2024-05-15T12:24:14.794 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-15T12:24:14.794 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} --owner scheduled_rzarzynski@teuthology --name rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi --archive /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707109 --verbose -- /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707109/orig.config.yaml 2024-05-15T12:24:14.838 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-15T12:24:14.924 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707109 branch: wip-osd-ec-partial-reads description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} email: null first_in_suite: false job_id: '7707109' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-osd-ec-partial-reads ceph: conf: client.2: debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms inject socket failures: 2500 ms tcp read timeout: 5 osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: min_in: 4 workunit: branch: wip-osd-ec-partial-reads sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 owner: scheduled_rzarzynski@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 8148 sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-osd-ec-partial-reads suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 targets: smithi027.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIVYUlhD9dqWRehm4Q46t1Kk7sxw0EQbzqqTnMx+36BFfa6KGUui96G6YBRSKXjGnhrX7oJQFjQaiPcv1Fvg01s= smithi120.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFuIH7kElwDHODlT/U2ii20ssku7k+Dlhp7Zk81sm9NOThO8WXeOC9rd+th2GofnTY+X4Guzw70dftyZK/zqeZ4= smithi177.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJSPjjHqUwsGcplWj9kv5whuGv6DCIPW7AoVbpKVSJfWctMFTZLktyilQaz8si1tBzwk1wAcWxORKGVom4prL+g= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: quincy exclude_packages: - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - ceph-base-debuginfo - ceph-common-debuginfo - ceph-immutable-object-cache-debuginfo - ceph-radosgw-debuginfo - ceph-test-debuginfo - ceph-base-debuginfo - ceph-mgr-debuginfo - ceph-mds-debuginfo - ceph-mon-debuginfo - ceph-osd-debuginfo - ceph-fuse-debuginfo - librados-devel-debuginfo - libcephfs2-debuginfo - librados2-debuginfo - librbd1-debuginfo - python3-cephfs-debuginfo - python3-rados-debuginfo - python3-rbd-debuginfo - python3-rgw-debuginfo - rbd-fuse-debuginfo - rbd-mirror-debuginfo - rbd-nbd-debuginfo - ceph-volume - cephadm: conf: mon: auth allow insecure global id reclaim: true - thrashosds: chance_pgnum_grow: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-15_11:53:37 tube: smithi user: rzarzynski verbose: false verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-15T12:24:14.924 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa; will attempt to use it 2024-05-15T12:24:14.925 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa/tasks 2024-05-15T12:24:14.925 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-15T12:24:14.927 INFO:teuthology.task.internal:Saving configuration 2024-05-15T12:24:14.944 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-15T12:24:14.950 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-15T12:24:14.973 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707109', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-15 12:17:44.501041', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIVYUlhD9dqWRehm4Q46t1Kk7sxw0EQbzqqTnMx+36BFfa6KGUui96G6YBRSKXjGnhrX7oJQFjQaiPcv1Fvg01s='} 2024-05-15T12:24:14.997 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi120.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707109', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-15 12:17:44.497013', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFuIH7kElwDHODlT/U2ii20ssku7k+Dlhp7Zk81sm9NOThO8WXeOC9rd+th2GofnTY+X4Guzw70dftyZK/zqeZ4='} 2024-05-15T12:24:15.019 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707109', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-15 12:17:44.499480', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJSPjjHqUwsGcplWj9kv5whuGv6DCIPW7AoVbpKVSJfWctMFTZLktyilQaz8si1tBzwk1wAcWxORKGVom4prL+g='} 2024-05-15T12:24:15.019 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-15T12:24:15.024 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-05-15T12:24:15.025 INFO:teuthology.task.internal:roles: ubuntu@smithi120.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-05-15T12:24:15.025 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-05-15T12:24:15.025 INFO:teuthology.run_tasks:Running task console_log... 2024-05-15T12:24:15.129 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdfc2bcc160>, signals=[15]) 2024-05-15T12:24:15.130 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-15T12:24:15.137 INFO:teuthology.task.internal:Opening connections... 2024-05-15T12:24:15.137 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi027.front.sepia.ceph.com 2024-05-15T12:24:15.140 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:24:15.228 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi120.front.sepia.ceph.com 2024-05-15T12:24:15.228 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:24:15.310 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi177.front.sepia.ceph.com 2024-05-15T12:24:15.311 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:24:15.398 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-15T12:24:15.405 DEBUG:teuthology.orchestra.run.smithi027:> uname -m 2024-05-15T12:24:15.429 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2024-05-15T12:24:15.429 DEBUG:teuthology.orchestra.run.smithi027:> cat /etc/os-release 2024-05-15T12:24:15.492 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="CentOS Stream" 2024-05-15T12:24:15.492 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="8" 2024-05-15T12:24:15.492 INFO:teuthology.orchestra.run.smithi027.stdout:ID="centos" 2024-05-15T12:24:15.492 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="rhel fedora" 2024-05-15T12:24:15.492 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="8" 2024-05-15T12:24:15.492 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el8" 2024-05-15T12:24:15.493 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="CentOS Stream 8" 2024-05-15T12:24:15.493 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31" 2024-05-15T12:24:15.493 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-05-15T12:24:15.493 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://centos.org/" 2024-05-15T12:24:15.493 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-15T12:24:15.493 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-05-15T12:24:15.493 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-15T12:24:15.494 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2024-05-15T12:24:15.521 DEBUG:teuthology.orchestra.run.smithi120:> uname -m 2024-05-15T12:24:15.543 INFO:teuthology.orchestra.run.smithi120.stdout:x86_64 2024-05-15T12:24:15.543 DEBUG:teuthology.orchestra.run.smithi120:> cat /etc/os-release 2024-05-15T12:24:15.605 INFO:teuthology.orchestra.run.smithi120.stdout:NAME="CentOS Stream" 2024-05-15T12:24:15.605 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION="8" 2024-05-15T12:24:15.605 INFO:teuthology.orchestra.run.smithi120.stdout:ID="centos" 2024-05-15T12:24:15.605 INFO:teuthology.orchestra.run.smithi120.stdout:ID_LIKE="rhel fedora" 2024-05-15T12:24:15.605 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_ID="8" 2024-05-15T12:24:15.606 INFO:teuthology.orchestra.run.smithi120.stdout:PLATFORM_ID="platform:el8" 2024-05-15T12:24:15.606 INFO:teuthology.orchestra.run.smithi120.stdout:PRETTY_NAME="CentOS Stream 8" 2024-05-15T12:24:15.606 INFO:teuthology.orchestra.run.smithi120.stdout:ANSI_COLOR="0;31" 2024-05-15T12:24:15.606 INFO:teuthology.orchestra.run.smithi120.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-05-15T12:24:15.606 INFO:teuthology.orchestra.run.smithi120.stdout:HOME_URL="https://centos.org/" 2024-05-15T12:24:15.606 INFO:teuthology.orchestra.run.smithi120.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-15T12:24:15.606 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-05-15T12:24:15.606 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-15T12:24:15.607 INFO:teuthology.lock.ops:Updating smithi120.front.sepia.ceph.com on lock server 2024-05-15T12:24:15.629 DEBUG:teuthology.orchestra.run.smithi177:> uname -m 2024-05-15T12:24:15.651 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2024-05-15T12:24:15.651 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2024-05-15T12:24:15.712 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="CentOS Stream" 2024-05-15T12:24:15.712 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="8" 2024-05-15T12:24:15.712 INFO:teuthology.orchestra.run.smithi177.stdout:ID="centos" 2024-05-15T12:24:15.713 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="rhel fedora" 2024-05-15T12:24:15.713 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="8" 2024-05-15T12:24:15.713 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el8" 2024-05-15T12:24:15.713 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="CentOS Stream 8" 2024-05-15T12:24:15.713 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31" 2024-05-15T12:24:15.713 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-05-15T12:24:15.713 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://centos.org/" 2024-05-15T12:24:15.713 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-15T12:24:15.713 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-05-15T12:24:15.713 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-15T12:24:15.713 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2024-05-15T12:24:15.737 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-15T12:24:15.746 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-15T12:24:15.750 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-15T12:24:15.751 DEBUG:teuthology.orchestra.run.smithi027:> test '!' -e /home/ubuntu/cephtest 2024-05-15T12:24:15.753 DEBUG:teuthology.orchestra.run.smithi120:> test '!' -e /home/ubuntu/cephtest 2024-05-15T12:24:15.755 DEBUG:teuthology.orchestra.run.smithi177:> test '!' -e /home/ubuntu/cephtest 2024-05-15T12:24:15.775 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-15T12:24:15.780 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-15T12:24:15.780 DEBUG:teuthology.orchestra.run.smithi027:> test -z $(ls -A /var/lib/ceph) 2024-05-15T12:24:15.815 DEBUG:teuthology.orchestra.run.smithi120:> test -z $(ls -A /var/lib/ceph) 2024-05-15T12:24:15.817 DEBUG:teuthology.orchestra.run.smithi177:> test -z $(ls -A /var/lib/ceph) 2024-05-15T12:24:15.868 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-15T12:24:15.948 INFO:teuthology.run_tasks:Running task kernel... 2024-05-15T12:24:15.961 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-15T12:24:15.961 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-15T12:24:15.962 DEBUG:teuthology.orchestra.run.smithi027:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-15T12:24:15.962 DEBUG:teuthology.orchestra.run.smithi120:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-15T12:24:15.963 DEBUG:teuthology.orchestra.run.smithi177:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-15T12:24:15.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:24:15.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:24:15.984 DEBUG:teuthology.orchestra.run.smithi120:> uname -r 2024-05-15T12:24:15.984 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-05-15T12:24:15.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:24:15.985 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-05-15T12:24:16.045 INFO:teuthology.orchestra.run.smithi177.stdout:4.18.0-489.el8.x86_64 2024-05-15T12:24:16.045 INFO:teuthology.task.kernel:Running kernel on smithi177: 4.18.0-489.el8.x86_64 2024-05-15T12:24:16.045 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum install -y kernel 2024-05-15T12:24:16.046 INFO:teuthology.orchestra.run.smithi120.stdout:4.18.0-489.el8.x86_64 2024-05-15T12:24:16.046 INFO:teuthology.task.kernel:Running kernel on smithi120: 4.18.0-489.el8.x86_64 2024-05-15T12:24:16.046 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum install -y kernel 2024-05-15T12:24:16.048 INFO:teuthology.orchestra.run.smithi027.stdout:4.18.0-489.el8.x86_64 2024-05-15T12:24:16.048 INFO:teuthology.task.kernel:Running kernel on smithi027: 4.18.0-489.el8.x86_64 2024-05-15T12:24:16.048 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum install -y kernel 2024-05-15T12:24:19.756 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 8 - AppStream 23 MB/s | 28 MB 00:01 2024-05-15T12:24:20.072 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - AppStream 21 MB/s | 28 MB 00:01 2024-05-15T12:24:20.822 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 8 - AppStream 13 MB/s | 28 MB 00:02 2024-05-15T12:24:26.948 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 8 - BaseOS 4.8 MB/s | 10 MB 00:02 2024-05-15T12:24:27.906 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - BaseOS 4.2 MB/s | 10 MB 00:02 2024-05-15T12:24:28.205 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 8 - BaseOS 4.4 MB/s | 10 MB 00:02 2024-05-15T12:24:28.939 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 8 - Extras 49 kB/s | 18 kB 00:00 2024-05-15T12:24:29.215 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 8 - Extras common packages 46 kB/s | 7.7 kB 00:00 2024-05-15T12:24:29.500 INFO:teuthology.orchestra.run.smithi177.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2024-05-15T12:24:30.156 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - Extras 35 kB/s | 18 kB 00:00 2024-05-15T12:24:30.322 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 8 - Extras 36 kB/s | 18 kB 00:00 2024-05-15T12:24:30.548 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - Extras common packages 28 kB/s | 7.7 kB 00:00 2024-05-15T12:24:30.573 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 8 - Extras common packages 54 kB/s | 7.7 kB 00:00 2024-05-15T12:24:30.784 INFO:teuthology.orchestra.run.smithi027.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-05-15T12:24:30.792 INFO:teuthology.orchestra.run.smithi120.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-05-15T12:24:32.309 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 9.5 MB/s | 14 MB 00:01 2024-05-15T12:24:36.718 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 656 kB/s | 24 kB 00:00 2024-05-15T12:24:39.956 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-05-15T12:24:39.957 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-05-15T12:24:40.081 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-15T12:24:40.082 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:24:40.082 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-05-15T12:24:40.082 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:24:40.083 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-05-15T12:24:40.083 INFO:teuthology.orchestra.run.smithi027.stdout: kernel x86_64 4.18.0-552.1.1.el8 baseos 10 M 2024-05-15T12:24:40.083 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-05-15T12:24:40.083 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-core x86_64 4.18.0-552.1.1.el8 baseos 44 M 2024-05-15T12:24:40.083 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules x86_64 4.18.0-552.1.1.el8 baseos 36 M 2024-05-15T12:24:40.083 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:24:40.083 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-05-15T12:24:40.083 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:24:40.083 INFO:teuthology.orchestra.run.smithi027.stdout:Install 3 Packages 2024-05-15T12:24:40.083 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:24:40.084 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 90 M 2024-05-15T12:24:40.084 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 96 M 2024-05-15T12:24:40.084 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-05-15T12:24:41.444 INFO:teuthology.orchestra.run.smithi027.stdout:(1/3): kernel-4.18.0-552.1.1.el8.x86_64.rpm 8.0 MB/s | 10 MB 00:01 2024-05-15T12:24:42.747 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 14 MB 00:13 2024-05-15T12:24:42.770 INFO:teuthology.orchestra.run.smithi027.stdout:(2/3): kernel-modules-4.18.0-552.1.1.el8.x86_64 14 MB/s | 36 MB 00:02 2024-05-15T12:24:43.278 INFO:teuthology.orchestra.run.smithi027.stdout:(3/3): kernel-core-4.18.0-552.1.1.el8.x86_64.rp 14 MB/s | 44 MB 00:03 2024-05-15T12:24:43.278 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:24:43.278 INFO:teuthology.orchestra.run.smithi027.stdout:Total 28 MB/s | 90 MB 00:03 2024-05-15T12:24:43.993 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-05-15T12:24:44.269 INFO:teuthology.orchestra.run.smithi120.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 14 MB 00:13 2024-05-15T12:24:44.439 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-05-15T12:24:44.439 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-05-15T12:24:45.760 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-05-15T12:24:45.781 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-05-15T12:24:46.982 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 413 kB/s | 24 kB 00:00 2024-05-15T12:24:48.515 INFO:teuthology.orchestra.run.smithi120.stdout:lab-extras 402 kB/s | 24 kB 00:00 2024-05-15T12:24:48.727 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-05-15T12:24:49.286 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-core-4.18.0-552.1.1.el8.x86_64 1/3 2024-05-15T12:24:50.103 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-05-15T12:24:50.104 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-05-15T12:24:50.216 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-05-15T12:24:50.216 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:24:50.217 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-05-15T12:24:50.217 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:24:50.217 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-05-15T12:24:50.217 INFO:teuthology.orchestra.run.smithi177.stdout: kernel x86_64 4.18.0-552.1.1.el8 baseos 10 M 2024-05-15T12:24:50.217 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-05-15T12:24:50.217 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-core x86_64 4.18.0-552.1.1.el8 baseos 44 M 2024-05-15T12:24:50.217 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules x86_64 4.18.0-552.1.1.el8 baseos 36 M 2024-05-15T12:24:50.217 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:24:50.217 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-05-15T12:24:50.217 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:24:50.217 INFO:teuthology.orchestra.run.smithi177.stdout:Install 3 Packages 2024-05-15T12:24:50.217 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:24:50.218 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 90 M 2024-05-15T12:24:50.218 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 96 M 2024-05-15T12:24:50.218 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-05-15T12:24:51.075 INFO:teuthology.orchestra.run.smithi177.stdout:(1/3): kernel-4.18.0-552.1.1.el8.x86_64.rpm 16 MB/s | 10 MB 00:00 2024-05-15T12:24:51.571 INFO:teuthology.orchestra.run.smithi120.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-05-15T12:24:51.571 INFO:teuthology.orchestra.run.smithi120.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-05-15T12:24:51.698 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-05-15T12:24:51.699 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:24:51.699 INFO:teuthology.orchestra.run.smithi120.stdout: Package Architecture Version Repository Size 2024-05-15T12:24:51.699 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:24:51.699 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-05-15T12:24:51.699 INFO:teuthology.orchestra.run.smithi120.stdout: kernel x86_64 4.18.0-552.1.1.el8 baseos 10 M 2024-05-15T12:24:51.699 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-05-15T12:24:51.699 INFO:teuthology.orchestra.run.smithi120.stdout: kernel-core x86_64 4.18.0-552.1.1.el8 baseos 44 M 2024-05-15T12:24:51.700 INFO:teuthology.orchestra.run.smithi120.stdout: kernel-modules x86_64 4.18.0-552.1.1.el8 baseos 36 M 2024-05-15T12:24:51.700 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:24:51.700 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-05-15T12:24:51.700 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:24:51.700 INFO:teuthology.orchestra.run.smithi120.stdout:Install 3 Packages 2024-05-15T12:24:51.700 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:24:51.700 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 90 M 2024-05-15T12:24:51.701 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 96 M 2024-05-15T12:24:51.701 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-05-15T12:24:52.117 INFO:teuthology.orchestra.run.smithi177.stdout:(2/3): kernel-core-4.18.0-552.1.1.el8.x86_64.rp 26 MB/s | 44 MB 00:01 2024-05-15T12:24:52.867 INFO:teuthology.orchestra.run.smithi120.stdout:(1/3): kernel-4.18.0-552.1.1.el8.x86_64.rpm 9.4 MB/s | 10 MB 00:01 2024-05-15T12:24:53.012 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-core-4.18.0-552.1.1.el8.x86_64 1/3 2024-05-15T12:24:53.452 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-modules-4.18.0-552.1.1.el8.x86_64 2/3 2024-05-15T12:24:53.550 INFO:teuthology.orchestra.run.smithi177.stdout:(3/3): kernel-modules-4.18.0-552.1.1.el8.x86_64 12 MB/s | 36 MB 00:03 2024-05-15T12:24:53.550 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:24:53.550 INFO:teuthology.orchestra.run.smithi177.stdout:Total 27 MB/s | 90 MB 00:03 2024-05-15T12:24:54.233 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-05-15T12:24:54.467 INFO:teuthology.orchestra.run.smithi120.stdout:(2/3): kernel-modules-4.18.0-552.1.1.el8.x86_64 13 MB/s | 36 MB 00:02 2024-05-15T12:24:54.682 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-05-15T12:24:54.682 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-05-15T12:24:55.218 INFO:teuthology.orchestra.run.smithi120.stdout:(3/3): kernel-core-4.18.0-552.1.1.el8.x86_64.rp 13 MB/s | 44 MB 00:03 2024-05-15T12:24:55.218 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:24:55.218 INFO:teuthology.orchestra.run.smithi120.stdout:Total 26 MB/s | 90 MB 00:03 2024-05-15T12:24:55.900 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-05-15T12:24:55.933 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-05-15T12:24:55.949 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-05-15T12:24:56.337 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-05-15T12:24:56.338 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-05-15T12:24:57.137 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-4.18.0-552.1.1.el8.x86_64 2/3 2024-05-15T12:24:57.437 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T12:24:57.573 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-05-15T12:24:57.592 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-05-15T12:24:58.756 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-05-15T12:24:59.244 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-core-4.18.0-552.1.1.el8.x86_64 1/3 2024-05-15T12:25:00.408 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-05-15T12:25:00.922 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : kernel-core-4.18.0-552.1.1.el8.x86_64 1/3 2024-05-15T12:25:02.813 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-core-4.18.0-552.1.1.el8.x86_64 1/3 2024-05-15T12:25:03.159 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-modules-4.18.0-552.1.1.el8.x86_64 2/3 2024-05-15T12:25:04.474 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: kernel-core-4.18.0-552.1.1.el8.x86_64 1/3 2024-05-15T12:25:04.855 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : kernel-modules-4.18.0-552.1.1.el8.x86_64 2/3 2024-05-15T12:25:06.601 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-4.18.0-552.1.1.el8.x86_64 2/3 2024-05-15T12:25:06.900 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T12:25:08.285 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: kernel-modules-4.18.0-552.1.1.el8.x86_64 2/3 2024-05-15T12:25:08.569 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : kernel-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T12:25:20.824 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-core-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T12:25:20.849 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T12:25:21.216 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T12:25:21.216 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-4.18.0-552.1.1.el8.x86_64 1/3 2024-05-15T12:25:21.217 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-core-4.18.0-552.1.1.el8.x86_64 2/3 2024-05-15T12:25:21.727 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-modules-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T12:25:21.727 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:25:21.727 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-05-15T12:25:21.727 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:21.727 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-core-4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:21.727 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:21.727 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:25:21.728 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-15T12:25:21.909 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel | sort -rV | head -n 1 2024-05-15T12:25:22.137 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:22.138 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:22.138 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:22.139 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-15T12:25:22.139 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-15T12:25:22.139 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-05-15T12:25:22.139 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi027.front.sepia.ceph.com, path=None, version=distro) 2024-05-15T12:25:22.139 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel | sort -rV | head -n 1 2024-05-15T12:25:22.359 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:22.359 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:22.359 DEBUG:teuthology.orchestra.run.smithi027:> sudo rpm -qi grub2-tools 2024-05-15T12:25:22.394 INFO:teuthology.orchestra.run.smithi027.stdout:Name : grub2-tools 2024-05-15T12:25:22.394 INFO:teuthology.orchestra.run.smithi027.stdout:Epoch : 1 2024-05-15T12:25:22.394 INFO:teuthology.orchestra.run.smithi027.stdout:Version : 2.02 2024-05-15T12:25:22.394 INFO:teuthology.orchestra.run.smithi027.stdout:Release : 129.el8 2024-05-15T12:25:22.394 INFO:teuthology.orchestra.run.smithi027.stdout:Architecture: x86_64 2024-05-15T12:25:22.394 INFO:teuthology.orchestra.run.smithi027.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-05-15T12:25:22.394 INFO:teuthology.orchestra.run.smithi027.stdout:Group : System Environment/Base 2024-05-15T12:25:22.394 INFO:teuthology.orchestra.run.smithi027.stdout:Size : 9265586 2024-05-15T12:25:22.394 INFO:teuthology.orchestra.run.smithi027.stdout:License : GPLv3+ 2024-05-15T12:25:22.394 INFO:teuthology.orchestra.run.smithi027.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-05-15T12:25:22.395 INFO:teuthology.orchestra.run.smithi027.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-05-15T12:25:22.395 INFO:teuthology.orchestra.run.smithi027.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-05-15T12:25:22.395 INFO:teuthology.orchestra.run.smithi027.stdout:Build Host : kbuilder.bsys.centos.org 2024-05-15T12:25:22.395 INFO:teuthology.orchestra.run.smithi027.stdout:Relocations : (not relocatable) 2024-05-15T12:25:22.395 INFO:teuthology.orchestra.run.smithi027.stdout:Packager : CentOS BuildSystem 2024-05-15T12:25:22.395 INFO:teuthology.orchestra.run.smithi027.stdout:Vendor : CentOS 2024-05-15T12:25:22.395 INFO:teuthology.orchestra.run.smithi027.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-15T12:25:22.395 INFO:teuthology.orchestra.run.smithi027.stdout:Summary : Support tools for GRUB. 2024-05-15T12:25:22.395 INFO:teuthology.orchestra.run.smithi027.stdout:Description : 2024-05-15T12:25:22.395 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:25:22.395 INFO:teuthology.orchestra.run.smithi027.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-15T12:25:22.395 INFO:teuthology.orchestra.run.smithi027.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-15T12:25:22.395 INFO:teuthology.orchestra.run.smithi027.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-15T12:25:22.395 INFO:teuthology.orchestra.run.smithi027.stdout:hardware devices. 2024-05-15T12:25:22.395 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:25:22.396 INFO:teuthology.orchestra.run.smithi027.stdout:This subpackage provides tools for support of all platforms. 2024-05-15T12:25:22.396 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-15T12:25:22.397 INFO:teuthology.task.kernel:Updating grub on smithi027 to boot 4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:22.397 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-15T12:25:23.798 INFO:teuthology.orchestra.run.smithi027.stderr:Generating grub configuration file ... 2024-05-15T12:25:25.505 INFO:teuthology.orchestra.run.smithi027.stderr:done 2024-05-15T12:25:25.507 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2024-05-15T12:25:25.536 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.SPZGYaL23o 2024-05-15T12:25:25.537 DEBUG:teuthology.orchestra.run.smithi027:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.SPZGYaL23o 2024-05-15T12:25:25.609 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.SPZGYaL23o 2024-05-15T12:25:25.762 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.SPZGYaL23o is 6KB 2024-05-15T12:25:25.811 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.SPZGYaL23o 2024-05-15T12:25:25.829 DEBUG:teuthology.orchestra.run.smithi027:> sudo /bin/ls /boot/loader/entries || true 2024-05-15T12:25:25.904 INFO:teuthology.orchestra.run.smithi027.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-05-15T12:25:25.904 INFO:teuthology.orchestra.run.smithi027.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-05-15T12:25:25.904 INFO:teuthology.orchestra.run.smithi027.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-05-15T12:25:25.905 INFO:teuthology.orchestra.run.smithi027.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.1.1.el8.x86_64.conf 2024-05-15T12:25:25.906 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:26.103 DEBUG:teuthology.orchestra.run.smithi027:> sudo shutdown -r now 2024-05-15T12:25:28.952 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-core-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T12:25:28.976 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T12:25:29.321 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T12:25:29.322 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-4.18.0-552.1.1.el8.x86_64 1/3 2024-05-15T12:25:29.322 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-core-4.18.0-552.1.1.el8.x86_64 2/3 2024-05-15T12:25:29.819 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-modules-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T12:25:29.819 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:25:29.820 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-05-15T12:25:29.820 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:29.820 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-core-4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:29.820 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:29.820 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:25:29.820 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-05-15T12:25:29.962 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2024-05-15T12:25:30.183 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:30.183 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:30.183 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:30.184 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-15T12:25:30.184 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-15T12:25:30.184 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-05-15T12:25:30.184 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi177.front.sepia.ceph.com, path=None, version=distro) 2024-05-15T12:25:30.184 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2024-05-15T12:25:30.295 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: kernel-core-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T12:25:30.320 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: kernel-modules-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T12:25:30.395 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:30.395 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:30.395 DEBUG:teuthology.orchestra.run.smithi177:> sudo rpm -qi grub2-tools 2024-05-15T12:25:30.428 INFO:teuthology.orchestra.run.smithi177.stdout:Name : grub2-tools 2024-05-15T12:25:30.428 INFO:teuthology.orchestra.run.smithi177.stdout:Epoch : 1 2024-05-15T12:25:30.428 INFO:teuthology.orchestra.run.smithi177.stdout:Version : 2.02 2024-05-15T12:25:30.428 INFO:teuthology.orchestra.run.smithi177.stdout:Release : 129.el8 2024-05-15T12:25:30.428 INFO:teuthology.orchestra.run.smithi177.stdout:Architecture: x86_64 2024-05-15T12:25:30.428 INFO:teuthology.orchestra.run.smithi177.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-05-15T12:25:30.428 INFO:teuthology.orchestra.run.smithi177.stdout:Group : System Environment/Base 2024-05-15T12:25:30.428 INFO:teuthology.orchestra.run.smithi177.stdout:Size : 9265586 2024-05-15T12:25:30.428 INFO:teuthology.orchestra.run.smithi177.stdout:License : GPLv3+ 2024-05-15T12:25:30.428 INFO:teuthology.orchestra.run.smithi177.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-05-15T12:25:30.429 INFO:teuthology.orchestra.run.smithi177.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-05-15T12:25:30.429 INFO:teuthology.orchestra.run.smithi177.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-05-15T12:25:30.429 INFO:teuthology.orchestra.run.smithi177.stdout:Build Host : kbuilder.bsys.centos.org 2024-05-15T12:25:30.429 INFO:teuthology.orchestra.run.smithi177.stdout:Relocations : (not relocatable) 2024-05-15T12:25:30.429 INFO:teuthology.orchestra.run.smithi177.stdout:Packager : CentOS BuildSystem 2024-05-15T12:25:30.429 INFO:teuthology.orchestra.run.smithi177.stdout:Vendor : CentOS 2024-05-15T12:25:30.429 INFO:teuthology.orchestra.run.smithi177.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-15T12:25:30.429 INFO:teuthology.orchestra.run.smithi177.stdout:Summary : Support tools for GRUB. 2024-05-15T12:25:30.429 INFO:teuthology.orchestra.run.smithi177.stdout:Description : 2024-05-15T12:25:30.429 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:25:30.429 INFO:teuthology.orchestra.run.smithi177.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-15T12:25:30.429 INFO:teuthology.orchestra.run.smithi177.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-15T12:25:30.429 INFO:teuthology.orchestra.run.smithi177.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-15T12:25:30.429 INFO:teuthology.orchestra.run.smithi177.stdout:hardware devices. 2024-05-15T12:25:30.429 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:25:30.430 INFO:teuthology.orchestra.run.smithi177.stdout:This subpackage provides tools for support of all platforms. 2024-05-15T12:25:30.430 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-15T12:25:30.430 INFO:teuthology.task.kernel:Updating grub on smithi177 to boot 4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:30.431 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-15T12:25:30.653 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: kernel-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T12:25:30.653 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : kernel-4.18.0-552.1.1.el8.x86_64 1/3 2024-05-15T12:25:30.653 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : kernel-core-4.18.0-552.1.1.el8.x86_64 2/3 2024-05-15T12:25:31.422 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : kernel-modules-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T12:25:31.422 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:25:31.422 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-05-15T12:25:31.422 INFO:teuthology.orchestra.run.smithi120.stdout: kernel-4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:31.422 INFO:teuthology.orchestra.run.smithi120.stdout: kernel-core-4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:31.422 INFO:teuthology.orchestra.run.smithi120.stdout: kernel-modules-4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:31.422 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:25:31.422 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-05-15T12:25:31.603 DEBUG:teuthology.orchestra.run.smithi120:> rpm -q kernel | sort -rV | head -n 1 2024-05-15T12:25:31.824 INFO:teuthology.orchestra.run.smithi120.stdout:kernel-4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:31.824 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:31.824 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:31.825 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-15T12:25:31.825 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-15T12:25:31.825 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-05-15T12:25:31.825 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi120.front.sepia.ceph.com, path=None, version=distro) 2024-05-15T12:25:31.825 DEBUG:teuthology.orchestra.run.smithi120:> rpm -q kernel | sort -rV | head -n 1 2024-05-15T12:25:32.038 INFO:teuthology.orchestra.run.smithi120.stdout:kernel-4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:32.038 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:32.038 DEBUG:teuthology.orchestra.run.smithi120:> sudo rpm -qi grub2-tools 2024-05-15T12:25:32.041 INFO:teuthology.orchestra.run.smithi177.stderr:Generating grub configuration file ... 2024-05-15T12:25:32.074 INFO:teuthology.orchestra.run.smithi120.stdout:Name : grub2-tools 2024-05-15T12:25:32.074 INFO:teuthology.orchestra.run.smithi120.stdout:Epoch : 1 2024-05-15T12:25:32.074 INFO:teuthology.orchestra.run.smithi120.stdout:Version : 2.02 2024-05-15T12:25:32.074 INFO:teuthology.orchestra.run.smithi120.stdout:Release : 129.el8 2024-05-15T12:25:32.074 INFO:teuthology.orchestra.run.smithi120.stdout:Architecture: x86_64 2024-05-15T12:25:32.074 INFO:teuthology.orchestra.run.smithi120.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-05-15T12:25:32.074 INFO:teuthology.orchestra.run.smithi120.stdout:Group : System Environment/Base 2024-05-15T12:25:32.075 INFO:teuthology.orchestra.run.smithi120.stdout:Size : 9265586 2024-05-15T12:25:32.075 INFO:teuthology.orchestra.run.smithi120.stdout:License : GPLv3+ 2024-05-15T12:25:32.075 INFO:teuthology.orchestra.run.smithi120.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-05-15T12:25:32.075 INFO:teuthology.orchestra.run.smithi120.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-05-15T12:25:32.075 INFO:teuthology.orchestra.run.smithi120.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-05-15T12:25:32.075 INFO:teuthology.orchestra.run.smithi120.stdout:Build Host : kbuilder.bsys.centos.org 2024-05-15T12:25:32.075 INFO:teuthology.orchestra.run.smithi120.stdout:Relocations : (not relocatable) 2024-05-15T12:25:32.075 INFO:teuthology.orchestra.run.smithi120.stdout:Packager : CentOS BuildSystem 2024-05-15T12:25:32.075 INFO:teuthology.orchestra.run.smithi120.stdout:Vendor : CentOS 2024-05-15T12:25:32.075 INFO:teuthology.orchestra.run.smithi120.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-15T12:25:32.075 INFO:teuthology.orchestra.run.smithi120.stdout:Summary : Support tools for GRUB. 2024-05-15T12:25:32.075 INFO:teuthology.orchestra.run.smithi120.stdout:Description : 2024-05-15T12:25:32.075 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:25:32.075 INFO:teuthology.orchestra.run.smithi120.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-15T12:25:32.076 INFO:teuthology.orchestra.run.smithi120.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-15T12:25:32.076 INFO:teuthology.orchestra.run.smithi120.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-15T12:25:32.076 INFO:teuthology.orchestra.run.smithi120.stdout:hardware devices. 2024-05-15T12:25:32.076 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:25:32.076 INFO:teuthology.orchestra.run.smithi120.stdout:This subpackage provides tools for support of all platforms. 2024-05-15T12:25:32.077 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-15T12:25:32.077 INFO:teuthology.task.kernel:Updating grub on smithi120 to boot 4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:32.077 DEBUG:teuthology.orchestra.run.smithi120:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-15T12:25:33.374 INFO:teuthology.orchestra.run.smithi120.stderr:Generating grub configuration file ... 2024-05-15T12:25:33.916 INFO:teuthology.orchestra.run.smithi177.stderr:done 2024-05-15T12:25:33.918 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-05-15T12:25:33.941 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.xz9hwe7gvB 2024-05-15T12:25:33.941 DEBUG:teuthology.orchestra.run.smithi177:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xz9hwe7gvB 2024-05-15T12:25:34.026 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.xz9hwe7gvB 2024-05-15T12:25:34.167 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.xz9hwe7gvB is 6KB 2024-05-15T12:25:34.215 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.xz9hwe7gvB 2024-05-15T12:25:34.232 DEBUG:teuthology.orchestra.run.smithi177:> sudo /bin/ls /boot/loader/entries || true 2024-05-15T12:25:34.299 INFO:teuthology.orchestra.run.smithi177.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-05-15T12:25:34.300 INFO:teuthology.orchestra.run.smithi177.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-05-15T12:25:34.300 INFO:teuthology.orchestra.run.smithi177.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-05-15T12:25:34.300 INFO:teuthology.orchestra.run.smithi177.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.1.1.el8.x86_64.conf 2024-05-15T12:25:34.301 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:34.477 DEBUG:teuthology.orchestra.run.smithi177:> sudo shutdown -r now 2024-05-15T12:25:35.294 INFO:teuthology.orchestra.run.smithi120.stderr:done 2024-05-15T12:25:35.295 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2024-05-15T12:25:35.328 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.OYHfHaQOb6 2024-05-15T12:25:35.328 DEBUG:teuthology.orchestra.run.smithi120:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.OYHfHaQOb6 2024-05-15T12:25:35.402 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.OYHfHaQOb6 2024-05-15T12:25:35.569 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.OYHfHaQOb6 is 6KB 2024-05-15T12:25:35.618 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.OYHfHaQOb6 2024-05-15T12:25:35.636 DEBUG:teuthology.orchestra.run.smithi120:> sudo /bin/ls /boot/loader/entries || true 2024-05-15T12:25:35.701 INFO:teuthology.orchestra.run.smithi120.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-05-15T12:25:35.702 INFO:teuthology.orchestra.run.smithi120.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-05-15T12:25:35.702 INFO:teuthology.orchestra.run.smithi120.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-05-15T12:25:35.702 INFO:teuthology.orchestra.run.smithi120.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.1.1.el8.x86_64.conf 2024-05-15T12:25:35.703 DEBUG:teuthology.orchestra.run.smithi120:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.1.1.el8.x86_64 2024-05-15T12:25:35.854 DEBUG:teuthology.orchestra.run.smithi120:> sudo shutdown -r now 2024-05-15T12:25:56.113 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-05-15T12:25:56.114 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-05-15T12:25:56.115 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:26:04.486 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-05-15T12:26:04.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-05-15T12:26:04.487 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:26:05.858 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-05-15T12:26:05.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-05-15T12:26:05.859 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:26:14.431 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-05-15T12:26:22.878 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-05-15T12:26:23.433 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-05-15T12:26:23.433 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:26:24.414 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.120 2024-05-15T12:26:26.494 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-05-15T12:26:31.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-05-15T12:26:31.883 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:26:33.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-05-15T12:26:33.416 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:26:34.942 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-05-15T12:26:36.482 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.120 2024-05-15T12:26:38.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-05-15T12:26:38.497 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:26:41.566 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-05-15T12:26:46.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-05-15T12:26:46.947 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:26:48.484 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-05-15T12:26:48.485 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:26:56.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-05-15T12:26:56.575 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:26:59.646 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-05-15T12:27:05.374 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-05-15T12:27:17.658 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-05-15T12:27:17.659 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:27:20.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-05-15T12:27:20.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:27:21.023 DEBUG:teuthology.orchestra.run.smithi120:> true 2024-05-15T12:27:21.193 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-05-15T12:27:21.366 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-05-15T12:27:21.366 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-552.1.1.el8.x86_64"... 2024-05-15T12:27:21.366 DEBUG:teuthology.orchestra.run.smithi120:> uname -r 2024-05-15T12:27:21.383 INFO:teuthology.orchestra.run.smithi120.stdout:4.18.0-552.1.1.el8.x86_64 2024-05-15T12:27:21.383 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.1.1.el8.x86_64 vs 4.18.0-552.1.1.el8.x86_64 2024-05-15T12:27:21.383 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-15T12:27:21.383 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-15T12:27:22.384 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-05-15T12:27:22.384 DEBUG:teuthology.orchestra.run.smithi120:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-15T12:27:22.465 INFO:teuthology.orchestra.run.smithi120.stdout:ttyS1 2024-05-15T12:27:22.481 DEBUG:teuthology.parallel:result is None 2024-05-15T12:27:22.487 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-05-15T12:27:22.487 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.18.0-552.1.1.el8.x86_64"... 2024-05-15T12:27:22.487 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-05-15T12:27:22.605 INFO:teuthology.orchestra.run.smithi177.stdout:4.18.0-552.1.1.el8.x86_64 2024-05-15T12:27:22.605 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.1.1.el8.x86_64 vs 4.18.0-552.1.1.el8.x86_64 2024-05-15T12:27:22.605 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-15T12:27:22.606 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-15T12:27:23.606 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-05-15T12:27:23.606 DEBUG:teuthology.orchestra.run.smithi177:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-15T12:27:23.660 INFO:teuthology.orchestra.run.smithi177.stdout:ttyS1 2024-05-15T12:27:23.675 DEBUG:teuthology.parallel:result is None 2024-05-15T12:27:36.098 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-05-15T12:27:57.100 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-05-15T12:27:57.101 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:27:58.056 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-05-15T12:27:58.903 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-05-15T12:27:58.903 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-552.1.1.el8.x86_64"... 2024-05-15T12:27:58.903 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-05-15T12:27:58.920 INFO:teuthology.orchestra.run.smithi027.stdout:4.18.0-552.1.1.el8.x86_64 2024-05-15T12:27:58.920 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.1.1.el8.x86_64 vs 4.18.0-552.1.1.el8.x86_64 2024-05-15T12:27:58.920 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-15T12:27:58.920 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-15T12:27:59.921 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-15T12:27:59.922 DEBUG:teuthology.orchestra.run.smithi027:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-15T12:27:59.993 INFO:teuthology.orchestra.run.smithi027.stdout:ttyS1 2024-05-15T12:28:00.006 DEBUG:teuthology.parallel:result is None 2024-05-15T12:28:00.006 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-15T12:28:00.016 INFO:teuthology.task.internal:Creating test directory... 2024-05-15T12:28:00.016 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-15T12:28:00.019 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-15T12:28:00.021 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-15T12:28:00.043 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-15T12:28:00.050 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-15T12:28:00.131 INFO:teuthology.task.internal:Creating archive directory... 2024-05-15T12:28:00.132 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-15T12:28:00.134 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-15T12:28:00.137 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-15T12:28:00.178 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-15T12:28:00.184 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-15T12:28:00.185 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-15T12:28:00.211 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-15T12:28:00.214 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-15T12:28:00.245 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:28:00.249 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:28:00.254 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:28:00.255 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:28:00.260 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:28:00.265 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:28:00.267 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-15T12:28:00.274 INFO:teuthology.task.internal:Configuring sudo... 2024-05-15T12:28:00.274 DEBUG:teuthology.orchestra.run.smithi027:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-15T12:28:00.298 DEBUG:teuthology.orchestra.run.smithi120:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-15T12:28:00.304 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-15T12:28:00.337 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-15T12:28:00.346 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-15T12:28:00.346 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-15T12:28:00.366 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-15T12:28:00.373 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-15T12:28:00.396 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T12:28:00.468 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T12:28:00.519 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T12:28:00.601 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T12:28:00.647 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-15T12:28:00.647 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-15T12:28:00.714 DEBUG:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T12:28:00.796 DEBUG:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T12:28:00.852 DEBUG:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T12:28:00.937 DEBUG:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T12:28:00.982 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-05-15T12:28:00.982 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-15T12:28:01.050 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T12:28:01.099 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T12:28:01.155 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T12:28:01.245 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T12:28:01.288 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-05-15T12:28:01.288 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-15T12:28:01.358 DEBUG:teuthology.orchestra.run.smithi027:> sudo service rsyslog restart 2024-05-15T12:28:01.360 DEBUG:teuthology.orchestra.run.smithi120:> sudo service rsyslog restart 2024-05-15T12:28:01.362 DEBUG:teuthology.orchestra.run.smithi177:> sudo service rsyslog restart 2024-05-15T12:28:01.415 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-15T12:28:01.441 INFO:teuthology.orchestra.run.smithi120.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-15T12:28:01.461 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-15T12:28:01.791 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-15T12:28:01.801 INFO:teuthology.task.internal:Starting timer... 2024-05-15T12:28:01.802 INFO:teuthology.run_tasks:Running task pcp... 2024-05-15T12:28:01.812 INFO:teuthology.run_tasks:Running task selinux... 2024-05-15T12:28:01.820 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-05-15T12:28:01.820 DEBUG:teuthology.orchestra.run.smithi027:> sudo service auditd rotate 2024-05-15T12:28:01.911 INFO:teuthology.orchestra.run.smithi027.stdout:Rotating logs: 2024-05-15T12:28:01.913 DEBUG:teuthology.orchestra.run.smithi120:> sudo service auditd rotate 2024-05-15T12:28:02.012 INFO:teuthology.orchestra.run.smithi120.stdout:Rotating logs: 2024-05-15T12:28:02.014 DEBUG:teuthology.orchestra.run.smithi177:> sudo service auditd rotate 2024-05-15T12:28:02.091 INFO:teuthology.orchestra.run.smithi177.stdout:Rotating logs: 2024-05-15T12:28:02.093 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-15T12:28:02.094 DEBUG:teuthology.orchestra.run.smithi027:> /usr/sbin/getenforce 2024-05-15T12:28:02.148 INFO:teuthology.orchestra.run.smithi027.stdout:Permissive 2024-05-15T12:28:02.149 DEBUG:teuthology.orchestra.run.smithi120:> /usr/sbin/getenforce 2024-05-15T12:28:02.213 INFO:teuthology.orchestra.run.smithi120.stdout:Permissive 2024-05-15T12:28:02.213 DEBUG:teuthology.orchestra.run.smithi177:> /usr/sbin/getenforce 2024-05-15T12:28:02.257 INFO:teuthology.orchestra.run.smithi177.stdout:Permissive 2024-05-15T12:28:02.258 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi027.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi120.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi177.front.sepia.ceph.com': 'permissive'} 2024-05-15T12:28:02.258 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-15T12:28:02.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:28:02.290 DEBUG:teuthology.orchestra.run.smithi120:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-15T12:28:02.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:28:02.318 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-15T12:28:02.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:28:02.346 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-15T12:28:02.346 DEBUG:teuthology.orchestra.run.smithi027:> sudo /usr/sbin/setenforce permissive 2024-05-15T12:28:02.379 DEBUG:teuthology.orchestra.run.smithi120:> sudo /usr/sbin/setenforce permissive 2024-05-15T12:28:02.406 DEBUG:teuthology.orchestra.run.smithi177:> sudo /usr/sbin/setenforce permissive 2024-05-15T12:28:02.432 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-15T12:28:02.448 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-05-15T12:28:02.453 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-05-15T12:28:02.591 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-15T12:28:02.609 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-15T12:28:02.610 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi027.front.sepia.ceph.com,smithi120.front.sepia.ceph.com,smithi177.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-15T12:34:05.079 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi027.front.sepia.ceph.com'), Remote(name='ubuntu@smithi120.front.sepia.ceph.com'), Remote(name='ubuntu@smithi177.front.sepia.ceph.com')] 2024-05-15T12:34:05.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-05-15T12:34:05.082 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:34:05.155 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-05-15T12:34:05.226 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-05-15T12:34:05.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-05-15T12:34:05.226 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:34:05.304 DEBUG:teuthology.orchestra.run.smithi120:> true 2024-05-15T12:34:05.331 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-05-15T12:34:05.331 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-05-15T12:34:05.332 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:34:05.407 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-05-15T12:34:05.477 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-05-15T12:34:05.478 INFO:teuthology.run_tasks:Running task clock... 2024-05-15T12:34:05.494 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-15T12:34:05.494 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-15T12:34:05.495 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T12:34:05.497 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-15T12:34:05.497 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T12:34:05.499 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-15T12:34:05.499 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T12:34:05.531 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-15T12:34:05.533 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-15T12:34:05.545 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-15T12:34:05.546 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-15T12:34:05.551 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-15T12:34:05.565 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-15T12:34:05.617 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: ntpd: command not found 2024-05-15T12:34:05.626 INFO:teuthology.orchestra.run.smithi120.stderr:sudo: ntpd: command not found 2024-05-15T12:34:05.629 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: ntpd: command not found 2024-05-15T12:34:05.629 INFO:teuthology.orchestra.run.smithi027.stdout:506 Cannot talk to daemon 2024-05-15T12:34:05.638 INFO:teuthology.orchestra.run.smithi120.stdout:506 Cannot talk to daemon 2024-05-15T12:34:05.639 INFO:teuthology.orchestra.run.smithi177.stdout:506 Cannot talk to daemon 2024-05-15T12:34:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-15T12:34:05.651 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-15T12:34:05.653 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-15T12:34:05.658 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-15T12:34:05.664 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-15T12:34:05.666 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-15T12:34:05.733 INFO:teuthology.orchestra.run.smithi120.stderr:bash: ntpq: command not found 2024-05-15T12:34:05.738 INFO:teuthology.orchestra.run.smithi120.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-15T12:34:05.738 INFO:teuthology.orchestra.run.smithi120.stdout:=============================================================================== 2024-05-15T12:34:05.738 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:34:05.738 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:34:05.738 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:34:05.738 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:34:05.739 INFO:teuthology.orchestra.run.smithi027.stderr:bash: ntpq: command not found 2024-05-15T12:34:05.740 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-15T12:34:05.741 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-05-15T12:34:05.741 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:34:05.741 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:34:05.741 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:34:05.741 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:34:05.742 INFO:teuthology.orchestra.run.smithi177.stderr:bash: ntpq: command not found 2024-05-15T12:34:05.744 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-15T12:34:05.744 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-05-15T12:34:05.744 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:34:05.744 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:34:05.744 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:34:05.744 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:34:05.745 INFO:teuthology.run_tasks:Running task pexec... 2024-05-15T12:34:05.755 INFO:teuthology.task.pexec:Executing custom commands... 2024-05-15T12:34:05.755 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi027.front.sepia.ceph.com 2024-05-15T12:34:05.756 DEBUG:teuthology.orchestra.run.smithi027:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-05-15T12:34:05.756 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi120.front.sepia.ceph.com 2024-05-15T12:34:05.756 DEBUG:teuthology.orchestra.run.smithi120:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-05-15T12:34:05.756 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi177.front.sepia.ceph.com 2024-05-15T12:34:05.756 DEBUG:teuthology.orchestra.run.smithi177:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-05-15T12:34:06.526 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:33 ago on Wed 15 May 2024 12:32:33 PM UTC. 2024-05-15T12:34:06.543 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:33 ago on Wed 15 May 2024 12:32:33 PM UTC. 2024-05-15T12:34:06.573 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:24 ago on Wed 15 May 2024 12:32:42 PM UTC. 2024-05-15T12:34:08.713 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-05-15T12:34:08.713 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:34:08.714 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-05-15T12:34:08.714 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:34:08.714 INFO:teuthology.orchestra.run.smithi177.stdout:Resetting modules: 2024-05-15T12:34:08.714 INFO:teuthology.orchestra.run.smithi177.stdout: container-tools 2024-05-15T12:34:08.714 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:34:08.714 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-05-15T12:34:08.714 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:34:08.714 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:34:08.722 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-05-15T12:34:08.723 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:34:08.723 INFO:teuthology.orchestra.run.smithi120.stdout: Package Architecture Version Repository Size 2024-05-15T12:34:08.723 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:34:08.723 INFO:teuthology.orchestra.run.smithi120.stdout:Resetting modules: 2024-05-15T12:34:08.723 INFO:teuthology.orchestra.run.smithi120.stdout: container-tools 2024-05-15T12:34:08.723 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:34:08.723 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-05-15T12:34:08.723 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:34:08.723 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:34:08.885 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-15T12:34:08.886 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:34:08.886 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-05-15T12:34:08.886 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:34:08.886 INFO:teuthology.orchestra.run.smithi027.stdout:Resetting modules: 2024-05-15T12:34:08.886 INFO:teuthology.orchestra.run.smithi027.stdout: container-tools 2024-05-15T12:34:08.886 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:34:08.886 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-05-15T12:34:08.886 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:34:08.887 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:34:08.963 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-05-15T12:34:08.977 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-05-15T12:34:09.194 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-15T12:34:09.759 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:36 ago on Wed 15 May 2024 12:32:33 PM UTC. 2024-05-15T12:34:09.774 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:36 ago on Wed 15 May 2024 12:32:33 PM UTC. 2024-05-15T12:34:10.019 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:27 ago on Wed 15 May 2024 12:32:42 PM UTC. 2024-05-15T12:34:11.909 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-05-15T12:34:11.931 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-05-15T12:34:11.946 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:34:11.947 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-05-15T12:34:11.947 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:34:11.947 INFO:teuthology.orchestra.run.smithi120.stdout:Installing group/module packages: 2024-05-15T12:34:11.947 INFO:teuthology.orchestra.run.smithi120.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-05-15T12:34:11.947 INFO:teuthology.orchestra.run.smithi120.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-05-15T12:34:11.947 INFO:teuthology.orchestra.run.smithi120.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-05-15T12:34:11.947 INFO:teuthology.orchestra.run.smithi120.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-05-15T12:34:11.947 INFO:teuthology.orchestra.run.smithi120.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-05-15T12:34:11.947 INFO:teuthology.orchestra.run.smithi120.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-05-15T12:34:11.947 INFO:teuthology.orchestra.run.smithi120.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-05-15T12:34:11.947 INFO:teuthology.orchestra.run.smithi120.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-05-15T12:34:11.947 INFO:teuthology.orchestra.run.smithi120.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-05-15T12:34:11.947 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-05-15T12:34:11.947 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-05-15T12:34:11.947 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-05-15T12:34:11.948 INFO:teuthology.orchestra.run.smithi120.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-05-15T12:34:11.948 INFO:teuthology.orchestra.run.smithi120.stdout:Installing weak dependencies: 2024-05-15T12:34:11.948 INFO:teuthology.orchestra.run.smithi120.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-05-15T12:34:11.948 INFO:teuthology.orchestra.run.smithi120.stdout:Installing module profiles: 2024-05-15T12:34:11.948 INFO:teuthology.orchestra.run.smithi120.stdout: container-tools/common 2024-05-15T12:34:11.948 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:34:11.948 INFO:teuthology.orchestra.run.smithi120.stdout:Enabling module streams: 2024-05-15T12:34:11.948 INFO:teuthology.orchestra.run.smithi120.stdout: container-tools rhel8 2024-05-15T12:34:11.948 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:34:11.948 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-05-15T12:34:11.948 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:34:11.948 INFO:teuthology.orchestra.run.smithi120.stdout:Install 13 Packages 2024-05-15T12:34:11.948 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:34:11.949 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 27 M 2024-05-15T12:34:11.949 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 91 M 2024-05-15T12:34:11.949 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-05-15T12:34:11.970 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:34:11.970 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-05-15T12:34:11.970 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:34:11.970 INFO:teuthology.orchestra.run.smithi177.stdout:Installing group/module packages: 2024-05-15T12:34:11.970 INFO:teuthology.orchestra.run.smithi177.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-05-15T12:34:11.970 INFO:teuthology.orchestra.run.smithi177.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-05-15T12:34:11.970 INFO:teuthology.orchestra.run.smithi177.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-05-15T12:34:11.970 INFO:teuthology.orchestra.run.smithi177.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-05-15T12:34:11.970 INFO:teuthology.orchestra.run.smithi177.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-05-15T12:34:11.970 INFO:teuthology.orchestra.run.smithi177.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-05-15T12:34:11.970 INFO:teuthology.orchestra.run.smithi177.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-05-15T12:34:11.971 INFO:teuthology.orchestra.run.smithi177.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-05-15T12:34:11.971 INFO:teuthology.orchestra.run.smithi177.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-05-15T12:34:11.971 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-05-15T12:34:11.971 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-05-15T12:34:11.971 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-05-15T12:34:11.971 INFO:teuthology.orchestra.run.smithi177.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-05-15T12:34:11.971 INFO:teuthology.orchestra.run.smithi177.stdout:Installing weak dependencies: 2024-05-15T12:34:11.971 INFO:teuthology.orchestra.run.smithi177.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-05-15T12:34:11.971 INFO:teuthology.orchestra.run.smithi177.stdout:Installing module profiles: 2024-05-15T12:34:11.971 INFO:teuthology.orchestra.run.smithi177.stdout: container-tools/common 2024-05-15T12:34:11.971 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:34:11.971 INFO:teuthology.orchestra.run.smithi177.stdout:Enabling module streams: 2024-05-15T12:34:11.971 INFO:teuthology.orchestra.run.smithi177.stdout: container-tools rhel8 2024-05-15T12:34:11.971 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:34:11.971 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-05-15T12:34:11.971 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:34:11.972 INFO:teuthology.orchestra.run.smithi177.stdout:Install 13 Packages 2024-05-15T12:34:11.972 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:34:11.972 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 27 M 2024-05-15T12:34:11.972 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 91 M 2024-05-15T12:34:11.972 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-05-15T12:34:12.295 INFO:teuthology.orchestra.run.smithi120.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 2.5 MB/s | 689 kB 00:00 2024-05-15T12:34:12.343 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-15T12:34:12.353 INFO:teuthology.orchestra.run.smithi120.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 3.0 MB/s | 1.0 MB 00:00 2024-05-15T12:34:12.370 INFO:teuthology.orchestra.run.smithi120.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 503 kB/s | 38 kB 00:00 2024-05-15T12:34:12.385 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:34:12.385 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-05-15T12:34:12.385 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:34:12.385 INFO:teuthology.orchestra.run.smithi027.stdout:Installing group/module packages: 2024-05-15T12:34:12.385 INFO:teuthology.orchestra.run.smithi027.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-05-15T12:34:12.385 INFO:teuthology.orchestra.run.smithi027.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-05-15T12:34:12.385 INFO:teuthology.orchestra.run.smithi027.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-05-15T12:34:12.385 INFO:teuthology.orchestra.run.smithi027.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-05-15T12:34:12.385 INFO:teuthology.orchestra.run.smithi027.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-05-15T12:34:12.385 INFO:teuthology.orchestra.run.smithi027.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-05-15T12:34:12.385 INFO:teuthology.orchestra.run.smithi027.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-05-15T12:34:12.385 INFO:teuthology.orchestra.run.smithi027.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-05-15T12:34:12.385 INFO:teuthology.orchestra.run.smithi027.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-05-15T12:34:12.385 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-05-15T12:34:12.386 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-05-15T12:34:12.386 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-05-15T12:34:12.386 INFO:teuthology.orchestra.run.smithi027.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-05-15T12:34:12.386 INFO:teuthology.orchestra.run.smithi027.stdout:Installing weak dependencies: 2024-05-15T12:34:12.386 INFO:teuthology.orchestra.run.smithi027.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-05-15T12:34:12.386 INFO:teuthology.orchestra.run.smithi027.stdout:Installing module profiles: 2024-05-15T12:34:12.386 INFO:teuthology.orchestra.run.smithi027.stdout: container-tools/common 2024-05-15T12:34:12.386 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:34:12.386 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling module streams: 2024-05-15T12:34:12.386 INFO:teuthology.orchestra.run.smithi027.stdout: container-tools rhel8 2024-05-15T12:34:12.386 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:34:12.386 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-05-15T12:34:12.386 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:34:12.386 INFO:teuthology.orchestra.run.smithi027.stdout:Install 13 Packages 2024-05-15T12:34:12.387 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:34:12.387 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 27 M 2024-05-15T12:34:12.387 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 91 M 2024-05-15T12:34:12.387 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-05-15T12:34:12.421 INFO:teuthology.orchestra.run.smithi120.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 3.8 MB/s | 258 kB 00:00 2024-05-15T12:34:12.444 INFO:teuthology.orchestra.run.smithi177.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 2.6 MB/s | 689 kB 00:00 2024-05-15T12:34:12.494 INFO:teuthology.orchestra.run.smithi177.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 760 kB/s | 38 kB 00:00 2024-05-15T12:34:12.496 INFO:teuthology.orchestra.run.smithi120.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 2.3 MB/s | 180 kB 00:00 2024-05-15T12:34:12.546 INFO:teuthology.orchestra.run.smithi027.stdout:[MIRROR] aardvark-dns-1.10.0-1.module_el8+871+552be958.x86_64.rpm: Status code: 403 for http://distro.ibiblio.org/centos/8-stream/AppStream/x86_64/os/Packages/aardvark-dns-1.10.0-1.module_el8%2b871%2b552be958.x86_64.rpm (IP: 152.19.134.43) 2024-05-15T12:34:12.569 INFO:teuthology.orchestra.run.smithi177.stdout:(3/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 3.4 MB/s | 258 kB 00:00 2024-05-15T12:34:12.571 INFO:teuthology.orchestra.run.smithi120.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 331 kB/s | 25 kB 00:00 2024-05-15T12:34:12.632 INFO:teuthology.orchestra.run.smithi120.stdout:(7/13): netavark-1.10.3-1.module_el8+951+32019c 16 MB/s | 4.1 MB 00:00 2024-05-15T12:34:12.649 INFO:teuthology.orchestra.run.smithi120.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.2 MB/s | 94 kB 00:00 2024-05-15T12:34:12.757 INFO:teuthology.orchestra.run.smithi027.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 2.3 MB/s | 689 kB 00:00 2024-05-15T12:34:12.758 INFO:teuthology.orchestra.run.smithi120.stdout:(9/13): buildah-1.33.6-2.module_el8+976+1136073 13 MB/s | 9.6 MB 00:00 2024-05-15T12:34:12.788 INFO:teuthology.orchestra.run.smithi027.stdout:[MIRROR] criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64.rpm: Status code: 403 for http://distro.ibiblio.org/centos/8-stream/AppStream/x86_64/os/Packages/criu-libs-3.15-3.module_el8.7.0%2b1216%2bb022c01d.x86_64.rpm (IP: 152.19.134.43) 2024-05-15T12:34:12.794 INFO:teuthology.orchestra.run.smithi177.stdout:(4/13): aardvark-dns-1.10.0-1.module_el8+871+55 1.6 MB/s | 1.0 MB 00:00 2024-05-15T12:34:12.802 INFO:teuthology.orchestra.run.smithi120.stdout:[MIRROR] udica-0.2.6-20.module_el8+804+f131391c.noarch.rpm: Status code: 403 for http://distro.ibiblio.org/centos/8-stream/AppStream/x86_64/os/Packages/udica-0.2.6-20.module_el8%2b804%2bf131391c.noarch.rpm (IP: 152.19.134.43) 2024-05-15T12:34:12.908 INFO:teuthology.orchestra.run.smithi027.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 252 kB/s | 38 kB 00:00 2024-05-15T12:34:12.945 INFO:teuthology.orchestra.run.smithi177.stdout:(5/13): netavark-1.10.3-1.module_el8+951+32019c 11 MB/s | 4.1 MB 00:00 2024-05-15T12:34:12.967 INFO:teuthology.orchestra.run.smithi120.stdout:(10/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 8.0 MB/s | 2.5 MB 00:00 2024-05-15T12:34:12.970 INFO:teuthology.orchestra.run.smithi177.stdout:(6/13): python3-podman-4.9.0-1.module_el8+847+7 1.0 MB/s | 180 kB 00:00 2024-05-15T12:34:12.992 INFO:teuthology.orchestra.run.smithi120.stdout:(11/13): udica-0.2.6-20.module_el8+804+f131391c 225 kB/s | 53 kB 00:00 2024-05-15T12:34:13.003 INFO:teuthology.orchestra.run.smithi177.stdout:(7/13): python3-pytoml-0.1.14-5.git7dea353.el8. 422 kB/s | 25 kB 00:00 2024-05-15T12:34:13.033 INFO:teuthology.orchestra.run.smithi027.stdout:(3/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 2.0 MB/s | 258 kB 00:00 2024-05-15T12:34:13.037 INFO:teuthology.orchestra.run.smithi177.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.4 MB/s | 94 kB 00:00 2024-05-15T12:34:13.075 INFO:teuthology.orchestra.run.smithi027.stdout:(4/13): aardvark-dns-1.10.0-1.module_el8+871+55 1.6 MB/s | 1.0 MB 00:00 2024-05-15T12:34:13.134 INFO:teuthology.orchestra.run.smithi120.stdout:(12/13): skopeo-1.14.3-0.1.module_el8+885+7da14 18 MB/s | 8.8 MB 00:00 2024-05-15T12:34:13.159 INFO:teuthology.orchestra.run.smithi120.stdout:(13/13): yajl-2.1.0-12.el8.x86_64.rpm 213 kB/s | 41 kB 00:00 2024-05-15T12:34:13.160 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:34:13.160 INFO:teuthology.orchestra.run.smithi120.stdout:Total 23 MB/s | 27 MB 00:01 2024-05-15T12:34:13.184 INFO:teuthology.orchestra.run.smithi027.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 1.6 MB/s | 180 kB 00:00 2024-05-15T12:34:13.217 INFO:teuthology.orchestra.run.smithi027.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 749 kB/s | 25 kB 00:00 2024-05-15T12:34:13.275 INFO:teuthology.orchestra.run.smithi027.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.6 MB/s | 94 kB 00:00 2024-05-15T12:34:13.428 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-05-15T12:34:13.448 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-05-15T12:34:13.448 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-05-15T12:34:13.462 INFO:teuthology.orchestra.run.smithi177.stdout:(9/13): skopeo-1.14.3-0.1.module_el8+885+7da147 19 MB/s | 8.8 MB 00:00 2024-05-15T12:34:13.513 INFO:teuthology.orchestra.run.smithi177.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 1.0 MB/s | 53 kB 00:00 2024-05-15T12:34:13.563 INFO:teuthology.orchestra.run.smithi177.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 818 kB/s | 41 kB 00:00 2024-05-15T12:34:13.584 INFO:teuthology.orchestra.run.smithi027.stdout:(8/13): buildah-1.33.6-2.module_el8+976+1136073 8.6 MB/s | 9.6 MB 00:01 2024-05-15T12:34:13.647 INFO:teuthology.orchestra.run.smithi177.stdout:(12/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 4.2 MB/s | 2.5 MB 00:00 2024-05-15T12:34:13.709 INFO:teuthology.orchestra.run.smithi027.stdout:(9/13): netavark-1.10.3-1.module_el8+951+32019c 6.1 MB/s | 4.1 MB 00:00 2024-05-15T12:34:13.715 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-05-15T12:34:13.718 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-05-15T12:34:13.752 INFO:teuthology.orchestra.run.smithi027.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 1.2 MB/s | 53 kB 00:00 2024-05-15T12:34:13.794 INFO:teuthology.orchestra.run.smithi027.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 984 kB/s | 41 kB 00:00 2024-05-15T12:34:14.138 INFO:teuthology.orchestra.run.smithi177.stdout:(13/13): buildah-1.33.6-2.module_el8+976+113607 4.9 MB/s | 9.6 MB 00:01 2024-05-15T12:34:14.139 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:34:14.139 INFO:teuthology.orchestra.run.smithi177.stdout:Total 13 MB/s | 27 MB 00:02 2024-05-15T12:34:14.286 INFO:teuthology.orchestra.run.smithi027.stdout:(12/13): skopeo-1.14.3-0.1.module_el8+885+7da14 8.7 MB/s | 8.8 MB 00:01 2024-05-15T12:34:14.332 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-05-15T12:34:14.361 INFO:teuthology.orchestra.run.smithi027.stdout:(13/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 3.3 MB/s | 2.5 MB 00:00 2024-05-15T12:34:14.361 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:34:14.361 INFO:teuthology.orchestra.run.smithi027.stdout:Total 14 MB/s | 27 MB 00:01 2024-05-15T12:34:14.407 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-05-15T12:34:14.428 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-05-15T12:34:14.429 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-05-15T12:34:14.642 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-05-15T12:34:14.664 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-05-15T12:34:14.664 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-05-15T12:34:14.683 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-05-15T12:34:14.687 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-05-15T12:34:14.938 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-05-15T12:34:14.942 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-05-15T12:34:15.286 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-05-15T12:34:15.288 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-05-15T12:34:15.557 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-05-15T12:34:15.625 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-05-15T12:34:15.719 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-05-15T12:34:15.962 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-05-15T12:34:16.225 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-05-15T12:34:16.419 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-05-15T12:34:16.511 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-05-15T12:34:16.589 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-05-15T12:34:16.640 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-05-15T12:34:16.776 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-05-15T12:34:16.895 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-05-15T12:34:16.990 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-05-15T12:34:17.048 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-05-15T12:34:17.054 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-05-15T12:34:17.288 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-05-15T12:34:17.506 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-05-15T12:34:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-05-15T12:34:18.033 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-05-15T12:34:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-05-15T12:34:18.320 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-05-15T12:34:18.461 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-05-15T12:34:18.532 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-05-15T12:34:18.682 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-05-15T12:34:18.894 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-05-15T12:34:18.951 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-05-15T12:34:19.153 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-05-15T12:34:19.175 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-05-15T12:34:19.357 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-05-15T12:34:19.476 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-05-15T12:34:19.610 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-05-15T12:34:19.696 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-05-15T12:34:19.715 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-05-15T12:34:19.907 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-05-15T12:34:20.028 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-05-15T12:34:20.028 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-05-15T12:34:20.028 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-05-15T12:34:20.028 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-05-15T12:34:20.028 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-05-15T12:34:20.028 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-05-15T12:34:20.028 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-05-15T12:34:20.028 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-05-15T12:34:20.028 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-05-15T12:34:20.028 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-05-15T12:34:20.028 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-05-15T12:34:20.028 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-05-15T12:34:20.028 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-05-15T12:34:20.127 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-05-15T12:34:20.176 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-05-15T12:34:20.363 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-05-15T12:34:20.417 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-05-15T12:34:20.582 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-05-15T12:34:20.582 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:34:20.582 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-05-15T12:34:20.582 INFO:teuthology.orchestra.run.smithi120.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-05-15T12:34:20.582 INFO:teuthology.orchestra.run.smithi120.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-05-15T12:34:20.582 INFO:teuthology.orchestra.run.smithi120.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-05-15T12:34:20.582 INFO:teuthology.orchestra.run.smithi120.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-05-15T12:34:20.582 INFO:teuthology.orchestra.run.smithi120.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-05-15T12:34:20.582 INFO:teuthology.orchestra.run.smithi120.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-05-15T12:34:20.583 INFO:teuthology.orchestra.run.smithi120.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-05-15T12:34:20.583 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-05-15T12:34:20.583 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-05-15T12:34:20.583 INFO:teuthology.orchestra.run.smithi120.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-05-15T12:34:20.583 INFO:teuthology.orchestra.run.smithi120.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-05-15T12:34:20.583 INFO:teuthology.orchestra.run.smithi120.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-05-15T12:34:20.583 INFO:teuthology.orchestra.run.smithi120.stdout: yajl-2.1.0-12.el8.x86_64 2024-05-15T12:34:20.583 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:34:20.583 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-05-15T12:34:20.643 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-05-15T12:34:20.793 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-05-15T12:34:20.793 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-05-15T12:34:20.793 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-05-15T12:34:20.793 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-05-15T12:34:20.793 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-05-15T12:34:20.793 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-05-15T12:34:20.793 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-05-15T12:34:20.794 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-05-15T12:34:20.794 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-05-15T12:34:20.794 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-05-15T12:34:20.794 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-05-15T12:34:20.794 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-05-15T12:34:20.794 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-05-15T12:34:20.864 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-05-15T12:34:20.884 DEBUG:teuthology.parallel:result is None 2024-05-15T12:34:21.256 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-05-15T12:34:21.256 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:34:21.256 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-05-15T12:34:21.256 INFO:teuthology.orchestra.run.smithi177.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-05-15T12:34:21.256 INFO:teuthology.orchestra.run.smithi177.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-05-15T12:34:21.256 INFO:teuthology.orchestra.run.smithi177.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-05-15T12:34:21.256 INFO:teuthology.orchestra.run.smithi177.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-05-15T12:34:21.256 INFO:teuthology.orchestra.run.smithi177.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-05-15T12:34:21.256 INFO:teuthology.orchestra.run.smithi177.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-05-15T12:34:21.257 INFO:teuthology.orchestra.run.smithi177.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-05-15T12:34:21.257 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-05-15T12:34:21.257 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-05-15T12:34:21.257 INFO:teuthology.orchestra.run.smithi177.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-05-15T12:34:21.257 INFO:teuthology.orchestra.run.smithi177.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-05-15T12:34:21.257 INFO:teuthology.orchestra.run.smithi177.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-05-15T12:34:21.257 INFO:teuthology.orchestra.run.smithi177.stdout: yajl-2.1.0-12.el8.x86_64 2024-05-15T12:34:21.257 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:34:21.257 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-05-15T12:34:21.338 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-05-15T12:34:21.338 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-05-15T12:34:21.338 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-05-15T12:34:21.338 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-05-15T12:34:21.338 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-05-15T12:34:21.339 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-05-15T12:34:21.339 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-05-15T12:34:21.339 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-05-15T12:34:21.339 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-05-15T12:34:21.339 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-05-15T12:34:21.339 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-05-15T12:34:21.339 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-05-15T12:34:21.339 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-05-15T12:34:21.557 DEBUG:teuthology.parallel:result is None 2024-05-15T12:34:21.893 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-05-15T12:34:21.893 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:34:21.894 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-05-15T12:34:21.894 INFO:teuthology.orchestra.run.smithi027.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-05-15T12:34:21.894 INFO:teuthology.orchestra.run.smithi027.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-05-15T12:34:21.894 INFO:teuthology.orchestra.run.smithi027.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-05-15T12:34:21.894 INFO:teuthology.orchestra.run.smithi027.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-05-15T12:34:21.894 INFO:teuthology.orchestra.run.smithi027.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-05-15T12:34:21.895 INFO:teuthology.orchestra.run.smithi027.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-05-15T12:34:21.895 INFO:teuthology.orchestra.run.smithi027.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-05-15T12:34:21.895 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-05-15T12:34:21.895 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-05-15T12:34:21.895 INFO:teuthology.orchestra.run.smithi027.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-05-15T12:34:21.895 INFO:teuthology.orchestra.run.smithi027.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-05-15T12:34:21.896 INFO:teuthology.orchestra.run.smithi027.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-05-15T12:34:21.896 INFO:teuthology.orchestra.run.smithi027.stdout: yajl-2.1.0-12.el8.x86_64 2024-05-15T12:34:21.896 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:34:21.896 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-15T12:34:22.179 DEBUG:teuthology.parallel:result is None 2024-05-15T12:34:22.180 INFO:teuthology.run_tasks:Running task install... 2024-05-15T12:34:22.188 DEBUG:teuthology.task.install:project ceph 2024-05-15T12:34:22.188 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314'}} 2024-05-15T12:34:22.188 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'flavor': 'default', 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314'} 2024-05-15T12:34:22.188 INFO:teuthology.task.install:Using flavor: default 2024-05-15T12:34:22.194 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-15T12:34:22.194 INFO:teuthology.task.install:extra packages: [] 2024-05-15T12:34:22.194 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314', 'tag': None, 'wait_for_package': False} 2024-05-15T12:34:22.195 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-15T12:34:22.195 INFO:teuthology.packaging:ref: None 2024-05-15T12:34:22.195 INFO:teuthology.packaging:tag: None 2024-05-15T12:34:22.195 INFO:teuthology.packaging:branch: quincy 2024-05-15T12:34:22.195 INFO:teuthology.packaging:sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:34:22.195 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-05-15T12:34:22.197 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314', 'tag': None, 'wait_for_package': False} 2024-05-15T12:34:22.198 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-15T12:34:22.198 INFO:teuthology.packaging:ref: None 2024-05-15T12:34:22.198 INFO:teuthology.packaging:tag: None 2024-05-15T12:34:22.198 INFO:teuthology.packaging:branch: quincy 2024-05-15T12:34:22.198 INFO:teuthology.packaging:sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:34:22.198 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-05-15T12:34:22.199 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314', 'tag': None, 'wait_for_package': False} 2024-05-15T12:34:22.199 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-15T12:34:22.199 INFO:teuthology.packaging:ref: None 2024-05-15T12:34:22.199 INFO:teuthology.packaging:tag: None 2024-05-15T12:34:22.199 INFO:teuthology.packaging:branch: quincy 2024-05-15T12:34:22.199 INFO:teuthology.packaging:sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:34:22.199 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-05-15T12:34:22.455 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/quincy/72b0874a0bf8d8bc27f1e304ccc2198707d4dad7/centos/8/flavors/default/ 2024-05-15T12:34:22.455 INFO:teuthology.task.install.rpm:Package version is 17.2.7-945.g72b0874a 2024-05-15T12:34:22.460 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/quincy/72b0874a0bf8d8bc27f1e304ccc2198707d4dad7/centos/8/flavors/default/ 2024-05-15T12:34:22.460 INFO:teuthology.task.install.rpm:Package version is 17.2.7-945.g72b0874a 2024-05-15T12:34:22.513 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/quincy/72b0874a0bf8d8bc27f1e304ccc2198707d4dad7/centos/8/flavors/default/ 2024-05-15T12:34:22.514 INFO:teuthology.task.install.rpm:Package version is 17.2.7-945.g72b0874a 2024-05-15T12:34:22.582 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72b0874a0bf8d8bc27f1e304ccc2198707d4dad7/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72b0874a0bf8d8bc27f1e304ccc2198707d4dad7/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72b0874a0bf8d8bc27f1e304ccc2198707d4dad7/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-15T12:34:22.582 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-05-15T12:34:22.583 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-15T12:34:22.613 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-05-15T12:34:22.613 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-15T12:34:22.613 INFO:teuthology.packaging:ref: None 2024-05-15T12:34:22.613 INFO:teuthology.packaging:tag: None 2024-05-15T12:34:22.613 INFO:teuthology.packaging:branch: quincy 2024-05-15T12:34:22.613 INFO:teuthology.packaging:sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:34:22.613 DEBUG:teuthology.orchestra.run.smithi177:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-15T12:34:22.624 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72b0874a0bf8d8bc27f1e304ccc2198707d4dad7/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72b0874a0bf8d8bc27f1e304ccc2198707d4dad7/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72b0874a0bf8d8bc27f1e304ccc2198707d4dad7/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-15T12:34:22.624 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-15T12:34:22.624 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-15T12:34:22.635 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72b0874a0bf8d8bc27f1e304ccc2198707d4dad7/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72b0874a0bf8d8bc27f1e304ccc2198707d4dad7/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72b0874a0bf8d8bc27f1e304ccc2198707d4dad7/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-15T12:34:22.635 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-05-15T12:34:22.635 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-15T12:34:22.653 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-05-15T12:34:22.653 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-15T12:34:22.654 INFO:teuthology.packaging:ref: None 2024-05-15T12:34:22.654 INFO:teuthology.packaging:tag: None 2024-05-15T12:34:22.654 INFO:teuthology.packaging:branch: quincy 2024-05-15T12:34:22.654 INFO:teuthology.packaging:sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:34:22.654 DEBUG:teuthology.orchestra.run.smithi027:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-15T12:34:22.668 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-05-15T12:34:22.668 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-15T12:34:22.668 INFO:teuthology.packaging:ref: None 2024-05-15T12:34:22.668 INFO:teuthology.packaging:tag: None 2024-05-15T12:34:22.668 INFO:teuthology.packaging:branch: quincy 2024-05-15T12:34:22.668 INFO:teuthology.packaging:sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:34:22.668 DEBUG:teuthology.orchestra.run.smithi120:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-15T12:34:22.683 DEBUG:teuthology.orchestra.run.smithi177:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-15T12:34:22.723 DEBUG:teuthology.orchestra.run.smithi027:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-15T12:34:22.739 DEBUG:teuthology.orchestra.run.smithi120:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-15T12:34:22.761 DEBUG:teuthology.orchestra.run.smithi177:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-15T12:34:22.802 DEBUG:teuthology.orchestra.run.smithi027:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-15T12:34:22.818 DEBUG:teuthology.orchestra.run.smithi120:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-15T12:34:22.828 INFO:teuthology.orchestra.run.smithi177.stdout:check_obsoletes = 1 2024-05-15T12:34:22.830 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum clean all 2024-05-15T12:34:22.870 INFO:teuthology.orchestra.run.smithi027.stdout:check_obsoletes = 1 2024-05-15T12:34:22.871 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum clean all 2024-05-15T12:34:22.885 INFO:teuthology.orchestra.run.smithi120.stdout:check_obsoletes = 1 2024-05-15T12:34:22.887 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum clean all 2024-05-15T12:34:23.097 INFO:teuthology.orchestra.run.smithi177.stdout:78 files removed 2024-05-15T12:34:23.125 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-radosgw 2024-05-15T12:34:23.150 INFO:teuthology.orchestra.run.smithi120.stdout:78 files removed 2024-05-15T12:34:23.151 INFO:teuthology.orchestra.run.smithi027.stdout:78 files removed 2024-05-15T12:34:23.178 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-radosgw 2024-05-15T12:34:23.179 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-radosgw 2024-05-15T12:34:23.854 INFO:teuthology.orchestra.run.smithi120.stdout:ceph packages for x86_64 286 kB/s | 79 kB 00:00 2024-05-15T12:34:23.953 INFO:teuthology.orchestra.run.smithi177.stdout:ceph packages for x86_64 186 kB/s | 79 kB 00:00 2024-05-15T12:34:24.082 INFO:teuthology.orchestra.run.smithi027.stdout:ceph packages for x86_64 165 kB/s | 79 kB 00:00 2024-05-15T12:34:24.163 INFO:teuthology.orchestra.run.smithi120.stdout:ceph noarch packages 39 kB/s | 11 kB 00:00 2024-05-15T12:34:24.337 INFO:teuthology.orchestra.run.smithi177.stdout:ceph noarch packages 31 kB/s | 11 kB 00:00 2024-05-15T12:34:24.405 INFO:teuthology.orchestra.run.smithi027.stdout:ceph noarch packages 38 kB/s | 11 kB 00:00 2024-05-15T12:34:24.574 INFO:teuthology.orchestra.run.smithi120.stdout:ceph source packages 3.8 kB/s | 1.5 kB 00:00 2024-05-15T12:34:24.775 INFO:teuthology.orchestra.run.smithi027.stdout:ceph source packages 4.3 kB/s | 1.5 kB 00:00 2024-05-15T12:34:24.923 INFO:teuthology.orchestra.run.smithi177.stdout:ceph source packages 2.6 kB/s | 1.5 kB 00:00 2024-05-15T12:34:25.740 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 8 - AppStream 25 MB/s | 28 MB 00:01 2024-05-15T12:34:26.789 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 8 - AppStream 15 MB/s | 28 MB 00:01 2024-05-15T12:34:26.891 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - AppStream 13 MB/s | 28 MB 00:02 2024-05-15T12:34:33.416 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 8 - BaseOS 3.9 MB/s | 10 MB 00:02 2024-05-15T12:34:34.122 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 8 - BaseOS 4.5 MB/s | 10 MB 00:02 2024-05-15T12:34:34.467 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - BaseOS 4.6 MB/s | 10 MB 00:02 2024-05-15T12:34:35.549 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 8 - Extras 34 kB/s | 18 kB 00:00 2024-05-15T12:34:35.958 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 8 - Extras common packages 26 kB/s | 7.7 kB 00:00 2024-05-15T12:34:36.248 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 8 - Extras 35 kB/s | 18 kB 00:00 2024-05-15T12:34:36.490 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 8 - Extras common packages 58 kB/s | 7.7 kB 00:00 2024-05-15T12:34:36.543 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - Extras 49 kB/s | 18 kB 00:00 2024-05-15T12:34:36.818 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - Extras common packages 48 kB/s | 7.7 kB 00:00 2024-05-15T12:34:36.951 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 8 - PowerTools 5.5 MB/s | 5.0 MB 00:00 2024-05-15T12:34:37.425 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 8 - PowerTools 5.9 MB/s | 5.0 MB 00:00 2024-05-15T12:34:37.628 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - PowerTools 7.0 MB/s | 5.0 MB 00:00 2024-05-15T12:34:38.012 INFO:teuthology.orchestra.run.smithi120.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-05-15T12:34:38.493 INFO:teuthology.orchestra.run.smithi177.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-05-15T12:34:38.781 INFO:teuthology.orchestra.run.smithi027.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-05-15T12:34:39.262 INFO:teuthology.orchestra.run.smithi120.stdout:Extra Packages for Enterprise Linux 12 MB/s | 14 MB 00:01 2024-05-15T12:34:40.041 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 12 MB/s | 14 MB 00:01 2024-05-15T12:34:40.204 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 8.4 MB/s | 14 MB 00:01 2024-05-15T12:34:43.513 INFO:teuthology.orchestra.run.smithi120.stdout:lab-extras 429 kB/s | 24 kB 00:00 2024-05-15T12:34:44.510 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 364 kB/s | 24 kB 00:00 2024-05-15T12:34:44.639 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 280 kB/s | 24 kB 00:00 2024-05-15T12:34:47.230 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-05-15T12:34:47.232 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:34:47.232 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-05-15T12:34:47.233 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:34:47.233 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-05-15T12:34:47.233 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw x86_64 2:17.2.7-945.g72b0874a.el8 ceph 17 M 2024-05-15T12:34:47.233 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrading: 2024-05-15T12:34:47.233 INFO:teuthology.orchestra.run.smithi120.stdout: librados2 x86_64 2:17.2.7-945.g72b0874a.el8 ceph 3.5 M 2024-05-15T12:34:47.233 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1 x86_64 2:17.2.7-945.g72b0874a.el8 ceph 3.6 M 2024-05-15T12:34:47.233 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-05-15T12:34:47.233 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base x86_64 2:17.2.7-945.g72b0874a.el8 ceph 5.3 M 2024-05-15T12:34:47.233 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common x86_64 2:17.2.7-945.g72b0874a.el8 ceph 24 M 2024-05-15T12:34:47.233 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-selinux x86_64 2:17.2.7-945.g72b0874a.el8 ceph 24 k 2024-05-15T12:34:47.233 INFO:teuthology.orchestra.run.smithi120.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-05-15T12:34:47.234 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2 x86_64 2:17.2.7-945.g72b0874a.el8 ceph 798 k 2024-05-15T12:34:47.234 INFO:teuthology.orchestra.run.smithi120.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-05-15T12:34:47.234 INFO:teuthology.orchestra.run.smithi120.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-05-15T12:34:47.234 INFO:teuthology.orchestra.run.smithi120.stdout: libradosstriper1 x86_64 2:17.2.7-945.g72b0874a.el8 ceph 505 k 2024-05-15T12:34:47.234 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-05-15T12:34:47.234 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2 x86_64 2:17.2.7-945.g72b0874a.el8 ceph 5.6 M 2024-05-15T12:34:47.234 INFO:teuthology.orchestra.run.smithi120.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-05-15T12:34:47.234 INFO:teuthology.orchestra.run.smithi120.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-05-15T12:34:47.234 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse x86_64 2:17.2.7-945.g72b0874a.el8 ceph 45 k 2024-05-15T12:34:47.234 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-common x86_64 2:17.2.7-945.g72b0874a.el8 ceph 117 k 2024-05-15T12:34:47.234 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs x86_64 2:17.2.7-945.g72b0874a.el8 ceph 190 k 2024-05-15T12:34:47.234 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-05-15T12:34:47.235 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados x86_64 2:17.2.7-945.g72b0874a.el8 ceph 371 k 2024-05-15T12:34:47.235 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd x86_64 2:17.2.7-945.g72b0874a.el8 ceph 358 k 2024-05-15T12:34:47.235 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw x86_64 2:17.2.7-945.g72b0874a.el8 ceph 107 k 2024-05-15T12:34:47.235 INFO:teuthology.orchestra.run.smithi120.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-05-15T12:34:47.235 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:34:47.235 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-05-15T12:34:47.235 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:34:47.235 INFO:teuthology.orchestra.run.smithi120.stdout:Install 21 Packages 2024-05-15T12:34:47.235 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrade 2 Packages 2024-05-15T12:34:47.235 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:34:47.236 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 64 M 2024-05-15T12:34:47.236 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-05-15T12:34:48.249 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-05-15T12:34:48.252 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:34:48.252 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-05-15T12:34:48.252 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:34:48.252 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-05-15T12:34:48.252 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw x86_64 2:17.2.7-945.g72b0874a.el8 ceph 17 M 2024-05-15T12:34:48.252 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrading: 2024-05-15T12:34:48.252 INFO:teuthology.orchestra.run.smithi177.stdout: librados2 x86_64 2:17.2.7-945.g72b0874a.el8 ceph 3.5 M 2024-05-15T12:34:48.252 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1 x86_64 2:17.2.7-945.g72b0874a.el8 ceph 3.6 M 2024-05-15T12:34:48.252 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-05-15T12:34:48.252 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base x86_64 2:17.2.7-945.g72b0874a.el8 ceph 5.3 M 2024-05-15T12:34:48.252 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common x86_64 2:17.2.7-945.g72b0874a.el8 ceph 24 M 2024-05-15T12:34:48.253 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux x86_64 2:17.2.7-945.g72b0874a.el8 ceph 24 k 2024-05-15T12:34:48.253 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-05-15T12:34:48.253 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2 x86_64 2:17.2.7-945.g72b0874a.el8 ceph 798 k 2024-05-15T12:34:48.253 INFO:teuthology.orchestra.run.smithi177.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-05-15T12:34:48.253 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-05-15T12:34:48.253 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1 x86_64 2:17.2.7-945.g72b0874a.el8 ceph 505 k 2024-05-15T12:34:48.253 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-05-15T12:34:48.253 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2 x86_64 2:17.2.7-945.g72b0874a.el8 ceph 5.6 M 2024-05-15T12:34:48.253 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-05-15T12:34:48.253 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-05-15T12:34:48.253 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse x86_64 2:17.2.7-945.g72b0874a.el8 ceph 45 k 2024-05-15T12:34:48.254 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common x86_64 2:17.2.7-945.g72b0874a.el8 ceph 117 k 2024-05-15T12:34:48.254 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs x86_64 2:17.2.7-945.g72b0874a.el8 ceph 190 k 2024-05-15T12:34:48.254 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-05-15T12:34:48.254 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados x86_64 2:17.2.7-945.g72b0874a.el8 ceph 371 k 2024-05-15T12:34:48.254 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd x86_64 2:17.2.7-945.g72b0874a.el8 ceph 358 k 2024-05-15T12:34:48.254 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw x86_64 2:17.2.7-945.g72b0874a.el8 ceph 107 k 2024-05-15T12:34:48.254 INFO:teuthology.orchestra.run.smithi177.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-05-15T12:34:48.254 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:34:48.254 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-05-15T12:34:48.254 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:34:48.254 INFO:teuthology.orchestra.run.smithi177.stdout:Install 21 Packages 2024-05-15T12:34:48.254 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrade 2 Packages 2024-05-15T12:34:48.254 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:34:48.255 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 64 M 2024-05-15T12:34:48.255 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-05-15T12:34:48.637 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-15T12:34:48.640 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:34:48.640 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-05-15T12:34:48.640 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:34:48.640 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-05-15T12:34:48.640 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw x86_64 2:17.2.7-945.g72b0874a.el8 ceph 17 M 2024-05-15T12:34:48.640 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrading: 2024-05-15T12:34:48.640 INFO:teuthology.orchestra.run.smithi027.stdout: librados2 x86_64 2:17.2.7-945.g72b0874a.el8 ceph 3.5 M 2024-05-15T12:34:48.640 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1 x86_64 2:17.2.7-945.g72b0874a.el8 ceph 3.6 M 2024-05-15T12:34:48.640 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-05-15T12:34:48.640 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base x86_64 2:17.2.7-945.g72b0874a.el8 ceph 5.3 M 2024-05-15T12:34:48.640 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common x86_64 2:17.2.7-945.g72b0874a.el8 ceph 24 M 2024-05-15T12:34:48.640 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux x86_64 2:17.2.7-945.g72b0874a.el8 ceph 24 k 2024-05-15T12:34:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-05-15T12:34:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2 x86_64 2:17.2.7-945.g72b0874a.el8 ceph 798 k 2024-05-15T12:34:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-05-15T12:34:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-05-15T12:34:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1 x86_64 2:17.2.7-945.g72b0874a.el8 ceph 505 k 2024-05-15T12:34:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-05-15T12:34:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2 x86_64 2:17.2.7-945.g72b0874a.el8 ceph 5.6 M 2024-05-15T12:34:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-05-15T12:34:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-05-15T12:34:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse x86_64 2:17.2.7-945.g72b0874a.el8 ceph 45 k 2024-05-15T12:34:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common x86_64 2:17.2.7-945.g72b0874a.el8 ceph 117 k 2024-05-15T12:34:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs x86_64 2:17.2.7-945.g72b0874a.el8 ceph 190 k 2024-05-15T12:34:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-05-15T12:34:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados x86_64 2:17.2.7-945.g72b0874a.el8 ceph 371 k 2024-05-15T12:34:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd x86_64 2:17.2.7-945.g72b0874a.el8 ceph 358 k 2024-05-15T12:34:48.642 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw x86_64 2:17.2.7-945.g72b0874a.el8 ceph 107 k 2024-05-15T12:34:48.642 INFO:teuthology.orchestra.run.smithi027.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-05-15T12:34:48.642 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:34:48.642 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-05-15T12:34:48.642 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:34:48.642 INFO:teuthology.orchestra.run.smithi027.stdout:Install 21 Packages 2024-05-15T12:34:48.642 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrade 2 Packages 2024-05-15T12:34:48.642 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:34:48.643 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 64 M 2024-05-15T12:34:48.643 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-05-15T12:34:48.998 INFO:teuthology.orchestra.run.smithi120.stdout:(1/23): ceph-common-17.2.7-945.g72b0874a.el8.x8 15 MB/s | 24 MB 00:01 2024-05-15T12:34:49.097 INFO:teuthology.orchestra.run.smithi120.stdout:(2/23): ceph-selinux-17.2.7-945.g72b0874a.el8.x 244 kB/s | 24 kB 00:00 2024-05-15T12:34:50.023 INFO:teuthology.orchestra.run.smithi120.stdout:(3/23): libcephfs2-17.2.7-945.g72b0874a.el8.x86 863 kB/s | 798 kB 00:00 2024-05-15T12:34:50.148 INFO:teuthology.orchestra.run.smithi120.stdout:(4/23): libradosstriper1-17.2.7-945.g72b0874a.e 4.0 MB/s | 505 kB 00:00 2024-05-15T12:34:50.203 INFO:teuthology.orchestra.run.smithi177.stdout:(1/23): ceph-common-17.2.7-945.g72b0874a.el8.x8 15 MB/s | 24 MB 00:01 2024-05-15T12:34:50.328 INFO:teuthology.orchestra.run.smithi177.stdout:(2/23): ceph-selinux-17.2.7-945.g72b0874a.el8.x 195 kB/s | 24 kB 00:00 2024-05-15T12:34:50.674 INFO:teuthology.orchestra.run.smithi120.stdout:(5/23): librgw2-17.2.7-945.g72b0874a.el8.x86_64 11 MB/s | 5.6 MB 00:00 2024-05-15T12:34:50.774 INFO:teuthology.orchestra.run.smithi120.stdout:(6/23): python3-ceph-argparse-17.2.7-945.g72b08 448 kB/s | 45 kB 00:00 2024-05-15T12:34:50.866 INFO:teuthology.orchestra.run.smithi120.stdout:(7/23): python3-ceph-common-17.2.7-945.g72b0874 1.2 MB/s | 117 kB 00:00 2024-05-15T12:34:50.878 INFO:teuthology.orchestra.run.smithi177.stdout:(3/23): libcephfs2-17.2.7-945.g72b0874a.el8.x86 1.4 MB/s | 798 kB 00:00 2024-05-15T12:34:50.958 INFO:teuthology.orchestra.run.smithi120.stdout:(8/23): python3-cephfs-17.2.7-945.g72b0874a.el8 2.0 MB/s | 190 kB 00:00 2024-05-15T12:34:50.995 INFO:teuthology.orchestra.run.smithi177.stdout:(4/23): libradosstriper1-17.2.7-945.g72b0874a.e 4.2 MB/s | 505 kB 00:00 2024-05-15T12:34:51.067 INFO:teuthology.orchestra.run.smithi120.stdout:(9/23): python3-rados-17.2.7-945.g72b0874a.el8. 3.3 MB/s | 371 kB 00:00 2024-05-15T12:34:51.141 INFO:teuthology.orchestra.run.smithi027.stdout:(1/23): ceph-base-17.2.7-945.g72b0874a.el8.x86_ 3.3 MB/s | 5.3 MB 00:01 2024-05-15T12:34:51.192 INFO:teuthology.orchestra.run.smithi120.stdout:(10/23): python3-rbd-17.2.7-945.g72b0874a.el8.x 2.8 MB/s | 358 kB 00:00 2024-05-15T12:34:51.266 INFO:teuthology.orchestra.run.smithi027.stdout:(2/23): ceph-selinux-17.2.7-945.g72b0874a.el8.x 195 kB/s | 24 kB 00:00 2024-05-15T12:34:51.300 INFO:teuthology.orchestra.run.smithi120.stdout:(11/23): python3-rgw-17.2.7-945.g72b0874a.el8.x 985 kB/s | 107 kB 00:00 2024-05-15T12:34:51.483 INFO:teuthology.orchestra.run.smithi027.stdout:(3/23): libcephfs2-17.2.7-945.g72b0874a.el8.x86 3.6 MB/s | 798 kB 00:00 2024-05-15T12:34:51.568 INFO:teuthology.orchestra.run.smithi120.stdout:(12/23): librdkafka-1.6.1-1.el8.x86_64.rpm 2.5 MB/s | 677 kB 00:00 2024-05-15T12:34:51.616 INFO:teuthology.orchestra.run.smithi027.stdout:(4/23): libradosstriper1-17.2.7-945.g72b0874a.e 3.7 MB/s | 505 kB 00:00 2024-05-15T12:34:51.643 INFO:teuthology.orchestra.run.smithi120.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 3.4 MB/s | 259 kB 00:00 2024-05-15T12:34:51.726 INFO:teuthology.orchestra.run.smithi120.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 529 kB/s | 44 kB 00:00 2024-05-15T12:34:51.843 INFO:teuthology.orchestra.run.smithi120.stdout:(15/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 403 kB/s | 47 kB 00:00 2024-05-15T12:34:51.996 INFO:teuthology.orchestra.run.smithi177.stdout:(5/23): librgw2-17.2.7-945.g72b0874a.el8.x86_64 5.6 MB/s | 5.6 MB 00:01 2024-05-15T12:34:52.077 INFO:teuthology.orchestra.run.smithi120.stdout:(16/23): ceph-base-17.2.7-945.g72b0874a.el8.x86 1.1 MB/s | 5.3 MB 00:04 2024-05-15T12:34:52.097 INFO:teuthology.orchestra.run.smithi177.stdout:(6/23): python3-ceph-argparse-17.2.7-945.g72b08 451 kB/s | 45 kB 00:00 2024-05-15T12:34:52.196 INFO:teuthology.orchestra.run.smithi177.stdout:(7/23): python3-ceph-common-17.2.7-945.g72b0874 1.1 MB/s | 117 kB 00:00 2024-05-15T12:34:52.252 INFO:teuthology.orchestra.run.smithi120.stdout:(17/23): gperftools-libs-2.7-9.el8.x86_64.rpm 749 kB/s | 306 kB 00:00 2024-05-15T12:34:52.280 INFO:teuthology.orchestra.run.smithi177.stdout:(8/23): python3-cephfs-17.2.7-945.g72b0874a.el8 2.2 MB/s | 190 kB 00:00 2024-05-15T12:34:52.333 INFO:teuthology.orchestra.run.smithi027.stdout:(5/23): librgw2-17.2.7-945.g72b0874a.el8.x86_64 7.8 MB/s | 5.6 MB 00:00 2024-05-15T12:34:52.336 INFO:teuthology.orchestra.run.smithi120.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 900 kB/s | 75 kB 00:00 2024-05-15T12:34:52.355 INFO:teuthology.orchestra.run.smithi177.stdout:(9/23): python3-rados-17.2.7-945.g72b0874a.el8. 4.8 MB/s | 371 kB 00:00 2024-05-15T12:34:52.362 INFO:teuthology.orchestra.run.smithi120.stdout:(19/23): liboath-2.6.2-3.el8.x86_64.rpm 207 kB/s | 59 kB 00:00 2024-05-15T12:34:52.442 INFO:teuthology.orchestra.run.smithi027.stdout:(6/23): python3-ceph-argparse-17.2.7-945.g72b08 413 kB/s | 45 kB 00:00 2024-05-15T12:34:52.447 INFO:teuthology.orchestra.run.smithi177.stdout:(10/23): python3-rbd-17.2.7-945.g72b0874a.el8.x 3.8 MB/s | 358 kB 00:00 2024-05-15T12:34:52.531 INFO:teuthology.orchestra.run.smithi177.stdout:(11/23): python3-rgw-17.2.7-945.g72b0874a.el8.x 1.3 MB/s | 107 kB 00:00 2024-05-15T12:34:52.550 INFO:teuthology.orchestra.run.smithi027.stdout:(7/23): python3-ceph-common-17.2.7-945.g72b0874 1.1 MB/s | 117 kB 00:00 2024-05-15T12:34:52.562 INFO:teuthology.orchestra.run.smithi120.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 7.7 MB/s | 1.7 MB 00:00 2024-05-15T12:34:52.676 INFO:teuthology.orchestra.run.smithi027.stdout:(8/23): python3-cephfs-17.2.7-945.g72b0874a.el8 1.5 MB/s | 190 kB 00:00 2024-05-15T12:34:52.756 INFO:teuthology.orchestra.run.smithi177.stdout:(12/23): librdkafka-1.6.1-1.el8.x86_64.rpm 2.9 MB/s | 677 kB 00:00 2024-05-15T12:34:52.784 INFO:teuthology.orchestra.run.smithi027.stdout:(9/23): python3-rados-17.2.7-945.g72b0874a.el8. 3.3 MB/s | 371 kB 00:00 2024-05-15T12:34:52.815 INFO:teuthology.orchestra.run.smithi177.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 4.4 MB/s | 259 kB 00:00 2024-05-15T12:34:52.865 INFO:teuthology.orchestra.run.smithi177.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 880 kB/s | 44 kB 00:00 2024-05-15T12:34:52.893 INFO:teuthology.orchestra.run.smithi027.stdout:(10/23): python3-rbd-17.2.7-945.g72b0874a.el8.x 3.2 MB/s | 358 kB 00:00 2024-05-15T12:34:52.965 INFO:teuthology.orchestra.run.smithi177.stdout:(15/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 471 kB/s | 47 kB 00:00 2024-05-15T12:34:53.035 INFO:teuthology.orchestra.run.smithi027.stdout:(11/23): python3-rgw-17.2.7-945.g72b0874a.el8.x 755 kB/s | 107 kB 00:00 2024-05-15T12:34:53.132 INFO:teuthology.orchestra.run.smithi177.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.8 MB/s | 306 kB 00:00 2024-05-15T12:34:53.166 INFO:teuthology.orchestra.run.smithi177.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 1.7 MB/s | 59 kB 00:00 2024-05-15T12:34:53.208 INFO:teuthology.orchestra.run.smithi177.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.8 MB/s | 75 kB 00:00 2024-05-15T12:34:53.288 INFO:teuthology.orchestra.run.smithi027.stdout:(12/23): librdkafka-1.6.1-1.el8.x86_64.rpm 2.6 MB/s | 677 kB 00:00 2024-05-15T12:34:53.352 INFO:teuthology.orchestra.run.smithi027.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 4.0 MB/s | 259 kB 00:00 2024-05-15T12:34:53.354 INFO:teuthology.orchestra.run.smithi120.stdout:(21/23): librbd1-17.2.7-945.g72b0874a.el8.x86_6 4.5 MB/s | 3.6 MB 00:00 2024-05-15T12:34:53.402 INFO:teuthology.orchestra.run.smithi027.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 877 kB/s | 44 kB 00:00 2024-05-15T12:34:53.517 INFO:teuthology.orchestra.run.smithi177.stdout:(19/23): thrift-0.13.0-2.el8.x86_64.rpm 5.6 MB/s | 1.7 MB 00:00 2024-05-15T12:34:53.520 INFO:teuthology.orchestra.run.smithi027.stdout:(15/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 403 kB/s | 47 kB 00:00 2024-05-15T12:34:53.684 INFO:teuthology.orchestra.run.smithi177.stdout:(20/23): ceph-base-17.2.7-945.g72b0874a.el8.x86 1.1 MB/s | 5.3 MB 00:05 2024-05-15T12:34:53.954 INFO:teuthology.orchestra.run.smithi027.stdout:(16/23): ceph-radosgw-17.2.7-945.g72b0874a.el8. 3.8 MB/s | 17 MB 00:04 2024-05-15T12:34:54.120 INFO:teuthology.orchestra.run.smithi027.stdout:(17/23): gperftools-libs-2.7-9.el8.x86_64.rpm 510 kB/s | 306 kB 00:00 2024-05-15T12:34:54.142 INFO:teuthology.orchestra.run.smithi177.stdout:(21/23): librados2-17.2.7-945.g72b0874a.el8.x86 5.5 MB/s | 3.5 MB 00:00 2024-05-15T12:34:54.229 INFO:teuthology.orchestra.run.smithi027.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 697 kB/s | 75 kB 00:00 2024-05-15T12:34:54.379 INFO:teuthology.orchestra.run.smithi027.stdout:(19/23): liboath-2.6.2-3.el8.x86_64.rpm 138 kB/s | 59 kB 00:00 2024-05-15T12:34:54.621 INFO:teuthology.orchestra.run.smithi027.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 4.4 MB/s | 1.7 MB 00:00 2024-05-15T12:34:55.205 INFO:teuthology.orchestra.run.smithi027.stdout:(21/23): librados2-17.2.7-945.g72b0874a.el8.x86 4.2 MB/s | 3.5 MB 00:00 2024-05-15T12:34:55.213 INFO:teuthology.orchestra.run.smithi120.stdout:(22/23): librados2-17.2.7-945.g72b0874a.el8.x86 1.2 MB/s | 3.5 MB 00:02 2024-05-15T12:34:55.530 INFO:teuthology.orchestra.run.smithi027.stdout:(22/23): librbd1-17.2.7-945.g72b0874a.el8.x86_6 3.9 MB/s | 3.6 MB 00:00 2024-05-15T12:34:56.375 INFO:teuthology.orchestra.run.smithi177.stdout:(22/23): librbd1-17.2.7-945.g72b0874a.el8.x86_6 1.3 MB/s | 3.6 MB 00:02 2024-05-15T12:34:58.223 INFO:teuthology.orchestra.run.smithi120.stdout:(23/23): ceph-radosgw-17.2.7-945.g72b0874a.el8. 1.6 MB/s | 17 MB 00:10 2024-05-15T12:34:58.227 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:34:58.227 INFO:teuthology.orchestra.run.smithi120.stdout:Total 5.8 MB/s | 64 MB 00:10 2024-05-15T12:34:58.270 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-05-15T12:34:58.316 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-05-15T12:34:58.316 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-05-15T12:34:58.695 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-05-15T12:34:58.698 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-05-15T12:34:59.735 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-05-15T12:34:59.986 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-05-15T12:35:00.179 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-05-15T12:35:00.560 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-05-15T12:35:00.770 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : librados2-2:17.2.7-945.g72b0874a.el8.x86_64 3/25 2024-05-15T12:35:00.864 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-2:17.2.7-945.g72b0874a.el8.x86_64 3/25 2024-05-15T12:35:01.125 INFO:teuthology.orchestra.run.smithi177.stdout:(23/23): ceph-radosgw-17.2.7-945.g72b0874a.el8. 1.3 MB/s | 17 MB 00:12 2024-05-15T12:35:01.126 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rados-2:17.2.7-945.g72b0874a.el8.x86_64 4/25 2024-05-15T12:35:01.128 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:35:01.128 INFO:teuthology.orchestra.run.smithi177.stdout:Total 5.0 MB/s | 64 MB 00:12 2024-05-15T12:35:01.170 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-05-15T12:35:01.216 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-05-15T12:35:01.216 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-05-15T12:35:01.320 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs2-2:17.2.7-945.g72b0874a.el8.x86_64 5/25 2024-05-15T12:35:01.572 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-05-15T12:35:01.575 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-05-15T12:35:01.697 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephfs2-2:17.2.7-945.g72b0874a.el8.x86_64 5/25 2024-05-15T12:35:01.903 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : librbd1-2:17.2.7-945.g72b0874a.el8.x86_64 6/25 2024-05-15T12:35:02.127 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-2:17.2.7-945.g72b0874a.el8.x86_64 6/25 2024-05-15T12:35:02.350 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2024-05-15T12:35:02.554 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-05-15T12:35:02.813 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-05-15T12:35:03.021 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-05-15T12:35:03.206 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-05-15T12:35:03.258 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-05-15T12:35:03.350 INFO:teuthology.orchestra.run.smithi027.stdout:(23/23): ceph-common-17.2.7-945.g72b0874a.el8.x 1.7 MB/s | 24 MB 00:13 2024-05-15T12:35:03.354 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:35:03.354 INFO:teuthology.orchestra.run.smithi027.stdout:Total 4.3 MB/s | 64 MB 00:14 2024-05-15T12:35:03.398 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-05-15T12:35:03.437 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librgw2-2:17.2.7-945.g72b0874a.el8.x86_64 10/25 2024-05-15T12:35:03.449 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-05-15T12:35:03.449 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-05-15T12:35:03.506 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librgw2-2:17.2.7-945.g72b0874a.el8.x86_64 10/25 2024-05-15T12:35:03.581 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-05-15T12:35:03.647 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ceph-argparse-2:17.2.7-945.g72b0874a.el8.x 11/25 2024-05-15T12:35:03.798 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librados2-2:17.2.7-945.g72b0874a.el8.x86_64 3/25 2024-05-15T12:35:03.846 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-05-15T12:35:03.849 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-05-15T12:35:03.893 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:17.2.7-945.g72b0874a.el8.x86_64 3/25 2024-05-15T12:35:04.032 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cephfs-2:17.2.7-945.g72b0874a.el8.x86_64 12/25 2024-05-15T12:35:04.129 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rados-2:17.2.7-945.g72b0874a.el8.x86_64 4/25 2024-05-15T12:35:04.239 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rgw-2:17.2.7-945.g72b0874a.el8.x86_64 13/25 2024-05-15T12:35:04.297 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs2-2:17.2.7-945.g72b0874a.el8.x86_64 5/25 2024-05-15T12:35:04.457 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rbd-2:17.2.7-945.g72b0874a.el8.x86_64 14/25 2024-05-15T12:35:04.620 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libradosstriper1-2:17.2.7-945.g72b0874a.el8.x86_64 15/25 2024-05-15T12:35:04.684 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libradosstriper1-2:17.2.7-945.g72b0874a.el8.x86_64 15/25 2024-05-15T12:35:04.691 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs2-2:17.2.7-945.g72b0874a.el8.x86_64 5/25 2024-05-15T12:35:04.905 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-05-15T12:35:04.922 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librbd1-2:17.2.7-945.g72b0874a.el8.x86_64 6/25 2024-05-15T12:35:04.954 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-05-15T12:35:05.130 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-05-15T12:35:05.136 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:17.2.7-945.g72b0874a.el8.x86_64 6/25 2024-05-15T12:35:05.155 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-05-15T12:35:05.324 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-05-15T12:35:05.359 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-05-15T12:35:05.369 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2024-05-15T12:35:05.504 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ceph-common-2:17.2.7-945.g72b0874a.el8.x86 19/25 2024-05-15T12:35:05.607 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-05-15T12:35:05.757 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-05-15T12:35:05.942 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librados2-2:17.2.7-945.g72b0874a.el8.x86_64 3/25 2024-05-15T12:35:06.078 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:17.2.7-945.g72b0874a.el8.x86_64 3/25 2024-05-15T12:35:06.252 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-05-15T12:35:06.306 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rados-2:17.2.7-945.g72b0874a.el8.x86_64 4/25 2024-05-15T12:35:06.414 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librgw2-2:17.2.7-945.g72b0874a.el8.x86_64 10/25 2024-05-15T12:35:06.467 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librgw2-2:17.2.7-945.g72b0874a.el8.x86_64 10/25 2024-05-15T12:35:06.475 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs2-2:17.2.7-945.g72b0874a.el8.x86_64 5/25 2024-05-15T12:35:06.615 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-argparse-2:17.2.7-945.g72b0874a.el8.x 11/25 2024-05-15T12:35:06.784 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cephfs-2:17.2.7-945.g72b0874a.el8.x86_64 12/25 2024-05-15T12:35:06.920 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs2-2:17.2.7-945.g72b0874a.el8.x86_64 5/25 2024-05-15T12:35:06.990 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rgw-2:17.2.7-945.g72b0874a.el8.x86_64 13/25 2024-05-15T12:35:07.208 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librbd1-2:17.2.7-945.g72b0874a.el8.x86_64 6/25 2024-05-15T12:35:07.217 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rbd-2:17.2.7-945.g72b0874a.el8.x86_64 14/25 2024-05-15T12:35:07.380 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libradosstriper1-2:17.2.7-945.g72b0874a.el8.x86_64 15/25 2024-05-15T12:35:07.443 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:17.2.7-945.g72b0874a.el8.x86_64 6/25 2024-05-15T12:35:07.487 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libradosstriper1-2:17.2.7-945.g72b0874a.el8.x86_64 15/25 2024-05-15T12:35:07.697 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2024-05-15T12:35:07.789 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-05-15T12:35:07.900 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-05-15T12:35:07.971 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-05-15T12:35:08.163 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-05-15T12:35:08.218 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-2:17.2.7-945.g72b0874a.el8.x86_64 20/25 2024-05-15T12:35:08.339 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-common-2:17.2.7-945.g72b0874a.el8.x86 19/25 2024-05-15T12:35:08.478 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-common-2:17.2.7-945.g72b0874a.el8.x86_64 20/25 2024-05-15T12:35:08.563 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-05-15T12:35:08.758 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librgw2-2:17.2.7-945.g72b0874a.el8.x86_64 10/25 2024-05-15T12:35:08.824 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librgw2-2:17.2.7-945.g72b0874a.el8.x86_64 10/25 2024-05-15T12:35:09.028 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-argparse-2:17.2.7-945.g72b0874a.el8.x 11/25 2024-05-15T12:35:09.079 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-2:17.2.7-945.g72b0874a.el8.x86_64 20/25 2024-05-15T12:35:09.231 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cephfs-2:17.2.7-945.g72b0874a.el8.x86_64 12/25 2024-05-15T12:35:09.429 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rgw-2:17.2.7-945.g72b0874a.el8.x86_64 13/25 2024-05-15T12:35:09.550 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-base-2:17.2.7-945.g72b0874a.el8.x86_64 21/25 2024-05-15T12:35:09.656 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rbd-2:17.2.7-945.g72b0874a.el8.x86_64 14/25 2024-05-15T12:35:09.685 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-base-2:17.2.7-945.g72b0874a.el8.x86_64 21/25 2024-05-15T12:35:09.841 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libradosstriper1-2:17.2.7-945.g72b0874a.el8.x86_64 15/25 2024-05-15T12:35:09.852 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-selinux-2:17.2.7-945.g72b0874a.el8.x86_64 22/25 2024-05-15T12:35:09.920 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libradosstriper1-2:17.2.7-945.g72b0874a.el8.x86_64 15/25 2024-05-15T12:35:10.161 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-05-15T12:35:10.350 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-05-15T12:35:10.580 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-05-15T12:35:10.817 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-common-2:17.2.7-945.g72b0874a.el8.x86 19/25 2024-05-15T12:35:10.988 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:17.2.7-945.g72b0874a.el8.x86_64 20/25 2024-05-15T12:35:11.245 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-common-2:17.2.7-945.g72b0874a.el8.x86_64 20/25 2024-05-15T12:35:11.834 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:17.2.7-945.g72b0874a.el8.x86_64 20/25 2024-05-15T12:35:12.094 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-base-2:17.2.7-945.g72b0874a.el8.x86_64 21/25 2024-05-15T12:35:12.204 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-base-2:17.2.7-945.g72b0874a.el8.x86_64 21/25 2024-05-15T12:35:12.388 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-selinux-2:17.2.7-945.g72b0874a.el8.x86_64 22/25 2024-05-15T12:35:13.737 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:17.2.7-945.g72b0874a.el8.x86_64 20/25 2024-05-15T12:35:14.019 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-common-2:17.2.7-945.g72b0874a.el8.x86_64 20/25 2024-05-15T12:35:14.643 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:17.2.7-945.g72b0874a.el8.x86_64 20/25 2024-05-15T12:35:14.948 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-base-2:17.2.7-945.g72b0874a.el8.x86_64 21/25 2024-05-15T12:35:15.050 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-base-2:17.2.7-945.g72b0874a.el8.x86_64 21/25 2024-05-15T12:35:15.224 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-selinux-2:17.2.7-945.g72b0874a.el8.x86_64 22/25 2024-05-15T12:35:29.098 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-selinux-2:17.2.7-945.g72b0874a.el8.x86_64 22/25 2024-05-15T12:35:29.098 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /sys 2024-05-15T12:35:29.098 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /proc 2024-05-15T12:35:29.098 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /dev 2024-05-15T12:35:29.098 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /run 2024-05-15T12:35:29.099 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /mnt 2024-05-15T12:35:29.099 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /var/tmp 2024-05-15T12:35:29.099 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /home 2024-05-15T12:35:29.099 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /tmp 2024-05-15T12:35:29.099 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /dev 2024-05-15T12:35:29.099 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:35:30.788 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-radosgw-2:17.2.7-945.g72b0874a.el8.x86_64 23/25 2024-05-15T12:35:30.864 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-945.g72b0874a.el8.x86_64 23/25 2024-05-15T12:35:30.865 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-05-15T12:35:31.068 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-05-15T12:35:31.068 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-05-15T12:35:31.410 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-selinux-2:17.2.7-945.g72b0874a.el8.x86_64 22/25 2024-05-15T12:35:31.411 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /sys 2024-05-15T12:35:31.411 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /proc 2024-05-15T12:35:31.411 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /dev 2024-05-15T12:35:31.411 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /run 2024-05-15T12:35:31.411 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /mnt 2024-05-15T12:35:31.411 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /var/tmp 2024-05-15T12:35:31.411 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /home 2024-05-15T12:35:31.411 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /tmp 2024-05-15T12:35:31.411 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /dev 2024-05-15T12:35:31.411 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:35:31.773 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-05-15T12:35:31.774 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-base-2:17.2.7-945.g72b0874a.el8.x86_64 1/25 2024-05-15T12:35:31.774 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-common-2:17.2.7-945.g72b0874a.el8.x86_64 2/25 2024-05-15T12:35:31.774 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-radosgw-2:17.2.7-945.g72b0874a.el8.x86_64 3/25 2024-05-15T12:35:31.774 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-selinux-2:17.2.7-945.g72b0874a.el8.x86_64 4/25 2024-05-15T12:35:31.774 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs2-2:17.2.7-945.g72b0874a.el8.x86_64 5/25 2024-05-15T12:35:31.774 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libradosstriper1-2:17.2.7-945.g72b0874a.el8.x86_64 6/25 2024-05-15T12:35:31.774 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librgw2-2:17.2.7-945.g72b0874a.el8.x86_64 7/25 2024-05-15T12:35:31.774 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ceph-argparse-2:17.2.7-945.g72b0874a.el8.x 8/25 2024-05-15T12:35:31.774 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ceph-common-2:17.2.7-945.g72b0874a.el8.x86 9/25 2024-05-15T12:35:31.774 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cephfs-2:17.2.7-945.g72b0874a.el8.x86_64 10/25 2024-05-15T12:35:31.774 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rados-2:17.2.7-945.g72b0874a.el8.x86_64 11/25 2024-05-15T12:35:31.774 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rbd-2:17.2.7-945.g72b0874a.el8.x86_64 12/25 2024-05-15T12:35:31.775 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rgw-2:17.2.7-945.g72b0874a.el8.x86_64 13/25 2024-05-15T12:35:31.775 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-05-15T12:35:31.775 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-05-15T12:35:31.775 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-05-15T12:35:31.775 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-05-15T12:35:31.775 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-05-15T12:35:31.775 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-05-15T12:35:31.775 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-05-15T12:35:31.775 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-05-15T12:35:31.775 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-2:17.2.7-945.g72b0874a.el8.x86_64 22/25 2024-05-15T12:35:31.775 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-05-15T12:35:31.775 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-2:17.2.7-945.g72b0874a.el8.x86_64 24/25 2024-05-15T12:35:32.666 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-05-15T12:35:32.666 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:35:32.666 INFO:teuthology.orchestra.run.smithi120.stdout:Upgraded: 2024-05-15T12:35:32.666 INFO:teuthology.orchestra.run.smithi120.stdout: librados2-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:32.666 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:32.666 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-05-15T12:35:32.666 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:32.667 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:32.667 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:32.667 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-selinux-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:32.667 INFO:teuthology.orchestra.run.smithi120.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-05-15T12:35:32.667 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:32.667 INFO:teuthology.orchestra.run.smithi120.stdout: liboath-2.6.2-3.el8.x86_64 2024-05-15T12:35:32.667 INFO:teuthology.orchestra.run.smithi120.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-05-15T12:35:32.667 INFO:teuthology.orchestra.run.smithi120.stdout: libradosstriper1-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:32.667 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-05-15T12:35:32.667 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:32.667 INFO:teuthology.orchestra.run.smithi120.stdout: libunwind-1.3.1-3.el8.x86_64 2024-05-15T12:35:32.667 INFO:teuthology.orchestra.run.smithi120.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-05-15T12:35:32.667 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:32.668 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-common-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:32.668 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:32.668 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-05-15T12:35:32.668 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:32.668 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:32.668 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:32.668 INFO:teuthology.orchestra.run.smithi120.stdout: thrift-0.13.0-2.el8.x86_64 2024-05-15T12:35:32.668 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:35:32.668 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-05-15T12:35:32.956 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-test 2024-05-15T12:35:33.098 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-radosgw-2:17.2.7-945.g72b0874a.el8.x86_64 23/25 2024-05-15T12:35:33.169 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-945.g72b0874a.el8.x86_64 23/25 2024-05-15T12:35:33.169 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-05-15T12:35:33.369 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-05-15T12:35:33.370 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-05-15T12:35:33.683 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:00:50 ago on Wed 15 May 2024 12:34:43 PM UTC. 2024-05-15T12:35:34.042 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-05-15T12:35:34.042 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-base-2:17.2.7-945.g72b0874a.el8.x86_64 1/25 2024-05-15T12:35:34.042 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-common-2:17.2.7-945.g72b0874a.el8.x86_64 2/25 2024-05-15T12:35:34.043 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-radosgw-2:17.2.7-945.g72b0874a.el8.x86_64 3/25 2024-05-15T12:35:34.043 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-selinux-2:17.2.7-945.g72b0874a.el8.x86_64 4/25 2024-05-15T12:35:34.043 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs2-2:17.2.7-945.g72b0874a.el8.x86_64 5/25 2024-05-15T12:35:34.043 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libradosstriper1-2:17.2.7-945.g72b0874a.el8.x86_64 6/25 2024-05-15T12:35:34.043 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librgw2-2:17.2.7-945.g72b0874a.el8.x86_64 7/25 2024-05-15T12:35:34.043 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-argparse-2:17.2.7-945.g72b0874a.el8.x 8/25 2024-05-15T12:35:34.043 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-common-2:17.2.7-945.g72b0874a.el8.x86 9/25 2024-05-15T12:35:34.043 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cephfs-2:17.2.7-945.g72b0874a.el8.x86_64 10/25 2024-05-15T12:35:34.043 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rados-2:17.2.7-945.g72b0874a.el8.x86_64 11/25 2024-05-15T12:35:34.043 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rbd-2:17.2.7-945.g72b0874a.el8.x86_64 12/25 2024-05-15T12:35:34.043 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rgw-2:17.2.7-945.g72b0874a.el8.x86_64 13/25 2024-05-15T12:35:34.043 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-05-15T12:35:34.043 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-05-15T12:35:34.043 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-05-15T12:35:34.044 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-05-15T12:35:34.044 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-05-15T12:35:34.044 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-05-15T12:35:34.044 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-05-15T12:35:34.044 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-05-15T12:35:34.044 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:17.2.7-945.g72b0874a.el8.x86_64 22/25 2024-05-15T12:35:34.044 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-05-15T12:35:34.044 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:17.2.7-945.g72b0874a.el8.x86_64 24/25 2024-05-15T12:35:34.880 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-05-15T12:35:34.880 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:35:34.880 INFO:teuthology.orchestra.run.smithi177.stdout:Upgraded: 2024-05-15T12:35:34.881 INFO:teuthology.orchestra.run.smithi177.stdout: librados2-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:34.881 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:34.881 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-05-15T12:35:34.881 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:34.881 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:34.881 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:34.881 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:34.881 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-05-15T12:35:34.881 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:34.881 INFO:teuthology.orchestra.run.smithi177.stdout: liboath-2.6.2-3.el8.x86_64 2024-05-15T12:35:34.881 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-05-15T12:35:34.881 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:34.881 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-05-15T12:35:34.881 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:34.881 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind-1.3.1-3.el8.x86_64 2024-05-15T12:35:34.882 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-05-15T12:35:34.882 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:34.882 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:34.882 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:34.882 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-05-15T12:35:34.882 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:34.882 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:34.882 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:34.882 INFO:teuthology.orchestra.run.smithi177.stdout: thrift-0.13.0-2.el8.x86_64 2024-05-15T12:35:34.882 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:35:34.882 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-05-15T12:35:35.156 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-test 2024-05-15T12:35:35.318 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-selinux-2:17.2.7-945.g72b0874a.el8.x86_64 22/25 2024-05-15T12:35:35.319 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /sys 2024-05-15T12:35:35.319 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /proc 2024-05-15T12:35:35.319 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /dev 2024-05-15T12:35:35.319 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /run 2024-05-15T12:35:35.319 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /mnt 2024-05-15T12:35:35.319 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /var/tmp 2024-05-15T12:35:35.319 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /home 2024-05-15T12:35:35.319 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /tmp 2024-05-15T12:35:35.319 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /dev 2024-05-15T12:35:35.319 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:35:35.339 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-05-15T12:35:35.340 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:35:35.340 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-05-15T12:35:35.340 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:35:35.340 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-05-15T12:35:35.340 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test x86_64 2:17.2.7-945.g72b0874a.el8 ceph 53 M 2024-05-15T12:35:35.340 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-05-15T12:35:35.340 INFO:teuthology.orchestra.run.smithi120.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-05-15T12:35:35.341 INFO:teuthology.orchestra.run.smithi120.stdout: libcephsqlite x86_64 2:17.2.7-945.g72b0874a.el8 ceph 179 k 2024-05-15T12:35:35.341 INFO:teuthology.orchestra.run.smithi120.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-05-15T12:35:35.341 INFO:teuthology.orchestra.run.smithi120.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-05-15T12:35:35.341 INFO:teuthology.orchestra.run.smithi120.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-05-15T12:35:35.341 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:35:35.341 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-05-15T12:35:35.341 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:35:35.341 INFO:teuthology.orchestra.run.smithi120.stdout:Install 6 Packages 2024-05-15T12:35:35.341 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:35:35.342 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 54 M 2024-05-15T12:35:35.342 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 236 M 2024-05-15T12:35:35.342 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-05-15T12:35:35.773 INFO:teuthology.orchestra.run.smithi120.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 849 kB/s | 203 kB 00:00 2024-05-15T12:35:35.887 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:51 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:35:35.890 INFO:teuthology.orchestra.run.smithi120.stdout:(2/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-05-15T12:35:35.949 INFO:teuthology.orchestra.run.smithi120.stdout:(3/6): libcephsqlite-17.2.7-945.g72b0874a.el8.x 431 kB/s | 179 kB 00:00 2024-05-15T12:35:35.999 INFO:teuthology.orchestra.run.smithi120.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.9 MB/s | 323 kB 00:00 2024-05-15T12:35:36.108 INFO:teuthology.orchestra.run.smithi120.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 445 kB/s | 71 kB 00:00 2024-05-15T12:35:37.136 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-radosgw-2:17.2.7-945.g72b0874a.el8.x86_64 23/25 2024-05-15T12:35:37.232 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-945.g72b0874a.el8.x86_64 23/25 2024-05-15T12:35:37.233 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-05-15T12:35:37.432 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-05-15T12:35:37.432 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-05-15T12:35:37.542 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-05-15T12:35:37.543 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:35:37.543 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-05-15T12:35:37.543 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:35:37.543 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-05-15T12:35:37.543 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test x86_64 2:17.2.7-945.g72b0874a.el8 ceph 53 M 2024-05-15T12:35:37.543 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-05-15T12:35:37.543 INFO:teuthology.orchestra.run.smithi177.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-05-15T12:35:37.543 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite x86_64 2:17.2.7-945.g72b0874a.el8 ceph 179 k 2024-05-15T12:35:37.543 INFO:teuthology.orchestra.run.smithi177.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-05-15T12:35:37.543 INFO:teuthology.orchestra.run.smithi177.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-05-15T12:35:37.543 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-05-15T12:35:37.544 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:35:37.544 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-05-15T12:35:37.544 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:35:37.544 INFO:teuthology.orchestra.run.smithi177.stdout:Install 6 Packages 2024-05-15T12:35:37.544 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:35:37.544 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 54 M 2024-05-15T12:35:37.544 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 236 M 2024-05-15T12:35:37.545 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-05-15T12:35:37.918 INFO:teuthology.orchestra.run.smithi177.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 703 kB/s | 203 kB 00:00 2024-05-15T12:35:38.051 INFO:teuthology.orchestra.run.smithi177.stdout:(2/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.4 MB/s | 188 kB 00:00 2024-05-15T12:35:38.151 INFO:teuthology.orchestra.run.smithi177.stdout:(3/6): libcephsqlite-17.2.7-945.g72b0874a.el8.x 342 kB/s | 179 kB 00:00 2024-05-15T12:35:38.168 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-05-15T12:35:38.169 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-base-2:17.2.7-945.g72b0874a.el8.x86_64 1/25 2024-05-15T12:35:38.169 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-common-2:17.2.7-945.g72b0874a.el8.x86_64 2/25 2024-05-15T12:35:38.169 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-radosgw-2:17.2.7-945.g72b0874a.el8.x86_64 3/25 2024-05-15T12:35:38.169 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-selinux-2:17.2.7-945.g72b0874a.el8.x86_64 4/25 2024-05-15T12:35:38.169 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs2-2:17.2.7-945.g72b0874a.el8.x86_64 5/25 2024-05-15T12:35:38.169 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libradosstriper1-2:17.2.7-945.g72b0874a.el8.x86_64 6/25 2024-05-15T12:35:38.169 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librgw2-2:17.2.7-945.g72b0874a.el8.x86_64 7/25 2024-05-15T12:35:38.169 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-argparse-2:17.2.7-945.g72b0874a.el8.x 8/25 2024-05-15T12:35:38.169 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-common-2:17.2.7-945.g72b0874a.el8.x86 9/25 2024-05-15T12:35:38.169 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cephfs-2:17.2.7-945.g72b0874a.el8.x86_64 10/25 2024-05-15T12:35:38.169 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rados-2:17.2.7-945.g72b0874a.el8.x86_64 11/25 2024-05-15T12:35:38.169 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rbd-2:17.2.7-945.g72b0874a.el8.x86_64 12/25 2024-05-15T12:35:38.169 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rgw-2:17.2.7-945.g72b0874a.el8.x86_64 13/25 2024-05-15T12:35:38.170 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-05-15T12:35:38.170 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-05-15T12:35:38.170 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-05-15T12:35:38.170 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-05-15T12:35:38.170 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-05-15T12:35:38.170 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-05-15T12:35:38.170 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-05-15T12:35:38.170 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-05-15T12:35:38.170 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:17.2.7-945.g72b0874a.el8.x86_64 22/25 2024-05-15T12:35:38.170 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-05-15T12:35:38.170 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:17.2.7-945.g72b0874a.el8.x86_64 24/25 2024-05-15T12:35:38.176 INFO:teuthology.orchestra.run.smithi177.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.5 MB/s | 323 kB 00:00 2024-05-15T12:35:38.302 INFO:teuthology.orchestra.run.smithi177.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 470 kB/s | 71 kB 00:00 2024-05-15T12:35:39.176 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-05-15T12:35:39.177 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:35:39.177 INFO:teuthology.orchestra.run.smithi027.stdout:Upgraded: 2024-05-15T12:35:39.177 INFO:teuthology.orchestra.run.smithi027.stdout: librados2-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:39.177 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:39.177 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-05-15T12:35:39.177 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:39.177 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:39.177 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:39.177 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:39.177 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-05-15T12:35:39.177 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:39.177 INFO:teuthology.orchestra.run.smithi027.stdout: liboath-2.6.2-3.el8.x86_64 2024-05-15T12:35:39.177 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-05-15T12:35:39.178 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:39.178 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-05-15T12:35:39.178 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:39.178 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind-1.3.1-3.el8.x86_64 2024-05-15T12:35:39.178 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-05-15T12:35:39.178 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:39.178 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:39.178 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:39.178 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-05-15T12:35:39.178 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:39.178 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:39.178 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:39.178 INFO:teuthology.orchestra.run.smithi027.stdout: thrift-0.13.0-2.el8.x86_64 2024-05-15T12:35:39.179 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:35:39.179 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-15T12:35:39.459 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-test 2024-05-15T12:35:40.217 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:56 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:35:41.085 INFO:teuthology.orchestra.run.smithi177.stdout:(6/6): ceph-test-17.2.7-945.g72b0874a.el8.x86_6 15 MB/s | 53 MB 00:03 2024-05-15T12:35:41.087 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:35:41.087 INFO:teuthology.orchestra.run.smithi177.stdout:Total 15 MB/s | 54 MB 00:03 2024-05-15T12:35:41.129 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-05-15T12:35:41.150 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-05-15T12:35:41.150 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-05-15T12:35:41.455 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-05-15T12:35:41.458 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-05-15T12:35:41.591 INFO:teuthology.orchestra.run.smithi120.stdout:(6/6): ceph-test-17.2.7-945.g72b0874a.el8.x86_6 8.7 MB/s | 53 MB 00:06 2024-05-15T12:35:41.593 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:35:41.593 INFO:teuthology.orchestra.run.smithi120.stdout:Total 8.6 MB/s | 54 MB 00:06 2024-05-15T12:35:41.634 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-05-15T12:35:41.655 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-05-15T12:35:41.655 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-05-15T12:35:41.937 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-05-15T12:35:41.974 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-05-15T12:35:41.977 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-05-15T12:35:42.000 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-15T12:35:42.001 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:35:42.001 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-05-15T12:35:42.001 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:35:42.001 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-05-15T12:35:42.001 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test x86_64 2:17.2.7-945.g72b0874a.el8 ceph 53 M 2024-05-15T12:35:42.001 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-05-15T12:35:42.001 INFO:teuthology.orchestra.run.smithi027.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-05-15T12:35:42.001 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite x86_64 2:17.2.7-945.g72b0874a.el8 ceph 179 k 2024-05-15T12:35:42.001 INFO:teuthology.orchestra.run.smithi027.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-05-15T12:35:42.001 INFO:teuthology.orchestra.run.smithi027.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-05-15T12:35:42.001 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-05-15T12:35:42.001 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:35:42.002 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-05-15T12:35:42.002 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:35:42.002 INFO:teuthology.orchestra.run.smithi027.stdout:Install 6 Packages 2024-05-15T12:35:42.002 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:35:42.002 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 54 M 2024-05-15T12:35:42.002 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 236 M 2024-05-15T12:35:42.003 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-05-15T12:35:42.195 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-05-15T12:35:42.353 INFO:teuthology.orchestra.run.smithi027.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 710 kB/s | 203 kB 00:00 2024-05-15T12:35:42.377 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-05-15T12:35:42.453 INFO:teuthology.orchestra.run.smithi027.stdout:(2/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-05-15T12:35:42.476 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-05-15T12:35:42.541 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-05-15T12:35:42.562 INFO:teuthology.orchestra.run.smithi027.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.9 MB/s | 323 kB 00:00 2024-05-15T12:35:42.606 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-05-15T12:35:42.637 INFO:teuthology.orchestra.run.smithi027.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 942 kB/s | 71 kB 00:00 2024-05-15T12:35:42.708 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-05-15T12:35:42.798 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-05-15T12:35:42.934 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-05-15T12:35:42.974 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephsqlite-2:17.2.7-945.g72b0874a.el8.x86_64 5/6 2024-05-15T12:35:42.979 INFO:teuthology.orchestra.run.smithi027.stdout:(5/6): libcephsqlite-17.2.7-945.g72b0874a.el8.x 196 kB/s | 179 kB 00:00 2024-05-15T12:35:43.080 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-05-15T12:35:43.162 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-05-15T12:35:43.371 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-05-15T12:35:43.538 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephsqlite-2:17.2.7-945.g72b0874a.el8.x86_64 5/6 2024-05-15T12:35:47.446 INFO:teuthology.orchestra.run.smithi027.stdout:(6/6): ceph-test-17.2.7-945.g72b0874a.el8.x86_6 9.8 MB/s | 53 MB 00:05 2024-05-15T12:35:47.447 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:35:47.448 INFO:teuthology.orchestra.run.smithi027.stdout:Total 9.8 MB/s | 54 MB 00:05 2024-05-15T12:35:47.490 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-05-15T12:35:47.512 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-05-15T12:35:47.513 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-05-15T12:35:47.840 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-05-15T12:35:47.843 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-05-15T12:35:48.061 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephsqlite-2:17.2.7-945.g72b0874a.el8.x86_64 5/6 2024-05-15T12:35:48.305 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-test-2:17.2.7-945.g72b0874a.el8.x86_64 6/6 2024-05-15T12:35:48.371 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-05-15T12:35:48.596 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-test-2:17.2.7-945.g72b0874a.el8.x86_64 6/6 2024-05-15T12:35:48.596 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-test-2:17.2.7-945.g72b0874a.el8.x86_64 1/6 2024-05-15T12:35:48.596 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephsqlite-2:17.2.7-945.g72b0874a.el8.x86_64 2/6 2024-05-15T12:35:48.596 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-05-15T12:35:48.596 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-05-15T12:35:48.596 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-05-15T12:35:48.618 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephsqlite-2:17.2.7-945.g72b0874a.el8.x86_64 5/6 2024-05-15T12:35:48.621 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-05-15T12:35:48.837 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-test-2:17.2.7-945.g72b0874a.el8.x86_64 6/6 2024-05-15T12:35:48.857 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-05-15T12:35:48.993 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-05-15T12:35:48.993 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:35:48.993 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-05-15T12:35:48.994 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:48.994 INFO:teuthology.orchestra.run.smithi177.stdout: jq-1.6-8.el8.x86_64 2024-05-15T12:35:48.994 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:48.994 INFO:teuthology.orchestra.run.smithi177.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-05-15T12:35:48.994 INFO:teuthology.orchestra.run.smithi177.stdout: socat-1.7.4.1-1.el8.x86_64 2024-05-15T12:35:48.994 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-05-15T12:35:48.994 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:35:48.994 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-05-15T12:35:49.003 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-05-15T12:35:49.101 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-05-15T12:35:49.115 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-test-2:17.2.7-945.g72b0874a.el8.x86_64 6/6 2024-05-15T12:35:49.115 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-test-2:17.2.7-945.g72b0874a.el8.x86_64 1/6 2024-05-15T12:35:49.115 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephsqlite-2:17.2.7-945.g72b0874a.el8.x86_64 2/6 2024-05-15T12:35:49.116 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-05-15T12:35:49.116 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-05-15T12:35:49.116 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-05-15T12:35:49.210 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph 2024-05-15T12:35:49.269 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-05-15T12:35:49.436 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephsqlite-2:17.2.7-945.g72b0874a.el8.x86_64 5/6 2024-05-15T12:35:49.559 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-05-15T12:35:49.559 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:35:49.559 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-05-15T12:35:49.559 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:49.559 INFO:teuthology.orchestra.run.smithi120.stdout: jq-1.6-8.el8.x86_64 2024-05-15T12:35:49.559 INFO:teuthology.orchestra.run.smithi120.stdout: libcephsqlite-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:49.559 INFO:teuthology.orchestra.run.smithi120.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-05-15T12:35:49.559 INFO:teuthology.orchestra.run.smithi120.stdout: socat-1.7.4.1-1.el8.x86_64 2024-05-15T12:35:49.560 INFO:teuthology.orchestra.run.smithi120.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-05-15T12:35:49.560 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:35:49.560 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-05-15T12:35:49.823 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph 2024-05-15T12:35:49.947 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:05 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:35:50.557 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:07 ago on Wed 15 May 2024 12:34:43 PM UTC. 2024-05-15T12:35:51.605 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-05-15T12:35:51.608 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:35:51.609 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-05-15T12:35:51.609 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:35:51.609 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-05-15T12:35:51.609 INFO:teuthology.orchestra.run.smithi177.stdout: ceph x86_64 2:17.2.7-945.g72b0874a.el8 ceph 6.4 k 2024-05-15T12:35:51.609 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-05-15T12:35:51.609 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds x86_64 2:17.2.7-945.g72b0874a.el8 ceph 2.3 M 2024-05-15T12:35:51.609 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr x86_64 2:17.2.7-945.g72b0874a.el8 ceph 1.6 M 2024-05-15T12:35:51.609 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core noarch 2:17.2.7-945.g72b0874a.el8 ceph-noarch 240 k 2024-05-15T12:35:51.609 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon x86_64 2:17.2.7-945.g72b0874a.el8 ceph 4.0 M 2024-05-15T12:35:51.609 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd x86_64 2:17.2.7-945.g72b0874a.el8 ceph 17 M 2024-05-15T12:35:51.609 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-05-15T12:35:51.609 INFO:teuthology.orchestra.run.smithi177.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-05-15T12:35:51.609 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-05-15T12:35:51.610 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-05-15T12:35:51.610 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-05-15T12:35:51.610 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-05-15T12:35:51.610 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-05-15T12:35:51.610 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-05-15T12:35:51.610 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-05-15T12:35:51.610 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-05-15T12:35:51.610 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-05-15T12:35:51.610 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-05-15T12:35:51.610 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-05-15T12:35:51.610 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-05-15T12:35:51.610 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-05-15T12:35:51.610 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-05-15T12:35:51.611 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-05-15T12:35:51.611 INFO:teuthology.orchestra.run.smithi177.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-05-15T12:35:51.611 INFO:teuthology.orchestra.run.smithi177.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-05-15T12:35:51.611 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-05-15T12:35:51.611 INFO:teuthology.orchestra.run.smithi177.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-05-15T12:35:51.611 INFO:teuthology.orchestra.run.smithi177.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-05-15T12:35:51.611 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-05-15T12:35:51.611 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-05-15T12:35:51.611 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-05-15T12:35:51.611 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-05-15T12:35:51.611 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:35:51.611 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-05-15T12:35:51.611 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:35:51.612 INFO:teuthology.orchestra.run.smithi177.stdout:Install 32 Packages 2024-05-15T12:35:51.612 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:35:51.612 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 29 M 2024-05-15T12:35:51.612 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 113 M 2024-05-15T12:35:51.612 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-05-15T12:35:52.218 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-05-15T12:35:52.221 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:35:52.221 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-05-15T12:35:52.221 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:35:52.221 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-05-15T12:35:52.221 INFO:teuthology.orchestra.run.smithi120.stdout: ceph x86_64 2:17.2.7-945.g72b0874a.el8 ceph 6.4 k 2024-05-15T12:35:52.222 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-05-15T12:35:52.222 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds x86_64 2:17.2.7-945.g72b0874a.el8 ceph 2.3 M 2024-05-15T12:35:52.222 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr x86_64 2:17.2.7-945.g72b0874a.el8 ceph 1.6 M 2024-05-15T12:35:52.222 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-modules-core noarch 2:17.2.7-945.g72b0874a.el8 ceph-noarch 240 k 2024-05-15T12:35:52.222 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon x86_64 2:17.2.7-945.g72b0874a.el8 ceph 4.0 M 2024-05-15T12:35:52.222 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd x86_64 2:17.2.7-945.g72b0874a.el8 ceph 17 M 2024-05-15T12:35:52.222 INFO:teuthology.orchestra.run.smithi120.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-05-15T12:35:52.222 INFO:teuthology.orchestra.run.smithi120.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-05-15T12:35:52.222 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-05-15T12:35:52.222 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-05-15T12:35:52.222 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-05-15T12:35:52.222 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-05-15T12:35:52.222 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-05-15T12:35:52.223 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-05-15T12:35:52.223 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-05-15T12:35:52.223 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-05-15T12:35:52.223 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-05-15T12:35:52.223 INFO:teuthology.orchestra.run.smithi120.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-05-15T12:35:52.223 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-05-15T12:35:52.223 INFO:teuthology.orchestra.run.smithi120.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-05-15T12:35:52.223 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-05-15T12:35:52.223 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-05-15T12:35:52.223 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-05-15T12:35:52.223 INFO:teuthology.orchestra.run.smithi120.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-05-15T12:35:52.223 INFO:teuthology.orchestra.run.smithi120.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-05-15T12:35:52.223 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-05-15T12:35:52.223 INFO:teuthology.orchestra.run.smithi120.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-05-15T12:35:52.223 INFO:teuthology.orchestra.run.smithi120.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-05-15T12:35:52.224 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-05-15T12:35:52.224 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-05-15T12:35:52.224 INFO:teuthology.orchestra.run.smithi120.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-05-15T12:35:52.224 INFO:teuthology.orchestra.run.smithi120.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-05-15T12:35:52.224 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:35:52.224 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-05-15T12:35:52.224 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:35:52.224 INFO:teuthology.orchestra.run.smithi120.stdout:Install 32 Packages 2024-05-15T12:35:52.224 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:35:52.225 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 29 M 2024-05-15T12:35:52.225 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 113 M 2024-05-15T12:35:52.225 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-05-15T12:35:52.245 INFO:teuthology.orchestra.run.smithi177.stdout:(1/32): ceph-17.2.7-945.g72b0874a.el8.x86_64.rp 24 kB/s | 6.4 kB 00:00 2024-05-15T12:35:52.453 INFO:teuthology.orchestra.run.smithi177.stdout:(2/32): ceph-mds-17.2.7-945.g72b0874a.el8.x86_6 4.9 MB/s | 2.3 MB 00:00 2024-05-15T12:35:52.813 INFO:teuthology.orchestra.run.smithi120.stdout:(1/32): ceph-17.2.7-945.g72b0874a.el8.x86_64.rp 24 kB/s | 6.4 kB 00:00 2024-05-15T12:35:53.370 INFO:teuthology.orchestra.run.smithi177.stdout:(3/32): ceph-osd-17.2.7-945.g72b0874a.el8.x86_6 19 MB/s | 17 MB 00:00 2024-05-15T12:35:53.445 INFO:teuthology.orchestra.run.smithi177.stdout:(4/32): ceph-mgr-modules-core-17.2.7-945.g72b08 3.1 MB/s | 240 kB 00:00 2024-05-15T12:35:53.487 INFO:teuthology.orchestra.run.smithi177.stdout:(5/32): ceph-mgr-17.2.7-945.g72b0874a.el8.x86_6 1.0 MB/s | 1.6 MB 00:01 2024-05-15T12:35:53.596 INFO:teuthology.orchestra.run.smithi177.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 359 kB/s | 39 kB 00:00 2024-05-15T12:35:53.621 INFO:teuthology.orchestra.run.smithi177.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 895 kB/s | 157 kB 00:00 2024-05-15T12:35:53.663 INFO:teuthology.orchestra.run.smithi177.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.3 MB/s | 57 kB 00:00 2024-05-15T12:35:53.667 INFO:teuthology.orchestra.run.smithi120.stdout:(2/32): ceph-mds-17.2.7-945.g72b0874a.el8.x86_6 2.1 MB/s | 2.3 MB 00:01 2024-05-15T12:35:53.688 INFO:teuthology.orchestra.run.smithi177.stdout:(9/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.1 MB/s | 103 kB 00:00 2024-05-15T12:35:53.747 INFO:teuthology.orchestra.run.smithi177.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 5.3 MB/s | 457 kB 00:00 2024-05-15T12:35:53.822 INFO:teuthology.orchestra.run.smithi177.stdout:(11/32): ceph-mon-17.2.7-945.g72b0874a.el8.x86_ 2.6 MB/s | 4.0 MB 00:01 2024-05-15T12:35:53.863 INFO:teuthology.orchestra.run.smithi120.stdout:(3/32): ceph-mgr-17.2.7-945.g72b0874a.el8.x86_6 1.2 MB/s | 1.6 MB 00:01 2024-05-15T12:35:53.889 INFO:teuthology.orchestra.run.smithi177.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.2 MB/s | 249 kB 00:00 2024-05-15T12:35:53.964 INFO:teuthology.orchestra.run.smithi120.stdout:(4/32): ceph-mgr-modules-core-17.2.7-945.g72b08 2.3 MB/s | 240 kB 00:00 2024-05-15T12:35:53.964 INFO:teuthology.orchestra.run.smithi177.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 769 kB/s | 109 kB 00:00 2024-05-15T12:35:54.023 INFO:teuthology.orchestra.run.smithi177.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 2.3 MB/s | 641 kB 00:00 2024-05-15T12:35:54.131 INFO:teuthology.orchestra.run.smithi120.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 942 kB/s | 157 kB 00:00 2024-05-15T12:35:54.149 INFO:teuthology.orchestra.run.smithi177.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 168 kB/s | 44 kB 00:00 2024-05-15T12:35:54.181 INFO:teuthology.orchestra.run.smithi120.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 774 kB/s | 39 kB 00:00 2024-05-15T12:35:54.248 INFO:teuthology.orchestra.run.smithi120.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.5 MB/s | 103 kB 00:00 2024-05-15T12:35:54.298 INFO:teuthology.orchestra.run.smithi120.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.1 MB/s | 57 kB 00:00 2024-05-15T12:35:54.324 INFO:teuthology.orchestra.run.smithi177.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 516 kB/s | 185 kB 00:00 2024-05-15T12:35:54.394 INFO:teuthology.orchestra.run.smithi120.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 4.9 MB/s | 457 kB 00:00 2024-05-15T12:35:54.399 INFO:teuthology.orchestra.run.smithi177.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.5 MB/s | 384 kB 00:00 2024-05-15T12:35:54.424 INFO:teuthology.orchestra.run.smithi177.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2024-05-15T12:35:54.450 INFO:teuthology.orchestra.run.smithi177.stdout:(19/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 410 kB/s | 173 kB 00:00 2024-05-15T12:35:54.483 INFO:teuthology.orchestra.run.smithi177.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 209 kB/s | 18 kB 00:00 2024-05-15T12:35:54.509 INFO:teuthology.orchestra.run.smithi177.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 580 kB/s | 49 kB 00:00 2024-05-15T12:35:54.576 INFO:teuthology.orchestra.run.smithi177.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 466 kB/s | 59 kB 00:00 2024-05-15T12:35:54.582 INFO:teuthology.orchestra.run.smithi120.stdout:(10/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.3 MB/s | 249 kB 00:00 2024-05-15T12:35:54.601 INFO:teuthology.orchestra.run.smithi177.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.4 MB/s | 283 kB 00:00 2024-05-15T12:35:54.618 INFO:teuthology.orchestra.run.smithi177.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 143 kB/s | 16 kB 00:00 2024-05-15T12:35:54.685 INFO:teuthology.orchestra.run.smithi177.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 172 kB/s | 19 kB 00:00 2024-05-15T12:35:54.702 INFO:teuthology.orchestra.run.smithi177.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 238 kB/s | 24 kB 00:00 2024-05-15T12:35:54.719 INFO:teuthology.orchestra.run.smithi177.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 291 kB/s | 29 kB 00:00 2024-05-15T12:35:54.769 INFO:teuthology.orchestra.run.smithi177.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 329 kB/s | 27 kB 00:00 2024-05-15T12:35:54.813 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephsqlite-2:17.2.7-945.g72b0874a.el8.x86_64 5/6 2024-05-15T12:35:54.844 INFO:teuthology.orchestra.run.smithi177.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.0 MB/s | 255 kB 00:00 2024-05-15T12:35:54.900 INFO:teuthology.orchestra.run.smithi120.stdout:(11/32): ceph-osd-17.2.7-945.g72b0874a.el8.x86_ 14 MB/s | 17 MB 00:01 2024-05-15T12:35:54.911 INFO:teuthology.orchestra.run.smithi177.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.3 MB/s | 251 kB 00:00 2024-05-15T12:35:54.936 INFO:teuthology.orchestra.run.smithi177.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 250 kB/s | 23 kB 00:00 2024-05-15T12:35:54.962 INFO:teuthology.orchestra.run.smithi177.stdout:(32/32): python3-webtest-2.0.33-1.el8.noarch.rp 444 kB/s | 85 kB 00:00 2024-05-15T12:35:54.966 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:35:54.967 INFO:teuthology.orchestra.run.smithi177.stdout:Total 8.6 MB/s | 29 MB 00:03 2024-05-15T12:35:55.048 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-05-15T12:35:55.050 INFO:teuthology.orchestra.run.smithi120.stdout:(12/32): ceph-mon-17.2.7-945.g72b0874a.el8.x86_ 1.8 MB/s | 4.0 MB 00:02 2024-05-15T12:35:55.051 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-test-2:17.2.7-945.g72b0874a.el8.x86_64 6/6 2024-05-15T12:35:55.075 INFO:teuthology.orchestra.run.smithi120.stdout:(13/32): python3-cryptography-3.2.1-7.el8.x86_6 1.3 MB/s | 641 kB 00:00 2024-05-15T12:35:55.085 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-05-15T12:35:55.085 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-05-15T12:35:55.101 INFO:teuthology.orchestra.run.smithi120.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 544 kB/s | 109 kB 00:00 2024-05-15T12:35:55.395 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-05-15T12:35:55.398 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-05-15T12:35:55.476 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-test-2:17.2.7-945.g72b0874a.el8.x86_64 6/6 2024-05-15T12:35:55.476 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-test-2:17.2.7-945.g72b0874a.el8.x86_64 1/6 2024-05-15T12:35:55.476 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephsqlite-2:17.2.7-945.g72b0874a.el8.x86_64 2/6 2024-05-15T12:35:55.476 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-05-15T12:35:55.477 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-05-15T12:35:55.477 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-05-15T12:35:55.501 INFO:teuthology.orchestra.run.smithi120.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 97 kB/s | 44 kB 00:00 2024-05-15T12:35:55.651 INFO:teuthology.orchestra.run.smithi120.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 322 kB/s | 185 kB 00:00 2024-05-15T12:35:55.676 INFO:teuthology.orchestra.run.smithi120.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 300 kB/s | 173 kB 00:00 2024-05-15T12:35:55.760 INFO:teuthology.orchestra.run.smithi120.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 102 kB/s | 11 kB 00:00 2024-05-15T12:35:55.777 INFO:teuthology.orchestra.run.smithi120.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 176 kB/s | 18 kB 00:00 2024-05-15T12:35:55.886 INFO:teuthology.orchestra.run.smithi120.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 389 kB/s | 49 kB 00:00 2024-05-15T12:35:55.898 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-05-15T12:35:55.898 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:35:55.898 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-05-15T12:35:55.898 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:55.898 INFO:teuthology.orchestra.run.smithi027.stdout: jq-1.6-8.el8.x86_64 2024-05-15T12:35:55.898 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:35:55.899 INFO:teuthology.orchestra.run.smithi027.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-05-15T12:35:55.899 INFO:teuthology.orchestra.run.smithi027.stdout: socat-1.7.4.1-1.el8.x86_64 2024-05-15T12:35:55.899 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-05-15T12:35:55.899 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:35:55.899 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-15T12:35:55.911 INFO:teuthology.orchestra.run.smithi120.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 439 kB/s | 59 kB 00:00 2024-05-15T12:35:55.968 INFO:teuthology.orchestra.run.smithi120.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 823 kB/s | 384 kB 00:00 2024-05-15T12:35:56.020 INFO:teuthology.orchestra.run.smithi120.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 143 kB/s | 16 kB 00:00 2024-05-15T12:35:56.079 INFO:teuthology.orchestra.run.smithi120.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2024-05-15T12:35:56.104 INFO:teuthology.orchestra.run.smithi120.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 138 kB/s | 19 kB 00:00 2024-05-15T12:35:56.138 INFO:teuthology.orchestra.run.smithi120.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 203 kB/s | 24 kB 00:00 2024-05-15T12:35:56.188 INFO:teuthology.orchestra.run.smithi120.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 268 kB/s | 29 kB 00:00 2024-05-15T12:35:56.213 INFO:teuthology.orchestra.run.smithi120.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 252 kB/s | 27 kB 00:00 2024-05-15T12:35:56.288 INFO:teuthology.orchestra.run.smithi120.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.7 MB/s | 255 kB 00:00 2024-05-15T12:35:56.303 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph 2024-05-15T12:35:56.322 INFO:teuthology.orchestra.run.smithi120.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.8 MB/s | 251 kB 00:00 2024-05-15T12:35:56.347 INFO:teuthology.orchestra.run.smithi120.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 637 kB/s | 85 kB 00:00 2024-05-15T12:35:56.398 INFO:teuthology.orchestra.run.smithi120.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 211 kB/s | 23 kB 00:00 2024-05-15T12:35:56.402 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:35:56.402 INFO:teuthology.orchestra.run.smithi120.stdout:Total 6.9 MB/s | 29 MB 00:04 2024-05-15T12:35:56.483 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-05-15T12:35:56.496 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-05-15T12:35:56.521 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-05-15T12:35:56.521 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-05-15T12:35:56.735 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-05-15T12:35:56.827 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-05-15T12:35:56.830 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-05-15T12:35:56.979 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-05-15T12:35:57.068 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:13 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:35:57.230 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-05-15T12:35:57.488 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-05-15T12:35:57.676 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-05-15T12:35:57.856 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-05-15T12:35:57.968 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-05-15T12:35:58.049 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-05-15T12:35:58.192 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-05-15T12:35:58.210 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-05-15T12:35:58.411 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-05-15T12:35:58.449 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-05-15T12:35:58.649 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-05-15T12:35:58.678 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-05-15T12:35:58.834 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-15T12:35:58.837 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:35:58.837 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-05-15T12:35:58.837 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:35:58.838 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-05-15T12:35:58.838 INFO:teuthology.orchestra.run.smithi027.stdout: ceph x86_64 2:17.2.7-945.g72b0874a.el8 ceph 6.4 k 2024-05-15T12:35:58.838 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-05-15T12:35:58.838 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds x86_64 2:17.2.7-945.g72b0874a.el8 ceph 2.3 M 2024-05-15T12:35:58.838 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr x86_64 2:17.2.7-945.g72b0874a.el8 ceph 1.6 M 2024-05-15T12:35:58.838 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core noarch 2:17.2.7-945.g72b0874a.el8 ceph-noarch 240 k 2024-05-15T12:35:58.839 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon x86_64 2:17.2.7-945.g72b0874a.el8 ceph 4.0 M 2024-05-15T12:35:58.839 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd x86_64 2:17.2.7-945.g72b0874a.el8 ceph 17 M 2024-05-15T12:35:58.839 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-05-15T12:35:58.839 INFO:teuthology.orchestra.run.smithi027.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-05-15T12:35:58.839 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-05-15T12:35:58.839 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-05-15T12:35:58.839 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-05-15T12:35:58.839 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-05-15T12:35:58.839 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-05-15T12:35:58.839 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-05-15T12:35:58.839 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-05-15T12:35:58.839 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-05-15T12:35:58.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-05-15T12:35:58.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-05-15T12:35:58.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-05-15T12:35:58.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-05-15T12:35:58.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-05-15T12:35:58.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-05-15T12:35:58.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-05-15T12:35:58.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-05-15T12:35:58.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-05-15T12:35:58.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-05-15T12:35:58.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-05-15T12:35:58.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-05-15T12:35:58.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-05-15T12:35:58.840 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-05-15T12:35:58.841 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-05-15T12:35:58.841 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-05-15T12:35:58.841 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:35:58.841 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-05-15T12:35:58.841 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:35:58.841 INFO:teuthology.orchestra.run.smithi027.stdout:Install 32 Packages 2024-05-15T12:35:58.841 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:35:58.842 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 29 M 2024-05-15T12:35:58.842 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 113 M 2024-05-15T12:35:58.842 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-05-15T12:35:58.886 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-05-15T12:35:58.975 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-05-15T12:35:59.135 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-05-15T12:35:59.158 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-05-15T12:35:59.322 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-05-15T12:35:59.506 INFO:teuthology.orchestra.run.smithi027.stdout:(1/32): ceph-17.2.7-945.g72b0874a.el8.x86_64.rp 17 kB/s | 6.4 kB 00:00 2024-05-15T12:35:59.543 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-05-15T12:35:59.548 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-05-15T12:35:59.701 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-05-15T12:35:59.773 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-05-15T12:35:59.790 INFO:teuthology.orchestra.run.smithi027.stdout:(2/32): ceph-mgr-17.2.7-945.g72b0874a.el8.x86_6 2.4 MB/s | 1.6 MB 00:00 2024-05-15T12:35:59.910 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-05-15T12:35:59.954 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-05-15T12:36:00.065 INFO:teuthology.orchestra.run.smithi027.stdout:(3/32): ceph-mds-17.2.7-945.g72b0874a.el8.x86_6 2.5 MB/s | 2.3 MB 00:00 2024-05-15T12:36:00.164 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-05-15T12:36:00.174 INFO:teuthology.orchestra.run.smithi027.stdout:(4/32): ceph-mgr-modules-core-17.2.7-945.g72b08 2.2 MB/s | 240 kB 00:00 2024-05-15T12:36:00.211 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-05-15T12:36:00.350 INFO:teuthology.orchestra.run.smithi027.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 899 kB/s | 157 kB 00:00 2024-05-15T12:36:00.400 INFO:teuthology.orchestra.run.smithi027.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 776 kB/s | 39 kB 00:00 2024-05-15T12:36:00.410 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-05-15T12:36:00.413 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-05-15T12:36:00.442 INFO:teuthology.orchestra.run.smithi027.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 2.4 MB/s | 103 kB 00:00 2024-05-15T12:36:00.509 INFO:teuthology.orchestra.run.smithi027.stdout:(8/32): ceph-mon-17.2.7-945.g72b0874a.el8.x86_6 4.0 MB/s | 4.0 MB 00:01 2024-05-15T12:36:00.560 INFO:teuthology.orchestra.run.smithi027.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 484 kB/s | 57 kB 00:00 2024-05-15T12:36:00.676 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-05-15T12:36:00.713 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-05-15T12:36:00.943 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-05-15T12:36:00.943 INFO:teuthology.orchestra.run.smithi027.stdout:(10/32): ceph-osd-17.2.7-945.g72b0874a.el8.x86_ 15 MB/s | 17 MB 00:01 2024-05-15T12:36:00.960 INFO:teuthology.orchestra.run.smithi027.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 623 kB/s | 249 kB 00:00 2024-05-15T12:36:00.985 INFO:teuthology.orchestra.run.smithi027.stdout:(12/32): python3-werkzeug-0.12.2-4.el8.noarch.r 962 kB/s | 457 kB 00:00 2024-05-15T12:36:01.081 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-05-15T12:36:01.106 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-05-15T12:36:01.144 INFO:teuthology.orchestra.run.smithi027.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 595 kB/s | 109 kB 00:00 2024-05-15T12:36:01.210 INFO:teuthology.orchestra.run.smithi027.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 193 kB/s | 44 kB 00:00 2024-05-15T12:36:01.309 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-05-15T12:36:01.327 INFO:teuthology.orchestra.run.smithi027.stdout:(15/32): python3-cryptography-3.2.1-7.el8.x86_6 1.7 MB/s | 641 kB 00:00 2024-05-15T12:36:01.356 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-05-15T12:36:01.444 INFO:teuthology.orchestra.run.smithi027.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 739 kB/s | 173 kB 00:00 2024-05-15T12:36:01.470 INFO:teuthology.orchestra.run.smithi027.stdout:(17/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 568 kB/s | 185 kB 00:00 2024-05-15T12:36:01.512 INFO:teuthology.orchestra.run.smithi027.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 166 kB/s | 11 kB 00:00 2024-05-15T12:36:01.537 INFO:teuthology.orchestra.run.smithi027.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 263 kB/s | 18 kB 00:00 2024-05-15T12:36:01.543 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-05-15T12:36:01.544 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-05-15T12:36:01.579 INFO:teuthology.orchestra.run.smithi027.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 728 kB/s | 49 kB 00:00 2024-05-15T12:36:01.638 INFO:teuthology.orchestra.run.smithi027.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 583 kB/s | 59 kB 00:00 2024-05-15T12:36:01.671 INFO:teuthology.orchestra.run.smithi027.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.1 MB/s | 384 kB 00:00 2024-05-15T12:36:01.705 INFO:teuthology.orchestra.run.smithi027.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2024-05-15T12:36:01.738 INFO:teuthology.orchestra.run.smithi027.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 280 kB/s | 19 kB 00:00 2024-05-15T12:36:01.766 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-05-15T12:36:01.772 INFO:teuthology.orchestra.run.smithi027.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 359 kB/s | 24 kB 00:00 2024-05-15T12:36:01.806 INFO:teuthology.orchestra.run.smithi027.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 435 kB/s | 29 kB 00:00 2024-05-15T12:36:01.838 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-05-15T12:36:01.839 INFO:teuthology.orchestra.run.smithi027.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 409 kB/s | 27 kB 00:00 2024-05-15T12:36:01.908 INFO:teuthology.orchestra.run.smithi027.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 861 kB/s | 283 kB 00:00 2024-05-15T12:36:01.940 INFO:teuthology.orchestra.run.smithi027.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.9 MB/s | 255 kB 00:00 2024-05-15T12:36:01.965 INFO:teuthology.orchestra.run.smithi027.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.0 MB/s | 251 kB 00:00 2024-05-15T12:36:01.997 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-05-15T12:36:02.007 INFO:teuthology.orchestra.run.smithi027.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 344 kB/s | 23 kB 00:00 2024-05-15T12:36:02.029 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-05-15T12:36:02.032 INFO:teuthology.orchestra.run.smithi027.stdout:(32/32): python3-webtest-2.0.33-1.el8.noarch.rp 686 kB/s | 85 kB 00:00 2024-05-15T12:36:02.035 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:36:02.035 INFO:teuthology.orchestra.run.smithi027.stdout:Total 9.1 MB/s | 29 MB 00:03 2024-05-15T12:36:02.122 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-05-15T12:36:02.163 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-05-15T12:36:02.163 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-05-15T12:36:02.292 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-05-15T12:36:02.335 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-05-15T12:36:02.483 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-05-15T12:36:02.486 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-05-15T12:36:02.559 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-05-15T12:36:02.608 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-05-15T12:36:02.722 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-05-15T12:36:02.934 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-05-15T12:36:02.965 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-945.g72b0874a.el8.n 27/32 2024-05-15T12:36:03.190 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-05-15T12:36:03.191 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-2:17.2.7-945.g72b0874a.el8.x86_64 28/32 2024-05-15T12:36:03.475 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-05-15T12:36:03.654 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-05-15T12:36:03.919 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-05-15T12:36:04.201 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-05-15T12:36:04.398 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-05-15T12:36:04.474 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-05-15T12:36:04.705 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-2:17.2.7-945.g72b0874a.el8.x86_64 28/32 2024-05-15T12:36:04.727 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-05-15T12:36:04.741 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-05-15T12:36:04.901 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-osd-2:17.2.7-945.g72b0874a.el8.x86_64 29/32 2024-05-15T12:36:04.948 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-05-15T12:36:05.058 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-05-15T12:36:05.105 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-05-15T12:36:05.302 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-osd-2:17.2.7-945.g72b0874a.el8.x86_64 29/32 2024-05-15T12:36:05.346 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-05-15T12:36:05.463 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-945.g72b0874a.el8.n 27/32 2024-05-15T12:36:05.478 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mon-2:17.2.7-945.g72b0874a.el8.x86_64 30/32 2024-05-15T12:36:05.566 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-05-15T12:36:05.683 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-2:17.2.7-945.g72b0874a.el8.x86_64 28/32 2024-05-15T12:36:05.696 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mon-2:17.2.7-945.g72b0874a.el8.x86_64 30/32 2024-05-15T12:36:05.795 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-05-15T12:36:05.870 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mds-2:17.2.7-945.g72b0874a.el8.x86_64 31/32 2024-05-15T12:36:05.888 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mds-2:17.2.7-945.g72b0874a.el8.x86_64 31/32 2024-05-15T12:36:06.065 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-05-15T12:36:06.353 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-05-15T12:36:06.398 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-2:17.2.7-945.g72b0874a.el8.x86_64 32/32 2024-05-15T12:36:06.654 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-05-15T12:36:07.023 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-2:17.2.7-945.g72b0874a.el8.x86_64 32/32 2024-05-15T12:36:07.024 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-2:17.2.7-945.g72b0874a.el8.x86_64 1/32 2024-05-15T12:36:07.024 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mds-2:17.2.7-945.g72b0874a.el8.x86_64 2/32 2024-05-15T12:36:07.024 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-2:17.2.7-945.g72b0874a.el8.x86_64 3/32 2024-05-15T12:36:07.024 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mon-2:17.2.7-945.g72b0874a.el8.x86_64 4/32 2024-05-15T12:36:07.024 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-osd-2:17.2.7-945.g72b0874a.el8.x86_64 5/32 2024-05-15T12:36:07.024 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-945.g72b0874a.el8.n 6/32 2024-05-15T12:36:07.024 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-05-15T12:36:07.024 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-05-15T12:36:07.024 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-05-15T12:36:07.024 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-05-15T12:36:07.024 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-05-15T12:36:07.025 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-05-15T12:36:07.025 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-05-15T12:36:07.025 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-05-15T12:36:07.025 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-05-15T12:36:07.025 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-05-15T12:36:07.025 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-05-15T12:36:07.025 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-05-15T12:36:07.025 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-05-15T12:36:07.025 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-05-15T12:36:07.025 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-05-15T12:36:07.025 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-05-15T12:36:07.025 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-05-15T12:36:07.025 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-05-15T12:36:07.025 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-05-15T12:36:07.025 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-05-15T12:36:07.026 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-05-15T12:36:07.026 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-05-15T12:36:07.026 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-05-15T12:36:07.026 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-05-15T12:36:07.026 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-05-15T12:36:07.071 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-05-15T12:36:07.198 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-2:17.2.7-945.g72b0874a.el8.x86_64 28/32 2024-05-15T12:36:07.297 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-05-15T12:36:07.383 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-osd-2:17.2.7-945.g72b0874a.el8.x86_64 29/32 2024-05-15T12:36:07.501 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-05-15T12:36:07.782 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-osd-2:17.2.7-945.g72b0874a.el8.x86_64 29/32 2024-05-15T12:36:07.798 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-05-15T12:36:07.886 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-05-15T12:36:07.886 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:07.886 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-05-15T12:36:07.886 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:07.886 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:07.886 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:07.886 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core-2:17.2.7-945.g72b0874a.el8.noarch 2024-05-15T12:36:07.887 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:07.887 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:07.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-05-15T12:36:07.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-05-15T12:36:07.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-05-15T12:36:07.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-05-15T12:36:07.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-05-15T12:36:07.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-05-15T12:36:07.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-6.2-6.el8.noarch 2024-05-15T12:36:07.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-05-15T12:36:07.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-05-15T12:36:07.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako-1.0.6-14.el8.noarch 2024-05-15T12:36:07.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-05-15T12:36:07.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-05-15T12:36:07.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-05-15T12:36:07.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend-2.6-1.el8.noarch 2024-05-15T12:36:07.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-05-15T12:36:07.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser-2.14-14.el8.noarch 2024-05-15T12:36:07.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz-2017.2-11.el8.noarch 2024-05-15T12:36:07.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-05-15T12:36:07.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-05-15T12:36:07.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-05-15T12:36:07.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-05-15T12:36:07.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-05-15T12:36:07.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-05-15T12:36:07.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-05-15T12:36:07.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-05-15T12:36:07.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-05-15T12:36:07.888 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:07.888 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-05-15T12:36:07.952 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mon-2:17.2.7-945.g72b0874a.el8.x86_64 30/32 2024-05-15T12:36:08.045 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-05-15T12:36:08.153 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-base 2024-05-15T12:36:08.169 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mon-2:17.2.7-945.g72b0874a.el8.x86_64 30/32 2024-05-15T12:36:08.352 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mds-2:17.2.7-945.g72b0874a.el8.x86_64 31/32 2024-05-15T12:36:08.370 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mds-2:17.2.7-945.g72b0874a.el8.x86_64 31/32 2024-05-15T12:36:08.412 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-05-15T12:36:08.493 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-2:17.2.7-945.g72b0874a.el8.x86_64 32/32 2024-05-15T12:36:08.675 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-05-15T12:36:08.879 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-05-15T12:36:08.897 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-2:17.2.7-945.g72b0874a.el8.x86_64 32/32 2024-05-15T12:36:08.898 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-2:17.2.7-945.g72b0874a.el8.x86_64 1/32 2024-05-15T12:36:08.898 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mds-2:17.2.7-945.g72b0874a.el8.x86_64 2/32 2024-05-15T12:36:08.898 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-2:17.2.7-945.g72b0874a.el8.x86_64 3/32 2024-05-15T12:36:08.898 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mon-2:17.2.7-945.g72b0874a.el8.x86_64 4/32 2024-05-15T12:36:08.898 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-osd-2:17.2.7-945.g72b0874a.el8.x86_64 5/32 2024-05-15T12:36:08.898 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-945.g72b0874a.el8.n 6/32 2024-05-15T12:36:08.898 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-05-15T12:36:08.898 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-05-15T12:36:08.898 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-05-15T12:36:08.898 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-05-15T12:36:08.898 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-05-15T12:36:08.899 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-05-15T12:36:08.899 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-05-15T12:36:08.899 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-05-15T12:36:08.899 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-05-15T12:36:08.899 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-05-15T12:36:08.899 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-05-15T12:36:08.899 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-05-15T12:36:08.899 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-05-15T12:36:08.899 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-05-15T12:36:08.899 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-05-15T12:36:08.899 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-05-15T12:36:08.899 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-05-15T12:36:08.899 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-05-15T12:36:08.899 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-05-15T12:36:08.902 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-05-15T12:36:08.902 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-05-15T12:36:08.903 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-05-15T12:36:08.903 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-05-15T12:36:08.903 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-05-15T12:36:08.903 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-05-15T12:36:08.906 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:24 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:09.123 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-05-15T12:36:09.412 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-05-15T12:36:09.693 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-05-15T12:36:09.818 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-05-15T12:36:09.818 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:09.818 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-05-15T12:36:09.818 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:09.818 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:09.818 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:09.818 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-modules-core-2:17.2.7-945.g72b0874a.el8.noarch 2024-05-15T12:36:09.819 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:09.819 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:09.819 INFO:teuthology.orchestra.run.smithi120.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-05-15T12:36:09.819 INFO:teuthology.orchestra.run.smithi120.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-05-15T12:36:09.819 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-05-15T12:36:09.819 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-05-15T12:36:09.819 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-05-15T12:36:09.819 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-05-15T12:36:09.819 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-6.2-6.el8.noarch 2024-05-15T12:36:09.819 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-05-15T12:36:09.819 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-05-15T12:36:09.819 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako-1.0.6-14.el8.noarch 2024-05-15T12:36:09.819 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-05-15T12:36:09.820 INFO:teuthology.orchestra.run.smithi120.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-05-15T12:36:09.820 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-05-15T12:36:09.820 INFO:teuthology.orchestra.run.smithi120.stdout: python3-portend-2.6-1.el8.noarch 2024-05-15T12:36:09.820 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-05-15T12:36:09.820 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pycparser-2.14-14.el8.noarch 2024-05-15T12:36:09.820 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytz-2017.2-11.el8.noarch 2024-05-15T12:36:09.820 INFO:teuthology.orchestra.run.smithi120.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-05-15T12:36:09.820 INFO:teuthology.orchestra.run.smithi120.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-05-15T12:36:09.820 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-05-15T12:36:09.820 INFO:teuthology.orchestra.run.smithi120.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-05-15T12:36:09.820 INFO:teuthology.orchestra.run.smithi120.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-05-15T12:36:09.820 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-05-15T12:36:09.820 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-05-15T12:36:09.820 INFO:teuthology.orchestra.run.smithi120.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-05-15T12:36:09.821 INFO:teuthology.orchestra.run.smithi120.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-05-15T12:36:09.821 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:09.821 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-05-15T12:36:09.956 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-05-15T12:36:10.058 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-base 2024-05-15T12:36:10.512 INFO:teuthology.orchestra.run.smithi177.stdout:Package ceph-base-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:10.561 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-05-15T12:36:10.562 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-05-15T12:36:10.562 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-05-15T12:36:10.675 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install cephadm 2024-05-15T12:36:10.813 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:27 ago on Wed 15 May 2024 12:34:43 PM UTC. 2024-05-15T12:36:10.929 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-05-15T12:36:11.282 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-05-15T12:36:11.420 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:27 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:11.864 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-945.g72b0874a.el8.n 27/32 2024-05-15T12:36:12.073 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-2:17.2.7-945.g72b0874a.el8.x86_64 28/32 2024-05-15T12:36:12.434 INFO:teuthology.orchestra.run.smithi120.stdout:Package ceph-base-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:12.482 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-05-15T12:36:12.483 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-05-15T12:36:12.483 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-05-15T12:36:12.590 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install cephadm 2024-05-15T12:36:13.072 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-05-15T12:36:13.072 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:36:13.073 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-05-15T12:36:13.073 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:36:13.073 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-05-15T12:36:13.073 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm noarch 2:17.2.7-945.g72b0874a.el8 ceph-noarch 82 k 2024-05-15T12:36:13.073 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:13.073 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-05-15T12:36:13.073 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:36:13.073 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-05-15T12:36:13.073 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:13.074 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 82 k 2024-05-15T12:36:13.074 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 372 k 2024-05-15T12:36:13.074 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-05-15T12:36:13.342 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:30 ago on Wed 15 May 2024 12:34:43 PM UTC. 2024-05-15T12:36:13.558 INFO:teuthology.orchestra.run.smithi177.stdout:cephadm-17.2.7-945.g72b0874a.el8.noarch.rpm 169 kB/s | 82 kB 00:00 2024-05-15T12:36:13.559 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:36:13.559 INFO:teuthology.orchestra.run.smithi177.stdout:Total 169 kB/s | 82 kB 00:00 2024-05-15T12:36:13.559 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-05-15T12:36:13.567 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-05-15T12:36:13.567 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-05-15T12:36:13.569 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-05-15T12:36:13.572 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-05-15T12:36:13.618 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-05-15T12:36:13.692 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-2:17.2.7-945.g72b0874a.el8.x86_64 28/32 2024-05-15T12:36:13.907 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-osd-2:17.2.7-945.g72b0874a.el8.x86_64 29/32 2024-05-15T12:36:14.078 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephadm-2:17.2.7-945.g72b0874a.el8.noarch 1/1 2024-05-15T12:36:14.229 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephadm-2:17.2.7-945.g72b0874a.el8.noarch 1/1 2024-05-15T12:36:14.332 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-osd-2:17.2.7-945.g72b0874a.el8.x86_64 29/32 2024-05-15T12:36:14.454 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephadm-2:17.2.7-945.g72b0874a.el8.noarch 1/1 2024-05-15T12:36:14.517 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mon-2:17.2.7-945.g72b0874a.el8.x86_64 30/32 2024-05-15T12:36:14.747 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mon-2:17.2.7-945.g72b0874a.el8.x86_64 30/32 2024-05-15T12:36:14.843 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephadm-2:17.2.7-945.g72b0874a.el8.noarch 1/1 2024-05-15T12:36:14.843 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:14.843 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-05-15T12:36:14.843 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm-2:17.2.7-945.g72b0874a.el8.noarch 2024-05-15T12:36:14.843 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:14.843 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-05-15T12:36:14.967 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mds-2:17.2.7-945.g72b0874a.el8.x86_64 31/32 2024-05-15T12:36:14.974 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-05-15T12:36:14.975 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:36:14.975 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-05-15T12:36:14.975 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:36:14.975 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-05-15T12:36:14.975 INFO:teuthology.orchestra.run.smithi120.stdout: cephadm noarch 2:17.2.7-945.g72b0874a.el8 ceph-noarch 82 k 2024-05-15T12:36:14.975 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:14.975 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-05-15T12:36:14.975 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:36:14.975 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-05-15T12:36:14.975 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:14.976 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 82 k 2024-05-15T12:36:14.976 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 372 k 2024-05-15T12:36:14.976 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-05-15T12:36:14.985 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mds-2:17.2.7-945.g72b0874a.el8.x86_64 31/32 2024-05-15T12:36:15.082 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-immutable-object-cache 2024-05-15T12:36:15.159 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-2:17.2.7-945.g72b0874a.el8.x86_64 32/32 2024-05-15T12:36:15.574 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-2:17.2.7-945.g72b0874a.el8.x86_64 32/32 2024-05-15T12:36:15.574 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-2:17.2.7-945.g72b0874a.el8.x86_64 1/32 2024-05-15T12:36:15.574 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mds-2:17.2.7-945.g72b0874a.el8.x86_64 2/32 2024-05-15T12:36:15.574 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-2:17.2.7-945.g72b0874a.el8.x86_64 3/32 2024-05-15T12:36:15.574 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mon-2:17.2.7-945.g72b0874a.el8.x86_64 4/32 2024-05-15T12:36:15.574 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-osd-2:17.2.7-945.g72b0874a.el8.x86_64 5/32 2024-05-15T12:36:15.574 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-945.g72b0874a.el8.n 6/32 2024-05-15T12:36:15.574 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-05-15T12:36:15.575 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-05-15T12:36:15.575 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-05-15T12:36:15.575 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-05-15T12:36:15.575 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-05-15T12:36:15.575 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-05-15T12:36:15.575 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-05-15T12:36:15.575 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-05-15T12:36:15.575 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-05-15T12:36:15.575 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-05-15T12:36:15.575 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-05-15T12:36:15.575 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-05-15T12:36:15.575 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-05-15T12:36:15.575 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-05-15T12:36:15.575 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-05-15T12:36:15.575 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-05-15T12:36:15.576 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-05-15T12:36:15.576 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-05-15T12:36:15.576 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-05-15T12:36:15.576 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-05-15T12:36:15.576 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-05-15T12:36:15.576 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-05-15T12:36:15.576 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-05-15T12:36:15.577 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-05-15T12:36:15.577 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-05-15T12:36:15.684 INFO:teuthology.orchestra.run.smithi120.stdout:cephadm-17.2.7-945.g72b0874a.el8.noarch.rpm 116 kB/s | 82 kB 00:00 2024-05-15T12:36:15.685 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:36:15.685 INFO:teuthology.orchestra.run.smithi120.stdout:Total 115 kB/s | 82 kB 00:00 2024-05-15T12:36:15.686 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-05-15T12:36:15.695 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-05-15T12:36:15.695 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-05-15T12:36:15.697 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-05-15T12:36:15.700 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-05-15T12:36:15.749 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-05-15T12:36:15.837 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:31 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:16.286 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cephadm-2:17.2.7-945.g72b0874a.el8.noarch 1/1 2024-05-15T12:36:16.453 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephadm-2:17.2.7-945.g72b0874a.el8.noarch 1/1 2024-05-15T12:36:16.578 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-05-15T12:36:16.579 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:36:16.579 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-05-15T12:36:16.579 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:16.579 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:16.579 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:16.579 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core-2:17.2.7-945.g72b0874a.el8.noarch 2024-05-15T12:36:16.579 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:16.579 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:16.579 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-05-15T12:36:16.579 INFO:teuthology.orchestra.run.smithi027.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-05-15T12:36:16.579 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-05-15T12:36:16.579 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-05-15T12:36:16.579 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-05-15T12:36:16.580 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-05-15T12:36:16.580 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-6.2-6.el8.noarch 2024-05-15T12:36:16.580 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-05-15T12:36:16.580 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-05-15T12:36:16.580 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako-1.0.6-14.el8.noarch 2024-05-15T12:36:16.580 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-05-15T12:36:16.580 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-05-15T12:36:16.580 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-05-15T12:36:16.580 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend-2.6-1.el8.noarch 2024-05-15T12:36:16.580 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-05-15T12:36:16.580 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser-2.14-14.el8.noarch 2024-05-15T12:36:16.580 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz-2017.2-11.el8.noarch 2024-05-15T12:36:16.580 INFO:teuthology.orchestra.run.smithi027.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-05-15T12:36:16.581 INFO:teuthology.orchestra.run.smithi027.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-05-15T12:36:16.581 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-05-15T12:36:16.581 INFO:teuthology.orchestra.run.smithi027.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-05-15T12:36:16.581 INFO:teuthology.orchestra.run.smithi027.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-05-15T12:36:16.581 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-05-15T12:36:16.581 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-05-15T12:36:16.581 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-05-15T12:36:16.581 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-05-15T12:36:16.581 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:36:16.581 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-15T12:36:16.686 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cephadm-2:17.2.7-945.g72b0874a.el8.noarch 1/1 2024-05-15T12:36:16.860 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-base 2024-05-15T12:36:17.084 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephadm-2:17.2.7-945.g72b0874a.el8.noarch 1/1 2024-05-15T12:36:17.084 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:17.084 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-05-15T12:36:17.084 INFO:teuthology.orchestra.run.smithi120.stdout: cephadm-2:17.2.7-945.g72b0874a.el8.noarch 2024-05-15T12:36:17.084 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:17.084 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-05-15T12:36:17.309 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-immutable-object-cache 2024-05-15T12:36:17.494 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-05-15T12:36:17.494 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:36:17.495 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-05-15T12:36:17.495 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:36:17.495 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-05-15T12:36:17.495 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-945.g72b0874a.el8 ceph 162 k 2024-05-15T12:36:17.495 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:17.495 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-05-15T12:36:17.495 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:36:17.495 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-05-15T12:36:17.495 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:17.496 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 162 k 2024-05-15T12:36:17.496 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 494 k 2024-05-15T12:36:17.496 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-05-15T12:36:17.647 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:33 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:18.068 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:35 ago on Wed 15 May 2024 12:34:43 PM UTC. 2024-05-15T12:36:18.569 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-immutable-object-cache-17.2.7-945.g72b0874 151 kB/s | 162 kB 00:01 2024-05-15T12:36:18.571 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:36:18.571 INFO:teuthology.orchestra.run.smithi177.stdout:Total 151 kB/s | 162 kB 00:01 2024-05-15T12:36:18.572 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-05-15T12:36:18.583 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-05-15T12:36:18.583 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-05-15T12:36:18.636 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-05-15T12:36:18.639 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-05-15T12:36:18.768 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-05-15T12:36:18.952 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-945.g72b0874a.e 1/1 2024-05-15T12:36:19.368 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-945.g72b0874a.e 1/1 2024-05-15T12:36:19.369 INFO:teuthology.orchestra.run.smithi027.stdout:Package ceph-base-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:19.422 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-15T12:36:19.423 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-05-15T12:36:19.423 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-15T12:36:19.527 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install cephadm 2024-05-15T12:36:19.727 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-05-15T12:36:19.727 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:36:19.727 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-05-15T12:36:19.727 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:36:19.727 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-05-15T12:36:19.727 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-945.g72b0874a.el8 ceph 162 k 2024-05-15T12:36:19.727 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:19.727 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-05-15T12:36:19.728 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:36:19.728 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-05-15T12:36:19.728 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:19.728 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 162 k 2024-05-15T12:36:19.728 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 494 k 2024-05-15T12:36:19.728 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-05-15T12:36:19.767 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-945.g72b0874a.e 1/1 2024-05-15T12:36:19.767 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:19.767 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-05-15T12:36:19.767 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:19.767 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:19.767 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-05-15T12:36:19.993 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr 2024-05-15T12:36:20.223 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-immutable-object-cache-17.2.7-945.g72b0874 327 kB/s | 162 kB 00:00 2024-05-15T12:36:20.224 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:36:20.224 INFO:teuthology.orchestra.run.smithi120.stdout:Total 327 kB/s | 162 kB 00:00 2024-05-15T12:36:20.224 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-05-15T12:36:20.235 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-05-15T12:36:20.235 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-05-15T12:36:20.282 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-05-15T12:36:20.285 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-05-15T12:36:20.318 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:36 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:20.394 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-05-15T12:36:20.751 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:36 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:21.915 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-945.g72b0874a.e 1/1 2024-05-15T12:36:22.102 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-15T12:36:22.103 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:36:22.103 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-05-15T12:36:22.103 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:36:22.103 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-05-15T12:36:22.103 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm noarch 2:17.2.7-945.g72b0874a.el8 ceph-noarch 82 k 2024-05-15T12:36:22.103 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:36:22.104 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-05-15T12:36:22.104 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:36:22.104 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-05-15T12:36:22.104 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:36:22.104 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 82 k 2024-05-15T12:36:22.104 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 372 k 2024-05-15T12:36:22.104 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-05-15T12:36:22.336 INFO:teuthology.orchestra.run.smithi177.stdout:Package ceph-mgr-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:22.380 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-945.g72b0874a.e 1/1 2024-05-15T12:36:22.385 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-05-15T12:36:22.386 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-05-15T12:36:22.386 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-05-15T12:36:22.397 INFO:teuthology.orchestra.run.smithi027.stdout:cephadm-17.2.7-945.g72b0874a.el8.noarch.rpm 279 kB/s | 82 kB 00:00 2024-05-15T12:36:22.397 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:36:22.398 INFO:teuthology.orchestra.run.smithi027.stdout:Total 278 kB/s | 82 kB 00:00 2024-05-15T12:36:22.398 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-05-15T12:36:22.406 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-05-15T12:36:22.406 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-05-15T12:36:22.409 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-05-15T12:36:22.412 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-05-15T12:36:22.460 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-05-15T12:36:22.495 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-fuse 2024-05-15T12:36:22.762 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-945.g72b0874a.e 1/1 2024-05-15T12:36:22.762 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:22.762 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-05-15T12:36:22.762 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:22.762 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:22.763 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-05-15T12:36:22.999 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mgr 2024-05-15T12:36:23.031 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:17.2.7-945.g72b0874a.el8.noarch 1/1 2024-05-15T12:36:23.235 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephadm-2:17.2.7-945.g72b0874a.el8.noarch 1/1 2024-05-15T12:36:23.239 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:39 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:23.476 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:17.2.7-945.g72b0874a.el8.noarch 1/1 2024-05-15T12:36:23.757 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:40 ago on Wed 15 May 2024 12:34:43 PM UTC. 2024-05-15T12:36:23.879 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephadm-2:17.2.7-945.g72b0874a.el8.noarch 1/1 2024-05-15T12:36:23.879 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:36:23.879 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-05-15T12:36:23.879 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm-2:17.2.7-945.g72b0874a.el8.noarch 2024-05-15T12:36:23.879 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:36:23.879 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-15T12:36:24.131 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-immutable-object-cache 2024-05-15T12:36:24.888 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-05-15T12:36:24.889 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:36:24.890 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-05-15T12:36:24.890 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:36:24.890 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-05-15T12:36:24.890 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse x86_64 2:17.2.7-945.g72b0874a.el8 ceph 876 k 2024-05-15T12:36:24.890 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-05-15T12:36:24.890 INFO:teuthology.orchestra.run.smithi177.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-05-15T12:36:24.890 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:24.890 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-05-15T12:36:24.890 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:36:24.890 INFO:teuthology.orchestra.run.smithi177.stdout:Install 2 Packages 2024-05-15T12:36:24.890 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:24.890 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 959 k 2024-05-15T12:36:24.891 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 3.1 M 2024-05-15T12:36:24.891 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-05-15T12:36:24.919 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:40 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:25.084 INFO:teuthology.orchestra.run.smithi177.stdout:(1/2): fuse-2.9.7-19.el8.x86_64.rpm 610 kB/s | 83 kB 00:00 2024-05-15T12:36:25.217 INFO:teuthology.orchestra.run.smithi177.stdout:(2/2): ceph-fuse-17.2.7-945.g72b0874a.el8.x86_6 3.2 MB/s | 876 kB 00:00 2024-05-15T12:36:25.217 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:36:25.218 INFO:teuthology.orchestra.run.smithi177.stdout:Total 2.9 MB/s | 959 kB 00:00 2024-05-15T12:36:25.228 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-05-15T12:36:25.249 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-05-15T12:36:25.249 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-05-15T12:36:25.332 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-05-15T12:36:25.335 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-05-15T12:36:25.364 INFO:teuthology.orchestra.run.smithi120.stdout:Package ceph-mgr-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:25.412 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-05-15T12:36:25.414 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-05-15T12:36:25.414 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-05-15T12:36:25.505 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-05-15T12:36:25.534 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-fuse 2024-05-15T12:36:25.784 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-05-15T12:36:26.018 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-fuse-2:17.2.7-945.g72b0874a.el8.x86_64 2/2 2024-05-15T12:36:26.281 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:43 ago on Wed 15 May 2024 12:34:43 PM UTC. 2024-05-15T12:36:26.408 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-fuse-2:17.2.7-945.g72b0874a.el8.x86_64 2/2 2024-05-15T12:36:26.408 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-fuse-2:17.2.7-945.g72b0874a.el8.x86_64 1/2 2024-05-15T12:36:26.683 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-15T12:36:26.684 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:36:26.684 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-05-15T12:36:26.685 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:36:26.685 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-05-15T12:36:26.685 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-945.g72b0874a.el8 ceph 162 k 2024-05-15T12:36:26.685 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:36:26.685 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-05-15T12:36:26.685 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:36:26.685 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-05-15T12:36:26.686 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:36:26.686 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 162 k 2024-05-15T12:36:26.686 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 494 k 2024-05-15T12:36:26.686 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-05-15T12:36:26.778 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-05-15T12:36:26.778 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:26.778 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-05-15T12:36:26.778 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse-2:17.2.7-945.g72b0874a.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-05-15T12:36:26.778 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:26.778 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-05-15T12:36:27.003 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librados-devel 2024-05-15T12:36:27.273 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-immutable-object-cache-17.2.7-945.g72b0874 275 kB/s | 162 kB 00:00 2024-05-15T12:36:27.273 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:36:27.274 INFO:teuthology.orchestra.run.smithi027.stdout:Total 275 kB/s | 162 kB 00:00 2024-05-15T12:36:27.274 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-05-15T12:36:27.288 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-05-15T12:36:27.288 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-05-15T12:36:27.344 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-05-15T12:36:27.347 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-05-15T12:36:27.769 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:43 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:27.785 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-05-15T12:36:27.946 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-05-15T12:36:27.946 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:36:27.947 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-05-15T12:36:27.947 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:36:27.947 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-05-15T12:36:27.947 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse x86_64 2:17.2.7-945.g72b0874a.el8 ceph 876 k 2024-05-15T12:36:27.947 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-05-15T12:36:27.947 INFO:teuthology.orchestra.run.smithi120.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-05-15T12:36:27.947 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:27.947 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-05-15T12:36:27.947 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:36:27.947 INFO:teuthology.orchestra.run.smithi120.stdout:Install 2 Packages 2024-05-15T12:36:27.947 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:27.948 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 959 k 2024-05-15T12:36:27.948 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 3.1 M 2024-05-15T12:36:27.948 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-05-15T12:36:27.999 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-945.g72b0874a.e 1/1 2024-05-15T12:36:28.278 INFO:teuthology.orchestra.run.smithi120.stdout:(1/2): fuse-2.9.7-19.el8.x86_64.rpm 614 kB/s | 83 kB 00:00 2024-05-15T12:36:28.387 INFO:teuthology.orchestra.run.smithi120.stdout:(2/2): ceph-fuse-17.2.7-945.g72b0874a.el8.x86_6 3.5 MB/s | 876 kB 00:00 2024-05-15T12:36:28.387 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:36:28.388 INFO:teuthology.orchestra.run.smithi120.stdout:Total 2.1 MB/s | 959 kB 00:00 2024-05-15T12:36:28.397 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-05-15T12:36:28.417 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-05-15T12:36:28.417 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-05-15T12:36:28.443 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-945.g72b0874a.e 1/1 2024-05-15T12:36:28.499 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-05-15T12:36:28.502 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-05-15T12:36:28.686 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-05-15T12:36:28.875 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-945.g72b0874a.e 1/1 2024-05-15T12:36:28.875 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:36:28.875 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-05-15T12:36:28.875 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:28.876 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:36:28.876 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-15T12:36:28.983 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-05-15T12:36:29.126 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr 2024-05-15T12:36:29.197 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-fuse-2:17.2.7-945.g72b0874a.el8.x86_64 2/2 2024-05-15T12:36:29.437 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-05-15T12:36:29.438 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:36:29.438 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-05-15T12:36:29.438 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:36:29.438 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-05-15T12:36:29.438 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel x86_64 2:17.2.7-945.g72b0874a.el8 ceph 130 k 2024-05-15T12:36:29.438 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:29.438 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-05-15T12:36:29.438 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:36:29.438 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-05-15T12:36:29.438 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:29.439 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 130 k 2024-05-15T12:36:29.439 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 464 k 2024-05-15T12:36:29.439 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-05-15T12:36:29.580 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-fuse-2:17.2.7-945.g72b0874a.el8.x86_64 2/2 2024-05-15T12:36:29.581 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-fuse-2:17.2.7-945.g72b0874a.el8.x86_64 1/2 2024-05-15T12:36:29.746 INFO:teuthology.orchestra.run.smithi177.stdout:librados-devel-17.2.7-945.g72b0874a.el8.x86_64. 422 kB/s | 130 kB 00:00 2024-05-15T12:36:29.747 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:36:29.747 INFO:teuthology.orchestra.run.smithi177.stdout:Total 420 kB/s | 130 kB 00:00 2024-05-15T12:36:29.747 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-05-15T12:36:29.753 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-05-15T12:36:29.753 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-05-15T12:36:29.802 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-05-15T12:36:29.805 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-05-15T12:36:29.914 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:45 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:29.921 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-05-15T12:36:29.947 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-05-15T12:36:29.947 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:29.947 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-05-15T12:36:29.947 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse-2:17.2.7-945.g72b0874a.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-05-15T12:36:29.947 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:29.947 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-05-15T12:36:30.139 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librados-devel-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:36:30.214 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install librados-devel 2024-05-15T12:36:30.435 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados-devel-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:36:30.836 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados-devel-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:36:30.836 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:30.836 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-05-15T12:36:30.836 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:30.836 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:30.837 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-05-15T12:36:30.973 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:47 ago on Wed 15 May 2024 12:34:43 PM UTC. 2024-05-15T12:36:31.059 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install libcephfs2 2024-05-15T12:36:31.629 INFO:teuthology.orchestra.run.smithi027.stdout:Package ceph-mgr-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:31.681 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-15T12:36:31.682 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-05-15T12:36:31.682 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-15T12:36:31.793 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-fuse 2024-05-15T12:36:31.815 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:47 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:32.578 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:48 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:32.643 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-05-15T12:36:32.643 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:36:32.644 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-05-15T12:36:32.644 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:36:32.644 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-05-15T12:36:32.644 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel x86_64 2:17.2.7-945.g72b0874a.el8 ceph 130 k 2024-05-15T12:36:32.644 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:32.644 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-05-15T12:36:32.644 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:36:32.645 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-05-15T12:36:32.645 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:32.646 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 130 k 2024-05-15T12:36:32.646 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 464 k 2024-05-15T12:36:32.646 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-05-15T12:36:32.828 INFO:teuthology.orchestra.run.smithi120.stdout:librados-devel-17.2.7-945.g72b0874a.el8.x86_64. 710 kB/s | 130 kB 00:00 2024-05-15T12:36:32.829 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:36:32.829 INFO:teuthology.orchestra.run.smithi120.stdout:Total 705 kB/s | 130 kB 00:00 2024-05-15T12:36:32.829 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-05-15T12:36:32.834 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-05-15T12:36:32.834 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-05-15T12:36:32.881 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-05-15T12:36:32.884 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-05-15T12:36:32.991 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-05-15T12:36:33.188 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librados-devel-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:36:33.390 INFO:teuthology.orchestra.run.smithi177.stdout:Package libcephfs2-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:33.439 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-05-15T12:36:33.441 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-05-15T12:36:33.441 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-05-15T12:36:33.474 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados-devel-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:36:33.548 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install libcephfs-devel 2024-05-15T12:36:33.837 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados-devel-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:36:33.837 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:33.837 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-05-15T12:36:33.837 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:33.837 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:33.838 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-05-15T12:36:34.074 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install libcephfs2 2024-05-15T12:36:34.310 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:50 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:34.347 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-15T12:36:34.348 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:36:34.348 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-05-15T12:36:34.349 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:36:34.349 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-05-15T12:36:34.349 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse x86_64 2:17.2.7-945.g72b0874a.el8 ceph 876 k 2024-05-15T12:36:34.349 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-05-15T12:36:34.349 INFO:teuthology.orchestra.run.smithi027.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-05-15T12:36:34.349 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:36:34.349 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-05-15T12:36:34.349 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:36:34.349 INFO:teuthology.orchestra.run.smithi027.stdout:Install 2 Packages 2024-05-15T12:36:34.349 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:36:34.350 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 959 k 2024-05-15T12:36:34.350 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 3.1 M 2024-05-15T12:36:34.350 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-05-15T12:36:34.551 INFO:teuthology.orchestra.run.smithi027.stdout:(1/2): fuse-2.9.7-19.el8.x86_64.rpm 577 kB/s | 83 kB 00:00 2024-05-15T12:36:34.835 INFO:teuthology.orchestra.run.smithi027.stdout:(2/2): ceph-fuse-17.2.7-945.g72b0874a.el8.x86_6 2.0 MB/s | 876 kB 00:00 2024-05-15T12:36:34.835 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:36:34.836 INFO:teuthology.orchestra.run.smithi027.stdout:Total 1.9 MB/s | 959 kB 00:00 2024-05-15T12:36:34.837 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:51 ago on Wed 15 May 2024 12:34:43 PM UTC. 2024-05-15T12:36:34.846 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-05-15T12:36:34.868 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-05-15T12:36:34.868 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-05-15T12:36:34.956 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-05-15T12:36:34.960 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-05-15T12:36:35.158 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-05-15T12:36:35.453 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-05-15T12:36:35.704 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-fuse-2:17.2.7-945.g72b0874a.el8.x86_64 2/2 2024-05-15T12:36:35.970 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-05-15T12:36:35.970 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:36:35.970 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-05-15T12:36:35.971 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:36:35.971 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-05-15T12:36:35.971 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel x86_64 2:17.2.7-945.g72b0874a.el8 ceph 25 k 2024-05-15T12:36:35.971 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:35.971 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-05-15T12:36:35.971 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:36:35.971 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-05-15T12:36:35.971 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:35.972 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 25 k 2024-05-15T12:36:35.972 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 109 k 2024-05-15T12:36:35.972 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-05-15T12:36:36.098 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-fuse-2:17.2.7-945.g72b0874a.el8.x86_64 2/2 2024-05-15T12:36:36.098 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-fuse-2:17.2.7-945.g72b0874a.el8.x86_64 1/2 2024-05-15T12:36:36.109 INFO:teuthology.orchestra.run.smithi177.stdout:libcephfs-devel-17.2.7-945.g72b0874a.el8.x86_64 181 kB/s | 25 kB 00:00 2024-05-15T12:36:36.110 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:36:36.110 INFO:teuthology.orchestra.run.smithi177.stdout:Total 179 kB/s | 25 kB 00:00 2024-05-15T12:36:36.110 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-05-15T12:36:36.114 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-05-15T12:36:36.114 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-05-15T12:36:36.115 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-05-15T12:36:36.118 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-05-15T12:36:36.172 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-05-15T12:36:36.329 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs-devel-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:36:36.441 INFO:teuthology.orchestra.run.smithi120.stdout:Package libcephfs2-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:36.490 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-05-15T12:36:36.492 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-05-15T12:36:36.492 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-05-15T12:36:36.501 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-05-15T12:36:36.501 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:36:36.501 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-05-15T12:36:36.501 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse-2:17.2.7-945.g72b0874a.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-05-15T12:36:36.501 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:36:36.501 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-15T12:36:36.587 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:36:36.606 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install libcephfs-devel 2024-05-15T12:36:36.722 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librados-devel 2024-05-15T12:36:36.941 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs-devel-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:36:36.941 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:36.942 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-05-15T12:36:36.942 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:36.942 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:36.942 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-05-15T12:36:37.073 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librados2 2024-05-15T12:36:37.355 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:54 ago on Wed 15 May 2024 12:34:43 PM UTC. 2024-05-15T12:36:37.517 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:53 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:37.830 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:53 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:38.994 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-05-15T12:36:38.995 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:36:38.995 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-05-15T12:36:38.995 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:36:38.995 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-05-15T12:36:38.995 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs-devel x86_64 2:17.2.7-945.g72b0874a.el8 ceph 25 k 2024-05-15T12:36:38.995 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:38.995 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-05-15T12:36:38.995 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:36:38.995 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-05-15T12:36:38.995 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:38.996 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 25 k 2024-05-15T12:36:38.996 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 109 k 2024-05-15T12:36:38.996 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-05-15T12:36:39.224 INFO:teuthology.orchestra.run.smithi120.stdout:libcephfs-devel-17.2.7-945.g72b0874a.el8.x86_64 109 kB/s | 25 kB 00:00 2024-05-15T12:36:39.225 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:36:39.225 INFO:teuthology.orchestra.run.smithi120.stdout:Total 108 kB/s | 25 kB 00:00 2024-05-15T12:36:39.225 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-05-15T12:36:39.229 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-05-15T12:36:39.229 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-05-15T12:36:39.230 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-05-15T12:36:39.234 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-05-15T12:36:39.285 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-05-15T12:36:39.305 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-15T12:36:39.305 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:36:39.305 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-05-15T12:36:39.305 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:36:39.306 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-05-15T12:36:39.306 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel x86_64 2:17.2.7-945.g72b0874a.el8 ceph 130 k 2024-05-15T12:36:39.306 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:36:39.306 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-05-15T12:36:39.306 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:36:39.306 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-05-15T12:36:39.306 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:36:39.307 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 130 k 2024-05-15T12:36:39.307 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 464 k 2024-05-15T12:36:39.307 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-05-15T12:36:39.441 INFO:teuthology.orchestra.run.smithi177.stdout:Package librados2-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:39.479 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs-devel-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:36:39.490 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-05-15T12:36:39.491 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-05-15T12:36:39.491 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-05-15T12:36:39.499 INFO:teuthology.orchestra.run.smithi027.stdout:librados-devel-17.2.7-945.g72b0874a.el8.x86_64. 674 kB/s | 130 kB 00:00 2024-05-15T12:36:39.500 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:36:39.500 INFO:teuthology.orchestra.run.smithi027.stdout:Total 670 kB/s | 130 kB 00:00 2024-05-15T12:36:39.500 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-05-15T12:36:39.506 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-05-15T12:36:39.506 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-05-15T12:36:39.558 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-05-15T12:36:39.561 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-05-15T12:36:39.612 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librbd1 2024-05-15T12:36:39.675 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-05-15T12:36:39.734 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:36:39.879 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librados-devel-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:36:40.085 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs-devel-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:36:40.085 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:40.085 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-05-15T12:36:40.085 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs-devel-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:40.085 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:40.085 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-05-15T12:36:40.219 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install librados2 2024-05-15T12:36:40.221 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados-devel-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:36:40.353 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:56 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:40.664 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados-devel-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:36:40.665 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:36:40.665 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-05-15T12:36:40.665 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:40.665 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:36:40.665 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-15T12:36:40.943 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install libcephfs2 2024-05-15T12:36:40.964 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:57 ago on Wed 15 May 2024 12:34:43 PM UTC. 2024-05-15T12:36:41.739 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:57 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:41.948 INFO:teuthology.orchestra.run.smithi177.stdout:Package librbd1-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:41.996 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-05-15T12:36:41.998 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-05-15T12:36:41.998 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-05-15T12:36:42.111 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rados 2024-05-15T12:36:42.580 INFO:teuthology.orchestra.run.smithi120.stdout:Package librados2-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:42.629 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-05-15T12:36:42.630 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-05-15T12:36:42.630 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-05-15T12:36:42.740 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install librbd1 2024-05-15T12:36:42.859 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:58 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:43.459 INFO:teuthology.orchestra.run.smithi027.stdout:Package libcephfs2-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:43.481 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:02:00 ago on Wed 15 May 2024 12:34:43 PM UTC. 2024-05-15T12:36:43.512 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-15T12:36:43.513 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-05-15T12:36:43.513 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-15T12:36:44.063 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install libcephfs-devel 2024-05-15T12:36:44.454 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-rados-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:44.503 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-05-15T12:36:44.504 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-05-15T12:36:44.504 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-05-15T12:36:44.605 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rgw 2024-05-15T12:36:44.857 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:00 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:45.076 INFO:teuthology.orchestra.run.smithi120.stdout:Package librbd1-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:45.124 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-05-15T12:36:45.126 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-05-15T12:36:45.126 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-05-15T12:36:45.244 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-rados 2024-05-15T12:36:45.355 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:02:01 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:45.987 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:02:02 ago on Wed 15 May 2024 12:34:43 PM UTC. 2024-05-15T12:36:46.637 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-15T12:36:46.638 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:36:46.638 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-05-15T12:36:46.638 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:36:46.638 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-05-15T12:36:46.638 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel x86_64 2:17.2.7-945.g72b0874a.el8 ceph 25 k 2024-05-15T12:36:46.638 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:36:46.638 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-05-15T12:36:46.638 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:36:46.638 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-05-15T12:36:46.638 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:36:46.639 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 25 k 2024-05-15T12:36:46.639 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 109 k 2024-05-15T12:36:46.639 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-05-15T12:36:46.814 INFO:teuthology.orchestra.run.smithi027.stdout:libcephfs-devel-17.2.7-945.g72b0874a.el8.x86_64 142 kB/s | 25 kB 00:00 2024-05-15T12:36:46.815 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:36:46.815 INFO:teuthology.orchestra.run.smithi027.stdout:Total 141 kB/s | 25 kB 00:00 2024-05-15T12:36:46.815 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-05-15T12:36:46.819 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-05-15T12:36:46.819 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-05-15T12:36:46.821 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-05-15T12:36:46.824 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-05-15T12:36:46.882 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-05-15T12:36:46.945 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-rgw-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:46.996 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-05-15T12:36:46.997 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-05-15T12:36:46.997 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-05-15T12:36:47.036 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs-devel-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:36:47.124 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-cephfs 2024-05-15T12:36:47.346 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:36:47.580 INFO:teuthology.orchestra.run.smithi120.stdout:Package python3-rados-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:47.628 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-05-15T12:36:47.630 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-05-15T12:36:47.630 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-05-15T12:36:47.733 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs-devel-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:36:47.733 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:36:47.733 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-05-15T12:36:47.733 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:47.733 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:36:47.733 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-15T12:36:47.749 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-rgw 2024-05-15T12:36:47.848 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librados2 2024-05-15T12:36:47.881 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:02:03 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:48.499 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:02:05 ago on Wed 15 May 2024 12:34:43 PM UTC. 2024-05-15T12:36:48.630 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:04 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:49.486 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-cephfs-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:49.536 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-05-15T12:36:49.537 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-05-15T12:36:49.537 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-05-15T12:36:49.639 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rbd 2024-05-15T12:36:50.108 INFO:teuthology.orchestra.run.smithi120.stdout:Package python3-rgw-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:50.157 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-05-15T12:36:50.158 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-05-15T12:36:50.158 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-05-15T12:36:50.287 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-cephfs 2024-05-15T12:36:50.362 INFO:teuthology.orchestra.run.smithi027.stdout:Package librados2-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:50.385 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:02:06 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:50.415 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-15T12:36:50.417 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-05-15T12:36:50.417 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-15T12:36:50.530 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librbd1 2024-05-15T12:36:51.034 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:02:08 ago on Wed 15 May 2024 12:34:43 PM UTC. 2024-05-15T12:36:51.313 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:07 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:51.978 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-rbd-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:52.027 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-05-15T12:36:52.028 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-05-15T12:36:52.028 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-05-15T12:36:52.138 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-fuse 2024-05-15T12:36:52.629 INFO:teuthology.orchestra.run.smithi120.stdout:Package python3-cephfs-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:52.678 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-05-15T12:36:52.680 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-05-15T12:36:52.680 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-05-15T12:36:52.797 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-rbd 2024-05-15T12:36:52.891 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:02:08 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:53.035 INFO:teuthology.orchestra.run.smithi027.stdout:Package librbd1-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:53.088 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-15T12:36:53.089 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-05-15T12:36:53.089 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-15T12:36:53.186 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rados 2024-05-15T12:36:53.545 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:02:10 ago on Wed 15 May 2024 12:34:43 PM UTC. 2024-05-15T12:36:53.971 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:09 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:54.546 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-05-15T12:36:54.549 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:36:54.549 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-05-15T12:36:54.549 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:36:54.550 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-05-15T12:36:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse x86_64 2:17.2.7-945.g72b0874a.el8 ceph 90 k 2024-05-15T12:36:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:54.550 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-05-15T12:36:54.550 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:36:54.550 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-05-15T12:36:54.550 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:54.550 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 90 k 2024-05-15T12:36:54.550 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 268 k 2024-05-15T12:36:54.550 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-05-15T12:36:54.936 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-fuse-17.2.7-945.g72b0874a.el8.x86_64.rpm 233 kB/s | 90 kB 00:00 2024-05-15T12:36:54.936 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:36:54.936 INFO:teuthology.orchestra.run.smithi177.stdout:Total 232 kB/s | 90 kB 00:00 2024-05-15T12:36:54.937 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-05-15T12:36:54.946 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-05-15T12:36:54.946 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-05-15T12:36:54.993 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-05-15T12:36:54.996 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-05-15T12:36:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-05-15T12:36:55.129 INFO:teuthology.orchestra.run.smithi120.stdout:Package python3-rbd-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:55.180 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-05-15T12:36:55.181 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-05-15T12:36:55.182 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-05-15T12:36:55.276 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-fuse-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:36:55.293 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install rbd-fuse 2024-05-15T12:36:55.548 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-fuse-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:36:55.687 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rados-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:55.741 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-15T12:36:55.742 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-05-15T12:36:55.742 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-15T12:36:55.844 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rgw 2024-05-15T12:36:55.928 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-fuse-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:36:55.929 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:55.929 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-05-15T12:36:55.929 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:55.929 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:55.929 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-05-15T12:36:56.054 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:02:13 ago on Wed 15 May 2024 12:34:43 PM UTC. 2024-05-15T12:36:56.161 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-mirror 2024-05-15T12:36:56.630 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:12 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:56.919 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:02:12 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:57.702 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-05-15T12:36:57.703 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:36:57.703 INFO:teuthology.orchestra.run.smithi120.stdout: Package Architecture Version Repository Size 2024-05-15T12:36:57.703 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:36:57.703 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-05-15T12:36:57.703 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse x86_64 2:17.2.7-945.g72b0874a.el8 ceph 90 k 2024-05-15T12:36:57.703 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:57.703 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-05-15T12:36:57.703 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:36:57.703 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-05-15T12:36:57.703 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:57.704 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 90 k 2024-05-15T12:36:57.704 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 268 k 2024-05-15T12:36:57.704 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-05-15T12:36:57.870 INFO:teuthology.orchestra.run.smithi120.stdout:rbd-fuse-17.2.7-945.g72b0874a.el8.x86_64.rpm 544 kB/s | 90 kB 00:00 2024-05-15T12:36:57.871 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:36:57.871 INFO:teuthology.orchestra.run.smithi120.stdout:Total 540 kB/s | 90 kB 00:00 2024-05-15T12:36:57.871 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-05-15T12:36:57.881 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-05-15T12:36:57.881 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-05-15T12:36:57.928 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-05-15T12:36:57.931 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-05-15T12:36:58.032 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-05-15T12:36:58.234 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-fuse-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:36:58.350 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rgw-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:36:58.404 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-15T12:36:58.405 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-05-15T12:36:58.405 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-15T12:36:58.509 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-cephfs 2024-05-15T12:36:58.520 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-fuse-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:36:58.580 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-05-15T12:36:58.581 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:36:58.581 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-05-15T12:36:58.581 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:36:58.581 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-05-15T12:36:58.581 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror x86_64 2:17.2.7-945.g72b0874a.el8 ceph 3.5 M 2024-05-15T12:36:58.581 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:58.581 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-05-15T12:36:58.581 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:36:58.581 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-05-15T12:36:58.582 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:36:58.582 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 3.5 M 2024-05-15T12:36:58.582 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 16 M 2024-05-15T12:36:58.582 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-05-15T12:36:58.888 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-fuse-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:36:58.888 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:58.888 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-05-15T12:36:58.888 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:36:58.888 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:36:58.888 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-05-15T12:36:59.121 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install rbd-mirror 2024-05-15T12:36:59.303 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:15 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:36:59.730 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-mirror-17.2.7-945.g72b0874a.el8.x86_64.rpm 3.0 MB/s | 3.5 MB 00:01 2024-05-15T12:36:59.731 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:36:59.731 INFO:teuthology.orchestra.run.smithi177.stdout:Total 3.0 MB/s | 3.5 MB 00:01 2024-05-15T12:36:59.731 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-05-15T12:36:59.744 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-05-15T12:36:59.744 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-05-15T12:36:59.807 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-05-15T12:36:59.811 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-05-15T12:36:59.877 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:02:16 ago on Wed 15 May 2024 12:34:43 PM UTC. 2024-05-15T12:37:00.243 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-05-15T12:37:00.444 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-mirror-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:37:00.866 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-mirror-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:37:01.021 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-cephfs-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:37:01.074 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-15T12:37:01.076 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-05-15T12:37:01.076 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-15T12:37:01.178 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rbd 2024-05-15T12:37:01.364 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-mirror-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:37:01.365 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:37:01.365 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-05-15T12:37:01.365 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:37:01.365 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:37:01.365 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-05-15T12:37:01.523 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-05-15T12:37:01.524 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:37:01.524 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-05-15T12:37:01.524 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:37:01.524 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-05-15T12:37:01.524 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror x86_64 2:17.2.7-945.g72b0874a.el8 ceph 3.5 M 2024-05-15T12:37:01.524 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:37:01.524 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-05-15T12:37:01.524 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:37:01.524 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-05-15T12:37:01.525 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:37:01.525 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 3.5 M 2024-05-15T12:37:01.525 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 16 M 2024-05-15T12:37:01.525 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-05-15T12:37:01.616 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-nbd 2024-05-15T12:37:01.968 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:17 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:37:02.373 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:02:18 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:37:02.658 INFO:teuthology.orchestra.run.smithi120.stdout:rbd-mirror-17.2.7-945.g72b0874a.el8.x86_64.rpm 3.1 MB/s | 3.5 MB 00:01 2024-05-15T12:37:02.658 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:37:02.659 INFO:teuthology.orchestra.run.smithi120.stdout:Total 3.0 MB/s | 3.5 MB 00:01 2024-05-15T12:37:02.659 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-05-15T12:37:02.672 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-05-15T12:37:02.672 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-05-15T12:37:02.735 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-05-15T12:37:02.739 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-05-15T12:37:03.178 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-05-15T12:37:03.344 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-mirror-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:37:03.677 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rbd-2:17.2.7-945.g72b0874a.el8.x86_64 is already installed. 2024-05-15T12:37:03.733 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-15T12:37:03.734 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-05-15T12:37:03.734 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-15T12:37:03.759 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-mirror-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:37:03.834 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-fuse 2024-05-15T12:37:04.028 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-05-15T12:37:04.028 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:37:04.029 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-05-15T12:37:04.029 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:37:04.029 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-05-15T12:37:04.029 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd x86_64 2:17.2.7-945.g72b0874a.el8 ceph 175 k 2024-05-15T12:37:04.029 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:37:04.029 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-05-15T12:37:04.029 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-05-15T12:37:04.029 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-05-15T12:37:04.029 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:37:04.030 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 175 k 2024-05-15T12:37:04.030 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 556 k 2024-05-15T12:37:04.030 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-05-15T12:37:04.147 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-mirror-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:37:04.147 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:37:04.147 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-05-15T12:37:04.147 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:37:04.147 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:37:04.147 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-05-15T12:37:04.359 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install rbd-nbd 2024-05-15T12:37:04.623 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:20 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:37:04.749 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-nbd-17.2.7-945.g72b0874a.el8.x86_64.rpm 243 kB/s | 175 kB 00:00 2024-05-15T12:37:04.749 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:37:04.750 INFO:teuthology.orchestra.run.smithi177.stdout:Total 242 kB/s | 175 kB 00:00 2024-05-15T12:37:04.750 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-05-15T12:37:04.761 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-05-15T12:37:04.761 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-05-15T12:37:04.810 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-05-15T12:37:04.813 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-05-15T12:37:04.925 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-05-15T12:37:05.115 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:02:22 ago on Wed 15 May 2024 12:34:43 PM UTC. 2024-05-15T12:37:05.123 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-nbd-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:37:05.416 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-nbd-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:37:05.785 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-nbd-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:37:05.785 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:37:05.785 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-05-15T12:37:05.785 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:37:05.785 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:37:05.785 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-05-15T12:37:06.014 DEBUG:teuthology.parallel:result is None 2024-05-15T12:37:06.428 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-15T12:37:06.429 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:37:06.429 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-05-15T12:37:06.429 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:37:06.429 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-05-15T12:37:06.429 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse x86_64 2:17.2.7-945.g72b0874a.el8 ceph 90 k 2024-05-15T12:37:06.429 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:37:06.429 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-05-15T12:37:06.429 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:37:06.429 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-05-15T12:37:06.429 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:37:06.430 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 90 k 2024-05-15T12:37:06.430 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 268 k 2024-05-15T12:37:06.430 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-05-15T12:37:06.790 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-05-15T12:37:06.791 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:37:06.791 INFO:teuthology.orchestra.run.smithi120.stdout: Package Architecture Version Repository Size 2024-05-15T12:37:06.791 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:37:06.791 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-05-15T12:37:06.792 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd x86_64 2:17.2.7-945.g72b0874a.el8 ceph 175 k 2024-05-15T12:37:06.792 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:37:06.792 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-05-15T12:37:06.792 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-05-15T12:37:06.792 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-05-15T12:37:06.792 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:37:06.792 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 175 k 2024-05-15T12:37:06.792 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 556 k 2024-05-15T12:37:06.792 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-05-15T12:37:06.832 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-fuse-17.2.7-945.g72b0874a.el8.x86_64.rpm 225 kB/s | 90 kB 00:00 2024-05-15T12:37:06.833 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:37:06.833 INFO:teuthology.orchestra.run.smithi027.stdout:Total 224 kB/s | 90 kB 00:00 2024-05-15T12:37:06.833 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-05-15T12:37:06.847 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-05-15T12:37:06.848 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-05-15T12:37:06.894 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-05-15T12:37:06.898 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-05-15T12:37:07.008 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-05-15T12:37:07.275 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-fuse-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:37:07.312 INFO:teuthology.orchestra.run.smithi120.stdout:rbd-nbd-17.2.7-945.g72b0874a.el8.x86_64.rpm 336 kB/s | 175 kB 00:00 2024-05-15T12:37:07.312 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:37:07.313 INFO:teuthology.orchestra.run.smithi120.stdout:Total 335 kB/s | 175 kB 00:00 2024-05-15T12:37:07.313 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-05-15T12:37:07.324 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-05-15T12:37:07.324 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-05-15T12:37:07.374 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-05-15T12:37:07.377 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-05-15T12:37:07.491 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-05-15T12:37:07.549 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-fuse-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:37:07.712 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-nbd-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:37:07.951 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-fuse-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:37:07.951 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:37:07.951 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-05-15T12:37:07.951 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:37:07.951 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:37:07.951 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-15T12:37:08.004 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-nbd-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:37:08.186 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-mirror 2024-05-15T12:37:08.397 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-nbd-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:37:08.398 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:37:08.398 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-05-15T12:37:08.398 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:37:08.398 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-05-15T12:37:08.398 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-05-15T12:37:08.645 DEBUG:teuthology.parallel:result is None 2024-05-15T12:37:08.979 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:24 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:37:10.756 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-15T12:37:10.757 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:37:10.757 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-05-15T12:37:10.757 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:37:10.757 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-05-15T12:37:10.757 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror x86_64 2:17.2.7-945.g72b0874a.el8 ceph 3.5 M 2024-05-15T12:37:10.757 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:37:10.757 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-05-15T12:37:10.757 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:37:10.757 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-05-15T12:37:10.758 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:37:10.758 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 3.5 M 2024-05-15T12:37:10.758 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 16 M 2024-05-15T12:37:10.758 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-05-15T12:37:11.444 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-mirror-17.2.7-945.g72b0874a.el8.x86_64.rpm 5.0 MB/s | 3.5 MB 00:00 2024-05-15T12:37:11.445 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:37:11.445 INFO:teuthology.orchestra.run.smithi027.stdout:Total 5.0 MB/s | 3.5 MB 00:00 2024-05-15T12:37:11.445 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-05-15T12:37:11.459 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-05-15T12:37:11.459 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-05-15T12:37:11.526 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-05-15T12:37:11.529 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-05-15T12:37:11.996 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-05-15T12:37:12.185 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-mirror-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:37:12.609 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-mirror-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:37:13.096 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-mirror-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:37:13.096 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:37:13.096 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-05-15T12:37:13.096 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:37:13.096 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:37:13.096 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-15T12:37:13.341 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-nbd 2024-05-15T12:37:14.128 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:30 ago on Wed 15 May 2024 12:34:44 PM UTC. 2024-05-15T12:37:15.900 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-05-15T12:37:15.901 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:37:15.901 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-05-15T12:37:15.901 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:37:15.901 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-05-15T12:37:15.901 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd x86_64 2:17.2.7-945.g72b0874a.el8 ceph 175 k 2024-05-15T12:37:15.901 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:37:15.901 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-05-15T12:37:15.901 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-05-15T12:37:15.901 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-05-15T12:37:15.901 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:37:15.902 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 175 k 2024-05-15T12:37:15.902 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 556 k 2024-05-15T12:37:15.902 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-05-15T12:37:16.198 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-nbd-17.2.7-945.g72b0874a.el8.x86_64.rpm 591 kB/s | 175 kB 00:00 2024-05-15T12:37:16.198 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:37:16.198 INFO:teuthology.orchestra.run.smithi027.stdout:Total 588 kB/s | 175 kB 00:00 2024-05-15T12:37:16.199 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-05-15T12:37:16.210 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-05-15T12:37:16.211 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-05-15T12:37:16.261 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-05-15T12:37:16.266 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-05-15T12:37:16.416 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-05-15T12:37:16.656 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-nbd-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:37:16.940 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-nbd-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:37:17.323 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-nbd-2:17.2.7-945.g72b0874a.el8.x86_64 1/1 2024-05-15T12:37:17.323 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:37:17.323 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-05-15T12:37:17.323 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd-2:17.2.7-945.g72b0874a.el8.x86_64 2024-05-15T12:37:17.323 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:37:17.324 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-05-15T12:37:17.580 DEBUG:teuthology.parallel:result is None 2024-05-15T12:37:17.581 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-15T12:37:17.581 INFO:teuthology.packaging:ref: None 2024-05-15T12:37:17.581 INFO:teuthology.packaging:tag: None 2024-05-15T12:37:17.581 INFO:teuthology.packaging:branch: quincy 2024-05-15T12:37:17.581 INFO:teuthology.packaging:sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:37:17.581 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-05-15T12:37:17.884 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-15T12:37:17.908 INFO:teuthology.orchestra.run.smithi027.stdout:17.2.7-945.g72b0874a.el8 2024-05-15T12:37:17.908 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-945.g72b0874a.el8 2024-05-15T12:37:17.908 INFO:teuthology.task.install:The correct ceph version 17.2.7-945.g72b0874a is installed. 2024-05-15T12:37:17.910 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-15T12:37:17.910 INFO:teuthology.packaging:ref: None 2024-05-15T12:37:17.910 INFO:teuthology.packaging:tag: None 2024-05-15T12:37:17.910 INFO:teuthology.packaging:branch: quincy 2024-05-15T12:37:17.910 INFO:teuthology.packaging:sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:37:17.910 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-05-15T12:37:18.206 DEBUG:teuthology.orchestra.run.smithi120:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-15T12:37:18.233 INFO:teuthology.orchestra.run.smithi120.stdout:17.2.7-945.g72b0874a.el8 2024-05-15T12:37:18.233 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-945.g72b0874a.el8 2024-05-15T12:37:18.233 INFO:teuthology.task.install:The correct ceph version 17.2.7-945.g72b0874a is installed. 2024-05-15T12:37:18.235 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-15T12:37:18.235 INFO:teuthology.packaging:ref: None 2024-05-15T12:37:18.235 INFO:teuthology.packaging:tag: None 2024-05-15T12:37:18.235 INFO:teuthology.packaging:branch: quincy 2024-05-15T12:37:18.235 INFO:teuthology.packaging:sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:37:18.235 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-05-15T12:37:18.455 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-15T12:37:18.482 INFO:teuthology.orchestra.run.smithi177.stdout:17.2.7-945.g72b0874a.el8 2024-05-15T12:37:18.482 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-945.g72b0874a.el8 2024-05-15T12:37:18.483 INFO:teuthology.task.install:The correct ceph version 17.2.7-945.g72b0874a is installed. 2024-05-15T12:37:18.484 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-15T12:37:18.484 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-15T12:37:18.485 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-15T12:37:18.518 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-05-15T12:37:18.518 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-15T12:37:18.551 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-05-15T12:37:18.551 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-15T12:37:18.583 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-15T12:37:18.583 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-15T12:37:18.584 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/daemon-helper 2024-05-15T12:37:18.615 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-15T12:37:18.685 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-05-15T12:37:18.685 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/daemon-helper 2024-05-15T12:37:18.716 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-15T12:37:18.785 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-05-15T12:37:18.785 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/daemon-helper 2024-05-15T12:37:18.817 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-15T12:37:18.889 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-15T12:37:18.889 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-15T12:37:18.889 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-15T12:37:18.922 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-15T12:37:18.993 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-05-15T12:37:18.994 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-15T12:37:19.024 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-15T12:37:19.093 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-05-15T12:37:19.094 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-15T12:37:19.126 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-15T12:37:19.196 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-15T12:37:19.196 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-15T12:37:19.196 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/stdin-killer 2024-05-15T12:37:19.230 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-15T12:37:19.302 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-05-15T12:37:19.302 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/stdin-killer 2024-05-15T12:37:19.334 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-15T12:37:19.404 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-05-15T12:37:19.404 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/stdin-killer 2024-05-15T12:37:19.435 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-15T12:37:19.505 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-15T12:37:19.606 INFO:tasks.cephadm:Config: {'conf': {'mon': {'auth allow insecure global id reclaim': True, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314'} 2024-05-15T12:37:19.607 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:37:19.607 INFO:tasks.cephadm:Cluster fsid is deaea04e-12b7-11ef-bc98-c7b262605968 2024-05-15T12:37:19.607 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-15T12:37:19.608 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.27', 'mon.b': '172.21.15.120', 'mon.c': '172.21.15.177'} 2024-05-15T12:37:19.608 INFO:tasks.cephadm:First mon is mon.a on smithi027 2024-05-15T12:37:19.608 INFO:tasks.cephadm:First mgr is y 2024-05-15T12:37:19.608 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-15T12:37:19.608 DEBUG:teuthology.orchestra.run.smithi027:> sudo hostname $(hostname -s) 2024-05-15T12:37:19.643 DEBUG:teuthology.orchestra.run.smithi120:> sudo hostname $(hostname -s) 2024-05-15T12:37:19.676 DEBUG:teuthology.orchestra.run.smithi177:> sudo hostname $(hostname -s) 2024-05-15T12:37:19.707 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-05-15T12:37:19.707 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:37:19.851 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/79382/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1882-gae248125', 'node_name': '172.21.5.31+adami01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1882.gae248125'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2024-05-14 20:15:07.104929', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-osd-ec-partial-reads', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/8/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-05-15T12:37:20.063 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-osd-ec-partial-reads, sha1 ae2481253f9b2fa3451954e615eedb67ebdd5314 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:37:20.064 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/8/x86_64/flavors/default/cephadm 2024-05-15T12:37:20.064 DEBUG:teuthology.orchestra.run.smithi027:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-15T12:37:20.542 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-rw-r--. 1 ubuntu ubuntu 2251844 May 15 12:37 /home/ubuntu/cephtest/cephadm 2024-05-15T12:37:20.543 DEBUG:teuthology.orchestra.run.smithi120:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-15T12:37:21.002 INFO:teuthology.orchestra.run.smithi120.stdout:-rw-rw-r--. 1 ubuntu ubuntu 2251844 May 15 12:37 /home/ubuntu/cephtest/cephadm 2024-05-15T12:37:21.002 DEBUG:teuthology.orchestra.run.smithi177:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-15T12:37:21.655 INFO:teuthology.orchestra.run.smithi177.stdout:-rw-rw-r--. 1 ubuntu ubuntu 2251844 May 15 12:37 /home/ubuntu/cephtest/cephadm 2024-05-15T12:37:21.655 DEBUG:teuthology.orchestra.run.smithi027:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-15T12:37:21.679 DEBUG:teuthology.orchestra.run.smithi120:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-15T12:37:21.701 DEBUG:teuthology.orchestra.run.smithi177:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-15T12:37:21.741 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 on all hosts... 2024-05-15T12:37:21.742 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 pull 2024-05-15T12:37:21.745 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 pull 2024-05-15T12:37:21.747 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 pull 2024-05-15T12:37:22.865 INFO:teuthology.orchestra.run.smithi120.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314... 2024-05-15T12:37:22.951 INFO:teuthology.orchestra.run.smithi177.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314... 2024-05-15T12:37:23.011 INFO:teuthology.orchestra.run.smithi027.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314... 2024-05-15T12:37:58.951 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T12:37:58.951 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_version": "ceph version 19.0.0-1882-gae248125 (ae2481253f9b2fa3451954e615eedb67ebdd5314) squid (dev)", 2024-05-15T12:37:58.951 INFO:teuthology.orchestra.run.smithi120.stdout: "image_id": "bce43dd9c04a5d1efc353fa6a8cab31ce3dda72b4628285173869789664d92a9", 2024-05-15T12:37:58.952 INFO:teuthology.orchestra.run.smithi120.stdout: "repo_digests": [ 2024-05-15T12:37:58.952 INFO:teuthology.orchestra.run.smithi120.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328" 2024-05-15T12:37:58.952 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:37:58.952 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T12:38:10.027 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T12:38:10.027 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-1882-gae248125 (ae2481253f9b2fa3451954e615eedb67ebdd5314) squid (dev)", 2024-05-15T12:38:10.027 INFO:teuthology.orchestra.run.smithi027.stdout: "image_id": "bce43dd9c04a5d1efc353fa6a8cab31ce3dda72b4628285173869789664d92a9", 2024-05-15T12:38:10.027 INFO:teuthology.orchestra.run.smithi027.stdout: "repo_digests": [ 2024-05-15T12:38:10.027 INFO:teuthology.orchestra.run.smithi027.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328" 2024-05-15T12:38:10.027 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:38:10.028 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T12:38:10.808 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T12:38:10.809 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_version": "ceph version 19.0.0-1882-gae248125 (ae2481253f9b2fa3451954e615eedb67ebdd5314) squid (dev)", 2024-05-15T12:38:10.809 INFO:teuthology.orchestra.run.smithi177.stdout: "image_id": "bce43dd9c04a5d1efc353fa6a8cab31ce3dda72b4628285173869789664d92a9", 2024-05-15T12:38:10.809 INFO:teuthology.orchestra.run.smithi177.stdout: "repo_digests": [ 2024-05-15T12:38:10.809 INFO:teuthology.orchestra.run.smithi177.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328" 2024-05-15T12:38:10.810 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:38:10.810 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T12:38:10.836 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph 2024-05-15T12:38:10.876 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /etc/ceph 2024-05-15T12:38:10.914 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph 2024-05-15T12:38:10.946 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 777 /etc/ceph 2024-05-15T12:38:10.979 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 777 /etc/ceph 2024-05-15T12:38:11.010 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 777 /etc/ceph 2024-05-15T12:38:11.038 INFO:tasks.cephadm:Writing seed config... 2024-05-15T12:38:11.039 INFO:tasks.cephadm: override: [mon] auth allow insecure global id reclaim = True 2024-05-15T12:38:11.040 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-15T12:38:11.040 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-15T12:38:11.040 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-15T12:38:11.040 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-05-15T12:38:11.040 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-05-15T12:38:11.040 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-05-15T12:38:11.041 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-05-15T12:38:11.041 INFO:tasks.cephadm: override: [mon] mon scrub interval = 300 2024-05-15T12:38:11.041 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-05-15T12:38:11.041 INFO:tasks.cephadm: override: [client.2] debug ms = 1 2024-05-15T12:38:11.041 INFO:tasks.cephadm: override: [client.2] debug objecter = 20 2024-05-15T12:38:11.041 INFO:tasks.cephadm: override: [client.2] debug rados = 20 2024-05-15T12:38:11.041 INFO:tasks.cephadm: override: [global] mon client directed command retry = 5 2024-05-15T12:38:11.042 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-05-15T12:38:11.042 INFO:tasks.cephadm: override: [global] ms inject socket failures = 2500 2024-05-15T12:38:11.042 INFO:tasks.cephadm: override: [global] ms tcp read timeout = 5 2024-05-15T12:38:11.042 INFO:tasks.cephadm: override: [global] osd_pool_default_min_size = 2 2024-05-15T12:38:11.042 INFO:tasks.cephadm: override: [global] osd_pool_default_size = 3 2024-05-15T12:38:11.042 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-15T12:38:11.042 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-15T12:38:11.043 INFO:tasks.cephadm: override: [osd] bluestore zero block detection = True 2024-05-15T12:38:11.043 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-15T12:38:11.043 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-15T12:38:11.043 INFO:tasks.cephadm: override: [osd] osd blocked scrub grace period = 3600 2024-05-15T12:38:11.043 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-05-15T12:38:11.043 INFO:tasks.cephadm: override: [osd] osd debug verify cached snaps = True 2024-05-15T12:38:11.044 INFO:tasks.cephadm: override: [osd] osd debug verify missing on start = True 2024-05-15T12:38:11.044 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-05-15T12:38:11.044 INFO:tasks.cephadm: override: [osd] osd max markdown count = 1000 2024-05-15T12:38:11.044 INFO:tasks.cephadm: override: [osd] osd mclock override recovery settings = True 2024-05-15T12:38:11.044 INFO:tasks.cephadm: override: [osd] osd mclock profile = high_recovery_ops 2024-05-15T12:38:11.044 INFO:tasks.cephadm: override: [osd] osd op queue = debug_random 2024-05-15T12:38:11.044 INFO:tasks.cephadm: override: [osd] osd op queue cut off = debug_random 2024-05-15T12:38:11.044 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-05-15T12:38:11.045 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-05-15T12:38:11.045 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-05-15T12:38:11.045 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-05-15T12:38:11.046 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-15T12:38:11.046 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-15T12:38:11.069 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = deaea04e-12b7-11ef-bc98-c7b262605968 mon client directed command retry = 5 mon election default strategy = 3 ms inject socket failures = 2500 ms tcp read timeout = 5 osd_pool_default_min_size = 2 osd_pool_default_size = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = True osd debug verify missing on start = True osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore zero block detection = True debug ms = 1 debug osd = 20 osd blocked scrub grace period = 3600 osd debug reject backfill probability = 0.3 osd max backfills = 3 osd max markdown count = 1000 osd mclock override recovery settings = True osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false auth allow insecure global id reclaim = True debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon scrub interval = 300 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.2] debug ms = 1 debug objecter = 20 debug rados = 20 2024-05-15T12:38:11.070 DEBUG:teuthology.orchestra.run.smithi027:mon.a> sudo journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mon.a.service 2024-05-15T12:38:11.112 DEBUG:teuthology.orchestra.run.smithi027:mgr.y> sudo journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mgr.y.service 2024-05-15T12:38:11.145 INFO:journalctl@ceph.mon.a.smithi027.stdout:-- Logs begin at Wed 2024-05-15 12:27:35 UTC. -- 2024-05-15T12:38:11.146 INFO:tasks.cephadm:Bootstrapping... 2024-05-15T12:38:11.146 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 -v bootstrap --fsid deaea04e-12b7-11ef-bc98-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.27 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-15T12:38:11.176 INFO:journalctl@ceph.mgr.y.smithi027.stdout:-- Logs begin at Wed 2024-05-15 12:27:35 UTC. -- 2024-05-15T12:38:11.366 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:38:11.367 INFO:teuthology.orchestra.run.smithi027.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314', '-v', 'bootstrap', '--fsid', 'deaea04e-12b7-11ef-bc98-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.27', '--skip-admin-label'] 2024-05-15T12:38:11.403 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-15T12:38:11.403 INFO:teuthology.orchestra.run.smithi027.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-05-15T12:38:11.404 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying podman|docker is present... 2024-05-15T12:38:11.433 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-15T12:38:11.433 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying lvm2 is present... 2024-05-15T12:38:11.433 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying time synchronization is in place... 2024-05-15T12:38:11.440 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-15T12:38:11.440 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-15T12:38:11.446 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-15T12:38:11.446 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-05-15T12:38:11.452 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout enabled 2024-05-15T12:38:11.458 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout active 2024-05-15T12:38:11.459 INFO:teuthology.orchestra.run.smithi027.stdout:Unit chronyd.service is enabled and running 2024-05-15T12:38:11.459 INFO:teuthology.orchestra.run.smithi027.stdout:Repeating the final host check... 2024-05-15T12:38:11.487 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-15T12:38:11.487 INFO:teuthology.orchestra.run.smithi027.stdout:podman (/bin/podman) version 4.9.4 is present 2024-05-15T12:38:11.487 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl is present 2024-05-15T12:38:11.487 INFO:teuthology.orchestra.run.smithi027.stdout:lvcreate is present 2024-05-15T12:38:11.493 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-15T12:38:11.494 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-15T12:38:11.499 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-15T12:38:11.499 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-05-15T12:38:11.506 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout enabled 2024-05-15T12:38:11.511 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout active 2024-05-15T12:38:11.511 INFO:teuthology.orchestra.run.smithi027.stdout:Unit chronyd.service is enabled and running 2024-05-15T12:38:11.511 INFO:teuthology.orchestra.run.smithi027.stdout:Host looks OK 2024-05-15T12:38:11.512 INFO:teuthology.orchestra.run.smithi027.stdout:Cluster fsid: deaea04e-12b7-11ef-bc98-c7b262605968 2024-05-15T12:38:11.512 INFO:teuthology.orchestra.run.smithi027.stdout:Acquiring lock 140168603079008 on /run/cephadm/deaea04e-12b7-11ef-bc98-c7b262605968.lock 2024-05-15T12:38:11.512 INFO:teuthology.orchestra.run.smithi027.stdout:Lock 140168603079008 acquired on /run/cephadm/deaea04e-12b7-11ef-bc98-c7b262605968.lock 2024-05-15T12:38:11.512 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying IP 172.21.15.27 port 3300 ... 2024-05-15T12:38:11.513 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying IP 172.21.15.27 port 6789 ... 2024-05-15T12:38:11.513 INFO:teuthology.orchestra.run.smithi027.stdout:Base mon IP(s) is [172.21.15.27:3300, 172.21.15.27:6789], mon addrv is [v2:172.21.15.27:3300,v1:172.21.15.27:6789] 2024-05-15T12:38:11.516 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.27 metric 100 2024-05-15T12:38:11.517 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.27 metric 100 2024-05-15T12:38:11.520 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-05-15T12:38:11.520 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-05-15T12:38:11.520 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1798sec hoplimit 64 pref medium 2024-05-15T12:38:11.522 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-15T12:38:11.523 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-05-15T12:38:11.523 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-15T12:38:11.523 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-05-15T12:38:11.523 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-05-15T12:38:11.523 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-15T12:38:11.524 INFO:teuthology.orchestra.run.smithi027.stdout:Mon IP `172.21.15.27` is in CIDR network `172.21.0.0/20` 2024-05-15T12:38:11.524 INFO:teuthology.orchestra.run.smithi027.stdout:Mon IP `172.21.15.27` is in CIDR network `172.21.0.0/20` 2024-05-15T12:38:11.524 INFO:teuthology.orchestra.run.smithi027.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-05-15T12:38:11.524 INFO:teuthology.orchestra.run.smithi027.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-15T12:38:11.525 INFO:teuthology.orchestra.run.smithi027.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314... 2024-05-15T12:38:12.016 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout bce43dd9c04a5d1efc353fa6a8cab31ce3dda72b4628285173869789664d92a9 2024-05-15T12:38:12.016 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314... 2024-05-15T12:38:12.016 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Getting image source signatures 2024-05-15T12:38:12.017 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Copying blob sha256:4502bcc40a1bb6c3aa83c1a9c4cc55deee56ecb239477549ee4bf66585c31c94 2024-05-15T12:38:12.017 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Copying blob sha256:d0e808cd455827ca3108db1b0125dcee9ffc148daa1d7dde963f550bc831d501 2024-05-15T12:38:12.017 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Copying config sha256:bce43dd9c04a5d1efc353fa6a8cab31ce3dda72b4628285173869789664d92a9 2024-05-15T12:38:12.017 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Writing manifest to image destination 2024-05-15T12:38:13.276 INFO:teuthology.orchestra.run.smithi027.stdout:ceph: stdout ceph version 19.0.0-1882-gae248125 (ae2481253f9b2fa3451954e615eedb67ebdd5314) squid (dev) 2024-05-15T12:38:13.276 INFO:teuthology.orchestra.run.smithi027.stdout:Ceph version: ceph version 19.0.0-1882-gae248125 (ae2481253f9b2fa3451954e615eedb67ebdd5314) squid (dev) 2024-05-15T12:38:13.276 INFO:teuthology.orchestra.run.smithi027.stdout:Extracting ceph user uid/gid from container image... 2024-05-15T12:38:14.243 INFO:teuthology.orchestra.run.smithi027.stdout:stat: stdout 167 167 2024-05-15T12:38:14.243 INFO:teuthology.orchestra.run.smithi027.stdout:Creating initial keys... 2024-05-15T12:38:15.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph-authtool: stdout AQC2rERmLfe7KxAA74hPEVTUVvsQZYekkwKX6w== 2024-05-15T12:38:16.610 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph-authtool: stdout AQC3rERmYWc9NBAAXcFcSEiadGiws23+oHsg0Q== 2024-05-15T12:38:17.670 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph-authtool: stdout AQC5rERmh+diBRAACFVkKBcvQm4Lvm1KmEzKHw== 2024-05-15T12:38:17.670 INFO:teuthology.orchestra.run.smithi027.stdout:Creating initial monmap... 2024-05-15T12:38:18.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-15T12:38:18.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-05-15T12:38:18.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to deaea04e-12b7-11ef-bc98-c7b262605968 2024-05-15T12:38:18.895 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-15T12:38:18.895 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool for a [v2:172.21.15.27:3300,v1:172.21.15.27:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-15T12:38:18.895 INFO:teuthology.orchestra.run.smithi027.stdout:setting min_mon_release = quincy 2024-05-15T12:38:18.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: set fsid to deaea04e-12b7-11ef-bc98-c7b262605968 2024-05-15T12:38:18.896 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-15T12:38:18.896 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:38:18.896 INFO:teuthology.orchestra.run.smithi027.stdout:Creating mon... 2024-05-15T12:38:20.036 INFO:teuthology.orchestra.run.smithi027.stdout:create mon.a on 2024-05-15T12:38:20.163 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-05-15T12:38:20.255 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-15T12:38:20.378 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-deaea04e-12b7-11ef-bc98-c7b262605968.target → /etc/systemd/system/ceph-deaea04e-12b7-11ef-bc98-c7b262605968.target. 2024-05-15T12:38:20.378 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-deaea04e-12b7-11ef-bc98-c7b262605968.target → /etc/systemd/system/ceph-deaea04e-12b7-11ef-bc98-c7b262605968.target. 2024-05-15T12:38:20.569 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mon.a 2024-05-15T12:38:20.569 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to reset failed state of unit ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mon.a.service: Unit ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mon.a.service not loaded. 2024-05-15T12:38:20.667 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-deaea04e-12b7-11ef-bc98-c7b262605968.target.wants/ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mon.a.service → /etc/systemd/system/ceph-deaea04e-12b7-11ef-bc98-c7b262605968@.service. 2024-05-15T12:38:21.321 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-15T12:38:21.321 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-05-15T12:38:21.330 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-15T12:38:21.330 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-05-15T12:38:21.330 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-05-15T12:38:21.330 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to enable service . firewalld.service is not available 2024-05-15T12:38:21.330 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mon to start... 2024-05-15T12:38:21.331 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mon... 2024-05-15T12:38:21.519 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:21 smithi027 bash[39272]: 7c6c5b447088af2feee3b700b2b29d15796509ec2670a1b8c765431f12c06623 2024-05-15T12:38:21.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:21 smithi027 systemd[1]: Started Ceph mon.a for deaea04e-12b7-11ef-bc98-c7b262605968. 2024-05-15T12:38:22.508 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout cluster: 2024-05-15T12:38:22.509 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout id: deaea04e-12b7-11ef-bc98-c7b262605968 2024-05-15T12:38:22.509 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-05-15T12:38:22.509 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-05-15T12:38:22.509 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout services: 2024-05-15T12:38:22.509 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.691414s) 2024-05-15T12:38:22.509 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-05-15T12:38:22.509 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-05-15T12:38:22.509 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-05-15T12:38:22.509 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout data: 2024-05-15T12:38:22.509 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-05-15T12:38:22.509 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-05-15T12:38:22.509 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-05-15T12:38:22.510 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout pgs: 2024-05-15T12:38:22.510 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-05-15T12:38:22.510 INFO:teuthology.orchestra.run.smithi027.stdout:mon is available 2024-05-15T12:38:22.510 INFO:teuthology.orchestra.run.smithi027.stdout:Assimilating anything we can from ceph.conf... 2024-05-15T12:38:22.510 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:22 smithi027 ceph-mon[39311]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-15T12:38:22.510 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:22 smithi027 ceph-mon[39311]: monmap e1: 1 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-15T12:38:22.510 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:22 smithi027 ceph-mon[39311]: fsmap 2024-05-15T12:38:22.510 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:22 smithi027 ceph-mon[39311]: osdmap e1: 0 total, 0 up, 0 in 2024-05-15T12:38:22.510 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:22 smithi027 ceph-mon[39311]: mgrmap e1: no daemons active 2024-05-15T12:38:22.511 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:22 smithi027 ceph-mon[39311]: from='client.? 172.21.15.27:0/773458427' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-15T12:38:23.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:23 smithi027 ceph-mon[39311]: from='client.? 172.21.15.27:0/697950280' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-15T12:38:23.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:23 smithi027 ceph-mon[39311]: from='client.? 172.21.15.27:0/697950280' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-15T12:38:23.939 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-05-15T12:38:23.941 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [global] 2024-05-15T12:38:23.941 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout fsid = deaea04e-12b7-11ef-bc98-c7b262605968 2024-05-15T12:38:23.941 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.27:3300,v1:172.21.15.27:6789] 2024-05-15T12:38:23.941 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-15T12:38:23.941 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-15T12:38:23.941 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-15T12:38:23.941 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ms_tcp_read_timeout = 5 2024-05-15T12:38:23.941 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-15T12:38:23.942 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-05-15T12:38:23.942 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-15T12:38:23.942 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-15T12:38:23.942 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-05-15T12:38:23.942 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [osd] 2024-05-15T12:38:23.942 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-15T12:38:23.942 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-15T12:38:23.942 INFO:teuthology.orchestra.run.smithi027.stdout:Generating new minimal ceph.conf... 2024-05-15T12:38:24.875 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:24 smithi027 ceph-mon[39311]: from='client.? 172.21.15.27:0/2295587190' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:38:25.161 INFO:teuthology.orchestra.run.smithi027.stdout:Restarting the monitor... 2024-05-15T12:38:25.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:25 smithi027 systemd[1]: Stopping Ceph mon.a for deaea04e-12b7-11ef-bc98-c7b262605968... 2024-05-15T12:38:25.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:25 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-a[39288]: 2024-05-15T12:38:25.420+0000 7fb2cb343700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T12:38:25.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:25 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-a[39288]: 2024-05-15T12:38:25.420+0000 7fb2cb343700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-15T12:38:26.212 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:25 smithi027 bash[39636]: ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-a 2024-05-15T12:38:26.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 systemd[1]: ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mon.a.service: Succeeded. 2024-05-15T12:38:26.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 systemd[1]: Stopped Ceph mon.a for deaea04e-12b7-11ef-bc98-c7b262605968. 2024-05-15T12:38:26.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 systemd[1]: Starting Ceph mon.a for deaea04e-12b7-11ef-bc98-c7b262605968... 2024-05-15T12:38:26.861 INFO:teuthology.orchestra.run.smithi027.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-05-15T12:38:27.127 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: set uid:gid to 167:167 (ceph:ceph) 2024-05-15T12:38:27.127 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: ceph version 19.0.0-1882-gae248125 (ae2481253f9b2fa3451954e615eedb67ebdd5314) squid (dev), process ceph-mon, pid 7 2024-05-15T12:38:27.128 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: pidfile_write: ignore empty --pid-file 2024-05-15T12:38:27.128 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: load: jerasure load: lrc 2024-05-15T12:38:27.128 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: RocksDB version: 7.9.2 2024-05-15T12:38:27.128 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Git sha 0 2024-05-15T12:38:27.128 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Compile date 2024-05-14 19:11:07 2024-05-15T12:38:27.128 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: DB SUMMARY 2024-05-15T12:38:27.128 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: DB Session ID: 37BVQ0Z3R9DBFFUYI6RW 2024-05-15T12:38:27.128 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: CURRENT file: CURRENT 2024-05-15T12:38:27.128 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: IDENTITY file: IDENTITY 2024-05-15T12:38:27.128 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-05-15T12:38:27.129 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-05-15T12:38:27.129 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 95183 ; 2024-05-15T12:38:27.129 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.error_if_exists: 0 2024-05-15T12:38:27.129 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.create_if_missing: 0 2024-05-15T12:38:27.129 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.paranoid_checks: 1 2024-05-15T12:38:27.129 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-15T12:38:27.129 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-15T12:38:27.129 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-15T12:38:27.129 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.env: 0x560323ddab20 2024-05-15T12:38:27.129 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.fs: PosixFileSystem 2024-05-15T12:38:27.129 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.info_log: 0x560325ffd900 2024-05-15T12:38:27.129 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_file_opening_threads: 16 2024-05-15T12:38:27.129 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.statistics: (nil) 2024-05-15T12:38:27.130 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.use_fsync: 0 2024-05-15T12:38:27.130 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_log_file_size: 0 2024-05-15T12:38:27.130 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-15T12:38:27.130 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-15T12:38:27.130 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.keep_log_file_num: 1000 2024-05-15T12:38:27.130 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.recycle_log_file_num: 0 2024-05-15T12:38:27.130 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.allow_fallocate: 1 2024-05-15T12:38:27.130 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.allow_mmap_reads: 0 2024-05-15T12:38:27.130 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.allow_mmap_writes: 0 2024-05-15T12:38:27.130 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.use_direct_reads: 0 2024-05-15T12:38:27.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-15T12:38:27.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.create_missing_column_families: 0 2024-05-15T12:38:27.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.db_log_dir: 2024-05-15T12:38:27.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.wal_dir: 2024-05-15T12:38:27.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-15T12:38:27.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-15T12:38:27.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-15T12:38:27.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-15T12:38:27.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-15T12:38:27.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-15T12:38:27.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.advise_random_on_open: 1 2024-05-15T12:38:27.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.db_write_buffer_size: 0 2024-05-15T12:38:27.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.write_buffer_manager: 0x560325fd1400 2024-05-15T12:38:27.132 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-15T12:38:27.132 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-15T12:38:27.132 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-15T12:38:27.132 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.rate_limiter: (nil) 2024-05-15T12:38:27.132 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-15T12:38:27.132 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.wal_recovery_mode: 2 2024-05-15T12:38:27.132 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.enable_thread_tracking: 0 2024-05-15T12:38:27.132 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.enable_pipelined_write: 0 2024-05-15T12:38:27.132 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.unordered_write: 0 2024-05-15T12:38:27.132 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-15T12:38:27.132 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-15T12:38:27.132 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-15T12:38:27.132 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-15T12:38:27.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.row_cache: None 2024-05-15T12:38:27.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.wal_filter: None 2024-05-15T12:38:27.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-15T12:38:27.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.allow_ingest_behind: 0 2024-05-15T12:38:27.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.two_write_queues: 0 2024-05-15T12:38:27.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.manual_wal_flush: 0 2024-05-15T12:38:27.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.wal_compression: 0 2024-05-15T12:38:27.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.atomic_flush: 0 2024-05-15T12:38:27.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-15T12:38:27.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-15T12:38:27.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-15T12:38:27.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.log_readahead_size: 0 2024-05-15T12:38:27.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-15T12:38:27.134 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.best_efforts_recovery: 0 2024-05-15T12:38:27.134 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-15T12:38:27.134 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-15T12:38:27.134 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.allow_data_in_errors: 0 2024-05-15T12:38:27.134 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.db_host_id: __hostname__ 2024-05-15T12:38:27.134 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-15T12:38:27.135 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_background_jobs: 2 2024-05-15T12:38:27.135 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_background_compactions: -1 2024-05-15T12:38:27.135 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_subcompactions: 1 2024-05-15T12:38:27.135 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-15T12:38:27.136 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-15T12:38:27.136 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-15T12:38:27.136 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_total_wal_size: 0 2024-05-15T12:38:27.136 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-15T12:38:27.136 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-15T12:38:27.136 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-15T12:38:27.136 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-15T12:38:27.136 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_open_files: -1 2024-05-15T12:38:27.136 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.bytes_per_sync: 0 2024-05-15T12:38:27.136 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-15T12:38:27.136 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-15T12:38:27.136 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.compaction_readahead_size: 0 2024-05-15T12:38:27.136 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_background_flushes: -1 2024-05-15T12:38:27.136 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Compression algorithms supported: 2024-05-15T12:38:27.137 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-15T12:38:27.137 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: kZSTD supported: 0 2024-05-15T12:38:27.137 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: kXpressCompression supported: 0 2024-05-15T12:38:27.137 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: kLZ4HCCompression supported: 1 2024-05-15T12:38:27.137 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: kZlibCompression supported: 1 2024-05-15T12:38:27.137 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: kSnappyCompression supported: 1 2024-05-15T12:38:27.137 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: kLZ4Compression supported: 1 2024-05-15T12:38:27.137 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: kBZip2Compression supported: 0 2024-05-15T12:38:27.137 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-15T12:38:27.137 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-15T12:38:27.137 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-05-15T12:38:27.137 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-15T12:38:27.137 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-15T12:38:27.137 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.merge_operator: 2024-05-15T12:38:27.138 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.compaction_filter: None 2024-05-15T12:38:27.138 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.compaction_filter_factory: None 2024-05-15T12:38:27.138 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.sst_partitioner_factory: None 2024-05-15T12:38:27.138 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-15T12:38:27.138 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-15T12:38:27.138 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560325ffda00) 2024-05-15T12:38:27.138 INFO:journalctl@ceph.mon.a.smithi027.stdout: cache_index_and_filter_blocks: 1 2024-05-15T12:38:27.138 INFO:journalctl@ceph.mon.a.smithi027.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-15T12:38:27.138 INFO:journalctl@ceph.mon.a.smithi027.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-15T12:38:27.138 INFO:journalctl@ceph.mon.a.smithi027.stdout: pin_top_level_index_and_filter: 1 2024-05-15T12:38:27.138 INFO:journalctl@ceph.mon.a.smithi027.stdout: index_type: 0 2024-05-15T12:38:27.138 INFO:journalctl@ceph.mon.a.smithi027.stdout: data_block_index_type: 0 2024-05-15T12:38:27.138 INFO:journalctl@ceph.mon.a.smithi027.stdout: index_shortening: 1 2024-05-15T12:38:27.138 INFO:journalctl@ceph.mon.a.smithi027.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-15T12:38:27.139 INFO:journalctl@ceph.mon.a.smithi027.stdout: checksum: 4 2024-05-15T12:38:27.139 INFO:journalctl@ceph.mon.a.smithi027.stdout: no_block_cache: 0 2024-05-15T12:38:27.139 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache: 0x560326003090 2024-05-15T12:38:27.139 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache_name: BinnedLRUCache 2024-05-15T12:38:27.139 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache_options: 2024-05-15T12:38:27.139 INFO:journalctl@ceph.mon.a.smithi027.stdout: capacity : 536870912 2024-05-15T12:38:27.139 INFO:journalctl@ceph.mon.a.smithi027.stdout: num_shard_bits : 4 2024-05-15T12:38:27.139 INFO:journalctl@ceph.mon.a.smithi027.stdout: strict_capacity_limit : 0 2024-05-15T12:38:27.139 INFO:journalctl@ceph.mon.a.smithi027.stdout: high_pri_pool_ratio: 0.000 2024-05-15T12:38:27.139 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache_compressed: (nil) 2024-05-15T12:38:27.139 INFO:journalctl@ceph.mon.a.smithi027.stdout: persistent_cache: (nil) 2024-05-15T12:38:27.139 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_size: 4096 2024-05-15T12:38:27.139 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_size_deviation: 10 2024-05-15T12:38:27.139 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_restart_interval: 16 2024-05-15T12:38:27.140 INFO:journalctl@ceph.mon.a.smithi027.stdout: index_block_restart_interval: 1 2024-05-15T12:38:27.140 INFO:journalctl@ceph.mon.a.smithi027.stdout: metadata_block_size: 4096 2024-05-15T12:38:27.140 INFO:journalctl@ceph.mon.a.smithi027.stdout: partition_filters: 0 2024-05-15T12:38:27.140 INFO:journalctl@ceph.mon.a.smithi027.stdout: use_delta_encoding: 1 2024-05-15T12:38:27.140 INFO:journalctl@ceph.mon.a.smithi027.stdout: filter_policy: bloomfilter 2024-05-15T12:38:27.140 INFO:journalctl@ceph.mon.a.smithi027.stdout: whole_key_filtering: 1 2024-05-15T12:38:27.140 INFO:journalctl@ceph.mon.a.smithi027.stdout: verify_compression: 0 2024-05-15T12:38:27.140 INFO:journalctl@ceph.mon.a.smithi027.stdout: read_amp_bytes_per_bit: 0 2024-05-15T12:38:27.140 INFO:journalctl@ceph.mon.a.smithi027.stdout: format_version: 5 2024-05-15T12:38:27.140 INFO:journalctl@ceph.mon.a.smithi027.stdout: enable_index_compression: 1 2024-05-15T12:38:27.140 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_align: 0 2024-05-15T12:38:27.140 INFO:journalctl@ceph.mon.a.smithi027.stdout: max_auto_readahead_size: 262144 2024-05-15T12:38:27.140 INFO:journalctl@ceph.mon.a.smithi027.stdout: prepopulate_block_cache: 0 2024-05-15T12:38:27.141 INFO:journalctl@ceph.mon.a.smithi027.stdout: initial_auto_readahead_size: 8192 2024-05-15T12:38:27.141 INFO:journalctl@ceph.mon.a.smithi027.stdout: num_file_reads_for_auto_readahead: 2 2024-05-15T12:38:27.141 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.write_buffer_size: 33554432 2024-05-15T12:38:27.141 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_write_buffer_number: 2 2024-05-15T12:38:27.141 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.compression: NoCompression 2024-05-15T12:38:27.141 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.bottommost_compression: Disabled 2024-05-15T12:38:27.141 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.prefix_extractor: nullptr 2024-05-15T12:38:27.141 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-15T12:38:27.141 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.num_levels: 7 2024-05-15T12:38:27.141 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-15T12:38:27.141 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-15T12:38:27.141 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-15T12:38:27.141 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-15T12:38:27.141 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-15T12:38:27.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-15T12:38:27.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-15T12:38:27.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-15T12:38:27.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-15T12:38:27.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-15T12:38:27.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-15T12:38:27.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-15T12:38:27.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-15T12:38:27.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.compression_opts.level: 32767 2024-05-15T12:38:27.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.compression_opts.strategy: 0 2024-05-15T12:38:27.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-15T12:38:27.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-15T12:38:27.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-15T12:38:27.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-15T12:38:27.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.compression_opts.enabled: false 2024-05-15T12:38:27.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-15T12:38:27.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-15T12:38:27.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-15T12:38:27.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-15T12:38:27.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.target_file_size_base: 67108864 2024-05-15T12:38:27.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-15T12:38:27.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-15T12:38:27.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-15T12:38:27.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-15T12:38:27.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-15T12:38:27.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-15T12:38:27.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-15T12:38:27.144 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-15T12:38:27.144 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-15T12:38:27.144 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-15T12:38:27.144 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-15T12:38:27.144 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-15T12:38:27.144 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-15T12:38:27.144 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-15T12:38:27.144 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.arena_block_size: 1048576 2024-05-15T12:38:27.144 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-15T12:38:27.144 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-15T12:38:27.144 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.disable_auto_compactions: 0 2024-05-15T12:38:27.144 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-15T12:38:27.144 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-15T12:38:27.144 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-15T12:38:27.144 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-15T12:38:27.145 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-15T12:38:27.145 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-15T12:38:27.145 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-15T12:38:27.145 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-15T12:38:27.145 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-15T12:38:27.145 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-15T12:38:27.145 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.table_properties_collectors: 2024-05-15T12:38:27.145 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.inplace_update_support: 0 2024-05-15T12:38:27.145 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-15T12:38:27.145 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-15T12:38:27.145 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-15T12:38:27.145 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-15T12:38:27.145 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.bloom_locality: 0 2024-05-15T12:38:27.145 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.max_successive_merges: 0 2024-05-15T12:38:27.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-15T12:38:27.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.paranoid_file_checks: 0 2024-05-15T12:38:27.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.force_consistency_checks: 1 2024-05-15T12:38:27.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.report_bg_io_stats: 0 2024-05-15T12:38:27.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.ttl: 2592000 2024-05-15T12:38:27.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-15T12:38:27.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-15T12:38:27.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-15T12:38:27.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.enable_blob_files: false 2024-05-15T12:38:27.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.min_blob_size: 0 2024-05-15T12:38:27.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.blob_file_size: 268435456 2024-05-15T12:38:27.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-15T12:38:27.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-15T12:38:27.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-15T12:38:27.147 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-15T12:38:27.147 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-15T12:38:27.147 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.blob_file_starting_level: 0 2024-05-15T12:38:27.147 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-15T12:38:27.147 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-05-15T12:38:27.147 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-05-15T12:38:27.147 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 121da0da-0b34-43a3-9995-75fc201232d6 2024-05-15T12:38:27.147 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715776706841260, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-05-15T12:38:27.147 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-05-15T12:38:27.147 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715776706842327, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 90657, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 294, "table_properties": {"data_size": 88743, "index_size": 260, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 12730, "raw_average_key_size": 50, "raw_value_size": 81811, "raw_average_value_size": 323, "num_data_blocks": 11, "num_entries": 253, "num_filter_entries": 253, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715776706, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "121da0da-0b34-43a3-9995-75fc201232d6", "db_session_id": "37BVQ0Z3R9DBFFUYI6RW", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-05-15T12:38:27.147 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715776706842415, "job": 1, "event": "recovery_finished"} 2024-05-15T12:38:27.147 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-05-15T12:38:27.147 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-15T12:38:27.147 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5603260f6000 2024-05-15T12:38:27.148 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: DB pointer 0x5603260e0000 2024-05-15T12:38:27.148 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-15T12:38:27.148 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-15T12:38:27.148 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** DB Stats ** 2024-05-15T12:38:27.148 INFO:journalctl@ceph.mon.a.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-15T12:38:27.148 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-15T12:38:27.148 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-15T12:38:27.148 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-15T12:38:27.148 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-15T12:38:27.148 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-15T12:38:27.148 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-15T12:38:27.148 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-05-15T12:38:27.148 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** Compaction Stats [default] ** 2024-05-15T12:38:27.149 INFO:journalctl@ceph.mon.a.smithi027.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-15T12:38:27.149 INFO:journalctl@ceph.mon.a.smithi027.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-15T12:38:27.149 INFO:journalctl@ceph.mon.a.smithi027.stdout: L0 2/0 90.37 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-15T12:38:27.149 INFO:journalctl@ceph.mon.a.smithi027.stdout: Sum 2/0 90.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-15T12:38:27.149 INFO:journalctl@ceph.mon.a.smithi027.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-15T12:38:27.149 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-05-15T12:38:27.149 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** Compaction Stats [default] ** 2024-05-15T12:38:27.149 INFO:journalctl@ceph.mon.a.smithi027.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-15T12:38:27.149 INFO:journalctl@ceph.mon.a.smithi027.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-15T12:38:27.149 INFO:journalctl@ceph.mon.a.smithi027.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 127.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-15T12:38:27.149 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-05-15T12:38:27.149 INFO:journalctl@ceph.mon.a.smithi027.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-15T12:38:27.149 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-05-15T12:38:27.149 INFO:journalctl@ceph.mon.a.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-15T12:38:27.150 INFO:journalctl@ceph.mon.a.smithi027.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-15T12:38:27.150 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-15T12:38:27.150 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-15T12:38:27.150 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-15T12:38:27.150 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-15T12:38:27.150 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative compaction: 0.00 GB write, 26.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-15T12:38:27.150 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval compaction: 0.00 GB write, 26.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-15T12:38:27.150 INFO:journalctl@ceph.mon.a.smithi027.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-15T12:38:27.150 INFO:journalctl@ceph.mon.a.smithi027.stdout: Block cache BinnedLRUCache@0x560326003090#7 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-05-15T12:38:27.150 INFO:journalctl@ceph.mon.a.smithi027.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-05-15T12:38:27.150 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-05-15T12:38:27.150 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-15T12:38:27.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: starting mon.a rank 0 at public addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] at bind addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid deaea04e-12b7-11ef-bc98-c7b262605968 2024-05-15T12:38:27.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: mon.a@-1(???) e1 preinit fsid deaea04e-12b7-11ef-bc98-c7b262605968 2024-05-15T12:38:27.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: mon.a@-1(???).mds e1 new map 2024-05-15T12:38:27.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: mon.a@-1(???).mds e1 print_map 2024-05-15T12:38:27.151 INFO:journalctl@ceph.mon.a.smithi027.stdout: e1 2024-05-15T12:38:27.151 INFO:journalctl@ceph.mon.a.smithi027.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-15T12:38:27.151 INFO:journalctl@ceph.mon.a.smithi027.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-05-15T12:38:27.151 INFO:journalctl@ceph.mon.a.smithi027.stdout: legacy client fscid: -1 2024-05-15T12:38:27.151 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-05-15T12:38:27.151 INFO:journalctl@ceph.mon.a.smithi027.stdout: No filesystems configured 2024-05-15T12:38:27.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-15T12:38:27.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-15T12:38:27.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-15T12:38:27.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-15T12:38:27.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-15T12:38:27.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 bash[39757]: 8954bca5df214f6c0e146e96a5b89474e1061cf548244217a70d7566ba20dd38 2024-05-15T12:38:27.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 systemd[1]: Started Ceph mon.a for deaea04e-12b7-11ef-bc98-c7b262605968. 2024-05-15T12:38:27.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-15T12:38:27.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: monmap e1: 1 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-15T12:38:27.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: fsmap 2024-05-15T12:38:27.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: osdmap e1: 0 total, 0 up, 0 in 2024-05-15T12:38:27.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:26 smithi027 ceph-mon[39795]: mgrmap e1: no daemons active 2024-05-15T12:38:28.124 INFO:teuthology.orchestra.run.smithi027.stdout:Wrote config to /etc/ceph/ceph.conf 2024-05-15T12:38:28.146 INFO:teuthology.orchestra.run.smithi027.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-15T12:38:28.146 INFO:teuthology.orchestra.run.smithi027.stdout:Creating mgr... 2024-05-15T12:38:28.147 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying port 0.0.0.0:9283 ... 2024-05-15T12:38:28.147 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying port 0.0.0.0:8765 ... 2024-05-15T12:38:28.357 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mgr.y 2024-05-15T12:38:28.357 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to reset failed state of unit ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mgr.y.service: Unit ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mgr.y.service not loaded. 2024-05-15T12:38:28.452 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-deaea04e-12b7-11ef-bc98-c7b262605968.target.wants/ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-deaea04e-12b7-11ef-bc98-c7b262605968@.service. 2024-05-15T12:38:28.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:28 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3792568761' entity='client.admin' 2024-05-15T12:38:28.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:28 smithi027 systemd[1]: Starting Ceph mgr.y for deaea04e-12b7-11ef-bc98-c7b262605968... 2024-05-15T12:38:29.041 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-15T12:38:29.041 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-05-15T12:38:29.049 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-15T12:38:29.049 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-05-15T12:38:29.049 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-05-15T12:38:29.049 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to enable service . firewalld.service is not available 2024-05-15T12:38:29.057 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-15T12:38:29.058 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-05-15T12:38:29.066 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-15T12:38:29.066 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-05-15T12:38:29.066 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-05-15T12:38:29.066 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-05-15T12:38:29.067 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr to start... 2024-05-15T12:38:29.067 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr... 2024-05-15T12:38:29.294 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:29 smithi027 bash[40026]: f0a3d1a755e66880f4bada2b93157b336045b47837815bf103a7884662d5b637 2024-05-15T12:38:29.295 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:29 smithi027 systemd[1]: Started Ceph mgr.y for deaea04e-12b7-11ef-bc98-c7b262605968. 2024-05-15T12:38:29.295 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:29 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:29.130+0000 7f94e0f6b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-15T12:38:29.662 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:29 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:29.318+0000 7f94e0f6b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-15T12:38:30.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:29 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/357715080' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-15T12:38:30.020 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:29 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:29.672+0000 7f94e0f6b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-15T12:38:30.463 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:30 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:30.220+0000 7f94e0f6b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-15T12:38:30.769 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:30 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:30.474+0000 7f94e0f6b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-15T12:38:31.269 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:30 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:30.934+0000 7f94e0f6b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-15T12:38:32.020 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:31 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:31.696+0000 7f94e0f6b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-15T12:38:32.020 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:31 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:31.787+0000 7f94e0f6b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-15T12:38:32.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:32 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:32.423+0000 7f94e0f6b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-15T12:38:32.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:32 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:32.514+0000 7f94e0f6b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-15T12:38:32.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:32 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:32.607+0000 7f94e0f6b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-15T12:38:32.793 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-05-15T12:38:32.793 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-05-15T12:38:32.793 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "deaea04e-12b7-11ef-bc98-c7b262605968", 2024-05-15T12:38:32.793 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-05-15T12:38:32.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-15T12:38:32.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-15T12:38:32.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-15T12:38:32.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:38:32.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-15T12:38:32.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-15T12:38:32.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-05-15T12:38:32.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-05-15T12:38:32.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-15T12:38:32.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-05-15T12:38:32.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-05-15T12:38:32.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-05-15T12:38:32.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-15T12:38:32.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T12:38:32.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-15T12:38:32.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-15T12:38:32.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:38:32.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-15T12:38:32.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T12:38:32.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-15T12:38:32.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-15T12:38:32.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-15T12:38:32.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-15T12:38:32.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-15T12:38:32.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-15T12:38:32.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:38:32.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-15T12:38:32.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-15T12:38:32.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-15T12:38:32.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-15T12:38:32.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-15T12:38:32.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-15T12:38:32.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-15T12:38:32.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-15T12:38:32.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-15T12:38:32.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:38:32.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-15T12:38:32.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T12:38:32.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-15T12:38:32.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-15T12:38:32.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:38:32.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-15T12:38:32.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-15T12:38:32.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-15T12:38:32.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-15T12:38:32.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-15T12:38:32.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-15T12:38:32.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-05-15T12:38:32.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-05-15T12:38:32.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-15T12:38:32.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:38:32.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-15T12:38:32.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T12:38:32.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-05-15T12:38:21.245019+0000", 2024-05-15T12:38:32.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-15T12:38:32.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:38:32.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-15T12:38:32.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-05-15T12:38:32.801 INFO:teuthology.orchestra.run.smithi027.stdout:mgr not available, waiting (1/15)... 2024-05-15T12:38:33.049 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:32 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:32.860+0000 7f94e0f6b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-15T12:38:33.049 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:32 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:32.959+0000 7f94e0f6b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-15T12:38:33.049 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:33 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:33.059+0000 7f94e0f6b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-15T12:38:33.303 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:33 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:33.313+0000 7f94e0f6b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-15T12:38:33.598 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:33 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:33.414+0000 7f94e0f6b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-15T12:38:33.598 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:33 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:33.608+0000 7f94e0f6b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-15T12:38:34.270 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:33 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:33.871+0000 7f94e0f6b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-15T12:38:34.769 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:34 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:34.489+0000 7f94e0f6b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-15T12:38:34.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:34 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:34.708+0000 7f94e0f6b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-15T12:38:35.270 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:34 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:34.922+0000 7f94e0f6b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:38:35.996 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:35 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:35.612+0000 7f94e0f6b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-15T12:38:35.997 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:35 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:35.817+0000 7f94e0f6b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:38:36.270 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:36 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:36.007+0000 7f94e0f6b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-15T12:38:36.270 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:36 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:36.127+0000 7f94e0f6b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-15T12:38:36.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:36 smithi027 ceph-mon[39795]: Activating manager daemon y 2024-05-15T12:38:36.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:36 smithi027 ceph-mon[39795]: mgrmap e2: y(active, starting, since 0.00328774s) 2024-05-15T12:38:36.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:36 smithi027 ceph-mon[39795]: from='mgr.14100 172.21.15.27:0/3876576893' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-15T12:38:36.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:36 smithi027 ceph-mon[39795]: from='mgr.14100 172.21.15.27:0/3876576893' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-15T12:38:36.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:36 smithi027 ceph-mon[39795]: from='mgr.14100 172.21.15.27:0/3876576893' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-15T12:38:36.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:36 smithi027 ceph-mon[39795]: from='mgr.14100 172.21.15.27:0/3876576893' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:38:36.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:36 smithi027 ceph-mon[39795]: from='mgr.14100 172.21.15.27:0/3876576893' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-15T12:38:36.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:36 smithi027 ceph-mon[39795]: Manager daemon y is now available 2024-05-15T12:38:36.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:36 smithi027 ceph-mon[39795]: from='mgr.14100 172.21.15.27:0/3876576893' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-15T12:38:36.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:36 smithi027 ceph-mon[39795]: from='mgr.14100 172.21.15.27:0/3876576893' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-15T12:38:36.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:36 smithi027 ceph-mon[39795]: from='mgr.14100 172.21.15.27:0/3876576893' entity='mgr.y' 2024-05-15T12:38:36.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:36 smithi027 ceph-mon[39795]: from='mgr.14100 172.21.15.27:0/3876576893' entity='mgr.y' 2024-05-15T12:38:36.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:36 smithi027 ceph-mon[39795]: from='mgr.14100 172.21.15.27:0/3876576893' entity='mgr.y' 2024-05-15T12:38:38.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:38 smithi027 ceph-mon[39795]: mgrmap e3: y(active, since 1.00721s) 2024-05-15T12:38:39.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:39 smithi027 ceph-mon[39795]: mgrmap e4: y(active, since 2s) 2024-05-15T12:38:39.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:39 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3150961394' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-15T12:38:41.439 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-05-15T12:38:41.441 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-05-15T12:38:41.442 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "deaea04e-12b7-11ef-bc98-c7b262605968", 2024-05-15T12:38:41.442 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-05-15T12:38:41.442 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-15T12:38:41.442 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-15T12:38:41.442 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-15T12:38:41.442 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:38:41.442 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-15T12:38:41.442 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-15T12:38:41.442 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-05-15T12:38:41.442 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-05-15T12:38:41.442 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-15T12:38:41.442 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-05-15T12:38:41.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-05-15T12:38:41.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-05-15T12:38:41.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-15T12:38:41.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T12:38:41.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-15T12:38:41.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-15T12:38:41.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:38:41.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-15T12:38:41.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T12:38:41.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-15T12:38:41.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-15T12:38:41.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-15T12:38:41.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-15T12:38:41.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-15T12:38:41.443 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-15T12:38:41.447 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:38:41.447 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-15T12:38:41.447 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-15T12:38:41.447 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-15T12:38:41.447 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-15T12:38:41.447 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-15T12:38:41.447 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-15T12:38:41.447 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-15T12:38:41.447 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-15T12:38:41.448 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-15T12:38:41.448 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:38:41.448 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-15T12:38:41.448 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T12:38:41.448 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-15T12:38:41.448 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-15T12:38:41.448 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:38:41.448 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-15T12:38:41.448 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-15T12:38:41.448 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-15T12:38:41.448 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-15T12:38:41.448 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-15T12:38:41.448 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-15T12:38:41.448 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-05-15T12:38:41.448 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-05-15T12:38:41.449 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-15T12:38:41.449 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:38:41.449 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-15T12:38:41.449 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T12:38:41.449 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-05-15T12:38:21.245019+0000", 2024-05-15T12:38:41.449 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-15T12:38:41.449 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:38:41.449 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-15T12:38:41.449 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-05-15T12:38:41.449 INFO:teuthology.orchestra.run.smithi027.stdout:mgr is available 2024-05-15T12:38:42.658 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:42 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3867926344' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-15T12:38:42.930 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-05-15T12:38:42.930 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [global] 2024-05-15T12:38:42.930 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout fsid = deaea04e-12b7-11ef-bc98-c7b262605968 2024-05-15T12:38:42.930 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-15T12:38:42.930 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-15T12:38:42.931 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-15T12:38:42.931 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ms_tcp_read_timeout = 5 2024-05-15T12:38:42.931 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-15T12:38:42.931 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-05-15T12:38:42.931 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-15T12:38:42.931 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-15T12:38:42.931 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-05-15T12:38:42.931 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [osd] 2024-05-15T12:38:42.931 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-15T12:38:42.931 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-15T12:38:42.931 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling cephadm module... 2024-05-15T12:38:44.145 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:43 smithi027 ceph-mon[39795]: from='client.14114 172.21.15.27:0/971326797' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-15T12:38:44.145 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:43 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: ignoring --setuser ceph since I am not root 2024-05-15T12:38:44.145 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:43 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: ignoring --setgroup ceph since I am not root 2024-05-15T12:38:44.145 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:43 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:43.971+0000 7fa9a8327200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-15T12:38:44.484 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:44 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:44.156+0000 7fa9a8327200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-15T12:38:44.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:44 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:44.495+0000 7fa9a8327200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-15T12:38:45.270 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:45 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:45.029+0000 7fa9a8327200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-15T12:38:45.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:44 smithi027 ceph-mon[39795]: from='client.14114 172.21.15.27:0/971326797' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-15T12:38:45.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:44 smithi027 ceph-mon[39795]: mgrmap e5: y(active, since 7s) 2024-05-15T12:38:45.588 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:45 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:45.300+0000 7fa9a8327200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-15T12:38:46.013 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:45 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1974390506' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-15T12:38:46.013 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:45 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:45.775+0000 7fa9a8327200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-15T12:38:46.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-05-15T12:38:46.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-05-15T12:38:46.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-15T12:38:46.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-05-15T12:38:46.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-15T12:38:46.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-05-15T12:38:46.409 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for the mgr to restart... 2024-05-15T12:38:46.409 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr epoch 5... 2024-05-15T12:38:46.683 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:46 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:46.535+0000 7fa9a8327200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-15T12:38:46.683 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:46 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:46.627+0000 7fa9a8327200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-15T12:38:47.520 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:47 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:47.266+0000 7fa9a8327200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-15T12:38:47.520 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:47 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:47.355+0000 7fa9a8327200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-15T12:38:47.520 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:47 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:47.448+0000 7fa9a8327200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-15T12:38:48.020 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:47 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:47.685+0000 7fa9a8327200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-15T12:38:48.020 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:47 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:47.781+0000 7fa9a8327200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-15T12:38:48.020 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:47 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:47.877+0000 7fa9a8327200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-15T12:38:48.402 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:48 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:48.119+0000 7fa9a8327200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-15T12:38:48.402 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:48 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:48.219+0000 7fa9a8327200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-15T12:38:48.402 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:48 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:48.412+0000 7fa9a8327200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-15T12:38:49.020 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:48 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:48.672+0000 7fa9a8327200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-15T12:38:49.700 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:49 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:49.284+0000 7fa9a8327200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-15T12:38:49.700 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:49 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:49.505+0000 7fa9a8327200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-15T12:38:49.700 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:49 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:49.709+0000 7fa9a8327200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:38:50.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:50 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:50.400+0000 7fa9a8327200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-15T12:38:50.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:50 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:50.607+0000 7fa9a8327200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:38:51.270 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:50 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:50.796+0000 7fa9a8327200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-15T12:38:51.270 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:50 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:38:50.917+0000 7fa9a8327200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-15T12:38:51.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:50 smithi027 ceph-mon[39795]: Active manager daemon y restarted 2024-05-15T12:38:51.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:50 smithi027 ceph-mon[39795]: Activating manager daemon y 2024-05-15T12:38:51.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:50 smithi027 ceph-mon[39795]: osdmap e2: 0 total, 0 up, 0 in 2024-05-15T12:38:51.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:50 smithi027 ceph-mon[39795]: mgrmap e6: y(active, starting, since 0.00580282s) 2024-05-15T12:38:51.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:50 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:38:51.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:50 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-15T12:38:51.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:50 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-15T12:38:51.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:50 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-15T12:38:51.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:50 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-15T12:38:51.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:50 smithi027 ceph-mon[39795]: Manager daemon y is now available 2024-05-15T12:38:51.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:50 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' 2024-05-15T12:38:52.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:51 smithi027 ceph-mon[39795]: Found migration_current of "None". Setting to last migration. 2024-05-15T12:38:52.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:51 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' 2024-05-15T12:38:52.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:51 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:38:52.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:51 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:38:52.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:51 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:38:52.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:51 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-15T12:38:52.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:51 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-15T12:38:52.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:51 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' 2024-05-15T12:38:52.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:51 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' 2024-05-15T12:38:52.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:51 smithi027 ceph-mon[39795]: mgrmap e7: y(active, since 1.00566s) 2024-05-15T12:38:52.540 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-05-15T12:38:52.540 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-05-15T12:38:52.540 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-15T12:38:52.540 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-05-15T12:38:52.540 INFO:teuthology.orchestra.run.smithi027.stdout:mgr epoch 5 is available 2024-05-15T12:38:52.540 INFO:teuthology.orchestra.run.smithi027.stdout:Setting orchestrator backend to cephadm... 2024-05-15T12:38:53.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:52 smithi027 ceph-mon[39795]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-15T12:38:53.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:52 smithi027 ceph-mon[39795]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-15T12:38:53.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:52 smithi027 ceph-mon[39795]: [15/May/2024:12:38:52] ENGINE Bus STARTING 2024-05-15T12:38:53.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:52 smithi027 ceph-mon[39795]: [15/May/2024:12:38:52] ENGINE Serving on https://172.21.15.27:7150 2024-05-15T12:38:53.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:52 smithi027 ceph-mon[39795]: [15/May/2024:12:38:52] ENGINE Serving on http://172.21.15.27:8765 2024-05-15T12:38:53.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:52 smithi027 ceph-mon[39795]: [15/May/2024:12:38:52] ENGINE Bus STARTED 2024-05-15T12:38:53.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:52 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:38:54.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:53 smithi027 ceph-mon[39795]: mgrmap e8: y(active, since 2s) 2024-05-15T12:38:54.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:53 smithi027 ceph-mon[39795]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:38:54.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:53 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' 2024-05-15T12:38:54.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:53 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:38:55.519 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout value unchanged 2024-05-15T12:38:55.520 INFO:teuthology.orchestra.run.smithi027.stdout:Generating ssh key... 2024-05-15T12:38:56.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:55 smithi027 ceph-mon[39795]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:38:56.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:56 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: Generating public/private rsa key pair. 2024-05-15T12:38:56.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:56 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: Your identification has been saved in /tmp/tmpytt3achw/key. 2024-05-15T12:38:56.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:56 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: Your public key has been saved in /tmp/tmpytt3achw/key.pub. 2024-05-15T12:38:56.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:56 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: The key fingerprint is: 2024-05-15T12:38:56.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:56 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: SHA256:u4TFwL5UQVtcIyNTy6ia0m4b5j54vRiiY6DQdttjO/A ceph-deaea04e-12b7-11ef-bc98-c7b262605968 2024-05-15T12:38:56.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:56 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: The key's randomart image is: 2024-05-15T12:38:56.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:56 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: +---[RSA 3072]----+ 2024-05-15T12:38:56.771 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:56 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: | .=o=.o | 2024-05-15T12:38:56.771 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:56 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: | . O.+ . | 2024-05-15T12:38:56.771 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:56 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: | o + o | 2024-05-15T12:38:56.771 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:56 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: | . = | 2024-05-15T12:38:56.771 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:56 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: | . + S | 2024-05-15T12:38:56.771 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:56 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: |o o.o+ + . | 2024-05-15T12:38:56.771 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:56 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: |+.ooB*o o | 2024-05-15T12:38:56.771 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:56 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: |oo.*=+E. . | 2024-05-15T12:38:56.771 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:56 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: |o. +*+o=. | 2024-05-15T12:38:56.771 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:38:56 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: +----[SHA256]-----+ 2024-05-15T12:38:57.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:57 smithi027 ceph-mon[39795]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:38:57.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:57 smithi027 ceph-mon[39795]: Generating ssh key... 2024-05-15T12:38:57.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:57 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' 2024-05-15T12:38:57.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:57 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' 2024-05-15T12:38:58.709 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+fQOZkoY90pnILValPfSe+lH2ORjb0hUJVZ0U8gcE+w187hqT/l8wuHfUqf2bHvaG7TAYvs3aNEP/B3Bcd+aAjT+BlFROc9Pg7znocmGBAYB5Qw7yG2l3Z31ayG8UkSbgkxGQjNkTSp3BVpuemm8fwTFsxWCP6ethR4yDR7knNu6RzQmqHDzyixDf73/QTSSqzrCDySHXLS4V3DErDG+iARTvxK+FGdRz2waFTcbXwNzwZpc0Tu8SKjTOYEpIOntyPJJCXXKFSj231dAdUXeSgTtg9umgoTNSIehBrpmt81FYjt6jlS2KiJWNVPFhLz38wv4NZUlRcfC3oBQpC8wTnuybPYXFgTVLu+nQdm5856P3xrNi5nNa3Nz3ZeIVlIK8HxYH21QBep53QVsVivaCPYBEw04Hpm28zy1Y6fkica2T7nDgDM7m8vClhJ/L2Ab9alVSyFVXMCCvVfWRKgVrWne6OpATV0dbRuITHEBPlF0+1nV2sZUl2CDUfbrkpJc= ceph-deaea04e-12b7-11ef-bc98-c7b262605968 2024-05-15T12:38:58.709 INFO:teuthology.orchestra.run.smithi027.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-15T12:38:58.709 INFO:teuthology.orchestra.run.smithi027.stdout:Adding key to root@localhost authorized_keys... 2024-05-15T12:38:58.711 INFO:teuthology.orchestra.run.smithi027.stdout:Adding host smithi027... 2024-05-15T12:38:58.990 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:58 smithi027 ceph-mon[39795]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:39:00.078 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:38:59 smithi027 ceph-mon[39795]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi027", "addr": "172.21.15.27", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:39:01.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:00 smithi027 ceph-mon[39795]: Deploying cephadm binary to smithi027 2024-05-15T12:39:04.016 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Added host 'smithi027' with addr '172.21.15.27' 2024-05-15T12:39:04.017 INFO:teuthology.orchestra.run.smithi027.stdout:Deploying unmanaged mon service... 2024-05-15T12:39:04.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:04 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' 2024-05-15T12:39:04.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:04 smithi027 ceph-mon[39795]: Added host smithi027 2024-05-15T12:39:04.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:04 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:39:05.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-05-15T12:39:05.856 INFO:teuthology.orchestra.run.smithi027.stdout:Deploying unmanaged mgr service... 2024-05-15T12:39:06.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:05 smithi027 ceph-mon[39795]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:39:06.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:05 smithi027 ceph-mon[39795]: Saving service mon spec with placement count:5 2024-05-15T12:39:06.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:05 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' 2024-05-15T12:39:07.680 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-05-15T12:39:08.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:07 smithi027 ceph-mon[39795]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:39:08.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:07 smithi027 ceph-mon[39795]: Saving service mgr spec with placement count:2 2024-05-15T12:39:08.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:07 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' 2024-05-15T12:39:08.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:07 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' 2024-05-15T12:39:08.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:07 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:39:08.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:07 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' 2024-05-15T12:39:09.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:09 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1303774121' entity='client.admin' 2024-05-15T12:39:09.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:09 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' 2024-05-15T12:39:10.980 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling the dashboard module... 2024-05-15T12:39:11.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:11 smithi027 ceph-mon[39795]: from='mgr.14116 172.21.15.27:0/1474379764' entity='mgr.y' 2024-05-15T12:39:11.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:11 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1797585596' entity='client.admin' 2024-05-15T12:39:12.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:12 smithi027 ceph-mon[39795]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:39:12.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:12 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/532652713' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-15T12:39:12.520 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:12 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: ignoring --setuser ceph since I am not root 2024-05-15T12:39:12.520 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:12 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: ignoring --setgroup ceph since I am not root 2024-05-15T12:39:12.521 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:12 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:12.440+0000 7f6abd3cf200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-15T12:39:12.961 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:12 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:12.627+0000 7f6abd3cf200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-15T12:39:13.247 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:12 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:12.971+0000 7f6abd3cf200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-15T12:39:13.516 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:13 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:13.492+0000 7f6abd3cf200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-15T12:39:13.516 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:13 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/532652713' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-15T12:39:13.517 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:13 smithi027 ceph-mon[39795]: mgrmap e9: y(active, since 21s) 2024-05-15T12:39:13.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:13 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:13.746+0000 7f6abd3cf200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-15T12:39:14.520 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:14 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:14.210+0000 7f6abd3cf200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-15T12:39:15.190 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:14 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3939109272' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-15T12:39:15.190 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:14 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:14.973+0000 7f6abd3cf200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-15T12:39:15.190 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:15 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:15.064+0000 7f6abd3cf200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-15T12:39:15.485 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-05-15T12:39:15.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-05-15T12:39:15.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-15T12:39:15.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-05-15T12:39:15.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-15T12:39:15.486 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-05-15T12:39:15.486 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for the mgr to restart... 2024-05-15T12:39:15.486 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr epoch 9... 2024-05-15T12:39:15.770 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:15 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:15.714+0000 7f6abd3cf200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-15T12:39:16.127 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:15 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:15.803+0000 7f6abd3cf200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-15T12:39:16.127 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:15 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:15.898+0000 7f6abd3cf200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-15T12:39:16.520 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:16 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:16.137+0000 7f6abd3cf200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-15T12:39:16.520 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:16 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:16.237+0000 7f6abd3cf200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-15T12:39:16.520 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:16 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:16.334+0000 7f6abd3cf200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-15T12:39:16.863 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:16 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:16.578+0000 7f6abd3cf200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-15T12:39:16.864 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:16 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:16.678+0000 7f6abd3cf200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-15T12:39:17.520 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:16 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:16.874+0000 7f6abd3cf200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-15T12:39:17.520 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:17 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:17.134+0000 7f6abd3cf200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-15T12:39:18.020 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:17 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:17.750+0000 7f6abd3cf200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-15T12:39:18.020 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:17 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:17.971+0000 7f6abd3cf200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-15T12:39:18.520 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:18 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:18.178+0000 7f6abd3cf200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:39:19.263 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:18 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:18.871+0000 7f6abd3cf200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-15T12:39:19.263 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:19 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:19.080+0000 7f6abd3cf200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:39:19.513 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:19 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:19.272+0000 7f6abd3cf200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-15T12:39:19.513 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 12:39:19 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T12:39:19.394+0000 7f6abd3cf200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-15T12:39:19.514 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:19 smithi027 ceph-mon[39795]: Active manager daemon y restarted 2024-05-15T12:39:19.514 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:19 smithi027 ceph-mon[39795]: Activating manager daemon y 2024-05-15T12:39:19.514 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:19 smithi027 ceph-mon[39795]: osdmap e3: 0 total, 0 up, 0 in 2024-05-15T12:39:19.514 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:19 smithi027 ceph-mon[39795]: mgrmap e10: y(active, starting, since 0.0034052s) 2024-05-15T12:39:19.514 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:19 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:39:19.514 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:19 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-15T12:39:19.514 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:19 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-15T12:39:19.514 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:19 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-15T12:39:19.514 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:19 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-15T12:39:19.514 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:19 smithi027 ceph-mon[39795]: Manager daemon y is now available 2024-05-15T12:39:19.514 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:19 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:39:19.515 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:19 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:39:20.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:20 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-15T12:39:20.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:20 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-15T12:39:20.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:20 smithi027 ceph-mon[39795]: mgrmap e11: y(active, since 1.00615s) 2024-05-15T12:39:21.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-05-15T12:39:21.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-05-15T12:39:21.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-15T12:39:21.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-05-15T12:39:21.430 INFO:teuthology.orchestra.run.smithi027.stdout:mgr epoch 9 is available 2024-05-15T12:39:21.430 INFO:teuthology.orchestra.run.smithi027.stdout:Generating a dashboard self-signed certificate... 2024-05-15T12:39:22.240 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:21 smithi027 ceph-mon[39795]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-15T12:39:22.240 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:21 smithi027 ceph-mon[39795]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-15T12:39:22.241 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:21 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:22.241 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:21 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:23.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:22 smithi027 ceph-mon[39795]: mgrmap e12: y(active, since 2s) 2024-05-15T12:39:23.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:22 smithi027 ceph-mon[39795]: from='client.14160 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:39:23.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:23.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:23.522 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-05-15T12:39:23.522 INFO:teuthology.orchestra.run.smithi027.stdout:Creating initial admin user... 2024-05-15T12:39:24.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:23 smithi027 ceph-mon[39795]: [15/May/2024:12:39:22] ENGINE Bus STARTING 2024-05-15T12:39:24.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:23 smithi027 ceph-mon[39795]: [15/May/2024:12:39:22] ENGINE Serving on https://172.21.15.27:7150 2024-05-15T12:39:24.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:23 smithi027 ceph-mon[39795]: [15/May/2024:12:39:22] ENGINE Serving on http://172.21.15.27:8765 2024-05-15T12:39:24.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:23 smithi027 ceph-mon[39795]: [15/May/2024:12:39:22] ENGINE Bus STARTED 2024-05-15T12:39:25.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$MlOtucp2OuMhmtMdIU15OO7oRx9SkhTQh6NNXU2eODDKKQJeB8kMu", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1715776764, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-15T12:39:25.182 INFO:teuthology.orchestra.run.smithi027.stdout:Fetching dashboard port number... 2024-05-15T12:39:25.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:25 smithi027 ceph-mon[39795]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:39:25.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:25 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:26.712 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 8443 2024-05-15T12:39:26.714 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:26 smithi027 ceph-mon[39795]: mgrmap e13: y(active, since 6s) 2024-05-15T12:39:26.715 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:26 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1176581782' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-15T12:39:26.723 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-15T12:39:26.723 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-05-15T12:39:26.732 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-15T12:39:26.732 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-05-15T12:39:26.732 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-05-15T12:39:26.732 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-15T12:39:26.734 INFO:teuthology.orchestra.run.smithi027.stdout:Ceph Dashboard is now available at: 2024-05-15T12:39:26.734 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:39:26.734 INFO:teuthology.orchestra.run.smithi027.stdout: URL: https://smithi027.front.sepia.ceph.com:8443/ 2024-05-15T12:39:26.735 INFO:teuthology.orchestra.run.smithi027.stdout: User: admin 2024-05-15T12:39:26.735 INFO:teuthology.orchestra.run.smithi027.stdout: Password: s986o5vlk8 2024-05-15T12:39:26.735 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:39:26.735 INFO:teuthology.orchestra.run.smithi027.stdout:Saving cluster configuration to /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/config directory 2024-05-15T12:39:26.735 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling autotune for osd_memory_target 2024-05-15T12:39:29.752 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-05-15T12:39:29.752 INFO:teuthology.orchestra.run.smithi027.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-05-15T12:39:29.752 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:39:29.752 INFO:teuthology.orchestra.run.smithi027.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-15T12:39:29.752 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:39:29.752 INFO:teuthology.orchestra.run.smithi027.stdout:Or, if you are only running a single cluster on this host: 2024-05-15T12:39:29.753 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:39:29.753 INFO:teuthology.orchestra.run.smithi027.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-05-15T12:39:29.753 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:39:29.753 INFO:teuthology.orchestra.run.smithi027.stdout:Please consider enabling telemetry to help improve Ceph: 2024-05-15T12:39:29.753 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:39:29.753 INFO:teuthology.orchestra.run.smithi027.stdout: ceph telemetry on 2024-05-15T12:39:29.753 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:39:29.753 INFO:teuthology.orchestra.run.smithi027.stdout:For more information see: 2024-05-15T12:39:29.753 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:39:29.753 INFO:teuthology.orchestra.run.smithi027.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-05-15T12:39:29.753 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:39:29.753 INFO:teuthology.orchestra.run.smithi027.stdout:Bootstrap complete. 2024-05-15T12:39:29.803 INFO:tasks.cephadm:Fetching config... 2024-05-15T12:39:29.803 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-15T12:39:29.803 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-15T12:39:29.823 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-15T12:39:29.823 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-15T12:39:29.823 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-15T12:39:29.882 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-15T12:39:29.882 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-15T12:39:29.882 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/keyring of=/dev/stdout 2024-05-15T12:39:29.951 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-15T12:39:29.951 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-15T12:39:29.951 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-15T12:39:30.011 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-15T12:39:30.011 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+fQOZkoY90pnILValPfSe+lH2ORjb0hUJVZ0U8gcE+w187hqT/l8wuHfUqf2bHvaG7TAYvs3aNEP/B3Bcd+aAjT+BlFROc9Pg7znocmGBAYB5Qw7yG2l3Z31ayG8UkSbgkxGQjNkTSp3BVpuemm8fwTFsxWCP6ethR4yDR7knNu6RzQmqHDzyixDf73/QTSSqzrCDySHXLS4V3DErDG+iARTvxK+FGdRz2waFTcbXwNzwZpc0Tu8SKjTOYEpIOntyPJJCXXKFSj231dAdUXeSgTtg9umgoTNSIehBrpmt81FYjt6jlS2KiJWNVPFhLz38wv4NZUlRcfC3oBQpC8wTnuybPYXFgTVLu+nQdm5856P3xrNi5nNa3Nz3ZeIVlIK8HxYH21QBep53QVsVivaCPYBEw04Hpm28zy1Y6fkica2T7nDgDM7m8vClhJ/L2Ab9alVSyFVXMCCvVfWRKgVrWne6OpATV0dbRuITHEBPlF0+1nV2sZUl2CDUfbrkpJc= ceph-deaea04e-12b7-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-15T12:39:30.088 INFO:teuthology.orchestra.run.smithi027.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+fQOZkoY90pnILValPfSe+lH2ORjb0hUJVZ0U8gcE+w187hqT/l8wuHfUqf2bHvaG7TAYvs3aNEP/B3Bcd+aAjT+BlFROc9Pg7znocmGBAYB5Qw7yG2l3Z31ayG8UkSbgkxGQjNkTSp3BVpuemm8fwTFsxWCP6ethR4yDR7knNu6RzQmqHDzyixDf73/QTSSqzrCDySHXLS4V3DErDG+iARTvxK+FGdRz2waFTcbXwNzwZpc0Tu8SKjTOYEpIOntyPJJCXXKFSj231dAdUXeSgTtg9umgoTNSIehBrpmt81FYjt6jlS2KiJWNVPFhLz38wv4NZUlRcfC3oBQpC8wTnuybPYXFgTVLu+nQdm5856P3xrNi5nNa3Nz3ZeIVlIK8HxYH21QBep53QVsVivaCPYBEw04Hpm28zy1Y6fkica2T7nDgDM7m8vClhJ/L2Ab9alVSyFVXMCCvVfWRKgVrWne6OpATV0dbRuITHEBPlF0+1nV2sZUl2CDUfbrkpJc= ceph-deaea04e-12b7-11ef-bc98-c7b262605968 2024-05-15T12:39:30.099 DEBUG:teuthology.orchestra.run.smithi120:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+fQOZkoY90pnILValPfSe+lH2ORjb0hUJVZ0U8gcE+w187hqT/l8wuHfUqf2bHvaG7TAYvs3aNEP/B3Bcd+aAjT+BlFROc9Pg7znocmGBAYB5Qw7yG2l3Z31ayG8UkSbgkxGQjNkTSp3BVpuemm8fwTFsxWCP6ethR4yDR7knNu6RzQmqHDzyixDf73/QTSSqzrCDySHXLS4V3DErDG+iARTvxK+FGdRz2waFTcbXwNzwZpc0Tu8SKjTOYEpIOntyPJJCXXKFSj231dAdUXeSgTtg9umgoTNSIehBrpmt81FYjt6jlS2KiJWNVPFhLz38wv4NZUlRcfC3oBQpC8wTnuybPYXFgTVLu+nQdm5856P3xrNi5nNa3Nz3ZeIVlIK8HxYH21QBep53QVsVivaCPYBEw04Hpm28zy1Y6fkica2T7nDgDM7m8vClhJ/L2Ab9alVSyFVXMCCvVfWRKgVrWne6OpATV0dbRuITHEBPlF0+1nV2sZUl2CDUfbrkpJc= ceph-deaea04e-12b7-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-15T12:39:30.143 INFO:teuthology.orchestra.run.smithi120.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+fQOZkoY90pnILValPfSe+lH2ORjb0hUJVZ0U8gcE+w187hqT/l8wuHfUqf2bHvaG7TAYvs3aNEP/B3Bcd+aAjT+BlFROc9Pg7znocmGBAYB5Qw7yG2l3Z31ayG8UkSbgkxGQjNkTSp3BVpuemm8fwTFsxWCP6ethR4yDR7knNu6RzQmqHDzyixDf73/QTSSqzrCDySHXLS4V3DErDG+iARTvxK+FGdRz2waFTcbXwNzwZpc0Tu8SKjTOYEpIOntyPJJCXXKFSj231dAdUXeSgTtg9umgoTNSIehBrpmt81FYjt6jlS2KiJWNVPFhLz38wv4NZUlRcfC3oBQpC8wTnuybPYXFgTVLu+nQdm5856P3xrNi5nNa3Nz3ZeIVlIK8HxYH21QBep53QVsVivaCPYBEw04Hpm28zy1Y6fkica2T7nDgDM7m8vClhJ/L2Ab9alVSyFVXMCCvVfWRKgVrWne6OpATV0dbRuITHEBPlF0+1nV2sZUl2CDUfbrkpJc= ceph-deaea04e-12b7-11ef-bc98-c7b262605968 2024-05-15T12:39:30.155 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+fQOZkoY90pnILValPfSe+lH2ORjb0hUJVZ0U8gcE+w187hqT/l8wuHfUqf2bHvaG7TAYvs3aNEP/B3Bcd+aAjT+BlFROc9Pg7znocmGBAYB5Qw7yG2l3Z31ayG8UkSbgkxGQjNkTSp3BVpuemm8fwTFsxWCP6ethR4yDR7knNu6RzQmqHDzyixDf73/QTSSqzrCDySHXLS4V3DErDG+iARTvxK+FGdRz2waFTcbXwNzwZpc0Tu8SKjTOYEpIOntyPJJCXXKFSj231dAdUXeSgTtg9umgoTNSIehBrpmt81FYjt6jlS2KiJWNVPFhLz38wv4NZUlRcfC3oBQpC8wTnuybPYXFgTVLu+nQdm5856P3xrNi5nNa3Nz3ZeIVlIK8HxYH21QBep53QVsVivaCPYBEw04Hpm28zy1Y6fkica2T7nDgDM7m8vClhJ/L2Ab9alVSyFVXMCCvVfWRKgVrWne6OpATV0dbRuITHEBPlF0+1nV2sZUl2CDUfbrkpJc= ceph-deaea04e-12b7-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-15T12:39:30.200 INFO:teuthology.orchestra.run.smithi177.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+fQOZkoY90pnILValPfSe+lH2ORjb0hUJVZ0U8gcE+w187hqT/l8wuHfUqf2bHvaG7TAYvs3aNEP/B3Bcd+aAjT+BlFROc9Pg7znocmGBAYB5Qw7yG2l3Z31ayG8UkSbgkxGQjNkTSp3BVpuemm8fwTFsxWCP6ethR4yDR7knNu6RzQmqHDzyixDf73/QTSSqzrCDySHXLS4V3DErDG+iARTvxK+FGdRz2waFTcbXwNzwZpc0Tu8SKjTOYEpIOntyPJJCXXKFSj231dAdUXeSgTtg9umgoTNSIehBrpmt81FYjt6jlS2KiJWNVPFhLz38wv4NZUlRcfC3oBQpC8wTnuybPYXFgTVLu+nQdm5856P3xrNi5nNa3Nz3ZeIVlIK8HxYH21QBep53QVsVivaCPYBEw04Hpm28zy1Y6fkica2T7nDgDM7m8vClhJ/L2Ab9alVSyFVXMCCvVfWRKgVrWne6OpATV0dbRuITHEBPlF0+1nV2sZUl2CDUfbrkpJc= ceph-deaea04e-12b7-11ef-bc98-c7b262605968 2024-05-15T12:39:30.212 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-15T12:39:30.497 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:39:30.519 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:30 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/753302442' entity='client.admin' 2024-05-15T12:39:33.230 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:33 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:33.230 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:33 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:33.230 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:33 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi027", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:39:33.230 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:33 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:33.230 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:33 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:39:33.230 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:33 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:34.781 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-15T12:39:34.782 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-15T12:39:35.058 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:39:35.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:34 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2464798031' entity='client.admin' 2024-05-15T12:39:35.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:34 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:39:35.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:34 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:37.923 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi120 2024-05-15T12:39:37.923 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-05-15T12:39:37.923 DEBUG:teuthology.orchestra.run.smithi120:> dd of=/etc/ceph/ceph.conf 2024-05-15T12:39:37.945 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-05-15T12:39:37.946 DEBUG:teuthology.orchestra.run.smithi120:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-15T12:39:38.006 INFO:tasks.cephadm:Adding host smithi120 to orchestrator... 2024-05-15T12:39:38.006 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph orch host add smithi120 2024-05-15T12:39:38.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:38 smithi027 ceph-mon[39795]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:39:38.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:38 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:38.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:38 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:39:38.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:38 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:39:38.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:38 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:39:38.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:38 smithi027 ceph-mon[39795]: Updating smithi027:/etc/ceph/ceph.conf 2024-05-15T12:39:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:39:39.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:39 smithi027 ceph-mon[39795]: Updating smithi027:/var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:39:39.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:39 smithi027 ceph-mon[39795]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-05-15T12:39:39.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:39 smithi027 ceph-mon[39795]: Updating smithi027:/var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-15T12:39:39.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:39 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:39.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:39 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:39.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:39 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:41.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:41 smithi027 ceph-mon[39795]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:39:41.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:41 smithi027 ceph-mon[39795]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi120", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:39:42.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:42 smithi027 ceph-mon[39795]: Deploying cephadm binary to smithi120 2024-05-15T12:39:43.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:43 smithi027 ceph-mon[39795]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:39:44.066 INFO:teuthology.orchestra.run.smithi027.stdout:Added host 'smithi120' with addr '172.21.15.120' 2024-05-15T12:39:44.760 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph orch host ls --format=json 2024-05-15T12:39:45.035 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:39:45.325 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:45 smithi027 ceph-mon[39795]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:39:45.325 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:45 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:45.325 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:45 smithi027 ceph-mon[39795]: Added host smithi120 2024-05-15T12:39:45.325 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:45 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:39:45.325 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:45 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:45.325 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:45 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:46.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:46 smithi027 ceph-mon[39795]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:39:46.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:46 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:46.927 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:39:46.927 INFO:teuthology.orchestra.run.smithi027.stdout:[{"addr": "172.21.15.27", "hostname": "smithi027", "labels": [], "status": ""}, {"addr": "172.21.15.120", "hostname": "smithi120", "labels": [], "status": ""}] 2024-05-15T12:39:47.577 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi177 2024-05-15T12:39:47.577 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-05-15T12:39:47.577 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/etc/ceph/ceph.conf 2024-05-15T12:39:47.599 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-05-15T12:39:47.599 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-15T12:39:47.660 INFO:tasks.cephadm:Adding host smithi177 to orchestrator... 2024-05-15T12:39:47.660 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph orch host add smithi177 2024-05-15T12:39:47.938 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:39:48.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:47 smithi027 ceph-mon[39795]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:39:48.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:48 smithi027 ceph-mon[39795]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:39:50.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:49 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:50.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:49 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:50.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:49 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:50.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:49 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:50.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:49 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi120", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:39:50.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:49 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:39:50.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:49 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:39:50.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:49 smithi027 ceph-mon[39795]: Updating smithi120:/etc/ceph/ceph.conf 2024-05-15T12:39:50.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:49 smithi027 ceph-mon[39795]: Updating smithi120:/var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:39:50.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:49 smithi027 ceph-mon[39795]: Updating smithi120:/etc/ceph/ceph.client.admin.keyring 2024-05-15T12:39:50.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:49 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:50.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:49 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:50.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:49 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:51.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:50 smithi027 ceph-mon[39795]: Updating smithi120:/var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-15T12:39:51.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:50 smithi027 ceph-mon[39795]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:39:51.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:50 smithi027 ceph-mon[39795]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi177", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:39:52.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:51 smithi027 ceph-mon[39795]: Deploying cephadm binary to smithi177 2024-05-15T12:39:53.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:52 smithi027 ceph-mon[39795]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:39:53.991 INFO:teuthology.orchestra.run.smithi027.stdout:Added host 'smithi177' with addr '172.21.15.177' 2024-05-15T12:39:54.706 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph orch host ls --format=json 2024-05-15T12:39:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:39:55.245 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:55 smithi027 ceph-mon[39795]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:39:55.245 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:55 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:55.245 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:55 smithi027 ceph-mon[39795]: Added host smithi177 2024-05-15T12:39:55.245 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:55 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:39:55.245 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:55 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:55.245 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:55 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:56.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:56 smithi027 ceph-mon[39795]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:39:56.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:56 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:56.921 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:39:56.921 INFO:teuthology.orchestra.run.smithi027.stdout:[{"addr": "172.21.15.27", "hostname": "smithi027", "labels": [], "status": ""}, {"addr": "172.21.15.120", "hostname": "smithi120", "labels": [], "status": ""}, {"addr": "172.21.15.177", "hostname": "smithi177", "labels": [], "status": ""}] 2024-05-15T12:39:57.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:57 smithi027 ceph-mon[39795]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:39:57.567 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-15T12:39:57.567 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd crush tunables default 2024-05-15T12:39:57.840 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:39:58.763 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:58 smithi027 ceph-mon[39795]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:39:59.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:59 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:59.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:59 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:59.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:59 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:59.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:59 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:59.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:59 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:39:59.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:59 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:39:59.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:59 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:39:59.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:59 smithi027 ceph-mon[39795]: Updating smithi177:/etc/ceph/ceph.conf 2024-05-15T12:39:59.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:59 smithi027 ceph-mon[39795]: Updating smithi177:/var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:39:59.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:59 smithi027 ceph-mon[39795]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-05-15T12:39:59.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:59 smithi027 ceph-mon[39795]: Updating smithi177:/var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-15T12:39:59.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:59 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:59.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:59 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:39:59.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:39:59 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:00.540 INFO:teuthology.orchestra.run.smithi027.stderr:adjusted tunables profile to default 2024-05-15T12:40:01.013 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:00 smithi027 ceph-mon[39795]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:01.013 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:00 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/526335356' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-15T12:40:01.343 INFO:tasks.cephadm:Adding mon.a on smithi027 2024-05-15T12:40:01.343 INFO:tasks.cephadm:Adding mon.b on smithi120 2024-05-15T12:40:01.343 INFO:tasks.cephadm:Adding mon.c on smithi177 2024-05-15T12:40:01.343 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph orch apply mon '3;smithi027:172.21.15.27=a;smithi120:172.21.15.120=b;smithi177:172.21.15.177=c' 2024-05-15T12:40:01.643 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:40:01.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:01 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/526335356' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-15T12:40:01.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:01 smithi027 ceph-mon[39795]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T12:40:03.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:02 smithi027 ceph-mon[39795]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:03.468 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled mon update... 2024-05-15T12:40:04.098 DEBUG:teuthology.orchestra.run.smithi120:mon.b> sudo journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mon.b.service 2024-05-15T12:40:04.101 DEBUG:teuthology.orchestra.run.smithi177:mon.c> sudo journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mon.c.service 2024-05-15T12:40:04.103 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-15T12:40:04.104 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph mon dump -f json 2024-05-15T12:40:04.131 INFO:journalctl@ceph.mon.c.smithi177.stdout:-- Logs begin at Wed 2024-05-15 12:27:00 UTC. -- 2024-05-15T12:40:04.137 INFO:journalctl@ceph.mon.b.smithi120.stdout:-- Logs begin at Wed 2024-05-15 12:26:58 UTC. -- 2024-05-15T12:40:04.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:04 smithi027 ceph-mon[39795]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:04.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:04 smithi027 ceph-mon[39795]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi027:172.21.15.27=a;smithi120:172.21.15.120=b;smithi177:172.21.15.177=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:40:04.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:04 smithi027 ceph-mon[39795]: Saving service mon spec with placement smithi027:172.21.15.27=a;smithi120:172.21.15.120=b;smithi177:172.21.15.177=c;count:3 2024-05-15T12:40:04.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:04 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:04.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:04 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:40:04.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:04 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:04.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:04 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:40:04.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:04 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:04.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:04 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:40:04.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:04 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:04.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:04 smithi027 ceph-mon[39795]: Deploying daemon mon.c on smithi177 2024-05-15T12:40:04.864 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:40:06.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:06 smithi027 ceph-mon[39795]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:07.765 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:40:07.765 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":1,"fsid":"deaea04e-12b7-11ef-bc98-c7b262605968","modified":"2024-05-15T12:38:18.186353Z","created":"2024-05-15T12:38:18.186353Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-15T12:40:07.767 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 1 2024-05-15T12:40:08.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:07 smithi027 ceph-mon[39795]: from='client.? 172.21.15.177:0/3290182628' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T12:40:09.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:08 smithi177 bash[44266]: 967c99f8db2cb1bd94f2bbbe27b2ffbe27f76c4b9abf2f2c05f7ed8ae5ff9470 2024-05-15T12:40:09.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:08 smithi177 systemd[1]: Started Ceph mon.c for deaea04e-12b7-11ef-bc98-c7b262605968. 2024-05-15T12:40:09.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:08 smithi027 ceph-mon[39795]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:09.502 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-15T12:40:09.503 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph mon dump -f json 2024-05-15T12:40:09.762 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:40:14.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:40:14.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:40:14.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:14 smithi177 ceph-mon[44303]: mon.a calling monitor election 2024-05-15T12:40:14.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:40:14.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:40:14.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:14 smithi177 ceph-mon[44303]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:14.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:40:14.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:40:14.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:14 smithi177 ceph-mon[44303]: mon.c calling monitor election 2024-05-15T12:40:14.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:14 smithi177 ceph-mon[44303]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:14.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:40:14.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:40:14.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:40:14.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:14 smithi177 ceph-mon[44303]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-15T12:40:14.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:14 smithi177 ceph-mon[44303]: monmap e2: 2 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],c=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-15T12:40:14.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:14 smithi177 ceph-mon[44303]: fsmap 2024-05-15T12:40:14.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:14 smithi177 ceph-mon[44303]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T12:40:14.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:14 smithi177 ceph-mon[44303]: mgrmap e13: y(active, since 54s) 2024-05-15T12:40:14.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:14 smithi177 ceph-mon[44303]: overall HEALTH_OK 2024-05-15T12:40:14.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:14.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:14.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:14.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:40:14.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:14.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:40:14.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:40:14.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:14 smithi027 ceph-mon[39795]: mon.a calling monitor election 2024-05-15T12:40:14.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:40:14.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:40:14.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:14 smithi027 ceph-mon[39795]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:14.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:40:14.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:40:14.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:14 smithi027 ceph-mon[39795]: mon.c calling monitor election 2024-05-15T12:40:14.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:14 smithi027 ceph-mon[39795]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:14.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:40:14.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:40:14.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:40:14.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:14 smithi027 ceph-mon[39795]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-15T12:40:14.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:14 smithi027 ceph-mon[39795]: monmap e2: 2 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],c=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-15T12:40:14.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:14 smithi027 ceph-mon[39795]: fsmap 2024-05-15T12:40:14.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:14 smithi027 ceph-mon[39795]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T12:40:14.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:14 smithi027 ceph-mon[39795]: mgrmap e13: y(active, since 54s) 2024-05-15T12:40:14.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:14 smithi027 ceph-mon[39795]: overall HEALTH_OK 2024-05-15T12:40:14.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:14.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:14.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:14.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:40:14.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:14.528 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:40:14.528 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":2,"fsid":"deaea04e-12b7-11ef-bc98-c7b262605968","modified":"2024-05-15T12:40:08.961850Z","created":"2024-05-15T12:38:18.186353Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-15T12:40:14.530 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 2 2024-05-15T12:40:15.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:15 smithi177 ceph-mon[44303]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:15.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:15 smithi177 ceph-mon[44303]: Deploying daemon mon.b on smithi120 2024-05-15T12:40:15.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:15 smithi177 ceph-mon[44303]: from='client.? 172.21.15.177:0/1275431788' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T12:40:15.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:15 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:40:15.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:15 smithi027 ceph-mon[39795]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:15.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:15 smithi027 ceph-mon[39795]: Deploying daemon mon.b on smithi120 2024-05-15T12:40:15.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:15 smithi027 ceph-mon[39795]: from='client.? 172.21.15.177:0/1275431788' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T12:40:15.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:15 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:40:16.224 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-15T12:40:16.224 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph mon dump -f json 2024-05-15T12:40:16.494 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:40:17.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:17 smithi177 ceph-mon[44303]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:17.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:17 smithi027 ceph-mon[39795]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:18.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:17 smithi120 bash[43923]: d81c257c3dcfa2805226a03a27d67be18397b6374decbfc6e11fbedd8898c3ab 2024-05-15T12:40:18.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:17 smithi120 systemd[1]: Started Ceph mon.b for deaea04e-12b7-11ef-bc98-c7b262605968. 2024-05-15T12:40:23.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:23.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:40:23.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: mon.c calling monitor election 2024-05-15T12:40:23.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: mon.a calling monitor election 2024-05-15T12:40:23.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:40:23.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:40:23.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:40:23.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:23.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:40:23.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:40:23.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:23.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:40:23.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:40:23.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-15T12:40:23.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: monmap e3: 3 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],b=[v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0],c=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-15T12:40:23.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: fsmap 2024-05-15T12:40:23.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T12:40:23.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: mgrmap e13: y(active, since 63s) 2024-05-15T12:40:23.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-05-15T12:40:23.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-05-15T12:40:23.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-05-15T12:40:23.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: mon.b (rank 2) addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] is down (out of quorum) 2024-05-15T12:40:23.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:23.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:23.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:23.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:22 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:23.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:23.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:40:23.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: mon.c calling monitor election 2024-05-15T12:40:23.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: mon.a calling monitor election 2024-05-15T12:40:23.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:40:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:40:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:40:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:40:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:40:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:40:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:40:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-15T12:40:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: monmap e3: 3 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],b=[v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0],c=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-15T12:40:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: fsmap 2024-05-15T12:40:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T12:40:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: mgrmap e13: y(active, since 63s) 2024-05-15T12:40:23.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-05-15T12:40:23.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-05-15T12:40:23.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-05-15T12:40:23.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: mon.b (rank 2) addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] is down (out of quorum) 2024-05-15T12:40:23.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:23.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:23.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:23.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:24.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:40:24.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:40:24.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:40:24.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:40:24.340 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-05-15T12:40:24.341 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":3,"fsid":"deaea04e-12b7-11ef-bc98-c7b262605968","modified":"2024-05-15T12:40:17.814390Z","created":"2024-05-15T12:38:18.186353Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:3300","nonce":0},{"type":"v1","addr":"172.21.15.120:6789","nonce":0}]},"addr":"172.21.15.120:6789/0","public_addr":"172.21.15.120:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-15T12:40:24.342 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 3 2024-05-15T12:40:25.061 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-15T12:40:25.061 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph config generate-minimal-conf 2024-05-15T12:40:25.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:25.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:40:25.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: mon.c calling monitor election 2024-05-15T12:40:25.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: mon.a calling monitor election 2024-05-15T12:40:25.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:40:25.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:40:25.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:40:25.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:25.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:40:25.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:40:25.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:25.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:40:25.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:40:25.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-15T12:40:25.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: monmap e3: 3 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],b=[v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0],c=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-15T12:40:25.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: fsmap 2024-05-15T12:40:25.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T12:40:25.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: mgrmap e13: y(active, since 63s) 2024-05-15T12:40:25.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-05-15T12:40:25.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-05-15T12:40:25.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-05-15T12:40:25.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: mon.b (rank 2) addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] is down (out of quorum) 2024-05-15T12:40:25.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:25.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:25.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:25.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:25.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:40:25.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:40:25.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:25 smithi027 ceph-mon[39795]: mon.b calling monitor election 2024-05-15T12:40:25.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:25 smithi027 ceph-mon[39795]: Updating smithi027:/var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:40:25.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:25 smithi027 ceph-mon[39795]: Updating smithi120:/var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:40:25.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:25 smithi027 ceph-mon[39795]: Updating smithi177:/var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:40:25.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:25 smithi027 ceph-mon[39795]: mon.b calling monitor election 2024-05-15T12:40:25.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:25 smithi027 ceph-mon[39795]: mon.c calling monitor election 2024-05-15T12:40:25.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:25 smithi027 ceph-mon[39795]: mon.a calling monitor election 2024-05-15T12:40:25.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:25 smithi027 ceph-mon[39795]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-15T12:40:25.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:25 smithi027 ceph-mon[39795]: monmap e3: 3 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],b=[v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0],c=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-15T12:40:25.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:25 smithi027 ceph-mon[39795]: fsmap 2024-05-15T12:40:25.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:25 smithi027 ceph-mon[39795]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T12:40:25.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:25 smithi027 ceph-mon[39795]: mgrmap e13: y(active, since 65s) 2024-05-15T12:40:25.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:25 smithi027 ceph-mon[39795]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-05-15T12:40:25.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:25 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T12:40:25.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:25 smithi027 ceph-mon[39795]: overall HEALTH_OK 2024-05-15T12:40:25.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:25 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:25.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:25 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:25.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:25 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:25.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:25 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:25.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:25 smithi027 ceph-mon[39795]: Reconfiguring mon.a (unknown last config time)... 2024-05-15T12:40:25.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:25 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:40:25.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:25 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T12:40:25.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:25 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:25.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:25 smithi027 ceph-mon[39795]: Reconfiguring daemon mon.a on smithi027 2024-05-15T12:40:26.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:25 smithi120 ceph-mon[43961]: mon.b calling monitor election 2024-05-15T12:40:26.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:25 smithi120 ceph-mon[43961]: Updating smithi027:/var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:40:26.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:25 smithi120 ceph-mon[43961]: Updating smithi120:/var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:40:26.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:25 smithi120 ceph-mon[43961]: Updating smithi177:/var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:40:26.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:25 smithi120 ceph-mon[43961]: mon.b calling monitor election 2024-05-15T12:40:26.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:25 smithi120 ceph-mon[43961]: mon.c calling monitor election 2024-05-15T12:40:26.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:25 smithi120 ceph-mon[43961]: mon.a calling monitor election 2024-05-15T12:40:26.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:25 smithi120 ceph-mon[43961]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-15T12:40:26.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:25 smithi120 ceph-mon[43961]: monmap e3: 3 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],b=[v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0],c=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-15T12:40:26.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:25 smithi120 ceph-mon[43961]: fsmap 2024-05-15T12:40:26.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:25 smithi120 ceph-mon[43961]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T12:40:26.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:25 smithi120 ceph-mon[43961]: mgrmap e13: y(active, since 65s) 2024-05-15T12:40:26.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:25 smithi120 ceph-mon[43961]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-05-15T12:40:26.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:25 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T12:40:26.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:25 smithi120 ceph-mon[43961]: overall HEALTH_OK 2024-05-15T12:40:26.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:25 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:26.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:25 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:26.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:25 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:26.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:25 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:26.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:25 smithi120 ceph-mon[43961]: Reconfiguring mon.a (unknown last config time)... 2024-05-15T12:40:26.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:25 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:40:26.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:25 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T12:40:26.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:25 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:26.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:25 smithi120 ceph-mon[43961]: Reconfiguring daemon mon.a on smithi027 2024-05-15T12:40:26.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:25 smithi177 ceph-mon[44303]: mon.b calling monitor election 2024-05-15T12:40:26.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:25 smithi177 ceph-mon[44303]: Updating smithi027:/var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:40:26.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:25 smithi177 ceph-mon[44303]: Updating smithi120:/var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:40:26.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:25 smithi177 ceph-mon[44303]: Updating smithi177:/var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:40:26.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:25 smithi177 ceph-mon[44303]: mon.b calling monitor election 2024-05-15T12:40:26.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:25 smithi177 ceph-mon[44303]: mon.c calling monitor election 2024-05-15T12:40:26.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:25 smithi177 ceph-mon[44303]: mon.a calling monitor election 2024-05-15T12:40:26.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:25 smithi177 ceph-mon[44303]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-15T12:40:26.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:25 smithi177 ceph-mon[44303]: monmap e3: 3 mons at {a=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],b=[v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0],c=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-15T12:40:26.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:25 smithi177 ceph-mon[44303]: fsmap 2024-05-15T12:40:26.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:25 smithi177 ceph-mon[44303]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T12:40:26.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:25 smithi177 ceph-mon[44303]: mgrmap e13: y(active, since 65s) 2024-05-15T12:40:26.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:25 smithi177 ceph-mon[44303]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-05-15T12:40:26.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:25 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T12:40:26.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:25 smithi177 ceph-mon[44303]: overall HEALTH_OK 2024-05-15T12:40:26.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:25 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:26.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:25 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:26.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:25 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:26.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:25 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:26.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:25 smithi177 ceph-mon[44303]: Reconfiguring mon.a (unknown last config time)... 2024-05-15T12:40:26.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:25 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:40:26.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:25 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T12:40:26.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:25 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:26.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:25 smithi177 ceph-mon[44303]: Reconfiguring daemon mon.a on smithi027 2024-05-15T12:40:26.448 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:40:26.992 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:26 smithi027 ceph-mon[39795]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:26.992 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:26 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:40:26.992 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:26 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:26.992 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:26 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:26.993 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:26 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:40:26.993 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:26 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T12:40:26.993 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:26 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:27.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:26 smithi120 ceph-mon[43961]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:27.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:26 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:40:27.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:26 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:27.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:26 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:27.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:26 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:40:27.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:26 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T12:40:27.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:26 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:27.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:26 smithi177 ceph-mon[44303]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:27.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:26 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:40:27.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:26 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:27.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:26 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:27.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:26 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:40:27.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:26 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T12:40:27.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:26 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:28.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:27 smithi027 ceph-mon[39795]: Reconfiguring mon.b (monmap changed)... 2024-05-15T12:40:28.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:27 smithi027 ceph-mon[39795]: Reconfiguring daemon mon.b on smithi120 2024-05-15T12:40:28.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:27 smithi120 ceph-mon[43961]: Reconfiguring mon.b (monmap changed)... 2024-05-15T12:40:28.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:27 smithi120 ceph-mon[43961]: Reconfiguring daemon mon.b on smithi120 2024-05-15T12:40:28.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:27 smithi177 ceph-mon[44303]: Reconfiguring mon.b (monmap changed)... 2024-05-15T12:40:28.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:27 smithi177 ceph-mon[44303]: Reconfiguring daemon mon.b on smithi120 2024-05-15T12:40:28.534 INFO:teuthology.orchestra.run.smithi027.stdout:# minimal ceph.conf for deaea04e-12b7-11ef-bc98-c7b262605968 2024-05-15T12:40:28.534 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2024-05-15T12:40:28.534 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = deaea04e-12b7-11ef-bc98-c7b262605968 2024-05-15T12:40:28.534 INFO:teuthology.orchestra.run.smithi027.stdout: mon_host = [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-05-15T12:40:29.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:28 smithi027 ceph-mon[39795]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:29.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:29.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:29.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:28 smithi027 ceph-mon[39795]: Reconfiguring mon.c (monmap changed)... 2024-05-15T12:40:29.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:40:29.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T12:40:29.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:29.147 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:28 smithi027 ceph-mon[39795]: Reconfiguring daemon mon.c on smithi177 2024-05-15T12:40:29.147 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:28 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4215313729' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:29.174 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-15T12:40:29.175 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-15T12:40:29.175 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.conf 2024-05-15T12:40:29.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:28 smithi177 ceph-mon[44303]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:29.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:29.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:29.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:28 smithi177 ceph-mon[44303]: Reconfiguring mon.c (monmap changed)... 2024-05-15T12:40:29.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:40:29.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T12:40:29.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:29.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:28 smithi177 ceph-mon[44303]: Reconfiguring daemon mon.c on smithi177 2024-05-15T12:40:29.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:28 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4215313729' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:29.207 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-15T12:40:29.207 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-15T12:40:29.275 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-05-15T12:40:29.275 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/ceph/ceph.conf 2024-05-15T12:40:29.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:28 smithi120 ceph-mon[43961]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:29.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:29.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:29.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:28 smithi120 ceph-mon[43961]: Reconfiguring mon.c (monmap changed)... 2024-05-15T12:40:29.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:40:29.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T12:40:29.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:29.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:28 smithi120 ceph-mon[43961]: Reconfiguring daemon mon.c on smithi177 2024-05-15T12:40:29.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:28 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4215313729' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:29.313 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-05-15T12:40:29.313 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-15T12:40:29.386 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-05-15T12:40:29.386 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.conf 2024-05-15T12:40:29.417 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-05-15T12:40:29.417 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-15T12:40:29.484 INFO:tasks.cephadm:Adding mgr.y on smithi027 2024-05-15T12:40:29.484 INFO:tasks.cephadm:Adding mgr.x on smithi120 2024-05-15T12:40:29.484 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph orch apply mgr '2;smithi027=y;smithi120=x' 2024-05-15T12:40:29.776 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:40:30.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:30 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:30.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:30 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:30.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:30 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:40:30.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:30 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:30.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:30 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:40:30.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:30 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:30.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:30 smithi177 ceph-mon[44303]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:30.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:30 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:30.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:30 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:30.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:30 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:40:30.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:30 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:30.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:30 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:40:30.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:30 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:30.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:30 smithi027 ceph-mon[39795]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:30.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:30 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:30.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:30 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:30.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:30 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:40:30.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:30 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:30.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:30 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:40:30.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:30 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:30.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:30 smithi120 ceph-mon[43961]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:31.575 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled mgr update... 2024-05-15T12:40:32.205 DEBUG:teuthology.orchestra.run.smithi120:mgr.x> sudo journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mgr.x.service 2024-05-15T12:40:32.208 INFO:tasks.cephadm:Deploying OSDs... 2024-05-15T12:40:32.208 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-15T12:40:32.208 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2024-05-15T12:40:32.231 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-15T12:40:32.231 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_1 2024-05-15T12:40:32.234 INFO:journalctl@ceph.mgr.x.smithi120.stdout:-- Logs begin at Wed 2024-05-15 12:26:58 UTC. -- 2024-05-15T12:40:32.294 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-15T12:40:32.294 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:40:32.294 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 164804 Links: 1 2024-05-15T12:40:32.294 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:40:32.294 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:40:32.294 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-05-15 12:39:33.016809472 +0000 2024-05-15T12:40:32.294 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-05-15 12:33:33.032318494 +0000 2024-05-15T12:40:32.294 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-05-15 12:33:33.032318494 +0000 2024-05-15T12:40:32.294 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2024-05-15T12:40:32.295 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-15T12:40:32.365 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-05-15T12:40:32.365 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-05-15T12:40:32.366 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000219537 s, 2.3 MB/s 2024-05-15T12:40:32.367 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-15T12:40:32.430 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_2 2024-05-15T12:40:32.491 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-15T12:40:32.491 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:40:32.491 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 168437 Links: 1 2024-05-15T12:40:32.491 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:40:32.491 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:40:32.491 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-05-15 12:39:33.057808439 +0000 2024-05-15T12:40:32.491 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-05-15 12:33:33.398309224 +0000 2024-05-15T12:40:32.491 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-05-15 12:33:33.398309224 +0000 2024-05-15T12:40:32.491 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2024-05-15T12:40:32.492 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-15T12:40:32.560 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-05-15T12:40:32.560 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-05-15T12:40:32.560 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000210664 s, 2.4 MB/s 2024-05-15T12:40:32.561 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-15T12:40:32.623 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_3 2024-05-15T12:40:32.682 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-15T12:40:32.682 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:40:32.682 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 164863 Links: 1 2024-05-15T12:40:32.682 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:40:32.682 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:40:32.682 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-05-15 12:39:33.102807305 +0000 2024-05-15T12:40:32.682 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-05-15 12:33:33.760300056 +0000 2024-05-15T12:40:32.683 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-05-15 12:33:33.760300056 +0000 2024-05-15T12:40:32.683 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2024-05-15T12:40:32.683 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-15T12:40:32.748 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:32 smithi027 ceph-mon[39795]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:32.748 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:32 smithi027 ceph-mon[39795]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi027=y;smithi120=x", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:40:32.748 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:32 smithi027 ceph-mon[39795]: Saving service mgr spec with placement smithi027=y;smithi120=x;count:2 2024-05-15T12:40:32.748 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:32 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:32.748 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:32 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:40:32.749 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:32 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:32.749 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:32 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:40:32.749 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:32 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:32.749 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:32 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-15T12:40:32.749 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:32 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-15T12:40:32.749 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:32 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-15T12:40:32.749 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:32 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:32.749 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:32 smithi027 ceph-mon[39795]: Deploying daemon mgr.x on smithi120 2024-05-15T12:40:32.750 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-05-15T12:40:32.750 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-05-15T12:40:32.750 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.00017601 s, 2.9 MB/s 2024-05-15T12:40:32.751 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-15T12:40:32.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:32 smithi120 ceph-mon[43961]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:32.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:32 smithi120 ceph-mon[43961]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi027=y;smithi120=x", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:40:32.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:32 smithi120 ceph-mon[43961]: Saving service mgr spec with placement smithi027=y;smithi120=x;count:2 2024-05-15T12:40:32.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:32 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:32.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:32 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:40:32.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:32 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:32.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:32 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:40:32.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:32 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:32.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:32 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-15T12:40:32.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:32 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-15T12:40:32.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:32 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-15T12:40:32.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:32 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:32.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:32 smithi120 ceph-mon[43961]: Deploying daemon mgr.x on smithi120 2024-05-15T12:40:32.811 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_4 2024-05-15T12:40:32.870 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-15T12:40:32.870 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:40:32.870 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 170354 Links: 1 2024-05-15T12:40:32.871 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:40:32.871 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:40:32.871 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-05-15 12:39:33.146806196 +0000 2024-05-15T12:40:32.871 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-05-15 12:33:34.121290913 +0000 2024-05-15T12:40:32.871 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-05-15 12:33:34.121290913 +0000 2024-05-15T12:40:32.871 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2024-05-15T12:40:32.871 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-15T12:40:32.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:32 smithi177 ceph-mon[44303]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:32.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:32 smithi177 ceph-mon[44303]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi027=y;smithi120=x", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:40:32.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:32 smithi177 ceph-mon[44303]: Saving service mgr spec with placement smithi027=y;smithi120=x;count:2 2024-05-15T12:40:32.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:32 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:32.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:32 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:40:32.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:32 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:32.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:32 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:40:32.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:32 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:32.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:32 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-15T12:40:32.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:32 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-15T12:40:32.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:32 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-15T12:40:32.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:32 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:32.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:32 smithi177 ceph-mon[44303]: Deploying daemon mgr.x on smithi120 2024-05-15T12:40:32.937 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-05-15T12:40:32.937 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-05-15T12:40:32.938 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000251234 s, 2.0 MB/s 2024-05-15T12:40:32.939 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-15T12:40:32.998 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-05-15T12:40:32.998 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/scratch_devs of=/dev/stdout 2024-05-15T12:40:33.016 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-15T12:40:33.016 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_1 2024-05-15T12:40:33.074 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-15T12:40:33.074 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:40:33.075 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 6h/6d Inode: 166764 Links: 1 2024-05-15T12:40:33.075 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:40:33.075 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:40:33.075 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-05-15 12:40:17.213343391 +0000 2024-05-15T12:40:33.075 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-05-15 12:33:23.226640349 +0000 2024-05-15T12:40:33.075 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-05-15 12:33:23.226640349 +0000 2024-05-15T12:40:33.075 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: - 2024-05-15T12:40:33.075 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-15T12:40:33.141 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-05-15T12:40:33.142 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-05-15T12:40:33.142 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000247309 s, 2.1 MB/s 2024-05-15T12:40:33.143 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-15T12:40:33.202 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_2 2024-05-15T12:40:33.262 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-15T12:40:33.262 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:40:33.262 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 6h/6d Inode: 165789 Links: 1 2024-05-15T12:40:33.262 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:40:33.262 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:40:33.262 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-05-15 12:40:33.266988351 +0000 2024-05-15T12:40:33.262 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-05-15 12:33:23.582632667 +0000 2024-05-15T12:40:33.262 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-05-15 12:33:23.582632667 +0000 2024-05-15T12:40:33.263 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: - 2024-05-15T12:40:33.263 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-15T12:40:33.330 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-05-15T12:40:33.330 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-05-15T12:40:33.330 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.0002364 s, 2.2 MB/s 2024-05-15T12:40:33.332 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-15T12:40:33.391 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_3 2024-05-15T12:40:33.449 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-15T12:40:33.449 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:40:33.449 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 6h/6d Inode: 165814 Links: 1 2024-05-15T12:40:33.450 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:40:33.450 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:40:33.450 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-05-15 12:40:33.398985431 +0000 2024-05-15T12:40:33.450 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-05-15 12:33:23.915625481 +0000 2024-05-15T12:40:33.450 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-05-15 12:33:23.915625481 +0000 2024-05-15T12:40:33.450 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: - 2024-05-15T12:40:33.450 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-15T12:40:33.518 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-05-15T12:40:33.518 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-05-15T12:40:33.518 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000174076 s, 2.9 MB/s 2024-05-15T12:40:33.519 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-15T12:40:33.578 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_4 2024-05-15T12:40:33.636 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-15T12:40:33.636 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:40:33.636 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 6h/6d Inode: 170495 Links: 1 2024-05-15T12:40:33.636 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:40:33.637 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:40:33.637 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-05-15 12:40:33.398985431 +0000 2024-05-15T12:40:33.637 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-05-15 12:33:24.250618252 +0000 2024-05-15T12:40:33.637 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-05-15 12:33:24.250618252 +0000 2024-05-15T12:40:33.637 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: - 2024-05-15T12:40:33.637 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-15T12:40:33.700 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:33 smithi120 systemd[1]: Starting Ceph mgr.x for deaea04e-12b7-11ef-bc98-c7b262605968... 2024-05-15T12:40:33.702 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-05-15T12:40:33.702 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-05-15T12:40:33.702 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000226995 s, 2.3 MB/s 2024-05-15T12:40:33.703 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-15T12:40:33.761 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-05-15T12:40:33.761 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/scratch_devs of=/dev/stdout 2024-05-15T12:40:33.784 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-15T12:40:33.784 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_1 2024-05-15T12:40:33.845 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-15T12:40:33.845 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:40:33.845 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 6h/6d Inode: 167895 Links: 1 2024-05-15T12:40:33.845 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:40:33.845 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:40:33.845 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-05-15 12:40:08.040487973 +0000 2024-05-15T12:40:33.845 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-05-15 12:33:23.248402915 +0000 2024-05-15T12:40:33.846 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-05-15 12:33:23.248402915 +0000 2024-05-15T12:40:33.846 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: - 2024-05-15T12:40:33.846 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-15T12:40:33.916 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-05-15T12:40:33.917 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-05-15T12:40:33.917 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000303233 s, 1.7 MB/s 2024-05-15T12:40:33.918 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-15T12:40:33.979 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_2 2024-05-15T12:40:34.038 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-15T12:40:34.038 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:40:34.038 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 6h/6d Inode: 165610 Links: 1 2024-05-15T12:40:34.038 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:40:34.039 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:40:34.039 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-05-15 12:40:08.041487946 +0000 2024-05-15T12:40:34.039 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-05-15 12:33:23.584393814 +0000 2024-05-15T12:40:34.039 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-05-15 12:33:23.584393814 +0000 2024-05-15T12:40:34.039 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: - 2024-05-15T12:40:34.039 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-15T12:40:34.106 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-05-15T12:40:34.106 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-05-15T12:40:34.106 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000200104 s, 2.6 MB/s 2024-05-15T12:40:34.107 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-15T12:40:34.168 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_3 2024-05-15T12:40:34.227 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-15T12:40:34.227 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:40:34.227 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 6h/6d Inode: 164468 Links: 1 2024-05-15T12:40:34.227 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:40:34.228 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:40:34.228 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-05-15 12:40:08.041487946 +0000 2024-05-15T12:40:34.228 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-05-15 12:33:23.933384361 +0000 2024-05-15T12:40:34.228 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-05-15 12:33:23.933384361 +0000 2024-05-15T12:40:34.228 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: - 2024-05-15T12:40:34.228 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-15T12:40:34.281 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:34 smithi120 ceph-mgr[45637]: -- 172.21.15.120:0/1238263089 <== mon.2 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c15dd9e5a0 con 0x55c15dd73000 2024-05-15T12:40:34.281 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:34 smithi120 bash[45503]: a678e97315e01ddd9827bd7ef8e8c9d3233164de33633c09f5202077e8407076 2024-05-15T12:40:34.281 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:34 smithi120 systemd[1]: Started Ceph mgr.x for deaea04e-12b7-11ef-bc98-c7b262605968. 2024-05-15T12:40:34.281 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:34 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:34.176+0000 7fce4a288200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-15T12:40:34.294 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-05-15T12:40:34.294 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-05-15T12:40:34.294 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000204358 s, 2.5 MB/s 2024-05-15T12:40:34.295 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-15T12:40:34.354 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_4 2024-05-15T12:40:34.412 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-15T12:40:34.412 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:40:34.412 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 6h/6d Inode: 165658 Links: 1 2024-05-15T12:40:34.412 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:40:34.412 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:40:34.412 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-05-15 12:40:08.041487946 +0000 2024-05-15T12:40:34.412 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-05-15 12:33:24.276375070 +0000 2024-05-15T12:40:34.412 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-05-15 12:33:24.276375070 +0000 2024-05-15T12:40:34.413 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: - 2024-05-15T12:40:34.413 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-15T12:40:34.478 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-05-15T12:40:34.478 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-05-15T12:40:34.478 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000235999 s, 2.2 MB/s 2024-05-15T12:40:34.479 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-15T12:40:34.537 INFO:tasks.cephadm:Deploying osd.0 on smithi027 with /dev/vg_nvme/lv_4... 2024-05-15T12:40:34.538 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-15T12:40:34.685 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:34 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:34.347+0000 7fce4a288200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-15T12:40:34.818 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:40:35.030 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:34 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:34.694+0000 7fce4a288200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-15T12:40:35.468 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:35 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:35.232+0000 7fce4a288200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-15T12:40:35.468 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:35 smithi120 ceph-mon[43961]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:35.468 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:35 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:35.468 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:35 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:35.468 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:35 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:35.469 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:35 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:35.469 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:35 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:40:35.469 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:35 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:35.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:35 smithi027 ceph-mon[39795]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:35.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:35 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:35.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:35 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:35.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:35 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:35.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:35 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:35.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:35 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:40:35.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:35 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:35.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:35 smithi177 ceph-mon[44303]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:35.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:35 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:35.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:35 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:35.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:35 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:35.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:35 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:35.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:35 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:40:35.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:35 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:35.780 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:35 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:35.477+0000 7fce4a288200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-15T12:40:36.280 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:35 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:35.922+0000 7fce4a288200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-15T12:40:36.745 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:36 smithi027 ceph-mon[39795]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:36.745 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:36 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:36.745 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:36 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:36.745 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:36 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:36.745 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:36 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:40:36.745 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:36 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:36.745 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:36 smithi027 ceph-mon[39795]: Reconfiguring mgr.y (unknown last config time)... 2024-05-15T12:40:36.745 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:36 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-15T12:40:36.746 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:36 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-15T12:40:36.746 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:36 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:36.746 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:36 smithi027 ceph-mon[39795]: Reconfiguring daemon mgr.y on smithi027 2024-05-15T12:40:36.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:36 smithi177 ceph-mon[44303]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:36.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:36 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:36.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:36 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:36.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:36 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:36.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:36 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:40:36.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:36 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:36.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:36 smithi177 ceph-mon[44303]: Reconfiguring mgr.y (unknown last config time)... 2024-05-15T12:40:36.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:36 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-15T12:40:36.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:36 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-15T12:40:36.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:36 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:36.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:36 smithi177 ceph-mon[44303]: Reconfiguring daemon mgr.y on smithi027 2024-05-15T12:40:37.030 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:36 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:36.630+0000 7fce4a288200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-15T12:40:37.030 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:36 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:36.716+0000 7fce4a288200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-15T12:40:37.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:36 smithi120 ceph-mon[43961]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:37.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:36 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:37.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:36 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:37.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:36 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:37.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:36 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:40:37.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:36 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:37.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:36 smithi120 ceph-mon[43961]: Reconfiguring mgr.y (unknown last config time)... 2024-05-15T12:40:37.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:36 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-15T12:40:37.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:36 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-15T12:40:37.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:36 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:37.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:36 smithi120 ceph-mon[43961]: Reconfiguring daemon mgr.y on smithi027 2024-05-15T12:40:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-15T12:40:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-15T12:40:37.487 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-05-15T12:40:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-05-15T12:40:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0108018 s, 971 MB/s 2024-05-15T12:40:37.487 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 2024-05-15T12:40:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-05-15T12:40:37.712 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:37 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:37.314+0000 7fce4a288200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-15T12:40:37.712 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:37 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:37.398+0000 7fce4a288200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-15T12:40:37.712 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:37 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:37.485+0000 7fce4a288200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-15T12:40:38.030 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:37 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:37.721+0000 7fce4a288200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-15T12:40:38.030 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:37 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:37.812+0000 7fce4a288200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-15T12:40:38.030 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:37 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:37.905+0000 7fce4a288200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-15T12:40:38.410 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:38 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:38.140+0000 7fce4a288200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-15T12:40:38.411 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:38 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:38.235+0000 7fce4a288200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-15T12:40:38.754 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_4 2024-05-15T12:40:38.780 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:38 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:38.420+0000 7fce4a288200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-15T12:40:38.780 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:38 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:38.668+0000 7fce4a288200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-15T12:40:39.026 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:40:39.238 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:38 smithi120 ceph-mon[43961]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:39.238 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:38 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:39.238 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:38 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:39.238 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:38 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:40:39.239 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:38 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:39.239 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:38 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:40:39.239 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:38 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:39.245 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:38 smithi027 ceph-mon[39795]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:39.245 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:38 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:39.245 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:38 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:39.245 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:38 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:40:39.246 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:38 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:39.246 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:38 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:40:39.246 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:38 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:39.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:38 smithi177 ceph-mon[44303]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:39.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:38 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:39.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:38 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:39.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:38 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:40:39.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:38 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:39.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:38 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:40:39.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:38 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:39.530 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:39 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:39.247+0000 7fce4a288200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-15T12:40:39.530 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:39 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:39.454+0000 7fce4a288200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-15T12:40:40.030 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:39 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:39.649+0000 7fce4a288200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:40:40.675 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:40 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:40.305+0000 7fce4a288200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-15T12:40:40.675 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:40 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:40.503+0000 7fce4a288200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:40:40.987 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:40 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:40.684+0000 7fce4a288200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-15T12:40:40.987 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 12:40:40 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x[45614]: 2024-05-15T12:40:40.801+0000 7fce4a288200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-15T12:40:41.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:40 smithi027 ceph-mon[39795]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:41.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:40 smithi027 ceph-mon[39795]: Standby manager daemon x started 2024-05-15T12:40:41.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:40 smithi027 ceph-mon[39795]: from='mgr.? 172.21.15.120:0/1653581496' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-15T12:40:41.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:40 smithi027 ceph-mon[39795]: from='mgr.? 172.21.15.120:0/1653581496' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-15T12:40:41.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:40 smithi027 ceph-mon[39795]: from='mgr.? 172.21.15.120:0/1653581496' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-15T12:40:41.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:40 smithi027 ceph-mon[39795]: from='mgr.? 172.21.15.120:0/1653581496' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-15T12:40:41.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:40 smithi120 ceph-mon[43961]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:41.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:40 smithi120 ceph-mon[43961]: Standby manager daemon x started 2024-05-15T12:40:41.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:40 smithi120 ceph-mon[43961]: from='mgr.? 172.21.15.120:0/1653581496' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-15T12:40:41.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:40 smithi120 ceph-mon[43961]: from='mgr.? 172.21.15.120:0/1653581496' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-15T12:40:41.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:40 smithi120 ceph-mon[43961]: from='mgr.? 172.21.15.120:0/1653581496' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-15T12:40:41.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:40 smithi120 ceph-mon[43961]: from='mgr.? 172.21.15.120:0/1653581496' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-15T12:40:41.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:40 smithi177 ceph-mon[44303]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:41.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:40 smithi177 ceph-mon[44303]: Standby manager daemon x started 2024-05-15T12:40:41.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:40 smithi177 ceph-mon[44303]: from='mgr.? 172.21.15.120:0/1653581496' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-15T12:40:41.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:40 smithi177 ceph-mon[44303]: from='mgr.? 172.21.15.120:0/1653581496' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-15T12:40:41.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:40 smithi177 ceph-mon[44303]: from='mgr.? 172.21.15.120:0/1653581496' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-15T12:40:41.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:40 smithi177 ceph-mon[44303]: from='mgr.? 172.21.15.120:0/1653581496' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-15T12:40:42.191 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:42 smithi027 ceph-mon[39795]: mgrmap e14: y(active, since 81s), standbys: x 2024-05-15T12:40:42.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:42 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-15T12:40:42.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:42 smithi027 ceph-mon[39795]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:40:42.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:42 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:40:42.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:42 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:40:42.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:42 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:42.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:42 smithi120 ceph-mon[43961]: mgrmap e14: y(active, since 81s), standbys: x 2024-05-15T12:40:42.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:42 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-15T12:40:42.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:42 smithi120 ceph-mon[43961]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:40:42.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:42 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:40:42.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:42 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:40:42.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:42 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:42.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:42 smithi177 ceph-mon[44303]: mgrmap e14: y(active, since 81s), standbys: x 2024-05-15T12:40:42.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:42 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-15T12:40:42.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:42 smithi177 ceph-mon[44303]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:40:42.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:42 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:40:42.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:42 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:40:42.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:42 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:43.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:43 smithi027 ceph-mon[39795]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:43.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:43 smithi120 ceph-mon[43961]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:43.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:43 smithi177 ceph-mon[44303]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:44.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:44 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1588042815' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64fb3ad2-08a7-4106-a07a-6c776c877faf"}]: dispatch 2024-05-15T12:40:44.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:44 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1588042815' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "64fb3ad2-08a7-4106-a07a-6c776c877faf"}]': finished 2024-05-15T12:40:44.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:44 smithi027 ceph-mon[39795]: osdmap e5: 1 total, 0 up, 1 in 2024-05-15T12:40:44.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:44 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:40:44.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:44 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1588042815' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64fb3ad2-08a7-4106-a07a-6c776c877faf"}]: dispatch 2024-05-15T12:40:44.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:44 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1588042815' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "64fb3ad2-08a7-4106-a07a-6c776c877faf"}]': finished 2024-05-15T12:40:44.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:44 smithi120 ceph-mon[43961]: osdmap e5: 1 total, 0 up, 1 in 2024-05-15T12:40:44.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:44 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:40:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:44 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1588042815' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64fb3ad2-08a7-4106-a07a-6c776c877faf"}]: dispatch 2024-05-15T12:40:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:44 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1588042815' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "64fb3ad2-08a7-4106-a07a-6c776c877faf"}]': finished 2024-05-15T12:40:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:44 smithi177 ceph-mon[44303]: osdmap e5: 1 total, 0 up, 1 in 2024-05-15T12:40:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:44 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:40:45.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:45 smithi027 ceph-mon[39795]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:45.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:45 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3026787037' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:40:45.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:45 smithi120 ceph-mon[43961]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:45.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:45 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3026787037' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:40:45.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:45 smithi177 ceph-mon[44303]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:45.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:45 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3026787037' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:40:47.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:47 smithi027 ceph-mon[39795]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:47.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:47 smithi120 ceph-mon[43961]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:47.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:47 smithi177 ceph-mon[44303]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:49.077 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:49 smithi027 ceph-mon[39795]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:49.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:49 smithi120 ceph-mon[43961]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:49.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:49 smithi177 ceph-mon[44303]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:51.219 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:51 smithi027 ceph-mon[39795]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:51.219 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:51 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-15T12:40:51.219 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:51 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:51.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:51 smithi120 ceph-mon[43961]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:51.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:51 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-15T12:40:51.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:51 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:51.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:51 smithi177 ceph-mon[44303]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:51.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:51 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-15T12:40:51.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:51 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:52.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:52 smithi027 ceph-mon[39795]: Deploying daemon osd.0 on smithi027 2024-05-15T12:40:52.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:52 smithi120 ceph-mon[43961]: Deploying daemon osd.0 on smithi027 2024-05-15T12:40:52.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:52 smithi177 ceph-mon[44303]: Deploying daemon osd.0 on smithi027 2024-05-15T12:40:53.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:53 smithi027 ceph-mon[39795]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:53.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:53 smithi120 ceph-mon[43961]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:53.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:53 smithi177 ceph-mon[44303]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:55.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:55 smithi027 ceph-mon[39795]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:55.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:55 smithi120 ceph-mon[43961]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:55.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:55 smithi177 ceph-mon[44303]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:57.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:57 smithi177 ceph-mon[44303]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:57.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:57 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:57.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:57 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:40:57.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:57 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:57.469 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:57 smithi027 ceph-mon[39795]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:57.469 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:57 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:57.469 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:57 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:40:57.469 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:57 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:57.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:57 smithi120 ceph-mon[43961]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:57.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:57 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:57.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:57 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:40:57.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:57 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:59.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:59 smithi177 ceph-mon[44303]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:59.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:59 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:59.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:59 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:59.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:59 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:59.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:59 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:40:59.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:40:59 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:59.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:59 smithi027 ceph-mon[39795]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:59.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:59 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:59.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:59 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:59.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:59 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:59.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:59 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:40:59.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:40:59 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:59.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:59 smithi120 ceph-mon[43961]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:40:59.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:59 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:59.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:59 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:40:59.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:59 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:40:59.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:59 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:40:59.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:40:59 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:00.202 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 0 on host 'smithi027' 2024-05-15T12:41:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:00 smithi177 ceph-mon[44303]: from='osd.0 [v2:172.21.15.27:6802/4239116935,v1:172.21.15.27:6803/4239116935]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-15T12:41:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:00 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:00 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:00 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:41:00.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:00 smithi027 ceph-mon[39795]: from='osd.0 [v2:172.21.15.27:6802/4239116935,v1:172.21.15.27:6803/4239116935]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-15T12:41:00.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:00 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:00.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:00 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:00.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:00 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:41:00.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:00 smithi120 ceph-mon[43961]: from='osd.0 [v2:172.21.15.27:6802/4239116935,v1:172.21.15.27:6803/4239116935]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-15T12:41:00.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:00 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:00 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:00 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:41:00.951 DEBUG:teuthology.orchestra.run.smithi027:osd.0> sudo journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.0.service 2024-05-15T12:41:00.953 INFO:tasks.cephadm:Deploying osd.1 on smithi027 with /dev/vg_nvme/lv_3... 2024-05-15T12:41:00.953 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-15T12:41:00.983 INFO:journalctl@ceph.osd.0.smithi027.stdout:-- Logs begin at Wed 2024-05-15 12:27:35 UTC. -- 2024-05-15T12:41:01.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:01 smithi027 ceph-mon[39795]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:41:01.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:01 smithi027 ceph-mon[39795]: from='osd.0 [v2:172.21.15.27:6802/4239116935,v1:172.21.15.27:6803/4239116935]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-15T12:41:01.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:01 smithi027 ceph-mon[39795]: osdmap e6: 1 total, 0 up, 1 in 2024-05-15T12:41:01.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:01 smithi027 ceph-mon[39795]: from='osd.0 [v2:172.21.15.27:6802/4239116935,v1:172.21.15.27:6803/4239116935]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-15T12:41:01.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:01 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:41:01.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:01 smithi120 ceph-mon[43961]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:41:01.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:01 smithi120 ceph-mon[43961]: from='osd.0 [v2:172.21.15.27:6802/4239116935,v1:172.21.15.27:6803/4239116935]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-15T12:41:01.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:01 smithi120 ceph-mon[43961]: osdmap e6: 1 total, 0 up, 1 in 2024-05-15T12:41:01.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:01 smithi120 ceph-mon[43961]: from='osd.0 [v2:172.21.15.27:6802/4239116935,v1:172.21.15.27:6803/4239116935]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-15T12:41:01.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:01 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:41:01.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:01 smithi177 ceph-mon[44303]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:41:01.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:01 smithi177 ceph-mon[44303]: from='osd.0 [v2:172.21.15.27:6802/4239116935,v1:172.21.15.27:6803/4239116935]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-15T12:41:01.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:01 smithi177 ceph-mon[44303]: osdmap e6: 1 total, 0 up, 1 in 2024-05-15T12:41:01.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:01 smithi177 ceph-mon[44303]: from='osd.0 [v2:172.21.15.27:6802/4239116935,v1:172.21.15.27:6803/4239116935]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-15T12:41:01.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:01 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:41:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:41:02.017 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 15 12:41:01 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-0[58372]: 2024-05-15T12:41:01.644+0000 7f89823f8700 -1 osd.0 0 waiting for initial osdmap 2024-05-15T12:41:02.017 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 15 12:41:01 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-0[58372]: 2024-05-15T12:41:01.647+0000 7f89799e2700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:41:02.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:02 smithi027 ceph-mon[39795]: from='osd.0 [v2:172.21.15.27:6802/4239116935,v1:172.21.15.27:6803/4239116935]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-05-15T12:41:02.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:02 smithi027 ceph-mon[39795]: osdmap e7: 1 total, 0 up, 1 in 2024-05-15T12:41:02.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:02 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:41:02.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:02 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:41:02.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:02 smithi027 ceph-mon[39795]: from='osd.0 [v2:172.21.15.27:6802/4239116935,v1:172.21.15.27:6803/4239116935]' entity='osd.0' 2024-05-15T12:41:02.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:02 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:41:02.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:02 smithi120 ceph-mon[43961]: from='osd.0 [v2:172.21.15.27:6802/4239116935,v1:172.21.15.27:6803/4239116935]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-05-15T12:41:02.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:02 smithi120 ceph-mon[43961]: osdmap e7: 1 total, 0 up, 1 in 2024-05-15T12:41:02.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:02 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:41:02.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:02 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:41:02.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:02 smithi120 ceph-mon[43961]: from='osd.0 [v2:172.21.15.27:6802/4239116935,v1:172.21.15.27:6803/4239116935]' entity='osd.0' 2024-05-15T12:41:02.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:02 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:41:02.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:02 smithi177 ceph-mon[44303]: from='osd.0 [v2:172.21.15.27:6802/4239116935,v1:172.21.15.27:6803/4239116935]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-05-15T12:41:02.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:02 smithi177 ceph-mon[44303]: osdmap e7: 1 total, 0 up, 1 in 2024-05-15T12:41:02.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:02 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:41:02.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:02 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:41:02.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:02 smithi177 ceph-mon[44303]: from='osd.0 [v2:172.21.15.27:6802/4239116935,v1:172.21.15.27:6803/4239116935]' entity='osd.0' 2024-05-15T12:41:02.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:02 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:41:03.469 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:03 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:41:03.470 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:03 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:41:03.470 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:03 smithi027 ceph-mon[39795]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:41:03.470 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:03 smithi027 ceph-mon[39795]: osd.0 [v2:172.21.15.27:6802/4239116935,v1:172.21.15.27:6803/4239116935] boot 2024-05-15T12:41:03.470 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:03 smithi027 ceph-mon[39795]: osdmap e8: 1 total, 1 up, 1 in 2024-05-15T12:41:03.470 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:03 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:41:03.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:03 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:41:03.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:03 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:41:03.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:03 smithi120 ceph-mon[43961]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:41:03.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:03 smithi120 ceph-mon[43961]: osd.0 [v2:172.21.15.27:6802/4239116935,v1:172.21.15.27:6803/4239116935] boot 2024-05-15T12:41:03.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:03 smithi120 ceph-mon[43961]: osdmap e8: 1 total, 1 up, 1 in 2024-05-15T12:41:03.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:03 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:41:03.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:03 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:41:03.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:03 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:41:03.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:03 smithi177 ceph-mon[44303]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:41:03.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:03 smithi177 ceph-mon[44303]: osd.0 [v2:172.21.15.27:6802/4239116935,v1:172.21.15.27:6803/4239116935] boot 2024-05-15T12:41:03.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:03 smithi177 ceph-mon[44303]: osdmap e8: 1 total, 1 up, 1 in 2024-05-15T12:41:03.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:03 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:41:04.829 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-15T12:41:04.829 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-15T12:41:04.829 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-05-15T12:41:04.829 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-05-15T12:41:04.830 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110131 s, 952 MB/s 2024-05-15T12:41:04.830 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-05-15T12:41:05.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:05 smithi027 ceph-mon[39795]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:05.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:05 smithi027 ceph-mon[39795]: osdmap e9: 1 total, 1 up, 1 in 2024-05-15T12:41:05.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:05 smithi120 ceph-mon[43961]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:05.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:05 smithi120 ceph-mon[43961]: osdmap e9: 1 total, 1 up, 1 in 2024-05-15T12:41:05.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:05 smithi177 ceph-mon[44303]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:05.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:05 smithi177 ceph-mon[44303]: osdmap e9: 1 total, 1 up, 1 in 2024-05-15T12:41:05.925 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_3 2024-05-15T12:41:06.196 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:41:06.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:06 smithi027 ceph-mon[39795]: Detected new or changed devices on smithi027 2024-05-15T12:41:06.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:06 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:06.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:06 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:06.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:06 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:41:06.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:06 smithi027 ceph-mon[39795]: Adjusting osd_memory_target on smithi027 to 17122M 2024-05-15T12:41:06.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:06 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:06.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:06 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:06.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:06 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:41:06.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:06 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:06.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:06 smithi120 ceph-mon[43961]: Detected new or changed devices on smithi027 2024-05-15T12:41:06.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:06 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:06.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:06 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:06.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:06 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:41:06.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:06 smithi120 ceph-mon[43961]: Adjusting osd_memory_target on smithi027 to 17122M 2024-05-15T12:41:06.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:06 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:06.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:06 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:06.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:06 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:41:06.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:06 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:06.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:06 smithi177 ceph-mon[44303]: Detected new or changed devices on smithi027 2024-05-15T12:41:06.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:06 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:06.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:06 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:06.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:06 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:41:06.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:06 smithi177 ceph-mon[44303]: Adjusting osd_memory_target on smithi027 to 17122M 2024-05-15T12:41:06.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:06 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:06.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:06 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:06.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:06 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:41:06.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:06 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:07.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:07 smithi027 ceph-mon[39795]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:07.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:07 smithi120 ceph-mon[43961]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:07.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:07 smithi177 ceph-mon[44303]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:09.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:09 smithi027 ceph-mon[39795]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:09.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:09 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:41:09.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:09 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:41:09.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:09 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:09.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:09 smithi120 ceph-mon[43961]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:09.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:09 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:41:09.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:09 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:41:09.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:09 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:09.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:09 smithi177 ceph-mon[44303]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:09.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:09 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:41:09.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:09 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:41:09.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:09 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:10.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:10 smithi027 ceph-mon[39795]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:41:10.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:10 smithi120 ceph-mon[43961]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:41:10.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:10 smithi177 ceph-mon[44303]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:41:11.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:11 smithi120 ceph-mon[43961]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:11.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:11 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2773303387' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c16dd90-9d51-48b8-a296-1c8879ba1bc8"}]: dispatch 2024-05-15T12:41:11.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:11 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2773303387' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c16dd90-9d51-48b8-a296-1c8879ba1bc8"}]': finished 2024-05-15T12:41:11.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:11 smithi120 ceph-mon[43961]: osdmap e10: 2 total, 1 up, 2 in 2024-05-15T12:41:11.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:11 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:41:11.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:11 smithi177 ceph-mon[44303]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:11.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:11 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2773303387' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c16dd90-9d51-48b8-a296-1c8879ba1bc8"}]: dispatch 2024-05-15T12:41:11.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:11 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2773303387' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c16dd90-9d51-48b8-a296-1c8879ba1bc8"}]': finished 2024-05-15T12:41:11.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:11 smithi177 ceph-mon[44303]: osdmap e10: 2 total, 1 up, 2 in 2024-05-15T12:41:11.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:11 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:41:11.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:11 smithi027 ceph-mon[39795]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:11.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:11 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2773303387' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c16dd90-9d51-48b8-a296-1c8879ba1bc8"}]: dispatch 2024-05-15T12:41:11.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:11 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2773303387' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c16dd90-9d51-48b8-a296-1c8879ba1bc8"}]': finished 2024-05-15T12:41:11.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:11 smithi027 ceph-mon[39795]: osdmap e10: 2 total, 1 up, 2 in 2024-05-15T12:41:11.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:11 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:41:12.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:12 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3260683668' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:41:12.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:12 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3260683668' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:41:12.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:12 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3260683668' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:41:13.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:13 smithi120 ceph-mon[43961]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:13.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:13 smithi177 ceph-mon[44303]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:13.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:13 smithi027 ceph-mon[39795]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:15.628 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:15 smithi027 ceph-mon[39795]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:15.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:15 smithi177 ceph-mon[44303]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:15.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:15 smithi120 ceph-mon[43961]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:17.419 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:17 smithi027 ceph-mon[39795]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:17.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:17 smithi177 ceph-mon[44303]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:17.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:17 smithi120 ceph-mon[43961]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:18.571 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-15T12:41:18.571 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:18.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-15T12:41:18.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:18.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-15T12:41:18.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:19.614 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:19 smithi027 ceph-mon[39795]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:19.614 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:19 smithi027 ceph-mon[39795]: Deploying daemon osd.1 on smithi027 2024-05-15T12:41:19.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:19 smithi177 ceph-mon[44303]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:19.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:19 smithi177 ceph-mon[44303]: Deploying daemon osd.1 on smithi027 2024-05-15T12:41:19.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:19 smithi120 ceph-mon[43961]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:19.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:19 smithi120 ceph-mon[43961]: Deploying daemon osd.1 on smithi027 2024-05-15T12:41:21.514 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:21 smithi027 ceph-mon[39795]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:21.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:21 smithi177 ceph-mon[44303]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:21.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:21 smithi120 ceph-mon[43961]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:23.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:23 smithi177 ceph-mon[44303]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:23.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:23 smithi027 ceph-mon[39795]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:23.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:23 smithi120 ceph-mon[43961]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:24.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:24 smithi027 ceph-mon[39795]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:24.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:24 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:24.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:24 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:24.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:24 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:41:24.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:24 smithi120 ceph-mon[43961]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:24.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:24.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:24.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:41:24.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:24 smithi177 ceph-mon[44303]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:24.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:24 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:24.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:24 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:24.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:24 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:41:26.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:26 smithi027 ceph-mon[39795]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:26.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:26 smithi120 ceph-mon[43961]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:26.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:26 smithi177 ceph-mon[44303]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:27.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:27 smithi027 ceph-mon[39795]: from='osd.1 [v2:172.21.15.27:6810/1466270637,v1:172.21.15.27:6811/1466270637]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-15T12:41:27.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:27 smithi027 ceph-mon[39795]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-15T12:41:27.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:27 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:27.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:27 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:27.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:27 smithi120 ceph-mon[43961]: from='osd.1 [v2:172.21.15.27:6810/1466270637,v1:172.21.15.27:6811/1466270637]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-15T12:41:27.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:27 smithi120 ceph-mon[43961]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-15T12:41:27.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:27 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:27.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:27 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:27.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:27 smithi177 ceph-mon[44303]: from='osd.1 [v2:172.21.15.27:6810/1466270637,v1:172.21.15.27:6811/1466270637]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-15T12:41:27.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:27 smithi177 ceph-mon[44303]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-15T12:41:27.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:27 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:27.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:27 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:27.992 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 1 on host 'smithi027' 2024-05-15T12:41:28.730 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:28 smithi027 ceph-mon[39795]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:28.730 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:28 smithi027 ceph-mon[39795]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-15T12:41:28.730 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:28 smithi027 ceph-mon[39795]: osdmap e11: 2 total, 1 up, 2 in 2024-05-15T12:41:28.730 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:41:28.730 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:28 smithi027 ceph-mon[39795]: from='osd.1 [v2:172.21.15.27:6810/1466270637,v1:172.21.15.27:6811/1466270637]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-15T12:41:28.730 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:28 smithi027 ceph-mon[39795]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-15T12:41:28.730 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:28.730 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:28.759 DEBUG:teuthology.orchestra.run.smithi027:osd.1> sudo journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.1.service 2024-05-15T12:41:28.761 INFO:tasks.cephadm:Deploying osd.2 on smithi027 with /dev/vg_nvme/lv_2... 2024-05-15T12:41:28.761 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-15T12:41:28.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:28 smithi120 ceph-mon[43961]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:28.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:28 smithi120 ceph-mon[43961]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-15T12:41:28.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:28 smithi120 ceph-mon[43961]: osdmap e11: 2 total, 1 up, 2 in 2024-05-15T12:41:28.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:41:28.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:28 smithi120 ceph-mon[43961]: from='osd.1 [v2:172.21.15.27:6810/1466270637,v1:172.21.15.27:6811/1466270637]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-15T12:41:28.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:28 smithi120 ceph-mon[43961]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-15T12:41:28.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:28.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:28.793 INFO:journalctl@ceph.osd.1.smithi027.stdout:-- Logs begin at Wed 2024-05-15 12:27:35 UTC. -- 2024-05-15T12:41:28.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:28 smithi177 ceph-mon[44303]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:28.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:28 smithi177 ceph-mon[44303]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-15T12:41:28.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:28 smithi177 ceph-mon[44303]: osdmap e11: 2 total, 1 up, 2 in 2024-05-15T12:41:28.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:41:28.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:28 smithi177 ceph-mon[44303]: from='osd.1 [v2:172.21.15.27:6810/1466270637,v1:172.21.15.27:6811/1466270637]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-15T12:41:28.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:28 smithi177 ceph-mon[44303]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-15T12:41:28.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:28.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:28.981 INFO:journalctl@ceph.osd.1.smithi027.stdout:May 15 12:41:28 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-1[66219]: 2024-05-15T12:41:28.988+0000 7f419c3a3700 -1 osd.1 0 waiting for initial osdmap 2024-05-15T12:41:29.270 INFO:journalctl@ceph.osd.1.smithi027.stdout:May 15 12:41:28 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-1[66219]: 2024-05-15T12:41:28.991+0000 7f419418e700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:41:29.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:29 smithi027 ceph-mon[39795]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-05-15T12:41:29.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:29 smithi027 ceph-mon[39795]: osdmap e12: 2 total, 1 up, 2 in 2024-05-15T12:41:29.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:29 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:41:29.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:29 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:41:29.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:29 smithi027 ceph-mon[39795]: from='osd.1 ' entity='osd.1' 2024-05-15T12:41:29.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:29 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:41:29.775 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:41:29.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:29 smithi120 ceph-mon[43961]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-05-15T12:41:29.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:29 smithi120 ceph-mon[43961]: osdmap e12: 2 total, 1 up, 2 in 2024-05-15T12:41:29.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:29 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:41:29.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:29 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:41:29.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:29 smithi120 ceph-mon[43961]: from='osd.1 ' entity='osd.1' 2024-05-15T12:41:29.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:29 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:41:29.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:29 smithi177 ceph-mon[44303]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-05-15T12:41:29.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:29 smithi177 ceph-mon[44303]: osdmap e12: 2 total, 1 up, 2 in 2024-05-15T12:41:29.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:29 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:41:29.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:29 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:41:29.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:29 smithi177 ceph-mon[44303]: from='osd.1 ' entity='osd.1' 2024-05-15T12:41:29.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:29 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:41:30.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:30 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:41:30.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:30 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:41:30.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:30 smithi027 ceph-mon[39795]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:30.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:30 smithi027 ceph-mon[39795]: osd.1 [v2:172.21.15.27:6810/1466270637,v1:172.21.15.27:6811/1466270637] boot 2024-05-15T12:41:30.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:30 smithi027 ceph-mon[39795]: osdmap e13: 2 total, 2 up, 2 in 2024-05-15T12:41:30.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:30 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:41:30.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:30 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:41:30.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:30 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:41:30.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:30 smithi120 ceph-mon[43961]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:30.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:30 smithi120 ceph-mon[43961]: osd.1 [v2:172.21.15.27:6810/1466270637,v1:172.21.15.27:6811/1466270637] boot 2024-05-15T12:41:30.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:30 smithi120 ceph-mon[43961]: osdmap e13: 2 total, 2 up, 2 in 2024-05-15T12:41:30.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:30 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:41:30.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:30 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:41:30.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:30 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:41:30.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:30 smithi177 ceph-mon[44303]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:41:30.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:30 smithi177 ceph-mon[44303]: osd.1 [v2:172.21.15.27:6810/1466270637,v1:172.21.15.27:6811/1466270637] boot 2024-05-15T12:41:30.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:30 smithi177 ceph-mon[44303]: osdmap e13: 2 total, 2 up, 2 in 2024-05-15T12:41:30.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:30 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:41:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-15T12:41:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-15T12:41:32.524 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-05-15T12:41:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-05-15T12:41:32.524 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113928 s, 920 MB/s 2024-05-15T12:41:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-05-15T12:41:32.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:32 smithi120 ceph-mon[43961]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:32.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:32 smithi120 ceph-mon[43961]: osdmap e14: 2 total, 2 up, 2 in 2024-05-15T12:41:32.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:32 smithi177 ceph-mon[44303]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:32.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:32 smithi177 ceph-mon[44303]: osdmap e14: 2 total, 2 up, 2 in 2024-05-15T12:41:33.002 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:32 smithi027 ceph-mon[39795]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:33.003 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:32 smithi027 ceph-mon[39795]: osdmap e14: 2 total, 2 up, 2 in 2024-05-15T12:41:33.335 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_2 2024-05-15T12:41:33.911 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:41:35.078 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:34 smithi027 ceph-mon[39795]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:35.079 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:34 smithi027 ceph-mon[39795]: Detected new or changed devices on smithi027 2024-05-15T12:41:35.079 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:34 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:35.079 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:34 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:35.079 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:34 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:41:35.079 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:34 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:41:35.079 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:34 smithi027 ceph-mon[39795]: Adjusting osd_memory_target on smithi027 to 8561M 2024-05-15T12:41:35.079 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:34 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:35.079 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:34 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:35.079 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:34 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:41:35.079 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:34 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:35.079 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:34 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:41:35.079 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:34 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:35.079 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:34 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:41:35.079 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:34 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:35.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:34 smithi120 ceph-mon[43961]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:35.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:34 smithi120 ceph-mon[43961]: Detected new or changed devices on smithi027 2024-05-15T12:41:35.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:34 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:35.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:34 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:35.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:34 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:41:35.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:34 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:41:35.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:34 smithi120 ceph-mon[43961]: Adjusting osd_memory_target on smithi027 to 8561M 2024-05-15T12:41:35.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:34 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:35.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:34 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:35.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:34 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:41:35.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:34 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:35.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:34 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:41:35.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:34 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:35.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:34 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:41:35.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:34 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:35.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:34 smithi177 ceph-mon[44303]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:35.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:34 smithi177 ceph-mon[44303]: Detected new or changed devices on smithi027 2024-05-15T12:41:35.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:34 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:35.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:34 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:35.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:34 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:41:35.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:34 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:41:35.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:34 smithi177 ceph-mon[44303]: Adjusting osd_memory_target on smithi027 to 8561M 2024-05-15T12:41:35.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:34 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:35.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:34 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:35.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:34 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:41:35.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:34 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:35.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:34 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:41:35.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:34 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:35.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:34 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:41:35.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:34 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:36.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:35 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:41:36.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:35 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:41:36.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:35 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:36.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:35 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:41:36.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:35 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:41:36.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:35 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:36.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:35 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:41:36.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:35 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:41:36.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:35 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:37.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:36 smithi027 ceph-mon[39795]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:37.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:36 smithi027 ceph-mon[39795]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:41:37.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:36 smithi120 ceph-mon[43961]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:37.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:36 smithi120 ceph-mon[43961]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:41:37.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:36 smithi177 ceph-mon[44303]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:37.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:36 smithi177 ceph-mon[44303]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:41:39.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:38 smithi027 ceph-mon[39795]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:39.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:38 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1333348103' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cbe3ba13-3ea3-4891-8d2d-6b85581a732c"}]: dispatch 2024-05-15T12:41:39.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:38 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1333348103' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cbe3ba13-3ea3-4891-8d2d-6b85581a732c"}]': finished 2024-05-15T12:41:39.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:38 smithi027 ceph-mon[39795]: osdmap e15: 3 total, 2 up, 3 in 2024-05-15T12:41:39.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:38 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:41:39.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:38 smithi120 ceph-mon[43961]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:39.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:38 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1333348103' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cbe3ba13-3ea3-4891-8d2d-6b85581a732c"}]: dispatch 2024-05-15T12:41:39.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:38 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1333348103' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cbe3ba13-3ea3-4891-8d2d-6b85581a732c"}]': finished 2024-05-15T12:41:39.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:38 smithi120 ceph-mon[43961]: osdmap e15: 3 total, 2 up, 3 in 2024-05-15T12:41:39.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:38 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:41:39.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:38 smithi177 ceph-mon[44303]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:39.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:38 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1333348103' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cbe3ba13-3ea3-4891-8d2d-6b85581a732c"}]: dispatch 2024-05-15T12:41:39.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:38 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1333348103' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cbe3ba13-3ea3-4891-8d2d-6b85581a732c"}]': finished 2024-05-15T12:41:39.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:38 smithi177 ceph-mon[44303]: osdmap e15: 3 total, 2 up, 3 in 2024-05-15T12:41:39.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:38 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:41:40.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:39 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3730574482' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:41:40.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:39 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3730574482' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:41:40.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:39 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3730574482' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:41:41.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:40 smithi027 ceph-mon[39795]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:41.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:40 smithi120 ceph-mon[43961]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:41.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:40 smithi177 ceph-mon[44303]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:43.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:42 smithi027 ceph-mon[39795]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:43.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:42 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:41:43.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:42 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:41:43.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:42 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:41:43.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:42 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:41:43.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:42 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:43.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:42 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:41:43.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:42 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:43.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:42 smithi120 ceph-mon[43961]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:43.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:42 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:41:43.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:42 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:41:43.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:42 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:41:43.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:42 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:41:43.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:42 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:43.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:42 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:41:43.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:42 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:43.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:42 smithi177 ceph-mon[44303]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:43.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:42 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:41:43.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:42 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:41:43.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:42 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:41:43.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:42 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:41:43.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:42 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:43.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:42 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:41:43.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:42 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:45.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:45 smithi177 ceph-mon[44303]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:45.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:45 smithi027 ceph-mon[39795]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:45.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:45 smithi120 ceph-mon[43961]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:46.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:46 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-15T12:41:46.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:46 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:46.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:46 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-15T12:41:46.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:46 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:46.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:46 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-15T12:41:46.596 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:46 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:47.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:47 smithi027 ceph-mon[39795]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:47.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:47 smithi027 ceph-mon[39795]: Deploying daemon osd.2 on smithi027 2024-05-15T12:41:47.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:47 smithi177 ceph-mon[44303]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:47.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:47 smithi177 ceph-mon[44303]: Deploying daemon osd.2 on smithi027 2024-05-15T12:41:47.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:47 smithi120 ceph-mon[43961]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:47.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:47 smithi120 ceph-mon[43961]: Deploying daemon osd.2 on smithi027 2024-05-15T12:41:49.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:49 smithi027 ceph-mon[39795]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:49.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:49 smithi177 ceph-mon[44303]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:49.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:49 smithi120 ceph-mon[43961]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:51.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:51 smithi177 ceph-mon[44303]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:51.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:51 smithi027 ceph-mon[39795]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:51.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:51 smithi120 ceph-mon[43961]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:52.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:52 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:52.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:52 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:52.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:52 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:41:52.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:52 smithi177 ceph-mon[44303]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:52.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:52 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:52.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:52 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:52.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:52 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:41:52.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:52 smithi027 ceph-mon[39795]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:52.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:52 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:52.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:52 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:52.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:52 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:41:52.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:52 smithi120 ceph-mon[43961]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:54.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:54 smithi027 ceph-mon[39795]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:54.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:54 smithi027 ceph-mon[39795]: from='osd.2 [v2:172.21.15.27:6818/3285413897,v1:172.21.15.27:6819/3285413897]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-15T12:41:54.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:54 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:54.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:54 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:54.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:54 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:54.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:54 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:41:54.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:54 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:54.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:54 smithi120 ceph-mon[43961]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:54.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:54 smithi120 ceph-mon[43961]: from='osd.2 [v2:172.21.15.27:6818/3285413897,v1:172.21.15.27:6819/3285413897]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-15T12:41:54.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:54 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:54.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:54 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:54.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:54 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:54.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:54 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:41:54.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:54 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:54.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:54 smithi177 ceph-mon[44303]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:54.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:54 smithi177 ceph-mon[44303]: from='osd.2 [v2:172.21.15.27:6818/3285413897,v1:172.21.15.27:6819/3285413897]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-15T12:41:54.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:54 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:54.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:54 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:54.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:54 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:41:54.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:54 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:41:54.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:54 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:55.265 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 2 on host 'smithi027' 2024-05-15T12:41:55.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:55 smithi027 ceph-mon[39795]: from='osd.2 [v2:172.21.15.27:6818/3285413897,v1:172.21.15.27:6819/3285413897]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-15T12:41:55.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:55 smithi027 ceph-mon[39795]: osdmap e16: 3 total, 2 up, 3 in 2024-05-15T12:41:55.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:55 smithi027 ceph-mon[39795]: from='osd.2 [v2:172.21.15.27:6818/3285413897,v1:172.21.15.27:6819/3285413897]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-15T12:41:55.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:55 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:41:55.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:55 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:55.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:55 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:55.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:55 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:41:55.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:55 smithi120 ceph-mon[43961]: from='osd.2 [v2:172.21.15.27:6818/3285413897,v1:172.21.15.27:6819/3285413897]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-15T12:41:55.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:55 smithi120 ceph-mon[43961]: osdmap e16: 3 total, 2 up, 3 in 2024-05-15T12:41:55.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:55 smithi120 ceph-mon[43961]: from='osd.2 [v2:172.21.15.27:6818/3285413897,v1:172.21.15.27:6819/3285413897]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-15T12:41:55.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:55 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:41:55.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:55 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:55.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:55 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:55.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:55 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:41:55.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:55 smithi177 ceph-mon[44303]: from='osd.2 [v2:172.21.15.27:6818/3285413897,v1:172.21.15.27:6819/3285413897]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-15T12:41:55.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:55 smithi177 ceph-mon[44303]: osdmap e16: 3 total, 2 up, 3 in 2024-05-15T12:41:55.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:55 smithi177 ceph-mon[44303]: from='osd.2 [v2:172.21.15.27:6818/3285413897,v1:172.21.15.27:6819/3285413897]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-15T12:41:55.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:55 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:41:55.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:55 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:55.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:55 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:41:55.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:55 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:41:56.030 DEBUG:teuthology.orchestra.run.smithi027:osd.2> sudo journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.2.service 2024-05-15T12:41:56.032 INFO:tasks.cephadm:Deploying osd.3 on smithi027 with /dev/vg_nvme/lv_1... 2024-05-15T12:41:56.033 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-15T12:41:56.069 INFO:journalctl@ceph.osd.2.smithi027.stdout:-- Logs begin at Wed 2024-05-15 12:27:35 UTC. -- 2024-05-15T12:41:56.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:56 smithi027 ceph-mon[39795]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:56.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:56 smithi027 ceph-mon[39795]: from='osd.2 [v2:172.21.15.27:6818/3285413897,v1:172.21.15.27:6819/3285413897]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-05-15T12:41:56.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:56 smithi027 ceph-mon[39795]: osdmap e17: 3 total, 2 up, 3 in 2024-05-15T12:41:56.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:56 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:41:56.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:56 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:41:56.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:56 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:41:56.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:56 smithi120 ceph-mon[43961]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:56.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:56 smithi120 ceph-mon[43961]: from='osd.2 [v2:172.21.15.27:6818/3285413897,v1:172.21.15.27:6819/3285413897]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-05-15T12:41:56.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:56 smithi120 ceph-mon[43961]: osdmap e17: 3 total, 2 up, 3 in 2024-05-15T12:41:56.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:56 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:41:56.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:56 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:41:56.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:56 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:41:56.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:56 smithi177 ceph-mon[44303]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:56.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:56 smithi177 ceph-mon[44303]: from='osd.2 [v2:172.21.15.27:6818/3285413897,v1:172.21.15.27:6819/3285413897]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-05-15T12:41:56.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:56 smithi177 ceph-mon[44303]: osdmap e17: 3 total, 2 up, 3 in 2024-05-15T12:41:56.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:56 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:41:56.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:56 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:41:56.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:56 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:41:57.127 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:41:57.624 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:57 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:41:57.624 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:57 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:41:57.624 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:57 smithi027 ceph-mon[39795]: osd.2 [v2:172.21.15.27:6818/3285413897,v1:172.21.15.27:6819/3285413897] boot 2024-05-15T12:41:57.624 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:57 smithi027 ceph-mon[39795]: osdmap e18: 3 total, 3 up, 3 in 2024-05-15T12:41:57.624 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:57 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:41:57.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:57 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:41:57.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:57 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:41:57.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:57 smithi120 ceph-mon[43961]: osd.2 [v2:172.21.15.27:6818/3285413897,v1:172.21.15.27:6819/3285413897] boot 2024-05-15T12:41:57.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:57 smithi120 ceph-mon[43961]: osdmap e18: 3 total, 3 up, 3 in 2024-05-15T12:41:57.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:57 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:41:57.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:57 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:41:57.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:57 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:41:57.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:57 smithi177 ceph-mon[44303]: osd.2 [v2:172.21.15.27:6818/3285413897,v1:172.21.15.27:6819/3285413897] boot 2024-05-15T12:41:57.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:57 smithi177 ceph-mon[44303]: osdmap e18: 3 total, 3 up, 3 in 2024-05-15T12:41:57.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:57 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:41:58.731 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:58 smithi027 ceph-mon[39795]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:58.732 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:58 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:41:58.732 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:58 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-15T12:41:58.732 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:58 smithi027 ceph-mon[39795]: osdmap e19: 3 total, 3 up, 3 in 2024-05-15T12:41:58.732 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:58 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:41:58.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:58 smithi120 ceph-mon[43961]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:58.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:58 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:41:58.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:58 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-15T12:41:58.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:58 smithi120 ceph-mon[43961]: osdmap e19: 3 total, 3 up, 3 in 2024-05-15T12:41:58.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:58 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:41:58.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:58 smithi177 ceph-mon[44303]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:41:58.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:58 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:41:58.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:58 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-15T12:41:58.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:58 smithi177 ceph-mon[44303]: osdmap e19: 3 total, 3 up, 3 in 2024-05-15T12:41:58.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:58 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:41:59.771 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 15 12:41:59 smithi027 sudo[75644]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-15T12:41:59.771 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 15 12:41:59 smithi027 sudo[75644]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T12:41:59.771 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 15 12:41:59 smithi027 sudo[75644]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T12:41:59.771 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 15 12:41:59 smithi027 sudo[75644]: pam_unix(sudo:session): session closed for user root 2024-05-15T12:42:00.102 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-15T12:42:00.102 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-15T12:42:00.102 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-05-15T12:42:00.102 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-05-15T12:42:00.102 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0152918 s, 686 MB/s 2024-05-15T12:42:00.102 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-05-15T12:42:00.232 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:59 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-15T12:42:00.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:41:59 smithi027 ceph-mon[39795]: osdmap e20: 3 total, 3 up, 3 in 2024-05-15T12:42:00.233 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 15 12:42:00 smithi027 sudo[77333]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-15T12:42:00.233 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 15 12:42:00 smithi027 sudo[77333]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T12:42:00.233 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 15 12:42:00 smithi027 sudo[77333]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T12:42:00.233 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 15 12:42:00 smithi027 sudo[77333]: pam_unix(sudo:session): session closed for user root 2024-05-15T12:42:00.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:59 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-15T12:42:00.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:41:59 smithi120 ceph-mon[43961]: osdmap e20: 3 total, 3 up, 3 in 2024-05-15T12:42:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:59 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-15T12:42:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:41:59 smithi177 ceph-mon[44303]: osdmap e20: 3 total, 3 up, 3 in 2024-05-15T12:42:00.770 INFO:journalctl@ceph.osd.1.smithi027.stdout:May 15 12:42:00 smithi027 sudo[78862]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-15T12:42:00.771 INFO:journalctl@ceph.osd.1.smithi027.stdout:May 15 12:42:00 smithi027 sudo[78862]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T12:42:00.771 INFO:journalctl@ceph.osd.1.smithi027.stdout:May 15 12:42:00 smithi027 sudo[78862]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T12:42:00.771 INFO:journalctl@ceph.osd.1.smithi027.stdout:May 15 12:42:00 smithi027 sudo[78862]: pam_unix(sudo:session): session closed for user root 2024-05-15T12:42:00.818 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_1 2024-05-15T12:42:01.142 INFO:journalctl@ceph.osd.1.smithi027.stdout:May 15 12:42:00 smithi027 sudo[78893]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-15T12:42:01.142 INFO:journalctl@ceph.osd.1.smithi027.stdout:May 15 12:42:00 smithi027 sudo[78893]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T12:42:01.142 INFO:journalctl@ceph.osd.1.smithi027.stdout:May 15 12:42:00 smithi027 sudo[78893]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T12:42:01.142 INFO:journalctl@ceph.osd.1.smithi027.stdout:May 15 12:42:00 smithi027 sudo[78893]: pam_unix(sudo:session): session closed for user root 2024-05-15T12:42:01.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:00 smithi027 ceph-mon[39795]: pgmap v90: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:01.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:00 smithi027 ceph-mon[39795]: osdmap e21: 3 total, 3 up, 3 in 2024-05-15T12:42:01.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:00 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:01.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:00 smithi120 ceph-mon[43961]: pgmap v90: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:01.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:00 smithi120 ceph-mon[43961]: osdmap e21: 3 total, 3 up, 3 in 2024-05-15T12:42:01.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:00 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:01.396 INFO:journalctl@ceph.osd.2.smithi027.stdout:May 15 12:42:01 smithi027 sudo[78928]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-15T12:42:01.396 INFO:journalctl@ceph.osd.2.smithi027.stdout:May 15 12:42:01 smithi027 sudo[78928]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T12:42:01.396 INFO:journalctl@ceph.osd.2.smithi027.stdout:May 15 12:42:01 smithi027 sudo[78928]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T12:42:01.396 INFO:journalctl@ceph.osd.2.smithi027.stdout:May 15 12:42:01 smithi027 sudo[78928]: pam_unix(sudo:session): session closed for user root 2024-05-15T12:42:01.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:00 smithi177 ceph-mon[44303]: pgmap v90: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:01.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:00 smithi177 ceph-mon[44303]: osdmap e21: 3 total, 3 up, 3 in 2024-05-15T12:42:01.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:00 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:42:02.119 INFO:journalctl@ceph.osd.2.smithi027.stdout:May 15 12:42:01 smithi027 sudo[79005]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-15T12:42:02.119 INFO:journalctl@ceph.osd.2.smithi027.stdout:May 15 12:42:01 smithi027 sudo[79005]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T12:42:02.119 INFO:journalctl@ceph.osd.2.smithi027.stdout:May 15 12:42:01 smithi027 sudo[79005]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T12:42:02.119 INFO:journalctl@ceph.osd.2.smithi027.stdout:May 15 12:42:01 smithi027 sudo[79005]: pam_unix(sudo:session): session closed for user root 2024-05-15T12:42:02.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:02 smithi027 sudo[79046]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-15T12:42:02.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:02 smithi027 sudo[79046]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T12:42:02.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:02 smithi027 sudo[79046]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T12:42:02.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:02 smithi027 sudo[79046]: pam_unix(sudo:session): session closed for user root 2024-05-15T12:42:03.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:02 smithi027 sudo[79057]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-15T12:42:03.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:02 smithi027 sudo[79057]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T12:42:03.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:02 smithi027 sudo[79057]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T12:42:03.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:02 smithi027 sudo[79057]: pam_unix(sudo:session): session closed for user root 2024-05-15T12:42:03.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:02 smithi027 ceph-mon[39795]: mgrmap e15: y(active, since 2m), standbys: x 2024-05-15T12:42:03.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:02 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:03.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:02 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:03.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:02 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:42:03.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:02 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:42:03.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:02 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:42:03.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:02 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:03.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:02 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:03.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:02 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:42:03.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:02 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:03.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:02 smithi027 ceph-mon[39795]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T12:42:03.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:02 smithi120 ceph-mon[43961]: mgrmap e15: y(active, since 2m), standbys: x 2024-05-15T12:42:03.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:02 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:03.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:02 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:03.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:02 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:42:03.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:02 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:42:03.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:02 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:42:03.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:02 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:03.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:02 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:03.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:02 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:42:03.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:02 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:03.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:02 smithi120 ceph-mon[43961]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T12:42:03.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:02 smithi177 ceph-mon[44303]: mgrmap e15: y(active, since 2m), standbys: x 2024-05-15T12:42:03.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:02 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:03.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:02 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:03.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:02 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:42:03.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:02 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:42:03.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:02 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:42:03.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:02 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:03.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:02 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:03.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:02 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:42:03.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:02 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:03.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:02 smithi177 ceph-mon[44303]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T12:42:03.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:02 smithi177 sudo[46782]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-15T12:42:03.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:02 smithi177 sudo[46782]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T12:42:03.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:02 smithi177 sudo[46782]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T12:42:03.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 sudo[46782]: pam_unix(sudo:session): session closed for user root 2024-05-15T12:42:03.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 sudo[46785]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-15T12:42:03.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 sudo[46785]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T12:42:03.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 sudo[46785]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T12:42:03.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 sudo[46785]: pam_unix(sudo:session): session closed for user root 2024-05-15T12:42:03.732 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 sudo[46444]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-15T12:42:03.732 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 sudo[46444]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T12:42:03.732 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 sudo[46444]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T12:42:03.732 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 sudo[46444]: pam_unix(sudo:session): session closed for user root 2024-05-15T12:42:03.746 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:03 smithi027 ceph-mon[39795]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:03.746 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:03 smithi027 ceph-mon[39795]: Detected new or changed devices on smithi027 2024-05-15T12:42:03.746 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:03 smithi027 ceph-mon[39795]: Adjusting osd_memory_target on smithi027 to 5707M 2024-05-15T12:42:03.747 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:03 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:42:03.747 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:03 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:42:03.747 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:03 smithi027 ceph-mon[39795]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T12:42:03.747 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:03 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:42:03.747 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:03 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:42:03.747 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:03 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:42:03.747 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:03 smithi027 ceph-mon[39795]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T12:42:03.747 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:03 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:42:03.747 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:03 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:42:03.747 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:03 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:42:03.747 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:03 smithi027 ceph-mon[39795]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T12:42:03.747 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:03 smithi027 ceph-mon[39795]: osdmap e22: 3 total, 3 up, 3 in 2024-05-15T12:42:03.747 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:03 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:03.747 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:03 smithi027 ceph-mon[39795]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T12:42:03.748 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:03 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:42:03.748 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:03 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:42:03.748 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:03 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:42:03.748 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:03 smithi027 ceph-mon[39795]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T12:42:04.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 sudo[46451]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-15T12:42:04.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 sudo[46451]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T12:42:04.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 sudo[46451]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T12:42:04.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 sudo[46451]: pam_unix(sudo:session): session closed for user root 2024-05-15T12:42:04.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 ceph-mon[43961]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:04.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 ceph-mon[43961]: Detected new or changed devices on smithi027 2024-05-15T12:42:04.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 ceph-mon[43961]: Adjusting osd_memory_target on smithi027 to 5707M 2024-05-15T12:42:04.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:42:04.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:42:04.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 ceph-mon[43961]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T12:42:04.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:42:04.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:42:04.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:42:04.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 ceph-mon[43961]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T12:42:04.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:42:04.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:42:04.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:42:04.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 ceph-mon[43961]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T12:42:04.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 ceph-mon[43961]: osdmap e22: 3 total, 3 up, 3 in 2024-05-15T12:42:04.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:04.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 ceph-mon[43961]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T12:42:04.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:42:04.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:42:04.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:42:04.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:03 smithi120 ceph-mon[43961]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T12:42:04.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 ceph-mon[44303]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:04.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 ceph-mon[44303]: Detected new or changed devices on smithi027 2024-05-15T12:42:04.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 ceph-mon[44303]: Adjusting osd_memory_target on smithi027 to 5707M 2024-05-15T12:42:04.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:42:04.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:42:04.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 ceph-mon[44303]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T12:42:04.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:42:04.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:42:04.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:42:04.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 ceph-mon[44303]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T12:42:04.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:42:04.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:42:04.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:42:04.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 ceph-mon[44303]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T12:42:04.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 ceph-mon[44303]: osdmap e22: 3 total, 3 up, 3 in 2024-05-15T12:42:04.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:04.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 ceph-mon[44303]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T12:42:04.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:42:04.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:42:04.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:42:04.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:03 smithi177 ceph-mon[44303]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T12:42:05.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:04 smithi027 ceph-mon[39795]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:05.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:04 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:05.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:04 smithi027 ceph-mon[39795]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:42:05.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:04 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:42:05.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:04 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:42:05.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:04 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:05.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:04 smithi120 ceph-mon[43961]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:05.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:04 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:05.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:04 smithi120 ceph-mon[43961]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:42:05.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:04 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:42:05.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:04 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:42:05.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:04 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:05.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:04 smithi177 ceph-mon[44303]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:05.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:04 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:05.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:04 smithi177 ceph-mon[44303]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:42:05.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:04 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:42:05.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:04 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:42:05.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:04 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:07.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:06 smithi027 ceph-mon[39795]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:07.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:06 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2583044739' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e83735b-b944-4f87-8080-b994d4b96897"}]: dispatch 2024-05-15T12:42:07.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:06 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2583044739' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e83735b-b944-4f87-8080-b994d4b96897"}]': finished 2024-05-15T12:42:07.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:06 smithi027 ceph-mon[39795]: osdmap e23: 4 total, 3 up, 4 in 2024-05-15T12:42:07.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:06 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:42:07.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:06 smithi120 ceph-mon[43961]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:07.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:06 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2583044739' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e83735b-b944-4f87-8080-b994d4b96897"}]: dispatch 2024-05-15T12:42:07.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:06 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2583044739' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e83735b-b944-4f87-8080-b994d4b96897"}]': finished 2024-05-15T12:42:07.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:06 smithi120 ceph-mon[43961]: osdmap e23: 4 total, 3 up, 4 in 2024-05-15T12:42:07.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:06 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:42:07.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:06 smithi177 ceph-mon[44303]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:07.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:06 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2583044739' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e83735b-b944-4f87-8080-b994d4b96897"}]: dispatch 2024-05-15T12:42:07.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:06 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2583044739' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e83735b-b944-4f87-8080-b994d4b96897"}]': finished 2024-05-15T12:42:07.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:06 smithi177 ceph-mon[44303]: osdmap e23: 4 total, 3 up, 4 in 2024-05-15T12:42:07.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:06 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:42:08.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:07 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3945017897' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:42:08.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:07 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3945017897' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:42:08.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:07 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3945017897' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:42:09.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:08 smithi027 ceph-mon[39795]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-05-15T12:42:09.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:08 smithi120 ceph-mon[43961]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-05-15T12:42:09.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:08 smithi177 ceph-mon[44303]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-05-15T12:42:11.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:10 smithi120 ceph-mon[43961]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 10 op/s 2024-05-15T12:42:11.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:10 smithi177 ceph-mon[44303]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 10 op/s 2024-05-15T12:42:11.225 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:10 smithi027 ceph-mon[39795]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 10 op/s 2024-05-15T12:42:13.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:12 smithi120 ceph-mon[43961]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 159 KiB/s wr, 10 op/s 2024-05-15T12:42:13.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:12 smithi177 ceph-mon[44303]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 159 KiB/s wr, 10 op/s 2024-05-15T12:42:13.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:12 smithi027 ceph-mon[39795]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 159 KiB/s wr, 10 op/s 2024-05-15T12:42:14.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:13 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-15T12:42:14.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:13 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:14.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:13 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-15T12:42:14.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:13 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:14.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:13 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-15T12:42:14.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:13 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:15.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:14 smithi120 ceph-mon[43961]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-05-15T12:42:15.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:14 smithi120 ceph-mon[43961]: Deploying daemon osd.3 on smithi027 2024-05-15T12:42:15.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:14 smithi177 ceph-mon[44303]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-05-15T12:42:15.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:14 smithi177 ceph-mon[44303]: Deploying daemon osd.3 on smithi027 2024-05-15T12:42:15.218 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:14 smithi027 ceph-mon[39795]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-05-15T12:42:15.218 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:14 smithi027 ceph-mon[39795]: Deploying daemon osd.3 on smithi027 2024-05-15T12:42:17.117 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:16 smithi027 ceph-mon[39795]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-05-15T12:42:17.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:16 smithi177 ceph-mon[44303]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-05-15T12:42:17.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:16 smithi120 ceph-mon[43961]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-05-15T12:42:19.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:18 smithi177 ceph-mon[44303]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:19.226 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:18 smithi027 ceph-mon[39795]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:19.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:18 smithi120 ceph-mon[43961]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:20.704 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:20 smithi027 ceph-mon[39795]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:20.704 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:20 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:20.705 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:20 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:20.705 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:20 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:42:20.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:20 smithi120 ceph-mon[43961]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:20.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:20 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:20.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:20 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:20.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:20 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:42:20.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:20 smithi177 ceph-mon[44303]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:20.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:20 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:20.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:20 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:20.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:20 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:42:22.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:22 smithi027 ceph-mon[39795]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:22.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:22 smithi120 ceph-mon[43961]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:22.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:22 smithi177 ceph-mon[44303]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:23.403 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 3 on host 'smithi027' 2024-05-15T12:42:23.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:23.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:23.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:23.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:42:23.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:23.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:23 smithi027 ceph-mon[39795]: from='osd.3 [v2:172.21.15.27:6826/3008975244,v1:172.21.15.27:6827/3008975244]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-15T12:42:23.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:23.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:23.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:42:23.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:23.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:23.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:23.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:42:23.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:23.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:23 smithi120 ceph-mon[43961]: from='osd.3 [v2:172.21.15.27:6826/3008975244,v1:172.21.15.27:6827/3008975244]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-15T12:42:23.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:23.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:23.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:42:23.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:23.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:23.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:23.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:42:23.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:23.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:23 smithi177 ceph-mon[44303]: from='osd.3 [v2:172.21.15.27:6826/3008975244,v1:172.21.15.27:6827/3008975244]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-15T12:42:23.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:23.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:23.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:42:24.111 DEBUG:teuthology.orchestra.run.smithi027:osd.3> sudo journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.3.service 2024-05-15T12:42:24.113 INFO:tasks.cephadm:Deploying osd.4 on smithi120 with /dev/vg_nvme/lv_4... 2024-05-15T12:42:24.114 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-15T12:42:24.146 INFO:journalctl@ceph.osd.3.smithi027.stdout:-- Logs begin at Wed 2024-05-15 12:27:35 UTC. -- 2024-05-15T12:42:24.381 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:42:24.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:24 smithi027 ceph-mon[39795]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:24.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:24 smithi027 ceph-mon[39795]: from='osd.3 [v2:172.21.15.27:6826/3008975244,v1:172.21.15.27:6827/3008975244]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-15T12:42:24.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:24 smithi027 ceph-mon[39795]: osdmap e24: 4 total, 3 up, 4 in 2024-05-15T12:42:24.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:24 smithi027 ceph-mon[39795]: from='osd.3 [v2:172.21.15.27:6826/3008975244,v1:172.21.15.27:6827/3008975244]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-15T12:42:24.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:24 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:42:24.892 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:24 smithi120 ceph-mon[43961]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:24.893 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:24 smithi120 ceph-mon[43961]: from='osd.3 [v2:172.21.15.27:6826/3008975244,v1:172.21.15.27:6827/3008975244]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-15T12:42:24.893 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:24 smithi120 ceph-mon[43961]: osdmap e24: 4 total, 3 up, 4 in 2024-05-15T12:42:24.893 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:24 smithi120 ceph-mon[43961]: from='osd.3 [v2:172.21.15.27:6826/3008975244,v1:172.21.15.27:6827/3008975244]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-15T12:42:24.893 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:42:24.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:24 smithi177 ceph-mon[44303]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:24.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:24 smithi177 ceph-mon[44303]: from='osd.3 [v2:172.21.15.27:6826/3008975244,v1:172.21.15.27:6827/3008975244]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-15T12:42:24.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:24 smithi177 ceph-mon[44303]: osdmap e24: 4 total, 3 up, 4 in 2024-05-15T12:42:24.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:24 smithi177 ceph-mon[44303]: from='osd.3 [v2:172.21.15.27:6826/3008975244,v1:172.21.15.27:6827/3008975244]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-05-15T12:42:24.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:24 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:42:25.188 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 15 12:42:24 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-3[81304]: 2024-05-15T12:42:24.945+0000 7fd2b3030700 -1 osd.3 0 waiting for initial osdmap 2024-05-15T12:42:25.188 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 15 12:42:24 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-3[81304]: 2024-05-15T12:42:24.949+0000 7fd2aae1b700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:42:25.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:25 smithi120 ceph-mon[43961]: from='osd.3 [v2:172.21.15.27:6826/3008975244,v1:172.21.15.27:6827/3008975244]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-05-15T12:42:25.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:25 smithi120 ceph-mon[43961]: osdmap e25: 4 total, 3 up, 4 in 2024-05-15T12:42:25.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:25 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:42:25.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:25 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:42:25.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:25 smithi120 ceph-mon[43961]: from='osd.3 [v2:172.21.15.27:6826/3008975244,v1:172.21.15.27:6827/3008975244]' entity='osd.3' 2024-05-15T12:42:25.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:25 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:42:25.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:25 smithi027 ceph-mon[39795]: from='osd.3 [v2:172.21.15.27:6826/3008975244,v1:172.21.15.27:6827/3008975244]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-05-15T12:42:25.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:25 smithi027 ceph-mon[39795]: osdmap e25: 4 total, 3 up, 4 in 2024-05-15T12:42:25.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:25 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:42:25.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:25 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:42:25.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:25 smithi027 ceph-mon[39795]: from='osd.3 [v2:172.21.15.27:6826/3008975244,v1:172.21.15.27:6827/3008975244]' entity='osd.3' 2024-05-15T12:42:25.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:25 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:42:25.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:25 smithi177 ceph-mon[44303]: from='osd.3 [v2:172.21.15.27:6826/3008975244,v1:172.21.15.27:6827/3008975244]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-05-15T12:42:25.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:25 smithi177 ceph-mon[44303]: osdmap e25: 4 total, 3 up, 4 in 2024-05-15T12:42:25.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:25 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:42:25.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:25 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:42:25.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:25 smithi177 ceph-mon[44303]: from='osd.3 [v2:172.21.15.27:6826/3008975244,v1:172.21.15.27:6827/3008975244]' entity='osd.3' 2024-05-15T12:42:25.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:25 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:42:26.672 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-15T12:42:26.672 INFO:teuthology.orchestra.run.smithi120.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-15T12:42:26.672 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10+0 records in 2024-05-15T12:42:26.672 INFO:teuthology.orchestra.run.smithi120.stderr:10+0 records out 2024-05-15T12:42:26.672 INFO:teuthology.orchestra.run.smithi120.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0143443 s, 731 MB/s 2024-05-15T12:42:26.672 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 2024-05-15T12:42:26.673 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping successful for: 2024-05-15T12:42:26.907 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:26 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:42:26.907 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:26 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:42:26.908 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:26 smithi027 ceph-mon[39795]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:26.908 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:26 smithi027 ceph-mon[39795]: osd.3 [v2:172.21.15.27:6826/3008975244,v1:172.21.15.27:6827/3008975244] boot 2024-05-15T12:42:26.908 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:26 smithi027 ceph-mon[39795]: osdmap e26: 4 total, 4 up, 4 in 2024-05-15T12:42:26.908 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:26 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:42:26.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:26 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:42:26.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:26 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:42:26.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:26 smithi177 ceph-mon[44303]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:26.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:26 smithi177 ceph-mon[44303]: osd.3 [v2:172.21.15.27:6826/3008975244,v1:172.21.15.27:6827/3008975244] boot 2024-05-15T12:42:26.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:26 smithi177 ceph-mon[44303]: osdmap e26: 4 total, 4 up, 4 in 2024-05-15T12:42:26.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:26 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:42:26.993 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:26 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:42:26.993 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:26 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:42:26.993 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:26 smithi120 ceph-mon[43961]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:42:26.993 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:26 smithi120 ceph-mon[43961]: osd.3 [v2:172.21.15.27:6826/3008975244,v1:172.21.15.27:6827/3008975244] boot 2024-05-15T12:42:26.993 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:26 smithi120 ceph-mon[43961]: osdmap e26: 4 total, 4 up, 4 in 2024-05-15T12:42:26.993 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:26 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:42:27.249 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi120:vg_nvme/lv_4 2024-05-15T12:42:27.507 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:42:28.706 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:28 smithi120 ceph-mon[43961]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:28.706 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:28 smithi120 ceph-mon[43961]: osdmap e27: 4 total, 4 up, 4 in 2024-05-15T12:42:28.706 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:28 smithi120 ceph-mon[43961]: Detected new or changed devices on smithi027 2024-05-15T12:42:28.706 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:28.706 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:28.706 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:42:28.706 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:42:28.706 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:42:28.707 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:42:28.707 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:28 smithi120 ceph-mon[43961]: Adjusting osd_memory_target on smithi027 to 4280M 2024-05-15T12:42:28.707 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:28.707 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:28.707 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:42:28.707 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:28.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:28 smithi177 ceph-mon[44303]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:28.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:28 smithi177 ceph-mon[44303]: osdmap e27: 4 total, 4 up, 4 in 2024-05-15T12:42:28.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:28 smithi177 ceph-mon[44303]: Detected new or changed devices on smithi027 2024-05-15T12:42:28.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:28.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:28.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:42:28.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:42:28.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:42:28.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:42:28.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:28 smithi177 ceph-mon[44303]: Adjusting osd_memory_target on smithi027 to 4280M 2024-05-15T12:42:28.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:28.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:28.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:42:28.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:29.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:28 smithi027 ceph-mon[39795]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:29.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:28 smithi027 ceph-mon[39795]: osdmap e27: 4 total, 4 up, 4 in 2024-05-15T12:42:29.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:28 smithi027 ceph-mon[39795]: Detected new or changed devices on smithi027 2024-05-15T12:42:29.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:29.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:29.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:42:29.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:42:29.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:42:29.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:42:29.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:28 smithi027 ceph-mon[39795]: Adjusting osd_memory_target on smithi027 to 4280M 2024-05-15T12:42:29.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:29.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:29.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:42:29.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:29.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:29 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:42:29.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:29 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:42:29.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:29 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:29.970 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:29 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:42:29.970 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:29 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:42:29.970 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:29 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:30.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:29 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:42:30.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:29 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:42:30.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:29 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:30.872 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:30 smithi120 ceph-mon[43961]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:42:30.872 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:30 smithi120 ceph-mon[43961]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-05-15T12:42:30.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:30 smithi177 ceph-mon[44303]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:42:30.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:30 smithi177 ceph-mon[44303]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-05-15T12:42:31.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:30 smithi027 ceph-mon[39795]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:42:31.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:30 smithi027 ceph-mon[39795]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-05-15T12:42:32.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:32 smithi177 ceph-mon[44303]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-05-15T12:42:32.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:32 smithi177 ceph-mon[44303]: from='client.? 172.21.15.120:0/3516081809' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c39c1dd-3867-46f7-8372-7688fa13b16a"}]: dispatch 2024-05-15T12:42:32.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:32 smithi177 ceph-mon[44303]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c39c1dd-3867-46f7-8372-7688fa13b16a"}]: dispatch 2024-05-15T12:42:32.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:32 smithi177 ceph-mon[44303]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c39c1dd-3867-46f7-8372-7688fa13b16a"}]': finished 2024-05-15T12:42:32.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:32 smithi177 ceph-mon[44303]: osdmap e28: 5 total, 4 up, 5 in 2024-05-15T12:42:32.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:32 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:42:33.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:32 smithi027 ceph-mon[39795]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-05-15T12:42:33.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:32 smithi027 ceph-mon[39795]: from='client.? 172.21.15.120:0/3516081809' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c39c1dd-3867-46f7-8372-7688fa13b16a"}]: dispatch 2024-05-15T12:42:33.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:32 smithi027 ceph-mon[39795]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c39c1dd-3867-46f7-8372-7688fa13b16a"}]: dispatch 2024-05-15T12:42:33.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:32 smithi027 ceph-mon[39795]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c39c1dd-3867-46f7-8372-7688fa13b16a"}]': finished 2024-05-15T12:42:33.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:32 smithi027 ceph-mon[39795]: osdmap e28: 5 total, 4 up, 5 in 2024-05-15T12:42:33.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:32 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:42:33.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:32 smithi120 ceph-mon[43961]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-05-15T12:42:33.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:32 smithi120 ceph-mon[43961]: from='client.? 172.21.15.120:0/3516081809' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c39c1dd-3867-46f7-8372-7688fa13b16a"}]: dispatch 2024-05-15T12:42:33.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:32 smithi120 ceph-mon[43961]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c39c1dd-3867-46f7-8372-7688fa13b16a"}]: dispatch 2024-05-15T12:42:33.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:32 smithi120 ceph-mon[43961]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c39c1dd-3867-46f7-8372-7688fa13b16a"}]': finished 2024-05-15T12:42:33.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:32 smithi120 ceph-mon[43961]: osdmap e28: 5 total, 4 up, 5 in 2024-05-15T12:42:33.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:32 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:42:33.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:33 smithi177 ceph-mon[44303]: from='client.? 172.21.15.120:0/693591568' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:42:34.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:33 smithi027 ceph-mon[39795]: from='client.? 172.21.15.120:0/693591568' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:42:34.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:33 smithi120 ceph-mon[43961]: from='client.? 172.21.15.120:0/693591568' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:42:34.885 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:34 smithi120 ceph-mon[43961]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:34.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:34 smithi177 ceph-mon[44303]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:35.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:34 smithi027 ceph-mon[39795]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:36.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:36 smithi177 ceph-mon[44303]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:36.969 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:36 smithi120 ceph-mon[43961]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:37.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:36 smithi027 ceph-mon[39795]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:39.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:38 smithi027 ceph-mon[39795]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:39.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:38 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-15T12:42:39.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:38 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:39.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:38 smithi027 ceph-mon[39795]: Deploying daemon osd.4 on smithi120 2024-05-15T12:42:39.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:38 smithi120 ceph-mon[43961]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:39.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:38 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-15T12:42:39.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:38 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:39.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:38 smithi120 ceph-mon[43961]: Deploying daemon osd.4 on smithi120 2024-05-15T12:42:39.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:38 smithi177 ceph-mon[44303]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:39.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:38 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-15T12:42:39.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:38 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:39.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:38 smithi177 ceph-mon[44303]: Deploying daemon osd.4 on smithi120 2024-05-15T12:42:41.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:40 smithi027 ceph-mon[39795]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:41.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:40 smithi120 ceph-mon[43961]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:41.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:40 smithi177 ceph-mon[44303]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:43.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:42 smithi027 ceph-mon[39795]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:43.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:42 smithi120 ceph-mon[43961]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:43.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:42 smithi177 ceph-mon[44303]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:44.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:44 smithi177 ceph-mon[44303]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:44.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:44 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:44.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:44 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:44.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:44 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:42:45.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:44 smithi027 ceph-mon[39795]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:45.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:44 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:45.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:44 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:45.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:44 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:42:45.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:44 smithi120 ceph-mon[43961]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:45.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:44 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:45.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:44 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:45.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:44 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:42:47.121 INFO:teuthology.orchestra.run.smithi120.stdout:Created osd(s) 4 on host 'smithi120' 2024-05-15T12:42:47.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:47 smithi027 ceph-mon[39795]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:47.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:47 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:47.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:47 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:47.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:47 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:47.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:47 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:42:47.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:47 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:47.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:47 smithi027 ceph-mon[39795]: from='osd.4 [v2:172.21.15.120:6800/3877842957,v1:172.21.15.120:6801/3877842957]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-15T12:42:47.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:47 smithi027 ceph-mon[39795]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-15T12:42:47.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:47 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:47.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:47 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:47.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:47 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:42:47.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:47 smithi120 ceph-mon[43961]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:47.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:47 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:47.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:47 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:47.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:47 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:47.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:47 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:42:47.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:47 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:47.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:47 smithi120 ceph-mon[43961]: from='osd.4 [v2:172.21.15.120:6800/3877842957,v1:172.21.15.120:6801/3877842957]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-15T12:42:47.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:47 smithi120 ceph-mon[43961]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-15T12:42:47.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:47 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:47.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:47 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:47.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:47 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:42:47.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:47 smithi177 ceph-mon[44303]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:47.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:47 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:47.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:47 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:47.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:47 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:47.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:47 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:42:47.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:47 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:47.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:47 smithi177 ceph-mon[44303]: from='osd.4 [v2:172.21.15.120:6800/3877842957,v1:172.21.15.120:6801/3877842957]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-15T12:42:47.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:47 smithi177 ceph-mon[44303]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-15T12:42:47.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:47 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:47.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:47 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:47.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:47 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:42:47.858 DEBUG:teuthology.orchestra.run.smithi120:osd.4> sudo journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.4.service 2024-05-15T12:42:47.861 INFO:tasks.cephadm:Deploying osd.5 on smithi120 with /dev/vg_nvme/lv_3... 2024-05-15T12:42:47.861 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-15T12:42:47.889 INFO:journalctl@ceph.osd.4.smithi120.stdout:-- Logs begin at Wed 2024-05-15 12:26:58 UTC. -- 2024-05-15T12:42:48.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:48 smithi120 ceph-mon[43961]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-15T12:42:48.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:48 smithi120 ceph-mon[43961]: from='osd.4 [v2:172.21.15.120:6800/3877842957,v1:172.21.15.120:6801/3877842957]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-05-15T12:42:48.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:48 smithi120 ceph-mon[43961]: osdmap e29: 5 total, 4 up, 5 in 2024-05-15T12:42:48.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:48 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:42:48.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:48 smithi120 ceph-mon[43961]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-05-15T12:42:48.565 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:42:48.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:48 smithi177 ceph-mon[44303]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-15T12:42:48.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:48 smithi177 ceph-mon[44303]: from='osd.4 [v2:172.21.15.120:6800/3877842957,v1:172.21.15.120:6801/3877842957]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-05-15T12:42:48.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:48 smithi177 ceph-mon[44303]: osdmap e29: 5 total, 4 up, 5 in 2024-05-15T12:42:48.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:48 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:42:48.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:48 smithi177 ceph-mon[44303]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-05-15T12:42:48.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:48 smithi027 ceph-mon[39795]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-15T12:42:48.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:48 smithi027 ceph-mon[39795]: from='osd.4 [v2:172.21.15.120:6800/3877842957,v1:172.21.15.120:6801/3877842957]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-05-15T12:42:48.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:48 smithi027 ceph-mon[39795]: osdmap e29: 5 total, 4 up, 5 in 2024-05-15T12:42:48.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:48 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:42:48.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:48 smithi027 ceph-mon[39795]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-05-15T12:42:49.031 INFO:journalctl@ceph.osd.4.smithi120.stdout:May 15 12:42:48 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-4[49322]: 2024-05-15T12:42:48.715+0000 7f9fa83b2700 -1 osd.4 0 waiting for initial osdmap 2024-05-15T12:42:49.031 INFO:journalctl@ceph.osd.4.smithi120.stdout:May 15 12:42:48 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-4[49322]: 2024-05-15T12:42:48.720+0000 7f9fa019d700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:42:49.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:49 smithi177 ceph-mon[44303]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:49.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:49 smithi177 ceph-mon[44303]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-05-15T12:42:49.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:49 smithi177 ceph-mon[44303]: osdmap e30: 5 total, 4 up, 5 in 2024-05-15T12:42:49.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:49 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:42:49.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:49 smithi177 ceph-mon[44303]: from='osd.4 ' entity='osd.4' 2024-05-15T12:42:49.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:49 smithi177 ceph-mon[44303]: osd.4 [v2:172.21.15.120:6800/3877842957,v1:172.21.15.120:6801/3877842957] boot 2024-05-15T12:42:49.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:49 smithi177 ceph-mon[44303]: osdmap e31: 5 total, 5 up, 5 in 2024-05-15T12:42:49.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:49 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:42:49.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:49 smithi027 ceph-mon[39795]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:49.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:49 smithi027 ceph-mon[39795]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-05-15T12:42:49.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:49 smithi027 ceph-mon[39795]: osdmap e30: 5 total, 4 up, 5 in 2024-05-15T12:42:49.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:49 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:42:49.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:49 smithi027 ceph-mon[39795]: from='osd.4 ' entity='osd.4' 2024-05-15T12:42:49.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:49 smithi027 ceph-mon[39795]: osd.4 [v2:172.21.15.120:6800/3877842957,v1:172.21.15.120:6801/3877842957] boot 2024-05-15T12:42:49.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:49 smithi027 ceph-mon[39795]: osdmap e31: 5 total, 5 up, 5 in 2024-05-15T12:42:49.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:49 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:42:49.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:49 smithi120 ceph-mon[43961]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:42:49.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:49 smithi120 ceph-mon[43961]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-05-15T12:42:49.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:49 smithi120 ceph-mon[43961]: osdmap e30: 5 total, 4 up, 5 in 2024-05-15T12:42:49.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:49 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:42:49.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:49 smithi120 ceph-mon[43961]: from='osd.4 ' entity='osd.4' 2024-05-15T12:42:49.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:49 smithi120 ceph-mon[43961]: osd.4 [v2:172.21.15.120:6800/3877842957,v1:172.21.15.120:6801/3877842957] boot 2024-05-15T12:42:49.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:49 smithi120 ceph-mon[43961]: osdmap e31: 5 total, 5 up, 5 in 2024-05-15T12:42:49.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:49 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:42:50.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:50 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:42:50.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:50 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:42:50.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:50 smithi177 ceph-mon[44303]: osdmap e32: 5 total, 5 up, 5 in 2024-05-15T12:42:50.714 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:50 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:42:50.714 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:50 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:42:50.714 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:50 smithi120 ceph-mon[43961]: osdmap e32: 5 total, 5 up, 5 in 2024-05-15T12:42:50.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:50 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:42:50.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:50 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:42:50.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:50 smithi027 ceph-mon[39795]: osdmap e32: 5 total, 5 up, 5 in 2024-05-15T12:42:51.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:51 smithi177 ceph-mon[44303]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:42:51.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:51 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:42:51.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:51 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:42:51.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:51 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:51.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:51 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:51.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:51 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:42:51.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:51 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:51.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:51 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:51.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:51 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:42:51.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:51 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:51.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:51 smithi177 ceph-mon[44303]: osdmap e33: 5 total, 5 up, 5 in 2024-05-15T12:42:51.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:51 smithi027 ceph-mon[39795]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:42:51.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:51 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:42:51.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:51 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:42:51.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:51 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:51.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:51 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:51.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:51 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:42:51.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:51 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:51.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:51 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:51.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:51 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:42:51.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:51 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:51.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:51 smithi027 ceph-mon[39795]: osdmap e33: 5 total, 5 up, 5 in 2024-05-15T12:42:51.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:51 smithi120 ceph-mon[43961]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:42:51.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:51 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:42:51.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:51 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:42:51.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:51 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:51.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:51 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:51.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:51 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:42:51.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:51 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:51.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:51 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:51.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:51 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:42:51.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:51 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:42:51.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:51 smithi120 ceph-mon[43961]: osdmap e33: 5 total, 5 up, 5 in 2024-05-15T12:42:52.476 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-15T12:42:52.477 INFO:teuthology.orchestra.run.smithi120.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-15T12:42:52.477 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10+0 records in 2024-05-15T12:42:52.477 INFO:teuthology.orchestra.run.smithi120.stderr:10+0 records out 2024-05-15T12:42:52.477 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0134282 s, 781 MB/s 2024-05-15T12:42:52.477 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping successful for: 2024-05-15T12:42:52.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:52 smithi177 ceph-mon[44303]: Detected new or changed devices on smithi120 2024-05-15T12:42:52.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:52 smithi177 ceph-mon[44303]: Adjusting osd_memory_target on smithi120 to 17122M 2024-05-15T12:42:52.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:52 smithi177 ceph-mon[44303]: osdmap e34: 5 total, 5 up, 5 in 2024-05-15T12:42:52.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:52 smithi027 ceph-mon[39795]: Detected new or changed devices on smithi120 2024-05-15T12:42:52.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:52 smithi027 ceph-mon[39795]: Adjusting osd_memory_target on smithi120 to 17122M 2024-05-15T12:42:52.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:52 smithi027 ceph-mon[39795]: osdmap e34: 5 total, 5 up, 5 in 2024-05-15T12:42:52.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:52 smithi120 ceph-mon[43961]: Detected new or changed devices on smithi120 2024-05-15T12:42:52.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:52 smithi120 ceph-mon[43961]: Adjusting osd_memory_target on smithi120 to 17122M 2024-05-15T12:42:52.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:52 smithi120 ceph-mon[43961]: osdmap e34: 5 total, 5 up, 5 in 2024-05-15T12:42:53.057 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi120:vg_nvme/lv_3 2024-05-15T12:42:53.318 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:42:53.588 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:53 smithi120 ceph-mon[43961]: pgmap v129: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:42:53.588 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:53 smithi120 ceph-mon[43961]: osdmap e35: 5 total, 5 up, 5 in 2024-05-15T12:42:53.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:53 smithi177 ceph-mon[44303]: pgmap v129: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:42:53.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:53 smithi177 ceph-mon[44303]: osdmap e35: 5 total, 5 up, 5 in 2024-05-15T12:42:53.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:53 smithi027 ceph-mon[39795]: pgmap v129: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:42:53.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:53 smithi027 ceph-mon[39795]: osdmap e35: 5 total, 5 up, 5 in 2024-05-15T12:42:55.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:55 smithi177 ceph-mon[44303]: pgmap v132: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:42:55.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:55 smithi027 ceph-mon[39795]: pgmap v132: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:42:55.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:55 smithi120 ceph-mon[43961]: pgmap v132: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:42:56.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:56 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:42:56.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:56 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:42:56.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:56 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:56.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:56 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:42:56.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:56 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:42:56.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:56 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:56.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:56 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:42:56.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:56 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:42:56.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:56 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:42:57.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:57 smithi120 ceph-mon[43961]: from='client.24236 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:42:57.477 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:57 smithi120 ceph-mon[43961]: pgmap v133: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:42:57.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:57 smithi177 ceph-mon[44303]: from='client.24236 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:42:57.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:57 smithi177 ceph-mon[44303]: pgmap v133: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:42:57.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:57 smithi027 ceph-mon[39795]: from='client.24236 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:42:57.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:57 smithi027 ceph-mon[39795]: pgmap v133: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:42:59.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:59 smithi177 ceph-mon[44303]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-05-15T12:42:59.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:59 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:42:59.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:59 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:42:59.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:59 smithi177 ceph-mon[44303]: osdmap e36: 5 total, 5 up, 5 in 2024-05-15T12:42:59.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:59 smithi177 ceph-mon[44303]: from='client.? 172.21.15.120:0/3525404732' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6dc220f-aec0-4454-9695-b44c0b9770e0"}]: dispatch 2024-05-15T12:42:59.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:59 smithi177 ceph-mon[44303]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6dc220f-aec0-4454-9695-b44c0b9770e0"}]: dispatch 2024-05-15T12:42:59.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:59 smithi177 ceph-mon[44303]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a6dc220f-aec0-4454-9695-b44c0b9770e0"}]': finished 2024-05-15T12:42:59.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:59 smithi177 ceph-mon[44303]: osdmap e37: 6 total, 5 up, 6 in 2024-05-15T12:42:59.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:59 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:42:59.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:42:59 smithi177 ceph-mon[44303]: from='client.? 172.21.15.120:0/2858055039' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:42:59.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:59 smithi027 ceph-mon[39795]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-05-15T12:42:59.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:59 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:42:59.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:59 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:42:59.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:59 smithi027 ceph-mon[39795]: osdmap e36: 5 total, 5 up, 5 in 2024-05-15T12:42:59.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:59 smithi027 ceph-mon[39795]: from='client.? 172.21.15.120:0/3525404732' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6dc220f-aec0-4454-9695-b44c0b9770e0"}]: dispatch 2024-05-15T12:42:59.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:59 smithi027 ceph-mon[39795]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6dc220f-aec0-4454-9695-b44c0b9770e0"}]: dispatch 2024-05-15T12:42:59.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:59 smithi027 ceph-mon[39795]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a6dc220f-aec0-4454-9695-b44c0b9770e0"}]': finished 2024-05-15T12:42:59.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:59 smithi027 ceph-mon[39795]: osdmap e37: 6 total, 5 up, 6 in 2024-05-15T12:42:59.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:59 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:42:59.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:42:59 smithi027 ceph-mon[39795]: from='client.? 172.21.15.120:0/2858055039' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:42:59.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:59 smithi120 ceph-mon[43961]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-05-15T12:42:59.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:59 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:42:59.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:59 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:42:59.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:59 smithi120 ceph-mon[43961]: osdmap e36: 5 total, 5 up, 5 in 2024-05-15T12:42:59.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:59 smithi120 ceph-mon[43961]: from='client.? 172.21.15.120:0/3525404732' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6dc220f-aec0-4454-9695-b44c0b9770e0"}]: dispatch 2024-05-15T12:42:59.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:59 smithi120 ceph-mon[43961]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6dc220f-aec0-4454-9695-b44c0b9770e0"}]: dispatch 2024-05-15T12:42:59.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:59 smithi120 ceph-mon[43961]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a6dc220f-aec0-4454-9695-b44c0b9770e0"}]': finished 2024-05-15T12:42:59.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:59 smithi120 ceph-mon[43961]: osdmap e37: 6 total, 5 up, 6 in 2024-05-15T12:42:59.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:59 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:42:59.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:42:59 smithi120 ceph-mon[43961]: from='client.? 172.21.15.120:0/2858055039' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:43:01.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:01 smithi120 ceph-mon[43961]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-05-15T12:43:01.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:01 smithi177 ceph-mon[44303]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-05-15T12:43:01.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:01 smithi027 ceph-mon[39795]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-05-15T12:43:03.668 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:03 smithi120 ceph-mon[43961]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-15T12:43:03.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:03 smithi177 ceph-mon[44303]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-15T12:43:03.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:03 smithi027 ceph-mon[39795]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-15T12:43:05.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:05 smithi120 ceph-mon[43961]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-15T12:43:05.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:05 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-15T12:43:05.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:05 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:05.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:05 smithi177 ceph-mon[44303]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-15T12:43:05.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:05 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-15T12:43:05.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:05 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:05.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:05 smithi027 ceph-mon[39795]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-15T12:43:05.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:05 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-15T12:43:05.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:05 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:06.597 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:06 smithi120 ceph-mon[43961]: Deploying daemon osd.5 on smithi120 2024-05-15T12:43:06.598 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:06 smithi120 ceph-mon[43961]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:43:06.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:06 smithi177 ceph-mon[44303]: Deploying daemon osd.5 on smithi120 2024-05-15T12:43:06.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:06 smithi177 ceph-mon[44303]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:43:06.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:06 smithi027 ceph-mon[39795]: Deploying daemon osd.5 on smithi120 2024-05-15T12:43:06.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:06 smithi027 ceph-mon[39795]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:43:08.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:08 smithi120 ceph-mon[43961]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:43:08.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:08 smithi027 ceph-mon[39795]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:43:08.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:08 smithi177 ceph-mon[44303]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:43:10.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:10 smithi027 ceph-mon[39795]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:43:10.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:10 smithi120 ceph-mon[43961]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:43:10.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:10 smithi177 ceph-mon[44303]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:43:12.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:11 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:12.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:11 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:12.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:11 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:43:12.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:11 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:12.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:11 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:12.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:11 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:43:12.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:11 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:12.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:11 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:12.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:11 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:43:12.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:12 smithi120 ceph-mon[43961]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:43:13.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:12 smithi027 ceph-mon[39795]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:43:13.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:12 smithi177 ceph-mon[44303]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:43:14.438 INFO:teuthology.orchestra.run.smithi120.stdout:Created osd(s) 5 on host 'smithi120' 2024-05-15T12:43:14.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:14.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:14.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:14.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:43:14.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:43:14.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:43:14.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:43:14.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:14.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:43:14.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:14.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:43:14.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:14.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:14 smithi027 ceph-mon[39795]: from='osd.5 [v2:172.21.15.120:6808/1600179183,v1:172.21.15.120:6809/1600179183]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-15T12:43:14.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:14.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:14.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:43:14.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:14.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:14.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:14.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:43:14.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:43:14.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:43:14.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:43:14.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:14.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:43:14.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:14.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:43:14.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:14.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:14 smithi120 ceph-mon[43961]: from='osd.5 [v2:172.21.15.120:6808/1600179183,v1:172.21.15.120:6809/1600179183]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-15T12:43:14.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:14.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:14.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:43:14.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:14.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:14.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:14.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:43:14.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:43:14.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:43:14.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:43:14.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:14.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:43:14.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:14.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:43:14.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:14.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:14 smithi177 ceph-mon[44303]: from='osd.5 [v2:172.21.15.120:6808/1600179183,v1:172.21.15.120:6809/1600179183]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-15T12:43:14.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:14.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:14.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:43:15.129 DEBUG:teuthology.orchestra.run.smithi120:osd.5> sudo journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.5.service 2024-05-15T12:43:15.132 INFO:tasks.cephadm:Deploying osd.6 on smithi120 with /dev/vg_nvme/lv_2... 2024-05-15T12:43:15.132 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-15T12:43:15.162 INFO:journalctl@ceph.osd.5.smithi120.stdout:-- Logs begin at Wed 2024-05-15 12:26:58 UTC. -- 2024-05-15T12:43:15.854 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:43:15.974 INFO:journalctl@ceph.osd.5.smithi120.stdout:May 15 12:43:15 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-5[56546]: 2024-05-15T12:43:15.732+0000 7f0f49721700 -1 osd.5 0 waiting for initial osdmap 2024-05-15T12:43:15.974 INFO:journalctl@ceph.osd.5.smithi120.stdout:May 15 12:43:15 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-5[56546]: 2024-05-15T12:43:15.737+0000 7f0f45514700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:43:15.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:15 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:43:15.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:15 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:43:15.975 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:15 smithi120 ceph-mon[43961]: from='osd.5 [v2:172.21.15.120:6808/1600179183,v1:172.21.15.120:6809/1600179183]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-15T12:43:15.975 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:15 smithi120 ceph-mon[43961]: osdmap e38: 6 total, 5 up, 6 in 2024-05-15T12:43:15.975 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:15 smithi120 ceph-mon[43961]: from='osd.5 [v2:172.21.15.120:6808/1600179183,v1:172.21.15.120:6809/1600179183]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-05-15T12:43:15.975 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:15 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:43:16.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:15 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:43:16.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:15 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:43:16.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:15 smithi027 ceph-mon[39795]: from='osd.5 [v2:172.21.15.120:6808/1600179183,v1:172.21.15.120:6809/1600179183]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-15T12:43:16.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:15 smithi027 ceph-mon[39795]: osdmap e38: 6 total, 5 up, 6 in 2024-05-15T12:43:16.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:15 smithi027 ceph-mon[39795]: from='osd.5 [v2:172.21.15.120:6808/1600179183,v1:172.21.15.120:6809/1600179183]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-05-15T12:43:16.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:15 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:43:16.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:15 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:43:16.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:15 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:43:16.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:15 smithi177 ceph-mon[44303]: from='osd.5 [v2:172.21.15.120:6808/1600179183,v1:172.21.15.120:6809/1600179183]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-15T12:43:16.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:15 smithi177 ceph-mon[44303]: osdmap e38: 6 total, 5 up, 6 in 2024-05-15T12:43:16.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:15 smithi177 ceph-mon[44303]: from='osd.5 [v2:172.21.15.120:6808/1600179183,v1:172.21.15.120:6809/1600179183]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-05-15T12:43:16.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:15 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:43:16.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:16 smithi120 ceph-mon[43961]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:43:16.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:16 smithi120 ceph-mon[43961]: from='osd.5 [v2:172.21.15.120:6808/1600179183,v1:172.21.15.120:6809/1600179183]' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-05-15T12:43:16.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:16 smithi120 ceph-mon[43961]: osdmap e39: 6 total, 5 up, 6 in 2024-05-15T12:43:16.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:16 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:43:16.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:16 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:43:17.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:16 smithi027 ceph-mon[39795]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:43:17.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:16 smithi027 ceph-mon[39795]: from='osd.5 [v2:172.21.15.120:6808/1600179183,v1:172.21.15.120:6809/1600179183]' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-05-15T12:43:17.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:16 smithi027 ceph-mon[39795]: osdmap e39: 6 total, 5 up, 6 in 2024-05-15T12:43:17.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:16 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:43:17.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:16 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:43:17.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:16 smithi177 ceph-mon[44303]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:43:17.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:16 smithi177 ceph-mon[44303]: from='osd.5 [v2:172.21.15.120:6808/1600179183,v1:172.21.15.120:6809/1600179183]' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-05-15T12:43:17.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:16 smithi177 ceph-mon[44303]: osdmap e39: 6 total, 5 up, 6 in 2024-05-15T12:43:17.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:16 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:43:17.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:16 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:43:17.949 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:17 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:43:17.949 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:17 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:43:17.949 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:17 smithi120 ceph-mon[43961]: osd.5 [v2:172.21.15.120:6808/1600179183,v1:172.21.15.120:6809/1600179183] boot 2024-05-15T12:43:17.949 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:17 smithi120 ceph-mon[43961]: osdmap e40: 6 total, 6 up, 6 in 2024-05-15T12:43:17.949 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:17 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:43:18.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:17 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:43:18.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:17 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:43:18.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:17 smithi027 ceph-mon[39795]: osd.5 [v2:172.21.15.120:6808/1600179183,v1:172.21.15.120:6809/1600179183] boot 2024-05-15T12:43:18.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:17 smithi027 ceph-mon[39795]: osdmap e40: 6 total, 6 up, 6 in 2024-05-15T12:43:18.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:17 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:43:18.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:17 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:43:18.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:17 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:43:18.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:17 smithi177 ceph-mon[44303]: osd.5 [v2:172.21.15.120:6808/1600179183,v1:172.21.15.120:6809/1600179183] boot 2024-05-15T12:43:18.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:17 smithi177 ceph-mon[44303]: osdmap e40: 6 total, 6 up, 6 in 2024-05-15T12:43:18.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:17 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:43:19.011 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-15T12:43:19.011 INFO:teuthology.orchestra.run.smithi120.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-15T12:43:19.011 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10+0 records in 2024-05-15T12:43:19.011 INFO:teuthology.orchestra.run.smithi120.stderr:10+0 records out 2024-05-15T12:43:19.011 INFO:teuthology.orchestra.run.smithi120.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0121085 s, 866 MB/s 2024-05-15T12:43:19.011 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping successful for: 2024-05-15T12:43:19.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:18 smithi027 ceph-mon[39795]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:43:19.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:18 smithi027 ceph-mon[39795]: osdmap e41: 6 total, 6 up, 6 in 2024-05-15T12:43:19.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:18 smithi027 ceph-mon[39795]: osdmap e42: 6 total, 6 up, 6 in 2024-05-15T12:43:19.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:18 smithi120 ceph-mon[43961]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:43:19.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:18 smithi120 ceph-mon[43961]: osdmap e41: 6 total, 6 up, 6 in 2024-05-15T12:43:19.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:18 smithi120 ceph-mon[43961]: osdmap e42: 6 total, 6 up, 6 in 2024-05-15T12:43:19.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:18 smithi177 ceph-mon[44303]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:43:19.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:18 smithi177 ceph-mon[44303]: osdmap e41: 6 total, 6 up, 6 in 2024-05-15T12:43:19.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:18 smithi177 ceph-mon[44303]: osdmap e42: 6 total, 6 up, 6 in 2024-05-15T12:43:19.811 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi120:vg_nvme/lv_2 2024-05-15T12:43:20.070 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:43:20.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:20 smithi027 ceph-mon[39795]: Detected new or changed devices on smithi120 2024-05-15T12:43:20.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:20 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:20.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:20 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:20.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:20 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:43:20.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:20 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:43:20.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:20 smithi027 ceph-mon[39795]: Adjusting osd_memory_target on smithi120 to 8561M 2024-05-15T12:43:20.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:20 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:20.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:20 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:20.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:20 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:43:20.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:20 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:20.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:20 smithi027 ceph-mon[39795]: osdmap e43: 6 total, 6 up, 6 in 2024-05-15T12:43:20.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:20 smithi120 ceph-mon[43961]: Detected new or changed devices on smithi120 2024-05-15T12:43:20.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:20 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:20.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:20 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:20.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:20 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:43:20.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:20 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:43:20.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:20 smithi120 ceph-mon[43961]: Adjusting osd_memory_target on smithi120 to 8561M 2024-05-15T12:43:20.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:20 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:20.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:20 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:20.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:20 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:43:20.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:20 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:20.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:20 smithi120 ceph-mon[43961]: osdmap e43: 6 total, 6 up, 6 in 2024-05-15T12:43:20.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:20 smithi177 ceph-mon[44303]: Detected new or changed devices on smithi120 2024-05-15T12:43:20.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:20 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:20.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:20 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:20.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:20 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:43:20.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:20 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:43:20.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:20 smithi177 ceph-mon[44303]: Adjusting osd_memory_target on smithi120 to 8561M 2024-05-15T12:43:20.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:20 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:20.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:20 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:20.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:20 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:43:20.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:20 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:20.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:20 smithi177 ceph-mon[44303]: osdmap e43: 6 total, 6 up, 6 in 2024-05-15T12:43:21.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:21 smithi120 ceph-mon[43961]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:43:21.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:21 smithi120 ceph-mon[43961]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-05-15T12:43:21.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:21 smithi120 ceph-mon[43961]: osdmap e44: 6 total, 6 up, 6 in 2024-05-15T12:43:21.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:21 smithi027 ceph-mon[39795]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:43:21.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:21 smithi027 ceph-mon[39795]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-05-15T12:43:21.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:21 smithi027 ceph-mon[39795]: osdmap e44: 6 total, 6 up, 6 in 2024-05-15T12:43:21.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:21 smithi177 ceph-mon[44303]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:43:21.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:21 smithi177 ceph-mon[44303]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-05-15T12:43:21.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:21 smithi177 ceph-mon[44303]: osdmap e44: 6 total, 6 up, 6 in 2024-05-15T12:43:22.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:43:22.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:43:22.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:22.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:22 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:43:22.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:22 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:43:22.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:22 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:22.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:22 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:43:22.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:22 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:43:22.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:22 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:23.496 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:23 smithi120 ceph-mon[43961]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:43:23.497 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:23 smithi120 ceph-mon[43961]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:43:23.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:23 smithi027 ceph-mon[39795]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:43:23.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:23 smithi027 ceph-mon[39795]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:43:23.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:23 smithi177 ceph-mon[44303]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:43:23.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:23 smithi177 ceph-mon[44303]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:43:24.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:24 smithi027 ceph-mon[39795]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-05-15T12:43:24.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:24 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T12:43:24.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:24 smithi120 ceph-mon[43961]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-05-15T12:43:24.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:24 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T12:43:24.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:24 smithi177 ceph-mon[44303]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-05-15T12:43:24.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:24 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T12:43:25.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:25 smithi027 ceph-mon[39795]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:43:25.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:25 smithi027 ceph-mon[39795]: from='client.? 172.21.15.120:0/1703311570' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b"}]: dispatch 2024-05-15T12:43:25.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:25 smithi027 ceph-mon[39795]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b"}]: dispatch 2024-05-15T12:43:25.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:25 smithi027 ceph-mon[39795]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b"}]': finished 2024-05-15T12:43:25.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:25 smithi027 ceph-mon[39795]: osdmap e45: 7 total, 6 up, 7 in 2024-05-15T12:43:25.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:25 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:43:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:25 smithi120 ceph-mon[43961]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:43:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:25 smithi120 ceph-mon[43961]: from='client.? 172.21.15.120:0/1703311570' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b"}]: dispatch 2024-05-15T12:43:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:25 smithi120 ceph-mon[43961]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b"}]: dispatch 2024-05-15T12:43:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:25 smithi120 ceph-mon[43961]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b"}]': finished 2024-05-15T12:43:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:25 smithi120 ceph-mon[43961]: osdmap e45: 7 total, 6 up, 7 in 2024-05-15T12:43:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:25 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:43:25.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:25 smithi177 ceph-mon[44303]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:43:25.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:25 smithi177 ceph-mon[44303]: from='client.? 172.21.15.120:0/1703311570' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b"}]: dispatch 2024-05-15T12:43:25.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:25 smithi177 ceph-mon[44303]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b"}]: dispatch 2024-05-15T12:43:25.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:25 smithi177 ceph-mon[44303]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b"}]': finished 2024-05-15T12:43:25.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:25 smithi177 ceph-mon[44303]: osdmap e45: 7 total, 6 up, 7 in 2024-05-15T12:43:25.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:25 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:43:26.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:26 smithi027 ceph-mon[39795]: from='client.? 172.21.15.120:0/1494118303' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:43:26.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:26 smithi120 ceph-mon[43961]: from='client.? 172.21.15.120:0/1494118303' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:43:26.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:26 smithi177 ceph-mon[44303]: from='client.? 172.21.15.120:0/1494118303' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:43:27.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:27 smithi027 ceph-mon[39795]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-15T12:43:27.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:27 smithi120 ceph-mon[43961]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-15T12:43:27.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:27 smithi177 ceph-mon[44303]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-15T12:43:29.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:29 smithi027 ceph-mon[39795]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-05-15T12:43:29.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:29 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T12:43:29.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:29 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T12:43:29.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:29 smithi120 ceph-mon[43961]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-05-15T12:43:29.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:29 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T12:43:29.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:29 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T12:43:29.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:29 smithi177 ceph-mon[44303]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-05-15T12:43:29.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:29 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T12:43:29.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:29 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T12:43:31.494 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:31 smithi120 ceph-mon[43961]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-05-15T12:43:31.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:31 smithi027 ceph-mon[39795]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-05-15T12:43:31.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:31 smithi177 ceph-mon[44303]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-05-15T12:43:32.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:32 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-15T12:43:32.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:32 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:32.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:32 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-15T12:43:32.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:32 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:32.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:32 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-15T12:43:32.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:32 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:33.393 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:33 smithi120 ceph-mon[43961]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-15T12:43:33.393 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:33 smithi120 ceph-mon[43961]: Deploying daemon osd.6 on smithi120 2024-05-15T12:43:33.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:33 smithi027 ceph-mon[39795]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-15T12:43:33.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:33 smithi027 ceph-mon[39795]: Deploying daemon osd.6 on smithi120 2024-05-15T12:43:33.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:33 smithi177 ceph-mon[44303]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-15T12:43:33.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:33 smithi177 ceph-mon[44303]: Deploying daemon osd.6 on smithi120 2024-05-15T12:43:35.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:35 smithi120 ceph-mon[43961]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:43:35.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:35 smithi177 ceph-mon[44303]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:43:35.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:35 smithi027 ceph-mon[39795]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:43:37.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:37 smithi120 ceph-mon[43961]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:43:37.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:37.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:37.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:43:37.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:37 smithi177 ceph-mon[44303]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:43:37.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:37.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:37.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:43:37.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:37 smithi027 ceph-mon[39795]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:43:37.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:37.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:37.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:43:39.400 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:39 smithi120 ceph-mon[43961]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:43:39.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:39 smithi177 ceph-mon[44303]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:43:39.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:39 smithi027 ceph-mon[39795]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:43:40.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:40 smithi177 ceph-mon[44303]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:43:40.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:40 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:40.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:40 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:40.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:40 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:40.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:40 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:43:40.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:40 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:40.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:40 smithi177 ceph-mon[44303]: from='osd.6 [v2:172.21.15.120:6816/3323706735,v1:172.21.15.120:6817/3323706735]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-15T12:43:40.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:40 smithi177 ceph-mon[44303]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-15T12:43:40.957 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:40 smithi120 ceph-mon[43961]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:43:40.957 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:40 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:40.957 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:40 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:40.957 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:40 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:40.957 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:40 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:43:40.957 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:40 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:40.957 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:40 smithi120 ceph-mon[43961]: from='osd.6 [v2:172.21.15.120:6816/3323706735,v1:172.21.15.120:6817/3323706735]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-15T12:43:40.957 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:40 smithi120 ceph-mon[43961]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-15T12:43:41.009 INFO:teuthology.orchestra.run.smithi120.stdout:Created osd(s) 6 on host 'smithi120' 2024-05-15T12:43:41.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:40 smithi027 ceph-mon[39795]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:43:41.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:40 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:41.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:40 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:41.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:40 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:41.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:40 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:43:41.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:40 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:41.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:40 smithi027 ceph-mon[39795]: from='osd.6 [v2:172.21.15.120:6816/3323706735,v1:172.21.15.120:6817/3323706735]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-15T12:43:41.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:40 smithi027 ceph-mon[39795]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-15T12:43:41.671 DEBUG:teuthology.orchestra.run.smithi120:osd.6> sudo journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.6.service 2024-05-15T12:43:41.673 INFO:tasks.cephadm:Deploying osd.7 on smithi120 with /dev/vg_nvme/lv_1... 2024-05-15T12:43:41.673 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-15T12:43:41.705 INFO:journalctl@ceph.osd.6.smithi120.stdout:-- Logs begin at Wed 2024-05-15 12:26:58 UTC. -- 2024-05-15T12:43:42.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:41 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:42.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:41 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:42.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:41 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:43:42.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:41 smithi027 ceph-mon[39795]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-15T12:43:42.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:41 smithi027 ceph-mon[39795]: osdmap e46: 7 total, 6 up, 7 in 2024-05-15T12:43:42.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:41 smithi027 ceph-mon[39795]: from='osd.6 [v2:172.21.15.120:6816/3323706735,v1:172.21.15.120:6817/3323706735]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-05-15T12:43:42.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:41 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:43:42.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:41 smithi027 ceph-mon[39795]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-05-15T12:43:42.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:41 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:42.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:41 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:42.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:41 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:43:42.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:41 smithi120 ceph-mon[43961]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-15T12:43:42.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:41 smithi120 ceph-mon[43961]: osdmap e46: 7 total, 6 up, 7 in 2024-05-15T12:43:42.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:41 smithi120 ceph-mon[43961]: from='osd.6 [v2:172.21.15.120:6816/3323706735,v1:172.21.15.120:6817/3323706735]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-05-15T12:43:42.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:41 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:43:42.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:41 smithi120 ceph-mon[43961]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-05-15T12:43:42.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:41 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:42.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:41 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:42.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:41 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:43:42.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:41 smithi177 ceph-mon[44303]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-15T12:43:42.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:41 smithi177 ceph-mon[44303]: osdmap e46: 7 total, 6 up, 7 in 2024-05-15T12:43:42.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:41 smithi177 ceph-mon[44303]: from='osd.6 [v2:172.21.15.120:6816/3323706735,v1:172.21.15.120:6817/3323706735]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-05-15T12:43:42.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:41 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:43:42.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:41 smithi177 ceph-mon[44303]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-05-15T12:43:42.621 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 12:43:42 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T12:43:42.296+0000 7fa32b448700 -1 osd.6 0 waiting for initial osdmap 2024-05-15T12:43:42.621 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 12:43:42 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T12:43:42.303+0000 7fa32723b700 -1 osd.6 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:43:42.902 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:43:43.545 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:43 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:43:43.546 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:43 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:43:43.546 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:43 smithi120 ceph-mon[43961]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:43:43.546 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:43 smithi120 ceph-mon[43961]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-05-15T12:43:43.546 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:43 smithi120 ceph-mon[43961]: osdmap e47: 7 total, 6 up, 7 in 2024-05-15T12:43:43.546 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:43 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:43:43.546 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:43 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:43:43.546 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:43 smithi120 ceph-mon[43961]: osd.6 [v2:172.21.15.120:6816/3323706735,v1:172.21.15.120:6817/3323706735] boot 2024-05-15T12:43:43.547 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:43 smithi120 ceph-mon[43961]: osdmap e48: 7 total, 7 up, 7 in 2024-05-15T12:43:43.547 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:43 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:43:43.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:43 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:43:43.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:43 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:43:43.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:43 smithi177 ceph-mon[44303]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:43:43.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:43 smithi177 ceph-mon[44303]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-05-15T12:43:43.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:43 smithi177 ceph-mon[44303]: osdmap e47: 7 total, 6 up, 7 in 2024-05-15T12:43:43.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:43 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:43:43.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:43 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:43:43.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:43 smithi177 ceph-mon[44303]: osd.6 [v2:172.21.15.120:6816/3323706735,v1:172.21.15.120:6817/3323706735] boot 2024-05-15T12:43:43.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:43 smithi177 ceph-mon[44303]: osdmap e48: 7 total, 7 up, 7 in 2024-05-15T12:43:43.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:43 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:43:43.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:43 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:43:43.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:43 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:43:43.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:43 smithi027 ceph-mon[39795]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:43:43.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:43 smithi027 ceph-mon[39795]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-05-15T12:43:43.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:43 smithi027 ceph-mon[39795]: osdmap e47: 7 total, 6 up, 7 in 2024-05-15T12:43:43.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:43 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:43:43.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:43 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:43:43.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:43 smithi027 ceph-mon[39795]: osd.6 [v2:172.21.15.120:6816/3323706735,v1:172.21.15.120:6817/3323706735] boot 2024-05-15T12:43:43.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:43 smithi027 ceph-mon[39795]: osdmap e48: 7 total, 7 up, 7 in 2024-05-15T12:43:43.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:43 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:43:45.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:44 smithi027 ceph-mon[39795]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:43:45.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:44 smithi027 ceph-mon[39795]: osdmap e49: 7 total, 7 up, 7 in 2024-05-15T12:43:45.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:44 smithi120 ceph-mon[43961]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:43:45.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:44 smithi120 ceph-mon[43961]: osdmap e49: 7 total, 7 up, 7 in 2024-05-15T12:43:45.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:44 smithi177 ceph-mon[44303]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:43:45.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:44 smithi177 ceph-mon[44303]: osdmap e49: 7 total, 7 up, 7 in 2024-05-15T12:43:45.364 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-15T12:43:45.364 INFO:teuthology.orchestra.run.smithi120.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-15T12:43:45.364 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10+0 records in 2024-05-15T12:43:45.364 INFO:teuthology.orchestra.run.smithi120.stderr:10+0 records out 2024-05-15T12:43:45.364 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011012 s, 952 MB/s 2024-05-15T12:43:45.364 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping successful for: 2024-05-15T12:43:46.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:45 smithi120 ceph-mon[43961]: osdmap e50: 7 total, 7 up, 7 in 2024-05-15T12:43:46.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:45 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:46.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:45 smithi177 ceph-mon[44303]: osdmap e50: 7 total, 7 up, 7 in 2024-05-15T12:43:46.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:45 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:46.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:45 smithi027 ceph-mon[39795]: osdmap e50: 7 total, 7 up, 7 in 2024-05-15T12:43:46.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:45 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:46.392 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi120:vg_nvme/lv_1 2024-05-15T12:43:46.652 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:43:47.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:46 smithi120 ceph-mon[43961]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:43:47.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:46 smithi120 ceph-mon[43961]: Detected new or changed devices on smithi120 2024-05-15T12:43:47.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:46 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:47.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:46 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:43:47.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:46 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:43:47.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:46 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:43:47.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:46 smithi120 ceph-mon[43961]: Adjusting osd_memory_target on smithi120 to 5707M 2024-05-15T12:43:47.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:46 smithi120 ceph-mon[43961]: osdmap e51: 7 total, 7 up, 7 in 2024-05-15T12:43:47.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:46 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:47.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:46 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:47.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:46 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:43:47.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:46 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:47.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:46 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:43:47.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:46 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:43:47.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:46 smithi177 ceph-mon[44303]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:43:47.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:46 smithi177 ceph-mon[44303]: Detected new or changed devices on smithi120 2024-05-15T12:43:47.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:46 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:47.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:46 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:43:47.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:46 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:43:47.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:46 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:43:47.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:46 smithi177 ceph-mon[44303]: Adjusting osd_memory_target on smithi120 to 5707M 2024-05-15T12:43:47.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:46 smithi177 ceph-mon[44303]: osdmap e51: 7 total, 7 up, 7 in 2024-05-15T12:43:47.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:46 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:47.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:46 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:47.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:46 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:43:47.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:46 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:47.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:46 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:43:47.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:46 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:43:47.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:46 smithi027 ceph-mon[39795]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:43:47.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:46 smithi027 ceph-mon[39795]: Detected new or changed devices on smithi120 2024-05-15T12:43:47.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:46 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:47.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:46 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:43:47.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:46 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:43:47.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:46 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:43:47.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:46 smithi027 ceph-mon[39795]: Adjusting osd_memory_target on smithi120 to 5707M 2024-05-15T12:43:47.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:46 smithi027 ceph-mon[39795]: osdmap e51: 7 total, 7 up, 7 in 2024-05-15T12:43:47.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:46 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:47.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:46 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:47.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:46 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:43:47.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:46 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:43:47.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:46 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:43:47.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:46 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:43:48.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:47 smithi120 ceph-mon[43961]: osdmap e52: 7 total, 7 up, 7 in 2024-05-15T12:43:48.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:47 smithi177 ceph-mon[44303]: osdmap e52: 7 total, 7 up, 7 in 2024-05-15T12:43:48.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:47 smithi027 ceph-mon[39795]: osdmap e52: 7 total, 7 up, 7 in 2024-05-15T12:43:49.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:48 smithi177 ceph-mon[44303]: pgmap v176: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:43:49.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:48 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:43:49.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:48 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:43:49.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:48 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:49.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:48 smithi027 ceph-mon[39795]: pgmap v176: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:43:49.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:48 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:43:49.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:48 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:43:49.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:48 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:49.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:48 smithi120 ceph-mon[43961]: pgmap v176: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:43:49.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:48 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:43:49.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:48 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:43:49.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:48 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:50.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:49 smithi177 ceph-mon[44303]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:43:50.218 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:49 smithi120 ceph-mon[43961]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:43:50.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:49 smithi027 ceph-mon[39795]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:43:51.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:50 smithi177 ceph-mon[44303]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-05-15T12:43:51.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:50 smithi027 ceph-mon[39795]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-05-15T12:43:51.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:50 smithi120 ceph-mon[43961]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-05-15T12:43:52.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:51 smithi177 ceph-mon[44303]: from='client.? 172.21.15.120:0/1785626026' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ef085d8-f59b-4033-8280-2c8f78693e30"}]: dispatch 2024-05-15T12:43:52.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:51 smithi177 ceph-mon[44303]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ef085d8-f59b-4033-8280-2c8f78693e30"}]: dispatch 2024-05-15T12:43:52.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:51 smithi177 ceph-mon[44303]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ef085d8-f59b-4033-8280-2c8f78693e30"}]': finished 2024-05-15T12:43:52.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:51 smithi177 ceph-mon[44303]: osdmap e53: 8 total, 7 up, 8 in 2024-05-15T12:43:52.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:51 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:43:52.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:51 smithi027 ceph-mon[39795]: from='client.? 172.21.15.120:0/1785626026' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ef085d8-f59b-4033-8280-2c8f78693e30"}]: dispatch 2024-05-15T12:43:52.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:51 smithi027 ceph-mon[39795]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ef085d8-f59b-4033-8280-2c8f78693e30"}]: dispatch 2024-05-15T12:43:52.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:51 smithi027 ceph-mon[39795]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ef085d8-f59b-4033-8280-2c8f78693e30"}]': finished 2024-05-15T12:43:52.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:51 smithi027 ceph-mon[39795]: osdmap e53: 8 total, 7 up, 8 in 2024-05-15T12:43:52.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:51 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:43:52.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:51 smithi120 ceph-mon[43961]: from='client.? 172.21.15.120:0/1785626026' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ef085d8-f59b-4033-8280-2c8f78693e30"}]: dispatch 2024-05-15T12:43:52.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:51 smithi120 ceph-mon[43961]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ef085d8-f59b-4033-8280-2c8f78693e30"}]: dispatch 2024-05-15T12:43:52.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:51 smithi120 ceph-mon[43961]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ef085d8-f59b-4033-8280-2c8f78693e30"}]': finished 2024-05-15T12:43:52.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:51 smithi120 ceph-mon[43961]: osdmap e53: 8 total, 7 up, 8 in 2024-05-15T12:43:52.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:51 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:43:53.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:52 smithi177 ceph-mon[44303]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-05-15T12:43:53.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:52 smithi177 ceph-mon[44303]: from='client.? 172.21.15.120:0/4164885074' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:43:53.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:52 smithi027 ceph-mon[39795]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-05-15T12:43:53.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:52 smithi027 ceph-mon[39795]: from='client.? 172.21.15.120:0/4164885074' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:43:53.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:52 smithi120 ceph-mon[43961]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-05-15T12:43:53.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:52 smithi120 ceph-mon[43961]: from='client.? 172.21.15.120:0/4164885074' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:43:55.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:54 smithi120 ceph-mon[43961]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-05-15T12:43:55.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:54 smithi177 ceph-mon[44303]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-05-15T12:43:55.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:54 smithi027 ceph-mon[39795]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-05-15T12:43:57.091 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:56 smithi120 ceph-mon[43961]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-05-15T12:43:57.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:56 smithi177 ceph-mon[44303]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-05-15T12:43:57.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:56 smithi027 ceph-mon[39795]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-05-15T12:43:59.130 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:58 smithi120 ceph-mon[43961]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-15T12:43:59.131 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:58 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-15T12:43:59.131 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:58 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:59.131 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:43:58 smithi120 ceph-mon[43961]: Deploying daemon osd.7 on smithi120 2024-05-15T12:43:59.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:58 smithi177 ceph-mon[44303]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-15T12:43:59.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:58 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-15T12:43:59.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:58 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:59.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:43:58 smithi177 ceph-mon[44303]: Deploying daemon osd.7 on smithi120 2024-05-15T12:43:59.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:58 smithi027 ceph-mon[39795]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-15T12:43:59.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:58 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-15T12:43:59.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:58 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:43:59.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:43:58 smithi027 ceph-mon[39795]: Deploying daemon osd.7 on smithi120 2024-05-15T12:44:01.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:00 smithi177 ceph-mon[44303]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:44:01.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:00 smithi027 ceph-mon[39795]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:44:01.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:00 smithi120 ceph-mon[43961]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:44:03.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:02 smithi177 ceph-mon[44303]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:44:03.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:02 smithi027 ceph-mon[39795]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:44:03.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:02 smithi120 ceph-mon[43961]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:44:05.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:05 smithi177 ceph-mon[44303]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:44:05.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:05 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:05.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:05 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:05.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:05 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:44:05.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:05 smithi027 ceph-mon[39795]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:44:05.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:05 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:05.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:05 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:05.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:05 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:44:05.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:05 smithi120 ceph-mon[43961]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:44:05.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:05 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:05.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:05 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:05.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:05 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:44:07.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:07 smithi177 ceph-mon[44303]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:44:07.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:07 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:07.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:07 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:07.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:07 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:07.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:07 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:44:07.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:07 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:07.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:07 smithi027 ceph-mon[39795]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:44:07.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:07 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:07.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:07 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:07.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:07 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:07.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:07 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:44:07.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:07 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:07.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:07 smithi120 ceph-mon[43961]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:44:07.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:07 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:07.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:07 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:07.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:07 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:07.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:07 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:44:07.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:07 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:08.178 INFO:teuthology.orchestra.run.smithi120.stdout:Created osd(s) 7 on host 'smithi120' 2024-05-15T12:44:08.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:08 smithi120 ceph-mon[43961]: from='osd.7 [v2:172.21.15.120:6824/2960616153,v1:172.21.15.120:6825/2960616153]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-15T12:44:08.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:08 smithi120 ceph-mon[43961]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-15T12:44:08.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:08 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:08.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:08 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:08.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:08 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:44:08.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:08 smithi177 ceph-mon[44303]: from='osd.7 [v2:172.21.15.120:6824/2960616153,v1:172.21.15.120:6825/2960616153]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-15T12:44:08.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:08 smithi177 ceph-mon[44303]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-15T12:44:08.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:08 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:08.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:08 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:08.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:08 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:44:08.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:08 smithi027 ceph-mon[39795]: from='osd.7 [v2:172.21.15.120:6824/2960616153,v1:172.21.15.120:6825/2960616153]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-15T12:44:08.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:08 smithi027 ceph-mon[39795]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-15T12:44:08.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:08 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:08.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:08 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:08.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:08 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:44:08.941 DEBUG:teuthology.orchestra.run.smithi120:osd.7> sudo journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.7.service 2024-05-15T12:44:08.943 INFO:tasks.cephadm:Deploying osd.8 on smithi177 with /dev/vg_nvme/lv_4... 2024-05-15T12:44:08.943 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-15T12:44:08.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:-- Logs begin at Wed 2024-05-15 12:26:58 UTC. -- 2024-05-15T12:44:09.211 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:44:09.464 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:09 smithi120 ceph-mon[43961]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:44:09.465 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:09 smithi120 ceph-mon[43961]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-15T12:44:09.465 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:09 smithi120 ceph-mon[43961]: from='osd.7 [v2:172.21.15.120:6824/2960616153,v1:172.21.15.120:6825/2960616153]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-05-15T12:44:09.465 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:09 smithi120 ceph-mon[43961]: osdmap e54: 8 total, 7 up, 8 in 2024-05-15T12:44:09.465 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:09 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:44:09.465 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:09 smithi120 ceph-mon[43961]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-05-15T12:44:09.669 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:09 smithi177 ceph-mon[44303]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:44:09.669 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:09 smithi177 ceph-mon[44303]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-15T12:44:09.669 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:09 smithi177 ceph-mon[44303]: from='osd.7 [v2:172.21.15.120:6824/2960616153,v1:172.21.15.120:6825/2960616153]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-05-15T12:44:09.669 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:09 smithi177 ceph-mon[44303]: osdmap e54: 8 total, 7 up, 8 in 2024-05-15T12:44:09.669 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:09 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:44:09.669 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:09 smithi177 ceph-mon[44303]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-05-15T12:44:09.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:09 smithi027 ceph-mon[39795]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:44:09.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:09 smithi027 ceph-mon[39795]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-15T12:44:09.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:09 smithi027 ceph-mon[39795]: from='osd.7 [v2:172.21.15.120:6824/2960616153,v1:172.21.15.120:6825/2960616153]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-05-15T12:44:09.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:09 smithi027 ceph-mon[39795]: osdmap e54: 8 total, 7 up, 8 in 2024-05-15T12:44:09.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:09 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:44:09.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:09 smithi027 ceph-mon[39795]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-05-15T12:44:10.200 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 12:44:09 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T12:44:09.799+0000 7f26e710d700 -1 osd.7 0 waiting for initial osdmap 2024-05-15T12:44:10.200 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 12:44:09 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T12:44:09.807+0000 7f26deef8700 -1 osd.7 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:44:10.515 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:10 smithi120 ceph-mon[43961]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-05-15T12:44:10.515 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:10 smithi120 ceph-mon[43961]: osdmap e55: 8 total, 7 up, 8 in 2024-05-15T12:44:10.515 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:10 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:44:10.515 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:10 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:44:10.515 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:10 smithi120 ceph-mon[43961]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:44:10.515 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:10 smithi120 ceph-mon[43961]: from='osd.7 ' entity='osd.7' 2024-05-15T12:44:10.652 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:10 smithi177 ceph-mon[44303]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-05-15T12:44:10.652 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:10 smithi177 ceph-mon[44303]: osdmap e55: 8 total, 7 up, 8 in 2024-05-15T12:44:10.652 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:10 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:44:10.652 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:10 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:44:10.652 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:10 smithi177 ceph-mon[44303]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:44:10.652 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:10 smithi177 ceph-mon[44303]: from='osd.7 ' entity='osd.7' 2024-05-15T12:44:10.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:10 smithi027 ceph-mon[39795]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-05-15T12:44:10.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:10 smithi027 ceph-mon[39795]: osdmap e55: 8 total, 7 up, 8 in 2024-05-15T12:44:10.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:10 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:44:10.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:10 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:44:10.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:10 smithi027 ceph-mon[39795]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:44:10.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:10 smithi027 ceph-mon[39795]: from='osd.7 ' entity='osd.7' 2024-05-15T12:44:11.230 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-15T12:44:11.230 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-15T12:44:11.230 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-05-15T12:44:11.231 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-05-15T12:44:11.231 INFO:teuthology.orchestra.run.smithi177.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0127584 s, 822 MB/s 2024-05-15T12:44:11.231 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-05-15T12:44:11.537 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:11 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:44:11.537 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:11 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:44:11.538 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:11 smithi177 ceph-mon[44303]: osd.7 [v2:172.21.15.120:6824/2960616153,v1:172.21.15.120:6825/2960616153] boot 2024-05-15T12:44:11.538 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:11 smithi177 ceph-mon[44303]: osdmap e56: 8 total, 8 up, 8 in 2024-05-15T12:44:11.538 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:11 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:44:11.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:11 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:44:11.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:11 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:44:11.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:11 smithi027 ceph-mon[39795]: osd.7 [v2:172.21.15.120:6824/2960616153,v1:172.21.15.120:6825/2960616153] boot 2024-05-15T12:44:11.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:11 smithi027 ceph-mon[39795]: osdmap e56: 8 total, 8 up, 8 in 2024-05-15T12:44:11.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:11 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:44:11.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:11 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:44:11.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:11 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:44:11.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:11 smithi120 ceph-mon[43961]: osd.7 [v2:172.21.15.120:6824/2960616153,v1:172.21.15.120:6825/2960616153] boot 2024-05-15T12:44:11.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:11 smithi120 ceph-mon[43961]: osdmap e56: 8 total, 8 up, 8 in 2024-05-15T12:44:11.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:11 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:44:11.885 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi177:vg_nvme/lv_4 2024-05-15T12:44:12.142 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:44:12.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:12 smithi120 ceph-mon[43961]: osdmap e57: 8 total, 8 up, 8 in 2024-05-15T12:44:12.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:12 smithi120 ceph-mon[43961]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 396 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:44:12.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:12 smithi120 ceph-mon[43961]: Detected new or changed devices on smithi120 2024-05-15T12:44:12.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:12 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:12.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:12 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:12.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:12 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:44:12.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:12 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:44:12.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:12 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:44:12.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:12 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:44:12.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:12 smithi120 ceph-mon[43961]: Adjusting osd_memory_target on smithi120 to 4280M 2024-05-15T12:44:12.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:12 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:12.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:12 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:12.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:12 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:44:12.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:12 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:12.675 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:12 smithi177 ceph-mon[44303]: osdmap e57: 8 total, 8 up, 8 in 2024-05-15T12:44:12.675 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:12 smithi177 ceph-mon[44303]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 396 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:44:12.676 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:12 smithi177 ceph-mon[44303]: Detected new or changed devices on smithi120 2024-05-15T12:44:12.676 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:12 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:12.676 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:12 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:12.676 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:12 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:44:12.676 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:12 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:44:12.676 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:12 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:44:12.676 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:12 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:44:12.676 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:12 smithi177 ceph-mon[44303]: Adjusting osd_memory_target on smithi120 to 4280M 2024-05-15T12:44:12.676 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:12 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:12.676 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:12 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:12.676 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:12 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:44:12.676 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:12 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:12.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:12 smithi027 ceph-mon[39795]: osdmap e57: 8 total, 8 up, 8 in 2024-05-15T12:44:12.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:12 smithi027 ceph-mon[39795]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 396 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:44:12.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:12 smithi027 ceph-mon[39795]: Detected new or changed devices on smithi120 2024-05-15T12:44:12.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:12 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:12.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:12 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:12.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:12 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:44:12.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:12 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:44:12.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:12 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:44:12.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:12 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:44:12.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:12 smithi027 ceph-mon[39795]: Adjusting osd_memory_target on smithi120 to 4280M 2024-05-15T12:44:12.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:12 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:12.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:12 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:12.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:12 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:44:12.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:12 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:13.597 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:13 smithi177 ceph-mon[44303]: osdmap e58: 8 total, 8 up, 8 in 2024-05-15T12:44:13.597 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:13 smithi177 ceph-mon[44303]: osdmap e59: 8 total, 8 up, 8 in 2024-05-15T12:44:13.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:13 smithi027 ceph-mon[39795]: osdmap e58: 8 total, 8 up, 8 in 2024-05-15T12:44:13.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:13 smithi027 ceph-mon[39795]: osdmap e59: 8 total, 8 up, 8 in 2024-05-15T12:44:13.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:13 smithi120 ceph-mon[43961]: osdmap e58: 8 total, 8 up, 8 in 2024-05-15T12:44:13.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:13 smithi120 ceph-mon[43961]: osdmap e59: 8 total, 8 up, 8 in 2024-05-15T12:44:14.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:14 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:44:14.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:14 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:44:14.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:14 smithi177 ceph-mon[44303]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 396 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:44:14.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:14 smithi177 ceph-mon[44303]: osdmap e60: 8 total, 8 up, 8 in 2024-05-15T12:44:14.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:14 smithi177 ceph-mon[44303]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:44:14.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:44:14.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:44:14.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:15.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:14 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:44:15.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:14 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:44:15.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:14 smithi120 ceph-mon[43961]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 396 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:44:15.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:14 smithi120 ceph-mon[43961]: osdmap e60: 8 total, 8 up, 8 in 2024-05-15T12:44:15.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:14 smithi120 ceph-mon[43961]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:44:15.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:44:15.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:44:15.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:15.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:14 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:44:15.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:14 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:44:15.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:14 smithi027 ceph-mon[39795]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 396 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:44:15.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:14 smithi027 ceph-mon[39795]: osdmap e60: 8 total, 8 up, 8 in 2024-05-15T12:44:15.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:14 smithi027 ceph-mon[39795]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:44:15.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:44:15.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:44:15.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:17.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:16 smithi177 ceph-mon[44303]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-05-15T12:44:17.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:16 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:44:17.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:16 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:44:17.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:16 smithi177 ceph-mon[44303]: from='client.? 172.21.15.177:0/3699352589' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26514b11-d02b-46e9-a4e8-3da193fcdbcc"}]: dispatch 2024-05-15T12:44:17.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:16 smithi177 ceph-mon[44303]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26514b11-d02b-46e9-a4e8-3da193fcdbcc"}]: dispatch 2024-05-15T12:44:17.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:16 smithi177 ceph-mon[44303]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26514b11-d02b-46e9-a4e8-3da193fcdbcc"}]': finished 2024-05-15T12:44:17.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:16 smithi177 ceph-mon[44303]: osdmap e61: 9 total, 8 up, 9 in 2024-05-15T12:44:17.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:16 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T12:44:17.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:16 smithi027 ceph-mon[39795]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-05-15T12:44:17.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:16 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:44:17.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:16 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:44:17.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:16 smithi027 ceph-mon[39795]: from='client.? 172.21.15.177:0/3699352589' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26514b11-d02b-46e9-a4e8-3da193fcdbcc"}]: dispatch 2024-05-15T12:44:17.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:16 smithi027 ceph-mon[39795]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26514b11-d02b-46e9-a4e8-3da193fcdbcc"}]: dispatch 2024-05-15T12:44:17.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:16 smithi027 ceph-mon[39795]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26514b11-d02b-46e9-a4e8-3da193fcdbcc"}]': finished 2024-05-15T12:44:17.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:16 smithi027 ceph-mon[39795]: osdmap e61: 9 total, 8 up, 9 in 2024-05-15T12:44:17.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:16 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T12:44:17.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:16 smithi120 ceph-mon[43961]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-05-15T12:44:17.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:16 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:44:17.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:16 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:44:17.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:16 smithi120 ceph-mon[43961]: from='client.? 172.21.15.177:0/3699352589' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26514b11-d02b-46e9-a4e8-3da193fcdbcc"}]: dispatch 2024-05-15T12:44:17.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:16 smithi120 ceph-mon[43961]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26514b11-d02b-46e9-a4e8-3da193fcdbcc"}]: dispatch 2024-05-15T12:44:17.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:16 smithi120 ceph-mon[43961]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26514b11-d02b-46e9-a4e8-3da193fcdbcc"}]': finished 2024-05-15T12:44:17.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:16 smithi120 ceph-mon[43961]: osdmap e61: 9 total, 8 up, 9 in 2024-05-15T12:44:17.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:16 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T12:44:18.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:17 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:44:18.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:17 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:44:18.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:17 smithi177 ceph-mon[44303]: from='client.? 172.21.15.177:0/773988483' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:44:18.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:17 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:44:18.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:17 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:44:18.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:17 smithi027 ceph-mon[39795]: from='client.? 172.21.15.177:0/773988483' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:44:18.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:17 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:44:18.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:17 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:44:18.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:17 smithi120 ceph-mon[43961]: from='client.? 172.21.15.177:0/773988483' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:44:19.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:18 smithi177 ceph-mon[44303]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-05-15T12:44:19.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:18 smithi027 ceph-mon[39795]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-05-15T12:44:19.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:18 smithi120 ceph-mon[43961]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-05-15T12:44:20.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:20 smithi177 ceph-mon[44303]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-05-15T12:44:21.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:20 smithi027 ceph-mon[39795]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-05-15T12:44:21.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:20 smithi120 ceph-mon[43961]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-05-15T12:44:22.846 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:22 smithi177 ceph-mon[44303]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-15T12:44:23.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:22 smithi027 ceph-mon[39795]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-15T12:44:23.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:22 smithi120 ceph-mon[43961]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-15T12:44:23.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-15T12:44:23.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:23.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:23 smithi177 ceph-mon[44303]: Deploying daemon osd.8 on smithi177 2024-05-15T12:44:23.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:23 smithi177 ceph-mon[44303]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-05-15T12:44:23.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:44:23.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:44:23.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:44:23.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:44:23.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:23.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:44:23.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:24.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-15T12:44:24.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:24.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:23 smithi027 ceph-mon[39795]: Deploying daemon osd.8 on smithi177 2024-05-15T12:44:24.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:23 smithi027 ceph-mon[39795]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-05-15T12:44:24.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:44:24.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:44:24.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:44:24.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:44:24.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:24.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:44:24.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:24.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-15T12:44:24.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:24.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:23 smithi120 ceph-mon[43961]: Deploying daemon osd.8 on smithi177 2024-05-15T12:44:24.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:23 smithi120 ceph-mon[43961]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-05-15T12:44:24.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:44:24.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:44:24.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:44:24.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:44:24.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:24.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:44:24.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:26.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:26 smithi027 ceph-mon[39795]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:44:26.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:26 smithi120 ceph-mon[43961]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:44:26.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:26 smithi177 ceph-mon[44303]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:44:28.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:28 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:44:28.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:28 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:44:28.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:28 smithi027 ceph-mon[39795]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:44:28.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:28 smithi027 ceph-mon[39795]: osdmap e62: 9 total, 8 up, 9 in 2024-05-15T12:44:28.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T12:44:28.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:28.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:28.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:44:28.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:28 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:44:28.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:28 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:44:28.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:28 smithi120 ceph-mon[43961]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:44:28.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:28 smithi120 ceph-mon[43961]: osdmap e62: 9 total, 8 up, 9 in 2024-05-15T12:44:28.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T12:44:28.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:28.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:28.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:44:28.857 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:28 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:44:28.857 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:28 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:44:28.858 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:28 smithi177 ceph-mon[44303]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:44:28.858 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:28 smithi177 ceph-mon[44303]: osdmap e62: 9 total, 8 up, 9 in 2024-05-15T12:44:28.858 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T12:44:28.858 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:28.858 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:28.858 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:44:30.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:30 smithi177 ceph-mon[44303]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:44:30.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:30 smithi027 ceph-mon[39795]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:44:30.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:30 smithi120 ceph-mon[43961]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:44:31.747 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 8 on host 'smithi177' 2024-05-15T12:44:32.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:31 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:32.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:31 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:32.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:31 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:32.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:31 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:44:32.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:31 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:32.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:31 smithi027 ceph-mon[39795]: from='osd.8 [v2:172.21.15.177:6800/1132459315,v1:172.21.15.177:6801/1132459315]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-15T12:44:32.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:31 smithi027 ceph-mon[39795]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-15T12:44:32.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:31 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:32.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:31 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:44:32.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:31 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:32.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:31 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:32.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:31 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:32.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:31 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:32.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:31 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:44:32.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:31 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:32.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:31 smithi120 ceph-mon[43961]: from='osd.8 [v2:172.21.15.177:6800/1132459315,v1:172.21.15.177:6801/1132459315]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-15T12:44:32.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:31 smithi120 ceph-mon[43961]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-15T12:44:32.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:31 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:32.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:31 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:44:32.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:31 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:32.352 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:31 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:32.352 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:31 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:32.352 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:31 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:32.352 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:31 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:44:32.352 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:31 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:32.352 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:31 smithi177 ceph-mon[44303]: from='osd.8 [v2:172.21.15.177:6800/1132459315,v1:172.21.15.177:6801/1132459315]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-15T12:44:32.353 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:31 smithi177 ceph-mon[44303]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-15T12:44:32.353 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:31 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:32.353 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:31 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:44:32.353 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:31 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:32.379 DEBUG:teuthology.orchestra.run.smithi177:osd.8> sudo journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.8.service 2024-05-15T12:44:32.382 INFO:tasks.cephadm:Deploying osd.9 on smithi177 with /dev/vg_nvme/lv_3... 2024-05-15T12:44:32.382 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-15T12:44:32.411 INFO:journalctl@ceph.osd.8.smithi177.stdout:-- Logs begin at Wed 2024-05-15 12:27:00 UTC. -- 2024-05-15T12:44:33.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:32 smithi027 ceph-mon[39795]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:44:33.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:32 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T12:44:33.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:32 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T12:44:33.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:32 smithi027 ceph-mon[39795]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-15T12:44:33.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:32 smithi027 ceph-mon[39795]: osdmap e63: 9 total, 8 up, 9 in 2024-05-15T12:44:33.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:32 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T12:44:33.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:32 smithi027 ceph-mon[39795]: from='osd.8 [v2:172.21.15.177:6800/1132459315,v1:172.21.15.177:6801/1132459315]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-05-15T12:44:33.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:32 smithi027 ceph-mon[39795]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-05-15T12:44:33.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:32 smithi120 ceph-mon[43961]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:44:33.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:32 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T12:44:33.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:32 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T12:44:33.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:32 smithi120 ceph-mon[43961]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-15T12:44:33.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:32 smithi120 ceph-mon[43961]: osdmap e63: 9 total, 8 up, 9 in 2024-05-15T12:44:33.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:32 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T12:44:33.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:32 smithi120 ceph-mon[43961]: from='osd.8 [v2:172.21.15.177:6800/1132459315,v1:172.21.15.177:6801/1132459315]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-05-15T12:44:33.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:32 smithi120 ceph-mon[43961]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-05-15T12:44:33.429 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 12:44:32 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T12:44:32.977+0000 7fce944ae700 -1 osd.8 0 waiting for initial osdmap 2024-05-15T12:44:33.429 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 12:44:32 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T12:44:32.978+0000 7fce8bc3c700 -1 osd.8 0 failed to load OSD map for epoch 10, got 0 bytes 2024-05-15T12:44:33.430 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 12:44:32 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T12:44:32.985+0000 7fce902a1700 -1 osd.8 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:44:33.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:32 smithi177 ceph-mon[44303]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:44:33.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:32 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T12:44:33.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:32 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T12:44:33.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:32 smithi177 ceph-mon[44303]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-15T12:44:33.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:32 smithi177 ceph-mon[44303]: osdmap e63: 9 total, 8 up, 9 in 2024-05-15T12:44:33.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:32 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T12:44:33.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:32 smithi177 ceph-mon[44303]: from='osd.8 [v2:172.21.15.177:6800/1132459315,v1:172.21.15.177:6801/1132459315]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-05-15T12:44:33.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:32 smithi177 ceph-mon[44303]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-05-15T12:44:33.461 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:44:34.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:33 smithi027 ceph-mon[39795]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-05-15T12:44:34.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:33 smithi027 ceph-mon[39795]: osdmap e64: 9 total, 8 up, 9 in 2024-05-15T12:44:34.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:33 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T12:44:34.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:33 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T12:44:34.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:33 smithi120 ceph-mon[43961]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-05-15T12:44:34.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:33 smithi120 ceph-mon[43961]: osdmap e64: 9 total, 8 up, 9 in 2024-05-15T12:44:34.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:33 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T12:44:34.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:33 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T12:44:34.321 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:33 smithi177 ceph-mon[44303]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-05-15T12:44:34.321 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:33 smithi177 ceph-mon[44303]: osdmap e64: 9 total, 8 up, 9 in 2024-05-15T12:44:34.321 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:33 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T12:44:34.321 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:33 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T12:44:35.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:34 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:44:35.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:34 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:44:35.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:34 smithi027 ceph-mon[39795]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:44:35.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:34 smithi027 ceph-mon[39795]: osd.8 [v2:172.21.15.177:6800/1132459315,v1:172.21.15.177:6801/1132459315] boot 2024-05-15T12:44:35.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:34 smithi027 ceph-mon[39795]: osdmap e65: 9 total, 9 up, 9 in 2024-05-15T12:44:35.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:34 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T12:44:35.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:34 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:44:35.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:34 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:44:35.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:34 smithi120 ceph-mon[43961]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:44:35.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:34 smithi120 ceph-mon[43961]: osd.8 [v2:172.21.15.177:6800/1132459315,v1:172.21.15.177:6801/1132459315] boot 2024-05-15T12:44:35.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:34 smithi120 ceph-mon[43961]: osdmap e65: 9 total, 9 up, 9 in 2024-05-15T12:44:35.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:34 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T12:44:35.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:34 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:44:35.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:34 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:44:35.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:34 smithi177 ceph-mon[44303]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:44:35.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:34 smithi177 ceph-mon[44303]: osd.8 [v2:172.21.15.177:6800/1132459315,v1:172.21.15.177:6801/1132459315] boot 2024-05-15T12:44:35.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:34 smithi177 ceph-mon[44303]: osdmap e65: 9 total, 9 up, 9 in 2024-05-15T12:44:35.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:34 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T12:44:35.821 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-15T12:44:35.821 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-15T12:44:35.821 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-05-15T12:44:35.821 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-05-15T12:44:35.821 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110462 s, 949 MB/s 2024-05-15T12:44:35.821 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-05-15T12:44:36.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:35 smithi177 ceph-mon[44303]: osdmap e66: 9 total, 9 up, 9 in 2024-05-15T12:44:36.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:35 smithi027 ceph-mon[39795]: osdmap e66: 9 total, 9 up, 9 in 2024-05-15T12:44:36.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:36 smithi120 ceph-mon[43961]: osdmap e66: 9 total, 9 up, 9 in 2024-05-15T12:44:37.157 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi177:vg_nvme/lv_3 2024-05-15T12:44:37.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:37 smithi027 ceph-mon[39795]: pgmap v214: 1 pgs: 1 remapped+peering; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:44:37.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:37 smithi027 ceph-mon[39795]: osdmap e67: 9 total, 9 up, 9 in 2024-05-15T12:44:37.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:37 smithi027 ceph-mon[39795]: Detected new or changed devices on smithi177 2024-05-15T12:44:37.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:37.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:37.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:44:37.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:37 smithi027 ceph-mon[39795]: Adjusting osd_memory_target on smithi177 to 21218M 2024-05-15T12:44:37.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:37.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:44:37.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:44:37.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:44:37.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:37.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:44:37.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:37.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:44:37.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:37.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:44:37.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:37.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:37 smithi120 ceph-mon[43961]: pgmap v214: 1 pgs: 1 remapped+peering; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:44:37.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:37 smithi120 ceph-mon[43961]: osdmap e67: 9 total, 9 up, 9 in 2024-05-15T12:44:37.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:37 smithi120 ceph-mon[43961]: Detected new or changed devices on smithi177 2024-05-15T12:44:37.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:37.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:37.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:44:37.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:37 smithi120 ceph-mon[43961]: Adjusting osd_memory_target on smithi177 to 21218M 2024-05-15T12:44:37.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:37.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:44:37.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:44:37.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:44:37.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:37.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:44:37.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:37.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:44:37.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:37.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:44:37.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:37.416 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:44:37.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:37 smithi177 ceph-mon[44303]: pgmap v214: 1 pgs: 1 remapped+peering; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:44:37.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:37 smithi177 ceph-mon[44303]: osdmap e67: 9 total, 9 up, 9 in 2024-05-15T12:44:37.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:37 smithi177 ceph-mon[44303]: Detected new or changed devices on smithi177 2024-05-15T12:44:37.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:37.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:37.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:44:37.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:37 smithi177 ceph-mon[44303]: Adjusting osd_memory_target on smithi177 to 21218M 2024-05-15T12:44:37.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:37.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:44:37.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:44:37.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:44:37.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:37.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:44:37.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:37.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:44:37.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:37.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:44:37.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:38.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:38 smithi177 ceph-mon[44303]: osdmap e68: 9 total, 9 up, 9 in 2024-05-15T12:44:38.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:38 smithi027 ceph-mon[39795]: osdmap e68: 9 total, 9 up, 9 in 2024-05-15T12:44:38.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:38 smithi120 ceph-mon[43961]: osdmap e68: 9 total, 9 up, 9 in 2024-05-15T12:44:39.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:39 smithi177 ceph-mon[44303]: pgmap v217: 1 pgs: 1 remapped+peering; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:44:39.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:39 smithi177 ceph-mon[44303]: osdmap e69: 9 total, 9 up, 9 in 2024-05-15T12:44:39.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:39 smithi027 ceph-mon[39795]: pgmap v217: 1 pgs: 1 remapped+peering; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:44:39.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:39 smithi027 ceph-mon[39795]: osdmap e69: 9 total, 9 up, 9 in 2024-05-15T12:44:39.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:39 smithi120 ceph-mon[43961]: pgmap v217: 1 pgs: 1 remapped+peering; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:44:39.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:39 smithi120 ceph-mon[43961]: osdmap e69: 9 total, 9 up, 9 in 2024-05-15T12:44:40.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:40 smithi177 ceph-mon[44303]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:44:40.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:40 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:44:40.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:40 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:44:40.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:40 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:40.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:40 smithi027 ceph-mon[39795]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:44:40.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:40 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:44:40.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:40 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:44:40.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:40 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:40.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:40 smithi120 ceph-mon[43961]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:44:40.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:40 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:44:40.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:40 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:44:40.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:40 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:41.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:41 smithi027 ceph-mon[39795]: pgmap v219: 1 pgs: 1 remapped+peering; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:44:41.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:41 smithi120 ceph-mon[43961]: pgmap v219: 1 pgs: 1 remapped+peering; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:44:41.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:41 smithi177 ceph-mon[44303]: pgmap v219: 1 pgs: 1 remapped+peering; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:44:42.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:42 smithi027 ceph-mon[39795]: from='client.? 172.21.15.177:0/3575054961' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9497bc-e962-409d-8a2c-0c634cc1124f"}]: dispatch 2024-05-15T12:44:42.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:42 smithi027 ceph-mon[39795]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9497bc-e962-409d-8a2c-0c634cc1124f"}]: dispatch 2024-05-15T12:44:42.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:42 smithi027 ceph-mon[39795]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f9497bc-e962-409d-8a2c-0c634cc1124f"}]': finished 2024-05-15T12:44:42.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:42 smithi027 ceph-mon[39795]: osdmap e70: 10 total, 9 up, 10 in 2024-05-15T12:44:42.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:42 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:44:42.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:42 smithi120 ceph-mon[43961]: from='client.? 172.21.15.177:0/3575054961' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9497bc-e962-409d-8a2c-0c634cc1124f"}]: dispatch 2024-05-15T12:44:42.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:42 smithi120 ceph-mon[43961]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9497bc-e962-409d-8a2c-0c634cc1124f"}]: dispatch 2024-05-15T12:44:42.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:42 smithi120 ceph-mon[43961]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f9497bc-e962-409d-8a2c-0c634cc1124f"}]': finished 2024-05-15T12:44:42.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:42 smithi120 ceph-mon[43961]: osdmap e70: 10 total, 9 up, 10 in 2024-05-15T12:44:42.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:42 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:44:42.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:42 smithi177 ceph-mon[44303]: from='client.? 172.21.15.177:0/3575054961' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9497bc-e962-409d-8a2c-0c634cc1124f"}]: dispatch 2024-05-15T12:44:42.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:42 smithi177 ceph-mon[44303]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9497bc-e962-409d-8a2c-0c634cc1124f"}]: dispatch 2024-05-15T12:44:42.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:42 smithi177 ceph-mon[44303]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f9497bc-e962-409d-8a2c-0c634cc1124f"}]': finished 2024-05-15T12:44:42.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:42 smithi177 ceph-mon[44303]: osdmap e70: 10 total, 9 up, 10 in 2024-05-15T12:44:42.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:42 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:44:43.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:43 smithi027 ceph-mon[39795]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-15T12:44:43.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:43 smithi027 ceph-mon[39795]: from='client.? 172.21.15.177:0/3628978176' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:44:43.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:43 smithi120 ceph-mon[43961]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-15T12:44:43.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:43 smithi120 ceph-mon[43961]: from='client.? 172.21.15.177:0/3628978176' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:44:43.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:43 smithi177 ceph-mon[44303]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-15T12:44:43.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:43 smithi177 ceph-mon[44303]: from='client.? 172.21.15.177:0/3628978176' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:44:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:44 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:44:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:44 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:44:44.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:44 smithi177 ceph-mon[44303]: osdmap e71: 10 total, 9 up, 10 in 2024-05-15T12:44:44.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:44 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:44:44.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:44 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:44:44.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:44 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:44:44.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:44 smithi027 ceph-mon[39795]: osdmap e71: 10 total, 9 up, 10 in 2024-05-15T12:44:44.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:44 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:44:44.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:44 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:44:44.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:44 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:44:44.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:44 smithi120 ceph-mon[43961]: osdmap e71: 10 total, 9 up, 10 in 2024-05-15T12:44:44.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:44 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:44:45.398 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:45 smithi177 ceph-mon[44303]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-15T12:44:45.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:45 smithi027 ceph-mon[39795]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-15T12:44:45.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:45 smithi120 ceph-mon[43961]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-15T12:44:47.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:47 smithi177 ceph-mon[44303]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-05-15T12:44:47.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:47 smithi027 ceph-mon[39795]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-05-15T12:44:47.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:47 smithi120 ceph-mon[43961]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-05-15T12:44:49.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:49 smithi177 ceph-mon[44303]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-15T12:44:49.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:49 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-15T12:44:49.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:49 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:49.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:49 smithi177 ceph-mon[44303]: Deploying daemon osd.9 on smithi177 2024-05-15T12:44:49.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:49 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:44:49.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:49 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:44:49.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:49 smithi027 ceph-mon[39795]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-15T12:44:49.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:49 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-15T12:44:49.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:49 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:49.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:49 smithi027 ceph-mon[39795]: Deploying daemon osd.9 on smithi177 2024-05-15T12:44:49.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:49 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:44:49.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:49 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:44:49.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:49 smithi120 ceph-mon[43961]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-15T12:44:49.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:49 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-15T12:44:49.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:49 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:49.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:49 smithi120 ceph-mon[43961]: Deploying daemon osd.9 on smithi177 2024-05-15T12:44:49.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:49 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:44:49.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:49 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:44:50.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:50 smithi177 ceph-mon[44303]: osdmap e72: 10 total, 9 up, 10 in 2024-05-15T12:44:50.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:50 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:44:50.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:50 smithi027 ceph-mon[39795]: osdmap e72: 10 total, 9 up, 10 in 2024-05-15T12:44:50.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:50 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:44:50.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:50 smithi120 ceph-mon[43961]: osdmap e72: 10 total, 9 up, 10 in 2024-05-15T12:44:50.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:50 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:44:51.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:51 smithi177 ceph-mon[44303]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:44:51.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:51 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:44:51.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:51 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:44:51.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:51 smithi177 ceph-mon[44303]: osdmap e73: 10 total, 9 up, 10 in 2024-05-15T12:44:51.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:51 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:44:51.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:51 smithi027 ceph-mon[39795]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:44:51.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:51 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:44:51.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:51 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:44:51.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:51 smithi027 ceph-mon[39795]: osdmap e73: 10 total, 9 up, 10 in 2024-05-15T12:44:51.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:51 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:44:51.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:51 smithi120 ceph-mon[43961]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:44:51.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:51 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:44:51.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:51 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:44:51.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:51 smithi120 ceph-mon[43961]: osdmap e73: 10 total, 9 up, 10 in 2024-05-15T12:44:51.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:51 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:44:53.338 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:53 smithi177 ceph-mon[44303]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:44:53.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:53 smithi027 ceph-mon[39795]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:44:53.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:53 smithi120 ceph-mon[43961]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:44:54.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:54 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:54.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:54 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:54.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:54 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:44:54.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:54 smithi177 ceph-mon[44303]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:44:54.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:54 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:54.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:54 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:54.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:54 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:44:54.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:54 smithi027 ceph-mon[39795]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:44:54.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:54 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:54.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:54 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:54.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:54 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:44:54.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:54 smithi120 ceph-mon[43961]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:44:57.285 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 9 on host 'smithi177' 2024-05-15T12:44:57.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:57 smithi177 ceph-mon[44303]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:44:57.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:57 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:57.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:57 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:57.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:57 smithi177 ceph-mon[44303]: from='osd.9 [v2:172.21.15.177:6808/2710457646,v1:172.21.15.177:6809/2710457646]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-15T12:44:57.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:57 smithi177 ceph-mon[44303]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-15T12:44:57.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:57 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:57.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:57 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:44:57.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:57 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:57.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:57 smithi027 ceph-mon[39795]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:44:57.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:57 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:57.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:57 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:57.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:57 smithi027 ceph-mon[39795]: from='osd.9 [v2:172.21.15.177:6808/2710457646,v1:172.21.15.177:6809/2710457646]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-15T12:44:57.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:57 smithi027 ceph-mon[39795]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-15T12:44:57.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:57 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:57.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:57 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:44:57.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:57 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:57.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:57 smithi120 ceph-mon[43961]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:44:57.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:57 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:57.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:57 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:57.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:57 smithi120 ceph-mon[43961]: from='osd.9 [v2:172.21.15.177:6808/2710457646,v1:172.21.15.177:6809/2710457646]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-15T12:44:57.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:57 smithi120 ceph-mon[43961]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-15T12:44:57.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:57 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:44:57.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:57 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:44:57.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:57 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:58.065 DEBUG:teuthology.orchestra.run.smithi177:osd.9> sudo journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.9.service 2024-05-15T12:44:58.068 INFO:tasks.cephadm:Deploying osd.10 on smithi177 with /dev/vg_nvme/lv_2... 2024-05-15T12:44:58.069 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-15T12:44:58.099 INFO:journalctl@ceph.osd.9.smithi177.stdout:-- Logs begin at Wed 2024-05-15 12:27:00 UTC. -- 2024-05-15T12:44:58.286 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 12:44:58 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T12:44:58.139+0000 7f798a091700 -1 osd.9 0 waiting for initial osdmap 2024-05-15T12:44:58.286 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 12:44:58 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T12:44:58.141+0000 7f798181f700 -1 osd.9 0 failed to load OSD map for epoch 20, got 0 bytes 2024-05-15T12:44:58.286 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 12:44:58 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T12:44:58.147+0000 7f7985e84700 -1 osd.9 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:44:58.290 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:58 smithi177 ceph-mon[44303]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-15T12:44:58.291 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:58 smithi177 ceph-mon[44303]: osdmap e74: 10 total, 9 up, 10 in 2024-05-15T12:44:58.291 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:58 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:44:58.291 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:58 smithi177 ceph-mon[44303]: from='osd.9 [v2:172.21.15.177:6808/2710457646,v1:172.21.15.177:6809/2710457646]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-05-15T12:44:58.291 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:58 smithi177 ceph-mon[44303]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-05-15T12:44:58.291 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:58 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:58.291 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:58 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:58.292 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:58 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:44:58.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:58 smithi027 ceph-mon[39795]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-15T12:44:58.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:58 smithi027 ceph-mon[39795]: osdmap e74: 10 total, 9 up, 10 in 2024-05-15T12:44:58.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:58 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:44:58.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:58 smithi027 ceph-mon[39795]: from='osd.9 [v2:172.21.15.177:6808/2710457646,v1:172.21.15.177:6809/2710457646]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-05-15T12:44:58.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:58 smithi027 ceph-mon[39795]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-05-15T12:44:58.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:58 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:58.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:58 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:58.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:58 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:44:58.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:58 smithi120 ceph-mon[43961]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-15T12:44:58.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:58 smithi120 ceph-mon[43961]: osdmap e74: 10 total, 9 up, 10 in 2024-05-15T12:44:58.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:58 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:44:58.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:58 smithi120 ceph-mon[43961]: from='osd.9 [v2:172.21.15.177:6808/2710457646,v1:172.21.15.177:6809/2710457646]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-05-15T12:44:58.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:58 smithi120 ceph-mon[43961]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-05-15T12:44:58.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:58 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:58.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:58 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:44:58.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:58 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:44:58.794 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:44:59.162 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:59 smithi177 ceph-mon[44303]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:44:59.162 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:59 smithi177 ceph-mon[44303]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-05-15T12:44:59.162 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:59 smithi177 ceph-mon[44303]: osdmap e75: 10 total, 9 up, 10 in 2024-05-15T12:44:59.162 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:59 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:44:59.162 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:44:59 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:44:59.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:59 smithi027 ceph-mon[39795]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:44:59.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:59 smithi027 ceph-mon[39795]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-05-15T12:44:59.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:59 smithi027 ceph-mon[39795]: osdmap e75: 10 total, 9 up, 10 in 2024-05-15T12:44:59.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:59 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:44:59.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:44:59 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:44:59.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:59 smithi120 ceph-mon[43961]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:44:59.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:59 smithi120 ceph-mon[43961]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-05-15T12:44:59.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:59 smithi120 ceph-mon[43961]: osdmap e75: 10 total, 9 up, 10 in 2024-05-15T12:44:59.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:59 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:44:59.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:44:59 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:45:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:00 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:45:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:00 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:45:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:00 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:45:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:00 smithi177 ceph-mon[44303]: osd.9 [v2:172.21.15.177:6808/2710457646,v1:172.21.15.177:6809/2710457646] boot 2024-05-15T12:45:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:00 smithi177 ceph-mon[44303]: osdmap e76: 10 total, 10 up, 10 in 2024-05-15T12:45:00.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:00 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:45:00.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:00 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:45:00.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:00 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:45:00.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:00 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:45:00.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:00 smithi027 ceph-mon[39795]: osd.9 [v2:172.21.15.177:6808/2710457646,v1:172.21.15.177:6809/2710457646] boot 2024-05-15T12:45:00.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:00 smithi027 ceph-mon[39795]: osdmap e76: 10 total, 10 up, 10 in 2024-05-15T12:45:00.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:00 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:45:00.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:00 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:45:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:00 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:45:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:00 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:45:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:00 smithi120 ceph-mon[43961]: osd.9 [v2:172.21.15.177:6808/2710457646,v1:172.21.15.177:6809/2710457646] boot 2024-05-15T12:45:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:00 smithi120 ceph-mon[43961]: osdmap e76: 10 total, 10 up, 10 in 2024-05-15T12:45:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:00 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T12:45:01.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:01 smithi177 ceph-mon[44303]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:45:01.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:01 smithi177 ceph-mon[44303]: osdmap e77: 10 total, 10 up, 10 in 2024-05-15T12:45:01.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:01 smithi027 ceph-mon[39795]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:45:01.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:01 smithi027 ceph-mon[39795]: osdmap e77: 10 total, 10 up, 10 in 2024-05-15T12:45:01.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:01 smithi120 ceph-mon[43961]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-05-15T12:45:01.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:01 smithi120 ceph-mon[43961]: osdmap e77: 10 total, 10 up, 10 in 2024-05-15T12:45:01.811 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-15T12:45:01.811 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-15T12:45:01.811 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-05-15T12:45:01.811 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-05-15T12:45:01.811 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0133731 s, 784 MB/s 2024-05-15T12:45:01.811 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-05-15T12:45:02.581 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi177:vg_nvme/lv_2 2024-05-15T12:45:02.842 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:45:03.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:02 smithi027 ceph-mon[39795]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:03.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:02 smithi027 ceph-mon[39795]: Detected new or changed devices on smithi177 2024-05-15T12:45:03.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:02 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:03.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:02 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:03.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:02 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:03.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:02 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:03.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:02 smithi027 ceph-mon[39795]: Adjusting osd_memory_target on smithi177 to 10609M 2024-05-15T12:45:03.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:02 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:03.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:02 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:03.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:02 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:45:03.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:02 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:03.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:02 smithi120 ceph-mon[43961]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:03.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:02 smithi120 ceph-mon[43961]: Detected new or changed devices on smithi177 2024-05-15T12:45:03.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:02 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:03.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:02 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:03.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:02 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:03.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:02 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:03.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:02 smithi120 ceph-mon[43961]: Adjusting osd_memory_target on smithi177 to 10609M 2024-05-15T12:45:03.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:02 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:03.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:02 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:03.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:02 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:45:03.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:02 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:03.334 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:02 smithi177 ceph-mon[44303]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:03.334 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:02 smithi177 ceph-mon[44303]: Detected new or changed devices on smithi177 2024-05-15T12:45:03.334 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:02 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:03.334 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:02 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:03.334 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:02 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:03.334 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:02 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:03.334 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:02 smithi177 ceph-mon[44303]: Adjusting osd_memory_target on smithi177 to 10609M 2024-05-15T12:45:03.334 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:02 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:03.335 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:02 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:03.335 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:02 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:45:03.335 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:02 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:05.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:05 smithi027 ceph-mon[39795]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:05.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:05 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:45:05.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:05 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:45:05.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:05 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:05.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:05 smithi120 ceph-mon[43961]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:05.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:05 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:45:05.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:05 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:45:05.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:05 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:05.387 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:05 smithi177 ceph-mon[44303]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:05.387 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:05 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:45:05.387 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:05 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:45:05.387 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:05 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:06.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:06 smithi027 ceph-mon[39795]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:45:06.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:06 smithi120 ceph-mon[43961]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:45:06.322 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:06 smithi177 ceph-mon[44303]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:45:07.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:07 smithi027 ceph-mon[39795]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:07.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:07 smithi120 ceph-mon[43961]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:07.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:07 smithi177 ceph-mon[44303]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:08.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:08 smithi027 ceph-mon[39795]: from='client.? 172.21.15.177:0/1747598273' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f852876f-c7d1-4039-90c4-5ab09dbd3e32"}]: dispatch 2024-05-15T12:45:08.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:08 smithi027 ceph-mon[39795]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f852876f-c7d1-4039-90c4-5ab09dbd3e32"}]: dispatch 2024-05-15T12:45:08.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:08 smithi027 ceph-mon[39795]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f852876f-c7d1-4039-90c4-5ab09dbd3e32"}]': finished 2024-05-15T12:45:08.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:08 smithi027 ceph-mon[39795]: osdmap e78: 11 total, 10 up, 11 in 2024-05-15T12:45:08.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:08 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T12:45:08.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:08 smithi120 ceph-mon[43961]: from='client.? 172.21.15.177:0/1747598273' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f852876f-c7d1-4039-90c4-5ab09dbd3e32"}]: dispatch 2024-05-15T12:45:08.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:08 smithi120 ceph-mon[43961]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f852876f-c7d1-4039-90c4-5ab09dbd3e32"}]: dispatch 2024-05-15T12:45:08.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:08 smithi120 ceph-mon[43961]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f852876f-c7d1-4039-90c4-5ab09dbd3e32"}]': finished 2024-05-15T12:45:08.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:08 smithi120 ceph-mon[43961]: osdmap e78: 11 total, 10 up, 11 in 2024-05-15T12:45:08.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:08 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T12:45:08.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:08 smithi177 ceph-mon[44303]: from='client.? 172.21.15.177:0/1747598273' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f852876f-c7d1-4039-90c4-5ab09dbd3e32"}]: dispatch 2024-05-15T12:45:08.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:08 smithi177 ceph-mon[44303]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f852876f-c7d1-4039-90c4-5ab09dbd3e32"}]: dispatch 2024-05-15T12:45:08.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:08 smithi177 ceph-mon[44303]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f852876f-c7d1-4039-90c4-5ab09dbd3e32"}]': finished 2024-05-15T12:45:08.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:08 smithi177 ceph-mon[44303]: osdmap e78: 11 total, 10 up, 11 in 2024-05-15T12:45:08.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:08 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T12:45:09.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:09 smithi027 ceph-mon[39795]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:09.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:09 smithi027 ceph-mon[39795]: from='client.? 172.21.15.177:0/106791006' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:45:09.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:09 smithi120 ceph-mon[43961]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:09.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:09 smithi120 ceph-mon[43961]: from='client.? 172.21.15.177:0/106791006' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:45:09.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:09 smithi177 ceph-mon[44303]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:09.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:09 smithi177 ceph-mon[44303]: from='client.? 172.21.15.177:0/106791006' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:45:11.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:11 smithi027 ceph-mon[39795]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:11.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:11 smithi120 ceph-mon[43961]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:11.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:11 smithi177 ceph-mon[44303]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:13.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:13 smithi027 ceph-mon[39795]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:13.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:13 smithi120 ceph-mon[43961]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:13.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:13 smithi177 ceph-mon[44303]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:14.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-15T12:45:14.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:14.274 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-15T12:45:14.274 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:14.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-15T12:45:14.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:15.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:15 smithi177 ceph-mon[44303]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:15.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:15 smithi177 ceph-mon[44303]: Deploying daemon osd.10 on smithi177 2024-05-15T12:45:15.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:15 smithi027 ceph-mon[39795]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:15.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:15 smithi027 ceph-mon[39795]: Deploying daemon osd.10 on smithi177 2024-05-15T12:45:15.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:15 smithi120 ceph-mon[43961]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:15.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:15 smithi120 ceph-mon[43961]: Deploying daemon osd.10 on smithi177 2024-05-15T12:45:17.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:17 smithi027 ceph-mon[39795]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:17.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:17 smithi120 ceph-mon[43961]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:17.380 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:17 smithi177 ceph-mon[44303]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:19.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:19 smithi027 ceph-mon[39795]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:19.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:19 smithi120 ceph-mon[43961]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:19.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:19 smithi177 ceph-mon[44303]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:20.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:20 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:20.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:20 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:20.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:20 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:45:20.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:20 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:20.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:20 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:20.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:20 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:45:20.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:20 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:20.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:20 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:20.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:20 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:45:21.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:21 smithi177 ceph-mon[44303]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:21.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:21 smithi027 ceph-mon[39795]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:21.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:21 smithi120 ceph-mon[43961]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:22.900 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 10 on host 'smithi177' 2024-05-15T12:45:23.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:22 smithi177 ceph-mon[44303]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:23.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:22 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:23.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:22 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:23.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:22 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:23.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:22 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:45:23.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:22 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:23.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:22 smithi177 ceph-mon[44303]: from='osd.10 [v2:172.21.15.177:6816/3236900263,v1:172.21.15.177:6817/3236900263]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-15T12:45:23.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:22 smithi177 ceph-mon[44303]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-15T12:45:23.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:22 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:23.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:22 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:23.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:22 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:45:23.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:22 smithi027 ceph-mon[39795]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:45:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:22 smithi027 ceph-mon[39795]: from='osd.10 [v2:172.21.15.177:6816/3236900263,v1:172.21.15.177:6817/3236900263]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-15T12:45:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:22 smithi027 ceph-mon[39795]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-15T12:45:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:22 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:45:23.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:22 smithi120 ceph-mon[43961]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:23.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:22 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:23.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:22 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:23.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:22 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:23.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:22 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:45:23.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:22 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:23.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:22 smithi120 ceph-mon[43961]: from='osd.10 [v2:172.21.15.177:6816/3236900263,v1:172.21.15.177:6817/3236900263]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-15T12:45:23.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:22 smithi120 ceph-mon[43961]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-15T12:45:23.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:22 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:23.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:22 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:23.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:22 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:45:23.555 DEBUG:teuthology.orchestra.run.smithi177:osd.10> sudo journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.10.service 2024-05-15T12:45:23.557 INFO:tasks.cephadm:Deploying osd.11 on smithi177 with /dev/vg_nvme/lv_1... 2024-05-15T12:45:23.558 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-15T12:45:23.587 INFO:journalctl@ceph.osd.10.smithi177.stdout:-- Logs begin at Wed 2024-05-15 12:27:00 UTC. -- 2024-05-15T12:45:24.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:24 smithi177 ceph-mon[44303]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-15T12:45:24.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:24 smithi177 ceph-mon[44303]: osdmap e79: 11 total, 10 up, 11 in 2024-05-15T12:45:24.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:24 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T12:45:24.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:24 smithi177 ceph-mon[44303]: from='osd.10 [v2:172.21.15.177:6816/3236900263,v1:172.21.15.177:6817/3236900263]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-05-15T12:45:24.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:24 smithi177 ceph-mon[44303]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-05-15T12:45:24.430 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 12:45:24 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T12:45:24.214+0000 7f5872c82700 -1 osd.10 0 waiting for initial osdmap 2024-05-15T12:45:24.430 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 12:45:24 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T12:45:24.218+0000 7f586a410700 -1 osd.10 0 failed to load OSD map for epoch 20, got 0 bytes 2024-05-15T12:45:24.430 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 12:45:24 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T12:45:24.224+0000 7f586ea75700 -1 osd.10 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:45:24.431 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:45:24.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:24 smithi027 ceph-mon[39795]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-15T12:45:24.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:24 smithi027 ceph-mon[39795]: osdmap e79: 11 total, 10 up, 11 in 2024-05-15T12:45:24.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:24 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T12:45:24.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:24 smithi027 ceph-mon[39795]: from='osd.10 [v2:172.21.15.177:6816/3236900263,v1:172.21.15.177:6817/3236900263]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-05-15T12:45:24.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:24 smithi027 ceph-mon[39795]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-05-15T12:45:24.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:24 smithi120 ceph-mon[43961]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-15T12:45:24.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:24 smithi120 ceph-mon[43961]: osdmap e79: 11 total, 10 up, 11 in 2024-05-15T12:45:24.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T12:45:24.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:24 smithi120 ceph-mon[43961]: from='osd.10 [v2:172.21.15.177:6816/3236900263,v1:172.21.15.177:6817/3236900263]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-05-15T12:45:24.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:24 smithi120 ceph-mon[43961]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-05-15T12:45:25.259 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:25.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-05-15T12:45:25.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: osdmap e80: 11 total, 10 up, 11 in 2024-05-15T12:45:25.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T12:45:25.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T12:45:25.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {auth=21,config=2,health=12,kv=60,logm=5} crc {auth=2763500184,config=1548763834,health=3897446751,kv=3761881296,logm=2965911461}) 2024-05-15T12:45:25.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4216734556}) 2024-05-15T12:45:25.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2519803900}) 2024-05-15T12:45:25.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4053685217}) 2024-05-15T12:45:25.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1663780687}) 2024-05-15T12:45:25.260 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3728619581}) 2024-05-15T12:45:25.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2201484603}) 2024-05-15T12:45:25.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3441760356}) 2024-05-15T12:45:25.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2706724295}) 2024-05-15T12:45:25.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=452280410}) 2024-05-15T12:45:25.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2454620920}) 2024-05-15T12:45:25.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2088980125}) 2024-05-15T12:45:25.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=914606595}) 2024-05-15T12:45:25.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3992577438}) 2024-05-15T12:45:25.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3229352522}) 2024-05-15T12:45:25.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=56,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=3} crc {logm=1287696432,mdsmap=3984425268,mgr=76093979,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=2152968406,mon_config_key=2723414102}) 2024-05-15T12:45:25.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=107424801}) 2024-05-15T12:45:25.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=663993044}) 2024-05-15T12:45:25.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=63,monmap=5,osd_metadata=10,osd_pg_creating=1,osdmap=21} crc {mon_config_key=1642719236,monmap=2435304556,osd_metadata=2262644424,osd_pg_creating=4193607613,osdmap=3491440269}) 2024-05-15T12:45:25.262 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4264519513}) 2024-05-15T12:45:25.262 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=2} crc {osdmap=3502786757}) 2024-05-15T12:45:25.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:25.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-05-15T12:45:25.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: osdmap e80: 11 total, 10 up, 11 in 2024-05-15T12:45:25.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T12:45:25.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T12:45:25.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {auth=21,config=2,health=12,kv=60,logm=5} crc {auth=2763500184,config=1548763834,health=3897446751,kv=3761881296,logm=2965911461}) 2024-05-15T12:45:25.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4216734556}) 2024-05-15T12:45:25.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2519803900}) 2024-05-15T12:45:25.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4053685217}) 2024-05-15T12:45:25.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1663780687}) 2024-05-15T12:45:25.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3728619581}) 2024-05-15T12:45:25.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2201484603}) 2024-05-15T12:45:25.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3441760356}) 2024-05-15T12:45:25.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2706724295}) 2024-05-15T12:45:25.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=452280410}) 2024-05-15T12:45:25.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2454620920}) 2024-05-15T12:45:25.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2088980125}) 2024-05-15T12:45:25.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=914606595}) 2024-05-15T12:45:25.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3992577438}) 2024-05-15T12:45:25.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3229352522}) 2024-05-15T12:45:25.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=56,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=3} crc {logm=1287696432,mdsmap=3984425268,mgr=76093979,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=2152968406,mon_config_key=2723414102}) 2024-05-15T12:45:25.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=107424801}) 2024-05-15T12:45:25.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=663993044}) 2024-05-15T12:45:25.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=63,monmap=5,osd_metadata=10,osd_pg_creating=1,osdmap=21} crc {mon_config_key=1642719236,monmap=2435304556,osd_metadata=2262644424,osd_pg_creating=4193607613,osdmap=3491440269}) 2024-05-15T12:45:25.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4264519513}) 2024-05-15T12:45:25.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=2} crc {osdmap=3502786757}) 2024-05-15T12:45:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-05-15T12:45:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: osdmap e80: 11 total, 10 up, 11 in 2024-05-15T12:45:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T12:45:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T12:45:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {auth=21,config=2,health=12,kv=60,logm=5} crc {auth=2763500184,config=1548763834,health=3897446751,kv=3761881296,logm=2965911461}) 2024-05-15T12:45:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4216734556}) 2024-05-15T12:45:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2519803900}) 2024-05-15T12:45:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4053685217}) 2024-05-15T12:45:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1663780687}) 2024-05-15T12:45:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3728619581}) 2024-05-15T12:45:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2201484603}) 2024-05-15T12:45:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3441760356}) 2024-05-15T12:45:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2706724295}) 2024-05-15T12:45:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=452280410}) 2024-05-15T12:45:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2454620920}) 2024-05-15T12:45:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2088980125}) 2024-05-15T12:45:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=914606595}) 2024-05-15T12:45:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3992577438}) 2024-05-15T12:45:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3229352522}) 2024-05-15T12:45:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=56,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=3} crc {logm=1287696432,mdsmap=3984425268,mgr=76093979,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=2152968406,mon_config_key=2723414102}) 2024-05-15T12:45:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=107424801}) 2024-05-15T12:45:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=663993044}) 2024-05-15T12:45:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=63,monmap=5,osd_metadata=10,osd_pg_creating=1,osdmap=21} crc {mon_config_key=1642719236,monmap=2435304556,osd_metadata=2262644424,osd_pg_creating=4193607613,osdmap=3491440269}) 2024-05-15T12:45:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4264519513}) 2024-05-15T12:45:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=2} crc {osdmap=3502786757}) 2024-05-15T12:45:26.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:26 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:45:26.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:26 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:45:26.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:26 smithi027 ceph-mon[39795]: osd.10 [v2:172.21.15.177:6816/3236900263,v1:172.21.15.177:6817/3236900263] boot 2024-05-15T12:45:26.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:26 smithi027 ceph-mon[39795]: osdmap e81: 11 total, 11 up, 11 in 2024-05-15T12:45:26.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:26 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T12:45:26.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:26 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:45:26.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:26 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:45:26.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:26 smithi120 ceph-mon[43961]: osd.10 [v2:172.21.15.177:6816/3236900263,v1:172.21.15.177:6817/3236900263] boot 2024-05-15T12:45:26.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:26 smithi120 ceph-mon[43961]: osdmap e81: 11 total, 11 up, 11 in 2024-05-15T12:45:26.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:26 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T12:45:26.558 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:26 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:45:26.558 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:26 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:45:26.558 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:26 smithi177 ceph-mon[44303]: osd.10 [v2:172.21.15.177:6816/3236900263,v1:172.21.15.177:6817/3236900263] boot 2024-05-15T12:45:26.558 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:26 smithi177 ceph-mon[44303]: osdmap e81: 11 total, 11 up, 11 in 2024-05-15T12:45:26.558 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:26 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T12:45:27.485 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-15T12:45:27.485 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-15T12:45:27.485 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-05-15T12:45:27.485 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-05-15T12:45:27.485 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110145 s, 952 MB/s 2024-05-15T12:45:27.485 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-05-15T12:45:27.519 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:27 smithi177 ceph-mon[44303]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:27.519 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:27 smithi177 ceph-mon[44303]: osdmap e82: 11 total, 11 up, 11 in 2024-05-15T12:45:27.519 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:27 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:45:27.519 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:27 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:45:27.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:27 smithi027 ceph-mon[39795]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:27.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:27 smithi027 ceph-mon[39795]: osdmap e82: 11 total, 11 up, 11 in 2024-05-15T12:45:27.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:27 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:45:27.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:27 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:45:27.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:27 smithi120 ceph-mon[43961]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-05-15T12:45:27.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:27 smithi120 ceph-mon[43961]: osdmap e82: 11 total, 11 up, 11 in 2024-05-15T12:45:27.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:27 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:45:27.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:27 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:45:28.328 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:28 smithi177 ceph-mon[44303]: osdmap e83: 11 total, 11 up, 11 in 2024-05-15T12:45:28.329 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:28.329 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:28.329 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:28.329 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:28.329 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:28.329 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:28.329 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:28.329 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:45:28.329 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:28.355 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi177:vg_nvme/lv_1 2024-05-15T12:45:28.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:28 smithi027 ceph-mon[39795]: osdmap e83: 11 total, 11 up, 11 in 2024-05-15T12:45:28.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:28.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:28.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:28.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:28.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:28.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:28.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:28.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:45:28.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:28.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:28 smithi120 ceph-mon[43961]: osdmap e83: 11 total, 11 up, 11 in 2024-05-15T12:45:28.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:28.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:28.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:28.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:28.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:28.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:28.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:28.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:45:28.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:28.614 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:45:29.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:29 smithi027 ceph-mon[39795]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:29.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:29 smithi027 ceph-mon[39795]: Detected new or changed devices on smithi177 2024-05-15T12:45:29.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:29 smithi027 ceph-mon[39795]: Adjusting osd_memory_target on smithi177 to 7072M 2024-05-15T12:45:29.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:29 smithi120 ceph-mon[43961]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:29.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:29 smithi120 ceph-mon[43961]: Detected new or changed devices on smithi177 2024-05-15T12:45:29.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:29 smithi120 ceph-mon[43961]: Adjusting osd_memory_target on smithi177 to 7072M 2024-05-15T12:45:29.612 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:29 smithi177 ceph-mon[44303]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:29.613 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:29 smithi177 ceph-mon[44303]: Detected new or changed devices on smithi177 2024-05-15T12:45:29.613 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:29 smithi177 ceph-mon[44303]: Adjusting osd_memory_target on smithi177 to 7072M 2024-05-15T12:45:31.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:31 smithi177 ceph-mon[44303]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:31.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:31 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:45:31.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:31 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:45:31.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:31 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:31.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:31 smithi027 ceph-mon[39795]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:31.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:31 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:45:31.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:31 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:45:31.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:31 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:31.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:31 smithi120 ceph-mon[43961]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:31.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:31 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:45:31.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:31 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:45:31.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:31 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:32.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:32 smithi027 ceph-mon[39795]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:45:32.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:32 smithi120 ceph-mon[43961]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:45:32.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:32 smithi177 ceph-mon[44303]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:45:33.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:33 smithi027 ceph-mon[39795]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:33.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:33 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:45:33.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:33 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:45:33.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:33 smithi027 ceph-mon[39795]: osdmap e84: 11 total, 11 up, 11 in 2024-05-15T12:45:33.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:33 smithi027 ceph-mon[39795]: from='client.? 172.21.15.177:0/4251783677' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cccc53c5-8476-4d55-b160-4bdc5cf92902"}]: dispatch 2024-05-15T12:45:33.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:33 smithi027 ceph-mon[39795]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cccc53c5-8476-4d55-b160-4bdc5cf92902"}]: dispatch 2024-05-15T12:45:33.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:33 smithi027 ceph-mon[39795]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cccc53c5-8476-4d55-b160-4bdc5cf92902"}]': finished 2024-05-15T12:45:33.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:33 smithi027 ceph-mon[39795]: osdmap e85: 12 total, 11 up, 12 in 2024-05-15T12:45:33.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:33 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T12:45:33.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:33 smithi120 ceph-mon[43961]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:33.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:33 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:45:33.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:33 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:45:33.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:33 smithi120 ceph-mon[43961]: osdmap e84: 11 total, 11 up, 11 in 2024-05-15T12:45:33.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:33 smithi120 ceph-mon[43961]: from='client.? 172.21.15.177:0/4251783677' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cccc53c5-8476-4d55-b160-4bdc5cf92902"}]: dispatch 2024-05-15T12:45:33.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:33 smithi120 ceph-mon[43961]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cccc53c5-8476-4d55-b160-4bdc5cf92902"}]: dispatch 2024-05-15T12:45:33.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:33 smithi120 ceph-mon[43961]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cccc53c5-8476-4d55-b160-4bdc5cf92902"}]': finished 2024-05-15T12:45:33.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:33 smithi120 ceph-mon[43961]: osdmap e85: 12 total, 11 up, 12 in 2024-05-15T12:45:33.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:33 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T12:45:33.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:33 smithi177 ceph-mon[44303]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:33.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:33 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:45:33.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:33 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:45:33.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:33 smithi177 ceph-mon[44303]: osdmap e84: 11 total, 11 up, 11 in 2024-05-15T12:45:33.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:33 smithi177 ceph-mon[44303]: from='client.? 172.21.15.177:0/4251783677' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cccc53c5-8476-4d55-b160-4bdc5cf92902"}]: dispatch 2024-05-15T12:45:33.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:33 smithi177 ceph-mon[44303]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cccc53c5-8476-4d55-b160-4bdc5cf92902"}]: dispatch 2024-05-15T12:45:33.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:33 smithi177 ceph-mon[44303]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cccc53c5-8476-4d55-b160-4bdc5cf92902"}]': finished 2024-05-15T12:45:33.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:33 smithi177 ceph-mon[44303]: osdmap e85: 12 total, 11 up, 12 in 2024-05-15T12:45:33.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:33 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T12:45:34.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:34 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:45:34.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:34 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:45:34.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:34 smithi027 ceph-mon[39795]: from='client.? 172.21.15.177:0/3020621960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:45:34.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:34 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:45:34.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:34 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:45:34.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:34 smithi120 ceph-mon[43961]: from='client.? 172.21.15.177:0/3020621960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:45:34.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:34 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:45:34.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:34 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:45:34.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:34 smithi177 ceph-mon[44303]: from='client.? 172.21.15.177:0/3020621960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:45:35.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:35 smithi027 ceph-mon[39795]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:35.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:35 smithi120 ceph-mon[43961]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:35.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:35 smithi177 ceph-mon[44303]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:37.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:37 smithi027 ceph-mon[39795]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:37.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:37 smithi120 ceph-mon[43961]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:37.607 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:37 smithi177 ceph-mon[44303]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:38.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:38 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:45:38.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:38 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:45:38.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:38 smithi177 ceph-mon[44303]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:38.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:38 smithi177 ceph-mon[44303]: osdmap e86: 12 total, 11 up, 12 in 2024-05-15T12:45:38.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:38 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T12:45:39.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:38 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:45:39.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:38 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:45:39.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:38 smithi027 ceph-mon[39795]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:39.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:38 smithi027 ceph-mon[39795]: osdmap e86: 12 total, 11 up, 12 in 2024-05-15T12:45:39.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:38 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T12:45:39.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:38 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:45:39.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:38 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:45:39.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:38 smithi120 ceph-mon[43961]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:39.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:38 smithi120 ceph-mon[43961]: osdmap e86: 12 total, 11 up, 12 in 2024-05-15T12:45:39.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:38 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T12:45:40.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:40 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:45:40.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:40 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:45:40.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:40 smithi027 ceph-mon[39795]: osdmap e87: 12 total, 11 up, 12 in 2024-05-15T12:45:40.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:40 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T12:45:40.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:40 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-15T12:45:40.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:40 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:40.524 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:40 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:45:40.524 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:40 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:45:40.524 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:40 smithi177 ceph-mon[44303]: osdmap e87: 12 total, 11 up, 12 in 2024-05-15T12:45:40.525 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:40 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T12:45:40.525 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:40 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-15T12:45:40.525 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:40 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:40.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:40 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:45:40.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:40 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:45:40.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:40 smithi120 ceph-mon[43961]: osdmap e87: 12 total, 11 up, 12 in 2024-05-15T12:45:40.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:40 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T12:45:40.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:40 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-15T12:45:40.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:40 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:41.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:41 smithi027 ceph-mon[39795]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:41.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:41 smithi027 ceph-mon[39795]: Deploying daemon osd.11 on smithi177 2024-05-15T12:45:41.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:41 smithi120 ceph-mon[43961]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:41.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:41 smithi120 ceph-mon[43961]: Deploying daemon osd.11 on smithi177 2024-05-15T12:45:41.581 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:41 smithi177 ceph-mon[44303]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:41.581 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:41 smithi177 ceph-mon[44303]: Deploying daemon osd.11 on smithi177 2024-05-15T12:45:43.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:43 smithi027 ceph-mon[39795]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:43.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:43 smithi120 ceph-mon[43961]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:43.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:43 smithi177 ceph-mon[44303]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:45.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:45 smithi177 ceph-mon[44303]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:45.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:45 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:45.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:45 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:45.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:45 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:45:45.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:45 smithi120 ceph-mon[43961]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:45.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:45 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:45.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:45 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:45.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:45 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:45:45.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:45 smithi027 ceph-mon[39795]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:45.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:45 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:45.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:45 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:45.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:45 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:45:47.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:47 smithi120 ceph-mon[43961]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:47.546 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:47 smithi177 ceph-mon[44303]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:47.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:47 smithi027 ceph-mon[39795]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:48.803 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 11 on host 'smithi177' 2024-05-15T12:45:49.153 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:48 smithi177 ceph-mon[44303]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:49.153 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:48 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:49.153 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:48 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:49.153 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:48 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:49.154 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:48 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:45:49.154 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:48 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:49.154 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:48 smithi177 ceph-mon[44303]: from='osd.11 [v2:172.21.15.177:6824/681011628,v1:172.21.15.177:6825/681011628]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-15T12:45:49.154 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:48 smithi177 ceph-mon[44303]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-15T12:45:49.154 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:48 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:49.154 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:48 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:49.154 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:48 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:45:49.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:48 smithi027 ceph-mon[39795]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:49.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:48 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:49.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:48 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:49.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:48 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:49.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:48 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:45:49.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:48 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:49.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:48 smithi027 ceph-mon[39795]: from='osd.11 [v2:172.21.15.177:6824/681011628,v1:172.21.15.177:6825/681011628]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-15T12:45:49.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:48 smithi027 ceph-mon[39795]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-15T12:45:49.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:48 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:49.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:48 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:49.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:48 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:45:49.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:48 smithi120 ceph-mon[43961]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:49.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:48 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:49.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:48 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:49.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:48 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:49.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:48 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:45:49.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:48 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:49.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:48 smithi120 ceph-mon[43961]: from='osd.11 [v2:172.21.15.177:6824/681011628,v1:172.21.15.177:6825/681011628]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-15T12:45:49.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:48 smithi120 ceph-mon[43961]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-15T12:45:49.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:48 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:49.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:48 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:49.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:48 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:45:49.508 DEBUG:teuthology.orchestra.run.smithi177:osd.11> sudo journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.11.service 2024-05-15T12:45:49.510 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-05-15T12:45:49.511 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd stat -f json 2024-05-15T12:45:49.543 INFO:journalctl@ceph.osd.11.smithi177.stdout:-- Logs begin at Wed 2024-05-15 12:27:00 UTC. -- 2024-05-15T12:45:49.798 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:45:50.040 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:49 smithi177 ceph-mon[44303]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-15T12:45:50.040 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:49 smithi177 ceph-mon[44303]: osdmap e88: 12 total, 11 up, 12 in 2024-05-15T12:45:50.040 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:49 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T12:45:50.040 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:49 smithi177 ceph-mon[44303]: from='osd.11 [v2:172.21.15.177:6824/681011628,v1:172.21.15.177:6825/681011628]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-05-15T12:45:50.040 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:49 smithi177 ceph-mon[44303]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-05-15T12:45:50.040 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 12:45:49 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T12:45:49.877+0000 7fc21947a700 -1 osd.11 0 waiting for initial osdmap 2024-05-15T12:45:50.040 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 12:45:49 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T12:45:49.878+0000 7fc210c08700 -1 osd.11 0 failed to load OSD map for epoch 34, got 0 bytes 2024-05-15T12:45:50.040 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 12:45:49 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T12:45:49.890+0000 7fc21526d700 -1 osd.11 89 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:45:50.130 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:49 smithi027 ceph-mon[39795]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-15T12:45:50.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:49 smithi027 ceph-mon[39795]: osdmap e88: 12 total, 11 up, 12 in 2024-05-15T12:45:50.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:49 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T12:45:50.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:49 smithi027 ceph-mon[39795]: from='osd.11 [v2:172.21.15.177:6824/681011628,v1:172.21.15.177:6825/681011628]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-05-15T12:45:50.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:49 smithi027 ceph-mon[39795]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-05-15T12:45:50.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:49 smithi120 ceph-mon[43961]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-15T12:45:50.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:49 smithi120 ceph-mon[43961]: osdmap e88: 12 total, 11 up, 12 in 2024-05-15T12:45:50.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:49 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T12:45:50.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:49 smithi120 ceph-mon[43961]: from='osd.11 [v2:172.21.15.177:6824/681011628,v1:172.21.15.177:6825/681011628]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-05-15T12:45:50.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:49 smithi120 ceph-mon[43961]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-05-15T12:45:51.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:50 smithi177 ceph-mon[44303]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:51.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:50 smithi177 ceph-mon[44303]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-05-15T12:45:51.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:50 smithi177 ceph-mon[44303]: osdmap e89: 12 total, 11 up, 12 in 2024-05-15T12:45:51.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:50 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T12:45:51.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:50 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T12:45:51.264 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:50 smithi027 ceph-mon[39795]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:51.265 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:50 smithi027 ceph-mon[39795]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-05-15T12:45:51.265 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:50 smithi027 ceph-mon[39795]: osdmap e89: 12 total, 11 up, 12 in 2024-05-15T12:45:51.265 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:50 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T12:45:51.265 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:50 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T12:45:51.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:50 smithi120 ceph-mon[43961]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:51.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:50 smithi120 ceph-mon[43961]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-05-15T12:45:51.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:50 smithi120 ceph-mon[43961]: osdmap e89: 12 total, 11 up, 12 in 2024-05-15T12:45:51.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:50 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T12:45:51.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:50 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T12:45:52.092 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:45:52.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:51 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:45:52.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:51 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:45:52.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:51 smithi177 ceph-mon[44303]: osd.11 [v2:172.21.15.177:6824/681011628,v1:172.21.15.177:6825/681011628] boot 2024-05-15T12:45:52.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:51 smithi177 ceph-mon[44303]: osdmap e90: 12 total, 12 up, 12 in 2024-05-15T12:45:52.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:51 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T12:45:52.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:51 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:45:52.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:51 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:45:52.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:51 smithi027 ceph-mon[39795]: osd.11 [v2:172.21.15.177:6824/681011628,v1:172.21.15.177:6825/681011628] boot 2024-05-15T12:45:52.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:51 smithi027 ceph-mon[39795]: osdmap e90: 12 total, 12 up, 12 in 2024-05-15T12:45:52.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:51 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T12:45:52.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:51 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:45:52.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:51 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:45:52.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:51 smithi120 ceph-mon[43961]: osd.11 [v2:172.21.15.177:6824/681011628,v1:172.21.15.177:6825/681011628] boot 2024-05-15T12:45:52.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:51 smithi120 ceph-mon[43961]: osdmap e90: 12 total, 12 up, 12 in 2024-05-15T12:45:52.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:51 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T12:45:52.914 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":91,"num_osds":12,"num_up_osds":12,"osd_up_since":1715777150,"num_in_osds":12,"osd_in_since":1715777133,"num_remapped_pgs":1} 2024-05-15T12:45:52.915 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-15T12:45:53.183 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:52 smithi177 ceph-mon[44303]: pgmap v276: 1 pgs: 1 peering; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:53.183 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:52 smithi177 ceph-mon[44303]: osdmap e91: 12 total, 12 up, 12 in 2024-05-15T12:45:53.183 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:52 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2998739558' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-15T12:45:53.183 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:52 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:45:53.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:52 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:45:53.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:52 smithi177 ceph-mon[44303]: Detected new or changed devices on smithi177 2024-05-15T12:45:53.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:52 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:53.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:52 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:53.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:52 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:53.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:52 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:53.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:52 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:53.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:52 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:53.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:52 smithi177 ceph-mon[44303]: Adjusting osd_memory_target on smithi177 to 5304M 2024-05-15T12:45:53.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:52 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:53.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:52 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:45:53.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:52 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:45:53.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:52 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:45:53.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:52 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:53.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:52 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:45:53.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:52 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:53.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:52 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:45:53.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:52 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:45:53.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:52 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:45:53.206 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:45:53.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:52 smithi027 ceph-mon[39795]: pgmap v276: 1 pgs: 1 peering; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:53.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:52 smithi027 ceph-mon[39795]: osdmap e91: 12 total, 12 up, 12 in 2024-05-15T12:45:53.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:52 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2998739558' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-15T12:45:53.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:52 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:45:53.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:52 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:45:53.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:52 smithi027 ceph-mon[39795]: Detected new or changed devices on smithi177 2024-05-15T12:45:53.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:52 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:53.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:52 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:53.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:52 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:53.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:52 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:53.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:52 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:53.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:52 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:53.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:52 smithi027 ceph-mon[39795]: Adjusting osd_memory_target on smithi177 to 5304M 2024-05-15T12:45:53.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:52 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:53.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:52 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:45:53.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:52 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:45:53.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:52 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:45:53.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:52 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:53.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:52 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:45:53.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:52 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:53.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:52 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:45:53.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:52 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:45:53.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:52 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:45:53.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:52 smithi120 ceph-mon[43961]: pgmap v276: 1 pgs: 1 peering; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-05-15T12:45:53.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:52 smithi120 ceph-mon[43961]: osdmap e91: 12 total, 12 up, 12 in 2024-05-15T12:45:53.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:52 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2998739558' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-15T12:45:53.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:52 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:45:53.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:52 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:45:53.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:52 smithi120 ceph-mon[43961]: Detected new or changed devices on smithi177 2024-05-15T12:45:53.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:52 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:53.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:52 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:53.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:52 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:53.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:52 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:53.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:52 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:53.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:52 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:45:53.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:52 smithi120 ceph-mon[43961]: Adjusting osd_memory_target on smithi177 to 5304M 2024-05-15T12:45:53.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:52 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:53.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:52 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:45:53.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:52 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:45:53.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:52 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:45:53.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:52 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:53.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:52 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:45:53.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:52 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:53.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:52 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:45:53.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:52 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:45:53.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:52 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:45:54.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:53 smithi177 ceph-mon[44303]: osdmap e92: 12 total, 12 up, 12 in 2024-05-15T12:45:54.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:53 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:45:54.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:53 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:54.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:53 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:45:54.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:53 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:54.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:53 smithi027 ceph-mon[39795]: osdmap e92: 12 total, 12 up, 12 in 2024-05-15T12:45:54.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:53 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:45:54.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:53 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:54.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:53 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:45:54.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:53 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:54.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:53 smithi120 ceph-mon[43961]: osdmap e92: 12 total, 12 up, 12 in 2024-05-15T12:45:54.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:53 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:45:54.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:53 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:45:54.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:53 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:45:54.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:53 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:45:55.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:55 smithi027 ceph-mon[39795]: pgmap v279: 1 pgs: 1 peering; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:45:55.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:55 smithi027 ceph-mon[39795]: osdmap e93: 12 total, 12 up, 12 in 2024-05-15T12:45:55.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:55 smithi120 ceph-mon[43961]: pgmap v279: 1 pgs: 1 peering; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:45:55.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:55 smithi120 ceph-mon[43961]: osdmap e93: 12 total, 12 up, 12 in 2024-05-15T12:45:55.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:55 smithi177 ceph-mon[44303]: pgmap v279: 1 pgs: 1 peering; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:45:55.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:55 smithi177 ceph-mon[44303]: osdmap e93: 12 total, 12 up, 12 in 2024-05-15T12:45:55.755 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:45:55.755 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":94,"fsid":"deaea04e-12b7-11ef-bc98-c7b262605968","created":"2024-05-15T12:38:21.244556+0000","modified":"2024-05-15T12:45:55.003011+0000","last_up_change":"2024-05-15T12:45:50.876090+0000","last_in_change":"2024-05-15T12:45:33.158047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T12:41:57.948613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"64fb3ad2-08a7-4106-a07a-6c776c877faf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4239116935},{"type":"v1","addr":"172.21.15.27:6803","nonce":4239116935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4239116935},{"type":"v1","addr":"172.21.15.27:6805","nonce":4239116935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4239116935},{"type":"v1","addr":"172.21.15.27:6809","nonce":4239116935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4239116935},{"type":"v1","addr":"172.21.15.27:6807","nonce":4239116935}]},"public_addr":"172.21.15.27:6803/4239116935","cluster_addr":"172.21.15.27:6805/4239116935","heartbeat_back_addr":"172.21.15.27:6809/4239116935","heartbeat_front_addr":"172.21.15.27:6807/4239116935","state":["exists","up"]},{"osd":1,"uuid":"9c16dd90-9d51-48b8-a296-1c8879ba1bc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1466270637},{"type":"v1","addr":"172.21.15.27:6811","nonce":1466270637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1466270637},{"type":"v1","addr":"172.21.15.27:6813","nonce":1466270637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1466270637},{"type":"v1","addr":"172.21.15.27:6817","nonce":1466270637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1466270637},{"type":"v1","addr":"172.21.15.27:6815","nonce":1466270637}]},"public_addr":"172.21.15.27:6811/1466270637","cluster_addr":"172.21.15.27:6813/1466270637","heartbeat_back_addr":"172.21.15.27:6817/1466270637","heartbeat_front_addr":"172.21.15.27:6815/1466270637","state":["exists","up"]},{"osd":2,"uuid":"cbe3ba13-3ea3-4891-8d2d-6b85581a732c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3285413897},{"type":"v1","addr":"172.21.15.27:6819","nonce":3285413897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3285413897},{"type":"v1","addr":"172.21.15.27:6821","nonce":3285413897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3285413897},{"type":"v1","addr":"172.21.15.27:6825","nonce":3285413897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3285413897},{"type":"v1","addr":"172.21.15.27:6823","nonce":3285413897}]},"public_addr":"172.21.15.27:6819/3285413897","cluster_addr":"172.21.15.27:6821/3285413897","heartbeat_back_addr":"172.21.15.27:6825/3285413897","heartbeat_front_addr":"172.21.15.27:6823/3285413897","state":["exists","up"]},{"osd":3,"uuid":"2e83735b-b944-4f87-8080-b994d4b96897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3008975244},{"type":"v1","addr":"172.21.15.27:6827","nonce":3008975244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3008975244},{"type":"v1","addr":"172.21.15.27:6829","nonce":3008975244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3008975244},{"type":"v1","addr":"172.21.15.27:6833","nonce":3008975244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3008975244},{"type":"v1","addr":"172.21.15.27:6831","nonce":3008975244}]},"public_addr":"172.21.15.27:6827/3008975244","cluster_addr":"172.21.15.27:6829/3008975244","heartbeat_back_addr":"172.21.15.27:6833/3008975244","heartbeat_front_addr":"172.21.15.27:6831/3008975244","state":["exists","up"]},{"osd":4,"uuid":"7c39c1dd-3867-46f7-8372-7688fa13b16a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3877842957},{"type":"v1","addr":"172.21.15.120:6801","nonce":3877842957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3877842957},{"type":"v1","addr":"172.21.15.120:6803","nonce":3877842957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3877842957},{"type":"v1","addr":"172.21.15.120:6807","nonce":3877842957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3877842957},{"type":"v1","addr":"172.21.15.120:6805","nonce":3877842957}]},"public_addr":"172.21.15.120:6801/3877842957","cluster_addr":"172.21.15.120:6803/3877842957","heartbeat_back_addr":"172.21.15.120:6807/3877842957","heartbeat_front_addr":"172.21.15.120:6805/3877842957","state":["exists","up"]},{"osd":5,"uuid":"a6dc220f-aec0-4454-9695-b44c0b9770e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1600179183},{"type":"v1","addr":"172.21.15.120:6809","nonce":1600179183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1600179183},{"type":"v1","addr":"172.21.15.120:6811","nonce":1600179183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1600179183},{"type":"v1","addr":"172.21.15.120:6815","nonce":1600179183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1600179183},{"type":"v1","addr":"172.21.15.120:6813","nonce":1600179183}]},"public_addr":"172.21.15.120:6809/1600179183","cluster_addr":"172.21.15.120:6811/1600179183","heartbeat_back_addr":"172.21.15.120:6815/1600179183","heartbeat_front_addr":"172.21.15.120:6813/1600179183","state":["exists","up"]},{"osd":6,"uuid":"9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":3323706735},{"type":"v1","addr":"172.21.15.120:6817","nonce":3323706735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":3323706735},{"type":"v1","addr":"172.21.15.120:6819","nonce":3323706735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":3323706735},{"type":"v1","addr":"172.21.15.120:6823","nonce":3323706735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":3323706735},{"type":"v1","addr":"172.21.15.120:6821","nonce":3323706735}]},"public_addr":"172.21.15.120:6817/3323706735","cluster_addr":"172.21.15.120:6819/3323706735","heartbeat_back_addr":"172.21.15.120:6823/3323706735","heartbeat_front_addr":"172.21.15.120:6821/3323706735","state":["exists","up"]},{"osd":7,"uuid":"6ef085d8-f59b-4033-8280-2c8f78693e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2960616153},{"type":"v1","addr":"172.21.15.120:6825","nonce":2960616153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2960616153},{"type":"v1","addr":"172.21.15.120:6827","nonce":2960616153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2960616153},{"type":"v1","addr":"172.21.15.120:6831","nonce":2960616153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2960616153},{"type":"v1","addr":"172.21.15.120:6829","nonce":2960616153}]},"public_addr":"172.21.15.120:6825/2960616153","cluster_addr":"172.21.15.120:6827/2960616153","heartbeat_back_addr":"172.21.15.120:6831/2960616153","heartbeat_front_addr":"172.21.15.120:6829/2960616153","state":["exists","up"]},{"osd":8,"uuid":"26514b11-d02b-46e9-a4e8-3da193fcdbcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1132459315},{"type":"v1","addr":"172.21.15.177:6801","nonce":1132459315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1132459315},{"type":"v1","addr":"172.21.15.177:6803","nonce":1132459315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1132459315},{"type":"v1","addr":"172.21.15.177:6807","nonce":1132459315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1132459315},{"type":"v1","addr":"172.21.15.177:6805","nonce":1132459315}]},"public_addr":"172.21.15.177:6801/1132459315","cluster_addr":"172.21.15.177:6803/1132459315","heartbeat_back_addr":"172.21.15.177:6807/1132459315","heartbeat_front_addr":"172.21.15.177:6805/1132459315","state":["exists","up"]},{"osd":9,"uuid":"6f9497bc-e962-409d-8a2c-0c634cc1124f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2710457646},{"type":"v1","addr":"172.21.15.177:6809","nonce":2710457646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2710457646},{"type":"v1","addr":"172.21.15.177:6811","nonce":2710457646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2710457646},{"type":"v1","addr":"172.21.15.177:6815","nonce":2710457646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2710457646},{"type":"v1","addr":"172.21.15.177:6813","nonce":2710457646}]},"public_addr":"172.21.15.177:6809/2710457646","cluster_addr":"172.21.15.177:6811/2710457646","heartbeat_back_addr":"172.21.15.177:6815/2710457646","heartbeat_front_addr":"172.21.15.177:6813/2710457646","state":["exists","up"]},{"osd":10,"uuid":"f852876f-c7d1-4039-90c4-5ab09dbd3e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":3236900263},{"type":"v1","addr":"172.21.15.177:6817","nonce":3236900263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3236900263},{"type":"v1","addr":"172.21.15.177:6819","nonce":3236900263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3236900263},{"type":"v1","addr":"172.21.15.177:6823","nonce":3236900263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3236900263},{"type":"v1","addr":"172.21.15.177:6821","nonce":3236900263}]},"public_addr":"172.21.15.177:6817/3236900263","cluster_addr":"172.21.15.177:6819/3236900263","heartbeat_back_addr":"172.21.15.177:6823/3236900263","heartbeat_front_addr":"172.21.15.177:6821/3236900263","state":["exists","up"]},{"osd":11,"uuid":"cccc53c5-8476-4d55-b160-4bdc5cf92902","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":90,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":681011628},{"type":"v1","addr":"172.21.15.177:6825","nonce":681011628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":681011628},{"type":"v1","addr":"172.21.15.177:6827","nonce":681011628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":681011628},{"type":"v1","addr":"172.21.15.177:6831","nonce":681011628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":681011628},{"type":"v1","addr":"172.21.15.177:6829","nonce":681011628}]},"public_addr":"172.21.15.177:6825/681011628","cluster_addr":"172.21.15.177:6827/681011628","heartbeat_back_addr":"172.21.15.177:6831/681011628","heartbeat_front_addr":"172.21.15.177:6829/681011628","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:44:42.917233+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:45:38.748016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:45:37.332618+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:44:48.221317+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:45:26.363198+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:45:32.127420+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:44:49.578317+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:45:32.922550+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:45:52.327609+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:44:57.537707+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:45:23.318781+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:45:49.287805+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6801/2530617473":"2024-05-16T12:39:19.409195+0000","172.21.15.27:6800/2530617473":"2024-05-16T12:39:19.409195+0000","172.21.15.27:0/1880261204":"2024-05-16T12:39:19.409195+0000","172.21.15.27:6801/3441136471":"2024-05-16T12:38:50.931011+0000","172.21.15.27:0/4291446247":"2024-05-16T12:39:19.409195+0000","172.21.15.27:0/580128209":"2024-05-16T12:38:50.931011+0000","172.21.15.27:6800/3441136471":"2024-05-16T12:38:50.931011+0000","172.21.15.27:0/3011229857":"2024-05-16T12:38:50.931011+0000","172.21.15.27:0/2109693913":"2024-05-16T12:38:50.931011+0000","172.21.15.27:0/2445663024":"2024-05-16T12:39:19.409195+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:45:56.204 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:56 smithi027 ceph-mon[39795]: osdmap e94: 12 total, 12 up, 12 in 2024-05-15T12:45:56.205 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:56 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/828172551' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T12:45:56.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:56 smithi120 ceph-mon[43961]: osdmap e94: 12 total, 12 up, 12 in 2024-05-15T12:45:56.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:56 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/828172551' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T12:45:56.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:56 smithi177 ceph-mon[44303]: osdmap e94: 12 total, 12 up, 12 in 2024-05-15T12:45:56.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:56 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/828172551' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T12:45:56.562 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-15T12:41:57.948613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-15T12:45:56.563 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-15T12:45:56.842 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:45:57.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:57 smithi027 ceph-mon[39795]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s, 0 objects/s recovering 2024-05-15T12:45:57.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:57 smithi120 ceph-mon[43961]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s, 0 objects/s recovering 2024-05-15T12:45:57.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:57 smithi177 ceph-mon[44303]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s, 0 objects/s recovering 2024-05-15T12:45:58.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:58 smithi027 ceph-mon[39795]: 1.0 deep-scrub starts 2024-05-15T12:45:58.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:58 smithi027 ceph-mon[39795]: 1.0 deep-scrub ok 2024-05-15T12:45:58.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:58 smithi120 ceph-mon[43961]: 1.0 deep-scrub starts 2024-05-15T12:45:58.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:58 smithi120 ceph-mon[43961]: 1.0 deep-scrub ok 2024-05-15T12:45:58.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:58 smithi177 ceph-mon[44303]: 1.0 deep-scrub starts 2024-05-15T12:45:58.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:58 smithi177 ceph-mon[44303]: 1.0 deep-scrub ok 2024-05-15T12:45:59.162 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-05-15T12:45:59.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:45:59 smithi027 ceph-mon[39795]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s, 0 objects/s recovering 2024-05-15T12:45:59.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:45:59 smithi120 ceph-mon[43961]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s, 0 objects/s recovering 2024-05-15T12:45:59.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:45:59 smithi177 ceph-mon[44303]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s, 0 objects/s recovering 2024-05-15T12:46:00.043 INFO:tasks.cephadm:Setting up client nodes... 2024-05-15T12:46:00.043 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-15T12:46:00.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:00 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3256150294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-15T12:46:00.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:00 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3256150294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-15T12:46:00.323 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:00 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3256150294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-15T12:46:01.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:01 smithi120 ceph-mon[43961]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s, 0 objects/s recovering 2024-05-15T12:46:01.301 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:01 smithi027 ceph-mon[39795]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s, 0 objects/s recovering 2024-05-15T12:46:01.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:01 smithi177 ceph-mon[44303]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s, 0 objects/s recovering 2024-05-15T12:46:02.698 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-05-15T12:46:02.698 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCKrkRm0AMjKhAAbPVFekdFGNkV7Uvspj8uOQ== 2024-05-15T12:46:03.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:03 smithi120 ceph-mon[43961]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-15T12:46:03.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:03 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2944768105' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T12:46:03.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:03 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2944768105' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-15T12:46:03.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:03 smithi177 ceph-mon[44303]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-15T12:46:03.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:03 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2944768105' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T12:46:03.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:03 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2944768105' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-15T12:46:03.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:03 smithi027 ceph-mon[39795]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-15T12:46:03.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:03 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2944768105' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T12:46:03.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:03 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2944768105' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-15T12:46:03.562 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-05-15T12:46:03.563 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-15T12:46:03.563 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-15T12:46:03.607 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-15T12:46:03.882 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:46:05.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:05 smithi120 ceph-mon[43961]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s, 0 objects/s recovering 2024-05-15T12:46:05.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:05 smithi177 ceph-mon[44303]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s, 0 objects/s recovering 2024-05-15T12:46:05.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:05 smithi027 ceph-mon[39795]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s, 0 objects/s recovering 2024-05-15T12:46:06.089 INFO:teuthology.orchestra.run.smithi120.stdout:[client.1] 2024-05-15T12:46:06.089 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQCOrkRmz8OzBRAAun7rUPxNVdP8w9cOqjwzpw== 2024-05-15T12:46:06.791 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-05-15T12:46:06.791 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-05-15T12:46:06.791 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-05-15T12:46:06.832 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-15T12:46:07.107 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:46:07.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:07 smithi177 ceph-mon[44303]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s, 0 objects/s recovering 2024-05-15T12:46:07.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:07 smithi177 ceph-mon[44303]: from='client.? 172.21.15.120:0/1706851128' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T12:46:07.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:07 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T12:46:07.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:07 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-15T12:46:07.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:07 smithi027 ceph-mon[39795]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s, 0 objects/s recovering 2024-05-15T12:46:07.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:07 smithi027 ceph-mon[39795]: from='client.? 172.21.15.120:0/1706851128' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T12:46:07.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:07 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T12:46:07.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:07 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-15T12:46:07.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:07 smithi120 ceph-mon[43961]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s, 0 objects/s recovering 2024-05-15T12:46:07.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:07 smithi120 ceph-mon[43961]: from='client.? 172.21.15.120:0/1706851128' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T12:46:07.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:07 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T12:46:07.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:07 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-15T12:46:08.279 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:08 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:46:08.280 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:08 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:46:08.280 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:08 smithi177 ceph-mon[44303]: osdmap e95: 12 total, 12 up, 12 in 2024-05-15T12:46:08.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:08 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:46:08.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:08 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:46:08.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:08 smithi027 ceph-mon[39795]: osdmap e95: 12 total, 12 up, 12 in 2024-05-15T12:46:08.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:08 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:46:08.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:08 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:46:08.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:08 smithi120 ceph-mon[43961]: osdmap e95: 12 total, 12 up, 12 in 2024-05-15T12:46:09.092 INFO:teuthology.orchestra.run.smithi177.stdout:[client.2] 2024-05-15T12:46:09.092 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQCRrkRmRG/RBRAAvO9f1PPi8InYbajatzENow== 2024-05-15T12:46:09.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:09 smithi177 ceph-mon[44303]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:09.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:09 smithi177 ceph-mon[44303]: from='client.? 172.21.15.177:0/3569727569' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T12:46:09.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:09 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T12:46:09.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:09 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-15T12:46:09.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:09 smithi027 ceph-mon[39795]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:09.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:09 smithi027 ceph-mon[39795]: from='client.? 172.21.15.177:0/3569727569' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T12:46:09.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:09 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T12:46:09.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:09 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-15T12:46:09.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:09 smithi120 ceph-mon[43961]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:09.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:09 smithi120 ceph-mon[43961]: from='client.? 172.21.15.177:0/3569727569' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T12:46:09.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:09 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T12:46:09.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:09 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-15T12:46:09.761 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-05-15T12:46:09.761 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-05-15T12:46:09.761 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-05-15T12:46:09.801 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-15T12:46:09.801 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-15T12:46:09.801 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph mgr dump --format=json 2024-05-15T12:46:10.087 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:11.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:11 smithi177 ceph-mon[44303]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:11.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:11 smithi027 ceph-mon[39795]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:11.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:11 smithi120 ceph-mon[43961]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:12.421 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:46:13.347 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:13 smithi027 ceph-mon[39795]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:13.347 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:13 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:46:13.348 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:13 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:46:13.348 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:13 smithi027 ceph-mon[39795]: osdmap e96: 12 total, 12 up, 12 in 2024-05-15T12:46:13.348 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:13 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2633144232' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-15T12:46:13.378 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"flags":0,"active_gid":14148,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3328392788},{"type":"v1","addr":"172.21.15.27:6801","nonce":3328392788}]},"active_addr":"172.21.15.27:6801/3328392788","active_change":"2024-05-15T12:39:19.409347+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24116,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.27:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":636811177}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":2912795986}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":2080608953}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":752459743}]}]} 2024-05-15T12:46:13.383 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-15T12:46:13.383 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-15T12:46:13.384 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-15T12:46:13.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:13 smithi177 ceph-mon[44303]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:13.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:13 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:46:13.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:13 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:46:13.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:13 smithi177 ceph-mon[44303]: osdmap e96: 12 total, 12 up, 12 in 2024-05-15T12:46:13.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:13 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2633144232' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-15T12:46:13.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:13 smithi120 ceph-mon[43961]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:13.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:13 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:46:13.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:13 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:46:13.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:13 smithi120 ceph-mon[43961]: osdmap e96: 12 total, 12 up, 12 in 2024-05-15T12:46:13.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:13 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2633144232' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-15T12:46:13.663 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:15.379 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:15 smithi027 ceph-mon[39795]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:15.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:15 smithi177 ceph-mon[44303]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:15.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:15 smithi120 ceph-mon[43961]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:15.866 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:46:15.866 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":96,"fsid":"deaea04e-12b7-11ef-bc98-c7b262605968","created":"2024-05-15T12:38:21.244556+0000","modified":"2024-05-15T12:46:12.108227+0000","last_up_change":"2024-05-15T12:45:50.876090+0000","last_in_change":"2024-05-15T12:45:33.158047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T12:41:57.948613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"64fb3ad2-08a7-4106-a07a-6c776c877faf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4239116935},{"type":"v1","addr":"172.21.15.27:6803","nonce":4239116935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4239116935},{"type":"v1","addr":"172.21.15.27:6805","nonce":4239116935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4239116935},{"type":"v1","addr":"172.21.15.27:6809","nonce":4239116935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4239116935},{"type":"v1","addr":"172.21.15.27:6807","nonce":4239116935}]},"public_addr":"172.21.15.27:6803/4239116935","cluster_addr":"172.21.15.27:6805/4239116935","heartbeat_back_addr":"172.21.15.27:6809/4239116935","heartbeat_front_addr":"172.21.15.27:6807/4239116935","state":["exists","up"]},{"osd":1,"uuid":"9c16dd90-9d51-48b8-a296-1c8879ba1bc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1466270637},{"type":"v1","addr":"172.21.15.27:6811","nonce":1466270637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1466270637},{"type":"v1","addr":"172.21.15.27:6813","nonce":1466270637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1466270637},{"type":"v1","addr":"172.21.15.27:6817","nonce":1466270637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1466270637},{"type":"v1","addr":"172.21.15.27:6815","nonce":1466270637}]},"public_addr":"172.21.15.27:6811/1466270637","cluster_addr":"172.21.15.27:6813/1466270637","heartbeat_back_addr":"172.21.15.27:6817/1466270637","heartbeat_front_addr":"172.21.15.27:6815/1466270637","state":["exists","up"]},{"osd":2,"uuid":"cbe3ba13-3ea3-4891-8d2d-6b85581a732c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3285413897},{"type":"v1","addr":"172.21.15.27:6819","nonce":3285413897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3285413897},{"type":"v1","addr":"172.21.15.27:6821","nonce":3285413897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3285413897},{"type":"v1","addr":"172.21.15.27:6825","nonce":3285413897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3285413897},{"type":"v1","addr":"172.21.15.27:6823","nonce":3285413897}]},"public_addr":"172.21.15.27:6819/3285413897","cluster_addr":"172.21.15.27:6821/3285413897","heartbeat_back_addr":"172.21.15.27:6825/3285413897","heartbeat_front_addr":"172.21.15.27:6823/3285413897","state":["exists","up"]},{"osd":3,"uuid":"2e83735b-b944-4f87-8080-b994d4b96897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3008975244},{"type":"v1","addr":"172.21.15.27:6827","nonce":3008975244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3008975244},{"type":"v1","addr":"172.21.15.27:6829","nonce":3008975244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3008975244},{"type":"v1","addr":"172.21.15.27:6833","nonce":3008975244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3008975244},{"type":"v1","addr":"172.21.15.27:6831","nonce":3008975244}]},"public_addr":"172.21.15.27:6827/3008975244","cluster_addr":"172.21.15.27:6829/3008975244","heartbeat_back_addr":"172.21.15.27:6833/3008975244","heartbeat_front_addr":"172.21.15.27:6831/3008975244","state":["exists","up"]},{"osd":4,"uuid":"7c39c1dd-3867-46f7-8372-7688fa13b16a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3877842957},{"type":"v1","addr":"172.21.15.120:6801","nonce":3877842957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3877842957},{"type":"v1","addr":"172.21.15.120:6803","nonce":3877842957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3877842957},{"type":"v1","addr":"172.21.15.120:6807","nonce":3877842957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3877842957},{"type":"v1","addr":"172.21.15.120:6805","nonce":3877842957}]},"public_addr":"172.21.15.120:6801/3877842957","cluster_addr":"172.21.15.120:6803/3877842957","heartbeat_back_addr":"172.21.15.120:6807/3877842957","heartbeat_front_addr":"172.21.15.120:6805/3877842957","state":["exists","up"]},{"osd":5,"uuid":"a6dc220f-aec0-4454-9695-b44c0b9770e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1600179183},{"type":"v1","addr":"172.21.15.120:6809","nonce":1600179183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1600179183},{"type":"v1","addr":"172.21.15.120:6811","nonce":1600179183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1600179183},{"type":"v1","addr":"172.21.15.120:6815","nonce":1600179183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1600179183},{"type":"v1","addr":"172.21.15.120:6813","nonce":1600179183}]},"public_addr":"172.21.15.120:6809/1600179183","cluster_addr":"172.21.15.120:6811/1600179183","heartbeat_back_addr":"172.21.15.120:6815/1600179183","heartbeat_front_addr":"172.21.15.120:6813/1600179183","state":["exists","up"]},{"osd":6,"uuid":"9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":3323706735},{"type":"v1","addr":"172.21.15.120:6817","nonce":3323706735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":3323706735},{"type":"v1","addr":"172.21.15.120:6819","nonce":3323706735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":3323706735},{"type":"v1","addr":"172.21.15.120:6823","nonce":3323706735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":3323706735},{"type":"v1","addr":"172.21.15.120:6821","nonce":3323706735}]},"public_addr":"172.21.15.120:6817/3323706735","cluster_addr":"172.21.15.120:6819/3323706735","heartbeat_back_addr":"172.21.15.120:6823/3323706735","heartbeat_front_addr":"172.21.15.120:6821/3323706735","state":["exists","up"]},{"osd":7,"uuid":"6ef085d8-f59b-4033-8280-2c8f78693e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2960616153},{"type":"v1","addr":"172.21.15.120:6825","nonce":2960616153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2960616153},{"type":"v1","addr":"172.21.15.120:6827","nonce":2960616153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2960616153},{"type":"v1","addr":"172.21.15.120:6831","nonce":2960616153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2960616153},{"type":"v1","addr":"172.21.15.120:6829","nonce":2960616153}]},"public_addr":"172.21.15.120:6825/2960616153","cluster_addr":"172.21.15.120:6827/2960616153","heartbeat_back_addr":"172.21.15.120:6831/2960616153","heartbeat_front_addr":"172.21.15.120:6829/2960616153","state":["exists","up"]},{"osd":8,"uuid":"26514b11-d02b-46e9-a4e8-3da193fcdbcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1132459315},{"type":"v1","addr":"172.21.15.177:6801","nonce":1132459315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1132459315},{"type":"v1","addr":"172.21.15.177:6803","nonce":1132459315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1132459315},{"type":"v1","addr":"172.21.15.177:6807","nonce":1132459315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1132459315},{"type":"v1","addr":"172.21.15.177:6805","nonce":1132459315}]},"public_addr":"172.21.15.177:6801/1132459315","cluster_addr":"172.21.15.177:6803/1132459315","heartbeat_back_addr":"172.21.15.177:6807/1132459315","heartbeat_front_addr":"172.21.15.177:6805/1132459315","state":["exists","up"]},{"osd":9,"uuid":"6f9497bc-e962-409d-8a2c-0c634cc1124f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2710457646},{"type":"v1","addr":"172.21.15.177:6809","nonce":2710457646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2710457646},{"type":"v1","addr":"172.21.15.177:6811","nonce":2710457646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2710457646},{"type":"v1","addr":"172.21.15.177:6815","nonce":2710457646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2710457646},{"type":"v1","addr":"172.21.15.177:6813","nonce":2710457646}]},"public_addr":"172.21.15.177:6809/2710457646","cluster_addr":"172.21.15.177:6811/2710457646","heartbeat_back_addr":"172.21.15.177:6815/2710457646","heartbeat_front_addr":"172.21.15.177:6813/2710457646","state":["exists","up"]},{"osd":10,"uuid":"f852876f-c7d1-4039-90c4-5ab09dbd3e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":3236900263},{"type":"v1","addr":"172.21.15.177:6817","nonce":3236900263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3236900263},{"type":"v1","addr":"172.21.15.177:6819","nonce":3236900263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3236900263},{"type":"v1","addr":"172.21.15.177:6823","nonce":3236900263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3236900263},{"type":"v1","addr":"172.21.15.177:6821","nonce":3236900263}]},"public_addr":"172.21.15.177:6817/3236900263","cluster_addr":"172.21.15.177:6819/3236900263","heartbeat_back_addr":"172.21.15.177:6823/3236900263","heartbeat_front_addr":"172.21.15.177:6821/3236900263","state":["exists","up"]},{"osd":11,"uuid":"cccc53c5-8476-4d55-b160-4bdc5cf92902","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":90,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":681011628},{"type":"v1","addr":"172.21.15.177:6825","nonce":681011628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":681011628},{"type":"v1","addr":"172.21.15.177:6827","nonce":681011628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":681011628},{"type":"v1","addr":"172.21.15.177:6831","nonce":681011628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":681011628},{"type":"v1","addr":"172.21.15.177:6829","nonce":681011628}]},"public_addr":"172.21.15.177:6825/681011628","cluster_addr":"172.21.15.177:6827/681011628","heartbeat_back_addr":"172.21.15.177:6831/681011628","heartbeat_front_addr":"172.21.15.177:6829/681011628","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:46:06.857716+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:45:38.748016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:45:37.332618+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:44:48.221317+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:45:26.363198+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:45:32.127420+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:44:49.578317+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:45:32.922550+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:45:52.327609+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:46:11.579496+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:45:23.318781+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:45:49.287805+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6801/2530617473":"2024-05-16T12:39:19.409195+0000","172.21.15.27:6800/2530617473":"2024-05-16T12:39:19.409195+0000","172.21.15.27:0/1880261204":"2024-05-16T12:39:19.409195+0000","172.21.15.27:6801/3441136471":"2024-05-16T12:38:50.931011+0000","172.21.15.27:0/4291446247":"2024-05-16T12:39:19.409195+0000","172.21.15.27:0/580128209":"2024-05-16T12:38:50.931011+0000","172.21.15.27:6800/3441136471":"2024-05-16T12:38:50.931011+0000","172.21.15.27:0/3011229857":"2024-05-16T12:38:50.931011+0000","172.21.15.27:0/2109693913":"2024-05-16T12:38:50.931011+0000","172.21.15.27:0/2445663024":"2024-05-16T12:39:19.409195+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:46:16.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:16 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2722145533' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T12:46:16.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:16 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2722145533' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T12:46:16.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:16 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2722145533' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T12:46:16.612 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-15T12:46:16.613 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-15T12:46:16.886 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:17.422 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:17 smithi027 ceph-mon[39795]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:17.422 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:17 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:46:17.422 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:17 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:46:17.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:17 smithi177 ceph-mon[44303]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:17.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:17 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:46:17.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:17 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:46:17.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:17 smithi120 ceph-mon[43961]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:17.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:17 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:46:17.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:17 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:46:18.260 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:18 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:46:18.261 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:18 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:46:18.261 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:18 smithi027 ceph-mon[39795]: osdmap e97: 12 total, 12 up, 12 in 2024-05-15T12:46:18.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:18 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:46:18.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:18 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:46:18.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:18 smithi177 ceph-mon[44303]: osdmap e97: 12 total, 12 up, 12 in 2024-05-15T12:46:18.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:18 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:46:18.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:18 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:46:18.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:18 smithi120 ceph-mon[43961]: osdmap e97: 12 total, 12 up, 12 in 2024-05-15T12:46:18.993 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:46:18.993 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":97,"fsid":"deaea04e-12b7-11ef-bc98-c7b262605968","created":"2024-05-15T12:38:21.244556+0000","modified":"2024-05-15T12:46:17.129220+0000","last_up_change":"2024-05-15T12:45:50.876090+0000","last_in_change":"2024-05-15T12:45:33.158047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T12:41:57.948613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"64fb3ad2-08a7-4106-a07a-6c776c877faf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4239116935},{"type":"v1","addr":"172.21.15.27:6803","nonce":4239116935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4239116935},{"type":"v1","addr":"172.21.15.27:6805","nonce":4239116935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4239116935},{"type":"v1","addr":"172.21.15.27:6809","nonce":4239116935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4239116935},{"type":"v1","addr":"172.21.15.27:6807","nonce":4239116935}]},"public_addr":"172.21.15.27:6803/4239116935","cluster_addr":"172.21.15.27:6805/4239116935","heartbeat_back_addr":"172.21.15.27:6809/4239116935","heartbeat_front_addr":"172.21.15.27:6807/4239116935","state":["exists","up"]},{"osd":1,"uuid":"9c16dd90-9d51-48b8-a296-1c8879ba1bc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1466270637},{"type":"v1","addr":"172.21.15.27:6811","nonce":1466270637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1466270637},{"type":"v1","addr":"172.21.15.27:6813","nonce":1466270637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1466270637},{"type":"v1","addr":"172.21.15.27:6817","nonce":1466270637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1466270637},{"type":"v1","addr":"172.21.15.27:6815","nonce":1466270637}]},"public_addr":"172.21.15.27:6811/1466270637","cluster_addr":"172.21.15.27:6813/1466270637","heartbeat_back_addr":"172.21.15.27:6817/1466270637","heartbeat_front_addr":"172.21.15.27:6815/1466270637","state":["exists","up"]},{"osd":2,"uuid":"cbe3ba13-3ea3-4891-8d2d-6b85581a732c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3285413897},{"type":"v1","addr":"172.21.15.27:6819","nonce":3285413897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3285413897},{"type":"v1","addr":"172.21.15.27:6821","nonce":3285413897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3285413897},{"type":"v1","addr":"172.21.15.27:6825","nonce":3285413897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3285413897},{"type":"v1","addr":"172.21.15.27:6823","nonce":3285413897}]},"public_addr":"172.21.15.27:6819/3285413897","cluster_addr":"172.21.15.27:6821/3285413897","heartbeat_back_addr":"172.21.15.27:6825/3285413897","heartbeat_front_addr":"172.21.15.27:6823/3285413897","state":["exists","up"]},{"osd":3,"uuid":"2e83735b-b944-4f87-8080-b994d4b96897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3008975244},{"type":"v1","addr":"172.21.15.27:6827","nonce":3008975244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3008975244},{"type":"v1","addr":"172.21.15.27:6829","nonce":3008975244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3008975244},{"type":"v1","addr":"172.21.15.27:6833","nonce":3008975244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3008975244},{"type":"v1","addr":"172.21.15.27:6831","nonce":3008975244}]},"public_addr":"172.21.15.27:6827/3008975244","cluster_addr":"172.21.15.27:6829/3008975244","heartbeat_back_addr":"172.21.15.27:6833/3008975244","heartbeat_front_addr":"172.21.15.27:6831/3008975244","state":["exists","up"]},{"osd":4,"uuid":"7c39c1dd-3867-46f7-8372-7688fa13b16a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3877842957},{"type":"v1","addr":"172.21.15.120:6801","nonce":3877842957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3877842957},{"type":"v1","addr":"172.21.15.120:6803","nonce":3877842957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3877842957},{"type":"v1","addr":"172.21.15.120:6807","nonce":3877842957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3877842957},{"type":"v1","addr":"172.21.15.120:6805","nonce":3877842957}]},"public_addr":"172.21.15.120:6801/3877842957","cluster_addr":"172.21.15.120:6803/3877842957","heartbeat_back_addr":"172.21.15.120:6807/3877842957","heartbeat_front_addr":"172.21.15.120:6805/3877842957","state":["exists","up"]},{"osd":5,"uuid":"a6dc220f-aec0-4454-9695-b44c0b9770e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1600179183},{"type":"v1","addr":"172.21.15.120:6809","nonce":1600179183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1600179183},{"type":"v1","addr":"172.21.15.120:6811","nonce":1600179183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1600179183},{"type":"v1","addr":"172.21.15.120:6815","nonce":1600179183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1600179183},{"type":"v1","addr":"172.21.15.120:6813","nonce":1600179183}]},"public_addr":"172.21.15.120:6809/1600179183","cluster_addr":"172.21.15.120:6811/1600179183","heartbeat_back_addr":"172.21.15.120:6815/1600179183","heartbeat_front_addr":"172.21.15.120:6813/1600179183","state":["exists","up"]},{"osd":6,"uuid":"9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":3323706735},{"type":"v1","addr":"172.21.15.120:6817","nonce":3323706735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":3323706735},{"type":"v1","addr":"172.21.15.120:6819","nonce":3323706735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":3323706735},{"type":"v1","addr":"172.21.15.120:6823","nonce":3323706735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":3323706735},{"type":"v1","addr":"172.21.15.120:6821","nonce":3323706735}]},"public_addr":"172.21.15.120:6817/3323706735","cluster_addr":"172.21.15.120:6819/3323706735","heartbeat_back_addr":"172.21.15.120:6823/3323706735","heartbeat_front_addr":"172.21.15.120:6821/3323706735","state":["exists","up"]},{"osd":7,"uuid":"6ef085d8-f59b-4033-8280-2c8f78693e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2960616153},{"type":"v1","addr":"172.21.15.120:6825","nonce":2960616153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2960616153},{"type":"v1","addr":"172.21.15.120:6827","nonce":2960616153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2960616153},{"type":"v1","addr":"172.21.15.120:6831","nonce":2960616153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2960616153},{"type":"v1","addr":"172.21.15.120:6829","nonce":2960616153}]},"public_addr":"172.21.15.120:6825/2960616153","cluster_addr":"172.21.15.120:6827/2960616153","heartbeat_back_addr":"172.21.15.120:6831/2960616153","heartbeat_front_addr":"172.21.15.120:6829/2960616153","state":["exists","up"]},{"osd":8,"uuid":"26514b11-d02b-46e9-a4e8-3da193fcdbcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1132459315},{"type":"v1","addr":"172.21.15.177:6801","nonce":1132459315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1132459315},{"type":"v1","addr":"172.21.15.177:6803","nonce":1132459315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1132459315},{"type":"v1","addr":"172.21.15.177:6807","nonce":1132459315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1132459315},{"type":"v1","addr":"172.21.15.177:6805","nonce":1132459315}]},"public_addr":"172.21.15.177:6801/1132459315","cluster_addr":"172.21.15.177:6803/1132459315","heartbeat_back_addr":"172.21.15.177:6807/1132459315","heartbeat_front_addr":"172.21.15.177:6805/1132459315","state":["exists","up"]},{"osd":9,"uuid":"6f9497bc-e962-409d-8a2c-0c634cc1124f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2710457646},{"type":"v1","addr":"172.21.15.177:6809","nonce":2710457646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2710457646},{"type":"v1","addr":"172.21.15.177:6811","nonce":2710457646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2710457646},{"type":"v1","addr":"172.21.15.177:6815","nonce":2710457646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2710457646},{"type":"v1","addr":"172.21.15.177:6813","nonce":2710457646}]},"public_addr":"172.21.15.177:6809/2710457646","cluster_addr":"172.21.15.177:6811/2710457646","heartbeat_back_addr":"172.21.15.177:6815/2710457646","heartbeat_front_addr":"172.21.15.177:6813/2710457646","state":["exists","up"]},{"osd":10,"uuid":"f852876f-c7d1-4039-90c4-5ab09dbd3e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":3236900263},{"type":"v1","addr":"172.21.15.177:6817","nonce":3236900263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3236900263},{"type":"v1","addr":"172.21.15.177:6819","nonce":3236900263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3236900263},{"type":"v1","addr":"172.21.15.177:6823","nonce":3236900263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3236900263},{"type":"v1","addr":"172.21.15.177:6821","nonce":3236900263}]},"public_addr":"172.21.15.177:6817/3236900263","cluster_addr":"172.21.15.177:6819/3236900263","heartbeat_back_addr":"172.21.15.177:6823/3236900263","heartbeat_front_addr":"172.21.15.177:6821/3236900263","state":["exists","up"]},{"osd":11,"uuid":"cccc53c5-8476-4d55-b160-4bdc5cf92902","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":90,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":681011628},{"type":"v1","addr":"172.21.15.177:6825","nonce":681011628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":681011628},{"type":"v1","addr":"172.21.15.177:6827","nonce":681011628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":681011628},{"type":"v1","addr":"172.21.15.177:6831","nonce":681011628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":681011628},{"type":"v1","addr":"172.21.15.177:6829","nonce":681011628}]},"public_addr":"172.21.15.177:6825/681011628","cluster_addr":"172.21.15.177:6827/681011628","heartbeat_back_addr":"172.21.15.177:6831/681011628","heartbeat_front_addr":"172.21.15.177:6829/681011628","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:46:06.857716+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:45:38.748016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:45:37.332618+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:46:16.484098+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:45:26.363198+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:45:32.127420+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:46:16.493580+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:45:32.922550+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:45:52.327609+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:46:11.579496+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:45:23.318781+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:45:49.287805+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6801/2530617473":"2024-05-16T12:39:19.409195+0000","172.21.15.27:6800/2530617473":"2024-05-16T12:39:19.409195+0000","172.21.15.27:0/1880261204":"2024-05-16T12:39:19.409195+0000","172.21.15.27:6801/3441136471":"2024-05-16T12:38:50.931011+0000","172.21.15.27:0/4291446247":"2024-05-16T12:39:19.409195+0000","172.21.15.27:0/580128209":"2024-05-16T12:38:50.931011+0000","172.21.15.27:6800/3441136471":"2024-05-16T12:38:50.931011+0000","172.21.15.27:0/3011229857":"2024-05-16T12:38:50.931011+0000","172.21.15.27:0/2109693913":"2024-05-16T12:38:50.931011+0000","172.21.15.27:0/2445663024":"2024-05-16T12:39:19.409195+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:46:19.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:19 smithi027 ceph-mon[39795]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:19.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:19 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/841194732' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T12:46:19.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:19 smithi177 ceph-mon[44303]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:19.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:19 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/841194732' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T12:46:19.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:19 smithi120 ceph-mon[43961]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:19.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:19 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/841194732' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T12:46:19.781 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-15T12:46:19.781 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-15T12:46:19.782 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-15T12:46:19.782 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-15T12:46:19.782 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-15T12:46:19.783 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-15T12:46:19.783 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-15T12:46:19.784 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-15T12:46:19.784 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-05-15T12:46:19.784 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-05-15T12:46:19.785 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-05-15T12:46:19.785 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-05-15T12:46:20.458 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:20.710 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:20.721 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:20.723 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:20.727 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:21.393 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:21 smithi027 ceph-mon[39795]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:21.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:21 smithi177 ceph-mon[44303]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:21.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:21 smithi120 ceph-mon[43961]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:23.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:23 smithi177 ceph-mon[44303]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:23.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:23 smithi027 ceph-mon[39795]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:23.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:23 smithi120 ceph-mon[43961]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:25.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:25 smithi027 ceph-mon[39795]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:25.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:25 smithi177 ceph-mon[44303]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:25 smithi120 ceph-mon[43961]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:25.599 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:25.600 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:25.602 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:25.603 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:25.605 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:25.609 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:26.324 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:26 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:46:26.325 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:26 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:46:26.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:26 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:46:26.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:26 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:46:26.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:26 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:46:26.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:26 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:46:27.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:27 smithi177 ceph-mon[44303]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:27.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:27 smithi177 ceph-mon[44303]: osdmap e98: 12 total, 12 up, 12 in 2024-05-15T12:46:27.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:27 smithi027 ceph-mon[39795]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:27.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:27 smithi027 ceph-mon[39795]: osdmap e98: 12 total, 12 up, 12 in 2024-05-15T12:46:27.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:27 smithi120 ceph-mon[43961]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:27.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:27 smithi120 ceph-mon[43961]: osdmap e98: 12 total, 12 up, 12 in 2024-05-15T12:46:29.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:29 smithi177 ceph-mon[44303]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:29.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:29 smithi027 ceph-mon[39795]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:29.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:29 smithi120 ceph-mon[43961]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:30.121 INFO:teuthology.orchestra.run.smithi027.stdout:386547056650 2024-05-15T12:46:30.122 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-05-15T12:46:31.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:31 smithi177 ceph-mon[44303]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:31.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:31 smithi027 ceph-mon[39795]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:31.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:31 smithi120 ceph-mon[43961]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:32.400 INFO:teuthology.orchestra.run.smithi027.stdout:55834574912 2024-05-15T12:46:32.400 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-15T12:46:33.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:33 smithi177 ceph-mon[44303]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:33.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:33 smithi027 ceph-mon[39795]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:33.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:33 smithi120 ceph-mon[43961]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:34.442 INFO:teuthology.orchestra.run.smithi027.stdout:34359738437 2024-05-15T12:46:34.443 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-15T12:46:35.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:35 smithi177 ceph-mon[44303]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:35.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:35 smithi027 ceph-mon[39795]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:35.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:35 smithi120 ceph-mon[43961]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:35.650 INFO:teuthology.orchestra.run.smithi027.stdout:326417514518 2024-05-15T12:46:35.650 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-05-15T12:46:36.137 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:36.168 INFO:teuthology.orchestra.run.smithi027.stdout:279172874267 2024-05-15T12:46:36.168 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-05-15T12:46:37.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:37 smithi027 ceph-mon[39795]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:37.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:37 smithi177 ceph-mon[44303]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:37.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:37 smithi120 ceph-mon[43961]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:39.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:39 smithi177 ceph-mon[44303]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:39.469 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:39 smithi027 ceph-mon[39795]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:39.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:39 smithi120 ceph-mon[43961]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:41.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:41 smithi027 ceph-mon[39795]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:41.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:41 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:46:41.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:41 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:46:41.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:41 smithi120 ceph-mon[43961]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:41.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:41 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:46:41.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:41 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:46:41.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:41 smithi177 ceph-mon[44303]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:41.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:41 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:46:41.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:41 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:46:42.456 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:42 smithi027 ceph-mon[39795]: osdmap e99: 12 total, 12 up, 12 in 2024-05-15T12:46:42.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:42 smithi120 ceph-mon[43961]: osdmap e99: 12 total, 12 up, 12 in 2024-05-15T12:46:42.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:42 smithi177 ceph-mon[44303]: osdmap e99: 12 total, 12 up, 12 in 2024-05-15T12:46:43.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:43 smithi027 ceph-mon[39795]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:43.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:43 smithi120 ceph-mon[43961]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:43.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:43 smithi177 ceph-mon[44303]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:44.396 INFO:teuthology.orchestra.run.smithi027.stdout:206158430246 2024-05-15T12:46:44.397 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-15T12:46:45.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:45 smithi027 ceph-mon[39795]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:45.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:45 smithi120 ceph-mon[43961]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:45.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:45 smithi177 ceph-mon[44303]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:45.997 INFO:teuthology.orchestra.run.smithi027.stdout:111669149751 2024-05-15T12:46:45.997 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-15T12:46:46.650 INFO:teuthology.orchestra.run.smithi027.stdout:77309411390 2024-05-15T12:46:46.650 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-15T12:46:47.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:47 smithi027 ceph-mon[39795]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:47.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:47 smithi120 ceph-mon[43961]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:47.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:47 smithi177 ceph-mon[44303]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:48.158 INFO:teuthology.orchestra.run.smithi027.stdout:240518168608 2024-05-15T12:46:48.158 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-15T12:46:48.265 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:48.686 INFO:teuthology.orchestra.run.smithi027.stdout:133143986226 2024-05-15T12:46:48.687 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-15T12:46:48.873 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:48.873 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:49.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:49 smithi027 ceph-mon[39795]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:49.327 INFO:teuthology.orchestra.run.smithi027.stdout:347892350996 2024-05-15T12:46:49.328 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-05-15T12:46:49.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:49 smithi120 ceph-mon[43961]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:49.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:49 smithi177 ceph-mon[44303]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:50.695 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:51.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:51 smithi027 ceph-mon[39795]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:51.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:51 smithi120 ceph-mon[43961]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:51.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:51 smithi177 ceph-mon[44303]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:51.752 INFO:teuthology.orchestra.run.smithi027.stdout:171798691885 2024-05-15T12:46:51.752 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-15T12:46:52.074 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:53.244 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:53 smithi027 ceph-mon[39795]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:53.244 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:53 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:46:53.294 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:53.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:53 smithi120 ceph-mon[43961]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:53.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:53 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:46:53.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:53 smithi177 ceph-mon[44303]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:53.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:53 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:46:54.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:54 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:46:54.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:54 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:46:54.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:54 smithi027 ceph-mon[39795]: osdmap e100: 12 total, 12 up, 12 in 2024-05-15T12:46:54.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:54 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:46:54.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:54 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:46:54.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:54 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:46:54.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:54 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:46:54.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:54 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:46:54.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:54 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4078992791' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-15T12:46:54.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:54 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:46:54.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:54 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:46:54.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:54 smithi120 ceph-mon[43961]: osdmap e100: 12 total, 12 up, 12 in 2024-05-15T12:46:54.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:54 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:46:54.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:54 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:46:54.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:54 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:46:54.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:54 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:46:54.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:54 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:46:54.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:54 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4078992791' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-15T12:46:54.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:54 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:46:54.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:54 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:46:54.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:54 smithi177 ceph-mon[44303]: osdmap e100: 12 total, 12 up, 12 in 2024-05-15T12:46:54.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:54 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:46:54.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:54 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:46:54.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:54 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:46:54.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:54 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:46:54.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:54 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:46:54.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:54 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4078992791' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-15T12:46:54.844 INFO:teuthology.orchestra.run.smithi027.stdout:386547056655 2024-05-15T12:46:55.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:55 smithi027 ceph-mon[39795]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:55.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:55 smithi027 ceph-mon[39795]: osdmap e101: 12 total, 12 up, 12 in 2024-05-15T12:46:55.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:55 smithi120 ceph-mon[43961]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:55.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:55 smithi120 ceph-mon[43961]: osdmap e101: 12 total, 12 up, 12 in 2024-05-15T12:46:55.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:55 smithi177 ceph-mon[44303]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:55.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:55 smithi177 ceph-mon[44303]: osdmap e101: 12 total, 12 up, 12 in 2024-05-15T12:46:55.716 INFO:tasks.cephadm.ceph_manager.ceph:need seq 386547056650 got 386547056655 for osd.11 2024-05-15T12:46:55.716 DEBUG:teuthology.parallel:result is None 2024-05-15T12:46:57.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:57 smithi027 ceph-mon[39795]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:57.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:57 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/353318332' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-15T12:46:57.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:57 smithi120 ceph-mon[43961]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:57.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:57 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/353318332' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-15T12:46:57.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:57 smithi177 ceph-mon[44303]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:57.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:57 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/353318332' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-15T12:46:57.699 INFO:teuthology.orchestra.run.smithi027.stdout:55834574916 2024-05-15T12:46:58.224 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574912 got 55834574916 for osd.1 2024-05-15T12:46:58.225 DEBUG:teuthology.parallel:result is None 2024-05-15T12:46:59.470 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:59 smithi027 ceph-mon[39795]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:59.470 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:59 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:46:59.470 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:59 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:46:59.470 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:59 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:46:59.470 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:59 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:46:59.470 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:46:59 smithi027 ceph-mon[39795]: osdmap e102: 12 total, 12 up, 12 in 2024-05-15T12:46:59.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:59 smithi120 ceph-mon[43961]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:59.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:59 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:46:59.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:59 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:46:59.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:59 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:46:59.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:59 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:46:59.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:46:59 smithi120 ceph-mon[43961]: osdmap e102: 12 total, 12 up, 12 in 2024-05-15T12:46:59.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:59 smithi177 ceph-mon[44303]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:46:59.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:59 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:46:59.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:59 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:46:59.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:59 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:46:59.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:59 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:46:59.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:46:59 smithi177 ceph-mon[44303]: osdmap e102: 12 total, 12 up, 12 in 2024-05-15T12:46:59.724 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:59.727 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:59.727 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:59.735 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:46:59.740 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:47:00.662 INFO:teuthology.orchestra.run.smithi027.stdout:326417514523 2024-05-15T12:47:01.489 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:01 smithi027 ceph-mon[39795]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:01.489 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:01 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/625143812' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-15T12:47:01.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:01 smithi120 ceph-mon[43961]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:01.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:01 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/625143812' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-15T12:47:01.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:01 smithi177 ceph-mon[44303]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:01.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:01 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/625143812' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-15T12:47:03.481 INFO:teuthology.orchestra.run.smithi027.stdout:347892350999 2024-05-15T12:47:03.502 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:03 smithi027 ceph-mon[39795]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:03.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:03 smithi120 ceph-mon[43961]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:03.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:03 smithi177 ceph-mon[44303]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:04.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:04 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1087300103' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-15T12:47:04.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:04 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1087300103' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-15T12:47:04.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:04 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1087300103' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-15T12:47:05.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:05 smithi027 ceph-mon[39795]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:05.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:05 smithi120 ceph-mon[43961]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:05.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:05 smithi177 ceph-mon[44303]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:06.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:06 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/503175457' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-15T12:47:06.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:06 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/503175457' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-15T12:47:06.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:06 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/503175457' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-15T12:47:06.807 INFO:teuthology.orchestra.run.smithi027.stdout:34359738444 2024-05-15T12:47:07.372 INFO:tasks.cephadm.ceph_manager.ceph:need seq 326417514518 got 326417514523 for osd.9 2024-05-15T12:47:07.373 DEBUG:teuthology.parallel:result is None 2024-05-15T12:47:07.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:07 smithi027 ceph-mon[39795]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:07.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:07 smithi120 ceph-mon[43961]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:07.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:07 smithi177 ceph-mon[44303]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:08.651 INFO:teuthology.orchestra.run.smithi027.stdout:279172874275 2024-05-15T12:47:08.844 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:47:08.876 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892350996 got 347892350999 for osd.10 2024-05-15T12:47:08.876 DEBUG:teuthology.parallel:result is None 2024-05-15T12:47:09.507 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:09 smithi027 ceph-mon[39795]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:09.507 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:09 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3081299612' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-15T12:47:09.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:09 smithi120 ceph-mon[43961]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:09.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:09 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3081299612' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-15T12:47:09.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:09 smithi177 ceph-mon[44303]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:09.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:09 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3081299612' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-15T12:47:09.987 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738437 got 34359738444 for osd.0 2024-05-15T12:47:09.987 DEBUG:teuthology.parallel:result is None 2024-05-15T12:47:11.148 INFO:teuthology.orchestra.run.smithi027.stdout:206158430252 2024-05-15T12:47:11.261 INFO:teuthology.orchestra.run.smithi027.stdout:77309411396 2024-05-15T12:47:11.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:11 smithi027 ceph-mon[39795]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:11.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:11 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2750743576' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-15T12:47:11.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:11 smithi120 ceph-mon[43961]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:11.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:11 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2750743576' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-15T12:47:11.646 INFO:teuthology.orchestra.run.smithi027.stdout:111669149756 2024-05-15T12:47:11.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:11 smithi177 ceph-mon[44303]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:11.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:11 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2750743576' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-15T12:47:12.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:12 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1685497554' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-15T12:47:12.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:12 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2619885020' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-15T12:47:12.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:12 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1685497554' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-15T12:47:12.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:12 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2619885020' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-15T12:47:12.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:12 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1685497554' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-15T12:47:12.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:12 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2619885020' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-15T12:47:12.997 INFO:teuthology.orchestra.run.smithi027.stdout:133143986231 2024-05-15T12:47:13.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:13 smithi027 ceph-mon[39795]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:13.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:13 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3863686679' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-15T12:47:13.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:13 smithi120 ceph-mon[43961]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:13.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:13 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3863686679' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-15T12:47:13.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:13 smithi177 ceph-mon[44303]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:13.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:13 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3863686679' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-15T12:47:14.185 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430246 got 206158430252 for osd.6 2024-05-15T12:47:14.185 DEBUG:teuthology.parallel:result is None 2024-05-15T12:47:15.154 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411390 got 77309411396 for osd.2 2024-05-15T12:47:15.154 DEBUG:teuthology.parallel:result is None 2024-05-15T12:47:15.375 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:15 smithi027 ceph-mon[39795]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:15.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:15 smithi120 ceph-mon[43961]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:15.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:15 smithi177 ceph-mon[44303]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:15.686 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874267 got 279172874275 for osd.8 2024-05-15T12:47:15.686 DEBUG:teuthology.parallel:result is None 2024-05-15T12:47:15.762 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149751 got 111669149756 for osd.3 2024-05-15T12:47:15.762 DEBUG:teuthology.parallel:result is None 2024-05-15T12:47:16.714 INFO:teuthology.orchestra.run.smithi027.stdout:240518168615 2024-05-15T12:47:17.176 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986226 got 133143986231 for osd.4 2024-05-15T12:47:17.176 DEBUG:teuthology.parallel:result is None 2024-05-15T12:47:17.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:17 smithi027 ceph-mon[39795]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:17.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:17 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/192636096' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-15T12:47:17.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:17 smithi120 ceph-mon[43961]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:17.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:17 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/192636096' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-15T12:47:17.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:17 smithi177 ceph-mon[44303]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:17.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:17 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/192636096' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-15T12:47:18.361 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168608 got 240518168615 for osd.7 2024-05-15T12:47:18.362 DEBUG:teuthology.parallel:result is None 2024-05-15T12:47:19.201 INFO:teuthology.orchestra.run.smithi027.stdout:171798691891 2024-05-15T12:47:19.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:19 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:47:19.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:19 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:47:19.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:19 smithi027 ceph-mon[39795]: pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:19.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:19 smithi027 ceph-mon[39795]: osdmap e103: 12 total, 12 up, 12 in 2024-05-15T12:47:19.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:19 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3996444157' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-15T12:47:19.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:19 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:47:19.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:19 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:47:19.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:19 smithi120 ceph-mon[43961]: pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:19.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:19 smithi120 ceph-mon[43961]: osdmap e103: 12 total, 12 up, 12 in 2024-05-15T12:47:19.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:19 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3996444157' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-15T12:47:19.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:19 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:47:19.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:19 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:47:19.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:19 smithi177 ceph-mon[44303]: pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:19.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:19 smithi177 ceph-mon[44303]: osdmap e103: 12 total, 12 up, 12 in 2024-05-15T12:47:19.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:19 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3996444157' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-15T12:47:20.010 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691885 got 171798691891 for osd.5 2024-05-15T12:47:20.010 DEBUG:teuthology.parallel:result is None 2024-05-15T12:47:20.010 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-15T12:47:20.010 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T12:47:20.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:20 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T12:47:20.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:20 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T12:47:20.290 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:47:20.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:20 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T12:47:20.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:20 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T12:47:20.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:20 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T12:47:20.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:20 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T12:47:21.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:21 smithi027 ceph-mon[39795]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:21.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:21 smithi120 ceph-mon[43961]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:21.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:21 smithi177 ceph-mon[44303]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:22.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:22 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:47:22.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:22 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:47:22.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:22 smithi027 ceph-mon[39795]: osdmap e104: 12 total, 12 up, 12 in 2024-05-15T12:47:22.625 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:47:22.628 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T12:47:22.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:22 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:47:22.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:22 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:47:22.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:22 smithi177 ceph-mon[44303]: osdmap e104: 12 total, 12 up, 12 in 2024-05-15T12:47:22.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:22 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:47:22.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:22 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:47:22.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:22 smithi120 ceph-mon[43961]: osdmap e104: 12 total, 12 up, 12 in 2024-05-15T12:47:23.442 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2024-05-15T12:47:21.504026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":342636,"kb_used_data":10080,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124599188,"statfs":{"total":1151940427776,"available":1151589568512,"internally_reserved":0,"allocated":10321920,"data_stored":6203910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":340374843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001860"},"pg_stats":[{"pgid":"1.0","version":"94'139","reported_seq":373,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-15T12:47:18.958463+0000","last_change":"2024-05-15T12:47:18.958496+0000","last_active":"2024-05-15T12:47:18.958463+0000","last_peered":"2024-05-15T12:47:18.958463+0000","last_clean":"2024-05-15T12:47:18.958463+0000","last_became_active":"2024-05-15T12:45:55.009444+0000","last_became_peered":"2024-05-15T12:45:55.009444+0000","last_unstale":"2024-05-15T12:47:18.958463+0000","last_undegraded":"2024-05-15T12:47:18.958463+0000","last_fullsized":"2024-05-15T12:47:18.958463+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T12:47:18.958392+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T12:47:18.958392+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:48:26.376618+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056661,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28244,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716908,"statfs":{"total":95995035648,"available":95966113792,"internally_reserved":0,"allocated":1253376,"data_stored":909985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351003,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27736,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717416,"statfs":{"total":95995035648,"available":95966633984,"internally_reserved":0,"allocated":733184,"data_stored":388434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514528,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27736,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717416,"statfs":{"total":95995035648,"available":95966633984,"internally_reserved":0,"allocated":733184,"data_stored":388434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874278,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27736,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717416,"statfs":{"total":95995035648,"available":95966633984,"internally_reserved":0,"allocated":733184,"data_stored":388454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168616,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28252,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716900,"statfs":{"total":95995035648,"available":95966105600,"internally_reserved":0,"allocated":1265664,"data_stored":917382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430254,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28244,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716908,"statfs":{"total":95995035648,"available":95966113792,"internally_reserved":0,"allocated":1253376,"data_stored":910005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31896,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713256,"statfs":{"total":95995035648,"available":95962374144,"internally_reserved":0,"allocated":733184,"data_stored":388454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31884,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713268,"statfs":{"total":95995035648,"available":95962386432,"internally_reserved":0,"allocated":720896,"data_stored":381077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411398,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27724,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717428,"statfs":{"total":95995035648,"available":95966646272,"internally_reserved":0,"allocated":720896,"data_stored":381077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27724,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717428,"statfs":{"total":95995035648,"available":95966646272,"internally_reserved":0,"allocated":720896,"data_stored":381077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986233,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27736,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717416,"statfs":{"total":95995035648,"available":95966633984,"internally_reserved":0,"allocated":733184,"data_stored":388454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27724,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717428,"statfs":{"total":95995035648,"available":95966646272,"internally_reserved":0,"allocated":720896,"data_stored":381077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T12:47:23.443 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T12:47:23.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:23 smithi177 ceph-mon[44303]: pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:47:23.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:23 smithi027 ceph-mon[39795]: pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:23.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:23 smithi120 ceph-mon[43961]: pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:24.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:24 smithi177 ceph-mon[44303]: from='client.14649 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:47:24.745 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:24 smithi027 ceph-mon[39795]: from='client.14649 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:47:24.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:24 smithi120 ceph-mon[43961]: from='client.14649 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:47:25.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:25 smithi027 ceph-mon[39795]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:25.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:25 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:47:25.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:25 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:47:25.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:25 smithi027 ceph-mon[39795]: osdmap e105: 12 total, 12 up, 12 in 2024-05-15T12:47:25.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:25 smithi177 ceph-mon[44303]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:25.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:25 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:47:25.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:25 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:47:25.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:25 smithi177 ceph-mon[44303]: osdmap e105: 12 total, 12 up, 12 in 2024-05-15T12:47:25.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:25 smithi120 ceph-mon[43961]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:25.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:25 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:47:25.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:25 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:47:25.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:25 smithi120 ceph-mon[43961]: osdmap e105: 12 total, 12 up, 12 in 2024-05-15T12:47:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:47:26.069 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T12:47:26.879 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":338,"stamp":"2024-05-15T12:47:25.504628+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":342820,"kb_used_data":10260,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124599004,"statfs":{"total":1151940427776,"available":1151589380096,"internally_reserved":0,"allocated":10506240,"data_stored":6314565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":340374843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.227318"},"pg_stats":[{"pgid":"1.0","version":"94'139","reported_seq":377,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-05-15T12:47:24.414666+0000","last_change":"2024-05-15T12:47:18.958496+0000","last_active":"2024-05-15T12:47:24.414666+0000","last_peered":"2024-05-15T12:47:24.414666+0000","last_clean":"2024-05-15T12:47:24.414666+0000","last_became_active":"2024-05-15T12:45:55.009444+0000","last_became_peered":"2024-05-15T12:45:55.009444+0000","last_unstale":"2024-05-15T12:47:24.414666+0000","last_undegraded":"2024-05-15T12:47:24.414666+0000","last_fullsized":"2024-05-15T12:47:24.414666+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T12:47:18.958392+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T12:47:18.958392+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:48:26.376618+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056662,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28256,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716896,"statfs":{"total":95995035648,"available":95966101504,"internally_reserved":0,"allocated":1265664,"data_stored":917362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351003,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27736,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717416,"statfs":{"total":95995035648,"available":95966633984,"internally_reserved":0,"allocated":733184,"data_stored":388434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514529,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27748,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717404,"statfs":{"total":95995035648,"available":95966621696,"internally_reserved":0,"allocated":745472,"data_stored":395811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874279,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27748,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717404,"statfs":{"total":95995035648,"available":95966621696,"internally_reserved":0,"allocated":745472,"data_stored":395831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168617,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28280,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716872,"statfs":{"total":95995035648,"available":95966076928,"internally_reserved":0,"allocated":1290240,"data_stored":932136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430255,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28268,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716884,"statfs":{"total":95995035648,"available":95966089216,"internally_reserved":0,"allocated":1277952,"data_stored":924759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31896,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713256,"statfs":{"total":95995035648,"available":95962374144,"internally_reserved":0,"allocated":733184,"data_stored":388454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31908,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713244,"statfs":{"total":95995035648,"available":95962361856,"internally_reserved":0,"allocated":745472,"data_stored":395831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27748,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717404,"statfs":{"total":95995035648,"available":95966621696,"internally_reserved":0,"allocated":745472,"data_stored":395831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149759,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27736,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717416,"statfs":{"total":95995035648,"available":95966633984,"internally_reserved":0,"allocated":733184,"data_stored":388454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986234,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27748,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717404,"statfs":{"total":95995035648,"available":95966621696,"internally_reserved":0,"allocated":745472,"data_stored":395831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27748,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717404,"statfs":{"total":95995035648,"available":95966621696,"internally_reserved":0,"allocated":745472,"data_stored":395831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T12:47:26.880 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-15T12:47:26.880 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-15T12:47:26.880 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-15T12:47:26.880 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph health --format=json 2024-05-15T12:47:27.154 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:47:27.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:27 smithi177 ceph-mon[44303]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:27.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:27 smithi177 ceph-mon[44303]: from='client.14655 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:47:27.731 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:27 smithi027 ceph-mon[39795]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:27.731 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:27 smithi027 ceph-mon[39795]: from='client.14655 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:47:27.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:27 smithi120 ceph-mon[43961]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:27.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:27 smithi120 ceph-mon[43961]: from='client.14655 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:47:29.428 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:47:29.428 INFO:teuthology.orchestra.run.smithi027.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-15T12:47:29.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:29 smithi177 ceph-mon[44303]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:29.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:29 smithi027 ceph-mon[39795]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:29.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:29 smithi120 ceph-mon[43961]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:30.117 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-15T12:47:30.117 INFO:tasks.cephadm:Setup complete, yielding 2024-05-15T12:47:30.117 INFO:teuthology.run_tasks:Running task thrashosds... 2024-05-15T12:47:30.128 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-05-15T12:47:30.129 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-osd-ec-partial-reads'}, 'ceph': {'conf': {'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-osd-ec-partial-reads', 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314'}} 2024-05-15T12:47:30.129 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'min_in': 4} 2024-05-15T12:47:30.129 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-05-15T12:47:30.405 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:47:30.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:30 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2442992631' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-15T12:47:30.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:30 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2442992631' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-15T12:47:30.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:30 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2442992631' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-15T12:47:31.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:31 smithi177 ceph-mon[44303]: pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:31.740 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:31 smithi027 ceph-mon[39795]: pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:31.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:31 smithi120 ceph-mon[43961]: pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:33.342 INFO:tasks.thrashosds:Beginning thrashosds... 2024-05-15T12:47:33.342 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-15T12:47:33.342 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T12:47:33.617 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:47:33.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:33 smithi177 ceph-mon[44303]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:33.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:33 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2664901429' entity='client.admin' 2024-05-15T12:47:33.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:33 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:47:33.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:33 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:47:33.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:33 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:47:33.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:33 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:47:33.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:33 smithi027 ceph-mon[39795]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:33.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:33 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2664901429' entity='client.admin' 2024-05-15T12:47:33.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:33 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:47:33.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:33 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:47:33.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:33 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:47:33.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:33 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:47:33.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:33 smithi120 ceph-mon[43961]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:33.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:33 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2664901429' entity='client.admin' 2024-05-15T12:47:33.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:33 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:47:33.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:33 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:47:33.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:33 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:47:33.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:33 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:47:35.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:35 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:47:35.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:35 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:47:35.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:35 smithi177 ceph-mon[44303]: pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:35.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:35 smithi177 ceph-mon[44303]: osdmap e106: 12 total, 12 up, 12 in 2024-05-15T12:47:35.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:35 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:47:35.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:35 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:47:35.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:35 smithi027 ceph-mon[39795]: pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:35.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:35 smithi027 ceph-mon[39795]: osdmap e106: 12 total, 12 up, 12 in 2024-05-15T12:47:35.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:35 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:47:35.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:35 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:47:35.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:35 smithi120 ceph-mon[43961]: pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:35.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:35 smithi120 ceph-mon[43961]: osdmap e106: 12 total, 12 up, 12 in 2024-05-15T12:47:35.858 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:47:35.860 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T12:47:36.659 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-05-15T12:47:35.506417+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":343016,"kb_used_data":10332,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124598808,"statfs":{"total":1151940427776,"available":1151589179392,"internally_reserved":0,"allocated":10579968,"data_stored":6379176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":340374843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001788"},"pg_stats":[{"pgid":"1.0","version":"94'139","reported_seq":379,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-05-15T12:47:35.016010+0000","last_change":"2024-05-15T12:47:18.958496+0000","last_active":"2024-05-15T12:47:35.016010+0000","last_peered":"2024-05-15T12:47:35.016010+0000","last_clean":"2024-05-15T12:47:35.016010+0000","last_became_active":"2024-05-15T12:45:55.009444+0000","last_became_peered":"2024-05-15T12:45:55.009444+0000","last_unstale":"2024-05-15T12:47:35.016010+0000","last_undegraded":"2024-05-15T12:47:35.016010+0000","last_fullsized":"2024-05-15T12:47:35.016010+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T12:47:18.958392+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T12:47:18.958392+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:48:26.376618+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056664,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28284,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716868,"statfs":{"total":95995035648,"available":95966072832,"internally_reserved":0,"allocated":1290240,"data_stored":932116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351005,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":403188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514531,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":403188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874281,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":403208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168619,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1179648,"data_stored":886092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430257,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28280,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716872,"statfs":{"total":95995035648,"available":95966076928,"internally_reserved":0,"allocated":1290240,"data_stored":932136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31920,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713232,"statfs":{"total":95995035648,"available":95962349568,"internally_reserved":0,"allocated":757760,"data_stored":403208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31920,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713232,"statfs":{"total":95995035648,"available":95962349568,"internally_reserved":0,"allocated":757760,"data_stored":403208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":403208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149761,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":403208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986236,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":403208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":403208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T12:47:36.660 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T12:47:36.938 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:47:37.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:37 smithi027 ceph-mon[39795]: pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:37.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:37 smithi027 ceph-mon[39795]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:47:37.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:37 smithi177 ceph-mon[44303]: pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:37.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:37 smithi177 ceph-mon[44303]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:47:37.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:37 smithi120 ceph-mon[43961]: pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:37.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:37 smithi120 ceph-mon[43961]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:47:38.585 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:38 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:47:38.586 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:38 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:47:38.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:38 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:47:38.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:38 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:47:38.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:38 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:47:38.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:38 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:47:39.036 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:47:39.038 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T12:47:39.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:39 smithi177 ceph-mon[44303]: pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:39.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:39 smithi177 ceph-mon[44303]: osdmap e107: 12 total, 12 up, 12 in 2024-05-15T12:47:39.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:39 smithi027 ceph-mon[39795]: pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:39.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:39 smithi027 ceph-mon[39795]: osdmap e107: 12 total, 12 up, 12 in 2024-05-15T12:47:39.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:39 smithi120 ceph-mon[43961]: pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:39.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:39 smithi120 ceph-mon[43961]: osdmap e107: 12 total, 12 up, 12 in 2024-05-15T12:47:39.848 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-05-15T12:47:38.335808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":343108,"kb_used_data":9468,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124598716,"statfs":{"total":1151940427776,"available":1151589085184,"internally_reserved":0,"allocated":9695232,"data_stored":6010824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":340374843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.830547"},"pg_stats":[{"pgid":"1.0","version":"94'139","reported_seq":379,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-05-15T12:47:35.016010+0000","last_change":"2024-05-15T12:47:18.958496+0000","last_active":"2024-05-15T12:47:35.016010+0000","last_peered":"2024-05-15T12:47:35.016010+0000","last_clean":"2024-05-15T12:47:35.016010+0000","last_became_active":"2024-05-15T12:45:55.009444+0000","last_became_peered":"2024-05-15T12:45:55.009444+0000","last_unstale":"2024-05-15T12:47:35.016010+0000","last_undegraded":"2024-05-15T12:47:35.016010+0000","last_fullsized":"2024-05-15T12:47:35.016010+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T12:47:18.958392+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T12:47:18.958392+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:48:26.376618+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056665,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1179648,"data_stored":886072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351006,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27772,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717380,"statfs":{"total":95995035648,"available":95966597120,"internally_reserved":0,"allocated":647168,"data_stored":357144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514531,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":403188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874281,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":403208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168619,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1179648,"data_stored":886092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430258,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1179648,"data_stored":886092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31932,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713220,"statfs":{"total":95995035648,"available":95962337280,"internally_reserved":0,"allocated":647168,"data_stored":357164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738452,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31932,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713220,"statfs":{"total":95995035648,"available":95962337280,"internally_reserved":0,"allocated":647168,"data_stored":357164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411402,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27772,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717380,"statfs":{"total":95995035648,"available":95966597120,"internally_reserved":0,"allocated":647168,"data_stored":357164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27772,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717380,"statfs":{"total":95995035648,"available":95966597120,"internally_reserved":0,"allocated":647168,"data_stored":357164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986236,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":403208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27772,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717380,"statfs":{"total":95995035648,"available":95966597120,"internally_reserved":0,"allocated":647168,"data_stored":357164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T12:47:39.849 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-15T12:47:39.849 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd dump 2024-05-15T12:47:40.125 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:47:40.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:40 smithi177 ceph-mon[44303]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:47:40.737 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:40 smithi027 ceph-mon[39795]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:47:40.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:40 smithi120 ceph-mon[43961]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:47:41.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:41 smithi027 ceph-mon[39795]: pgmap v347: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:41.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:41 smithi177 ceph-mon[44303]: pgmap v347: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:41.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:41 smithi120 ceph-mon[43961]: pgmap v347: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:42.280 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 107 2024-05-15T12:47:42.281 INFO:teuthology.orchestra.run.smithi027.stdout:fsid deaea04e-12b7-11ef-bc98-c7b262605968 2024-05-15T12:47:42.281 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-05-15T12:38:21.244556+0000 2024-05-15T12:47:42.281 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-05-15T12:47:38.330709+0000 2024-05-15T12:47:42.281 INFO:teuthology.orchestra.run.smithi027.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-05-15T12:47:42.281 INFO:teuthology.orchestra.run.smithi027.stdout:crush_version 26 2024-05-15T12:47:42.281 INFO:teuthology.orchestra.run.smithi027.stdout:full_ratio 0.95 2024-05-15T12:47:42.281 INFO:teuthology.orchestra.run.smithi027.stdout:backfillfull_ratio 0.9 2024-05-15T12:47:42.281 INFO:teuthology.orchestra.run.smithi027.stdout:nearfull_ratio 0.85 2024-05-15T12:47:42.281 INFO:teuthology.orchestra.run.smithi027.stdout:require_min_compat_client luminous 2024-05-15T12:47:42.281 INFO:teuthology.orchestra.run.smithi027.stdout:min_compat_client jewel 2024-05-15T12:47:42.282 INFO:teuthology.orchestra.run.smithi027.stdout:require_osd_release squid 2024-05-15T12:47:42.282 INFO:teuthology.orchestra.run.smithi027.stdout:stretch_mode_enabled false 2024-05-15T12:47:42.282 INFO:teuthology.orchestra.run.smithi027.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-05-15T12:47:42.282 INFO:teuthology.orchestra.run.smithi027.stdout:max_osd 12 2024-05-15T12:47:42.282 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 up in weight 1 up_from 8 up_thru 58 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6802/4239116935,v1:172.21.15.27:6803/4239116935] [v2:172.21.15.27:6804/4239116935,v1:172.21.15.27:6805/4239116935] exists,up 64fb3ad2-08a7-4106-a07a-6c776c877faf 2024-05-15T12:47:42.282 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 up in weight 1 up_from 13 up_thru 34 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6810/1466270637,v1:172.21.15.27:6811/1466270637] [v2:172.21.15.27:6812/1466270637,v1:172.21.15.27:6813/1466270637] exists,up 9c16dd90-9d51-48b8-a296-1c8879ba1bc8 2024-05-15T12:47:42.282 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6818/3285413897,v1:172.21.15.27:6819/3285413897] [v2:172.21.15.27:6820/3285413897,v1:172.21.15.27:6821/3285413897] exists,up cbe3ba13-3ea3-4891-8d2d-6b85581a732c 2024-05-15T12:47:42.282 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6826/3008975244,v1:172.21.15.27:6827/3008975244] [v2:172.21.15.27:6828/3008975244,v1:172.21.15.27:6829/3008975244] exists,up 2e83735b-b944-4f87-8080-b994d4b96897 2024-05-15T12:47:42.282 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.120:6800/3877842957,v1:172.21.15.120:6801/3877842957] [v2:172.21.15.120:6802/3877842957,v1:172.21.15.120:6803/3877842957] exists,up 7c39c1dd-3867-46f7-8372-7688fa13b16a 2024-05-15T12:47:42.282 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 up in weight 1 up_from 40 up_thru 43 down_at 0 last_clean_interval [0,0) [v2:172.21.15.120:6808/1600179183,v1:172.21.15.120:6809/1600179183] [v2:172.21.15.120:6810/1600179183,v1:172.21.15.120:6811/1600179183] exists,up a6dc220f-aec0-4454-9695-b44c0b9770e0 2024-05-15T12:47:42.282 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6 up in weight 1 up_from 48 up_thru 51 down_at 0 last_clean_interval [0,0) [v2:172.21.15.120:6816/3323706735,v1:172.21.15.120:6817/3323706735] [v2:172.21.15.120:6818/3323706735,v1:172.21.15.120:6819/3323706735] exists,up 9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b 2024-05-15T12:47:42.283 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7 up in weight 1 up_from 56 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.120:6824/2960616153,v1:172.21.15.120:6825/2960616153] [v2:172.21.15.120:6826/2960616153,v1:172.21.15.120:6827/2960616153] exists,up 6ef085d8-f59b-4033-8280-2c8f78693e30 2024-05-15T12:47:42.283 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8 up in weight 1 up_from 65 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6800/1132459315,v1:172.21.15.177:6801/1132459315] [v2:172.21.15.177:6802/1132459315,v1:172.21.15.177:6803/1132459315] exists,up 26514b11-d02b-46e9-a4e8-3da193fcdbcc 2024-05-15T12:47:42.283 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9 up in weight 1 up_from 76 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6808/2710457646,v1:172.21.15.177:6809/2710457646] [v2:172.21.15.177:6810/2710457646,v1:172.21.15.177:6811/2710457646] exists,up 6f9497bc-e962-409d-8a2c-0c634cc1124f 2024-05-15T12:47:42.283 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10 up in weight 1 up_from 81 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6816/3236900263,v1:172.21.15.177:6817/3236900263] [v2:172.21.15.177:6818/3236900263,v1:172.21.15.177:6819/3236900263] exists,up f852876f-c7d1-4039-90c4-5ab09dbd3e32 2024-05-15T12:47:42.283 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11 up in weight 1 up_from 90 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6824/681011628,v1:172.21.15.177:6825/681011628] [v2:172.21.15.177:6826/681011628,v1:172.21.15.177:6827/681011628] exists,up cccc53c5-8476-4d55-b160-4bdc5cf92902 2024-05-15T12:47:42.283 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:6801/2530617473 expires 2024-05-16T12:39:19.409195+0000 2024-05-15T12:47:42.283 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:6800/2530617473 expires 2024-05-16T12:39:19.409195+0000 2024-05-15T12:47:42.283 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/1880261204 expires 2024-05-16T12:39:19.409195+0000 2024-05-15T12:47:42.283 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:6801/3441136471 expires 2024-05-16T12:38:50.931011+0000 2024-05-15T12:47:42.283 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/4291446247 expires 2024-05-16T12:39:19.409195+0000 2024-05-15T12:47:42.283 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/580128209 expires 2024-05-16T12:38:50.931011+0000 2024-05-15T12:47:42.283 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:6800/3441136471 expires 2024-05-16T12:38:50.931011+0000 2024-05-15T12:47:42.284 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/3011229857 expires 2024-05-16T12:38:50.931011+0000 2024-05-15T12:47:42.284 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/2109693913 expires 2024-05-16T12:38:50.931011+0000 2024-05-15T12:47:42.284 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/2445663024 expires 2024-05-16T12:39:19.409195+0000 2024-05-15T12:47:42.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:42 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2734329585' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-15T12:47:42.725 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:42 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2734329585' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-15T12:47:42.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:42 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2734329585' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-15T12:47:43.049 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 58 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6802/4239116935,v1:172.21.15.27:6803/4239116935] [v2:172.21.15.27:6804/4239116935,v1:172.21.15.27:6805/4239116935] exists,up 64fb3ad2-08a7-4106-a07a-6c776c877faf', 'osd.1 up in weight 1 up_from 13 up_thru 34 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6810/1466270637,v1:172.21.15.27:6811/1466270637] [v2:172.21.15.27:6812/1466270637,v1:172.21.15.27:6813/1466270637] exists,up 9c16dd90-9d51-48b8-a296-1c8879ba1bc8', 'osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6818/3285413897,v1:172.21.15.27:6819/3285413897] [v2:172.21.15.27:6820/3285413897,v1:172.21.15.27:6821/3285413897] exists,up cbe3ba13-3ea3-4891-8d2d-6b85581a732c', 'osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6826/3008975244,v1:172.21.15.27:6827/3008975244] [v2:172.21.15.27:6828/3008975244,v1:172.21.15.27:6829/3008975244] exists,up 2e83735b-b944-4f87-8080-b994d4b96897', 'osd.4 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.120:6800/3877842957,v1:172.21.15.120:6801/3877842957] [v2:172.21.15.120:6802/3877842957,v1:172.21.15.120:6803/3877842957] exists,up 7c39c1dd-3867-46f7-8372-7688fa13b16a', 'osd.5 up in weight 1 up_from 40 up_thru 43 down_at 0 last_clean_interval [0,0) [v2:172.21.15.120:6808/1600179183,v1:172.21.15.120:6809/1600179183] [v2:172.21.15.120:6810/1600179183,v1:172.21.15.120:6811/1600179183] exists,up a6dc220f-aec0-4454-9695-b44c0b9770e0', 'osd.6 up in weight 1 up_from 48 up_thru 51 down_at 0 last_clean_interval [0,0) [v2:172.21.15.120:6816/3323706735,v1:172.21.15.120:6817/3323706735] [v2:172.21.15.120:6818/3323706735,v1:172.21.15.120:6819/3323706735] exists,up 9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b', 'osd.7 up in weight 1 up_from 56 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.120:6824/2960616153,v1:172.21.15.120:6825/2960616153] [v2:172.21.15.120:6826/2960616153,v1:172.21.15.120:6827/2960616153] exists,up 6ef085d8-f59b-4033-8280-2c8f78693e30', 'osd.8 up in weight 1 up_from 65 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6800/1132459315,v1:172.21.15.177:6801/1132459315] [v2:172.21.15.177:6802/1132459315,v1:172.21.15.177:6803/1132459315] exists,up 26514b11-d02b-46e9-a4e8-3da193fcdbcc', 'osd.9 up in weight 1 up_from 76 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6808/2710457646,v1:172.21.15.177:6809/2710457646] [v2:172.21.15.177:6810/2710457646,v1:172.21.15.177:6811/2710457646] exists,up 6f9497bc-e962-409d-8a2c-0c634cc1124f', 'osd.10 up in weight 1 up_from 81 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6816/3236900263,v1:172.21.15.177:6817/3236900263] [v2:172.21.15.177:6818/3236900263,v1:172.21.15.177:6819/3236900263] exists,up f852876f-c7d1-4039-90c4-5ab09dbd3e32', 'osd.11 up in weight 1 up_from 90 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6824/681011628,v1:172.21.15.177:6825/681011628] [v2:172.21.15.177:6826/681011628,v1:172.21.15.177:6827/681011628] exists,up cccc53c5-8476-4d55-b160-4bdc5cf92902'] 2024-05-15T12:47:43.050 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon mon.a config show 2024-05-15T12:47:43.326 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:47:43.611 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:43 smithi027 ceph-mon[39795]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:43.612 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:43 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:47:43.612 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:43 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:47:43.612 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:43 smithi027 ceph-mon[39795]: osdmap e108: 12 total, 12 up, 12 in 2024-05-15T12:47:43.612 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:43 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:47:43.612 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:43 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:47:43.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:43 smithi177 ceph-mon[44303]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:43.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:43 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:47:43.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:43 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:47:43.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:43 smithi177 ceph-mon[44303]: osdmap e108: 12 total, 12 up, 12 in 2024-05-15T12:47:43.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:43 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:47:43.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:43 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:47:43.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:43 smithi120 ceph-mon[43961]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:43.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:43 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:47:43.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:43 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:47:43.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:43 smithi120 ceph-mon[43961]: osdmap e108: 12 total, 12 up, 12 in 2024-05-15T12:47:43.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:43 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:47:43.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:43 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:47:44.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:44 smithi177 ceph-mon[44303]: osdmap e109: 12 total, 12 up, 12 in 2024-05-15T12:47:44.714 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:44 smithi027 ceph-mon[39795]: osdmap e109: 12 total, 12 up, 12 in 2024-05-15T12:47:44.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:44 smithi120 ceph-mon[43961]: osdmap e109: 12 total, 12 up, 12 in 2024-05-15T12:47:45.375 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T12:47:45.376 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mon.a", 2024-05-15T12:47:45.376 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster": "ceph", 2024-05-15T12:47:45.376 INFO:teuthology.orchestra.run.smithi027.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-05-15T12:47:45.376 INFO:teuthology.orchestra.run.smithi027.stdout: "admin_socket_mode": "", 2024-05-15T12:47:45.376 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-05-15T12:47:45.376 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_client_required": "cephx, none", 2024-05-15T12:47:45.376 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_cluster_required": "cephx", 2024-05-15T12:47:45.376 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_debug": "true", 2024-05-15T12:47:45.376 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-05-15T12:47:45.376 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-05-15T12:47:45.376 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_service_required": "cephx", 2024-05-15T12:47:45.377 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_service_ticket_ttl": "240.000000", 2024-05-15T12:47:45.377 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_supported": "", 2024-05-15T12:47:45.377 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_aio": "true", 2024-05-15T12:47:45.377 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_aio_max_queue_depth": "1024", 2024-05-15T12:47:45.377 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_aio_poll_ms": "250", 2024-05-15T12:47:45.377 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_aio_reap_max": "16", 2024-05-15T12:47:45.377 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_async_discard": "false", 2024-05-15T12:47:45.377 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_block_size": "4096", 2024-05-15T12:47:45.377 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_debug_aio": "false", 2024-05-15T12:47:45.377 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-05-15T12:47:45.377 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-05-15T12:47:45.377 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_debug_inflight_ios": "false", 2024-05-15T12:47:45.377 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_enable_discard": "false", 2024-05-15T12:47:45.378 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_flock_retry": "3", 2024-05-15T12:47:45.378 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_flock_retry_interval": "0.100000", 2024-05-15T12:47:45.378 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_inject_crash": "0", 2024-05-15T12:47:45.378 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_inject_crash_flush_delay": "2", 2024-05-15T12:47:45.378 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_ioring": "false", 2024-05-15T12:47:45.378 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_ioring_hipri": "false", 2024-05-15T12:47:45.378 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_ioring_sqthread_poll": "false", 2024-05-15T12:47:45.378 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-05-15T12:47:45.378 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_read_buffer_alignment": "4096", 2024-05-15T12:47:45.378 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-05-15T12:47:45.378 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_type": "", 2024-05-15T12:47:45.378 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_alloc_size": "1048576", 2024-05-15T12:47:45.378 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_allocator": "hybrid", 2024-05-15T12:47:45.379 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_buffered_io": "true", 2024-05-15T12:47:45.379 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_check_for_zeros": "false", 2024-05-15T12:47:45.379 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_check_volume_selector_often": "false", 2024-05-15T12:47:45.379 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-05-15T12:47:45.379 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_compact_log_sync": "false", 2024-05-15T12:47:45.379 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-05-15T12:47:45.379 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-05-15T12:47:45.379 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_log_compact_min_size": "16777216", 2024-05-15T12:47:45.379 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_log_replay_check_allocations": "true", 2024-05-15T12:47:45.379 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_max_log_runway": "4194304", 2024-05-15T12:47:45.379 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_max_prefetch": "1048576", 2024-05-15T12:47:45.379 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_min_flush_size": "524288", 2024-05-15T12:47:45.379 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_min_log_runway": "1048576", 2024-05-15T12:47:45.380 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_replay_recovery": "false", 2024-05-15T12:47:45.380 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-05-15T12:47:45.380 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_shared_alloc_size": "65536", 2024-05-15T12:47:45.380 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_sync_write": "false", 2024-05-15T12:47:45.380 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-05-15T12:47:45.380 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-05-15T12:47:45.380 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-05-15T12:47:45.381 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_allocation_from_file": "true", 2024-05-15T12:47:45.381 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_allocator": "hybrid", 2024-05-15T12:47:45.381 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-05-15T12:47:45.381 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-05-15T12:47:45.381 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-05-15T12:47:45.381 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-05-15T12:47:45.382 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-05-15T12:47:45.382 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-05-15T12:47:45.382 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blobid_prealloc": "10240", 2024-05-15T12:47:45.382 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_block_create": "true", 2024-05-15T12:47:45.382 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_block_db_create": "false", 2024-05-15T12:47:45.382 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_block_db_path": "", 2024-05-15T12:47:45.382 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_block_db_size": "0", 2024-05-15T12:47:45.382 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_block_path": "", 2024-05-15T12:47:45.382 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_block_preallocate_file": "false", 2024-05-15T12:47:45.382 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_block_size": "107374182400", 2024-05-15T12:47:45.382 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_block_wal_create": "false", 2024-05-15T12:47:45.382 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_block_wal_path": "", 2024-05-15T12:47:45.383 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_block_wal_size": "100663296", 2024-05-15T12:47:45.383 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_bluefs": "true", 2024-05-15T12:47:45.383 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-05-15T12:47:45.383 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_bluefs_env_mirror": "false", 2024-05-15T12:47:45.383 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-05-15T12:47:45.383 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-05-15T12:47:45.383 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-05-15T12:47:45.383 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-05-15T12:47:45.383 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-05-15T12:47:45.383 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-05-15T12:47:45.383 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_autotune": "true", 2024-05-15T12:47:45.383 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-05-15T12:47:45.383 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-05-15T12:47:45.384 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-05-15T12:47:45.384 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-05-15T12:47:45.384 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_size": "0", 2024-05-15T12:47:45.384 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-05-15T12:47:45.384 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-05-15T12:47:45.384 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-05-15T12:47:45.384 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-05-15T12:47:45.384 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_type": "2q", 2024-05-15T12:47:45.384 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-05-15T12:47:45.384 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_clone_cow": "true", 2024-05-15T12:47:45.384 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_compression_algorithm": "snappy", 2024-05-15T12:47:45.384 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_compression_max_blob_size": "0", 2024-05-15T12:47:45.384 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-05-15T12:47:45.384 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-05-15T12:47:45.385 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_compression_min_blob_size": "0", 2024-05-15T12:47:45.385 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-05-15T12:47:45.385 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-05-15T12:47:45.385 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_compression_mode": "none", 2024-05-15T12:47:45.385 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-05-15T12:47:45.385 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_csum_type": "crc32c", 2024-05-15T12:47:45.385 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-05-15T12:47:45.385 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_enforce_settings": "default", 2024-05-15T12:47:45.385 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_freelist": "false", 2024-05-15T12:47:45.385 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_fsck_abort": "false", 2024-05-15T12:47:45.385 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-05-15T12:47:45.385 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-05-15T12:47:45.385 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_inject_read_err": "false", 2024-05-15T12:47:45.386 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_legacy_omap": "false", 2024-05-15T12:47:45.386 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-05-15T12:47:45.386 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-05-15T12:47:45.386 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-05-15T12:47:45.386 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-05-15T12:47:45.386 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_prefill": "0.000000", 2024-05-15T12:47:45.386 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-05-15T12:47:45.386 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-05-15T12:47:45.386 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-05-15T12:47:45.386 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_small_allocations": "0", 2024-05-15T12:47:45.386 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-05-15T12:47:45.386 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_default_buffered_read": "true", 2024-05-15T12:47:45.386 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_default_buffered_write": "false", 2024-05-15T12:47:45.387 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_deferred_batch_ops": "0", 2024-05-15T12:47:45.387 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-05-15T12:47:45.387 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-05-15T12:47:45.387 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_elastic_shared_blobs": "true", 2024-05-15T12:47:45.387 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-05-15T12:47:45.387 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-05-15T12:47:45.387 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-05-15T12:47:45.387 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-05-15T12:47:45.387 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-05-15T12:47:45.387 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fail_eio": "false", 2024-05-15T12:47:45.387 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-05-15T12:47:45.387 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-05-15T12:47:45.388 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-05-15T12:47:45.388 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-05-15T12:47:45.388 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_on_mkfs": "true", 2024-05-15T12:47:45.388 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-05-15T12:47:45.388 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_on_mount": "false", 2024-05-15T12:47:45.388 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-05-15T12:47:45.388 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_on_umount": "false", 2024-05-15T12:47:45.388 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-05-15T12:47:45.388 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-05-15T12:47:45.388 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-05-15T12:47:45.388 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-05-15T12:47:45.388 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-05-15T12:47:45.388 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-05-15T12:47:45.388 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-05-15T12:47:45.388 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-05-15T12:47:45.389 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_ignore_data_csum": "false", 2024-05-15T12:47:45.389 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-05-15T12:47:45.389 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_kvbackend": "rocksdb", 2024-05-15T12:47:45.389 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-05-15T12:47:45.389 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-05-15T12:47:45.389 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_log_op_age": "5.000000", 2024-05-15T12:47:45.389 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_max_alloc_size": "0", 2024-05-15T12:47:45.389 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_max_blob_size": "0", 2024-05-15T12:47:45.389 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-05-15T12:47:45.389 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-05-15T12:47:45.389 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_max_defer_interval": "3.000000", 2024-05-15T12:47:45.389 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_max_deferred_txc": "32", 2024-05-15T12:47:45.389 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_min_alloc_size": "0", 2024-05-15T12:47:45.390 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-05-15T12:47:45.390 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-05-15T12:47:45.390 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_nid_prealloc": "1024", 2024-05-15T12:47:45.390 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_prefer_deferred_size": "0", 2024-05-15T12:47:45.390 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-05-15T12:47:45.390 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-05-15T12:47:45.390 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_qfsck_on_mount": "true", 2024-05-15T12:47:45.390 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_retry_disk_reads": "3", 2024-05-15T12:47:45.390 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_rocksdb_cf": "true", 2024-05-15T12:47:45.390 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-05-15T12:47:45.390 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-05-15T12:47:45.390 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_rocksdb_options_annex": "", 2024-05-15T12:47:45.391 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_spdk_coremask": "0x1", 2024-05-15T12:47:45.391 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_spdk_io_sleep": "5", 2024-05-15T12:47:45.391 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_spdk_max_io_completion": "0", 2024-05-15T12:47:45.391 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_spdk_mem": "512", 2024-05-15T12:47:45.391 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_sync_submit_transaction": "false", 2024-05-15T12:47:45.391 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_throttle_bytes": "67108864", 2024-05-15T12:47:45.391 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_throttle_cost_per_io": "0", 2024-05-15T12:47:45.391 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-05-15T12:47:45.391 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-05-15T12:47:45.391 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-05-15T12:47:45.391 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-05-15T12:47:45.391 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_tracing": "false", 2024-05-15T12:47:45.391 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-05-15T12:47:45.391 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-05-15T12:47:45.392 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_volume_selection_reserved": "0", 2024-05-15T12:47:45.392 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-05-15T12:47:45.392 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-05-15T12:47:45.392 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-05-15T12:47:45.392 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-05-15T12:47:45.392 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-05-15T12:47:45.392 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-05-15T12:47:45.392 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_zero_block_detection": "false", 2024-05-15T12:47:45.392 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-05-15T12:47:45.392 INFO:teuthology.orchestra.run.smithi027.stdout: "cephfs_mirror_action_update_interval": "2", 2024-05-15T12:47:45.392 INFO:teuthology.orchestra.run.smithi027.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-05-15T12:47:45.392 INFO:teuthology.orchestra.run.smithi027.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-05-15T12:47:45.392 INFO:teuthology.orchestra.run.smithi027.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-05-15T12:47:45.392 INFO:teuthology.orchestra.run.smithi027.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-05-15T12:47:45.393 INFO:teuthology.orchestra.run.smithi027.stdout: "cephfs_mirror_mount_timeout": "10", 2024-05-15T12:47:45.393 INFO:teuthology.orchestra.run.smithi027.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-05-15T12:47:45.393 INFO:teuthology.orchestra.run.smithi027.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-05-15T12:47:45.393 INFO:teuthology.orchestra.run.smithi027.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-05-15T12:47:45.393 INFO:teuthology.orchestra.run.smithi027.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-05-15T12:47:45.393 INFO:teuthology.orchestra.run.smithi027.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-05-15T12:47:45.393 INFO:teuthology.orchestra.run.smithi027.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-05-15T12:47:45.393 INFO:teuthology.orchestra.run.smithi027.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-05-15T12:47:45.393 INFO:teuthology.orchestra.run.smithi027.stdout: "cephx_cluster_require_signatures": "false", 2024-05-15T12:47:45.393 INFO:teuthology.orchestra.run.smithi027.stdout: "cephx_cluster_require_version": "2", 2024-05-15T12:47:45.393 INFO:teuthology.orchestra.run.smithi027.stdout: "cephx_require_signatures": "false", 2024-05-15T12:47:45.393 INFO:teuthology.orchestra.run.smithi027.stdout: "cephx_require_version": "2", 2024-05-15T12:47:45.393 INFO:teuthology.orchestra.run.smithi027.stdout: "cephx_service_require_signatures": "false", 2024-05-15T12:47:45.393 INFO:teuthology.orchestra.run.smithi027.stdout: "cephx_service_require_version": "2", 2024-05-15T12:47:45.394 INFO:teuthology.orchestra.run.smithi027.stdout: "cephx_sign_messages": "true", 2024-05-15T12:47:45.394 INFO:teuthology.orchestra.run.smithi027.stdout: "chdir": "", 2024-05-15T12:47:45.394 INFO:teuthology.orchestra.run.smithi027.stdout: "client_acl_type": "", 2024-05-15T12:47:45.394 INFO:teuthology.orchestra.run.smithi027.stdout: "client_asio_thread_count": "2", 2024-05-15T12:47:45.394 INFO:teuthology.orchestra.run.smithi027.stdout: "client_cache_mid": "0.750000", 2024-05-15T12:47:45.394 INFO:teuthology.orchestra.run.smithi027.stdout: "client_cache_size": "16384", 2024-05-15T12:47:45.394 INFO:teuthology.orchestra.run.smithi027.stdout: "client_caps_release_delay": "5", 2024-05-15T12:47:45.394 INFO:teuthology.orchestra.run.smithi027.stdout: "client_check_pool_perm": "true", 2024-05-15T12:47:45.394 INFO:teuthology.orchestra.run.smithi027.stdout: "client_collect_and_send_global_metrics": "false", 2024-05-15T12:47:45.394 INFO:teuthology.orchestra.run.smithi027.stdout: "client_debug_force_sync_read": "false", 2024-05-15T12:47:45.394 INFO:teuthology.orchestra.run.smithi027.stdout: "client_debug_getattr_caps": "false", 2024-05-15T12:47:45.394 INFO:teuthology.orchestra.run.smithi027.stdout: "client_debug_inject_tick_delay": "0", 2024-05-15T12:47:45.395 INFO:teuthology.orchestra.run.smithi027.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-05-15T12:47:45.395 INFO:teuthology.orchestra.run.smithi027.stdout: "client_die_on_failed_remount": "false", 2024-05-15T12:47:45.395 INFO:teuthology.orchestra.run.smithi027.stdout: "client_dirsize_rbytes": "true", 2024-05-15T12:47:45.395 INFO:teuthology.orchestra.run.smithi027.stdout: "client_force_lazyio": "false", 2024-05-15T12:47:45.395 INFO:teuthology.orchestra.run.smithi027.stdout: "client_fs": "", 2024-05-15T12:47:45.395 INFO:teuthology.orchestra.run.smithi027.stdout: "client_inject_fixed_oldest_tid": "false", 2024-05-15T12:47:45.395 INFO:teuthology.orchestra.run.smithi027.stdout: "client_inject_release_failure": "false", 2024-05-15T12:47:45.395 INFO:teuthology.orchestra.run.smithi027.stdout: "client_max_inline_size": "4096", 2024-05-15T12:47:45.395 INFO:teuthology.orchestra.run.smithi027.stdout: "client_max_retries_on_remount_failure": "5", 2024-05-15T12:47:45.395 INFO:teuthology.orchestra.run.smithi027.stdout: "client_mds_namespace": "", 2024-05-15T12:47:45.395 INFO:teuthology.orchestra.run.smithi027.stdout: "client_metadata": "", 2024-05-15T12:47:45.395 INFO:teuthology.orchestra.run.smithi027.stdout: "client_mount_gid": "-1", 2024-05-15T12:47:45.395 INFO:teuthology.orchestra.run.smithi027.stdout: "client_mount_timeout": "300", 2024-05-15T12:47:45.395 INFO:teuthology.orchestra.run.smithi027.stdout: "client_mount_uid": "-1", 2024-05-15T12:47:45.396 INFO:teuthology.orchestra.run.smithi027.stdout: "client_mountpoint": "/", 2024-05-15T12:47:45.396 INFO:teuthology.orchestra.run.smithi027.stdout: "client_notify_timeout": "10", 2024-05-15T12:47:45.396 INFO:teuthology.orchestra.run.smithi027.stdout: "client_oc": "true", 2024-05-15T12:47:45.396 INFO:teuthology.orchestra.run.smithi027.stdout: "client_oc_max_dirty": "104857600", 2024-05-15T12:47:45.396 INFO:teuthology.orchestra.run.smithi027.stdout: "client_oc_max_dirty_age": "5.000000", 2024-05-15T12:47:45.396 INFO:teuthology.orchestra.run.smithi027.stdout: "client_oc_max_objects": "1000", 2024-05-15T12:47:45.396 INFO:teuthology.orchestra.run.smithi027.stdout: "client_oc_size": "209715200", 2024-05-15T12:47:45.396 INFO:teuthology.orchestra.run.smithi027.stdout: "client_oc_target_dirty": "8388608", 2024-05-15T12:47:45.396 INFO:teuthology.orchestra.run.smithi027.stdout: "client_permissions": "true", 2024-05-15T12:47:45.396 INFO:teuthology.orchestra.run.smithi027.stdout: "client_quota": "true", 2024-05-15T12:47:45.396 INFO:teuthology.orchestra.run.smithi027.stdout: "client_quota_df": "true", 2024-05-15T12:47:45.396 INFO:teuthology.orchestra.run.smithi027.stdout: "client_readahead_max_bytes": "0", 2024-05-15T12:47:45.396 INFO:teuthology.orchestra.run.smithi027.stdout: "client_readahead_max_periods": "4", 2024-05-15T12:47:45.396 INFO:teuthology.orchestra.run.smithi027.stdout: "client_readahead_min": "131072", 2024-05-15T12:47:45.397 INFO:teuthology.orchestra.run.smithi027.stdout: "client_reconnect_stale": "false", 2024-05-15T12:47:45.397 INFO:teuthology.orchestra.run.smithi027.stdout: "client_shutdown_timeout": "30", 2024-05-15T12:47:45.397 INFO:teuthology.orchestra.run.smithi027.stdout: "client_snapdir": ".snap", 2024-05-15T12:47:45.397 INFO:teuthology.orchestra.run.smithi027.stdout: "client_tick_interval": "1", 2024-05-15T12:47:45.397 INFO:teuthology.orchestra.run.smithi027.stdout: "client_trace": "", 2024-05-15T12:47:45.397 INFO:teuthology.orchestra.run.smithi027.stdout: "client_try_dentry_invalidate": "false", 2024-05-15T12:47:45.397 INFO:teuthology.orchestra.run.smithi027.stdout: "client_use_faked_inos": "false", 2024-05-15T12:47:45.397 INFO:teuthology.orchestra.run.smithi027.stdout: "client_use_random_mds": "false", 2024-05-15T12:47:45.397 INFO:teuthology.orchestra.run.smithi027.stdout: "clog_to_graylog": "false", 2024-05-15T12:47:45.397 INFO:teuthology.orchestra.run.smithi027.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-05-15T12:47:45.397 INFO:teuthology.orchestra.run.smithi027.stdout: "clog_to_graylog_port": "12201", 2024-05-15T12:47:45.397 INFO:teuthology.orchestra.run.smithi027.stdout: "clog_to_monitors": "default=true", 2024-05-15T12:47:45.397 INFO:teuthology.orchestra.run.smithi027.stdout: "clog_to_syslog": "false", 2024-05-15T12:47:45.398 INFO:teuthology.orchestra.run.smithi027.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-05-15T12:47:45.398 INFO:teuthology.orchestra.run.smithi027.stdout: "clog_to_syslog_level": "info", 2024-05-15T12:47:45.398 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "-", 2024-05-15T12:47:45.398 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_network": "", 2024-05-15T12:47:45.398 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_network_interface": "", 2024-05-15T12:47:45.398 INFO:teuthology.orchestra.run.smithi027.stdout: "compressor_zlib_isal": "false", 2024-05-15T12:47:45.398 INFO:teuthology.orchestra.run.smithi027.stdout: "compressor_zlib_level": "5", 2024-05-15T12:47:45.398 INFO:teuthology.orchestra.run.smithi027.stdout: "compressor_zlib_winsize": "-15", 2024-05-15T12:47:45.398 INFO:teuthology.orchestra.run.smithi027.stdout: "compressor_zstd_level": "1", 2024-05-15T12:47:45.398 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328", 2024-05-15T12:47:45.398 INFO:teuthology.orchestra.run.smithi027.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-05-15T12:47:45.398 INFO:teuthology.orchestra.run.smithi027.stdout: "crimson_alien_op_num_threads": "6", 2024-05-15T12:47:45.399 INFO:teuthology.orchestra.run.smithi027.stdout: "crimson_alien_thread_cpu_cores": "", 2024-05-15T12:47:45.399 INFO:teuthology.orchestra.run.smithi027.stdout: "crimson_osd_obc_lru_size": "512", 2024-05-15T12:47:45.399 INFO:teuthology.orchestra.run.smithi027.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-05-15T12:47:45.399 INFO:teuthology.orchestra.run.smithi027.stdout: "crimson_seastar_cpu_cores": "", 2024-05-15T12:47:45.399 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_location": "", 2024-05-15T12:47:45.399 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_location_hook": "", 2024-05-15T12:47:45.399 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_location_hook_timeout": "10", 2024-05-15T12:47:45.399 INFO:teuthology.orchestra.run.smithi027.stdout: "daemonize": "false", 2024-05-15T12:47:45.399 INFO:teuthology.orchestra.run.smithi027.stdout: "daos_pool": "tank", 2024-05-15T12:47:45.399 INFO:teuthology.orchestra.run.smithi027.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-05-15T12:47:45.399 INFO:teuthology.orchestra.run.smithi027.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-05-15T12:47:45.400 INFO:teuthology.orchestra.run.smithi027.stdout: "dbstore_db_name_prefix": "dbstore", 2024-05-15T12:47:45.400 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_alienstore": "0/5", 2024-05-15T12:47:45.400 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_allow_any_pool_priority": "false", 2024-05-15T12:47:45.400 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_asok": "1/5", 2024-05-15T12:47:45.400 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_asok_assert_abort": "false", 2024-05-15T12:47:45.400 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_asserts_on_shutdown": "true", 2024-05-15T12:47:45.400 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_auth": "1/5", 2024-05-15T12:47:45.400 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_bdev": "1/3", 2024-05-15T12:47:45.400 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_bluefs": "1/5", 2024-05-15T12:47:45.400 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_bluestore": "1/5", 2024-05-15T12:47:45.400 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_buffer": "0/1", 2024-05-15T12:47:45.400 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_ceph_exporter": "1/5", 2024-05-15T12:47:45.400 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_cephfs_mirror": "0/5", 2024-05-15T12:47:45.400 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_cephsqlite": "0/5", 2024-05-15T12:47:45.401 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_client": "0/5", 2024-05-15T12:47:45.401 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_compressor": "1/5", 2024-05-15T12:47:45.401 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_context": "0/1", 2024-05-15T12:47:45.401 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_crimson_interrupt": "0/5", 2024-05-15T12:47:45.401 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_crush": "1/1", 2024-05-15T12:47:45.401 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_crypto": "1/5", 2024-05-15T12:47:45.401 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_cyanstore": "0/5", 2024-05-15T12:47:45.401 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_deliberately_leak_memory": "false", 2024-05-15T12:47:45.401 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_disable_randomized_ping": "false", 2024-05-15T12:47:45.401 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_dpdk": "1/5", 2024-05-15T12:47:45.401 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_eventtrace": "1/5", 2024-05-15T12:47:45.401 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_filer": "0/1", 2024-05-15T12:47:45.401 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_filestore": "1/3", 2024-05-15T12:47:45.401 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_finisher": "1/1", 2024-05-15T12:47:45.402 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_fuse": "1/5", 2024-05-15T12:47:45.402 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_heartbeat_testing_span": "0", 2024-05-15T12:47:45.402 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_heartbeatmap": "1/5", 2024-05-15T12:47:45.402 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_immutable_obj_cache": "0/5", 2024-05-15T12:47:45.402 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_javaclient": "1/5", 2024-05-15T12:47:45.402 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_journal": "1/3", 2024-05-15T12:47:45.402 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_journaler": "0/5", 2024-05-15T12:47:45.402 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_kstore": "1/5", 2024-05-15T12:47:45.402 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_lockdep": "0/1", 2024-05-15T12:47:45.402 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_mclock": "1/5", 2024-05-15T12:47:45.402 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_mds": "1/5", 2024-05-15T12:47:45.402 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_mds_balancer": "1/5", 2024-05-15T12:47:45.402 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_mds_locker": "1/5", 2024-05-15T12:47:45.402 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_mds_log": "1/5", 2024-05-15T12:47:45.403 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_mds_log_expire": "1/5", 2024-05-15T12:47:45.403 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_mds_migrator": "1/5", 2024-05-15T12:47:45.403 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_memstore": "1/5", 2024-05-15T12:47:45.403 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_mgr": "2/5", 2024-05-15T12:47:45.403 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_mgrc": "1/5", 2024-05-15T12:47:45.403 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_mon": "20/20", 2024-05-15T12:47:45.403 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_monc": "0/10", 2024-05-15T12:47:45.403 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_ms": "1/1", 2024-05-15T12:47:45.403 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_none": "0/5", 2024-05-15T12:47:45.403 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_objclass": "0/5", 2024-05-15T12:47:45.403 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_objectcacher": "0/5", 2024-05-15T12:47:45.403 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_objecter": "0/1", 2024-05-15T12:47:45.403 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_optracker": "0/5", 2024-05-15T12:47:45.404 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_osd": "1/5", 2024-05-15T12:47:45.404 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_paxos": "20/20", 2024-05-15T12:47:45.404 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_perfcounter": "1/5", 2024-05-15T12:47:45.404 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_prioritycache": "1/5", 2024-05-15T12:47:45.404 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rados": "0/5", 2024-05-15T12:47:45.404 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rbd": "0/5", 2024-05-15T12:47:45.404 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rbd_mirror": "0/5", 2024-05-15T12:47:45.404 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rbd_pwl": "0/5", 2024-05-15T12:47:45.404 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rbd_replay": "0/5", 2024-05-15T12:47:45.404 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_refs": "0/0", 2024-05-15T12:47:45.404 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_reserver": "1/1", 2024-05-15T12:47:45.404 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rgw": "1/5", 2024-05-15T12:47:45.404 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rgw_access": "1/5", 2024-05-15T12:47:45.404 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rgw_datacache": "1/5", 2024-05-15T12:47:45.405 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rgw_dbstore": "1/5", 2024-05-15T12:47:45.405 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rgw_flight": "1/5", 2024-05-15T12:47:45.405 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rgw_lifecycle": "1/5", 2024-05-15T12:47:45.405 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rgw_sync": "1/5", 2024-05-15T12:47:45.405 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rocksdb": "4/5", 2024-05-15T12:47:45.405 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore": "0/5", 2024-05-15T12:47:45.405 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_backref": "0/5", 2024-05-15T12:47:45.405 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_cache": "0/5", 2024-05-15T12:47:45.405 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_cleaner": "0/5", 2024-05-15T12:47:45.405 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_device": "0/5", 2024-05-15T12:47:45.405 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_epm": "0/5", 2024-05-15T12:47:45.405 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-05-15T12:47:45.405 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_journal": "0/5", 2024-05-15T12:47:45.405 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_lba": "0/5", 2024-05-15T12:47:45.406 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_odata": "0/5", 2024-05-15T12:47:45.406 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_omap": "0/5", 2024-05-15T12:47:45.406 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_onode": "0/5", 2024-05-15T12:47:45.406 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_t": "0/5", 2024-05-15T12:47:45.406 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_tm": "0/5", 2024-05-15T12:47:45.406 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_striper": "0/1", 2024-05-15T12:47:45.406 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_test": "0/5", 2024-05-15T12:47:45.406 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_throttle": "1/1", 2024-05-15T12:47:45.406 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_timer": "0/1", 2024-05-15T12:47:45.406 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_tp": "0/5", 2024-05-15T12:47:45.406 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_trace": "1/5", 2024-05-15T12:47:45.406 INFO:teuthology.orchestra.run.smithi027.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-05-15T12:47:45.406 INFO:teuthology.orchestra.run.smithi027.stdout: "device_failure_prediction_mode": "none", 2024-05-15T12:47:45.407 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-05-15T12:47:45.407 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-05-15T12:47:45.407 INFO:teuthology.orchestra.run.smithi027.stdout: "err_to_graylog": "false", 2024-05-15T12:47:45.407 INFO:teuthology.orchestra.run.smithi027.stdout: "err_to_journald": "false", 2024-05-15T12:47:45.407 INFO:teuthology.orchestra.run.smithi027.stdout: "err_to_stderr": "true", 2024-05-15T12:47:45.407 INFO:teuthology.orchestra.run.smithi027.stdout: "err_to_syslog": "false", 2024-05-15T12:47:45.407 INFO:teuthology.orchestra.run.smithi027.stdout: "event_tracing": "false", 2024-05-15T12:47:45.407 INFO:teuthology.orchestra.run.smithi027.stdout: "exporter_addr": "0.0.0.0", 2024-05-15T12:47:45.407 INFO:teuthology.orchestra.run.smithi027.stdout: "exporter_http_port": "9926", 2024-05-15T12:47:45.407 INFO:teuthology.orchestra.run.smithi027.stdout: "exporter_prio_limit": "5", 2024-05-15T12:47:45.407 INFO:teuthology.orchestra.run.smithi027.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-05-15T12:47:45.407 INFO:teuthology.orchestra.run.smithi027.stdout: "exporter_sort_metrics": "true", 2024-05-15T12:47:45.407 INFO:teuthology.orchestra.run.smithi027.stdout: "exporter_stats_period": "5", 2024-05-15T12:47:45.408 INFO:teuthology.orchestra.run.smithi027.stdout: "fake_statfs_for_testing": "0", 2024-05-15T12:47:45.408 INFO:teuthology.orchestra.run.smithi027.stdout: "fatal_signal_handlers": "true", 2024-05-15T12:47:45.408 INFO:teuthology.orchestra.run.smithi027.stdout: "filer_max_purge_ops": "10", 2024-05-15T12:47:45.408 INFO:teuthology.orchestra.run.smithi027.stdout: "filer_max_truncate_ops": "128", 2024-05-15T12:47:45.408 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_apply_finisher_threads": "1", 2024-05-15T12:47:45.408 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_blackhole": "false", 2024-05-15T12:47:45.408 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_btrfs_clone_range": "true", 2024-05-15T12:47:45.408 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_btrfs_snap": "true", 2024-05-15T12:47:45.408 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_caller_concurrency": "10", 2024-05-15T12:47:45.408 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_collect_device_partition_information": "true", 2024-05-15T12:47:45.408 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_commit_timeout": "600.000000", 2024-05-15T12:47:45.408 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_debug_inject_read_err": "false", 2024-05-15T12:47:45.409 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_debug_omap_check": "false", 2024-05-15T12:47:45.409 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_debug_random_read_err": "0.000000", 2024-05-15T12:47:45.409 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_debug_verify_split": "false", 2024-05-15T12:47:45.409 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_dump_file": "", 2024-05-15T12:47:45.409 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-05-15T12:47:45.409 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-05-15T12:47:45.409 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_fadvise": "true", 2024-05-15T12:47:45.409 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_fail_eio": "true", 2024-05-15T12:47:45.409 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_fd_cache_shards": "16", 2024-05-15T12:47:45.409 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_fd_cache_size": "128", 2024-05-15T12:47:45.409 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_fiemap": "false", 2024-05-15T12:47:45.409 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_fiemap_threshold": "4096", 2024-05-15T12:47:45.409 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-05-15T12:47:45.409 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_index_retry_probability": "0.000000", 2024-05-15T12:47:45.410 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_inject_stall": "0", 2024-05-15T12:47:45.410 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_journal_parallel": "false", 2024-05-15T12:47:45.410 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_journal_trailing": "false", 2024-05-15T12:47:45.410 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_journal_writeahead": "false", 2024-05-15T12:47:45.410 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_kill_at": "0", 2024-05-15T12:47:45.410 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-05-15T12:47:45.410 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_inline_xattr_size": "0", 2024-05-15T12:47:45.410 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-05-15T12:47:45.410 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-05-15T12:47:45.410 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-05-15T12:47:45.410 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_inline_xattrs": "0", 2024-05-15T12:47:45.410 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-05-15T12:47:45.410 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_inline_xattrs_other": "2", 2024-05-15T12:47:45.410 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-05-15T12:47:45.411 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_sync_interval": "5.000000", 2024-05-15T12:47:45.411 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_xattr_value_size": "0", 2024-05-15T12:47:45.411 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-05-15T12:47:45.411 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-05-15T12:47:45.411 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-05-15T12:47:45.411 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_merge_threshold": "-10", 2024-05-15T12:47:45.411 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_min_sync_interval": "0.010000", 2024-05-15T12:47:45.411 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_odsync_write": "false", 2024-05-15T12:47:45.411 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_omap_backend": "rocksdb", 2024-05-15T12:47:45.411 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_omap_backend_path": "", 2024-05-15T12:47:45.411 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_omap_header_cache_size": "1024", 2024-05-15T12:47:45.411 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_ondisk_finisher_threads": "1", 2024-05-15T12:47:45.411 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-05-15T12:47:45.411 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_op_thread_timeout": "60", 2024-05-15T12:47:45.412 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_op_threads": "2", 2024-05-15T12:47:45.412 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_punch_hole": "false", 2024-05-15T12:47:45.412 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-05-15T12:47:45.412 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-05-15T12:47:45.412 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-05-15T12:47:45.412 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-05-15T12:47:45.412 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-05-15T12:47:45.412 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_queue_max_bytes": "104857600", 2024-05-15T12:47:45.412 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-05-15T12:47:45.412 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-05-15T12:47:45.412 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-05-15T12:47:45.412 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_queue_max_ops": "50", 2024-05-15T12:47:45.412 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-05-15T12:47:45.412 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_seek_data_hole": "false", 2024-05-15T12:47:45.413 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_sloppy_crc": "false", 2024-05-15T12:47:45.413 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-05-15T12:47:45.413 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_splice": "false", 2024-05-15T12:47:45.413 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_split_multiple": "2", 2024-05-15T12:47:45.413 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_split_rand_factor": "20", 2024-05-15T12:47:45.413 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_update_to": "1000", 2024-05-15T12:47:45.413 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-05-15T12:47:45.413 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-05-15T12:47:45.413 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-05-15T12:47:45.413 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-05-15T12:47:45.413 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-05-15T12:47:45.413 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-05-15T12:47:45.413 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_enable": "true", 2024-05-15T12:47:45.414 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-05-15T12:47:45.414 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-05-15T12:47:45.414 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-05-15T12:47:45.414 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-05-15T12:47:45.414 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-05-15T12:47:45.414 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-05-15T12:47:45.414 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_xfs_extsize": "false", 2024-05-15T12:47:45.414 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_zfs_snap": "false", 2024-05-15T12:47:45.414 INFO:teuthology.orchestra.run.smithi027.stdout: "fio_dir": "/tmp/fio", 2024-05-15T12:47:45.414 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "deaea04e-12b7-11ef-bc98-c7b262605968", 2024-05-15T12:47:45.414 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_allow_other": "true", 2024-05-15T12:47:45.414 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_atomic_o_trunc": "true", 2024-05-15T12:47:45.414 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_big_writes": "true", 2024-05-15T12:47:45.414 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_debug": "false", 2024-05-15T12:47:45.415 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_default_permissions": "false", 2024-05-15T12:47:45.415 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_disable_pagecache": "false", 2024-05-15T12:47:45.415 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_max_write": "0", 2024-05-15T12:47:45.415 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_multithreaded": "true", 2024-05-15T12:47:45.415 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_require_active_mds": "true", 2024-05-15T12:47:45.415 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_set_user_groups": "true", 2024-05-15T12:47:45.415 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_splice_move": "true", 2024-05-15T12:47:45.415 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_splice_read": "true", 2024-05-15T12:47:45.415 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_splice_write": "true", 2024-05-15T12:47:45.415 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_syncfs_on_mksnap": "true", 2024-05-15T12:47:45.415 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_use_invalidate_cb": "true", 2024-05-15T12:47:45.415 INFO:teuthology.orchestra.run.smithi027.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-05-15T12:47:45.415 INFO:teuthology.orchestra.run.smithi027.stdout: "gss_target_name": "ceph", 2024-05-15T12:47:45.415 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_file": "", 2024-05-15T12:47:45.416 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_inject_failure": "0", 2024-05-15T12:47:45.416 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_interval": "5", 2024-05-15T12:47:45.416 INFO:teuthology.orchestra.run.smithi027.stdout: "host": "smithi027", 2024-05-15T12:47:45.416 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-05-15T12:47:45.416 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-05-15T12:47:45.416 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_max_size": "1073741824", 2024-05-15T12:47:45.416 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-05-15T12:47:45.416 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-05-15T12:47:45.416 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-05-15T12:47:45.416 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-05-15T12:47:45.416 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-05-15T12:47:45.416 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-05-15T12:47:45.416 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-05-15T12:47:45.417 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-05-15T12:47:45.417 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-05-15T12:47:45.417 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_watermark": "0.900000", 2024-05-15T12:47:45.417 INFO:teuthology.orchestra.run.smithi027.stdout: "inject_early_sigterm": "false", 2024-05-15T12:47:45.417 INFO:teuthology.orchestra.run.smithi027.stdout: "jaeger_agent_port": "6799", 2024-05-15T12:47:45.417 INFO:teuthology.orchestra.run.smithi027.stdout: "jaeger_tracing_enable": "false", 2024-05-15T12:47:45.417 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_aio": "true", 2024-05-15T12:47:45.417 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_align_min_size": "65536", 2024-05-15T12:47:45.417 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_block_align": "true", 2024-05-15T12:47:45.417 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_block_size": "4096", 2024-05-15T12:47:45.417 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_dio": "true", 2024-05-15T12:47:45.418 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_discard": "false", 2024-05-15T12:47:45.418 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_force_aio": "false", 2024-05-15T12:47:45.418 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_ignore_corruption": "false", 2024-05-15T12:47:45.418 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_max_write_bytes": "10485760", 2024-05-15T12:47:45.418 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_max_write_entries": "100", 2024-05-15T12:47:45.418 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_replay_from": "0", 2024-05-15T12:47:45.418 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_throttle_high_multiple": "0.000000", 2024-05-15T12:47:45.418 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-05-15T12:47:45.418 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-05-15T12:47:45.418 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_throttle_max_multiple": "0.000000", 2024-05-15T12:47:45.418 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_write_header_frequency": "0", 2024-05-15T12:47:45.418 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_zero_on_create": "false", 2024-05-15T12:47:45.418 INFO:teuthology.orchestra.run.smithi027.stdout: "journaler_prefetch_periods": "10", 2024-05-15T12:47:45.419 INFO:teuthology.orchestra.run.smithi027.stdout: "journaler_prezero_periods": "5", 2024-05-15T12:47:45.419 INFO:teuthology.orchestra.run.smithi027.stdout: "journaler_write_head_interval": "15", 2024-05-15T12:47:45.419 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "", 2024-05-15T12:47:45.419 INFO:teuthology.orchestra.run.smithi027.stdout: "keyfile": "", 2024-05-15T12:47:45.419 INFO:teuthology.orchestra.run.smithi027.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-05-15T12:47:45.419 INFO:teuthology.orchestra.run.smithi027.stdout: "kstore_backend": "rocksdb", 2024-05-15T12:47:45.419 INFO:teuthology.orchestra.run.smithi027.stdout: "kstore_default_stripe_size": "65536", 2024-05-15T12:47:45.419 INFO:teuthology.orchestra.run.smithi027.stdout: "kstore_fsck_on_mount": "false", 2024-05-15T12:47:45.419 INFO:teuthology.orchestra.run.smithi027.stdout: "kstore_fsck_on_mount_deep": "true", 2024-05-15T12:47:45.419 INFO:teuthology.orchestra.run.smithi027.stdout: "kstore_max_bytes": "67108864", 2024-05-15T12:47:45.419 INFO:teuthology.orchestra.run.smithi027.stdout: "kstore_max_ops": "512", 2024-05-15T12:47:45.419 INFO:teuthology.orchestra.run.smithi027.stdout: "kstore_nid_prealloc": "1024", 2024-05-15T12:47:45.419 INFO:teuthology.orchestra.run.smithi027.stdout: "kstore_onode_map_size": "1024", 2024-05-15T12:47:45.420 INFO:teuthology.orchestra.run.smithi027.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-05-15T12:47:45.420 INFO:teuthology.orchestra.run.smithi027.stdout: "kstore_sync_submit_transaction": "false", 2024-05-15T12:47:45.420 INFO:teuthology.orchestra.run.smithi027.stdout: "kstore_sync_transaction": "false", 2024-05-15T12:47:45.420 INFO:teuthology.orchestra.run.smithi027.stdout: "librados_thread_count": "2", 2024-05-15T12:47:45.420 INFO:teuthology.orchestra.run.smithi027.stdout: "lockdep": "false", 2024-05-15T12:47:45.420 INFO:teuthology.orchestra.run.smithi027.stdout: "lockdep_force_backtrace": "false", 2024-05-15T12:47:45.420 INFO:teuthology.orchestra.run.smithi027.stdout: "log_coarse_timestamps": "true", 2024-05-15T12:47:45.420 INFO:teuthology.orchestra.run.smithi027.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-05-15T12:47:45.420 INFO:teuthology.orchestra.run.smithi027.stdout: "log_flush_on_exit": "false", 2024-05-15T12:47:45.420 INFO:teuthology.orchestra.run.smithi027.stdout: "log_graylog_host": "127.0.0.1", 2024-05-15T12:47:45.420 INFO:teuthology.orchestra.run.smithi027.stdout: "log_graylog_port": "12201", 2024-05-15T12:47:45.420 INFO:teuthology.orchestra.run.smithi027.stdout: "log_max_new": "1000", 2024-05-15T12:47:45.420 INFO:teuthology.orchestra.run.smithi027.stdout: "log_max_recent": "10000", 2024-05-15T12:47:45.420 INFO:teuthology.orchestra.run.smithi027.stdout: "log_stderr_prefix": "", 2024-05-15T12:47:45.421 INFO:teuthology.orchestra.run.smithi027.stdout: "log_stop_at_utilization": "0.970000", 2024-05-15T12:47:45.421 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": "true", 2024-05-15T12:47:45.421 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_graylog": "false", 2024-05-15T12:47:45.421 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_journald": "false", 2024-05-15T12:47:45.421 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_stderr": "false", 2024-05-15T12:47:45.421 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_syslog": "false", 2024-05-15T12:47:45.421 INFO:teuthology.orchestra.run.smithi027.stdout: "max_rotating_auth_attempts": "10", 2024-05-15T12:47:45.421 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-05-15T12:47:45.421 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_action_on_write_error": "1", 2024-05-15T12:47:45.421 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_alternate_name_max": "8192", 2024-05-15T12:47:45.421 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_asio_thread_count": "2", 2024-05-15T12:47:45.421 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_export_pin": "true", 2024-05-15T12:47:45.421 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_fragment_dirs": "true", 2024-05-15T12:47:45.421 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-05-15T12:47:45.422 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_fragment_interval": "5", 2024-05-15T12:47:45.422 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_fragment_size_max": "100000", 2024-05-15T12:47:45.422 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_idle_threshold": "0.000000", 2024-05-15T12:47:45.422 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_interval": "10", 2024-05-15T12:47:45.422 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_max": "-1", 2024-05-15T12:47:45.422 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_max_until": "-1", 2024-05-15T12:47:45.422 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_merge_size": "50", 2024-05-15T12:47:45.422 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_midchunk": "0.300000", 2024-05-15T12:47:45.422 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_min_rebalance": "0.100000", 2024-05-15T12:47:45.422 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_min_start": "0.200000", 2024-05-15T12:47:45.422 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_minchunk": "0.001000", 2024-05-15T12:47:45.422 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_mode": "0", 2024-05-15T12:47:45.422 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_need_max": "1.200000", 2024-05-15T12:47:45.423 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_need_min": "0.800000", 2024-05-15T12:47:45.423 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_overload_epochs": "2", 2024-05-15T12:47:45.423 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-05-15T12:47:45.423 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_sample_interval": "3.000000", 2024-05-15T12:47:45.423 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_split_bits": "3", 2024-05-15T12:47:45.423 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_split_rd": "25000.000000", 2024-05-15T12:47:45.423 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_split_size": "10000", 2024-05-15T12:47:45.423 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_split_wr": "10000.000000", 2024-05-15T12:47:45.423 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_target_decay": "10.000000", 2024-05-15T12:47:45.423 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-05-15T12:47:45.423 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_beacon_grace": "15.000000", 2024-05-15T12:47:45.423 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_beacon_interval": "4.000000", 2024-05-15T12:47:45.424 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_beacon_mon_down_grace": "60", 2024-05-15T12:47:45.424 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache_memory_limit": "4294967296", 2024-05-15T12:47:45.424 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache_mid": "0.700000", 2024-05-15T12:47:45.424 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache_release_free_interval": "10", 2024-05-15T12:47:45.424 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache_reservation": "0.050000", 2024-05-15T12:47:45.424 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-05-15T12:47:45.424 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache_trim_interval": "1", 2024-05-15T12:47:45.424 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache_trim_threshold": "262144", 2024-05-15T12:47:45.424 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-05-15T12:47:45.424 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-05-15T12:47:45.424 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_client_delegate_inos_pct": "50", 2024-05-15T12:47:45.425 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_client_prealloc_inos": "1000", 2024-05-15T12:47:45.425 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-05-15T12:47:45.425 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_connect_bootstrapping": "false", 2024-05-15T12:47:45.425 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_damage_table_max_entries": "10000", 2024-05-15T12:47:45.425 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-05-15T12:47:45.425 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_debug_auth_pins": "false", 2024-05-15T12:47:45.425 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_debug_frag": "false", 2024-05-15T12:47:45.425 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_debug_scatterstat": "false", 2024-05-15T12:47:45.425 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_debug_subtrees": "false", 2024-05-15T12:47:45.425 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_decay_halflife": "5.000000", 2024-05-15T12:47:45.425 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_default_dir_hash": "2", 2024-05-15T12:47:45.425 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_defer_session_stale": "true", 2024-05-15T12:47:45.425 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_deny_all_reconnect": "false", 2024-05-15T12:47:45.425 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_dir_keys_per_op": "16384", 2024-05-15T12:47:45.426 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_dir_max_commit_size": "10", 2024-05-15T12:47:45.426 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_dir_max_entries": "0", 2024-05-15T12:47:45.426 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_dir_prefetch": "true", 2024-05-15T12:47:45.426 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_dirstat_min_interval": "1.000000", 2024-05-15T12:47:45.426 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_dump_cache_after_rejoin": "false", 2024-05-15T12:47:45.426 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_dump_cache_on_map": "false", 2024-05-15T12:47:45.426 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_dump_cache_threshold_file": "0", 2024-05-15T12:47:45.426 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-05-15T12:47:45.426 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_early_reply": "true", 2024-05-15T12:47:45.426 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_enable_op_tracker": "true", 2024-05-15T12:47:45.426 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_enforce_unique_name": "true", 2024-05-15T12:47:45.426 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_export_ephemeral_distributed": "true", 2024-05-15T12:47:45.426 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-05-15T12:47:45.427 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_export_ephemeral_random": "true", 2024-05-15T12:47:45.427 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-05-15T12:47:45.427 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-05-15T12:47:45.427 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_forward_all_requests_to_auth": "false", 2024-05-15T12:47:45.427 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-05-15T12:47:45.427 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-05-15T12:47:45.427 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-05-15T12:47:45.427 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-05-15T12:47:45.428 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_health_cache_threshold": "1.500000", 2024-05-15T12:47:45.428 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_health_summarize_threshold": "10", 2024-05-15T12:47:45.428 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_heartbeat_grace": "15.000000", 2024-05-15T12:47:45.428 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_heartbeat_reset_grace": "1000", 2024-05-15T12:47:45.428 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_inject_health_dummy": "false", 2024-05-15T12:47:45.428 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-05-15T12:47:45.428 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_inject_migrator_session_race": "false", 2024-05-15T12:47:45.428 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-05-15T12:47:45.428 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-05-15T12:47:45.428 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-05-15T12:47:45.428 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_join_fs": "", 2024-05-15T12:47:45.428 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_journal_format": "1", 2024-05-15T12:47:45.428 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_create_at": "0", 2024-05-15T12:47:45.429 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_export_at": "0", 2024-05-15T12:47:45.429 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_import_at": "0", 2024-05-15T12:47:45.429 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_journal_at": "0", 2024-05-15T12:47:45.429 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_journal_expire_at": "0", 2024-05-15T12:47:45.429 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_journal_replay_at": "0", 2024-05-15T12:47:45.429 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_link_at": "0", 2024-05-15T12:47:45.429 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_mdstable_at": "0", 2024-05-15T12:47:45.429 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_openc_at": "0", 2024-05-15T12:47:45.429 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_rename_at": "0", 2024-05-15T12:47:45.429 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_shutdown_at": "0", 2024-05-15T12:47:45.429 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-05-15T12:47:45.429 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_event_large_threshold": "524288", 2024-05-15T12:47:45.429 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_events_per_segment": "1024", 2024-05-15T12:47:45.430 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_major_segment_event_ratio": "12", 2024-05-15T12:47:45.430 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_max_events": "-1", 2024-05-15T12:47:45.430 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_max_segments": "128", 2024-05-15T12:47:45.430 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_pause": "false", 2024-05-15T12:47:45.430 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_segment_size": "0", 2024-05-15T12:47:45.430 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_skip_corrupt_events": "false", 2024-05-15T12:47:45.430 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_skip_unbounded_events": "false", 2024-05-15T12:47:45.430 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-05-15T12:47:45.430 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_trim_threshold": "128", 2024-05-15T12:47:45.430 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-05-15T12:47:45.430 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_warn_factor": "2.000000", 2024-05-15T12:47:45.431 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_max_caps_per_client": "1048576", 2024-05-15T12:47:45.431 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_max_completed_flushes": "100000", 2024-05-15T12:47:45.431 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_max_completed_requests": "100000", 2024-05-15T12:47:45.431 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_max_export_size": "20971520", 2024-05-15T12:47:45.431 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_max_file_recover": "32", 2024-05-15T12:47:45.431 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_max_purge_files": "64", 2024-05-15T12:47:45.431 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_max_purge_ops": "8192", 2024-05-15T12:47:45.431 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-05-15T12:47:45.431 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-05-15T12:47:45.431 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_max_snaps_per_dir": "100", 2024-05-15T12:47:45.431 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_metrics_update_interval": "2", 2024-05-15T12:47:45.431 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_min_caps_per_client": "100", 2024-05-15T12:47:45.431 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_min_caps_working_set": "10000", 2024-05-15T12:47:45.431 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-05-15T12:47:45.432 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_numa_node": "-1", 2024-05-15T12:47:45.432 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-05-15T12:47:45.432 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_op_complaint_time": "30.000000", 2024-05-15T12:47:45.432 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_op_history_duration": "600", 2024-05-15T12:47:45.432 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_op_history_size": "20", 2024-05-15T12:47:45.432 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_op_history_slow_op_size": "20", 2024-05-15T12:47:45.432 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_op_history_slow_op_threshold": "10", 2024-05-15T12:47:45.432 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_op_log_threshold": "5", 2024-05-15T12:47:45.432 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_ping_grace": "15", 2024-05-15T12:47:45.432 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_ping_interval": "5", 2024-05-15T12:47:45.432 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-05-15T12:47:45.432 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-05-15T12:47:45.432 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_recall_max_caps": "30000", 2024-05-15T12:47:45.432 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-05-15T12:47:45.433 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_recall_max_decay_threshold": "131072", 2024-05-15T12:47:45.433 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-05-15T12:47:45.433 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_recall_warning_threshold": "262144", 2024-05-15T12:47:45.433 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_reconnect_timeout": "45.000000", 2024-05-15T12:47:45.433 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_replay_interval": "1.000000", 2024-05-15T12:47:45.433 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-05-15T12:47:45.433 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-05-15T12:47:45.433 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_root_ino_gid": "0", 2024-05-15T12:47:45.433 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_root_ino_uid": "0", 2024-05-15T12:47:45.433 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-05-15T12:47:45.433 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_session_blocklist_on_evict": "true", 2024-05-15T12:47:45.433 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_session_blocklist_on_timeout": "true", 2024-05-15T12:47:45.433 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-05-15T12:47:45.433 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-05-15T12:47:45.434 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-05-15T12:47:45.434 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-05-15T12:47:45.434 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-05-15T12:47:45.434 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_session_metadata_threshold": "16777216", 2024-05-15T12:47:45.434 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-05-15T12:47:45.434 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_shutdown_check": "0", 2024-05-15T12:47:45.434 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_skip_ino": "0", 2024-05-15T12:47:45.434 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_sleep_rank_change": "0.000000", 2024-05-15T12:47:45.434 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_snap_max_uid": "4294967294", 2024-05-15T12:47:45.434 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_snap_min_uid": "0", 2024-05-15T12:47:45.434 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_snap_rstat": "false", 2024-05-15T12:47:45.434 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_standby_replay_damaged": "false", 2024-05-15T12:47:45.435 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_symlink_recovery": "true", 2024-05-15T12:47:45.435 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_task_status_update_interval": "2.000000", 2024-05-15T12:47:45.435 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_thrash_exports": "0", 2024-05-15T12:47:45.435 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_thrash_fragments": "0", 2024-05-15T12:47:45.435 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_tick_interval": "5.000000", 2024-05-15T12:47:45.435 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_valgrind_exit": "false", 2024-05-15T12:47:45.435 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_verify_backtrace": "1", 2024-05-15T12:47:45.435 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_verify_scatter": "false", 2024-05-15T12:47:45.435 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_wipe_ino_prealloc": "false", 2024-05-15T12:47:45.435 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_wipe_sessions": "false", 2024-05-15T12:47:45.435 INFO:teuthology.orchestra.run.smithi027.stdout: "mempool_debug": "false", 2024-05-15T12:47:45.435 INFO:teuthology.orchestra.run.smithi027.stdout: "memstore_debug_omit_block_device_write": "false", 2024-05-15T12:47:45.435 INFO:teuthology.orchestra.run.smithi027.stdout: "memstore_device_bytes": "1073741824", 2024-05-15T12:47:45.435 INFO:teuthology.orchestra.run.smithi027.stdout: "memstore_page_set": "false", 2024-05-15T12:47:45.436 INFO:teuthology.orchestra.run.smithi027.stdout: "memstore_page_size": "65536", 2024-05-15T12:47:45.436 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_client_bytes": "134217728", 2024-05-15T12:47:45.436 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_client_messages": "512", 2024-05-15T12:47:45.436 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-05-15T12:47:45.436 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_connect_retry_interval": "1.000000", 2024-05-15T12:47:45.436 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-05-15T12:47:45.436 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-05-15T12:47:45.436 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_disabled_modules": "", 2024-05-15T12:47:45.436 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-05-15T12:47:45.436 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_max_pg_num_change": "128", 2024-05-15T12:47:45.436 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_mds_bytes": "134217728", 2024-05-15T12:47:45.436 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_mds_messages": "128", 2024-05-15T12:47:45.436 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-05-15T12:47:45.436 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_mon_bytes": "134217728", 2024-05-15T12:47:45.437 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_mon_messages": "128", 2024-05-15T12:47:45.437 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_osd_bytes": "536870912", 2024-05-15T12:47:45.437 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_osd_messages": "8192", 2024-05-15T12:47:45.437 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_pool": "true", 2024-05-15T12:47:45.437 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_service_beacon_grace": "60.000000", 2024-05-15T12:47:45.437 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_standby_modules": "true", 2024-05-15T12:47:45.437 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_stats_period": "5", 2024-05-15T12:47:45.437 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_stats_threshold": "5", 2024-05-15T12:47:45.437 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_tick_period": "2", 2024-05-15T12:47:45.437 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-05-15T12:47:45.437 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_accept_timeout_factor": "2.000000", 2024-05-15T12:47:45.437 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_allow_pool_delete": "true", 2024-05-15T12:47:45.437 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_allow_pool_size_one": "false", 2024-05-15T12:47:45.437 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_auth_validate_all_caps": "true", 2024-05-15T12:47:45.438 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-05-15T12:47:45.438 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-05-15T12:47:45.438 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_client_bytes": "104857600", 2024-05-15T12:47:45.438 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_client_directed_command_retry": "5", 2024-05-15T12:47:45.438 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_client_hunt_interval": "3.000000", 2024-05-15T12:47:45.438 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-05-15T12:47:45.438 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-05-15T12:47:45.438 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-05-15T12:47:45.438 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_client_hunt_parallel": "3", 2024-05-15T12:47:45.439 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_client_log_interval": "1.000000", 2024-05-15T12:47:45.439 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-05-15T12:47:45.439 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_client_ping_interval": "10.000000", 2024-05-15T12:47:45.439 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_client_ping_timeout": "30.000000", 2024-05-15T12:47:45.439 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_client_target_rank": "-1", 2024-05-15T12:47:45.439 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_clock_drift_allowed": "1.000000", 2024-05-15T12:47:45.439 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-05-15T12:47:45.439 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-05-15T12:47:45.439 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cluster_log_file_level": "debug", 2024-05-15T12:47:45.439 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cluster_log_to_file": "true", 2024-05-15T12:47:45.439 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cluster_log_to_graylog": "false", 2024-05-15T12:47:45.439 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-05-15T12:47:45.439 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-05-15T12:47:45.439 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cluster_log_to_journald": "true", 2024-05-15T12:47:45.440 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cluster_log_to_stderr": "false", 2024-05-15T12:47:45.440 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-05-15T12:47:45.440 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-05-15T12:47:45.440 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-05-15T12:47:45.440 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_compact_on_bootstrap": "false", 2024-05-15T12:47:45.440 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_compact_on_start": "false", 2024-05-15T12:47:45.440 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_compact_on_trim": "true", 2024-05-15T12:47:45.440 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_con_tracker_persist_interval": "10", 2024-05-15T12:47:45.440 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_con_tracker_score_halflife": "43200", 2024-05-15T12:47:45.440 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_config_key_max_entry_size": "65536", 2024-05-15T12:47:45.440 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cpu_threads": "4", 2024-05-15T12:47:45.440 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_crush_min_required_version": "hammer", 2024-05-15T12:47:45.441 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_daemon_bytes": "419430400", 2024-05-15T12:47:45.441 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-05-15T12:47:45.441 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_data_avail_crit": "5", 2024-05-15T12:47:45.441 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_data_avail_warn": "5", 2024-05-15T12:47:45.441 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_data_size_warn": "16106127360", 2024-05-15T12:47:45.441 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_debug_block_osdmap_trim": "false", 2024-05-15T12:47:45.441 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-05-15T12:47:45.441 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_debug_dump_json": "false", 2024-05-15T12:47:45.441 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-05-15T12:47:45.441 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_debug_dump_transactions": "false", 2024-05-15T12:47:45.441 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_debug_extra_checks": "false", 2024-05-15T12:47:45.441 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-05-15T12:47:45.441 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-05-15T12:47:45.441 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_debug_no_require_reef": "false", 2024-05-15T12:47:45.442 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_debug_no_require_squid": "false", 2024-05-15T12:47:45.442 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-05-15T12:47:45.442 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_delta_reset_interval": "10.000000", 2024-05-15T12:47:45.442 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_dns_srv_name": "ceph-mon", 2024-05-15T12:47:45.442 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_down_mkfs_grace": "60", 2024-05-15T12:47:45.442 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_election_default_strategy": "3", 2024-05-15T12:47:45.442 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_election_timeout": "5.000000", 2024-05-15T12:47:45.442 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-05-15T12:47:45.442 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_elector_ping_divisor": "2", 2024-05-15T12:47:45.442 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_elector_ping_timeout": "2.000000", 2024-05-15T12:47:45.442 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_enable_op_tracker": "true", 2024-05-15T12:47:45.443 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_fake_pool_delete": "false", 2024-05-15T12:47:45.443 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_force_quorum_join": "false", 2024-05-15T12:47:45.443 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_globalid_prealloc": "10000", 2024-05-15T12:47:45.443 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_health_detail_to_clog": "true", 2024-05-15T12:47:45.488 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_health_log_update_period": "5", 2024-05-15T12:47:45.488 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_health_max_detail": "50", 2024-05-15T12:47:45.488 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_health_to_clog": "true", 2024-05-15T12:47:45.488 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_health_to_clog_interval": "600", 2024-05-15T12:47:45.488 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-05-15T12:47:45.488 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_host": "[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0]", 2024-05-15T12:47:45.488 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_host_override": "", 2024-05-15T12:47:45.489 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_initial_members": "", 2024-05-15T12:47:45.489 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-05-15T12:47:45.489 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-05-15T12:47:45.489 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-05-15T12:47:45.489 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-05-15T12:47:45.489 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_keyvaluedb": "rocksdb", 2024-05-15T12:47:45.489 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_lease": "5.000000", 2024-05-15T12:47:45.489 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-05-15T12:47:45.489 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-05-15T12:47:45.489 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_log_full_interval": "50", 2024-05-15T12:47:45.489 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_log_max": "10000", 2024-05-15T12:47:45.490 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_log_max_summary": "50", 2024-05-15T12:47:45.490 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_max_log_entries_per_event": "4096", 2024-05-15T12:47:45.490 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_max_log_epochs": "500", 2024-05-15T12:47:45.490 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_max_mdsmap_epochs": "500", 2024-05-15T12:47:45.490 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_max_mgrmap_epochs": "500", 2024-05-15T12:47:45.490 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_max_osd": "10000", 2024-05-15T12:47:45.490 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_max_pg_per_osd": "10000", 2024-05-15T12:47:45.490 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_max_pool_pg_num": "65536", 2024-05-15T12:47:45.490 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-05-15T12:47:45.490 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-05-15T12:47:45.490 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_mds_force_trim_to": "0", 2024-05-15T12:47:45.490 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_mds_skip_sanity": "false", 2024-05-15T12:47:45.490 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_memory_autotune": "true", 2024-05-15T12:47:45.491 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_memory_target": "2147483648", 2024-05-15T12:47:45.491 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_mgr_beacon_grace": "30", 2024-05-15T12:47:45.491 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-05-15T12:47:45.491 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_mgr_digest_period": "5", 2024-05-15T12:47:45.491 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_mgr_inactive_grace": "60", 2024-05-15T12:47:45.491 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_mgr_mkfs_grace": "240", 2024-05-15T12:47:45.491 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-05-15T12:47:45.491 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_min_osdmap_epochs": "50", 2024-05-15T12:47:45.491 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_op_complaint_time": "30", 2024-05-15T12:47:45.491 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_op_history_duration": "600", 2024-05-15T12:47:45.491 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_op_history_size": "20", 2024-05-15T12:47:45.492 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_op_history_slow_op_size": "20", 2024-05-15T12:47:45.492 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_op_history_slow_op_threshold": "10", 2024-05-15T12:47:45.492 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_op_log_threshold": "5", 2024-05-15T12:47:45.492 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-05-15T12:47:45.492 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-05-15T12:47:45.492 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-05-15T12:47:45.492 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_auto_mark_in": "false", 2024-05-15T12:47:45.492 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_auto_mark_new_in": "true", 2024-05-15T12:47:45.492 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-05-15T12:47:45.493 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-05-15T12:47:45.493 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_cache_size": "500", 2024-05-15T12:47:45.493 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_cache_size_min": "134217728", 2024-05-15T12:47:45.493 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_crush_smoke_test": "true", 2024-05-15T12:47:45.493 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_destroyed_out_interval": "600", 2024-05-15T12:47:45.493 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_down_out_interval": "600", 2024-05-15T12:47:45.493 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-05-15T12:47:45.493 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-05-15T12:47:45.493 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_force_trim_to": "0", 2024-05-15T12:47:45.493 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_full_ratio": "0.950000", 2024-05-15T12:47:45.493 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-05-15T12:47:45.493 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_laggy_halflife": "3600", 2024-05-15T12:47:45.493 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_laggy_max_interval": "300", 2024-05-15T12:47:45.494 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_laggy_weight": "0.300000", 2024-05-15T12:47:45.494 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-05-15T12:47:45.494 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_max_creating_pgs": "1024", 2024-05-15T12:47:45.494 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_max_initial_pgs": "1024", 2024-05-15T12:47:45.494 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_min_down_reporters": "2", 2024-05-15T12:47:45.494 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-05-15T12:47:45.494 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-05-15T12:47:45.494 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-05-15T12:47:45.494 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_prime_pg_temp": "true", 2024-05-15T12:47:45.494 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-05-15T12:47:45.494 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-05-15T12:47:45.494 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_report_timeout": "900", 2024-05-15T12:47:45.494 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-05-15T12:47:45.494 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-05-15T12:47:45.495 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_warn_num_repaired": "10", 2024-05-15T12:47:45.495 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_warn_op_age": "32.000000", 2024-05-15T12:47:45.495 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-05-15T12:47:45.495 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osdmap_full_prune_interval": "2", 2024-05-15T12:47:45.495 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osdmap_full_prune_min": "15", 2024-05-15T12:47:45.495 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-05-15T12:47:45.495 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-05-15T12:47:45.495 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_pg_stuck_threshold": "60", 2024-05-15T12:47:45.495 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-05-15T12:47:45.495 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_pg_warn_min_objects": "10000", 2024-05-15T12:47:45.495 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_pg_warn_min_per_osd": "0", 2024-05-15T12:47:45.495 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-05-15T12:47:45.495 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_pool_quota_crit_threshold": "0", 2024-05-15T12:47:45.496 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_pool_quota_warn_threshold": "0", 2024-05-15T12:47:45.496 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_probe_timeout": "2.000000", 2024-05-15T12:47:45.496 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_reweight_max_change": "0.050000", 2024-05-15T12:47:45.496 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_reweight_max_osds": "4", 2024-05-15T12:47:45.496 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-05-15T12:47:45.496 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-05-15T12:47:45.496 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-05-15T12:47:45.496 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-05-15T12:47:45.496 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-05-15T12:47:45.496 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_scrub_interval": "300", 2024-05-15T12:47:45.496 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_scrub_max_keys": "100", 2024-05-15T12:47:45.496 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_scrub_timeout": "300", 2024-05-15T12:47:45.496 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_session_timeout": "300", 2024-05-15T12:47:45.496 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_smart_report_timeout": "5", 2024-05-15T12:47:45.497 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_stat_smooth_intervals": "6", 2024-05-15T12:47:45.497 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-05-15T12:47:45.497 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_stretch_pool_min_size": "2", 2024-05-15T12:47:45.497 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_stretch_pool_size": "4", 2024-05-15T12:47:45.497 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-05-15T12:47:45.497 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_subscribe_interval": "86400.000000", 2024-05-15T12:47:45.497 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_sync_debug": "false", 2024-05-15T12:47:45.497 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_sync_max_payload_keys": "2000", 2024-05-15T12:47:45.497 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_sync_max_payload_size": "1048576", 2024-05-15T12:47:45.497 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_sync_provider_kill_at": "0", 2024-05-15T12:47:45.497 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_sync_requester_kill_at": "0", 2024-05-15T12:47:45.497 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_sync_timeout": "60.000000", 2024-05-15T12:47:45.497 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_target_pg_per_osd": "100", 2024-05-15T12:47:45.497 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_tick_interval": "5", 2024-05-15T12:47:45.497 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_timecheck_interval": "300.000000", 2024-05-15T12:47:45.498 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-05-15T12:47:45.498 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_use_min_delay_socket": "false", 2024-05-15T12:47:45.498 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_older_version_delay": "604800", 2024-05-15T12:47:45.498 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-05-15T12:47:45.498 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-05-15T12:47:45.498 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-05-15T12:47:45.498 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_filestore_osds": "true", 2024-05-15T12:47:45.498 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-05-15T12:47:45.498 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-05-15T12:47:45.498 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-05-15T12:47:45.498 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_misplaced": "false", 2024-05-15T12:47:45.498 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-05-15T12:47:45.499 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_older_version": "true", 2024-05-15T12:47:45.499 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-05-15T12:47:45.499 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_pool_no_app": "true", 2024-05-15T12:47:45.499 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-05-15T12:47:45.499 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-05-15T12:47:45.499 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-05-15T12:47:45.499 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-05-15T12:47:45.499 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_too_few_osds": "false", 2024-05-15T12:47:45.499 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-05-15T12:47:45.499 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-05-15T12:47:45.499 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": "", 2024-05-15T12:47:45.499 INFO:teuthology.orchestra.run.smithi027.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-05-15T12:47:45.499 INFO:teuthology.orchestra.run.smithi027.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-05-15T12:47:45.499 INFO:teuthology.orchestra.run.smithi027.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-05-15T12:47:45.499 INFO:teuthology.orchestra.run.smithi027.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-05-15T12:47:45.500 INFO:teuthology.orchestra.run.smithi027.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-05-15T12:47:45.500 INFO:teuthology.orchestra.run.smithi027.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-05-15T12:47:45.500 INFO:teuthology.orchestra.run.smithi027.stdout: "motr_tracing_enabled": "false", 2024-05-15T12:47:45.500 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_op_threads": "3", 2024-05-15T12:47:45.500 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_buffer_size": "131072", 2024-05-15T12:47:45.500 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_cm": "false", 2024-05-15T12:47:45.500 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_device_name": "", 2024-05-15T12:47:45.500 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_dscp": "96", 2024-05-15T12:47:45.500 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-05-15T12:47:45.500 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_gid_idx": "0", 2024-05-15T12:47:45.500 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_local_gid": "", 2024-05-15T12:47:45.500 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_polling_us": "1000", 2024-05-15T12:47:45.500 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_port_num": "1", 2024-05-15T12:47:45.500 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-05-15T12:47:45.501 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-05-15T12:47:45.501 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_roce_ver": "1", 2024-05-15T12:47:45.501 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_send_buffers": "1024", 2024-05-15T12:47:45.501 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_sl": "3", 2024-05-15T12:47:45.501 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_support_srq": "true", 2024-05-15T12:47:45.501 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_type": "ib", 2024-05-15T12:47:45.501 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_reap_threshold": "5", 2024-05-15T12:47:45.501 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_bind_before_connect": "false", 2024-05-15T12:47:45.501 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_bind_ipv4": "true", 2024-05-15T12:47:45.501 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_bind_ipv6": "false", 2024-05-15T12:47:45.502 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_bind_msgr1": "true", 2024-05-15T12:47:45.502 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_bind_msgr2": "true", 2024-05-15T12:47:45.502 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_bind_port_max": "7568", 2024-05-15T12:47:45.502 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_bind_port_min": "6800", 2024-05-15T12:47:45.502 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_bind_prefer_ipv4": "false", 2024-05-15T12:47:45.502 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_bind_retry_count": "3", 2024-05-15T12:47:45.502 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_bind_retry_delay": "5", 2024-05-15T12:47:45.502 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_blackhole_client": "false", 2024-05-15T12:47:45.502 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_blackhole_mds": "false", 2024-05-15T12:47:45.502 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_blackhole_mgr": "false", 2024-05-15T12:47:45.502 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_blackhole_mon": "false", 2024-05-15T12:47:45.502 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_blackhole_osd": "false", 2024-05-15T12:47:45.503 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_client_mode": "crc secure", 2024-05-15T12:47:45.503 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_cluster_mode": "crc secure", 2024-05-15T12:47:45.503 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_cluster_type": "", 2024-05-15T12:47:45.503 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_compress_secure": "false", 2024-05-15T12:47:45.503 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_connection_idle_timeout": "900", 2024-05-15T12:47:45.503 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_connection_ready_timeout": "10", 2024-05-15T12:47:45.503 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_crc_data": "true", 2024-05-15T12:47:45.503 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_crc_header": "true", 2024-05-15T12:47:45.503 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_die_on_bad_msg": "false", 2024-05-15T12:47:45.503 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_die_on_bug": "true", 2024-05-15T12:47:45.504 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_die_on_old_message": "true", 2024-05-15T12:47:45.504 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_die_on_skipped_message": "false", 2024-05-15T12:47:45.504 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_die_on_unhandled_msg": "false", 2024-05-15T12:47:45.504 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-05-15T12:47:45.504 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_coremask": "0xF", 2024-05-15T12:47:45.504 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-05-15T12:47:45.504 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_devs_allowlist": "", 2024-05-15T12:47:45.504 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_enable_tso": "true", 2024-05-15T12:47:45.504 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-05-15T12:47:45.504 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-05-15T12:47:45.504 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_hugepages": "", 2024-05-15T12:47:45.504 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_hw_flow_control": "true", 2024-05-15T12:47:45.504 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-05-15T12:47:45.505 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_lro": "true", 2024-05-15T12:47:45.505 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_memory_channel": "4", 2024-05-15T12:47:45.505 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-05-15T12:47:45.505 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_pmd": "", 2024-05-15T12:47:45.505 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_port_id": "0", 2024-05-15T12:47:45.505 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-05-15T12:47:45.505 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dump_corrupt_message_level": "1", 2024-05-15T12:47:45.505 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dump_on_send": "false", 2024-05-15T12:47:45.505 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_initial_backoff": "0.200000", 2024-05-15T12:47:45.505 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_inject_delay_max": "1.000000", 2024-05-15T12:47:45.505 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_inject_delay_probability": "0.000000", 2024-05-15T12:47:45.505 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_inject_delay_type": "", 2024-05-15T12:47:45.505 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_inject_internal_delays": "0.000000", 2024-05-15T12:47:45.505 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_inject_network_congestion": "0", 2024-05-15T12:47:45.506 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_inject_socket_failures": "2500", 2024-05-15T12:47:45.506 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_learn_addr_from_peer": "true", 2024-05-15T12:47:45.506 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_max_accept_failures": "4", 2024-05-15T12:47:45.506 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_max_backoff": "15.000000", 2024-05-15T12:47:45.506 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_mon_client_mode": "secure crc", 2024-05-15T12:47:45.506 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_mon_cluster_mode": "secure crc", 2024-05-15T12:47:45.506 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_mon_service_mode": "secure crc", 2024-05-15T12:47:45.506 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_osd_compress_min_size": "1024", 2024-05-15T12:47:45.506 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_osd_compress_mode": "none", 2024-05-15T12:47:45.506 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_osd_compression_algorithm": "snappy", 2024-05-15T12:47:45.507 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-05-15T12:47:45.507 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_pq_min_cost": "65536", 2024-05-15T12:47:45.507 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_public_type": "", 2024-05-15T12:47:45.507 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_service_mode": "crc secure", 2024-05-15T12:47:45.507 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_tcp_listen_backlog": "512", 2024-05-15T12:47:45.507 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_tcp_nodelay": "true", 2024-05-15T12:47:45.507 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-05-15T12:47:45.507 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_tcp_rcvbuf": "0", 2024-05-15T12:47:45.507 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_type": "async+posix", 2024-05-15T12:47:45.507 INFO:teuthology.orchestra.run.smithi027.stdout: "no_config_file": "false", 2024-05-15T12:47:45.507 INFO:teuthology.orchestra.run.smithi027.stdout: "objecter_completion_locks_per_session": "32", 2024-05-15T12:47:45.507 INFO:teuthology.orchestra.run.smithi027.stdout: "objecter_debug_inject_relock_delay": "false", 2024-05-15T12:47:45.507 INFO:teuthology.orchestra.run.smithi027.stdout: "objecter_inflight_op_bytes": "104857600", 2024-05-15T12:47:45.507 INFO:teuthology.orchestra.run.smithi027.stdout: "objecter_inflight_ops": "1024", 2024-05-15T12:47:45.507 INFO:teuthology.orchestra.run.smithi027.stdout: "objecter_inject_no_watch_ping": "false", 2024-05-15T12:47:45.508 INFO:teuthology.orchestra.run.smithi027.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-05-15T12:47:45.508 INFO:teuthology.orchestra.run.smithi027.stdout: "objecter_tick_interval": "5.000000", 2024-05-15T12:47:45.508 INFO:teuthology.orchestra.run.smithi027.stdout: "objecter_timeout": "10.000000", 2024-05-15T12:47:45.508 INFO:teuthology.orchestra.run.smithi027.stdout: "objectstore_blackhole": "false", 2024-05-15T12:47:45.508 INFO:teuthology.orchestra.run.smithi027.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-05-15T12:47:45.508 INFO:teuthology.orchestra.run.smithi027.stdout: "openssl_engine_opts": "", 2024-05-15T12:47:45.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_agent_delay_time": "5.000000", 2024-05-15T12:47:45.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_agent_hist_halflife": "1000", 2024-05-15T12:47:45.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_agent_max_low_ops": "2", 2024-05-15T12:47:45.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_agent_max_ops": "4", 2024-05-15T12:47:45.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-05-15T12:47:45.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_agent_quantize_effort": "0.100000", 2024-05-15T12:47:45.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_agent_slop": "0.020000", 2024-05-15T12:47:45.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-05-15T12:47:45.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_allow_recovery_below_min_size": "true", 2024-05-15T12:47:45.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_asio_thread_count": "2", 2024-05-15T12:47:45.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_async_recovery_min_cost": "100", 2024-05-15T12:47:45.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_auto_mark_unfound_lost": "false", 2024-05-15T12:47:45.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_backfill_retry_interval": "30.000000", 2024-05-15T12:47:45.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_backfill_scan_max": "512", 2024-05-15T12:47:45.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_backfill_scan_min": "64", 2024-05-15T12:47:45.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_backoff_on_degraded": "false", 2024-05-15T12:47:45.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_backoff_on_peering": "false", 2024-05-15T12:47:45.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_backoff_on_unfound": "true", 2024-05-15T12:47:45.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_beacon_report_interval": "300", 2024-05-15T12:47:45.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_bench_duration": "30", 2024-05-15T12:47:45.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-05-15T12:47:45.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_bench_max_block_size": "67108864", 2024-05-15T12:47:45.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_bench_small_size_max_iops": "100", 2024-05-15T12:47:45.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_blkin_trace_all": "false", 2024-05-15T12:47:45.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_blocked_scrub_grace_period": "120", 2024-05-15T12:47:45.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-05-15T12:47:45.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-05-15T12:47:45.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-05-15T12:47:45.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_check_for_log_corruption": "false", 2024-05-15T12:47:45.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-05-15T12:47:45.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-05-15T12:47:45.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-05-15T12:47:45.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-05-15T12:47:45.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_class_update_on_start": "true", 2024-05-15T12:47:45.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_client_message_cap": "256", 2024-05-15T12:47:45.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_client_message_size_cap": "524288000", 2024-05-15T12:47:45.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_client_op_priority": "63", 2024-05-15T12:47:45.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_client_watch_timeout": "30", 2024-05-15T12:47:45.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_command_max_records": "256", 2024-05-15T12:47:45.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_compact_on_start": "false", 2024-05-15T12:47:45.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-05-15T12:47:45.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_crush_chooseleaf_type": "1", 2024-05-15T12:47:45.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_crush_initial_weight": "-1.000000", 2024-05-15T12:47:45.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_crush_update_on_start": "true", 2024-05-15T12:47:45.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_crush_update_weight_set": "true", 2024-05-15T12:47:45.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-05-15T12:47:45.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-05-15T12:47:45.512 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-05-15T12:47:45.512 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_drop_ping_duration": "0", 2024-05-15T12:47:45.512 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-05-15T12:47:45.512 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_feed_pullee": "-1", 2024-05-15T12:47:45.512 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-05-15T12:47:45.512 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-05-15T12:47:45.512 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-05-15T12:47:45.512 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_misdirected_ops": "false", 2024-05-15T12:47:45.512 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_no_acting_change": "false", 2024-05-15T12:47:45.512 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_no_purge_strays": "false", 2024-05-15T12:47:45.513 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_op_order": "false", 2024-05-15T12:47:45.513 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_pg_log_writeout": "false", 2024-05-15T12:47:45.513 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_pretend_recovery_active": "false", 2024-05-15T12:47:45.513 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-05-15T12:47:45.513 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-05-15T12:47:45.513 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_shutdown": "false", 2024-05-15T12:47:45.513 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-05-15T12:47:45.513 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-05-15T12:47:45.513 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_trim_objects": "false", 2024-05-15T12:47:45.513 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_verify_cached_snaps": "false", 2024-05-15T12:47:45.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_verify_missing_on_start": "false", 2024-05-15T12:47:45.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_verify_snaps": "false", 2024-05-15T12:47:45.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-05-15T12:47:45.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-05-15T12:47:45.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_deep_scrub_keys": "1024", 2024-05-15T12:47:45.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-05-15T12:47:45.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-05-15T12:47:45.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-05-15T12:47:45.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_deep_scrub_stride": "524288", 2024-05-15T12:47:45.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-05-15T12:47:45.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_default_data_pool_replay_window": "45", 2024-05-15T12:47:45.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_default_notify_timeout": "30", 2024-05-15T12:47:45.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_delete_sleep": "0.000000", 2024-05-15T12:47:45.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-05-15T12:47:45.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-05-15T12:47:45.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-05-15T12:47:45.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_discard_disconnected_ops": "true", 2024-05-15T12:47:45.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_ec_partial_reads": "true", 2024-05-15T12:47:45.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_enable_op_tracker": "true", 2024-05-15T12:47:45.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-05-15T12:47:45.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_extblkdev_plugins": "vdo", 2024-05-15T12:47:45.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-05-15T12:47:45.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-05-15T12:47:45.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_fast_info": "true", 2024-05-15T12:47:45.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_fast_shutdown": "true", 2024-05-15T12:47:45.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-05-15T12:47:45.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_fast_shutdown_timeout": "15", 2024-05-15T12:47:45.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-05-15T12:47:45.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-05-15T12:47:45.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-05-15T12:47:45.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_function_tracing": "false", 2024-05-15T12:47:45.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_heartbeat_grace": "20", 2024-05-15T12:47:45.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_heartbeat_interval": "6", 2024-05-15T12:47:45.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-05-15T12:47:45.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_heartbeat_min_peers": "10", 2024-05-15T12:47:45.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_heartbeat_min_size": "2000", 2024-05-15T12:47:45.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_heartbeat_stale": "600", 2024-05-15T12:47:45.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-05-15T12:47:45.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_hit_set_max_size": "100000", 2024-05-15T12:47:45.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_hit_set_min_size": "1000", 2024-05-15T12:47:45.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-05-15T12:47:45.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-05-15T12:47:45.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-05-15T12:47:45.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-05-15T12:47:45.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-05-15T12:47:45.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_journal_flush_on_shutdown": "true", 2024-05-15T12:47:45.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_journal_size": "5120", 2024-05-15T12:47:45.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_kill_backfill_at": "0", 2024-05-15T12:47:45.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_loop_before_reset_tphandle": "64", 2024-05-15T12:47:45.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_map_cache_size": "50", 2024-05-15T12:47:45.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_map_dedup": "true", 2024-05-15T12:47:45.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_map_message_max": "40", 2024-05-15T12:47:45.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_map_message_max_bytes": "10485760", 2024-05-15T12:47:45.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_map_share_max_epochs": "40", 2024-05-15T12:47:45.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_attr_name_len": "100", 2024-05-15T12:47:45.518 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_attr_size": "0", 2024-05-15T12:47:45.518 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_backfills": "1", 2024-05-15T12:47:45.518 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_markdown_count": "5", 2024-05-15T12:47:45.518 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_markdown_period": "600", 2024-05-15T12:47:45.518 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_object_name_len": "2048", 2024-05-15T12:47:45.518 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_object_namespace_len": "256", 2024-05-15T12:47:45.518 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_object_size": "134217728", 2024-05-15T12:47:45.518 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-05-15T12:47:45.518 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_omap_entries_per_request": "1024", 2024-05-15T12:47:45.518 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_pg_blocked_by": "16", 2024-05-15T12:47:45.518 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_pg_log_entries": "10000", 2024-05-15T12:47:45.518 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-05-15T12:47:45.519 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_pgls": "1024", 2024-05-15T12:47:45.519 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_push_cost": "8388608", 2024-05-15T12:47:45.519 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_push_objects": "10", 2024-05-15T12:47:45.519 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_scrubs": "3", 2024-05-15T12:47:45.519 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-05-15T12:47:45.519 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_trimming_pgs": "2", 2024-05-15T12:47:45.519 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_write_op_reply_len": "64", 2024-05-15T12:47:45.519 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_write_size": "90", 2024-05-15T12:47:45.519 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-05-15T12:47:45.519 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-05-15T12:47:45.519 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-05-15T12:47:45.519 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-05-15T12:47:45.519 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-05-15T12:47:45.520 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-05-15T12:47:45.520 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-05-15T12:47:45.520 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_override_recovery_settings": "false", 2024-05-15T12:47:45.520 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_profile": "balanced", 2024-05-15T12:47:45.520 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-05-15T12:47:45.520 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-05-15T12:47:45.520 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-05-15T12:47:45.520 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-05-15T12:47:45.520 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-05-15T12:47:45.520 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-05-15T12:47:45.520 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-05-15T12:47:45.520 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-05-15T12:47:45.520 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-05-15T12:47:45.520 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-05-15T12:47:45.520 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_skip_benchmark": "false", 2024-05-15T12:47:45.521 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_memory_base": "805306368", 2024-05-15T12:47:45.521 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_memory_cache_min": "134217728", 2024-05-15T12:47:45.521 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-05-15T12:47:45.521 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-05-15T12:47:45.521 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_memory_target": "4294967296", 2024-05-15T12:47:45.521 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_memory_target_autotune": "false", 2024-05-15T12:47:45.521 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-05-15T12:47:45.521 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_min_pg_log_entries": "250", 2024-05-15T12:47:45.521 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_min_recovery_priority": "0", 2024-05-15T12:47:45.521 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mon_heartbeat_interval": "30", 2024-05-15T12:47:45.521 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-05-15T12:47:45.521 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mon_report_interval": "5", 2024-05-15T12:47:45.522 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mon_report_max_in_flight": "2", 2024-05-15T12:47:45.522 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-05-15T12:47:45.522 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_num_cache_shards": "32", 2024-05-15T12:47:45.522 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_num_op_tracker_shard": "32", 2024-05-15T12:47:45.522 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_numa_auto_affinity": "true", 2024-05-15T12:47:45.522 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_numa_node": "-1", 2024-05-15T12:47:45.522 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_numa_prefer_iface": "true", 2024-05-15T12:47:45.523 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-05-15T12:47:45.523 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_objecter_finishers": "1", 2024-05-15T12:47:45.523 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_objectstore": "bluestore", 2024-05-15T12:47:45.523 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_objectstore_fuse": "false", 2024-05-15T12:47:45.523 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_objectstore_tracing": "false", 2024-05-15T12:47:45.523 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_complaint_time": "30.000000", 2024-05-15T12:47:45.523 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_history_duration": "600", 2024-05-15T12:47:45.523 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_history_size": "20", 2024-05-15T12:47:45.523 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_history_slow_op_size": "20", 2024-05-15T12:47:45.523 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-05-15T12:47:45.523 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_log_threshold": "5", 2024-05-15T12:47:45.523 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_num_shards": "0", 2024-05-15T12:47:45.524 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_num_shards_hdd": "5", 2024-05-15T12:47:45.524 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_num_shards_ssd": "8", 2024-05-15T12:47:45.524 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_num_threads_per_shard": "0", 2024-05-15T12:47:45.524 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-05-15T12:47:45.524 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-05-15T12:47:45.524 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-05-15T12:47:45.524 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_pq_min_cost": "65536", 2024-05-15T12:47:45.524 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_queue": "mclock_scheduler", 2024-05-15T12:47:45.524 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_queue_cut_off": "high", 2024-05-15T12:47:45.524 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_thread_suicide_timeout": "150", 2024-05-15T12:47:45.524 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_thread_timeout": "15", 2024-05-15T12:47:45.524 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_open_classes_on_start": "true", 2024-05-15T12:47:45.524 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_os_flags": "0", 2024-05-15T12:47:45.525 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_peering_op_priority": "255", 2024-05-15T12:47:45.525 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pg_delete_cost": "1048576", 2024-05-15T12:47:45.525 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pg_delete_priority": "5", 2024-05-15T12:47:45.525 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-05-15T12:47:45.525 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-05-15T12:47:45.525 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pg_log_dups_tracked": "3000", 2024-05-15T12:47:45.525 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pg_log_trim_max": "10000", 2024-05-15T12:47:45.525 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pg_log_trim_min": "100", 2024-05-15T12:47:45.525 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-05-15T12:47:45.525 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pg_object_context_cache_count": "64", 2024-05-15T12:47:45.525 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-05-15T12:47:45.525 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-05-15T12:47:45.525 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-05-15T12:47:45.525 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-05-15T12:47:45.525 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-05-15T12:47:45.526 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-05-15T12:47:45.526 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-05-15T12:47:45.526 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-05-15T12:47:45.526 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_crimson": "false", 2024-05-15T12:47:45.526 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_crush_rule": "-1", 2024-05-15T12:47:45.526 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_ec_fast_read": "false", 2024-05-15T12:47:45.526 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-05-15T12:47:45.526 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_flag_bulk": "false", 2024-05-15T12:47:45.526 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-05-15T12:47:45.526 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_flag_nodelete": "false", 2024-05-15T12:47:45.526 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-05-15T12:47:45.526 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-05-15T12:47:45.526 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_flags": "0", 2024-05-15T12:47:45.526 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-05-15T12:47:45.527 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_min_size": "2", 2024-05-15T12:47:45.527 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-05-15T12:47:45.527 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_pg_num": "32", 2024-05-15T12:47:45.527 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_pgp_num": "0", 2024-05-15T12:47:45.527 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-05-15T12:47:45.527 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_read_ratio": "70", 2024-05-15T12:47:45.527 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_size": "3", 2024-05-15T12:47:45.527 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_type": "replicated", 2024-05-15T12:47:45.527 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-05-15T12:47:45.527 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_use_gmt_hitset": "true", 2024-05-15T12:47:45.527 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_push_per_object_cost": "1000", 2024-05-15T12:47:45.527 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_read_ec_check_for_errors": "false", 2024-05-15T12:47:45.527 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recover_clone_overlap": "true", 2024-05-15T12:47:45.527 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recover_clone_overlap_limit": "10", 2024-05-15T12:47:45.528 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_cost": "20971520", 2024-05-15T12:47:45.528 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_delay_start": "0.000000", 2024-05-15T12:47:45.528 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_max_active": "0", 2024-05-15T12:47:45.528 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_max_active_hdd": "3", 2024-05-15T12:47:45.528 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_max_active_ssd": "10", 2024-05-15T12:47:45.528 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_max_chunk": "8388608", 2024-05-15T12:47:45.528 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-05-15T12:47:45.528 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_max_single_start": "1", 2024-05-15T12:47:45.528 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_op_priority": "3", 2024-05-15T12:47:45.528 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_op_warn_multiple": "16", 2024-05-15T12:47:45.528 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_priority": "5", 2024-05-15T12:47:45.528 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_retry_interval": "30.000000", 2024-05-15T12:47:45.528 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_sleep": "0.000000", 2024-05-15T12:47:45.528 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-05-15T12:47:45.529 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-05-15T12:47:45.529 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-05-15T12:47:45.529 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_repair_during_recovery": "false", 2024-05-15T12:47:45.529 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_requested_scrub_priority": "120", 2024-05-15T12:47:45.529 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-05-15T12:47:45.529 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_rollback_to_cluster_snap": "", 2024-05-15T12:47:45.529 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_auto_repair": "false", 2024-05-15T12:47:45.529 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-05-15T12:47:45.529 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-05-15T12:47:45.529 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_begin_hour": "0", 2024-05-15T12:47:45.529 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_begin_week_day": "0", 2024-05-15T12:47:45.529 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_chunk_max": "25", 2024-05-15T12:47:45.529 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_chunk_min": "5", 2024-05-15T12:47:45.529 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_cost": "52428800", 2024-05-15T12:47:45.530 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-05-15T12:47:45.530 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_during_recovery": "false", 2024-05-15T12:47:45.530 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_end_hour": "0", 2024-05-15T12:47:45.530 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_end_week_day": "0", 2024-05-15T12:47:45.530 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_event_cost": "4096", 2024-05-15T12:47:45.530 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-05-15T12:47:45.530 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-05-15T12:47:45.530 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_invalid_stats": "true", 2024-05-15T12:47:45.530 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_load_threshold": "0.500000", 2024-05-15T12:47:45.530 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_max_interval": "604800.000000", 2024-05-15T12:47:45.530 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_max_preemptions": "5", 2024-05-15T12:47:45.530 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_min_interval": "86400.000000", 2024-05-15T12:47:45.530 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_priority": "5", 2024-05-15T12:47:45.530 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_reservation_timeout": "300000", 2024-05-15T12:47:45.531 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_sleep": "0.000000", 2024-05-15T12:47:45.531 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-05-15T12:47:45.531 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-05-15T12:47:45.531 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-05-15T12:47:45.531 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_shutdown_pgref_assert": "false", 2024-05-15T12:47:45.531 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-05-15T12:47:45.531 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_skip_data_digest": "false", 2024-05-15T12:47:45.531 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_smart_report_timeout": "5", 2024-05-15T12:47:45.531 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_snap_trim_cost": "1048576", 2024-05-15T12:47:45.531 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_snap_trim_priority": "5", 2024-05-15T12:47:45.531 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_snap_trim_sleep": "0.000000", 2024-05-15T12:47:45.531 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-05-15T12:47:45.531 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-05-15T12:47:45.531 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-05-15T12:47:45.532 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-05-15T12:47:45.532 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_stats_update_period_scrubbing": "15", 2024-05-15T12:47:45.532 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-05-15T12:47:45.532 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_target_transaction_size": "30", 2024-05-15T12:47:45.532 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-05-15T12:47:45.532 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-05-15T12:47:45.532 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-05-15T12:47:45.532 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-05-15T12:47:45.532 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-05-15T12:47:45.532 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-05-15T12:47:45.532 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-05-15T12:47:45.532 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_tier_default_cache_mode": "writeback", 2024-05-15T12:47:45.532 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-05-15T12:47:45.533 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-05-15T12:47:45.533 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_tracing": "false", 2024-05-15T12:47:45.533 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_use_stale_snap": "false", 2024-05-15T12:47:45.533 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-05-15T12:47:45.533 INFO:teuthology.orchestra.run.smithi027.stdout: "osdc_blkin_trace_all": "false", 2024-05-15T12:47:45.533 INFO:teuthology.orchestra.run.smithi027.stdout: "paxos_kill_at": "0", 2024-05-15T12:47:45.533 INFO:teuthology.orchestra.run.smithi027.stdout: "paxos_max_join_drift": "10", 2024-05-15T12:47:45.533 INFO:teuthology.orchestra.run.smithi027.stdout: "paxos_min": "500", 2024-05-15T12:47:45.533 INFO:teuthology.orchestra.run.smithi027.stdout: "paxos_min_wait": "0.050000", 2024-05-15T12:47:45.533 INFO:teuthology.orchestra.run.smithi027.stdout: "paxos_propose_interval": "1.000000", 2024-05-15T12:47:45.533 INFO:teuthology.orchestra.run.smithi027.stdout: "paxos_service_trim_max": "500", 2024-05-15T12:47:45.533 INFO:teuthology.orchestra.run.smithi027.stdout: "paxos_service_trim_max_multiplier": "20", 2024-05-15T12:47:45.533 INFO:teuthology.orchestra.run.smithi027.stdout: "paxos_service_trim_min": "10", 2024-05-15T12:47:45.534 INFO:teuthology.orchestra.run.smithi027.stdout: "paxos_stash_full_interval": "25", 2024-05-15T12:47:45.534 INFO:teuthology.orchestra.run.smithi027.stdout: "paxos_trim_max": "500", 2024-05-15T12:47:45.534 INFO:teuthology.orchestra.run.smithi027.stdout: "paxos_trim_min": "250", 2024-05-15T12:47:45.534 INFO:teuthology.orchestra.run.smithi027.stdout: "perf": "true", 2024-05-15T12:47:45.534 INFO:teuthology.orchestra.run.smithi027.stdout: "pid_file": "", 2024-05-15T12:47:45.534 INFO:teuthology.orchestra.run.smithi027.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-05-15T12:47:45.534 INFO:teuthology.orchestra.run.smithi027.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-05-15T12:47:45.534 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "-", 2024-05-15T12:47:45.534 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrv": "", 2024-05-15T12:47:45.534 INFO:teuthology.orchestra.run.smithi027.stdout: "public_bind_addr": "-", 2024-05-15T12:47:45.534 INFO:teuthology.orchestra.run.smithi027.stdout: "public_network": "", 2024-05-15T12:47:45.534 INFO:teuthology.orchestra.run.smithi027.stdout: "public_network_interface": "", 2024-05-15T12:47:45.534 INFO:teuthology.orchestra.run.smithi027.stdout: "qat_compressor_enabled": "false", 2024-05-15T12:47:45.535 INFO:teuthology.orchestra.run.smithi027.stdout: "qat_compressor_session_max_number": "256", 2024-05-15T12:47:45.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rados_mon_op_timeout": "0", 2024-05-15T12:47:45.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rados_osd_op_timeout": "0", 2024-05-15T12:47:45.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rados_tracing": "false", 2024-05-15T12:47:45.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_atime_update_interval": "60", 2024-05-15T12:47:45.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-05-15T12:47:45.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_balance_parent_reads": "false", 2024-05-15T12:47:45.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_balance_snap_reads": "false", 2024-05-15T12:47:45.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_blkin_trace_all": "false", 2024-05-15T12:47:45.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_blocklist_expire_seconds": "0", 2024-05-15T12:47:45.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_blocklist_on_break_lock": "true", 2024-05-15T12:47:45.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_cache": "true", 2024-05-15T12:47:45.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_cache_block_writes_upfront": "false", 2024-05-15T12:47:45.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_cache_max_dirty": "25165824", 2024-05-15T12:47:45.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-05-15T12:47:45.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_cache_max_dirty_object": "0", 2024-05-15T12:47:45.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_cache_policy": "writearound", 2024-05-15T12:47:45.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_cache_size": "33554432", 2024-05-15T12:47:45.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_cache_target_dirty": "16777216", 2024-05-15T12:47:45.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-05-15T12:47:45.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_clone_copy_on_read": "false", 2024-05-15T12:47:45.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_compression_hint": "none", 2024-05-15T12:47:45.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_concurrent_management_ops": "10", 2024-05-15T12:47:45.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-05-15T12:47:45.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_default_clone_format": "auto", 2024-05-15T12:47:45.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_default_data_pool": "", 2024-05-15T12:47:45.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_default_features": "61", 2024-05-15T12:47:45.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_default_format": "2", 2024-05-15T12:47:45.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_default_map_options": "", 2024-05-15T12:47:45.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_default_order": "22", 2024-05-15T12:47:45.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_default_pool": "rbd", 2024-05-15T12:47:45.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-05-15T12:47:45.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_default_stripe_count": "0", 2024-05-15T12:47:45.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_default_stripe_unit": "0", 2024-05-15T12:47:45.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_disable_zero_copy_writes": "true", 2024-05-15T12:47:45.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_discard_granularity_bytes": "65536", 2024-05-15T12:47:45.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-05-15T12:47:45.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_enable_alloc_hint": "true", 2024-05-15T12:47:45.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-05-15T12:47:45.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_io_scheduler": "simple", 2024-05-15T12:47:45.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-05-15T12:47:45.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_journal_commit_age": "5.000000", 2024-05-15T12:47:45.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-05-15T12:47:45.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-05-15T12:47:45.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-05-15T12:47:45.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-05-15T12:47:45.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_journal_object_flush_interval": "0", 2024-05-15T12:47:45.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-05-15T12:47:45.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-05-15T12:47:45.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_journal_order": "24", 2024-05-15T12:47:45.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_journal_pool": "", 2024-05-15T12:47:45.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_journal_splay_width": "4", 2024-05-15T12:47:45.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_localize_parent_reads": "false", 2024-05-15T12:47:45.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_localize_snap_reads": "false", 2024-05-15T12:47:45.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-05-15T12:47:45.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-05-15T12:47:45.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-05-15T12:47:45.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-05-15T12:47:45.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-05-15T12:47:45.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-05-15T12:47:45.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_image_policy_type": "simple", 2024-05-15T12:47:45.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-05-15T12:47:45.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-05-15T12:47:45.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-05-15T12:47:45.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-05-15T12:47:45.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-05-15T12:47:45.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-05-15T12:47:45.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-05-15T12:47:45.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_memory_autotune": "true", 2024-05-15T12:47:45.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_memory_base": "805306368", 2024-05-15T12:47:45.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-05-15T12:47:45.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-05-15T12:47:45.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-05-15T12:47:45.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-05-15T12:47:45.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_memory_target": "4294967296", 2024-05-15T12:47:45.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-05-15T12:47:45.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-05-15T12:47:45.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-05-15T12:47:45.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirroring_delete_delay": "0", 2024-05-15T12:47:45.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-05-15T12:47:45.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirroring_replay_delay": "0", 2024-05-15T12:47:45.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-05-15T12:47:45.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-05-15T12:47:45.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_move_to_trash_on_remove": "false", 2024-05-15T12:47:45.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-05-15T12:47:45.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mtime_update_interval": "60", 2024-05-15T12:47:45.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_non_blocking_aio": "true", 2024-05-15T12:47:45.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_op_thread_timeout": "60", 2024-05-15T12:47:45.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_op_threads": "1", 2024-05-15T12:47:45.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_parent_cache_enabled": "false", 2024-05-15T12:47:45.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_persistent_cache_mode": "disabled", 2024-05-15T12:47:45.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_persistent_cache_path": "/tmp", 2024-05-15T12:47:45.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_persistent_cache_size": "1073741824", 2024-05-15T12:47:45.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_plugins": "", 2024-05-15T12:47:45.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_bps_burst": "0", 2024-05-15T12:47:45.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-05-15T12:47:45.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_bps_limit": "0", 2024-05-15T12:47:45.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_exclude_ops": "0", 2024-05-15T12:47:45.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_iops_burst": "0", 2024-05-15T12:47:45.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-05-15T12:47:45.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_iops_limit": "0", 2024-05-15T12:47:45.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_read_bps_burst": "0", 2024-05-15T12:47:45.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-05-15T12:47:45.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_read_bps_limit": "0", 2024-05-15T12:47:45.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_read_iops_burst": "0", 2024-05-15T12:47:45.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-05-15T12:47:45.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_read_iops_limit": "0", 2024-05-15T12:47:45.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_schedule_tick_min": "50", 2024-05-15T12:47:45.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_write_bps_burst": "0", 2024-05-15T12:47:45.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-05-15T12:47:45.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_write_bps_limit": "0", 2024-05-15T12:47:45.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_write_iops_burst": "0", 2024-05-15T12:47:45.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-05-15T12:47:45.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_write_iops_limit": "0", 2024-05-15T12:47:45.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_quiesce_notification_attempts": "10", 2024-05-15T12:47:45.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_read_from_replica_policy": "default", 2024-05-15T12:47:45.543 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-05-15T12:47:45.543 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_readahead_max_bytes": "524288", 2024-05-15T12:47:45.543 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_readahead_trigger_requests": "10", 2024-05-15T12:47:45.543 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_request_timed_out_seconds": "30", 2024-05-15T12:47:45.543 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_skip_partial_discard": "true", 2024-05-15T12:47:45.543 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-05-15T12:47:45.543 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_tracing": "false", 2024-05-15T12:47:45.543 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_validate_names": "true", 2024-05-15T12:47:45.543 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_validate_pool": "true", 2024-05-15T12:47:45.544 INFO:teuthology.orchestra.run.smithi027.stdout: "restapi_base_url": "", 2024-05-15T12:47:45.544 INFO:teuthology.orchestra.run.smithi027.stdout: "restapi_log_level": "", 2024-05-15T12:47:45.544 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_acl_grants_max_num": "100", 2024-05-15T12:47:45.544 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_admin_entry": "admin", 2024-05-15T12:47:45.544 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-05-15T12:47:45.544 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_backend_store": "rados", 2024-05-15T12:47:45.544 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_barbican_url": "", 2024-05-15T12:47:45.544 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_beast_enable_async": "true", 2024-05-15T12:47:45.544 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_bucket_counters_cache": "false", 2024-05-15T12:47:45.544 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-05-15T12:47:45.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-05-15T12:47:45.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-05-15T12:47:45.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_bucket_index_max_aio": "128", 2024-05-15T12:47:45.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-05-15T12:47:45.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-05-15T12:47:45.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_bucket_quota_ttl": "600", 2024-05-15T12:47:45.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-05-15T12:47:45.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_cache_enabled": "true", 2024-05-15T12:47:45.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_cache_expiry_interval": "900", 2024-05-15T12:47:45.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_cache_lru_size": "10000", 2024-05-15T12:47:45.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_config_store": "rados", 2024-05-15T12:47:45.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_content_length_compat": "false", 2024-05-15T12:47:45.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_copy_obj_progress": "true", 2024-05-15T12:47:45.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-05-15T12:47:45.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_cors_rules_max_num": "100", 2024-05-15T12:47:45.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_cross_domain_policy": "", 2024-05-15T12:47:45.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_default_encryption_key": "", 2024-05-15T12:47:45.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_kmip_addr": "", 2024-05-15T12:47:45.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_kmip_ca_path": "", 2024-05-15T12:47:45.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_kmip_client_cert": "", 2024-05-15T12:47:45.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_kmip_client_key": "", 2024-05-15T12:47:45.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-05-15T12:47:45.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_kmip_password": "", 2024-05-15T12:47:45.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-05-15T12:47:45.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_kmip_username": "", 2024-05-15T12:47:45.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_require_ssl": "true", 2024-05-15T12:47:45.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-05-15T12:47:45.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-05-15T12:47:45.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-05-15T12:47:45.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-05-15T12:47:45.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-05-15T12:47:45.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-05-15T12:47:45.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-05-15T12:47:45.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-05-15T12:47:45.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-05-15T12:47:45.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-05-15T12:47:45.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-05-15T12:47:45.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-05-15T12:47:45.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-05-15T12:47:45.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-05-15T12:47:45.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_suppress_logs": "true", 2024-05-15T12:47:45.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_vault_addr": "", 2024-05-15T12:47:45.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_vault_auth": "token", 2024-05-15T12:47:45.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_vault_namespace": "", 2024-05-15T12:47:45.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_vault_prefix": "", 2024-05-15T12:47:45.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-05-15T12:47:45.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-05-15T12:47:45.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-05-15T12:47:45.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-05-15T12:47:45.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_vault_token_file": "", 2024-05-15T12:47:45.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-05-15T12:47:45.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_curl_buffersize": "524288", 2024-05-15T12:47:45.549 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_curl_low_speed_limit": "1024", 2024-05-15T12:47:45.549 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_curl_low_speed_time": "300", 2024-05-15T12:47:45.549 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_curl_tcp_keepalive": "0", 2024-05-15T12:47:45.549 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-05-15T12:47:45.549 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-05-15T12:47:45.549 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-05-15T12:47:45.549 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-05-15T12:47:45.549 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-05-15T12:47:45.549 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d3n_l1_fadvise": "4", 2024-05-15T12:47:45.549 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-05-15T12:47:45.549 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-05-15T12:47:45.549 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-05-15T12:47:45.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d4n_host": "127.0.0.1", 2024-05-15T12:47:45.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d4n_port": "6379", 2024-05-15T12:47:45.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-05-15T12:47:45.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_data_log_changes_size": "1000", 2024-05-15T12:47:45.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_data_log_num_shards": "128", 2024-05-15T12:47:45.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-05-15T12:47:45.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_data_log_window": "30", 2024-05-15T12:47:45.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_data_notify_interval_msec": "0", 2024-05-15T12:47:45.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_data_sync_poll_interval": "20", 2024-05-15T12:47:45.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_data_sync_spawn_window": "20", 2024-05-15T12:47:45.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-05-15T12:47:45.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-05-15T12:47:45.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-05-15T12:47:45.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_default_data_log_backing": "fifo", 2024-05-15T12:47:45.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-05-15T12:47:45.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_default_region_info_oid": "default.region", 2024-05-15T12:47:45.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-05-15T12:47:45.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-05-15T12:47:45.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_defer_to_bucket_acls": "", 2024-05-15T12:47:45.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-05-15T12:47:45.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-05-15T12:47:45.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-05-15T12:47:45.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-05-15T12:47:45.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-05-15T12:47:45.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-05-15T12:47:45.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-05-15T12:47:45.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-05-15T12:47:45.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dmclock_data_res": "500.000000", 2024-05-15T12:47:45.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-05-15T12:47:45.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-05-15T12:47:45.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-05-15T12:47:45.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-05-15T12:47:45.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dns_name": "", 2024-05-15T12:47:45.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dns_s3website_name": "", 2024-05-15T12:47:45.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dynamic_resharding": "true", 2024-05-15T12:47:45.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-05-15T12:47:45.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_enable_gc_threads": "true", 2024-05-15T12:47:45.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_enable_lc_threads": "true", 2024-05-15T12:47:45.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_enable_ops_log": "false", 2024-05-15T12:47:45.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_enable_quota_threads": "true", 2024-05-15T12:47:45.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_enable_static_website": "false", 2024-05-15T12:47:45.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_enable_usage_log": "false", 2024-05-15T12:47:45.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_enforce_swift_acls": "true", 2024-05-15T12:47:45.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_exit_timeout_secs": "120", 2024-05-15T12:47:45.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_expose_bucket": "false", 2024-05-15T12:47:45.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_extended_http_attrs": "", 2024-05-15T12:47:45.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_filter": "none", 2024-05-15T12:47:45.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-05-15T12:47:45.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_frontends": "beast port=7480", 2024-05-15T12:47:45.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_gc_max_concurrent_io": "10", 2024-05-15T12:47:45.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_gc_max_deferred": "50", 2024-05-15T12:47:45.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-05-15T12:47:45.554 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_gc_max_objs": "32", 2024-05-15T12:47:45.554 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_gc_max_queue_size": "134213632", 2024-05-15T12:47:45.554 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_gc_max_trim_chunk": "16", 2024-05-15T12:47:45.554 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_gc_obj_min_wait": "7200", 2024-05-15T12:47:45.554 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_gc_processor_max_time": "3600", 2024-05-15T12:47:45.554 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_gc_processor_period": "3600", 2024-05-15T12:47:45.554 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-05-15T12:47:45.554 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_get_obj_window_size": "16777216", 2024-05-15T12:47:45.554 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_healthcheck_disabling_path": "", 2024-05-15T12:47:45.554 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_ignore_get_invalid_range": "false", 2024-05-15T12:47:45.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_init_timeout": "300", 2024-05-15T12:47:45.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-05-15T12:47:45.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-05-15T12:47:45.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_kafka_connection_idle": "30", 2024-05-15T12:47:45.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_kafka_sleep_timeout": "10", 2024-05-15T12:47:45.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-05-15T12:47:45.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-05-15T12:47:45.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-05-15T12:47:45.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_admin_domain": "", 2024-05-15T12:47:45.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_admin_password": "", 2024-05-15T12:47:45.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_admin_password_path": "", 2024-05-15T12:47:45.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_admin_project": "", 2024-05-15T12:47:45.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_admin_tenant": "", 2024-05-15T12:47:45.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_admin_token": "", 2024-05-15T12:47:45.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_admin_token_path": "", 2024-05-15T12:47:45.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_admin_user": "", 2024-05-15T12:47:45.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_api_version": "2", 2024-05-15T12:47:45.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_barbican_domain": "", 2024-05-15T12:47:45.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_barbican_password": "", 2024-05-15T12:47:45.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_barbican_project": "", 2024-05-15T12:47:45.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_barbican_tenant": "", 2024-05-15T12:47:45.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_barbican_user": "", 2024-05-15T12:47:45.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-05-15T12:47:45.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_implicit_tenants": "false", 2024-05-15T12:47:45.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-05-15T12:47:45.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_service_token_enabled": "false", 2024-05-15T12:47:45.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_token_cache_size": "10000", 2024-05-15T12:47:45.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_url": "", 2024-05-15T12:47:45.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_verify_ssl": "true", 2024-05-15T12:47:45.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_lc_debug_interval": "-1", 2024-05-15T12:47:45.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_lc_lock_max_time": "90", 2024-05-15T12:47:45.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_lc_max_objs": "32", 2024-05-15T12:47:45.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_lc_max_rules": "1000", 2024-05-15T12:47:45.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_lc_max_worker": "3", 2024-05-15T12:47:45.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_lc_max_wp_worker": "3", 2024-05-15T12:47:45.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_lc_thread_delay": "0", 2024-05-15T12:47:45.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-05-15T12:47:45.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_ldap_dnattr": "uid", 2024-05-15T12:47:45.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-05-15T12:47:45.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_ldap_searchfilter": "", 2024-05-15T12:47:45.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-05-15T12:47:45.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_ldap_uri": "ldaps://", 2024-05-15T12:47:45.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-05-15T12:47:45.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-05-15T12:47:45.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-05-15T12:47:45.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_log_http_headers": "", 2024-05-15T12:47:45.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_log_nonexistent_bucket": "false", 2024-05-15T12:47:45.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-05-15T12:47:45.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_log_object_name_utc": "false", 2024-05-15T12:47:45.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-05-15T12:47:45.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-05-15T12:47:45.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_attr_name_len": "0", 2024-05-15T12:47:45.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_attr_size": "0", 2024-05-15T12:47:45.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_attrs_num_in_req": "0", 2024-05-15T12:47:45.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_chunk_size": "4194304", 2024-05-15T12:47:45.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_concurrent_requests": "1024", 2024-05-15T12:47:45.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-05-15T12:47:45.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_dynamic_shards": "1999", 2024-05-15T12:47:45.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_listing_results": "1000", 2024-05-15T12:47:45.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_notify_retries": "10", 2024-05-15T12:47:45.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_objs_per_shard": "100000", 2024-05-15T12:47:45.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_put_param_size": "1048576", 2024-05-15T12:47:45.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_put_size": "5368709120", 2024-05-15T12:47:45.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_slo_entries": "1000", 2024-05-15T12:47:45.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_md_log_max_shards": "64", 2024-05-15T12:47:45.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_md_notify_interval_msec": "200", 2024-05-15T12:47:45.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_meta_sync_poll_interval": "20", 2024-05-15T12:47:45.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_meta_sync_spawn_window": "20", 2024-05-15T12:47:45.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-05-15T12:47:45.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_mp_lock_max_time": "600", 2024-05-15T12:47:45.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-05-15T12:47:45.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_multipart_min_part_size": "5242880", 2024-05-15T12:47:45.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-05-15T12:47:45.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-05-15T12:47:45.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_fhcache_size": "2017", 2024-05-15T12:47:45.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-05-15T12:47:45.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-05-15T12:47:45.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_lru_lanes": "5", 2024-05-15T12:47:45.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_max_gc": "300", 2024-05-15T12:47:45.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-05-15T12:47:45.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_run_gc_threads": "false", 2024-05-15T12:47:45.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_run_lc_threads": "false", 2024-05-15T12:47:45.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_run_quota_threads": "false", 2024-05-15T12:47:45.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_run_sync_thread": "false", 2024-05-15T12:47:45.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-05-15T12:47:45.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-05-15T12:47:45.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_num_async_rados_threads": "32", 2024-05-15T12:47:45.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_num_control_oids": "8", 2024-05-15T12:47:45.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_numa_node": "-1", 2024-05-15T12:47:45.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_obj_stripe_size": "4194304", 2024-05-15T12:47:45.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-05-15T12:47:45.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_objexp_chunk_size": "100", 2024-05-15T12:47:45.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_objexp_gc_interval": "600", 2024-05-15T12:47:45.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_objexp_hints_num_shards": "127", 2024-05-15T12:47:45.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-05-15T12:47:45.562 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-05-15T12:47:45.562 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_op_thread_timeout": "600", 2024-05-15T12:47:45.562 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_op_tracing": "false", 2024-05-15T12:47:45.562 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_opa_token": "", 2024-05-15T12:47:45.562 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_opa_url": "", 2024-05-15T12:47:45.562 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_opa_verify_ssl": "true", 2024-05-15T12:47:45.562 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-05-15T12:47:45.562 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-05-15T12:47:45.562 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_ops_log_rados": "false", 2024-05-15T12:47:45.562 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_ops_log_socket_path": "", 2024-05-15T12:47:45.562 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-05-15T12:47:45.563 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-05-15T12:47:45.563 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-05-15T12:47:45.563 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_period_push_interval": "2.000000", 2024-05-15T12:47:45.563 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_period_push_interval_max": "30.000000", 2024-05-15T12:47:45.563 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_period_root_pool": ".rgw.root", 2024-05-15T12:47:45.563 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-05-15T12:47:45.563 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-05-15T12:47:45.563 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_posix_cache_lanes": "3", 2024-05-15T12:47:45.563 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-05-15T12:47:45.563 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_posix_cache_max_buckets": "100", 2024-05-15T12:47:45.563 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_posix_cache_partitions": "3", 2024-05-15T12:47:45.563 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-05-15T12:47:45.563 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_print_continue": "true", 2024-05-15T12:47:45.564 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_print_prohibited_content_length": "false", 2024-05-15T12:47:45.564 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-05-15T12:47:45.564 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-05-15T12:47:45.564 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-05-15T12:47:45.564 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-05-15T12:47:45.564 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_rados_tracing": "false", 2024-05-15T12:47:45.564 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_realm": "", 2024-05-15T12:47:45.564 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_realm_id": "", 2024-05-15T12:47:45.564 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-05-15T12:47:45.564 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_region": "", 2024-05-15T12:47:45.565 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_region_root_pool": ".rgw.root", 2024-05-15T12:47:45.565 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_relaxed_region_enforcement": "false", 2024-05-15T12:47:45.565 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-05-15T12:47:45.565 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-05-15T12:47:45.565 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_request_uri": "", 2024-05-15T12:47:45.565 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_reshard_batch_size": "64", 2024-05-15T12:47:45.565 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-05-15T12:47:45.565 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_reshard_max_aio": "128", 2024-05-15T12:47:45.565 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_reshard_num_logs": "16", 2024-05-15T12:47:45.565 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_reshard_thread_interval": "600", 2024-05-15T12:47:45.565 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_resolve_cname": "false", 2024-05-15T12:47:45.566 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_rest_getusage_op_compat": "false", 2024-05-15T12:47:45.566 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_run_sync_thread": "true", 2024-05-15T12:47:45.566 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-05-15T12:47:45.566 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_s3_auth_use_keystone": "false", 2024-05-15T12:47:45.566 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_s3_auth_use_ldap": "false", 2024-05-15T12:47:45.566 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_s3_auth_use_rados": "true", 2024-05-15T12:47:45.566 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_s3_auth_use_sts": "false", 2024-05-15T12:47:45.566 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-05-15T12:47:45.566 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_s3_success_create_obj_status": "0", 2024-05-15T12:47:45.566 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-05-15T12:47:45.566 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_scheduler_type": "throttler", 2024-05-15T12:47:45.566 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_script_uri": "", 2024-05-15T12:47:45.566 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_service_provider_name": "", 2024-05-15T12:47:45.567 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-05-15T12:47:45.567 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sts_client_id": "", 2024-05-15T12:47:45.567 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sts_client_secret": "", 2024-05-15T12:47:45.567 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sts_entry": "sts", 2024-05-15T12:47:45.567 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sts_key": "sts", 2024-05-15T12:47:45.567 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sts_max_session_duration": "43200", 2024-05-15T12:47:45.567 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sts_min_session_duration": "900", 2024-05-15T12:47:45.567 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sts_token_introspection_url": "", 2024-05-15T12:47:45.567 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_swift_account_in_url": "false", 2024-05-15T12:47:45.567 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_swift_auth_entry": "auth", 2024-05-15T12:47:45.567 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_swift_auth_url": "", 2024-05-15T12:47:45.567 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_swift_custom_header": "", 2024-05-15T12:47:45.567 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_swift_enforce_content_length": "false", 2024-05-15T12:47:45.567 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_swift_need_stats": "true", 2024-05-15T12:47:45.567 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_swift_tenant_name": "", 2024-05-15T12:47:45.568 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_swift_token_expiration": "86400", 2024-05-15T12:47:45.568 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_swift_url": "", 2024-05-15T12:47:45.568 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_swift_url_prefix": "swift", 2024-05-15T12:47:45.568 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_swift_versioning_enabled": "false", 2024-05-15T12:47:45.568 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-05-15T12:47:45.568 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-05-15T12:47:45.568 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sync_lease_period": "120", 2024-05-15T12:47:45.568 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-05-15T12:47:45.568 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sync_log_trim_interval": "1200", 2024-05-15T12:47:45.568 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-05-15T12:47:45.568 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-05-15T12:47:45.568 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-05-15T12:47:45.568 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sync_obj_etag_verify": "false", 2024-05-15T12:47:45.568 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sync_trace_history_size": "4096", 2024-05-15T12:47:45.568 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-05-15T12:47:45.569 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-05-15T12:47:45.569 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_thread_pool_size": "512", 2024-05-15T12:47:45.569 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_topic_persistency_max_retries": "0", 2024-05-15T12:47:45.569 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-05-15T12:47:45.569 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-05-15T12:47:45.569 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_topic_require_publish_policy": "false", 2024-05-15T12:47:45.569 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_torrent_comment": "", 2024-05-15T12:47:45.569 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_torrent_createby": "", 2024-05-15T12:47:45.569 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_torrent_encoding": "", 2024-05-15T12:47:45.569 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_torrent_flag": "false", 2024-05-15T12:47:45.569 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_torrent_max_size": "5368709120", 2024-05-15T12:47:45.569 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_torrent_origin": "", 2024-05-15T12:47:45.569 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_torrent_sha_unit": "524288", 2024-05-15T12:47:45.569 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_torrent_tracker": "", 2024-05-15T12:47:45.569 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_trust_forwarded_https": "false", 2024-05-15T12:47:45.569 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-05-15T12:47:45.570 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_usage_log_tick_interval": "30", 2024-05-15T12:47:45.570 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_usage_max_shards": "32", 2024-05-15T12:47:45.570 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_usage_max_user_shards": "1", 2024-05-15T12:47:45.570 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_use_opa_authz": "false", 2024-05-15T12:47:45.570 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_user_counters_cache": "false", 2024-05-15T12:47:45.570 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_user_counters_cache_size": "10000", 2024-05-15T12:47:45.570 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-05-15T12:47:45.570 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_user_default_quota_max_size": "-1", 2024-05-15T12:47:45.570 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_user_max_buckets": "1000", 2024-05-15T12:47:45.570 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_user_policies_max_num": "100", 2024-05-15T12:47:45.570 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-05-15T12:47:45.570 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-05-15T12:47:45.570 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_user_quota_sync_interval": "86400", 2024-05-15T12:47:45.570 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-05-15T12:47:45.571 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_user_unique_email": "true", 2024-05-15T12:47:45.571 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_verify_ssl": "true", 2024-05-15T12:47:45.571 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_website_routing_rules_max_num": "50", 2024-05-15T12:47:45.571 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_zone": "", 2024-05-15T12:47:45.571 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_zone_id": "", 2024-05-15T12:47:45.571 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-05-15T12:47:45.571 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_zonegroup": "", 2024-05-15T12:47:45.571 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_zonegroup_id": "", 2024-05-15T12:47:45.571 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-05-15T12:47:45.571 INFO:teuthology.orchestra.run.smithi027.stdout: "rgwlc_auto_session_clear": "true", 2024-05-15T12:47:45.571 INFO:teuthology.orchestra.run.smithi027.stdout: "rgwlc_skip_bucket_step": "false", 2024-05-15T12:47:45.571 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_block_size": "4096", 2024-05-15T12:47:45.571 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-05-15T12:47:45.572 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-05-15T12:47:45.572 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-05-15T12:47:45.572 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-05-15T12:47:45.572 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_cache_shard_bits": "4", 2024-05-15T12:47:45.572 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_cache_size": "536870912", 2024-05-15T12:47:45.572 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_cache_type": "binned_lru", 2024-05-15T12:47:45.572 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-05-15T12:47:45.572 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-05-15T12:47:45.572 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-05-15T12:47:45.572 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_collect_compaction_stats": "false", 2024-05-15T12:47:45.572 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_collect_extended_stats": "false", 2024-05-15T12:47:45.572 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_collect_memory_stats": "false", 2024-05-15T12:47:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-05-15T12:47:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_index_type": "binary_search", 2024-05-15T12:47:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_log_to_ceph_log": "true", 2024-05-15T12:47:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_metadata_block_size": "4096", 2024-05-15T12:47:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_partition_filters": "false", 2024-05-15T12:47:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_perf": "false", 2024-05-15T12:47:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-05-15T12:47:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-05-15T12:47:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "rotating_keys_renewal_timeout": "10", 2024-05-15T12:47:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "run_dir": "/var/run/ceph", 2024-05-15T12:47:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_block_create": "true", 2024-05-15T12:47:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_cache_lru_size": "67108864", 2024-05-15T12:47:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_cbjournal_size": "5368709120", 2024-05-15T12:47:45.574 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_data_delta_based_overwrite": "0", 2024-05-15T12:47:45.574 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_default_max_object_size": "16777216", 2024-05-15T12:47:45.574 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-05-15T12:47:45.574 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_device_size": "53687091200", 2024-05-15T12:47:45.574 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_journal_batch_capacity": "16", 2024-05-15T12:47:45.574 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-05-15T12:47:45.574 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-05-15T12:47:45.574 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_journal_iodepth_limit": "5", 2024-05-15T12:47:45.574 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_main_device_type": "SSD", 2024-05-15T12:47:45.574 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_max_concurrent_transactions": "8", 2024-05-15T12:47:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-05-15T12:47:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-05-15T12:47:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-05-15T12:47:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-05-15T12:47:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_segment_size": "67108864", 2024-05-15T12:47:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: "set_keepcaps": "false", 2024-05-15T12:47:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: "setgroup": "ceph", 2024-05-15T12:47:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: "setuser": "ceph", 2024-05-15T12:47:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: "setuser_match_path": "", 2024-05-15T12:47:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_misplaced_ratio": "0.050000", 2024-05-15T12:47:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: "thp": "false", 2024-05-15T12:47:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: "threadpool_default_timeout": "60", 2024-05-15T12:47:45.576 INFO:teuthology.orchestra.run.smithi027.stdout: "threadpool_empty_queue_max_wait": "2", 2024-05-15T12:47:45.576 INFO:teuthology.orchestra.run.smithi027.stdout: "throttler_perf_counter": "true" 2024-05-15T12:47:45.576 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T12:47:45.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:45 smithi177 ceph-mon[44303]: pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:45.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:45 smithi027 ceph-mon[39795]: pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:45.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:45 smithi120 ceph-mon[43961]: pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:46.363 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-05-15T12:47:46.641 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:47:47.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:47 smithi027 ceph-mon[39795]: pgmap v352: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:47.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:47 smithi177 ceph-mon[44303]: pgmap v352: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:47.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:47 smithi120 ceph-mon[43961]: pgmap v352: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:48.604 INFO:teuthology.orchestra.run.smithi027.stdout:mon.a: {} 2024-05-15T12:47:48.614 INFO:teuthology.orchestra.run.smithi027.stdout:mon.c: {} 2024-05-15T12:47:48.625 INFO:teuthology.orchestra.run.smithi027.stdout:mon.b: {} 2024-05-15T12:47:48.626 INFO:teuthology.orchestra.run.smithi027.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-05-15T12:47:48.626 INFO:teuthology.orchestra.run.smithi027.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-05-15T12:47:48.627 INFO:teuthology.orchestra.run.smithi027.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-05-15T12:47:49.391 INFO:teuthology.run_tasks:Running task full_sequential... 2024-05-15T12:47:49.399 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-05-15T12:47:49.401 INFO:tasks.radosbench:Beginning radosbench... 2024-05-15T12:47:49.401 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-05-15T12:47:49.401 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-05-15T12:47:49.402 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-05-15T12:47:49.402 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T12:47:49.402 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T12:47:49.402 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-05-15T12:47:49.402 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd dump -f json-pretty 2024-05-15T12:47:49.403 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-05-15T12:47:49.403 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:47:49.403 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-05-15T12:47:49.403 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:47:49.404 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-05-15T12:47:49.404 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-05-15T12:47:49.404 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-05-15T12:47:49.404 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T12:47:49.450 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:49.486 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:47:49.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:49.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:47:49.592 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:47:49.622 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:49 smithi120 ceph-mon[43961]: pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:49.622 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:49 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1786374245' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T12:47:49.625 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:49.647 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:47:49.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:49.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:47:49.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:49 smithi177 ceph-mon[44303]: pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:49.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:49 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1786374245' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T12:47:49.729 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:47:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:47:49.751 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:47:49.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:49 smithi027 ceph-mon[39795]: pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:49.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:49 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1786374245' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T12:47:49.784 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:47:49.785 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:49.807 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:47:49.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:49.810 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:47:49.910 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:47:49.939 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:50.045 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:47:50.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:50.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:47:50.149 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:47:50.182 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:50.203 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:47:50.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:50.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:47:50.307 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:47:50.336 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:50.389 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:47:50.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:50.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:47:50.494 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:47:50.526 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:50.547 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:47:50.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:50.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:47:50.651 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:47:50.681 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:50.702 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:47:50.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:50.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:47:50.806 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:47:50.838 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:50.859 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:47:50.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:50.862 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:47:50.963 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:47:50.993 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:51.013 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:47:51.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:51.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:47:51.116 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:47:51.145 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:51.164 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:47:51.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:51.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:47:51.268 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:47:51.294 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:51.313 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:47:51.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:51.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:47:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:47:51.417 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:47:51.453 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:51.664 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:51 smithi027 ceph-mon[39795]: pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:51.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:51 smithi177 ceph-mon[44303]: pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:51.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:51 smithi120 ceph-mon[43961]: pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:52.306 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:47:52.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:52.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:47:52.410 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:47:52.443 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:52.464 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:47:52.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:52.467 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:47:52.568 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:47:52.599 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:52.619 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:47:52.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:52.623 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:47:52.723 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:47:52.751 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:52.771 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:47:52.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:52.774 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:47:52.874 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:47:52.907 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:52.928 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:47:52.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:52.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:47:53.032 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:47:53.062 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:53.082 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:47:53.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:53.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:47:53.186 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:47:53.215 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:53.236 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:47:53.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:53.239 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:47:53.340 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:47:53.372 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:53.392 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:47:53.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:53.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:47:53.490 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:47:53.496 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:47:53.526 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:53.546 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:47:53.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:53.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:47:53.650 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:47:53.676 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:53 smithi120 ceph-mon[43961]: pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:53.678 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:53.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:53 smithi177 ceph-mon[44303]: pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:53.698 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:47:53.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:53.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:47:53.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:53 smithi027 ceph-mon[39795]: pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:53.801 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:47:53.833 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:53.854 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:47:53.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:53.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:47:53.959 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:47:53.987 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:54.006 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:47:54.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:54.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:47:54.110 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:47:54.143 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:54.164 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:47:54.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:54.167 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:47:54.268 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:47:54.298 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:54.318 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:47:54.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:54.322 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:47:54.423 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:47:54.455 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:54.476 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:47:54.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:54.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:47:54.581 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:47:54.612 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:54.634 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:47:54.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:54.637 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:47:54.738 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:47:54.767 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:54.786 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:47:54.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:54.790 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:47:54.891 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:47:54.919 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:54.939 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:47:54.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:54.942 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:47:55.042 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:47:55.074 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:55.095 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:47:55.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:55.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:47:55.199 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:47:55.232 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:55.337 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:47:55.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:55.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:47:55.442 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:47:55.474 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:55.495 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:47:55.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:55.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:47:55.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:55 smithi027 ceph-mon[39795]: pgmap v356: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:55.600 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:47:55.630 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:55 smithi177 ceph-mon[44303]: pgmap v356: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:55.632 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:55.653 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:47:55.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:55.657 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:47:55.758 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:47:55.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:55 smithi120 ceph-mon[43961]: pgmap v356: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:55.789 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:55.811 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:47:55.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:55.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:47:55.915 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:47:55.947 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:55.968 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:47:55.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:55.971 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:47:56.072 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:47:56.101 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:56.122 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:47:56.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:56.125 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:47:56.226 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:47:56.258 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:56.278 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:47:56.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:56.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:47:56.382 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:47:56.411 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:56.431 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:47:56.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:56.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:47:56.535 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:47:56.564 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:56.584 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:47:56.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:56.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:47:56.688 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:47:56.717 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:56.735 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:47:56.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:56.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:47:56.839 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:47:56.871 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:56.893 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:47:56.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:56.897 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:47:56.997 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:47:57.027 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:57.046 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:47:57.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:57.049 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:47:57.150 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:47:57.178 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:57.197 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:47:57.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:57.200 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:47:57.301 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:47:57.333 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:57.354 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:47:57.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:57.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:47:57.387 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:47:57.407 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:47:57.409 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:47:57.410 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T12:47:57.410 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 109, 2024-05-15T12:47:57.410 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "deaea04e-12b7-11ef-bc98-c7b262605968", 2024-05-15T12:47:57.410 INFO:teuthology.orchestra.run.smithi027.stdout: "created": "2024-05-15T12:38:21.244556+0000", 2024-05-15T12:47:57.410 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-05-15T12:47:43.349784+0000", 2024-05-15T12:47:57.410 INFO:teuthology.orchestra.run.smithi027.stdout: "last_up_change": "2024-05-15T12:45:50.876090+0000", 2024-05-15T12:47:57.410 INFO:teuthology.orchestra.run.smithi027.stdout: "last_in_change": "2024-05-15T12:45:33.158047+0000", 2024-05-15T12:47:57.410 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-15T12:47:57.410 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_num": 5799936, 2024-05-15T12:47:57.410 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_set": [ 2024-05-15T12:47:57.410 INFO:teuthology.orchestra.run.smithi027.stdout: "pglog_hardlimit", 2024-05-15T12:47:57.410 INFO:teuthology.orchestra.run.smithi027.stdout: "purged_snapdirs", 2024-05-15T12:47:57.410 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_deletes", 2024-05-15T12:47:57.411 INFO:teuthology.orchestra.run.smithi027.stdout: "sortbitwise" 2024-05-15T12:47:57.411 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T12:47:57.411 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_version": 26, 2024-05-15T12:47:57.411 INFO:teuthology.orchestra.run.smithi027.stdout: "full_ratio": 0.94999998807907104, 2024-05-15T12:47:57.411 INFO:teuthology.orchestra.run.smithi027.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-05-15T12:47:57.411 INFO:teuthology.orchestra.run.smithi027.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-15T12:47:57.411 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_snapshot": "", 2024-05-15T12:47:57.411 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_max": 1, 2024-05-15T12:47:57.411 INFO:teuthology.orchestra.run.smithi027.stdout: "max_osd": 12, 2024-05-15T12:47:57.411 INFO:teuthology.orchestra.run.smithi027.stdout: "require_min_compat_client": "luminous", 2024-05-15T12:47:57.411 INFO:teuthology.orchestra.run.smithi027.stdout: "min_compat_client": "jewel", 2024-05-15T12:47:57.411 INFO:teuthology.orchestra.run.smithi027.stdout: "require_osd_release": "squid", 2024-05-15T12:47:57.411 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_crimson": false, 2024-05-15T12:47:57.411 INFO:teuthology.orchestra.run.smithi027.stdout: "pools": [ 2024-05-15T12:47:57.412 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.412 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 1, 2024-05-15T12:47:57.412 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": ".mgr", 2024-05-15T12:47:57.412 INFO:teuthology.orchestra.run.smithi027.stdout: "create_time": "2024-05-15T12:41:57.948613+0000", 2024-05-15T12:47:57.412 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-15T12:47:57.412 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_names": "hashpspool", 2024-05-15T12:47:57.412 INFO:teuthology.orchestra.run.smithi027.stdout: "type": 1, 2024-05-15T12:47:57.412 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 3, 2024-05-15T12:47:57.412 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": 2, 2024-05-15T12:47:57.412 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_rule": 0, 2024-05-15T12:47:57.412 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_count": 0, 2024-05-15T12:47:57.412 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_target": 0, 2024-05-15T12:47:57.412 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T12:47:57.412 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T12:47:57.412 INFO:teuthology.orchestra.run.smithi027.stdout: "object_hash": 2, 2024-05-15T12:47:57.413 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscale_mode": "off", 2024-05-15T12:47:57.413 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": 1, 2024-05-15T12:47:57.413 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num": 1, 2024-05-15T12:47:57.413 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num_target": 1, 2024-05-15T12:47:57.413 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_target": 1, 2024-05-15T12:47:57.413 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_pending": 1, 2024-05-15T12:47:57.413 INFO:teuthology.orchestra.run.smithi027.stdout: "last_pg_merge_meta": { 2024-05-15T12:47:57.413 INFO:teuthology.orchestra.run.smithi027.stdout: "source_pgid": "0.0", 2024-05-15T12:47:57.413 INFO:teuthology.orchestra.run.smithi027.stdout: "ready_epoch": 0, 2024-05-15T12:47:57.413 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_started": 0, 2024-05-15T12:47:57.413 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_clean": 0, 2024-05-15T12:47:57.413 INFO:teuthology.orchestra.run.smithi027.stdout: "source_version": "0'0", 2024-05-15T12:47:57.413 INFO:teuthology.orchestra.run.smithi027.stdout: "target_version": "0'0" 2024-05-15T12:47:57.413 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.413 INFO:teuthology.orchestra.run.smithi027.stdout: "last_change": "21", 2024-05-15T12:47:57.414 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend": "0", 2024-05-15T12:47:57.414 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T12:47:57.414 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T12:47:57.414 INFO:teuthology.orchestra.run.smithi027.stdout: "auid": 0, 2024-05-15T12:47:57.414 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_mode": "selfmanaged", 2024-05-15T12:47:57.414 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_seq": 0, 2024-05-15T12:47:57.414 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_epoch": 0, 2024-05-15T12:47:57.414 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_snaps": [], 2024-05-15T12:47:57.414 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps": "[]", 2024-05-15T12:47:57.414 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_bytes": 0, 2024-05-15T12:47:57.414 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_objects": 0, 2024-05-15T12:47:57.414 INFO:teuthology.orchestra.run.smithi027.stdout: "tiers": [], 2024-05-15T12:47:57.414 INFO:teuthology.orchestra.run.smithi027.stdout: "tier_of": -1, 2024-05-15T12:47:57.414 INFO:teuthology.orchestra.run.smithi027.stdout: "read_tier": -1, 2024-05-15T12:47:57.415 INFO:teuthology.orchestra.run.smithi027.stdout: "write_tier": -1, 2024-05-15T12:47:57.415 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_mode": "none", 2024-05-15T12:47:57.415 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_bytes": 0, 2024-05-15T12:47:57.415 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_objects": 0, 2024-05-15T12:47:57.415 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T12:47:57.415 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T12:47:57.415 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T12:47:57.415 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_flush_age": 0, 2024-05-15T12:47:57.415 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_evict_age": 0, 2024-05-15T12:47:57.415 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profile": "", 2024-05-15T12:47:57.415 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_params": { 2024-05-15T12:47:57.415 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "none" 2024-05-15T12:47:57.415 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.415 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_period": 0, 2024-05-15T12:47:57.415 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_count": 0, 2024-05-15T12:47:57.416 INFO:teuthology.orchestra.run.smithi027.stdout: "use_gmt_hitset": true, 2024-05-15T12:47:57.416 INFO:teuthology.orchestra.run.smithi027.stdout: "min_read_recency_for_promote": 0, 2024-05-15T12:47:57.416 INFO:teuthology.orchestra.run.smithi027.stdout: "min_write_recency_for_promote": 0, 2024-05-15T12:47:57.416 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T12:47:57.416 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_search_last_n": 0, 2024-05-15T12:47:57.416 INFO:teuthology.orchestra.run.smithi027.stdout: "grade_table": [], 2024-05-15T12:47:57.416 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_width": 0, 2024-05-15T12:47:57.416 INFO:teuthology.orchestra.run.smithi027.stdout: "expected_num_objects": 0, 2024-05-15T12:47:57.416 INFO:teuthology.orchestra.run.smithi027.stdout: "fast_read": false, 2024-05-15T12:47:57.416 INFO:teuthology.orchestra.run.smithi027.stdout: "options": { 2024-05-15T12:47:57.416 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_max": 32, 2024-05-15T12:47:57.416 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_min": 1 2024-05-15T12:47:57.416 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.417 INFO:teuthology.orchestra.run.smithi027.stdout: "application_metadata": { 2024-05-15T12:47:57.417 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": {} 2024-05-15T12:47:57.417 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.417 INFO:teuthology.orchestra.run.smithi027.stdout: "read_balance": { 2024-05-15T12:47:57.417 INFO:teuthology.orchestra.run.smithi027.stdout: "score_type": "Fair distribution", 2024-05-15T12:47:57.417 INFO:teuthology.orchestra.run.smithi027.stdout: "score_acting": 12, 2024-05-15T12:47:57.417 INFO:teuthology.orchestra.run.smithi027.stdout: "score_stable": 12, 2024-05-15T12:47:57.417 INFO:teuthology.orchestra.run.smithi027.stdout: "optimal_score": 0.25, 2024-05-15T12:47:57.417 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_acting": 3, 2024-05-15T12:47:57.417 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_stable": 3, 2024-05-15T12:47:57.417 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-05-15T12:47:57.417 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity": 1, 2024-05-15T12:47:57.417 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity_weighted": 1 2024-05-15T12:47:57.417 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.418 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.418 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T12:47:57.418 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T12:47:57.418 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.418 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 0, 2024-05-15T12:47:57.418 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "64fb3ad2-08a7-4106-a07a-6c776c877faf", 2024-05-15T12:47:57.418 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:47:57.418 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:47:57.418 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:47:57.418 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:47:57.418 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:47:57.418 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:47:57.418 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 8, 2024-05-15T12:47:57.419 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 58, 2024-05-15T12:47:57.419 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:47:57.419 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:47:57.419 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:47:57.419 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.419 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.419 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.419 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6802", 2024-05-15T12:47:57.419 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:47:57.419 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.419 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.419 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.419 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6803", 2024-05-15T12:47:57.420 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:47:57.420 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.420 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.420 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.420 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:47:57.420 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.420 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.420 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.420 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6804", 2024-05-15T12:47:57.420 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:47:57.420 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.420 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.420 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.420 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6805", 2024-05-15T12:47:57.421 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:47:57.421 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.421 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.421 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.421 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:47:57.421 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.421 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.421 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.421 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6808", 2024-05-15T12:47:57.421 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:47:57.421 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.421 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.421 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.421 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6809", 2024-05-15T12:47:57.421 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:47:57.422 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.422 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.422 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.422 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:47:57.422 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.422 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.422 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.422 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6806", 2024-05-15T12:47:57.422 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:47:57.422 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.422 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.422 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.422 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6807", 2024-05-15T12:47:57.422 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:47:57.422 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.423 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.423 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.423 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6803/4239116935", 2024-05-15T12:47:57.423 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6805/4239116935", 2024-05-15T12:47:57.423 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6809/4239116935", 2024-05-15T12:47:57.423 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6807/4239116935", 2024-05-15T12:47:57.423 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:47:57.423 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:47:57.423 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:47:57.423 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.423 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.423 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.423 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 1, 2024-05-15T12:47:57.423 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "9c16dd90-9d51-48b8-a296-1c8879ba1bc8", 2024-05-15T12:47:57.424 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:47:57.424 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:47:57.424 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:47:57.424 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:47:57.424 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:47:57.424 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:47:57.424 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 13, 2024-05-15T12:47:57.424 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 34, 2024-05-15T12:47:57.424 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:47:57.424 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:47:57.424 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:47:57.424 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.424 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.424 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.424 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6810", 2024-05-15T12:47:57.424 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:47:57.425 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.425 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.425 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.425 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6811", 2024-05-15T12:47:57.425 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:47:57.425 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.425 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.425 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.425 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:47:57.425 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.425 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.425 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.425 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6812", 2024-05-15T12:47:57.426 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:47:57.426 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.426 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.426 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.426 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6813", 2024-05-15T12:47:57.426 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:47:57.426 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.426 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.426 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.426 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:47:57.426 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.426 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.426 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.427 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6816", 2024-05-15T12:47:57.427 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:47:57.427 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.427 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.427 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.427 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6817", 2024-05-15T12:47:57.427 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:47:57.427 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.427 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.427 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.427 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:47:57.427 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.427 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.428 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.428 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6814", 2024-05-15T12:47:57.428 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:47:57.428 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.428 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.428 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.428 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6815", 2024-05-15T12:47:57.428 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:47:57.428 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.428 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.428 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.428 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6811/1466270637", 2024-05-15T12:47:57.428 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6813/1466270637", 2024-05-15T12:47:57.428 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6817/1466270637", 2024-05-15T12:47:57.429 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6815/1466270637", 2024-05-15T12:47:57.429 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:47:57.429 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:47:57.429 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:47:57.429 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.429 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.429 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.429 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 2, 2024-05-15T12:47:57.429 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "cbe3ba13-3ea3-4891-8d2d-6b85581a732c", 2024-05-15T12:47:57.429 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:47:57.429 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:47:57.429 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:47:57.429 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:47:57.429 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:47:57.430 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:47:57.430 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 18, 2024-05-15T12:47:57.430 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 0, 2024-05-15T12:47:57.430 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:47:57.430 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:47:57.430 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:47:57.430 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.430 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.430 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.430 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6818", 2024-05-15T12:47:57.430 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:47:57.430 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.430 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.430 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.431 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6819", 2024-05-15T12:47:57.431 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:47:57.431 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.431 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.431 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.431 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:47:57.431 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.431 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.431 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.431 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6820", 2024-05-15T12:47:57.431 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:47:57.431 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.431 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.431 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.431 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6821", 2024-05-15T12:47:57.431 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:47:57.432 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.432 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.432 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.432 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:47:57.432 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.432 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.432 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.432 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6824", 2024-05-15T12:47:57.432 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:47:57.432 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.432 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.432 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.432 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6825", 2024-05-15T12:47:57.432 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:47:57.432 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.433 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.433 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.433 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:47:57.433 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.433 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.433 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.433 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6822", 2024-05-15T12:47:57.433 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:47:57.433 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.433 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.433 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.433 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6823", 2024-05-15T12:47:57.433 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:47:57.433 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.433 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.434 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.434 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6819/3285413897", 2024-05-15T12:47:57.434 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6821/3285413897", 2024-05-15T12:47:57.434 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6825/3285413897", 2024-05-15T12:47:57.434 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6823/3285413897", 2024-05-15T12:47:57.434 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:47:57.434 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:47:57.434 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:47:57.434 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.434 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.434 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.434 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 3, 2024-05-15T12:47:57.434 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "2e83735b-b944-4f87-8080-b994d4b96897", 2024-05-15T12:47:57.434 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:47:57.435 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:47:57.435 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:47:57.435 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:47:57.435 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:47:57.435 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:47:57.435 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 26, 2024-05-15T12:47:57.435 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 0, 2024-05-15T12:47:57.435 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:47:57.435 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:47:57.435 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:47:57.435 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.435 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.435 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.435 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6826", 2024-05-15T12:47:57.435 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:47:57.436 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.436 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.436 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.436 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6827", 2024-05-15T12:47:57.436 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:47:57.436 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.436 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.436 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.436 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:47:57.436 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.436 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.436 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.436 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6828", 2024-05-15T12:47:57.436 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:47:57.437 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.437 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.437 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.437 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6829", 2024-05-15T12:47:57.437 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:47:57.437 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.437 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.437 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.437 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:47:57.437 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.437 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.437 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.437 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6832", 2024-05-15T12:47:57.437 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:47:57.437 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.438 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.438 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.438 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6833", 2024-05-15T12:47:57.438 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:47:57.438 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.438 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.438 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.438 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:47:57.438 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.438 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.441 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.441 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6830", 2024-05-15T12:47:57.442 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:47:57.442 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.442 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.442 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.442 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6831", 2024-05-15T12:47:57.442 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:47:57.442 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.442 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.442 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.442 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6827/3008975244", 2024-05-15T12:47:57.442 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6829/3008975244", 2024-05-15T12:47:57.442 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6833/3008975244", 2024-05-15T12:47:57.442 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6831/3008975244", 2024-05-15T12:47:57.442 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:47:57.443 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:47:57.443 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:47:57.443 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.443 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.443 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.443 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 4, 2024-05-15T12:47:57.443 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "7c39c1dd-3867-46f7-8372-7688fa13b16a", 2024-05-15T12:47:57.444 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:47:57.444 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:47:57.444 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:47:57.444 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:47:57.444 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:47:57.444 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:47:57.444 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 31, 2024-05-15T12:47:57.444 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 0, 2024-05-15T12:47:57.444 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:47:57.444 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:47:57.444 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:47:57.444 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.444 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.444 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.445 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6800", 2024-05-15T12:47:57.445 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:47:57.445 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.445 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.445 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.445 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6801", 2024-05-15T12:47:57.445 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:47:57.445 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.445 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.445 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.445 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:47:57.445 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.446 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.446 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.446 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6802", 2024-05-15T12:47:57.446 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:47:57.446 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.446 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.446 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.446 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6803", 2024-05-15T12:47:57.446 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:47:57.446 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.446 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.446 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.446 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:47:57.446 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.447 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.447 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.447 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6806", 2024-05-15T12:47:57.447 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:47:57.447 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.447 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.447 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.447 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6807", 2024-05-15T12:47:57.447 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:47:57.447 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.447 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.447 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.447 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:47:57.447 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.447 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.448 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.448 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6804", 2024-05-15T12:47:57.448 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:47:57.448 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.448 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.448 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.448 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6805", 2024-05-15T12:47:57.448 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:47:57.448 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.448 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.448 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.448 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6801/3877842957", 2024-05-15T12:47:57.448 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6803/3877842957", 2024-05-15T12:47:57.448 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6807/3877842957", 2024-05-15T12:47:57.449 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6805/3877842957", 2024-05-15T12:47:57.449 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:47:57.449 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:47:57.449 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:47:57.449 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.449 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.449 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.449 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 5, 2024-05-15T12:47:57.449 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "a6dc220f-aec0-4454-9695-b44c0b9770e0", 2024-05-15T12:47:57.449 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:47:57.449 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:47:57.449 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:47:57.449 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:47:57.449 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:47:57.450 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:47:57.450 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 40, 2024-05-15T12:47:57.450 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 43, 2024-05-15T12:47:57.450 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:47:57.450 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:47:57.450 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:47:57.450 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.450 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.450 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.450 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6808", 2024-05-15T12:47:57.450 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:47:57.450 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.450 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.450 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.450 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6809", 2024-05-15T12:47:57.451 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:47:57.451 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.451 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.451 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.451 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:47:57.451 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.451 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.451 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.451 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6810", 2024-05-15T12:47:57.451 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:47:57.451 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.451 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.451 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.451 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6811", 2024-05-15T12:47:57.452 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:47:57.452 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.452 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.452 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.452 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:47:57.452 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.452 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.452 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.452 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6814", 2024-05-15T12:47:57.452 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:47:57.452 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.452 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.452 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.452 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6815", 2024-05-15T12:47:57.452 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:47:57.453 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.453 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.453 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.453 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:47:57.453 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.453 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.453 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.453 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6812", 2024-05-15T12:47:57.453 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:47:57.453 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.453 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.453 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.453 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6813", 2024-05-15T12:47:57.453 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:47:57.453 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.453 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.454 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.454 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6809/1600179183", 2024-05-15T12:47:57.454 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6811/1600179183", 2024-05-15T12:47:57.454 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6815/1600179183", 2024-05-15T12:47:57.454 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6813/1600179183", 2024-05-15T12:47:57.454 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:47:57.454 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:47:57.454 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:47:57.454 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.454 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.454 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.454 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 6, 2024-05-15T12:47:57.454 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b", 2024-05-15T12:47:57.454 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:47:57.454 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:47:57.455 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:47:57.455 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:47:57.455 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:47:57.455 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:47:57.455 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 48, 2024-05-15T12:47:57.455 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 51, 2024-05-15T12:47:57.455 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:47:57.455 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:47:57.455 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:47:57.455 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.455 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.455 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.455 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6816", 2024-05-15T12:47:57.455 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:47:57.455 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.455 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.456 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.456 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6817", 2024-05-15T12:47:57.456 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:47:57.456 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.456 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.456 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.456 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:47:57.456 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.456 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.456 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.456 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6818", 2024-05-15T12:47:57.456 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:47:57.456 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.456 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.456 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.457 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6819", 2024-05-15T12:47:57.457 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:47:57.457 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.457 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.457 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.457 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:47:57.457 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.457 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.457 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.457 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6822", 2024-05-15T12:47:57.457 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:47:57.457 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.457 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.457 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.458 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6823", 2024-05-15T12:47:57.458 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:47:57.458 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.458 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.458 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.458 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:47:57.458 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.458 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.458 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.458 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6820", 2024-05-15T12:47:57.458 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:47:57.458 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.459 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.459 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.459 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6821", 2024-05-15T12:47:57.459 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:47:57.459 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.459 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.459 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.459 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6817/3323706735", 2024-05-15T12:47:57.459 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6819/3323706735", 2024-05-15T12:47:57.459 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6823/3323706735", 2024-05-15T12:47:57.459 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6821/3323706735", 2024-05-15T12:47:57.459 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:47:57.459 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:47:57.459 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:47:57.460 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.460 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.460 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.460 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 7, 2024-05-15T12:47:57.460 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "6ef085d8-f59b-4033-8280-2c8f78693e30", 2024-05-15T12:47:57.460 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:47:57.460 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:47:57.460 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:47:57.460 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:47:57.460 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:47:57.460 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:47:57.460 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 56, 2024-05-15T12:47:57.460 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 93, 2024-05-15T12:47:57.460 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:47:57.460 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:47:57.460 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:47:57.461 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.461 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.461 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.461 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6824", 2024-05-15T12:47:57.461 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:47:57.461 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.461 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.461 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.461 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6825", 2024-05-15T12:47:57.461 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:47:57.461 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.461 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.461 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.461 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:47:57.461 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.461 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.462 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.462 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6826", 2024-05-15T12:47:57.462 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:47:57.462 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.462 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.462 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.462 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6827", 2024-05-15T12:47:57.462 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:47:57.462 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.462 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.462 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.462 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:47:57.462 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.462 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.463 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.463 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6830", 2024-05-15T12:47:57.463 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:47:57.463 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.463 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.463 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.463 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6831", 2024-05-15T12:47:57.463 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:47:57.463 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.463 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.463 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.463 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:47:57.463 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.463 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.463 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.464 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6828", 2024-05-15T12:47:57.464 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:47:57.464 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.464 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.464 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.464 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6829", 2024-05-15T12:47:57.464 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:47:57.464 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.464 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.464 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.464 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6825/2960616153", 2024-05-15T12:47:57.464 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6827/2960616153", 2024-05-15T12:47:57.464 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6831/2960616153", 2024-05-15T12:47:57.464 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6829/2960616153", 2024-05-15T12:47:57.464 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:47:57.465 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:47:57.465 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:47:57.465 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.465 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.465 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.465 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 8, 2024-05-15T12:47:57.465 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "26514b11-d02b-46e9-a4e8-3da193fcdbcc", 2024-05-15T12:47:57.465 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:47:57.465 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:47:57.465 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:47:57.465 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:47:57.465 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:47:57.465 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:47:57.465 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 65, 2024-05-15T12:47:57.465 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 0, 2024-05-15T12:47:57.466 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:47:57.466 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:47:57.466 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:47:57.466 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.466 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.466 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.466 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6800", 2024-05-15T12:47:57.466 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:47:57.466 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.466 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.466 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.466 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6801", 2024-05-15T12:47:57.466 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:47:57.466 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.467 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.467 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.467 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:47:57.467 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.467 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.467 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.467 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6802", 2024-05-15T12:47:57.467 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:47:57.467 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.467 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.467 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.467 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6803", 2024-05-15T12:47:57.467 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:47:57.468 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.468 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.468 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.468 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:47:57.468 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.468 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.468 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.468 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6806", 2024-05-15T12:47:57.468 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:47:57.468 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.468 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.468 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.468 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6807", 2024-05-15T12:47:57.469 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:47:57.469 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.469 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.469 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.469 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:47:57.469 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.469 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.469 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.469 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6804", 2024-05-15T12:47:57.469 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:47:57.469 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.469 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.469 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.469 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6805", 2024-05-15T12:47:57.469 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:47:57.470 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.470 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.470 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.470 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6801/1132459315", 2024-05-15T12:47:57.470 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6803/1132459315", 2024-05-15T12:47:57.470 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6807/1132459315", 2024-05-15T12:47:57.470 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6805/1132459315", 2024-05-15T12:47:57.470 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:47:57.470 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:47:57.470 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:47:57.470 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.470 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.470 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.470 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 9, 2024-05-15T12:47:57.471 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "6f9497bc-e962-409d-8a2c-0c634cc1124f", 2024-05-15T12:47:57.471 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:47:57.471 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:47:57.471 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:47:57.471 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:47:57.471 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:47:57.471 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:47:57.471 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 76, 2024-05-15T12:47:57.471 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 0, 2024-05-15T12:47:57.471 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:47:57.471 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:47:57.471 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:47:57.471 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.471 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.472 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.472 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6808", 2024-05-15T12:47:57.472 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:47:57.472 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.472 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.472 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.472 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6809", 2024-05-15T12:47:57.472 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:47:57.472 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.472 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.472 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.472 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:47:57.472 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.472 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.472 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.473 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6810", 2024-05-15T12:47:57.473 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:47:57.473 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.473 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.473 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.473 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6811", 2024-05-15T12:47:57.473 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:47:57.473 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.473 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.473 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.473 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:47:57.473 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.473 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.473 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.473 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6814", 2024-05-15T12:47:57.474 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:47:57.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.474 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.474 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.474 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6815", 2024-05-15T12:47:57.474 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:47:57.474 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.474 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.474 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:47:57.474 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.474 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.475 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.475 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6812", 2024-05-15T12:47:57.475 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:47:57.475 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.475 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.475 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.475 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6813", 2024-05-15T12:47:57.475 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:47:57.475 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.475 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.475 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.475 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6809/2710457646", 2024-05-15T12:47:57.475 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6811/2710457646", 2024-05-15T12:47:57.475 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6815/2710457646", 2024-05-15T12:47:57.476 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6813/2710457646", 2024-05-15T12:47:57.476 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:47:57.476 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:47:57.476 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:47:57.476 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.476 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.476 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 10, 2024-05-15T12:47:57.476 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "f852876f-c7d1-4039-90c4-5ab09dbd3e32", 2024-05-15T12:47:57.476 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:47:57.476 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:47:57.476 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:47:57.476 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:47:57.476 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:47:57.477 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:47:57.477 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 81, 2024-05-15T12:47:57.477 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 0, 2024-05-15T12:47:57.477 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:47:57.477 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:47:57.477 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:47:57.477 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.477 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.477 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.477 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6816", 2024-05-15T12:47:57.477 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:47:57.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.477 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.478 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.478 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6817", 2024-05-15T12:47:57.478 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:47:57.478 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.478 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.478 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.478 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:47:57.478 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.478 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.478 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6818", 2024-05-15T12:47:57.479 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:47:57.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.479 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.479 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6819", 2024-05-15T12:47:57.479 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:47:57.479 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.479 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.479 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:47:57.479 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.479 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.479 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6822", 2024-05-15T12:47:57.480 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:47:57.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.480 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.480 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.480 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6823", 2024-05-15T12:47:57.480 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:47:57.480 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.480 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.480 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:47:57.480 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.480 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.480 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.481 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6820", 2024-05-15T12:47:57.481 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:47:57.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.481 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.481 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6821", 2024-05-15T12:47:57.481 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:47:57.481 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.481 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.481 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6817/3236900263", 2024-05-15T12:47:57.481 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6819/3236900263", 2024-05-15T12:47:57.481 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6823/3236900263", 2024-05-15T12:47:57.481 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6821/3236900263", 2024-05-15T12:47:57.482 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:47:57.482 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:47:57.482 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:47:57.482 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.482 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 11, 2024-05-15T12:47:57.482 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "cccc53c5-8476-4d55-b160-4bdc5cf92902", 2024-05-15T12:47:57.482 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:47:57.482 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:47:57.482 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:47:57.482 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:47:57.482 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:47:57.482 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:47:57.483 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 90, 2024-05-15T12:47:57.483 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 0, 2024-05-15T12:47:57.483 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:47:57.483 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:47:57.483 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:47:57.483 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.483 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.483 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6824", 2024-05-15T12:47:57.483 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:47:57.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.483 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.483 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6825", 2024-05-15T12:47:57.484 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:47:57.484 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.484 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.484 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:47:57.484 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.484 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.484 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6826", 2024-05-15T12:47:57.484 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:47:57.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.484 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.484 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6827", 2024-05-15T12:47:57.484 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:47:57.485 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.485 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.485 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:47:57.485 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.485 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.485 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6830", 2024-05-15T12:47:57.485 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:47:57.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.485 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.485 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6831", 2024-05-15T12:47:57.485 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:47:57.485 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.486 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.486 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:47:57.486 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:47:57.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.486 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:47:57.486 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6828", 2024-05-15T12:47:57.486 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:47:57.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.486 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:47:57.486 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6829", 2024-05-15T12:47:57.486 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:47:57.486 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.487 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.487 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6825/681011628", 2024-05-15T12:47:57.487 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6827/681011628", 2024-05-15T12:47:57.487 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6831/681011628", 2024-05-15T12:47:57.487 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6829/681011628", 2024-05-15T12:47:57.487 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:47:57.487 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:47:57.487 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:47:57.487 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:47:57.487 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.487 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T12:47:57.487 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_xinfo": [ 2024-05-15T12:47:57.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.488 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 0, 2024-05-15T12:47:57.488 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:47:57.488 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:47:57.488 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:47:57.488 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:47:57.488 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:47:57.488 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:47:33.401880+0000", 2024-05-15T12:47:57.488 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:47:57.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.488 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 1, 2024-05-15T12:47:57.488 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:47:57.488 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:47:57.489 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:47:57.489 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:47:57.489 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:47:57.489 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:46:58.205736+0000", 2024-05-15T12:47:57.489 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:47:57.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.489 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 2, 2024-05-15T12:47:57.489 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:47:57.489 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:47:57.489 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:47:57.489 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:47:57.489 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:47:57.490 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:46:57.679599+0000", 2024-05-15T12:47:57.490 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:47:57.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.490 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 3, 2024-05-15T12:47:57.490 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:47:57.490 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:47:57.490 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:47:57.490 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:47:57.490 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:47:57.490 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:47:37.360495+0000", 2024-05-15T12:47:57.490 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:47:57.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.491 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 4, 2024-05-15T12:47:57.491 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:47:57.491 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:47:57.491 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:47:57.491 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:47:57.491 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:47:57.491 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:46:40.442595+0000", 2024-05-15T12:47:57.491 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:47:57.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.491 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 5, 2024-05-15T12:47:57.491 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:47:57.492 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:47:57.492 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:47:57.492 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:47:57.492 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:47:57.492 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:46:53.473605+0000", 2024-05-15T12:47:57.492 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:47:57.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.492 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 6, 2024-05-15T12:47:57.492 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:47:57.492 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:47:57.492 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:47:57.492 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:47:57.493 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:47:57.493 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:47:42.382426+0000", 2024-05-15T12:47:57.493 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:47:57.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.493 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 7, 2024-05-15T12:47:57.493 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:47:57.493 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:47:57.493 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:47:57.493 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:47:57.493 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:47:57.493 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:46:53.203182+0000", 2024-05-15T12:47:57.493 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:47:57.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.494 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 8, 2024-05-15T12:47:57.494 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:47:57.494 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:47:57.494 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:47:57.494 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:47:57.494 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:47:57.494 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:47:20.504432+0000", 2024-05-15T12:47:57.494 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:47:57.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.494 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 9, 2024-05-15T12:47:57.494 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:47:57.495 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:47:57.495 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:47:57.495 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:47:57.495 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:47:57.495 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:47:23.974039+0000", 2024-05-15T12:47:57.495 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:47:57.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.495 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 10, 2024-05-15T12:47:57.495 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:47:57.495 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:47:57.495 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:47:57.495 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:47:57.495 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:47:57.496 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:47:41.837735+0000", 2024-05-15T12:47:57.496 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:47:57.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:47:57.496 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 11, 2024-05-15T12:47:57.496 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:47:57.496 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:47:57.496 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:47:57.496 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:47:57.496 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:47:57.496 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:47:17.427055+0000", 2024-05-15T12:47:57.496 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:47:57.496 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.497 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T12:47:57.497 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap": [], 2024-05-15T12:47:57.497 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap_items": [], 2024-05-15T12:47:57.497 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap_primaries": [], 2024-05-15T12:47:57.497 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_temp": [], 2024-05-15T12:47:57.497 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_temp": [], 2024-05-15T12:47:57.497 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklist": { 2024-05-15T12:47:57.497 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6801/2530617473": "2024-05-16T12:39:19.409195+0000", 2024-05-15T12:47:57.497 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6800/2530617473": "2024-05-16T12:39:19.409195+0000", 2024-05-15T12:47:57.497 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/1880261204": "2024-05-16T12:39:19.409195+0000", 2024-05-15T12:47:57.497 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6801/3441136471": "2024-05-16T12:38:50.931011+0000", 2024-05-15T12:47:57.497 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/4291446247": "2024-05-16T12:39:19.409195+0000", 2024-05-15T12:47:57.497 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/580128209": "2024-05-16T12:38:50.931011+0000", 2024-05-15T12:47:57.497 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6800/3441136471": "2024-05-16T12:38:50.931011+0000", 2024-05-15T12:47:57.498 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/3011229857": "2024-05-16T12:38:50.931011+0000", 2024-05-15T12:47:57.498 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/2109693913": "2024-05-16T12:38:50.931011+0000", 2024-05-15T12:47:57.498 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/2445663024": "2024-05-16T12:39:19.409195+0000" 2024-05-15T12:47:57.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.498 INFO:teuthology.orchestra.run.smithi027.stdout: "range_blocklist": {}, 2024-05-15T12:47:57.498 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profiles": { 2024-05-15T12:47:57.498 INFO:teuthology.orchestra.run.smithi027.stdout: "default": { 2024-05-15T12:47:57.498 INFO:teuthology.orchestra.run.smithi027.stdout: "crush-failure-domain": "osd", 2024-05-15T12:47:57.498 INFO:teuthology.orchestra.run.smithi027.stdout: "k": "2", 2024-05-15T12:47:57.498 INFO:teuthology.orchestra.run.smithi027.stdout: "m": "1", 2024-05-15T12:47:57.498 INFO:teuthology.orchestra.run.smithi027.stdout: "plugin": "jerasure", 2024-05-15T12:47:57.498 INFO:teuthology.orchestra.run.smithi027.stdout: "technique": "reed_sol_van" 2024-05-15T12:47:57.498 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:47:57.499 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps_queue": [], 2024-05-15T12:47:57.499 INFO:teuthology.orchestra.run.smithi027.stdout: "new_removed_snaps": [], 2024-05-15T12:47:57.499 INFO:teuthology.orchestra.run.smithi027.stdout: "new_purged_snaps": [], 2024-05-15T12:47:57.499 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_node_flags": {}, 2024-05-15T12:47:57.499 INFO:teuthology.orchestra.run.smithi027.stdout: "device_class_flags": {}, 2024-05-15T12:47:57.499 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode": { 2024-05-15T12:47:57.499 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode_enabled": false, 2024-05-15T12:47:57.499 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_bucket_count": 0, 2024-05-15T12:47:57.499 INFO:teuthology.orchestra.run.smithi027.stdout: "degraded_stretch_mode": 0, 2024-05-15T12:47:57.499 INFO:teuthology.orchestra.run.smithi027.stdout: "recovering_stretch_mode": 0, 2024-05-15T12:47:57.499 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode_bucket": 0 2024-05-15T12:47:57.499 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:47:57.500 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T12:47:57.500 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:47:57.501 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:57 smithi177 ceph-mon[44303]: pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:57.501 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:57 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2809462007' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T12:47:57.501 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:57 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/39026711' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-05-15T12:47:57.502 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:57 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3253103499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:47:57.502 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:47:57.502 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:47:57.502 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:47:57.502 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:47:57.508 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:47:57.528 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:47:57.531 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:57.548 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:47:57.551 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:47:57.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:57.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:47:57.567 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:47:57.587 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:47:57.607 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:47:57.609 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:47:57.609 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:47:57.609 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:47:57.609 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:47:57.609 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:47:57.609 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:47:57.609 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:47:57.609 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:47:57.609 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:47:57.609 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:47:57.609 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:47:57.609 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:47:57.628 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:57 smithi027 ceph-mon[39795]: pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:57.628 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:57 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2809462007' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T12:47:57.628 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:57 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/39026711' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-05-15T12:47:57.629 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:57 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3253103499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:47:57.655 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:47:57.681 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:57 smithi120 ceph-mon[43961]: pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:57.681 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:57 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2809462007' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T12:47:57.681 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:57 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/39026711' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-05-15T12:47:57.681 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:57 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3253103499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:47:57.683 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:57.703 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:47:57.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:57.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:47:57.806 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:47:57.834 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:57.853 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:47:57.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:57.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:47:57.957 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:47:57.983 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:58.002 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:47:58.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:58.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:47:58.054 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_0' created 2024-05-15T12:47:58.105 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:47:58.132 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:58.151 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:47:58.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:58.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:47:58.255 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:47:58.282 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:58.300 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:47:58.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:58.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:47:58.406 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:47:58.432 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:58.451 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:47:58.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:58.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:47:58.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:58 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/39026711' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-05-15T12:47:58.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:58 smithi120 ceph-mon[43961]: osdmap e110: 12 total, 12 up, 12 in 2024-05-15T12:47:58.555 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:47:58.588 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:58.608 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:47:58.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:58.611 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:47:58.675 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:58 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/39026711' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-05-15T12:47:58.675 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:58 smithi027 ceph-mon[39795]: osdmap e110: 12 total, 12 up, 12 in 2024-05-15T12:47:58.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:58 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/39026711' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-05-15T12:47:58.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:58 smithi177 ceph-mon[44303]: osdmap e110: 12 total, 12 up, 12 in 2024-05-15T12:47:58.714 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:47:58.748 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:58.976 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:47:58.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:58.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:47:59.081 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:47:59.114 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:59.134 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:47:59.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:59.137 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:47:59.238 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:47:59.269 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:59.290 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:47:59.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:59.293 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:47:59.386 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T12:47:59.394 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:47:59.423 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:59.451 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:47:59.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:59.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:47:59.555 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:47:59.587 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:59.608 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:47:59.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:59.611 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:47:59.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:59 smithi177 ceph-mon[44303]: pgmap v359: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:59.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:59 smithi177 ceph-mon[44303]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T12:47:59.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:59 smithi177 ceph-mon[44303]: osdmap e111: 12 total, 12 up, 12 in 2024-05-15T12:47:59.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:47:59 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3678960125' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:47:59.712 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:47:59.738 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:59 smithi027 ceph-mon[39795]: pgmap v359: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:59.738 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:59 smithi027 ceph-mon[39795]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T12:47:59.738 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:59 smithi027 ceph-mon[39795]: osdmap e111: 12 total, 12 up, 12 in 2024-05-15T12:47:59.738 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:47:59 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3678960125' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:47:59.739 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:59.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:59 smithi120 ceph-mon[43961]: pgmap v359: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:47:59.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:59 smithi120 ceph-mon[43961]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T12:47:59.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:59 smithi120 ceph-mon[43961]: osdmap e111: 12 total, 12 up, 12 in 2024-05-15T12:47:59.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:47:59 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3678960125' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:47:59.813 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:47:59.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:59.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:47:59.919 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:47:59.951 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:47:59.973 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:47:59.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:47:59.976 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:48:00.077 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:48:00.105 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:00.126 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:48:00.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:00.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:48:00.148 INFO:tasks.thrashosds.thrasher:j is {'epoch': 109, 'fsid': 'deaea04e-12b7-11ef-bc98-c7b262605968', 'created': '2024-05-15T12:38:21.244556+0000', 'modified': '2024-05-15T12:47:43.349784+0000', 'last_up_change': '2024-05-15T12:45:50.876090+0000', 'last_in_change': '2024-05-15T12:45:33.158047+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 1, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-15T12:41:57.948613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '64fb3ad2-08a7-4106-a07a-6c776c877faf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 58, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6802', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6803', 'nonce': 4239116935}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6804', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6805', 'nonce': 4239116935}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6808', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6809', 'nonce': 4239116935}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6806', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6807', 'nonce': 4239116935}]}, 'public_addr': '172.21.15.27:6803/4239116935', 'cluster_addr': '172.21.15.27:6805/4239116935', 'heartbeat_back_addr': '172.21.15.27:6809/4239116935', 'heartbeat_front_addr': '172.21.15.27:6807/4239116935', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '9c16dd90-9d51-48b8-a296-1c8879ba1bc8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6810', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6811', 'nonce': 1466270637}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6812', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6813', 'nonce': 1466270637}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6816', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6817', 'nonce': 1466270637}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6814', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6815', 'nonce': 1466270637}]}, 'public_addr': '172.21.15.27:6811/1466270637', 'cluster_addr': '172.21.15.27:6813/1466270637', 'heartbeat_back_addr': '172.21.15.27:6817/1466270637', 'heartbeat_front_addr': '172.21.15.27:6815/1466270637', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'cbe3ba13-3ea3-4891-8d2d-6b85581a732c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6818', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6819', 'nonce': 3285413897}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6820', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6821', 'nonce': 3285413897}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6824', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6825', 'nonce': 3285413897}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6822', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6823', 'nonce': 3285413897}]}, 'public_addr': '172.21.15.27:6819/3285413897', 'cluster_addr': '172.21.15.27:6821/3285413897', 'heartbeat_back_addr': '172.21.15.27:6825/3285413897', 'heartbeat_front_addr': '172.21.15.27:6823/3285413897', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2e83735b-b944-4f87-8080-b994d4b96897', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6826', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6827', 'nonce': 3008975244}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6828', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6829', 'nonce': 3008975244}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6833', 'nonce': 3008975244}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6830', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6831', 'nonce': 3008975244}]}, 'public_addr': '172.21.15.27:6827/3008975244', 'cluster_addr': '172.21.15.27:6829/3008975244', 'heartbeat_back_addr': '172.21.15.27:6833/3008975244', 'heartbeat_front_addr': '172.21.15.27:6831/3008975244', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7c39c1dd-3867-46f7-8372-7688fa13b16a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6800', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6801', 'nonce': 3877842957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6802', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6803', 'nonce': 3877842957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6806', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6807', 'nonce': 3877842957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6804', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6805', 'nonce': 3877842957}]}, 'public_addr': '172.21.15.120:6801/3877842957', 'cluster_addr': '172.21.15.120:6803/3877842957', 'heartbeat_back_addr': '172.21.15.120:6807/3877842957', 'heartbeat_front_addr': '172.21.15.120:6805/3877842957', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a6dc220f-aec0-4454-9695-b44c0b9770e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 43, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6808', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6809', 'nonce': 1600179183}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6810', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6811', 'nonce': 1600179183}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6814', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6815', 'nonce': 1600179183}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6812', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6813', 'nonce': 1600179183}]}, 'public_addr': '172.21.15.120:6809/1600179183', 'cluster_addr': '172.21.15.120:6811/1600179183', 'heartbeat_back_addr': '172.21.15.120:6815/1600179183', 'heartbeat_front_addr': '172.21.15.120:6813/1600179183', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 51, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6816', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6817', 'nonce': 3323706735}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6818', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6819', 'nonce': 3323706735}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6822', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6823', 'nonce': 3323706735}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6820', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6821', 'nonce': 3323706735}]}, 'public_addr': '172.21.15.120:6817/3323706735', 'cluster_addr': '172.21.15.120:6819/3323706735', 'heartbeat_back_addr': '172.21.15.120:6823/3323706735', 'heartbeat_front_addr': '172.21.15.120:6821/3323706735', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6ef085d8-f59b-4033-8280-2c8f78693e30', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 93, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6824', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6825', 'nonce': 2960616153}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6826', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6827', 'nonce': 2960616153}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6830', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6831', 'nonce': 2960616153}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6828', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6829', 'nonce': 2960616153}]}, 'public_addr': '172.21.15.120:6825/2960616153', 'cluster_addr': '172.21.15.120:6827/2960616153', 'heartbeat_back_addr': '172.21.15.120:6831/2960616153', 'heartbeat_front_addr': '172.21.15.120:6829/2960616153', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '26514b11-d02b-46e9-a4e8-3da193fcdbcc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6800', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6801', 'nonce': 1132459315}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6802', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6803', 'nonce': 1132459315}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6806', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6807', 'nonce': 1132459315}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6804', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6805', 'nonce': 1132459315}]}, 'public_addr': '172.21.15.177:6801/1132459315', 'cluster_addr': '172.21.15.177:6803/1132459315', 'heartbeat_back_addr': '172.21.15.177:6807/1132459315', 'heartbeat_front_addr': '172.21.15.177:6805/1132459315', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '6f9497bc-e962-409d-8a2c-0c634cc1124f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6808', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6809', 'nonce': 2710457646}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6810', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6811', 'nonce': 2710457646}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6814', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6815', 'nonce': 2710457646}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6812', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6813', 'nonce': 2710457646}]}, 'public_addr': '172.21.15.177:6809/2710457646', 'cluster_addr': '172.21.15.177:6811/2710457646', 'heartbeat_back_addr': '172.21.15.177:6815/2710457646', 'heartbeat_front_addr': '172.21.15.177:6813/2710457646', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'f852876f-c7d1-4039-90c4-5ab09dbd3e32', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 81, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6816', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6817', 'nonce': 3236900263}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6818', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6819', 'nonce': 3236900263}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6822', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6823', 'nonce': 3236900263}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6820', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6821', 'nonce': 3236900263}]}, 'public_addr': '172.21.15.177:6817/3236900263', 'cluster_addr': '172.21.15.177:6819/3236900263', 'heartbeat_back_addr': '172.21.15.177:6823/3236900263', 'heartbeat_front_addr': '172.21.15.177:6821/3236900263', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'cccc53c5-8476-4d55-b160-4bdc5cf92902', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 90, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6824', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6825', 'nonce': 681011628}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6826', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6827', 'nonce': 681011628}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6830', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6831', 'nonce': 681011628}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6828', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6829', 'nonce': 681011628}]}, 'public_addr': '172.21.15.177:6825/681011628', 'cluster_addr': '172.21.15.177:6827/681011628', 'heartbeat_back_addr': '172.21.15.177:6831/681011628', 'heartbeat_front_addr': '172.21.15.177:6829/681011628', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:47:33.401880+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:46:58.205736+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:46:57.679599+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:47:37.360495+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:46:40.442595+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:46:53.473605+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:47:42.382426+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:46:53.203182+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:47:20.504432+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:47:23.974039+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:47:41.837735+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:47:17.427055+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.27:6801/2530617473': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:6800/2530617473': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:0/1880261204': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:6801/3441136471': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/4291446247': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:0/580128209': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:6800/3441136471': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/3011229857': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/2109693913': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/2445663024': '2024-05-16T12:39:19.409195+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-05-15T12:48:00.148 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-05-15T12:48:00.230 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:48:00.256 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:00.657 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:00 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:48:00.657 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:00 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3678960125' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:48:00.657 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:00 smithi027 ceph-mon[39795]: osdmap e112: 12 total, 12 up, 12 in 2024-05-15T12:48:00.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:00 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:48:00.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:00 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3678960125' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:48:00.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:00 smithi177 ceph-mon[44303]: osdmap e112: 12 total, 12 up, 12 in 2024-05-15T12:48:00.682 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-05-15T12:48:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:48:00.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:00.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:48:00.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:00 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:48:00.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:00 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3678960125' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:48:00.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:00 smithi120 ceph-mon[43961]: osdmap e112: 12 total, 12 up, 12 in 2024-05-15T12:48:00.808 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:48:00.840 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:00.844 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T12:48:00.844 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:48:00.861 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:48:00.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:00.864 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:48:00.965 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:48:00.994 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:01.014 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:48:01.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:01.017 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:48:01.118 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:48:01.146 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:01.174 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:48:01.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:01.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:48:01.286 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:48:01.313 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:01.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:01 smithi027 ceph-mon[39795]: pgmap v362: 17 pgs: 5 creating+peering, 1 active+clean, 11 unknown; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:48:01.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:01 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:48:01.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:01 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:48:01.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:01 smithi027 ceph-mon[39795]: osdmap e113: 12 total, 12 up, 12 in 2024-05-15T12:48:01.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:01 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:48:01.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:01 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:48:01.482 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:48:01.561 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:48:01.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:01.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:48:01.566 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-05-15T12:48:01.665 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:48:01.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:01 smithi177 ceph-mon[44303]: pgmap v362: 17 pgs: 5 creating+peering, 1 active+clean, 11 unknown; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:48:01.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:01 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:48:01.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:01 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:48:01.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:01 smithi177 ceph-mon[44303]: osdmap e113: 12 total, 12 up, 12 in 2024-05-15T12:48:01.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:01 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:48:01.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:01 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:48:01.697 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:01.718 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:48:01.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:01.721 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:48:01.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:01 smithi120 ceph-mon[43961]: pgmap v362: 17 pgs: 5 creating+peering, 1 active+clean, 11 unknown; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:48:01.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:01 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:48:01.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:01 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:48:01.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:01 smithi120 ceph-mon[43961]: osdmap e113: 12 total, 12 up, 12 in 2024-05-15T12:48:01.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:01 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:48:01.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:01 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:48:01.822 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:48:01.852 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:01.872 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:48:01.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:01.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:48:01.976 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:48:02.004 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:02.023 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:48:02.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:02.026 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:48:02.127 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:48:02.159 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:02.181 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:48:02.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:02.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:48:02.210 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:48:02.238 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:48:02.248 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:48:02.285 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:48:02.314 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:02.334 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:48:02.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:02.337 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:48:02.438 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:48:02.465 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:02 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:48:02.465 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:02 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:48:02.465 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:02 smithi027 ceph-mon[39795]: osdmap e114: 12 total, 12 up, 12 in 2024-05-15T12:48:02.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:02 smithi027 ceph-mon[39795]: pgmap v365: 17 pgs: 10 active+clean, 5 creating+peering, 2 unknown; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:48:02.468 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:02.488 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:48:02.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:02.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:48:02.592 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:48:02.621 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:02.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:02 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:48:02.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:02 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:48:02.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:02 smithi177 ceph-mon[44303]: osdmap e114: 12 total, 12 up, 12 in 2024-05-15T12:48:02.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:02 smithi177 ceph-mon[44303]: pgmap v365: 17 pgs: 10 active+clean, 5 creating+peering, 2 unknown; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:48:02.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:02 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:48:02.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:02 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:48:02.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:02 smithi120 ceph-mon[43961]: osdmap e114: 12 total, 12 up, 12 in 2024-05-15T12:48:02.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:02 smithi120 ceph-mon[43961]: pgmap v365: 17 pgs: 10 active+clean, 5 creating+peering, 2 unknown; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:48:04.203 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T12:48:04.672 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:48:04.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:04.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:48:04.776 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:48:04.805 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:04 smithi120 ceph-mon[43961]: pgmap v366: 17 pgs: 17 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:48:04.808 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:04.828 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:48:04.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:04.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:48:04.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:04 smithi177 ceph-mon[44303]: pgmap v366: 17 pgs: 17 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:48:04.932 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:48:04.962 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:04.982 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:48:04.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:04.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:48:05.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:04 smithi027 ceph-mon[39795]: pgmap v366: 17 pgs: 17 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:48:05.086 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:48:05.118 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:05.140 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:48:05.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:05.143 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:48:05.149 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T12:48:05.149 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T12:48:05.149 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-05-15T12:48:05.150 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd dump -f json-pretty 2024-05-15T12:48:05.244 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:48:05.271 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:06.815 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:06 smithi027 ceph-mon[39795]: pgmap v367: 17 pgs: 17 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:48:06.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:06 smithi177 ceph-mon[44303]: pgmap v367: 17 pgs: 17 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:48:07.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:06 smithi120 ceph-mon[43961]: pgmap v367: 17 pgs: 17 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:48:07.098 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T12:48:07.098 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:48:07.482 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:48:07.482 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:48:07.482 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:48:07.483 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:48:07.483 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:48:07.483 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:48:07.483 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:48:07.483 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:48:07.483 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:48:07.483 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:48:07.483 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:48:07.483 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:48:07.484 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:48:07.484 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:48:07.484 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:48:07.484 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:48:07.484 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:48:07.484 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:48:07.484 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:48:07.484 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:48:07.484 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:48:07.484 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:48:07.484 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:48:07.484 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:48:07.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:07 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1932705565' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:48:07.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:07 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1932705565' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:48:08.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:07 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1932705565' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:48:08.351 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-05-15T12:48:08.352 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:48:08.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:08.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:48:08.456 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:48:08.484 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:08.823 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-05-15T12:48:08.935 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:48:08.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:08.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:48:09.040 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:48:09.070 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:08 smithi177 ceph-mon[44303]: pgmap v368: 17 pgs: 17 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:48:09.070 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:08 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1870706099' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:48:09.070 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:08 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:48:09.070 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:08 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:48:09.072 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:09.093 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:48:09.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:09.096 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:48:09.169 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:48:09.197 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:48:09.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:08 smithi027 ceph-mon[39795]: pgmap v368: 17 pgs: 17 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:48:09.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:08 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1870706099' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:48:09.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:08 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:48:09.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:08 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:48:09.224 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:48:09.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:09.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:48:09.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:08 smithi120 ceph-mon[43961]: pgmap v368: 17 pgs: 17 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:48:09.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:08 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1870706099' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:48:09.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:08 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:48:09.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:08 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:48:09.363 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:48:09.390 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:09.733 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:48:09.734 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:48:09.739 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:48:09.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:09.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:48:09.844 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:48:09.871 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:09.994 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:48:09.996 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:09 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1870706099' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T12:48:09.996 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:09 smithi027 ceph-mon[39795]: osdmap e115: 12 total, 12 up, 12 in 2024-05-15T12:48:09.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:09.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:48:10.098 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:48:10.128 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:09 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1870706099' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T12:48:10.128 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:09 smithi120 ceph-mon[43961]: osdmap e115: 12 total, 12 up, 12 in 2024-05-15T12:48:10.130 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:10.151 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:48:10.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:10.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:48:10.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:09 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1870706099' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T12:48:10.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:09 smithi177 ceph-mon[44303]: osdmap e115: 12 total, 12 up, 12 in 2024-05-15T12:48:10.255 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:48:10.285 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:10.305 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:48:10.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:10.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:48:10.410 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:48:10.442 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:10.463 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:48:10.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:10.466 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:48:10.567 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:48:10.599 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:10.620 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:48:10.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:10.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:48:10.692 DEBUG:teuthology.orchestra.run.smithi177:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-05-15T12:48:10.695 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-05-15T12:48:10.695 INFO:teuthology.orchestra.run:waiting for 3000 2024-05-15T12:48:10.724 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:48:10.752 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:10.766 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:hints = 1 2024-05-15T12:48:10.767 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-05-15T12:48:10.767 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Object prefix: benchmark_data_smithi177_78104 2024-05-15T12:48:10.767 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T12:48:10.767 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 0 0 0 0 0 0 - 0 2024-05-15T12:48:10.775 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:48:10.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:10.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:48:10.828 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:10 smithi177 ceph-mon[44303]: pgmap v370: 17 pgs: 17 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:48:10.828 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:10 smithi177 ceph-mon[44303]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T12:48:10.888 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:48:10.931 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:10.962 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:48:10.964 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:48:10.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:10.966 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:48:11.010 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:10 smithi027 ceph-mon[39795]: pgmap v370: 17 pgs: 17 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:48:11.010 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:10 smithi027 ceph-mon[39795]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T12:48:11.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:10 smithi120 ceph-mon[43961]: pgmap v370: 17 pgs: 17 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:48:11.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:10 smithi120 ceph-mon[43961]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T12:48:11.067 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:48:11.110 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:11.138 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:48:11.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:11.141 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:48:11.242 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:48:11.277 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:11.301 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:48:11.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:11.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:48:11.406 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:48:11.452 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:11.768 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 1 15 4203 4188 261.727 261.75 0.00533779 0.00380973 2024-05-15T12:48:12.510 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:48:12.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:12.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:48:12.615 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:48:12.664 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:12.767 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 2 16 7721 7705 240.757 219.812 0.00218213 0.00414437 2024-05-15T12:48:12.873 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:48:12.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:12.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:48:12.979 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:48:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:13.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:12 smithi177 ceph-mon[44303]: pgmap v371: 17 pgs: 17 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:48:13.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:12 smithi027 ceph-mon[39795]: pgmap v371: 17 pgs: 17 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:48:13.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:12 smithi120 ceph-mon[43961]: pgmap v371: 17 pgs: 17 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T12:48:13.767 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 3 15 11550 11535 240.29 239.375 0.00216241 0.00415446 2024-05-15T12:48:14.478 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T12:48:14.478 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T12:48:14.478 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T12:48:14.478 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T12:48:14.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.479 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:6658 2.5 e115/110 2:a936a168:::benchmark_data_smithi177_78104_object6657:head v 115'370, mlcod=115'368)", 2024-05-15T12:48:14.479 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:12.513327+0000", 2024-05-15T12:48:14.479 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 1.9718958450000001, 2024-05-15T12:48:14.479 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0073214489999999998, 2024-05-15T12:48:14.479 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:14.479 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:14.479 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:14.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.479 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:14.479 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.513327+0000", 2024-05-15T12:48:14.479 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.480 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:14.480 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.513327+0000", 2024-05-15T12:48:14.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.480 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.480 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:14.480 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.513328+0000", 2024-05-15T12:48:14.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7199999999999999e-07 2024-05-15T12:48:14.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.480 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.480 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:14.480 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.513430+0000", 2024-05-15T12:48:14.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000102082 2024-05-15T12:48:14.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:14.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.513442+0000", 2024-05-15T12:48:14.481 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1816e-05 2024-05-15T12:48:14.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:14.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.513446+0000", 2024-05-15T12:48:14.481 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4360000000000004e-06 2024-05-15T12:48:14.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:14.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.513503+0000", 2024-05-15T12:48:14.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6378999999999997e-05 2024-05-15T12:48:14.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.482 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:14.482 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.513515+0000", 2024-05-15T12:48:14.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2697999999999999e-05 2024-05-15T12:48:14.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.482 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:14.482 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.520601+0000", 2024-05-15T12:48:14.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0070852729999999996 2024-05-15T12:48:14.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.483 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:14.483 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.520649+0000", 2024-05-15T12:48:14.483 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8192999999999998e-05 2024-05-15T12:48:14.483 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.483 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.483 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.483 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:8407 2.1 e115/110 2:8fae52de:::benchmark_data_smithi177_78104_object8406:head v 115'549, mlcod=115'547)", 2024-05-15T12:48:14.483 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:12.950405+0000", 2024-05-15T12:48:14.483 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 1.534818572, 2024-05-15T12:48:14.483 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0068986280000000004, 2024-05-15T12:48:14.483 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:14.483 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:14.484 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:14.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.484 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:14.484 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.950405+0000", 2024-05-15T12:48:14.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.484 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:14.484 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.950405+0000", 2024-05-15T12:48:14.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.484 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:14.484 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.950405+0000", 2024-05-15T12:48:14.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6899999999999997e-07 2024-05-15T12:48:14.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:14.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.950431+0000", 2024-05-15T12:48:14.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5329000000000001e-05 2024-05-15T12:48:14.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:14.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.950443+0000", 2024-05-15T12:48:14.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2061e-05 2024-05-15T12:48:14.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:14.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.950449+0000", 2024-05-15T12:48:14.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2759999999999997e-06 2024-05-15T12:48:14.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:14.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.950600+0000", 2024-05-15T12:48:14.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015056800000000001 2024-05-15T12:48:14.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:14.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.950619+0000", 2024-05-15T12:48:14.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.982e-05 2024-05-15T12:48:14.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:14.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.957251+0000", 2024-05-15T12:48:14.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0066312389999999997 2024-05-15T12:48:14.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:14.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.957303+0000", 2024-05-15T12:48:14.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2666e-05 2024-05-15T12:48:14.487 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.487 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.487 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.488 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:10087 2.5 e115/110 2:aaa7f0c3:::benchmark_data_smithi177_78104_object10086:head v 115'588, mlcod=115'586)", 2024-05-15T12:48:14.488 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:13.386523+0000", 2024-05-15T12:48:14.488 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 1.098700555, 2024-05-15T12:48:14.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0069207219999999998, 2024-05-15T12:48:14.488 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:14.488 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:14.488 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:14.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:14.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.386523+0000", 2024-05-15T12:48:14.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:14.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.386523+0000", 2024-05-15T12:48:14.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:14.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.386523+0000", 2024-05-15T12:48:14.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8999999999999995e-07 2024-05-15T12:48:14.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:14.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.386717+0000", 2024-05-15T12:48:14.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00019357099999999999 2024-05-15T12:48:14.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:14.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.386732+0000", 2024-05-15T12:48:14.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4790000000000001e-05 2024-05-15T12:48:14.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:14.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.386740+0000", 2024-05-15T12:48:14.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0469999999999994e-06 2024-05-15T12:48:14.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:14.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.386810+0000", 2024-05-15T12:48:14.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0519999999999996e-05 2024-05-15T12:48:14.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:14.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.386861+0000", 2024-05-15T12:48:14.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0553999999999997e-05 2024-05-15T12:48:14.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.491 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:14.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.393384+0000", 2024-05-15T12:48:14.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0065227940000000002 2024-05-15T12:48:14.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.491 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:14.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.393443+0000", 2024-05-15T12:48:14.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9655999999999999e-05 2024-05-15T12:48:14.491 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.491 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.491 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.492 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:10787 2.c e115/110 2:329e7f77:::benchmark_data_smithi177_78104_object10786:head v 115'682, mlcod=115'679)", 2024-05-15T12:48:14.492 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:13.566578+0000", 2024-05-15T12:48:14.492 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.91864550899999997, 2024-05-15T12:48:14.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0075447250000000004, 2024-05-15T12:48:14.492 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:14.492 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:14.492 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:14.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:14.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.566578+0000", 2024-05-15T12:48:14.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.493 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:14.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.566578+0000", 2024-05-15T12:48:14.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.493 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:14.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.566579+0000", 2024-05-15T12:48:14.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1500000000000003e-07 2024-05-15T12:48:14.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.493 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:14.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.566657+0000", 2024-05-15T12:48:14.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8421e-05 2024-05-15T12:48:14.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.566672+0000", 2024-05-15T12:48:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5421e-05 2024-05-15T12:48:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.566680+0000", 2024-05-15T12:48:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5059999999999998e-06 2024-05-15T12:48:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.566917+0000", 2024-05-15T12:48:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00023749900000000001 2024-05-15T12:48:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.566935+0000", 2024-05-15T12:48:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8085e-05 2024-05-15T12:48:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.574059+0000", 2024-05-15T12:48:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0071240110000000004 2024-05-15T12:48:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.574122+0000", 2024-05-15T12:48:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2966999999999995e-05 2024-05-15T12:48:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:10802 2.c e115/110 2:33b4c310:::benchmark_data_smithi177_78104_object10801:head v 115'684, mlcod=115'680)", 2024-05-15T12:48:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:13.570819+0000", 2024-05-15T12:48:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.91440444200000004, 2024-05-15T12:48:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090015330000000008, 2024-05-15T12:48:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.570819+0000", 2024-05-15T12:48:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.570819+0000", 2024-05-15T12:48:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.570820+0000", 2024-05-15T12:48:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5499999999999997e-07 2024-05-15T12:48:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.570919+0000", 2024-05-15T12:48:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9419999999999993e-05 2024-05-15T12:48:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.570928+0000", 2024-05-15T12:48:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9020000000000005e-06 2024-05-15T12:48:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.570933+0000", 2024-05-15T12:48:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3090000000000003e-06 2024-05-15T12:48:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.571822+0000", 2024-05-15T12:48:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00088850799999999998 2024-05-15T12:48:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.571840+0000", 2024-05-15T12:48:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.859e-05 2024-05-15T12:48:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.579765+0000", 2024-05-15T12:48:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0079244959999999996 2024-05-15T12:48:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:13.579820+0000", 2024-05-15T12:48:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5452999999999999e-05 2024-05-15T12:48:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:12714 2.c e115/110 2:3a3c3ced:::benchmark_data_smithi177_78104_object12713:head v 115'813, mlcod=115'811)", 2024-05-15T12:48:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:14.242562+0000", 2024-05-15T12:48:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.242661667, 2024-05-15T12:48:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090746460000000004, 2024-05-15T12:48:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.242562+0000", 2024-05-15T12:48:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.242562+0000", 2024-05-15T12:48:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.242562+0000", 2024-05-15T12:48:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9399999999999995e-07 2024-05-15T12:48:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.247175+0000", 2024-05-15T12:48:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046130750000000003 2024-05-15T12:48:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.247188+0000", 2024-05-15T12:48:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2441e-05 2024-05-15T12:48:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.247196+0000", 2024-05-15T12:48:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9920000000000007e-06 2024-05-15T12:48:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.247249+0000", 2024-05-15T12:48:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3454e-05 2024-05-15T12:48:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:14.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.247269+0000", 2024-05-15T12:48:14.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0290000000000001e-05 2024-05-15T12:48:14.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:14.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.251604+0000", 2024-05-15T12:48:14.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0043345409999999999 2024-05-15T12:48:14.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:14.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.251636+0000", 2024-05-15T12:48:14.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2359e-05 2024-05-15T12:48:14.505 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.505 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.505 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.505 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:12941 2.8 e115/110 2:1f1f6f70:::benchmark_data_smithi177_78104_object12940:head v 115'866, mlcod=115'864)", 2024-05-15T12:48:14.505 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:14.318127+0000", 2024-05-15T12:48:14.505 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.16709597300000001, 2024-05-15T12:48:14.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010223439000000001, 2024-05-15T12:48:14.505 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:14.505 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:14.505 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:14.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:14.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.318127+0000", 2024-05-15T12:48:14.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:14.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.318127+0000", 2024-05-15T12:48:14.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:14.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.318128+0000", 2024-05-15T12:48:14.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6800000000000001e-07 2024-05-15T12:48:14.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:14.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.318170+0000", 2024-05-15T12:48:14.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2386999999999999e-05 2024-05-15T12:48:14.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:14.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.318179+0000", 2024-05-15T12:48:14.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8650000000000007e-06 2024-05-15T12:48:14.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:14.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.318185+0000", 2024-05-15T12:48:14.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.105e-06 2024-05-15T12:48:14.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.508 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:14.508 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.318253+0000", 2024-05-15T12:48:14.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8138000000000004e-05 2024-05-15T12:48:14.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.508 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:14.508 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.318276+0000", 2024-05-15T12:48:14.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2906000000000001e-05 2024-05-15T12:48:14.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.508 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:14.508 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.328290+0000", 2024-05-15T12:48:14.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010013404 2024-05-15T12:48:14.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:14.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.328351+0000", 2024-05-15T12:48:14.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1166000000000004e-05 2024-05-15T12:48:14.509 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.509 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.509 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.509 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:12935 2.4 e115/110 2:2f6292ff:::benchmark_data_smithi177_78104_object12934:head v 115'776, mlcod=115'773)", 2024-05-15T12:48:14.509 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:14.318241+0000", 2024-05-15T12:48:14.509 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.166982724, 2024-05-15T12:48:14.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011318933, 2024-05-15T12:48:14.509 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:14.510 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:14.510 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:14.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:14.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.318241+0000", 2024-05-15T12:48:14.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:14.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.318241+0000", 2024-05-15T12:48:14.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:14.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.318241+0000", 2024-05-15T12:48:14.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7400000000000003e-07 2024-05-15T12:48:14.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:14.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.318265+0000", 2024-05-15T12:48:14.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4206000000000002e-05 2024-05-15T12:48:14.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:14.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.318275+0000", 2024-05-15T12:48:14.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5689999999999994e-06 2024-05-15T12:48:14.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:14.512 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.318281+0000", 2024-05-15T12:48:14.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3280000000000002e-06 2024-05-15T12:48:14.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:14.512 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.318369+0000", 2024-05-15T12:48:14.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7975999999999999e-05 2024-05-15T12:48:14.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:14.512 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.318390+0000", 2024-05-15T12:48:14.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0378e-05 2024-05-15T12:48:14.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:14.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.329479+0000", 2024-05-15T12:48:14.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011089477 2024-05-15T12:48:14.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:14.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.329559+0000", 2024-05-15T12:48:14.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0325000000000001e-05 2024-05-15T12:48:14.513 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.513 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.513 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.513 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:12936 2.4 e115/110 2:2014b124:::benchmark_data_smithi177_78104_object12935:head v 115'777, mlcod=115'773)", 2024-05-15T12:48:14.513 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:14.318293+0000", 2024-05-15T12:48:14.514 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.16692997500000001, 2024-05-15T12:48:14.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.01130832, 2024-05-15T12:48:14.514 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:14.514 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:14.514 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:14.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.514 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:14.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.318293+0000", 2024-05-15T12:48:14.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.514 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:14.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.318293+0000", 2024-05-15T12:48:14.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:14.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.318294+0000", 2024-05-15T12:48:14.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.58e-07 2024-05-15T12:48:14.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:14.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.318316+0000", 2024-05-15T12:48:14.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2212e-05 2024-05-15T12:48:14.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:14.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.318325+0000", 2024-05-15T12:48:14.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0939999999999997e-06 2024-05-15T12:48:14.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:14.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.318331+0000", 2024-05-15T12:48:14.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0599999999999996e-06 2024-05-15T12:48:14.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:14.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.318899+0000", 2024-05-15T12:48:14.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00056773800000000003 2024-05-15T12:48:14.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.517 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:14.517 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.318917+0000", 2024-05-15T12:48:14.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8502e-05 2024-05-15T12:48:14.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.517 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:14.517 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.329569+0000", 2024-05-15T12:48:14.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010651561 2024-05-15T12:48:14.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.517 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:14.517 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.329602+0000", 2024-05-15T12:48:14.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2795e-05 2024-05-15T12:48:14.518 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.518 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.518 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.518 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:12953 2.8 e115/110 2:145a7a04:::benchmark_data_smithi177_78104_object12952:head v 115'867, mlcod=115'864)", 2024-05-15T12:48:14.518 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:14.319612+0000", 2024-05-15T12:48:14.518 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.16561135099999999, 2024-05-15T12:48:14.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0094931000000000008, 2024-05-15T12:48:14.518 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:14.518 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:14.518 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:14.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:14.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.319612+0000", 2024-05-15T12:48:14.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:14.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.319612+0000", 2024-05-15T12:48:14.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:14.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.319613+0000", 2024-05-15T12:48:14.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8400000000000004e-07 2024-05-15T12:48:14.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:14.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.319729+0000", 2024-05-15T12:48:14.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011606499999999999 2024-05-15T12:48:14.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:14.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.319742+0000", 2024-05-15T12:48:14.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3799e-05 2024-05-15T12:48:14.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:14.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.319749+0000", 2024-05-15T12:48:14.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9369999999999998e-06 2024-05-15T12:48:14.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:14.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.319822+0000", 2024-05-15T12:48:14.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2563000000000003e-05 2024-05-15T12:48:14.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:14.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.319842+0000", 2024-05-15T12:48:14.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9905e-05 2024-05-15T12:48:14.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:14.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.329069+0000", 2024-05-15T12:48:14.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0092269670000000008 2024-05-15T12:48:14.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.522 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.522 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:14.522 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.329105+0000", 2024-05-15T12:48:14.522 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6180000000000003e-05 2024-05-15T12:48:14.522 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.522 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.522 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.522 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.522 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:12944 2.4 e115/110 2:260789f1:::benchmark_data_smithi177_78104_object12943:head v 115'778, mlcod=115'774)", 2024-05-15T12:48:14.522 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:14.321556+0000", 2024-05-15T12:48:14.522 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.16366685, 2024-05-15T12:48:14.522 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0080733690000000004, 2024-05-15T12:48:14.523 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:14.523 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:14.523 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:14.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:14.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.321556+0000", 2024-05-15T12:48:14.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:14.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.321556+0000", 2024-05-15T12:48:14.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:14.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.321558+0000", 2024-05-15T12:48:14.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.172e-06 2024-05-15T12:48:14.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:14.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.321608+0000", 2024-05-15T12:48:14.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0519000000000002e-05 2024-05-15T12:48:14.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:14.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.321635+0000", 2024-05-15T12:48:14.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7379e-05 2024-05-15T12:48:14.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:14.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.321644+0000", 2024-05-15T12:48:14.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1890000000000007e-06 2024-05-15T12:48:14.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:14.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.321724+0000", 2024-05-15T12:48:14.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0593999999999994e-05 2024-05-15T12:48:14.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:14.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.321742+0000", 2024-05-15T12:48:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7456999999999999e-05 2024-05-15T12:48:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.329604+0000", 2024-05-15T12:48:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0078626200000000007 2024-05-15T12:48:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.329630+0000", 2024-05-15T12:48:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5439000000000002e-05 2024-05-15T12:48:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.526 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.527 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:13363 2.4 e115/110 2:26a8221b:::benchmark_data_smithi177_78104_object13362:head v 115'805, mlcod=115'803)", 2024-05-15T12:48:14.527 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:14.455510+0000", 2024-05-15T12:48:14.527 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.029713148000000002, 2024-05-15T12:48:14.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011718444999999999, 2024-05-15T12:48:14.527 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:14.527 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:14.527 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:14.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.527 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:14.527 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.455510+0000", 2024-05-15T12:48:14.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:14.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.455510+0000", 2024-05-15T12:48:14.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:14.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.455511+0000", 2024-05-15T12:48:14.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7100000000000001e-07 2024-05-15T12:48:14.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:14.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.455551+0000", 2024-05-15T12:48:14.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9974e-05 2024-05-15T12:48:14.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:14.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.455564+0000", 2024-05-15T12:48:14.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3441000000000001e-05 2024-05-15T12:48:14.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:14.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.455572+0000", 2024-05-15T12:48:14.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9540000000000004e-06 2024-05-15T12:48:14.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:14.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.455633+0000", 2024-05-15T12:48:14.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0933000000000001e-05 2024-05-15T12:48:14.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:14.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.455653+0000", 2024-05-15T12:48:14.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9752e-05 2024-05-15T12:48:14.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:14.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.467174+0000", 2024-05-15T12:48:14.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.01152125 2024-05-15T12:48:14.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:14.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.467229+0000", 2024-05-15T12:48:14.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4370000000000003e-05 2024-05-15T12:48:14.531 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.531 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.531 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.531 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:13352 2.c e115/110 2:3f931087:::benchmark_data_smithi177_78104_object13351:head v 115'851, mlcod=115'848)", 2024-05-15T12:48:14.531 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:14.457015+0000", 2024-05-15T12:48:14.531 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.028208502, 2024-05-15T12:48:14.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010512674, 2024-05-15T12:48:14.532 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:14.532 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:14.532 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:14.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:14.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.457015+0000", 2024-05-15T12:48:14.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:14.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.457015+0000", 2024-05-15T12:48:14.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:14.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.457015+0000", 2024-05-15T12:48:14.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.37e-07 2024-05-15T12:48:14.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:14.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.457067+0000", 2024-05-15T12:48:14.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1251000000000001e-05 2024-05-15T12:48:14.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:14.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.457079+0000", 2024-05-15T12:48:14.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2838e-05 2024-05-15T12:48:14.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:14.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.457087+0000", 2024-05-15T12:48:14.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5909999999999998e-06 2024-05-15T12:48:14.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:14.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.457232+0000", 2024-05-15T12:48:14.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014540700000000001 2024-05-15T12:48:14.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:14.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.457270+0000", 2024-05-15T12:48:14.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7944000000000001e-05 2024-05-15T12:48:14.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:14.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.467478+0000", 2024-05-15T12:48:14.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010207759 2024-05-15T12:48:14.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:14.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.467527+0000", 2024-05-15T12:48:14.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9246999999999998e-05 2024-05-15T12:48:14.535 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.535 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.535 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.536 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:13355 2.c e115/110 2:3c6ff1cd:::benchmark_data_smithi177_78104_object13354:head v 115'852, mlcod=115'848)", 2024-05-15T12:48:14.536 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:14.457107+0000", 2024-05-15T12:48:14.536 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.028116717999999999, 2024-05-15T12:48:14.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010454328000000001, 2024-05-15T12:48:14.536 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:14.536 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:14.536 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:14.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.536 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:14.536 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.457107+0000", 2024-05-15T12:48:14.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:14.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.457107+0000", 2024-05-15T12:48:14.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:14.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.457107+0000", 2024-05-15T12:48:14.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3099999999999999e-07 2024-05-15T12:48:14.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:14.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.457139+0000", 2024-05-15T12:48:14.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2366000000000001e-05 2024-05-15T12:48:14.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:14.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.457149+0000", 2024-05-15T12:48:14.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3729999999999999e-06 2024-05-15T12:48:14.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:14.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.457155+0000", 2024-05-15T12:48:14.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0360000000000003e-06 2024-05-15T12:48:14.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:14.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.457884+0000", 2024-05-15T12:48:14.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00072907199999999999 2024-05-15T12:48:14.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:14.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.457905+0000", 2024-05-15T12:48:14.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0931999999999999e-05 2024-05-15T12:48:14.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:14.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.467530+0000", 2024-05-15T12:48:14.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0096254099999999992 2024-05-15T12:48:14.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:14.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.467561+0000", 2024-05-15T12:48:14.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0808e-05 2024-05-15T12:48:14.540 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.540 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.540 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.540 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:13370 2.5 e115/110 2:a54f0567:::benchmark_data_smithi177_78104_object13369:head v 115'789, mlcod=115'785)", 2024-05-15T12:48:14.540 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:14.458393+0000", 2024-05-15T12:48:14.540 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.026829950000000002, 2024-05-15T12:48:14.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0091706879999999998, 2024-05-15T12:48:14.540 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:14.540 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:14.540 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:14.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:14.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.458393+0000", 2024-05-15T12:48:14.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:14.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.458393+0000", 2024-05-15T12:48:14.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:14.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.458394+0000", 2024-05-15T12:48:14.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3300000000000002e-07 2024-05-15T12:48:14.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:14.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.458441+0000", 2024-05-15T12:48:14.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6989e-05 2024-05-15T12:48:14.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:14.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.458465+0000", 2024-05-15T12:48:14.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4088999999999999e-05 2024-05-15T12:48:14.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:14.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.458474+0000", 2024-05-15T12:48:14.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3659999999999998e-06 2024-05-15T12:48:14.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:14.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.458536+0000", 2024-05-15T12:48:14.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1939000000000004e-05 2024-05-15T12:48:14.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:14.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.458570+0000", 2024-05-15T12:48:14.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3411000000000002e-05 2024-05-15T12:48:14.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:14.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.467524+0000", 2024-05-15T12:48:14.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0089546130000000002 2024-05-15T12:48:14.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:14.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.467564+0000", 2024-05-15T12:48:14.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9647999999999997e-05 2024-05-15T12:48:14.544 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.544 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.544 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.544 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:13371 2.5 e115/110 2:ad2df9cc:::benchmark_data_smithi177_78104_object13370:head v 115'790, mlcod=115'785)", 2024-05-15T12:48:14.545 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:14.458498+0000", 2024-05-15T12:48:14.545 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.026724784000000001, 2024-05-15T12:48:14.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090913060000000004, 2024-05-15T12:48:14.545 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:14.545 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:14.545 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:14.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:14.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.458498+0000", 2024-05-15T12:48:14.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:14.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.458498+0000", 2024-05-15T12:48:14.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:14.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.458499+0000", 2024-05-15T12:48:14.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8200000000000001e-07 2024-05-15T12:48:14.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:14.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.458533+0000", 2024-05-15T12:48:14.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4267000000000002e-05 2024-05-15T12:48:14.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:14.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.458550+0000", 2024-05-15T12:48:14.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6603999999999999e-05 2024-05-15T12:48:14.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:14.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.458557+0000", 2024-05-15T12:48:14.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9970000000000001e-06 2024-05-15T12:48:14.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:14.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.459068+0000", 2024-05-15T12:48:14.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00051153699999999995 2024-05-15T12:48:14.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:14.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.459089+0000", 2024-05-15T12:48:14.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0632000000000002e-05 2024-05-15T12:48:14.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:14.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.467567+0000", 2024-05-15T12:48:14.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0084784720000000008 2024-05-15T12:48:14.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.549 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:14.549 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.467590+0000", 2024-05-15T12:48:14.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2415000000000001e-05 2024-05-15T12:48:14.549 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.549 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.549 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.549 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:13368 2.4 e115/110 2:22793b36:::benchmark_data_smithi177_78104_object13367:head v 115'806, mlcod=115'803)", 2024-05-15T12:48:14.549 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:14.458591+0000", 2024-05-15T12:48:14.549 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.026632119999999999, 2024-05-15T12:48:14.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090007850000000007, 2024-05-15T12:48:14.549 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:14.549 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:14.550 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:14.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:14.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.458591+0000", 2024-05-15T12:48:14.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:14.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.458591+0000", 2024-05-15T12:48:14.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:14.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.458592+0000", 2024-05-15T12:48:14.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0399999999999996e-07 2024-05-15T12:48:14.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:14.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.458615+0000", 2024-05-15T12:48:14.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3022000000000001e-05 2024-05-15T12:48:14.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:14.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.458624+0000", 2024-05-15T12:48:14.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5599999999999999e-06 2024-05-15T12:48:14.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:14.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.458630+0000", 2024-05-15T12:48:14.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1319999999999999e-06 2024-05-15T12:48:14.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:14.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.458746+0000", 2024-05-15T12:48:14.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000115423 2024-05-15T12:48:14.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:14.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.458783+0000", 2024-05-15T12:48:14.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7069e-05 2024-05-15T12:48:14.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:14.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.467564+0000", 2024-05-15T12:48:14.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0087810969999999999 2024-05-15T12:48:14.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:14.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.467592+0000", 2024-05-15T12:48:14.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7977999999999999e-05 2024-05-15T12:48:14.553 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.553 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.553 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.553 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:13377 2.c e115/110 2:3940eb1e:::benchmark_data_smithi177_78104_object13376:head v 115'853, mlcod=115'849)", 2024-05-15T12:48:14.554 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:14.459310+0000", 2024-05-15T12:48:14.554 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.025913093000000002, 2024-05-15T12:48:14.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0083137630000000001, 2024-05-15T12:48:14.554 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:14.554 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:14.554 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:14.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:14.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.459310+0000", 2024-05-15T12:48:14.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:14.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.459310+0000", 2024-05-15T12:48:14.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:14.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.459311+0000", 2024-05-15T12:48:14.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0500000000000003e-07 2024-05-15T12:48:14.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:14.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.459344+0000", 2024-05-15T12:48:14.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2925000000000001e-05 2024-05-15T12:48:14.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:14.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.459356+0000", 2024-05-15T12:48:14.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2139e-05 2024-05-15T12:48:14.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:14.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.459364+0000", 2024-05-15T12:48:14.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5730000000000008e-06 2024-05-15T12:48:14.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:14.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.459449+0000", 2024-05-15T12:48:14.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4394999999999994e-05 2024-05-15T12:48:14.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:14.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.459471+0000", 2024-05-15T12:48:14.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2121e-05 2024-05-15T12:48:14.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:14.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.467594+0000", 2024-05-15T12:48:14.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0081227250000000008 2024-05-15T12:48:14.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:14.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.467624+0000", 2024-05-15T12:48:14.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0179999999999999e-05 2024-05-15T12:48:14.558 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.558 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.558 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.558 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:13369 2.4 e115/110 2:241ecf82:::benchmark_data_smithi177_78104_object13368:head v 115'807, mlcod=115'803)", 2024-05-15T12:48:14.558 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:14.460532+0000", 2024-05-15T12:48:14.558 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.024690798, 2024-05-15T12:48:14.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0071198119999999997, 2024-05-15T12:48:14.558 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:14.558 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:14.558 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:14.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:14.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.460532+0000", 2024-05-15T12:48:14.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:14.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.460532+0000", 2024-05-15T12:48:14.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:14.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.460533+0000", 2024-05-15T12:48:14.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4600000000000004e-07 2024-05-15T12:48:14.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:14.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.460561+0000", 2024-05-15T12:48:14.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7483000000000001e-05 2024-05-15T12:48:14.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:14.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.460574+0000", 2024-05-15T12:48:14.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3189e-05 2024-05-15T12:48:14.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:14.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.460582+0000", 2024-05-15T12:48:14.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6920000000000002e-06 2024-05-15T12:48:14.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:14.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.460665+0000", 2024-05-15T12:48:14.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3681000000000001e-05 2024-05-15T12:48:14.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:14.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.460724+0000", 2024-05-15T12:48:14.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9011999999999997e-05 2024-05-15T12:48:14.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:14.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.467626+0000", 2024-05-15T12:48:14.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.006901524 2024-05-15T12:48:14.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:14.562 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.467652+0000", 2024-05-15T12:48:14.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6485000000000001e-05 2024-05-15T12:48:14.562 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.562 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.562 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.562 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:13381 2.8 e115/110 2:1ae58552:::benchmark_data_smithi177_78104_object13380:head v 115'897, mlcod=115'895)", 2024-05-15T12:48:14.562 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:14.461133+0000", 2024-05-15T12:48:14.562 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.024090113999999999, 2024-05-15T12:48:14.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0071008080000000001, 2024-05-15T12:48:14.562 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:14.562 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:14.563 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:14.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:14.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.461133+0000", 2024-05-15T12:48:14.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:14.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.461133+0000", 2024-05-15T12:48:14.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:14.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:14.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.461134+0000", 2024-05-15T12:48:14.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5000000000000002e-07 2024-05-15T12:48:14.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:14.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.461236+0000", 2024-05-15T12:48:14.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000101903 2024-05-15T12:48:14.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:14.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.461244+0000", 2024-05-15T12:48:14.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3229999999999998e-06 2024-05-15T12:48:14.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:14.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.461249+0000", 2024-05-15T12:48:14.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1229999999999999e-06 2024-05-15T12:48:14.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:14.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.461299+0000", 2024-05-15T12:48:14.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9858999999999998e-05 2024-05-15T12:48:14.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:14.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.461318+0000", 2024-05-15T12:48:14.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9369999999999999e-05 2024-05-15T12:48:14.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:14.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.468190+0000", 2024-05-15T12:48:14.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0068720279999999996 2024-05-15T12:48:14.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:14.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:14.468234+0000", 2024-05-15T12:48:14.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3451999999999997e-05 2024-05-15T12:48:14.566 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.566 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.566 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.566 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.566 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.566 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T12:48:14.621 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:48:14.621 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T12:48:14.621 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 115, 2024-05-15T12:48:14.621 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "deaea04e-12b7-11ef-bc98-c7b262605968", 2024-05-15T12:48:14.621 INFO:teuthology.orchestra.run.smithi027.stdout: "created": "2024-05-15T12:38:21.244556+0000", 2024-05-15T12:48:14.621 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-05-15T12:48:08.824732+0000", 2024-05-15T12:48:14.621 INFO:teuthology.orchestra.run.smithi027.stdout: "last_up_change": "2024-05-15T12:45:50.876090+0000", 2024-05-15T12:48:14.621 INFO:teuthology.orchestra.run.smithi027.stdout: "last_in_change": "2024-05-15T12:45:33.158047+0000", 2024-05-15T12:48:14.622 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-15T12:48:14.622 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_num": 5801984, 2024-05-15T12:48:14.622 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_set": [ 2024-05-15T12:48:14.622 INFO:teuthology.orchestra.run.smithi027.stdout: "noscrub", 2024-05-15T12:48:14.622 INFO:teuthology.orchestra.run.smithi027.stdout: "pglog_hardlimit", 2024-05-15T12:48:14.622 INFO:teuthology.orchestra.run.smithi027.stdout: "purged_snapdirs", 2024-05-15T12:48:14.622 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_deletes", 2024-05-15T12:48:14.622 INFO:teuthology.orchestra.run.smithi027.stdout: "sortbitwise" 2024-05-15T12:48:14.622 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T12:48:14.622 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_version": 26, 2024-05-15T12:48:14.622 INFO:teuthology.orchestra.run.smithi027.stdout: "full_ratio": 0.94999998807907104, 2024-05-15T12:48:14.622 INFO:teuthology.orchestra.run.smithi027.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-05-15T12:48:14.622 INFO:teuthology.orchestra.run.smithi027.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-15T12:48:14.623 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_snapshot": "", 2024-05-15T12:48:14.623 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_max": 2, 2024-05-15T12:48:14.623 INFO:teuthology.orchestra.run.smithi027.stdout: "max_osd": 12, 2024-05-15T12:48:14.623 INFO:teuthology.orchestra.run.smithi027.stdout: "require_min_compat_client": "luminous", 2024-05-15T12:48:14.623 INFO:teuthology.orchestra.run.smithi027.stdout: "min_compat_client": "jewel", 2024-05-15T12:48:14.623 INFO:teuthology.orchestra.run.smithi027.stdout: "require_osd_release": "squid", 2024-05-15T12:48:14.623 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_crimson": false, 2024-05-15T12:48:14.623 INFO:teuthology.orchestra.run.smithi027.stdout: "pools": [ 2024-05-15T12:48:14.623 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.623 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 1, 2024-05-15T12:48:14.623 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": ".mgr", 2024-05-15T12:48:14.623 INFO:teuthology.orchestra.run.smithi027.stdout: "create_time": "2024-05-15T12:41:57.948613+0000", 2024-05-15T12:48:14.623 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-15T12:48:14.623 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_names": "hashpspool", 2024-05-15T12:48:14.624 INFO:teuthology.orchestra.run.smithi027.stdout: "type": 1, 2024-05-15T12:48:14.624 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 3, 2024-05-15T12:48:14.624 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": 2, 2024-05-15T12:48:14.624 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_rule": 0, 2024-05-15T12:48:14.624 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_count": 0, 2024-05-15T12:48:14.624 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_target": 0, 2024-05-15T12:48:14.624 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T12:48:14.624 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T12:48:14.624 INFO:teuthology.orchestra.run.smithi027.stdout: "object_hash": 2, 2024-05-15T12:48:14.624 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscale_mode": "off", 2024-05-15T12:48:14.624 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": 1, 2024-05-15T12:48:14.624 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num": 1, 2024-05-15T12:48:14.624 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num_target": 1, 2024-05-15T12:48:14.625 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_target": 1, 2024-05-15T12:48:14.625 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_pending": 1, 2024-05-15T12:48:14.625 INFO:teuthology.orchestra.run.smithi027.stdout: "last_pg_merge_meta": { 2024-05-15T12:48:14.625 INFO:teuthology.orchestra.run.smithi027.stdout: "source_pgid": "0.0", 2024-05-15T12:48:14.625 INFO:teuthology.orchestra.run.smithi027.stdout: "ready_epoch": 0, 2024-05-15T12:48:14.625 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_started": 0, 2024-05-15T12:48:14.625 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_clean": 0, 2024-05-15T12:48:14.625 INFO:teuthology.orchestra.run.smithi027.stdout: "source_version": "0'0", 2024-05-15T12:48:14.625 INFO:teuthology.orchestra.run.smithi027.stdout: "target_version": "0'0" 2024-05-15T12:48:14.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.625 INFO:teuthology.orchestra.run.smithi027.stdout: "last_change": "21", 2024-05-15T12:48:14.625 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend": "0", 2024-05-15T12:48:14.625 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T12:48:14.625 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T12:48:14.626 INFO:teuthology.orchestra.run.smithi027.stdout: "auid": 0, 2024-05-15T12:48:14.626 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_mode": "selfmanaged", 2024-05-15T12:48:14.626 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_seq": 0, 2024-05-15T12:48:14.626 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_epoch": 0, 2024-05-15T12:48:14.626 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_snaps": [], 2024-05-15T12:48:14.626 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps": "[]", 2024-05-15T12:48:14.626 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_bytes": 0, 2024-05-15T12:48:14.626 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_objects": 0, 2024-05-15T12:48:14.626 INFO:teuthology.orchestra.run.smithi027.stdout: "tiers": [], 2024-05-15T12:48:14.626 INFO:teuthology.orchestra.run.smithi027.stdout: "tier_of": -1, 2024-05-15T12:48:14.626 INFO:teuthology.orchestra.run.smithi027.stdout: "read_tier": -1, 2024-05-15T12:48:14.626 INFO:teuthology.orchestra.run.smithi027.stdout: "write_tier": -1, 2024-05-15T12:48:14.626 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_mode": "none", 2024-05-15T12:48:14.627 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_bytes": 0, 2024-05-15T12:48:14.627 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_objects": 0, 2024-05-15T12:48:14.627 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T12:48:14.627 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T12:48:14.627 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T12:48:14.627 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_flush_age": 0, 2024-05-15T12:48:14.627 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_evict_age": 0, 2024-05-15T12:48:14.627 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profile": "", 2024-05-15T12:48:14.627 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_params": { 2024-05-15T12:48:14.627 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "none" 2024-05-15T12:48:14.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.627 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_period": 0, 2024-05-15T12:48:14.627 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_count": 0, 2024-05-15T12:48:14.627 INFO:teuthology.orchestra.run.smithi027.stdout: "use_gmt_hitset": true, 2024-05-15T12:48:14.628 INFO:teuthology.orchestra.run.smithi027.stdout: "min_read_recency_for_promote": 0, 2024-05-15T12:48:14.628 INFO:teuthology.orchestra.run.smithi027.stdout: "min_write_recency_for_promote": 0, 2024-05-15T12:48:14.628 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T12:48:14.628 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_search_last_n": 0, 2024-05-15T12:48:14.628 INFO:teuthology.orchestra.run.smithi027.stdout: "grade_table": [], 2024-05-15T12:48:14.628 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_width": 0, 2024-05-15T12:48:14.628 INFO:teuthology.orchestra.run.smithi027.stdout: "expected_num_objects": 0, 2024-05-15T12:48:14.628 INFO:teuthology.orchestra.run.smithi027.stdout: "fast_read": false, 2024-05-15T12:48:14.628 INFO:teuthology.orchestra.run.smithi027.stdout: "options": { 2024-05-15T12:48:14.628 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_max": 32, 2024-05-15T12:48:14.628 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_min": 1 2024-05-15T12:48:14.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.628 INFO:teuthology.orchestra.run.smithi027.stdout: "application_metadata": { 2024-05-15T12:48:14.629 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": {} 2024-05-15T12:48:14.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.629 INFO:teuthology.orchestra.run.smithi027.stdout: "read_balance": { 2024-05-15T12:48:14.629 INFO:teuthology.orchestra.run.smithi027.stdout: "score_type": "Fair distribution", 2024-05-15T12:48:14.629 INFO:teuthology.orchestra.run.smithi027.stdout: "score_acting": 12, 2024-05-15T12:48:14.629 INFO:teuthology.orchestra.run.smithi027.stdout: "score_stable": 12, 2024-05-15T12:48:14.629 INFO:teuthology.orchestra.run.smithi027.stdout: "optimal_score": 0.25, 2024-05-15T12:48:14.629 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_acting": 3, 2024-05-15T12:48:14.629 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_stable": 3, 2024-05-15T12:48:14.629 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-05-15T12:48:14.629 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity": 1, 2024-05-15T12:48:14.629 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity_weighted": 1 2024-05-15T12:48:14.629 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.630 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.630 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 2, 2024-05-15T12:48:14.630 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": "unique_pool_0", 2024-05-15T12:48:14.630 INFO:teuthology.orchestra.run.smithi027.stdout: "create_time": "2024-05-15T12:47:57.372152+0000", 2024-05-15T12:48:14.630 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-15T12:48:14.630 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_names": "hashpspool", 2024-05-15T12:48:14.630 INFO:teuthology.orchestra.run.smithi027.stdout: "type": 1, 2024-05-15T12:48:14.630 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 3, 2024-05-15T12:48:14.630 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": 2, 2024-05-15T12:48:14.630 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_rule": 0, 2024-05-15T12:48:14.630 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_count": 0, 2024-05-15T12:48:14.630 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_target": 0, 2024-05-15T12:48:14.630 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T12:48:14.631 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T12:48:14.631 INFO:teuthology.orchestra.run.smithi027.stdout: "object_hash": 2, 2024-05-15T12:48:14.631 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscale_mode": "off", 2024-05-15T12:48:14.631 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": 16, 2024-05-15T12:48:14.631 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num": 16, 2024-05-15T12:48:14.631 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num_target": 16, 2024-05-15T12:48:14.631 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_target": 16, 2024-05-15T12:48:14.631 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_pending": 16, 2024-05-15T12:48:14.631 INFO:teuthology.orchestra.run.smithi027.stdout: "last_pg_merge_meta": { 2024-05-15T12:48:14.631 INFO:teuthology.orchestra.run.smithi027.stdout: "source_pgid": "0.0", 2024-05-15T12:48:14.631 INFO:teuthology.orchestra.run.smithi027.stdout: "ready_epoch": 0, 2024-05-15T12:48:14.631 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_started": 0, 2024-05-15T12:48:14.631 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_clean": 0, 2024-05-15T12:48:14.632 INFO:teuthology.orchestra.run.smithi027.stdout: "source_version": "0'0", 2024-05-15T12:48:14.632 INFO:teuthology.orchestra.run.smithi027.stdout: "target_version": "0'0" 2024-05-15T12:48:14.632 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.632 INFO:teuthology.orchestra.run.smithi027.stdout: "last_change": "115", 2024-05-15T12:48:14.632 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend": "0", 2024-05-15T12:48:14.632 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T12:48:14.632 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T12:48:14.632 INFO:teuthology.orchestra.run.smithi027.stdout: "auid": 0, 2024-05-15T12:48:14.632 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_mode": "selfmanaged", 2024-05-15T12:48:14.632 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_seq": 0, 2024-05-15T12:48:14.632 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_epoch": 0, 2024-05-15T12:48:14.632 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_snaps": [], 2024-05-15T12:48:14.632 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps": "[]", 2024-05-15T12:48:14.632 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_bytes": 0, 2024-05-15T12:48:14.633 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_objects": 0, 2024-05-15T12:48:14.633 INFO:teuthology.orchestra.run.smithi027.stdout: "tiers": [], 2024-05-15T12:48:14.633 INFO:teuthology.orchestra.run.smithi027.stdout: "tier_of": -1, 2024-05-15T12:48:14.633 INFO:teuthology.orchestra.run.smithi027.stdout: "read_tier": -1, 2024-05-15T12:48:14.633 INFO:teuthology.orchestra.run.smithi027.stdout: "write_tier": -1, 2024-05-15T12:48:14.633 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_mode": "none", 2024-05-15T12:48:14.633 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_bytes": 0, 2024-05-15T12:48:14.633 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_objects": 0, 2024-05-15T12:48:14.633 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T12:48:14.633 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T12:48:14.633 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T12:48:14.633 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_flush_age": 0, 2024-05-15T12:48:14.633 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_evict_age": 0, 2024-05-15T12:48:14.633 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profile": "", 2024-05-15T12:48:14.634 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_params": { 2024-05-15T12:48:14.634 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "none" 2024-05-15T12:48:14.634 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.634 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_period": 0, 2024-05-15T12:48:14.634 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_count": 0, 2024-05-15T12:48:14.634 INFO:teuthology.orchestra.run.smithi027.stdout: "use_gmt_hitset": true, 2024-05-15T12:48:14.634 INFO:teuthology.orchestra.run.smithi027.stdout: "min_read_recency_for_promote": 0, 2024-05-15T12:48:14.634 INFO:teuthology.orchestra.run.smithi027.stdout: "min_write_recency_for_promote": 0, 2024-05-15T12:48:14.634 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T12:48:14.634 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_search_last_n": 0, 2024-05-15T12:48:14.634 INFO:teuthology.orchestra.run.smithi027.stdout: "grade_table": [], 2024-05-15T12:48:14.634 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_width": 0, 2024-05-15T12:48:14.634 INFO:teuthology.orchestra.run.smithi027.stdout: "expected_num_objects": 0, 2024-05-15T12:48:14.634 INFO:teuthology.orchestra.run.smithi027.stdout: "fast_read": false, 2024-05-15T12:48:14.635 INFO:teuthology.orchestra.run.smithi027.stdout: "options": {}, 2024-05-15T12:48:14.635 INFO:teuthology.orchestra.run.smithi027.stdout: "application_metadata": { 2024-05-15T12:48:14.635 INFO:teuthology.orchestra.run.smithi027.stdout: "rados": {} 2024-05-15T12:48:14.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.635 INFO:teuthology.orchestra.run.smithi027.stdout: "read_balance": { 2024-05-15T12:48:14.635 INFO:teuthology.orchestra.run.smithi027.stdout: "score_type": "Fair distribution", 2024-05-15T12:48:14.635 INFO:teuthology.orchestra.run.smithi027.stdout: "score_acting": 2.25, 2024-05-15T12:48:14.635 INFO:teuthology.orchestra.run.smithi027.stdout: "score_stable": 2.25, 2024-05-15T12:48:14.635 INFO:teuthology.orchestra.run.smithi027.stdout: "optimal_score": 1, 2024-05-15T12:48:14.635 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_acting": 2.25, 2024-05-15T12:48:14.635 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_stable": 2.25, 2024-05-15T12:48:14.635 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-05-15T12:48:14.635 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity": 1, 2024-05-15T12:48:14.635 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity_weighted": 1 2024-05-15T12:48:14.635 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.636 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.636 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T12:48:14.636 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T12:48:14.636 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.636 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 0, 2024-05-15T12:48:14.636 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "64fb3ad2-08a7-4106-a07a-6c776c877faf", 2024-05-15T12:48:14.636 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:48:14.636 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:48:14.636 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:48:14.636 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:48:14.636 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:48:14.636 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:48:14.636 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 8, 2024-05-15T12:48:14.636 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 58, 2024-05-15T12:48:14.637 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:48:14.637 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:48:14.637 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:48:14.637 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.637 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.637 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.637 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6802", 2024-05-15T12:48:14.637 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:48:14.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.637 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.637 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.637 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6803", 2024-05-15T12:48:14.637 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:48:14.637 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.638 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.638 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.638 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:48:14.638 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.638 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.638 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.638 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6804", 2024-05-15T12:48:14.638 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:48:14.638 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.638 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.638 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.639 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6805", 2024-05-15T12:48:14.639 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:48:14.639 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.639 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.639 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.639 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:48:14.640 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.640 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.640 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.640 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6808", 2024-05-15T12:48:14.640 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:48:14.640 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.640 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.640 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.640 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6809", 2024-05-15T12:48:14.640 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:48:14.640 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.640 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.640 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.640 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:48:14.640 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.641 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.641 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.641 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6806", 2024-05-15T12:48:14.641 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:48:14.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.641 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.641 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.641 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6807", 2024-05-15T12:48:14.641 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:48:14.641 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.641 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.641 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6803/4239116935", 2024-05-15T12:48:14.641 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6805/4239116935", 2024-05-15T12:48:14.642 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6809/4239116935", 2024-05-15T12:48:14.642 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6807/4239116935", 2024-05-15T12:48:14.642 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:48:14.642 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:48:14.642 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:48:14.642 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.642 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.642 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 1, 2024-05-15T12:48:14.642 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "9c16dd90-9d51-48b8-a296-1c8879ba1bc8", 2024-05-15T12:48:14.642 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:48:14.642 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:48:14.642 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:48:14.642 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:48:14.643 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:48:14.643 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:48:14.643 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 13, 2024-05-15T12:48:14.643 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 110, 2024-05-15T12:48:14.643 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:48:14.643 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:48:14.643 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:48:14.643 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.643 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.643 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6810", 2024-05-15T12:48:14.643 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:48:14.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.644 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.644 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6811", 2024-05-15T12:48:14.644 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:48:14.644 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.644 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.644 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:48:14.644 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.644 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.644 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.644 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6812", 2024-05-15T12:48:14.644 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:48:14.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.644 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.644 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.645 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6813", 2024-05-15T12:48:14.645 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:48:14.645 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.645 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.645 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:48:14.645 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.645 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.645 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.645 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6816", 2024-05-15T12:48:14.645 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:48:14.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.645 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.645 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.645 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6817", 2024-05-15T12:48:14.646 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:48:14.646 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.646 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.646 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.646 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:48:14.646 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.646 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.646 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.646 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6814", 2024-05-15T12:48:14.646 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:48:14.646 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.646 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.646 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.646 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6815", 2024-05-15T12:48:14.647 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:48:14.647 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.647 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.647 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6811/1466270637", 2024-05-15T12:48:14.647 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6813/1466270637", 2024-05-15T12:48:14.647 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6817/1466270637", 2024-05-15T12:48:14.647 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6815/1466270637", 2024-05-15T12:48:14.647 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:48:14.647 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:48:14.647 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:48:14.647 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.647 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.648 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 2, 2024-05-15T12:48:14.648 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "cbe3ba13-3ea3-4891-8d2d-6b85581a732c", 2024-05-15T12:48:14.648 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:48:14.648 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:48:14.648 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:48:14.648 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:48:14.648 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:48:14.648 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:48:14.648 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 18, 2024-05-15T12:48:14.648 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 110, 2024-05-15T12:48:14.648 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:48:14.648 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:48:14.648 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:48:14.648 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.648 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.649 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.649 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6818", 2024-05-15T12:48:14.649 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:48:14.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.649 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.649 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.649 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6819", 2024-05-15T12:48:14.649 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:48:14.649 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.649 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.649 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:48:14.649 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.649 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.650 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.650 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6820", 2024-05-15T12:48:14.650 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:48:14.650 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.650 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.650 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.650 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6821", 2024-05-15T12:48:14.650 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:48:14.650 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.650 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.650 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.650 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:48:14.650 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.651 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.651 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.651 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6824", 2024-05-15T12:48:14.651 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:48:14.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.651 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.651 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.651 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6825", 2024-05-15T12:48:14.651 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:48:14.651 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.651 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.651 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:48:14.651 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.652 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.652 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.652 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6822", 2024-05-15T12:48:14.652 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:48:14.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.652 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.652 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.652 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6823", 2024-05-15T12:48:14.652 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:48:14.652 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.652 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.652 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6819/3285413897", 2024-05-15T12:48:14.653 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6821/3285413897", 2024-05-15T12:48:14.653 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6825/3285413897", 2024-05-15T12:48:14.653 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6823/3285413897", 2024-05-15T12:48:14.653 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:48:14.653 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:48:14.653 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:48:14.653 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.653 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.653 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 3, 2024-05-15T12:48:14.653 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "2e83735b-b944-4f87-8080-b994d4b96897", 2024-05-15T12:48:14.653 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:48:14.653 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:48:14.653 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:48:14.653 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:48:14.654 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:48:14.654 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:48:14.654 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 26, 2024-05-15T12:48:14.654 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 0, 2024-05-15T12:48:14.654 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:48:14.654 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:48:14.654 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:48:14.654 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.654 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.654 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.654 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6826", 2024-05-15T12:48:14.654 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:48:14.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.654 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.655 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.655 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6827", 2024-05-15T12:48:14.655 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:48:14.655 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.655 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.655 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:48:14.655 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.655 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.655 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.655 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6828", 2024-05-15T12:48:14.655 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:48:14.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.655 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.655 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.656 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6829", 2024-05-15T12:48:14.656 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:48:14.656 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.656 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.656 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:48:14.656 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.656 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.656 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.656 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6832", 2024-05-15T12:48:14.656 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:48:14.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.656 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.657 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.657 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6833", 2024-05-15T12:48:14.657 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:48:14.657 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.657 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.657 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:48:14.657 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.657 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.657 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.657 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6830", 2024-05-15T12:48:14.657 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:48:14.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.657 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.657 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.658 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6831", 2024-05-15T12:48:14.658 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:48:14.658 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.658 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.658 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6827/3008975244", 2024-05-15T12:48:14.658 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6829/3008975244", 2024-05-15T12:48:14.658 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6833/3008975244", 2024-05-15T12:48:14.658 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6831/3008975244", 2024-05-15T12:48:14.658 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:48:14.658 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:48:14.658 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:48:14.658 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.659 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.659 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 4, 2024-05-15T12:48:14.659 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "7c39c1dd-3867-46f7-8372-7688fa13b16a", 2024-05-15T12:48:14.659 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:48:14.659 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:48:14.659 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:48:14.659 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:48:14.659 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:48:14.659 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:48:14.659 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 31, 2024-05-15T12:48:14.659 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 0, 2024-05-15T12:48:14.659 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:48:14.659 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:48:14.659 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:48:14.660 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.660 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.660 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.660 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6800", 2024-05-15T12:48:14.660 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:48:14.660 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.660 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.660 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.660 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6801", 2024-05-15T12:48:14.660 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:48:14.660 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.660 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.660 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.660 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:48:14.661 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.661 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.661 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.661 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6802", 2024-05-15T12:48:14.661 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:48:14.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.661 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.661 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.661 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6803", 2024-05-15T12:48:14.661 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:48:14.661 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.661 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.661 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:48:14.661 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.661 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.662 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.662 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6806", 2024-05-15T12:48:14.662 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:48:14.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.662 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.662 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.662 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6807", 2024-05-15T12:48:14.662 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:48:14.662 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.662 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.662 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:48:14.662 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.663 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.663 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.663 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6804", 2024-05-15T12:48:14.663 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:48:14.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.663 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.663 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.663 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6805", 2024-05-15T12:48:14.663 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:48:14.663 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.663 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.663 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6801/3877842957", 2024-05-15T12:48:14.663 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6803/3877842957", 2024-05-15T12:48:14.663 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6807/3877842957", 2024-05-15T12:48:14.664 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6805/3877842957", 2024-05-15T12:48:14.664 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:48:14.664 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:48:14.664 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:48:14.664 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.664 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.664 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 5, 2024-05-15T12:48:14.664 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "a6dc220f-aec0-4454-9695-b44c0b9770e0", 2024-05-15T12:48:14.664 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:48:14.664 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:48:14.664 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:48:14.664 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:48:14.664 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:48:14.665 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:48:14.665 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 40, 2024-05-15T12:48:14.665 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 110, 2024-05-15T12:48:14.665 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:48:14.665 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:48:14.665 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:48:14.665 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.665 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.665 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.665 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6808", 2024-05-15T12:48:14.665 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:48:14.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.665 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.665 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.665 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6809", 2024-05-15T12:48:14.666 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:48:14.666 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.666 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.666 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:48:14.666 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.666 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.666 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.666 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6810", 2024-05-15T12:48:14.666 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:48:14.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.666 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.666 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.667 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6811", 2024-05-15T12:48:14.667 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:48:14.667 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.667 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.667 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:48:14.667 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.667 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.667 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.667 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6814", 2024-05-15T12:48:14.667 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:48:14.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.667 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.667 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.667 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6815", 2024-05-15T12:48:14.667 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:48:14.668 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.668 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.668 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:48:14.668 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.668 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.668 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.668 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6812", 2024-05-15T12:48:14.668 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:48:14.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.668 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.668 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.668 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6813", 2024-05-15T12:48:14.668 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:48:14.668 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.669 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.669 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6809/1600179183", 2024-05-15T12:48:14.669 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6811/1600179183", 2024-05-15T12:48:14.669 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6815/1600179183", 2024-05-15T12:48:14.669 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6813/1600179183", 2024-05-15T12:48:14.669 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:48:14.669 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:48:14.669 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:48:14.669 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.669 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.669 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 6, 2024-05-15T12:48:14.669 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b", 2024-05-15T12:48:14.669 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:48:14.670 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:48:14.670 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:48:14.670 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:48:14.670 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:48:14.670 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:48:14.670 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 48, 2024-05-15T12:48:14.670 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 110, 2024-05-15T12:48:14.670 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:48:14.670 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:48:14.670 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:48:14.670 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.670 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.670 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.671 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6816", 2024-05-15T12:48:14.671 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:48:14.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.671 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.671 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.671 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6817", 2024-05-15T12:48:14.671 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:48:14.671 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.671 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.671 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:48:14.671 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.671 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.671 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.671 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6818", 2024-05-15T12:48:14.671 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:48:14.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.672 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.672 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.672 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6819", 2024-05-15T12:48:14.672 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:48:14.672 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.672 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.672 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:48:14.672 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.672 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.672 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.672 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6822", 2024-05-15T12:48:14.672 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:48:14.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.673 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.673 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.673 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6823", 2024-05-15T12:48:14.673 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:48:14.673 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.673 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.673 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:48:14.673 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.673 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.673 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.673 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6820", 2024-05-15T12:48:14.673 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:48:14.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.673 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.674 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.674 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6821", 2024-05-15T12:48:14.674 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:48:14.674 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.674 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.674 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6817/3323706735", 2024-05-15T12:48:14.674 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6819/3323706735", 2024-05-15T12:48:14.674 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6823/3323706735", 2024-05-15T12:48:14.674 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6821/3323706735", 2024-05-15T12:48:14.674 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:48:14.674 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:48:14.674 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:48:14.674 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.675 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.675 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 7, 2024-05-15T12:48:14.675 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "6ef085d8-f59b-4033-8280-2c8f78693e30", 2024-05-15T12:48:14.675 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:48:14.675 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:48:14.675 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:48:14.675 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:48:14.675 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:48:14.675 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:48:14.675 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 56, 2024-05-15T12:48:14.675 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 110, 2024-05-15T12:48:14.675 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:48:14.675 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:48:14.675 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:48:14.675 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.676 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.676 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.676 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6824", 2024-05-15T12:48:14.676 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:48:14.676 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.676 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.676 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.676 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6825", 2024-05-15T12:48:14.676 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:48:14.676 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.676 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.676 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.676 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:48:14.676 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.676 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.677 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.677 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6826", 2024-05-15T12:48:14.677 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:48:14.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.677 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.677 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6827", 2024-05-15T12:48:14.677 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:48:14.677 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.677 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.677 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:48:14.677 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.678 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.678 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6830", 2024-05-15T12:48:14.678 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:48:14.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.678 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.678 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6831", 2024-05-15T12:48:14.678 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:48:14.678 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.678 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.678 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:48:14.678 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.679 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.679 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6828", 2024-05-15T12:48:14.679 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:48:14.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.679 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.679 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.679 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6829", 2024-05-15T12:48:14.679 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:48:14.679 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.679 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.679 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6825/2960616153", 2024-05-15T12:48:14.679 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6827/2960616153", 2024-05-15T12:48:14.679 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6831/2960616153", 2024-05-15T12:48:14.679 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6829/2960616153", 2024-05-15T12:48:14.680 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:48:14.680 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:48:14.680 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:48:14.680 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.680 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.680 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.680 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 8, 2024-05-15T12:48:14.680 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "26514b11-d02b-46e9-a4e8-3da193fcdbcc", 2024-05-15T12:48:14.680 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:48:14.680 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:48:14.680 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:48:14.680 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:48:14.680 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:48:14.680 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:48:14.681 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 65, 2024-05-15T12:48:14.681 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 110, 2024-05-15T12:48:14.681 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:48:14.681 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:48:14.681 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:48:14.681 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.681 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.681 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.681 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6800", 2024-05-15T12:48:14.681 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:48:14.681 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.681 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.681 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.681 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6801", 2024-05-15T12:48:14.681 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:48:14.682 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.682 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.682 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:48:14.682 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.682 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.682 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.682 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6802", 2024-05-15T12:48:14.682 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:48:14.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.682 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.682 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.682 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6803", 2024-05-15T12:48:14.683 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:48:14.683 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.683 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.683 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:48:14.683 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.683 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.683 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.683 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6806", 2024-05-15T12:48:14.683 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:48:14.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.683 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.683 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.683 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6807", 2024-05-15T12:48:14.683 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:48:14.684 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.684 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.684 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:48:14.684 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.684 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.684 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.684 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6804", 2024-05-15T12:48:14.684 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:48:14.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.684 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.684 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.684 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6805", 2024-05-15T12:48:14.684 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:48:14.685 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.685 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.685 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6801/1132459315", 2024-05-15T12:48:14.685 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6803/1132459315", 2024-05-15T12:48:14.685 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6807/1132459315", 2024-05-15T12:48:14.685 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6805/1132459315", 2024-05-15T12:48:14.685 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:48:14.685 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:48:14.685 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:48:14.685 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.685 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.685 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 9, 2024-05-15T12:48:14.685 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "6f9497bc-e962-409d-8a2c-0c634cc1124f", 2024-05-15T12:48:14.686 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:48:14.686 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:48:14.686 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:48:14.686 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:48:14.686 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:48:14.686 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:48:14.686 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 76, 2024-05-15T12:48:14.686 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 110, 2024-05-15T12:48:14.686 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:48:14.686 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:48:14.686 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:48:14.686 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.686 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.686 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.687 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6808", 2024-05-15T12:48:14.687 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:48:14.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.687 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.687 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.687 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6809", 2024-05-15T12:48:14.687 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:48:14.687 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.687 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.687 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:48:14.687 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.687 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.687 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.687 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6810", 2024-05-15T12:48:14.687 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:48:14.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.688 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.688 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.688 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6811", 2024-05-15T12:48:14.688 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:48:14.688 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.688 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.688 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:48:14.688 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.688 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.688 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.688 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6814", 2024-05-15T12:48:14.688 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:48:14.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.689 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.689 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.689 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6815", 2024-05-15T12:48:14.689 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:48:14.689 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.689 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.689 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:48:14.689 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.689 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.689 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.689 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6812", 2024-05-15T12:48:14.689 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:48:14.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.689 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.690 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.690 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6813", 2024-05-15T12:48:14.690 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:48:14.690 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.690 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.690 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6809/2710457646", 2024-05-15T12:48:14.690 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6811/2710457646", 2024-05-15T12:48:14.690 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6815/2710457646", 2024-05-15T12:48:14.690 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6813/2710457646", 2024-05-15T12:48:14.690 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:48:14.690 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:48:14.691 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:48:14.691 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.691 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.691 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 10, 2024-05-15T12:48:14.691 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "f852876f-c7d1-4039-90c4-5ab09dbd3e32", 2024-05-15T12:48:14.691 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:48:14.691 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:48:14.691 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:48:14.691 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:48:14.691 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:48:14.691 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:48:14.691 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 81, 2024-05-15T12:48:14.692 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 0, 2024-05-15T12:48:14.692 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:48:14.692 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:48:14.692 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:48:14.692 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.692 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.692 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6816", 2024-05-15T12:48:14.692 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:48:14.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.692 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.692 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6817", 2024-05-15T12:48:14.693 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:48:14.693 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.693 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.693 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:48:14.693 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.693 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.693 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.693 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6818", 2024-05-15T12:48:14.693 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:48:14.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.693 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.693 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.693 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6819", 2024-05-15T12:48:14.694 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:48:14.694 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.694 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.694 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:48:14.694 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.694 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.694 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.694 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6822", 2024-05-15T12:48:14.694 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:48:14.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.694 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.695 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.695 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6823", 2024-05-15T12:48:14.695 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:48:14.695 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.695 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.695 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:48:14.695 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.695 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.695 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.695 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6820", 2024-05-15T12:48:14.695 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:48:14.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.695 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.696 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.696 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6821", 2024-05-15T12:48:14.696 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:48:14.696 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.696 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.696 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.696 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6817/3236900263", 2024-05-15T12:48:14.696 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6819/3236900263", 2024-05-15T12:48:14.696 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6823/3236900263", 2024-05-15T12:48:14.696 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6821/3236900263", 2024-05-15T12:48:14.696 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:48:14.696 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:48:14.696 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:48:14.697 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.697 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.697 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.697 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 11, 2024-05-15T12:48:14.697 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "cccc53c5-8476-4d55-b160-4bdc5cf92902", 2024-05-15T12:48:14.697 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:48:14.697 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:48:14.697 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:48:14.697 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:48:14.697 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:48:14.697 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:48:14.697 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 90, 2024-05-15T12:48:14.697 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 110, 2024-05-15T12:48:14.697 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:48:14.698 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:48:14.698 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:48:14.698 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.698 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.698 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.698 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6824", 2024-05-15T12:48:14.698 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:48:14.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.698 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.698 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.698 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6825", 2024-05-15T12:48:14.698 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:48:14.699 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.699 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.699 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:48:14.699 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.699 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.699 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.699 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6826", 2024-05-15T12:48:14.699 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:48:14.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.699 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.699 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.699 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6827", 2024-05-15T12:48:14.699 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:48:14.699 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.700 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.700 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:48:14.700 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.700 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.700 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.700 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6830", 2024-05-15T12:48:14.700 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:48:14.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.700 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.700 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.700 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6831", 2024-05-15T12:48:14.700 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:48:14.700 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.701 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.701 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:48:14.701 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:14.701 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.701 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:14.701 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6828", 2024-05-15T12:48:14.701 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:48:14.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.701 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.701 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:14.701 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6829", 2024-05-15T12:48:14.701 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:48:14.701 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.702 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.702 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6825/681011628", 2024-05-15T12:48:14.702 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6827/681011628", 2024-05-15T12:48:14.702 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6831/681011628", 2024-05-15T12:48:14.702 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6829/681011628", 2024-05-15T12:48:14.702 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:48:14.702 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:48:14.702 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:48:14.702 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:14.702 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.702 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T12:48:14.702 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_xinfo": [ 2024-05-15T12:48:14.702 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.703 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 0, 2024-05-15T12:48:14.703 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:48:14.703 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:48:14.703 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:48:14.703 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:48:14.703 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:48:14.703 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:47:33.401880+0000", 2024-05-15T12:48:14.703 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:48:14.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.703 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.703 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 1, 2024-05-15T12:48:14.703 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:48:14.703 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:48:14.703 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:48:14.704 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:48:14.704 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:48:14.704 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:48:01.354812+0000", 2024-05-15T12:48:14.704 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:48:14.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.704 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.704 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 2, 2024-05-15T12:48:14.704 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:48:14.704 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:48:14.704 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:48:14.704 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:48:14.704 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:48:14.704 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:48:00.750859+0000", 2024-05-15T12:48:14.704 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:48:14.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.705 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.705 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 3, 2024-05-15T12:48:14.705 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:48:14.705 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:48:14.705 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:48:14.705 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:48:14.705 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:48:14.705 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:47:37.360495+0000", 2024-05-15T12:48:14.705 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:48:14.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.705 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.705 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 4, 2024-05-15T12:48:14.705 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:48:14.705 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:48:14.706 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:48:14.706 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:48:14.706 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:48:14.706 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:48:08.671701+0000", 2024-05-15T12:48:14.706 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:48:14.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.706 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.706 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 5, 2024-05-15T12:48:14.706 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:48:14.706 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:48:14.706 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:48:14.706 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:48:14.706 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:48:14.706 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:46:53.473605+0000", 2024-05-15T12:48:14.707 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:48:14.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.707 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.707 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 6, 2024-05-15T12:48:14.707 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:48:14.707 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:48:14.707 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:48:14.707 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:48:14.707 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:48:14.707 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:47:42.382426+0000", 2024-05-15T12:48:14.707 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:48:14.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.707 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.707 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 7, 2024-05-15T12:48:14.707 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:48:14.708 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:48:14.708 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:48:14.708 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:48:14.708 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:48:14.708 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:48:00.104152+0000", 2024-05-15T12:48:14.708 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:48:14.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.708 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.708 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 8, 2024-05-15T12:48:14.708 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:48:14.708 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:48:14.708 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:48:14.708 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:48:14.708 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:48:14.708 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:47:20.504432+0000", 2024-05-15T12:48:14.708 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:48:14.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.709 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.709 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 9, 2024-05-15T12:48:14.709 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:48:14.709 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:48:14.709 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:48:14.709 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:48:14.709 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:48:14.709 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:47:23.974039+0000", 2024-05-15T12:48:14.709 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:48:14.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.709 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.709 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 10, 2024-05-15T12:48:14.709 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:48:14.709 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:48:14.710 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:48:14.710 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:48:14.710 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:48:14.710 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:47:41.837735+0000", 2024-05-15T12:48:14.710 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:48:14.710 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.710 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:14.710 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 11, 2024-05-15T12:48:14.710 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:48:14.710 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:48:14.710 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:48:14.710 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:48:14.710 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:48:14.710 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:47:17.427055+0000", 2024-05-15T12:48:14.711 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:48:14.711 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.711 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T12:48:14.711 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap": [], 2024-05-15T12:48:14.711 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap_items": [], 2024-05-15T12:48:14.711 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap_primaries": [], 2024-05-15T12:48:14.711 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_temp": [], 2024-05-15T12:48:14.711 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_temp": [], 2024-05-15T12:48:14.711 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklist": { 2024-05-15T12:48:14.711 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6801/2530617473": "2024-05-16T12:39:19.409195+0000", 2024-05-15T12:48:14.711 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6800/2530617473": "2024-05-16T12:39:19.409195+0000", 2024-05-15T12:48:14.711 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/1880261204": "2024-05-16T12:39:19.409195+0000", 2024-05-15T12:48:14.711 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6801/3441136471": "2024-05-16T12:38:50.931011+0000", 2024-05-15T12:48:14.711 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/4291446247": "2024-05-16T12:39:19.409195+0000", 2024-05-15T12:48:14.711 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/580128209": "2024-05-16T12:38:50.931011+0000", 2024-05-15T12:48:14.712 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6800/3441136471": "2024-05-16T12:38:50.931011+0000", 2024-05-15T12:48:14.712 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/3011229857": "2024-05-16T12:38:50.931011+0000", 2024-05-15T12:48:14.712 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/2109693913": "2024-05-16T12:38:50.931011+0000", 2024-05-15T12:48:14.712 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/2445663024": "2024-05-16T12:39:19.409195+0000" 2024-05-15T12:48:14.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.712 INFO:teuthology.orchestra.run.smithi027.stdout: "range_blocklist": {}, 2024-05-15T12:48:14.712 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profiles": { 2024-05-15T12:48:14.712 INFO:teuthology.orchestra.run.smithi027.stdout: "default": { 2024-05-15T12:48:14.712 INFO:teuthology.orchestra.run.smithi027.stdout: "crush-failure-domain": "osd", 2024-05-15T12:48:14.712 INFO:teuthology.orchestra.run.smithi027.stdout: "k": "2", 2024-05-15T12:48:14.712 INFO:teuthology.orchestra.run.smithi027.stdout: "m": "1", 2024-05-15T12:48:14.712 INFO:teuthology.orchestra.run.smithi027.stdout: "plugin": "jerasure", 2024-05-15T12:48:14.712 INFO:teuthology.orchestra.run.smithi027.stdout: "technique": "reed_sol_van" 2024-05-15T12:48:14.712 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:14.713 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps_queue": [], 2024-05-15T12:48:14.713 INFO:teuthology.orchestra.run.smithi027.stdout: "new_removed_snaps": [], 2024-05-15T12:48:14.713 INFO:teuthology.orchestra.run.smithi027.stdout: "new_purged_snaps": [], 2024-05-15T12:48:14.713 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_node_flags": {}, 2024-05-15T12:48:14.713 INFO:teuthology.orchestra.run.smithi027.stdout: "device_class_flags": {}, 2024-05-15T12:48:14.713 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode": { 2024-05-15T12:48:14.713 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode_enabled": false, 2024-05-15T12:48:14.713 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_bucket_count": 0, 2024-05-15T12:48:14.713 INFO:teuthology.orchestra.run.smithi027.stdout: "degraded_stretch_mode": 0, 2024-05-15T12:48:14.713 INFO:teuthology.orchestra.run.smithi027.stdout: "recovering_stretch_mode": 0, 2024-05-15T12:48:14.713 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode_bucket": 0 2024-05-15T12:48:14.713 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:14.713 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T12:48:14.767 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 4 15 14474 14459 225.901 182.75 0.00326321 0.00442088 2024-05-15T12:48:15.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:14 smithi177 ceph-mon[44303]: pgmap v372: 17 pgs: 17 active+clean; 187 MiB data, 714 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 297 op/s 2024-05-15T12:48:15.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:14 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2693521895' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T12:48:15.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:14 smithi027 ceph-mon[39795]: pgmap v372: 17 pgs: 17 active+clean; 187 MiB data, 714 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 297 op/s 2024-05-15T12:48:15.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:14 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2693521895' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T12:48:15.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:14 smithi120 ceph-mon[43961]: pgmap v372: 17 pgs: 17 active+clean; 187 MiB data, 714 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 297 op/s 2024-05-15T12:48:15.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:14 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2693521895' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T12:48:15.589 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:48:15.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:15.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:48:15.701 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:48:15.737 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:15.761 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:48:15.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:15.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:48:15.767 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 5 16 18032 18016 225.18 222.312 0.00294994 0.00443566 2024-05-15T12:48:15.868 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:48:15.920 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:15.983 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:48:15.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:15.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:48:16.089 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:48:16.129 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:16.151 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:48:16.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:16.154 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:48:16.255 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:48:16.280 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:16.299 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:48:16.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:16.302 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:48:16.403 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:48:16.436 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:16.461 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:48:16.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:16.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:48:16.566 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:48:16.568 INFO:tasks.thrashosds.thrasher:j is {'epoch': 115, 'fsid': 'deaea04e-12b7-11ef-bc98-c7b262605968', 'created': '2024-05-15T12:38:21.244556+0000', 'modified': '2024-05-15T12:48:08.824732+0000', 'last_up_change': '2024-05-15T12:45:50.876090+0000', 'last_in_change': '2024-05-15T12:45:33.158047+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-15T12:41:57.948613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-05-15T12:47:57.372152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '115', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '64fb3ad2-08a7-4106-a07a-6c776c877faf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 58, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6802', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6803', 'nonce': 4239116935}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6804', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6805', 'nonce': 4239116935}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6808', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6809', 'nonce': 4239116935}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6806', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6807', 'nonce': 4239116935}]}, 'public_addr': '172.21.15.27:6803/4239116935', 'cluster_addr': '172.21.15.27:6805/4239116935', 'heartbeat_back_addr': '172.21.15.27:6809/4239116935', 'heartbeat_front_addr': '172.21.15.27:6807/4239116935', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '9c16dd90-9d51-48b8-a296-1c8879ba1bc8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6810', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6811', 'nonce': 1466270637}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6812', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6813', 'nonce': 1466270637}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6816', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6817', 'nonce': 1466270637}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6814', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6815', 'nonce': 1466270637}]}, 'public_addr': '172.21.15.27:6811/1466270637', 'cluster_addr': '172.21.15.27:6813/1466270637', 'heartbeat_back_addr': '172.21.15.27:6817/1466270637', 'heartbeat_front_addr': '172.21.15.27:6815/1466270637', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'cbe3ba13-3ea3-4891-8d2d-6b85581a732c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6818', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6819', 'nonce': 3285413897}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6820', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6821', 'nonce': 3285413897}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6824', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6825', 'nonce': 3285413897}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6822', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6823', 'nonce': 3285413897}]}, 'public_addr': '172.21.15.27:6819/3285413897', 'cluster_addr': '172.21.15.27:6821/3285413897', 'heartbeat_back_addr': '172.21.15.27:6825/3285413897', 'heartbeat_front_addr': '172.21.15.27:6823/3285413897', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2e83735b-b944-4f87-8080-b994d4b96897', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6826', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6827', 'nonce': 3008975244}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6828', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6829', 'nonce': 3008975244}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6833', 'nonce': 3008975244}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6830', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6831', 'nonce': 3008975244}]}, 'public_addr': '172.21.15.27:6827/3008975244', 'cluster_addr': '172.21.15.27:6829/3008975244', 'heartbeat_back_addr': '172.21.15.27:6833/3008975244', 'heartbeat_front_addr': '172.21.15.27:6831/3008975244', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7c39c1dd-3867-46f7-8372-7688fa13b16a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6800', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6801', 'nonce': 3877842957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6802', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6803', 'nonce': 3877842957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6806', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6807', 'nonce': 3877842957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6804', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6805', 'nonce': 3877842957}]}, 'public_addr': '172.21.15.120:6801/3877842957', 'cluster_addr': '172.21.15.120:6803/3877842957', 'heartbeat_back_addr': '172.21.15.120:6807/3877842957', 'heartbeat_front_addr': '172.21.15.120:6805/3877842957', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a6dc220f-aec0-4454-9695-b44c0b9770e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6808', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6809', 'nonce': 1600179183}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6810', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6811', 'nonce': 1600179183}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6814', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6815', 'nonce': 1600179183}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6812', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6813', 'nonce': 1600179183}]}, 'public_addr': '172.21.15.120:6809/1600179183', 'cluster_addr': '172.21.15.120:6811/1600179183', 'heartbeat_back_addr': '172.21.15.120:6815/1600179183', 'heartbeat_front_addr': '172.21.15.120:6813/1600179183', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6816', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6817', 'nonce': 3323706735}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6818', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6819', 'nonce': 3323706735}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6822', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6823', 'nonce': 3323706735}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6820', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6821', 'nonce': 3323706735}]}, 'public_addr': '172.21.15.120:6817/3323706735', 'cluster_addr': '172.21.15.120:6819/3323706735', 'heartbeat_back_addr': '172.21.15.120:6823/3323706735', 'heartbeat_front_addr': '172.21.15.120:6821/3323706735', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6ef085d8-f59b-4033-8280-2c8f78693e30', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6824', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6825', 'nonce': 2960616153}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6826', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6827', 'nonce': 2960616153}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6830', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6831', 'nonce': 2960616153}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6828', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6829', 'nonce': 2960616153}]}, 'public_addr': '172.21.15.120:6825/2960616153', 'cluster_addr': '172.21.15.120:6827/2960616153', 'heartbeat_back_addr': '172.21.15.120:6831/2960616153', 'heartbeat_front_addr': '172.21.15.120:6829/2960616153', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '26514b11-d02b-46e9-a4e8-3da193fcdbcc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6800', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6801', 'nonce': 1132459315}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6802', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6803', 'nonce': 1132459315}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6806', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6807', 'nonce': 1132459315}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6804', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6805', 'nonce': 1132459315}]}, 'public_addr': '172.21.15.177:6801/1132459315', 'cluster_addr': '172.21.15.177:6803/1132459315', 'heartbeat_back_addr': '172.21.15.177:6807/1132459315', 'heartbeat_front_addr': '172.21.15.177:6805/1132459315', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '6f9497bc-e962-409d-8a2c-0c634cc1124f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6808', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6809', 'nonce': 2710457646}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6810', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6811', 'nonce': 2710457646}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6814', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6815', 'nonce': 2710457646}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6812', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6813', 'nonce': 2710457646}]}, 'public_addr': '172.21.15.177:6809/2710457646', 'cluster_addr': '172.21.15.177:6811/2710457646', 'heartbeat_back_addr': '172.21.15.177:6815/2710457646', 'heartbeat_front_addr': '172.21.15.177:6813/2710457646', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'f852876f-c7d1-4039-90c4-5ab09dbd3e32', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 81, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6816', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6817', 'nonce': 3236900263}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6818', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6819', 'nonce': 3236900263}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6822', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6823', 'nonce': 3236900263}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6820', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6821', 'nonce': 3236900263}]}, 'public_addr': '172.21.15.177:6817/3236900263', 'cluster_addr': '172.21.15.177:6819/3236900263', 'heartbeat_back_addr': '172.21.15.177:6823/3236900263', 'heartbeat_front_addr': '172.21.15.177:6821/3236900263', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'cccc53c5-8476-4d55-b160-4bdc5cf92902', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 90, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6824', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6825', 'nonce': 681011628}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6826', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6827', 'nonce': 681011628}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6830', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6831', 'nonce': 681011628}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6828', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6829', 'nonce': 681011628}]}, 'public_addr': '172.21.15.177:6825/681011628', 'cluster_addr': '172.21.15.177:6827/681011628', 'heartbeat_back_addr': '172.21.15.177:6831/681011628', 'heartbeat_front_addr': '172.21.15.177:6829/681011628', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:47:33.401880+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:48:01.354812+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:48:00.750859+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:47:37.360495+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:48:08.671701+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:46:53.473605+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:47:42.382426+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:48:00.104152+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:47:20.504432+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:47:23.974039+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:47:41.837735+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:47:17.427055+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.27:6801/2530617473': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:6800/2530617473': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:0/1880261204': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:6801/3441136471': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/4291446247': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:0/580128209': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:6800/3441136471': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/3011229857': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/2109693913': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/2445663024': '2024-05-16T12:39:19.409195+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-05-15T12:48:16.568 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-05-15T12:48:16.608 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:16.635 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:48:16.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:16.640 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:48:16.741 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:48:16.767 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 6 16 20728 20712 215.731 168.5 0.0049962 0.00462971 2024-05-15T12:48:16.777 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:16.800 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:48:16.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:16.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:48:16.818 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:48:16.905 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:48:16.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:16 smithi177 ceph-mon[44303]: pgmap v373: 17 pgs: 17 active+clean; 562 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 897 op/s 2024-05-15T12:48:16.950 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:16.984 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:48:16.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:16.988 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:48:17.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:16 smithi120 ceph-mon[43961]: pgmap v373: 17 pgs: 17 active+clean; 562 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 897 op/s 2024-05-15T12:48:17.089 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:48:17.115 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:17.133 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:48:17.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:17.136 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:48:17.237 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:48:17.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:16 smithi027 ceph-mon[39795]: pgmap v373: 17 pgs: 17 active+clean; 562 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 897 op/s 2024-05-15T12:48:17.271 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:17.295 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:48:17.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:17.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:48:17.400 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:48:17.441 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:17.467 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:48:17.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:17.471 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:48:17.572 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:48:17.607 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:17.639 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:48:17.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:17.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:48:17.743 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:48:17.754 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-05-15T12:48:17.768 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 7 16 23128 23112 206.32 150 0.0051787 0.00484151 2024-05-15T12:48:17.786 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:17.812 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:48:17.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:17.815 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:48:17.875 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T12:48:17.915 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:48:17.944 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:17 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1448456624' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:48:17.946 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:17.972 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:48:17.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:17.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:48:18.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:17 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1448456624' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:48:18.075 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:48:18.110 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:18.141 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:48:18.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:18.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:48:18.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:17 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1448456624' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:48:18.244 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:48:18.274 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:18.327 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:48:18.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:18.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:48:18.433 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:48:18.479 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:18.507 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:48:18.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:18.510 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:48:18.611 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:48:18.660 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:18.768 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 8 16 25480 25464 198.904 147 0.00454102 0.00502252 2024-05-15T12:48:18.916 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:48:18.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:18.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:48:19.029 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:48:19.063 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:19.088 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:48:19.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:19.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:48:19.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:18 smithi177 ceph-mon[44303]: pgmap v374: 17 pgs: 17 active+clean; 887 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-05-15T12:48:19.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:18 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:48:19.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:18 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:48:19.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:18 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:48:19.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:18 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1448456624' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:48:19.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:18 smithi177 ceph-mon[44303]: osdmap e116: 12 total, 12 up, 12 in 2024-05-15T12:48:19.192 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:48:19.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:18 smithi027 ceph-mon[39795]: pgmap v374: 17 pgs: 17 active+clean; 887 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-05-15T12:48:19.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:18 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:48:19.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:18 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:48:19.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:18 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:48:19.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:18 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1448456624' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:48:19.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:18 smithi027 ceph-mon[39795]: osdmap e116: 12 total, 12 up, 12 in 2024-05-15T12:48:19.236 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:19.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:18 smithi120 ceph-mon[43961]: pgmap v374: 17 pgs: 17 active+clean; 887 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-05-15T12:48:19.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:18 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:48:19.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:18 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:48:19.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:18 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:48:19.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:18 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1448456624' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:48:19.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:18 smithi120 ceph-mon[43961]: osdmap e116: 12 total, 12 up, 12 in 2024-05-15T12:48:19.768 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 9 16 28361 28345 196.809 180.062 0.00263884 0.00505305 2024-05-15T12:48:19.885 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:48:19.895 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:48:19.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:19.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:48:20.003 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:48:20.037 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:20.064 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:48:20.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:20.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:48:20.168 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:48:20.215 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:20.245 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:48:20.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:20.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:48:20.349 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:48:20.377 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:20.526 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:48:20.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:20.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:48:20.630 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:48:20.658 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:20.769 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 10 16 29546 29530 184.534 74.0625 0.0016355 0.00513927 2024-05-15T12:48:20.960 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:48:20.960 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:48:20.960 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:48:20.964 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T12:48:20.980 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:48:21.002 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:48:21.022 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:48:21.043 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:48:21.063 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:48:21.083 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:48:21.104 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:48:21.124 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:48:21.145 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:48:21.146 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:48:21.146 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:48:21.146 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:48:21.146 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:48:21.146 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:48:21.147 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:48:21.147 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:48:21.147 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:48:21.147 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:48:21.147 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:48:21.147 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:48:21.147 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:48:21.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:20 smithi027 ceph-mon[39795]: pgmap v376: 17 pgs: 17 active+clean; 1.2 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 119 MiB/s wr, 1.90k op/s 2024-05-15T12:48:21.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:20 smithi177 ceph-mon[44303]: pgmap v376: 17 pgs: 17 active+clean; 1.2 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 119 MiB/s wr, 1.90k op/s 2024-05-15T12:48:21.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:20 smithi120 ceph-mon[43961]: pgmap v376: 17 pgs: 17 active+clean; 1.2 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 119 MiB/s wr, 1.90k op/s 2024-05-15T12:48:21.569 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T12:48:21.570 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T12:48:21.570 INFO:tasks.thrashosds.thrasher:Removing osd 10, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T12:48:21.570 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd out 10 2024-05-15T12:48:21.769 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 11 16 29546 29530 167.758 0 - 0.00513927 2024-05-15T12:48:22.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:21 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3808356159' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:48:22.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:21 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3808356159' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:48:22.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:21 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3808356159' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:48:22.769 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 12 16 29546 29530 153.778 0 - 0.00513927 2024-05-15T12:48:22.953 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:48:22.995 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T12:48:22.995 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:48:23.018 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:22 smithi027 ceph-mon[39795]: pgmap v377: 17 pgs: 17 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 164 MiB/s wr, 2.62k op/s 2024-05-15T12:48:23.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:22 smithi177 ceph-mon[44303]: pgmap v377: 17 pgs: 17 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 164 MiB/s wr, 2.62k op/s 2024-05-15T12:48:23.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:22 smithi120 ceph-mon[43961]: pgmap v377: 17 pgs: 17 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 164 MiB/s wr, 2.62k op/s 2024-05-15T12:48:23.610 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:48:23.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:23.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:48:23.714 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:48:23.746 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:23.767 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:48:23.769 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 13 16 29546 29530 141.949 0 - 0.00513927 2024-05-15T12:48:23.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:23.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:48:23.871 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:48:23.901 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:23.921 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:48:23.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:23.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:48:24.025 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:48:24.056 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:24.077 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:48:24.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:24.080 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:48:24.181 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:48:24.210 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:24.229 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:48:24.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:24.232 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:48:24.333 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:48:24.361 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:24.381 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:48:24.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:24.384 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:48:24.451 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:48:24.484 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:48:24.511 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:24.531 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:48:24.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:24.533 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:48:24.634 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:48:24.666 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:24.686 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:48:24.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:24.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:48:24.769 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 14 16 29546 29530 131.81 0 - 0.00513927 2024-05-15T12:48:24.790 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:48:24.818 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:25.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:24 smithi120 ceph-mon[43961]: pgmap v378: 17 pgs: 17 active+clean; 1.8 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 165 MiB/s wr, 2.63k op/s 2024-05-15T12:48:25.032 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:48:25.062 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-05-15T12:48:25.127 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:24 smithi027 ceph-mon[39795]: pgmap v378: 17 pgs: 17 active+clean; 1.8 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 165 MiB/s wr, 2.63k op/s 2024-05-15T12:48:25.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:24 smithi177 ceph-mon[44303]: pgmap v378: 17 pgs: 17 active+clean; 1.8 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 165 MiB/s wr, 2.63k op/s 2024-05-15T12:48:25.769 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 15 16 29546 29530 123.023 0 - 0.00513927 2024-05-15T12:48:25.797 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:48:25.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:25.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:48:25.902 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:48:25.932 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:26.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:26 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-a[39772]: 2024-05-15T12:48:26.133+0000 7f8daffd2700 -1 mon.a@0(leader).osd e116 definitely_dead 0 2024-05-15T12:48:26.769 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 16 16 29546 29530 115.334 0 - 0.00513927 2024-05-15T12:48:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:48:26.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:26.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:48:26.802 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:48:26.805 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:48:26.876 INFO:teuthology.orchestra.run.smithi027.stderr:marked out osd.10. 2024-05-15T12:48:26.894 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:48:26.924 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:26.945 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:48:26.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:26.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:48:27.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:26 smithi027 ceph-mon[39795]: pgmap v379: 17 pgs: 17 active+clean; 1.8 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s wr, 2.05k op/s 2024-05-15T12:48:27.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:26 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3749995532' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-05-15T12:48:27.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:26 smithi027 ceph-mon[39795]: Client client.admin marked osd.10 out, while it was still marked up 2024-05-15T12:48:27.049 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:48:27.080 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:27.102 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:48:27.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:27.104 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:48:27.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:26 smithi177 ceph-mon[44303]: pgmap v379: 17 pgs: 17 active+clean; 1.8 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s wr, 2.05k op/s 2024-05-15T12:48:27.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:26 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3749995532' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-05-15T12:48:27.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:26 smithi177 ceph-mon[44303]: Client client.admin marked osd.10 out, while it was still marked up 2024-05-15T12:48:27.206 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:48:27.235 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:27.256 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:48:27.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:27.258 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:48:27.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:26 smithi120 ceph-mon[43961]: pgmap v379: 17 pgs: 17 active+clean; 1.8 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s wr, 2.05k op/s 2024-05-15T12:48:27.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:26 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3749995532' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-05-15T12:48:27.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:26 smithi120 ceph-mon[43961]: Client client.admin marked osd.10 out, while it was still marked up 2024-05-15T12:48:27.359 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:48:27.389 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:27.442 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:48:27.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:27.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:48:27.546 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:48:27.575 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:27.674 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:48:27.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:27.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:48:27.770 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 17 16 29598 29582 108.741 0.464286 0.00201857 0.00893713 2024-05-15T12:48:27.783 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:48:27.815 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:27.819 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T12:48:27.837 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:48:27.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:27.840 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:48:27.941 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:48:27.969 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:27.989 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:48:27.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:27.991 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:48:28.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:27 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:48:28.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:27 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:48:28.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:27 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3749995532' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-05-15T12:48:28.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:27 smithi027 ceph-mon[39795]: osdmap e117: 12 total, 12 up, 11 in 2024-05-15T12:48:28.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:27 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/625731036' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:48:28.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:27 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:48:28.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:27 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/625731036' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:48:28.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:27 smithi027 ceph-mon[39795]: osdmap e118: 12 total, 12 up, 11 in 2024-05-15T12:48:28.092 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:48:28.121 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:28.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:27 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:48:28.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:27 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:48:28.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:27 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3749995532' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-05-15T12:48:28.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:27 smithi177 ceph-mon[44303]: osdmap e117: 12 total, 12 up, 11 in 2024-05-15T12:48:28.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:27 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/625731036' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:48:28.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:27 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:48:28.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:27 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/625731036' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:48:28.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:27 smithi177 ceph-mon[44303]: osdmap e118: 12 total, 12 up, 11 in 2024-05-15T12:48:28.270 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:48:28.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:28.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:48:28.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:27 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:48:28.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:27 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:48:28.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:27 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3749995532' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-05-15T12:48:28.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:27 smithi120 ceph-mon[43961]: osdmap e117: 12 total, 12 up, 11 in 2024-05-15T12:48:28.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:27 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/625731036' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:48:28.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:27 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:48:28.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:27 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/625731036' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:48:28.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:27 smithi120 ceph-mon[43961]: osdmap e118: 12 total, 12 up, 11 in 2024-05-15T12:48:28.374 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:48:28.406 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:28.427 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:48:28.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:28.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:48:28.530 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:48:28.563 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:28.584 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:48:28.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:28.587 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:48:28.688 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:48:28.717 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:28.770 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 18 16 29598 29582 102.7 0 - 0.00893713 2024-05-15T12:48:28.969 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:48:28.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:28.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:48:29.075 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:48:29.119 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:29.148 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:48:29.152 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:28 smithi177 ceph-mon[44303]: pgmap v381: 17 pgs: 17 active+clean; 1.8 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-05-15T12:48:29.152 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:28 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T12:48:29.152 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:28 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T12:48:29.152 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:28 smithi177 ceph-mon[44303]: osdmap e119: 12 total, 12 up, 11 in 2024-05-15T12:48:29.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:29.153 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:48:29.254 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:48:29.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:28 smithi027 ceph-mon[39795]: pgmap v381: 17 pgs: 17 active+clean; 1.8 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-05-15T12:48:29.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:28 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T12:48:29.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:28 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T12:48:29.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:28 smithi027 ceph-mon[39795]: osdmap e119: 12 total, 12 up, 11 in 2024-05-15T12:48:29.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:28 smithi120 ceph-mon[43961]: pgmap v381: 17 pgs: 17 active+clean; 1.8 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-05-15T12:48:29.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:28 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T12:48:29.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:28 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T12:48:29.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:28 smithi120 ceph-mon[43961]: osdmap e119: 12 total, 12 up, 11 in 2024-05-15T12:48:29.285 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:29.341 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:48:29.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:29.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:48:29.445 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:48:29.471 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:29.490 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:48:29.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:29.492 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:48:29.593 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:48:29.636 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:29.663 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:48:29.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:29.666 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:48:29.767 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:48:29.770 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 19 16 30663 30647 100.798 33.2812 0.00215047 0.00991577 2024-05-15T12:48:29.803 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:29.833 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:48:29.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:29.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:48:29.940 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:48:29.996 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:30.219 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:48:30.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:30.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:48:30.326 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:48:30.383 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:30.770 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T12:48:30.781282+0000 min lat: 0.00104218 max lat: 7.07499 avg lat: 0.00982126 2024-05-15T12:48:30.770 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T12:48:30.770 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 20 16 32586 32570 101.767 120.188 0.00527362 0.00982126 2024-05-15T12:48:30.883 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:30 smithi027 ceph-mon[39795]: pgmap v384: 17 pgs: 1 remapped+peering, 16 active+clean; 1.8 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 2.3 MiB/s wr, 37 op/s 2024-05-15T12:48:31.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:30 smithi177 ceph-mon[44303]: pgmap v384: 17 pgs: 1 remapped+peering, 16 active+clean; 1.8 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 2.3 MiB/s wr, 37 op/s 2024-05-15T12:48:31.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:30 smithi120 ceph-mon[43961]: pgmap v384: 17 pgs: 1 remapped+peering, 16 active+clean; 1.8 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 2.3 MiB/s wr, 37 op/s 2024-05-15T12:48:31.524 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:48:31.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:31.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:48:31.630 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:48:31.667 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:31.698 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:48:31.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:31.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:48:31.770 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 21 16 34683 34667 103.161 131.062 0.00877519 0.0096876 2024-05-15T12:48:31.802 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:48:31.881 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:32.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:31 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:48:32.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:31 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:48:32.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:31 smithi120 ceph-mon[43961]: osdmap e120: 12 total, 12 up, 11 in 2024-05-15T12:48:32.098 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:48:32.100 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:31 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:48:32.100 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:31 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:48:32.100 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:31 smithi027 ceph-mon[39795]: osdmap e120: 12 total, 12 up, 11 in 2024-05-15T12:48:32.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:32.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:48:32.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:31 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:48:32.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:31 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:48:32.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:31 smithi177 ceph-mon[44303]: osdmap e120: 12 total, 12 up, 11 in 2024-05-15T12:48:32.203 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:48:32.240 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:32.271 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:48:32.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:32.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:48:32.377 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:48:32.427 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:32.498 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T12:48:32.498 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:48:32.520 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:48:32.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:32.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:48:32.626 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:48:32.666 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:32.689 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:48:32.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:32.691 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:48:32.721 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:48:32.760 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:48:32.770 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 22 16 36530 36514 103.719 115.438 0.0163735 0.00963579 2024-05-15T12:48:32.793 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:48:32.805 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:48:32.830 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:32.847 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:48:32.860 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:48:32.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:32.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:48:32.893 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:48:32.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:32 smithi177 ceph-mon[44303]: pgmap v386: 17 pgs: 2 active+recovering+undersized+remapped, 1 remapped+peering, 14 active+clean; 1.9 GiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 18 MiB/s wr, 280 op/s; 3351/92502 objects misplaced (3.623%); 5.2 MiB/s, 83 objects/s recovering 2024-05-15T12:48:32.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:32 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1350851404' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:48:32.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:32 smithi177 ceph-mon[44303]: osdmap e121: 12 total, 12 up, 11 in 2024-05-15T12:48:32.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:32 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:48:32.945 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:48:32.964 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:48:32.993 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:48:33.002 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T12:48:33.005 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:33.030 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:48:33.036 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:48:33.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:33.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:48:33.069 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:48:33.110 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:48:33.141 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:48:33.151 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:48:33.181 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:33.209 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:48:33.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:33.210 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:48:33.223 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:48:33.237 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:48:33.237 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:48:33.237 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:48:33.237 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:48:33.237 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:48:33.237 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:48:33.237 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:48:33.237 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:48:33.237 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:48:33.238 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:48:33.238 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:48:33.238 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:48:33.266 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:32 smithi027 ceph-mon[39795]: pgmap v386: 17 pgs: 2 active+recovering+undersized+remapped, 1 remapped+peering, 14 active+clean; 1.9 GiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 18 MiB/s wr, 280 op/s; 3351/92502 objects misplaced (3.623%); 5.2 MiB/s, 83 objects/s recovering 2024-05-15T12:48:33.267 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:32 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1350851404' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:48:33.267 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:32 smithi027 ceph-mon[39795]: osdmap e121: 12 total, 12 up, 11 in 2024-05-15T12:48:33.267 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:32 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:48:33.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:32 smithi120 ceph-mon[43961]: pgmap v386: 17 pgs: 2 active+recovering+undersized+remapped, 1 remapped+peering, 14 active+clean; 1.9 GiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 18 MiB/s wr, 280 op/s; 3351/92502 objects misplaced (3.623%); 5.2 MiB/s, 83 objects/s recovering 2024-05-15T12:48:33.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:32 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1350851404' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:48:33.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:32 smithi120 ceph-mon[43961]: osdmap e121: 12 total, 12 up, 11 in 2024-05-15T12:48:33.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:32 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:48:33.311 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:48:33.349 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:33.373 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:48:33.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:33.377 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:48:33.478 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:48:33.510 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:33.538 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:48:33.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:33.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:48:33.645 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:48:33.680 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:33.710 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:48:33.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:33.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:48:33.736 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T12:48:33.737 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T12:48:33.737 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-05-15T12:48:33.737 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd dump -f json-pretty 2024-05-15T12:48:33.770 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 23 16 38173 38157 103.673 102.688 0.0190838 0.00963999 2024-05-15T12:48:33.815 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:48:33.827 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-05-15T12:48:33.847 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:33.871 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:48:33.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:33.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:48:33.976 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:48:34.009 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:34.033 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:48:34.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:34.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:48:34.138 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:48:34.170 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:34.199 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:48:34.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:34.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:48:34.305 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:48:34.367 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:34.581 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:48:34.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:34.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:48:34.614 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:48:34.621 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:48:34.687 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:48:34.725 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:34.751 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:48:34.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:34.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:48:34.770 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 24 16 39744 39728 103.444 98.1875 0.00257554 0.00966173 2024-05-15T12:48:34.855 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:48:34.911 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:35.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:34 smithi120 ceph-mon[43961]: pgmap v388: 17 pgs: 3 active+recovering+undersized+remapped, 14 active+clean; 2.1 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 45 MiB/s wr, 717 op/s; 4722/100920 objects misplaced (4.679%); 9.5 MiB/s, 151 objects/s recovering 2024-05-15T12:48:35.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:34 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:48:35.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:34 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:48:35.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:34 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:48:35.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:34 smithi177 ceph-mon[44303]: pgmap v388: 17 pgs: 3 active+recovering+undersized+remapped, 14 active+clean; 2.1 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 45 MiB/s wr, 717 op/s; 4722/100920 objects misplaced (4.679%); 9.5 MiB/s, 151 objects/s recovering 2024-05-15T12:48:35.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:34 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:48:35.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:34 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:48:35.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:34 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:48:35.246 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:34 smithi027 ceph-mon[39795]: pgmap v388: 17 pgs: 3 active+recovering+undersized+remapped, 14 active+clean; 2.1 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 45 MiB/s wr, 717 op/s; 4722/100920 objects misplaced (4.679%); 9.5 MiB/s, 151 objects/s recovering 2024-05-15T12:48:35.246 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:34 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:48:35.247 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:34 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:48:35.247 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:34 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:48:35.770 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 25 16 41631 41615 104.024 117.938 0.00762987 0.00960709 2024-05-15T12:48:35.976 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:48:35.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:35.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:48:36.085 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:48:36.117 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:36.142 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:48:36.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:36.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:48:36.187 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:48:36.247 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:48:36.300 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:36.360 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:48:36.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:36.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:48:36.473 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:48:36.509 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:36.534 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:48:36.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:36.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:48:36.617 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:48:36.641 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:48:36.675 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:36.702 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:48:36.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:36.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:48:36.770 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 26 16 43516 43500 104.554 117.812 0.0113232 0.00955796 2024-05-15T12:48:36.807 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:48:36.836 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:36.861 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:48:36.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:36.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:48:36.937 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:36 smithi027 ceph-mon[39795]: pgmap v389: 17 pgs: 3 active+recovering+undersized+remapped, 14 active+clean; 2.3 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s; 3946/111135 objects misplaced (3.551%); 15 MiB/s, 245 objects/s recovering 2024-05-15T12:48:36.966 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:48:37.007 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:37.264 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:48:37.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:37.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:48:37.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:36 smithi120 ceph-mon[43961]: pgmap v389: 17 pgs: 3 active+recovering+undersized+remapped, 14 active+clean; 2.3 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s; 3946/111135 objects misplaced (3.551%); 15 MiB/s, 245 objects/s recovering 2024-05-15T12:48:37.373 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:48:37.412 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:36 smithi177 ceph-mon[44303]: pgmap v389: 17 pgs: 3 active+recovering+undersized+remapped, 14 active+clean; 2.3 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s; 3946/111135 objects misplaced (3.551%); 15 MiB/s, 245 objects/s recovering 2024-05-15T12:48:37.422 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:37.440 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:48:37.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:37.445 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:48:37.546 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:48:37.579 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:37.602 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:48:37.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:37.606 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:48:37.707 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:48:37.754 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:37.770 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 27 16 45220 45204 104.625 106.5 0.00590679 0.00955157 2024-05-15T12:48:37.818 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:48:37.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:37.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:48:37.924 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:48:37.958 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:38.617 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:48:38.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:38.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:48:38.723 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:48:38.761 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:38.770 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 28 16 46881 46865 104.596 103.812 0.0100955 0.00955525 2024-05-15T12:48:38.786 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:48:38.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:38.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:48:38.851 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:38 smithi027 ceph-mon[39795]: pgmap v390: 17 pgs: 3 active+recovering+undersized+remapped, 14 active+clean; 2.4 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 79 MiB/s wr, 1.27k op/s; 3593/119151 objects misplaced (3.016%); 16 MiB/s, 249 objects/s recovering 2024-05-15T12:48:38.851 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:38 smithi027 ceph-mon[39795]: osdmap e122: 12 total, 12 up, 11 in 2024-05-15T12:48:38.890 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:48:38.963 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:39.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:38 smithi120 ceph-mon[43961]: pgmap v390: 17 pgs: 3 active+recovering+undersized+remapped, 14 active+clean; 2.4 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 79 MiB/s wr, 1.27k op/s; 3593/119151 objects misplaced (3.016%); 16 MiB/s, 249 objects/s recovering 2024-05-15T12:48:39.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:38 smithi120 ceph-mon[43961]: osdmap e122: 12 total, 12 up, 11 in 2024-05-15T12:48:39.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:38 smithi177 ceph-mon[44303]: pgmap v390: 17 pgs: 3 active+recovering+undersized+remapped, 14 active+clean; 2.4 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 79 MiB/s wr, 1.27k op/s; 3593/119151 objects misplaced (3.016%); 16 MiB/s, 249 objects/s recovering 2024-05-15T12:48:39.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:38 smithi177 ceph-mon[44303]: osdmap e122: 12 total, 12 up, 11 in 2024-05-15T12:48:39.771 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 29 16 48449 48433 104.368 98 0.00913099 0.00957675 2024-05-15T12:48:40.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:39 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1416047618' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T12:48:40.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:39 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1416047618' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T12:48:40.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:39 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1416047618' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T12:48:40.771 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 30 16 48821 48805 101.664 23.25 0.00199785 0.00956833 2024-05-15T12:48:41.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:40 smithi027 ceph-mon[39795]: pgmap v392: 17 pgs: 3 active+recovering+undersized+remapped, 14 active+clean; 2.6 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 91 MiB/s wr, 1.45k op/s; 3082/127350 objects misplaced (2.420%); 17 MiB/s, 264 objects/s recovering 2024-05-15T12:48:41.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:40 smithi027 ceph-mon[39795]: osdmap e123: 12 total, 12 up, 11 in 2024-05-15T12:48:41.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:40 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4085732478' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:48:41.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:40 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:48:41.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:40 smithi177 ceph-mon[44303]: pgmap v392: 17 pgs: 3 active+recovering+undersized+remapped, 14 active+clean; 2.6 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 91 MiB/s wr, 1.45k op/s; 3082/127350 objects misplaced (2.420%); 17 MiB/s, 264 objects/s recovering 2024-05-15T12:48:41.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:40 smithi177 ceph-mon[44303]: osdmap e123: 12 total, 12 up, 11 in 2024-05-15T12:48:41.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:40 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4085732478' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:48:41.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:40 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:48:41.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:40 smithi120 ceph-mon[43961]: pgmap v392: 17 pgs: 3 active+recovering+undersized+remapped, 14 active+clean; 2.6 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 91 MiB/s wr, 1.45k op/s; 3082/127350 objects misplaced (2.420%); 17 MiB/s, 264 objects/s recovering 2024-05-15T12:48:41.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:40 smithi120 ceph-mon[43961]: osdmap e123: 12 total, 12 up, 11 in 2024-05-15T12:48:41.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:40 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4085732478' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:48:41.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:40 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:48:41.771 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 31 16 50392 50376 101.552 98.1875 0.00564868 0.0098422 2024-05-15T12:48:42.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:41 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:48:42.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:41 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T12:48:42.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:41 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:48:42.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:41 smithi177 ceph-mon[44303]: osdmap e124: 12 total, 12 up, 11 in 2024-05-15T12:48:42.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:41 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:48:42.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:41 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T12:48:42.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:41 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:48:42.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:41 smithi027 ceph-mon[39795]: osdmap e124: 12 total, 12 up, 11 in 2024-05-15T12:48:42.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:41 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:48:42.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:41 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T12:48:42.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:41 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:48:42.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:41 smithi120 ceph-mon[43961]: osdmap e124: 12 total, 12 up, 11 in 2024-05-15T12:48:42.771 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 32 16 51893 51877 101.31 93.8125 0.00259699 0.00986578 2024-05-15T12:48:43.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:42 smithi027 ceph-mon[39795]: pgmap v395: 17 pgs: 1 peering, 2 active+recovering+undersized+remapped, 14 active+clean; 2.9 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 104 MiB/s wr, 1.66k op/s; 1713/141057 objects misplaced (1.214%); 23 MiB/s, 371 objects/s recovering 2024-05-15T12:48:43.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:42 smithi027 ceph-mon[39795]: osdmap e125: 12 total, 12 up, 11 in 2024-05-15T12:48:43.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:42 smithi120 ceph-mon[43961]: pgmap v395: 17 pgs: 1 peering, 2 active+recovering+undersized+remapped, 14 active+clean; 2.9 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 104 MiB/s wr, 1.66k op/s; 1713/141057 objects misplaced (1.214%); 23 MiB/s, 371 objects/s recovering 2024-05-15T12:48:43.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:42 smithi120 ceph-mon[43961]: osdmap e125: 12 total, 12 up, 11 in 2024-05-15T12:48:43.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:42 smithi177 ceph-mon[44303]: pgmap v395: 17 pgs: 1 peering, 2 active+recovering+undersized+remapped, 14 active+clean; 2.9 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 104 MiB/s wr, 1.66k op/s; 1713/141057 objects misplaced (1.214%); 23 MiB/s, 371 objects/s recovering 2024-05-15T12:48:43.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:42 smithi177 ceph-mon[44303]: osdmap e125: 12 total, 12 up, 11 in 2024-05-15T12:48:43.772 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 33 16 53602 53586 101.476 106.812 0.00241203 0.00984214 2024-05-15T12:48:44.380 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T12:48:44.771 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 34 16 54061 54045 99.3352 28.6875 0.00281428 0.0098415 2024-05-15T12:48:45.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:44 smithi027 ceph-mon[39795]: pgmap v397: 17 pgs: 1 peering, 2 active+recovering+undersized+remapped, 14 active+clean; 3.1 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 89 MiB/s wr, 1.42k op/s; 888/150054 objects misplaced (0.592%); 30 MiB/s, 475 objects/s recovering 2024-05-15T12:48:45.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:44 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:48:45.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:44 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:48:45.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:44 smithi027 ceph-mon[39795]: osdmap e126: 12 total, 12 up, 11 in 2024-05-15T12:48:45.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:44 smithi120 ceph-mon[43961]: pgmap v397: 17 pgs: 1 peering, 2 active+recovering+undersized+remapped, 14 active+clean; 3.1 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 89 MiB/s wr, 1.42k op/s; 888/150054 objects misplaced (0.592%); 30 MiB/s, 475 objects/s recovering 2024-05-15T12:48:45.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:44 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:48:45.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:44 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:48:45.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:44 smithi120 ceph-mon[43961]: osdmap e126: 12 total, 12 up, 11 in 2024-05-15T12:48:45.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:44 smithi177 ceph-mon[44303]: pgmap v397: 17 pgs: 1 peering, 2 active+recovering+undersized+remapped, 14 active+clean; 3.1 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 89 MiB/s wr, 1.42k op/s; 888/150054 objects misplaced (0.592%); 30 MiB/s, 475 objects/s recovering 2024-05-15T12:48:45.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:44 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:48:45.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:44 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:48:45.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:44 smithi177 ceph-mon[44303]: osdmap e126: 12 total, 12 up, 11 in 2024-05-15T12:48:45.771 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 35 16 55478 55462 99.0271 88.5625 0.00912574 0.010094 2024-05-15T12:48:46.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:45 smithi027 ceph-mon[39795]: osdmap e127: 12 total, 12 up, 11 in 2024-05-15T12:48:46.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:45 smithi120 ceph-mon[43961]: osdmap e127: 12 total, 12 up, 11 in 2024-05-15T12:48:46.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:45 smithi177 ceph-mon[44303]: osdmap e127: 12 total, 12 up, 11 in 2024-05-15T12:48:46.771 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 36 16 57267 57251 99.382 111.812 0.0076082 0.0100577 2024-05-15T12:48:46.867 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:48:46.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:46.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:48:46.974 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:48:47.028 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:47.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:46 smithi027 ceph-mon[39795]: pgmap v400: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 3.2 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s; 888/156261 objects misplaced (0.568%); 11 MiB/s, 177 objects/s recovering 2024-05-15T12:48:47.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:46 smithi120 ceph-mon[43961]: pgmap v400: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 3.2 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s; 888/156261 objects misplaced (0.568%); 11 MiB/s, 177 objects/s recovering 2024-05-15T12:48:47.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:46 smithi177 ceph-mon[44303]: pgmap v400: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 3.2 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s; 888/156261 objects misplaced (0.568%); 11 MiB/s, 177 objects/s recovering 2024-05-15T12:48:47.772 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 37 16 57888 57872 97.7449 38.8125 0.00251085 0.0100271 2024-05-15T12:48:48.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:47 smithi027 ceph-mon[39795]: osdmap e128: 12 total, 12 up, 11 in 2024-05-15T12:48:48.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:47 smithi120 ceph-mon[43961]: osdmap e128: 12 total, 12 up, 11 in 2024-05-15T12:48:48.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:47 smithi177 ceph-mon[44303]: osdmap e128: 12 total, 12 up, 11 in 2024-05-15T12:48:48.772 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 38 15 59105 59090 97.1758 76.125 0.00404862 0.0102865 2024-05-15T12:48:48.795 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:48:48.795 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T12:48:48.795 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 122, 2024-05-15T12:48:48.795 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "deaea04e-12b7-11ef-bc98-c7b262605968", 2024-05-15T12:48:48.795 INFO:teuthology.orchestra.run.smithi027.stdout: "created": "2024-05-15T12:38:21.244556+0000", 2024-05-15T12:48:48.795 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-05-15T12:48:37.830846+0000", 2024-05-15T12:48:48.796 INFO:teuthology.orchestra.run.smithi027.stdout: "last_up_change": "2024-05-15T12:45:50.876090+0000", 2024-05-15T12:48:48.796 INFO:teuthology.orchestra.run.smithi027.stdout: "last_in_change": "2024-05-15T12:48:26.875487+0000", 2024-05-15T12:48:48.796 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-15T12:48:48.796 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_num": 5804032, 2024-05-15T12:48:48.796 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_set": [ 2024-05-15T12:48:48.796 INFO:teuthology.orchestra.run.smithi027.stdout: "nodeep-scrub", 2024-05-15T12:48:48.796 INFO:teuthology.orchestra.run.smithi027.stdout: "pglog_hardlimit", 2024-05-15T12:48:48.796 INFO:teuthology.orchestra.run.smithi027.stdout: "purged_snapdirs", 2024-05-15T12:48:48.796 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_deletes", 2024-05-15T12:48:48.796 INFO:teuthology.orchestra.run.smithi027.stdout: "sortbitwise" 2024-05-15T12:48:48.796 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T12:48:48.796 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_version": 26, 2024-05-15T12:48:48.796 INFO:teuthology.orchestra.run.smithi027.stdout: "full_ratio": 0.94999998807907104, 2024-05-15T12:48:48.797 INFO:teuthology.orchestra.run.smithi027.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-05-15T12:48:48.797 INFO:teuthology.orchestra.run.smithi027.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-15T12:48:48.797 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_snapshot": "", 2024-05-15T12:48:48.797 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_max": 2, 2024-05-15T12:48:48.797 INFO:teuthology.orchestra.run.smithi027.stdout: "max_osd": 12, 2024-05-15T12:48:48.797 INFO:teuthology.orchestra.run.smithi027.stdout: "require_min_compat_client": "luminous", 2024-05-15T12:48:48.797 INFO:teuthology.orchestra.run.smithi027.stdout: "min_compat_client": "jewel", 2024-05-15T12:48:48.797 INFO:teuthology.orchestra.run.smithi027.stdout: "require_osd_release": "squid", 2024-05-15T12:48:48.797 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_crimson": false, 2024-05-15T12:48:48.797 INFO:teuthology.orchestra.run.smithi027.stdout: "pools": [ 2024-05-15T12:48:48.797 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.797 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 1, 2024-05-15T12:48:48.797 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": ".mgr", 2024-05-15T12:48:48.798 INFO:teuthology.orchestra.run.smithi027.stdout: "create_time": "2024-05-15T12:41:57.948613+0000", 2024-05-15T12:48:48.798 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-15T12:48:48.798 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_names": "hashpspool", 2024-05-15T12:48:48.798 INFO:teuthology.orchestra.run.smithi027.stdout: "type": 1, 2024-05-15T12:48:48.798 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 3, 2024-05-15T12:48:48.798 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": 2, 2024-05-15T12:48:48.798 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_rule": 0, 2024-05-15T12:48:48.798 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_count": 0, 2024-05-15T12:48:48.798 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_target": 0, 2024-05-15T12:48:48.798 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T12:48:48.798 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T12:48:48.798 INFO:teuthology.orchestra.run.smithi027.stdout: "object_hash": 2, 2024-05-15T12:48:48.798 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscale_mode": "off", 2024-05-15T12:48:48.798 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": 1, 2024-05-15T12:48:48.799 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num": 1, 2024-05-15T12:48:48.799 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num_target": 1, 2024-05-15T12:48:48.799 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_target": 1, 2024-05-15T12:48:48.799 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_pending": 1, 2024-05-15T12:48:48.799 INFO:teuthology.orchestra.run.smithi027.stdout: "last_pg_merge_meta": { 2024-05-15T12:48:48.799 INFO:teuthology.orchestra.run.smithi027.stdout: "source_pgid": "0.0", 2024-05-15T12:48:48.799 INFO:teuthology.orchestra.run.smithi027.stdout: "ready_epoch": 0, 2024-05-15T12:48:48.799 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_started": 0, 2024-05-15T12:48:48.799 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_clean": 0, 2024-05-15T12:48:48.799 INFO:teuthology.orchestra.run.smithi027.stdout: "source_version": "0'0", 2024-05-15T12:48:48.799 INFO:teuthology.orchestra.run.smithi027.stdout: "target_version": "0'0" 2024-05-15T12:48:48.799 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.799 INFO:teuthology.orchestra.run.smithi027.stdout: "last_change": "21", 2024-05-15T12:48:48.799 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend": "0", 2024-05-15T12:48:48.800 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T12:48:48.800 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T12:48:48.800 INFO:teuthology.orchestra.run.smithi027.stdout: "auid": 0, 2024-05-15T12:48:48.800 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_mode": "selfmanaged", 2024-05-15T12:48:48.800 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_seq": 0, 2024-05-15T12:48:48.800 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_epoch": 0, 2024-05-15T12:48:48.800 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_snaps": [], 2024-05-15T12:48:48.800 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps": "[]", 2024-05-15T12:48:48.800 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_bytes": 0, 2024-05-15T12:48:48.800 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_objects": 0, 2024-05-15T12:48:48.800 INFO:teuthology.orchestra.run.smithi027.stdout: "tiers": [], 2024-05-15T12:48:48.800 INFO:teuthology.orchestra.run.smithi027.stdout: "tier_of": -1, 2024-05-15T12:48:48.800 INFO:teuthology.orchestra.run.smithi027.stdout: "read_tier": -1, 2024-05-15T12:48:48.800 INFO:teuthology.orchestra.run.smithi027.stdout: "write_tier": -1, 2024-05-15T12:48:48.801 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_mode": "none", 2024-05-15T12:48:48.801 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_bytes": 0, 2024-05-15T12:48:48.801 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_objects": 0, 2024-05-15T12:48:48.801 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T12:48:48.801 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T12:48:48.801 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T12:48:48.801 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_flush_age": 0, 2024-05-15T12:48:48.801 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_evict_age": 0, 2024-05-15T12:48:48.801 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profile": "", 2024-05-15T12:48:48.801 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_params": { 2024-05-15T12:48:48.801 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "none" 2024-05-15T12:48:48.801 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.801 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_period": 0, 2024-05-15T12:48:48.801 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_count": 0, 2024-05-15T12:48:48.802 INFO:teuthology.orchestra.run.smithi027.stdout: "use_gmt_hitset": true, 2024-05-15T12:48:48.802 INFO:teuthology.orchestra.run.smithi027.stdout: "min_read_recency_for_promote": 0, 2024-05-15T12:48:48.802 INFO:teuthology.orchestra.run.smithi027.stdout: "min_write_recency_for_promote": 0, 2024-05-15T12:48:48.802 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T12:48:48.802 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_search_last_n": 0, 2024-05-15T12:48:48.802 INFO:teuthology.orchestra.run.smithi027.stdout: "grade_table": [], 2024-05-15T12:48:48.802 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_width": 0, 2024-05-15T12:48:48.802 INFO:teuthology.orchestra.run.smithi027.stdout: "expected_num_objects": 0, 2024-05-15T12:48:48.802 INFO:teuthology.orchestra.run.smithi027.stdout: "fast_read": false, 2024-05-15T12:48:48.802 INFO:teuthology.orchestra.run.smithi027.stdout: "options": { 2024-05-15T12:48:48.802 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_max": 32, 2024-05-15T12:48:48.802 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_min": 1 2024-05-15T12:48:48.802 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.803 INFO:teuthology.orchestra.run.smithi027.stdout: "application_metadata": { 2024-05-15T12:48:48.803 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": {} 2024-05-15T12:48:48.803 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.803 INFO:teuthology.orchestra.run.smithi027.stdout: "read_balance": { 2024-05-15T12:48:48.803 INFO:teuthology.orchestra.run.smithi027.stdout: "score_type": "Fair distribution", 2024-05-15T12:48:48.803 INFO:teuthology.orchestra.run.smithi027.stdout: "score_acting": 12, 2024-05-15T12:48:48.803 INFO:teuthology.orchestra.run.smithi027.stdout: "score_stable": 12, 2024-05-15T12:48:48.803 INFO:teuthology.orchestra.run.smithi027.stdout: "optimal_score": 0.25, 2024-05-15T12:48:48.803 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_acting": 3, 2024-05-15T12:48:48.803 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_stable": 3, 2024-05-15T12:48:48.803 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-05-15T12:48:48.803 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity": 1, 2024-05-15T12:48:48.803 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity_weighted": 1 2024-05-15T12:48:48.803 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.804 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.804 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.804 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 2, 2024-05-15T12:48:48.804 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": "unique_pool_0", 2024-05-15T12:48:48.804 INFO:teuthology.orchestra.run.smithi027.stdout: "create_time": "2024-05-15T12:47:57.372152+0000", 2024-05-15T12:48:48.804 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-15T12:48:48.804 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_names": "hashpspool", 2024-05-15T12:48:48.804 INFO:teuthology.orchestra.run.smithi027.stdout: "type": 1, 2024-05-15T12:48:48.804 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 3, 2024-05-15T12:48:48.804 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": 2, 2024-05-15T12:48:48.804 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_rule": 0, 2024-05-15T12:48:48.804 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_count": 0, 2024-05-15T12:48:48.804 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_target": 0, 2024-05-15T12:48:48.804 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T12:48:48.805 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T12:48:48.805 INFO:teuthology.orchestra.run.smithi027.stdout: "object_hash": 2, 2024-05-15T12:48:48.805 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscale_mode": "off", 2024-05-15T12:48:48.805 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": 16, 2024-05-15T12:48:48.805 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num": 16, 2024-05-15T12:48:48.805 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num_target": 16, 2024-05-15T12:48:48.805 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_target": 16, 2024-05-15T12:48:48.805 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_pending": 16, 2024-05-15T12:48:48.805 INFO:teuthology.orchestra.run.smithi027.stdout: "last_pg_merge_meta": { 2024-05-15T12:48:48.805 INFO:teuthology.orchestra.run.smithi027.stdout: "source_pgid": "0.0", 2024-05-15T12:48:48.805 INFO:teuthology.orchestra.run.smithi027.stdout: "ready_epoch": 0, 2024-05-15T12:48:48.805 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_started": 0, 2024-05-15T12:48:48.805 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_clean": 0, 2024-05-15T12:48:48.806 INFO:teuthology.orchestra.run.smithi027.stdout: "source_version": "0'0", 2024-05-15T12:48:48.806 INFO:teuthology.orchestra.run.smithi027.stdout: "target_version": "0'0" 2024-05-15T12:48:48.806 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.806 INFO:teuthology.orchestra.run.smithi027.stdout: "last_change": "115", 2024-05-15T12:48:48.806 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend": "0", 2024-05-15T12:48:48.806 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T12:48:48.806 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T12:48:48.806 INFO:teuthology.orchestra.run.smithi027.stdout: "auid": 0, 2024-05-15T12:48:48.806 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_mode": "selfmanaged", 2024-05-15T12:48:48.806 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_seq": 0, 2024-05-15T12:48:48.806 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_epoch": 0, 2024-05-15T12:48:48.806 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_snaps": [], 2024-05-15T12:48:48.806 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps": "[]", 2024-05-15T12:48:48.806 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_bytes": 0, 2024-05-15T12:48:48.807 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_objects": 0, 2024-05-15T12:48:48.807 INFO:teuthology.orchestra.run.smithi027.stdout: "tiers": [], 2024-05-15T12:48:48.807 INFO:teuthology.orchestra.run.smithi027.stdout: "tier_of": -1, 2024-05-15T12:48:48.807 INFO:teuthology.orchestra.run.smithi027.stdout: "read_tier": -1, 2024-05-15T12:48:48.807 INFO:teuthology.orchestra.run.smithi027.stdout: "write_tier": -1, 2024-05-15T12:48:48.807 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_mode": "none", 2024-05-15T12:48:48.807 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_bytes": 0, 2024-05-15T12:48:48.807 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_objects": 0, 2024-05-15T12:48:48.807 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T12:48:48.807 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T12:48:48.807 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T12:48:48.807 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_flush_age": 0, 2024-05-15T12:48:48.807 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_evict_age": 0, 2024-05-15T12:48:48.807 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profile": "", 2024-05-15T12:48:48.808 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_params": { 2024-05-15T12:48:48.808 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "none" 2024-05-15T12:48:48.808 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.808 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_period": 0, 2024-05-15T12:48:48.808 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_count": 0, 2024-05-15T12:48:48.808 INFO:teuthology.orchestra.run.smithi027.stdout: "use_gmt_hitset": true, 2024-05-15T12:48:48.808 INFO:teuthology.orchestra.run.smithi027.stdout: "min_read_recency_for_promote": 0, 2024-05-15T12:48:48.808 INFO:teuthology.orchestra.run.smithi027.stdout: "min_write_recency_for_promote": 0, 2024-05-15T12:48:48.808 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T12:48:48.808 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_search_last_n": 0, 2024-05-15T12:48:48.808 INFO:teuthology.orchestra.run.smithi027.stdout: "grade_table": [], 2024-05-15T12:48:48.808 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_width": 0, 2024-05-15T12:48:48.808 INFO:teuthology.orchestra.run.smithi027.stdout: "expected_num_objects": 0, 2024-05-15T12:48:48.808 INFO:teuthology.orchestra.run.smithi027.stdout: "fast_read": false, 2024-05-15T12:48:48.809 INFO:teuthology.orchestra.run.smithi027.stdout: "options": {}, 2024-05-15T12:48:48.809 INFO:teuthology.orchestra.run.smithi027.stdout: "application_metadata": { 2024-05-15T12:48:48.809 INFO:teuthology.orchestra.run.smithi027.stdout: "rados": {} 2024-05-15T12:48:48.809 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.809 INFO:teuthology.orchestra.run.smithi027.stdout: "read_balance": { 2024-05-15T12:48:48.809 INFO:teuthology.orchestra.run.smithi027.stdout: "score_type": "Fair distribution", 2024-05-15T12:48:48.809 INFO:teuthology.orchestra.run.smithi027.stdout: "score_acting": 2.2400000095367432, 2024-05-15T12:48:48.809 INFO:teuthology.orchestra.run.smithi027.stdout: "score_stable": 2.2400000095367432, 2024-05-15T12:48:48.809 INFO:teuthology.orchestra.run.smithi027.stdout: "optimal_score": 0.92000001668930054, 2024-05-15T12:48:48.809 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_acting": 2.059999942779541, 2024-05-15T12:48:48.809 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_stable": 2.059999942779541, 2024-05-15T12:48:48.809 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-05-15T12:48:48.809 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity": 1, 2024-05-15T12:48:48.809 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity_weighted": 1 2024-05-15T12:48:48.810 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.810 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.810 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T12:48:48.810 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T12:48:48.810 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.810 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 0, 2024-05-15T12:48:48.810 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "64fb3ad2-08a7-4106-a07a-6c776c877faf", 2024-05-15T12:48:48.810 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:48:48.810 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:48:48.810 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:48:48.810 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:48:48.810 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:48:48.810 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:48:48.811 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 8, 2024-05-15T12:48:48.811 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 58, 2024-05-15T12:48:48.811 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:48:48.811 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:48:48.811 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:48:48.811 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.811 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.811 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.811 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6802", 2024-05-15T12:48:48.811 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:48:48.811 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.811 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.811 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.811 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6803", 2024-05-15T12:48:48.811 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:48:48.812 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.812 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.812 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:48:48.812 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.812 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.812 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.812 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6804", 2024-05-15T12:48:48.812 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:48:48.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.812 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.812 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.812 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6805", 2024-05-15T12:48:48.813 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:48:48.813 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.813 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.813 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.813 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:48:48.813 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.813 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.813 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.813 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6808", 2024-05-15T12:48:48.813 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:48:48.813 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.813 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.813 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.813 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6809", 2024-05-15T12:48:48.813 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:48:48.814 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.814 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.814 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.814 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:48:48.814 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.814 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.814 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.814 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6806", 2024-05-15T12:48:48.814 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:48:48.814 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.814 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.814 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.814 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6807", 2024-05-15T12:48:48.815 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:48:48.815 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.815 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.815 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.815 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6803/4239116935", 2024-05-15T12:48:48.815 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6805/4239116935", 2024-05-15T12:48:48.815 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6809/4239116935", 2024-05-15T12:48:48.815 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6807/4239116935", 2024-05-15T12:48:48.815 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:48:48.815 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:48:48.815 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:48:48.815 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.815 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.815 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.816 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 1, 2024-05-15T12:48:48.816 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "9c16dd90-9d51-48b8-a296-1c8879ba1bc8", 2024-05-15T12:48:48.816 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:48:48.816 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:48:48.816 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:48:48.816 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:48:48.816 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:48:48.816 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:48:48.816 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 13, 2024-05-15T12:48:48.816 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 118, 2024-05-15T12:48:48.816 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:48:48.816 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:48:48.816 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:48:48.816 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.817 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.817 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.817 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6810", 2024-05-15T12:48:48.817 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:48:48.817 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.817 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.817 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.817 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6811", 2024-05-15T12:48:48.817 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:48:48.817 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.818 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.818 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.818 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:48:48.818 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.818 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.818 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.818 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6812", 2024-05-15T12:48:48.818 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:48:48.818 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.818 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.818 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.818 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6813", 2024-05-15T12:48:48.818 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:48:48.818 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.819 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.819 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.819 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:48:48.819 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.819 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.819 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.819 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6816", 2024-05-15T12:48:48.819 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:48:48.819 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.819 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.819 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.819 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6817", 2024-05-15T12:48:48.819 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:48:48.819 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.820 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.820 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.820 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:48:48.820 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.820 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.820 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.820 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6814", 2024-05-15T12:48:48.820 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:48:48.820 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.820 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.820 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.820 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6815", 2024-05-15T12:48:48.820 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:48:48.820 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.821 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.821 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.821 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6811/1466270637", 2024-05-15T12:48:48.821 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6813/1466270637", 2024-05-15T12:48:48.821 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6817/1466270637", 2024-05-15T12:48:48.821 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6815/1466270637", 2024-05-15T12:48:48.821 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:48:48.821 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:48:48.821 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:48:48.821 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.821 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.821 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.821 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 2, 2024-05-15T12:48:48.821 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "cbe3ba13-3ea3-4891-8d2d-6b85581a732c", 2024-05-15T12:48:48.822 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:48:48.822 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:48:48.822 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:48:48.822 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:48:48.822 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:48:48.822 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:48:48.822 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 18, 2024-05-15T12:48:48.822 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 110, 2024-05-15T12:48:48.822 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:48:48.822 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:48:48.822 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:48:48.822 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.822 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.822 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.823 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6818", 2024-05-15T12:48:48.823 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:48:48.823 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.823 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.823 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.823 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6819", 2024-05-15T12:48:48.823 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:48:48.823 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.823 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.823 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.823 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:48:48.823 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.823 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.824 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.824 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6820", 2024-05-15T12:48:48.824 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:48:48.824 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.824 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.824 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.824 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6821", 2024-05-15T12:48:48.824 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:48:48.824 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.824 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.824 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.824 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:48:48.824 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.824 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.824 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.825 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6824", 2024-05-15T12:48:48.825 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:48:48.825 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.825 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.825 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.825 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6825", 2024-05-15T12:48:48.825 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:48:48.825 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.825 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.825 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.825 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:48:48.825 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.825 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.825 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.826 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6822", 2024-05-15T12:48:48.826 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:48:48.826 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.826 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.826 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.826 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6823", 2024-05-15T12:48:48.826 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:48:48.826 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.826 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.826 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.826 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6819/3285413897", 2024-05-15T12:48:48.826 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6821/3285413897", 2024-05-15T12:48:48.826 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6825/3285413897", 2024-05-15T12:48:48.827 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6823/3285413897", 2024-05-15T12:48:48.827 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:48:48.827 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:48:48.827 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:48:48.827 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.827 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.827 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.827 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 3, 2024-05-15T12:48:48.827 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "2e83735b-b944-4f87-8080-b994d4b96897", 2024-05-15T12:48:48.827 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:48:48.827 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:48:48.827 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:48:48.827 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:48:48.827 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:48:48.828 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:48:48.828 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 26, 2024-05-15T12:48:48.828 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 0, 2024-05-15T12:48:48.828 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:48:48.828 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:48:48.828 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:48:48.828 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.828 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.828 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.828 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6826", 2024-05-15T12:48:48.828 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:48:48.828 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.828 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.828 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.829 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6827", 2024-05-15T12:48:48.829 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:48:48.829 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.829 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.829 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.829 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:48:48.829 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.829 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.829 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.829 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6828", 2024-05-15T12:48:48.829 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:48:48.829 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.829 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.829 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.829 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6829", 2024-05-15T12:48:48.830 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:48:48.830 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.830 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.830 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.830 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:48:48.830 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.830 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.830 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.830 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6832", 2024-05-15T12:48:48.830 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:48:48.830 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.830 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.830 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.831 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6833", 2024-05-15T12:48:48.831 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:48:48.831 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.831 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.831 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.831 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:48:48.831 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.831 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.831 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.831 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6830", 2024-05-15T12:48:48.831 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:48:48.831 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.831 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.831 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.831 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6831", 2024-05-15T12:48:48.832 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:48:48.832 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.832 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.832 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.832 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6827/3008975244", 2024-05-15T12:48:48.832 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6829/3008975244", 2024-05-15T12:48:48.832 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6833/3008975244", 2024-05-15T12:48:48.832 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6831/3008975244", 2024-05-15T12:48:48.832 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:48:48.832 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:48:48.832 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:48:48.832 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.832 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.832 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.833 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 4, 2024-05-15T12:48:48.833 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "7c39c1dd-3867-46f7-8372-7688fa13b16a", 2024-05-15T12:48:48.833 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:48:48.833 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:48:48.833 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:48:48.833 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:48:48.833 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:48:48.833 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:48:48.833 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 31, 2024-05-15T12:48:48.833 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 0, 2024-05-15T12:48:48.833 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:48:48.833 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:48:48.833 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:48:48.833 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.834 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.834 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.834 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6800", 2024-05-15T12:48:48.834 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:48:48.834 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.834 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.834 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.834 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6801", 2024-05-15T12:48:48.834 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:48:48.834 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.834 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.834 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.834 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:48:48.835 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.835 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.835 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.835 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6802", 2024-05-15T12:48:48.835 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:48:48.835 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.835 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.835 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.835 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6803", 2024-05-15T12:48:48.835 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:48:48.835 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.835 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.835 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.835 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:48:48.836 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.836 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.836 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.836 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6806", 2024-05-15T12:48:48.836 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:48:48.836 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.836 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.836 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.836 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6807", 2024-05-15T12:48:48.836 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:48:48.836 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.836 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.836 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.837 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:48:48.837 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.837 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.837 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.837 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6804", 2024-05-15T12:48:48.837 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:48:48.837 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.837 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.837 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.837 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6805", 2024-05-15T12:48:48.837 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:48:48.837 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.837 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.837 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.838 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6801/3877842957", 2024-05-15T12:48:48.838 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6803/3877842957", 2024-05-15T12:48:48.838 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6807/3877842957", 2024-05-15T12:48:48.838 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6805/3877842957", 2024-05-15T12:48:48.838 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:48:48.838 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:48:48.838 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:48:48.838 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.838 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.838 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.838 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 5, 2024-05-15T12:48:48.838 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "a6dc220f-aec0-4454-9695-b44c0b9770e0", 2024-05-15T12:48:48.838 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:48:48.838 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:48:48.839 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:48:48.839 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:48:48.839 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:48:48.839 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:48:48.839 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 40, 2024-05-15T12:48:48.839 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 110, 2024-05-15T12:48:48.839 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:48:48.839 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:48:48.839 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:48:48.839 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.839 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.839 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.839 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6808", 2024-05-15T12:48:48.840 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:48:48.840 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.840 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.840 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.840 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6809", 2024-05-15T12:48:48.840 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:48:48.840 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.840 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.840 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.840 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:48:48.840 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.840 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.840 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.840 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6810", 2024-05-15T12:48:48.841 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:48:48.841 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.841 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.841 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.841 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6811", 2024-05-15T12:48:48.841 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:48:48.841 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.841 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.841 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.841 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:48:48.841 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.841 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.841 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.841 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6814", 2024-05-15T12:48:48.842 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:48:48.842 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.842 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.842 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.842 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6815", 2024-05-15T12:48:48.842 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:48:48.842 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.842 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.842 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.842 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:48:48.842 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.842 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.842 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.842 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6812", 2024-05-15T12:48:48.843 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:48:48.843 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.843 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.843 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.843 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6813", 2024-05-15T12:48:48.843 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:48:48.843 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.843 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.843 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.843 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6809/1600179183", 2024-05-15T12:48:48.843 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6811/1600179183", 2024-05-15T12:48:48.843 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6815/1600179183", 2024-05-15T12:48:48.843 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6813/1600179183", 2024-05-15T12:48:48.843 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:48:48.843 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:48:48.844 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:48:48.844 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.844 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.844 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.844 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 6, 2024-05-15T12:48:48.844 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b", 2024-05-15T12:48:48.844 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:48:48.844 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:48:48.844 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:48:48.844 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:48:48.844 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:48:48.844 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:48:48.844 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 48, 2024-05-15T12:48:48.844 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 118, 2024-05-15T12:48:48.845 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:48:48.845 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:48:48.845 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:48:48.845 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.845 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.845 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.845 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6816", 2024-05-15T12:48:48.845 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:48:48.845 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.845 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.845 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.845 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6817", 2024-05-15T12:48:48.845 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:48:48.845 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.846 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.846 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.846 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:48:48.846 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.846 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.846 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.846 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6818", 2024-05-15T12:48:48.846 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:48:48.846 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.846 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.846 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.846 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6819", 2024-05-15T12:48:48.846 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:48:48.846 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.847 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.847 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.847 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:48:48.847 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.847 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.847 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.847 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6822", 2024-05-15T12:48:48.847 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:48:48.847 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.847 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.847 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.847 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6823", 2024-05-15T12:48:48.847 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:48:48.847 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.848 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.848 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.848 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:48:48.848 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.848 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.848 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.848 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6820", 2024-05-15T12:48:48.848 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:48:48.848 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.848 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.848 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.848 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6821", 2024-05-15T12:48:48.848 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:48:48.848 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.849 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.849 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.849 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6817/3323706735", 2024-05-15T12:48:48.849 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6819/3323706735", 2024-05-15T12:48:48.849 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6823/3323706735", 2024-05-15T12:48:48.849 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6821/3323706735", 2024-05-15T12:48:48.849 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:48:48.849 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:48:48.849 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:48:48.849 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.849 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.849 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.849 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 7, 2024-05-15T12:48:48.849 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "6ef085d8-f59b-4033-8280-2c8f78693e30", 2024-05-15T12:48:48.850 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:48:48.850 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:48:48.850 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:48:48.850 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:48:48.850 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:48:48.850 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:48:48.850 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 56, 2024-05-15T12:48:48.850 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 118, 2024-05-15T12:48:48.850 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:48:48.850 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:48:48.850 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:48:48.850 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.850 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.851 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.851 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6824", 2024-05-15T12:48:48.851 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:48:48.851 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.851 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.851 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.851 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6825", 2024-05-15T12:48:48.851 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:48:48.851 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.851 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.851 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.851 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:48:48.851 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.852 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.852 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.852 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6826", 2024-05-15T12:48:48.852 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:48:48.852 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.852 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.852 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.852 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6827", 2024-05-15T12:48:48.852 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:48:48.852 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.852 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.852 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.853 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:48:48.853 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.853 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.853 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.853 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6830", 2024-05-15T12:48:48.853 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:48:48.853 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.853 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.853 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.853 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6831", 2024-05-15T12:48:48.853 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:48:48.853 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.853 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.853 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.854 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:48:48.854 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.854 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.854 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.854 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6828", 2024-05-15T12:48:48.854 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:48:48.854 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.854 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.854 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.854 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6829", 2024-05-15T12:48:48.854 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:48:48.854 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.854 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.854 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.855 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6825/2960616153", 2024-05-15T12:48:48.855 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6827/2960616153", 2024-05-15T12:48:48.855 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6831/2960616153", 2024-05-15T12:48:48.855 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6829/2960616153", 2024-05-15T12:48:48.855 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:48:48.855 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:48:48.855 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:48:48.855 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.855 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.855 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.855 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 8, 2024-05-15T12:48:48.855 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "26514b11-d02b-46e9-a4e8-3da193fcdbcc", 2024-05-15T12:48:48.855 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:48:48.855 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:48:48.856 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:48:48.856 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:48:48.856 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:48:48.856 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:48:48.856 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 65, 2024-05-15T12:48:48.856 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 110, 2024-05-15T12:48:48.856 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:48:48.856 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:48:48.856 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:48:48.856 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.856 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.856 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.856 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6800", 2024-05-15T12:48:48.856 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:48:48.857 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.857 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.857 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.857 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6801", 2024-05-15T12:48:48.857 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:48:48.857 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.857 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.857 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.857 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:48:48.857 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.857 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.857 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.857 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6802", 2024-05-15T12:48:48.858 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:48:48.858 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.858 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.858 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.858 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6803", 2024-05-15T12:48:48.858 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:48:48.858 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.858 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.858 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.858 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:48:48.858 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.858 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.858 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.859 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6806", 2024-05-15T12:48:48.859 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:48:48.859 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.859 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.859 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.859 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6807", 2024-05-15T12:48:48.859 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:48:48.859 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.859 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.859 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.859 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:48:48.859 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.859 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.859 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.860 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6804", 2024-05-15T12:48:48.860 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:48:48.860 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.860 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.860 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.860 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6805", 2024-05-15T12:48:48.860 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:48:48.860 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.860 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.860 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.860 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6801/1132459315", 2024-05-15T12:48:48.860 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6803/1132459315", 2024-05-15T12:48:48.860 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6807/1132459315", 2024-05-15T12:48:48.860 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6805/1132459315", 2024-05-15T12:48:48.861 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:48:48.861 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:48:48.861 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:48:48.861 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.861 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.861 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.861 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 9, 2024-05-15T12:48:48.861 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "6f9497bc-e962-409d-8a2c-0c634cc1124f", 2024-05-15T12:48:48.861 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:48:48.861 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:48:48.861 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:48:48.861 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:48:48.861 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:48:48.861 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:48:48.861 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 76, 2024-05-15T12:48:48.862 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 110, 2024-05-15T12:48:48.862 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:48:48.862 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:48:48.862 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:48:48.862 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.862 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.862 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.862 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6808", 2024-05-15T12:48:48.862 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:48:48.862 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.862 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.862 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.862 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6809", 2024-05-15T12:48:48.863 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:48:48.863 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.863 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.863 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.863 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:48:48.863 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.863 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.863 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.863 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6810", 2024-05-15T12:48:48.863 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:48:48.863 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.863 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.863 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.863 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6811", 2024-05-15T12:48:48.863 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:48:48.864 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.864 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.864 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:48:48.864 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.864 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.864 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.864 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6814", 2024-05-15T12:48:48.864 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:48:48.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.864 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.864 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.864 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6815", 2024-05-15T12:48:48.864 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:48:48.865 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.865 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.865 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.865 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:48:48.865 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.865 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.865 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.865 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6812", 2024-05-15T12:48:48.865 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:48:48.865 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.865 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.865 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.865 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6813", 2024-05-15T12:48:48.865 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:48:48.866 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.866 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.866 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.866 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6809/2710457646", 2024-05-15T12:48:48.866 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6811/2710457646", 2024-05-15T12:48:48.866 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6815/2710457646", 2024-05-15T12:48:48.866 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6813/2710457646", 2024-05-15T12:48:48.866 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:48:48.866 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:48:48.866 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:48:48.866 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.866 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.866 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.866 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 10, 2024-05-15T12:48:48.867 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "f852876f-c7d1-4039-90c4-5ab09dbd3e32", 2024-05-15T12:48:48.867 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:48:48.867 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 0, 2024-05-15T12:48:48.867 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 0, 2024-05-15T12:48:48.867 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:48:48.867 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:48:48.867 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:48:48.867 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 81, 2024-05-15T12:48:48.867 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 0, 2024-05-15T12:48:48.867 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:48:48.867 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:48:48.867 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:48:48.867 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.867 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.867 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.868 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6816", 2024-05-15T12:48:48.868 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:48:48.868 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.868 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.868 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.868 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6817", 2024-05-15T12:48:48.868 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:48:48.868 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.868 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.868 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.868 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:48:48.868 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.868 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.868 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.868 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6818", 2024-05-15T12:48:48.869 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:48:48.869 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.869 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.869 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.869 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6819", 2024-05-15T12:48:48.869 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:48:48.869 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.869 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.869 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.869 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:48:48.869 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.869 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.869 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.869 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6822", 2024-05-15T12:48:48.870 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:48:48.870 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.870 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.870 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.870 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6823", 2024-05-15T12:48:48.870 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:48:48.870 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.870 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.870 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.870 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:48:48.870 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.870 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.870 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.871 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6820", 2024-05-15T12:48:48.871 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:48:48.871 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.871 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.871 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.871 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6821", 2024-05-15T12:48:48.871 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:48:48.871 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.871 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.871 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.871 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6817/3236900263", 2024-05-15T12:48:48.871 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6819/3236900263", 2024-05-15T12:48:48.871 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6823/3236900263", 2024-05-15T12:48:48.871 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6821/3236900263", 2024-05-15T12:48:48.872 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:48:48.872 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:48:48.872 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:48:48.872 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.872 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.872 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.872 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 11, 2024-05-15T12:48:48.872 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "cccc53c5-8476-4d55-b160-4bdc5cf92902", 2024-05-15T12:48:48.872 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:48:48.872 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:48:48.872 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:48:48.872 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:48:48.872 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:48:48.872 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:48:48.873 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 90, 2024-05-15T12:48:48.873 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 110, 2024-05-15T12:48:48.873 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:48:48.873 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:48:48.873 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:48:48.873 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.873 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.873 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.873 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6824", 2024-05-15T12:48:48.873 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:48:48.873 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.873 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.873 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.873 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6825", 2024-05-15T12:48:48.874 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:48:48.874 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.874 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.874 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.874 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:48:48.874 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.874 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.874 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.874 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6826", 2024-05-15T12:48:48.874 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:48:48.874 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.874 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.874 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.874 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6827", 2024-05-15T12:48:48.875 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:48:48.875 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.875 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.875 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.875 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:48:48.875 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.875 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.875 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.875 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6830", 2024-05-15T12:48:48.875 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:48:48.875 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.875 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.875 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.875 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6831", 2024-05-15T12:48:48.876 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:48:48.876 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.876 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.876 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.876 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:48:48.876 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:48:48.876 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.876 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:48:48.876 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6828", 2024-05-15T12:48:48.876 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:48:48.876 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.876 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.876 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:48:48.876 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6829", 2024-05-15T12:48:48.876 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:48:48.877 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.877 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.877 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.877 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6825/681011628", 2024-05-15T12:48:48.877 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6827/681011628", 2024-05-15T12:48:48.877 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6831/681011628", 2024-05-15T12:48:48.877 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6829/681011628", 2024-05-15T12:48:48.877 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:48:48.877 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:48:48.877 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:48:48.877 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.877 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.877 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T12:48:48.877 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_xinfo": [ 2024-05-15T12:48:48.877 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.878 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 0, 2024-05-15T12:48:48.878 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:48:48.878 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:48:48.878 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:48:48.878 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:48:48.878 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:48:48.878 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:47:33.401880+0000", 2024-05-15T12:48:48.878 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:48:48.878 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.878 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.878 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 1, 2024-05-15T12:48:48.878 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:48:48.878 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:48:48.878 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:48:48.879 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:48:48.879 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:48:48.879 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:48:01.354812+0000", 2024-05-15T12:48:48.879 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:48:48.879 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.879 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.879 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 2, 2024-05-15T12:48:48.879 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:48:48.879 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:48:48.879 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:48:48.879 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:48:48.879 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:48:48.879 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:48:00.750859+0000", 2024-05-15T12:48:48.879 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:48:48.879 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.880 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.880 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 3, 2024-05-15T12:48:48.880 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:48:48.880 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:48:48.880 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:48:48.880 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:48:48.880 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:48:48.880 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:47:37.360495+0000", 2024-05-15T12:48:48.880 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:48:48.880 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.880 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.880 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 4, 2024-05-15T12:48:48.880 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:48:48.880 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:48:48.881 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:48:48.881 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:48:48.881 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:48:48.881 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:48:08.671701+0000", 2024-05-15T12:48:48.881 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:48:48.881 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.881 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.881 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 5, 2024-05-15T12:48:48.881 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:48:48.881 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:48:48.881 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:48:48.881 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:48:48.881 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:48:48.881 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:48:17.811164+0000", 2024-05-15T12:48:48.881 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:48:48.882 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.882 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.882 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 6, 2024-05-15T12:48:48.882 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:48:48.882 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:48:48.882 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:48:48.882 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:48:48.882 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:48:48.882 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:47:42.382426+0000", 2024-05-15T12:48:48.882 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:48:48.882 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.882 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.882 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 7, 2024-05-15T12:48:48.882 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:48:48.883 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:48:48.883 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:48:48.883 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:48:48.883 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:48:48.883 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:48:00.104152+0000", 2024-05-15T12:48:48.883 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:48:48.883 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.883 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.883 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 8, 2024-05-15T12:48:48.883 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:48:48.883 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:48:48.883 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:48:48.883 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:48:48.883 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:48:48.883 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:48:30.527775+0000", 2024-05-15T12:48:48.884 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:48:48.884 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.884 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.884 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 9, 2024-05-15T12:48:48.884 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:48:48.884 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:48:48.884 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:48:48.884 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:48:48.884 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:48:48.884 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:47:23.974039+0000", 2024-05-15T12:48:48.884 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:48:48.884 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.884 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.885 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 10, 2024-05-15T12:48:48.885 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:48:48.885 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:48:48.885 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:48:48.885 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:48:48.885 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 65536, 2024-05-15T12:48:48.885 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:47:41.837735+0000", 2024-05-15T12:48:48.885 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:48:48.885 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.885 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.885 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 11, 2024-05-15T12:48:48.885 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:48:48.885 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:48:48.885 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:48:48.886 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:48:48.886 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:48:48.886 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:48:26.576644+0000", 2024-05-15T12:48:48.886 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:48:48.886 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.886 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T12:48:48.886 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap": [], 2024-05-15T12:48:48.886 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap_items": [], 2024-05-15T12:48:48.886 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap_primaries": [], 2024-05-15T12:48:48.886 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_temp": [ 2024-05-15T12:48:48.886 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.886 INFO:teuthology.orchestra.run.smithi027.stdout: "pgid": "2.0", 2024-05-15T12:48:48.886 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T12:48:48.886 INFO:teuthology.orchestra.run.smithi027.stdout: 7, 2024-05-15T12:48:48.887 INFO:teuthology.orchestra.run.smithi027.stdout: 9 2024-05-15T12:48:48.887 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.887 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.887 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.887 INFO:teuthology.orchestra.run.smithi027.stdout: "pgid": "2.4", 2024-05-15T12:48:48.887 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T12:48:48.887 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-05-15T12:48:48.887 INFO:teuthology.orchestra.run.smithi027.stdout: 0 2024-05-15T12:48:48.887 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.887 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.887 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:48.887 INFO:teuthology.orchestra.run.smithi027.stdout: "pgid": "2.7", 2024-05-15T12:48:48.887 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T12:48:48.887 INFO:teuthology.orchestra.run.smithi027.stdout: 6, 2024-05-15T12:48:48.888 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-05-15T12:48:48.888 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:48.888 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.888 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T12:48:48.888 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_temp": [], 2024-05-15T12:48:48.888 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklist": { 2024-05-15T12:48:48.888 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6801/2530617473": "2024-05-16T12:39:19.409195+0000", 2024-05-15T12:48:48.888 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6800/2530617473": "2024-05-16T12:39:19.409195+0000", 2024-05-15T12:48:48.888 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/1880261204": "2024-05-16T12:39:19.409195+0000", 2024-05-15T12:48:48.888 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6801/3441136471": "2024-05-16T12:38:50.931011+0000", 2024-05-15T12:48:48.888 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/4291446247": "2024-05-16T12:39:19.409195+0000", 2024-05-15T12:48:48.888 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/580128209": "2024-05-16T12:38:50.931011+0000", 2024-05-15T12:48:48.888 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6800/3441136471": "2024-05-16T12:38:50.931011+0000", 2024-05-15T12:48:48.888 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/3011229857": "2024-05-16T12:38:50.931011+0000", 2024-05-15T12:48:48.889 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/2109693913": "2024-05-16T12:38:50.931011+0000", 2024-05-15T12:48:48.889 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/2445663024": "2024-05-16T12:39:19.409195+0000" 2024-05-15T12:48:48.889 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.889 INFO:teuthology.orchestra.run.smithi027.stdout: "range_blocklist": {}, 2024-05-15T12:48:48.889 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profiles": { 2024-05-15T12:48:48.889 INFO:teuthology.orchestra.run.smithi027.stdout: "default": { 2024-05-15T12:48:48.889 INFO:teuthology.orchestra.run.smithi027.stdout: "crush-failure-domain": "osd", 2024-05-15T12:48:48.889 INFO:teuthology.orchestra.run.smithi027.stdout: "k": "2", 2024-05-15T12:48:48.889 INFO:teuthology.orchestra.run.smithi027.stdout: "m": "1", 2024-05-15T12:48:48.889 INFO:teuthology.orchestra.run.smithi027.stdout: "plugin": "jerasure", 2024-05-15T12:48:48.889 INFO:teuthology.orchestra.run.smithi027.stdout: "technique": "reed_sol_van" 2024-05-15T12:48:48.889 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.889 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:48.889 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps_queue": [], 2024-05-15T12:48:48.890 INFO:teuthology.orchestra.run.smithi027.stdout: "new_removed_snaps": [], 2024-05-15T12:48:48.890 INFO:teuthology.orchestra.run.smithi027.stdout: "new_purged_snaps": [], 2024-05-15T12:48:48.890 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_node_flags": {}, 2024-05-15T12:48:48.890 INFO:teuthology.orchestra.run.smithi027.stdout: "device_class_flags": {}, 2024-05-15T12:48:48.890 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode": { 2024-05-15T12:48:48.890 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode_enabled": false, 2024-05-15T12:48:48.890 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_bucket_count": 0, 2024-05-15T12:48:48.890 INFO:teuthology.orchestra.run.smithi027.stdout: "degraded_stretch_mode": 0, 2024-05-15T12:48:48.890 INFO:teuthology.orchestra.run.smithi027.stdout: "recovering_stretch_mode": 0, 2024-05-15T12:48:48.890 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode_bucket": 0 2024-05-15T12:48:48.890 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:48.890 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T12:48:49.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:48 smithi027 ceph-mon[39795]: pgmap v402: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 3.3 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 60 MiB/s wr, 956 op/s; 136/162201 objects misplaced (0.084%); 21 MiB/s, 336 objects/s recovering 2024-05-15T12:48:49.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:48 smithi027 ceph-mon[39795]: osdmap e129: 12 total, 12 up, 11 in 2024-05-15T12:48:49.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:48 smithi120 ceph-mon[43961]: pgmap v402: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 3.3 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 60 MiB/s wr, 956 op/s; 136/162201 objects misplaced (0.084%); 21 MiB/s, 336 objects/s recovering 2024-05-15T12:48:49.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:48 smithi120 ceph-mon[43961]: osdmap e129: 12 total, 12 up, 11 in 2024-05-15T12:48:49.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:48 smithi177 ceph-mon[44303]: pgmap v402: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 3.3 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 60 MiB/s wr, 956 op/s; 136/162201 objects misplaced (0.084%); 21 MiB/s, 336 objects/s recovering 2024-05-15T12:48:49.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:48 smithi177 ceph-mon[44303]: osdmap e129: 12 total, 12 up, 11 in 2024-05-15T12:48:49.772 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 39 16 60796 60780 97.3921 105.625 0.00261452 0.0102613 2024-05-15T12:48:50.097 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:48:50.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:50.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:48:50.202 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:48:50.251 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:50.280 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:48:50.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:50.286 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:48:50.387 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:48:50.423 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:50.445 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:48:50.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:50.450 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:48:50.551 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:48:50.577 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:50.598 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:48:50.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:50.601 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:48:50.653 INFO:tasks.thrashosds.thrasher:j is {'epoch': 122, 'fsid': 'deaea04e-12b7-11ef-bc98-c7b262605968', 'created': '2024-05-15T12:38:21.244556+0000', 'modified': '2024-05-15T12:48:37.830846+0000', 'last_up_change': '2024-05-15T12:45:50.876090+0000', 'last_in_change': '2024-05-15T12:48:26.875487+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-15T12:41:57.948613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-05-15T12:47:57.372152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '115', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '64fb3ad2-08a7-4106-a07a-6c776c877faf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 58, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6802', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6803', 'nonce': 4239116935}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6804', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6805', 'nonce': 4239116935}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6808', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6809', 'nonce': 4239116935}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6806', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6807', 'nonce': 4239116935}]}, 'public_addr': '172.21.15.27:6803/4239116935', 'cluster_addr': '172.21.15.27:6805/4239116935', 'heartbeat_back_addr': '172.21.15.27:6809/4239116935', 'heartbeat_front_addr': '172.21.15.27:6807/4239116935', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '9c16dd90-9d51-48b8-a296-1c8879ba1bc8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 118, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6810', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6811', 'nonce': 1466270637}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6812', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6813', 'nonce': 1466270637}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6816', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6817', 'nonce': 1466270637}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6814', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6815', 'nonce': 1466270637}]}, 'public_addr': '172.21.15.27:6811/1466270637', 'cluster_addr': '172.21.15.27:6813/1466270637', 'heartbeat_back_addr': '172.21.15.27:6817/1466270637', 'heartbeat_front_addr': '172.21.15.27:6815/1466270637', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'cbe3ba13-3ea3-4891-8d2d-6b85581a732c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6818', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6819', 'nonce': 3285413897}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6820', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6821', 'nonce': 3285413897}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6824', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6825', 'nonce': 3285413897}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6822', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6823', 'nonce': 3285413897}]}, 'public_addr': '172.21.15.27:6819/3285413897', 'cluster_addr': '172.21.15.27:6821/3285413897', 'heartbeat_back_addr': '172.21.15.27:6825/3285413897', 'heartbeat_front_addr': '172.21.15.27:6823/3285413897', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2e83735b-b944-4f87-8080-b994d4b96897', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6826', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6827', 'nonce': 3008975244}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6828', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6829', 'nonce': 3008975244}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6833', 'nonce': 3008975244}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6830', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6831', 'nonce': 3008975244}]}, 'public_addr': '172.21.15.27:6827/3008975244', 'cluster_addr': '172.21.15.27:6829/3008975244', 'heartbeat_back_addr': '172.21.15.27:6833/3008975244', 'heartbeat_front_addr': '172.21.15.27:6831/3008975244', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7c39c1dd-3867-46f7-8372-7688fa13b16a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6800', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6801', 'nonce': 3877842957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6802', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6803', 'nonce': 3877842957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6806', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6807', 'nonce': 3877842957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6804', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6805', 'nonce': 3877842957}]}, 'public_addr': '172.21.15.120:6801/3877842957', 'cluster_addr': '172.21.15.120:6803/3877842957', 'heartbeat_back_addr': '172.21.15.120:6807/3877842957', 'heartbeat_front_addr': '172.21.15.120:6805/3877842957', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a6dc220f-aec0-4454-9695-b44c0b9770e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6808', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6809', 'nonce': 1600179183}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6810', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6811', 'nonce': 1600179183}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6814', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6815', 'nonce': 1600179183}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6812', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6813', 'nonce': 1600179183}]}, 'public_addr': '172.21.15.120:6809/1600179183', 'cluster_addr': '172.21.15.120:6811/1600179183', 'heartbeat_back_addr': '172.21.15.120:6815/1600179183', 'heartbeat_front_addr': '172.21.15.120:6813/1600179183', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 118, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6816', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6817', 'nonce': 3323706735}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6818', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6819', 'nonce': 3323706735}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6822', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6823', 'nonce': 3323706735}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6820', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6821', 'nonce': 3323706735}]}, 'public_addr': '172.21.15.120:6817/3323706735', 'cluster_addr': '172.21.15.120:6819/3323706735', 'heartbeat_back_addr': '172.21.15.120:6823/3323706735', 'heartbeat_front_addr': '172.21.15.120:6821/3323706735', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6ef085d8-f59b-4033-8280-2c8f78693e30', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 118, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6824', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6825', 'nonce': 2960616153}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6826', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6827', 'nonce': 2960616153}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6830', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6831', 'nonce': 2960616153}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6828', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6829', 'nonce': 2960616153}]}, 'public_addr': '172.21.15.120:6825/2960616153', 'cluster_addr': '172.21.15.120:6827/2960616153', 'heartbeat_back_addr': '172.21.15.120:6831/2960616153', 'heartbeat_front_addr': '172.21.15.120:6829/2960616153', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '26514b11-d02b-46e9-a4e8-3da193fcdbcc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6800', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6801', 'nonce': 1132459315}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6802', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6803', 'nonce': 1132459315}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6806', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6807', 'nonce': 1132459315}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6804', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6805', 'nonce': 1132459315}]}, 'public_addr': '172.21.15.177:6801/1132459315', 'cluster_addr': '172.21.15.177:6803/1132459315', 'heartbeat_back_addr': '172.21.15.177:6807/1132459315', 'heartbeat_front_addr': '172.21.15.177:6805/1132459315', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '6f9497bc-e962-409d-8a2c-0c634cc1124f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6808', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6809', 'nonce': 2710457646}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6810', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6811', 'nonce': 2710457646}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6814', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6815', 'nonce': 2710457646}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6812', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6813', 'nonce': 2710457646}]}, 'public_addr': '172.21.15.177:6809/2710457646', 'cluster_addr': '172.21.15.177:6811/2710457646', 'heartbeat_back_addr': '172.21.15.177:6815/2710457646', 'heartbeat_front_addr': '172.21.15.177:6813/2710457646', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'f852876f-c7d1-4039-90c4-5ab09dbd3e32', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 81, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6816', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6817', 'nonce': 3236900263}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6818', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6819', 'nonce': 3236900263}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6822', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6823', 'nonce': 3236900263}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6820', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6821', 'nonce': 3236900263}]}, 'public_addr': '172.21.15.177:6817/3236900263', 'cluster_addr': '172.21.15.177:6819/3236900263', 'heartbeat_back_addr': '172.21.15.177:6823/3236900263', 'heartbeat_front_addr': '172.21.15.177:6821/3236900263', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'cccc53c5-8476-4d55-b160-4bdc5cf92902', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 90, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6824', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6825', 'nonce': 681011628}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6826', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6827', 'nonce': 681011628}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6830', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6831', 'nonce': 681011628}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6828', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6829', 'nonce': 681011628}]}, 'public_addr': '172.21.15.177:6825/681011628', 'cluster_addr': '172.21.15.177:6827/681011628', 'heartbeat_back_addr': '172.21.15.177:6831/681011628', 'heartbeat_front_addr': '172.21.15.177:6829/681011628', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:47:33.401880+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:48:01.354812+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:48:00.750859+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:47:37.360495+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:48:08.671701+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:48:17.811164+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:47:42.382426+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:48:00.104152+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:48:30.527775+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:47:23.974039+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-15T12:47:41.837735+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:48:26.576644+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '2.0', 'osds': [7, 9]}, {'pgid': '2.4', 'osds': [1, 0]}, {'pgid': '2.7', 'osds': [6, 2]}], 'primary_temp': [], 'blocklist': {'172.21.15.27:6801/2530617473': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:6800/2530617473': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:0/1880261204': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:6801/3441136471': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/4291446247': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:0/580128209': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:6800/3441136471': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/3011229857': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/2109693913': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/2445663024': '2024-05-16T12:39:19.409195+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-05-15T12:48:50.653 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-05-15T12:48:50.702 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:48:50.741 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:50.768 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:48:50.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:50.771 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:48:50.772 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T12:48:50.783196+0000 min lat: 0.00104218 max lat: 7.07499 avg lat: 0.0102939 2024-05-15T12:48:50.772 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T12:48:50.772 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 40 16 62167 62151 97.0993 85.6875 0.0123754 0.0102939 2024-05-15T12:48:50.872 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:48:50.941 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:51.124 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:48:51.128 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:48:51.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:51.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:48:51.238 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:48:51.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:51 smithi027 ceph-mon[39795]: pgmap v404: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.4 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s; 101/168018 objects misplaced (0.060%); 8.8 MiB/s, 141 objects/s recovering 2024-05-15T12:48:51.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:51 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:48:51.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:51 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:48:51.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:51 smithi027 ceph-mon[39795]: osdmap e130: 12 total, 12 up, 11 in 2024-05-15T12:48:51.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:51 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:48:51.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:51 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:48:51.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:51 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:48:51.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:51 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:48:51.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:51 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:48:51.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:51 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:48:51.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:51 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:48:51.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:51 smithi120 ceph-mon[43961]: pgmap v404: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.4 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s; 101/168018 objects misplaced (0.060%); 8.8 MiB/s, 141 objects/s recovering 2024-05-15T12:48:51.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:51 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:48:51.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:51 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:48:51.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:51 smithi120 ceph-mon[43961]: osdmap e130: 12 total, 12 up, 11 in 2024-05-15T12:48:51.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:51 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:48:51.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:51 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:48:51.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:51 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:48:51.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:51 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:48:51.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:51 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:48:51.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:51 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:48:51.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:51 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:48:51.284 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:51.350 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:48:51.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:51.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:48:51.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:51 smithi177 ceph-mon[44303]: pgmap v404: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.4 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s; 101/168018 objects misplaced (0.060%); 8.8 MiB/s, 141 objects/s recovering 2024-05-15T12:48:51.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:51 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:48:51.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:51 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:48:51.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:51 smithi177 ceph-mon[44303]: osdmap e130: 12 total, 12 up, 11 in 2024-05-15T12:48:51.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:51 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:48:51.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:51 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:48:51.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:51 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:48:51.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:51 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:48:51.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:51 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:48:51.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:51 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:48:51.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:51 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:48:51.456 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:48:51.493 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:51.519 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:48:51.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:51.522 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:48:51.623 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:48:51.685 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:51.772 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 41 16 63738 63722 97.1256 98.1875 0.00407902 0.0102921 2024-05-15T12:48:51.831 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:48:51.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:51.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:48:51.937 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:48:51.974 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:52.002 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:48:52.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:52.006 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:48:52.019 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:52 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:48:52.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:52 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:48:52.107 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:48:52.148 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:52.178 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:48:52.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:52.181 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:48:52.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:52 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:48:52.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:52 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:48:52.282 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:48:52.324 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:52.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:52 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:48:52.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:52 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:48:52.551 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:48:52.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:52.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:48:52.655 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:48:52.697 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:52.722 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:48:52.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:52.726 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:48:52.772 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 42 16 65116 65100 96.8635 86.125 0.00300293 0.0103193 2024-05-15T12:48:52.826 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:48:52.889 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:53.023 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:53 smithi177 ceph-mon[44303]: pgmap v406: 17 pgs: 17 active+clean; 3.7 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s; 9.2 MiB/s, 147 objects/s recovering 2024-05-15T12:48:53.023 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:53 smithi177 ceph-mon[44303]: osdmap e131: 12 total, 12 up, 11 in 2024-05-15T12:48:53.023 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:53 smithi177 ceph-mon[44303]: osdmap e132: 12 total, 12 up, 11 in 2024-05-15T12:48:53.054 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:48:53.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:53.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:48:53.160 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:48:53.171 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T12:48:53.208 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:53.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:53 smithi027 ceph-mon[39795]: pgmap v406: 17 pgs: 17 active+clean; 3.7 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s; 9.2 MiB/s, 147 objects/s recovering 2024-05-15T12:48:53.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:53 smithi027 ceph-mon[39795]: osdmap e131: 12 total, 12 up, 11 in 2024-05-15T12:48:53.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:53 smithi027 ceph-mon[39795]: osdmap e132: 12 total, 12 up, 11 in 2024-05-15T12:48:53.279 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T12:48:53.279 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T12:48:53.279 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T12:48:53.280 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T12:48:53.280 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.280 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:4419 2.9 2:9adaa6b3:::benchmark_data_smithi177_78104_object4418:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-05-15T12:48:53.280 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:11.825393+0000", 2024-05-15T12:48:53.280 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 40.366545618000004, 2024-05-15T12:48:53.280 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20793916200000001, 2024-05-15T12:48:53.280 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:53.280 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:53.280 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:48:53.280 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:48:53.280 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:48:53.280 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 4419 2024-05-15T12:48:53.280 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.281 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:53.281 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.281 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:53.281 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:11.825393+0000", 2024-05-15T12:48:53.281 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.281 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.281 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.281 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:53.281 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:11.825393+0000", 2024-05-15T12:48:53.281 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.281 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.281 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.281 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:53.281 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:11.825394+0000", 2024-05-15T12:48:53.282 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3e-07 2024-05-15T12:48:53.282 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.282 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.282 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:53.282 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:11.825423+0000", 2024-05-15T12:48:53.282 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8955999999999999e-05 2024-05-15T12:48:53.282 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.282 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.282 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:53.282 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:11.825429+0000", 2024-05-15T12:48:53.282 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2840000000000003e-06 2024-05-15T12:48:53.283 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.283 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.283 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:53.283 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:11.825433+0000", 2024-05-15T12:48:53.283 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9639999999999997e-06 2024-05-15T12:48:53.283 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.283 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.283 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.283 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:11.825516+0000", 2024-05-15T12:48:53.283 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3288999999999995e-05 2024-05-15T12:48:53.283 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.283 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.284 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:53.284 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:11.825659+0000", 2024-05-15T12:48:53.284 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014327099999999999 2024-05-15T12:48:53.284 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.284 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.284 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,7", 2024-05-15T12:48:53.284 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:11.825790+0000", 2024-05-15T12:48:53.284 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000130636 2024-05-15T12:48:53.284 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.284 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.284 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.284 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:11.827345+0000", 2024-05-15T12:48:53.284 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001554932 2024-05-15T12:48:53.285 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.285 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.285 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:48:53.285 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:11.828045+0000", 2024-05-15T12:48:53.285 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00069973599999999996 2024-05-15T12:48:53.285 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.285 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.285 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.285 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.033247+0000", 2024-05-15T12:48:53.285 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.205202786 2024-05-15T12:48:53.285 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.285 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.286 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:53.286 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.033286+0000", 2024-05-15T12:48:53.286 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8767999999999999e-05 2024-05-15T12:48:53.286 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.286 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.286 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:53.286 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.033332+0000", 2024-05-15T12:48:53.286 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5909999999999999e-05 2024-05-15T12:48:53.286 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.286 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:53.286 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.286 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.287 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.287 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:4429 2.9 2:95d1949a:::benchmark_data_smithi177_78104_object4428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-05-15T12:48:53.287 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:11.827313+0000", 2024-05-15T12:48:53.287 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 40.364626088000001, 2024-05-15T12:48:53.287 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20690971999999999, 2024-05-15T12:48:53.287 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:53.287 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:53.287 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:48:53.287 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:48:53.287 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:48:53.287 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 4429 2024-05-15T12:48:53.287 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.287 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:53.288 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.288 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:53.288 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:11.827313+0000", 2024-05-15T12:48:53.288 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.288 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.288 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.288 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:53.288 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:11.827313+0000", 2024-05-15T12:48:53.288 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.288 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.288 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.288 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:53.288 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:11.827313+0000", 2024-05-15T12:48:53.288 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7700000000000004e-07 2024-05-15T12:48:53.289 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.289 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.289 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:53.289 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:11.827342+0000", 2024-05-15T12:48:53.289 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8822e-05 2024-05-15T12:48:53.289 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.289 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.289 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:53.289 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:11.827350+0000", 2024-05-15T12:48:53.289 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6159999999999996e-06 2024-05-15T12:48:53.289 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.289 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.289 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:53.289 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:11.827356+0000", 2024-05-15T12:48:53.290 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8430000000000004e-06 2024-05-15T12:48:53.290 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.290 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.290 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.290 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:11.827402+0000", 2024-05-15T12:48:53.290 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5349000000000002e-05 2024-05-15T12:48:53.290 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.290 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.290 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:53.290 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:11.827556+0000", 2024-05-15T12:48:53.290 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015470600000000001 2024-05-15T12:48:53.290 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.290 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.290 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,7", 2024-05-15T12:48:53.290 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:11.827691+0000", 2024-05-15T12:48:53.291 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013408699999999999 2024-05-15T12:48:53.291 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.291 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.291 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:48:53.291 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:11.828702+0000", 2024-05-15T12:48:53.291 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010117189999999999 2024-05-15T12:48:53.291 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.291 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.291 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.291 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:11.829586+0000", 2024-05-15T12:48:53.291 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00088355200000000001 2024-05-15T12:48:53.291 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.291 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.291 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.292 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.034125+0000", 2024-05-15T12:48:53.292 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20453919000000001 2024-05-15T12:48:53.292 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.292 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.292 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:53.292 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.034179+0000", 2024-05-15T12:48:53.292 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3519e-05 2024-05-15T12:48:53.292 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.292 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.292 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:53.292 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:12.034222+0000", 2024-05-15T12:48:53.292 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3739999999999998e-05 2024-05-15T12:48:53.292 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.292 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:53.292 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.293 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.293 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.293 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:24287 2.4 2:21f3f7db:::benchmark_data_smithi177_78104_object24286:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-05-15T12:48:53.293 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:18.199331+0000", 2024-05-15T12:48:53.293 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 33.992607384999999, 2024-05-15T12:48:53.293 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20862608199999999, 2024-05-15T12:48:53.293 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:53.293 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:53.293 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:48:53.293 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:48:53.293 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:48:53.293 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 24287 2024-05-15T12:48:53.293 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.293 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:53.294 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.294 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:53.294 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.199331+0000", 2024-05-15T12:48:53.294 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.294 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.294 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.294 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:53.294 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.199331+0000", 2024-05-15T12:48:53.294 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.294 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.294 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.294 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:53.294 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.199332+0000", 2024-05-15T12:48:53.295 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2300000000000001e-07 2024-05-15T12:48:53.295 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.295 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.295 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:53.295 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.199368+0000", 2024-05-15T12:48:53.295 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6041999999999998e-05 2024-05-15T12:48:53.295 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.295 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.295 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:53.295 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.199379+0000", 2024-05-15T12:48:53.295 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1686e-05 2024-05-15T12:48:53.295 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.295 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.295 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:53.295 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.199386+0000", 2024-05-15T12:48:53.296 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4110000000000001e-06 2024-05-15T12:48:53.296 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.296 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.296 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.296 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.201415+0000", 2024-05-15T12:48:53.296 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020287640000000002 2024-05-15T12:48:53.296 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.296 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.296 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:53.296 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.201489+0000", 2024-05-15T12:48:53.296 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4021000000000002e-05 2024-05-15T12:48:53.296 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.296 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.297 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,10", 2024-05-15T12:48:53.297 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.201571+0000", 2024-05-15T12:48:53.297 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1965999999999994e-05 2024-05-15T12:48:53.297 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.297 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.297 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:48:53.297 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.203905+0000", 2024-05-15T12:48:53.297 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002334367 2024-05-15T12:48:53.297 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.297 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.297 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.297 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.206183+0000", 2024-05-15T12:48:53.298 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002277542 2024-05-15T12:48:53.298 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.298 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.298 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.298 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.407871+0000", 2024-05-15T12:48:53.298 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20168798900000001 2024-05-15T12:48:53.298 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.298 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.298 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:53.298 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.407913+0000", 2024-05-15T12:48:53.298 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2883000000000003e-05 2024-05-15T12:48:53.298 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.298 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.299 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:53.299 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.407957+0000", 2024-05-15T12:48:53.299 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3887999999999997e-05 2024-05-15T12:48:53.299 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.299 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:53.299 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.299 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.299 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.299 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:24290 2.4 2:20cce92a:::benchmark_data_smithi177_78104_object24289:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-05-15T12:48:53.299 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:18.199811+0000", 2024-05-15T12:48:53.299 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 33.992127578000002, 2024-05-15T12:48:53.299 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20826777699999999, 2024-05-15T12:48:53.299 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:53.299 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:53.300 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:48:53.300 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:48:53.300 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:48:53.300 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 24290 2024-05-15T12:48:53.300 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.300 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:53.300 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.300 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:53.300 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.199811+0000", 2024-05-15T12:48:53.300 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.300 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.300 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.300 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:53.300 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.199811+0000", 2024-05-15T12:48:53.300 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.301 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.301 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.301 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:53.301 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.199812+0000", 2024-05-15T12:48:53.301 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2200000000000003e-07 2024-05-15T12:48:53.301 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.301 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.301 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:53.301 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.199864+0000", 2024-05-15T12:48:53.301 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1919000000000002e-05 2024-05-15T12:48:53.301 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.301 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.301 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:53.301 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.199876+0000", 2024-05-15T12:48:53.302 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1956e-05 2024-05-15T12:48:53.302 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.302 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.302 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:53.302 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.199884+0000", 2024-05-15T12:48:53.302 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7600000000000002e-06 2024-05-15T12:48:53.302 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.302 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.302 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.302 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.202395+0000", 2024-05-15T12:48:53.302 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025110879999999999 2024-05-15T12:48:53.302 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.302 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.302 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:53.303 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.202503+0000", 2024-05-15T12:48:53.303 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000108354 2024-05-15T12:48:53.303 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.303 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.303 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,10", 2024-05-15T12:48:53.303 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.202632+0000", 2024-05-15T12:48:53.303 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012932199999999999 2024-05-15T12:48:53.303 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.303 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.303 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:48:53.303 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.203909+0000", 2024-05-15T12:48:53.303 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0012762419999999999 2024-05-15T12:48:53.303 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.303 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.303 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.304 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.206732+0000", 2024-05-15T12:48:53.304 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0028230999999999998 2024-05-15T12:48:53.304 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.304 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.304 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.304 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.408013+0000", 2024-05-15T12:48:53.304 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20128122100000001 2024-05-15T12:48:53.304 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.304 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.304 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:53.304 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.408054+0000", 2024-05-15T12:48:53.304 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1248000000000002e-05 2024-05-15T12:48:53.304 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.304 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.305 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:53.305 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:18.408079+0000", 2024-05-15T12:48:53.305 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4745000000000001e-05 2024-05-15T12:48:53.305 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.305 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:53.305 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.305 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.305 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.305 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:57615 2.4 2:22ab3194:::benchmark_data_smithi177_78104_object57614:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-05-15T12:48:53.305 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:46.975644+0000", 2024-05-15T12:48:53.305 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.216294338, 2024-05-15T12:48:53.305 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0212203049999999, 2024-05-15T12:48:53.305 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:53.305 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T12:48:53.305 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:48:53.306 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:48:53.306 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:48:53.306 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 57615 2024-05-15T12:48:53.306 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.306 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:53.306 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.306 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:53.306 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.975644+0000", 2024-05-15T12:48:53.306 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.306 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.306 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.306 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:53.306 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.975644+0000", 2024-05-15T12:48:53.306 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.307 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.307 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.307 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:53.307 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.975645+0000", 2024-05-15T12:48:53.307 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4199999999999996e-07 2024-05-15T12:48:53.307 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.307 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.307 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:53.307 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.979851+0000", 2024-05-15T12:48:53.307 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0042065310000000003 2024-05-15T12:48:53.307 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.307 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.307 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:53.307 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.979862+0000", 2024-05-15T12:48:53.308 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0302000000000001e-05 2024-05-15T12:48:53.308 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.308 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.308 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:53.308 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.979867+0000", 2024-05-15T12:48:53.308 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8109999999999999e-06 2024-05-15T12:48:53.308 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.308 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.308 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.308 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.981001+0000", 2024-05-15T12:48:53.308 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001133352 2024-05-15T12:48:53.308 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.308 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.309 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:53.309 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.981143+0000", 2024-05-15T12:48:53.309 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014233900000000001 2024-05-15T12:48:53.309 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.309 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.309 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2", 2024-05-15T12:48:53.309 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.981283+0000", 2024-05-15T12:48:53.309 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013992200000000001 2024-05-15T12:48:53.309 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.309 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.309 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.309 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.991284+0000", 2024-05-15T12:48:53.309 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010000587999999999 2024-05-15T12:48:53.309 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.309 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.310 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.310 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.993679+0000", 2024-05-15T12:48:53.310 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002395253 2024-05-15T12:48:53.310 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.310 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.310 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:48:53.310 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.993687+0000", 2024-05-15T12:48:53.310 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3149999999999992e-06 2024-05-15T12:48:53.310 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.310 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.310 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.310 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.996780+0000", 2024-05-15T12:48:53.310 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0030931240000001 2024-05-15T12:48:53.310 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.311 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.311 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:53.311 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.996865+0000", 2024-05-15T12:48:53.311 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4226000000000003e-05 2024-05-15T12:48:53.311 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.311 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:53.311 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.311 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.311 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.311 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:57622 2.4 2:2213f24f:::benchmark_data_smithi177_78104_object57621:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-05-15T12:48:53.311 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:46.979893+0000", 2024-05-15T12:48:53.311 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.2120456060000002, 2024-05-15T12:48:53.311 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0174245150000001, 2024-05-15T12:48:53.311 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:53.311 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T12:48:53.311 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:48:53.312 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:48:53.312 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:48:53.312 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 57622 2024-05-15T12:48:53.312 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.312 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:53.312 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.312 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:53.312 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.979893+0000", 2024-05-15T12:48:53.312 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.312 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.312 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.312 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:53.312 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.979893+0000", 2024-05-15T12:48:53.312 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.312 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.313 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.313 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:53.313 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.979894+0000", 2024-05-15T12:48:53.313 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5499999999999997e-07 2024-05-15T12:48:53.313 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.313 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.313 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:53.313 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.979933+0000", 2024-05-15T12:48:53.313 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9326000000000003e-05 2024-05-15T12:48:53.313 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.313 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.313 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:53.313 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.979940+0000", 2024-05-15T12:48:53.313 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1040000000000001e-06 2024-05-15T12:48:53.314 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.314 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.314 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:53.314 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.979944+0000", 2024-05-15T12:48:53.314 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9899999999999999e-06 2024-05-15T12:48:53.314 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.314 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.314 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.314 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.984956+0000", 2024-05-15T12:48:53.314 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050114959999999998 2024-05-15T12:48:53.314 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.314 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.314 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:53.314 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.985089+0000", 2024-05-15T12:48:53.315 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000133387 2024-05-15T12:48:53.315 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.315 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.315 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2", 2024-05-15T12:48:53.315 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.985255+0000", 2024-05-15T12:48:53.315 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016599799999999999 2024-05-15T12:48:53.315 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.315 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.315 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.315 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.993733+0000", 2024-05-15T12:48:53.315 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0084779260000000002 2024-05-15T12:48:53.315 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.315 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.315 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:48:53.315 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.993744+0000", 2024-05-15T12:48:53.315 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0727000000000001e-05 2024-05-15T12:48:53.316 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.316 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.316 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.316 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.997244+0000", 2024-05-15T12:48:53.316 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0035007199999999 2024-05-15T12:48:53.316 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.316 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.316 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:53.316 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.997318+0000", 2024-05-15T12:48:53.316 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3085999999999998e-05 2024-05-15T12:48:53.316 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.316 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:53.316 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.316 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.316 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.316 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:57647 2.4 2:257b14ee:::benchmark_data_smithi177_78104_object57646:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-05-15T12:48:53.316 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:46.984210+0000", 2024-05-15T12:48:53.317 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.2077287190000003, 2024-05-15T12:48:53.317 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0131975980000001, 2024-05-15T12:48:53.317 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:53.317 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T12:48:53.317 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:48:53.317 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:48:53.317 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:48:53.317 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 57647 2024-05-15T12:48:53.317 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.317 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:53.317 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.317 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:53.317 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.984210+0000", 2024-05-15T12:48:53.317 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.317 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.317 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.318 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:53.318 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.984210+0000", 2024-05-15T12:48:53.318 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.318 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.318 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.318 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:53.318 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.984210+0000", 2024-05-15T12:48:53.318 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1399999999999997e-07 2024-05-15T12:48:53.318 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.318 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.318 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:53.318 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.984244+0000", 2024-05-15T12:48:53.318 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3646000000000003e-05 2024-05-15T12:48:53.319 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.319 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.319 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:53.319 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.984255+0000", 2024-05-15T12:48:53.319 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0633e-05 2024-05-15T12:48:53.319 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.319 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.319 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:53.319 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.984261+0000", 2024-05-15T12:48:53.319 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7889999999999997e-06 2024-05-15T12:48:53.319 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.319 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.319 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.319 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.987175+0000", 2024-05-15T12:48:53.319 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029139259999999998 2024-05-15T12:48:53.319 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.320 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.320 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:53.320 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.987340+0000", 2024-05-15T12:48:53.320 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000164415 2024-05-15T12:48:53.320 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.320 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.320 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2", 2024-05-15T12:48:53.320 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.987462+0000", 2024-05-15T12:48:53.320 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000122423 2024-05-15T12:48:53.320 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.320 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.320 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.320 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.993831+0000", 2024-05-15T12:48:53.320 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0063684739999999998 2024-05-15T12:48:53.321 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.321 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.321 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:48:53.321 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:46.993841+0000", 2024-05-15T12:48:53.321 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0057999999999999e-05 2024-05-15T12:48:53.321 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.321 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.321 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.321 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.997353+0000", 2024-05-15T12:48:53.321 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0035124070000001 2024-05-15T12:48:53.321 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.321 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.321 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:53.321 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.997408+0000", 2024-05-15T12:48:53.321 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4313e-05 2024-05-15T12:48:53.322 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.322 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:53.322 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.322 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.322 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.322 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:57675 2.4 2:28307a7c:::benchmark_data_smithi177_78104_object57674:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-05-15T12:48:53.322 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:47.003083+0000", 2024-05-15T12:48:53.322 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.188855234, 2024-05-15T12:48:53.322 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.004079613, 2024-05-15T12:48:53.322 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:53.322 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:53.322 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:48:53.322 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:48:53.322 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:48:53.322 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 57675 2024-05-15T12:48:53.323 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.323 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:53.323 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.323 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:53.323 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.003083+0000", 2024-05-15T12:48:53.323 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.323 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.323 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.323 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:53.323 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.003083+0000", 2024-05-15T12:48:53.323 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.323 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.323 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.323 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:53.323 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.003085+0000", 2024-05-15T12:48:53.323 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1829999999999999e-06 2024-05-15T12:48:53.323 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.324 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.324 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:53.324 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.003176+0000", 2024-05-15T12:48:53.324 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1800999999999997e-05 2024-05-15T12:48:53.324 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.324 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.324 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:53.324 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.003191+0000", 2024-05-15T12:48:53.324 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4443999999999999e-05 2024-05-15T12:48:53.324 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.324 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.324 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:53.324 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.003199+0000", 2024-05-15T12:48:53.324 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2190000000000004e-06 2024-05-15T12:48:53.325 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.325 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.325 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.325 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.003512+0000", 2024-05-15T12:48:53.325 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00031312499999999998 2024-05-15T12:48:53.325 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.325 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.325 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:48:53.325 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.003525+0000", 2024-05-15T12:48:53.325 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3268000000000001e-05 2024-05-15T12:48:53.325 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.325 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.325 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.325 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.997457+0000", 2024-05-15T12:48:53.325 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99393162199999996 2024-05-15T12:48:53.326 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.326 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.326 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:53.326 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.997572+0000", 2024-05-15T12:48:53.326 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000115041 2024-05-15T12:48:53.326 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.326 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.326 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2", 2024-05-15T12:48:53.326 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.997689+0000", 2024-05-15T12:48:53.326 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000116531 2024-05-15T12:48:53.326 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.326 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.326 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.326 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.003634+0000", 2024-05-15T12:48:53.327 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.005945489 2024-05-15T12:48:53.327 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.327 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.327 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:48:53.327 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.003649+0000", 2024-05-15T12:48:53.327 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4924e-05 2024-05-15T12:48:53.327 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.327 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.327 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.327 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.007038+0000", 2024-05-15T12:48:53.327 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033889100000000002 2024-05-15T12:48:53.327 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.327 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.327 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:53.327 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.007099+0000", 2024-05-15T12:48:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1160000000000004e-05 2024-05-15T12:48:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.007163+0000", 2024-05-15T12:48:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3895999999999999e-05 2024-05-15T12:48:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:57684 2.4 2:222de3c8:::benchmark_data_smithi177_78104_object57683:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-05-15T12:48:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:47.010090+0000", 2024-05-15T12:48:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.1818481600000004, 2024-05-15T12:48:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0018080039999999, 2024-05-15T12:48:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:48:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:48:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:48:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 57684 2024-05-15T12:48:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.010090+0000", 2024-05-15T12:48:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:53.330 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.010090+0000", 2024-05-15T12:48:53.330 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.330 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.330 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.330 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:53.330 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.010092+0000", 2024-05-15T12:48:53.330 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8160000000000001e-06 2024-05-15T12:48:53.330 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.330 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.330 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:53.330 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.010143+0000", 2024-05-15T12:48:53.330 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0657e-05 2024-05-15T12:48:53.330 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.330 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.330 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:53.331 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.010160+0000", 2024-05-15T12:48:53.331 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6789000000000001e-05 2024-05-15T12:48:53.331 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.331 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.331 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:53.331 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.010169+0000", 2024-05-15T12:48:53.331 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4429999999999992e-06 2024-05-15T12:48:53.331 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.331 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.331 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.331 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.010231+0000", 2024-05-15T12:48:53.331 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2081000000000004e-05 2024-05-15T12:48:53.331 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.332 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.332 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:48:53.332 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.010239+0000", 2024-05-15T12:48:53.332 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4730000000000003e-06 2024-05-15T12:48:53.332 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.332 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.332 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.332 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.999526+0000", 2024-05-15T12:48:53.332 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.98928724499999998 2024-05-15T12:48:53.332 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.332 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.332 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:53.332 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.999649+0000", 2024-05-15T12:48:53.332 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000122846 2024-05-15T12:48:53.333 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.333 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.333 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2", 2024-05-15T12:48:53.333 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.999794+0000", 2024-05-15T12:48:53.333 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014538600000000001 2024-05-15T12:48:53.333 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.333 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.333 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.333 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.003677+0000", 2024-05-15T12:48:53.333 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038832810000000001 2024-05-15T12:48:53.333 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.333 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.333 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:48:53.333 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.003737+0000", 2024-05-15T12:48:53.334 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9095000000000003e-05 2024-05-15T12:48:53.334 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.334 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.334 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.334 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.007211+0000", 2024-05-15T12:48:53.334 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034740690000000002 2024-05-15T12:48:53.334 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.334 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.334 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:53.334 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.011855+0000", 2024-05-15T12:48:53.334 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046439039999999999 2024-05-15T12:48:53.334 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.334 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.335 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:53.335 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.011898+0000", 2024-05-15T12:48:53.335 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3918999999999997e-05 2024-05-15T12:48:53.335 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.335 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:53.335 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.335 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.335 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.335 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:57738 2.4 2:236cd0fa:::benchmark_data_smithi177_78104_object57737:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-05-15T12:48:53.335 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:47.040207+0000", 2024-05-15T12:48:53.335 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.1517320209999999, 2024-05-15T12:48:53.335 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.97650864199999998, 2024-05-15T12:48:53.335 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:53.335 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:53.336 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:48:53.336 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:48:53.336 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:48:53.336 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 57738 2024-05-15T12:48:53.336 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.336 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:53.336 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.336 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:53.336 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.040207+0000", 2024-05-15T12:48:53.336 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.336 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.336 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.336 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:53.336 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.040207+0000", 2024-05-15T12:48:53.337 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.337 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.337 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.337 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:53.337 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.040209+0000", 2024-05-15T12:48:53.337 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0880000000000002e-06 2024-05-15T12:48:53.337 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.337 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.337 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:53.337 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.040259+0000", 2024-05-15T12:48:53.337 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0781000000000001e-05 2024-05-15T12:48:53.337 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.338 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.338 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:53.338 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.040277+0000", 2024-05-15T12:48:53.338 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7220000000000001e-05 2024-05-15T12:48:53.338 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.338 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.338 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:53.338 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.040285+0000", 2024-05-15T12:48:53.338 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0309999999999998e-06 2024-05-15T12:48:53.338 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.338 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.338 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.338 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.040346+0000", 2024-05-15T12:48:53.339 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1202000000000001e-05 2024-05-15T12:48:53.339 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.339 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.339 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:48:53.339 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.040356+0000", 2024-05-15T12:48:53.339 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0091e-05 2024-05-15T12:48:53.339 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.339 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.339 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.339 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.001557+0000", 2024-05-15T12:48:53.339 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.96120145700000004 2024-05-15T12:48:53.339 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.339 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.339 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:53.340 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.001682+0000", 2024-05-15T12:48:53.340 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012416 2024-05-15T12:48:53.340 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.340 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.340 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2", 2024-05-15T12:48:53.340 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.001823+0000", 2024-05-15T12:48:53.340 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000141378 2024-05-15T12:48:53.340 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.340 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.340 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.340 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.003724+0000", 2024-05-15T12:48:53.340 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019008989999999999 2024-05-15T12:48:53.340 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.340 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.340 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.341 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.011996+0000", 2024-05-15T12:48:53.341 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0082718679999999999 2024-05-15T12:48:53.341 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.341 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.341 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:48:53.341 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.016588+0000", 2024-05-15T12:48:53.341 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0045925189999999998 2024-05-15T12:48:53.341 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.341 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.341 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:53.341 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.016652+0000", 2024-05-15T12:48:53.341 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4090000000000005e-05 2024-05-15T12:48:53.341 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.341 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.341 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:53.342 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.016715+0000", 2024-05-15T12:48:53.342 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2858e-05 2024-05-15T12:48:53.342 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.342 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:53.342 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.342 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.342 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.342 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:57741 2.4 2:2296d063:::benchmark_data_smithi177_78104_object57740:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-05-15T12:48:53.342 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:47.042528+0000", 2024-05-15T12:48:53.342 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.1494105560000003, 2024-05-15T12:48:53.342 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.97438154700000001, 2024-05-15T12:48:53.342 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:53.342 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:53.342 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:48:53.343 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:48:53.343 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:48:53.343 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 57741 2024-05-15T12:48:53.343 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.343 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:53.343 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.343 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:53.343 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.042528+0000", 2024-05-15T12:48:53.343 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.343 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.343 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.343 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:53.343 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.042528+0000", 2024-05-15T12:48:53.343 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.344 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.344 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.344 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:53.344 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.042530+0000", 2024-05-15T12:48:53.344 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2859999999999998e-06 2024-05-15T12:48:53.344 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.344 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.344 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:53.344 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.042590+0000", 2024-05-15T12:48:53.344 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9676000000000003e-05 2024-05-15T12:48:53.344 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.344 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.344 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:53.344 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.042609+0000", 2024-05-15T12:48:53.344 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8473000000000001e-05 2024-05-15T12:48:53.345 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.345 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.345 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:53.345 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.042618+0000", 2024-05-15T12:48:53.345 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9729999999999992e-06 2024-05-15T12:48:53.345 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.345 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.345 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.345 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.042701+0000", 2024-05-15T12:48:53.345 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2157999999999995e-05 2024-05-15T12:48:53.345 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.345 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.345 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:48:53.345 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.042714+0000", 2024-05-15T12:48:53.346 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3556e-05 2024-05-15T12:48:53.346 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.346 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.346 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.346 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.003767+0000", 2024-05-15T12:48:53.346 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.96105238900000001 2024-05-15T12:48:53.346 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.346 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.346 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:53.346 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.003844+0000", 2024-05-15T12:48:53.346 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7574e-05 2024-05-15T12:48:53.347 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.347 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.347 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2", 2024-05-15T12:48:53.347 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.003933+0000", 2024-05-15T12:48:53.347 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8795999999999995e-05 2024-05-15T12:48:53.347 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.347 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.347 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.347 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.011952+0000", 2024-05-15T12:48:53.347 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0080195279999999997 2024-05-15T12:48:53.347 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.347 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.347 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:48:53.347 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.016729+0000", 2024-05-15T12:48:53.347 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0047766659999999997 2024-05-15T12:48:53.347 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.348 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.348 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.348 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.016800+0000", 2024-05-15T12:48:53.348 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0499000000000004e-05 2024-05-15T12:48:53.348 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.348 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.348 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:53.348 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.016865+0000", 2024-05-15T12:48:53.348 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5148999999999996e-05 2024-05-15T12:48:53.348 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.348 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.348 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:53.348 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.016910+0000", 2024-05-15T12:48:53.348 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4824000000000003e-05 2024-05-15T12:48:53.349 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.349 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:53.349 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.349 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.349 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.349 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:57751 2.4 2:255d6e35:::benchmark_data_smithi177_78104_object57750:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-05-15T12:48:53.349 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:47.048913+0000", 2024-05-15T12:48:53.349 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.1430258760000003, 2024-05-15T12:48:53.349 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.96819326299999997, 2024-05-15T12:48:53.349 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:53.349 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:53.349 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:48:53.349 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:48:53.349 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:48:53.349 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 57751 2024-05-15T12:48:53.349 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.350 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:53.350 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.350 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:53.350 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.048913+0000", 2024-05-15T12:48:53.350 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.350 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.350 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.350 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:53.350 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.048913+0000", 2024-05-15T12:48:53.350 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.350 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.350 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.350 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:53.350 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.048914+0000", 2024-05-15T12:48:53.350 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3689999999999999e-06 2024-05-15T12:48:53.350 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.351 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.351 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:53.351 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.048968+0000", 2024-05-15T12:48:53.351 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3893000000000001e-05 2024-05-15T12:48:53.351 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.351 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.351 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:53.351 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.048979+0000", 2024-05-15T12:48:53.351 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1209000000000001e-05 2024-05-15T12:48:53.351 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.351 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.351 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:53.351 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.048988+0000", 2024-05-15T12:48:53.351 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1379999999999996e-06 2024-05-15T12:48:53.351 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.351 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.352 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.352 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.049071+0000", 2024-05-15T12:48:53.352 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2791000000000001e-05 2024-05-15T12:48:53.352 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.352 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.352 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:48:53.352 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.049084+0000", 2024-05-15T12:48:53.352 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2605e-05 2024-05-15T12:48:53.352 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.352 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.352 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.352 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.005494+0000", 2024-05-15T12:48:53.352 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.95641035100000005 2024-05-15T12:48:53.352 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.352 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.352 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:53.353 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.005568+0000", 2024-05-15T12:48:53.353 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3755000000000001e-05 2024-05-15T12:48:53.353 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.353 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.353 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2", 2024-05-15T12:48:53.353 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.005669+0000", 2024-05-15T12:48:53.353 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000101339 2024-05-15T12:48:53.353 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.353 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.353 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:48:53.353 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.016736+0000", 2024-05-15T12:48:53.353 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011067151000000001 2024-05-15T12:48:53.353 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.353 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.353 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.353 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.016958+0000", 2024-05-15T12:48:53.354 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000221189 2024-05-15T12:48:53.354 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.354 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.354 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.354 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.017005+0000", 2024-05-15T12:48:53.354 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7945999999999998e-05 2024-05-15T12:48:53.354 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.354 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.354 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:53.354 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.017062+0000", 2024-05-15T12:48:53.354 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6616999999999997e-05 2024-05-15T12:48:53.354 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.354 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.354 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:53.355 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.017106+0000", 2024-05-15T12:48:53.355 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3909999999999998e-05 2024-05-15T12:48:53.355 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.355 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:53.355 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.355 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.355 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.355 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:57777 2.4 2:2ca01416:::benchmark_data_smithi177_78104_object57776:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-05-15T12:48:53.355 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:47.066611+0000", 2024-05-15T12:48:53.355 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.1253277730000004, 2024-05-15T12:48:53.355 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.95077564699999995, 2024-05-15T12:48:53.355 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:53.355 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:53.355 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:48:53.356 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:48:53.356 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:48:53.356 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 57777 2024-05-15T12:48:53.356 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.356 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:53.356 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.356 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:53.356 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.066611+0000", 2024-05-15T12:48:53.356 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.356 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.356 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.356 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:53.356 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.066611+0000", 2024-05-15T12:48:53.356 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.356 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.356 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.357 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:53.357 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.066612+0000", 2024-05-15T12:48:53.357 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.26e-06 2024-05-15T12:48:53.357 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.357 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.357 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:53.357 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.066659+0000", 2024-05-15T12:48:53.357 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6832999999999997e-05 2024-05-15T12:48:53.357 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.357 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.357 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:53.357 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.066671+0000", 2024-05-15T12:48:53.357 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2164000000000001e-05 2024-05-15T12:48:53.357 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.357 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.357 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:53.357 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.066678+0000", 2024-05-15T12:48:53.358 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9829999999999999e-06 2024-05-15T12:48:53.358 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.358 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.358 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.358 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.066792+0000", 2024-05-15T12:48:53.358 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000114329 2024-05-15T12:48:53.358 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.358 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.358 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:48:53.358 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.066820+0000", 2024-05-15T12:48:53.358 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7355000000000002e-05 2024-05-15T12:48:53.358 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.358 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.359 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.359 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.012028+0000", 2024-05-15T12:48:53.359 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.94520853800000004 2024-05-15T12:48:53.359 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.359 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.359 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:53.359 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.012188+0000", 2024-05-15T12:48:53.359 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015977799999999999 2024-05-15T12:48:53.359 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.359 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.359 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2", 2024-05-15T12:48:53.359 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.012349+0000", 2024-05-15T12:48:53.359 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016132599999999999 2024-05-15T12:48:53.359 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.359 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.360 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:48:53.360 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.016967+0000", 2024-05-15T12:48:53.360 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046175970000000002 2024-05-15T12:48:53.360 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.360 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.360 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.360 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.017227+0000", 2024-05-15T12:48:53.360 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00025989900000000001 2024-05-15T12:48:53.360 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.360 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.360 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.360 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.017283+0000", 2024-05-15T12:48:53.360 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6419000000000003e-05 2024-05-15T12:48:53.360 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.361 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.361 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:53.361 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.017340+0000", 2024-05-15T12:48:53.361 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6925999999999997e-05 2024-05-15T12:48:53.361 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.361 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.361 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:53.361 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.017387+0000", 2024-05-15T12:48:53.361 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6239999999999998e-05 2024-05-15T12:48:53.361 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.361 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:53.361 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.361 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.361 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.361 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:57789 2.4 2:227d9c3c:::benchmark_data_smithi177_78104_object57788:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-05-15T12:48:53.361 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:47.074434+0000", 2024-05-15T12:48:53.362 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.117505038, 2024-05-15T12:48:53.362 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.943250904, 2024-05-15T12:48:53.362 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:53.362 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:53.362 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:48:53.362 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:48:53.362 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:48:53.362 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 57789 2024-05-15T12:48:53.362 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.362 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:53.362 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.362 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:53.362 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.074434+0000", 2024-05-15T12:48:53.362 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.362 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.363 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.363 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:53.363 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.074434+0000", 2024-05-15T12:48:53.363 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.363 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.363 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.363 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:53.363 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.074435+0000", 2024-05-15T12:48:53.363 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0240000000000001e-06 2024-05-15T12:48:53.363 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.363 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.363 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:53.363 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.074503+0000", 2024-05-15T12:48:53.363 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7941999999999994e-05 2024-05-15T12:48:53.363 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.363 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.364 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:53.364 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.074512+0000", 2024-05-15T12:48:53.364 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3780000000000007e-06 2024-05-15T12:48:53.364 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.364 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.364 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:53.364 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.074518+0000", 2024-05-15T12:48:53.364 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7379999999999999e-06 2024-05-15T12:48:53.364 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.364 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.364 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.364 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.074588+0000", 2024-05-15T12:48:53.364 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0160000000000006e-05 2024-05-15T12:48:53.365 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.365 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.365 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:48:53.365 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.074597+0000", 2024-05-15T12:48:53.365 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9080000000000001e-06 2024-05-15T12:48:53.365 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.365 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.365 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.365 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.014455+0000", 2024-05-15T12:48:53.365 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.93985867999999995 2024-05-15T12:48:53.365 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.365 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.365 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:53.365 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.014597+0000", 2024-05-15T12:48:53.366 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014107799999999999 2024-05-15T12:48:53.366 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.366 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.366 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2", 2024-05-15T12:48:53.366 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.014750+0000", 2024-05-15T12:48:53.366 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000153327 2024-05-15T12:48:53.366 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.366 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.366 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:48:53.366 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.017235+0000", 2024-05-15T12:48:53.366 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002485541 2024-05-15T12:48:53.366 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.366 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.366 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.367 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.017447+0000", 2024-05-15T12:48:53.367 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00021176100000000001 2024-05-15T12:48:53.367 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.367 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.367 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.367 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.017591+0000", 2024-05-15T12:48:53.367 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014429699999999999 2024-05-15T12:48:53.367 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.367 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.367 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:53.367 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.017643+0000", 2024-05-15T12:48:53.367 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1903000000000001e-05 2024-05-15T12:48:53.367 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.367 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.367 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:53.367 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.017684+0000", 2024-05-15T12:48:53.368 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1167000000000001e-05 2024-05-15T12:48:53.368 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.368 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:53.368 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.368 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.368 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.368 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:57792 2.4 2:25a21404:::benchmark_data_smithi177_78104_object57791:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-05-15T12:48:53.368 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:47.076130+0000", 2024-05-15T12:48:53.368 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.1158084339999998, 2024-05-15T12:48:53.368 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.957195974, 2024-05-15T12:48:53.368 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:53.368 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:53.368 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:48:53.369 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:48:53.369 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:48:53.369 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 57792 2024-05-15T12:48:53.369 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.369 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:53.369 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.369 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:53.369 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.076130+0000", 2024-05-15T12:48:53.369 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.369 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.369 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.369 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:53.369 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.076130+0000", 2024-05-15T12:48:53.370 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.370 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.370 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.370 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:53.370 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.076131+0000", 2024-05-15T12:48:53.370 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2900000000000003e-07 2024-05-15T12:48:53.370 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.370 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.370 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:53.370 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.076177+0000", 2024-05-15T12:48:53.370 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6381000000000002e-05 2024-05-15T12:48:53.370 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.370 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.371 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:53.371 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.076186+0000", 2024-05-15T12:48:53.371 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.67e-06 2024-05-15T12:48:53.371 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.371 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.371 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:53.371 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.076192+0000", 2024-05-15T12:48:53.371 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7339999999999996e-06 2024-05-15T12:48:53.371 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.371 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.371 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.371 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.076244+0000", 2024-05-15T12:48:53.371 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2240999999999997e-05 2024-05-15T12:48:53.371 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.372 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.372 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:48:53.372 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.076253+0000", 2024-05-15T12:48:53.372 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6600000000000001e-06 2024-05-15T12:48:53.372 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.372 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.372 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.372 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.017737+0000", 2024-05-15T12:48:53.372 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.94148459500000004 2024-05-15T12:48:53.372 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.372 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.373 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:53.373 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.017857+0000", 2024-05-15T12:48:53.373 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000119868 2024-05-15T12:48:53.373 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.373 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.373 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2", 2024-05-15T12:48:53.373 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.018010+0000", 2024-05-15T12:48:53.373 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000152652 2024-05-15T12:48:53.373 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.373 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.373 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:48:53.373 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.024227+0000", 2024-05-15T12:48:53.373 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0062170819999999996 2024-05-15T12:48:53.374 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.374 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.374 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.374 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.032945+0000", 2024-05-15T12:48:53.374 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.008718646 2024-05-15T12:48:53.374 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.374 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.374 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.374 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.033074+0000", 2024-05-15T12:48:53.374 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012825799999999999 2024-05-15T12:48:53.374 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.374 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.374 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:53.374 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.033198+0000", 2024-05-15T12:48:53.375 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000124809 2024-05-15T12:48:53.375 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.375 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.375 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:53.375 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.033326+0000", 2024-05-15T12:48:53.375 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000127749 2024-05-15T12:48:53.375 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.375 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:53.375 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.375 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.375 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.375 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:57808 2.4 2:2bf4618f:::benchmark_data_smithi177_78104_object57807:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-05-15T12:48:53.375 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:47.086498+0000", 2024-05-15T12:48:53.376 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.1054408799999997, 2024-05-15T12:48:53.376 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.94697629000000005, 2024-05-15T12:48:53.376 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:53.376 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:53.376 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:48:53.376 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:48:53.376 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:48:53.376 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 57808 2024-05-15T12:48:53.376 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.376 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:53.376 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.376 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:53.376 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.086498+0000", 2024-05-15T12:48:53.376 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.377 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.377 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.377 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:53.377 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.086498+0000", 2024-05-15T12:48:53.377 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.377 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.377 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.377 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:53.377 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.086499+0000", 2024-05-15T12:48:53.377 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.344e-06 2024-05-15T12:48:53.377 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.377 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.377 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:53.377 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.086560+0000", 2024-05-15T12:48:53.377 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0857999999999999e-05 2024-05-15T12:48:53.378 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.378 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.378 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:53.378 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.086573+0000", 2024-05-15T12:48:53.378 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.307e-05 2024-05-15T12:48:53.378 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.378 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.378 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:53.378 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.086580+0000", 2024-05-15T12:48:53.378 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6250000000000001e-06 2024-05-15T12:48:53.378 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.378 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.379 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.379 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.086691+0000", 2024-05-15T12:48:53.379 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000110919 2024-05-15T12:48:53.379 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.379 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.379 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:48:53.379 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.086704+0000", 2024-05-15T12:48:53.379 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3327e-05 2024-05-15T12:48:53.379 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.379 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.379 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.379 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.020041+0000", 2024-05-15T12:48:53.379 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.93333712199999996 2024-05-15T12:48:53.379 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.379 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.380 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:53.380 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.020158+0000", 2024-05-15T12:48:53.380 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000117007 2024-05-15T12:48:53.380 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.380 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.380 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2", 2024-05-15T12:48:53.380 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.020285+0000", 2024-05-15T12:48:53.380 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000127093 2024-05-15T12:48:53.380 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.380 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.380 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:48:53.380 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.026337+0000", 2024-05-15T12:48:53.381 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0060521230000000004 2024-05-15T12:48:53.381 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.381 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.381 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.381 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.032986+0000", 2024-05-15T12:48:53.381 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0066486239999999997 2024-05-15T12:48:53.381 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.381 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.381 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.381 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.033383+0000", 2024-05-15T12:48:53.381 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00039699499999999998 2024-05-15T12:48:53.381 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.381 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.381 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:53.382 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.033434+0000", 2024-05-15T12:48:53.382 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1369000000000002e-05 2024-05-15T12:48:53.382 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.382 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.382 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:53.382 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.033474+0000", 2024-05-15T12:48:53.382 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9814000000000002e-05 2024-05-15T12:48:53.382 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.382 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:53.382 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.382 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.382 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.382 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:57810 2.4 2:2a27d57c:::benchmark_data_smithi177_78104_object57809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-05-15T12:48:53.383 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:47.087338+0000", 2024-05-15T12:48:53.383 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.1046007160000002, 2024-05-15T12:48:53.383 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.94626223700000001, 2024-05-15T12:48:53.383 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:53.383 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:53.383 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:48:53.383 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:48:53.383 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:48:53.383 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 57810 2024-05-15T12:48:53.383 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.383 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:53.383 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.383 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:53.383 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.087338+0000", 2024-05-15T12:48:53.384 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.384 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.384 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.384 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:53.384 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.087338+0000", 2024-05-15T12:48:53.384 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.384 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.384 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.384 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:53.384 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.087339+0000", 2024-05-15T12:48:53.384 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.54e-07 2024-05-15T12:48:53.384 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.384 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.384 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:53.384 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.087401+0000", 2024-05-15T12:48:53.384 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2238999999999998e-05 2024-05-15T12:48:53.385 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.385 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.385 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:53.385 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.087412+0000", 2024-05-15T12:48:53.385 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0626e-05 2024-05-15T12:48:53.385 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.385 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.385 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:53.385 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.087420+0000", 2024-05-15T12:48:53.385 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0099999999999995e-06 2024-05-15T12:48:53.385 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.385 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.385 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.385 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.087475+0000", 2024-05-15T12:48:53.385 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4923000000000003e-05 2024-05-15T12:48:53.386 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.386 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.386 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:48:53.386 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.087484+0000", 2024-05-15T12:48:53.386 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4690000000000003e-06 2024-05-15T12:48:53.386 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.386 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.386 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.386 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.022165+0000", 2024-05-15T12:48:53.386 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.93468094000000002 2024-05-15T12:48:53.386 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.386 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.386 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:53.386 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.022311+0000", 2024-05-15T12:48:53.386 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014643600000000001 2024-05-15T12:48:53.387 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.387 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.387 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2", 2024-05-15T12:48:53.387 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.022454+0000", 2024-05-15T12:48:53.387 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000142805 2024-05-15T12:48:53.387 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.387 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.387 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:48:53.387 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.028604+0000", 2024-05-15T12:48:53.387 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.006149893 2024-05-15T12:48:53.387 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.387 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.387 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.387 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.033023+0000", 2024-05-15T12:48:53.387 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0044185279999999997 2024-05-15T12:48:53.388 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.388 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.388 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.388 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.033515+0000", 2024-05-15T12:48:53.388 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00049273399999999999 2024-05-15T12:48:53.388 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.388 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.388 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:53.388 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.033564+0000", 2024-05-15T12:48:53.388 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9110000000000002e-05 2024-05-15T12:48:53.388 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.388 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.388 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:53.388 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.033600+0000", 2024-05-15T12:48:53.389 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5670000000000002e-05 2024-05-15T12:48:53.389 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.389 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:53.389 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.389 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.389 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.389 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:57842 2.4 2:2c07f715:::benchmark_data_smithi177_78104_object57841:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-05-15T12:48:53.389 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:47.117064+0000", 2024-05-15T12:48:53.389 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.0748748150000003, 2024-05-15T12:48:53.389 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.91748106299999999, 2024-05-15T12:48:53.389 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:53.389 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:53.389 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:48:53.389 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:48:53.390 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:48:53.390 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 57842 2024-05-15T12:48:53.390 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.390 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:53.390 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.390 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:53.390 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.117064+0000", 2024-05-15T12:48:53.390 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.390 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.390 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.390 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:53.390 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.117064+0000", 2024-05-15T12:48:53.390 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.390 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.391 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.391 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:53.391 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.117065+0000", 2024-05-15T12:48:53.391 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2099999999999995e-07 2024-05-15T12:48:53.391 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.391 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.391 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:53.391 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.117145+0000", 2024-05-15T12:48:53.391 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0620999999999999e-05 2024-05-15T12:48:53.391 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.391 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.391 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:53.391 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.117153+0000", 2024-05-15T12:48:53.391 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0250000000000002e-06 2024-05-15T12:48:53.392 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.392 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.392 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:53.392 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.117158+0000", 2024-05-15T12:48:53.392 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9389999999999996e-06 2024-05-15T12:48:53.392 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.392 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.392 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.392 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.117209+0000", 2024-05-15T12:48:53.392 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1044000000000001e-05 2024-05-15T12:48:53.392 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.392 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.392 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:48:53.392 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.117216+0000", 2024-05-15T12:48:53.393 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.126e-06 2024-05-15T12:48:53.393 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.393 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.393 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.393 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.024251+0000", 2024-05-15T12:48:53.393 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.90703503500000005 2024-05-15T12:48:53.393 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.393 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.393 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:53.393 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.024324+0000", 2024-05-15T12:48:53.393 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2831999999999997e-05 2024-05-15T12:48:53.393 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.393 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.393 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2", 2024-05-15T12:48:53.393 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.024408+0000", 2024-05-15T12:48:53.393 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4134000000000005e-05 2024-05-15T12:48:53.394 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.394 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.394 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:48:53.394 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.030751+0000", 2024-05-15T12:48:53.394 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0063428779999999997 2024-05-15T12:48:53.394 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.394 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.394 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.394 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.034354+0000", 2024-05-15T12:48:53.394 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0036028290000000001 2024-05-15T12:48:53.394 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.394 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.394 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.394 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.034447+0000", 2024-05-15T12:48:53.395 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2621000000000006e-05 2024-05-15T12:48:53.395 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.395 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.395 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:53.395 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.034503+0000", 2024-05-15T12:48:53.395 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6098999999999999e-05 2024-05-15T12:48:53.395 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.395 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.395 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:53.395 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.034545+0000", 2024-05-15T12:48:53.395 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2058999999999998e-05 2024-05-15T12:48:53.395 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.395 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:53.395 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.396 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.396 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.396 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:57858 2.4 2:2e13da18:::benchmark_data_smithi177_78104_object57857:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-05-15T12:48:53.396 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:47.135380+0000", 2024-05-15T12:48:53.396 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.056558409, 2024-05-15T12:48:53.396 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.89927543099999996, 2024-05-15T12:48:53.396 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:53.396 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:53.396 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:48:53.396 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:48:53.396 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:48:53.396 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 57858 2024-05-15T12:48:53.396 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.397 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:53.397 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.397 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:53.397 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.135380+0000", 2024-05-15T12:48:53.397 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.397 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.397 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.397 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:53.397 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.135380+0000", 2024-05-15T12:48:53.397 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.397 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.397 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.397 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:53.397 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.135382+0000", 2024-05-15T12:48:53.397 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.812e-06 2024-05-15T12:48:53.397 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.398 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.398 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:53.398 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.135453+0000", 2024-05-15T12:48:53.398 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1215000000000002e-05 2024-05-15T12:48:53.398 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.398 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.398 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:53.398 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.135470+0000", 2024-05-15T12:48:53.398 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7079999999999999e-05 2024-05-15T12:48:53.398 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.398 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.398 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:53.398 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.135480+0000", 2024-05-15T12:48:53.398 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5519999999999993e-06 2024-05-15T12:48:53.399 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.399 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.399 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.399 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.135543+0000", 2024-05-15T12:48:53.399 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3230000000000003e-05 2024-05-15T12:48:53.399 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.399 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.399 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:48:53.399 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.135551+0000", 2024-05-15T12:48:53.399 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6680000000000001e-06 2024-05-15T12:48:53.399 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.399 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.399 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.399 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.026368+0000", 2024-05-15T12:48:53.399 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.89081749799999999 2024-05-15T12:48:53.399 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.400 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.400 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:53.400 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.026486+0000", 2024-05-15T12:48:53.400 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011757899999999999 2024-05-15T12:48:53.400 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.400 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.400 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2", 2024-05-15T12:48:53.400 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.026626+0000", 2024-05-15T12:48:53.400 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014021500000000001 2024-05-15T12:48:53.400 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.400 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.400 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:48:53.400 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.034370+0000", 2024-05-15T12:48:53.400 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0077441139999999999 2024-05-15T12:48:53.400 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.400 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.401 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.401 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.034405+0000", 2024-05-15T12:48:53.401 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4647999999999998e-05 2024-05-15T12:48:53.401 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.401 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.401 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.401 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.034600+0000", 2024-05-15T12:48:53.401 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00019504099999999999 2024-05-15T12:48:53.401 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.401 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.401 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:53.401 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.034633+0000", 2024-05-15T12:48:53.401 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3092000000000001e-05 2024-05-15T12:48:53.401 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.401 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.401 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:53.402 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.034656+0000", 2024-05-15T12:48:53.402 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2687000000000001e-05 2024-05-15T12:48:53.402 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.402 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:53.402 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.402 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.402 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.402 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:57888 2.4 2:2a9dd420:::benchmark_data_smithi177_78104_object57887:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-05-15T12:48:53.402 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:47.210254+0000", 2024-05-15T12:48:53.402 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.9816845890000003, 2024-05-15T12:48:53.402 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.82927775599999998, 2024-05-15T12:48:53.402 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:48:53.402 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:48:53.402 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:48:53.402 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:48:53.403 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:48:53.403 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 57888 2024-05-15T12:48:53.403 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.403 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:48:53.403 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.403 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:48:53.403 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.210254+0000", 2024-05-15T12:48:53.403 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.403 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.403 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.403 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:48:53.403 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.210254+0000", 2024-05-15T12:48:53.403 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:48:53.403 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.403 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.404 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:48:53.404 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.210255+0000", 2024-05-15T12:48:53.404 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0189999999999999e-06 2024-05-15T12:48:53.404 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.404 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.404 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:48:53.404 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.210323+0000", 2024-05-15T12:48:53.404 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8107000000000004e-05 2024-05-15T12:48:53.404 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.404 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.404 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:48:53.404 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.210335+0000", 2024-05-15T12:48:53.404 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1464999999999999e-05 2024-05-15T12:48:53.404 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.404 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.404 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:48:53.405 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.210340+0000", 2024-05-15T12:48:53.405 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7169999999999996e-06 2024-05-15T12:48:53.405 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.405 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.405 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.405 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.210394+0000", 2024-05-15T12:48:53.405 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3198999999999997e-05 2024-05-15T12:48:53.405 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.405 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.405 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:48:53.405 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:47.210401+0000", 2024-05-15T12:48:53.405 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9800000000000001e-06 2024-05-15T12:48:53.405 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.405 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.405 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:48:53.405 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.028643+0000", 2024-05-15T12:48:53.406 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.81824265799999996 2024-05-15T12:48:53.406 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.406 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.406 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:48:53.406 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.028779+0000", 2024-05-15T12:48:53.406 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013562099999999999 2024-05-15T12:48:53.406 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.406 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.406 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2", 2024-05-15T12:48:53.406 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.028966+0000", 2024-05-15T12:48:53.406 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000186703 2024-05-15T12:48:53.406 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.406 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.406 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:48:53.406 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.034558+0000", 2024-05-15T12:48:53.407 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0055927490000000002 2024-05-15T12:48:53.407 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.407 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.407 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.407 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.034710+0000", 2024-05-15T12:48:53.407 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000151643 2024-05-15T12:48:53.407 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.407 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.407 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:48:53.407 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.039429+0000", 2024-05-15T12:48:53.407 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0047195689999999998 2024-05-15T12:48:53.407 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.407 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.407 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:48:53.407 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.039481+0000", 2024-05-15T12:48:53.407 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1375999999999997e-05 2024-05-15T12:48:53.408 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:48:53.408 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:48:53.408 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:48:53.408 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.039532+0000", 2024-05-15T12:48:53.408 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0949999999999998e-05 2024-05-15T12:48:53.408 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.408 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:53.408 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.408 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:48:53.408 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:48:53.408 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T12:48:53.409 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:53 smithi120 ceph-mon[43961]: pgmap v406: 17 pgs: 17 active+clean; 3.7 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s; 9.2 MiB/s, 147 objects/s recovering 2024-05-15T12:48:53.409 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:53 smithi120 ceph-mon[43961]: osdmap e131: 12 total, 12 up, 11 in 2024-05-15T12:48:53.409 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:53 smithi120 ceph-mon[43961]: osdmap e132: 12 total, 12 up, 11 in 2024-05-15T12:48:53.772 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 43 16 66650 66634 96.8404 95.875 0.0154592 0.0103217 2024-05-15T12:48:54.052 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-05-15T12:48:54.070 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:48:54.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:54.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:48:54.107 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:48:54.176 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:48:54.207 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:54.236 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:48:54.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:54.239 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:48:54.340 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:48:54.408 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:54.772 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 44 16 68099 68083 96.6975 90.5625 0.0111221 0.0103373 2024-05-15T12:48:54.778 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:48:54.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:54.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:48:54.884 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:48:54.945 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:54.989 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:48:55.029 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:48:55.031 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:48:55.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:55.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:48:55.073 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:48:55.105 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:48:55.135 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:48:55.153 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:48:55.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:55 smithi027 ceph-mon[39795]: pgmap v409: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s; 1.5 MiB/s, 24 objects/s recovering 2024-05-15T12:48:55.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:55 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:48:55.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:55 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:48:55.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:55 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1003507596' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:48:55.195 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:55.206 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:48:55.236 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:48:55.270 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:48:55.309 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:48:55.337 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:48:55.412 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:48:55.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:55.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:48:55.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:55 smithi177 ceph-mon[44303]: pgmap v409: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s; 1.5 MiB/s, 24 objects/s recovering 2024-05-15T12:48:55.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:55 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:48:55.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:55 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:48:55.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:55 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1003507596' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:48:55.517 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:48:55.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:55 smithi120 ceph-mon[43961]: pgmap v409: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s; 1.5 MiB/s, 24 objects/s recovering 2024-05-15T12:48:55.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:55 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:48:55.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:55 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:48:55.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:55 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1003507596' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:48:55.582 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:48:55.588 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:55.621 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:48:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:48:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:48:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:48:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:48:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:48:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:48:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:48:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:48:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:48:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:48:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:48:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:48:55.653 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T12:48:55.654 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T12:48:55.654 INFO:tasks.thrashosds.thrasher:Removing osd 11, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] 2024-05-15T12:48:55.654 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd out 11 2024-05-15T12:48:55.772 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 45 16 69485 69469 96.4736 86.625 0.00691651 0.0103616 2024-05-15T12:48:56.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:56 smithi027 ceph-mon[39795]: osdmap e133: 12 total, 12 up, 11 in 2024-05-15T12:48:56.404 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:48:56.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:56.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:48:56.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:56 smithi177 ceph-mon[44303]: osdmap e133: 12 total, 12 up, 11 in 2024-05-15T12:48:56.511 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:48:56.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:56 smithi120 ceph-mon[43961]: osdmap e133: 12 total, 12 up, 11 in 2024-05-15T12:48:56.548 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:56.577 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:48:56.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:56.578 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:48:56.679 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:48:56.746 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:56.772 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 46 16 70960 70944 96.3803 92.1875 0.0112691 0.0103706 2024-05-15T12:48:56.819 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:48:56.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:56.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:48:56.929 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:48:56.968 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:56.994 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:48:56.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:56.999 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:48:57.100 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:48:57.137 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:57.160 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:48:57.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:57.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:48:57.215 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:48:57.230 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:48:57.265 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:48:57.293 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:57.320 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:48:57.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:57.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:48:57.325 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:57 smithi120 ceph-mon[43961]: pgmap v411: 17 pgs: 17 active+clean; 4.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 94 MiB/s wr, 1.51k op/s; 1.1 MiB/s, 18 objects/s recovering 2024-05-15T12:48:57.426 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:48:57.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:57 smithi177 ceph-mon[44303]: pgmap v411: 17 pgs: 17 active+clean; 4.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 94 MiB/s wr, 1.51k op/s; 1.1 MiB/s, 18 objects/s recovering 2024-05-15T12:48:57.469 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:57.497 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:48:57.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:57.500 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:48:57.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:57 smithi027 ceph-mon[39795]: pgmap v411: 17 pgs: 17 active+clean; 4.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 94 MiB/s wr, 1.51k op/s; 1.1 MiB/s, 18 objects/s recovering 2024-05-15T12:48:57.601 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:48:57.638 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:57.663 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:48:57.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:57.668 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:48:57.769 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:48:57.772 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 47 16 72405 72389 96.2511 90.3125 0.0128958 0.0103851 2024-05-15T12:48:57.808 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:57.831 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:48:57.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:57.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:48:57.852 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T12:48:57.937 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:48:57.963 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:57.984 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:48:57.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:57.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:48:58.088 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:48:58.124 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:58.151 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:48:58.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:58.158 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:48:58.259 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:48:58.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:58 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/407698385' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:48:58.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:58 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:48:58.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:58 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:48:58.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:58 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:48:58.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:58 smithi027 ceph-mon[39795]: osdmap e134: 12 total, 12 up, 11 in 2024-05-15T12:48:58.294 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:58.320 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:48:58.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:58.324 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:48:58.425 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:48:58.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:58 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/407698385' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:48:58.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:58 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:48:58.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:58 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:48:58.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:58 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:48:58.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:58 smithi177 ceph-mon[44303]: osdmap e134: 12 total, 12 up, 11 in 2024-05-15T12:48:58.455 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:58 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/407698385' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:48:58.455 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:58 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:48:58.455 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:58 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:48:58.455 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:58 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:48:58.455 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:58 smithi120 ceph-mon[43961]: osdmap e134: 12 total, 12 up, 11 in 2024-05-15T12:48:58.458 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:58.483 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:48:58.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:58.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:48:58.588 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:48:58.621 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:58.648 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:48:58.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:58.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:48:58.752 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:48:58.772 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 48 16 73914 73898 96.2105 94.3125 0.0157265 0.0103891 2024-05-15T12:48:58.809 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:59.084 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:48:59 smithi027 ceph-mon[39795]: pgmap v412: 17 pgs: 17 active+clean; 4.1 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-05-15T12:48:59.203 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:48:59.280 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:48:59.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:59.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:48:59.396 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:48:59.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:48:59 smithi177 ceph-mon[44303]: pgmap v412: 17 pgs: 17 active+clean; 4.1 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-05-15T12:48:59.435 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:59.465 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:48:59.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:59.470 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:48:59.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:48:59 smithi120 ceph-mon[43961]: pgmap v412: 17 pgs: 17 active+clean; 4.1 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-05-15T12:48:59.570 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:48:59.622 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:48:59.685 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:48:59.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:48:59.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:48:59.772 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 49 16 75363 75347 96.0951 90.5625 0.00374819 0.010402 2024-05-15T12:48:59.791 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:48:59.849 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:00.734 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:49:00.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:00.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:49:00.773 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 50 16 76876 76860 96.0643 94.5625 0.0129277 0.010404 2024-05-15T12:49:00.848 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:49:00.888 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:00.916 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:49:00.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:00.923 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:49:01.024 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:49:01.063 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:01.089 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:49:01.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:01.092 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:49:01.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:01 smithi177 ceph-mon[44303]: pgmap v414: 17 pgs: 17 active+clean; 4.3 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-05-15T12:49:01.193 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:49:01.226 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:01.255 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:49:01.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:01.262 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:49:01.363 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:49:01.427 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:01 smithi027 ceph-mon[39795]: pgmap v414: 17 pgs: 17 active+clean; 4.3 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-05-15T12:49:01.429 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:01.441 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:49:01.491 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T12:49:01.491 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:49:01.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:01 smithi120 ceph-mon[43961]: pgmap v414: 17 pgs: 17 active+clean; 4.3 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-05-15T12:49:01.773 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 51 16 78281 78265 95.9024 87.8125 0.00368851 0.0104232 2024-05-15T12:49:01.922 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:49:01.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:01.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:49:02.028 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:49:02.057 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:02.080 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:49:02.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:02.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:49:02.185 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:49:02.216 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:02.242 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:49:02.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:02.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:49:02.347 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:49:02.383 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:02.408 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:49:02.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:02.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:49:02.474 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-05-15T12:49:02.514 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:49:02.542 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:02.565 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:49:02.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:02.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:49:02.670 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:49:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:02.773 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 52 16 79668 79652 95.7251 86.6875 0.0298524 0.0104423 2024-05-15T12:49:02.774 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:49:02.782 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T12:49:02.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:02.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:49:02.886 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:49:02.913 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:02.931 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:49:02.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:02.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:49:03.035 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:49:03.067 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:03.091 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:49:03.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:03.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:49:03.196 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:49:03.232 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:03.256 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:49:03.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:03.258 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:49:03.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:03 smithi120 ceph-mon[43961]: pgmap v415: 17 pgs: 17 active+clean; 4.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-05-15T12:49:03.359 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:49:03.418 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:03 smithi027 ceph-mon[39795]: pgmap v415: 17 pgs: 17 active+clean; 4.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-05-15T12:49:03.422 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:03.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:03 smithi177 ceph-mon[44303]: pgmap v415: 17 pgs: 17 active+clean; 4.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-05-15T12:49:03.773 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 53 16 80660 80644 95.0886 62 0.0029061 0.0104955 2024-05-15T12:49:04.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:03 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-a[39772]: 2024-05-15T12:49:03.775+0000 7f8daffd2700 -1 mon.a@0(leader).osd e134 definitely_dead 0 2024-05-15T12:49:04.139 INFO:teuthology.orchestra.run.smithi027.stderr:marked out osd.11. 2024-05-15T12:49:04.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:04 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3386866005' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-05-15T12:49:04.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:04 smithi177 ceph-mon[44303]: Client client.admin marked osd.11 out, while it was still marked up 2024-05-15T12:49:04.505 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:49:04.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:04 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3386866005' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-05-15T12:49:04.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:04 smithi027 ceph-mon[39795]: Client client.admin marked osd.11 out, while it was still marked up 2024-05-15T12:49:04.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:04 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3386866005' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-05-15T12:49:04.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:04 smithi120 ceph-mon[43961]: Client client.admin marked osd.11 out, while it was still marked up 2024-05-15T12:49:04.773 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 54 16 81077 81061 93.8102 26.0625 0.00305412 0.0105305 2024-05-15T12:49:05.208 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:49:05.212 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:05 smithi027 ceph-mon[39795]: pgmap v416: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-05-15T12:49:05.212 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:05 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3386866005' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-05-15T12:49:05.212 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:05 smithi027 ceph-mon[39795]: osdmap e135: 12 total, 12 up, 10 in 2024-05-15T12:49:05.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:05 smithi177 ceph-mon[44303]: pgmap v416: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-05-15T12:49:05.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:05 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3386866005' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-05-15T12:49:05.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:05 smithi177 ceph-mon[44303]: osdmap e135: 12 total, 12 up, 10 in 2024-05-15T12:49:05.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:05 smithi120 ceph-mon[43961]: pgmap v416: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-05-15T12:49:05.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:05 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3386866005' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-05-15T12:49:05.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:05 smithi120 ceph-mon[43961]: osdmap e135: 12 total, 12 up, 10 in 2024-05-15T12:49:05.773 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 55 16 81077 81061 92.1045 0 - 0.0105305 2024-05-15T12:49:06.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:06 smithi027 ceph-mon[39795]: osdmap e136: 12 total, 12 up, 10 in 2024-05-15T12:49:06.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:06 smithi177 ceph-mon[44303]: osdmap e136: 12 total, 12 up, 10 in 2024-05-15T12:49:06.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:06 smithi120 ceph-mon[43961]: osdmap e136: 12 total, 12 up, 10 in 2024-05-15T12:49:06.689 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:49:06.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:06.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:49:06.773 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 56 16 81545 81529 90.9822 14.625 0.0344366 0.0109865 2024-05-15T12:49:06.797 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:49:06.826 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:06.850 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:49:06.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:06.855 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:49:06.956 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:49:06.986 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:07.011 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:49:07.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:07.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:49:07.086 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:49:07.115 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:49:07.147 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:49:07.180 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:49:07.192 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:07.242 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:49:07.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:07 smithi120 ceph-mon[43961]: pgmap v419: 17 pgs: 2 remapped+peering, 1 peering, 14 active+clean; 4.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-05-15T12:49:07.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:07 smithi120 ceph-mon[43961]: osdmap e137: 12 total, 12 up, 10 in 2024-05-15T12:49:07.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:07 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1374365491' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:49:07.281 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:49:07.334 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:49:07.375 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:49:07.379 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:49:07.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:07.390 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:07 smithi027 ceph-mon[39795]: pgmap v419: 17 pgs: 2 remapped+peering, 1 peering, 14 active+clean; 4.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-05-15T12:49:07.391 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:07 smithi027 ceph-mon[39795]: osdmap e137: 12 total, 12 up, 10 in 2024-05-15T12:49:07.391 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:07 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1374365491' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:49:07.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:49:07.428 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:49:07.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:07 smithi177 ceph-mon[44303]: pgmap v419: 17 pgs: 2 remapped+peering, 1 peering, 14 active+clean; 4.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-05-15T12:49:07.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:07 smithi177 ceph-mon[44303]: osdmap e137: 12 total, 12 up, 10 in 2024-05-15T12:49:07.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:07 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1374365491' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:49:07.462 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:49:07.492 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:49:07.499 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:49:07.541 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:49:07.567 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:07.615 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:49:07.625 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:49:07.625 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:49:07.625 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:49:07.625 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:49:07.625 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:49:07.625 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:49:07.626 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:49:07.626 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:49:07.626 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:49:07.626 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:49:07.626 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:49:07.626 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:49:07.777 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 57 16 82433 82417 90.3535 55.5 0.00637926 0.0110634 2024-05-15T12:49:08.148 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:49:08.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:08.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:49:08.262 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:49:08.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:08 smithi027 ceph-mon[39795]: osdmap e138: 12 total, 12 up, 10 in 2024-05-15T12:49:08.299 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:08.325 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:49:08.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:08.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:49:08.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:08 smithi177 ceph-mon[44303]: osdmap e138: 12 total, 12 up, 10 in 2024-05-15T12:49:08.435 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:49:08.466 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:08 smithi120 ceph-mon[43961]: osdmap e138: 12 total, 12 up, 10 in 2024-05-15T12:49:08.469 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:08.488 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:49:08.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:08.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:49:08.592 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:49:08.623 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:08.645 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:49:08.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:08.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:49:08.751 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:49:08.777 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 58 16 83376 83360 89.8118 58.9375 0.0109848 0.0111299 2024-05-15T12:49:08.824 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:09.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:09 smithi027 ceph-mon[39795]: pgmap v422: 17 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 1 peering, 13 active+clean; 4.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 47 MiB/s wr, 751 op/s; 5060/241461 objects misplaced (2.096%); 192 KiB/s, 2 objects/s recovering 2024-05-15T12:49:09.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:09 smithi027 ceph-mon[39795]: osdmap e139: 12 total, 12 up, 10 in 2024-05-15T12:49:09.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:09 smithi177 ceph-mon[44303]: pgmap v422: 17 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 1 peering, 13 active+clean; 4.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 47 MiB/s wr, 751 op/s; 5060/241461 objects misplaced (2.096%); 192 KiB/s, 2 objects/s recovering 2024-05-15T12:49:09.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:09 smithi177 ceph-mon[44303]: osdmap e139: 12 total, 12 up, 10 in 2024-05-15T12:49:09.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:09 smithi120 ceph-mon[43961]: pgmap v422: 17 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 1 peering, 13 active+clean; 4.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 47 MiB/s wr, 751 op/s; 5060/241461 objects misplaced (2.096%); 192 KiB/s, 2 objects/s recovering 2024-05-15T12:49:09.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:09 smithi120 ceph-mon[43961]: osdmap e139: 12 total, 12 up, 10 in 2024-05-15T12:49:09.777 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 59 16 84281 84265 89.2483 56.5625 0.0227866 0.0111996 2024-05-15T12:49:09.860 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T12:49:09.860 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T12:49:09.860 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-05-15T12:49:09.860 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-05-15T12:49:09.860 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T12:49:09.982 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] out_osds: [10, 11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T12:49:09.982 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T12:49:09.982 INFO:tasks.thrashosds.thrasher:Adding osd 10 2024-05-15T12:49:09.982 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd in 10 2024-05-15T12:49:10.182 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:10 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:49:10.182 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:10 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:49:10.183 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:10 smithi027 ceph-mon[39795]: osdmap e140: 12 total, 12 up, 10 in 2024-05-15T12:49:10.183 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:10 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2028020226' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:49:10.183 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:10 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:49:10.262 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T12:49:10.325 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:49:10.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:10.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:49:10.441 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:49:10.476 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:10 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:49:10.476 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:10 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:49:10.476 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:10 smithi177 ceph-mon[44303]: osdmap e140: 12 total, 12 up, 10 in 2024-05-15T12:49:10.476 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:10 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2028020226' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:49:10.476 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:10 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:49:10.478 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:10.501 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:49:10.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:10.504 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:49:10.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:10 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:49:10.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:10 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:49:10.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:10 smithi120 ceph-mon[43961]: osdmap e140: 12 total, 12 up, 10 in 2024-05-15T12:49:10.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:10 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2028020226' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:49:10.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:10 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:49:10.604 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:49:10.648 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:10.677 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:49:10.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:10.680 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:49:10.777 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T12:49:10.788738+0000 min lat: 0.00104218 max lat: 7.07499 avg lat: 0.0112712 2024-05-15T12:49:10.778 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T12:49:10.778 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 60 16 85163 85147 88.6795 55.125 0.0164944 0.0112712 2024-05-15T12:49:10.781 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:49:10.782 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-05-15T12:49:10.816 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:10.841 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:49:10.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:10.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:49:10.946 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:49:10.978 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:10.998 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:49:11.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:11.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:49:11.102 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:49:11.133 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:11.157 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:49:11.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:11.162 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:49:11.186 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:11 smithi120 ceph-mon[43961]: pgmap v425: 17 pgs: 2 active+recovering+undersized+remapped, 2 remapped+peering, 1 peering, 12 active+clean; 5.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 34 MiB/s wr, 548 op/s; 9909/244857 objects misplaced (4.047%); 1.5 MiB/s, 24 objects/s recovering 2024-05-15T12:49:11.186 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:11 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:49:11.187 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:11 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:49:11.187 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:11 smithi120 ceph-mon[43961]: osdmap e141: 12 total, 12 up, 10 in 2024-05-15T12:49:11.263 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:49:11.301 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:11.332 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:49:11.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:11.337 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:49:11.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:11 smithi177 ceph-mon[44303]: pgmap v425: 17 pgs: 2 active+recovering+undersized+remapped, 2 remapped+peering, 1 peering, 12 active+clean; 5.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 34 MiB/s wr, 548 op/s; 9909/244857 objects misplaced (4.047%); 1.5 MiB/s, 24 objects/s recovering 2024-05-15T12:49:11.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:11 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:49:11.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:11 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:49:11.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:11 smithi177 ceph-mon[44303]: osdmap e141: 12 total, 12 up, 10 in 2024-05-15T12:49:11.438 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:49:11.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:11 smithi027 ceph-mon[39795]: pgmap v425: 17 pgs: 2 active+recovering+undersized+remapped, 2 remapped+peering, 1 peering, 12 active+clean; 5.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 34 MiB/s wr, 548 op/s; 9909/244857 objects misplaced (4.047%); 1.5 MiB/s, 24 objects/s recovering 2024-05-15T12:49:11.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:11 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:49:11.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:11 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:49:11.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:11 smithi027 ceph-mon[39795]: osdmap e141: 12 total, 12 up, 10 in 2024-05-15T12:49:11.531 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:11.683 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:49:11.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:11.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:49:11.778 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 61 16 86084 86068 88.1694 57.5625 0.0038721 0.0113369 2024-05-15T12:49:11.796 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:49:11.868 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:12.229 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:49:12.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:12.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:49:12.337 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:49:12.343 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:49:12.362 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:49:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:12.676 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:49:12.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:12.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:49:12.778 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 62 16 86994 86978 87.6646 56.875 0.01561 0.0114017 2024-05-15T12:49:12.784 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:49:12.815 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:12.839 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:49:12.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:12.844 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:49:12.945 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:49:13.012 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:13.050 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:49:13.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:13.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:49:13.157 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:49:13.191 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:13.212 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:49:13.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:13.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:49:13.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:13 smithi027 ceph-mon[39795]: pgmap v427: 17 pgs: 4 active+recovering+undersized+remapped, 13 active+clean; 5.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 46 MiB/s wr, 733 op/s; 19801/251106 objects misplaced (7.886%); 5.7 MiB/s, 90 objects/s recovering 2024-05-15T12:49:13.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:13 smithi027 ceph-mon[39795]: osdmap e142: 12 total, 12 up, 10 in 2024-05-15T12:49:13.315 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:49:13.357 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:13.381 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:49:13.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:13.384 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:49:13.485 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:49:13.519 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:13 smithi177 ceph-mon[44303]: pgmap v427: 17 pgs: 4 active+recovering+undersized+remapped, 13 active+clean; 5.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 46 MiB/s wr, 733 op/s; 19801/251106 objects misplaced (7.886%); 5.7 MiB/s, 90 objects/s recovering 2024-05-15T12:49:13.519 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:13 smithi177 ceph-mon[44303]: osdmap e142: 12 total, 12 up, 10 in 2024-05-15T12:49:13.521 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:13.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:13 smithi120 ceph-mon[43961]: pgmap v427: 17 pgs: 4 active+recovering+undersized+remapped, 13 active+clean; 5.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 46 MiB/s wr, 733 op/s; 19801/251106 objects misplaced (7.886%); 5.7 MiB/s, 90 objects/s recovering 2024-05-15T12:49:13.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:13 smithi120 ceph-mon[43961]: osdmap e142: 12 total, 12 up, 10 in 2024-05-15T12:49:13.545 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:49:13.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:13.549 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:49:13.650 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:49:13.689 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:13.714 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:49:13.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:13.718 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:49:13.734 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:49:13.778 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 63 16 87899 87883 87.1709 56.5625 0.0047631 0.0114673 2024-05-15T12:49:13.818 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:49:13.893 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:14.257 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T12:49:14.282 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:49:14.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:14.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:49:14.392 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:49:14.424 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:14.446 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:49:14.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:14.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:49:14.557 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:49:14.588 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:14.609 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:49:14.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:14.612 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:49:14.713 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:49:14.743 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:14.769 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:49:14.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:14.774 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:49:14.778 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 64 16 88713 88697 86.6037 50.875 0.0186862 0.0115415 2024-05-15T12:49:14.876 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:49:14.908 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:14.935 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:49:14.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:14.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:49:15.041 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:49:15.115 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:15.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:15 smithi027 ceph-mon[39795]: pgmap v429: 17 pgs: 4 active+recovering+undersized+remapped, 13 active+clean; 5.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 52 MiB/s wr, 830 op/s; 19393/256992 objects misplaced (7.546%); 9.4 MiB/s, 149 objects/s recovering 2024-05-15T12:49:15.342 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:49:15.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:15.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:49:15.450 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:49:15.481 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:15 smithi177 ceph-mon[44303]: pgmap v429: 17 pgs: 4 active+recovering+undersized+remapped, 13 active+clean; 5.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 52 MiB/s wr, 830 op/s; 19393/256992 objects misplaced (7.546%); 9.4 MiB/s, 149 objects/s recovering 2024-05-15T12:49:15.483 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:15.509 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:49:15.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:15.514 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:49:15.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:15 smithi120 ceph-mon[43961]: pgmap v429: 17 pgs: 4 active+recovering+undersized+remapped, 13 active+clean; 5.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 52 MiB/s wr, 830 op/s; 19393/256992 objects misplaced (7.546%); 9.4 MiB/s, 149 objects/s recovering 2024-05-15T12:49:15.615 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:49:15.650 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:15.673 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:49:15.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:15.676 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:49:15.777 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:49:15.778 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 65 16 89590 89574 86.1146 54.8125 0.0493015 0.0116076 2024-05-15T12:49:15.816 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:15.842 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:49:15.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:15.846 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:49:15.947 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:49:15.977 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:15.999 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:49:16.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:16.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:49:16.103 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:49:16.159 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T12:49:16.160 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T12:49:16.160 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T12:49:16.160 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T12:49:16.160 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.160 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:51713 2.1 2:8f803db6:::benchmark_data_smithi177_78104_object51712:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-15T12:49:16.160 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:42.601217+0000", 2024-05-15T12:49:16.160 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 33.511151185999999, 2024-05-15T12:49:16.160 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.098120633999999998, 2024-05-15T12:49:16.160 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:49:16.160 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:49:16.160 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:49:16.160 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:49:16.160 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:49:16.161 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 51713 2024-05-15T12:49:16.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.161 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:49:16.161 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.161 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:49:16.161 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.601217+0000", 2024-05-15T12:49:16.161 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.161 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.161 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:49:16.161 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.601217+0000", 2024-05-15T12:49:16.161 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.162 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:49:16.162 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.601218+0000", 2024-05-15T12:49:16.162 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0589999999999999e-06 2024-05-15T12:49:16.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.162 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:49:16.162 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.601366+0000", 2024-05-15T12:49:16.162 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014783099999999999 2024-05-15T12:49:16.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.162 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:49:16.162 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.601377+0000", 2024-05-15T12:49:16.163 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.077e-05 2024-05-15T12:49:16.163 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.163 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.163 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:49:16.163 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.601386+0000", 2024-05-15T12:49:16.163 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7069999999999998e-06 2024-05-15T12:49:16.163 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.163 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.163 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:49:16.163 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.602166+0000", 2024-05-15T12:49:16.164 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00078028799999999997 2024-05-15T12:49:16.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.164 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.164 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:49:16.164 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.602279+0000", 2024-05-15T12:49:16.164 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000112859 2024-05-15T12:49:16.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,3", 2024-05-15T12:49:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.602412+0000", 2024-05-15T12:49:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013315899999999999 2024-05-15T12:49:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.690434+0000", 2024-05-15T12:49:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.088022144999999996 2024-05-15T12:49:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.690477+0000", 2024-05-15T12:49:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2521999999999997e-05 2024-05-15T12:49:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:49:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.699238+0000", 2024-05-15T12:49:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0087611070000000006 2024-05-15T12:49:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:49:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.699290+0000", 2024-05-15T12:49:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1999999999999997e-05 2024-05-15T12:49:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:49:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.699338+0000", 2024-05-15T12:49:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8186999999999998e-05 2024-05-15T12:49:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:49:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:58488 2.1 2:8c133fc7:::benchmark_data_smithi177_78104_object58487:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-15T12:49:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:48.359181+0000", 2024-05-15T12:49:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 27.753187691000001, 2024-05-15T12:49:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.105747596, 2024-05-15T12:49:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:49:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:49:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:49:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:49:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:49:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 58488 2024-05-15T12:49:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:49:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:49:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359181+0000", 2024-05-15T12:49:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:49:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359181+0000", 2024-05-15T12:49:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:49:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359182+0000", 2024-05-15T12:49:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1659999999999999e-06 2024-05-15T12:49:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:49:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359212+0000", 2024-05-15T12:49:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9657000000000001e-05 2024-05-15T12:49:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:49:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359220+0000", 2024-05-15T12:49:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2810000000000008e-06 2024-05-15T12:49:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:49:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359226+0000", 2024-05-15T12:49:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2169999999999999e-06 2024-05-15T12:49:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:49:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359282+0000", 2024-05-15T12:49:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5878000000000003e-05 2024-05-15T12:49:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:49:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359431+0000", 2024-05-15T12:49:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014895899999999999 2024-05-15T12:49:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,3", 2024-05-15T12:49:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359608+0000", 2024-05-15T12:49:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017731299999999999 2024-05-15T12:49:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.363541+0000", 2024-05-15T12:49:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039324939999999999 2024-05-15T12:49:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:49:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.363625+0000", 2024-05-15T12:49:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3719000000000003e-05 2024-05-15T12:49:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464802+0000", 2024-05-15T12:49:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.101177645 2024-05-15T12:49:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:49:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464874+0000", 2024-05-15T12:49:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2324999999999996e-05 2024-05-15T12:49:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:49:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464928+0000", 2024-05-15T12:49:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3942e-05 2024-05-15T12:49:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:49:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:58490 2.1 2:80921832:::benchmark_data_smithi177_78104_object58489:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-15T12:49:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:48.359238+0000", 2024-05-15T12:49:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 27.753131011000001, 2024-05-15T12:49:16.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10584242100000001, 2024-05-15T12:49:16.174 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:49:16.174 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:49:16.174 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:49:16.174 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:49:16.174 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:49:16.174 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 58490 2024-05-15T12:49:16.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.174 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:49:16.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:49:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359238+0000", 2024-05-15T12:49:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:49:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359238+0000", 2024-05-15T12:49:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:49:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359238+0000", 2024-05-15T12:49:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3500000000000002e-07 2024-05-15T12:49:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:49:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359388+0000", 2024-05-15T12:49:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015008 2024-05-15T12:49:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:49:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359398+0000", 2024-05-15T12:49:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7569999999999999e-06 2024-05-15T12:49:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:49:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359405+0000", 2024-05-15T12:49:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7970000000000003e-06 2024-05-15T12:49:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:49:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361472+0000", 2024-05-15T12:49:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020671880000000002 2024-05-15T12:49:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:49:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361583+0000", 2024-05-15T12:49:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000110745 2024-05-15T12:49:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,3", 2024-05-15T12:49:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361714+0000", 2024-05-15T12:49:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013185399999999999 2024-05-15T12:49:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.178 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.178 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.363611+0000", 2024-05-15T12:49:16.178 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0018970689999999999 2024-05-15T12:49:16.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.178 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:49:16.178 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367554+0000", 2024-05-15T12:49:16.178 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039426879999999997 2024-05-15T12:49:16.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.178 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464987+0000", 2024-05-15T12:49:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.097432747 2024-05-15T12:49:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:49:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465043+0000", 2024-05-15T12:49:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.588e-05 2024-05-15T12:49:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:49:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465080+0000", 2024-05-15T12:49:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7181000000000002e-05 2024-05-15T12:49:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:49:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:58493 2.1 2:81994a98:::benchmark_data_smithi177_78104_object58492:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-15T12:49:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:48.359509+0000", 2024-05-15T12:49:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 27.752859612999998, 2024-05-15T12:49:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.105753467, 2024-05-15T12:49:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:49:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:49:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:49:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:49:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:49:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 58493 2024-05-15T12:49:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:49:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:49:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359509+0000", 2024-05-15T12:49:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:49:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359509+0000", 2024-05-15T12:49:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:49:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359510+0000", 2024-05-15T12:49:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8599999999999997e-07 2024-05-15T12:49:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:49:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359561+0000", 2024-05-15T12:49:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0909999999999999e-05 2024-05-15T12:49:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:49:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359570+0000", 2024-05-15T12:49:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9579999999999996e-06 2024-05-15T12:49:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:49:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359576+0000", 2024-05-15T12:49:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6320000000000002e-06 2024-05-15T12:49:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:49:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.363643+0000", 2024-05-15T12:49:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0040667409999999996 2024-05-15T12:49:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:49:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.363730+0000", 2024-05-15T12:49:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6646999999999999e-05 2024-05-15T12:49:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,3", 2024-05-15T12:49:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.363815+0000", 2024-05-15T12:49:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5136000000000006e-05 2024-05-15T12:49:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:49:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464946+0000", 2024-05-15T12:49:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.101130962 2024-05-15T12:49:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465138+0000", 2024-05-15T12:49:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00019219199999999999 2024-05-15T12:49:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465177+0000", 2024-05-15T12:49:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9573999999999998e-05 2024-05-15T12:49:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:49:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465225+0000", 2024-05-15T12:49:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7426999999999998e-05 2024-05-15T12:49:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:49:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465262+0000", 2024-05-15T12:49:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7502000000000001e-05 2024-05-15T12:49:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:49:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:58498 2.1 2:8b03664e:::benchmark_data_smithi177_78104_object58497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-15T12:49:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:48.361834+0000", 2024-05-15T12:49:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 27.750534545000001, 2024-05-15T12:49:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10363879400000001, 2024-05-15T12:49:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:49:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:49:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:49:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:49:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:49:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 58498 2024-05-15T12:49:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:49:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:49:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361834+0000", 2024-05-15T12:49:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:49:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361834+0000", 2024-05-15T12:49:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:49:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361835+0000", 2024-05-15T12:49:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.155e-06 2024-05-15T12:49:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:49:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361876+0000", 2024-05-15T12:49:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1337000000000001e-05 2024-05-15T12:49:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:49:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361887+0000", 2024-05-15T12:49:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0589e-05 2024-05-15T12:49:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:49:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361894+0000", 2024-05-15T12:49:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.289e-06 2024-05-15T12:49:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:49:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365486+0000", 2024-05-15T12:49:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035911979999999999 2024-05-15T12:49:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:49:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365624+0000", 2024-05-15T12:49:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000138281 2024-05-15T12:49:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,3", 2024-05-15T12:49:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365802+0000", 2024-05-15T12:49:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017837500000000001 2024-05-15T12:49:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:49:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465090+0000", 2024-05-15T12:49:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.099288247999999996 2024-05-15T12:49:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465303+0000", 2024-05-15T12:49:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00021270499999999999 2024-05-15T12:49:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465390+0000", 2024-05-15T12:49:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6355999999999998e-05 2024-05-15T12:49:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:49:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465437+0000", 2024-05-15T12:49:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7642000000000002e-05 2024-05-15T12:49:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:49:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465473+0000", 2024-05-15T12:49:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5618999999999999e-05 2024-05-15T12:49:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:49:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.192 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.192 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.24562.0:58488 2.1 e129/110 ondisk, result = 0)", 2024-05-15T12:49:16.192 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:48.365352+0000", 2024-05-15T12:49:16.192 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 27.747016715000001, 2024-05-15T12:49:16.192 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.099595227999999994, 2024-05-15T12:49:16.192 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:49:16.192 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T12:49:16.192 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:49:16.192 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.192 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:49:16.192 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365352+0000", 2024-05-15T12:49:16.192 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.192 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.192 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.193 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:49:16.193 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365352+0000", 2024-05-15T12:49:16.193 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.193 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.193 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.193 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:49:16.193 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365352+0000", 2024-05-15T12:49:16.193 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5699999999999998e-07 2024-05-15T12:49:16.193 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.193 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.193 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:49:16.193 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365360+0000", 2024-05-15T12:49:16.193 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6690000000000006e-06 2024-05-15T12:49:16.193 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.194 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.194 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:49:16.194 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365367+0000", 2024-05-15T12:49:16.194 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4939999999999997e-06 2024-05-15T12:49:16.194 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.194 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.194 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:49:16.194 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365378+0000", 2024-05-15T12:49:16.194 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.066e-05 2024-05-15T12:49:16.194 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.194 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.194 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:49:16.194 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464785+0000", 2024-05-15T12:49:16.194 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.099406904000000004 2024-05-15T12:49:16.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.195 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:49:16.195 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464795+0000", 2024-05-15T12:49:16.195 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0076e-05 2024-05-15T12:49:16.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.195 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:49:16.195 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464947+0000", 2024-05-15T12:49:16.195 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015206799999999999 2024-05-15T12:49:16.196 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.196 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:49:16.196 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.196 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.196 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.196 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:58501 2.4 e129/128 2:2c05dc93:::benchmark_data_smithi177_78104_object58500:head v 129'3650, mlcod=129'3648)", 2024-05-15T12:49:16.196 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:48.365547+0000", 2024-05-15T12:49:16.196 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 27.746821758999999, 2024-05-15T12:49:16.196 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10243205599999999, 2024-05-15T12:49:16.196 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:49:16.196 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:49:16.196 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:49:16.196 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.196 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:49:16.197 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365547+0000", 2024-05-15T12:49:16.197 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.197 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.197 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:49:16.197 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365547+0000", 2024-05-15T12:49:16.197 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.197 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.197 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:49:16.197 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365547+0000", 2024-05-15T12:49:16.197 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0800000000000004e-07 2024-05-15T12:49:16.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.197 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.197 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:49:16.198 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365571+0000", 2024-05-15T12:49:16.198 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3992e-05 2024-05-15T12:49:16.198 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.198 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.198 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:49:16.198 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365582+0000", 2024-05-15T12:49:16.198 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0929e-05 2024-05-15T12:49:16.198 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.198 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.198 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:49:16.198 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365589+0000", 2024-05-15T12:49:16.198 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.545e-06 2024-05-15T12:49:16.199 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.199 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.199 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:49:16.202 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365641+0000", 2024-05-15T12:49:16.202 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1969000000000003e-05 2024-05-15T12:49:16.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.202 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.202 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:49:16.202 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365664+0000", 2024-05-15T12:49:16.202 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3442999999999999e-05 2024-05-15T12:49:16.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.202 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.202 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:49:16.202 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.467932+0000", 2024-05-15T12:49:16.203 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.102268092 2024-05-15T12:49:16.203 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.203 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.203 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:49:16.203 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.467979+0000", 2024-05-15T12:49:16.203 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6378000000000002e-05 2024-05-15T12:49:16.203 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.203 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:49:16.203 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.203 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.203 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.203 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:58506 2.e 2:73b262f3:::benchmark_data_smithi177_78104_object58505:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-15T12:49:16.203 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:48.365633+0000", 2024-05-15T12:49:16.203 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 27.746735314999999, 2024-05-15T12:49:16.203 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.106977115, 2024-05-15T12:49:16.204 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:49:16.204 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:49:16.204 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:49:16.204 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:49:16.204 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:49:16.204 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 58506 2024-05-15T12:49:16.204 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.204 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:49:16.204 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.204 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:49:16.204 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365633+0000", 2024-05-15T12:49:16.204 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.204 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.204 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.205 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:49:16.205 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365633+0000", 2024-05-15T12:49:16.205 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.205 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.205 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.205 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:49:16.205 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365634+0000", 2024-05-15T12:49:16.205 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1289999999999999e-06 2024-05-15T12:49:16.205 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.205 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.205 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:49:16.206 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367598+0000", 2024-05-15T12:49:16.206 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019633670000000002 2024-05-15T12:49:16.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.206 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.206 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:49:16.206 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367609+0000", 2024-05-15T12:49:16.206 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1491000000000001e-05 2024-05-15T12:49:16.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.206 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.206 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:49:16.206 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367617+0000", 2024-05-15T12:49:16.206 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9270000000000005e-06 2024-05-15T12:49:16.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.206 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.207 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:49:16.207 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367669+0000", 2024-05-15T12:49:16.207 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2287000000000003e-05 2024-05-15T12:49:16.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.207 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.207 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:49:16.207 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367777+0000", 2024-05-15T12:49:16.207 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000107895 2024-05-15T12:49:16.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.207 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.207 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:49:16.207 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464792+0000", 2024-05-15T12:49:16.207 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.097014787000000005 2024-05-15T12:49:16.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.208 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.208 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:49:16.208 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.469572+0000", 2024-05-15T12:49:16.208 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0047797839999999996 2024-05-15T12:49:16.208 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.208 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.208 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.208 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.471658+0000", 2024-05-15T12:49:16.208 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020863610000000001 2024-05-15T12:49:16.208 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.208 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.208 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.208 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.472506+0000", 2024-05-15T12:49:16.208 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00084819400000000003 2024-05-15T12:49:16.208 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.209 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.209 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:49:16.209 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.472563+0000", 2024-05-15T12:49:16.209 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6164e-05 2024-05-15T12:49:16.209 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.209 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.209 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:49:16.209 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.472610+0000", 2024-05-15T12:49:16.209 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7729000000000003e-05 2024-05-15T12:49:16.209 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.209 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:49:16.209 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.209 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.209 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.210 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:58508 2.1 2:8ad9250e:::benchmark_data_smithi177_78104_object58507:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-15T12:49:16.210 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:48.367638+0000", 2024-05-15T12:49:16.210 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 27.744730125, 2024-05-15T12:49:16.210 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.107729793, 2024-05-15T12:49:16.210 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:49:16.210 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:49:16.210 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:49:16.210 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:49:16.210 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:49:16.210 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 58508 2024-05-15T12:49:16.210 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.210 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:49:16.210 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.210 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:49:16.211 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367638+0000", 2024-05-15T12:49:16.211 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.211 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.211 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.211 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:49:16.211 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367638+0000", 2024-05-15T12:49:16.211 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.211 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.211 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.211 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:49:16.211 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367640+0000", 2024-05-15T12:49:16.211 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4389999999999999e-06 2024-05-15T12:49:16.211 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.211 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.212 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:49:16.212 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367672+0000", 2024-05-15T12:49:16.212 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2190999999999997e-05 2024-05-15T12:49:16.212 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.212 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.212 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:49:16.212 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367681+0000", 2024-05-15T12:49:16.212 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2110000000000004e-06 2024-05-15T12:49:16.212 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.212 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.212 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:49:16.212 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367688+0000", 2024-05-15T12:49:16.212 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8800000000000002e-06 2024-05-15T12:49:16.212 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.212 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.213 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:49:16.213 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465627+0000", 2024-05-15T12:49:16.213 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.097938774000000006 2024-05-15T12:49:16.213 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.213 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.213 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:49:16.213 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465788+0000", 2024-05-15T12:49:16.213 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000160631 2024-05-15T12:49:16.213 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.213 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.213 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,3", 2024-05-15T12:49:16.213 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465946+0000", 2024-05-15T12:49:16.213 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000158918 2024-05-15T12:49:16.213 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.213 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.214 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:49:16.214 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.469751+0000", 2024-05-15T12:49:16.214 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038043389999999999 2024-05-15T12:49:16.214 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.214 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.214 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.214 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.473812+0000", 2024-05-15T12:49:16.214 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0040614559999999997 2024-05-15T12:49:16.214 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.214 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.214 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.214 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.475272+0000", 2024-05-15T12:49:16.214 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0014601359999999999 2024-05-15T12:49:16.214 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.215 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.215 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:49:16.215 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.475329+0000", 2024-05-15T12:49:16.215 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.622e-05 2024-05-15T12:49:16.215 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.215 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.215 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:49:16.215 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.475368+0000", 2024-05-15T12:49:16.215 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9598000000000003e-05 2024-05-15T12:49:16.215 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.215 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:49:16.215 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.215 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.215 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.215 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89600 2.e 2:763b1fd1:::benchmark_data_smithi177_78104_object89599:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:49:16.216 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.804057+0000", 2024-05-15T12:49:16.216 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.30831163700000003, 2024-05-15T12:49:16.216 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.21480517800000001, 2024-05-15T12:49:16.216 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:49:16.216 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:49:16.216 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:49:16.216 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:49:16.216 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:49:16.216 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89600 2024-05-15T12:49:16.216 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.216 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:49:16.217 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.217 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:49:16.217 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804057+0000", 2024-05-15T12:49:16.217 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.217 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.217 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.217 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:49:16.217 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804057+0000", 2024-05-15T12:49:16.217 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.217 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.217 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.217 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:49:16.217 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804058+0000", 2024-05-15T12:49:16.217 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0690000000000001e-06 2024-05-15T12:49:16.218 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.218 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.218 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:49:16.218 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804179+0000", 2024-05-15T12:49:16.218 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120938 2024-05-15T12:49:16.218 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.218 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.218 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:49:16.218 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804191+0000", 2024-05-15T12:49:16.218 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1868000000000001e-05 2024-05-15T12:49:16.218 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.218 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.218 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:49:16.218 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804197+0000", 2024-05-15T12:49:16.219 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2550000000000003e-06 2024-05-15T12:49:16.219 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.219 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.219 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:49:16.219 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804252+0000", 2024-05-15T12:49:16.219 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4953000000000001e-05 2024-05-15T12:49:16.219 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.219 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.219 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:49:16.219 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804392+0000", 2024-05-15T12:49:16.219 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000139772 2024-05-15T12:49:16.219 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.219 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.220 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:49:16.220 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804513+0000", 2024-05-15T12:49:16.220 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120763 2024-05-15T12:49:16.220 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.220 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.220 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:49:16.220 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.808055+0000", 2024-05-15T12:49:16.220 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035428830000000001 2024-05-15T12:49:16.220 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.220 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.220 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.220 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.819733+0000", 2024-05-15T12:49:16.220 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011677682999999999 2024-05-15T12:49:16.220 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.221 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.221 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.221 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.018742+0000", 2024-05-15T12:49:16.221 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19900900199999999 2024-05-15T12:49:16.221 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.221 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.221 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:49:16.221 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.018812+0000", 2024-05-15T12:49:16.221 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9924000000000003e-05 2024-05-15T12:49:16.221 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.221 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.221 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:49:16.221 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.018862+0000", 2024-05-15T12:49:16.221 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0068000000000002e-05 2024-05-15T12:49:16.221 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.222 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:49:16.222 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.222 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.222 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.222 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89634 2.e 2:7b9cebfb:::benchmark_data_smithi177_78104_object89633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:49:16.222 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.841129+0000", 2024-05-15T12:49:16.222 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.27123940099999999, 2024-05-15T12:49:16.222 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.18161068899999999, 2024-05-15T12:49:16.222 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:49:16.222 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:49:16.222 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:49:16.222 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:49:16.222 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:49:16.223 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89634 2024-05-15T12:49:16.223 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.223 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:49:16.223 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.223 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:49:16.223 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841129+0000", 2024-05-15T12:49:16.223 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.223 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.223 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.223 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:49:16.223 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841129+0000", 2024-05-15T12:49:16.223 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.223 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.224 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.224 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:49:16.224 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841131+0000", 2024-05-15T12:49:16.224 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.699e-06 2024-05-15T12:49:16.224 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.224 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.224 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:49:16.224 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841163+0000", 2024-05-15T12:49:16.224 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2499999999999997e-05 2024-05-15T12:49:16.224 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.224 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.224 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:49:16.224 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841181+0000", 2024-05-15T12:49:16.224 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7436e-05 2024-05-15T12:49:16.225 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.225 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.225 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:49:16.225 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841190+0000", 2024-05-15T12:49:16.225 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.02e-06 2024-05-15T12:49:16.225 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.225 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.225 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:49:16.225 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841264+0000", 2024-05-15T12:49:16.225 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3866000000000001e-05 2024-05-15T12:49:16.225 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.225 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.225 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:49:16.225 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841398+0000", 2024-05-15T12:49:16.225 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013442900000000001 2024-05-15T12:49:16.226 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.226 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.226 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:49:16.226 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841544+0000", 2024-05-15T12:49:16.226 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014611800000000001 2024-05-15T12:49:16.226 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.226 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.226 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:49:16.226 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.844757+0000", 2024-05-15T12:49:16.226 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032125180000000001 2024-05-15T12:49:16.227 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.227 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.227 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.227 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.850454+0000", 2024-05-15T12:49:16.227 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.005697667 2024-05-15T12:49:16.227 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.227 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.227 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.227 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.022628+0000", 2024-05-15T12:49:16.227 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.17217374299999999 2024-05-15T12:49:16.227 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.227 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.227 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:49:16.227 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.022690+0000", 2024-05-15T12:49:16.227 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1566e-05 2024-05-15T12:49:16.227 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.228 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.228 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:49:16.228 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.022740+0000", 2024-05-15T12:49:16.228 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0127000000000003e-05 2024-05-15T12:49:16.228 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.228 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:49:16.228 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.228 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.228 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.228 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89645 2.e 2:7ffe4884:::benchmark_data_smithi177_78104_object89644:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:49:16.228 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.858184+0000", 2024-05-15T12:49:16.228 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.25418413899999998, 2024-05-15T12:49:16.228 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.164674923, 2024-05-15T12:49:16.228 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:49:16.228 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:49:16.229 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:49:16.229 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:49:16.229 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:49:16.229 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89645 2024-05-15T12:49:16.229 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.229 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:49:16.229 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.229 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:49:16.229 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858184+0000", 2024-05-15T12:49:16.229 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.229 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.229 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.229 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:49:16.229 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858184+0000", 2024-05-15T12:49:16.229 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.229 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.230 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.230 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:49:16.230 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858186+0000", 2024-05-15T12:49:16.230 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5850000000000001e-06 2024-05-15T12:49:16.230 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.230 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.230 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:49:16.230 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858245+0000", 2024-05-15T12:49:16.230 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8729e-05 2024-05-15T12:49:16.230 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.230 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.230 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:49:16.230 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858258+0000", 2024-05-15T12:49:16.230 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3767999999999999e-05 2024-05-15T12:49:16.231 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.231 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.231 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:49:16.231 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858265+0000", 2024-05-15T12:49:16.231 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.725e-06 2024-05-15T12:49:16.231 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.231 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.231 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:49:16.231 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858324+0000", 2024-05-15T12:49:16.231 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8910999999999999e-05 2024-05-15T12:49:16.231 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.231 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.232 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:49:16.232 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858435+0000", 2024-05-15T12:49:16.232 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000111068 2024-05-15T12:49:16.232 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.232 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.232 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:49:16.232 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858592+0000", 2024-05-15T12:49:16.232 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000157083 2024-05-15T12:49:16.232 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.232 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.232 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:49:16.232 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.871361+0000", 2024-05-15T12:49:16.232 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012768662 2024-05-15T12:49:16.232 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.233 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.233 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.233 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.872157+0000", 2024-05-15T12:49:16.233 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00079593199999999995 2024-05-15T12:49:16.233 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.233 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.233 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.233 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.022780+0000", 2024-05-15T12:49:16.233 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.150623014 2024-05-15T12:49:16.233 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.233 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.233 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:49:16.233 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.022825+0000", 2024-05-15T12:49:16.233 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5626e-05 2024-05-15T12:49:16.234 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.234 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.234 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:49:16.234 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.022859+0000", 2024-05-15T12:49:16.234 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3819999999999998e-05 2024-05-15T12:49:16.234 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.234 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:49:16.234 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.234 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.234 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.234 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89654 2.e 2:7e1530a2:::benchmark_data_smithi177_78104_object89653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:49:16.234 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.874001+0000", 2024-05-15T12:49:16.235 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.238367883, 2024-05-15T12:49:16.235 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.15396952, 2024-05-15T12:49:16.235 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:49:16.235 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:49:16.235 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:49:16.235 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:49:16.235 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:49:16.235 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89654 2024-05-15T12:49:16.235 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.235 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:49:16.235 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.235 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:49:16.235 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874001+0000", 2024-05-15T12:49:16.236 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.236 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.236 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.236 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:49:16.236 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874001+0000", 2024-05-15T12:49:16.236 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.236 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.236 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.236 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:49:16.236 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874002+0000", 2024-05-15T12:49:16.236 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6199999999999999e-06 2024-05-15T12:49:16.236 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.237 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.237 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:49:16.237 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874045+0000", 2024-05-15T12:49:16.237 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3191e-05 2024-05-15T12:49:16.237 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.237 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.237 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:49:16.237 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874061+0000", 2024-05-15T12:49:16.237 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6002999999999999e-05 2024-05-15T12:49:16.237 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.237 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.238 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:49:16.238 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874072+0000", 2024-05-15T12:49:16.238 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0647e-05 2024-05-15T12:49:16.238 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.238 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.238 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:49:16.238 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874137+0000", 2024-05-15T12:49:16.238 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4941e-05 2024-05-15T12:49:16.238 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.238 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.239 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:49:16.239 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874222+0000", 2024-05-15T12:49:16.239 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4581999999999998e-05 2024-05-15T12:49:16.239 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.239 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.239 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:49:16.240 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874308+0000", 2024-05-15T12:49:16.240 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6658999999999998e-05 2024-05-15T12:49:16.240 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.240 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.240 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:49:16.240 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.876658+0000", 2024-05-15T12:49:16.240 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0023498080000000001 2024-05-15T12:49:16.240 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.240 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.240 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.240 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.893634+0000", 2024-05-15T12:49:16.241 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016976104999999998 2024-05-15T12:49:16.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.241 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.241 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.241 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.027858+0000", 2024-05-15T12:49:16.241 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.134223696 2024-05-15T12:49:16.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.241 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.241 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:49:16.241 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.027919+0000", 2024-05-15T12:49:16.241 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1464000000000006e-05 2024-05-15T12:49:16.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.241 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.241 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:49:16.241 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.027970+0000", 2024-05-15T12:49:16.242 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0803999999999997e-05 2024-05-15T12:49:16.242 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.242 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:49:16.242 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.242 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.242 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.242 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89661 2.e 2:7462a28b:::benchmark_data_smithi177_78104_object89660:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:49:16.242 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.887128+0000", 2024-05-15T12:49:16.242 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.22524065500000001, 2024-05-15T12:49:16.242 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.14095980499999999, 2024-05-15T12:49:16.242 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:49:16.242 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:49:16.242 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:49:16.243 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:49:16.243 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:49:16.243 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89661 2024-05-15T12:49:16.243 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.243 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:49:16.243 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.243 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:49:16.243 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887128+0000", 2024-05-15T12:49:16.243 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.243 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.243 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.243 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:49:16.243 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887128+0000", 2024-05-15T12:49:16.243 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.243 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.244 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.244 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:49:16.244 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887130+0000", 2024-05-15T12:49:16.244 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7969999999999999e-06 2024-05-15T12:49:16.244 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.244 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.244 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:49:16.244 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887162+0000", 2024-05-15T12:49:16.244 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2780000000000001e-05 2024-05-15T12:49:16.244 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.244 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.245 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:49:16.245 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887180+0000", 2024-05-15T12:49:16.245 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7187999999999999e-05 2024-05-15T12:49:16.245 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.245 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.245 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:49:16.245 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887190+0000", 2024-05-15T12:49:16.245 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8980000000000008e-06 2024-05-15T12:49:16.245 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.245 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.245 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:49:16.245 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887253+0000", 2024-05-15T12:49:16.245 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3724000000000002e-05 2024-05-15T12:49:16.245 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.246 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.246 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:49:16.246 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887391+0000", 2024-05-15T12:49:16.246 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013803700000000001 2024-05-15T12:49:16.246 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.246 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.246 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:49:16.246 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887549+0000", 2024-05-15T12:49:16.246 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015766599999999999 2024-05-15T12:49:16.246 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.246 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.246 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:49:16.246 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.891069+0000", 2024-05-15T12:49:16.246 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035196989999999998 2024-05-15T12:49:16.247 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.247 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.247 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.247 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.894586+0000", 2024-05-15T12:49:16.247 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035178269999999998 2024-05-15T12:49:16.247 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.247 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.247 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.248 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.028009+0000", 2024-05-15T12:49:16.248 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.13342280400000001 2024-05-15T12:49:16.248 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.248 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.248 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:49:16.248 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.028054+0000", 2024-05-15T12:49:16.248 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5207000000000003e-05 2024-05-15T12:49:16.248 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.248 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.248 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:49:16.248 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.028088+0000", 2024-05-15T12:49:16.249 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3178e-05 2024-05-15T12:49:16.249 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.249 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:49:16.249 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.250 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.250 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.250 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89672 2.e 2:7301c129:::benchmark_data_smithi177_78104_object89671:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:49:16.250 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.899549+0000", 2024-05-15T12:49:16.250 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.212819547, 2024-05-15T12:49:16.250 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.13251970099999999, 2024-05-15T12:49:16.250 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:49:16.250 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:49:16.250 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:49:16.251 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:49:16.251 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:49:16.251 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89672 2024-05-15T12:49:16.251 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.251 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:49:16.251 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.251 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:49:16.251 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.899549+0000", 2024-05-15T12:49:16.251 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.251 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.251 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.251 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:49:16.251 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.899549+0000", 2024-05-15T12:49:16.251 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.252 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.252 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.252 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:49:16.252 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.899551+0000", 2024-05-15T12:49:16.252 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.826e-06 2024-05-15T12:49:16.252 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.252 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.252 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:49:16.252 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.899591+0000", 2024-05-15T12:49:16.252 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0422e-05 2024-05-15T12:49:16.252 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.252 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.252 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:49:16.252 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.899606+0000", 2024-05-15T12:49:16.252 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5233e-05 2024-05-15T12:49:16.253 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.253 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.253 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:49:16.253 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.899617+0000", 2024-05-15T12:49:16.254 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0186999999999999e-05 2024-05-15T12:49:16.254 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.254 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.254 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:49:16.254 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.899682+0000", 2024-05-15T12:49:16.254 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4869000000000005e-05 2024-05-15T12:49:16.254 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.254 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.254 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:49:16.254 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.899838+0000", 2024-05-15T12:49:16.254 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015638300000000001 2024-05-15T12:49:16.254 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.254 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.255 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:49:16.255 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.900004+0000", 2024-05-15T12:49:16.255 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000165835 2024-05-15T12:49:16.255 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.255 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.255 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.255 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.911032+0000", 2024-05-15T12:49:16.255 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011028692999999999 2024-05-15T12:49:16.255 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.255 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.255 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:49:16.255 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.917176+0000", 2024-05-15T12:49:16.255 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.006143333 2024-05-15T12:49:16.255 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.256 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.256 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.256 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.031955+0000", 2024-05-15T12:49:16.256 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.114779144 2024-05-15T12:49:16.256 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.256 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.256 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:49:16.256 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032020+0000", 2024-05-15T12:49:16.256 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4707000000000002e-05 2024-05-15T12:49:16.256 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.256 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.256 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:49:16.256 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032069+0000", 2024-05-15T12:49:16.256 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9069000000000001e-05 2024-05-15T12:49:16.257 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.257 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:49:16.257 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.257 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.257 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.257 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89679 2.e 2:74cc89e8:::benchmark_data_smithi177_78104_object89678:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:49:16.257 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.910531+0000", 2024-05-15T12:49:16.257 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.20183705499999999, 2024-05-15T12:49:16.257 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.121660637, 2024-05-15T12:49:16.257 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:49:16.257 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:49:16.257 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:49:16.257 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:49:16.257 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:49:16.258 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89679 2024-05-15T12:49:16.258 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.258 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:49:16.258 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.258 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:49:16.258 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910531+0000", 2024-05-15T12:49:16.258 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.258 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.258 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.258 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:49:16.258 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910531+0000", 2024-05-15T12:49:16.258 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.258 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.259 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.259 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:49:16.259 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910533+0000", 2024-05-15T12:49:16.259 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.773e-06 2024-05-15T12:49:16.259 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.259 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.259 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:49:16.259 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910580+0000", 2024-05-15T12:49:16.259 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6854000000000002e-05 2024-05-15T12:49:16.259 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.259 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.259 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:49:16.259 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910595+0000", 2024-05-15T12:49:16.259 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5082999999999999e-05 2024-05-15T12:49:16.259 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.259 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.260 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:49:16.260 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910605+0000", 2024-05-15T12:49:16.260 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6029999999999994e-06 2024-05-15T12:49:16.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.260 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.260 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:49:16.260 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.911065+0000", 2024-05-15T12:49:16.260 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00046001800000000003 2024-05-15T12:49:16.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.260 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.260 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:49:16.260 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.911211+0000", 2024-05-15T12:49:16.260 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014574400000000001 2024-05-15T12:49:16.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.260 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.261 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:49:16.261 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.911363+0000", 2024-05-15T12:49:16.261 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015289500000000001 2024-05-15T12:49:16.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.261 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.261 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:49:16.261 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.924113+0000", 2024-05-15T12:49:16.261 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012749286 2024-05-15T12:49:16.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.261 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.261 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.261 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.931398+0000", 2024-05-15T12:49:16.261 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0072853279999999998 2024-05-15T12:49:16.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.262 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.262 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.262 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032107+0000", 2024-05-15T12:49:16.262 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.100708456 2024-05-15T12:49:16.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.262 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.262 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:49:16.262 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032159+0000", 2024-05-15T12:49:16.262 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2234000000000002e-05 2024-05-15T12:49:16.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.262 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.262 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:49:16.262 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032192+0000", 2024-05-15T12:49:16.263 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3362999999999999e-05 2024-05-15T12:49:16.263 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.263 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:49:16.263 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.263 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.263 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.263 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89680 2.e 2:71e6b157:::benchmark_data_smithi177_78104_object89679:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:49:16.263 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.910622+0000", 2024-05-15T12:49:16.263 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.201746959, 2024-05-15T12:49:16.263 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12167915, 2024-05-15T12:49:16.263 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:49:16.263 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:49:16.263 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:49:16.263 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:49:16.264 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:49:16.264 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89680 2024-05-15T12:49:16.264 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.264 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:49:16.264 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.264 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:49:16.264 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910622+0000", 2024-05-15T12:49:16.264 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.264 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.264 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.264 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:49:16.264 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910622+0000", 2024-05-15T12:49:16.264 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.265 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.265 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.265 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:49:16.265 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910622+0000", 2024-05-15T12:49:16.265 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6499999999999999e-07 2024-05-15T12:49:16.265 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.265 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.265 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:49:16.265 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910662+0000", 2024-05-15T12:49:16.265 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9436e-05 2024-05-15T12:49:16.265 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.265 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.265 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:49:16.266 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910670+0000", 2024-05-15T12:49:16.266 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4940000000000003e-06 2024-05-15T12:49:16.266 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.266 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.266 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:49:16.267 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910677+0000", 2024-05-15T12:49:16.267 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6039999999999998e-06 2024-05-15T12:49:16.267 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.267 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.267 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:49:16.267 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.914062+0000", 2024-05-15T12:49:16.267 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003385117 2024-05-15T12:49:16.267 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.267 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.267 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:49:16.267 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.914206+0000", 2024-05-15T12:49:16.267 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014453299999999999 2024-05-15T12:49:16.267 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.268 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.268 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:49:16.268 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.914360+0000", 2024-05-15T12:49:16.268 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000154146 2024-05-15T12:49:16.268 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.268 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.268 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:49:16.268 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.924120+0000", 2024-05-15T12:49:16.268 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0097599209999999995 2024-05-15T12:49:16.268 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.268 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.268 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.268 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.934155+0000", 2024-05-15T12:49:16.268 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0100348 2024-05-15T12:49:16.268 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.269 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.269 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.269 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032226+0000", 2024-05-15T12:49:16.269 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.098070750999999998 2024-05-15T12:49:16.269 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.269 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.269 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:49:16.269 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032269+0000", 2024-05-15T12:49:16.269 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3513000000000002e-05 2024-05-15T12:49:16.269 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.269 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.269 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:49:16.269 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032301+0000", 2024-05-15T12:49:16.269 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1269999999999997e-05 2024-05-15T12:49:16.270 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.270 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:49:16.270 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.270 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.270 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.270 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89690 2.e 2:7ba0758f:::benchmark_data_smithi177_78104_object89689:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:49:16.270 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.928382+0000", 2024-05-15T12:49:16.270 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.18398693799999999, 2024-05-15T12:49:16.270 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10402787500000001, 2024-05-15T12:49:16.270 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:49:16.270 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:49:16.270 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:49:16.270 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:49:16.271 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:49:16.271 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89690 2024-05-15T12:49:16.271 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.271 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:49:16.271 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.271 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:49:16.271 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.928382+0000", 2024-05-15T12:49:16.271 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.271 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.271 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.271 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:49:16.271 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.928382+0000", 2024-05-15T12:49:16.271 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.272 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.272 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.272 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:49:16.272 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.928383+0000", 2024-05-15T12:49:16.272 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8610000000000001e-06 2024-05-15T12:49:16.272 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.272 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.272 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:49:16.272 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.928422+0000", 2024-05-15T12:49:16.272 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8152000000000003e-05 2024-05-15T12:49:16.272 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.272 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.272 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:49:16.273 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.928444+0000", 2024-05-15T12:49:16.273 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2830999999999999e-05 2024-05-15T12:49:16.273 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.273 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.273 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:49:16.273 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.928454+0000", 2024-05-15T12:49:16.273 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8740000000000007e-06 2024-05-15T12:49:16.273 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.273 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.273 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:49:16.273 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.931422+0000", 2024-05-15T12:49:16.273 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029681500000000001 2024-05-15T12:49:16.273 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.273 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.274 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:49:16.274 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.931545+0000", 2024-05-15T12:49:16.274 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000122574 2024-05-15T12:49:16.274 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.274 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.274 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:49:16.274 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.931686+0000", 2024-05-15T12:49:16.274 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014140100000000001 2024-05-15T12:49:16.274 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.274 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.274 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:49:16.274 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935371+0000", 2024-05-15T12:49:16.274 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0036848359999999999 2024-05-15T12:49:16.274 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.275 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.275 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.275 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.938923+0000", 2024-05-15T12:49:16.275 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035515859999999998 2024-05-15T12:49:16.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.275 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.275 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.275 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032334+0000", 2024-05-15T12:49:16.275 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.093410842999999993 2024-05-15T12:49:16.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.275 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.275 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:49:16.275 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032378+0000", 2024-05-15T12:49:16.275 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.384e-05 2024-05-15T12:49:16.276 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.276 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.276 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:49:16.276 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032409+0000", 2024-05-15T12:49:16.276 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1927000000000001e-05 2024-05-15T12:49:16.276 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.276 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:49:16.276 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.276 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.276 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.276 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89694 2.e 2:7ceb574e:::benchmark_data_smithi177_78104_object89693:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:49:16.276 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.935396+0000", 2024-05-15T12:49:16.276 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.17697279699999999, 2024-05-15T12:49:16.276 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10441977, 2024-05-15T12:49:16.277 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:49:16.277 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:49:16.277 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:49:16.277 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:49:16.277 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:49:16.277 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89694 2024-05-15T12:49:16.277 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.277 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:49:16.277 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.277 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:49:16.277 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935396+0000", 2024-05-15T12:49:16.277 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.277 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.277 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.278 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:49:16.278 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935396+0000", 2024-05-15T12:49:16.278 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.278 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.278 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.278 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:49:16.278 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935397+0000", 2024-05-15T12:49:16.278 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.353e-06 2024-05-15T12:49:16.278 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.278 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.278 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:49:16.278 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935423+0000", 2024-05-15T12:49:16.278 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6291e-05 2024-05-15T12:49:16.279 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.279 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.279 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:49:16.279 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935436+0000", 2024-05-15T12:49:16.279 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2184000000000001e-05 2024-05-15T12:49:16.279 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.279 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.279 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:49:16.279 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935445+0000", 2024-05-15T12:49:16.279 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4020000000000008e-06 2024-05-15T12:49:16.279 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.279 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.279 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:49:16.279 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935511+0000", 2024-05-15T12:49:16.280 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5832000000000003e-05 2024-05-15T12:49:16.280 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.280 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.280 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:49:16.280 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935684+0000", 2024-05-15T12:49:16.280 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017362399999999999 2024-05-15T12:49:16.280 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.280 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.280 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:49:16.280 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935863+0000", 2024-05-15T12:49:16.280 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017888299999999999 2024-05-15T12:49:16.280 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.280 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.281 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:49:16.281 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.942570+0000", 2024-05-15T12:49:16.281 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0067064869999999997 2024-05-15T12:49:16.281 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.281 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.281 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.281 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.943101+0000", 2024-05-15T12:49:16.281 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00053091900000000003 2024-05-15T12:49:16.281 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.281 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.281 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.281 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.039698+0000", 2024-05-15T12:49:16.281 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.096597075000000004 2024-05-15T12:49:16.281 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.282 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.282 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:49:16.282 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.039761+0000", 2024-05-15T12:49:16.282 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2772000000000001e-05 2024-05-15T12:49:16.282 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.282 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.282 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:49:16.282 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.039816+0000", 2024-05-15T12:49:16.282 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4948000000000003e-05 2024-05-15T12:49:16.282 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.282 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:49:16.282 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.282 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.283 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.283 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89699 2.e 2:7fa3b3a9:::benchmark_data_smithi177_78104_object89698:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:49:16.283 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.939371+0000", 2024-05-15T12:49:16.283 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.17299704399999999, 2024-05-15T12:49:16.283 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10057566799999999, 2024-05-15T12:49:16.283 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:49:16.283 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:49:16.283 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:49:16.283 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:49:16.283 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:49:16.283 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89699 2024-05-15T12:49:16.283 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.283 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:49:16.283 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.284 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:49:16.284 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939371+0000", 2024-05-15T12:49:16.284 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.284 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.284 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.284 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:49:16.284 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939371+0000", 2024-05-15T12:49:16.284 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:49:16.284 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.284 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.284 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:49:16.284 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939373+0000", 2024-05-15T12:49:16.284 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6700000000000001e-06 2024-05-15T12:49:16.284 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.285 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.285 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:49:16.285 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939415+0000", 2024-05-15T12:49:16.285 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1551000000000002e-05 2024-05-15T12:49:16.285 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.285 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.285 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:49:16.285 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939432+0000", 2024-05-15T12:49:16.285 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.768e-05 2024-05-15T12:49:16.285 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.285 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.285 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:49:16.285 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939441+0000", 2024-05-15T12:49:16.285 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7299999999999994e-06 2024-05-15T12:49:16.286 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.286 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.286 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:49:16.286 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939527+0000", 2024-05-15T12:49:16.286 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6098000000000001e-05 2024-05-15T12:49:16.286 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.286 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.286 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:49:16.286 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939679+0000", 2024-05-15T12:49:16.286 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015150299999999999 2024-05-15T12:49:16.286 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.286 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.286 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:49:16.286 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939825+0000", 2024-05-15T12:49:16.287 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000146725 2024-05-15T12:49:16.287 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.287 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.287 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:49:16.287 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.942947+0000", 2024-05-15T12:49:16.287 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031219279999999999 2024-05-15T12:49:16.287 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.287 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.287 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.287 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.943208+0000", 2024-05-15T12:49:16.287 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00026073099999999999 2024-05-15T12:49:16.287 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.287 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.287 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:49:16.288 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.039860+0000", 2024-05-15T12:49:16.288 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.096651440000000005 2024-05-15T12:49:16.288 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.288 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.288 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:49:16.288 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.039911+0000", 2024-05-15T12:49:16.288 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.13e-05 2024-05-15T12:49:16.288 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:49:16.288 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:49:16.288 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:49:16.288 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.039947+0000", 2024-05-15T12:49:16.288 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6312000000000001e-05 2024-05-15T12:49:16.288 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.288 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:49:16.289 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.289 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:49:16.289 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:49:16.289 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T12:49:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:49:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:16.389 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:49:16.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:16.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:49:16.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:16 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-a[39772]: 2024-05-15T12:49:16.316+0000 7f8daffd2700 -1 mon.a@0(leader).osd e142 definitely_dead 0 2024-05-15T12:49:16.433 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:49:16.496 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:49:16.563 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:16.778 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 66 16 90363 90347 85.5418 48.3125 0.0102672 0.0116857 2024-05-15T12:49:17.259 INFO:teuthology.orchestra.run.smithi027.stderr:marked in osd.10. 2024-05-15T12:49:17.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:17 smithi027 ceph-mon[39795]: pgmap v430: 17 pgs: 4 active+recovering+undersized+remapped, 13 active+clean; 5.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 56 MiB/s wr, 897 op/s; 18959/261984 objects misplaced (7.237%); 11 MiB/s, 181 objects/s recovering 2024-05-15T12:49:17.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:17 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4072830696' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-05-15T12:49:17.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:17 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-05-15T12:49:17.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:17 smithi120 ceph-mon[43961]: pgmap v430: 17 pgs: 4 active+recovering+undersized+remapped, 13 active+clean; 5.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 56 MiB/s wr, 897 op/s; 18959/261984 objects misplaced (7.237%); 11 MiB/s, 181 objects/s recovering 2024-05-15T12:49:17.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:17 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4072830696' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-05-15T12:49:17.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:17 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-05-15T12:49:17.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:17 smithi177 ceph-mon[44303]: pgmap v430: 17 pgs: 4 active+recovering+undersized+remapped, 13 active+clean; 5.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 56 MiB/s wr, 897 op/s; 18959/261984 objects misplaced (7.237%); 11 MiB/s, 181 objects/s recovering 2024-05-15T12:49:17.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:17 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4072830696' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-05-15T12:49:17.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:17 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-05-15T12:49:17.778 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 67 16 90861 90845 84.7297 31.125 0.010708 0.0117088 2024-05-15T12:49:18.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:18 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-05-15T12:49:18.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:18 smithi027 ceph-mon[39795]: osdmap e143: 12 total, 12 up, 11 in 2024-05-15T12:49:18.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:18 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:49:18.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:18 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:49:18.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:18 smithi027 ceph-mon[39795]: osdmap e144: 12 total, 12 up, 11 in 2024-05-15T12:49:18.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:18 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-05-15T12:49:18.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:18 smithi120 ceph-mon[43961]: osdmap e143: 12 total, 12 up, 11 in 2024-05-15T12:49:18.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:18 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:49:18.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:18 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:49:18.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:18 smithi120 ceph-mon[43961]: osdmap e144: 12 total, 12 up, 11 in 2024-05-15T12:49:18.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:18 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-05-15T12:49:18.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:18 smithi177 ceph-mon[44303]: osdmap e143: 12 total, 12 up, 11 in 2024-05-15T12:49:18.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:18 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:49:18.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:18 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:49:18.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:18 smithi177 ceph-mon[44303]: osdmap e144: 12 total, 12 up, 11 in 2024-05-15T12:49:18.778 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 68 16 90861 90845 83.4837 0 - 0.0117088 2024-05-15T12:49:19.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:19 smithi027 ceph-mon[39795]: pgmap v432: 17 pgs: 4 active+recovering+undersized+remapped, 13 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 51 MiB/s wr, 811 op/s; 18766/265254 objects misplaced (7.075%); 10 MiB/s, 162 objects/s recovering 2024-05-15T12:49:19.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:19 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:49:19.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:19 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:49:19.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:19 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:49:19.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:19 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:49:19.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:19 smithi027 ceph-mon[39795]: osdmap e145: 12 total, 12 up, 11 in 2024-05-15T12:49:19.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:19 smithi120 ceph-mon[43961]: pgmap v432: 17 pgs: 4 active+recovering+undersized+remapped, 13 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 51 MiB/s wr, 811 op/s; 18766/265254 objects misplaced (7.075%); 10 MiB/s, 162 objects/s recovering 2024-05-15T12:49:19.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:19 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:49:19.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:19 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:49:19.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:19 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:49:19.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:19 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:49:19.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:19 smithi120 ceph-mon[43961]: osdmap e145: 12 total, 12 up, 11 in 2024-05-15T12:49:19.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:19 smithi177 ceph-mon[44303]: pgmap v432: 17 pgs: 4 active+recovering+undersized+remapped, 13 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 51 MiB/s wr, 811 op/s; 18766/265254 objects misplaced (7.075%); 10 MiB/s, 162 objects/s recovering 2024-05-15T12:49:19.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:19 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:49:19.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:19 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:49:19.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:19 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:49:19.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:19 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:49:19.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:19 smithi177 ceph-mon[44303]: osdmap e145: 12 total, 12 up, 11 in 2024-05-15T12:49:19.778 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 69 16 91163 91147 82.5474 9.4375 0.0106412 0.0121056 2024-05-15T12:49:20.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:20 smithi027 ceph-mon[39795]: 2.7 starting backfill to osd.10 from (0'0,116'1848] MIN to 142'5644 2024-05-15T12:49:20.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:20 smithi027 ceph-mon[39795]: 2.0 starting backfill to osd.10 from (0'0,116'1884] MIN to 142'5728 2024-05-15T12:49:20.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:20 smithi027 ceph-mon[39795]: 2.4 starting backfill to osd.10 from (0'0,116'1855] MIN to 142'5651 2024-05-15T12:49:20.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:20 smithi120 ceph-mon[43961]: 2.7 starting backfill to osd.10 from (0'0,116'1848] MIN to 142'5644 2024-05-15T12:49:20.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:20 smithi120 ceph-mon[43961]: 2.0 starting backfill to osd.10 from (0'0,116'1884] MIN to 142'5728 2024-05-15T12:49:20.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:20 smithi120 ceph-mon[43961]: 2.4 starting backfill to osd.10 from (0'0,116'1855] MIN to 142'5651 2024-05-15T12:49:20.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:20 smithi177 ceph-mon[44303]: 2.7 starting backfill to osd.10 from (0'0,116'1848] MIN to 142'5644 2024-05-15T12:49:20.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:20 smithi177 ceph-mon[44303]: 2.0 starting backfill to osd.10 from (0'0,116'1884] MIN to 142'5728 2024-05-15T12:49:20.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:20 smithi177 ceph-mon[44303]: 2.4 starting backfill to osd.10 from (0'0,116'1855] MIN to 142'5651 2024-05-15T12:49:20.780 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 70 16 91934 91918 82.0565 48.1875 0.0131057 0.0121821 2024-05-15T12:49:21.347 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:49:21.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:21.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:49:21.400 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-05-15T12:49:21.457 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:49:21.484 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:21.507 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:49:21.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:21.517 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:49:21.518 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:21 smithi177 ceph-mon[44303]: pgmap v435: 17 pgs: 1 remapped+peering, 4 active+recovering+undersized+remapped, 12 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 44 MiB/s wr, 700 op/s; 18493/269607 objects misplaced (6.859%); 9.4 MiB/s, 149 objects/s recovering 2024-05-15T12:49:21.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:21 smithi027 ceph-mon[39795]: pgmap v435: 17 pgs: 1 remapped+peering, 4 active+recovering+undersized+remapped, 12 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 44 MiB/s wr, 700 op/s; 18493/269607 objects misplaced (6.859%); 9.4 MiB/s, 149 objects/s recovering 2024-05-15T12:49:21.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:21 smithi120 ceph-mon[43961]: pgmap v435: 17 pgs: 1 remapped+peering, 4 active+recovering+undersized+remapped, 12 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 44 MiB/s wr, 700 op/s; 18493/269607 objects misplaced (6.859%); 9.4 MiB/s, 149 objects/s recovering 2024-05-15T12:49:21.618 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:49:21.653 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:21.677 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:49:21.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:21.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:49:21.778 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 71 16 92736 92720 81.6068 50.125 0.0332526 0.012249 2024-05-15T12:49:21.782 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:49:21.823 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:21.846 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:49:21.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:21.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:49:21.956 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:49:21.997 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:22.035 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:49:22.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:22.039 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:49:22.140 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:49:22.176 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:22.204 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:49:22.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:22.211 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:49:22.312 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:49:22.349 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:22.371 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:49:22.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:22.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:49:22.476 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:49:22.503 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:22.779 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 72 16 93571 93555 81.1981 52.1875 0.0267011 0.012311 2024-05-15T12:49:23.487 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:49:23.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:23.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:49:23.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:23 smithi027 ceph-mon[39795]: pgmap v436: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 remapped+peering, 3 active+recovering+undersized+remapped, 11 active+clean; 5.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 40 MiB/s wr, 646 op/s; 23699/273615 objects misplaced (8.661%); 10 MiB/s, 161 objects/s recovering 2024-05-15T12:49:23.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:23 smithi027 ceph-mon[39795]: osdmap e146: 12 total, 12 up, 11 in 2024-05-15T12:49:23.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:23 smithi120 ceph-mon[43961]: pgmap v436: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 remapped+peering, 3 active+recovering+undersized+remapped, 11 active+clean; 5.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 40 MiB/s wr, 646 op/s; 23699/273615 objects misplaced (8.661%); 10 MiB/s, 161 objects/s recovering 2024-05-15T12:49:23.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:23 smithi120 ceph-mon[43961]: osdmap e146: 12 total, 12 up, 11 in 2024-05-15T12:49:23.596 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:49:23.660 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:23.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:23 smithi177 ceph-mon[44303]: pgmap v436: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 remapped+peering, 3 active+recovering+undersized+remapped, 11 active+clean; 5.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 40 MiB/s wr, 646 op/s; 23699/273615 objects misplaced (8.661%); 10 MiB/s, 161 objects/s recovering 2024-05-15T12:49:23.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:23 smithi177 ceph-mon[44303]: osdmap e146: 12 total, 12 up, 11 in 2024-05-15T12:49:23.778 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 73 16 94358 94342 80.7596 49.1875 0.00325332 0.0123778 2024-05-15T12:49:24.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:24 smithi177 ceph-mon[44303]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T12:49:24.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:24 smithi027 ceph-mon[39795]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T12:49:24.780 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 74 16 95209 95193 80.3855 53.1875 0.00710089 0.0124347 2024-05-15T12:49:24.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:24 smithi120 ceph-mon[43961]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T12:49:24.950 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:49:24.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:24.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:49:25.061 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:49:25.092 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:25.117 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:49:25.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:25.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:49:25.221 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:49:25.269 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:25.306 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:49:25.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:25.312 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:49:25.413 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:49:25.495 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:25.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:25 smithi120 ceph-mon[43961]: pgmap v438: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 42 MiB/s wr, 672 op/s; 28936/277368 objects misplaced (10.432%); 14 MiB/s, 227 objects/s recovering 2024-05-15T12:49:25.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:25 smithi177 ceph-mon[44303]: pgmap v438: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 42 MiB/s wr, 672 op/s; 28936/277368 objects misplaced (10.432%); 14 MiB/s, 227 objects/s recovering 2024-05-15T12:49:25.739 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:25 smithi027 ceph-mon[39795]: pgmap v438: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 42 MiB/s wr, 672 op/s; 28936/277368 objects misplaced (10.432%); 14 MiB/s, 227 objects/s recovering 2024-05-15T12:49:25.780 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 75 16 95985 95969 79.9604 48.5 0.0278333 0.0124983 2024-05-15T12:49:25.792 INFO:tasks.thrashosds.thrasher:Added osd 10 2024-05-15T12:49:26.168 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:49:26.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:26.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:49:26.284 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:49:26.352 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:26.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:26 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:49:26.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:26 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:49:26.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:26 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:49:26.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:26 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:49:26.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:26 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:49:26.576 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:49:26.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:26.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:49:26.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:26 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:49:26.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:26 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:49:26.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:26 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:49:26.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:26 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:49:26.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:26 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:49:26.682 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:49:26.719 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:26.745 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:49:26.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:26.749 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:49:26.780 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 76 16 96757 96741 79.5431 48.25 0.00705264 0.0125663 2024-05-15T12:49:26.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:26 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:49:26.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:26 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:49:26.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:26 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:49:26.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:26 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:49:26.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:26 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:49:26.849 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:49:26.918 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:27.292 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:49:27.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:27.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:49:27.398 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:49:27.453 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:27.540 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:49:27.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:27.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:49:27.555 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:27 smithi027 ceph-mon[39795]: pgmap v439: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 37 MiB/s wr, 588 op/s; 28505/281952 objects misplaced (10.110%); 15 MiB/s, 233 objects/s recovering 2024-05-15T12:49:27.555 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:27 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:49:27.555 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:27 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:49:27.555 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:27 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:49:27.555 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:27 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:49:27.650 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:49:27.678 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:27 smithi120 ceph-mon[43961]: pgmap v439: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 37 MiB/s wr, 588 op/s; 28505/281952 objects misplaced (10.110%); 15 MiB/s, 233 objects/s recovering 2024-05-15T12:49:27.678 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:27 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:49:27.678 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:27 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:49:27.678 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:27 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:49:27.678 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:27 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:49:27.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:27 smithi177 ceph-mon[44303]: pgmap v439: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 37 MiB/s wr, 588 op/s; 28505/281952 objects misplaced (10.110%); 15 MiB/s, 233 objects/s recovering 2024-05-15T12:49:27.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:27 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:49:27.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:27 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:49:27.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:27 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:49:27.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:27 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:49:27.680 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:27.704 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:49:27.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:27.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:49:27.780 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 77 15 97508 97493 79.1205 47 0.0340993 0.0126342 2024-05-15T12:49:27.807 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:49:27.829 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:49:27.845 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:27.879 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:49:27.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:27.883 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:49:27.894 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:49:27.930 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:49:27.952 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:49:27.984 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:49:28.012 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:49:28.016 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:28.042 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:49:28.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:28.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:49:28.055 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:49:28.109 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:49:28.146 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:49:28.180 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:28.204 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:49:28.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:28.213 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:49:28.314 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:49:28.353 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:49:28.389 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:49:28.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:28 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/906834184' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:49:28.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:28 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/617226505' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:49:28.432 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:49:28.449 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:28.478 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:49:28.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:28.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:49:28.510 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:49:28.553 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:49:28.584 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:49:28.597 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:49:28.618 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:49:28.618 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:49:28.618 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:49:28.618 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:49:28.618 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:49:28.618 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:49:28.618 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:49:28.618 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:49:28.618 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:49:28.618 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:49:28.619 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:49:28.619 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:49:28.644 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:28 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/906834184' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:49:28.644 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:28 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/617226505' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:49:28.670 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:28.780 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 78 16 98251 98235 78.7007 46.375 0.0222185 0.012701 2024-05-15T12:49:28.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:28 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/906834184' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:49:28.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:28 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/617226505' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:49:28.891 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T12:49:29.264 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:49:29.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:29.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:49:29.372 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:49:29.416 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:29.454 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:49:29.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:29.459 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:49:29.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:29 smithi027 ceph-mon[39795]: pgmap v440: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 37 MiB/s wr, 585 op/s; 28199/284856 objects misplaced (9.899%); 15 MiB/s, 236 objects/s recovering 2024-05-15T12:49:29.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:29 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:49:29.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:29 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/617226505' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:49:29.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:29 smithi027 ceph-mon[39795]: osdmap e147: 12 total, 12 up, 11 in 2024-05-15T12:49:29.560 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:49:29.631 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:29.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:29 smithi177 ceph-mon[44303]: pgmap v440: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 37 MiB/s wr, 585 op/s; 28199/284856 objects misplaced (9.899%); 15 MiB/s, 236 objects/s recovering 2024-05-15T12:49:29.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:29 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:49:29.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:29 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/617226505' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:49:29.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:29 smithi177 ceph-mon[44303]: osdmap e147: 12 total, 12 up, 11 in 2024-05-15T12:49:29.780 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 79 16 99062 99046 78.346 50.6875 0.0161205 0.0127592 2024-05-15T12:49:29.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:29 smithi120 ceph-mon[43961]: pgmap v440: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 37 MiB/s wr, 585 op/s; 28199/284856 objects misplaced (9.899%); 15 MiB/s, 236 objects/s recovering 2024-05-15T12:49:29.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:29 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:49:29.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:29 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/617226505' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:49:29.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:29 smithi120 ceph-mon[43961]: osdmap e147: 12 total, 12 up, 11 in 2024-05-15T12:49:30.701 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:49:30.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:30.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:49:30.780 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T12:49:30.791729+0000 min lat: 0.00104218 max lat: 7.07499 avg lat: 0.0128082 2024-05-15T12:49:30.780 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T12:49:30.780 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 80 16 99930 99914 78.0448 54.25 0.00669337 0.0128082 2024-05-15T12:49:30.793 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] out_osds: [11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T12:49:30.793 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T12:49:30.793 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T12:49:30.807 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:49:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:31.299 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:49:31.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:31.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:49:31.410 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:49:31.411 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:49:31.490 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:31.769 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:49:31.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:31 smithi027 ceph-mon[39795]: pgmap v442: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 41 MiB/s wr, 654 op/s; 28213/289341 objects misplaced (9.751%); 14 MiB/s, 220 objects/s recovering 2024-05-15T12:49:31.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:31.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:49:31.781 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 81 16 100646 100630 77.6337 44.75 0.016944 0.0128764 2024-05-15T12:49:31.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:31 smithi120 ceph-mon[43961]: pgmap v442: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 41 MiB/s wr, 654 op/s; 28213/289341 objects misplaced (9.751%); 14 MiB/s, 220 objects/s recovering 2024-05-15T12:49:31.878 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:49:31.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:31 smithi177 ceph-mon[44303]: pgmap v442: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 41 MiB/s wr, 654 op/s; 28213/289341 objects misplaced (9.751%); 14 MiB/s, 220 objects/s recovering 2024-05-15T12:49:31.937 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:32.075 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:49:32.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:32.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:49:32.153 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T12:49:32.194 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:49:32.227 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:32.250 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:49:32.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:32.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:49:32.290 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T12:49:32.290 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:49:32.356 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:49:32.393 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:32.419 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:49:32.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:32.424 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:49:32.525 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:49:32.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:32 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:49:32.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:32 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:49:32.568 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:32.597 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:49:32.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:32.602 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:49:32.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:32 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:49:32.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:32 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:49:32.702 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:49:32.712 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:49:32.741 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:32.771 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:49:32.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:32.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:49:32.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:32 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:49:32.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:32 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:49:32.780 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 82 16 101354 101338 77.2266 44.25 0.00951513 0.0129439 2024-05-15T12:49:32.877 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:49:32.911 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:32.934 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:49:32.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:32.939 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:49:33.040 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:49:33.067 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:33.092 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:49:33.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:33.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:49:33.196 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:49:33.225 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:33.246 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:49:33.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:33.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:49:33.351 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:49:33.384 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:33.411 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:49:33.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:33.420 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:49:33.521 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:49:33.578 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:33.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:33 smithi177 ceph-mon[44303]: pgmap v443: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.0 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 47 MiB/s wr, 758 op/s; 27692/295851 objects misplaced (9.360%); 16 MiB/s, 255 objects/s recovering 2024-05-15T12:49:33.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:33 smithi177 ceph-mon[44303]: osdmap e148: 12 total, 12 up, 11 in 2024-05-15T12:49:33.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:33 smithi177 ceph-mon[44303]: 2.1 scrub starts 2024-05-15T12:49:33.722 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:49:33.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:33.729 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:33 smithi027 ceph-mon[39795]: pgmap v443: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.0 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 47 MiB/s wr, 758 op/s; 27692/295851 objects misplaced (9.360%); 16 MiB/s, 255 objects/s recovering 2024-05-15T12:49:33.729 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:33 smithi027 ceph-mon[39795]: osdmap e148: 12 total, 12 up, 11 in 2024-05-15T12:49:33.729 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:33 smithi027 ceph-mon[39795]: 2.1 scrub starts 2024-05-15T12:49:33.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:49:33.770 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-05-15T12:49:33.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:33 smithi120 ceph-mon[43961]: pgmap v443: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.0 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 47 MiB/s wr, 758 op/s; 27692/295851 objects misplaced (9.360%); 16 MiB/s, 255 objects/s recovering 2024-05-15T12:49:33.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:33 smithi120 ceph-mon[43961]: osdmap e148: 12 total, 12 up, 11 in 2024-05-15T12:49:33.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:33 smithi120 ceph-mon[43961]: 2.1 scrub starts 2024-05-15T12:49:33.781 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 83 16 102009 101993 76.7894 40.9375 0.0154533 0.0130168 2024-05-15T12:49:33.831 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:49:33.858 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:33.881 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:49:33.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:33.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:49:33.987 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:49:34.016 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:34.041 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:49:34.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:34.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:49:34.146 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:49:34.174 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:34.198 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:49:34.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:34.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:49:34.302 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:49:34.334 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:34.354 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:49:34.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:34.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:49:34.372 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:49:34.460 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:49:34.536 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:34.658 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:49:34.660 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:49:34.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:34.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:49:34.682 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:34 smithi027 ceph-mon[39795]: pgmap v445: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 49 MiB/s wr, 784 op/s; 27506/300813 objects misplaced (9.144%); 15 MiB/s, 241 objects/s recovering 2024-05-15T12:49:34.771 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:49:34.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:34 smithi120 ceph-mon[43961]: pgmap v445: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 49 MiB/s wr, 784 op/s; 27506/300813 objects misplaced (9.144%); 15 MiB/s, 241 objects/s recovering 2024-05-15T12:49:34.781 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 84 16 102670 102654 76.3671 41.3125 0.0125204 0.0130896 2024-05-15T12:49:34.799 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:34 smithi177 ceph-mon[44303]: pgmap v445: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 49 MiB/s wr, 784 op/s; 27506/300813 objects misplaced (9.144%); 15 MiB/s, 241 objects/s recovering 2024-05-15T12:49:34.802 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:34.829 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:49:34.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:34.833 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:49:34.935 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:49:34.969 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:34.996 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:49:34.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:34.999 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:49:35.100 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:49:35.127 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:35.150 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:49:35.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:35.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:49:35.257 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:49:35.338 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:35.411 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:49:35.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:35.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:49:35.525 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:49:35.596 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:49:35.609 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:35.610 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:35 smithi027 ceph-mon[39795]: 2.1 scrub ok 2024-05-15T12:49:35.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:35 smithi120 ceph-mon[43961]: 2.1 scrub ok 2024-05-15T12:49:35.781 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 85 16 103402 103386 76.0068 45.75 0.00872983 0.0131518 2024-05-15T12:49:35.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:35 smithi177 ceph-mon[44303]: 2.1 scrub ok 2024-05-15T12:49:36.497 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:49:36.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:36.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:49:36.602 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:49:36.632 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:36.654 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:49:36.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:36.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:49:36.758 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:49:36.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:36 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:49:36.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:36 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:49:36.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:36 smithi027 ceph-mon[39795]: osdmap e149: 12 total, 12 up, 11 in 2024-05-15T12:49:36.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:36 smithi027 ceph-mon[39795]: pgmap v447: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 48 MiB/s wr, 773 op/s; 27015/304845 objects misplaced (8.862%); 16 MiB/s, 251 objects/s recovering 2024-05-15T12:49:36.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:36 smithi027 ceph-mon[39795]: 2.e scrub starts 2024-05-15T12:49:36.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:36 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:49:36.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:36 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:49:36.781 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 86 16 104117 104101 75.6427 44.6875 0.0108973 0.0132152 2024-05-15T12:49:36.791 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:36.816 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:49:36.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:36 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:49:36.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:36 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:49:36.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:36 smithi177 ceph-mon[44303]: osdmap e149: 12 total, 12 up, 11 in 2024-05-15T12:49:36.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:36 smithi177 ceph-mon[44303]: pgmap v447: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 48 MiB/s wr, 773 op/s; 27015/304845 objects misplaced (8.862%); 16 MiB/s, 251 objects/s recovering 2024-05-15T12:49:36.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:36 smithi177 ceph-mon[44303]: 2.e scrub starts 2024-05-15T12:49:36.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:36 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:49:36.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:36 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:49:36.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:36.820 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:49:36.921 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:49:36.955 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:36.980 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:49:36.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:36.983 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:49:37.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:36 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:49:37.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:36 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:49:37.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:36 smithi120 ceph-mon[43961]: osdmap e149: 12 total, 12 up, 11 in 2024-05-15T12:49:37.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:36 smithi120 ceph-mon[43961]: pgmap v447: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 48 MiB/s wr, 773 op/s; 27015/304845 objects misplaced (8.862%); 16 MiB/s, 251 objects/s recovering 2024-05-15T12:49:37.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:36 smithi120 ceph-mon[43961]: 2.e scrub starts 2024-05-15T12:49:37.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:36 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:49:37.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:36 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:49:37.084 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:49:37.114 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:37.134 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:49:37.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:37.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:49:37.238 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:49:37.266 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:37.288 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:49:37.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:37.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:49:37.394 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:49:37.519 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:37.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:37 smithi027 ceph-mon[39795]: osdmap e150: 12 total, 12 up, 11 in 2024-05-15T12:49:37.781 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 87 16 104765 104749 75.2387 40.5 0.0192701 0.0132851 2024-05-15T12:49:37.798 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:49:37.823 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T12:49:37.893 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:49:37.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:37.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:49:37.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:37 smithi177 ceph-mon[44303]: osdmap e150: 12 total, 12 up, 11 in 2024-05-15T12:49:38.007 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:49:38.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:37 smithi120 ceph-mon[43961]: osdmap e150: 12 total, 12 up, 11 in 2024-05-15T12:49:38.037 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:38.059 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:49:38.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:38.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:49:38.162 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:49:38.199 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:38.225 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:49:38.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:38.231 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:49:38.332 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:49:38.362 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:38.385 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:49:38.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:38.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:49:38.488 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:49:38.554 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:38.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:38 smithi027 ceph-mon[39795]: pgmap v449: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.3 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 40 MiB/s wr, 644 op/s; 26687/307467 objects misplaced (8.680%); 15 MiB/s, 239 objects/s recovering 2024-05-15T12:49:38.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:38 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:49:38.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:38 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:49:38.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:38 smithi027 ceph-mon[39795]: from='client.24622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:49:38.781 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 88 16 105492 105476 74.9 45.4375 0.0202342 0.0133457 2024-05-15T12:49:38.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:38 smithi177 ceph-mon[44303]: pgmap v449: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.3 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 40 MiB/s wr, 644 op/s; 26687/307467 objects misplaced (8.680%); 15 MiB/s, 239 objects/s recovering 2024-05-15T12:49:38.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:38 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:49:38.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:38 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:49:38.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:38 smithi177 ceph-mon[44303]: from='client.24622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:49:39.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:38 smithi120 ceph-mon[43961]: pgmap v449: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.3 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 40 MiB/s wr, 644 op/s; 26687/307467 objects misplaced (8.680%); 15 MiB/s, 239 objects/s recovering 2024-05-15T12:49:39.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:38 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:49:39.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:38 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:49:39.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:38 smithi120 ceph-mon[43961]: from='client.24622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:49:39.745 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:39 smithi027 ceph-mon[39795]: 2.e scrub ok 2024-05-15T12:49:39.746 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:39 smithi027 ceph-mon[39795]: osdmap e151: 12 total, 12 up, 11 in 2024-05-15T12:49:39.781 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 89 16 106338 106322 74.6525 52.875 0.0267013 0.0133908 2024-05-15T12:49:39.790 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":449,"stamp":"2024-05-15T12:49:37.555993+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6717178400,"num_objects":102489,"num_object_clones":0,"num_object_copies":307467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":26687,"num_objects_unfound":0,"num_objects_dirty":102489,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":102668,"num_write_kb":6562106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11777,"num_bytes_recovered":774204608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102626,"ondisk_log_size":102626,"up":51,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":19644128,"kb_used_data":18962924,"kb_used_omap":17,"kb_used_meta":680174,"kb_avail":1011552544,"statfs":{"total":1055945392128,"available":1035829805056,"internally_reserved":0,"allocated":19418034176,"data_stored":19414660120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":696499061},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":253755392,"num_objects":3872,"num_object_clones":0,"num_object_copies":11616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1005,"num_objects_unfound":0,"num_objects_dirty":3872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3872,"num_write_kb":247808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1435,"num_bytes_recovered":94044160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.003989"},"pg_stats":[{"pgid":"2.f","version":"148'6407","reported_seq":6493,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-05-15T12:49:35.060331+0000","last_change":"2024-05-15T12:47:58.402003+0000","last_active":"2024-05-15T12:49:35.060331+0000","last_peered":"2024-05-15T12:49:35.060331+0000","last_clean":"2024-05-15T12:49:35.060331+0000","last_became_active":"2024-05-15T12:47:58.401685+0000","last_became_peered":"2024-05-15T12:47:58.401685+0000","last_unstale":"2024-05-15T12:49:35.060331+0000","last_undegraded":"2024-05-15T12:49:35.060331+0000","last_fullsized":"2024-05-15T12:49:35.060331+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:47:57.380220+0000","objects_scrubbed":0,"log_size":6407,"log_dups_size":0,"ondisk_log_size":6407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419889152,"num_objects":6407,"num_object_clones":0,"num_object_copies":19221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6407,"num_write_kb":410048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"149'6410","reported_seq":12168,"reported_epoch":149,"state":"active+remapped+backfilling","last_fresh":"2024-05-15T12:49:35.634751+0000","last_change":"2024-05-15T12:49:19.400155+0000","last_active":"2024-05-15T12:49:35.634751+0000","last_peered":"2024-05-15T12:49:35.634751+0000","last_clean":"2024-05-15T12:49:17.180821+0000","last_became_active":"2024-05-15T12:49:19.375415+0000","last_became_peered":"2024-05-15T12:49:19.375415+0000","last_unstale":"2024-05-15T12:49:35.634751+0000","last_undegraded":"2024-05-15T12:49:35.634751+0000","last_fullsized":"2024-05-15T12:49:35.634751+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:47:57.380220+0000","objects_scrubbed":0,"log_size":6410,"log_dups_size":0,"ondisk_log_size":6410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420085760,"num_objects":6410,"num_object_clones":0,"num_object_copies":19230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5382,"num_objects_unfound":0,"num_objects_dirty":6410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6410,"num_write_kb":410240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2828,"num_bytes_recovered":185335808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,2],"avail_no_missing":["1","0","2"],"object_location_counts":[{"shards":"0,1,2","objects":6410}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"148'6251","reported_seq":6336,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-05-15T12:49:32.747645+0000","last_change":"2024-05-15T12:47:58.401301+0000","last_active":"2024-05-15T12:49:32.747645+0000","last_peered":"2024-05-15T12:49:32.747645+0000","last_clean":"2024-05-15T12:49:32.747645+0000","last_became_active":"2024-05-15T12:47:58.401043+0000","last_became_peered":"2024-05-15T12:47:58.401043+0000","last_unstale":"2024-05-15T12:49:32.747645+0000","last_undegraded":"2024-05-15T12:49:32.747645+0000","last_fullsized":"2024-05-15T12:49:32.747645+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:47:57.380220+0000","objects_scrubbed":0,"log_size":6251,"log_dups_size":0,"ondisk_log_size":6251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409665536,"num_objects":6251,"num_object_clones":0,"num_object_copies":18753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6251,"num_write_kb":400064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"148'6331","reported_seq":6416,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-05-15T12:49:32.560849+0000","last_change":"2024-05-15T12:47:59.089978+0000","last_active":"2024-05-15T12:49:32.560849+0000","last_peered":"2024-05-15T12:49:32.560849+0000","last_clean":"2024-05-15T12:49:32.560849+0000","last_became_active":"2024-05-15T12:47:59.089659+0000","last_became_peered":"2024-05-15T12:47:59.089659+0000","last_unstale":"2024-05-15T12:49:32.560849+0000","last_undegraded":"2024-05-15T12:49:32.560849+0000","last_fullsized":"2024-05-15T12:49:32.560849+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:47:57.380220+0000","objects_scrubbed":0,"log_size":6331,"log_dups_size":0,"ondisk_log_size":6331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414908416,"num_objects":6331,"num_object_clones":0,"num_object_copies":18993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6331,"num_write_kb":405184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"148'6287","reported_seq":6372,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-05-15T12:49:32.753168+0000","last_change":"2024-05-15T12:47:58.401363+0000","last_active":"2024-05-15T12:49:32.753168+0000","last_peered":"2024-05-15T12:49:32.753168+0000","last_clean":"2024-05-15T12:49:32.753168+0000","last_became_active":"2024-05-15T12:47:58.401049+0000","last_became_peered":"2024-05-15T12:47:58.401049+0000","last_unstale":"2024-05-15T12:49:32.753168+0000","last_undegraded":"2024-05-15T12:49:32.753168+0000","last_fullsized":"2024-05-15T12:49:32.753168+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:47:57.380220+0000","objects_scrubbed":0,"log_size":6287,"log_dups_size":0,"ondisk_log_size":6287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412024832,"num_objects":6287,"num_object_clones":0,"num_object_copies":18861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6287,"num_write_kb":402368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":485,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-05-15T12:49:32.471270+0000","last_change":"2024-05-15T12:49:08.168761+0000","last_active":"2024-05-15T12:49:32.471270+0000","last_peered":"2024-05-15T12:49:32.471270+0000","last_clean":"2024-05-15T12:49:32.471270+0000","last_became_active":"2024-05-15T12:49:08.168334+0000","last_became_peered":"2024-05-15T12:49:08.168334+0000","last_unstale":"2024-05-15T12:49:32.471270+0000","last_undegraded":"2024-05-15T12:49:32.471270+0000","last_fullsized":"2024-05-15T12:49:32.471270+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T12:48:28.015538+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T12:48:28.015538+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:49:53.429425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"148'6509","reported_seq":12367,"reported_epoch":148,"state":"active+recovering+remapped","last_fresh":"2024-05-15T12:49:35.431633+0000","last_change":"2024-05-15T12:49:19.399185+0000","last_active":"2024-05-15T12:49:35.431633+0000","last_peered":"2024-05-15T12:49:35.431633+0000","last_clean":"2024-05-15T12:49:04.115103+0000","last_became_active":"2024-05-15T12:49:19.369517+0000","last_became_peered":"2024-05-15T12:49:19.369517+0000","last_unstale":"2024-05-15T12:49:35.431633+0000","last_undegraded":"2024-05-15T12:49:35.431633+0000","last_fullsized":"2024-05-15T12:49:35.431633+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:47:57.380220+0000","objects_scrubbed":0,"log_size":6509,"log_dups_size":0,"ondisk_log_size":6509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426573824,"num_objects":6509,"num_object_clones":0,"num_object_copies":19527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4129,"num_objects_unfound":0,"num_objects_dirty":6509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6509,"num_write_kb":416576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2877,"num_bytes_recovered":188547072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,9,3],"avail_no_missing":["7","9","11"],"object_location_counts":[{"shards":"3,7,9","objects":2380},{"shards":"7,9,11","objects":4129}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"148'6441","reported_seq":6527,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-05-15T12:49:35.061218+0000","last_change":"2024-05-15T12:47:59.090166+0000","last_active":"2024-05-15T12:49:35.061218+0000","last_peered":"2024-05-15T12:49:35.061218+0000","last_clean":"2024-05-15T12:49:35.061218+0000","last_became_active":"2024-05-15T12:47:59.089807+0000","last_became_peered":"2024-05-15T12:47:59.089807+0000","last_unstale":"2024-05-15T12:49:35.061218+0000","last_undegraded":"2024-05-15T12:49:35.061218+0000","last_fullsized":"2024-05-15T12:49:35.061218+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:47:57.380220+0000","objects_scrubbed":0,"log_size":6441,"log_dups_size":0,"ondisk_log_size":6441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422117376,"num_objects":6441,"num_object_clones":0,"num_object_copies":19323,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6441,"num_write_kb":412224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"149'6431","reported_seq":6519,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-15T12:49:35.632725+0000","last_change":"2024-05-15T12:47:58.444913+0000","last_active":"2024-05-15T12:49:35.632725+0000","last_peered":"2024-05-15T12:49:35.632725+0000","last_clean":"2024-05-15T12:49:35.632725+0000","last_became_active":"2024-05-15T12:47:58.444441+0000","last_became_peered":"2024-05-15T12:47:58.444441+0000","last_unstale":"2024-05-15T12:49:35.632725+0000","last_undegraded":"2024-05-15T12:49:35.632725+0000","last_fullsized":"2024-05-15T12:49:35.632725+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:47:57.380220+0000","objects_scrubbed":0,"log_size":6431,"log_dups_size":0,"ondisk_log_size":6431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421462016,"num_objects":6431,"num_object_clones":0,"num_object_copies":19293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6431,"num_write_kb":411584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"148'6302","reported_seq":10099,"reported_epoch":148,"state":"active+remapped+backfill_toofull","last_fresh":"2024-05-15T12:49:33.196187+0000","last_change":"2024-05-15T12:49:19.390372+0000","last_active":"2024-05-15T12:49:33.196187+0000","last_peered":"2024-05-15T12:49:33.196187+0000","last_clean":"2024-05-15T12:49:17.241459+0000","last_became_active":"2024-05-15T12:49:19.375525+0000","last_became_peered":"2024-05-15T12:49:19.375525+0000","last_unstale":"2024-05-15T12:49:33.196187+0000","last_undegraded":"2024-05-15T12:49:33.196187+0000","last_fullsized":"2024-05-15T12:49:33.196187+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:47:57.380220+0000","objects_scrubbed":0,"log_size":6302,"log_dups_size":0,"ondisk_log_size":6302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413007872,"num_objects":6302,"num_object_clones":0,"num_object_copies":18906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6302,"num_objects_unfound":0,"num_objects_dirty":6302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6302,"num_write_kb":403328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1848,"num_bytes_recovered":121110528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,2,3],"avail_no_missing":["6","2","3"],"object_location_counts":[{"shards":"2,3,6","objects":6302}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"148'6445","reported_seq":6530,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-05-15T12:49:35.401888+0000","last_change":"2024-05-15T12:47:59.089926+0000","last_active":"2024-05-15T12:49:35.401888+0000","last_peered":"2024-05-15T12:49:35.401888+0000","last_clean":"2024-05-15T12:49:35.401888+0000","last_became_active":"2024-05-15T12:47:59.089595+0000","last_became_peered":"2024-05-15T12:47:59.089595+0000","last_unstale":"2024-05-15T12:49:35.401888+0000","last_undegraded":"2024-05-15T12:49:35.401888+0000","last_fullsized":"2024-05-15T12:49:35.401888+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:47:57.380220+0000","objects_scrubbed":0,"log_size":6445,"log_dups_size":0,"ondisk_log_size":6445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422379520,"num_objects":6445,"num_object_clones":0,"num_object_copies":19335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6445,"num_write_kb":412480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"149'6451","reported_seq":6539,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-15T12:49:35.562053+0000","last_change":"2024-05-15T12:47:58.445838+0000","last_active":"2024-05-15T12:49:35.562053+0000","last_peered":"2024-05-15T12:49:35.562053+0000","last_clean":"2024-05-15T12:49:35.562053+0000","last_became_active":"2024-05-15T12:47:58.445362+0000","last_became_peered":"2024-05-15T12:47:58.445362+0000","last_unstale":"2024-05-15T12:49:35.562053+0000","last_undegraded":"2024-05-15T12:49:35.562053+0000","last_fullsized":"2024-05-15T12:49:35.562053+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:47:57.380220+0000","objects_scrubbed":0,"log_size":6451,"log_dups_size":0,"ondisk_log_size":6451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422772736,"num_objects":6451,"num_object_clones":0,"num_object_copies":19353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6451,"num_write_kb":412864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"150'6488","reported_seq":9295,"reported_epoch":150,"state":"active+recovering+undersized+remapped","last_fresh":"2024-05-15T12:49:36.720755+0000","last_change":"2024-05-15T12:49:06.337580+0000","last_active":"2024-05-15T12:49:36.720755+0000","last_peered":"2024-05-15T12:49:36.720755+0000","last_clean":"2024-05-15T12:49:04.090764+0000","last_became_active":"2024-05-15T12:49:06.283897+0000","last_became_peered":"2024-05-15T12:49:06.283897+0000","last_unstale":"2024-05-15T12:49:36.720755+0000","last_undegraded":"2024-05-15T12:49:36.720755+0000","last_fullsized":"2024-05-15T12:49:06.132295+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:47:57.380220+0000","objects_scrubbed":0,"log_size":6488,"log_dups_size":0,"ondisk_log_size":6488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425197568,"num_objects":6488,"num_object_clones":0,"num_object_copies":19464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3715,"num_objects_unfound":0,"num_objects_dirty":6488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6488,"num_write_kb":415232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1357,"num_bytes_recovered":88932352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,8],"acting":[3,9],"avail_no_missing":["3","9","11"],"object_location_counts":[{"shards":"3,9","objects":2773},{"shards":"3,9,11","objects":3715}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"148'6450","reported_seq":9676,"reported_epoch":148,"state":"active+recovering+undersized+remapped","last_fresh":"2024-05-15T12:49:35.073262+0000","last_change":"2024-05-15T12:49:06.411720+0000","last_active":"2024-05-15T12:49:35.073262+0000","last_peered":"2024-05-15T12:49:35.073262+0000","last_clean":"2024-05-15T12:49:04.115809+0000","last_became_active":"2024-05-15T12:49:06.376843+0000","last_became_peered":"2024-05-15T12:49:06.376843+0000","last_unstale":"2024-05-15T12:49:35.073262+0000","last_undegraded":"2024-05-15T12:49:35.073262+0000","last_fullsized":"2024-05-15T12:49:06.131044+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:47:57.380220+0000","objects_scrubbed":0,"log_size":6450,"log_dups_size":0,"ondisk_log_size":6450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422707200,"num_objects":6450,"num_object_clones":0,"num_object_copies":19350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":6450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6450,"num_write_kb":412800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1568,"num_bytes_recovered":102760448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,5],"acting":[8,7],"avail_no_missing":["8","7","11"],"object_location_counts":[{"shards":"7,8","objects":2940},{"shards":"7,8,11","objects":3510}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"148'6287","reported_seq":8949,"reported_epoch":148,"state":"active+recovering+undersized+remapped","last_fresh":"2024-05-15T12:49:33.337966+0000","last_change":"2024-05-15T12:49:06.323824+0000","last_active":"2024-05-15T12:49:33.337966+0000","last_peered":"2024-05-15T12:49:33.337966+0000","last_clean":"2024-05-15T12:49:04.107375+0000","last_became_active":"2024-05-15T12:49:06.278186+0000","last_became_peered":"2024-05-15T12:49:06.278186+0000","last_unstale":"2024-05-15T12:49:33.337966+0000","last_undegraded":"2024-05-15T12:49:33.337966+0000","last_fullsized":"2024-05-15T12:49:06.132270+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:47:57.380220+0000","objects_scrubbed":0,"log_size":6287,"log_dups_size":0,"ondisk_log_size":6287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412024832,"num_objects":6287,"num_object_clones":0,"num_object_copies":18861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3649,"num_objects_unfound":0,"num_objects_dirty":6287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6287,"num_write_kb":402368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1287,"num_bytes_recovered":84344832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,8],"acting":[0,1],"avail_no_missing":["0","1","11"],"object_location_counts":[{"shards":"0,1","objects":2638},{"shards":"0,1,11","objects":3649}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"148'6476","reported_seq":6562,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-05-15T12:49:35.160026+0000","last_change":"2024-05-15T12:47:58.445060+0000","last_active":"2024-05-15T12:49:35.160026+0000","last_peered":"2024-05-15T12:49:35.160026+0000","last_clean":"2024-05-15T12:49:35.160026+0000","last_became_active":"2024-05-15T12:47:58.444625+0000","last_became_peered":"2024-05-15T12:47:58.444625+0000","last_unstale":"2024-05-15T12:49:35.160026+0000","last_undegraded":"2024-05-15T12:49:35.160026+0000","last_fullsized":"2024-05-15T12:49:35.160026+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:47:57.380220+0000","objects_scrubbed":0,"log_size":6476,"log_dups_size":0,"ondisk_log_size":6476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424411136,"num_objects":6476,"num_object_clones":0,"num_object_copies":19428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6476,"num_write_kb":414464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"148'6521","reported_seq":6606,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-05-15T12:49:32.534127+0000","last_change":"2024-05-15T12:47:58.444842+0000","last_active":"2024-05-15T12:49:32.534127+0000","last_peered":"2024-05-15T12:49:32.534127+0000","last_clean":"2024-05-15T12:49:32.534127+0000","last_became_active":"2024-05-15T12:47:58.444477+0000","last_became_peered":"2024-05-15T12:47:58.444477+0000","last_unstale":"2024-05-15T12:49:32.534127+0000","last_undegraded":"2024-05-15T12:49:32.534127+0000","last_fullsized":"2024-05-15T12:49:32.534127+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:47:57.380220+0000","objects_scrubbed":0,"log_size":6521,"log_dups_size":0,"ondisk_log_size":6521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427360256,"num_objects":6521,"num_object_clones":0,"num_object_copies":19563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6521,"num_write_kb":417344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":6716588032,"num_objects":102487,"num_object_clones":0,"num_object_copies":307461,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":26687,"num_objects_unfound":0,"num_objects_dirty":102487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102487,"num_write_kb":6559168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11765,"num_bytes_recovered":771031040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20787625984,"data_stored":20787625984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102487,"ondisk_log_size":102487,"up":48,"acting":45,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":0,"seq":386547056689,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351030,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":287504,"kb_used_data":256328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93457648,"statfs":{"total":95995035648,"available":95700631552,"internally_reserved":0,"allocated":262479872,"data_stored":262177090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514555,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2547416,"kb_used_data":2475984,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91197736,"statfs":{"total":95995035648,"available":93386481664,"internally_reserved":0,"allocated":2535407616,"data_stored":2535101282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":73136576},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874305,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2053576,"kb_used_data":1990280,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91691576,"statfs":{"total":95995035648,"available":93892173824,"internally_reserved":0,"allocated":2038046720,"data_stored":2037743958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168643,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2135448,"kb_used_data":2059856,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91609704,"statfs":{"total":95995035648,"available":93808336896,"internally_reserved":0,"allocated":2109292544,"data_stored":2108986230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430281,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":829456,"kb_used_data":802256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":92915696,"statfs":{"total":95995035648,"available":95145672704,"internally_reserved":0,"allocated":821510144,"data_stored":821203830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574949,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2110928,"kb_used_data":2047816,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91634224,"statfs":{"total":95995035648,"available":93833445376,"internally_reserved":0,"allocated":2096963584,"data_stored":2096660822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738476,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2092808,"kb_used_data":2025352,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91652344,"statfs":{"total":95995035648,"available":93852000256,"internally_reserved":0,"allocated":2073960448,"data_stored":2073657686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411425,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2082552,"kb_used_data":2019440,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91662600,"statfs":{"total":95995035648,"available":93862502400,"internally_reserved":0,"allocated":2067906560,"data_stored":2067526780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149786,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3190440,"kb_used_data":3082908,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":90554712,"statfs":{"total":95995035648,"available":92728025088,"internally_reserved":0,"allocated":3156897792,"data_stored":3156639070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":110098887},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1287304,"kb_used_data":1229576,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92457848,"statfs":{"total":95995035648,"available":94676836352,"internally_reserved":0,"allocated":1259085824,"data_stored":1258783062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691919,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1026696,"kb_used_data":973128,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92718456,"statfs":{"total":95995035648,"available":94943698944,"internally_reserved":0,"allocated":996483072,"data_stored":996180310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2076901376,"data_stored":2076901376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2100232192,"data_stored":2100232192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2076180480,"data_stored":2076180480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3159359488,"data_stored":3159359488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1259864064,"data_stored":1259864064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":998834176,"data_stored":998834176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":824573952,"data_stored":824573952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2114060288,"data_stored":2114060288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":2038956032,"data_stored":2038956032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2548563968,"data_stored":2548563968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":265748480,"data_stored":265748480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1324351488,"data_stored":1324351488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T12:49:39.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:39 smithi177 ceph-mon[44303]: 2.e scrub ok 2024-05-15T12:49:39.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:39 smithi177 ceph-mon[44303]: osdmap e151: 12 total, 12 up, 11 in 2024-05-15T12:49:40.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:39 smithi120 ceph-mon[43961]: 2.e scrub ok 2024-05-15T12:49:40.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:39 smithi120 ceph-mon[43961]: osdmap e151: 12 total, 12 up, 11 in 2024-05-15T12:49:40.781 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Total time run: 90.013 2024-05-15T12:49:40.781 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Total writes made: 107130 2024-05-15T12:49:40.782 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Write size: 65536 2024-05-15T12:49:40.782 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Object size: 65536 2024-05-15T12:49:40.782 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Bandwidth (MB/sec): 74.3851 2024-05-15T12:49:40.782 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Stddev Bandwidth: 54.8435 2024-05-15T12:49:40.782 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Max bandwidth (MB/sec): 261.75 2024-05-15T12:49:40.782 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Min bandwidth (MB/sec): 0 2024-05-15T12:49:40.782 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Average IOPS: 1190 2024-05-15T12:49:40.782 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Stddev IOPS: 877.507 2024-05-15T12:49:40.782 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Max IOPS: 4188 2024-05-15T12:49:40.782 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Min IOPS: 0 2024-05-15T12:49:40.782 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Average Latency(s): 0.0134401 2024-05-15T12:49:40.782 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Stddev Latency(s): 0.0984634 2024-05-15T12:49:40.782 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Max latency(s): 7.07499 2024-05-15T12:49:40.783 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Min latency(s): 0.00104218 2024-05-15T12:49:40.786 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Cleaning up (deleting benchmark objects) 2024-05-15T12:49:40.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:40 smithi177 ceph-mon[44303]: pgmap v451: 17 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 10 active+clean; 6.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 37 MiB/s wr, 590 op/s; 26714/311445 objects misplaced (8.577%); 13 MiB/s, 203 objects/s recovering 2024-05-15T12:49:41.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:40 smithi027 ceph-mon[39795]: pgmap v451: 17 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 10 active+clean; 6.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 37 MiB/s wr, 590 op/s; 26714/311445 objects misplaced (8.577%); 13 MiB/s, 203 objects/s recovering 2024-05-15T12:49:41.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:40 smithi120 ceph-mon[43961]: pgmap v451: 17 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 10 active+clean; 6.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 37 MiB/s wr, 590 op/s; 26714/311445 objects misplaced (8.577%); 13 MiB/s, 203 objects/s recovering 2024-05-15T12:49:42.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:42 smithi177 ceph-mon[44303]: pgmap v452: 17 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 10 active+clean; 6.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 45 MiB/s wr, 714 op/s; 26095/317796 objects misplaced (8.211%); 16 MiB/s, 257 objects/s recovering 2024-05-15T12:49:43.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:42 smithi027 ceph-mon[39795]: pgmap v452: 17 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 10 active+clean; 6.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 45 MiB/s wr, 714 op/s; 26095/317796 objects misplaced (8.211%); 16 MiB/s, 257 objects/s recovering 2024-05-15T12:49:43.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:42 smithi120 ceph-mon[43961]: pgmap v452: 17 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 10 active+clean; 6.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 45 MiB/s wr, 714 op/s; 26095/317796 objects misplaced (8.211%); 16 MiB/s, 257 objects/s recovering 2024-05-15T12:49:44.793 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] out_osds: [11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T12:49:44.793 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 0 1 2024-05-15T12:49:45.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:44 smithi027 ceph-mon[39795]: pgmap v453: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.5 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 42 MiB/s wr, 758 op/s; 25593/318735 objects misplaced (8.030%); 16 MiB/s, 253 objects/s recovering 2024-05-15T12:49:45.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:44 smithi120 ceph-mon[43961]: pgmap v453: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.5 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 42 MiB/s wr, 758 op/s; 25593/318735 objects misplaced (8.030%); 16 MiB/s, 253 objects/s recovering 2024-05-15T12:49:45.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:44 smithi177 ceph-mon[44303]: pgmap v453: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.5 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 42 MiB/s wr, 758 op/s; 25593/318735 objects misplaced (8.030%); 16 MiB/s, 253 objects/s recovering 2024-05-15T12:49:47.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:46 smithi027 ceph-mon[39795]: pgmap v454: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 113 B/s rd, 34 MiB/s wr, 774 op/s; 25069/314922 objects misplaced (7.960%); 16 MiB/s, 251 objects/s recovering 2024-05-15T12:49:47.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:46 smithi120 ceph-mon[43961]: pgmap v454: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 113 B/s rd, 34 MiB/s wr, 774 op/s; 25069/314922 objects misplaced (7.960%); 16 MiB/s, 251 objects/s recovering 2024-05-15T12:49:47.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:46 smithi177 ceph-mon[44303]: pgmap v454: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 113 B/s rd, 34 MiB/s wr, 774 op/s; 25069/314922 objects misplaced (7.960%); 16 MiB/s, 251 objects/s recovering 2024-05-15T12:49:47.407 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:49:47.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:47.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:49:47.513 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:49:47.530 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:49:47.550 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:47.573 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:49:47.580 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:49:47.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:47.586 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:49:47.614 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:49:47.686 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:49:47.703 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:49:47.718 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:47 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2450246812' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:49:47.724 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:47.741 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:49:47.753 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:49:47.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:47.762 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:49:47.791 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:49:47.860 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:49:47.863 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:49:47.924 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:49:47.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:47 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2450246812' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:49:47.962 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:47.972 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:49:48.029 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:49:48.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:47 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2450246812' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:49:48.069 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:49:48.121 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:49:48.136 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:49:48.136 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:49:48.136 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:49:48.136 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:49:48.136 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:49:48.136 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:49:48.136 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:49:48.136 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:49:48.136 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:49:48.136 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:49:48.136 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:49:48.136 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:49:48.359 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:49:48.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:48.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:49:48.467 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:49:48.545 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:48.600 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:49:48.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:48.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:49:48.709 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:49:48.742 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:48.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:48 smithi027 ceph-mon[39795]: pgmap v455: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 102 B/s rd, 29 MiB/s wr, 774 op/s; 24313/312090 objects misplaced (7.790%); 16 MiB/s, 261 objects/s recovering 2024-05-15T12:49:48.773 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:49:48.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:49:48.777 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:49:48.878 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:49:48.956 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:49:49.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:48 smithi120 ceph-mon[43961]: pgmap v455: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 102 B/s rd, 29 MiB/s wr, 774 op/s; 24313/312090 objects misplaced (7.790%); 16 MiB/s, 261 objects/s recovering 2024-05-15T12:49:49.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:48 smithi177 ceph-mon[44303]: pgmap v455: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 102 B/s rd, 29 MiB/s wr, 774 op/s; 24313/312090 objects misplaced (7.790%); 16 MiB/s, 261 objects/s recovering 2024-05-15T12:49:49.748 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T12:49:50.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:49 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3503211272' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:49:50.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:49 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:49:50.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:49 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3503211272' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:49:50.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:49 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:49:50.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:49 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3503211272' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:49:50.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:49 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:49:50.742 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:50 smithi027 ceph-mon[39795]: pgmap v456: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 93 B/s rd, 19 MiB/s wr, 727 op/s; 23833/307518 objects misplaced (7.750%); 15 MiB/s, 241 objects/s recovering 2024-05-15T12:49:50.742 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:50 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:49:50.742 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:50 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:49:50.743 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:50 smithi027 ceph-mon[39795]: osdmap e152: 12 total, 12 up, 11 in 2024-05-15T12:49:51.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:50 smithi120 ceph-mon[43961]: pgmap v456: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 93 B/s rd, 19 MiB/s wr, 727 op/s; 23833/307518 objects misplaced (7.750%); 15 MiB/s, 241 objects/s recovering 2024-05-15T12:49:51.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:50 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:49:51.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:50 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:49:51.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:50 smithi120 ceph-mon[43961]: osdmap e152: 12 total, 12 up, 11 in 2024-05-15T12:49:51.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:50 smithi177 ceph-mon[44303]: pgmap v456: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 93 B/s rd, 19 MiB/s wr, 727 op/s; 23833/307518 objects misplaced (7.750%); 15 MiB/s, 241 objects/s recovering 2024-05-15T12:49:51.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:50 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:49:51.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:50 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:49:51.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:50 smithi177 ceph-mon[44303]: osdmap e152: 12 total, 12 up, 11 in 2024-05-15T12:49:53.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:52 smithi027 ceph-mon[39795]: pgmap v458: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.1 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 102 B/s rd, 7.5 MiB/s wr, 798 op/s; 22788/301044 objects misplaced (7.570%); 17 MiB/s, 268 objects/s recovering 2024-05-15T12:49:53.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:52 smithi120 ceph-mon[43961]: pgmap v458: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.1 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 102 B/s rd, 7.5 MiB/s wr, 798 op/s; 22788/301044 objects misplaced (7.570%); 17 MiB/s, 268 objects/s recovering 2024-05-15T12:49:53.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:52 smithi177 ceph-mon[44303]: pgmap v458: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.1 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 102 B/s rd, 7.5 MiB/s wr, 798 op/s; 22788/301044 objects misplaced (7.570%); 17 MiB/s, 268 objects/s recovering 2024-05-15T12:49:54.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:53 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:49:54.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:53 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:49:54.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:53 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:49:54.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:53 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:49:54.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:53 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:49:54.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:53 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:49:55.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:54 smithi177 ceph-mon[44303]: pgmap v459: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 102 B/s rd, 1.0 MiB/s wr, 800 op/s; 22050/295722 objects misplaced (7.456%); 17 MiB/s, 269 objects/s recovering 2024-05-15T12:49:55.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:54 smithi177 ceph-mon[44303]: osdmap e153: 12 total, 12 up, 11 in 2024-05-15T12:49:55.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:54 smithi027 ceph-mon[39795]: pgmap v459: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 102 B/s rd, 1.0 MiB/s wr, 800 op/s; 22050/295722 objects misplaced (7.456%); 17 MiB/s, 269 objects/s recovering 2024-05-15T12:49:55.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:54 smithi027 ceph-mon[39795]: osdmap e153: 12 total, 12 up, 11 in 2024-05-15T12:49:55.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:54 smithi120 ceph-mon[43961]: pgmap v459: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 6.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 102 B/s rd, 1.0 MiB/s wr, 800 op/s; 22050/295722 objects misplaced (7.456%); 17 MiB/s, 269 objects/s recovering 2024-05-15T12:49:55.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:54 smithi120 ceph-mon[43961]: osdmap e153: 12 total, 12 up, 11 in 2024-05-15T12:49:57.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:56 smithi177 ceph-mon[44303]: pgmap v461: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.9 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 873 op/s; 21449/291108 objects misplaced (7.368%); 17 MiB/s, 274 objects/s recovering 2024-05-15T12:49:57.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:56 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:49:57.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:56 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:49:57.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:56 smithi027 ceph-mon[39795]: pgmap v461: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.9 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 873 op/s; 21449/291108 objects misplaced (7.368%); 17 MiB/s, 274 objects/s recovering 2024-05-15T12:49:57.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:56 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:49:57.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:56 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:49:57.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:56 smithi120 ceph-mon[43961]: pgmap v461: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.9 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 873 op/s; 21449/291108 objects misplaced (7.368%); 17 MiB/s, 274 objects/s recovering 2024-05-15T12:49:57.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:56 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:49:57.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:56 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:49:57.376 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T12:49:57.377 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T12:49:57.377 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-05-15T12:49:57.377 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-05-15T12:49:57.377 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T12:49:58.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:57 smithi177 ceph-mon[44303]: osdmap e154: 12 total, 12 up, 11 in 2024-05-15T12:49:58.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:57 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:49:58.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:57 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:49:58.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:57 smithi027 ceph-mon[39795]: osdmap e154: 12 total, 12 up, 11 in 2024-05-15T12:49:58.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:57 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:49:58.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:57 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:49:58.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:57 smithi120 ceph-mon[43961]: osdmap e154: 12 total, 12 up, 11 in 2024-05-15T12:49:58.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:57 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:49:58.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:57 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:49:59.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:58 smithi177 ceph-mon[44303]: pgmap v463: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.9 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 817 op/s; 20625/288333 objects misplaced (7.153%); 21 MiB/s, 333 objects/s recovering 2024-05-15T12:49:59.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:49:58 smithi177 ceph-mon[44303]: osdmap e155: 12 total, 12 up, 11 in 2024-05-15T12:49:59.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:58 smithi027 ceph-mon[39795]: pgmap v463: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.9 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 817 op/s; 20625/288333 objects misplaced (7.153%); 21 MiB/s, 333 objects/s recovering 2024-05-15T12:49:59.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:49:58 smithi027 ceph-mon[39795]: osdmap e155: 12 total, 12 up, 11 in 2024-05-15T12:49:59.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:58 smithi120 ceph-mon[43961]: pgmap v463: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.9 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 817 op/s; 20625/288333 objects misplaced (7.153%); 21 MiB/s, 333 objects/s recovering 2024-05-15T12:49:59.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:49:58 smithi120 ceph-mon[43961]: osdmap e155: 12 total, 12 up, 11 in 2024-05-15T12:50:01.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:00 smithi177 ceph-mon[44303]: pgmap v465: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 558 op/s; 19934/285663 objects misplaced (6.978%); 19 MiB/s, 301 objects/s recovering 2024-05-15T12:50:01.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:00 smithi177 ceph-mon[44303]: Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-05-15T12:50:01.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:00 smithi177 ceph-mon[44303]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-05-15T12:50:01.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:00 smithi177 ceph-mon[44303]: pg 2.7 is active+remapped+backfill_toofull, acting [6,2,3] 2024-05-15T12:50:01.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:00 smithi027 ceph-mon[39795]: pgmap v465: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 558 op/s; 19934/285663 objects misplaced (6.978%); 19 MiB/s, 301 objects/s recovering 2024-05-15T12:50:01.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:00 smithi027 ceph-mon[39795]: Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-05-15T12:50:01.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:00 smithi027 ceph-mon[39795]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-05-15T12:50:01.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:00 smithi027 ceph-mon[39795]: pg 2.7 is active+remapped+backfill_toofull, acting [6,2,3] 2024-05-15T12:50:01.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:00 smithi120 ceph-mon[43961]: pgmap v465: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 558 op/s; 19934/285663 objects misplaced (6.978%); 19 MiB/s, 301 objects/s recovering 2024-05-15T12:50:01.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:00 smithi120 ceph-mon[43961]: Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-05-15T12:50:01.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:00 smithi120 ceph-mon[43961]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-05-15T12:50:01.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:00 smithi120 ceph-mon[43961]: pg 2.7 is active+remapped+backfill_toofull, acting [6,2,3] 2024-05-15T12:50:01.725 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:50:01.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:01.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:50:01.829 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:50:01.864 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:03.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:02 smithi027 ceph-mon[39795]: pgmap v466: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 465 op/s; 18483/285018 objects misplaced (6.485%); 26 MiB/s, 423 objects/s recovering 2024-05-15T12:50:03.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:02 smithi177 ceph-mon[44303]: pgmap v466: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 465 op/s; 18483/285018 objects misplaced (6.485%); 26 MiB/s, 423 objects/s recovering 2024-05-15T12:50:03.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:02 smithi120 ceph-mon[43961]: pgmap v466: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 465 op/s; 18483/285018 objects misplaced (6.485%); 26 MiB/s, 423 objects/s recovering 2024-05-15T12:50:04.777 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:50:05.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:04 smithi177 ceph-mon[44303]: pgmap v467: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 253 op/s; 16609/285018 objects misplaced (5.827%); 35 MiB/s, 565 objects/s recovering 2024-05-15T12:50:05.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:04 smithi027 ceph-mon[39795]: pgmap v467: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 253 op/s; 16609/285018 objects misplaced (5.827%); 35 MiB/s, 565 objects/s recovering 2024-05-15T12:50:05.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:04 smithi120 ceph-mon[43961]: pgmap v467: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 11 active+clean; 5.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 253 op/s; 16609/285018 objects misplaced (5.827%); 35 MiB/s, 565 objects/s recovering 2024-05-15T12:50:07.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:06 smithi177 ceph-mon[44303]: pgmap v468: 17 pgs: 1 active+undersized+remapped, 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 11 active+clean; 5.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 127 op/s; 14844/285018 objects misplaced (5.208%); 41 MiB/s, 651 objects/s recovering 2024-05-15T12:50:07.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:06 smithi177 ceph-mon[44303]: osdmap e156: 12 total, 12 up, 11 in 2024-05-15T12:50:07.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:06 smithi027 ceph-mon[39795]: pgmap v468: 17 pgs: 1 active+undersized+remapped, 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 11 active+clean; 5.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 127 op/s; 14844/285018 objects misplaced (5.208%); 41 MiB/s, 651 objects/s recovering 2024-05-15T12:50:07.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:06 smithi027 ceph-mon[39795]: osdmap e156: 12 total, 12 up, 11 in 2024-05-15T12:50:07.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:06 smithi120 ceph-mon[43961]: pgmap v468: 17 pgs: 1 active+undersized+remapped, 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 11 active+clean; 5.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 127 op/s; 14844/285018 objects misplaced (5.208%); 41 MiB/s, 651 objects/s recovering 2024-05-15T12:50:07.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:06 smithi120 ceph-mon[43961]: osdmap e156: 12 total, 12 up, 11 in 2024-05-15T12:50:08.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:07 smithi177 ceph-mon[44303]: osdmap e157: 12 total, 12 up, 11 in 2024-05-15T12:50:08.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:07 smithi177 ceph-mon[44303]: osdmap e158: 12 total, 12 up, 11 in 2024-05-15T12:50:08.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:07 smithi027 ceph-mon[39795]: osdmap e157: 12 total, 12 up, 11 in 2024-05-15T12:50:08.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:07 smithi027 ceph-mon[39795]: osdmap e158: 12 total, 12 up, 11 in 2024-05-15T12:50:08.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:07 smithi120 ceph-mon[43961]: osdmap e157: 12 total, 12 up, 11 in 2024-05-15T12:50:08.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:07 smithi120 ceph-mon[43961]: osdmap e158: 12 total, 12 up, 11 in 2024-05-15T12:50:09.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:08 smithi177 ceph-mon[44303]: pgmap v471: 17 pgs: 1 active+remapped+backfilling, 1 active+undersized+remapped, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 11 active+clean; 5.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 26 op/s; 13951/285018 objects misplaced (4.895%); 47 MiB/s, 746 objects/s recovering 2024-05-15T12:50:09.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:08 smithi177 ceph-mon[44303]: Health check failed: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-05-15T12:50:09.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:08 smithi177 ceph-mon[44303]: osdmap e159: 12 total, 12 up, 11 in 2024-05-15T12:50:09.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:08 smithi027 ceph-mon[39795]: pgmap v471: 17 pgs: 1 active+remapped+backfilling, 1 active+undersized+remapped, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 11 active+clean; 5.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 26 op/s; 13951/285018 objects misplaced (4.895%); 47 MiB/s, 746 objects/s recovering 2024-05-15T12:50:09.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:08 smithi027 ceph-mon[39795]: Health check failed: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-05-15T12:50:09.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:08 smithi027 ceph-mon[39795]: osdmap e159: 12 total, 12 up, 11 in 2024-05-15T12:50:09.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:08 smithi120 ceph-mon[43961]: pgmap v471: 17 pgs: 1 active+remapped+backfilling, 1 active+undersized+remapped, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 11 active+clean; 5.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 26 op/s; 13951/285018 objects misplaced (4.895%); 47 MiB/s, 746 objects/s recovering 2024-05-15T12:50:09.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:08 smithi120 ceph-mon[43961]: Health check failed: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-05-15T12:50:09.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:08 smithi120 ceph-mon[43961]: osdmap e159: 12 total, 12 up, 11 in 2024-05-15T12:50:11.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:10 smithi027 ceph-mon[39795]: pgmap v474: 17 pgs: 1 peering, 11 active+clean, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 1 active+undersized+remapped, 1 active+remapped+backfilling; 5.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 12883/285018 objects misplaced (4.520%); 31 MiB/s, 490 objects/s recovering 2024-05-15T12:50:11.211 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:10 smithi027 ceph-mon[39795]: 2.c scrub starts 2024-05-15T12:50:11.212 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-05-15T12:50:11.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:10 smithi120 ceph-mon[43961]: pgmap v474: 17 pgs: 1 peering, 11 active+clean, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 1 active+undersized+remapped, 1 active+remapped+backfilling; 5.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 12883/285018 objects misplaced (4.520%); 31 MiB/s, 490 objects/s recovering 2024-05-15T12:50:11.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:10 smithi120 ceph-mon[43961]: 2.c scrub starts 2024-05-15T12:50:11.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:10 smithi177 ceph-mon[44303]: pgmap v474: 17 pgs: 1 peering, 11 active+clean, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering+remapped, 1 active+undersized+remapped, 1 active+remapped+backfilling; 5.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 12883/285018 objects misplaced (4.520%); 31 MiB/s, 490 objects/s recovering 2024-05-15T12:50:11.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:10 smithi177 ceph-mon[44303]: 2.c scrub starts 2024-05-15T12:50:12.249 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:11 smithi027 ceph-mon[39795]: 2.c scrub ok 2024-05-15T12:50:12.249 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:11 smithi027 ceph-mon[39795]: 2.b scrub starts 2024-05-15T12:50:12.249 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:11 smithi027 ceph-mon[39795]: osdmap e160: 12 total, 12 up, 11 in 2024-05-15T12:50:12.250 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:11 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:50:12.250 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:11 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:50:12.265 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:50:12.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:12.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:50:12.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:11 smithi120 ceph-mon[43961]: 2.c scrub ok 2024-05-15T12:50:12.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:11 smithi120 ceph-mon[43961]: 2.b scrub starts 2024-05-15T12:50:12.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:11 smithi120 ceph-mon[43961]: osdmap e160: 12 total, 12 up, 11 in 2024-05-15T12:50:12.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:11 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:50:12.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:11 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:50:12.306 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:50:12.348 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:50:12.356 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:50:12.369 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:50:12.398 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:11 smithi177 ceph-mon[44303]: 2.c scrub ok 2024-05-15T12:50:12.398 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:11 smithi177 ceph-mon[44303]: 2.b scrub starts 2024-05-15T12:50:12.398 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:11 smithi177 ceph-mon[44303]: osdmap e160: 12 total, 12 up, 11 in 2024-05-15T12:50:12.398 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:11 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:50:12.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:11 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:50:12.401 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:12.425 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:50:12.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:12.428 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:50:12.529 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:50:12.561 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:12.583 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:50:12.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:12.587 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:50:12.688 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:50:12.718 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:12.749 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:50:12.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:12.753 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:50:12.854 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:50:12.898 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:12.916 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:50:12.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:12.919 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:50:13.020 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:50:13.050 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:12 smithi177 ceph-mon[44303]: 2.b scrub ok 2024-05-15T12:50:13.050 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:12 smithi177 ceph-mon[44303]: pgmap v476: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 peering, 12 active+clean, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull; 5.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1 op/s; 11772/284991 objects misplaced (4.131%); 29 MiB/s, 468 objects/s recovering 2024-05-15T12:50:13.050 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:12 smithi177 ceph-mon[44303]: osdmap e161: 12 total, 12 up, 11 in 2024-05-15T12:50:13.050 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:12 smithi177 ceph-mon[44303]: osdmap e162: 12 total, 12 up, 11 in 2024-05-15T12:50:13.052 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:13.074 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:50:13.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:13.077 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:50:13.178 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:50:13.210 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:13.233 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:50:13.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:13.236 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:50:13.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:12 smithi027 ceph-mon[39795]: 2.b scrub ok 2024-05-15T12:50:13.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:12 smithi027 ceph-mon[39795]: pgmap v476: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 peering, 12 active+clean, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull; 5.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1 op/s; 11772/284991 objects misplaced (4.131%); 29 MiB/s, 468 objects/s recovering 2024-05-15T12:50:13.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:12 smithi027 ceph-mon[39795]: osdmap e161: 12 total, 12 up, 11 in 2024-05-15T12:50:13.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:12 smithi027 ceph-mon[39795]: osdmap e162: 12 total, 12 up, 11 in 2024-05-15T12:50:13.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:12 smithi120 ceph-mon[43961]: 2.b scrub ok 2024-05-15T12:50:13.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:12 smithi120 ceph-mon[43961]: pgmap v476: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 peering, 12 active+clean, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull; 5.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1 op/s; 11772/284991 objects misplaced (4.131%); 29 MiB/s, 468 objects/s recovering 2024-05-15T12:50:13.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:12 smithi120 ceph-mon[43961]: osdmap e161: 12 total, 12 up, 11 in 2024-05-15T12:50:13.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:12 smithi120 ceph-mon[43961]: osdmap e162: 12 total, 12 up, 11 in 2024-05-15T12:50:13.337 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:50:13.397 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:14.250 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T12:50:14.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:14 smithi120 ceph-mon[43961]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-05-15T12:50:14.289 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:50:14.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:14.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:50:14.293 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:14 smithi027 ceph-mon[39795]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-05-15T12:50:14.394 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:50:14.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:14 smithi177 ceph-mon[44303]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-05-15T12:50:14.430 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:14.458 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:50:14.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:14.464 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:50:14.565 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:50:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:14.733 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:50:14.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:14.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:50:14.840 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:50:14.870 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:14.891 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:50:14.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:14.900 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:50:15.001 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:50:15.064 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:15.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:15 smithi177 ceph-mon[44303]: pgmap v479: 17 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 peering, 13 active+clean; 5.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 121 op/s; 10665/283311 objects misplaced (3.764%); 28 MiB/s, 445 objects/s recovering 2024-05-15T12:50:15.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:15 smithi177 ceph-mon[44303]: 2.f scrub starts 2024-05-15T12:50:15.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:15 smithi027 ceph-mon[39795]: pgmap v479: 17 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 peering, 13 active+clean; 5.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 121 op/s; 10665/283311 objects misplaced (3.764%); 28 MiB/s, 445 objects/s recovering 2024-05-15T12:50:15.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:15 smithi027 ceph-mon[39795]: 2.f scrub starts 2024-05-15T12:50:15.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:15 smithi120 ceph-mon[43961]: pgmap v479: 17 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 peering, 13 active+clean; 5.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 121 op/s; 10665/283311 objects misplaced (3.764%); 28 MiB/s, 445 objects/s recovering 2024-05-15T12:50:15.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:15 smithi120 ceph-mon[43961]: 2.f scrub starts 2024-05-15T12:50:16.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:16 smithi027 ceph-mon[39795]: 2.f scrub ok 2024-05-15T12:50:16.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:16 smithi027 ceph-mon[39795]: 2.3 scrub starts 2024-05-15T12:50:16.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:16 smithi177 ceph-mon[44303]: 2.f scrub ok 2024-05-15T12:50:16.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:16 smithi177 ceph-mon[44303]: 2.3 scrub starts 2024-05-15T12:50:16.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:16 smithi120 ceph-mon[43961]: 2.f scrub ok 2024-05-15T12:50:16.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:16 smithi120 ceph-mon[43961]: 2.3 scrub starts 2024-05-15T12:50:16.750 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:50:16.757 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:50:16.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:16.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:50:16.865 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:50:16.896 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:16.919 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:50:16.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:16.922 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:50:17.023 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:50:17.055 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:17.078 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:50:17.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:17.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:50:17.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:17 smithi177 ceph-mon[44303]: pgmap v480: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 peering, 13 active+clean; 5.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 314 op/s; 10241/279360 objects misplaced (3.666%); 26 MiB/s, 418 objects/s recovering 2024-05-15T12:50:17.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:17 smithi177 ceph-mon[44303]: 2.3 scrub ok 2024-05-15T12:50:17.182 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:50:17.250 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:17.323 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:17 smithi027 ceph-mon[39795]: pgmap v480: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 peering, 13 active+clean; 5.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 314 op/s; 10241/279360 objects misplaced (3.666%); 26 MiB/s, 418 objects/s recovering 2024-05-15T12:50:17.324 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:17 smithi027 ceph-mon[39795]: 2.3 scrub ok 2024-05-15T12:50:17.358 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:50:17.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:17.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:50:17.463 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:50:17.497 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:17 smithi120 ceph-mon[43961]: pgmap v480: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 peering, 13 active+clean; 5.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 314 op/s; 10241/279360 objects misplaced (3.666%); 26 MiB/s, 418 objects/s recovering 2024-05-15T12:50:17.497 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:17 smithi120 ceph-mon[43961]: 2.3 scrub ok 2024-05-15T12:50:17.499 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:17.524 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:50:17.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:17.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:50:17.628 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:50:17.706 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:17.742 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:50:17.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:17.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:50:17.850 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:50:17.884 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:17.906 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:50:17.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:17.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:50:18.009 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:50:18.049 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T12:50:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T12:50:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T12:50:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T12:50:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:91491 2.a 2:54e8cb2a:::benchmark_data_smithi177_78104_object91490:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-05-15T12:50:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:20.185232+0000", 2024-05-15T12:50:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 57.855314317000001, 2024-05-15T12:50:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.094834012999999995, 2024-05-15T12:50:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:50:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:50:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:50:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:50:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 91491 2024-05-15T12:50:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:50:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:50:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.185232+0000", 2024-05-15T12:50:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:50:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.185232+0000", 2024-05-15T12:50:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:50:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.185234+0000", 2024-05-15T12:50:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.198e-06 2024-05-15T12:50:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:50:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.185274+0000", 2024-05-15T12:50:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9966999999999999e-05 2024-05-15T12:50:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:50:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.185284+0000", 2024-05-15T12:50:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0696999999999999e-05 2024-05-15T12:50:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:50:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.185291+0000", 2024-05-15T12:50:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7329999999999997e-06 2024-05-15T12:50:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.186523+0000", 2024-05-15T12:50:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001231733 2024-05-15T12:50:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:50:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.186690+0000", 2024-05-15T12:50:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016688 2024-05-15T12:50:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,9", 2024-05-15T12:50:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.186883+0000", 2024-05-15T12:50:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00019343499999999999 2024-05-15T12:50:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:50:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.191634+0000", 2024-05-15T12:50:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0047507649999999997 2024-05-15T12:50:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.277923+0000", 2024-05-15T12:50:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.086289489999999996 2024-05-15T12:50:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.277977+0000", 2024-05-15T12:50:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3706999999999999e-05 2024-05-15T12:50:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:50:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.278043+0000", 2024-05-15T12:50:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6007999999999995e-05 2024-05-15T12:50:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:50:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.280066+0000", 2024-05-15T12:50:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020233999999999999 2024-05-15T12:50:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:50:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:91478 2.0 e145/144 2:0e661660:::benchmark_data_smithi177_78104_object91477:head v 145'5771, mlcod=115'451)", 2024-05-15T12:50:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:20.190369+0000", 2024-05-15T12:50:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 57.850177293000002, 2024-05-15T12:50:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.093120244000000005, 2024-05-15T12:50:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:50:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:50:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:50:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.190369+0000", 2024-05-15T12:50:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:50:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.190369+0000", 2024-05-15T12:50:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:50:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.190370+0000", 2024-05-15T12:50:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6000000000000003e-07 2024-05-15T12:50:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:50:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.190473+0000", 2024-05-15T12:50:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000102741 2024-05-15T12:50:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:50:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.190482+0000", 2024-05-15T12:50:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4660000000000005e-06 2024-05-15T12:50:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:50:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.190489+0000", 2024-05-15T12:50:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5710000000000003e-06 2024-05-15T12:50:18.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.059 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.059 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.190537+0000", 2024-05-15T12:50:18.059 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8180999999999998e-05 2024-05-15T12:50:18.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.059 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:50:18.059 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.190554+0000", 2024-05-15T12:50:18.059 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6765e-05 2024-05-15T12:50:18.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.059 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:50:18.059 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.283406+0000", 2024-05-15T12:50:18.060 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.092852377 2024-05-15T12:50:18.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.060 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:50:18.060 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.283490+0000", 2024-05-15T12:50:18.060 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3382999999999998e-05 2024-05-15T12:50:18.060 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.060 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:50:18.060 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.060 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:91480 2.0 e145/144 2:05578dff:::benchmark_data_smithi177_78104_object91479:head v 145'5772, mlcod=115'451)", 2024-05-15T12:50:18.060 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:20.193004+0000", 2024-05-15T12:50:18.061 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 57.847543213999998, 2024-05-15T12:50:18.061 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.090549389999999993, 2024-05-15T12:50:18.061 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:50:18.061 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:18.061 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:50:18.061 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.061 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:50:18.061 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.193004+0000", 2024-05-15T12:50:18.061 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.061 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.061 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:50:18.061 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.193004+0000", 2024-05-15T12:50:18.061 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.062 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.062 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:50:18.062 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.193005+0000", 2024-05-15T12:50:18.062 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0380000000000001e-06 2024-05-15T12:50:18.062 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.062 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.062 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:50:18.062 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.193132+0000", 2024-05-15T12:50:18.062 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012757799999999999 2024-05-15T12:50:18.062 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.062 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.062 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:50:18.062 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.273531+0000", 2024-05-15T12:50:18.062 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.080398481999999993 2024-05-15T12:50:18.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.063 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:50:18.063 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.273545+0000", 2024-05-15T12:50:18.063 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4591e-05 2024-05-15T12:50:18.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.063 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.063 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.277339+0000", 2024-05-15T12:50:18.063 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003794216 2024-05-15T12:50:18.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.063 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:50:18.064 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.277369+0000", 2024-05-15T12:50:18.064 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9830000000000001e-05 2024-05-15T12:50:18.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.064 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.064 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:50:18.064 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.283493+0000", 2024-05-15T12:50:18.064 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0061238159999999998 2024-05-15T12:50:18.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.064 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.064 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:50:18.064 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.283553+0000", 2024-05-15T12:50:18.064 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9839000000000001e-05 2024-05-15T12:50:18.064 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.065 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:50:18.065 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.065 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.065 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.065 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:91494 2.a 2:590d89f2:::benchmark_data_smithi177_78104_object91493:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-05-15T12:50:18.065 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:20.193839+0000", 2024-05-15T12:50:18.065 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 57.846707866999999, 2024-05-15T12:50:18.065 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.099715965000000004, 2024-05-15T12:50:18.065 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:50:18.065 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:18.065 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:50:18.065 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:50:18.065 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:50:18.066 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 91494 2024-05-15T12:50:18.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.066 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:50:18.066 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.066 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:50:18.066 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.193839+0000", 2024-05-15T12:50:18.066 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.066 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.066 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:50:18.066 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.193839+0000", 2024-05-15T12:50:18.066 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.066 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.067 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:50:18.067 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.193840+0000", 2024-05-15T12:50:18.067 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5880000000000001e-06 2024-05-15T12:50:18.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.067 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.067 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:50:18.067 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.193921+0000", 2024-05-15T12:50:18.067 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0900000000000001e-05 2024-05-15T12:50:18.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.067 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.067 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:50:18.067 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.273938+0000", 2024-05-15T12:50:18.067 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.080016176999999994 2024-05-15T12:50:18.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.068 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.068 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:50:18.068 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.273948+0000", 2024-05-15T12:50:18.068 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0515e-05 2024-05-15T12:50:18.068 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.068 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.068 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.068 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.280134+0000", 2024-05-15T12:50:18.068 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.006185439 2024-05-15T12:50:18.068 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.068 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.068 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:50:18.068 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.280321+0000", 2024-05-15T12:50:18.069 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00018760599999999999 2024-05-15T12:50:18.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.069 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,9", 2024-05-15T12:50:18.069 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.280547+0000", 2024-05-15T12:50:18.069 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00022572099999999999 2024-05-15T12:50:18.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.069 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:50:18.069 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.287725+0000", 2024-05-15T12:50:18.069 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0071782499999999997 2024-05-15T12:50:18.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.069 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.069 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.289787+0000", 2024-05-15T12:50:18.070 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002061415 2024-05-15T12:50:18.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.070 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.070 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.070 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.291880+0000", 2024-05-15T12:50:18.070 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020931470000000001 2024-05-15T12:50:18.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.070 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.070 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:50:18.070 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.291945+0000", 2024-05-15T12:50:18.070 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5096000000000002e-05 2024-05-15T12:50:18.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.070 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.071 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:50:18.071 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:20.293555+0000", 2024-05-15T12:50:18.071 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001610111 2024-05-15T12:50:18.071 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.071 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:50:18.071 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.071 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.071 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.071 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119622 2.a 2:5d9d6e4f:::benchmark_data_smithi177_78104_object12489:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-05-15T12:50:18.071 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:50:11.032834+0000", 2024-05-15T12:50:18.071 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 7.0077127279999996, 2024-05-15T12:50:18.071 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.97383829600000005, 2024-05-15T12:50:18.071 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:50:18.071 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:18.072 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:50:18.072 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:50:18.072 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:50:18.072 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119622 2024-05-15T12:50:18.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.072 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:50:18.072 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.072 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:50:18.072 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.032834+0000", 2024-05-15T12:50:18.072 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.072 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.072 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:50:18.073 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.032834+0000", 2024-05-15T12:50:18.073 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.073 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.073 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.073 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:50:18.073 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.032836+0000", 2024-05-15T12:50:18.073 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6300000000000001e-06 2024-05-15T12:50:18.073 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.073 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.073 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:50:18.073 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.032842+0000", 2024-05-15T12:50:18.073 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4339999999999997e-06 2024-05-15T12:50:18.073 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.073 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.074 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:50:18.074 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.032862+0000", 2024-05-15T12:50:18.074 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9578000000000001e-05 2024-05-15T12:50:18.074 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.074 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.074 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:50:18.074 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.032871+0000", 2024-05-15T12:50:18.074 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9379999999999998e-06 2024-05-15T12:50:18.074 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.074 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.074 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.074 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.032940+0000", 2024-05-15T12:50:18.074 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9434e-05 2024-05-15T12:50:18.075 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.075 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.075 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:50:18.075 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.032952+0000", 2024-05-15T12:50:18.075 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.236e-05 2024-05-15T12:50:18.075 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.075 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.075 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.075 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.997358+0000", 2024-05-15T12:50:18.075 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.96440568000000004 2024-05-15T12:50:18.075 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.075 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.075 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:50:18.075 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.997468+0000", 2024-05-15T12:50:18.076 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000110383 2024-05-15T12:50:18.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.076 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.076 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,9", 2024-05-15T12:50:18.076 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.997582+0000", 2024-05-15T12:50:18.076 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000113451 2024-05-15T12:50:18.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.076 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.076 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.076 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.000658+0000", 2024-05-15T12:50:18.076 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0030763449999999999 2024-05-15T12:50:18.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.076 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.076 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:50:18.077 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.003499+0000", 2024-05-15T12:50:18.077 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0028403489999999998 2024-05-15T12:50:18.077 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.077 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.077 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.077 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.006604+0000", 2024-05-15T12:50:18.077 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031058499999999998 2024-05-15T12:50:18.077 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.077 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.077 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:50:18.077 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.006644+0000", 2024-05-15T12:50:18.077 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9437999999999997e-05 2024-05-15T12:50:18.077 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.078 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.078 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:50:18.078 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.006672+0000", 2024-05-15T12:50:18.078 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8425999999999999e-05 2024-05-15T12:50:18.078 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.078 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:50:18.078 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.078 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.078 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.078 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119629 2.a 2:5e62ecf9:::benchmark_data_smithi177_78104_object12496:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-05-15T12:50:18.078 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:50:11.042700+0000", 2024-05-15T12:50:18.078 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 6.9978468869999997, 2024-05-15T12:50:18.078 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.96409628400000003, 2024-05-15T12:50:18.079 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:50:18.079 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:18.079 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:50:18.079 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:50:18.079 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:50:18.079 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119629 2024-05-15T12:50:18.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.079 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:50:18.079 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.079 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:50:18.079 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.042700+0000", 2024-05-15T12:50:18.079 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.080 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.080 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:50:18.080 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.042700+0000", 2024-05-15T12:50:18.080 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.080 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.080 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.080 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:50:18.080 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.042701+0000", 2024-05-15T12:50:18.080 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5400000000000001e-06 2024-05-15T12:50:18.080 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.080 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.080 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:50:18.080 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.042710+0000", 2024-05-15T12:50:18.081 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3510000000000002e-06 2024-05-15T12:50:18.081 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.081 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.081 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:50:18.081 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.042721+0000", 2024-05-15T12:50:18.081 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1656e-05 2024-05-15T12:50:18.081 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.081 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.081 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:50:18.081 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.042729+0000", 2024-05-15T12:50:18.081 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.396e-06 2024-05-15T12:50:18.081 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.081 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.081 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.082 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.042795+0000", 2024-05-15T12:50:18.082 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5988000000000006e-05 2024-05-15T12:50:18.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.082 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.082 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:50:18.082 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.042805+0000", 2024-05-15T12:50:18.082 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0366999999999999e-05 2024-05-15T12:50:18.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.082 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.082 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.082 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.000690+0000", 2024-05-15T12:50:18.082 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.95788437599999998 2024-05-15T12:50:18.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.083 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.083 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:50:18.083 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.000760+0000", 2024-05-15T12:50:18.083 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0604000000000004e-05 2024-05-15T12:50:18.083 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.083 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.083 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,9", 2024-05-15T12:50:18.083 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.000824+0000", 2024-05-15T12:50:18.083 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3477000000000002e-05 2024-05-15T12:50:18.083 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.083 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.083 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.083 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.006709+0000", 2024-05-15T12:50:18.083 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0058858239999999996 2024-05-15T12:50:18.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.084 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.084 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:50:18.084 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.006718+0000", 2024-05-15T12:50:18.084 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9469999999999992e-06 2024-05-15T12:50:18.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.084 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.084 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.084 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.006749+0000", 2024-05-15T12:50:18.084 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0528999999999999e-05 2024-05-15T12:50:18.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.084 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.084 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:50:18.084 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.006775+0000", 2024-05-15T12:50:18.085 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6420000000000001e-05 2024-05-15T12:50:18.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.085 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.085 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:50:18.085 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.006796+0000", 2024-05-15T12:50:18.085 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0809e-05 2024-05-15T12:50:18.085 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.085 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:50:18.085 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.085 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.085 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119634 2.a 2:5fd9716d:::benchmark_data_smithi177_78104_object12501:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-05-15T12:50:18.085 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:50:11.043155+0000", 2024-05-15T12:50:18.085 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 6.997391447, 2024-05-15T12:50:18.085 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.96375531000000003, 2024-05-15T12:50:18.086 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:50:18.086 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:18.086 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:50:18.086 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:50:18.086 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:50:18.086 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119634 2024-05-15T12:50:18.086 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.086 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:50:18.086 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.086 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:50:18.086 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.043155+0000", 2024-05-15T12:50:18.086 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.086 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.087 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:50:18.087 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.043155+0000", 2024-05-15T12:50:18.087 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.087 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:50:18.087 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.043156+0000", 2024-05-15T12:50:18.087 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7100000000000001e-07 2024-05-15T12:50:18.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.087 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:50:18.087 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.043159+0000", 2024-05-15T12:50:18.087 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3589999999999999e-06 2024-05-15T12:50:18.088 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.088 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.088 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:50:18.088 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.043168+0000", 2024-05-15T12:50:18.088 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.439e-06 2024-05-15T12:50:18.088 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.088 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.088 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:50:18.088 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.043174+0000", 2024-05-15T12:50:18.088 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0039999999999996e-06 2024-05-15T12:50:18.088 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.088 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.088 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.088 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.043224+0000", 2024-05-15T12:50:18.088 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0223999999999999e-05 2024-05-15T12:50:18.089 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.089 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.089 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:50:18.089 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.043236+0000", 2024-05-15T12:50:18.089 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1702000000000001e-05 2024-05-15T12:50:18.089 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.089 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.089 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.089 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.003528+0000", 2024-05-15T12:50:18.089 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.96029225200000001 2024-05-15T12:50:18.089 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.089 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.089 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:50:18.089 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.003637+0000", 2024-05-15T12:50:18.089 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010853699999999999 2024-05-15T12:50:18.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.090 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.090 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,9", 2024-05-15T12:50:18.090 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.003748+0000", 2024-05-15T12:50:18.090 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000111739 2024-05-15T12:50:18.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.090 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.090 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.090 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.006830+0000", 2024-05-15T12:50:18.090 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0030816070000000001 2024-05-15T12:50:18.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.090 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.090 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:50:18.090 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.006837+0000", 2024-05-15T12:50:18.090 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2450000000000001e-06 2024-05-15T12:50:18.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.091 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.091 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.006866+0000", 2024-05-15T12:50:18.091 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9170999999999999e-05 2024-05-15T12:50:18.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.091 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:50:18.091 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.006892+0000", 2024-05-15T12:50:18.091 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5432e-05 2024-05-15T12:50:18.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.091 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:50:18.092 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.006911+0000", 2024-05-15T12:50:18.092 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8828e-05 2024-05-15T12:50:18.092 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.092 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:50:18.092 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.092 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119635 2.a 2:590400db:::benchmark_data_smithi177_78104_object12502:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-05-15T12:50:18.092 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:50:11.047419+0000", 2024-05-15T12:50:18.092 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 6.9931275280000005, 2024-05-15T12:50:18.092 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.96939879900000003, 2024-05-15T12:50:18.092 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:50:18.092 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:18.092 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:50:18.092 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:50:18.092 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:50:18.093 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119635 2024-05-15T12:50:18.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.093 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:50:18.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.093 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:50:18.093 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.047419+0000", 2024-05-15T12:50:18.093 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.093 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:50:18.093 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.047419+0000", 2024-05-15T12:50:18.093 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.094 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:50:18.094 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.047421+0000", 2024-05-15T12:50:18.094 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.519e-06 2024-05-15T12:50:18.094 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.094 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.094 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:50:18.094 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.047425+0000", 2024-05-15T12:50:18.094 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0199999999999996e-06 2024-05-15T12:50:18.094 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.094 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.094 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:50:18.094 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.047442+0000", 2024-05-15T12:50:18.094 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6769000000000002e-05 2024-05-15T12:50:18.094 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.095 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.095 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:50:18.095 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.047450+0000", 2024-05-15T12:50:18.095 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4999999999999999e-06 2024-05-15T12:50:18.095 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.095 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.095 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.095 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.047537+0000", 2024-05-15T12:50:18.095 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7074e-05 2024-05-15T12:50:18.095 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.095 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.096 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:50:18.096 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.047550+0000", 2024-05-15T12:50:18.096 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3271999999999999e-05 2024-05-15T12:50:18.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.096 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.096 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.006937+0000", 2024-05-15T12:50:18.096 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.95938654999999995 2024-05-15T12:50:18.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.096 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:50:18.096 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.007009+0000", 2024-05-15T12:50:18.097 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2201999999999998e-05 2024-05-15T12:50:18.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.097 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,9", 2024-05-15T12:50:18.097 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.007066+0000", 2024-05-15T12:50:18.097 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7079e-05 2024-05-15T12:50:18.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.097 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.097 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.016661+0000", 2024-05-15T12:50:18.097 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0095945539999999999 2024-05-15T12:50:18.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.097 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:50:18.097 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.016671+0000", 2024-05-15T12:50:18.098 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.063e-05 2024-05-15T12:50:18.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.098 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.098 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.098 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.016704+0000", 2024-05-15T12:50:18.098 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2938000000000002e-05 2024-05-15T12:50:18.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.098 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.098 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:50:18.098 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.016771+0000", 2024-05-15T12:50:18.098 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6945000000000004e-05 2024-05-15T12:50:18.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.098 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.098 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:50:18.099 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.016818+0000", 2024-05-15T12:50:18.099 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6746999999999998e-05 2024-05-15T12:50:18.099 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.099 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:50:18.099 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.099 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119641 2.a 2:538069bf:::benchmark_data_smithi177_78104_object12508:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-05-15T12:50:18.099 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:50:11.048535+0000", 2024-05-15T12:50:18.099 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 6.9920114670000002, 2024-05-15T12:50:18.099 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.975497795, 2024-05-15T12:50:18.099 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:50:18.099 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:18.100 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:50:18.100 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:50:18.100 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:50:18.100 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119641 2024-05-15T12:50:18.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.100 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:50:18.100 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.100 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:50:18.100 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.048535+0000", 2024-05-15T12:50:18.100 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.100 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.100 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:50:18.100 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.048535+0000", 2024-05-15T12:50:18.100 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.101 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:50:18.101 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.048536+0000", 2024-05-15T12:50:18.101 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4300000000000001e-07 2024-05-15T12:50:18.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.101 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:50:18.101 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.048541+0000", 2024-05-15T12:50:18.101 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7940000000000002e-06 2024-05-15T12:50:18.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.101 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:50:18.101 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.048558+0000", 2024-05-15T12:50:18.102 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6595e-05 2024-05-15T12:50:18.102 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.102 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.102 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:50:18.102 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.048566+0000", 2024-05-15T12:50:18.102 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7830000000000004e-06 2024-05-15T12:50:18.102 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.102 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.102 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.102 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.048628+0000", 2024-05-15T12:50:18.102 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1472999999999999e-05 2024-05-15T12:50:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:50:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.048638+0000", 2024-05-15T12:50:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6409999999999997e-06 2024-05-15T12:50:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.010043+0000", 2024-05-15T12:50:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.96140517800000003 2024-05-15T12:50:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.104 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:50:18.104 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.010156+0000", 2024-05-15T12:50:18.104 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000113428 2024-05-15T12:50:18.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.104 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.104 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,9", 2024-05-15T12:50:18.104 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.010261+0000", 2024-05-15T12:50:18.104 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000104378 2024-05-15T12:50:18.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.104 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.104 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.105 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.016857+0000", 2024-05-15T12:50:18.105 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0065966380000000002 2024-05-15T12:50:18.105 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.105 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.105 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:50:18.105 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.016867+0000", 2024-05-15T12:50:18.105 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8870000000000004e-06 2024-05-15T12:50:18.105 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.105 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.105 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.023923+0000", 2024-05-15T12:50:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.007056199 2024-05-15T12:50:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:50:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.023986+0000", 2024-05-15T12:50:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3058999999999994e-05 2024-05-15T12:50:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:50:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.024033+0000", 2024-05-15T12:50:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6798999999999997e-05 2024-05-15T12:50:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:50:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119663 2.a 2:5719cd00:::benchmark_data_smithi177_78104_object12530:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-05-15T12:50:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:50:11.076128+0000", 2024-05-15T12:50:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 6.9644189619999999, 2024-05-15T12:50:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.94806049000000003, 2024-05-15T12:50:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:50:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:50:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:50:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:50:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119663 2024-05-15T12:50:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:50:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:50:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.076128+0000", 2024-05-15T12:50:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:50:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.076128+0000", 2024-05-15T12:50:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:50:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.076130+0000", 2024-05-15T12:50:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.764e-06 2024-05-15T12:50:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:50:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.076139+0000", 2024-05-15T12:50:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2240000000000001e-06 2024-05-15T12:50:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:50:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.076152+0000", 2024-05-15T12:50:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2738000000000001e-05 2024-05-15T12:50:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:50:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.076161+0000", 2024-05-15T12:50:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2229999999999996e-06 2024-05-15T12:50:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.076242+0000", 2024-05-15T12:50:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0777000000000003e-05 2024-05-15T12:50:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:50:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.076252+0000", 2024-05-15T12:50:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0557000000000001e-05 2024-05-15T12:50:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.013480+0000", 2024-05-15T12:50:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.93722830499999998 2024-05-15T12:50:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:50:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.013612+0000", 2024-05-15T12:50:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013185200000000001 2024-05-15T12:50:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,9", 2024-05-15T12:50:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.013827+0000", 2024-05-15T12:50:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00021431999999999999 2024-05-15T12:50:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.016901+0000", 2024-05-15T12:50:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0030740250000000002 2024-05-15T12:50:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:50:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.020420+0000", 2024-05-15T12:50:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035195769999999999 2024-05-15T12:50:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.024097+0000", 2024-05-15T12:50:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003676449 2024-05-15T12:50:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:50:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.024145+0000", 2024-05-15T12:50:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8273999999999999e-05 2024-05-15T12:50:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:50:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.024188+0000", 2024-05-15T12:50:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3405000000000002e-05 2024-05-15T12:50:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:50:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119675 2.a 2:52e72d5d:::benchmark_data_smithi177_78104_object12542:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-05-15T12:50:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:50:11.086840+0000", 2024-05-15T12:50:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 6.9537062880000002, 2024-05-15T12:50:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.93750450299999999, 2024-05-15T12:50:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:50:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:50:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:50:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:50:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119675 2024-05-15T12:50:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:50:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:50:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.086840+0000", 2024-05-15T12:50:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:50:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.086840+0000", 2024-05-15T12:50:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:50:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.086842+0000", 2024-05-15T12:50:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.894e-06 2024-05-15T12:50:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:50:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.086849+0000", 2024-05-15T12:50:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.251e-06 2024-05-15T12:50:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:50:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.086862+0000", 2024-05-15T12:50:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3838e-05 2024-05-15T12:50:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:50:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.086872+0000", 2024-05-15T12:50:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7710000000000001e-06 2024-05-15T12:50:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.086942+0000", 2024-05-15T12:50:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9357999999999996e-05 2024-05-15T12:50:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:50:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.086953+0000", 2024-05-15T12:50:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1171e-05 2024-05-15T12:50:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.016935+0000", 2024-05-15T12:50:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.92998227700000002 2024-05-15T12:50:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:50:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.017011+0000", 2024-05-15T12:50:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5689e-05 2024-05-15T12:50:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,9", 2024-05-15T12:50:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.017087+0000", 2024-05-15T12:50:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6204000000000004e-05 2024-05-15T12:50:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.024239+0000", 2024-05-15T12:50:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0071524049999999997 2024-05-15T12:50:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:50:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.024248+0000", 2024-05-15T12:50:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1859999999999992e-06 2024-05-15T12:50:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.024271+0000", 2024-05-15T12:50:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3804000000000001e-05 2024-05-15T12:50:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:50:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.024306+0000", 2024-05-15T12:50:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4542000000000002e-05 2024-05-15T12:50:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:50:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.024345+0000", 2024-05-15T12:50:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9113000000000003e-05 2024-05-15T12:50:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:50:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119784 2.a 2:5248a020:::benchmark_data_smithi177_78104_object12651:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-05-15T12:50:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:50:11.223718+0000", 2024-05-15T12:50:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 6.8168288459999999, 2024-05-15T12:50:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.81142093999999998, 2024-05-15T12:50:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:50:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:50:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:50:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:50:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119784 2024-05-15T12:50:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:50:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:50:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.223718+0000", 2024-05-15T12:50:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:50:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.223718+0000", 2024-05-15T12:50:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:50:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.223719+0000", 2024-05-15T12:50:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5209999999999999e-06 2024-05-15T12:50:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:50:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.223727+0000", 2024-05-15T12:50:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1199999999999996e-06 2024-05-15T12:50:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:50:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.223738+0000", 2024-05-15T12:50:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1756999999999999e-05 2024-05-15T12:50:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:50:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.223746+0000", 2024-05-15T12:50:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4059999999999999e-06 2024-05-15T12:50:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.223810+0000", 2024-05-15T12:50:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4565000000000003e-05 2024-05-15T12:50:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:50:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.223817+0000", 2024-05-15T12:50:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.058e-06 2024-05-15T12:50:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.020470+0000", 2024-05-15T12:50:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.79665279200000005 2024-05-15T12:50:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:50:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.020586+0000", 2024-05-15T12:50:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011563199999999999 2024-05-15T12:50:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,9", 2024-05-15T12:50:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.020695+0000", 2024-05-15T12:50:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000109275 2024-05-15T12:50:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.024407+0000", 2024-05-15T12:50:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003711953 2024-05-15T12:50:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:50:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.024422+0000", 2024-05-15T12:50:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4673e-05 2024-05-15T12:50:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.035016+0000", 2024-05-15T12:50:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010594790999999999 2024-05-15T12:50:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:50:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.035076+0000", 2024-05-15T12:50:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9312999999999999e-05 2024-05-15T12:50:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:50:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.035139+0000", 2024-05-15T12:50:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3083999999999994e-05 2024-05-15T12:50:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:50:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119816 2.a 2:565d65fd:::benchmark_data_smithi177_78104_object12683:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-05-15T12:50:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:50:11.274343+0000", 2024-05-15T12:50:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 6.7662040320000001, 2024-05-15T12:50:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.76783549299999998, 2024-05-15T12:50:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:50:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:50:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:50:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:50:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119816 2024-05-15T12:50:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:50:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:50:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.274343+0000", 2024-05-15T12:50:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:50:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.274343+0000", 2024-05-15T12:50:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:50:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.274344+0000", 2024-05-15T12:50:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.327e-06 2024-05-15T12:50:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:50:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.274351+0000", 2024-05-15T12:50:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.063e-06 2024-05-15T12:50:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:50:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.274360+0000", 2024-05-15T12:50:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3109999999999995e-06 2024-05-15T12:50:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:50:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.274367+0000", 2024-05-15T12:50:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1129999999999997e-06 2024-05-15T12:50:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.274419+0000", 2024-05-15T12:50:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2435000000000002e-05 2024-05-15T12:50:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:50:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.274426+0000", 2024-05-15T12:50:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9450000000000004e-06 2024-05-15T12:50:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.024446+0000", 2024-05-15T12:50:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.75002002499999998 2024-05-15T12:50:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:50:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.024589+0000", 2024-05-15T12:50:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014268199999999999 2024-05-15T12:50:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,9", 2024-05-15T12:50:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.024693+0000", 2024-05-15T12:50:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000104526 2024-05-15T12:50:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.034961+0000", 2024-05-15T12:50:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010268279 2024-05-15T12:50:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.035221+0000", 2024-05-15T12:50:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00025979300000000002 2024-05-15T12:50:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:50:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.042102+0000", 2024-05-15T12:50:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0068810829999999996 2024-05-15T12:50:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:50:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.042144+0000", 2024-05-15T12:50:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1696999999999997e-05 2024-05-15T12:50:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:50:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.042178+0000", 2024-05-15T12:50:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4214000000000001e-05 2024-05-15T12:50:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:50:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119818 2.a 2:576b6356:::benchmark_data_smithi177_78104_object12685:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-05-15T12:50:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:50:11.274551+0000", 2024-05-15T12:50:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 6.7659961900000001, 2024-05-15T12:50:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.76777247699999995, 2024-05-15T12:50:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:50:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:50:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:50:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:50:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119818 2024-05-15T12:50:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:50:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:50:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.274551+0000", 2024-05-15T12:50:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:50:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.274551+0000", 2024-05-15T12:50:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:50:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.274551+0000", 2024-05-15T12:50:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4799999999999998e-07 2024-05-15T12:50:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:50:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.274554+0000", 2024-05-15T12:50:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2660000000000002e-06 2024-05-15T12:50:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:50:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.274564+0000", 2024-05-15T12:50:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2050000000000008e-06 2024-05-15T12:50:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:50:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.274568+0000", 2024-05-15T12:50:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6439999999999999e-06 2024-05-15T12:50:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.274605+0000", 2024-05-15T12:50:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6591000000000003e-05 2024-05-15T12:50:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:50:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.274610+0000", 2024-05-15T12:50:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9389999999999996e-06 2024-05-15T12:50:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.028214+0000", 2024-05-15T12:50:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.75360372099999995 2024-05-15T12:50:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:50:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.028311+0000", 2024-05-15T12:50:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6903000000000004e-05 2024-05-15T12:50:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,9", 2024-05-15T12:50:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.028440+0000", 2024-05-15T12:50:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000129152 2024-05-15T12:50:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.035182+0000", 2024-05-15T12:50:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.006742044 2024-05-15T12:50:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:50:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.042187+0000", 2024-05-15T12:50:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0070051560000000002 2024-05-15T12:50:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.042239+0000", 2024-05-15T12:50:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2261e-05 2024-05-15T12:50:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:50:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.042301+0000", 2024-05-15T12:50:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1449000000000001e-05 2024-05-15T12:50:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:50:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.042323+0000", 2024-05-15T12:50:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2498e-05 2024-05-15T12:50:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:50:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119839 2.a 2:5c693fb5:::benchmark_data_smithi177_78104_object12706:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-05-15T12:50:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:50:11.313146+0000", 2024-05-15T12:50:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 6.7274006650000002, 2024-05-15T12:50:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.72930229099999999, 2024-05-15T12:50:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:50:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:50:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:50:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:50:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119839 2024-05-15T12:50:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:50:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:50:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.313146+0000", 2024-05-15T12:50:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:50:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.313146+0000", 2024-05-15T12:50:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:50:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.313148+0000", 2024-05-15T12:50:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.533e-06 2024-05-15T12:50:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:50:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.313153+0000", 2024-05-15T12:50:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1189999999999996e-06 2024-05-15T12:50:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:50:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.313165+0000", 2024-05-15T12:50:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2340999999999999e-05 2024-05-15T12:50:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:50:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.313173+0000", 2024-05-15T12:50:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7389999999999999e-06 2024-05-15T12:50:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.313236+0000", 2024-05-15T12:50:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3521000000000004e-05 2024-05-15T12:50:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:50:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.313248+0000", 2024-05-15T12:50:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1559000000000001e-05 2024-05-15T12:50:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.031835+0000", 2024-05-15T12:50:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.71858677100000001 2024-05-15T12:50:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:50:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.031918+0000", 2024-05-15T12:50:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2890000000000001e-05 2024-05-15T12:50:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,9", 2024-05-15T12:50:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.031985+0000", 2024-05-15T12:50:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7625000000000004e-05 2024-05-15T12:50:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.035263+0000", 2024-05-15T12:50:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032775819999999998 2024-05-15T12:50:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.042362+0000", 2024-05-15T12:50:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0070990289999999998 2024-05-15T12:50:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:50:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.042372+0000", 2024-05-15T12:50:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6989999999999998e-06 2024-05-15T12:50:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:50:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.042411+0000", 2024-05-15T12:50:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9625000000000001e-05 2024-05-15T12:50:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:50:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.042448+0000", 2024-05-15T12:50:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7258000000000001e-05 2024-05-15T12:50:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:50:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119843 2.a 2:54bbb5e6:::benchmark_data_smithi177_78104_object12710:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-05-15T12:50:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:50:11.315152+0000", 2024-05-15T12:50:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 6.7253949999999998, 2024-05-15T12:50:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.72745872600000006, 2024-05-15T12:50:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:50:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:50:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:50:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:50:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119843 2024-05-15T12:50:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:50:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:50:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.315152+0000", 2024-05-15T12:50:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:50:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.315152+0000", 2024-05-15T12:50:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:50:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.315153+0000", 2024-05-15T12:50:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6279999999999999e-06 2024-05-15T12:50:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:50:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.315159+0000", 2024-05-15T12:50:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5319999999999997e-06 2024-05-15T12:50:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:50:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.315171+0000", 2024-05-15T12:50:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2160999999999999e-05 2024-05-15T12:50:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:50:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.315179+0000", 2024-05-15T12:50:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8630000000000008e-06 2024-05-15T12:50:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.315235+0000", 2024-05-15T12:50:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6245000000000001e-05 2024-05-15T12:50:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:50:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.315245+0000", 2024-05-15T12:50:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5990000000000008e-06 2024-05-15T12:50:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.035299+0000", 2024-05-15T12:50:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.720054211 2024-05-15T12:50:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:50:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.035427+0000", 2024-05-15T12:50:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012761399999999999 2024-05-15T12:50:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,9", 2024-05-15T12:50:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.035547+0000", 2024-05-15T12:50:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120386 2024-05-15T12:50:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:50:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.042458+0000", 2024-05-15T12:50:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0069112610000000001 2024-05-15T12:50:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.042516+0000", 2024-05-15T12:50:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8140999999999998e-05 2024-05-15T12:50:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.042559+0000", 2024-05-15T12:50:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2929999999999997e-05 2024-05-15T12:50:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:50:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.042588+0000", 2024-05-15T12:50:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9108999999999999e-05 2024-05-15T12:50:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:50:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.042611+0000", 2024-05-15T12:50:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2045999999999999e-05 2024-05-15T12:50:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:50:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119870 2.a 2:55ab654c:::benchmark_data_smithi177_78104_object12737:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-05-15T12:50:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:50:11.364706+0000", 2024-05-15T12:50:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 6.675841234, 2024-05-15T12:50:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.68902259300000002, 2024-05-15T12:50:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:50:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:50:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:50:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:50:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119870 2024-05-15T12:50:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:50:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:50:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.364706+0000", 2024-05-15T12:50:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:50:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.364706+0000", 2024-05-15T12:50:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:50:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.364707+0000", 2024-05-15T12:50:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4410000000000001e-06 2024-05-15T12:50:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:50:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.364715+0000", 2024-05-15T12:50:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1810000000000001e-06 2024-05-15T12:50:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:50:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.364728+0000", 2024-05-15T12:50:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3148e-05 2024-05-15T12:50:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:50:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.364736+0000", 2024-05-15T12:50:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1240000000000005e-06 2024-05-15T12:50:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.364836+0000", 2024-05-15T12:50:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9711999999999997e-05 2024-05-15T12:50:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:50:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.364848+0000", 2024-05-15T12:50:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2217e-05 2024-05-15T12:50:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.038693+0000", 2024-05-15T12:50:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.67384458300000005 2024-05-15T12:50:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:50:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.038811+0000", 2024-05-15T12:50:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011830800000000001 2024-05-15T12:50:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,9", 2024-05-15T12:50:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.038922+0000", 2024-05-15T12:50:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011103900000000001 2024-05-15T12:50:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:50:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.042617+0000", 2024-05-15T12:50:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0036951190000000002 2024-05-15T12:50:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.042651+0000", 2024-05-15T12:50:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3751999999999998e-05 2024-05-15T12:50:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.053627+0000", 2024-05-15T12:50:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010975393999999999 2024-05-15T12:50:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:50:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.053682+0000", 2024-05-15T12:50:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5374000000000002e-05 2024-05-15T12:50:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:50:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.053728+0000", 2024-05-15T12:50:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6201e-05 2024-05-15T12:50:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:50:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119872 2.a 2:5c1a4a40:::benchmark_data_smithi177_78104_object12739:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-05-15T12:50:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:50:11.368572+0000", 2024-05-15T12:50:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 6.6719752559999996, 2024-05-15T12:50:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.68527095999999998, 2024-05-15T12:50:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:50:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:50:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:50:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:50:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119872 2024-05-15T12:50:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:50:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:50:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.368572+0000", 2024-05-15T12:50:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:50:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.368572+0000", 2024-05-15T12:50:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:50:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.368573+0000", 2024-05-15T12:50:18.163 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7710000000000001e-06 2024-05-15T12:50:18.163 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.163 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.163 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:50:18.163 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.368580+0000", 2024-05-15T12:50:18.163 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.241e-06 2024-05-15T12:50:18.163 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.163 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.163 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:50:18.163 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.368597+0000", 2024-05-15T12:50:18.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6971999999999999e-05 2024-05-15T12:50:18.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:50:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.368606+0000", 2024-05-15T12:50:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7869999999999996e-06 2024-05-15T12:50:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.368673+0000", 2024-05-15T12:50:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6804999999999995e-05 2024-05-15T12:50:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:50:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.368683+0000", 2024-05-15T12:50:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0034000000000001e-05 2024-05-15T12:50:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.042689+0000", 2024-05-15T12:50:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.67400573100000005 2024-05-15T12:50:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:50:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.042770+0000", 2024-05-15T12:50:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0667999999999994e-05 2024-05-15T12:50:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,9", 2024-05-15T12:50:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.042852+0000", 2024-05-15T12:50:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2782999999999997e-05 2024-05-15T12:50:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.053535+0000", 2024-05-15T12:50:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010682695000000001 2024-05-15T12:50:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:50:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.053591+0000", 2024-05-15T12:50:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6150999999999998e-05 2024-05-15T12:50:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.053765+0000", 2024-05-15T12:50:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000173389 2024-05-15T12:50:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:50:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.053804+0000", 2024-05-15T12:50:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9830999999999999e-05 2024-05-15T12:50:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:50:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.053842+0000", 2024-05-15T12:50:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8102000000000002e-05 2024-05-15T12:50:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:50:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119874 2.a 2:55c229b4:::benchmark_data_smithi177_78104_object12741:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-05-15T12:50:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:50:11.375148+0000", 2024-05-15T12:50:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 6.6653986730000003, 2024-05-15T12:50:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.67880697999999995, 2024-05-15T12:50:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:50:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:50:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:50:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:50:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119874 2024-05-15T12:50:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:50:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:50:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.375148+0000", 2024-05-15T12:50:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:50:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.375148+0000", 2024-05-15T12:50:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:50:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.375150+0000", 2024-05-15T12:50:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.697e-06 2024-05-15T12:50:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:50:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.375156+0000", 2024-05-15T12:50:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0390000000000001e-06 2024-05-15T12:50:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:50:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.375173+0000", 2024-05-15T12:50:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6701000000000002e-05 2024-05-15T12:50:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:50:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.375198+0000", 2024-05-15T12:50:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5449e-05 2024-05-15T12:50:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.375269+0000", 2024-05-15T12:50:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1007000000000006e-05 2024-05-15T12:50:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:50:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.375283+0000", 2024-05-15T12:50:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3805000000000001e-05 2024-05-15T12:50:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.046313+0000", 2024-05-15T12:50:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.67103036800000004 2024-05-15T12:50:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:50:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.046434+0000", 2024-05-15T12:50:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000121107 2024-05-15T12:50:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,9", 2024-05-15T12:50:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.046549+0000", 2024-05-15T12:50:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000115072 2024-05-15T12:50:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.053580+0000", 2024-05-15T12:50:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0070308829999999999 2024-05-15T12:50:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:50:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.053596+0000", 2024-05-15T12:50:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5537e-05 2024-05-15T12:50:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.053881+0000", 2024-05-15T12:50:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00028479299999999998 2024-05-15T12:50:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:50:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.053922+0000", 2024-05-15T12:50:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0991000000000001e-05 2024-05-15T12:50:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:50:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.053955+0000", 2024-05-15T12:50:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3531000000000001e-05 2024-05-15T12:50:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:50:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119884 2.a 2:5be312c7:::benchmark_data_smithi177_78104_object12751:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-05-15T12:50:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:50:11.415558+0000", 2024-05-15T12:50:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 6.6249884850000003, 2024-05-15T12:50:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.65326022900000003, 2024-05-15T12:50:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:50:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:50:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:50:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:50:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119884 2024-05-15T12:50:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:50:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:50:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.415558+0000", 2024-05-15T12:50:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:50:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.415558+0000", 2024-05-15T12:50:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:50:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:50:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.415560+0000", 2024-05-15T12:50:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.265e-06 2024-05-15T12:50:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:50:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.415566+0000", 2024-05-15T12:50:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1500000000000004e-06 2024-05-15T12:50:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:50:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.415575+0000", 2024-05-15T12:50:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3239999999999992e-06 2024-05-15T12:50:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:50:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.415580+0000", 2024-05-15T12:50:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3199999999999999e-06 2024-05-15T12:50:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.415629+0000", 2024-05-15T12:50:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8925000000000003e-05 2024-05-15T12:50:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:50:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:11.415636+0000", 2024-05-15T12:50:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1550000000000001e-06 2024-05-15T12:50:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:50:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.050040+0000", 2024-05-15T12:50:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.63440378900000005 2024-05-15T12:50:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:50:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.050147+0000", 2024-05-15T12:50:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000107036 2024-05-15T12:50:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,9", 2024-05-15T12:50:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.050246+0000", 2024-05-15T12:50:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8523999999999994e-05 2024-05-15T12:50:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.061240+0000", 2024-05-15T12:50:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010994715 2024-05-15T12:50:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:50:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.061309+0000", 2024-05-15T12:50:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8116999999999999e-05 2024-05-15T12:50:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:50:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.068721+0000", 2024-05-15T12:50:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.007412231 2024-05-15T12:50:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:50:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.068771+0000", 2024-05-15T12:50:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.977e-05 2024-05-15T12:50:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:50:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:50:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:50:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:12.068819+0000", 2024-05-15T12:50:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7908000000000003e-05 2024-05-15T12:50:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:50:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:50:18.184 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:50:18.184 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T12:50:18.184 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:18.669 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:50:18.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:18.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:50:18.777 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:50:18.825 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:18.851 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:50:18.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:18.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:50:18.955 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:50:18.990 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:19.022 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:50:19.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:19.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:50:19.128 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:50:19.163 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:18 smithi120 ceph-mon[43961]: pgmap v481: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 14 active+clean; 5.6 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 434 op/s; 9651/276363 objects misplaced (3.492%); 17 MiB/s, 279 objects/s recovering 2024-05-15T12:50:19.163 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:18 smithi120 ceph-mon[43961]: 2.5 scrub starts 2024-05-15T12:50:19.163 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:18 smithi120 ceph-mon[43961]: osdmap e163: 12 total, 12 up, 11 in 2024-05-15T12:50:19.165 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:19.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:18 smithi177 ceph-mon[44303]: pgmap v481: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 14 active+clean; 5.6 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 434 op/s; 9651/276363 objects misplaced (3.492%); 17 MiB/s, 279 objects/s recovering 2024-05-15T12:50:19.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:18 smithi177 ceph-mon[44303]: 2.5 scrub starts 2024-05-15T12:50:19.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:18 smithi177 ceph-mon[44303]: osdmap e163: 12 total, 12 up, 11 in 2024-05-15T12:50:19.191 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:50:19.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:19.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:50:19.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:18 smithi027 ceph-mon[39795]: pgmap v481: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 14 active+clean; 5.6 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 434 op/s; 9651/276363 objects misplaced (3.492%); 17 MiB/s, 279 objects/s recovering 2024-05-15T12:50:19.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:18 smithi027 ceph-mon[39795]: 2.5 scrub starts 2024-05-15T12:50:19.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:18 smithi027 ceph-mon[39795]: osdmap e163: 12 total, 12 up, 11 in 2024-05-15T12:50:19.296 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:50:19.327 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:19.350 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:50:19.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:19.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:50:19.454 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:50:19.496 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:19.527 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:50:19.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:19.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:50:19.634 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:50:19.743 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:20.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:19 smithi027 ceph-mon[39795]: 2.2 scrub starts 2024-05-15T12:50:20.024 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:19 smithi027 ceph-mon[39795]: 2.5 scrub ok 2024-05-15T12:50:20.024 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:19 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2758596208' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T12:50:20.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:19 smithi177 ceph-mon[44303]: 2.2 scrub starts 2024-05-15T12:50:20.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:19 smithi177 ceph-mon[44303]: 2.5 scrub ok 2024-05-15T12:50:20.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:19 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2758596208' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T12:50:20.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:19 smithi120 ceph-mon[43961]: 2.2 scrub starts 2024-05-15T12:50:20.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:19 smithi120 ceph-mon[43961]: 2.5 scrub ok 2024-05-15T12:50:20.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:19 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2758596208' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T12:50:20.344 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.0 to 1 (10000) 2024-05-15T12:50:21.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:20 smithi177 ceph-mon[44303]: pgmap v483: 17 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+recovering+remapped, 14 active+clean; 5.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 599 op/s; 9368/271236 objects misplaced (3.454%); 7.2 MiB/s, 114 objects/s recovering 2024-05-15T12:50:21.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:20 smithi177 ceph-mon[44303]: 2.2 scrub ok 2024-05-15T12:50:21.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:20 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2758596208' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T12:50:21.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:20 smithi177 ceph-mon[44303]: osdmap e164: 12 total, 12 up, 11 in 2024-05-15T12:50:21.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:20 smithi027 ceph-mon[39795]: pgmap v483: 17 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+recovering+remapped, 14 active+clean; 5.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 599 op/s; 9368/271236 objects misplaced (3.454%); 7.2 MiB/s, 114 objects/s recovering 2024-05-15T12:50:21.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:20 smithi027 ceph-mon[39795]: 2.2 scrub ok 2024-05-15T12:50:21.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:20 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2758596208' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T12:50:21.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:20 smithi027 ceph-mon[39795]: osdmap e164: 12 total, 12 up, 11 in 2024-05-15T12:50:21.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:20 smithi120 ceph-mon[43961]: pgmap v483: 17 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+recovering+remapped, 14 active+clean; 5.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 599 op/s; 9368/271236 objects misplaced (3.454%); 7.2 MiB/s, 114 objects/s recovering 2024-05-15T12:50:21.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:20 smithi120 ceph-mon[43961]: 2.2 scrub ok 2024-05-15T12:50:21.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:20 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2758596208' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T12:50:21.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:20 smithi120 ceph-mon[43961]: osdmap e164: 12 total, 12 up, 11 in 2024-05-15T12:50:22.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:21 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:50:22.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:21 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:50:22.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:21 smithi027 ceph-mon[39795]: osdmap e165: 12 total, 12 up, 11 in 2024-05-15T12:50:22.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:21 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:50:22.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:21 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:50:22.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:21 smithi120 ceph-mon[43961]: osdmap e165: 12 total, 12 up, 11 in 2024-05-15T12:50:22.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:21 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:50:22.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:21 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:50:22.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:21 smithi177 ceph-mon[44303]: osdmap e165: 12 total, 12 up, 11 in 2024-05-15T12:50:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:22 smithi027 ceph-mon[39795]: pgmap v486: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 14 active+clean; 5.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 857 op/s; 8551/263913 objects misplaced (3.240%); 12 MiB/s, 184 objects/s recovering 2024-05-15T12:50:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:22 smithi027 ceph-mon[39795]: osdmap e166: 12 total, 12 up, 11 in 2024-05-15T12:50:23.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:22 smithi120 ceph-mon[43961]: pgmap v486: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 14 active+clean; 5.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 857 op/s; 8551/263913 objects misplaced (3.240%); 12 MiB/s, 184 objects/s recovering 2024-05-15T12:50:23.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:22 smithi120 ceph-mon[43961]: osdmap e166: 12 total, 12 up, 11 in 2024-05-15T12:50:23.331 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 1 1 2024-05-15T12:50:23.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:22 smithi177 ceph-mon[44303]: pgmap v486: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 14 active+clean; 5.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 857 op/s; 8551/263913 objects misplaced (3.240%); 12 MiB/s, 184 objects/s recovering 2024-05-15T12:50:23.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:22 smithi177 ceph-mon[44303]: osdmap e166: 12 total, 12 up, 11 in 2024-05-15T12:50:24.241 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:23 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:50:24.241 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:23 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:50:24.277 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-05-15T12:50:24.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:23 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:50:24.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:23 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:50:24.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:23 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:50:24.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:23 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:50:24.507 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:50:24.507 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:50:24.507 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:50:24.507 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:50:24.507 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:50:24.507 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:50:24.507 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:50:24.507 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:50:24.507 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:50:24.514 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:50:24.546 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:50:24.590 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:50:24.594 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:50:24.594 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:50:24.595 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:50:24.595 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:50:24.595 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:50:24.595 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:50:24.595 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:50:24.595 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:50:24.595 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:50:24.595 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:50:24.595 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:50:24.595 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:50:24.631 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:50:24.958 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:24 smithi027 ceph-mon[39795]: pgmap v488: 17 pgs: 2 active+remapped+backfilling, 1 active+recovering+remapped, 14 active+clean; 5.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 852 op/s; 7836/258354 objects misplaced (3.033%); 13 MiB/s, 201 objects/s recovering 2024-05-15T12:50:24.958 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:24 smithi027 ceph-mon[39795]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-05-15T12:50:24.958 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:24 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T12:50:24.959 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:24 smithi027 ceph-mon[39795]: osdmap e167: 12 total, 12 up, 11 in 2024-05-15T12:50:24.959 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:24 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2555867946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:50:24.959 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:24 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=1521683048,config=1059521453,health=3897446751,kv=652887828}) 2024-05-15T12:50:24.959 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:24 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {kv=4,logm=96} crc {kv=2599466666,logm=410752864}) 2024-05-15T12:50:24.959 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:24 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2309776181}) 2024-05-15T12:50:24.959 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:24 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=952196978}) 2024-05-15T12:50:24.959 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:24 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2762789687}) 2024-05-15T12:50:24.959 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:24 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936003465}) 2024-05-15T12:50:24.959 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:24 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2671964272}) 2024-05-15T12:50:24.960 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:24 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1839189699}) 2024-05-15T12:50:24.960 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:24 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2102224047}) 2024-05-15T12:50:24.960 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:24 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2448478298}) 2024-05-15T12:50:24.960 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:24 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1856342594}) 2024-05-15T12:50:24.960 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:24 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4256712967}) 2024-05-15T12:50:24.960 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:24 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3749378147}) 2024-05-15T12:50:24.960 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:24 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3769130080}) 2024-05-15T12:50:24.960 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:24 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=445539370}) 2024-05-15T12:50:24.961 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:24 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=269475152}) 2024-05-15T12:50:24.961 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:24 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2196281583}) 2024-05-15T12:50:24.963 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:50:24.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:24.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:50:25.072 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:50:25.143 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:24 smithi120 ceph-mon[43961]: pgmap v488: 17 pgs: 2 active+remapped+backfilling, 1 active+recovering+remapped, 14 active+clean; 5.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 852 op/s; 7836/258354 objects misplaced (3.033%); 13 MiB/s, 201 objects/s recovering 2024-05-15T12:50:25.143 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:24 smithi120 ceph-mon[43961]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-05-15T12:50:25.143 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:24 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T12:50:25.143 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:24 smithi120 ceph-mon[43961]: osdmap e167: 12 total, 12 up, 11 in 2024-05-15T12:50:25.143 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:24 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2555867946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:50:25.143 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:24 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=1521683048,config=1059521453,health=3897446751,kv=652887828}) 2024-05-15T12:50:25.143 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:24 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {kv=4,logm=96} crc {kv=2599466666,logm=410752864}) 2024-05-15T12:50:25.144 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:24 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2309776181}) 2024-05-15T12:50:25.144 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:24 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=952196978}) 2024-05-15T12:50:25.144 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:24 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2762789687}) 2024-05-15T12:50:25.144 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:24 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936003465}) 2024-05-15T12:50:25.144 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:24 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2671964272}) 2024-05-15T12:50:25.144 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:24 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1839189699}) 2024-05-15T12:50:25.144 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:24 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2102224047}) 2024-05-15T12:50:25.144 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:24 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2448478298}) 2024-05-15T12:50:25.144 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:24 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1856342594}) 2024-05-15T12:50:25.144 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:24 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4256712967}) 2024-05-15T12:50:25.144 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:24 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3749378147}) 2024-05-15T12:50:25.144 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:24 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3769130080}) 2024-05-15T12:50:25.144 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:24 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=445539370}) 2024-05-15T12:50:25.144 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:24 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=269475152}) 2024-05-15T12:50:25.144 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:24 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2196281583}) 2024-05-15T12:50:25.163 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:25.249 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:50:25.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:25.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:50:25.358 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:50:25.393 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:24 smithi177 ceph-mon[44303]: pgmap v488: 17 pgs: 2 active+remapped+backfilling, 1 active+recovering+remapped, 14 active+clean; 5.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 852 op/s; 7836/258354 objects misplaced (3.033%); 13 MiB/s, 201 objects/s recovering 2024-05-15T12:50:25.393 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:24 smithi177 ceph-mon[44303]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-05-15T12:50:25.393 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:24 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T12:50:25.393 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:24 smithi177 ceph-mon[44303]: osdmap e167: 12 total, 12 up, 11 in 2024-05-15T12:50:25.394 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:24 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2555867946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:50:25.394 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:24 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=1521683048,config=1059521453,health=3897446751,kv=652887828}) 2024-05-15T12:50:25.394 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:24 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {kv=4,logm=96} crc {kv=2599466666,logm=410752864}) 2024-05-15T12:50:25.394 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:24 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2309776181}) 2024-05-15T12:50:25.394 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:24 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=952196978}) 2024-05-15T12:50:25.394 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:24 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2762789687}) 2024-05-15T12:50:25.394 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:24 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936003465}) 2024-05-15T12:50:25.394 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:24 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2671964272}) 2024-05-15T12:50:25.394 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:24 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1839189699}) 2024-05-15T12:50:25.394 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:24 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2102224047}) 2024-05-15T12:50:25.394 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:24 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2448478298}) 2024-05-15T12:50:25.395 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:24 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1856342594}) 2024-05-15T12:50:25.395 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:24 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4256712967}) 2024-05-15T12:50:25.395 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:24 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3749378147}) 2024-05-15T12:50:25.395 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:24 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3769130080}) 2024-05-15T12:50:25.395 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:24 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=445539370}) 2024-05-15T12:50:25.395 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:24 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=269475152}) 2024-05-15T12:50:25.395 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:24 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2196281583}) 2024-05-15T12:50:25.396 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:25.429 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:50:25.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:25.432 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:50:25.533 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:50:25.564 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:25.583 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:50:25.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:25.586 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:50:25.687 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:50:25.765 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:25.865 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:50:25.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:25.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:50:25.970 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:50:26.006 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:26.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3795383125}) 2024-05-15T12:50:26.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=336187104}) 2024-05-15T12:50:26.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1062473678}) 2024-05-15T12:50:26.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3127181941}) 2024-05-15T12:50:26.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1403669791}) 2024-05-15T12:50:26.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2009925847}) 2024-05-15T12:50:26.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=930158798,mdsmap=3984425268,mgr=1522752385}) 2024-05-15T12:50:26.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mgr=11,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=72} crc {mgr=807179156,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=3526048306,mon_config_key=3254280539}) 2024-05-15T12:50:26.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1630816165}) 2024-05-15T12:50:26.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3043646949}) 2024-05-15T12:50:26.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=3,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=79} crc {mon_config_key=2714342900,monmap=2435304556,osd_metadata=1212035326,osd_pg_creating=3841215591,osdmap=142328752}) 2024-05-15T12:50:26.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=59} crc {osdmap=1750001293}) 2024-05-15T12:50:26.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3795383125}) 2024-05-15T12:50:26.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=336187104}) 2024-05-15T12:50:26.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1062473678}) 2024-05-15T12:50:26.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3127181941}) 2024-05-15T12:50:26.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1403669791}) 2024-05-15T12:50:26.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2009925847}) 2024-05-15T12:50:26.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=930158798,mdsmap=3984425268,mgr=1522752385}) 2024-05-15T12:50:26.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mgr=11,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=72} crc {mgr=807179156,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=3526048306,mon_config_key=3254280539}) 2024-05-15T12:50:26.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1630816165}) 2024-05-15T12:50:26.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3043646949}) 2024-05-15T12:50:26.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=3,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=79} crc {mon_config_key=2714342900,monmap=2435304556,osd_metadata=1212035326,osd_pg_creating=3841215591,osdmap=142328752}) 2024-05-15T12:50:26.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=59} crc {osdmap=1750001293}) 2024-05-15T12:50:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:50:26.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3795383125}) 2024-05-15T12:50:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=336187104}) 2024-05-15T12:50:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1062473678}) 2024-05-15T12:50:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3127181941}) 2024-05-15T12:50:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1403669791}) 2024-05-15T12:50:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2009925847}) 2024-05-15T12:50:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=930158798,mdsmap=3984425268,mgr=1522752385}) 2024-05-15T12:50:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mgr=11,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=72} crc {mgr=807179156,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=3526048306,mon_config_key=3254280539}) 2024-05-15T12:50:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1630816165}) 2024-05-15T12:50:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3043646949}) 2024-05-15T12:50:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=3,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=79} crc {mon_config_key=2714342900,monmap=2435304556,osd_metadata=1212035326,osd_pg_creating=3841215591,osdmap=142328752}) 2024-05-15T12:50:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=59} crc {osdmap=1750001293}) 2024-05-15T12:50:27.000 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:27 smithi027 ceph-mon[39795]: pgmap v490: 17 pgs: 1 active+recovering+remapped, 2 active+remapped+backfilling, 14 active+clean; 5.2 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s; 7508/253287 objects misplaced (2.964%); 16 MiB/s, 259 objects/s recovering 2024-05-15T12:50:27.000 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:27 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:50:27.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:27 smithi120 ceph-mon[43961]: pgmap v490: 17 pgs: 1 active+recovering+remapped, 2 active+remapped+backfilling, 14 active+clean; 5.2 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s; 7508/253287 objects misplaced (2.964%); 16 MiB/s, 259 objects/s recovering 2024-05-15T12:50:27.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:27 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:50:27.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:27 smithi177 ceph-mon[44303]: pgmap v490: 17 pgs: 1 active+recovering+remapped, 2 active+remapped+backfilling, 14 active+clean; 5.2 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s; 7508/253287 objects misplaced (2.964%); 16 MiB/s, 259 objects/s recovering 2024-05-15T12:50:27.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:27 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:50:27.661 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:50:27.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:27.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:50:27.765 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:50:27.794 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:27.915 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:50:28.238 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T12:50:28.266 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:28 smithi027 ceph-mon[39795]: 2.8 scrub starts 2024-05-15T12:50:28.266 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:28 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:50:28.266 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:28 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:50:28.266 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:28 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/347378076' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:50:28.266 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:28 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:50:28.266 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:28 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:50:28.266 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:28 smithi027 ceph-mon[39795]: 2.a scrub starts 2024-05-15T12:50:28.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:28 smithi177 ceph-mon[44303]: 2.8 scrub starts 2024-05-15T12:50:28.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:28 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:50:28.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:28 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:50:28.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:28 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/347378076' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:50:28.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:28 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:50:28.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:28 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:50:28.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:28 smithi177 ceph-mon[44303]: 2.a scrub starts 2024-05-15T12:50:28.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:28 smithi120 ceph-mon[43961]: 2.8 scrub starts 2024-05-15T12:50:28.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:28 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:50:28.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:28 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:50:28.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:28 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/347378076' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:50:28.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:28 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:50:28.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:28 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:50:28.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:28 smithi120 ceph-mon[43961]: 2.a scrub starts 2024-05-15T12:50:29.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:29 smithi120 ceph-mon[43961]: pgmap v491: 17 pgs: 1 active+recovering+remapped, 2 active+remapped+backfilling, 14 active+clean; 5.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 696 op/s; 6974/249990 objects misplaced (2.790%); 11 MiB/s, 178 objects/s recovering 2024-05-15T12:50:29.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:29 smithi120 ceph-mon[43961]: 2.d deep-scrub starts 2024-05-15T12:50:29.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:29 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:50:29.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:29 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/347378076' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:50:29.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:29 smithi120 ceph-mon[43961]: osdmap e168: 12 total, 12 up, 11 in 2024-05-15T12:50:29.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:29 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:50:29.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:29 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:50:29.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:29 smithi177 ceph-mon[44303]: pgmap v491: 17 pgs: 1 active+recovering+remapped, 2 active+remapped+backfilling, 14 active+clean; 5.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 696 op/s; 6974/249990 objects misplaced (2.790%); 11 MiB/s, 178 objects/s recovering 2024-05-15T12:50:29.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:29 smithi177 ceph-mon[44303]: 2.d deep-scrub starts 2024-05-15T12:50:29.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:29 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:50:29.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:29 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/347378076' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:50:29.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:29 smithi177 ceph-mon[44303]: osdmap e168: 12 total, 12 up, 11 in 2024-05-15T12:50:29.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:29 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:50:29.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:29 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:50:29.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:29 smithi027 ceph-mon[39795]: pgmap v491: 17 pgs: 1 active+recovering+remapped, 2 active+remapped+backfilling, 14 active+clean; 5.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 696 op/s; 6974/249990 objects misplaced (2.790%); 11 MiB/s, 178 objects/s recovering 2024-05-15T12:50:29.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:29 smithi027 ceph-mon[39795]: 2.d deep-scrub starts 2024-05-15T12:50:29.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:29 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:50:29.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:29 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/347378076' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:50:29.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:29 smithi027 ceph-mon[39795]: osdmap e168: 12 total, 12 up, 11 in 2024-05-15T12:50:29.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:29 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:50:29.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:29 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:50:29.764 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:50:30.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:30 smithi027 ceph-mon[39795]: osdmap e169: 12 total, 12 up, 11 in 2024-05-15T12:50:30.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:30 smithi120 ceph-mon[43961]: osdmap e169: 12 total, 12 up, 11 in 2024-05-15T12:50:30.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:30 smithi177 ceph-mon[44303]: osdmap e169: 12 total, 12 up, 11 in 2024-05-15T12:50:31.242 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.1 to 1 (10000) 2024-05-15T12:50:31.302 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T12:50:31.362 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:31 smithi027 ceph-mon[39795]: pgmap v494: 17 pgs: 1 active+recovering+remapped, 2 active+remapped+backfilling, 14 active+clean; 5.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 714 op/s; 6115/245487 objects misplaced (2.491%); 14 MiB/s, 226 objects/s recovering 2024-05-15T12:50:31.362 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:31 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2964633286' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T12:50:31.362 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:31 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T12:50:31.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:31 smithi120 ceph-mon[43961]: pgmap v494: 17 pgs: 1 active+recovering+remapped, 2 active+remapped+backfilling, 14 active+clean; 5.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 714 op/s; 6115/245487 objects misplaced (2.491%); 14 MiB/s, 226 objects/s recovering 2024-05-15T12:50:31.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:31 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2964633286' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T12:50:31.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:31 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T12:50:31.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:31 smithi177 ceph-mon[44303]: pgmap v494: 17 pgs: 1 active+recovering+remapped, 2 active+remapped+backfilling, 14 active+clean; 5.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 714 op/s; 6115/245487 objects misplaced (2.491%); 14 MiB/s, 226 objects/s recovering 2024-05-15T12:50:31.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:31 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2964633286' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T12:50:31.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:31 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T12:50:32.233 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 2 1 2024-05-15T12:50:32.357 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:50:32.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:32.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:50:32.462 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:50:32.476 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:32 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T12:50:32.476 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:32 smithi027 ceph-mon[39795]: osdmap e170: 12 total, 12 up, 11 in 2024-05-15T12:50:32.498 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:32 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T12:50:32.498 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:32 smithi177 ceph-mon[44303]: osdmap e170: 12 total, 12 up, 11 in 2024-05-15T12:50:32.498 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:32.525 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:50:32.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:32.528 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:50:32.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:32 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T12:50:32.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:32 smithi120 ceph-mon[43961]: osdmap e170: 12 total, 12 up, 11 in 2024-05-15T12:50:32.628 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:50:32.686 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:32.875 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:50:32.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:32.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:50:32.930 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:50:32.941 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:50:32.981 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:50:32.992 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:50:32.994 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T12:50:32.994 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:50:33.026 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:33.045 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:50:33.101 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:50:33.158 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:50:33.193 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:50:33.228 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:50:33.264 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:50:33.296 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:50:33.300 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:50:33.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:33.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:50:33.350 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:50:33.393 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:50:33.407 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:50:33.439 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:33.440 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:50:33.482 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:50:33.482 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:50:33.482 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:50:33.482 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:50:33.482 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:50:33.482 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:50:33.482 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:50:33.483 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:50:33.483 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:50:33.483 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:50:33.483 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:50:33.483 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:50:33.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:33 smithi027 ceph-mon[39795]: pgmap v496: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 13 active+clean; 4.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 784 op/s; 10461/239148 objects misplaced (4.374%); 16 MiB/s, 254 objects/s recovering 2024-05-15T12:50:33.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:33 smithi027 ceph-mon[39795]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T12:50:33.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:33 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:50:33.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:33 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:50:33.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:33 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:50:33.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:33 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3425784018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:50:33.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:33 smithi120 ceph-mon[43961]: pgmap v496: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 13 active+clean; 4.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 784 op/s; 10461/239148 objects misplaced (4.374%); 16 MiB/s, 254 objects/s recovering 2024-05-15T12:50:33.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:33 smithi120 ceph-mon[43961]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T12:50:33.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:33 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:50:33.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:33 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:50:33.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:33 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:50:33.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:33 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3425784018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:50:33.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:33 smithi177 ceph-mon[44303]: pgmap v496: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 13 active+clean; 4.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 784 op/s; 10461/239148 objects misplaced (4.374%); 16 MiB/s, 254 objects/s recovering 2024-05-15T12:50:33.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:33 smithi177 ceph-mon[44303]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T12:50:33.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:33 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:50:33.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:33 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:50:33.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:33 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:50:33.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:33 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3425784018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:50:33.799 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:50:33.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:33.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:50:33.809 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-05-15T12:50:33.904 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:50:33.944 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:33.968 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:50:33.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:33.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:50:34.077 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:50:34.161 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:34.205 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:50:34.224 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:50:34.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:34.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:50:34.330 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:50:34.390 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:34.557 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:50:34.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:34.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:50:34.665 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:50:34.700 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:34.727 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:50:34.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:34.733 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:50:34.834 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:50:34.895 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:35.162 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:50:35.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:35.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:50:35.271 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:50:35.330 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:35.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:35 smithi027 ceph-mon[39795]: pgmap v497: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 13 active+clean; 4.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 864 op/s; 9542/234420 objects misplaced (4.070%); 20 MiB/s, 315 objects/s recovering 2024-05-15T12:50:35.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:35 smithi120 ceph-mon[43961]: pgmap v497: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 13 active+clean; 4.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 864 op/s; 9542/234420 objects misplaced (4.070%); 20 MiB/s, 315 objects/s recovering 2024-05-15T12:50:35.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:35 smithi177 ceph-mon[44303]: pgmap v497: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 13 active+clean; 4.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 864 op/s; 9542/234420 objects misplaced (4.070%); 20 MiB/s, 315 objects/s recovering 2024-05-15T12:50:36.007 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:50:36.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:36.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:50:36.111 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:50:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:36.250 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T12:50:36.250 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T12:50:36.251 INFO:teuthology.orchestra.run.smithi120.stdout: "complaint_time": 30, 2024-05-15T12:50:36.251 INFO:teuthology.orchestra.run.smithi120.stdout: "num_blocked_ops": 0 2024-05-15T12:50:36.251 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T12:50:36.352 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:50:36.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:36.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:50:36.371 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:50:36.461 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:50:36.526 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:36.775 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:50:36.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:36.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:50:36.882 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:50:36.980 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:37.000 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-05-15T12:50:37.040 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:50:37.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:37.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:50:37.144 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:50:37.220 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:37.304 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T12:50:37.330 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:37 smithi027 ceph-mon[39795]: pgmap v498: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 13 active+clean; 4.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 749 op/s; 9323/230292 objects misplaced (4.048%); 16 MiB/s, 256 objects/s recovering 2024-05-15T12:50:37.330 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:37 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1284529424' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:50:37.330 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:37 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:50:37.395 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:50:37.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:37 smithi120 ceph-mon[43961]: pgmap v498: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 13 active+clean; 4.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 749 op/s; 9323/230292 objects misplaced (4.048%); 16 MiB/s, 256 objects/s recovering 2024-05-15T12:50:37.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:37 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1284529424' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:50:37.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:37 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:50:37.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:37 smithi177 ceph-mon[44303]: pgmap v498: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 13 active+clean; 4.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 749 op/s; 9323/230292 objects misplaced (4.048%); 16 MiB/s, 256 objects/s recovering 2024-05-15T12:50:37.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:37 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1284529424' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:50:37.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:37 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:50:37.708 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:50:37.709 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:50:37.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:37.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:50:37.819 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:50:37.848 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:38.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:38 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:50:38.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:38 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:50:38.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:38 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:50:38.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:38 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:50:38.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:38 smithi027 ceph-mon[39795]: osdmap e171: 12 total, 12 up, 11 in 2024-05-15T12:50:38.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:38 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:50:38.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:38 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:50:38.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:38 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:50:38.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:38 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:50:38.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:38 smithi177 ceph-mon[44303]: osdmap e171: 12 total, 12 up, 11 in 2024-05-15T12:50:38.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:38 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:50:38.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:38 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:50:38.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:38 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:50:38.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:38 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:50:38.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:38 smithi120 ceph-mon[43961]: osdmap e171: 12 total, 12 up, 11 in 2024-05-15T12:50:38.700 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:50:38.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:38.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:50:38.805 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:50:38.884 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:38.922 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:50:38.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:38.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:50:38.931 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:50:39.029 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:50:39.069 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:39.103 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:50:39.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:39.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:50:39.207 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:50:39.246 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:39.317 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:39 smithi120 ceph-mon[43961]: pgmap v500: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 13 active+clean; 4.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 746 op/s; 8745/227568 objects misplaced (3.843%); 14 MiB/s, 219 objects/s recovering 2024-05-15T12:50:39.342 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:50:39.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:39.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:50:39.396 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T12:50:39.396 INFO:teuthology.orchestra.run.smithi120.stdout: "size": 20, 2024-05-15T12:50:39.396 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 600, 2024-05-15T12:50:39.396 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [ 2024-05-15T12:50:39.396 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.396 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(osd.9.0:9585 2.d e170/110 2:b2f6fa8a:::benchmark_data_smithi177_78104_object51448:head v 170'9586, mlcod=170'9572)", 2024-05-15T12:50:39.396 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:50:34.208296+0000", 2024-05-15T12:50:39.396 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 5.1933638909999997, 2024-05-15T12:50:39.396 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.055687590000000002, 2024-05-15T12:50:39.397 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:50:39.397 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:39.397 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:50:39.397 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.397 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:50:39.397 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.208296+0000", 2024-05-15T12:50:39.397 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.397 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.397 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.397 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:50:39.397 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.208296+0000", 2024-05-15T12:50:39.397 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.397 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.397 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.398 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:50:39.398 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.208297+0000", 2024-05-15T12:50:39.398 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0200000000000001e-07 2024-05-15T12:50:39.398 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.398 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.398 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:50:39.398 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.208303+0000", 2024-05-15T12:50:39.398 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.5350000000000001e-06 2024-05-15T12:50:39.398 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.398 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.398 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:50:39.399 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.208314+0000", 2024-05-15T12:50:39.399 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0924000000000001e-05 2024-05-15T12:50:39.399 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.399 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.399 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:50:39.399 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.208321+0000", 2024-05-15T12:50:39.399 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.075e-06 2024-05-15T12:50:39.399 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.399 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.399 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:50:39.399 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.213456+0000", 2024-05-15T12:50:39.399 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0051341609999999999 2024-05-15T12:50:39.399 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.399 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.400 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:50:39.400 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.213469+0000", 2024-05-15T12:50:39.400 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.33e-05 2024-05-15T12:50:39.400 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.400 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.400 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:50:39.400 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.263952+0000", 2024-05-15T12:50:39.400 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.050482958000000001 2024-05-15T12:50:39.400 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.400 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.400 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:50:39.400 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.263984+0000", 2024-05-15T12:50:39.400 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.1835000000000003e-05 2024-05-15T12:50:39.400 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.401 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:50:39.401 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.401 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.401 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.401 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(osd.9.0:9586 2.d e170/110 2:b2f87b44:::benchmark_data_smithi177_78104_object36900:head v 170'9587, mlcod=170'9572)", 2024-05-15T12:50:39.401 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:50:34.212256+0000", 2024-05-15T12:50:39.401 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 5.1894041590000004, 2024-05-15T12:50:39.401 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.054389024000000001, 2024-05-15T12:50:39.401 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:50:39.401 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:39.401 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:50:39.401 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.401 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:50:39.402 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.212256+0000", 2024-05-15T12:50:39.402 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.402 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.402 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.402 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:50:39.402 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.212256+0000", 2024-05-15T12:50:39.402 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.402 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.402 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.402 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:50:39.402 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.212257+0000", 2024-05-15T12:50:39.402 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1299999999999998e-07 2024-05-15T12:50:39.402 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.403 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.403 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:50:39.403 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.212264+0000", 2024-05-15T12:50:39.403 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3599999999999998e-06 2024-05-15T12:50:39.403 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.403 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.403 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:50:39.403 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.212273+0000", 2024-05-15T12:50:39.403 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.003e-06 2024-05-15T12:50:39.403 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.403 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.403 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:50:39.403 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.212279+0000", 2024-05-15T12:50:39.404 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9549999999999997e-06 2024-05-15T12:50:39.404 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.404 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.404 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:50:39.404 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.217204+0000", 2024-05-15T12:50:39.404 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0049246710000000003 2024-05-15T12:50:39.404 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.404 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.404 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:50:39.404 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.217224+0000", 2024-05-15T12:50:39.404 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0465000000000001e-05 2024-05-15T12:50:39.404 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.404 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.404 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:50:39.405 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.266612+0000", 2024-05-15T12:50:39.405 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.049387777000000001 2024-05-15T12:50:39.405 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.405 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.405 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:50:39.405 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.266645+0000", 2024-05-15T12:50:39.405 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.2879999999999997e-05 2024-05-15T12:50:39.405 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.405 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:50:39.405 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.405 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.405 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.405 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.24562.0:138151 2.5 e170/110 2:ad6fb18c:::benchmark_data_smithi177_78104_object31018:head v 170'8595, mlcod=170'8591)", 2024-05-15T12:50:39.406 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:50:34.214011+0000", 2024-05-15T12:50:39.406 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 5.1876486210000001, 2024-05-15T12:50:39.406 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.050013580000000002, 2024-05-15T12:50:39.406 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:50:39.406 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:39.406 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:50:39.406 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.406 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:50:39.406 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.214011+0000", 2024-05-15T12:50:39.406 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.406 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.406 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.406 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:50:39.406 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.214011+0000", 2024-05-15T12:50:39.407 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.407 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.407 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.407 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:50:39.407 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.214012+0000", 2024-05-15T12:50:39.407 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9999999999999997e-07 2024-05-15T12:50:39.407 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.407 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.407 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:50:39.407 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.214019+0000", 2024-05-15T12:50:39.407 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8680000000000001e-06 2024-05-15T12:50:39.407 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.407 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.408 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:50:39.408 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.214030+0000", 2024-05-15T12:50:39.408 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1429e-05 2024-05-15T12:50:39.408 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.408 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.408 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:50:39.408 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.214037+0000", 2024-05-15T12:50:39.408 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9110000000000004e-06 2024-05-15T12:50:39.408 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.408 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.408 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:50:39.408 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.217278+0000", 2024-05-15T12:50:39.408 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0032411789999999998 2024-05-15T12:50:39.408 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.409 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.409 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:50:39.409 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.217298+0000", 2024-05-15T12:50:39.409 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.914e-05 2024-05-15T12:50:39.409 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.409 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.409 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:50:39.409 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.263987+0000", 2024-05-15T12:50:39.409 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.046689754 2024-05-15T12:50:39.409 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.409 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.409 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:50:39.409 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.264025+0000", 2024-05-15T12:50:39.410 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7598999999999997e-05 2024-05-15T12:50:39.410 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.410 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:50:39.410 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.410 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.410 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.410 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(osd.9.0:9589 2.d e170/110 2:b2fda54f:::benchmark_data_smithi177_78104_object36191:head v 170'9590, mlcod=170'9572)", 2024-05-15T12:50:39.410 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:50:34.224099+0000", 2024-05-15T12:50:39.410 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 5.1775611430000001, 2024-05-15T12:50:39.410 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.050663008000000002, 2024-05-15T12:50:39.410 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:50:39.410 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:39.410 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:50:39.411 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.411 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:50:39.411 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.224099+0000", 2024-05-15T12:50:39.411 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.411 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.411 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.411 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:50:39.411 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.224099+0000", 2024-05-15T12:50:39.411 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.411 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.411 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.411 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:50:39.412 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.224100+0000", 2024-05-15T12:50:39.412 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.6500000000000008e-07 2024-05-15T12:50:39.412 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.412 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.412 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:50:39.412 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.224106+0000", 2024-05-15T12:50:39.412 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1319999999999999e-06 2024-05-15T12:50:39.412 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.412 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.412 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:50:39.412 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.224116+0000", 2024-05-15T12:50:39.412 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.9599999999999995e-06 2024-05-15T12:50:39.412 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.412 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.413 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:50:39.413 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.224122+0000", 2024-05-15T12:50:39.413 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7270000000000004e-06 2024-05-15T12:50:39.413 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.413 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.413 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:50:39.413 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.233546+0000", 2024-05-15T12:50:39.413 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0094241140000000008 2024-05-15T12:50:39.413 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.413 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.413 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:50:39.413 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.233570+0000", 2024-05-15T12:50:39.413 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.4440999999999999e-05 2024-05-15T12:50:39.413 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.414 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.414 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:50:39.414 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.274714+0000", 2024-05-15T12:50:39.414 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.041143515999999998 2024-05-15T12:50:39.414 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.414 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.414 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:50:39.414 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:34.274762+0000", 2024-05-15T12:50:39.414 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8152999999999998e-05 2024-05-15T12:50:39.414 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.414 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:50:39.414 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.414 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.415 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.415 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(osd.9.0:9851 2.d e170/110 2:b390f4a8:::benchmark_data_smithi177_78104_object33480:head v 170'9852, mlcod=170'9846)", 2024-05-15T12:50:39.415 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:50:35.619658+0000", 2024-05-15T12:50:39.415 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 3.782001926, 2024-05-15T12:50:39.415 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.070279356000000001, 2024-05-15T12:50:39.415 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:50:39.415 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:39.415 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:50:39.415 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.415 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:50:39.415 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.619658+0000", 2024-05-15T12:50:39.415 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.415 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.416 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.416 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:50:39.416 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.619658+0000", 2024-05-15T12:50:39.416 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.416 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.416 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.416 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:50:39.416 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.619659+0000", 2024-05-15T12:50:39.416 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.78e-07 2024-05-15T12:50:39.416 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.416 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.416 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:50:39.416 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.619666+0000", 2024-05-15T12:50:39.417 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7730000000000002e-06 2024-05-15T12:50:39.417 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.417 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.417 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:50:39.417 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.619680+0000", 2024-05-15T12:50:39.417 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.449e-05 2024-05-15T12:50:39.417 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.417 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.417 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:50:39.417 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.619687+0000", 2024-05-15T12:50:39.417 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0709999999999997e-06 2024-05-15T12:50:39.417 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.417 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.418 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:50:39.418 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.622597+0000", 2024-05-15T12:50:39.418 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0029095890000000002 2024-05-15T12:50:39.418 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.418 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.418 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:50:39.418 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.622616+0000", 2024-05-15T12:50:39.418 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9048000000000001e-05 2024-05-15T12:50:39.418 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.418 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.418 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:50:39.418 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.689899+0000", 2024-05-15T12:50:39.418 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.067283180999999997 2024-05-15T12:50:39.418 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.419 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.419 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:50:39.419 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.689937+0000", 2024-05-15T12:50:39.419 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.8325999999999999e-05 2024-05-15T12:50:39.419 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.419 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:50:39.419 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.419 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.419 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.419 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(osd.9.0:9852 2.d e170/110 2:b3923f06:::benchmark_data_smithi177_78104_object88011:head v 170'9853, mlcod=170'9846)", 2024-05-15T12:50:39.419 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:50:35.622709+0000", 2024-05-15T12:50:39.419 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 3.77895102, 2024-05-15T12:50:39.419 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.067249639999999999, 2024-05-15T12:50:39.420 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:50:39.420 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:39.420 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:50:39.420 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.420 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:50:39.420 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.622709+0000", 2024-05-15T12:50:39.420 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.420 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.420 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.420 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:50:39.420 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.622709+0000", 2024-05-15T12:50:39.420 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.420 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.420 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.421 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:50:39.421 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.622710+0000", 2024-05-15T12:50:39.421 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.5300000000000003e-07 2024-05-15T12:50:39.421 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.421 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.421 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:50:39.421 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.622718+0000", 2024-05-15T12:50:39.421 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9240000000000007e-06 2024-05-15T12:50:39.421 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.421 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.421 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:50:39.421 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.622732+0000", 2024-05-15T12:50:39.421 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4440000000000001e-05 2024-05-15T12:50:39.421 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.422 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.422 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:50:39.422 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.622740+0000", 2024-05-15T12:50:39.422 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3200000000000002e-06 2024-05-15T12:50:39.422 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.422 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.422 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:50:39.422 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.626996+0000", 2024-05-15T12:50:39.422 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0042562709999999998 2024-05-15T12:50:39.422 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.422 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.422 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:50:39.422 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.627016+0000", 2024-05-15T12:50:39.422 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9981000000000001e-05 2024-05-15T12:50:39.423 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.423 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.423 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:50:39.423 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.689942+0000", 2024-05-15T12:50:39.423 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.062925755999999999 2024-05-15T12:50:39.423 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.423 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.423 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:50:39.423 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.689959+0000", 2024-05-15T12:50:39.423 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7095000000000001e-05 2024-05-15T12:50:39.423 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.423 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:50:39.423 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.423 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.423 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.424 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(osd.9.0:9853 2.d e170/110 2:b39271bf:::benchmark_data_smithi177_78104_object93300:head v 170'9854, mlcod=170'9846)", 2024-05-15T12:50:39.424 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:50:35.626861+0000", 2024-05-15T12:50:39.424 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 3.7747988810000002, 2024-05-15T12:50:39.424 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.063114156000000005, 2024-05-15T12:50:39.424 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:50:39.424 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:39.424 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:50:39.424 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.424 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:50:39.424 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.626861+0000", 2024-05-15T12:50:39.424 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.424 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.424 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.424 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:50:39.425 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.626861+0000", 2024-05-15T12:50:39.425 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.425 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.425 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.425 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:50:39.425 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.626862+0000", 2024-05-15T12:50:39.425 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2999999999999999e-07 2024-05-15T12:50:39.425 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.425 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.425 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:50:39.425 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.626870+0000", 2024-05-15T12:50:39.425 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9920000000000007e-06 2024-05-15T12:50:39.425 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.425 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.425 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:50:39.426 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.626886+0000", 2024-05-15T12:50:39.426 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6283e-05 2024-05-15T12:50:39.426 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.426 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.426 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:50:39.426 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.626894+0000", 2024-05-15T12:50:39.426 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1110000000000008e-06 2024-05-15T12:50:39.426 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.426 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.426 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:50:39.426 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.631427+0000", 2024-05-15T12:50:39.427 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0045329100000000002 2024-05-15T12:50:39.427 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.427 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.427 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:50:39.427 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.631441+0000", 2024-05-15T12:50:39.427 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3884e-05 2024-05-15T12:50:39.427 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.427 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.427 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:50:39.427 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.689961+0000", 2024-05-15T12:50:39.427 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.058519996999999997 2024-05-15T12:50:39.427 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.427 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.428 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:50:39.428 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.689975+0000", 2024-05-15T12:50:39.428 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4149e-05 2024-05-15T12:50:39.428 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.428 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:50:39.428 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.428 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.428 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.428 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(osd.9.0:9854 2.d e170/110 2:b393f104:::benchmark_data_smithi177_78104_object62448:head v 170'9855, mlcod=170'9846)", 2024-05-15T12:50:39.428 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:50:35.629925+0000", 2024-05-15T12:50:39.428 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 3.7717346329999999, 2024-05-15T12:50:39.428 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.060065440999999997, 2024-05-15T12:50:39.428 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:50:39.428 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:39.428 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:50:39.429 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.429 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:50:39.429 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.629925+0000", 2024-05-15T12:50:39.429 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.429 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.429 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.429 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:50:39.429 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.629925+0000", 2024-05-15T12:50:39.429 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.429 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.429 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.429 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:50:39.429 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.629926+0000", 2024-05-15T12:50:39.430 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7199999999999997e-07 2024-05-15T12:50:39.430 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.430 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.430 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:50:39.430 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.629934+0000", 2024-05-15T12:50:39.430 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2560000000000002e-06 2024-05-15T12:50:39.430 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.430 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.430 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:50:39.430 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.629949+0000", 2024-05-15T12:50:39.430 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4905e-05 2024-05-15T12:50:39.430 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.430 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.431 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:50:39.431 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.629958+0000", 2024-05-15T12:50:39.431 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.3969999999999994e-06 2024-05-15T12:50:39.431 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.431 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.431 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:50:39.431 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.635586+0000", 2024-05-15T12:50:39.431 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0056279590000000001 2024-05-15T12:50:39.431 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.431 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.431 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:50:39.431 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.635609+0000", 2024-05-15T12:50:39.431 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.3003e-05 2024-05-15T12:50:39.432 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.432 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.432 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:50:39.432 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.689978+0000", 2024-05-15T12:50:39.432 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.054368839000000002 2024-05-15T12:50:39.432 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.432 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.432 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:50:39.432 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.689991+0000", 2024-05-15T12:50:39.432 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3210000000000001e-05 2024-05-15T12:50:39.432 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.432 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:50:39.433 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.433 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.433 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.433 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.24562.0:139208 2.5 e170/110 2:a80fb3b0:::benchmark_data_smithi177_78104_object32075:head v 170'8674, mlcod=170'8670)", 2024-05-15T12:50:39.433 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:50:35.632499+0000", 2024-05-15T12:50:39.433 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 3.7691605560000001, 2024-05-15T12:50:39.433 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.060422312999999998, 2024-05-15T12:50:39.433 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:50:39.433 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:39.433 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:50:39.433 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.433 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:50:39.433 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.632499+0000", 2024-05-15T12:50:39.433 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.433 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.434 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.434 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:50:39.434 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.632499+0000", 2024-05-15T12:50:39.434 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.434 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.434 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.434 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:50:39.434 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.632500+0000", 2024-05-15T12:50:39.434 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0599999999999999e-07 2024-05-15T12:50:39.434 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.434 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.434 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:50:39.434 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.632509+0000", 2024-05-15T12:50:39.434 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2390000000000002e-06 2024-05-15T12:50:39.435 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.435 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.435 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:50:39.435 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.632523+0000", 2024-05-15T12:50:39.435 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4654999999999999e-05 2024-05-15T12:50:39.435 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.435 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.435 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:50:39.435 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.632531+0000", 2024-05-15T12:50:39.435 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1259999999999998e-06 2024-05-15T12:50:39.435 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.435 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.435 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:50:39.435 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.639524+0000", 2024-05-15T12:50:39.435 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0069925960000000002 2024-05-15T12:50:39.436 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.436 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.436 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:50:39.436 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.639546+0000", 2024-05-15T12:50:39.436 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2237000000000001e-05 2024-05-15T12:50:39.436 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.436 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.436 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:50:39.436 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.692877+0000", 2024-05-15T12:50:39.436 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.053331049999999998 2024-05-15T12:50:39.436 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.436 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.436 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:50:39.436 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.692922+0000", 2024-05-15T12:50:39.436 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4503999999999999e-05 2024-05-15T12:50:39.436 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.437 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:50:39.437 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.437 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.437 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.437 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(osd.9.0:9855 2.d e170/110 2:b394ec14:::benchmark_data_smithi177_78104_object68127:head v 170'9856, mlcod=170'9846)", 2024-05-15T12:50:39.437 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:50:35.633840+0000", 2024-05-15T12:50:39.437 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 3.7678197579999999, 2024-05-15T12:50:39.437 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.059342087000000002, 2024-05-15T12:50:39.437 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:50:39.437 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:39.437 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:50:39.437 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.437 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:50:39.437 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.633840+0000", 2024-05-15T12:50:39.438 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.438 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.438 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.438 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:50:39.438 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.633840+0000", 2024-05-15T12:50:39.438 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.438 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.438 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.438 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:50:39.438 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.633841+0000", 2024-05-15T12:50:39.438 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.4e-07 2024-05-15T12:50:39.439 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.439 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.439 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:50:39.439 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.633849+0000", 2024-05-15T12:50:39.439 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7810000000000005e-06 2024-05-15T12:50:39.439 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.439 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.439 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:50:39.439 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.633863+0000", 2024-05-15T12:50:39.439 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4168000000000001e-05 2024-05-15T12:50:39.439 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.439 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.439 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:50:39.439 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.633871+0000", 2024-05-15T12:50:39.439 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.6429999999999995e-06 2024-05-15T12:50:39.440 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.440 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.440 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:50:39.440 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.642761+0000", 2024-05-15T12:50:39.440 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0088908100000000007 2024-05-15T12:50:39.440 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.440 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.440 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:50:39.440 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.642789+0000", 2024-05-15T12:50:39.440 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.7396999999999999e-05 2024-05-15T12:50:39.440 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.440 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.440 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:50:39.440 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.693149+0000", 2024-05-15T12:50:39.440 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.050359721000000003 2024-05-15T12:50:39.440 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.441 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.441 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:50:39.441 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.693182+0000", 2024-05-15T12:50:39.441 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.3726999999999997e-05 2024-05-15T12:50:39.441 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.441 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:50:39.441 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.441 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.441 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.441 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(osd.9.0:9856 2.d e170/110 2:b3962985:::benchmark_data_smithi177_78104_object74909:head v 170'9857, mlcod=170'9846)", 2024-05-15T12:50:39.441 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:50:35.638015+0000", 2024-05-15T12:50:39.441 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 3.763644663, 2024-05-15T12:50:39.441 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.055186849000000003, 2024-05-15T12:50:39.441 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:50:39.441 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:39.441 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:50:39.442 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.442 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:50:39.442 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.638015+0000", 2024-05-15T12:50:39.442 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.442 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.442 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.442 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:50:39.442 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.638015+0000", 2024-05-15T12:50:39.442 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.442 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.442 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.442 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:50:39.442 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.638017+0000", 2024-05-15T12:50:39.443 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.277e-06 2024-05-15T12:50:39.443 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.443 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.443 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:50:39.443 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.638027+0000", 2024-05-15T12:50:39.443 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0158e-05 2024-05-15T12:50:39.443 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.443 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.443 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:50:39.443 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.638044+0000", 2024-05-15T12:50:39.443 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7631000000000001e-05 2024-05-15T12:50:39.443 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.443 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.443 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:50:39.444 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.638055+0000", 2024-05-15T12:50:39.444 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0835e-05 2024-05-15T12:50:39.444 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.444 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.444 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:50:39.444 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.647312+0000", 2024-05-15T12:50:39.444 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0092571540000000001 2024-05-15T12:50:39.444 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.444 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.444 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:50:39.444 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.647336+0000", 2024-05-15T12:50:39.444 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.3609e-05 2024-05-15T12:50:39.444 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.444 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.445 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:50:39.445 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.693185+0000", 2024-05-15T12:50:39.445 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.045849419000000002 2024-05-15T12:50:39.445 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.445 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.445 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:50:39.445 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.693202+0000", 2024-05-15T12:50:39.445 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6765999999999998e-05 2024-05-15T12:50:39.445 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.445 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:50:39.445 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.445 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.445 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.445 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(osd.9.0:9857 2.d e170/110 2:b396e8c5:::benchmark_data_smithi177_78104_object45291:head v 170'9858, mlcod=170'9846)", 2024-05-15T12:50:39.445 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:50:35.641850+0000", 2024-05-15T12:50:39.446 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 3.7598102349999998, 2024-05-15T12:50:39.446 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.055383934000000003, 2024-05-15T12:50:39.446 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:50:39.446 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:39.446 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:50:39.446 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.446 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:50:39.446 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.641850+0000", 2024-05-15T12:50:39.446 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.446 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.446 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.446 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:50:39.446 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.641850+0000", 2024-05-15T12:50:39.446 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.446 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.447 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.447 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:50:39.447 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.641851+0000", 2024-05-15T12:50:39.447 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.9300000000000006e-07 2024-05-15T12:50:39.447 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.447 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.447 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:50:39.447 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.641859+0000", 2024-05-15T12:50:39.447 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.4870000000000002e-06 2024-05-15T12:50:39.447 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.447 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.447 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:50:39.447 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.641873+0000", 2024-05-15T12:50:39.447 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.358e-05 2024-05-15T12:50:39.447 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.448 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.448 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:50:39.448 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.641881+0000", 2024-05-15T12:50:39.448 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.6260000000000001e-06 2024-05-15T12:50:39.448 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.448 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.448 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:50:39.448 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.655768+0000", 2024-05-15T12:50:39.448 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.013886157999999999 2024-05-15T12:50:39.448 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.448 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.448 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:50:39.449 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.655792+0000", 2024-05-15T12:50:39.449 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.425e-05 2024-05-15T12:50:39.449 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.449 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.449 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:50:39.449 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.697183+0000", 2024-05-15T12:50:39.449 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.041390892999999998 2024-05-15T12:50:39.449 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.449 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.449 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:50:39.449 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.697234+0000", 2024-05-15T12:50:39.449 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0946999999999999e-05 2024-05-15T12:50:39.449 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.449 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:50:39.449 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.449 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.450 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.450 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(osd.9.0:9858 2.d e170/110 2:b3972515:::benchmark_data_smithi177_78104_object38117:head v 170'9859, mlcod=170'9846)", 2024-05-15T12:50:39.450 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:50:35.646163+0000", 2024-05-15T12:50:39.450 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 3.755496618, 2024-05-15T12:50:39.450 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.051094342000000001, 2024-05-15T12:50:39.450 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:50:39.450 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:39.450 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:50:39.450 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.450 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:50:39.450 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.646163+0000", 2024-05-15T12:50:39.450 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.450 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.450 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.450 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:50:39.451 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.646163+0000", 2024-05-15T12:50:39.451 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.451 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.451 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.451 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:50:39.451 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.646165+0000", 2024-05-15T12:50:39.451 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1319999999999999e-06 2024-05-15T12:50:39.451 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.451 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.451 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:50:39.451 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.646172+0000", 2024-05-15T12:50:39.451 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.1980000000000004e-06 2024-05-15T12:50:39.451 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.451 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.451 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:50:39.451 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.646182+0000", 2024-05-15T12:50:39.452 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0532e-05 2024-05-15T12:50:39.452 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.452 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.452 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:50:39.452 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.646189+0000", 2024-05-15T12:50:39.452 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9619999999999996e-06 2024-05-15T12:50:39.452 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.452 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.452 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:50:39.452 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.659821+0000", 2024-05-15T12:50:39.452 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.01363194 2024-05-15T12:50:39.452 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.452 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.452 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:50:39.452 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.659841+0000", 2024-05-15T12:50:39.453 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0140999999999999e-05 2024-05-15T12:50:39.453 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.453 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.453 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:50:39.453 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.697238+0000", 2024-05-15T12:50:39.453 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.037396566999999999 2024-05-15T12:50:39.453 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.453 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.453 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:50:39.453 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:35.697258+0000", 2024-05-15T12:50:39.453 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9870000000000001e-05 2024-05-15T12:50:39.453 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.453 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:50:39.453 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.453 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.453 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.453 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(osd.9.0:10132 2.d e170/110 2:b44553d6:::benchmark_data_smithi177_78104_object57886:head v 170'10133, mlcod=170'10114)", 2024-05-15T12:50:39.454 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:50:37.266752+0000", 2024-05-15T12:50:39.454 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 2.1349075750000002, 2024-05-15T12:50:39.454 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.058646983, 2024-05-15T12:50:39.454 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:50:39.454 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:39.454 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:50:39.454 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.454 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:50:39.454 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.266752+0000", 2024-05-15T12:50:39.454 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.454 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.454 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.454 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:50:39.455 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.266752+0000", 2024-05-15T12:50:39.455 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.455 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.455 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.455 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:50:39.455 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.266753+0000", 2024-05-15T12:50:39.455 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9299999999999997e-07 2024-05-15T12:50:39.455 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.455 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.455 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:50:39.455 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.266760+0000", 2024-05-15T12:50:39.455 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0099999999999998e-06 2024-05-15T12:50:39.455 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.455 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.455 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:50:39.456 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.266771+0000", 2024-05-15T12:50:39.456 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0382e-05 2024-05-15T12:50:39.456 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.456 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.456 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:50:39.456 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.266779+0000", 2024-05-15T12:50:39.456 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2689999999999999e-06 2024-05-15T12:50:39.456 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.456 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.456 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:50:39.456 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.281776+0000", 2024-05-15T12:50:39.456 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.014997421 2024-05-15T12:50:39.457 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.457 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.457 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:50:39.457 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.281799+0000", 2024-05-15T12:50:39.457 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2809000000000001e-05 2024-05-15T12:50:39.457 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.457 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.457 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:50:39.457 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.325329+0000", 2024-05-15T12:50:39.457 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.043529619999999998 2024-05-15T12:50:39.457 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.457 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.457 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:50:39.458 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.325399+0000", 2024-05-15T12:50:39.458 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0679000000000005e-05 2024-05-15T12:50:39.458 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.458 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:50:39.458 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.458 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.458 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.458 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(osd.9.0:10133 2.d e170/110 2:b44569a4:::benchmark_data_smithi177_78104_object74783:head v 170'10134, mlcod=170'10114)", 2024-05-15T12:50:39.458 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:50:37.270442+0000", 2024-05-15T12:50:39.458 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 2.1312177050000001, 2024-05-15T12:50:39.458 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.054976323000000001, 2024-05-15T12:50:39.458 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:50:39.458 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:39.459 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:50:39.459 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.459 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:50:39.459 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.270442+0000", 2024-05-15T12:50:39.459 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.459 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.459 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.459 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:50:39.459 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.270442+0000", 2024-05-15T12:50:39.459 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.459 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.459 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.459 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:50:39.459 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.270443+0000", 2024-05-15T12:50:39.460 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.6099999999999999e-07 2024-05-15T12:50:39.460 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.460 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.460 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:50:39.460 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.270451+0000", 2024-05-15T12:50:39.460 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.5859999999999999e-06 2024-05-15T12:50:39.460 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.460 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.460 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:50:39.460 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.270461+0000", 2024-05-15T12:50:39.460 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0394000000000001e-05 2024-05-15T12:50:39.460 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.460 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.460 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:50:39.461 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.270467+0000", 2024-05-15T12:50:39.461 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8189999999999997e-06 2024-05-15T12:50:39.461 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.461 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.461 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:50:39.461 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.285264+0000", 2024-05-15T12:50:39.461 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.014796604 2024-05-15T12:50:39.461 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.461 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.461 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:50:39.461 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.285276+0000", 2024-05-15T12:50:39.461 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2554e-05 2024-05-15T12:50:39.461 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.461 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.462 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:50:39.462 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.325403+0000", 2024-05-15T12:50:39.462 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.040126863999999998 2024-05-15T12:50:39.462 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.462 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.462 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:50:39.462 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.325419+0000", 2024-05-15T12:50:39.462 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5641000000000002e-05 2024-05-15T12:50:39.462 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.462 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:50:39.462 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.462 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.462 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.463 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(osd.9.0:10134 2.d e170/110 2:b4464636:::benchmark_data_smithi177_78104_object42468:head v 170'10135, mlcod=170'10114)", 2024-05-15T12:50:39.463 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:50:37.273214+0000", 2024-05-15T12:50:39.463 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 2.128445718, 2024-05-15T12:50:39.463 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.063160114000000003, 2024-05-15T12:50:39.463 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:50:39.463 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:39.463 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:50:39.463 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.463 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:50:39.463 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.273214+0000", 2024-05-15T12:50:39.463 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.463 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.463 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.463 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:50:39.464 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.273214+0000", 2024-05-15T12:50:39.464 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.464 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.464 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.464 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:50:39.464 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.273215+0000", 2024-05-15T12:50:39.464 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.9899999999999999e-07 2024-05-15T12:50:39.464 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.464 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.464 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:50:39.464 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.273222+0000", 2024-05-15T12:50:39.464 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7070000000000003e-06 2024-05-15T12:50:39.464 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.465 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.465 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:50:39.465 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.273235+0000", 2024-05-15T12:50:39.465 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2935000000000001e-05 2024-05-15T12:50:39.465 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.465 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.465 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:50:39.465 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.273242+0000", 2024-05-15T12:50:39.465 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2189999999999998e-06 2024-05-15T12:50:39.465 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.465 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.465 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:50:39.465 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.288382+0000", 2024-05-15T12:50:39.465 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.015139878000000001 2024-05-15T12:50:39.465 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.465 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.466 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:50:39.466 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.288396+0000", 2024-05-15T12:50:39.466 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4137e-05 2024-05-15T12:50:39.466 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.466 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.466 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:50:39.466 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.336329+0000", 2024-05-15T12:50:39.466 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.047933335000000001 2024-05-15T12:50:39.466 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.466 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.466 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:50:39.466 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.336374+0000", 2024-05-15T12:50:39.466 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.5003999999999998e-05 2024-05-15T12:50:39.467 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.467 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:50:39.467 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.467 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.467 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.467 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(osd.9.0:10135 2.d e170/110 2:b447186c:::benchmark_data_smithi177_78104_object73359:head v 170'10136, mlcod=170'10114)", 2024-05-15T12:50:39.467 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:50:37.275308+0000", 2024-05-15T12:50:39.467 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 2.1263521019999998, 2024-05-15T12:50:39.467 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.061092263000000001, 2024-05-15T12:50:39.467 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:50:39.467 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:39.467 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:50:39.467 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.468 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:50:39.468 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.275308+0000", 2024-05-15T12:50:39.468 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.468 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.468 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.468 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:50:39.468 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.275308+0000", 2024-05-15T12:50:39.468 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.468 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.468 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.468 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:50:39.468 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.275309+0000", 2024-05-15T12:50:39.468 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1900000000000001e-07 2024-05-15T12:50:39.469 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.469 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.469 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:50:39.469 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.275316+0000", 2024-05-15T12:50:39.469 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.17e-06 2024-05-15T12:50:39.469 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.469 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.469 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:50:39.469 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.275328+0000", 2024-05-15T12:50:39.469 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2155e-05 2024-05-15T12:50:39.469 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.469 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.469 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:50:39.470 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.275336+0000", 2024-05-15T12:50:39.470 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9650000000000008e-06 2024-05-15T12:50:39.470 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.470 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.470 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:50:39.470 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.292157+0000", 2024-05-15T12:50:39.470 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.016821165999999999 2024-05-15T12:50:39.470 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.470 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.470 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:50:39.470 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.292178+0000", 2024-05-15T12:50:39.470 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0611e-05 2024-05-15T12:50:39.471 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.471 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.471 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:50:39.471 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.336377+0000", 2024-05-15T12:50:39.471 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.044199062999999997 2024-05-15T12:50:39.471 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.471 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.471 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:50:39.471 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.336400+0000", 2024-05-15T12:50:39.471 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.3213999999999998e-05 2024-05-15T12:50:39.471 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.471 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:50:39.471 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.472 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.472 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.472 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(osd.9.0:10136 2.d e170/110 2:b4472a13:::benchmark_data_smithi177_78104_object61314:head v 170'10137, mlcod=170'10114)", 2024-05-15T12:50:39.472 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:50:37.278255+0000", 2024-05-15T12:50:39.472 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 2.1234048799999998, 2024-05-15T12:50:39.472 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.058167473999999997, 2024-05-15T12:50:39.472 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:50:39.472 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:39.472 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:50:39.472 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.472 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:50:39.472 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.278255+0000", 2024-05-15T12:50:39.472 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.473 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.473 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.473 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:50:39.473 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.278255+0000", 2024-05-15T12:50:39.473 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.473 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.473 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.473 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:50:39.473 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.278256+0000", 2024-05-15T12:50:39.473 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.5899999999999995e-07 2024-05-15T12:50:39.473 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.473 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.473 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:50:39.474 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.278263+0000", 2024-05-15T12:50:39.474 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4780000000000002e-06 2024-05-15T12:50:39.474 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.474 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.474 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:50:39.474 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.278274+0000", 2024-05-15T12:50:39.474 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0954e-05 2024-05-15T12:50:39.474 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.474 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.474 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:50:39.474 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.278282+0000", 2024-05-15T12:50:39.474 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0700000000000007e-06 2024-05-15T12:50:39.474 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.475 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.475 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:50:39.475 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.296254+0000", 2024-05-15T12:50:39.475 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.017971493000000002 2024-05-15T12:50:39.475 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.475 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.475 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:50:39.475 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.296273+0000", 2024-05-15T12:50:39.475 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9239e-05 2024-05-15T12:50:39.475 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.475 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.475 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:50:39.475 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.336402+0000", 2024-05-15T12:50:39.476 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.040128865999999999 2024-05-15T12:50:39.476 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.476 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.476 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:50:39.476 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.336423+0000", 2024-05-15T12:50:39.476 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0514999999999999e-05 2024-05-15T12:50:39.476 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.476 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:50:39.476 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.476 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.476 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.476 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(osd.9.0:10137 2.d e170/110 2:b4473398:::benchmark_data_smithi177_78104_object95946:head v 170'10138, mlcod=170'10114)", 2024-05-15T12:50:39.476 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:50:37.282392+0000", 2024-05-15T12:50:39.477 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 2.1192682810000001, 2024-05-15T12:50:39.477 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.054053206999999999, 2024-05-15T12:50:39.477 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:50:39.477 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:39.477 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:50:39.477 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.477 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:50:39.477 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.282392+0000", 2024-05-15T12:50:39.477 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.477 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.477 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.477 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:50:39.478 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.282392+0000", 2024-05-15T12:50:39.478 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.478 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.478 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.478 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:50:39.478 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.282393+0000", 2024-05-15T12:50:39.478 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1489999999999999e-06 2024-05-15T12:50:39.478 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.478 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.478 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:50:39.478 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.282401+0000", 2024-05-15T12:50:39.478 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.585e-06 2024-05-15T12:50:39.478 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.478 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.479 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:50:39.479 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.282412+0000", 2024-05-15T12:50:39.479 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0988999999999999e-05 2024-05-15T12:50:39.479 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.479 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.479 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:50:39.479 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.282421+0000", 2024-05-15T12:50:39.479 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.3890000000000005e-06 2024-05-15T12:50:39.479 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.479 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.479 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:50:39.479 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.300656+0000", 2024-05-15T12:50:39.479 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.018235083999999999 2024-05-15T12:50:39.479 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.480 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.480 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:50:39.480 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.300675+0000", 2024-05-15T12:50:39.480 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.8938000000000001e-05 2024-05-15T12:50:39.480 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.480 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.480 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:50:39.480 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.336424+0000", 2024-05-15T12:50:39.480 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.035749597000000001 2024-05-15T12:50:39.480 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.480 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.480 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:50:39.480 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.336445+0000", 2024-05-15T12:50:39.481 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0475999999999998e-05 2024-05-15T12:50:39.481 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.481 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:50:39.481 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.481 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.481 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.481 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(osd.9.0:10138 2.d e170/110 2:b449adfa:::benchmark_data_smithi177_78104_object95549:head v 170'10139, mlcod=170'10114)", 2024-05-15T12:50:39.481 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:50:37.286413+0000", 2024-05-15T12:50:39.481 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 2.1152468149999999, 2024-05-15T12:50:39.481 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.050053680000000003, 2024-05-15T12:50:39.481 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:50:39.481 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:50:39.481 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:50:39.481 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.481 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:50:39.482 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.286413+0000", 2024-05-15T12:50:39.482 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.482 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.482 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.482 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:50:39.482 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.286413+0000", 2024-05-15T12:50:39.482 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:50:39.482 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.482 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.482 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:50:39.482 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.286414+0000", 2024-05-15T12:50:39.482 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4900000000000005e-07 2024-05-15T12:50:39.482 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.483 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.483 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:50:39.483 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.286420+0000", 2024-05-15T12:50:39.483 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4479999999999999e-06 2024-05-15T12:50:39.483 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.483 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.483 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:50:39.483 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.286435+0000", 2024-05-15T12:50:39.483 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4263e-05 2024-05-15T12:50:39.483 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.483 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.483 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:50:39.483 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.286442+0000", 2024-05-15T12:50:39.483 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.5120000000000002e-06 2024-05-15T12:50:39.483 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.483 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.484 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:50:39.484 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.304978+0000", 2024-05-15T12:50:39.484 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.018536037000000002 2024-05-15T12:50:39.484 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.484 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.484 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:50:39.484 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.304996+0000", 2024-05-15T12:50:39.484 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.791e-05 2024-05-15T12:50:39.484 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.484 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.484 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:50:39.484 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.336446+0000", 2024-05-15T12:50:39.484 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.031450293999999997 2024-05-15T12:50:39.484 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:50:39.485 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:50:39.485 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:50:39.485 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:50:37.336467+0000", 2024-05-15T12:50:39.485 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0466999999999999e-05 2024-05-15T12:50:39.485 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.485 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:50:39.485 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.485 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:50:39.485 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:50:39.485 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T12:50:39.485 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:50:39.560 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:39.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:39 smithi177 ceph-mon[44303]: pgmap v500: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 13 active+clean; 4.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 746 op/s; 8745/227568 objects misplaced (3.843%); 14 MiB/s, 219 objects/s recovering 2024-05-15T12:50:39.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:39 smithi027 ceph-mon[39795]: pgmap v500: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 13 active+clean; 4.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 746 op/s; 8745/227568 objects misplaced (3.843%); 14 MiB/s, 219 objects/s recovering 2024-05-15T12:50:40.196 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-05-15T12:50:40.279 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T12:50:40.508 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:50:41.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:41 smithi177 ceph-mon[44303]: pgmap v501: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 13 active+clean; 4.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 624 op/s; 7755/223488 objects misplaced (3.470%); 16 MiB/s, 255 objects/s recovering 2024-05-15T12:50:41.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:41 smithi027 ceph-mon[39795]: pgmap v501: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 13 active+clean; 4.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 624 op/s; 7755/223488 objects misplaced (3.470%); 16 MiB/s, 255 objects/s recovering 2024-05-15T12:50:41.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:41 smithi120 ceph-mon[43961]: pgmap v501: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 13 active+clean; 4.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 624 op/s; 7755/223488 objects misplaced (3.470%); 16 MiB/s, 255 objects/s recovering 2024-05-15T12:50:41.914 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:50:41.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:41.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:50:42.097 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:50:42.152 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:42.236 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:50:42.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:42.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:50:42.343 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:50:42.384 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:42.412 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:50:42.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:42.419 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:50:42.520 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:50:42.576 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:42.594 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T12:50:42.595 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T12:50:42.595 INFO:teuthology.orchestra.run.smithi120.stdout: "num_ops": 0 2024-05-15T12:50:42.595 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T12:50:42.602 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:50:42.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:42.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:50:42.710 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:50:42.754 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:42.822 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:50:42.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:42.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:50:42.930 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:50:42.963 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:43.166 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:50:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.2 to 1 (10000) 2024-05-15T12:50:43.630 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:43 smithi027 ceph-mon[39795]: pgmap v502: 17 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 14 active+clean; 4.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 733 op/s; 6850/217128 objects misplaced (3.155%); 17 MiB/s, 277 objects/s recovering 2024-05-15T12:50:43.630 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:43 smithi027 ceph-mon[39795]: from='client.14838 172.21.15.27:0/3002503702' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T12:50:43.631 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:43 smithi027 ceph-mon[39795]: from='client.14838 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T12:50:43.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:43 smithi177 ceph-mon[44303]: pgmap v502: 17 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 14 active+clean; 4.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 733 op/s; 6850/217128 objects misplaced (3.155%); 17 MiB/s, 277 objects/s recovering 2024-05-15T12:50:43.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:43 smithi177 ceph-mon[44303]: from='client.14838 172.21.15.27:0/3002503702' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T12:50:43.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:43 smithi177 ceph-mon[44303]: from='client.14838 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T12:50:43.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:43 smithi120 ceph-mon[43961]: pgmap v502: 17 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 14 active+clean; 4.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 733 op/s; 6850/217128 objects misplaced (3.155%); 17 MiB/s, 277 objects/s recovering 2024-05-15T12:50:43.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:43 smithi120 ceph-mon[43961]: from='client.14838 172.21.15.27:0/3002503702' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T12:50:43.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:43 smithi120 ceph-mon[43961]: from='client.14838 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T12:50:44.488 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 3 1 2024-05-15T12:50:44.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:44 smithi177 ceph-mon[44303]: from='client.14838 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T12:50:44.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:44 smithi177 ceph-mon[44303]: osdmap e172: 12 total, 12 up, 11 in 2024-05-15T12:50:44.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:44 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:50:44.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:44 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:50:44.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:44 smithi027 ceph-mon[39795]: from='client.14838 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T12:50:44.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:44 smithi027 ceph-mon[39795]: osdmap e172: 12 total, 12 up, 11 in 2024-05-15T12:50:44.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:44 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:50:44.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:44 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:50:44.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:44 smithi120 ceph-mon[43961]: from='client.14838 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T12:50:44.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:44 smithi120 ceph-mon[43961]: osdmap e172: 12 total, 12 up, 11 in 2024-05-15T12:50:44.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:44 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:50:44.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:44 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:50:44.814 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:50:44.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:44.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:50:44.918 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:50:44.945 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:44.964 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:50:44.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:44.967 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:50:45.068 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:50:45.096 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:45.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:45 smithi177 ceph-mon[44303]: pgmap v504: 17 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 14 active+clean; 4.3 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 766 op/s; 5795/211896 objects misplaced (2.735%); 23 MiB/s, 373 objects/s recovering 2024-05-15T12:50:45.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:45 smithi177 ceph-mon[44303]: osdmap e173: 12 total, 12 up, 11 in 2024-05-15T12:50:45.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:45 smithi027 ceph-mon[39795]: pgmap v504: 17 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 14 active+clean; 4.3 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 766 op/s; 5795/211896 objects misplaced (2.735%); 23 MiB/s, 373 objects/s recovering 2024-05-15T12:50:45.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:45 smithi027 ceph-mon[39795]: osdmap e173: 12 total, 12 up, 11 in 2024-05-15T12:50:45.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:45 smithi120 ceph-mon[43961]: pgmap v504: 17 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 14 active+clean; 4.3 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 766 op/s; 5795/211896 objects misplaced (2.735%); 23 MiB/s, 373 objects/s recovering 2024-05-15T12:50:45.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:45 smithi120 ceph-mon[43961]: osdmap e173: 12 total, 12 up, 11 in 2024-05-15T12:50:45.906 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-05-15T12:50:45.966 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:50:45.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:45.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:50:45.972 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:50:46.073 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:50:46.131 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:46.161 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:50:46.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:46.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:50:46.268 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:50:46.296 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:50:46.310 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:46.337 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:50:46.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:46.340 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:50:46.441 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:50:46.507 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:46.554 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:50:46.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:46.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:50:46.657 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:46 smithi027 ceph-mon[39795]: osdmap e174: 12 total, 12 up, 11 in 2024-05-15T12:50:46.657 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:46 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:50:46.657 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:46 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:50:46.660 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:50:46.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:46 smithi177 ceph-mon[44303]: osdmap e174: 12 total, 12 up, 11 in 2024-05-15T12:50:46.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:46 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:50:46.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:46 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:50:46.687 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:46 smithi120 ceph-mon[43961]: osdmap e174: 12 total, 12 up, 11 in 2024-05-15T12:50:46.687 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:46 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:50:46.687 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:46 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:50:46.690 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:46.756 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:50:46.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:46.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:50:46.861 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:50:46.933 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:47.089 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:50:47.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:47.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:50:47.199 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:50:47.199 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:50:47.212 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:50:47.261 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:47.270 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:50:47.313 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:50:47.358 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:50:47.409 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:50:47.445 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:50:47.485 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:50:47.522 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:50:47.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:47 smithi027 ceph-mon[39795]: pgmap v507: 17 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 14 active+clean; 4.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 870 op/s; 5583/207822 objects misplaced (2.686%); 17 MiB/s, 266 objects/s recovering 2024-05-15T12:50:47.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:47 smithi027 ceph-mon[39795]: osdmap e175: 12 total, 12 up, 11 in 2024-05-15T12:50:47.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:47 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4192976244' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:50:47.557 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:50:47.591 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:50:47.618 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:50:47.636 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:47 smithi120 ceph-mon[43961]: pgmap v507: 17 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 14 active+clean; 4.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 870 op/s; 5583/207822 objects misplaced (2.686%); 17 MiB/s, 266 objects/s recovering 2024-05-15T12:50:47.636 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:47 smithi120 ceph-mon[43961]: osdmap e175: 12 total, 12 up, 11 in 2024-05-15T12:50:47.636 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:47 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4192976244' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:50:47.638 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:50:47.638 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:50:47.638 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:50:47.638 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:50:47.638 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:50:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:50:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:50:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:50:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:50:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:50:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:50:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:50:47.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:47 smithi177 ceph-mon[44303]: pgmap v507: 17 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 14 active+clean; 4.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 870 op/s; 5583/207822 objects misplaced (2.686%); 17 MiB/s, 266 objects/s recovering 2024-05-15T12:50:47.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:47 smithi177 ceph-mon[44303]: osdmap e175: 12 total, 12 up, 11 in 2024-05-15T12:50:47.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:47 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4192976244' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:50:48.199 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T12:50:48.199 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:50:48.390 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:48 smithi027 ceph-mon[39795]: osdmap e176: 12 total, 12 up, 11 in 2024-05-15T12:50:48.472 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:50:48.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:48.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:50:48.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:48 smithi120 ceph-mon[43961]: osdmap e176: 12 total, 12 up, 11 in 2024-05-15T12:50:48.577 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:50:48.639 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:48.674 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:50:48.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:48.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:50:48.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:48 smithi177 ceph-mon[44303]: osdmap e176: 12 total, 12 up, 11 in 2024-05-15T12:50:48.779 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:50:48.812 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:48.837 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:50:48.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:48.850 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:50:48.951 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:50:48.983 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:49.015 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:50:49.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:49.018 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:50:49.021 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-05-15T12:50:49.119 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:50:49.151 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:49.174 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:50:49.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:49.178 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:50:49.279 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:50:49.325 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:49.362 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:50:49.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:49.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:50:49.391 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:50:49.423 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T12:50:49.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:49 smithi177 ceph-mon[44303]: pgmap v510: 17 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 15 active+clean; 4.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 564 op/s; 5228/205122 objects misplaced (2.549%); 5.5 MiB/s, 87 objects/s recovering 2024-05-15T12:50:49.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:49 smithi177 ceph-mon[44303]: osdmap e177: 12 total, 12 up, 11 in 2024-05-15T12:50:49.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:49 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1487727322' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:50:49.470 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:50:49.497 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:49.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:49 smithi120 ceph-mon[43961]: pgmap v510: 17 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 15 active+clean; 4.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 564 op/s; 5228/205122 objects misplaced (2.549%); 5.5 MiB/s, 87 objects/s recovering 2024-05-15T12:50:49.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:49 smithi120 ceph-mon[43961]: osdmap e177: 12 total, 12 up, 11 in 2024-05-15T12:50:49.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:49 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1487727322' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:50:49.697 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:50:49.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:49.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:50:49.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:49 smithi027 ceph-mon[39795]: pgmap v510: 17 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 15 active+clean; 4.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 564 op/s; 5228/205122 objects misplaced (2.549%); 5.5 MiB/s, 87 objects/s recovering 2024-05-15T12:50:49.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:49 smithi027 ceph-mon[39795]: osdmap e177: 12 total, 12 up, 11 in 2024-05-15T12:50:49.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:49 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1487727322' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:50:49.803 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:50:49.878 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:50.472 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:50:50.675 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:50 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:50:50.675 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:50 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1487727322' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:50:50.675 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:50 smithi027 ceph-mon[39795]: osdmap e178: 12 total, 12 up, 11 in 2024-05-15T12:50:50.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:50 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:50:50.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:50 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1487727322' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:50:50.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:50 smithi177 ceph-mon[44303]: osdmap e178: 12 total, 12 up, 11 in 2024-05-15T12:50:50.758 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:50:50.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:50.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:50:50.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:50 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:50:50.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:50 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1487727322' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:50:50.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:50 smithi120 ceph-mon[43961]: osdmap e178: 12 total, 12 up, 11 in 2024-05-15T12:50:50.869 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:50:50.949 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:51.078 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:50:51.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:51.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:50:51.187 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:50:51.217 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:51.250 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:50:51.486 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:51 smithi120 ceph-mon[43961]: pgmap v513: 17 pgs: 1 peering, 1 active+remapped+backfill_toofull, 15 active+clean; 4.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 551 op/s; 4271/201207 objects misplaced (2.123%); 20 MiB/s, 322 objects/s recovering 2024-05-15T12:50:51.486 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:51 smithi120 ceph-mon[43961]: 2.9 scrub starts 2024-05-15T12:50:51.486 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:51 smithi120 ceph-mon[43961]: 2.a scrub starts 2024-05-15T12:50:51.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:51 smithi027 ceph-mon[39795]: pgmap v513: 17 pgs: 1 peering, 1 active+remapped+backfill_toofull, 15 active+clean; 4.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 551 op/s; 4271/201207 objects misplaced (2.123%); 20 MiB/s, 322 objects/s recovering 2024-05-15T12:50:51.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:51 smithi027 ceph-mon[39795]: 2.9 scrub starts 2024-05-15T12:50:51.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:51 smithi027 ceph-mon[39795]: 2.a scrub starts 2024-05-15T12:50:51.774 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:50:51.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:51.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:50:51.810 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T12:50:51.810 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:50:51.879 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:50:51.928 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:51.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:51 smithi177 ceph-mon[44303]: pgmap v513: 17 pgs: 1 peering, 1 active+remapped+backfill_toofull, 15 active+clean; 4.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 551 op/s; 4271/201207 objects misplaced (2.123%); 20 MiB/s, 322 objects/s recovering 2024-05-15T12:50:51.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:51 smithi177 ceph-mon[44303]: 2.9 scrub starts 2024-05-15T12:50:51.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:51 smithi177 ceph-mon[44303]: 2.a scrub starts 2024-05-15T12:50:52.091 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:50:52.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:52.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:50:52.201 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:50:52.265 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:52.314 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:50:52.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:52.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:50:52.426 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:50:52.453 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T12:50:52.480 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.3 to 1 (10000) 2024-05-15T12:50:52.514 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:52.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:52 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T12:50:52.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:52 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T12:50:52.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:52 smithi027 ceph-mon[39795]: 2.4 scrub starts 2024-05-15T12:50:52.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:52 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1085708810' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T12:50:52.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:52 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T12:50:52.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:52 smithi027 ceph-mon[39795]: 2.9 scrub ok 2024-05-15T12:50:52.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:52 smithi027 ceph-mon[39795]: 2.7 scrub starts 2024-05-15T12:50:52.617 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-05-15T12:50:52.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:52 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T12:50:52.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:52 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T12:50:52.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:52 smithi120 ceph-mon[43961]: 2.4 scrub starts 2024-05-15T12:50:52.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:52 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1085708810' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T12:50:52.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:52 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T12:50:52.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:52 smithi120 ceph-mon[43961]: 2.9 scrub ok 2024-05-15T12:50:52.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:52 smithi120 ceph-mon[43961]: 2.7 scrub starts 2024-05-15T12:50:52.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:52 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T12:50:52.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:52 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T12:50:52.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:52 smithi177 ceph-mon[44303]: 2.4 scrub starts 2024-05-15T12:50:52.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:52 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1085708810' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T12:50:52.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:52 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T12:50:52.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:52 smithi177 ceph-mon[44303]: 2.9 scrub ok 2024-05-15T12:50:52.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:52 smithi177 ceph-mon[44303]: 2.7 scrub starts 2024-05-15T12:50:52.968 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:50:53.503 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:53 smithi120 ceph-mon[43961]: pgmap v514: 17 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 1 peering, 14 active+clean; 4.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 694 op/s; 4035/194973 objects misplaced (2.070%); 11 MiB/s, 181 objects/s recovering 2024-05-15T12:50:53.503 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:53 smithi120 ceph-mon[43961]: 2.a scrub ok 2024-05-15T12:50:53.503 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:53 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T12:50:53.503 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:53 smithi120 ceph-mon[43961]: osdmap e179: 12 total, 12 up, 11 in 2024-05-15T12:50:53.503 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:53 smithi120 ceph-mon[43961]: 2.7 scrub ok 2024-05-15T12:50:53.503 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:53 smithi120 ceph-mon[43961]: 2.4 scrub ok 2024-05-15T12:50:53.731 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:53 smithi027 ceph-mon[39795]: pgmap v514: 17 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 1 peering, 14 active+clean; 4.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 694 op/s; 4035/194973 objects misplaced (2.070%); 11 MiB/s, 181 objects/s recovering 2024-05-15T12:50:53.731 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:53 smithi027 ceph-mon[39795]: 2.a scrub ok 2024-05-15T12:50:53.731 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:53 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T12:50:53.732 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:53 smithi027 ceph-mon[39795]: osdmap e179: 12 total, 12 up, 11 in 2024-05-15T12:50:53.732 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:53 smithi027 ceph-mon[39795]: 2.7 scrub ok 2024-05-15T12:50:53.732 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:53 smithi027 ceph-mon[39795]: 2.4 scrub ok 2024-05-15T12:50:53.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:53 smithi177 ceph-mon[44303]: pgmap v514: 17 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 1 peering, 14 active+clean; 4.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 694 op/s; 4035/194973 objects misplaced (2.070%); 11 MiB/s, 181 objects/s recovering 2024-05-15T12:50:53.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:53 smithi177 ceph-mon[44303]: 2.a scrub ok 2024-05-15T12:50:53.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:53 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T12:50:53.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:53 smithi177 ceph-mon[44303]: osdmap e179: 12 total, 12 up, 11 in 2024-05-15T12:50:53.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:53 smithi177 ceph-mon[44303]: 2.7 scrub ok 2024-05-15T12:50:53.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:53 smithi177 ceph-mon[44303]: 2.4 scrub ok 2024-05-15T12:50:54.747 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:54 smithi120 ceph-mon[43961]: 2.6 scrub starts 2024-05-15T12:50:54.747 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:54 smithi120 ceph-mon[43961]: 2.e scrub starts 2024-05-15T12:50:54.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:54 smithi027 ceph-mon[39795]: 2.6 scrub starts 2024-05-15T12:50:54.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:54 smithi027 ceph-mon[39795]: 2.e scrub starts 2024-05-15T12:50:54.885 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 4 1 2024-05-15T12:50:54.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:54 smithi177 ceph-mon[44303]: 2.6 scrub starts 2024-05-15T12:50:54.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:54 smithi177 ceph-mon[44303]: 2.e scrub starts 2024-05-15T12:50:55.053 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:50:55.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:55.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:50:55.077 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T12:50:55.077 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:50:55.169 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:50:55.211 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:55.362 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:50:55.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:55.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:50:55.467 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:50:55.486 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:50:55.504 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:50:55.525 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:50:55.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:55.528 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:50:55.535 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:50:55.563 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:50:55.589 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:50:55.620 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:50:55.629 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:50:55.656 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:55 smithi120 ceph-mon[43961]: pgmap v516: 17 pgs: 2 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 14 active+clean; 3.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 829 op/s; 4035/190188 objects misplaced (2.122%); 9.8 MiB/s, 156 objects/s recovering 2024-05-15T12:50:55.656 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:55 smithi120 ceph-mon[43961]: 2.6 scrub ok 2024-05-15T12:50:55.656 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:55 smithi120 ceph-mon[43961]: 2.e scrub ok 2024-05-15T12:50:55.657 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:55 smithi120 ceph-mon[43961]: from='client.24695 172.21.15.27:0/4204785078' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:50:55.657 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:55 smithi120 ceph-mon[43961]: from='client.24695 172.21.15.27:0/4204785078' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:50:55.659 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:50:55.659 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:55.708 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:50:55.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:55 smithi027 ceph-mon[39795]: pgmap v516: 17 pgs: 2 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 14 active+clean; 3.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 829 op/s; 4035/190188 objects misplaced (2.122%); 9.8 MiB/s, 156 objects/s recovering 2024-05-15T12:50:55.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:55 smithi027 ceph-mon[39795]: 2.6 scrub ok 2024-05-15T12:50:55.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:55 smithi027 ceph-mon[39795]: 2.e scrub ok 2024-05-15T12:50:55.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:55 smithi027 ceph-mon[39795]: from='client.24695 172.21.15.27:0/4204785078' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:50:55.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:55 smithi027 ceph-mon[39795]: from='client.24695 172.21.15.27:0/4204785078' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:50:55.776 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:50:55.817 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:50:55.819 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:50:55.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:55.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:50:55.846 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:50:55.863 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-05-15T12:50:55.895 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:50:55.926 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:50:55.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:55 smithi177 ceph-mon[44303]: pgmap v516: 17 pgs: 2 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 14 active+clean; 3.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 829 op/s; 4035/190188 objects misplaced (2.122%); 9.8 MiB/s, 156 objects/s recovering 2024-05-15T12:50:55.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:55 smithi177 ceph-mon[44303]: 2.6 scrub ok 2024-05-15T12:50:55.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:55 smithi177 ceph-mon[44303]: 2.e scrub ok 2024-05-15T12:50:55.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:55 smithi177 ceph-mon[44303]: from='client.24695 172.21.15.27:0/4204785078' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:50:55.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:55 smithi177 ceph-mon[44303]: from='client.24695 172.21.15.27:0/4204785078' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:50:55.945 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:50:55.961 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:55.981 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:50:55.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:55.989 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:50:56.006 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:50:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:50:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:50:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:50:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:50:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:50:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:50:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:50:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:50:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:50:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:50:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:50:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:50:56.090 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:50:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:56.249 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:50:56.320 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:50:56.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:56.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:50:56.424 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:50:56.460 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:56.702 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:56 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:50:56.703 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:56 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:50:56.709 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:56 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:50:56.709 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:56 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:50:56.723 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:50:56.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:56.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:50:56.829 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:50:56.871 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:56.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:56 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:50:56.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:56 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:50:57.346 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:50:57.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:57.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:50:57.451 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:50:57.496 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:57.668 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:50:57.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:57.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:50:57.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:57 smithi027 ceph-mon[39795]: pgmap v517: 17 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 14 active+clean; 3.8 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 744 op/s; 3809/185715 objects misplaced (2.051%) 2024-05-15T12:50:57.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:57 smithi027 ceph-mon[39795]: osdmap e180: 12 total, 12 up, 11 in 2024-05-15T12:50:57.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:57 smithi120 ceph-mon[43961]: pgmap v517: 17 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 14 active+clean; 3.8 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 744 op/s; 3809/185715 objects misplaced (2.051%) 2024-05-15T12:50:57.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:57 smithi120 ceph-mon[43961]: osdmap e180: 12 total, 12 up, 11 in 2024-05-15T12:50:57.786 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:50:57.824 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:57.850 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:50:57.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:57.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:50:57.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:57 smithi177 ceph-mon[44303]: pgmap v517: 17 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 14 active+clean; 3.8 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 744 op/s; 3809/185715 objects misplaced (2.051%) 2024-05-15T12:50:57.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:57 smithi177 ceph-mon[44303]: osdmap e180: 12 total, 12 up, 11 in 2024-05-15T12:50:57.960 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:50:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:58.282 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:50:58.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:58.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:50:58.389 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:50:58.446 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:58.474 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:50:58.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:58.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:50:58.535 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T12:50:58.536 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T12:50:58.536 INFO:teuthology.orchestra.run.smithi120.stdout: "complaint_time": 30, 2024-05-15T12:50:58.536 INFO:teuthology.orchestra.run.smithi120.stdout: "num_blocked_ops": 0 2024-05-15T12:50:58.536 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T12:50:58.579 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:50:58.619 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:58.642 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:50:58.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:58.645 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:50:58.747 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:50:58.783 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:58.808 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:50:58.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:58.813 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:50:58.914 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:50:58.947 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:58.973 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:50:58.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:50:58.979 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:50:59.080 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:50:59.146 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:50:59.313 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-05-15T12:50:59.680 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:50:59.704 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:50:59.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:50:59 smithi177 ceph-mon[44303]: pgmap v519: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 773 op/s; 3809/182637 objects misplaced (2.086%) 2024-05-15T12:50:59.930 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:50:59 smithi120 ceph-mon[43961]: pgmap v519: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 773 op/s; 3809/182637 objects misplaced (2.086%) 2024-05-15T12:51:00.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:50:59 smithi027 ceph-mon[39795]: pgmap v519: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 773 op/s; 3809/182637 objects misplaced (2.086%) 2024-05-15T12:51:01.429 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:51:01.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:01.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:51:01.538 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:51:01.582 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:01.610 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:51:01.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:01.617 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:51:01.684 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T12:51:01.684 INFO:teuthology.orchestra.run.smithi120.stdout: "size": 20, 2024-05-15T12:51:01.684 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 600, 2024-05-15T12:51:01.684 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [ 2024-05-15T12:51:01.684 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.684 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:53860 2.7 2:eacb7370:::benchmark_data_smithi177_78104_object53859:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-05-15T12:51:01.685 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:43.957376+0000", 2024-05-15T12:51:01.685 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 137.73412883200001, 2024-05-15T12:51:01.685 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.011513259, 2024-05-15T12:51:01.685 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:51:01.685 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T12:51:01.685 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:51:01.685 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:51:01.685 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:01.685 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 53860 2024-05-15T12:51:01.685 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.685 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:51:01.685 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.686 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:51:01.686 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.957376+0000", 2024-05-15T12:51:01.686 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.686 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.686 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.686 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:51:01.686 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.957376+0000", 2024-05-15T12:51:01.686 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.686 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.686 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.686 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:51:01.686 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.957377+0000", 2024-05-15T12:51:01.686 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.412e-06 2024-05-15T12:51:01.687 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.687 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.687 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:51:01.687 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.957429+0000", 2024-05-15T12:51:01.687 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2389999999999998e-05 2024-05-15T12:51:01.687 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.687 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.687 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:51:01.687 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.957441+0000", 2024-05-15T12:51:01.687 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1965e-05 2024-05-15T12:51:01.687 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.687 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.687 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:51:01.687 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.957448+0000", 2024-05-15T12:51:01.687 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0389999999999999e-06 2024-05-15T12:51:01.688 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.688 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.688 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.688 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.957509+0000", 2024-05-15T12:51:01.688 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.0643999999999997e-05 2024-05-15T12:51:01.688 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.688 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.688 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:51:01.688 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.957667+0000", 2024-05-15T12:51:01.688 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000157524 2024-05-15T12:51:01.688 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.688 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.688 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3", 2024-05-15T12:51:01.688 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.957826+0000", 2024-05-15T12:51:01.688 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015924900000000001 2024-05-15T12:51:01.689 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.689 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.689 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:51:01.689 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.959444+0000", 2024-05-15T12:51:01.689 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0016178220000000001 2024-05-15T12:51:01.689 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.689 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.689 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.689 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.963233+0000", 2024-05-15T12:51:01.689 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0037892590000000001 2024-05-15T12:51:01.689 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.689 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.689 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.689 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.963246+0000", 2024-05-15T12:51:01.690 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3461999999999999e-05 2024-05-15T12:51:01.690 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.690 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.690 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.690 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.968787+0000", 2024-05-15T12:51:01.690 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.005540694 2024-05-15T12:51:01.690 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.690 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.690 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:51:01.690 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.968889+0000", 2024-05-15T12:51:01.690 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00010179900000000001 2024-05-15T12:51:01.690 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.690 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:51:01.691 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.691 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.691 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.691 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:53866 2.7 2:e7adacc6:::benchmark_data_smithi177_78104_object53865:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-05-15T12:51:01.691 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:43.962362+0000", 2024-05-15T12:51:01.691 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 137.72914268700001, 2024-05-15T12:51:01.691 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0124753230000001, 2024-05-15T12:51:01.692 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:51:01.692 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:01.692 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:51:01.692 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:51:01.692 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:01.692 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 53866 2024-05-15T12:51:01.692 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.693 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:51:01.693 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.693 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:51:01.693 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.962362+0000", 2024-05-15T12:51:01.693 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.693 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.694 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.694 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:51:01.694 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.962362+0000", 2024-05-15T12:51:01.694 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.694 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.694 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.694 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:51:01.694 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.962363+0000", 2024-05-15T12:51:01.695 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.688e-06 2024-05-15T12:51:01.695 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.695 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.695 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:51:01.695 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.962407+0000", 2024-05-15T12:51:01.695 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3659000000000003e-05 2024-05-15T12:51:01.695 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.695 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.696 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:51:01.696 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.962421+0000", 2024-05-15T12:51:01.696 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3434e-05 2024-05-15T12:51:01.696 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.696 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.696 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:51:01.696 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.962429+0000", 2024-05-15T12:51:01.697 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.9190000000000005e-06 2024-05-15T12:51:01.697 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.697 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.697 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.697 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.963343+0000", 2024-05-15T12:51:01.697 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00091340599999999996 2024-05-15T12:51:01.697 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.698 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.698 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.698 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.963352+0000", 2024-05-15T12:51:01.698 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0520000000000007e-06 2024-05-15T12:51:01.698 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.698 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.698 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.698 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.969278+0000", 2024-05-15T12:51:01.698 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0059264640000001 2024-05-15T12:51:01.699 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.699 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.699 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:51:01.699 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.969425+0000", 2024-05-15T12:51:01.699 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000146625 2024-05-15T12:51:01.699 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.699 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.699 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3", 2024-05-15T12:51:01.699 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.969587+0000", 2024-05-15T12:51:01.699 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00016151199999999999 2024-05-15T12:51:01.699 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.699 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.699 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.699 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.974642+0000", 2024-05-15T12:51:01.700 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0050549879999999998 2024-05-15T12:51:01.700 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.700 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.700 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.700 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.974697+0000", 2024-05-15T12:51:01.700 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5390999999999999e-05 2024-05-15T12:51:01.700 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.700 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.700 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:51:01.700 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.974775+0000", 2024-05-15T12:51:01.700 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8360000000000002e-05 2024-05-15T12:51:01.700 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.700 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.700 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:51:01.700 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.974813+0000", 2024-05-15T12:51:01.700 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.8211999999999999e-05 2024-05-15T12:51:01.701 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.701 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.701 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:51:01.701 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.974837+0000", 2024-05-15T12:51:01.701 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.3612999999999999e-05 2024-05-15T12:51:01.701 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.701 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:51:01.701 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.701 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.701 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.701 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:53873 2.7 2:e517f039:::benchmark_data_smithi177_78104_object53872:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-05-15T12:51:01.701 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:43.964338+0000", 2024-05-15T12:51:01.701 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 137.72716614300001, 2024-05-15T12:51:01.701 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0106468, 2024-05-15T12:51:01.701 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:51:01.701 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:01.702 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:51:01.702 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:51:01.702 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:01.702 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 53873 2024-05-15T12:51:01.702 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.702 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:51:01.702 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.702 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:51:01.702 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.964338+0000", 2024-05-15T12:51:01.702 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.702 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.702 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.702 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:51:01.703 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.964338+0000", 2024-05-15T12:51:01.703 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.703 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.703 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.703 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:51:01.703 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.964340+0000", 2024-05-15T12:51:01.703 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3069999999999999e-06 2024-05-15T12:51:01.703 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.703 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.703 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:51:01.703 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.964391+0000", 2024-05-15T12:51:01.703 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1861999999999999e-05 2024-05-15T12:51:01.703 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.703 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.703 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:51:01.704 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.964405+0000", 2024-05-15T12:51:01.704 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3689999999999999e-05 2024-05-15T12:51:01.704 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.704 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.704 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:51:01.704 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.964413+0000", 2024-05-15T12:51:01.704 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2900000000000002e-06 2024-05-15T12:51:01.704 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.704 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.704 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.704 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.964471+0000", 2024-05-15T12:51:01.704 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7661999999999998e-05 2024-05-15T12:51:01.704 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.704 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.704 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.705 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.964482+0000", 2024-05-15T12:51:01.705 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1137e-05 2024-05-15T12:51:01.705 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.705 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.705 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.705 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.971323+0000", 2024-05-15T12:51:01.705 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0068405309999999 2024-05-15T12:51:01.705 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.705 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.705 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:51:01.705 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.971441+0000", 2024-05-15T12:51:01.705 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00011856199999999999 2024-05-15T12:51:01.705 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.705 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.705 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3", 2024-05-15T12:51:01.706 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.971594+0000", 2024-05-15T12:51:01.706 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015217200000000001 2024-05-15T12:51:01.706 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.706 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.706 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.706 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.974888+0000", 2024-05-15T12:51:01.706 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00329415 2024-05-15T12:51:01.706 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.706 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.706 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.706 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.974928+0000", 2024-05-15T12:51:01.706 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0568000000000002e-05 2024-05-15T12:51:01.706 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.707 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.707 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:51:01.707 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.974936+0000", 2024-05-15T12:51:01.707 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8259999999999993e-06 2024-05-15T12:51:01.707 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.707 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.707 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:51:01.707 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.974964+0000", 2024-05-15T12:51:01.707 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.7440000000000002e-05 2024-05-15T12:51:01.707 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.707 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.707 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:51:01.707 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.974985+0000", 2024-05-15T12:51:01.707 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1603e-05 2024-05-15T12:51:01.707 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.708 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:51:01.708 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.708 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.708 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.708 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:53902 2.7 2:e70705ef:::benchmark_data_smithi177_78104_object53901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-05-15T12:51:01.708 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:43.982287+0000", 2024-05-15T12:51:01.708 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 137.70921779099999, 2024-05-15T12:51:01.708 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.99522317100000002, 2024-05-15T12:51:01.708 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:51:01.708 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:01.708 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:51:01.708 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:51:01.709 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:01.709 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 53902 2024-05-15T12:51:01.709 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.709 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:51:01.709 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.709 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:51:01.709 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.982287+0000", 2024-05-15T12:51:01.709 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.709 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.709 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.709 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:51:01.709 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.982287+0000", 2024-05-15T12:51:01.710 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.710 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.710 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.710 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:51:01.710 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.982288+0000", 2024-05-15T12:51:01.710 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5510000000000001e-06 2024-05-15T12:51:01.710 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.710 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.710 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:51:01.711 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.982331+0000", 2024-05-15T12:51:01.711 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2959e-05 2024-05-15T12:51:01.711 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.711 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.711 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:51:01.711 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.982346+0000", 2024-05-15T12:51:01.711 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5301000000000001e-05 2024-05-15T12:51:01.711 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.711 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.711 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:51:01.711 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.982354+0000", 2024-05-15T12:51:01.711 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7519999999999996e-06 2024-05-15T12:51:01.711 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.711 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.711 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.712 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.982420+0000", 2024-05-15T12:51:01.712 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.5413999999999994e-05 2024-05-15T12:51:01.712 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.712 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.712 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.712 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.982431+0000", 2024-05-15T12:51:01.712 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.183e-05 2024-05-15T12:51:01.712 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.712 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.712 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.712 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.973398+0000", 2024-05-15T12:51:01.712 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.99096646200000005 2024-05-15T12:51:01.713 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.713 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.713 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:51:01.713 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.973512+0000", 2024-05-15T12:51:01.713 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000114487 2024-05-15T12:51:01.713 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.713 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.713 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3", 2024-05-15T12:51:01.713 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.973619+0000", 2024-05-15T12:51:01.713 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00010624 2024-05-15T12:51:01.713 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.713 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.713 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:51:01.713 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.974991+0000", 2024-05-15T12:51:01.713 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0013723190000000001 2024-05-15T12:51:01.714 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.714 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.714 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.714 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.977396+0000", 2024-05-15T12:51:01.714 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0024052710000000001 2024-05-15T12:51:01.714 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.714 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.714 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.714 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.977437+0000", 2024-05-15T12:51:01.714 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0538999999999999e-05 2024-05-15T12:51:01.714 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.714 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.714 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:51:01.715 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.977474+0000", 2024-05-15T12:51:01.715 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.6897000000000003e-05 2024-05-15T12:51:01.715 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.715 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.715 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:51:01.715 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.977510+0000", 2024-05-15T12:51:01.715 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.6149000000000002e-05 2024-05-15T12:51:01.715 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.715 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:51:01.715 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.715 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.715 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.715 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:53914 2.7 2:ee8bfed2:::benchmark_data_smithi177_78104_object53913:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-05-15T12:51:01.715 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:43.987976+0000", 2024-05-15T12:51:01.716 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 137.70352817, 2024-05-15T12:51:01.716 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.99186516999999996, 2024-05-15T12:51:01.716 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:51:01.716 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:01.716 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:51:01.716 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:51:01.716 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:01.716 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 53914 2024-05-15T12:51:01.716 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.716 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:51:01.716 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.716 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:51:01.716 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.987976+0000", 2024-05-15T12:51:01.717 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.717 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.717 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.717 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:51:01.717 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.987976+0000", 2024-05-15T12:51:01.717 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.717 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.717 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.717 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:51:01.717 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.987978+0000", 2024-05-15T12:51:01.717 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5200000000000001e-06 2024-05-15T12:51:01.717 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.718 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.718 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:51:01.718 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.988022+0000", 2024-05-15T12:51:01.718 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4493999999999998e-05 2024-05-15T12:51:01.718 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.718 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.718 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:51:01.718 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.988038+0000", 2024-05-15T12:51:01.718 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5709000000000002e-05 2024-05-15T12:51:01.718 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.718 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.718 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:51:01.718 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.988047+0000", 2024-05-15T12:51:01.719 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.799e-06 2024-05-15T12:51:01.719 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.719 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.719 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.719 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.988112+0000", 2024-05-15T12:51:01.719 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.5016000000000002e-05 2024-05-15T12:51:01.719 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.719 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.719 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.719 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.988122+0000", 2024-05-15T12:51:01.719 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0592e-05 2024-05-15T12:51:01.719 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.719 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.719 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.720 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.975019+0000", 2024-05-15T12:51:01.720 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.98689622200000005 2024-05-15T12:51:01.720 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.720 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.720 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:51:01.720 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.975097+0000", 2024-05-15T12:51:01.720 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8658000000000005e-05 2024-05-15T12:51:01.720 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.720 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.720 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3", 2024-05-15T12:51:01.720 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.975190+0000", 2024-05-15T12:51:01.720 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.2530999999999999e-05 2024-05-15T12:51:01.720 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.720 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.720 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.720 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.977548+0000", 2024-05-15T12:51:01.721 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0023580670000000001 2024-05-15T12:51:01.721 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.721 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.721 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:51:01.721 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.977556+0000", 2024-05-15T12:51:01.721 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.6650000000000003e-06 2024-05-15T12:51:01.721 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.721 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.721 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.721 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.979776+0000", 2024-05-15T12:51:01.721 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0022206050000000001 2024-05-15T12:51:01.721 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.721 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.721 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:51:01.721 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.979810+0000", 2024-05-15T12:51:01.721 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.4156999999999998e-05 2024-05-15T12:51:01.722 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.722 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.722 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:51:01.722 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.979841+0000", 2024-05-15T12:51:01.722 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.1134999999999999e-05 2024-05-15T12:51:01.722 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.722 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:51:01.722 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.722 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.722 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.722 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:53924 2.7 2:e120d2c9:::benchmark_data_smithi177_78104_object53923:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-05-15T12:51:01.722 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:43.994171+0000", 2024-05-15T12:51:01.722 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 137.69733381099999, 2024-05-15T12:51:01.722 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.99687578600000004, 2024-05-15T12:51:01.723 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:51:01.723 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:01.723 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:51:01.723 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:51:01.723 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:01.723 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 53924 2024-05-15T12:51:01.723 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.723 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:51:01.723 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.723 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:51:01.723 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.994171+0000", 2024-05-15T12:51:01.723 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.723 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.724 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.724 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:51:01.724 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.994171+0000", 2024-05-15T12:51:01.724 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.724 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.724 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.724 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:51:01.724 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.994172+0000", 2024-05-15T12:51:01.724 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.64e-06 2024-05-15T12:51:01.724 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.724 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.724 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:51:01.724 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.994259+0000", 2024-05-15T12:51:01.724 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7101000000000005e-05 2024-05-15T12:51:01.725 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.725 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.725 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:51:01.725 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.994286+0000", 2024-05-15T12:51:01.725 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.6465000000000002e-05 2024-05-15T12:51:01.725 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.725 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.725 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:51:01.725 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.994296+0000", 2024-05-15T12:51:01.725 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.9129999999999998e-06 2024-05-15T12:51:01.725 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.725 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.725 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.725 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.994359+0000", 2024-05-15T12:51:01.726 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3508000000000002e-05 2024-05-15T12:51:01.726 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.726 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.726 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.726 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:43.994370+0000", 2024-05-15T12:51:01.726 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0263e-05 2024-05-15T12:51:01.726 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.726 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.726 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.726 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.976345+0000", 2024-05-15T12:51:01.726 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.98197519499999997 2024-05-15T12:51:01.726 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.726 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.727 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:51:01.727 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.976427+0000", 2024-05-15T12:51:01.727 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2409000000000003e-05 2024-05-15T12:51:01.727 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.727 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.727 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3", 2024-05-15T12:51:01.727 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.976531+0000", 2024-05-15T12:51:01.727 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00010341200000000001 2024-05-15T12:51:01.727 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.727 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.727 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:51:01.727 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.979732+0000", 2024-05-15T12:51:01.727 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0032018699999999999 2024-05-15T12:51:01.727 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.728 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.728 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.728 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.979887+0000", 2024-05-15T12:51:01.728 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015474900000000001 2024-05-15T12:51:01.728 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.728 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.728 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.728 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.990959+0000", 2024-05-15T12:51:01.728 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.011071575 2024-05-15T12:51:01.728 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.728 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.728 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:51:01.728 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.991005+0000", 2024-05-15T12:51:01.728 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6127000000000001e-05 2024-05-15T12:51:01.729 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.729 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.729 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:51:01.729 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.991046+0000", 2024-05-15T12:51:01.729 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.1559e-05 2024-05-15T12:51:01.729 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.729 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:51:01.729 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.729 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.729 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.729 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:53943 2.7 2:e56c7dc7:::benchmark_data_smithi177_78104_object53942:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-05-15T12:51:01.729 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:44.007234+0000", 2024-05-15T12:51:01.729 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 137.68427056799999, 2024-05-15T12:51:01.730 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.98401325399999995, 2024-05-15T12:51:01.730 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:51:01.730 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:01.730 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:51:01.730 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:51:01.730 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:01.730 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 53943 2024-05-15T12:51:01.730 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.730 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:51:01.730 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.730 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:51:01.730 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.007234+0000", 2024-05-15T12:51:01.730 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.730 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.731 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.731 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:51:01.731 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.007234+0000", 2024-05-15T12:51:01.731 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.731 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.731 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.731 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:51:01.731 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.007235+0000", 2024-05-15T12:51:01.731 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.19e-06 2024-05-15T12:51:01.731 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.731 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.731 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:51:01.731 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.007324+0000", 2024-05-15T12:51:01.731 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8889999999999998e-05 2024-05-15T12:51:01.732 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.732 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.732 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:51:01.732 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.007335+0000", 2024-05-15T12:51:01.732 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0754000000000001e-05 2024-05-15T12:51:01.732 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.732 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.732 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:51:01.732 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.007342+0000", 2024-05-15T12:51:01.732 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8449999999999997e-06 2024-05-15T12:51:01.732 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.732 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.732 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.732 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.007404+0000", 2024-05-15T12:51:01.733 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.2912999999999999e-05 2024-05-15T12:51:01.733 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.733 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.733 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.733 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.007417+0000", 2024-05-15T12:51:01.733 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2798e-05 2024-05-15T12:51:01.733 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.733 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.733 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.733 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.977581+0000", 2024-05-15T12:51:01.733 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.97016336599999997 2024-05-15T12:51:01.733 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.733 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.733 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:51:01.734 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.977668+0000", 2024-05-15T12:51:01.734 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7540999999999994e-05 2024-05-15T12:51:01.734 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.734 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.734 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3", 2024-05-15T12:51:01.734 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.977774+0000", 2024-05-15T12:51:01.734 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000106178 2024-05-15T12:51:01.734 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.734 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.734 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:51:01.734 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.979895+0000", 2024-05-15T12:51:01.734 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.002120823 2024-05-15T12:51:01.735 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.735 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.735 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.735 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.987571+0000", 2024-05-15T12:51:01.735 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.007676039 2024-05-15T12:51:01.735 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.735 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.736 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.736 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.991111+0000", 2024-05-15T12:51:01.736 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0035397279999999998 2024-05-15T12:51:01.736 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.736 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.736 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:51:01.736 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.991184+0000", 2024-05-15T12:51:01.736 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3510000000000006e-05 2024-05-15T12:51:01.736 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.736 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.736 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:51:01.736 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.991247+0000", 2024-05-15T12:51:01.736 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.2679000000000001e-05 2024-05-15T12:51:01.736 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.737 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:51:01.737 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.737 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.737 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.737 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:53950 2.7 2:ebafda16:::benchmark_data_smithi177_78104_object53949:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-05-15T12:51:01.737 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:44.010201+0000", 2024-05-15T12:51:01.737 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 137.681303305, 2024-05-15T12:51:01.737 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.98117490200000002, 2024-05-15T12:51:01.737 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:51:01.737 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:01.737 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:51:01.737 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:51:01.737 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:01.737 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 53950 2024-05-15T12:51:01.738 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.738 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:51:01.738 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.738 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:51:01.738 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.010201+0000", 2024-05-15T12:51:01.738 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.738 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.738 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.738 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:51:01.738 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.010201+0000", 2024-05-15T12:51:01.738 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.738 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.738 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.738 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:51:01.739 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.010202+0000", 2024-05-15T12:51:01.739 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2389999999999999e-06 2024-05-15T12:51:01.739 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.739 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.739 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:51:01.739 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.010317+0000", 2024-05-15T12:51:01.739 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000114508 2024-05-15T12:51:01.739 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.739 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.739 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:51:01.739 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.010330+0000", 2024-05-15T12:51:01.739 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2704000000000001e-05 2024-05-15T12:51:01.739 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.739 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.740 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:51:01.740 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.010337+0000", 2024-05-15T12:51:01.740 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4769999999999997e-06 2024-05-15T12:51:01.740 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.740 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.740 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.740 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.010393+0000", 2024-05-15T12:51:01.740 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5593000000000001e-05 2024-05-15T12:51:01.740 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.740 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.740 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.740 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.010402+0000", 2024-05-15T12:51:01.740 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.4830000000000005e-06 2024-05-15T12:51:01.740 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.740 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.741 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.741 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.978685+0000", 2024-05-15T12:51:01.741 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.96828286500000005 2024-05-15T12:51:01.741 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.741 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.741 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:51:01.741 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.978772+0000", 2024-05-15T12:51:01.741 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7282999999999998e-05 2024-05-15T12:51:01.741 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.741 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.741 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3", 2024-05-15T12:51:01.741 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.978863+0000", 2024-05-15T12:51:01.741 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0649000000000006e-05 2024-05-15T12:51:01.741 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.741 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.742 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:51:01.742 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.983711+0000", 2024-05-15T12:51:01.742 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0048480950000000002 2024-05-15T12:51:01.742 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.742 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.742 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.742 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.987627+0000", 2024-05-15T12:51:01.742 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0039160250000000001 2024-05-15T12:51:01.742 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.742 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.742 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.742 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.991304+0000", 2024-05-15T12:51:01.742 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0036772530000000001 2024-05-15T12:51:01.742 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.743 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.743 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:51:01.743 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.991345+0000", 2024-05-15T12:51:01.743 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0540000000000001e-05 2024-05-15T12:51:01.743 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.743 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.743 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:51:01.743 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.991376+0000", 2024-05-15T12:51:01.743 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.1188e-05 2024-05-15T12:51:01.743 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.743 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:51:01.743 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.743 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.743 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.744 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:53970 2.7 2:eb1123f2:::benchmark_data_smithi177_78104_object53969:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-05-15T12:51:01.744 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:44.021992+0000", 2024-05-15T12:51:01.744 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 137.66951236099999, 2024-05-15T12:51:01.744 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.96960066899999997, 2024-05-15T12:51:01.744 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:51:01.744 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:01.744 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:51:01.744 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:51:01.744 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:01.744 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 53970 2024-05-15T12:51:01.744 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.744 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:51:01.744 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.744 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:51:01.745 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.021992+0000", 2024-05-15T12:51:01.745 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.745 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.745 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.745 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:51:01.745 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.021992+0000", 2024-05-15T12:51:01.745 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.745 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.745 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.745 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:51:01.745 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.021993+0000", 2024-05-15T12:51:01.746 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2330000000000001e-06 2024-05-15T12:51:01.746 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.746 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.746 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:51:01.746 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.022053+0000", 2024-05-15T12:51:01.746 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9552000000000002e-05 2024-05-15T12:51:01.746 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.746 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.746 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:51:01.746 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.022064+0000", 2024-05-15T12:51:01.747 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1547e-05 2024-05-15T12:51:01.747 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.747 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.747 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:51:01.747 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.022071+0000", 2024-05-15T12:51:01.747 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.161e-06 2024-05-15T12:51:01.747 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.747 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.747 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.747 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.022139+0000", 2024-05-15T12:51:01.747 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7954999999999996e-05 2024-05-15T12:51:01.747 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.747 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.747 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.748 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.022151+0000", 2024-05-15T12:51:01.748 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2870999999999999e-05 2024-05-15T12:51:01.748 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.748 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.748 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.748 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.979910+0000", 2024-05-15T12:51:01.748 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.95775874299999997 2024-05-15T12:51:01.748 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.748 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.748 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:51:01.748 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.979992+0000", 2024-05-15T12:51:01.748 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2089000000000006e-05 2024-05-15T12:51:01.748 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.749 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.749 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3", 2024-05-15T12:51:01.749 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.980098+0000", 2024-05-15T12:51:01.749 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000106147 2024-05-15T12:51:01.749 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.749 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.749 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:51:01.749 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.983718+0000", 2024-05-15T12:51:01.749 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0036195509999999999 2024-05-15T12:51:01.749 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.749 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.749 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.749 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.987661+0000", 2024-05-15T12:51:01.749 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0039434129999999998 2024-05-15T12:51:01.750 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.750 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.750 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.750 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.991436+0000", 2024-05-15T12:51:01.750 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0037748510000000001 2024-05-15T12:51:01.750 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.750 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.750 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:51:01.750 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.991512+0000", 2024-05-15T12:51:01.750 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.5842999999999999e-05 2024-05-15T12:51:01.750 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.750 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.750 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:51:01.751 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.991593+0000", 2024-05-15T12:51:01.751 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0712999999999998e-05 2024-05-15T12:51:01.751 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.751 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:51:01.751 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.751 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.751 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.751 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:53983 2.7 2:e4149689:::benchmark_data_smithi177_78104_object53982:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-05-15T12:51:01.751 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:44.032746+0000", 2024-05-15T12:51:01.751 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 137.658758557, 2024-05-15T12:51:01.751 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.95906061600000003, 2024-05-15T12:51:01.751 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:51:01.751 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:01.751 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:51:01.751 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:51:01.752 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:01.752 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 53983 2024-05-15T12:51:01.752 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.752 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:51:01.752 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.752 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:51:01.752 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.032746+0000", 2024-05-15T12:51:01.752 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.752 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.752 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.752 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:51:01.752 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.032746+0000", 2024-05-15T12:51:01.752 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.752 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.753 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.753 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:51:01.753 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.032747+0000", 2024-05-15T12:51:01.753 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.423e-06 2024-05-15T12:51:01.753 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.753 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.753 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:51:01.753 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.032798+0000", 2024-05-15T12:51:01.753 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0441e-05 2024-05-15T12:51:01.753 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.753 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.753 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:51:01.753 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.032815+0000", 2024-05-15T12:51:01.753 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7516999999999998e-05 2024-05-15T12:51:01.753 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.754 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.754 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:51:01.754 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.032824+0000", 2024-05-15T12:51:01.754 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.354e-06 2024-05-15T12:51:01.754 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.754 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.754 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.754 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.032892+0000", 2024-05-15T12:51:01.754 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8227999999999998e-05 2024-05-15T12:51:01.754 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.754 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.754 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.754 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.032905+0000", 2024-05-15T12:51:01.754 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2758000000000001e-05 2024-05-15T12:51:01.755 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.755 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.755 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.755 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.981285+0000", 2024-05-15T12:51:01.755 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.94838048500000005 2024-05-15T12:51:01.755 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.755 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.755 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:51:01.755 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.981377+0000", 2024-05-15T12:51:01.755 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1618000000000003e-05 2024-05-15T12:51:01.755 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.755 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.755 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3", 2024-05-15T12:51:01.755 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.981487+0000", 2024-05-15T12:51:01.756 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000110665 2024-05-15T12:51:01.756 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.756 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.756 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:51:01.756 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.986247+0000", 2024-05-15T12:51:01.756 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0047600170000000001 2024-05-15T12:51:01.756 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.756 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.756 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.756 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.987743+0000", 2024-05-15T12:51:01.756 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0014957410000000001 2024-05-15T12:51:01.756 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.756 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.756 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.757 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.991671+0000", 2024-05-15T12:51:01.757 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0039280319999999997 2024-05-15T12:51:01.757 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.757 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.757 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:51:01.757 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.991758+0000", 2024-05-15T12:51:01.757 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7239999999999998e-05 2024-05-15T12:51:01.757 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.757 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.757 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:51:01.757 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.991807+0000", 2024-05-15T12:51:01.757 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8096999999999997e-05 2024-05-15T12:51:01.757 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.758 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:51:01.758 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.758 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.758 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.758 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:53987 2.7 2:ef8f42e2:::benchmark_data_smithi177_78104_object53986:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-05-15T12:51:01.758 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:44.035509+0000", 2024-05-15T12:51:01.758 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 137.65599589000001, 2024-05-15T12:51:01.758 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.95648043699999996, 2024-05-15T12:51:01.758 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:51:01.758 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:01.758 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:51:01.758 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:51:01.758 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:01.758 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 53987 2024-05-15T12:51:01.759 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.759 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:51:01.759 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.759 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:51:01.759 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.035509+0000", 2024-05-15T12:51:01.759 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.759 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.759 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.759 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:51:01.759 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.035509+0000", 2024-05-15T12:51:01.759 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.759 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.759 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.760 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:51:01.760 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.035510+0000", 2024-05-15T12:51:01.760 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.035e-06 2024-05-15T12:51:01.760 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.760 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.760 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:51:01.760 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.035597+0000", 2024-05-15T12:51:01.760 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7658000000000007e-05 2024-05-15T12:51:01.760 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.760 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.760 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:51:01.760 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.035607+0000", 2024-05-15T12:51:01.760 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.6590000000000002e-06 2024-05-15T12:51:01.760 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.761 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.761 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:51:01.761 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.035613+0000", 2024-05-15T12:51:01.761 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.767e-06 2024-05-15T12:51:01.761 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.761 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.761 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.761 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.035668+0000", 2024-05-15T12:51:01.761 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5488000000000001e-05 2024-05-15T12:51:01.761 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.761 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.761 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.761 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.035678+0000", 2024-05-15T12:51:01.761 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0064000000000001e-05 2024-05-15T12:51:01.762 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.762 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.762 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.762 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.982551+0000", 2024-05-15T12:51:01.762 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.94687267799999997 2024-05-15T12:51:01.762 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.762 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.762 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:51:01.762 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.982628+0000", 2024-05-15T12:51:01.762 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7051999999999993e-05 2024-05-15T12:51:01.762 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.762 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.762 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3", 2024-05-15T12:51:01.763 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.982785+0000", 2024-05-15T12:51:01.763 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000156589 2024-05-15T12:51:01.763 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.763 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.763 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:51:01.763 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.987582+0000", 2024-05-15T12:51:01.763 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0047977159999999996 2024-05-15T12:51:01.763 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.763 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.763 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.763 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.990055+0000", 2024-05-15T12:51:01.763 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0024728520000000002 2024-05-15T12:51:01.763 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.764 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.764 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.764 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.991879+0000", 2024-05-15T12:51:01.764 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001824244 2024-05-15T12:51:01.764 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.764 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.764 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:51:01.764 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.991940+0000", 2024-05-15T12:51:01.764 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1074999999999994e-05 2024-05-15T12:51:01.764 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.764 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.764 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:51:01.764 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:44.991989+0000", 2024-05-15T12:51:01.764 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8560000000000003e-05 2024-05-15T12:51:01.765 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.765 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:51:01.765 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.765 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.765 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.765 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:90791 2.7 2:ede8a1cf:::benchmark_data_smithi177_78104_object90790:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:51:01.765 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:49:17.233646+0000", 2024-05-15T12:51:01.765 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 104.457858027, 2024-05-15T12:51:01.765 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1459873439999999, 2024-05-15T12:51:01.765 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:51:01.765 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T12:51:01.765 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:51:01.765 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:51:01.765 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:01.766 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 90791 2024-05-15T12:51:01.766 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.766 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:51:01.766 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.766 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:51:01.766 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.233646+0000", 2024-05-15T12:51:01.766 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.766 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.766 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.766 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:51:01.766 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.233646+0000", 2024-05-15T12:51:01.766 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.766 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.767 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.767 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:51:01.767 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.233647+0000", 2024-05-15T12:51:01.767 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.71e-07 2024-05-15T12:51:01.767 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.767 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.767 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:51:01.767 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.233716+0000", 2024-05-15T12:51:01.767 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8796999999999999e-05 2024-05-15T12:51:01.767 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.767 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.767 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:51:01.767 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.233737+0000", 2024-05-15T12:51:01.767 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0421e-05 2024-05-15T12:51:01.768 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.768 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.768 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:51:01.768 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.233744+0000", 2024-05-15T12:51:01.768 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0539999999999997e-06 2024-05-15T12:51:01.768 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.768 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.768 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.768 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.236353+0000", 2024-05-15T12:51:01.768 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0026095129999999999 2024-05-15T12:51:01.768 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.768 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.768 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:51:01.768 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.236477+0000", 2024-05-15T12:51:01.768 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000123428 2024-05-15T12:51:01.769 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.769 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.769 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3", 2024-05-15T12:51:01.769 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.236620+0000", 2024-05-15T12:51:01.769 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014386299999999999 2024-05-15T12:51:01.769 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.769 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.769 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:51:01.769 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.241511+0000", 2024-05-15T12:51:01.769 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0048901989999999996 2024-05-15T12:51:01.769 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.769 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.769 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.770 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.243337+0000", 2024-05-15T12:51:01.770 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001826041 2024-05-15T12:51:01.770 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.770 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.770 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.770 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.249571+0000", 2024-05-15T12:51:01.770 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0062345100000000004 2024-05-15T12:51:01.770 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.770 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.770 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.770 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.249584+0000", 2024-05-15T12:51:01.770 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3359e-05 2024-05-15T12:51:01.770 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.770 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.771 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.771 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.855870+0000", 2024-05-15T12:51:01.771 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.60628580300000001 2024-05-15T12:51:01.771 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.771 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.771 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.771 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.855883+0000", 2024-05-15T12:51:01.771 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2294e-05 2024-05-15T12:51:01.771 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.771 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.771 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.771 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.379560+0000", 2024-05-15T12:51:01.771 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5236770769999999 2024-05-15T12:51:01.771 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.772 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.772 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:51:01.772 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.379634+0000", 2024-05-15T12:51:01.772 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4114000000000002e-05 2024-05-15T12:51:01.772 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.772 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:51:01.772 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.772 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.772 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.772 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:90794 2.7 2:ee131b04:::benchmark_data_smithi177_78104_object90793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:51:01.772 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:49:17.238186+0000", 2024-05-15T12:51:01.772 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 104.453318861, 2024-05-15T12:51:01.772 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1417517950000002, 2024-05-15T12:51:01.773 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:51:01.773 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T12:51:01.773 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:51:01.773 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:51:01.773 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:01.773 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 90794 2024-05-15T12:51:01.773 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.773 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:51:01.773 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.773 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:51:01.773 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.238186+0000", 2024-05-15T12:51:01.773 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.773 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.773 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.774 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:51:01.774 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.238186+0000", 2024-05-15T12:51:01.774 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.774 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.774 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.774 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:51:01.774 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.238187+0000", 2024-05-15T12:51:01.774 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2890000000000001e-06 2024-05-15T12:51:01.774 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.774 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.774 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:51:01.774 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.238230+0000", 2024-05-15T12:51:01.774 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2707999999999998e-05 2024-05-15T12:51:01.775 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.775 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.775 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:51:01.775 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.238246+0000", 2024-05-15T12:51:01.775 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6101999999999999e-05 2024-05-15T12:51:01.775 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.775 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.775 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:51:01.775 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.238253+0000", 2024-05-15T12:51:01.775 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7279999999999995e-06 2024-05-15T12:51:01.775 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.775 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.775 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.775 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.239026+0000", 2024-05-15T12:51:01.776 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00077243500000000003 2024-05-15T12:51:01.776 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.776 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.776 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:51:01.776 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.239133+0000", 2024-05-15T12:51:01.776 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000107557 2024-05-15T12:51:01.776 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.776 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.776 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3", 2024-05-15T12:51:01.776 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.239228+0000", 2024-05-15T12:51:01.776 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.4159000000000004e-05 2024-05-15T12:51:01.776 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.776 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.776 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:51:01.777 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.241664+0000", 2024-05-15T12:51:01.777 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0024366359999999998 2024-05-15T12:51:01.777 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.777 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.777 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.777 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.249638+0000", 2024-05-15T12:51:01.777 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0079736939999999999 2024-05-15T12:51:01.777 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.777 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.777 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.777 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.249649+0000", 2024-05-15T12:51:01.777 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0832000000000001e-05 2024-05-15T12:51:01.777 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.777 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.778 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.778 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.855991+0000", 2024-05-15T12:51:01.778 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.60634224400000003 2024-05-15T12:51:01.778 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.778 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.778 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.778 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856003+0000", 2024-05-15T12:51:01.778 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.183e-05 2024-05-15T12:51:01.778 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.778 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.778 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.778 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.379870+0000", 2024-05-15T12:51:01.778 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5238669819999999 2024-05-15T12:51:01.778 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.779 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.779 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:51:01.779 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.379937+0000", 2024-05-15T12:51:01.779 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7599000000000001e-05 2024-05-15T12:51:01.779 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.779 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:51:01.779 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.779 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.779 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.779 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:90802 2.7 2:e02a2e8d:::benchmark_data_smithi177_78104_object90801:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-05-15T12:51:01.779 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:49:17.246620+0000", 2024-05-15T12:51:01.779 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 104.444884661, 2024-05-15T12:51:01.779 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1558356180000002, 2024-05-15T12:51:01.779 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:51:01.780 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:01.780 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:51:01.780 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:51:01.780 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:01.780 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 90802 2024-05-15T12:51:01.780 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.780 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:51:01.780 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.780 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:51:01.780 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.246620+0000", 2024-05-15T12:51:01.780 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.780 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.780 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.780 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:51:01.781 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.246620+0000", 2024-05-15T12:51:01.781 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.781 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.781 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.781 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:51:01.781 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.246621+0000", 2024-05-15T12:51:01.781 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3349999999999999e-06 2024-05-15T12:51:01.781 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.781 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.781 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:51:01.781 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.246683+0000", 2024-05-15T12:51:01.781 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1421999999999996e-05 2024-05-15T12:51:01.781 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.781 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.782 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:51:01.782 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.246698+0000", 2024-05-15T12:51:01.782 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5852e-05 2024-05-15T12:51:01.782 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.782 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.782 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:51:01.782 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.246706+0000", 2024-05-15T12:51:01.782 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7379999999999994e-06 2024-05-15T12:51:01.782 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.782 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.782 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.782 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.246773+0000", 2024-05-15T12:51:01.782 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6762999999999998e-05 2024-05-15T12:51:01.782 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.783 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.783 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op must wait for map", 2024-05-15T12:51:01.783 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.246780+0000", 2024-05-15T12:51:01.783 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2980000000000003e-06 2024-05-15T12:51:01.783 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.783 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.783 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.783 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.249680+0000", 2024-05-15T12:51:01.783 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0028993619999999999 2024-05-15T12:51:01.783 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.783 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.783 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.783 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.249685+0000", 2024-05-15T12:51:01.785 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2530000000000004e-06 2024-05-15T12:51:01.785 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.785 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.785 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.785 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856040+0000", 2024-05-15T12:51:01.785 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.60635493299999998 2024-05-15T12:51:01.785 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.785 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.785 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.786 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856050+0000", 2024-05-15T12:51:01.786 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0504e-05 2024-05-15T12:51:01.786 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.786 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.786 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.786 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.380002+0000", 2024-05-15T12:51:01.786 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5239516829999999 2024-05-15T12:51:01.786 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.786 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.786 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:51:01.786 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.380127+0000", 2024-05-15T12:51:01.786 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000125389 2024-05-15T12:51:01.786 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.786 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.787 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3,10", 2024-05-15T12:51:01.787 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.380288+0000", 2024-05-15T12:51:01.787 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000160163 2024-05-15T12:51:01.787 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.787 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.787 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.787 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.385190+0000", 2024-05-15T12:51:01.787 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0049021289999999999 2024-05-15T12:51:01.787 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.787 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.787 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:51:01.787 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.387946+0000", 2024-05-15T12:51:01.787 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.002756859 2024-05-15T12:51:01.787 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.788 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.788 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.788 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.400172+0000", 2024-05-15T12:51:01.788 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.012225234 2024-05-15T12:51:01.788 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.788 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.788 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.788 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.402313+0000", 2024-05-15T12:51:01.788 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.002141146 2024-05-15T12:51:01.788 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.788 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.788 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:51:01.788 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.402375+0000", 2024-05-15T12:51:01.788 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.2288999999999999e-05 2024-05-15T12:51:01.789 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.789 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.789 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:51:01.789 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.402455+0000", 2024-05-15T12:51:01.789 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0265999999999993e-05 2024-05-15T12:51:01.789 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.789 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:51:01.789 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.789 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.789 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.789 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:90803 2.7 2:ec92ac45:::benchmark_data_smithi177_78104_object90802:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-05-15T12:51:01.789 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:49:17.250972+0000", 2024-05-15T12:51:01.789 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 104.440532198, 2024-05-15T12:51:01.789 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1517382629999999, 2024-05-15T12:51:01.789 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:51:01.790 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:01.790 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:51:01.790 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:51:01.790 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:01.790 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 90803 2024-05-15T12:51:01.790 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.790 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:51:01.790 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.790 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:51:01.790 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.250972+0000", 2024-05-15T12:51:01.790 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.790 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.790 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.791 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:51:01.791 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.250972+0000", 2024-05-15T12:51:01.791 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.791 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.791 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.791 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:51:01.791 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.250973+0000", 2024-05-15T12:51:01.791 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.226e-06 2024-05-15T12:51:01.791 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.791 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.791 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:51:01.791 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.251042+0000", 2024-05-15T12:51:01.791 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8552000000000004e-05 2024-05-15T12:51:01.791 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.791 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.792 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:51:01.792 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.251053+0000", 2024-05-15T12:51:01.792 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0567000000000001e-05 2024-05-15T12:51:01.792 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.792 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.792 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:51:01.792 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.251059+0000", 2024-05-15T12:51:01.792 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3459999999999999e-06 2024-05-15T12:51:01.792 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.792 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.792 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.792 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.251103+0000", 2024-05-15T12:51:01.792 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4314999999999998e-05 2024-05-15T12:51:01.792 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.793 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.793 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.793 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.251113+0000", 2024-05-15T12:51:01.793 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.4169999999999998e-06 2024-05-15T12:51:01.793 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.793 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.793 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.793 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856075+0000", 2024-05-15T12:51:01.793 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.60496192999999998 2024-05-15T12:51:01.793 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.793 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.793 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.793 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856082+0000", 2024-05-15T12:51:01.794 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4490000000000002e-06 2024-05-15T12:51:01.794 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.794 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.794 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.794 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.382594+0000", 2024-05-15T12:51:01.794 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.526511927 2024-05-15T12:51:01.794 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.794 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.794 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:51:01.794 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.382695+0000", 2024-05-15T12:51:01.794 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000101407 2024-05-15T12:51:01.794 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.794 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.794 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3,10", 2024-05-15T12:51:01.795 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.382806+0000", 2024-05-15T12:51:01.795 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000110837 2024-05-15T12:51:01.795 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.795 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.795 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.795 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.385250+0000", 2024-05-15T12:51:01.795 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0024435669999999998 2024-05-15T12:51:01.795 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.795 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.795 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:51:01.795 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.387954+0000", 2024-05-15T12:51:01.795 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0027041000000000001 2024-05-15T12:51:01.795 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.795 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.795 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.796 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.400230+0000", 2024-05-15T12:51:01.796 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.012276545999999999 2024-05-15T12:51:01.796 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.796 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.796 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.796 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.402518+0000", 2024-05-15T12:51:01.796 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00228707 2024-05-15T12:51:01.796 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.796 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.796 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:51:01.796 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.402585+0000", 2024-05-15T12:51:01.796 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6982000000000003e-05 2024-05-15T12:51:01.796 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.796 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.797 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:51:01.797 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.402711+0000", 2024-05-15T12:51:01.797 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012602499999999999 2024-05-15T12:51:01.797 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.797 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:51:01.797 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.797 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.797 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.797 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:90810 2.7 2:eb1c506f:::benchmark_data_smithi177_78104_object90809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-05-15T12:51:01.797 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:49:17.261408+0000", 2024-05-15T12:51:01.797 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 104.43009611399999, 2024-05-15T12:51:01.797 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.141452954, 2024-05-15T12:51:01.797 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:51:01.797 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:01.798 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:51:01.798 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:51:01.798 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:01.798 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 90810 2024-05-15T12:51:01.798 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.798 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:51:01.798 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.798 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:51:01.798 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261408+0000", 2024-05-15T12:51:01.798 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.798 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.798 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.798 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:51:01.798 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261408+0000", 2024-05-15T12:51:01.798 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.799 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.799 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.799 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:51:01.799 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261410+0000", 2024-05-15T12:51:01.799 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5629999999999999e-06 2024-05-15T12:51:01.799 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.799 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.799 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:51:01.799 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261473+0000", 2024-05-15T12:51:01.799 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3189999999999996e-05 2024-05-15T12:51:01.799 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.799 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.799 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:51:01.800 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261485+0000", 2024-05-15T12:51:01.800 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2306e-05 2024-05-15T12:51:01.800 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.800 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.800 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:51:01.800 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261494+0000", 2024-05-15T12:51:01.800 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.9150000000000002e-06 2024-05-15T12:51:01.800 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.800 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.800 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.800 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261546+0000", 2024-05-15T12:51:01.800 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1975999999999998e-05 2024-05-15T12:51:01.800 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.800 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.800 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.801 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261553+0000", 2024-05-15T12:51:01.801 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.1779999999999997e-06 2024-05-15T12:51:01.801 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.801 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.801 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.801 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856116+0000", 2024-05-15T12:51:01.801 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.59456220500000001 2024-05-15T12:51:01.801 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.801 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.801 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.801 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856121+0000", 2024-05-15T12:51:01.801 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5300000000000004e-06 2024-05-15T12:51:01.801 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.801 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.802 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.802 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.385305+0000", 2024-05-15T12:51:01.802 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.529184074 2024-05-15T12:51:01.802 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.802 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.802 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:51:01.802 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.385421+0000", 2024-05-15T12:51:01.802 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00011531699999999999 2024-05-15T12:51:01.802 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.802 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.802 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3,10", 2024-05-15T12:51:01.802 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.385558+0000", 2024-05-15T12:51:01.802 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013759800000000001 2024-05-15T12:51:01.802 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.803 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.803 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.803 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.390650+0000", 2024-05-15T12:51:01.803 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0050915530000000004 2024-05-15T12:51:01.803 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.803 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.803 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:51:01.803 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.390728+0000", 2024-05-15T12:51:01.803 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8359e-05 2024-05-15T12:51:01.803 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.803 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.803 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.803 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.400258+0000", 2024-05-15T12:51:01.803 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0095295580000000005 2024-05-15T12:51:01.804 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.804 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.804 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.804 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.402758+0000", 2024-05-15T12:51:01.804 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.002500071 2024-05-15T12:51:01.804 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.804 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.804 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:51:01.804 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.402795+0000", 2024-05-15T12:51:01.804 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7286000000000002e-05 2024-05-15T12:51:01.804 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.804 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.804 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:51:01.804 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.402861+0000", 2024-05-15T12:51:01.805 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6274999999999998e-05 2024-05-15T12:51:01.805 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.805 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:51:01.805 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.805 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.805 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.805 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:90812 2.7 2:e1d3b6e4:::benchmark_data_smithi177_78104_object90811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-05-15T12:51:01.805 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:49:17.261825+0000", 2024-05-15T12:51:01.805 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 104.429679562, 2024-05-15T12:51:01.805 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1412600579999999, 2024-05-15T12:51:01.805 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:51:01.805 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:01.805 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:51:01.805 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:51:01.806 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:01.806 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 90812 2024-05-15T12:51:01.806 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.806 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:51:01.806 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.806 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:51:01.806 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261825+0000", 2024-05-15T12:51:01.806 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.806 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.806 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.806 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:51:01.806 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261825+0000", 2024-05-15T12:51:01.806 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.806 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.807 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.807 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:51:01.807 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261826+0000", 2024-05-15T12:51:01.807 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7999999999999995e-07 2024-05-15T12:51:01.807 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.807 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.807 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:51:01.807 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261892+0000", 2024-05-15T12:51:01.807 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6837999999999999e-05 2024-05-15T12:51:01.807 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.807 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.807 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:51:01.807 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261902+0000", 2024-05-15T12:51:01.807 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.9229999999999997e-06 2024-05-15T12:51:01.807 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.808 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.808 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:51:01.808 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261908+0000", 2024-05-15T12:51:01.808 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.366e-06 2024-05-15T12:51:01.808 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.808 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.808 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.808 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261952+0000", 2024-05-15T12:51:01.808 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4563e-05 2024-05-15T12:51:01.808 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.808 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.808 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.808 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261959+0000", 2024-05-15T12:51:01.808 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7789999999999998e-06 2024-05-15T12:51:01.809 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.809 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.809 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.809 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856157+0000", 2024-05-15T12:51:01.809 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.59419784499999995 2024-05-15T12:51:01.809 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.809 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.809 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.809 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856173+0000", 2024-05-15T12:51:01.809 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5930000000000002e-05 2024-05-15T12:51:01.809 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.809 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.809 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.809 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.387998+0000", 2024-05-15T12:51:01.809 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5318250820000001 2024-05-15T12:51:01.810 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.810 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.810 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:51:01.810 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.388139+0000", 2024-05-15T12:51:01.810 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014140499999999999 2024-05-15T12:51:01.810 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.810 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.810 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3,10", 2024-05-15T12:51:01.810 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.388305+0000", 2024-05-15T12:51:01.810 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00016605 2024-05-15T12:51:01.810 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.810 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.810 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.811 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.390707+0000", 2024-05-15T12:51:01.811 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0024013989999999998 2024-05-15T12:51:01.811 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.811 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.811 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:51:01.811 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.390736+0000", 2024-05-15T12:51:01.811 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.9150999999999999e-05 2024-05-15T12:51:01.811 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.811 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.811 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.811 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.400280+0000", 2024-05-15T12:51:01.811 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0095440190000000008 2024-05-15T12:51:01.811 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.811 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.811 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.812 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.402906+0000", 2024-05-15T12:51:01.812 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0026255850000000002 2024-05-15T12:51:01.812 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.812 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.812 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:51:01.812 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.402967+0000", 2024-05-15T12:51:01.812 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1748000000000006e-05 2024-05-15T12:51:01.812 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.812 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.812 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:51:01.812 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.403085+0000", 2024-05-15T12:51:01.812 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000117695 2024-05-15T12:51:01.812 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.812 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:51:01.813 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.813 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.813 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.813 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:90830 2.7 2:e14b3430:::benchmark_data_smithi177_78104_object90829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-05-15T12:51:01.813 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:49:17.287740+0000", 2024-05-15T12:51:01.813 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 104.403764325, 2024-05-15T12:51:01.813 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1222937009999998, 2024-05-15T12:51:01.813 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:51:01.813 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:01.813 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:51:01.813 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:51:01.813 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:01.813 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 90830 2024-05-15T12:51:01.813 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.814 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:51:01.814 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.814 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:51:01.814 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.287740+0000", 2024-05-15T12:51:01.814 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.814 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.814 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.814 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:51:01.814 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.287740+0000", 2024-05-15T12:51:01.814 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.814 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.814 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.814 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:51:01.815 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.287742+0000", 2024-05-15T12:51:01.815 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.278e-06 2024-05-15T12:51:01.815 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.815 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.815 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:51:01.815 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.287776+0000", 2024-05-15T12:51:01.815 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.4022000000000001e-05 2024-05-15T12:51:01.815 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.815 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.815 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:51:01.815 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.287798+0000", 2024-05-15T12:51:01.815 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1107999999999998e-05 2024-05-15T12:51:01.815 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.815 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.816 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:51:01.816 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.287808+0000", 2024-05-15T12:51:01.816 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0091e-05 2024-05-15T12:51:01.816 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.816 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.816 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.816 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.287874+0000", 2024-05-15T12:51:01.816 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.5922999999999999e-05 2024-05-15T12:51:01.816 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.816 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.816 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.816 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.287883+0000", 2024-05-15T12:51:01.816 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.8840000000000006e-06 2024-05-15T12:51:01.816 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.817 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.817 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.817 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856234+0000", 2024-05-15T12:51:01.817 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.568351037 2024-05-15T12:51:01.817 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.817 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.817 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.817 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856240+0000", 2024-05-15T12:51:01.817 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.361e-06 2024-05-15T12:51:01.817 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.817 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.817 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.817 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.390752+0000", 2024-05-15T12:51:01.817 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5345125959999999 2024-05-15T12:51:01.817 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.818 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.818 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:51:01.818 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.390833+0000", 2024-05-15T12:51:01.818 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0131999999999998e-05 2024-05-15T12:51:01.818 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.818 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.818 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3,10", 2024-05-15T12:51:01.818 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.390936+0000", 2024-05-15T12:51:01.818 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000103118 2024-05-15T12:51:01.818 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.818 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.818 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.818 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.397027+0000", 2024-05-15T12:51:01.819 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0060916310000000001 2024-05-15T12:51:01.819 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.819 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.819 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:51:01.819 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.397044+0000", 2024-05-15T12:51:01.819 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6484e-05 2024-05-15T12:51:01.819 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.819 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.819 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.819 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.400304+0000", 2024-05-15T12:51:01.819 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0032599310000000002 2024-05-15T12:51:01.819 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.819 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.819 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.819 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.409895+0000", 2024-05-15T12:51:01.820 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0095915229999999994 2024-05-15T12:51:01.820 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.820 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.820 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:51:01.820 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.409938+0000", 2024-05-15T12:51:01.820 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2889999999999998e-05 2024-05-15T12:51:01.820 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.820 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.820 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:51:01.820 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.410034+0000", 2024-05-15T12:51:01.820 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.5692000000000005e-05 2024-05-15T12:51:01.820 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.820 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:51:01.820 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.821 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.821 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.821 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:90834 2.7 2:eb30fe0e:::benchmark_data_smithi177_78104_object90833:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-05-15T12:51:01.821 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:49:17.293510+0000", 2024-05-15T12:51:01.821 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 104.397994373, 2024-05-15T12:51:01.821 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1167819890000001, 2024-05-15T12:51:01.821 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:51:01.821 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:01.821 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:51:01.821 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:51:01.821 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:01.821 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 90834 2024-05-15T12:51:01.821 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.821 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:51:01.821 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.821 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:51:01.822 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.293510+0000", 2024-05-15T12:51:01.822 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.822 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.822 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.822 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:51:01.822 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.293510+0000", 2024-05-15T12:51:01.822 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.822 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.822 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.822 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:51:01.822 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.293512+0000", 2024-05-15T12:51:01.822 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.44e-06 2024-05-15T12:51:01.822 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.822 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.823 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:51:01.823 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.293552+0000", 2024-05-15T12:51:01.823 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0200999999999997e-05 2024-05-15T12:51:01.823 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.823 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.823 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:51:01.823 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.293567+0000", 2024-05-15T12:51:01.823 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5213e-05 2024-05-15T12:51:01.823 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.823 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.823 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:51:01.823 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.293576+0000", 2024-05-15T12:51:01.823 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.9369999999999993e-06 2024-05-15T12:51:01.823 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.824 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.824 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.824 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.293645+0000", 2024-05-15T12:51:01.824 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9261e-05 2024-05-15T12:51:01.824 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.824 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.824 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.824 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.293654+0000", 2024-05-15T12:51:01.824 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0799999999999995e-06 2024-05-15T12:51:01.824 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.824 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.824 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.824 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856256+0000", 2024-05-15T12:51:01.824 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.56260164999999995 2024-05-15T12:51:01.825 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.825 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.825 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.825 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856260+0000", 2024-05-15T12:51:01.825 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0060000000000003e-06 2024-05-15T12:51:01.825 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.825 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.825 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.825 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.392511+0000", 2024-05-15T12:51:01.825 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.536250683 2024-05-15T12:51:01.825 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.825 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.825 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:51:01.826 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.392632+0000", 2024-05-15T12:51:01.826 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012167000000000001 2024-05-15T12:51:01.826 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.826 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.826 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3,10", 2024-05-15T12:51:01.826 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.392749+0000", 2024-05-15T12:51:01.826 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00011702899999999999 2024-05-15T12:51:01.826 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.826 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.826 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:51:01.826 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.397054+0000", 2024-05-15T12:51:01.826 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0043043259999999998 2024-05-15T12:51:01.826 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.827 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.827 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.827 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.397784+0000", 2024-05-15T12:51:01.827 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00073027400000000005 2024-05-15T12:51:01.827 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.827 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.827 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.827 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.400327+0000", 2024-05-15T12:51:01.827 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0025431529999999998 2024-05-15T12:51:01.827 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.827 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.827 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.827 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.410094+0000", 2024-05-15T12:51:01.827 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0097673640000000006 2024-05-15T12:51:01.828 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.828 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.828 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:51:01.828 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.410168+0000", 2024-05-15T12:51:01.828 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3959000000000001e-05 2024-05-15T12:51:01.828 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.828 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.828 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:51:01.828 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.410292+0000", 2024-05-15T12:51:01.828 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000123743 2024-05-15T12:51:01.828 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.828 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:51:01.828 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.828 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.829 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.829 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:90857 2.7 2:e20f5260:::benchmark_data_smithi177_78104_object90856:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-05-15T12:51:01.829 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:49:17.347336+0000", 2024-05-15T12:51:01.829 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 104.34416802299999, 2024-05-15T12:51:01.829 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0631595960000002, 2024-05-15T12:51:01.829 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:51:01.829 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:01.829 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:51:01.829 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:51:01.829 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:01.829 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 90857 2024-05-15T12:51:01.829 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.829 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:51:01.829 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.830 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:51:01.830 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.347336+0000", 2024-05-15T12:51:01.830 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.830 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.830 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.830 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:51:01.830 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.347336+0000", 2024-05-15T12:51:01.830 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:51:01.830 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.830 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.830 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:51:01.830 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.347338+0000", 2024-05-15T12:51:01.830 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9240000000000001e-06 2024-05-15T12:51:01.831 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.831 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.831 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:51:01.831 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.347368+0000", 2024-05-15T12:51:01.831 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.9694000000000001e-05 2024-05-15T12:51:01.831 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.831 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.831 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:51:01.831 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.347385+0000", 2024-05-15T12:51:01.831 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6595999999999998e-05 2024-05-15T12:51:01.831 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.831 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.831 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:51:01.831 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.347392+0000", 2024-05-15T12:51:01.832 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7219999999999999e-06 2024-05-15T12:51:01.832 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.832 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.832 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.832 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.347455+0000", 2024-05-15T12:51:01.832 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.2726999999999998e-05 2024-05-15T12:51:01.832 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.832 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.832 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.832 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.347464+0000", 2024-05-15T12:51:01.832 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8100000000000004e-06 2024-05-15T12:51:01.832 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.832 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.832 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.833 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856284+0000", 2024-05-15T12:51:01.833 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.50881985900000004 2024-05-15T12:51:01.833 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.833 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.833 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:51:01.833 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856294+0000", 2024-05-15T12:51:01.833 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0404000000000001e-05 2024-05-15T12:51:01.833 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.833 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.833 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:51:01.833 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.394917+0000", 2024-05-15T12:51:01.833 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5386228559999999 2024-05-15T12:51:01.833 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.833 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.833 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:51:01.834 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.395044+0000", 2024-05-15T12:51:01.834 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012691499999999999 2024-05-15T12:51:01.834 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.834 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.834 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3,10", 2024-05-15T12:51:01.834 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.395187+0000", 2024-05-15T12:51:01.834 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000143468 2024-05-15T12:51:01.834 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.834 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.834 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:51:01.834 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.397060+0000", 2024-05-15T12:51:01.834 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0018721020000000001 2024-05-15T12:51:01.834 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.835 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.835 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.835 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.397823+0000", 2024-05-15T12:51:01.835 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00076354299999999999 2024-05-15T12:51:01.835 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.835 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.835 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.835 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.403147+0000", 2024-05-15T12:51:01.835 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0053240650000000002 2024-05-15T12:51:01.835 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.835 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.835 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:01.835 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.410339+0000", 2024-05-15T12:51:01.835 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0071920480000000004 2024-05-15T12:51:01.836 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.836 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.836 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:51:01.836 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.410392+0000", 2024-05-15T12:51:01.836 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2840999999999998e-05 2024-05-15T12:51:01.836 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:51:01.836 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:51:01.836 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:51:01.836 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.410496+0000", 2024-05-15T12:51:01.836 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000104022 2024-05-15T12:51:01.836 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.836 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:51:01.836 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.836 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:51:01.836 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:51:01.837 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T12:51:01.837 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:51:01.838 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:01 smithi027 ceph-mon[39795]: pgmap v520: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 3.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 709 op/s; 3809/177948 objects misplaced (2.141%) 2024-05-15T12:51:01.838 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:01 smithi027 ceph-mon[39795]: 2.d scrub starts 2024-05-15T12:51:01.838 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:01 smithi120 ceph-mon[43961]: pgmap v520: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 3.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 709 op/s; 3809/177948 objects misplaced (2.141%) 2024-05-15T12:51:01.838 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:01 smithi120 ceph-mon[43961]: 2.d scrub starts 2024-05-15T12:51:01.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:01 smithi177 ceph-mon[44303]: pgmap v520: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 3.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 709 op/s; 3809/177948 objects misplaced (2.141%) 2024-05-15T12:51:01.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:01 smithi177 ceph-mon[44303]: 2.d scrub starts 2024-05-15T12:51:01.934 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:02.004 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:51:02.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:02.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:51:02.112 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:51:02.206 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:02.621 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.4 to 1 (10000) 2024-05-15T12:51:02.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:02 smithi177 ceph-mon[44303]: 2.d scrub ok 2024-05-15T12:51:02.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:02 smithi177 ceph-mon[44303]: 2.1 scrub starts 2024-05-15T12:51:02.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:02 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:51:02.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:02 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:51:02.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:02 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1554922868' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T12:51:03.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:02 smithi027 ceph-mon[39795]: 2.d scrub ok 2024-05-15T12:51:03.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:02 smithi027 ceph-mon[39795]: 2.1 scrub starts 2024-05-15T12:51:03.024 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:02 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:51:03.024 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:02 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:51:03.024 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:02 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1554922868' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T12:51:03.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:02 smithi120 ceph-mon[43961]: 2.d scrub ok 2024-05-15T12:51:03.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:02 smithi120 ceph-mon[43961]: 2.1 scrub starts 2024-05-15T12:51:03.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:02 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:51:03.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:02 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:51:03.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:02 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1554922868' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T12:51:03.484 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:51:03.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:03.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:51:03.533 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 5 1 2024-05-15T12:51:03.595 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:51:03.634 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:03.662 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:51:03.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:03.668 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:51:03.766 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:03 smithi027 ceph-mon[39795]: pgmap v521: 17 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 15 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 858 op/s; 3170/170778 objects misplaced (1.856%); 2.7 MiB/s, 42 objects/s recovering 2024-05-15T12:51:03.766 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:03 smithi027 ceph-mon[39795]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-05-15T12:51:03.766 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:03 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1554922868' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T12:51:03.766 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:03 smithi027 ceph-mon[39795]: osdmap e181: 12 total, 12 up, 11 in 2024-05-15T12:51:03.766 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:03 smithi027 ceph-mon[39795]: osdmap e182: 12 total, 12 up, 11 in 2024-05-15T12:51:03.766 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:03 smithi027 ceph-mon[39795]: 2.1 scrub ok 2024-05-15T12:51:03.769 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:51:03.797 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:03.824 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:51:03.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:03.829 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:51:03.833 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:51:03.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:03 smithi177 ceph-mon[44303]: pgmap v521: 17 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 15 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 858 op/s; 3170/170778 objects misplaced (1.856%); 2.7 MiB/s, 42 objects/s recovering 2024-05-15T12:51:03.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:03 smithi177 ceph-mon[44303]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-05-15T12:51:03.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:03 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1554922868' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T12:51:03.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:03 smithi177 ceph-mon[44303]: osdmap e181: 12 total, 12 up, 11 in 2024-05-15T12:51:03.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:03 smithi177 ceph-mon[44303]: osdmap e182: 12 total, 12 up, 11 in 2024-05-15T12:51:03.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:03 smithi177 ceph-mon[44303]: 2.1 scrub ok 2024-05-15T12:51:03.930 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:51:03.968 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:03.999 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:51:04.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:04.003 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:51:04.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:03 smithi120 ceph-mon[43961]: pgmap v521: 17 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 15 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 858 op/s; 3170/170778 objects misplaced (1.856%); 2.7 MiB/s, 42 objects/s recovering 2024-05-15T12:51:04.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:03 smithi120 ceph-mon[43961]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-05-15T12:51:04.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:03 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1554922868' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T12:51:04.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:03 smithi120 ceph-mon[43961]: osdmap e181: 12 total, 12 up, 11 in 2024-05-15T12:51:04.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:03 smithi120 ceph-mon[43961]: osdmap e182: 12 total, 12 up, 11 in 2024-05-15T12:51:04.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:03 smithi120 ceph-mon[43961]: 2.1 scrub ok 2024-05-15T12:51:04.103 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:51:04.196 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:04.321 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:51:04.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:04.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:51:04.427 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:51:04.457 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:04.479 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:51:04.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:04.485 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:51:04.586 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:51:04.665 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:04.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:04 smithi177 ceph-mon[44303]: pgmap v524: 17 pgs: 2 active+clean+scrubbing, 1 active+remapped+backfilling, 14 active+clean; 3.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 861 op/s; 3170/165162 objects misplaced (1.919%); 3.5 MiB/s, 55 objects/s recovering 2024-05-15T12:51:05.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:04 smithi027 ceph-mon[39795]: pgmap v524: 17 pgs: 2 active+clean+scrubbing, 1 active+remapped+backfilling, 14 active+clean; 3.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 861 op/s; 3170/165162 objects misplaced (1.919%); 3.5 MiB/s, 55 objects/s recovering 2024-05-15T12:51:05.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:04 smithi120 ceph-mon[43961]: pgmap v524: 17 pgs: 2 active+clean+scrubbing, 1 active+remapped+backfilling, 14 active+clean; 3.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 861 op/s; 3170/165162 objects misplaced (1.919%); 3.5 MiB/s, 55 objects/s recovering 2024-05-15T12:51:05.355 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-05-15T12:51:05.625 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:51:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T12:51:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:51:05.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:05.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:51:05.724 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:51:05.761 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:51:05.844 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:05.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:05 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4252564601' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:51:05.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:05 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:51:05.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:05 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:51:05.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:05 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T12:51:06.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:05 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4252564601' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:51:06.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:05 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:51:06.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:05 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:51:06.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:05 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T12:51:06.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:05 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4252564601' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:51:06.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:05 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:51:06.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:05 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:51:06.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:05 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T12:51:06.743 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:51:06.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:06.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:51:06.851 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:51:06.923 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:06 smithi027 ceph-mon[39795]: pgmap v525: 17 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 15 active+clean; 3.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 908 op/s; 2286/160818 objects misplaced (1.421%); 8.6 MiB/s, 136 objects/s recovering 2024-05-15T12:51:06.923 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:06 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:51:06.923 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:06 smithi027 ceph-mon[39795]: osdmap e183: 12 total, 12 up, 11 in 2024-05-15T12:51:06.929 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:06.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:06 smithi177 ceph-mon[44303]: pgmap v525: 17 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 15 active+clean; 3.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 908 op/s; 2286/160818 objects misplaced (1.421%); 8.6 MiB/s, 136 objects/s recovering 2024-05-15T12:51:06.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:06 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:51:06.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:06 smithi177 ceph-mon[44303]: osdmap e183: 12 total, 12 up, 11 in 2024-05-15T12:51:07.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:06 smithi120 ceph-mon[43961]: pgmap v525: 17 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 15 active+clean; 3.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 908 op/s; 2286/160818 objects misplaced (1.421%); 8.6 MiB/s, 136 objects/s recovering 2024-05-15T12:51:07.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:06 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:51:07.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:06 smithi120 ceph-mon[43961]: osdmap e183: 12 total, 12 up, 11 in 2024-05-15T12:51:07.169 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:51:07.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:07.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:51:07.274 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:51:07.349 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:07.920 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:51:07.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:07.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:51:08.047 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:51:08.110 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:08.584 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:51:08.584 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:51:08.584 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:51:08.584 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:51:08.584 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:51:08.584 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:51:08.584 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:51:08.588 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:51:08.607 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:51:08.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:08.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:51:08.626 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:51:08.663 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:51:08.700 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:51:08.723 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:51:08.759 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:08.866 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:51:08.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:08.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:51:08.946 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:51:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:51:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:51:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:51:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:51:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:51:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:51:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:51:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:51:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:51:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:51:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:51:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:51:08.970 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:51:09.025 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:09.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:08 smithi027 ceph-mon[39795]: pgmap v527: 17 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 15 active+clean; 3.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 719 op/s; 2286/157833 objects misplaced (1.448%); 7.3 MiB/s, 116 objects/s recovering 2024-05-15T12:51:09.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:08 smithi027 ceph-mon[39795]: osdmap e184: 12 total, 12 up, 11 in 2024-05-15T12:51:09.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:08 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/239843948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:51:09.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:08 smithi120 ceph-mon[43961]: pgmap v527: 17 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 15 active+clean; 3.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 719 op/s; 2286/157833 objects misplaced (1.448%); 7.3 MiB/s, 116 objects/s recovering 2024-05-15T12:51:09.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:08 smithi120 ceph-mon[43961]: osdmap e184: 12 total, 12 up, 11 in 2024-05-15T12:51:09.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:08 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/239843948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:51:09.309 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:51:09.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:09.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:51:09.415 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:51:09.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:08 smithi177 ceph-mon[44303]: pgmap v527: 17 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 15 active+clean; 3.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 719 op/s; 2286/157833 objects misplaced (1.448%); 7.3 MiB/s, 116 objects/s recovering 2024-05-15T12:51:09.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:08 smithi177 ceph-mon[44303]: osdmap e184: 12 total, 12 up, 11 in 2024-05-15T12:51:09.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:08 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/239843948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:51:09.504 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:09.792 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T12:51:09.947 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.5 to 1 (10000) 2024-05-15T12:51:10.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:09 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2705242845' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T12:51:10.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:09 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T12:51:10.141 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T12:51:10.141 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:51:10.162 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:09 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2705242845' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T12:51:10.163 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:09 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T12:51:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:51:10.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:10.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:51:10.311 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:51:10.383 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:10.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:09 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2705242845' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T12:51:10.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:09 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T12:51:10.441 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:51:10.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:10.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:51:10.551 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:51:10.590 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:10.903 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-05-15T12:51:10.944 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:51:10.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:10.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:51:11.047 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:51:11.100 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 6 1 2024-05-15T12:51:11.101 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:11.122 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:51:11.124 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:10 smithi120 ceph-mon[43961]: pgmap v529: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.1 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 672 op/s; 2286/153213 objects misplaced (1.492%); 6.6 MiB/s, 105 objects/s recovering 2024-05-15T12:51:11.124 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:10 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T12:51:11.124 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:10 smithi120 ceph-mon[43961]: osdmap e185: 12 total, 12 up, 11 in 2024-05-15T12:51:11.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:11.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:51:11.181 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:51:11.226 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:51:11.267 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:11.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:10 smithi027 ceph-mon[39795]: pgmap v529: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.1 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 672 op/s; 2286/153213 objects misplaced (1.492%); 6.6 MiB/s, 105 objects/s recovering 2024-05-15T12:51:11.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:10 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T12:51:11.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:10 smithi027 ceph-mon[39795]: osdmap e185: 12 total, 12 up, 11 in 2024-05-15T12:51:11.297 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:51:11.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:11.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:51:11.304 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:51:11.403 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:51:11.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:10 smithi177 ceph-mon[44303]: pgmap v529: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.1 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 672 op/s; 2286/153213 objects misplaced (1.492%); 6.6 MiB/s, 105 objects/s recovering 2024-05-15T12:51:11.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:10 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T12:51:11.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:10 smithi177 ceph-mon[44303]: osdmap e185: 12 total, 12 up, 11 in 2024-05-15T12:51:11.434 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:11.457 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:51:11.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:11.460 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:51:11.561 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:51:11.608 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:11.642 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:51:11.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:11.646 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:51:11.748 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:51:11.789 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:11.815 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:51:11.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:11.822 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:51:11.923 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:51:11.960 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:11.989 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:51:11.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:11.993 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:51:12.094 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:51:12.123 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:12 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:51:12.123 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:12 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:51:12.126 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:12.160 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:51:12.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:12.165 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:51:12.241 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:51:12.265 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:51:12.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:11 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:51:12.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:11 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:51:12.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:11 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:51:12.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:11 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:51:12.303 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:12.865 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:51:12.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:12.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:51:12.971 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:51:13.024 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:13.111 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:13 smithi120 ceph-mon[43961]: pgmap v531: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 813 op/s; 1358/146169 objects misplaced (0.929%); 8.0 MiB/s, 127 objects/s recovering 2024-05-15T12:51:13.112 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:13 smithi120 ceph-mon[43961]: osdmap e186: 12 total, 12 up, 11 in 2024-05-15T12:51:13.112 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:13 smithi120 ceph-mon[43961]: osdmap e187: 12 total, 12 up, 11 in 2024-05-15T12:51:13.163 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:51:13.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:13.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:51:13.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:13 smithi027 ceph-mon[39795]: pgmap v531: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 813 op/s; 1358/146169 objects misplaced (0.929%); 8.0 MiB/s, 127 objects/s recovering 2024-05-15T12:51:13.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:13 smithi027 ceph-mon[39795]: osdmap e186: 12 total, 12 up, 11 in 2024-05-15T12:51:13.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:13 smithi027 ceph-mon[39795]: osdmap e187: 12 total, 12 up, 11 in 2024-05-15T12:51:13.271 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:51:13.301 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:13 smithi177 ceph-mon[44303]: pgmap v531: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 813 op/s; 1358/146169 objects misplaced (0.929%); 8.0 MiB/s, 127 objects/s recovering 2024-05-15T12:51:13.301 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:13 smithi177 ceph-mon[44303]: osdmap e186: 12 total, 12 up, 11 in 2024-05-15T12:51:13.301 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:13 smithi177 ceph-mon[44303]: osdmap e187: 12 total, 12 up, 11 in 2024-05-15T12:51:13.303 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:13.332 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:51:13.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:13.336 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:51:13.436 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:51:13.471 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T12:51:13.471 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:51:13.519 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:13.826 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:51:13.909 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:51:14.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:14 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:51:14.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:14 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:51:14.198 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:14 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:51:14.199 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:14 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:51:14.203 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:51:14.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:14.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:51:14.232 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-05-15T12:51:14.312 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:51:14.362 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:14.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:14 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:51:14.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:14 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:51:14.576 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:51:14.835 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:51:14.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:14.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:51:14.941 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:51:14.998 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:15.064 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:51:15.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:15.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:51:15.168 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:51:15.238 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:15 smithi027 ceph-mon[39795]: pgmap v534: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 815 op/s; 1358/140871 objects misplaced (0.964%); 8.5 MiB/s, 135 objects/s recovering 2024-05-15T12:51:15.238 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:15 smithi027 ceph-mon[39795]: osdmap e188: 12 total, 12 up, 11 in 2024-05-15T12:51:15.252 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:15.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:15 smithi120 ceph-mon[43961]: pgmap v534: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 815 op/s; 1358/140871 objects misplaced (0.964%); 8.5 MiB/s, 135 objects/s recovering 2024-05-15T12:51:15.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:15 smithi120 ceph-mon[43961]: osdmap e188: 12 total, 12 up, 11 in 2024-05-15T12:51:15.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:15 smithi177 ceph-mon[44303]: pgmap v534: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 815 op/s; 1358/140871 objects misplaced (0.964%); 8.5 MiB/s, 135 objects/s recovering 2024-05-15T12:51:15.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:15 smithi177 ceph-mon[44303]: osdmap e188: 12 total, 12 up, 11 in 2024-05-15T12:51:16.034 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T12:51:16.227 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:16 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3607065540' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:51:16.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:16 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3607065540' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:51:16.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:16 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3607065540' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:51:16.544 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T12:51:16.545 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:51:17.002 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:51:17.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:17.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:51:17.117 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:51:17.153 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:17.176 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:51:17.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:17.179 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:51:17.201 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:17 smithi120 ceph-mon[43961]: pgmap v536: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.8 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 998 op/s; 383/136257 objects misplaced (0.281%); 18 MiB/s, 295 objects/s recovering 2024-05-15T12:51:17.201 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:17 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:51:17.201 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:17 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3607065540' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:51:17.201 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:17 smithi120 ceph-mon[43961]: osdmap e189: 12 total, 12 up, 11 in 2024-05-15T12:51:17.231 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-05-15T12:51:17.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:17 smithi027 ceph-mon[39795]: pgmap v536: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.8 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 998 op/s; 383/136257 objects misplaced (0.281%); 18 MiB/s, 295 objects/s recovering 2024-05-15T12:51:17.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:17 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:51:17.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:17 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3607065540' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:51:17.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:17 smithi027 ceph-mon[39795]: osdmap e189: 12 total, 12 up, 11 in 2024-05-15T12:51:17.280 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:51:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:17.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:17 smithi177 ceph-mon[44303]: pgmap v536: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.8 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 998 op/s; 383/136257 objects misplaced (0.281%); 18 MiB/s, 295 objects/s recovering 2024-05-15T12:51:17.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:17 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:51:17.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:17 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3607065540' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:51:17.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:17 smithi177 ceph-mon[44303]: osdmap e189: 12 total, 12 up, 11 in 2024-05-15T12:51:17.509 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:51:17.669 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:51:17.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:17.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:51:17.773 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:51:17.800 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:18.042 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:18 smithi177 ceph-mon[44303]: osdmap e190: 12 total, 12 up, 11 in 2024-05-15T12:51:18.043 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:18 smithi177 ceph-mon[44303]: osdmap e191: 12 total, 12 up, 11 in 2024-05-15T12:51:18.049 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:51:18.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:18.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:51:18.160 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:51:18.200 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:18.264 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:51:18.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:18.268 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:51:18.369 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:51:18.430 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:18 smithi027 ceph-mon[39795]: osdmap e190: 12 total, 12 up, 11 in 2024-05-15T12:51:18.430 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:18 smithi027 ceph-mon[39795]: osdmap e191: 12 total, 12 up, 11 in 2024-05-15T12:51:18.432 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:18.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:18 smithi120 ceph-mon[43961]: osdmap e190: 12 total, 12 up, 11 in 2024-05-15T12:51:18.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:18 smithi120 ceph-mon[43961]: osdmap e191: 12 total, 12 up, 11 in 2024-05-15T12:51:19.147 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:19 smithi177 ceph-mon[44303]: pgmap v539: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.7 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 674 op/s; 383/132972 objects misplaced (0.288%); 12 MiB/s, 194 objects/s recovering 2024-05-15T12:51:19.427 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T12:51:19.427 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:51:19.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:19 smithi027 ceph-mon[39795]: pgmap v539: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.7 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 674 op/s; 383/132972 objects misplaced (0.288%); 12 MiB/s, 194 objects/s recovering 2024-05-15T12:51:19.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:19 smithi120 ceph-mon[43961]: pgmap v539: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.7 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 674 op/s; 383/132972 objects misplaced (0.288%); 12 MiB/s, 194 objects/s recovering 2024-05-15T12:51:19.619 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T12:51:20.072 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-05-15T12:51:20.334 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:51:20.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:20 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:51:20.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:20 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:51:20.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:20 smithi177 ceph-mon[44303]: osdmap e192: 12 total, 12 up, 11 in 2024-05-15T12:51:20.433 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:20 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:51:20.433 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:20 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:51:20.433 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:20 smithi027 ceph-mon[39795]: osdmap e192: 12 total, 12 up, 11 in 2024-05-15T12:51:20.511 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:51:20.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:20.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:51:20.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:20 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:51:20.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:20 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:51:20.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:20 smithi120 ceph-mon[43961]: osdmap e192: 12 total, 12 up, 11 in 2024-05-15T12:51:20.616 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:51:20.646 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:20.664 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:51:20.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:20.667 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:51:20.768 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:51:20.805 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:20.826 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:51:20.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:20.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:51:20.930 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:51:20.963 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:20.984 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:51:20.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:20.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:51:21.088 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:51:21.117 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:21 smithi027 ceph-mon[39795]: pgmap v542: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.6 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 623 op/s; 383/128769 objects misplaced (0.297%) 2024-05-15T12:51:21.121 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:21.159 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:51:21.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:21.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:51:21.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:21 smithi177 ceph-mon[44303]: pgmap v542: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.6 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 623 op/s; 383/128769 objects misplaced (0.297%) 2024-05-15T12:51:21.263 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:51:21.289 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:21.290 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:51:21.290 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:51:21.290 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:51:21.290 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:51:21.290 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:51:21.290 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:51:21.296 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:51:21.316 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:51:21.337 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:51:21.357 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:51:21.377 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:51:21.402 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:51:21.404 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:51:21.404 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:51:21.404 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:51:21.404 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:51:21.404 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:51:21.404 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:51:21.404 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:51:21.404 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:51:21.405 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:51:21.405 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:51:21.405 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:51:21.405 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:51:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:51:21.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:21 smithi120 ceph-mon[43961]: pgmap v542: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.6 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 623 op/s; 383/128769 objects misplaced (0.297%) 2024-05-15T12:51:21.838 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:51:21.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:21.841 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:51:21.942 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:51:21.971 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:22.059 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.6 to 1 (10000) 2024-05-15T12:51:22.111 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:51:22.113 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:22 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1712022305' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:51:22.113 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:22 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4058368815' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T12:51:22.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:22.114 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:51:22.127 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T12:51:22.127 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T12:51:22.127 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-05-15T12:51:22.127 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-05-15T12:51:22.127 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T12:51:22.215 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:51:22.242 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:22 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1712022305' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:51:22.242 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:22 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4058368815' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T12:51:22.244 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:22.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:22 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1712022305' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:51:22.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:22 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4058368815' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T12:51:22.752 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-05-15T12:51:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:51:22.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:22.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:51:22.997 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:51:23.025 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:23.057 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:51:23.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:23.060 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:51:23.061 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:51:23.160 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:51:23.190 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:23.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:23 smithi027 ceph-mon[39795]: pgmap v543: 17 pgs: 17 active+clean; 2.5 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 565 op/s; 4.2 MiB/s, 67 objects/s recovering 2024-05-15T12:51:23.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:23 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4058368815' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T12:51:23.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:23 smithi027 ceph-mon[39795]: osdmap e193: 12 total, 12 up, 11 in 2024-05-15T12:51:23.299 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:23 smithi177 ceph-mon[44303]: pgmap v543: 17 pgs: 17 active+clean; 2.5 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 565 op/s; 4.2 MiB/s, 67 objects/s recovering 2024-05-15T12:51:23.299 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:23 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4058368815' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T12:51:23.299 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:23 smithi177 ceph-mon[44303]: osdmap e193: 12 total, 12 up, 11 in 2024-05-15T12:51:23.312 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:51:23.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:23.315 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:51:23.416 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:51:23.444 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:23.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:23 smithi120 ceph-mon[43961]: pgmap v543: 17 pgs: 17 active+clean; 2.5 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 565 op/s; 4.2 MiB/s, 67 objects/s recovering 2024-05-15T12:51:23.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:23 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4058368815' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T12:51:23.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:23 smithi120 ceph-mon[43961]: osdmap e193: 12 total, 12 up, 11 in 2024-05-15T12:51:23.543 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:51:23.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:23.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:51:23.647 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:51:23.674 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:24.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:24 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:51:24.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:24 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:51:24.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:24 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:51:24.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:24 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:51:24.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:24 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:51:24.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:24 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:51:24.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:24 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:51:24.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:24 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:51:24.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:24 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:51:24.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:24 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:51:24.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:24 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:51:24.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:24 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:51:24.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:24 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:51:24.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:24 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:51:24.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:51:24.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:51:24.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:51:24.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:51:24.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:51:24.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:51:24.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:51:24.635 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:51:24.715 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 7 1 2024-05-15T12:51:24.766 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T12:51:24.766 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-05-15T12:51:24.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-05-15T12:51:24.766 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T12:51:24.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.767 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:91859 2.b 2:d55ed149:::benchmark_data_smithi177_78104_object91858:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-05-15T12:51:24.767 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:49:20.681451+0000", 2024-05-15T12:51:24.767 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 124.092711178, 2024-05-15T12:51:24.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.073830701999999998, 2024-05-15T12:51:24.767 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:24.767 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:24.767 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:24.767 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:24.767 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:24.767 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 91859 2024-05-15T12:51:24.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.767 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:24.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:24.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.681451+0000", 2024-05-15T12:51:24.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:24.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.681451+0000", 2024-05-15T12:51:24.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:24.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.681453+0000", 2024-05-15T12:51:24.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4950000000000001e-06 2024-05-15T12:51:24.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:24.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.681475+0000", 2024-05-15T12:51:24.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2308999999999999e-05 2024-05-15T12:51:24.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:24.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.681485+0000", 2024-05-15T12:51:24.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0135e-05 2024-05-15T12:51:24.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:24.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.681492+0000", 2024-05-15T12:51:24.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6930000000000001e-06 2024-05-15T12:51:24.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:24.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.685871+0000", 2024-05-15T12:51:24.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004379137 2024-05-15T12:51:24.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:24.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.685965+0000", 2024-05-15T12:51:24.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4016000000000002e-05 2024-05-15T12:51:24.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.771 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:51:24.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.686073+0000", 2024-05-15T12:51:24.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010843199999999999 2024-05-15T12:51:24.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.771 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.752407+0000", 2024-05-15T12:51:24.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.066333475000000003 2024-05-15T12:51:24.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.771 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.752494+0000", 2024-05-15T12:51:24.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6837999999999998e-05 2024-05-15T12:51:24.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:24.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.752507+0000", 2024-05-15T12:51:24.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3447e-05 2024-05-15T12:51:24.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:24.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.752541+0000", 2024-05-15T12:51:24.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3871000000000001e-05 2024-05-15T12:51:24.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:24.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.755282+0000", 2024-05-15T12:51:24.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002740854 2024-05-15T12:51:24.773 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.773 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:24.773 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.773 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:91860 2.b 2:de072342:::benchmark_data_smithi177_78104_object91859:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-05-15T12:51:24.773 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:49:20.684580+0000", 2024-05-15T12:51:24.773 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 124.08958206, 2024-05-15T12:51:24.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.073318634999999993, 2024-05-15T12:51:24.773 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:24.773 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:24.773 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:24.773 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:24.773 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:24.774 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 91860 2024-05-15T12:51:24.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.774 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:24.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:24.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.684580+0000", 2024-05-15T12:51:24.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:24.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.684580+0000", 2024-05-15T12:51:24.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:24.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.684582+0000", 2024-05-15T12:51:24.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3629999999999999e-06 2024-05-15T12:51:24.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:24.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.684604+0000", 2024-05-15T12:51:24.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.198e-05 2024-05-15T12:51:24.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:24.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.684613+0000", 2024-05-15T12:51:24.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2180000000000005e-06 2024-05-15T12:51:24.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:24.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.684619+0000", 2024-05-15T12:51:24.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5830000000000003e-06 2024-05-15T12:51:24.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:24.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.688168+0000", 2024-05-15T12:51:24.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035486620000000002 2024-05-15T12:51:24.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:24.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.688262+0000", 2024-05-15T12:51:24.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3703999999999996e-05 2024-05-15T12:51:24.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:51:24.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.688360+0000", 2024-05-15T12:51:24.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8342000000000001e-05 2024-05-15T12:51:24.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.752453+0000", 2024-05-15T12:51:24.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.064092957000000006 2024-05-15T12:51:24.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:24.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.755295+0000", 2024-05-15T12:51:24.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002841971 2024-05-15T12:51:24.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.755347+0000", 2024-05-15T12:51:24.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1613000000000002e-05 2024-05-15T12:51:24.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:24.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.755406+0000", 2024-05-15T12:51:24.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8959000000000003e-05 2024-05-15T12:51:24.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:24.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.757899+0000", 2024-05-15T12:51:24.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024932829999999998 2024-05-15T12:51:24.779 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.779 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:24.779 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.779 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:91861 2.f 2:f2e5831b:::benchmark_data_smithi177_78104_object91860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-05-15T12:51:24.779 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:49:20.688856+0000", 2024-05-15T12:51:24.779 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 124.08530591100001, 2024-05-15T12:51:24.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.062972200000000006, 2024-05-15T12:51:24.779 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:24.779 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:24.779 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:24.779 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:24.779 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:24.779 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 91861 2024-05-15T12:51:24.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.780 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:24.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:24.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.688856+0000", 2024-05-15T12:51:24.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:24.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.688856+0000", 2024-05-15T12:51:24.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:24.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.688858+0000", 2024-05-15T12:51:24.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1570000000000001e-06 2024-05-15T12:51:24.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:24.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.688878+0000", 2024-05-15T12:51:24.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0820999999999999e-05 2024-05-15T12:51:24.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:24.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.743264+0000", 2024-05-15T12:51:24.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.054385971999999998 2024-05-15T12:51:24.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:24.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.743284+0000", 2024-05-15T12:51:24.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9601000000000001e-05 2024-05-15T12:51:24.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:24.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.743352+0000", 2024-05-15T12:51:24.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7638000000000005e-05 2024-05-15T12:51:24.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:24.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.743518+0000", 2024-05-15T12:51:24.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016677 2024-05-15T12:51:24.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 4,9", 2024-05-15T12:51:24.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.745025+0000", 2024-05-15T12:51:24.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015062789999999999 2024-05-15T12:51:24.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.750803+0000", 2024-05-15T12:51:24.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0057781819999999998 2024-05-15T12:51:24.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:24.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.750815+0000", 2024-05-15T12:51:24.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2413e-05 2024-05-15T12:51:24.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.751741+0000", 2024-05-15T12:51:24.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00092586899999999995 2024-05-15T12:51:24.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:24.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.751782+0000", 2024-05-15T12:51:24.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0531999999999998e-05 2024-05-15T12:51:24.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:24.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.751829+0000", 2024-05-15T12:51:24.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6965999999999997e-05 2024-05-15T12:51:24.784 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.784 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:24.784 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.784 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.24562.0:91859 2.b e145/136 ondisk, result = 0)", 2024-05-15T12:51:24.784 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:49:20.690813+0000", 2024-05-15T12:51:24.785 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 124.083349114, 2024-05-15T12:51:24.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.061603545000000003, 2024-05-15T12:51:24.785 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:24.785 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T12:51:24.785 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:24.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:24.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.690813+0000", 2024-05-15T12:51:24.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:24.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.690813+0000", 2024-05-15T12:51:24.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:24.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.690814+0000", 2024-05-15T12:51:24.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2799999999999996e-07 2024-05-15T12:51:24.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:24.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.690818+0000", 2024-05-15T12:51:24.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5020000000000003e-06 2024-05-15T12:51:24.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:24.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.743290+0000", 2024-05-15T12:51:24.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.052471379999999998 2024-05-15T12:51:24.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:24.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.743300+0000", 2024-05-15T12:51:24.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0637e-05 2024-05-15T12:51:24.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:24.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.752391+0000", 2024-05-15T12:51:24.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0090908719999999998 2024-05-15T12:51:24.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:24.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.752400+0000", 2024-05-15T12:51:24.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8990000000000007e-06 2024-05-15T12:51:24.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:24.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:49:20.752417+0000", 2024-05-15T12:51:24.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6626999999999999e-05 2024-05-15T12:51:24.788 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.788 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:24.788 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.788 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125021 2.b 2:d6c97047:::benchmark_data_smithi177_78104_object17888:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:51:24.788 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.708938+0000", 2024-05-15T12:51:24.788 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 67.065224474999994, 2024-05-15T12:51:24.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21075210699999999, 2024-05-15T12:51:24.788 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:24.788 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:24.788 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:24.788 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:24.789 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:24.789 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125021 2024-05-15T12:51:24.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.789 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:24.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:24.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.708938+0000", 2024-05-15T12:51:24.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:24.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.708938+0000", 2024-05-15T12:51:24.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:24.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.708939+0000", 2024-05-15T12:51:24.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.31e-06 2024-05-15T12:51:24.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:24.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.708944+0000", 2024-05-15T12:51:24.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2020000000000003e-06 2024-05-15T12:51:24.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:24.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.708956+0000", 2024-05-15T12:51:24.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1430000000000001e-05 2024-05-15T12:51:24.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:24.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.708963+0000", 2024-05-15T12:51:24.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5639999999999999e-06 2024-05-15T12:51:24.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:24.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.709010+0000", 2024-05-15T12:51:24.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6993000000000002e-05 2024-05-15T12:51:24.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:24.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.709098+0000", 2024-05-15T12:51:24.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7961999999999996e-05 2024-05-15T12:51:24.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:51:24.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.709168+0000", 2024-05-15T12:51:24.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9677000000000004e-05 2024-05-15T12:51:24.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:24.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.712329+0000", 2024-05-15T12:51:24.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003161245 2024-05-15T12:51:24.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.712501+0000", 2024-05-15T12:51:24.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017174700000000001 2024-05-15T12:51:24.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.919583+0000", 2024-05-15T12:51:24.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20708225 2024-05-15T12:51:24.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:24.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.919642+0000", 2024-05-15T12:51:24.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8273000000000003e-05 2024-05-15T12:51:24.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:24.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.919690+0000", 2024-05-15T12:51:24.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8454000000000001e-05 2024-05-15T12:51:24.793 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.793 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:24.794 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.794 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125026 2.b 2:d5f79b62:::benchmark_data_smithi177_78104_object17893:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:51:24.794 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.718595+0000", 2024-05-15T12:51:24.794 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 67.055567475999993, 2024-05-15T12:51:24.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20594219999999999, 2024-05-15T12:51:24.794 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:24.794 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:24.794 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:24.794 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:24.794 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:24.794 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125026 2024-05-15T12:51:24.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.795 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:24.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:24.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.718595+0000", 2024-05-15T12:51:24.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:24.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.718595+0000", 2024-05-15T12:51:24.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.718596+0000", 2024-05-15T12:51:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1170000000000001e-06 2024-05-15T12:51:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.718601+0000", 2024-05-15T12:51:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0089999999999998e-06 2024-05-15T12:51:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.718611+0000", 2024-05-15T12:51:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6719999999999999e-06 2024-05-15T12:51:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:24.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.718617+0000", 2024-05-15T12:51:24.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9989999999999997e-06 2024-05-15T12:51:24.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:24.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.718668+0000", 2024-05-15T12:51:24.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0936000000000002e-05 2024-05-15T12:51:24.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:24.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.718800+0000", 2024-05-15T12:51:24.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013213799999999999 2024-05-15T12:51:24.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:51:24.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.718911+0000", 2024-05-15T12:51:24.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011174499999999999 2024-05-15T12:51:24.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:24.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.722009+0000", 2024-05-15T12:51:24.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003097739 2024-05-15T12:51:24.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.722395+0000", 2024-05-15T12:51:24.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00038567000000000001 2024-05-15T12:51:24.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.924476+0000", 2024-05-15T12:51:24.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.202080975 2024-05-15T12:51:24.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:24.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.924509+0000", 2024-05-15T12:51:24.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3284999999999997e-05 2024-05-15T12:51:24.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:24.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.924537+0000", 2024-05-15T12:51:24.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7915e-05 2024-05-15T12:51:24.799 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.800 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:24.800 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.800 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125038 2.b 2:d70abfc3:::benchmark_data_smithi177_78104_object17905:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:51:24.800 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.727604+0000", 2024-05-15T12:51:24.800 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 67.046558071000007, 2024-05-15T12:51:24.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.201815784, 2024-05-15T12:51:24.800 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:24.800 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:24.800 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:24.800 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:24.800 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:24.800 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125038 2024-05-15T12:51:24.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.801 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:24.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:24.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.727604+0000", 2024-05-15T12:51:24.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:24.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.727604+0000", 2024-05-15T12:51:24.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:24.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.727605+0000", 2024-05-15T12:51:24.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2e-07 2024-05-15T12:51:24.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:24.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.727608+0000", 2024-05-15T12:51:24.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1310000000000002e-06 2024-05-15T12:51:24.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:24.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.727614+0000", 2024-05-15T12:51:24.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.353e-06 2024-05-15T12:51:24.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:24.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.727620+0000", 2024-05-15T12:51:24.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4790000000000004e-06 2024-05-15T12:51:24.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:24.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.727667+0000", 2024-05-15T12:51:24.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7219000000000003e-05 2024-05-15T12:51:24.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:24.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.727743+0000", 2024-05-15T12:51:24.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5912000000000001e-05 2024-05-15T12:51:24.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:51:24.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.727818+0000", 2024-05-15T12:51:24.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5493000000000004e-05 2024-05-15T12:51:24.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.731339+0000", 2024-05-15T12:51:24.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035210580000000001 2024-05-15T12:51:24.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:24.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.731402+0000", 2024-05-15T12:51:24.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2527999999999995e-05 2024-05-15T12:51:24.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.929301+0000", 2024-05-15T12:51:24.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.197899305 2024-05-15T12:51:24.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:24.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.929360+0000", 2024-05-15T12:51:24.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8541000000000001e-05 2024-05-15T12:51:24.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:24.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.929420+0000", 2024-05-15T12:51:24.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0244999999999997e-05 2024-05-15T12:51:24.806 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.806 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:24.806 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.806 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125040 2.b 2:d223540b:::benchmark_data_smithi177_78104_object17907:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:51:24.806 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.733348+0000", 2024-05-15T12:51:24.806 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 67.040814562999998, 2024-05-15T12:51:24.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19620225999999999, 2024-05-15T12:51:24.806 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:24.806 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:24.806 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:24.806 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:24.807 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:24.807 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125040 2024-05-15T12:51:24.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.807 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:24.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:24.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.733348+0000", 2024-05-15T12:51:24.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:24.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.733348+0000", 2024-05-15T12:51:24.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:24.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.733349+0000", 2024-05-15T12:51:24.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1680000000000001e-06 2024-05-15T12:51:24.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:24.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.733355+0000", 2024-05-15T12:51:24.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7599999999999999e-06 2024-05-15T12:51:24.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:24.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.733365+0000", 2024-05-15T12:51:24.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0562e-05 2024-05-15T12:51:24.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:24.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.733381+0000", 2024-05-15T12:51:24.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5469999999999999e-05 2024-05-15T12:51:24.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:24.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.733440+0000", 2024-05-15T12:51:24.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9667000000000003e-05 2024-05-15T12:51:24.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:24.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.733570+0000", 2024-05-15T12:51:24.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000129794 2024-05-15T12:51:24.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:51:24.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.733689+0000", 2024-05-15T12:51:24.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011836999999999999 2024-05-15T12:51:24.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:24.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.737518+0000", 2024-05-15T12:51:24.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003829598 2024-05-15T12:51:24.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.737968+0000", 2024-05-15T12:51:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00045025700000000002 2024-05-15T12:51:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.929466+0000", 2024-05-15T12:51:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19149776499999999 2024-05-15T12:51:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.929508+0000", 2024-05-15T12:51:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1690999999999998e-05 2024-05-15T12:51:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.929550+0000", 2024-05-15T12:51:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2157999999999999e-05 2024-05-15T12:51:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125083 2.b 2:db4ca6e9:::benchmark_data_smithi177_78104_object17950:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:51:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.785463+0000", 2024-05-15T12:51:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 66.988699393000005, 2024-05-15T12:51:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.15291454400000001, 2024-05-15T12:51:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:24.812 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125083 2024-05-15T12:51:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.785463+0000", 2024-05-15T12:51:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.785463+0000", 2024-05-15T12:51:24.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.785464+0000", 2024-05-15T12:51:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.488e-06 2024-05-15T12:51:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.785470+0000", 2024-05-15T12:51:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8340000000000004e-06 2024-05-15T12:51:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.785485+0000", 2024-05-15T12:51:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5099000000000001e-05 2024-05-15T12:51:24.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.785494+0000", 2024-05-15T12:51:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3189999999999995e-06 2024-05-15T12:51:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.785549+0000", 2024-05-15T12:51:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5670000000000001e-05 2024-05-15T12:51:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.785738+0000", 2024-05-15T12:51:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000188677 2024-05-15T12:51:24.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:51:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.785868+0000", 2024-05-15T12:51:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012986700000000001 2024-05-15T12:51:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.789091+0000", 2024-05-15T12:51:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0032231680000000002 2024-05-15T12:51:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:24.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.789135+0000", 2024-05-15T12:51:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3782999999999997e-05 2024-05-15T12:51:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.938308+0000", 2024-05-15T12:51:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14917336 2024-05-15T12:51:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.938349+0000", 2024-05-15T12:51:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0386999999999998e-05 2024-05-15T12:51:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.938378+0000", 2024-05-15T12:51:24.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8892000000000001e-05 2024-05-15T12:51:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125086 2.b 2:d1b0dce9:::benchmark_data_smithi177_78104_object17953:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:51:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.792066+0000", 2024-05-15T12:51:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 66.982096318000004, 2024-05-15T12:51:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.146447893, 2024-05-15T12:51:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:24.818 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125086 2024-05-15T12:51:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.792066+0000", 2024-05-15T12:51:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.792066+0000", 2024-05-15T12:51:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.792067+0000", 2024-05-15T12:51:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1689999999999999e-06 2024-05-15T12:51:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.792072+0000", 2024-05-15T12:51:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2599999999999996e-06 2024-05-15T12:51:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.792083+0000", 2024-05-15T12:51:24.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0508e-05 2024-05-15T12:51:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.792089+0000", 2024-05-15T12:51:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6710000000000004e-06 2024-05-15T12:51:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.792134+0000", 2024-05-15T12:51:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5732000000000002e-05 2024-05-15T12:51:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.792252+0000", 2024-05-15T12:51:24.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117772 2024-05-15T12:51:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:51:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.792367+0000", 2024-05-15T12:51:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114957 2024-05-15T12:51:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.795925+0000", 2024-05-15T12:51:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035583559999999999 2024-05-15T12:51:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.797215+0000", 2024-05-15T12:51:24.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012897679999999999 2024-05-15T12:51:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.938419+0000", 2024-05-15T12:51:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14120342899999999 2024-05-15T12:51:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.938471+0000", 2024-05-15T12:51:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2422000000000001e-05 2024-05-15T12:51:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.938514+0000", 2024-05-15T12:51:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2849000000000003e-05 2024-05-15T12:51:24.823 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125099 2.b 2:da4fcb81:::benchmark_data_smithi177_78104_object17966:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:51:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.803901+0000", 2024-05-15T12:51:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 66.970261324000006, 2024-05-15T12:51:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.138301853, 2024-05-15T12:51:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125099 2024-05-15T12:51:24.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.803901+0000", 2024-05-15T12:51:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.803901+0000", 2024-05-15T12:51:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.803902+0000", 2024-05-15T12:51:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9400000000000004e-07 2024-05-15T12:51:24.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.803907+0000", 2024-05-15T12:51:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.186e-06 2024-05-15T12:51:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.803920+0000", 2024-05-15T12:51:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3227000000000001e-05 2024-05-15T12:51:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.803927+0000", 2024-05-15T12:51:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2400000000000004e-06 2024-05-15T12:51:24.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.803977+0000", 2024-05-15T12:51:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0000000000000002e-05 2024-05-15T12:51:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.804106+0000", 2024-05-15T12:51:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012985699999999999 2024-05-15T12:51:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:51:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.804229+0000", 2024-05-15T12:51:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012295599999999999 2024-05-15T12:51:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.807117+0000", 2024-05-15T12:51:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0028879969999999998 2024-05-15T12:51:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.807404+0000", 2024-05-15T12:51:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028621799999999998 2024-05-15T12:51:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.942098+0000", 2024-05-15T12:51:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.134694803 2024-05-15T12:51:24.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.942156+0000", 2024-05-15T12:51:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7138000000000001e-05 2024-05-15T12:51:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.942203+0000", 2024-05-15T12:51:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7336999999999998e-05 2024-05-15T12:51:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125102 2.b 2:da660b72:::benchmark_data_smithi177_78104_object17969:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:51:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.809757+0000", 2024-05-15T12:51:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 66.964405717999995, 2024-05-15T12:51:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13260296999999999, 2024-05-15T12:51:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125102 2024-05-15T12:51:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809757+0000", 2024-05-15T12:51:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809757+0000", 2024-05-15T12:51:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809757+0000", 2024-05-15T12:51:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6800000000000001e-07 2024-05-15T12:51:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809760+0000", 2024-05-15T12:51:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.847e-06 2024-05-15T12:51:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809766+0000", 2024-05-15T12:51:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1310000000000003e-06 2024-05-15T12:51:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809771+0000", 2024-05-15T12:51:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8729999999999998e-06 2024-05-15T12:51:24.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809813+0000", 2024-05-15T12:51:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2419000000000002e-05 2024-05-15T12:51:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809921+0000", 2024-05-15T12:51:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010755300000000001 2024-05-15T12:51:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:51:24.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.810010+0000", 2024-05-15T12:51:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8954000000000002e-05 2024-05-15T12:51:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.816996+0000", 2024-05-15T12:51:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0069862789999999998 2024-05-15T12:51:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.817033+0000", 2024-05-15T12:51:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6776999999999997e-05 2024-05-15T12:51:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.942257+0000", 2024-05-15T12:51:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.125224213 2024-05-15T12:51:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.942313+0000", 2024-05-15T12:51:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6221000000000003e-05 2024-05-15T12:51:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.942360+0000", 2024-05-15T12:51:24.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6235e-05 2024-05-15T12:51:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125103 2.b 2:d6ba212d:::benchmark_data_smithi177_78104_object17970:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:51:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.809893+0000", 2024-05-15T12:51:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 66.964269130999995, 2024-05-15T12:51:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.132601633, 2024-05-15T12:51:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125103 2024-05-15T12:51:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809893+0000", 2024-05-15T12:51:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809893+0000", 2024-05-15T12:51:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809894+0000", 2024-05-15T12:51:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8700000000000004e-07 2024-05-15T12:51:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809898+0000", 2024-05-15T12:51:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2699999999999998e-06 2024-05-15T12:51:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809912+0000", 2024-05-15T12:51:24.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3572e-05 2024-05-15T12:51:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809919+0000", 2024-05-15T12:51:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.221e-06 2024-05-15T12:51:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.813691+0000", 2024-05-15T12:51:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037719440000000002 2024-05-15T12:51:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.839 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.813839+0000", 2024-05-15T12:51:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000148369 2024-05-15T12:51:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:51:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.813913+0000", 2024-05-15T12:51:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3009999999999994e-05 2024-05-15T12:51:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.817441+0000", 2024-05-15T12:51:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035280620000000002 2024-05-15T12:51:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.822286+0000", 2024-05-15T12:51:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0048452429999999999 2024-05-15T12:51:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.942407+0000", 2024-05-15T12:51:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.120121518 2024-05-15T12:51:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.942457+0000", 2024-05-15T12:51:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9818999999999999e-05 2024-05-15T12:51:24.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.942495+0000", 2024-05-15T12:51:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7718e-05 2024-05-15T12:51:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125109 2.b 2:dc68fa46:::benchmark_data_smithi177_78104_object17976:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:51:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.818774+0000", 2024-05-15T12:51:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 66.955388411000001, 2024-05-15T12:51:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.12385244300000001, 2024-05-15T12:51:24.842 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125109 2024-05-15T12:51:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.818774+0000", 2024-05-15T12:51:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:24.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.818774+0000", 2024-05-15T12:51:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.818775+0000", 2024-05-15T12:51:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4219999999999999e-06 2024-05-15T12:51:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.818781+0000", 2024-05-15T12:51:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.468e-06 2024-05-15T12:51:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.818792+0000", 2024-05-15T12:51:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1504e-05 2024-05-15T12:51:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.818800+0000", 2024-05-15T12:51:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.695e-06 2024-05-15T12:51:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.818867+0000", 2024-05-15T12:51:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6550999999999994e-05 2024-05-15T12:51:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.818990+0000", 2024-05-15T12:51:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012320199999999999 2024-05-15T12:51:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:51:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.819107+0000", 2024-05-15T12:51:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000116922 2024-05-15T12:51:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.823726+0000", 2024-05-15T12:51:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0046193049999999998 2024-05-15T12:51:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.824529+0000", 2024-05-15T12:51:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00080272500000000001 2024-05-15T12:51:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.942538+0000", 2024-05-15T12:51:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.118008848 2024-05-15T12:51:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.942593+0000", 2024-05-15T12:51:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5300000000000002e-05 2024-05-15T12:51:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.942626+0000", 2024-05-15T12:51:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3501000000000003e-05 2024-05-15T12:51:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125120 2.b 2:dccc152d:::benchmark_data_smithi177_78104_object17987:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:51:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.833738+0000", 2024-05-15T12:51:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 66.940424852000007, 2024-05-15T12:51:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11326000999999999, 2024-05-15T12:51:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:24.848 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125120 2024-05-15T12:51:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.833738+0000", 2024-05-15T12:51:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.833738+0000", 2024-05-15T12:51:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.833739+0000", 2024-05-15T12:51:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2109999999999999e-06 2024-05-15T12:51:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.833743+0000", 2024-05-15T12:51:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.262e-06 2024-05-15T12:51:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.833753+0000", 2024-05-15T12:51:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0013000000000001e-05 2024-05-15T12:51:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.833758+0000", 2024-05-15T12:51:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2480000000000004e-06 2024-05-15T12:51:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.833812+0000", 2024-05-15T12:51:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4203999999999998e-05 2024-05-15T12:51:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.833936+0000", 2024-05-15T12:51:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012344799999999999 2024-05-15T12:51:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:51:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.834057+0000", 2024-05-15T12:51:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120771 2024-05-15T12:51:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.838256+0000", 2024-05-15T12:51:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0041994550000000004 2024-05-15T12:51:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.838594+0000", 2024-05-15T12:51:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000337812 2024-05-15T12:51:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.852 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.946914+0000", 2024-05-15T12:51:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.108320473 2024-05-15T12:51:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.946963+0000", 2024-05-15T12:51:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8916999999999999e-05 2024-05-15T12:51:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.946998+0000", 2024-05-15T12:51:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4196000000000003e-05 2024-05-15T12:51:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125133 2.b 2:de1aeb7b:::benchmark_data_smithi177_78104_object18000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:51:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.853949+0000", 2024-05-15T12:51:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 66.920213836000002, 2024-05-15T12:51:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.093161534000000004, 2024-05-15T12:51:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125133 2024-05-15T12:51:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.853949+0000", 2024-05-15T12:51:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.853949+0000", 2024-05-15T12:51:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.853950+0000", 2024-05-15T12:51:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1790000000000001e-06 2024-05-15T12:51:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:24.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.853955+0000", 2024-05-15T12:51:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9910000000000002e-06 2024-05-15T12:51:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.853966+0000", 2024-05-15T12:51:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.148e-05 2024-05-15T12:51:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.853972+0000", 2024-05-15T12:51:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1800000000000001e-06 2024-05-15T12:51:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.854014+0000", 2024-05-15T12:51:24.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2104999999999998e-05 2024-05-15T12:51:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.854106+0000", 2024-05-15T12:51:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1762000000000007e-05 2024-05-15T12:51:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:51:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.854175+0000", 2024-05-15T12:51:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8411999999999995e-05 2024-05-15T12:51:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.861727+0000", 2024-05-15T12:51:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.007552614 2024-05-15T12:51:24.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.861777+0000", 2024-05-15T12:51:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9790000000000003e-05 2024-05-15T12:51:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.947049+0000", 2024-05-15T12:51:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.085271745999999995 2024-05-15T12:51:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:24.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.947087+0000", 2024-05-15T12:51:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8327000000000001e-05 2024-05-15T12:51:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.947110+0000", 2024-05-15T12:51:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2948000000000001e-05 2024-05-15T12:51:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125134 2.b 2:d9aab623:::benchmark_data_smithi177_78104_object18001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:51:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.855290+0000", 2024-05-15T12:51:24.859 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 66.918871929999995, 2024-05-15T12:51:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.091940785999999997, 2024-05-15T12:51:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125134 2024-05-15T12:51:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.855290+0000", 2024-05-15T12:51:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.855290+0000", 2024-05-15T12:51:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.855291+0000", 2024-05-15T12:51:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5799999999999999e-07 2024-05-15T12:51:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.855295+0000", 2024-05-15T12:51:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1420000000000003e-06 2024-05-15T12:51:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.855304+0000", 2024-05-15T12:51:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.064e-06 2024-05-15T12:51:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.855311+0000", 2024-05-15T12:51:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1189999999999999e-06 2024-05-15T12:51:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.857963+0000", 2024-05-15T12:51:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00265146 2024-05-15T12:51:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.858068+0000", 2024-05-15T12:51:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010539000000000001 2024-05-15T12:51:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:51:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.858182+0000", 2024-05-15T12:51:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011352499999999999 2024-05-15T12:51:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.861765+0000", 2024-05-15T12:51:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035833850000000001 2024-05-15T12:51:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.861835+0000", 2024-05-15T12:51:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0312e-05 2024-05-15T12:51:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.947152+0000", 2024-05-15T12:51:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.085316980000000001 2024-05-15T12:51:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.947196+0000", 2024-05-15T12:51:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3112000000000003e-05 2024-05-15T12:51:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.947231+0000", 2024-05-15T12:51:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5639000000000002e-05 2024-05-15T12:51:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125142 2.b 2:d25c7bf0:::benchmark_data_smithi177_78104_object18009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:51:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.871504+0000", 2024-05-15T12:51:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 66.902658607000006, 2024-05-15T12:51:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.075815470999999995, 2024-05-15T12:51:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:24.865 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125142 2024-05-15T12:51:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.871504+0000", 2024-05-15T12:51:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.871504+0000", 2024-05-15T12:51:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.871505+0000", 2024-05-15T12:51:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.372e-06 2024-05-15T12:51:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.871511+0000", 2024-05-15T12:51:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1129999999999997e-06 2024-05-15T12:51:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.871523+0000", 2024-05-15T12:51:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1342000000000001e-05 2024-05-15T12:51:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.871530+0000", 2024-05-15T12:51:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0550000000000002e-06 2024-05-15T12:51:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.871585+0000", 2024-05-15T12:51:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5365999999999998e-05 2024-05-15T12:51:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.871714+0000", 2024-05-15T12:51:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012897700000000001 2024-05-15T12:51:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:51:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.871828+0000", 2024-05-15T12:51:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011437700000000001 2024-05-15T12:51:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.875275+0000", 2024-05-15T12:51:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0034469370000000002 2024-05-15T12:51:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.875524+0000", 2024-05-15T12:51:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024909599999999997 2024-05-15T12:51:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.947263+0000", 2024-05-15T12:51:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.071738407000000004 2024-05-15T12:51:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.947298+0000", 2024-05-15T12:51:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5076000000000001e-05 2024-05-15T12:51:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.947319+0000", 2024-05-15T12:51:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1353e-05 2024-05-15T12:51:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:24.870 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:139208 2.5 2:a80fb3b0:::benchmark_data_smithi177_78104_object32075:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-05-15T12:51:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:35.632019+0000", 2024-05-15T12:51:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 49.142143111999999, 2024-05-15T12:51:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.061144566999999997, 2024-05-15T12:51:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 139208 2024-05-15T12:51:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.871 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:35.632019+0000", 2024-05-15T12:51:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:35.632019+0000", 2024-05-15T12:51:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:35.632021+0000", 2024-05-15T12:51:24.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6670000000000001e-06 2024-05-15T12:51:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:35.632027+0000", 2024-05-15T12:51:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1040000000000003e-06 2024-05-15T12:51:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:35.632042+0000", 2024-05-15T12:51:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5204999999999999e-05 2024-05-15T12:51:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:35.632051+0000", 2024-05-15T12:51:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8680000000000005e-06 2024-05-15T12:51:24.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:35.632106+0000", 2024-05-15T12:51:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5352999999999997e-05 2024-05-15T12:51:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:35.632240+0000", 2024-05-15T12:51:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013376800000000001 2024-05-15T12:51:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,4", 2024-05-15T12:51:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:35.632356+0000", 2024-05-15T12:51:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011599600000000001 2024-05-15T12:51:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:35.635972+0000", 2024-05-15T12:51:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0036156790000000001 2024-05-15T12:51:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:35.637278+0000", 2024-05-15T12:51:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013062799999999999 2024-05-15T12:51:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:35.693079+0000", 2024-05-15T12:51:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.055800711000000003 2024-05-15T12:51:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:35.693126+0000", 2024-05-15T12:51:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7070999999999997e-05 2024-05-15T12:51:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:35.693164+0000", 2024-05-15T12:51:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7864999999999997e-05 2024-05-15T12:51:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:24.876 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:159800 2.b 2:d1b7b24d:::benchmark_data_smithi177_78104_object52667:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-05-15T12:51:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:51:02.205853+0000", 2024-05-15T12:51:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 22.568309391, 2024-05-15T12:51:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.085231990999999993, 2024-05-15T12:51:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 159800 2024-05-15T12:51:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:24.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:02.205853+0000", 2024-05-15T12:51:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:02.205853+0000", 2024-05-15T12:51:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:02.205854+0000", 2024-05-15T12:51:24.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.026e-06 2024-05-15T12:51:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:02.205859+0000", 2024-05-15T12:51:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1050000000000003e-06 2024-05-15T12:51:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:02.205873+0000", 2024-05-15T12:51:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3429e-05 2024-05-15T12:51:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:02.205880+0000", 2024-05-15T12:51:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9049999999999997e-06 2024-05-15T12:51:24.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:02.205937+0000", 2024-05-15T12:51:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6323000000000003e-05 2024-05-15T12:51:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:02.206051+0000", 2024-05-15T12:51:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114354 2024-05-15T12:51:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:51:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:02.206173+0000", 2024-05-15T12:51:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000121927 2024-05-15T12:51:24.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:02.211126+0000", 2024-05-15T12:51:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0049525869999999996 2024-05-15T12:51:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:02.212795+0000", 2024-05-15T12:51:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001668959 2024-05-15T12:51:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:02.290974+0000", 2024-05-15T12:51:24.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.078179810000000002 2024-05-15T12:51:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:02.291033+0000", 2024-05-15T12:51:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8816000000000001e-05 2024-05-15T12:51:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:02.291085+0000", 2024-05-15T12:51:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1749999999999997e-05 2024-05-15T12:51:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.882 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:24.883 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:24.883 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T12:51:25.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:25 smithi027 ceph-mon[39795]: pgmap v545: 17 pgs: 17 active+clean; 2.5 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 555 op/s; 3.9 MiB/s, 62 objects/s recovering 2024-05-15T12:51:25.273 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:51:25.273 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:51:25.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:25 smithi177 ceph-mon[44303]: pgmap v545: 17 pgs: 17 active+clean; 2.5 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 555 op/s; 3.9 MiB/s, 62 objects/s recovering 2024-05-15T12:51:25.492 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-05-15T12:51:25.525 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:51:25.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:25.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:51:25.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:25 smithi120 ceph-mon[43961]: pgmap v545: 17 pgs: 17 active+clean; 2.5 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 555 op/s; 3.9 MiB/s, 62 objects/s recovering 2024-05-15T12:51:25.629 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:51:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:25.749 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:51:25.910 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:51:25.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:25.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:51:26.014 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:51:26.044 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:26.107 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T12:51:26.262 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:26 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:51:26.263 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:26 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:51:26.263 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:26 smithi177 ceph-mon[44303]: osdmap e194: 12 total, 12 up, 11 in 2024-05-15T12:51:26.263 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:26 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2885383538' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:51:26.263 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:26 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:51:26.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:26 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:51:26.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:26 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:51:26.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:26 smithi027 ceph-mon[39795]: osdmap e194: 12 total, 12 up, 11 in 2024-05-15T12:51:26.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:26 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2885383538' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:51:26.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:26 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:51:26.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:26 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:51:26.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:26 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:51:26.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:26 smithi120 ceph-mon[43961]: osdmap e194: 12 total, 12 up, 11 in 2024-05-15T12:51:26.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:26 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2885383538' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:51:26.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:26 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:51:27.213 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:27 smithi177 ceph-mon[44303]: pgmap v547: 17 pgs: 17 active+clean; 2.5 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 294 op/s; 3.6 MiB/s, 57 objects/s recovering 2024-05-15T12:51:27.213 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:27 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:51:27.213 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:27 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:51:27.213 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:27 smithi177 ceph-mon[44303]: osdmap e195: 12 total, 12 up, 11 in 2024-05-15T12:51:27.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:27 smithi027 ceph-mon[39795]: pgmap v547: 17 pgs: 17 active+clean; 2.5 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 294 op/s; 3.6 MiB/s, 57 objects/s recovering 2024-05-15T12:51:27.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:27 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:51:27.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:27 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:51:27.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:27 smithi027 ceph-mon[39795]: osdmap e195: 12 total, 12 up, 11 in 2024-05-15T12:51:27.438 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T12:51:27.438 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T12:51:27.438 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-05-15T12:51:27.438 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T12:51:27.495 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:51:27.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:27.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:51:27.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:27 smithi120 ceph-mon[43961]: pgmap v547: 17 pgs: 17 active+clean; 2.5 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 294 op/s; 3.6 MiB/s, 57 objects/s recovering 2024-05-15T12:51:27.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:27 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:51:27.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:27 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:51:27.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:27 smithi120 ceph-mon[43961]: osdmap e195: 12 total, 12 up, 11 in 2024-05-15T12:51:27.600 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:51:27.630 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:27.703 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:51:27.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:27.706 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:51:27.807 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:51:27.841 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:27.871 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:51:27.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:27.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:51:27.985 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:51:28.018 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:28.039 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:51:28.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:28.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:51:28.143 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:51:28.147 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-05-15T12:51:28.176 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:28.376 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:51:28.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:28.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:51:28.403 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:51:28.480 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:51:28.513 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:28.533 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:51:28.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:28.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:51:28.638 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:51:28.665 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:28.683 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:51:28.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:28.687 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:51:28.788 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:51:28.814 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:28.833 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:51:28.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:28.836 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:51:28.937 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:51:28.969 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:28.990 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:51:28.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:28.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:51:29.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:28 smithi027 ceph-mon[39795]: pgmap v549: 17 pgs: 17 active+clean; 2.5 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 77 op/s 2024-05-15T12:51:29.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:28 smithi027 ceph-mon[39795]: osdmap e196: 12 total, 12 up, 11 in 2024-05-15T12:51:29.094 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:51:29.130 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:29.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:28 smithi177 ceph-mon[44303]: pgmap v549: 17 pgs: 17 active+clean; 2.5 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 77 op/s 2024-05-15T12:51:29.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:28 smithi177 ceph-mon[44303]: osdmap e196: 12 total, 12 up, 11 in 2024-05-15T12:51:29.186 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:51:29.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:29.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:51:29.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:28 smithi120 ceph-mon[43961]: pgmap v549: 17 pgs: 17 active+clean; 2.5 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 77 op/s 2024-05-15T12:51:29.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:28 smithi120 ceph-mon[43961]: osdmap e196: 12 total, 12 up, 11 in 2024-05-15T12:51:29.290 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:51:29.321 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:29.342 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:51:29.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:29.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:51:29.446 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:51:29.476 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:29.495 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:51:29.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:29.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:51:29.600 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:51:29.628 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:29.733 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:51:29.752 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:51:29.771 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:51:29.791 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:51:29.811 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:51:29.831 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:51:29.852 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:51:29.872 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:51:29.876 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:51:29.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:29.879 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:51:29.892 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:51:29.912 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:51:29.937 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:51:29.961 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:51:29.962 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:51:29.963 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:51:29.963 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:51:29.963 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:51:29.963 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:51:29.963 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:51:29.963 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:51:29.963 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:51:29.963 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:51:29.963 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:51:29.963 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:51:29.963 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:51:29.980 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:51:30.013 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:30.038 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:51:30.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:30.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:51:30.139 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T12:51:30.139 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:51:30.144 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:51:30.160 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:29 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1314050723' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:51:30.176 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:30.197 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:51:30.199 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:29 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1314050723' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:51:30.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:30.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:51:30.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:29 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1314050723' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:51:30.301 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:51:30.330 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:30.349 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:51:30.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:30.351 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:51:30.379 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T12:51:30.452 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:51:30.478 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:30.496 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:51:30.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:30.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:51:30.600 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:51:30.632 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:30.653 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:51:30.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:30.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:51:30.757 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:51:30.786 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:30.806 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:51:30.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:30.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:51:30.882 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:51:30.909 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:51:30.933 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.7 to 1 (10000) 2024-05-15T12:51:30.939 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:31.138 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:51:31.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:31.142 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:51:31.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:30 smithi177 ceph-mon[44303]: pgmap v551: 17 pgs: 17 active+clean; 2.5 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1 op/s 2024-05-15T12:51:31.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:30 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:51:31.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:30 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:51:31.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:30 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1107202728' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T12:51:31.191 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:30 smithi027 ceph-mon[39795]: pgmap v551: 17 pgs: 17 active+clean; 2.5 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1 op/s 2024-05-15T12:51:31.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:30 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:51:31.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:30 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:51:31.192 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:30 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1107202728' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T12:51:31.243 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:51:31.272 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:31.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:30 smithi120 ceph-mon[43961]: pgmap v551: 17 pgs: 17 active+clean; 2.5 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1 op/s 2024-05-15T12:51:31.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:30 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:51:31.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:30 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:51:31.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:30 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1107202728' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T12:51:31.470 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:51:31.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:31.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:51:31.574 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:51:31.585 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-05-15T12:51:31.606 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:31.627 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:51:31.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:31.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:51:31.731 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:51:31.762 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:31.782 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:51:31.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:31.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:51:31.884 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:51:31.886 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:51:31.915 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:31.935 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:51:31.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:31.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:51:32.036 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 8 1 2024-05-15T12:51:32.039 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:51:32.065 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:32.138 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:51:32.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:32.140 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:51:32.200 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:31 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1107202728' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T12:51:32.201 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:31 smithi177 ceph-mon[44303]: osdmap e197: 12 total, 12 up, 11 in 2024-05-15T12:51:32.225 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:31 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1107202728' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T12:51:32.225 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:31 smithi027 ceph-mon[39795]: osdmap e197: 12 total, 12 up, 11 in 2024-05-15T12:51:32.241 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:51:32.268 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:32.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:31 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1107202728' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T12:51:32.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:31 smithi120 ceph-mon[43961]: osdmap e197: 12 total, 12 up, 11 in 2024-05-15T12:51:32.617 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:51:32.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:32.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:51:32.721 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:51:32.753 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:32.774 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:51:32.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:32.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:51:32.828 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:51:32.879 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:51:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:33.021 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:32 smithi177 ceph-mon[44303]: pgmap v553: 17 pgs: 17 active+clean; 2.5 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1 op/s 2024-05-15T12:51:33.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:32 smithi120 ceph-mon[43961]: pgmap v553: 17 pgs: 17 active+clean; 2.5 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1 op/s 2024-05-15T12:51:33.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:32 smithi027 ceph-mon[39795]: pgmap v553: 17 pgs: 17 active+clean; 2.5 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1 op/s 2024-05-15T12:51:33.272 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:51:33.538 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T12:51:33.538 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:51:33.959 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:51:33.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:33.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:51:34.063 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:51:34.098 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:34.124 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:51:34.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:34.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:51:34.228 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:51:34.277 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-05-15T12:51:34.295 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:34.325 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:51:34.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:34.334 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:51:34.435 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:51:34.478 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:34.507 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:51:34.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:34.513 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:51:34.614 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:51:34.670 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:51:34.683 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:34.773 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:51:34.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:34.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:51:34.909 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:51:34.952 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:34 smithi027 ceph-mon[39795]: pgmap v554: 17 pgs: 17 active+clean; 2.5 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail 2024-05-15T12:51:34.958 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:35.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:34 smithi177 ceph-mon[44303]: pgmap v554: 17 pgs: 17 active+clean; 2.5 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail 2024-05-15T12:51:35.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:34 smithi120 ceph-mon[43961]: pgmap v554: 17 pgs: 17 active+clean; 2.5 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail 2024-05-15T12:51:35.332 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:51:35.349 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:51:35.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:35.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:51:35.459 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:51:35.498 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:35.677 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:51:35.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:35.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:51:35.781 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:51:35.830 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:35.956 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:35 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3438373566' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:51:35.956 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:35 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:51:36.005 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T12:51:36.036 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:51:36.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:36.040 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:51:36.141 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:51:36.179 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:36.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:35 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3438373566' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:51:36.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:35 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:51:36.277 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:51:36.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:35 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3438373566' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:51:36.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:35 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:51:36.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:36.294 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:51:36.395 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:51:36.444 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:36.736 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T12:51:36.736 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:51:37.159 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:36 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:51:37.159 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:36 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:51:37.159 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:36 smithi177 ceph-mon[44303]: pgmap v555: 17 pgs: 17 active+clean; 2.5 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 51 op/s 2024-05-15T12:51:37.159 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:36 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:51:37.159 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:36 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:51:37.159 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:36 smithi177 ceph-mon[44303]: osdmap e198: 12 total, 12 up, 11 in 2024-05-15T12:51:37.160 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:36 smithi177 ceph-mon[44303]: 2.f scrub starts 2024-05-15T12:51:37.160 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:36 smithi177 ceph-mon[44303]: 2.3 scrub starts 2024-05-15T12:51:37.160 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:36 smithi177 ceph-mon[44303]: 2.3 scrub ok 2024-05-15T12:51:37.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:36 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:51:37.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:36 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:51:37.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:36 smithi027 ceph-mon[39795]: pgmap v555: 17 pgs: 17 active+clean; 2.5 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 51 op/s 2024-05-15T12:51:37.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:36 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:51:37.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:36 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:51:37.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:36 smithi027 ceph-mon[39795]: osdmap e198: 12 total, 12 up, 11 in 2024-05-15T12:51:37.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:36 smithi027 ceph-mon[39795]: 2.f scrub starts 2024-05-15T12:51:37.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:36 smithi027 ceph-mon[39795]: 2.3 scrub starts 2024-05-15T12:51:37.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:36 smithi027 ceph-mon[39795]: 2.3 scrub ok 2024-05-15T12:51:37.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:36 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:51:37.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:36 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:51:37.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:36 smithi120 ceph-mon[43961]: pgmap v555: 17 pgs: 17 active+clean; 2.5 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 51 op/s 2024-05-15T12:51:37.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:36 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:51:37.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:36 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:51:37.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:36 smithi120 ceph-mon[43961]: osdmap e198: 12 total, 12 up, 11 in 2024-05-15T12:51:37.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:36 smithi120 ceph-mon[43961]: 2.f scrub starts 2024-05-15T12:51:37.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:36 smithi120 ceph-mon[43961]: 2.3 scrub starts 2024-05-15T12:51:37.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:36 smithi120 ceph-mon[43961]: 2.3 scrub ok 2024-05-15T12:51:37.497 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-05-15T12:51:37.893 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:51:37.940 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.8 to 1 (10000) 2024-05-15T12:51:37.972 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:37 smithi027 ceph-mon[39795]: 2.f scrub ok 2024-05-15T12:51:37.972 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:37 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/248550874' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T12:51:37.972 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:37 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T12:51:37.972 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:37 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T12:51:37.972 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:37 smithi027 ceph-mon[39795]: osdmap e199: 12 total, 12 up, 11 in 2024-05-15T12:51:38.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:37 smithi120 ceph-mon[43961]: 2.f scrub ok 2024-05-15T12:51:38.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:37 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/248550874' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T12:51:38.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:37 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T12:51:38.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:37 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T12:51:38.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:37 smithi120 ceph-mon[43961]: osdmap e199: 12 total, 12 up, 11 in 2024-05-15T12:51:38.408 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:37 smithi177 ceph-mon[44303]: 2.f scrub ok 2024-05-15T12:51:38.408 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:37 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/248550874' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T12:51:38.409 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:37 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T12:51:38.409 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:37 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T12:51:38.409 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:37 smithi177 ceph-mon[44303]: osdmap e199: 12 total, 12 up, 11 in 2024-05-15T12:51:39.245 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:38 smithi177 ceph-mon[44303]: pgmap v557: 17 pgs: 17 active+clean; 2.4 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 107 op/s 2024-05-15T12:51:39.245 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:38 smithi177 ceph-mon[44303]: 2.0 scrub starts 2024-05-15T12:51:39.245 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:38 smithi177 ceph-mon[44303]: 2.0 scrub ok 2024-05-15T12:51:39.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:38 smithi027 ceph-mon[39795]: pgmap v557: 17 pgs: 17 active+clean; 2.4 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 107 op/s 2024-05-15T12:51:39.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:38 smithi027 ceph-mon[39795]: 2.0 scrub starts 2024-05-15T12:51:39.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:38 smithi027 ceph-mon[39795]: 2.0 scrub ok 2024-05-15T12:51:39.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:38 smithi120 ceph-mon[43961]: pgmap v557: 17 pgs: 17 active+clean; 2.4 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 107 op/s 2024-05-15T12:51:39.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:38 smithi120 ceph-mon[43961]: 2.0 scrub starts 2024-05-15T12:51:39.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:38 smithi120 ceph-mon[43961]: 2.0 scrub ok 2024-05-15T12:51:39.628 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 9 1 2024-05-15T12:51:39.747 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:51:39.748 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:51:39.748 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:51:39.748 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:51:39.748 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:51:39.748 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:51:39.748 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:51:39.748 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:51:39.748 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:51:39.748 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:51:39.748 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:51:39.748 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:51:39.749 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:51:39.749 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:51:39.749 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:51:39.749 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:51:39.749 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:51:39.749 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:51:39.749 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:51:39.749 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:51:39.749 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:51:39.749 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:51:39.749 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:51:39.749 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:51:39.884 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T12:51:39.885 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T12:51:39.885 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-05-15T12:51:39.885 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-05-15T12:51:39.885 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T12:51:39.999 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:40 smithi027 ceph-mon[39795]: 2.8 scrub starts 2024-05-15T12:51:39.999 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:40 smithi027 ceph-mon[39795]: 2.c scrub starts 2024-05-15T12:51:39.999 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:40 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3638796436' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:51:39.999 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:40 smithi027 ceph-mon[39795]: 2.8 scrub ok 2024-05-15T12:51:40.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:40 smithi177 ceph-mon[44303]: 2.8 scrub starts 2024-05-15T12:51:40.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:40 smithi177 ceph-mon[44303]: 2.c scrub starts 2024-05-15T12:51:40.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:40 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3638796436' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:51:40.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:40 smithi177 ceph-mon[44303]: 2.8 scrub ok 2024-05-15T12:51:40.246 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:51:40.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:40.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:51:40.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:40 smithi120 ceph-mon[43961]: 2.8 scrub starts 2024-05-15T12:51:40.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:40 smithi120 ceph-mon[43961]: 2.c scrub starts 2024-05-15T12:51:40.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:40 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3638796436' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:51:40.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:40 smithi120 ceph-mon[43961]: 2.8 scrub ok 2024-05-15T12:51:40.349 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:51:40.397 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:40.563 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:51:40.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:40.570 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:51:40.600 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-05-15T12:51:40.671 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:51:40.724 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:40.758 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:51:40.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:40.762 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:51:40.863 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:51:40.915 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:40.942 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:51:40.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:40.947 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:51:41.007 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:41 smithi027 ceph-mon[39795]: pgmap v559: 17 pgs: 17 active+clean; 2.4 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 261 op/s 2024-05-15T12:51:41.008 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:41 smithi027 ceph-mon[39795]: 2.c scrub ok 2024-05-15T12:51:41.008 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:41 smithi027 ceph-mon[39795]: 2.b scrub starts 2024-05-15T12:51:41.018 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:51:41.048 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:51:41.090 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:41.137 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:51:41.169 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:51:41.172 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:41 smithi177 ceph-mon[44303]: pgmap v559: 17 pgs: 17 active+clean; 2.4 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 261 op/s 2024-05-15T12:51:41.172 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:41 smithi177 ceph-mon[44303]: 2.c scrub ok 2024-05-15T12:51:41.172 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:41 smithi177 ceph-mon[44303]: 2.b scrub starts 2024-05-15T12:51:41.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:41.173 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:51:41.274 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:51:41.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:41 smithi120 ceph-mon[43961]: pgmap v559: 17 pgs: 17 active+clean; 2.4 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 261 op/s 2024-05-15T12:51:41.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:41 smithi120 ceph-mon[43961]: 2.c scrub ok 2024-05-15T12:51:41.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:41 smithi120 ceph-mon[43961]: 2.b scrub starts 2024-05-15T12:51:41.304 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:41.328 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:51:41.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:41.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:51:41.432 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:51:41.480 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:41.501 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:51:41.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:41.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:51:41.608 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:51:41.661 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:41.837 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:51:41.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:41.840 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:51:41.941 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:51:41.974 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:41.993 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:51:41.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:41.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:51:42.098 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:51:42.126 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:42.145 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:51:42.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:42.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:51:42.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:42 smithi177 ceph-mon[44303]: 2.b scrub ok 2024-05-15T12:51:42.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:42 smithi177 ceph-mon[44303]: 2.5 scrub starts 2024-05-15T12:51:42.242 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T12:51:42.247 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:51:42.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:42 smithi027 ceph-mon[39795]: 2.b scrub ok 2024-05-15T12:51:42.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:42 smithi027 ceph-mon[39795]: 2.5 scrub starts 2024-05-15T12:51:42.279 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:42 smithi120 ceph-mon[43961]: 2.b scrub ok 2024-05-15T12:51:42.279 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:42 smithi120 ceph-mon[43961]: 2.5 scrub starts 2024-05-15T12:51:42.281 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:42.308 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:51:42.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:42.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:51:42.415 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:51:42.442 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:42.466 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:51:42.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:42.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:51:42.570 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:51:42.610 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:42.715 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:51:42.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:42.720 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:51:42.821 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:51:42.865 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:42.892 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:51:42.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:42.895 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:51:42.996 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:51:43.033 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:43 smithi177 ceph-mon[44303]: pgmap v560: 17 pgs: 17 active+clean; 2.2 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 557 op/s 2024-05-15T12:51:43.033 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:43 smithi177 ceph-mon[44303]: 2.5 scrub ok 2024-05-15T12:51:43.041 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:43.070 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:51:43.072 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T12:51:43.072 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-05-15T12:51:43.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-05-15T12:51:43.072 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T12:51:43.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.072 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:119150 2.4 e153/144 2:27ad4cd4:::benchmark_data_smithi177_78104_object12017:head v 153'7379, mlcod=153'7369)", 2024-05-15T12:51:43.073 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:10.624006+0000", 2024-05-15T12:51:43.073 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.449367348999999, 2024-05-15T12:51:43.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021522151, 2024-05-15T12:51:43.073 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:43.073 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:43.073 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:43.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:43.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624006+0000", 2024-05-15T12:51:43.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:43.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624006+0000", 2024-05-15T12:51:43.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:43.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624006+0000", 2024-05-15T12:51:43.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.99e-07 2024-05-15T12:51:43.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:43.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624013+0000", 2024-05-15T12:51:43.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0539999999999997e-06 2024-05-15T12:51:43.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:43.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624019+0000", 2024-05-15T12:51:43.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6350000000000003e-06 2024-05-15T12:51:43.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:43.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624024+0000", 2024-05-15T12:51:43.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8790000000000002e-06 2024-05-15T12:51:43.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:43.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.635988+0000", 2024-05-15T12:51:43.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011964011 2024-05-15T12:51:43.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:43.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.636008+0000", 2024-05-15T12:51:43.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0837999999999999e-05 2024-05-15T12:51:43.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:43.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.645483+0000", 2024-05-15T12:51:43.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0094744570000000004 2024-05-15T12:51:43.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:43.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.645528+0000", 2024-05-15T12:51:43.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5077999999999998e-05 2024-05-15T12:51:43.077 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.077 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:43.077 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.077 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:119168 2.4 e153/144 2:2f7ee5d7:::benchmark_data_smithi177_78104_object12035:head v 153'7380, mlcod=153'7369)", 2024-05-15T12:51:43.077 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:10.624029+0000", 2024-05-15T12:51:43.077 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.449343739, 2024-05-15T12:51:43.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021544813999999999, 2024-05-15T12:51:43.077 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:43.077 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:43.078 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:43.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:43.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624029+0000", 2024-05-15T12:51:43.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:43.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624029+0000", 2024-05-15T12:51:43.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:43.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624030+0000", 2024-05-15T12:51:43.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.66e-07 2024-05-15T12:51:43.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:43.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624033+0000", 2024-05-15T12:51:43.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9500000000000001e-06 2024-05-15T12:51:43.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:43.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624038+0000", 2024-05-15T12:51:43.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0499999999999999e-06 2024-05-15T12:51:43.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:43.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624042+0000", 2024-05-15T12:51:43.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2069999999999997e-06 2024-05-15T12:51:43.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:43.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.637448+0000", 2024-05-15T12:51:43.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013406114 2024-05-15T12:51:43.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:43.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.637471+0000", 2024-05-15T12:51:43.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3288000000000001e-05 2024-05-15T12:51:43.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:43.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.645532+0000", 2024-05-15T12:51:43.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0080605499999999997 2024-05-15T12:51:43.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:43.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.645574+0000", 2024-05-15T12:51:43.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2488999999999999e-05 2024-05-15T12:51:43.081 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.081 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:43.081 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.082 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:119181 2.4 e153/144 2:297c213d:::benchmark_data_smithi177_78104_object12048:head v 153'7381, mlcod=153'7369)", 2024-05-15T12:51:43.082 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:10.624047+0000", 2024-05-15T12:51:43.082 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.449326459999995, 2024-05-15T12:51:43.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028294039999999999, 2024-05-15T12:51:43.082 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:43.082 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:43.082 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:43.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:43.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624047+0000", 2024-05-15T12:51:43.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:43.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624047+0000", 2024-05-15T12:51:43.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:43.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624047+0000", 2024-05-15T12:51:43.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5599999999999999e-07 2024-05-15T12:51:43.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:43.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624050+0000", 2024-05-15T12:51:43.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.869e-06 2024-05-15T12:51:43.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:43.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624054+0000", 2024-05-15T12:51:43.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4410000000000003e-06 2024-05-15T12:51:43.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:43.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624059+0000", 2024-05-15T12:51:43.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5009999999999998e-06 2024-05-15T12:51:43.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:43.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.638902+0000", 2024-05-15T12:51:43.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014843294999999999 2024-05-15T12:51:43.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:43.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.638917+0000", 2024-05-15T12:51:43.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5122e-05 2024-05-15T12:51:43.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:43.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652308+0000", 2024-05-15T12:51:43.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013390786999999999 2024-05-15T12:51:43.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:43.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652341+0000", 2024-05-15T12:51:43.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2869e-05 2024-05-15T12:51:43.086 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.086 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:43.086 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.086 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:119196 2.4 e153/144 2:2646c09a:::benchmark_data_smithi177_78104_object12063:head v 153'7382, mlcod=153'7369)", 2024-05-15T12:51:43.086 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:10.624063+0000", 2024-05-15T12:51:43.086 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.449310186999995, 2024-05-15T12:51:43.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028311266000000002, 2024-05-15T12:51:43.086 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:43.086 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:43.086 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:43.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:43.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624063+0000", 2024-05-15T12:51:43.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:43.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624063+0000", 2024-05-15T12:51:43.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:43.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624063+0000", 2024-05-15T12:51:43.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.05e-07 2024-05-15T12:51:43.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:43.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624071+0000", 2024-05-15T12:51:43.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3069999999999997e-06 2024-05-15T12:51:43.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:43.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624076+0000", 2024-05-15T12:51:43.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2830000000000001e-06 2024-05-15T12:51:43.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:43.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624080+0000", 2024-05-15T12:51:43.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4360000000000004e-06 2024-05-15T12:51:43.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:43.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.640837+0000", 2024-05-15T12:51:43.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.016756356 2024-05-15T12:51:43.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:43.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.640857+0000", 2024-05-15T12:51:43.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0786999999999999e-05 2024-05-15T12:51:43.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:43.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652344+0000", 2024-05-15T12:51:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011486876 2024-05-15T12:51:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652374+0000", 2024-05-15T12:51:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0015999999999999e-05 2024-05-15T12:51:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:119223 2.4 e153/144 2:2892d1d8:::benchmark_data_smithi177_78104_object12090:head v 153'7383, mlcod=153'7369)", 2024-05-15T12:51:43.090 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:10.624086+0000", 2024-05-15T12:51:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.449287095000003, 2024-05-15T12:51:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028317624999999999, 2024-05-15T12:51:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624086+0000", 2024-05-15T12:51:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624086+0000", 2024-05-15T12:51:43.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:43.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624086+0000", 2024-05-15T12:51:43.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8699999999999999e-07 2024-05-15T12:51:43.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:43.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624089+0000", 2024-05-15T12:51:43.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2219999999999998e-06 2024-05-15T12:51:43.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:43.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624095+0000", 2024-05-15T12:51:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0000000000000002e-06 2024-05-15T12:51:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624099+0000", 2024-05-15T12:51:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9840000000000003e-06 2024-05-15T12:51:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.643649+0000", 2024-05-15T12:51:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.019549818 2024-05-15T12:51:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.643672+0000", 2024-05-15T12:51:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2259000000000002e-05 2024-05-15T12:51:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652379+0000", 2024-05-15T12:51:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0087074639999999998 2024-05-15T12:51:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652404+0000", 2024-05-15T12:51:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4691000000000001e-05 2024-05-15T12:51:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.094 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:119236 2.4 e153/144 2:222c6e22:::benchmark_data_smithi177_78104_object12103:head v 153'7384, mlcod=153'7369)", 2024-05-15T12:51:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:10.624106+0000", 2024-05-15T12:51:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.449267403999997, 2024-05-15T12:51:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028323482000000001, 2024-05-15T12:51:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624106+0000", 2024-05-15T12:51:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624106+0000", 2024-05-15T12:51:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624106+0000", 2024-05-15T12:51:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5200000000000001e-07 2024-05-15T12:51:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624109+0000", 2024-05-15T12:51:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2540000000000001e-06 2024-05-15T12:51:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624115+0000", 2024-05-15T12:51:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1829999999999999e-06 2024-05-15T12:51:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624120+0000", 2024-05-15T12:51:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1670000000000001e-06 2024-05-15T12:51:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.645610+0000", 2024-05-15T12:51:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021490510000000001 2024-05-15T12:51:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.645624+0000", 2024-05-15T12:51:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4372999999999999e-05 2024-05-15T12:51:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652406+0000", 2024-05-15T12:51:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0067820729999999996 2024-05-15T12:51:43.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652429+0000", 2024-05-15T12:51:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2770000000000001e-05 2024-05-15T12:51:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:119240 2.4 e153/144 2:2ed0c2c5:::benchmark_data_smithi177_78104_object12107:head v 153'7385, mlcod=153'7369)", 2024-05-15T12:51:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:10.624126+0000", 2024-05-15T12:51:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.449247413999998, 2024-05-15T12:51:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028331313, 2024-05-15T12:51:43.099 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624126+0000", 2024-05-15T12:51:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624126+0000", 2024-05-15T12:51:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624126+0000", 2024-05-15T12:51:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.97e-07 2024-05-15T12:51:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624132+0000", 2024-05-15T12:51:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0939999999999996e-06 2024-05-15T12:51:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624137+0000", 2024-05-15T12:51:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.417e-06 2024-05-15T12:51:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624142+0000", 2024-05-15T12:51:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0679999999999998e-06 2024-05-15T12:51:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.647795+0000", 2024-05-15T12:51:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.023653422 2024-05-15T12:51:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.647819+0000", 2024-05-15T12:51:43.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3946000000000001e-05 2024-05-15T12:51:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652435+0000", 2024-05-15T12:51:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004615638 2024-05-15T12:51:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652457+0000", 2024-05-15T12:51:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2531000000000002e-05 2024-05-15T12:51:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.104 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:119259 2.4 e153/144 2:21f9b17e:::benchmark_data_smithi177_78104_object12126:head v 153'7386, mlcod=153'7369)", 2024-05-15T12:51:43.104 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:10.624148+0000", 2024-05-15T12:51:43.104 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.449225522000006, 2024-05-15T12:51:43.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028337528000000001, 2024-05-15T12:51:43.104 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:43.104 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:43.104 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:43.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:43.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624148+0000", 2024-05-15T12:51:43.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:43.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624148+0000", 2024-05-15T12:51:43.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:43.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624148+0000", 2024-05-15T12:51:43.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.98e-07 2024-05-15T12:51:43.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:43.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624151+0000", 2024-05-15T12:51:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2270000000000002e-06 2024-05-15T12:51:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624157+0000", 2024-05-15T12:51:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9379999999999997e-06 2024-05-15T12:51:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624161+0000", 2024-05-15T12:51:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9249999999999997e-06 2024-05-15T12:51:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.650016+0000", 2024-05-15T12:51:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.025854855 2024-05-15T12:51:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.650040+0000", 2024-05-15T12:51:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4402000000000001e-05 2024-05-15T12:51:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652461+0000", 2024-05-15T12:51:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024205590000000001 2024-05-15T12:51:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652485+0000", 2024-05-15T12:51:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4423999999999999e-05 2024-05-15T12:51:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(2.4 153/144 [PushOp(2:25f9c844:::benchmark_data_smithi177_78104_object18168:head, version: 115'1116, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:25f9c844:::benchmark_data_smithi177_78104_object18168:head@115'1116, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-05-15T12:51:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:10.624171+0000", 2024-05-15T12:51:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.449201880000004, 2024-05-15T12:51:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028502006999999999, 2024-05-15T12:51:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:43.109 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T12:51:43.110 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:43.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:43.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624171+0000", 2024-05-15T12:51:43.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:43.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624171+0000", 2024-05-15T12:51:43.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:43.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624171+0000", 2024-05-15T12:51:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3999999999999995e-08 2024-05-15T12:51:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624204+0000", 2024-05-15T12:51:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2954999999999999e-05 2024-05-15T12:51:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624218+0000", 2024-05-15T12:51:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3172e-05 2024-05-15T12:51:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:43.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624227+0000", 2024-05-15T12:51:43.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4890000000000001e-06 2024-05-15T12:51:43.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:43.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652529+0000", 2024-05-15T12:51:43.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028302054 2024-05-15T12:51:43.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:43.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652538+0000", 2024-05-15T12:51:43.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9260000000000006e-06 2024-05-15T12:51:43.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652673+0000", 2024-05-15T12:51:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013531700000000001 2024-05-15T12:51:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:157241 2.0 e180/144 2:0be9a21b:::benchmark_data_smithi177_78104_object50108:head v 180'9907, mlcod=180'9903)", 2024-05-15T12:51:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:59.308865+0000", 2024-05-15T12:51:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 43.764508071999998, 2024-05-15T12:51:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021685632999999999, 2024-05-15T12:51:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:43.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308865+0000", 2024-05-15T12:51:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308865+0000", 2024-05-15T12:51:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308865+0000", 2024-05-15T12:51:43.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.41e-07 2024-05-15T12:51:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308869+0000", 2024-05-15T12:51:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8070000000000001e-06 2024-05-15T12:51:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308876+0000", 2024-05-15T12:51:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.092e-06 2024-05-15T12:51:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308881+0000", 2024-05-15T12:51:43.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7360000000000001e-06 2024-05-15T12:51:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.315813+0000", 2024-05-15T12:51:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.006931698 2024-05-15T12:51:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.315828+0000", 2024-05-15T12:51:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5548000000000001e-05 2024-05-15T12:51:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.330522+0000", 2024-05-15T12:51:43.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014693733 2024-05-15T12:51:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.330551+0000", 2024-05-15T12:51:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8778000000000001e-05 2024-05-15T12:51:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:157283 2.0 e180/144 2:08447ce8:::benchmark_data_smithi177_78104_object50150:head v 180'9908, mlcod=180'9903)", 2024-05-15T12:51:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:59.308891+0000", 2024-05-15T12:51:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 43.764482016000002, 2024-05-15T12:51:43.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021701286, 2024-05-15T12:51:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308891+0000", 2024-05-15T12:51:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308891+0000", 2024-05-15T12:51:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:43.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308891+0000", 2024-05-15T12:51:43.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4999999999999999e-07 2024-05-15T12:51:43.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:43.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308895+0000", 2024-05-15T12:51:43.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.36e-06 2024-05-15T12:51:43.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:43.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308902+0000", 2024-05-15T12:51:43.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2790000000000001e-06 2024-05-15T12:51:43.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:43.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308907+0000", 2024-05-15T12:51:43.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8520000000000003e-06 2024-05-15T12:51:43.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:43.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.318873+0000", 2024-05-15T12:51:43.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0099664569999999997 2024-05-15T12:51:43.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:43.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.318899+0000", 2024-05-15T12:51:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6088000000000002e-05 2024-05-15T12:51:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.330555+0000", 2024-05-15T12:51:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011655096 2024-05-15T12:51:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.330592+0000", 2024-05-15T12:51:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7904000000000002e-05 2024-05-15T12:51:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:157300 2.0 e180/144 2:061be614:::benchmark_data_smithi177_78104_object50167:head v 180'9909, mlcod=180'9903)", 2024-05-15T12:51:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:59.308914+0000", 2024-05-15T12:51:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 43.764459135999999, 2024-05-15T12:51:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.031795017000000002, 2024-05-15T12:51:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308914+0000", 2024-05-15T12:51:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308914+0000", 2024-05-15T12:51:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308914+0000", 2024-05-15T12:51:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.03e-07 2024-05-15T12:51:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308923+0000", 2024-05-15T12:51:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7080000000000003e-06 2024-05-15T12:51:43.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:43.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308929+0000", 2024-05-15T12:51:43.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0109999999999997e-06 2024-05-15T12:51:43.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:43.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308935+0000", 2024-05-15T12:51:43.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9159999999999998e-06 2024-05-15T12:51:43.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:43.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.322985+0000", 2024-05-15T12:51:43.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014050104000000001 2024-05-15T12:51:43.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.323000+0000", 2024-05-15T12:51:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4973e-05 2024-05-15T12:51:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.340664+0000", 2024-05-15T12:51:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017664418000000001 2024-05-15T12:51:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.340709+0000", 2024-05-15T12:51:43.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4684000000000001e-05 2024-05-15T12:51:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:157344 2.0 e180/144 2:025f35be:::benchmark_data_smithi177_78104_object50211:head v 180'9910, mlcod=180'9903)", 2024-05-15T12:51:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:59.308941+0000", 2024-05-15T12:51:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 43.764432636000002, 2024-05-15T12:51:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.035675354999999999, 2024-05-15T12:51:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308941+0000", 2024-05-15T12:51:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308941+0000", 2024-05-15T12:51:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308941+0000", 2024-05-15T12:51:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.98e-07 2024-05-15T12:51:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308944+0000", 2024-05-15T12:51:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5190000000000001e-06 2024-05-15T12:51:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308950+0000", 2024-05-15T12:51:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5709999999999997e-06 2024-05-15T12:51:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308955+0000", 2024-05-15T12:51:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8230000000000002e-06 2024-05-15T12:51:43.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.326782+0000", 2024-05-15T12:51:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.01782688 2024-05-15T12:51:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.326806+0000", 2024-05-15T12:51:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4964e-05 2024-05-15T12:51:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.344552+0000", 2024-05-15T12:51:43.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017745133999999999 2024-05-15T12:51:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.344616+0000", 2024-05-15T12:51:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4265999999999998e-05 2024-05-15T12:51:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:157358 2.0 e180/144 2:0b657229:::benchmark_data_smithi177_78104_object50225:head v 180'9911, mlcod=180'9903)", 2024-05-15T12:51:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:59.308961+0000", 2024-05-15T12:51:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 43.764412014000001, 2024-05-15T12:51:43.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.035687647000000003, 2024-05-15T12:51:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308961+0000", 2024-05-15T12:51:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308961+0000", 2024-05-15T12:51:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:43.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308961+0000", 2024-05-15T12:51:43.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.55e-07 2024-05-15T12:51:43.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:43.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308965+0000", 2024-05-15T12:51:43.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3840000000000001e-06 2024-05-15T12:51:43.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:43.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308970+0000", 2024-05-15T12:51:43.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.536e-06 2024-05-15T12:51:43.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308975+0000", 2024-05-15T12:51:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.454e-06 2024-05-15T12:51:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.330632+0000", 2024-05-15T12:51:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021657725999999999 2024-05-15T12:51:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.330655+0000", 2024-05-15T12:51:43.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.279e-05 2024-05-15T12:51:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.344620+0000", 2024-05-15T12:51:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013964682000000001 2024-05-15T12:51:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.344649+0000", 2024-05-15T12:51:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8920000000000001e-05 2024-05-15T12:51:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.135 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:157360 2.0 e180/144 2:03d63169:::benchmark_data_smithi177_78104_object50227:head v 180'9912, mlcod=180'9903)", 2024-05-15T12:51:43.135 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:59.308981+0000", 2024-05-15T12:51:43.135 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 43.764392188000002, 2024-05-15T12:51:43.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.035698306999999999, 2024-05-15T12:51:43.135 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:43.135 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:43.135 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:43.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:43.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308981+0000", 2024-05-15T12:51:43.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308981+0000", 2024-05-15T12:51:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308981+0000", 2024-05-15T12:51:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0200000000000001e-07 2024-05-15T12:51:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308988+0000", 2024-05-15T12:51:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1960000000000002e-06 2024-05-15T12:51:43.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308995+0000", 2024-05-15T12:51:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7649999999999996e-06 2024-05-15T12:51:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.309000+0000", 2024-05-15T12:51:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7040000000000002e-06 2024-05-15T12:51:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.333688+0000", 2024-05-15T12:51:43.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.024688627000000001 2024-05-15T12:51:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.333711+0000", 2024-05-15T12:51:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2520000000000001e-05 2024-05-15T12:51:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.344651+0000", 2024-05-15T12:51:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010940340999999999 2024-05-15T12:51:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.344679+0000", 2024-05-15T12:51:43.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7951999999999999e-05 2024-05-15T12:51:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:157385 2.0 e180/144 2:0bdb78a8:::benchmark_data_smithi177_78104_object50252:head v 180'9913, mlcod=180'9903)", 2024-05-15T12:51:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:59.309006+0000", 2024-05-15T12:51:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 43.764366897999999, 2024-05-15T12:51:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.035703602000000001, 2024-05-15T12:51:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.309006+0000", 2024-05-15T12:51:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.309006+0000", 2024-05-15T12:51:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.309006+0000", 2024-05-15T12:51:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8699999999999999e-07 2024-05-15T12:51:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.309010+0000", 2024-05-15T12:51:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2100000000000002e-06 2024-05-15T12:51:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.309016+0000", 2024-05-15T12:51:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6710000000000002e-06 2024-05-15T12:51:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.309021+0000", 2024-05-15T12:51:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6040000000000003e-06 2024-05-15T12:51:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:43.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.337608+0000", 2024-05-15T12:51:43.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028586711000000001 2024-05-15T12:51:43.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:43.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.337632+0000", 2024-05-15T12:51:43.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4108e-05 2024-05-15T12:51:43.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:43.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.344683+0000", 2024-05-15T12:51:43.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0070516449999999996 2024-05-15T12:51:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.344710+0000", 2024-05-15T12:51:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6466e-05 2024-05-15T12:51:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(2.0 180/144 [PushOp(2:01179965:::benchmark_data_smithi177_78104_object56722:head, version: 127'3544, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:01179965:::benchmark_data_smithi177_78104_object56722:head@127'3544, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-05-15T12:51:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:59.312898+0000", 2024-05-15T12:51:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 43.760474766999998, 2024-05-15T12:51:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.032109265999999997, 2024-05-15T12:51:43.143 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T12:51:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.312898+0000", 2024-05-15T12:51:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.312898+0000", 2024-05-15T12:51:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:43.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.312899+0000", 2024-05-15T12:51:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.16e-07 2024-05-15T12:51:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.313019+0000", 2024-05-15T12:51:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000119511 2024-05-15T12:51:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.313041+0000", 2024-05-15T12:51:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2566999999999999e-05 2024-05-15T12:51:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.313057+0000", 2024-05-15T12:51:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5845000000000002e-05 2024-05-15T12:51:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.344767+0000", 2024-05-15T12:51:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.031709933000000003 2024-05-15T12:51:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.344781+0000", 2024-05-15T12:51:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4307999999999999e-05 2024-05-15T12:51:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.345008+0000", 2024-05-15T12:51:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022628599999999999 2024-05-15T12:51:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "pg_backfill(progress 2.0 e 180/180 lb 2:01160b9d:::benchmark_data_smithi177_78104_object97643:head)", 2024-05-15T12:51:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:59.313079+0000", 2024-05-15T12:51:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 43.760293703999999, 2024-05-15T12:51:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.027718355, 2024-05-15T12:51:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:43.147 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T12:51:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.313079+0000", 2024-05-15T12:51:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.313079+0000", 2024-05-15T12:51:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.313080+0000", 2024-05-15T12:51:43.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.46e-07 2024-05-15T12:51:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.313087+0000", 2024-05-15T12:51:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3830000000000003e-06 2024-05-15T12:51:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.313096+0000", 2024-05-15T12:51:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3019999999999995e-06 2024-05-15T12:51:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:43.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.313101+0000", 2024-05-15T12:51:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7300000000000002e-06 2024-05-15T12:51:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.340752+0000", 2024-05-15T12:51:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.027650464 2024-05-15T12:51:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.340762+0000", 2024-05-15T12:51:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0125e-05 2024-05-15T12:51:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:43.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.340798+0000", 2024-05-15T12:51:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6004999999999998e-05 2024-05-15T12:51:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:157403 2.0 e180/144 2:0bc41ae4:::benchmark_data_smithi177_78104_object50270:head v 180'9914, mlcod=180'9903)", 2024-05-15T12:51:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:59.324335+0000", 2024-05-15T12:51:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 43.749038431000002, 2024-05-15T12:51:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.022944787000000001, 2024-05-15T12:51:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:43.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.324335+0000", 2024-05-15T12:51:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.324335+0000", 2024-05-15T12:51:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.324336+0000", 2024-05-15T12:51:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3099999999999996e-07 2024-05-15T12:51:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.324344+0000", 2024-05-15T12:51:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8979999999999996e-06 2024-05-15T12:51:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.324357+0000", 2024-05-15T12:51:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3190000000000001e-05 2024-05-15T12:51:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.324364+0000", 2024-05-15T12:51:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2960000000000001e-06 2024-05-15T12:51:43.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.340841+0000", 2024-05-15T12:51:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.016476764000000001 2024-05-15T12:51:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.340855+0000", 2024-05-15T12:51:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3903999999999999e-05 2024-05-15T12:51:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.347235+0000", 2024-05-15T12:51:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0063801379999999996 2024-05-15T12:51:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.347280+0000", 2024-05-15T12:51:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4666000000000002e-05 2024-05-15T12:51:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:179738 2.0 e199/190 2:0e26ef05:::benchmark_data_smithi177_78104_object72605:head v 199'11295, mlcod=199'11293)", 2024-05-15T12:51:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:51:41.692674+0000", 2024-05-15T12:51:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.380698942, 2024-05-15T12:51:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021588042000000002, 2024-05-15T12:51:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:41.692674+0000", 2024-05-15T12:51:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:41.692674+0000", 2024-05-15T12:51:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:41.692675+0000", 2024-05-15T12:51:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4600000000000003e-07 2024-05-15T12:51:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:41.692683+0000", 2024-05-15T12:51:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2570000000000007e-06 2024-05-15T12:51:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:41.692698+0000", 2024-05-15T12:51:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4848000000000001e-05 2024-05-15T12:51:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:43.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:41.692707+0000", 2024-05-15T12:51:43.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5529999999999993e-06 2024-05-15T12:51:43.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:43.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:41.692827+0000", 2024-05-15T12:51:43.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000119846 2024-05-15T12:51:43.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:43.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:41.692850+0000", 2024-05-15T12:51:43.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3076000000000001e-05 2024-05-15T12:51:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:41.714216+0000", 2024-05-15T12:51:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021366369 2024-05-15T12:51:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:41.714262+0000", 2024-05-15T12:51:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6146999999999997e-05 2024-05-15T12:51:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:43.159 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:43.160 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T12:51:43.160 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:51:43.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:43.161 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:51:43.262 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:51:43.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:43 smithi120 ceph-mon[43961]: pgmap v560: 17 pgs: 17 active+clean; 2.2 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 557 op/s 2024-05-15T12:51:43.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:43 smithi120 ceph-mon[43961]: 2.5 scrub ok 2024-05-15T12:51:43.299 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:43.323 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:51:43.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:43.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:51:43.429 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:51:43.499 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:43 smithi027 ceph-mon[39795]: pgmap v560: 17 pgs: 17 active+clean; 2.2 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 557 op/s 2024-05-15T12:51:43.499 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:43 smithi027 ceph-mon[39795]: 2.5 scrub ok 2024-05-15T12:51:43.501 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:43.759 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-05-15T12:51:44.133 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:51:45.047 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:45 smithi177 ceph-mon[44303]: pgmap v561: 17 pgs: 17 active+clean; 2.1 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 724 op/s 2024-05-15T12:51:45.047 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:45 smithi177 ceph-mon[44303]: 2.2 scrub starts 2024-05-15T12:51:45.047 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:45 smithi177 ceph-mon[44303]: 2.2 scrub ok 2024-05-15T12:51:45.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:45 smithi027 ceph-mon[39795]: pgmap v561: 17 pgs: 17 active+clean; 2.1 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 724 op/s 2024-05-15T12:51:45.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:45 smithi027 ceph-mon[39795]: 2.2 scrub starts 2024-05-15T12:51:45.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:45 smithi027 ceph-mon[39795]: 2.2 scrub ok 2024-05-15T12:51:45.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:45 smithi120 ceph-mon[43961]: pgmap v561: 17 pgs: 17 active+clean; 2.1 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 724 op/s 2024-05-15T12:51:45.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:45 smithi120 ceph-mon[43961]: 2.2 scrub starts 2024-05-15T12:51:45.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:45 smithi120 ceph-mon[43961]: 2.2 scrub ok 2024-05-15T12:51:46.132 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T12:51:46.132 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T12:51:46.132 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-05-15T12:51:46.132 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T12:51:47.340 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:47 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:51:47.340 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:47 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:51:47.340 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:47 smithi027 ceph-mon[39795]: pgmap v562: 17 pgs: 17 active+clean; 2.0 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 746 op/s 2024-05-15T12:51:47.340 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:47 smithi027 ceph-mon[39795]: osdmap e200: 12 total, 12 up, 11 in 2024-05-15T12:51:47.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:47 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:51:47.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:47 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:51:47.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:47 smithi177 ceph-mon[44303]: pgmap v562: 17 pgs: 17 active+clean; 2.0 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 746 op/s 2024-05-15T12:51:47.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:47 smithi177 ceph-mon[44303]: osdmap e200: 12 total, 12 up, 11 in 2024-05-15T12:51:47.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:47 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:51:47.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:47 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:51:47.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:47 smithi120 ceph-mon[43961]: pgmap v562: 17 pgs: 17 active+clean; 2.0 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 746 op/s 2024-05-15T12:51:47.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:47 smithi120 ceph-mon[43961]: osdmap e200: 12 total, 12 up, 11 in 2024-05-15T12:51:47.985 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-05-15T12:51:48.369 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:51:49.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:48 smithi027 ceph-mon[39795]: pgmap v564: 17 pgs: 17 active+clean; 2.0 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 753 op/s 2024-05-15T12:51:49.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:48 smithi027 ceph-mon[39795]: osdmap e201: 12 total, 12 up, 11 in 2024-05-15T12:51:49.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:48 smithi120 ceph-mon[43961]: pgmap v564: 17 pgs: 17 active+clean; 2.0 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 753 op/s 2024-05-15T12:51:49.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:48 smithi120 ceph-mon[43961]: osdmap e201: 12 total, 12 up, 11 in 2024-05-15T12:51:49.354 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:48 smithi177 ceph-mon[44303]: pgmap v564: 17 pgs: 17 active+clean; 2.0 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 753 op/s 2024-05-15T12:51:49.354 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:48 smithi177 ceph-mon[44303]: osdmap e201: 12 total, 12 up, 11 in 2024-05-15T12:51:50.370 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T12:51:50.370 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T12:51:50.370 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-05-15T12:51:50.370 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-05-15T12:51:50.370 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T12:51:51.049 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:50 smithi177 ceph-mon[44303]: pgmap v566: 17 pgs: 17 active+clean; 1.9 GiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 757 op/s 2024-05-15T12:51:51.091 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-05-15T12:51:51.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:50 smithi027 ceph-mon[39795]: pgmap v566: 17 pgs: 17 active+clean; 1.9 GiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 757 op/s 2024-05-15T12:51:51.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:50 smithi120 ceph-mon[43961]: pgmap v566: 17 pgs: 17 active+clean; 1.9 GiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 757 op/s 2024-05-15T12:51:51.472 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:51:52.021 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:51:52.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:52.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:51:52.069 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:51:52.128 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:51:52.131 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:51:52.163 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:52.373 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:51:52.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:52.378 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:51:52.479 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:51:52.514 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:52.547 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:51:52.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:52.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:51:52.654 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:51:52.682 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:52.708 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:51:52.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:52.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:51:52.817 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:51:52.901 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:53.140 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:52 smithi177 ceph-mon[44303]: pgmap v567: 17 pgs: 17 active+clean; 1.7 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 855 op/s 2024-05-15T12:51:53.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:52 smithi027 ceph-mon[39795]: pgmap v567: 17 pgs: 17 active+clean; 1.7 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 855 op/s 2024-05-15T12:51:53.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:52 smithi120 ceph-mon[43961]: pgmap v567: 17 pgs: 17 active+clean; 1.7 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 855 op/s 2024-05-15T12:51:53.485 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T12:51:53.485 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-05-15T12:51:53.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-05-15T12:51:53.485 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T12:51:53.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.485 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:3432 2.c 2:302aa9a3:::benchmark_data_smithi177_78104_object3431:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-05-15T12:51:53.486 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:48:11.559331+0000", 2024-05-15T12:51:53.486 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 221.934260427, 2024-05-15T12:51:53.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.058142196, 2024-05-15T12:51:53.486 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:53.486 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:53.486 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:53.486 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:53.486 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:53.486 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 3432 2024-05-15T12:51:53.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.486 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:53.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.487 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:53.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.559331+0000", 2024-05-15T12:51:53.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.487 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:53.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.559331+0000", 2024-05-15T12:51:53.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.487 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:53.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.559332+0000", 2024-05-15T12:51:53.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.316e-06 2024-05-15T12:51:53.488 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.488 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:53.488 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.559355+0000", 2024-05-15T12:51:53.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3051e-05 2024-05-15T12:51:53.488 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.488 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:53.488 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.559367+0000", 2024-05-15T12:51:53.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2026e-05 2024-05-15T12:51:53.488 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.488 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:53.488 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.559375+0000", 2024-05-15T12:51:53.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6459999999999993e-06 2024-05-15T12:51:53.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:53.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.559432+0000", 2024-05-15T12:51:53.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7568000000000002e-05 2024-05-15T12:51:53.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:53.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.559553+0000", 2024-05-15T12:51:53.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012045 2024-05-15T12:51:53.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T12:51:53.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.559700+0000", 2024-05-15T12:51:53.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014723899999999999 2024-05-15T12:51:53.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:53.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.560824+0000", 2024-05-15T12:51:53.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011238210000000001 2024-05-15T12:51:53.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.560857+0000", 2024-05-15T12:51:53.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2914999999999999e-05 2024-05-15T12:51:53.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.617360+0000", 2024-05-15T12:51:53.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.056503383999999997 2024-05-15T12:51:53.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:53.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.617429+0000", 2024-05-15T12:51:53.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8856999999999995e-05 2024-05-15T12:51:53.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:53.492 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.617473+0000", 2024-05-15T12:51:53.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3922999999999999e-05 2024-05-15T12:51:53.492 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.492 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:53.492 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.492 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.492 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:3442 2.c 2:305588c6:::benchmark_data_smithi177_78104_object3441:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-05-15T12:51:53.492 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:48:11.560184+0000", 2024-05-15T12:51:53.492 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 221.93340708299999, 2024-05-15T12:51:53.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.059728746999999999, 2024-05-15T12:51:53.492 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:53.492 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:53.493 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:53.493 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:53.493 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:53.493 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 3442 2024-05-15T12:51:53.493 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.493 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:53.493 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.493 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:53.493 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.560184+0000", 2024-05-15T12:51:53.493 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.493 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.493 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:53.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.560184+0000", 2024-05-15T12:51:53.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:53.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.560185+0000", 2024-05-15T12:51:53.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7300000000000004e-07 2024-05-15T12:51:53.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:53.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.560204+0000", 2024-05-15T12:51:53.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9375999999999999e-05 2024-05-15T12:51:53.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:53.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.560217+0000", 2024-05-15T12:51:53.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2182e-05 2024-05-15T12:51:53.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:53.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.560222+0000", 2024-05-15T12:51:53.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8069999999999996e-06 2024-05-15T12:51:53.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:53.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.560268+0000", 2024-05-15T12:51:53.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5195999999999999e-05 2024-05-15T12:51:53.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:53.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.560352+0000", 2024-05-15T12:51:53.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4000999999999998e-05 2024-05-15T12:51:53.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T12:51:53.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.560481+0000", 2024-05-15T12:51:53.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012969500000000001 2024-05-15T12:51:53.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:53.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.561019+0000", 2024-05-15T12:51:53.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00053762999999999999 2024-05-15T12:51:53.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.561615+0000", 2024-05-15T12:51:53.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00059566199999999995 2024-05-15T12:51:53.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.619823+0000", 2024-05-15T12:51:53.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.058208727000000002 2024-05-15T12:51:53.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.498 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:53.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.619873+0000", 2024-05-15T12:51:53.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9975999999999997e-05 2024-05-15T12:51:53.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.498 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:53.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:11.619913+0000", 2024-05-15T12:51:53.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9521999999999999e-05 2024-05-15T12:51:53.498 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.498 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:53.498 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.499 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:47933 2.c 2:30a59271:::benchmark_data_smithi177_78104_object47932:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-05-15T12:51:53.499 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:48:39.433311+0000", 2024-05-15T12:51:53.499 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 194.06027990699999, 2024-05-15T12:51:53.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.057850240999999997, 2024-05-15T12:51:53.499 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:53.499 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:53.499 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:53.499 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:53.501 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:53.501 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 47933 2024-05-15T12:51:53.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.501 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:53.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:53.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.433311+0000", 2024-05-15T12:51:53.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:53.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.433311+0000", 2024-05-15T12:51:53.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:53.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.433312+0000", 2024-05-15T12:51:53.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.238e-06 2024-05-15T12:51:53.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:53.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.433339+0000", 2024-05-15T12:51:53.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6610000000000001e-05 2024-05-15T12:51:53.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:53.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.433349+0000", 2024-05-15T12:51:53.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0171e-05 2024-05-15T12:51:53.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:53.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.433355+0000", 2024-05-15T12:51:53.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7969999999999997e-06 2024-05-15T12:51:53.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:53.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.433411+0000", 2024-05-15T12:51:53.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6175999999999999e-05 2024-05-15T12:51:53.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:53.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.433533+0000", 2024-05-15T12:51:53.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000122077 2024-05-15T12:51:53.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T12:51:53.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.433696+0000", 2024-05-15T12:51:53.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016224699999999999 2024-05-15T12:51:53.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:53.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.436222+0000", 2024-05-15T12:51:53.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025261849999999998 2024-05-15T12:51:53.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.439064+0000", 2024-05-15T12:51:53.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0028417719999999998 2024-05-15T12:51:53.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.491096+0000", 2024-05-15T12:51:53.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.052032800999999997 2024-05-15T12:51:53.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.506 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:53.506 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.491133+0000", 2024-05-15T12:51:53.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7125000000000001e-05 2024-05-15T12:51:53.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.506 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:53.506 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.491162+0000", 2024-05-15T12:51:53.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8042e-05 2024-05-15T12:51:53.506 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.506 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:53.506 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.507 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:47937 2.c 2:32eb6495:::benchmark_data_smithi177_78104_object47936:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-05-15T12:51:53.507 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:48:39.434189+0000", 2024-05-15T12:51:53.507 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 194.05940245299999, 2024-05-15T12:51:53.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.057095474, 2024-05-15T12:51:53.507 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:53.507 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:53.507 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:53.507 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:53.507 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:53.507 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 47937 2024-05-15T12:51:53.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.507 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:53.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:53.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.434189+0000", 2024-05-15T12:51:53.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:53.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.434189+0000", 2024-05-15T12:51:53.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:53.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.434190+0000", 2024-05-15T12:51:53.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6000000000000002e-07 2024-05-15T12:51:53.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:53.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.434210+0000", 2024-05-15T12:51:53.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0398e-05 2024-05-15T12:51:53.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:53.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.434222+0000", 2024-05-15T12:51:53.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1701e-05 2024-05-15T12:51:53.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:53.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.434229+0000", 2024-05-15T12:51:53.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3189999999999997e-06 2024-05-15T12:51:53.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:53.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.434993+0000", 2024-05-15T12:51:53.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00076364399999999995 2024-05-15T12:51:53.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:53.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.435111+0000", 2024-05-15T12:51:53.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000118415 2024-05-15T12:51:53.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T12:51:53.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.435249+0000", 2024-05-15T12:51:53.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013775500000000001 2024-05-15T12:51:53.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:53.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.436324+0000", 2024-05-15T12:51:53.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010755280000000001 2024-05-15T12:51:53.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.442666+0000", 2024-05-15T12:51:53.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0063412360000000001 2024-05-15T12:51:53.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.512 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.491206+0000", 2024-05-15T12:51:53.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.048540522000000003 2024-05-15T12:51:53.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.512 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:53.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.491247+0000", 2024-05-15T12:51:53.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.104e-05 2024-05-15T12:51:53.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.512 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:53.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:39.491284+0000", 2024-05-15T12:51:53.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7055999999999999e-05 2024-05-15T12:51:53.513 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.513 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:53.513 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.513 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:49608 2.c 2:3eab675c:::benchmark_data_smithi177_78104_object49607:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-15T12:51:53.513 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:48:41.302830+0000", 2024-05-15T12:51:53.513 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 192.19076084599999, 2024-05-15T12:51:53.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.097721886999999993, 2024-05-15T12:51:53.513 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:53.513 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:53.513 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:53.513 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:53.514 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:53.514 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 49608 2024-05-15T12:51:53.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.514 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:53.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:53.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.302830+0000", 2024-05-15T12:51:53.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:53.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.302830+0000", 2024-05-15T12:51:53.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:53.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.302832+0000", 2024-05-15T12:51:53.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5820000000000001e-06 2024-05-15T12:51:53.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:53.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.302854+0000", 2024-05-15T12:51:53.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1889e-05 2024-05-15T12:51:53.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:53.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.302867+0000", 2024-05-15T12:51:53.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3277e-05 2024-05-15T12:51:53.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:53.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.302875+0000", 2024-05-15T12:51:53.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8709999999999997e-06 2024-05-15T12:51:53.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:53.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.302928+0000", 2024-05-15T12:51:53.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2877999999999998e-05 2024-05-15T12:51:53.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:53.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.303052+0000", 2024-05-15T12:51:53.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012456899999999999 2024-05-15T12:51:53.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T12:51:53.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.303190+0000", 2024-05-15T12:51:53.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013791899999999999 2024-05-15T12:51:53.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:53.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.304873+0000", 2024-05-15T12:51:53.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001682303 2024-05-15T12:51:53.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.310652+0000", 2024-05-15T12:51:53.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0057797539999999998 2024-05-15T12:51:53.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.400487+0000", 2024-05-15T12:51:53.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.089834174000000003 2024-05-15T12:51:53.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:53.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.400523+0000", 2024-05-15T12:51:53.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6551999999999998e-05 2024-05-15T12:51:53.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:53.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.400552+0000", 2024-05-15T12:51:53.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9119e-05 2024-05-15T12:51:53.519 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.519 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:53.519 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.519 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:50572 2.c 2:3664cb9e:::benchmark_data_smithi177_78104_object50571:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-15T12:51:53.519 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:48:41.874674+0000", 2024-05-15T12:51:53.519 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 191.618917396, 2024-05-15T12:51:53.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.054020788, 2024-05-15T12:51:53.520 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:53.520 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:53.520 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:53.520 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:53.520 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:53.520 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 50572 2024-05-15T12:51:53.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.520 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:53.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:53.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.874674+0000", 2024-05-15T12:51:53.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:53.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.874674+0000", 2024-05-15T12:51:53.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:53.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.874675+0000", 2024-05-15T12:51:53.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6800000000000009e-07 2024-05-15T12:51:53.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:53.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.874693+0000", 2024-05-15T12:51:53.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8549000000000001e-05 2024-05-15T12:51:53.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:53.522 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.874704+0000", 2024-05-15T12:51:53.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0886999999999999e-05 2024-05-15T12:51:53.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:53.522 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.874710+0000", 2024-05-15T12:51:53.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8799999999999996e-06 2024-05-15T12:51:53.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:53.522 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.874760+0000", 2024-05-15T12:51:53.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0173000000000002e-05 2024-05-15T12:51:53.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:53.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.874913+0000", 2024-05-15T12:51:53.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015227200000000001 2024-05-15T12:51:53.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T12:51:53.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.875038+0000", 2024-05-15T12:51:53.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012546000000000001 2024-05-15T12:51:53.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:53.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.877988+0000", 2024-05-15T12:51:53.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0029504650000000002 2024-05-15T12:51:53.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.880241+0000", 2024-05-15T12:51:53.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0022520750000000001 2024-05-15T12:51:53.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.928584+0000", 2024-05-15T12:51:53.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.048343991000000003 2024-05-15T12:51:53.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:53.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.928648+0000", 2024-05-15T12:51:53.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3145000000000006e-05 2024-05-15T12:51:53.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:53.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.928695+0000", 2024-05-15T12:51:53.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6922999999999998e-05 2024-05-15T12:51:53.525 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.525 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:53.525 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.525 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:50585 2.c 2:3df9101a:::benchmark_data_smithi177_78104_object50584:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-15T12:51:53.526 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:48:41.881494+0000", 2024-05-15T12:51:53.526 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 191.612097361, 2024-05-15T12:51:53.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.057883693999999999, 2024-05-15T12:51:53.526 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:53.526 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:53.526 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:53.526 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:53.526 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:53.526 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 50585 2024-05-15T12:51:53.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.526 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:53.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:53.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.881494+0000", 2024-05-15T12:51:53.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.527 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:53.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.881494+0000", 2024-05-15T12:51:53.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.527 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:53.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.881495+0000", 2024-05-15T12:51:53.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5099999999999998e-07 2024-05-15T12:51:53.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:53.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.881512+0000", 2024-05-15T12:51:53.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7186000000000001e-05 2024-05-15T12:51:53.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:53.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.881522+0000", 2024-05-15T12:51:53.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7820000000000005e-06 2024-05-15T12:51:53.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:53.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.881530+0000", 2024-05-15T12:51:53.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8129999999999996e-06 2024-05-15T12:51:53.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:53.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.881591+0000", 2024-05-15T12:51:53.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1420999999999994e-05 2024-05-15T12:51:53.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:53.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.881708+0000", 2024-05-15T12:51:53.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117284 2024-05-15T12:51:53.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T12:51:53.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.881851+0000", 2024-05-15T12:51:53.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014277300000000001 2024-05-15T12:51:53.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:53.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.883430+0000", 2024-05-15T12:51:53.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001578964 2024-05-15T12:51:53.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.883816+0000", 2024-05-15T12:51:53.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00038558199999999998 2024-05-15T12:51:53.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.939264+0000", 2024-05-15T12:51:53.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.055448451000000003 2024-05-15T12:51:53.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:53.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.939330+0000", 2024-05-15T12:51:53.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5727000000000003e-05 2024-05-15T12:51:53.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:53.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:41.939378+0000", 2024-05-15T12:51:53.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7759999999999997e-05 2024-05-15T12:51:53.531 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.531 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:53.532 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.532 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:50910 2.c 2:3816627d:::benchmark_data_smithi177_78104_object50909:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-15T12:51:53.532 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:48:42.103428+0000", 2024-05-15T12:51:53.532 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 191.39016324400001, 2024-05-15T12:51:53.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.069616993000000002, 2024-05-15T12:51:53.532 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:53.532 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:53.532 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:53.532 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:53.532 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:53.532 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 50910 2024-05-15T12:51:53.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.533 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:53.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:53.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.103428+0000", 2024-05-15T12:51:53.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:53.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.103428+0000", 2024-05-15T12:51:53.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:53.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.103429+0000", 2024-05-15T12:51:53.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.55e-06 2024-05-15T12:51:53.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:53.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.103460+0000", 2024-05-15T12:51:53.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0518999999999997e-05 2024-05-15T12:51:53.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:53.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.103478+0000", 2024-05-15T12:51:53.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8249000000000001e-05 2024-05-15T12:51:53.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:53.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.103486+0000", 2024-05-15T12:51:53.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7940000000000003e-06 2024-05-15T12:51:53.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:53.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.103541+0000", 2024-05-15T12:51:53.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4975000000000001e-05 2024-05-15T12:51:53.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:53.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.103667+0000", 2024-05-15T12:51:53.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012552099999999999 2024-05-15T12:51:53.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T12:51:53.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.103813+0000", 2024-05-15T12:51:53.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014647299999999999 2024-05-15T12:51:53.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:53.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.107027+0000", 2024-05-15T12:51:53.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0032143430000000001 2024-05-15T12:51:53.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.110634+0000", 2024-05-15T12:51:53.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0036070220000000001 2024-05-15T12:51:53.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.172971+0000", 2024-05-15T12:51:53.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.062336183000000003 2024-05-15T12:51:53.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:53.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.173015+0000", 2024-05-15T12:51:53.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4518000000000003e-05 2024-05-15T12:51:53.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.538 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:53.538 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.173045+0000", 2024-05-15T12:51:53.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9845999999999999e-05 2024-05-15T12:51:53.538 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.538 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:53.538 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.538 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:50912 2.c 2:37e60f04:::benchmark_data_smithi177_78104_object50911:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-15T12:51:53.538 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:48:42.104168+0000", 2024-05-15T12:51:53.538 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 191.38942280500001, 2024-05-15T12:51:53.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.069031324000000005, 2024-05-15T12:51:53.538 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:53.539 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:53.539 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:53.539 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:53.539 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:53.539 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 50912 2024-05-15T12:51:53.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.539 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:53.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:53.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.104168+0000", 2024-05-15T12:51:53.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:53.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.104168+0000", 2024-05-15T12:51:53.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:53.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.104169+0000", 2024-05-15T12:51:53.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3900000000000003e-07 2024-05-15T12:51:53.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:53.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.104190+0000", 2024-05-15T12:51:53.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0965999999999999e-05 2024-05-15T12:51:53.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:53.541 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.104202+0000", 2024-05-15T12:51:53.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1574e-05 2024-05-15T12:51:53.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:53.541 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.104210+0000", 2024-05-15T12:51:53.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0760000000000003e-06 2024-05-15T12:51:53.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:53.541 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.105363+0000", 2024-05-15T12:51:53.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001153449 2024-05-15T12:51:53.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:53.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.105480+0000", 2024-05-15T12:51:53.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000116526 2024-05-15T12:51:53.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T12:51:53.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.105620+0000", 2024-05-15T12:51:53.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014034599999999999 2024-05-15T12:51:53.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:53.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.107211+0000", 2024-05-15T12:51:53.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001590466 2024-05-15T12:51:53.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.110666+0000", 2024-05-15T12:51:53.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0034554550000000001 2024-05-15T12:51:53.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.173099+0000", 2024-05-15T12:51:53.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.062432643000000003 2024-05-15T12:51:53.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:53.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.173158+0000", 2024-05-15T12:51:53.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9333000000000003e-05 2024-05-15T12:51:53.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:53.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:42.173200+0000", 2024-05-15T12:51:53.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1551000000000002e-05 2024-05-15T12:51:53.544 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.544 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:53.545 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.545 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:53583 2.c 2:347fbf01:::benchmark_data_smithi177_78104_object53582:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-05-15T12:51:53.545 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:48:43.748792+0000", 2024-05-15T12:51:53.545 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 189.74479940200001, 2024-05-15T12:51:53.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.068598224999999999, 2024-05-15T12:51:53.545 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:53.545 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:53.545 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:53.545 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:53.545 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:53.545 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 53583 2024-05-15T12:51:53.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.546 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:53.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:53.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:43.748792+0000", 2024-05-15T12:51:53.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:53.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:43.748792+0000", 2024-05-15T12:51:53.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:53.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:43.748793+0000", 2024-05-15T12:51:53.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.53e-06 2024-05-15T12:51:53.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:53.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:43.748815+0000", 2024-05-15T12:51:53.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1234999999999999e-05 2024-05-15T12:51:53.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:53.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:43.748828+0000", 2024-05-15T12:51:53.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.31e-05 2024-05-15T12:51:53.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:53.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:43.748835+0000", 2024-05-15T12:51:53.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5340000000000002e-06 2024-05-15T12:51:53.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:53.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:43.748890+0000", 2024-05-15T12:51:53.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5185000000000001e-05 2024-05-15T12:51:53.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:53.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:43.749014+0000", 2024-05-15T12:51:53.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012337200000000001 2024-05-15T12:51:53.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T12:51:53.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:43.749153+0000", 2024-05-15T12:51:53.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013879499999999999 2024-05-15T12:51:53.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:53.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:43.750793+0000", 2024-05-15T12:51:53.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016401320000000001 2024-05-15T12:51:53.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.550 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.550 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.550 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:43.751840+0000", 2024-05-15T12:51:53.550 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010474309999999999 2024-05-15T12:51:53.550 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.550 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.550 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.550 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:43.817311+0000", 2024-05-15T12:51:53.550 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.065471014999999994 2024-05-15T12:51:53.550 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.550 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.550 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:53.550 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:43.817348+0000", 2024-05-15T12:51:53.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6752000000000003e-05 2024-05-15T12:51:53.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:53.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:43.817390+0000", 2024-05-15T12:51:53.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2144000000000002e-05 2024-05-15T12:51:53.551 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.551 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:53.551 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.551 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:54781 2.c 2:3ce3b6b1:::benchmark_data_smithi177_78104_object54780:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-05-15T12:51:53.551 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:48:45.361581+0000", 2024-05-15T12:51:53.552 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 188.13200998100001, 2024-05-15T12:51:53.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.069199384000000003, 2024-05-15T12:51:53.552 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:53.552 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:53.552 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:53.552 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:53.552 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:53.552 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54781 2024-05-15T12:51:53.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.552 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:53.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:53.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.361581+0000", 2024-05-15T12:51:53.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:53.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.361581+0000", 2024-05-15T12:51:53.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:53.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.361582+0000", 2024-05-15T12:51:53.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9800000000000003e-07 2024-05-15T12:51:53.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:53.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.361601+0000", 2024-05-15T12:51:53.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9630999999999999e-05 2024-05-15T12:51:53.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.554 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:53.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.361608+0000", 2024-05-15T12:51:53.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8739999999999997e-06 2024-05-15T12:51:53.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.554 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:53.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.361614+0000", 2024-05-15T12:51:53.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7350000000000001e-06 2024-05-15T12:51:53.555 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.555 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:53.555 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.361652+0000", 2024-05-15T12:51:53.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7926000000000002e-05 2024-05-15T12:51:53.555 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.555 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:53.555 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.361750+0000", 2024-05-15T12:51:53.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7582000000000002e-05 2024-05-15T12:51:53.555 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.555 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T12:51:53.555 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.361834+0000", 2024-05-15T12:51:53.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4751000000000002e-05 2024-05-15T12:51:53.556 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.556 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:53.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.363593+0000", 2024-05-15T12:51:53.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017590590000000001 2024-05-15T12:51:53.556 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.556 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.366026+0000", 2024-05-15T12:51:53.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024329500000000001 2024-05-15T12:51:53.556 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.556 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.557 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.557 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.430714+0000", 2024-05-15T12:51:53.557 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.064688205999999998 2024-05-15T12:51:53.557 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.557 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.557 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:53.557 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.430750+0000", 2024-05-15T12:51:53.557 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5435999999999998e-05 2024-05-15T12:51:53.557 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.557 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.557 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:53.557 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.430781+0000", 2024-05-15T12:51:53.557 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0636000000000003e-05 2024-05-15T12:51:53.558 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.558 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:53.558 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.558 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.558 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.558 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:54800 2.c 2:3d26f005:::benchmark_data_smithi177_78104_object54799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-05-15T12:51:53.558 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:48:45.369924+0000", 2024-05-15T12:51:53.558 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 188.12366682699999, 2024-05-15T12:51:53.558 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.068007463000000004, 2024-05-15T12:51:53.558 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:53.558 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:53.558 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:53.558 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:53.559 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:53.559 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54800 2024-05-15T12:51:53.559 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.559 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:53.559 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.559 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:53.559 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.369924+0000", 2024-05-15T12:51:53.559 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.559 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.559 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.559 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:53.559 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.369924+0000", 2024-05-15T12:51:53.560 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.560 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.560 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.560 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:53.560 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.369925+0000", 2024-05-15T12:51:53.560 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9399999999999993e-07 2024-05-15T12:51:53.560 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.560 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.560 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:53.560 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.369948+0000", 2024-05-15T12:51:53.560 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3068999999999999e-05 2024-05-15T12:51:53.560 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.560 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.560 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:53.561 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.369959+0000", 2024-05-15T12:51:53.561 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0864e-05 2024-05-15T12:51:53.561 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.561 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.561 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:53.561 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.369967+0000", 2024-05-15T12:51:53.561 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.695e-06 2024-05-15T12:51:53.562 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.562 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.562 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:53.562 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.370020+0000", 2024-05-15T12:51:53.562 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3038000000000003e-05 2024-05-15T12:51:53.562 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.562 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.562 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:53.562 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.370152+0000", 2024-05-15T12:51:53.562 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013174199999999999 2024-05-15T12:51:53.562 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.562 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.562 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T12:51:53.563 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.370277+0000", 2024-05-15T12:51:53.563 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012557099999999999 2024-05-15T12:51:53.563 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.563 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.563 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.563 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.373651+0000", 2024-05-15T12:51:53.563 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0033736500000000002 2024-05-15T12:51:53.563 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.563 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.563 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:53.563 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.373666+0000", 2024-05-15T12:51:53.563 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5088e-05 2024-05-15T12:51:53.564 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.564 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.564 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.564 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.437816+0000", 2024-05-15T12:51:53.564 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.064150219999999994 2024-05-15T12:51:53.564 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.564 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.564 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:53.564 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.437880+0000", 2024-05-15T12:51:53.564 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3837000000000006e-05 2024-05-15T12:51:53.564 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.564 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.564 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:53.565 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.437932+0000", 2024-05-15T12:51:53.565 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1694999999999999e-05 2024-05-15T12:51:53.565 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.565 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:53.565 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.565 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.565 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.565 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:54801 2.c 2:30f57b79:::benchmark_data_smithi177_78104_object54800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-05-15T12:51:53.565 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:48:45.371872+0000", 2024-05-15T12:51:53.565 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 188.12171941700001, 2024-05-15T12:51:53.565 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.066199939999999999, 2024-05-15T12:51:53.565 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:53.566 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:53.566 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:53.566 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:53.566 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:53.566 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54801 2024-05-15T12:51:53.566 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.566 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:53.566 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.566 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:53.566 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.371872+0000", 2024-05-15T12:51:53.566 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.566 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.567 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.567 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:53.567 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.371872+0000", 2024-05-15T12:51:53.567 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.567 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.567 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.567 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:53.567 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.371873+0000", 2024-05-15T12:51:53.567 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.015e-06 2024-05-15T12:51:53.567 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.567 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.567 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:53.567 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.371893+0000", 2024-05-15T12:51:53.567 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0360000000000002e-05 2024-05-15T12:51:53.568 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.568 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.568 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:53.568 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.371902+0000", 2024-05-15T12:51:53.568 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8920000000000006e-06 2024-05-15T12:51:53.568 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.568 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.568 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:53.568 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.371908+0000", 2024-05-15T12:51:53.568 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4990000000000002e-06 2024-05-15T12:51:53.568 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.568 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.569 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:53.569 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.372063+0000", 2024-05-15T12:51:53.569 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000155057 2024-05-15T12:51:53.569 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.569 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.569 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:53.569 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.372189+0000", 2024-05-15T12:51:53.569 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012628800000000001 2024-05-15T12:51:53.569 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.569 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.569 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T12:51:53.569 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.372367+0000", 2024-05-15T12:51:53.569 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017768699999999999 2024-05-15T12:51:53.570 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.570 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.570 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.570 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.373798+0000", 2024-05-15T12:51:53.570 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001431472 2024-05-15T12:51:53.570 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.570 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.570 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:53.570 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.373810+0000", 2024-05-15T12:51:53.570 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1688999999999999e-05 2024-05-15T12:51:53.570 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.570 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.570 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.571 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.437979+0000", 2024-05-15T12:51:53.571 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.064169344000000003 2024-05-15T12:51:53.571 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.571 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.571 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:53.571 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.438036+0000", 2024-05-15T12:51:53.571 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6502000000000002e-05 2024-05-15T12:51:53.571 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.571 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.571 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:53.571 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.438072+0000", 2024-05-15T12:51:53.571 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6134999999999999e-05 2024-05-15T12:51:53.571 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.572 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:53.572 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.572 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.572 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.572 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:54810 2.c 2:37c5e231:::benchmark_data_smithi177_78104_object54809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-05-15T12:51:53.572 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:48:45.378952+0000", 2024-05-15T12:51:53.572 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 188.11463962400001, 2024-05-15T12:51:53.572 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.063644131000000007, 2024-05-15T12:51:53.572 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:53.572 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:53.572 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:53.572 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:53.573 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:53.573 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 54810 2024-05-15T12:51:53.573 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.573 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:53.573 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.573 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:53.573 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.378952+0000", 2024-05-15T12:51:53.573 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.573 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.573 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.573 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:53.573 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.378952+0000", 2024-05-15T12:51:53.574 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.574 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.574 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.574 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:53.574 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.378953+0000", 2024-05-15T12:51:53.574 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1629999999999999e-06 2024-05-15T12:51:53.574 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.574 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.574 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:53.574 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.378971+0000", 2024-05-15T12:51:53.574 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8233e-05 2024-05-15T12:51:53.574 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.574 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.575 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:53.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.378983+0000", 2024-05-15T12:51:53.575 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1703e-05 2024-05-15T12:51:53.575 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.575 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.575 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:53.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.378988+0000", 2024-05-15T12:51:53.578 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8159999999999999e-06 2024-05-15T12:51:53.578 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.579 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.579 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:53.579 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.379031+0000", 2024-05-15T12:51:53.579 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2438999999999998e-05 2024-05-15T12:51:53.579 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.579 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.579 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:53.579 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.379118+0000", 2024-05-15T12:51:53.579 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7565999999999995e-05 2024-05-15T12:51:53.579 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.579 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.579 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T12:51:53.579 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.379211+0000", 2024-05-15T12:51:53.580 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2932999999999999e-05 2024-05-15T12:51:53.580 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.580 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.580 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.580 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.380444+0000", 2024-05-15T12:51:53.580 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001232981 2024-05-15T12:51:53.580 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.580 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.580 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:53.581 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.380547+0000", 2024-05-15T12:51:53.581 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102849 2024-05-15T12:51:53.581 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.581 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.581 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.581 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.442482+0000", 2024-05-15T12:51:53.581 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.061934706999999999 2024-05-15T12:51:53.581 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.581 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.581 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:53.581 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.442541+0000", 2024-05-15T12:51:53.581 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9302000000000002e-05 2024-05-15T12:51:53.582 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.582 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.582 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:53.582 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.442596+0000", 2024-05-15T12:51:53.582 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4438999999999998e-05 2024-05-15T12:51:53.582 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.582 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:53.582 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.582 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.582 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.582 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:55804 2.c 2:3dacca5d:::benchmark_data_smithi177_78104_object55803:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-05-15T12:51:53.582 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:48:45.954056+0000", 2024-05-15T12:51:53.583 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 187.539535387, 2024-05-15T12:51:53.583 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.080491396000000007, 2024-05-15T12:51:53.583 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:53.583 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:53.583 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:53.583 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:53.583 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:53.583 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 55804 2024-05-15T12:51:53.583 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.583 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:53.583 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.584 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:53.584 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.954056+0000", 2024-05-15T12:51:53.584 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.584 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.584 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.584 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:53.584 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.954056+0000", 2024-05-15T12:51:53.584 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.584 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.584 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.584 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:53.584 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.954057+0000", 2024-05-15T12:51:53.585 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1829999999999999e-06 2024-05-15T12:51:53.585 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.585 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.585 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:53.585 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.954077+0000", 2024-05-15T12:51:53.585 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0339999999999998e-05 2024-05-15T12:51:53.585 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.585 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.585 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:53.585 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.954090+0000", 2024-05-15T12:51:53.588 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.308e-05 2024-05-15T12:51:53.588 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.588 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.588 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:53.588 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.954098+0000", 2024-05-15T12:51:53.588 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5639999999999999e-06 2024-05-15T12:51:53.588 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.588 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.588 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:53.588 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.954143+0000", 2024-05-15T12:51:53.588 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5080000000000002e-05 2024-05-15T12:51:53.589 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.589 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.589 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:53.589 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.954225+0000", 2024-05-15T12:51:53.589 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2348000000000005e-05 2024-05-15T12:51:53.589 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.589 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.589 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T12:51:53.589 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.954310+0000", 2024-05-15T12:51:53.589 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4596000000000001e-05 2024-05-15T12:51:53.589 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.589 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.589 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:53.590 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.957311+0000", 2024-05-15T12:51:53.590 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003001231 2024-05-15T12:51:53.590 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.590 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.590 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.590 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.957371+0000", 2024-05-15T12:51:53.590 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0232999999999997e-05 2024-05-15T12:51:53.590 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.590 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.590 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.590 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:46.034474+0000", 2024-05-15T12:51:53.590 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.077102466999999994 2024-05-15T12:51:53.590 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.591 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.591 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:53.591 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:46.034514+0000", 2024-05-15T12:51:53.591 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0271000000000001e-05 2024-05-15T12:51:53.591 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.591 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.591 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:53.591 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:46.034547+0000", 2024-05-15T12:51:53.591 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3003000000000002e-05 2024-05-15T12:51:53.591 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.591 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:53.591 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.591 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.591 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.592 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:55805 2.c 2:333e4218:::benchmark_data_smithi177_78104_object55804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-05-15T12:51:53.592 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:48:45.954861+0000", 2024-05-15T12:51:53.592 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 187.53873028999999, 2024-05-15T12:51:53.592 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.087714845, 2024-05-15T12:51:53.592 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:53.592 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:53.592 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:53.592 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:53.592 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:53.592 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 55805 2024-05-15T12:51:53.592 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.592 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:53.592 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.593 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:53.593 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.954861+0000", 2024-05-15T12:51:53.593 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.593 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.593 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.593 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:53.593 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.954861+0000", 2024-05-15T12:51:53.593 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.593 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.593 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.593 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:53.593 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.954862+0000", 2024-05-15T12:51:53.593 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8100000000000002e-07 2024-05-15T12:51:53.593 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.594 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.594 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:53.594 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.954882+0000", 2024-05-15T12:51:53.594 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0193000000000001e-05 2024-05-15T12:51:53.594 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.594 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.594 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:53.594 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.954894+0000", 2024-05-15T12:51:53.594 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1994000000000001e-05 2024-05-15T12:51:53.594 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.594 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.594 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:53.594 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.954901+0000", 2024-05-15T12:51:53.595 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3459999999999996e-06 2024-05-15T12:51:53.595 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.595 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.595 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:53.595 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.955765+0000", 2024-05-15T12:51:53.595 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00086348299999999996 2024-05-15T12:51:53.595 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.595 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.595 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:53.595 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.955873+0000", 2024-05-15T12:51:53.595 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108417 2024-05-15T12:51:53.595 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.595 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.595 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T12:51:53.596 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.955956+0000", 2024-05-15T12:51:53.596 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3184999999999997e-05 2024-05-15T12:51:53.596 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.596 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.596 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.596 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.957423+0000", 2024-05-15T12:51:53.596 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014669139999999999 2024-05-15T12:51:53.596 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.596 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.596 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:53.596 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:45.957437+0000", 2024-05-15T12:51:53.596 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3339000000000001e-05 2024-05-15T12:51:53.596 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.596 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.597 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.597 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:46.042475+0000", 2024-05-15T12:51:53.597 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.085038458999999997 2024-05-15T12:51:53.597 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.597 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.597 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:53.597 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:46.042518+0000", 2024-05-15T12:51:53.597 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2599000000000003e-05 2024-05-15T12:51:53.597 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.597 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.597 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:53.597 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:46.042576+0000", 2024-05-15T12:51:53.597 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8134999999999998e-05 2024-05-15T12:51:53.598 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.598 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:53.598 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.598 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.598 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.598 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:58514 2.c 2:32061fa1:::benchmark_data_smithi177_78104_object58513:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-15T12:51:53.598 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:48:48.372260+0000", 2024-05-15T12:51:53.598 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 185.121330946, 2024-05-15T12:51:53.598 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.099972444999999993, 2024-05-15T12:51:53.598 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:53.598 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:53.598 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:53.598 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:53.598 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:53.599 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 58514 2024-05-15T12:51:53.599 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.599 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:53.599 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.599 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:53.599 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:48.372260+0000", 2024-05-15T12:51:53.599 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.599 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.599 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.599 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:53.599 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:48.372260+0000", 2024-05-15T12:51:53.599 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.599 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.599 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.600 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:53.600 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:48.372262+0000", 2024-05-15T12:51:53.600 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.655e-06 2024-05-15T12:51:53.600 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.600 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.600 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:53.600 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:48.372287+0000", 2024-05-15T12:51:53.600 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4698e-05 2024-05-15T12:51:53.600 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.600 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.600 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:53.600 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:48.372295+0000", 2024-05-15T12:51:53.600 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2910000000000007e-06 2024-05-15T12:51:53.601 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.601 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.601 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:53.601 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:48.372300+0000", 2024-05-15T12:51:53.601 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4949999999999999e-06 2024-05-15T12:51:53.601 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.601 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.601 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:53.601 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:48.372341+0000", 2024-05-15T12:51:53.601 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0459e-05 2024-05-15T12:51:53.601 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.601 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.601 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:53.601 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:48.372442+0000", 2024-05-15T12:51:53.602 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000101162 2024-05-15T12:51:53.602 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.602 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.602 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T12:51:53.602 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:48.372525+0000", 2024-05-15T12:51:53.602 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2546000000000006e-05 2024-05-15T12:51:53.602 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.602 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.602 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:53.602 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:48.374099+0000", 2024-05-15T12:51:53.602 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001574949 2024-05-15T12:51:53.602 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.602 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.603 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.603 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:48.374648+0000", 2024-05-15T12:51:53.603 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00054900599999999999 2024-05-15T12:51:53.603 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.603 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.603 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.603 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:48.472125+0000", 2024-05-15T12:51:53.603 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.097476549999999995 2024-05-15T12:51:53.603 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.603 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.603 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:53.603 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:48.472181+0000", 2024-05-15T12:51:53.603 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6458000000000001e-05 2024-05-15T12:51:53.603 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.604 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.604 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:53.604 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:48.472233+0000", 2024-05-15T12:51:53.604 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1175999999999999e-05 2024-05-15T12:51:53.604 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.604 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:53.604 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.604 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.604 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.604 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:60766 2.c 2:357f23c4:::benchmark_data_smithi177_78104_object60765:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-15T12:51:53.604 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:48:49.762721+0000", 2024-05-15T12:51:53.604 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 183.73087018699999, 2024-05-15T12:51:53.604 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.10909801399999999, 2024-05-15T12:51:53.604 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:53.605 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:53.605 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:53.605 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:53.605 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:53.605 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60766 2024-05-15T12:51:53.605 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.605 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:53.605 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.605 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:53.605 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.762721+0000", 2024-05-15T12:51:53.605 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.605 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.605 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.605 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:53.606 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.762721+0000", 2024-05-15T12:51:53.606 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.606 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.606 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.606 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:53.606 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.762722+0000", 2024-05-15T12:51:53.606 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1689999999999999e-06 2024-05-15T12:51:53.606 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.606 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.606 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:53.606 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.762745+0000", 2024-05-15T12:51:53.606 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2674e-05 2024-05-15T12:51:53.606 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.607 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.607 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:53.607 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.762757+0000", 2024-05-15T12:51:53.607 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2323e-05 2024-05-15T12:51:53.607 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.607 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.607 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:53.607 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.762766+0000", 2024-05-15T12:51:53.607 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3699999999999995e-06 2024-05-15T12:51:53.607 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.607 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.607 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:53.607 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.762859+0000", 2024-05-15T12:51:53.607 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3215999999999997e-05 2024-05-15T12:51:53.608 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.608 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.608 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:53.608 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.762969+0000", 2024-05-15T12:51:53.608 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011024 2024-05-15T12:51:53.608 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.608 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.608 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T12:51:53.608 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.763078+0000", 2024-05-15T12:51:53.608 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109319 2024-05-15T12:51:53.608 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.608 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.608 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:53.609 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.765523+0000", 2024-05-15T12:51:53.609 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024449430000000002 2024-05-15T12:51:53.609 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.609 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.609 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.609 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.768086+0000", 2024-05-15T12:51:53.609 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025625930000000002 2024-05-15T12:51:53.609 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.609 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.609 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.609 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.871753+0000", 2024-05-15T12:51:53.609 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.10366738 2024-05-15T12:51:53.609 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.610 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.610 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:53.610 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.871790+0000", 2024-05-15T12:51:53.610 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6718000000000003e-05 2024-05-15T12:51:53.610 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.610 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.610 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:53.610 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.871819+0000", 2024-05-15T12:51:53.610 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9068999999999999e-05 2024-05-15T12:51:53.610 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.610 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:53.610 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.610 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.611 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.611 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:60774 2.c 2:3ffdc7a0:::benchmark_data_smithi177_78104_object60773:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-15T12:51:53.611 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:48:49.765726+0000", 2024-05-15T12:51:53.611 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 183.72786534599999, 2024-05-15T12:51:53.611 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.112588415, 2024-05-15T12:51:53.611 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:53.611 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:53.611 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:53.611 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:53.611 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:53.611 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60774 2024-05-15T12:51:53.611 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.611 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:53.611 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.612 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:53.612 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.765726+0000", 2024-05-15T12:51:53.612 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.612 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.612 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.612 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:53.612 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.765726+0000", 2024-05-15T12:51:53.612 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.612 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.612 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.612 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:53.612 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.765727+0000", 2024-05-15T12:51:53.612 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8599999999999997e-07 2024-05-15T12:51:53.613 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.613 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.613 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:53.613 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.765746+0000", 2024-05-15T12:51:53.613 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9377000000000001e-05 2024-05-15T12:51:53.613 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.613 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.613 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:53.613 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.765755+0000", 2024-05-15T12:51:53.613 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3230000000000004e-06 2024-05-15T12:51:53.613 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.613 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.613 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:53.613 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.765761+0000", 2024-05-15T12:51:53.614 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3070000000000002e-06 2024-05-15T12:51:53.614 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.614 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.614 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:53.614 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.765796+0000", 2024-05-15T12:51:53.614 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5298e-05 2024-05-15T12:51:53.614 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.614 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.614 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:53.614 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.765874+0000", 2024-05-15T12:51:53.614 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8133000000000006e-05 2024-05-15T12:51:53.614 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.615 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.615 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T12:51:53.615 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.765961+0000", 2024-05-15T12:51:53.615 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6954999999999997e-05 2024-05-15T12:51:53.615 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.615 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.615 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.615 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.768448+0000", 2024-05-15T12:51:53.615 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024872269999999998 2024-05-15T12:51:53.615 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.615 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.615 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:53.615 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.768526+0000", 2024-05-15T12:51:53.616 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8076000000000003e-05 2024-05-15T12:51:53.616 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.616 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.616 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.616 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.878221+0000", 2024-05-15T12:51:53.616 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.10969509500000001 2024-05-15T12:51:53.616 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.616 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.616 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:53.616 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.878263+0000", 2024-05-15T12:51:53.616 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2038e-05 2024-05-15T12:51:53.616 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.616 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.616 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:53.617 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.878314+0000", 2024-05-15T12:51:53.617 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0800000000000002e-05 2024-05-15T12:51:53.617 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.617 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:53.617 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.617 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.617 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.617 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:60783 2.c 2:34e37738:::benchmark_data_smithi177_78104_object60782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-15T12:51:53.617 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:48:49.771876+0000", 2024-05-15T12:51:53.617 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 183.72171477099999, 2024-05-15T12:51:53.617 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.10759964499999999, 2024-05-15T12:51:53.617 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:51:53.617 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:51:53.617 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:51:53.618 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:51:53.618 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:51:53.618 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 60783 2024-05-15T12:51:53.618 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.618 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:51:53.618 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.618 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:51:53.618 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.771876+0000", 2024-05-15T12:51:53.618 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.618 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.618 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.618 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:51:53.618 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.771876+0000", 2024-05-15T12:51:53.618 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:51:53.619 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.619 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.619 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:51:53.619 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.771878+0000", 2024-05-15T12:51:53.619 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3680000000000001e-06 2024-05-15T12:51:53.619 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.619 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.619 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:51:53.619 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.771904+0000", 2024-05-15T12:51:53.619 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6211999999999999e-05 2024-05-15T12:51:53.619 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.619 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.619 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:51:53.619 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.771925+0000", 2024-05-15T12:51:53.620 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1409000000000001e-05 2024-05-15T12:51:53.620 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.620 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.620 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:51:53.620 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.771934+0000", 2024-05-15T12:51:53.620 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1429999999999998e-06 2024-05-15T12:51:53.620 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.620 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.620 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:51:53.620 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.771991+0000", 2024-05-15T12:51:53.620 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7499e-05 2024-05-15T12:51:53.620 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.621 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.621 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:51:53.621 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.772131+0000", 2024-05-15T12:51:53.621 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000139837 2024-05-15T12:51:53.621 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.621 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.621 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T12:51:53.621 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.772277+0000", 2024-05-15T12:51:53.621 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014650700000000001 2024-05-15T12:51:53.621 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.621 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.621 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:51:53.621 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.774087+0000", 2024-05-15T12:51:53.621 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001810065 2024-05-15T12:51:53.621 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.622 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.622 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.622 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.774773+0000", 2024-05-15T12:51:53.622 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000685533 2024-05-15T12:51:53.622 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.622 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.622 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:51:53.622 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.879366+0000", 2024-05-15T12:51:53.622 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.104592564 2024-05-15T12:51:53.622 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.622 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.622 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:51:53.622 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.879429+0000", 2024-05-15T12:51:53.622 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3878000000000001e-05 2024-05-15T12:51:53.623 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:51:53.623 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:51:53.623 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:51:53.623 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:48:49.879476+0000", 2024-05-15T12:51:53.623 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6629999999999999e-05 2024-05-15T12:51:53.623 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.623 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:53.623 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.623 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:51:53.623 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:51:53.623 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T12:51:54.201 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-05-15T12:51:54.974 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:54 smithi027 ceph-mon[39795]: pgmap v568: 17 pgs: 17 active+clean; 1.6 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 890 op/s 2024-05-15T12:51:55.186 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:51:55.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:55.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:51:55.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:54 smithi120 ceph-mon[43961]: pgmap v568: 17 pgs: 17 active+clean; 1.6 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 890 op/s 2024-05-15T12:51:55.293 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:51:55.335 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:54 smithi177 ceph-mon[44303]: pgmap v568: 17 pgs: 17 active+clean; 1.6 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 890 op/s 2024-05-15T12:51:55.340 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:55.363 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:51:55.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:55.368 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:51:55.469 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:51:55.553 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:56.014 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.9 to 1 (10000) 2024-05-15T12:51:56.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:55 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3279000048' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T12:51:56.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:55 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3279000048' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T12:51:56.376 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:51:56.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:56.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:51:56.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:55 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3279000048' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T12:51:56.479 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:51:56.566 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:57.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:56 smithi027 ceph-mon[39795]: pgmap v569: 17 pgs: 17 active+clean; 1.5 GiB data, 5.8 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 845 op/s 2024-05-15T12:51:57.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:56 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3279000048' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T12:51:57.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:56 smithi027 ceph-mon[39795]: osdmap e202: 12 total, 12 up, 11 in 2024-05-15T12:51:57.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:57 smithi120 ceph-mon[43961]: pgmap v569: 17 pgs: 17 active+clean; 1.5 GiB data, 5.8 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 845 op/s 2024-05-15T12:51:57.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:57 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3279000048' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T12:51:57.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:57 smithi120 ceph-mon[43961]: osdmap e202: 12 total, 12 up, 11 in 2024-05-15T12:51:57.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:56 smithi177 ceph-mon[44303]: pgmap v569: 17 pgs: 17 active+clean; 1.5 GiB data, 5.8 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 845 op/s 2024-05-15T12:51:57.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:56 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3279000048' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T12:51:57.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:57 smithi177 ceph-mon[44303]: osdmap e202: 12 total, 12 up, 11 in 2024-05-15T12:51:57.583 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:51:57.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:57.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:51:57.692 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:51:57.736 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:57.770 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:51:57.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:57.777 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:51:57.878 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:51:57.914 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:57.942 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:51:57.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:57.947 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:51:57.969 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T12:51:58.001 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:58 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4216258103' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:51:58.001 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:58 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:51:58.001 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:58 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:51:58.001 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:58 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T12:51:58.001 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:58 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:51:58.001 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:58 smithi027 ceph-mon[39795]: osdmap e203: 12 total, 12 up, 11 in 2024-05-15T12:51:58.048 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:51:58.081 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:58.104 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:51:58.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:58.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:51:58.209 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:51:58.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:58 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4216258103' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:51:58.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:58 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:51:58.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:58 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:51:58.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:58 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T12:51:58.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:58 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:51:58.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:58 smithi120 ceph-mon[43961]: osdmap e203: 12 total, 12 up, 11 in 2024-05-15T12:51:58.293 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:58.390 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:51:58.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:58.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:51:58.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:58 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4216258103' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:51:58.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:58 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:51:58.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:58 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:51:58.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:58 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T12:51:58.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:58 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:51:58.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:58 smithi177 ceph-mon[44303]: osdmap e203: 12 total, 12 up, 11 in 2024-05-15T12:51:58.508 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:51:58.555 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:58.583 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:51:58.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:58.587 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:51:58.688 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:51:58.716 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:58.748 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:51:58.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:58.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:51:58.854 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:51:58.888 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:58.915 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:51:58.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:58.919 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:51:59.020 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:51:59.063 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:59.086 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:51:59.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:59.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:51:59.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:59 smithi177 ceph-mon[44303]: pgmap v571: 17 pgs: 17 active+clean; 1.4 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 808 op/s 2024-05-15T12:51:59.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:59 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:51:59.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:51:59 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:51:59.191 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:51:59.226 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:59.247 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:51:59.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:59.250 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:51:59.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:59 smithi027 ceph-mon[39795]: pgmap v571: 17 pgs: 17 active+clean; 1.4 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 808 op/s 2024-05-15T12:51:59.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:59 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:51:59.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:51:59 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:51:59.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:59 smithi120 ceph-mon[43961]: pgmap v571: 17 pgs: 17 active+clean; 1.4 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 808 op/s 2024-05-15T12:51:59.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:59 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:51:59.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:51:59 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:51:59.351 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:51:59.380 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:59.406 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:51:59.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:59.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:51:59.474 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 10 1 2024-05-15T12:51:59.511 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:51:59.538 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:59.560 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:51:59.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:59.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:51:59.666 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:51:59.694 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:59.718 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:51:59.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:59.724 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:51:59.825 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:51:59.862 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:51:59.892 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:51:59.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:51:59.898 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:51:59.999 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:52:00.027 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:00.050 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:52:00.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:00.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:52:00.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:52:00.231 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:00.276 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:52:00.277 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:52:00.277 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:52:00.277 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:52:00.277 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:52:00.277 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:52:00.277 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:52:00.277 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:52:00.277 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:52:00.277 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:52:00.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:00 smithi120 ceph-mon[43961]: osdmap e204: 12 total, 12 up, 11 in 2024-05-15T12:52:00.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:00 smithi120 ceph-mon[43961]: pgmap v574: 17 pgs: 17 active+clean; 1.3 GiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 736 op/s 2024-05-15T12:52:00.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:00 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/928916132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:52:00.293 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:52:00.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:00.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:52:00.300 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:00 smithi027 ceph-mon[39795]: osdmap e204: 12 total, 12 up, 11 in 2024-05-15T12:52:00.300 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:00 smithi027 ceph-mon[39795]: pgmap v574: 17 pgs: 17 active+clean; 1.3 GiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 736 op/s 2024-05-15T12:52:00.300 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:00 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/928916132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:52:00.306 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:52:00.337 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:52:00.380 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:52:00.398 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:52:00.398 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:52:00.398 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:52:00.398 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:52:00.398 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:52:00.398 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:52:00.398 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:52:00.398 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:52:00.399 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:52:00.399 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:52:00.399 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:52:00.399 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:52:00.400 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:52:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:00 smithi177 ceph-mon[44303]: osdmap e204: 12 total, 12 up, 11 in 2024-05-15T12:52:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:00 smithi177 ceph-mon[44303]: pgmap v574: 17 pgs: 17 active+clean; 1.3 GiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 736 op/s 2024-05-15T12:52:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:00 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/928916132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:52:00.473 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:00.544 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:52:00.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:00.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:52:00.650 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:52:00.683 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:00.709 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:52:00.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:00.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:52:00.813 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:52:00.889 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:01.167 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:52:01.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:01.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:52:01.278 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:52:01.311 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:01.342 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:52:01.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:01.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:52:01.445 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:52:01.597 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:01.662 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:52:01.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:01.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:52:01.771 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:52:01.847 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:01 smithi027 ceph-mon[39795]: pgmap v575: 17 pgs: 17 active+clean; 1.2 GiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 846 op/s 2024-05-15T12:52:01.852 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:02.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:01 smithi120 ceph-mon[43961]: pgmap v575: 17 pgs: 17 active+clean; 1.2 GiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 846 op/s 2024-05-15T12:52:02.080 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:52:02.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:02.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:52:02.174 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T12:52:02.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:01 smithi177 ceph-mon[44303]: pgmap v575: 17 pgs: 17 active+clean; 1.2 GiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 846 op/s 2024-05-15T12:52:02.192 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:52:02.284 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:02.777 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:52:02.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:02.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:52:02.814 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:52:02.884 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:52:02.885 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:52:02.914 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:02.936 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:52:02.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:02.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:52:03.041 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:52:03.068 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:03.090 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:52:03.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:03.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:52:03.196 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:52:03.232 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:03.255 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:52:03.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:03.262 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:52:03.362 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:52:03.393 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:03.426 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:52:03.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:03.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:52:03.531 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:52:03.607 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:03.718 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:52:04.025 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:03 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T12:52:04.025 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:03 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T12:52:04.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:03 smithi027 ceph-mon[39795]: osdmap e205: 12 total, 12 up, 11 in 2024-05-15T12:52:04.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:03 smithi027 ceph-mon[39795]: pgmap v577: 17 pgs: 17 active+clean; 1.1 GiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 985 op/s 2024-05-15T12:52:04.195 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:52:04.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:04.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:52:04.256 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T12:52:04.256 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:52:04.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:03 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T12:52:04.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:03 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T12:52:04.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:03 smithi120 ceph-mon[43961]: osdmap e205: 12 total, 12 up, 11 in 2024-05-15T12:52:04.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:03 smithi120 ceph-mon[43961]: pgmap v577: 17 pgs: 17 active+clean; 1.1 GiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 985 op/s 2024-05-15T12:52:04.303 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:52:04.336 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:03 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T12:52:04.336 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:03 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T12:52:04.336 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:03 smithi177 ceph-mon[44303]: osdmap e205: 12 total, 12 up, 11 in 2024-05-15T12:52:04.336 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:03 smithi177 ceph-mon[44303]: pgmap v577: 17 pgs: 17 active+clean; 1.1 GiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 985 op/s 2024-05-15T12:52:04.338 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:04.369 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:52:04.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:04.372 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:52:04.473 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:52:04.541 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:05.603 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:05 smithi027 ceph-mon[39795]: 2.e scrub starts 2024-05-15T12:52:05.603 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:05 smithi027 ceph-mon[39795]: 2.e scrub ok 2024-05-15T12:52:05.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:05 smithi177 ceph-mon[44303]: 2.e scrub starts 2024-05-15T12:52:05.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:05 smithi177 ceph-mon[44303]: 2.e scrub ok 2024-05-15T12:52:06.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:05 smithi120 ceph-mon[43961]: 2.e scrub starts 2024-05-15T12:52:06.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:05 smithi120 ceph-mon[43961]: 2.e scrub ok 2024-05-15T12:52:06.437 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:52:06.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:06.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:52:06.548 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:52:06.584 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:06.612 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:52:06.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:06.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:52:06.718 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:52:06.748 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:06.769 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:52:06.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:06 smithi027 ceph-mon[39795]: pgmap v578: 17 pgs: 17 active+clean; 979 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 832 op/s 2024-05-15T12:52:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:06.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:52:06.873 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:52:06.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:06 smithi177 ceph-mon[44303]: pgmap v578: 17 pgs: 17 active+clean; 979 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 832 op/s 2024-05-15T12:52:06.959 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:07.008 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:52:07.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:07.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:52:07.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:06 smithi120 ceph-mon[43961]: pgmap v578: 17 pgs: 17 active+clean; 979 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 832 op/s 2024-05-15T12:52:07.114 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:52:07.190 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:07.639 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-05-15T12:52:07.877 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:52:07.905 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:52:07.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:07.915 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:07 smithi027 ceph-mon[39795]: pgmap v579: 17 pgs: 17 active+clean; 913 MiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 805 op/s 2024-05-15T12:52:07.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:52:08.017 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:52:08.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:07 smithi120 ceph-mon[43961]: pgmap v579: 17 pgs: 17 active+clean; 913 MiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 805 op/s 2024-05-15T12:52:08.055 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:07 smithi177 ceph-mon[44303]: pgmap v579: 17 pgs: 17 active+clean; 913 MiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 805 op/s 2024-05-15T12:52:08.056 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:08.081 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:52:08.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:08.085 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:52:08.186 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:52:08.215 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:08.244 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:52:08.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:08.248 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:52:08.349 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:52:08.383 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:08.405 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:52:08.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:08.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:52:08.512 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:52:08.565 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:08.595 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:52:08.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:08.598 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:52:08.699 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:52:08.738 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:08.762 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:52:08.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:08.765 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:52:08.866 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:52:08.993 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.10 to 1 (10000) 2024-05-15T12:52:08.995 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:09.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:09 smithi027 ceph-mon[39795]: 2.4 scrub starts 2024-05-15T12:52:09.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:09 smithi027 ceph-mon[39795]: 2.4 scrub ok 2024-05-15T12:52:09.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:09 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2736321045' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T12:52:09.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:09 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T12:52:09.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:09 smithi027 ceph-mon[39795]: 2.d scrub starts 2024-05-15T12:52:09.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:09 smithi120 ceph-mon[43961]: 2.4 scrub starts 2024-05-15T12:52:09.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:09 smithi120 ceph-mon[43961]: 2.4 scrub ok 2024-05-15T12:52:09.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:09 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2736321045' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T12:52:09.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:09 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T12:52:09.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:09 smithi120 ceph-mon[43961]: 2.d scrub starts 2024-05-15T12:52:09.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:09 smithi177 ceph-mon[44303]: 2.4 scrub starts 2024-05-15T12:52:09.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:09 smithi177 ceph-mon[44303]: 2.4 scrub ok 2024-05-15T12:52:09.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:09 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2736321045' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T12:52:09.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:09 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T12:52:09.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:09 smithi177 ceph-mon[44303]: 2.d scrub starts 2024-05-15T12:52:10.026 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T12:52:10.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:10 smithi027 ceph-mon[39795]: 2.6 scrub starts 2024-05-15T12:52:10.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:10 smithi027 ceph-mon[39795]: 2.6 scrub ok 2024-05-15T12:52:10.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:10 smithi027 ceph-mon[39795]: 2.d scrub ok 2024-05-15T12:52:10.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:10 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T12:52:10.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:10 smithi027 ceph-mon[39795]: osdmap e206: 12 total, 12 up, 11 in 2024-05-15T12:52:10.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:10 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/782141233' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:52:10.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:10 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:52:10.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:10 smithi027 ceph-mon[39795]: pgmap v581: 17 pgs: 17 active+clean; 809 MiB data, 3.7 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 777 op/s 2024-05-15T12:52:10.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:10 smithi120 ceph-mon[43961]: 2.6 scrub starts 2024-05-15T12:52:10.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:10 smithi120 ceph-mon[43961]: 2.6 scrub ok 2024-05-15T12:52:10.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:10 smithi120 ceph-mon[43961]: 2.d scrub ok 2024-05-15T12:52:10.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:10 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T12:52:10.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:10 smithi120 ceph-mon[43961]: osdmap e206: 12 total, 12 up, 11 in 2024-05-15T12:52:10.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:10 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/782141233' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:52:10.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:10 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:52:10.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:10 smithi120 ceph-mon[43961]: pgmap v581: 17 pgs: 17 active+clean; 809 MiB data, 3.7 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 777 op/s 2024-05-15T12:52:10.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:10 smithi177 ceph-mon[44303]: 2.6 scrub starts 2024-05-15T12:52:10.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:10 smithi177 ceph-mon[44303]: 2.6 scrub ok 2024-05-15T12:52:10.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:10 smithi177 ceph-mon[44303]: 2.d scrub ok 2024-05-15T12:52:10.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:10 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T12:52:10.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:10 smithi177 ceph-mon[44303]: osdmap e206: 12 total, 12 up, 11 in 2024-05-15T12:52:10.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:10 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/782141233' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:52:10.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:10 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:52:10.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:10 smithi177 ceph-mon[44303]: pgmap v581: 17 pgs: 17 active+clean; 809 MiB data, 3.7 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 777 op/s 2024-05-15T12:52:10.487 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:52:10.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:10.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:52:10.598 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:52:10.647 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:10.676 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:52:10.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:10.680 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:52:10.781 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:52:10.816 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:10.841 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:52:10.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:10.846 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:52:10.947 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:52:10.979 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:11.013 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:52:11.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:11.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:52:11.121 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:52:11.179 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:11 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:52:11.179 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:11 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:52:11.180 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:11 smithi027 ceph-mon[39795]: osdmap e207: 12 total, 12 up, 11 in 2024-05-15T12:52:11.180 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:11.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:11 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:52:11.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:11 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:52:11.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:11 smithi120 ceph-mon[43961]: osdmap e207: 12 total, 12 up, 11 in 2024-05-15T12:52:11.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:11 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:52:11.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:11 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:52:11.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:11 smithi177 ceph-mon[44303]: osdmap e207: 12 total, 12 up, 11 in 2024-05-15T12:52:11.502 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-05-15T12:52:11.502 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T12:52:11.567 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:52:11.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:11.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:52:11.679 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:52:11.710 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:11.734 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:52:11.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:11.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:52:11.838 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:52:11.902 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:12.243 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:12 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:52:12.243 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:12 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:52:12.243 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:12 smithi027 ceph-mon[39795]: osdmap e208: 12 total, 12 up, 11 in 2024-05-15T12:52:12.243 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:12 smithi027 ceph-mon[39795]: pgmap v584: 17 pgs: 17 active+clean; 662 MiB data, 3.3 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 845 op/s 2024-05-15T12:52:12.246 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:52:12.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:12 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:52:12.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:12 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:52:12.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:12 smithi120 ceph-mon[43961]: osdmap e208: 12 total, 12 up, 11 in 2024-05-15T12:52:12.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:12 smithi120 ceph-mon[43961]: pgmap v584: 17 pgs: 17 active+clean; 662 MiB data, 3.3 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 845 op/s 2024-05-15T12:52:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:52:12.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:12.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:52:12.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:12 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:52:12.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:12 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:52:12.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:12 smithi177 ceph-mon[44303]: osdmap e208: 12 total, 12 up, 11 in 2024-05-15T12:52:12.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:12 smithi177 ceph-mon[44303]: pgmap v584: 17 pgs: 17 active+clean; 662 MiB data, 3.3 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 845 op/s 2024-05-15T12:52:12.508 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:52:12.570 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:12.665 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:52:12.724 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:52:12.765 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:52:12.811 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:52:12.841 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:52:12.860 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:52:12.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:12.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:52:12.880 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:52:12.934 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:52:12.969 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:52:12.978 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:52:13.011 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:13.025 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:52:13.033 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:52:13.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:13.037 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:52:13.083 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:52:13.124 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:52:13.138 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:52:13.166 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:13 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1140114355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:52:13.177 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:52:13.186 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:52:13.186 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:52:13.186 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:52:13.186 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:52:13.186 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:52:13.186 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:52:13.186 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:52:13.187 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:52:13.187 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:52:13.187 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:52:13.187 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:52:13.187 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:52:13.241 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:13.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:13 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1140114355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:52:13.484 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:52:13.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:13.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:52:13.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:13 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1140114355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:52:13.588 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:52:13.632 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:13.838 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:52:13.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:13.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:52:13.945 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:52:13.980 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:14.011 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:52:14.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:14.018 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:52:14.119 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:52:14.166 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:14.196 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:52:14.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:14.203 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:52:14.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:14 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:52:14.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:14 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:52:14.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:14 smithi027 ceph-mon[39795]: pgmap v585: 17 pgs: 17 active+clean; 546 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 979 op/s 2024-05-15T12:52:14.292 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T12:52:14.304 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:52:14.333 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:14.339 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:14 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:52:14.339 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:14 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:52:14.339 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:14 smithi177 ceph-mon[44303]: pgmap v585: 17 pgs: 17 active+clean; 546 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 979 op/s 2024-05-15T12:52:14.358 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:52:14.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:14.361 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:52:14.462 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:52:14.494 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:14.525 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:52:14.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:14.529 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:52:14.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:14 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:52:14.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:14 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:52:14.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:14 smithi120 ceph-mon[43961]: pgmap v585: 17 pgs: 17 active+clean; 546 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 979 op/s 2024-05-15T12:52:14.629 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:52:14.723 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:14.993 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:52:15.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:15.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:52:15.104 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:52:15.196 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:15.257 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:52:15.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:15.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:52:15.263 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:15 smithi027 ceph-mon[39795]: osdmap e209: 12 total, 12 up, 11 in 2024-05-15T12:52:15.361 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:52:15.388 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:15 smithi177 ceph-mon[44303]: osdmap e209: 12 total, 12 up, 11 in 2024-05-15T12:52:15.390 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:15.436 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:52:15.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:15.444 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:52:15.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:15 smithi120 ceph-mon[43961]: osdmap e209: 12 total, 12 up, 11 in 2024-05-15T12:52:15.545 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:52:15.582 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:15.608 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:52:15.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:15.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:52:15.712 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:52:15.736 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:52:15.736 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T12:52:15.736 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T12:52:15.736 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-05-15T12:52:15.736 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-05-15T12:52:15.737 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T12:52:15.791 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:15.811 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:52:16.089 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:16 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:52:16.089 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:16 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:52:16.089 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:16 smithi027 ceph-mon[39795]: osdmap e210: 12 total, 12 up, 11 in 2024-05-15T12:52:16.089 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:16 smithi027 ceph-mon[39795]: pgmap v588: 17 pgs: 17 active+clean; 444 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s 2024-05-15T12:52:16.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:16 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:52:16.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:16 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:52:16.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:16 smithi177 ceph-mon[44303]: osdmap e210: 12 total, 12 up, 11 in 2024-05-15T12:52:16.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:16 smithi177 ceph-mon[44303]: pgmap v588: 17 pgs: 17 active+clean; 444 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s 2024-05-15T12:52:16.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:16 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:52:16.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:16 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:52:16.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:16 smithi120 ceph-mon[43961]: osdmap e210: 12 total, 12 up, 11 in 2024-05-15T12:52:16.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:16 smithi120 ceph-mon[43961]: pgmap v588: 17 pgs: 17 active+clean; 444 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s 2024-05-15T12:52:17.044 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:52:18.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:17 smithi027 ceph-mon[39795]: pgmap v589: 17 pgs: 17 active+clean; 377 MiB data, 2.4 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 690 op/s 2024-05-15T12:52:18.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:17 smithi120 ceph-mon[43961]: pgmap v589: 17 pgs: 17 active+clean; 377 MiB data, 2.4 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 690 op/s 2024-05-15T12:52:18.183 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:17 smithi177 ceph-mon[44303]: pgmap v589: 17 pgs: 17 active+clean; 377 MiB data, 2.4 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 690 op/s 2024-05-15T12:52:18.614 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:52:18.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:18.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:52:18.730 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:52:18.760 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:18.784 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:52:18.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:18.788 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:52:18.889 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:52:18.925 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:18.954 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:52:18.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:18.958 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:52:19.059 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:52:19.100 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:19.135 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:52:19.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:19.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:52:19.241 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:52:19.279 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:19.301 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:52:19.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:19.306 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:52:19.407 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:52:19.440 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:19.465 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:52:19.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:19.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:52:19.572 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:52:19.631 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:19.780 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:52:19.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:19.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:52:19.885 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:52:19.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:19 smithi027 ceph-mon[39795]: pgmap v590: 17 pgs: 17 active+clean; 277 MiB data, 2.1 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 770 op/s 2024-05-15T12:52:19.957 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:20.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:19 smithi120 ceph-mon[43961]: pgmap v590: 17 pgs: 17 active+clean; 277 MiB data, 2.1 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 770 op/s 2024-05-15T12:52:20.168 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:52:20.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:20.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:52:20.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:19 smithi177 ceph-mon[44303]: pgmap v590: 17 pgs: 17 active+clean; 277 MiB data, 2.1 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 770 op/s 2024-05-15T12:52:20.275 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:52:20.351 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:20.388 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:52:20.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:20.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:52:20.510 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:52:20.541 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:20.573 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:52:20.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:20.579 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:52:20.680 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:52:20.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:20 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:52:20.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:20 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:52:20.733 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:20.759 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:52:20.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:20.766 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:52:20.867 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:52:20.894 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:20.924 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:52:20.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:20.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:52:20.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:20 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:52:20.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:20 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:52:20.965 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-05-15T12:52:21.030 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:52:21.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:20 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:52:21.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:20 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:52:21.061 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:21.089 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:52:21.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:21.092 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:52:21.193 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:52:21.232 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:21.259 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:52:21.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:21.262 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:52:21.363 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:52:21.408 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:21.440 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:52:21.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:21.443 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:52:21.544 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:52:21.622 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:21.653 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Removed 107130 objects 2024-05-15T12:52:21.654 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Clean up completed and total clean up time :160.868 2024-05-15T12:52:21.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:21 smithi027 ceph-mon[39795]: osdmap e211: 12 total, 12 up, 11 in 2024-05-15T12:52:21.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:21 smithi027 ceph-mon[39795]: pgmap v592: 17 pgs: 17 active+clean; 124 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 890 op/s 2024-05-15T12:52:22.026 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:52:22.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:21 smithi120 ceph-mon[43961]: osdmap e211: 12 total, 12 up, 11 in 2024-05-15T12:52:22.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:21 smithi120 ceph-mon[43961]: pgmap v592: 17 pgs: 17 active+clean; 124 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 890 op/s 2024-05-15T12:52:22.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:21 smithi177 ceph-mon[44303]: osdmap e211: 12 total, 12 up, 11 in 2024-05-15T12:52:22.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:21 smithi177 ceph-mon[44303]: pgmap v592: 17 pgs: 17 active+clean; 124 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 890 op/s 2024-05-15T12:52:22.381 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:52:22.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:22.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:52:22.485 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:52:22.517 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:22.538 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:52:22.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:22.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:52:22.642 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:52:22.673 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:23.824 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:52:23.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:23.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:52:23.929 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:52:23.957 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:23.976 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:52:23.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:23.980 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:52:23.995 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:52:23.997 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T12:52:24.081 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:52:24.107 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:24.125 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:52:24.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:24.128 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:52:24.184 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:52:24.184 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:23 smithi027 ceph-mon[39795]: pgmap v593: 17 pgs: 17 active+clean; 35 MiB data, 1.4 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 954 op/s 2024-05-15T12:52:24.229 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:52:24.253 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:52:24.253 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:23 smithi177 ceph-mon[44303]: pgmap v593: 17 pgs: 17 active+clean; 35 MiB data, 1.4 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 954 op/s 2024-05-15T12:52:24.255 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:24.274 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:52:24.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:24.277 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:52:24.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:52:24.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:23 smithi120 ceph-mon[43961]: pgmap v593: 17 pgs: 17 active+clean; 35 MiB data, 1.4 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 954 op/s 2024-05-15T12:52:24.379 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:52:24.411 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:24.432 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:52:24.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:24.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:52:24.536 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:52:24.565 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:24.585 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:52:24.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:24.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:52:24.689 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:52:24.716 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:24.734 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:52:24.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:24.738 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:52:24.839 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:52:24.871 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:24.970 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:52:24.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:24.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:52:24.976 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T12:52:25.000 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":594,"stamp":"2024-05-15T12:52:23.974213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":36569656,"num_objects":552,"num_object_clones":0,"num_object_copies":1656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":552,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":213893,"num_write_kb":6859259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38976,"num_bytes_recovered":2556718272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":52,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":1496116,"kb_used_data":287956,"kb_used_omap":17,"kb_used_meta":1207726,"kb_avail":1029700556,"statfs":{"total":1055945392128,"available":1054413369344,"internally_reserved":0,"allocated":294866944,"data_stored":291956123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":1236712309},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,3,5,9,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-429129728,"num_objects":-6548,"num_object_clones":0,"num_object_copies":-19644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6548,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.340023"},"pg_stats":[{"pgid":"2.f","version":"210'13279","reported_seq":13587,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-15T12:52:20.202635+0000","last_change":"2024-05-15T12:51:36.662714+0000","last_active":"2024-05-15T12:52:20.202635+0000","last_peered":"2024-05-15T12:52:20.202635+0000","last_clean":"2024-05-15T12:52:20.202635+0000","last_became_active":"2024-05-15T12:47:58.401685+0000","last_became_peered":"2024-05-15T12:47:58.401685+0000","last_unstale":"2024-05-15T12:52:20.202635+0000","last_undegraded":"2024-05-15T12:52:20.202635+0000","last_fullsized":"2024-05-15T12:52:20.202635+0000","mapping_epoch":110,"log_start":"124'3200","ondisk_log_start":"124'3200","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'10955","last_scrub_stamp":"2024-05-15T12:51:36.662404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:51:36.662404+0000","objects_scrubbed":2415,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:53:04.071424+0000","scrub_duration":536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4784128,"num_objects":73,"num_object_clones":0,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13279,"num_write_kb":427264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"211'13276","reported_seq":28013,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-05-15T12:52:20.910783+0000","last_change":"2024-05-15T12:52:07.535000+0000","last_active":"2024-05-15T12:52:20.910783+0000","last_peered":"2024-05-15T12:52:20.910783+0000","last_clean":"2024-05-15T12:52:20.910783+0000","last_became_active":"2024-05-15T12:50:45.374644+0000","last_became_peered":"2024-05-15T12:50:45.374644+0000","last_unstale":"2024-05-15T12:52:20.910783+0000","last_undegraded":"2024-05-15T12:52:20.910783+0000","last_fullsized":"2024-05-15T12:52:20.910783+0000","mapping_epoch":173,"log_start":"124'3200","ondisk_log_start":"124'3200","created":110,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12553","last_scrub_stamp":"2024-05-15T12:52:07.534725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:52:07.534725+0000","objects_scrubbed":774,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:53:18.408005+0000","scrub_duration":355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2752512,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13276,"num_write_kb":426176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7219,"num_bytes_recovered":473104384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"211'13254","reported_seq":13558,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-05-15T12:52:21.608549+0000","last_change":"2024-05-15T12:51:44.611596+0000","last_active":"2024-05-15T12:52:21.608549+0000","last_peered":"2024-05-15T12:52:21.608549+0000","last_clean":"2024-05-15T12:52:21.608549+0000","last_became_active":"2024-05-15T12:47:58.401043+0000","last_became_peered":"2024-05-15T12:47:58.401043+0000","last_unstale":"2024-05-15T12:52:21.608549+0000","last_undegraded":"2024-05-15T12:52:21.608549+0000","last_fullsized":"2024-05-15T12:52:21.608549+0000","mapping_epoch":110,"log_start":"125'3200","ondisk_log_start":"125'3200","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'11200","last_scrub_stamp":"2024-05-15T12:51:44.611436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:51:44.611436+0000","objects_scrubbed":2066,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:53:14.472861+0000","scrub_duration":459,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13254,"num_write_kb":424128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"211'13414","reported_seq":13743,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-05-15T12:52:21.636548+0000","last_change":"2024-05-15T12:51:03.134479+0000","last_active":"2024-05-15T12:52:21.636548+0000","last_peered":"2024-05-15T12:52:21.636548+0000","last_clean":"2024-05-15T12:52:21.636548+0000","last_became_active":"2024-05-15T12:47:59.089659+0000","last_became_peered":"2024-05-15T12:47:59.089659+0000","last_unstale":"2024-05-15T12:52:21.636548+0000","last_undegraded":"2024-05-15T12:52:21.636548+0000","last_fullsized":"2024-05-15T12:52:21.636548+0000","mapping_epoch":110,"log_start":"127'3400","ondisk_log_start":"127'3400","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'9994","last_scrub_stamp":"2024-05-15T12:51:03.134217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:51:03.134217+0000","objects_scrubbed":3456,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1384,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13414,"num_write_kb":429248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"211'13268","reported_seq":13577,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-05-15T12:52:21.656671+0000","last_change":"2024-05-15T12:51:36.698048+0000","last_active":"2024-05-15T12:52:21.656671+0000","last_peered":"2024-05-15T12:52:21.656671+0000","last_clean":"2024-05-15T12:52:21.656671+0000","last_became_active":"2024-05-15T12:47:58.401049+0000","last_became_peered":"2024-05-15T12:47:58.401049+0000","last_unstale":"2024-05-15T12:52:21.656671+0000","last_undegraded":"2024-05-15T12:52:21.656671+0000","last_fullsized":"2024-05-15T12:52:21.656671+0000","mapping_epoch":110,"log_start":"124'3200","ondisk_log_start":"124'3200","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'10931","last_scrub_stamp":"2024-05-15T12:51:36.697819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:51:36.697819+0000","objects_scrubbed":2348,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:53:00.881405+0000","scrub_duration":526,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13268,"num_write_kb":424576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":627,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-15T12:52:15.079872+0000","last_change":"2024-05-15T12:52:02.886508+0000","last_active":"2024-05-15T12:52:15.079872+0000","last_peered":"2024-05-15T12:52:15.079872+0000","last_clean":"2024-05-15T12:52:15.079872+0000","last_became_active":"2024-05-15T12:49:08.168334+0000","last_became_peered":"2024-05-15T12:49:08.168334+0000","last_unstale":"2024-05-15T12:52:15.079872+0000","last_undegraded":"2024-05-15T12:52:15.079872+0000","last_fullsized":"2024-05-15T12:52:15.079872+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T12:52:02.886389+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T12:52:02.886389+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:53:20.689368+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"210'13464","reported_seq":34044,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-15T12:52:20.505385+0000","last_change":"2024-05-15T12:51:38.513245+0000","last_active":"2024-05-15T12:52:20.505385+0000","last_peered":"2024-05-15T12:52:20.505385+0000","last_clean":"2024-05-15T12:52:20.505385+0000","last_became_active":"2024-05-15T12:51:17.929582+0000","last_became_peered":"2024-05-15T12:51:17.929582+0000","last_unstale":"2024-05-15T12:52:20.505385+0000","last_undegraded":"2024-05-15T12:52:20.505385+0000","last_fullsized":"2024-05-15T12:52:20.505385+0000","mapping_epoch":190,"log_start":"127'3400","ondisk_log_start":"127'3400","created":110,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'11129","last_scrub_stamp":"2024-05-15T12:51:38.512996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:51:38.512996+0000","objects_scrubbed":2417,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:53:07.028342+0000","scrub_duration":676,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3670016,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13464,"num_write_kb":432640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10151,"num_bytes_recovered":665255936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"210'13339","reported_seq":13644,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-15T12:52:20.143285+0000","last_change":"2024-05-15T12:51:41.798194+0000","last_active":"2024-05-15T12:52:20.143285+0000","last_peered":"2024-05-15T12:52:20.143285+0000","last_clean":"2024-05-15T12:52:20.143285+0000","last_became_active":"2024-05-15T12:47:59.089807+0000","last_became_peered":"2024-05-15T12:47:59.089807+0000","last_unstale":"2024-05-15T12:52:20.143285+0000","last_undegraded":"2024-05-15T12:52:20.143285+0000","last_fullsized":"2024-05-15T12:52:20.143285+0000","mapping_epoch":110,"log_start":"125'3300","ondisk_log_start":"125'3300","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'11284","last_scrub_stamp":"2024-05-15T12:51:41.797972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:51:41.797972+0000","objects_scrubbed":2157,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:53:05.487584+0000","scrub_duration":579,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5308416,"num_objects":81,"num_object_clones":0,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13339,"num_write_kb":429440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"211'13263","reported_seq":13538,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-05-15T12:52:20.912354+0000","last_change":"2024-05-15T12:52:08.430907+0000","last_active":"2024-05-15T12:52:20.912354+0000","last_peered":"2024-05-15T12:52:20.912354+0000","last_clean":"2024-05-15T12:52:20.912354+0000","last_became_active":"2024-05-15T12:47:58.444441+0000","last_became_peered":"2024-05-15T12:47:58.444441+0000","last_unstale":"2024-05-15T12:52:20.912354+0000","last_undegraded":"2024-05-15T12:52:20.912354+0000","last_fullsized":"2024-05-15T12:52:20.912354+0000","mapping_epoch":110,"log_start":"124'3200","ondisk_log_start":"124'3200","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12619","last_scrub_stamp":"2024-05-15T12:52:08.430641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:52:08.430641+0000","objects_scrubbed":686,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:53:17.854752+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2424832,"num_objects":37,"num_object_clones":0,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13263,"num_write_kb":425600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"211'13354","reported_seq":26660,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-05-15T12:52:21.644717+0000","last_change":"2024-05-15T12:50:52.684827+0000","last_active":"2024-05-15T12:52:21.644717+0000","last_peered":"2024-05-15T12:52:21.644717+0000","last_clean":"2024-05-15T12:52:21.644717+0000","last_became_active":"2024-05-15T12:50:48.402873+0000","last_became_peered":"2024-05-15T12:50:48.402873+0000","last_unstale":"2024-05-15T12:52:21.644717+0000","last_undegraded":"2024-05-15T12:52:21.644717+0000","last_fullsized":"2024-05-15T12:52:21.644717+0000","mapping_epoch":176,"log_start":"125'3300","ondisk_log_start":"125'3300","created":110,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'9459","last_scrub_stamp":"2024-05-15T12:50:52.684491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:50:52.684491+0000","objects_scrubbed":3922,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":890,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13354,"num_write_kb":427328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6508,"num_bytes_recovered":426508288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"210'13323","reported_seq":13605,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-15T12:52:20.541472+0000","last_change":"2024-05-15T12:51:39.397895+0000","last_active":"2024-05-15T12:52:20.541472+0000","last_peered":"2024-05-15T12:52:20.541472+0000","last_clean":"2024-05-15T12:52:20.541472+0000","last_became_active":"2024-05-15T12:47:59.089595+0000","last_became_peered":"2024-05-15T12:47:59.089595+0000","last_unstale":"2024-05-15T12:52:20.541472+0000","last_undegraded":"2024-05-15T12:52:20.541472+0000","last_fullsized":"2024-05-15T12:52:20.541472+0000","mapping_epoch":110,"log_start":"124'3300","ondisk_log_start":"124'3300","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'11159","last_scrub_stamp":"2024-05-15T12:51:39.397762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:51:39.397762+0000","objects_scrubbed":2232,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:52:57.578486+0000","scrub_duration":517,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3735552,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13323,"num_write_kb":428160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"211'13351","reported_seq":13613,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-05-15T12:52:20.864643+0000","last_change":"2024-05-15T12:50:51.797101+0000","last_active":"2024-05-15T12:52:20.864643+0000","last_peered":"2024-05-15T12:52:20.864643+0000","last_clean":"2024-05-15T12:52:20.864643+0000","last_became_active":"2024-05-15T12:47:58.445362+0000","last_became_peered":"2024-05-15T12:47:58.445362+0000","last_unstale":"2024-05-15T12:52:20.864643+0000","last_undegraded":"2024-05-15T12:52:20.864643+0000","last_fullsized":"2024-05-15T12:52:20.864643+0000","mapping_epoch":110,"log_start":"125'3300","ondisk_log_start":"125'3300","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'9378","last_scrub_stamp":"2024-05-15T12:50:51.796727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:50:51.796727+0000","objects_scrubbed":4041,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:52:21.109006+0000","scrub_duration":1627,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2424832,"num_objects":37,"num_object_clones":0,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13351,"num_write_kb":428416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"211'13324","reported_seq":23756,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-05-15T12:52:21.597654+0000","last_change":"2024-05-15T12:50:52.144944+0000","last_active":"2024-05-15T12:52:21.597654+0000","last_peered":"2024-05-15T12:52:21.597654+0000","last_clean":"2024-05-15T12:52:21.597654+0000","last_became_active":"2024-05-15T12:50:11.967389+0000","last_became_peered":"2024-05-15T12:50:11.967389+0000","last_unstale":"2024-05-15T12:52:21.597654+0000","last_undegraded":"2024-05-15T12:52:21.597654+0000","last_fullsized":"2024-05-15T12:52:21.597654+0000","mapping_epoch":160,"log_start":"125'3300","ondisk_log_start":"125'3300","created":110,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'9401","last_scrub_stamp":"2024-05-15T12:50:52.144769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:50:52.144769+0000","objects_scrubbed":3950,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:52:23.798102+0000","scrub_duration":1350,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13324,"num_write_kb":426368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5072,"num_bytes_recovered":332398592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,8],"acting":[3,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"210'13345","reported_seq":23820,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-15T12:52:20.170580+0000","last_change":"2024-05-15T12:51:40.671087+0000","last_active":"2024-05-15T12:52:20.170580+0000","last_peered":"2024-05-15T12:52:20.170580+0000","last_clean":"2024-05-15T12:52:20.170580+0000","last_became_active":"2024-05-15T12:50:06.930143+0000","last_became_peered":"2024-05-15T12:50:06.930143+0000","last_unstale":"2024-05-15T12:52:20.170580+0000","last_undegraded":"2024-05-15T12:52:20.170580+0000","last_fullsized":"2024-05-15T12:52:20.170580+0000","mapping_epoch":156,"log_start":"125'3300","ondisk_log_start":"125'3300","created":110,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'11207","last_scrub_stamp":"2024-05-15T12:51:40.670610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:51:40.670610+0000","objects_scrubbed":2239,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:52:44.163791+0000","scrub_duration":497,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5439488,"num_objects":83,"num_object_clones":0,"num_object_copies":249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13345,"num_write_kb":429696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5078,"num_bytes_recovered":332791808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,5],"acting":[8,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"211'13244","reported_seq":23439,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-05-15T12:52:21.641016+0000","last_change":"2024-05-15T12:51:40.072607+0000","last_active":"2024-05-15T12:52:21.641016+0000","last_peered":"2024-05-15T12:52:21.641016+0000","last_clean":"2024-05-15T12:52:21.641016+0000","last_became_active":"2024-05-15T12:50:08.886465+0000","last_became_peered":"2024-05-15T12:50:08.886465+0000","last_unstale":"2024-05-15T12:52:21.641016+0000","last_undegraded":"2024-05-15T12:52:21.641016+0000","last_fullsized":"2024-05-15T12:52:21.641016+0000","mapping_epoch":158,"log_start":"124'3200","ondisk_log_start":"124'3200","created":110,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'10960","last_scrub_stamp":"2024-05-15T12:51:40.072467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:51:40.072467+0000","objects_scrubbed":2307,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:52:43.778477+0000","scrub_duration":1066,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13244,"num_write_kb":423808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4936,"num_bytes_recovered":323485696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,8],"acting":[0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"210'14723","reported_seq":15050,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-05-15T12:52:20.267882+0000","last_change":"2024-05-15T12:52:08.591560+0000","last_active":"2024-05-15T12:52:20.267882+0000","last_peered":"2024-05-15T12:52:20.267882+0000","last_clean":"2024-05-15T12:52:20.267882+0000","last_became_active":"2024-05-15T12:47:58.444625+0000","last_became_peered":"2024-05-15T12:47:58.444625+0000","last_unstale":"2024-05-15T12:52:20.267882+0000","last_undegraded":"2024-05-15T12:52:20.267882+0000","last_fullsized":"2024-05-15T12:52:20.267882+0000","mapping_epoch":110,"log_start":"134'4700","ondisk_log_start":"134'4700","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'14054","last_scrub_stamp":"2024-05-15T12:52:08.591420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:52:08.591420+0000","objects_scrubbed":754,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:53:15.664439+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5439512,"num_objects":84,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13418,"num_write_kb":432001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"211'13796","reported_seq":14151,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-05-15T12:52:21.662694+0000","last_change":"2024-05-15T12:52:05.112514+0000","last_active":"2024-05-15T12:52:21.662694+0000","last_peered":"2024-05-15T12:52:21.662694+0000","last_clean":"2024-05-15T12:52:21.662694+0000","last_became_active":"2024-05-15T12:47:58.444477+0000","last_became_peered":"2024-05-15T12:47:58.444477+0000","last_unstale":"2024-05-15T12:52:21.662694+0000","last_undegraded":"2024-05-15T12:52:21.662694+0000","last_fullsized":"2024-05-15T12:52:21.662694+0000","mapping_epoch":110,"log_start":"127'3700","ondisk_log_start":"127'3700","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12902","last_scrub_stamp":"2024-05-15T12:52:05.112295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:52:05.112295+0000","objects_scrubbed":899,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:53:30.483122+0000","scrub_duration":299,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13796,"num_write_kb":441472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":35979288,"num_objects":550,"num_object_clones":0,"num_object_copies":1650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":550,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":213712,"num_write_kb":6856321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38964,"num_bytes_recovered":2553544704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":623058944,"data_stored":623050800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160817,"ondisk_log_size":160817,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":0,"seq":386547056723,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351063,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":146096,"kb_used_data":10792,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":93599056,"statfs":{"total":95995035648,"available":95845433344,"internally_reserved":0,"allocated":11051008,"data_stored":10790147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":138541504},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514588,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":178428,"kb_used_data":48692,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93566724,"statfs":{"total":95995035648,"available":95812325376,"internally_reserved":0,"allocated":49860608,"data_stored":49592123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":132839872},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874338,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":195632,"kb_used_data":52008,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93549520,"statfs":{"total":95995035648,"available":95794708480,"internally_reserved":0,"allocated":53256192,"data_stored":52995351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168677,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":128056,"kb_used_data":28720,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93617096,"statfs":{"total":95995035648,"available":95863906304,"internally_reserved":0,"allocated":29409280,"data_stored":29144887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":101579210},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430315,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72644,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93672508,"statfs":{"total":95995035648,"available":95920648192,"internally_reserved":0,"allocated":1241088,"data_stored":972149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574982,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":137264,"kb_used_data":25384,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93607888,"statfs":{"total":95995035648,"available":95854477312,"internally_reserved":0,"allocated":25993216,"data_stored":25732375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738510,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":104064,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93641088,"statfs":{"total":95995035648,"available":95888474112,"internally_reserved":0,"allocated":708608,"data_stored":443221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411459,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":190912,"kb_used_data":87540,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93554240,"statfs":{"total":95995035648,"available":95799541760,"internally_reserved":0,"allocated":89640960,"data_stored":89375573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149819,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":176252,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93568900,"statfs":{"total":95995035648,"available":95814553600,"internally_reserved":0,"allocated":708608,"data_stored":443221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":179763655},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986293,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94644,"kb_used_data":31532,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93650508,"statfs":{"total":95995035648,"available":95898120192,"internally_reserved":0,"allocated":32288768,"data_stored":32023855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691953,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72124,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93673028,"statfs":{"total":95995035648,"available":95921180672,"internally_reserved":0,"allocated":708608,"data_stored":443221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13107200,"data_stored":13107200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86966272,"data_stored":86966272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":25563136,"data_stored":25559064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":19660800,"data_stored":19660800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":30670848,"data_stored":30670848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":29233152,"data_stored":29229080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":3473408,"data_stored":3473408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":414384128,"data_stored":414384128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T12:52:25.002 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T12:52:25.076 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:52:25.108 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:25.129 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:52:25.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:25.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:52:25.233 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:52:25.260 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:24 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:52:25.260 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:24 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:52:25.260 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:24 smithi120 ceph-mon[43961]: osdmap e212: 12 total, 12 up, 11 in 2024-05-15T12:52:25.260 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:24 smithi120 ceph-mon[43961]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:52:25.260 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:24 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4110801496' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:52:25.260 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:24 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:52:25.260 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:24 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:52:25.262 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:25.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:24 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:52:25.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:24 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:52:25.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:24 smithi027 ceph-mon[39795]: osdmap e212: 12 total, 12 up, 11 in 2024-05-15T12:52:25.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:24 smithi027 ceph-mon[39795]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:52:25.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:24 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4110801496' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:52:25.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:24 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:52:25.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:24 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:52:25.281 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:52:25.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:25.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:52:25.385 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:52:25.414 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:25.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:24 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:52:25.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:24 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:52:25.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:24 smithi177 ceph-mon[44303]: osdmap e212: 12 total, 12 up, 11 in 2024-05-15T12:52:25.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:24 smithi177 ceph-mon[44303]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:52:25.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:24 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4110801496' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:52:25.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:24 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:52:25.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:24 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:52:25.537 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:52:25.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:25.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:52:25.541 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:52:25.641 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:52:25.674 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:25.695 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:52:25.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:25.698 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:52:25.799 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:52:25.832 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:25.853 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:52:25.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:25.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:52:25.957 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:52:25.987 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:26.007 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:52:26.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:26.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:52:26.111 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:52:26.137 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:25 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:52:26.137 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:25 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4110801496' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:52:26.137 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:25 smithi120 ceph-mon[43961]: osdmap e213: 12 total, 12 up, 11 in 2024-05-15T12:52:26.137 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:25 smithi120 ceph-mon[43961]: pgmap v596: 17 pgs: 17 active+clean; 7.8 MiB data, 1.3 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 716 op/s 2024-05-15T12:52:26.139 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:26.158 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:52:26.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:26.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:52:26.262 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:52:26.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:25 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:52:26.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:25 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4110801496' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:52:26.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:25 smithi027 ceph-mon[39795]: osdmap e213: 12 total, 12 up, 11 in 2024-05-15T12:52:26.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:25 smithi027 ceph-mon[39795]: pgmap v596: 17 pgs: 17 active+clean; 7.8 MiB data, 1.3 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 716 op/s 2024-05-15T12:52:26.292 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:26.391 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:52:26.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:26.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:52:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:25 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:52:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:25 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4110801496' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:52:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:25 smithi177 ceph-mon[44303]: osdmap e213: 12 total, 12 up, 11 in 2024-05-15T12:52:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:25 smithi177 ceph-mon[44303]: pgmap v596: 17 pgs: 17 active+clean; 7.8 MiB data, 1.3 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 716 op/s 2024-05-15T12:52:26.496 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:52:26.528 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:26.549 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:52:26.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:26.552 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:52:26.653 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:52:26.685 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:26.706 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:52:26.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:26.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:52:26.810 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:52:26.841 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:26.956 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-05-15T12:52:26.957 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-05-15T12:52:26.965 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:52:26.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:26.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:52:27.071 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:52:27.116 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:52:27.116 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:52:27.133 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:52:27.143 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:27.160 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:52:27.169 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:52:27.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:27.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:52:27.187 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:52:27.207 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:52:27.227 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:52:27.247 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:52:27.268 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:52:27.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:27 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/969962945' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:52:27.274 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:52:27.288 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:52:27.306 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:27.308 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:52:27.327 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:52:27.328 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:52:27.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:27.330 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:52:27.331 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:52:27.331 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:52:27.331 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:52:27.331 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:52:27.331 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:52:27.331 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:52:27.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:52:27.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:52:27.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:52:27.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:52:27.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:52:27.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:52:27.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:27 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/969962945' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:52:27.431 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:52:27.460 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:27.479 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:52:27.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:27.482 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:52:27.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:27 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/969962945' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:52:27.583 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:52:27.615 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:27.636 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:52:27.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:27.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:52:27.740 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:52:27.769 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:27.788 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:52:27.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:27.791 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:52:27.892 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:52:27.921 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:28.409 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T12:52:28.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:28 smithi177 ceph-mon[44303]: pgmap v597: 17 pgs: 17 active+clean; 577 KiB data, 1.3 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 404 op/s 2024-05-15T12:52:28.435 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:28 smithi027 ceph-mon[39795]: pgmap v597: 17 pgs: 17 active+clean; 577 KiB data, 1.3 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 404 op/s 2024-05-15T12:52:28.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:28 smithi120 ceph-mon[43961]: pgmap v597: 17 pgs: 17 active+clean; 577 KiB data, 1.3 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 404 op/s 2024-05-15T12:52:28.748 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:52:28.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:28.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:52:28.852 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:52:28.884 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:28.905 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:52:28.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:28.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:52:29.009 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:52:29.039 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:29.059 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:52:29.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:29.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:52:29.163 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:52:29.191 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:29.211 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:52:29.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:29.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:52:29.315 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:52:29.347 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:29.367 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:52:29.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:29.370 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:52:29.471 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:52:29.501 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:29.554 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:52:29.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:29.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:52:29.660 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:52:29.679 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T12:52:29.679 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:52:29.689 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:29.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:29 smithi027 ceph-mon[39795]: pgmap v598: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 246 op/s 2024-05-15T12:52:30.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:29 smithi120 ceph-mon[43961]: pgmap v598: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 246 op/s 2024-05-15T12:52:30.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:29 smithi177 ceph-mon[44303]: pgmap v598: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 246 op/s 2024-05-15T12:52:30.253 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:52:30.338 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:52:30.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:30.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:52:30.354 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:52:30.358 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:52:30.401 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:52:30.443 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:52:30.471 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:30.921 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:52:30.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:30.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:52:31.025 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:52:31.054 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:31.185 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:52:31.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:31.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:52:31.289 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:52:31.317 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:32.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:32 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:52:32.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:32 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:52:32.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:32 smithi177 ceph-mon[44303]: osdmap e214: 12 total, 12 up, 11 in 2024-05-15T12:52:32.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:32 smithi177 ceph-mon[44303]: pgmap v600: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 71 op/s 2024-05-15T12:52:32.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:32 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:52:32.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:32 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:52:32.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:32 smithi027 ceph-mon[39795]: osdmap e214: 12 total, 12 up, 11 in 2024-05-15T12:52:32.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:32 smithi027 ceph-mon[39795]: pgmap v600: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 71 op/s 2024-05-15T12:52:32.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:32 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:52:32.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:32 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:52:32.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:32 smithi120 ceph-mon[43961]: osdmap e214: 12 total, 12 up, 11 in 2024-05-15T12:52:32.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:32 smithi120 ceph-mon[43961]: pgmap v600: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 71 op/s 2024-05-15T12:52:34.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:33 smithi027 ceph-mon[39795]: pgmap v601: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 63 op/s 2024-05-15T12:52:34.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:33 smithi120 ceph-mon[43961]: pgmap v601: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 63 op/s 2024-05-15T12:52:34.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:33 smithi177 ceph-mon[44303]: pgmap v601: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 63 op/s 2024-05-15T12:52:35.096 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:52:35.096 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T12:52:35.246 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T12:52:35.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:35 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3171565860' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:52:35.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:35 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:52:35.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:35 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3171565860' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:52:35.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:35 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:52:35.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:35 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3171565860' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:52:35.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:35 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:52:36.172 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-05-15T12:52:36.253 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:36 smithi027 ceph-mon[39795]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:52:36.253 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:36 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:52:36.253 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:36 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:52:36.253 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:36 smithi027 ceph-mon[39795]: osdmap e215: 12 total, 12 up, 11 in 2024-05-15T12:52:36.253 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:36 smithi027 ceph-mon[39795]: pgmap v603: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:36.473 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:52:36.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:36.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:52:36.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:36 smithi120 ceph-mon[43961]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:52:36.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:36 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:52:36.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:36 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:52:36.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:36 smithi120 ceph-mon[43961]: osdmap e215: 12 total, 12 up, 11 in 2024-05-15T12:52:36.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:36 smithi120 ceph-mon[43961]: pgmap v603: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:36.577 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:52:36.605 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:36 smithi177 ceph-mon[44303]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:52:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:36 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:52:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:36 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:52:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:36 smithi177 ceph-mon[44303]: osdmap e215: 12 total, 12 up, 11 in 2024-05-15T12:52:36.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:36 smithi177 ceph-mon[44303]: pgmap v603: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:36.712 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:52:36.729 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:52:36.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:36.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:52:36.835 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:52:36.867 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:36.887 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:52:36.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:36.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:52:36.991 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:52:37.023 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:37.044 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:52:37.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:37.048 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:52:37.148 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:52:37.178 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:37.252 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:52:37.255 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:37 smithi027 ceph-mon[39795]: 2.a scrub starts 2024-05-15T12:52:37.255 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:37 smithi027 ceph-mon[39795]: 2.a scrub ok 2024-05-15T12:52:37.255 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:37 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3666667037' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T12:52:37.255 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:52:37.255 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:52:37.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:37.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:52:37.269 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_0' removed 2024-05-15T12:52:37.357 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:52:37.382 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:37 smithi177 ceph-mon[44303]: 2.a scrub starts 2024-05-15T12:52:37.382 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:37 smithi177 ceph-mon[44303]: 2.a scrub ok 2024-05-15T12:52:37.383 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:37 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3666667037' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T12:52:37.383 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:52:37.383 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:52:37.384 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:37.404 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:52:37.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:37.407 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:52:37.508 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:52:37.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:37 smithi120 ceph-mon[43961]: 2.a scrub starts 2024-05-15T12:52:37.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:37 smithi120 ceph-mon[43961]: 2.a scrub ok 2024-05-15T12:52:37.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:37 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3666667037' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T12:52:37.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:52:37.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:52:37.534 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:37.552 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:52:37.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:37.555 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:52:37.656 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:52:37.682 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:37.705 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:52:37.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:37.708 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:52:37.712 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":601,"stamp":"2024-05-15T12:52:33.638599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":214443,"num_write_kb":6859259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38976,"num_bytes_recovered":2556718272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":52,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":1299736,"kb_used_data":75200,"kb_used_omap":17,"kb_used_meta":1224366,"kb_avail":1029896936,"statfs":{"total":1055945392128,"available":1054614462464,"internally_reserved":0,"allocated":77004800,"data_stored":73961197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":1253751669},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-35979288,"num_objects":-550,"num_object_clones":0,"num_object_copies":-1650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":550,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.644962"},"pg_stats":[{"pgid":"2.f","version":"211'13352","reported_seq":13667,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-05-15T12:52:25.894979+0000","last_change":"2024-05-15T12:51:36.662714+0000","last_active":"2024-05-15T12:52:25.894979+0000","last_peered":"2024-05-15T12:52:25.894979+0000","last_clean":"2024-05-15T12:52:25.894979+0000","last_became_active":"2024-05-15T12:47:58.401685+0000","last_became_peered":"2024-05-15T12:47:58.401685+0000","last_unstale":"2024-05-15T12:52:25.894979+0000","last_undegraded":"2024-05-15T12:52:25.894979+0000","last_fullsized":"2024-05-15T12:52:25.894979+0000","mapping_epoch":110,"log_start":"125'3300","ondisk_log_start":"125'3300","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'10955","last_scrub_stamp":"2024-05-15T12:51:36.662404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:51:36.662404+0000","objects_scrubbed":2415,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:53:04.071424+0000","scrub_duration":536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13352,"num_write_kb":427264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"211'13318","reported_seq":28059,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-05-15T12:52:25.894767+0000","last_change":"2024-05-15T12:52:07.535000+0000","last_active":"2024-05-15T12:52:25.894767+0000","last_peered":"2024-05-15T12:52:25.894767+0000","last_clean":"2024-05-15T12:52:25.894767+0000","last_became_active":"2024-05-15T12:50:45.374644+0000","last_became_peered":"2024-05-15T12:50:45.374644+0000","last_unstale":"2024-05-15T12:52:25.894767+0000","last_undegraded":"2024-05-15T12:52:25.894767+0000","last_fullsized":"2024-05-15T12:52:25.894767+0000","mapping_epoch":173,"log_start":"125'3300","ondisk_log_start":"125'3300","created":110,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12553","last_scrub_stamp":"2024-05-15T12:52:07.534725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:52:07.534725+0000","objects_scrubbed":774,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:53:18.408005+0000","scrub_duration":355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13318,"num_write_kb":426176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7219,"num_bytes_recovered":473104384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"211'13254","reported_seq":13562,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-05-15T12:52:25.895018+0000","last_change":"2024-05-15T12:51:44.611596+0000","last_active":"2024-05-15T12:52:25.895018+0000","last_peered":"2024-05-15T12:52:25.895018+0000","last_clean":"2024-05-15T12:52:25.895018+0000","last_became_active":"2024-05-15T12:47:58.401043+0000","last_became_peered":"2024-05-15T12:47:58.401043+0000","last_unstale":"2024-05-15T12:52:25.895018+0000","last_undegraded":"2024-05-15T12:52:25.895018+0000","last_fullsized":"2024-05-15T12:52:25.895018+0000","mapping_epoch":110,"log_start":"125'3200","ondisk_log_start":"125'3200","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'11200","last_scrub_stamp":"2024-05-15T12:51:44.611436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:51:44.611436+0000","objects_scrubbed":2066,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:53:14.472861+0000","scrub_duration":459,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13254,"num_write_kb":424128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"211'13414","reported_seq":13749,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-05-15T12:52:31.335066+0000","last_change":"2024-05-15T12:51:03.134479+0000","last_active":"2024-05-15T12:52:31.335066+0000","last_peered":"2024-05-15T12:52:31.335066+0000","last_clean":"2024-05-15T12:52:31.335066+0000","last_became_active":"2024-05-15T12:47:59.089659+0000","last_became_peered":"2024-05-15T12:47:59.089659+0000","last_unstale":"2024-05-15T12:52:31.335066+0000","last_undegraded":"2024-05-15T12:52:31.335066+0000","last_fullsized":"2024-05-15T12:52:31.335066+0000","mapping_epoch":110,"log_start":"127'3400","ondisk_log_start":"127'3400","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'9994","last_scrub_stamp":"2024-05-15T12:51:03.134217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:51:03.134217+0000","objects_scrubbed":3456,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:52:31.966162+0000","scrub_duration":1384,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13414,"num_write_kb":429248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"211'13268","reported_seq":13581,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-05-15T12:52:25.895036+0000","last_change":"2024-05-15T12:51:36.698048+0000","last_active":"2024-05-15T12:52:25.895036+0000","last_peered":"2024-05-15T12:52:25.895036+0000","last_clean":"2024-05-15T12:52:25.895036+0000","last_became_active":"2024-05-15T12:47:58.401049+0000","last_became_peered":"2024-05-15T12:47:58.401049+0000","last_unstale":"2024-05-15T12:52:25.895036+0000","last_undegraded":"2024-05-15T12:52:25.895036+0000","last_fullsized":"2024-05-15T12:52:25.895036+0000","mapping_epoch":110,"log_start":"124'3200","ondisk_log_start":"124'3200","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'10931","last_scrub_stamp":"2024-05-15T12:51:36.697819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:51:36.697819+0000","objects_scrubbed":2348,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:53:00.881405+0000","scrub_duration":526,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13268,"num_write_kb":424576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":633,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-05-15T12:52:25.894835+0000","last_change":"2024-05-15T12:52:02.886508+0000","last_active":"2024-05-15T12:52:25.894835+0000","last_peered":"2024-05-15T12:52:25.894835+0000","last_clean":"2024-05-15T12:52:25.894835+0000","last_became_active":"2024-05-15T12:49:08.168334+0000","last_became_peered":"2024-05-15T12:49:08.168334+0000","last_unstale":"2024-05-15T12:52:25.894835+0000","last_undegraded":"2024-05-15T12:52:25.894835+0000","last_fullsized":"2024-05-15T12:52:25.894835+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T12:52:02.886389+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T12:52:02.886389+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:53:20.689368+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"211'13520","reported_seq":34106,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-05-15T12:52:25.894980+0000","last_change":"2024-05-15T12:51:38.513245+0000","last_active":"2024-05-15T12:52:25.894980+0000","last_peered":"2024-05-15T12:52:25.894980+0000","last_clean":"2024-05-15T12:52:25.894980+0000","last_became_active":"2024-05-15T12:51:17.929582+0000","last_became_peered":"2024-05-15T12:51:17.929582+0000","last_unstale":"2024-05-15T12:52:25.894980+0000","last_undegraded":"2024-05-15T12:52:25.894980+0000","last_fullsized":"2024-05-15T12:52:25.894980+0000","mapping_epoch":190,"log_start":"127'3500","ondisk_log_start":"127'3500","created":110,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'11129","last_scrub_stamp":"2024-05-15T12:51:38.512996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:51:38.512996+0000","objects_scrubbed":2417,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:53:07.028342+0000","scrub_duration":676,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13520,"num_write_kb":432640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10151,"num_bytes_recovered":665255936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"211'13420","reported_seq":13731,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-05-15T12:52:25.894828+0000","last_change":"2024-05-15T12:51:41.798194+0000","last_active":"2024-05-15T12:52:25.894828+0000","last_peered":"2024-05-15T12:52:25.894828+0000","last_clean":"2024-05-15T12:52:25.894828+0000","last_became_active":"2024-05-15T12:47:59.089807+0000","last_became_peered":"2024-05-15T12:47:59.089807+0000","last_unstale":"2024-05-15T12:52:25.894828+0000","last_undegraded":"2024-05-15T12:52:25.894828+0000","last_fullsized":"2024-05-15T12:52:25.894828+0000","mapping_epoch":110,"log_start":"126'3400","ondisk_log_start":"126'3400","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'11284","last_scrub_stamp":"2024-05-15T12:51:41.797972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:51:41.797972+0000","objects_scrubbed":2157,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:53:05.487584+0000","scrub_duration":579,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13420,"num_write_kb":429440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"211'13300","reported_seq":13579,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-05-15T12:52:25.894638+0000","last_change":"2024-05-15T12:52:08.430907+0000","last_active":"2024-05-15T12:52:25.894638+0000","last_peered":"2024-05-15T12:52:25.894638+0000","last_clean":"2024-05-15T12:52:25.894638+0000","last_became_active":"2024-05-15T12:47:58.444441+0000","last_became_peered":"2024-05-15T12:47:58.444441+0000","last_unstale":"2024-05-15T12:52:25.894638+0000","last_undegraded":"2024-05-15T12:52:25.894638+0000","last_fullsized":"2024-05-15T12:52:25.894638+0000","mapping_epoch":110,"log_start":"124'3200","ondisk_log_start":"124'3200","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12619","last_scrub_stamp":"2024-05-15T12:52:08.430641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:52:08.430641+0000","objects_scrubbed":686,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:53:17.854752+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13300,"num_write_kb":425600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"211'13354","reported_seq":26666,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-05-15T12:52:31.368922+0000","last_change":"2024-05-15T12:50:52.684827+0000","last_active":"2024-05-15T12:52:31.368922+0000","last_peered":"2024-05-15T12:52:31.368922+0000","last_clean":"2024-05-15T12:52:31.368922+0000","last_became_active":"2024-05-15T12:50:48.402873+0000","last_became_peered":"2024-05-15T12:50:48.402873+0000","last_unstale":"2024-05-15T12:52:31.368922+0000","last_undegraded":"2024-05-15T12:52:31.368922+0000","last_fullsized":"2024-05-15T12:52:31.368922+0000","mapping_epoch":176,"log_start":"125'3300","ondisk_log_start":"125'3300","created":110,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'9459","last_scrub_stamp":"2024-05-15T12:50:52.684491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:50:52.684491+0000","objects_scrubbed":3922,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:52:34.577252+0000","scrub_duration":890,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13354,"num_write_kb":427328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6508,"num_bytes_recovered":426508288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"211'13380","reported_seq":13669,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-05-15T12:52:25.894831+0000","last_change":"2024-05-15T12:51:39.397895+0000","last_active":"2024-05-15T12:52:25.894831+0000","last_peered":"2024-05-15T12:52:25.894831+0000","last_clean":"2024-05-15T12:52:25.894831+0000","last_became_active":"2024-05-15T12:47:59.089595+0000","last_became_peered":"2024-05-15T12:47:59.089595+0000","last_unstale":"2024-05-15T12:52:25.894831+0000","last_undegraded":"2024-05-15T12:52:25.894831+0000","last_fullsized":"2024-05-15T12:52:25.894831+0000","mapping_epoch":110,"log_start":"124'3300","ondisk_log_start":"124'3300","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'11159","last_scrub_stamp":"2024-05-15T12:51:39.397762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:51:39.397762+0000","objects_scrubbed":2232,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:52:57.578486+0000","scrub_duration":517,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13380,"num_write_kb":428160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"211'13388","reported_seq":13654,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-05-15T12:52:25.894653+0000","last_change":"2024-05-15T12:50:51.797101+0000","last_active":"2024-05-15T12:52:25.894653+0000","last_peered":"2024-05-15T12:52:25.894653+0000","last_clean":"2024-05-15T12:52:25.894653+0000","last_became_active":"2024-05-15T12:47:58.445362+0000","last_became_peered":"2024-05-15T12:47:58.445362+0000","last_unstale":"2024-05-15T12:52:25.894653+0000","last_undegraded":"2024-05-15T12:52:25.894653+0000","last_fullsized":"2024-05-15T12:52:25.894653+0000","mapping_epoch":110,"log_start":"125'3300","ondisk_log_start":"125'3300","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'9378","last_scrub_stamp":"2024-05-15T12:50:51.796727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:50:51.796727+0000","objects_scrubbed":4041,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:52:26.182635+0000","scrub_duration":1627,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13388,"num_write_kb":428416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"211'13324","reported_seq":23760,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-05-15T12:52:25.894504+0000","last_change":"2024-05-15T12:50:52.144944+0000","last_active":"2024-05-15T12:52:25.894504+0000","last_peered":"2024-05-15T12:52:25.894504+0000","last_clean":"2024-05-15T12:52:25.894504+0000","last_became_active":"2024-05-15T12:50:11.967389+0000","last_became_peered":"2024-05-15T12:50:11.967389+0000","last_unstale":"2024-05-15T12:52:25.894504+0000","last_undegraded":"2024-05-15T12:52:25.894504+0000","last_fullsized":"2024-05-15T12:52:25.894504+0000","mapping_epoch":160,"log_start":"125'3300","ondisk_log_start":"125'3300","created":110,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'9401","last_scrub_stamp":"2024-05-15T12:50:52.144769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:50:52.144769+0000","objects_scrubbed":3950,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:52:29.736243+0000","scrub_duration":1350,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13324,"num_write_kb":426368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5072,"num_bytes_recovered":332398592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,8],"acting":[3,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"211'13428","reported_seq":23910,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-05-15T12:52:25.894921+0000","last_change":"2024-05-15T12:51:40.671087+0000","last_active":"2024-05-15T12:52:25.894921+0000","last_peered":"2024-05-15T12:52:25.894921+0000","last_clean":"2024-05-15T12:52:25.894921+0000","last_became_active":"2024-05-15T12:50:06.930143+0000","last_became_peered":"2024-05-15T12:50:06.930143+0000","last_unstale":"2024-05-15T12:52:25.894921+0000","last_undegraded":"2024-05-15T12:52:25.894921+0000","last_fullsized":"2024-05-15T12:52:25.894921+0000","mapping_epoch":156,"log_start":"127'3400","ondisk_log_start":"127'3400","created":110,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'11207","last_scrub_stamp":"2024-05-15T12:51:40.670610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:51:40.670610+0000","objects_scrubbed":2239,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:52:44.163791+0000","scrub_duration":497,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13428,"num_write_kb":429696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5078,"num_bytes_recovered":332791808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,5],"acting":[8,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"211'13244","reported_seq":23445,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-05-15T12:52:32.968027+0000","last_change":"2024-05-15T12:51:40.072607+0000","last_active":"2024-05-15T12:52:32.968027+0000","last_peered":"2024-05-15T12:52:32.968027+0000","last_clean":"2024-05-15T12:52:32.968027+0000","last_became_active":"2024-05-15T12:50:08.886465+0000","last_became_peered":"2024-05-15T12:50:08.886465+0000","last_unstale":"2024-05-15T12:52:32.968027+0000","last_undegraded":"2024-05-15T12:52:32.968027+0000","last_fullsized":"2024-05-15T12:52:32.968027+0000","mapping_epoch":158,"log_start":"124'3200","ondisk_log_start":"124'3200","created":110,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'10960","last_scrub_stamp":"2024-05-15T12:51:40.072467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:51:40.072467+0000","objects_scrubbed":2307,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:52:43.778477+0000","scrub_duration":1066,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13244,"num_write_kb":423808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4936,"num_bytes_recovered":323485696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,8],"acting":[0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"211'14807","reported_seq":15140,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-05-15T12:52:25.895139+0000","last_change":"2024-05-15T12:52:08.591560+0000","last_active":"2024-05-15T12:52:25.895139+0000","last_peered":"2024-05-15T12:52:25.895139+0000","last_clean":"2024-05-15T12:52:25.895139+0000","last_became_active":"2024-05-15T12:47:58.444625+0000","last_became_peered":"2024-05-15T12:47:58.444625+0000","last_unstale":"2024-05-15T12:52:25.895139+0000","last_undegraded":"2024-05-15T12:52:25.895139+0000","last_fullsized":"2024-05-15T12:52:25.895139+0000","mapping_epoch":110,"log_start":"134'4800","ondisk_log_start":"134'4800","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'14054","last_scrub_stamp":"2024-05-15T12:52:08.591420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:52:08.591420+0000","objects_scrubbed":754,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:53:15.664439+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13502,"num_write_kb":432001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"211'13796","reported_seq":14157,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-05-15T12:52:31.335191+0000","last_change":"2024-05-15T12:52:05.112514+0000","last_active":"2024-05-15T12:52:31.335191+0000","last_peered":"2024-05-15T12:52:31.335191+0000","last_clean":"2024-05-15T12:52:31.335191+0000","last_became_active":"2024-05-15T12:47:58.444477+0000","last_became_peered":"2024-05-15T12:47:58.444477+0000","last_unstale":"2024-05-15T12:52:31.335191+0000","last_undegraded":"2024-05-15T12:52:31.335191+0000","last_fullsized":"2024-05-15T12:52:31.335191+0000","mapping_epoch":110,"log_start":"127'3700","ondisk_log_start":"127'3700","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12902","last_scrub_stamp":"2024-05-15T12:52:05.112295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:47:57.380220+0000","last_clean_scrub_stamp":"2024-05-15T12:52:05.112295+0000","objects_scrubbed":899,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:53:30.483122+0000","scrub_duration":299,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13796,"num_write_kb":441472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":214262,"num_write_kb":6856321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38964,"num_bytes_recovered":2553544704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":403046400,"data_stored":403046400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160767,"ondisk_log_size":160767,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":0,"seq":386547056725,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351065,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":140188,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93604964,"statfs":{"total":95995035648,"available":95851483136,"internally_reserved":0,"allocated":733184,"data_stored":458685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":142801344},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514590,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":135136,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":133886,"kb_avail":93610016,"statfs":{"total":95995035648,"available":95856656384,"internally_reserved":0,"allocated":1265664,"data_stored":987613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":137099712},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874340,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":148508,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93596644,"statfs":{"total":95995035648,"available":95842963456,"internally_reserved":0,"allocated":733184,"data_stored":458705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168679,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":100452,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93644700,"statfs":{"total":95995035648,"available":95892172800,"internally_reserved":0,"allocated":1265664,"data_stored":987633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":101579210},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430317,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72684,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93672468,"statfs":{"total":95995035648,"available":95920607232,"internally_reserved":0,"allocated":1277952,"data_stored":995375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574984,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112408,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93632744,"statfs":{"total":95995035648,"available":95879929856,"internally_reserved":0,"allocated":733184,"data_stored":458705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738512,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":104100,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93641052,"statfs":{"total":95995035648,"available":95888437248,"internally_reserved":0,"allocated":745472,"data_stored":466447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411461,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":169828,"kb_used_data":66456,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93575324,"statfs":{"total":95995035648,"available":95821131776,"internally_reserved":0,"allocated":68050944,"data_stored":67771919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149821,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":176284,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93568868,"statfs":{"total":95995035648,"available":95814520832,"internally_reserved":0,"allocated":733184,"data_stored":458705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":179763655},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986295,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67996,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93677156,"statfs":{"total":95995035648,"available":95925407744,"internally_reserved":0,"allocated":733184,"data_stored":458705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691955,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72152,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93673000,"statfs":{"total":95995035648,"available":95921152000,"internally_reserved":0,"allocated":733184,"data_stored":458705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":67305472,"data_stored":67305472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":335740928,"data_stored":335740928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T12:52:37.713 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-05-15T12:52:37.713 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T12:52:37.714 INFO:tasks.thrashosds.thrasher:inject_pause on osd.10 2024-05-15T12:52:37.714 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-15T12:52:37.714 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-15T12:52:37.714 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 config set filestore_inject_stall 3 2024-05-15T12:52:37.809 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:52:37.838 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:38.005 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:52:38.454 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:38 smithi027 ceph-mon[39795]: 2.1 scrub starts 2024-05-15T12:52:38.455 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:38 smithi027 ceph-mon[39795]: 2.1 scrub ok 2024-05-15T12:52:38.455 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:38 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3666667037' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T12:52:38.455 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:38 smithi027 ceph-mon[39795]: osdmap e216: 12 total, 12 up, 11 in 2024-05-15T12:52:38.455 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:38 smithi027 ceph-mon[39795]: pgmap v605: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:38.455 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:38 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:52:38.455 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:38 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:52:38.455 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:38 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:52:38.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:38 smithi120 ceph-mon[43961]: 2.1 scrub starts 2024-05-15T12:52:38.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:38 smithi120 ceph-mon[43961]: 2.1 scrub ok 2024-05-15T12:52:38.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:38 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3666667037' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T12:52:38.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:38 smithi120 ceph-mon[43961]: osdmap e216: 12 total, 12 up, 11 in 2024-05-15T12:52:38.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:38 smithi120 ceph-mon[43961]: pgmap v605: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:38.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:38 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:52:38.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:38 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:52:38.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:38 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:52:38.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:38 smithi177 ceph-mon[44303]: 2.1 scrub starts 2024-05-15T12:52:38.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:38 smithi177 ceph-mon[44303]: 2.1 scrub ok 2024-05-15T12:52:38.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:38 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3666667037' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T12:52:38.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:38 smithi177 ceph-mon[44303]: osdmap e216: 12 total, 12 up, 11 in 2024-05-15T12:52:38.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:38 smithi177 ceph-mon[44303]: pgmap v605: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:38.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:38 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:52:38.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:38 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:52:38.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:38 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:52:39.185 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:52:39.187 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:52:39.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:39.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:52:39.292 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:52:39.320 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:39.442 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:52:39.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:39.445 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:52:39.546 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:52:39.575 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:39.697 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:52:39.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:39.701 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:52:39.713 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T12:52:39.714 INFO:teuthology.orchestra.run.smithi177.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-05-15T12:52:39.714 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T12:52:39.802 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:52:39.834 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:39.855 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:52:39.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:39.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:52:39.959 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:52:39.992 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:39 smithi027 ceph-mon[39795]: pgmap v606: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:39.994 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:40.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:39 smithi120 ceph-mon[43961]: pgmap v606: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:40.044 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:39 smithi177 ceph-mon[44303]: pgmap v606: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:40.147 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:52:40.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:40.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:52:40.177 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-05-15T12:52:40.177 INFO:tasks.radosbench:Beginning radosbench... 2024-05-15T12:52:40.178 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-05-15T12:52:40.178 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-05-15T12:52:40.254 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:52:40.285 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:40.306 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:52:40.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:40.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:52:40.410 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:52:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:40.459 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:52:40.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:40.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:52:40.482 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T12:52:40.563 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:52:40.595 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:40.616 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:52:40.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:40.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:52:40.720 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:52:40.749 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:40.769 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:52:40.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:40.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:52:40.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:40 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:52:40.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:40 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:52:40.872 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:52:40.897 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:40 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:52:40.897 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:40 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:52:40.899 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:40.918 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:52:40.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:40.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:52:41.022 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:52:41.050 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:40 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:52:41.050 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:40 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:52:41.052 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:41.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:41.076 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:52:41.076 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:52:41.177 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:52:41.205 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:41.224 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:52:41.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:41.227 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:52:41.328 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:52:41.356 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:41.376 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:52:41.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:41.379 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:52:41.480 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:52:41.512 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:41.533 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:52:41.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:41.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:52:41.617 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:52:41.625 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:52:41.629 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:52:41.637 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:52:41.639 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:52:41.666 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:52:41.671 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:41.688 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:52:41.708 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:52:41.729 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:52:41.749 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:52:41.769 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:52:41.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:41 smithi120 ceph-mon[43961]: osdmap e217: 12 total, 12 up, 11 in 2024-05-15T12:52:41.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:41 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2124307923' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:52:41.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:41 smithi120 ceph-mon[43961]: pgmap v608: 1 pgs: 1 active+clean; 577 KiB data, 1.3 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:41.789 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:52:41.810 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:52:41.830 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:52:41.850 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:52:41.851 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:52:41.851 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:52:41.851 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:52:41.852 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:52:41.852 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:52:41.852 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:52:41.852 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:52:41.852 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:52:41.852 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:52:41.852 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:52:41.852 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:52:41.852 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:52:41.925 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:52:41.926 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:41 smithi027 ceph-mon[39795]: osdmap e217: 12 total, 12 up, 11 in 2024-05-15T12:52:41.926 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:41 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2124307923' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:52:41.926 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:41 smithi027 ceph-mon[39795]: pgmap v608: 1 pgs: 1 active+clean; 577 KiB data, 1.3 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:41.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:41.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:52:42.029 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:52:42.058 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:42.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:41 smithi177 ceph-mon[44303]: osdmap e217: 12 total, 12 up, 11 in 2024-05-15T12:52:42.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:41 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2124307923' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:52:42.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:41 smithi177 ceph-mon[44303]: pgmap v608: 1 pgs: 1 active+clean; 577 KiB data, 1.3 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:43.337 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:52:43.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:43.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:52:43.443 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:52:43.476 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:43.497 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:52:43.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:43.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:52:43.601 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:52:43.635 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:43.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:43 smithi120 ceph-mon[43961]: pgmap v609: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:44.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:43 smithi027 ceph-mon[39795]: pgmap v609: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:44.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:43 smithi177 ceph-mon[44303]: pgmap v609: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:52:44.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:44.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:52:44.296 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:52:44.330 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:44.351 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:52:44.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:44.355 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:52:44.456 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:52:44.485 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:44.733 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:52:44.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:44.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:52:44.837 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:52:44.869 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:44.891 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:52:44.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:44.894 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:52:44.995 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:52:45.026 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:45.047 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:52:45.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:45.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:52:45.151 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:52:45.182 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:45.369 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] out_osds: [11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T12:52:45.370 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 0 1 2024-05-15T12:52:46.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:45 smithi027 ceph-mon[39795]: pgmap v610: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:46.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:45 smithi120 ceph-mon[43961]: pgmap v610: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:46.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:45 smithi177 ceph-mon[44303]: pgmap v610: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:48.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:47 smithi027 ceph-mon[39795]: pgmap v611: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:48.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:47 smithi120 ceph-mon[43961]: pgmap v611: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:48.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:47 smithi177 ceph-mon[44303]: pgmap v611: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:50.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:49 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:52:50.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:49 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:52:50.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:49 smithi120 ceph-mon[43961]: osdmap e218: 12 total, 12 up, 11 in 2024-05-15T12:52:50.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:49 smithi120 ceph-mon[43961]: pgmap v613: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:50.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:49 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:52:50.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:49 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:52:50.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:49 smithi177 ceph-mon[44303]: osdmap e218: 12 total, 12 up, 11 in 2024-05-15T12:52:50.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:49 smithi177 ceph-mon[44303]: pgmap v613: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:50.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:49 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:52:50.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:49 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:52:50.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:49 smithi027 ceph-mon[39795]: osdmap e218: 12 total, 12 up, 11 in 2024-05-15T12:52:50.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:49 smithi027 ceph-mon[39795]: pgmap v613: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:52.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:51 smithi027 ceph-mon[39795]: pgmap v614: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:52.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:51 smithi120 ceph-mon[43961]: pgmap v614: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:52.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:51 smithi177 ceph-mon[44303]: pgmap v614: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:54.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:53 smithi027 ceph-mon[39795]: pgmap v615: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:54.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:53 smithi120 ceph-mon[43961]: pgmap v615: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:54.318 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:52:54.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:54.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:52:54.423 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:52:54.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:53 smithi177 ceph-mon[44303]: pgmap v615: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:54.457 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:54.479 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:52:54.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:54.482 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:52:54.583 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:52:54.616 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:54.637 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:52:54.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:54.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:52:54.742 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:52:54.773 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:55.155 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_1' created 2024-05-15T12:52:55.265 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T12:52:55.265 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T12:52:55.266 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-05-15T12:52:55.266 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T12:52:55.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:55 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2033669120' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-05-15T12:52:55.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:55 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2033669120' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-05-15T12:52:55.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:55 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2033669120' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-05-15T12:52:55.623 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:52:55.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:55.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:52:55.737 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:52:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:56.163 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T12:52:56.171 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-05-15T12:52:56.174 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:52:56.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:56 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2033669120' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-05-15T12:52:56.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:56 smithi177 ceph-mon[44303]: osdmap e219: 12 total, 12 up, 11 in 2024-05-15T12:52:56.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:56 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3225368294' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:52:56.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:56 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:52:56.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:56 smithi177 ceph-mon[44303]: pgmap v617: 17 pgs: 1 active+clean, 4 creating+peering, 12 unknown; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:56.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:56 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:52:56.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:56 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T12:52:56.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:56 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:52:56.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:56 smithi177 ceph-mon[44303]: osdmap e220: 12 total, 12 up, 11 in 2024-05-15T12:52:56.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:56 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2033669120' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-05-15T12:52:56.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:56 smithi027 ceph-mon[39795]: osdmap e219: 12 total, 12 up, 11 in 2024-05-15T12:52:56.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:56 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3225368294' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:52:56.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:56 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:52:56.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:56 smithi027 ceph-mon[39795]: pgmap v617: 17 pgs: 1 active+clean, 4 creating+peering, 12 unknown; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:56.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:56 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:52:56.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:56 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T12:52:56.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:56 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:52:56.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:56 smithi027 ceph-mon[39795]: osdmap e220: 12 total, 12 up, 11 in 2024-05-15T12:52:56.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:56 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2033669120' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-05-15T12:52:56.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:56 smithi120 ceph-mon[43961]: osdmap e219: 12 total, 12 up, 11 in 2024-05-15T12:52:56.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:56 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3225368294' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:52:56.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:56 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:52:56.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:56 smithi120 ceph-mon[43961]: pgmap v617: 17 pgs: 1 active+clean, 4 creating+peering, 12 unknown; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:56.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:56 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:52:56.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:56 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T12:52:56.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:56 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:52:56.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:56 smithi120 ceph-mon[43961]: osdmap e220: 12 total, 12 up, 11 in 2024-05-15T12:52:56.646 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:52:57.185 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-05-15T12:52:57.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:57 smithi177 ceph-mon[44303]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T12:52:57.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:57 smithi177 ceph-mon[44303]: osdmap e221: 12 total, 12 up, 11 in 2024-05-15T12:52:57.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:57 smithi027 ceph-mon[39795]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T12:52:57.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:57 smithi027 ceph-mon[39795]: osdmap e221: 12 total, 12 up, 11 in 2024-05-15T12:52:57.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:57 smithi120 ceph-mon[43961]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T12:52:57.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:57 smithi120 ceph-mon[43961]: osdmap e221: 12 total, 12 up, 11 in 2024-05-15T12:52:57.617 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:52:57.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:57.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:52:57.721 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:52:57.753 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:57.774 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:52:57.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:57.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:52:57.914 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:52:57.944 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:57.964 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:52:57.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:57.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:52:58.068 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:52:58.100 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:58.122 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:52:58.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:58.126 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:52:58.227 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:52:58.257 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:58.278 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:52:58.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:58.280 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:52:58.381 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:52:58.414 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:58.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:58 smithi177 ceph-mon[44303]: pgmap v620: 17 pgs: 4 active+clean, 6 creating+peering, 7 unknown; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:58.435 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:52:58.437 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:58 smithi120 ceph-mon[43961]: pgmap v620: 17 pgs: 4 active+clean, 6 creating+peering, 7 unknown; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:58.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:58.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:52:58.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:58 smithi027 ceph-mon[39795]: pgmap v620: 17 pgs: 4 active+clean, 6 creating+peering, 7 unknown; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:52:58.539 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:52:58.569 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:58.589 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:52:58.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:58.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:52:58.643 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:52:58.644 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:52:58.686 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:52:58.693 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:52:58.723 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:58.971 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:52:58.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:58.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:52:59.076 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:52:59.108 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:59.129 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:52:59.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:59.132 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:52:59.231 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:59 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:52:59.231 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:52:59 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:52:59.233 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:52:59.262 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:59.281 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:52:59.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:59.285 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:52:59.386 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:52:59.418 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:59.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:59 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:52:59.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:52:59 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:52:59.440 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:52:59.441 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:59 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:52:59.441 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:52:59 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:52:59.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:59.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:52:59.544 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:52:59.574 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:52:59.593 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:52:59.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:52:59.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:52:59.697 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:52:59.726 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:00.202 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T12:53:00.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:00 smithi027 ceph-mon[39795]: osdmap e222: 12 total, 12 up, 11 in 2024-05-15T12:53:00.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:00 smithi027 ceph-mon[39795]: pgmap v622: 17 pgs: 11 active+clean, 6 creating+peering; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:53:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:00 smithi120 ceph-mon[43961]: osdmap e222: 12 total, 12 up, 11 in 2024-05-15T12:53:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:00 smithi120 ceph-mon[43961]: pgmap v622: 17 pgs: 11 active+clean, 6 creating+peering; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:53:00.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:00 smithi177 ceph-mon[44303]: osdmap e222: 12 total, 12 up, 11 in 2024-05-15T12:53:00.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:00 smithi177 ceph-mon[44303]: pgmap v622: 17 pgs: 11 active+clean, 6 creating+peering; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:53:01.578 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:53:01.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:01.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:53:01.683 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:53:01.715 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:01.736 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:53:01.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:01.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:53:01.840 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:53:01.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:01 smithi027 ceph-mon[39795]: pgmap v623: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:53:01.871 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:02.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:01 smithi120 ceph-mon[43961]: pgmap v623: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:53:02.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:01 smithi177 ceph-mon[44303]: pgmap v623: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:53:03.122 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:53:03.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:03.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:53:03.227 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:53:03.259 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:03.280 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:53:03.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:03.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:53:03.384 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:53:03.417 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:03.457 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:53:03.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:03.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:53:03.563 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:53:03.579 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:53:03.595 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:03.598 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:53:03.615 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:53:03.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:03.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:53:03.619 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.0 to 1 (10000) 2024-05-15T12:53:03.619 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:53:03.640 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:53:03.661 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:53:03.682 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:53:03.702 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:53:03.711 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:03 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/283445922' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:53:03.711 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:03 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1205846973' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T12:53:03.720 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:53:03.724 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:53:03.745 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:53:03.752 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:03.765 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:53:03.773 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:53:03.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:03.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:53:03.786 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:53:03.806 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:53:03.807 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:53:03.807 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:53:03.808 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:53:03.808 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:53:03.808 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:53:03.808 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:53:03.808 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:53:03.808 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:53:03.808 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:53:03.808 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:53:03.808 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:53:03.808 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:53:03.877 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:53:03.905 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:03 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/283445922' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:53:03.906 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:03 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1205846973' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T12:53:03.907 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:03.929 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:53:03.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:03.932 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:53:04.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:03 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/283445922' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:53:04.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:03 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1205846973' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T12:53:04.032 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:53:04.061 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:04.080 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:53:04.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:04.083 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:53:04.184 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:53:04.216 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:04.238 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:53:04.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:04.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:53:04.342 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:53:04.371 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:04.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:04 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1205846973' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T12:53:04.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:04 smithi177 ceph-mon[44303]: osdmap e223: 12 total, 12 up, 11 in 2024-05-15T12:53:04.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:04 smithi177 ceph-mon[44303]: pgmap v625: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:53:05.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:04 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1205846973' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T12:53:05.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:04 smithi027 ceph-mon[39795]: osdmap e223: 12 total, 12 up, 11 in 2024-05-15T12:53:05.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:04 smithi027 ceph-mon[39795]: pgmap v625: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:53:05.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:04 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1205846973' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T12:53:05.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:04 smithi120 ceph-mon[43961]: osdmap e223: 12 total, 12 up, 11 in 2024-05-15T12:53:05.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:04 smithi120 ceph-mon[43961]: pgmap v625: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:53:05.970 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T12:53:05.970 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:53:05.991 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:05 smithi027 ceph-mon[39795]: pgmap v626: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:53:06.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:05 smithi120 ceph-mon[43961]: pgmap v626: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:53:06.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:05 smithi177 ceph-mon[44303]: pgmap v626: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:53:06.722 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-05-15T12:53:06.856 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:53:06.884 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:06 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2611355654' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:53:07.010 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-05-15T12:53:07.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:06 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2611355654' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:53:07.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:06 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2611355654' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:53:07.223 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:53:07.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:07.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:53:07.327 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:53:07.358 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:07.379 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:53:07.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:07.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:53:07.483 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:53:07.511 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:07.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:07 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2611355654' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T12:53:07.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:07 smithi027 ceph-mon[39795]: osdmap e224: 12 total, 12 up, 11 in 2024-05-15T12:53:07.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:07 smithi027 ceph-mon[39795]: pgmap v628: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:53:08.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:07 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2611355654' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T12:53:08.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:07 smithi120 ceph-mon[43961]: osdmap e224: 12 total, 12 up, 11 in 2024-05-15T12:53:08.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:07 smithi120 ceph-mon[43961]: pgmap v628: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:53:08.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:07 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2611355654' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T12:53:08.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:07 smithi177 ceph-mon[44303]: osdmap e224: 12 total, 12 up, 11 in 2024-05-15T12:53:08.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:07 smithi177 ceph-mon[44303]: pgmap v628: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-05-15T12:53:08.415 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:53:08.457 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:53:08.459 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 1 1 2024-05-15T12:53:08.462 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:53:08.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:08.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:53:08.566 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:53:08.599 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:08.620 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:53:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:08.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:53:08.724 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:53:08.756 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:08.775 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:53:08.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:08.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:53:08.803 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:53:08.878 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:53:08.907 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:08 smithi120 ceph-mon[43961]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T12:53:08.908 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:08 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T12:53:08.910 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:08.933 DEBUG:teuthology.orchestra.run.smithi177:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-05-15T12:53:08.934 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:53:08.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:08.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:53:08.937 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-05-15T12:53:08.937 INFO:teuthology.orchestra.run:waiting for 3000 2024-05-15T12:53:08.993 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:hints = 1 2024-05-15T12:53:08.993 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-05-15T12:53:08.993 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Object prefix: benchmark_data_smithi177_88104 2024-05-15T12:53:08.993 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T12:53:08.993 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 0 0 0 0 0 0 - 0 2024-05-15T12:53:09.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:08 smithi027 ceph-mon[39795]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T12:53:09.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:08 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T12:53:09.036 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:53:09.070 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:08 smithi177 ceph-mon[44303]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T12:53:09.071 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:08 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T12:53:09.073 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:09.104 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:53:09.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:09.106 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:53:09.206 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:53:09.245 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:53:09.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:09.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:53:09.712 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:53:09.767 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:09.993 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 1 15 3555 3540 221.238 221.25 0.00275428 0.00450563 2024-05-15T12:53:10.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:09 smithi027 ceph-mon[39795]: pgmap v629: 17 pgs: 17 active+clean; 11 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 1.3 MiB/s wr, 21 op/s 2024-05-15T12:53:10.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:09 smithi120 ceph-mon[43961]: pgmap v629: 17 pgs: 17 active+clean; 11 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 1.3 MiB/s wr, 21 op/s 2024-05-15T12:53:10.120 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:53:10.134 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:53:10.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:10.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:53:10.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:09 smithi177 ceph-mon[44303]: pgmap v629: 17 pgs: 17 active+clean; 11 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 1.3 MiB/s wr, 21 op/s 2024-05-15T12:53:10.240 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:53:10.266 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:10.285 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:53:10.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:10.288 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:53:10.389 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:53:10.416 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:10.972 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:53:10.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:10.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:53:10.993 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 2 16 6423 6407 200.197 179.188 0.00124324 0.00460797 2024-05-15T12:53:11.078 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:53:11.129 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:11.164 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:53:11.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:11.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:53:11.268 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:53:11.309 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:11.338 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:53:11.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:11.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:53:11.441 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:53:11.477 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:11.498 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:53:11.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:11.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:53:11.602 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:53:11.651 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:11.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:11 smithi027 ceph-mon[39795]: pgmap v630: 17 pgs: 17 active+clean; 134 MiB data, 1.4 GiB used, 982 GiB / 983 GiB avail; 17 MiB/s wr, 266 op/s 2024-05-15T12:53:11.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:11 smithi120 ceph-mon[43961]: pgmap v630: 17 pgs: 17 active+clean; 134 MiB data, 1.4 GiB used, 982 GiB / 983 GiB avail; 17 MiB/s wr, 266 op/s 2024-05-15T12:53:11.795 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:53:11.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:11.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:53:11.901 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:53:11.937 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:11.964 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:53:11.966 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:11 smithi177 ceph-mon[44303]: pgmap v630: 17 pgs: 17 active+clean; 134 MiB data, 1.4 GiB used, 982 GiB / 983 GiB avail; 17 MiB/s wr, 266 op/s 2024-05-15T12:53:11.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:11.967 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:53:11.993 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 3 16 9133 9117 189.918 169.375 0.00290149 0.00525846 2024-05-15T12:53:12.068 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:53:12.102 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:12.125 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:53:12.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:12.128 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:53:12.229 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:53:12.268 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:12.304 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:53:12.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:12.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:53:12.409 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:53:12.447 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:12.475 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:53:12.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:12.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:53:12.580 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:53:12.637 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:12.720 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:53:12.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:12.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:53:12.825 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:53:12.852 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:12.871 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:53:12.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:12.874 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:53:12.908 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:12 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3422100135' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:53:12.975 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:53:12.993 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 4 16 12113 12097 188.997 186.25 0.00820719 0.00528269 2024-05-15T12:53:13.026 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:13.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:12 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3422100135' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:53:13.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:12 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3422100135' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:53:13.385 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:53:13.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:13.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:53:13.490 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:53:13.522 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:13.543 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:53:13.543 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:53:13.544 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:53:13.544 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:53:13.544 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:53:13.544 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:53:13.544 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:53:13.544 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:53:13.544 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:53:13.544 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:53:13.544 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:53:13.544 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:53:13.544 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:53:13.545 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:53:13.545 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:53:13.545 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:53:13.545 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:53:13.545 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:53:13.545 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:53:13.545 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:53:13.545 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:53:13.545 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:53:13.545 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:53:13.545 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:53:13.547 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:53:13.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:13.551 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:53:13.652 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:53:13.703 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:13.994 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 5 16 14449 14433 180.396 146 0.00561708 0.00553648 2024-05-15T12:53:13.997 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:14 smithi027 ceph-mon[39795]: pgmap v631: 17 pgs: 17 active+clean; 490 MiB data, 2.3 GiB used, 981 GiB / 983 GiB avail; 49 MiB/s wr, 782 op/s 2024-05-15T12:53:14.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:14 smithi120 ceph-mon[43961]: pgmap v631: 17 pgs: 17 active+clean; 490 MiB data, 2.3 GiB used, 981 GiB / 983 GiB avail; 49 MiB/s wr, 782 op/s 2024-05-15T12:53:14.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:14 smithi177 ceph-mon[44303]: pgmap v631: 17 pgs: 17 active+clean; 490 MiB data, 2.3 GiB used, 981 GiB / 983 GiB avail; 49 MiB/s wr, 782 op/s 2024-05-15T12:53:14.994 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 6 16 17415 17399 181.223 185.375 0.00444287 0.00551309 2024-05-15T12:53:15.162 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:53:15.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:15.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:53:15.170 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:15 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:53:15.170 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:15 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:53:15.271 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:53:15.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:15 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:53:15.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:15 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:53:15.303 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:15 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:53:15.303 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:15 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:53:15.306 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:15.332 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:53:15.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:15.336 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:53:15.437 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:53:15.486 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:15.693 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:53:15.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:15.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:53:15.797 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:53:15.829 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:15.853 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:53:15.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:15.857 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:53:15.957 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:53:15.989 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:15.994 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 7 16 19978 19962 178.216 160.188 0.00408712 0.00560539 2024-05-15T12:53:16.011 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:53:16.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:16.015 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:53:16.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:16 smithi027 ceph-mon[39795]: osdmap e225: 12 total, 12 up, 11 in 2024-05-15T12:53:16.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:16 smithi027 ceph-mon[39795]: pgmap v633: 17 pgs: 17 active+clean; 822 MiB data, 3.4 GiB used, 980 GiB / 983 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-05-15T12:53:16.048 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T12:53:16.048 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T12:53:16.048 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T12:53:16.048 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T12:53:16.048 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.048 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:90800 2.4 2:2acfc200:::benchmark_data_smithi177_78104_object90799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-05-15T12:53:16.048 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:17.243564+0000", 2024-05-15T12:53:16.048 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 238.80963859900001, 2024-05-15T12:53:16.048 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2451092909999999, 2024-05-15T12:53:16.048 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:16.049 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:16.049 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:16.049 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:16.049 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:16.049 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 90800 2024-05-15T12:53:16.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.049 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:16.049 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.049 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:16.049 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.243564+0000", 2024-05-15T12:53:16.049 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.049 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.050 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:16.050 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.243564+0000", 2024-05-15T12:53:16.050 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.050 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:16.050 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.243566+0000", 2024-05-15T12:53:16.050 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.669e-06 2024-05-15T12:53:16.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.050 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:16.050 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.243631+0000", 2024-05-15T12:53:16.051 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5320999999999993e-05 2024-05-15T12:53:16.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.051 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:16.051 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.243649+0000", 2024-05-15T12:53:16.051 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7788e-05 2024-05-15T12:53:16.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.051 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:16.051 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.243659+0000", 2024-05-15T12:53:16.051 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9310000000000003e-06 2024-05-15T12:53:16.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.052 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.052 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.243727+0000", 2024-05-15T12:53:16.052 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8182999999999994e-05 2024-05-15T12:53:16.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.052 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-05-15T12:53:16.052 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.243733+0000", 2024-05-15T12:53:16.052 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9730000000000002e-06 2024-05-15T12:53:16.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.052 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.052 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.248328+0000", 2024-05-15T12:53:16.052 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0045947999999999996 2024-05-15T12:53:16.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.053 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:53:16.053 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.248339+0000", 2024-05-15T12:53:16.053 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1358e-05 2024-05-15T12:53:16.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.053 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.053 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.855990+0000", 2024-05-15T12:53:16.053 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.60765039899999995 2024-05-15T12:53:16.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.054 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:53:16.054 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.856005+0000", 2024-05-15T12:53:16.054 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5495e-05 2024-05-15T12:53:16.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.054 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.054 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.054 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.379629+0000", 2024-05-15T12:53:16.054 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.523623607 2024-05-15T12:53:16.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.054 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.054 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:16.054 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.379779+0000", 2024-05-15T12:53:16.055 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015046100000000001 2024-05-15T12:53:16.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.055 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:53:16.055 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.379942+0000", 2024-05-15T12:53:16.055 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000162962 2024-05-15T12:53:16.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.055 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.055 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.390721+0000", 2024-05-15T12:53:16.055 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010778914000000001 2024-05-15T12:53:16.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.056 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:16.056 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.390736+0000", 2024-05-15T12:53:16.056 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4922e-05 2024-05-15T12:53:16.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.056 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.056 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.390775+0000", 2024-05-15T12:53:16.056 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.913e-05 2024-05-15T12:53:16.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.056 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.056 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.488483+0000", 2024-05-15T12:53:16.056 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.097707752999999994 2024-05-15T12:53:16.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.057 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:16.057 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.488545+0000", 2024-05-15T12:53:16.057 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1901000000000002e-05 2024-05-15T12:53:16.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.057 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:16.057 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.488674+0000", 2024-05-15T12:53:16.057 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012872400000000001 2024-05-15T12:53:16.057 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.058 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:16.058 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.058 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.058 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:90822 2.4 2:28e8288f:::benchmark_data_smithi177_78104_object90821:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-05-15T12:53:16.058 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:17.272538+0000", 2024-05-15T12:53:16.058 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 238.78066485900001, 2024-05-15T12:53:16.058 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.216411146, 2024-05-15T12:53:16.058 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:16.058 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:16.058 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:16.058 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:16.059 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:16.059 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 90822 2024-05-15T12:53:16.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.059 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:16.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.059 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:16.059 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.272538+0000", 2024-05-15T12:53:16.059 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.059 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:16.059 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.272538+0000", 2024-05-15T12:53:16.059 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.060 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:16.060 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.272540+0000", 2024-05-15T12:53:16.060 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.984e-06 2024-05-15T12:53:16.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.060 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:16.060 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.272587+0000", 2024-05-15T12:53:16.060 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6814000000000003e-05 2024-05-15T12:53:16.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.060 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:16.060 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.272609+0000", 2024-05-15T12:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1979000000000001e-05 2024-05-15T12:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.272619+0000", 2024-05-15T12:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0539e-05 2024-05-15T12:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.272698+0000", 2024-05-15T12:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8716999999999999e-05 2024-05-15T12:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:53:16.062 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.272713+0000", 2024-05-15T12:53:16.062 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4662e-05 2024-05-15T12:53:16.062 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.062 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.062 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.062 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.856063+0000", 2024-05-15T12:53:16.062 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.58335017700000003 2024-05-15T12:53:16.062 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.062 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.062 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:53:16.062 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.856071+0000", 2024-05-15T12:53:16.062 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2130000000000008e-06 2024-05-15T12:53:16.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.063 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.063 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.383908+0000", 2024-05-15T12:53:16.063 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.527836432 2024-05-15T12:53:16.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.063 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:16.063 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.384046+0000", 2024-05-15T12:53:16.063 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013840900000000001 2024-05-15T12:53:16.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.063 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:53:16.064 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.384225+0000", 2024-05-15T12:53:16.064 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000179218 2024-05-15T12:53:16.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.064 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.064 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.064 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.394448+0000", 2024-05-15T12:53:16.064 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010223101 2024-05-15T12:53:16.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.064 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.064 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:16.064 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.394463+0000", 2024-05-15T12:53:16.064 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5175e-05 2024-05-15T12:53:16.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.065 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.065 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.065 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.400394+0000", 2024-05-15T12:53:16.065 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0059300280000000004 2024-05-15T12:53:16.065 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.065 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.065 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.065 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.488739+0000", 2024-05-15T12:53:16.065 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.088345173999999999 2024-05-15T12:53:16.065 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.065 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.066 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:16.066 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.488813+0000", 2024-05-15T12:53:16.066 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4237000000000001e-05 2024-05-15T12:53:16.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.066 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.066 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:16.066 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.488949+0000", 2024-05-15T12:53:16.066 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013628699999999999 2024-05-15T12:53:16.066 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.066 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:16.066 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.067 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.067 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119049 2.4 2:2045346f:::benchmark_data_smithi177_78104_object11916:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:53:16.067 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.627777+0000", 2024-05-15T12:53:16.067 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 200.42542611299999, 2024-05-15T12:53:16.067 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 15.003778057, 2024-05-15T12:53:16.067 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:16.067 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:16.067 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:16.067 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:16.067 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:16.067 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119049 2024-05-15T12:53:16.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.067 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:16.067 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.068 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:16.068 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.627777+0000", 2024-05-15T12:53:16.068 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.068 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.068 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.068 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:16.068 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.627777+0000", 2024-05-15T12:53:16.068 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.068 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.068 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.068 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:16.068 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.627777+0000", 2024-05-15T12:53:16.068 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0200000000000002e-07 2024-05-15T12:53:16.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.069 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:16.069 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.627780+0000", 2024-05-15T12:53:16.069 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0000000000000001e-06 2024-05-15T12:53:16.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.069 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:16.069 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.627788+0000", 2024-05-15T12:53:16.069 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9640000000000003e-06 2024-05-15T12:53:16.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.069 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:16.069 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.627794+0000", 2024-05-15T12:53:16.070 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4430000000000002e-06 2024-05-15T12:53:16.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.070 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.070 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.070 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.629174+0000", 2024-05-15T12:53:16.070 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001380364 2024-05-15T12:53:16.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.070 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.070 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:16.070 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.629297+0000", 2024-05-15T12:53:16.070 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000123041 2024-05-15T12:53:16.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.071 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.071 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:53:16.071 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.629415+0000", 2024-05-15T12:53:16.071 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000118213 2024-05-15T12:53:16.071 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.071 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.071 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.071 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640876+0000", 2024-05-15T12:53:16.071 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011460568000000001 2024-05-15T12:53:16.071 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.071 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.071 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:16.071 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640891+0000", 2024-05-15T12:53:16.071 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5194000000000001e-05 2024-05-15T12:53:16.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.072 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.072 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.072 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.646262+0000", 2024-05-15T12:53:16.072 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0053713809999999997 2024-05-15T12:53:16.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.072 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.072 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.072 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.631408+0000", 2024-05-15T12:53:16.072 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.985145664000001 2024-05-15T12:53:16.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.072 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.072 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:16.072 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.631460+0000", 2024-05-15T12:53:16.073 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1654999999999999e-05 2024-05-15T12:53:16.073 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.073 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.073 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:16.073 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.631555+0000", 2024-05-15T12:53:16.073 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5068000000000005e-05 2024-05-15T12:53:16.073 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.073 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:16.073 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.073 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.073 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.073 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119051 2.4 2:2ff5910d:::benchmark_data_smithi177_78104_object11918:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:53:16.073 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.631462+0000", 2024-05-15T12:53:16.073 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 200.421740427, 2024-05-15T12:53:16.074 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 15.000378482, 2024-05-15T12:53:16.074 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:16.074 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:16.074 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:16.074 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:16.074 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:16.074 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119051 2024-05-15T12:53:16.074 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.074 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:16.074 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.074 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:16.074 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.631462+0000", 2024-05-15T12:53:16.074 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.075 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.075 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.075 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:16.075 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.631462+0000", 2024-05-15T12:53:16.075 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.075 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.075 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.075 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:16.075 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.631464+0000", 2024-05-15T12:53:16.075 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.415e-06 2024-05-15T12:53:16.075 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.075 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.075 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:16.075 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.631470+0000", 2024-05-15T12:53:16.076 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2650000000000002e-06 2024-05-15T12:53:16.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.076 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.076 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:16.076 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.631480+0000", 2024-05-15T12:53:16.076 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0020000000000001e-05 2024-05-15T12:53:16.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.076 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.076 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:16.076 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.631487+0000", 2024-05-15T12:53:16.076 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8120000000000002e-06 2024-05-15T12:53:16.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.076 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.076 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.077 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.633185+0000", 2024-05-15T12:53:16.077 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016980470000000001 2024-05-15T12:53:16.077 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.077 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.077 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:16.077 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.633311+0000", 2024-05-15T12:53:16.077 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012555699999999999 2024-05-15T12:53:16.077 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.077 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.077 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:53:16.078 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.633434+0000", 2024-05-15T12:53:16.078 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000123177 2024-05-15T12:53:16.078 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.078 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.078 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:16.078 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640898+0000", 2024-05-15T12:53:16.078 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0074641630000000002 2024-05-15T12:53:16.078 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.078 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.078 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.078 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.641342+0000", 2024-05-15T12:53:16.079 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00044389700000000001 2024-05-15T12:53:16.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.079 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.079 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.079 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.651030+0000", 2024-05-15T12:53:16.079 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0096885540000000003 2024-05-15T12:53:16.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.079 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.079 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.079 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.631621+0000", 2024-05-15T12:53:16.079 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.980590125999999 2024-05-15T12:53:16.080 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.080 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.080 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:16.080 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.631691+0000", 2024-05-15T12:53:16.080 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0535000000000001e-05 2024-05-15T12:53:16.080 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.080 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.080 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:16.080 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.631841+0000", 2024-05-15T12:53:16.080 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014991399999999999 2024-05-15T12:53:16.080 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.081 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:16.081 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.081 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.081 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.081 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119049 2.4 2:2045346f:::benchmark_data_smithi177_78104_object11916:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:53:16.081 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.640268+0000", 2024-05-15T12:53:16.081 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 200.412934828, 2024-05-15T12:53:16.081 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.991202053, 2024-05-15T12:53:16.081 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:16.081 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "waiting for ondisk", 2024-05-15T12:53:16.081 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:16.081 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:16.081 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:16.081 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119049 2024-05-15T12:53:16.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.082 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:16.082 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.082 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:16.082 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640268+0000", 2024-05-15T12:53:16.082 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.082 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.082 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:16.082 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640268+0000", 2024-05-15T12:53:16.082 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.082 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.083 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:16.083 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640269+0000", 2024-05-15T12:53:16.083 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4999999999999998e-07 2024-05-15T12:53:16.083 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.083 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.083 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:16.083 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640271+0000", 2024-05-15T12:53:16.083 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6129999999999998e-06 2024-05-15T12:53:16.083 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.083 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.083 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:16.083 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640277+0000", 2024-05-15T12:53:16.083 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2260000000000002e-06 2024-05-15T12:53:16.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.084 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.084 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:16.084 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640283+0000", 2024-05-15T12:53:16.084 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2349999999999999e-06 2024-05-15T12:53:16.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.084 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.084 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.084 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640928+0000", 2024-05-15T12:53:16.084 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00064552499999999998 2024-05-15T12:53:16.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.084 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.084 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for ondisk", 2024-05-15T12:53:16.085 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640997+0000", 2024-05-15T12:53:16.085 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8717e-05 2024-05-15T12:53:16.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.085 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.085 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:16.085 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.631470+0000", 2024-05-15T12:53:16.085 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.990473287 2024-05-15T12:53:16.085 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.085 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:16.085 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.085 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.085 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119051 2.4 2:2ff5910d:::benchmark_data_smithi177_78104_object11918:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:53:16.086 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.640291+0000", 2024-05-15T12:53:16.086 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 200.412912396, 2024-05-15T12:53:16.086 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.991410879, 2024-05-15T12:53:16.086 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:16.086 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "waiting for ondisk", 2024-05-15T12:53:16.086 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:16.086 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:16.086 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:16.086 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119051 2024-05-15T12:53:16.086 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.086 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:16.086 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.087 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:16.087 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640291+0000", 2024-05-15T12:53:16.087 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.087 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:16.087 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640291+0000", 2024-05-15T12:53:16.087 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.087 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:16.087 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640291+0000", 2024-05-15T12:53:16.087 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0100000000000001e-07 2024-05-15T12:53:16.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.088 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.088 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:16.088 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640293+0000", 2024-05-15T12:53:16.088 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0700000000000001e-06 2024-05-15T12:53:16.088 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.088 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.088 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:16.088 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640298+0000", 2024-05-15T12:53:16.088 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.558e-06 2024-05-15T12:53:16.088 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.088 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.088 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:16.089 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640303+0000", 2024-05-15T12:53:16.089 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7890000000000002e-06 2024-05-15T12:53:16.089 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.089 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.089 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.089 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.641024+0000", 2024-05-15T12:53:16.089 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00072119299999999996 2024-05-15T12:53:16.089 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.089 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.089 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for ondisk", 2024-05-15T12:53:16.089 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.641086+0000", 2024-05-15T12:53:16.089 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1248999999999996e-05 2024-05-15T12:53:16.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.090 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.090 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:16.090 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.631701+0000", 2024-05-15T12:53:16.090 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.990615719000001 2024-05-15T12:53:16.090 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.090 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:16.090 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.090 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.090 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119069 2.4 2:2e7a257b:::benchmark_data_smithi177_78104_object11936:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:53:16.090 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.651569+0000", 2024-05-15T12:53:16.091 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 200.40163415699999, 2024-05-15T12:53:16.091 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.980443061999999, 2024-05-15T12:53:16.091 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:16.091 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:16.091 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:16.091 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:16.091 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:16.091 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119069 2024-05-15T12:53:16.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.091 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:16.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.091 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:16.091 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.651569+0000", 2024-05-15T12:53:16.091 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.092 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:16.092 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.651569+0000", 2024-05-15T12:53:16.092 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.092 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:16.092 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.651571+0000", 2024-05-15T12:53:16.092 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9069999999999999e-06 2024-05-15T12:53:16.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.093 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:16.093 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.651581+0000", 2024-05-15T12:53:16.093 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.011e-05 2024-05-15T12:53:16.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.093 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:16.093 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.651594+0000", 2024-05-15T12:53:16.093 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3159999999999999e-05 2024-05-15T12:53:16.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.093 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:16.093 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.651602+0000", 2024-05-15T12:53:16.094 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6490000000000008e-06 2024-05-15T12:53:16.094 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.094 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.094 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.094 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.651661+0000", 2024-05-15T12:53:16.094 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9160000000000003e-05 2024-05-15T12:53:16.094 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.094 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.094 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:16.094 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.651804+0000", 2024-05-15T12:53:16.094 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000142764 2024-05-15T12:53:16.094 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.095 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.095 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:53:16.095 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.651938+0000", 2024-05-15T12:53:16.095 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013402499999999999 2024-05-15T12:53:16.095 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.095 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.095 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:16.095 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.660292+0000", 2024-05-15T12:53:16.095 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0083543000000000003 2024-05-15T12:53:16.095 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.095 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.095 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.095 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.663547+0000", 2024-05-15T12:53:16.095 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003255553 2024-05-15T12:53:16.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.096 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.096 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.667485+0000", 2024-05-15T12:53:16.096 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039378340000000003 2024-05-15T12:53:16.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.096 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.096 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.631879+0000", 2024-05-15T12:53:16.096 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.964394083 2024-05-15T12:53:16.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.096 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:16.097 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.631914+0000", 2024-05-15T12:53:16.097 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4369000000000002e-05 2024-05-15T12:53:16.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.097 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:16.097 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.632012+0000", 2024-05-15T12:53:16.097 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8147999999999996e-05 2024-05-15T12:53:16.097 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.097 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:16.097 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.097 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119086 2.4 2:211895ac:::benchmark_data_smithi177_78104_object11953:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:53:16.097 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.667377+0000", 2024-05-15T12:53:16.098 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 200.385825842, 2024-05-15T12:53:16.098 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.967394268, 2024-05-15T12:53:16.098 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:16.098 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:16.098 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:16.098 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:16.098 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:16.098 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119086 2024-05-15T12:53:16.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.098 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:16.098 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.098 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:16.099 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.667377+0000", 2024-05-15T12:53:16.099 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.099 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:16.099 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.667377+0000", 2024-05-15T12:53:16.099 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.099 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:16.099 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.667378+0000", 2024-05-15T12:53:16.099 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.128e-06 2024-05-15T12:53:16.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.100 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.100 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:16.100 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.667386+0000", 2024-05-15T12:53:16.100 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4410000000000004e-06 2024-05-15T12:53:16.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.100 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.100 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:16.100 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.667396+0000", 2024-05-15T12:53:16.100 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0172e-05 2024-05-15T12:53:16.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.100 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.100 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:16.100 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.667406+0000", 2024-05-15T12:53:16.101 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7259999999999997e-06 2024-05-15T12:53:16.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.101 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.101 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.667522+0000", 2024-05-15T12:53:16.101 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000116402 2024-05-15T12:53:16.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.101 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:16.101 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.667651+0000", 2024-05-15T12:53:16.101 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012925 2024-05-15T12:53:16.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.102 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:53:16.102 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.667785+0000", 2024-05-15T12:53:16.102 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000133714 2024-05-15T12:53:16.102 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.102 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.102 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:16.102 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.672146+0000", 2024-05-15T12:53:16.102 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.004361452 2024-05-15T12:53:16.102 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.102 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.102 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.102 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.679409+0000", 2024-05-15T12:53:16.102 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0072626799999999997 2024-05-15T12:53:16.103 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.103 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.103 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.103 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.686927+0000", 2024-05-15T12:53:16.103 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0075182699999999996 2024-05-15T12:53:16.103 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.103 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.103 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.103 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.634563+0000", 2024-05-15T12:53:16.103 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.947635259 2024-05-15T12:53:16.103 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.103 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.103 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:16.103 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.634639+0000", 2024-05-15T12:53:16.104 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6280999999999997e-05 2024-05-15T12:53:16.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.104 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.104 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:16.104 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.634771+0000", 2024-05-15T12:53:16.104 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013249299999999999 2024-05-15T12:53:16.104 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.104 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:16.104 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.104 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.104 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119093 2.4 2:24c03606:::benchmark_data_smithi177_78104_object11960:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:53:16.104 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.675073+0000", 2024-05-15T12:53:16.105 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 200.37812950700001, 2024-05-15T12:53:16.105 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.963393344, 2024-05-15T12:53:16.105 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:16.105 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:16.105 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:16.105 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:16.105 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:16.105 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119093 2024-05-15T12:53:16.105 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.105 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:16.105 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.105 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:16.105 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.675073+0000", 2024-05-15T12:53:16.105 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.106 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.106 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:16.106 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.675073+0000", 2024-05-15T12:53:16.106 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.106 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.106 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:16.106 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.675080+0000", 2024-05-15T12:53:16.106 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6989999999999997e-06 2024-05-15T12:53:16.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.106 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.106 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:16.107 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.675086+0000", 2024-05-15T12:53:16.107 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4299999999999997e-06 2024-05-15T12:53:16.107 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.107 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.107 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:16.107 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.675100+0000", 2024-05-15T12:53:16.107 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4211e-05 2024-05-15T12:53:16.107 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.107 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.107 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:16.107 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.675120+0000", 2024-05-15T12:53:16.107 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9868999999999999e-05 2024-05-15T12:53:16.107 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.107 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.108 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.108 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.675208+0000", 2024-05-15T12:53:16.108 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8237000000000002e-05 2024-05-15T12:53:16.108 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.108 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.108 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:16.108 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.675333+0000", 2024-05-15T12:53:16.108 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000125072 2024-05-15T12:53:16.108 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.108 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.108 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:53:16.108 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.675454+0000", 2024-05-15T12:53:16.108 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000121144 2024-05-15T12:53:16.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.109 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.109 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:16.109 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.679006+0000", 2024-05-15T12:53:16.109 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003551528 2024-05-15T12:53:16.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.109 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.109 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.109 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.686976+0000", 2024-05-15T12:53:16.109 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0079701870000000001 2024-05-15T12:53:16.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.109 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.110 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.110 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.691978+0000", 2024-05-15T12:53:16.110 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050020200000000003 2024-05-15T12:53:16.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.110 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.110 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.110 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.638261+0000", 2024-05-15T12:53:16.110 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.946283504 2024-05-15T12:53:16.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.110 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.110 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:16.110 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.638326+0000", 2024-05-15T12:53:16.111 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5178000000000005e-05 2024-05-15T12:53:16.111 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.111 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.111 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:16.111 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.638467+0000", 2024-05-15T12:53:16.111 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014026500000000001 2024-05-15T12:53:16.111 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.111 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:16.111 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.111 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.111 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.111 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119105 2.4 2:2b391885:::benchmark_data_smithi177_78104_object11972:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:53:16.111 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.695763+0000", 2024-05-15T12:53:16.112 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 200.35743987999999, 2024-05-15T12:53:16.112 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.942991198, 2024-05-15T12:53:16.112 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:16.112 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:16.112 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:16.112 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:16.112 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:16.113 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119105 2024-05-15T12:53:16.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.113 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:16.113 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.113 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:16.113 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.695763+0000", 2024-05-15T12:53:16.113 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.113 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.113 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:16.113 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.695763+0000", 2024-05-15T12:53:16.113 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.114 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.114 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:16.114 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.695764+0000", 2024-05-15T12:53:16.114 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.158e-06 2024-05-15T12:53:16.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.114 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.114 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:16.114 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.695770+0000", 2024-05-15T12:53:16.114 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0939999999999996e-06 2024-05-15T12:53:16.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.114 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.115 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:16.115 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.695781+0000", 2024-05-15T12:53:16.115 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0526000000000001e-05 2024-05-15T12:53:16.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.115 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:16.115 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.695788+0000", 2024-05-15T12:53:16.115 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8950000000000001e-06 2024-05-15T12:53:16.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.115 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.115 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.695916+0000", 2024-05-15T12:53:16.115 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012823000000000001 2024-05-15T12:53:16.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.116 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.116 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:16.116 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.696037+0000", 2024-05-15T12:53:16.116 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000121317 2024-05-15T12:53:16.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.116 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.116 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:53:16.116 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.696164+0000", 2024-05-15T12:53:16.116 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000126499 2024-05-15T12:53:16.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.116 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.116 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.116 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.700131+0000", 2024-05-15T12:53:16.117 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039675190000000001 2024-05-15T12:53:16.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.117 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:16.117 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.700158+0000", 2024-05-15T12:53:16.117 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6407999999999999e-05 2024-05-15T12:53:16.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.117 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.117 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.703940+0000", 2024-05-15T12:53:16.117 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003782058 2024-05-15T12:53:16.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.118 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.118 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.638526+0000", 2024-05-15T12:53:16.118 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.934585805999999 2024-05-15T12:53:16.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.118 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.118 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:16.118 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.638603+0000", 2024-05-15T12:53:16.118 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7503999999999995e-05 2024-05-15T12:53:16.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.118 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.118 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:16.118 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.638754+0000", 2024-05-15T12:53:16.119 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015118400000000001 2024-05-15T12:53:16.119 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.119 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:16.119 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.119 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.119 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119126 2.4 2:2fb2b24a:::benchmark_data_smithi177_78104_object11993:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:53:16.119 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.722151+0000", 2024-05-15T12:53:16.119 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 200.33105231799999, 2024-05-15T12:53:16.119 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.919300895999999, 2024-05-15T12:53:16.119 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:16.119 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:16.119 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:16.119 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:16.120 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:16.120 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119126 2024-05-15T12:53:16.120 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.120 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:16.120 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.120 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:16.120 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.722151+0000", 2024-05-15T12:53:16.120 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.120 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.120 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.120 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:16.120 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.722151+0000", 2024-05-15T12:53:16.120 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.120 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.121 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.121 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:16.121 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.722152+0000", 2024-05-15T12:53:16.121 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.201e-06 2024-05-15T12:53:16.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.121 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.121 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:16.121 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.722160+0000", 2024-05-15T12:53:16.121 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.388e-06 2024-05-15T12:53:16.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.121 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.121 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:16.121 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.722175+0000", 2024-05-15T12:53:16.121 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4363e-05 2024-05-15T12:53:16.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.122 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.122 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:16.122 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.722182+0000", 2024-05-15T12:53:16.122 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6799999999999993e-06 2024-05-15T12:53:16.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.122 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.122 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.122 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.722300+0000", 2024-05-15T12:53:16.122 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011772700000000001 2024-05-15T12:53:16.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.122 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.123 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:16.123 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.722462+0000", 2024-05-15T12:53:16.123 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016200399999999999 2024-05-15T12:53:16.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.123 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.123 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:53:16.123 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.722587+0000", 2024-05-15T12:53:16.123 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012485299999999999 2024-05-15T12:53:16.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.123 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.123 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.123 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.732412+0000", 2024-05-15T12:53:16.123 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0098255870000000002 2024-05-15T12:53:16.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.124 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.124 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:16.124 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.733985+0000", 2024-05-15T12:53:16.124 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001572389 2024-05-15T12:53:16.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.124 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.124 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.124 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.736745+0000", 2024-05-15T12:53:16.124 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0027598760000000001 2024-05-15T12:53:16.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.124 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.124 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.124 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.641257+0000", 2024-05-15T12:53:16.125 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.904512132000001 2024-05-15T12:53:16.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.125 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:16.125 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.641323+0000", 2024-05-15T12:53:16.125 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6563999999999995e-05 2024-05-15T12:53:16.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.125 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:16.125 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.641452+0000", 2024-05-15T12:53:16.125 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012813200000000001 2024-05-15T12:53:16.125 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.125 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:16.125 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.126 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.126 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119147 2.4 2:244e395e:::benchmark_data_smithi177_78104_object12014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:53:16.126 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.748010+0000", 2024-05-15T12:53:16.126 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 200.305193116, 2024-05-15T12:53:16.126 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.893615745, 2024-05-15T12:53:16.126 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:16.126 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:16.126 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:16.126 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:16.126 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:16.126 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119147 2024-05-15T12:53:16.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.127 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:16.127 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.127 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:16.127 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.748010+0000", 2024-05-15T12:53:16.127 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.127 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.127 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:16.127 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.748010+0000", 2024-05-15T12:53:16.127 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.127 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.128 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:16.128 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.748011+0000", 2024-05-15T12:53:16.128 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.609e-06 2024-05-15T12:53:16.128 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.128 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.128 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:16.128 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.748019+0000", 2024-05-15T12:53:16.128 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9529999999999999e-06 2024-05-15T12:53:16.128 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.128 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.128 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:16.128 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.748029+0000", 2024-05-15T12:53:16.128 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2299999999999997e-06 2024-05-15T12:53:16.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.129 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.129 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:16.129 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.748036+0000", 2024-05-15T12:53:16.129 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7430000000000002e-06 2024-05-15T12:53:16.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.129 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.129 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.129 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.748100+0000", 2024-05-15T12:53:16.129 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3907999999999999e-05 2024-05-15T12:53:16.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.129 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.129 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:16.130 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.748274+0000", 2024-05-15T12:53:16.130 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000174112 2024-05-15T12:53:16.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.130 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.130 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:53:16.130 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.748382+0000", 2024-05-15T12:53:16.130 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000107799 2024-05-15T12:53:16.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.130 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.130 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.130 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.752266+0000", 2024-05-15T12:53:16.130 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038838089999999998 2024-05-15T12:53:16.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.131 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.131 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:16.131 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.752326+0000", 2024-05-15T12:53:16.131 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0316999999999999e-05 2024-05-15T12:53:16.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.131 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.131 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.131 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.754303+0000", 2024-05-15T12:53:16.131 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001976233 2024-05-15T12:53:16.132 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.132 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.132 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.132 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.641493+0000", 2024-05-15T12:53:16.132 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.887190867999999 2024-05-15T12:53:16.132 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.132 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.132 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:16.133 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.641547+0000", 2024-05-15T12:53:16.133 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3338000000000003e-05 2024-05-15T12:53:16.133 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.133 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.133 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:16.133 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.641626+0000", 2024-05-15T12:53:16.133 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8825999999999994e-05 2024-05-15T12:53:16.133 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.133 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:16.133 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.133 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.133 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.133 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119150 2.4 2:27ad4cd4:::benchmark_data_smithi177_78104_object12017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:53:16.133 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.755494+0000", 2024-05-15T12:53:16.134 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 200.29770874900001, 2024-05-15T12:53:16.134 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.890362140000001, 2024-05-15T12:53:16.134 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:16.134 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:16.134 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:16.134 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:16.134 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:16.134 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119150 2024-05-15T12:53:16.134 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.134 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:16.134 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.134 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:16.134 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.755494+0000", 2024-05-15T12:53:16.135 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.135 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:16.135 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.755494+0000", 2024-05-15T12:53:16.135 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.135 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:16.135 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.755495+0000", 2024-05-15T12:53:16.135 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.066e-06 2024-05-15T12:53:16.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.135 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:16.136 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.755500+0000", 2024-05-15T12:53:16.136 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8640000000000004e-06 2024-05-15T12:53:16.136 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.136 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.136 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:16.136 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.755519+0000", 2024-05-15T12:53:16.136 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8470000000000001e-05 2024-05-15T12:53:16.136 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.136 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.136 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:16.136 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.755527+0000", 2024-05-15T12:53:16.136 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7909999999999993e-06 2024-05-15T12:53:16.136 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.136 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.137 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.137 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.755574+0000", 2024-05-15T12:53:16.137 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6794e-05 2024-05-15T12:53:16.137 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.137 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.137 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:16.137 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.755726+0000", 2024-05-15T12:53:16.137 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000151772 2024-05-15T12:53:16.137 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.137 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.137 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:53:16.137 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.755810+0000", 2024-05-15T12:53:16.137 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4252999999999994e-05 2024-05-15T12:53:16.138 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.138 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.138 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.138 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.759504+0000", 2024-05-15T12:53:16.138 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003693446 2024-05-15T12:53:16.138 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.138 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.138 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:16.138 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.759573+0000", 2024-05-15T12:53:16.138 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9017999999999995e-05 2024-05-15T12:53:16.138 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.138 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.138 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.139 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.761070+0000", 2024-05-15T12:53:16.139 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0014970020000000001 2024-05-15T12:53:16.139 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.139 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.139 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.139 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.645664+0000", 2024-05-15T12:53:16.139 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.884594117000001 2024-05-15T12:53:16.139 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.139 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.139 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:16.139 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.645723+0000", 2024-05-15T12:53:16.139 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8897000000000002e-05 2024-05-15T12:53:16.139 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.139 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.140 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:16.140 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.645856+0000", 2024-05-15T12:53:16.140 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013365 2024-05-15T12:53:16.140 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.140 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:16.140 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.140 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.140 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.140 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119168 2.4 2:2f7ee5d7:::benchmark_data_smithi177_78104_object12035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:53:16.140 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.789664+0000", 2024-05-15T12:53:16.140 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 200.26353876900001, 2024-05-15T12:53:16.140 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.856372735000001, 2024-05-15T12:53:16.140 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:16.140 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:16.141 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:16.141 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:16.141 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:16.141 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119168 2024-05-15T12:53:16.141 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.141 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:16.141 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.141 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:16.141 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.789664+0000", 2024-05-15T12:53:16.141 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.141 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.141 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.141 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:16.141 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.789664+0000", 2024-05-15T12:53:16.142 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.142 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.142 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.142 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:16.142 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.789666+0000", 2024-05-15T12:53:16.142 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6479999999999999e-06 2024-05-15T12:53:16.142 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.142 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.142 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:16.142 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.789673+0000", 2024-05-15T12:53:16.142 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2049999999999996e-06 2024-05-15T12:53:16.142 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.142 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.143 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:16.143 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.789686+0000", 2024-05-15T12:53:16.143 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3461999999999999e-05 2024-05-15T12:53:16.143 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.143 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.143 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:16.143 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.789694+0000", 2024-05-15T12:53:16.143 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5320000000000001e-06 2024-05-15T12:53:16.143 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.143 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.143 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.143 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.789776+0000", 2024-05-15T12:53:16.143 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1644999999999995e-05 2024-05-15T12:53:16.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.144 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.144 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:16.144 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.789886+0000", 2024-05-15T12:53:16.144 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000110123 2024-05-15T12:53:16.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.144 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.144 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:53:16.144 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.789975+0000", 2024-05-15T12:53:16.144 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8914999999999998e-05 2024-05-15T12:53:16.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.144 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.145 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:16.145 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.793355+0000", 2024-05-15T12:53:16.145 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003380508 2024-05-15T12:53:16.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.145 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.145 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.145 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.793536+0000", 2024-05-15T12:53:16.145 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00018034799999999999 2024-05-15T12:53:16.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.145 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.145 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.145 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.794419+0000", 2024-05-15T12:53:16.145 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00088304699999999998 2024-05-15T12:53:16.146 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.146 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.146 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.146 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.645905+0000", 2024-05-15T12:53:16.146 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.851486226 2024-05-15T12:53:16.146 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.146 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.146 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:16.146 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.645942+0000", 2024-05-15T12:53:16.146 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6860000000000003e-05 2024-05-15T12:53:16.146 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.146 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.146 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:16.147 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.646037+0000", 2024-05-15T12:53:16.147 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5216000000000005e-05 2024-05-15T12:53:16.147 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.147 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:16.147 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.147 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.147 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.147 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119181 2.4 2:297c213d:::benchmark_data_smithi177_78104_object12048:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:53:16.147 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.807440+0000", 2024-05-15T12:53:16.147 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 200.24576249500001, 2024-05-15T12:53:16.147 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.848339640000001, 2024-05-15T12:53:16.147 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:16.147 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:16.147 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:16.148 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:16.148 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:16.148 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119181 2024-05-15T12:53:16.148 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.148 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:16.148 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.148 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:16.148 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.807440+0000", 2024-05-15T12:53:16.148 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.148 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.148 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.148 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:16.148 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.807440+0000", 2024-05-15T12:53:16.148 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.149 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.149 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:16.149 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.807442+0000", 2024-05-15T12:53:16.149 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5409999999999999e-06 2024-05-15T12:53:16.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.149 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.149 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:16.149 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.807448+0000", 2024-05-15T12:53:16.149 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.942e-06 2024-05-15T12:53:16.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.149 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.149 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:16.149 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.807460+0000", 2024-05-15T12:53:16.150 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1881e-05 2024-05-15T12:53:16.150 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.150 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.150 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:16.150 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.807467+0000", 2024-05-15T12:53:16.150 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.627e-06 2024-05-15T12:53:16.150 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.150 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.150 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.150 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.807533+0000", 2024-05-15T12:53:16.150 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5537999999999995e-05 2024-05-15T12:53:16.150 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.150 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.151 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:16.151 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.807685+0000", 2024-05-15T12:53:16.151 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000152177 2024-05-15T12:53:16.151 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.151 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.151 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:53:16.151 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.807817+0000", 2024-05-15T12:53:16.151 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013225800000000001 2024-05-15T12:53:16.151 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.151 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.151 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:16.152 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.813241+0000", 2024-05-15T12:53:16.152 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0054233650000000003 2024-05-15T12:53:16.152 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.152 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.152 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.152 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.815003+0000", 2024-05-15T12:53:16.152 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00176184 2024-05-15T12:53:16.152 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.152 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.152 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.152 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.815055+0000", 2024-05-15T12:53:16.152 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2493e-05 2024-05-15T12:53:16.153 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.153 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.153 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.153 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.655595+0000", 2024-05-15T12:53:16.153 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.840539567 2024-05-15T12:53:16.153 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.153 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.153 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:16.153 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.655658+0000", 2024-05-15T12:53:16.153 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3590000000000006e-05 2024-05-15T12:53:16.154 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.154 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.154 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:16.154 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.655780+0000", 2024-05-15T12:53:16.154 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000121821 2024-05-15T12:53:16.154 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.154 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:16.154 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.154 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.154 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.154 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119196 2.4 2:2646c09a:::benchmark_data_smithi177_78104_object12063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:53:16.154 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.832154+0000", 2024-05-15T12:53:16.155 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 200.221048483, 2024-05-15T12:53:16.155 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.823821685, 2024-05-15T12:53:16.155 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:16.155 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:16.155 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:16.155 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:16.155 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:16.155 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119196 2024-05-15T12:53:16.155 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.155 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:16.155 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.156 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:16.156 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.832154+0000", 2024-05-15T12:53:16.156 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.156 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.156 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.156 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:16.156 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.832154+0000", 2024-05-15T12:53:16.156 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.156 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.156 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.156 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:16.156 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.832156+0000", 2024-05-15T12:53:16.156 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9609999999999999e-06 2024-05-15T12:53:16.157 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.157 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.157 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:16.157 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.832165+0000", 2024-05-15T12:53:16.157 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8310000000000007e-06 2024-05-15T12:53:16.157 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.157 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.157 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:16.157 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.832187+0000", 2024-05-15T12:53:16.157 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2093e-05 2024-05-15T12:53:16.158 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.158 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.158 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:16.158 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.832198+0000", 2024-05-15T12:53:16.158 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0516000000000001e-05 2024-05-15T12:53:16.158 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.158 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.158 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.158 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.832258+0000", 2024-05-15T12:53:16.159 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0010000000000003e-05 2024-05-15T12:53:16.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.159 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.159 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:16.159 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.832426+0000", 2024-05-15T12:53:16.159 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016802599999999999 2024-05-15T12:53:16.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.160 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.160 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:53:16.160 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.832593+0000", 2024-05-15T12:53:16.160 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000167071 2024-05-15T12:53:16.160 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.160 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.161 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:16.161 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.837325+0000", 2024-05-15T12:53:16.161 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0047316420000000003 2024-05-15T12:53:16.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.161 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.161 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.162 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.839375+0000", 2024-05-15T12:53:16.162 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020508560000000002 2024-05-15T12:53:16.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.162 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.162 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.843832+0000", 2024-05-15T12:53:16.162 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0044566340000000001 2024-05-15T12:53:16.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.162 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.163 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.655823+0000", 2024-05-15T12:53:16.163 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.811990489999999 2024-05-15T12:53:16.163 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.163 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.163 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:16.163 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.655872+0000", 2024-05-15T12:53:16.163 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9601999999999997e-05 2024-05-15T12:53:16.163 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.163 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.163 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:16.163 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.655976+0000", 2024-05-15T12:53:16.163 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000103953 2024-05-15T12:53:16.163 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.163 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:16.164 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.164 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.164 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119223 2.4 2:2892d1d8:::benchmark_data_smithi177_78104_object12090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:53:16.164 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.884405+0000", 2024-05-15T12:53:16.164 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 200.16879818500001, 2024-05-15T12:53:16.164 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.771771241, 2024-05-15T12:53:16.164 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:16.164 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:16.164 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:16.164 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:16.164 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:16.164 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119223 2024-05-15T12:53:16.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.884405+0000", 2024-05-15T12:53:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.884405+0000", 2024-05-15T12:53:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.884406+0000", 2024-05-15T12:53:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.469e-06 2024-05-15T12:53:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.884413+0000", 2024-05-15T12:53:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3890000000000002e-06 2024-05-15T12:53:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.884430+0000", 2024-05-15T12:53:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7638e-05 2024-05-15T12:53:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.884439+0000", 2024-05-15T12:53:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5699999999999993e-06 2024-05-15T12:53:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.884495+0000", 2024-05-15T12:53:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6388999999999998e-05 2024-05-15T12:53:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:16.167 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.884622+0000", 2024-05-15T12:53:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012727999999999999 2024-05-15T12:53:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:53:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.884715+0000", 2024-05-15T12:53:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2476e-05 2024-05-15T12:53:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.887826+0000", 2024-05-15T12:53:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031109029999999999 2024-05-15T12:53:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.168 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.888005+0000", 2024-05-15T12:53:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017930100000000001 2024-05-15T12:53:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.888537+0000", 2024-05-15T12:53:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00053228399999999995 2024-05-15T12:53:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.656015+0000", 2024-05-15T12:53:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.767477478 2024-05-15T12:53:16.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.656064+0000", 2024-05-15T12:53:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9458e-05 2024-05-15T12:53:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.656176+0000", 2024-05-15T12:53:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000111606 2024-05-15T12:53:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119236 2.4 2:222c6e22:::benchmark_data_smithi177_78104_object12103:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:53:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.915558+0000", 2024-05-15T12:53:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 200.13764501, 2024-05-15T12:53:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.740808504, 2024-05-15T12:53:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119236 2024-05-15T12:53:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:16.171 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.915558+0000", 2024-05-15T12:53:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.915558+0000", 2024-05-15T12:53:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.915559+0000", 2024-05-15T12:53:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4789999999999999e-06 2024-05-15T12:53:16.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.915565+0000", 2024-05-15T12:53:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6930000000000004e-06 2024-05-15T12:53:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.915578+0000", 2024-05-15T12:53:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3244000000000001e-05 2024-05-15T12:53:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.915586+0000", 2024-05-15T12:53:16.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4359999999999996e-06 2024-05-15T12:53:16.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.174 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.174 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.915637+0000", 2024-05-15T12:53:16.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1054999999999998e-05 2024-05-15T12:53:16.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.174 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:16.174 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.915778+0000", 2024-05-15T12:53:16.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014136800000000001 2024-05-15T12:53:16.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.174 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:53:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.915903+0000", 2024-05-15T12:53:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000124354 2024-05-15T12:53:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.919310+0000", 2024-05-15T12:53:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003407392 2024-05-15T12:53:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.919841+0000", 2024-05-15T12:53:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00053115300000000003 2024-05-15T12:53:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.924424+0000", 2024-05-15T12:53:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0045829829999999997 2024-05-15T12:53:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.656216+0000", 2024-05-15T12:53:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.731791498 2024-05-15T12:53:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.656264+0000", 2024-05-15T12:53:16.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8350000000000003e-05 2024-05-15T12:53:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.656366+0000", 2024-05-15T12:53:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000102499 2024-05-15T12:53:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119240 2.4 2:2ed0c2c5:::benchmark_data_smithi177_78104_object12107:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:53:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.924347+0000", 2024-05-15T12:53:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 200.128855872, 2024-05-15T12:53:16.177 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.732209246, 2024-05-15T12:53:16.178 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:16.178 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:16.178 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:16.178 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:16.178 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:16.178 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119240 2024-05-15T12:53:16.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.178 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:16.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.178 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.924347+0000", 2024-05-15T12:53:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.924347+0000", 2024-05-15T12:53:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.924348+0000", 2024-05-15T12:53:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4100000000000001e-06 2024-05-15T12:53:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.924356+0000", 2024-05-15T12:53:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3780000000000003e-06 2024-05-15T12:53:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.924371+0000", 2024-05-15T12:53:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4783000000000001e-05 2024-05-15T12:53:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.924381+0000", 2024-05-15T12:53:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0166999999999999e-05 2024-05-15T12:53:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.924454+0000", 2024-05-15T12:53:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3535999999999995e-05 2024-05-15T12:53:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.924553+0000", 2024-05-15T12:53:16.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8793000000000001e-05 2024-05-15T12:53:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:53:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.924634+0000", 2024-05-15T12:53:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0436e-05 2024-05-15T12:53:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.928341+0000", 2024-05-15T12:53:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0037073509999999998 2024-05-15T12:53:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.182 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.928376+0000", 2024-05-15T12:53:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4686e-05 2024-05-15T12:53:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.928434+0000", 2024-05-15T12:53:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7958999999999999e-05 2024-05-15T12:53:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.656405+0000", 2024-05-15T12:53:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.727971643 2024-05-15T12:53:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.656454+0000", 2024-05-15T12:53:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9110000000000002e-05 2024-05-15T12:53:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.656556+0000", 2024-05-15T12:53:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000101994 2024-05-15T12:53:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.184 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119259 2.4 2:21f9b17e:::benchmark_data_smithi177_78104_object12126:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:53:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:56.045969+0000", 2024-05-15T12:53:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 200.00723377899999, 2024-05-15T12:53:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.610775774, 2024-05-15T12:53:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119259 2024-05-15T12:53:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.185 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:56.045969+0000", 2024-05-15T12:53:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:56.045969+0000", 2024-05-15T12:53:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:56.045970+0000", 2024-05-15T12:53:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.125e-06 2024-05-15T12:53:16.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:56.045976+0000", 2024-05-15T12:53:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5380000000000002e-06 2024-05-15T12:53:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:56.045989+0000", 2024-05-15T12:53:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3305999999999999e-05 2024-05-15T12:53:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:56.045996+0000", 2024-05-15T12:53:16.187 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1359999999999999e-06 2024-05-15T12:53:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:56.046051+0000", 2024-05-15T12:53:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4388999999999997e-05 2024-05-15T12:53:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:56.046223+0000", 2024-05-15T12:53:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017190599999999999 2024-05-15T12:53:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:53:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:56.046387+0000", 2024-05-15T12:53:16.188 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000164451 2024-05-15T12:53:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:56.050121+0000", 2024-05-15T12:53:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003734163 2024-05-15T12:53:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:56.050203+0000", 2024-05-15T12:53:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2077999999999996e-05 2024-05-15T12:53:16.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:56.050256+0000", 2024-05-15T12:53:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2388000000000001e-05 2024-05-15T12:53:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.656595+0000", 2024-05-15T12:53:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.606338887 2024-05-15T12:53:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.656644+0000", 2024-05-15T12:53:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9110999999999998e-05 2024-05-15T12:53:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.656745+0000", 2024-05-15T12:53:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000101296 2024-05-15T12:53:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:16.191 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:16.191 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T12:53:16.192 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:53:16.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:16 smithi177 ceph-mon[44303]: osdmap e225: 12 total, 12 up, 11 in 2024-05-15T12:53:16.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:16 smithi177 ceph-mon[44303]: pgmap v633: 17 pgs: 17 active+clean; 822 MiB data, 3.4 GiB used, 980 GiB / 983 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-05-15T12:53:16.238 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:16.263 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:53:16.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:16 smithi120 ceph-mon[43961]: osdmap e225: 12 total, 12 up, 11 in 2024-05-15T12:53:16.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:16 smithi120 ceph-mon[43961]: pgmap v633: 17 pgs: 17 active+clean; 822 MiB data, 3.4 GiB used, 980 GiB / 983 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-05-15T12:53:16.450 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:53:16.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:16.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:53:16.556 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:53:16.586 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:16.608 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:53:16.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:16.611 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:53:16.712 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:53:16.765 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:16.978 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:53:16.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:16.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:53:16.994 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 8 15 22869 22854 178.531 180.75 0.00674399 0.00559557 2024-05-15T12:53:17.054 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T12:53:17.088 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:53:17.120 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:17.143 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:53:17.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:17.147 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:53:17.248 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:53:17.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:17 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/455273313' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:53:17.288 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:17 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/455273313' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:53:17.291 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:17.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:17 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/455273313' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:53:17.749 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-05-15T12:53:17.955 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:53:17.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:17.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:53:17.994 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 9 16 25689 25673 178.269 176.188 0.00650722 0.00560536 2024-05-15T12:53:18.062 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:53:18.106 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:18.144 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:53:18.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:18.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:53:18.248 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:53:18.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:18 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:53:18.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:18 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/455273313' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:53:18.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:18 smithi027 ceph-mon[39795]: osdmap e226: 12 total, 12 up, 11 in 2024-05-15T12:53:18.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:18 smithi027 ceph-mon[39795]: pgmap v635: 17 pgs: 17 active+clean; 1.1 GiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 135 MiB/s wr, 2.16k op/s 2024-05-15T12:53:18.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:18 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:53:18.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:18 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/455273313' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:53:18.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:18 smithi120 ceph-mon[43961]: osdmap e226: 12 total, 12 up, 11 in 2024-05-15T12:53:18.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:18 smithi120 ceph-mon[43961]: pgmap v635: 17 pgs: 17 active+clean; 1.1 GiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 135 MiB/s wr, 2.16k op/s 2024-05-15T12:53:18.290 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:18.318 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:53:18.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:18.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:53:18.424 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:53:18.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:18 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:53:18.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:18 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/455273313' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:53:18.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:18 smithi177 ceph-mon[44303]: osdmap e226: 12 total, 12 up, 11 in 2024-05-15T12:53:18.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:18 smithi177 ceph-mon[44303]: pgmap v635: 17 pgs: 17 active+clean; 1.1 GiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 135 MiB/s wr, 2.16k op/s 2024-05-15T12:53:18.456 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:18.479 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:53:18.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:18.482 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:53:18.582 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:53:18.627 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:18.665 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:53:18.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:18.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:53:18.771 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:53:18.819 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:18.844 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:53:18.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:18.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:53:18.950 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:53:18.994 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 10 16 28291 28275 176.703 162.625 0.00766601 0.00565466 2024-05-15T12:53:18.996 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:19.030 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:53:19.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:19.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:53:19.106 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:53:19.109 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:53:19.134 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:53:19.172 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:19.212 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:53:19.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:19.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:53:19.316 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:53:19.349 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:19.373 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:53:19.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:19.376 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:53:19.477 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:53:19.540 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:19.994 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 11 15 30694 30679 174.297 150.25 0.00594121 0.00573364 2024-05-15T12:53:20.065 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.1 to 1 (10000) 2024-05-15T12:53:20.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:20 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:53:20.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:20 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:53:20.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:20 smithi177 ceph-mon[44303]: osdmap e227: 12 total, 12 up, 11 in 2024-05-15T12:53:20.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:20 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1410881007' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T12:53:20.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:20 smithi177 ceph-mon[44303]: pgmap v637: 17 pgs: 17 active+clean; 1.4 GiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 161 MiB/s wr, 2.57k op/s 2024-05-15T12:53:20.453 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T12:53:20.457 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:20 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:53:20.457 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:20 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:53:20.457 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:20 smithi027 ceph-mon[39795]: osdmap e227: 12 total, 12 up, 11 in 2024-05-15T12:53:20.457 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:20 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1410881007' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T12:53:20.457 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:20 smithi027 ceph-mon[39795]: pgmap v637: 17 pgs: 17 active+clean; 1.4 GiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 161 MiB/s wr, 2.57k op/s 2024-05-15T12:53:20.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:20 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:53:20.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:20 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:53:20.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:20 smithi120 ceph-mon[43961]: osdmap e227: 12 total, 12 up, 11 in 2024-05-15T12:53:20.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:20 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1410881007' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T12:53:20.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:20 smithi120 ceph-mon[43961]: pgmap v637: 17 pgs: 17 active+clean; 1.4 GiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 161 MiB/s wr, 2.57k op/s 2024-05-15T12:53:20.994 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 12 16 33185 33169 172.74 155.625 0.00454862 0.00578451 2024-05-15T12:53:21.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:21 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1410881007' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T12:53:21.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:21 smithi177 ceph-mon[44303]: osdmap e228: 12 total, 12 up, 11 in 2024-05-15T12:53:21.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:21 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1410881007' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T12:53:21.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:21 smithi027 ceph-mon[39795]: osdmap e228: 12 total, 12 up, 11 in 2024-05-15T12:53:21.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:21 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1410881007' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T12:53:21.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:21 smithi120 ceph-mon[43961]: osdmap e228: 12 total, 12 up, 11 in 2024-05-15T12:53:21.994 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 13 16 35200 35184 169.139 125.938 0.00664412 0.00590768 2024-05-15T12:53:22.244 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:22 smithi027 ceph-mon[39795]: pgmap v639: 17 pgs: 17 active+clean; 1.7 GiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 157 MiB/s wr, 2.50k op/s 2024-05-15T12:53:22.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:22 smithi177 ceph-mon[44303]: pgmap v639: 17 pgs: 17 active+clean; 1.7 GiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 157 MiB/s wr, 2.50k op/s 2024-05-15T12:53:22.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:22 smithi120 ceph-mon[43961]: pgmap v639: 17 pgs: 17 active+clean; 1.7 GiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 157 MiB/s wr, 2.50k op/s 2024-05-15T12:53:22.994 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 14 16 37229 37213 166.115 126.812 0.00195845 0.00594566 2024-05-15T12:53:23.995 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 15 16 38844 38828 161.769 100.938 0.00401923 0.00617784 2024-05-15T12:53:24.216 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:53:24.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:24.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:53:24.321 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:53:24.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:24 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:53:24.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:24 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:53:24.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:24 smithi027 ceph-mon[39795]: osdmap e229: 12 total, 12 up, 11 in 2024-05-15T12:53:24.365 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:24 smithi027 ceph-mon[39795]: pgmap v641: 17 pgs: 17 active+clean; 2.1 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 173 MiB/s wr, 2.76k op/s 2024-05-15T12:53:24.366 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:24.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:24 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:53:24.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:24 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:53:24.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:24 smithi120 ceph-mon[43961]: osdmap e229: 12 total, 12 up, 11 in 2024-05-15T12:53:24.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:24 smithi120 ceph-mon[43961]: pgmap v641: 17 pgs: 17 active+clean; 2.1 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 173 MiB/s wr, 2.76k op/s 2024-05-15T12:53:24.579 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:53:24.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:24.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:53:24.621 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 2 1 2024-05-15T12:53:24.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:24 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:53:24.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:24 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:53:24.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:24 smithi177 ceph-mon[44303]: osdmap e229: 12 total, 12 up, 11 in 2024-05-15T12:53:24.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:24 smithi177 ceph-mon[44303]: pgmap v641: 17 pgs: 17 active+clean; 2.1 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 173 MiB/s wr, 2.76k op/s 2024-05-15T12:53:24.685 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:53:24.729 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:24.765 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:53:24.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:24.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:53:24.874 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:53:24.942 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:24.994 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 16 16 41004 40988 160.095 135 0.0135964 0.0062399 2024-05-15T12:53:25.416 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:53:25.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:25.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:53:25.513 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:53:25.520 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:53:25.553 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:53:25.590 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:25.597 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:53:25.639 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:53:25.683 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:53:25.698 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:53:25.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:25.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:53:25.731 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:53:25.753 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T12:53:25.753 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:53:25.765 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:53:25.776 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:53:25.805 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:53:25.815 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:53:25.843 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:25.855 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:53:25.867 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:53:25.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:25.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:53:25.880 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:53:25.915 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:53:25.950 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:53:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:53:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:53:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:53:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:53:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:53:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:53:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:53:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:53:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:53:25.960 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:53:25.960 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:53:25.960 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:53:25.972 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:53:25.995 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 17 16 42945 42929 157.813 121.312 0.00389348 0.0063325 2024-05-15T12:53:26.003 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:26.032 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:53:26.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:26.034 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:53:26.135 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:53:26.178 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:26.208 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:53:26.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:26.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:53:26.275 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:26 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:53:26.275 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:26 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:53:26.275 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:26 smithi177 ceph-mon[44303]: osdmap e230: 12 total, 12 up, 11 in 2024-05-15T12:53:26.275 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:26 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3914936706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:53:26.275 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:26 smithi177 ceph-mon[44303]: pgmap v643: 17 pgs: 17 active+clean; 2.3 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-05-15T12:53:26.314 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:53:26.369 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:26.407 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:53:26.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:26.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:53:26.512 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:53:26.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:26 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:53:26.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:26 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:53:26.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:26 smithi120 ceph-mon[43961]: osdmap e230: 12 total, 12 up, 11 in 2024-05-15T12:53:26.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:26 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3914936706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:53:26.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:26 smithi120 ceph-mon[43961]: pgmap v643: 17 pgs: 17 active+clean; 2.3 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-05-15T12:53:26.551 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:26.583 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:53:26.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:26.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:53:26.636 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:26 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:53:26.637 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:26 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:53:26.637 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:26 smithi027 ceph-mon[39795]: osdmap e230: 12 total, 12 up, 11 in 2024-05-15T12:53:26.637 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:26 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3914936706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:53:26.637 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:26 smithi027 ceph-mon[39795]: pgmap v643: 17 pgs: 17 active+clean; 2.3 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-05-15T12:53:26.691 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:53:26.746 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:26.995 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 18 16 44863 44847 155.705 119.875 0.0101066 0.00641816 2024-05-15T12:53:27.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:27 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:53:27.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:27 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:53:27.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:27 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:53:27.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:27 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:53:27.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:27 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:53:27.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:27 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:53:27.995 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 19 16 46998 46982 154.533 133.438 0.00469015 0.00646606 2024-05-15T12:53:28.108 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:53:28.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:28.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:53:28.212 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:53:28.245 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:28.269 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:53:28.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:28.272 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:53:28.373 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:53:28.406 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:28.426 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:53:28.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:28.429 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:53:28.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:28 smithi027 ceph-mon[39795]: osdmap e231: 12 total, 12 up, 11 in 2024-05-15T12:53:28.548 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:28 smithi027 ceph-mon[39795]: pgmap v645: 17 pgs: 17 active+clean; 2.5 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-05-15T12:53:28.549 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:28 smithi027 ceph-mon[39795]: osdmap e232: 12 total, 12 up, 11 in 2024-05-15T12:53:28.549 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:53:28.597 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:28.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:28 smithi177 ceph-mon[44303]: osdmap e231: 12 total, 12 up, 11 in 2024-05-15T12:53:28.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:28 smithi177 ceph-mon[44303]: pgmap v645: 17 pgs: 17 active+clean; 2.5 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-05-15T12:53:28.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:28 smithi177 ceph-mon[44303]: osdmap e232: 12 total, 12 up, 11 in 2024-05-15T12:53:28.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:28 smithi120 ceph-mon[43961]: osdmap e231: 12 total, 12 up, 11 in 2024-05-15T12:53:28.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:28 smithi120 ceph-mon[43961]: pgmap v645: 17 pgs: 17 active+clean; 2.5 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-05-15T12:53:28.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:28 smithi120 ceph-mon[43961]: osdmap e232: 12 total, 12 up, 11 in 2024-05-15T12:53:28.995 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T12:53:29.006177+0000 min lat: 0.00112991 max lat: 0.214075 avg lat: 0.00658695 2024-05-15T12:53:28.995 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T12:53:28.995 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 20 15 48465 48450 151.393 91.75 0.00492239 0.00658695 2024-05-15T12:53:29.995 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 21 16 49962 49946 148.635 93.5 0.00261134 0.0067069 2024-05-15T12:53:30.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:30 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:53:30.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:30 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:53:30.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:30 smithi177 ceph-mon[44303]: osdmap e233: 12 total, 12 up, 11 in 2024-05-15T12:53:30.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:30 smithi177 ceph-mon[44303]: pgmap v648: 17 pgs: 17 active+clean; 2.8 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-05-15T12:53:30.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:30 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:53:30.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:30 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:53:30.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:30 smithi027 ceph-mon[39795]: osdmap e233: 12 total, 12 up, 11 in 2024-05-15T12:53:30.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:30 smithi027 ceph-mon[39795]: pgmap v648: 17 pgs: 17 active+clean; 2.8 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-05-15T12:53:30.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:30 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:53:30.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:30 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:53:30.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:30 smithi120 ceph-mon[43961]: osdmap e233: 12 total, 12 up, 11 in 2024-05-15T12:53:30.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:30 smithi120 ceph-mon[43961]: pgmap v648: 17 pgs: 17 active+clean; 2.8 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-05-15T12:53:30.995 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 22 15 51546 51531 146.382 99.0625 0.00856201 0.00682735 2024-05-15T12:53:31.995 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 23 16 53494 53478 145.308 121.688 0.00806041 0.00687777 2024-05-15T12:53:32.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:32 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:53:32.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:32 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:53:32.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:32 smithi177 ceph-mon[44303]: osdmap e234: 12 total, 12 up, 11 in 2024-05-15T12:53:32.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:32 smithi177 ceph-mon[44303]: pgmap v650: 17 pgs: 17 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-05-15T12:53:32.995 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 24 16 55064 55048 143.341 98.125 0.0139627 0.0069721 2024-05-15T12:53:33.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:32 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:53:33.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:32 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:53:33.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:32 smithi027 ceph-mon[39795]: osdmap e234: 12 total, 12 up, 11 in 2024-05-15T12:53:33.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:32 smithi027 ceph-mon[39795]: pgmap v650: 17 pgs: 17 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-05-15T12:53:33.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:32 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:53:33.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:32 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:53:33.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:32 smithi120 ceph-mon[43961]: osdmap e234: 12 total, 12 up, 11 in 2024-05-15T12:53:33.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:32 smithi120 ceph-mon[43961]: pgmap v650: 17 pgs: 17 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-05-15T12:53:33.158 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-05-15T12:53:33.274 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:53:33.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:33.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:53:33.384 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:53:33.432 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:33.467 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:53:33.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:33.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:53:33.574 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:53:33.615 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:33.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:33 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:53:33.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:33 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:53:33.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:33 smithi027 ceph-mon[39795]: osdmap e235: 12 total, 12 up, 11 in 2024-05-15T12:53:33.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:33 smithi027 ceph-mon[39795]: osdmap e236: 12 total, 12 up, 11 in 2024-05-15T12:53:33.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:33 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:53:33.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:33 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:53:33.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:33 smithi120 ceph-mon[43961]: osdmap e235: 12 total, 12 up, 11 in 2024-05-15T12:53:33.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:33 smithi120 ceph-mon[43961]: osdmap e236: 12 total, 12 up, 11 in 2024-05-15T12:53:33.880 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:53:33.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:33 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:53:33.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:33 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:53:33.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:33 smithi177 ceph-mon[44303]: osdmap e235: 12 total, 12 up, 11 in 2024-05-15T12:53:33.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:33 smithi177 ceph-mon[44303]: osdmap e236: 12 total, 12 up, 11 in 2024-05-15T12:53:33.995 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 25 16 56355 56339 140.835 80.6875 0.00640641 0.00709617 2024-05-15T12:53:34.276 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:53:34.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:34.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:53:34.383 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:53:34.428 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:34.464 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:53:34.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:34.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:53:34.572 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:53:34.603 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:34.625 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:53:34.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:34.630 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:53:34.731 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:53:34.771 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:34.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:34 smithi120 ceph-mon[43961]: pgmap v653: 17 pgs: 17 active+clean; 3.2 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-05-15T12:53:34.885 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:53:34.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:34.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:53:34.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:34 smithi177 ceph-mon[44303]: pgmap v653: 17 pgs: 17 active+clean; 3.2 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-05-15T12:53:34.992 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:53:34.995 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 26 16 58095 58079 139.601 108.75 0.00280109 0.00714313 2024-05-15T12:53:35.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:34 smithi027 ceph-mon[39795]: pgmap v653: 17 pgs: 17 active+clean; 3.2 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-05-15T12:53:35.022 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:35.045 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:53:35.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:35.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:53:35.150 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:53:35.181 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:35.204 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:53:35.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:35.209 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:53:35.310 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:53:35.321 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:53:35.367 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:35.401 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:53:35.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:35.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:53:35.509 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:53:35.537 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:35.561 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:53:35.561 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:53:35.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:35.564 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:53:35.665 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:53:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:35.841 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:53:35.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:35.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:53:35.950 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:53:35.995 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 27 16 59704 59688 138.155 100.562 0.00866025 0.00723439 2024-05-15T12:53:36.009 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:36.041 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:53:36.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:36.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:53:36.144 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:53:36.206 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:36.568 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:53:36.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:36.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:53:36.685 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:53:36.728 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:36.766 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:53:36.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:36.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:53:36.870 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:53:36.900 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:36.918 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:53:36.919 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:36 smithi177 ceph-mon[44303]: pgmap v654: 17 pgs: 17 active+clean; 3.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-05-15T12:53:36.919 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:36 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3667522871' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:53:36.920 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:36 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:53:36.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:36.921 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:53:36.957 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T12:53:36.996 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 28 16 61211 61195 136.584 94.1875 0.00377543 0.0073176 2024-05-15T12:53:37.022 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:53:37.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:36 smithi027 ceph-mon[39795]: pgmap v654: 17 pgs: 17 active+clean; 3.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-05-15T12:53:37.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:36 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3667522871' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:53:37.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:36 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:53:37.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:36 smithi120 ceph-mon[43961]: pgmap v654: 17 pgs: 17 active+clean; 3.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-05-15T12:53:37.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:36 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3667522871' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:53:37.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:36 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:53:37.072 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:37.106 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:53:37.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:37.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:53:37.212 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:53:37.244 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:37.267 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:53:37.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:37.270 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:53:37.371 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:53:37.400 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:37.424 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:53:37.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:37.427 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:53:37.528 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:53:37.560 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:37.585 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:53:37.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:37.587 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:53:37.689 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:53:37.719 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:37.870 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:53:37.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:37.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:53:37.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:37 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:53:37.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:37 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:53:37.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:37 smithi177 ceph-mon[44303]: osdmap e237: 12 total, 12 up, 11 in 2024-05-15T12:53:37.981 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:53:37.996 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 29 16 62601 62585 134.87 86.875 0.00911818 0.00741028 2024-05-15T12:53:38.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:37 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:53:38.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:37 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:53:38.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:37 smithi027 ceph-mon[39795]: osdmap e237: 12 total, 12 up, 11 in 2024-05-15T12:53:38.023 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:37 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:53:38.023 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:37 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:53:38.023 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:37 smithi120 ceph-mon[43961]: osdmap e237: 12 total, 12 up, 11 in 2024-05-15T12:53:38.024 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:38.053 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:53:38.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:38.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:53:38.162 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:53:38.192 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:38.219 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:53:38.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:38.223 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:53:38.324 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:53:38.376 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:38.657 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:53:38.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:38.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:53:38.764 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:53:38.827 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:38.846 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:53:38.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:38.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:53:38.959 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:53:38.996 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 30 16 64033 64017 133.357 89.5 0.0157396 0.00749445 2024-05-15T12:53:39.001 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:38 smithi120 ceph-mon[43961]: pgmap v656: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-05-15T12:53:39.001 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:38 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:53:39.003 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:39.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:38 smithi027 ceph-mon[39795]: pgmap v656: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-05-15T12:53:39.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:38 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:53:39.031 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:53:39.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:39.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:53:39.136 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:53:39.165 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:39.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:38 smithi177 ceph-mon[44303]: pgmap v656: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-05-15T12:53:39.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:38 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:53:39.184 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:53:39.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:39.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:53:39.288 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:53:39.358 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:39.686 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:39 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:53:39.686 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:39 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:53:39.686 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:39 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:53:39.686 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:39 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3474570752' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T12:53:39.686 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:39 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T12:53:39.686 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:39 smithi027 ceph-mon[39795]: pgmap v657: 17 pgs: 17 active+clean; 3.7 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-05-15T12:53:39.887 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.2 to 1 (10000) 2024-05-15T12:53:39.914 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T12:53:39.996 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 31 16 65554 65538 132.122 95.0625 0.00903583 0.00756394 2024-05-15T12:53:40.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:39 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:53:40.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:39 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:53:40.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:39 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:53:40.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:39 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3474570752' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T12:53:40.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:39 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T12:53:40.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:39 smithi120 ceph-mon[43961]: pgmap v657: 17 pgs: 17 active+clean; 3.7 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-05-15T12:53:40.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:39 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:53:40.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:39 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:53:40.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:39 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:53:40.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:39 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3474570752' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T12:53:40.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:39 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T12:53:40.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:39 smithi177 ceph-mon[44303]: pgmap v657: 17 pgs: 17 active+clean; 3.7 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-05-15T12:53:40.487 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T12:53:40.488 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:53:40.525 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:53:40.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:40.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:53:40.632 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:53:40.664 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:40.689 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:53:40.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:40.693 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:53:40.794 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:53:40.823 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:40.848 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:53:40.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:40.854 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:53:40.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:40 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:53:40.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:40 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:53:40.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:40 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T12:53:40.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:40 smithi177 ceph-mon[44303]: osdmap e238: 12 total, 12 up, 11 in 2024-05-15T12:53:40.955 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:53:40.996 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 32 16 67155 67139 131.12 100.062 0.00677409 0.00762182 2024-05-15T12:53:41.006 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:41.035 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:53:41.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:41.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:53:41.096 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:53:41.136 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:40 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:53:41.136 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:40 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:53:41.136 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:40 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T12:53:41.136 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:40 smithi027 ceph-mon[39795]: osdmap e238: 12 total, 12 up, 11 in 2024-05-15T12:53:41.137 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 3 1 2024-05-15T12:53:41.148 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:53:41.242 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:41.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:40 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:53:41.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:40 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:53:41.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:40 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T12:53:41.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:40 smithi120 ceph-mon[43961]: osdmap e238: 12 total, 12 up, 11 in 2024-05-15T12:53:41.355 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:53:41.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:41.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:53:41.463 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:53:41.492 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:41.514 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:53:41.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:41.517 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:53:41.618 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:53:41.650 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:41.682 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:53:41.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:41.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:53:41.787 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:53:41.850 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:41.876 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:53:41.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:41.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:53:41.984 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:53:41.996 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 33 16 68565 68549 129.816 88.125 0.00860763 0.00769961 2024-05-15T12:53:42.029 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:42.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:41 smithi120 ceph-mon[43961]: pgmap v659: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-05-15T12:53:42.112 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:53:42.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:42.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:53:42.166 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:41 smithi027 ceph-mon[39795]: pgmap v659: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-05-15T12:53:42.167 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-05-15T12:53:42.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:41 smithi177 ceph-mon[44303]: pgmap v659: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-05-15T12:53:42.219 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:53:42.258 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:42.298 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:53:42.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:42.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:53:42.404 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:53:42.472 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:42.810 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:53:42.833 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:53:42.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:42.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:53:42.938 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:53:42.989 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:42.996 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 34 15 70066 70051 128.758 93.875 0.0129486 0.0077625 2024-05-15T12:53:43.036 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:53:43.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:43.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:53:43.144 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:53:43.196 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:43.230 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:53:43.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:43.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:53:43.335 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:53:43.384 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:43.412 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:53:43.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:43.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:53:43.514 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:53:43.557 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:43.588 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:53:43.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:43.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:53:43.696 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:53:43.744 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:43.779 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:53:43.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:43.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:53:43.885 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:53:43.949 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:43.978 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:53:43.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:43.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:53:43.996 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 35 16 71572 71556 127.767 94.0625 0.00870022 0.00782344 2024-05-15T12:53:44.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:43 smithi120 ceph-mon[43961]: osdmap e239: 12 total, 12 up, 11 in 2024-05-15T12:53:44.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:43 smithi120 ceph-mon[43961]: pgmap v661: 17 pgs: 17 active+clean; 4.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-05-15T12:53:44.082 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:53:44.130 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:44.168 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:53:44.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:44.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:53:44.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:43 smithi027 ceph-mon[39795]: osdmap e239: 12 total, 12 up, 11 in 2024-05-15T12:53:44.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:43 smithi027 ceph-mon[39795]: pgmap v661: 17 pgs: 17 active+clean; 4.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-05-15T12:53:44.274 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:53:44.303 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:43 smithi177 ceph-mon[44303]: osdmap e239: 12 total, 12 up, 11 in 2024-05-15T12:53:44.303 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:43 smithi177 ceph-mon[44303]: pgmap v661: 17 pgs: 17 active+clean; 4.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-05-15T12:53:44.305 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:44.326 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:53:44.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:44.329 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:53:44.430 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:53:44.465 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:44.486 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:53:44.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:44.490 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:53:44.591 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:53:44.643 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:44.683 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:53:44.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:44.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:53:44.790 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:53:44.820 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:44.843 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:53:44.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:44.845 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:53:44.946 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:53:44.996 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 36 16 73079 73063 126.834 94.1875 0.00598114 0.0078802 2024-05-15T12:53:45.019 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:45.158 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:45 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2934329110' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:53:45.162 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:53:45.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:45.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:53:45.284 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:53:45.312 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:45.333 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:53:45.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:45.336 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:53:45.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:45 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2934329110' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:53:45.437 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:53:45.480 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:45 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2934329110' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:53:45.483 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:45.509 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:53:45.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:45.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:53:45.619 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:53:45.647 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:45.669 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:53:45.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:45.672 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:53:45.773 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:53:45.806 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:45.828 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:53:45.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:45.831 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:53:45.903 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:53:45.903 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:53:45.903 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:53:45.903 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:53:45.903 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:53:45.903 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:53:45.903 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:53:45.903 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:53:45.903 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:53:45.904 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:53:45.904 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:53:45.904 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:53:45.904 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:53:45.904 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:53:45.904 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:53:45.904 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:53:45.904 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:53:45.904 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:53:45.904 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:53:45.904 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:53:45.905 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:53:45.905 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:53:45.905 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:53:45.905 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:53:45.931 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:53:45.961 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:45.988 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:53:45.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:45.993 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:53:45.996 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 37 16 74539 74523 125.872 91.25 0.0102804 0.00794077 2024-05-15T12:53:46.093 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:53:46.123 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:46.133 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:53:46.136 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:53:46.143 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:53:46.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:46.146 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:53:46.247 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:53:46.297 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:46.330 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:53:46.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:46.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:53:46.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:46 smithi177 ceph-mon[44303]: pgmap v662: 17 pgs: 17 active+clean; 4.3 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-05-15T12:53:46.439 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:53:46.479 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:46 smithi120 ceph-mon[43961]: pgmap v662: 17 pgs: 17 active+clean; 4.3 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-05-15T12:53:46.486 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:46.507 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:53:46.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:46.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:53:46.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:46 smithi027 ceph-mon[39795]: pgmap v662: 17 pgs: 17 active+clean; 4.3 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-05-15T12:53:46.615 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:53:46.655 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:46.686 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:53:46.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:46.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:53:46.791 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:53:46.840 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:46.997 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 38 16 75998 75982 124.959 91.1875 0.0172921 0.00799907 2024-05-15T12:53:47.242 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:53:47.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:47.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:53:47.348 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:53:47.420 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:47.997 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 39 16 77455 77439 124.09 91.0625 0.00746967 0.00805498 2024-05-15T12:53:48.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:47 smithi027 ceph-mon[39795]: pgmap v663: 17 pgs: 17 active+clean; 4.5 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-05-15T12:53:48.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:47 smithi120 ceph-mon[43961]: pgmap v663: 17 pgs: 17 active+clean; 4.5 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-05-15T12:53:48.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:47 smithi177 ceph-mon[44303]: pgmap v663: 17 pgs: 17 active+clean; 4.5 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-05-15T12:53:48.681 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:53:48.997 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T12:53:49.008054+0000 min lat: 0.00112991 max lat: 0.214075 avg lat: 0.008107 2024-05-15T12:53:48.997 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T12:53:48.997 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 40 16 78929 78913 123.29 92.125 0.0104457 0.008107 2024-05-15T12:53:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:53:49.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:49.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:53:49.997 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 41 16 80307 80291 122.384 86.125 0.0123714 0.00816744 2024-05-15T12:53:50.009 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:53:50.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:49 smithi027 ceph-mon[39795]: pgmap v664: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-05-15T12:53:50.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:49 smithi120 ceph-mon[43961]: pgmap v664: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-05-15T12:53:50.070 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:50.087 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T12:53:50.087 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T12:53:50.087 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T12:53:50.087 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T12:53:50.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.088 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:51713 2.1 2:8f803db6:::benchmark_data_smithi177_78104_object51712:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-15T12:53:50.088 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:42.601217+0000", 2024-05-15T12:53:50.088 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 307.49324418700002, 2024-05-15T12:53:50.088 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.098120633999999998, 2024-05-15T12:53:50.088 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:50.088 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:50.088 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:50.088 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:50.088 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:50.088 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 51713 2024-05-15T12:53:50.088 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.088 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:50.089 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.089 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:50.089 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.601217+0000", 2024-05-15T12:53:50.089 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.089 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.089 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.089 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:50.089 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.601217+0000", 2024-05-15T12:53:50.089 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.089 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.089 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.089 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:50.089 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.601218+0000", 2024-05-15T12:53:50.090 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0589999999999999e-06 2024-05-15T12:53:50.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.090 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.090 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:50.090 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.601366+0000", 2024-05-15T12:53:50.090 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014783099999999999 2024-05-15T12:53:50.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.090 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.090 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:50.090 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.601377+0000", 2024-05-15T12:53:50.090 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.077e-05 2024-05-15T12:53:50.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.090 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.091 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:50.091 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.601386+0000", 2024-05-15T12:53:50.091 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7069999999999998e-06 2024-05-15T12:53:50.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.091 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:50.091 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.602166+0000", 2024-05-15T12:53:50.091 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00078028799999999997 2024-05-15T12:53:50.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.091 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:50.091 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.602279+0000", 2024-05-15T12:53:50.091 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000112859 2024-05-15T12:53:50.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.092 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,3", 2024-05-15T12:53:50.092 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.602412+0000", 2024-05-15T12:53:50.092 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013315899999999999 2024-05-15T12:53:50.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.092 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.092 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.690434+0000", 2024-05-15T12:53:50.092 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.088022144999999996 2024-05-15T12:53:50.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.093 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.093 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.690477+0000", 2024-05-15T12:53:50.093 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2521999999999997e-05 2024-05-15T12:53:50.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.093 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:50.093 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.699238+0000", 2024-05-15T12:53:50.093 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0087611070000000006 2024-05-15T12:53:50.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.093 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:50.093 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.699290+0000", 2024-05-15T12:53:50.093 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1999999999999997e-05 2024-05-15T12:53:50.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.094 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.094 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:50.094 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.699338+0000", 2024-05-15T12:53:50.094 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8186999999999998e-05 2024-05-15T12:53:50.094 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.094 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:50.094 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.094 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.094 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.094 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:58488 2.1 2:8c133fc7:::benchmark_data_smithi177_78104_object58487:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-15T12:53:50.094 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:48.359181+0000", 2024-05-15T12:53:50.094 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 301.735280692, 2024-05-15T12:53:50.094 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.105747596, 2024-05-15T12:53:50.095 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:50.095 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:50.095 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:50.095 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:50.095 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:50.095 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 58488 2024-05-15T12:53:50.095 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.095 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:50.095 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.095 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:50.095 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359181+0000", 2024-05-15T12:53:50.095 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.096 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:50.096 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359181+0000", 2024-05-15T12:53:50.096 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.096 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:50.096 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359182+0000", 2024-05-15T12:53:50.096 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1659999999999999e-06 2024-05-15T12:53:50.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.096 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:50.097 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359212+0000", 2024-05-15T12:53:50.097 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9657000000000001e-05 2024-05-15T12:53:50.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.097 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:50.097 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359220+0000", 2024-05-15T12:53:50.097 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2810000000000008e-06 2024-05-15T12:53:50.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.097 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:50.097 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359226+0000", 2024-05-15T12:53:50.097 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2169999999999999e-06 2024-05-15T12:53:50.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.098 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:50.098 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359282+0000", 2024-05-15T12:53:50.098 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5878000000000003e-05 2024-05-15T12:53:50.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.098 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.098 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:50.098 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359431+0000", 2024-05-15T12:53:50.098 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014895899999999999 2024-05-15T12:53:50.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.098 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.098 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,3", 2024-05-15T12:53:50.098 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359608+0000", 2024-05-15T12:53:50.098 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017731299999999999 2024-05-15T12:53:50.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.099 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.099 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.363541+0000", 2024-05-15T12:53:50.099 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039324939999999999 2024-05-15T12:53:50.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.099 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:50.099 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.363625+0000", 2024-05-15T12:53:50.099 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3719000000000003e-05 2024-05-15T12:53:50.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.099 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.099 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464802+0000", 2024-05-15T12:53:50.099 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.101177645 2024-05-15T12:53:50.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.100 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.100 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:50.100 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464874+0000", 2024-05-15T12:53:50.100 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2324999999999996e-05 2024-05-15T12:53:50.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.100 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.100 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:50.100 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464928+0000", 2024-05-15T12:53:50.100 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3942e-05 2024-05-15T12:53:50.100 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.100 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:50.100 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.101 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:58490 2.1 2:80921832:::benchmark_data_smithi177_78104_object58489:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-15T12:53:50.101 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:48.359238+0000", 2024-05-15T12:53:50.101 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 301.735224012, 2024-05-15T12:53:50.101 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10584242100000001, 2024-05-15T12:53:50.101 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:50.101 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:50.101 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:50.101 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:50.101 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:50.101 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 58490 2024-05-15T12:53:50.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.101 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:50.102 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.102 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:50.102 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359238+0000", 2024-05-15T12:53:50.102 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.102 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.102 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.102 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:50.102 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359238+0000", 2024-05-15T12:53:50.102 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.102 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.102 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.102 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:50.102 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359238+0000", 2024-05-15T12:53:50.103 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3500000000000002e-07 2024-05-15T12:53:50.103 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.103 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.103 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:50.103 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359388+0000", 2024-05-15T12:53:50.103 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015008 2024-05-15T12:53:50.103 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.103 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.103 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:50.103 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359398+0000", 2024-05-15T12:53:50.103 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7569999999999999e-06 2024-05-15T12:53:50.103 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.103 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.103 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:50.104 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359405+0000", 2024-05-15T12:53:50.104 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7970000000000003e-06 2024-05-15T12:53:50.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.104 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.104 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:50.104 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361472+0000", 2024-05-15T12:53:50.104 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020671880000000002 2024-05-15T12:53:50.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.104 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.104 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:50.104 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361583+0000", 2024-05-15T12:53:50.104 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000110745 2024-05-15T12:53:50.105 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.105 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.105 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,3", 2024-05-15T12:53:50.105 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361714+0000", 2024-05-15T12:53:50.105 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013185399999999999 2024-05-15T12:53:50.105 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.105 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.105 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.105 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.363611+0000", 2024-05-15T12:53:50.105 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0018970689999999999 2024-05-15T12:53:50.105 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.105 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.105 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:50.106 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367554+0000", 2024-05-15T12:53:50.106 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039426879999999997 2024-05-15T12:53:50.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.106 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.106 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.106 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464987+0000", 2024-05-15T12:53:50.106 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.097432747 2024-05-15T12:53:50.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.106 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.106 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:50.106 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465043+0000", 2024-05-15T12:53:50.106 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.588e-05 2024-05-15T12:53:50.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.106 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.107 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:50.107 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465080+0000", 2024-05-15T12:53:50.107 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7181000000000002e-05 2024-05-15T12:53:50.107 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.107 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:50.107 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.107 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.107 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.107 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:58493 2.1 2:81994a98:::benchmark_data_smithi177_78104_object58492:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-15T12:53:50.107 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:48.359509+0000", 2024-05-15T12:53:50.107 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 301.73495261400001, 2024-05-15T12:53:50.107 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.105753467, 2024-05-15T12:53:50.107 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:50.107 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:50.108 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:50.108 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:50.108 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:50.108 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 58493 2024-05-15T12:53:50.108 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.108 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:50.108 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.108 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:50.108 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359509+0000", 2024-05-15T12:53:50.108 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.108 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.108 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.108 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:50.109 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359509+0000", 2024-05-15T12:53:50.109 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.109 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.109 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:50.109 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359510+0000", 2024-05-15T12:53:50.109 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8599999999999997e-07 2024-05-15T12:53:50.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.109 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.109 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:50.109 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359561+0000", 2024-05-15T12:53:50.109 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0909999999999999e-05 2024-05-15T12:53:50.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.109 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.110 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:50.110 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359570+0000", 2024-05-15T12:53:50.110 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9579999999999996e-06 2024-05-15T12:53:50.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.110 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.110 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:50.110 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359576+0000", 2024-05-15T12:53:50.110 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6320000000000002e-06 2024-05-15T12:53:50.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.110 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.110 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:50.110 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.363643+0000", 2024-05-15T12:53:50.110 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0040667409999999996 2024-05-15T12:53:50.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.111 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.111 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:50.111 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.363730+0000", 2024-05-15T12:53:50.111 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6646999999999999e-05 2024-05-15T12:53:50.111 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.111 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.111 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,3", 2024-05-15T12:53:50.111 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.363815+0000", 2024-05-15T12:53:50.111 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5136000000000006e-05 2024-05-15T12:53:50.111 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.111 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.111 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:50.111 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464946+0000", 2024-05-15T12:53:50.111 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.101130962 2024-05-15T12:53:50.112 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.112 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.112 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.112 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465138+0000", 2024-05-15T12:53:50.112 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00019219199999999999 2024-05-15T12:53:50.112 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.112 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.112 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.112 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465177+0000", 2024-05-15T12:53:50.112 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9573999999999998e-05 2024-05-15T12:53:50.112 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.112 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.112 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:50.112 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465225+0000", 2024-05-15T12:53:50.113 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7426999999999998e-05 2024-05-15T12:53:50.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.113 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.113 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:50.113 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465262+0000", 2024-05-15T12:53:50.113 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7502000000000001e-05 2024-05-15T12:53:50.113 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.113 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:50.113 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.113 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.113 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:58498 2.1 2:8b03664e:::benchmark_data_smithi177_78104_object58497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-15T12:53:50.113 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:48.361834+0000", 2024-05-15T12:53:50.113 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 301.732627546, 2024-05-15T12:53:50.114 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10363879400000001, 2024-05-15T12:53:50.114 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:50.114 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:50.114 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:50.114 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:50.114 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:50.114 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 58498 2024-05-15T12:53:50.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.114 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:50.114 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.114 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:50.114 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361834+0000", 2024-05-15T12:53:50.114 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.115 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:50.115 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361834+0000", 2024-05-15T12:53:50.115 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.115 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:50.115 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361835+0000", 2024-05-15T12:53:50.115 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.155e-06 2024-05-15T12:53:50.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.115 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:50.115 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361876+0000", 2024-05-15T12:53:50.116 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1337000000000001e-05 2024-05-15T12:53:50.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.116 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.116 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:50.116 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361887+0000", 2024-05-15T12:53:50.116 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0589e-05 2024-05-15T12:53:50.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.116 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.116 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:50.116 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361894+0000", 2024-05-15T12:53:50.116 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.289e-06 2024-05-15T12:53:50.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.116 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.116 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:50.117 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365486+0000", 2024-05-15T12:53:50.117 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035911979999999999 2024-05-15T12:53:50.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.117 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:50.117 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365624+0000", 2024-05-15T12:53:50.117 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000138281 2024-05-15T12:53:50.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.117 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,3", 2024-05-15T12:53:50.117 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365802+0000", 2024-05-15T12:53:50.117 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017837500000000001 2024-05-15T12:53:50.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.118 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:50.118 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465090+0000", 2024-05-15T12:53:50.118 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.099288247999999996 2024-05-15T12:53:50.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.118 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.118 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.118 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465303+0000", 2024-05-15T12:53:50.118 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00021270499999999999 2024-05-15T12:53:50.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.118 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.118 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.118 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465390+0000", 2024-05-15T12:53:50.118 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6355999999999998e-05 2024-05-15T12:53:50.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.119 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.119 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:50.119 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465437+0000", 2024-05-15T12:53:50.119 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7642000000000002e-05 2024-05-15T12:53:50.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.119 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.119 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:50.119 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465473+0000", 2024-05-15T12:53:50.119 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5618999999999999e-05 2024-05-15T12:53:50.119 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.119 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:50.119 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.119 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.120 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.24562.0:58488 2.1 e129/110 ondisk, result = 0)", 2024-05-15T12:53:50.120 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:48.365352+0000", 2024-05-15T12:53:50.120 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 301.72910971599998, 2024-05-15T12:53:50.120 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.099595227999999994, 2024-05-15T12:53:50.120 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:50.120 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T12:53:50.120 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:50.120 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.120 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:50.120 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365352+0000", 2024-05-15T12:53:50.120 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.120 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.120 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.120 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:50.121 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365352+0000", 2024-05-15T12:53:50.121 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.121 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.121 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:50.121 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365352+0000", 2024-05-15T12:53:50.121 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5699999999999998e-07 2024-05-15T12:53:50.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.121 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.121 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:50.121 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365360+0000", 2024-05-15T12:53:50.121 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6690000000000006e-06 2024-05-15T12:53:50.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.121 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.122 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:50.122 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365367+0000", 2024-05-15T12:53:50.122 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4939999999999997e-06 2024-05-15T12:53:50.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.122 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.122 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:50.122 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365378+0000", 2024-05-15T12:53:50.122 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.066e-05 2024-05-15T12:53:50.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.122 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.122 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:50.122 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464785+0000", 2024-05-15T12:53:50.122 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.099406904000000004 2024-05-15T12:53:50.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.123 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.123 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:50.123 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464795+0000", 2024-05-15T12:53:50.123 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0076e-05 2024-05-15T12:53:50.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.123 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.123 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:50.123 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464947+0000", 2024-05-15T12:53:50.123 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015206799999999999 2024-05-15T12:53:50.123 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.123 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:50.123 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.124 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.124 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:58501 2.4 e129/128 2:2c05dc93:::benchmark_data_smithi177_78104_object58500:head v 129'3650, mlcod=129'3648)", 2024-05-15T12:53:50.124 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:48.365547+0000", 2024-05-15T12:53:50.124 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 301.72891476000001, 2024-05-15T12:53:50.124 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10243205599999999, 2024-05-15T12:53:50.124 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:50.124 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:50.124 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:50.124 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.124 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:50.124 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365547+0000", 2024-05-15T12:53:50.124 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.124 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.125 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:50.125 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365547+0000", 2024-05-15T12:53:50.125 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.125 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:50.125 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365547+0000", 2024-05-15T12:53:50.125 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0800000000000004e-07 2024-05-15T12:53:50.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.125 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:50.125 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365571+0000", 2024-05-15T12:53:50.125 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3992e-05 2024-05-15T12:53:50.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.126 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:50.126 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365582+0000", 2024-05-15T12:53:50.126 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0929e-05 2024-05-15T12:53:50.126 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.126 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.126 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:50.126 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365589+0000", 2024-05-15T12:53:50.126 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.545e-06 2024-05-15T12:53:50.126 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.126 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.126 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:50.126 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365641+0000", 2024-05-15T12:53:50.126 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1969000000000003e-05 2024-05-15T12:53:50.126 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.127 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.127 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:50.127 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365664+0000", 2024-05-15T12:53:50.127 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3442999999999999e-05 2024-05-15T12:53:50.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.127 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.127 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:50.127 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.467932+0000", 2024-05-15T12:53:50.127 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.102268092 2024-05-15T12:53:50.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.127 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.127 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:50.127 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.467979+0000", 2024-05-15T12:53:50.127 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6378000000000002e-05 2024-05-15T12:53:50.127 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.128 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:50.128 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.128 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.128 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.128 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:58506 2.e 2:73b262f3:::benchmark_data_smithi177_78104_object58505:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-15T12:53:50.128 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:48.365633+0000", 2024-05-15T12:53:50.128 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 301.72882831599998, 2024-05-15T12:53:50.128 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.106977115, 2024-05-15T12:53:50.128 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:50.128 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:50.128 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:50.128 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:50.128 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:50.128 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 58506 2024-05-15T12:53:50.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.129 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:50.129 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.129 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:50.129 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365633+0000", 2024-05-15T12:53:50.129 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.129 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.129 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:50.129 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365633+0000", 2024-05-15T12:53:50.129 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.129 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.129 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:50.130 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365634+0000", 2024-05-15T12:53:50.130 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1289999999999999e-06 2024-05-15T12:53:50.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.130 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.130 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:50.130 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367598+0000", 2024-05-15T12:53:50.130 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019633670000000002 2024-05-15T12:53:50.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.130 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.130 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:50.130 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367609+0000", 2024-05-15T12:53:50.130 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1491000000000001e-05 2024-05-15T12:53:50.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.130 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.131 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:50.131 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367617+0000", 2024-05-15T12:53:50.131 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9270000000000005e-06 2024-05-15T12:53:50.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.131 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.131 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:50.131 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367669+0000", 2024-05-15T12:53:50.131 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2287000000000003e-05 2024-05-15T12:53:50.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.131 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.131 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:50.131 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367777+0000", 2024-05-15T12:53:50.131 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000107895 2024-05-15T12:53:50.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.132 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.132 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:53:50.132 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464792+0000", 2024-05-15T12:53:50.132 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.097014787000000005 2024-05-15T12:53:50.132 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.132 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.132 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:50.132 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.469572+0000", 2024-05-15T12:53:50.132 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0047797839999999996 2024-05-15T12:53:50.132 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.133 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.133 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.133 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.471658+0000", 2024-05-15T12:53:50.133 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020863610000000001 2024-05-15T12:53:50.133 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.133 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.133 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.133 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.472506+0000", 2024-05-15T12:53:50.133 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00084819400000000003 2024-05-15T12:53:50.133 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.133 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.133 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:50.134 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.472563+0000", 2024-05-15T12:53:50.134 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6164e-05 2024-05-15T12:53:50.134 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.134 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.134 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:50.134 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.472610+0000", 2024-05-15T12:53:50.134 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7729000000000003e-05 2024-05-15T12:53:50.134 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.134 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:50.135 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.135 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:58508 2.1 2:8ad9250e:::benchmark_data_smithi177_78104_object58507:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-15T12:53:50.135 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:48.367638+0000", 2024-05-15T12:53:50.135 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 301.726823126, 2024-05-15T12:53:50.135 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.107729793, 2024-05-15T12:53:50.135 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:50.135 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:50.135 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:50.135 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:50.135 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:50.135 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 58508 2024-05-15T12:53:50.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.136 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:50.136 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.136 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:50.136 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367638+0000", 2024-05-15T12:53:50.136 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.136 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.136 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.136 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:50.136 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367638+0000", 2024-05-15T12:53:50.136 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.136 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.136 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.136 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:50.136 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367640+0000", 2024-05-15T12:53:50.137 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4389999999999999e-06 2024-05-15T12:53:50.137 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.137 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.137 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:50.137 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367672+0000", 2024-05-15T12:53:50.137 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2190999999999997e-05 2024-05-15T12:53:50.137 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.137 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.137 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:50.137 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367681+0000", 2024-05-15T12:53:50.137 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2110000000000004e-06 2024-05-15T12:53:50.137 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.137 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.138 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:50.138 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367688+0000", 2024-05-15T12:53:50.138 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8800000000000002e-06 2024-05-15T12:53:50.138 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.138 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.138 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:50.138 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465627+0000", 2024-05-15T12:53:50.138 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.097938774000000006 2024-05-15T12:53:50.138 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.138 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.138 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:50.139 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465788+0000", 2024-05-15T12:53:50.139 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000160631 2024-05-15T12:53:50.139 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.139 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.139 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,3", 2024-05-15T12:53:50.139 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465946+0000", 2024-05-15T12:53:50.139 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000158918 2024-05-15T12:53:50.139 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.139 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.139 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:50.139 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.469751+0000", 2024-05-15T12:53:50.140 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038043389999999999 2024-05-15T12:53:50.140 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.140 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.140 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.140 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.473812+0000", 2024-05-15T12:53:50.140 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0040614559999999997 2024-05-15T12:53:50.140 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.140 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.140 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.140 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.475272+0000", 2024-05-15T12:53:50.140 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0014601359999999999 2024-05-15T12:53:50.140 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.141 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.141 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:50.141 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.475329+0000", 2024-05-15T12:53:50.141 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.622e-05 2024-05-15T12:53:50.141 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.141 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.141 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:50.141 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.475368+0000", 2024-05-15T12:53:50.141 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9598000000000003e-05 2024-05-15T12:53:50.141 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.141 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:50.141 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.141 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.141 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.142 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89600 2.e 2:763b1fd1:::benchmark_data_smithi177_78104_object89599:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:53:50.142 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.804057+0000", 2024-05-15T12:53:50.142 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 274.29040463799998, 2024-05-15T12:53:50.142 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.21480517800000001, 2024-05-15T12:53:50.142 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:50.142 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:50.142 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:50.142 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:50.142 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:50.142 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89600 2024-05-15T12:53:50.142 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.142 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:50.142 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.143 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:50.143 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804057+0000", 2024-05-15T12:53:50.143 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.143 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.143 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.143 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:50.143 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804057+0000", 2024-05-15T12:53:50.143 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.143 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.143 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.143 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:50.143 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804058+0000", 2024-05-15T12:53:50.143 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0690000000000001e-06 2024-05-15T12:53:50.143 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.143 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.144 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:50.144 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804179+0000", 2024-05-15T12:53:50.144 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120938 2024-05-15T12:53:50.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.144 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.144 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:50.144 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804191+0000", 2024-05-15T12:53:50.144 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1868000000000001e-05 2024-05-15T12:53:50.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.144 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.144 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:50.144 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804197+0000", 2024-05-15T12:53:50.144 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2550000000000003e-06 2024-05-15T12:53:50.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.145 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.145 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:50.145 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804252+0000", 2024-05-15T12:53:50.145 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4953000000000001e-05 2024-05-15T12:53:50.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.145 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.145 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:50.145 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804392+0000", 2024-05-15T12:53:50.145 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000139772 2024-05-15T12:53:50.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.145 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.145 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:53:50.145 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804513+0000", 2024-05-15T12:53:50.145 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120763 2024-05-15T12:53:50.146 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.146 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.146 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:50.146 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.808055+0000", 2024-05-15T12:53:50.146 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035428830000000001 2024-05-15T12:53:50.146 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.146 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.146 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.146 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.819733+0000", 2024-05-15T12:53:50.146 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011677682999999999 2024-05-15T12:53:50.146 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.146 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.146 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.147 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.018742+0000", 2024-05-15T12:53:50.147 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19900900199999999 2024-05-15T12:53:50.147 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.147 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.147 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:50.147 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.018812+0000", 2024-05-15T12:53:50.147 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9924000000000003e-05 2024-05-15T12:53:50.147 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.147 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.147 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:50.147 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.018862+0000", 2024-05-15T12:53:50.147 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0068000000000002e-05 2024-05-15T12:53:50.147 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.148 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:50.148 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.148 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.148 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.148 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89634 2.e 2:7b9cebfb:::benchmark_data_smithi177_78104_object89633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:53:50.148 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.841129+0000", 2024-05-15T12:53:50.148 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 274.25333240200001, 2024-05-15T12:53:50.148 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.18161068899999999, 2024-05-15T12:53:50.148 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:50.148 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:50.148 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:50.148 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:50.148 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:50.148 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89634 2024-05-15T12:53:50.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.149 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:50.149 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.149 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:50.149 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841129+0000", 2024-05-15T12:53:50.149 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.149 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.149 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:50.149 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841129+0000", 2024-05-15T12:53:50.149 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.149 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.149 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:50.149 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841131+0000", 2024-05-15T12:53:50.150 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.699e-06 2024-05-15T12:53:50.150 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.150 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.150 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:50.150 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841163+0000", 2024-05-15T12:53:50.150 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2499999999999997e-05 2024-05-15T12:53:50.150 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.150 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.150 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:50.150 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841181+0000", 2024-05-15T12:53:50.150 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7436e-05 2024-05-15T12:53:50.150 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.150 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.151 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:50.151 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841190+0000", 2024-05-15T12:53:50.151 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.02e-06 2024-05-15T12:53:50.151 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.151 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.151 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:50.151 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841264+0000", 2024-05-15T12:53:50.151 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3866000000000001e-05 2024-05-15T12:53:50.151 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.151 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.151 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:50.151 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841398+0000", 2024-05-15T12:53:50.151 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013442900000000001 2024-05-15T12:53:50.151 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.152 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.152 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:53:50.152 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841544+0000", 2024-05-15T12:53:50.152 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014611800000000001 2024-05-15T12:53:50.152 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.152 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.152 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:50.152 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.844757+0000", 2024-05-15T12:53:50.152 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032125180000000001 2024-05-15T12:53:50.152 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.152 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.152 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.152 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.850454+0000", 2024-05-15T12:53:50.152 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.005697667 2024-05-15T12:53:50.153 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.153 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.153 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.153 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.022628+0000", 2024-05-15T12:53:50.153 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.17217374299999999 2024-05-15T12:53:50.153 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.153 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.153 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:50.153 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.022690+0000", 2024-05-15T12:53:50.153 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1566e-05 2024-05-15T12:53:50.153 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.153 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.153 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:50.153 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.022740+0000", 2024-05-15T12:53:50.154 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0127000000000003e-05 2024-05-15T12:53:50.154 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.154 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:50.154 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.154 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.154 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.154 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89645 2.e 2:7ffe4884:::benchmark_data_smithi177_78104_object89644:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:53:50.154 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.858184+0000", 2024-05-15T12:53:50.154 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 274.23627714000003, 2024-05-15T12:53:50.154 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.164674923, 2024-05-15T12:53:50.154 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:50.154 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:50.154 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:50.154 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:50.155 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:50.155 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89645 2024-05-15T12:53:50.155 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.155 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:50.155 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.155 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:50.155 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858184+0000", 2024-05-15T12:53:50.155 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.155 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.155 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.155 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:50.155 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858184+0000", 2024-05-15T12:53:50.155 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.155 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.156 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.156 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:50.156 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858186+0000", 2024-05-15T12:53:50.156 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5850000000000001e-06 2024-05-15T12:53:50.156 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.156 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.156 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:50.156 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858245+0000", 2024-05-15T12:53:50.156 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8729e-05 2024-05-15T12:53:50.156 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.156 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.156 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:50.156 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858258+0000", 2024-05-15T12:53:50.156 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3767999999999999e-05 2024-05-15T12:53:50.156 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.157 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.157 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:50.157 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858265+0000", 2024-05-15T12:53:50.157 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.725e-06 2024-05-15T12:53:50.157 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.157 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.157 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:50.157 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858324+0000", 2024-05-15T12:53:50.157 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8910999999999999e-05 2024-05-15T12:53:50.157 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.157 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.157 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:50.157 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858435+0000", 2024-05-15T12:53:50.158 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000111068 2024-05-15T12:53:50.158 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.158 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.158 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:53:50.158 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858592+0000", 2024-05-15T12:53:50.158 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000157083 2024-05-15T12:53:50.158 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.158 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.158 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:50.158 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.871361+0000", 2024-05-15T12:53:50.158 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012768662 2024-05-15T12:53:50.158 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.158 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.158 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.159 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.872157+0000", 2024-05-15T12:53:50.159 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00079593199999999995 2024-05-15T12:53:50.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.159 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.159 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.159 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.022780+0000", 2024-05-15T12:53:50.159 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.150623014 2024-05-15T12:53:50.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.159 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.159 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:50.159 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.022825+0000", 2024-05-15T12:53:50.159 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5626e-05 2024-05-15T12:53:50.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.160 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.160 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:50.160 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.022859+0000", 2024-05-15T12:53:50.160 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3819999999999998e-05 2024-05-15T12:53:50.160 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.160 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:50.160 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.160 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.160 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.160 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89654 2.e 2:7e1530a2:::benchmark_data_smithi177_78104_object89653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:53:50.160 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.874001+0000", 2024-05-15T12:53:50.160 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 274.22046088399998, 2024-05-15T12:53:50.160 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.15396952, 2024-05-15T12:53:50.160 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:50.161 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:50.161 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:50.161 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:50.161 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:50.161 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89654 2024-05-15T12:53:50.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.161 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:50.161 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.161 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:50.161 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874001+0000", 2024-05-15T12:53:50.161 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.161 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.162 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:50.162 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874001+0000", 2024-05-15T12:53:50.162 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.162 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:50.162 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874002+0000", 2024-05-15T12:53:50.162 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6199999999999999e-06 2024-05-15T12:53:50.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.162 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:50.162 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874045+0000", 2024-05-15T12:53:50.162 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3191e-05 2024-05-15T12:53:50.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.163 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.163 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:50.163 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874061+0000", 2024-05-15T12:53:50.163 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6002999999999999e-05 2024-05-15T12:53:50.163 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.163 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.163 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:50.163 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874072+0000", 2024-05-15T12:53:50.163 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0647e-05 2024-05-15T12:53:50.163 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.163 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.163 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:50.163 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874137+0000", 2024-05-15T12:53:50.164 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4941e-05 2024-05-15T12:53:50.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.164 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.164 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:50.164 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874222+0000", 2024-05-15T12:53:50.164 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4581999999999998e-05 2024-05-15T12:53:50.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.164 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.164 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:53:50.164 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874308+0000", 2024-05-15T12:53:50.164 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6658999999999998e-05 2024-05-15T12:53:50.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.164 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.165 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:50.165 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.876658+0000", 2024-05-15T12:53:50.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0023498080000000001 2024-05-15T12:53:50.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.165 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.165 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.165 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.893634+0000", 2024-05-15T12:53:50.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016976104999999998 2024-05-15T12:53:50.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.165 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.165 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.165 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.027858+0000", 2024-05-15T12:53:50.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.134223696 2024-05-15T12:53:50.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.166 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.166 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:50.166 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.027919+0000", 2024-05-15T12:53:50.166 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1464000000000006e-05 2024-05-15T12:53:50.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.166 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.166 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:50.166 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.027970+0000", 2024-05-15T12:53:50.166 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0803999999999997e-05 2024-05-15T12:53:50.166 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.166 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:50.166 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.167 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.167 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89661 2.e 2:7462a28b:::benchmark_data_smithi177_78104_object89660:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:53:50.167 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.887128+0000", 2024-05-15T12:53:50.167 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 274.207333656, 2024-05-15T12:53:50.167 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.14095980499999999, 2024-05-15T12:53:50.167 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:50.167 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:50.167 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:50.167 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:50.167 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:50.167 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89661 2024-05-15T12:53:50.167 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.167 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:50.167 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.168 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:50.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887128+0000", 2024-05-15T12:53:50.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.168 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:50.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887128+0000", 2024-05-15T12:53:50.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.168 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:50.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887130+0000", 2024-05-15T12:53:50.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7969999999999999e-06 2024-05-15T12:53:50.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:50.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887162+0000", 2024-05-15T12:53:50.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2780000000000001e-05 2024-05-15T12:53:50.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:50.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887180+0000", 2024-05-15T12:53:50.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7187999999999999e-05 2024-05-15T12:53:50.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:50.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887190+0000", 2024-05-15T12:53:50.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8980000000000008e-06 2024-05-15T12:53:50.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:50.170 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887253+0000", 2024-05-15T12:53:50.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3724000000000002e-05 2024-05-15T12:53:50.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:50.170 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887391+0000", 2024-05-15T12:53:50.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013803700000000001 2024-05-15T12:53:50.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:53:50.171 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887549+0000", 2024-05-15T12:53:50.171 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015766599999999999 2024-05-15T12:53:50.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.171 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.171 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:50.171 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.891069+0000", 2024-05-15T12:53:50.171 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035196989999999998 2024-05-15T12:53:50.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.171 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.171 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.171 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.894586+0000", 2024-05-15T12:53:50.171 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035178269999999998 2024-05-15T12:53:50.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.171 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.172 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.172 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.028009+0000", 2024-05-15T12:53:50.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.13342280400000001 2024-05-15T12:53:50.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.172 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:50.172 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.028054+0000", 2024-05-15T12:53:50.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5207000000000003e-05 2024-05-15T12:53:50.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.172 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:50.172 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.028088+0000", 2024-05-15T12:53:50.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3178e-05 2024-05-15T12:53:50.172 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.172 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:50.173 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.173 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89672 2.e 2:7301c129:::benchmark_data_smithi177_78104_object89671:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:53:50.173 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.899549+0000", 2024-05-15T12:53:50.173 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 274.19491254799999, 2024-05-15T12:53:50.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.13251970099999999, 2024-05-15T12:53:50.173 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:50.173 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:50.173 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:50.173 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:50.173 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:50.173 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89672 2024-05-15T12:53:50.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.174 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:50.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.174 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:50.174 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.899549+0000", 2024-05-15T12:53:50.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.174 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:50.174 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.899549+0000", 2024-05-15T12:53:50.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:50.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.899551+0000", 2024-05-15T12:53:50.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.826e-06 2024-05-15T12:53:50.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:50.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.899591+0000", 2024-05-15T12:53:50.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0422e-05 2024-05-15T12:53:50.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:50.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.899606+0000", 2024-05-15T12:53:50.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5233e-05 2024-05-15T12:53:50.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:50.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.899617+0000", 2024-05-15T12:53:50.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0186999999999999e-05 2024-05-15T12:53:50.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:50.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.899682+0000", 2024-05-15T12:53:50.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4869000000000005e-05 2024-05-15T12:53:50.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:50.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.899838+0000", 2024-05-15T12:53:50.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015638300000000001 2024-05-15T12:53:50.177 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.177 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.177 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:53:50.177 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.900004+0000", 2024-05-15T12:53:50.177 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000165835 2024-05-15T12:53:50.177 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.177 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.177 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.177 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.911032+0000", 2024-05-15T12:53:50.177 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011028692999999999 2024-05-15T12:53:50.177 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.177 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.177 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:50.177 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.917176+0000", 2024-05-15T12:53:50.178 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.006143333 2024-05-15T12:53:50.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.178 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.178 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.031955+0000", 2024-05-15T12:53:50.178 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.114779144 2024-05-15T12:53:50.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.178 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:50.178 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032020+0000", 2024-05-15T12:53:50.178 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4707000000000002e-05 2024-05-15T12:53:50.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.179 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:50.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032069+0000", 2024-05-15T12:53:50.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9069000000000001e-05 2024-05-15T12:53:50.179 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.179 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:50.179 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.179 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89679 2.e 2:74cc89e8:::benchmark_data_smithi177_78104_object89678:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:53:50.179 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.910531+0000", 2024-05-15T12:53:50.179 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 274.18393005600001, 2024-05-15T12:53:50.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.121660637, 2024-05-15T12:53:50.179 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:50.179 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:50.180 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:50.180 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:50.180 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:50.180 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89679 2024-05-15T12:53:50.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.180 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:50.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:50.180 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910531+0000", 2024-05-15T12:53:50.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:50.180 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910531+0000", 2024-05-15T12:53:50.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:50.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910533+0000", 2024-05-15T12:53:50.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.773e-06 2024-05-15T12:53:50.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:50.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910580+0000", 2024-05-15T12:53:50.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6854000000000002e-05 2024-05-15T12:53:50.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:50.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910595+0000", 2024-05-15T12:53:50.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5082999999999999e-05 2024-05-15T12:53:50.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.182 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:50.182 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910605+0000", 2024-05-15T12:53:50.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6029999999999994e-06 2024-05-15T12:53:50.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.182 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:50.182 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.911065+0000", 2024-05-15T12:53:50.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00046001800000000003 2024-05-15T12:53:50.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:50.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.911211+0000", 2024-05-15T12:53:50.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014574400000000001 2024-05-15T12:53:50.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:53:50.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.911363+0000", 2024-05-15T12:53:50.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015289500000000001 2024-05-15T12:53:50.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:50.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.924113+0000", 2024-05-15T12:53:50.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012749286 2024-05-15T12:53:50.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.184 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.184 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.184 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.931398+0000", 2024-05-15T12:53:50.184 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0072853279999999998 2024-05-15T12:53:50.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.184 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.184 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.184 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032107+0000", 2024-05-15T12:53:50.184 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.100708456 2024-05-15T12:53:50.185 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.185 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.185 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:50.185 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032159+0000", 2024-05-15T12:53:50.185 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2234000000000002e-05 2024-05-15T12:53:50.185 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.185 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.185 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:50.185 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032192+0000", 2024-05-15T12:53:50.185 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3362999999999999e-05 2024-05-15T12:53:50.185 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.185 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:50.185 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.186 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.186 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89680 2.e 2:71e6b157:::benchmark_data_smithi177_78104_object89679:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:53:50.186 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.910622+0000", 2024-05-15T12:53:50.186 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 274.18383996, 2024-05-15T12:53:50.186 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12167915, 2024-05-15T12:53:50.186 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:50.186 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:50.186 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:50.186 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:50.186 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:50.186 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89680 2024-05-15T12:53:50.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.186 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:50.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.187 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:50.187 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910622+0000", 2024-05-15T12:53:50.187 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.187 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:50.187 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910622+0000", 2024-05-15T12:53:50.187 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.187 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:50.187 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910622+0000", 2024-05-15T12:53:50.187 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6499999999999999e-07 2024-05-15T12:53:50.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:50.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910662+0000", 2024-05-15T12:53:50.188 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9436e-05 2024-05-15T12:53:50.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:50.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910670+0000", 2024-05-15T12:53:50.188 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4940000000000003e-06 2024-05-15T12:53:50.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:50.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910677+0000", 2024-05-15T12:53:50.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6039999999999998e-06 2024-05-15T12:53:50.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.189 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:50.189 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.914062+0000", 2024-05-15T12:53:50.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003385117 2024-05-15T12:53:50.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.189 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:50.189 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.914206+0000", 2024-05-15T12:53:50.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014453299999999999 2024-05-15T12:53:50.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.189 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:53:50.190 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.914360+0000", 2024-05-15T12:53:50.190 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000154146 2024-05-15T12:53:50.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:50.190 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.924120+0000", 2024-05-15T12:53:50.190 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0097599209999999995 2024-05-15T12:53:50.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.190 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.934155+0000", 2024-05-15T12:53:50.190 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0100348 2024-05-15T12:53:50.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.191 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.191 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032226+0000", 2024-05-15T12:53:50.191 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.098070750999999998 2024-05-15T12:53:50.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.191 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:50.191 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032269+0000", 2024-05-15T12:53:50.191 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3513000000000002e-05 2024-05-15T12:53:50.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.191 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:50.191 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032301+0000", 2024-05-15T12:53:50.191 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1269999999999997e-05 2024-05-15T12:53:50.191 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.192 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:50.192 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.192 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.192 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.192 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89690 2.e 2:7ba0758f:::benchmark_data_smithi177_78104_object89689:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:53:50.192 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.928382+0000", 2024-05-15T12:53:50.192 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 274.16607993899999, 2024-05-15T12:53:50.192 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10402787500000001, 2024-05-15T12:53:50.192 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:50.192 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:50.192 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:50.192 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:50.192 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:50.192 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89690 2024-05-15T12:53:50.193 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.193 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:50.193 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.193 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:50.193 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.928382+0000", 2024-05-15T12:53:50.193 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.193 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.193 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.193 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:50.193 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.928382+0000", 2024-05-15T12:53:50.193 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.193 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.193 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.193 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:50.194 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.928383+0000", 2024-05-15T12:53:50.194 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8610000000000001e-06 2024-05-15T12:53:50.194 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.194 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.194 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:50.194 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.928422+0000", 2024-05-15T12:53:50.194 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8152000000000003e-05 2024-05-15T12:53:50.194 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.194 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.194 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:50.194 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.928444+0000", 2024-05-15T12:53:50.194 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2830999999999999e-05 2024-05-15T12:53:50.194 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.194 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.195 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:50.195 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.928454+0000", 2024-05-15T12:53:50.195 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8740000000000007e-06 2024-05-15T12:53:50.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.195 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:50.195 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.931422+0000", 2024-05-15T12:53:50.195 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029681500000000001 2024-05-15T12:53:50.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.195 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:50.195 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.931545+0000", 2024-05-15T12:53:50.195 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000122574 2024-05-15T12:53:50.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.196 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:53:50.196 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.931686+0000", 2024-05-15T12:53:50.196 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014140100000000001 2024-05-15T12:53:50.196 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.196 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.196 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:50.196 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935371+0000", 2024-05-15T12:53:50.196 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0036848359999999999 2024-05-15T12:53:50.196 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.196 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.196 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.196 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.938923+0000", 2024-05-15T12:53:50.196 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035515859999999998 2024-05-15T12:53:50.196 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.196 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.197 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.197 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032334+0000", 2024-05-15T12:53:50.197 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.093410842999999993 2024-05-15T12:53:50.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.197 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.197 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:50.197 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032378+0000", 2024-05-15T12:53:50.197 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.384e-05 2024-05-15T12:53:50.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.197 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.197 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:50.197 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032409+0000", 2024-05-15T12:53:50.197 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1927000000000001e-05 2024-05-15T12:53:50.197 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.198 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:50.198 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.198 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.198 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.198 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89694 2.e 2:7ceb574e:::benchmark_data_smithi177_78104_object89693:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:53:50.198 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.935396+0000", 2024-05-15T12:53:50.198 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 274.15906579799997, 2024-05-15T12:53:50.198 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10441977, 2024-05-15T12:53:50.198 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:50.198 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:50.198 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:50.198 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:50.198 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:50.199 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89694 2024-05-15T12:53:50.199 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.199 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:50.199 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.199 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:50.199 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935396+0000", 2024-05-15T12:53:50.199 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.199 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.199 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.199 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:50.199 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935396+0000", 2024-05-15T12:53:50.199 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.199 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.199 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.200 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:50.200 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935397+0000", 2024-05-15T12:53:50.200 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.353e-06 2024-05-15T12:53:50.200 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.200 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.200 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:50.200 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935423+0000", 2024-05-15T12:53:50.200 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6291e-05 2024-05-15T12:53:50.200 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.200 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.200 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:50.200 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935436+0000", 2024-05-15T12:53:50.200 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2184000000000001e-05 2024-05-15T12:53:50.201 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.201 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.201 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:50.201 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935445+0000", 2024-05-15T12:53:50.201 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4020000000000008e-06 2024-05-15T12:53:50.201 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.201 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.201 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:50.201 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935511+0000", 2024-05-15T12:53:50.201 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5832000000000003e-05 2024-05-15T12:53:50.201 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.201 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.201 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:50.201 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935684+0000", 2024-05-15T12:53:50.202 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017362399999999999 2024-05-15T12:53:50.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.202 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.202 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:53:50.202 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935863+0000", 2024-05-15T12:53:50.202 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017888299999999999 2024-05-15T12:53:50.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.202 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.202 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:50.202 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.942570+0000", 2024-05-15T12:53:50.202 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0067064869999999997 2024-05-15T12:53:50.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.202 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.203 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.203 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.943101+0000", 2024-05-15T12:53:50.203 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00053091900000000003 2024-05-15T12:53:50.203 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.203 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.203 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.203 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.039698+0000", 2024-05-15T12:53:50.203 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.096597075000000004 2024-05-15T12:53:50.203 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.203 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.203 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:50.203 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.039761+0000", 2024-05-15T12:53:50.203 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2772000000000001e-05 2024-05-15T12:53:50.203 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.204 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.204 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:50.204 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.039816+0000", 2024-05-15T12:53:50.204 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4948000000000003e-05 2024-05-15T12:53:50.204 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.204 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:50.204 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.204 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.204 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.204 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89699 2.e 2:7fa3b3a9:::benchmark_data_smithi177_78104_object89698:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:53:50.204 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.939371+0000", 2024-05-15T12:53:50.204 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 274.15509004500001, 2024-05-15T12:53:50.204 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10057566799999999, 2024-05-15T12:53:50.204 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:53:50.205 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:53:50.205 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:53:50.205 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:53:50.205 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:53:50.205 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89699 2024-05-15T12:53:50.205 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.205 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:53:50.205 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.205 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:53:50.205 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939371+0000", 2024-05-15T12:53:50.205 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.205 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.205 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.205 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:53:50.205 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939371+0000", 2024-05-15T12:53:50.206 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:53:50.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.206 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.206 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:53:50.206 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939373+0000", 2024-05-15T12:53:50.206 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6700000000000001e-06 2024-05-15T12:53:50.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.206 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.206 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:53:50.206 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939415+0000", 2024-05-15T12:53:50.206 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1551000000000002e-05 2024-05-15T12:53:50.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.206 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.206 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:53:50.207 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939432+0000", 2024-05-15T12:53:50.207 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.768e-05 2024-05-15T12:53:50.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.207 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.207 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:53:50.207 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939441+0000", 2024-05-15T12:53:50.207 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7299999999999994e-06 2024-05-15T12:53:50.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.207 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.207 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:53:50.207 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939527+0000", 2024-05-15T12:53:50.207 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6098000000000001e-05 2024-05-15T12:53:50.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.207 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.207 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:53:50.208 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939679+0000", 2024-05-15T12:53:50.208 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015150299999999999 2024-05-15T12:53:50.208 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.208 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.208 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:53:50.208 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939825+0000", 2024-05-15T12:53:50.208 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000146725 2024-05-15T12:53:50.208 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.208 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.208 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:53:50.208 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.942947+0000", 2024-05-15T12:53:50.208 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031219279999999999 2024-05-15T12:53:50.208 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.208 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.208 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.209 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.943208+0000", 2024-05-15T12:53:50.209 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00026073099999999999 2024-05-15T12:53:50.209 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.209 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.209 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:53:50.209 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.039860+0000", 2024-05-15T12:53:50.209 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.096651440000000005 2024-05-15T12:53:50.209 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.209 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.209 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:53:50.209 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.039911+0000", 2024-05-15T12:53:50.209 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.13e-05 2024-05-15T12:53:50.209 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:53:50.209 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:53:50.210 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:53:50.210 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.039947+0000", 2024-05-15T12:53:50.210 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6312000000000001e-05 2024-05-15T12:53:50.210 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.210 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:50.210 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.210 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:53:50.210 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:53:50.210 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T12:53:50.211 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:49 smithi177 ceph-mon[44303]: pgmap v664: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-05-15T12:53:50.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:50.212 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:53:50.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:53:50.313 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:53:50.344 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:50.366 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:53:50.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:50.369 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:53:50.470 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:53:50.527 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:50.559 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:53:50.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:50.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:53:50.669 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:53:50.730 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:50.746 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T12:53:50.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:50 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4247075349' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:53:50.997 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 42 16 81721 81705 121.574 88.375 0.00940931 0.00822154 2024-05-15T12:53:51.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:50 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4247075349' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:53:51.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:50 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4247075349' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:53:51.200 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-05-15T12:53:51.512 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:53:51.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:51.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:53:51.618 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:53:51.670 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:51.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:51 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:53:51.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:51 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4247075349' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:53:51.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:51 smithi027 ceph-mon[39795]: osdmap e240: 12 total, 12 up, 11 in 2024-05-15T12:53:51.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:51 smithi027 ceph-mon[39795]: pgmap v666: 17 pgs: 17 active+clean; 4.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-05-15T12:53:51.997 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 43 16 83108 83092 120.762 86.6875 0.00925944 0.00827697 2024-05-15T12:53:52.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:51 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:53:52.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:51 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4247075349' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:53:52.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:51 smithi120 ceph-mon[43961]: osdmap e240: 12 total, 12 up, 11 in 2024-05-15T12:53:52.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:51 smithi120 ceph-mon[43961]: pgmap v666: 17 pgs: 17 active+clean; 4.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-05-15T12:53:52.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:51 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:53:52.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:51 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4247075349' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:53:52.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:51 smithi177 ceph-mon[44303]: osdmap e240: 12 total, 12 up, 11 in 2024-05-15T12:53:52.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:51 smithi177 ceph-mon[44303]: pgmap v666: 17 pgs: 17 active+clean; 4.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-05-15T12:53:52.244 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:53:52.997 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 44 16 84499 84483 119.994 86.9375 0.00791418 0.00832968 2024-05-15T12:53:53.997 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 45 16 85917 85901 119.296 88.625 0.0111843 0.00837919 2024-05-15T12:53:54.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:54 smithi027 ceph-mon[39795]: osdmap e241: 12 total, 12 up, 11 in 2024-05-15T12:53:54.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:54 smithi027 ceph-mon[39795]: pgmap v668: 17 pgs: 17 active+clean; 5.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-05-15T12:53:54.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:54 smithi120 ceph-mon[43961]: osdmap e241: 12 total, 12 up, 11 in 2024-05-15T12:53:54.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:54 smithi120 ceph-mon[43961]: pgmap v668: 17 pgs: 17 active+clean; 5.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-05-15T12:53:54.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:54 smithi177 ceph-mon[44303]: osdmap e241: 12 total, 12 up, 11 in 2024-05-15T12:53:54.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:54 smithi177 ceph-mon[44303]: pgmap v668: 17 pgs: 17 active+clean; 5.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-05-15T12:53:54.745 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T12:53:54.997 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 46 16 87302 87286 118.584 86.5625 0.0110451 0.00842824 2024-05-15T12:53:55.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:55 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:53:55.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:55 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:53:55.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:55 smithi027 ceph-mon[39795]: osdmap e242: 12 total, 12 up, 11 in 2024-05-15T12:53:55.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:55 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T12:53:55.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:55 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T12:53:55.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:55 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:53:55.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:55 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:53:55.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:55 smithi120 ceph-mon[43961]: osdmap e242: 12 total, 12 up, 11 in 2024-05-15T12:53:55.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:55 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T12:53:55.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:55 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T12:53:55.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:55 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:53:55.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:55 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:53:55.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:55 smithi177 ceph-mon[44303]: osdmap e242: 12 total, 12 up, 11 in 2024-05-15T12:53:55.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:55 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T12:53:55.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:55 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T12:53:55.998 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 47 16 88464 88448 117.606 72.625 0.00329495 0.00846895 2024-05-15T12:53:56.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:56 smithi120 ceph-mon[43961]: pgmap v670: 17 pgs: 17 active+clean; 5.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-05-15T12:53:56.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:56 smithi177 ceph-mon[44303]: pgmap v670: 17 pgs: 17 active+clean; 5.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-05-15T12:53:56.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:56 smithi027 ceph-mon[39795]: pgmap v670: 17 pgs: 17 active+clean; 5.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-05-15T12:53:56.762 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:53:56.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:56.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:53:56.875 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:53:56.907 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:56.934 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:53:56.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:56.937 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:53:56.997 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 48 16 89692 89676 116.755 76.75 0.0144773 0.00856089 2024-05-15T12:53:57.038 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:53:57.066 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:57.088 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:53:57.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:57.091 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:53:57.192 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:53:57.230 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:57.296 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:53:57.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:57.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:53:57.402 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:53:57.432 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:57.455 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:53:57.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:57.458 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:53:57.559 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:53:57.586 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:57.607 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:53:57.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:57.610 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:53:57.711 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:53:57.770 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:57.803 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:53:57.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:57.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:53:57.915 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:53:57.981 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:57.998 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 49 16 90875 90859 115.881 73.9375 0.00504836 0.0086252 2024-05-15T12:53:58.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:57 smithi027 ceph-mon[39795]: pgmap v671: 17 pgs: 17 active+clean; 5.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-05-15T12:53:58.021 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:53:58.022 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:57 smithi120 ceph-mon[43961]: pgmap v671: 17 pgs: 17 active+clean; 5.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-05-15T12:53:58.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:58.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:53:58.124 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:53:58.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:57 smithi177 ceph-mon[44303]: pgmap v671: 17 pgs: 17 active+clean; 5.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-05-15T12:53:58.181 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:58.207 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:53:58.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:58.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:53:58.312 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:53:58.390 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:58.431 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:53:58.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:58.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:53:58.444 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:53:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:53:58.533 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:53:58.545 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:53:58.574 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:53:58.599 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:58.626 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:53:58.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:58.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:53:58.726 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:58 smithi027 ceph-mon[39795]: 3.e scrub starts 2024-05-15T12:53:58.726 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:58 smithi027 ceph-mon[39795]: osdmap e243: 12 total, 12 up, 11 in 2024-05-15T12:53:58.726 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:58 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2097961602' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T12:53:58.726 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:58 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T12:53:58.726 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:58 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2387235823' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:53:58.730 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:53:58.761 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:58.782 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:53:58.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:58.785 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:53:58.807 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:53:58.848 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:53:58.882 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:53:58.886 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:53:58.915 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:58.927 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:53:58.937 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:53:58.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:58.940 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:53:58.969 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:53:58.998 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 50 16 92016 92000 114.99 71.3125 0.00540893 0.00869128 2024-05-15T12:53:59.013 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:53:59.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:58 smithi120 ceph-mon[43961]: 3.e scrub starts 2024-05-15T12:53:59.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:58 smithi120 ceph-mon[43961]: osdmap e243: 12 total, 12 up, 11 in 2024-05-15T12:53:59.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:58 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2097961602' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T12:53:59.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:58 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T12:53:59.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:58 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2387235823' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:53:59.042 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:53:59.042 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.3 to 1 (10000) 2024-05-15T12:53:59.058 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:53:59.098 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:53:59.099 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:59.128 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:53:59.130 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:53:59.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:53:59.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:53:59.172 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:53:59.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:58 smithi177 ceph-mon[44303]: 3.e scrub starts 2024-05-15T12:53:59.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:58 smithi177 ceph-mon[44303]: osdmap e243: 12 total, 12 up, 11 in 2024-05-15T12:53:59.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:58 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2097961602' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T12:53:59.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:58 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T12:53:59.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:58 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2387235823' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:53:59.181 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:53:59.181 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:53:59.182 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:53:59.182 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:53:59.182 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:53:59.182 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:53:59.182 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:53:59.182 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:53:59.182 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:53:59.182 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:53:59.182 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:53:59.182 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:53:59.239 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:53:59.309 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:53:59.998 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 51 16 93267 93251 114.268 78.1875 0.00497523 0.00874802 2024-05-15T12:54:00.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:59 smithi027 ceph-mon[39795]: 3.e scrub ok 2024-05-15T12:54:00.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:59 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T12:54:00.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:59 smithi027 ceph-mon[39795]: osdmap e244: 12 total, 12 up, 11 in 2024-05-15T12:54:00.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:53:59 smithi027 ceph-mon[39795]: pgmap v674: 17 pgs: 17 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-05-15T12:54:00.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:59 smithi120 ceph-mon[43961]: 3.e scrub ok 2024-05-15T12:54:00.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:59 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T12:54:00.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:59 smithi120 ceph-mon[43961]: osdmap e244: 12 total, 12 up, 11 in 2024-05-15T12:54:00.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:53:59 smithi120 ceph-mon[43961]: pgmap v674: 17 pgs: 17 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-05-15T12:54:00.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:59 smithi177 ceph-mon[44303]: 3.e scrub ok 2024-05-15T12:54:00.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:59 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T12:54:00.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:59 smithi177 ceph-mon[44303]: osdmap e244: 12 total, 12 up, 11 in 2024-05-15T12:54:00.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:53:59 smithi177 ceph-mon[44303]: pgmap v674: 17 pgs: 17 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-05-15T12:54:00.883 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:54:00.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:00.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:54:00.991 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:54:00.998 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 52 16 94576 94560 113.644 81.8125 0.0145799 0.00879577 2024-05-15T12:54:01.037 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:01.074 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:54:01.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:01.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:54:01.180 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:54:01.228 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:01.998 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 53 16 95855 95839 113.008 79.9375 0.0124068 0.00884515 2024-05-15T12:54:02.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:01 smithi027 ceph-mon[39795]: pgmap v675: 17 pgs: 17 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-05-15T12:54:02.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:01 smithi120 ceph-mon[43961]: pgmap v675: 17 pgs: 17 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-05-15T12:54:02.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:01 smithi177 ceph-mon[44303]: pgmap v675: 17 pgs: 17 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-05-15T12:54:02.998 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 54 16 97003 96987 112.244 71.75 0.0200726 0.00890485 2024-05-15T12:54:03.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:02 smithi027 ceph-mon[39795]: 3.7 scrub starts 2024-05-15T12:54:03.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:02 smithi120 ceph-mon[43961]: 3.7 scrub starts 2024-05-15T12:54:03.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:02 smithi177 ceph-mon[44303]: 3.7 scrub starts 2024-05-15T12:54:03.597 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:54:03.998 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 55 16 97924 97908 111.249 57.5625 0.00774765 0.0089847 2024-05-15T12:54:04.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:04 smithi027 ceph-mon[39795]: 3.b scrub starts 2024-05-15T12:54:04.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:04 smithi027 ceph-mon[39795]: osdmap e245: 12 total, 12 up, 11 in 2024-05-15T12:54:04.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:04 smithi027 ceph-mon[39795]: pgmap v677: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-05-15T12:54:04.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:04 smithi120 ceph-mon[43961]: 3.b scrub starts 2024-05-15T12:54:04.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:04 smithi120 ceph-mon[43961]: osdmap e245: 12 total, 12 up, 11 in 2024-05-15T12:54:04.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:04 smithi120 ceph-mon[43961]: pgmap v677: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-05-15T12:54:04.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:04 smithi177 ceph-mon[44303]: 3.b scrub starts 2024-05-15T12:54:04.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:04 smithi177 ceph-mon[44303]: osdmap e245: 12 total, 12 up, 11 in 2024-05-15T12:54:04.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:04 smithi177 ceph-mon[44303]: pgmap v677: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-05-15T12:54:04.998 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 56 16 99087 99071 110.561 72.6875 0.0176928 0.00904131 2024-05-15T12:54:05.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:05 smithi177 ceph-mon[44303]: 3.7 scrub ok 2024-05-15T12:54:05.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:05 smithi177 ceph-mon[44303]: 3.b scrub ok 2024-05-15T12:54:05.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:05 smithi027 ceph-mon[39795]: 3.7 scrub ok 2024-05-15T12:54:05.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:05 smithi027 ceph-mon[39795]: 3.b scrub ok 2024-05-15T12:54:05.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:05 smithi120 ceph-mon[43961]: 3.7 scrub ok 2024-05-15T12:54:05.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:05 smithi120 ceph-mon[43961]: 3.b scrub ok 2024-05-15T12:54:05.998 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 57 16 100352 100336 110.008 79.0625 0.00935874 0.00908624 2024-05-15T12:54:06.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:06 smithi177 ceph-mon[44303]: pgmap v678: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-05-15T12:54:06.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:06 smithi027 ceph-mon[39795]: pgmap v678: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-05-15T12:54:06.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:06 smithi120 ceph-mon[43961]: pgmap v678: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-05-15T12:54:06.994 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:54:06.998 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 58 16 101569 101553 109.423 76.0625 0.0148501 0.00913506 2024-05-15T12:54:07.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:07.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:54:07.212 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:54:07.245 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:07.270 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:54:07.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:07.276 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:54:07.377 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:54:07.439 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:07.468 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:54:07.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:07.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:54:07.575 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:54:07.634 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:07.998 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 59 16 102673 102657 108.737 69 0.0119355 0.00919296 2024-05-15T12:54:08.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:08 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:54:08.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:08 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:54:08.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:08 smithi177 ceph-mon[44303]: osdmap e246: 12 total, 12 up, 11 in 2024-05-15T12:54:08.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:08 smithi177 ceph-mon[44303]: 3.4 scrub starts 2024-05-15T12:54:08.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:08 smithi177 ceph-mon[44303]: pgmap v680: 17 pgs: 17 active+clean; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-05-15T12:54:08.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:08 smithi177 ceph-mon[44303]: osdmap e247: 12 total, 12 up, 11 in 2024-05-15T12:54:08.470 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:08 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:54:08.470 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:08 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:54:08.470 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:08 smithi027 ceph-mon[39795]: osdmap e246: 12 total, 12 up, 11 in 2024-05-15T12:54:08.470 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:08 smithi027 ceph-mon[39795]: 3.4 scrub starts 2024-05-15T12:54:08.470 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:08 smithi027 ceph-mon[39795]: pgmap v680: 17 pgs: 17 active+clean; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-05-15T12:54:08.470 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:08 smithi027 ceph-mon[39795]: osdmap e247: 12 total, 12 up, 11 in 2024-05-15T12:54:08.509 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:54:08.510 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 4 1 2024-05-15T12:54:08.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:08.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:54:08.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:08 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:54:08.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:08 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:54:08.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:08 smithi120 ceph-mon[43961]: osdmap e246: 12 total, 12 up, 11 in 2024-05-15T12:54:08.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:08 smithi120 ceph-mon[43961]: 3.4 scrub starts 2024-05-15T12:54:08.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:08 smithi120 ceph-mon[43961]: pgmap v680: 17 pgs: 17 active+clean; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-05-15T12:54:08.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:08 smithi120 ceph-mon[43961]: osdmap e247: 12 total, 12 up, 11 in 2024-05-15T12:54:08.618 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:54:08.645 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:08.666 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:54:08.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:08.669 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:54:08.748 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:54:08.770 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:54:08.804 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:08.829 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:54:08.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:08.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:54:08.936 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:54:08.980 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:08.998 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T12:54:09.009653+0000 min lat: 0.00112991 max lat: 0.224156 avg lat: 0.00926692 2024-05-15T12:54:08.999 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T12:54:08.999 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 60 16 103571 103555 107.86 56.125 0.00823699 0.00926692 2024-05-15T12:54:09.012 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:54:09.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:09.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:54:09.120 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:54:09.162 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:09.198 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:54:09.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:09.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:54:09.303 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:54:09.333 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:09.356 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:54:09.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:09.362 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:54:09.463 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:54:09.491 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:09.512 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:54:09.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:09.516 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:54:09.617 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:54:09.635 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:09 smithi027 ceph-mon[39795]: 3.f scrub starts 2024-05-15T12:54:09.669 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:09.720 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:54:09.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:09.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:54:09.740 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T12:54:09.740 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:54:09.827 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:54:09.855 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:09.876 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:54:09.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:09.879 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:54:09.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:09 smithi177 ceph-mon[44303]: 3.f scrub starts 2024-05-15T12:54:09.980 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:54:09.998 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 61 16 104603 104587 107.15 64.5 0.0221919 0.00932889 2024-05-15T12:54:10.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:09 smithi120 ceph-mon[43961]: 3.f scrub starts 2024-05-15T12:54:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:10.151 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:54:10.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:10.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:54:10.262 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:54:10.293 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:10.323 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:54:10.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:10.326 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:54:10.427 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:54:10.458 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:10.480 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:54:10.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:10.484 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:54:10.585 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:54:10.637 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:10.681 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T12:54:10.714 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:10 smithi027 ceph-mon[39795]: 3.4 scrub ok 2024-05-15T12:54:10.714 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:10 smithi027 ceph-mon[39795]: 3.8 scrub starts 2024-05-15T12:54:10.715 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:10 smithi027 ceph-mon[39795]: pgmap v682: 17 pgs: 17 active+clean; 6.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-05-15T12:54:10.715 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:10 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3657043047' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:54:10.715 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:10 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:54:10.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:10 smithi177 ceph-mon[44303]: 3.4 scrub ok 2024-05-15T12:54:10.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:10 smithi177 ceph-mon[44303]: 3.8 scrub starts 2024-05-15T12:54:10.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:10 smithi177 ceph-mon[44303]: pgmap v682: 17 pgs: 17 active+clean; 6.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-05-15T12:54:10.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:10 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3657043047' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:54:10.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:10 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:54:10.999 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 62 16 105616 105600 106.442 63.3125 0.00867824 0.00939085 2024-05-15T12:54:11.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:10 smithi120 ceph-mon[43961]: 3.4 scrub ok 2024-05-15T12:54:11.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:10 smithi120 ceph-mon[43961]: 3.8 scrub starts 2024-05-15T12:54:11.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:10 smithi120 ceph-mon[43961]: pgmap v682: 17 pgs: 17 active+clean; 6.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-05-15T12:54:11.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:10 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3657043047' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:54:11.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:10 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:54:11.117 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-05-15T12:54:11.408 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:54:11.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:11.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:54:11.518 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:54:11.547 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:11.573 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:54:11.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:11.576 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:54:11.677 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:54:11.706 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:11.727 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:54:11.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:11.730 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:54:11.831 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:54:11.857 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:11 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:54:11.857 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:11 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T12:54:11.857 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:11 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:54:11.858 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:11 smithi177 ceph-mon[44303]: osdmap e248: 12 total, 12 up, 11 in 2024-05-15T12:54:11.858 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:11 smithi177 ceph-mon[44303]: 3.f scrub ok 2024-05-15T12:54:11.859 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:11.874 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:54:11.882 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:54:11.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:11.887 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:54:11.988 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:54:11.998 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 63 16 106724 106708 105.852 69.25 0.0201578 0.00944329 2024-05-15T12:54:12.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:11 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:54:12.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:11 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T12:54:12.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:11 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:54:12.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:11 smithi027 ceph-mon[39795]: osdmap e248: 12 total, 12 up, 11 in 2024-05-15T12:54:12.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:11 smithi027 ceph-mon[39795]: 3.f scrub ok 2024-05-15T12:54:12.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:11 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:54:12.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:11 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T12:54:12.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:11 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:54:12.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:11 smithi120 ceph-mon[43961]: osdmap e248: 12 total, 12 up, 11 in 2024-05-15T12:54:12.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:11 smithi120 ceph-mon[43961]: 3.f scrub ok 2024-05-15T12:54:12.057 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:12.095 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:54:12.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:12.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:54:12.202 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:54:12.252 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:12.288 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:54:12.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:12.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:54:12.396 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:54:12.439 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:12.482 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:54:12.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:12.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:54:12.587 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:54:12.663 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:12.756 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:54:12.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:12.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:54:12.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:12 smithi120 ceph-mon[43961]: 3.8 scrub ok 2024-05-15T12:54:12.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:12 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:54:12.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:12 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:54:12.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:12 smithi120 ceph-mon[43961]: osdmap e249: 12 total, 12 up, 11 in 2024-05-15T12:54:12.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:12 smithi120 ceph-mon[43961]: pgmap v685: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 62 MiB/s wr, 992 op/s 2024-05-15T12:54:12.864 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:54:12.905 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:12.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:12 smithi177 ceph-mon[44303]: 3.8 scrub ok 2024-05-15T12:54:12.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:12 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:54:12.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:12 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:54:12.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:12 smithi177 ceph-mon[44303]: osdmap e249: 12 total, 12 up, 11 in 2024-05-15T12:54:12.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:12 smithi177 ceph-mon[44303]: pgmap v685: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 62 MiB/s wr, 992 op/s 2024-05-15T12:54:12.931 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:54:12.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:12.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:54:12.999 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 64 16 107834 107818 105.282 69.375 0.0172711 0.00949432 2024-05-15T12:54:13.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:12 smithi027 ceph-mon[39795]: 3.8 scrub ok 2024-05-15T12:54:13.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:12 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:54:13.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:12 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:54:13.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:12 smithi027 ceph-mon[39795]: osdmap e249: 12 total, 12 up, 11 in 2024-05-15T12:54:13.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:12 smithi027 ceph-mon[39795]: pgmap v685: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 62 MiB/s wr, 992 op/s 2024-05-15T12:54:13.038 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:54:13.066 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:13.239 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:54:13.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:13.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:54:13.344 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:54:13.391 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:13.414 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:54:13.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:13.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:54:13.520 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:54:13.553 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:13.576 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:54:13.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:13.579 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:54:13.680 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:54:13.708 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:13.733 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:54:13.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:13.738 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:54:13.839 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:54:13.901 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:13.999 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 65 16 108908 108892 104.695 67.125 0.00780933 0.00954756 2024-05-15T12:54:14.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:14 smithi027 ceph-mon[39795]: pgmap v686: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-05-15T12:54:14.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:14 smithi120 ceph-mon[43961]: pgmap v686: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-05-15T12:54:14.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:14 smithi177 ceph-mon[44303]: pgmap v686: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-05-15T12:54:14.592 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:54:14.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:14.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:54:14.701 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:54:14.732 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:14.755 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:54:14.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:14.758 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:54:14.812 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T12:54:14.859 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:54:14.926 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:14.999 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 66 16 110003 109987 104.145 68.4375 0.014517 0.00959808 2024-05-15T12:54:15.018 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:15 smithi177 ceph-mon[44303]: 3.3 scrub starts 2024-05-15T12:54:15.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:15 smithi027 ceph-mon[39795]: 3.3 scrub starts 2024-05-15T12:54:15.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:15 smithi120 ceph-mon[43961]: 3.3 scrub starts 2024-05-15T12:54:15.505 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:54:15.506 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:54:15.506 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:54:15.511 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:54:15.545 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:54:15.587 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:54:15.631 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:54:15.691 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:54:15.721 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:54:15.757 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:54:15.800 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:54:15.803 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:54:15.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:15.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:54:15.843 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:54:15.858 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:54:15.858 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:54:15.858 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:54:15.858 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:54:15.858 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:54:15.858 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:54:15.858 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:54:15.859 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:54:15.859 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:54:15.859 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:54:15.859 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:54:15.859 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:54:15.884 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:54:15.910 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:54:15.993 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:15.999 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 67 16 110947 110931 103.471 59 0.0176066 0.00965938 2024-05-15T12:54:16.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:16 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3053847515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:54:16.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:16 smithi027 ceph-mon[39795]: pgmap v687: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-05-15T12:54:16.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:16 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3053847515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:54:16.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:16 smithi177 ceph-mon[44303]: pgmap v687: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-05-15T12:54:16.454 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:54:16.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:16.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:54:16.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:16 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3053847515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:54:16.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:16 smithi120 ceph-mon[43961]: pgmap v687: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-05-15T12:54:16.559 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:54:16.612 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:16.657 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:54:16.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:16.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:54:16.762 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:54:16.789 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:16.812 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:54:16.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:16.815 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:54:16.916 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:54:16.976 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:16.999 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 68 15 112011 111996 102.929 66.5625 0.0147941 0.00971199 2024-05-15T12:54:17.002 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:54:17.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:17.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:54:17.054 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:17 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3122530189' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T12:54:17.054 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:17 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T12:54:17.054 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:17 smithi177 ceph-mon[44303]: 3.3 scrub ok 2024-05-15T12:54:17.069 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.4 to 1 (10000) 2024-05-15T12:54:17.111 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:54:17.137 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:54:17.163 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:17.189 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:54:17.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:17.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:54:17.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:17 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3122530189' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T12:54:17.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:17 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T12:54:17.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:17 smithi120 ceph-mon[43961]: 3.3 scrub ok 2024-05-15T12:54:17.295 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:54:17.323 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:17.346 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:54:17.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:17.350 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:54:17.451 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:54:17.481 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:17.505 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:54:17.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:17.508 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:54:17.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:17 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3122530189' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T12:54:17.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:17 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T12:54:17.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:17 smithi027 ceph-mon[39795]: 3.3 scrub ok 2024-05-15T12:54:17.609 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:54:17.663 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:17.693 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:54:17.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:17.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:54:17.799 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:54:17.861 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:17.999 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 69 16 113094 113078 102.417 67.625 0.00859138 0.00975936 2024-05-15T12:54:18.236 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:54:18.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:18.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:54:18.348 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:54:18.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:18 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T12:54:18.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:18 smithi027 ceph-mon[39795]: osdmap e250: 12 total, 12 up, 11 in 2024-05-15T12:54:18.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:18 smithi027 ceph-mon[39795]: pgmap v689: 17 pgs: 17 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 62 MiB/s wr, 984 op/s 2024-05-15T12:54:18.405 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:18.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:18 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T12:54:18.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:18 smithi177 ceph-mon[44303]: osdmap e250: 12 total, 12 up, 11 in 2024-05-15T12:54:18.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:18 smithi177 ceph-mon[44303]: pgmap v689: 17 pgs: 17 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 62 MiB/s wr, 984 op/s 2024-05-15T12:54:18.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:18 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T12:54:18.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:18 smithi120 ceph-mon[43961]: osdmap e250: 12 total, 12 up, 11 in 2024-05-15T12:54:18.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:18 smithi120 ceph-mon[43961]: pgmap v689: 17 pgs: 17 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 62 MiB/s wr, 984 op/s 2024-05-15T12:54:18.820 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 5 1 2024-05-15T12:54:18.999 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 70 16 114169 114153 101.914 67.1875 0.0115025 0.00980787 2024-05-15T12:54:19.370 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:54:19.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:19.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:54:19.481 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:54:19.533 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:19.656 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:19 smithi027 ceph-mon[39795]: 3.d scrub starts 2024-05-15T12:54:19.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:19 smithi177 ceph-mon[44303]: 3.d scrub starts 2024-05-15T12:54:20.001 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 71 16 115145 115129 101.337 61 0.0175675 0.00986407 2024-05-15T12:54:20.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:19 smithi120 ceph-mon[43961]: 3.d scrub starts 2024-05-15T12:54:20.276 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:54:20.506 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:54:20.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:20.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:54:20.612 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:54:20.647 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:20.774 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:54:20.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:20.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:54:20.880 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:54:20.925 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:20 smithi027 ceph-mon[39795]: 3.a scrub starts 2024-05-15T12:54:20.925 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:20 smithi027 ceph-mon[39795]: pgmap v690: 17 pgs: 17 active+clean; 6.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-05-15T12:54:20.925 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:20 smithi027 ceph-mon[39795]: 3.9 scrub starts 2024-05-15T12:54:20.927 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:20.999 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 72 16 115977 115961 100.652 52 0.00667898 0.00992866 2024-05-15T12:54:21.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:20 smithi120 ceph-mon[43961]: 3.a scrub starts 2024-05-15T12:54:21.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:20 smithi120 ceph-mon[43961]: pgmap v690: 17 pgs: 17 active+clean; 6.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-05-15T12:54:21.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:20 smithi120 ceph-mon[43961]: 3.9 scrub starts 2024-05-15T12:54:21.150 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:54:21.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:21.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:54:21.164 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:54:21.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:20 smithi177 ceph-mon[44303]: 3.a scrub starts 2024-05-15T12:54:21.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:20 smithi177 ceph-mon[44303]: pgmap v690: 17 pgs: 17 active+clean; 6.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-05-15T12:54:21.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:20 smithi177 ceph-mon[44303]: 3.9 scrub starts 2024-05-15T12:54:21.255 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:54:21.284 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:21.306 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:54:21.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:21.309 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:54:21.410 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:54:21.487 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:21.546 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:54:21.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:21.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:54:21.657 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:54:21.681 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T12:54:21.681 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T12:54:21.682 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-05-15T12:54:21.682 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-05-15T12:54:21.682 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T12:54:21.693 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:21.716 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:54:21.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:21.719 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:54:21.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:21 smithi027 ceph-mon[39795]: 3.1 scrub starts 2024-05-15T12:54:21.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:21 smithi027 ceph-mon[39795]: 3.d scrub ok 2024-05-15T12:54:21.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:21 smithi027 ceph-mon[39795]: 3.9 scrub ok 2024-05-15T12:54:21.819 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:54:21.862 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:21.904 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:54:21.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:21.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:54:22.000 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 73 16 116771 116755 99.9529 49.625 0.00517374 0.010001 2024-05-15T12:54:22.011 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:54:22.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:21 smithi120 ceph-mon[43961]: 3.1 scrub starts 2024-05-15T12:54:22.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:21 smithi120 ceph-mon[43961]: 3.d scrub ok 2024-05-15T12:54:22.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:21 smithi120 ceph-mon[43961]: 3.9 scrub ok 2024-05-15T12:54:22.055 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:22.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:21 smithi177 ceph-mon[44303]: 3.1 scrub starts 2024-05-15T12:54:22.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:21 smithi177 ceph-mon[44303]: 3.d scrub ok 2024-05-15T12:54:22.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:21 smithi177 ceph-mon[44303]: 3.9 scrub ok 2024-05-15T12:54:22.265 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:54:22.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:22.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:54:22.374 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:54:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:22.746 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T12:54:22.914 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-05-15T12:54:23.000 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 74 16 117772 117756 99.4475 62.5625 0.0242618 0.0100517 2024-05-15T12:54:23.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:22 smithi027 ceph-mon[39795]: 3.1 scrub ok 2024-05-15T12:54:23.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:22 smithi027 ceph-mon[39795]: pgmap v691: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.0 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-05-15T12:54:23.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:22 smithi027 ceph-mon[39795]: 3.a scrub ok 2024-05-15T12:54:23.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:22 smithi027 ceph-mon[39795]: 3.5 scrub starts 2024-05-15T12:54:23.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:22 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1785284467' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:54:23.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:22 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:54:23.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:22 smithi120 ceph-mon[43961]: 3.1 scrub ok 2024-05-15T12:54:23.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:22 smithi120 ceph-mon[43961]: pgmap v691: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.0 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-05-15T12:54:23.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:22 smithi120 ceph-mon[43961]: 3.a scrub ok 2024-05-15T12:54:23.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:22 smithi120 ceph-mon[43961]: 3.5 scrub starts 2024-05-15T12:54:23.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:22 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1785284467' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:54:23.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:22 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:54:23.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:22 smithi177 ceph-mon[44303]: 3.1 scrub ok 2024-05-15T12:54:23.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:22 smithi177 ceph-mon[44303]: pgmap v691: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.0 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-05-15T12:54:23.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:22 smithi177 ceph-mon[44303]: 3.a scrub ok 2024-05-15T12:54:23.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:22 smithi177 ceph-mon[44303]: 3.5 scrub starts 2024-05-15T12:54:23.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:22 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1785284467' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:54:23.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:22 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:54:23.911 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:54:24.000 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 75 16 118838 118822 99.0098 66.625 0.0186083 0.0100959 2024-05-15T12:54:24.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:23 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:54:24.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:23 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:54:24.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:23 smithi027 ceph-mon[39795]: osdmap e251: 12 total, 12 up, 11 in 2024-05-15T12:54:24.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:23 smithi027 ceph-mon[39795]: pgmap v693: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.1 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-05-15T12:54:24.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:23 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:54:24.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:23 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:54:24.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:23 smithi120 ceph-mon[43961]: osdmap e251: 12 total, 12 up, 11 in 2024-05-15T12:54:24.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:23 smithi120 ceph-mon[43961]: pgmap v693: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.1 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-05-15T12:54:24.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:23 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:54:24.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:23 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:54:24.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:23 smithi177 ceph-mon[44303]: osdmap e251: 12 total, 12 up, 11 in 2024-05-15T12:54:24.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:23 smithi177 ceph-mon[44303]: pgmap v693: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.1 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-05-15T12:54:25.000 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 76 16 119823 119807 98.517 61.5625 0.0132676 0.0101467 2024-05-15T12:54:25.059 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:54:25.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:25.067 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:25 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:54:25.067 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:25 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:54:25.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:54:25.168 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:54:25.203 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:25.231 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:54:25.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:25.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:54:25.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:25 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:54:25.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:25 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:54:25.339 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:54:25.366 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:25 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:54:25.366 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:25 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:54:25.368 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:25.390 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:54:25.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:25.396 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:54:25.497 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:54:25.566 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:25.612 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:54:25.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:25.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:54:25.718 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:54:25.777 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:25.815 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:54:25.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:25.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:54:25.924 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:54:25.977 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:26.000 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 77 16 120873 120857 98.0898 65.625 0.0115264 0.0101911 2024-05-15T12:54:26.256 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:54:26.261 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:26 smithi027 ceph-mon[39795]: osdmap e252: 12 total, 12 up, 11 in 2024-05-15T12:54:26.261 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:26 smithi027 ceph-mon[39795]: pgmap v695: 17 pgs: 17 active+clean; 7.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-05-15T12:54:26.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:26.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:54:26.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:26 smithi120 ceph-mon[43961]: osdmap e252: 12 total, 12 up, 11 in 2024-05-15T12:54:26.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:26 smithi120 ceph-mon[43961]: pgmap v695: 17 pgs: 17 active+clean; 7.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-05-15T12:54:26.363 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:54:26.428 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:26.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:26 smithi177 ceph-mon[44303]: osdmap e252: 12 total, 12 up, 11 in 2024-05-15T12:54:26.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:26 smithi177 ceph-mon[44303]: pgmap v695: 17 pgs: 17 active+clean; 7.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-05-15T12:54:26.513 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:54:26.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:26.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:54:26.622 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:54:26.707 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:26.755 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:54:26.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:26.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:54:26.870 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:54:26.897 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:26.917 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:54:26.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:26.920 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:54:27.000 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 78 16 121903 121887 97.6575 64.375 0.0121946 0.0102363 2024-05-15T12:54:27.020 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:54:27.076 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:27.094 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.5 to 1 (10000) 2024-05-15T12:54:27.103 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:54:27.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:27.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:54:27.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:27 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2622824685' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T12:54:27.211 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:54:27.282 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:27.306 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:54:27.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:27.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:54:27.411 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:54:27.463 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:27 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2622824685' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T12:54:27.468 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:54:27.469 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:27.497 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:54:27.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:27.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:54:27.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:27 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2622824685' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T12:54:27.602 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:54:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:27.702 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:54:27.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:27.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:54:27.812 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:54:27.841 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T12:54:27.854 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:27.890 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:54:27.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:27.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:54:27.997 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:54:28.000 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 79 16 122917 122901 97.2235 63.375 0.0183556 0.0102816 2024-05-15T12:54:28.029 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:28.048 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:54:28.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:28.052 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:54:28.152 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:54:28.193 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:28 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2622824685' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T12:54:28.193 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:28 smithi120 ceph-mon[43961]: osdmap e253: 12 total, 12 up, 11 in 2024-05-15T12:54:28.193 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:28 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:54:28.194 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:28 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:54:28.194 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:28 smithi120 ceph-mon[43961]: pgmap v697: 17 pgs: 17 active+clean; 7.3 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 60 MiB/s wr, 962 op/s 2024-05-15T12:54:28.194 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:28.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:28 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2622824685' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T12:54:28.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:28 smithi027 ceph-mon[39795]: osdmap e253: 12 total, 12 up, 11 in 2024-05-15T12:54:28.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:28 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:54:28.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:28 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:54:28.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:28 smithi027 ceph-mon[39795]: pgmap v697: 17 pgs: 17 active+clean; 7.3 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 60 MiB/s wr, 962 op/s 2024-05-15T12:54:28.233 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:54:28.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:28.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:54:28.266 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 6 1 2024-05-15T12:54:28.339 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:54:28.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:28 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2622824685' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T12:54:28.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:28 smithi177 ceph-mon[44303]: osdmap e253: 12 total, 12 up, 11 in 2024-05-15T12:54:28.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:28 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:54:28.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:28 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:54:28.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:28 smithi177 ceph-mon[44303]: pgmap v697: 17 pgs: 17 active+clean; 7.3 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 60 MiB/s wr, 962 op/s 2024-05-15T12:54:28.368 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:28.388 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:54:28.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:28.391 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:54:28.492 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:54:28.542 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:28.577 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:54:28.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:28.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:54:28.685 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:54:28.716 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:28.740 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:54:28.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:28.743 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:54:28.844 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:54:28.906 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:28.948 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:54:28.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:28.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:54:29.000 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T12:54:29.011298+0000 min lat: 0.00112991 max lat: 0.225661 avg lat: 0.0103303 2024-05-15T12:54:29.000 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T12:54:29.000 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 80 16 123886 123870 96.7651 60.5625 0.012919 0.0103303 2024-05-15T12:54:29.058 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:54:29.086 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:29.108 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:54:29.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:29.112 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:54:29.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:29 smithi027 ceph-mon[39795]: osdmap e254: 12 total, 12 up, 11 in 2024-05-15T12:54:29.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:29 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2017761986' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:54:29.213 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:54:29.243 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:29.265 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:54:29.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:29.267 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:54:29.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:29 smithi120 ceph-mon[43961]: osdmap e254: 12 total, 12 up, 11 in 2024-05-15T12:54:29.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:29 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2017761986' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:54:29.368 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:54:29.406 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:29.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:29 smithi177 ceph-mon[44303]: osdmap e254: 12 total, 12 up, 11 in 2024-05-15T12:54:29.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:29 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2017761986' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:54:29.438 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:54:29.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:29.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:54:29.546 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:54:29.633 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:29.656 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:54:29.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:29.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:54:29.765 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:54:29.822 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:29.854 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:54:29.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:29.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:54:29.959 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:54:30.001 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 81 16 124891 124875 96.3459 62.8125 0.0135885 0.0103758 2024-05-15T12:54:30.066 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:30.098 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:54:30.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:30.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:54:30.208 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:54:30.239 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:30.262 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:54:30.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:30.265 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:54:30.366 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:54:30.378 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:54:30.378 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:54:30.378 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:54:30.379 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:54:30.379 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:54:30.379 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:54:30.379 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:54:30.379 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:54:30.379 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:54:30.379 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:54:30.379 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:54:30.379 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:54:30.379 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:54:30.379 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:54:30.379 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:54:30.380 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:54:30.380 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:54:30.380 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:54:30.380 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:54:30.380 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:54:30.380 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:54:30.380 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:54:30.380 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:54:30.380 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:54:30.403 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:30 smithi120 ceph-mon[43961]: pgmap v699: 17 pgs: 17 active+clean; 7.5 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 58 MiB/s wr, 931 op/s 2024-05-15T12:54:30.405 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:30.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:30 smithi177 ceph-mon[44303]: pgmap v699: 17 pgs: 17 active+clean; 7.5 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 58 MiB/s wr, 931 op/s 2024-05-15T12:54:30.450 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:54:30.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:30.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:54:30.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:30 smithi027 ceph-mon[39795]: pgmap v699: 17 pgs: 17 active+clean; 7.5 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 58 MiB/s wr, 931 op/s 2024-05-15T12:54:30.566 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:54:30.595 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:30.622 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:54:30.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:30.625 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:54:30.726 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:54:30.754 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:30.775 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:54:30.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:30.779 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:54:30.880 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:54:30.909 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:30.930 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:54:30.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:30.933 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:54:31.000 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 82 16 125892 125876 95.9339 62.5625 0.0137565 0.0104199 2024-05-15T12:54:31.034 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:54:31.114 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:31.140 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:54:31.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:31.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:54:31.249 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:54:31.305 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:31.746 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:31 smithi027 ceph-mon[39795]: pgmap v700: 17 pgs: 17 active+clean; 7.6 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-05-15T12:54:32.000 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 83 16 126897 126881 95.5347 62.8125 0.0046459 0.0104623 2024-05-15T12:54:32.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:31 smithi120 ceph-mon[43961]: pgmap v700: 17 pgs: 17 active+clean; 7.6 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-05-15T12:54:32.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:31 smithi177 ceph-mon[44303]: pgmap v700: 17 pgs: 17 active+clean; 7.6 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-05-15T12:54:32.281 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:54:32.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:32.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:54:32.390 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:54:32.439 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:32.956 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:54:33.002 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 84 16 127897 127881 95.1414 62.5 0.0364309 0.0105063 2024-05-15T12:54:33.350 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:54:33.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:33.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:54:33.457 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:54:33.508 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:34.001 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 85 16 128923 128907 94.7764 64.125 0.0197914 0.0105473 2024-05-15T12:54:34.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:34 smithi027 ceph-mon[39795]: pgmap v701: 17 pgs: 17 active+clean; 7.7 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-05-15T12:54:34.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:34 smithi120 ceph-mon[43961]: pgmap v701: 17 pgs: 17 active+clean; 7.7 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-05-15T12:54:34.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:34 smithi177 ceph-mon[44303]: pgmap v701: 17 pgs: 17 active+clean; 7.7 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-05-15T12:54:34.993 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:54:35.000 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 86 16 129932 129916 94.4076 63.0625 0.0134946 0.0105881 2024-05-15T12:54:35.987 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T12:54:35.987 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:54:36.001 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 87 16 130862 130846 93.9904 58.125 0.00889091 0.010627 2024-05-15T12:54:36.219 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:36 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:54:36.220 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:36 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:54:36.220 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:36 smithi027 ceph-mon[39795]: osdmap e255: 12 total, 12 up, 11 in 2024-05-15T12:54:36.220 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:36 smithi027 ceph-mon[39795]: pgmap v703: 17 pgs: 17 active+clean; 7.8 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-05-15T12:54:36.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:36 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:54:36.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:36 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:54:36.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:36 smithi120 ceph-mon[43961]: osdmap e255: 12 total, 12 up, 11 in 2024-05-15T12:54:36.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:36 smithi120 ceph-mon[43961]: pgmap v703: 17 pgs: 17 active+clean; 7.8 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-05-15T12:54:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:36 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:54:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:36 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:54:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:36 smithi177 ceph-mon[44303]: osdmap e255: 12 total, 12 up, 11 in 2024-05-15T12:54:36.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:36 smithi177 ceph-mon[44303]: pgmap v703: 17 pgs: 17 active+clean; 7.8 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-05-15T12:54:36.803 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:54:36.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:36.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:54:36.855 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-05-15T12:54:36.919 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:54:36.953 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:54:36.964 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:37.001 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 88 15 131751 131736 93.5544 55.625 0.0208263 0.010685 2024-05-15T12:54:37.004 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:54:37.009 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:54:37.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:37.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:54:37.121 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:54:37.171 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:37.234 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:37 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4144129532' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:54:37.240 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:54:37.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:37.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:54:37.260 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T12:54:37.291 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:54:37.348 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:54:37.404 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:37.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:37 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4144129532' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:54:37.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:37 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4144129532' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:54:38.001 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 89 16 132687 132671 93.1598 58.4375 0.0207396 0.0107305 2024-05-15T12:54:38.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:38 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:54:38.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:38 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4144129532' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:54:38.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:38 smithi027 ceph-mon[39795]: osdmap e256: 12 total, 12 up, 11 in 2024-05-15T12:54:38.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:38 smithi027 ceph-mon[39795]: pgmap v705: 17 pgs: 17 active+clean; 7.9 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 61 MiB/s wr, 977 op/s 2024-05-15T12:54:38.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:38 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:54:38.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:38 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4144129532' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:54:38.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:38 smithi120 ceph-mon[43961]: osdmap e256: 12 total, 12 up, 11 in 2024-05-15T12:54:38.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:38 smithi120 ceph-mon[43961]: pgmap v705: 17 pgs: 17 active+clean; 7.9 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 61 MiB/s wr, 977 op/s 2024-05-15T12:54:38.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:38 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:54:38.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:38 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4144129532' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:54:38.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:38 smithi177 ceph-mon[44303]: osdmap e256: 12 total, 12 up, 11 in 2024-05-15T12:54:38.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:38 smithi177 ceph-mon[44303]: pgmap v705: 17 pgs: 17 active+clean; 7.9 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 61 MiB/s wr, 977 op/s 2024-05-15T12:54:39.001 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 90 3 133619 133616 92.7809 59.0625 0.0193108 0.0107746 2024-05-15T12:54:39.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:39 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:54:39.394 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T12:54:39.395 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:54:39.417 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:39 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:54:39.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:39 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:54:40.001 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Total time run: 90.0088 2024-05-15T12:54:40.002 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Total writes made: 133619 2024-05-15T12:54:40.002 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Write size: 65536 2024-05-15T12:54:40.002 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Object size: 65536 2024-05-15T12:54:40.002 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Bandwidth (MB/sec): 92.7819 2024-05-15T12:54:40.002 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Stddev Bandwidth: 37.4339 2024-05-15T12:54:40.002 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Max bandwidth (MB/sec): 221.25 2024-05-15T12:54:40.002 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Min bandwidth (MB/sec): 49.625 2024-05-15T12:54:40.002 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Average IOPS: 1484 2024-05-15T12:54:40.002 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Stddev IOPS: 598.943 2024-05-15T12:54:40.002 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Max IOPS: 3540 2024-05-15T12:54:40.002 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Min IOPS: 794 2024-05-15T12:54:40.002 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Average Latency(s): 0.0107747 2024-05-15T12:54:40.003 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Stddev Latency(s): 0.0123649 2024-05-15T12:54:40.003 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Max latency(s): 0.225661 2024-05-15T12:54:40.003 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Min latency(s): 0.00112991 2024-05-15T12:54:40.011 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Cleaning up (deleting benchmark objects) 2024-05-15T12:54:40.061 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-05-15T12:54:40.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:40 smithi027 ceph-mon[39795]: pgmap v706: 17 pgs: 17 active+clean; 8.1 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 62 MiB/s wr, 991 op/s 2024-05-15T12:54:40.272 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:40 smithi120 ceph-mon[43961]: pgmap v706: 17 pgs: 17 active+clean; 8.1 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 62 MiB/s wr, 991 op/s 2024-05-15T12:54:40.372 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:54:40.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:40.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:54:40.482 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:54:40.522 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:40.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:40 smithi177 ceph-mon[44303]: pgmap v706: 17 pgs: 17 active+clean; 8.1 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 62 MiB/s wr, 991 op/s 2024-05-15T12:54:40.985 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:54:41.759 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:41 smithi027 ceph-mon[39795]: pgmap v707: 17 pgs: 17 active+clean; 8.1 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 127 B/s rd, 52 MiB/s wr, 858 op/s 2024-05-15T12:54:41.759 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:41 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:54:41.759 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:41 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:54:41.759 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:41 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:54:41.759 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:41 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:54:41.759 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:41 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:54:42.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:41 smithi120 ceph-mon[43961]: pgmap v707: 17 pgs: 17 active+clean; 8.1 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 127 B/s rd, 52 MiB/s wr, 858 op/s 2024-05-15T12:54:42.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:41 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:54:42.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:41 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:54:42.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:41 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:54:42.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:41 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:54:42.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:41 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:54:42.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:41 smithi177 ceph-mon[44303]: pgmap v707: 17 pgs: 17 active+clean; 8.1 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 127 B/s rd, 52 MiB/s wr, 858 op/s 2024-05-15T12:54:42.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:41 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:54:42.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:41 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:54:42.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:41 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:54:42.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:41 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:54:42.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:41 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:54:42.522 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:54:42.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:42.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:54:42.631 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:54:42.685 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:42.805 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:54:42.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:42.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:54:42.913 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:54:42.952 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:43.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:42 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:54:43.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:42 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:54:43.169 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T12:54:43.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:42 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:54:43.599 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:54:43.601 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:54:43.655 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:54:43.697 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:54:43.738 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:54:43.773 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:54:43.804 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:54:43.853 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:54:43.886 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:54:43.920 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:54:43.966 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:54:43.995 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:54:44.041 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:54:44.045 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.6 to 1 (10000) 2024-05-15T12:54:44.062 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:54:44.062 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:54:44.062 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:54:44.062 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:54:44.062 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:54:44.063 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:54:44.063 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:54:44.063 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:54:44.063 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:54:44.063 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:54:44.063 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:54:44.063 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:54:44.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:44 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4036934513' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T12:54:44.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:44 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T12:54:44.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:44 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/333937073' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:54:44.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:44 smithi027 ceph-mon[39795]: pgmap v708: 17 pgs: 17 active+clean; 8.1 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 120 B/s rd, 48 MiB/s wr, 957 op/s 2024-05-15T12:54:44.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:44 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4036934513' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T12:54:44.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:44 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T12:54:44.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:44 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/333937073' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:54:44.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:44 smithi120 ceph-mon[43961]: pgmap v708: 17 pgs: 17 active+clean; 8.1 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 120 B/s rd, 48 MiB/s wr, 957 op/s 2024-05-15T12:54:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:44 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4036934513' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T12:54:44.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:44 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T12:54:44.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:44 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/333937073' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:54:44.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:44 smithi177 ceph-mon[44303]: pgmap v708: 17 pgs: 17 active+clean; 8.1 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 120 B/s rd, 48 MiB/s wr, 957 op/s 2024-05-15T12:54:45.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:45 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T12:54:45.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:45 smithi120 ceph-mon[43961]: osdmap e257: 12 total, 12 up, 11 in 2024-05-15T12:54:45.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:45 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:54:45.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:45 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:54:45.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:45 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:54:45.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:45 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:54:45.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:45 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:54:45.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:45 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:54:45.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:45 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:54:45.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:45 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:54:45.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:45 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:54:45.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:45 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T12:54:45.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:45 smithi177 ceph-mon[44303]: osdmap e257: 12 total, 12 up, 11 in 2024-05-15T12:54:45.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:45 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:54:45.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:45 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:54:45.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:45 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:54:45.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:45 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:54:45.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:45 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:54:45.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:45 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:54:45.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:45 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:54:45.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:45 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:54:45.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:45 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:54:45.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:45 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T12:54:45.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:45 smithi027 ceph-mon[39795]: osdmap e257: 12 total, 12 up, 11 in 2024-05-15T12:54:45.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:45 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:54:45.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:45 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:54:45.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:45 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:54:45.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:45 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:54:45.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:45 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:54:45.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:45 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:54:45.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:45 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:54:45.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:45 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:54:45.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:45 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:54:45.649 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:54:45.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:45.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:54:45.756 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:54:45.786 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:45.809 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:54:45.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:45.812 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:54:45.913 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:54:45.942 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:45.962 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:54:45.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:45.967 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:54:46.068 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:54:46.127 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:46.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:46 smithi177 ceph-mon[44303]: osdmap e258: 12 total, 12 up, 11 in 2024-05-15T12:54:46.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:46 smithi177 ceph-mon[44303]: pgmap v711: 17 pgs: 17 active+clean; 8.0 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 127 B/s rd, 29 MiB/s wr, 868 op/s 2024-05-15T12:54:46.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:46 smithi027 ceph-mon[39795]: osdmap e258: 12 total, 12 up, 11 in 2024-05-15T12:54:46.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:46 smithi027 ceph-mon[39795]: pgmap v711: 17 pgs: 17 active+clean; 8.0 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 127 B/s rd, 29 MiB/s wr, 868 op/s 2024-05-15T12:54:46.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:46 smithi120 ceph-mon[43961]: osdmap e258: 12 total, 12 up, 11 in 2024-05-15T12:54:46.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:46 smithi120 ceph-mon[43961]: pgmap v711: 17 pgs: 17 active+clean; 8.0 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 127 B/s rd, 29 MiB/s wr, 868 op/s 2024-05-15T12:54:47.157 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:54:47.512 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:47 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4217581949' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:54:47.517 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T12:54:47.626 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 7 1 2024-05-15T12:54:47.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:47 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4217581949' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:54:47.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:47 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4217581949' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:54:47.940 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:54:48.523 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:54:48.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:48.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:54:48.637 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:54:48.664 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:48.688 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:54:48.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:48.690 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:54:48.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:48 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:54:48.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:48 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4217581949' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:54:48.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:48 smithi027 ceph-mon[39795]: osdmap e259: 12 total, 12 up, 11 in 2024-05-15T12:54:48.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:48 smithi027 ceph-mon[39795]: pgmap v713: 17 pgs: 17 active+clean; 7.9 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 5.6 MiB/s wr, 838 op/s 2024-05-15T12:54:48.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:48 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:54:48.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:48 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4217581949' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:54:48.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:48 smithi120 ceph-mon[43961]: osdmap e259: 12 total, 12 up, 11 in 2024-05-15T12:54:48.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:48 smithi120 ceph-mon[43961]: pgmap v713: 17 pgs: 17 active+clean; 7.9 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 5.6 MiB/s wr, 838 op/s 2024-05-15T12:54:48.791 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:54:48.838 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:48.907 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:54:48.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:48.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:54:48.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:48 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:54:48.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:48 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4217581949' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:54:48.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:48 smithi177 ceph-mon[44303]: osdmap e259: 12 total, 12 up, 11 in 2024-05-15T12:54:48.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:48 smithi177 ceph-mon[44303]: pgmap v713: 17 pgs: 17 active+clean; 7.9 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 5.6 MiB/s wr, 838 op/s 2024-05-15T12:54:49.011 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T12:54:49.012 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T12:54:49.012 INFO:teuthology.orchestra.run.smithi120.stdout: "complaint_time": 30, 2024-05-15T12:54:49.012 INFO:teuthology.orchestra.run.smithi120.stdout: "num_blocked_ops": 0 2024-05-15T12:54:49.012 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T12:54:49.012 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:54:49.073 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:49.286 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:54:49.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:49.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:54:49.391 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:54:49.442 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:49.755 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:49 smithi027 ceph-mon[39795]: pgmap v714: 17 pgs: 17 active+clean; 7.7 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 859 op/s 2024-05-15T12:54:49.814 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-05-15T12:54:50.001 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:54:50.011 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:54:50.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:50.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:54:50.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:49 smithi120 ceph-mon[43961]: pgmap v714: 17 pgs: 17 active+clean; 7.7 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 859 op/s 2024-05-15T12:54:50.122 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:54:50.148 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:49 smithi177 ceph-mon[44303]: pgmap v714: 17 pgs: 17 active+clean; 7.7 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 859 op/s 2024-05-15T12:54:50.150 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:50.172 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:54:50.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:50.175 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:54:50.267 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:54:50.276 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:54:50.303 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:50.325 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:54:50.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:50.328 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:54:50.429 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:54:50.482 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:50.569 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:54:50.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:50.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:54:50.675 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:54:50.703 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:50.722 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:54:50.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:50.725 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:54:50.767 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:50 smithi120 ceph-mon[43961]: 3.2 scrub starts 2024-05-15T12:54:50.826 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:54:50.855 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:50.876 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:54:50.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:50.879 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:54:50.980 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:54:51.007 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:50 smithi177 ceph-mon[44303]: 3.2 scrub starts 2024-05-15T12:54:51.009 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:51.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:50 smithi027 ceph-mon[39795]: 3.2 scrub starts 2024-05-15T12:54:51.029 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:54:51.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:51.032 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:54:51.075 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:54:51.111 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:54:51.132 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:54:51.149 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:54:51.196 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:51.197 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:54:51.253 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:54:51.319 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:54:51.358 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:54:51.393 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:54:51.397 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T12:54:51.437 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:54:51.472 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:54:51.530 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:54:51.564 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:54:51.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:51.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:54:51.578 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:54:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:54:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:54:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:54:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:54:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:54:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:54:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:54:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:54:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:54:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:54:51.602 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:54:51.602 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:54:51.671 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:54:51.726 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:52.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:51 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/432517977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:54:52.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:51 smithi027 ceph-mon[39795]: pgmap v715: 17 pgs: 17 active+clean; 7.6 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 900 op/s 2024-05-15T12:54:52.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:51 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/432517977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:54:52.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:51 smithi120 ceph-mon[43961]: pgmap v715: 17 pgs: 17 active+clean; 7.6 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 900 op/s 2024-05-15T12:54:52.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:51 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/432517977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:54:52.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:51 smithi177 ceph-mon[44303]: pgmap v715: 17 pgs: 17 active+clean; 7.6 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 900 op/s 2024-05-15T12:54:52.506 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T12:54:52.506 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:54:52.814 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:54:52.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:52.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:54:52.921 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:54:52.948 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:52 smithi120 ceph-mon[43961]: 3.5 scrub starts 2024-05-15T12:54:52.948 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:52 smithi120 ceph-mon[43961]: 3.6 scrub starts 2024-05-15T12:54:52.973 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:53.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:52 smithi177 ceph-mon[44303]: 3.5 scrub starts 2024-05-15T12:54:53.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:52 smithi177 ceph-mon[44303]: 3.6 scrub starts 2024-05-15T12:54:53.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:52 smithi027 ceph-mon[39795]: 3.5 scrub starts 2024-05-15T12:54:53.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:52 smithi027 ceph-mon[39795]: 3.6 scrub starts 2024-05-15T12:54:53.354 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:54:53.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:53.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:54:53.373 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-05-15T12:54:53.460 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:54:53.523 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:53.586 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:54:53.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:53.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:54:53.693 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:54:53.734 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:54:53.760 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:53.858 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:54:54.073 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.7 to 1 (10000) 2024-05-15T12:54:54.121 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:54:54.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:54.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:54:54.228 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:54:54.260 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:54.291 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:54:54.295 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:54 smithi177 ceph-mon[44303]: 3.2 scrub ok 2024-05-15T12:54:54.295 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:54 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:54:54.295 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:54 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:54:54.295 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:54 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4120801702' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T12:54:54.295 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:54 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T12:54:54.296 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:54 smithi177 ceph-mon[44303]: pgmap v716: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.5 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 984 op/s 2024-05-15T12:54:54.296 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:54 smithi177 ceph-mon[44303]: 3.5 scrub ok 2024-05-15T12:54:54.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:54.297 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:54:54.398 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:54:54.429 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:54.451 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:54:54.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:54.454 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:54:54.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:54 smithi027 ceph-mon[39795]: 3.2 scrub ok 2024-05-15T12:54:54.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:54 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:54:54.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:54 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:54:54.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:54 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4120801702' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T12:54:54.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:54 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T12:54:54.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:54 smithi027 ceph-mon[39795]: pgmap v716: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.5 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 984 op/s 2024-05-15T12:54:54.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:54 smithi027 ceph-mon[39795]: 3.5 scrub ok 2024-05-15T12:54:54.527 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:54 smithi120 ceph-mon[43961]: 3.2 scrub ok 2024-05-15T12:54:54.527 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:54 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:54:54.527 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:54 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:54:54.527 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:54 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4120801702' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T12:54:54.527 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:54 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T12:54:54.527 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:54 smithi120 ceph-mon[43961]: pgmap v716: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.5 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 984 op/s 2024-05-15T12:54:54.528 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:54 smithi120 ceph-mon[43961]: 3.5 scrub ok 2024-05-15T12:54:54.555 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:54:54.581 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:54.605 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:54:54.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:54.608 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:54:54.709 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:54:54.767 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:54.851 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:54:54.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:54.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:54:54.960 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:54:55.015 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:55.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:55 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:54:55.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:55 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:54:55.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:55 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T12:54:55.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:55 smithi027 ceph-mon[39795]: osdmap e260: 12 total, 12 up, 11 in 2024-05-15T12:54:55.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:55 smithi027 ceph-mon[39795]: 3.6 scrub ok 2024-05-15T12:54:55.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:55 smithi027 ceph-mon[39795]: 3.c scrub starts 2024-05-15T12:54:55.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:55 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:54:55.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:55 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:54:55.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:55 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T12:54:55.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:55 smithi120 ceph-mon[43961]: osdmap e260: 12 total, 12 up, 11 in 2024-05-15T12:54:55.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:55 smithi120 ceph-mon[43961]: 3.6 scrub ok 2024-05-15T12:54:55.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:55 smithi120 ceph-mon[43961]: 3.c scrub starts 2024-05-15T12:54:55.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:55 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:54:55.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:55 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:54:55.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:55 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T12:54:55.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:55 smithi177 ceph-mon[44303]: osdmap e260: 12 total, 12 up, 11 in 2024-05-15T12:54:55.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:55 smithi177 ceph-mon[44303]: 3.6 scrub ok 2024-05-15T12:54:55.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:55 smithi177 ceph-mon[44303]: 3.c scrub starts 2024-05-15T12:54:55.517 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:54:55.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:55.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:54:55.530 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:54:55.629 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:54:55.692 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:55.764 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:54:55.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:55.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:54:55.873 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:54:55.927 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:56.099 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:54:56.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:56.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:54:56.142 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T12:54:56.142 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:54:56.175 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:56 smithi120 ceph-mon[43961]: 3.0 scrub starts 2024-05-15T12:54:56.175 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:56 smithi120 ceph-mon[43961]: pgmap v718: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.4 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 901 op/s 2024-05-15T12:54:56.207 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:54:56.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:56 smithi027 ceph-mon[39795]: 3.0 scrub starts 2024-05-15T12:54:56.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:56 smithi027 ceph-mon[39795]: pgmap v718: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.4 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 901 op/s 2024-05-15T12:54:56.301 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:56.400 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 8 1 2024-05-15T12:54:56.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:56 smithi177 ceph-mon[44303]: 3.0 scrub starts 2024-05-15T12:54:56.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:56 smithi177 ceph-mon[44303]: pgmap v718: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.4 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 901 op/s 2024-05-15T12:54:56.460 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:54:56.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:56.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:54:56.566 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:54:56.628 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:57.012 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-05-15T12:54:57.013 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:54:57.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:57.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:54:57.101 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:54:57.106 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:54:57.123 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:54:57.168 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:57.225 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:54:57.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:57.232 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:57 smithi120 ceph-mon[43961]: 3.0 scrub ok 2024-05-15T12:54:57.232 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:57 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:54:57.232 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:57 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:54:57.232 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:57 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:54:57.232 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:57 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:54:57.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:54:57.334 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:54:57.377 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:57.400 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:54:57.414 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:54:57.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:57.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:54:57.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:57 smithi177 ceph-mon[44303]: 3.0 scrub ok 2024-05-15T12:54:57.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:57 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:54:57.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:57 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:54:57.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:57 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:54:57.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:57 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:54:57.437 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:57 smithi027 ceph-mon[39795]: 3.0 scrub ok 2024-05-15T12:54:57.437 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:57 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:54:57.437 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:57 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:54:57.438 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:57 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:54:57.438 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:57 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:54:57.517 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:54:57.547 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:57.567 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:54:57.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:57.570 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:54:57.671 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:54:57.717 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:58.160 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T12:54:58.385 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:54:58.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:58.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:54:58.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:58 smithi027 ceph-mon[39795]: 3.c scrub ok 2024-05-15T12:54:58.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:58 smithi027 ceph-mon[39795]: osdmap e261: 12 total, 12 up, 11 in 2024-05-15T12:54:58.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:58 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/257134972' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:54:58.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:58 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:54:58.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:58 smithi027 ceph-mon[39795]: pgmap v720: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.3 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 823 op/s 2024-05-15T12:54:58.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:58 smithi177 ceph-mon[44303]: 3.c scrub ok 2024-05-15T12:54:58.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:58 smithi177 ceph-mon[44303]: osdmap e261: 12 total, 12 up, 11 in 2024-05-15T12:54:58.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:58 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/257134972' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:54:58.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:58 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:54:58.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:58 smithi177 ceph-mon[44303]: pgmap v720: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.3 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 823 op/s 2024-05-15T12:54:58.467 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:58 smithi120 ceph-mon[43961]: 3.c scrub ok 2024-05-15T12:54:58.467 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:58 smithi120 ceph-mon[43961]: osdmap e261: 12 total, 12 up, 11 in 2024-05-15T12:54:58.467 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:58 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/257134972' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:54:58.467 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:58 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:54:58.467 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:58 smithi120 ceph-mon[43961]: pgmap v720: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.3 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 823 op/s 2024-05-15T12:54:58.494 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:54:58.525 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:58.545 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:54:58.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:58.548 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:54:58.649 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:54:58.677 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:58.701 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:54:58.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:58.705 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:54:58.806 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:54:58.836 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:58.856 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:54:58.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:58.859 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:54:58.960 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:54:59.037 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:59.073 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:54:59.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:59.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:54:59.148 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:59 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:54:59.149 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:59 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T12:54:59.149 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:59 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:54:59.180 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:54:59.227 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:59.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:59 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:54:59.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:59 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T12:54:59.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:59 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:54:59.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:54:59 smithi177 ceph-mon[44303]: osdmap e262: 12 total, 12 up, 11 in 2024-05-15T12:54:59.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:59 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:54:59.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:59 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T12:54:59.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:59 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:54:59.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:54:59 smithi027 ceph-mon[39795]: osdmap e262: 12 total, 12 up, 11 in 2024-05-15T12:54:59.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:54:59 smithi120 ceph-mon[43961]: osdmap e262: 12 total, 12 up, 11 in 2024-05-15T12:54:59.591 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:54:59.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:59.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:54:59.609 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T12:54:59.609 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:54:59.696 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:54:59.752 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:54:59.791 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:54:59.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:54:59.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:54:59.907 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:54:59.948 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:00.206 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:55:00.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:00.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:55:00.212 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:00 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:55:00.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:00 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:55:00.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:00 smithi027 ceph-mon[39795]: pgmap v722: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.2 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 749 op/s 2024-05-15T12:55:00.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:00 smithi027 ceph-mon[39795]: 3.e scrub starts 2024-05-15T12:55:00.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:00 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:55:00.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:00 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:55:00.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:00 smithi120 ceph-mon[43961]: pgmap v722: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.2 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 749 op/s 2024-05-15T12:55:00.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:00 smithi120 ceph-mon[43961]: 3.e scrub starts 2024-05-15T12:55:00.313 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:55:00.385 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:00.409 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-05-15T12:55:00.422 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:55:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:00 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:55:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:00 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:55:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:00 smithi177 ceph-mon[44303]: pgmap v722: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.2 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 749 op/s 2024-05-15T12:55:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:00 smithi177 ceph-mon[44303]: 3.e scrub starts 2024-05-15T12:55:00.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:00.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:55:00.535 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:55:00.566 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:00.601 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:55:00.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:00.605 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:55:00.706 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:55:00.765 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:00.802 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:55:00.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:00.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:55:00.905 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:55:00.934 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:55:00.995 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:01.283 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:55:01.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:01.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:55:01.391 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:55:01.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:01 smithi177 ceph-mon[44303]: osdmap e263: 12 total, 12 up, 11 in 2024-05-15T12:55:01.448 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:01 smithi120 ceph-mon[43961]: osdmap e263: 12 total, 12 up, 11 in 2024-05-15T12:55:01.457 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:01.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:01 smithi027 ceph-mon[39795]: osdmap e263: 12 total, 12 up, 11 in 2024-05-15T12:55:02.062 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:55:02.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:02.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:55:02.178 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:55:02.240 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:02.405 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:02 smithi120 ceph-mon[43961]: pgmap v724: 17 pgs: 17 active+clean; 7.1 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 769 op/s 2024-05-15T12:55:02.416 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:55:02.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:02.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:55:02.520 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:55:02.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:02 smithi027 ceph-mon[39795]: pgmap v724: 17 pgs: 17 active+clean; 7.1 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 769 op/s 2024-05-15T12:55:02.594 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:02.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:02 smithi177 ceph-mon[44303]: pgmap v724: 17 pgs: 17 active+clean; 7.1 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 769 op/s 2024-05-15T12:55:02.761 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:55:02.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:02.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:55:02.868 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:55:02.922 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:02.964 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:55:02.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:02.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:55:03.071 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:55:03.099 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:03.122 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:55:03.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:03.124 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:55:03.146 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T12:55:03.147 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:55:03.200 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.8 to 1 (10000) 2024-05-15T12:55:03.225 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:55:03.281 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:03.402 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:55:03.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:03.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:55:03.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:03 smithi177 ceph-mon[44303]: 3.e scrub ok 2024-05-15T12:55:03.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:03 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2661675652' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T12:55:03.511 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:55:03.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:03 smithi027 ceph-mon[39795]: 3.e scrub ok 2024-05-15T12:55:03.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:03 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2661675652' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T12:55:03.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:03 smithi120 ceph-mon[43961]: 3.e scrub ok 2024-05-15T12:55:03.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:03 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2661675652' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T12:55:03.569 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:03.666 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:55:03.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:03.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:55:03.770 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:55:03.797 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:03.819 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:55:03.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:03.822 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:55:03.893 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:55:03.894 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:55:03.894 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:55:03.894 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:55:03.894 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:55:03.894 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:55:03.894 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:55:03.894 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:55:03.897 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:55:03.923 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:55:03.931 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:55:03.949 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-05-15T12:55:03.967 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:55:03.982 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:03.995 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:55:04.015 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:55:04.015 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:55:04.015 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:55:04.015 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:55:04.015 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:55:04.015 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:55:04.015 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:55:04.015 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:55:04.015 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:55:04.016 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:55:04.016 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:55:04.016 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:55:04.033 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:55:04.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:04.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:55:04.135 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:55:04.164 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:04.185 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:55:04.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:04.188 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:55:04.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:04 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2661675652' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T12:55:04.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:04 smithi120 ceph-mon[43961]: osdmap e264: 12 total, 12 up, 11 in 2024-05-15T12:55:04.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:04 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/238163693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:55:04.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:04 smithi120 ceph-mon[43961]: pgmap v726: 17 pgs: 17 active+clean; 7.0 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 920 op/s 2024-05-15T12:55:04.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:04 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2661675652' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T12:55:04.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:04 smithi027 ceph-mon[39795]: osdmap e264: 12 total, 12 up, 11 in 2024-05-15T12:55:04.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:04 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/238163693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:55:04.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:04 smithi027 ceph-mon[39795]: pgmap v726: 17 pgs: 17 active+clean; 7.0 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 920 op/s 2024-05-15T12:55:04.289 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:55:04.362 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:04.412 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:55:04.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:04.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:55:04.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:04 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2661675652' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T12:55:04.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:04 smithi177 ceph-mon[44303]: osdmap e264: 12 total, 12 up, 11 in 2024-05-15T12:55:04.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:04 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/238163693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:55:04.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:04 smithi177 ceph-mon[44303]: pgmap v726: 17 pgs: 17 active+clean; 7.0 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 920 op/s 2024-05-15T12:55:04.456 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 9 1 2024-05-15T12:55:04.503 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:55:04.520 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:55:04.578 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:04.760 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:55:04.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:04.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:55:04.866 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:55:04.893 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:04.914 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:55:04.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:04.919 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:55:04.921 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T12:55:05.020 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:55:05.075 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:05.325 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:55:05.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:05.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:55:05.389 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:55:05.413 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:55:05.431 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:55:05.458 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:05.477 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:55:05.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:05.480 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:55:05.581 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:55:05.632 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:05.771 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:05 smithi177 ceph-mon[44303]: pgmap v727: 17 pgs: 17 active+clean; 6.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 875 op/s 2024-05-15T12:55:05.851 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:55:05.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:05.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:55:05.967 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:55:05.987 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:05 smithi027 ceph-mon[39795]: pgmap v727: 17 pgs: 17 active+clean; 6.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 875 op/s 2024-05-15T12:55:06.016 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:05 smithi120 ceph-mon[43961]: pgmap v727: 17 pgs: 17 active+clean; 6.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 875 op/s 2024-05-15T12:55:06.021 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:06.192 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:55:06.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:06.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:55:06.300 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:55:06.328 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:06.347 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:55:06.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:06.350 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:55:06.451 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:55:06.480 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:06.502 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:55:06.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:06.506 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:55:06.581 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T12:55:06.582 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [ 2024-05-15T12:55:06.582 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.582 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:156632 3.a 3:59c2d8ac:::benchmark_data_smithi177_88104_object23010:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e264)", 2024-05-15T12:55:06.582 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:55:06.575112+0000", 2024-05-15T12:55:06.582 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.016297243999999999, 2024-05-15T12:55:06.582 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.016337154, 2024-05-15T12:55:06.582 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:06.582 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "waiting for sub ops", 2024-05-15T12:55:06.582 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:55:06.582 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T12:55:06.582 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:55:06.583 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 156632 2024-05-15T12:55:06.583 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.583 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:06.583 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.583 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:06.583 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.575112+0000", 2024-05-15T12:55:06.583 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:06.583 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.583 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.583 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:06.583 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.575112+0000", 2024-05-15T12:55:06.583 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:06.583 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.583 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.584 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:06.584 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.575114+0000", 2024-05-15T12:55:06.584 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.643e-06 2024-05-15T12:55:06.584 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.584 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.584 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:06.584 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.575120+0000", 2024-05-15T12:55:06.584 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5369999999999997e-06 2024-05-15T12:55:06.584 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.584 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.584 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:06.584 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.575132+0000", 2024-05-15T12:55:06.584 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2616000000000001e-05 2024-05-15T12:55:06.585 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.585 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.585 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:06.585 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.575140+0000", 2024-05-15T12:55:06.585 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0690000000000002e-06 2024-05-15T12:55:06.585 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.585 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.585 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:06.585 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.575206+0000", 2024-05-15T12:55:06.585 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.5850000000000001e-05 2024-05-15T12:55:06.585 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.585 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.585 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:55:06.586 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.575312+0000", 2024-05-15T12:55:06.586 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00010618799999999999 2024-05-15T12:55:06.586 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.586 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.586 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,4", 2024-05-15T12:55:06.586 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.575412+0000", 2024-05-15T12:55:06.586 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000100021 2024-05-15T12:55:06.586 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.586 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.586 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:55:06.586 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.583811+0000", 2024-05-15T12:55:06.586 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0083985339999999992 2024-05-15T12:55:06.587 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:06.587 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:06.587 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:06.587 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.587 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.587 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:156637 3.a 3:5183db70:::benchmark_data_smithi177_88104_object23015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e264)", 2024-05-15T12:55:06.587 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:55:06.583745+0000", 2024-05-15T12:55:06.587 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.0076650449999999997, 2024-05-15T12:55:06.587 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0077932540000000003, 2024-05-15T12:55:06.587 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:06.587 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "waiting for sub ops", 2024-05-15T12:55:06.587 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:55:06.587 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T12:55:06.587 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:55:06.588 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 156637 2024-05-15T12:55:06.588 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.588 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:06.588 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.588 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:06.588 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.583745+0000", 2024-05-15T12:55:06.588 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:06.588 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.588 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.588 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:06.588 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.583745+0000", 2024-05-15T12:55:06.588 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:06.588 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.588 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.588 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:06.589 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.583746+0000", 2024-05-15T12:55:06.589 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.28e-06 2024-05-15T12:55:06.589 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.589 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.589 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:06.589 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.583752+0000", 2024-05-15T12:55:06.589 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9669999999999998e-06 2024-05-15T12:55:06.589 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.589 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.589 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:06.589 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.583767+0000", 2024-05-15T12:55:06.589 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4786e-05 2024-05-15T12:55:06.589 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.589 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.589 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:06.590 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.583776+0000", 2024-05-15T12:55:06.590 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.7589999999999992e-06 2024-05-15T12:55:06.590 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.590 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.590 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:06.590 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.583851+0000", 2024-05-15T12:55:06.590 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4277999999999996e-05 2024-05-15T12:55:06.590 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.590 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.590 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:55:06.590 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.583985+0000", 2024-05-15T12:55:06.590 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013438499999999999 2024-05-15T12:55:06.590 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.591 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.591 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,4", 2024-05-15T12:55:06.591 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.584098+0000", 2024-05-15T12:55:06.591 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00011297400000000001 2024-05-15T12:55:06.591 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.591 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.591 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:55:06.591 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588943+0000", 2024-05-15T12:55:06.591 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0048451930000000002 2024-05-15T12:55:06.591 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:06.591 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:06.591 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:06.591 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.591 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.591 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.24832.0:156623 3.c e264/219 3:3c04b9c3:::benchmark_data_smithi177_88104_object23001:head v 264'9759, mlcod=264'9755)", 2024-05-15T12:55:06.592 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:55:06.588841+0000", 2024-05-15T12:55:06.592 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.0025687399999999999, 2024-05-15T12:55:06.592 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0027755050000000002, 2024-05-15T12:55:06.592 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:06.592 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "started", 2024-05-15T12:55:06.592 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:06.592 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.592 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:06.592 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588841+0000", 2024-05-15T12:55:06.592 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:06.592 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.592 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.592 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:06.592 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588841+0000", 2024-05-15T12:55:06.593 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:06.593 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.593 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.593 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:06.593 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588842+0000", 2024-05-15T12:55:06.593 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8899999999999998e-07 2024-05-15T12:55:06.593 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.593 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.593 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:06.593 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588849+0000", 2024-05-15T12:55:06.593 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2060000000000001e-06 2024-05-15T12:55:06.593 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.593 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.593 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:06.593 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588858+0000", 2024-05-15T12:55:06.593 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.3729999999999999e-06 2024-05-15T12:55:06.594 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.594 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.594 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:06.594 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588865+0000", 2024-05-15T12:55:06.594 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0620000000000003e-06 2024-05-15T12:55:06.594 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.594 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.594 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:06.594 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.589029+0000", 2024-05-15T12:55:06.594 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00016328999999999999 2024-05-15T12:55:06.594 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.594 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.594 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:55:06.594 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.589048+0000", 2024-05-15T12:55:06.595 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9854999999999999e-05 2024-05-15T12:55:06.595 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:06.595 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:06.595 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:06.595 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.595 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.595 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.24832.0:156633 3.c e264/219)", 2024-05-15T12:55:06.595 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:55:06.588887+0000", 2024-05-15T12:55:06.595 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.0025223189999999999, 2024-05-15T12:55:06.595 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0027840809999999999, 2024-05-15T12:55:06.595 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:06.595 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "queued for pg", 2024-05-15T12:55:06.595 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:06.595 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.595 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:06.596 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588887+0000", 2024-05-15T12:55:06.596 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:06.596 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.596 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.596 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:06.596 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588887+0000", 2024-05-15T12:55:06.596 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:06.596 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.596 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.596 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:06.596 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588888+0000", 2024-05-15T12:55:06.596 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.2599999999999998e-07 2024-05-15T12:55:06.596 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.596 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.596 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:06.597 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588892+0000", 2024-05-15T12:55:06.597 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0029999999999996e-06 2024-05-15T12:55:06.597 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.597 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.597 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:06.597 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588898+0000", 2024-05-15T12:55:06.597 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1240000000000002e-06 2024-05-15T12:55:06.597 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.597 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.597 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:06.597 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588903+0000", 2024-05-15T12:55:06.597 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9250000000000003e-06 2024-05-15T12:55:06.597 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:06.597 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:06.597 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:06.598 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.598 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.598 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop_reply(client.24832.0:156632 3.a e264/219)", 2024-05-15T12:55:06.598 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:55:06.588931+0000", 2024-05-15T12:55:06.598 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.002478357, 2024-05-15T12:55:06.598 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.002785339, 2024-05-15T12:55:06.598 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:06.598 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "queued for pg", 2024-05-15T12:55:06.598 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:06.598 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.598 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:06.598 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588931+0000", 2024-05-15T12:55:06.598 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:06.598 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.599 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.599 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:06.599 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588931+0000", 2024-05-15T12:55:06.599 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:06.599 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.599 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.599 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:06.599 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588932+0000", 2024-05-15T12:55:06.599 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4700000000000002e-07 2024-05-15T12:55:06.599 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.599 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.599 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:06.599 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588935+0000", 2024-05-15T12:55:06.599 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.3560000000000001e-06 2024-05-15T12:55:06.599 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.599 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.600 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:06.600 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588942+0000", 2024-05-15T12:55:06.600 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3350000000000001e-06 2024-05-15T12:55:06.600 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.600 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.600 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:06.600 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588948+0000", 2024-05-15T12:55:06.600 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3210000000000004e-06 2024-05-15T12:55:06.600 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:06.600 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:06.600 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:06.600 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.600 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.600 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:156639 3.a 3.f9b6f94a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e264)", 2024-05-15T12:55:06.600 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:55:06.588967+0000", 2024-05-15T12:55:06.600 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.0024424, 2024-05-15T12:55:06.601 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0028171870000000001, 2024-05-15T12:55:06.601 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:06.601 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "queued for pg", 2024-05-15T12:55:06.601 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:55:06.601 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T12:55:06.601 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:55:06.601 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 156639 2024-05-15T12:55:06.601 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.601 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:06.601 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.601 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:06.601 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588967+0000", 2024-05-15T12:55:06.601 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:06.601 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.602 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.602 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:06.602 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588967+0000", 2024-05-15T12:55:06.602 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:06.602 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.602 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.602 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:06.602 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588968+0000", 2024-05-15T12:55:06.602 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.243e-06 2024-05-15T12:55:06.602 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.602 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.602 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:06.602 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588973+0000", 2024-05-15T12:55:06.602 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3089999999999997e-06 2024-05-15T12:55:06.603 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.603 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.603 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:06.603 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588983+0000", 2024-05-15T12:55:06.603 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.7329999999999998e-06 2024-05-15T12:55:06.603 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:06.603 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:06.603 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:06.603 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:55:06.588989+0000", 2024-05-15T12:55:06.603 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3119999999999999e-06 2024-05-15T12:55:06.603 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:06.603 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:06.603 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:06.603 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:06.603 INFO:teuthology.orchestra.run.smithi120.stdout: ], 2024-05-15T12:55:06.603 INFO:teuthology.orchestra.run.smithi120.stdout: "num_ops": 6 2024-05-15T12:55:06.603 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T12:55:06.606 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:55:06.632 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:06.652 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:55:06.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:06.655 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:55:06.756 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:55:06.812 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:06.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:06 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T12:55:06.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:06 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T12:55:06.981 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:55:06.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:06.990 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:06 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T12:55:06.990 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:06 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T12:55:06.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:55:07.009 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:06 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T12:55:07.009 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:06 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T12:55:07.092 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:55:07.152 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:07.358 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:55:07.392 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:55:07.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:07.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:55:07.412 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-05-15T12:55:07.507 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:55:07.578 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:07.636 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:55:07.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:07.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:55:07.739 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:55:07.766 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:07.788 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:55:07.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:07.791 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:55:07.892 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:55:07.893 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:55:07.919 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:07.941 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:55:07.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:07.943 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:55:08.044 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:55:08.097 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:07 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:55:08.097 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:07 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:55:08.098 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:07 smithi120 ceph-mon[43961]: pgmap v728: 17 pgs: 17 active+clean; 6.8 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 827 op/s 2024-05-15T12:55:08.103 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:08.177 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:55:08.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:07 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:55:08.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:07 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:55:08.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:07 smithi177 ceph-mon[44303]: pgmap v728: 17 pgs: 17 active+clean; 6.8 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 827 op/s 2024-05-15T12:55:08.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:08.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:55:08.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:07 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:55:08.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:07 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:55:08.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:07 smithi027 ceph-mon[39795]: pgmap v728: 17 pgs: 17 active+clean; 6.8 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 827 op/s 2024-05-15T12:55:08.290 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:55:08.353 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:08.474 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:55:08.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:08.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:55:08.577 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:55:08.603 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:08.624 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:55:08.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:08.629 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:55:08.730 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:55:08.775 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:08.825 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:55:08.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:08.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:55:08.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:08 smithi177 ceph-mon[44303]: osdmap e265: 12 total, 12 up, 11 in 2024-05-15T12:55:08.931 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:55:08.970 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:09.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:08 smithi027 ceph-mon[39795]: osdmap e265: 12 total, 12 up, 11 in 2024-05-15T12:55:09.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:08 smithi120 ceph-mon[43961]: osdmap e265: 12 total, 12 up, 11 in 2024-05-15T12:55:09.394 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:55:09.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:09.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:55:09.408 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:55:09.503 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:55:09.533 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:09.556 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:55:09.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:09.560 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:55:09.661 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:55:09.714 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:09.830 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T12:55:09.831 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:09 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3874475421' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:55:09.831 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:09 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:55:09.831 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:09 smithi177 ceph-mon[44303]: pgmap v730: 17 pgs: 17 active+clean; 6.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 850 op/s 2024-05-15T12:55:09.930 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:55:09.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:09.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:55:09.970 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:09 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3874475421' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:55:09.971 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:09 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:55:09.971 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:09 smithi120 ceph-mon[43961]: pgmap v730: 17 pgs: 17 active+clean; 6.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 850 op/s 2024-05-15T12:55:10.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:09 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3874475421' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:55:10.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:09 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:55:10.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:09 smithi027 ceph-mon[39795]: pgmap v730: 17 pgs: 17 active+clean; 6.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 850 op/s 2024-05-15T12:55:10.035 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:55:10.092 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:10.329 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T12:55:10.329 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T12:55:10.329 INFO:teuthology.orchestra.run.smithi120.stdout: "complaint_time": 30, 2024-05-15T12:55:10.329 INFO:teuthology.orchestra.run.smithi120.stdout: "num_blocked_ops": 0 2024-05-15T12:55:10.329 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T12:55:10.756 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:55:10.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:10.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:55:10.852 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.9 to 1 (10000) 2024-05-15T12:55:10.863 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:55:10.915 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:11.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:10 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:55:11.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:10 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:55:11.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:10 smithi027 ceph-mon[39795]: osdmap e266: 12 total, 12 up, 11 in 2024-05-15T12:55:11.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:10 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2354760197' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T12:55:11.129 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:10 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:55:11.129 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:10 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:55:11.129 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:10 smithi120 ceph-mon[43961]: osdmap e266: 12 total, 12 up, 11 in 2024-05-15T12:55:11.129 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:10 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2354760197' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T12:55:11.160 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-05-15T12:55:11.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:10 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:55:11.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:10 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:55:11.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:10 smithi177 ceph-mon[44303]: osdmap e266: 12 total, 12 up, 11 in 2024-05-15T12:55:11.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:10 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2354760197' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T12:55:11.688 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:55:11.789 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:55:11.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:11.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:55:11.902 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:55:11.955 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:11.993 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:55:12.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:12.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:55:12.101 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:55:12.128 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:11 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2354760197' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T12:55:12.128 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:11 smithi177 ceph-mon[44303]: osdmap e267: 12 total, 12 up, 11 in 2024-05-15T12:55:12.128 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:11 smithi177 ceph-mon[44303]: pgmap v733: 17 pgs: 17 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 787 op/s 2024-05-15T12:55:12.130 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:12.150 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:55:12.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:12.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:55:12.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:11 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2354760197' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T12:55:12.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:11 smithi027 ceph-mon[39795]: osdmap e267: 12 total, 12 up, 11 in 2024-05-15T12:55:12.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:11 smithi027 ceph-mon[39795]: pgmap v733: 17 pgs: 17 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 787 op/s 2024-05-15T12:55:12.254 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:55:12.276 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:11 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2354760197' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T12:55:12.276 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:11 smithi120 ceph-mon[43961]: osdmap e267: 12 total, 12 up, 11 in 2024-05-15T12:55:12.276 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:11 smithi120 ceph-mon[43961]: pgmap v733: 17 pgs: 17 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 787 op/s 2024-05-15T12:55:12.295 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:12.338 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:55:12.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:12.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:55:12.448 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:55:12.528 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:12.655 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:55:12.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:12.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:55:12.723 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 10 1 2024-05-15T12:55:12.768 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:55:12.835 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:12.919 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:55:12.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:12.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:55:13.032 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:55:13.086 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:13.171 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:55:13.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:13.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:55:13.281 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:55:13.310 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:13.334 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:55:13.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:13.338 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:55:13.356 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:55:13.397 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:55:13.433 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:55:13.439 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:55:13.505 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:55:13.521 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:55:13.526 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:13.546 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:55:13.567 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:55:13.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:13.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:55:13.587 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:55:13.623 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:55:13.661 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:55:13.674 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:55:13.694 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:55:13.722 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:55:13.748 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:13.754 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:55:13.780 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:55:13.781 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:55:13.781 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:55:13.781 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:55:13.781 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:55:13.781 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:55:13.781 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:55:13.781 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:55:13.781 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:55:13.782 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:55:13.782 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:55:13.782 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:55:13.782 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:55:13.784 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:55:13.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:13.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:55:13.894 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:55:13.954 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:13 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:55:13.954 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:13 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:55:13.955 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:13 smithi120 ceph-mon[43961]: osdmap e268: 12 total, 12 up, 11 in 2024-05-15T12:55:13.955 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:13 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2818095927' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:55:13.955 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:13 smithi120 ceph-mon[43961]: pgmap v735: 17 pgs: 17 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 935 op/s 2024-05-15T12:55:13.957 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:14.017 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:55:14.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:14.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:55:14.052 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T12:55:14.052 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:55:14.121 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:55:14.168 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:13 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:55:14.168 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:13 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:55:14.168 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:13 smithi027 ceph-mon[39795]: osdmap e268: 12 total, 12 up, 11 in 2024-05-15T12:55:14.168 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:13 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2818095927' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:55:14.168 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:13 smithi027 ceph-mon[39795]: pgmap v735: 17 pgs: 17 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 935 op/s 2024-05-15T12:55:14.178 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:14.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:13 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:55:14.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:13 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:55:14.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:13 smithi177 ceph-mon[44303]: osdmap e268: 12 total, 12 up, 11 in 2024-05-15T12:55:14.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:13 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2818095927' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:55:14.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:13 smithi177 ceph-mon[44303]: pgmap v735: 17 pgs: 17 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 935 op/s 2024-05-15T12:55:14.194 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T12:55:14.247 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:55:14.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:14.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:55:14.357 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:55:14.415 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:14.883 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:55:14.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:14.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:55:14.890 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-05-15T12:55:14.988 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:55:15.022 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:15.396 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:55:15.893 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:55:15.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:15.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:55:15.955 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:55:16.000 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:55:16.029 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:16.050 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:55:16.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:16.053 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:55:16.154 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:55:16.184 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:16.203 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:55:16.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:16.206 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:55:16.307 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:55:16.336 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:16 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:55:16.336 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:16 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:55:16.336 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:16 smithi177 ceph-mon[44303]: osdmap e269: 12 total, 12 up, 11 in 2024-05-15T12:55:16.336 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:16 smithi177 ceph-mon[44303]: pgmap v737: 17 pgs: 17 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 888 op/s 2024-05-15T12:55:16.338 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:16.358 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:55:16.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:16.361 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:55:16.462 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:55:16.502 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:16 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:55:16.502 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:16 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:55:16.503 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:16 smithi027 ceph-mon[39795]: osdmap e269: 12 total, 12 up, 11 in 2024-05-15T12:55:16.503 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:16 smithi027 ceph-mon[39795]: pgmap v737: 17 pgs: 17 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 888 op/s 2024-05-15T12:55:16.505 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:16.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:16 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:55:16.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:16 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:55:16.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:16 smithi120 ceph-mon[43961]: osdmap e269: 12 total, 12 up, 11 in 2024-05-15T12:55:16.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:16 smithi120 ceph-mon[43961]: pgmap v737: 17 pgs: 17 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 888 op/s 2024-05-15T12:55:17.543 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:55:17.714 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.10 to 1 (10000) 2024-05-15T12:55:17.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:17 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1933462753' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T12:55:17.993 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:55:18.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:18.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:55:18.012 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:17 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1933462753' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T12:55:18.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:17 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1933462753' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T12:55:18.112 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:55:18.177 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:18.451 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:55:18.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:18.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:55:18.558 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:55:18.584 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:18.604 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:55:18.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:18.607 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:55:18.708 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:55:18.744 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:18 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1933462753' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T12:55:18.745 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:18 smithi120 ceph-mon[43961]: osdmap e270: 12 total, 12 up, 11 in 2024-05-15T12:55:18.745 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:18 smithi120 ceph-mon[43961]: pgmap v739: 17 pgs: 17 active+clean; 6.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 803 op/s 2024-05-15T12:55:18.748 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:55:18.753 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:18.789 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:55:18.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:18.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:55:18.894 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:55:18.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:18 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1933462753' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T12:55:18.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:18 smithi177 ceph-mon[44303]: osdmap e270: 12 total, 12 up, 11 in 2024-05-15T12:55:18.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:18 smithi177 ceph-mon[44303]: pgmap v739: 17 pgs: 17 active+clean; 6.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 803 op/s 2024-05-15T12:55:18.965 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:19.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:18 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1933462753' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T12:55:19.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:18 smithi027 ceph-mon[39795]: osdmap e270: 12 total, 12 up, 11 in 2024-05-15T12:55:19.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:18 smithi027 ceph-mon[39795]: pgmap v739: 17 pgs: 17 active+clean; 6.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 803 op/s 2024-05-15T12:55:19.125 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:55:19.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:19.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:55:19.237 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:55:19.286 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:19.656 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:55:19.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:19.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:55:19.773 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:55:19.839 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:19.968 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:55:19.972 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:19 smithi120 ceph-mon[43961]: pgmap v740: 17 pgs: 17 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 867 op/s 2024-05-15T12:55:19.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:19.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:55:20.078 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:55:20.104 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:19 smithi177 ceph-mon[44303]: pgmap v740: 17 pgs: 17 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 867 op/s 2024-05-15T12:55:20.106 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:20.126 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:55:20.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:20.128 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:55:20.229 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:55:20.255 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:20.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:19 smithi027 ceph-mon[39795]: pgmap v740: 17 pgs: 17 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 867 op/s 2024-05-15T12:55:20.276 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:55:20.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:20.279 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:55:20.360 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T12:55:20.360 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:55:20.380 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:55:20.408 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:20.429 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:55:20.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:20.432 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:55:20.533 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:55:20.559 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:20.582 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:55:20.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:20.585 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:55:20.686 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:55:20.735 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:20.769 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:55:20.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:20.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:55:20.806 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T12:55:20.884 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:55:20.938 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:21.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:20 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1086641142' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:55:21.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:20 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:55:21.160 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-05-15T12:55:21.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:20 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1086641142' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:55:21.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:20 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:55:21.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:20 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1086641142' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:55:21.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:20 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:55:21.625 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:55:22.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:21 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:55:22.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:21 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:55:22.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:21 smithi027 ceph-mon[39795]: osdmap e271: 12 total, 12 up, 11 in 2024-05-15T12:55:22.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:21 smithi027 ceph-mon[39795]: pgmap v742: 17 pgs: 17 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 855 op/s 2024-05-15T12:55:22.015 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:55:22.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:22.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:55:22.060 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-05-15T12:55:22.061 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T12:55:22.132 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:55:22.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:21 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:55:22.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:21 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:55:22.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:21 smithi177 ceph-mon[44303]: osdmap e271: 12 total, 12 up, 11 in 2024-05-15T12:55:22.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:21 smithi177 ceph-mon[44303]: pgmap v742: 17 pgs: 17 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 855 op/s 2024-05-15T12:55:22.204 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:22.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:21 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:55:22.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:21 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:55:22.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:21 smithi120 ceph-mon[43961]: osdmap e271: 12 total, 12 up, 11 in 2024-05-15T12:55:22.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:21 smithi120 ceph-mon[43961]: pgmap v742: 17 pgs: 17 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 855 op/s 2024-05-15T12:55:22.498 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:55:22.499 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:55:22.507 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:55:22.584 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:55:22.585 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:55:22.636 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:55:22.673 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:55:22.724 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:55:22.758 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:55:22.834 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:55:22.835 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:55:22.850 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:55:22.851 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:55:22.851 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:55:22.851 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:55:22.851 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:55:22.851 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:55:22.851 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:55:22.851 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:55:22.852 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:55:22.852 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:55:22.852 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:55:22.852 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:55:22.852 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:55:23.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:22 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1561931004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:55:23.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:22 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1561931004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:55:23.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:22 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1561931004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:55:23.861 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T12:55:23.861 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T12:55:23.861 INFO:teuthology.orchestra.run.smithi120.stdout: "complaint_time": 30, 2024-05-15T12:55:23.861 INFO:teuthology.orchestra.run.smithi120.stdout: "num_blocked_ops": 0 2024-05-15T12:55:23.861 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T12:55:24.250 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:24 smithi120 ceph-mon[43961]: pgmap v743: 17 pgs: 17 active+clean; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 825 op/s 2024-05-15T12:55:24.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:24 smithi177 ceph-mon[44303]: pgmap v743: 17 pgs: 17 active+clean; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 825 op/s 2024-05-15T12:55:24.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:24 smithi027 ceph-mon[39795]: pgmap v743: 17 pgs: 17 active+clean; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 825 op/s 2024-05-15T12:55:24.639 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-05-15T12:55:25.087 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:55:25.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=53,logm=4} crc {auth=2415987967,config=1059521453,health=179246142,kv=4233349660,logm=3774817477}) 2024-05-15T12:55:25.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2711979579}) 2024-05-15T12:55:25.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1789645826}) 2024-05-15T12:55:25.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2322648897}) 2024-05-15T12:55:25.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4104565502}) 2024-05-15T12:55:25.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3327343107}) 2024-05-15T12:55:25.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1065649752}) 2024-05-15T12:55:25.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4266042033}) 2024-05-15T12:55:25.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2925535695}) 2024-05-15T12:55:25.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4215085994}) 2024-05-15T12:55:25.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2136045958}) 2024-05-15T12:55:25.448 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=53,logm=4} crc {auth=2415987967,config=1059521453,health=179246142,kv=4233349660,logm=3774817477}) 2024-05-15T12:55:25.448 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2711979579}) 2024-05-15T12:55:25.448 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1789645826}) 2024-05-15T12:55:25.448 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2322648897}) 2024-05-15T12:55:25.448 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4104565502}) 2024-05-15T12:55:25.449 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3327343107}) 2024-05-15T12:55:25.449 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1065649752}) 2024-05-15T12:55:25.449 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4266042033}) 2024-05-15T12:55:25.449 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2925535695}) 2024-05-15T12:55:25.449 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4215085994}) 2024-05-15T12:55:25.449 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2136045958}) 2024-05-15T12:55:25.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=53,logm=4} crc {auth=2415987967,config=1059521453,health=179246142,kv=4233349660,logm=3774817477}) 2024-05-15T12:55:25.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2711979579}) 2024-05-15T12:55:25.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1789645826}) 2024-05-15T12:55:25.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2322648897}) 2024-05-15T12:55:25.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4104565502}) 2024-05-15T12:55:25.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3327343107}) 2024-05-15T12:55:25.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1065649752}) 2024-05-15T12:55:25.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4266042033}) 2024-05-15T12:55:25.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2925535695}) 2024-05-15T12:55:25.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4215085994}) 2024-05-15T12:55:25.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2136045958}) 2024-05-15T12:55:26.320 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3092522679}) 2024-05-15T12:55:26.320 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2447229457}) 2024-05-15T12:55:26.320 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1270629206}) 2024-05-15T12:55:26.320 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1299215093}) 2024-05-15T12:55:26.320 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2059728455}) 2024-05-15T12:55:26.320 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3624118011}) 2024-05-15T12:55:26.320 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2075562622}) 2024-05-15T12:55:26.321 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2587056106}) 2024-05-15T12:55:26.321 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2490164203}) 2024-05-15T12:55:26.321 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=910921930}) 2024-05-15T12:55:26.321 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2374751343}) 2024-05-15T12:55:26.321 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=916426931}) 2024-05-15T12:55:26.321 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=630051579}) 2024-05-15T12:55:26.321 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2613859207}) 2024-05-15T12:55:26.321 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3675549082}) 2024-05-15T12:55:26.321 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=787558570}) 2024-05-15T12:55:26.321 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2610018071}) 2024-05-15T12:55:26.321 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=263040210}) 2024-05-15T12:55:26.321 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1179330171}) 2024-05-15T12:55:26.321 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=63,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=13} crc {logm=402090617,mdsmap=3984425268,mgr=76093979,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=730470246}) 2024-05-15T12:55:26.322 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=1,mon_config_key=99} crc {mgrstat=1896939529,mon_config_key=3488710819}) 2024-05-15T12:55:26.322 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3743217500}) 2024-05-15T12:55:26.322 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=76,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=6} crc {mon_config_key=2222729488,monmap=2435304556,osd_metadata=1212035326,osd_pg_creating=1443027706,osdmap=427232963}) 2024-05-15T12:55:26.322 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=20690031}) 2024-05-15T12:55:26.322 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=1} crc {osdmap=3496029462}) 2024-05-15T12:55:26.322 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:26 smithi120 ceph-mon[43961]: pgmap v744: 17 pgs: 17 active+clean; 5.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 857 op/s 2024-05-15T12:55:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3092522679}) 2024-05-15T12:55:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2447229457}) 2024-05-15T12:55:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1270629206}) 2024-05-15T12:55:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1299215093}) 2024-05-15T12:55:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2059728455}) 2024-05-15T12:55:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3624118011}) 2024-05-15T12:55:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2075562622}) 2024-05-15T12:55:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2587056106}) 2024-05-15T12:55:26.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2490164203}) 2024-05-15T12:55:26.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=910921930}) 2024-05-15T12:55:26.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2374751343}) 2024-05-15T12:55:26.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=916426931}) 2024-05-15T12:55:26.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=630051579}) 2024-05-15T12:55:26.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2613859207}) 2024-05-15T12:55:26.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3675549082}) 2024-05-15T12:55:26.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=787558570}) 2024-05-15T12:55:26.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2610018071}) 2024-05-15T12:55:26.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=263040210}) 2024-05-15T12:55:26.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1179330171}) 2024-05-15T12:55:26.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=63,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=13} crc {logm=402090617,mdsmap=3984425268,mgr=76093979,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=730470246}) 2024-05-15T12:55:26.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=1,mon_config_key=99} crc {mgrstat=1896939529,mon_config_key=3488710819}) 2024-05-15T12:55:26.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3743217500}) 2024-05-15T12:55:26.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=76,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=6} crc {mon_config_key=2222729488,monmap=2435304556,osd_metadata=1212035326,osd_pg_creating=1443027706,osdmap=427232963}) 2024-05-15T12:55:26.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=20690031}) 2024-05-15T12:55:26.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=1} crc {osdmap=3496029462}) 2024-05-15T12:55:26.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:26 smithi177 ceph-mon[44303]: pgmap v744: 17 pgs: 17 active+clean; 5.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 857 op/s 2024-05-15T12:55:26.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3092522679}) 2024-05-15T12:55:26.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2447229457}) 2024-05-15T12:55:26.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1270629206}) 2024-05-15T12:55:26.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1299215093}) 2024-05-15T12:55:26.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2059728455}) 2024-05-15T12:55:26.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3624118011}) 2024-05-15T12:55:26.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2075562622}) 2024-05-15T12:55:26.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2587056106}) 2024-05-15T12:55:26.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2490164203}) 2024-05-15T12:55:26.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=910921930}) 2024-05-15T12:55:26.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2374751343}) 2024-05-15T12:55:26.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=916426931}) 2024-05-15T12:55:26.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=630051579}) 2024-05-15T12:55:26.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2613859207}) 2024-05-15T12:55:26.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3675549082}) 2024-05-15T12:55:26.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=787558570}) 2024-05-15T12:55:26.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2610018071}) 2024-05-15T12:55:26.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=263040210}) 2024-05-15T12:55:26.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1179330171}) 2024-05-15T12:55:26.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=63,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=13} crc {logm=402090617,mdsmap=3984425268,mgr=76093979,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=730470246}) 2024-05-15T12:55:26.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=1,mon_config_key=99} crc {mgrstat=1896939529,mon_config_key=3488710819}) 2024-05-15T12:55:26.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3743217500}) 2024-05-15T12:55:26.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=76,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=6} crc {mon_config_key=2222729488,monmap=2435304556,osd_metadata=1212035326,osd_pg_creating=1443027706,osdmap=427232963}) 2024-05-15T12:55:26.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=20690031}) 2024-05-15T12:55:26.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=1} crc {osdmap=3496029462}) 2024-05-15T12:55:26.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:26 smithi027 ceph-mon[39795]: pgmap v744: 17 pgs: 17 active+clean; 5.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 857 op/s 2024-05-15T12:55:27.609 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T12:55:27.609 INFO:teuthology.orchestra.run.smithi120.stdout: "size": 20, 2024-05-15T12:55:27.610 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 600, 2024-05-15T12:55:27.610 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [ 2024-05-15T12:55:27.610 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.610 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:48588 2.0 2:0abdfbd3:::benchmark_data_smithi177_78104_object48587:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-05-15T12:55:27.610 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:39.871751+0000", 2024-05-15T12:55:27.610 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 407.73628308899998, 2024-05-15T12:55:27.610 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.009894506, 2024-05-15T12:55:27.610 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:27.610 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:27.610 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:55:27.610 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:55:27.610 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:27.611 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 48588 2024-05-15T12:55:27.611 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.611 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:27.611 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.611 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:27.611 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.871751+0000", 2024-05-15T12:55:27.611 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.611 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.611 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.611 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:27.611 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.871751+0000", 2024-05-15T12:55:27.611 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.611 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.611 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.612 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:27.612 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.871752+0000", 2024-05-15T12:55:27.612 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.173e-06 2024-05-15T12:55:27.612 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.612 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.612 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:27.612 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.871799+0000", 2024-05-15T12:55:27.612 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6832000000000002e-05 2024-05-15T12:55:27.612 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.612 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.612 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:27.612 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.871809+0000", 2024-05-15T12:55:27.612 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0266e-05 2024-05-15T12:55:27.613 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.613 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.613 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:27.613 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.871815+0000", 2024-05-15T12:55:27.613 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.613e-06 2024-05-15T12:55:27.613 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.613 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.613 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.613 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.871860+0000", 2024-05-15T12:55:27.613 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4818999999999999e-05 2024-05-15T12:55:27.613 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.613 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.613 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:55:27.614 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.871870+0000", 2024-05-15T12:55:27.614 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.961e-06 2024-05-15T12:55:27.614 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.614 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.614 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.614 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.877374+0000", 2024-05-15T12:55:27.614 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0055037609999999 2024-05-15T12:55:27.614 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.614 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.614 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:55:27.614 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.877499+0000", 2024-05-15T12:55:27.614 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000124759 2024-05-15T12:55:27.614 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.615 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.615 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 9,11", 2024-05-15T12:55:27.615 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.877661+0000", 2024-05-15T12:55:27.615 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000161953 2024-05-15T12:55:27.615 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.615 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.615 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:55:27.615 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.880240+0000", 2024-05-15T12:55:27.615 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.002579605 2024-05-15T12:55:27.615 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.615 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.615 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.615 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.881461+0000", 2024-05-15T12:55:27.616 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0012207489999999999 2024-05-15T12:55:27.616 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.616 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.616 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.616 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.881581+0000", 2024-05-15T12:55:27.616 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000120297 2024-05-15T12:55:27.616 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.616 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.616 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:55:27.616 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.881615+0000", 2024-05-15T12:55:27.616 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.3599999999999997e-05 2024-05-15T12:55:27.616 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.616 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.617 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:55:27.617 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.881645+0000", 2024-05-15T12:55:27.617 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.0117999999999999e-05 2024-05-15T12:55:27.617 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.617 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:27.617 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.617 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.617 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.617 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:48608 2.0 2:0f2ee1c7:::benchmark_data_smithi177_78104_object48607:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-05-15T12:55:27.617 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:39.880196+0000", 2024-05-15T12:55:27.617 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 407.727838045, 2024-05-15T12:55:27.617 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.003832877, 2024-05-15T12:55:27.617 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:27.617 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:27.618 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:55:27.618 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:55:27.618 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:27.618 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 48608 2024-05-15T12:55:27.618 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.618 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:27.618 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.618 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:27.618 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.880196+0000", 2024-05-15T12:55:27.618 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.618 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.618 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.619 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:27.619 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.880196+0000", 2024-05-15T12:55:27.619 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.619 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.619 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.619 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:27.619 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.880196+0000", 2024-05-15T12:55:27.619 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.1900000000000002e-07 2024-05-15T12:55:27.619 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.619 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.619 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:27.620 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.880291+0000", 2024-05-15T12:55:27.620 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.4562999999999996e-05 2024-05-15T12:55:27.620 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.620 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.620 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:27.620 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.880301+0000", 2024-05-15T12:55:27.620 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0229e-05 2024-05-15T12:55:27.620 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.620 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.620 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:27.620 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.880307+0000", 2024-05-15T12:55:27.620 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5709999999999997e-06 2024-05-15T12:55:27.620 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.620 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.620 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.621 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.880362+0000", 2024-05-15T12:55:27.621 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5504000000000003e-05 2024-05-15T12:55:27.621 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.621 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.621 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:55:27.621 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.880372+0000", 2024-05-15T12:55:27.621 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0220999999999999e-05 2024-05-15T12:55:27.621 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.621 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.621 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.621 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.879197+0000", 2024-05-15T12:55:27.621 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.99882503499999997 2024-05-15T12:55:27.621 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.621 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.622 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:55:27.622 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.879319+0000", 2024-05-15T12:55:27.622 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012148 2024-05-15T12:55:27.622 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.622 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.622 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 9,11", 2024-05-15T12:55:27.622 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.879462+0000", 2024-05-15T12:55:27.622 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000142686 2024-05-15T12:55:27.622 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.622 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.622 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:55:27.622 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.882697+0000", 2024-05-15T12:55:27.622 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.003235057 2024-05-15T12:55:27.623 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.623 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.623 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.623 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.883899+0000", 2024-05-15T12:55:27.623 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001202276 2024-05-15T12:55:27.623 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.623 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.623 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.623 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.883954+0000", 2024-05-15T12:55:27.623 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5198000000000002e-05 2024-05-15T12:55:27.623 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.623 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.623 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:55:27.623 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.883989+0000", 2024-05-15T12:55:27.623 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5131e-05 2024-05-15T12:55:27.624 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.624 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.624 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:55:27.624 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.884028+0000", 2024-05-15T12:55:27.624 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.9206999999999999e-05 2024-05-15T12:55:27.624 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.624 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:27.624 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.624 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.624 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.624 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:48660 2.0 2:0a7e60a3:::benchmark_data_smithi177_78104_object48659:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-05-15T12:55:27.624 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:39.913625+0000", 2024-05-15T12:55:27.624 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 407.694408893, 2024-05-15T12:55:27.624 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.97062420400000005, 2024-05-15T12:55:27.625 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:27.625 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:27.625 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:55:27.625 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:55:27.625 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:27.625 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 48660 2024-05-15T12:55:27.625 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.625 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:27.625 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.625 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:27.625 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.913625+0000", 2024-05-15T12:55:27.625 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.625 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.625 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.625 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:27.626 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.913625+0000", 2024-05-15T12:55:27.626 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.626 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.626 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.626 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:27.626 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.913625+0000", 2024-05-15T12:55:27.626 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7199999999999998e-07 2024-05-15T12:55:27.626 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.626 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.626 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:27.626 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.913721+0000", 2024-05-15T12:55:27.626 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.6016999999999999e-05 2024-05-15T12:55:27.626 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.627 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.627 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:27.627 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.913732+0000", 2024-05-15T12:55:27.627 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0839000000000001e-05 2024-05-15T12:55:27.627 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.627 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.627 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:27.627 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.913736+0000", 2024-05-15T12:55:27.627 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.211e-06 2024-05-15T12:55:27.627 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.627 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.627 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.627 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.913777+0000", 2024-05-15T12:55:27.627 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0380999999999998e-05 2024-05-15T12:55:27.627 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.628 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.628 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:55:27.628 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.913783+0000", 2024-05-15T12:55:27.628 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9560000000000002e-06 2024-05-15T12:55:27.628 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.628 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.628 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.628 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.880272+0000", 2024-05-15T12:55:27.628 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.96648962199999999 2024-05-15T12:55:27.628 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.628 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.628 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:55:27.628 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.880376+0000", 2024-05-15T12:55:27.629 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00010384500000000001 2024-05-15T12:55:27.629 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.629 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.629 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 9,11", 2024-05-15T12:55:27.629 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.880526+0000", 2024-05-15T12:55:27.629 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014945900000000001 2024-05-15T12:55:27.629 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.629 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.629 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:55:27.629 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.883913+0000", 2024-05-15T12:55:27.629 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0033875060000000002 2024-05-15T12:55:27.629 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.629 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.629 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.630 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.884081+0000", 2024-05-15T12:55:27.630 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00016768800000000001 2024-05-15T12:55:27.630 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.630 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.630 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.630 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.884178+0000", 2024-05-15T12:55:27.630 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.7449999999999997e-05 2024-05-15T12:55:27.630 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.630 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.630 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:55:27.630 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.884222+0000", 2024-05-15T12:55:27.630 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3696999999999999e-05 2024-05-15T12:55:27.630 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.630 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.631 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:55:27.631 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.884249+0000", 2024-05-15T12:55:27.631 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.6860999999999999e-05 2024-05-15T12:55:27.631 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.631 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:27.631 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.631 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.631 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.631 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:48686 2.0 2:0963877c:::benchmark_data_smithi177_78104_object48685:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-05-15T12:55:27.631 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:39.927128+0000", 2024-05-15T12:55:27.631 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 407.68090519899999, 2024-05-15T12:55:27.631 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.95972319299999997, 2024-05-15T12:55:27.631 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:27.631 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:27.632 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:55:27.632 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:55:27.632 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:27.632 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 48686 2024-05-15T12:55:27.632 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.632 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:27.632 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.632 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:27.632 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.927128+0000", 2024-05-15T12:55:27.632 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.632 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.632 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.632 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:27.632 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.927128+0000", 2024-05-15T12:55:27.632 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.633 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.633 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.633 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:27.633 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.927132+0000", 2024-05-15T12:55:27.633 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0219999999999998e-06 2024-05-15T12:55:27.633 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.633 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.633 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:27.633 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.927244+0000", 2024-05-15T12:55:27.633 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000111086 2024-05-15T12:55:27.633 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.633 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.633 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:27.633 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.927253+0000", 2024-05-15T12:55:27.634 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1330000000000005e-06 2024-05-15T12:55:27.634 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.634 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.634 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:27.634 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.927261+0000", 2024-05-15T12:55:27.634 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9820000000000008e-06 2024-05-15T12:55:27.634 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.634 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.634 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.634 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.927324+0000", 2024-05-15T12:55:27.634 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3847e-05 2024-05-15T12:55:27.634 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.634 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.634 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:55:27.635 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.927336+0000", 2024-05-15T12:55:27.635 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1096e-05 2024-05-15T12:55:27.635 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.635 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.635 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.635 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.881687+0000", 2024-05-15T12:55:27.635 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.95435096600000002 2024-05-15T12:55:27.635 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.635 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.635 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:55:27.635 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.881809+0000", 2024-05-15T12:55:27.635 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000122001 2024-05-15T12:55:27.635 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.635 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.635 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 9,11", 2024-05-15T12:55:27.636 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.881951+0000", 2024-05-15T12:55:27.636 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014225699999999999 2024-05-15T12:55:27.636 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.636 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.636 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:55:27.636 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.884094+0000", 2024-05-15T12:55:27.636 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0021427529999999998 2024-05-15T12:55:27.636 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.636 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.636 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.636 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.884128+0000", 2024-05-15T12:55:27.636 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.4662000000000001e-05 2024-05-15T12:55:27.636 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.636 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.637 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.637 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.886734+0000", 2024-05-15T12:55:27.637 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0026059500000000001 2024-05-15T12:55:27.637 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.637 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.637 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:55:27.637 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.886795+0000", 2024-05-15T12:55:27.637 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1197000000000004e-05 2024-05-15T12:55:27.637 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.637 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.637 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:55:27.637 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.886852+0000", 2024-05-15T12:55:27.637 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6240999999999999e-05 2024-05-15T12:55:27.638 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.638 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:27.638 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.638 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.638 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.638 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:48687 2.0 2:06ab9848:::benchmark_data_smithi177_78104_object48686:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-05-15T12:55:27.638 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:39.927279+0000", 2024-05-15T12:55:27.638 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 407.68075424599999, 2024-05-15T12:55:27.638 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.95978231899999999, 2024-05-15T12:55:27.638 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:27.638 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:27.638 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:55:27.639 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:55:27.639 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:27.639 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 48687 2024-05-15T12:55:27.639 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.639 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:27.639 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.639 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:27.639 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.927279+0000", 2024-05-15T12:55:27.639 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.639 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.639 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.640 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:27.640 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.927279+0000", 2024-05-15T12:55:27.640 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.640 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.640 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.640 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:27.640 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.927280+0000", 2024-05-15T12:55:27.640 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.0800000000000004e-07 2024-05-15T12:55:27.640 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.640 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.640 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:27.640 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.927315+0000", 2024-05-15T12:55:27.640 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5111999999999999e-05 2024-05-15T12:55:27.641 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.641 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.641 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:27.641 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.927323+0000", 2024-05-15T12:55:27.641 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7789999999999996e-06 2024-05-15T12:55:27.641 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.641 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.641 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:27.641 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.927329+0000", 2024-05-15T12:55:27.641 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.0680000000000002e-06 2024-05-15T12:55:27.641 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.641 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.641 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.641 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.927367+0000", 2024-05-15T12:55:27.642 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.8367e-05 2024-05-15T12:55:27.642 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.642 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.642 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:55:27.642 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.927375+0000", 2024-05-15T12:55:27.642 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8709999999999997e-06 2024-05-15T12:55:27.642 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.642 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.642 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.642 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.882750+0000", 2024-05-15T12:55:27.642 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.95537470499999999 2024-05-15T12:55:27.642 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.643 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.643 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:55:27.643 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.882846+0000", 2024-05-15T12:55:27.643 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.6124000000000003e-05 2024-05-15T12:55:27.643 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.643 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.643 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 9,11", 2024-05-15T12:55:27.643 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.882972+0000", 2024-05-15T12:55:27.643 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000126291 2024-05-15T12:55:27.643 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.643 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.643 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:55:27.644 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.884140+0000", 2024-05-15T12:55:27.644 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001167974 2024-05-15T12:55:27.644 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.644 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.644 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.644 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.886907+0000", 2024-05-15T12:55:27.644 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0027663309999999999 2024-05-15T12:55:27.644 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.644 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.644 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.644 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.886960+0000", 2024-05-15T12:55:27.644 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3151e-05 2024-05-15T12:55:27.644 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.645 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.645 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:55:27.645 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.887013+0000", 2024-05-15T12:55:27.645 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3677000000000001e-05 2024-05-15T12:55:27.645 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.645 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.645 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:55:27.645 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.887062+0000", 2024-05-15T12:55:27.645 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8260999999999998e-05 2024-05-15T12:55:27.645 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.645 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:27.645 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.646 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.646 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.646 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:48699 2.0 2:0dd8cb65:::benchmark_data_smithi177_78104_object48698:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-05-15T12:55:27.646 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:39.934915+0000", 2024-05-15T12:55:27.646 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 407.67311878700002, 2024-05-15T12:55:27.646 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.95509175300000004, 2024-05-15T12:55:27.646 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:27.646 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:27.646 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:55:27.646 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:55:27.646 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:27.647 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 48699 2024-05-15T12:55:27.647 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.647 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:27.647 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.647 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:27.647 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.934915+0000", 2024-05-15T12:55:27.647 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.647 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.647 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.647 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:27.647 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.934915+0000", 2024-05-15T12:55:27.647 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.648 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.648 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.648 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:27.648 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.934916+0000", 2024-05-15T12:55:27.648 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.119e-06 2024-05-15T12:55:27.648 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.648 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.648 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:27.648 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.934972+0000", 2024-05-15T12:55:27.648 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5711000000000002e-05 2024-05-15T12:55:27.648 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.648 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.648 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:27.649 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.934983+0000", 2024-05-15T12:55:27.649 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1398e-05 2024-05-15T12:55:27.649 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.649 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.649 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:27.649 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.934989+0000", 2024-05-15T12:55:27.649 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6899999999999997e-06 2024-05-15T12:55:27.649 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.649 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.649 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.649 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.935037+0000", 2024-05-15T12:55:27.649 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7862999999999999e-05 2024-05-15T12:55:27.649 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.649 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.650 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:55:27.650 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.935044+0000", 2024-05-15T12:55:27.650 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.374e-06 2024-05-15T12:55:27.650 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.650 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.650 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.650 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.884287+0000", 2024-05-15T12:55:27.650 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.94924350400000002 2024-05-15T12:55:27.650 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.650 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.650 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:55:27.650 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.884401+0000", 2024-05-15T12:55:27.650 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000113368 2024-05-15T12:55:27.650 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.651 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.651 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 9,11", 2024-05-15T12:55:27.651 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.884549+0000", 2024-05-15T12:55:27.651 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000148047 2024-05-15T12:55:27.651 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.651 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.651 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:55:27.651 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.887074+0000", 2024-05-15T12:55:27.651 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0025255339999999999 2024-05-15T12:55:27.651 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.651 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.651 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.651 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.887128+0000", 2024-05-15T12:55:27.651 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3628000000000002e-05 2024-05-15T12:55:27.651 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.652 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.652 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.652 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.889947+0000", 2024-05-15T12:55:27.652 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0028193469999999998 2024-05-15T12:55:27.652 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.652 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.652 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:55:27.652 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.889981+0000", 2024-05-15T12:55:27.652 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.3149999999999999e-05 2024-05-15T12:55:27.652 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.652 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.652 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:55:27.652 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.890007+0000", 2024-05-15T12:55:27.652 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.6020000000000002e-05 2024-05-15T12:55:27.652 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.653 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:27.653 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.653 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.653 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.653 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:48702 2.0 2:0e29b217:::benchmark_data_smithi177_78104_object48701:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-05-15T12:55:27.653 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:39.937593+0000", 2024-05-15T12:55:27.653 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 407.67044034499997, 2024-05-15T12:55:27.653 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.95254754799999997, 2024-05-15T12:55:27.653 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:27.653 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:27.653 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:55:27.653 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:55:27.653 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:27.653 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 48702 2024-05-15T12:55:27.653 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.654 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:27.654 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.654 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:27.654 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.937593+0000", 2024-05-15T12:55:27.654 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.654 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.654 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.654 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:27.654 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.937593+0000", 2024-05-15T12:55:27.654 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.654 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.654 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.654 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:27.654 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.937594+0000", 2024-05-15T12:55:27.654 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1099999999999996e-07 2024-05-15T12:55:27.655 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.655 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.655 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:27.655 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.937688+0000", 2024-05-15T12:55:27.655 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.3807000000000005e-05 2024-05-15T12:55:27.655 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.655 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.655 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:27.655 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.937693+0000", 2024-05-15T12:55:27.655 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5779999999999998e-06 2024-05-15T12:55:27.655 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.655 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.655 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:27.655 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.937697+0000", 2024-05-15T12:55:27.656 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.9029999999999997e-06 2024-05-15T12:55:27.656 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.656 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.656 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.656 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.937747+0000", 2024-05-15T12:55:27.656 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0413e-05 2024-05-15T12:55:27.656 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.656 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.656 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:55:27.656 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.937758+0000", 2024-05-15T12:55:27.656 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.044e-05 2024-05-15T12:55:27.656 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.656 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.656 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.657 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.885360+0000", 2024-05-15T12:55:27.657 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.94760217599999996 2024-05-15T12:55:27.657 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.657 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.657 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:55:27.657 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.885475+0000", 2024-05-15T12:55:27.657 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000114432 2024-05-15T12:55:27.657 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.657 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.657 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 9,11", 2024-05-15T12:55:27.657 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.885618+0000", 2024-05-15T12:55:27.657 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014369799999999999 2024-05-15T12:55:27.657 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.657 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.658 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:55:27.658 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.887082+0000", 2024-05-15T12:55:27.658 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0014637470000000001 2024-05-15T12:55:27.658 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.658 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.658 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.658 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.890051+0000", 2024-05-15T12:55:27.658 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0029686410000000002 2024-05-15T12:55:27.658 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.658 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.658 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.658 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.890091+0000", 2024-05-15T12:55:27.658 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.9969000000000003e-05 2024-05-15T12:55:27.658 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.659 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.659 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:55:27.659 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.890119+0000", 2024-05-15T12:55:27.659 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.8884999999999999e-05 2024-05-15T12:55:27.659 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.659 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.659 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:55:27.659 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.890141+0000", 2024-05-15T12:55:27.659 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1348e-05 2024-05-15T12:55:27.659 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.659 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:27.659 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.659 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.660 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.660 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:48703 2.0 2:061e4c88:::benchmark_data_smithi177_78104_object48702:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-05-15T12:55:27.660 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:39.937705+0000", 2024-05-15T12:55:27.660 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 407.67032851900001, 2024-05-15T12:55:27.660 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.95255814100000002, 2024-05-15T12:55:27.660 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:27.660 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:27.660 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:55:27.660 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:55:27.660 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:27.660 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 48703 2024-05-15T12:55:27.660 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.660 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:27.660 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.661 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:27.661 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.937705+0000", 2024-05-15T12:55:27.661 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.661 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.661 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.661 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:27.661 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.937705+0000", 2024-05-15T12:55:27.661 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.661 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.661 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.661 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:27.661 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.937706+0000", 2024-05-15T12:55:27.661 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3700000000000001e-07 2024-05-15T12:55:27.661 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.661 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.662 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:27.662 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.937762+0000", 2024-05-15T12:55:27.662 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6974e-05 2024-05-15T12:55:27.662 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.662 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.662 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:27.662 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.937771+0000", 2024-05-15T12:55:27.662 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.4460000000000001e-06 2024-05-15T12:55:27.662 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.662 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.662 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:27.662 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.937775+0000", 2024-05-15T12:55:27.662 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.9349999999999996e-06 2024-05-15T12:55:27.662 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.662 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.663 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.663 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.937807+0000", 2024-05-15T12:55:27.663 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.1677000000000002e-05 2024-05-15T12:55:27.663 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.663 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.663 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:55:27.663 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.937814+0000", 2024-05-15T12:55:27.663 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4889999999999998e-06 2024-05-15T12:55:27.663 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.663 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.663 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.663 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.887169+0000", 2024-05-15T12:55:27.663 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.949354476 2024-05-15T12:55:27.663 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.664 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.664 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:55:27.664 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.887277+0000", 2024-05-15T12:55:27.664 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0001088 2024-05-15T12:55:27.664 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.664 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.664 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 9,11", 2024-05-15T12:55:27.664 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.887418+0000", 2024-05-15T12:55:27.664 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014022500000000001 2024-05-15T12:55:27.664 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.664 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.664 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:55:27.664 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.890065+0000", 2024-05-15T12:55:27.664 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0026473669999999999 2024-05-15T12:55:27.664 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.665 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.665 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.665 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.890174+0000", 2024-05-15T12:55:27.665 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000109208 2024-05-15T12:55:27.665 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.665 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.665 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.665 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.890211+0000", 2024-05-15T12:55:27.665 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7137e-05 2024-05-15T12:55:27.665 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.665 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.665 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:55:27.666 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.890241+0000", 2024-05-15T12:55:27.666 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.9961999999999999e-05 2024-05-15T12:55:27.666 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.666 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.666 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:55:27.666 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.890263+0000", 2024-05-15T12:55:27.666 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2008e-05 2024-05-15T12:55:27.666 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.666 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:27.666 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.666 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.666 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.666 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:48730 2.0 2:00ff964d:::benchmark_data_smithi177_78104_object48729:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-05-15T12:55:27.666 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:39.961610+0000", 2024-05-15T12:55:27.667 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 407.64642398500001, 2024-05-15T12:55:27.667 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.93792978000000005, 2024-05-15T12:55:27.667 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:27.667 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:27.667 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:55:27.667 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:55:27.667 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:27.667 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 48730 2024-05-15T12:55:27.667 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.667 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:27.667 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.667 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:27.668 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.961610+0000", 2024-05-15T12:55:27.668 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.668 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.668 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.668 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:27.668 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.961610+0000", 2024-05-15T12:55:27.668 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.668 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.668 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.668 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:27.668 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.961611+0000", 2024-05-15T12:55:27.668 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0139999999999999e-06 2024-05-15T12:55:27.668 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.668 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.669 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:27.669 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.961683+0000", 2024-05-15T12:55:27.669 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2333e-05 2024-05-15T12:55:27.669 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.669 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.669 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:27.669 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.961693+0000", 2024-05-15T12:55:27.669 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0519e-05 2024-05-15T12:55:27.669 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.669 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.669 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:27.669 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.961699+0000", 2024-05-15T12:55:27.669 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.4639999999999997e-06 2024-05-15T12:55:27.669 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.670 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.670 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.670 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.961753+0000", 2024-05-15T12:55:27.670 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.4017000000000001e-05 2024-05-15T12:55:27.670 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.670 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.670 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:55:27.670 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.961761+0000", 2024-05-15T12:55:27.670 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7179999999999996e-06 2024-05-15T12:55:27.670 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.670 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.670 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.670 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.888761+0000", 2024-05-15T12:55:27.671 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.92700030700000002 2024-05-15T12:55:27.671 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.671 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.671 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:55:27.671 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.888839+0000", 2024-05-15T12:55:27.671 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8276999999999996e-05 2024-05-15T12:55:27.671 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.671 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.671 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 9,11", 2024-05-15T12:55:27.671 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.888923+0000", 2024-05-15T12:55:27.671 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.3798e-05 2024-05-15T12:55:27.671 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.671 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.671 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:55:27.672 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.890183+0000", 2024-05-15T12:55:27.672 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001259722 2024-05-15T12:55:27.672 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.672 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.672 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.672 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.892423+0000", 2024-05-15T12:55:27.672 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0022402609999999999 2024-05-15T12:55:27.672 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.672 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.672 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.672 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.899434+0000", 2024-05-15T12:55:27.672 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0070111959999999999 2024-05-15T12:55:27.672 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.673 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.673 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:55:27.673 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.899493+0000", 2024-05-15T12:55:27.673 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8631000000000002e-05 2024-05-15T12:55:27.673 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.673 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.673 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:55:27.673 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.899539+0000", 2024-05-15T12:55:27.673 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6523000000000002e-05 2024-05-15T12:55:27.673 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.673 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:27.673 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.673 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.674 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.674 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:48743 2.0 2:0f363763:::benchmark_data_smithi177_78104_object48742:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-05-15T12:55:27.674 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:39.975705+0000", 2024-05-15T12:55:27.674 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 407.63232846300002, 2024-05-15T12:55:27.674 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.92393715600000004, 2024-05-15T12:55:27.674 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:27.674 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:27.674 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:55:27.674 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:55:27.674 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:27.674 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 48743 2024-05-15T12:55:27.674 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.674 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:27.674 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.675 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:27.675 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.975705+0000", 2024-05-15T12:55:27.675 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.675 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.675 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.675 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:27.675 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.975705+0000", 2024-05-15T12:55:27.675 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.675 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.675 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.675 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:27.675 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.975706+0000", 2024-05-15T12:55:27.675 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.5799999999999998e-07 2024-05-15T12:55:27.675 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.676 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.676 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:27.676 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.975785+0000", 2024-05-15T12:55:27.676 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8504000000000006e-05 2024-05-15T12:55:27.676 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.676 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.676 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:27.676 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.975795+0000", 2024-05-15T12:55:27.676 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0669999999999999e-05 2024-05-15T12:55:27.676 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.676 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.676 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:27.676 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.975801+0000", 2024-05-15T12:55:27.677 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1800000000000001e-06 2024-05-15T12:55:27.677 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.677 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.677 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.677 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.975860+0000", 2024-05-15T12:55:27.677 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8167000000000001e-05 2024-05-15T12:55:27.677 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.677 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.677 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:55:27.677 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.975870+0000", 2024-05-15T12:55:27.677 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0380999999999999e-05 2024-05-15T12:55:27.677 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.677 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.677 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.678 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.890294+0000", 2024-05-15T12:55:27.678 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.91442419500000005 2024-05-15T12:55:27.678 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.678 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.678 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:55:27.678 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.890359+0000", 2024-05-15T12:55:27.678 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4956000000000006e-05 2024-05-15T12:55:27.678 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.678 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.678 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 9,11", 2024-05-15T12:55:27.678 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.890445+0000", 2024-05-15T12:55:27.678 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.6089999999999997e-05 2024-05-15T12:55:27.678 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.679 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.679 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:55:27.679 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.893665+0000", 2024-05-15T12:55:27.679 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0032193209999999998 2024-05-15T12:55:27.679 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.679 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.679 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.679 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.894686+0000", 2024-05-15T12:55:27.679 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001021419 2024-05-15T12:55:27.679 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.679 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.679 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.679 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.899580+0000", 2024-05-15T12:55:27.680 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0048937060000000003 2024-05-15T12:55:27.680 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.680 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.680 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:55:27.680 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.899614+0000", 2024-05-15T12:55:27.680 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.3880000000000001e-05 2024-05-15T12:55:27.680 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.680 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.680 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:55:27.680 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.899642+0000", 2024-05-15T12:55:27.680 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.8728999999999999e-05 2024-05-15T12:55:27.680 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.680 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:27.680 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.681 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.681 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.681 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:48761 2.0 2:0a067cc4:::benchmark_data_smithi177_78104_object48760:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-05-15T12:55:27.681 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:39.988407+0000", 2024-05-15T12:55:27.681 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 407.619626681, 2024-05-15T12:55:27.681 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.91137896500000004, 2024-05-15T12:55:27.681 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:27.681 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:27.681 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:55:27.681 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:55:27.681 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:27.681 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 48761 2024-05-15T12:55:27.681 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.681 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:27.681 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.682 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:27.682 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.988407+0000", 2024-05-15T12:55:27.682 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.682 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.682 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.682 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:27.682 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.988407+0000", 2024-05-15T12:55:27.682 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.682 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.682 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.682 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:27.682 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.988408+0000", 2024-05-15T12:55:27.682 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3430000000000001e-06 2024-05-15T12:55:27.682 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.683 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.683 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:27.683 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.988499+0000", 2024-05-15T12:55:27.683 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1027999999999997e-05 2024-05-15T12:55:27.683 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.683 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.683 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:27.683 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.988510+0000", 2024-05-15T12:55:27.683 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1092e-05 2024-05-15T12:55:27.683 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.683 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.683 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:27.683 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.988518+0000", 2024-05-15T12:55:27.683 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8650000000000001e-06 2024-05-15T12:55:27.684 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.684 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.684 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.684 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.988584+0000", 2024-05-15T12:55:27.684 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6240000000000003e-05 2024-05-15T12:55:27.684 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.684 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.684 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:55:27.684 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.988595+0000", 2024-05-15T12:55:27.684 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0560999999999999e-05 2024-05-15T12:55:27.684 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.684 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.684 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.684 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.891316+0000", 2024-05-15T12:55:27.684 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.90272076999999995 2024-05-15T12:55:27.685 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.685 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.685 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:55:27.685 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.891401+0000", 2024-05-15T12:55:27.685 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.4695000000000001e-05 2024-05-15T12:55:27.685 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.685 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.685 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 9,11", 2024-05-15T12:55:27.685 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.891514+0000", 2024-05-15T12:55:27.685 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00011394499999999999 2024-05-15T12:55:27.685 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.685 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.685 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:55:27.685 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.894694+0000", 2024-05-15T12:55:27.685 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.003179313 2024-05-15T12:55:27.686 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.686 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.686 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.686 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.897285+0000", 2024-05-15T12:55:27.686 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0025908540000000001 2024-05-15T12:55:27.686 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.686 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.686 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.686 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.899681+0000", 2024-05-15T12:55:27.686 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0023962229999999998 2024-05-15T12:55:27.686 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.686 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.686 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:55:27.686 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.899745+0000", 2024-05-15T12:55:27.687 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3887999999999995e-05 2024-05-15T12:55:27.687 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.687 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.687 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:55:27.687 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.899786+0000", 2024-05-15T12:55:27.687 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.1148e-05 2024-05-15T12:55:27.687 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.687 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:27.687 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.687 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.687 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.687 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:48762 2.0 2:046869aa:::benchmark_data_smithi177_78104_object48761:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-05-15T12:55:27.687 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:39.988538+0000", 2024-05-15T12:55:27.687 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 407.61949600299999, 2024-05-15T12:55:27.688 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.91574645700000001, 2024-05-15T12:55:27.688 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:27.688 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:27.688 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:55:27.688 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:55:27.688 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:27.688 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 48762 2024-05-15T12:55:27.688 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.688 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:27.688 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.688 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:27.688 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.988538+0000", 2024-05-15T12:55:27.688 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.688 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.689 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.689 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:27.689 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.988538+0000", 2024-05-15T12:55:27.689 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.689 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.689 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.689 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:27.689 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.988538+0000", 2024-05-15T12:55:27.689 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6800000000000001e-07 2024-05-15T12:55:27.689 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.689 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.689 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:27.689 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.988616+0000", 2024-05-15T12:55:27.689 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8234999999999999e-05 2024-05-15T12:55:27.689 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.689 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.690 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:27.690 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.988626+0000", 2024-05-15T12:55:27.690 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.3579999999999992e-06 2024-05-15T12:55:27.690 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.690 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.690 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:27.690 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.988632+0000", 2024-05-15T12:55:27.690 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6449999999999999e-06 2024-05-15T12:55:27.690 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.690 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.690 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.690 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.988683+0000", 2024-05-15T12:55:27.691 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0464000000000003e-05 2024-05-15T12:55:27.691 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.691 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.691 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:55:27.691 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.988693+0000", 2024-05-15T12:55:27.691 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.9189999999999994e-06 2024-05-15T12:55:27.691 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.691 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.691 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.691 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.892471+0000", 2024-05-15T12:55:27.691 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.90377877299999998 2024-05-15T12:55:27.691 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.691 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.691 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:55:27.691 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.892589+0000", 2024-05-15T12:55:27.691 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000117773 2024-05-15T12:55:27.692 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.692 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.692 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 9,11", 2024-05-15T12:55:27.692 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.892744+0000", 2024-05-15T12:55:27.692 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015463799999999999 2024-05-15T12:55:27.692 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.692 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.692 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:55:27.692 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.896370+0000", 2024-05-15T12:55:27.692 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.003626125 2024-05-15T12:55:27.692 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.692 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.692 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.692 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.900801+0000", 2024-05-15T12:55:27.693 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0044307870000000003 2024-05-15T12:55:27.693 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.693 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.693 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.693 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.904195+0000", 2024-05-15T12:55:27.693 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0033943240000000002 2024-05-15T12:55:27.693 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.693 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.693 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:55:27.693 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.904247+0000", 2024-05-15T12:55:27.693 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1678999999999998e-05 2024-05-15T12:55:27.693 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.693 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.693 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:55:27.694 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.904284+0000", 2024-05-15T12:55:27.694 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7268999999999998e-05 2024-05-15T12:55:27.694 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.694 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:27.694 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.694 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.694 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.694 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:48763 2.0 2:0448c495:::benchmark_data_smithi177_78104_object48762:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-05-15T12:55:27.694 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:39.988660+0000", 2024-05-15T12:55:27.694 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 407.61937321099998, 2024-05-15T12:55:27.694 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.91574130799999998, 2024-05-15T12:55:27.694 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:27.694 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:27.694 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:55:27.695 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:55:27.695 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:27.695 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 48763 2024-05-15T12:55:27.695 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.695 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:27.695 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.695 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:27.695 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.988660+0000", 2024-05-15T12:55:27.695 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.695 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.695 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.695 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:27.695 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.988660+0000", 2024-05-15T12:55:27.695 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.695 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.696 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.696 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:27.696 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.988661+0000", 2024-05-15T12:55:27.696 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0800000000000004e-07 2024-05-15T12:55:27.696 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.696 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.696 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:27.696 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.988819+0000", 2024-05-15T12:55:27.696 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015820299999999999 2024-05-15T12:55:27.696 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.696 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.696 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:27.696 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.988829+0000", 2024-05-15T12:55:27.696 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.8519999999999999e-06 2024-05-15T12:55:27.696 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.697 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.697 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:27.697 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.988837+0000", 2024-05-15T12:55:27.697 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.763e-06 2024-05-15T12:55:27.697 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.697 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.697 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.697 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.988873+0000", 2024-05-15T12:55:27.697 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.6047000000000002e-05 2024-05-15T12:55:27.697 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.697 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.697 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:55:27.697 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:39.988878+0000", 2024-05-15T12:55:27.697 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8049999999999997e-06 2024-05-15T12:55:27.697 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.698 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.698 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.698 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.893696+0000", 2024-05-15T12:55:27.698 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.90481815300000001 2024-05-15T12:55:27.698 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.698 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.699 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:55:27.699 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.893776+0000", 2024-05-15T12:55:27.699 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.051e-05 2024-05-15T12:55:27.699 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.699 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.699 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 9,11", 2024-05-15T12:55:27.699 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.893865+0000", 2024-05-15T12:55:27.699 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8936000000000004e-05 2024-05-15T12:55:27.699 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.699 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.699 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:55:27.699 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.897294+0000", 2024-05-15T12:55:27.699 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0034285959999999999 2024-05-15T12:55:27.699 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.700 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.700 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.700 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.900843+0000", 2024-05-15T12:55:27.700 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0035486350000000001 2024-05-15T12:55:27.700 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.700 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.700 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.700 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.904330+0000", 2024-05-15T12:55:27.700 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0034877969999999999 2024-05-15T12:55:27.700 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.700 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.700 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:55:27.700 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.904366+0000", 2024-05-15T12:55:27.700 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5850999999999999e-05 2024-05-15T12:55:27.700 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.701 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.701 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:55:27.701 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.904402+0000", 2024-05-15T12:55:27.701 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5451999999999999e-05 2024-05-15T12:55:27.701 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.701 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:27.701 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.701 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.701 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.701 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:48776 2.0 2:005123fb:::benchmark_data_smithi177_78104_object48775:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-05-15T12:55:27.701 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:40.006877+0000", 2024-05-15T12:55:27.701 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 407.60115667700001, 2024-05-15T12:55:27.701 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.89765720500000001, 2024-05-15T12:55:27.701 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:27.701 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:27.702 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:55:27.702 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:55:27.702 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:27.702 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 48776 2024-05-15T12:55:27.702 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.702 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:27.702 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.702 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:27.702 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.006877+0000", 2024-05-15T12:55:27.702 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.702 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.702 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.702 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:27.702 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.006877+0000", 2024-05-15T12:55:27.702 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.702 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.703 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.703 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:27.703 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.006878+0000", 2024-05-15T12:55:27.703 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3990000000000001e-06 2024-05-15T12:55:27.703 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.703 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.703 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:27.703 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.006962+0000", 2024-05-15T12:55:27.703 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.3300000000000005e-05 2024-05-15T12:55:27.703 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.703 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.703 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:27.703 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.006974+0000", 2024-05-15T12:55:27.703 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1950000000000001e-05 2024-05-15T12:55:27.703 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.704 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.704 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:27.704 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.006981+0000", 2024-05-15T12:55:27.704 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.5939999999999996e-06 2024-05-15T12:55:27.704 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.704 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.704 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.704 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.007033+0000", 2024-05-15T12:55:27.704 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2225000000000002e-05 2024-05-15T12:55:27.704 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.704 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.704 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:55:27.704 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.007040+0000", 2024-05-15T12:55:27.704 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0450000000000003e-06 2024-05-15T12:55:27.704 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.704 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.705 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.705 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.894719+0000", 2024-05-15T12:55:27.705 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.88767858499999996 2024-05-15T12:55:27.705 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.705 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.705 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:55:27.705 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.894791+0000", 2024-05-15T12:55:27.705 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2393999999999998e-05 2024-05-15T12:55:27.705 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.705 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.705 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 9,11", 2024-05-15T12:55:27.705 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.894874+0000", 2024-05-15T12:55:27.705 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2960999999999994e-05 2024-05-15T12:55:27.705 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.705 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.706 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:55:27.706 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.897300+0000", 2024-05-15T12:55:27.706 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0024253220000000002 2024-05-15T12:55:27.706 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.706 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.706 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.706 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.900883+0000", 2024-05-15T12:55:27.706 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.003583143 2024-05-15T12:55:27.706 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.706 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.706 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.706 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.904449+0000", 2024-05-15T12:55:27.706 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0035656899999999998 2024-05-15T12:55:27.707 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.707 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.707 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:55:27.707 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.904499+0000", 2024-05-15T12:55:27.707 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0141e-05 2024-05-15T12:55:27.707 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.707 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.707 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:55:27.707 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.904534+0000", 2024-05-15T12:55:27.707 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5456000000000001e-05 2024-05-15T12:55:27.707 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.707 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:27.707 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.707 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.708 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.708 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:48810 2.0 2:00ffcdc5:::benchmark_data_smithi177_78104_object48809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-05-15T12:55:27.708 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:40.053610+0000", 2024-05-15T12:55:27.708 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 407.55442328999999, 2024-05-15T12:55:27.708 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.85107060199999995, 2024-05-15T12:55:27.708 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:27.708 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:27.708 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:55:27.708 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:55:27.708 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:27.708 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 48810 2024-05-15T12:55:27.708 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.708 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:27.708 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.708 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:27.709 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.053610+0000", 2024-05-15T12:55:27.709 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.709 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.709 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.709 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:27.709 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.053610+0000", 2024-05-15T12:55:27.709 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.709 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.709 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.709 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:27.709 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.053611+0000", 2024-05-15T12:55:27.709 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0299999999999998e-07 2024-05-15T12:55:27.709 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.709 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.709 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:27.710 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.053662+0000", 2024-05-15T12:55:27.710 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0429000000000001e-05 2024-05-15T12:55:27.710 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.710 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.710 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:27.710 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.053674+0000", 2024-05-15T12:55:27.710 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2867999999999999e-05 2024-05-15T12:55:27.710 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.710 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.710 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:27.710 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.053683+0000", 2024-05-15T12:55:27.710 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.5909999999999996e-06 2024-05-15T12:55:27.710 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.710 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.710 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.711 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.053738+0000", 2024-05-15T12:55:27.711 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5081000000000003e-05 2024-05-15T12:55:27.711 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.711 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.711 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:55:27.711 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.053746+0000", 2024-05-15T12:55:27.711 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0220000000000004e-06 2024-05-15T12:55:27.711 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.711 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.711 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.711 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.895529+0000", 2024-05-15T12:55:27.711 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.84178270899999996 2024-05-15T12:55:27.711 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.711 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.711 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:55:27.712 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.895636+0000", 2024-05-15T12:55:27.712 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000107426 2024-05-15T12:55:27.712 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.712 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.712 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 9,11", 2024-05-15T12:55:27.712 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.895719+0000", 2024-05-15T12:55:27.712 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2785000000000001e-05 2024-05-15T12:55:27.712 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.712 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.712 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:55:27.712 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.899060+0000", 2024-05-15T12:55:27.712 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0033408050000000001 2024-05-15T12:55:27.712 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.712 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.712 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.713 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.903323+0000", 2024-05-15T12:55:27.713 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0042630610000000003 2024-05-15T12:55:27.713 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.713 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.713 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.713 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.904583+0000", 2024-05-15T12:55:27.713 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001260464 2024-05-15T12:55:27.713 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.713 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.713 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:55:27.713 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.904635+0000", 2024-05-15T12:55:27.713 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2082e-05 2024-05-15T12:55:27.713 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.713 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.713 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:55:27.713 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.904681+0000", 2024-05-15T12:55:27.714 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.5476000000000003e-05 2024-05-15T12:55:27.714 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.714 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:27.714 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.714 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.714 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.714 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:48821 2.0 2:099b45b0:::benchmark_data_smithi177_78104_object48820:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-05-15T12:55:27.714 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:48:40.285676+0000", 2024-05-15T12:55:27.714 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 407.322357969, 2024-05-15T12:55:27.714 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.61915024600000002, 2024-05-15T12:55:27.714 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:27.714 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:27.714 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:55:27.714 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:55:27.715 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:27.715 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 48821 2024-05-15T12:55:27.715 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.715 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:27.715 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.715 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:27.715 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.285676+0000", 2024-05-15T12:55:27.715 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.715 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.715 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.715 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:27.715 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.285676+0000", 2024-05-15T12:55:27.715 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.715 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.715 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.716 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:27.716 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.285677+0000", 2024-05-15T12:55:27.716 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1689999999999999e-06 2024-05-15T12:55:27.716 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.716 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.716 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:27.716 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.285765+0000", 2024-05-15T12:55:27.716 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7748000000000001e-05 2024-05-15T12:55:27.716 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.716 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.716 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:27.716 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.285777+0000", 2024-05-15T12:55:27.716 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2184000000000001e-05 2024-05-15T12:55:27.716 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.716 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.716 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:27.717 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.285785+0000", 2024-05-15T12:55:27.717 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9659999999999996e-06 2024-05-15T12:55:27.717 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.717 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.717 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.717 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.285842+0000", 2024-05-15T12:55:27.717 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6972999999999998e-05 2024-05-15T12:55:27.717 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.717 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.717 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:55:27.717 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.285850+0000", 2024-05-15T12:55:27.717 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2579999999999995e-06 2024-05-15T12:55:27.717 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.717 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.717 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.718 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.896400+0000", 2024-05-15T12:55:27.718 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.61055055700000005 2024-05-15T12:55:27.718 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.718 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.718 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:55:27.718 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.896485+0000", 2024-05-15T12:55:27.718 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.4437999999999994e-05 2024-05-15T12:55:27.718 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.718 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.718 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 9,11", 2024-05-15T12:55:27.718 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.896567+0000", 2024-05-15T12:55:27.718 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1724000000000006e-05 2024-05-15T12:55:27.718 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.718 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.719 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:55:27.719 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.899064+0000", 2024-05-15T12:55:27.719 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0024975309999999999 2024-05-15T12:55:27.719 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.719 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.719 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.719 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.903376+0000", 2024-05-15T12:55:27.719 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0043121469999999997 2024-05-15T12:55:27.719 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.719 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.719 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.719 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.904737+0000", 2024-05-15T12:55:27.719 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0013604929999999999 2024-05-15T12:55:27.719 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.719 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.720 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:55:27.720 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.904787+0000", 2024-05-15T12:55:27.720 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9707999999999999e-05 2024-05-15T12:55:27.720 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.720 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.720 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:55:27.720 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:48:40.904826+0000", 2024-05-15T12:55:27.720 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.9350000000000001e-05 2024-05-15T12:55:27.720 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.720 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:27.720 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.720 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.720 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.720 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:90788 2.0 2:09a2a14f:::benchmark_data_smithi177_78104_object90787:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:55:27.721 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:49:17.232266+0000", 2024-05-15T12:55:27.721 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 370.37576798200001, 2024-05-15T12:55:27.721 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1434281610000001, 2024-05-15T12:55:27.721 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:27.721 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T12:55:27.721 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:55:27.721 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:55:27.721 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:27.721 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 90788 2024-05-15T12:55:27.721 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.721 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:27.721 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.721 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:27.721 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.232266+0000", 2024-05-15T12:55:27.721 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.721 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.722 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.722 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:27.722 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.232266+0000", 2024-05-15T12:55:27.722 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.722 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.722 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.722 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:27.722 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.232267+0000", 2024-05-15T12:55:27.722 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.074e-06 2024-05-15T12:55:27.722 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.722 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.722 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:27.722 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.232321+0000", 2024-05-15T12:55:27.722 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.4721e-05 2024-05-15T12:55:27.722 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.723 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.723 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:27.723 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.232332+0000", 2024-05-15T12:55:27.723 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0648999999999999e-05 2024-05-15T12:55:27.723 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.723 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.723 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:27.723 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.232338+0000", 2024-05-15T12:55:27.723 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.626e-06 2024-05-15T12:55:27.723 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.723 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.723 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.723 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.236599+0000", 2024-05-15T12:55:27.723 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0042610699999999996 2024-05-15T12:55:27.724 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.724 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.724 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:55:27.724 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.236756+0000", 2024-05-15T12:55:27.724 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015688599999999999 2024-05-15T12:55:27.724 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.724 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.724 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:55:27.724 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.236969+0000", 2024-05-15T12:55:27.724 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00021367599999999999 2024-05-15T12:55:27.724 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.724 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.724 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.724 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.245963+0000", 2024-05-15T12:55:27.724 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0089941210000000008 2024-05-15T12:55:27.725 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.725 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.725 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:55:27.725 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.245974+0000", 2024-05-15T12:55:27.725 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0417e-05 2024-05-15T12:55:27.725 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.725 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.725 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.725 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.258040+0000", 2024-05-15T12:55:27.725 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.012066346 2024-05-15T12:55:27.725 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.725 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.725 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:55:27.725 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.258056+0000", 2024-05-15T12:55:27.725 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6274999999999999e-05 2024-05-15T12:55:27.725 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.726 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.726 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.726 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.854407+0000", 2024-05-15T12:55:27.726 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.59635021200000005 2024-05-15T12:55:27.726 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.726 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.726 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:55:27.726 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.854424+0000", 2024-05-15T12:55:27.726 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7231000000000002e-05 2024-05-15T12:55:27.726 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.726 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.726 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.726 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.375630+0000", 2024-05-15T12:55:27.726 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5212062669999999 2024-05-15T12:55:27.726 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.727 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.727 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:55:27.727 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.375694+0000", 2024-05-15T12:55:27.727 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3590000000000006e-05 2024-05-15T12:55:27.727 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.727 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:27.727 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.727 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.727 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.727 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:90796 2.0 2:0a122ee9:::benchmark_data_smithi177_78104_object90795:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:55:27.727 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:49:17.240737+0000", 2024-05-15T12:55:27.727 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 370.367296189, 2024-05-15T12:55:27.727 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1369414010000001, 2024-05-15T12:55:27.727 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:27.727 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T12:55:27.728 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:55:27.728 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:55:27.728 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:27.728 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 90796 2024-05-15T12:55:27.728 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.728 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:27.728 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.728 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:27.728 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.240737+0000", 2024-05-15T12:55:27.728 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.728 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.728 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.728 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:27.728 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.240737+0000", 2024-05-15T12:55:27.729 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.729 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.729 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.729 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:27.729 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.240739+0000", 2024-05-15T12:55:27.729 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.658e-06 2024-05-15T12:55:27.729 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.729 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.729 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:27.729 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.240832+0000", 2024-05-15T12:55:27.729 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.3258999999999996e-05 2024-05-15T12:55:27.729 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.729 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.729 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:27.730 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.240844+0000", 2024-05-15T12:55:27.730 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.201e-05 2024-05-15T12:55:27.730 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.730 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.730 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:27.730 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.240853+0000", 2024-05-15T12:55:27.730 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.4719999999999995e-06 2024-05-15T12:55:27.730 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.730 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.730 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.730 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.243408+0000", 2024-05-15T12:55:27.730 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0025555560000000001 2024-05-15T12:55:27.730 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.730 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.731 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:55:27.731 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.243526+0000", 2024-05-15T12:55:27.731 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00011812899999999999 2024-05-15T12:55:27.731 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.731 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.731 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:55:27.731 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.243649+0000", 2024-05-15T12:55:27.731 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000122399 2024-05-15T12:55:27.731 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.731 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.731 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.731 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.253803+0000", 2024-05-15T12:55:27.731 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.010154058000000001 2024-05-15T12:55:27.731 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.732 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.732 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:55:27.732 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.253822+0000", 2024-05-15T12:55:27.732 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.8700999999999999e-05 2024-05-15T12:55:27.732 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.732 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.732 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.732 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.258222+0000", 2024-05-15T12:55:27.732 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0044000430000000002 2024-05-15T12:55:27.732 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.732 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.732 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:55:27.732 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.258233+0000", 2024-05-15T12:55:27.732 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1766000000000001e-05 2024-05-15T12:55:27.733 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.733 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.733 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.733 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.854479+0000", 2024-05-15T12:55:27.733 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.59624525799999994 2024-05-15T12:55:27.733 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.733 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.733 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:55:27.733 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.854489+0000", 2024-05-15T12:55:27.733 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0495e-05 2024-05-15T12:55:27.733 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.733 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.733 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.734 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.377627+0000", 2024-05-15T12:55:27.734 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.523137433 2024-05-15T12:55:27.734 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.734 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.734 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:55:27.734 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.377679+0000", 2024-05-15T12:55:27.734 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2163999999999997e-05 2024-05-15T12:55:27.734 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.734 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:27.734 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.734 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.735 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.735 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:90855 2.0 2:00a11c06:::benchmark_data_smithi177_78104_object90854:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-05-15T12:55:27.735 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:49:17.346247+0000", 2024-05-15T12:55:27.735 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 370.26178695599998, 2024-05-15T12:55:27.735 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0421678860000001, 2024-05-15T12:55:27.735 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:27.735 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:27.735 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:55:27.735 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:55:27.735 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:27.735 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 90855 2024-05-15T12:55:27.735 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.735 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:27.735 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.736 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:27.736 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.346247+0000", 2024-05-15T12:55:27.736 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.736 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.736 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.736 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:27.736 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.346247+0000", 2024-05-15T12:55:27.736 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.736 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.736 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.736 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:27.736 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.346248+0000", 2024-05-15T12:55:27.736 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5120000000000001e-06 2024-05-15T12:55:27.736 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.737 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.737 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:27.737 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.346294+0000", 2024-05-15T12:55:27.737 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.5668999999999999e-05 2024-05-15T12:55:27.737 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.737 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.737 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:27.737 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.346309+0000", 2024-05-15T12:55:27.737 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.56e-05 2024-05-15T12:55:27.737 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.737 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.737 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:27.737 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.346317+0000", 2024-05-15T12:55:27.738 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8380000000000002e-06 2024-05-15T12:55:27.738 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.738 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.738 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.738 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.346374+0000", 2024-05-15T12:55:27.738 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6620999999999999e-05 2024-05-15T12:55:27.738 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.738 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.738 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:55:27.738 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.346383+0000", 2024-05-15T12:55:27.738 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.3049999999999998e-06 2024-05-15T12:55:27.738 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.738 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.739 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.739 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.854553+0000", 2024-05-15T12:55:27.739 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.50816969099999998 2024-05-15T12:55:27.739 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.739 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.739 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:55:27.739 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.854564+0000", 2024-05-15T12:55:27.739 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.102e-05 2024-05-15T12:55:27.739 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.739 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.739 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.739 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.377698+0000", 2024-05-15T12:55:27.740 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5231341970000001 2024-05-15T12:55:27.740 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.740 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.740 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:55:27.740 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.377795+0000", 2024-05-15T12:55:27.740 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.6576999999999994e-05 2024-05-15T12:55:27.740 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.740 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.740 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 3,9,10", 2024-05-15T12:55:27.740 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.377924+0000", 2024-05-15T12:55:27.740 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012965700000000001 2024-05-15T12:55:27.740 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.740 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.741 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:55:27.741 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.383361+0000", 2024-05-15T12:55:27.741 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0054363409999999999 2024-05-15T12:55:27.741 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.741 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.741 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.741 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.383422+0000", 2024-05-15T12:55:27.741 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1773999999999995e-05 2024-05-15T12:55:27.741 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.741 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.741 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.741 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.384704+0000", 2024-05-15T12:55:27.741 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001281623 2024-05-15T12:55:27.741 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.742 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.742 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.742 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.385119+0000", 2024-05-15T12:55:27.742 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00041488000000000001 2024-05-15T12:55:27.742 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.742 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.742 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:55:27.742 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.385190+0000", 2024-05-15T12:55:27.742 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.1460999999999999e-05 2024-05-15T12:55:27.742 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.742 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.742 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:55:27.743 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.388415+0000", 2024-05-15T12:55:27.743 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00322412 2024-05-15T12:55:27.743 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.743 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:27.743 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.743 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.743 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.743 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:90861 2.0 2:0771628c:::benchmark_data_smithi177_78104_object90860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-05-15T12:55:27.743 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:49:17.367962+0000", 2024-05-15T12:55:27.743 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 370.240071602, 2024-05-15T12:55:27.743 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0340376849999999, 2024-05-15T12:55:27.743 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:55:27.743 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:27.743 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:55:27.744 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:55:27.744 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:27.744 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 90861 2024-05-15T12:55:27.744 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.744 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:55:27.744 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.744 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:55:27.744 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.367962+0000", 2024-05-15T12:55:27.744 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.744 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.744 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.744 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:55:27.744 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.367962+0000", 2024-05-15T12:55:27.745 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:55:27.745 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.745 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.745 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:55:27.745 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.367964+0000", 2024-05-15T12:55:27.745 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2000000000000001e-06 2024-05-15T12:55:27.745 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.745 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.745 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:55:27.745 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.368038+0000", 2024-05-15T12:55:27.745 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4111999999999998e-05 2024-05-15T12:55:27.745 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.745 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.746 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:55:27.746 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.368078+0000", 2024-05-15T12:55:27.746 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.9721999999999997e-05 2024-05-15T12:55:27.746 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.746 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.746 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:55:27.746 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.368114+0000", 2024-05-15T12:55:27.746 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5848e-05 2024-05-15T12:55:27.746 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.746 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.746 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.746 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.368175+0000", 2024-05-15T12:55:27.746 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1483999999999996e-05 2024-05-15T12:55:27.747 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.747 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.747 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:55:27.747 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.368184+0000", 2024-05-15T12:55:27.747 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7450000000000001e-06 2024-05-15T12:55:27.747 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.747 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.747 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.747 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.854585+0000", 2024-05-15T12:55:27.747 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.48640091899999999 2024-05-15T12:55:27.747 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.747 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.747 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:55:27.747 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.854590+0000", 2024-05-15T12:55:27.748 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0719999999999999e-06 2024-05-15T12:55:27.748 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.748 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.748 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:55:27.748 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.380723+0000", 2024-05-15T12:55:27.748 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.526132496 2024-05-15T12:55:27.748 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.748 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.748 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:55:27.748 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.380864+0000", 2024-05-15T12:55:27.748 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014107900000000001 2024-05-15T12:55:27.748 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.748 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.748 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 3,9,10", 2024-05-15T12:55:27.749 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.381047+0000", 2024-05-15T12:55:27.749 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000183392 2024-05-15T12:55:27.749 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.749 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.749 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.749 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.383494+0000", 2024-05-15T12:55:27.749 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0024469930000000002 2024-05-15T12:55:27.749 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.749 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.749 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:55:27.749 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.383591+0000", 2024-05-15T12:55:27.749 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.6531000000000001e-05 2024-05-15T12:55:27.750 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.750 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.750 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.750 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.388480+0000", 2024-05-15T12:55:27.750 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0048892720000000001 2024-05-15T12:55:27.750 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.750 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.750 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:27.750 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.399142+0000", 2024-05-15T12:55:27.750 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.010662486000000001 2024-05-15T12:55:27.750 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.750 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.750 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:55:27.750 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.399183+0000", 2024-05-15T12:55:27.751 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0878999999999999e-05 2024-05-15T12:55:27.751 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:55:27.751 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:55:27.751 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:55:27.751 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.402000+0000", 2024-05-15T12:55:27.751 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0028164549999999998 2024-05-15T12:55:27.751 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.751 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:27.751 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.751 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:55:27.751 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:55:27.751 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T12:55:27.977 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:27 smithi120 ceph-mon[43961]: pgmap v745: 17 pgs: 17 active+clean; 5.8 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 823 op/s 2024-05-15T12:55:28.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:27 smithi177 ceph-mon[44303]: pgmap v745: 17 pgs: 17 active+clean; 5.8 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 823 op/s 2024-05-15T12:55:28.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:27 smithi027 ceph-mon[39795]: pgmap v745: 17 pgs: 17 active+clean; 5.8 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 823 op/s 2024-05-15T12:55:28.358 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-05-15T12:55:28.665 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:55:29.137 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:55:29.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:29.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:55:29.246 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:55:29.295 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:29.321 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:55:29.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:29.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:55:29.431 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:55:29.464 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:29.490 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:55:29.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:29.493 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:55:29.594 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:55:29.624 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:29.682 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:55:29.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:29.692 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:55:29.778 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:29 smithi120 ceph-mon[43961]: pgmap v746: 17 pgs: 17 active+clean; 5.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 824 op/s 2024-05-15T12:55:29.793 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:55:29.842 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:30.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:29 smithi177 ceph-mon[44303]: pgmap v746: 17 pgs: 17 active+clean; 5.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 824 op/s 2024-05-15T12:55:30.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:29 smithi027 ceph-mon[39795]: pgmap v746: 17 pgs: 17 active+clean; 5.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 824 op/s 2024-05-15T12:55:30.650 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T12:55:30.651 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T12:55:30.651 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.651 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24832.0:176610 3.9 3:9a6aec8a:::benchmark_data_smithi177_88104_object42988:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-05-15T12:55:30.651 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:55:30.650307+0000", 2024-05-15T12:55:30.651 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0070614010000000001, 2024-05-15T12:55:30.651 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0071018380000000001, 2024-05-15T12:55:30.651 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:30.651 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-05-15T12:55:30.651 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:30.651 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24832", 2024-05-15T12:55:30.651 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:55:30.651 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 176610 2024-05-15T12:55:30.652 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.652 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:30.652 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.652 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:30.652 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.650307+0000", 2024-05-15T12:55:30.652 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:30.652 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.652 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.652 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:30.652 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.650307+0000", 2024-05-15T12:55:30.652 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:30.652 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.652 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.652 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:30.652 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.650308+0000", 2024-05-15T12:55:30.653 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3179999999999999e-06 2024-05-15T12:55:30.653 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.653 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.653 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:30.653 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.650314+0000", 2024-05-15T12:55:30.653 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1750000000000002e-06 2024-05-15T12:55:30.653 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.653 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.653 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:30.653 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.650327+0000", 2024-05-15T12:55:30.653 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2301e-05 2024-05-15T12:55:30.653 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.653 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.654 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:30.654 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.650335+0000", 2024-05-15T12:55:30.654 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9780000000000005e-06 2024-05-15T12:55:30.654 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.654 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.654 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:30.654 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.650382+0000", 2024-05-15T12:55:30.654 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7425000000000001e-05 2024-05-15T12:55:30.654 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.654 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.654 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:30.654 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.650476+0000", 2024-05-15T12:55:30.655 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3788000000000004e-05 2024-05-15T12:55:30.655 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.655 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.655 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7,9", 2024-05-15T12:55:30.655 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.650557+0000", 2024-05-15T12:55:30.655 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1088999999999996e-05 2024-05-15T12:55:30.655 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.655 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.655 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:55:30.655 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.654396+0000", 2024-05-15T12:55:30.655 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00383922 2024-05-15T12:55:30.655 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.655 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.655 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:30.656 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.654548+0000", 2024-05-15T12:55:30.656 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000151936 2024-05-15T12:55:30.656 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:30.656 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:30.656 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:30.656 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.656 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.656 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24832.0:176612 3.f e271/219 3:f3172f36:::benchmark_data_smithi177_88104_object42990:head v 271'11000, mlcod=271'10996)", 2024-05-15T12:55:30.656 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:55:30.654360+0000", 2024-05-15T12:55:30.656 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0030080670000000001, 2024-05-15T12:55:30.656 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003140777, 2024-05-15T12:55:30.656 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:30.656 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T12:55:30.657 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:30.657 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.657 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:30.657 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.654360+0000", 2024-05-15T12:55:30.657 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:30.657 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.657 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.657 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:30.657 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.654360+0000", 2024-05-15T12:55:30.657 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:30.657 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.657 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.657 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:30.657 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.654361+0000", 2024-05-15T12:55:30.657 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2399999999999996e-07 2024-05-15T12:55:30.658 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.658 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.658 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:30.658 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.654369+0000", 2024-05-15T12:55:30.658 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.926e-06 2024-05-15T12:55:30.658 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.658 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.658 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:30.658 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.654381+0000", 2024-05-15T12:55:30.658 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1622e-05 2024-05-15T12:55:30.658 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.658 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.658 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:30.659 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.654389+0000", 2024-05-15T12:55:30.659 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1710000000000002e-06 2024-05-15T12:55:30.659 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.659 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.659 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:30.659 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.654447+0000", 2024-05-15T12:55:30.659 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7751999999999999e-05 2024-05-15T12:55:30.659 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.659 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.659 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:30.659 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.654470+0000", 2024-05-15T12:55:30.659 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3736000000000001e-05 2024-05-15T12:55:30.659 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:30.659 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:30.659 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:30.660 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.660 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.660 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24832.0:176615 3.9 3:98c054a8:::benchmark_data_smithi177_88104_object42993:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-05-15T12:55:30.660 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:55:30.655417+0000", 2024-05-15T12:55:30.660 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0019516469999999999, 2024-05-15T12:55:30.660 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002145409, 2024-05-15T12:55:30.660 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:30.660 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-05-15T12:55:30.660 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:30.660 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24832", 2024-05-15T12:55:30.660 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:55:30.660 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 176615 2024-05-15T12:55:30.660 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.660 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:30.660 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.660 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:30.661 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.655417+0000", 2024-05-15T12:55:30.661 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:30.661 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.661 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.661 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:30.661 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.655417+0000", 2024-05-15T12:55:30.661 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:30.661 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.661 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.661 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:30.661 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.655418+0000", 2024-05-15T12:55:30.661 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2249999999999999e-06 2024-05-15T12:55:30.661 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.661 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.661 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:30.662 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.655423+0000", 2024-05-15T12:55:30.662 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5829999999999997e-06 2024-05-15T12:55:30.662 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.662 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.662 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:30.662 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.655437+0000", 2024-05-15T12:55:30.662 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3308e-05 2024-05-15T12:55:30.662 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.662 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.662 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:30.662 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.655444+0000", 2024-05-15T12:55:30.662 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6399999999999997e-06 2024-05-15T12:55:30.662 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.662 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.663 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:30.663 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.655496+0000", 2024-05-15T12:55:30.663 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1434999999999998e-05 2024-05-15T12:55:30.663 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.663 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.663 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:30.663 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.655639+0000", 2024-05-15T12:55:30.663 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000142764 2024-05-15T12:55:30.663 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:30.663 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:30.663 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7,9", 2024-05-15T12:55:30.663 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:30.655761+0000", 2024-05-15T12:55:30.663 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000122772 2024-05-15T12:55:30.664 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:30.664 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:30.664 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:30.664 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:30.664 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-05-15T12:55:30.664 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 3 2024-05-15T12:55:30.664 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T12:55:31.173 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T12:55:31.369 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-05-15T12:55:31.691 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:55:32.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:31 smithi177 ceph-mon[44303]: pgmap v747: 17 pgs: 17 active+clean; 5.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 816 op/s 2024-05-15T12:55:32.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:31 smithi027 ceph-mon[39795]: pgmap v747: 17 pgs: 17 active+clean; 5.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 816 op/s 2024-05-15T12:55:32.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:31 smithi120 ceph-mon[43961]: pgmap v747: 17 pgs: 17 active+clean; 5.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 816 op/s 2024-05-15T12:55:33.409 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:55:33.419 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:55:33.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:33.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:55:33.471 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:55:33.528 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:55:33.538 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T12:55:33.539 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T12:55:33.539 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-05-15T12:55:33.539 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-05-15T12:55:33.539 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T12:55:33.560 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:33.581 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:55:33.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:33.584 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:55:33.685 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:55:33.743 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:33.800 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:55:33.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:33.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:55:33.912 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:55:33.943 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:34.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:34 smithi177 ceph-mon[44303]: pgmap v748: 17 pgs: 17 active+clean; 5.5 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 878 op/s 2024-05-15T12:55:34.290 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-05-15T12:55:34.296 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:55:34.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:34.299 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:55:34.400 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:55:34.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:34 smithi027 ceph-mon[39795]: pgmap v748: 17 pgs: 17 active+clean; 5.5 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 878 op/s 2024-05-15T12:55:34.446 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:34.500 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:55:34.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:34.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:55:34.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:34 smithi120 ceph-mon[43961]: pgmap v748: 17 pgs: 17 active+clean; 5.5 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 878 op/s 2024-05-15T12:55:34.602 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:55:34.606 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:55:34.656 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:34.880 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:55:34.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:34.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:55:34.989 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:55:35.079 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:35.130 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:55:35.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:35.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:55:35.236 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:55:35.272 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:35.380 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:55:35.477 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:55:35.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:35.480 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:55:35.581 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:55:35.663 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:35.792 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:35 smithi177 ceph-mon[44303]: pgmap v749: 17 pgs: 17 active+clean; 5.4 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 842 op/s 2024-05-15T12:55:36.208 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:35 smithi027 ceph-mon[39795]: pgmap v749: 17 pgs: 17 active+clean; 5.4 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 842 op/s 2024-05-15T12:55:36.241 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:55:36.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:36.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:55:36.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:35 smithi120 ceph-mon[43961]: pgmap v749: 17 pgs: 17 active+clean; 5.4 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 842 op/s 2024-05-15T12:55:36.347 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:55:36.383 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:36.974 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:55:36.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:36.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:55:37.058 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T12:55:37.058 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-05-15T12:55:37.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-05-15T12:55:37.058 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T12:55:37.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.058 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125021 2.b 2:d6c97047:::benchmark_data_smithi177_78104_object17888:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:55:37.058 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.708938+0000", 2024-05-15T12:55:37.059 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 319.35589931599998, 2024-05-15T12:55:37.059 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21075210699999999, 2024-05-15T12:55:37.059 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:37.059 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:37.059 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:37.059 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:55:37.059 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:37.059 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125021 2024-05-15T12:55:37.059 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.059 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:37.059 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:37.060 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.708938+0000", 2024-05-15T12:55:37.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:37.060 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.708938+0000", 2024-05-15T12:55:37.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:37.060 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.708939+0000", 2024-05-15T12:55:37.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.31e-06 2024-05-15T12:55:37.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:37.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.708944+0000", 2024-05-15T12:55:37.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2020000000000003e-06 2024-05-15T12:55:37.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:37.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.708956+0000", 2024-05-15T12:55:37.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1430000000000001e-05 2024-05-15T12:55:37.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:37.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.708963+0000", 2024-05-15T12:55:37.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5639999999999999e-06 2024-05-15T12:55:37.062 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.062 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:37.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.709010+0000", 2024-05-15T12:55:37.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6993000000000002e-05 2024-05-15T12:55:37.062 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.062 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:37.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.709098+0000", 2024-05-15T12:55:37.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7961999999999996e-05 2024-05-15T12:55:37.062 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.063 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:55:37.063 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.709168+0000", 2024-05-15T12:55:37.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9677000000000004e-05 2024-05-15T12:55:37.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.063 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:55:37.063 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.712329+0000", 2024-05-15T12:55:37.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003161245 2024-05-15T12:55:37.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.063 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.063 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.712501+0000", 2024-05-15T12:55:37.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017174700000000001 2024-05-15T12:55:37.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.919583+0000", 2024-05-15T12:55:37.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20708225 2024-05-15T12:55:37.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:37.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.919642+0000", 2024-05-15T12:55:37.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8273000000000003e-05 2024-05-15T12:55:37.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:37.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.919690+0000", 2024-05-15T12:55:37.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8454000000000001e-05 2024-05-15T12:55:37.065 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.065 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:37.065 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.065 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125026 2.b 2:d5f79b62:::benchmark_data_smithi177_78104_object17893:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:55:37.065 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.718595+0000", 2024-05-15T12:55:37.065 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 319.34624231700002, 2024-05-15T12:55:37.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20594219999999999, 2024-05-15T12:55:37.065 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:37.065 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:37.066 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:37.066 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:55:37.066 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:37.066 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125026 2024-05-15T12:55:37.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.066 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:37.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:37.066 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.718595+0000", 2024-05-15T12:55:37.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:37.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.718595+0000", 2024-05-15T12:55:37.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.067 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:37.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.718596+0000", 2024-05-15T12:55:37.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1170000000000001e-06 2024-05-15T12:55:37.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.067 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:37.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.718601+0000", 2024-05-15T12:55:37.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0089999999999998e-06 2024-05-15T12:55:37.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:37.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.718611+0000", 2024-05-15T12:55:37.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6719999999999999e-06 2024-05-15T12:55:37.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:37.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.718617+0000", 2024-05-15T12:55:37.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9989999999999997e-06 2024-05-15T12:55:37.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:37.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.718668+0000", 2024-05-15T12:55:37.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0936000000000002e-05 2024-05-15T12:55:37.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:37.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.718800+0000", 2024-05-15T12:55:37.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013213799999999999 2024-05-15T12:55:37.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:55:37.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.718911+0000", 2024-05-15T12:55:37.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011174499999999999 2024-05-15T12:55:37.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:55:37.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.722009+0000", 2024-05-15T12:55:37.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003097739 2024-05-15T12:55:37.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.722395+0000", 2024-05-15T12:55:37.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00038567000000000001 2024-05-15T12:55:37.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.924476+0000", 2024-05-15T12:55:37.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.202080975 2024-05-15T12:55:37.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:37.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.924509+0000", 2024-05-15T12:55:37.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3284999999999997e-05 2024-05-15T12:55:37.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:37.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.924537+0000", 2024-05-15T12:55:37.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7915e-05 2024-05-15T12:55:37.071 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.071 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:37.072 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.072 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125038 2.b 2:d70abfc3:::benchmark_data_smithi177_78104_object17905:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:55:37.072 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.727604+0000", 2024-05-15T12:55:37.072 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 319.33723291199999, 2024-05-15T12:55:37.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.201815784, 2024-05-15T12:55:37.072 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:37.072 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:37.072 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:37.072 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:55:37.072 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:37.072 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125038 2024-05-15T12:55:37.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.072 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:37.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:37.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.727604+0000", 2024-05-15T12:55:37.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:37.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.727604+0000", 2024-05-15T12:55:37.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:37.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.727605+0000", 2024-05-15T12:55:37.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2e-07 2024-05-15T12:55:37.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:37.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.727608+0000", 2024-05-15T12:55:37.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1310000000000002e-06 2024-05-15T12:55:37.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:37.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.727614+0000", 2024-05-15T12:55:37.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.353e-06 2024-05-15T12:55:37.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:37.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.727620+0000", 2024-05-15T12:55:37.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4790000000000004e-06 2024-05-15T12:55:37.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:37.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.727667+0000", 2024-05-15T12:55:37.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7219000000000003e-05 2024-05-15T12:55:37.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:37.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.727743+0000", 2024-05-15T12:55:37.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5912000000000001e-05 2024-05-15T12:55:37.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:55:37.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.727818+0000", 2024-05-15T12:55:37.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5493000000000004e-05 2024-05-15T12:55:37.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.731339+0000", 2024-05-15T12:55:37.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035210580000000001 2024-05-15T12:55:37.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:55:37.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.731402+0000", 2024-05-15T12:55:37.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2527999999999995e-05 2024-05-15T12:55:37.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.929301+0000", 2024-05-15T12:55:37.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.197899305 2024-05-15T12:55:37.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:37.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.929360+0000", 2024-05-15T12:55:37.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8541000000000001e-05 2024-05-15T12:55:37.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:37.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.929420+0000", 2024-05-15T12:55:37.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0244999999999997e-05 2024-05-15T12:55:37.077 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.077 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:37.078 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.078 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125040 2.b 2:d223540b:::benchmark_data_smithi177_78104_object17907:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:55:37.078 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.733348+0000", 2024-05-15T12:55:37.078 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 319.33148940400002, 2024-05-15T12:55:37.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19620225999999999, 2024-05-15T12:55:37.078 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:37.078 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:37.078 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:37.078 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:55:37.078 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:37.078 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125040 2024-05-15T12:55:37.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.079 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:37.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:37.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.733348+0000", 2024-05-15T12:55:37.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:37.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.733348+0000", 2024-05-15T12:55:37.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:37.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.733349+0000", 2024-05-15T12:55:37.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1680000000000001e-06 2024-05-15T12:55:37.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:37.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.733355+0000", 2024-05-15T12:55:37.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7599999999999999e-06 2024-05-15T12:55:37.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:37.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.733365+0000", 2024-05-15T12:55:37.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0562e-05 2024-05-15T12:55:37.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:37.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.733381+0000", 2024-05-15T12:55:37.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5469999999999999e-05 2024-05-15T12:55:37.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:37.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.733440+0000", 2024-05-15T12:55:37.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9667000000000003e-05 2024-05-15T12:55:37.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:37.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.733570+0000", 2024-05-15T12:55:37.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000129794 2024-05-15T12:55:37.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:55:37.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.733689+0000", 2024-05-15T12:55:37.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011836999999999999 2024-05-15T12:55:37.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:55:37.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.737518+0000", 2024-05-15T12:55:37.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003829598 2024-05-15T12:55:37.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.737968+0000", 2024-05-15T12:55:37.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00045025700000000002 2024-05-15T12:55:37.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.929466+0000", 2024-05-15T12:55:37.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19149776499999999 2024-05-15T12:55:37.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:37.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.929508+0000", 2024-05-15T12:55:37.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1690999999999998e-05 2024-05-15T12:55:37.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:37.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.929550+0000", 2024-05-15T12:55:37.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2157999999999999e-05 2024-05-15T12:55:37.084 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.084 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:37.084 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.085 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125083 2.b 2:db4ca6e9:::benchmark_data_smithi177_78104_object17950:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:55:37.085 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.785463+0000", 2024-05-15T12:55:37.085 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 319.27937423399999, 2024-05-15T12:55:37.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.15291454400000001, 2024-05-15T12:55:37.085 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:37.085 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:37.085 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:37.085 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:55:37.085 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:37.085 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125083 2024-05-15T12:55:37.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.085 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:37.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:37.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.785463+0000", 2024-05-15T12:55:37.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:37.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.785463+0000", 2024-05-15T12:55:37.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:37.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.785464+0000", 2024-05-15T12:55:37.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.488e-06 2024-05-15T12:55:37.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:37.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.785470+0000", 2024-05-15T12:55:37.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8340000000000004e-06 2024-05-15T12:55:37.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.785485+0000", 2024-05-15T12:55:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5099000000000001e-05 2024-05-15T12:55:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.785494+0000", 2024-05-15T12:55:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3189999999999995e-06 2024-05-15T12:55:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.785549+0000", 2024-05-15T12:55:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5670000000000001e-05 2024-05-15T12:55:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.785738+0000", 2024-05-15T12:55:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000188677 2024-05-15T12:55:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:55:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.785868+0000", 2024-05-15T12:55:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012986700000000001 2024-05-15T12:55:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.789091+0000", 2024-05-15T12:55:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0032231680000000002 2024-05-15T12:55:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:55:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.789135+0000", 2024-05-15T12:55:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3782999999999997e-05 2024-05-15T12:55:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.938308+0000", 2024-05-15T12:55:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14917336 2024-05-15T12:55:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.938349+0000", 2024-05-15T12:55:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0386999999999998e-05 2024-05-15T12:55:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.938378+0000", 2024-05-15T12:55:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8892000000000001e-05 2024-05-15T12:55:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.094 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125086 2.b 2:d1b0dce9:::benchmark_data_smithi177_78104_object17953:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:55:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.792066+0000", 2024-05-15T12:55:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 319.272771159, 2024-05-15T12:55:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.146447893, 2024-05-15T12:55:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:55:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125086 2024-05-15T12:55:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:37.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.792066+0000", 2024-05-15T12:55:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.792066+0000", 2024-05-15T12:55:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.792067+0000", 2024-05-15T12:55:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1689999999999999e-06 2024-05-15T12:55:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.792072+0000", 2024-05-15T12:55:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2599999999999996e-06 2024-05-15T12:55:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.792083+0000", 2024-05-15T12:55:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0508e-05 2024-05-15T12:55:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.792089+0000", 2024-05-15T12:55:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6710000000000004e-06 2024-05-15T12:55:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.792134+0000", 2024-05-15T12:55:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5732000000000002e-05 2024-05-15T12:55:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.792252+0000", 2024-05-15T12:55:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117772 2024-05-15T12:55:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:55:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.792367+0000", 2024-05-15T12:55:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114957 2024-05-15T12:55:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:55:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.795925+0000", 2024-05-15T12:55:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035583559999999999 2024-05-15T12:55:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.797215+0000", 2024-05-15T12:55:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012897679999999999 2024-05-15T12:55:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.938419+0000", 2024-05-15T12:55:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14120342899999999 2024-05-15T12:55:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.938471+0000", 2024-05-15T12:55:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2422000000000001e-05 2024-05-15T12:55:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.938514+0000", 2024-05-15T12:55:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2849000000000003e-05 2024-05-15T12:55:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125099 2.b 2:da4fcb81:::benchmark_data_smithi177_78104_object17966:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:55:37.100 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.803901+0000", 2024-05-15T12:55:37.101 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 319.26093616499998, 2024-05-15T12:55:37.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.138301853, 2024-05-15T12:55:37.101 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:37.101 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:37.101 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:37.101 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:55:37.101 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:37.101 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125099 2024-05-15T12:55:37.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.101 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.803901+0000", 2024-05-15T12:55:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.803901+0000", 2024-05-15T12:55:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:37.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.803902+0000", 2024-05-15T12:55:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9400000000000004e-07 2024-05-15T12:55:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.803907+0000", 2024-05-15T12:55:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.186e-06 2024-05-15T12:55:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.803920+0000", 2024-05-15T12:55:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3227000000000001e-05 2024-05-15T12:55:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:37.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.803927+0000", 2024-05-15T12:55:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2400000000000004e-06 2024-05-15T12:55:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.803977+0000", 2024-05-15T12:55:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0000000000000002e-05 2024-05-15T12:55:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.804106+0000", 2024-05-15T12:55:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012985699999999999 2024-05-15T12:55:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:55:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.804229+0000", 2024-05-15T12:55:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012295599999999999 2024-05-15T12:55:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:55:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.807117+0000", 2024-05-15T12:55:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0028879969999999998 2024-05-15T12:55:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.807404+0000", 2024-05-15T12:55:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028621799999999998 2024-05-15T12:55:37.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.942098+0000", 2024-05-15T12:55:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.134694803 2024-05-15T12:55:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.942156+0000", 2024-05-15T12:55:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7138000000000001e-05 2024-05-15T12:55:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:37.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.942203+0000", 2024-05-15T12:55:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7336999999999998e-05 2024-05-15T12:55:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125102 2.b 2:da660b72:::benchmark_data_smithi177_78104_object17969:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:55:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.809757+0000", 2024-05-15T12:55:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 319.25508055900002, 2024-05-15T12:55:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13260296999999999, 2024-05-15T12:55:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:37.107 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:55:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125102 2024-05-15T12:55:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809757+0000", 2024-05-15T12:55:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809757+0000", 2024-05-15T12:55:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809757+0000", 2024-05-15T12:55:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6800000000000001e-07 2024-05-15T12:55:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809760+0000", 2024-05-15T12:55:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.847e-06 2024-05-15T12:55:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:37.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809766+0000", 2024-05-15T12:55:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1310000000000003e-06 2024-05-15T12:55:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809771+0000", 2024-05-15T12:55:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8729999999999998e-06 2024-05-15T12:55:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809813+0000", 2024-05-15T12:55:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2419000000000002e-05 2024-05-15T12:55:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:37.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809921+0000", 2024-05-15T12:55:37.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010755300000000001 2024-05-15T12:55:37.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:55:37.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.810010+0000", 2024-05-15T12:55:37.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8954000000000002e-05 2024-05-15T12:55:37.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.816996+0000", 2024-05-15T12:55:37.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0069862789999999998 2024-05-15T12:55:37.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:55:37.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.817033+0000", 2024-05-15T12:55:37.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6776999999999997e-05 2024-05-15T12:55:37.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.942257+0000", 2024-05-15T12:55:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.125224213 2024-05-15T12:55:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.942313+0000", 2024-05-15T12:55:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6221000000000003e-05 2024-05-15T12:55:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.942360+0000", 2024-05-15T12:55:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6235e-05 2024-05-15T12:55:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.113 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125103 2.b 2:d6ba212d:::benchmark_data_smithi177_78104_object17970:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:55:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.809893+0000", 2024-05-15T12:55:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 319.25494397199998, 2024-05-15T12:55:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.132601633, 2024-05-15T12:55:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:55:37.114 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125103 2024-05-15T12:55:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809893+0000", 2024-05-15T12:55:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809893+0000", 2024-05-15T12:55:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809894+0000", 2024-05-15T12:55:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8700000000000004e-07 2024-05-15T12:55:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809898+0000", 2024-05-15T12:55:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2699999999999998e-06 2024-05-15T12:55:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809912+0000", 2024-05-15T12:55:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3572e-05 2024-05-15T12:55:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:37.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.809919+0000", 2024-05-15T12:55:37.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.221e-06 2024-05-15T12:55:37.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:37.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.813691+0000", 2024-05-15T12:55:37.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037719440000000002 2024-05-15T12:55:37.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:37.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.813839+0000", 2024-05-15T12:55:37.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000148369 2024-05-15T12:55:37.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:55:37.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.813913+0000", 2024-05-15T12:55:37.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3009999999999994e-05 2024-05-15T12:55:37.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:55:37.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.817441+0000", 2024-05-15T12:55:37.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035280620000000002 2024-05-15T12:55:37.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.822286+0000", 2024-05-15T12:55:37.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0048452429999999999 2024-05-15T12:55:37.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.942407+0000", 2024-05-15T12:55:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.120121518 2024-05-15T12:55:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.942457+0000", 2024-05-15T12:55:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9818999999999999e-05 2024-05-15T12:55:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.942495+0000", 2024-05-15T12:55:37.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7718e-05 2024-05-15T12:55:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125109 2.b 2:dc68fa46:::benchmark_data_smithi177_78104_object17976:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:55:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.818774+0000", 2024-05-15T12:55:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 319.246063252, 2024-05-15T12:55:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.12385244300000001, 2024-05-15T12:55:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:55:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:37.120 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125109 2024-05-15T12:55:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.818774+0000", 2024-05-15T12:55:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.818774+0000", 2024-05-15T12:55:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:37.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.818775+0000", 2024-05-15T12:55:37.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4219999999999999e-06 2024-05-15T12:55:37.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:37.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.818781+0000", 2024-05-15T12:55:37.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.468e-06 2024-05-15T12:55:37.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:37.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.818792+0000", 2024-05-15T12:55:37.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1504e-05 2024-05-15T12:55:37.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:37.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.818800+0000", 2024-05-15T12:55:37.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.695e-06 2024-05-15T12:55:37.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.818867+0000", 2024-05-15T12:55:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6550999999999994e-05 2024-05-15T12:55:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.818990+0000", 2024-05-15T12:55:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012320199999999999 2024-05-15T12:55:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:55:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.819107+0000", 2024-05-15T12:55:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000116922 2024-05-15T12:55:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:55:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.823726+0000", 2024-05-15T12:55:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0046193049999999998 2024-05-15T12:55:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.824529+0000", 2024-05-15T12:55:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00080272500000000001 2024-05-15T12:55:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.942538+0000", 2024-05-15T12:55:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.118008848 2024-05-15T12:55:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.942593+0000", 2024-05-15T12:55:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5300000000000002e-05 2024-05-15T12:55:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.942626+0000", 2024-05-15T12:55:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3501000000000003e-05 2024-05-15T12:55:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.126 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:125120 2.b 2:dccc152d:::benchmark_data_smithi177_78104_object17987:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-05-15T12:55:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:17.833738+0000", 2024-05-15T12:55:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 319.23109969299998, 2024-05-15T12:55:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11326000999999999, 2024-05-15T12:55:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:55:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125120 2024-05-15T12:55:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.833738+0000", 2024-05-15T12:55:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:37.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.833738+0000", 2024-05-15T12:55:37.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:37.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.833739+0000", 2024-05-15T12:55:37.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2109999999999999e-06 2024-05-15T12:55:37.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:37.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.833743+0000", 2024-05-15T12:55:37.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.262e-06 2024-05-15T12:55:37.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:37.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.833753+0000", 2024-05-15T12:55:37.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0013000000000001e-05 2024-05-15T12:55:37.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:37.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.833758+0000", 2024-05-15T12:55:37.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2480000000000004e-06 2024-05-15T12:55:37.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:37.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.833812+0000", 2024-05-15T12:55:37.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4203999999999998e-05 2024-05-15T12:55:37.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:37.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.833936+0000", 2024-05-15T12:55:37.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012344799999999999 2024-05-15T12:55:37.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:55:37.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.834057+0000", 2024-05-15T12:55:37.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120771 2024-05-15T12:55:37.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.838256+0000", 2024-05-15T12:55:37.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0041994550000000004 2024-05-15T12:55:37.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:55:37.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.838594+0000", 2024-05-15T12:55:37.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000337812 2024-05-15T12:55:37.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.946914+0000", 2024-05-15T12:55:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.108320473 2024-05-15T12:55:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.946963+0000", 2024-05-15T12:55:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8916999999999999e-05 2024-05-15T12:55:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:17.946998+0000", 2024-05-15T12:55:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4196000000000003e-05 2024-05-15T12:55:37.132 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:180380 2.b 2:db605072:::benchmark_data_smithi177_78104_object73247:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-05-15T12:55:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:51:42.456884+0000", 2024-05-15T12:55:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 234.60795273900001, 2024-05-15T12:55:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21694975, 2024-05-15T12:55:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:55:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:37.133 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 180380 2024-05-15T12:55:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.456884+0000", 2024-05-15T12:55:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.456884+0000", 2024-05-15T12:55:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.456885+0000", 2024-05-15T12:55:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9100000000000004e-07 2024-05-15T12:55:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.456887+0000", 2024-05-15T12:55:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2960000000000001e-06 2024-05-15T12:55:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.456893+0000", 2024-05-15T12:55:37.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1639999999999998e-06 2024-05-15T12:55:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.456899+0000", 2024-05-15T12:55:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1390000000000003e-06 2024-05-15T12:55:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.456960+0000", 2024-05-15T12:55:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1657999999999998e-05 2024-05-15T12:55:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.457101+0000", 2024-05-15T12:55:37.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000140304 2024-05-15T12:55:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:55:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.457177+0000", 2024-05-15T12:55:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6148000000000003e-05 2024-05-15T12:55:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.462026+0000", 2024-05-15T12:55:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0048497949999999996 2024-05-15T12:55:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:55:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.466659+0000", 2024-05-15T12:55:37.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0046324010000000004 2024-05-15T12:55:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.673761+0000", 2024-05-15T12:55:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20710241099999999 2024-05-15T12:55:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.673797+0000", 2024-05-15T12:55:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5936000000000003e-05 2024-05-15T12:55:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.673834+0000", 2024-05-15T12:55:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7007e-05 2024-05-15T12:55:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:180400 2.b 2:dabf837e:::benchmark_data_smithi177_78104_object73267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-05-15T12:55:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:51:42.483556+0000", 2024-05-15T12:55:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 234.58128168100001, 2024-05-15T12:55:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.193595298, 2024-05-15T12:55:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:37.139 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:55:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 180400 2024-05-15T12:55:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.483556+0000", 2024-05-15T12:55:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.483556+0000", 2024-05-15T12:55:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.483557+0000", 2024-05-15T12:55:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.221e-06 2024-05-15T12:55:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.483562+0000", 2024-05-15T12:55:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4639999999999997e-06 2024-05-15T12:55:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.483584+0000", 2024-05-15T12:55:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2218e-05 2024-05-15T12:55:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.483592+0000", 2024-05-15T12:55:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5320000000000001e-06 2024-05-15T12:55:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.483645+0000", 2024-05-15T12:55:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3507000000000001e-05 2024-05-15T12:55:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.483770+0000", 2024-05-15T12:55:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000124268 2024-05-15T12:55:37.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:55:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.483888+0000", 2024-05-15T12:55:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011801199999999999 2024-05-15T12:55:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:55:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.488970+0000", 2024-05-15T12:55:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0050817659999999997 2024-05-15T12:55:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.495691+0000", 2024-05-15T12:55:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0067215560000000001 2024-05-15T12:55:37.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.677040+0000", 2024-05-15T12:55:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18134873600000001 2024-05-15T12:55:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.677106+0000", 2024-05-15T12:55:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5786999999999999e-05 2024-05-15T12:55:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.677151+0000", 2024-05-15T12:55:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5231000000000001e-05 2024-05-15T12:55:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.144 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:180428 2.b 2:d4cfe823:::benchmark_data_smithi177_78104_object73295:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-05-15T12:55:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:51:42.517245+0000", 2024-05-15T12:55:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 234.547591756, 2024-05-15T12:55:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16003568900000001, 2024-05-15T12:55:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:55:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 180428 2024-05-15T12:55:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:37.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.517245+0000", 2024-05-15T12:55:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.517245+0000", 2024-05-15T12:55:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.517246+0000", 2024-05-15T12:55:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.015e-06 2024-05-15T12:55:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.517252+0000", 2024-05-15T12:55:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1959999999999999e-06 2024-05-15T12:55:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.517263+0000", 2024-05-15T12:55:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1228e-05 2024-05-15T12:55:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.517271+0000", 2024-05-15T12:55:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2239999999999995e-06 2024-05-15T12:55:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.517332+0000", 2024-05-15T12:55:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1099000000000006e-05 2024-05-15T12:55:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.517450+0000", 2024-05-15T12:55:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117315 2024-05-15T12:55:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:55:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.517562+0000", 2024-05-15T12:55:37.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000112915 2024-05-15T12:55:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:55:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.521408+0000", 2024-05-15T12:55:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0038454740000000002 2024-05-15T12:55:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.526550+0000", 2024-05-15T12:55:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0051420249999999997 2024-05-15T12:55:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.677201+0000", 2024-05-15T12:55:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.15065061199999999 2024-05-15T12:55:37.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.677236+0000", 2024-05-15T12:55:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5207999999999999e-05 2024-05-15T12:55:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.677281+0000", 2024-05-15T12:55:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5377999999999998e-05 2024-05-15T12:55:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.150 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:180435 2.b 2:d62c3e2d:::benchmark_data_smithi177_78104_object73302:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-05-15T12:55:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:51:42.526543+0000", 2024-05-15T12:55:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 234.538294366, 2024-05-15T12:55:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.15591202400000001, 2024-05-15T12:55:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:55:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 180435 2024-05-15T12:55:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:37.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.526543+0000", 2024-05-15T12:55:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.526543+0000", 2024-05-15T12:55:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.526544+0000", 2024-05-15T12:55:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.226e-06 2024-05-15T12:55:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.526550+0000", 2024-05-15T12:55:37.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.353e-06 2024-05-15T12:55:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.526562+0000", 2024-05-15T12:55:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1266e-05 2024-05-15T12:55:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.526575+0000", 2024-05-15T12:55:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3232e-05 2024-05-15T12:55:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.526619+0000", 2024-05-15T12:55:37.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4233000000000002e-05 2024-05-15T12:55:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.526754+0000", 2024-05-15T12:55:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013522100000000001 2024-05-15T12:55:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:55:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.526860+0000", 2024-05-15T12:55:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105246 2024-05-15T12:55:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.531900+0000", 2024-05-15T12:55:37.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0050400410000000003 2024-05-15T12:55:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:55:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.532106+0000", 2024-05-15T12:55:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000206422 2024-05-15T12:55:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.682380+0000", 2024-05-15T12:55:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.150274202 2024-05-15T12:55:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.682418+0000", 2024-05-15T12:55:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7465000000000001e-05 2024-05-15T12:55:37.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.682455+0000", 2024-05-15T12:55:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7116999999999997e-05 2024-05-15T12:55:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:180445 2.b 2:d5fe51a2:::benchmark_data_smithi177_78104_object73312:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-05-15T12:55:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:51:42.534617+0000", 2024-05-15T12:55:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 234.530220383, 2024-05-15T12:55:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.15372477500000001, 2024-05-15T12:55:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:37.156 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:37.157 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:37.157 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:55:37.157 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:37.157 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 180445 2024-05-15T12:55:37.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.157 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:37.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:37.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.534617+0000", 2024-05-15T12:55:37.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:37.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.534617+0000", 2024-05-15T12:55:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.534619+0000", 2024-05-15T12:55:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.784e-06 2024-05-15T12:55:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.534624+0000", 2024-05-15T12:55:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.536e-06 2024-05-15T12:55:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.534637+0000", 2024-05-15T12:55:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3161e-05 2024-05-15T12:55:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.534645+0000", 2024-05-15T12:55:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3919999999999997e-06 2024-05-15T12:55:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.534701+0000", 2024-05-15T12:55:37.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5949000000000003e-05 2024-05-15T12:55:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.534828+0000", 2024-05-15T12:55:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000127498 2024-05-15T12:55:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:55:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.534954+0000", 2024-05-15T12:55:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012631799999999999 2024-05-15T12:55:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:55:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.539494+0000", 2024-05-15T12:55:37.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0045394759999999998 2024-05-15T12:55:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.547892+0000", 2024-05-15T12:55:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0083977840000000002 2024-05-15T12:55:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.688245+0000", 2024-05-15T12:55:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14035356900000001 2024-05-15T12:55:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.688299+0000", 2024-05-15T12:55:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3458999999999998e-05 2024-05-15T12:55:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.688342+0000", 2024-05-15T12:55:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2849000000000003e-05 2024-05-15T12:55:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:180462 2.b 2:dd3006b4:::benchmark_data_smithi177_78104_object73329:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-05-15T12:55:37.162 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:51:42.556441+0000", 2024-05-15T12:55:37.163 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 234.508395971, 2024-05-15T12:55:37.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13820427900000001, 2024-05-15T12:55:37.163 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:37.163 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:37.163 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:37.163 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:55:37.163 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:37.163 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 180462 2024-05-15T12:55:37.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.163 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:37.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:37.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.556441+0000", 2024-05-15T12:55:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.556441+0000", 2024-05-15T12:55:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.556442+0000", 2024-05-15T12:55:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3699999999999999e-07 2024-05-15T12:55:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.556447+0000", 2024-05-15T12:55:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6670000000000004e-06 2024-05-15T12:55:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.556458+0000", 2024-05-15T12:55:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0954e-05 2024-05-15T12:55:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.556463+0000", 2024-05-15T12:55:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5589999999999996e-06 2024-05-15T12:55:37.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.556504+0000", 2024-05-15T12:55:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0315999999999998e-05 2024-05-15T12:55:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.556604+0000", 2024-05-15T12:55:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010042100000000001 2024-05-15T12:55:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.166 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:55:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.556678+0000", 2024-05-15T12:55:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3886000000000004e-05 2024-05-15T12:55:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:55:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.563719+0000", 2024-05-15T12:55:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0070414800000000001 2024-05-15T12:55:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.564920+0000", 2024-05-15T12:55:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001200895 2024-05-15T12:55:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.167 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.694575+0000", 2024-05-15T12:55:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.12965502400000001 2024-05-15T12:55:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.694612+0000", 2024-05-15T12:55:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6769e-05 2024-05-15T12:55:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.694646+0000", 2024-05-15T12:55:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3470999999999999e-05 2024-05-15T12:55:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:37.168 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:180465 2.b 2:db125771:::benchmark_data_smithi177_78104_object73332:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-05-15T12:55:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:51:42.556798+0000", 2024-05-15T12:55:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 234.50803895000001, 2024-05-15T12:55:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13799846299999999, 2024-05-15T12:55:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:55:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 180465 2024-05-15T12:55:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.169 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.556798+0000", 2024-05-15T12:55:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.556798+0000", 2024-05-15T12:55:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.556799+0000", 2024-05-15T12:55:37.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4600000000000004e-07 2024-05-15T12:55:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.556804+0000", 2024-05-15T12:55:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5310000000000003e-06 2024-05-15T12:55:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.556814+0000", 2024-05-15T12:55:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0407e-05 2024-05-15T12:55:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:37.171 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.556827+0000", 2024-05-15T12:55:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2835e-05 2024-05-15T12:55:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.560227+0000", 2024-05-15T12:55:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0034007180000000001 2024-05-15T12:55:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.560351+0000", 2024-05-15T12:55:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000123857 2024-05-15T12:55:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:55:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.560479+0000", 2024-05-15T12:55:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012803599999999999 2024-05-15T12:55:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:55:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.563875+0000", 2024-05-15T12:55:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0033952940000000001 2024-05-15T12:55:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.565020+0000", 2024-05-15T12:55:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011455840000000001 2024-05-15T12:55:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.694693+0000", 2024-05-15T12:55:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.12967251699999999 2024-05-15T12:55:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.694751+0000", 2024-05-15T12:55:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8198000000000001e-05 2024-05-15T12:55:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:37.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.694797+0000", 2024-05-15T12:55:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5739999999999999e-05 2024-05-15T12:55:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:180473 2.b 2:dc8ba0f9:::benchmark_data_smithi177_78104_object73340:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-05-15T12:55:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:51:42.570600+0000", 2024-05-15T12:55:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 234.494237193, 2024-05-15T12:55:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.128188262, 2024-05-15T12:55:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:37.175 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:55:37.176 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:37.176 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 180473 2024-05-15T12:55:37.176 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.176 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:37.176 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.176 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:37.176 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.570600+0000", 2024-05-15T12:55:37.176 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.176 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.176 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.176 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.570600+0000", 2024-05-15T12:55:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.570601+0000", 2024-05-15T12:55:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5499999999999998e-07 2024-05-15T12:55:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.570605+0000", 2024-05-15T12:55:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.997e-06 2024-05-15T12:55:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.570616+0000", 2024-05-15T12:55:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1625e-05 2024-05-15T12:55:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.570624+0000", 2024-05-15T12:55:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6299999999999998e-06 2024-05-15T12:55:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.570684+0000", 2024-05-15T12:55:37.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0217999999999998e-05 2024-05-15T12:55:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.570784+0000", 2024-05-15T12:55:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9531000000000006e-05 2024-05-15T12:55:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:55:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.571365+0000", 2024-05-15T12:55:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00058150499999999996 2024-05-15T12:55:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.576070+0000", 2024-05-15T12:55:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0047050140000000004 2024-05-15T12:55:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:55:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.576240+0000", 2024-05-15T12:55:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016976799999999999 2024-05-15T12:55:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.698718+0000", 2024-05-15T12:55:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.122478528 2024-05-15T12:55:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.180 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.698753+0000", 2024-05-15T12:55:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4272e-05 2024-05-15T12:55:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.698788+0000", 2024-05-15T12:55:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5519000000000003e-05 2024-05-15T12:55:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.181 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24562.0:180488 2.b 2:dffad04d:::benchmark_data_smithi177_78104_object73355:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-05-15T12:55:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:51:42.590937+0000", 2024-05-15T12:55:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 234.47390054100001, 2024-05-15T12:55:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.107988853, 2024-05-15T12:55:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24562", 2024-05-15T12:55:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:55:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 180488 2024-05-15T12:55:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:37.182 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.590937+0000", 2024-05-15T12:55:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.590937+0000", 2024-05-15T12:55:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.590938+0000", 2024-05-15T12:55:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.452e-06 2024-05-15T12:55:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.183 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.590946+0000", 2024-05-15T12:55:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1300000000000001e-06 2024-05-15T12:55:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.590964+0000", 2024-05-15T12:55:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7878999999999999e-05 2024-05-15T12:55:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.590975+0000", 2024-05-15T12:55:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0498e-05 2024-05-15T12:55:37.184 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.591041+0000", 2024-05-15T12:55:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6518000000000002e-05 2024-05-15T12:55:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.591176+0000", 2024-05-15T12:55:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013470100000000001 2024-05-15T12:55:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T12:55:37.185 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.591303+0000", 2024-05-15T12:55:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012666500000000001 2024-05-15T12:55:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.596505+0000", 2024-05-15T12:55:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0052020130000000001 2024-05-15T12:55:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:55:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.596611+0000", 2024-05-15T12:55:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010685500000000001 2024-05-15T12:55:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.698843+0000", 2024-05-15T12:55:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.102231945 2024-05-15T12:55:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.698896+0000", 2024-05-15T12:55:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2741000000000002e-05 2024-05-15T12:55:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:42.698926+0000", 2024-05-15T12:55:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9456000000000001e-05 2024-05-15T12:55:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.187 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:37.188 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.188 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:37.188 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:37.188 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T12:55:37.188 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:55:37.189 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:55:37.220 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:37.282 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:55:37.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:37.285 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:55:37.386 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:55:37.415 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:37.452 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:55:37.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:37.462 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:55:37.492 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:55:37.511 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T12:55:37.563 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:55:37.591 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:37.671 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:55:37.671 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:55:37.671 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:55:37.671 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:55:37.671 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:55:37.671 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:55:37.671 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:55:37.724 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:55:37.724 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:55:37.724 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:55:37.724 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:55:37.724 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:55:37.724 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:55:37.724 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:55:37.742 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:55:37.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:37.749 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:55:37.769 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-05-15T12:55:37.850 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:55:37.919 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:37.956 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:55:37.961 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:55:37.961 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:55:37.961 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:55:37.961 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:55:37.962 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:55:37.962 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:55:37.962 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:55:37.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:37.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:55:38.064 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:55:38.096 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:38.100 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:55:38.142 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:55:38.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:38.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:55:38.254 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:55:38.283 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:38.383 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:55:38.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:38.390 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:55:38.491 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:55:38.559 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:38.593 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:55:38.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:38.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:55:38.700 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:55:38.750 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:38.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:38 smithi177 ceph-mon[44303]: pgmap v750: 17 pgs: 17 active+clean; 5.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 826 op/s 2024-05-15T12:55:39.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:38 smithi027 ceph-mon[39795]: pgmap v750: 17 pgs: 17 active+clean; 5.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 826 op/s 2024-05-15T12:55:39.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:38 smithi120 ceph-mon[43961]: pgmap v750: 17 pgs: 17 active+clean; 5.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 826 op/s 2024-05-15T12:55:39.456 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:55:39.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:39.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:55:39.561 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:55:39.602 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:39.629 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:55:39.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:39.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:55:39.715 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T12:55:39.738 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:55:39.781 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:39 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:55:39.782 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:39 smithi177 ceph-mon[44303]: osdmap e272: 12 total, 12 up, 11 in 2024-05-15T12:55:39.782 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:39 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1176727653' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:55:39.782 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:39 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:55:39.787 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:40.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:39 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:55:40.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:39 smithi027 ceph-mon[39795]: osdmap e272: 12 total, 12 up, 11 in 2024-05-15T12:55:40.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:39 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1176727653' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:55:40.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:39 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:55:40.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:39 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:55:40.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:39 smithi120 ceph-mon[43961]: osdmap e272: 12 total, 12 up, 11 in 2024-05-15T12:55:40.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:39 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1176727653' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:55:40.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:39 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:55:40.102 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T12:55:40.102 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T12:55:40.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.102 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24832.0:184684 3.0 3:05e5ef19:::benchmark_data_smithi177_88104_object51062:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e273)", 2024-05-15T12:55:40.103 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:55:40.093601+0000", 2024-05-15T12:55:40.103 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.018547369000000001, 2024-05-15T12:55:40.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.018579894999999999, 2024-05-15T12:55:40.103 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:40.103 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-05-15T12:55:40.103 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:40.103 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24832", 2024-05-15T12:55:40.103 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:55:40.103 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 184684 2024-05-15T12:55:40.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.103 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:40.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:40.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.093601+0000", 2024-05-15T12:55:40.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:40.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:40.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.093601+0000", 2024-05-15T12:55:40.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:40.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:40.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.093602+0000", 2024-05-15T12:55:40.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0920000000000001e-06 2024-05-15T12:55:40.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:40.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.093607+0000", 2024-05-15T12:55:40.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3430000000000003e-06 2024-05-15T12:55:40.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:40.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.093620+0000", 2024-05-15T12:55:40.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2948e-05 2024-05-15T12:55:40.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:40.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.093629+0000", 2024-05-15T12:55:40.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7099999999999996e-06 2024-05-15T12:55:40.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:40.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.096749+0000", 2024-05-15T12:55:40.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003119691 2024-05-15T12:55:40.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:40.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.096881+0000", 2024-05-15T12:55:40.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013264400000000001 2024-05-15T12:55:40.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,6", 2024-05-15T12:55:40.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.097011+0000", 2024-05-15T12:55:40.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012992400000000001 2024-05-15T12:55:40.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:55:40.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.101037+0000", 2024-05-15T12:55:40.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0040255969999999997 2024-05-15T12:55:40.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:40.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.105671+0000", 2024-05-15T12:55:40.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0046339429999999997 2024-05-15T12:55:40.107 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:40.108 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:40.108 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:40.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.108 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24832.0:184693 3.0 3:0198fe30:::benchmark_data_smithi177_88104_object51071:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e273)", 2024-05-15T12:55:40.108 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:55:40.106014+0000", 2024-05-15T12:55:40.108 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0061338749999999996, 2024-05-15T12:55:40.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0062616989999999999, 2024-05-15T12:55:40.108 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:40.108 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-05-15T12:55:40.108 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:40.108 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24832", 2024-05-15T12:55:40.108 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:55:40.108 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 184693 2024-05-15T12:55:40.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.109 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:40.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:40.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.106014+0000", 2024-05-15T12:55:40.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:40.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:40.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.106014+0000", 2024-05-15T12:55:40.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:40.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:40.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.106016+0000", 2024-05-15T12:55:40.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.68e-06 2024-05-15T12:55:40.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:40.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.106022+0000", 2024-05-15T12:55:40.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4910000000000002e-06 2024-05-15T12:55:40.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:40.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.106038+0000", 2024-05-15T12:55:40.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5466999999999999e-05 2024-05-15T12:55:40.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:40.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.106047+0000", 2024-05-15T12:55:40.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3600000000000002e-06 2024-05-15T12:55:40.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:40.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.106098+0000", 2024-05-15T12:55:40.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0250000000000002e-05 2024-05-15T12:55:40.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:40.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.106210+0000", 2024-05-15T12:55:40.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111996 2024-05-15T12:55:40.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,6", 2024-05-15T12:55:40.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.106319+0000", 2024-05-15T12:55:40.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109522 2024-05-15T12:55:40.112 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:40.112 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:40.112 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:40.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.112 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24832.0:184694 3.0 3:0f49f58f:::benchmark_data_smithi177_88104_object51072:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e273)", 2024-05-15T12:55:40.112 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:55:40.106324+0000", 2024-05-15T12:55:40.112 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0058246050000000001, 2024-05-15T12:55:40.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0060267810000000002, 2024-05-15T12:55:40.112 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:40.113 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-05-15T12:55:40.113 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:40.113 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24832", 2024-05-15T12:55:40.113 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:55:40.113 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 184694 2024-05-15T12:55:40.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.113 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:40.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:40.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.106324+0000", 2024-05-15T12:55:40.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:40.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:40.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.106324+0000", 2024-05-15T12:55:40.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:40.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:40.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.106325+0000", 2024-05-15T12:55:40.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0130000000000001e-06 2024-05-15T12:55:40.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:40.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.106329+0000", 2024-05-15T12:55:40.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8199999999999996e-06 2024-05-15T12:55:40.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:40.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.106340+0000", 2024-05-15T12:55:40.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0787e-05 2024-05-15T12:55:40.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:40.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.106349+0000", 2024-05-15T12:55:40.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7649999999999999e-06 2024-05-15T12:55:40.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:40.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.109849+0000", 2024-05-15T12:55:40.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035002990000000001 2024-05-15T12:55:40.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:40.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.109931+0000", 2024-05-15T12:55:40.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2239999999999999e-05 2024-05-15T12:55:40.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,6", 2024-05-15T12:55:40.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.110005+0000", 2024-05-15T12:55:40.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3907999999999998e-05 2024-05-15T12:55:40.116 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:40.116 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:40.116 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:40.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.116 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24832.0:184696 3.f e273/219 3:ff029751:::benchmark_data_smithi177_88104_object51074:head v 273'11497, mlcod=273'11495)", 2024-05-15T12:55:40.116 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:55:40.110771+0000", 2024-05-15T12:55:40.116 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0013766679999999999, 2024-05-15T12:55:40.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001653552, 2024-05-15T12:55:40.117 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:40.117 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T12:55:40.117 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:40.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:40.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.110771+0000", 2024-05-15T12:55:40.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:40.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:40.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.110771+0000", 2024-05-15T12:55:40.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:40.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:40.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.110772+0000", 2024-05-15T12:55:40.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9500000000000002e-07 2024-05-15T12:55:40.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:40.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.110776+0000", 2024-05-15T12:55:40.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1250000000000003e-06 2024-05-15T12:55:40.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:40.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.110786+0000", 2024-05-15T12:55:40.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5000000000000005e-06 2024-05-15T12:55:40.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:40.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.110793+0000", 2024-05-15T12:55:40.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3950000000000003e-06 2024-05-15T12:55:40.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:40.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.110854+0000", 2024-05-15T12:55:40.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0747e-05 2024-05-15T12:55:40.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:40.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.110876+0000", 2024-05-15T12:55:40.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2018000000000002e-05 2024-05-15T12:55:40.119 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:40.120 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:40.120 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:40.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.120 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24832.0:184683 3.0 3:0f35010c:::benchmark_data_smithi177_88104_object51061:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e273)", 2024-05-15T12:55:40.120 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:55:40.091805+0000", 2024-05-15T12:55:40.120 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.020343422999999999, 2024-05-15T12:55:40.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.020682323999999998, 2024-05-15T12:55:40.120 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:40.120 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-05-15T12:55:40.120 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T12:55:40.120 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24832", 2024-05-15T12:55:40.120 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:55:40.120 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 184683 2024-05-15T12:55:40.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.120 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:40.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:40.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.091805+0000", 2024-05-15T12:55:40.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:40.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:40.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.091805+0000", 2024-05-15T12:55:40.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:40.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:40.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.091806+0000", 2024-05-15T12:55:40.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.24e-06 2024-05-15T12:55:40.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:40.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.091811+0000", 2024-05-15T12:55:40.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7940000000000002e-06 2024-05-15T12:55:40.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:40.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.091820+0000", 2024-05-15T12:55:40.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5079999999999994e-06 2024-05-15T12:55:40.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:40.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.091828+0000", 2024-05-15T12:55:40.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8229999999999995e-06 2024-05-15T12:55:40.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:40.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.091883+0000", 2024-05-15T12:55:40.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5367e-05 2024-05-15T12:55:40.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:40.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.092004+0000", 2024-05-15T12:55:40.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120099 2024-05-15T12:55:40.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,6", 2024-05-15T12:55:40.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.092107+0000", 2024-05-15T12:55:40.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103094 2024-05-15T12:55:40.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:55:40.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.100900+0000", 2024-05-15T12:55:40.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0087933549999999992 2024-05-15T12:55:40.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:40.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:40.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T12:55:40.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:55:40.100916+0000", 2024-05-15T12:55:40.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6028999999999998e-05 2024-05-15T12:55:40.124 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:40.125 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:40.125 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:40.125 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:40.125 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-05-15T12:55:40.125 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 5 2024-05-15T12:55:40.125 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T12:55:40.468 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:55:40.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:40.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:55:40.579 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:55:40.637 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:40.665 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":749,"stamp":"2024-05-15T12:55:35.712191+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5778309688,"num_objects":88164,"num_object_clones":0,"num_object_copies":264492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88164,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":179259,"num_write_kb":8554555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161117,"ondisk_log_size":161117,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":19076560,"kb_used_data":17004920,"kb_used_omap":17,"kb_used_meta":2071214,"kb_avail":1012120112,"statfs":{"total":1055945392128,"available":1036410994688,"internally_reserved":0,"allocated":17413038080,"data_stored":17410284550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":2120924021},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,6,14,12,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-663420928,"num_objects":-10123,"num_object_clones":0,"num_object_copies":-30369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10123,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011010"},"pg_stats":[{"pgid":"3.f","version":"271'11270","reported_seq":11459,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T12:55:35.637807+0000","last_change":"2024-05-15T12:54:10.808118+0000","last_active":"2024-05-15T12:55:35.637807+0000","last_peered":"2024-05-15T12:55:35.637807+0000","last_clean":"2024-05-15T12:55:35.637807+0000","last_became_active":"2024-05-15T12:52:56.175163+0000","last_became_peered":"2024-05-15T12:52:56.175163+0000","last_unstale":"2024-05-15T12:55:35.637807+0000","last_undegraded":"2024-05-15T12:55:35.637807+0000","last_fullsized":"2024-05-15T12:55:35.637807+0000","mapping_epoch":219,"log_start":"225'1200","ondisk_log_start":"225'1200","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'6601","last_scrub_stamp":"2024-05-15T12:54:10.807993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:10.807993+0000","objects_scrubbed":6559,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:55:39.403253+0000","scrub_duration":1479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349306880,"num_objects":5330,"num_object_clones":0,"num_object_copies":15990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11270,"num_write_kb":531200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,9],"acting":[7,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"271'11554","reported_seq":11817,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T12:55:34.465703+0000","last_change":"2024-05-15T12:55:02.145527+0000","last_active":"2024-05-15T12:55:34.465703+0000","last_peered":"2024-05-15T12:55:34.465703+0000","last_clean":"2024-05-15T12:55:34.465703+0000","last_became_active":"2024-05-15T12:52:56.170845+0000","last_became_peered":"2024-05-15T12:52:56.170845+0000","last_unstale":"2024-05-15T12:55:34.465703+0000","last_undegraded":"2024-05-15T12:55:34.465703+0000","last_fullsized":"2024-05-15T12:55:34.465703+0000","mapping_epoch":219,"log_start":"226'1500","ondisk_log_start":"226'1500","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'9806","last_scrub_stamp":"2024-05-15T12:55:02.145251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:55:02.145251+0000","objects_scrubbed":7368,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:56:10.554313+0000","scrub_duration":2271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363855872,"num_objects":5552,"num_object_clones":0,"num_object_copies":16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11554,"num_write_kb":547392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,9],"acting":[4,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"271'11286","reported_seq":11481,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T12:55:35.658789+0000","last_change":"2024-05-15T12:54:21.111762+0000","last_active":"2024-05-15T12:55:35.658789+0000","last_peered":"2024-05-15T12:55:35.658789+0000","last_clean":"2024-05-15T12:55:35.658789+0000","last_became_active":"2024-05-15T12:52:56.175090+0000","last_became_peered":"2024-05-15T12:52:56.175090+0000","last_unstale":"2024-05-15T12:55:35.658789+0000","last_undegraded":"2024-05-15T12:55:35.658789+0000","last_fullsized":"2024-05-15T12:55:35.658789+0000","mapping_epoch":219,"log_start":"225'1200","ondisk_log_start":"225'1200","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'7179","last_scrub_stamp":"2024-05-15T12:54:21.111584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:21.111584+0000","objects_scrubbed":7133,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:55:39.403291+0000","scrub_duration":1752,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352649240,"num_objects":5382,"num_object_clones":0,"num_object_copies":16146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5382,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11286,"num_write_kb":533313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"271'11134","reported_seq":11333,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T12:55:32.891520+0000","last_change":"2024-05-15T12:54:57.015086+0000","last_active":"2024-05-15T12:55:32.891520+0000","last_peered":"2024-05-15T12:55:32.891520+0000","last_clean":"2024-05-15T12:55:32.891520+0000","last_became_active":"2024-05-15T12:52:56.176874+0000","last_became_peered":"2024-05-15T12:52:56.176874+0000","last_unstale":"2024-05-15T12:55:32.891520+0000","last_undegraded":"2024-05-15T12:55:32.891520+0000","last_fullsized":"2024-05-15T12:55:32.891520+0000","mapping_epoch":219,"log_start":"225'1100","ondisk_log_start":"225'1100","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'9251","last_scrub_stamp":"2024-05-15T12:54:57.014861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:57.014861+0000","objects_scrubbed":7496,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:56:12.719100+0000","scrub_duration":2328,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364380160,"num_objects":5560,"num_object_clones":0,"num_object_copies":16680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11134,"num_write_kb":534208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"271'11271","reported_seq":11456,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T12:55:32.124964+0000","last_change":"2024-05-15T12:54:04.886544+0000","last_active":"2024-05-15T12:55:32.124964+0000","last_peered":"2024-05-15T12:55:32.124964+0000","last_clean":"2024-05-15T12:55:32.124964+0000","last_became_active":"2024-05-15T12:52:56.170556+0000","last_became_peered":"2024-05-15T12:52:56.170556+0000","last_unstale":"2024-05-15T12:55:32.124964+0000","last_undegraded":"2024-05-15T12:55:32.124964+0000","last_fullsized":"2024-05-15T12:55:32.124964+0000","mapping_epoch":219,"log_start":"225'1200","ondisk_log_start":"225'1200","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'6237","last_scrub_stamp":"2024-05-15T12:54:04.886417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:04.886417+0000","objects_scrubbed":6175,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:55:35.776027+0000","scrub_duration":1939,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371916800,"num_objects":5675,"num_object_clones":0,"num_object_copies":17025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11271,"num_write_kb":542272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"271'11268","reported_seq":11464,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T12:55:33.508758+0000","last_change":"2024-05-15T12:54:21.749033+0000","last_active":"2024-05-15T12:55:33.508758+0000","last_peered":"2024-05-15T12:55:33.508758+0000","last_clean":"2024-05-15T12:55:33.508758+0000","last_became_active":"2024-05-15T12:52:56.176405+0000","last_became_peered":"2024-05-15T12:52:56.176405+0000","last_unstale":"2024-05-15T12:55:33.508758+0000","last_undegraded":"2024-05-15T12:55:33.508758+0000","last_fullsized":"2024-05-15T12:55:33.508758+0000","mapping_epoch":219,"log_start":"225'1200","ondisk_log_start":"225'1200","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'7332","last_scrub_stamp":"2024-05-15T12:54:21.748890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:21.748890+0000","objects_scrubbed":7266,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:55:51.430245+0000","scrub_duration":2221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365690880,"num_objects":5580,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11268,"num_write_kb":539136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"271'11592","reported_seq":11789,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T12:55:35.238017+0000","last_change":"2024-05-15T12:54:21.235627+0000","last_active":"2024-05-15T12:55:35.238017+0000","last_peered":"2024-05-15T12:55:35.238017+0000","last_clean":"2024-05-15T12:55:35.238017+0000","last_became_active":"2024-05-15T12:52:56.173402+0000","last_became_peered":"2024-05-15T12:52:56.173402+0000","last_unstale":"2024-05-15T12:55:35.238017+0000","last_undegraded":"2024-05-15T12:55:35.238017+0000","last_fullsized":"2024-05-15T12:55:35.238017+0000","mapping_epoch":219,"log_start":"226'1500","ondisk_log_start":"226'1500","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'7443","last_scrub_stamp":"2024-05-15T12:54:21.235476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:21.235476+0000","objects_scrubbed":7425,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:55:39.899885+0000","scrub_duration":1082,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359268352,"num_objects":5482,"num_object_clones":0,"num_object_copies":16446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11592,"num_write_kb":546368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,9],"acting":[8,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"271'10970","reported_seq":11159,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T12:55:31.030772+0000","last_change":"2024-05-15T12:54:11.330474+0000","last_active":"2024-05-15T12:55:31.030772+0000","last_peered":"2024-05-15T12:55:31.030772+0000","last_clean":"2024-05-15T12:55:31.030772+0000","last_became_active":"2024-05-15T12:52:56.174589+0000","last_became_peered":"2024-05-15T12:52:56.174589+0000","last_unstale":"2024-05-15T12:55:31.030772+0000","last_undegraded":"2024-05-15T12:55:31.030772+0000","last_fullsized":"2024-05-15T12:55:31.030772+0000","mapping_epoch":219,"log_start":"224'900","ondisk_log_start":"224'900","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'6555","last_scrub_stamp":"2024-05-15T12:54:11.330191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:11.330191+0000","objects_scrubbed":6496,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:55:35.627460+0000","scrub_duration":1897,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370802688,"num_objects":5658,"num_object_clones":0,"num_object_copies":16974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10970,"num_write_kb":532096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"271'10924","reported_seq":11108,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T12:55:32.066221+0000","last_change":"2024-05-15T12:54:04.170429+0000","last_active":"2024-05-15T12:55:32.066221+0000","last_peered":"2024-05-15T12:55:32.066221+0000","last_clean":"2024-05-15T12:55:32.066221+0000","last_became_active":"2024-05-15T12:52:56.175515+0000","last_became_peered":"2024-05-15T12:52:56.175515+0000","last_unstale":"2024-05-15T12:55:32.066221+0000","last_undegraded":"2024-05-15T12:55:32.066221+0000","last_fullsized":"2024-05-15T12:55:32.066221+0000","mapping_epoch":219,"log_start":"224'900","ondisk_log_start":"224'900","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'5999","last_scrub_stamp":"2024-05-15T12:54:04.170185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:04.170185+0000","objects_scrubbed":5903,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:55:35.783874+0000","scrub_duration":2224,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358612992,"num_objects":5472,"num_object_clones":0,"num_object_copies":16416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10924,"num_write_kb":524672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"271'11307","reported_seq":11508,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T12:55:33.571684+0000","last_change":"2024-05-15T12:54:54.253282+0000","last_active":"2024-05-15T12:55:33.571684+0000","last_peered":"2024-05-15T12:55:33.571684+0000","last_clean":"2024-05-15T12:55:33.571684+0000","last_became_active":"2024-05-15T12:52:56.169719+0000","last_became_peered":"2024-05-15T12:52:56.169719+0000","last_unstale":"2024-05-15T12:55:33.571684+0000","last_undegraded":"2024-05-15T12:55:33.571684+0000","last_fullsized":"2024-05-15T12:55:33.571684+0000","mapping_epoch":219,"log_start":"225'1300","ondisk_log_start":"225'1300","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'9231","last_scrub_stamp":"2024-05-15T12:54:54.252859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:54.252859+0000","objects_scrubbed":7684,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:55:55.071670+0000","scrub_duration":2207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363528192,"num_objects":5547,"num_object_clones":0,"num_object_copies":16641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11307,"num_write_kb":539328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"271'11083","reported_seq":11320,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T12:55:32.899031+0000","last_change":"2024-05-15T12:54:53.922409+0000","last_active":"2024-05-15T12:55:32.899031+0000","last_peered":"2024-05-15T12:55:32.899031+0000","last_clean":"2024-05-15T12:55:32.899031+0000","last_became_active":"2024-05-15T12:52:56.273601+0000","last_became_peered":"2024-05-15T12:52:56.273601+0000","last_unstale":"2024-05-15T12:55:32.899031+0000","last_undegraded":"2024-05-15T12:55:32.899031+0000","last_fullsized":"2024-05-15T12:55:32.899031+0000","mapping_epoch":219,"log_start":"224'1000","ondisk_log_start":"224'1000","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'9107","last_scrub_stamp":"2024-05-15T12:54:53.922303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:53.922303+0000","objects_scrubbed":7674,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:56:04.424915+0000","scrub_duration":2205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369033216,"num_objects":5631,"num_object_clones":0,"num_object_copies":16893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11083,"num_write_kb":534848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"271'10955","reported_seq":11141,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T12:55:31.026449+0000","last_change":"2024-05-15T12:54:09.050583+0000","last_active":"2024-05-15T12:55:31.026449+0000","last_peered":"2024-05-15T12:55:31.026449+0000","last_clean":"2024-05-15T12:55:31.026449+0000","last_became_active":"2024-05-15T12:52:56.273535+0000","last_became_peered":"2024-05-15T12:52:56.273535+0000","last_unstale":"2024-05-15T12:55:31.026449+0000","last_undegraded":"2024-05-15T12:55:31.026449+0000","last_fullsized":"2024-05-15T12:55:31.026449+0000","mapping_epoch":219,"log_start":"224'900","ondisk_log_start":"224'900","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'6434","last_scrub_stamp":"2024-05-15T12:54:09.050363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:09.050363+0000","objects_scrubbed":6370,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:55:34.760521+0000","scrub_duration":1655,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370475008,"num_objects":5653,"num_object_clones":0,"num_object_copies":16959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10955,"num_write_kb":531456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"271'10978","reported_seq":11167,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T12:55:34.495712+0000","last_change":"2024-05-15T12:54:16.516142+0000","last_active":"2024-05-15T12:55:34.495712+0000","last_peered":"2024-05-15T12:55:34.495712+0000","last_clean":"2024-05-15T12:55:34.495712+0000","last_became_active":"2024-05-15T12:52:56.174822+0000","last_became_peered":"2024-05-15T12:52:56.174822+0000","last_unstale":"2024-05-15T12:55:34.495712+0000","last_undegraded":"2024-05-15T12:55:34.495712+0000","last_fullsized":"2024-05-15T12:55:34.495712+0000","mapping_epoch":219,"log_start":"224'900","ondisk_log_start":"224'900","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'6783","last_scrub_stamp":"2024-05-15T12:54:16.515960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:16.515960+0000","objects_scrubbed":6740,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:55:34.878891+0000","scrub_duration":1694,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348651520,"num_objects":5320,"num_object_clones":0,"num_object_copies":15960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10978,"num_write_kb":521536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"271'10959","reported_seq":11152,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T12:55:33.577253+0000","last_change":"2024-05-15T12:54:21.477942+0000","last_active":"2024-05-15T12:55:33.577253+0000","last_peered":"2024-05-15T12:55:33.577253+0000","last_clean":"2024-05-15T12:55:33.577253+0000","last_became_active":"2024-05-15T12:52:56.273621+0000","last_became_peered":"2024-05-15T12:52:56.273621+0000","last_unstale":"2024-05-15T12:55:33.577253+0000","last_undegraded":"2024-05-15T12:55:33.577253+0000","last_fullsized":"2024-05-15T12:55:33.577253+0000","mapping_epoch":219,"log_start":"224'900","ondisk_log_start":"224'900","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'7086","last_scrub_stamp":"2024-05-15T12:54:21.477706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:21.477706+0000","objects_scrubbed":7054,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:55:33.786720+0000","scrub_duration":1461,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343867392,"num_objects":5247,"num_object_clones":0,"num_object_copies":15741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10959,"num_write_kb":518592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10],"acting":[0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"271'11329","reported_seq":11526,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T12:55:35.349110+0000","last_change":"2024-05-15T12:54:56.371913+0000","last_active":"2024-05-15T12:55:35.349110+0000","last_peered":"2024-05-15T12:55:35.349110+0000","last_clean":"2024-05-15T12:55:35.349110+0000","last_became_active":"2024-05-15T12:52:56.173177+0000","last_became_peered":"2024-05-15T12:52:56.173177+0000","last_unstale":"2024-05-15T12:55:35.349110+0000","last_undegraded":"2024-05-15T12:55:35.349110+0000","last_fullsized":"2024-05-15T12:55:35.349110+0000","mapping_epoch":219,"log_start":"225'1300","ondisk_log_start":"225'1300","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'9339","last_scrub_stamp":"2024-05-15T12:54:56.371625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:56.371625+0000","objects_scrubbed":7499,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:56:15.477068+0000","scrub_duration":1588,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358285312,"num_objects":5467,"num_object_clones":0,"num_object_copies":16401,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11329,"num_write_kb":537472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,1],"acting":[9,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2","version":"271'11198","reported_seq":11396,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T12:55:32.179099+0000","last_change":"2024-05-15T12:54:52.113647+0000","last_active":"2024-05-15T12:55:32.179099+0000","last_peered":"2024-05-15T12:55:32.179099+0000","last_clean":"2024-05-15T12:55:32.179099+0000","last_became_active":"2024-05-15T12:52:56.173445+0000","last_became_peered":"2024-05-15T12:52:56.173445+0000","last_unstale":"2024-05-15T12:55:32.179099+0000","last_undegraded":"2024-05-15T12:55:32.179099+0000","last_fullsized":"2024-05-15T12:55:32.179099+0000","mapping_epoch":219,"log_start":"224'1100","ondisk_log_start":"224'1100","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'9122","last_scrub_stamp":"2024-05-15T12:54:52.113308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:52.113308+0000","objects_scrubbed":7745,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:56:05.645218+0000","scrub_duration":2293,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367394816,"num_objects":5606,"num_object_clones":0,"num_object_copies":16818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11198,"num_write_kb":537728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":766,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T12:55:20.818887+0000","last_change":"2024-05-15T12:55:06.370818+0000","last_active":"2024-05-15T12:55:20.818887+0000","last_peered":"2024-05-15T12:55:20.818887+0000","last_clean":"2024-05-15T12:55:20.818887+0000","last_became_active":"2024-05-15T12:49:08.168334+0000","last_became_peered":"2024-05-15T12:49:08.168334+0000","last_unstale":"2024-05-15T12:55:20.818887+0000","last_undegraded":"2024-05-15T12:55:20.818887+0000","last_fullsized":"2024-05-15T12:55:20.818887+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T12:55:06.370673+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T12:55:06.370673+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:56:30.073725+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":5777719320,"num_objects":88162,"num_object_clones":0,"num_object_copies":264486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88162,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179078,"num_write_kb":8551617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17307152384,"data_stored":17307140168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160978,"ondisk_log_size":160978,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":0,"seq":386547056761,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351101,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":510276,"kb_used_data":344444,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":93234876,"statfs":{"total":95995035648,"available":95472513024,"internally_reserved":0,"allocated":352710656,"data_stored":352462398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":169802176},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514627,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2352784,"kb_used_data":2100872,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":91392368,"statfs":{"total":95995035648,"available":93585784832,"internally_reserved":0,"allocated":2151292928,"data_stored":2151037046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":257948096},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874377,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":872324,"kb_used_data":696764,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":92872828,"statfs":{"total":95995035648,"available":95101775872,"internally_reserved":0,"allocated":713486336,"data_stored":713238098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168716,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1557328,"kb_used_data":1377608,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92187824,"statfs":{"total":95995035648,"available":94400331776,"internally_reserved":0,"allocated":1410670592,"data_stored":1410414730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":184023498},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430353,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1896076,"kb_used_data":1766340,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":91849076,"statfs":{"total":95995035648,"available":94053453824,"internally_reserved":0,"allocated":1808732160,"data_stored":1808480370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575020,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2004484,"kb_used_data":1828796,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":91740668,"statfs":{"total":95995035648,"available":93942444032,"internally_reserved":0,"allocated":1872687104,"data_stored":1872438866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738548,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2350724,"kb_used_data":2112700,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":91394428,"statfs":{"total":95995035648,"available":93587894272,"internally_reserved":0,"allocated":2163404800,"data_stored":2163156562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411497,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":862724,"kb_used_data":719100,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92882428,"statfs":{"total":95995035648,"available":95111606272,"internally_reserved":0,"allocated":736358400,"data_stored":736110162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149857,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2091076,"kb_used_data":1803004,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":91654076,"statfs":{"total":95995035648,"available":93853773824,"internally_reserved":0,"allocated":1846276096,"data_stored":1846027858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":294910407},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986333,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2631876,"kb_used_data":2456188,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":91113276,"statfs":{"total":95995035648,"available":93299994624,"internally_reserved":0,"allocated":2515136512,"data_stored":2514888274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691991,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1946888,"kb_used_data":1799104,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":91798264,"statfs":{"total":95995035648,"available":94001422336,"internally_reserved":0,"allocated":1842282496,"data_stored":1842030186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2147221504,"data_stored":2147221504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1860370432,"data_stored":1860370432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":733478912,"data_stored":733478912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1837105152,"data_stored":1837105152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2497380352,"data_stored":2497380352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1823477760,"data_stored":1823473688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1805189120,"data_stored":1805189120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1405423616,"data_stored":1405419544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":709951488,"data_stored":709951488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2135691264,"data_stored":2135687192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":351862784,"data_stored":351862784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T12:55:40.667 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T12:55:40.675 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:55:40.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:40.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:55:40.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:40 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:55:40.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:40 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:55:40.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:40 smithi027 ceph-mon[39795]: pgmap v752: 17 pgs: 17 active+clean; 5.2 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 868 op/s 2024-05-15T12:55:40.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:40 smithi027 ceph-mon[39795]: osdmap e273: 12 total, 12 up, 11 in 2024-05-15T12:55:40.776 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-05-15T12:55:40.786 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:55:40.832 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:40.877 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:55:40.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:40.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:55:40.988 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:55:41.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:40 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:55:41.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:40 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:55:41.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:40 smithi120 ceph-mon[43961]: pgmap v752: 17 pgs: 17 active+clean; 5.2 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 868 op/s 2024-05-15T12:55:41.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:40 smithi120 ceph-mon[43961]: osdmap e273: 12 total, 12 up, 11 in 2024-05-15T12:55:41.065 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:41.074 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:55:41.133 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:55:41.169 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:55:41.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:40 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:55:41.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:40 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:55:41.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:40 smithi177 ceph-mon[44303]: pgmap v752: 17 pgs: 17 active+clean; 5.2 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 868 op/s 2024-05-15T12:55:41.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:40 smithi177 ceph-mon[44303]: osdmap e273: 12 total, 12 up, 11 in 2024-05-15T12:55:41.212 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:55:41.249 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:55:41.280 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:55:41.318 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:55:41.367 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:55:41.416 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:55:41.446 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:55:41.474 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:55:41.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:41.490 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:55:41.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:55:41.509 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:55:41.557 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:55:41.592 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:55:41.596 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:55:41.610 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:55:41.610 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:55:41.610 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:55:41.610 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:55:41.610 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:55:41.610 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:55:41.610 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:55:41.611 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:55:41.611 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:55:41.611 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:55:41.611 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:55:41.611 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:55:41.671 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:41.747 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:41 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3639267026' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:55:42.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:41 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3639267026' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:55:42.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:41 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3639267026' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:55:42.243 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:55:42.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:42.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:55:42.378 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:55:42.411 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:42.435 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:55:42.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:42.440 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:55:42.541 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:55:42.588 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:42.619 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:55:42.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:42.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:55:42.722 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:55:42.756 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:42.810 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:55:42.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:42.814 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:55:42.915 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:55:42.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:42 smithi177 ceph-mon[44303]: 3.4 scrub starts 2024-05-15T12:55:42.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:42 smithi177 ceph-mon[44303]: pgmap v754: 17 pgs: 17 active+clean; 5.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 802 op/s 2024-05-15T12:55:42.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:42 smithi177 ceph-mon[44303]: 3.b scrub starts 2024-05-15T12:55:42.937 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T12:55:42.937 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:55:42.967 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:43.002 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:55:43.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:43.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:55:43.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:42 smithi027 ceph-mon[39795]: 3.4 scrub starts 2024-05-15T12:55:43.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:42 smithi027 ceph-mon[39795]: pgmap v754: 17 pgs: 17 active+clean; 5.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 802 op/s 2024-05-15T12:55:43.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:42 smithi027 ceph-mon[39795]: 3.b scrub starts 2024-05-15T12:55:43.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:42 smithi120 ceph-mon[43961]: 3.4 scrub starts 2024-05-15T12:55:43.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:42 smithi120 ceph-mon[43961]: pgmap v754: 17 pgs: 17 active+clean; 5.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 802 op/s 2024-05-15T12:55:43.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:42 smithi120 ceph-mon[43961]: 3.b scrub starts 2024-05-15T12:55:43.107 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:55:43.138 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:43.172 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:55:43.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:43.176 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:55:43.277 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:55:43.342 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:43.383 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:55:43.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:43.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:55:43.489 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:55:43.518 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T12:55:43.559 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:43.615 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:55:43.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:43.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:55:43.698 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-05-15T12:55:43.746 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:55:43.775 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:43.799 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:55:43.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:43.802 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:55:43.902 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:55:43.952 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:43.982 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:55:43.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:43.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:55:44.002 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:55:44.088 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:55:44.117 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:44.267 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:55:44.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:44.269 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:55:44.277 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:55:44.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:44 smithi120 ceph-mon[43961]: 3.8 scrub starts 2024-05-15T12:55:44.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:44 smithi120 ceph-mon[43961]: 3.7 scrub starts 2024-05-15T12:55:44.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:44 smithi120 ceph-mon[43961]: 3.4 scrub ok 2024-05-15T12:55:44.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:44 smithi120 ceph-mon[43961]: pgmap v755: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 844 op/s 2024-05-15T12:55:44.310 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:44 smithi177 ceph-mon[44303]: 3.8 scrub starts 2024-05-15T12:55:44.310 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:44 smithi177 ceph-mon[44303]: 3.7 scrub starts 2024-05-15T12:55:44.311 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:44 smithi177 ceph-mon[44303]: 3.4 scrub ok 2024-05-15T12:55:44.311 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:44 smithi177 ceph-mon[44303]: pgmap v755: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 844 op/s 2024-05-15T12:55:44.370 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:55:44.415 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:44.494 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:55:44.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:44.497 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:55:44.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:44 smithi027 ceph-mon[39795]: 3.8 scrub starts 2024-05-15T12:55:44.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:44 smithi027 ceph-mon[39795]: 3.7 scrub starts 2024-05-15T12:55:44.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:44 smithi027 ceph-mon[39795]: 3.4 scrub ok 2024-05-15T12:55:44.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:44 smithi027 ceph-mon[39795]: pgmap v755: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 844 op/s 2024-05-15T12:55:44.597 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:55:44.625 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:44.817 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:55:44.828 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:55:44.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:44.831 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:55:44.931 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:55:44.998 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:45.044 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:55:45.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:45.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:55:45.082 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:45 smithi177 ceph-mon[44303]: 3.b scrub ok 2024-05-15T12:55:45.082 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:45 smithi177 ceph-mon[44303]: 3.1 scrub starts 2024-05-15T12:55:45.082 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:45 smithi177 ceph-mon[44303]: 3.8 scrub ok 2024-05-15T12:55:45.082 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:45 smithi177 ceph-mon[44303]: 3.3 scrub starts 2024-05-15T12:55:45.150 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:55:45.180 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:45.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:45 smithi027 ceph-mon[39795]: 3.b scrub ok 2024-05-15T12:55:45.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:45 smithi027 ceph-mon[39795]: 3.1 scrub starts 2024-05-15T12:55:45.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:45 smithi027 ceph-mon[39795]: 3.8 scrub ok 2024-05-15T12:55:45.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:45 smithi027 ceph-mon[39795]: 3.3 scrub starts 2024-05-15T12:55:45.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:45 smithi120 ceph-mon[43961]: 3.b scrub ok 2024-05-15T12:55:45.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:45 smithi120 ceph-mon[43961]: 3.1 scrub starts 2024-05-15T12:55:45.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:45 smithi120 ceph-mon[43961]: 3.8 scrub ok 2024-05-15T12:55:45.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:45 smithi120 ceph-mon[43961]: 3.3 scrub starts 2024-05-15T12:55:45.283 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:55:45.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:45.287 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:55:45.344 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:55:45.348 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T12:55:45.388 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:55:45.447 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:45.477 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:55:45.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:45.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:55:45.586 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:55:45.617 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:45.645 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:55:45.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:45.648 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:55:45.749 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:55:45.814 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:45.912 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T12:55:45.912 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:55:46.023 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:55:46.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:46.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:55:46.127 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:55:46.204 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:46.234 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:55:46.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:46.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:55:46.331 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:46 smithi177 ceph-mon[44303]: 3.7 scrub ok 2024-05-15T12:55:46.331 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:46 smithi177 ceph-mon[44303]: 3.9 scrub starts 2024-05-15T12:55:46.332 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:46 smithi177 ceph-mon[44303]: 3.1 scrub ok 2024-05-15T12:55:46.332 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:46 smithi177 ceph-mon[44303]: from='client.25012 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:55:46.332 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:46 smithi177 ceph-mon[44303]: pgmap v756: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 843 op/s 2024-05-15T12:55:46.342 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:55:46.377 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:46 smithi027 ceph-mon[39795]: 3.7 scrub ok 2024-05-15T12:55:46.378 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:46 smithi027 ceph-mon[39795]: 3.9 scrub starts 2024-05-15T12:55:46.378 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:46 smithi027 ceph-mon[39795]: 3.1 scrub ok 2024-05-15T12:55:46.378 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:46 smithi027 ceph-mon[39795]: from='client.25012 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:55:46.378 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:46 smithi027 ceph-mon[39795]: pgmap v756: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 843 op/s 2024-05-15T12:55:46.380 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:46.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:46 smithi120 ceph-mon[43961]: 3.7 scrub ok 2024-05-15T12:55:46.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:46 smithi120 ceph-mon[43961]: 3.9 scrub starts 2024-05-15T12:55:46.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:46 smithi120 ceph-mon[43961]: 3.1 scrub ok 2024-05-15T12:55:46.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:46 smithi120 ceph-mon[43961]: from='client.25012 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:55:46.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:46 smithi120 ceph-mon[43961]: pgmap v756: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 843 op/s 2024-05-15T12:55:46.704 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-05-15T12:55:46.830 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":755,"stamp":"2024-05-15T12:55:43.723048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5335024184,"num_objects":81400,"num_object_clones":0,"num_object_copies":244200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81400,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":186023,"num_write_kb":8554555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160981,"ondisk_log_size":160981,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":17817644,"kb_used_data":15712692,"kb_used_omap":17,"kb_used_meta":2104558,"kb_avail":1013379028,"statfs":{"total":1055945392128,"available":1037700124672,"internally_reserved":0,"allocated":16089796608,"data_stored":16086947112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":2155068277},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,3,5,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-443285504,"num_objects":-6764,"num_object_clones":0,"num_object_copies":-20292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6764,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.010856"},"pg_stats":[{"pgid":"3.f","version":"273'11523","reported_seq":11715,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-05-15T12:55:40.633534+0000","last_change":"2024-05-15T12:54:10.808118+0000","last_active":"2024-05-15T12:55:40.633534+0000","last_peered":"2024-05-15T12:55:40.633534+0000","last_clean":"2024-05-15T12:55:40.633534+0000","last_became_active":"2024-05-15T12:52:56.175163+0000","last_became_peered":"2024-05-15T12:52:56.175163+0000","last_unstale":"2024-05-15T12:55:40.633534+0000","last_undegraded":"2024-05-15T12:55:40.633534+0000","last_fullsized":"2024-05-15T12:55:40.633534+0000","mapping_epoch":219,"log_start":"226'1500","ondisk_log_start":"226'1500","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'6601","last_scrub_stamp":"2024-05-15T12:54:10.807993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:10.807993+0000","objects_scrubbed":6559,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332726272,"num_objects":5077,"num_object_clones":0,"num_object_copies":15231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11523,"num_write_kb":531200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,9],"acting":[7,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"272'11841","reported_seq":12105,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T12:55:39.491943+0000","last_change":"2024-05-15T12:55:02.145527+0000","last_active":"2024-05-15T12:55:39.491943+0000","last_peered":"2024-05-15T12:55:39.491943+0000","last_clean":"2024-05-15T12:55:39.491943+0000","last_became_active":"2024-05-15T12:52:56.170845+0000","last_became_peered":"2024-05-15T12:52:56.170845+0000","last_unstale":"2024-05-15T12:55:39.491943+0000","last_undegraded":"2024-05-15T12:55:39.491943+0000","last_fullsized":"2024-05-15T12:55:39.491943+0000","mapping_epoch":219,"log_start":"226'1800","ondisk_log_start":"226'1800","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'9806","last_scrub_stamp":"2024-05-15T12:55:02.145251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:55:02.145251+0000","objects_scrubbed":7368,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:56:10.554313+0000","scrub_duration":2271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345047040,"num_objects":5265,"num_object_clones":0,"num_object_copies":15795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11841,"num_write_kb":547392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,9],"acting":[4,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"273'11538","reported_seq":11737,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-05-15T12:55:40.662530+0000","last_change":"2024-05-15T12:54:21.111762+0000","last_active":"2024-05-15T12:55:40.662530+0000","last_peered":"2024-05-15T12:55:40.662530+0000","last_clean":"2024-05-15T12:55:40.662530+0000","last_became_active":"2024-05-15T12:52:56.175090+0000","last_became_peered":"2024-05-15T12:52:56.175090+0000","last_unstale":"2024-05-15T12:55:40.662530+0000","last_undegraded":"2024-05-15T12:55:40.662530+0000","last_fullsized":"2024-05-15T12:55:40.662530+0000","mapping_epoch":219,"log_start":"226'1500","ondisk_log_start":"226'1500","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'7179","last_scrub_stamp":"2024-05-15T12:54:21.111584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:21.111584+0000","objects_scrubbed":7133,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1752,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336134168,"num_objects":5130,"num_object_clones":0,"num_object_copies":15390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5130,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11538,"num_write_kb":533313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"273'11651","reported_seq":11854,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-05-15T12:55:42.719454+0000","last_change":"2024-05-15T12:54:57.015086+0000","last_active":"2024-05-15T12:55:42.719454+0000","last_peered":"2024-05-15T12:55:42.719454+0000","last_clean":"2024-05-15T12:55:42.719454+0000","last_became_active":"2024-05-15T12:52:56.176874+0000","last_became_peered":"2024-05-15T12:52:56.176874+0000","last_unstale":"2024-05-15T12:55:42.719454+0000","last_undegraded":"2024-05-15T12:55:42.719454+0000","last_fullsized":"2024-05-15T12:55:42.719454+0000","mapping_epoch":219,"log_start":"226'1600","ondisk_log_start":"226'1600","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'9251","last_scrub_stamp":"2024-05-15T12:54:57.014861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:57.014861+0000","objects_scrubbed":7496,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:56:12.719100+0000","scrub_duration":2328,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330498048,"num_objects":5043,"num_object_clones":0,"num_object_copies":15129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11651,"num_write_kb":534208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"273'11789","reported_seq":11991,"reported_epoch":273,"state":"active+clean+scrubbing","last_fresh":"2024-05-15T12:55:42.190872+0000","last_change":"2024-05-15T12:55:41.745788+0000","last_active":"2024-05-15T12:55:42.190872+0000","last_peered":"2024-05-15T12:55:42.190872+0000","last_clean":"2024-05-15T12:55:42.190872+0000","last_became_active":"2024-05-15T12:52:56.170556+0000","last_became_peered":"2024-05-15T12:52:56.170556+0000","last_unstale":"2024-05-15T12:55:42.190872+0000","last_undegraded":"2024-05-15T12:55:42.190872+0000","last_fullsized":"2024-05-15T12:55:42.190872+0000","mapping_epoch":219,"log_start":"226'1700","ondisk_log_start":"226'1700","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'6237","last_scrub_stamp":"2024-05-15T12:54:04.886417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:04.886417+0000","objects_scrubbed":1199,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"scrubbing for 0s","scrub_duration":1939,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337969152,"num_objects":5157,"num_object_clones":0,"num_object_copies":15471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11789,"num_write_kb":542272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"273'11751","reported_seq":11951,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-05-15T12:55:43.510174+0000","last_change":"2024-05-15T12:54:21.749033+0000","last_active":"2024-05-15T12:55:43.510174+0000","last_peered":"2024-05-15T12:55:43.510174+0000","last_clean":"2024-05-15T12:55:43.510174+0000","last_became_active":"2024-05-15T12:52:56.176405+0000","last_became_peered":"2024-05-15T12:52:56.176405+0000","last_unstale":"2024-05-15T12:55:43.510174+0000","last_undegraded":"2024-05-15T12:55:43.510174+0000","last_fullsized":"2024-05-15T12:55:43.510174+0000","mapping_epoch":219,"log_start":"226'1700","ondisk_log_start":"226'1700","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'7332","last_scrub_stamp":"2024-05-15T12:54:21.748890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:21.748890+0000","objects_scrubbed":7266,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:55:51.430245+0000","scrub_duration":2221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334036992,"num_objects":5097,"num_object_clones":0,"num_object_copies":15291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11751,"num_write_kb":539136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"273'11854","reported_seq":12055,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-05-15T12:55:40.245008+0000","last_change":"2024-05-15T12:54:21.235627+0000","last_active":"2024-05-15T12:55:40.245008+0000","last_peered":"2024-05-15T12:55:40.245008+0000","last_clean":"2024-05-15T12:55:40.245008+0000","last_became_active":"2024-05-15T12:52:56.173402+0000","last_became_peered":"2024-05-15T12:52:56.173402+0000","last_unstale":"2024-05-15T12:55:40.245008+0000","last_undegraded":"2024-05-15T12:55:40.245008+0000","last_fullsized":"2024-05-15T12:55:40.245008+0000","mapping_epoch":219,"log_start":"226'1800","ondisk_log_start":"226'1800","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'7443","last_scrub_stamp":"2024-05-15T12:54:21.235476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:21.235476+0000","objects_scrubbed":7425,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1082,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342097920,"num_objects":5220,"num_object_clones":0,"num_object_copies":15660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11854,"num_write_kb":546368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,9],"acting":[8,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"273'11496","reported_seq":11689,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-05-15T12:55:41.072434+0000","last_change":"2024-05-15T12:54:11.330474+0000","last_active":"2024-05-15T12:55:41.072434+0000","last_peered":"2024-05-15T12:55:41.072434+0000","last_clean":"2024-05-15T12:55:41.072434+0000","last_became_active":"2024-05-15T12:52:56.174589+0000","last_became_peered":"2024-05-15T12:52:56.174589+0000","last_unstale":"2024-05-15T12:55:41.072434+0000","last_undegraded":"2024-05-15T12:55:41.072434+0000","last_fullsized":"2024-05-15T12:55:41.072434+0000","mapping_epoch":219,"log_start":"226'1400","ondisk_log_start":"226'1400","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'6555","last_scrub_stamp":"2024-05-15T12:54:11.330191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:11.330191+0000","objects_scrubbed":6496,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1897,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336330752,"num_objects":5132,"num_object_clones":0,"num_object_copies":15396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11496,"num_write_kb":532096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"273'11435","reported_seq":11623,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-05-15T12:55:42.157134+0000","last_change":"2024-05-15T12:54:04.170429+0000","last_active":"2024-05-15T12:55:42.157134+0000","last_peered":"2024-05-15T12:55:42.157134+0000","last_clean":"2024-05-15T12:55:42.157134+0000","last_became_active":"2024-05-15T12:52:56.175515+0000","last_became_peered":"2024-05-15T12:52:56.175515+0000","last_unstale":"2024-05-15T12:55:42.157134+0000","last_undegraded":"2024-05-15T12:55:42.157134+0000","last_fullsized":"2024-05-15T12:55:42.157134+0000","mapping_epoch":219,"log_start":"225'1400","ondisk_log_start":"225'1400","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'5999","last_scrub_stamp":"2024-05-15T12:54:04.170185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:04.170185+0000","objects_scrubbed":5903,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2224,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325124096,"num_objects":4961,"num_object_clones":0,"num_object_copies":14883,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11435,"num_write_kb":524672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"273'11850","reported_seq":12055,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-05-15T12:55:43.553616+0000","last_change":"2024-05-15T12:54:54.253282+0000","last_active":"2024-05-15T12:55:43.553616+0000","last_peered":"2024-05-15T12:55:43.553616+0000","last_clean":"2024-05-15T12:55:43.553616+0000","last_became_active":"2024-05-15T12:52:56.169719+0000","last_became_peered":"2024-05-15T12:52:56.169719+0000","last_unstale":"2024-05-15T12:55:43.553616+0000","last_undegraded":"2024-05-15T12:55:43.553616+0000","last_fullsized":"2024-05-15T12:55:43.553616+0000","mapping_epoch":219,"log_start":"227'1800","ondisk_log_start":"227'1800","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'9231","last_scrub_stamp":"2024-05-15T12:54:54.252859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:54.252859+0000","objects_scrubbed":7684,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:55:55.071670+0000","scrub_duration":2207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327942144,"num_objects":5004,"num_object_clones":0,"num_object_copies":15012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11850,"num_write_kb":539328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"273'11619","reported_seq":11860,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-05-15T12:55:42.744063+0000","last_change":"2024-05-15T12:54:53.922409+0000","last_active":"2024-05-15T12:55:42.744063+0000","last_peered":"2024-05-15T12:55:42.744063+0000","last_clean":"2024-05-15T12:55:42.744063+0000","last_became_active":"2024-05-15T12:52:56.273601+0000","last_became_peered":"2024-05-15T12:52:56.273601+0000","last_unstale":"2024-05-15T12:55:42.744063+0000","last_undegraded":"2024-05-15T12:55:42.744063+0000","last_fullsized":"2024-05-15T12:55:42.744063+0000","mapping_epoch":219,"log_start":"226'1600","ondisk_log_start":"226'1600","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'9107","last_scrub_stamp":"2024-05-15T12:54:53.922303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:53.922303+0000","objects_scrubbed":7674,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:56:04.424915+0000","scrub_duration":2205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333905920,"num_objects":5095,"num_object_clones":0,"num_object_copies":15285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11619,"num_write_kb":534848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"273'11475","reported_seq":11663,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-05-15T12:55:41.078091+0000","last_change":"2024-05-15T12:54:09.050583+0000","last_active":"2024-05-15T12:55:41.078091+0000","last_peered":"2024-05-15T12:55:41.078091+0000","last_clean":"2024-05-15T12:55:41.078091+0000","last_became_active":"2024-05-15T12:52:56.273535+0000","last_became_peered":"2024-05-15T12:52:56.273535+0000","last_unstale":"2024-05-15T12:55:41.078091+0000","last_undegraded":"2024-05-15T12:55:41.078091+0000","last_fullsized":"2024-05-15T12:55:41.078091+0000","mapping_epoch":219,"log_start":"225'1400","ondisk_log_start":"225'1400","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'6434","last_scrub_stamp":"2024-05-15T12:54:09.050363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:09.050363+0000","objects_scrubbed":6370,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1655,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336396288,"num_objects":5133,"num_object_clones":0,"num_object_copies":15399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11475,"num_write_kb":531456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"272'11214","reported_seq":11406,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T12:55:39.433433+0000","last_change":"2024-05-15T12:54:16.516142+0000","last_active":"2024-05-15T12:55:39.433433+0000","last_peered":"2024-05-15T12:55:39.433433+0000","last_clean":"2024-05-15T12:55:39.433433+0000","last_became_active":"2024-05-15T12:52:56.174822+0000","last_became_peered":"2024-05-15T12:52:56.174822+0000","last_unstale":"2024-05-15T12:55:39.433433+0000","last_undegraded":"2024-05-15T12:55:39.433433+0000","last_fullsized":"2024-05-15T12:55:39.433433+0000","mapping_epoch":219,"log_start":"225'1200","ondisk_log_start":"225'1200","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'6783","last_scrub_stamp":"2024-05-15T12:54:16.515960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:16.515960+0000","objects_scrubbed":6740,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:55:43.835195+0000","scrub_duration":1694,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333185024,"num_objects":5084,"num_object_clones":0,"num_object_copies":15252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11214,"num_write_kb":521536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"273'11447","reported_seq":11644,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-05-15T12:55:43.600841+0000","last_change":"2024-05-15T12:54:21.477942+0000","last_active":"2024-05-15T12:55:43.600841+0000","last_peered":"2024-05-15T12:55:43.600841+0000","last_clean":"2024-05-15T12:55:43.600841+0000","last_became_active":"2024-05-15T12:52:56.273621+0000","last_became_peered":"2024-05-15T12:52:56.273621+0000","last_unstale":"2024-05-15T12:55:43.600841+0000","last_undegraded":"2024-05-15T12:55:43.600841+0000","last_fullsized":"2024-05-15T12:55:43.600841+0000","mapping_epoch":219,"log_start":"225'1400","ondisk_log_start":"225'1400","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'7086","last_scrub_stamp":"2024-05-15T12:54:21.477706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:21.477706+0000","objects_scrubbed":7054,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1461,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311885824,"num_objects":4759,"num_object_clones":0,"num_object_copies":14277,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11447,"num_write_kb":518592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10],"acting":[0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"273'11595","reported_seq":11796,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-05-15T12:55:40.324044+0000","last_change":"2024-05-15T12:54:56.371913+0000","last_active":"2024-05-15T12:55:40.324044+0000","last_peered":"2024-05-15T12:55:40.324044+0000","last_clean":"2024-05-15T12:55:40.324044+0000","last_became_active":"2024-05-15T12:52:56.173177+0000","last_became_peered":"2024-05-15T12:52:56.173177+0000","last_unstale":"2024-05-15T12:55:40.324044+0000","last_undegraded":"2024-05-15T12:55:40.324044+0000","last_fullsized":"2024-05-15T12:55:40.324044+0000","mapping_epoch":219,"log_start":"226'1500","ondisk_log_start":"226'1500","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'9339","last_scrub_stamp":"2024-05-15T12:54:56.371625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:56.371625+0000","objects_scrubbed":7499,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:56:15.477068+0000","scrub_duration":1588,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340852736,"num_objects":5201,"num_object_clones":0,"num_object_copies":15603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":537472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,1],"acting":[9,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2","version":"273'11764","reported_seq":11968,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-05-15T12:55:42.163743+0000","last_change":"2024-05-15T12:54:52.113647+0000","last_active":"2024-05-15T12:55:42.163743+0000","last_peered":"2024-05-15T12:55:42.163743+0000","last_clean":"2024-05-15T12:55:42.163743+0000","last_became_active":"2024-05-15T12:52:56.173445+0000","last_became_peered":"2024-05-15T12:52:56.173445+0000","last_unstale":"2024-05-15T12:55:42.163743+0000","last_undegraded":"2024-05-15T12:55:42.163743+0000","last_fullsized":"2024-05-15T12:55:42.163743+0000","mapping_epoch":219,"log_start":"226'1700","ondisk_log_start":"226'1700","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'9122","last_scrub_stamp":"2024-05-15T12:54:52.113308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:52:55.160452+0000","last_clean_scrub_stamp":"2024-05-15T12:54:52.113308+0000","objects_scrubbed":7745,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:56:05.645218+0000","scrub_duration":2293,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330301440,"num_objects":5040,"num_object_clones":0,"num_object_copies":15120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11764,"num_write_kb":537728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":770,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-05-15T12:55:39.719587+0000","last_change":"2024-05-15T12:55:06.370818+0000","last_active":"2024-05-15T12:55:39.719587+0000","last_peered":"2024-05-15T12:55:39.719587+0000","last_clean":"2024-05-15T12:55:39.719587+0000","last_became_active":"2024-05-15T12:49:08.168334+0000","last_became_peered":"2024-05-15T12:49:08.168334+0000","last_unstale":"2024-05-15T12:55:39.719587+0000","last_undegraded":"2024-05-15T12:55:39.719587+0000","last_fullsized":"2024-05-15T12:55:39.719587+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T12:55:06.370673+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T12:55:06.370673+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:56:30.073725+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":5334433816,"num_objects":81398,"num_object_clones":0,"num_object_copies":244194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81398,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185842,"num_write_kb":8551617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15995580416,"data_stored":15995568200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160842,"ondisk_log_size":160842,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":0,"seq":386547056763,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351103,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":464992,"kb_used_data":311636,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93280160,"statfs":{"total":95995035648,"available":95518883840,"internally_reserved":0,"allocated":319115264,"data_stored":318857864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":157022656},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514628,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2252336,"kb_used_data":1996256,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":91492816,"statfs":{"total":95995035648,"available":93688643584,"internally_reserved":0,"allocated":2044166144,"data_stored":2043901120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":262207936},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874378,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":843616,"kb_used_data":663892,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92901536,"statfs":{"total":95995035648,"available":95131172864,"internally_reserved":0,"allocated":679825408,"data_stored":679568028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168717,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1497776,"kb_used_data":1313888,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92247376,"statfs":{"total":95995035648,"available":94461313024,"internally_reserved":0,"allocated":1345421312,"data_stored":1345156308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":188283338},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430355,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1752548,"kb_used_data":1608924,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":91992604,"statfs":{"total":95995035648,"available":94200426496,"internally_reserved":0,"allocated":1647538176,"data_stored":1647277244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":147061190},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575022,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1827488,"kb_used_data":1656020,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":91917664,"statfs":{"total":95995035648,"available":94123687936,"internally_reserved":0,"allocated":1695764480,"data_stored":1695507100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738550,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2160668,"kb_used_data":1912916,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":91584484,"statfs":{"total":95995035648,"available":93782511616,"internally_reserved":0,"allocated":1958825984,"data_stored":1958568604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411499,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":798492,"kb_used_data":650708,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92946660,"statfs":{"total":95995035648,"available":95177379840,"internally_reserved":0,"allocated":666324992,"data_stored":666067612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1915100,"kb_used_data":1631124,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":91830052,"statfs":{"total":95995035648,"available":94033973248,"internally_reserved":0,"allocated":1670270976,"data_stored":1670013596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":290650567},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986334,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2515924,"kb_used_data":2336136,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":91229228,"statfs":{"total":95995035648,"available":93418729472,"internally_reserved":0,"allocated":2392203264,"data_stored":2391950480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691993,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1788704,"kb_used_data":1631192,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":91956448,"statfs":{"total":95995035648,"available":94163402752,"internally_reserved":0,"allocated":1670340608,"data_stored":1670079156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1946222592,"data_stored":1946222592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1687224320,"data_stored":1687224320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":664403968,"data_stored":664403968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1661796352,"data_stored":1661796352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2376531968,"data_stored":2376531968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1654198272,"data_stored":1654194200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1641414656,"data_stored":1641414656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1338970112,"data_stored":1338966040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":675938304,"data_stored":675938304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2030637056,"data_stored":2030632984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":318242816,"data_stored":318242816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T12:55:46.831 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-05-15T12:55:46.831 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T12:55:46.832 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-05-15T12:55:46.832 INFO:tasks.thrashosds.thrasher:Setting osd 9 primary_affinity to 0.906214 2024-05-15T12:55:46.832 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd primary-affinity 9 0.9062144341189233 2024-05-15T12:55:46.843 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:55:46.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:46.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:55:46.948 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:55:47.006 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:47.021 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:55:47.056 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:55:47.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:47.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:55:47.115 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:47 smithi027 ceph-mon[39795]: 3.9 scrub ok 2024-05-15T12:55:47.116 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:47 smithi027 ceph-mon[39795]: 3.3 scrub ok 2024-05-15T12:55:47.116 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:47 smithi027 ceph-mon[39795]: 3.f scrub starts 2024-05-15T12:55:47.166 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:55:47.206 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:55:47.261 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:55:47.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:47.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:55:47.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:47 smithi120 ceph-mon[43961]: 3.9 scrub ok 2024-05-15T12:55:47.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:47 smithi120 ceph-mon[43961]: 3.3 scrub ok 2024-05-15T12:55:47.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:47 smithi120 ceph-mon[43961]: 3.f scrub starts 2024-05-15T12:55:47.368 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:55:47.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:47 smithi177 ceph-mon[44303]: 3.9 scrub ok 2024-05-15T12:55:47.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:47 smithi177 ceph-mon[44303]: 3.3 scrub ok 2024-05-15T12:55:47.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:47 smithi177 ceph-mon[44303]: 3.f scrub starts 2024-05-15T12:55:47.402 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:47.533 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:55:47.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:47.535 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:55:47.636 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:55:47.689 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:48.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:48 smithi027 ceph-mon[39795]: 3.d scrub starts 2024-05-15T12:55:48.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:48 smithi027 ceph-mon[39795]: 3.f scrub ok 2024-05-15T12:55:48.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:48 smithi027 ceph-mon[39795]: pgmap v757: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.8 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 730 op/s 2024-05-15T12:55:48.364 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:48 smithi177 ceph-mon[44303]: 3.d scrub starts 2024-05-15T12:55:48.364 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:48 smithi177 ceph-mon[44303]: 3.f scrub ok 2024-05-15T12:55:48.364 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:48 smithi177 ceph-mon[44303]: pgmap v757: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.8 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 730 op/s 2024-05-15T12:55:48.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:48 smithi120 ceph-mon[43961]: 3.d scrub starts 2024-05-15T12:55:48.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:48 smithi120 ceph-mon[43961]: 3.f scrub ok 2024-05-15T12:55:48.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:48 smithi120 ceph-mon[43961]: pgmap v757: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.8 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 730 op/s 2024-05-15T12:55:48.853 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:55:48.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:48.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:55:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:55:48.959 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:55:48.973 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T12:55:48.973 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:55:49.024 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:49.064 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:55:49.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:49.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:55:49.153 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T12:55:49.187 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:55:49.220 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:49.241 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:55:49.246 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:49 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2146649875' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:55:49.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:49.247 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:55:49.348 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:55:49.431 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:49.431 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:49 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2146649875' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:55:49.457 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:55:49.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:49.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:55:49.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:49 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2146649875' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:55:49.567 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:55:49.623 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:49.663 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:55:49.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:49.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:55:49.730 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-05-15T12:55:49.775 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:55:49.816 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:50.033 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:55:50.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:50 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:55:50.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:50 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:55:50.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:50 smithi027 ceph-mon[39795]: 3.d scrub ok 2024-05-15T12:55:50.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:50 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:55:50.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:50 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T12:55:50.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:50 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2146649875' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:55:50.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:50 smithi027 ceph-mon[39795]: osdmap e274: 12 total, 12 up, 11 in 2024-05-15T12:55:50.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:50 smithi027 ceph-mon[39795]: pgmap v759: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 773 op/s 2024-05-15T12:55:50.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:50 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:55:50.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:50 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:55:50.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:50 smithi177 ceph-mon[44303]: 3.d scrub ok 2024-05-15T12:55:50.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:50 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:55:50.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:50 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T12:55:50.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:50 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2146649875' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:55:50.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:50 smithi177 ceph-mon[44303]: osdmap e274: 12 total, 12 up, 11 in 2024-05-15T12:55:50.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:50 smithi177 ceph-mon[44303]: pgmap v759: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 773 op/s 2024-05-15T12:55:50.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:50 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:55:50.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:50 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:55:50.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:50 smithi120 ceph-mon[43961]: 3.d scrub ok 2024-05-15T12:55:50.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:50 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:55:50.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:50 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T12:55:50.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:50 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2146649875' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:55:50.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:50 smithi120 ceph-mon[43961]: osdmap e274: 12 total, 12 up, 11 in 2024-05-15T12:55:50.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:50 smithi120 ceph-mon[43961]: pgmap v759: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 773 op/s 2024-05-15T12:55:51.188 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:55:51.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:51.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:55:51.298 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:55:51.335 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:51.363 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:55:51.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:51.369 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:55:51.470 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:55:51.521 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:51.546 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:55:51.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:51.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:55:51.655 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:55:51.717 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:51.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:51.770 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:55:51.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:55:51.871 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:55:51.904 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:51 smithi177 ceph-mon[44303]: pgmap v760: 17 pgs: 17 active+clean; 4.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 772 op/s 2024-05-15T12:55:51.906 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:51.937 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:55:51.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:51.939 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:55:52.041 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:55:52.070 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:52.083 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T12:55:52.083 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:55:52.094 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:55:52.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:52.097 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:55:52.122 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:51 smithi027 ceph-mon[39795]: pgmap v760: 17 pgs: 17 active+clean; 4.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 772 op/s 2024-05-15T12:55:52.198 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:55:52.247 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:51 smithi120 ceph-mon[43961]: pgmap v760: 17 pgs: 17 active+clean; 4.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 772 op/s 2024-05-15T12:55:52.260 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:52.296 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:55:52.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:52.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:55:52.305 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:55:52.305 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:55:52.305 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:55:52.305 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:55:52.305 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:55:52.305 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:55:52.305 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:55:52.305 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:55:52.305 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:55:52.305 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:55:52.305 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:55:52.347 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:55:52.355 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:55:52.355 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:55:52.355 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:55:52.355 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:55:52.355 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:55:52.355 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:55:52.355 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:55:52.355 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:55:52.355 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:55:52.355 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:55:52.355 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:55:52.356 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:55:52.401 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:55:52.430 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:52.783 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:55:52.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:52.787 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:55:52.813 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-05-15T12:55:52.888 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:55:52.940 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:52 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3460400381' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:55:52.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:52 smithi027 ceph-mon[39795]: 3.a scrub starts 2024-05-15T12:55:52.950 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:53.155 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:55:53.178 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:55:53.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:52 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3460400381' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:55:53.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:52 smithi177 ceph-mon[44303]: 3.a scrub starts 2024-05-15T12:55:53.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:53.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:55:53.225 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T12:55:53.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:52 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3460400381' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:55:53.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:52 smithi120 ceph-mon[43961]: 3.a scrub starts 2024-05-15T12:55:53.285 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:55:53.336 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:53.388 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:55:53.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:53.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:55:53.495 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:55:53.526 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:53.982 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:55:53.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:53.985 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:55:54.080 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.9 primary-affinity to 0.906214 (8593892) 2024-05-15T12:55:54.086 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:55:54.157 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:54.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:54 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3425121505' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 0.9062144341189233}]: dispatch 2024-05-15T12:55:54.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:54 smithi177 ceph-mon[44303]: pgmap v761: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 781 op/s 2024-05-15T12:55:54.435 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:55:54.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:54.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:54 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3425121505' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 0.9062144341189233}]: dispatch 2024-05-15T12:55:54.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:54 smithi027 ceph-mon[39795]: pgmap v761: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 781 op/s 2024-05-15T12:55:54.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:55:54.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:54 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3425121505' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 0.9062144341189233}]: dispatch 2024-05-15T12:55:54.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:54 smithi120 ceph-mon[43961]: pgmap v761: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 781 op/s 2024-05-15T12:55:54.545 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:55:54.595 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:55.066 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T12:55:55.066 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-05-15T12:55:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-05-15T12:55:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T12:55:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:119223 2.4 e153/144 2:2892d1d8:::benchmark_data_smithi177_78104_object12090:head v 153'7383, mlcod=153'7369)", 2024-05-15T12:55:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:10.624086+0000", 2024-05-15T12:55:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 344.45069677999999, 2024-05-15T12:55:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028317624999999999, 2024-05-15T12:55:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:55.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624086+0000", 2024-05-15T12:55:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624086+0000", 2024-05-15T12:55:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624086+0000", 2024-05-15T12:55:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8699999999999999e-07 2024-05-15T12:55:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624089+0000", 2024-05-15T12:55:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2219999999999998e-06 2024-05-15T12:55:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624095+0000", 2024-05-15T12:55:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0000000000000002e-06 2024-05-15T12:55:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624099+0000", 2024-05-15T12:55:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9840000000000003e-06 2024-05-15T12:55:55.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.643649+0000", 2024-05-15T12:55:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.019549818 2024-05-15T12:55:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.643672+0000", 2024-05-15T12:55:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2259000000000002e-05 2024-05-15T12:55:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:55.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652379+0000", 2024-05-15T12:55:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0087074639999999998 2024-05-15T12:55:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652404+0000", 2024-05-15T12:55:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4691000000000001e-05 2024-05-15T12:55:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:119236 2.4 e153/144 2:222c6e22:::benchmark_data_smithi177_78104_object12103:head v 153'7384, mlcod=153'7369)", 2024-05-15T12:55:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:10.624106+0000", 2024-05-15T12:55:55.071 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 344.45067708900001, 2024-05-15T12:55:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028323482000000001, 2024-05-15T12:55:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624106+0000", 2024-05-15T12:55:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:55.072 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624106+0000", 2024-05-15T12:55:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624106+0000", 2024-05-15T12:55:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5200000000000001e-07 2024-05-15T12:55:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624109+0000", 2024-05-15T12:55:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2540000000000001e-06 2024-05-15T12:55:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624115+0000", 2024-05-15T12:55:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1829999999999999e-06 2024-05-15T12:55:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624120+0000", 2024-05-15T12:55:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1670000000000001e-06 2024-05-15T12:55:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.645610+0000", 2024-05-15T12:55:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021490510000000001 2024-05-15T12:55:55.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.645624+0000", 2024-05-15T12:55:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4372999999999999e-05 2024-05-15T12:55:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652406+0000", 2024-05-15T12:55:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0067820729999999996 2024-05-15T12:55:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:55.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652429+0000", 2024-05-15T12:55:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2770000000000001e-05 2024-05-15T12:55:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:119240 2.4 e153/144 2:2ed0c2c5:::benchmark_data_smithi177_78104_object12107:head v 153'7385, mlcod=153'7369)", 2024-05-15T12:55:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:10.624126+0000", 2024-05-15T12:55:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 344.45065709900001, 2024-05-15T12:55:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028331313, 2024-05-15T12:55:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:55.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624126+0000", 2024-05-15T12:55:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624126+0000", 2024-05-15T12:55:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624126+0000", 2024-05-15T12:55:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.97e-07 2024-05-15T12:55:55.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624132+0000", 2024-05-15T12:55:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0939999999999996e-06 2024-05-15T12:55:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624137+0000", 2024-05-15T12:55:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.417e-06 2024-05-15T12:55:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:55.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624142+0000", 2024-05-15T12:55:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0679999999999998e-06 2024-05-15T12:55:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.647795+0000", 2024-05-15T12:55:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.023653422 2024-05-15T12:55:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.647819+0000", 2024-05-15T12:55:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3946000000000001e-05 2024-05-15T12:55:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:55.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652435+0000", 2024-05-15T12:55:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004615638 2024-05-15T12:55:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652457+0000", 2024-05-15T12:55:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2531000000000002e-05 2024-05-15T12:55:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:119259 2.4 e153/144 2:21f9b17e:::benchmark_data_smithi177_78104_object12126:head v 153'7386, mlcod=153'7369)", 2024-05-15T12:55:55.080 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:10.624148+0000", 2024-05-15T12:55:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 344.450635207, 2024-05-15T12:55:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028337528000000001, 2024-05-15T12:55:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624148+0000", 2024-05-15T12:55:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624148+0000", 2024-05-15T12:55:55.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624148+0000", 2024-05-15T12:55:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.98e-07 2024-05-15T12:55:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624151+0000", 2024-05-15T12:55:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2270000000000002e-06 2024-05-15T12:55:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624157+0000", 2024-05-15T12:55:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9379999999999997e-06 2024-05-15T12:55:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624161+0000", 2024-05-15T12:55:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9249999999999997e-06 2024-05-15T12:55:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.650016+0000", 2024-05-15T12:55:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.025854855 2024-05-15T12:55:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.650040+0000", 2024-05-15T12:55:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4402000000000001e-05 2024-05-15T12:55:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652461+0000", 2024-05-15T12:55:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024205590000000001 2024-05-15T12:55:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652485+0000", 2024-05-15T12:55:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4423999999999999e-05 2024-05-15T12:55:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(2.4 153/144 [PushOp(2:25f9c844:::benchmark_data_smithi177_78104_object18168:head, version: 115'1116, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:25f9c844:::benchmark_data_smithi177_78104_object18168:head@115'1116, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-05-15T12:55:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:10.624171+0000", 2024-05-15T12:55:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 344.45061156499997, 2024-05-15T12:55:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028502006999999999, 2024-05-15T12:55:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T12:55:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624171+0000", 2024-05-15T12:55:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624171+0000", 2024-05-15T12:55:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624171+0000", 2024-05-15T12:55:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3999999999999995e-08 2024-05-15T12:55:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624204+0000", 2024-05-15T12:55:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2954999999999999e-05 2024-05-15T12:55:55.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624218+0000", 2024-05-15T12:55:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3172e-05 2024-05-15T12:55:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.624227+0000", 2024-05-15T12:55:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4890000000000001e-06 2024-05-15T12:55:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652529+0000", 2024-05-15T12:55:55.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028302054 2024-05-15T12:55:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652538+0000", 2024-05-15T12:55:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9260000000000006e-06 2024-05-15T12:55:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:10.652673+0000", 2024-05-15T12:55:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013531700000000001 2024-05-15T12:55:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:157300 2.0 e180/144 2:061be614:::benchmark_data_smithi177_78104_object50167:head v 180'9909, mlcod=180'9903)", 2024-05-15T12:55:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:59.308914+0000", 2024-05-15T12:55:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 295.76586882100003, 2024-05-15T12:55:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.031795017000000002, 2024-05-15T12:55:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308914+0000", 2024-05-15T12:55:55.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308914+0000", 2024-05-15T12:55:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308914+0000", 2024-05-15T12:55:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.03e-07 2024-05-15T12:55:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308923+0000", 2024-05-15T12:55:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7080000000000003e-06 2024-05-15T12:55:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308929+0000", 2024-05-15T12:55:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0109999999999997e-06 2024-05-15T12:55:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308935+0000", 2024-05-15T12:55:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9159999999999998e-06 2024-05-15T12:55:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.322985+0000", 2024-05-15T12:55:55.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014050104000000001 2024-05-15T12:55:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.323000+0000", 2024-05-15T12:55:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4973e-05 2024-05-15T12:55:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.340664+0000", 2024-05-15T12:55:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017664418000000001 2024-05-15T12:55:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.340709+0000", 2024-05-15T12:55:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4684000000000001e-05 2024-05-15T12:55:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:157344 2.0 e180/144 2:025f35be:::benchmark_data_smithi177_78104_object50211:head v 180'9910, mlcod=180'9903)", 2024-05-15T12:55:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:59.308941+0000", 2024-05-15T12:55:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 295.76584232099998, 2024-05-15T12:55:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.035675354999999999, 2024-05-15T12:55:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:55.095 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308941+0000", 2024-05-15T12:55:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308941+0000", 2024-05-15T12:55:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308941+0000", 2024-05-15T12:55:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.98e-07 2024-05-15T12:55:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308944+0000", 2024-05-15T12:55:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5190000000000001e-06 2024-05-15T12:55:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308950+0000", 2024-05-15T12:55:55.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5709999999999997e-06 2024-05-15T12:55:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308955+0000", 2024-05-15T12:55:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8230000000000002e-06 2024-05-15T12:55:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.326782+0000", 2024-05-15T12:55:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.01782688 2024-05-15T12:55:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.326806+0000", 2024-05-15T12:55:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4964e-05 2024-05-15T12:55:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.344552+0000", 2024-05-15T12:55:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017745133999999999 2024-05-15T12:55:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.344616+0000", 2024-05-15T12:55:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4265999999999998e-05 2024-05-15T12:55:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.099 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:157358 2.0 e180/144 2:0b657229:::benchmark_data_smithi177_78104_object50225:head v 180'9911, mlcod=180'9903)", 2024-05-15T12:55:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:59.308961+0000", 2024-05-15T12:55:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 295.76582169900001, 2024-05-15T12:55:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.035687647000000003, 2024-05-15T12:55:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:55.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308961+0000", 2024-05-15T12:55:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308961+0000", 2024-05-15T12:55:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308961+0000", 2024-05-15T12:55:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.55e-07 2024-05-15T12:55:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308965+0000", 2024-05-15T12:55:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3840000000000001e-06 2024-05-15T12:55:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308970+0000", 2024-05-15T12:55:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.536e-06 2024-05-15T12:55:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308975+0000", 2024-05-15T12:55:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.454e-06 2024-05-15T12:55:55.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.330632+0000", 2024-05-15T12:55:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021657725999999999 2024-05-15T12:55:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.330655+0000", 2024-05-15T12:55:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.279e-05 2024-05-15T12:55:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:55.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.344620+0000", 2024-05-15T12:55:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013964682000000001 2024-05-15T12:55:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.344649+0000", 2024-05-15T12:55:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8920000000000001e-05 2024-05-15T12:55:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:157360 2.0 e180/144 2:03d63169:::benchmark_data_smithi177_78104_object50227:head v 180'9912, mlcod=180'9903)", 2024-05-15T12:55:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:59.308981+0000", 2024-05-15T12:55:55.104 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 295.76580187299999, 2024-05-15T12:55:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.035698306999999999, 2024-05-15T12:55:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308981+0000", 2024-05-15T12:55:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308981+0000", 2024-05-15T12:55:55.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308981+0000", 2024-05-15T12:55:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0200000000000001e-07 2024-05-15T12:55:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308988+0000", 2024-05-15T12:55:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1960000000000002e-06 2024-05-15T12:55:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:55.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.308995+0000", 2024-05-15T12:55:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7649999999999996e-06 2024-05-15T12:55:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.309000+0000", 2024-05-15T12:55:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7040000000000002e-06 2024-05-15T12:55:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.333688+0000", 2024-05-15T12:55:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.024688627000000001 2024-05-15T12:55:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.333711+0000", 2024-05-15T12:55:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2520000000000001e-05 2024-05-15T12:55:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.344651+0000", 2024-05-15T12:55:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010940340999999999 2024-05-15T12:55:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.344679+0000", 2024-05-15T12:55:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7951999999999999e-05 2024-05-15T12:55:55.108 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:157385 2.0 e180/144 2:0bdb78a8:::benchmark_data_smithi177_78104_object50252:head v 180'9913, mlcod=180'9903)", 2024-05-15T12:55:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:59.309006+0000", 2024-05-15T12:55:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 295.76577658299999, 2024-05-15T12:55:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.035703602000000001, 2024-05-15T12:55:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:55.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.309006+0000", 2024-05-15T12:55:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.309006+0000", 2024-05-15T12:55:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.309006+0000", 2024-05-15T12:55:55.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8699999999999999e-07 2024-05-15T12:55:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.309010+0000", 2024-05-15T12:55:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2100000000000002e-06 2024-05-15T12:55:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.309016+0000", 2024-05-15T12:55:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6710000000000002e-06 2024-05-15T12:55:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.309021+0000", 2024-05-15T12:55:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6040000000000003e-06 2024-05-15T12:55:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.337608+0000", 2024-05-15T12:55:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028586711000000001 2024-05-15T12:55:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.337632+0000", 2024-05-15T12:55:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4108e-05 2024-05-15T12:55:55.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.344683+0000", 2024-05-15T12:55:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0070516449999999996 2024-05-15T12:55:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.344710+0000", 2024-05-15T12:55:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6466e-05 2024-05-15T12:55:55.113 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(2.0 180/144 [PushOp(2:01179965:::benchmark_data_smithi177_78104_object56722:head, version: 127'3544, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:01179965:::benchmark_data_smithi177_78104_object56722:head@127'3544, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-05-15T12:55:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:50:59.312898+0000", 2024-05-15T12:55:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 295.761884452, 2024-05-15T12:55:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.032109265999999997, 2024-05-15T12:55:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T12:55:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.312898+0000", 2024-05-15T12:55:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.312898+0000", 2024-05-15T12:55:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.312899+0000", 2024-05-15T12:55:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.16e-07 2024-05-15T12:55:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.313019+0000", 2024-05-15T12:55:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000119511 2024-05-15T12:55:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.313041+0000", 2024-05-15T12:55:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2566999999999999e-05 2024-05-15T12:55:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.313057+0000", 2024-05-15T12:55:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5845000000000002e-05 2024-05-15T12:55:55.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.344767+0000", 2024-05-15T12:55:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.031709933000000003 2024-05-15T12:55:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.344781+0000", 2024-05-15T12:55:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4307999999999999e-05 2024-05-15T12:55:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:50:59.345008+0000", 2024-05-15T12:55:55.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022628599999999999 2024-05-15T12:55:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:185550 2.4 e201/173 2:28ea92eb:::benchmark_data_smithi177_78104_object78417:head v 201'11545, mlcod=201'11541)", 2024-05-15T12:55:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:51:48.390480+0000", 2024-05-15T12:55:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 246.68430284300001, 2024-05-15T12:55:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028378520000000001, 2024-05-15T12:55:55.118 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:48.390480+0000", 2024-05-15T12:55:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:48.390480+0000", 2024-05-15T12:55:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:48.390481+0000", 2024-05-15T12:55:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.421e-06 2024-05-15T12:55:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:48.390492+0000", 2024-05-15T12:55:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0907000000000001e-05 2024-05-15T12:55:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:55.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:48.390506+0000", 2024-05-15T12:55:55.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3339999999999999e-05 2024-05-15T12:55:55.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:55.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:48.390516+0000", 2024-05-15T12:55:55.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.995e-06 2024-05-15T12:55:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:48.395450+0000", 2024-05-15T12:55:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004934473 2024-05-15T12:55:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:48.395470+0000", 2024-05-15T12:55:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9831e-05 2024-05-15T12:55:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:55.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:48.418814+0000", 2024-05-15T12:55:55.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.023343605999999999 2024-05-15T12:55:55.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:55.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:48.418859+0000", 2024-05-15T12:55:55.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4947000000000002e-05 2024-05-15T12:55:55.123 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.123 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:55.123 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:185854 2.0 e201/190 2:0d45165c:::benchmark_data_smithi177_78104_object78721:head v 201'11722, mlcod=201'11717)", 2024-05-15T12:55:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:51:48.711609+0000", 2024-05-15T12:55:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 246.36317353499999, 2024-05-15T12:55:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.029987149000000001, 2024-05-15T12:55:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:48.711609+0000", 2024-05-15T12:55:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:55.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:48.711609+0000", 2024-05-15T12:55:55.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:55.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:48.711610+0000", 2024-05-15T12:55:55.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8800000000000003e-07 2024-05-15T12:55:55.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:48.711615+0000", 2024-05-15T12:55:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8760000000000004e-06 2024-05-15T12:55:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:48.711622+0000", 2024-05-15T12:55:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7030000000000006e-06 2024-05-15T12:55:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:48.711629+0000", 2024-05-15T12:55:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3149999999999997e-06 2024-05-15T12:55:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:48.717493+0000", 2024-05-15T12:55:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0058643929999999999 2024-05-15T12:55:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:48.717512+0000", 2024-05-15T12:55:55.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8865999999999999e-05 2024-05-15T12:55:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:48.741547+0000", 2024-05-15T12:55:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.024035451999999999 2024-05-15T12:55:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:51:48.741596+0000", 2024-05-15T12:55:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9055999999999999e-05 2024-05-15T12:55:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:207270 2.4 e208/173 2:20cb6b62:::benchmark_data_smithi177_78104_object100137:head v 208'12869, mlcod=208'12863)", 2024-05-15T12:55:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:52:13.692083+0000", 2024-05-15T12:55:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 221.38270015800001, 2024-05-15T12:55:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028925467999999999, 2024-05-15T12:55:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692083+0000", 2024-05-15T12:55:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:55.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692083+0000", 2024-05-15T12:55:55.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:55.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692083+0000", 2024-05-15T12:55:55.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9000000000000001e-07 2024-05-15T12:55:55.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692086+0000", 2024-05-15T12:55:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.791e-06 2024-05-15T12:55:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692092+0000", 2024-05-15T12:55:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.981e-06 2024-05-15T12:55:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692096+0000", 2024-05-15T12:55:55.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8380000000000004e-06 2024-05-15T12:55:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.706170+0000", 2024-05-15T12:55:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014074769000000001 2024-05-15T12:55:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.706193+0000", 2024-05-15T12:55:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2323e-05 2024-05-15T12:55:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:55.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.720969+0000", 2024-05-15T12:55:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014776529 2024-05-15T12:55:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.721008+0000", 2024-05-15T12:55:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9047000000000001e-05 2024-05-15T12:55:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:207290 2.4 e208/173 2:2147ec8c:::benchmark_data_smithi177_78104_object100157:head v 208'12870, mlcod=208'12863)", 2024-05-15T12:55:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:52:13.692100+0000", 2024-05-15T12:55:55.133 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 221.38268243799999, 2024-05-15T12:55:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028943515999999999, 2024-05-15T12:55:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692100+0000", 2024-05-15T12:55:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692100+0000", 2024-05-15T12:55:55.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:55.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692101+0000", 2024-05-15T12:55:55.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5200000000000001e-07 2024-05-15T12:55:55.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:55.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692106+0000", 2024-05-15T12:55:55.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.237e-06 2024-05-15T12:55:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692118+0000", 2024-05-15T12:55:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.234e-05 2024-05-15T12:55:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692122+0000", 2024-05-15T12:55:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1540000000000004e-06 2024-05-15T12:55:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.709506+0000", 2024-05-15T12:55:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017384001999999999 2024-05-15T12:55:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.709530+0000", 2024-05-15T12:55:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4156999999999999e-05 2024-05-15T12:55:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.721012+0000", 2024-05-15T12:55:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011481158999999999 2024-05-15T12:55:55.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.721044+0000", 2024-05-15T12:55:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2314999999999998e-05 2024-05-15T12:55:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:207292 2.4 e208/173 2:2783f900:::benchmark_data_smithi177_78104_object100159:head v 208'12871, mlcod=208'12863)", 2024-05-15T12:55:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:52:13.692129+0000", 2024-05-15T12:55:55.138 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 221.382654016, 2024-05-15T12:55:55.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.035840193999999999, 2024-05-15T12:55:55.139 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:55.139 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:55.139 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:55.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:55.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692129+0000", 2024-05-15T12:55:55.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692129+0000", 2024-05-15T12:55:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692129+0000", 2024-05-15T12:55:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1400000000000001e-07 2024-05-15T12:55:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692145+0000", 2024-05-15T12:55:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6336000000000001e-05 2024-05-15T12:55:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:55.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692151+0000", 2024-05-15T12:55:55.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0599999999999996e-06 2024-05-15T12:55:55.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:55.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692155+0000", 2024-05-15T12:55:55.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9689999999999996e-06 2024-05-15T12:55:55.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.712708+0000", 2024-05-15T12:55:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.020552369000000001 2024-05-15T12:55:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.712721+0000", 2024-05-15T12:55:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2830000000000001e-05 2024-05-15T12:55:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.727941+0000", 2024-05-15T12:55:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015220529 2024-05-15T12:55:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.727969+0000", 2024-05-15T12:55:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7886999999999999e-05 2024-05-15T12:55:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:207295 2.4 e208/173 2:277dd002:::benchmark_data_smithi177_78104_object100162:head v 208'12872, mlcod=208'12863)", 2024-05-15T12:55:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:52:13.692161+0000", 2024-05-15T12:55:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 221.382622129, 2024-05-15T12:55:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.035826047, 2024-05-15T12:55:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692161+0000", 2024-05-15T12:55:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:55.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692161+0000", 2024-05-15T12:55:55.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:55.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692161+0000", 2024-05-15T12:55:55.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.72e-07 2024-05-15T12:55:55.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692167+0000", 2024-05-15T12:55:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2260000000000002e-06 2024-05-15T12:55:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692172+0000", 2024-05-15T12:55:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9830000000000004e-06 2024-05-15T12:55:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692176+0000", 2024-05-15T12:55:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.67e-06 2024-05-15T12:55:55.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.715642+0000", 2024-05-15T12:55:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.023466629999999999 2024-05-15T12:55:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.715660+0000", 2024-05-15T12:55:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7189000000000001e-05 2024-05-15T12:55:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:55.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.727972+0000", 2024-05-15T12:55:55.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012312236000000001 2024-05-15T12:55:55.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:55.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.727987+0000", 2024-05-15T12:55:55.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4941e-05 2024-05-15T12:55:55.148 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.148 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:55.148 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.148 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:207341 2.4 e208/173 2:25f9374d:::benchmark_data_smithi177_78104_object100208:head v 208'12873, mlcod=208'12863)", 2024-05-15T12:55:55.148 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:52:13.692180+0000", 2024-05-15T12:55:55.148 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 221.38260236900001, 2024-05-15T12:55:55.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.041290383, 2024-05-15T12:55:55.149 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:55.149 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:55.149 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:55.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:55.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692180+0000", 2024-05-15T12:55:55.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:55.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692180+0000", 2024-05-15T12:55:55.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:55.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692181+0000", 2024-05-15T12:55:55.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.66e-07 2024-05-15T12:55:55.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:55.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692183+0000", 2024-05-15T12:55:55.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5409999999999999e-06 2024-05-15T12:55:55.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:55.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692189+0000", 2024-05-15T12:55:55.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7629999999999997e-06 2024-05-15T12:55:55.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:55.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692193+0000", 2024-05-15T12:55:55.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.602e-06 2024-05-15T12:55:55.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:55.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.718428+0000", 2024-05-15T12:55:55.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.026235764000000002 2024-05-15T12:55:55.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:55.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.718449+0000", 2024-05-15T12:55:55.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0846e-05 2024-05-15T12:55:55.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:55.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.733439+0000", 2024-05-15T12:55:55.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014989715000000001 2024-05-15T12:55:55.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:55.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.733471+0000", 2024-05-15T12:55:55.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1986000000000002e-05 2024-05-15T12:55:55.152 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.153 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:55.153 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.153 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:207343 2.4 e208/173 2:2ac4541f:::benchmark_data_smithi177_78104_object100210:head v 208'12874, mlcod=208'12863)", 2024-05-15T12:55:55.153 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:52:13.692197+0000", 2024-05-15T12:55:55.153 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 221.38258547000001, 2024-05-15T12:55:55.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.041300275999999997, 2024-05-15T12:55:55.153 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:55.153 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:55.153 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:55.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:55.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692197+0000", 2024-05-15T12:55:55.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:55.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692197+0000", 2024-05-15T12:55:55.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:55.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692198+0000", 2024-05-15T12:55:55.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7700000000000001e-07 2024-05-15T12:55:55.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:55.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692200+0000", 2024-05-15T12:55:55.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5790000000000002e-06 2024-05-15T12:55:55.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:55.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692206+0000", 2024-05-15T12:55:55.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.665e-06 2024-05-15T12:55:55.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:55.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692209+0000", 2024-05-15T12:55:55.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.67e-06 2024-05-15T12:55:55.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:55.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.721081+0000", 2024-05-15T12:55:55.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028871206999999999 2024-05-15T12:55:55.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:55.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.721103+0000", 2024-05-15T12:55:55.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2319000000000001e-05 2024-05-15T12:55:55.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:55.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.733474+0000", 2024-05-15T12:55:55.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012371147000000001 2024-05-15T12:55:55.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:55.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.733498+0000", 2024-05-15T12:55:55.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3512000000000001e-05 2024-05-15T12:55:55.157 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.157 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:55.157 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.158 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.24562.0:207360 2.4 e208/173 2:2db4f081:::benchmark_data_smithi177_78104_object100227:head v 208'12875, mlcod=208'12863)", 2024-05-15T12:55:55.158 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:52:13.692214+0000", 2024-05-15T12:55:55.158 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 221.38256879100001, 2024-05-15T12:55:55.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.041308655999999999, 2024-05-15T12:55:55.158 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:55:55.158 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:55:55.158 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:55:55.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:55:55.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692214+0000", 2024-05-15T12:55:55.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:55:55.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692214+0000", 2024-05-15T12:55:55.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:55:55.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:55:55.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692214+0000", 2024-05-15T12:55:55.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5599999999999999e-07 2024-05-15T12:55:55.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:55:55.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692219+0000", 2024-05-15T12:55:55.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7469999999999996e-06 2024-05-15T12:55:55.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:55:55.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692225+0000", 2024-05-15T12:55:55.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7849999999999997e-06 2024-05-15T12:55:55.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:55:55.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.692229+0000", 2024-05-15T12:55:55.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8489999999999999e-06 2024-05-15T12:55:55.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:55:55.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.723420+0000", 2024-05-15T12:55:55.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.031191872999999998 2024-05-15T12:55:55.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:55:55.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.723450+0000", 2024-05-15T12:55:55.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9158000000000001e-05 2024-05-15T12:55:55.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T12:55:55.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.733500+0000", 2024-05-15T12:55:55.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010050085 2024-05-15T12:55:55.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:55:55.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:55:55.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T12:55:55.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:52:13.733523+0000", 2024-05-15T12:55:55.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3003e-05 2024-05-15T12:55:55.162 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.162 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:55.162 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.162 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:55:55.162 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:55:55.162 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T12:55:55.163 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:55:55.183 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:55:55.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:55.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:55:55.291 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:55:55.325 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:55.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:55 smithi177 ceph-mon[44303]: 3.a scrub ok 2024-05-15T12:55:55.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:55 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3425121505' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 0.9062144341189233}]': finished 2024-05-15T12:55:55.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:55 smithi177 ceph-mon[44303]: osdmap e275: 12 total, 12 up, 11 in 2024-05-15T12:55:55.460 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:55 smithi027 ceph-mon[39795]: 3.a scrub ok 2024-05-15T12:55:55.460 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:55 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3425121505' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 0.9062144341189233}]': finished 2024-05-15T12:55:55.460 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:55 smithi027 ceph-mon[39795]: osdmap e275: 12 total, 12 up, 11 in 2024-05-15T12:55:55.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:55 smithi120 ceph-mon[43961]: 3.a scrub ok 2024-05-15T12:55:55.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:55 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3425121505' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 0.9062144341189233}]': finished 2024-05-15T12:55:55.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:55 smithi120 ceph-mon[43961]: osdmap e275: 12 total, 12 up, 11 in 2024-05-15T12:55:55.785 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-05-15T12:55:55.786 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:55:55.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:55.788 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:55:55.889 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:55:55.925 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:55.946 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:55:55.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:55.951 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:55:56.052 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:55:56.090 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:56.114 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:55:56.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:56.123 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:55:56.163 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:55:56.224 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:55:56.234 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:55:56.251 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:56.427 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:56 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:55:56.427 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:56 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:55:56.427 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:56 smithi177 ceph-mon[44303]: pgmap v763: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 844 op/s 2024-05-15T12:55:56.452 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:55:56.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:56.455 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:55:56.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:56 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:55:56.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:56 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:55:56.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:56 smithi027 ceph-mon[39795]: pgmap v763: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 844 op/s 2024-05-15T12:55:56.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:56 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:55:56.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:56 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:55:56.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:56 smithi120 ceph-mon[43961]: pgmap v763: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 844 op/s 2024-05-15T12:55:56.555 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:55:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:56.857 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:55:56.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:56.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:55:56.965 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:55:56.992 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:55:56.994 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:57.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:57 smithi027 ceph-mon[39795]: osdmap e276: 12 total, 12 up, 11 in 2024-05-15T12:55:57.346 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:55:57.349 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:57 smithi177 ceph-mon[44303]: osdmap e276: 12 total, 12 up, 11 in 2024-05-15T12:55:57.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:57.350 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:55:57.451 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:55:57.516 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:57.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:57 smithi120 ceph-mon[43961]: osdmap e276: 12 total, 12 up, 11 in 2024-05-15T12:55:58.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:58 smithi177 ceph-mon[44303]: pgmap v765: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 802 op/s 2024-05-15T12:55:58.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:58 smithi027 ceph-mon[39795]: pgmap v765: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 802 op/s 2024-05-15T12:55:58.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:58 smithi120 ceph-mon[43961]: pgmap v765: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 802 op/s 2024-05-15T12:55:59.123 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:55:59.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:59.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:55:59.152 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T12:55:59.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:59 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1355403476' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:55:59.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:55:59 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:55:59.231 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T12:55:59.231 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T12:55:59.231 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-05-15T12:55:59.231 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T12:55:59.235 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:55:59.284 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:59.326 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:55:59.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:59.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:55:59.434 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:55:59.469 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:59 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1355403476' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:55:59.469 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:55:59 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:55:59.471 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:59.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:59 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1355403476' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:55:59.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:55:59 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:55:59.692 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:55:59.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:59.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:55:59.796 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:55:59.857 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:55:59.912 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:55:59.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:55:59.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:55:59.930 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-05-15T12:56:00.018 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:56:00.064 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:00.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:00 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:56:00.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:00 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:56:00.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:00 smithi027 ceph-mon[39795]: 3.6 scrub starts 2024-05-15T12:56:00.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:00 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:56:00.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:00 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:56:00.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:00 smithi027 ceph-mon[39795]: osdmap e277: 12 total, 12 up, 11 in 2024-05-15T12:56:00.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:00 smithi027 ceph-mon[39795]: pgmap v767: 17 pgs: 17 active+clean; 4.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 793 op/s 2024-05-15T12:56:00.164 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] out_osds: [11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T12:56:00.165 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T12:56:00.165 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] 2024-05-15T12:56:00.165 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd out 7 2024-05-15T12:56:00.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:00 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:56:00.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:00 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:56:00.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:00 smithi177 ceph-mon[44303]: 3.6 scrub starts 2024-05-15T12:56:00.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:00 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:56:00.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:00 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:56:00.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:00 smithi177 ceph-mon[44303]: osdmap e277: 12 total, 12 up, 11 in 2024-05-15T12:56:00.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:00 smithi177 ceph-mon[44303]: pgmap v767: 17 pgs: 17 active+clean; 4.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 793 op/s 2024-05-15T12:56:00.257 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:56:00.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:00 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:56:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:00 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:56:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:00 smithi120 ceph-mon[43961]: 3.6 scrub starts 2024-05-15T12:56:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:00 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:56:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:00 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:56:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:00 smithi120 ceph-mon[43961]: osdmap e277: 12 total, 12 up, 11 in 2024-05-15T12:56:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:00 smithi120 ceph-mon[43961]: pgmap v767: 17 pgs: 17 active+clean; 4.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 793 op/s 2024-05-15T12:56:00.826 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:56:00.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:00.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:56:00.836 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:56:00.837 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:56:00.873 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:56:00.904 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:56:00.937 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:56:00.946 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:56:00.966 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:00.996 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:56:01.041 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:56:01.083 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:56:01.093 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:56:01.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:01.096 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:56:01.115 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:56:01.147 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:01 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3264860457' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:56:01.152 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:56:01.187 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:56:01.197 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:56:01.219 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:56:01.240 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:01.257 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:56:01.259 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:56:01.259 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:56:01.259 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:56:01.259 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:56:01.259 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:56:01.259 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:56:01.259 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:56:01.259 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:56:01.259 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:56:01.259 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:56:01.259 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:56:01.259 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:56:01.273 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:56:01.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:01.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:56:01.375 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:56:01.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:01 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3264860457' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:56:01.439 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:01 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3264860457' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:56:01.443 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:01.483 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:56:01.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:01.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:56:01.593 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:56:01.640 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:01.780 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:56:01.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:01.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:56:01.886 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:56:01.936 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:02.175 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:02 smithi027 ceph-mon[39795]: pgmap v768: 17 pgs: 17 active+clean; 4.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 832 op/s 2024-05-15T12:56:02.223 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T12:56:02.223 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:56:02.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:02 smithi177 ceph-mon[44303]: pgmap v768: 17 pgs: 17 active+clean; 4.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 832 op/s 2024-05-15T12:56:02.506 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T12:56:02.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:02 smithi120 ceph-mon[43961]: pgmap v768: 17 pgs: 17 active+clean; 4.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 832 op/s 2024-05-15T12:56:02.996 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-05-15T12:56:03.262 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:56:03.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:03.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:56:03.309 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:56:03.372 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:56:03.403 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:03.608 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:56:03.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:03.611 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:56:03.712 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:56:03.768 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:03.906 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:56:03.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:03.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:56:04.015 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:56:04.024 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:03 smithi027 ceph-mon[39795]: pgmap v769: 17 pgs: 17 active+clean; 4.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 868 op/s 2024-05-15T12:56:04.055 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:04.090 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:56:04.091 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:03 smithi120 ceph-mon[43961]: pgmap v769: 17 pgs: 17 active+clean; 4.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 868 op/s 2024-05-15T12:56:04.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:04.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:56:04.101 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:56:04.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:03 smithi177 ceph-mon[44303]: pgmap v769: 17 pgs: 17 active+clean; 4.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 868 op/s 2024-05-15T12:56:04.193 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:56:04.223 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:04.243 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:56:04.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:04.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:56:04.349 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:56:04.419 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:04.514 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:56:04.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:04.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:56:04.625 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:56:04.655 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:04.683 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:56:04.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:04.687 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:56:04.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:04 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-a[39772]: 2024-05-15T12:56:04.464+0000 7f8daffd2700 -1 mon.a@0(leader).osd e277 definitely_dead 0 2024-05-15T12:56:04.788 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:56:04.835 INFO:teuthology.orchestra.run.smithi027.stderr:marked out osd.7. 2024-05-15T12:56:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:04.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:04 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:56:04.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:04 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:56:04.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:04 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2815325553' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-05-15T12:56:04.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:04 smithi177 ceph-mon[44303]: Client client.admin marked osd.7 out, while it was still marked up 2024-05-15T12:56:05.242 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T12:56:05.242 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:56:05.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:04 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:56:05.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:04 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:56:05.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:04 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2815325553' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-05-15T12:56:05.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:04 smithi027 ceph-mon[39795]: Client client.admin marked osd.7 out, while it was still marked up 2024-05-15T12:56:05.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:04 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:56:05.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:04 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:56:05.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:04 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2815325553' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-05-15T12:56:05.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:04 smithi120 ceph-mon[43961]: Client client.admin marked osd.7 out, while it was still marked up 2024-05-15T12:56:05.303 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:56:05.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:05.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:56:05.408 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:56:05.437 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:05.457 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:56:05.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:05.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:56:05.561 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:56:05.593 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:05.720 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:56:05.832 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:05 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2815325553' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-05-15T12:56:05.833 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:05 smithi177 ceph-mon[44303]: osdmap e278: 12 total, 12 up, 10 in 2024-05-15T12:56:05.833 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:05 smithi177 ceph-mon[44303]: pgmap v771: 17 pgs: 1 unknown, 1 peering, 15 active+clean; 3.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 860 op/s 2024-05-15T12:56:05.848 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:05 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2815325553' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-05-15T12:56:05.848 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:05 smithi027 ceph-mon[39795]: osdmap e278: 12 total, 12 up, 10 in 2024-05-15T12:56:05.848 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:05 smithi027 ceph-mon[39795]: pgmap v771: 17 pgs: 1 unknown, 1 peering, 15 active+clean; 3.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 860 op/s 2024-05-15T12:56:05.862 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-05-15T12:56:06.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:05 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2815325553' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-05-15T12:56:06.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:05 smithi120 ceph-mon[43961]: osdmap e278: 12 total, 12 up, 10 in 2024-05-15T12:56:06.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:05 smithi120 ceph-mon[43961]: pgmap v771: 17 pgs: 1 unknown, 1 peering, 15 active+clean; 3.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 860 op/s 2024-05-15T12:56:06.452 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:56:06.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:06.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:56:06.556 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:56:06.592 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:06.613 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:56:06.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:06.616 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:56:06.687 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:56:06.717 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:56:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:06.818 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:56:06.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:06.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:56:06.922 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:56:06.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:06 smithi177 ceph-mon[44303]: osdmap e279: 12 total, 12 up, 10 in 2024-05-15T12:56:06.953 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:06 smithi027 ceph-mon[39795]: osdmap e279: 12 total, 12 up, 10 in 2024-05-15T12:56:06.955 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:07.165 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:56:07.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:07.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:56:07.182 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:56:07.279 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:56:07.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:06 smithi120 ceph-mon[43961]: osdmap e279: 12 total, 12 up, 10 in 2024-05-15T12:56:07.354 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:07.503 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:56:07.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:07.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:56:07.607 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:56:07.662 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:07.860 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T12:56:07.925 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:07 smithi027 ceph-mon[39795]: osdmap e280: 12 total, 12 up, 10 in 2024-05-15T12:56:07.925 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:07 smithi027 ceph-mon[39795]: 3.d starting backfill to osd.6 from (0'0,0'0] MAX to 277'12745 2024-05-15T12:56:07.925 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:07 smithi027 ceph-mon[39795]: 3.7 starting backfill to osd.9 from (0'0,0'0] MAX to 277'12543 2024-05-15T12:56:07.925 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:07 smithi027 ceph-mon[39795]: 3.9 starting backfill to osd.5 from (0'0,0'0] MAX to 277'13103 2024-05-15T12:56:07.925 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:07 smithi027 ceph-mon[39795]: 3.f starting backfill to osd.10 from (0'0,0'0] MAX to 277'12773 2024-05-15T12:56:07.925 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:07 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/678806306' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:56:07.925 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:07 smithi027 ceph-mon[39795]: pgmap v774: 17 pgs: 1 active+remapped+backfilling, 1 unknown, 1 peering, 14 active+clean; 3.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 708 op/s; 3819/188877 objects misplaced (2.022%); 213 KiB/s, 3 objects/s recovering 2024-05-15T12:56:07.934 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:56:07.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:07.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:56:08.051 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:56:08.123 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:07 smithi120 ceph-mon[43961]: osdmap e280: 12 total, 12 up, 10 in 2024-05-15T12:56:08.123 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:07 smithi120 ceph-mon[43961]: 3.d starting backfill to osd.6 from (0'0,0'0] MAX to 277'12745 2024-05-15T12:56:08.123 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:07 smithi120 ceph-mon[43961]: 3.7 starting backfill to osd.9 from (0'0,0'0] MAX to 277'12543 2024-05-15T12:56:08.123 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:07 smithi120 ceph-mon[43961]: 3.9 starting backfill to osd.5 from (0'0,0'0] MAX to 277'13103 2024-05-15T12:56:08.123 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:07 smithi120 ceph-mon[43961]: 3.f starting backfill to osd.10 from (0'0,0'0] MAX to 277'12773 2024-05-15T12:56:08.123 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:07 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/678806306' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:56:08.123 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:07 smithi120 ceph-mon[43961]: pgmap v774: 17 pgs: 1 active+remapped+backfilling, 1 unknown, 1 peering, 14 active+clean; 3.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 708 op/s; 3819/188877 objects misplaced (2.022%); 213 KiB/s, 3 objects/s recovering 2024-05-15T12:56:08.130 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:08.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:07 smithi177 ceph-mon[44303]: osdmap e280: 12 total, 12 up, 10 in 2024-05-15T12:56:08.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:07 smithi177 ceph-mon[44303]: 3.d starting backfill to osd.6 from (0'0,0'0] MAX to 277'12745 2024-05-15T12:56:08.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:07 smithi177 ceph-mon[44303]: 3.7 starting backfill to osd.9 from (0'0,0'0] MAX to 277'12543 2024-05-15T12:56:08.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:07 smithi177 ceph-mon[44303]: 3.9 starting backfill to osd.5 from (0'0,0'0] MAX to 277'13103 2024-05-15T12:56:08.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:07 smithi177 ceph-mon[44303]: 3.f starting backfill to osd.10 from (0'0,0'0] MAX to 277'12773 2024-05-15T12:56:08.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:07 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/678806306' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:56:08.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:07 smithi177 ceph-mon[44303]: pgmap v774: 17 pgs: 1 active+remapped+backfilling, 1 unknown, 1 peering, 14 active+clean; 3.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 708 op/s; 3819/188877 objects misplaced (2.022%); 213 KiB/s, 3 objects/s recovering 2024-05-15T12:56:08.193 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:56:08.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:08.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:56:08.299 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:56:08.361 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:08.747 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:56:08.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:08.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:56:08.856 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:56:08.914 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:08.951 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:56:08.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:08.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:56:09.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:08 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:56:09.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:08 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/678806306' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:56:09.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:08 smithi027 ceph-mon[39795]: osdmap e281: 12 total, 12 up, 10 in 2024-05-15T12:56:09.060 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:56:09.114 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:09.141 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:56:09.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:09.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:56:09.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:08 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:56:09.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:08 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/678806306' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:56:09.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:08 smithi177 ceph-mon[44303]: osdmap e281: 12 total, 12 up, 10 in 2024-05-15T12:56:09.249 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:56:09.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:08 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:56:09.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:08 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/678806306' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:56:09.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:08 smithi120 ceph-mon[43961]: osdmap e281: 12 total, 12 up, 10 in 2024-05-15T12:56:09.283 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:09.307 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:56:09.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:09.315 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:56:09.416 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:56:09.489 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:09.902 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:56:09.918 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:56:09.958 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:56:09.984 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:56:10.030 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:56:10.075 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:56:10.140 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:56:10.179 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:56:10.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:09 smithi177 ceph-mon[44303]: osdmap e282: 12 total, 12 up, 10 in 2024-05-15T12:56:10.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:09 smithi177 ceph-mon[44303]: pgmap v777: 17 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 unknown, 2 peering, 12 active+clean; 3.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 536 op/s; 7695/185046 objects misplaced (4.158%); 320 KiB/s, 4 objects/s recovering 2024-05-15T12:56:10.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:09 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3376366525' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:56:10.221 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:56:10.255 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:56:10.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:09 smithi027 ceph-mon[39795]: osdmap e282: 12 total, 12 up, 10 in 2024-05-15T12:56:10.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:09 smithi027 ceph-mon[39795]: pgmap v777: 17 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 unknown, 2 peering, 12 active+clean; 3.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 536 op/s; 7695/185046 objects misplaced (4.158%); 320 KiB/s, 4 objects/s recovering 2024-05-15T12:56:10.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:09 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3376366525' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:56:10.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:09 smithi120 ceph-mon[43961]: osdmap e282: 12 total, 12 up, 10 in 2024-05-15T12:56:10.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:09 smithi120 ceph-mon[43961]: pgmap v777: 17 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 unknown, 2 peering, 12 active+clean; 3.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 536 op/s; 7695/185046 objects misplaced (4.158%); 320 KiB/s, 4 objects/s recovering 2024-05-15T12:56:10.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:09 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3376366525' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:56:10.283 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:56:10.316 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:56:10.336 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:56:10.336 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:56:10.336 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:56:10.337 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:56:10.337 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:56:10.337 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:56:10.337 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:56:10.337 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:56:10.337 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:56:10.337 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:56:10.337 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:56:10.337 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:56:10.965 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:56:10.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:10.976 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:10 smithi027 ceph-mon[39795]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T12:56:10.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:56:11.077 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:56:11.121 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:11.149 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:56:11.153 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:10 smithi177 ceph-mon[44303]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T12:56:11.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:11.154 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:56:11.254 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:56:11.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:10 smithi120 ceph-mon[43961]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T12:56:11.287 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:11.313 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:56:11.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:11.318 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:56:11.383 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10] out_osds: [11, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T12:56:11.384 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 0 1 2024-05-15T12:56:11.420 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:56:11.467 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:11.495 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:56:11.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:11.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:56:11.602 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:56:11.659 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:11.729 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:56:11.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:11.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:56:11.837 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:56:11.916 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:11.954 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:56:11.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:11.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:56:12.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:11 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:56:12.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:11 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:56:12.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:11 smithi027 ceph-mon[39795]: pgmap v778: 17 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 1 peering, 12 active+clean; 3.7 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 520 op/s; 13596/182286 objects misplaced (7.459%); 17 MiB/s, 276 objects/s recovering 2024-05-15T12:56:12.067 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:56:12.130 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:12.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:11 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:56:12.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:11 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:56:12.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:11 smithi177 ceph-mon[44303]: pgmap v778: 17 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 1 peering, 12 active+clean; 3.7 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 520 op/s; 13596/182286 objects misplaced (7.459%); 17 MiB/s, 276 objects/s recovering 2024-05-15T12:56:12.197 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:56:12.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:12.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:56:12.202 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:11 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:56:12.202 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:11 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:56:12.202 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:11 smithi120 ceph-mon[43961]: pgmap v778: 17 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 1 peering, 12 active+clean; 3.7 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 520 op/s; 13596/182286 objects misplaced (7.459%); 17 MiB/s, 276 objects/s recovering 2024-05-15T12:56:12.302 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:56:12.365 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:12.410 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:56:12.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:12.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:56:12.421 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:56:12.437 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T12:56:12.437 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T12:56:12.438 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.438 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24832.0:209225 3.3 e283/219 3:c10e7f9d:::benchmark_data_smithi177_88104_object75603:head v 283'12772, mlcod=283'12769)", 2024-05-15T12:56:12.438 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:12.432299+0000", 2024-05-15T12:56:12.438 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.012997364000000001, 2024-05-15T12:56:12.438 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013041211, 2024-05-15T12:56:12.438 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:12.438 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T12:56:12.438 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:12.438 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.438 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:12.438 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.432299+0000", 2024-05-15T12:56:12.438 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:12.439 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.439 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.439 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:12.439 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.432299+0000", 2024-05-15T12:56:12.439 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:12.439 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.439 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.439 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:12.439 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.432300+0000", 2024-05-15T12:56:12.439 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6200000000000006e-07 2024-05-15T12:56:12.439 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.439 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.439 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:12.439 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.432313+0000", 2024-05-15T12:56:12.440 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2357e-05 2024-05-15T12:56:12.440 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.440 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.440 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:12.440 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.432327+0000", 2024-05-15T12:56:12.440 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4484000000000001e-05 2024-05-15T12:56:12.440 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.440 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.440 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:12.440 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.432335+0000", 2024-05-15T12:56:12.440 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3070000000000003e-06 2024-05-15T12:56:12.440 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.440 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.441 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:12.441 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.432596+0000", 2024-05-15T12:56:12.441 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00026087199999999999 2024-05-15T12:56:12.441 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.441 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.441 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:12.441 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.432618+0000", 2024-05-15T12:56:12.441 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2147999999999999e-05 2024-05-15T12:56:12.441 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:12.441 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:12.441 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:12.441 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.441 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.441 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:209227 3.1 3:81f84649:::benchmark_data_smithi177_88104_object75605:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-05-15T12:56:12.442 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:12.434001+0000", 2024-05-15T12:56:12.442 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.01129526, 2024-05-15T12:56:12.442 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011412083999999999, 2024-05-15T12:56:12.442 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:12.442 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "waiting for sub ops", 2024-05-15T12:56:12.442 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:56:12.442 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T12:56:12.442 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:56:12.442 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 209227 2024-05-15T12:56:12.442 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.442 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:12.442 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.442 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:12.443 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.434001+0000", 2024-05-15T12:56:12.443 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:12.443 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.443 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.443 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:12.443 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.434001+0000", 2024-05-15T12:56:12.443 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:12.443 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.443 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.443 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:12.443 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.434003+0000", 2024-05-15T12:56:12.443 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5740000000000001e-06 2024-05-15T12:56:12.443 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.443 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.444 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:12.444 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.434011+0000", 2024-05-15T12:56:12.444 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3229999999999998e-06 2024-05-15T12:56:12.444 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.444 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.444 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:12.444 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.434025+0000", 2024-05-15T12:56:12.444 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3458000000000001e-05 2024-05-15T12:56:12.444 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.444 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.444 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:12.444 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.434033+0000", 2024-05-15T12:56:12.444 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1030000000000002e-06 2024-05-15T12:56:12.444 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.445 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.445 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:12.445 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.434095+0000", 2024-05-15T12:56:12.445 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2422999999999995e-05 2024-05-15T12:56:12.445 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.445 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.445 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:12.445 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.434241+0000", 2024-05-15T12:56:12.445 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014598599999999999 2024-05-15T12:56:12.445 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.445 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.445 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 4,10", 2024-05-15T12:56:12.445 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.434357+0000", 2024-05-15T12:56:12.446 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000116229 2024-05-15T12:56:12.446 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.446 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.446 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:56:12.446 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.444009+0000", 2024-05-15T12:56:12.446 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0096519229999999998 2024-05-15T12:56:12.446 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:12.446 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:12.446 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:12.446 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.446 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.446 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24832.0:209220 3.b e283/219 3:dbfa5cf0:::benchmark_data_smithi177_88104_object75598:head v 283'13244, mlcod=283'13241)", 2024-05-15T12:56:12.447 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:12.439290+0000", 2024-05-15T12:56:12.447 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.0060064560000000003, 2024-05-15T12:56:12.447 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0062062860000000001, 2024-05-15T12:56:12.447 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:12.447 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T12:56:12.447 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:12.447 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.447 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:12.447 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.439290+0000", 2024-05-15T12:56:12.447 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:12.447 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.447 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.447 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:12.448 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.439290+0000", 2024-05-15T12:56:12.448 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:12.448 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.448 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.448 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:12.448 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.439291+0000", 2024-05-15T12:56:12.448 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.328e-06 2024-05-15T12:56:12.448 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.448 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.448 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:12.448 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.439302+0000", 2024-05-15T12:56:12.448 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0219e-05 2024-05-15T12:56:12.448 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.448 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.449 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:12.449 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.439314+0000", 2024-05-15T12:56:12.449 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.225e-05 2024-05-15T12:56:12.449 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.449 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.449 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:12.449 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.439323+0000", 2024-05-15T12:56:12.449 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1500000000000005e-06 2024-05-15T12:56:12.449 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.449 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.449 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:12.449 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.443989+0000", 2024-05-15T12:56:12.449 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046662589999999999 2024-05-15T12:56:12.449 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.450 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.450 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:12.450 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.444029+0000", 2024-05-15T12:56:12.450 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9493999999999998e-05 2024-05-15T12:56:12.450 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:12.450 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:12.450 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:12.450 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.450 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.450 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24832.0:209229 3.e e283/219 3:7789f40b:::benchmark_data_smithi177_88104_object75607:head v 283'13434, mlcod=283'13432)", 2024-05-15T12:56:12.450 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:12.439439+0000", 2024-05-15T12:56:12.451 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.0058573200000000001, 2024-05-15T12:56:12.453 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0061167340000000004, 2024-05-15T12:56:12.453 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:12.453 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T12:56:12.453 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:12.453 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.453 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:12.453 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.439439+0000", 2024-05-15T12:56:12.454 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:12.454 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.454 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.454 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:12.454 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.439439+0000", 2024-05-15T12:56:12.454 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:12.454 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.454 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.454 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:12.454 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.439439+0000", 2024-05-15T12:56:12.454 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3799999999999999e-07 2024-05-15T12:56:12.454 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.454 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.454 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:12.455 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.439443+0000", 2024-05-15T12:56:12.455 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9960000000000004e-06 2024-05-15T12:56:12.455 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.455 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.455 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:12.455 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.439451+0000", 2024-05-15T12:56:12.455 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5429999999999996e-06 2024-05-15T12:56:12.455 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.455 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.455 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:12.455 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.439455+0000", 2024-05-15T12:56:12.455 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2599999999999999e-06 2024-05-15T12:56:12.455 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.455 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.456 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:12.456 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.439693+0000", 2024-05-15T12:56:12.456 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00023738599999999999 2024-05-15T12:56:12.456 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.456 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.456 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:12.456 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.439713+0000", 2024-05-15T12:56:12.456 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0344999999999999e-05 2024-05-15T12:56:12.456 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:12.456 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:12.456 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:12.456 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.457 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.457 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:209233 3.6 3:6ffe55bb:::benchmark_data_smithi177_88104_object75611:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-05-15T12:56:12.457 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:12.442068+0000", 2024-05-15T12:56:12.457 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.0032288579999999998, 2024-05-15T12:56:12.457 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035505810000000001, 2024-05-15T12:56:12.457 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:12.457 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "waiting for sub ops", 2024-05-15T12:56:12.457 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:56:12.457 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T12:56:12.457 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:56:12.457 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 209233 2024-05-15T12:56:12.457 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.457 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:12.458 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.458 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:12.458 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.442068+0000", 2024-05-15T12:56:12.458 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:12.458 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.458 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.458 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:12.458 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.442068+0000", 2024-05-15T12:56:12.458 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:12.458 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.458 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.458 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:12.458 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.442070+0000", 2024-05-15T12:56:12.458 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.216e-06 2024-05-15T12:56:12.459 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.459 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.459 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:12.459 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.442078+0000", 2024-05-15T12:56:12.459 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1650000000000006e-06 2024-05-15T12:56:12.459 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.459 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.459 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:12.459 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.442097+0000", 2024-05-15T12:56:12.459 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9310999999999999e-05 2024-05-15T12:56:12.459 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.459 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.459 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:12.460 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.442194+0000", 2024-05-15T12:56:12.460 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6996000000000004e-05 2024-05-15T12:56:12.460 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.460 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.460 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:12.460 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.444702+0000", 2024-05-15T12:56:12.460 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025079630000000002 2024-05-15T12:56:12.460 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.460 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.460 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:12.460 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.444851+0000", 2024-05-15T12:56:12.460 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014877900000000001 2024-05-15T12:56:12.460 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.460 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.460 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T12:56:12.461 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.444978+0000", 2024-05-15T12:56:12.461 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012662400000000001 2024-05-15T12:56:12.461 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:12.461 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:12.461 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:12.461 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.461 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.461 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24832.0:209222 3.b e283/219)", 2024-05-15T12:56:12.461 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:12.442785+0000", 2024-05-15T12:56:12.461 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.002511255, 2024-05-15T12:56:12.461 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029025240000000001, 2024-05-15T12:56:12.461 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:12.461 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "queued for pg", 2024-05-15T12:56:12.461 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:12.462 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.462 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:12.462 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.442785+0000", 2024-05-15T12:56:12.462 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:12.462 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.462 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.462 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:12.462 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.442785+0000", 2024-05-15T12:56:12.462 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:12.462 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.462 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.462 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:12.462 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.442786+0000", 2024-05-15T12:56:12.462 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.02e-07 2024-05-15T12:56:12.462 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.463 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.463 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:12.463 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.442794+0000", 2024-05-15T12:56:12.463 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8590000000000005e-06 2024-05-15T12:56:12.463 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.463 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.463 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:12.463 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.442814+0000", 2024-05-15T12:56:12.463 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0032000000000001e-05 2024-05-15T12:56:12.463 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:12.463 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:12.463 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:12.463 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:12.442822+0000", 2024-05-15T12:56:12.464 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6009999999999997e-06 2024-05-15T12:56:12.464 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:12.464 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:12.464 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:12.464 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:12.464 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T12:56:12.464 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 6 2024-05-15T12:56:12.464 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T12:56:12.520 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:56:12.558 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:12.590 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:56:12.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:12.599 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:56:12.700 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:56:12.751 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:12.780 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:56:12.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:12.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:56:12.890 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:56:12.923 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:12 smithi177 ceph-mon[44303]: osdmap e283: 12 total, 12 up, 10 in 2024-05-15T12:56:12.928 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:13.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:12 smithi027 ceph-mon[39795]: osdmap e283: 12 total, 12 up, 10 in 2024-05-15T12:56:13.107 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:56:13.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:13.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:56:13.222 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:56:13.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:12 smithi120 ceph-mon[43961]: osdmap e283: 12 total, 12 up, 10 in 2024-05-15T12:56:13.282 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:13.560 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:56:13.663 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:56:13.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:13.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:56:13.746 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T12:56:13.768 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:56:13.822 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:13.864 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:56:13.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:13.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:56:13.974 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:56:14.012 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:14.050 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:56:14.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:14.056 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:56:14.157 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:56:14.190 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:14.215 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:56:14.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:14.217 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:56:14.276 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-05-15T12:56:14.318 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:56:14.358 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:14 smithi027 ceph-mon[39795]: pgmap v780: 17 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 13 active+clean; 3.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 736 op/s; 12264/175611 objects misplaced (6.984%); 26 MiB/s, 413 objects/s recovering 2024-05-15T12:56:14.360 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:14.367 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:14 smithi120 ceph-mon[43961]: pgmap v780: 17 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 13 active+clean; 3.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 736 op/s; 12264/175611 objects misplaced (6.984%); 26 MiB/s, 413 objects/s recovering 2024-05-15T12:56:14.388 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:56:14.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:14.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:56:14.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:14 smithi177 ceph-mon[44303]: pgmap v780: 17 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 13 active+clean; 3.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 736 op/s; 12264/175611 objects misplaced (6.984%); 26 MiB/s, 413 objects/s recovering 2024-05-15T12:56:14.492 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:56:14.525 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:14.552 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:56:14.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:14.555 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:56:14.656 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:56:14.699 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:15.037 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:56:15.066 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:56:15.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:15.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:56:15.174 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:56:15.232 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:15.267 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:56:15.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:15.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:56:15.375 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:56:15.437 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:15.482 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:56:15.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:15.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:56:15.590 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:56:15.639 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:15.687 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:56:15.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:15.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:56:15.801 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:56:15.836 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:15.861 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:56:15.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:15.868 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:56:15.969 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:56:16.013 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:15 smithi027 ceph-mon[39795]: pgmap v781: 17 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 13 active+clean; 3.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 719 op/s; 9412/171876 objects misplaced (5.476%); 40 MiB/s, 638 objects/s recovering 2024-05-15T12:56:16.039 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:16.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:15 smithi177 ceph-mon[44303]: pgmap v781: 17 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 13 active+clean; 3.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 719 op/s; 9412/171876 objects misplaced (5.476%); 40 MiB/s, 638 objects/s recovering 2024-05-15T12:56:16.279 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:56:16.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:15 smithi120 ceph-mon[43961]: pgmap v781: 17 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 13 active+clean; 3.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 719 op/s; 9412/171876 objects misplaced (5.476%); 40 MiB/s, 638 objects/s recovering 2024-05-15T12:56:16.411 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:56:16.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:16.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:56:16.523 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:56:16.582 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:16.616 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:56:16.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:16.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:56:16.721 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:56:16.783 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:16.882 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.0 to 1 (10000) 2024-05-15T12:56:17.025 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:16 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1673818383' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T12:56:17.025 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:16 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T12:56:17.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:16 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1673818383' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T12:56:17.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:16 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T12:56:17.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:16 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1673818383' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T12:56:17.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:16 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T12:56:18.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:17 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:56:18.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:17 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:56:18.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:17 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T12:56:18.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:17 smithi177 ceph-mon[44303]: osdmap e284: 12 total, 12 up, 10 in 2024-05-15T12:56:18.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:17 smithi177 ceph-mon[44303]: pgmap v783: 17 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 13 active+clean; 3.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 696 op/s; 8417/168330 objects misplaced (5.000%); 46 MiB/s, 734 objects/s recovering 2024-05-15T12:56:18.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:17 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:56:18.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:17 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:56:18.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:17 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T12:56:18.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:17 smithi027 ceph-mon[39795]: osdmap e284: 12 total, 12 up, 10 in 2024-05-15T12:56:18.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:17 smithi027 ceph-mon[39795]: pgmap v783: 17 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 13 active+clean; 3.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 696 op/s; 8417/168330 objects misplaced (5.000%); 46 MiB/s, 734 objects/s recovering 2024-05-15T12:56:18.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:17 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:56:18.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:17 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:56:18.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:17 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T12:56:18.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:17 smithi120 ceph-mon[43961]: osdmap e284: 12 total, 12 up, 10 in 2024-05-15T12:56:18.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:17 smithi120 ceph-mon[43961]: pgmap v783: 17 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 13 active+clean; 3.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 696 op/s; 8417/168330 objects misplaced (5.000%); 46 MiB/s, 734 objects/s recovering 2024-05-15T12:56:18.778 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:56:18.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:18.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:56:18.896 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:56:18.925 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:18.954 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:56:18.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:18.957 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:56:19.058 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:56:19.138 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:18 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:56:19.138 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:18 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:56:19.138 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:18 smithi120 ceph-mon[43961]: osdmap e285: 12 total, 12 up, 10 in 2024-05-15T12:56:19.143 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:19.149 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:18 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:56:19.149 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:18 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:56:19.149 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:18 smithi027 ceph-mon[39795]: osdmap e285: 12 total, 12 up, 10 in 2024-05-15T12:56:19.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:18 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:56:19.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:18 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:56:19.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:18 smithi177 ceph-mon[44303]: osdmap e285: 12 total, 12 up, 10 in 2024-05-15T12:56:19.201 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:56:19.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:19.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:56:19.306 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:56:19.350 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:19.622 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 1 1 2024-05-15T12:56:19.891 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:19 smithi027 ceph-mon[39795]: osdmap e286: 12 total, 12 up, 10 in 2024-05-15T12:56:19.891 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:19 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:56:19.891 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:19 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:56:19.891 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:19 smithi027 ceph-mon[39795]: pgmap v786: 17 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 13 active+clean; 3.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 700 op/s; 8164/162993 objects misplaced (5.009%); 35 MiB/s, 565 objects/s recovering 2024-05-15T12:56:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:56:19.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:19.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:56:20.045 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:56:20.124 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:20.158 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:56:20.158 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:19 smithi120 ceph-mon[43961]: osdmap e286: 12 total, 12 up, 10 in 2024-05-15T12:56:20.158 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:19 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:56:20.158 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:19 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:56:20.159 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:19 smithi120 ceph-mon[43961]: pgmap v786: 17 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 13 active+clean; 3.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 700 op/s; 8164/162993 objects misplaced (5.009%); 35 MiB/s, 565 objects/s recovering 2024-05-15T12:56:20.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:20.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:56:20.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:19 smithi177 ceph-mon[44303]: osdmap e286: 12 total, 12 up, 10 in 2024-05-15T12:56:20.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:19 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:56:20.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:19 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:56:20.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:19 smithi177 ceph-mon[44303]: pgmap v786: 17 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 13 active+clean; 3.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 700 op/s; 8164/162993 objects misplaced (5.009%); 35 MiB/s, 565 objects/s recovering 2024-05-15T12:56:20.260 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:56:20.291 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:20.310 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:56:20.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:20.314 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:56:20.415 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:56:20.493 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:20.527 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:56:20.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:20.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:56:20.633 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:56:20.710 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:21.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:20 smithi027 ceph-mon[39795]: osdmap e287: 12 total, 12 up, 10 in 2024-05-15T12:56:21.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:20 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2864300717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:56:21.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:20 smithi177 ceph-mon[44303]: osdmap e287: 12 total, 12 up, 10 in 2024-05-15T12:56:21.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:20 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2864300717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:56:21.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:20 smithi120 ceph-mon[43961]: osdmap e287: 12 total, 12 up, 10 in 2024-05-15T12:56:21.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:20 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2864300717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:56:21.748 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:56:21.748 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:56:21.748 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:56:21.748 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:56:21.748 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:56:21.748 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:56:21.748 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:56:21.748 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:56:21.748 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:56:21.749 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:56:21.749 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:56:21.749 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:56:21.749 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:56:21.749 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:56:21.749 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:56:21.749 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:56:21.749 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:56:21.749 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:56:21.749 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:56:21.749 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:56:21.750 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:56:21.750 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:56:21.750 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:56:21.750 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:56:21.966 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T12:56:22.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:21 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3366942922' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:56:22.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:21 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:56:22.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:21 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:56:22.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:21 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:56:22.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:21 smithi027 ceph-mon[39795]: pgmap v788: 17 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 14 active+clean; 3.3 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 726 op/s; 6411/159792 objects misplaced (4.012%); 32 MiB/s, 505 objects/s recovering 2024-05-15T12:56:22.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:21 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3366942922' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:56:22.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:21 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:56:22.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:21 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:56:22.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:21 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:56:22.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:21 smithi120 ceph-mon[43961]: pgmap v788: 17 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 14 active+clean; 3.3 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 726 op/s; 6411/159792 objects misplaced (4.012%); 32 MiB/s, 505 objects/s recovering 2024-05-15T12:56:22.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:21 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3366942922' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:56:22.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:21 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:56:22.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:21 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:56:22.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:21 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:56:22.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:21 smithi177 ceph-mon[44303]: pgmap v788: 17 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 14 active+clean; 3.3 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 726 op/s; 6411/159792 objects misplaced (4.012%); 32 MiB/s, 505 objects/s recovering 2024-05-15T12:56:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:22 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:56:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:22 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:56:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:22 smithi027 ceph-mon[39795]: osdmap e288: 12 total, 12 up, 10 in 2024-05-15T12:56:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:22 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T12:56:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:22 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T12:56:23.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:22 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:56:23.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:22 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:56:23.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:22 smithi120 ceph-mon[43961]: osdmap e288: 12 total, 12 up, 10 in 2024-05-15T12:56:23.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:22 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T12:56:23.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:22 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T12:56:23.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:22 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:56:23.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:22 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:56:23.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:22 smithi177 ceph-mon[44303]: osdmap e288: 12 total, 12 up, 10 in 2024-05-15T12:56:23.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:22 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T12:56:23.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:22 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T12:56:24.023 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:56:24.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:24.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:56:24.135 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:56:24.176 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:24.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:24 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:56:24.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:24 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:56:24.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:24 smithi177 ceph-mon[44303]: pgmap v790: 17 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 14 active+clean; 3.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 819 op/s; 4867/153942 objects misplaced (3.162%); 31 MiB/s, 495 objects/s recovering 2024-05-15T12:56:24.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:24 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:56:24.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:24 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:56:24.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:24 smithi027 ceph-mon[39795]: pgmap v790: 17 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 14 active+clean; 3.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 819 op/s; 4867/153942 objects misplaced (3.162%); 31 MiB/s, 495 objects/s recovering 2024-05-15T12:56:24.536 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:24 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:56:24.536 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:24 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:56:24.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:24 smithi120 ceph-mon[43961]: pgmap v790: 17 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 14 active+clean; 3.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 819 op/s; 4867/153942 objects misplaced (3.162%); 31 MiB/s, 495 objects/s recovering 2024-05-15T12:56:25.137 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:56:25.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:25.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:56:25.248 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:56:25.280 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:25.304 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:56:25.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:25.309 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:56:25.410 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:56:25.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:25 smithi177 ceph-mon[44303]: osdmap e289: 12 total, 12 up, 10 in 2024-05-15T12:56:25.439 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:25.459 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:56:25.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:25.463 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:56:25.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:25 smithi027 ceph-mon[39795]: osdmap e289: 12 total, 12 up, 10 in 2024-05-15T12:56:25.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:25 smithi120 ceph-mon[43961]: osdmap e289: 12 total, 12 up, 10 in 2024-05-15T12:56:25.564 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:56:25.599 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:26.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:26 smithi177 ceph-mon[44303]: 3.6 scrub starts 2024-05-15T12:56:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:26 smithi177 ceph-mon[44303]: osdmap e290: 12 total, 12 up, 10 in 2024-05-15T12:56:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:26 smithi177 ceph-mon[44303]: 3.6 scrub ok 2024-05-15T12:56:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:26 smithi177 ceph-mon[44303]: pgmap v793: 17 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 14 active+clean; 3.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 743 op/s; 4056/150030 objects misplaced (2.703%); 39 MiB/s, 628 objects/s recovering 2024-05-15T12:56:26.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:26 smithi027 ceph-mon[39795]: 3.6 scrub starts 2024-05-15T12:56:26.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:26 smithi027 ceph-mon[39795]: osdmap e290: 12 total, 12 up, 10 in 2024-05-15T12:56:26.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:26 smithi027 ceph-mon[39795]: 3.6 scrub ok 2024-05-15T12:56:26.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:26 smithi027 ceph-mon[39795]: pgmap v793: 17 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 14 active+clean; 3.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 743 op/s; 4056/150030 objects misplaced (2.703%); 39 MiB/s, 628 objects/s recovering 2024-05-15T12:56:26.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:26 smithi120 ceph-mon[43961]: 3.6 scrub starts 2024-05-15T12:56:26.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:26 smithi120 ceph-mon[43961]: osdmap e290: 12 total, 12 up, 10 in 2024-05-15T12:56:26.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:26 smithi120 ceph-mon[43961]: 3.6 scrub ok 2024-05-15T12:56:26.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:26 smithi120 ceph-mon[43961]: pgmap v793: 17 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 14 active+clean; 3.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 743 op/s; 4056/150030 objects misplaced (2.703%); 39 MiB/s, 628 objects/s recovering 2024-05-15T12:56:27.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:27 smithi177 ceph-mon[44303]: 3.2 scrub starts 2024-05-15T12:56:27.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:27 smithi177 ceph-mon[44303]: 3.0 scrub starts 2024-05-15T12:56:27.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:27 smithi177 ceph-mon[44303]: 3.2 scrub ok 2024-05-15T12:56:27.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:27 smithi177 ceph-mon[44303]: osdmap e291: 12 total, 12 up, 10 in 2024-05-15T12:56:27.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:27 smithi177 ceph-mon[44303]: 3.0 scrub ok 2024-05-15T12:56:27.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:27 smithi027 ceph-mon[39795]: 3.2 scrub starts 2024-05-15T12:56:27.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:27 smithi027 ceph-mon[39795]: 3.0 scrub starts 2024-05-15T12:56:27.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:27 smithi027 ceph-mon[39795]: 3.2 scrub ok 2024-05-15T12:56:27.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:27 smithi027 ceph-mon[39795]: osdmap e291: 12 total, 12 up, 10 in 2024-05-15T12:56:27.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:27 smithi027 ceph-mon[39795]: 3.0 scrub ok 2024-05-15T12:56:27.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:27 smithi120 ceph-mon[43961]: 3.2 scrub starts 2024-05-15T12:56:27.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:27 smithi120 ceph-mon[43961]: 3.0 scrub starts 2024-05-15T12:56:27.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:27 smithi120 ceph-mon[43961]: 3.2 scrub ok 2024-05-15T12:56:27.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:27 smithi120 ceph-mon[43961]: osdmap e291: 12 total, 12 up, 10 in 2024-05-15T12:56:27.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:27 smithi120 ceph-mon[43961]: 3.0 scrub ok 2024-05-15T12:56:28.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:28 smithi177 ceph-mon[44303]: osdmap e292: 12 total, 12 up, 10 in 2024-05-15T12:56:28.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:28 smithi177 ceph-mon[44303]: pgmap v796: 17 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 15 active+clean; 3.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 605 op/s; 3649/146670 objects misplaced (2.488%); 18 MiB/s, 287 objects/s recovering 2024-05-15T12:56:28.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:28 smithi027 ceph-mon[39795]: osdmap e292: 12 total, 12 up, 10 in 2024-05-15T12:56:28.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:28 smithi027 ceph-mon[39795]: pgmap v796: 17 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 15 active+clean; 3.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 605 op/s; 3649/146670 objects misplaced (2.488%); 18 MiB/s, 287 objects/s recovering 2024-05-15T12:56:28.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:28 smithi120 ceph-mon[43961]: osdmap e292: 12 total, 12 up, 10 in 2024-05-15T12:56:28.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:28 smithi120 ceph-mon[43961]: pgmap v796: 17 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 15 active+clean; 3.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 605 op/s; 3649/146670 objects misplaced (2.488%); 18 MiB/s, 287 objects/s recovering 2024-05-15T12:56:29.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:29 smithi177 ceph-mon[44303]: osdmap e293: 12 total, 12 up, 10 in 2024-05-15T12:56:29.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:29 smithi027 ceph-mon[39795]: osdmap e293: 12 total, 12 up, 10 in 2024-05-15T12:56:29.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:29 smithi120 ceph-mon[43961]: osdmap e293: 12 total, 12 up, 10 in 2024-05-15T12:56:30.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:30 smithi177 ceph-mon[44303]: 3.e scrub starts 2024-05-15T12:56:30.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:30 smithi177 ceph-mon[44303]: pgmap v798: 17 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 14 active+clean; 2.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 745 op/s; 3457/142089 objects misplaced (2.433%); 16 MiB/s, 248 objects/s recovering 2024-05-15T12:56:30.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:30 smithi027 ceph-mon[39795]: 3.e scrub starts 2024-05-15T12:56:30.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:30 smithi027 ceph-mon[39795]: pgmap v798: 17 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 14 active+clean; 2.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 745 op/s; 3457/142089 objects misplaced (2.433%); 16 MiB/s, 248 objects/s recovering 2024-05-15T12:56:30.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:30 smithi120 ceph-mon[43961]: 3.e scrub starts 2024-05-15T12:56:30.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:30 smithi120 ceph-mon[43961]: pgmap v798: 17 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 14 active+clean; 2.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 745 op/s; 3457/142089 objects misplaced (2.433%); 16 MiB/s, 248 objects/s recovering 2024-05-15T12:56:31.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:31 smithi177 ceph-mon[44303]: 3.e scrub ok 2024-05-15T12:56:31.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:31 smithi027 ceph-mon[39795]: 3.e scrub ok 2024-05-15T12:56:31.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:31 smithi120 ceph-mon[43961]: 3.e scrub ok 2024-05-15T12:56:32.321 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:56:32.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:32 smithi177 ceph-mon[44303]: pgmap v799: 17 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 15 active+clean; 2.8 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 611 op/s; 2972/139017 objects misplaced (2.138%); 9.2 MiB/s, 146 objects/s recovering 2024-05-15T12:56:32.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:32 smithi027 ceph-mon[39795]: pgmap v799: 17 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 15 active+clean; 2.8 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 611 op/s; 2972/139017 objects misplaced (2.138%); 9.2 MiB/s, 146 objects/s recovering 2024-05-15T12:56:32.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:32 smithi120 ceph-mon[43961]: pgmap v799: 17 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 15 active+clean; 2.8 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 611 op/s; 2972/139017 objects misplaced (2.138%); 9.2 MiB/s, 146 objects/s recovering 2024-05-15T12:56:33.258 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:56:33.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:33.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:56:33.287 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:56:33.371 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:56:33.433 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:33.462 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:56:33.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:33.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:56:33.566 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:56:33.620 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:33.650 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:56:33.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:33.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:56:33.753 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:56:33.822 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:34.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:34 smithi027 ceph-mon[39795]: pgmap v800: 17 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 15 active+clean; 2.7 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 733 op/s; 2713/131979 objects misplaced (2.056%); 7.2 MiB/s, 115 objects/s recovering 2024-05-15T12:56:34.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:34 smithi027 ceph-mon[39795]: 3.5 scrub starts 2024-05-15T12:56:34.281 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:56:34.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:34.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:56:34.345 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:56:34.389 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:56:34.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:34 smithi177 ceph-mon[44303]: pgmap v800: 17 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 15 active+clean; 2.7 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 733 op/s; 2713/131979 objects misplaced (2.056%); 7.2 MiB/s, 115 objects/s recovering 2024-05-15T12:56:34.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:34 smithi177 ceph-mon[44303]: 3.5 scrub starts 2024-05-15T12:56:34.466 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:34.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:34 smithi120 ceph-mon[43961]: pgmap v800: 17 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 15 active+clean; 2.7 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 733 op/s; 2713/131979 objects misplaced (2.056%); 7.2 MiB/s, 115 objects/s recovering 2024-05-15T12:56:34.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:34 smithi120 ceph-mon[43961]: 3.5 scrub starts 2024-05-15T12:56:35.283 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T12:56:35.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:35 smithi177 ceph-mon[44303]: 3.5 scrub ok 2024-05-15T12:56:35.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:35 smithi177 ceph-mon[44303]: 3.c scrub starts 2024-05-15T12:56:35.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:35 smithi027 ceph-mon[39795]: 3.5 scrub ok 2024-05-15T12:56:35.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:35 smithi027 ceph-mon[39795]: 3.c scrub starts 2024-05-15T12:56:35.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:35 smithi120 ceph-mon[43961]: 3.5 scrub ok 2024-05-15T12:56:35.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:35 smithi120 ceph-mon[43961]: 3.c scrub starts 2024-05-15T12:56:35.736 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:56:35.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:35.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:56:35.852 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:56:35.901 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:36.085 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:56:36.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:36.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:56:36.196 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:56:36.235 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:36.266 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:56:36.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:36.269 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:56:36.356 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:36 smithi027 ceph-mon[39795]: pgmap v801: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 2.6 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 875 op/s; 2713/126177 objects misplaced (2.150%); 6.4 MiB/s, 102 objects/s recovering 2024-05-15T12:56:36.357 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:36 smithi027 ceph-mon[39795]: 3.c scrub ok 2024-05-15T12:56:36.370 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:56:36.407 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:36 smithi120 ceph-mon[43961]: pgmap v801: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 2.6 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 875 op/s; 2713/126177 objects misplaced (2.150%); 6.4 MiB/s, 102 objects/s recovering 2024-05-15T12:56:36.407 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:36 smithi120 ceph-mon[43961]: 3.c scrub ok 2024-05-15T12:56:36.413 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:36.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:36 smithi177 ceph-mon[44303]: pgmap v801: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 2.6 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 875 op/s; 2713/126177 objects misplaced (2.150%); 6.4 MiB/s, 102 objects/s recovering 2024-05-15T12:56:36.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:36 smithi177 ceph-mon[44303]: 3.c scrub ok 2024-05-15T12:56:36.460 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:56:36.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:36.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:56:36.563 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:56:36.624 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:36.656 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:56:36.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:36.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:56:36.763 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:56:36.810 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:36.842 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T12:56:36.842 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:56:36.861 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:56:36.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:36.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:56:36.966 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:56:37.010 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:37.061 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:56:37.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:37.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:56:37.164 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:56:37.192 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:37.211 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:56:37.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:37.218 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:56:37.319 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:56:37.348 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:37.378 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:56:37.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:37.385 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:56:37.486 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:56:37.560 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:38.110 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:37 smithi027 ceph-mon[39795]: pgmap v802: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 2.5 GiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 758 op/s; 2713/123912 objects misplaced (2.189%); 3.0 MiB/s, 47 objects/s recovering 2024-05-15T12:56:38.136 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:56:38.144 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-05-15T12:56:38.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:38.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:56:38.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:37 smithi177 ceph-mon[44303]: pgmap v802: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 2.5 GiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 758 op/s; 2713/123912 objects misplaced (2.189%); 3.0 MiB/s, 47 objects/s recovering 2024-05-15T12:56:38.247 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:56:38.281 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:56:38.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:37 smithi120 ceph-mon[43961]: pgmap v802: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 2.5 GiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 758 op/s; 2713/123912 objects misplaced (2.189%); 3.0 MiB/s, 47 objects/s recovering 2024-05-15T12:56:38.285 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:38.303 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:56:38.330 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:56:38.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:38.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:56:38.435 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:56:38.466 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:38.855 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:38 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:56:38.856 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:38 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1273398188' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T12:56:38.856 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:38 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T12:56:38.924 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:56:38.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:38.928 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:56:38.934 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:38 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:56:38.934 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:38 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1273398188' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T12:56:38.934 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:38 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T12:56:39.029 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:56:39.092 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:39.115 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.1 to 1 (10000) 2024-05-15T12:56:39.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:38 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:56:39.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:38 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1273398188' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T12:56:39.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:38 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T12:56:39.769 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:56:39.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:39.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:56:39.875 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:56:39.926 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:39.952 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:56:39.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:39.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:56:40.072 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:56:40.111 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:40.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:40 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T12:56:40.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:40 smithi177 ceph-mon[44303]: osdmap e294: 12 total, 12 up, 10 in 2024-05-15T12:56:40.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:40 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:56:40.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:40 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:56:40.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:40 smithi177 ceph-mon[44303]: pgmap v804: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 2.4 GiB data, 9.2 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 809 op/s; 2466/117801 objects misplaced (2.093%); 3.0 MiB/s, 47 objects/s recovering 2024-05-15T12:56:40.479 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:56:40.481 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:40 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T12:56:40.481 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:40 smithi027 ceph-mon[39795]: osdmap e294: 12 total, 12 up, 10 in 2024-05-15T12:56:40.481 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:40 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:56:40.481 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:40 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:56:40.481 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:40 smithi027 ceph-mon[39795]: pgmap v804: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 2.4 GiB data, 9.2 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 809 op/s; 2466/117801 objects misplaced (2.093%); 3.0 MiB/s, 47 objects/s recovering 2024-05-15T12:56:40.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:40.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:56:40.523 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 2 1 2024-05-15T12:56:40.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:40 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T12:56:40.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:40 smithi120 ceph-mon[43961]: osdmap e294: 12 total, 12 up, 10 in 2024-05-15T12:56:40.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:40 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:56:40.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:40 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:56:40.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:40 smithi120 ceph-mon[43961]: pgmap v804: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 2.4 GiB data, 9.2 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 809 op/s; 2466/117801 objects misplaced (2.093%); 3.0 MiB/s, 47 objects/s recovering 2024-05-15T12:56:40.583 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:56:40.673 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:41.141 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:56:41.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:41.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:56:41.246 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:56:41.289 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:41.356 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:56:41.366 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:41 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:56:41.366 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:41 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:56:41.366 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:41 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:56:41.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:41.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:56:41.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:41 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:56:41.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:41 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:56:41.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:41 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:56:41.468 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:56:41.517 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:41.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:41 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:56:41.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:41 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:56:41.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:41 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:56:41.555 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:56:41.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:41.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:56:41.667 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:56:41.713 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:41.740 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:56:41.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:41.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:56:41.845 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:56:41.935 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:41.963 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:56:41.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:41.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:56:42.076 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:56:42.128 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:42 smithi027 ceph-mon[39795]: pgmap v805: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 2.3 GiB data, 9.0 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 837 op/s; 2466/113862 objects misplaced (2.166%) 2024-05-15T12:56:42.129 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:42.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:42 smithi120 ceph-mon[43961]: pgmap v805: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 2.3 GiB data, 9.0 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 837 op/s; 2466/113862 objects misplaced (2.166%) 2024-05-15T12:56:42.304 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:56:42.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:42.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:56:42.310 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:56:42.371 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:56:42.412 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:56:42.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:42 smithi177 ceph-mon[44303]: pgmap v805: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 2.3 GiB data, 9.0 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 837 op/s; 2466/113862 objects misplaced (2.166%) 2024-05-15T12:56:42.448 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:42.476 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:56:42.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:42.479 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:56:42.580 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:56:42.648 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:42.688 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:56:42.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:42.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:56:42.797 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:56:42.843 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:42.888 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:56:42.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:42.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:56:43.002 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:56:43.034 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:43.055 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:56:43.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:43.058 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:56:43.159 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:56:43.222 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:43 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:56:43.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:43 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:56:43.225 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:43.259 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:56:43.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:43.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:56:43.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:43 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:56:43.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:43 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:56:43.370 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:56:43.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:43 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:56:43.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:43 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:56:43.460 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:43.669 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:56:43.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:43.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:56:43.779 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:56:43.807 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:43.828 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:56:43.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:43.835 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:56:43.938 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:56:43.967 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:43.994 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:56:43.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:43.996 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:56:44.097 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:56:44.124 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:44.145 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:56:44.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:44.148 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:56:44.180 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T12:56:44.248 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:56:44.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:44 smithi027 ceph-mon[39795]: osdmap e295: 12 total, 12 up, 10 in 2024-05-15T12:56:44.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:44 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/96681260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:56:44.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:44 smithi027 ceph-mon[39795]: pgmap v807: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 2.2 GiB data, 8.6 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 823 op/s; 2174/106395 objects misplaced (2.043%) 2024-05-15T12:56:44.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:44 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/512249143' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:56:44.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:44 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:56:44.279 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:44.300 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:56:44.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:44.303 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:56:44.404 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:56:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:44 smithi177 ceph-mon[44303]: osdmap e295: 12 total, 12 up, 10 in 2024-05-15T12:56:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:44 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/96681260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:56:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:44 smithi177 ceph-mon[44303]: pgmap v807: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 2.2 GiB data, 8.6 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 823 op/s; 2174/106395 objects misplaced (2.043%) 2024-05-15T12:56:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:44 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/512249143' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:56:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:44 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:56:44.456 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:44.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:44.495 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:56:44.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:56:44.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:44 smithi120 ceph-mon[43961]: osdmap e295: 12 total, 12 up, 10 in 2024-05-15T12:56:44.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:44 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/96681260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:56:44.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:44 smithi120 ceph-mon[43961]: pgmap v807: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 2.2 GiB data, 8.6 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 823 op/s; 2174/106395 objects misplaced (2.043%) 2024-05-15T12:56:44.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:44 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/512249143' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:56:44.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:44 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:56:44.596 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:56:44.600 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:56:44.600 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:56:44.600 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:56:44.600 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:56:44.600 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:56:44.600 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:56:44.600 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:56:44.600 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:56:44.600 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:56:44.601 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:56:44.601 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:56:44.601 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:56:44.601 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:56:44.601 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:56:44.601 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:56:44.601 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:56:44.601 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:56:44.601 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:56:44.601 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:56:44.601 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:56:44.602 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:56:44.602 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:56:44.602 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:56:44.602 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:56:44.637 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:44.666 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:56:44.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:44.668 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:56:44.769 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:56:44.807 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:44.833 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:56:44.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:44.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:56:44.938 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:56:44.990 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:45.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:45 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:56:45.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:45 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:56:45.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:45 smithi027 ceph-mon[39795]: osdmap e296: 12 total, 12 up, 10 in 2024-05-15T12:56:45.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:45 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:56:45.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:45 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:56:45.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:45 smithi177 ceph-mon[44303]: osdmap e296: 12 total, 12 up, 10 in 2024-05-15T12:56:45.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:45 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:56:45.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:45 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:56:45.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:45 smithi120 ceph-mon[43961]: osdmap e296: 12 total, 12 up, 10 in 2024-05-15T12:56:46.066 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:56:46.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:46.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:56:46.171 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:56:46.199 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:46.225 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:56:46.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:46.228 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:56:46.329 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:56:46.363 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:46.383 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:56:46.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:46.386 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:56:46.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:46 smithi177 ceph-mon[44303]: pgmap v809: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 2.1 GiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 923 op/s; 2174/100977 objects misplaced (2.153%) 2024-05-15T12:56:46.477 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:46 smithi027 ceph-mon[39795]: pgmap v809: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 2.1 GiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 923 op/s; 2174/100977 objects misplaced (2.153%) 2024-05-15T12:56:46.487 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:56:46.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:46 smithi120 ceph-mon[43961]: pgmap v809: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 2.1 GiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 923 op/s; 2174/100977 objects misplaced (2.153%) 2024-05-15T12:56:46.537 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:46.584 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:56:46.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:46.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:56:46.694 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:56:46.746 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:46.782 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:56:46.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:46.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:56:46.888 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:56:46.938 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:46.965 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:56:46.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:46.968 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:56:47.069 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:56:47.143 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:47.803 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:56:47.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:47.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:56:47.912 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:56:47.940 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:47.960 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:56:47.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:47.965 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:56:48.067 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:56:48.110 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:47 smithi027 ceph-mon[39795]: pgmap v810: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 2.0 GiB data, 8.1 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 796 op/s; 2174/98670 objects misplaced (2.203%) 2024-05-15T12:56:48.114 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:48.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:47 smithi177 ceph-mon[44303]: pgmap v810: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 2.0 GiB data, 8.1 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 796 op/s; 2174/98670 objects misplaced (2.203%) 2024-05-15T12:56:48.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:47 smithi120 ceph-mon[43961]: pgmap v810: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 2.0 GiB data, 8.1 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 796 op/s; 2174/98670 objects misplaced (2.203%) 2024-05-15T12:56:48.486 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:56:48.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:48.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:56:48.553 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T12:56:48.554 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T12:56:48.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T12:56:48.554 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T12:56:48.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.554 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:37183 3.6 3:6cf935c9:::benchmark_data_smithi177_88104_object37182:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-05-15T12:56:48.554 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:53:22.828029+0000", 2024-05-15T12:56:48.554 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 205.73107559300001, 2024-05-15T12:56:48.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.211649965, 2024-05-15T12:56:48.554 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:48.555 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:56:48.555 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:56:48.555 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T12:56:48.555 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:56:48.555 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 37183 2024-05-15T12:56:48.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.555 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:48.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:48.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828029+0000", 2024-05-15T12:56:48.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:48.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828029+0000", 2024-05-15T12:56:48.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:48.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828030+0000", 2024-05-15T12:56:48.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5149999999999999e-06 2024-05-15T12:56:48.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:48.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828157+0000", 2024-05-15T12:56:48.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000127119 2024-05-15T12:56:48.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:48.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828171+0000", 2024-05-15T12:56:48.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4134999999999999e-05 2024-05-15T12:56:48.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:48.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828180+0000", 2024-05-15T12:56:48.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6910000000000003e-06 2024-05-15T12:56:48.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:48.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828229+0000", 2024-05-15T12:56:48.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9320999999999998e-05 2024-05-15T12:56:48.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:48.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828370+0000", 2024-05-15T12:56:48.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000140705 2024-05-15T12:56:48.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T12:56:48.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828519+0000", 2024-05-15T12:56:48.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014910699999999999 2024-05-15T12:56:48.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.831299+0000", 2024-05-15T12:56:48.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0027803599999999999 2024-05-15T12:56:48.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:56:48.562 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.831307+0000", 2024-05-15T12:56:48.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1269999999999997e-06 2024-05-15T12:56:48.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.039610+0000", 2024-05-15T12:56:48.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.208303346 2024-05-15T12:56:48.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:56:48.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.039648+0000", 2024-05-15T12:56:48.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8115999999999999e-05 2024-05-15T12:56:48.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:56:48.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.039679+0000", 2024-05-15T12:56:48.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0423e-05 2024-05-15T12:56:48.564 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.564 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:48.565 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.565 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:37184 3.6 3:64ac7aa0:::benchmark_data_smithi177_88104_object37183:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-05-15T12:56:48.565 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:53:22.828198+0000", 2024-05-15T12:56:48.565 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 205.730905855, 2024-05-15T12:56:48.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.21159296799999999, 2024-05-15T12:56:48.565 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:48.566 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:56:48.566 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:56:48.566 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T12:56:48.566 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:56:48.566 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 37184 2024-05-15T12:56:48.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.566 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:48.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:48.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828198+0000", 2024-05-15T12:56:48.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:48.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828198+0000", 2024-05-15T12:56:48.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:48.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828199+0000", 2024-05-15T12:56:48.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6199999999999998e-07 2024-05-15T12:56:48.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:48.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828231+0000", 2024-05-15T12:56:48.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2506999999999999e-05 2024-05-15T12:56:48.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:48.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828239+0000", 2024-05-15T12:56:48.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7419999999999997e-06 2024-05-15T12:56:48.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:48.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828245+0000", 2024-05-15T12:56:48.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6359999999999999e-06 2024-05-15T12:56:48.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.570 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:48.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.829838+0000", 2024-05-15T12:56:48.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001593391 2024-05-15T12:56:48.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.570 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:48.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.829960+0000", 2024-05-15T12:56:48.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000121637 2024-05-15T12:56:48.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T12:56:48.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.830089+0000", 2024-05-15T12:56:48.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012972999999999999 2024-05-15T12:56:48.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:56:48.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.831422+0000", 2024-05-15T12:56:48.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0013328280000000001 2024-05-15T12:56:48.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.572 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.831931+0000", 2024-05-15T12:56:48.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00050825999999999996 2024-05-15T12:56:48.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.572 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.039722+0000", 2024-05-15T12:56:48.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20779120300000001 2024-05-15T12:56:48.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.573 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:56:48.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.039762+0000", 2024-05-15T12:56:48.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9885000000000002e-05 2024-05-15T12:56:48.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:56:48.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.039791+0000", 2024-05-15T12:56:48.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9587e-05 2024-05-15T12:56:48.574 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.574 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:48.574 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.575 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:37200 3.1 3:8e4b7950:::benchmark_data_smithi177_88104_object37199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-05-15T12:56:48.575 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:53:22.834283+0000", 2024-05-15T12:56:48.575 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 205.724821273, 2024-05-15T12:56:48.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.208503207, 2024-05-15T12:56:48.575 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:48.575 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:56:48.575 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:56:48.575 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T12:56:48.576 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:56:48.576 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 37200 2024-05-15T12:56:48.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.576 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:48.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:48.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.834283+0000", 2024-05-15T12:56:48.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:48.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.834283+0000", 2024-05-15T12:56:48.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:48.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.834284+0000", 2024-05-15T12:56:48.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.624e-06 2024-05-15T12:56:48.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:48.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.834373+0000", 2024-05-15T12:56:48.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8338999999999996e-05 2024-05-15T12:56:48.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:48.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.834385+0000", 2024-05-15T12:56:48.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2238e-05 2024-05-15T12:56:48.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:48.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.834391+0000", 2024-05-15T12:56:48.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7790000000000001e-06 2024-05-15T12:56:48.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:48.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.834437+0000", 2024-05-15T12:56:48.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5831999999999998e-05 2024-05-15T12:56:48.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:48.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.834528+0000", 2024-05-15T12:56:48.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1253000000000002e-05 2024-05-15T12:56:48.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 4,10", 2024-05-15T12:56:48.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.834619+0000", 2024-05-15T12:56:48.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0716000000000004e-05 2024-05-15T12:56:48.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:56:48.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.835998+0000", 2024-05-15T12:56:48.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001379433 2024-05-15T12:56:48.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.836563+0000", 2024-05-15T12:56:48.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00056531099999999998 2024-05-15T12:56:48.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.042703+0000", 2024-05-15T12:56:48.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20613922700000001 2024-05-15T12:56:48.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:56:48.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.042754+0000", 2024-05-15T12:56:48.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1468000000000003e-05 2024-05-15T12:56:48.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:56:48.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.042786+0000", 2024-05-15T12:56:48.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1986999999999997e-05 2024-05-15T12:56:48.584 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.584 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:48.584 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.584 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:37202 3.6 3:6388ea14:::benchmark_data_smithi177_88104_object37201:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-05-15T12:56:48.584 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:53:22.835082+0000", 2024-05-15T12:56:48.584 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 205.72402216, 2024-05-15T12:56:48.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20765689300000001, 2024-05-15T12:56:48.585 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:48.585 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:56:48.585 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:56:48.585 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T12:56:48.585 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:56:48.585 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 37202 2024-05-15T12:56:48.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.585 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:48.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:48.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.835082+0000", 2024-05-15T12:56:48.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:48.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.835082+0000", 2024-05-15T12:56:48.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:48.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.835083+0000", 2024-05-15T12:56:48.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8599999999999997e-07 2024-05-15T12:56:48.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:48.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.835189+0000", 2024-05-15T12:56:48.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010657 2024-05-15T12:56:48.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:48.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.835205+0000", 2024-05-15T12:56:48.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5693999999999999e-05 2024-05-15T12:56:48.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:48.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.835214+0000", 2024-05-15T12:56:48.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2639999999999997e-06 2024-05-15T12:56:48.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:48.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.835276+0000", 2024-05-15T12:56:48.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1495000000000007e-05 2024-05-15T12:56:48.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:48.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.835405+0000", 2024-05-15T12:56:48.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012882400000000001 2024-05-15T12:56:48.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T12:56:48.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.835547+0000", 2024-05-15T12:56:48.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000142268 2024-05-15T12:56:48.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:56:48.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.837000+0000", 2024-05-15T12:56:48.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001453024 2024-05-15T12:56:48.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.837078+0000", 2024-05-15T12:56:48.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7657000000000005e-05 2024-05-15T12:56:48.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.042639+0000", 2024-05-15T12:56:48.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20556112200000001 2024-05-15T12:56:48.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:56:48.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.042699+0000", 2024-05-15T12:56:48.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0133000000000002e-05 2024-05-15T12:56:48.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:56:48.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.042739+0000", 2024-05-15T12:56:48.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0055999999999997e-05 2024-05-15T12:56:48.593 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.593 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:48.593 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.594 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:37206 3.6 3:66563e40:::benchmark_data_smithi177_88104_object37205:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-05-15T12:56:48.594 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:53:22.837645+0000", 2024-05-15T12:56:48.594 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 205.72145958999999, 2024-05-15T12:56:48.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20581651500000001, 2024-05-15T12:56:48.594 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:48.594 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:56:48.594 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:56:48.594 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T12:56:48.595 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:56:48.595 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 37206 2024-05-15T12:56:48.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.595 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:48.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:48.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.837645+0000", 2024-05-15T12:56:48.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:48.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.837645+0000", 2024-05-15T12:56:48.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:48.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.837645+0000", 2024-05-15T12:56:48.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5000000000000001e-07 2024-05-15T12:56:48.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.597 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:48.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.837700+0000", 2024-05-15T12:56:48.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4256999999999999e-05 2024-05-15T12:56:48.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.598 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:48.598 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.837710+0000", 2024-05-15T12:56:48.598 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0057e-05 2024-05-15T12:56:48.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.598 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:48.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.837717+0000", 2024-05-15T12:56:48.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9650000000000002e-06 2024-05-15T12:56:48.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:48.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.837754+0000", 2024-05-15T12:56:48.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7404000000000003e-05 2024-05-15T12:56:48.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:48.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.837837+0000", 2024-05-15T12:56:48.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2468999999999999e-05 2024-05-15T12:56:48.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T12:56:48.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.837924+0000", 2024-05-15T12:56:48.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7070000000000005e-05 2024-05-15T12:56:48.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:56:48.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.838965+0000", 2024-05-15T12:56:48.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010411470000000001 2024-05-15T12:56:48.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.839177+0000", 2024-05-15T12:56:48.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000211914 2024-05-15T12:56:48.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.602 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.043375+0000", 2024-05-15T12:56:48.602 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.204198555 2024-05-15T12:56:48.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.602 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:56:48.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.043430+0000", 2024-05-15T12:56:48.603 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5133000000000002e-05 2024-05-15T12:56:48.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.603 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.603 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:56:48.603 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.043461+0000", 2024-05-15T12:56:48.603 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0593999999999999e-05 2024-05-15T12:56:48.603 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.603 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:48.604 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.604 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.604 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:37215 3.1 3:8a061330:::benchmark_data_smithi177_88104_object37214:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-05-15T12:56:48.604 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:53:22.842190+0000", 2024-05-15T12:56:48.604 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 205.71691385299999, 2024-05-15T12:56:48.604 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.200671393, 2024-05-15T12:56:48.604 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:48.604 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:56:48.605 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:56:48.605 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T12:56:48.605 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:56:48.605 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 37215 2024-05-15T12:56:48.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.605 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:48.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.605 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:48.605 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.842190+0000", 2024-05-15T12:56:48.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.606 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:48.606 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.842190+0000", 2024-05-15T12:56:48.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.606 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:48.606 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.842192+0000", 2024-05-15T12:56:48.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2389999999999999e-06 2024-05-15T12:56:48.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.607 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.607 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:48.607 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.842263+0000", 2024-05-15T12:56:48.607 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1099999999999994e-05 2024-05-15T12:56:48.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.607 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.607 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:48.607 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.842274+0000", 2024-05-15T12:56:48.607 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1844e-05 2024-05-15T12:56:48.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.607 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:48.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.842283+0000", 2024-05-15T12:56:48.608 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0970000000000006e-06 2024-05-15T12:56:48.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:48.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.842342+0000", 2024-05-15T12:56:48.608 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9846000000000003e-05 2024-05-15T12:56:48.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.609 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.609 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:48.609 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.842478+0000", 2024-05-15T12:56:48.609 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013552200000000001 2024-05-15T12:56:48.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.609 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.609 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 4,10", 2024-05-15T12:56:48.609 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.842632+0000", 2024-05-15T12:56:48.609 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015451500000000001 2024-05-15T12:56:48.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.610 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.610 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:56:48.610 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.843799+0000", 2024-05-15T12:56:48.610 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0011665709999999999 2024-05-15T12:56:48.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.610 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.611 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.611 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.844098+0000", 2024-05-15T12:56:48.611 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00029868400000000002 2024-05-15T12:56:48.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.611 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.611 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.611 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.042815+0000", 2024-05-15T12:56:48.611 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19871739699999999 2024-05-15T12:56:48.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.611 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.612 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:56:48.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.042842+0000", 2024-05-15T12:56:48.612 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7098e-05 2024-05-15T12:56:48.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.612 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.612 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:56:48.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.042862+0000", 2024-05-15T12:56:48.612 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.948e-05 2024-05-15T12:56:48.612 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.612 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:48.613 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.613 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.613 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:37220 3.1 3:868c097d:::benchmark_data_smithi177_88104_object37219:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-05-15T12:56:48.613 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:53:22.846699+0000", 2024-05-15T12:56:48.613 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 205.71240537700001, 2024-05-15T12:56:48.613 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19624024700000001, 2024-05-15T12:56:48.614 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:48.614 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:56:48.614 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:56:48.614 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T12:56:48.614 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:56:48.615 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 37220 2024-05-15T12:56:48.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.615 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:48.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.615 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:48.615 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.846699+0000", 2024-05-15T12:56:48.615 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.616 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:48.616 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.846699+0000", 2024-05-15T12:56:48.616 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.616 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:48.616 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.846700+0000", 2024-05-15T12:56:48.616 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1289999999999999e-06 2024-05-15T12:56:48.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.616 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:48.617 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.846787+0000", 2024-05-15T12:56:48.617 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7292000000000004e-05 2024-05-15T12:56:48.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.617 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:48.617 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.846798+0000", 2024-05-15T12:56:48.617 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0771000000000001e-05 2024-05-15T12:56:48.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.618 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.618 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:48.618 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.846804+0000", 2024-05-15T12:56:48.618 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1720000000000004e-06 2024-05-15T12:56:48.618 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.618 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.618 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:48.619 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.846861+0000", 2024-05-15T12:56:48.619 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6418000000000001e-05 2024-05-15T12:56:48.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.619 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.619 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:48.619 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.846999+0000", 2024-05-15T12:56:48.619 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013880000000000001 2024-05-15T12:56:48.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.619 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.620 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 4,10", 2024-05-15T12:56:48.620 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.847174+0000", 2024-05-15T12:56:48.620 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017457900000000001 2024-05-15T12:56:48.620 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.620 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.620 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.620 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.848585+0000", 2024-05-15T12:56:48.620 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001411151 2024-05-15T12:56:48.620 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.621 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.621 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:56:48.621 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.848604+0000", 2024-05-15T12:56:48.621 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8672e-05 2024-05-15T12:56:48.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.621 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.621 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.621 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.042884+0000", 2024-05-15T12:56:48.621 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.194280602 2024-05-15T12:56:48.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.621 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.622 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:56:48.622 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.042908+0000", 2024-05-15T12:56:48.622 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4019000000000002e-05 2024-05-15T12:56:48.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.622 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.622 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:56:48.622 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.042939+0000", 2024-05-15T12:56:48.622 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0642000000000002e-05 2024-05-15T12:56:48.622 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.622 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:48.623 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.623 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.623 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:113042 3.6 3:6a7f7ff0:::benchmark_data_smithi177_88104_object113041:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-05-15T12:56:48.623 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:54:17.961745+0000", 2024-05-15T12:56:48.623 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 150.59735878399999, 2024-05-15T12:56:48.623 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22458128399999999, 2024-05-15T12:56:48.623 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:48.623 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:56:48.624 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:56:48.624 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T12:56:48.624 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:56:48.624 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 113042 2024-05-15T12:56:48.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.624 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:48.624 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.624 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:48.624 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.961745+0000", 2024-05-15T12:56:48.624 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.625 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.625 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:48.625 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.961745+0000", 2024-05-15T12:56:48.625 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.625 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.625 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:48.625 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.961747+0000", 2024-05-15T12:56:48.625 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.736e-06 2024-05-15T12:56:48.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.626 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.626 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:48.626 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.961778+0000", 2024-05-15T12:56:48.626 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1078000000000003e-05 2024-05-15T12:56:48.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.626 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.626 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:48.626 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.961791+0000", 2024-05-15T12:56:48.627 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2998e-05 2024-05-15T12:56:48.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.627 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.627 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:48.627 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.961800+0000", 2024-05-15T12:56:48.627 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8200000000000003e-06 2024-05-15T12:56:48.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.627 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.627 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:48.627 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.962010+0000", 2024-05-15T12:56:48.627 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00021018499999999999 2024-05-15T12:56:48.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.628 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.628 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:48.628 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.962152+0000", 2024-05-15T12:56:48.628 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000141638 2024-05-15T12:56:48.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.628 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.628 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T12:56:48.628 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.962291+0000", 2024-05-15T12:56:48.628 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013942199999999999 2024-05-15T12:56:48.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.628 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.628 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:56:48.629 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.969770+0000", 2024-05-15T12:56:48.629 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0074782750000000004 2024-05-15T12:56:48.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.629 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.629 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.629 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.969801+0000", 2024-05-15T12:56:48.629 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1565e-05 2024-05-15T12:56:48.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.629 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.629 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.630 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.186218+0000", 2024-05-15T12:56:48.630 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.216416627 2024-05-15T12:56:48.630 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.630 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.630 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:56:48.630 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.186276+0000", 2024-05-15T12:56:48.630 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8455999999999997e-05 2024-05-15T12:56:48.630 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.630 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.630 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:56:48.630 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.186327+0000", 2024-05-15T12:56:48.630 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0484e-05 2024-05-15T12:56:48.631 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.631 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:48.631 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.631 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.631 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:113044 3.6 3:660f5aad:::benchmark_data_smithi177_88104_object113043:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-05-15T12:56:48.631 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:54:17.962563+0000", 2024-05-15T12:56:48.631 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 150.59654144000001, 2024-05-15T12:56:48.631 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22388765999999999, 2024-05-15T12:56:48.631 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:48.631 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:56:48.632 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:56:48.632 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T12:56:48.632 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:56:48.632 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 113044 2024-05-15T12:56:48.632 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.632 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:48.632 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.632 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:48.633 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.962563+0000", 2024-05-15T12:56:48.633 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.633 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.633 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.633 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:48.633 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.962563+0000", 2024-05-15T12:56:48.633 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.633 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.633 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.633 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:48.633 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.962564+0000", 2024-05-15T12:56:48.633 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.78e-07 2024-05-15T12:56:48.634 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.634 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.634 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:48.634 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.962621+0000", 2024-05-15T12:56:48.634 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7899999999999998e-05 2024-05-15T12:56:48.634 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.634 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.634 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:48.634 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.962630+0000", 2024-05-15T12:56:48.634 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9870000000000005e-06 2024-05-15T12:56:48.634 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.634 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.635 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:48.635 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.962637+0000", 2024-05-15T12:56:48.635 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4300000000000003e-06 2024-05-15T12:56:48.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.635 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.635 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:48.635 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.965856+0000", 2024-05-15T12:56:48.635 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032189359999999999 2024-05-15T12:56:48.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.635 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.635 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:48.635 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.966002+0000", 2024-05-15T12:56:48.635 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014567599999999999 2024-05-15T12:56:48.636 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.636 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.636 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T12:56:48.636 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.966173+0000", 2024-05-15T12:56:48.636 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017171299999999999 2024-05-15T12:56:48.636 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.636 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.636 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.636 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.969945+0000", 2024-05-15T12:56:48.636 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0037720589999999999 2024-05-15T12:56:48.636 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.636 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.636 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:56:48.637 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.970373+0000", 2024-05-15T12:56:48.637 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00042798199999999998 2024-05-15T12:56:48.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.637 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.637 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.637 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.186365+0000", 2024-05-15T12:56:48.637 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.21599131499999999 2024-05-15T12:56:48.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.637 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.637 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:56:48.637 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.186414+0000", 2024-05-15T12:56:48.637 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9097000000000001e-05 2024-05-15T12:56:48.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.638 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.638 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:56:48.638 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.186450+0000", 2024-05-15T12:56:48.638 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6687000000000003e-05 2024-05-15T12:56:48.638 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.638 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:48.638 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.638 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.638 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.638 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:113073 3.6 3:610a8193:::benchmark_data_smithi177_88104_object113072:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-05-15T12:56:48.638 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:54:17.989442+0000", 2024-05-15T12:56:48.639 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 150.56966215899999, 2024-05-15T12:56:48.639 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.200631694, 2024-05-15T12:56:48.639 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:48.639 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:56:48.639 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:56:48.639 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T12:56:48.639 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:56:48.639 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 113073 2024-05-15T12:56:48.639 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.639 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:48.640 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.640 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:48.640 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989442+0000", 2024-05-15T12:56:48.640 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.640 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.640 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.640 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:48.640 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989442+0000", 2024-05-15T12:56:48.640 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.640 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.640 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989443+0000", 2024-05-15T12:56:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.274e-06 2024-05-15T12:56:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989506+0000", 2024-05-15T12:56:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3244000000000006e-05 2024-05-15T12:56:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:48.641 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989520+0000", 2024-05-15T12:56:48.642 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3532e-05 2024-05-15T12:56:48.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.642 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.642 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:48.642 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989526+0000", 2024-05-15T12:56:48.642 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0440000000000001e-06 2024-05-15T12:56:48.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.642 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:48.643 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989602+0000", 2024-05-15T12:56:48.643 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6211999999999995e-05 2024-05-15T12:56:48.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:48.643 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989736+0000", 2024-05-15T12:56:48.643 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013396999999999999 2024-05-15T12:56:48.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T12:56:48.643 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989890+0000", 2024-05-15T12:56:48.643 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015393899999999999 2024-05-15T12:56:48.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.644 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.644 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:56:48.644 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.000201+0000", 2024-05-15T12:56:48.644 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010310332 2024-05-15T12:56:48.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.644 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.644 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.644 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.000307+0000", 2024-05-15T12:56:48.644 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000106217 2024-05-15T12:56:48.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.644 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.644 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.644 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.189969+0000", 2024-05-15T12:56:48.645 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.18966257 2024-05-15T12:56:48.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.645 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.645 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:56:48.645 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.190028+0000", 2024-05-15T12:56:48.645 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9178000000000002e-05 2024-05-15T12:56:48.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.645 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.645 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:56:48.645 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.190074+0000", 2024-05-15T12:56:48.645 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5182000000000002e-05 2024-05-15T12:56:48.645 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.645 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:48.646 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.646 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.646 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.646 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:113074 3.6 3:64bf8e0c:::benchmark_data_smithi177_88104_object113073:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-05-15T12:56:48.646 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:54:17.989550+0000", 2024-05-15T12:56:48.646 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 150.56955386499999, 2024-05-15T12:56:48.646 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.200655903, 2024-05-15T12:56:48.646 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:48.646 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:56:48.646 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:56:48.646 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T12:56:48.646 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:56:48.646 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 113074 2024-05-15T12:56:48.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.647 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:48.647 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.647 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:48.647 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989550+0000", 2024-05-15T12:56:48.647 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.647 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.647 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:48.647 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989550+0000", 2024-05-15T12:56:48.647 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.647 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.647 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:48.648 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989551+0000", 2024-05-15T12:56:48.648 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8599999999999997e-07 2024-05-15T12:56:48.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.648 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.648 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:48.648 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989590+0000", 2024-05-15T12:56:48.648 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9113000000000003e-05 2024-05-15T12:56:48.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.648 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.648 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:48.648 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989600+0000", 2024-05-15T12:56:48.648 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4890000000000001e-06 2024-05-15T12:56:48.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.649 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.649 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:48.649 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989607+0000", 2024-05-15T12:56:48.649 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9979999999999997e-06 2024-05-15T12:56:48.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.649 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.649 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:48.649 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.993352+0000", 2024-05-15T12:56:48.649 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0037452230000000002 2024-05-15T12:56:48.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.649 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.649 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:48.649 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.993454+0000", 2024-05-15T12:56:48.650 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000102515 2024-05-15T12:56:48.650 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.650 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.650 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T12:56:48.650 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.993553+0000", 2024-05-15T12:56:48.650 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8948000000000002e-05 2024-05-15T12:56:48.650 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.650 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.650 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:56:48.650 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.000211+0000", 2024-05-15T12:56:48.650 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0066575269999999999 2024-05-15T12:56:48.650 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.650 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.651 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.651 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.000339+0000", 2024-05-15T12:56:48.651 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012788900000000001 2024-05-15T12:56:48.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.651 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.651 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.651 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.190133+0000", 2024-05-15T12:56:48.651 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.18979415099999999 2024-05-15T12:56:48.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.651 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.651 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:56:48.651 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.190168+0000", 2024-05-15T12:56:48.651 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5438999999999997e-05 2024-05-15T12:56:48.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.652 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.652 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:56:48.652 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.190206+0000", 2024-05-15T12:56:48.652 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7824999999999998e-05 2024-05-15T12:56:48.652 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.652 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:48.652 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.652 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.652 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:113076 3.6 3:698d8b94:::benchmark_data_smithi177_88104_object113075:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-05-15T12:56:48.652 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:54:17.993185+0000", 2024-05-15T12:56:48.652 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 150.56591958600001, 2024-05-15T12:56:48.652 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.2013375, 2024-05-15T12:56:48.652 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:48.653 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:56:48.653 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:56:48.653 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T12:56:48.653 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:56:48.653 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 113076 2024-05-15T12:56:48.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.653 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:48.653 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.653 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:48.653 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.993185+0000", 2024-05-15T12:56:48.653 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.653 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.654 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:48.654 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.993185+0000", 2024-05-15T12:56:48.654 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.654 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.654 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:48.654 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.993186+0000", 2024-05-15T12:56:48.654 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4890000000000001e-06 2024-05-15T12:56:48.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.654 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.654 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:48.654 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.993241+0000", 2024-05-15T12:56:48.654 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5182000000000001e-05 2024-05-15T12:56:48.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.655 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.655 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:48.655 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.993256+0000", 2024-05-15T12:56:48.655 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4749999999999999e-05 2024-05-15T12:56:48.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.655 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.655 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:48.655 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.993264+0000", 2024-05-15T12:56:48.655 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3070000000000003e-06 2024-05-15T12:56:48.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.656 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.656 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:48.656 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.996424+0000", 2024-05-15T12:56:48.656 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031594259999999999 2024-05-15T12:56:48.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.656 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.656 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:48.656 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.996497+0000", 2024-05-15T12:56:48.656 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3330000000000004e-05 2024-05-15T12:56:48.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.657 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.657 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T12:56:48.657 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.996636+0000", 2024-05-15T12:56:48.657 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013876000000000001 2024-05-15T12:56:48.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.657 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.657 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:56:48.657 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.001265+0000", 2024-05-15T12:56:48.657 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046289249999999999 2024-05-15T12:56:48.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.658 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.658 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.658 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.006975+0000", 2024-05-15T12:56:48.658 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0057106680000000003 2024-05-15T12:56:48.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.658 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.658 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.658 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.194420+0000", 2024-05-15T12:56:48.658 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.18744422599999999 2024-05-15T12:56:48.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.658 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.659 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:56:48.659 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.194476+0000", 2024-05-15T12:56:48.659 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6212000000000003e-05 2024-05-15T12:56:48.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.659 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.659 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:56:48.659 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.194522+0000", 2024-05-15T12:56:48.659 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6224999999999999e-05 2024-05-15T12:56:48.659 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.659 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:48.659 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.659 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.660 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:113090 3.6 3:6cf5823a:::benchmark_data_smithi177_88104_object113089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-05-15T12:56:48.660 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:54:18.002227+0000", 2024-05-15T12:56:48.660 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 150.55687717199999, 2024-05-15T12:56:48.660 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19613824999999999, 2024-05-15T12:56:48.660 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:48.660 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:56:48.660 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:56:48.660 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T12:56:48.660 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:56:48.660 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 113090 2024-05-15T12:56:48.660 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.660 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:48.661 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.661 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:48.661 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.002227+0000", 2024-05-15T12:56:48.661 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.661 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.661 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:48.661 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.002227+0000", 2024-05-15T12:56:48.661 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.661 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.661 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:48.661 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.002228+0000", 2024-05-15T12:56:48.662 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.266e-06 2024-05-15T12:56:48.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.662 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.662 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:48.662 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.002301+0000", 2024-05-15T12:56:48.662 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2909000000000003e-05 2024-05-15T12:56:48.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.662 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.662 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:48.662 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.002318+0000", 2024-05-15T12:56:48.662 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6526000000000001e-05 2024-05-15T12:56:48.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.662 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.663 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:48.663 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.002326+0000", 2024-05-15T12:56:48.663 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3340000000000002e-06 2024-05-15T12:56:48.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.663 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.663 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:48.663 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.002405+0000", 2024-05-15T12:56:48.663 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9157000000000001e-05 2024-05-15T12:56:48.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.663 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.663 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:48.663 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.002518+0000", 2024-05-15T12:56:48.663 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011309 2024-05-15T12:56:48.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.664 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.664 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T12:56:48.664 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.002653+0000", 2024-05-15T12:56:48.664 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013430899999999999 2024-05-15T12:56:48.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.664 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.664 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:56:48.664 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.007906+0000", 2024-05-15T12:56:48.664 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.005253443 2024-05-15T12:56:48.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.664 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.664 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.664 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.010931+0000", 2024-05-15T12:56:48.664 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0030248190000000002 2024-05-15T12:56:48.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.665 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.665 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.665 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.198239+0000", 2024-05-15T12:56:48.665 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.187308266 2024-05-15T12:56:48.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.665 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.665 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:56:48.665 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.198312+0000", 2024-05-15T12:56:48.665 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2860000000000004e-05 2024-05-15T12:56:48.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.665 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.665 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:56:48.665 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.198365+0000", 2024-05-15T12:56:48.666 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3270999999999999e-05 2024-05-15T12:56:48.666 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.666 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:48.666 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.666 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.666 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:113101 3.6 3:6a24830d:::benchmark_data_smithi177_88104_object113100:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-05-15T12:56:48.666 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:54:18.016206+0000", 2024-05-15T12:56:48.666 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 150.54289771099999, 2024-05-15T12:56:48.666 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19138814700000001, 2024-05-15T12:56:48.666 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:48.666 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:56:48.666 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:56:48.667 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T12:56:48.667 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:56:48.667 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 113101 2024-05-15T12:56:48.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.667 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:48.667 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.667 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:48.667 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.016206+0000", 2024-05-15T12:56:48.667 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.667 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.667 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:48.667 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.016206+0000", 2024-05-15T12:56:48.667 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.668 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.668 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:48.668 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.016208+0000", 2024-05-15T12:56:48.668 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6509999999999999e-06 2024-05-15T12:56:48.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.668 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.668 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:48.668 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.016277+0000", 2024-05-15T12:56:48.668 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8569e-05 2024-05-15T12:56:48.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.668 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.669 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:48.669 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.016295+0000", 2024-05-15T12:56:48.669 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8224999999999999e-05 2024-05-15T12:56:48.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.669 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.669 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:48.669 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.016303+0000", 2024-05-15T12:56:48.669 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5029999999999997e-06 2024-05-15T12:56:48.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.669 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.669 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:48.669 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.016363+0000", 2024-05-15T12:56:48.669 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0074000000000001e-05 2024-05-15T12:56:48.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.670 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.670 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:48.670 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.016509+0000", 2024-05-15T12:56:48.670 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014593200000000001 2024-05-15T12:56:48.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.670 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.670 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T12:56:48.670 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.016656+0000", 2024-05-15T12:56:48.670 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014697899999999999 2024-05-15T12:56:48.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.670 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.670 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:56:48.671 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.019804+0000", 2024-05-15T12:56:48.671 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031474430000000002 2024-05-15T12:56:48.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.671 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.671 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.671 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.023329+0000", 2024-05-15T12:56:48.671 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003524957 2024-05-15T12:56:48.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.671 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.671 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.671 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.207502+0000", 2024-05-15T12:56:48.671 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.184172786 2024-05-15T12:56:48.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.672 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.672 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:56:48.672 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.207560+0000", 2024-05-15T12:56:48.672 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8508999999999998e-05 2024-05-15T12:56:48.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.672 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.672 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:56:48.672 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.207595+0000", 2024-05-15T12:56:48.672 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4518999999999999e-05 2024-05-15T12:56:48.672 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.672 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:48.673 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.673 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.673 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:234133 3.6 3:6bfad9ff:::benchmark_data_smithi177_88104_object100511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T12:56:48.673 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:44.636070+0000", 2024-05-15T12:56:48.673 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 3.9230341649999998, 2024-05-15T12:56:48.673 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22423902700000001, 2024-05-15T12:56:48.673 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:48.673 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:56:48.673 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:56:48.673 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T12:56:48.673 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:56:48.673 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 234133 2024-05-15T12:56:48.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.674 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:48.674 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.674 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:48.674 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636070+0000", 2024-05-15T12:56:48.674 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.674 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.674 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:48.674 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636070+0000", 2024-05-15T12:56:48.674 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.674 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.674 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:48.675 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636071+0000", 2024-05-15T12:56:48.675 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3900000000000004e-07 2024-05-15T12:56:48.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.675 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.675 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:48.675 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636076+0000", 2024-05-15T12:56:48.675 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7040000000000002e-06 2024-05-15T12:56:48.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.675 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.675 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:48.675 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636087+0000", 2024-05-15T12:56:48.675 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1718e-05 2024-05-15T12:56:48.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.675 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.676 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:48.676 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636098+0000", 2024-05-15T12:56:48.676 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.084e-05 2024-05-15T12:56:48.676 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.676 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.676 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:48.676 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.641220+0000", 2024-05-15T12:56:48.676 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0051218169999999999 2024-05-15T12:56:48.676 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.676 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.677 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:48.677 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.641326+0000", 2024-05-15T12:56:48.677 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010569900000000001 2024-05-15T12:56:48.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.677 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T12:56:48.677 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.641435+0000", 2024-05-15T12:56:48.677 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000109162 2024-05-15T12:56:48.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.677 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.677 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.655524+0000", 2024-05-15T12:56:48.677 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014089121 2024-05-15T12:56:48.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.678 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:56:48.678 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.662050+0000", 2024-05-15T12:56:48.678 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0065262009999999997 2024-05-15T12:56:48.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.678 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.678 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.860197+0000", 2024-05-15T12:56:48.678 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19814722300000001 2024-05-15T12:56:48.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.678 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:56:48.678 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.860258+0000", 2024-05-15T12:56:48.679 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0826999999999999e-05 2024-05-15T12:56:48.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.679 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.679 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:56:48.679 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.860309+0000", 2024-05-15T12:56:48.679 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0875999999999999e-05 2024-05-15T12:56:48.679 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.679 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:48.679 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.679 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.679 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:234137 3.6 3:64a7160a:::benchmark_data_smithi177_88104_object100515:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T12:56:48.679 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:44.636929+0000", 2024-05-15T12:56:48.679 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 3.9221755319999998, 2024-05-15T12:56:48.680 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22352234700000001, 2024-05-15T12:56:48.680 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:48.680 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:56:48.680 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:56:48.680 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T12:56:48.680 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:56:48.680 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 234137 2024-05-15T12:56:48.680 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.680 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:48.680 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.680 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:48.680 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636929+0000", 2024-05-15T12:56:48.680 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.681 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.681 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.681 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:48.681 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636929+0000", 2024-05-15T12:56:48.681 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.681 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.681 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.681 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:48.681 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636930+0000", 2024-05-15T12:56:48.681 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.057e-06 2024-05-15T12:56:48.681 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.681 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.681 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:48.682 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636934+0000", 2024-05-15T12:56:48.682 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4750000000000004e-06 2024-05-15T12:56:48.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.682 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.682 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:48.682 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636945+0000", 2024-05-15T12:56:48.682 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.094e-05 2024-05-15T12:56:48.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.682 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.682 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:48.682 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636953+0000", 2024-05-15T12:56:48.682 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7400000000000004e-06 2024-05-15T12:56:48.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.683 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.683 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:48.683 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650374+0000", 2024-05-15T12:56:48.683 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013421535 2024-05-15T12:56:48.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.683 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.683 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:48.683 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650489+0000", 2024-05-15T12:56:48.683 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000114919 2024-05-15T12:56:48.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.683 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.684 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T12:56:48.684 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650628+0000", 2024-05-15T12:56:48.684 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013872799999999999 2024-05-15T12:56:48.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.684 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.684 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.684 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.657205+0000", 2024-05-15T12:56:48.684 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0065768520000000002 2024-05-15T12:56:48.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.684 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.684 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:56:48.684 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.666120+0000", 2024-05-15T12:56:48.685 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0089147159999999996 2024-05-15T12:56:48.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.685 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.685 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.685 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.860355+0000", 2024-05-15T12:56:48.685 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19423544000000001 2024-05-15T12:56:48.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.685 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.685 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:56:48.685 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.860409+0000", 2024-05-15T12:56:48.686 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4172999999999998e-05 2024-05-15T12:56:48.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.686 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.686 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:56:48.686 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.860451+0000", 2024-05-15T12:56:48.686 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1771999999999999e-05 2024-05-15T12:56:48.686 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.686 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:48.686 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.686 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.686 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:234139 3.6 3:67f4ccfa:::benchmark_data_smithi177_88104_object100517:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T12:56:48.686 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:44.642307+0000", 2024-05-15T12:56:48.687 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 3.9167974540000001, 2024-05-15T12:56:48.687 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22900473900000001, 2024-05-15T12:56:48.687 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:48.687 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:56:48.687 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:56:48.687 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T12:56:48.687 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:56:48.687 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 234139 2024-05-15T12:56:48.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.687 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:48.687 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.687 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:48.688 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.642307+0000", 2024-05-15T12:56:48.688 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.688 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.688 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:48.688 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.642307+0000", 2024-05-15T12:56:48.688 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.688 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.688 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:48.688 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.642308+0000", 2024-05-15T12:56:48.688 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2100000000000001e-06 2024-05-15T12:56:48.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.689 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.689 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:48.689 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.642311+0000", 2024-05-15T12:56:48.689 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4630000000000001e-06 2024-05-15T12:56:48.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.689 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.689 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:48.689 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.642320+0000", 2024-05-15T12:56:48.689 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3119999999999994e-06 2024-05-15T12:56:48.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.689 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.689 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:48.689 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.642326+0000", 2024-05-15T12:56:48.690 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1709999999999999e-06 2024-05-15T12:56:48.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.690 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.690 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:48.690 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.657235+0000", 2024-05-15T12:56:48.690 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014908889999999999 2024-05-15T12:56:48.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.690 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.690 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:48.690 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.657392+0000", 2024-05-15T12:56:48.690 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015708900000000001 2024-05-15T12:56:48.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.690 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.691 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T12:56:48.691 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.657525+0000", 2024-05-15T12:56:48.691 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000132764 2024-05-15T12:56:48.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.691 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.691 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.691 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.661929+0000", 2024-05-15T12:56:48.691 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0044041469999999997 2024-05-15T12:56:48.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.691 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.691 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:56:48.691 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.666168+0000", 2024-05-15T12:56:48.691 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0042391920000000001 2024-05-15T12:56:48.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.692 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.692 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.871182+0000", 2024-05-15T12:56:48.692 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20501413499999999 2024-05-15T12:56:48.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.692 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:56:48.692 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.871250+0000", 2024-05-15T12:56:48.692 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8338999999999997e-05 2024-05-15T12:56:48.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.692 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:56:48.692 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.871311+0000", 2024-05-15T12:56:48.693 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1026999999999997e-05 2024-05-15T12:56:48.693 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.693 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:48.693 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.693 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.693 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:234142 3.6 3:6ff8b345:::benchmark_data_smithi177_88104_object100520:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T12:56:48.693 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:44.650217+0000", 2024-05-15T12:56:48.693 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 3.908887317, 2024-05-15T12:56:48.693 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22408456900000001, 2024-05-15T12:56:48.693 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:48.693 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:56:48.693 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:56:48.693 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T12:56:48.694 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:56:48.694 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 234142 2024-05-15T12:56:48.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.694 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:48.694 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.694 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:48.694 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650217+0000", 2024-05-15T12:56:48.694 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.694 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.694 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:48.694 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650217+0000", 2024-05-15T12:56:48.694 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650218+0000", 2024-05-15T12:56:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.316e-06 2024-05-15T12:56:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650222+0000", 2024-05-15T12:56:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5319999999999998e-06 2024-05-15T12:56:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650230+0000", 2024-05-15T12:56:48.696 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7090000000000008e-06 2024-05-15T12:56:48.696 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.696 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:56:48.697 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.697 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:48.697 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650236+0000", 2024-05-15T12:56:48.697 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6899999999999997e-06 2024-05-15T12:56:48.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.698 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.698 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:48.698 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.662077+0000", 2024-05-15T12:56:48.698 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.01184057 2024-05-15T12:56:48.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.698 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.698 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:48.698 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.662237+0000", 2024-05-15T12:56:48.698 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016033200000000001 2024-05-15T12:56:48.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.698 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.698 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T12:56:48.699 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.662332+0000", 2024-05-15T12:56:48.699 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5199999999999997e-05 2024-05-15T12:56:48.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.699 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.699 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:56:48.699 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.666359+0000", 2024-05-15T12:56:48.699 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.004027325 2024-05-15T12:56:48.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.699 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.699 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.699 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.668031+0000", 2024-05-15T12:56:48.699 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016717629999999999 2024-05-15T12:56:48.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.700 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.700 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.700 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.874188+0000", 2024-05-15T12:56:48.700 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20615639699999999 2024-05-15T12:56:48.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.700 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.700 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:56:48.700 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.874247+0000", 2024-05-15T12:56:48.700 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9775000000000003e-05 2024-05-15T12:56:48.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.700 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.700 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:56:48.700 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.874301+0000", 2024-05-15T12:56:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3959999999999998e-05 2024-05-15T12:56:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:234166 3.6 3:65e82527:::benchmark_data_smithi177_88104_object100544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T12:56:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:44.676834+0000", 2024-05-15T12:56:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 3.8822703409999999, 2024-05-15T12:56:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.214610197, 2024-05-15T12:56:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:56:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:56:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T12:56:48.702 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:56:48.702 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 234166 2024-05-15T12:56:48.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.702 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:48.702 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.702 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:48.702 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.676834+0000", 2024-05-15T12:56:48.702 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.702 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.702 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:48.702 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.676834+0000", 2024-05-15T12:56:48.702 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.703 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.703 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:48.703 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.676836+0000", 2024-05-15T12:56:48.703 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0260000000000002e-06 2024-05-15T12:56:48.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.703 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.703 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:48.703 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.676852+0000", 2024-05-15T12:56:48.703 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.647e-05 2024-05-15T12:56:48.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.703 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.703 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:48.703 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.676869+0000", 2024-05-15T12:56:48.704 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6915999999999999e-05 2024-05-15T12:56:48.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.704 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.704 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:48.704 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.676888+0000", 2024-05-15T12:56:48.704 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8437999999999999e-05 2024-05-15T12:56:48.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.704 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.704 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:48.704 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.676954+0000", 2024-05-15T12:56:48.704 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6723000000000005e-05 2024-05-15T12:56:48.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.704 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.705 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:48.705 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.677089+0000", 2024-05-15T12:56:48.705 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013486300000000001 2024-05-15T12:56:48.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.705 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.705 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T12:56:48.705 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.677239+0000", 2024-05-15T12:56:48.705 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014941600000000001 2024-05-15T12:56:48.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.705 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.705 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.705 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.683263+0000", 2024-05-15T12:56:48.705 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0060247510000000001 2024-05-15T12:56:48.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.706 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.706 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:56:48.706 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.683276+0000", 2024-05-15T12:56:48.706 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2281e-05 2024-05-15T12:56:48.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.706 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.706 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.706 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.891351+0000", 2024-05-15T12:56:48.706 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20807553100000001 2024-05-15T12:56:48.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.706 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.706 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:56:48.706 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.891400+0000", 2024-05-15T12:56:48.706 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8640999999999997e-05 2024-05-15T12:56:48.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.707 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.707 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:56:48.707 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.891444+0000", 2024-05-15T12:56:48.707 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4141000000000003e-05 2024-05-15T12:56:48.707 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.707 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:48.707 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.707 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.707 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:234181 3.6 3:635178a9:::benchmark_data_smithi177_88104_object100559:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T12:56:48.707 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:44.691125+0000", 2024-05-15T12:56:48.707 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 3.867978753, 2024-05-15T12:56:48.707 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20043533499999999, 2024-05-15T12:56:48.708 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:48.708 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:56:48.708 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:56:48.708 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T12:56:48.708 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:56:48.708 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 234181 2024-05-15T12:56:48.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.708 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:48.708 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.708 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:48.708 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691125+0000", 2024-05-15T12:56:48.708 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.708 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.709 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:48.709 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691125+0000", 2024-05-15T12:56:48.709 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:48.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.709 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.709 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:48.709 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691127+0000", 2024-05-15T12:56:48.709 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.392e-06 2024-05-15T12:56:48.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.709 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.709 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:48.709 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691131+0000", 2024-05-15T12:56:48.709 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5020000000000003e-06 2024-05-15T12:56:48.710 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.710 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.710 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:48.710 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691142+0000", 2024-05-15T12:56:48.710 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0478e-05 2024-05-15T12:56:48.710 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.710 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.710 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:48.710 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691149+0000", 2024-05-15T12:56:48.710 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9689999999999997e-06 2024-05-15T12:56:48.710 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.710 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.710 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:48.710 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691205+0000", 2024-05-15T12:56:48.711 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6137999999999997e-05 2024-05-15T12:56:48.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.711 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.711 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:48.711 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691332+0000", 2024-05-15T12:56:48.711 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012724599999999999 2024-05-15T12:56:48.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.711 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.711 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T12:56:48.711 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691458+0000", 2024-05-15T12:56:48.711 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012549999999999999 2024-05-15T12:56:48.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.711 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.712 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.712 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.696662+0000", 2024-05-15T12:56:48.712 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0052043760000000001 2024-05-15T12:56:48.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.712 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.712 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:56:48.712 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.697207+0000", 2024-05-15T12:56:48.712 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00054491700000000001 2024-05-15T12:56:48.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.712 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.712 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:56:48.712 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.891478+0000", 2024-05-15T12:56:48.712 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19427066900000001 2024-05-15T12:56:48.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.713 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.713 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:56:48.713 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.891527+0000", 2024-05-15T12:56:48.713 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9308999999999998e-05 2024-05-15T12:56:48.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:48.713 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:48.713 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:56:48.713 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.891561+0000", 2024-05-15T12:56:48.713 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3838999999999999e-05 2024-05-15T12:56:48.713 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.713 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:48.713 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.713 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:48.714 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:48.714 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T12:56:48.742 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:48.772 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:56:48.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:48.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:56:48.879 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:56:48.907 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:48.927 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T12:56:48.930 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:56:48.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:48.933 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:56:49.035 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:56:49.067 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:49.098 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:56:49.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:49.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:56:49.206 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:56:49.262 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:49.477 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:56:49.750 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:56:49.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:49.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:56:49.766 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-05-15T12:56:49.844 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:49 smithi027 ceph-mon[39795]: pgmap v811: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.9 GiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 890 op/s; 1955/92463 objects misplaced (2.114%) 2024-05-15T12:56:49.861 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:56:49.888 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:49.910 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:56:49.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:49.916 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:56:50.017 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:56:50.101 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:50.156 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:56:50.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:50.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:56:50.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:49 smithi177 ceph-mon[44303]: pgmap v811: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.9 GiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 890 op/s; 1955/92463 objects misplaced (2.114%) 2024-05-15T12:56:50.261 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:56:50.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:49 smithi120 ceph-mon[43961]: pgmap v811: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.9 GiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 890 op/s; 1955/92463 objects misplaced (2.114%) 2024-05-15T12:56:50.290 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:50.314 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:56:50.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:50.316 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:56:50.417 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:56:50.504 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:50.527 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:56:50.546 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:56:50.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:50.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:56:50.600 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:56:50.651 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:56:50.658 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:56:50.696 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:50.731 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:56:50.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:50.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:56:50.840 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:56:50.894 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.2 to 1 (10000) 2024-05-15T12:56:50.894 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:50.946 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:56:50.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:50.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:56:51.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:50 smithi120 ceph-mon[43961]: 3.4 scrub starts 2024-05-15T12:56:51.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:50 smithi120 ceph-mon[43961]: 3.4 scrub ok 2024-05-15T12:56:51.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:50 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1887031867' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T12:56:51.050 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:56:51.135 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:51.168 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:56:51.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:51.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:56:51.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:50 smithi177 ceph-mon[44303]: 3.4 scrub starts 2024-05-15T12:56:51.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:50 smithi177 ceph-mon[44303]: 3.4 scrub ok 2024-05-15T12:56:51.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:50 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1887031867' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T12:56:51.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:50 smithi027 ceph-mon[39795]: 3.4 scrub starts 2024-05-15T12:56:51.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:50 smithi027 ceph-mon[39795]: 3.4 scrub ok 2024-05-15T12:56:51.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:50 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1887031867' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T12:56:51.278 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:56:51.319 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:51.351 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:56:51.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:51.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:56:51.463 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:56:51.499 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:51.528 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:56:51.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:51.531 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:56:51.632 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:56:51.700 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:51.854 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:51 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1887031867' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T12:56:51.855 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:51 smithi177 ceph-mon[44303]: osdmap e297: 12 total, 12 up, 10 in 2024-05-15T12:56:51.855 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:51 smithi177 ceph-mon[44303]: pgmap v813: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.8 GiB data, 7.5 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 741 op/s; 1955/88593 objects misplaced (2.207%) 2024-05-15T12:56:51.931 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:56:51.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:51.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:56:51.985 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:51 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1887031867' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T12:56:51.985 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:51 smithi027 ceph-mon[39795]: osdmap e297: 12 total, 12 up, 10 in 2024-05-15T12:56:51.985 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:51 smithi027 ceph-mon[39795]: pgmap v813: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.8 GiB data, 7.5 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 741 op/s; 1955/88593 objects misplaced (2.207%) 2024-05-15T12:56:52.046 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:56:52.107 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:52.129 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:56:52.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:52.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:56:52.135 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:51 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1887031867' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T12:56:52.135 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:51 smithi120 ceph-mon[43961]: osdmap e297: 12 total, 12 up, 10 in 2024-05-15T12:56:52.135 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:51 smithi120 ceph-mon[43961]: pgmap v813: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.8 GiB data, 7.5 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 741 op/s; 1955/88593 objects misplaced (2.207%) 2024-05-15T12:56:52.235 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:56:52.276 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:52.311 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:56:52.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:52.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:56:52.421 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:56:52.464 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:52.485 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:56:52.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:52.488 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:56:52.589 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:56:52.621 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:52.644 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:56:52.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:52.647 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:56:52.748 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:56:52.774 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:52.800 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:56:52.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:52.803 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:56:52.887 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 3 1 2024-05-15T12:56:52.904 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:56:52.947 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:52.973 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:56:52.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:52.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:56:53.078 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:56:53.107 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:53.107 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:52 smithi177 ceph-mon[44303]: 3.9 scrub starts 2024-05-15T12:56:53.107 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:52 smithi177 ceph-mon[44303]: 3.9 scrub ok 2024-05-15T12:56:53.113 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:52 smithi027 ceph-mon[39795]: 3.9 scrub starts 2024-05-15T12:56:53.113 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:52 smithi027 ceph-mon[39795]: 3.9 scrub ok 2024-05-15T12:56:53.126 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:56:53.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:53.129 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:56:53.230 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:56:53.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:52 smithi120 ceph-mon[43961]: 3.9 scrub starts 2024-05-15T12:56:53.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:52 smithi120 ceph-mon[43961]: 3.9 scrub ok 2024-05-15T12:56:53.292 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:53.318 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:56:53.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:53.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:56:53.427 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:56:53.497 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:53.531 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:56:53.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:53.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:56:53.639 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:56:53.670 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:53.696 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:56:53.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:53.700 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:56:53.801 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:56:53.853 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:53.885 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:56:53.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:53.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:56:53.995 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:56:54.043 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:56:54.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:54.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:56:54.130 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:54 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:56:54.130 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:54 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:56:54.131 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:54 smithi120 ceph-mon[43961]: pgmap v814: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.7 GiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 807 op/s; 1694/83151 objects misplaced (2.037%) 2024-05-15T12:56:54.187 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:56:54.227 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:54.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:54 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:56:54.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:54 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:56:54.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:54 smithi027 ceph-mon[39795]: pgmap v814: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.7 GiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 807 op/s; 1694/83151 objects misplaced (2.037%) 2024-05-15T12:56:54.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:54 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:56:54.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:54 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:56:54.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:54 smithi177 ceph-mon[44303]: pgmap v814: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.7 GiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 807 op/s; 1694/83151 objects misplaced (2.037%) 2024-05-15T12:56:54.587 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:56:54.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:54.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:56:54.695 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:56:54.742 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:54.773 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:56:54.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:54.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:56:54.881 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:56:54.912 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:54.937 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:56:54.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:54.950 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:56:55.051 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:56:55.139 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:55.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:55 smithi177 ceph-mon[44303]: osdmap e298: 12 total, 12 up, 10 in 2024-05-15T12:56:55.291 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:56:55.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:55.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:56:55.396 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:56:55.397 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:55 smithi027 ceph-mon[39795]: osdmap e298: 12 total, 12 up, 10 in 2024-05-15T12:56:55.459 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:55.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:55 smithi120 ceph-mon[43961]: osdmap e298: 12 total, 12 up, 10 in 2024-05-15T12:56:55.677 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:56:55.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:55.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:56:55.785 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:56:55.811 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:55.834 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:56:55.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:55.838 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:56:55.851 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T12:56:55.851 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T12:56:55.851 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.851 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:243504 3.8 3.d6ddf4c8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-05-15T12:56:55.851 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:55.801203+0000", 2024-05-15T12:56:55.851 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.0018387869999999999, 2024-05-15T12:56:55.851 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0018667759999999999, 2024-05-15T12:56:55.851 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:55.851 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "queued for pg", 2024-05-15T12:56:55.852 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:56:55.852 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T12:56:55.852 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:56:55.852 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 243504 2024-05-15T12:56:55.852 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.852 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:55.852 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.852 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:55.852 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801203+0000", 2024-05-15T12:56:55.852 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:55.852 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.852 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.852 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:55.852 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801203+0000", 2024-05-15T12:56:55.853 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:55.853 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.853 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.853 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:55.853 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801203+0000", 2024-05-15T12:56:55.853 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2099999999999997e-07 2024-05-15T12:56:55.853 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.853 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.853 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:55.853 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801206+0000", 2024-05-15T12:56:55.853 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9610000000000001e-06 2024-05-15T12:56:55.854 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.854 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.854 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:55.854 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801214+0000", 2024-05-15T12:56:55.854 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6279999999999996e-06 2024-05-15T12:56:55.854 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.854 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.854 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:55.854 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801219+0000", 2024-05-15T12:56:55.854 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.48e-06 2024-05-15T12:56:55.854 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:55.854 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:55.855 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:55.855 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.855 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.855 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24832.0:243497 3.6 e298/219)", 2024-05-15T12:56:55.855 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:55.801210+0000", 2024-05-15T12:56:55.855 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.001831137, 2024-05-15T12:56:55.855 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019352659999999999, 2024-05-15T12:56:55.855 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:55.855 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "queued for pg", 2024-05-15T12:56:55.855 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:55.855 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.855 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:55.855 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801210+0000", 2024-05-15T12:56:55.855 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:55.856 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.856 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.856 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:55.856 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801210+0000", 2024-05-15T12:56:55.856 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:55.856 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.856 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.856 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:55.856 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801211+0000", 2024-05-15T12:56:55.856 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8999999999999995e-07 2024-05-15T12:56:55.856 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.856 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.856 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:55.856 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801219+0000", 2024-05-15T12:56:55.857 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4649999999999997e-06 2024-05-15T12:56:55.857 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.857 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.857 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:55.857 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801230+0000", 2024-05-15T12:56:55.857 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1259e-05 2024-05-15T12:56:55.857 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.857 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.857 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:55.857 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801237+0000", 2024-05-15T12:56:55.857 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3379999999999999e-06 2024-05-15T12:56:55.857 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:55.857 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:55.858 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:55.858 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.858 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.858 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24832.0:243495 3.a e298/219 3:597ee718:::benchmark_data_smithi177_88104_object109873:head v 298'15322, mlcod=298'15319)", 2024-05-15T12:56:55.858 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:55.801262+0000", 2024-05-15T12:56:55.858 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.001779154, 2024-05-15T12:56:55.858 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019332749999999999, 2024-05-15T12:56:55.858 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:55.858 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T12:56:55.858 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:55.858 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.858 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:55.858 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801262+0000", 2024-05-15T12:56:55.858 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:55.859 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.859 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.859 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:55.859 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801262+0000", 2024-05-15T12:56:55.859 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:55.859 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.859 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.859 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:55.859 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801263+0000", 2024-05-15T12:56:55.859 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2100000000000002e-07 2024-05-15T12:56:55.859 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.859 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.859 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:55.860 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801267+0000", 2024-05-15T12:56:55.860 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2989999999999998e-06 2024-05-15T12:56:55.860 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.860 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.860 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:55.860 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801274+0000", 2024-05-15T12:56:55.860 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7959999999999998e-06 2024-05-15T12:56:55.860 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.860 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.860 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:55.861 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801278+0000", 2024-05-15T12:56:55.861 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6829999999999999e-06 2024-05-15T12:56:55.861 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.861 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.861 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:55.861 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801324+0000", 2024-05-15T12:56:55.861 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5677000000000003e-05 2024-05-15T12:56:55.861 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.861 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.861 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:55.861 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801343+0000", 2024-05-15T12:56:55.861 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9301999999999999e-05 2024-05-15T12:56:55.861 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:55.862 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:55.862 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:55.862 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.862 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.862 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24832.0:243492 3.a e298/219 3:59a2d01c:::benchmark_data_smithi177_88104_object109870:head v 298'15321, mlcod=298'15319)", 2024-05-15T12:56:55.862 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:55.795998+0000", 2024-05-15T12:56:55.862 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.0070436830000000002, 2024-05-15T12:56:55.862 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0072844370000000004, 2024-05-15T12:56:55.862 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:55.862 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T12:56:55.862 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:55.862 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.862 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:55.863 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.795998+0000", 2024-05-15T12:56:55.863 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:55.863 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.863 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.863 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:55.863 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.795998+0000", 2024-05-15T12:56:55.863 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:55.863 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.863 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.863 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:55.863 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.795999+0000", 2024-05-15T12:56:55.863 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.124e-06 2024-05-15T12:56:55.863 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.863 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.864 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:55.864 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.796007+0000", 2024-05-15T12:56:55.864 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8320000000000006e-06 2024-05-15T12:56:55.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.864 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.864 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:55.864 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.796019+0000", 2024-05-15T12:56:55.864 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2816999999999999e-05 2024-05-15T12:56:55.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.864 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.864 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:55.864 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.796027+0000", 2024-05-15T12:56:55.864 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0809999999999996e-06 2024-05-15T12:56:55.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.864 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.865 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:55.865 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.796083+0000", 2024-05-15T12:56:55.865 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6563e-05 2024-05-15T12:56:55.865 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.865 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.865 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:55.865 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.796125+0000", 2024-05-15T12:56:55.865 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1992e-05 2024-05-15T12:56:55.865 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:55.865 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:55.865 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:55.865 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.865 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.865 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:243503 3.4 3:2d0b9ebd:::benchmark_data_smithi177_88104_object109881:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-05-15T12:56:55.865 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:55.801145+0000", 2024-05-15T12:56:55.866 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.0018962989999999999, 2024-05-15T12:56:55.866 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0021989319999999998, 2024-05-15T12:56:55.866 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:56:55.866 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "waiting for sub ops", 2024-05-15T12:56:55.866 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:56:55.866 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T12:56:55.866 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:56:55.866 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 243503 2024-05-15T12:56:55.866 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.866 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:56:55.866 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.866 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:56:55.866 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801145+0000", 2024-05-15T12:56:55.867 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:55.867 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.867 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.867 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:56:55.867 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801145+0000", 2024-05-15T12:56:55.867 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:56:55.867 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.867 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.867 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:56:55.867 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801147+0000", 2024-05-15T12:56:55.867 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6050000000000001e-06 2024-05-15T12:56:55.867 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.867 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.867 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:56:55.867 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801153+0000", 2024-05-15T12:56:55.868 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7700000000000004e-06 2024-05-15T12:56:55.868 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.868 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.868 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:56:55.868 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801171+0000", 2024-05-15T12:56:55.868 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7750000000000001e-05 2024-05-15T12:56:55.868 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.868 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.868 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:56:55.868 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801180+0000", 2024-05-15T12:56:55.868 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7329999999999992e-06 2024-05-15T12:56:55.868 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.868 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.868 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:56:55.868 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801234+0000", 2024-05-15T12:56:55.869 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3925999999999998e-05 2024-05-15T12:56:55.869 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.869 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.869 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:56:55.869 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801343+0000", 2024-05-15T12:56:55.869 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000108816 2024-05-15T12:56:55.869 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:56:55.869 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:56:55.869 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2,5", 2024-05-15T12:56:55.869 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:55.801458+0000", 2024-05-15T12:56:55.869 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011552099999999999 2024-05-15T12:56:55.869 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:55.869 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:56:55.869 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:55.870 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:56:55.870 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T12:56:55.870 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 5 2024-05-15T12:56:55.870 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T12:56:55.938 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:56:55.969 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:55.992 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:56:55.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:55.997 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:56:56.098 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:56:56.133 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:56.156 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:56:56.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:56.159 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:56:56.260 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:56:56.290 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:56.312 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:56:56.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:56.315 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:56:56.416 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:56:56.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:56 smithi177 ceph-mon[44303]: pgmap v816: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.5 GiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 940 op/s; 1694/76086 objects misplaced (2.226%) 2024-05-15T12:56:56.448 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:56.472 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:56:56.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:56.475 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:56:56.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:56 smithi027 ceph-mon[39795]: pgmap v816: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.5 GiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 940 op/s; 1694/76086 objects misplaced (2.226%) 2024-05-15T12:56:56.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:56 smithi120 ceph-mon[43961]: pgmap v816: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.5 GiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 940 op/s; 1694/76086 objects misplaced (2.226%) 2024-05-15T12:56:56.576 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:56:56.604 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:56.625 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:56:56.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:56.627 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:56:56.728 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:56:56.788 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:56.818 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:56:56.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:56.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:56:56.928 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:56:56.956 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:56.978 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:56:56.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:56.982 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:56:56.984 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-05-15T12:56:57.082 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:56:57.157 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:57.196 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:56:57.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:57.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:56:57.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:57 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3405879449' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:56:57.301 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:56:57.327 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:57 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3405879449' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:56:57.329 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:57.351 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:56:57.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:57.354 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:56:57.455 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:56:57.484 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:57.506 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:56:57.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:57.512 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:56:57.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:57 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3405879449' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:56:57.613 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:56:57.647 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:57.675 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:56:57.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:57.679 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:56:57.780 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:56:57.816 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:57.842 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:56:57.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:57.847 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:56:57.948 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:56:57.968 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:56:57.968 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:56:57.968 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:56:57.968 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:56:57.968 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:56:57.968 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:56:57.968 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:56:57.968 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:56:57.969 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:56:57.969 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:56:57.969 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:56:57.969 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:56:57.969 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:56:57.969 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:56:57.969 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:56:57.969 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:56:57.969 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:56:57.969 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:56:57.969 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:56:57.969 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:56:57.969 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:56:57.970 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:56:57.970 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:56:57.970 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:56:58.002 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:58.039 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:56:58.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:58.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:56:58.144 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:56:58.172 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:58.195 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:56:58.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:58.197 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:56:58.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:58 smithi120 ceph-mon[43961]: 3.8 scrub starts 2024-05-15T12:56:58.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:58 smithi120 ceph-mon[43961]: pgmap v817: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.5 GiB data, 6.6 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 790 op/s; 1694/73491 objects misplaced (2.305%) 2024-05-15T12:56:58.298 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:56:58.353 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:58.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:58 smithi177 ceph-mon[44303]: 3.8 scrub starts 2024-05-15T12:56:58.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:58 smithi177 ceph-mon[44303]: pgmap v817: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.5 GiB data, 6.6 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 790 op/s; 1694/73491 objects misplaced (2.305%) 2024-05-15T12:56:58.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:58 smithi027 ceph-mon[39795]: 3.8 scrub starts 2024-05-15T12:56:58.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:58 smithi027 ceph-mon[39795]: pgmap v817: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.5 GiB data, 6.6 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 790 op/s; 1694/73491 objects misplaced (2.305%) 2024-05-15T12:56:58.825 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:56:58.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:58.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:56:58.932 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:56:58.963 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:58.981 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:56:58.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:58.984 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:56:59.085 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:56:59.136 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:59.174 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:56:59.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:59.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:56:59.277 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:56:59.327 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:59.374 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:56:59.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:59.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:56:59.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:56:59 smithi177 ceph-mon[44303]: 3.8 scrub ok 2024-05-15T12:56:59.480 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:56:59.509 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:59.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:56:59 smithi027 ceph-mon[39795]: 3.8 scrub ok 2024-05-15T12:56:59.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:56:59 smithi120 ceph-mon[43961]: 3.8 scrub ok 2024-05-15T12:56:59.531 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:56:59.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:59.534 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:56:59.545 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:56:59.545 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:56:59.635 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:56:59.683 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:56:59.868 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:56:59.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:56:59.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:56:59.979 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:57:00.034 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:00.270 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T12:57:00.298 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:00 smithi027 ceph-mon[39795]: pgmap v818: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.4 GiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 805 op/s; 1444/67050 objects misplaced (2.154%) 2024-05-15T12:57:00.298 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:00 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/270947473' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:57:00.298 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:00 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:57:00.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:00 smithi120 ceph-mon[43961]: pgmap v818: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.4 GiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 805 op/s; 1444/67050 objects misplaced (2.154%) 2024-05-15T12:57:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:00 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/270947473' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:57:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:00 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:57:00.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:00 smithi177 ceph-mon[44303]: pgmap v818: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.4 GiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 805 op/s; 1444/67050 objects misplaced (2.154%) 2024-05-15T12:57:00.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:00 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/270947473' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:57:00.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:00 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:57:00.863 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:57:01.122 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:57:01.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:01.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:57:01.229 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:57:01.283 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:01.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:01 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:57:01.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:01 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:57:01.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:01 smithi027 ceph-mon[39795]: osdmap e299: 12 total, 12 up, 10 in 2024-05-15T12:57:01.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:01 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:57:01.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:01 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:57:01.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:01 smithi120 ceph-mon[43961]: osdmap e299: 12 total, 12 up, 10 in 2024-05-15T12:57:01.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:01 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:57:01.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:01 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:57:01.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:01 smithi177 ceph-mon[44303]: osdmap e299: 12 total, 12 up, 10 in 2024-05-15T12:57:02.155 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:57:02.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:02.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:57:02.259 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:57:02.311 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:02.333 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:57:02.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:02.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:57:02.437 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:57:02.493 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:02 smithi027 ceph-mon[39795]: pgmap v820: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.3 GiB data, 5.9 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 827 op/s; 1444/63261 objects misplaced (2.283%) 2024-05-15T12:57:02.495 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:02.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:02 smithi120 ceph-mon[43961]: pgmap v820: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.3 GiB data, 5.9 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 827 op/s; 1444/63261 objects misplaced (2.283%) 2024-05-15T12:57:02.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:02 smithi177 ceph-mon[44303]: pgmap v820: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.3 GiB data, 5.9 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 827 op/s; 1444/63261 objects misplaced (2.283%) 2024-05-15T12:57:02.868 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:57:02.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:02.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:57:02.976 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:57:03.011 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:03.048 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:57:03.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:03.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:57:03.158 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:57:03.196 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:03.228 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:57:03.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:03.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:57:03.341 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:57:03.376 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:03.401 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:57:03.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:03.405 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:57:03.506 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:57:03.547 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:03.766 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:57:03.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:03.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:57:03.876 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:57:03.932 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:03.985 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:57:03.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:03.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:57:04.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:03 smithi027 ceph-mon[39795]: pgmap v821: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.2 GiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 831 op/s; 1155/57519 objects misplaced (2.008%) 2024-05-15T12:57:04.091 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:57:04.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:03 smithi177 ceph-mon[44303]: pgmap v821: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.2 GiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 831 op/s; 1155/57519 objects misplaced (2.008%) 2024-05-15T12:57:04.195 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:04.206 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:57:04.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:03 smithi120 ceph-mon[43961]: pgmap v821: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.2 GiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 831 op/s; 1155/57519 objects misplaced (2.008%) 2024-05-15T12:57:04.874 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.3 to 1 (10000) 2024-05-15T12:57:05.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:04 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2385039536' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T12:57:05.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:04 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T12:57:05.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:04 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2385039536' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T12:57:05.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:04 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T12:57:05.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:04 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2385039536' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T12:57:05.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:04 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T12:57:05.481 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T12:57:05.858 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:05 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T12:57:05.858 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:05 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:57:05.858 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:05 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:57:05.858 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:05 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:57:05.858 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:05 smithi027 ceph-mon[39795]: osdmap e300: 12 total, 12 up, 10 in 2024-05-15T12:57:05.858 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:05 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:57:05.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:05 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:57:05.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:05 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:57:06.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:05 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T12:57:06.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:05 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:57:06.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:05 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:57:06.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:05 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:57:06.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:05 smithi177 ceph-mon[44303]: osdmap e300: 12 total, 12 up, 10 in 2024-05-15T12:57:06.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:05 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:57:06.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:05 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:57:06.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:05 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:57:06.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:05 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:57:06.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:05 smithi177 ceph-mon[44303]: pgmap v823: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.0 GiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 959 op/s; 1155/50457 objects misplaced (2.289%) 2024-05-15T12:57:06.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:05 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:57:06.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:05 smithi027 ceph-mon[39795]: pgmap v823: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.0 GiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 959 op/s; 1155/50457 objects misplaced (2.289%) 2024-05-15T12:57:06.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:05 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T12:57:06.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:05 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:57:06.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:05 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:57:06.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:05 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:57:06.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:05 smithi120 ceph-mon[43961]: osdmap e300: 12 total, 12 up, 10 in 2024-05-15T12:57:06.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:05 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:57:06.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:05 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:57:06.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:05 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:57:06.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:05 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:57:06.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:05 smithi120 ceph-mon[43961]: pgmap v823: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 1.0 GiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 959 op/s; 1155/50457 objects misplaced (2.289%) 2024-05-15T12:57:07.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:06 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:57:07.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:06 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:57:07.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:06 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:57:07.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:06 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:57:07.304 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:57:07.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:07.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:57:07.409 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:57:07.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:06 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:57:07.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:06 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:57:07.448 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:07.554 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:57:07.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:07.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:57:07.662 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:57:07.690 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:07.713 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:57:07.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:07.723 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:57:07.824 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:57:07.856 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:07.885 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:57:07.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:07.890 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:57:07.991 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:57:07.993 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:08 smithi027 ceph-mon[39795]: osdmap e301: 12 total, 12 up, 10 in 2024-05-15T12:57:07.993 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:08 smithi027 ceph-mon[39795]: pgmap v825: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 999 MiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 846 op/s; 1155/47928 objects misplaced (2.410%) 2024-05-15T12:57:08.034 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:08.074 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:57:08.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:08.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:57:08.180 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:57:08.208 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:08.229 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:57:08.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:08.233 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:57:08.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:08 smithi120 ceph-mon[43961]: osdmap e301: 12 total, 12 up, 10 in 2024-05-15T12:57:08.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:08 smithi120 ceph-mon[43961]: pgmap v825: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 999 MiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 846 op/s; 1155/47928 objects misplaced (2.410%) 2024-05-15T12:57:08.334 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:57:08.393 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:08.410 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T12:57:08.410 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:57:08.426 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:57:08.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:08.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:57:08.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:08 smithi177 ceph-mon[44303]: osdmap e301: 12 total, 12 up, 10 in 2024-05-15T12:57:08.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:08 smithi177 ceph-mon[44303]: pgmap v825: 17 pgs: 1 active+remapped+backfill_toofull, 16 active+clean; 999 MiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 846 op/s; 1155/47928 objects misplaced (2.410%) 2024-05-15T12:57:08.531 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:57:08.573 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:08.607 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:57:08.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:08.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:57:08.714 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:57:08.741 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:08.765 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:57:08.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:08.769 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:57:08.870 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:57:08.898 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:08.917 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:57:08.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:08.920 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:57:09.021 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:57:09.072 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:09.773 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:57:09.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:09.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:57:09.815 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 4 1 2024-05-15T12:57:09.886 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:57:09.914 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:09.942 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:57:09.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:09.945 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:57:10.046 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:57:10.079 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:10.102 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:57:10.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:10.108 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:57:10.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:10 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:57:10.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:10 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:57:10.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:10 smithi177 ceph-mon[44303]: osdmap e302: 12 total, 12 up, 10 in 2024-05-15T12:57:10.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:10 smithi177 ceph-mon[44303]: pgmap v827: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 862 MiB data, 4.7 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 897 op/s; 758/41355 objects misplaced (1.833%); 1.2 MiB/s, 19 objects/s recovering 2024-05-15T12:57:10.209 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:57:10.258 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:10.385 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:57:10.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:10.389 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:10 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:57:10.390 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:10 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:57:10.390 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:10 smithi027 ceph-mon[39795]: osdmap e302: 12 total, 12 up, 10 in 2024-05-15T12:57:10.390 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:10 smithi027 ceph-mon[39795]: pgmap v827: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 862 MiB data, 4.7 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 897 op/s; 758/41355 objects misplaced (1.833%); 1.2 MiB/s, 19 objects/s recovering 2024-05-15T12:57:10.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:57:10.491 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:57:10.514 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:57:10.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:10 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:57:10.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:10 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:57:10.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:10 smithi120 ceph-mon[43961]: osdmap e302: 12 total, 12 up, 10 in 2024-05-15T12:57:10.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:10 smithi120 ceph-mon[43961]: pgmap v827: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 862 MiB data, 4.7 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 897 op/s; 758/41355 objects misplaced (1.833%); 1.2 MiB/s, 19 objects/s recovering 2024-05-15T12:57:10.556 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:10.559 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:57:10.585 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:57:10.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:10.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:57:10.711 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:57:10.768 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:10.796 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:57:10.817 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:57:10.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:10.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:57:10.830 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:57:10.878 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:57:10.920 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:57:10.927 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:57:10.977 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:10.977 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:57:11.019 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:57:11.053 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:57:11.084 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:57:11.090 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:57:11.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:11.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:57:11.132 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:57:11.139 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-05-15T12:57:11.158 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:57:11.167 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:57:11.177 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:57:11.182 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:57:11.182 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:57:11.182 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:57:11.182 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:57:11.182 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:57:11.182 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:57:11.182 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:57:11.183 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:57:11.183 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:57:11.183 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:57:11.183 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:57:11.183 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:57:11.199 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:57:11.229 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:11.230 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:11 smithi027 ceph-mon[39795]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-05-15T12:57:11.230 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:11 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2634522535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:57:11.248 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:57:11.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:11.251 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:57:11.352 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:57:11.379 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:11.400 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:57:11.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:11 smithi177 ceph-mon[44303]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-05-15T12:57:11.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:11 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2634522535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:57:11.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:11.404 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:57:11.504 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:57:11.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:11 smithi120 ceph-mon[43961]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-05-15T12:57:11.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:11 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2634522535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:57:11.534 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:11.556 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:57:11.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:11.560 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:57:11.661 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:57:11.729 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:11.778 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:57:11.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:11.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:57:11.892 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:57:11.944 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:11.987 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:57:11.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:11.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:57:12.092 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:57:12.162 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:12.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:12 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:57:12.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:12 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:57:12.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:12 smithi027 ceph-mon[39795]: osdmap e303: 12 total, 12 up, 10 in 2024-05-15T12:57:12.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:12 smithi027 ceph-mon[39795]: pgmap v829: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 777 MiB data, 4.5 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 732 op/s; 758/37257 objects misplaced (2.035%); 1.2 MiB/s, 19 objects/s recovering 2024-05-15T12:57:12.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:12 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:57:12.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:12 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:57:12.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:12 smithi120 ceph-mon[43961]: osdmap e303: 12 total, 12 up, 10 in 2024-05-15T12:57:12.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:12 smithi120 ceph-mon[43961]: pgmap v829: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 777 MiB data, 4.5 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 732 op/s; 758/37257 objects misplaced (2.035%); 1.2 MiB/s, 19 objects/s recovering 2024-05-15T12:57:12.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:12 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:57:12.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:12 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:57:12.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:12 smithi177 ceph-mon[44303]: osdmap e303: 12 total, 12 up, 10 in 2024-05-15T12:57:12.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:12 smithi177 ceph-mon[44303]: pgmap v829: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 777 MiB data, 4.5 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 732 op/s; 758/37257 objects misplaced (2.035%); 1.2 MiB/s, 19 objects/s recovering 2024-05-15T12:57:12.536 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:57:12.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:12.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:57:12.642 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:57:12.708 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:12.737 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:57:12.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:12.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:57:12.844 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:57:12.894 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:12.954 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:57:12.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:12.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:57:13.062 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:57:13.097 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:13.125 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:57:13.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:13.132 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:57:13.233 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:57:13.270 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:13.828 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:57:13.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:13.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:57:13.934 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:57:13.961 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:13.980 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:57:13.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:13.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:57:14.084 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:57:14.112 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:14.132 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:57:14.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:14.135 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:57:14.236 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:57:14.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:14 smithi120 ceph-mon[43961]: osdmap e304: 12 total, 12 up, 10 in 2024-05-15T12:57:14.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:14 smithi120 ceph-mon[43961]: pgmap v831: 17 pgs: 1 active+clean+remapped, 16 active+clean; 661 MiB data, 4.2 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 900 op/s; 7.5 MiB/s, 120 objects/s recovering 2024-05-15T12:57:14.300 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:14.324 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:57:14.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:14.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:57:14.429 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:57:14.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:14 smithi177 ceph-mon[44303]: osdmap e304: 12 total, 12 up, 10 in 2024-05-15T12:57:14.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:14 smithi177 ceph-mon[44303]: pgmap v831: 17 pgs: 1 active+clean+remapped, 16 active+clean; 661 MiB data, 4.2 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 900 op/s; 7.5 MiB/s, 120 objects/s recovering 2024-05-15T12:57:14.455 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:14.473 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:57:14.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:14.476 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:57:14.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:14 smithi027 ceph-mon[39795]: osdmap e304: 12 total, 12 up, 10 in 2024-05-15T12:57:14.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:14 smithi027 ceph-mon[39795]: pgmap v831: 17 pgs: 1 active+clean+remapped, 16 active+clean; 661 MiB data, 4.2 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 900 op/s; 7.5 MiB/s, 120 objects/s recovering 2024-05-15T12:57:14.577 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:57:14.668 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:15.029 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:57:15.043 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:57:15.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:15.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:57:15.143 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.4 to 1 (10000) 2024-05-15T12:57:15.154 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:57:15.227 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:15.274 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:57:15.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:15.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:57:15.386 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:57:15.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:15 smithi177 ceph-mon[44303]: osdmap e305: 12 total, 12 up, 10 in 2024-05-15T12:57:15.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:15 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4138393327' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T12:57:15.442 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:15 smithi120 ceph-mon[43961]: osdmap e305: 12 total, 12 up, 10 in 2024-05-15T12:57:15.442 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:15 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4138393327' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T12:57:15.446 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:15.482 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:57:15.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:15.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:57:15.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:15 smithi027 ceph-mon[39795]: osdmap e305: 12 total, 12 up, 10 in 2024-05-15T12:57:15.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:15 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4138393327' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T12:57:15.589 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:57:15.610 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:57:15.653 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:15.810 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:57:15.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:15.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:57:15.916 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:57:15.981 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:16.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:16 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4138393327' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T12:57:16.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:16 smithi027 ceph-mon[39795]: osdmap e306: 12 total, 12 up, 10 in 2024-05-15T12:57:16.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:16 smithi027 ceph-mon[39795]: pgmap v834: 17 pgs: 1 active+clean+remapped, 16 active+clean; 537 MiB data, 3.8 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 831 op/s; 8.2 MiB/s, 131 objects/s recovering 2024-05-15T12:57:16.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:16 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4138393327' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T12:57:16.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:16 smithi177 ceph-mon[44303]: osdmap e306: 12 total, 12 up, 10 in 2024-05-15T12:57:16.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:16 smithi177 ceph-mon[44303]: pgmap v834: 17 pgs: 1 active+clean+remapped, 16 active+clean; 537 MiB data, 3.8 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 831 op/s; 8.2 MiB/s, 131 objects/s recovering 2024-05-15T12:57:16.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:16 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4138393327' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T12:57:16.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:16 smithi120 ceph-mon[43961]: osdmap e306: 12 total, 12 up, 10 in 2024-05-15T12:57:16.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:16 smithi120 ceph-mon[43961]: pgmap v834: 17 pgs: 1 active+clean+remapped, 16 active+clean; 537 MiB data, 3.8 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 831 op/s; 8.2 MiB/s, 131 objects/s recovering 2024-05-15T12:57:16.558 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:57:16.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:16.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:57:16.598 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 5 1 2024-05-15T12:57:16.666 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:57:16.698 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:16.718 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:57:16.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:16.722 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:57:16.823 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:57:16.851 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:16.872 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:57:16.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:16.877 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:57:16.978 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:57:17.026 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:17.070 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:57:17.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:17.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:57:17.178 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:57:17.205 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:17.227 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:57:17.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:17.230 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:57:17.331 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:57:17.410 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:17.441 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:57:17.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:17.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:57:17.545 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:57:17.609 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:17.636 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:57:17.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:17.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:57:17.744 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:57:17.791 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:17.831 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:57:17.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:17.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:57:17.935 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:57:17.969 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:17.989 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:57:17.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:17.999 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:57:18.100 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:57:18.138 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:17 smithi027 ceph-mon[39795]: pgmap v835: 17 pgs: 1 active+clean+remapped, 16 active+clean; 496 MiB data, 3.7 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 749 op/s; 6.3 MiB/s, 100 objects/s recovering 2024-05-15T12:57:18.169 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:18.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:17 smithi177 ceph-mon[44303]: pgmap v835: 17 pgs: 1 active+clean+remapped, 16 active+clean; 496 MiB data, 3.7 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 749 op/s; 6.3 MiB/s, 100 objects/s recovering 2024-05-15T12:57:18.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:17 smithi120 ceph-mon[43961]: pgmap v835: 17 pgs: 1 active+clean+remapped, 16 active+clean; 496 MiB data, 3.7 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 749 op/s; 6.3 MiB/s, 100 objects/s recovering 2024-05-15T12:57:18.541 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:57:18.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:18.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:57:18.552 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:57:18.639 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:57:18.649 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:57:18.706 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:18.886 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T12:57:19.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:18 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1221792760' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:57:19.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:18 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:57:19.279 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:18 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1221792760' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:57:19.279 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:18 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:57:19.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:18 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1221792760' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:57:19.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:18 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:57:19.783 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:57:19.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:19.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:57:19.897 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:57:19.920 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T12:57:19.920 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T12:57:19.920 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T12:57:19.920 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T12:57:19.921 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.921 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:90800 2.4 2:2acfc200:::benchmark_data_smithi177_78104_object90799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-05-15T12:57:19.921 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:17.243564+0000", 2024-05-15T12:57:19.921 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 482.29265999099999, 2024-05-15T12:57:19.921 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2451092909999999, 2024-05-15T12:57:19.921 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:57:19.921 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:57:19.921 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:57:19.921 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:57:19.921 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:57:19.921 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 90800 2024-05-15T12:57:19.922 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.922 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:57:19.922 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.922 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:57:19.922 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.243564+0000", 2024-05-15T12:57:19.922 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:19.922 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.922 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.922 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:57:19.922 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.243564+0000", 2024-05-15T12:57:19.922 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:19.922 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.922 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.923 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:57:19.923 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.243566+0000", 2024-05-15T12:57:19.923 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.669e-06 2024-05-15T12:57:19.923 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.923 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.923 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:57:19.923 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.243631+0000", 2024-05-15T12:57:19.923 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5320999999999993e-05 2024-05-15T12:57:19.923 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.923 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.923 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:57:19.923 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.243649+0000", 2024-05-15T12:57:19.923 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7788e-05 2024-05-15T12:57:19.923 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.924 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.924 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:57:19.924 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.243659+0000", 2024-05-15T12:57:19.924 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9310000000000003e-06 2024-05-15T12:57:19.924 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.924 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.924 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:19.924 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.243727+0000", 2024-05-15T12:57:19.924 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8182999999999994e-05 2024-05-15T12:57:19.924 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.924 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.924 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-05-15T12:57:19.925 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.243733+0000", 2024-05-15T12:57:19.925 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9730000000000002e-06 2024-05-15T12:57:19.925 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.925 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.925 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:19.925 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.248328+0000", 2024-05-15T12:57:19.925 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0045947999999999996 2024-05-15T12:57:19.925 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.925 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.925 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:57:19.925 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.248339+0000", 2024-05-15T12:57:19.925 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1358e-05 2024-05-15T12:57:19.925 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.925 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.926 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:19.926 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.855990+0000", 2024-05-15T12:57:19.926 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.60765039899999995 2024-05-15T12:57:19.926 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.926 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.926 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:57:19.926 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.856005+0000", 2024-05-15T12:57:19.926 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5495e-05 2024-05-15T12:57:19.926 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.926 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.926 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:19.926 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.379629+0000", 2024-05-15T12:57:19.927 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.523623607 2024-05-15T12:57:19.927 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.927 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.927 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:57:19.927 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.379779+0000", 2024-05-15T12:57:19.927 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015046100000000001 2024-05-15T12:57:19.927 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.927 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.927 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:57:19.927 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.379942+0000", 2024-05-15T12:57:19.928 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000162962 2024-05-15T12:57:19.928 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.928 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.928 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.928 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.390721+0000", 2024-05-15T12:57:19.928 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010778914000000001 2024-05-15T12:57:19.928 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.928 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.928 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:57:19.928 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.390736+0000", 2024-05-15T12:57:19.928 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4922e-05 2024-05-15T12:57:19.928 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.928 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.928 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.929 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.390775+0000", 2024-05-15T12:57:19.929 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.913e-05 2024-05-15T12:57:19.929 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.929 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.929 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.929 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.488483+0000", 2024-05-15T12:57:19.929 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.097707752999999994 2024-05-15T12:57:19.929 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.929 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.929 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:57:19.929 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.488545+0000", 2024-05-15T12:57:19.929 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1901000000000002e-05 2024-05-15T12:57:19.929 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.930 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.930 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:57:19.930 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.488674+0000", 2024-05-15T12:57:19.930 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012872400000000001 2024-05-15T12:57:19.930 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:19.930 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:57:19.930 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:19.930 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.930 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.930 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:90822 2.4 2:28e8288f:::benchmark_data_smithi177_78104_object90821:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-05-15T12:57:19.931 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:17.272538+0000", 2024-05-15T12:57:19.931 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 482.26368625100002, 2024-05-15T12:57:19.931 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.216411146, 2024-05-15T12:57:19.931 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:57:19.931 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:57:19.931 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:57:19.931 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:57:19.931 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:57:19.931 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 90822 2024-05-15T12:57:19.931 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.931 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:57:19.931 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.931 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:57:19.932 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.272538+0000", 2024-05-15T12:57:19.932 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:19.932 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.932 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.932 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:57:19.932 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.272538+0000", 2024-05-15T12:57:19.932 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:19.932 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.932 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.932 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:57:19.932 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.272540+0000", 2024-05-15T12:57:19.933 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.984e-06 2024-05-15T12:57:19.933 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.933 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.933 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:57:19.933 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.272587+0000", 2024-05-15T12:57:19.933 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6814000000000003e-05 2024-05-15T12:57:19.933 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.933 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.933 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:57:19.933 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.272609+0000", 2024-05-15T12:57:19.933 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1979000000000001e-05 2024-05-15T12:57:19.933 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.933 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.933 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:57:19.934 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.272619+0000", 2024-05-15T12:57:19.934 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0539e-05 2024-05-15T12:57:19.934 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.934 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.934 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:19.934 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.272698+0000", 2024-05-15T12:57:19.934 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8716999999999999e-05 2024-05-15T12:57:19.934 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.934 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.935 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:57:19.935 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.272713+0000", 2024-05-15T12:57:19.935 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4662e-05 2024-05-15T12:57:19.935 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.935 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.935 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:19.935 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.856063+0000", 2024-05-15T12:57:19.935 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.58335017700000003 2024-05-15T12:57:19.935 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.935 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.935 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T12:57:19.935 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:17.856071+0000", 2024-05-15T12:57:19.935 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2130000000000008e-06 2024-05-15T12:57:19.935 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.936 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.936 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:19.936 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.383908+0000", 2024-05-15T12:57:19.936 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.527836432 2024-05-15T12:57:19.936 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.936 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.936 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:57:19.936 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.384046+0000", 2024-05-15T12:57:19.936 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013840900000000001 2024-05-15T12:57:19.936 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.936 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.936 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:57:19.936 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.384225+0000", 2024-05-15T12:57:19.937 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000179218 2024-05-15T12:57:19.937 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.937 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.937 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.937 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.394448+0000", 2024-05-15T12:57:19.937 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010223101 2024-05-15T12:57:19.937 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.937 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.937 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:57:19.938 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.394463+0000", 2024-05-15T12:57:19.938 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5175e-05 2024-05-15T12:57:19.938 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.938 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.938 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.938 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.400394+0000", 2024-05-15T12:57:19.938 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0059300280000000004 2024-05-15T12:57:19.938 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.938 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.939 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.939 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.488739+0000", 2024-05-15T12:57:19.939 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.088345173999999999 2024-05-15T12:57:19.939 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.939 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.939 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:57:19.939 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.488813+0000", 2024-05-15T12:57:19.939 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4237000000000001e-05 2024-05-15T12:57:19.939 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.939 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.939 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:57:19.939 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:19.488949+0000", 2024-05-15T12:57:19.939 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013628699999999999 2024-05-15T12:57:19.939 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:19.939 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:57:19.940 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:19.940 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.940 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.940 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119049 2.4 2:2045346f:::benchmark_data_smithi177_78104_object11916:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:57:19.940 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.627777+0000", 2024-05-15T12:57:19.940 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 443.90844750500003, 2024-05-15T12:57:19.940 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 15.003778057, 2024-05-15T12:57:19.940 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:57:19.940 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:57:19.940 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:57:19.940 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:57:19.940 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:57:19.940 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119049 2024-05-15T12:57:19.940 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.941 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:57:19.941 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.941 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:57:19.941 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.627777+0000", 2024-05-15T12:57:19.941 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:19.941 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.941 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.941 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:57:19.941 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.627777+0000", 2024-05-15T12:57:19.941 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:19.941 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.942 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.942 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:57:19.942 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.627777+0000", 2024-05-15T12:57:19.942 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0200000000000002e-07 2024-05-15T12:57:19.942 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.942 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.942 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:57:19.942 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.627780+0000", 2024-05-15T12:57:19.942 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0000000000000001e-06 2024-05-15T12:57:19.942 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.942 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.942 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:57:19.942 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.627788+0000", 2024-05-15T12:57:19.943 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9640000000000003e-06 2024-05-15T12:57:19.943 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.943 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.943 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:57:19.943 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.627794+0000", 2024-05-15T12:57:19.943 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4430000000000002e-06 2024-05-15T12:57:19.943 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.943 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.943 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:19.943 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.629174+0000", 2024-05-15T12:57:19.943 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001380364 2024-05-15T12:57:19.943 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.943 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.943 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:57:19.943 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.629297+0000", 2024-05-15T12:57:19.944 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000123041 2024-05-15T12:57:19.944 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.944 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.944 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:57:19.944 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.629415+0000", 2024-05-15T12:57:19.944 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000118213 2024-05-15T12:57:19.944 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.944 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.944 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.944 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640876+0000", 2024-05-15T12:57:19.944 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011460568000000001 2024-05-15T12:57:19.944 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.944 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.944 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:57:19.944 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640891+0000", 2024-05-15T12:57:19.945 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5194000000000001e-05 2024-05-15T12:57:19.945 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.945 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.945 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.945 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.646262+0000", 2024-05-15T12:57:19.945 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0053713809999999997 2024-05-15T12:57:19.945 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.945 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.945 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.945 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.631408+0000", 2024-05-15T12:57:19.945 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.985145664000001 2024-05-15T12:57:19.945 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.945 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.945 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:57:19.945 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.631460+0000", 2024-05-15T12:57:19.946 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1654999999999999e-05 2024-05-15T12:57:19.946 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.946 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.946 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:57:19.946 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.631555+0000", 2024-05-15T12:57:19.946 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5068000000000005e-05 2024-05-15T12:57:19.946 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:19.946 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:57:19.946 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:19.946 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.946 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.946 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119051 2.4 2:2ff5910d:::benchmark_data_smithi177_78104_object11918:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:57:19.946 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.631462+0000", 2024-05-15T12:57:19.946 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 443.90476181899999, 2024-05-15T12:57:19.947 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 15.000378482, 2024-05-15T12:57:19.947 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:57:19.947 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:57:19.947 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:57:19.947 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:57:19.947 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:57:19.947 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119051 2024-05-15T12:57:19.947 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.947 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:57:19.947 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.947 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:57:19.947 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.631462+0000", 2024-05-15T12:57:19.947 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:19.947 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.947 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.948 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:57:19.948 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.631462+0000", 2024-05-15T12:57:19.948 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:19.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.948 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.948 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:57:19.948 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.631464+0000", 2024-05-15T12:57:19.948 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.415e-06 2024-05-15T12:57:19.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.949 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.949 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:57:19.949 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.631470+0000", 2024-05-15T12:57:19.949 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2650000000000002e-06 2024-05-15T12:57:19.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.949 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.949 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:57:19.949 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.631480+0000", 2024-05-15T12:57:19.949 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0020000000000001e-05 2024-05-15T12:57:19.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.949 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.950 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:57:19.950 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.631487+0000", 2024-05-15T12:57:19.950 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8120000000000002e-06 2024-05-15T12:57:19.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.950 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.950 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:19.950 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.633185+0000", 2024-05-15T12:57:19.950 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016980470000000001 2024-05-15T12:57:19.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.950 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.951 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:57:19.951 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.633311+0000", 2024-05-15T12:57:19.951 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012555699999999999 2024-05-15T12:57:19.951 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.951 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.951 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:57:19.951 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.633434+0000", 2024-05-15T12:57:19.951 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000123177 2024-05-15T12:57:19.951 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.951 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.951 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:57:19.951 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640898+0000", 2024-05-15T12:57:19.952 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0074641630000000002 2024-05-15T12:57:19.952 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.952 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.952 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.952 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.641342+0000", 2024-05-15T12:57:19.952 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00044389700000000001 2024-05-15T12:57:19.952 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.952 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.952 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.952 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.651030+0000", 2024-05-15T12:57:19.952 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0096885540000000003 2024-05-15T12:57:19.952 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.953 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.953 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.953 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.631621+0000", 2024-05-15T12:57:19.953 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.980590125999999 2024-05-15T12:57:19.953 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.953 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.954 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:57:19.954 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.631691+0000", 2024-05-15T12:57:19.954 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0535000000000001e-05 2024-05-15T12:57:19.954 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.954 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.955 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:57:19.955 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.631841+0000", 2024-05-15T12:57:19.955 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014991399999999999 2024-05-15T12:57:19.955 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:19.955 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:57:19.955 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:19.955 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.955 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.955 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119049 2.4 2:2045346f:::benchmark_data_smithi177_78104_object11916:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:57:19.955 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.640268+0000", 2024-05-15T12:57:19.956 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 443.89595622000002, 2024-05-15T12:57:19.956 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.991202053, 2024-05-15T12:57:19.956 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:57:19.956 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "waiting for ondisk", 2024-05-15T12:57:19.956 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:57:19.956 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:57:19.956 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:57:19.956 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119049 2024-05-15T12:57:19.956 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.956 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:57:19.956 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.956 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:57:19.956 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640268+0000", 2024-05-15T12:57:19.956 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:19.957 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.957 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.957 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:57:19.957 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640268+0000", 2024-05-15T12:57:19.957 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:19.957 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.957 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.957 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:57:19.957 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640269+0000", 2024-05-15T12:57:19.957 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4999999999999998e-07 2024-05-15T12:57:19.957 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.957 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.957 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:57:19.957 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640271+0000", 2024-05-15T12:57:19.958 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6129999999999998e-06 2024-05-15T12:57:19.958 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.958 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.958 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:57:19.958 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640277+0000", 2024-05-15T12:57:19.958 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2260000000000002e-06 2024-05-15T12:57:19.958 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.958 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.958 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:57:19.959 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640283+0000", 2024-05-15T12:57:19.959 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2349999999999999e-06 2024-05-15T12:57:19.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.959 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.959 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:19.959 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640928+0000", 2024-05-15T12:57:19.959 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00064552499999999998 2024-05-15T12:57:19.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.959 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.959 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for ondisk", 2024-05-15T12:57:19.959 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640997+0000", 2024-05-15T12:57:19.959 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8717e-05 2024-05-15T12:57:19.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.959 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.959 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:57:19.960 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.631470+0000", 2024-05-15T12:57:19.960 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.990473287 2024-05-15T12:57:19.960 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:19.961 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:57:19.961 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:19.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.961 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.961 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119051 2.4 2:2ff5910d:::benchmark_data_smithi177_78104_object11918:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:57:19.961 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.640291+0000", 2024-05-15T12:57:19.962 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 443.89593378799998, 2024-05-15T12:57:19.962 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.991410879, 2024-05-15T12:57:19.962 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:57:19.962 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "waiting for ondisk", 2024-05-15T12:57:19.962 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:57:19.962 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:57:19.962 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:57:19.962 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119051 2024-05-15T12:57:19.962 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.962 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:57:19.962 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.962 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:57:19.962 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640291+0000", 2024-05-15T12:57:19.963 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:19.963 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.963 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.963 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:57:19.963 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640291+0000", 2024-05-15T12:57:19.963 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:19.963 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.963 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.963 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:57:19.963 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640291+0000", 2024-05-15T12:57:19.963 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0100000000000001e-07 2024-05-15T12:57:19.963 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.963 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.964 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:57:19.964 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640293+0000", 2024-05-15T12:57:19.964 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0700000000000001e-06 2024-05-15T12:57:19.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.964 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.964 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:57:19.964 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640298+0000", 2024-05-15T12:57:19.964 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.558e-06 2024-05-15T12:57:19.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.964 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.964 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:57:19.964 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.640303+0000", 2024-05-15T12:57:19.964 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7890000000000002e-06 2024-05-15T12:57:19.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.964 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.965 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:19.965 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.641024+0000", 2024-05-15T12:57:19.965 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00072119299999999996 2024-05-15T12:57:19.965 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.965 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.965 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for ondisk", 2024-05-15T12:57:19.965 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.641086+0000", 2024-05-15T12:57:19.965 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1248999999999996e-05 2024-05-15T12:57:19.965 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.965 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.965 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:57:19.965 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.631701+0000", 2024-05-15T12:57:19.965 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.990615719000001 2024-05-15T12:57:19.965 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:19.965 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:57:19.966 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:19.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.966 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.966 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119069 2.4 2:2e7a257b:::benchmark_data_smithi177_78104_object11936:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:57:19.966 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.651569+0000", 2024-05-15T12:57:19.966 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 443.884655549, 2024-05-15T12:57:19.966 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.980443061999999, 2024-05-15T12:57:19.966 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:57:19.966 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:57:19.966 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:57:19.966 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:57:19.966 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:57:19.966 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119069 2024-05-15T12:57:19.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.967 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:57:19.967 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.967 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:57:19.967 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.651569+0000", 2024-05-15T12:57:19.967 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:19.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.967 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.967 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:57:19.967 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.651569+0000", 2024-05-15T12:57:19.967 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:19.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.967 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.967 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:57:19.968 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.651571+0000", 2024-05-15T12:57:19.968 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9069999999999999e-06 2024-05-15T12:57:19.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.968 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:57:19.968 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.651581+0000", 2024-05-15T12:57:19.968 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.011e-05 2024-05-15T12:57:19.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.968 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:57:19.968 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.651594+0000", 2024-05-15T12:57:19.968 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3159999999999999e-05 2024-05-15T12:57:19.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.969 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:57:19.969 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.651602+0000", 2024-05-15T12:57:19.969 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6490000000000008e-06 2024-05-15T12:57:19.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.969 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.969 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:19.969 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.651661+0000", 2024-05-15T12:57:19.969 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9160000000000003e-05 2024-05-15T12:57:19.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.969 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.969 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:57:19.969 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.651804+0000", 2024-05-15T12:57:19.970 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000142764 2024-05-15T12:57:19.970 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.970 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.970 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:57:19.970 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.651938+0000", 2024-05-15T12:57:19.970 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013402499999999999 2024-05-15T12:57:19.970 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.970 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.970 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:57:19.970 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.660292+0000", 2024-05-15T12:57:19.970 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0083543000000000003 2024-05-15T12:57:19.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.971 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.971 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.971 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.663547+0000", 2024-05-15T12:57:19.971 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003255553 2024-05-15T12:57:19.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.971 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.971 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.971 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.667485+0000", 2024-05-15T12:57:19.971 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039378340000000003 2024-05-15T12:57:19.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.971 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.971 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.971 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.631879+0000", 2024-05-15T12:57:19.972 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.964394083 2024-05-15T12:57:19.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.972 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.972 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:57:19.972 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.631914+0000", 2024-05-15T12:57:19.972 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4369000000000002e-05 2024-05-15T12:57:19.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.972 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.972 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:57:19.972 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.632012+0000", 2024-05-15T12:57:19.972 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8147999999999996e-05 2024-05-15T12:57:19.972 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:19.972 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:57:19.972 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:19.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.972 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.973 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119086 2.4 2:211895ac:::benchmark_data_smithi177_78104_object11953:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:57:19.973 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.667377+0000", 2024-05-15T12:57:19.973 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 443.86884723399999, 2024-05-15T12:57:19.973 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.967394268, 2024-05-15T12:57:19.973 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:57:19.973 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:57:19.973 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:57:19.973 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:57:19.973 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:57:19.973 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119086 2024-05-15T12:57:19.973 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.973 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:57:19.973 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.973 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:57:19.973 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.667377+0000", 2024-05-15T12:57:19.974 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:19.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.974 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.974 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:57:19.974 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.667377+0000", 2024-05-15T12:57:19.974 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:19.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.974 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.974 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:57:19.974 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.667378+0000", 2024-05-15T12:57:19.974 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.128e-06 2024-05-15T12:57:19.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.974 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.975 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:57:19.975 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.667386+0000", 2024-05-15T12:57:19.975 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4410000000000004e-06 2024-05-15T12:57:19.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.975 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.975 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:57:19.975 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.667396+0000", 2024-05-15T12:57:19.975 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0172e-05 2024-05-15T12:57:19.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.975 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.975 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:57:19.975 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.667406+0000", 2024-05-15T12:57:19.975 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7259999999999997e-06 2024-05-15T12:57:19.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.975 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.975 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:19.976 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.667522+0000", 2024-05-15T12:57:19.976 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000116402 2024-05-15T12:57:19.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.976 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.976 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:57:19.976 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.667651+0000", 2024-05-15T12:57:19.976 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012925 2024-05-15T12:57:19.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.976 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.976 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:57:19.976 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.667785+0000", 2024-05-15T12:57:19.976 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000133714 2024-05-15T12:57:19.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.976 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.976 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:57:19.977 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.672146+0000", 2024-05-15T12:57:19.977 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.004361452 2024-05-15T12:57:19.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.977 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.977 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.977 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.679409+0000", 2024-05-15T12:57:19.977 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0072626799999999997 2024-05-15T12:57:19.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.977 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.977 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.977 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.686927+0000", 2024-05-15T12:57:19.977 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0075182699999999996 2024-05-15T12:57:19.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.977 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.977 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.978 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.634563+0000", 2024-05-15T12:57:19.978 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.947635259 2024-05-15T12:57:19.978 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.978 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.978 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:57:19.978 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.634639+0000", 2024-05-15T12:57:19.978 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6280999999999997e-05 2024-05-15T12:57:19.978 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.978 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.978 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:57:19.978 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.634771+0000", 2024-05-15T12:57:19.978 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013249299999999999 2024-05-15T12:57:19.979 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:19.979 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:57:19.979 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:19.979 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.979 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.979 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119093 2.4 2:24c03606:::benchmark_data_smithi177_78104_object11960:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:57:19.979 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.675073+0000", 2024-05-15T12:57:19.979 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 443.86115089899999, 2024-05-15T12:57:19.979 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.963393344, 2024-05-15T12:57:19.980 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:57:19.980 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:57:19.980 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:57:19.980 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:57:19.980 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:57:19.980 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119093 2024-05-15T12:57:19.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.980 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:57:19.980 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.980 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:57:19.980 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.675073+0000", 2024-05-15T12:57:19.980 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:19.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.980 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.980 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:57:19.981 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.675073+0000", 2024-05-15T12:57:19.981 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:19.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.981 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.981 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:57:19.981 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.675080+0000", 2024-05-15T12:57:19.981 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6989999999999997e-06 2024-05-15T12:57:19.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.981 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.981 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:57:19.981 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.675086+0000", 2024-05-15T12:57:19.982 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4299999999999997e-06 2024-05-15T12:57:19.982 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.982 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.982 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:57:19.982 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.675100+0000", 2024-05-15T12:57:19.982 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4211e-05 2024-05-15T12:57:19.982 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.982 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.982 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:57:19.982 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.675120+0000", 2024-05-15T12:57:19.983 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9868999999999999e-05 2024-05-15T12:57:19.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.983 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.983 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:19.983 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.675208+0000", 2024-05-15T12:57:19.983 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8237000000000002e-05 2024-05-15T12:57:19.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.983 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.983 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:57:19.983 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.675333+0000", 2024-05-15T12:57:19.983 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000125072 2024-05-15T12:57:19.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.983 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.983 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:57:19.983 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.675454+0000", 2024-05-15T12:57:19.984 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000121144 2024-05-15T12:57:19.984 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.984 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.984 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:57:19.984 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.679006+0000", 2024-05-15T12:57:19.984 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003551528 2024-05-15T12:57:19.984 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.984 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.984 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.984 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.686976+0000", 2024-05-15T12:57:19.984 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0079701870000000001 2024-05-15T12:57:19.984 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.984 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.984 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.984 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.691978+0000", 2024-05-15T12:57:19.985 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050020200000000003 2024-05-15T12:57:19.985 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.985 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.985 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.985 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.638261+0000", 2024-05-15T12:57:19.985 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.946283504 2024-05-15T12:57:19.985 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.985 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.985 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:57:19.985 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.638326+0000", 2024-05-15T12:57:19.985 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5178000000000005e-05 2024-05-15T12:57:19.985 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.985 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.985 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:57:19.985 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.638467+0000", 2024-05-15T12:57:19.986 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014026500000000001 2024-05-15T12:57:19.986 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:19.986 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:57:19.986 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:19.986 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.986 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.986 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119105 2.4 2:2b391885:::benchmark_data_smithi177_78104_object11972:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:57:19.986 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.695763+0000", 2024-05-15T12:57:19.986 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 443.84046127200003, 2024-05-15T12:57:19.986 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.942991198, 2024-05-15T12:57:19.986 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:57:19.987 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:57:19.987 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:57:19.987 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:57:19.987 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:57:19.987 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119105 2024-05-15T12:57:19.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.987 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:57:19.987 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.987 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:57:19.987 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.695763+0000", 2024-05-15T12:57:19.987 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:19.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.987 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.987 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:57:19.987 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.695763+0000", 2024-05-15T12:57:19.987 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:19.988 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.988 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.988 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:57:19.988 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.695764+0000", 2024-05-15T12:57:19.988 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.158e-06 2024-05-15T12:57:19.988 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.988 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.988 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:57:19.988 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.695770+0000", 2024-05-15T12:57:19.988 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0939999999999996e-06 2024-05-15T12:57:19.988 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.988 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.988 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:57:19.988 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.695781+0000", 2024-05-15T12:57:19.989 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0526000000000001e-05 2024-05-15T12:57:19.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.989 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.989 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:57:19.989 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.695788+0000", 2024-05-15T12:57:19.989 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8950000000000001e-06 2024-05-15T12:57:19.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.989 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.989 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:19.989 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.695916+0000", 2024-05-15T12:57:19.989 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012823000000000001 2024-05-15T12:57:19.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.989 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.989 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:57:19.990 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.696037+0000", 2024-05-15T12:57:19.990 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000121317 2024-05-15T12:57:19.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.990 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.990 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:57:19.990 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.696164+0000", 2024-05-15T12:57:19.990 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000126499 2024-05-15T12:57:19.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.990 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.990 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.991 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.700131+0000", 2024-05-15T12:57:19.991 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039675190000000001 2024-05-15T12:57:19.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.991 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.991 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:57:19.991 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.700158+0000", 2024-05-15T12:57:19.991 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6407999999999999e-05 2024-05-15T12:57:19.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.991 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.991 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.991 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.703940+0000", 2024-05-15T12:57:19.991 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003782058 2024-05-15T12:57:19.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.991 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.991 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.992 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.638526+0000", 2024-05-15T12:57:19.992 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.934585805999999 2024-05-15T12:57:19.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.992 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.992 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:57:19.992 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.638603+0000", 2024-05-15T12:57:19.992 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7503999999999995e-05 2024-05-15T12:57:19.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.992 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.992 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:57:19.992 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.638754+0000", 2024-05-15T12:57:19.992 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015118400000000001 2024-05-15T12:57:19.992 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:19.992 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:57:19.992 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:19.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.993 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.993 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119126 2.4 2:2fb2b24a:::benchmark_data_smithi177_78104_object11993:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:57:19.993 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.722151+0000", 2024-05-15T12:57:19.993 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 443.81407371, 2024-05-15T12:57:19.993 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.919300895999999, 2024-05-15T12:57:19.993 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:57:19.993 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:57:19.993 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:57:19.993 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:57:19.993 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:57:19.993 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119126 2024-05-15T12:57:19.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.994 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:57:19.994 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.994 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:57:19.994 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.722151+0000", 2024-05-15T12:57:19.994 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:19.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.994 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.994 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:57:19.994 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.722151+0000", 2024-05-15T12:57:19.994 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:19.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.994 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.994 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:57:19.995 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.722152+0000", 2024-05-15T12:57:19.995 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.201e-06 2024-05-15T12:57:19.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.995 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.995 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:57:19.995 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.722160+0000", 2024-05-15T12:57:19.995 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.388e-06 2024-05-15T12:57:19.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.996 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.996 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:57:19.996 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.722175+0000", 2024-05-15T12:57:19.996 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4363e-05 2024-05-15T12:57:19.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.996 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.996 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:57:19.996 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.722182+0000", 2024-05-15T12:57:19.996 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6799999999999993e-06 2024-05-15T12:57:19.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.996 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.997 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:19.997 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.722300+0000", 2024-05-15T12:57:19.997 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011772700000000001 2024-05-15T12:57:19.997 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.997 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.997 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:57:19.998 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.722462+0000", 2024-05-15T12:57:19.998 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016200399999999999 2024-05-15T12:57:19.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.998 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.998 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:57:19.998 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.722587+0000", 2024-05-15T12:57:19.998 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012485299999999999 2024-05-15T12:57:19.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.998 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.998 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.998 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.732412+0000", 2024-05-15T12:57:19.999 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0098255870000000002 2024-05-15T12:57:19.999 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.999 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.999 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:57:19.999 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.733985+0000", 2024-05-15T12:57:19.999 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001572389 2024-05-15T12:57:19.999 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.999 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.999 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:19.999 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.736745+0000", 2024-05-15T12:57:19.999 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0027598760000000001 2024-05-15T12:57:19.999 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:19.999 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:19.999 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.000 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.641257+0000", 2024-05-15T12:57:20.000 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.904512132000001 2024-05-15T12:57:20.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.000 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.000 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:57:20.000 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.641323+0000", 2024-05-15T12:57:20.000 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6563999999999995e-05 2024-05-15T12:57:20.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.000 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.000 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:57:20.001 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.641452+0000", 2024-05-15T12:57:20.001 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012813200000000001 2024-05-15T12:57:20.001 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:20.001 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:57:20.001 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:20.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.001 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119147 2.4 2:244e395e:::benchmark_data_smithi177_78104_object12014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:57:20.001 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.748010+0000", 2024-05-15T12:57:20.001 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 443.78821450800001, 2024-05-15T12:57:20.001 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.893615745, 2024-05-15T12:57:20.001 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:57:20.001 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:57:20.002 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:57:20.002 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:57:20.002 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:57:20.002 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119147 2024-05-15T12:57:20.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.002 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:57:20.002 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.002 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:57:20.002 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.748010+0000", 2024-05-15T12:57:20.002 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:20.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.002 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.002 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:57:20.003 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.748010+0000", 2024-05-15T12:57:20.003 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:20.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.003 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.003 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:57:20.003 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.748011+0000", 2024-05-15T12:57:20.003 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.609e-06 2024-05-15T12:57:20.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.003 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.003 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:57:20.003 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.748019+0000", 2024-05-15T12:57:20.003 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9529999999999999e-06 2024-05-15T12:57:20.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.004 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.004 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:57:20.004 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.748029+0000", 2024-05-15T12:57:20.004 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2299999999999997e-06 2024-05-15T12:57:20.004 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.004 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.004 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:57:20.004 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.748036+0000", 2024-05-15T12:57:20.004 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7430000000000002e-06 2024-05-15T12:57:20.004 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.004 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.004 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:20.004 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.748100+0000", 2024-05-15T12:57:20.005 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3907999999999999e-05 2024-05-15T12:57:20.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.005 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.005 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:57:20.005 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.748274+0000", 2024-05-15T12:57:20.005 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000174112 2024-05-15T12:57:20.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.005 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.005 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:57:20.005 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.748382+0000", 2024-05-15T12:57:20.005 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000107799 2024-05-15T12:57:20.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.005 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.005 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.006 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.752266+0000", 2024-05-15T12:57:20.006 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038838089999999998 2024-05-15T12:57:20.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.006 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.006 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:57:20.006 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.752326+0000", 2024-05-15T12:57:20.006 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0316999999999999e-05 2024-05-15T12:57:20.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.006 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.006 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.006 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.754303+0000", 2024-05-15T12:57:20.006 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001976233 2024-05-15T12:57:20.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.007 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.007 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.007 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.641493+0000", 2024-05-15T12:57:20.007 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.887190867999999 2024-05-15T12:57:20.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.007 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.007 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:57:20.007 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.641547+0000", 2024-05-15T12:57:20.007 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3338000000000003e-05 2024-05-15T12:57:20.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.007 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.007 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:57:20.007 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.641626+0000", 2024-05-15T12:57:20.008 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8825999999999994e-05 2024-05-15T12:57:20.008 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:20.008 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:57:20.008 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:20.008 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.008 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.008 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119150 2.4 2:27ad4cd4:::benchmark_data_smithi177_78104_object12017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:57:20.008 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.755494+0000", 2024-05-15T12:57:20.008 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 443.78073014099999, 2024-05-15T12:57:20.008 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.890362140000001, 2024-05-15T12:57:20.008 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:57:20.008 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:57:20.009 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:57:20.009 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:57:20.009 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:57:20.009 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119150 2024-05-15T12:57:20.009 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.009 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:57:20.009 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.009 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:57:20.009 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.755494+0000", 2024-05-15T12:57:20.009 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:20.009 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.009 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.009 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:57:20.009 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.755494+0000", 2024-05-15T12:57:20.009 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:20.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.010 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.010 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:57:20.010 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.755495+0000", 2024-05-15T12:57:20.010 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.066e-06 2024-05-15T12:57:20.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.010 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.010 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:57:20.010 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.755500+0000", 2024-05-15T12:57:20.010 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8640000000000004e-06 2024-05-15T12:57:20.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.011 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.011 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:57:20.011 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.755519+0000", 2024-05-15T12:57:20.011 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8470000000000001e-05 2024-05-15T12:57:20.011 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.011 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.011 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:57:20.011 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.755527+0000", 2024-05-15T12:57:20.011 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7909999999999993e-06 2024-05-15T12:57:20.011 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.011 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.012 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:20.012 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.755574+0000", 2024-05-15T12:57:20.012 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6794e-05 2024-05-15T12:57:20.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.012 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.012 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:57:20.012 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.755726+0000", 2024-05-15T12:57:20.012 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000151772 2024-05-15T12:57:20.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.012 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.012 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:57:20.012 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.755810+0000", 2024-05-15T12:57:20.012 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4252999999999994e-05 2024-05-15T12:57:20.013 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.013 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.013 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.013 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.759504+0000", 2024-05-15T12:57:20.013 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003693446 2024-05-15T12:57:20.013 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.013 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.013 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:57:20.013 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.759573+0000", 2024-05-15T12:57:20.013 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9017999999999995e-05 2024-05-15T12:57:20.013 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.013 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.013 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.014 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.761070+0000", 2024-05-15T12:57:20.014 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0014970020000000001 2024-05-15T12:57:20.014 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.014 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.014 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.014 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.645664+0000", 2024-05-15T12:57:20.014 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.884594117000001 2024-05-15T12:57:20.014 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.014 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.014 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:57:20.014 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.645723+0000", 2024-05-15T12:57:20.014 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8897000000000002e-05 2024-05-15T12:57:20.014 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.015 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.015 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:57:20.015 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.645856+0000", 2024-05-15T12:57:20.015 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013365 2024-05-15T12:57:20.015 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:20.015 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:57:20.015 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:20.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.015 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.015 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119168 2.4 2:2f7ee5d7:::benchmark_data_smithi177_78104_object12035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:57:20.015 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.789664+0000", 2024-05-15T12:57:20.015 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 443.74656016099999, 2024-05-15T12:57:20.015 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.856372735000001, 2024-05-15T12:57:20.015 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:57:20.016 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:57:20.016 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:57:20.016 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:57:20.016 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:57:20.016 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119168 2024-05-15T12:57:20.016 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.016 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:57:20.016 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.016 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:57:20.016 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.789664+0000", 2024-05-15T12:57:20.016 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:20.017 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.017 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.017 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:57:20.017 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.789664+0000", 2024-05-15T12:57:20.017 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:20.017 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.017 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.017 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:57:20.017 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.789666+0000", 2024-05-15T12:57:20.017 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6479999999999999e-06 2024-05-15T12:57:20.017 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.017 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.018 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:57:20.018 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.789673+0000", 2024-05-15T12:57:20.018 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2049999999999996e-06 2024-05-15T12:57:20.018 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.018 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.018 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:57:20.018 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.789686+0000", 2024-05-15T12:57:20.018 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3461999999999999e-05 2024-05-15T12:57:20.018 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.018 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.018 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:57:20.018 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.789694+0000", 2024-05-15T12:57:20.019 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5320000000000001e-06 2024-05-15T12:57:20.019 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.019 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.019 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:20.019 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.789776+0000", 2024-05-15T12:57:20.019 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1644999999999995e-05 2024-05-15T12:57:20.019 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.019 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.019 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:57:20.019 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.789886+0000", 2024-05-15T12:57:20.019 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000110123 2024-05-15T12:57:20.019 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.019 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.019 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:57:20.020 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.789975+0000", 2024-05-15T12:57:20.020 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8914999999999998e-05 2024-05-15T12:57:20.020 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.020 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.020 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:57:20.020 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.793355+0000", 2024-05-15T12:57:20.020 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003380508 2024-05-15T12:57:20.020 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.020 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.020 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.020 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.793536+0000", 2024-05-15T12:57:20.020 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00018034799999999999 2024-05-15T12:57:20.020 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.021 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.021 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.021 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.794419+0000", 2024-05-15T12:57:20.021 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00088304699999999998 2024-05-15T12:57:20.021 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.021 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.021 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.021 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.645905+0000", 2024-05-15T12:57:20.022 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.851486226 2024-05-15T12:57:20.022 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.022 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.022 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:57:20.022 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.645942+0000", 2024-05-15T12:57:20.022 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6860000000000003e-05 2024-05-15T12:57:20.022 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.022 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.022 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:57:20.022 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.646037+0000", 2024-05-15T12:57:20.023 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5216000000000005e-05 2024-05-15T12:57:20.023 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:20.023 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:57:20.023 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:20.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.023 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.023 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119181 2.4 2:297c213d:::benchmark_data_smithi177_78104_object12048:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:57:20.023 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.807440+0000", 2024-05-15T12:57:20.023 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 443.72878388700002, 2024-05-15T12:57:20.023 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.848339640000001, 2024-05-15T12:57:20.023 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:57:20.023 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:57:20.023 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:57:20.023 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:57:20.024 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:57:20.024 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119181 2024-05-15T12:57:20.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.024 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:57:20.024 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.024 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:57:20.024 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.807440+0000", 2024-05-15T12:57:20.024 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:20.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.024 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.025 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:57:20.025 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.807440+0000", 2024-05-15T12:57:20.025 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:20.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.025 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.025 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:57:20.025 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.807442+0000", 2024-05-15T12:57:20.025 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5409999999999999e-06 2024-05-15T12:57:20.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.025 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.025 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:57:20.025 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.807448+0000", 2024-05-15T12:57:20.026 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.942e-06 2024-05-15T12:57:20.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.026 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.026 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:57:20.026 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.807460+0000", 2024-05-15T12:57:20.027 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1881e-05 2024-05-15T12:57:20.027 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.027 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.027 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:57:20.027 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.807467+0000", 2024-05-15T12:57:20.027 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.627e-06 2024-05-15T12:57:20.027 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.027 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.027 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:20.027 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.807533+0000", 2024-05-15T12:57:20.027 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5537999999999995e-05 2024-05-15T12:57:20.028 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.028 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.028 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:57:20.028 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.807685+0000", 2024-05-15T12:57:20.028 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000152177 2024-05-15T12:57:20.028 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.028 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.028 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:57:20.028 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.807817+0000", 2024-05-15T12:57:20.028 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013225800000000001 2024-05-15T12:57:20.028 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.028 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.028 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:57:20.029 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.813241+0000", 2024-05-15T12:57:20.029 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0054233650000000003 2024-05-15T12:57:20.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.029 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.029 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.029 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.815003+0000", 2024-05-15T12:57:20.029 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00176184 2024-05-15T12:57:20.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.029 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.029 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.029 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.815055+0000", 2024-05-15T12:57:20.029 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2493e-05 2024-05-15T12:57:20.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.030 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.030 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.030 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.655595+0000", 2024-05-15T12:57:20.030 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.840539567 2024-05-15T12:57:20.030 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.030 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.030 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:57:20.030 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.655658+0000", 2024-05-15T12:57:20.030 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3590000000000006e-05 2024-05-15T12:57:20.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.031 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.031 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:57:20.031 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.655780+0000", 2024-05-15T12:57:20.031 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000121821 2024-05-15T12:57:20.031 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:20.031 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:57:20.031 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:20.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.031 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.031 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119196 2.4 2:2646c09a:::benchmark_data_smithi177_78104_object12063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:57:20.031 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.832154+0000", 2024-05-15T12:57:20.032 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 443.70406987500002, 2024-05-15T12:57:20.032 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.823821685, 2024-05-15T12:57:20.032 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:57:20.032 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:57:20.032 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:57:20.032 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:57:20.032 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:57:20.032 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119196 2024-05-15T12:57:20.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.032 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:57:20.033 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.033 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:57:20.033 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.832154+0000", 2024-05-15T12:57:20.033 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:20.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.033 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.033 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:57:20.033 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.832154+0000", 2024-05-15T12:57:20.033 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:20.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.033 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.033 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:57:20.034 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.832156+0000", 2024-05-15T12:57:20.034 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9609999999999999e-06 2024-05-15T12:57:20.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.034 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.034 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:57:20.034 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.832165+0000", 2024-05-15T12:57:20.034 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8310000000000007e-06 2024-05-15T12:57:20.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.034 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.034 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:57:20.034 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.832187+0000", 2024-05-15T12:57:20.034 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2093e-05 2024-05-15T12:57:20.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.035 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.035 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:57:20.035 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.832198+0000", 2024-05-15T12:57:20.035 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0516000000000001e-05 2024-05-15T12:57:20.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.035 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.035 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:20.035 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.832258+0000", 2024-05-15T12:57:20.035 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0010000000000003e-05 2024-05-15T12:57:20.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.035 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.035 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:57:20.035 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.832426+0000", 2024-05-15T12:57:20.036 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016802599999999999 2024-05-15T12:57:20.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.036 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.036 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:57:20.036 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.832593+0000", 2024-05-15T12:57:20.036 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000167071 2024-05-15T12:57:20.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.036 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.036 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:57:20.036 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.837325+0000", 2024-05-15T12:57:20.036 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0047316420000000003 2024-05-15T12:57:20.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.036 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.036 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.037 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.839375+0000", 2024-05-15T12:57:20.037 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020508560000000002 2024-05-15T12:57:20.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.037 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.037 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.037 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.843832+0000", 2024-05-15T12:57:20.037 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0044566340000000001 2024-05-15T12:57:20.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.037 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.037 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.037 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.655823+0000", 2024-05-15T12:57:20.037 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.811990489999999 2024-05-15T12:57:20.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.037 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.037 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:57:20.038 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.655872+0000", 2024-05-15T12:57:20.038 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9601999999999997e-05 2024-05-15T12:57:20.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.038 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:57:20.038 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.655976+0000", 2024-05-15T12:57:20.038 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000103953 2024-05-15T12:57:20.038 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:20.038 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:57:20.038 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:20.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.039 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119223 2.4 2:2892d1d8:::benchmark_data_smithi177_78104_object12090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:57:20.039 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.884405+0000", 2024-05-15T12:57:20.039 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 443.65181957700003, 2024-05-15T12:57:20.039 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.771771241, 2024-05-15T12:57:20.039 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:57:20.039 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:57:20.039 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:57:20.039 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:57:20.040 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:57:20.040 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119223 2024-05-15T12:57:20.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.040 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:57:20.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.040 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:57:20.040 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.884405+0000", 2024-05-15T12:57:20.040 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:20.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.040 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:57:20.040 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.884405+0000", 2024-05-15T12:57:20.040 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:20.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.040 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:57:20.041 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.884406+0000", 2024-05-15T12:57:20.041 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.469e-06 2024-05-15T12:57:20.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.041 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:57:20.041 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.884413+0000", 2024-05-15T12:57:20.041 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3890000000000002e-06 2024-05-15T12:57:20.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.041 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:57:20.041 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.884430+0000", 2024-05-15T12:57:20.041 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7638e-05 2024-05-15T12:57:20.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.042 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:57:20.042 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.884439+0000", 2024-05-15T12:57:20.042 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5699999999999993e-06 2024-05-15T12:57:20.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.042 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.042 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:20.042 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.884495+0000", 2024-05-15T12:57:20.042 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6388999999999998e-05 2024-05-15T12:57:20.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.042 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.042 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:57:20.043 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.884622+0000", 2024-05-15T12:57:20.043 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012727999999999999 2024-05-15T12:57:20.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.043 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.043 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:57:20.043 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.884715+0000", 2024-05-15T12:57:20.043 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2476e-05 2024-05-15T12:57:20.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.043 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.044 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:57:20.044 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.887826+0000", 2024-05-15T12:57:20.044 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031109029999999999 2024-05-15T12:57:20.044 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.044 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.044 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.044 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.888005+0000", 2024-05-15T12:57:20.044 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017930100000000001 2024-05-15T12:57:20.044 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.044 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.044 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.044 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.888537+0000", 2024-05-15T12:57:20.044 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00053228399999999995 2024-05-15T12:57:20.044 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.044 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.045 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.045 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.656015+0000", 2024-05-15T12:57:20.045 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.767477478 2024-05-15T12:57:20.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.045 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:57:20.045 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.656064+0000", 2024-05-15T12:57:20.045 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9458e-05 2024-05-15T12:57:20.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.045 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:57:20.045 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.656176+0000", 2024-05-15T12:57:20.045 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000111606 2024-05-15T12:57:20.045 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:20.046 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:57:20.046 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:20.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.046 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.046 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119236 2.4 2:222c6e22:::benchmark_data_smithi177_78104_object12103:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:57:20.046 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.915558+0000", 2024-05-15T12:57:20.046 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 443.62066640199998, 2024-05-15T12:57:20.046 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.740808504, 2024-05-15T12:57:20.046 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:57:20.046 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:57:20.046 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:57:20.046 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:57:20.046 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:57:20.047 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119236 2024-05-15T12:57:20.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.047 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:57:20.047 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.047 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:57:20.047 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.915558+0000", 2024-05-15T12:57:20.047 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:20.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.047 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.047 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:57:20.047 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.915558+0000", 2024-05-15T12:57:20.047 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:20.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.047 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.048 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:57:20.048 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.915559+0000", 2024-05-15T12:57:20.048 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4789999999999999e-06 2024-05-15T12:57:20.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.048 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.048 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:57:20.048 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.915565+0000", 2024-05-15T12:57:20.048 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6930000000000004e-06 2024-05-15T12:57:20.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.048 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.048 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:57:20.048 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.915578+0000", 2024-05-15T12:57:20.048 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3244000000000001e-05 2024-05-15T12:57:20.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.049 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.049 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:57:20.049 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.915586+0000", 2024-05-15T12:57:20.049 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4359999999999996e-06 2024-05-15T12:57:20.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.049 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.049 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:20.049 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.915637+0000", 2024-05-15T12:57:20.049 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1054999999999998e-05 2024-05-15T12:57:20.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.049 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.049 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:57:20.049 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.915778+0000", 2024-05-15T12:57:20.050 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014136800000000001 2024-05-15T12:57:20.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.050 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:57:20.050 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.915903+0000", 2024-05-15T12:57:20.050 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000124354 2024-05-15T12:57:20.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.050 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:57:20.050 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.919310+0000", 2024-05-15T12:57:20.050 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003407392 2024-05-15T12:57:20.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.051 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.051 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.919841+0000", 2024-05-15T12:57:20.051 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00053115300000000003 2024-05-15T12:57:20.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.051 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.051 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.924424+0000", 2024-05-15T12:57:20.051 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0045829829999999997 2024-05-15T12:57:20.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.051 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.052 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.656216+0000", 2024-05-15T12:57:20.052 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.731791498 2024-05-15T12:57:20.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.052 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:57:20.052 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.656264+0000", 2024-05-15T12:57:20.052 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8350000000000003e-05 2024-05-15T12:57:20.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.052 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:57:20.052 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.656366+0000", 2024-05-15T12:57:20.052 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000102499 2024-05-15T12:57:20.052 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:20.052 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:57:20.053 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:20.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.053 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119240 2.4 2:2ed0c2c5:::benchmark_data_smithi177_78104_object12107:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:57:20.053 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:55.924347+0000", 2024-05-15T12:57:20.053 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 443.61187726399999, 2024-05-15T12:57:20.053 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.732209246, 2024-05-15T12:57:20.053 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:57:20.053 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:57:20.053 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:57:20.054 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:57:20.054 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:57:20.054 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119240 2024-05-15T12:57:20.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.054 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:57:20.054 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.054 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:57:20.054 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.924347+0000", 2024-05-15T12:57:20.054 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:20.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.054 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.054 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:57:20.055 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.924347+0000", 2024-05-15T12:57:20.055 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:20.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.055 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:57:20.055 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.924348+0000", 2024-05-15T12:57:20.055 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4100000000000001e-06 2024-05-15T12:57:20.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.055 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:57:20.055 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.924356+0000", 2024-05-15T12:57:20.055 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3780000000000003e-06 2024-05-15T12:57:20.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.056 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:57:20.056 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.924371+0000", 2024-05-15T12:57:20.056 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4783000000000001e-05 2024-05-15T12:57:20.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.056 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:57:20.056 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.924381+0000", 2024-05-15T12:57:20.056 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0166999999999999e-05 2024-05-15T12:57:20.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.056 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:20.056 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.924454+0000", 2024-05-15T12:57:20.056 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3535999999999995e-05 2024-05-15T12:57:20.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.057 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:57:20.057 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.924553+0000", 2024-05-15T12:57:20.057 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8793000000000001e-05 2024-05-15T12:57:20.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.057 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:57:20.057 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.924634+0000", 2024-05-15T12:57:20.057 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0436e-05 2024-05-15T12:57:20.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.057 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.057 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.928341+0000", 2024-05-15T12:57:20.057 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0037073509999999998 2024-05-15T12:57:20.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.058 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.058 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.058 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.928376+0000", 2024-05-15T12:57:20.058 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4686e-05 2024-05-15T12:57:20.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.058 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.058 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:57:20.058 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:55.928434+0000", 2024-05-15T12:57:20.058 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7958999999999999e-05 2024-05-15T12:57:20.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.058 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.058 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.058 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.656405+0000", 2024-05-15T12:57:20.059 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.727971643 2024-05-15T12:57:20.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.059 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:57:20.059 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.656454+0000", 2024-05-15T12:57:20.059 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9110000000000002e-05 2024-05-15T12:57:20.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.059 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:57:20.059 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.656556+0000", 2024-05-15T12:57:20.059 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000101994 2024-05-15T12:57:20.059 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:20.059 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:57:20.059 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:20.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.060 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:119259 2.4 2:21f9b17e:::benchmark_data_smithi177_78104_object12126:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-05-15T12:57:20.060 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:56.045969+0000", 2024-05-15T12:57:20.060 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 443.490255171, 2024-05-15T12:57:20.060 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.610775774, 2024-05-15T12:57:20.060 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:57:20.060 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:57:20.060 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:57:20.060 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:57:20.060 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:57:20.060 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 119259 2024-05-15T12:57:20.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.060 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:57:20.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.061 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:57:20.061 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:56.045969+0000", 2024-05-15T12:57:20.061 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:20.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.061 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.061 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:57:20.061 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:56.045969+0000", 2024-05-15T12:57:20.061 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:57:20.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.061 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.061 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:57:20.061 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:56.045970+0000", 2024-05-15T12:57:20.061 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.125e-06 2024-05-15T12:57:20.062 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.062 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.062 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:57:20.062 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:56.045976+0000", 2024-05-15T12:57:20.062 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5380000000000002e-06 2024-05-15T12:57:20.062 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.062 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.062 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:57:20.062 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:56.045989+0000", 2024-05-15T12:57:20.062 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3305999999999999e-05 2024-05-15T12:57:20.062 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.062 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.062 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:57:20.062 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:56.045996+0000", 2024-05-15T12:57:20.063 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1359999999999999e-06 2024-05-15T12:57:20.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.063 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:57:20.063 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:56.046051+0000", 2024-05-15T12:57:20.063 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4388999999999997e-05 2024-05-15T12:57:20.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.063 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:57:20.063 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:56.046223+0000", 2024-05-15T12:57:20.063 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017190599999999999 2024-05-15T12:57:20.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.063 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,2,10", 2024-05-15T12:57:20.064 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:56.046387+0000", 2024-05-15T12:57:20.064 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000164451 2024-05-15T12:57:20.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.064 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.064 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.064 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:56.050121+0000", 2024-05-15T12:57:20.064 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003734163 2024-05-15T12:57:20.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.064 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.064 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:57:20.065 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:56.050203+0000", 2024-05-15T12:57:20.065 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2077999999999996e-05 2024-05-15T12:57:20.065 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.065 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.065 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.065 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:56.050256+0000", 2024-05-15T12:57:20.065 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2388000000000001e-05 2024-05-15T12:57:20.065 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.065 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.065 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:57:20.065 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.656595+0000", 2024-05-15T12:57:20.065 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 14.606338887 2024-05-15T12:57:20.065 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.065 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.066 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:57:20.066 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.656644+0000", 2024-05-15T12:57:20.066 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9110999999999998e-05 2024-05-15T12:57:20.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:57:20.066 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:57:20.066 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:57:20.066 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:50:10.656745+0000", 2024-05-15T12:57:20.066 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000101296 2024-05-15T12:57:20.066 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:20.066 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:57:20.066 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:20.066 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:57:20.066 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:57:20.067 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T12:57:20.067 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:20.068 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:19 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:57:20.068 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:19 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:57:20.068 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:19 smithi027 ceph-mon[39795]: osdmap e307: 12 total, 12 up, 10 in 2024-05-15T12:57:20.068 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:19 smithi027 ceph-mon[39795]: pgmap v837: 17 pgs: 17 active+clean; 377 MiB data, 3.3 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 758 op/s 2024-05-15T12:57:20.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:19 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:57:20.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:19 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:57:20.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:19 smithi177 ceph-mon[44303]: osdmap e307: 12 total, 12 up, 10 in 2024-05-15T12:57:20.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:19 smithi177 ceph-mon[44303]: pgmap v837: 17 pgs: 17 active+clean; 377 MiB data, 3.3 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 758 op/s 2024-05-15T12:57:20.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:19 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:57:20.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:19 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:57:20.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:19 smithi120 ceph-mon[43961]: osdmap e307: 12 total, 12 up, 10 in 2024-05-15T12:57:20.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:19 smithi120 ceph-mon[43961]: pgmap v837: 17 pgs: 17 active+clean; 377 MiB data, 3.3 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 758 op/s 2024-05-15T12:57:22.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:21 smithi177 ceph-mon[44303]: pgmap v838: 17 pgs: 17 active+clean; 288 MiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 720 op/s 2024-05-15T12:57:22.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:21 smithi027 ceph-mon[39795]: pgmap v838: 17 pgs: 17 active+clean; 288 MiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 720 op/s 2024-05-15T12:57:22.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:21 smithi120 ceph-mon[43961]: pgmap v838: 17 pgs: 17 active+clean; 288 MiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 720 op/s 2024-05-15T12:57:22.646 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T12:57:23.827 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:57:23.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:23.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:57:23.850 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-05-15T12:57:23.938 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:57:24.008 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:24.139 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:24 smithi027 ceph-mon[39795]: pgmap v839: 17 pgs: 17 active+clean; 167 MiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 812 op/s 2024-05-15T12:57:24.163 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:57:24.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:24.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:57:24.270 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:57:24.299 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:24.313 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Removed 133619 objects 2024-05-15T12:57:24.314 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Clean up completed and total clean up time :164.303 2024-05-15T12:57:24.320 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:57:24.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:24.323 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:57:24.425 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:57:24.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:24 smithi177 ceph-mon[44303]: pgmap v839: 17 pgs: 17 active+clean; 167 MiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 812 op/s 2024-05-15T12:57:24.451 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:24.470 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:57:24.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:24.474 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:57:24.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:24 smithi120 ceph-mon[43961]: pgmap v839: 17 pgs: 17 active+clean; 167 MiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 812 op/s 2024-05-15T12:57:24.575 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:57:24.607 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:24.626 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:57:24.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:24.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:57:24.730 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:57:24.762 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:24.782 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:57:24.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:24.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:57:24.886 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:57:24.915 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:25.149 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.5 to 1 (10000) 2024-05-15T12:57:25.157 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:57:25.157 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:57:25.164 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:57:25.166 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:25 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:57:25.166 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:25 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:57:25.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:25 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2119911492' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T12:57:25.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:25 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1352581230' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:57:25.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:25.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:57:25.170 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:57:25.202 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:57:25.207 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:57:25.229 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:57:25.250 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:57:25.268 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:57:25.270 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:57:25.290 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:57:25.300 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:25.302 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-05-15T12:57:25.303 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-05-15T12:57:25.310 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:57:25.322 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:57:25.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:25.325 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:57:25.333 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:57:25.373 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:57:25.393 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:57:25.395 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:57:25.395 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:57:25.395 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:57:25.395 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:57:25.395 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:57:25.395 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:57:25.395 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:57:25.395 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:57:25.396 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:57:25.396 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:57:25.396 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:57:25.396 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:57:25.426 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:57:25.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:25 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:57:25.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:25 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:57:25.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:25 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2119911492' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T12:57:25.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:25 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1352581230' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:57:25.456 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:25 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:57:25.456 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:25 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:57:25.456 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:25 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2119911492' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T12:57:25.456 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:25 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1352581230' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:57:25.458 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:25.479 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:57:25.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:25.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:57:25.583 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:57:25.610 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:25.732 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:57:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:57:25.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:25.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:57:25.836 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:57:25.868 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:25.889 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:57:25.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:25.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:57:25.993 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:57:26.025 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:26.046 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:57:26.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:26.049 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:57:26.150 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:57:26.181 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:26.202 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:57:26.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:26.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:57:26.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:26 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2119911492' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T12:57:26.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:26 smithi027 ceph-mon[39795]: osdmap e308: 12 total, 12 up, 10 in 2024-05-15T12:57:26.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:26 smithi027 ceph-mon[39795]: pgmap v841: 17 pgs: 17 active+clean; 36 MiB data, 2.4 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 917 op/s 2024-05-15T12:57:26.305 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:57:26.334 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:26 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2119911492' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T12:57:26.334 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:26 smithi177 ceph-mon[44303]: osdmap e308: 12 total, 12 up, 10 in 2024-05-15T12:57:26.334 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:26 smithi177 ceph-mon[44303]: pgmap v841: 17 pgs: 17 active+clean; 36 MiB data, 2.4 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 917 op/s 2024-05-15T12:57:26.337 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:26.358 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:57:26.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:26.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:57:26.463 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:57:26.491 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:26.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:26.523 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:57:26.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:57:26.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:26 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2119911492' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T12:57:26.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:26 smithi120 ceph-mon[43961]: osdmap e308: 12 total, 12 up, 10 in 2024-05-15T12:57:26.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:26 smithi120 ceph-mon[43961]: pgmap v841: 17 pgs: 17 active+clean; 36 MiB data, 2.4 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 917 op/s 2024-05-15T12:57:26.624 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:57:26.651 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:57:26.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:26.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:57:27.003 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:57:27.031 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:27.179 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:57:27.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:27.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:57:27.289 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:57:27.317 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:57:27.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:27.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:57:27.770 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:57:27.799 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:27.852 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:57:27.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:27.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:57:27.956 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:57:27.988 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:28.010 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:57:28.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:28.013 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:57:28.114 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:57:28.143 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:28.163 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:57:28.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:28.166 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:57:28.267 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:57:28.300 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:28.322 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:57:28.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:28.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:57:28.426 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:57:28.455 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:28.476 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:57:28.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:28.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:57:28.580 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:57:28.612 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:28.632 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:57:28.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:28.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:57:28.736 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:57:28.769 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:28.789 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:57:28.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:28.792 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:57:28.893 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:57:28.923 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:28.943 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:57:28.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:28.946 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:57:29.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:28 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:57:29.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:28 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:57:29.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:28 smithi027 ceph-mon[39795]: pgmap v842: 17 pgs: 17 active+clean; 17 MiB data, 2.3 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 707 op/s 2024-05-15T12:57:29.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:28 smithi027 ceph-mon[39795]: osdmap e309: 12 total, 12 up, 10 in 2024-05-15T12:57:29.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:28 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:57:29.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:28 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:57:29.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:28 smithi120 ceph-mon[43961]: pgmap v842: 17 pgs: 17 active+clean; 17 MiB data, 2.3 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 707 op/s 2024-05-15T12:57:29.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:28 smithi120 ceph-mon[43961]: osdmap e309: 12 total, 12 up, 10 in 2024-05-15T12:57:29.047 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:57:29.079 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:29.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:28 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:57:29.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:28 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:57:29.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:28 smithi177 ceph-mon[44303]: pgmap v842: 17 pgs: 17 active+clean; 17 MiB data, 2.3 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 707 op/s 2024-05-15T12:57:29.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:28 smithi177 ceph-mon[44303]: osdmap e309: 12 total, 12 up, 10 in 2024-05-15T12:57:29.833 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:57:29.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:29.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:57:29.842 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:57:29.876 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 6 1 2024-05-15T12:57:29.938 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:57:29.970 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:29.991 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:57:29.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:29.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:57:30.095 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:57:30.125 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:30.146 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:57:30.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:30.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:57:30.251 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:57:30.282 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:30.303 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:57:30.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:30.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:57:30.407 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:57:30.435 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:30.814 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T12:57:31.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:30 smithi177 ceph-mon[44303]: pgmap v844: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 574 op/s 2024-05-15T12:57:31.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:30 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3538104816' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:57:31.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:30 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:57:31.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:30 smithi027 ceph-mon[39795]: pgmap v844: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 574 op/s 2024-05-15T12:57:31.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:30 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3538104816' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:57:31.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:30 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:57:31.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:30 smithi120 ceph-mon[43961]: pgmap v844: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 574 op/s 2024-05-15T12:57:31.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:30 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3538104816' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:57:31.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:30 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:57:31.430 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:57:31.486 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:57:31.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:31.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:57:31.590 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:57:31.617 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:31.743 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:57:31.744 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:57:31.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:31.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:57:31.848 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:57:31.877 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:31.906 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:57:31.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:31.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:57:32.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:31 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:57:32.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:31 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:57:32.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:31 smithi027 ceph-mon[39795]: osdmap e310: 12 total, 12 up, 10 in 2024-05-15T12:57:32.029 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:57:32.059 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:31 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:57:32.059 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:31 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:57:32.059 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:31 smithi177 ceph-mon[44303]: osdmap e310: 12 total, 12 up, 10 in 2024-05-15T12:57:32.061 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:32.083 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:57:32.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:32.086 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:57:32.187 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:57:32.216 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:31 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:57:32.216 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:31 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:57:32.216 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:31 smithi120 ceph-mon[43961]: osdmap e310: 12 total, 12 up, 10 in 2024-05-15T12:57:32.219 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:32.239 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:57:32.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:32.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:57:32.344 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:57:32.376 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:32.398 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:57:32.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:32.401 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:57:32.502 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:57:32.534 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:32.554 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:57:32.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:32.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:57:32.658 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:57:32.687 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:32.707 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:57:32.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:32.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:57:32.811 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:57:32.839 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:32.859 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:57:32.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:32.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:57:32.962 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:57:32.991 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:33.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:32 smithi177 ceph-mon[44303]: pgmap v846: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 244 op/s 2024-05-15T12:57:33.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:32 smithi027 ceph-mon[39795]: pgmap v846: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 244 op/s 2024-05-15T12:57:33.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:32 smithi120 ceph-mon[43961]: pgmap v846: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 244 op/s 2024-05-15T12:57:33.340 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:57:33.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:33.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:57:33.443 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:57:33.476 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:33.497 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:57:33.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:33.500 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:57:33.601 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:57:33.630 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:33.972 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T12:57:33.972 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:57:34.376 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:34 smithi027 ceph-mon[39795]: pgmap v847: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 71 op/s 2024-05-15T12:57:34.378 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:57:34.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:34.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:57:34.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:34 smithi177 ceph-mon[44303]: pgmap v847: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 71 op/s 2024-05-15T12:57:34.488 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:57:34.506 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T12:57:34.520 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:34.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:34 smithi120 ceph-mon[43961]: pgmap v847: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 71 op/s 2024-05-15T12:57:34.541 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:57:34.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:34.545 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:57:34.646 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:57:34.675 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:34.696 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:57:34.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:34.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:57:34.701 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-05-15T12:57:34.703 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:57:34.801 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:57:34.833 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:34.854 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:57:34.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:34.857 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:57:34.958 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:57:34.986 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:35.011 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:57:35.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:35.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:57:35.116 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:57:35.148 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:35.168 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:57:35.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:35.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:57:35.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:35 smithi027 ceph-mon[39795]: 3.3 scrub starts 2024-05-15T12:57:35.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:35 smithi027 ceph-mon[39795]: 3.3 scrub ok 2024-05-15T12:57:35.272 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:57:35.304 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:35.427 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:57:35.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:35 smithi177 ceph-mon[44303]: 3.3 scrub starts 2024-05-15T12:57:35.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:35 smithi177 ceph-mon[44303]: 3.3 scrub ok 2024-05-15T12:57:35.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:35.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:57:35.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:35 smithi120 ceph-mon[43961]: 3.3 scrub starts 2024-05-15T12:57:35.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:35 smithi120 ceph-mon[43961]: 3.3 scrub ok 2024-05-15T12:57:35.547 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:57:35.579 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:35.600 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:57:35.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:35.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:57:35.704 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:57:35.735 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:35.755 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:57:35.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:35.759 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:57:35.859 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:57:35.890 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:35.910 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:57:35.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:35.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:57:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:57:36.014 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:57:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:36.136 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_1' removed 2024-05-15T12:57:36.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:36 smithi027 ceph-mon[39795]: 3.f scrub starts 2024-05-15T12:57:36.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:36 smithi027 ceph-mon[39795]: 3.f scrub ok 2024-05-15T12:57:36.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:36 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3612449333' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T12:57:36.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:36 smithi027 ceph-mon[39795]: 3.0 scrub starts 2024-05-15T12:57:36.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:36 smithi027 ceph-mon[39795]: 3.1 scrub starts 2024-05-15T12:57:36.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:36 smithi027 ceph-mon[39795]: 3.1 scrub ok 2024-05-15T12:57:36.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:36 smithi027 ceph-mon[39795]: pgmap v848: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 32 op/s 2024-05-15T12:57:36.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:36 smithi120 ceph-mon[43961]: 3.f scrub starts 2024-05-15T12:57:36.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:36 smithi120 ceph-mon[43961]: 3.f scrub ok 2024-05-15T12:57:36.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:36 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3612449333' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T12:57:36.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:36 smithi120 ceph-mon[43961]: 3.0 scrub starts 2024-05-15T12:57:36.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:36 smithi120 ceph-mon[43961]: 3.1 scrub starts 2024-05-15T12:57:36.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:36 smithi120 ceph-mon[43961]: 3.1 scrub ok 2024-05-15T12:57:36.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:36 smithi120 ceph-mon[43961]: pgmap v848: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 32 op/s 2024-05-15T12:57:36.331 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:57:36.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:36.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:57:36.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:36 smithi177 ceph-mon[44303]: 3.f scrub starts 2024-05-15T12:57:36.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:36 smithi177 ceph-mon[44303]: 3.f scrub ok 2024-05-15T12:57:36.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:36 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3612449333' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T12:57:36.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:36 smithi177 ceph-mon[44303]: 3.0 scrub starts 2024-05-15T12:57:36.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:36 smithi177 ceph-mon[44303]: 3.1 scrub starts 2024-05-15T12:57:36.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:36 smithi177 ceph-mon[44303]: 3.1 scrub ok 2024-05-15T12:57:36.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:36 smithi177 ceph-mon[44303]: pgmap v848: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 32 op/s 2024-05-15T12:57:36.434 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:57:36.460 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:36.479 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:57:36.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:36.482 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:57:36.583 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:57:36.612 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:36.630 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:57:36.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:36.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:57:36.734 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:57:36.764 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:37.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:37 smithi177 ceph-mon[44303]: 3.0 scrub ok 2024-05-15T12:57:37.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:37 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3612449333' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T12:57:37.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:37 smithi177 ceph-mon[44303]: osdmap e311: 12 total, 12 up, 10 in 2024-05-15T12:57:37.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:37 smithi027 ceph-mon[39795]: 3.0 scrub ok 2024-05-15T12:57:37.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:37 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3612449333' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T12:57:37.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:37 smithi027 ceph-mon[39795]: osdmap e311: 12 total, 12 up, 10 in 2024-05-15T12:57:37.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:37 smithi120 ceph-mon[43961]: 3.0 scrub ok 2024-05-15T12:57:37.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:37 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3612449333' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T12:57:37.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:37 smithi120 ceph-mon[43961]: osdmap e311: 12 total, 12 up, 10 in 2024-05-15T12:57:37.648 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-05-15T12:57:37.648 INFO:tasks.radosbench:Beginning radosbench... 2024-05-15T12:57:37.649 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-05-15T12:57:37.649 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-05-15T12:57:38.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:38 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:57:38.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:38 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:57:38.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:38 smithi027 ceph-mon[39795]: osdmap e312: 12 total, 12 up, 10 in 2024-05-15T12:57:38.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:38 smithi027 ceph-mon[39795]: pgmap v851: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:38.314 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:57:38.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:38.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:57:38.418 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:57:38.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:38 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:57:38.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:38 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:57:38.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:38 smithi177 ceph-mon[44303]: osdmap e312: 12 total, 12 up, 10 in 2024-05-15T12:57:38.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:38 smithi177 ceph-mon[44303]: pgmap v851: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:38.450 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:38.471 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:57:38.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:38.475 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:57:38.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:38 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:57:38.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:38 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:57:38.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:38 smithi120 ceph-mon[43961]: osdmap e312: 12 total, 12 up, 10 in 2024-05-15T12:57:38.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:38 smithi120 ceph-mon[43961]: pgmap v851: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:38.575 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:57:38.605 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:38.624 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:57:38.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:38.627 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:57:38.728 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:57:38.759 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:39.154 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.6 to 1 (10000) 2024-05-15T12:57:39.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:39 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2536872228' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T12:57:39.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:39 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2536872228' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T12:57:39.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:39 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2536872228' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T12:57:40.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:40 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:57:40.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:40 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:57:40.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:40 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2536872228' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T12:57:40.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:40 smithi177 ceph-mon[44303]: osdmap e313: 12 total, 12 up, 10 in 2024-05-15T12:57:40.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:40 smithi177 ceph-mon[44303]: pgmap v853: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:40.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:40 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:57:40.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:40 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:57:40.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:40 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2536872228' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T12:57:40.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:40 smithi027 ceph-mon[39795]: osdmap e313: 12 total, 12 up, 10 in 2024-05-15T12:57:40.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:40 smithi027 ceph-mon[39795]: pgmap v853: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:40.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:40 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:57:40.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:40 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:57:40.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:40 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2536872228' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T12:57:40.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:40 smithi120 ceph-mon[43961]: osdmap e313: 12 total, 12 up, 10 in 2024-05-15T12:57:40.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:40 smithi120 ceph-mon[43961]: pgmap v853: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:42.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:42 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:57:42.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:42 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:57:42.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:42 smithi027 ceph-mon[39795]: osdmap e314: 12 total, 12 up, 10 in 2024-05-15T12:57:42.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:42 smithi027 ceph-mon[39795]: pgmap v855: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:42.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:42 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:57:42.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:42 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:57:42.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:42 smithi120 ceph-mon[43961]: osdmap e314: 12 total, 12 up, 10 in 2024-05-15T12:57:42.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:42 smithi120 ceph-mon[43961]: pgmap v855: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:42.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:42 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:57:42.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:42 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:57:42.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:42 smithi177 ceph-mon[44303]: osdmap e314: 12 total, 12 up, 10 in 2024-05-15T12:57:42.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:42 smithi177 ceph-mon[44303]: pgmap v855: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:43.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:43 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:57:43.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:43 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:57:43.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:43 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:57:43.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:43 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:57:43.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:43 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:57:43.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:43 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:57:44.614 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:57:44.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:44.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:57:44.719 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:57:44.752 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:44.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:44 smithi027 ceph-mon[39795]: osdmap e315: 12 total, 12 up, 10 in 2024-05-15T12:57:44.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:44 smithi027 ceph-mon[39795]: pgmap v857: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:44.773 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:57:44.774 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:44 smithi177 ceph-mon[44303]: osdmap e315: 12 total, 12 up, 10 in 2024-05-15T12:57:44.774 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:44 smithi177 ceph-mon[44303]: pgmap v857: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:44.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:44.776 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:57:44.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:44 smithi120 ceph-mon[43961]: osdmap e315: 12 total, 12 up, 10 in 2024-05-15T12:57:44.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:44 smithi120 ceph-mon[43961]: pgmap v857: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:44.877 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:57:44.909 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:44.931 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:57:44.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:44.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:57:45.035 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:57:45.064 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:45.083 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:57:45.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:45.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:57:45.187 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:57:45.210 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 8 1 2024-05-15T12:57:45.215 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:45.234 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:57:45.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:45.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:57:45.338 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:57:45.366 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:45.514 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:57:45.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:45.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:57:45.619 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:57:45.651 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:45.672 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:57:45.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:45.676 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:57:45.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:45 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:57:45.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:45 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:57:45.776 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:57:45.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:45 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:57:45.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:45 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:57:45.804 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:45 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:57:45.804 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:45 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:57:45.805 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:45.825 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:57:45.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:45.828 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:57:45.838 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:57:45.852 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:57:45.871 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:57:45.891 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:57:45.911 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:57:45.929 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:57:45.931 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:57:45.952 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:57:45.957 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:45.972 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:57:45.976 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:57:45.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:45.979 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:57:45.994 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:57:46.015 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:57:46.036 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:57:46.056 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:57:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:57:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:57:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:57:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:57:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:57:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:57:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:57:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:57:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:57:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:57:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:57:46.080 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:57:46.110 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:46.350 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:57:46.359 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:57:46.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:46.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:57:46.464 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:57:46.493 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:46.512 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:57:46.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:46.516 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:57:46.617 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:57:46.643 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:46 smithi027 ceph-mon[39795]: osdmap e316: 12 total, 12 up, 10 in 2024-05-15T12:57:46.643 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:46 smithi027 ceph-mon[39795]: pgmap v859: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:46.643 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:46 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2451462320' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:57:46.645 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:46.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:46 smithi120 ceph-mon[43961]: osdmap e316: 12 total, 12 up, 10 in 2024-05-15T12:57:46.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:46 smithi120 ceph-mon[43961]: pgmap v859: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:46.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:46 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2451462320' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:57:46.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:46 smithi177 ceph-mon[44303]: osdmap e316: 12 total, 12 up, 10 in 2024-05-15T12:57:46.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:46 smithi177 ceph-mon[44303]: pgmap v859: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:46.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:46 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2451462320' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:57:47.073 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:57:47.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:47.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:57:47.177 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:57:47.210 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:47.231 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:57:47.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:47.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:57:47.336 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:57:47.365 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:47.385 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:57:47.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:47.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:57:47.489 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:57:47.518 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:47.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:47 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T12:57:47.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:47 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T12:57:47.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:47 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T12:57:47.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:47 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T12:57:47.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:47 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T12:57:47.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:47 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T12:57:48.501 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:57:48.501 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:57:48.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:48 smithi027 ceph-mon[39795]: pgmap v860: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:48.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:48 smithi177 ceph-mon[44303]: pgmap v860: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:49.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:48 smithi120 ceph-mon[43961]: pgmap v860: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:49.068 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:57:49.174 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T12:57:49.174 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T12:57:49.174 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-05-15T12:57:49.174 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-05-15T12:57:49.174 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T12:57:49.670 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:57:49.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:49.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:57:49.775 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:57:49.804 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:50.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:49 smithi027 ceph-mon[39795]: pgmap v861: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:50.154 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:57:50.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:50.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:57:50.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:49 smithi177 ceph-mon[44303]: pgmap v861: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:50.259 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:57:50.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:49 smithi120 ceph-mon[43961]: pgmap v861: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:50.291 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:50.312 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:57:50.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:50.315 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:57:50.416 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:57:50.445 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:52.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:51 smithi027 ceph-mon[39795]: pgmap v862: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:52.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-05-15T12:57:52.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:51 smithi177 ceph-mon[44303]: pgmap v862: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:52.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:51 smithi120 ceph-mon[43961]: pgmap v862: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:52.997 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:57:52.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:53.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:57:53.101 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:57:53.134 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:53.156 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:57:53.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:53.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:57:53.260 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:57:53.293 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:53.499 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:57:53.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:53.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:57:53.604 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:57:53.631 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:53.781 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:57:53.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:53.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:57:53.885 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:57:53.914 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:54.141 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_2' created 2024-05-15T12:57:54.170 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:54 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/227796879' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-05-15T12:57:54.170 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:54 smithi027 ceph-mon[39795]: pgmap v863: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:54.170 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:54 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/967815697' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:57:54.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:54 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/227796879' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-05-15T12:57:54.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:54 smithi177 ceph-mon[44303]: pgmap v863: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:54.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:54 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/967815697' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:57:54.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:54 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/227796879' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-05-15T12:57:54.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:54 smithi120 ceph-mon[43961]: pgmap v863: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:54.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:54 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/967815697' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:57:54.863 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:57:54.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:54.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:57:54.973 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:57:55.005 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:55.026 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:57:55.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:55.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:57:55.130 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:57:55.150 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.8 to 1 (10000) 2024-05-15T12:57:55.163 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:55.183 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:57:55.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:55.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:57:55.279 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-05-15T12:57:55.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:55 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:57:55.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:55 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T12:57:55.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:55 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/227796879' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-05-15T12:57:55.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:55 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/967815697' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:57:55.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:55 smithi120 ceph-mon[43961]: osdmap e317: 12 total, 12 up, 10 in 2024-05-15T12:57:55.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:55 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3061709313' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T12:57:55.287 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:57:55.315 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:55.336 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:57:55.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:55.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:57:55.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:55 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:57:55.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:55 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T12:57:55.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:55 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/227796879' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-05-15T12:57:55.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:55 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/967815697' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:57:55.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:55 smithi177 ceph-mon[44303]: osdmap e317: 12 total, 12 up, 10 in 2024-05-15T12:57:55.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:55 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3061709313' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T12:57:55.439 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:57:55.467 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:55.486 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:57:55.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:55.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:57:55.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:55 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:57:55.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:55 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T12:57:55.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:55 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/227796879' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-05-15T12:57:55.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:55 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/967815697' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:57:55.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:55 smithi027 ceph-mon[39795]: osdmap e317: 12 total, 12 up, 10 in 2024-05-15T12:57:55.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:55 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3061709313' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T12:57:55.589 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:57:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:55.784 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T12:57:56.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:56 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3061709313' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T12:57:56.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:56 smithi177 ceph-mon[44303]: osdmap e318: 12 total, 12 up, 10 in 2024-05-15T12:57:56.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:56 smithi177 ceph-mon[44303]: pgmap v866: 17 pgs: 3 creating+peering, 7 unknown, 7 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:56.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:56 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3061709313' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T12:57:56.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:56 smithi027 ceph-mon[39795]: osdmap e318: 12 total, 12 up, 10 in 2024-05-15T12:57:56.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:56 smithi027 ceph-mon[39795]: pgmap v866: 17 pgs: 3 creating+peering, 7 unknown, 7 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:56.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:56 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3061709313' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T12:57:56.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:56 smithi120 ceph-mon[43961]: osdmap e318: 12 total, 12 up, 10 in 2024-05-15T12:57:56.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:56 smithi120 ceph-mon[43961]: pgmap v866: 17 pgs: 3 creating+peering, 7 unknown, 7 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:56.567 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:57:56.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:56.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:57:56.673 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:57:56.705 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:56.726 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:57:56.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:56.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:57:56.830 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:57:56.863 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:56.885 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:57:56.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:56.888 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:57:56.989 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:57:57.019 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:57.039 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:57:57.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:57.042 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:57:57.100 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 9 1 2024-05-15T12:57:57.110 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:57:57.142 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:57:57.157 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:57:57.170 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:57.190 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:57:57.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:57.193 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:57:57.293 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:57:57.326 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:57.347 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:57:57.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:57.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:57:57.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:57 smithi177 ceph-mon[44303]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T12:57:57.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:57 smithi177 ceph-mon[44303]: osdmap e319: 12 total, 12 up, 10 in 2024-05-15T12:57:57.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:57 smithi027 ceph-mon[39795]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T12:57:57.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:57 smithi027 ceph-mon[39795]: osdmap e319: 12 total, 12 up, 10 in 2024-05-15T12:57:57.450 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:57:57.479 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:57.490 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:57:57.499 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:57:57.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:57.502 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:57:57.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:57 smithi120 ceph-mon[43961]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T12:57:57.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:57 smithi120 ceph-mon[43961]: osdmap e319: 12 total, 12 up, 10 in 2024-05-15T12:57:57.603 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:57:57.635 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:57.657 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:57:57.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:57.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:57:57.761 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:57:57.790 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:57.809 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:57:57.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:57.813 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:57:57.914 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:57:57.946 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:57.967 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:57:57.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:57.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:57:58.071 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:57:58.098 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:58.118 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:57:58.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:58.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:57:58.222 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:57:58.252 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:58 smithi120 ceph-mon[43961]: pgmap v868: 17 pgs: 3 creating+peering, 6 unknown, 8 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:58.254 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:58.276 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:57:58.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:57:58.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:57:58.380 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:57:58.405 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:58 smithi027 ceph-mon[39795]: pgmap v868: 17 pgs: 3 creating+peering, 6 unknown, 8 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:58.408 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:57:58.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:58 smithi177 ceph-mon[44303]: pgmap v868: 17 pgs: 3 creating+peering, 6 unknown, 8 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:57:58.580 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T12:58:00.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:57:59 smithi177 ceph-mon[44303]: pgmap v869: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:58:00.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:57:59 smithi027 ceph-mon[39795]: pgmap v869: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:58:00.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:57:59 smithi120 ceph-mon[43961]: pgmap v869: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:58:01.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:00 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:58:01.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:00 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:58:01.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:00 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:58:01.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:00 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:58:01.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:00 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:58:01.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:00 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:58:01.661 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:58:01.664 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:58:01.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:01.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:58:01.698 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:58:01.766 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:58:01.798 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:01.818 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:58:01.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:01.822 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:58:01.879 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-05-15T12:58:01.923 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:58:01.933 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:01 smithi027 ceph-mon[39795]: osdmap e320: 12 total, 12 up, 10 in 2024-05-15T12:58:01.933 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:01 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2973834091' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:58:01.934 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:01 smithi027 ceph-mon[39795]: pgmap v871: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:58:01.952 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:01.973 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:58:01.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:01.976 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:58:02.077 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:58:02.110 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:02.131 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:58:02.132 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:01 smithi120 ceph-mon[43961]: osdmap e320: 12 total, 12 up, 10 in 2024-05-15T12:58:02.133 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:01 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2973834091' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:58:02.133 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:01 smithi120 ceph-mon[43961]: pgmap v871: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:58:02.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:02.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:58:02.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:01 smithi177 ceph-mon[44303]: osdmap e320: 12 total, 12 up, 10 in 2024-05-15T12:58:02.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:01 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2973834091' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:58:02.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:01 smithi177 ceph-mon[44303]: pgmap v871: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:58:02.235 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:58:02.266 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:03.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:02 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2973834091' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T12:58:03.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:02 smithi177 ceph-mon[44303]: osdmap e321: 12 total, 12 up, 10 in 2024-05-15T12:58:03.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:02 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2973834091' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T12:58:03.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:02 smithi027 ceph-mon[39795]: osdmap e321: 12 total, 12 up, 10 in 2024-05-15T12:58:03.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:02 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2973834091' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T12:58:03.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:02 smithi120 ceph-mon[43961]: osdmap e321: 12 total, 12 up, 10 in 2024-05-15T12:58:03.547 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:58:03.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:03.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:58:03.652 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:58:03.684 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:03.705 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:58:03.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:03.708 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:58:03.809 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:58:03.841 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:03.862 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:58:03.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:03.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:58:03.966 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:58:03.996 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:03 smithi177 ceph-mon[44303]: pgmap v873: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:58:03.998 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:04.019 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:58:04.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:03 smithi027 ceph-mon[39795]: pgmap v873: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:58:04.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:04.022 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:58:04.123 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:58:04.154 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:04.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:03 smithi120 ceph-mon[43961]: pgmap v873: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:58:04.475 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T12:58:04.475 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T12:58:04.475 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T12:58:04.475 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T12:58:04.475 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.475 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:51713 2.1 2:8f803db6:::benchmark_data_smithi177_78104_object51712:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-05-15T12:58:04.476 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:42.601217+0000", 2024-05-15T12:58:04.476 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 560.72735518299999, 2024-05-15T12:58:04.476 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.098120633999999998, 2024-05-15T12:58:04.476 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:58:04.476 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:04.476 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:58:04.476 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:58:04.476 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:58:04.476 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 51713 2024-05-15T12:58:04.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.476 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:58:04.476 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.476 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:58:04.477 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.601217+0000", 2024-05-15T12:58:04.477 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.477 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.477 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:58:04.477 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.601217+0000", 2024-05-15T12:58:04.477 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.477 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.477 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:58:04.477 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.601218+0000", 2024-05-15T12:58:04.477 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0589999999999999e-06 2024-05-15T12:58:04.478 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.478 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:58:04.478 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.601366+0000", 2024-05-15T12:58:04.478 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014783099999999999 2024-05-15T12:58:04.478 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.478 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:58:04.478 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.601377+0000", 2024-05-15T12:58:04.478 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.077e-05 2024-05-15T12:58:04.478 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.479 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:58:04.479 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.601386+0000", 2024-05-15T12:58:04.479 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7069999999999998e-06 2024-05-15T12:58:04.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.479 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:58:04.479 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.602166+0000", 2024-05-15T12:58:04.479 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00078028799999999997 2024-05-15T12:58:04.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.479 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:58:04.479 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.602279+0000", 2024-05-15T12:58:04.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000112859 2024-05-15T12:58:04.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.480 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.480 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,3", 2024-05-15T12:58:04.480 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.602412+0000", 2024-05-15T12:58:04.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013315899999999999 2024-05-15T12:58:04.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.480 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.480 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.480 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.690434+0000", 2024-05-15T12:58:04.481 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.088022144999999996 2024-05-15T12:58:04.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.690477+0000", 2024-05-15T12:58:04.481 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2521999999999997e-05 2024-05-15T12:58:04.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:58:04.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.699238+0000", 2024-05-15T12:58:04.481 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0087611070000000006 2024-05-15T12:58:04.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.482 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:58:04.482 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.699290+0000", 2024-05-15T12:58:04.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1999999999999997e-05 2024-05-15T12:58:04.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.482 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:58:04.482 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:42.699338+0000", 2024-05-15T12:58:04.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8186999999999998e-05 2024-05-15T12:58:04.482 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.482 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:58:04.483 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.483 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:58488 2.1 2:8c133fc7:::benchmark_data_smithi177_78104_object58487:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-15T12:58:04.483 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:48.359181+0000", 2024-05-15T12:58:04.483 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 554.96939168799997, 2024-05-15T12:58:04.483 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.105747596, 2024-05-15T12:58:04.483 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:58:04.483 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:04.483 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:58:04.483 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:58:04.483 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:58:04.483 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 58488 2024-05-15T12:58:04.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.484 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:58:04.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.484 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:58:04.484 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359181+0000", 2024-05-15T12:58:04.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.484 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:58:04.484 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359181+0000", 2024-05-15T12:58:04.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:58:04.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359182+0000", 2024-05-15T12:58:04.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1659999999999999e-06 2024-05-15T12:58:04.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:58:04.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359212+0000", 2024-05-15T12:58:04.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9657000000000001e-05 2024-05-15T12:58:04.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:58:04.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359220+0000", 2024-05-15T12:58:04.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2810000000000008e-06 2024-05-15T12:58:04.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:58:04.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359226+0000", 2024-05-15T12:58:04.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2169999999999999e-06 2024-05-15T12:58:04.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:58:04.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359282+0000", 2024-05-15T12:58:04.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5878000000000003e-05 2024-05-15T12:58:04.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:58:04.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359431+0000", 2024-05-15T12:58:04.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014895899999999999 2024-05-15T12:58:04.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,3", 2024-05-15T12:58:04.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359608+0000", 2024-05-15T12:58:04.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017731299999999999 2024-05-15T12:58:04.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.363541+0000", 2024-05-15T12:58:04.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039324939999999999 2024-05-15T12:58:04.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:58:04.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.363625+0000", 2024-05-15T12:58:04.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3719000000000003e-05 2024-05-15T12:58:04.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464802+0000", 2024-05-15T12:58:04.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.101177645 2024-05-15T12:58:04.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:58:04.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464874+0000", 2024-05-15T12:58:04.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2324999999999996e-05 2024-05-15T12:58:04.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:58:04.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464928+0000", 2024-05-15T12:58:04.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3942e-05 2024-05-15T12:58:04.489 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.489 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:58:04.490 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.490 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:58490 2.1 2:80921832:::benchmark_data_smithi177_78104_object58489:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-15T12:58:04.490 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:48.359238+0000", 2024-05-15T12:58:04.490 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 554.96933500800003, 2024-05-15T12:58:04.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10584242100000001, 2024-05-15T12:58:04.490 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:58:04.490 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:04.490 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:58:04.490 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:58:04.490 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:58:04.490 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 58490 2024-05-15T12:58:04.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.491 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:58:04.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.491 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:58:04.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359238+0000", 2024-05-15T12:58:04.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.491 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:58:04.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359238+0000", 2024-05-15T12:58:04.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:58:04.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359238+0000", 2024-05-15T12:58:04.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3500000000000002e-07 2024-05-15T12:58:04.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:58:04.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359388+0000", 2024-05-15T12:58:04.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015008 2024-05-15T12:58:04.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:58:04.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359398+0000", 2024-05-15T12:58:04.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7569999999999999e-06 2024-05-15T12:58:04.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.493 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:58:04.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359405+0000", 2024-05-15T12:58:04.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7970000000000003e-06 2024-05-15T12:58:04.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.493 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:58:04.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361472+0000", 2024-05-15T12:58:04.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020671880000000002 2024-05-15T12:58:04.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.493 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:58:04.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361583+0000", 2024-05-15T12:58:04.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000110745 2024-05-15T12:58:04.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,3", 2024-05-15T12:58:04.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361714+0000", 2024-05-15T12:58:04.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013185399999999999 2024-05-15T12:58:04.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.363611+0000", 2024-05-15T12:58:04.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0018970689999999999 2024-05-15T12:58:04.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:58:04.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367554+0000", 2024-05-15T12:58:04.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039426879999999997 2024-05-15T12:58:04.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464987+0000", 2024-05-15T12:58:04.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.097432747 2024-05-15T12:58:04.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:58:04.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465043+0000", 2024-05-15T12:58:04.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.588e-05 2024-05-15T12:58:04.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:58:04.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465080+0000", 2024-05-15T12:58:04.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7181000000000002e-05 2024-05-15T12:58:04.496 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.497 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:58:04.497 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.497 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:58493 2.1 2:81994a98:::benchmark_data_smithi177_78104_object58492:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-15T12:58:04.497 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:48.359509+0000", 2024-05-15T12:58:04.497 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 554.96906361000003, 2024-05-15T12:58:04.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.105753467, 2024-05-15T12:58:04.497 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:58:04.497 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:04.497 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:58:04.497 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:58:04.497 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:58:04.498 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 58493 2024-05-15T12:58:04.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.498 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:58:04.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:58:04.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359509+0000", 2024-05-15T12:58:04.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:58:04.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359509+0000", 2024-05-15T12:58:04.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:58:04.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359510+0000", 2024-05-15T12:58:04.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8599999999999997e-07 2024-05-15T12:58:04.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:58:04.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359561+0000", 2024-05-15T12:58:04.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0909999999999999e-05 2024-05-15T12:58:04.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:58:04.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359570+0000", 2024-05-15T12:58:04.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9579999999999996e-06 2024-05-15T12:58:04.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:58:04.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.359576+0000", 2024-05-15T12:58:04.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6320000000000002e-06 2024-05-15T12:58:04.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:58:04.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.363643+0000", 2024-05-15T12:58:04.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0040667409999999996 2024-05-15T12:58:04.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:58:04.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.363730+0000", 2024-05-15T12:58:04.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6646999999999999e-05 2024-05-15T12:58:04.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,3", 2024-05-15T12:58:04.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.363815+0000", 2024-05-15T12:58:04.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5136000000000006e-05 2024-05-15T12:58:04.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:58:04.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464946+0000", 2024-05-15T12:58:04.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.101130962 2024-05-15T12:58:04.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465138+0000", 2024-05-15T12:58:04.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00019219199999999999 2024-05-15T12:58:04.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465177+0000", 2024-05-15T12:58:04.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9573999999999998e-05 2024-05-15T12:58:04.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:58:04.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465225+0000", 2024-05-15T12:58:04.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7426999999999998e-05 2024-05-15T12:58:04.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:58:04.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465262+0000", 2024-05-15T12:58:04.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7502000000000001e-05 2024-05-15T12:58:04.504 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.505 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:58:04.505 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.505 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:58498 2.1 2:8b03664e:::benchmark_data_smithi177_78104_object58497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-15T12:58:04.505 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:48.361834+0000", 2024-05-15T12:58:04.505 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 554.96673854200003, 2024-05-15T12:58:04.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10363879400000001, 2024-05-15T12:58:04.505 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:58:04.505 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:04.505 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:58:04.505 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:58:04.505 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:58:04.505 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 58498 2024-05-15T12:58:04.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.505 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:58:04.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:58:04.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361834+0000", 2024-05-15T12:58:04.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:58:04.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361834+0000", 2024-05-15T12:58:04.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:58:04.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361835+0000", 2024-05-15T12:58:04.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.155e-06 2024-05-15T12:58:04.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:58:04.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361876+0000", 2024-05-15T12:58:04.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1337000000000001e-05 2024-05-15T12:58:04.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:58:04.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361887+0000", 2024-05-15T12:58:04.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0589e-05 2024-05-15T12:58:04.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:58:04.508 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.361894+0000", 2024-05-15T12:58:04.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.289e-06 2024-05-15T12:58:04.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.508 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:58:04.508 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365486+0000", 2024-05-15T12:58:04.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035911979999999999 2024-05-15T12:58:04.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.508 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:58:04.508 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365624+0000", 2024-05-15T12:58:04.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000138281 2024-05-15T12:58:04.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,3", 2024-05-15T12:58:04.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365802+0000", 2024-05-15T12:58:04.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017837500000000001 2024-05-15T12:58:04.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:58:04.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465090+0000", 2024-05-15T12:58:04.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.099288247999999996 2024-05-15T12:58:04.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465303+0000", 2024-05-15T12:58:04.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00021270499999999999 2024-05-15T12:58:04.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465390+0000", 2024-05-15T12:58:04.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6355999999999998e-05 2024-05-15T12:58:04.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:58:04.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465437+0000", 2024-05-15T12:58:04.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7642000000000002e-05 2024-05-15T12:58:04.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:58:04.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465473+0000", 2024-05-15T12:58:04.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5618999999999999e-05 2024-05-15T12:58:04.511 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.511 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:58:04.511 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.511 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.24562.0:58488 2.1 e129/110 ondisk, result = 0)", 2024-05-15T12:58:04.511 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:48.365352+0000", 2024-05-15T12:58:04.511 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 554.96322071199995, 2024-05-15T12:58:04.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.099595227999999994, 2024-05-15T12:58:04.511 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:58:04.511 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T12:58:04.512 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:58:04.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:58:04.512 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365352+0000", 2024-05-15T12:58:04.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:58:04.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365352+0000", 2024-05-15T12:58:04.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:58:04.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365352+0000", 2024-05-15T12:58:04.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5699999999999998e-07 2024-05-15T12:58:04.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:58:04.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365360+0000", 2024-05-15T12:58:04.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6690000000000006e-06 2024-05-15T12:58:04.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.514 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:58:04.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365367+0000", 2024-05-15T12:58:04.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4939999999999997e-06 2024-05-15T12:58:04.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.514 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:58:04.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365378+0000", 2024-05-15T12:58:04.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.066e-05 2024-05-15T12:58:04.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.514 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:58:04.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464785+0000", 2024-05-15T12:58:04.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.099406904000000004 2024-05-15T12:58:04.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:58:04.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464795+0000", 2024-05-15T12:58:04.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0076e-05 2024-05-15T12:58:04.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:58:04.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464947+0000", 2024-05-15T12:58:04.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015206799999999999 2024-05-15T12:58:04.515 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.516 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:58:04.516 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.516 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.24562.0:58501 2.4 e129/128 2:2c05dc93:::benchmark_data_smithi177_78104_object58500:head v 129'3650, mlcod=129'3648)", 2024-05-15T12:58:04.516 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:48.365547+0000", 2024-05-15T12:58:04.516 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 554.96302575599998, 2024-05-15T12:58:04.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10243205599999999, 2024-05-15T12:58:04.516 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:58:04.516 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:04.516 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:58:04.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.517 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:58:04.517 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365547+0000", 2024-05-15T12:58:04.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.517 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:58:04.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365547+0000", 2024-05-15T12:58:04.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:58:04.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365547+0000", 2024-05-15T12:58:04.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0800000000000004e-07 2024-05-15T12:58:04.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:58:04.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365571+0000", 2024-05-15T12:58:04.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3992e-05 2024-05-15T12:58:04.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:58:04.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365582+0000", 2024-05-15T12:58:04.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0929e-05 2024-05-15T12:58:04.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:58:04.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365589+0000", 2024-05-15T12:58:04.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.545e-06 2024-05-15T12:58:04.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:58:04.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365641+0000", 2024-05-15T12:58:04.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1969000000000003e-05 2024-05-15T12:58:04.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:58:04.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365664+0000", 2024-05-15T12:58:04.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3442999999999999e-05 2024-05-15T12:58:04.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:58:04.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.467932+0000", 2024-05-15T12:58:04.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.102268092 2024-05-15T12:58:04.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:58:04.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.467979+0000", 2024-05-15T12:58:04.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6378000000000002e-05 2024-05-15T12:58:04.522 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.522 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:58:04.522 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.522 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.522 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:58506 2.e 2:73b262f3:::benchmark_data_smithi177_78104_object58505:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-15T12:58:04.522 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:48.365633+0000", 2024-05-15T12:58:04.522 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 554.962939312, 2024-05-15T12:58:04.522 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.106977115, 2024-05-15T12:58:04.522 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:58:04.523 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:04.523 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:58:04.523 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:58:04.523 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:58:04.523 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 58506 2024-05-15T12:58:04.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.523 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:58:04.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:58:04.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365633+0000", 2024-05-15T12:58:04.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:58:04.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365633+0000", 2024-05-15T12:58:04.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:58:04.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.365634+0000", 2024-05-15T12:58:04.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1289999999999999e-06 2024-05-15T12:58:04.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:58:04.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367598+0000", 2024-05-15T12:58:04.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019633670000000002 2024-05-15T12:58:04.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:58:04.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367609+0000", 2024-05-15T12:58:04.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1491000000000001e-05 2024-05-15T12:58:04.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:58:04.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367617+0000", 2024-05-15T12:58:04.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9270000000000005e-06 2024-05-15T12:58:04.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:58:04.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367669+0000", 2024-05-15T12:58:04.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2287000000000003e-05 2024-05-15T12:58:04.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:58:04.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367777+0000", 2024-05-15T12:58:04.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000107895 2024-05-15T12:58:04.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:58:04.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.464792+0000", 2024-05-15T12:58:04.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.097014787000000005 2024-05-15T12:58:04.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.527 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:58:04.527 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.469572+0000", 2024-05-15T12:58:04.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0047797839999999996 2024-05-15T12:58:04.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.527 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.527 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.471658+0000", 2024-05-15T12:58:04.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020863610000000001 2024-05-15T12:58:04.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.527 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.527 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.472506+0000", 2024-05-15T12:58:04.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00084819400000000003 2024-05-15T12:58:04.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:58:04.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.472563+0000", 2024-05-15T12:58:04.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6164e-05 2024-05-15T12:58:04.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:58:04.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.472610+0000", 2024-05-15T12:58:04.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7729000000000003e-05 2024-05-15T12:58:04.528 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.528 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:58:04.528 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.529 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:58508 2.1 2:8ad9250e:::benchmark_data_smithi177_78104_object58507:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-05-15T12:58:04.529 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:48:48.367638+0000", 2024-05-15T12:58:04.529 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 554.96093412200003, 2024-05-15T12:58:04.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.107729793, 2024-05-15T12:58:04.529 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:58:04.529 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:04.529 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:58:04.529 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:58:04.529 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:58:04.529 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 58508 2024-05-15T12:58:04.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.529 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:58:04.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:58:04.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367638+0000", 2024-05-15T12:58:04.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:58:04.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367638+0000", 2024-05-15T12:58:04.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:58:04.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367640+0000", 2024-05-15T12:58:04.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4389999999999999e-06 2024-05-15T12:58:04.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:58:04.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367672+0000", 2024-05-15T12:58:04.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2190999999999997e-05 2024-05-15T12:58:04.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:58:04.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367681+0000", 2024-05-15T12:58:04.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2110000000000004e-06 2024-05-15T12:58:04.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:58:04.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.367688+0000", 2024-05-15T12:58:04.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8800000000000002e-06 2024-05-15T12:58:04.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:58:04.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465627+0000", 2024-05-15T12:58:04.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.097938774000000006 2024-05-15T12:58:04.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:58:04.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465788+0000", 2024-05-15T12:58:04.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000160631 2024-05-15T12:58:04.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,3", 2024-05-15T12:58:04.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.465946+0000", 2024-05-15T12:58:04.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000158918 2024-05-15T12:58:04.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:58:04.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.469751+0000", 2024-05-15T12:58:04.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038043389999999999 2024-05-15T12:58:04.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.473812+0000", 2024-05-15T12:58:04.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0040614559999999997 2024-05-15T12:58:04.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.475272+0000", 2024-05-15T12:58:04.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0014601359999999999 2024-05-15T12:58:04.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:58:04.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.475329+0000", 2024-05-15T12:58:04.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.622e-05 2024-05-15T12:58:04.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:58:04.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:48:48.475368+0000", 2024-05-15T12:58:04.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9598000000000003e-05 2024-05-15T12:58:04.535 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.535 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:58:04.535 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.535 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89600 2.e 2:763b1fd1:::benchmark_data_smithi177_78104_object89599:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:58:04.536 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.804057+0000", 2024-05-15T12:58:04.536 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 527.52451563399995, 2024-05-15T12:58:04.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.21480517800000001, 2024-05-15T12:58:04.536 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:58:04.536 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:04.536 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:58:04.536 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:58:04.536 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:58:04.536 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89600 2024-05-15T12:58:04.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.536 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:58:04.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.536 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:58:04.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804057+0000", 2024-05-15T12:58:04.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:58:04.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804057+0000", 2024-05-15T12:58:04.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:58:04.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804058+0000", 2024-05-15T12:58:04.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0690000000000001e-06 2024-05-15T12:58:04.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:58:04.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804179+0000", 2024-05-15T12:58:04.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120938 2024-05-15T12:58:04.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:58:04.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804191+0000", 2024-05-15T12:58:04.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1868000000000001e-05 2024-05-15T12:58:04.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:58:04.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804197+0000", 2024-05-15T12:58:04.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2550000000000003e-06 2024-05-15T12:58:04.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:58:04.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804252+0000", 2024-05-15T12:58:04.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4953000000000001e-05 2024-05-15T12:58:04.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:58:04.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804392+0000", 2024-05-15T12:58:04.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000139772 2024-05-15T12:58:04.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:58:04.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.804513+0000", 2024-05-15T12:58:04.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120763 2024-05-15T12:58:04.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:58:04.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.808055+0000", 2024-05-15T12:58:04.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035428830000000001 2024-05-15T12:58:04.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.819733+0000", 2024-05-15T12:58:04.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011677682999999999 2024-05-15T12:58:04.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.018742+0000", 2024-05-15T12:58:04.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19900900199999999 2024-05-15T12:58:04.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:58:04.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.018812+0000", 2024-05-15T12:58:04.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9924000000000003e-05 2024-05-15T12:58:04.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:58:04.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.018862+0000", 2024-05-15T12:58:04.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0068000000000002e-05 2024-05-15T12:58:04.542 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.542 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:58:04.542 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.542 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89634 2.e 2:7b9cebfb:::benchmark_data_smithi177_78104_object89633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:58:04.542 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.841129+0000", 2024-05-15T12:58:04.542 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 527.48744339799998, 2024-05-15T12:58:04.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.18161068899999999, 2024-05-15T12:58:04.542 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:58:04.542 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:04.542 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:58:04.543 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:58:04.543 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:58:04.543 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89634 2024-05-15T12:58:04.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.543 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:58:04.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:58:04.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841129+0000", 2024-05-15T12:58:04.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:58:04.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841129+0000", 2024-05-15T12:58:04.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:58:04.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841131+0000", 2024-05-15T12:58:04.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.699e-06 2024-05-15T12:58:04.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:58:04.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841163+0000", 2024-05-15T12:58:04.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2499999999999997e-05 2024-05-15T12:58:04.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:58:04.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841181+0000", 2024-05-15T12:58:04.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7436e-05 2024-05-15T12:58:04.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:58:04.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841190+0000", 2024-05-15T12:58:04.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.02e-06 2024-05-15T12:58:04.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:58:04.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841264+0000", 2024-05-15T12:58:04.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3866000000000001e-05 2024-05-15T12:58:04.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:58:04.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841398+0000", 2024-05-15T12:58:04.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013442900000000001 2024-05-15T12:58:04.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:58:04.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.841544+0000", 2024-05-15T12:58:04.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014611800000000001 2024-05-15T12:58:04.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:58:04.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.844757+0000", 2024-05-15T12:58:04.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032125180000000001 2024-05-15T12:58:04.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.850454+0000", 2024-05-15T12:58:04.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.005697667 2024-05-15T12:58:04.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.022628+0000", 2024-05-15T12:58:04.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.17217374299999999 2024-05-15T12:58:04.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:58:04.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.022690+0000", 2024-05-15T12:58:04.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1566e-05 2024-05-15T12:58:04.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:58:04.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.022740+0000", 2024-05-15T12:58:04.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0127000000000003e-05 2024-05-15T12:58:04.549 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.549 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:58:04.549 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.549 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89645 2.e 2:7ffe4884:::benchmark_data_smithi177_78104_object89644:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:58:04.549 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.858184+0000", 2024-05-15T12:58:04.549 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 527.470388136, 2024-05-15T12:58:04.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.164674923, 2024-05-15T12:58:04.549 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:58:04.549 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:04.549 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:58:04.549 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:58:04.549 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:58:04.549 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89645 2024-05-15T12:58:04.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.550 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:58:04.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:58:04.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858184+0000", 2024-05-15T12:58:04.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:58:04.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858184+0000", 2024-05-15T12:58:04.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:58:04.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858186+0000", 2024-05-15T12:58:04.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5850000000000001e-06 2024-05-15T12:58:04.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:58:04.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858245+0000", 2024-05-15T12:58:04.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8729e-05 2024-05-15T12:58:04.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:58:04.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858258+0000", 2024-05-15T12:58:04.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3767999999999999e-05 2024-05-15T12:58:04.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:58:04.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858265+0000", 2024-05-15T12:58:04.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.725e-06 2024-05-15T12:58:04.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:58:04.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858324+0000", 2024-05-15T12:58:04.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8910999999999999e-05 2024-05-15T12:58:04.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:58:04.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858435+0000", 2024-05-15T12:58:04.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000111068 2024-05-15T12:58:04.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:58:04.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.858592+0000", 2024-05-15T12:58:04.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000157083 2024-05-15T12:58:04.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:58:04.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.871361+0000", 2024-05-15T12:58:04.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012768662 2024-05-15T12:58:04.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.872157+0000", 2024-05-15T12:58:04.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00079593199999999995 2024-05-15T12:58:04.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.022780+0000", 2024-05-15T12:58:04.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.150623014 2024-05-15T12:58:04.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:58:04.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.022825+0000", 2024-05-15T12:58:04.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5626e-05 2024-05-15T12:58:04.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:58:04.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.022859+0000", 2024-05-15T12:58:04.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3819999999999998e-05 2024-05-15T12:58:04.555 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.555 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:58:04.555 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.555 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89654 2.e 2:7e1530a2:::benchmark_data_smithi177_78104_object89653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:58:04.555 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.874001+0000", 2024-05-15T12:58:04.556 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 527.45457188, 2024-05-15T12:58:04.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.15396952, 2024-05-15T12:58:04.556 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:58:04.556 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:04.556 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:58:04.556 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:58:04.556 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:58:04.556 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89654 2024-05-15T12:58:04.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.556 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:58:04.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:58:04.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874001+0000", 2024-05-15T12:58:04.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:58:04.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874001+0000", 2024-05-15T12:58:04.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:58:04.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874002+0000", 2024-05-15T12:58:04.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6199999999999999e-06 2024-05-15T12:58:04.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:58:04.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874045+0000", 2024-05-15T12:58:04.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3191e-05 2024-05-15T12:58:04.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:58:04.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874061+0000", 2024-05-15T12:58:04.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6002999999999999e-05 2024-05-15T12:58:04.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:58:04.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874072+0000", 2024-05-15T12:58:04.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0647e-05 2024-05-15T12:58:04.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:58:04.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874137+0000", 2024-05-15T12:58:04.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4941e-05 2024-05-15T12:58:04.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:58:04.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874222+0000", 2024-05-15T12:58:04.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4581999999999998e-05 2024-05-15T12:58:04.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:58:04.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.874308+0000", 2024-05-15T12:58:04.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6658999999999998e-05 2024-05-15T12:58:04.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:58:04.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.876658+0000", 2024-05-15T12:58:04.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0023498080000000001 2024-05-15T12:58:04.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.893634+0000", 2024-05-15T12:58:04.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016976104999999998 2024-05-15T12:58:04.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.027858+0000", 2024-05-15T12:58:04.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.134223696 2024-05-15T12:58:04.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:58:04.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.027919+0000", 2024-05-15T12:58:04.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1464000000000006e-05 2024-05-15T12:58:04.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:58:04.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.027970+0000", 2024-05-15T12:58:04.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0803999999999997e-05 2024-05-15T12:58:04.561 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.561 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:58:04.561 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.561 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89661 2.e 2:7462a28b:::benchmark_data_smithi177_78104_object89660:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:58:04.562 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.887128+0000", 2024-05-15T12:58:04.562 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 527.44144465199997, 2024-05-15T12:58:04.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.14095980499999999, 2024-05-15T12:58:04.562 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:58:04.562 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:04.562 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:58:04.562 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:58:04.562 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:58:04.562 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89661 2024-05-15T12:58:04.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.563 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:58:04.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:58:04.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887128+0000", 2024-05-15T12:58:04.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:58:04.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887128+0000", 2024-05-15T12:58:04.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:58:04.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887130+0000", 2024-05-15T12:58:04.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7969999999999999e-06 2024-05-15T12:58:04.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:58:04.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887162+0000", 2024-05-15T12:58:04.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2780000000000001e-05 2024-05-15T12:58:04.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:58:04.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887180+0000", 2024-05-15T12:58:04.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7187999999999999e-05 2024-05-15T12:58:04.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:58:04.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887190+0000", 2024-05-15T12:58:04.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8980000000000008e-06 2024-05-15T12:58:04.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:58:04.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887253+0000", 2024-05-15T12:58:04.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3724000000000002e-05 2024-05-15T12:58:04.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:58:04.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887391+0000", 2024-05-15T12:58:04.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013803700000000001 2024-05-15T12:58:04.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:58:04.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.887549+0000", 2024-05-15T12:58:04.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015766599999999999 2024-05-15T12:58:04.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:58:04.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.891069+0000", 2024-05-15T12:58:04.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035196989999999998 2024-05-15T12:58:04.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.570 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.894586+0000", 2024-05-15T12:58:04.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035178269999999998 2024-05-15T12:58:04.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.028009+0000", 2024-05-15T12:58:04.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.13342280400000001 2024-05-15T12:58:04.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:58:04.572 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.028054+0000", 2024-05-15T12:58:04.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5207000000000003e-05 2024-05-15T12:58:04.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.573 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:58:04.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.028088+0000", 2024-05-15T12:58:04.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3178e-05 2024-05-15T12:58:04.573 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.573 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:58:04.573 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.574 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89672 2.e 2:7301c129:::benchmark_data_smithi177_78104_object89671:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:58:04.574 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.899549+0000", 2024-05-15T12:58:04.574 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 527.42902354399996, 2024-05-15T12:58:04.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.13251970099999999, 2024-05-15T12:58:04.574 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:58:04.574 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:04.574 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:58:04.574 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:58:04.575 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:58:04.575 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89672 2024-05-15T12:58:04.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.575 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:58:04.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:58:04.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.899549+0000", 2024-05-15T12:58:04.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:58:04.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.899549+0000", 2024-05-15T12:58:04.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:58:04.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.899551+0000", 2024-05-15T12:58:04.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.826e-06 2024-05-15T12:58:04.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:58:04.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.899591+0000", 2024-05-15T12:58:04.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0422e-05 2024-05-15T12:58:04.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:58:04.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.899606+0000", 2024-05-15T12:58:04.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5233e-05 2024-05-15T12:58:04.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:58:04.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.899617+0000", 2024-05-15T12:58:04.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0186999999999999e-05 2024-05-15T12:58:04.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:58:04.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.899682+0000", 2024-05-15T12:58:04.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4869000000000005e-05 2024-05-15T12:58:04.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:58:04.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.899838+0000", 2024-05-15T12:58:04.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015638300000000001 2024-05-15T12:58:04.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:58:04.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.900004+0000", 2024-05-15T12:58:04.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000165835 2024-05-15T12:58:04.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.911032+0000", 2024-05-15T12:58:04.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011028692999999999 2024-05-15T12:58:04.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:58:04.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.917176+0000", 2024-05-15T12:58:04.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.006143333 2024-05-15T12:58:04.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.031955+0000", 2024-05-15T12:58:04.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.114779144 2024-05-15T12:58:04.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:58:04.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032020+0000", 2024-05-15T12:58:04.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4707000000000002e-05 2024-05-15T12:58:04.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:58:04.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032069+0000", 2024-05-15T12:58:04.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9069000000000001e-05 2024-05-15T12:58:04.583 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.584 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:58:04.584 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.584 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89679 2.e 2:74cc89e8:::benchmark_data_smithi177_78104_object89678:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:58:04.584 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.910531+0000", 2024-05-15T12:58:04.584 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 527.41804105200004, 2024-05-15T12:58:04.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.121660637, 2024-05-15T12:58:04.584 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:58:04.584 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:04.584 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:58:04.584 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:58:04.584 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:58:04.584 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89679 2024-05-15T12:58:04.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.585 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:58:04.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:58:04.585 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910531+0000", 2024-05-15T12:58:04.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:58:04.585 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910531+0000", 2024-05-15T12:58:04.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:58:04.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910533+0000", 2024-05-15T12:58:04.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.773e-06 2024-05-15T12:58:04.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:58:04.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910580+0000", 2024-05-15T12:58:04.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6854000000000002e-05 2024-05-15T12:58:04.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:58:04.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910595+0000", 2024-05-15T12:58:04.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5082999999999999e-05 2024-05-15T12:58:04.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:58:04.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910605+0000", 2024-05-15T12:58:04.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6029999999999994e-06 2024-05-15T12:58:04.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:58:04.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.911065+0000", 2024-05-15T12:58:04.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00046001800000000003 2024-05-15T12:58:04.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:58:04.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.911211+0000", 2024-05-15T12:58:04.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014574400000000001 2024-05-15T12:58:04.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:58:04.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.911363+0000", 2024-05-15T12:58:04.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015289500000000001 2024-05-15T12:58:04.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:58:04.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.924113+0000", 2024-05-15T12:58:04.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012749286 2024-05-15T12:58:04.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.931398+0000", 2024-05-15T12:58:04.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0072853279999999998 2024-05-15T12:58:04.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032107+0000", 2024-05-15T12:58:04.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.100708456 2024-05-15T12:58:04.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:58:04.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032159+0000", 2024-05-15T12:58:04.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2234000000000002e-05 2024-05-15T12:58:04.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:58:04.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032192+0000", 2024-05-15T12:58:04.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3362999999999999e-05 2024-05-15T12:58:04.590 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.590 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:58:04.590 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.590 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89680 2.e 2:71e6b157:::benchmark_data_smithi177_78104_object89679:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:58:04.590 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.910622+0000", 2024-05-15T12:58:04.590 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 527.41795095600003, 2024-05-15T12:58:04.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12167915, 2024-05-15T12:58:04.590 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:58:04.590 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:04.590 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:58:04.590 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:58:04.591 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:58:04.591 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89680 2024-05-15T12:58:04.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.591 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:58:04.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:58:04.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910622+0000", 2024-05-15T12:58:04.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:58:04.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910622+0000", 2024-05-15T12:58:04.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:58:04.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910622+0000", 2024-05-15T12:58:04.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6499999999999999e-07 2024-05-15T12:58:04.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:58:04.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910662+0000", 2024-05-15T12:58:04.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9436e-05 2024-05-15T12:58:04.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:58:04.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910670+0000", 2024-05-15T12:58:04.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4940000000000003e-06 2024-05-15T12:58:04.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:58:04.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.910677+0000", 2024-05-15T12:58:04.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6039999999999998e-06 2024-05-15T12:58:04.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:58:04.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.914062+0000", 2024-05-15T12:58:04.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003385117 2024-05-15T12:58:04.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:58:04.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.914206+0000", 2024-05-15T12:58:04.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014453299999999999 2024-05-15T12:58:04.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:58:04.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.914360+0000", 2024-05-15T12:58:04.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000154146 2024-05-15T12:58:04.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:58:04.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.924120+0000", 2024-05-15T12:58:04.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0097599209999999995 2024-05-15T12:58:04.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.934155+0000", 2024-05-15T12:58:04.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0100348 2024-05-15T12:58:04.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032226+0000", 2024-05-15T12:58:04.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.098070750999999998 2024-05-15T12:58:04.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:58:04.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032269+0000", 2024-05-15T12:58:04.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3513000000000002e-05 2024-05-15T12:58:04.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:58:04.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032301+0000", 2024-05-15T12:58:04.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1269999999999997e-05 2024-05-15T12:58:04.597 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.597 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:58:04.597 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.597 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89690 2.e 2:7ba0758f:::benchmark_data_smithi177_78104_object89689:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:58:04.597 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.928382+0000", 2024-05-15T12:58:04.597 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 527.40019093499996, 2024-05-15T12:58:04.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10402787500000001, 2024-05-15T12:58:04.597 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:58:04.597 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:04.597 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:58:04.598 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:58:04.598 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:58:04.598 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89690 2024-05-15T12:58:04.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.598 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:58:04.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.598 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:58:04.598 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.928382+0000", 2024-05-15T12:58:04.598 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:58:04.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.928382+0000", 2024-05-15T12:58:04.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:58:04.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.928383+0000", 2024-05-15T12:58:04.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8610000000000001e-06 2024-05-15T12:58:04.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:58:04.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.928422+0000", 2024-05-15T12:58:04.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8152000000000003e-05 2024-05-15T12:58:04.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:58:04.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.928444+0000", 2024-05-15T12:58:04.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2830999999999999e-05 2024-05-15T12:58:04.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:58:04.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.928454+0000", 2024-05-15T12:58:04.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8740000000000007e-06 2024-05-15T12:58:04.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:58:04.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.931422+0000", 2024-05-15T12:58:04.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029681500000000001 2024-05-15T12:58:04.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:58:04.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.931545+0000", 2024-05-15T12:58:04.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000122574 2024-05-15T12:58:04.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:58:04.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.931686+0000", 2024-05-15T12:58:04.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014140100000000001 2024-05-15T12:58:04.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:58:04.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935371+0000", 2024-05-15T12:58:04.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0036848359999999999 2024-05-15T12:58:04.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.938923+0000", 2024-05-15T12:58:04.602 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035515859999999998 2024-05-15T12:58:04.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.602 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032334+0000", 2024-05-15T12:58:04.602 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.093410842999999993 2024-05-15T12:58:04.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.603 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.603 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:58:04.603 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032378+0000", 2024-05-15T12:58:04.603 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.384e-05 2024-05-15T12:58:04.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.603 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.603 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:58:04.603 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.032409+0000", 2024-05-15T12:58:04.604 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1927000000000001e-05 2024-05-15T12:58:04.604 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.604 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:58:04.604 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.604 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.604 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89694 2.e 2:7ceb574e:::benchmark_data_smithi177_78104_object89693:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:58:04.604 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.935396+0000", 2024-05-15T12:58:04.604 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 527.39317679400006, 2024-05-15T12:58:04.604 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10441977, 2024-05-15T12:58:04.604 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:58:04.604 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:04.604 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:58:04.604 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:58:04.605 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:58:04.605 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89694 2024-05-15T12:58:04.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.605 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:58:04.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.605 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:58:04.605 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935396+0000", 2024-05-15T12:58:04.605 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.605 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:58:04.605 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935396+0000", 2024-05-15T12:58:04.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.606 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:58:04.606 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935397+0000", 2024-05-15T12:58:04.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.353e-06 2024-05-15T12:58:04.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.606 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:58:04.607 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935423+0000", 2024-05-15T12:58:04.607 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6291e-05 2024-05-15T12:58:04.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.607 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.607 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:58:04.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935436+0000", 2024-05-15T12:58:04.608 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2184000000000001e-05 2024-05-15T12:58:04.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:58:04.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935445+0000", 2024-05-15T12:58:04.609 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4020000000000008e-06 2024-05-15T12:58:04.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.609 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.609 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:58:04.609 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935511+0000", 2024-05-15T12:58:04.610 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5832000000000003e-05 2024-05-15T12:58:04.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.610 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.610 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:58:04.610 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935684+0000", 2024-05-15T12:58:04.611 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017362399999999999 2024-05-15T12:58:04.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.611 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.611 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:58:04.611 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.935863+0000", 2024-05-15T12:58:04.611 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017888299999999999 2024-05-15T12:58:04.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.612 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.612 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:58:04.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.942570+0000", 2024-05-15T12:58:04.612 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0067064869999999997 2024-05-15T12:58:04.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.613 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.613 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.613 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.943101+0000", 2024-05-15T12:58:04.613 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00053091900000000003 2024-05-15T12:58:04.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.614 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.614 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.614 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.039698+0000", 2024-05-15T12:58:04.614 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.096597075000000004 2024-05-15T12:58:04.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.615 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:58:04.615 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.039761+0000", 2024-05-15T12:58:04.615 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2772000000000001e-05 2024-05-15T12:58:04.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.616 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:58:04.616 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.039816+0000", 2024-05-15T12:58:04.616 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4948000000000003e-05 2024-05-15T12:58:04.616 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.616 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:58:04.617 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.617 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24562.0:89699 2.e 2:7fa3b3a9:::benchmark_data_smithi177_78104_object89698:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:58:04.617 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:49:15.939371+0000", 2024-05-15T12:58:04.618 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 527.38920104099998, 2024-05-15T12:58:04.618 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10057566799999999, 2024-05-15T12:58:04.618 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T12:58:04.618 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:04.618 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T12:58:04.618 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24562", 2024-05-15T12:58:04.619 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:58:04.619 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 89699 2024-05-15T12:58:04.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.619 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T12:58:04.619 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.620 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T12:58:04.620 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939371+0000", 2024-05-15T12:58:04.620 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.620 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.620 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.620 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T12:58:04.621 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939371+0000", 2024-05-15T12:58:04.621 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T12:58:04.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.621 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.621 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T12:58:04.622 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939373+0000", 2024-05-15T12:58:04.622 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6700000000000001e-06 2024-05-15T12:58:04.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.622 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.622 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T12:58:04.622 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939415+0000", 2024-05-15T12:58:04.623 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1551000000000002e-05 2024-05-15T12:58:04.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.623 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.623 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T12:58:04.623 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939432+0000", 2024-05-15T12:58:04.623 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.768e-05 2024-05-15T12:58:04.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.624 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.624 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T12:58:04.624 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939441+0000", 2024-05-15T12:58:04.624 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7299999999999994e-06 2024-05-15T12:58:04.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.625 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.625 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T12:58:04.625 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939527+0000", 2024-05-15T12:58:04.625 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6098000000000001e-05 2024-05-15T12:58:04.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.626 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.626 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T12:58:04.626 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939679+0000", 2024-05-15T12:58:04.626 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015150299999999999 2024-05-15T12:58:04.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.627 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.627 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T12:58:04.627 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.939825+0000", 2024-05-15T12:58:04.627 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000146725 2024-05-15T12:58:04.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.627 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.628 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T12:58:04.628 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.942947+0000", 2024-05-15T12:58:04.628 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031219279999999999 2024-05-15T12:58:04.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.628 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.628 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.629 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:15.943208+0000", 2024-05-15T12:58:04.629 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00026073099999999999 2024-05-15T12:58:04.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.629 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.629 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:04.630 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.039860+0000", 2024-05-15T12:58:04.630 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.096651440000000005 2024-05-15T12:58:04.630 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.630 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.630 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T12:58:04.630 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.039911+0000", 2024-05-15T12:58:04.630 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.13e-05 2024-05-15T12:58:04.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:58:04.631 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:58:04.631 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T12:58:04.631 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:49:16.039947+0000", 2024-05-15T12:58:04.631 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6312000000000001e-05 2024-05-15T12:58:04.631 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.631 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:58:04.631 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.631 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:58:04.631 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:58:04.631 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T12:58:05.038 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:58:05.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:05.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:58:05.143 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:58:05.173 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:04 smithi120 ceph-mon[43961]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T12:58:05.174 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:04 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T12:58:05.176 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:05.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:04 smithi177 ceph-mon[44303]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T12:58:05.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:04 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T12:58:05.197 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:58:05.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:05.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:58:05.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:04 smithi027 ceph-mon[39795]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T12:58:05.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:04 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T12:58:05.301 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:58:05.330 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:05.349 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:58:05.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:05.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:58:05.360 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:58:05.361 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:58:05.382 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:58:05.402 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:58:05.422 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:58:05.442 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:58:05.453 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:58:05.462 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:58:05.482 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:05.512 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:58:05.706 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:58:05.726 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:58:05.746 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:58:05.767 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:58:05.768 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:58:05.768 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:58:05.768 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:58:05.768 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:58:05.768 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:58:05.768 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:58:05.768 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:58:05.768 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:58:05.769 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:58:05.769 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:58:05.769 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:58:05.769 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:58:05.897 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-05-15T12:58:06.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:05 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:58:06.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:05 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2488770800' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:58:06.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:05 smithi177 ceph-mon[44303]: pgmap v874: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:58:06.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:05 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:58:06.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:05 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:58:06.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:05 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:58:06.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:05 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:58:06.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:05 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2488770800' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:58:06.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:05 smithi027 ceph-mon[39795]: pgmap v874: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:58:06.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:05 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:58:06.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:05 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:58:06.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:05 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:58:06.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:05 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:58:06.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:05 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2488770800' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:58:06.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:05 smithi120 ceph-mon[43961]: pgmap v874: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:58:06.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:05 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:58:06.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:05 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:58:06.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:05 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T12:58:06.935 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:58:06.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:06.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:58:07.039 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:58:07.070 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:07.328 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:58:07.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:07.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:58:07.432 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:58:07.464 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:07.485 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:58:07.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:07.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:58:07.589 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:58:07.620 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:07.641 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:58:07.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:07.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:58:07.745 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:58:07.778 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:07.799 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:58:07.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:07.802 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:58:07.903 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:58:07.935 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:08.033 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:58:08.033 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:58:08.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:08.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:58:08.138 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:58:08.166 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:07 smithi120 ceph-mon[43961]: pgmap v875: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:58:08.168 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:08.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:07 smithi177 ceph-mon[44303]: pgmap v875: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:58:08.189 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:58:08.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:08.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:58:08.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:07 smithi027 ceph-mon[39795]: pgmap v875: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 892 GiB / 894 GiB avail 2024-05-15T12:58:08.293 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:58:08.325 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:08.347 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:58:08.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:08.350 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:58:08.451 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:58:08.483 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:08.504 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:58:08.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:08.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:58:08.608 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:58:08.639 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:08.895 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:58:09.068 DEBUG:teuthology.orchestra.run.smithi177:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-05-15T12:58:09.071 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-05-15T12:58:09.071 INFO:teuthology.orchestra.run:waiting for 3000 2024-05-15T12:58:09.088 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:58:09.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:09.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:58:09.126 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:hints = 1 2024-05-15T12:58:09.126 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-05-15T12:58:09.126 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Object prefix: benchmark_data_smithi177_99756 2024-05-15T12:58:09.127 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T12:58:09.127 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 0 0 0 0 0 0 - 0 2024-05-15T12:58:09.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:09 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2697970726' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T12:58:09.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:09 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:58:09.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:09 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:58:09.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:09 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:58:09.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:09 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:58:09.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:09 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:58:09.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:09 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:58:09.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:09 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:58:09.134 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:09 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2463767191' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:58:09.150 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.9 to 1 (10000) 2024-05-15T12:58:09.150 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T12:58:09.192 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:58:09.243 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:09.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:09 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2697970726' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T12:58:09.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:09 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:58:09.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:09 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:58:09.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:09 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:58:09.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:09 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:58:09.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:09 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:58:09.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:09 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:58:09.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:09 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:58:09.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:09 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2463767191' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:58:09.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:09 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2697970726' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T12:58:09.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:09 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:58:09.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:09 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:58:09.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:09 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:58:09.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:09 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:58:09.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:09 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:58:09.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:09 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:58:09.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:09 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:58:09.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:09 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2463767191' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:58:10.127 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 1 16 3592 3576 223.478 223.5 0.00135206 0.00377442 2024-05-15T12:58:10.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:10 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:58:10.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:10 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2697970726' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T12:58:10.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:10 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2463767191' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:58:10.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:10 smithi177 ceph-mon[44303]: osdmap e322: 12 total, 12 up, 10 in 2024-05-15T12:58:10.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:10 smithi177 ceph-mon[44303]: pgmap v877: 17 pgs: 17 active+clean; 28 MiB data, 2.2 GiB used, 892 GiB / 894 GiB avail; 3.4 MiB/s wr, 54 op/s 2024-05-15T12:58:10.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:10 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:58:10.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:10 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2697970726' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T12:58:10.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:10 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2463767191' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:58:10.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:10 smithi027 ceph-mon[39795]: osdmap e322: 12 total, 12 up, 10 in 2024-05-15T12:58:10.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:10 smithi027 ceph-mon[39795]: pgmap v877: 17 pgs: 17 active+clean; 28 MiB data, 2.2 GiB used, 892 GiB / 894 GiB avail; 3.4 MiB/s wr, 54 op/s 2024-05-15T12:58:10.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:10 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:58:10.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:10 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2697970726' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T12:58:10.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:10 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2463767191' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:58:10.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:10 smithi120 ceph-mon[43961]: osdmap e322: 12 total, 12 up, 10 in 2024-05-15T12:58:10.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:10 smithi120 ceph-mon[43961]: pgmap v877: 17 pgs: 17 active+clean; 28 MiB data, 2.2 GiB used, 892 GiB / 894 GiB avail; 3.4 MiB/s wr, 54 op/s 2024-05-15T12:58:11.127 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 2 16 7582 7566 236.413 249.375 0.00298174 0.00422355 2024-05-15T12:58:11.216 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 10 1 2024-05-15T12:58:12.023 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:58:12.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:12.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:58:12.127 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 3 16 11033 11017 229.498 215.688 0.00371273 0.00434879 2024-05-15T12:58:12.133 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:58:12.172 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:11 smithi027 ceph-mon[39795]: pgmap v878: 17 pgs: 17 active+clean; 187 MiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 19 MiB/s wr, 300 op/s 2024-05-15T12:58:12.176 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:12.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:11 smithi177 ceph-mon[44303]: pgmap v878: 17 pgs: 17 active+clean; 187 MiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 19 MiB/s wr, 300 op/s 2024-05-15T12:58:12.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:11 smithi120 ceph-mon[43961]: pgmap v878: 17 pgs: 17 active+clean; 187 MiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 19 MiB/s wr, 300 op/s 2024-05-15T12:58:12.309 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:58:12.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:12.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:58:12.413 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:58:12.452 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:12.479 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:58:12.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:12.482 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:58:12.583 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:58:12.618 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:12.641 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:58:12.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:12.645 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:58:12.657 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:58:12.704 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:58:12.746 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:58:12.772 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T12:58:12.772 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:58:12.782 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:12.808 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:58:12.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:12.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:58:12.912 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:58:12.960 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:13.118 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:58:13.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:13.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:58:13.127 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 4 16 14255 14239 222.463 201.375 0.00326297 0.00448795 2024-05-15T12:58:13.224 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:58:13.271 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:13.389 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:58:13.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:13.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:58:13.497 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:58:13.529 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:13.556 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:58:13.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:13.560 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:58:13.661 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:58:13.700 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:13.725 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:58:13.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:13.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:58:13.829 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:58:13.855 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:13.873 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:58:13.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:13.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:58:13.977 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:58:14.028 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:14.056 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T12:58:14.127 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 5 16 16947 16931 211.618 168.25 0.00591908 0.00471914 2024-05-15T12:58:14.320 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:14 smithi027 ceph-mon[39795]: pgmap v879: 17 pgs: 17 active+clean; 479 MiB data, 3.3 GiB used, 891 GiB / 894 GiB avail; 48 MiB/s wr, 764 op/s 2024-05-15T12:58:14.357 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-05-15T12:58:14.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:14 smithi177 ceph-mon[44303]: pgmap v879: 17 pgs: 17 active+clean; 479 MiB data, 3.3 GiB used, 891 GiB / 894 GiB avail; 48 MiB/s wr, 764 op/s 2024-05-15T12:58:14.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:14 smithi120 ceph-mon[43961]: pgmap v879: 17 pgs: 17 active+clean; 479 MiB data, 3.3 GiB used, 891 GiB / 894 GiB avail; 48 MiB/s wr, 764 op/s 2024-05-15T12:58:14.933 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:58:14.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:14.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:58:15.039 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:58:15.086 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:15.128 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 6 16 19712 19696 205.122 172.812 0.00436039 0.00487015 2024-05-15T12:58:16.128 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 7 16 22257 22241 198.541 159.062 0.00835538 0.00503268 2024-05-15T12:58:16.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:16 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:58:16.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:16 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:58:16.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:16 smithi027 ceph-mon[39795]: osdmap e323: 12 total, 12 up, 10 in 2024-05-15T12:58:16.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:16 smithi027 ceph-mon[39795]: pgmap v881: 17 pgs: 17 active+clean; 1.1 GiB data, 4.9 GiB used, 889 GiB / 894 GiB avail; 136 MiB/s wr, 2.18k op/s 2024-05-15T12:58:16.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:16 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:58:16.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:16 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:58:16.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:16 smithi177 ceph-mon[44303]: osdmap e323: 12 total, 12 up, 10 in 2024-05-15T12:58:16.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:16 smithi177 ceph-mon[44303]: pgmap v881: 17 pgs: 17 active+clean; 1.1 GiB data, 4.9 GiB used, 889 GiB / 894 GiB avail; 136 MiB/s wr, 2.18k op/s 2024-05-15T12:58:16.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:16 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:58:16.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:16 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:58:16.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:16 smithi120 ceph-mon[43961]: osdmap e323: 12 total, 12 up, 10 in 2024-05-15T12:58:16.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:16 smithi120 ceph-mon[43961]: pgmap v881: 17 pgs: 17 active+clean; 1.1 GiB data, 4.9 GiB used, 889 GiB / 894 GiB avail; 136 MiB/s wr, 2.18k op/s 2024-05-15T12:58:17.128 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 8 16 25045 25029 195.503 174.25 0.0027276 0.00511049 2024-05-15T12:58:18.128 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 9 16 26348 26332 182.829 81.4375 0.00405444 0.00546471 2024-05-15T12:58:18.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:17 smithi177 ceph-mon[44303]: pgmap v882: 17 pgs: 17 active+clean; 1.2 GiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-05-15T12:58:18.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:17 smithi027 ceph-mon[39795]: pgmap v882: 17 pgs: 17 active+clean; 1.2 GiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-05-15T12:58:18.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:17 smithi120 ceph-mon[43961]: pgmap v882: 17 pgs: 17 active+clean; 1.2 GiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-05-15T12:58:19.129 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 10 16 28789 28773 179.783 152.562 0.00485956 0.00555776 2024-05-15T12:58:20.129 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 11 16 31203 31187 177.154 150.875 0.00342584 0.00564058 2024-05-15T12:58:20.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:19 smithi177 ceph-mon[44303]: pgmap v883: 17 pgs: 17 active+clean; 1.5 GiB data, 6.6 GiB used, 887 GiB / 894 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-05-15T12:58:20.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:19 smithi027 ceph-mon[39795]: pgmap v883: 17 pgs: 17 active+clean; 1.5 GiB data, 6.6 GiB used, 887 GiB / 894 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-05-15T12:58:20.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:19 smithi120 ceph-mon[43961]: pgmap v883: 17 pgs: 17 active+clean; 1.5 GiB data, 6.6 GiB used, 887 GiB / 894 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-05-15T12:58:20.963 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:58:20.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:20.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:58:21.071 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:58:21.105 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:21.130 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 12 16 33738 33722 175.593 158.438 0.00525001 0.00569054 2024-05-15T12:58:21.130 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:58:21.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:21.135 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:58:21.236 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:58:21.293 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:21.787 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:58:21.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:21.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:58:21.887 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:21 smithi027 ceph-mon[39795]: pgmap v884: 17 pgs: 17 active+clean; 1.8 GiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 169 MiB/s wr, 2.70k op/s 2024-05-15T12:58:21.900 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:58:21.930 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:21.954 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:58:21.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:21.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:58:22.058 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:58:22.124 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:22.129 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 13 16 35931 35915 172.629 137.062 0.00847825 0.00578496 2024-05-15T12:58:22.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:21 smithi177 ceph-mon[44303]: pgmap v884: 17 pgs: 17 active+clean; 1.8 GiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 169 MiB/s wr, 2.70k op/s 2024-05-15T12:58:22.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:21 smithi120 ceph-mon[43961]: pgmap v884: 17 pgs: 17 active+clean; 1.8 GiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 169 MiB/s wr, 2.70k op/s 2024-05-15T12:58:22.486 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:58:22.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:22.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:58:22.589 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:58:22.621 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:22.646 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:58:22.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:22.650 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:58:22.751 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:58:22.797 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:22.825 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:58:22.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:22.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:58:22.928 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:58:22.958 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:22.982 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:58:22.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:22.985 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:58:23.029 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:58:23.034 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:58:23.086 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:58:23.117 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:23.129 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 14 16 37444 37428 167.053 94.5625 0.00223923 0.00598018 2024-05-15T12:58:23.139 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:58:23.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:23.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:58:23.175 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:58:23.223 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:58:23.245 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:58:23.254 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:58:23.271 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:23 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2934764054' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:58:23.273 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:23.282 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:58:23.293 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:58:23.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:23.296 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:58:23.305 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:58:23.328 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:58:23.352 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:58:23.373 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:58:23.394 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:58:23.396 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:58:23.414 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:58:23.433 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:23.447 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:58:23.489 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:58:23.501 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:58:23.502 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:58:23.502 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:58:23.502 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:58:23.502 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:58:23.502 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:58:23.502 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:58:23.502 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:58:23.502 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:58:23.502 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:58:23.502 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:58:23.502 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:58:23.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:23 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2934764054' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:58:23.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:23 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2934764054' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:58:23.713 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:58:23.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:23.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:58:23.822 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:58:23.869 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:24.130 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 15 16 39120 39104 162.9 104.75 0.00621556 0.00613391 2024-05-15T12:58:24.208 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.10 to 1 (10000) 2024-05-15T12:58:24.338 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:58:24.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:24.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:58:24.441 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:58:24.495 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:24 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/216732793' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T12:58:24.495 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:24 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T12:58:24.495 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:24 smithi027 ceph-mon[39795]: pgmap v885: 17 pgs: 17 active+clean; 2.0 GiB data, 8.3 GiB used, 886 GiB / 894 GiB avail; 161 MiB/s wr, 2.58k op/s 2024-05-15T12:58:24.497 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:24.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:24 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/216732793' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T12:58:24.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:24 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T12:58:24.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:24 smithi120 ceph-mon[43961]: pgmap v885: 17 pgs: 17 active+clean; 2.0 GiB data, 8.3 GiB used, 886 GiB / 894 GiB avail; 161 MiB/s wr, 2.58k op/s 2024-05-15T12:58:24.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:24 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/216732793' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T12:58:24.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:24 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T12:58:24.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:24 smithi177 ceph-mon[44303]: pgmap v885: 17 pgs: 17 active+clean; 2.0 GiB data, 8.3 GiB used, 886 GiB / 894 GiB avail; 161 MiB/s wr, 2.58k op/s 2024-05-15T12:58:25.130 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 16 16 41218 41202 160.913 131.125 0.00202736 0.00619572 2024-05-15T12:58:25.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:25 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:58:25.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:25 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:58:25.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:25 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T12:58:25.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:25 smithi027 ceph-mon[39795]: osdmap e324: 12 total, 12 up, 10 in 2024-05-15T12:58:25.465 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:58:25.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:25.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:58:25.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:25 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:58:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:25 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:58:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:25 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T12:58:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:25 smithi120 ceph-mon[43961]: osdmap e324: 12 total, 12 up, 10 in 2024-05-15T12:58:25.573 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:58:25.623 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:25.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:25 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:58:25.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:25 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:58:25.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:25 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T12:58:25.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:25 smithi177 ceph-mon[44303]: osdmap e324: 12 total, 12 up, 10 in 2024-05-15T12:58:26.130 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 17 16 43092 43076 158.336 117.125 0.00895224 0.00631098 2024-05-15T12:58:26.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:26 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:58:26.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:26 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:58:26.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:26 smithi027 ceph-mon[39795]: pgmap v887: 17 pgs: 17 active+clean; 2.4 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-05-15T12:58:26.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:26 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:58:26.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:26 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:58:26.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:26 smithi120 ceph-mon[43961]: pgmap v887: 17 pgs: 17 active+clean; 2.4 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-05-15T12:58:26.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:26 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:58:26.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:26 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:58:26.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:26 smithi177 ceph-mon[44303]: pgmap v887: 17 pgs: 17 active+clean; 2.4 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-05-15T12:58:27.130 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 18 16 45199 45183 156.856 131.688 0.00856295 0.00637104 2024-05-15T12:58:27.471 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:58:27.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:27 smithi027 ceph-mon[39795]: osdmap e325: 12 total, 12 up, 10 in 2024-05-15T12:58:27.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:27 smithi120 ceph-mon[43961]: osdmap e325: 12 total, 12 up, 10 in 2024-05-15T12:58:27.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:27 smithi177 ceph-mon[44303]: osdmap e325: 12 total, 12 up, 10 in 2024-05-15T12:58:27.797 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-05-15T12:58:27.797 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T12:58:27.837 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:58:27.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:27.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:58:27.949 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:58:27.987 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:28.013 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:58:28.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:28.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:58:28.119 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:58:28.130 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 19 16 46997 46981 154.514 112.375 0.00961253 0.00646637 2024-05-15T12:58:28.150 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:28.171 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:58:28.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:28.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:58:28.277 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:58:28.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:28 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:58:28.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:28 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:58:28.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:28 smithi120 ceph-mon[43961]: osdmap e326: 12 total, 12 up, 10 in 2024-05-15T12:58:28.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:28 smithi120 ceph-mon[43961]: pgmap v890: 17 pgs: 17 active+clean; 2.5 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 118 MiB/s wr, 1.88k op/s 2024-05-15T12:58:28.347 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:58:28.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:28.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:58:28.491 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:28 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:58:28.491 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:28 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:58:28.491 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:28 smithi027 ceph-mon[39795]: osdmap e326: 12 total, 12 up, 10 in 2024-05-15T12:58:28.491 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:28 smithi027 ceph-mon[39795]: pgmap v890: 17 pgs: 17 active+clean; 2.5 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 118 MiB/s wr, 1.88k op/s 2024-05-15T12:58:28.591 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:58:28.623 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:28.647 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:58:28.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:28.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:58:28.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:28 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:58:28.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:28 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:58:28.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:28 smithi177 ceph-mon[44303]: osdmap e326: 12 total, 12 up, 10 in 2024-05-15T12:58:28.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:28 smithi177 ceph-mon[44303]: pgmap v890: 17 pgs: 17 active+clean; 2.5 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 118 MiB/s wr, 1.88k op/s 2024-05-15T12:58:28.751 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:58:28.806 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:28.854 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:58:28.859 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:58:28.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:28.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:58:28.965 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:58:28.996 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:29.018 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:58:29.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:29.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:58:29.124 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:58:29.130 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T12:58:29.141442+0000 min lat: 0.00111471 max lat: 0.608324 avg lat: 0.00657889 2024-05-15T12:58:29.130 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T12:58:29.130 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 20 16 48627 48611 151.882 101.875 0.00918601 0.00657889 2024-05-15T12:58:29.155 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:29.179 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:58:29.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:29.183 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:58:29.284 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:58:29.294 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T12:58:29.294 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T12:58:29.294 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-05-15T12:58:29.294 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-05-15T12:58:29.294 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T12:58:29.325 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:29.345 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T12:58:29.352 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:58:29.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:29.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:58:29.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:29 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3912367018' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:58:29.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:29 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:58:29.456 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:58:29.485 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:29.510 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:58:29.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:29.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:58:29.615 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:58:29.641 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:29.659 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:58:29.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:29.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:58:29.763 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:58:29.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:29 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3912367018' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:58:29.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:29 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:58:29.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:29 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3912367018' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:58:29.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:29 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:58:29.806 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:29.840 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:58:29.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:29.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:58:29.946 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:58:29.990 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:30.130 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 21 16 50088 50072 148.998 91.3125 0.00917904 0.00670716 2024-05-15T12:58:30.185 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-05-15T12:58:30.520 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:58:30.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:30 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:58:30.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:30 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:58:30.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:30 smithi027 ceph-mon[39795]: osdmap e327: 12 total, 12 up, 10 in 2024-05-15T12:58:30.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:30 smithi027 ceph-mon[39795]: pgmap v892: 17 pgs: 17 active+clean; 2.8 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-05-15T12:58:30.664 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:58:30.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:30.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:58:30.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:30 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:58:30.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:30 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:58:30.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:30 smithi177 ceph-mon[44303]: osdmap e327: 12 total, 12 up, 10 in 2024-05-15T12:58:30.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:30 smithi177 ceph-mon[44303]: pgmap v892: 17 pgs: 17 active+clean; 2.8 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-05-15T12:58:30.773 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:58:30.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:30 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:58:30.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:30 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:58:30.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:30 smithi120 ceph-mon[43961]: osdmap e327: 12 total, 12 up, 10 in 2024-05-15T12:58:30.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:30 smithi120 ceph-mon[43961]: pgmap v892: 17 pgs: 17 active+clean; 2.8 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-05-15T12:58:30.805 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:30.825 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:58:30.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:30.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:58:30.929 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:58:30.962 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:30.988 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:58:30.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:30.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:58:31.093 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:58:31.126 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:31.130 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 22 16 51827 51811 147.165 108.688 0.00897475 0.00679088 2024-05-15T12:58:31.155 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:58:31.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:31.159 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:58:31.259 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:58:31.298 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:31.329 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:58:31.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:31.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:58:31.433 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:58:31.468 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:31.495 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:58:31.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:31.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:58:31.604 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:58:31.660 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:32.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:31 smithi027 ceph-mon[39795]: pgmap v893: 17 pgs: 17 active+clean; 3.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-05-15T12:58:32.130 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 23 16 53581 53565 145.533 109.625 0.00332265 0.00686713 2024-05-15T12:58:32.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:31 smithi177 ceph-mon[44303]: pgmap v893: 17 pgs: 17 active+clean; 3.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-05-15T12:58:32.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:31 smithi120 ceph-mon[43961]: pgmap v893: 17 pgs: 17 active+clean; 3.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-05-15T12:58:32.734 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:58:32.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:32.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:58:32.840 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:58:32.872 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:32.898 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:58:32.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:32.901 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:58:33.002 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:58:33.052 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:33.084 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:58:33.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:33.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:58:33.130 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 24 16 55352 55336 144.081 110.688 0.00442361 0.00693701 2024-05-15T12:58:33.190 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:58:33.224 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:33.249 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:58:33.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:33.254 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:58:33.355 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:58:33.418 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:33.846 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:58:33.866 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:58:33.900 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:58:33.933 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:58:33.972 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:58:34.007 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:58:34.038 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:58:34.071 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:58:34.097 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:58:34.122 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:58:34.130 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 25 16 56727 56711 141.755 85.9375 0.00252959 0.00701727 2024-05-15T12:58:34.150 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:58:34.188 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:58:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:58:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:58:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:58:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:58:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:58:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:58:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:58:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:58:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:58:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:58:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:58:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:58:34.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:34 smithi177 ceph-mon[44303]: pgmap v894: 17 pgs: 17 active+clean; 3.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-05-15T12:58:34.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:34 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/716542319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:58:34.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:34 smithi027 ceph-mon[39795]: pgmap v894: 17 pgs: 17 active+clean; 3.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-05-15T12:58:34.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:34 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/716542319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:58:34.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:34 smithi120 ceph-mon[43961]: pgmap v894: 17 pgs: 17 active+clean; 3.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-05-15T12:58:34.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:34 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/716542319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:58:34.731 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T12:58:34.885 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:58:34.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:34.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:58:34.989 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:58:35.017 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:35.131 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 26 16 57446 57430 138.031 44.9375 0.00211656 0.00707673 2024-05-15T12:58:35.166 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:58:35.175 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:58:35.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:35.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:58:35.279 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:58:35.320 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:35.360 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:58:35.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:35.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:58:35.464 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:58:35.512 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:35.534 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:58:35.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:35.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:58:35.637 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:58:35.672 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:35.701 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:58:35.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:35.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:58:35.806 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:58:35.836 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:35.863 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:58:35.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:35.866 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:58:35.968 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:58:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:36.131 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 27 15 58957 58942 136.419 94.5 0.00930244 0.00732545 2024-05-15T12:58:36.171 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:58:36.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:36.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:58:36.177 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:35 smithi027 ceph-mon[39795]: pgmap v895: 17 pgs: 17 active+clean; 3.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-05-15T12:58:36.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:35 smithi177 ceph-mon[44303]: pgmap v895: 17 pgs: 17 active+clean; 3.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-05-15T12:58:36.227 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:58:36.230 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T12:58:36.277 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:58:36.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:35 smithi120 ceph-mon[43961]: pgmap v895: 17 pgs: 17 active+clean; 3.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-05-15T12:58:36.308 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:36.329 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:58:36.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:36.332 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:58:36.433 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:58:36.488 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:37.131 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 28 16 60545 60529 135.089 99.1875 0.00931867 0.00739735 2024-05-15T12:58:37.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:36 smithi177 ceph-mon[44303]: from='client.25136 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:58:37.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:36 smithi027 ceph-mon[39795]: from='client.25136 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:58:37.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:36 smithi120 ceph-mon[43961]: from='client.25136 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:58:37.861 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:58:37.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:37.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:58:37.877 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":895,"stamp":"2024-05-15T12:58:35.807406+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3718316576,"num_objects":56730,"num_object_clones":0,"num_object_copies":170190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56730,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":56909,"num_write_kb":3633530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56867,"ondisk_log_size":56867,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":13166876,"kb_used_data":10610248,"kb_used_omap":15,"kb_used_meta":2556528,"kb_avail":924284644,"statfs":{"total":959950356480,"available":946467475456,"internally_reserved":0,"allocated":10864893952,"data_stored":10861938266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15957,"internal_metadata":2617885099},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,15,5,8],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1086849024,"num_objects":16584,"num_object_clones":0,"num_object_copies":49752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16584,"num_write_kb":1061376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.582017"},"pg_stats":[{"pgid":"4.f","version":"327'3548","reported_seq":3578,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-05-15T12:58:35.586941+0000","last_change":"2024-05-15T12:57:55.158090+0000","last_active":"2024-05-15T12:58:35.586941+0000","last_peered":"2024-05-15T12:58:35.586941+0000","last_clean":"2024-05-15T12:58:35.586941+0000","last_became_active":"2024-05-15T12:57:55.157850+0000","last_became_peered":"2024-05-15T12:57:55.157850+0000","last_unstale":"2024-05-15T12:58:35.586941+0000","last_undegraded":"2024-05-15T12:58:35.586941+0000","last_fullsized":"2024-05-15T12:58:35.586941+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:06.150569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232521728,"num_objects":3548,"num_object_clones":0,"num_object_copies":10644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3548,"num_write_kb":227072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,9],"acting":[1,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"327'3610","reported_seq":3640,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-05-15T12:58:34.537469+0000","last_change":"2024-05-15T12:57:55.242710+0000","last_active":"2024-05-15T12:58:34.537469+0000","last_peered":"2024-05-15T12:58:34.537469+0000","last_clean":"2024-05-15T12:58:34.537469+0000","last_became_active":"2024-05-15T12:57:55.242367+0000","last_became_peered":"2024-05-15T12:57:55.242367+0000","last_unstale":"2024-05-15T12:58:34.537469+0000","last_undegraded":"2024-05-15T12:58:34.537469+0000","last_fullsized":"2024-05-15T12:58:34.537469+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":3610,"log_dups_size":0,"ondisk_log_size":3610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:20.835457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236584960,"num_objects":3610,"num_object_clones":0,"num_object_copies":10830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3610,"num_write_kb":231040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"327'3530","reported_seq":3560,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-05-15T12:58:34.551034+0000","last_change":"2024-05-15T12:57:55.155833+0000","last_active":"2024-05-15T12:58:34.551034+0000","last_peered":"2024-05-15T12:58:34.551034+0000","last_clean":"2024-05-15T12:58:34.551034+0000","last_became_active":"2024-05-15T12:57:55.155480+0000","last_became_peered":"2024-05-15T12:57:55.155480+0000","last_unstale":"2024-05-15T12:58:34.551034+0000","last_undegraded":"2024-05-15T12:58:34.551034+0000","last_fullsized":"2024-05-15T12:58:34.551034+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":3530,"log_dups_size":0,"ondisk_log_size":3530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:11.204322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231342080,"num_objects":3530,"num_object_clones":0,"num_object_copies":10590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3530,"num_write_kb":225920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"327'3643","reported_seq":3673,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-05-15T12:58:34.557973+0000","last_change":"2024-05-15T12:57:55.157244+0000","last_active":"2024-05-15T12:58:34.557973+0000","last_peered":"2024-05-15T12:58:34.557973+0000","last_clean":"2024-05-15T12:58:34.557973+0000","last_became_active":"2024-05-15T12:57:55.156859+0000","last_became_peered":"2024-05-15T12:57:55.156859+0000","last_unstale":"2024-05-15T12:58:34.557973+0000","last_undegraded":"2024-05-15T12:58:34.557973+0000","last_fullsized":"2024-05-15T12:58:34.557973+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":3643,"log_dups_size":0,"ondisk_log_size":3643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:12.960095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238747648,"num_objects":3643,"num_object_clones":0,"num_object_copies":10929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3643,"num_write_kb":233152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"327'3531","reported_seq":3561,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-05-15T12:58:33.652874+0000","last_change":"2024-05-15T12:57:55.160034+0000","last_active":"2024-05-15T12:58:33.652874+0000","last_peered":"2024-05-15T12:58:33.652874+0000","last_clean":"2024-05-15T12:58:33.652874+0000","last_became_active":"2024-05-15T12:57:55.159699+0000","last_became_peered":"2024-05-15T12:57:55.159699+0000","last_unstale":"2024-05-15T12:58:33.652874+0000","last_undegraded":"2024-05-15T12:58:33.652874+0000","last_fullsized":"2024-05-15T12:58:33.652874+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":3531,"log_dups_size":0,"ondisk_log_size":3531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:12.991722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231407616,"num_objects":3531,"num_object_clones":0,"num_object_copies":10593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3531,"num_write_kb":225984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"327'3516","reported_seq":3546,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-05-15T12:58:33.640900+0000","last_change":"2024-05-15T12:57:55.242821+0000","last_active":"2024-05-15T12:58:33.640900+0000","last_peered":"2024-05-15T12:58:33.640900+0000","last_clean":"2024-05-15T12:58:33.640900+0000","last_became_active":"2024-05-15T12:57:55.242372+0000","last_became_peered":"2024-05-15T12:57:55.242372+0000","last_unstale":"2024-05-15T12:58:33.640900+0000","last_undegraded":"2024-05-15T12:58:33.640900+0000","last_fullsized":"2024-05-15T12:58:33.640900+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":3516,"log_dups_size":0,"ondisk_log_size":3516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:15.409516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230424576,"num_objects":3516,"num_object_clones":0,"num_object_copies":10548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3516,"num_write_kb":225024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"327'3649","reported_seq":3679,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-05-15T12:58:35.556450+0000","last_change":"2024-05-15T12:57:55.157796+0000","last_active":"2024-05-15T12:58:35.556450+0000","last_peered":"2024-05-15T12:58:35.556450+0000","last_clean":"2024-05-15T12:58:35.556450+0000","last_became_active":"2024-05-15T12:57:55.157422+0000","last_became_peered":"2024-05-15T12:57:55.157422+0000","last_unstale":"2024-05-15T12:58:35.556450+0000","last_undegraded":"2024-05-15T12:58:35.556450+0000","last_fullsized":"2024-05-15T12:58:35.556450+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":3649,"log_dups_size":0,"ondisk_log_size":3649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:23.446009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239140864,"num_objects":3649,"num_object_clones":0,"num_object_copies":10947,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3649,"num_write_kb":233536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"327'3624","reported_seq":3654,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-05-15T12:58:33.658286+0000","last_change":"2024-05-15T12:57:55.242739+0000","last_active":"2024-05-15T12:58:33.658286+0000","last_peered":"2024-05-15T12:58:33.658286+0000","last_clean":"2024-05-15T12:58:33.658286+0000","last_became_active":"2024-05-15T12:57:55.242387+0000","last_became_peered":"2024-05-15T12:57:55.242387+0000","last_unstale":"2024-05-15T12:58:33.658286+0000","last_undegraded":"2024-05-15T12:58:33.658286+0000","last_fullsized":"2024-05-15T12:58:33.658286+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":3624,"log_dups_size":0,"ondisk_log_size":3624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:20.234355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237502464,"num_objects":3624,"num_object_clones":0,"num_object_copies":10872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3624,"num_write_kb":231936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,10],"acting":[6,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"327'3607","reported_seq":3637,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-05-15T12:58:35.603340+0000","last_change":"2024-05-15T12:57:55.242791+0000","last_active":"2024-05-15T12:58:35.603340+0000","last_peered":"2024-05-15T12:58:35.603340+0000","last_clean":"2024-05-15T12:58:35.603340+0000","last_became_active":"2024-05-15T12:57:55.242453+0000","last_became_peered":"2024-05-15T12:57:55.242453+0000","last_unstale":"2024-05-15T12:58:35.603340+0000","last_undegraded":"2024-05-15T12:58:35.603340+0000","last_fullsized":"2024-05-15T12:58:35.603340+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":3607,"log_dups_size":0,"ondisk_log_size":3607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:11.830376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236388352,"num_objects":3607,"num_object_clones":0,"num_object_copies":10821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3607,"num_write_kb":230848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"327'3624","reported_seq":3654,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-05-15T12:58:35.438166+0000","last_change":"2024-05-15T12:57:55.157512+0000","last_active":"2024-05-15T12:58:35.438166+0000","last_peered":"2024-05-15T12:58:35.438166+0000","last_clean":"2024-05-15T12:58:35.438166+0000","last_became_active":"2024-05-15T12:57:55.157131+0000","last_became_peered":"2024-05-15T12:57:55.157131+0000","last_unstale":"2024-05-15T12:58:35.438166+0000","last_undegraded":"2024-05-15T12:58:35.438166+0000","last_fullsized":"2024-05-15T12:58:35.438166+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":3624,"log_dups_size":0,"ondisk_log_size":3624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:12.418393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237502464,"num_objects":3624,"num_object_clones":0,"num_object_copies":10872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3624,"num_write_kb":231936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"327'3647","reported_seq":3677,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-05-15T12:58:35.586514+0000","last_change":"2024-05-15T12:57:55.158173+0000","last_active":"2024-05-15T12:58:35.586514+0000","last_peered":"2024-05-15T12:58:35.586514+0000","last_clean":"2024-05-15T12:58:35.586514+0000","last_became_active":"2024-05-15T12:57:55.157465+0000","last_became_peered":"2024-05-15T12:57:55.157465+0000","last_unstale":"2024-05-15T12:58:35.586514+0000","last_undegraded":"2024-05-15T12:58:35.586514+0000","last_fullsized":"2024-05-15T12:58:35.586514+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":3647,"log_dups_size":0,"ondisk_log_size":3647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:18.405669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239009792,"num_objects":3647,"num_object_clones":0,"num_object_copies":10941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3647,"num_write_kb":233408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"327'3581","reported_seq":3611,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-05-15T12:58:35.604252+0000","last_change":"2024-05-15T12:57:55.158034+0000","last_active":"2024-05-15T12:58:35.604252+0000","last_peered":"2024-05-15T12:58:35.604252+0000","last_clean":"2024-05-15T12:58:35.604252+0000","last_became_active":"2024-05-15T12:57:55.157716+0000","last_became_peered":"2024-05-15T12:57:55.157716+0000","last_unstale":"2024-05-15T12:58:35.604252+0000","last_undegraded":"2024-05-15T12:58:35.604252+0000","last_fullsized":"2024-05-15T12:58:35.604252+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":3581,"log_dups_size":0,"ondisk_log_size":3581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:05.527560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234684416,"num_objects":3581,"num_object_clones":0,"num_object_copies":10743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3581,"num_write_kb":229184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"327'3455","reported_seq":3485,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-05-15T12:58:32.716464+0000","last_change":"2024-05-15T12:57:55.158066+0000","last_active":"2024-05-15T12:58:32.716464+0000","last_peered":"2024-05-15T12:58:32.716464+0000","last_clean":"2024-05-15T12:58:32.716464+0000","last_became_active":"2024-05-15T12:57:55.157243+0000","last_became_peered":"2024-05-15T12:57:55.157243+0000","last_unstale":"2024-05-15T12:58:32.716464+0000","last_undegraded":"2024-05-15T12:58:32.716464+0000","last_fullsized":"2024-05-15T12:58:32.716464+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":3455,"log_dups_size":0,"ondisk_log_size":3455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:21.941110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226426880,"num_objects":3455,"num_object_clones":0,"num_object_copies":10365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3455,"num_write_kb":221120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,8],"acting":[5,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"327'3388","reported_seq":3418,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-05-15T12:58:32.722510+0000","last_change":"2024-05-15T12:57:55.158553+0000","last_active":"2024-05-15T12:58:32.722510+0000","last_peered":"2024-05-15T12:58:32.722510+0000","last_clean":"2024-05-15T12:58:32.722510+0000","last_became_active":"2024-05-15T12:57:55.158245+0000","last_became_peered":"2024-05-15T12:57:55.158245+0000","last_unstale":"2024-05-15T12:58:32.722510+0000","last_undegraded":"2024-05-15T12:58:32.722510+0000","last_fullsized":"2024-05-15T12:58:32.722510+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":3388,"log_dups_size":0,"ondisk_log_size":3388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:58:55.122035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222035968,"num_objects":3388,"num_object_clones":0,"num_object_copies":10164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3388,"num_write_kb":216832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"327'3332","reported_seq":3362,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-05-15T12:58:31.512677+0000","last_change":"2024-05-15T12:57:55.160182+0000","last_active":"2024-05-15T12:58:31.512677+0000","last_peered":"2024-05-15T12:58:31.512677+0000","last_clean":"2024-05-15T12:58:31.512677+0000","last_became_active":"2024-05-15T12:57:55.159918+0000","last_became_peered":"2024-05-15T12:57:55.159918+0000","last_unstale":"2024-05-15T12:58:31.512677+0000","last_undegraded":"2024-05-15T12:58:31.512677+0000","last_fullsized":"2024-05-15T12:58:31.512677+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":3332,"log_dups_size":0,"ondisk_log_size":3332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:04.512522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218365952,"num_objects":3332,"num_object_clones":0,"num_object_copies":9996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3332,"num_write_kb":213248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"327'3443","reported_seq":3473,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-05-15T12:58:33.666631+0000","last_change":"2024-05-15T12:57:55.243127+0000","last_active":"2024-05-15T12:58:33.666631+0000","last_peered":"2024-05-15T12:58:33.666631+0000","last_clean":"2024-05-15T12:58:33.666631+0000","last_became_active":"2024-05-15T12:57:55.242403+0000","last_became_peered":"2024-05-15T12:57:55.242403+0000","last_unstale":"2024-05-15T12:58:33.666631+0000","last_undegraded":"2024-05-15T12:58:33.666631+0000","last_fullsized":"2024-05-15T12:58:33.666631+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":3443,"log_dups_size":0,"ondisk_log_size":3443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:18.967030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225640448,"num_objects":3443,"num_object_clones":0,"num_object_copies":10329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3443,"num_write_kb":220352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,0],"acting":[6,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":473,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-05-15T12:58:29.336254+0000","last_change":"2024-05-15T12:57:46.629237+0000","last_active":"2024-05-15T12:58:29.336254+0000","last_peered":"2024-05-15T12:58:29.336254+0000","last_clean":"2024-05-15T12:58:29.336254+0000","last_became_active":"2024-05-15T12:56:08.882123+0000","last_became_peered":"2024-05-15T12:56:08.882123+0000","last_unstale":"2024-05-15T12:58:29.336254+0000","last_undegraded":"2024-05-15T12:58:29.336254+0000","last_fullsized":"2024-05-15T12:58:29.336254+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T12:57:46.629099+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T12:57:46.629099+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:03.416075+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,10],"acting":[6,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":3717726208,"num_objects":56728,"num_object_clones":0,"num_object_copies":170184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56728,"num_write_kb":3630592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11013914624,"data_stored":11013914624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56728,"ondisk_log_size":56728,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":0,"seq":386547056797,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351137,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1254152,"kb_used_data":1034176,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":92491000,"statfs":{"total":95995035648,"available":94710784000,"internally_reserved":0,"allocated":1058996224,"data_stored":1058698153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":225245632},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514663,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1484360,"kb_used_data":1150528,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":92260792,"statfs":{"total":95995035648,"available":94475051008,"internally_reserved":0,"allocated":1178140672,"data_stored":1177842601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":341834176},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874413,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":667904,"kb_used_data":465976,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":93077248,"statfs":{"total":95995035648,"available":95311101952,"internally_reserved":0,"allocated":477159424,"data_stored":476864925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":240518168752,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430389,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1732680,"kb_used_data":1543232,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":92012472,"statfs":{"total":95995035648,"available":94220771328,"internally_reserved":0,"allocated":1580269568,"data_stored":1579971517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":193984966},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575057,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2363904,"kb_used_data":2057848,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":91381248,"statfs":{"total":95995035648,"available":93574397952,"internally_reserved":0,"allocated":2107236352,"data_stored":2106941853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738584,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1415552,"kb_used_data":1119224,"kb_used_omap":1,"kb_used_meta":296318,"kb_avail":92329600,"statfs":{"total":95995035648,"available":94545510400,"internally_reserved":0,"allocated":1146085376,"data_stored":1145790877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":303430085},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411533,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":828032,"kb_used_data":652472,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":92917120,"statfs":{"total":95995035648,"available":95147130880,"internally_reserved":0,"allocated":668131328,"data_stored":667836829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149894,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1366020,"kb_used_data":1023864,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92379132,"statfs":{"total":95995035648,"available":94596231168,"internally_reserved":0,"allocated":1048436736,"data_stored":1048142237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":350353863},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986370,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1414848,"kb_used_data":1135160,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":92330304,"statfs":{"total":95995035648,"available":94546231296,"internally_reserved":0,"allocated":1162403840,"data_stored":1162109341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":286390725},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692027,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":639424,"kb_used_data":427768,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93105728,"statfs":{"total":95995035648,"available":95340265472,"internally_reserved":0,"allocated":438034432,"data_stored":437739933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1149304832,"data_stored":1149304832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2132541440,"data_stored":2132541440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":672923648,"data_stored":672923648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1083637760,"data_stored":1083637760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1177092096,"data_stored":1177092096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":448462848,"data_stored":448462848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1619525632,"data_stored":1619525632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":478347264,"data_stored":478347264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1184825344,"data_stored":1184825344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1067253760,"data_stored":1067253760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T12:58:37.879 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T12:58:37.969 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:58:38.000 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:38.022 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:58:38.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:38.025 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:58:38.126 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:58:38.131 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 29 16 62177 62161 133.948 102 0.0168216 0.00746063 2024-05-15T12:58:38.159 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:37 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:58:38.159 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:37 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:58:38.159 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:37 smithi120 ceph-mon[43961]: osdmap e328: 12 total, 12 up, 10 in 2024-05-15T12:58:38.159 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:37 smithi120 ceph-mon[43961]: pgmap v897: 17 pgs: 17 active+clean; 3.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-05-15T12:58:38.163 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:38.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:37 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:58:38.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:37 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:58:38.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:37 smithi177 ceph-mon[44303]: osdmap e328: 12 total, 12 up, 10 in 2024-05-15T12:58:38.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:37 smithi177 ceph-mon[44303]: pgmap v897: 17 pgs: 17 active+clean; 3.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-05-15T12:58:38.188 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:58:38.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:38.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:58:38.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:37 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:58:38.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:37 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:58:38.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:37 smithi027 ceph-mon[39795]: osdmap e328: 12 total, 12 up, 10 in 2024-05-15T12:58:38.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:37 smithi027 ceph-mon[39795]: pgmap v897: 17 pgs: 17 active+clean; 3.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-05-15T12:58:38.296 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:58:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:58:38.336 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:38.359 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:58:38.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:38.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:58:38.464 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:58:38.497 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:38.524 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:58:38.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:38.531 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:58:38.632 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:58:38.709 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:38.927 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:38 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:58:38.927 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:38 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:58:38.970 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:58:39.131 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 30 16 63715 63699 132.687 96.125 0.0053823 0.00753312 2024-05-15T12:58:39.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:38 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:58:39.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:38 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:58:39.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:38 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:58:39.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:38 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:58:39.883 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:58:39.887 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:58:39.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:39.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:58:39.993 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:58:40.030 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:40.058 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:58:40.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:40.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:58:40.131 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 31 16 65227 65211 131.455 94.5 0.00310197 0.00760287 2024-05-15T12:58:40.164 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:58:40.201 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:39 smithi120 ceph-mon[43961]: osdmap e329: 12 total, 12 up, 10 in 2024-05-15T12:58:40.201 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:39 smithi120 ceph-mon[43961]: pgmap v899: 17 pgs: 17 active+clean; 3.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-05-15T12:58:40.206 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:40.236 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:58:40.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:40.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:58:40.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:39 smithi027 ceph-mon[39795]: osdmap e329: 12 total, 12 up, 10 in 2024-05-15T12:58:40.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:39 smithi027 ceph-mon[39795]: pgmap v899: 17 pgs: 17 active+clean; 3.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-05-15T12:58:40.343 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:58:40.405 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:40.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:39 smithi177 ceph-mon[44303]: osdmap e329: 12 total, 12 up, 10 in 2024-05-15T12:58:40.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:39 smithi177 ceph-mon[44303]: pgmap v899: 17 pgs: 17 active+clean; 3.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-05-15T12:58:40.437 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:58:40.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:40.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:58:40.546 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:58:40.577 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:40.597 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:58:40.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:40.601 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:58:40.702 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:58:40.736 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:40.760 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:58:40.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:40.763 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:58:40.864 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:58:40.921 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:41.095 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:58:41.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:41.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:58:41.131 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 32 16 66704 66688 130.231 92.3125 0.0130573 0.0076725 2024-05-15T12:58:41.168 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T12:58:41.168 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:58:41.201 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:58:41.235 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:41.258 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:58:41.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:41.262 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:58:41.363 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:58:41.400 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:41.428 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:58:41.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:41.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:58:41.533 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:58:41.579 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:41.853 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:58:41.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:41.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:58:41.964 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:58:41.994 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:42.014 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:58:42.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:42.018 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:58:42.119 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:58:42.132 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 33 16 68133 68117 128.987 89.3125 0.00744682 0.007749 2024-05-15T12:58:42.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:41 smithi177 ceph-mon[44303]: pgmap v900: 17 pgs: 17 active+clean; 3.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-05-15T12:58:42.151 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:42.175 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:58:42.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:42.178 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:58:42.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:41 smithi027 ceph-mon[39795]: pgmap v900: 17 pgs: 17 active+clean; 3.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-05-15T12:58:42.279 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:58:42.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:41 smithi120 ceph-mon[43961]: pgmap v900: 17 pgs: 17 active+clean; 3.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-05-15T12:58:42.321 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:42.346 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:58:42.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:42.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:58:42.449 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:58:42.516 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:42.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:42.581 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:58:42.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:58:42.616 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-05-15T12:58:42.682 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:58:42.715 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:42.741 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:58:42.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:42.744 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:58:42.845 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:58:42.899 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:42 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1224872790' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:58:42.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:42 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:58:42.901 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:42 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1224872790' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:58:42.901 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:42 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:58:42.906 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:42.971 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T12:58:42.985 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:58:43.132 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 34 16 69571 69555 127.837 89.875 0.00585108 0.00781828 2024-05-15T12:58:43.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:42 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1224872790' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:58:43.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:42 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:58:43.786 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:58:43.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:43.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:58:43.899 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:58:43.925 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:43.944 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:58:43.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:43.946 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:58:44.047 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:58:44.059 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:43 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:58:44.059 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:43 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:58:44.059 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:43 smithi027 ceph-mon[39795]: osdmap e330: 12 total, 12 up, 10 in 2024-05-15T12:58:44.059 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:43 smithi027 ceph-mon[39795]: pgmap v902: 17 pgs: 17 active+clean; 4.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-05-15T12:58:44.088 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:44.132 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 35 16 70469 70453 125.788 56.125 0.00463123 0.00794618 2024-05-15T12:58:44.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:43 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:58:44.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:43 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:58:44.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:43 smithi177 ceph-mon[44303]: osdmap e330: 12 total, 12 up, 10 in 2024-05-15T12:58:44.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:43 smithi177 ceph-mon[44303]: pgmap v902: 17 pgs: 17 active+clean; 4.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-05-15T12:58:44.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:43 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:58:44.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:43 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:58:44.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:43 smithi120 ceph-mon[43961]: osdmap e330: 12 total, 12 up, 10 in 2024-05-15T12:58:44.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:43 smithi120 ceph-mon[43961]: pgmap v902: 17 pgs: 17 active+clean; 4.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-05-15T12:58:44.544 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:58:44.744 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:58:44.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:44.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:58:44.848 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:58:44.883 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:44.905 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:58:44.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:44.909 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:58:45.010 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:58:45.044 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:45.071 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:58:45.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:45.075 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:58:45.132 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 36 16 71955 71939 124.873 92.875 0.00815175 0.0080043 2024-05-15T12:58:45.177 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:58:45.212 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:45.233 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:58:45.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:45.237 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:58:45.338 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:58:45.369 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:45.388 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:58:45.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:45.390 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:58:45.395 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T12:58:45.395 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:58:45.491 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:58:45.521 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:45.542 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:58:45.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:45.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:58:45.646 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:58:45.706 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:45.753 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:58:45.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:45.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:58:45.858 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:58:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:46.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:45 smithi120 ceph-mon[43961]: pgmap v903: 17 pgs: 17 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-05-15T12:58:46.104 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-05-15T12:58:46.133 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 37 16 73112 73096 123.453 72.3125 0.0234187 0.00809576 2024-05-15T12:58:46.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:45 smithi177 ceph-mon[44303]: pgmap v903: 17 pgs: 17 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-05-15T12:58:46.200 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:58:46.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:46.206 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:45 smithi027 ceph-mon[39795]: pgmap v903: 17 pgs: 17 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-05-15T12:58:46.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:58:46.308 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:58:46.344 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:46.370 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:58:46.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:46.374 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:58:46.467 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:58:46.475 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:58:46.507 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:46.532 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:58:46.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:46.535 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:58:46.636 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:58:46.697 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:47.002 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T12:58:47.074 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:58:47.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:47.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:58:47.133 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 38 16 74315 74299 122.182 75.1875 0.00571244 0.00814955 2024-05-15T12:58:47.178 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:58:47.238 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:47.251 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:58:47.251 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T12:58:47.515 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:58:47.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:47.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:58:47.620 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:58:47.654 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:47.676 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:58:47.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:47.680 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:58:47.781 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:58:47.832 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:47.865 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:58:47.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:47.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:58:47.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:47 smithi177 ceph-mon[44303]: from='client.25145 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:58:47.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:47 smithi177 ceph-mon[44303]: pgmap v904: 17 pgs: 17 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-05-15T12:58:47.970 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:58:47.998 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:48.026 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:58:48.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:48.029 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:58:48.130 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:58:48.133 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 39 16 75656 75640 121.198 83.8125 0.00386777 0.00823742 2024-05-15T12:58:48.141 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:47 smithi027 ceph-mon[39795]: from='client.25145 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:58:48.141 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:47 smithi027 ceph-mon[39795]: pgmap v904: 17 pgs: 17 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-05-15T12:58:48.142 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":903,"stamp":"2024-05-15T12:58:45.809454+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4639162912,"num_objects":70781,"num_object_clones":0,"num_object_copies":212343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70781,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":70960,"num_write_kb":4532794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70918,"ondisk_log_size":70918,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":16065448,"kb_used_data":13365824,"kb_used_omap":15,"kb_used_meta":2699504,"kb_avail":921386072,"statfs":{"total":959950356480,"available":943499337728,"internally_reserved":0,"allocated":13686603776,"data_stored":13683530842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15957,"internal_metadata":2764292523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":835256320,"num_objects":12745,"num_object_clones":0,"num_object_copies":38235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12745,"num_write_kb":815680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001610"},"pg_stats":[{"pgid":"4.f","version":"330'4404","reported_seq":4437,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-05-15T12:58:45.603474+0000","last_change":"2024-05-15T12:57:55.158090+0000","last_active":"2024-05-15T12:58:45.603474+0000","last_peered":"2024-05-15T12:58:45.603474+0000","last_clean":"2024-05-15T12:58:45.603474+0000","last_became_active":"2024-05-15T12:57:55.157850+0000","last_became_peered":"2024-05-15T12:57:55.157850+0000","last_unstale":"2024-05-15T12:58:45.603474+0000","last_undegraded":"2024-05-15T12:58:45.603474+0000","last_fullsized":"2024-05-15T12:58:45.603474+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":4404,"log_dups_size":0,"ondisk_log_size":4404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:06.150569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288620544,"num_objects":4404,"num_object_clones":0,"num_object_copies":13212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4404,"num_write_kb":281856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,9],"acting":[1,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"330'4431","reported_seq":4467,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-05-15T12:58:44.685265+0000","last_change":"2024-05-15T12:57:55.242710+0000","last_active":"2024-05-15T12:58:44.685265+0000","last_peered":"2024-05-15T12:58:44.685265+0000","last_clean":"2024-05-15T12:58:44.685265+0000","last_became_active":"2024-05-15T12:57:55.242367+0000","last_became_peered":"2024-05-15T12:57:55.242367+0000","last_unstale":"2024-05-15T12:58:44.685265+0000","last_undegraded":"2024-05-15T12:58:44.685265+0000","last_fullsized":"2024-05-15T12:58:44.685265+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":4431,"log_dups_size":0,"ondisk_log_size":4431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:20.835457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290390016,"num_objects":4431,"num_object_clones":0,"num_object_copies":13293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4431,"num_write_kb":283584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"330'4384","reported_seq":4420,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-05-15T12:58:44.659344+0000","last_change":"2024-05-15T12:57:55.155833+0000","last_active":"2024-05-15T12:58:44.659344+0000","last_peered":"2024-05-15T12:58:44.659344+0000","last_clean":"2024-05-15T12:58:44.659344+0000","last_became_active":"2024-05-15T12:57:55.155480+0000","last_became_peered":"2024-05-15T12:57:55.155480+0000","last_unstale":"2024-05-15T12:58:44.659344+0000","last_undegraded":"2024-05-15T12:58:44.659344+0000","last_fullsized":"2024-05-15T12:58:44.659344+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":4384,"log_dups_size":0,"ondisk_log_size":4384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:11.204322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287309824,"num_objects":4384,"num_object_clones":0,"num_object_copies":13152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4384,"num_write_kb":280576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"330'4574","reported_seq":4610,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-05-15T12:58:44.661010+0000","last_change":"2024-05-15T12:57:55.157244+0000","last_active":"2024-05-15T12:58:44.661010+0000","last_peered":"2024-05-15T12:58:44.661010+0000","last_clean":"2024-05-15T12:58:44.661010+0000","last_became_active":"2024-05-15T12:57:55.156859+0000","last_became_peered":"2024-05-15T12:57:55.156859+0000","last_unstale":"2024-05-15T12:58:44.661010+0000","last_undegraded":"2024-05-15T12:58:44.661010+0000","last_fullsized":"2024-05-15T12:58:44.661010+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":4574,"log_dups_size":0,"ondisk_log_size":4574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:12.960095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299761664,"num_objects":4574,"num_object_clones":0,"num_object_copies":13722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4574,"num_write_kb":292736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"330'4445","reported_seq":4481,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-05-15T12:58:43.817976+0000","last_change":"2024-05-15T12:57:55.160034+0000","last_active":"2024-05-15T12:58:43.817976+0000","last_peered":"2024-05-15T12:58:43.817976+0000","last_clean":"2024-05-15T12:58:43.817976+0000","last_became_active":"2024-05-15T12:57:55.159699+0000","last_became_peered":"2024-05-15T12:57:55.159699+0000","last_unstale":"2024-05-15T12:58:43.817976+0000","last_undegraded":"2024-05-15T12:58:43.817976+0000","last_fullsized":"2024-05-15T12:58:43.817976+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":4445,"log_dups_size":0,"ondisk_log_size":4445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:12.991722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291307520,"num_objects":4445,"num_object_clones":0,"num_object_copies":13335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4445,"num_write_kb":284480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"330'4335","reported_seq":4371,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-05-15T12:58:43.240688+0000","last_change":"2024-05-15T12:57:55.242821+0000","last_active":"2024-05-15T12:58:43.240688+0000","last_peered":"2024-05-15T12:58:43.240688+0000","last_clean":"2024-05-15T12:58:43.240688+0000","last_became_active":"2024-05-15T12:57:55.242372+0000","last_became_peered":"2024-05-15T12:57:55.242372+0000","last_unstale":"2024-05-15T12:58:43.240688+0000","last_undegraded":"2024-05-15T12:58:43.240688+0000","last_fullsized":"2024-05-15T12:58:43.240688+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":4335,"log_dups_size":0,"ondisk_log_size":4335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:15.409516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284098560,"num_objects":4335,"num_object_clones":0,"num_object_copies":13005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4335,"num_write_kb":277440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"330'4467","reported_seq":4503,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-05-15T12:58:45.578772+0000","last_change":"2024-05-15T12:57:55.157796+0000","last_active":"2024-05-15T12:58:45.578772+0000","last_peered":"2024-05-15T12:58:45.578772+0000","last_clean":"2024-05-15T12:58:45.578772+0000","last_became_active":"2024-05-15T12:57:55.157422+0000","last_became_peered":"2024-05-15T12:57:55.157422+0000","last_unstale":"2024-05-15T12:58:45.578772+0000","last_undegraded":"2024-05-15T12:58:45.578772+0000","last_fullsized":"2024-05-15T12:58:45.578772+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":4467,"log_dups_size":0,"ondisk_log_size":4467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:23.446009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292749312,"num_objects":4467,"num_object_clones":0,"num_object_copies":13401,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4467,"num_write_kb":285888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"330'4480","reported_seq":4516,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-05-15T12:58:43.264043+0000","last_change":"2024-05-15T12:57:55.242739+0000","last_active":"2024-05-15T12:58:43.264043+0000","last_peered":"2024-05-15T12:58:43.264043+0000","last_clean":"2024-05-15T12:58:43.264043+0000","last_became_active":"2024-05-15T12:57:55.242387+0000","last_became_peered":"2024-05-15T12:57:55.242387+0000","last_unstale":"2024-05-15T12:58:43.264043+0000","last_undegraded":"2024-05-15T12:58:43.264043+0000","last_fullsized":"2024-05-15T12:58:43.264043+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":4480,"log_dups_size":0,"ondisk_log_size":4480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:20.234355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293601280,"num_objects":4480,"num_object_clones":0,"num_object_copies":13440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4480,"num_write_kb":286720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,10],"acting":[6,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"330'4491","reported_seq":4526,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-05-15T12:58:45.610144+0000","last_change":"2024-05-15T12:57:55.242791+0000","last_active":"2024-05-15T12:58:45.610144+0000","last_peered":"2024-05-15T12:58:45.610144+0000","last_clean":"2024-05-15T12:58:45.610144+0000","last_became_active":"2024-05-15T12:57:55.242453+0000","last_became_peered":"2024-05-15T12:57:55.242453+0000","last_unstale":"2024-05-15T12:58:45.610144+0000","last_undegraded":"2024-05-15T12:58:45.610144+0000","last_fullsized":"2024-05-15T12:58:45.610144+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":4491,"log_dups_size":0,"ondisk_log_size":4491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:11.830376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294322176,"num_objects":4491,"num_object_clones":0,"num_object_copies":13473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4491,"num_write_kb":287424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"330'4584","reported_seq":4620,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-05-15T12:58:45.366381+0000","last_change":"2024-05-15T12:57:55.157512+0000","last_active":"2024-05-15T12:58:45.366381+0000","last_peered":"2024-05-15T12:58:45.366381+0000","last_clean":"2024-05-15T12:58:45.366381+0000","last_became_active":"2024-05-15T12:57:55.157131+0000","last_became_peered":"2024-05-15T12:57:55.157131+0000","last_unstale":"2024-05-15T12:58:45.366381+0000","last_undegraded":"2024-05-15T12:58:45.366381+0000","last_fullsized":"2024-05-15T12:58:45.366381+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":4584,"log_dups_size":0,"ondisk_log_size":4584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:12.418393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300417024,"num_objects":4584,"num_object_clones":0,"num_object_copies":13752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4584,"num_write_kb":293376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"330'4584","reported_seq":4619,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-05-15T12:58:45.616416+0000","last_change":"2024-05-15T12:57:55.158173+0000","last_active":"2024-05-15T12:58:45.616416+0000","last_peered":"2024-05-15T12:58:45.616416+0000","last_clean":"2024-05-15T12:58:45.616416+0000","last_became_active":"2024-05-15T12:57:55.157465+0000","last_became_peered":"2024-05-15T12:57:55.157465+0000","last_unstale":"2024-05-15T12:58:45.616416+0000","last_undegraded":"2024-05-15T12:58:45.616416+0000","last_fullsized":"2024-05-15T12:58:45.616416+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":4584,"log_dups_size":0,"ondisk_log_size":4584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:18.405669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300417024,"num_objects":4584,"num_object_clones":0,"num_object_copies":13752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4584,"num_write_kb":293376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"330'4433","reported_seq":4469,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-05-15T12:58:45.586018+0000","last_change":"2024-05-15T12:57:55.158034+0000","last_active":"2024-05-15T12:58:45.586018+0000","last_peered":"2024-05-15T12:58:45.586018+0000","last_clean":"2024-05-15T12:58:45.586018+0000","last_became_active":"2024-05-15T12:57:55.157716+0000","last_became_peered":"2024-05-15T12:57:55.157716+0000","last_unstale":"2024-05-15T12:58:45.586018+0000","last_undegraded":"2024-05-15T12:58:45.586018+0000","last_fullsized":"2024-05-15T12:58:45.586018+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":4433,"log_dups_size":0,"ondisk_log_size":4433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:05.527560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290521088,"num_objects":4433,"num_object_clones":0,"num_object_copies":13299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4433,"num_write_kb":283712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"329'4337","reported_seq":4371,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-05-15T12:58:42.730387+0000","last_change":"2024-05-15T12:57:55.158066+0000","last_active":"2024-05-15T12:58:42.730387+0000","last_peered":"2024-05-15T12:58:42.730387+0000","last_clean":"2024-05-15T12:58:42.730387+0000","last_became_active":"2024-05-15T12:57:55.157243+0000","last_became_peered":"2024-05-15T12:57:55.157243+0000","last_unstale":"2024-05-15T12:58:42.730387+0000","last_undegraded":"2024-05-15T12:58:42.730387+0000","last_fullsized":"2024-05-15T12:58:42.730387+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":4337,"log_dups_size":0,"ondisk_log_size":4337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:21.941110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284229632,"num_objects":4337,"num_object_clones":0,"num_object_copies":13011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4337,"num_write_kb":277568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,8],"acting":[5,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"329'4305","reported_seq":4339,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-05-15T12:58:42.722589+0000","last_change":"2024-05-15T12:57:55.158553+0000","last_active":"2024-05-15T12:58:42.722589+0000","last_peered":"2024-05-15T12:58:42.722589+0000","last_clean":"2024-05-15T12:58:42.722589+0000","last_became_active":"2024-05-15T12:57:55.158245+0000","last_became_peered":"2024-05-15T12:57:55.158245+0000","last_unstale":"2024-05-15T12:58:42.722589+0000","last_undegraded":"2024-05-15T12:58:42.722589+0000","last_fullsized":"2024-05-15T12:58:42.722589+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:58:55.122035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282132480,"num_objects":4305,"num_object_clones":0,"num_object_copies":12915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4305,"num_write_kb":275520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"329'4259","reported_seq":4293,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-05-15T12:58:41.489356+0000","last_change":"2024-05-15T12:57:55.160182+0000","last_active":"2024-05-15T12:58:41.489356+0000","last_peered":"2024-05-15T12:58:41.489356+0000","last_clean":"2024-05-15T12:58:41.489356+0000","last_became_active":"2024-05-15T12:57:55.159918+0000","last_became_peered":"2024-05-15T12:57:55.159918+0000","last_unstale":"2024-05-15T12:58:41.489356+0000","last_undegraded":"2024-05-15T12:58:41.489356+0000","last_fullsized":"2024-05-15T12:58:41.489356+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:04.512522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279117824,"num_objects":4259,"num_object_clones":0,"num_object_copies":12777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4259,"num_write_kb":272576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"330'4266","reported_seq":4302,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-05-15T12:58:43.263980+0000","last_change":"2024-05-15T12:57:55.243127+0000","last_active":"2024-05-15T12:58:43.263980+0000","last_peered":"2024-05-15T12:58:43.263980+0000","last_clean":"2024-05-15T12:58:43.263980+0000","last_became_active":"2024-05-15T12:57:55.242403+0000","last_became_peered":"2024-05-15T12:57:55.242403+0000","last_unstale":"2024-05-15T12:58:43.263980+0000","last_undegraded":"2024-05-15T12:58:43.263980+0000","last_fullsized":"2024-05-15T12:58:43.263980+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":4266,"log_dups_size":0,"ondisk_log_size":4266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:18.967030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279576576,"num_objects":4266,"num_object_clones":0,"num_object_copies":12798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4266,"num_write_kb":273024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,0],"acting":[6,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":479,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-05-15T12:58:42.909311+0000","last_change":"2024-05-15T12:57:46.629237+0000","last_active":"2024-05-15T12:58:42.909311+0000","last_peered":"2024-05-15T12:58:42.909311+0000","last_clean":"2024-05-15T12:58:42.909311+0000","last_became_active":"2024-05-15T12:56:08.882123+0000","last_became_peered":"2024-05-15T12:56:08.882123+0000","last_unstale":"2024-05-15T12:58:42.909311+0000","last_undegraded":"2024-05-15T12:58:42.909311+0000","last_fullsized":"2024-05-15T12:58:42.909311+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T12:57:46.629099+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T12:57:46.629099+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T12:59:03.416075+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,10],"acting":[6,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":4638572544,"num_objects":70779,"num_object_clones":0,"num_object_copies":212337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70779,"num_write_kb":4529856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13804503040,"data_stored":13804503040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70779,"ondisk_log_size":70779,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":0,"seq":386547056799,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351139,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1553952,"kb_used_data":1333976,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":92191200,"statfs":{"total":95995035648,"available":94403788800,"internally_reserved":0,"allocated":1365991424,"data_stored":1365684357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":225245632},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514665,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1787884,"kb_used_data":1436004,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":91957268,"statfs":{"total":95995035648,"available":94164242432,"internally_reserved":0,"allocated":1470468096,"data_stored":1470156481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":360315328},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874415,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":767976,"kb_used_data":574364,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":92977176,"statfs":{"total":95995035648,"available":95208628224,"internally_reserved":0,"allocated":588148736,"data_stored":587840693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":198244805},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":240518168754,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430391,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2216752,"kb_used_data":1946788,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":91528400,"statfs":{"total":95995035648,"available":93725081600,"internally_reserved":0,"allocated":1993510912,"data_stored":1993199317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":276429254},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575059,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2896680,"kb_used_data":2568412,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":90848472,"statfs":{"total":95995035648,"available":93028835328,"internally_reserved":0,"allocated":2630053888,"data_stored":2629745845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738586,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1707752,"kb_used_data":1397532,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":92037400,"statfs":{"total":95995035648,"available":94246297600,"internally_reserved":0,"allocated":1431072768,"data_stored":1430764725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411535,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1012760,"kb_used_data":828880,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92732392,"statfs":{"total":95995035648,"available":94957969408,"internally_reserved":0,"allocated":848773120,"data_stored":848469625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149896,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1664988,"kb_used_data":1322832,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92080164,"statfs":{"total":95995035648,"available":94290087936,"internally_reserved":0,"allocated":1354579968,"data_stored":1354276473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":350353863},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986372,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1690920,"kb_used_data":1411228,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":92054232,"statfs":{"total":95995035648,"available":94263533568,"internally_reserved":0,"allocated":1445097472,"data_stored":1444789429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":286390725},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692029,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":765784,"kb_used_data":545808,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":92979368,"statfs":{"total":95995035648,"available":95210872832,"internally_reserved":0,"allocated":558907392,"data_stored":558603897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1431764992,"data_stored":1431764992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2653880320,"data_stored":2653880320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":852099072,"data_stored":852099072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1382612992,"data_stored":1382612992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1465122816,"data_stored":1465122816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":566493184,"data_stored":566493184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2004680704,"data_stored":2004680704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":594935808,"data_stored":594935808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1482424320,"data_stored":1482424320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1370488832,"data_stored":1370488832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T12:58:48.143 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-05-15T12:58:48.143 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T12:58:48.144 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-05-15T12:58:48.144 INFO:tasks.thrashosds.thrasher:Setting osd 10 primary_affinity to 0.452084 2024-05-15T12:58:48.144 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd primary-affinity 10 0.45208364879774454 2024-05-15T12:58:48.154 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:47 smithi120 ceph-mon[43961]: from='client.25145 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:58:48.154 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:47 smithi120 ceph-mon[43961]: pgmap v904: 17 pgs: 17 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-05-15T12:58:48.160 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:48.184 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:58:48.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:48.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:58:48.287 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:58:48.345 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:48.446 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T12:58:48.446 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:58:49.133 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T12:58:49.144327+0000 min lat: 0.00111471 max lat: 0.620182 avg lat: 0.00831562 2024-05-15T12:58:49.133 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T12:58:49.133 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 40 16 76621 76605 119.676 60.3125 0.00295014 0.00831562 2024-05-15T12:58:49.205 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-05-15T12:58:49.555 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:58:50.133 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 41 16 77902 77886 118.71 80.0625 0.00742936 0.00842019 2024-05-15T12:58:50.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:49 smithi177 ceph-mon[44303]: pgmap v905: 17 pgs: 17 active+clean; 4.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-05-15T12:58:50.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:49 smithi027 ceph-mon[39795]: pgmap v905: 17 pgs: 17 active+clean; 4.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-05-15T12:58:50.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:49 smithi120 ceph-mon[43961]: pgmap v905: 17 pgs: 17 active+clean; 4.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-05-15T12:58:50.731 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:58:50.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:50.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:58:50.841 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:58:50.871 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:50.896 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:58:50.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:50.900 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:58:51.001 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:58:51.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:50 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:58:51.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:50 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:58:51.031 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:51.054 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:58:51.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:51.060 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:58:51.133 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 42 16 79124 79108 117.702 76.375 0.0127291 0.00849174 2024-05-15T12:58:51.160 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:58:51.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:50 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:58:51.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:50 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:58:51.210 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:58:51.216 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:58:51.222 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:58:51.241 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:51.278 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:58:51.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:50 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:58:51.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:50 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:58:51.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:51.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:58:51.314 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:58:51.337 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:58:51.356 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:58:51.387 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:58:51.410 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:58:51.455 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:51.457 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:58:51.517 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:58:51.559 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:58:51.593 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:58:51.595 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:58:51.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:51.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:58:51.631 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:58:51.669 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:58:51.701 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:58:51.707 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:58:51.716 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:58:51.716 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:58:51.716 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:58:51.716 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:58:51.717 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:58:51.717 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:58:51.717 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:58:51.717 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:58:51.717 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:58:51.717 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:58:51.717 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:58:51.717 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:58:51.738 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:51.763 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:58:51.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:51.768 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:58:51.869 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:58:51.907 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:51.913 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:51 smithi027 ceph-mon[39795]: osdmap e331: 12 total, 12 up, 10 in 2024-05-15T12:58:51.913 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:51 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4236109487' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:58:51.913 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:51 smithi027 ceph-mon[39795]: pgmap v907: 17 pgs: 17 active+clean; 4.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-05-15T12:58:52.133 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 43 16 80240 80224 116.586 69.75 0.00503288 0.00857311 2024-05-15T12:58:52.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:51 smithi177 ceph-mon[44303]: osdmap e331: 12 total, 12 up, 10 in 2024-05-15T12:58:52.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:51 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4236109487' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:58:52.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:51 smithi177 ceph-mon[44303]: pgmap v907: 17 pgs: 17 active+clean; 4.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-05-15T12:58:52.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:51 smithi120 ceph-mon[43961]: osdmap e331: 12 total, 12 up, 10 in 2024-05-15T12:58:52.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:51 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4236109487' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:58:52.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:51 smithi120 ceph-mon[43961]: pgmap v907: 17 pgs: 17 active+clean; 4.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-05-15T12:58:52.660 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:58:52.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:52.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:58:52.766 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:58:52.801 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:52.831 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:58:52.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:52.835 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:58:52.936 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:58:52.984 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:53.133 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 44 16 81595 81579 115.861 84.6875 0.0107926 0.00862697 2024-05-15T12:58:53.194 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:58:53.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:53.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:58:53.298 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:58:53.331 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:53.792 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:58:53.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:53.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:58:53.899 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:58:53.927 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:53.952 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:58:53.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:53.956 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:58:54.058 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:58:54.133 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 45 16 82972 82956 115.199 86.0625 0.00909509 0.00867682 2024-05-15T12:58:54.135 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:54.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:54 smithi177 ceph-mon[44303]: osdmap e332: 12 total, 12 up, 10 in 2024-05-15T12:58:54.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:54 smithi177 ceph-mon[44303]: pgmap v909: 17 pgs: 17 active+clean; 4.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 65 MiB/s wr, 1.05k op/s 2024-05-15T12:58:54.378 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T12:58:54.379 INFO:teuthology.orchestra.run.smithi120.stdout: "size": 20, 2024-05-15T12:58:54.379 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 600, 2024-05-15T12:58:54.379 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [ 2024-05-15T12:58:54.379 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.379 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:37196 3.3 3:c87619e1:::benchmark_data_smithi177_88104_object37195:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-05-15T12:58:54.379 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:53:22.833950+0000", 2024-05-15T12:58:54.379 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 331.55296864000002, 2024-05-15T12:58:54.379 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.20909245800000001, 2024-05-15T12:58:54.379 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:58:54.379 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:54.380 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:58:54.380 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T12:58:54.380 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:58:54.380 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 37196 2024-05-15T12:58:54.380 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.380 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:58:54.380 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.380 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:58:54.380 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.833950+0000", 2024-05-15T12:58:54.380 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.380 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.380 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.380 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:58:54.381 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.833950+0000", 2024-05-15T12:58:54.381 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.381 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.381 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.381 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:58:54.381 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.833951+0000", 2024-05-15T12:58:54.381 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2240000000000001e-06 2024-05-15T12:58:54.381 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.381 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.381 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:58:54.381 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.833995+0000", 2024-05-15T12:58:54.381 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4548000000000001e-05 2024-05-15T12:58:54.381 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.382 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.382 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:58:54.382 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.834006+0000", 2024-05-15T12:58:54.382 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0129000000000001e-05 2024-05-15T12:58:54.382 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.382 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.382 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:58:54.382 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.834014+0000", 2024-05-15T12:58:54.382 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8400000000000001e-06 2024-05-15T12:58:54.382 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.382 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.382 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:58:54.382 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.834389+0000", 2024-05-15T12:58:54.383 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00037509499999999999 2024-05-15T12:58:54.383 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.383 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.383 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:58:54.383 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.834524+0000", 2024-05-15T12:58:54.383 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013477300000000001 2024-05-15T12:58:54.383 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.383 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.383 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,6", 2024-05-15T12:58:54.383 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.834661+0000", 2024-05-15T12:58:54.383 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013650700000000001 2024-05-15T12:58:54.383 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.383 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.383 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:58:54.384 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.838567+0000", 2024-05-15T12:58:54.384 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0039065389999999997 2024-05-15T12:58:54.384 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.384 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.384 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.384 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.842964+0000", 2024-05-15T12:58:54.384 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0043965710000000002 2024-05-15T12:58:54.384 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.384 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.384 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.384 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.042957+0000", 2024-05-15T12:58:54.384 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.19999308199999999 2024-05-15T12:58:54.384 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.385 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.385 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:58:54.385 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.043010+0000", 2024-05-15T12:58:54.385 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3068000000000001e-05 2024-05-15T12:58:54.385 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.385 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.385 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:58:54.385 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.043042+0000", 2024-05-15T12:58:54.385 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.2082000000000002e-05 2024-05-15T12:58:54.385 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.385 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:58:54.385 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.385 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.385 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.386 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:37198 3.3 3:cebe8428:::benchmark_data_smithi177_88104_object37197:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-05-15T12:58:54.386 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:53:22.834037+0000", 2024-05-15T12:58:54.386 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 331.552881652, 2024-05-15T12:58:54.386 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.209117732, 2024-05-15T12:58:54.386 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:58:54.386 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:54.386 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:58:54.386 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T12:58:54.386 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:58:54.386 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 37198 2024-05-15T12:58:54.386 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.386 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:58:54.386 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.387 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:58:54.387 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.834037+0000", 2024-05-15T12:58:54.387 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.387 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.387 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.387 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:58:54.387 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.834037+0000", 2024-05-15T12:58:54.387 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.387 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.387 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.387 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:58:54.387 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.834037+0000", 2024-05-15T12:58:54.387 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3300000000000002e-07 2024-05-15T12:58:54.388 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.388 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.388 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:58:54.388 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.834068+0000", 2024-05-15T12:58:54.388 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.0781999999999998e-05 2024-05-15T12:58:54.388 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.388 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.388 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:58:54.388 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.834075+0000", 2024-05-15T12:58:54.388 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2779999999999996e-06 2024-05-15T12:58:54.388 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.388 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.388 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:58:54.388 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.834081+0000", 2024-05-15T12:58:54.388 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.4530000000000001e-06 2024-05-15T12:58:54.389 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.389 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.389 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:58:54.389 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.835992+0000", 2024-05-15T12:58:54.389 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001910796 2024-05-15T12:58:54.389 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.389 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.389 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:58:54.389 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.836107+0000", 2024-05-15T12:58:54.389 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000115788 2024-05-15T12:58:54.389 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.389 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.389 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,6", 2024-05-15T12:58:54.389 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.836232+0000", 2024-05-15T12:58:54.390 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012419500000000001 2024-05-15T12:58:54.390 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.390 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.390 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:58:54.390 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.838573+0000", 2024-05-15T12:58:54.390 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0023413010000000001 2024-05-15T12:58:54.390 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.390 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.390 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.390 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.842983+0000", 2024-05-15T12:58:54.390 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0044104959999999999 2024-05-15T12:58:54.390 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.390 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.390 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.391 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.043089+0000", 2024-05-15T12:58:54.391 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.20010586399999999 2024-05-15T12:58:54.391 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.391 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.391 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:58:54.391 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.043126+0000", 2024-05-15T12:58:54.391 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7237999999999998e-05 2024-05-15T12:58:54.391 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.391 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.391 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:58:54.391 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.043154+0000", 2024-05-15T12:58:54.391 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.7908000000000001e-05 2024-05-15T12:58:54.391 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.391 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:58:54.392 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.392 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.392 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.392 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:37204 3.e 3:7a1d919b:::benchmark_data_smithi177_88104_object37203:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-05-15T12:58:54.392 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:53:22.836648+0000", 2024-05-15T12:58:54.392 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 331.55027074100002, 2024-05-15T12:58:54.392 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.21055301500000001, 2024-05-15T12:58:54.392 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:58:54.392 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:54.392 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:58:54.392 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T12:58:54.392 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:58:54.392 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 37204 2024-05-15T12:58:54.392 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.393 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:58:54.393 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.393 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:58:54.393 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.836648+0000", 2024-05-15T12:58:54.393 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.393 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.393 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.393 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:58:54.393 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.836648+0000", 2024-05-15T12:58:54.393 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.393 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.393 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.393 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:58:54.393 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.836649+0000", 2024-05-15T12:58:54.394 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.705e-06 2024-05-15T12:58:54.394 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.394 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.394 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:58:54.394 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.836696+0000", 2024-05-15T12:58:54.394 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6588999999999997e-05 2024-05-15T12:58:54.394 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.394 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.394 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:58:54.394 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.836708+0000", 2024-05-15T12:58:54.394 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2349e-05 2024-05-15T12:58:54.394 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.394 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.394 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:58:54.395 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.836719+0000", 2024-05-15T12:58:54.395 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0804e-05 2024-05-15T12:58:54.395 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.395 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.395 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:58:54.395 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.836847+0000", 2024-05-15T12:58:54.395 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000128188 2024-05-15T12:58:54.395 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.395 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.395 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:58:54.395 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.836989+0000", 2024-05-15T12:58:54.395 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014147199999999999 2024-05-15T12:58:54.395 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.395 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.396 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,9", 2024-05-15T12:58:54.396 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.837116+0000", 2024-05-15T12:58:54.396 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012755399999999999 2024-05-15T12:58:54.396 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.396 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.396 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:58:54.396 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.839467+0000", 2024-05-15T12:58:54.396 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0023508890000000001 2024-05-15T12:58:54.396 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.396 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.396 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.396 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.841466+0000", 2024-05-15T12:58:54.396 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0019989130000000002 2024-05-15T12:58:54.396 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.397 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.397 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.397 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.047109+0000", 2024-05-15T12:58:54.397 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.205643404 2024-05-15T12:58:54.397 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.397 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.397 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:58:54.397 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.047167+0000", 2024-05-15T12:58:54.397 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7773999999999999e-05 2024-05-15T12:58:54.397 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.397 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.397 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:58:54.397 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.047201+0000", 2024-05-15T12:58:54.398 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.3374000000000003e-05 2024-05-15T12:58:54.398 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.398 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:58:54.398 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.398 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.398 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.398 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:37205 3.3 3:c446999b:::benchmark_data_smithi177_88104_object37204:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-05-15T12:58:54.398 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:53:22.836765+0000", 2024-05-15T12:58:54.398 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 331.550153287, 2024-05-15T12:58:54.398 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.206496607, 2024-05-15T12:58:54.398 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:58:54.398 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:54.398 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:58:54.398 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T12:58:54.399 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:58:54.399 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 37205 2024-05-15T12:58:54.399 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.399 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:58:54.399 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.399 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:58:54.399 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.836765+0000", 2024-05-15T12:58:54.399 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.399 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.399 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.399 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:58:54.399 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.836765+0000", 2024-05-15T12:58:54.399 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.400 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.400 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.400 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:58:54.400 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.836765+0000", 2024-05-15T12:58:54.400 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9100000000000004e-07 2024-05-15T12:58:54.400 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.400 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.400 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:58:54.400 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.836834+0000", 2024-05-15T12:58:54.400 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8535999999999996e-05 2024-05-15T12:58:54.400 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.400 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.400 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:58:54.400 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.836843+0000", 2024-05-15T12:58:54.401 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1900000000000001e-06 2024-05-15T12:58:54.401 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.401 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.401 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:58:54.401 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.836850+0000", 2024-05-15T12:58:54.401 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.272e-06 2024-05-15T12:58:54.401 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.401 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.401 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:58:54.401 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.837336+0000", 2024-05-15T12:58:54.401 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000485673 2024-05-15T12:58:54.401 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.401 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.401 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:58:54.402 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.837436+0000", 2024-05-15T12:58:54.402 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00010008100000000001 2024-05-15T12:58:54.402 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.402 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.402 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,6", 2024-05-15T12:58:54.402 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.837565+0000", 2024-05-15T12:58:54.402 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012830099999999999 2024-05-15T12:58:54.402 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.402 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.402 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:58:54.402 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.839407+0000", 2024-05-15T12:58:54.402 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0018427000000000001 2024-05-15T12:58:54.402 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.403 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.403 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.403 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.843002+0000", 2024-05-15T12:58:54.403 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0035952240000000002 2024-05-15T12:58:54.403 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.403 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.403 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.403 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.043183+0000", 2024-05-15T12:58:54.403 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.200180947 2024-05-15T12:58:54.403 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.403 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.403 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:58:54.403 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.043228+0000", 2024-05-15T12:58:54.403 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4969000000000003e-05 2024-05-15T12:58:54.404 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.404 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.404 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:58:54.404 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.043262+0000", 2024-05-15T12:58:54.404 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.3222999999999997e-05 2024-05-15T12:58:54.404 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.404 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:58:54.404 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.404 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.404 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.404 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:37209 3.e 3:7e724444:::benchmark_data_smithi177_88104_object37208:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-05-15T12:58:54.404 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:53:22.839749+0000", 2024-05-15T12:58:54.404 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 331.54716968299999, 2024-05-15T12:58:54.404 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.20756075600000001, 2024-05-15T12:58:54.405 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:58:54.405 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:54.405 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:58:54.405 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T12:58:54.405 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:58:54.405 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 37209 2024-05-15T12:58:54.405 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.405 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:58:54.405 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.405 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:58:54.405 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.839749+0000", 2024-05-15T12:58:54.405 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.405 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.405 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.405 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:58:54.406 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.839749+0000", 2024-05-15T12:58:54.406 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.406 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.406 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.406 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:58:54.406 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.839750+0000", 2024-05-15T12:58:54.406 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3030000000000001e-06 2024-05-15T12:58:54.406 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.406 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.406 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:58:54.406 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.839785+0000", 2024-05-15T12:58:54.406 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5136999999999999e-05 2024-05-15T12:58:54.406 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.407 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.407 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:58:54.407 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.839802+0000", 2024-05-15T12:58:54.407 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7144000000000001e-05 2024-05-15T12:58:54.407 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.407 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.407 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:58:54.407 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.839813+0000", 2024-05-15T12:58:54.407 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0468999999999999e-05 2024-05-15T12:58:54.407 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.407 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.407 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:58:54.407 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.839878+0000", 2024-05-15T12:58:54.407 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.5563999999999998e-05 2024-05-15T12:58:54.408 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.408 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.408 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:58:54.408 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.840000+0000", 2024-05-15T12:58:54.408 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000121839 2024-05-15T12:58:54.408 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.408 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.408 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,9", 2024-05-15T12:58:54.408 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.840136+0000", 2024-05-15T12:58:54.408 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013551599999999999 2024-05-15T12:58:54.408 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.408 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.408 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:58:54.409 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.841927+0000", 2024-05-15T12:58:54.409 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0017918260000000001 2024-05-15T12:58:54.409 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.409 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.409 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.409 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.842164+0000", 2024-05-15T12:58:54.409 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00023689100000000001 2024-05-15T12:58:54.409 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.409 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.409 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.409 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.047230+0000", 2024-05-15T12:58:54.409 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.20506569699999999 2024-05-15T12:58:54.409 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.409 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.410 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:58:54.410 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.047268+0000", 2024-05-15T12:58:54.410 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.8383999999999997e-05 2024-05-15T12:58:54.410 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.410 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.410 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:58:54.410 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.047309+0000", 2024-05-15T12:58:54.410 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0985999999999997e-05 2024-05-15T12:58:54.410 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.410 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:58:54.410 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.410 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.410 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.410 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:37211 3.3 3:c098b45d:::benchmark_data_smithi177_88104_object37210:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-05-15T12:58:54.411 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:53:22.839880+0000", 2024-05-15T12:58:54.411 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 331.54703850999999, 2024-05-15T12:58:54.411 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.20742344500000001, 2024-05-15T12:58:54.411 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:58:54.411 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:54.411 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:58:54.411 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T12:58:54.411 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:58:54.411 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 37211 2024-05-15T12:58:54.411 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.411 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:58:54.411 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.411 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:58:54.411 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.839880+0000", 2024-05-15T12:58:54.412 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.412 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.412 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.412 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:58:54.412 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.839880+0000", 2024-05-15T12:58:54.412 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.412 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.412 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.412 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:58:54.412 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.839881+0000", 2024-05-15T12:58:54.412 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.5099999999999999e-07 2024-05-15T12:58:54.412 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.412 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.412 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:58:54.413 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.839996+0000", 2024-05-15T12:58:54.413 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000115196 2024-05-15T12:58:54.413 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.413 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.413 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:58:54.413 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.840004+0000", 2024-05-15T12:58:54.413 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9449999999999993e-06 2024-05-15T12:58:54.413 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.413 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.413 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:58:54.413 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.840009+0000", 2024-05-15T12:58:54.413 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0259999999999998e-06 2024-05-15T12:58:54.413 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.413 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.413 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:58:54.414 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.840060+0000", 2024-05-15T12:58:54.414 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0974999999999999e-05 2024-05-15T12:58:54.414 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.414 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.414 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:58:54.414 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.840177+0000", 2024-05-15T12:58:54.414 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00011702899999999999 2024-05-15T12:58:54.414 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.414 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.414 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,6", 2024-05-15T12:58:54.414 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.840296+0000", 2024-05-15T12:58:54.414 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000119109 2024-05-15T12:58:54.414 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.414 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.415 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:58:54.415 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.841706+0000", 2024-05-15T12:58:54.415 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0014100460000000001 2024-05-15T12:58:54.415 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.415 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.415 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.415 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.844278+0000", 2024-05-15T12:58:54.415 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.002572399 2024-05-15T12:58:54.415 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.415 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.415 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.415 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.047205+0000", 2024-05-15T12:58:54.415 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.20292704 2024-05-15T12:58:54.415 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.416 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.416 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:58:54.416 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.047263+0000", 2024-05-15T12:58:54.416 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7840000000000002e-05 2024-05-15T12:58:54.416 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.416 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.416 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:58:54.416 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.047303+0000", 2024-05-15T12:58:54.416 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0089000000000002e-05 2024-05-15T12:58:54.416 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.416 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:58:54.416 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.416 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.416 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.417 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:37222 3.e 3:7848fe48:::benchmark_data_smithi177_88104_object37221:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-05-15T12:58:54.417 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:53:22.847087+0000", 2024-05-15T12:58:54.417 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 331.53983113800001, 2024-05-15T12:58:54.417 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.200364338, 2024-05-15T12:58:54.417 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:58:54.417 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:54.417 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:58:54.417 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T12:58:54.417 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:58:54.417 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 37222 2024-05-15T12:58:54.417 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.417 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:58:54.417 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.417 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:58:54.417 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.847087+0000", 2024-05-15T12:58:54.418 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.418 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.418 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.418 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:58:54.418 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.847087+0000", 2024-05-15T12:58:54.418 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.418 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.418 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.418 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:58:54.418 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.847089+0000", 2024-05-15T12:58:54.418 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6360000000000001e-06 2024-05-15T12:58:54.418 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.418 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.418 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:58:54.419 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.847153+0000", 2024-05-15T12:58:54.419 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4455999999999994e-05 2024-05-15T12:58:54.419 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.419 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.419 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:58:54.419 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.847168+0000", 2024-05-15T12:58:54.419 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4731e-05 2024-05-15T12:58:54.419 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.419 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.419 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:58:54.419 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.847176+0000", 2024-05-15T12:58:54.419 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.4370000000000007e-06 2024-05-15T12:58:54.419 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.419 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.420 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:58:54.420 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.847232+0000", 2024-05-15T12:58:54.420 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5408000000000002e-05 2024-05-15T12:58:54.420 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.420 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.420 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:58:54.420 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.847368+0000", 2024-05-15T12:58:54.420 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000135916 2024-05-15T12:58:54.420 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.420 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.420 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,9", 2024-05-15T12:58:54.420 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.847511+0000", 2024-05-15T12:58:54.420 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014329300000000001 2024-05-15T12:58:54.421 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.421 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.421 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:58:54.421 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.848807+0000", 2024-05-15T12:58:54.421 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0012961400000000001 2024-05-15T12:58:54.421 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.421 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.421 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.421 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.848899+0000", 2024-05-15T12:58:54.421 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.2287000000000006e-05 2024-05-15T12:58:54.421 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.421 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.421 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.421 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.047357+0000", 2024-05-15T12:58:54.422 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.198457309 2024-05-15T12:58:54.422 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.422 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.422 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:58:54.422 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.047414+0000", 2024-05-15T12:58:54.422 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7543000000000001e-05 2024-05-15T12:58:54.422 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.422 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.422 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:58:54.422 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.047451+0000", 2024-05-15T12:58:54.422 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7181999999999997e-05 2024-05-15T12:58:54.422 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.422 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:58:54.423 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.423 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.423 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.423 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:37227 3.3 3:cc55a26a:::benchmark_data_smithi177_88104_object37226:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-05-15T12:58:54.423 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:53:22.851948+0000", 2024-05-15T12:58:54.423 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 331.53497075500002, 2024-05-15T12:58:54.423 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.19550398899999999, 2024-05-15T12:58:54.423 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:58:54.423 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:54.423 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:58:54.423 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T12:58:54.423 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:58:54.423 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 37227 2024-05-15T12:58:54.423 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.423 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:58:54.424 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.424 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:58:54.424 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.851948+0000", 2024-05-15T12:58:54.424 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.424 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.424 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.424 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:58:54.424 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.851948+0000", 2024-05-15T12:58:54.424 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.424 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.424 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.424 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:58:54.424 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.851949+0000", 2024-05-15T12:58:54.425 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1370000000000001e-06 2024-05-15T12:58:54.425 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.425 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.425 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:58:54.425 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.852013+0000", 2024-05-15T12:58:54.425 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4277999999999997e-05 2024-05-15T12:58:54.425 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.425 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.425 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:58:54.425 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.852023+0000", 2024-05-15T12:58:54.425 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0497e-05 2024-05-15T12:58:54.425 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.425 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.425 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:58:54.426 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.852029+0000", 2024-05-15T12:58:54.426 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9830000000000001e-06 2024-05-15T12:58:54.426 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.426 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.426 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:58:54.426 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.852083+0000", 2024-05-15T12:58:54.426 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3529000000000002e-05 2024-05-15T12:58:54.426 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.426 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.426 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:58:54.426 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.852188+0000", 2024-05-15T12:58:54.426 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000104796 2024-05-15T12:58:54.426 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.426 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.427 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,6", 2024-05-15T12:58:54.427 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.852278+0000", 2024-05-15T12:58:54.427 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0282000000000001e-05 2024-05-15T12:58:54.427 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.427 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.427 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:58:54.427 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.853377+0000", 2024-05-15T12:58:54.427 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0010989769999999999 2024-05-15T12:58:54.427 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.427 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.427 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.427 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.853642+0000", 2024-05-15T12:58:54.427 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00026526399999999997 2024-05-15T12:58:54.427 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.428 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.428 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.428 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.047364+0000", 2024-05-15T12:58:54.428 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.19372135300000001 2024-05-15T12:58:54.428 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.428 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.428 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:58:54.428 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.047415+0000", 2024-05-15T12:58:54.428 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1357999999999999e-05 2024-05-15T12:58:54.428 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.428 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.428 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:58:54.428 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.047451+0000", 2024-05-15T12:58:54.428 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.6535000000000002e-05 2024-05-15T12:58:54.428 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.429 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:58:54.429 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.429 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.429 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.429 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:37229 3.e 3:78549371:::benchmark_data_smithi177_88104_object37228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-05-15T12:58:54.429 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:53:22.855770+0000", 2024-05-15T12:58:54.429 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 331.53114873099997, 2024-05-15T12:58:54.429 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.19330467800000001, 2024-05-15T12:58:54.429 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:58:54.429 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:54.429 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:58:54.429 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T12:58:54.429 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:58:54.429 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 37229 2024-05-15T12:58:54.429 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.430 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:58:54.430 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.430 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:58:54.430 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.855770+0000", 2024-05-15T12:58:54.430 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.430 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.430 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.430 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:58:54.430 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.855770+0000", 2024-05-15T12:58:54.430 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.430 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.430 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.430 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:58:54.430 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.855770+0000", 2024-05-15T12:58:54.431 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1399999999999995e-07 2024-05-15T12:58:54.431 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.431 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.431 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:58:54.431 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.855803+0000", 2024-05-15T12:58:54.431 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.2237999999999999e-05 2024-05-15T12:58:54.431 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.431 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.431 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:58:54.431 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.855813+0000", 2024-05-15T12:58:54.431 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0803e-05 2024-05-15T12:58:54.431 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.431 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.431 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:58:54.431 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.855819+0000", 2024-05-15T12:58:54.432 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.4870000000000002e-06 2024-05-15T12:58:54.432 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.432 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.432 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:58:54.432 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.855864+0000", 2024-05-15T12:58:54.432 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4876999999999997e-05 2024-05-15T12:58:54.432 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.432 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.432 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:58:54.432 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.855952+0000", 2024-05-15T12:58:54.432 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7972000000000004e-05 2024-05-15T12:58:54.432 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.432 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.432 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,9", 2024-05-15T12:58:54.432 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.856064+0000", 2024-05-15T12:58:54.433 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000112038 2024-05-15T12:58:54.433 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.433 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.433 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:58:54.433 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.857164+0000", 2024-05-15T12:58:54.433 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0011000000000000001 2024-05-15T12:58:54.433 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.433 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.433 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.433 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:22.857381+0000", 2024-05-15T12:58:54.433 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00021671599999999999 2024-05-15T12:58:54.433 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.433 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.434 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.434 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.048974+0000", 2024-05-15T12:58:54.434 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.191593403 2024-05-15T12:58:54.434 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.434 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.434 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:58:54.434 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.049031+0000", 2024-05-15T12:58:54.434 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7361999999999997e-05 2024-05-15T12:58:54.434 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.434 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.434 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:58:54.434 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.049074+0000", 2024-05-15T12:58:54.434 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2867999999999997e-05 2024-05-15T12:58:54.434 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.435 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:58:54.435 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.435 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.435 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.435 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:38507 3.e 3:72b0fd85:::benchmark_data_smithi177_88104_object38506:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-05-15T12:58:54.435 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:53:23.744369+0000", 2024-05-15T12:58:54.435 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 330.64254957899999, 2024-05-15T12:58:54.435 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.20654434399999999, 2024-05-15T12:58:54.435 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:58:54.435 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:54.435 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:58:54.435 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T12:58:54.435 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:58:54.435 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 38507 2024-05-15T12:58:54.435 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.436 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:58:54.436 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.436 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:58:54.436 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.744369+0000", 2024-05-15T12:58:54.436 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.436 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.436 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.436 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:58:54.436 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.744369+0000", 2024-05-15T12:58:54.436 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.436 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.436 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.436 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:58:54.436 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.744370+0000", 2024-05-15T12:58:54.437 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.19e-06 2024-05-15T12:58:54.437 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.437 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.437 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:58:54.437 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.744403+0000", 2024-05-15T12:58:54.437 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.2978000000000002e-05 2024-05-15T12:58:54.437 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.437 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.437 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:58:54.437 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.744415+0000", 2024-05-15T12:58:54.437 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1780000000000001e-05 2024-05-15T12:58:54.437 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.437 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.437 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:58:54.438 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.744421+0000", 2024-05-15T12:58:54.438 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8580000000000002e-06 2024-05-15T12:58:54.438 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.438 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.438 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:58:54.438 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.744771+0000", 2024-05-15T12:58:54.438 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00034953099999999998 2024-05-15T12:58:54.438 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.438 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.438 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:58:54.438 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.744888+0000", 2024-05-15T12:58:54.438 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000116663 2024-05-15T12:58:54.438 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.439 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.439 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,9", 2024-05-15T12:58:54.439 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.745014+0000", 2024-05-15T12:58:54.439 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012643600000000001 2024-05-15T12:58:54.439 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.439 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.439 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.439 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.748394+0000", 2024-05-15T12:58:54.439 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0033801690000000001 2024-05-15T12:58:54.439 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.439 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.439 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:58:54.439 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.750131+0000", 2024-05-15T12:58:54.439 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0017368679999999999 2024-05-15T12:58:54.440 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.440 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.440 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.440 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.950826+0000", 2024-05-15T12:58:54.440 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.20069492999999999 2024-05-15T12:58:54.440 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.440 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.440 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:58:54.440 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.950864+0000", 2024-05-15T12:58:54.440 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7811000000000001e-05 2024-05-15T12:58:54.440 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.440 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.440 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:58:54.440 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.950913+0000", 2024-05-15T12:58:54.441 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9129999999999999e-05 2024-05-15T12:58:54.441 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.441 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:58:54.441 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.441 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.441 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.441 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:38528 3.e 3:76984a6d:::benchmark_data_smithi177_88104_object38527:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-05-15T12:58:54.441 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:53:23.753560+0000", 2024-05-15T12:58:54.441 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 330.63335829599998, 2024-05-15T12:58:54.441 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.198543895, 2024-05-15T12:58:54.441 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:58:54.441 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:54.441 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:58:54.441 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T12:58:54.441 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:58:54.442 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 38528 2024-05-15T12:58:54.442 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.442 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:58:54.442 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.442 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:58:54.442 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.753560+0000", 2024-05-15T12:58:54.442 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.442 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.442 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.442 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:58:54.442 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.753560+0000", 2024-05-15T12:58:54.442 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.442 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.442 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.443 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:58:54.443 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.753561+0000", 2024-05-15T12:58:54.443 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.359e-06 2024-05-15T12:58:54.443 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.443 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.443 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:58:54.443 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.753644+0000", 2024-05-15T12:58:54.443 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.3166999999999998e-05 2024-05-15T12:58:54.443 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.443 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.443 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:58:54.443 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.753653+0000", 2024-05-15T12:58:54.443 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7860000000000002e-06 2024-05-15T12:58:54.443 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.443 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.444 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:58:54.444 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.753659+0000", 2024-05-15T12:58:54.444 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2689999999999999e-06 2024-05-15T12:58:54.444 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.444 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.444 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:58:54.444 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.753698+0000", 2024-05-15T12:58:54.444 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.9140999999999997e-05 2024-05-15T12:58:54.444 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.444 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.444 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:58:54.444 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.753824+0000", 2024-05-15T12:58:54.444 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012633300000000001 2024-05-15T12:58:54.444 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.445 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.445 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,9", 2024-05-15T12:58:54.445 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.753969+0000", 2024-05-15T12:58:54.445 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000145224 2024-05-15T12:58:54.445 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.445 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.445 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.445 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.757896+0000", 2024-05-15T12:58:54.445 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.003926461 2024-05-15T12:58:54.445 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.445 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.445 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:58:54.445 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.757908+0000", 2024-05-15T12:58:54.445 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1892999999999999e-05 2024-05-15T12:58:54.445 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.446 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.446 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.446 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.952004+0000", 2024-05-15T12:58:54.446 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.194096034 2024-05-15T12:58:54.446 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.446 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.446 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:58:54.446 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.952060+0000", 2024-05-15T12:58:54.446 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6827000000000003e-05 2024-05-15T12:58:54.446 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.446 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.446 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:58:54.447 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.952104+0000", 2024-05-15T12:58:54.447 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3401e-05 2024-05-15T12:58:54.447 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.447 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:58:54.447 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.447 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.447 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.447 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:38531 3.e 3:75e48f60:::benchmark_data_smithi177_88104_object38530:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-05-15T12:58:54.447 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:53:23.754756+0000", 2024-05-15T12:58:54.447 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 330.632161887, 2024-05-15T12:58:54.447 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.198476027, 2024-05-15T12:58:54.447 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:58:54.447 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:54.447 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:58:54.447 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T12:58:54.448 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:58:54.448 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 38531 2024-05-15T12:58:54.448 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.448 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:58:54.448 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.448 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:58:54.448 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.754756+0000", 2024-05-15T12:58:54.448 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.448 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.448 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.448 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:58:54.448 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.754756+0000", 2024-05-15T12:58:54.448 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.448 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.449 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.449 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:58:54.449 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.754757+0000", 2024-05-15T12:58:54.449 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1000000000000001e-06 2024-05-15T12:58:54.449 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.449 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.449 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:58:54.449 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.754781+0000", 2024-05-15T12:58:54.449 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.3745999999999999e-05 2024-05-15T12:58:54.449 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.449 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.449 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:58:54.449 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.754792+0000", 2024-05-15T12:58:54.449 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0822000000000001e-05 2024-05-15T12:58:54.450 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.450 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.450 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:58:54.450 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.754800+0000", 2024-05-15T12:58:54.450 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7950000000000008e-06 2024-05-15T12:58:54.450 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.450 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.450 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:58:54.450 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.756275+0000", 2024-05-15T12:58:54.450 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0014756330000000001 2024-05-15T12:58:54.450 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.450 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.450 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:58:54.450 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.756392+0000", 2024-05-15T12:58:54.451 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00011675699999999999 2024-05-15T12:58:54.451 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.451 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.451 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,9", 2024-05-15T12:58:54.451 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.756523+0000", 2024-05-15T12:58:54.451 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013040999999999999 2024-05-15T12:58:54.451 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.451 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.451 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:58:54.451 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.758764+0000", 2024-05-15T12:58:54.451 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.002241332 2024-05-15T12:58:54.451 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.451 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.451 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.451 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.759999+0000", 2024-05-15T12:58:54.452 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0012353430000000001 2024-05-15T12:58:54.452 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.452 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.452 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.452 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.953173+0000", 2024-05-15T12:58:54.452 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.19317409999999999 2024-05-15T12:58:54.452 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.452 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.452 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:58:54.452 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.953207+0000", 2024-05-15T12:58:54.452 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.3235000000000003e-05 2024-05-15T12:58:54.452 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.452 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.452 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:58:54.452 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.953232+0000", 2024-05-15T12:58:54.453 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.5754000000000001e-05 2024-05-15T12:58:54.453 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.453 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:58:54.453 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.453 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.453 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.453 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:38544 3.e 3:7a5f9f57:::benchmark_data_smithi177_88104_object38543:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-05-15T12:58:54.453 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:53:23.759919+0000", 2024-05-15T12:58:54.453 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 330.62699894899998, 2024-05-15T12:58:54.453 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.19343480499999999, 2024-05-15T12:58:54.453 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:58:54.453 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:54.453 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:58:54.453 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T12:58:54.454 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:58:54.454 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 38544 2024-05-15T12:58:54.454 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.454 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:58:54.454 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.454 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:58:54.454 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.759919+0000", 2024-05-15T12:58:54.454 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.454 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.454 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.454 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:58:54.454 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.759919+0000", 2024-05-15T12:58:54.454 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.454 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.455 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.455 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:58:54.455 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.759921+0000", 2024-05-15T12:58:54.455 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5519999999999999e-06 2024-05-15T12:58:54.455 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.455 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.455 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:58:54.455 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.759997+0000", 2024-05-15T12:58:54.455 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.5838000000000001e-05 2024-05-15T12:58:54.455 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.455 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.455 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:58:54.455 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.760008+0000", 2024-05-15T12:58:54.455 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1429e-05 2024-05-15T12:58:54.455 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.455 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.456 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:58:54.456 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.760016+0000", 2024-05-15T12:58:54.456 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3880000000000002e-06 2024-05-15T12:58:54.456 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.456 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.456 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:58:54.456 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.760774+0000", 2024-05-15T12:58:54.456 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00075851399999999995 2024-05-15T12:58:54.456 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.456 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.456 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:58:54.456 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.760900+0000", 2024-05-15T12:58:54.456 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000126024 2024-05-15T12:58:54.456 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.456 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.456 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,9", 2024-05-15T12:58:54.457 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.761032+0000", 2024-05-15T12:58:54.457 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000132297 2024-05-15T12:58:54.457 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.457 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.457 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:58:54.457 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.763888+0000", 2024-05-15T12:58:54.457 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0028555059999999998 2024-05-15T12:58:54.457 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.457 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.457 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.457 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.766468+0000", 2024-05-15T12:58:54.457 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0025797049999999998 2024-05-15T12:58:54.457 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.457 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.457 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.457 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.953271+0000", 2024-05-15T12:58:54.458 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.18680387900000001 2024-05-15T12:58:54.458 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.458 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.458 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:58:54.458 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.953321+0000", 2024-05-15T12:58:54.458 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9583999999999998e-05 2024-05-15T12:58:54.458 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.458 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.458 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:58:54.458 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.953354+0000", 2024-05-15T12:58:54.458 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.3089000000000001e-05 2024-05-15T12:58:54.458 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.458 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:58:54.458 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.458 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.459 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.459 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:38549 3.e 3:739679a1:::benchmark_data_smithi177_88104_object38548:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-05-15T12:58:54.459 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:53:23.761422+0000", 2024-05-15T12:58:54.459 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 330.62549603999997, 2024-05-15T12:58:54.459 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.19315933800000001, 2024-05-15T12:58:54.459 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:58:54.459 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:54.459 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:58:54.459 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T12:58:54.459 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:58:54.459 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 38549 2024-05-15T12:58:54.459 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.459 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:58:54.459 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.459 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:58:54.460 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.761422+0000", 2024-05-15T12:58:54.460 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.460 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.460 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.460 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:58:54.460 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.761422+0000", 2024-05-15T12:58:54.460 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.460 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.460 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.460 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:58:54.460 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.761424+0000", 2024-05-15T12:58:54.460 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.328e-06 2024-05-15T12:58:54.460 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.460 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.460 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:58:54.461 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.761490+0000", 2024-05-15T12:58:54.461 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6911999999999999e-05 2024-05-15T12:58:54.461 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.461 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.461 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:58:54.461 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.761501+0000", 2024-05-15T12:58:54.461 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0407999999999999e-05 2024-05-15T12:58:54.461 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.461 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.461 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:58:54.461 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.761508+0000", 2024-05-15T12:58:54.461 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4080000000000001e-06 2024-05-15T12:58:54.461 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.461 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.461 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:58:54.461 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.762469+0000", 2024-05-15T12:58:54.462 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00096047899999999998 2024-05-15T12:58:54.462 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.462 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.462 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:58:54.462 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.762585+0000", 2024-05-15T12:58:54.462 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00011637999999999999 2024-05-15T12:58:54.462 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.462 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.462 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,9", 2024-05-15T12:58:54.462 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.762689+0000", 2024-05-15T12:58:54.462 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000104314 2024-05-15T12:58:54.462 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.462 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.462 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:58:54.463 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.764202+0000", 2024-05-15T12:58:54.463 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00151304 2024-05-15T12:58:54.463 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.463 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.463 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.463 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.766903+0000", 2024-05-15T12:58:54.463 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0027001350000000002 2024-05-15T12:58:54.463 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.463 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.463 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.463 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.954491+0000", 2024-05-15T12:58:54.463 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.18758851400000001 2024-05-15T12:58:54.463 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.463 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.464 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:58:54.464 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.954544+0000", 2024-05-15T12:58:54.464 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2426999999999998e-05 2024-05-15T12:58:54.464 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.464 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.464 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:58:54.464 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:53:23.954582+0000", 2024-05-15T12:58:54.464 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7993e-05 2024-05-15T12:58:54.464 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.464 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:58:54.464 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.464 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.464 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.464 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:115932 3.3 3:c02471c3:::benchmark_data_smithi177_88104_object115931:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-05-15T12:58:54.465 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:54:20.949443+0000", 2024-05-15T12:58:54.465 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 273.43747490700002, 2024-05-15T12:58:54.465 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.20726293900000001, 2024-05-15T12:58:54.465 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:58:54.465 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:54.465 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:58:54.465 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T12:58:54.465 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:58:54.465 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 115932 2024-05-15T12:58:54.465 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.465 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:58:54.465 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.465 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:58:54.465 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.949443+0000", 2024-05-15T12:58:54.465 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.466 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.466 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.466 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:58:54.466 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.949443+0000", 2024-05-15T12:58:54.466 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.466 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.466 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.466 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:58:54.466 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.949445+0000", 2024-05-15T12:58:54.466 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4649999999999999e-06 2024-05-15T12:58:54.466 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.466 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.466 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:58:54.466 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.949479+0000", 2024-05-15T12:58:54.467 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.4536000000000003e-05 2024-05-15T12:58:54.467 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.467 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.467 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:58:54.467 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.949493+0000", 2024-05-15T12:58:54.467 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3186999999999999e-05 2024-05-15T12:58:54.467 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.467 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.467 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:58:54.467 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.949500+0000", 2024-05-15T12:58:54.467 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7589999999999997e-06 2024-05-15T12:58:54.467 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.467 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.467 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:58:54.467 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.949582+0000", 2024-05-15T12:58:54.468 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1532000000000005e-05 2024-05-15T12:58:54.468 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.468 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.468 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:58:54.468 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.949717+0000", 2024-05-15T12:58:54.468 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013520399999999999 2024-05-15T12:58:54.468 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.468 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.468 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,6", 2024-05-15T12:58:54.468 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.949875+0000", 2024-05-15T12:58:54.468 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000158095 2024-05-15T12:58:54.468 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.468 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.468 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:58:54.469 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.953760+0000", 2024-05-15T12:58:54.469 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.003884387 2024-05-15T12:58:54.469 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.469 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.469 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.469 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.959907+0000", 2024-05-15T12:58:54.469 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0061473710000000004 2024-05-15T12:58:54.469 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.469 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.469 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.469 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:21.156603+0000", 2024-05-15T12:58:54.469 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.196695915 2024-05-15T12:58:54.469 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.469 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.469 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:58:54.470 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:21.156661+0000", 2024-05-15T12:58:54.470 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8357999999999999e-05 2024-05-15T12:58:54.470 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.470 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.470 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:58:54.470 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:21.156706+0000", 2024-05-15T12:58:54.470 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.5130000000000003e-05 2024-05-15T12:58:54.470 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.470 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:58:54.470 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.470 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.470 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.470 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:115943 3.3 3:cdfa4b09:::benchmark_data_smithi177_88104_object115942:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-05-15T12:58:54.470 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:54:20.959743+0000", 2024-05-15T12:58:54.471 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 273.427175519, 2024-05-15T12:58:54.471 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.21044579499999999, 2024-05-15T12:58:54.471 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:58:54.471 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:54.471 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:58:54.471 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T12:58:54.471 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:58:54.471 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 115943 2024-05-15T12:58:54.471 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.471 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:58:54.471 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.471 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:58:54.471 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.959743+0000", 2024-05-15T12:58:54.471 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.472 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.472 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.472 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:58:54.472 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.959743+0000", 2024-05-15T12:58:54.472 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.472 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.472 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.472 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:58:54.472 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.959744+0000", 2024-05-15T12:58:54.472 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7010000000000001e-06 2024-05-15T12:58:54.472 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.472 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.473 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:58:54.473 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.959787+0000", 2024-05-15T12:58:54.473 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3034000000000002e-05 2024-05-15T12:58:54.473 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.473 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.473 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:58:54.473 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.959800+0000", 2024-05-15T12:58:54.473 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2945999999999999e-05 2024-05-15T12:58:54.473 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.473 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.473 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:58:54.473 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.959811+0000", 2024-05-15T12:58:54.473 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0353999999999999e-05 2024-05-15T12:58:54.473 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.474 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.474 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:58:54.474 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.959937+0000", 2024-05-15T12:58:54.474 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000125778 2024-05-15T12:58:54.474 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.474 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.474 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:58:54.474 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.960049+0000", 2024-05-15T12:58:54.474 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000112215 2024-05-15T12:58:54.474 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.474 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.474 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,6", 2024-05-15T12:58:54.475 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.960175+0000", 2024-05-15T12:58:54.475 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012574400000000001 2024-05-15T12:58:54.475 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.475 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.475 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:58:54.475 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.965232+0000", 2024-05-15T12:58:54.475 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0050574449999999998 2024-05-15T12:58:54.475 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.475 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.475 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.475 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.967332+0000", 2024-05-15T12:58:54.475 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0021000950000000002 2024-05-15T12:58:54.475 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.475 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.476 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.476 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:21.170081+0000", 2024-05-15T12:58:54.476 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.20274848500000001 2024-05-15T12:58:54.476 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.476 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.476 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:58:54.476 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:21.170140+0000", 2024-05-15T12:58:54.476 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9641e-05 2024-05-15T12:58:54.476 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.476 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.476 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:58:54.476 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:21.170189+0000", 2024-05-15T12:58:54.476 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8356999999999998e-05 2024-05-15T12:58:54.476 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.477 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:58:54.477 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.477 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.477 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.477 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:115944 3.3 3:c958536f:::benchmark_data_smithi177_88104_object115943:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-05-15T12:58:54.477 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:54:20.967757+0000", 2024-05-15T12:58:54.477 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 273.41916162400003, 2024-05-15T12:58:54.477 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.208189398, 2024-05-15T12:58:54.477 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:58:54.477 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:54.477 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:58:54.477 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T12:58:54.477 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:58:54.478 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 115944 2024-05-15T12:58:54.478 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.478 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:58:54.478 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.478 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:58:54.478 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.967757+0000", 2024-05-15T12:58:54.478 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.478 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.478 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.478 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:58:54.478 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.967757+0000", 2024-05-15T12:58:54.478 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.478 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.478 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.479 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:58:54.479 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.967758+0000", 2024-05-15T12:58:54.479 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3710000000000001e-06 2024-05-15T12:58:54.479 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.479 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.479 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:58:54.479 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.967792+0000", 2024-05-15T12:58:54.479 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.4270999999999998e-05 2024-05-15T12:58:54.479 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.479 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.479 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:58:54.479 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.967807+0000", 2024-05-15T12:58:54.479 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5075999999999999e-05 2024-05-15T12:58:54.480 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.480 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.480 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:58:54.480 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.967816+0000", 2024-05-15T12:58:54.480 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8769999999999999e-06 2024-05-15T12:58:54.480 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.480 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.480 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:58:54.480 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.967903+0000", 2024-05-15T12:58:54.480 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7100000000000003e-05 2024-05-15T12:58:54.480 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.480 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.480 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:58:54.480 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.968018+0000", 2024-05-15T12:58:54.481 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00011472999999999999 2024-05-15T12:58:54.481 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.481 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.481 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,6", 2024-05-15T12:58:54.481 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.968145+0000", 2024-05-15T12:58:54.481 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000127195 2024-05-15T12:58:54.481 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.481 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.481 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:58:54.481 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.975213+0000", 2024-05-15T12:58:54.481 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0070678920000000001 2024-05-15T12:58:54.481 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.481 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.481 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.481 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.975331+0000", 2024-05-15T12:58:54.482 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000117958 2024-05-15T12:58:54.482 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.482 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.482 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.482 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:21.175834+0000", 2024-05-15T12:58:54.482 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.20050268800000001 2024-05-15T12:58:54.482 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.482 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.482 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:58:54.482 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:21.175893+0000", 2024-05-15T12:58:54.482 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9441999999999998e-05 2024-05-15T12:58:54.482 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.482 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.482 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:58:54.483 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:21.175946+0000", 2024-05-15T12:58:54.483 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2797999999999998e-05 2024-05-15T12:58:54.483 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.483 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:58:54.483 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.483 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.483 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.483 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:115947 3.3 3:ca17562c:::benchmark_data_smithi177_88104_object115946:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-05-15T12:58:54.483 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:54:20.967836+0000", 2024-05-15T12:58:54.483 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 273.419082699, 2024-05-15T12:58:54.483 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.20840172000000001, 2024-05-15T12:58:54.483 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:58:54.483 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:54.483 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:58:54.484 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T12:58:54.484 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:58:54.484 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 115947 2024-05-15T12:58:54.484 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.484 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:58:54.484 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.484 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:58:54.484 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.967836+0000", 2024-05-15T12:58:54.484 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.484 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.484 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.484 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:58:54.484 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.967836+0000", 2024-05-15T12:58:54.484 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.485 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.485 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.485 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:58:54.485 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.967836+0000", 2024-05-15T12:58:54.485 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9500000000000002e-07 2024-05-15T12:58:54.485 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.485 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.485 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:58:54.485 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.967866+0000", 2024-05-15T12:58:54.485 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.9671000000000001e-05 2024-05-15T12:58:54.485 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.485 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.485 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:58:54.485 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.967874+0000", 2024-05-15T12:58:54.486 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0870000000000007e-06 2024-05-15T12:58:54.486 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.486 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.486 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:58:54.486 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.967880+0000", 2024-05-15T12:58:54.486 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4810000000000003e-06 2024-05-15T12:58:54.486 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.486 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.486 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:58:54.486 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.971593+0000", 2024-05-15T12:58:54.486 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0037129770000000001 2024-05-15T12:58:54.486 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.486 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.487 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:58:54.487 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.971732+0000", 2024-05-15T12:58:54.487 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013872099999999999 2024-05-15T12:58:54.487 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.487 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.487 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,6", 2024-05-15T12:58:54.487 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.971890+0000", 2024-05-15T12:58:54.487 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015837699999999999 2024-05-15T12:58:54.487 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.487 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.487 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:58:54.487 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.982840+0000", 2024-05-15T12:58:54.487 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.010949379 2024-05-15T12:58:54.487 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.488 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.488 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.488 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.983783+0000", 2024-05-15T12:58:54.488 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000943052 2024-05-15T12:58:54.488 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.488 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.488 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.488 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:21.176143+0000", 2024-05-15T12:58:54.488 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.19236035500000001 2024-05-15T12:58:54.488 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.488 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.488 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:58:54.488 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:21.176201+0000", 2024-05-15T12:58:54.488 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8077e-05 2024-05-15T12:58:54.489 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.489 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.489 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:58:54.489 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:21.176237+0000", 2024-05-15T12:58:54.489 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5948000000000002e-05 2024-05-15T12:58:54.489 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.489 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:58:54.489 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.489 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.489 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.489 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:115954 3.3 3:ce6ced4c:::benchmark_data_smithi177_88104_object115953:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-05-15T12:58:54.489 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:54:20.973558+0000", 2024-05-15T12:58:54.489 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 273.41336014400002, 2024-05-15T12:58:54.489 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.202832656, 2024-05-15T12:58:54.489 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:58:54.490 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:54.490 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:58:54.490 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T12:58:54.490 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:58:54.490 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 115954 2024-05-15T12:58:54.490 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.490 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:58:54.490 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.490 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:58:54.490 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.973558+0000", 2024-05-15T12:58:54.490 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.490 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.490 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.490 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:58:54.491 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.973558+0000", 2024-05-15T12:58:54.491 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.491 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.491 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.491 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:58:54.491 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.973560+0000", 2024-05-15T12:58:54.491 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.44e-06 2024-05-15T12:58:54.491 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.491 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.491 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:58:54.491 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.973600+0000", 2024-05-15T12:58:54.491 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0410000000000001e-05 2024-05-15T12:58:54.491 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.491 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.492 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:58:54.492 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.973612+0000", 2024-05-15T12:58:54.492 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2245999999999999e-05 2024-05-15T12:58:54.492 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.492 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.492 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:58:54.492 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.973619+0000", 2024-05-15T12:58:54.492 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4710000000000004e-06 2024-05-15T12:58:54.492 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.492 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.492 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:58:54.492 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.975367+0000", 2024-05-15T12:58:54.492 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001747983 2024-05-15T12:58:54.492 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.493 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.493 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:58:54.493 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.975472+0000", 2024-05-15T12:58:54.493 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000105475 2024-05-15T12:58:54.493 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.493 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.493 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,6", 2024-05-15T12:58:54.493 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.975604+0000", 2024-05-15T12:58:54.493 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013223799999999999 2024-05-15T12:58:54.493 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.493 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.493 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:58:54.493 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.983804+0000", 2024-05-15T12:58:54.493 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0081992360000000004 2024-05-15T12:58:54.494 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.494 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.494 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.494 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.985283+0000", 2024-05-15T12:58:54.494 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001478891 2024-05-15T12:58:54.494 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.494 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.494 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.494 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:21.176273+0000", 2024-05-15T12:58:54.494 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.190990048 2024-05-15T12:58:54.494 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.494 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.494 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:58:54.494 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:21.176346+0000", 2024-05-15T12:58:54.494 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3527000000000003e-05 2024-05-15T12:58:54.495 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.495 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.495 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:58:54.495 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:21.176391+0000", 2024-05-15T12:58:54.495 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4691000000000003e-05 2024-05-15T12:58:54.495 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.495 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:58:54.495 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.495 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.495 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.495 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:115959 3.3 3:c4137197:::benchmark_data_smithi177_88104_object115958:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-05-15T12:58:54.495 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:54:20.978334+0000", 2024-05-15T12:58:54.495 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 273.408584039, 2024-05-15T12:58:54.495 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.19822703, 2024-05-15T12:58:54.496 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:58:54.496 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:58:54.496 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:58:54.496 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T12:58:54.496 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T12:58:54.496 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 115959 2024-05-15T12:58:54.496 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.496 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:58:54.496 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.496 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:58:54.496 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.978334+0000", 2024-05-15T12:58:54.496 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.496 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.497 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.497 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:58:54.497 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.978334+0000", 2024-05-15T12:58:54.497 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:58:54.497 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.497 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.497 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:58:54.497 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.978336+0000", 2024-05-15T12:58:54.497 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6199999999999999e-06 2024-05-15T12:58:54.497 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.497 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.497 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:58:54.497 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.978392+0000", 2024-05-15T12:58:54.497 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5733999999999999e-05 2024-05-15T12:58:54.497 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.498 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.498 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:58:54.498 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.978411+0000", 2024-05-15T12:58:54.498 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9524000000000001e-05 2024-05-15T12:58:54.498 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.498 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.498 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:58:54.498 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.978420+0000", 2024-05-15T12:58:54.498 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1349999999999998e-06 2024-05-15T12:58:54.498 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.498 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.498 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:58:54.498 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.978995+0000", 2024-05-15T12:58:54.498 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00057431400000000003 2024-05-15T12:58:54.499 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.499 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.499 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:58:54.499 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.979122+0000", 2024-05-15T12:58:54.499 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012702899999999999 2024-05-15T12:58:54.499 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.499 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.499 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,6", 2024-05-15T12:58:54.499 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.979312+0000", 2024-05-15T12:58:54.499 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00019014499999999999 2024-05-15T12:58:54.499 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.499 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.499 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:58:54.499 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.985352+0000", 2024-05-15T12:58:54.500 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0060401830000000002 2024-05-15T12:58:54.500 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.500 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.500 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.500 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:20.985430+0000", 2024-05-15T12:58:54.500 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7730000000000003e-05 2024-05-15T12:58:54.500 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.500 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.500 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:58:54.500 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:21.176440+0000", 2024-05-15T12:58:54.500 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.191009964 2024-05-15T12:58:54.500 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.500 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.500 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:58:54.500 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:21.176518+0000", 2024-05-15T12:58:54.501 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8793000000000003e-05 2024-05-15T12:58:54.501 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:58:54.501 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:58:54.501 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:58:54.501 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:54:21.176561+0000", 2024-05-15T12:58:54.501 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2858999999999997e-05 2024-05-15T12:58:54.501 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.501 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:58:54.501 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.501 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:58:54.501 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:58:54.501 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T12:58:54.502 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:54 smithi027 ceph-mon[39795]: osdmap e332: 12 total, 12 up, 10 in 2024-05-15T12:58:54.502 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:54 smithi027 ceph-mon[39795]: pgmap v909: 17 pgs: 17 active+clean; 4.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 65 MiB/s wr, 1.05k op/s 2024-05-15T12:58:54.502 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:54 smithi120 ceph-mon[43961]: osdmap e332: 12 total, 12 up, 10 in 2024-05-15T12:58:54.503 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:54 smithi120 ceph-mon[43961]: pgmap v909: 17 pgs: 17 active+clean; 4.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 65 MiB/s wr, 1.05k op/s 2024-05-15T12:58:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:58:54.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:54.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:58:54.609 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:58:54.670 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:55.133 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 46 16 84058 84042 114.17 67.875 0.0109667 0.00875463 2024-05-15T12:58:55.174 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-05-15T12:58:55.547 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:58:56.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:55 smithi027 ceph-mon[39795]: pgmap v910: 17 pgs: 17 active+clean; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-05-15T12:58:56.090 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:58:56.105 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:55 smithi120 ceph-mon[43961]: pgmap v910: 17 pgs: 17 active+clean; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-05-15T12:58:56.133 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 47 16 85289 85273 113.378 76.9375 0.0109856 0.00881585 2024-05-15T12:58:56.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:55 smithi177 ceph-mon[44303]: pgmap v910: 17 pgs: 17 active+clean; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-05-15T12:58:56.240 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:58:56.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:56.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:58:56.349 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:58:56.383 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:56.406 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:58:56.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:56.410 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:58:56.511 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:58:56.540 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:56.564 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:58:56.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:56.567 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:58:56.639 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:58:56.668 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:58:56.697 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:56.719 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:58:56.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:56.723 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:58:56.824 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:58:56.855 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:56.880 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:58:56.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:56.884 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:58:56.927 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T12:58:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.10 primary-affinity to 0.452084 (8296272) 2024-05-15T12:58:56.985 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:58:57.012 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:56 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1613683993' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 0.45208364879774454}]: dispatch 2024-05-15T12:58:57.013 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:56 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 0.45208364879774454}]: dispatch 2024-05-15T12:58:57.013 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:56 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/248937498' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:58:57.014 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:57.037 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:58:57.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:57.040 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:58:57.134 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 48 15 86406 86391 112.472 69.875 0.0129669 0.0088872 2024-05-15T12:58:57.141 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:58:57.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:56 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1613683993' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 0.45208364879774454}]: dispatch 2024-05-15T12:58:57.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:56 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 0.45208364879774454}]: dispatch 2024-05-15T12:58:57.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:56 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/248937498' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:58:57.174 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:57.196 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:58:57.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:57.201 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:58:57.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:56 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1613683993' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 0.45208364879774454}]: dispatch 2024-05-15T12:58:57.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:56 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 0.45208364879774454}]: dispatch 2024-05-15T12:58:57.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:56 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/248937498' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:58:57.301 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:58:57.334 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:57.355 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:58:57.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:57.360 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:58:57.461 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:58:57.500 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T12:58:57.500 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T12:58:57.500 INFO:teuthology.orchestra.run.smithi120.stdout: "num_ops": 0 2024-05-15T12:58:57.500 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T12:58:57.506 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:57.625 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:58:57.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:57.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:58:57.736 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:58:57.766 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:57.791 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:58:57.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:57.796 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:58:57.897 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:58:57.933 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:58.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:57 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:58:58.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:57 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T12:58:58.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:57 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 0.45208364879774454}]': finished 2024-05-15T12:58:58.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:57 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/248937498' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:58:58.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:57 smithi027 ceph-mon[39795]: osdmap e333: 12 total, 12 up, 10 in 2024-05-15T12:58:58.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:57 smithi027 ceph-mon[39795]: pgmap v912: 17 pgs: 17 active+clean; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-05-15T12:58:58.134 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 49 16 87603 87587 111.702 74.75 0.00969837 0.00894805 2024-05-15T12:58:58.179 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:57 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:58:58.179 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:57 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T12:58:58.179 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:57 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 0.45208364879774454}]': finished 2024-05-15T12:58:58.180 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:57 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/248937498' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:58:58.180 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:57 smithi120 ceph-mon[43961]: osdmap e333: 12 total, 12 up, 10 in 2024-05-15T12:58:58.180 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:57 smithi120 ceph-mon[43961]: pgmap v912: 17 pgs: 17 active+clean; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-05-15T12:58:58.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:57 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T12:58:58.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:57 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T12:58:58.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:57 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 0.45208364879774454}]': finished 2024-05-15T12:58:58.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:57 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/248937498' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T12:58:58.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:57 smithi177 ceph-mon[44303]: osdmap e333: 12 total, 12 up, 10 in 2024-05-15T12:58:58.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:57 smithi177 ceph-mon[44303]: pgmap v912: 17 pgs: 17 active+clean; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-05-15T12:58:58.189 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:58:58.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:58:58.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:58:58.213 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-05-15T12:58:58.297 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:58:58.348 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:58:58.590 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:58:59.134 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 50 16 88873 88857 111.055 79.375 0.00875621 0.009001 2024-05-15T12:58:59.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:59 smithi177 ceph-mon[44303]: osdmap e334: 12 total, 12 up, 10 in 2024-05-15T12:58:59.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:58:59 smithi177 ceph-mon[44303]: 4.1 scrub starts 2024-05-15T12:58:59.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:59 smithi027 ceph-mon[39795]: osdmap e334: 12 total, 12 up, 10 in 2024-05-15T12:58:59.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:58:59 smithi027 ceph-mon[39795]: 4.1 scrub starts 2024-05-15T12:58:59.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:59 smithi120 ceph-mon[43961]: osdmap e334: 12 total, 12 up, 10 in 2024-05-15T12:58:59.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:58:59 smithi120 ceph-mon[43961]: 4.1 scrub starts 2024-05-15T12:59:00.134 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 51 16 90117 90101 110.402 77.75 0.0166921 0.00905282 2024-05-15T12:59:00.153 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:59:00.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:00.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:59:00.261 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:59:00.323 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:00 smithi177 ceph-mon[44303]: pgmap v914: 17 pgs: 17 active+clean; 5.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-05-15T12:59:00.435 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:59:00.440 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:00 smithi027 ceph-mon[39795]: pgmap v914: 17 pgs: 17 active+clean; 5.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-05-15T12:59:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:00.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:59:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:00 smithi120 ceph-mon[43961]: pgmap v914: 17 pgs: 17 active+clean; 5.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-05-15T12:59:00.542 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:59:00.544 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:59:00.569 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T12:59:00.569 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T12:59:00.569 INFO:teuthology.orchestra.run.smithi120.stdout: "complaint_time": 30, 2024-05-15T12:59:00.569 INFO:teuthology.orchestra.run.smithi120.stdout: "num_blocked_ops": 0 2024-05-15T12:59:00.569 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T12:59:00.574 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:00.580 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:59:00.596 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:59:00.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:00.599 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:59:00.615 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:59:00.657 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:59:00.699 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:59:00.718 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:59:00.760 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:00.762 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:59:00.786 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:59:00.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:00.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:59:00.813 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:59:00.855 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:59:00.891 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:59:00.903 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:59:00.924 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:00.947 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:59:00.984 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:59:01.027 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:59:01.037 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:59:01.037 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:59:01.037 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:59:01.037 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:59:01.037 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:59:01.037 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:59:01.038 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:59:01.038 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:59:01.038 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:59:01.038 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:59:01.038 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:59:01.038 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:59:01.134 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 52 16 91278 91262 109.674 72.5625 0.0119319 0.0091141 2024-05-15T12:59:01.394 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:59:01.399 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-05-15T12:59:01.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:01.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:59:01.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:01 smithi177 ceph-mon[44303]: 4.1 scrub ok 2024-05-15T12:59:01.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:01 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3692733996' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:59:01.483 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:01 smithi120 ceph-mon[43961]: 4.1 scrub ok 2024-05-15T12:59:01.484 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:01 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3692733996' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:59:01.501 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:59:01.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:01 smithi027 ceph-mon[39795]: 4.1 scrub ok 2024-05-15T12:59:01.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:01 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3692733996' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:59:01.558 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:01.817 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:59:01.875 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T12:59:02.033 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:59:02.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:02.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:59:02.134 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 53 16 92513 92497 109.061 77.1875 0.0160298 0.00916505 2024-05-15T12:59:02.140 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:59:02.170 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:02.195 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:59:02.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:02.198 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:59:02.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:02 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:59:02.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:02 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:59:02.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:02 smithi027 ceph-mon[39795]: osdmap e335: 12 total, 12 up, 10 in 2024-05-15T12:59:02.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:02 smithi027 ceph-mon[39795]: pgmap v916: 17 pgs: 17 active+clean; 5.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-05-15T12:59:02.299 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:59:02.343 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:02 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:59:02.343 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:02 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:59:02.343 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:02 smithi120 ceph-mon[43961]: osdmap e335: 12 total, 12 up, 10 in 2024-05-15T12:59:02.344 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:02 smithi120 ceph-mon[43961]: pgmap v916: 17 pgs: 17 active+clean; 5.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-05-15T12:59:02.348 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:02.360 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:59:02.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:02 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:59:02.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:02 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:59:02.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:02 smithi177 ceph-mon[44303]: osdmap e335: 12 total, 12 up, 10 in 2024-05-15T12:59:02.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:02 smithi177 ceph-mon[44303]: pgmap v916: 17 pgs: 17 active+clean; 5.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-05-15T12:59:03.134 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 54 16 93721 93705 108.44 75.5 0.0239686 0.0092179 2024-05-15T12:59:03.217 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10] out_osds: [11, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T12:59:03.217 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T12:59:03.217 INFO:tasks.thrashosds.thrasher:Adding osd 11 2024-05-15T12:59:03.218 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd in 11 2024-05-15T12:59:03.409 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:59:03.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:03.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:59:03.522 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:59:03.620 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:03.982 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:59:03.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:03.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:59:03.994 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T12:59:03.994 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:59:04.050 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:59:04.081 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:59:04.086 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:59:04.134 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 55 16 94825 94809 107.722 69 0.00389172 0.00927941 2024-05-15T12:59:04.152 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:04.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:04 smithi120 ceph-mon[43961]: osdmap e336: 12 total, 12 up, 10 in 2024-05-15T12:59:04.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:04 smithi120 ceph-mon[43961]: pgmap v918: 17 pgs: 17 active+clean; 5.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-05-15T12:59:04.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:04 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:59:04.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:04 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:59:04.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:04 smithi027 ceph-mon[39795]: osdmap e336: 12 total, 12 up, 10 in 2024-05-15T12:59:04.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:04 smithi027 ceph-mon[39795]: pgmap v918: 17 pgs: 17 active+clean; 5.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-05-15T12:59:04.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:04 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:59:04.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:04 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:59:04.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:04 smithi177 ceph-mon[44303]: osdmap e336: 12 total, 12 up, 10 in 2024-05-15T12:59:04.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:04 smithi177 ceph-mon[44303]: pgmap v918: 17 pgs: 17 active+clean; 5.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-05-15T12:59:04.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:04 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:59:04.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:04 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:59:04.731 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:59:04.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:04.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:59:04.838 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:59:04.841 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-05-15T12:59:04.872 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:04.906 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:59:04.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:04.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:59:05.013 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:59:05.080 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:05.121 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:59:05.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:05.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:59:05.134 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 56 16 95863 95847 106.957 64.875 0.00730513 0.00934583 2024-05-15T12:59:05.229 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:59:05.236 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:59:05.246 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T12:59:05.291 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:05.498 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:05 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T12:59:05.498 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:05 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T12:59:05.498 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:05 smithi120 ceph-mon[43961]: osdmap e337: 12 total, 12 up, 10 in 2024-05-15T12:59:05.499 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:05 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2027574070' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:59:05.499 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:05 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:59:05.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:05 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T12:59:05.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:05 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T12:59:05.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:05 smithi027 ceph-mon[39795]: osdmap e337: 12 total, 12 up, 10 in 2024-05-15T12:59:05.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:05 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2027574070' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:59:05.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:05 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:59:05.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:05 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T12:59:05.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:05 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T12:59:05.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:05 smithi177 ceph-mon[44303]: osdmap e337: 12 total, 12 up, 10 in 2024-05-15T12:59:05.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:05 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2027574070' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:59:05.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:05 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T12:59:06.134 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 57 16 97062 97046 106.395 74.9375 0.00826181 0.00939444 2024-05-15T12:59:06.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:06 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:59:06.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:06 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:59:06.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:06 smithi027 ceph-mon[39795]: osdmap e338: 12 total, 12 up, 10 in 2024-05-15T12:59:06.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:06 smithi027 ceph-mon[39795]: pgmap v921: 17 pgs: 17 active+clean; 5.8 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-05-15T12:59:06.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:06 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:59:06.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:06 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:59:06.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:06 smithi120 ceph-mon[43961]: osdmap e338: 12 total, 12 up, 10 in 2024-05-15T12:59:06.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:06 smithi120 ceph-mon[43961]: pgmap v921: 17 pgs: 17 active+clean; 5.8 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-05-15T12:59:06.671 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:59:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:06.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:59:06.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:06 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:59:06.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:06 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T12:59:06.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:06 smithi177 ceph-mon[44303]: osdmap e338: 12 total, 12 up, 10 in 2024-05-15T12:59:06.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:06 smithi177 ceph-mon[44303]: pgmap v921: 17 pgs: 17 active+clean; 5.8 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-05-15T12:59:06.779 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:59:06.813 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:06.841 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:59:06.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:06.845 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:59:06.947 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:59:06.995 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:07.020 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:59:07.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:07.027 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:59:07.129 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:59:07.134 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 58 16 98302 98286 105.897 77.5 0.010503 0.00943959 2024-05-15T12:59:07.187 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:07.214 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:59:07.236 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:59:07.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:07.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:59:07.296 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T12:59:07.297 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:59:07.342 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:59:07.393 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:07.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:07 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:59:07.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:07 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:59:07.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:07 smithi027 ceph-mon[39795]: osdmap e339: 12 total, 12 up, 10 in 2024-05-15T12:59:07.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:07 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:59:07.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:07 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:59:07.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:07 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:59:07.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:07 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:59:07.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:07 smithi120 ceph-mon[43961]: osdmap e339: 12 total, 12 up, 10 in 2024-05-15T12:59:07.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:07 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:59:07.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:07 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:59:07.576 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:59:07.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:07.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:59:07.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:07 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:59:07.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:07 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:59:07.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:07 smithi177 ceph-mon[44303]: osdmap e339: 12 total, 12 up, 10 in 2024-05-15T12:59:07.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:07 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:59:07.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:07 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:59:07.681 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:59:07.732 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:07.877 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:59:07.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:07.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:59:07.985 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:59:08.015 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:08.135 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 59 15 99241 99226 105.098 58.75 0.0174211 0.0095108 2024-05-15T12:59:08.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:08 smithi027 ceph-mon[39795]: osdmap e340: 12 total, 12 up, 10 in 2024-05-15T12:59:08.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:08 smithi027 ceph-mon[39795]: pgmap v924: 17 pgs: 17 active+clean; 5.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-05-15T12:59:08.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:08 smithi027 ceph-mon[39795]: osdmap e341: 12 total, 12 up, 10 in 2024-05-15T12:59:08.324 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-05-15T12:59:08.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:08 smithi120 ceph-mon[43961]: osdmap e340: 12 total, 12 up, 10 in 2024-05-15T12:59:08.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:08 smithi120 ceph-mon[43961]: pgmap v924: 17 pgs: 17 active+clean; 5.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-05-15T12:59:08.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:08 smithi120 ceph-mon[43961]: osdmap e341: 12 total, 12 up, 10 in 2024-05-15T12:59:08.565 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:59:08.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:08.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:59:08.679 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:59:08.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:08 smithi177 ceph-mon[44303]: osdmap e340: 12 total, 12 up, 10 in 2024-05-15T12:59:08.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:08 smithi177 ceph-mon[44303]: pgmap v924: 17 pgs: 17 active+clean; 5.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-05-15T12:59:08.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:08 smithi177 ceph-mon[44303]: osdmap e341: 12 total, 12 up, 10 in 2024-05-15T12:59:08.689 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:59:08.715 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T12:59:08.717 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:08.738 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:59:08.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:08.741 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:59:08.843 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:59:08.872 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:08.895 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:59:08.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:08.899 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:59:09.000 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:59:09.029 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:09.054 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:59:09.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:09.057 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:59:09.135 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T12:59:09.145943+0000 min lat: 0.00111471 max lat: 0.620182 avg lat: 0.00956531 2024-05-15T12:59:09.135 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T12:59:09.135 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 60 16 100350 100334 104.5 69.25 0.00733481 0.00956531 2024-05-15T12:59:09.157 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:59:09.186 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:09.227 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:59:09.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:09.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:59:09.341 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:59:09.389 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:09.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:09 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:59:09.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:09 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:59:09.647 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:59:09.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:09.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:59:09.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:09 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:59:09.751 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:59:09.782 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:09.804 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:59:09.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:09.809 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:59:09.910 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:59:09.941 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:09.989 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:59:09.992 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:59:10.025 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:09 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-a[39772]: 2024-05-15T12:59:09.732+0000 7f8daffd2700 -1 mon.a@0(leader).osd e341 definitely_dead 0 2024-05-15T12:59:10.071 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:59:10.117 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:59:10.135 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 61 16 101183 101167 103.641 52.0625 0.00352636 0.00964365 2024-05-15T12:59:10.165 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:59:10.195 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:59:10.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:10.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:59:10.211 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:59:10.246 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:59:10.288 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:59:10.299 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:59:10.322 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:59:10.326 INFO:teuthology.orchestra.run.smithi027.stderr:marked in osd.11. 2024-05-15T12:59:10.326 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:10.347 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:59:10.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:10.350 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:59:10.356 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:59:10.377 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:59:10.398 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:59:10.418 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:59:10.419 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:59:10.419 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:59:10.420 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:59:10.420 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:59:10.420 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:59:10.420 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:59:10.420 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:59:10.420 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:59:10.420 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:59:10.420 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:59:10.420 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:59:10.420 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:59:10.451 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:59:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:10.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:10 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3145215645' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-05-15T12:59:10.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:10 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-05-15T12:59:10.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:10 smithi120 ceph-mon[43961]: pgmap v926: 17 pgs: 17 active+clean; 6.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-05-15T12:59:10.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:10 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:59:10.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:10 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:59:10.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:10 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:59:10.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:10 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2260734732' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:59:10.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:10 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3145215645' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-05-15T12:59:10.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:10 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-05-15T12:59:10.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:10 smithi177 ceph-mon[44303]: pgmap v926: 17 pgs: 17 active+clean; 6.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-05-15T12:59:10.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:10 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:59:10.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:10 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:59:10.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:10 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:59:10.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:10 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2260734732' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:59:10.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:10 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3145215645' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-05-15T12:59:10.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:10 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-05-15T12:59:10.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:10 smithi027 ceph-mon[39795]: pgmap v926: 17 pgs: 17 active+clean; 6.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-05-15T12:59:10.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:10 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:59:10.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:10 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:59:10.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:10 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T12:59:10.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:10 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2260734732' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:59:10.796 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T12:59:10.796 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T12:59:10.796 INFO:teuthology.orchestra.run.smithi120.stdout: "complaint_time": 30, 2024-05-15T12:59:10.797 INFO:teuthology.orchestra.run.smithi120.stdout: "num_blocked_ops": 0 2024-05-15T12:59:10.797 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T12:59:11.135 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 62 16 101391 101375 102.179 13 0.00484989 0.00965322 2024-05-15T12:59:11.456 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:11 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-05-15T12:59:11.456 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:11 smithi120 ceph-mon[43961]: osdmap e342: 12 total, 12 up, 11 in 2024-05-15T12:59:11.481 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-05-15T12:59:11.615 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:11 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-05-15T12:59:11.616 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:11 smithi027 ceph-mon[39795]: osdmap e342: 12 total, 12 up, 11 in 2024-05-15T12:59:11.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:11 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-05-15T12:59:11.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:11 smithi177 ceph-mon[44303]: osdmap e342: 12 total, 12 up, 11 in 2024-05-15T12:59:11.741 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:59:12.135 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 63 16 101391 101375 100.557 0 - 0.00965322 2024-05-15T12:59:12.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:12 smithi177 ceph-mon[44303]: osdmap e343: 12 total, 12 up, 11 in 2024-05-15T12:59:12.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:12 smithi177 ceph-mon[44303]: pgmap v929: 17 pgs: 1 peering, 16 active+clean; 6.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-05-15T12:59:12.716 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:12 smithi120 ceph-mon[43961]: osdmap e343: 12 total, 12 up, 11 in 2024-05-15T12:59:12.716 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:12 smithi120 ceph-mon[43961]: pgmap v929: 17 pgs: 1 peering, 16 active+clean; 6.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-05-15T12:59:12.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:12 smithi027 ceph-mon[39795]: osdmap e343: 12 total, 12 up, 11 in 2024-05-15T12:59:12.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:12 smithi027 ceph-mon[39795]: pgmap v929: 17 pgs: 1 peering, 16 active+clean; 6.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-05-15T12:59:13.135 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 64 16 101624 101608 99.2133 7.28125 0.00849366 0.0100741 2024-05-15T12:59:13.416 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:13 smithi120 ceph-mon[43961]: osdmap e344: 12 total, 12 up, 11 in 2024-05-15T12:59:13.416 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:13 smithi120 ceph-mon[43961]: osdmap e345: 12 total, 12 up, 11 in 2024-05-15T12:59:13.469 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T12:59:13.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:13 smithi177 ceph-mon[44303]: osdmap e344: 12 total, 12 up, 11 in 2024-05-15T12:59:13.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:13 smithi177 ceph-mon[44303]: osdmap e345: 12 total, 12 up, 11 in 2024-05-15T12:59:13.767 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:59:13.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:13.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:13 smithi027 ceph-mon[39795]: osdmap e344: 12 total, 12 up, 11 in 2024-05-15T12:59:13.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:13 smithi027 ceph-mon[39795]: osdmap e345: 12 total, 12 up, 11 in 2024-05-15T12:59:13.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:59:13.818 INFO:tasks.thrashosds.thrasher:Added osd 11 2024-05-15T12:59:13.877 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:59:13.925 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:13.957 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:59:13.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:13.961 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:59:14.062 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:59:14.088 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:14.114 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:59:14.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:14.117 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:59:14.135 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 65 16 102349 102333 98.384 45.3125 0.0209709 0.0101594 2024-05-15T12:59:14.218 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:59:14.272 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:14.310 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:59:14.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:14.315 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:59:14.416 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:59:14.456 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:14.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:14 smithi177 ceph-mon[44303]: pgmap v932: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 peering, 13 active+clean; 6.1 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 32 MiB/s wr, 506 op/s; 12865/301443 objects degraded (4.268%); 6371/301443 objects misplaced (2.114%); 192 KiB/s, 2 objects/s recovering 2024-05-15T12:59:14.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:14 smithi177 ceph-mon[44303]: osdmap e346: 12 total, 12 up, 11 in 2024-05-15T12:59:14.706 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T12:59:14.706 INFO:teuthology.orchestra.run.smithi120.stdout: "size": 20, 2024-05-15T12:59:14.707 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 600, 2024-05-15T12:59:14.707 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [ 2024-05-15T12:59:14.707 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.707 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:90791 2.7 2:ede8a1cf:::benchmark_data_smithi177_78104_object90790:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:59:14.707 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:49:17.233646+0000", 2024-05-15T12:59:14.707 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 596.523057386, 2024-05-15T12:59:14.707 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1459873439999999, 2024-05-15T12:59:14.707 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:59:14.707 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T12:59:14.707 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:59:14.707 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:59:14.707 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:59:14.707 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 90791 2024-05-15T12:59:14.707 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.708 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:59:14.708 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.708 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:59:14.708 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.233646+0000", 2024-05-15T12:59:14.708 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.708 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.708 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.708 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:59:14.708 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.233646+0000", 2024-05-15T12:59:14.708 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.708 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.708 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.708 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:59:14.708 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.233647+0000", 2024-05-15T12:59:14.709 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.71e-07 2024-05-15T12:59:14.709 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.709 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.709 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:59:14.709 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.233716+0000", 2024-05-15T12:59:14.709 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8796999999999999e-05 2024-05-15T12:59:14.709 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.709 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.709 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:59:14.709 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.233737+0000", 2024-05-15T12:59:14.709 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0421e-05 2024-05-15T12:59:14.709 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.709 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.709 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:59:14.710 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.233744+0000", 2024-05-15T12:59:14.710 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0539999999999997e-06 2024-05-15T12:59:14.710 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.710 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.710 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.710 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.236353+0000", 2024-05-15T12:59:14.711 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0026095129999999999 2024-05-15T12:59:14.711 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.711 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.711 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:59:14.711 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.236477+0000", 2024-05-15T12:59:14.711 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000123428 2024-05-15T12:59:14.711 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.711 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.711 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3", 2024-05-15T12:59:14.711 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.236620+0000", 2024-05-15T12:59:14.711 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014386299999999999 2024-05-15T12:59:14.711 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.712 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.712 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:59:14.712 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.241511+0000", 2024-05-15T12:59:14.712 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0048901989999999996 2024-05-15T12:59:14.712 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.713 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.713 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.713 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.243337+0000", 2024-05-15T12:59:14.713 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001826041 2024-05-15T12:59:14.713 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.713 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.713 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.713 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.249571+0000", 2024-05-15T12:59:14.713 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0062345100000000004 2024-05-15T12:59:14.713 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.713 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.713 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:59:14.713 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.249584+0000", 2024-05-15T12:59:14.714 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3359e-05 2024-05-15T12:59:14.714 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.714 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.714 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.714 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.855870+0000", 2024-05-15T12:59:14.714 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.60628580300000001 2024-05-15T12:59:14.714 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.714 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.714 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:59:14.714 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.855883+0000", 2024-05-15T12:59:14.714 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2294e-05 2024-05-15T12:59:14.714 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.714 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.715 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.715 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.379560+0000", 2024-05-15T12:59:14.715 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5236770769999999 2024-05-15T12:59:14.715 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.715 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.715 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:59:14.715 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.379634+0000", 2024-05-15T12:59:14.715 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4114000000000002e-05 2024-05-15T12:59:14.715 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.715 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:59:14.715 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.715 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.715 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.716 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:90794 2.7 2:ee131b04:::benchmark_data_smithi177_78104_object90793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-05-15T12:59:14.716 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:49:17.238186+0000", 2024-05-15T12:59:14.716 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 596.51851822000003, 2024-05-15T12:59:14.716 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1417517950000002, 2024-05-15T12:59:14.716 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:59:14.716 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T12:59:14.716 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:59:14.716 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:59:14.716 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:59:14.716 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 90794 2024-05-15T12:59:14.716 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.716 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:59:14.716 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.716 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:59:14.717 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.238186+0000", 2024-05-15T12:59:14.717 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.717 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.717 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.717 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:59:14.717 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.238186+0000", 2024-05-15T12:59:14.717 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.717 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.717 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.717 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:59:14.717 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.238187+0000", 2024-05-15T12:59:14.717 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2890000000000001e-06 2024-05-15T12:59:14.717 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.717 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.718 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:59:14.718 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.238230+0000", 2024-05-15T12:59:14.718 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2707999999999998e-05 2024-05-15T12:59:14.718 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.718 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.718 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:59:14.718 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.238246+0000", 2024-05-15T12:59:14.718 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6101999999999999e-05 2024-05-15T12:59:14.718 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.718 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.719 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:59:14.719 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.238253+0000", 2024-05-15T12:59:14.719 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7279999999999995e-06 2024-05-15T12:59:14.719 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.719 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.719 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.719 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.239026+0000", 2024-05-15T12:59:14.719 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00077243500000000003 2024-05-15T12:59:14.719 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.719 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.719 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:59:14.719 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.239133+0000", 2024-05-15T12:59:14.719 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000107557 2024-05-15T12:59:14.719 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.720 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.720 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3", 2024-05-15T12:59:14.720 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.239228+0000", 2024-05-15T12:59:14.720 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.4159000000000004e-05 2024-05-15T12:59:14.720 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.720 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.720 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:59:14.720 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.241664+0000", 2024-05-15T12:59:14.720 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0024366359999999998 2024-05-15T12:59:14.720 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.720 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.720 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.720 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.249638+0000", 2024-05-15T12:59:14.720 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0079736939999999999 2024-05-15T12:59:14.721 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.721 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.721 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:59:14.721 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.249649+0000", 2024-05-15T12:59:14.721 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0832000000000001e-05 2024-05-15T12:59:14.721 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.721 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.721 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.721 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.855991+0000", 2024-05-15T12:59:14.721 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.60634224400000003 2024-05-15T12:59:14.721 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.721 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.721 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:59:14.721 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856003+0000", 2024-05-15T12:59:14.722 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.183e-05 2024-05-15T12:59:14.722 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.722 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.722 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.722 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.379870+0000", 2024-05-15T12:59:14.722 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5238669819999999 2024-05-15T12:59:14.723 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.723 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.723 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:59:14.723 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.379937+0000", 2024-05-15T12:59:14.723 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7599000000000001e-05 2024-05-15T12:59:14.723 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.723 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:59:14.723 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.723 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.723 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.724 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:90802 2.7 2:e02a2e8d:::benchmark_data_smithi177_78104_object90801:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-05-15T12:59:14.724 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:49:17.246620+0000", 2024-05-15T12:59:14.724 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 596.51008402000002, 2024-05-15T12:59:14.724 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1558356180000002, 2024-05-15T12:59:14.724 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:59:14.724 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:59:14.724 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:59:14.724 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:59:14.724 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:59:14.724 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 90802 2024-05-15T12:59:14.724 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.724 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:59:14.724 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.724 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:59:14.725 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.246620+0000", 2024-05-15T12:59:14.725 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.725 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.725 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.725 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:59:14.725 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.246620+0000", 2024-05-15T12:59:14.725 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.725 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.725 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.725 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:59:14.725 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.246621+0000", 2024-05-15T12:59:14.726 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3349999999999999e-06 2024-05-15T12:59:14.726 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.726 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.726 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:59:14.726 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.246683+0000", 2024-05-15T12:59:14.726 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1421999999999996e-05 2024-05-15T12:59:14.726 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.726 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.726 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:59:14.726 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.246698+0000", 2024-05-15T12:59:14.726 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5852e-05 2024-05-15T12:59:14.726 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.726 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.727 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:59:14.727 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.246706+0000", 2024-05-15T12:59:14.727 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7379999999999994e-06 2024-05-15T12:59:14.727 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.727 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.727 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.727 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.246773+0000", 2024-05-15T12:59:14.727 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6762999999999998e-05 2024-05-15T12:59:14.727 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.727 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.727 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op must wait for map", 2024-05-15T12:59:14.727 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.246780+0000", 2024-05-15T12:59:14.727 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2980000000000003e-06 2024-05-15T12:59:14.727 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.728 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.728 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.728 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.249680+0000", 2024-05-15T12:59:14.728 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0028993619999999999 2024-05-15T12:59:14.728 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.728 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.728 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:59:14.728 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.249685+0000", 2024-05-15T12:59:14.728 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2530000000000004e-06 2024-05-15T12:59:14.728 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.728 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.728 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.728 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856040+0000", 2024-05-15T12:59:14.728 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.60635493299999998 2024-05-15T12:59:14.729 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.729 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.729 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:59:14.729 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856050+0000", 2024-05-15T12:59:14.729 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0504e-05 2024-05-15T12:59:14.729 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.729 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.729 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.729 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.380002+0000", 2024-05-15T12:59:14.729 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5239516829999999 2024-05-15T12:59:14.729 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.729 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.729 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:59:14.729 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.380127+0000", 2024-05-15T12:59:14.730 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000125389 2024-05-15T12:59:14.730 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.730 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.730 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3,10", 2024-05-15T12:59:14.730 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.380288+0000", 2024-05-15T12:59:14.730 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000160163 2024-05-15T12:59:14.730 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.730 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.730 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.730 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.385190+0000", 2024-05-15T12:59:14.730 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0049021289999999999 2024-05-15T12:59:14.730 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.730 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.731 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:59:14.731 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.387946+0000", 2024-05-15T12:59:14.731 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.002756859 2024-05-15T12:59:14.731 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.731 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.731 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.731 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.400172+0000", 2024-05-15T12:59:14.731 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.012225234 2024-05-15T12:59:14.731 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.731 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.731 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.731 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.402313+0000", 2024-05-15T12:59:14.731 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.002141146 2024-05-15T12:59:14.732 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.732 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.732 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:59:14.732 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.402375+0000", 2024-05-15T12:59:14.732 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.2288999999999999e-05 2024-05-15T12:59:14.732 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.732 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.732 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:59:14.732 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.402455+0000", 2024-05-15T12:59:14.732 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0265999999999993e-05 2024-05-15T12:59:14.732 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.732 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:59:14.733 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.733 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.733 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.733 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:90803 2.7 2:ec92ac45:::benchmark_data_smithi177_78104_object90802:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-05-15T12:59:14.733 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:49:17.250972+0000", 2024-05-15T12:59:14.733 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 596.50573155699999, 2024-05-15T12:59:14.733 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1517382629999999, 2024-05-15T12:59:14.733 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:59:14.733 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:59:14.733 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:59:14.733 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:59:14.733 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:59:14.734 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 90803 2024-05-15T12:59:14.734 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.734 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:59:14.734 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.734 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:59:14.734 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.250972+0000", 2024-05-15T12:59:14.734 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.734 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.734 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.734 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:59:14.734 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.250972+0000", 2024-05-15T12:59:14.734 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.734 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.735 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.735 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:59:14.735 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.250973+0000", 2024-05-15T12:59:14.735 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.226e-06 2024-05-15T12:59:14.735 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.735 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.735 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:59:14.735 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.251042+0000", 2024-05-15T12:59:14.735 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8552000000000004e-05 2024-05-15T12:59:14.735 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.735 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.735 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:59:14.735 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.251053+0000", 2024-05-15T12:59:14.735 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0567000000000001e-05 2024-05-15T12:59:14.736 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.736 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.736 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:59:14.736 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.251059+0000", 2024-05-15T12:59:14.736 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3459999999999999e-06 2024-05-15T12:59:14.736 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.736 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.736 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.736 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.251103+0000", 2024-05-15T12:59:14.736 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4314999999999998e-05 2024-05-15T12:59:14.736 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.736 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.736 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:59:14.736 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.251113+0000", 2024-05-15T12:59:14.737 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.4169999999999998e-06 2024-05-15T12:59:14.737 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.737 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.737 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.737 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856075+0000", 2024-05-15T12:59:14.737 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.60496192999999998 2024-05-15T12:59:14.737 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.737 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.737 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:59:14.737 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856082+0000", 2024-05-15T12:59:14.737 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4490000000000002e-06 2024-05-15T12:59:14.737 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.737 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.738 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.738 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.382594+0000", 2024-05-15T12:59:14.738 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.526511927 2024-05-15T12:59:14.738 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.738 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.738 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:59:14.738 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.382695+0000", 2024-05-15T12:59:14.738 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000101407 2024-05-15T12:59:14.738 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.738 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.738 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3,10", 2024-05-15T12:59:14.738 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.382806+0000", 2024-05-15T12:59:14.738 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000110837 2024-05-15T12:59:14.739 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.739 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.739 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.739 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.385250+0000", 2024-05-15T12:59:14.739 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0024435669999999998 2024-05-15T12:59:14.739 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.739 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.739 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:59:14.739 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.387954+0000", 2024-05-15T12:59:14.739 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0027041000000000001 2024-05-15T12:59:14.739 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.739 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.739 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.739 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.400230+0000", 2024-05-15T12:59:14.739 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.012276545999999999 2024-05-15T12:59:14.740 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.740 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.740 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.740 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.402518+0000", 2024-05-15T12:59:14.740 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00228707 2024-05-15T12:59:14.740 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.740 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.740 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:59:14.740 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.402585+0000", 2024-05-15T12:59:14.740 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6982000000000003e-05 2024-05-15T12:59:14.740 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.740 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.740 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:59:14.740 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.402711+0000", 2024-05-15T12:59:14.741 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012602499999999999 2024-05-15T12:59:14.741 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.741 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:59:14.741 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.741 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.741 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.741 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:90810 2.7 2:eb1c506f:::benchmark_data_smithi177_78104_object90809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-05-15T12:59:14.741 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:49:17.261408+0000", 2024-05-15T12:59:14.741 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 596.49529547300006, 2024-05-15T12:59:14.741 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.141452954, 2024-05-15T12:59:14.741 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:59:14.741 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:59:14.741 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:59:14.741 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:59:14.741 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:59:14.742 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 90810 2024-05-15T12:59:14.742 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.742 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:59:14.742 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.742 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:59:14.742 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261408+0000", 2024-05-15T12:59:14.742 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.742 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.742 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.742 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:59:14.742 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261408+0000", 2024-05-15T12:59:14.742 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.742 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.743 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.743 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:59:14.743 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261410+0000", 2024-05-15T12:59:14.743 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5629999999999999e-06 2024-05-15T12:59:14.743 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.743 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.743 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:59:14.743 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261473+0000", 2024-05-15T12:59:14.743 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3189999999999996e-05 2024-05-15T12:59:14.743 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.743 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.743 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:59:14.743 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261485+0000", 2024-05-15T12:59:14.743 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2306e-05 2024-05-15T12:59:14.743 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.744 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.744 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:59:14.744 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261494+0000", 2024-05-15T12:59:14.744 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.9150000000000002e-06 2024-05-15T12:59:14.744 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.744 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.744 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.744 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261546+0000", 2024-05-15T12:59:14.744 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1975999999999998e-05 2024-05-15T12:59:14.744 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.744 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.744 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:59:14.744 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261553+0000", 2024-05-15T12:59:14.744 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.1779999999999997e-06 2024-05-15T12:59:14.745 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.745 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.745 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.745 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856116+0000", 2024-05-15T12:59:14.745 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.59456220500000001 2024-05-15T12:59:14.745 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.745 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.745 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:59:14.745 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856121+0000", 2024-05-15T12:59:14.745 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5300000000000004e-06 2024-05-15T12:59:14.745 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.745 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.745 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.746 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.385305+0000", 2024-05-15T12:59:14.746 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.529184074 2024-05-15T12:59:14.746 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.746 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.746 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:59:14.746 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.385421+0000", 2024-05-15T12:59:14.746 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00011531699999999999 2024-05-15T12:59:14.746 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.746 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.746 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3,10", 2024-05-15T12:59:14.746 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.385558+0000", 2024-05-15T12:59:14.746 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013759800000000001 2024-05-15T12:59:14.746 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.746 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.747 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.747 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.390650+0000", 2024-05-15T12:59:14.747 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0050915530000000004 2024-05-15T12:59:14.747 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.747 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.747 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:59:14.747 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.390728+0000", 2024-05-15T12:59:14.747 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8359e-05 2024-05-15T12:59:14.747 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.747 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.747 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.747 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.400258+0000", 2024-05-15T12:59:14.747 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0095295580000000005 2024-05-15T12:59:14.747 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.748 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.748 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.748 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.402758+0000", 2024-05-15T12:59:14.748 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.002500071 2024-05-15T12:59:14.748 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.748 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.748 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:59:14.748 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.402795+0000", 2024-05-15T12:59:14.748 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7286000000000002e-05 2024-05-15T12:59:14.748 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.748 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.748 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:59:14.748 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.402861+0000", 2024-05-15T12:59:14.748 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6274999999999998e-05 2024-05-15T12:59:14.749 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.749 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:59:14.749 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.749 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.749 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.749 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:90812 2.7 2:e1d3b6e4:::benchmark_data_smithi177_78104_object90811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-05-15T12:59:14.749 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:49:17.261825+0000", 2024-05-15T12:59:14.749 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 596.49487892100001, 2024-05-15T12:59:14.749 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1412600579999999, 2024-05-15T12:59:14.749 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:59:14.749 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:59:14.749 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:59:14.749 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:59:14.749 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:59:14.750 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 90812 2024-05-15T12:59:14.750 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.750 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:59:14.750 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.750 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:59:14.750 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261825+0000", 2024-05-15T12:59:14.750 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.750 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.750 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.750 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:59:14.750 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261825+0000", 2024-05-15T12:59:14.750 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.750 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.751 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.751 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:59:14.751 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261826+0000", 2024-05-15T12:59:14.751 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7999999999999995e-07 2024-05-15T12:59:14.751 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.751 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.751 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:59:14.751 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261892+0000", 2024-05-15T12:59:14.751 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6837999999999999e-05 2024-05-15T12:59:14.751 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.751 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.751 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:59:14.751 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261902+0000", 2024-05-15T12:59:14.751 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.9229999999999997e-06 2024-05-15T12:59:14.751 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.752 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.752 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:59:14.752 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261908+0000", 2024-05-15T12:59:14.752 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.366e-06 2024-05-15T12:59:14.752 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.752 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.752 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.752 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261952+0000", 2024-05-15T12:59:14.752 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4563e-05 2024-05-15T12:59:14.752 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.752 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.752 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:59:14.752 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.261959+0000", 2024-05-15T12:59:14.752 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7789999999999998e-06 2024-05-15T12:59:14.753 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.753 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.753 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.753 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856157+0000", 2024-05-15T12:59:14.753 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.59419784499999995 2024-05-15T12:59:14.753 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.753 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.753 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:59:14.753 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856173+0000", 2024-05-15T12:59:14.753 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5930000000000002e-05 2024-05-15T12:59:14.753 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.753 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.753 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.753 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.387998+0000", 2024-05-15T12:59:14.754 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5318250820000001 2024-05-15T12:59:14.754 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.754 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.754 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:59:14.754 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.388139+0000", 2024-05-15T12:59:14.754 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014140499999999999 2024-05-15T12:59:14.754 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.754 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.754 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3,10", 2024-05-15T12:59:14.754 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.388305+0000", 2024-05-15T12:59:14.754 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00016605 2024-05-15T12:59:14.754 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.754 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.754 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.754 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.390707+0000", 2024-05-15T12:59:14.755 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0024013989999999998 2024-05-15T12:59:14.755 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.755 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.755 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:59:14.755 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.390736+0000", 2024-05-15T12:59:14.755 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.9150999999999999e-05 2024-05-15T12:59:14.755 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.755 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.755 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.755 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.400280+0000", 2024-05-15T12:59:14.755 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0095440190000000008 2024-05-15T12:59:14.755 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.755 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.755 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.756 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.402906+0000", 2024-05-15T12:59:14.756 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0026255850000000002 2024-05-15T12:59:14.756 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.756 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.756 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:59:14.756 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.402967+0000", 2024-05-15T12:59:14.756 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1748000000000006e-05 2024-05-15T12:59:14.756 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.756 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.756 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:59:14.756 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.403085+0000", 2024-05-15T12:59:14.756 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000117695 2024-05-15T12:59:14.756 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.756 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:59:14.757 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.757 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.757 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.757 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:90830 2.7 2:e14b3430:::benchmark_data_smithi177_78104_object90829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-05-15T12:59:14.757 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:49:17.287740+0000", 2024-05-15T12:59:14.757 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 596.46896368399996, 2024-05-15T12:59:14.757 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1222937009999998, 2024-05-15T12:59:14.757 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:59:14.757 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:59:14.757 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:59:14.757 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:59:14.757 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:59:14.757 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 90830 2024-05-15T12:59:14.757 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.758 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:59:14.758 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.758 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:59:14.758 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.287740+0000", 2024-05-15T12:59:14.758 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.758 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.758 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.758 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:59:14.758 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.287740+0000", 2024-05-15T12:59:14.758 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.758 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.758 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.758 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:59:14.758 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.287742+0000", 2024-05-15T12:59:14.759 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.278e-06 2024-05-15T12:59:14.759 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.759 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.759 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:59:14.759 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.287776+0000", 2024-05-15T12:59:14.759 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.4022000000000001e-05 2024-05-15T12:59:14.759 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.759 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.759 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:59:14.759 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.287798+0000", 2024-05-15T12:59:14.759 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1107999999999998e-05 2024-05-15T12:59:14.759 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.759 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.759 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:59:14.760 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.287808+0000", 2024-05-15T12:59:14.760 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0091e-05 2024-05-15T12:59:14.760 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.760 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.760 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.760 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.287874+0000", 2024-05-15T12:59:14.760 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.5922999999999999e-05 2024-05-15T12:59:14.760 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.760 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.760 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:59:14.760 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.287883+0000", 2024-05-15T12:59:14.760 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.8840000000000006e-06 2024-05-15T12:59:14.760 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.760 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.761 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.761 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856234+0000", 2024-05-15T12:59:14.761 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.568351037 2024-05-15T12:59:14.761 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.761 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.761 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:59:14.761 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856240+0000", 2024-05-15T12:59:14.761 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.361e-06 2024-05-15T12:59:14.761 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.761 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.761 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.761 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.390752+0000", 2024-05-15T12:59:14.761 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5345125959999999 2024-05-15T12:59:14.761 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.762 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.762 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:59:14.762 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.390833+0000", 2024-05-15T12:59:14.762 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0131999999999998e-05 2024-05-15T12:59:14.762 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.762 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.762 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3,10", 2024-05-15T12:59:14.762 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.390936+0000", 2024-05-15T12:59:14.762 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000103118 2024-05-15T12:59:14.762 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.762 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.762 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.762 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.397027+0000", 2024-05-15T12:59:14.762 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0060916310000000001 2024-05-15T12:59:14.763 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.763 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.763 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:59:14.763 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.397044+0000", 2024-05-15T12:59:14.763 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6484e-05 2024-05-15T12:59:14.763 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.763 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.763 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.763 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.400304+0000", 2024-05-15T12:59:14.763 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0032599310000000002 2024-05-15T12:59:14.763 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.763 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.763 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.764 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.409895+0000", 2024-05-15T12:59:14.764 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0095915229999999994 2024-05-15T12:59:14.764 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.764 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.764 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:59:14.764 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.409938+0000", 2024-05-15T12:59:14.764 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2889999999999998e-05 2024-05-15T12:59:14.764 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.764 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.764 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:59:14.764 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.410034+0000", 2024-05-15T12:59:14.764 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.5692000000000005e-05 2024-05-15T12:59:14.764 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.764 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:59:14.765 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.765 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.765 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.765 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:90834 2.7 2:eb30fe0e:::benchmark_data_smithi177_78104_object90833:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-05-15T12:59:14.765 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:49:17.293510+0000", 2024-05-15T12:59:14.765 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 596.46319373200004, 2024-05-15T12:59:14.765 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1167819890000001, 2024-05-15T12:59:14.765 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:59:14.765 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:59:14.765 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:59:14.765 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:59:14.765 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:59:14.765 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 90834 2024-05-15T12:59:14.765 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.765 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:59:14.766 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.766 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:59:14.766 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.293510+0000", 2024-05-15T12:59:14.766 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.766 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.766 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.766 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:59:14.766 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.293510+0000", 2024-05-15T12:59:14.766 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.766 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.766 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.766 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:59:14.766 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.293512+0000", 2024-05-15T12:59:14.767 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.44e-06 2024-05-15T12:59:14.767 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.767 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.767 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:59:14.767 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.293552+0000", 2024-05-15T12:59:14.767 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0200999999999997e-05 2024-05-15T12:59:14.767 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.767 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.767 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:59:14.767 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.293567+0000", 2024-05-15T12:59:14.767 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5213e-05 2024-05-15T12:59:14.767 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.767 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.767 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:59:14.767 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.293576+0000", 2024-05-15T12:59:14.768 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.9369999999999993e-06 2024-05-15T12:59:14.768 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.768 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.768 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.768 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.293645+0000", 2024-05-15T12:59:14.768 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9261e-05 2024-05-15T12:59:14.768 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.768 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.768 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:59:14.768 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.293654+0000", 2024-05-15T12:59:14.768 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0799999999999995e-06 2024-05-15T12:59:14.768 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.768 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.769 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.769 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856256+0000", 2024-05-15T12:59:14.769 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.56260164999999995 2024-05-15T12:59:14.769 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.769 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.769 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:59:14.769 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856260+0000", 2024-05-15T12:59:14.769 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0060000000000003e-06 2024-05-15T12:59:14.769 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.769 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.769 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.769 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.392511+0000", 2024-05-15T12:59:14.769 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.536250683 2024-05-15T12:59:14.769 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.769 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.770 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:59:14.770 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.392632+0000", 2024-05-15T12:59:14.770 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012167000000000001 2024-05-15T12:59:14.770 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.770 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.770 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3,10", 2024-05-15T12:59:14.770 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.392749+0000", 2024-05-15T12:59:14.770 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00011702899999999999 2024-05-15T12:59:14.770 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.770 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.770 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:59:14.770 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.397054+0000", 2024-05-15T12:59:14.770 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0043043259999999998 2024-05-15T12:59:14.770 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.771 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.771 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.771 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.397784+0000", 2024-05-15T12:59:14.771 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00073027400000000005 2024-05-15T12:59:14.771 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.771 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.771 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.771 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.400327+0000", 2024-05-15T12:59:14.771 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0025431529999999998 2024-05-15T12:59:14.771 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.771 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.771 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.771 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.410094+0000", 2024-05-15T12:59:14.771 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0097673640000000006 2024-05-15T12:59:14.772 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.772 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.772 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:59:14.772 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.410168+0000", 2024-05-15T12:59:14.772 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3959000000000001e-05 2024-05-15T12:59:14.772 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.772 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.772 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:59:14.772 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.410292+0000", 2024-05-15T12:59:14.772 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000123743 2024-05-15T12:59:14.772 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.772 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:59:14.773 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.773 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.773 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.773 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24562.0:90857 2.7 2:e20f5260:::benchmark_data_smithi177_78104_object90856:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-05-15T12:59:14.773 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:49:17.347336+0000", 2024-05-15T12:59:14.773 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 596.40936738200003, 2024-05-15T12:59:14.773 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0631595960000002, 2024-05-15T12:59:14.773 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:59:14.773 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:59:14.773 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:59:14.773 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24562", 2024-05-15T12:59:14.773 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/811327388", 2024-05-15T12:59:14.773 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 90857 2024-05-15T12:59:14.773 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.773 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:59:14.773 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.774 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:59:14.774 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.347336+0000", 2024-05-15T12:59:14.774 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.774 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.774 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.774 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:59:14.774 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.347336+0000", 2024-05-15T12:59:14.774 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.774 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.774 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.774 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:59:14.774 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.347338+0000", 2024-05-15T12:59:14.774 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9240000000000001e-06 2024-05-15T12:59:14.775 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.775 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.775 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:59:14.775 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.347368+0000", 2024-05-15T12:59:14.775 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.9694000000000001e-05 2024-05-15T12:59:14.775 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.775 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.775 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:59:14.775 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.347385+0000", 2024-05-15T12:59:14.775 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6595999999999998e-05 2024-05-15T12:59:14.775 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.775 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.775 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:59:14.775 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.347392+0000", 2024-05-15T12:59:14.775 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7219999999999999e-06 2024-05-15T12:59:14.775 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.776 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.776 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.776 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.347455+0000", 2024-05-15T12:59:14.776 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.2726999999999998e-05 2024-05-15T12:59:14.776 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.776 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.776 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:59:14.776 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.347464+0000", 2024-05-15T12:59:14.776 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8100000000000004e-06 2024-05-15T12:59:14.776 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.776 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.776 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.776 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856284+0000", 2024-05-15T12:59:14.776 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.50881985900000004 2024-05-15T12:59:14.776 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.777 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.777 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:59:14.777 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:17.856294+0000", 2024-05-15T12:59:14.777 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0404000000000001e-05 2024-05-15T12:59:14.777 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.777 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.777 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.777 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.394917+0000", 2024-05-15T12:59:14.777 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5386228559999999 2024-05-15T12:59:14.777 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.777 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.777 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:59:14.777 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.395044+0000", 2024-05-15T12:59:14.777 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012691499999999999 2024-05-15T12:59:14.778 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.778 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.778 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3,10", 2024-05-15T12:59:14.778 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.395187+0000", 2024-05-15T12:59:14.778 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000143468 2024-05-15T12:59:14.778 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.778 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.778 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:59:14.778 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.397060+0000", 2024-05-15T12:59:14.778 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0018721020000000001 2024-05-15T12:59:14.778 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.778 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.778 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.779 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.397823+0000", 2024-05-15T12:59:14.779 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00076354299999999999 2024-05-15T12:59:14.779 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.779 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.779 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.779 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.403147+0000", 2024-05-15T12:59:14.779 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0053240650000000002 2024-05-15T12:59:14.779 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.779 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.779 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.779 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.410339+0000", 2024-05-15T12:59:14.779 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0071920480000000004 2024-05-15T12:59:14.779 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.779 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.779 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:59:14.779 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.410392+0000", 2024-05-15T12:59:14.779 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2840999999999998e-05 2024-05-15T12:59:14.780 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.780 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.780 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:59:14.780 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:49:19.410496+0000", 2024-05-15T12:59:14.780 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000104022 2024-05-15T12:59:14.780 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.780 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:59:14.780 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.780 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.780 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.780 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:79596 4.a 4:506f4fa1:::benchmark_data_smithi177_99756_object79595:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e331)", 2024-05-15T12:59:14.780 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:58:51.634707+0000", 2024-05-15T12:59:14.780 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 22.121997046000001, 2024-05-15T12:59:14.780 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.050835246000000001, 2024-05-15T12:59:14.781 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:59:14.781 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:59:14.781 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:59:14.781 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T12:59:14.781 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T12:59:14.781 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 79596 2024-05-15T12:59:14.781 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.781 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:59:14.781 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.781 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:59:14.781 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:58:51.634707+0000", 2024-05-15T12:59:14.781 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.781 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.781 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.781 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:59:14.781 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:58:51.634707+0000", 2024-05-15T12:59:14.782 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.782 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.782 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.782 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:59:14.782 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:58:51.634708+0000", 2024-05-15T12:59:14.782 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.083e-06 2024-05-15T12:59:14.782 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.782 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.782 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:59:14.782 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:58:51.634746+0000", 2024-05-15T12:59:14.782 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.8448000000000002e-05 2024-05-15T12:59:14.782 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.782 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.782 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:59:14.783 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:58:51.634755+0000", 2024-05-15T12:59:14.783 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.3000000000000002e-06 2024-05-15T12:59:14.783 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.783 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.783 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:59:14.783 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:58:51.634760+0000", 2024-05-15T12:59:14.783 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6670000000000001e-06 2024-05-15T12:59:14.783 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.783 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.783 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.783 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:58:51.634803+0000", 2024-05-15T12:59:14.783 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2577999999999998e-05 2024-05-15T12:59:14.783 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.783 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.783 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:59:14.784 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:58:51.634886+0000", 2024-05-15T12:59:14.784 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.3432999999999999e-05 2024-05-15T12:59:14.784 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.784 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.784 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,10", 2024-05-15T12:59:14.784 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:58:51.634966+0000", 2024-05-15T12:59:14.784 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0154999999999994e-05 2024-05-15T12:59:14.784 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.784 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.784 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:59:14.784 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:58:51.638550+0000", 2024-05-15T12:59:14.784 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.003583925 2024-05-15T12:59:14.784 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.784 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.785 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.785 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:58:51.638577+0000", 2024-05-15T12:59:14.785 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.6701000000000001e-05 2024-05-15T12:59:14.785 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.785 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.785 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.785 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:58:51.685445+0000", 2024-05-15T12:59:14.785 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.046867974 2024-05-15T12:59:14.785 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.785 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.785 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:59:14.785 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:58:51.685500+0000", 2024-05-15T12:59:14.785 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5158999999999998e-05 2024-05-15T12:59:14.785 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.785 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.786 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:59:14.786 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:58:51.685542+0000", 2024-05-15T12:59:14.786 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.1823000000000003e-05 2024-05-15T12:59:14.786 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.786 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:59:14.786 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.786 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.786 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.786 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:101368 4.5 4:abf3cbb1:::benchmark_data_smithi177_99756_object101367:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-05-15T12:59:14.786 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:59:10.304149+0000", 2024-05-15T12:59:14.786 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 3.45255439, 2024-05-15T12:59:14.786 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.5083956729999999, 2024-05-15T12:59:14.786 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:59:14.786 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T12:59:14.787 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:59:14.787 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T12:59:14.787 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T12:59:14.787 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 101368 2024-05-15T12:59:14.787 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.787 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:59:14.787 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.787 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:59:14.787 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.304149+0000", 2024-05-15T12:59:14.787 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.787 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.787 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.787 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:59:14.787 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.304149+0000", 2024-05-15T12:59:14.787 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.787 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.788 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.788 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:59:14.788 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.304150+0000", 2024-05-15T12:59:14.788 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2099999999999997e-07 2024-05-15T12:59:14.788 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.788 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.788 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:59:14.788 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.304218+0000", 2024-05-15T12:59:14.788 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8374999999999995e-05 2024-05-15T12:59:14.788 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.788 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.788 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:59:14.788 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.304226+0000", 2024-05-15T12:59:14.788 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3590000000000002e-06 2024-05-15T12:59:14.788 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.788 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.789 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:59:14.789 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.304232+0000", 2024-05-15T12:59:14.789 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1659999999999999e-06 2024-05-15T12:59:14.789 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.789 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.789 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.789 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.304280+0000", 2024-05-15T12:59:14.789 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7682000000000002e-05 2024-05-15T12:59:14.789 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.789 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.789 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:59:14.789 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.304384+0000", 2024-05-15T12:59:14.789 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00010478499999999999 2024-05-15T12:59:14.789 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.789 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.789 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,10", 2024-05-15T12:59:14.790 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.304491+0000", 2024-05-15T12:59:14.790 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000106982 2024-05-15T12:59:14.790 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.790 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.790 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:59:14.790 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.307821+0000", 2024-05-15T12:59:14.790 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0033297019999999999 2024-05-15T12:59:14.790 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.790 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.790 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.790 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.308306+0000", 2024-05-15T12:59:14.790 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00048475299999999999 2024-05-15T12:59:14.790 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.790 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.791 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.791 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.315153+0000", 2024-05-15T12:59:14.791 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.006847517 2024-05-15T12:59:14.791 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.791 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.791 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:59:14.791 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.315161+0000", 2024-05-15T12:59:14.791 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.255e-06 2024-05-15T12:59:14.791 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.791 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.791 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.791 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312207+0000", 2024-05-15T12:59:14.791 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.99704610400000004 2024-05-15T12:59:14.791 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.791 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.791 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:59:14.792 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312223+0000", 2024-05-15T12:59:14.792 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6019999999999999e-05 2024-05-15T12:59:14.792 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.792 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.792 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.792 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.812476+0000", 2024-05-15T12:59:14.792 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.500253813 2024-05-15T12:59:14.792 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.792 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.792 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:59:14.792 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.812545+0000", 2024-05-15T12:59:14.792 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8639000000000005e-05 2024-05-15T12:59:14.792 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.792 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:59:14.793 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.793 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.793 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.793 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:101391 4.8 4:1d63c299:::benchmark_data_smithi177_99756_object101390:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e343)", 2024-05-15T12:59:14.793 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:59:11.312996+0000", 2024-05-15T12:59:14.793 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 2.4437075899999998, 2024-05-15T12:59:14.793 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.572125896, 2024-05-15T12:59:14.793 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:59:14.793 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:59:14.793 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:59:14.793 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T12:59:14.793 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T12:59:14.793 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 101391 2024-05-15T12:59:14.793 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.794 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:59:14.794 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.794 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:59:14.794 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312996+0000", 2024-05-15T12:59:14.794 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.794 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.794 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.794 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:59:14.794 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312996+0000", 2024-05-15T12:59:14.794 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.794 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.794 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.794 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:59:14.795 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312998+0000", 2024-05-15T12:59:14.795 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6750000000000001e-06 2024-05-15T12:59:14.795 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.795 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.795 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:59:14.795 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313176+0000", 2024-05-15T12:59:14.795 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00017834599999999999 2024-05-15T12:59:14.795 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.795 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.795 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:59:14.795 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313191+0000", 2024-05-15T12:59:14.795 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4741e-05 2024-05-15T12:59:14.795 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.795 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.796 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:59:14.796 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313201+0000", 2024-05-15T12:59:14.796 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.5370000000000003e-06 2024-05-15T12:59:14.796 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.796 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.796 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.796 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313254+0000", 2024-05-15T12:59:14.796 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3526999999999998e-05 2024-05-15T12:59:14.796 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.796 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.796 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T12:59:14.796 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313260+0000", 2024-05-15T12:59:14.796 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8590000000000001e-06 2024-05-15T12:59:14.796 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.797 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.797 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.797 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.867433+0000", 2024-05-15T12:59:14.797 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5541727939999999 2024-05-15T12:59:14.797 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.797 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.797 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:59:14.797 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.867550+0000", 2024-05-15T12:59:14.797 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00011683 2024-05-15T12:59:14.797 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.797 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.797 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 9,11", 2024-05-15T12:59:14.797 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.867653+0000", 2024-05-15T12:59:14.798 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000103139 2024-05-15T12:59:14.798 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.798 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.798 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.798 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.878453+0000", 2024-05-15T12:59:14.798 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.010800149 2024-05-15T12:59:14.798 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.798 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.798 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:59:14.798 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.880516+0000", 2024-05-15T12:59:14.798 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0020627269999999999 2024-05-15T12:59:14.798 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.798 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.798 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.799 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.880623+0000", 2024-05-15T12:59:14.799 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000107155 2024-05-15T12:59:14.799 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.799 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.799 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:59:14.799 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.880670+0000", 2024-05-15T12:59:14.799 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6801999999999997e-05 2024-05-15T12:59:14.799 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.799 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.799 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:59:14.799 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.885122+0000", 2024-05-15T12:59:14.799 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.004452615 2024-05-15T12:59:14.799 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.799 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:59:14.800 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.800 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.800 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.800 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:101723 4.5 4:a65f5301:::benchmark_data_smithi177_99756_object101722:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-05-15T12:59:14.800 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:59:13.267425+0000", 2024-05-15T12:59:14.800 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.489279254, 2024-05-15T12:59:14.800 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.063578888, 2024-05-15T12:59:14.800 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:59:14.800 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:59:14.800 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:59:14.800 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T12:59:14.800 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T12:59:14.800 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 101723 2024-05-15T12:59:14.800 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.801 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:59:14.801 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.801 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:59:14.801 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.267425+0000", 2024-05-15T12:59:14.801 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.801 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.801 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.801 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:59:14.801 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.267425+0000", 2024-05-15T12:59:14.801 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.801 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.801 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.801 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:59:14.801 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.267426+0000", 2024-05-15T12:59:14.802 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0780000000000001e-06 2024-05-15T12:59:14.802 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.802 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.802 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:59:14.802 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.267485+0000", 2024-05-15T12:59:14.802 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9134e-05 2024-05-15T12:59:14.802 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.802 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.802 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:59:14.802 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.267500+0000", 2024-05-15T12:59:14.802 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5160000000000001e-05 2024-05-15T12:59:14.802 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.802 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.802 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:59:14.803 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.267508+0000", 2024-05-15T12:59:14.803 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9470000000000003e-06 2024-05-15T12:59:14.803 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.803 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.803 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.803 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.280816+0000", 2024-05-15T12:59:14.803 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.013308154000000001 2024-05-15T12:59:14.803 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.803 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.803 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:59:14.803 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.280967+0000", 2024-05-15T12:59:14.803 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015127700000000001 2024-05-15T12:59:14.803 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.803 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.804 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 10,11", 2024-05-15T12:59:14.804 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.281156+0000", 2024-05-15T12:59:14.804 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00018889699999999999 2024-05-15T12:59:14.804 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.804 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.804 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:59:14.804 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.295582+0000", 2024-05-15T12:59:14.804 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.014425879000000001 2024-05-15T12:59:14.804 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.804 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.804 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.804 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.295651+0000", 2024-05-15T12:59:14.804 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8696999999999996e-05 2024-05-15T12:59:14.804 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.805 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.805 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.805 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.327869+0000", 2024-05-15T12:59:14.805 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.032217810999999999 2024-05-15T12:59:14.805 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.805 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.805 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:59:14.805 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.327945+0000", 2024-05-15T12:59:14.805 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.6693999999999994e-05 2024-05-15T12:59:14.805 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.805 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.805 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:59:14.805 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.331003+0000", 2024-05-15T12:59:14.805 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0030581599999999999 2024-05-15T12:59:14.805 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.806 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:59:14.806 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.806 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.806 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.806 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:101740 4.5 4:a6534756:::benchmark_data_smithi177_99756_object101739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-05-15T12:59:14.806 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:59:13.284316+0000", 2024-05-15T12:59:14.806 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.47238761699999998, 2024-05-15T12:59:14.806 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.055135057000000001, 2024-05-15T12:59:14.806 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:59:14.806 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:59:14.806 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:59:14.806 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T12:59:14.806 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T12:59:14.807 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 101740 2024-05-15T12:59:14.807 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.807 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:59:14.807 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.807 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:59:14.807 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.284316+0000", 2024-05-15T12:59:14.807 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.807 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.807 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.807 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:59:14.807 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.284316+0000", 2024-05-15T12:59:14.807 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.807 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.807 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.808 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:59:14.808 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.284318+0000", 2024-05-15T12:59:14.808 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5099999999999999e-06 2024-05-15T12:59:14.808 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.808 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.808 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:59:14.808 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.284379+0000", 2024-05-15T12:59:14.808 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.0883e-05 2024-05-15T12:59:14.808 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.808 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.808 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:59:14.808 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.284391+0000", 2024-05-15T12:59:14.808 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2802e-05 2024-05-15T12:59:14.808 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.808 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.809 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:59:14.809 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.284399+0000", 2024-05-15T12:59:14.809 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8690000000000004e-06 2024-05-15T12:59:14.809 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.809 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.809 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.809 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.297688+0000", 2024-05-15T12:59:14.809 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.013288400000000001 2024-05-15T12:59:14.809 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.809 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.809 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:59:14.809 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.297809+0000", 2024-05-15T12:59:14.809 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000121249 2024-05-15T12:59:14.809 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.810 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.810 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 10,11", 2024-05-15T12:59:14.810 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.297922+0000", 2024-05-15T12:59:14.810 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000113433 2024-05-15T12:59:14.810 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.810 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.810 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:59:14.810 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.303229+0000", 2024-05-15T12:59:14.810 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0053063199999999998 2024-05-15T12:59:14.810 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.810 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.810 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.810 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.317876+0000", 2024-05-15T12:59:14.810 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.014647403 2024-05-15T12:59:14.811 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.811 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.811 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.811 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.336470+0000", 2024-05-15T12:59:14.811 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.018593810999999998 2024-05-15T12:59:14.811 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.811 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.811 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:59:14.811 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.336530+0000", 2024-05-15T12:59:14.811 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.0485000000000001e-05 2024-05-15T12:59:14.811 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.811 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.811 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:59:14.811 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.339451+0000", 2024-05-15T12:59:14.811 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.002920892 2024-05-15T12:59:14.811 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.812 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:59:14.812 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.812 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.812 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.812 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:101741 4.5 4:a293ac2b:::benchmark_data_smithi177_99756_object101740:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-05-15T12:59:14.812 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:59:13.284710+0000", 2024-05-15T12:59:14.812 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.47199407799999998, 2024-05-15T12:59:14.812 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.057751365999999998, 2024-05-15T12:59:14.812 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:59:14.812 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:59:14.812 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:59:14.812 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T12:59:14.812 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T12:59:14.812 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 101741 2024-05-15T12:59:14.812 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.813 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:59:14.813 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.813 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:59:14.813 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.284710+0000", 2024-05-15T12:59:14.813 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.813 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.813 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.813 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:59:14.813 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.284710+0000", 2024-05-15T12:59:14.813 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.813 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.813 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.813 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:59:14.813 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.284711+0000", 2024-05-15T12:59:14.813 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0800000000000004e-07 2024-05-15T12:59:14.813 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.814 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.814 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:59:14.814 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.284780+0000", 2024-05-15T12:59:14.814 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9277999999999996e-05 2024-05-15T12:59:14.814 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.814 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.814 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:59:14.814 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.284790+0000", 2024-05-15T12:59:14.814 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0390999999999999e-05 2024-05-15T12:59:14.814 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.814 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.814 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:59:14.814 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.284798+0000", 2024-05-15T12:59:14.815 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.6699999999999994e-06 2024-05-15T12:59:14.815 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.815 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.815 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.815 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.300540+0000", 2024-05-15T12:59:14.815 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.015742335999999999 2024-05-15T12:59:14.815 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.815 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.815 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:59:14.815 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.300698+0000", 2024-05-15T12:59:14.815 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015750599999999999 2024-05-15T12:59:14.815 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.815 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.815 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 10,11", 2024-05-15T12:59:14.816 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.300872+0000", 2024-05-15T12:59:14.816 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00017409000000000001 2024-05-15T12:59:14.816 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.816 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.816 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:59:14.816 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.304992+0000", 2024-05-15T12:59:14.816 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0041198019999999997 2024-05-15T12:59:14.816 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.816 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.816 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.816 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.317929+0000", 2024-05-15T12:59:14.816 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.012936902 2024-05-15T12:59:14.816 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.817 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.817 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.817 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.339517+0000", 2024-05-15T12:59:14.817 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.021588145999999999 2024-05-15T12:59:14.817 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.817 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.817 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:59:14.817 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.339556+0000", 2024-05-15T12:59:14.817 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.9323999999999998e-05 2024-05-15T12:59:14.818 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.818 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.818 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:59:14.818 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.342461+0000", 2024-05-15T12:59:14.818 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.002905113 2024-05-15T12:59:14.818 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.818 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:59:14.818 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.818 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.818 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.818 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:101796 4.5 4:afa0a283:::benchmark_data_smithi177_99756_object101795:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-05-15T12:59:14.819 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:59:13.359227+0000", 2024-05-15T12:59:14.819 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.39747719100000001, 2024-05-15T12:59:14.819 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.071920303000000005, 2024-05-15T12:59:14.819 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:59:14.819 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:59:14.819 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:59:14.819 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T12:59:14.819 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T12:59:14.819 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 101796 2024-05-15T12:59:14.819 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.819 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:59:14.819 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.819 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:59:14.819 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.359227+0000", 2024-05-15T12:59:14.819 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.819 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.820 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.820 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:59:14.820 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.359227+0000", 2024-05-15T12:59:14.820 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.820 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.820 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.820 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:59:14.820 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.359228+0000", 2024-05-15T12:59:14.820 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.043e-06 2024-05-15T12:59:14.820 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.820 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.820 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:59:14.820 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.359325+0000", 2024-05-15T12:59:14.820 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.7481999999999999e-05 2024-05-15T12:59:14.820 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.820 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.821 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:59:14.821 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.359335+0000", 2024-05-15T12:59:14.821 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.8020000000000003e-06 2024-05-15T12:59:14.821 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.821 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.821 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:59:14.821 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.359342+0000", 2024-05-15T12:59:14.821 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8140000000000003e-06 2024-05-15T12:59:14.821 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.821 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.821 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.821 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.366338+0000", 2024-05-15T12:59:14.821 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0069957409999999998 2024-05-15T12:59:14.821 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.822 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.822 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:59:14.822 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.366491+0000", 2024-05-15T12:59:14.822 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015328299999999999 2024-05-15T12:59:14.822 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.822 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.822 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 10,11", 2024-05-15T12:59:14.822 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.366664+0000", 2024-05-15T12:59:14.822 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00017284099999999999 2024-05-15T12:59:14.822 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.822 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.822 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:59:14.822 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.380128+0000", 2024-05-15T12:59:14.822 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.013464166 2024-05-15T12:59:14.823 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.823 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.823 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.823 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.380207+0000", 2024-05-15T12:59:14.823 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8802999999999998e-05 2024-05-15T12:59:14.823 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.823 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.823 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.823 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.426879+0000", 2024-05-15T12:59:14.823 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.046672294000000003 2024-05-15T12:59:14.823 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.823 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.823 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:59:14.824 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.426946+0000", 2024-05-15T12:59:14.824 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6717999999999994e-05 2024-05-15T12:59:14.824 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.824 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.824 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:59:14.824 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.431147+0000", 2024-05-15T12:59:14.824 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0042013160000000001 2024-05-15T12:59:14.824 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.824 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:59:14.824 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.824 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.824 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.824 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:101801 4.5 4:af0d9b21:::benchmark_data_smithi177_99756_object101800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-05-15T12:59:14.824 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:59:13.364572+0000", 2024-05-15T12:59:14.824 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.39213145300000002, 2024-05-15T12:59:14.825 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.069812989000000006, 2024-05-15T12:59:14.825 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:59:14.825 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:59:14.825 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:59:14.825 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T12:59:14.825 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T12:59:14.825 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 101801 2024-05-15T12:59:14.825 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.825 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:59:14.825 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.825 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:59:14.825 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.364572+0000", 2024-05-15T12:59:14.825 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.825 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.825 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.825 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:59:14.826 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.364572+0000", 2024-05-15T12:59:14.826 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.826 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.826 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.826 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:59:14.826 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.364573+0000", 2024-05-15T12:59:14.826 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.08e-06 2024-05-15T12:59:14.826 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.826 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.826 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:59:14.826 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.364605+0000", 2024-05-15T12:59:14.826 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.1350999999999998e-05 2024-05-15T12:59:14.826 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.827 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.827 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:59:14.827 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.364615+0000", 2024-05-15T12:59:14.827 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0444e-05 2024-05-15T12:59:14.827 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.827 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.827 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:59:14.827 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.364621+0000", 2024-05-15T12:59:14.827 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9309999999999996e-06 2024-05-15T12:59:14.827 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.827 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.827 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.827 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.369677+0000", 2024-05-15T12:59:14.827 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0050561369999999996 2024-05-15T12:59:14.827 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.827 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.828 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:59:14.828 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.369794+0000", 2024-05-15T12:59:14.828 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000116286 2024-05-15T12:59:14.828 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.828 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.828 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 10,11", 2024-05-15T12:59:14.828 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.369912+0000", 2024-05-15T12:59:14.828 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000118281 2024-05-15T12:59:14.828 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.828 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.828 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:59:14.828 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.380136+0000", 2024-05-15T12:59:14.828 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.010223727 2024-05-15T12:59:14.828 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.828 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.828 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.829 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.382893+0000", 2024-05-15T12:59:14.829 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0027569700000000001 2024-05-15T12:59:14.829 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.829 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.829 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.829 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.431218+0000", 2024-05-15T12:59:14.829 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.048325878000000003 2024-05-15T12:59:14.829 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.829 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.829 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:59:14.829 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.431288+0000", 2024-05-15T12:59:14.829 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9729999999999998e-05 2024-05-15T12:59:14.829 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.830 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.830 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:59:14.830 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.434385+0000", 2024-05-15T12:59:14.830 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0030971739999999998 2024-05-15T12:59:14.830 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.830 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:59:14.830 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.830 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.830 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.830 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:101804 4.5 4:a0b3bf2a:::benchmark_data_smithi177_99756_object101803:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-05-15T12:59:14.830 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:59:13.364753+0000", 2024-05-15T12:59:14.830 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.39195128400000001, 2024-05-15T12:59:14.830 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.073640627, 2024-05-15T12:59:14.830 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:59:14.831 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:59:14.831 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:59:14.831 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T12:59:14.831 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T12:59:14.831 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 101804 2024-05-15T12:59:14.831 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.831 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:59:14.831 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.831 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:59:14.831 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.364753+0000", 2024-05-15T12:59:14.831 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.831 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.831 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.831 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:59:14.831 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.364753+0000", 2024-05-15T12:59:14.832 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.832 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.832 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.832 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:59:14.832 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.364753+0000", 2024-05-15T12:59:14.832 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.5400000000000002e-07 2024-05-15T12:59:14.832 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.832 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.832 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:59:14.832 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.364820+0000", 2024-05-15T12:59:14.832 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7408000000000002e-05 2024-05-15T12:59:14.832 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.832 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.832 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:59:14.833 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.364830+0000", 2024-05-15T12:59:14.833 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0023999999999999e-05 2024-05-15T12:59:14.833 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.833 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.833 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:59:14.833 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.364837+0000", 2024-05-15T12:59:14.833 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.212e-06 2024-05-15T12:59:14.833 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.833 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.833 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.833 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.372096+0000", 2024-05-15T12:59:14.833 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0072589610000000004 2024-05-15T12:59:14.833 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.833 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.834 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:59:14.834 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.372248+0000", 2024-05-15T12:59:14.834 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000152664 2024-05-15T12:59:14.834 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.834 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.834 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 10,11", 2024-05-15T12:59:14.834 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.372439+0000", 2024-05-15T12:59:14.834 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00019026700000000001 2024-05-15T12:59:14.834 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.834 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.834 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:59:14.834 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.380145+0000", 2024-05-15T12:59:14.834 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0077068809999999996 2024-05-15T12:59:14.835 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.835 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.835 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.835 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.382933+0000", 2024-05-15T12:59:14.835 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0027873389999999998 2024-05-15T12:59:14.835 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.835 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.835 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.835 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.434435+0000", 2024-05-15T12:59:14.835 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.051501795000000003 2024-05-15T12:59:14.835 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.835 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.835 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:59:14.836 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.434500+0000", 2024-05-15T12:59:14.836 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.5273999999999999e-05 2024-05-15T12:59:14.836 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.836 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.836 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:59:14.836 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.438393+0000", 2024-05-15T12:59:14.836 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.003893348 2024-05-15T12:59:14.836 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.836 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:59:14.836 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.836 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.836 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.836 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:101805 4.5 4:a87ccb52:::benchmark_data_smithi177_99756_object101804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-05-15T12:59:14.836 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:59:13.364844+0000", 2024-05-15T12:59:14.836 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.39185952299999999, 2024-05-15T12:59:14.837 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.077223218999999996, 2024-05-15T12:59:14.837 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:59:14.837 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:59:14.837 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:59:14.837 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T12:59:14.837 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T12:59:14.837 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 101805 2024-05-15T12:59:14.837 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.837 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:59:14.837 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.837 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:59:14.837 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.364844+0000", 2024-05-15T12:59:14.837 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.838 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.838 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.838 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:59:14.838 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.364844+0000", 2024-05-15T12:59:14.838 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.838 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.838 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.838 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:59:14.838 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.364845+0000", 2024-05-15T12:59:14.838 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.65e-07 2024-05-15T12:59:14.838 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.838 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.838 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:59:14.839 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.364875+0000", 2024-05-15T12:59:14.839 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.0110000000000001e-05 2024-05-15T12:59:14.839 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.839 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.839 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:59:14.839 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.364881+0000", 2024-05-15T12:59:14.839 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8159999999999999e-06 2024-05-15T12:59:14.839 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.839 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.839 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:59:14.839 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.364884+0000", 2024-05-15T12:59:14.839 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.585e-06 2024-05-15T12:59:14.839 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.839 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.840 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.840 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.374583+0000", 2024-05-15T12:59:14.840 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0096988850000000008 2024-05-15T12:59:14.840 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.840 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.840 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:59:14.840 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.374764+0000", 2024-05-15T12:59:14.840 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00018114800000000001 2024-05-15T12:59:14.840 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.840 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.840 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 10,11", 2024-05-15T12:59:14.840 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.375006+0000", 2024-05-15T12:59:14.840 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00024128800000000001 2024-05-15T12:59:14.840 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.841 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.841 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:59:14.841 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.380154+0000", 2024-05-15T12:59:14.841 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0051483529999999996 2024-05-15T12:59:14.841 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.841 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.841 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.841 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.382961+0000", 2024-05-15T12:59:14.841 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0028070899999999999 2024-05-15T12:59:14.841 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.841 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.841 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.841 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.438460+0000", 2024-05-15T12:59:14.841 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.055498935999999999 2024-05-15T12:59:14.841 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.842 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.842 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:59:14.842 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.438533+0000", 2024-05-15T12:59:14.842 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3056e-05 2024-05-15T12:59:14.842 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.842 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.842 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:59:14.842 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.442068+0000", 2024-05-15T12:59:14.842 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0035345870000000001 2024-05-15T12:59:14.842 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.842 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:59:14.842 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.842 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.842 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.843 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:101806 4.5 4:a9245b89:::benchmark_data_smithi177_99756_object101805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-05-15T12:59:14.843 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:59:13.368833+0000", 2024-05-15T12:59:14.843 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.38787094999999999, 2024-05-15T12:59:14.843 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.077197417000000004, 2024-05-15T12:59:14.843 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T12:59:14.843 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T12:59:14.843 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T12:59:14.843 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T12:59:14.843 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T12:59:14.843 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 101806 2024-05-15T12:59:14.843 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.843 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T12:59:14.843 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.844 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T12:59:14.844 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.368833+0000", 2024-05-15T12:59:14.844 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.844 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.844 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.844 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T12:59:14.844 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.368833+0000", 2024-05-15T12:59:14.844 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T12:59:14.844 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.844 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.844 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T12:59:14.844 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.368834+0000", 2024-05-15T12:59:14.844 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2419999999999999e-06 2024-05-15T12:59:14.844 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.844 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.844 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T12:59:14.845 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.368886+0000", 2024-05-15T12:59:14.845 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1894999999999997e-05 2024-05-15T12:59:14.845 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.845 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.845 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T12:59:14.845 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.368897+0000", 2024-05-15T12:59:14.845 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1379999999999999e-05 2024-05-15T12:59:14.845 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.845 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.845 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T12:59:14.845 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.368905+0000", 2024-05-15T12:59:14.845 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4070000000000004e-06 2024-05-15T12:59:14.845 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.845 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.845 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T12:59:14.846 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.377637+0000", 2024-05-15T12:59:14.846 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0087321740000000005 2024-05-15T12:59:14.846 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.846 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.846 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T12:59:14.846 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.377796+0000", 2024-05-15T12:59:14.846 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000158529 2024-05-15T12:59:14.846 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.846 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.846 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 10,11", 2024-05-15T12:59:14.846 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.377977+0000", 2024-05-15T12:59:14.846 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00018155800000000001 2024-05-15T12:59:14.846 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.846 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.847 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T12:59:14.847 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.380247+0000", 2024-05-15T12:59:14.847 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.002269999 2024-05-15T12:59:14.847 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.847 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.847 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.847 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.393277+0000", 2024-05-15T12:59:14.847 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.013030026 2024-05-15T12:59:14.847 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.847 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.847 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T12:59:14.847 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.442131+0000", 2024-05-15T12:59:14.847 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.048854017 2024-05-15T12:59:14.847 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.847 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.848 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T12:59:14.848 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.442170+0000", 2024-05-15T12:59:14.848 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.9267000000000002e-05 2024-05-15T12:59:14.848 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T12:59:14.848 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T12:59:14.848 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T12:59:14.848 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:13.446030+0000", 2024-05-15T12:59:14.848 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0038599229999999999 2024-05-15T12:59:14.848 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.848 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:59:14.848 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.848 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T12:59:14.848 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T12:59:14.848 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T12:59:14.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:14 smithi027 ceph-mon[39795]: pgmap v932: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 peering, 13 active+clean; 6.1 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 32 MiB/s wr, 506 op/s; 12865/301443 objects degraded (4.268%); 6371/301443 objects misplaced (2.114%); 192 KiB/s, 2 objects/s recovering 2024-05-15T12:59:14.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:14 smithi027 ceph-mon[39795]: osdmap e346: 12 total, 12 up, 11 in 2024-05-15T12:59:14.850 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:14 smithi120 ceph-mon[43961]: pgmap v932: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 peering, 13 active+clean; 6.1 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 32 MiB/s wr, 506 op/s; 12865/301443 objects degraded (4.268%); 6371/301443 objects misplaced (2.114%); 192 KiB/s, 2 objects/s recovering 2024-05-15T12:59:14.850 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:14 smithi120 ceph-mon[43961]: osdmap e346: 12 total, 12 up, 11 in 2024-05-15T12:59:15.135 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 66 16 103105 103089 97.6093 47.25 0.0423426 0.01024 2024-05-15T12:59:15.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:15 smithi177 ceph-mon[44303]: Health check failed: Degraded data redundancy: 12865/301443 objects degraded (4.268%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:15.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:15 smithi027 ceph-mon[39795]: Health check failed: Degraded data redundancy: 12865/301443 objects degraded (4.268%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:15.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:15 smithi120 ceph-mon[43961]: Health check failed: Degraded data redundancy: 12865/301443 objects degraded (4.268%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:16.135 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 67 16 103808 103792 96.8082 43.9375 0.0432872 0.0103247 2024-05-15T12:59:16.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:16 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:59:16.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:16 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:59:16.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:16 smithi177 ceph-mon[44303]: osdmap e347: 12 total, 12 up, 11 in 2024-05-15T12:59:16.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:16 smithi177 ceph-mon[44303]: pgmap v935: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 10 active+clean; 6.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 41 MiB/s wr, 659 op/s; 19153/307608 objects degraded (6.226%); 25075/307608 objects misplaced (8.152%); 1.8 MiB/s, 28 objects/s recovering 2024-05-15T12:59:16.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:16 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:59:16.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:16 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:59:16.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:16 smithi027 ceph-mon[39795]: osdmap e347: 12 total, 12 up, 11 in 2024-05-15T12:59:16.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:16 smithi027 ceph-mon[39795]: pgmap v935: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 10 active+clean; 6.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 41 MiB/s wr, 659 op/s; 19153/307608 objects degraded (6.226%); 25075/307608 objects misplaced (8.152%); 1.8 MiB/s, 28 objects/s recovering 2024-05-15T12:59:16.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:16 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:59:16.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:16 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:59:16.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:16 smithi120 ceph-mon[43961]: osdmap e347: 12 total, 12 up, 11 in 2024-05-15T12:59:16.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:16 smithi120 ceph-mon[43961]: pgmap v935: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 10 active+clean; 6.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 41 MiB/s wr, 659 op/s; 19153/307608 objects degraded (6.226%); 25075/307608 objects misplaced (8.152%); 1.8 MiB/s, 28 objects/s recovering 2024-05-15T12:59:17.135 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 68 16 104586 104570 96.0996 48.625 0.0489589 0.0104014 2024-05-15T12:59:18.136 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 69 16 105255 105239 95.3128 41.8125 0.00996046 0.0104879 2024-05-15T12:59:18.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:17 smithi177 ceph-mon[44303]: pgmap v936: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.3 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 35 MiB/s wr, 552 op/s; 19153/308817 objects degraded (6.202%); 25075/308817 objects misplaced (8.120%); 1.4 MiB/s, 21 objects/s recovering 2024-05-15T12:59:18.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:17 smithi027 ceph-mon[39795]: pgmap v936: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.3 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 35 MiB/s wr, 552 op/s; 19153/308817 objects degraded (6.202%); 25075/308817 objects misplaced (8.120%); 1.4 MiB/s, 21 objects/s recovering 2024-05-15T12:59:18.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:17 smithi120 ceph-mon[43961]: pgmap v936: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.3 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 35 MiB/s wr, 552 op/s; 19153/308817 objects degraded (6.202%); 25075/308817 objects misplaced (8.120%); 1.4 MiB/s, 21 objects/s recovering 2024-05-15T12:59:18.819 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T12:59:18.819 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T12:59:18.819 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-05-15T12:59:18.819 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd dump -f json-pretty 2024-05-15T12:59:18.937 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:59:18.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:18.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:59:19.041 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:59:19.097 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:19.135 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 70 16 106001 105985 94.6172 46.625 0.0113376 0.0105638 2024-05-15T12:59:19.140 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:59:19.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:19.145 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:59:19.246 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:59:19.300 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:19.328 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:59:19.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:19.337 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:59:19.439 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:59:19.471 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:20.136 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 71 16 106648 106632 93.8541 40.4375 0.0337945 0.0106491 2024-05-15T12:59:20.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:19 smithi177 ceph-mon[44303]: pgmap v937: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.4 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 46 MiB/s wr, 743 op/s; 19153/314949 objects degraded (6.081%); 24735/314949 objects misplaced (7.854%); 4.3 MiB/s, 68 objects/s recovering 2024-05-15T12:59:20.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:19 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 19153/314949 objects degraded (6.081%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:20.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:19 smithi027 ceph-mon[39795]: pgmap v937: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.4 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 46 MiB/s wr, 743 op/s; 19153/314949 objects degraded (6.081%); 24735/314949 objects misplaced (7.854%); 4.3 MiB/s, 68 objects/s recovering 2024-05-15T12:59:20.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:19 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 19153/314949 objects degraded (6.081%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:20.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:19 smithi120 ceph-mon[43961]: pgmap v937: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.4 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 46 MiB/s wr, 743 op/s; 19153/314949 objects degraded (6.081%); 24735/314949 objects misplaced (7.854%); 4.3 MiB/s, 68 objects/s recovering 2024-05-15T12:59:20.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:19 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 19153/314949 objects degraded (6.081%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:21.136 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 72 16 107363 107347 93.1712 44.6875 0.0497604 0.0107254 2024-05-15T12:59:22.136 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 73 16 107970 107954 92.4146 37.9375 0.0233268 0.0108161 2024-05-15T12:59:22.154 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-05-15T12:59:22.540 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:59:23.136 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 74 16 108736 108720 91.8127 47.875 0.0109486 0.0108875 2024-05-15T12:59:23.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:22 smithi177 ceph-mon[44303]: pgmap v938: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.5 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 46 MiB/s wr, 728 op/s; 19153/318933 objects degraded (6.005%); 24606/318933 objects misplaced (7.715%); 4.5 MiB/s, 71 objects/s recovering 2024-05-15T12:59:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:22 smithi027 ceph-mon[39795]: pgmap v938: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.5 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 46 MiB/s wr, 728 op/s; 19153/318933 objects degraded (6.005%); 24606/318933 objects misplaced (7.715%); 4.5 MiB/s, 71 objects/s recovering 2024-05-15T12:59:23.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:22 smithi120 ceph-mon[43961]: pgmap v938: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.5 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 46 MiB/s wr, 728 op/s; 19153/318933 objects degraded (6.005%); 24606/318933 objects misplaced (7.715%); 4.5 MiB/s, 71 objects/s recovering 2024-05-15T12:59:24.136 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 75 16 109474 109458 91.2035 46.125 0.0306518 0.0109602 2024-05-15T12:59:24.960 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:59:24.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:24.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:59:25.068 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:59:25.069 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:59:25.107 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:59:25.126 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:24 smithi027 ceph-mon[39795]: pgmap v939: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.5 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 40 MiB/s wr, 646 op/s; 19153/321546 objects degraded (5.957%); 24420/321546 objects misplaced (7.595%); 4.8 MiB/s, 76 objects/s recovering 2024-05-15T12:59:25.128 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:25.136 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 76 16 110193 110177 90.5947 44.9375 0.0166461 0.0110325 2024-05-15T12:59:25.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:24 smithi177 ceph-mon[44303]: pgmap v939: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.5 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 40 MiB/s wr, 646 op/s; 19153/321546 objects degraded (5.957%); 24420/321546 objects misplaced (7.595%); 4.8 MiB/s, 76 objects/s recovering 2024-05-15T12:59:25.252 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:59:25.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:24 smithi120 ceph-mon[43961]: pgmap v939: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.5 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 40 MiB/s wr, 646 op/s; 19153/321546 objects degraded (5.957%); 24420/321546 objects misplaced (7.595%); 4.8 MiB/s, 76 objects/s recovering 2024-05-15T12:59:25.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:25.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:59:25.382 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:59:25.429 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:25.519 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:59:25.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:25.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:59:25.622 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:59:25.668 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:25.687 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:59:25.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:25.689 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:59:25.790 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:59:25.850 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:25.883 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:59:25.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:25.891 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:59:25.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:25 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:59:25.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:25 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:59:25.992 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:59:26.027 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:26.048 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:59:26.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:26.051 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:59:26.136 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 77 16 110870 110854 89.9676 42.3125 0.0468226 0.0111097 2024-05-15T12:59:26.152 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:59:26.181 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:25 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:59:26.182 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:25 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:59:26.185 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:26.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:25 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:59:26.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:25 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:59:26.376 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:59:26.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:26.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:59:26.483 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T12:59:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:26.892 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:26 smithi027 ceph-mon[39795]: pgmap v940: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.7 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 48 MiB/s wr, 765 op/s; 19153/328848 objects degraded (5.824%); 24118/328848 objects misplaced (7.334%); 6.2 MiB/s, 98 objects/s recovering 2024-05-15T12:59:26.892 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:26 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 19153/328848 objects degraded (5.824%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:26.892 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:26 smithi027 ceph-mon[39795]: osdmap e348: 12 total, 12 up, 11 in 2024-05-15T12:59:27.136 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 78 16 111595 111579 89.3951 45.3125 0.0256168 0.0111814 2024-05-15T12:59:27.177 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T12:59:27.177 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T12:59:27.177 INFO:teuthology.orchestra.run.smithi120.stdout: "num_ops": 0 2024-05-15T12:59:27.177 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T12:59:27.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:26 smithi177 ceph-mon[44303]: pgmap v940: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.7 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 48 MiB/s wr, 765 op/s; 19153/328848 objects degraded (5.824%); 24118/328848 objects misplaced (7.334%); 6.2 MiB/s, 98 objects/s recovering 2024-05-15T12:59:27.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:26 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 19153/328848 objects degraded (5.824%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:27.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:26 smithi177 ceph-mon[44303]: osdmap e348: 12 total, 12 up, 11 in 2024-05-15T12:59:27.199 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:26 smithi120 ceph-mon[43961]: pgmap v940: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.7 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 48 MiB/s wr, 765 op/s; 19153/328848 objects degraded (5.824%); 24118/328848 objects misplaced (7.334%); 6.2 MiB/s, 98 objects/s recovering 2024-05-15T12:59:27.199 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:26 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 19153/328848 objects degraded (5.824%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:27.199 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:26 smithi120 ceph-mon[43961]: osdmap e348: 12 total, 12 up, 11 in 2024-05-15T12:59:27.910 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:27 smithi120 ceph-mon[43961]: pgmap v942: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.7 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 44 MiB/s wr, 707 op/s; 19153/330051 objects degraded (5.803%); 24118/330051 objects misplaced (7.307%); 6.0 MiB/s, 95 objects/s recovering 2024-05-15T12:59:27.911 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:27 smithi027 ceph-mon[39795]: pgmap v942: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.7 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 44 MiB/s wr, 707 op/s; 19153/330051 objects degraded (5.803%); 24118/330051 objects misplaced (7.307%); 6.0 MiB/s, 95 objects/s recovering 2024-05-15T12:59:27.953 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-05-15T12:59:28.136 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 79 16 112312 112296 88.8307 44.8125 0.010024 0.011252 2024-05-15T12:59:28.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:27 smithi177 ceph-mon[44303]: pgmap v942: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.7 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 44 MiB/s wr, 707 op/s; 19153/330051 objects degraded (5.803%); 24118/330051 objects misplaced (7.307%); 6.0 MiB/s, 95 objects/s recovering 2024-05-15T12:59:28.318 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:59:29.136 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T12:59:29.147761+0000 min lat: 0.00111471 max lat: 2.60882 avg lat: 0.0113205 2024-05-15T12:59:29.137 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T12:59:29.137 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 80 16 113047 113031 88.2945 45.9375 0.0201496 0.0113205 2024-05-15T12:59:29.193 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:29 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3485605343' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:59:29.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:29 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3485605343' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:59:29.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:29 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3485605343' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T12:59:29.684 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T12:59:29.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:29.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T12:59:29.789 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T12:59:29.824 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:30.136 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 81 16 113768 113752 87.7608 45.0625 0.0103551 0.0113899 2024-05-15T12:59:30.281 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T12:59:30.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:30.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T12:59:30.386 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:59:30.427 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:30.443 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T12:59:30.443 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T12:59:30.443 INFO:teuthology.orchestra.run.smithi120.stdout: "complaint_time": 30, 2024-05-15T12:59:30.443 INFO:teuthology.orchestra.run.smithi120.stdout: "num_blocked_ops": 0 2024-05-15T12:59:30.443 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T12:59:30.456 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:59:30.458 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:30 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:59:30.458 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:30 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3485605343' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:59:30.458 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:30 smithi177 ceph-mon[44303]: osdmap e349: 12 total, 12 up, 11 in 2024-05-15T12:59:30.458 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:30 smithi177 ceph-mon[44303]: pgmap v944: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.8 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 44 MiB/s wr, 709 op/s; 19153/335985 objects degraded (5.701%); 23784/335985 objects misplaced (7.079%); 6.4 MiB/s, 102 objects/s recovering 2024-05-15T12:59:30.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:30.459 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:59:30.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:30 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:59:30.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:30 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3485605343' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:59:30.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:30 smithi120 ceph-mon[43961]: osdmap e349: 12 total, 12 up, 11 in 2024-05-15T12:59:30.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:30 smithi120 ceph-mon[43961]: pgmap v944: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.8 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 44 MiB/s wr, 709 op/s; 19153/335985 objects degraded (5.701%); 23784/335985 objects misplaced (7.079%); 6.4 MiB/s, 102 objects/s recovering 2024-05-15T12:59:30.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:30 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:59:30.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:30 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3485605343' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T12:59:30.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:30 smithi027 ceph-mon[39795]: osdmap e349: 12 total, 12 up, 11 in 2024-05-15T12:59:30.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:30 smithi027 ceph-mon[39795]: pgmap v944: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.8 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 44 MiB/s wr, 709 op/s; 19153/335985 objects degraded (5.701%); 23784/335985 objects misplaced (7.079%); 6.4 MiB/s, 102 objects/s recovering 2024-05-15T12:59:30.560 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:59:30.638 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:30.700 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T12:59:31.137 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 82 16 114510 114494 87.256 46.375 0.0476301 0.0114559 2024-05-15T12:59:31.310 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-05-15T12:59:31.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:31 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/574499727' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T12:59:31.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:31 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/574499727' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T12:59:31.684 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:59:31.817 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:59:31.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:31.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:59:31.921 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:59:31.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:31 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/574499727' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T12:59:31.980 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:32.020 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:59:32.020 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T12:59:32.020 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 349, 2024-05-15T12:59:32.020 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "deaea04e-12b7-11ef-bc98-c7b262605968", 2024-05-15T12:59:32.020 INFO:teuthology.orchestra.run.smithi027.stdout: "created": "2024-05-15T12:38:21.244556+0000", 2024-05-15T12:59:32.021 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-05-15T12:59:29.199164+0000", 2024-05-15T12:59:32.021 INFO:teuthology.orchestra.run.smithi027.stdout: "last_up_change": "2024-05-15T12:45:50.876090+0000", 2024-05-15T12:59:32.021 INFO:teuthology.orchestra.run.smithi027.stdout: "last_in_change": "2024-05-15T12:59:10.294721+0000", 2024-05-15T12:59:32.021 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-15T12:59:32.021 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_num": 5806080, 2024-05-15T12:59:32.021 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_set": [ 2024-05-15T12:59:32.021 INFO:teuthology.orchestra.run.smithi027.stdout: "nodeep-scrub", 2024-05-15T12:59:32.021 INFO:teuthology.orchestra.run.smithi027.stdout: "noscrub", 2024-05-15T12:59:32.021 INFO:teuthology.orchestra.run.smithi027.stdout: "pglog_hardlimit", 2024-05-15T12:59:32.021 INFO:teuthology.orchestra.run.smithi027.stdout: "purged_snapdirs", 2024-05-15T12:59:32.021 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_deletes", 2024-05-15T12:59:32.021 INFO:teuthology.orchestra.run.smithi027.stdout: "sortbitwise" 2024-05-15T12:59:32.022 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T12:59:32.022 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_version": 26, 2024-05-15T12:59:32.022 INFO:teuthology.orchestra.run.smithi027.stdout: "full_ratio": 0.94999998807907104, 2024-05-15T12:59:32.022 INFO:teuthology.orchestra.run.smithi027.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-05-15T12:59:32.022 INFO:teuthology.orchestra.run.smithi027.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-15T12:59:32.022 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_snapshot": "", 2024-05-15T12:59:32.022 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_max": 4, 2024-05-15T12:59:32.022 INFO:teuthology.orchestra.run.smithi027.stdout: "max_osd": 12, 2024-05-15T12:59:32.022 INFO:teuthology.orchestra.run.smithi027.stdout: "require_min_compat_client": "luminous", 2024-05-15T12:59:32.022 INFO:teuthology.orchestra.run.smithi027.stdout: "min_compat_client": "jewel", 2024-05-15T12:59:32.022 INFO:teuthology.orchestra.run.smithi027.stdout: "require_osd_release": "squid", 2024-05-15T12:59:32.023 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_crimson": false, 2024-05-15T12:59:32.023 INFO:teuthology.orchestra.run.smithi027.stdout: "pools": [ 2024-05-15T12:59:32.023 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.023 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 1, 2024-05-15T12:59:32.023 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": ".mgr", 2024-05-15T12:59:32.023 INFO:teuthology.orchestra.run.smithi027.stdout: "create_time": "2024-05-15T12:41:57.948613+0000", 2024-05-15T12:59:32.023 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-15T12:59:32.023 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_names": "hashpspool", 2024-05-15T12:59:32.023 INFO:teuthology.orchestra.run.smithi027.stdout: "type": 1, 2024-05-15T12:59:32.023 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 3, 2024-05-15T12:59:32.023 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": 2, 2024-05-15T12:59:32.023 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_rule": 0, 2024-05-15T12:59:32.024 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_count": 0, 2024-05-15T12:59:32.024 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_target": 0, 2024-05-15T12:59:32.024 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T12:59:32.024 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T12:59:32.024 INFO:teuthology.orchestra.run.smithi027.stdout: "object_hash": 2, 2024-05-15T12:59:32.024 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscale_mode": "off", 2024-05-15T12:59:32.024 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": 1, 2024-05-15T12:59:32.024 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num": 1, 2024-05-15T12:59:32.024 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num_target": 1, 2024-05-15T12:59:32.024 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_target": 1, 2024-05-15T12:59:32.024 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_pending": 1, 2024-05-15T12:59:32.024 INFO:teuthology.orchestra.run.smithi027.stdout: "last_pg_merge_meta": { 2024-05-15T12:59:32.024 INFO:teuthology.orchestra.run.smithi027.stdout: "source_pgid": "0.0", 2024-05-15T12:59:32.025 INFO:teuthology.orchestra.run.smithi027.stdout: "ready_epoch": 0, 2024-05-15T12:59:32.025 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_started": 0, 2024-05-15T12:59:32.025 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_clean": 0, 2024-05-15T12:59:32.025 INFO:teuthology.orchestra.run.smithi027.stdout: "source_version": "0'0", 2024-05-15T12:59:32.025 INFO:teuthology.orchestra.run.smithi027.stdout: "target_version": "0'0" 2024-05-15T12:59:32.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.025 INFO:teuthology.orchestra.run.smithi027.stdout: "last_change": "21", 2024-05-15T12:59:32.025 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend": "0", 2024-05-15T12:59:32.025 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T12:59:32.025 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T12:59:32.025 INFO:teuthology.orchestra.run.smithi027.stdout: "auid": 0, 2024-05-15T12:59:32.026 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_mode": "selfmanaged", 2024-05-15T12:59:32.026 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_seq": 0, 2024-05-15T12:59:32.026 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_epoch": 0, 2024-05-15T12:59:32.026 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_snaps": [], 2024-05-15T12:59:32.026 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps": "[]", 2024-05-15T12:59:32.026 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_bytes": 0, 2024-05-15T12:59:32.026 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_objects": 0, 2024-05-15T12:59:32.026 INFO:teuthology.orchestra.run.smithi027.stdout: "tiers": [], 2024-05-15T12:59:32.026 INFO:teuthology.orchestra.run.smithi027.stdout: "tier_of": -1, 2024-05-15T12:59:32.026 INFO:teuthology.orchestra.run.smithi027.stdout: "read_tier": -1, 2024-05-15T12:59:32.026 INFO:teuthology.orchestra.run.smithi027.stdout: "write_tier": -1, 2024-05-15T12:59:32.026 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_mode": "none", 2024-05-15T12:59:32.026 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_bytes": 0, 2024-05-15T12:59:32.027 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_objects": 0, 2024-05-15T12:59:32.027 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T12:59:32.027 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T12:59:32.027 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T12:59:32.027 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_flush_age": 0, 2024-05-15T12:59:32.027 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_evict_age": 0, 2024-05-15T12:59:32.027 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profile": "", 2024-05-15T12:59:32.027 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_params": { 2024-05-15T12:59:32.027 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "none" 2024-05-15T12:59:32.027 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.027 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_period": 0, 2024-05-15T12:59:32.027 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_count": 0, 2024-05-15T12:59:32.027 INFO:teuthology.orchestra.run.smithi027.stdout: "use_gmt_hitset": true, 2024-05-15T12:59:32.027 INFO:teuthology.orchestra.run.smithi027.stdout: "min_read_recency_for_promote": 0, 2024-05-15T12:59:32.028 INFO:teuthology.orchestra.run.smithi027.stdout: "min_write_recency_for_promote": 0, 2024-05-15T12:59:32.028 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T12:59:32.028 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_search_last_n": 0, 2024-05-15T12:59:32.028 INFO:teuthology.orchestra.run.smithi027.stdout: "grade_table": [], 2024-05-15T12:59:32.028 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_width": 0, 2024-05-15T12:59:32.028 INFO:teuthology.orchestra.run.smithi027.stdout: "expected_num_objects": 0, 2024-05-15T12:59:32.028 INFO:teuthology.orchestra.run.smithi027.stdout: "fast_read": false, 2024-05-15T12:59:32.028 INFO:teuthology.orchestra.run.smithi027.stdout: "options": { 2024-05-15T12:59:32.028 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_max": 32, 2024-05-15T12:59:32.028 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_min": 1 2024-05-15T12:59:32.028 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.028 INFO:teuthology.orchestra.run.smithi027.stdout: "application_metadata": { 2024-05-15T12:59:32.028 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": {} 2024-05-15T12:59:32.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.029 INFO:teuthology.orchestra.run.smithi027.stdout: "read_balance": { 2024-05-15T12:59:32.029 INFO:teuthology.orchestra.run.smithi027.stdout: "score_type": "Fair distribution", 2024-05-15T12:59:32.029 INFO:teuthology.orchestra.run.smithi027.stdout: "score_acting": 12, 2024-05-15T12:59:32.029 INFO:teuthology.orchestra.run.smithi027.stdout: "score_stable": 12, 2024-05-15T12:59:32.029 INFO:teuthology.orchestra.run.smithi027.stdout: "optimal_score": 0.25, 2024-05-15T12:59:32.029 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_acting": 3, 2024-05-15T12:59:32.029 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_stable": 3, 2024-05-15T12:59:32.029 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity_weighted": 0.94652295112609863, 2024-05-15T12:59:32.029 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity": 0.94999998807907104, 2024-05-15T12:59:32.029 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-05-15T12:59:32.029 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.029 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.029 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 4, 2024-05-15T12:59:32.030 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": "unique_pool_2", 2024-05-15T12:59:32.030 INFO:teuthology.orchestra.run.smithi027.stdout: "create_time": "2024-05-15T12:57:53.956942+0000", 2024-05-15T12:59:32.030 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-15T12:59:32.030 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_names": "hashpspool", 2024-05-15T12:59:32.030 INFO:teuthology.orchestra.run.smithi027.stdout: "type": 1, 2024-05-15T12:59:32.030 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 3, 2024-05-15T12:59:32.030 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": 2, 2024-05-15T12:59:32.030 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_rule": 0, 2024-05-15T12:59:32.030 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_count": 0, 2024-05-15T12:59:32.030 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_target": 0, 2024-05-15T12:59:32.030 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T12:59:32.030 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T12:59:32.031 INFO:teuthology.orchestra.run.smithi027.stdout: "object_hash": 2, 2024-05-15T12:59:32.031 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscale_mode": "off", 2024-05-15T12:59:32.031 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": 16, 2024-05-15T12:59:32.031 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num": 16, 2024-05-15T12:59:32.031 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num_target": 16, 2024-05-15T12:59:32.031 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_target": 16, 2024-05-15T12:59:32.031 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_pending": 16, 2024-05-15T12:59:32.031 INFO:teuthology.orchestra.run.smithi027.stdout: "last_pg_merge_meta": { 2024-05-15T12:59:32.031 INFO:teuthology.orchestra.run.smithi027.stdout: "source_pgid": "0.0", 2024-05-15T12:59:32.031 INFO:teuthology.orchestra.run.smithi027.stdout: "ready_epoch": 0, 2024-05-15T12:59:32.031 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_started": 0, 2024-05-15T12:59:32.031 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_clean": 0, 2024-05-15T12:59:32.031 INFO:teuthology.orchestra.run.smithi027.stdout: "source_version": "0'0", 2024-05-15T12:59:32.031 INFO:teuthology.orchestra.run.smithi027.stdout: "target_version": "0'0" 2024-05-15T12:59:32.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.032 INFO:teuthology.orchestra.run.smithi027.stdout: "last_change": "321", 2024-05-15T12:59:32.032 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend": "0", 2024-05-15T12:59:32.032 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T12:59:32.032 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T12:59:32.032 INFO:teuthology.orchestra.run.smithi027.stdout: "auid": 0, 2024-05-15T12:59:32.032 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_mode": "selfmanaged", 2024-05-15T12:59:32.032 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_seq": 0, 2024-05-15T12:59:32.032 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_epoch": 0, 2024-05-15T12:59:32.032 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_snaps": [], 2024-05-15T12:59:32.032 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps": "[]", 2024-05-15T12:59:32.032 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_bytes": 0, 2024-05-15T12:59:32.032 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_objects": 0, 2024-05-15T12:59:32.032 INFO:teuthology.orchestra.run.smithi027.stdout: "tiers": [], 2024-05-15T12:59:32.033 INFO:teuthology.orchestra.run.smithi027.stdout: "tier_of": -1, 2024-05-15T12:59:32.033 INFO:teuthology.orchestra.run.smithi027.stdout: "read_tier": -1, 2024-05-15T12:59:32.033 INFO:teuthology.orchestra.run.smithi027.stdout: "write_tier": -1, 2024-05-15T12:59:32.033 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_mode": "none", 2024-05-15T12:59:32.033 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_bytes": 0, 2024-05-15T12:59:32.033 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_objects": 0, 2024-05-15T12:59:32.033 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T12:59:32.033 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T12:59:32.033 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T12:59:32.033 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_flush_age": 0, 2024-05-15T12:59:32.033 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_evict_age": 0, 2024-05-15T12:59:32.034 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profile": "", 2024-05-15T12:59:32.034 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_params": { 2024-05-15T12:59:32.034 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "none" 2024-05-15T12:59:32.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.034 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_period": 0, 2024-05-15T12:59:32.034 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_count": 0, 2024-05-15T12:59:32.034 INFO:teuthology.orchestra.run.smithi027.stdout: "use_gmt_hitset": true, 2024-05-15T12:59:32.034 INFO:teuthology.orchestra.run.smithi027.stdout: "min_read_recency_for_promote": 0, 2024-05-15T12:59:32.034 INFO:teuthology.orchestra.run.smithi027.stdout: "min_write_recency_for_promote": 0, 2024-05-15T12:59:32.034 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T12:59:32.034 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_search_last_n": 0, 2024-05-15T12:59:32.035 INFO:teuthology.orchestra.run.smithi027.stdout: "grade_table": [], 2024-05-15T12:59:32.035 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_width": 0, 2024-05-15T12:59:32.035 INFO:teuthology.orchestra.run.smithi027.stdout: "expected_num_objects": 0, 2024-05-15T12:59:32.035 INFO:teuthology.orchestra.run.smithi027.stdout: "fast_read": false, 2024-05-15T12:59:32.035 INFO:teuthology.orchestra.run.smithi027.stdout: "options": {}, 2024-05-15T12:59:32.035 INFO:teuthology.orchestra.run.smithi027.stdout: "application_metadata": { 2024-05-15T12:59:32.035 INFO:teuthology.orchestra.run.smithi027.stdout: "rados": {} 2024-05-15T12:59:32.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.035 INFO:teuthology.orchestra.run.smithi027.stdout: "read_balance": { 2024-05-15T12:59:32.035 INFO:teuthology.orchestra.run.smithi027.stdout: "score_type": "Fair distribution", 2024-05-15T12:59:32.035 INFO:teuthology.orchestra.run.smithi027.stdout: "score_acting": 2.9900000095367432, 2024-05-15T12:59:32.035 INFO:teuthology.orchestra.run.smithi027.stdout: "score_stable": 2.9900000095367432, 2024-05-15T12:59:32.035 INFO:teuthology.orchestra.run.smithi027.stdout: "optimal_score": 0.92000001668930054, 2024-05-15T12:59:32.035 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_acting": 2.75, 2024-05-15T12:59:32.036 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_stable": 2.75, 2024-05-15T12:59:32.036 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity_weighted": 0.94652295112609863, 2024-05-15T12:59:32.036 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity": 0.94999998807907104, 2024-05-15T12:59:32.036 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-05-15T12:59:32.036 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.036 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.036 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T12:59:32.036 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T12:59:32.036 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.036 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 0, 2024-05-15T12:59:32.036 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "64fb3ad2-08a7-4106-a07a-6c776c877faf", 2024-05-15T12:59:32.037 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:59:32.037 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:59:32.037 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:59:32.037 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:59:32.037 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:59:32.037 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:59:32.037 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 8, 2024-05-15T12:59:32.037 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 343, 2024-05-15T12:59:32.037 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:59:32.037 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:59:32.037 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:59:32.037 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.037 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.038 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.038 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6802", 2024-05-15T12:59:32.038 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:59:32.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.038 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.038 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6803", 2024-05-15T12:59:32.038 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:59:32.038 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.038 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.038 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:59:32.038 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.039 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.039 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6804", 2024-05-15T12:59:32.039 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:59:32.039 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.039 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.039 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.039 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6805", 2024-05-15T12:59:32.039 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:59:32.039 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.039 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.039 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.040 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:59:32.040 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.040 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.040 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6808", 2024-05-15T12:59:32.040 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:59:32.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.040 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.041 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6809", 2024-05-15T12:59:32.041 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:59:32.041 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.041 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.041 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:59:32.041 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.041 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.041 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6806", 2024-05-15T12:59:32.041 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:59:32.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.042 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.042 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.042 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6807", 2024-05-15T12:59:32.042 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T12:59:32.042 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.042 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.042 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6803/4239116935", 2024-05-15T12:59:32.042 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6805/4239116935", 2024-05-15T12:59:32.042 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6809/4239116935", 2024-05-15T12:59:32.042 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6807/4239116935", 2024-05-15T12:59:32.042 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:59:32.042 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:59:32.043 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:59:32.043 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.043 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.043 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 1, 2024-05-15T12:59:32.043 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "9c16dd90-9d51-48b8-a296-1c8879ba1bc8", 2024-05-15T12:59:32.043 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:59:32.043 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:59:32.043 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:59:32.043 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:59:32.043 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:59:32.043 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:59:32.044 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 13, 2024-05-15T12:59:32.044 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 343, 2024-05-15T12:59:32.044 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:59:32.044 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:59:32.044 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:59:32.044 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.044 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.044 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.044 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6810", 2024-05-15T12:59:32.044 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:59:32.044 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.044 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.044 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.045 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6811", 2024-05-15T12:59:32.045 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:59:32.045 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.045 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.045 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:59:32.045 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.045 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.045 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6812", 2024-05-15T12:59:32.045 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:59:32.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.045 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.045 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6813", 2024-05-15T12:59:32.046 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:59:32.046 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.046 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.046 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:59:32.046 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.046 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.046 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.046 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6816", 2024-05-15T12:59:32.046 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:59:32.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.046 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.046 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.047 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6817", 2024-05-15T12:59:32.047 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:59:32.047 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.047 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.047 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:59:32.047 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.047 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.047 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.047 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6814", 2024-05-15T12:59:32.047 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:59:32.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.047 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.047 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.048 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6815", 2024-05-15T12:59:32.048 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T12:59:32.048 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.048 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.048 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6811/1466270637", 2024-05-15T12:59:32.048 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6813/1466270637", 2024-05-15T12:59:32.048 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6817/1466270637", 2024-05-15T12:59:32.048 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6815/1466270637", 2024-05-15T12:59:32.048 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:59:32.048 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:59:32.048 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:59:32.048 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.049 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.049 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 2, 2024-05-15T12:59:32.049 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "cbe3ba13-3ea3-4891-8d2d-6b85581a732c", 2024-05-15T12:59:32.049 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:59:32.049 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:59:32.049 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:59:32.049 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:59:32.049 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:59:32.049 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:59:32.049 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 18, 2024-05-15T12:59:32.049 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 110, 2024-05-15T12:59:32.050 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:59:32.050 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:59:32.050 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:59:32.050 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.050 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.050 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6818", 2024-05-15T12:59:32.050 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:59:32.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.050 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.050 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6819", 2024-05-15T12:59:32.050 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:59:32.051 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.051 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.051 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:59:32.051 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.051 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.051 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6820", 2024-05-15T12:59:32.051 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:59:32.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.051 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.051 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6821", 2024-05-15T12:59:32.052 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:59:32.052 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.052 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.052 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:59:32.052 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.052 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.052 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6824", 2024-05-15T12:59:32.052 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:59:32.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.052 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.053 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6825", 2024-05-15T12:59:32.053 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:59:32.053 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.053 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.053 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:59:32.053 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.053 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.053 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6822", 2024-05-15T12:59:32.053 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:59:32.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.054 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.054 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.054 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6823", 2024-05-15T12:59:32.054 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T12:59:32.054 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.054 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.055 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6819/3285413897", 2024-05-15T12:59:32.055 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6821/3285413897", 2024-05-15T12:59:32.055 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6825/3285413897", 2024-05-15T12:59:32.055 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6823/3285413897", 2024-05-15T12:59:32.055 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:59:32.055 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:59:32.055 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:59:32.055 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.056 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 3, 2024-05-15T12:59:32.056 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "2e83735b-b944-4f87-8080-b994d4b96897", 2024-05-15T12:59:32.056 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:59:32.056 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:59:32.056 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:59:32.056 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:59:32.056 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:59:32.056 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:59:32.057 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 26, 2024-05-15T12:59:32.057 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 317, 2024-05-15T12:59:32.057 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:59:32.057 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:59:32.057 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:59:32.057 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.057 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.057 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6826", 2024-05-15T12:59:32.057 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:59:32.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.057 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.058 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6827", 2024-05-15T12:59:32.058 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:59:32.058 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.058 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.058 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:59:32.058 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.058 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.058 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.058 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6828", 2024-05-15T12:59:32.058 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:59:32.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.058 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.059 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.059 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6829", 2024-05-15T12:59:32.059 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:59:32.059 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.059 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.059 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:59:32.059 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.059 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.059 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6832", 2024-05-15T12:59:32.059 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:59:32.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.060 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.060 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6833", 2024-05-15T12:59:32.060 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:59:32.060 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.060 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.060 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:59:32.060 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.060 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.060 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6830", 2024-05-15T12:59:32.060 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:59:32.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.061 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.061 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6831", 2024-05-15T12:59:32.061 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T12:59:32.061 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.061 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.061 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6827/3008975244", 2024-05-15T12:59:32.061 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6829/3008975244", 2024-05-15T12:59:32.061 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6833/3008975244", 2024-05-15T12:59:32.061 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6831/3008975244", 2024-05-15T12:59:32.061 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:59:32.061 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:59:32.061 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:59:32.061 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.062 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.062 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.062 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 4, 2024-05-15T12:59:32.062 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "7c39c1dd-3867-46f7-8372-7688fa13b16a", 2024-05-15T12:59:32.062 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:59:32.062 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:59:32.062 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:59:32.062 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:59:32.062 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:59:32.062 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:59:32.062 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 31, 2024-05-15T12:59:32.062 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 343, 2024-05-15T12:59:32.063 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:59:32.063 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:59:32.063 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:59:32.063 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.063 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.063 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6800", 2024-05-15T12:59:32.063 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:59:32.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.063 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.063 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6801", 2024-05-15T12:59:32.063 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:59:32.063 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.064 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.064 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:59:32.064 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.064 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.064 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.064 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6802", 2024-05-15T12:59:32.064 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:59:32.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.064 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.064 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.064 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6803", 2024-05-15T12:59:32.064 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:59:32.064 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.065 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.065 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.065 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:59:32.065 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.065 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.065 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.065 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6806", 2024-05-15T12:59:32.065 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:59:32.065 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.065 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.065 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.066 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6807", 2024-05-15T12:59:32.066 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:59:32.066 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.066 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.066 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:59:32.066 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.066 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.066 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.066 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6804", 2024-05-15T12:59:32.066 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:59:32.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.067 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.067 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.067 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6805", 2024-05-15T12:59:32.067 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T12:59:32.067 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.067 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.067 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6801/3877842957", 2024-05-15T12:59:32.067 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6803/3877842957", 2024-05-15T12:59:32.067 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6807/3877842957", 2024-05-15T12:59:32.067 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6805/3877842957", 2024-05-15T12:59:32.067 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:59:32.067 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:59:32.068 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:59:32.068 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.068 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.068 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.068 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 5, 2024-05-15T12:59:32.068 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "a6dc220f-aec0-4454-9695-b44c0b9770e0", 2024-05-15T12:59:32.068 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:59:32.068 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:59:32.068 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:59:32.068 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:59:32.068 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:59:32.068 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:59:32.069 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 40, 2024-05-15T12:59:32.069 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 343, 2024-05-15T12:59:32.069 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:59:32.069 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:59:32.069 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:59:32.069 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.069 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.069 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6808", 2024-05-15T12:59:32.069 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:59:32.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.069 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.070 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6809", 2024-05-15T12:59:32.070 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:59:32.070 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.070 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.070 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:59:32.070 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.070 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.070 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.070 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6810", 2024-05-15T12:59:32.070 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:59:32.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.071 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.071 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.071 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6811", 2024-05-15T12:59:32.071 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:59:32.071 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.071 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.071 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.071 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:59:32.071 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.071 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.071 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.071 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6814", 2024-05-15T12:59:32.071 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:59:32.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.072 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.072 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.072 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6815", 2024-05-15T12:59:32.072 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:59:32.072 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.072 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.072 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:59:32.072 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.072 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.072 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.072 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6812", 2024-05-15T12:59:32.072 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:59:32.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.073 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.073 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.073 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6813", 2024-05-15T12:59:32.073 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T12:59:32.073 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.073 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.073 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.073 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6809/1600179183", 2024-05-15T12:59:32.073 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6811/1600179183", 2024-05-15T12:59:32.073 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6815/1600179183", 2024-05-15T12:59:32.073 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6813/1600179183", 2024-05-15T12:59:32.074 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:59:32.074 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:59:32.074 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:59:32.074 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.074 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.074 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.074 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 6, 2024-05-15T12:59:32.074 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b", 2024-05-15T12:59:32.074 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:59:32.074 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:59:32.074 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:59:32.075 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:59:32.075 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:59:32.075 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:59:32.075 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 48, 2024-05-15T12:59:32.075 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 344, 2024-05-15T12:59:32.075 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:59:32.075 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:59:32.075 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:59:32.075 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.075 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.075 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.075 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6816", 2024-05-15T12:59:32.075 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:59:32.075 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.076 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.076 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.076 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6817", 2024-05-15T12:59:32.076 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:59:32.076 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.076 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.076 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:59:32.076 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.076 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.076 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.076 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6818", 2024-05-15T12:59:32.076 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:59:32.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.076 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.077 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.077 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6819", 2024-05-15T12:59:32.077 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:59:32.077 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.077 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.077 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.077 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:59:32.077 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.077 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.077 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.077 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6822", 2024-05-15T12:59:32.077 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:59:32.077 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.077 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.078 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.078 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6823", 2024-05-15T12:59:32.078 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:59:32.078 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.078 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.078 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.078 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:59:32.078 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.078 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.078 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.079 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6820", 2024-05-15T12:59:32.079 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:59:32.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.079 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.079 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.079 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6821", 2024-05-15T12:59:32.079 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T12:59:32.079 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.079 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.080 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6817/3323706735", 2024-05-15T12:59:32.080 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6819/3323706735", 2024-05-15T12:59:32.080 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6823/3323706735", 2024-05-15T12:59:32.080 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6821/3323706735", 2024-05-15T12:59:32.080 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:59:32.080 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:59:32.080 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:59:32.080 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.080 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.080 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.080 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 7, 2024-05-15T12:59:32.080 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "6ef085d8-f59b-4033-8280-2c8f78693e30", 2024-05-15T12:59:32.080 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:59:32.081 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 0, 2024-05-15T12:59:32.081 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 0, 2024-05-15T12:59:32.081 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:59:32.081 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:59:32.081 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:59:32.081 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 56, 2024-05-15T12:59:32.081 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 219, 2024-05-15T12:59:32.081 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:59:32.081 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:59:32.081 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:59:32.081 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.082 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.082 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.082 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6824", 2024-05-15T12:59:32.082 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:59:32.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.082 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.082 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.082 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6825", 2024-05-15T12:59:32.082 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:59:32.082 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.082 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.082 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:59:32.083 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.083 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.083 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.083 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6826", 2024-05-15T12:59:32.083 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:59:32.083 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.083 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.083 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.083 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6827", 2024-05-15T12:59:32.084 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:59:32.084 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.084 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.084 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:59:32.084 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.084 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.084 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.084 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6830", 2024-05-15T12:59:32.085 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:59:32.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.085 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.085 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.085 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6831", 2024-05-15T12:59:32.085 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:59:32.085 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.085 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.085 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:59:32.085 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.085 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.085 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.086 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6828", 2024-05-15T12:59:32.086 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:59:32.086 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.086 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.086 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.086 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6829", 2024-05-15T12:59:32.086 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T12:59:32.086 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.086 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.086 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.086 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6825/2960616153", 2024-05-15T12:59:32.086 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6827/2960616153", 2024-05-15T12:59:32.087 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6831/2960616153", 2024-05-15T12:59:32.087 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6829/2960616153", 2024-05-15T12:59:32.087 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:59:32.087 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:59:32.087 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:59:32.087 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.087 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 8, 2024-05-15T12:59:32.087 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "26514b11-d02b-46e9-a4e8-3da193fcdbcc", 2024-05-15T12:59:32.088 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:59:32.088 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:59:32.088 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:59:32.088 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:59:32.088 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:59:32.088 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:59:32.088 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 65, 2024-05-15T12:59:32.088 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 292, 2024-05-15T12:59:32.089 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:59:32.089 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:59:32.089 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:59:32.089 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.089 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.089 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.089 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6800", 2024-05-15T12:59:32.089 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:59:32.089 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.089 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.089 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.090 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6801", 2024-05-15T12:59:32.090 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:59:32.090 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.090 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.090 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:59:32.090 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.090 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.091 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.091 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6802", 2024-05-15T12:59:32.091 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:59:32.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.091 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.091 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6803", 2024-05-15T12:59:32.091 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:59:32.091 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.091 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.092 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:59:32.092 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.092 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.092 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6806", 2024-05-15T12:59:32.092 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:59:32.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.092 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.092 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6807", 2024-05-15T12:59:32.092 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:59:32.093 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.093 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.093 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:59:32.093 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.093 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.093 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6804", 2024-05-15T12:59:32.093 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:59:32.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.093 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.093 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6805", 2024-05-15T12:59:32.093 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T12:59:32.094 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.094 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.094 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.094 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6801/1132459315", 2024-05-15T12:59:32.094 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6803/1132459315", 2024-05-15T12:59:32.094 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6807/1132459315", 2024-05-15T12:59:32.094 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6805/1132459315", 2024-05-15T12:59:32.094 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:59:32.094 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:59:32.094 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:59:32.094 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.094 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.094 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.095 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 9, 2024-05-15T12:59:32.095 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "6f9497bc-e962-409d-8a2c-0c634cc1124f", 2024-05-15T12:59:32.095 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:59:32.095 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:59:32.095 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:59:32.095 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 0.9062042236328125, 2024-05-15T12:59:32.095 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:59:32.095 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:59:32.095 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 76, 2024-05-15T12:59:32.095 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 317, 2024-05-15T12:59:32.095 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:59:32.095 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:59:32.095 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:59:32.095 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.096 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.096 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6808", 2024-05-15T12:59:32.096 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:59:32.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.096 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.096 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6809", 2024-05-15T12:59:32.096 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:59:32.096 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.096 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.096 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:59:32.097 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.097 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.097 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6810", 2024-05-15T12:59:32.097 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:59:32.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.097 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.097 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6811", 2024-05-15T12:59:32.097 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:59:32.097 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.097 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.097 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:59:32.098 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.098 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.098 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.098 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6814", 2024-05-15T12:59:32.098 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:59:32.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.098 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.098 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.098 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6815", 2024-05-15T12:59:32.098 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:59:32.098 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.099 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.099 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:59:32.099 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.099 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.099 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6812", 2024-05-15T12:59:32.099 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:59:32.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.099 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.099 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6813", 2024-05-15T12:59:32.099 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T12:59:32.100 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.100 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.100 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6809/2710457646", 2024-05-15T12:59:32.100 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6811/2710457646", 2024-05-15T12:59:32.100 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6815/2710457646", 2024-05-15T12:59:32.100 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6813/2710457646", 2024-05-15T12:59:32.100 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:59:32.100 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:59:32.101 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:59:32.101 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.101 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 10, 2024-05-15T12:59:32.101 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "f852876f-c7d1-4039-90c4-5ab09dbd3e32", 2024-05-15T12:59:32.101 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:59:32.101 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:59:32.101 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:59:32.101 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 0.4520721435546875, 2024-05-15T12:59:32.101 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:59:32.101 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:59:32.102 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 81, 2024-05-15T12:59:32.102 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 0, 2024-05-15T12:59:32.102 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:59:32.102 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:59:32.102 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:59:32.102 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.102 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.102 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.102 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6816", 2024-05-15T12:59:32.102 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:59:32.102 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.102 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.102 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.103 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6817", 2024-05-15T12:59:32.103 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:59:32.103 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.103 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.103 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.103 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:59:32.103 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.103 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.103 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.103 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6818", 2024-05-15T12:59:32.103 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:59:32.103 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.104 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.104 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.104 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6819", 2024-05-15T12:59:32.104 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:59:32.104 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.104 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.104 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:59:32.104 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.104 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.104 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.104 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6822", 2024-05-15T12:59:32.104 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:59:32.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.104 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.105 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.105 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6823", 2024-05-15T12:59:32.105 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:59:32.105 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.105 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.105 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.105 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:59:32.105 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.105 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.105 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.105 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6820", 2024-05-15T12:59:32.105 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:59:32.105 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.105 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.105 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.106 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6821", 2024-05-15T12:59:32.106 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T12:59:32.106 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.106 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.106 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6817/3236900263", 2024-05-15T12:59:32.106 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6819/3236900263", 2024-05-15T12:59:32.106 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6823/3236900263", 2024-05-15T12:59:32.106 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6821/3236900263", 2024-05-15T12:59:32.106 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:59:32.106 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:59:32.106 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:59:32.106 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.107 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.107 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.107 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 11, 2024-05-15T12:59:32.107 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "cccc53c5-8476-4d55-b160-4bdc5cf92902", 2024-05-15T12:59:32.107 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T12:59:32.107 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T12:59:32.107 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T12:59:32.107 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T12:59:32.107 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T12:59:32.107 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T12:59:32.107 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 90, 2024-05-15T12:59:32.107 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 345, 2024-05-15T12:59:32.107 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T12:59:32.107 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T12:59:32.108 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T12:59:32.108 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.108 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.108 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.108 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6824", 2024-05-15T12:59:32.108 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:59:32.108 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.108 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.108 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.108 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6825", 2024-05-15T12:59:32.108 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:59:32.108 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.109 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.109 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T12:59:32.109 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.109 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.109 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.109 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6826", 2024-05-15T12:59:32.109 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:59:32.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.109 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.109 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.109 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6827", 2024-05-15T12:59:32.109 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:59:32.110 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.110 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.110 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T12:59:32.110 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.110 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.110 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.110 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6830", 2024-05-15T12:59:32.110 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:59:32.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.110 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.110 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.111 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6831", 2024-05-15T12:59:32.111 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:59:32.111 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.111 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.111 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.111 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T12:59:32.111 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T12:59:32.111 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.111 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T12:59:32.112 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6828", 2024-05-15T12:59:32.112 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:59:32.112 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.112 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.112 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T12:59:32.112 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6829", 2024-05-15T12:59:32.112 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T12:59:32.112 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.112 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.112 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.112 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6825/681011628", 2024-05-15T12:59:32.112 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6827/681011628", 2024-05-15T12:59:32.112 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6831/681011628", 2024-05-15T12:59:32.113 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6829/681011628", 2024-05-15T12:59:32.113 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T12:59:32.113 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T12:59:32.113 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T12:59:32.113 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.113 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.113 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T12:59:32.113 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_xinfo": [ 2024-05-15T12:59:32.113 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.113 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 0, 2024-05-15T12:59:32.114 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:59:32.114 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:59:32.114 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:59:32.114 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:59:32.114 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:59:32.114 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:58:36.082224+0000", 2024-05-15T12:59:32.114 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:59:32.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.114 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.114 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 1, 2024-05-15T12:59:32.115 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:59:32.115 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:59:32.115 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:59:32.115 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:59:32.115 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:59:32.115 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:59:05.617321+0000", 2024-05-15T12:59:32.115 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:59:32.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.115 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 2, 2024-05-15T12:59:32.115 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:59:32.115 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:59:32.115 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:59:32.116 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:59:32.116 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:59:32.116 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:58:38.235991+0000", 2024-05-15T12:59:32.116 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:59:32.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.116 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.116 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 3, 2024-05-15T12:59:32.116 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:59:32.116 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:59:32.116 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:59:32.116 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:59:32.116 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:59:32.117 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:58:27.105368+0000", 2024-05-15T12:59:32.117 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:59:32.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.117 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 4, 2024-05-15T12:59:32.117 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:59:32.117 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:59:32.117 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:59:32.117 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:59:32.117 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:59:32.117 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:59:03.869628+0000", 2024-05-15T12:59:32.118 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:59:32.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.118 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.118 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 5, 2024-05-15T12:59:32.118 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:59:32.118 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:59:32.118 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:59:32.118 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:59:32.118 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:59:32.118 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:59:06.639157+0000", 2024-05-15T12:59:32.118 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:59:32.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.118 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.119 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 6, 2024-05-15T12:59:32.119 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:59:32.119 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:59:32.119 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:59:32.119 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:59:32.119 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:59:32.119 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:58:49.959878+0000", 2024-05-15T12:59:32.119 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:59:32.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.119 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.119 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 7, 2024-05-15T12:59:32.119 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:59:32.119 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:59:32.119 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:59:32.119 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:59:32.120 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 65536, 2024-05-15T12:59:32.120 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:58:23.906382+0000", 2024-05-15T12:59:32.120 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:59:32.120 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.120 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.120 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 8, 2024-05-15T12:59:32.120 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:59:32.120 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:59:32.120 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:59:32.120 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:59:32.120 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:59:32.120 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:59:25.546686+0000", 2024-05-15T12:59:32.120 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:59:32.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.121 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.121 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 9, 2024-05-15T12:59:32.121 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:59:32.121 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:59:32.121 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:59:32.121 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:59:32.121 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:59:32.121 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:59:14.791769+0000", 2024-05-15T12:59:32.121 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:59:32.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.121 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.121 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 10, 2024-05-15T12:59:32.121 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:59:32.122 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:59:32.122 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:59:32.122 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:59:32.122 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:59:32.122 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:58:14.898165+0000", 2024-05-15T12:59:32.122 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:59:32.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.122 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.122 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 11, 2024-05-15T12:59:32.122 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T12:59:32.122 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T12:59:32.123 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T12:59:32.123 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T12:59:32.123 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T12:59:32.123 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:59:00.958960+0000", 2024-05-15T12:59:32.123 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T12:59:32.123 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.123 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T12:59:32.123 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap": [], 2024-05-15T12:59:32.123 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap_items": [], 2024-05-15T12:59:32.123 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap_primaries": [], 2024-05-15T12:59:32.123 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_temp": [ 2024-05-15T12:59:32.123 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.124 INFO:teuthology.orchestra.run.smithi027.stdout: "pgid": "4.1", 2024-05-15T12:59:32.124 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T12:59:32.124 INFO:teuthology.orchestra.run.smithi027.stdout: 4, 2024-05-15T12:59:32.124 INFO:teuthology.orchestra.run.smithi027.stdout: 5, 2024-05-15T12:59:32.124 INFO:teuthology.orchestra.run.smithi027.stdout: 6 2024-05-15T12:59:32.124 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.124 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.124 INFO:teuthology.orchestra.run.smithi027.stdout: "pgid": "4.2", 2024-05-15T12:59:32.124 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T12:59:32.125 INFO:teuthology.orchestra.run.smithi027.stdout: 5, 2024-05-15T12:59:32.125 INFO:teuthology.orchestra.run.smithi027.stdout: 9 2024-05-15T12:59:32.125 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.125 INFO:teuthology.orchestra.run.smithi027.stdout: "pgid": "4.3", 2024-05-15T12:59:32.125 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T12:59:32.125 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-05-15T12:59:32.125 INFO:teuthology.orchestra.run.smithi027.stdout: 1 2024-05-15T12:59:32.125 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.126 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.126 INFO:teuthology.orchestra.run.smithi027.stdout: "pgid": "4.5", 2024-05-15T12:59:32.126 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T12:59:32.126 INFO:teuthology.orchestra.run.smithi027.stdout: 6, 2024-05-15T12:59:32.126 INFO:teuthology.orchestra.run.smithi027.stdout: 10 2024-05-15T12:59:32.126 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.126 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.126 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.126 INFO:teuthology.orchestra.run.smithi027.stdout: "pgid": "4.8", 2024-05-15T12:59:32.126 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T12:59:32.126 INFO:teuthology.orchestra.run.smithi027.stdout: 6, 2024-05-15T12:59:32.126 INFO:teuthology.orchestra.run.smithi027.stdout: 9 2024-05-15T12:59:32.126 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.127 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.127 INFO:teuthology.orchestra.run.smithi027.stdout: "pgid": "4.9", 2024-05-15T12:59:32.127 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T12:59:32.127 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-05-15T12:59:32.127 INFO:teuthology.orchestra.run.smithi027.stdout: 3 2024-05-15T12:59:32.127 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.127 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T12:59:32.127 INFO:teuthology.orchestra.run.smithi027.stdout: "pgid": "4.f", 2024-05-15T12:59:32.127 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T12:59:32.127 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-05-15T12:59:32.127 INFO:teuthology.orchestra.run.smithi027.stdout: 3 2024-05-15T12:59:32.127 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T12:59:32.128 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.128 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T12:59:32.128 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_temp": [], 2024-05-15T12:59:32.128 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklist": { 2024-05-15T12:59:32.128 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6801/2530617473": "2024-05-16T12:39:19.409195+0000", 2024-05-15T12:59:32.128 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6800/2530617473": "2024-05-16T12:39:19.409195+0000", 2024-05-15T12:59:32.128 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/1880261204": "2024-05-16T12:39:19.409195+0000", 2024-05-15T12:59:32.128 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6801/3441136471": "2024-05-16T12:38:50.931011+0000", 2024-05-15T12:59:32.128 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/4291446247": "2024-05-16T12:39:19.409195+0000", 2024-05-15T12:59:32.128 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/580128209": "2024-05-16T12:38:50.931011+0000", 2024-05-15T12:59:32.128 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6800/3441136471": "2024-05-16T12:38:50.931011+0000", 2024-05-15T12:59:32.128 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/3011229857": "2024-05-16T12:38:50.931011+0000", 2024-05-15T12:59:32.128 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/2109693913": "2024-05-16T12:38:50.931011+0000", 2024-05-15T12:59:32.128 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/2445663024": "2024-05-16T12:39:19.409195+0000" 2024-05-15T12:59:32.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.129 INFO:teuthology.orchestra.run.smithi027.stdout: "range_blocklist": {}, 2024-05-15T12:59:32.129 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profiles": { 2024-05-15T12:59:32.129 INFO:teuthology.orchestra.run.smithi027.stdout: "default": { 2024-05-15T12:59:32.129 INFO:teuthology.orchestra.run.smithi027.stdout: "crush-failure-domain": "osd", 2024-05-15T12:59:32.129 INFO:teuthology.orchestra.run.smithi027.stdout: "k": "2", 2024-05-15T12:59:32.129 INFO:teuthology.orchestra.run.smithi027.stdout: "m": "1", 2024-05-15T12:59:32.129 INFO:teuthology.orchestra.run.smithi027.stdout: "plugin": "jerasure", 2024-05-15T12:59:32.129 INFO:teuthology.orchestra.run.smithi027.stdout: "technique": "reed_sol_van" 2024-05-15T12:59:32.129 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T12:59:32.129 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps_queue": [], 2024-05-15T12:59:32.129 INFO:teuthology.orchestra.run.smithi027.stdout: "new_removed_snaps": [], 2024-05-15T12:59:32.130 INFO:teuthology.orchestra.run.smithi027.stdout: "new_purged_snaps": [], 2024-05-15T12:59:32.130 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_node_flags": {}, 2024-05-15T12:59:32.130 INFO:teuthology.orchestra.run.smithi027.stdout: "device_class_flags": {}, 2024-05-15T12:59:32.130 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode": { 2024-05-15T12:59:32.130 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode_enabled": false, 2024-05-15T12:59:32.130 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_bucket_count": 0, 2024-05-15T12:59:32.130 INFO:teuthology.orchestra.run.smithi027.stdout: "degraded_stretch_mode": 0, 2024-05-15T12:59:32.130 INFO:teuthology.orchestra.run.smithi027.stdout: "recovering_stretch_mode": 0, 2024-05-15T12:59:32.130 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode_bucket": 0 2024-05-15T12:59:32.130 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T12:59:32.130 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T12:59:32.136 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 83 16 115113 115097 86.6588 37.6875 0.00427238 0.0115131 2024-05-15T12:59:32.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:32 smithi027 ceph-mon[39795]: pgmap v945: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.9 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 48 MiB/s wr, 773 op/s; 19153/340140 objects degraded (5.631%); 23659/340140 objects misplaced (6.956%); 5.9 MiB/s, 94 objects/s recovering 2024-05-15T12:59:32.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:32 smithi120 ceph-mon[43961]: pgmap v945: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.9 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 48 MiB/s wr, 773 op/s; 19153/340140 objects degraded (5.631%); 23659/340140 objects misplaced (6.956%); 5.9 MiB/s, 94 objects/s recovering 2024-05-15T12:59:32.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:32 smithi177 ceph-mon[44303]: pgmap v945: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.9 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 48 MiB/s wr, 773 op/s; 19153/340140 objects degraded (5.631%); 23659/340140 objects misplaced (6.956%); 5.9 MiB/s, 94 objects/s recovering 2024-05-15T12:59:33.137 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 84 16 115817 115801 86.1509 44 0.040271 0.0116029 2024-05-15T12:59:33.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:33 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 19153/340140 objects degraded (5.631%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:33.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:33 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 19153/340140 objects degraded (5.631%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:33.797 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T12:59:33.797 INFO:teuthology.orchestra.run.smithi120.stdout: "size": 20, 2024-05-15T12:59:33.797 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 600, 2024-05-15T12:59:33.797 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [] 2024-05-15T12:59:33.797 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T12:59:33.852 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T12:59:33.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:33 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 19153/340140 objects degraded (5.631%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:34.137 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 85 16 116466 116450 85.6146 40.5625 0.00823197 0.0116758 2024-05-15T12:59:34.551 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-05-15T12:59:34.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:34 smithi027 ceph-mon[39795]: pgmap v946: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.0 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 36 MiB/s wr, 579 op/s; 19153/342780 objects degraded (5.588%); 23471/342780 objects misplaced (6.847%); 5.0 MiB/s, 80 objects/s recovering 2024-05-15T12:59:34.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:34 smithi120 ceph-mon[43961]: pgmap v946: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.0 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 36 MiB/s wr, 579 op/s; 19153/342780 objects degraded (5.588%); 23471/342780 objects misplaced (6.847%); 5.0 MiB/s, 80 objects/s recovering 2024-05-15T12:59:34.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:34 smithi177 ceph-mon[44303]: pgmap v946: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.0 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 36 MiB/s wr, 579 op/s; 19153/342780 objects degraded (5.588%); 23471/342780 objects misplaced (6.847%); 5.0 MiB/s, 80 objects/s recovering 2024-05-15T12:59:34.956 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:59:35.137 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 86 16 117169 117153 85.1299 43.9375 0.0129001 0.0117424 2024-05-15T12:59:36.137 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 87 16 117911 117895 84.6844 46.375 0.0188792 0.0118041 2024-05-15T12:59:36.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:35 smithi027 ceph-mon[39795]: pgmap v947: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.1 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 44 MiB/s wr, 705 op/s; 19153/349875 objects degraded (5.474%); 23175/349875 objects misplaced (6.624%); 5.9 MiB/s, 94 objects/s recovering 2024-05-15T12:59:36.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:35 smithi120 ceph-mon[43961]: pgmap v947: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.1 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 44 MiB/s wr, 705 op/s; 19153/349875 objects degraded (5.474%); 23175/349875 objects misplaced (6.624%); 5.9 MiB/s, 94 objects/s recovering 2024-05-15T12:59:36.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:35 smithi177 ceph-mon[44303]: pgmap v947: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.1 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 44 MiB/s wr, 705 op/s; 19153/349875 objects degraded (5.474%); 23175/349875 objects misplaced (6.624%); 5.9 MiB/s, 94 objects/s recovering 2024-05-15T12:59:37.137 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 88 16 118623 118607 84.2278 44.5 0.0341152 0.011868 2024-05-15T12:59:37.774 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:59:37.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:37.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:59:37.817 INFO:tasks.thrashosds.thrasher:j is {'epoch': 349, 'fsid': 'deaea04e-12b7-11ef-bc98-c7b262605968', 'created': '2024-05-15T12:38:21.244556+0000', 'modified': '2024-05-15T12:59:29.199164+0000', 'last_up_change': '2024-05-15T12:45:50.876090+0000', 'last_in_change': '2024-05-15T12:59:10.294721+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-15T12:41:57.948613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9465229511260986, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 4, 'pool_name': 'unique_pool_2', 'create_time': '2024-05-15T12:57:53.956942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '321', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.990000009536743, 'score_stable': 2.990000009536743, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.75, 'raw_score_stable': 2.75, 'primary_affinity_weighted': 0.9465229511260986, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': '64fb3ad2-08a7-4106-a07a-6c776c877faf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 343, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6802', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6803', 'nonce': 4239116935}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6804', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6805', 'nonce': 4239116935}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6808', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6809', 'nonce': 4239116935}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6806', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6807', 'nonce': 4239116935}]}, 'public_addr': '172.21.15.27:6803/4239116935', 'cluster_addr': '172.21.15.27:6805/4239116935', 'heartbeat_back_addr': '172.21.15.27:6809/4239116935', 'heartbeat_front_addr': '172.21.15.27:6807/4239116935', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '9c16dd90-9d51-48b8-a296-1c8879ba1bc8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 343, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6810', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6811', 'nonce': 1466270637}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6812', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6813', 'nonce': 1466270637}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6816', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6817', 'nonce': 1466270637}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6814', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6815', 'nonce': 1466270637}]}, 'public_addr': '172.21.15.27:6811/1466270637', 'cluster_addr': '172.21.15.27:6813/1466270637', 'heartbeat_back_addr': '172.21.15.27:6817/1466270637', 'heartbeat_front_addr': '172.21.15.27:6815/1466270637', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'cbe3ba13-3ea3-4891-8d2d-6b85581a732c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6818', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6819', 'nonce': 3285413897}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6820', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6821', 'nonce': 3285413897}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6824', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6825', 'nonce': 3285413897}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6822', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6823', 'nonce': 3285413897}]}, 'public_addr': '172.21.15.27:6819/3285413897', 'cluster_addr': '172.21.15.27:6821/3285413897', 'heartbeat_back_addr': '172.21.15.27:6825/3285413897', 'heartbeat_front_addr': '172.21.15.27:6823/3285413897', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2e83735b-b944-4f87-8080-b994d4b96897', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 317, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6826', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6827', 'nonce': 3008975244}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6828', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6829', 'nonce': 3008975244}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6833', 'nonce': 3008975244}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6830', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6831', 'nonce': 3008975244}]}, 'public_addr': '172.21.15.27:6827/3008975244', 'cluster_addr': '172.21.15.27:6829/3008975244', 'heartbeat_back_addr': '172.21.15.27:6833/3008975244', 'heartbeat_front_addr': '172.21.15.27:6831/3008975244', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7c39c1dd-3867-46f7-8372-7688fa13b16a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 343, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6800', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6801', 'nonce': 3877842957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6802', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6803', 'nonce': 3877842957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6806', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6807', 'nonce': 3877842957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6804', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6805', 'nonce': 3877842957}]}, 'public_addr': '172.21.15.120:6801/3877842957', 'cluster_addr': '172.21.15.120:6803/3877842957', 'heartbeat_back_addr': '172.21.15.120:6807/3877842957', 'heartbeat_front_addr': '172.21.15.120:6805/3877842957', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a6dc220f-aec0-4454-9695-b44c0b9770e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 343, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6808', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6809', 'nonce': 1600179183}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6810', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6811', 'nonce': 1600179183}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6814', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6815', 'nonce': 1600179183}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6812', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6813', 'nonce': 1600179183}]}, 'public_addr': '172.21.15.120:6809/1600179183', 'cluster_addr': '172.21.15.120:6811/1600179183', 'heartbeat_back_addr': '172.21.15.120:6815/1600179183', 'heartbeat_front_addr': '172.21.15.120:6813/1600179183', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 344, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6816', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6817', 'nonce': 3323706735}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6818', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6819', 'nonce': 3323706735}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6822', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6823', 'nonce': 3323706735}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6820', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6821', 'nonce': 3323706735}]}, 'public_addr': '172.21.15.120:6817/3323706735', 'cluster_addr': '172.21.15.120:6819/3323706735', 'heartbeat_back_addr': '172.21.15.120:6823/3323706735', 'heartbeat_front_addr': '172.21.15.120:6821/3323706735', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6ef085d8-f59b-4033-8280-2c8f78693e30', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 219, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6824', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6825', 'nonce': 2960616153}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6826', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6827', 'nonce': 2960616153}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6830', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6831', 'nonce': 2960616153}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6828', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6829', 'nonce': 2960616153}]}, 'public_addr': '172.21.15.120:6825/2960616153', 'cluster_addr': '172.21.15.120:6827/2960616153', 'heartbeat_back_addr': '172.21.15.120:6831/2960616153', 'heartbeat_front_addr': '172.21.15.120:6829/2960616153', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '26514b11-d02b-46e9-a4e8-3da193fcdbcc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 292, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6800', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6801', 'nonce': 1132459315}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6802', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6803', 'nonce': 1132459315}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6806', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6807', 'nonce': 1132459315}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6804', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6805', 'nonce': 1132459315}]}, 'public_addr': '172.21.15.177:6801/1132459315', 'cluster_addr': '172.21.15.177:6803/1132459315', 'heartbeat_back_addr': '172.21.15.177:6807/1132459315', 'heartbeat_front_addr': '172.21.15.177:6805/1132459315', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '6f9497bc-e962-409d-8a2c-0c634cc1124f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9062042236328125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 317, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6808', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6809', 'nonce': 2710457646}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6810', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6811', 'nonce': 2710457646}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6814', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6815', 'nonce': 2710457646}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6812', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6813', 'nonce': 2710457646}]}, 'public_addr': '172.21.15.177:6809/2710457646', 'cluster_addr': '172.21.15.177:6811/2710457646', 'heartbeat_back_addr': '172.21.15.177:6815/2710457646', 'heartbeat_front_addr': '172.21.15.177:6813/2710457646', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'f852876f-c7d1-4039-90c4-5ab09dbd3e32', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.4520721435546875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 81, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6816', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6817', 'nonce': 3236900263}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6818', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6819', 'nonce': 3236900263}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6822', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6823', 'nonce': 3236900263}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6820', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6821', 'nonce': 3236900263}]}, 'public_addr': '172.21.15.177:6817/3236900263', 'cluster_addr': '172.21.15.177:6819/3236900263', 'heartbeat_back_addr': '172.21.15.177:6823/3236900263', 'heartbeat_front_addr': '172.21.15.177:6821/3236900263', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'cccc53c5-8476-4d55-b160-4bdc5cf92902', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 90, 'up_thru': 345, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6824', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6825', 'nonce': 681011628}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6826', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6827', 'nonce': 681011628}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6830', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6831', 'nonce': 681011628}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6828', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6829', 'nonce': 681011628}]}, 'public_addr': '172.21.15.177:6825/681011628', 'cluster_addr': '172.21.15.177:6827/681011628', 'heartbeat_back_addr': '172.21.15.177:6831/681011628', 'heartbeat_front_addr': '172.21.15.177:6829/681011628', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:58:36.082224+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:59:05.617321+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:58:38.235991+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:58:27.105368+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:59:03.869628+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:59:06.639157+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:58:49.959878+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-15T12:58:23.906382+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:59:25.546686+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:59:14.791769+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:58:14.898165+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:59:00.958960+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '4.1', 'osds': [4, 5, 6]}, {'pgid': '4.2', 'osds': [5, 9]}, {'pgid': '4.3', 'osds': [0, 1]}, {'pgid': '4.5', 'osds': [6, 10]}, {'pgid': '4.8', 'osds': [6, 9]}, {'pgid': '4.9', 'osds': [1, 3]}, {'pgid': '4.f', 'osds': [1, 3]}], 'primary_temp': [], 'blocklist': {'172.21.15.27:6801/2530617473': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:6800/2530617473': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:0/1880261204': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:6801/3441136471': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/4291446247': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:0/580128209': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:6800/3441136471': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/3011229857': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/2109693913': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/2445663024': '2024-05-16T12:39:19.409195+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-05-15T12:59:37.818 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T12:59:37.885 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:59:37.925 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:37.949 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:59:37.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:37.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:59:37.975 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T12:59:37.976 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T12:59:37.976 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:59:37.976 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25180.0:119225 4.0 e349/317 4:032f1029:::benchmark_data_smithi177_99756_object119224:head v 349'7523, mlcod=349'7521)", 2024-05-15T12:59:37.976 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:59:37.984133+0000", 2024-05-15T12:59:37.976 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0018903290000000001, 2024-05-15T12:59:37.976 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00192095, 2024-05-15T12:59:37.976 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T12:59:37.976 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T12:59:37.976 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T12:59:37.976 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:59:37.976 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T12:59:37.976 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:37.984133+0000", 2024-05-15T12:59:37.977 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:59:37.977 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:59:37.977 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:59:37.977 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T12:59:37.977 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:37.984133+0000", 2024-05-15T12:59:37.977 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T12:59:37.977 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:59:37.977 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:59:37.977 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T12:59:37.977 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:37.984135+0000", 2024-05-15T12:59:37.977 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5009999999999999e-06 2024-05-15T12:59:37.977 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:59:37.977 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:59:37.977 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T12:59:37.977 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:37.984198+0000", 2024-05-15T12:59:37.978 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3434000000000003e-05 2024-05-15T12:59:37.978 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:59:37.978 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:59:37.978 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T12:59:37.978 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:37.984214+0000", 2024-05-15T12:59:37.978 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5532e-05 2024-05-15T12:59:37.978 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:59:37.978 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:59:37.978 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T12:59:37.978 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:37.984223+0000", 2024-05-15T12:59:37.978 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7280000000000007e-06 2024-05-15T12:59:37.978 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:59:37.979 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:59:37.979 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T12:59:37.979 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:37.984287+0000", 2024-05-15T12:59:37.979 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3342000000000004e-05 2024-05-15T12:59:37.979 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T12:59:37.979 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T12:59:37.979 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T12:59:37.979 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:37.984311+0000", 2024-05-15T12:59:37.979 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3983000000000001e-05 2024-05-15T12:59:37.979 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:59:37.979 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T12:59:37.979 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:59:37.980 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T12:59:37.980 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-05-15T12:59:37.980 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 1 2024-05-15T12:59:37.980 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T12:59:38.054 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:59:38.056 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:37 smithi027 ceph-mon[39795]: pgmap v948: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.1 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 44 MiB/s wr, 703 op/s; 19153/351171 objects degraded (5.454%); 23175/351171 objects misplaced (6.599%); 5.9 MiB/s, 94 objects/s recovering 2024-05-15T12:59:38.056 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:37 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 19153/351171 objects degraded (5.454%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:38.101 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:38.137 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 89 16 119338 119322 83.7835 44.6875 0.00867751 0.0119302 2024-05-15T12:59:38.219 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:59:38.222 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:37 smithi177 ceph-mon[44303]: pgmap v948: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.1 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 44 MiB/s wr, 703 op/s; 19153/351171 objects degraded (5.454%); 23175/351171 objects misplaced (6.599%); 5.9 MiB/s, 94 objects/s recovering 2024-05-15T12:59:38.222 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:37 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 19153/351171 objects degraded (5.454%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:38.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:38.223 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:59:38.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:37 smithi120 ceph-mon[43961]: pgmap v948: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.1 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 44 MiB/s wr, 703 op/s; 19153/351171 objects degraded (5.454%); 23175/351171 objects misplaced (6.599%); 5.9 MiB/s, 94 objects/s recovering 2024-05-15T12:59:38.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:37 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 19153/351171 objects degraded (5.454%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:38.323 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:59:38.382 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:38.424 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T12:59:38.432 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:59:38.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:38.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:59:38.444 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:59:38.468 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T12:59:38.500 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T12:59:38.519 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:59:38.539 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T12:59:38.541 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:59:38.575 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:38.582 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T12:59:38.613 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T12:59:38.646 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T12:59:38.687 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T12:59:38.695 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:59:38.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:38.702 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:59:38.725 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T12:59:38.752 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-05-15T12:59:38.763 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T12:59:38.802 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:59:38.827 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T12:59:38.876 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T12:59:38.877 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:38.878 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T12:59:38.878 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T12:59:38.878 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T12:59:38.878 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T12:59:38.878 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T12:59:38.878 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T12:59:38.878 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T12:59:38.878 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T12:59:38.878 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T12:59:38.878 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T12:59:38.879 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T12:59:38.879 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T12:59:38.911 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:59:38.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:38.912 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:59:39.013 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:59:39.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:38 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1133990572' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:59:39.046 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:39.075 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:59:39.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:39.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:59:39.140 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 90 7 119969 119962 83.2942 40 0.024004 0.0120007 2024-05-15T12:59:39.175 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:59:39.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:38 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1133990572' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:59:39.180 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:59:39.211 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:39.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:38 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1133990572' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T12:59:39.461 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:59:39.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:39.465 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:59:39.566 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:59:39.593 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:39.612 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:59:39.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:39.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:59:39.715 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:59:39.741 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:39.761 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:59:39.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:39.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:59:39.864 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:59:39.930 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:39.991 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:59:39.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:39.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:59:40.099 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:59:40.127 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:39 smithi027 ceph-mon[39795]: pgmap v949: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.3 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 46 MiB/s wr, 735 op/s; 19153/356931 objects degraded (5.366%); 22843/356931 objects misplaced (6.400%); 5.5 MiB/s, 88 objects/s recovering 2024-05-15T12:59:40.129 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:40.140 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Total time run: 90.027 2024-05-15T12:59:40.141 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Total writes made: 119969 2024-05-15T12:59:40.141 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Write size: 65536 2024-05-15T12:59:40.141 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Object size: 65536 2024-05-15T12:59:40.141 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Bandwidth (MB/sec): 83.2868 2024-05-15T12:59:40.141 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Stddev Bandwidth: 47.5224 2024-05-15T12:59:40.141 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Max bandwidth (MB/sec): 249.375 2024-05-15T12:59:40.141 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Min bandwidth (MB/sec): 0 2024-05-15T12:59:40.141 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Average IOPS: 1332 2024-05-15T12:59:40.141 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Stddev IOPS: 760.367 2024-05-15T12:59:40.141 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Max IOPS: 3990 2024-05-15T12:59:40.141 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Min IOPS: 0 2024-05-15T12:59:40.141 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Average Latency(s): 0.012002 2024-05-15T12:59:40.142 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Stddev Latency(s): 0.0337151 2024-05-15T12:59:40.142 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Max latency(s): 2.60882 2024-05-15T12:59:40.142 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Min latency(s): 0.00111471 2024-05-15T12:59:40.146 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Cleaning up (deleting benchmark objects) 2024-05-15T12:59:40.215 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:39 smithi177 ceph-mon[44303]: pgmap v949: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.3 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 46 MiB/s wr, 735 op/s; 19153/356931 objects degraded (5.366%); 22843/356931 objects misplaced (6.400%); 5.5 MiB/s, 88 objects/s recovering 2024-05-15T12:59:40.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:39 smithi120 ceph-mon[43961]: pgmap v949: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.3 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 46 MiB/s wr, 735 op/s; 19153/356931 objects degraded (5.366%); 22843/356931 objects misplaced (6.400%); 5.5 MiB/s, 88 objects/s recovering 2024-05-15T12:59:40.352 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:59:40.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:40.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:59:40.458 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:59:40.501 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:40.536 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:59:40.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:40.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:59:40.639 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:59:40.681 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:40.783 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:59:40.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:40.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:59:40.891 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:59:40.959 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:40.995 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:59:40.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:40.998 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:59:41.099 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T12:59:41.135 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:40 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:59:41.135 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:40 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:59:41.137 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:41.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:40 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:59:41.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:40 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:59:41.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:40 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:59:41.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:40 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:59:41.298 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T12:59:41.306 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T12:59:41.306 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:59:41.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:41.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T12:59:41.409 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T12:59:41.449 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:41.568 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T12:59:41.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:41.579 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T12:59:41.680 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:59:41.713 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:41.741 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:59:41.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:41.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:59:41.845 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:59:41.860 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T12:59:41.907 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:42.023 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:59:42.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:42.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:59:42.047 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-05-15T12:59:42.134 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:59:42.171 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:42.194 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:59:42.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:42.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:59:42.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:41 smithi027 ceph-mon[39795]: osdmap e350: 12 total, 12 up, 11 in 2024-05-15T12:59:42.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:41 smithi027 ceph-mon[39795]: pgmap v951: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.3 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 40 MiB/s wr, 662 op/s; 19152/358890 objects degraded (5.336%); 22676/358890 objects misplaced (6.318%); 6.1 MiB/s, 98 objects/s recovering 2024-05-15T12:59:42.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:41 smithi120 ceph-mon[43961]: osdmap e350: 12 total, 12 up, 11 in 2024-05-15T12:59:42.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:41 smithi120 ceph-mon[43961]: pgmap v951: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.3 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 40 MiB/s wr, 662 op/s; 19152/358890 objects degraded (5.336%); 22676/358890 objects misplaced (6.318%); 6.1 MiB/s, 98 objects/s recovering 2024-05-15T12:59:42.298 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T12:59:42.331 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:42.354 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T12:59:42.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:42.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T12:59:42.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:41 smithi177 ceph-mon[44303]: osdmap e350: 12 total, 12 up, 11 in 2024-05-15T12:59:42.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:41 smithi177 ceph-mon[44303]: pgmap v951: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.3 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 40 MiB/s wr, 662 op/s; 19152/358890 objects degraded (5.336%); 22676/358890 objects misplaced (6.318%); 6.1 MiB/s, 98 objects/s recovering 2024-05-15T12:59:42.449 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:59:42.459 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:59:42.504 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:43.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:43 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1778939226' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:59:43.433 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T12:59:43.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:43 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1778939226' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:59:43.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:43 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1778939226' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T12:59:43.686 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:59:43.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:43.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:59:43.729 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:59:43.792 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T12:59:43.838 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:44.230 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T12:59:44.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:44.237 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T12:59:44.338 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:59:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:44 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:59:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:44 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:59:44.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:44 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:59:44.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:44 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1778939226' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:59:44.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:44 smithi177 ceph-mon[44303]: osdmap e351: 12 total, 12 up, 11 in 2024-05-15T12:59:44.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:44 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 19152/358890 objects degraded (5.336%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:44.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:44 smithi177 ceph-mon[44303]: pgmap v953: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.3 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 26 MiB/s wr, 505 op/s; 19149/357735 objects degraded (5.353%); 22430/357735 objects misplaced (6.270%); 5.8 MiB/s, 93 objects/s recovering 2024-05-15T12:59:44.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:44 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:59:44.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:44 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:59:44.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:44 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:59:44.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:44 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1778939226' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:59:44.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:44 smithi027 ceph-mon[39795]: osdmap e351: 12 total, 12 up, 11 in 2024-05-15T12:59:44.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:44 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 19152/358890 objects degraded (5.336%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:44.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:44 smithi027 ceph-mon[39795]: pgmap v953: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.3 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 26 MiB/s wr, 505 op/s; 19149/357735 objects degraded (5.353%); 22430/357735 objects misplaced (6.270%); 5.8 MiB/s, 93 objects/s recovering 2024-05-15T12:59:44.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:44 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:59:44.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:44 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:59:44.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:44 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T12:59:44.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:44 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1778939226' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T12:59:44.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:44 smithi120 ceph-mon[43961]: osdmap e351: 12 total, 12 up, 11 in 2024-05-15T12:59:44.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:44 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 19152/358890 objects degraded (5.336%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:44.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:44 smithi120 ceph-mon[43961]: pgmap v953: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.3 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 26 MiB/s wr, 505 op/s; 19149/357735 objects degraded (5.353%); 22430/357735 objects misplaced (6.270%); 5.8 MiB/s, 93 objects/s recovering 2024-05-15T12:59:44.836 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T12:59:44.836 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:59:45.512 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T12:59:45.512 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T12:59:45.584 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:45.650 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-05-15T12:59:46.066 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:59:46.893 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:46 smithi177 ceph-mon[44303]: from='client.15357 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:59:46.893 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:46 smithi177 ceph-mon[44303]: pgmap v954: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.1 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 127 B/s rd, 23 MiB/s wr, 728 op/s; 19147/351165 objects degraded (5.452%); 22089/351165 objects misplaced (6.290%); 8.4 MiB/s, 136 objects/s recovering 2024-05-15T12:59:47.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:46 smithi120 ceph-mon[43961]: from='client.15357 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:59:47.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:46 smithi120 ceph-mon[43961]: pgmap v954: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.1 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 127 B/s rd, 23 MiB/s wr, 728 op/s; 19147/351165 objects degraded (5.452%); 22089/351165 objects misplaced (6.290%); 8.4 MiB/s, 136 objects/s recovering 2024-05-15T12:59:47.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:46 smithi027 ceph-mon[39795]: from='client.15357 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:59:47.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:46 smithi027 ceph-mon[39795]: pgmap v954: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.1 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 127 B/s rd, 23 MiB/s wr, 728 op/s; 19147/351165 objects degraded (5.452%); 22089/351165 objects misplaced (6.290%); 8.4 MiB/s, 136 objects/s recovering 2024-05-15T12:59:47.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:47 smithi177 ceph-mon[44303]: pgmap v955: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.1 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 127 B/s rd, 7.8 MiB/s wr, 545 op/s; 19147/349812 objects degraded (5.474%); 22089/349812 objects misplaced (6.315%); 5.8 MiB/s, 94 objects/s recovering 2024-05-15T12:59:47.991 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T12:59:47.991 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:59:48.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:47 smithi027 ceph-mon[39795]: pgmap v955: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.1 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 127 B/s rd, 7.8 MiB/s wr, 545 op/s; 19147/349812 objects degraded (5.474%); 22089/349812 objects misplaced (6.315%); 5.8 MiB/s, 94 objects/s recovering 2024-05-15T12:59:48.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:47 smithi120 ceph-mon[43961]: pgmap v955: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.1 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 127 B/s rd, 7.8 MiB/s wr, 545 op/s; 19147/349812 objects degraded (5.474%); 22089/349812 objects misplaced (6.315%); 5.8 MiB/s, 94 objects/s recovering 2024-05-15T12:59:48.708 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-05-15T12:59:48.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:48 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 19147/349812 objects degraded (5.474%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:49.150 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:59:49.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:48 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 19147/349812 objects degraded (5.474%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:49.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:48 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 19147/349812 objects degraded (5.474%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:50.102 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:49 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:59:50.102 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:49 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:59:50.102 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:49 smithi177 ceph-mon[44303]: pgmap v956: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.0 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 115 B/s rd, 1.7 MiB/s wr, 609 op/s; 19146/344040 objects degraded (5.565%); 21708/344040 objects misplaced (6.310%); 6.8 MiB/s, 109 objects/s recovering 2024-05-15T12:59:50.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:49 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:59:50.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:49 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:59:50.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:49 smithi027 ceph-mon[39795]: pgmap v956: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.0 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 115 B/s rd, 1.7 MiB/s wr, 609 op/s; 19146/344040 objects degraded (5.565%); 21708/344040 objects misplaced (6.310%); 6.8 MiB/s, 109 objects/s recovering 2024-05-15T12:59:50.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:49 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:59:50.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:49 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:59:50.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:49 smithi120 ceph-mon[43961]: pgmap v956: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 7.0 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 115 B/s rd, 1.7 MiB/s wr, 609 op/s; 19146/344040 objects degraded (5.565%); 21708/344040 objects misplaced (6.310%); 6.8 MiB/s, 109 objects/s recovering 2024-05-15T12:59:51.146 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T12:59:51.146 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:59:51.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:50 smithi177 ceph-mon[44303]: osdmap e352: 12 total, 12 up, 11 in 2024-05-15T12:59:51.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:50 smithi027 ceph-mon[39795]: osdmap e352: 12 total, 12 up, 11 in 2024-05-15T12:59:51.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:50 smithi120 ceph-mon[43961]: osdmap e352: 12 total, 12 up, 11 in 2024-05-15T12:59:51.955 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-05-15T12:59:52.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:51 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:59:52.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:51 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:59:52.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:51 smithi177 ceph-mon[44303]: osdmap e353: 12 total, 12 up, 11 in 2024-05-15T12:59:52.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:51 smithi177 ceph-mon[44303]: pgmap v959: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.9 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 127 B/s rd, 112 KiB/s wr, 736 op/s; 19139/340158 objects degraded (5.627%); 21575/340158 objects misplaced (6.343%); 6.6 MiB/s, 108 objects/s recovering 2024-05-15T12:59:52.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:51 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:59:52.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:51 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:59:52.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:51 smithi027 ceph-mon[39795]: osdmap e353: 12 total, 12 up, 11 in 2024-05-15T12:59:52.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:51 smithi027 ceph-mon[39795]: pgmap v959: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.9 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 127 B/s rd, 112 KiB/s wr, 736 op/s; 19139/340158 objects degraded (5.627%); 21575/340158 objects misplaced (6.343%); 6.6 MiB/s, 108 objects/s recovering 2024-05-15T12:59:52.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:51 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:59:52.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:51 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:59:52.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:51 smithi120 ceph-mon[43961]: osdmap e353: 12 total, 12 up, 11 in 2024-05-15T12:59:52.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:51 smithi120 ceph-mon[43961]: pgmap v959: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.9 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 127 B/s rd, 112 KiB/s wr, 736 op/s; 19139/340158 objects degraded (5.627%); 21575/340158 objects misplaced (6.343%); 6.6 MiB/s, 108 objects/s recovering 2024-05-15T12:59:52.372 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:59:53.142 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T12:59:53.256 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:52 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:59:53.256 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:52 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:59:53.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:53.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T12:59:53.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:52 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:59:53.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:52 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:59:53.364 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T12:59:53.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:52 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:59:53.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:52 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:59:53.402 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:53.768 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T12:59:53.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:53.781 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T12:59:53.882 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T12:59:53.918 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:53.943 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T12:59:53.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:53.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T12:59:54.046 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:59:54.100 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:54 smithi177 ceph-mon[44303]: osdmap e354: 12 total, 12 up, 11 in 2024-05-15T12:59:54.100 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:54 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 19139/340158 objects degraded (5.627%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:54.101 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:54 smithi177 ceph-mon[44303]: pgmap v961: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.9 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 675 op/s; 19135/337653 objects degraded (5.667%); 21355/337653 objects misplaced (6.325%); 7.6 MiB/s, 124 objects/s recovering 2024-05-15T12:59:54.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:54 smithi027 ceph-mon[39795]: osdmap e354: 12 total, 12 up, 11 in 2024-05-15T12:59:54.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:54 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 19139/340158 objects degraded (5.627%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:54.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:54 smithi027 ceph-mon[39795]: pgmap v961: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.9 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 675 op/s; 19135/337653 objects degraded (5.667%); 21355/337653 objects misplaced (6.325%); 7.6 MiB/s, 124 objects/s recovering 2024-05-15T12:59:54.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:54 smithi120 ceph-mon[43961]: osdmap e354: 12 total, 12 up, 11 in 2024-05-15T12:59:54.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:54 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 19139/340158 objects degraded (5.627%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:54.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:54 smithi120 ceph-mon[43961]: pgmap v961: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.9 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 675 op/s; 19135/337653 objects degraded (5.667%); 21355/337653 objects misplaced (6.325%); 7.6 MiB/s, 124 objects/s recovering 2024-05-15T12:59:54.433 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T12:59:54.433 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:59:54.675 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:55.147 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:59:55.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:55.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:59:55.252 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:59:55.255 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-05-15T12:59:55.264 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T12:59:55.289 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:55.319 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:59:55.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:55.322 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:59:55.423 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T12:59:55.457 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:55.483 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T12:59:55.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:55.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T12:59:55.589 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T12:59:55.646 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:55.686 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T12:59:55.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:55.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T12:59:55.700 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:59:55.793 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T12:59:55.864 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:55.907 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T12:59:55.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T12:59:55.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T12:59:56.013 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T12:59:56.073 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T12:59:57.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:56 smithi027 ceph-mon[39795]: pgmap v962: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.8 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 597 op/s; 19135/333273 objects degraded (5.742%); 21176/333273 objects misplaced (6.354%); 5.5 MiB/s, 90 objects/s recovering 2024-05-15T12:59:57.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:56 smithi177 ceph-mon[44303]: pgmap v962: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.8 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 597 op/s; 19135/333273 objects degraded (5.742%); 21176/333273 objects misplaced (6.354%); 5.5 MiB/s, 90 objects/s recovering 2024-05-15T12:59:57.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:56 smithi120 ceph-mon[43961]: pgmap v962: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.8 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 597 op/s; 19135/333273 objects degraded (5.742%); 21176/333273 objects misplaced (6.354%); 5.5 MiB/s, 90 objects/s recovering 2024-05-15T12:59:57.942 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T12:59:57.942 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T12:59:58.183 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:57 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T12:59:58.183 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:57 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T12:59:58.183 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:57 smithi177 ceph-mon[44303]: pgmap v963: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.7 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 618 op/s; 19131/329361 objects degraded (5.809%); 21036/329361 objects misplaced (6.387%); 5.3 MiB/s, 86 objects/s recovering 2024-05-15T12:59:58.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:57 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T12:59:58.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:57 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T12:59:58.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:57 smithi027 ceph-mon[39795]: pgmap v963: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.7 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 618 op/s; 19131/329361 objects degraded (5.809%); 21036/329361 objects misplaced (6.387%); 5.3 MiB/s, 86 objects/s recovering 2024-05-15T12:59:58.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:57 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T12:59:58.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:57 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T12:59:58.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:57 smithi120 ceph-mon[43961]: pgmap v963: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.7 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 618 op/s; 19131/329361 objects degraded (5.809%); 21036/329361 objects misplaced (6.387%); 5.3 MiB/s, 86 objects/s recovering 2024-05-15T12:59:58.705 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-05-15T12:59:58.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:58 smithi177 ceph-mon[44303]: osdmap e355: 12 total, 12 up, 11 in 2024-05-15T12:59:58.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:58 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 19131/329361 objects degraded (5.809%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:59.176 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T12:59:59.276 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:58 smithi027 ceph-mon[39795]: osdmap e355: 12 total, 12 up, 11 in 2024-05-15T12:59:59.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:58 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 19131/329361 objects degraded (5.809%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T12:59:59.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:58 smithi120 ceph-mon[43961]: osdmap e355: 12 total, 12 up, 11 in 2024-05-15T12:59:59.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:58 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 19131/329361 objects degraded (5.809%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:00:00.044 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 12:59:59 smithi177 ceph-mon[44303]: pgmap v965: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.6 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 682 op/s; 19130/323781 objects degraded (5.908%); 20637/323781 objects misplaced (6.374%); 7.3 MiB/s, 118 objects/s recovering 2024-05-15T13:00:00.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 12:59:59 smithi027 ceph-mon[39795]: pgmap v965: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.6 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 682 op/s; 19130/323781 objects degraded (5.908%); 20637/323781 objects misplaced (6.374%); 7.3 MiB/s, 118 objects/s recovering 2024-05-15T13:00:00.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 12:59:59 smithi120 ceph-mon[43961]: pgmap v965: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.6 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 682 op/s; 19130/323781 objects degraded (5.908%); 20637/323781 objects misplaced (6.374%); 7.3 MiB/s, 118 objects/s recovering 2024-05-15T13:00:01.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:00 smithi177 ceph-mon[44303]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 19130/323781 objects degraded (5.908%), 4 pgs degraded 2024-05-15T13:00:01.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:00 smithi177 ceph-mon[44303]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-05-15T13:00:01.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:00 smithi177 ceph-mon[44303]: [WRN] PG_DEGRADED: Degraded data redundancy: 19130/323781 objects degraded (5.908%), 4 pgs degraded 2024-05-15T13:00:01.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:00 smithi177 ceph-mon[44303]: pg 4.2 is active+recovery_wait+undersized+degraded+remapped, acting [5,9] 2024-05-15T13:00:01.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:00 smithi177 ceph-mon[44303]: pg 4.8 is active+recovery_wait+undersized+degraded+remapped, acting [6,9] 2024-05-15T13:00:01.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:00 smithi177 ceph-mon[44303]: pg 4.9 is active+recovery_wait+undersized+degraded+remapped, acting [1,3] 2024-05-15T13:00:01.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:00 smithi177 ceph-mon[44303]: pg 4.f is active+recovering+undersized+degraded+remapped, acting [1,3] 2024-05-15T13:00:01.239 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:00:01.240 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:00:01.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:00 smithi027 ceph-mon[39795]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 19130/323781 objects degraded (5.908%), 4 pgs degraded 2024-05-15T13:00:01.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:00 smithi027 ceph-mon[39795]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-05-15T13:00:01.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:00 smithi027 ceph-mon[39795]: [WRN] PG_DEGRADED: Degraded data redundancy: 19130/323781 objects degraded (5.908%), 4 pgs degraded 2024-05-15T13:00:01.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:00 smithi027 ceph-mon[39795]: pg 4.2 is active+recovery_wait+undersized+degraded+remapped, acting [5,9] 2024-05-15T13:00:01.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:00 smithi027 ceph-mon[39795]: pg 4.8 is active+recovery_wait+undersized+degraded+remapped, acting [6,9] 2024-05-15T13:00:01.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:00 smithi027 ceph-mon[39795]: pg 4.9 is active+recovery_wait+undersized+degraded+remapped, acting [1,3] 2024-05-15T13:00:01.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:00 smithi027 ceph-mon[39795]: pg 4.f is active+recovering+undersized+degraded+remapped, acting [1,3] 2024-05-15T13:00:01.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:00 smithi120 ceph-mon[43961]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 19130/323781 objects degraded (5.908%), 4 pgs degraded 2024-05-15T13:00:01.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:00 smithi120 ceph-mon[43961]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-05-15T13:00:01.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:00 smithi120 ceph-mon[43961]: [WRN] PG_DEGRADED: Degraded data redundancy: 19130/323781 objects degraded (5.908%), 4 pgs degraded 2024-05-15T13:00:01.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:00 smithi120 ceph-mon[43961]: pg 4.2 is active+recovery_wait+undersized+degraded+remapped, acting [5,9] 2024-05-15T13:00:01.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:00 smithi120 ceph-mon[43961]: pg 4.8 is active+recovery_wait+undersized+degraded+remapped, acting [6,9] 2024-05-15T13:00:01.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:00 smithi120 ceph-mon[43961]: pg 4.9 is active+recovery_wait+undersized+degraded+remapped, acting [1,3] 2024-05-15T13:00:01.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:00 smithi120 ceph-mon[43961]: pg 4.f is active+recovering+undersized+degraded+remapped, acting [1,3] 2024-05-15T13:00:01.990 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-05-15T13:00:02.027 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":953,"stamp":"2024-05-15T12:59:43.841170+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7815299616,"num_objects":119245,"num_object_clones":0,"num_object_copies":357735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19149,"num_objects_misplaced":22430,"num_objects_unfound":0,"num_objects_dirty":119245,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":120848,"num_write_kb":7680058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2777,"num_bytes_recovered":184516320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120806,"ondisk_log_size":120806,"up":51,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":26230396,"kb_used_data":23182820,"kb_used_omap":17,"kb_used_meta":3047214,"kb_avail":1004966276,"statfs":{"total":1055945392128,"available":1029085466624,"internally_reserved":0,"allocated":23739207680,"data_stored":23736355884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17551,"internal_metadata":3120348017},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,1,3,10,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":171704320,"num_objects":2620,"num_object_clones":0,"num_object_copies":7860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":-745,"num_objects_unfound":0,"num_objects_dirty":2620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4044,"num_write_kb":213248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":749,"num_bytes_recovered":48824320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003008"},"pg_stats":[{"pgid":"4.f","version":"349'7374","reported_seq":8976,"reported_epoch":349,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-05-15T12:59:40.789930+0000","last_change":"2024-05-15T12:59:12.836225+0000","last_active":"2024-05-15T12:59:40.789930+0000","last_peered":"2024-05-15T12:59:40.789930+0000","last_clean":"2024-05-15T12:59:10.259362+0000","last_became_active":"2024-05-15T12:59:12.819643+0000","last_became_peered":"2024-05-15T12:59:12.819643+0000","last_unstale":"2024-05-15T12:59:40.789930+0000","last_undegraded":"2024-05-15T12:59:12.358355+0000","last_fullsized":"2024-05-15T12:59:12.315304+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":7374,"log_dups_size":0,"ondisk_log_size":7374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479330304,"num_objects":7314,"num_object_clones":0,"num_object_copies":21942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":5403,"num_objects_unfound":0,"num_objects_dirty":7314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7374,"num_write_kb":470016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":764,"num_bytes_recovered":50069504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3],"avail_no_missing":["1","3","9"],"object_location_counts":[{"shards":"1,3","objects":1908},{"shards":"1,3,9","objects":5403}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"349'7550","reported_seq":7624,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-05-15T12:59:39.126947+0000","last_change":"2024-05-15T12:57:55.242710+0000","last_active":"2024-05-15T12:59:39.126947+0000","last_peered":"2024-05-15T12:59:39.126947+0000","last_clean":"2024-05-15T12:59:39.126947+0000","last_became_active":"2024-05-15T12:57:55.242367+0000","last_became_peered":"2024-05-15T12:57:55.242367+0000","last_unstale":"2024-05-15T12:59:39.126947+0000","last_undegraded":"2024-05-15T12:59:39.126947+0000","last_fullsized":"2024-05-15T12:59:39.126947+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":7550,"log_dups_size":0,"ondisk_log_size":7550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":494796800,"num_objects":7550,"num_object_clones":0,"num_object_copies":22650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7550,"num_write_kb":483200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"349'7378","reported_seq":7452,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-05-15T12:59:39.117025+0000","last_change":"2024-05-15T12:57:55.155833+0000","last_active":"2024-05-15T12:59:39.117025+0000","last_peered":"2024-05-15T12:59:39.117025+0000","last_clean":"2024-05-15T12:59:39.117025+0000","last_became_active":"2024-05-15T12:57:55.155480+0000","last_became_peered":"2024-05-15T12:57:55.155480+0000","last_unstale":"2024-05-15T12:59:39.117025+0000","last_undegraded":"2024-05-15T12:59:39.117025+0000","last_fullsized":"2024-05-15T12:59:39.117025+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":7378,"log_dups_size":0,"ondisk_log_size":7378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483524608,"num_objects":7378,"num_object_clones":0,"num_object_copies":22134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7378,"num_write_kb":472192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"349'7490","reported_seq":7564,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-05-15T12:59:39.068050+0000","last_change":"2024-05-15T12:57:55.157244+0000","last_active":"2024-05-15T12:59:39.068050+0000","last_peered":"2024-05-15T12:59:39.068050+0000","last_clean":"2024-05-15T12:59:39.068050+0000","last_became_active":"2024-05-15T12:57:55.156859+0000","last_became_peered":"2024-05-15T12:57:55.156859+0000","last_unstale":"2024-05-15T12:59:39.068050+0000","last_undegraded":"2024-05-15T12:59:39.068050+0000","last_fullsized":"2024-05-15T12:59:39.068050+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":7490,"log_dups_size":0,"ondisk_log_size":7490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":490864640,"num_objects":7490,"num_object_clones":0,"num_object_copies":22470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7490,"num_write_kb":479360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"349'7486","reported_seq":7560,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-05-15T12:59:38.913461+0000","last_change":"2024-05-15T12:57:55.160034+0000","last_active":"2024-05-15T12:59:38.913461+0000","last_peered":"2024-05-15T12:59:38.913461+0000","last_clean":"2024-05-15T12:59:38.913461+0000","last_became_active":"2024-05-15T12:57:55.159699+0000","last_became_peered":"2024-05-15T12:57:55.159699+0000","last_unstale":"2024-05-15T12:59:38.913461+0000","last_undegraded":"2024-05-15T12:59:38.913461+0000","last_fullsized":"2024-05-15T12:59:38.913461+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":7486,"log_dups_size":0,"ondisk_log_size":7486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":490602496,"num_objects":7486,"num_object_clones":0,"num_object_copies":22458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7486,"num_write_kb":479104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"351'7610","reported_seq":7688,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-05-15T12:59:43.406821+0000","last_change":"2024-05-15T12:57:55.242821+0000","last_active":"2024-05-15T12:59:43.406821+0000","last_peered":"2024-05-15T12:59:43.406821+0000","last_clean":"2024-05-15T12:59:43.406821+0000","last_became_active":"2024-05-15T12:57:55.242372+0000","last_became_peered":"2024-05-15T12:57:55.242372+0000","last_unstale":"2024-05-15T12:59:43.406821+0000","last_undegraded":"2024-05-15T12:59:43.406821+0000","last_fullsized":"2024-05-15T12:59:43.406821+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":7610,"log_dups_size":0,"ondisk_log_size":7610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482607104,"num_objects":7364,"num_object_clones":0,"num_object_copies":22092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7610,"num_write_kb":479168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"349'7488","reported_seq":7570,"reported_epoch":349,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-15T12:59:40.724730+0000","last_change":"2024-05-15T12:59:12.862170+0000","last_active":"2024-05-15T12:59:40.724730+0000","last_peered":"2024-05-15T12:59:40.724730+0000","last_clean":"2024-05-15T12:59:10.318134+0000","last_became_active":"2024-05-15T12:59:12.847627+0000","last_became_peered":"2024-05-15T12:59:12.847627+0000","last_unstale":"2024-05-15T12:59:40.724730+0000","last_undegraded":"2024-05-15T12:59:12.454722+0000","last_fullsized":"2024-05-15T12:59:12.315028+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":7488,"log_dups_size":0,"ondisk_log_size":7488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487718912,"num_objects":7442,"num_object_clones":0,"num_object_copies":22326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7488,"num_write_kb":477760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[1,3],"avail_no_missing":["1","3","6"],"object_location_counts":[{"shards":"1,3","objects":1158},{"shards":"1,3,6","objects":6262}],"blocked_by":[],"up_primary":11,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"351'7873","reported_seq":7959,"reported_epoch":351,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-15T12:59:43.391358+0000","last_change":"2024-05-15T12:59:12.870715+0000","last_active":"2024-05-15T12:59:43.391358+0000","last_peered":"2024-05-15T12:59:43.391358+0000","last_clean":"2024-05-15T12:59:10.299125+0000","last_became_active":"2024-05-15T12:59:12.854810+0000","last_became_peered":"2024-05-15T12:59:12.854810+0000","last_unstale":"2024-05-15T12:59:43.391358+0000","last_undegraded":"2024-05-15T12:59:12.358625+0000","last_fullsized":"2024-05-15T12:59:12.315099+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":7873,"log_dups_size":0,"ondisk_log_size":7873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":494600192,"num_objects":7547,"num_object_clones":0,"num_object_copies":22641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7873,"num_write_kb":493440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,9],"acting":[6,9],"avail_no_missing":["6","9","10"],"object_location_counts":[{"shards":"6,9","objects":1045},{"shards":"6,9,10","objects":6342}],"blocked_by":[],"up_primary":11,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"349'7466","reported_seq":7540,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-05-15T12:59:40.750577+0000","last_change":"2024-05-15T12:57:55.242791+0000","last_active":"2024-05-15T12:59:40.750577+0000","last_peered":"2024-05-15T12:59:40.750577+0000","last_clean":"2024-05-15T12:59:40.750577+0000","last_became_active":"2024-05-15T12:57:55.242453+0000","last_became_peered":"2024-05-15T12:57:55.242453+0000","last_unstale":"2024-05-15T12:59:40.750577+0000","last_undegraded":"2024-05-15T12:59:40.750577+0000","last_fullsized":"2024-05-15T12:59:40.750577+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":7466,"log_dups_size":0,"ondisk_log_size":7466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485621760,"num_objects":7410,"num_object_clones":0,"num_object_copies":22230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7466,"num_write_kb":476032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"349'7559","reported_seq":7633,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-05-15T12:59:40.465236+0000","last_change":"2024-05-15T12:57:55.157512+0000","last_active":"2024-05-15T12:59:40.465236+0000","last_peered":"2024-05-15T12:59:40.465236+0000","last_clean":"2024-05-15T12:59:40.465236+0000","last_became_active":"2024-05-15T12:57:55.157131+0000","last_became_peered":"2024-05-15T12:57:55.157131+0000","last_unstale":"2024-05-15T12:59:40.465236+0000","last_undegraded":"2024-05-15T12:59:40.465236+0000","last_fullsized":"2024-05-15T12:59:40.465236+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":7559,"log_dups_size":0,"ondisk_log_size":7559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":493551616,"num_objects":7531,"num_object_clones":0,"num_object_copies":22593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7559,"num_write_kb":482880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"349'7496","reported_seq":7570,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-05-15T12:59:40.741207+0000","last_change":"2024-05-15T12:57:55.158173+0000","last_active":"2024-05-15T12:59:40.741207+0000","last_peered":"2024-05-15T12:59:40.741207+0000","last_clean":"2024-05-15T12:59:40.741207+0000","last_became_active":"2024-05-15T12:57:55.157465+0000","last_became_peered":"2024-05-15T12:57:55.157465+0000","last_unstale":"2024-05-15T12:59:40.741207+0000","last_undegraded":"2024-05-15T12:59:40.741207+0000","last_fullsized":"2024-05-15T12:59:40.741207+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":7496,"log_dups_size":0,"ondisk_log_size":7496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487456768,"num_objects":7438,"num_object_clones":0,"num_object_copies":22314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7496,"num_write_kb":477888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"349'7467","reported_seq":9194,"reported_epoch":349,"state":"active+recovering+undersized+remapped","last_fresh":"2024-05-15T12:59:38.947515+0000","last_change":"2024-05-15T12:59:12.892718+0000","last_active":"2024-05-15T12:59:38.947515+0000","last_peered":"2024-05-15T12:59:38.947515+0000","last_clean":"2024-05-15T12:59:10.291321+0000","last_became_active":"2024-05-15T12:59:12.843364+0000","last_became_peered":"2024-05-15T12:59:12.843364+0000","last_unstale":"2024-05-15T12:59:38.947515+0000","last_undegraded":"2024-05-15T12:59:38.947515+0000","last_fullsized":"2024-05-15T12:59:12.316583+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":7467,"log_dups_size":0,"ondisk_log_size":7467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489357312,"num_objects":7467,"num_object_clones":0,"num_object_copies":22401,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5425,"num_objects_unfound":0,"num_objects_dirty":7467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7467,"num_write_kb":477888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":825,"num_bytes_recovered":54067200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[0,1],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1","objects":2042},{"shards":"0,1,2","objects":5425}],"blocked_by":[],"up_primary":11,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"350'7603","reported_seq":7682,"reported_epoch":350,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-15T12:59:42.692708+0000","last_change":"2024-05-15T12:59:12.864423+0000","last_active":"2024-05-15T12:59:42.692708+0000","last_peered":"2024-05-15T12:59:42.692708+0000","last_clean":"2024-05-15T12:59:10.299975+0000","last_became_active":"2024-05-15T12:59:12.847592+0000","last_became_peered":"2024-05-15T12:59:12.847592+0000","last_unstale":"2024-05-15T12:59:42.692708+0000","last_undegraded":"2024-05-15T12:59:12.358581+0000","last_fullsized":"2024-05-15T12:59:12.320946+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":7603,"log_dups_size":0,"ondisk_log_size":7603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483196928,"num_objects":7373,"num_object_clones":0,"num_object_copies":22119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6360,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7603,"num_write_kb":479232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,11],"acting":[5,9],"avail_no_missing":["5","8","9"],"object_location_counts":[{"shards":"5,8,9","objects":6245},{"shards":"5,9","objects":1013}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"349'7557","reported_seq":7699,"reported_epoch":349,"state":"active+recovery_wait+remapped","last_fresh":"2024-05-15T12:59:39.152339+0000","last_change":"2024-05-15T12:59:12.857109+0000","last_active":"2024-05-15T12:59:39.152339+0000","last_peered":"2024-05-15T12:59:39.152339+0000","last_clean":"2024-05-15T12:59:10.257764+0000","last_became_active":"2024-05-15T12:59:12.843041+0000","last_became_peered":"2024-05-15T12:59:12.843041+0000","last_unstale":"2024-05-15T12:59:39.152339+0000","last_undegraded":"2024-05-15T12:59:39.152339+0000","last_fullsized":"2024-05-15T12:59:39.152339+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'5644","last_scrub_stamp":"2024-05-15T12:59:00.172752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:59:00.172752+0000","objects_scrubbed":5592,"log_size":7557,"log_dups_size":0,"ondisk_log_size":7557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":495255552,"num_objects":7557,"num_object_clones":0,"num_object_copies":22671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6387,"num_objects_unfound":0,"num_objects_dirty":7557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7557,"num_write_kb":483648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[4,5,6],"avail_no_missing":["4","5","6"],"object_location_counts":[{"shards":"4,5,6","objects":1170}],"blocked_by":[],"up_primary":11,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"350'7631","reported_seq":7707,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-05-15T12:59:41.420943+0000","last_change":"2024-05-15T12:57:55.160182+0000","last_active":"2024-05-15T12:59:41.420943+0000","last_peered":"2024-05-15T12:59:41.420943+0000","last_clean":"2024-05-15T12:59:41.420943+0000","last_became_active":"2024-05-15T12:57:55.159918+0000","last_became_peered":"2024-05-15T12:57:55.159918+0000","last_unstale":"2024-05-15T12:59:41.420943+0000","last_undegraded":"2024-05-15T12:59:41.420943+0000","last_fullsized":"2024-05-15T12:59:41.420943+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":7631,"log_dups_size":0,"ondisk_log_size":7631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491585536,"num_objects":7501,"num_object_clones":0,"num_object_copies":22503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7631,"num_write_kb":484224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"351'7639","reported_seq":10055,"reported_epoch":351,"state":"active+recovering+undersized+remapped","last_fresh":"2024-05-15T12:59:43.397014+0000","last_change":"2024-05-15T12:59:12.844855+0000","last_active":"2024-05-15T12:59:43.397014+0000","last_peered":"2024-05-15T12:59:43.397014+0000","last_clean":"2024-05-15T12:59:10.303422+0000","last_became_active":"2024-05-15T12:59:12.805581+0000","last_became_peered":"2024-05-15T12:59:12.805581+0000","last_unstale":"2024-05-15T12:59:43.397014+0000","last_undegraded":"2024-05-15T12:59:43.397014+0000","last_fullsized":"2024-05-15T12:59:12.315125+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":7639,"log_dups_size":0,"ondisk_log_size":7639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484638720,"num_objects":7395,"num_object_clones":0,"num_object_copies":22185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5215,"num_objects_unfound":0,"num_objects_dirty":7395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7639,"num_write_kb":481088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1168,"num_bytes_recovered":76546048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10],"avail_no_missing":["6","0","10"],"object_location_counts":[{"shards":"0,6,10","objects":5215},{"shards":"6,10","objects":2180}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":531,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-05-15T12:59:40.973157+0000","last_change":"2024-05-15T12:59:14.208120+0000","last_active":"2024-05-15T12:59:40.973157+0000","last_peered":"2024-05-15T12:59:40.973157+0000","last_clean":"2024-05-15T12:59:40.973157+0000","last_became_active":"2024-05-15T12:59:14.207746+0000","last_became_peered":"2024-05-15T12:59:14.207746+0000","last_unstale":"2024-05-15T12:59:40.973157+0000","last_undegraded":"2024-05-15T12:59:40.973157+0000","last_fullsized":"2024-05-15T12:59:40.973157+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T12:59:04.037279+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T12:59:04.037279+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:00:25.638859+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,9],"acting":[11,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":7814709248,"num_objects":119243,"num_object_clones":0,"num_object_copies":357729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19149,"num_objects_misplaced":22430,"num_objects_unfound":0,"num_objects_dirty":119243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120667,"num_write_kb":7677120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2763,"num_bytes_recovered":180813824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23696777216,"data_stored":23696769072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120667,"ondisk_log_size":120667,"up":48,"acting":42,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056811,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":819640,"kb_used_data":698208,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":92925512,"statfs":{"total":95995035648,"available":95155724288,"internally_reserved":0,"allocated":714964992,"data_stored":714703612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124320198},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351151,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2578160,"kb_used_data":2326232,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":91166992,"statfs":{"total":95995035648,"available":93354999808,"internally_reserved":0,"allocated":2382061568,"data_stored":2381803740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":257948096},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514676,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2697704,"kb_used_data":2323616,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":91047448,"statfs":{"total":95995035648,"available":93232586752,"internally_reserved":0,"allocated":2379382784,"data_stored":2379121180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":383056320},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874426,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1107744,"kb_used_data":891928,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":92637408,"statfs":{"total":95995035648,"available":94860705792,"internally_reserved":0,"allocated":913334272,"data_stored":913076240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":240518168765,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430403,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3539576,"kb_used_data":3255712,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":90205576,"statfs":{"total":95995035648,"available":92370509824,"internally_reserved":0,"allocated":3333849088,"data_stored":3333587728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":290650566},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575070,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4650020,"kb_used_data":4293980,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":89095132,"statfs":{"total":95995035648,"available":91233415168,"internally_reserved":0,"allocated":4397035520,"data_stored":4396773416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738597,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2647648,"kb_used_data":2323544,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":91097504,"statfs":{"total":95995035648,"available":93283844096,"internally_reserved":0,"allocated":2379309056,"data_stored":2379051024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,9,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411547,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1551072,"kb_used_data":1349144,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":92194080,"statfs":{"total":95995035648,"available":94406737920,"internally_reserved":0,"allocated":1381523456,"data_stored":1381265648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149908,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2755760,"kb_used_data":2373144,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":90989392,"statfs":{"total":95995035648,"available":93173137408,"internally_reserved":0,"allocated":2430099456,"data_stored":2429841648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":391576007},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986383,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2718496,"kb_used_data":2394392,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":91026656,"statfs":{"total":95995035648,"available":93211295744,"internally_reserved":0,"allocated":2451857408,"data_stored":2451599376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692041,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1164576,"kb_used_data":952920,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92580576,"statfs":{"total":95995035648,"available":94802509824,"internally_reserved":0,"allocated":975790080,"data_stored":975532272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2380333056,"data_stored":2380333056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4381347840,"data_stored":4381343768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1378877440,"data_stored":1378877440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2426798080,"data_stored":2426798080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2455437312,"data_stored":2455437312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":971440128,"data_stored":971440128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3326279680,"data_stored":3326279680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":912064512,"data_stored":912064512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2373521408,"data_stored":2373517336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2378694656,"data_stored":2378694656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":711983104,"data_stored":711983104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:00:02.029 INFO:tasks.thrashosds.thrasher:Setting 1.0 to [1, 6, 10] 2024-05-15T13:00:02.029 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.0', '1', '6', '10'] 2024-05-15T13:00:02.030 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pg-upmap 1.0 1 6 10 2024-05-15T13:00:02.062 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:00:02.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:02.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:00:02.174 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:00:02.208 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:01 smithi120 ceph-mon[43961]: pgmap v966: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.5 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 736 op/s; 19125/320019 objects degraded (5.976%); 20503/320019 objects misplaced (6.407%); 7.5 MiB/s, 122 objects/s recovering 2024-05-15T13:00:02.220 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:02.243 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:00:02.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:02.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:00:02.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:01 smithi027 ceph-mon[39795]: pgmap v966: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.5 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 736 op/s; 19125/320019 objects degraded (5.976%); 20503/320019 objects misplaced (6.407%); 7.5 MiB/s, 122 objects/s recovering 2024-05-15T13:00:02.346 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:00:02.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:01 smithi177 ceph-mon[44303]: pgmap v966: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.5 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 736 op/s; 19125/320019 objects degraded (5.976%); 20503/320019 objects misplaced (6.407%); 7.5 MiB/s, 122 objects/s recovering 2024-05-15T13:00:02.407 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:00:02.407 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:02.438 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:00:02.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:02.443 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:00:02.544 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:00:02.599 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:02.738 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:00:02.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:02.743 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:00:02.844 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:00:02.894 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:03.000 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:00:03.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:03.005 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:00:03.106 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:00:03.156 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:03.374 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:00:03.378 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:03 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 19125/320019 objects degraded (5.976%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:00:03.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:03.379 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:00:03.480 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:00:03.512 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:03 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 19125/320019 objects degraded (5.976%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:00:03.514 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:03.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:03 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 19125/320019 objects degraded (5.976%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:00:03.535 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:00:03.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:03.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:00:03.639 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:00:03.690 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:04.154 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:00:04.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:04.161 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:00:04.262 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:00:04.301 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:04.328 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:00:04.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:04.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:00:04.431 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:00:04.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:04 smithi177 ceph-mon[44303]: pgmap v967: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.5 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 672 op/s; 19125/317460 objects degraded (6.024%); 20283/317460 objects misplaced (6.389%); 6.7 MiB/s, 108 objects/s recovering 2024-05-15T13:00:04.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:04 smithi027 ceph-mon[39795]: pgmap v967: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.5 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 672 op/s; 19125/317460 objects degraded (6.024%); 20283/317460 objects misplaced (6.389%); 6.7 MiB/s, 108 objects/s recovering 2024-05-15T13:00:04.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:04 smithi120 ceph-mon[43961]: pgmap v967: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.5 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 672 op/s; 19125/317460 objects degraded (6.024%); 20283/317460 objects misplaced (6.389%); 6.7 MiB/s, 108 objects/s recovering 2024-05-15T13:00:04.552 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:04.828 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:00:04.828 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:00:05.440 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:00:05.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:05.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:00:05.449 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:00:05.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:05 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:00:05.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:05 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:00:05.547 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:00:05.548 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:05 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:00:05.548 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:05 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:00:05.582 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:05.586 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-05-15T13:00:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:00:05.707 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:00:05.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:05.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:00:05.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:05 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:00:05.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:05 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:00:05.836 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:00:05.874 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:05.896 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:00:05.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:05.904 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:00:05.945 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:00:06.005 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:00:06.053 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:06.127 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:00:06.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:06.134 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:00:06.236 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:00:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:06.360 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:00:06.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:06.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:00:06.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:06 smithi177 ceph-mon[44303]: osdmap e356: 12 total, 12 up, 11 in 2024-05-15T13:00:06.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:06 smithi177 ceph-mon[44303]: pgmap v969: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.4 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 680 op/s; 19125/313017 objects degraded (6.110%); 20100/313017 objects misplaced (6.421%); 7.3 MiB/s, 117 objects/s recovering 2024-05-15T13:00:06.467 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:00:06.529 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:06 smithi027 ceph-mon[39795]: osdmap e356: 12 total, 12 up, 11 in 2024-05-15T13:00:06.529 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:06 smithi027 ceph-mon[39795]: pgmap v969: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.4 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 680 op/s; 19125/313017 objects degraded (6.110%); 20100/313017 objects misplaced (6.421%); 7.3 MiB/s, 117 objects/s recovering 2024-05-15T13:00:06.530 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:06.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:06 smithi120 ceph-mon[43961]: osdmap e356: 12 total, 12 up, 11 in 2024-05-15T13:00:06.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:06 smithi120 ceph-mon[43961]: pgmap v969: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.4 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 680 op/s; 19125/313017 objects degraded (6.110%); 20100/313017 objects misplaced (6.421%); 7.3 MiB/s, 117 objects/s recovering 2024-05-15T13:00:07.796 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:00:07.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:07.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:00:07.908 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:00:07.989 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:08.012 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:00:08.012 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:00:08.020 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:00:08.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:08.024 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:00:08.125 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:00:08.166 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:08.275 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:00:08.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:08.280 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:00:08.380 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:00:08.431 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:08.458 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:00:08.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:08.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:00:08.568 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:00:08.604 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:08.635 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:00:08.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:08.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:00:08.670 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:00:08.725 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:00:08.742 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:00:08.757 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:00:08.790 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-05-15T13:00:08.798 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:08.800 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:00:08.822 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:00:08.829 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:00:08.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:08.833 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:00:08.852 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:00:08.899 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:00:08.933 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:00:08.933 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:00:08.978 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:08.982 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:00:09.011 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:00:09.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:09.017 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:00:09.017 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:08 smithi177 ceph-mon[44303]: pgmap v970: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.3 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 669 op/s; 19117/309432 objects degraded (6.178%); 19941/309432 objects misplaced (6.444%); 6.9 MiB/s, 111 objects/s recovering 2024-05-15T13:00:09.017 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:08 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 19117/309432 objects degraded (6.178%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:00:09.017 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:08 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1869913942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:00:09.018 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:00:09.060 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:00:09.097 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:00:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:00:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:00:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:00:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:00:09.099 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:00:09.099 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:00:09.099 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:00:09.099 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:00:09.099 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:00:09.099 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:00:09.099 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:00:09.099 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:00:09.118 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:00:09.178 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:09.213 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:00:09.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:09.223 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:00:09.235 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:00:09.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:08 smithi027 ceph-mon[39795]: pgmap v970: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.3 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 669 op/s; 19117/309432 objects degraded (6.178%); 19941/309432 objects misplaced (6.444%); 6.9 MiB/s, 111 objects/s recovering 2024-05-15T13:00:09.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:08 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 19117/309432 objects degraded (6.178%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:00:09.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:08 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1869913942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:00:09.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:08 smithi120 ceph-mon[43961]: pgmap v970: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.3 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 669 op/s; 19117/309432 objects degraded (6.178%); 19941/309432 objects misplaced (6.444%); 6.9 MiB/s, 111 objects/s recovering 2024-05-15T13:00:09.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:08 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 19117/309432 objects degraded (6.178%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:00:09.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:08 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1869913942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:00:09.323 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:00:09.401 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:09.437 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:00:09.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:09.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:00:09.544 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:00:09.593 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:10.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:09 smithi027 ceph-mon[39795]: pgmap v971: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.2 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 660 op/s; 19116/303942 objects degraded (6.289%); 19519/303942 objects misplaced (6.422%); 6.9 MiB/s, 113 objects/s recovering 2024-05-15T13:00:10.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:09 smithi120 ceph-mon[43961]: pgmap v971: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.2 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 660 op/s; 19116/303942 objects degraded (6.289%); 19519/303942 objects misplaced (6.422%); 6.9 MiB/s, 113 objects/s recovering 2024-05-15T13:00:10.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:09 smithi177 ceph-mon[44303]: pgmap v971: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.2 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 660 op/s; 19116/303942 objects degraded (6.289%); 19519/303942 objects misplaced (6.422%); 6.9 MiB/s, 113 objects/s recovering 2024-05-15T13:00:11.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:10 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:00:11.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:10 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:00:11.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:10 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:00:11.954 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:00:11.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:11.961 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:00:12.062 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:00:12.094 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:12.119 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:00:12.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:12.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:00:12.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:12 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:00:12.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:12 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:00:12.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:12 smithi177 ceph-mon[44303]: osdmap e357: 12 total, 12 up, 11 in 2024-05-15T13:00:12.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:12 smithi177 ceph-mon[44303]: pgmap v973: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.1 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 709 op/s; 19116/300423 objects degraded (6.363%); 19367/300423 objects misplaced (6.447%); 7.1 MiB/s, 115 objects/s recovering 2024-05-15T13:00:12.225 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:00:12.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:12 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:00:12.279 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:12 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:00:12.279 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:12 smithi027 ceph-mon[39795]: osdmap e357: 12 total, 12 up, 11 in 2024-05-15T13:00:12.279 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:12 smithi027 ceph-mon[39795]: pgmap v973: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.1 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 709 op/s; 19116/300423 objects degraded (6.363%); 19367/300423 objects misplaced (6.447%); 7.1 MiB/s, 115 objects/s recovering 2024-05-15T13:00:12.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:12 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:00:12.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:12 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:00:12.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:12 smithi120 ceph-mon[43961]: osdmap e357: 12 total, 12 up, 11 in 2024-05-15T13:00:12.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:12 smithi120 ceph-mon[43961]: pgmap v973: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.1 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 709 op/s; 19116/300423 objects degraded (6.363%); 19367/300423 objects misplaced (6.447%); 7.1 MiB/s, 115 objects/s recovering 2024-05-15T13:00:12.288 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:00:12.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:12.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:00:12.672 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:00:12.745 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:13.727 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:00:13.727 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T13:00:13.727 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-05-15T13:00:13.727 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-05-15T13:00:13.727 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:00:14.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:14 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 19116/300423 objects degraded (6.363%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:00:14.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:14 smithi177 ceph-mon[44303]: pgmap v974: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.1 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 610 op/s; 19115/297969 objects degraded (6.415%); 19133/297969 objects misplaced (6.421%); 7.0 MiB/s, 113 objects/s recovering 2024-05-15T13:00:14.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:14 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 19116/300423 objects degraded (6.363%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:00:14.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:14 smithi027 ceph-mon[39795]: pgmap v974: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.1 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 610 op/s; 19115/297969 objects degraded (6.415%); 19133/297969 objects misplaced (6.421%); 7.0 MiB/s, 113 objects/s recovering 2024-05-15T13:00:14.502 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-05-15T13:00:14.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:14 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 19116/300423 objects degraded (6.363%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:00:14.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:14 smithi120 ceph-mon[43961]: pgmap v974: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.1 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 610 op/s; 19115/297969 objects degraded (6.415%); 19133/297969 objects misplaced (6.421%); 7.0 MiB/s, 113 objects/s recovering 2024-05-15T13:00:14.928 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:00:16.161 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:00:16.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:16.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:00:16.276 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:00:16.317 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:16.346 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:00:16.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:16.360 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:00:16.461 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:00:16.521 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:16.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:16 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:00:16.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:16 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:00:16.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:16 smithi027 ceph-mon[39795]: osdmap e358: 12 total, 12 up, 11 in 2024-05-15T13:00:16.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:16 smithi027 ceph-mon[39795]: pgmap v976: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.0 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 650 op/s; 19115/293817 objects degraded (6.506%); 18933/293817 objects misplaced (6.444%); 7.8 MiB/s, 126 objects/s recovering 2024-05-15T13:00:16.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:16 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:00:16.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:16 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:00:16.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:16 smithi177 ceph-mon[44303]: osdmap e358: 12 total, 12 up, 11 in 2024-05-15T13:00:16.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:16 smithi177 ceph-mon[44303]: pgmap v976: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.0 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 650 op/s; 19115/293817 objects degraded (6.506%); 18933/293817 objects misplaced (6.444%); 7.8 MiB/s, 126 objects/s recovering 2024-05-15T13:00:16.943 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:00:16.944 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-05-15T13:00:16.944 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-05-15T13:00:16.944 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T13:00:16.944 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.944 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(4.5 347/343 [PushOp(4:a2501b4b:::benchmark_data_smithi177_99756_object2809:head, version: 322'155, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a2501b4b:::benchmark_data_smithi177_99756_object2809:head@322'155, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-05-15T13:00:16.944 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:59:17.544182+0000", 2024-05-15T13:00:16.944 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 59.408036074999998, 2024-05-15T13:00:16.944 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.088824388000000004, 2024-05-15T13:00:16.944 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:00:16.944 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T13:00:16.944 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:00:16.944 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.944 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:00:16.944 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:17.544182+0000", 2024-05-15T13:00:16.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.945 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.945 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:00:16.945 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:17.544182+0000", 2024-05-15T13:00:16.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.945 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.945 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:00:16.945 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:17.544184+0000", 2024-05-15T13:00:16.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3319999999999999e-06 2024-05-15T13:00:16.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.945 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.945 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:00:16.945 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:17.544240+0000", 2024-05-15T13:00:16.946 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5985e-05 2024-05-15T13:00:16.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.946 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.946 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:00:16.946 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:17.544263+0000", 2024-05-15T13:00:16.946 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3643999999999999e-05 2024-05-15T13:00:16.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.946 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.946 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:00:16.946 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:17.544280+0000", 2024-05-15T13:00:16.946 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6458999999999999e-05 2024-05-15T13:00:16.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:00:16.947 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:17.630673+0000", 2024-05-15T13:00:16.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.086393259999999999 2024-05-15T13:00:16.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:00:16.947 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:17.630696+0000", 2024-05-15T13:00:16.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3190999999999999e-05 2024-05-15T13:00:16.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:00:16.947 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:17.633007+0000", 2024-05-15T13:00:16.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023105170000000002 2024-05-15T13:00:16.948 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:16.948 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:00:16.948 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:16.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.948 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.948 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25180.0:109923 4.9 e347/343 4:9a4923ed:::benchmark_data_smithi177_99756_object109922:head v 347'6800, mlcod=0'0)", 2024-05-15T13:00:16.948 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:59:24.741305+0000", 2024-05-15T13:00:16.948 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 52.210913202999997, 2024-05-15T13:00:16.948 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.060121438999999999, 2024-05-15T13:00:16.948 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:00:16.948 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:16.948 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:00:16.948 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.948 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:00:16.948 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.741305+0000", 2024-05-15T13:00:16.948 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.949 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.949 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:00:16.949 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.741305+0000", 2024-05-15T13:00:16.949 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.949 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.949 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:00:16.949 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.741307+0000", 2024-05-15T13:00:16.949 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.446e-06 2024-05-15T13:00:16.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.949 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.949 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:00:16.949 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.741354+0000", 2024-05-15T13:00:16.949 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7363e-05 2024-05-15T13:00:16.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:00:16.950 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.741369+0000", 2024-05-15T13:00:16.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4381e-05 2024-05-15T13:00:16.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:00:16.950 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.741380+0000", 2024-05-15T13:00:16.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1229999999999999e-05 2024-05-15T13:00:16.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:00:16.950 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.741440+0000", 2024-05-15T13:00:16.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0560000000000003e-05 2024-05-15T13:00:16.952 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.952 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.952 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:00:16.952 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.741463+0000", 2024-05-15T13:00:16.952 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2696000000000001e-05 2024-05-15T13:00:16.952 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.952 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.952 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:00:16.952 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.801375+0000", 2024-05-15T13:00:16.952 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.059911851000000002 2024-05-15T13:00:16.953 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.953 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.953 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:00:16.953 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.801427+0000", 2024-05-15T13:00:16.953 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1912e-05 2024-05-15T13:00:16.953 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:16.953 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:00:16.953 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:16.953 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.953 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.953 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25180.0:109928 4.9 e347/343 4:9547408d:::benchmark_data_smithi177_99756_object109927:head v 347'6802, mlcod=0'0)", 2024-05-15T13:00:16.953 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:59:24.750779+0000", 2024-05-15T13:00:16.953 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 52.201439794999999, 2024-05-15T13:00:16.953 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.068091583999999997, 2024-05-15T13:00:16.953 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:00:16.954 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:16.954 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:00:16.954 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.954 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:00:16.954 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.750779+0000", 2024-05-15T13:00:16.954 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.954 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.954 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.954 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:00:16.954 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.750779+0000", 2024-05-15T13:00:16.954 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.954 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.954 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.955 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:00:16.955 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.750779+0000", 2024-05-15T13:00:16.955 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4299999999999999e-07 2024-05-15T13:00:16.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.955 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.955 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:00:16.955 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.750862+0000", 2024-05-15T13:00:16.955 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3393999999999995e-05 2024-05-15T13:00:16.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.955 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.955 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:00:16.955 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.750873+0000", 2024-05-15T13:00:16.955 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1043000000000001e-05 2024-05-15T13:00:16.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.955 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.955 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:00:16.956 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.750880+0000", 2024-05-15T13:00:16.956 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4869999999999999e-06 2024-05-15T13:00:16.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.956 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.956 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:00:16.956 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.760208+0000", 2024-05-15T13:00:16.956 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0093278400000000004 2024-05-15T13:00:16.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.956 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.956 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:00:16.956 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.760237+0000", 2024-05-15T13:00:16.956 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9374999999999999e-05 2024-05-15T13:00:16.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.957 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.957 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:00:16.957 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.818826+0000", 2024-05-15T13:00:16.957 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.058588564000000003 2024-05-15T13:00:16.957 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.957 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.957 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:00:16.957 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.818870+0000", 2024-05-15T13:00:16.957 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4638000000000002e-05 2024-05-15T13:00:16.957 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:16.957 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:00:16.957 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:16.957 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.957 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.958 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25180.0:109937 4.9 e347/343 4:90de552a:::benchmark_data_smithi177_99756_object109936:head v 347'6803, mlcod=0'0)", 2024-05-15T13:00:16.958 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:59:24.752664+0000", 2024-05-15T13:00:16.958 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 52.199554227, 2024-05-15T13:00:16.958 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.066239174999999997, 2024-05-15T13:00:16.958 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:00:16.958 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:16.958 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:00:16.958 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.958 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:00:16.958 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.752664+0000", 2024-05-15T13:00:16.958 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.958 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:00:16.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.752664+0000", 2024-05-15T13:00:16.959 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:00:16.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.752665+0000", 2024-05-15T13:00:16.959 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3399999999999998e-07 2024-05-15T13:00:16.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:00:16.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.752730+0000", 2024-05-15T13:00:16.959 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5368000000000002e-05 2024-05-15T13:00:16.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.960 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.960 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:00:16.960 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.752742+0000", 2024-05-15T13:00:16.960 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1503e-05 2024-05-15T13:00:16.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.960 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.960 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:00:16.960 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.752750+0000", 2024-05-15T13:00:16.960 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7859999999999997e-06 2024-05-15T13:00:16.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.960 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.960 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:00:16.960 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.767553+0000", 2024-05-15T13:00:16.961 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014803271999999999 2024-05-15T13:00:16.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.961 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:00:16.961 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.767584+0000", 2024-05-15T13:00:16.961 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0595000000000001e-05 2024-05-15T13:00:16.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.961 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:00:16.961 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.818874+0000", 2024-05-15T13:00:16.961 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.051290007999999998 2024-05-15T13:00:16.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.961 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:00:16.962 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.818903+0000", 2024-05-15T13:00:16.962 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9808999999999999e-05 2024-05-15T13:00:16.962 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:16.962 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:00:16.962 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:16.962 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.962 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.962 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25180.0:109946 4.9 e347/343 4:9c7704e2:::benchmark_data_smithi177_99756_object109945:head v 347'6804, mlcod=0'0)", 2024-05-15T13:00:16.962 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:59:24.762063+0000", 2024-05-15T13:00:16.962 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 52.190155410999999, 2024-05-15T13:00:16.962 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.056919483999999999, 2024-05-15T13:00:16.962 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:00:16.962 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:16.963 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:00:16.963 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.963 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:00:16.963 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.762063+0000", 2024-05-15T13:00:16.963 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.963 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.963 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.963 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:00:16.963 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.762063+0000", 2024-05-15T13:00:16.963 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.963 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.963 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.963 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:00:16.964 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.762064+0000", 2024-05-15T13:00:16.964 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3629999999999999e-06 2024-05-15T13:00:16.964 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.964 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.964 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:00:16.964 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.762124+0000", 2024-05-15T13:00:16.964 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9406e-05 2024-05-15T13:00:16.964 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.964 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.964 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:00:16.964 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.762148+0000", 2024-05-15T13:00:16.964 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3642000000000002e-05 2024-05-15T13:00:16.964 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.965 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.965 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:00:16.965 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.762159+0000", 2024-05-15T13:00:16.965 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1877e-05 2024-05-15T13:00:16.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.965 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.965 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:00:16.965 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.784217+0000", 2024-05-15T13:00:16.965 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.022057869000000001 2024-05-15T13:00:16.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.965 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.965 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:00:16.965 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.784241+0000", 2024-05-15T13:00:16.965 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4213e-05 2024-05-15T13:00:16.966 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.966 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.966 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:00:16.966 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.818947+0000", 2024-05-15T13:00:16.966 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.034705158 2024-05-15T13:00:16.966 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.966 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.966 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:00:16.966 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:24.818983+0000", 2024-05-15T13:00:16.966 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5955999999999999e-05 2024-05-15T13:00:16.966 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:16.966 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:00:16.966 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:16.967 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.967 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.967 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25180.0:110465 4.9 e347/343 4:904edaf6:::benchmark_data_smithi177_99756_object110464:head v 347'6840, mlcod=0'0)", 2024-05-15T13:00:16.967 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:59:25.518908+0000", 2024-05-15T13:00:16.967 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 51.433310986999999, 2024-05-15T13:00:16.967 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.056955438999999997, 2024-05-15T13:00:16.967 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:00:16.967 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:16.967 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:00:16.967 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.967 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:00:16.967 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:25.518908+0000", 2024-05-15T13:00:16.967 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.967 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.968 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.968 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:00:16.968 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:25.518908+0000", 2024-05-15T13:00:16.968 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.968 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.968 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.968 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:00:16.968 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:25.518909+0000", 2024-05-15T13:00:16.968 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2130000000000001e-06 2024-05-15T13:00:16.968 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.968 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.968 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:00:16.968 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:25.518942+0000", 2024-05-15T13:00:16.969 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3519000000000002e-05 2024-05-15T13:00:16.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.969 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.969 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:00:16.969 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:25.518954+0000", 2024-05-15T13:00:16.969 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1796e-05 2024-05-15T13:00:16.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.969 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.969 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:00:16.969 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:25.518962+0000", 2024-05-15T13:00:16.969 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7880000000000007e-06 2024-05-15T13:00:16.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.969 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.969 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:00:16.970 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:25.536663+0000", 2024-05-15T13:00:16.970 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017701313999999999 2024-05-15T13:00:16.970 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.970 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.970 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:00:16.970 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:25.536681+0000", 2024-05-15T13:00:16.970 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7785e-05 2024-05-15T13:00:16.970 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.970 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.970 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:00:16.970 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:25.575832+0000", 2024-05-15T13:00:16.970 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.039150898000000003 2024-05-15T13:00:16.970 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.970 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.970 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:00:16.971 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:25.575863+0000", 2024-05-15T13:00:16.971 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1126e-05 2024-05-15T13:00:16.971 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:16.971 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:00:16.971 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:16.971 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.971 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.971 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25180.0:111197 4.9 e348/343 4:9fb75b57:::benchmark_data_smithi177_99756_object111196:head v 348'6896, mlcod=0'0)", 2024-05-15T13:00:16.971 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:59:26.606023+0000", 2024-05-15T13:00:16.971 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 50.346195621, 2024-05-15T13:00:16.971 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.057861062999999997, 2024-05-15T13:00:16.971 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:00:16.971 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:16.971 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:00:16.972 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.972 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:00:16.972 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.606023+0000", 2024-05-15T13:00:16.972 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.972 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.972 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.972 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:00:16.972 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.606023+0000", 2024-05-15T13:00:16.972 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.972 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.972 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.972 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:00:16.972 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.606024+0000", 2024-05-15T13:00:16.973 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5700000000000002e-07 2024-05-15T13:00:16.973 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.973 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.973 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:00:16.973 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.606083+0000", 2024-05-15T13:00:16.973 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9500000000000003e-05 2024-05-15T13:00:16.973 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.973 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.973 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:00:16.973 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.606097+0000", 2024-05-15T13:00:16.973 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4163e-05 2024-05-15T13:00:16.973 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.973 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.973 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:00:16.973 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.606109+0000", 2024-05-15T13:00:16.974 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1727e-05 2024-05-15T13:00:16.974 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.974 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.974 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:00:16.974 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.622656+0000", 2024-05-15T13:00:16.974 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.016546752000000001 2024-05-15T13:00:16.974 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.974 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.974 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:00:16.974 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.622683+0000", 2024-05-15T13:00:16.974 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7682999999999999e-05 2024-05-15T13:00:16.974 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.974 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.975 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:00:16.975 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.663832+0000", 2024-05-15T13:00:16.975 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.041149046000000002 2024-05-15T13:00:16.975 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.975 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.975 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:00:16.975 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.663884+0000", 2024-05-15T13:00:16.975 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1535e-05 2024-05-15T13:00:16.975 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:16.975 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:00:16.975 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:16.975 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.975 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.975 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25180.0:111201 4.1 e348/343 4:8f0774b3:::benchmark_data_smithi177_99756_object111200:head v 348'6999, mlcod=0'0)", 2024-05-15T13:00:16.976 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:59:26.610591+0000", 2024-05-15T13:00:16.976 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 50.341627387999999, 2024-05-15T13:00:16.976 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.061656157000000003, 2024-05-15T13:00:16.976 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:00:16.976 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:16.976 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:00:16.976 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.976 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:00:16.976 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.610591+0000", 2024-05-15T13:00:16.976 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.976 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.976 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.976 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:00:16.977 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.610591+0000", 2024-05-15T13:00:16.977 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.977 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.977 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.977 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:00:16.977 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.610592+0000", 2024-05-15T13:00:16.977 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3499999999999995e-07 2024-05-15T13:00:16.977 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.977 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.977 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:00:16.977 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.610625+0000", 2024-05-15T13:00:16.977 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2858000000000003e-05 2024-05-15T13:00:16.977 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.978 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.978 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:00:16.978 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.610635+0000", 2024-05-15T13:00:16.978 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8349999999999999e-06 2024-05-15T13:00:16.978 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.978 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.978 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:00:16.978 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.610641+0000", 2024-05-15T13:00:16.978 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1700000000000002e-06 2024-05-15T13:00:16.978 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.978 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.978 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:00:16.978 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.630175+0000", 2024-05-15T13:00:16.978 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.019533991000000001 2024-05-15T13:00:16.979 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.979 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.979 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:00:16.979 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.630206+0000", 2024-05-15T13:00:16.979 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1446999999999999e-05 2024-05-15T13:00:16.979 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.979 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.979 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:00:16.979 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.672201+0000", 2024-05-15T13:00:16.979 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.041995027999999997 2024-05-15T13:00:16.979 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.979 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.979 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:00:16.980 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.672247+0000", 2024-05-15T13:00:16.980 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6093000000000001e-05 2024-05-15T13:00:16.980 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:16.980 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:00:16.980 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:16.980 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.980 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.980 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25180.0:111202 4.1 e348/343 4:889f5bb4:::benchmark_data_smithi177_99756_object111201:head v 348'7000, mlcod=0'0)", 2024-05-15T13:00:16.980 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:59:26.615492+0000", 2024-05-15T13:00:16.980 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 50.336726841000001, 2024-05-15T13:00:16.980 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.056786515000000003, 2024-05-15T13:00:16.980 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:00:16.980 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:16.981 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:00:16.981 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.981 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:00:16.981 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.615492+0000", 2024-05-15T13:00:16.981 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.981 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.981 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.981 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:00:16.981 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.615492+0000", 2024-05-15T13:00:16.981 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.981 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.981 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.981 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:00:16.982 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.615492+0000", 2024-05-15T13:00:16.982 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6100000000000001e-07 2024-05-15T13:00:16.982 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.982 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.982 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:00:16.982 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.615563+0000", 2024-05-15T13:00:16.982 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0870000000000004e-05 2024-05-15T13:00:16.982 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.982 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.982 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:00:16.982 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.615586+0000", 2024-05-15T13:00:16.982 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2952e-05 2024-05-15T13:00:16.983 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.983 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.983 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:00:16.983 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.615593+0000", 2024-05-15T13:00:16.983 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5239999999999997e-06 2024-05-15T13:00:16.983 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.983 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.983 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:00:16.983 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.639317+0000", 2024-05-15T13:00:16.983 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.023724655000000001 2024-05-15T13:00:16.983 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.983 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.983 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:00:16.983 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.639337+0000", 2024-05-15T13:00:16.984 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9715999999999999e-05 2024-05-15T13:00:16.984 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.984 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.984 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:00:16.984 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.672252+0000", 2024-05-15T13:00:16.984 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.032914994000000003 2024-05-15T13:00:16.984 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.984 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.984 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:00:16.984 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:26.672278+0000", 2024-05-15T13:00:16.984 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6242999999999999e-05 2024-05-15T13:00:16.984 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:16.984 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:00:16.984 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:16.984 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.985 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.985 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25180.0:111604 4.5 e348/343 4:a84de7e5:::benchmark_data_smithi177_99756_object111603:head v 348'7023, mlcod=322'497)", 2024-05-15T13:00:16.985 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:59:27.179873+0000", 2024-05-15T13:00:16.985 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 49.772345754, 2024-05-15T13:00:16.985 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.058511604000000002, 2024-05-15T13:00:16.985 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:00:16.985 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:16.985 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:00:16.985 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.985 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:00:16.985 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:27.179873+0000", 2024-05-15T13:00:16.986 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.986 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.986 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.986 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:00:16.986 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:27.179873+0000", 2024-05-15T13:00:16.986 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.986 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.986 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.986 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:00:16.986 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:27.179874+0000", 2024-05-15T13:00:16.986 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3099999999999996e-07 2024-05-15T13:00:16.986 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.987 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.987 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:00:16.987 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:27.179946+0000", 2024-05-15T13:00:16.987 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2515000000000006e-05 2024-05-15T13:00:16.987 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.987 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.987 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:00:16.987 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:27.179958+0000", 2024-05-15T13:00:16.987 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2007999999999999e-05 2024-05-15T13:00:16.987 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.987 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.987 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:00:16.987 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:27.179966+0000", 2024-05-15T13:00:16.987 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8650000000000001e-06 2024-05-15T13:00:16.987 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.988 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.988 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:00:16.988 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:27.191123+0000", 2024-05-15T13:00:16.988 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011157343 2024-05-15T13:00:16.988 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.988 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.988 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:00:16.988 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:27.191154+0000", 2024-05-15T13:00:16.988 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0685000000000002e-05 2024-05-15T13:00:16.988 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.988 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.988 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:00:16.988 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:27.238333+0000", 2024-05-15T13:00:16.988 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.047178605999999998 2024-05-15T13:00:16.989 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.989 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.989 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:00:16.989 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:27.238384+0000", 2024-05-15T13:00:16.989 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1650999999999997e-05 2024-05-15T13:00:16.989 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:16.989 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:00:16.989 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:16.989 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.989 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.989 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25180.0:111610 4.5 e348/343 4:a780be61:::benchmark_data_smithi177_99756_object111609:head v 348'7024, mlcod=322'497)", 2024-05-15T13:00:16.989 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:59:27.182367+0000", 2024-05-15T13:00:16.989 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 49.769851031000002, 2024-05-15T13:00:16.989 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.056044218999999999, 2024-05-15T13:00:16.990 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:00:16.990 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:16.990 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:00:16.990 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.990 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:00:16.990 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:27.182367+0000", 2024-05-15T13:00:16.990 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.990 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.990 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.990 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:00:16.990 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:27.182367+0000", 2024-05-15T13:00:16.990 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.990 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.990 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.991 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:00:16.991 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:27.182368+0000", 2024-05-15T13:00:16.991 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7199999999999997e-07 2024-05-15T13:00:16.991 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.991 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.991 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:00:16.991 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:27.183743+0000", 2024-05-15T13:00:16.991 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00137418 2024-05-15T13:00:16.991 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.991 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.991 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:00:16.991 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:27.183751+0000", 2024-05-15T13:00:16.991 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7050000000000005e-06 2024-05-15T13:00:16.991 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.992 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.992 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:00:16.992 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:27.183757+0000", 2024-05-15T13:00:16.992 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4450000000000004e-06 2024-05-15T13:00:16.992 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.992 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.992 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:00:16.992 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:27.199096+0000", 2024-05-15T13:00:16.992 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015338899 2024-05-15T13:00:16.992 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.992 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.992 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:00:16.992 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:27.199122+0000", 2024-05-15T13:00:16.993 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6418e-05 2024-05-15T13:00:16.993 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.993 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.993 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:00:16.993 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:27.238388+0000", 2024-05-15T13:00:16.993 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.039265503 2024-05-15T13:00:16.993 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.993 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.993 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:00:16.993 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:27.238412+0000", 2024-05-15T13:00:16.993 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4196999999999999e-05 2024-05-15T13:00:16.993 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:16.993 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:00:16.994 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:16.994 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.994 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.994 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25180.0:119327 4.3 e349/343 4:c6f1e10a:::benchmark_data_smithi177_99756_object119326:head v 349'7430, mlcod=322'804)", 2024-05-15T13:00:16.994 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:59:38.140836+0000", 2024-05-15T13:00:16.994 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 38.811382969, 2024-05-15T13:00:16.994 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.062326701999999998, 2024-05-15T13:00:16.994 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:00:16.994 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:16.994 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:00:16.994 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.994 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:00:16.994 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.140836+0000", 2024-05-15T13:00:16.995 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.995 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.995 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.995 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:00:16.995 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.140836+0000", 2024-05-15T13:00:16.995 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.995 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.995 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.995 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:00:16.995 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.140836+0000", 2024-05-15T13:00:16.995 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.15e-07 2024-05-15T13:00:16.995 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.995 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.996 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:00:16.996 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.140866+0000", 2024-05-15T13:00:16.996 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0309000000000001e-05 2024-05-15T13:00:16.996 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.996 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.996 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:00:16.996 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.140872+0000", 2024-05-15T13:00:16.996 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7060000000000001e-06 2024-05-15T13:00:16.996 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.996 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.996 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:00:16.996 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.140876+0000", 2024-05-15T13:00:16.996 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4510000000000002e-06 2024-05-15T13:00:16.996 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.997 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.997 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:00:16.997 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.152346+0000", 2024-05-15T13:00:16.997 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011470108999999999 2024-05-15T13:00:16.997 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.997 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.997 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:00:16.997 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.152374+0000", 2024-05-15T13:00:16.997 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7461999999999999e-05 2024-05-15T13:00:16.997 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.997 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.997 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:00:16.997 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.203101+0000", 2024-05-15T13:00:16.998 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.050727581000000001 2024-05-15T13:00:16.998 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.998 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.998 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:00:16.998 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.203162+0000", 2024-05-15T13:00:16.998 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0769e-05 2024-05-15T13:00:16.998 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:16.998 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:00:16.998 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:16.998 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.998 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.998 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25180.0:119332 4.3 e349/343 4:c5e297c3:::benchmark_data_smithi177_99756_object119331:head v 349'7431, mlcod=322'804)", 2024-05-15T13:00:16.998 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:59:38.142640+0000", 2024-05-15T13:00:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 38.809578356000003, 2024-05-15T13:00:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.060556947, 2024-05-15T13:00:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:00:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:00:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:00:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.142640+0000", 2024-05-15T13:00:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:00:16.999 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.142640+0000", 2024-05-15T13:00:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:00:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.142641+0000", 2024-05-15T13:00:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.156e-06 2024-05-15T13:00:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:00:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.142691+0000", 2024-05-15T13:00:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0028999999999998e-05 2024-05-15T13:00:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:00:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.142707+0000", 2024-05-15T13:00:17.000 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5485000000000002e-05 2024-05-15T13:00:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:00:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.142715+0000", 2024-05-15T13:00:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4880000000000007e-06 2024-05-15T13:00:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:00:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.159154+0000", 2024-05-15T13:00:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.016438357000000001 2024-05-15T13:00:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:00:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.159178+0000", 2024-05-15T13:00:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4227e-05 2024-05-15T13:00:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:00:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.203168+0000", 2024-05-15T13:00:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.043990475000000001 2024-05-15T13:00:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:00:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.203197+0000", 2024-05-15T13:00:17.002 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8730000000000001e-05 2024-05-15T13:00:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:00:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25180.0:119333 4.3 e349/343 4:c36bca73:::benchmark_data_smithi177_99756_object119332:head v 349'7432, mlcod=322'804)", 2024-05-15T13:00:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:59:38.146201+0000", 2024-05-15T13:00:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 38.806017433999997, 2024-05-15T13:00:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.057026300000000002, 2024-05-15T13:00:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:00:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:00:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.003 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:00:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.146201+0000", 2024-05-15T13:00:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:00:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.146201+0000", 2024-05-15T13:00:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:00:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.146202+0000", 2024-05-15T13:00:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.155e-06 2024-05-15T13:00:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.004 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:00:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.146240+0000", 2024-05-15T13:00:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7526e-05 2024-05-15T13:00:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:00:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.146252+0000", 2024-05-15T13:00:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2655e-05 2024-05-15T13:00:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:00:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.146261+0000", 2024-05-15T13:00:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9870000000000005e-06 2024-05-15T13:00:17.005 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:00:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.165989+0000", 2024-05-15T13:00:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.019727718000000002 2024-05-15T13:00:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:00:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.166011+0000", 2024-05-15T13:00:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1840000000000001e-05 2024-05-15T13:00:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:00:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.203200+0000", 2024-05-15T13:00:17.006 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.037189298000000003 2024-05-15T13:00:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:00:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:59:38.203227+0000", 2024-05-15T13:00:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7121e-05 2024-05-15T13:00:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:00:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25180.0:141912 4.8 e357/343 4:13e52297:::benchmark_data_smithi177_99756_object21940:head v 357'9105, mlcod=323'1308)", 2024-05-15T13:00:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:00:13.041771+0000", 2024-05-15T13:00:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.9104473679999998, 2024-05-15T13:00:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.073007498000000004, 2024-05-15T13:00:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:00:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:00:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:00:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.041771+0000", 2024-05-15T13:00:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:00:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.041771+0000", 2024-05-15T13:00:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.008 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:00:17.009 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.041772+0000", 2024-05-15T13:00:17.009 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0499999999999999e-06 2024-05-15T13:00:17.009 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.009 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.009 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:00:17.009 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.041781+0000", 2024-05-15T13:00:17.009 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9490000000000002e-06 2024-05-15T13:00:17.009 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.009 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.009 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:00:17.009 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.041793+0000", 2024-05-15T13:00:17.009 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1997999999999999e-05 2024-05-15T13:00:17.009 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.009 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.009 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:00:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.041801+0000", 2024-05-15T13:00:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1030000000000002e-06 2024-05-15T13:00:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:00:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.053009+0000", 2024-05-15T13:00:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011207287999999999 2024-05-15T13:00:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:00:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.053030+0000", 2024-05-15T13:00:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1557999999999999e-05 2024-05-15T13:00:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.010 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:00:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.114727+0000", 2024-05-15T13:00:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.061697212000000001 2024-05-15T13:00:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:00:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.114779+0000", 2024-05-15T13:00:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.134e-05 2024-05-15T13:00:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:00:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25180.0:141916 4.8 e357/343 4:1643dfd2:::benchmark_data_smithi177_99756_object21944:head v 357'9106, mlcod=323'1308)", 2024-05-15T13:00:17.011 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:00:13.045706+0000", 2024-05-15T13:00:17.012 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.906512523, 2024-05-15T13:00:17.012 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.069106219999999996, 2024-05-15T13:00:17.012 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:00:17.012 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:17.012 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:00:17.012 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.012 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:00:17.012 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.045706+0000", 2024-05-15T13:00:17.012 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:17.012 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.012 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.012 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:00:17.012 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.045706+0000", 2024-05-15T13:00:17.012 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:17.012 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:00:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.045707+0000", 2024-05-15T13:00:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.344e-06 2024-05-15T13:00:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:00:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.045718+0000", 2024-05-15T13:00:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0509999999999999e-05 2024-05-15T13:00:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:00:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.045730+0000", 2024-05-15T13:00:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2357e-05 2024-05-15T13:00:17.013 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:00:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.045738+0000", 2024-05-15T13:00:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5290000000000003e-06 2024-05-15T13:00:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:00:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.061625+0000", 2024-05-15T13:00:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015886891 2024-05-15T13:00:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:00:17.014 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.061655+0000", 2024-05-15T13:00:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.027e-05 2024-05-15T13:00:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:00:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.114782+0000", 2024-05-15T13:00:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.053126803 2024-05-15T13:00:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:00:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.114812+0000", 2024-05-15T13:00:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0516000000000001e-05 2024-05-15T13:00:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:00:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25180.0:141917 4.8 e357/343 4:15c403a7:::benchmark_data_smithi177_99756_object21945:head v 357'9107, mlcod=323'1308)", 2024-05-15T13:00:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:00:13.059027+0000", 2024-05-15T13:00:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 3.8931917619999998, 2024-05-15T13:00:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.055818169000000001, 2024-05-15T13:00:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:00:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:00:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:00:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.059027+0000", 2024-05-15T13:00:17.016 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:17.017 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.017 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.017 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:00:17.017 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.059027+0000", 2024-05-15T13:00:17.017 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:17.017 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.017 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.017 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:00:17.017 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.059028+0000", 2024-05-15T13:00:17.017 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0079999999999999e-06 2024-05-15T13:00:17.017 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.017 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.017 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:00:17.017 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.059035+0000", 2024-05-15T13:00:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5410000000000003e-06 2024-05-15T13:00:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:00:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.059052+0000", 2024-05-15T13:00:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6385999999999999e-05 2024-05-15T13:00:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:00:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.059061+0000", 2024-05-15T13:00:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9500000000000007e-06 2024-05-15T13:00:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.018 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:00:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.069810+0000", 2024-05-15T13:00:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010749788 2024-05-15T13:00:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:00:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.069842+0000", 2024-05-15T13:00:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1443999999999999e-05 2024-05-15T13:00:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:00:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.114816+0000", 2024-05-15T13:00:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.044974517999999998 2024-05-15T13:00:17.019 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:00:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:13.114845+0000", 2024-05-15T13:00:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8534000000000002e-05 2024-05-15T13:00:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:00:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25180.0:143379 4.8 e357/343 4:1ab9a300:::benchmark_data_smithi177_99756_object23407:head v 357'9206, mlcod=323'1308)", 2024-05-15T13:00:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:00:15.326620+0000", 2024-05-15T13:00:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.625598243, 2024-05-15T13:00:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.058161299999999999, 2024-05-15T13:00:17.020 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:00:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:00:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:00:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.326620+0000", 2024-05-15T13:00:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:00:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.326620+0000", 2024-05-15T13:00:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.021 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:00:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.326621+0000", 2024-05-15T13:00:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.001e-06 2024-05-15T13:00:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:00:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.326630+0000", 2024-05-15T13:00:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3559999999999993e-06 2024-05-15T13:00:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:00:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.326640+0000", 2024-05-15T13:00:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0643e-05 2024-05-15T13:00:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:00:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.326648+0000", 2024-05-15T13:00:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0069999999999997e-06 2024-05-15T13:00:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:00:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.329844+0000", 2024-05-15T13:00:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031961310000000001 2024-05-15T13:00:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:00:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.329874+0000", 2024-05-15T13:00:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9295e-05 2024-05-15T13:00:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.023 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:00:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.384754+0000", 2024-05-15T13:00:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.054880578999999999 2024-05-15T13:00:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:00:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.384782+0000", 2024-05-15T13:00:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7288e-05 2024-05-15T13:00:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:00:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.024 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25180.0:143381 4.8 e357/343 4:1163b2cb:::benchmark_data_smithi177_99756_object23409:head v 357'9207, mlcod=323'1308)", 2024-05-15T13:00:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:00:15.329400+0000", 2024-05-15T13:00:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.6228183760000001, 2024-05-15T13:00:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.055402137999999997, 2024-05-15T13:00:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:00:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:00:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:00:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.329400+0000", 2024-05-15T13:00:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:00:17.025 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.329400+0000", 2024-05-15T13:00:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:00:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.329401+0000", 2024-05-15T13:00:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3200000000000004e-07 2024-05-15T13:00:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:00:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.329407+0000", 2024-05-15T13:00:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1480000000000003e-06 2024-05-15T13:00:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.026 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:00:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.329419+0000", 2024-05-15T13:00:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2028000000000001e-05 2024-05-15T13:00:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:00:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.329426+0000", 2024-05-15T13:00:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2320000000000004e-06 2024-05-15T13:00:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:00:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.338555+0000", 2024-05-15T13:00:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0091286779999999994 2024-05-15T13:00:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.027 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:00:17.028 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.338591+0000", 2024-05-15T13:00:17.028 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5519000000000003e-05 2024-05-15T13:00:17.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.028 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.028 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:00:17.028 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.384784+0000", 2024-05-15T13:00:17.028 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.046193639000000002 2024-05-15T13:00:17.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.028 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.028 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:00:17.028 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.384802+0000", 2024-05-15T13:00:17.028 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8062000000000001e-05 2024-05-15T13:00:17.028 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:00:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25180.0:143384 4.8 e357/343 4:15b4278a:::benchmark_data_smithi177_99756_object23412:head v 357'9209, mlcod=323'1308)", 2024-05-15T13:00:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:00:15.343910+0000", 2024-05-15T13:00:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.608308163, 2024-05-15T13:00:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.057262446000000002, 2024-05-15T13:00:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:00:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:00:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:00:17.029 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.343910+0000", 2024-05-15T13:00:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:00:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.343910+0000", 2024-05-15T13:00:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:00:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:00:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.343912+0000", 2024-05-15T13:00:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.762e-06 2024-05-15T13:00:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.030 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:00:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.343920+0000", 2024-05-15T13:00:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3559999999999993e-06 2024-05-15T13:00:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:00:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.343930+0000", 2024-05-15T13:00:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9159999999999996e-06 2024-05-15T13:00:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:00:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.343941+0000", 2024-05-15T13:00:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0208e-05 2024-05-15T13:00:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.031 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:00:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.359066+0000", 2024-05-15T13:00:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015125761 2024-05-15T13:00:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:00:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.359095+0000", 2024-05-15T13:00:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8839e-05 2024-05-15T13:00:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:00:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.401135+0000", 2024-05-15T13:00:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.04203966 2024-05-15T13:00:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:00:17.032 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:00:17.033 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:00:17.033 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:15.401173+0000", 2024-05-15T13:00:17.033 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7944000000000001e-05 2024-05-15T13:00:17.033 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:17.033 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:00:17.033 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:17.033 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:00:17.033 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:00:17.033 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:00:17.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:16 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:00:17.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:16 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:00:17.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:16 smithi120 ceph-mon[43961]: osdmap e358: 12 total, 12 up, 11 in 2024-05-15T13:00:17.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:16 smithi120 ceph-mon[43961]: pgmap v976: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 6.0 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 650 op/s; 19115/293817 objects degraded (6.506%); 18933/293817 objects misplaced (6.444%); 7.8 MiB/s, 126 objects/s recovering 2024-05-15T13:00:17.677 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-05-15T13:00:17.731 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:00:17.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:17.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:00:17.754 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:00:17.845 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:00:17.915 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:17.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:17 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2072011865' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:00:17.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:17 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:00:17.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:17 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:00:18.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:17 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2072011865' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:00:18.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:17 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:00:18.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:17 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:00:18.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:17 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2072011865' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:00:18.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:17 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:00:18.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:17 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:00:18.100 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:00:18.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:18.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:00:18.213 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:00:18.248 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:18.274 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:00:18.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:18.277 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:00:18.378 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:00:18.414 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:18.446 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:00:18.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:18.450 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:00:18.551 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:00:18.594 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:18.624 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:00:18.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:18.628 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:00:18.712 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:18 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:00:18.712 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:18 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2072011865' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:00:18.713 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:18 smithi027 ceph-mon[39795]: osdmap e359: 12 total, 12 up, 11 in 2024-05-15T13:00:18.713 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:18 smithi027 ceph-mon[39795]: pgmap v978: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 519 op/s; 19109/290340 objects degraded (6.582%); 18766/290340 objects misplaced (6.463%); 5.4 MiB/s, 88 objects/s recovering 2024-05-15T13:00:18.713 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:18 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 19115/293817 objects degraded (6.506%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:18.713 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:00:18.713 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:00:18.713 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:00:18.730 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:00:18.776 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:18.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:18 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:00:18.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:18 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2072011865' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:00:18.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:18 smithi177 ceph-mon[44303]: osdmap e359: 12 total, 12 up, 11 in 2024-05-15T13:00:18.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:18 smithi177 ceph-mon[44303]: pgmap v978: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 519 op/s; 19109/290340 objects degraded (6.582%); 18766/290340 objects misplaced (6.463%); 5.4 MiB/s, 88 objects/s recovering 2024-05-15T13:00:18.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:18 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 19115/293817 objects degraded (6.506%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:18.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:00:18.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:00:18.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:00:19.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:18 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:00:19.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:18 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2072011865' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:00:19.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:18 smithi120 ceph-mon[43961]: osdmap e359: 12 total, 12 up, 11 in 2024-05-15T13:00:19.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:18 smithi120 ceph-mon[43961]: pgmap v978: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 519 op/s; 19109/290340 objects degraded (6.582%); 18766/290340 objects misplaced (6.463%); 5.4 MiB/s, 88 objects/s recovering 2024-05-15T13:00:19.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:18 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 19115/293817 objects degraded (6.506%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:19.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:00:19.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:00:19.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:00:19.722 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:00:20.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:20 smithi177 ceph-mon[44303]: pgmap v979: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.8 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 626 op/s; 19109/285369 objects degraded (6.696%); 18282/285369 objects misplaced (6.406%); 8.4 MiB/s, 136 objects/s recovering 2024-05-15T13:00:20.956 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:00:20.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:20.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:00:21.021 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:00:21.042 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:00:21.064 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:00:21.097 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:21.123 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:00:21.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:21.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:00:21.228 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:00:21.254 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:20 smithi120 ceph-mon[43961]: pgmap v979: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.8 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 626 op/s; 19109/285369 objects degraded (6.696%); 18282/285369 objects misplaced (6.406%); 8.4 MiB/s, 136 objects/s recovering 2024-05-15T13:00:21.257 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:21.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:20 smithi027 ceph-mon[39795]: pgmap v979: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.8 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 626 op/s; 19109/285369 objects degraded (6.696%); 18282/285369 objects misplaced (6.406%); 8.4 MiB/s, 136 objects/s recovering 2024-05-15T13:00:21.280 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:00:21.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:21.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:00:21.386 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:00:21.420 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:21.441 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:00:21.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:21.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:00:21.550 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:00:21.614 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:21.649 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:00:21.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:21.655 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:00:21.756 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:00:21.797 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:21.825 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:00:21.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:21.828 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:00:21.929 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:00:21.965 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:21.985 INFO:teuthology.orchestra.run.smithi027.stderr:set 1.0 pg_upmap mapping to [1,6,10] 2024-05-15T13:00:21.994 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:00:21.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:22.000 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:00:22.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:21 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2339523254' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [1, 6, 10]}]: dispatch 2024-05-15T13:00:22.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:21 smithi027 ceph-mon[39795]: pgmap v980: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 649 op/s; 19106/282363 objects degraded (6.766%); 18093/282363 objects misplaced (6.408%); 8.1 MiB/s, 130 objects/s recovering 2024-05-15T13:00:22.100 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:00:22.137 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:22.161 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:00:22.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:22.164 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:00:22.265 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:00:22.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:21 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2339523254' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [1, 6, 10]}]: dispatch 2024-05-15T13:00:22.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:21 smithi120 ceph-mon[43961]: pgmap v980: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 649 op/s; 19106/282363 objects degraded (6.766%); 18093/282363 objects misplaced (6.408%); 8.1 MiB/s, 130 objects/s recovering 2024-05-15T13:00:22.324 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:22.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:21 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2339523254' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [1, 6, 10]}]: dispatch 2024-05-15T13:00:22.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:21 smithi177 ceph-mon[44303]: pgmap v980: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 649 op/s; 19106/282363 objects degraded (6.766%); 18093/282363 objects misplaced (6.408%); 8.1 MiB/s, 130 objects/s recovering 2024-05-15T13:00:22.836 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:00:23.108 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:00:23.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:23.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:00:23.222 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:00:23.248 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:22 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2339523254' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [1, 6, 10]}]': finished 2024-05-15T13:00:23.248 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:22 smithi027 ceph-mon[39795]: osdmap e360: 12 total, 12 up, 11 in 2024-05-15T13:00:23.261 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:22 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2339523254' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [1, 6, 10]}]': finished 2024-05-15T13:00:23.261 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:22 smithi120 ceph-mon[43961]: osdmap e360: 12 total, 12 up, 11 in 2024-05-15T13:00:23.264 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:23.284 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:00:23.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:23.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:00:23.388 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:00:23.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:22 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2339523254' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [1, 6, 10]}]': finished 2024-05-15T13:00:23.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:22 smithi177 ceph-mon[44303]: osdmap e360: 12 total, 12 up, 11 in 2024-05-15T13:00:23.437 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:23.468 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:00:23.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:23.476 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:00:23.577 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:00:23.605 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:23.637 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:00:23.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:23.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:00:23.740 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:00:23.803 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:23.963 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:23 smithi120 ceph-mon[43961]: osdmap e361: 12 total, 12 up, 11 in 2024-05-15T13:00:23.963 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:23 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 19106/282363 objects degraded (6.766%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:24.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:23 smithi027 ceph-mon[39795]: osdmap e361: 12 total, 12 up, 11 in 2024-05-15T13:00:24.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:23 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 19106/282363 objects degraded (6.766%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:24.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:23 smithi027 ceph-mon[39795]: pgmap v983: 17 pgs: 1 remapped+peering, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+clean; 5.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 584 op/s; 19099/280110 objects degraded (6.818%); 17810/280110 objects misplaced (6.358%); 9.7 MiB/s, 157 objects/s recovering 2024-05-15T13:00:24.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:23 smithi120 ceph-mon[43961]: pgmap v983: 17 pgs: 1 remapped+peering, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+clean; 5.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 584 op/s; 19099/280110 objects degraded (6.818%); 17810/280110 objects misplaced (6.358%); 9.7 MiB/s, 157 objects/s recovering 2024-05-15T13:00:24.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:23 smithi177 ceph-mon[44303]: osdmap e361: 12 total, 12 up, 11 in 2024-05-15T13:00:24.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:23 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 19106/282363 objects degraded (6.766%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:24.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:23 smithi177 ceph-mon[44303]: pgmap v983: 17 pgs: 1 remapped+peering, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+clean; 5.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 584 op/s; 19099/280110 objects degraded (6.818%); 17810/280110 objects misplaced (6.358%); 9.7 MiB/s, 157 objects/s recovering 2024-05-15T13:00:25.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:24 smithi027 ceph-mon[39795]: osdmap e362: 12 total, 12 up, 11 in 2024-05-15T13:00:25.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:24 smithi027 ceph-mon[39795]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-05-15T13:00:25.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:24 smithi120 ceph-mon[43961]: osdmap e362: 12 total, 12 up, 11 in 2024-05-15T13:00:25.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:24 smithi120 ceph-mon[43961]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-05-15T13:00:25.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:24 smithi177 ceph-mon[44303]: osdmap e362: 12 total, 12 up, 11 in 2024-05-15T13:00:25.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:24 smithi177 ceph-mon[44303]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-05-15T13:00:25.582 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:00:25.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:25.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:00:25.689 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:00:25.733 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:25.764 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:00:25.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:25.767 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:00:25.868 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:00:25.920 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:26.027 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: osdmap e363: 12 total, 12 up, 11 in 2024-05-15T13:00:26.027 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {auth=31,config=2,health=12,kv=54,logm=1} crc {auth=736313099,config=1059521453,health=1320705514,kv=1699799615,logm=2400247494}) 2024-05-15T13:00:26.027 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2876488869}) 2024-05-15T13:00:26.027 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=686199501}) 2024-05-15T13:00:26.027 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1616148306}) 2024-05-15T13:00:26.028 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4275824167}) 2024-05-15T13:00:26.028 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3998821739}) 2024-05-15T13:00:26.028 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3001491736}) 2024-05-15T13:00:26.028 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2451772996}) 2024-05-15T13:00:26.028 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1597541375}) 2024-05-15T13:00:26.028 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3119014430}) 2024-05-15T13:00:26.028 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=425805924}) 2024-05-15T13:00:26.028 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=690354095}) 2024-05-15T13:00:26.028 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=659643006}) 2024-05-15T13:00:26.028 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487252965}) 2024-05-15T13:00:26.028 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2782855003}) 2024-05-15T13:00:26.028 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2568277089}) 2024-05-15T13:00:26.028 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3484924554}) 2024-05-15T13:00:26.028 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3695821502}) 2024-05-15T13:00:26.029 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=864006334}) 2024-05-15T13:00:26.029 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2296682603}) 2024-05-15T13:00:26.029 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3396739540}) 2024-05-15T13:00:26.029 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3472198773}) 2024-05-15T13:00:26.029 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=997402248}) 2024-05-15T13:00:26.029 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1431518687}) 2024-05-15T13:00:26.029 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=963994662}) 2024-05-15T13:00:26.029 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1664765251}) 2024-05-15T13:00:26.029 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=594639042}) 2024-05-15T13:00:26.029 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1784648504}) 2024-05-15T13:00:26.029 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=447657171}) 2024-05-15T13:00:26.029 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=558600328}) 2024-05-15T13:00:26.029 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4186764182}) 2024-05-15T13:00:26.030 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=695669480}) 2024-05-15T13:00:26.030 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4179973952}) 2024-05-15T13:00:26.030 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4139883268}) 2024-05-15T13:00:26.030 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=249190038}) 2024-05-15T13:00:26.030 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2831802327}) 2024-05-15T13:00:26.030 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=1229981835,mdsmap=3984425268,mgr=1024455899}) 2024-05-15T13:00:26.030 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mgr=12,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=71} crc {mgr=490527139,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=2779858940,mon_config_key=639212462}) 2024-05-15T13:00:26.030 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1343966446}) 2024-05-15T13:00:26.030 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3861605617}) 2024-05-15T13:00:26.030 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=4,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=78} crc {mon_config_key=2572886359,monmap=2435304556,osd_metadata=1212035326,osd_pg_creating=3300993390,osdmap=766446339}) 2024-05-15T13:00:26.030 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=39} crc {osdmap=2724119868}) 2024-05-15T13:00:26.030 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:26 smithi177 ceph-mon[44303]: pgmap v986: 17 pgs: 1 active+recovery_wait+remapped, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+clean; 5.6 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 525 op/s; 19099/276057 objects degraded (6.918%); 17605/276057 objects misplaced (6.377%); 7.6 MiB/s, 123 objects/s recovering 2024-05-15T13:00:26.221 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:00:26.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:26.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:00:26.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: osdmap e363: 12 total, 12 up, 11 in 2024-05-15T13:00:26.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {auth=31,config=2,health=12,kv=54,logm=1} crc {auth=736313099,config=1059521453,health=1320705514,kv=1699799615,logm=2400247494}) 2024-05-15T13:00:26.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2876488869}) 2024-05-15T13:00:26.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=686199501}) 2024-05-15T13:00:26.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1616148306}) 2024-05-15T13:00:26.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4275824167}) 2024-05-15T13:00:26.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3998821739}) 2024-05-15T13:00:26.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3001491736}) 2024-05-15T13:00:26.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2451772996}) 2024-05-15T13:00:26.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1597541375}) 2024-05-15T13:00:26.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3119014430}) 2024-05-15T13:00:26.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=425805924}) 2024-05-15T13:00:26.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=690354095}) 2024-05-15T13:00:26.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=659643006}) 2024-05-15T13:00:26.234 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487252965}) 2024-05-15T13:00:26.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2782855003}) 2024-05-15T13:00:26.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2568277089}) 2024-05-15T13:00:26.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3484924554}) 2024-05-15T13:00:26.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3695821502}) 2024-05-15T13:00:26.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=864006334}) 2024-05-15T13:00:26.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2296682603}) 2024-05-15T13:00:26.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3396739540}) 2024-05-15T13:00:26.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3472198773}) 2024-05-15T13:00:26.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=997402248}) 2024-05-15T13:00:26.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1431518687}) 2024-05-15T13:00:26.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=963994662}) 2024-05-15T13:00:26.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1664765251}) 2024-05-15T13:00:26.235 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=594639042}) 2024-05-15T13:00:26.236 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1784648504}) 2024-05-15T13:00:26.236 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=447657171}) 2024-05-15T13:00:26.236 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=558600328}) 2024-05-15T13:00:26.236 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4186764182}) 2024-05-15T13:00:26.236 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=695669480}) 2024-05-15T13:00:26.236 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4179973952}) 2024-05-15T13:00:26.236 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4139883268}) 2024-05-15T13:00:26.236 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=249190038}) 2024-05-15T13:00:26.236 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2831802327}) 2024-05-15T13:00:26.236 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=1229981835,mdsmap=3984425268,mgr=1024455899}) 2024-05-15T13:00:26.236 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mgr=12,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=71} crc {mgr=490527139,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=2779858940,mon_config_key=639212462}) 2024-05-15T13:00:26.236 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1343966446}) 2024-05-15T13:00:26.236 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3861605617}) 2024-05-15T13:00:26.236 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=4,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=78} crc {mon_config_key=2572886359,monmap=2435304556,osd_metadata=1212035326,osd_pg_creating=3300993390,osdmap=766446339}) 2024-05-15T13:00:26.236 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=39} crc {osdmap=2724119868}) 2024-05-15T13:00:26.237 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:26 smithi027 ceph-mon[39795]: pgmap v986: 17 pgs: 1 active+recovery_wait+remapped, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+clean; 5.6 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 525 op/s; 19099/276057 objects degraded (6.918%); 17605/276057 objects misplaced (6.377%); 7.6 MiB/s, 123 objects/s recovering 2024-05-15T13:00:26.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: osdmap e363: 12 total, 12 up, 11 in 2024-05-15T13:00:26.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {auth=31,config=2,health=12,kv=54,logm=1} crc {auth=736313099,config=1059521453,health=1320705514,kv=1699799615,logm=2400247494}) 2024-05-15T13:00:26.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2876488869}) 2024-05-15T13:00:26.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=686199501}) 2024-05-15T13:00:26.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1616148306}) 2024-05-15T13:00:26.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4275824167}) 2024-05-15T13:00:26.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3998821739}) 2024-05-15T13:00:26.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3001491736}) 2024-05-15T13:00:26.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2451772996}) 2024-05-15T13:00:26.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1597541375}) 2024-05-15T13:00:26.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3119014430}) 2024-05-15T13:00:26.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=425805924}) 2024-05-15T13:00:26.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=690354095}) 2024-05-15T13:00:26.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=659643006}) 2024-05-15T13:00:26.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487252965}) 2024-05-15T13:00:26.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2782855003}) 2024-05-15T13:00:26.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2568277089}) 2024-05-15T13:00:26.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3484924554}) 2024-05-15T13:00:26.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3695821502}) 2024-05-15T13:00:26.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=864006334}) 2024-05-15T13:00:26.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2296682603}) 2024-05-15T13:00:26.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3396739540}) 2024-05-15T13:00:26.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3472198773}) 2024-05-15T13:00:26.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=997402248}) 2024-05-15T13:00:26.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1431518687}) 2024-05-15T13:00:26.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=963994662}) 2024-05-15T13:00:26.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1664765251}) 2024-05-15T13:00:26.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=594639042}) 2024-05-15T13:00:26.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1784648504}) 2024-05-15T13:00:26.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=447657171}) 2024-05-15T13:00:26.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=558600328}) 2024-05-15T13:00:26.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4186764182}) 2024-05-15T13:00:26.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=695669480}) 2024-05-15T13:00:26.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4179973952}) 2024-05-15T13:00:26.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4139883268}) 2024-05-15T13:00:26.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=249190038}) 2024-05-15T13:00:26.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2831802327}) 2024-05-15T13:00:26.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=1229981835,mdsmap=3984425268,mgr=1024455899}) 2024-05-15T13:00:26.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mgr=12,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=71} crc {mgr=490527139,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=2779858940,mon_config_key=639212462}) 2024-05-15T13:00:26.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1343966446}) 2024-05-15T13:00:26.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3861605617}) 2024-05-15T13:00:26.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=4,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=78} crc {mon_config_key=2572886359,monmap=2435304556,osd_metadata=1212035326,osd_pg_creating=3300993390,osdmap=766446339}) 2024-05-15T13:00:26.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=39} crc {osdmap=2724119868}) 2024-05-15T13:00:26.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:26 smithi120 ceph-mon[43961]: pgmap v986: 17 pgs: 1 active+recovery_wait+remapped, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+clean; 5.6 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 525 op/s; 19099/276057 objects degraded (6.918%); 17605/276057 objects misplaced (6.377%); 7.6 MiB/s, 123 objects/s recovering 2024-05-15T13:00:26.334 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:00:26.395 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:26.431 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:00:26.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:26.436 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:00:26.537 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:00:26.569 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:26.592 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:00:26.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:26.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:00:26.696 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:00:26.725 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:26.766 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:00:26.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:26.771 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:00:26.872 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:00:26.910 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:26.930 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:00:26.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:26.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:00:27.024 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:27 smithi177 ceph-mon[44303]: osdmap e364: 12 total, 12 up, 11 in 2024-05-15T13:00:27.034 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:00:27.104 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:27.227 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:00:27.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:27.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:00:27.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:27 smithi120 ceph-mon[43961]: osdmap e364: 12 total, 12 up, 11 in 2024-05-15T13:00:27.332 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:00:27.407 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:27 smithi027 ceph-mon[39795]: osdmap e364: 12 total, 12 up, 11 in 2024-05-15T13:00:27.410 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:27.426 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:00:27.426 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:00:27.426 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.426 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:150503 4.b 4:d68935f2:::benchmark_data_smithi177_99756_object30531:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-05-15T13:00:27.426 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:27.401120+0000", 2024-05-15T13:00:27.426 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.033761357999999998, 2024-05-15T13:00:27.426 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.033797467999999997, 2024-05-15T13:00:27.426 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:27.426 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:00:27.427 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:00:27.427 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:00:27.427 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:00:27.427 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 150503 2024-05-15T13:00:27.427 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.427 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:27.427 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.427 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:27.427 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.401120+0000", 2024-05-15T13:00:27.427 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:27.427 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.427 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.428 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:27.428 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.401120+0000", 2024-05-15T13:00:27.428 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:27.428 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.428 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.428 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:27.428 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.401122+0000", 2024-05-15T13:00:27.428 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6050000000000001e-06 2024-05-15T13:00:27.428 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.428 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.428 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:27.428 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.401129+0000", 2024-05-15T13:00:27.429 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9940000000000003e-06 2024-05-15T13:00:27.429 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.429 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.429 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:27.429 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.401674+0000", 2024-05-15T13:00:27.429 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00054572799999999999 2024-05-15T13:00:27.429 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.429 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.429 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:27.429 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.401686+0000", 2024-05-15T13:00:27.429 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1342000000000001e-05 2024-05-15T13:00:27.429 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.429 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.429 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:00:27.430 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.402195+0000", 2024-05-15T13:00:27.430 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00050954699999999995 2024-05-15T13:00:27.430 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.430 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.430 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:00:27.430 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.402319+0000", 2024-05-15T13:00:27.430 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012414899999999999 2024-05-15T13:00:27.430 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.430 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.430 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T13:00:27.430 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.402431+0000", 2024-05-15T13:00:27.430 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000111815 2024-05-15T13:00:27.430 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.430 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.431 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:27.431 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.418182+0000", 2024-05-15T13:00:27.431 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.015751052000000001 2024-05-15T13:00:27.431 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.431 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.431 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:00:27.431 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.418195+0000", 2024-05-15T13:00:27.431 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3278999999999999e-05 2024-05-15T13:00:27.431 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:27.431 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:27.431 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:27.431 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.431 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.431 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:150509 4.3 4:c5d0dd37:::benchmark_data_smithi177_99756_object30537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-05-15T13:00:27.432 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:27.415035+0000", 2024-05-15T13:00:27.432 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.019846424000000001, 2024-05-15T13:00:27.432 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.019975112, 2024-05-15T13:00:27.432 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:27.432 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:00:27.432 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:00:27.432 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:00:27.432 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:00:27.432 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 150509 2024-05-15T13:00:27.432 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.432 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:27.432 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.432 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:27.432 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.415035+0000", 2024-05-15T13:00:27.432 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:27.433 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.433 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.433 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:27.433 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.415035+0000", 2024-05-15T13:00:27.433 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:27.433 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.433 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.433 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:27.433 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.415036+0000", 2024-05-15T13:00:27.433 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.204e-06 2024-05-15T13:00:27.433 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.433 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.433 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:27.433 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.415045+0000", 2024-05-15T13:00:27.433 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8959999999999992e-06 2024-05-15T13:00:27.434 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.434 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.434 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:27.434 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.415055+0000", 2024-05-15T13:00:27.434 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0159999999999999e-05 2024-05-15T13:00:27.434 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.434 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.434 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:27.434 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.415061+0000", 2024-05-15T13:00:27.434 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0410000000000003e-06 2024-05-15T13:00:27.434 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.434 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.434 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:00:27.434 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.421759+0000", 2024-05-15T13:00:27.435 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0066974979999999996 2024-05-15T13:00:27.435 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.435 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.435 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:00:27.435 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.421900+0000", 2024-05-15T13:00:27.435 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000141507 2024-05-15T13:00:27.435 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.435 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.435 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,11", 2024-05-15T13:00:27.435 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.422021+0000", 2024-05-15T13:00:27.435 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120767 2024-05-15T13:00:27.435 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.435 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.435 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:00:27.436 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.430700+0000", 2024-05-15T13:00:27.436 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0086794509999999995 2024-05-15T13:00:27.436 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:27.436 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:27.436 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:27.436 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.436 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.436 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:150510 4.3 4:c9252df9:::benchmark_data_smithi177_99756_object30538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-05-15T13:00:27.436 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:27.415085+0000", 2024-05-15T13:00:27.436 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.019795821000000002, 2024-05-15T13:00:27.436 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.020015505999999999, 2024-05-15T13:00:27.436 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:27.436 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:00:27.436 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:00:27.437 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:00:27.437 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:00:27.437 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 150510 2024-05-15T13:00:27.437 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.437 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:27.437 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.437 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:27.437 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.415085+0000", 2024-05-15T13:00:27.437 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:27.437 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.437 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.437 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:27.437 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.415085+0000", 2024-05-15T13:00:27.437 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:27.438 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.438 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.438 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:27.438 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.415086+0000", 2024-05-15T13:00:27.438 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.37e-07 2024-05-15T13:00:27.438 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.438 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.438 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:27.438 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.415089+0000", 2024-05-15T13:00:27.438 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0680000000000001e-06 2024-05-15T13:00:27.438 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.438 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.438 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:27.438 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.415096+0000", 2024-05-15T13:00:27.439 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2740000000000002e-06 2024-05-15T13:00:27.439 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.439 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.439 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:27.439 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.415103+0000", 2024-05-15T13:00:27.439 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2910000000000004e-06 2024-05-15T13:00:27.439 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.439 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.439 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:00:27.439 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.426255+0000", 2024-05-15T13:00:27.439 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.01115211 2024-05-15T13:00:27.439 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.439 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.439 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:00:27.439 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.426393+0000", 2024-05-15T13:00:27.440 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000138529 2024-05-15T13:00:27.440 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.440 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.440 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,11", 2024-05-15T13:00:27.440 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.426477+0000", 2024-05-15T13:00:27.440 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3566999999999995e-05 2024-05-15T13:00:27.440 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.440 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.440 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:00:27.440 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.432451+0000", 2024-05-15T13:00:27.440 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0059738020000000003 2024-05-15T13:00:27.440 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:27.440 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:27.441 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:27.441 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.441 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.441 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.25180.0:150503 4.b e364/317 ondisk, result = 0)", 2024-05-15T13:00:27.441 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:27.434844+0000", 2024-05-15T13:00:27.441 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 3.7240000000000003e-05, 2024-05-15T13:00:27.441 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00031004999999999997, 2024-05-15T13:00:27.441 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:27.441 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:00:27.441 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:27.441 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.441 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:27.441 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.434844+0000", 2024-05-15T13:00:27.441 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:27.442 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.442 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.442 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:27.442 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.434844+0000", 2024-05-15T13:00:27.442 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:27.442 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.442 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.442 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:27.442 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.434845+0000", 2024-05-15T13:00:27.442 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.02e-06 2024-05-15T13:00:27.442 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.442 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.442 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:27.442 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.434852+0000", 2024-05-15T13:00:27.443 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3039999999999999e-06 2024-05-15T13:00:27.443 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.443 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.443 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:27.443 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.434866+0000", 2024-05-15T13:00:27.443 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3395e-05 2024-05-15T13:00:27.443 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.443 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.443 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:27.443 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.434875+0000", 2024-05-15T13:00:27.443 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5119999999999997e-06 2024-05-15T13:00:27.443 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.443 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.443 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:00:27.444 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.434932+0000", 2024-05-15T13:00:27.444 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6505999999999997e-05 2024-05-15T13:00:27.444 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.444 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.444 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:00:27.444 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.434941+0000", 2024-05-15T13:00:27.444 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1819999999999995e-06 2024-05-15T13:00:27.444 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:27.444 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:27.444 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:27.444 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.444 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.444 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.25180.0:150509 4.3 e364/343)", 2024-05-15T13:00:27.444 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:27.434909+0000", 2024-05-15T13:00:27.445 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4294967295.9999723, 2024-05-15T13:00:27.445 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00027869499999999997, 2024-05-15T13:00:27.445 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:27.445 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "queued for pg", 2024-05-15T13:00:27.445 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:27.445 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.445 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:27.445 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.434909+0000", 2024-05-15T13:00:27.445 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:27.445 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.445 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.445 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:27.445 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.434909+0000", 2024-05-15T13:00:27.445 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:27.445 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.446 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.446 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:27.446 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.434909+0000", 2024-05-15T13:00:27.446 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4400000000000001e-07 2024-05-15T13:00:27.446 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.446 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.446 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:27.446 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.434913+0000", 2024-05-15T13:00:27.446 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7290000000000002e-06 2024-05-15T13:00:27.446 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.446 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.446 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:27.446 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.434921+0000", 2024-05-15T13:00:27.446 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5700000000000004e-06 2024-05-15T13:00:27.447 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.447 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.447 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:27.447 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.434926+0000", 2024-05-15T13:00:27.447 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7760000000000003e-06 2024-05-15T13:00:27.447 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:27.447 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:27.447 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:27.447 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.447 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.447 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPushReply(4.3 364/343 [PushReplyOp(4:c93ba5d1:::benchmark_data_smithi177_99756_object43247:head)])", 2024-05-15T13:00:27.447 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:27.434946+0000", 2024-05-15T13:00:27.447 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4294967295.9999352, 2024-05-15T13:00:27.447 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00028270699999999999, 2024-05-15T13:00:27.448 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:27.448 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "queued for pg", 2024-05-15T13:00:27.448 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:27.448 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.448 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:27.448 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.434946+0000", 2024-05-15T13:00:27.448 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:27.448 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.448 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.448 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:27.448 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.434946+0000", 2024-05-15T13:00:27.448 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:27.448 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.448 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.448 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:27.449 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.434946+0000", 2024-05-15T13:00:27.449 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.03e-07 2024-05-15T13:00:27.449 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.449 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.449 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:27.449 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.434951+0000", 2024-05-15T13:00:27.449 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8539999999999996e-06 2024-05-15T13:00:27.449 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.449 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.449 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:27.449 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.434960+0000", 2024-05-15T13:00:27.449 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1770000000000004e-06 2024-05-15T13:00:27.449 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:27.449 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:27.449 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:27.450 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:27.434968+0000", 2024-05-15T13:00:27.450 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8760000000000005e-06 2024-05-15T13:00:27.450 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:27.450 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:27.450 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:27.450 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:27.450 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:00:27.450 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 6 2024-05-15T13:00:27.450 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:00:27.561 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:00:27.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:27.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:00:27.670 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:00:27.703 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:27.731 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:00:27.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:27.737 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:00:27.838 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:00:27.908 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:28.259 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:28 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/364506496' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:00:28.260 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:28 smithi027 ceph-mon[39795]: pgmap v988: 17 pgs: 1 peering, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+clean; 5.5 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 630 op/s; 19092/272526 objects degraded (7.006%); 17447/272526 objects misplaced (6.402%); 8.2 MiB/s, 134 objects/s recovering 2024-05-15T13:00:28.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:28 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/364506496' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:00:28.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:28 smithi177 ceph-mon[44303]: pgmap v988: 17 pgs: 1 peering, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+clean; 5.5 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 630 op/s; 19092/272526 objects degraded (7.006%); 17447/272526 objects misplaced (6.402%); 8.2 MiB/s, 134 objects/s recovering 2024-05-15T13:00:28.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:28 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/364506496' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:00:28.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:28 smithi120 ceph-mon[43961]: pgmap v988: 17 pgs: 1 peering, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+clean; 5.5 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 630 op/s; 19092/272526 objects degraded (7.006%); 17447/272526 objects misplaced (6.402%); 8.2 MiB/s, 134 objects/s recovering 2024-05-15T13:00:28.909 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:00:28.910 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:00:28.910 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:00:28.910 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:00:28.910 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:00:28.910 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:00:28.910 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:00:28.910 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:00:28.910 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:00:28.910 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:00:28.910 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:00:28.911 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:00:28.911 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:00:28.911 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:00:28.911 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:00:28.911 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:00:28.911 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:00:28.911 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:00:28.911 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:00:28.911 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:00:28.911 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:00:28.911 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:00:28.912 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:00:28.912 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:00:29.382 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:00:29.387 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:29 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 19092/272526 objects degraded (7.006%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:29.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:29.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:00:29.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:29 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 19092/272526 objects degraded (7.006%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:29.490 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:00:29.525 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:29 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 19092/272526 objects degraded (7.006%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:29.527 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:29.570 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:00:29.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:29.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:00:29.676 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:00:29.713 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:29.737 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:00:29.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:29.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:00:29.845 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:00:29.877 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:29.899 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:00:29.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:29.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:00:30.004 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:00:30.049 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:30.063 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:00:30.090 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:00:30.092 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:30 smithi027 ceph-mon[39795]: pgmap v989: 17 pgs: 1 active+recovery_wait+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+clean; 5.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 703 op/s; 19089/267441 objects degraded (7.138%); 16931/267441 objects misplaced (6.331%); 9.1 MiB/s, 148 objects/s recovering 2024-05-15T13:00:30.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:30.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:00:30.195 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:00:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:30.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:30 smithi120 ceph-mon[43961]: pgmap v989: 17 pgs: 1 active+recovery_wait+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+clean; 5.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 703 op/s; 19089/267441 objects degraded (7.138%); 16931/267441 objects misplaced (6.331%); 9.1 MiB/s, 148 objects/s recovering 2024-05-15T13:00:30.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:30 smithi177 ceph-mon[44303]: pgmap v989: 17 pgs: 1 active+recovery_wait+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 9 active+clean; 5.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 703 op/s; 19089/267441 objects degraded (7.138%); 16931/267441 objects misplaced (6.331%); 9.1 MiB/s, 148 objects/s recovering 2024-05-15T13:00:30.660 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:00:30.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:30.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:00:30.717 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-05-15T13:00:30.774 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:00:30.866 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:30.915 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:00:30.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:30.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:00:31.022 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:00:31.075 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:31.111 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:00:31.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:31.116 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:00:31.217 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:00:31.270 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:31.271 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:00:31.272 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:00:31.272 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-05-15T13:00:31.272 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd dump -f json-pretty 2024-05-15T13:00:31.444 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:00:31.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:31.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:00:31.552 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:00:31.588 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:31.613 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:00:31.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:31.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:00:31.717 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:00:31.783 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:31.812 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:00:31.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:31.818 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:00:31.876 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:00:31.918 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:00:31.949 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:31.971 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:00:31.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:31.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:00:32.063 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:00:32.076 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:00:32.109 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:00:32.160 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:32.331 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:00:32.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:32.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:00:32.444 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:00:32.482 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:32.507 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:00:32.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:32.514 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:00:32.616 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:00:32.647 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:32.676 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:00:32.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:32.682 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:00:32.783 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:00:32.841 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:32.876 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:00:32.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:32.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:00:32.982 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:00:33.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:32 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:00:33.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:32 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:00:33.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:32 smithi027 ceph-mon[39795]: pgmap v990: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 682 op/s; 19089/263955 objects degraded (7.232%); 16756/263955 objects misplaced (6.348%); 8.3 MiB/s, 134 objects/s recovering 2024-05-15T13:00:33.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:32 smithi027 ceph-mon[39795]: osdmap e365: 12 total, 12 up, 11 in 2024-05-15T13:00:33.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:32 smithi027 ceph-mon[39795]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-05-15T13:00:33.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:32 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:00:33.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:32 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:00:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:33.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:32 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:00:33.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:32 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:00:33.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:32 smithi177 ceph-mon[44303]: pgmap v990: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 682 op/s; 19089/263955 objects degraded (7.232%); 16756/263955 objects misplaced (6.348%); 8.3 MiB/s, 134 objects/s recovering 2024-05-15T13:00:33.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:32 smithi177 ceph-mon[44303]: osdmap e365: 12 total, 12 up, 11 in 2024-05-15T13:00:33.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:32 smithi177 ceph-mon[44303]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-05-15T13:00:33.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:32 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:00:33.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:32 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:00:33.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:32 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:00:33.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:32 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:00:33.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:32 smithi120 ceph-mon[43961]: pgmap v990: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 682 op/s; 19089/263955 objects degraded (7.232%); 16756/263955 objects misplaced (6.348%); 8.3 MiB/s, 134 objects/s recovering 2024-05-15T13:00:33.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:32 smithi120 ceph-mon[43961]: osdmap e365: 12 total, 12 up, 11 in 2024-05-15T13:00:33.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:32 smithi120 ceph-mon[43961]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-05-15T13:00:33.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:32 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:00:33.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:32 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:00:33.951 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:00:34.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:33 smithi177 ceph-mon[44303]: osdmap e366: 12 total, 12 up, 11 in 2024-05-15T13:00:34.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:33 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2090038077' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:00:34.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:33 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:00:34.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:33 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 19089/263955 objects degraded (7.232%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:34.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:33 smithi027 ceph-mon[39795]: osdmap e366: 12 total, 12 up, 11 in 2024-05-15T13:00:34.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:33 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2090038077' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:00:34.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:33 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:00:34.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:33 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 19089/263955 objects degraded (7.232%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:34.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:33 smithi120 ceph-mon[43961]: osdmap e366: 12 total, 12 up, 11 in 2024-05-15T13:00:34.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:33 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2090038077' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:00:34.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:33 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:00:34.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:33 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 19089/263955 objects degraded (7.232%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:35.063 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:00:35.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:35.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:00:35.167 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:00:35.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:34 smithi177 ceph-mon[44303]: pgmap v993: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 513 op/s; 19086/261618 objects degraded (7.295%); 16443/261618 objects misplaced (6.285%); 8.0 MiB/s, 128 objects/s recovering 2024-05-15T13:00:35.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:34 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:00:35.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:34 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:00:35.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:34 smithi177 ceph-mon[44303]: osdmap e367: 12 total, 12 up, 11 in 2024-05-15T13:00:35.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:34 smithi027 ceph-mon[39795]: pgmap v993: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 513 op/s; 19086/261618 objects degraded (7.295%); 16443/261618 objects misplaced (6.285%); 8.0 MiB/s, 128 objects/s recovering 2024-05-15T13:00:35.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:34 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:00:35.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:34 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:00:35.199 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:34 smithi027 ceph-mon[39795]: osdmap e367: 12 total, 12 up, 11 in 2024-05-15T13:00:35.215 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:35.238 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:00:35.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:35.243 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:00:35.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:34 smithi120 ceph-mon[43961]: pgmap v993: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 513 op/s; 19086/261618 objects degraded (7.295%); 16443/261618 objects misplaced (6.285%); 8.0 MiB/s, 128 objects/s recovering 2024-05-15T13:00:35.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:34 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:00:35.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:34 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:00:35.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:34 smithi120 ceph-mon[43961]: osdmap e367: 12 total, 12 up, 11 in 2024-05-15T13:00:35.343 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:00:35.378 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:35.406 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:00:35.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:35.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:00:35.510 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:00:35.542 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:35.573 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:00:35.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:35.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:00:35.677 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:00:35.715 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:35.746 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:00:35.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:35.757 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:00:35.858 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:00:35.899 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:35.950 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:00:35.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:35.953 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:00:35.984 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:35 smithi177 ceph-mon[44303]: pgmap v995: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 536 op/s; 19086/257778 objects degraded (7.404%); 16230/257778 objects misplaced (6.296%); 7.3 MiB/s, 117 objects/s recovering 2024-05-15T13:00:36.054 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:00:36.127 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:36.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:35 smithi027 ceph-mon[39795]: pgmap v995: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 536 op/s; 19086/257778 objects degraded (7.404%); 16230/257778 objects misplaced (6.296%); 7.3 MiB/s, 117 objects/s recovering 2024-05-15T13:00:36.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:35 smithi120 ceph-mon[43961]: pgmap v995: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 536 op/s; 19086/257778 objects degraded (7.404%); 16230/257778 objects misplaced (6.296%); 7.3 MiB/s, 117 objects/s recovering 2024-05-15T13:00:37.909 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:00:37.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:37.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:00:38.022 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:00:38.064 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:38.957 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:00:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:38.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:00:39.066 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:00:39.114 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:39.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:38 smithi177 ceph-mon[44303]: pgmap v996: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 533 op/s; 19083/254343 objects degraded (7.503%); 16070/254343 objects misplaced (6.318%); 7.1 MiB/s, 115 objects/s recovering 2024-05-15T13:00:39.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:38 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 19083/254343 objects degraded (7.503%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:39.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:38 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:00:39.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:38 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:00:39.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:38 smithi027 ceph-mon[39795]: pgmap v996: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 533 op/s; 19083/254343 objects degraded (7.503%); 16070/254343 objects misplaced (6.318%); 7.1 MiB/s, 115 objects/s recovering 2024-05-15T13:00:39.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:38 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 19083/254343 objects degraded (7.503%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:39.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:38 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:00:39.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:38 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:00:39.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:38 smithi120 ceph-mon[43961]: pgmap v996: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 533 op/s; 19083/254343 objects degraded (7.503%); 16070/254343 objects misplaced (6.318%); 7.1 MiB/s, 115 objects/s recovering 2024-05-15T13:00:39.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:38 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 19083/254343 objects degraded (7.503%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:39.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:38 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:00:39.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:38 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:00:39.324 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:00:39.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:39.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:00:39.435 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:00:39.471 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:40.010 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:00:40.080 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:00:40.081 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:00:40.328 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:00:40.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:40.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:00:40.435 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:00:40.480 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:40 smithi177 ceph-mon[44303]: osdmap e368: 12 total, 12 up, 11 in 2024-05-15T13:00:40.480 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:40 smithi177 ceph-mon[44303]: pgmap v998: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 704 op/s; 19082/248637 objects degraded (7.675%); 15552/248637 objects misplaced (6.255%); 11 MiB/s, 174 objects/s recovering 2024-05-15T13:00:40.483 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:40.488 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:40 smithi027 ceph-mon[39795]: osdmap e368: 12 total, 12 up, 11 in 2024-05-15T13:00:40.489 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:40 smithi027 ceph-mon[39795]: pgmap v998: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 704 op/s; 19082/248637 objects degraded (7.675%); 15552/248637 objects misplaced (6.255%); 11 MiB/s, 174 objects/s recovering 2024-05-15T13:00:40.511 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:00:40.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:40.515 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:00:40.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:40 smithi120 ceph-mon[43961]: osdmap e368: 12 total, 12 up, 11 in 2024-05-15T13:00:40.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:40 smithi120 ceph-mon[43961]: pgmap v998: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 704 op/s; 19082/248637 objects degraded (7.675%); 15552/248637 objects misplaced (6.255%); 11 MiB/s, 174 objects/s recovering 2024-05-15T13:00:40.615 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:00:40.689 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:40.967 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:00:41.091 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:00:41.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:41.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:00:41.199 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:00:41.276 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:42.024 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:00:42.024 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:00:42.024 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 368, 2024-05-15T13:00:42.025 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "deaea04e-12b7-11ef-bc98-c7b262605968", 2024-05-15T13:00:42.025 INFO:teuthology.orchestra.run.smithi027.stdout: "created": "2024-05-15T12:38:21.244556+0000", 2024-05-15T13:00:42.025 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-05-15T13:00:39.222468+0000", 2024-05-15T13:00:42.025 INFO:teuthology.orchestra.run.smithi027.stdout: "last_up_change": "2024-05-15T12:45:50.876090+0000", 2024-05-15T13:00:42.025 INFO:teuthology.orchestra.run.smithi027.stdout: "last_in_change": "2024-05-15T12:59:10.294721+0000", 2024-05-15T13:00:42.025 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-15T13:00:42.025 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_num": 5801984, 2024-05-15T13:00:42.025 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_set": [ 2024-05-15T13:00:42.025 INFO:teuthology.orchestra.run.smithi027.stdout: "noscrub", 2024-05-15T13:00:42.025 INFO:teuthology.orchestra.run.smithi027.stdout: "pglog_hardlimit", 2024-05-15T13:00:42.025 INFO:teuthology.orchestra.run.smithi027.stdout: "purged_snapdirs", 2024-05-15T13:00:42.025 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_deletes", 2024-05-15T13:00:42.025 INFO:teuthology.orchestra.run.smithi027.stdout: "sortbitwise" 2024-05-15T13:00:42.026 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:00:42.026 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_version": 26, 2024-05-15T13:00:42.026 INFO:teuthology.orchestra.run.smithi027.stdout: "full_ratio": 0.94999998807907104, 2024-05-15T13:00:42.026 INFO:teuthology.orchestra.run.smithi027.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-05-15T13:00:42.026 INFO:teuthology.orchestra.run.smithi027.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-15T13:00:42.026 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_snapshot": "", 2024-05-15T13:00:42.026 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_max": 4, 2024-05-15T13:00:42.026 INFO:teuthology.orchestra.run.smithi027.stdout: "max_osd": 12, 2024-05-15T13:00:42.026 INFO:teuthology.orchestra.run.smithi027.stdout: "require_min_compat_client": "luminous", 2024-05-15T13:00:42.027 INFO:teuthology.orchestra.run.smithi027.stdout: "min_compat_client": "luminous", 2024-05-15T13:00:42.027 INFO:teuthology.orchestra.run.smithi027.stdout: "require_osd_release": "squid", 2024-05-15T13:00:42.027 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_crimson": false, 2024-05-15T13:00:42.027 INFO:teuthology.orchestra.run.smithi027.stdout: "pools": [ 2024-05-15T13:00:42.027 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.027 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 1, 2024-05-15T13:00:42.027 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": ".mgr", 2024-05-15T13:00:42.027 INFO:teuthology.orchestra.run.smithi027.stdout: "create_time": "2024-05-15T12:41:57.948613+0000", 2024-05-15T13:00:42.027 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-15T13:00:42.027 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_names": "hashpspool", 2024-05-15T13:00:42.027 INFO:teuthology.orchestra.run.smithi027.stdout: "type": 1, 2024-05-15T13:00:42.027 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 3, 2024-05-15T13:00:42.027 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": 2, 2024-05-15T13:00:42.027 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_rule": 0, 2024-05-15T13:00:42.028 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_count": 0, 2024-05-15T13:00:42.028 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_target": 0, 2024-05-15T13:00:42.028 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T13:00:42.028 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T13:00:42.028 INFO:teuthology.orchestra.run.smithi027.stdout: "object_hash": 2, 2024-05-15T13:00:42.028 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscale_mode": "off", 2024-05-15T13:00:42.028 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": 1, 2024-05-15T13:00:42.028 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num": 1, 2024-05-15T13:00:42.028 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num_target": 1, 2024-05-15T13:00:42.028 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_target": 1, 2024-05-15T13:00:42.028 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_pending": 1, 2024-05-15T13:00:42.028 INFO:teuthology.orchestra.run.smithi027.stdout: "last_pg_merge_meta": { 2024-05-15T13:00:42.028 INFO:teuthology.orchestra.run.smithi027.stdout: "source_pgid": "0.0", 2024-05-15T13:00:42.028 INFO:teuthology.orchestra.run.smithi027.stdout: "ready_epoch": 0, 2024-05-15T13:00:42.029 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_started": 0, 2024-05-15T13:00:42.029 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_clean": 0, 2024-05-15T13:00:42.029 INFO:teuthology.orchestra.run.smithi027.stdout: "source_version": "0'0", 2024-05-15T13:00:42.029 INFO:teuthology.orchestra.run.smithi027.stdout: "target_version": "0'0" 2024-05-15T13:00:42.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.029 INFO:teuthology.orchestra.run.smithi027.stdout: "last_change": "21", 2024-05-15T13:00:42.029 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend": "0", 2024-05-15T13:00:42.029 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T13:00:42.029 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T13:00:42.029 INFO:teuthology.orchestra.run.smithi027.stdout: "auid": 0, 2024-05-15T13:00:42.029 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_mode": "selfmanaged", 2024-05-15T13:00:42.029 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_seq": 0, 2024-05-15T13:00:42.029 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_epoch": 0, 2024-05-15T13:00:42.029 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_snaps": [], 2024-05-15T13:00:42.030 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps": "[]", 2024-05-15T13:00:42.030 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_bytes": 0, 2024-05-15T13:00:42.030 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_objects": 0, 2024-05-15T13:00:42.031 INFO:teuthology.orchestra.run.smithi027.stdout: "tiers": [], 2024-05-15T13:00:42.031 INFO:teuthology.orchestra.run.smithi027.stdout: "tier_of": -1, 2024-05-15T13:00:42.031 INFO:teuthology.orchestra.run.smithi027.stdout: "read_tier": -1, 2024-05-15T13:00:42.031 INFO:teuthology.orchestra.run.smithi027.stdout: "write_tier": -1, 2024-05-15T13:00:42.031 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_mode": "none", 2024-05-15T13:00:42.031 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_bytes": 0, 2024-05-15T13:00:42.031 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_objects": 0, 2024-05-15T13:00:42.031 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T13:00:42.031 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T13:00:42.031 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T13:00:42.031 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_flush_age": 0, 2024-05-15T13:00:42.031 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_evict_age": 0, 2024-05-15T13:00:42.032 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profile": "", 2024-05-15T13:00:42.032 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_params": { 2024-05-15T13:00:42.032 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "none" 2024-05-15T13:00:42.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.032 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_period": 0, 2024-05-15T13:00:42.032 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_count": 0, 2024-05-15T13:00:42.032 INFO:teuthology.orchestra.run.smithi027.stdout: "use_gmt_hitset": true, 2024-05-15T13:00:42.032 INFO:teuthology.orchestra.run.smithi027.stdout: "min_read_recency_for_promote": 0, 2024-05-15T13:00:42.032 INFO:teuthology.orchestra.run.smithi027.stdout: "min_write_recency_for_promote": 0, 2024-05-15T13:00:42.032 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T13:00:42.032 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_search_last_n": 0, 2024-05-15T13:00:42.032 INFO:teuthology.orchestra.run.smithi027.stdout: "grade_table": [], 2024-05-15T13:00:42.032 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_width": 0, 2024-05-15T13:00:42.032 INFO:teuthology.orchestra.run.smithi027.stdout: "expected_num_objects": 0, 2024-05-15T13:00:42.033 INFO:teuthology.orchestra.run.smithi027.stdout: "fast_read": false, 2024-05-15T13:00:42.033 INFO:teuthology.orchestra.run.smithi027.stdout: "options": { 2024-05-15T13:00:42.033 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_max": 32, 2024-05-15T13:00:42.033 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_min": 1 2024-05-15T13:00:42.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.033 INFO:teuthology.orchestra.run.smithi027.stdout: "application_metadata": { 2024-05-15T13:00:42.033 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": {} 2024-05-15T13:00:42.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.033 INFO:teuthology.orchestra.run.smithi027.stdout: "read_balance": { 2024-05-15T13:00:42.033 INFO:teuthology.orchestra.run.smithi027.stdout: "score_type": "Fair distribution", 2024-05-15T13:00:42.033 INFO:teuthology.orchestra.run.smithi027.stdout: "score_acting": 12, 2024-05-15T13:00:42.033 INFO:teuthology.orchestra.run.smithi027.stdout: "score_stable": 12, 2024-05-15T13:00:42.033 INFO:teuthology.orchestra.run.smithi027.stdout: "optimal_score": 0.25, 2024-05-15T13:00:42.033 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_acting": 3, 2024-05-15T13:00:42.034 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_stable": 3, 2024-05-15T13:00:42.034 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity_weighted": 0.94652295112609863, 2024-05-15T13:00:42.034 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity": 0.94999998807907104, 2024-05-15T13:00:42.034 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-05-15T13:00:42.034 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.034 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.034 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 4, 2024-05-15T13:00:42.035 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": "unique_pool_2", 2024-05-15T13:00:42.035 INFO:teuthology.orchestra.run.smithi027.stdout: "create_time": "2024-05-15T12:57:53.956942+0000", 2024-05-15T13:00:42.035 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-15T13:00:42.035 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_names": "hashpspool", 2024-05-15T13:00:42.035 INFO:teuthology.orchestra.run.smithi027.stdout: "type": 1, 2024-05-15T13:00:42.035 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 3, 2024-05-15T13:00:42.035 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": 2, 2024-05-15T13:00:42.035 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_rule": 0, 2024-05-15T13:00:42.035 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_count": 0, 2024-05-15T13:00:42.035 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_target": 0, 2024-05-15T13:00:42.035 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T13:00:42.035 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T13:00:42.035 INFO:teuthology.orchestra.run.smithi027.stdout: "object_hash": 2, 2024-05-15T13:00:42.036 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscale_mode": "off", 2024-05-15T13:00:42.036 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": 16, 2024-05-15T13:00:42.036 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num": 16, 2024-05-15T13:00:42.036 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num_target": 16, 2024-05-15T13:00:42.036 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_target": 16, 2024-05-15T13:00:42.036 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_pending": 16, 2024-05-15T13:00:42.036 INFO:teuthology.orchestra.run.smithi027.stdout: "last_pg_merge_meta": { 2024-05-15T13:00:42.036 INFO:teuthology.orchestra.run.smithi027.stdout: "source_pgid": "0.0", 2024-05-15T13:00:42.036 INFO:teuthology.orchestra.run.smithi027.stdout: "ready_epoch": 0, 2024-05-15T13:00:42.036 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_started": 0, 2024-05-15T13:00:42.036 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_clean": 0, 2024-05-15T13:00:42.036 INFO:teuthology.orchestra.run.smithi027.stdout: "source_version": "0'0", 2024-05-15T13:00:42.037 INFO:teuthology.orchestra.run.smithi027.stdout: "target_version": "0'0" 2024-05-15T13:00:42.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.037 INFO:teuthology.orchestra.run.smithi027.stdout: "last_change": "321", 2024-05-15T13:00:42.037 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend": "0", 2024-05-15T13:00:42.037 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T13:00:42.037 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T13:00:42.037 INFO:teuthology.orchestra.run.smithi027.stdout: "auid": 0, 2024-05-15T13:00:42.037 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_mode": "selfmanaged", 2024-05-15T13:00:42.037 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_seq": 0, 2024-05-15T13:00:42.037 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_epoch": 0, 2024-05-15T13:00:42.037 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_snaps": [], 2024-05-15T13:00:42.037 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps": "[]", 2024-05-15T13:00:42.038 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_bytes": 0, 2024-05-15T13:00:42.038 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_objects": 0, 2024-05-15T13:00:42.038 INFO:teuthology.orchestra.run.smithi027.stdout: "tiers": [], 2024-05-15T13:00:42.038 INFO:teuthology.orchestra.run.smithi027.stdout: "tier_of": -1, 2024-05-15T13:00:42.038 INFO:teuthology.orchestra.run.smithi027.stdout: "read_tier": -1, 2024-05-15T13:00:42.038 INFO:teuthology.orchestra.run.smithi027.stdout: "write_tier": -1, 2024-05-15T13:00:42.038 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_mode": "none", 2024-05-15T13:00:42.038 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_bytes": 0, 2024-05-15T13:00:42.038 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_objects": 0, 2024-05-15T13:00:42.038 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T13:00:42.038 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T13:00:42.038 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T13:00:42.038 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_flush_age": 0, 2024-05-15T13:00:42.039 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_evict_age": 0, 2024-05-15T13:00:42.039 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profile": "", 2024-05-15T13:00:42.039 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_params": { 2024-05-15T13:00:42.039 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "none" 2024-05-15T13:00:42.039 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.039 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_period": 0, 2024-05-15T13:00:42.039 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_count": 0, 2024-05-15T13:00:42.039 INFO:teuthology.orchestra.run.smithi027.stdout: "use_gmt_hitset": true, 2024-05-15T13:00:42.039 INFO:teuthology.orchestra.run.smithi027.stdout: "min_read_recency_for_promote": 0, 2024-05-15T13:00:42.039 INFO:teuthology.orchestra.run.smithi027.stdout: "min_write_recency_for_promote": 0, 2024-05-15T13:00:42.039 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T13:00:42.039 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_search_last_n": 0, 2024-05-15T13:00:42.039 INFO:teuthology.orchestra.run.smithi027.stdout: "grade_table": [], 2024-05-15T13:00:42.039 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_width": 0, 2024-05-15T13:00:42.039 INFO:teuthology.orchestra.run.smithi027.stdout: "expected_num_objects": 0, 2024-05-15T13:00:42.040 INFO:teuthology.orchestra.run.smithi027.stdout: "fast_read": false, 2024-05-15T13:00:42.040 INFO:teuthology.orchestra.run.smithi027.stdout: "options": {}, 2024-05-15T13:00:42.040 INFO:teuthology.orchestra.run.smithi027.stdout: "application_metadata": { 2024-05-15T13:00:42.040 INFO:teuthology.orchestra.run.smithi027.stdout: "rados": {} 2024-05-15T13:00:42.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.040 INFO:teuthology.orchestra.run.smithi027.stdout: "read_balance": { 2024-05-15T13:00:42.040 INFO:teuthology.orchestra.run.smithi027.stdout: "score_type": "Fair distribution", 2024-05-15T13:00:42.040 INFO:teuthology.orchestra.run.smithi027.stdout: "score_acting": 2.9900000095367432, 2024-05-15T13:00:42.040 INFO:teuthology.orchestra.run.smithi027.stdout: "score_stable": 2.9900000095367432, 2024-05-15T13:00:42.040 INFO:teuthology.orchestra.run.smithi027.stdout: "optimal_score": 0.92000001668930054, 2024-05-15T13:00:42.040 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_acting": 2.75, 2024-05-15T13:00:42.041 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_stable": 2.75, 2024-05-15T13:00:42.041 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity_weighted": 0.94652295112609863, 2024-05-15T13:00:42.041 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity": 0.94999998807907104, 2024-05-15T13:00:42.041 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-05-15T13:00:42.041 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.041 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.041 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:00:42.041 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T13:00:42.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.041 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 0, 2024-05-15T13:00:42.041 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "64fb3ad2-08a7-4106-a07a-6c776c877faf", 2024-05-15T13:00:42.041 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:00:42.041 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:00:42.041 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:00:42.042 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:00:42.042 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:00:42.042 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:00:42.042 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 8, 2024-05-15T13:00:42.042 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 343, 2024-05-15T13:00:42.042 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:00:42.042 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:00:42.042 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:00:42.042 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.042 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.042 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.042 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6802", 2024-05-15T13:00:42.042 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:00:42.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.043 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.043 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.043 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6803", 2024-05-15T13:00:42.043 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:00:42.043 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.043 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.043 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:00:42.043 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.043 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.043 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.043 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6804", 2024-05-15T13:00:42.043 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:00:42.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.044 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.044 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.044 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6805", 2024-05-15T13:00:42.044 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:00:42.044 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.044 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.044 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.045 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:00:42.045 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.045 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.045 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6808", 2024-05-15T13:00:42.045 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:00:42.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.046 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.046 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.046 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6809", 2024-05-15T13:00:42.046 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:00:42.046 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.047 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.047 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:00:42.047 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.047 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.047 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.047 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6806", 2024-05-15T13:00:42.048 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:00:42.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.048 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.048 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.048 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6807", 2024-05-15T13:00:42.048 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:00:42.049 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.049 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.049 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6803/4239116935", 2024-05-15T13:00:42.049 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6805/4239116935", 2024-05-15T13:00:42.050 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6809/4239116935", 2024-05-15T13:00:42.050 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6807/4239116935", 2024-05-15T13:00:42.050 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:00:42.050 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:00:42.050 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:00:42.050 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.051 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 1, 2024-05-15T13:00:42.051 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "9c16dd90-9d51-48b8-a296-1c8879ba1bc8", 2024-05-15T13:00:42.051 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:00:42.051 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:00:42.051 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:00:42.052 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:00:42.052 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:00:42.052 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:00:42.052 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 13, 2024-05-15T13:00:42.052 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 363, 2024-05-15T13:00:42.052 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:00:42.053 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:00:42.053 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:00:42.053 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.053 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.053 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6810", 2024-05-15T13:00:42.054 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:00:42.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.054 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.054 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.054 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6811", 2024-05-15T13:00:42.054 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:00:42.055 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.055 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.055 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:00:42.055 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.055 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.055 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6812", 2024-05-15T13:00:42.056 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:00:42.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.056 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.056 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6813", 2024-05-15T13:00:42.056 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:00:42.056 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.057 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.057 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:00:42.057 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.057 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.058 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6816", 2024-05-15T13:00:42.058 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:00:42.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.058 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.058 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.058 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6817", 2024-05-15T13:00:42.059 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:00:42.059 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.059 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.059 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:00:42.059 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.060 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.060 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6814", 2024-05-15T13:00:42.060 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:00:42.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.061 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.061 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6815", 2024-05-15T13:00:42.061 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:00:42.061 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.061 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.062 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6811/1466270637", 2024-05-15T13:00:42.062 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6813/1466270637", 2024-05-15T13:00:42.062 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6817/1466270637", 2024-05-15T13:00:42.062 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6815/1466270637", 2024-05-15T13:00:42.062 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:00:42.062 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:00:42.063 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:00:42.063 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.063 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 2, 2024-05-15T13:00:42.064 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "cbe3ba13-3ea3-4891-8d2d-6b85581a732c", 2024-05-15T13:00:42.064 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:00:42.064 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:00:42.064 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:00:42.064 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:00:42.064 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:00:42.064 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:00:42.065 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 18, 2024-05-15T13:00:42.065 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 110, 2024-05-15T13:00:42.065 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:00:42.065 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:00:42.065 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:00:42.066 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.066 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.066 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.066 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6818", 2024-05-15T13:00:42.066 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:00:42.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.066 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.067 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.067 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6819", 2024-05-15T13:00:42.067 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:00:42.067 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.067 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.068 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:00:42.068 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.068 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.068 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.068 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6820", 2024-05-15T13:00:42.068 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:00:42.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.069 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.069 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6821", 2024-05-15T13:00:42.069 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:00:42.069 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.069 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.070 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:00:42.070 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.070 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.070 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.070 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6824", 2024-05-15T13:00:42.070 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:00:42.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.070 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.071 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.071 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6825", 2024-05-15T13:00:42.071 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:00:42.071 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.071 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.071 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.071 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:00:42.071 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.071 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.072 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.072 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6822", 2024-05-15T13:00:42.072 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:00:42.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.072 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.072 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.072 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6823", 2024-05-15T13:00:42.072 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:00:42.073 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.073 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.073 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.073 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6819/3285413897", 2024-05-15T13:00:42.073 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6821/3285413897", 2024-05-15T13:00:42.073 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6825/3285413897", 2024-05-15T13:00:42.073 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6823/3285413897", 2024-05-15T13:00:42.074 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:00:42.074 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:00:42.074 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:00:42.074 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.074 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.074 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.074 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 3, 2024-05-15T13:00:42.074 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "2e83735b-b944-4f87-8080-b994d4b96897", 2024-05-15T13:00:42.074 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:00:42.074 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:00:42.074 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:00:42.075 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:00:42.075 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:00:42.075 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:00:42.075 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 26, 2024-05-15T13:00:42.075 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 317, 2024-05-15T13:00:42.075 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:00:42.075 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:00:42.075 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:00:42.075 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.075 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.075 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.076 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6826", 2024-05-15T13:00:42.076 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:00:42.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.076 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.076 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.076 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6827", 2024-05-15T13:00:42.076 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:00:42.076 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.076 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.076 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:00:42.076 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.076 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.076 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.077 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6828", 2024-05-15T13:00:42.077 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:00:42.077 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.077 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.077 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.077 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6829", 2024-05-15T13:00:42.077 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:00:42.077 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.077 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.077 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.077 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:00:42.077 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.077 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.078 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.078 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6832", 2024-05-15T13:00:42.078 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:00:42.078 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.078 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.078 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.078 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6833", 2024-05-15T13:00:42.078 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:00:42.078 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.078 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.078 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.078 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:00:42.078 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.079 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.079 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.079 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6830", 2024-05-15T13:00:42.079 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:00:42.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.079 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.079 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.079 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6831", 2024-05-15T13:00:42.079 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:00:42.079 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.079 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.079 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6827/3008975244", 2024-05-15T13:00:42.079 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6829/3008975244", 2024-05-15T13:00:42.080 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6833/3008975244", 2024-05-15T13:00:42.080 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6831/3008975244", 2024-05-15T13:00:42.080 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:00:42.080 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:00:42.080 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:00:42.080 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.080 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.080 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.080 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 4, 2024-05-15T13:00:42.080 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "7c39c1dd-3867-46f7-8372-7688fa13b16a", 2024-05-15T13:00:42.080 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:00:42.080 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:00:42.080 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:00:42.080 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:00:42.081 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:00:42.081 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:00:42.081 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 31, 2024-05-15T13:00:42.081 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 343, 2024-05-15T13:00:42.081 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:00:42.081 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:00:42.081 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:00:42.081 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.081 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.081 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.081 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6800", 2024-05-15T13:00:42.081 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:00:42.081 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.081 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.082 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.082 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6801", 2024-05-15T13:00:42.082 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:00:42.082 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.082 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.082 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:00:42.082 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.082 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.082 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.082 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6802", 2024-05-15T13:00:42.082 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:00:42.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.083 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.083 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.083 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6803", 2024-05-15T13:00:42.083 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:00:42.083 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.083 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.083 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.083 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:00:42.083 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.083 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.083 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.083 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6806", 2024-05-15T13:00:42.083 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:00:42.083 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.084 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.084 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.084 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6807", 2024-05-15T13:00:42.084 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:00:42.084 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.084 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.084 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:00:42.084 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.084 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.084 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.084 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6804", 2024-05-15T13:00:42.084 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:00:42.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.085 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.085 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.085 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6805", 2024-05-15T13:00:42.085 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:00:42.085 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.085 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.085 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6801/3877842957", 2024-05-15T13:00:42.085 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6803/3877842957", 2024-05-15T13:00:42.085 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6807/3877842957", 2024-05-15T13:00:42.085 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6805/3877842957", 2024-05-15T13:00:42.085 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:00:42.085 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:00:42.085 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:00:42.086 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.086 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.086 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.086 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 5, 2024-05-15T13:00:42.086 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "a6dc220f-aec0-4454-9695-b44c0b9770e0", 2024-05-15T13:00:42.086 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:00:42.086 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:00:42.086 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:00:42.086 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:00:42.086 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:00:42.086 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:00:42.086 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 40, 2024-05-15T13:00:42.086 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 343, 2024-05-15T13:00:42.087 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:00:42.087 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:00:42.087 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:00:42.087 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.087 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.087 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6808", 2024-05-15T13:00:42.087 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:00:42.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.087 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.087 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6809", 2024-05-15T13:00:42.087 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:00:42.087 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.088 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.088 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.088 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:00:42.088 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.088 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.088 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.088 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6810", 2024-05-15T13:00:42.088 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:00:42.089 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.089 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.089 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.089 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6811", 2024-05-15T13:00:42.089 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:00:42.089 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.089 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.089 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.090 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:00:42.090 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.090 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.090 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.090 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6814", 2024-05-15T13:00:42.090 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:00:42.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.091 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.091 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6815", 2024-05-15T13:00:42.091 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:00:42.091 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.092 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.092 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:00:42.092 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.092 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.092 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6812", 2024-05-15T13:00:42.092 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:00:42.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.093 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.093 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6813", 2024-05-15T13:00:42.093 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:00:42.093 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.093 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.094 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.094 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6809/1600179183", 2024-05-15T13:00:42.094 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6811/1600179183", 2024-05-15T13:00:42.094 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6815/1600179183", 2024-05-15T13:00:42.094 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6813/1600179183", 2024-05-15T13:00:42.094 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:00:42.094 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:00:42.094 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:00:42.095 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.095 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.095 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.095 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 6, 2024-05-15T13:00:42.095 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b", 2024-05-15T13:00:42.095 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:00:42.095 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:00:42.096 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:00:42.096 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:00:42.096 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:00:42.096 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:00:42.096 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 48, 2024-05-15T13:00:42.096 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 362, 2024-05-15T13:00:42.096 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:00:42.097 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:00:42.097 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:00:42.097 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.097 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.097 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6816", 2024-05-15T13:00:42.098 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:00:42.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.098 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.098 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.098 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6817", 2024-05-15T13:00:42.098 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:00:42.099 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.099 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.099 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:00:42.099 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.099 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.099 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6818", 2024-05-15T13:00:42.099 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:00:42.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.099 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.099 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6819", 2024-05-15T13:00:42.100 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:00:42.100 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.100 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.100 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:00:42.100 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.100 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.100 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.100 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6822", 2024-05-15T13:00:42.100 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:00:42.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.100 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.100 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.100 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6823", 2024-05-15T13:00:42.101 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:00:42.101 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.101 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.101 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:00:42.101 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.101 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.101 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6820", 2024-05-15T13:00:42.101 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:00:42.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.101 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.101 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6821", 2024-05-15T13:00:42.102 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:00:42.102 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.102 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.102 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.102 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6817/3323706735", 2024-05-15T13:00:42.102 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6819/3323706735", 2024-05-15T13:00:42.102 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6823/3323706735", 2024-05-15T13:00:42.102 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6821/3323706735", 2024-05-15T13:00:42.102 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:00:42.102 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:00:42.102 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:00:42.102 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.102 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.102 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.103 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 7, 2024-05-15T13:00:42.103 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "6ef085d8-f59b-4033-8280-2c8f78693e30", 2024-05-15T13:00:42.103 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:00:42.103 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 0, 2024-05-15T13:00:42.103 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 0, 2024-05-15T13:00:42.103 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:00:42.103 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:00:42.103 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:00:42.103 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 56, 2024-05-15T13:00:42.103 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 219, 2024-05-15T13:00:42.103 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:00:42.103 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:00:42.103 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:00:42.103 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.104 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.104 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.104 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6824", 2024-05-15T13:00:42.104 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:00:42.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.104 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.104 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.104 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6825", 2024-05-15T13:00:42.104 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:00:42.104 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.104 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.104 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:00:42.104 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.105 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.105 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.105 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6826", 2024-05-15T13:00:42.105 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:00:42.105 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.105 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.105 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.105 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6827", 2024-05-15T13:00:42.105 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:00:42.105 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.105 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.105 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.105 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:00:42.105 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.105 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.106 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.106 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6830", 2024-05-15T13:00:42.106 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:00:42.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.106 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.106 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.106 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6831", 2024-05-15T13:00:42.106 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:00:42.106 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.106 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.106 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:00:42.106 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.107 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.107 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.107 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6828", 2024-05-15T13:00:42.107 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:00:42.107 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.107 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.107 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.107 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6829", 2024-05-15T13:00:42.107 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:00:42.107 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.107 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.107 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.107 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6825/2960616153", 2024-05-15T13:00:42.110 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6827/2960616153", 2024-05-15T13:00:42.111 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6831/2960616153", 2024-05-15T13:00:42.111 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6829/2960616153", 2024-05-15T13:00:42.111 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:00:42.111 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:00:42.111 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:00:42.111 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.112 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.112 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.112 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 8, 2024-05-15T13:00:42.112 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "26514b11-d02b-46e9-a4e8-3da193fcdbcc", 2024-05-15T13:00:42.112 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:00:42.112 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:00:42.112 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:00:42.112 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:00:42.112 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:00:42.112 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:00:42.112 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 65, 2024-05-15T13:00:42.112 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 292, 2024-05-15T13:00:42.113 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:00:42.113 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:00:42.113 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:00:42.113 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.113 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.113 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.113 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6800", 2024-05-15T13:00:42.113 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:00:42.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.113 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.113 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.113 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6801", 2024-05-15T13:00:42.113 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:00:42.113 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.114 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.114 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:00:42.114 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.114 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.114 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.114 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6802", 2024-05-15T13:00:42.114 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:00:42.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.114 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.114 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.114 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6803", 2024-05-15T13:00:42.114 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:00:42.115 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.115 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.115 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:00:42.115 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.115 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.115 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6806", 2024-05-15T13:00:42.115 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:00:42.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.115 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.115 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6807", 2024-05-15T13:00:42.115 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:00:42.115 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.116 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.116 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:00:42.116 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.116 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.116 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.116 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6804", 2024-05-15T13:00:42.116 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:00:42.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.116 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.116 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.116 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6805", 2024-05-15T13:00:42.116 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:00:42.116 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.117 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.117 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6801/1132459315", 2024-05-15T13:00:42.117 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6803/1132459315", 2024-05-15T13:00:42.117 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6807/1132459315", 2024-05-15T13:00:42.117 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6805/1132459315", 2024-05-15T13:00:42.117 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:00:42.117 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:00:42.117 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:00:42.117 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.117 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 9, 2024-05-15T13:00:42.117 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "6f9497bc-e962-409d-8a2c-0c634cc1124f", 2024-05-15T13:00:42.117 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:00:42.118 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:00:42.118 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:00:42.118 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 0.9062042236328125, 2024-05-15T13:00:42.118 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:00:42.118 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:00:42.118 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 76, 2024-05-15T13:00:42.118 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 317, 2024-05-15T13:00:42.118 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:00:42.118 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:00:42.118 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:00:42.118 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.118 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.118 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.118 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6808", 2024-05-15T13:00:42.119 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:00:42.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.119 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.119 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.119 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6809", 2024-05-15T13:00:42.119 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:00:42.119 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.119 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.119 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:00:42.119 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.119 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.119 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.119 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6810", 2024-05-15T13:00:42.120 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:00:42.120 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.120 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.120 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.120 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6811", 2024-05-15T13:00:42.120 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:00:42.120 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.120 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.120 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.120 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:00:42.120 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.120 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.120 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.120 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6814", 2024-05-15T13:00:42.120 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:00:42.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.121 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.121 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.121 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6815", 2024-05-15T13:00:42.121 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:00:42.121 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.121 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.121 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:00:42.121 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.121 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.121 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.121 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6812", 2024-05-15T13:00:42.121 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:00:42.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.122 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.122 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.122 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6813", 2024-05-15T13:00:42.122 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:00:42.122 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.122 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.122 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6809/2710457646", 2024-05-15T13:00:42.122 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6811/2710457646", 2024-05-15T13:00:42.122 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6815/2710457646", 2024-05-15T13:00:42.122 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6813/2710457646", 2024-05-15T13:00:42.122 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:00:42.122 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:00:42.123 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:00:42.123 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.123 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.123 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 10, 2024-05-15T13:00:42.123 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "f852876f-c7d1-4039-90c4-5ab09dbd3e32", 2024-05-15T13:00:42.123 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:00:42.123 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:00:42.123 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:00:42.123 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 0.4520721435546875, 2024-05-15T13:00:42.123 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:00:42.123 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:00:42.123 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 81, 2024-05-15T13:00:42.123 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 0, 2024-05-15T13:00:42.124 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:00:42.124 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:00:42.124 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:00:42.124 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.124 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.124 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.124 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6816", 2024-05-15T13:00:42.124 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:00:42.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.124 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.124 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.124 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6817", 2024-05-15T13:00:42.124 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:00:42.125 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.125 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.125 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:00:42.125 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.125 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.125 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6818", 2024-05-15T13:00:42.125 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:00:42.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.125 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.125 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6819", 2024-05-15T13:00:42.125 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:00:42.125 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.126 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.126 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.126 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:00:42.126 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.126 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.126 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.126 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6822", 2024-05-15T13:00:42.126 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:00:42.126 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.126 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.126 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.126 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6823", 2024-05-15T13:00:42.126 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:00:42.127 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.127 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.127 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:00:42.127 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.127 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.127 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.127 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6820", 2024-05-15T13:00:42.127 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:00:42.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.127 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.127 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.127 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6821", 2024-05-15T13:00:42.127 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:00:42.127 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.128 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.128 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.128 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6817/3236900263", 2024-05-15T13:00:42.128 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6819/3236900263", 2024-05-15T13:00:42.128 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6823/3236900263", 2024-05-15T13:00:42.128 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6821/3236900263", 2024-05-15T13:00:42.128 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:00:42.128 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:00:42.128 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:00:42.128 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.128 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.128 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.128 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 11, 2024-05-15T13:00:42.128 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "cccc53c5-8476-4d55-b160-4bdc5cf92902", 2024-05-15T13:00:42.129 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:00:42.129 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:00:42.129 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:00:42.129 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:00:42.129 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:00:42.129 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:00:42.129 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 90, 2024-05-15T13:00:42.129 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 345, 2024-05-15T13:00:42.129 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:00:42.129 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:00:42.129 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:00:42.129 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.129 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.130 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.130 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6824", 2024-05-15T13:00:42.130 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:00:42.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.130 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.130 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.130 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6825", 2024-05-15T13:00:42.130 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:00:42.130 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.130 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.130 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:00:42.130 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.130 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.131 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.131 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6826", 2024-05-15T13:00:42.131 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:00:42.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.131 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.131 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.131 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6827", 2024-05-15T13:00:42.131 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:00:42.131 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.131 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.131 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:00:42.131 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.131 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.131 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.132 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6830", 2024-05-15T13:00:42.132 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:00:42.132 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.132 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.132 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.132 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6831", 2024-05-15T13:00:42.132 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:00:42.132 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.132 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.132 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.132 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:00:42.132 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:00:42.132 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.132 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:00:42.133 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6828", 2024-05-15T13:00:42.133 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:00:42.133 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.133 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.133 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:00:42.133 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6829", 2024-05-15T13:00:42.133 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:00:42.133 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.133 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.133 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.133 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6825/681011628", 2024-05-15T13:00:42.133 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6827/681011628", 2024-05-15T13:00:42.133 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6831/681011628", 2024-05-15T13:00:42.133 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6829/681011628", 2024-05-15T13:00:42.134 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:00:42.134 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:00:42.134 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:00:42.134 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.134 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.134 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:00:42.134 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_xinfo": [ 2024-05-15T13:00:42.134 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.134 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 0, 2024-05-15T13:00:42.134 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:00:42.134 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:00:42.134 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:00:42.134 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:00:42.134 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:00:42.135 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:59:43.077075+0000", 2024-05-15T13:00:42.135 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:00:42.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.135 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 1, 2024-05-15T13:00:42.135 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:00:42.135 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:00:42.135 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:00:42.135 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:00:42.135 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:00:42.135 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:00:31.854618+0000", 2024-05-15T13:00:42.135 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:00:42.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.136 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 2, 2024-05-15T13:00:42.136 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:00:42.136 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:00:42.136 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:00:42.136 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:00:42.136 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:00:42.136 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:59:57.743220+0000", 2024-05-15T13:00:42.136 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:00:42.136 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.136 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.136 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 3, 2024-05-15T13:00:42.136 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:00:42.136 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:00:42.136 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:00:42.136 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:00:42.136 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:00:42.137 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:59:52.263164+0000", 2024-05-15T13:00:42.137 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:00:42.137 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.137 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.137 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 4, 2024-05-15T13:00:42.137 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:00:42.137 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:00:42.137 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:00:42.137 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:00:42.137 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:00:42.137 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:00:05.117774+0000", 2024-05-15T13:00:42.137 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:00:42.137 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.137 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.138 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 5, 2024-05-15T13:00:42.138 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:00:42.138 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:00:42.138 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:00:42.138 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:00:42.138 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:00:42.138 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:00:15.644018+0000", 2024-05-15T13:00:42.138 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:00:42.138 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.138 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.138 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 6, 2024-05-15T13:00:42.138 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:00:42.138 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:00:42.138 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:00:42.139 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:00:42.139 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:00:42.139 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:59:50.811831+0000", 2024-05-15T13:00:42.139 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:00:42.139 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.139 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.139 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 7, 2024-05-15T13:00:42.139 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:00:42.139 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:00:42.139 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:00:42.139 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:00:42.139 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 65536, 2024-05-15T13:00:42.139 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:59:49.579189+0000", 2024-05-15T13:00:42.139 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:00:42.140 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.140 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.140 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 8, 2024-05-15T13:00:42.140 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:00:42.140 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:00:42.140 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:00:42.140 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:00:42.140 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:00:42.140 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:00:38.448698+0000", 2024-05-15T13:00:42.140 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:00:42.140 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.140 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.140 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 9, 2024-05-15T13:00:42.140 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:00:42.141 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:00:42.141 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:00:42.141 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:00:42.141 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:00:42.141 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:00:32.401461+0000", 2024-05-15T13:00:42.141 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:00:42.141 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.141 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.141 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 10, 2024-05-15T13:00:42.141 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:00:42.141 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:00:42.141 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:00:42.141 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:00:42.141 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:00:42.142 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T12:59:39.949283+0000", 2024-05-15T13:00:42.142 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:00:42.142 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.142 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.142 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 11, 2024-05-15T13:00:42.142 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:00:42.142 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:00:42.142 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:00:42.142 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:00:42.142 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:00:42.142 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:00:10.897851+0000", 2024-05-15T13:00:42.142 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:00:42.142 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.142 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:00:42.143 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap": [ 2024-05-15T13:00:42.143 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.143 INFO:teuthology.orchestra.run.smithi027.stdout: "pgid": "1.0", 2024-05-15T13:00:42.143 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T13:00:42.143 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-05-15T13:00:42.143 INFO:teuthology.orchestra.run.smithi027.stdout: 6, 2024-05-15T13:00:42.143 INFO:teuthology.orchestra.run.smithi027.stdout: 10 2024-05-15T13:00:42.143 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.143 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.143 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:00:42.143 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap_items": [], 2024-05-15T13:00:42.143 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap_primaries": [], 2024-05-15T13:00:42.143 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_temp": [ 2024-05-15T13:00:42.143 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.144 INFO:teuthology.orchestra.run.smithi027.stdout: "pgid": "4.1", 2024-05-15T13:00:42.144 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T13:00:42.144 INFO:teuthology.orchestra.run.smithi027.stdout: 4, 2024-05-15T13:00:42.144 INFO:teuthology.orchestra.run.smithi027.stdout: 5, 2024-05-15T13:00:42.144 INFO:teuthology.orchestra.run.smithi027.stdout: 6 2024-05-15T13:00:42.144 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.144 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.144 INFO:teuthology.orchestra.run.smithi027.stdout: "pgid": "4.2", 2024-05-15T13:00:42.144 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T13:00:42.144 INFO:teuthology.orchestra.run.smithi027.stdout: 5, 2024-05-15T13:00:42.144 INFO:teuthology.orchestra.run.smithi027.stdout: 9 2024-05-15T13:00:42.144 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.144 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.145 INFO:teuthology.orchestra.run.smithi027.stdout: "pgid": "4.3", 2024-05-15T13:00:42.145 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T13:00:42.145 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-05-15T13:00:42.145 INFO:teuthology.orchestra.run.smithi027.stdout: 1 2024-05-15T13:00:42.145 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.145 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.145 INFO:teuthology.orchestra.run.smithi027.stdout: "pgid": "4.5", 2024-05-15T13:00:42.145 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T13:00:42.145 INFO:teuthology.orchestra.run.smithi027.stdout: 6, 2024-05-15T13:00:42.145 INFO:teuthology.orchestra.run.smithi027.stdout: 10 2024-05-15T13:00:42.145 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.145 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.146 INFO:teuthology.orchestra.run.smithi027.stdout: "pgid": "4.8", 2024-05-15T13:00:42.146 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T13:00:42.146 INFO:teuthology.orchestra.run.smithi027.stdout: 6, 2024-05-15T13:00:42.146 INFO:teuthology.orchestra.run.smithi027.stdout: 9 2024-05-15T13:00:42.146 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.146 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.146 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.146 INFO:teuthology.orchestra.run.smithi027.stdout: "pgid": "4.9", 2024-05-15T13:00:42.146 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T13:00:42.146 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-05-15T13:00:42.146 INFO:teuthology.orchestra.run.smithi027.stdout: 3 2024-05-15T13:00:42.146 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.146 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.146 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:42.147 INFO:teuthology.orchestra.run.smithi027.stdout: "pgid": "4.f", 2024-05-15T13:00:42.147 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T13:00:42.147 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-05-15T13:00:42.147 INFO:teuthology.orchestra.run.smithi027.stdout: 3 2024-05-15T13:00:42.147 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:42.147 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.147 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:00:42.147 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_temp": [], 2024-05-15T13:00:42.147 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklist": { 2024-05-15T13:00:42.147 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6801/2530617473": "2024-05-16T12:39:19.409195+0000", 2024-05-15T13:00:42.147 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6800/2530617473": "2024-05-16T12:39:19.409195+0000", 2024-05-15T13:00:42.147 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/1880261204": "2024-05-16T12:39:19.409195+0000", 2024-05-15T13:00:42.147 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6801/3441136471": "2024-05-16T12:38:50.931011+0000", 2024-05-15T13:00:42.147 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/4291446247": "2024-05-16T12:39:19.409195+0000", 2024-05-15T13:00:42.148 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/580128209": "2024-05-16T12:38:50.931011+0000", 2024-05-15T13:00:42.148 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6800/3441136471": "2024-05-16T12:38:50.931011+0000", 2024-05-15T13:00:42.148 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/3011229857": "2024-05-16T12:38:50.931011+0000", 2024-05-15T13:00:42.148 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/2109693913": "2024-05-16T12:38:50.931011+0000", 2024-05-15T13:00:42.148 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/2445663024": "2024-05-16T12:39:19.409195+0000" 2024-05-15T13:00:42.148 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.148 INFO:teuthology.orchestra.run.smithi027.stdout: "range_blocklist": {}, 2024-05-15T13:00:42.148 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profiles": { 2024-05-15T13:00:42.148 INFO:teuthology.orchestra.run.smithi027.stdout: "default": { 2024-05-15T13:00:42.148 INFO:teuthology.orchestra.run.smithi027.stdout: "crush-failure-domain": "osd", 2024-05-15T13:00:42.148 INFO:teuthology.orchestra.run.smithi027.stdout: "k": "2", 2024-05-15T13:00:42.148 INFO:teuthology.orchestra.run.smithi027.stdout: "m": "1", 2024-05-15T13:00:42.148 INFO:teuthology.orchestra.run.smithi027.stdout: "plugin": "jerasure", 2024-05-15T13:00:42.148 INFO:teuthology.orchestra.run.smithi027.stdout: "technique": "reed_sol_van" 2024-05-15T13:00:42.149 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:42.149 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps_queue": [], 2024-05-15T13:00:42.149 INFO:teuthology.orchestra.run.smithi027.stdout: "new_removed_snaps": [], 2024-05-15T13:00:42.149 INFO:teuthology.orchestra.run.smithi027.stdout: "new_purged_snaps": [], 2024-05-15T13:00:42.149 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_node_flags": {}, 2024-05-15T13:00:42.149 INFO:teuthology.orchestra.run.smithi027.stdout: "device_class_flags": {}, 2024-05-15T13:00:42.149 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode": { 2024-05-15T13:00:42.149 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode_enabled": false, 2024-05-15T13:00:42.149 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_bucket_count": 0, 2024-05-15T13:00:42.149 INFO:teuthology.orchestra.run.smithi027.stdout: "degraded_stretch_mode": 0, 2024-05-15T13:00:42.149 INFO:teuthology.orchestra.run.smithi027.stdout: "recovering_stretch_mode": 0, 2024-05-15T13:00:42.149 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode_bucket": 0 2024-05-15T13:00:42.149 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:42.149 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:00:43.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:42 smithi177 ceph-mon[44303]: pgmap v999: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 689 op/s; 19082/245070 objects degraded (7.786%); 15375/245070 objects misplaced (6.274%); 8.3 MiB/s, 133 objects/s recovering 2024-05-15T13:00:43.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:42 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/917151027' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T13:00:43.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:42 smithi027 ceph-mon[39795]: pgmap v999: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 689 op/s; 19082/245070 objects degraded (7.786%); 15375/245070 objects misplaced (6.274%); 8.3 MiB/s, 133 objects/s recovering 2024-05-15T13:00:43.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:42 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/917151027' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T13:00:43.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:42 smithi120 ceph-mon[43961]: pgmap v999: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 5.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 689 op/s; 19082/245070 objects degraded (7.786%); 15375/245070 objects misplaced (6.274%); 8.3 MiB/s, 133 objects/s recovering 2024-05-15T13:00:43.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:42 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/917151027' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T13:00:44.101 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:00:44.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:44.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:00:44.213 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:00:44.263 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:43 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 19082/245070 objects degraded (7.786%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:44.267 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:44.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:43 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 19082/245070 objects degraded (7.786%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:43 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 19082/245070 objects degraded (7.786%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:45.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:44 smithi027 ceph-mon[39795]: pgmap v1000: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 631 op/s; 19078/242805 objects degraded (7.857%); 15026/242805 objects misplaced (6.189%); 8.9 MiB/s, 143 objects/s recovering 2024-05-15T13:00:45.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:44 smithi120 ceph-mon[43961]: pgmap v1000: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 631 op/s; 19078/242805 objects degraded (7.857%); 15026/242805 objects misplaced (6.189%); 8.9 MiB/s, 143 objects/s recovering 2024-05-15T13:00:45.435 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:44 smithi177 ceph-mon[44303]: pgmap v1000: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 631 op/s; 19078/242805 objects degraded (7.857%); 15026/242805 objects misplaced (6.189%); 8.9 MiB/s, 143 objects/s recovering 2024-05-15T13:00:45.799 INFO:tasks.thrashosds.thrasher:j is {'epoch': 368, 'fsid': 'deaea04e-12b7-11ef-bc98-c7b262605968', 'created': '2024-05-15T12:38:21.244556+0000', 'modified': '2024-05-15T13:00:39.222468+0000', 'last_up_change': '2024-05-15T12:45:50.876090+0000', 'last_in_change': '2024-05-15T12:59:10.294721+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-15T12:41:57.948613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9465229511260986, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 4, 'pool_name': 'unique_pool_2', 'create_time': '2024-05-15T12:57:53.956942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '321', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.990000009536743, 'score_stable': 2.990000009536743, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.75, 'raw_score_stable': 2.75, 'primary_affinity_weighted': 0.9465229511260986, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': '64fb3ad2-08a7-4106-a07a-6c776c877faf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 343, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6802', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6803', 'nonce': 4239116935}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6804', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6805', 'nonce': 4239116935}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6808', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6809', 'nonce': 4239116935}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6806', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6807', 'nonce': 4239116935}]}, 'public_addr': '172.21.15.27:6803/4239116935', 'cluster_addr': '172.21.15.27:6805/4239116935', 'heartbeat_back_addr': '172.21.15.27:6809/4239116935', 'heartbeat_front_addr': '172.21.15.27:6807/4239116935', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '9c16dd90-9d51-48b8-a296-1c8879ba1bc8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 363, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6810', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6811', 'nonce': 1466270637}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6812', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6813', 'nonce': 1466270637}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6816', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6817', 'nonce': 1466270637}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6814', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6815', 'nonce': 1466270637}]}, 'public_addr': '172.21.15.27:6811/1466270637', 'cluster_addr': '172.21.15.27:6813/1466270637', 'heartbeat_back_addr': '172.21.15.27:6817/1466270637', 'heartbeat_front_addr': '172.21.15.27:6815/1466270637', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'cbe3ba13-3ea3-4891-8d2d-6b85581a732c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6818', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6819', 'nonce': 3285413897}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6820', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6821', 'nonce': 3285413897}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6824', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6825', 'nonce': 3285413897}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6822', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6823', 'nonce': 3285413897}]}, 'public_addr': '172.21.15.27:6819/3285413897', 'cluster_addr': '172.21.15.27:6821/3285413897', 'heartbeat_back_addr': '172.21.15.27:6825/3285413897', 'heartbeat_front_addr': '172.21.15.27:6823/3285413897', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2e83735b-b944-4f87-8080-b994d4b96897', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 317, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6826', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6827', 'nonce': 3008975244}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6828', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6829', 'nonce': 3008975244}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6833', 'nonce': 3008975244}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6830', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6831', 'nonce': 3008975244}]}, 'public_addr': '172.21.15.27:6827/3008975244', 'cluster_addr': '172.21.15.27:6829/3008975244', 'heartbeat_back_addr': '172.21.15.27:6833/3008975244', 'heartbeat_front_addr': '172.21.15.27:6831/3008975244', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7c39c1dd-3867-46f7-8372-7688fa13b16a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 343, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6800', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6801', 'nonce': 3877842957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6802', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6803', 'nonce': 3877842957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6806', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6807', 'nonce': 3877842957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6804', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6805', 'nonce': 3877842957}]}, 'public_addr': '172.21.15.120:6801/3877842957', 'cluster_addr': '172.21.15.120:6803/3877842957', 'heartbeat_back_addr': '172.21.15.120:6807/3877842957', 'heartbeat_front_addr': '172.21.15.120:6805/3877842957', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a6dc220f-aec0-4454-9695-b44c0b9770e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 343, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6808', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6809', 'nonce': 1600179183}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6810', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6811', 'nonce': 1600179183}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6814', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6815', 'nonce': 1600179183}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6812', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6813', 'nonce': 1600179183}]}, 'public_addr': '172.21.15.120:6809/1600179183', 'cluster_addr': '172.21.15.120:6811/1600179183', 'heartbeat_back_addr': '172.21.15.120:6815/1600179183', 'heartbeat_front_addr': '172.21.15.120:6813/1600179183', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 362, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6816', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6817', 'nonce': 3323706735}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6818', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6819', 'nonce': 3323706735}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6822', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6823', 'nonce': 3323706735}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6820', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6821', 'nonce': 3323706735}]}, 'public_addr': '172.21.15.120:6817/3323706735', 'cluster_addr': '172.21.15.120:6819/3323706735', 'heartbeat_back_addr': '172.21.15.120:6823/3323706735', 'heartbeat_front_addr': '172.21.15.120:6821/3323706735', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6ef085d8-f59b-4033-8280-2c8f78693e30', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 219, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6824', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6825', 'nonce': 2960616153}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6826', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6827', 'nonce': 2960616153}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6830', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6831', 'nonce': 2960616153}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6828', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6829', 'nonce': 2960616153}]}, 'public_addr': '172.21.15.120:6825/2960616153', 'cluster_addr': '172.21.15.120:6827/2960616153', 'heartbeat_back_addr': '172.21.15.120:6831/2960616153', 'heartbeat_front_addr': '172.21.15.120:6829/2960616153', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '26514b11-d02b-46e9-a4e8-3da193fcdbcc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 292, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6800', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6801', 'nonce': 1132459315}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6802', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6803', 'nonce': 1132459315}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6806', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6807', 'nonce': 1132459315}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6804', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6805', 'nonce': 1132459315}]}, 'public_addr': '172.21.15.177:6801/1132459315', 'cluster_addr': '172.21.15.177:6803/1132459315', 'heartbeat_back_addr': '172.21.15.177:6807/1132459315', 'heartbeat_front_addr': '172.21.15.177:6805/1132459315', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '6f9497bc-e962-409d-8a2c-0c634cc1124f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9062042236328125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 317, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6808', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6809', 'nonce': 2710457646}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6810', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6811', 'nonce': 2710457646}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6814', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6815', 'nonce': 2710457646}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6812', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6813', 'nonce': 2710457646}]}, 'public_addr': '172.21.15.177:6809/2710457646', 'cluster_addr': '172.21.15.177:6811/2710457646', 'heartbeat_back_addr': '172.21.15.177:6815/2710457646', 'heartbeat_front_addr': '172.21.15.177:6813/2710457646', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'f852876f-c7d1-4039-90c4-5ab09dbd3e32', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.4520721435546875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 81, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6816', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6817', 'nonce': 3236900263}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6818', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6819', 'nonce': 3236900263}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6822', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6823', 'nonce': 3236900263}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6820', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6821', 'nonce': 3236900263}]}, 'public_addr': '172.21.15.177:6817/3236900263', 'cluster_addr': '172.21.15.177:6819/3236900263', 'heartbeat_back_addr': '172.21.15.177:6823/3236900263', 'heartbeat_front_addr': '172.21.15.177:6821/3236900263', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'cccc53c5-8476-4d55-b160-4bdc5cf92902', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 90, 'up_thru': 345, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6824', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6825', 'nonce': 681011628}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6826', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6827', 'nonce': 681011628}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6830', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6831', 'nonce': 681011628}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6828', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6829', 'nonce': 681011628}]}, 'public_addr': '172.21.15.177:6825/681011628', 'cluster_addr': '172.21.15.177:6827/681011628', 'heartbeat_back_addr': '172.21.15.177:6831/681011628', 'heartbeat_front_addr': '172.21.15.177:6829/681011628', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:59:43.077075+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:00:31.854618+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:59:57.743220+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:59:52.263164+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:00:05.117774+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:00:15.644018+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:59:50.811831+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-15T12:59:49.579189+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:00:38.448698+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:00:32.401461+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T12:59:39.949283+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:00:10.897851+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [1, 6, 10]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '4.1', 'osds': [4, 5, 6]}, {'pgid': '4.2', 'osds': [5, 9]}, {'pgid': '4.3', 'osds': [0, 1]}, {'pgid': '4.5', 'osds': [6, 10]}, {'pgid': '4.8', 'osds': [6, 9]}, {'pgid': '4.9', 'osds': [1, 3]}, {'pgid': '4.f', 'osds': [1, 3]}], 'primary_temp': [], 'blocklist': {'172.21.15.27:6801/2530617473': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:6800/2530617473': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:0/1880261204': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:6801/3441136471': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/4291446247': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:0/580128209': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:6800/3441136471': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/3011229857': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/2109693913': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/2445663024': '2024-05-16T12:39:19.409195+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-05-15T13:00:45.799 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:00:45.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:45 smithi027 ceph-mon[39795]: pgmap v1001: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 632 op/s; 19078/238803 objects degraded (7.989%); 14802/238803 objects misplaced (6.198%); 8.9 MiB/s, 143 objects/s recovering 2024-05-15T13:00:46.245 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-05-15T13:00:46.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:45 smithi120 ceph-mon[43961]: pgmap v1001: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 632 op/s; 19078/238803 objects degraded (7.989%); 14802/238803 objects misplaced (6.198%); 8.9 MiB/s, 143 objects/s recovering 2024-05-15T13:00:46.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:45 smithi177 ceph-mon[44303]: pgmap v1001: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 632 op/s; 19078/238803 objects degraded (7.989%); 14802/238803 objects misplaced (6.198%); 8.9 MiB/s, 143 objects/s recovering 2024-05-15T13:00:46.757 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:00:46.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:46.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:00:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:00:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:00:46.864 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:00:46.907 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:46.944 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:00:46.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:46.948 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:00:47.049 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:00:47.093 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:47.123 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:00:47.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:47.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:00:47.227 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:00:47.295 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:47.321 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:00:47.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:47.325 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:00:47.426 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:00:47.457 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:47.484 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:00:47.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:47.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:00:47.594 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:00:47.636 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:47.689 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:00:47.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:47.693 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:00:47.795 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:00:47.831 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:47.862 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:00:47.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:47.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:00:47.972 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:00:48.015 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:48.046 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:00:48.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:48.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:00:48.151 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:00:48.182 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:48.210 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:00:48.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:48.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:00:48.324 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:00:48.352 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:48.374 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:00:48.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:48.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:00:48.479 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:00:48.531 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:48.560 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:00:48.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:48.564 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:00:48.665 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:00:48.697 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:48.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:48 smithi177 ceph-mon[44303]: pgmap v1002: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 19070/235179 objects degraded (8.109%); 14621/235179 objects misplaced (6.217%); 9.0 MiB/s, 146 objects/s recovering 2024-05-15T13:00:48.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:48 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 19070/235179 objects degraded (8.109%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:49.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:48 smithi027 ceph-mon[39795]: pgmap v1002: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 19070/235179 objects degraded (8.109%); 14621/235179 objects misplaced (6.217%); 9.0 MiB/s, 146 objects/s recovering 2024-05-15T13:00:49.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:48 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 19070/235179 objects degraded (8.109%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:49.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:48 smithi120 ceph-mon[43961]: pgmap v1002: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 19070/235179 objects degraded (8.109%); 14621/235179 objects misplaced (6.217%); 9.0 MiB/s, 146 objects/s recovering 2024-05-15T13:00:49.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:48 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 19070/235179 objects degraded (8.109%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:50.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:49 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4201876367' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:00:50.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:49 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4201876367' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:00:50.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:49 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4201876367' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:00:50.871 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:00:51.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:50 smithi027 ceph-mon[39795]: pgmap v1003: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 656 op/s; 19068/230118 objects degraded (8.286%); 14023/230118 objects misplaced (6.094%); 8.9 MiB/s, 145 objects/s recovering 2024-05-15T13:00:51.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:50 smithi120 ceph-mon[43961]: pgmap v1003: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 656 op/s; 19068/230118 objects degraded (8.286%); 14023/230118 objects misplaced (6.094%); 8.9 MiB/s, 145 objects/s recovering 2024-05-15T13:00:51.419 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:00:51.419 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:00:51.419 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:00:51.419 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:00:51.419 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:00:51.419 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:00:51.419 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:00:51.419 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:00:51.419 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:00:51.420 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:00:51.420 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:00:51.420 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:00:51.420 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:00:51.420 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:00:51.420 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:00:51.420 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:00:51.420 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:00:51.420 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:00:51.420 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:00:51.420 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:00:51.421 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:00:51.421 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:00:51.421 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:00:51.421 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:00:51.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:50 smithi177 ceph-mon[44303]: pgmap v1003: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 656 op/s; 19068/230118 objects degraded (8.286%); 14023/230118 objects misplaced (6.094%); 8.9 MiB/s, 145 objects/s recovering 2024-05-15T13:00:52.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:51 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:00:52.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:51 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:00:52.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:51 smithi027 ceph-mon[39795]: pgmap v1004: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 613 op/s; 19064/226530 objects degraded (8.416%); 13855/226530 objects misplaced (6.116%); 8.8 MiB/s, 142 objects/s recovering 2024-05-15T13:00:52.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:51 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:00:52.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:51 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:00:52.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:51 smithi120 ceph-mon[43961]: pgmap v1004: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 613 op/s; 19064/226530 objects degraded (8.416%); 13855/226530 objects misplaced (6.116%); 8.8 MiB/s, 142 objects/s recovering 2024-05-15T13:00:52.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:51 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:00:52.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:51 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:00:52.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:51 smithi177 ceph-mon[44303]: pgmap v1004: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 613 op/s; 19064/226530 objects degraded (8.416%); 13855/226530 objects misplaced (6.116%); 8.8 MiB/s, 142 objects/s recovering 2024-05-15T13:00:53.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:52 smithi027 ceph-mon[39795]: osdmap e369: 12 total, 12 up, 11 in 2024-05-15T13:00:53.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:52 smithi120 ceph-mon[43961]: osdmap e369: 12 total, 12 up, 11 in 2024-05-15T13:00:53.418 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:00:53.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:53.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:00:53.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:52 smithi177 ceph-mon[44303]: osdmap e369: 12 total, 12 up, 11 in 2024-05-15T13:00:53.524 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:00:53.641 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:54.258 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:00:54.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:53 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:00:54.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:53 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:00:54.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:53 smithi027 ceph-mon[39795]: osdmap e370: 12 total, 12 up, 11 in 2024-05-15T13:00:54.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:53 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 19064/226530 objects degraded (8.416%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:54.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:53 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1801122803' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:00:54.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:53 smithi027 ceph-mon[39795]: pgmap v1007: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 611 op/s; 19062/224124 objects degraded (8.505%); 13454/224124 objects misplaced (6.003%); 10 MiB/s, 170 objects/s recovering 2024-05-15T13:00:54.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:53 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:00:54.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:53 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:00:54.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:53 smithi120 ceph-mon[43961]: osdmap e370: 12 total, 12 up, 11 in 2024-05-15T13:00:54.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:53 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 19064/226530 objects degraded (8.416%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:54.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:53 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1801122803' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:00:54.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:53 smithi120 ceph-mon[43961]: pgmap v1007: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 611 op/s; 19062/224124 objects degraded (8.505%); 13454/224124 objects misplaced (6.003%); 10 MiB/s, 170 objects/s recovering 2024-05-15T13:00:54.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:53 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:00:54.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:53 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:00:54.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:53 smithi177 ceph-mon[44303]: osdmap e370: 12 total, 12 up, 11 in 2024-05-15T13:00:54.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:53 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 19064/226530 objects degraded (8.416%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:54.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:53 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1801122803' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:00:54.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:53 smithi177 ceph-mon[44303]: pgmap v1007: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 611 op/s; 19062/224124 objects degraded (8.505%); 13454/224124 objects misplaced (6.003%); 10 MiB/s, 170 objects/s recovering 2024-05-15T13:00:55.196 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:00:55.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:55 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:00:55.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:55 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1801122803' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:00:55.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:55 smithi027 ceph-mon[39795]: osdmap e371: 12 total, 12 up, 11 in 2024-05-15T13:00:55.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:55 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:00:55.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:55 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1801122803' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:00:55.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:55 smithi120 ceph-mon[43961]: osdmap e371: 12 total, 12 up, 11 in 2024-05-15T13:00:55.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:55 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:00:55.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:55 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1801122803' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:00:55.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:55 smithi177 ceph-mon[44303]: osdmap e371: 12 total, 12 up, 11 in 2024-05-15T13:00:56.060 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:56 smithi027 ceph-mon[39795]: pgmap v1009: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 560 op/s; 19062/220029 objects degraded (8.663%); 13219/220029 objects misplaced (6.008%); 8.3 MiB/s, 134 objects/s recovering 2024-05-15T13:00:56.278 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:00:56.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:56 smithi120 ceph-mon[43961]: pgmap v1009: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 560 op/s; 19062/220029 objects degraded (8.663%); 13219/220029 objects misplaced (6.008%); 8.3 MiB/s, 134 objects/s recovering 2024-05-15T13:00:56.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:56.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:00:56.385 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:00:56.412 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:56 smithi177 ceph-mon[44303]: pgmap v1009: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 560 op/s; 19062/220029 objects degraded (8.663%); 13219/220029 objects misplaced (6.008%); 8.3 MiB/s, 134 objects/s recovering 2024-05-15T13:00:56.415 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:56.442 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:00:56.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:56.449 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:00:56.494 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:00:56.494 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T13:00:56.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T13:00:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:00:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:37183 3.6 3:6cf935c9:::benchmark_data_smithi177_88104_object37182:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-05-15T13:00:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:53:22.828029+0000", 2024-05-15T13:00:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 453.66328497900003, 2024-05-15T13:00:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.211649965, 2024-05-15T13:00:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:00:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:00:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:00:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 37183 2024-05-15T13:00:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:56.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:56.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828029+0000", 2024-05-15T13:00:56.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:56.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828029+0000", 2024-05-15T13:00:56.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:56.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828030+0000", 2024-05-15T13:00:56.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5149999999999999e-06 2024-05-15T13:00:56.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:56.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828157+0000", 2024-05-15T13:00:56.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000127119 2024-05-15T13:00:56.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:56.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828171+0000", 2024-05-15T13:00:56.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4134999999999999e-05 2024-05-15T13:00:56.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:56.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828180+0000", 2024-05-15T13:00:56.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6910000000000003e-06 2024-05-15T13:00:56.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:00:56.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828229+0000", 2024-05-15T13:00:56.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9320999999999998e-05 2024-05-15T13:00:56.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:00:56.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828370+0000", 2024-05-15T13:00:56.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000140705 2024-05-15T13:00:56.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T13:00:56.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828519+0000", 2024-05-15T13:00:56.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014910699999999999 2024-05-15T13:00:56.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.831299+0000", 2024-05-15T13:00:56.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0027803599999999999 2024-05-15T13:00:56.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:00:56.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.831307+0000", 2024-05-15T13:00:56.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1269999999999997e-06 2024-05-15T13:00:56.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.039610+0000", 2024-05-15T13:00:56.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.208303346 2024-05-15T13:00:56.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:00:56.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.039648+0000", 2024-05-15T13:00:56.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8115999999999999e-05 2024-05-15T13:00:56.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:00:56.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.039679+0000", 2024-05-15T13:00:56.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0423e-05 2024-05-15T13:00:56.502 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.502 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:56.502 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.502 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:37184 3.6 3:64ac7aa0:::benchmark_data_smithi177_88104_object37183:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-05-15T13:00:56.502 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:53:22.828198+0000", 2024-05-15T13:00:56.502 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 453.66311524100001, 2024-05-15T13:00:56.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.21159296799999999, 2024-05-15T13:00:56.502 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:56.502 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:56.502 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:00:56.502 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:00:56.503 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:00:56.503 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 37184 2024-05-15T13:00:56.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.503 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:56.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:56.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828198+0000", 2024-05-15T13:00:56.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:56.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828198+0000", 2024-05-15T13:00:56.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:56.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828199+0000", 2024-05-15T13:00:56.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6199999999999998e-07 2024-05-15T13:00:56.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:56.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828231+0000", 2024-05-15T13:00:56.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2506999999999999e-05 2024-05-15T13:00:56.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:56.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828239+0000", 2024-05-15T13:00:56.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7419999999999997e-06 2024-05-15T13:00:56.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:56.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.828245+0000", 2024-05-15T13:00:56.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6359999999999999e-06 2024-05-15T13:00:56.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:00:56.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.829838+0000", 2024-05-15T13:00:56.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001593391 2024-05-15T13:00:56.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:00:56.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.829960+0000", 2024-05-15T13:00:56.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000121637 2024-05-15T13:00:56.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T13:00:56.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.830089+0000", 2024-05-15T13:00:56.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012972999999999999 2024-05-15T13:00:56.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:00:56.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.831422+0000", 2024-05-15T13:00:56.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0013328280000000001 2024-05-15T13:00:56.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.831931+0000", 2024-05-15T13:00:56.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00050825999999999996 2024-05-15T13:00:56.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.039722+0000", 2024-05-15T13:00:56.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20779120300000001 2024-05-15T13:00:56.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:00:56.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.039762+0000", 2024-05-15T13:00:56.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9885000000000002e-05 2024-05-15T13:00:56.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:00:56.508 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.039791+0000", 2024-05-15T13:00:56.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9587e-05 2024-05-15T13:00:56.508 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.508 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:56.508 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.508 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:37200 3.1 3:8e4b7950:::benchmark_data_smithi177_88104_object37199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-05-15T13:00:56.508 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:53:22.834283+0000", 2024-05-15T13:00:56.508 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 453.65703065899999, 2024-05-15T13:00:56.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.208503207, 2024-05-15T13:00:56.508 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:56.508 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:56.509 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:00:56.509 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:00:56.509 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:00:56.509 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 37200 2024-05-15T13:00:56.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.509 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:56.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:56.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.834283+0000", 2024-05-15T13:00:56.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:56.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.834283+0000", 2024-05-15T13:00:56.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:56.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.834284+0000", 2024-05-15T13:00:56.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.624e-06 2024-05-15T13:00:56.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:56.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.834373+0000", 2024-05-15T13:00:56.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8338999999999996e-05 2024-05-15T13:00:56.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:56.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.834385+0000", 2024-05-15T13:00:56.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2238e-05 2024-05-15T13:00:56.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:56.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.834391+0000", 2024-05-15T13:00:56.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7790000000000001e-06 2024-05-15T13:00:56.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:00:56.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.834437+0000", 2024-05-15T13:00:56.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5831999999999998e-05 2024-05-15T13:00:56.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:00:56.512 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.834528+0000", 2024-05-15T13:00:56.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1253000000000002e-05 2024-05-15T13:00:56.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 4,10", 2024-05-15T13:00:56.512 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.834619+0000", 2024-05-15T13:00:56.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0716000000000004e-05 2024-05-15T13:00:56.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:00:56.512 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.835998+0000", 2024-05-15T13:00:56.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001379433 2024-05-15T13:00:56.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.836563+0000", 2024-05-15T13:00:56.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00056531099999999998 2024-05-15T13:00:56.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.042703+0000", 2024-05-15T13:00:56.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20613922700000001 2024-05-15T13:00:56.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:00:56.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.042754+0000", 2024-05-15T13:00:56.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1468000000000003e-05 2024-05-15T13:00:56.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.514 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:00:56.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.042786+0000", 2024-05-15T13:00:56.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1986999999999997e-05 2024-05-15T13:00:56.514 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.514 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:56.514 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.514 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:37202 3.6 3:6388ea14:::benchmark_data_smithi177_88104_object37201:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-05-15T13:00:56.514 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:53:22.835082+0000", 2024-05-15T13:00:56.514 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 453.65623154600001, 2024-05-15T13:00:56.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20765689300000001, 2024-05-15T13:00:56.515 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:56.515 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:56.515 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:00:56.515 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:00:56.515 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:00:56.515 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 37202 2024-05-15T13:00:56.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.515 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:56.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:56.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.835082+0000", 2024-05-15T13:00:56.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:56.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.835082+0000", 2024-05-15T13:00:56.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:56.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.835083+0000", 2024-05-15T13:00:56.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8599999999999997e-07 2024-05-15T13:00:56.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:56.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.835189+0000", 2024-05-15T13:00:56.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010657 2024-05-15T13:00:56.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.517 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:56.517 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.835205+0000", 2024-05-15T13:00:56.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5693999999999999e-05 2024-05-15T13:00:56.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.517 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:56.517 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.835214+0000", 2024-05-15T13:00:56.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2639999999999997e-06 2024-05-15T13:00:56.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.517 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:00:56.517 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.835276+0000", 2024-05-15T13:00:56.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1495000000000007e-05 2024-05-15T13:00:56.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:00:56.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.835405+0000", 2024-05-15T13:00:56.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012882400000000001 2024-05-15T13:00:56.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T13:00:56.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.835547+0000", 2024-05-15T13:00:56.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000142268 2024-05-15T13:00:56.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:00:56.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.837000+0000", 2024-05-15T13:00:56.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001453024 2024-05-15T13:00:56.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.837078+0000", 2024-05-15T13:00:56.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7657000000000005e-05 2024-05-15T13:00:56.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.042639+0000", 2024-05-15T13:00:56.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20556112200000001 2024-05-15T13:00:56.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:00:56.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.042699+0000", 2024-05-15T13:00:56.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0133000000000002e-05 2024-05-15T13:00:56.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:00:56.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.042739+0000", 2024-05-15T13:00:56.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0055999999999997e-05 2024-05-15T13:00:56.520 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.520 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:56.520 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.520 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:37206 3.6 3:66563e40:::benchmark_data_smithi177_88104_object37205:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-05-15T13:00:56.521 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:53:22.837645+0000", 2024-05-15T13:00:56.521 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 453.65366897600001, 2024-05-15T13:00:56.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20581651500000001, 2024-05-15T13:00:56.521 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:56.521 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:56.521 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:00:56.521 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:00:56.521 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:00:56.521 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 37206 2024-05-15T13:00:56.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.521 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:56.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:56.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.837645+0000", 2024-05-15T13:00:56.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.522 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.522 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:56.522 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.837645+0000", 2024-05-15T13:00:56.522 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.522 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.522 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:56.522 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.837645+0000", 2024-05-15T13:00:56.522 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5000000000000001e-07 2024-05-15T13:00:56.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.522 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.522 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:56.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.837700+0000", 2024-05-15T13:00:56.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4256999999999999e-05 2024-05-15T13:00:56.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:56.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.837710+0000", 2024-05-15T13:00:56.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0057e-05 2024-05-15T13:00:56.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:56.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.837717+0000", 2024-05-15T13:00:56.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9650000000000002e-06 2024-05-15T13:00:56.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:00:56.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.837754+0000", 2024-05-15T13:00:56.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7404000000000003e-05 2024-05-15T13:00:56.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:00:56.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.837837+0000", 2024-05-15T13:00:56.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2468999999999999e-05 2024-05-15T13:00:56.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T13:00:56.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.837924+0000", 2024-05-15T13:00:56.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7070000000000005e-05 2024-05-15T13:00:56.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:00:56.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.838965+0000", 2024-05-15T13:00:56.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010411470000000001 2024-05-15T13:00:56.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.839177+0000", 2024-05-15T13:00:56.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000211914 2024-05-15T13:00:56.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.043375+0000", 2024-05-15T13:00:56.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.204198555 2024-05-15T13:00:56.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:00:56.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.043430+0000", 2024-05-15T13:00:56.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5133000000000002e-05 2024-05-15T13:00:56.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:00:56.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.043461+0000", 2024-05-15T13:00:56.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0593999999999999e-05 2024-05-15T13:00:56.526 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.526 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:56.526 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.527 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:37215 3.1 3:8a061330:::benchmark_data_smithi177_88104_object37214:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-05-15T13:00:56.527 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:53:22.842190+0000", 2024-05-15T13:00:56.527 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 453.64912323900001, 2024-05-15T13:00:56.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.200671393, 2024-05-15T13:00:56.527 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:56.527 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:56.527 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:00:56.527 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:00:56.527 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:00:56.527 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 37215 2024-05-15T13:00:56.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.527 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:56.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:56.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.842190+0000", 2024-05-15T13:00:56.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:56.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.842190+0000", 2024-05-15T13:00:56.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:56.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.842192+0000", 2024-05-15T13:00:56.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2389999999999999e-06 2024-05-15T13:00:56.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:56.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.842263+0000", 2024-05-15T13:00:56.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1099999999999994e-05 2024-05-15T13:00:56.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:56.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.842274+0000", 2024-05-15T13:00:56.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1844e-05 2024-05-15T13:00:56.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:56.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.842283+0000", 2024-05-15T13:00:56.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0970000000000006e-06 2024-05-15T13:00:56.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:00:56.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.842342+0000", 2024-05-15T13:00:56.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9846000000000003e-05 2024-05-15T13:00:56.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:00:56.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.842478+0000", 2024-05-15T13:00:56.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013552200000000001 2024-05-15T13:00:56.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 4,10", 2024-05-15T13:00:56.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.842632+0000", 2024-05-15T13:00:56.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015451500000000001 2024-05-15T13:00:56.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:00:56.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.843799+0000", 2024-05-15T13:00:56.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0011665709999999999 2024-05-15T13:00:56.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.844098+0000", 2024-05-15T13:00:56.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00029868400000000002 2024-05-15T13:00:56.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.042815+0000", 2024-05-15T13:00:56.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19871739699999999 2024-05-15T13:00:56.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:00:56.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.042842+0000", 2024-05-15T13:00:56.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7098e-05 2024-05-15T13:00:56.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:00:56.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.042862+0000", 2024-05-15T13:00:56.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.948e-05 2024-05-15T13:00:56.532 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.533 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:56.533 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.533 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:37220 3.1 3:868c097d:::benchmark_data_smithi177_88104_object37219:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-05-15T13:00:56.533 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:53:22.846699+0000", 2024-05-15T13:00:56.533 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 453.64461476299999, 2024-05-15T13:00:56.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19624024700000001, 2024-05-15T13:00:56.533 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:56.533 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:56.533 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:00:56.533 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:00:56.533 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:00:56.533 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 37220 2024-05-15T13:00:56.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.534 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:56.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:56.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.846699+0000", 2024-05-15T13:00:56.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:56.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.846699+0000", 2024-05-15T13:00:56.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:56.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.846700+0000", 2024-05-15T13:00:56.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1289999999999999e-06 2024-05-15T13:00:56.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:56.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.846787+0000", 2024-05-15T13:00:56.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7292000000000004e-05 2024-05-15T13:00:56.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:56.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.846798+0000", 2024-05-15T13:00:56.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0771000000000001e-05 2024-05-15T13:00:56.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:56.536 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.846804+0000", 2024-05-15T13:00:56.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1720000000000004e-06 2024-05-15T13:00:56.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.536 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:00:56.536 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.846861+0000", 2024-05-15T13:00:56.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6418000000000001e-05 2024-05-15T13:00:56.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.536 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:00:56.536 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.846999+0000", 2024-05-15T13:00:56.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013880000000000001 2024-05-15T13:00:56.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 4,10", 2024-05-15T13:00:56.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.847174+0000", 2024-05-15T13:00:56.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017457900000000001 2024-05-15T13:00:56.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.848585+0000", 2024-05-15T13:00:56.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001411151 2024-05-15T13:00:56.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:00:56.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:22.848604+0000", 2024-05-15T13:00:56.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8672e-05 2024-05-15T13:00:56.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.042884+0000", 2024-05-15T13:00:56.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.194280602 2024-05-15T13:00:56.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:00:56.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.042908+0000", 2024-05-15T13:00:56.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4019000000000002e-05 2024-05-15T13:00:56.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:00:56.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:23.042939+0000", 2024-05-15T13:00:56.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0642000000000002e-05 2024-05-15T13:00:56.539 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.539 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:56.539 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.539 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:113042 3.6 3:6a7f7ff0:::benchmark_data_smithi177_88104_object113041:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-05-15T13:00:56.539 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:54:17.961745+0000", 2024-05-15T13:00:56.539 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 398.52956817, 2024-05-15T13:00:56.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22458128399999999, 2024-05-15T13:00:56.539 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:56.539 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:56.539 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:00:56.539 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:00:56.540 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:00:56.540 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 113042 2024-05-15T13:00:56.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.540 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:56.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:56.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.961745+0000", 2024-05-15T13:00:56.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:56.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.961745+0000", 2024-05-15T13:00:56.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:56.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.961747+0000", 2024-05-15T13:00:56.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.736e-06 2024-05-15T13:00:56.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:56.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.961778+0000", 2024-05-15T13:00:56.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1078000000000003e-05 2024-05-15T13:00:56.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:56.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.961791+0000", 2024-05-15T13:00:56.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2998e-05 2024-05-15T13:00:56.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:56.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.961800+0000", 2024-05-15T13:00:56.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8200000000000003e-06 2024-05-15T13:00:56.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:00:56.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.962010+0000", 2024-05-15T13:00:56.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00021018499999999999 2024-05-15T13:00:56.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:00:56.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.962152+0000", 2024-05-15T13:00:56.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000141638 2024-05-15T13:00:56.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T13:00:56.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.962291+0000", 2024-05-15T13:00:56.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013942199999999999 2024-05-15T13:00:56.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:00:56.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.969770+0000", 2024-05-15T13:00:56.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0074782750000000004 2024-05-15T13:00:56.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.969801+0000", 2024-05-15T13:00:56.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1565e-05 2024-05-15T13:00:56.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.186218+0000", 2024-05-15T13:00:56.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.216416627 2024-05-15T13:00:56.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:00:56.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.186276+0000", 2024-05-15T13:00:56.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8455999999999997e-05 2024-05-15T13:00:56.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:00:56.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.186327+0000", 2024-05-15T13:00:56.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0484e-05 2024-05-15T13:00:56.545 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.545 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:56.545 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.545 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:113044 3.6 3:660f5aad:::benchmark_data_smithi177_88104_object113043:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-05-15T13:00:56.545 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:54:17.962563+0000", 2024-05-15T13:00:56.545 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 398.52875082600002, 2024-05-15T13:00:56.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22388765999999999, 2024-05-15T13:00:56.545 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:56.545 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:56.545 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:00:56.546 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:00:56.546 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:00:56.546 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 113044 2024-05-15T13:00:56.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.546 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:56.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:56.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.962563+0000", 2024-05-15T13:00:56.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:56.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.962563+0000", 2024-05-15T13:00:56.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:56.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.962564+0000", 2024-05-15T13:00:56.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.78e-07 2024-05-15T13:00:56.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:56.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.962621+0000", 2024-05-15T13:00:56.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7899999999999998e-05 2024-05-15T13:00:56.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:56.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.962630+0000", 2024-05-15T13:00:56.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9870000000000005e-06 2024-05-15T13:00:56.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:56.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.962637+0000", 2024-05-15T13:00:56.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4300000000000003e-06 2024-05-15T13:00:56.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:00:56.549 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.965856+0000", 2024-05-15T13:00:56.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032189359999999999 2024-05-15T13:00:56.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.549 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:00:56.549 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.966002+0000", 2024-05-15T13:00:56.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014567599999999999 2024-05-15T13:00:56.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.549 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T13:00:56.549 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.966173+0000", 2024-05-15T13:00:56.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017171299999999999 2024-05-15T13:00:56.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.969945+0000", 2024-05-15T13:00:56.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0037720589999999999 2024-05-15T13:00:56.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:00:56.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.970373+0000", 2024-05-15T13:00:56.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00042798199999999998 2024-05-15T13:00:56.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.186365+0000", 2024-05-15T13:00:56.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.21599131499999999 2024-05-15T13:00:56.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:00:56.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.186414+0000", 2024-05-15T13:00:56.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9097000000000001e-05 2024-05-15T13:00:56.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:00:56.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.186450+0000", 2024-05-15T13:00:56.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6687000000000003e-05 2024-05-15T13:00:56.551 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.551 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:56.551 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.552 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:113073 3.6 3:610a8193:::benchmark_data_smithi177_88104_object113072:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-05-15T13:00:56.552 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:54:17.989442+0000", 2024-05-15T13:00:56.552 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 398.50187154499997, 2024-05-15T13:00:56.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.200631694, 2024-05-15T13:00:56.552 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:56.552 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:56.552 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:00:56.552 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:00:56.552 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:00:56.552 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 113073 2024-05-15T13:00:56.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.552 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:56.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:56.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989442+0000", 2024-05-15T13:00:56.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:56.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989442+0000", 2024-05-15T13:00:56.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:56.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989443+0000", 2024-05-15T13:00:56.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.274e-06 2024-05-15T13:00:56.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:56.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989506+0000", 2024-05-15T13:00:56.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3244000000000006e-05 2024-05-15T13:00:56.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:56.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989520+0000", 2024-05-15T13:00:56.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3532e-05 2024-05-15T13:00:56.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:56.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989526+0000", 2024-05-15T13:00:56.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0440000000000001e-06 2024-05-15T13:00:56.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:00:56.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989602+0000", 2024-05-15T13:00:56.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6211999999999995e-05 2024-05-15T13:00:56.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:00:56.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989736+0000", 2024-05-15T13:00:56.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013396999999999999 2024-05-15T13:00:56.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T13:00:56.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989890+0000", 2024-05-15T13:00:56.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015393899999999999 2024-05-15T13:00:56.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:00:56.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.000201+0000", 2024-05-15T13:00:56.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010310332 2024-05-15T13:00:56.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.000307+0000", 2024-05-15T13:00:56.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000106217 2024-05-15T13:00:56.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.189969+0000", 2024-05-15T13:00:56.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.18966257 2024-05-15T13:00:56.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:00:56.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.190028+0000", 2024-05-15T13:00:56.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9178000000000002e-05 2024-05-15T13:00:56.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:00:56.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.190074+0000", 2024-05-15T13:00:56.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5182000000000002e-05 2024-05-15T13:00:56.557 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.557 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:56.557 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.558 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:113074 3.6 3:64bf8e0c:::benchmark_data_smithi177_88104_object113073:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-05-15T13:00:56.558 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:54:17.989550+0000", 2024-05-15T13:00:56.558 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 398.501763251, 2024-05-15T13:00:56.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.200655903, 2024-05-15T13:00:56.558 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:56.558 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:56.558 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:00:56.558 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:00:56.558 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:00:56.559 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 113074 2024-05-15T13:00:56.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.559 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:56.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:56.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989550+0000", 2024-05-15T13:00:56.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:56.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989550+0000", 2024-05-15T13:00:56.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:56.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989551+0000", 2024-05-15T13:00:56.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8599999999999997e-07 2024-05-15T13:00:56.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:56.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989590+0000", 2024-05-15T13:00:56.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9113000000000003e-05 2024-05-15T13:00:56.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:56.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989600+0000", 2024-05-15T13:00:56.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4890000000000001e-06 2024-05-15T13:00:56.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:56.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.989607+0000", 2024-05-15T13:00:56.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9979999999999997e-06 2024-05-15T13:00:56.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:00:56.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.993352+0000", 2024-05-15T13:00:56.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0037452230000000002 2024-05-15T13:00:56.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:00:56.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.993454+0000", 2024-05-15T13:00:56.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000102515 2024-05-15T13:00:56.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T13:00:56.562 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.993553+0000", 2024-05-15T13:00:56.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8948000000000002e-05 2024-05-15T13:00:56.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:00:56.562 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.000211+0000", 2024-05-15T13:00:56.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0066575269999999999 2024-05-15T13:00:56.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.562 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.000339+0000", 2024-05-15T13:00:56.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012788900000000001 2024-05-15T13:00:56.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.190133+0000", 2024-05-15T13:00:56.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.18979415099999999 2024-05-15T13:00:56.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:00:56.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.190168+0000", 2024-05-15T13:00:56.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5438999999999997e-05 2024-05-15T13:00:56.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:00:56.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.190206+0000", 2024-05-15T13:00:56.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7824999999999998e-05 2024-05-15T13:00:56.564 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.564 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:56.564 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.564 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:113076 3.6 3:698d8b94:::benchmark_data_smithi177_88104_object113075:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-05-15T13:00:56.564 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:54:17.993185+0000", 2024-05-15T13:00:56.564 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 398.49812897200002, 2024-05-15T13:00:56.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.2013375, 2024-05-15T13:00:56.564 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:56.564 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:56.564 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:00:56.564 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:00:56.565 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:00:56.565 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 113076 2024-05-15T13:00:56.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.565 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:56.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:56.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.993185+0000", 2024-05-15T13:00:56.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:56.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.993185+0000", 2024-05-15T13:00:56.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:56.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.993186+0000", 2024-05-15T13:00:56.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4890000000000001e-06 2024-05-15T13:00:56.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:56.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.993241+0000", 2024-05-15T13:00:56.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5182000000000001e-05 2024-05-15T13:00:56.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:56.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.993256+0000", 2024-05-15T13:00:56.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4749999999999999e-05 2024-05-15T13:00:56.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:56.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.993264+0000", 2024-05-15T13:00:56.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3070000000000003e-06 2024-05-15T13:00:56.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:00:56.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.996424+0000", 2024-05-15T13:00:56.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031594259999999999 2024-05-15T13:00:56.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:00:56.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.996497+0000", 2024-05-15T13:00:56.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3330000000000004e-05 2024-05-15T13:00:56.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T13:00:56.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:17.996636+0000", 2024-05-15T13:00:56.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013876000000000001 2024-05-15T13:00:56.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:00:56.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.001265+0000", 2024-05-15T13:00:56.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046289249999999999 2024-05-15T13:00:56.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.006975+0000", 2024-05-15T13:00:56.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0057106680000000003 2024-05-15T13:00:56.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.194420+0000", 2024-05-15T13:00:56.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.18744422599999999 2024-05-15T13:00:56.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:00:56.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.194476+0000", 2024-05-15T13:00:56.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6212000000000003e-05 2024-05-15T13:00:56.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:00:56.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.194522+0000", 2024-05-15T13:00:56.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6224999999999999e-05 2024-05-15T13:00:56.570 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.570 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:56.570 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.570 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:113090 3.6 3:6cf5823a:::benchmark_data_smithi177_88104_object113089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-05-15T13:00:56.570 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:54:18.002227+0000", 2024-05-15T13:00:56.570 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 398.489086558, 2024-05-15T13:00:56.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19613824999999999, 2024-05-15T13:00:56.570 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:56.570 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:56.570 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:00:56.571 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:00:56.571 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:00:56.571 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 113090 2024-05-15T13:00:56.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.571 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:56.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:56.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.002227+0000", 2024-05-15T13:00:56.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:56.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.002227+0000", 2024-05-15T13:00:56.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:56.572 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.002228+0000", 2024-05-15T13:00:56.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.266e-06 2024-05-15T13:00:56.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:56.572 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.002301+0000", 2024-05-15T13:00:56.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2909000000000003e-05 2024-05-15T13:00:56.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:56.572 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.002318+0000", 2024-05-15T13:00:56.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6526000000000001e-05 2024-05-15T13:00:56.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.573 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:56.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.002326+0000", 2024-05-15T13:00:56.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3340000000000002e-06 2024-05-15T13:00:56.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.573 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:00:56.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.002405+0000", 2024-05-15T13:00:56.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9157000000000001e-05 2024-05-15T13:00:56.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.573 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:00:56.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.002518+0000", 2024-05-15T13:00:56.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011309 2024-05-15T13:00:56.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T13:00:56.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.002653+0000", 2024-05-15T13:00:56.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013430899999999999 2024-05-15T13:00:56.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:00:56.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.007906+0000", 2024-05-15T13:00:56.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.005253443 2024-05-15T13:00:56.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.010931+0000", 2024-05-15T13:00:56.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0030248190000000002 2024-05-15T13:00:56.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.198239+0000", 2024-05-15T13:00:56.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.187308266 2024-05-15T13:00:56.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:00:56.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.198312+0000", 2024-05-15T13:00:56.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2860000000000004e-05 2024-05-15T13:00:56.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:00:56.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.198365+0000", 2024-05-15T13:00:56.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3270999999999999e-05 2024-05-15T13:00:56.576 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.576 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:56.576 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.576 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:113101 3.6 3:6a24830d:::benchmark_data_smithi177_88104_object113100:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-05-15T13:00:56.576 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:54:18.016206+0000", 2024-05-15T13:00:56.576 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 398.47510709699998, 2024-05-15T13:00:56.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19138814700000001, 2024-05-15T13:00:56.576 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:56.576 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:56.577 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:00:56.577 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:00:56.577 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:00:56.577 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 113101 2024-05-15T13:00:56.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.577 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:56.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:56.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.016206+0000", 2024-05-15T13:00:56.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:56.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.016206+0000", 2024-05-15T13:00:56.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:56.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.016208+0000", 2024-05-15T13:00:56.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6509999999999999e-06 2024-05-15T13:00:56.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:56.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.016277+0000", 2024-05-15T13:00:56.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8569e-05 2024-05-15T13:00:56.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:56.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.016295+0000", 2024-05-15T13:00:56.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8224999999999999e-05 2024-05-15T13:00:56.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:56.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.016303+0000", 2024-05-15T13:00:56.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5029999999999997e-06 2024-05-15T13:00:56.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:00:56.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.016363+0000", 2024-05-15T13:00:56.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0074000000000001e-05 2024-05-15T13:00:56.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:00:56.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.016509+0000", 2024-05-15T13:00:56.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014593200000000001 2024-05-15T13:00:56.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T13:00:56.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.016656+0000", 2024-05-15T13:00:56.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014697899999999999 2024-05-15T13:00:56.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:00:56.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.019804+0000", 2024-05-15T13:00:56.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031474430000000002 2024-05-15T13:00:56.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.023329+0000", 2024-05-15T13:00:56.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003524957 2024-05-15T13:00:56.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.207502+0000", 2024-05-15T13:00:56.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.184172786 2024-05-15T13:00:56.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:00:56.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.207560+0000", 2024-05-15T13:00:56.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8508999999999998e-05 2024-05-15T13:00:56.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:00:56.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:54:18.207595+0000", 2024-05-15T13:00:56.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4518999999999999e-05 2024-05-15T13:00:56.582 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.582 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:56.582 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.582 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:234133 3.6 3:6bfad9ff:::benchmark_data_smithi177_88104_object100511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T13:00:56.582 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:44.636070+0000", 2024-05-15T13:00:56.582 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 251.855243551, 2024-05-15T13:00:56.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22423902700000001, 2024-05-15T13:00:56.583 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:56.583 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:56.583 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:00:56.583 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:00:56.583 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:00:56.583 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 234133 2024-05-15T13:00:56.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.583 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:56.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:56.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636070+0000", 2024-05-15T13:00:56.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:56.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636070+0000", 2024-05-15T13:00:56.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.584 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:56.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636071+0000", 2024-05-15T13:00:56.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3900000000000004e-07 2024-05-15T13:00:56.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.584 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:56.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636076+0000", 2024-05-15T13:00:56.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7040000000000002e-06 2024-05-15T13:00:56.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:56.585 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636087+0000", 2024-05-15T13:00:56.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1718e-05 2024-05-15T13:00:56.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:56.585 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636098+0000", 2024-05-15T13:00:56.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.084e-05 2024-05-15T13:00:56.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:00:56.585 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.641220+0000", 2024-05-15T13:00:56.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0051218169999999999 2024-05-15T13:00:56.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:00:56.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.641326+0000", 2024-05-15T13:00:56.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010569900000000001 2024-05-15T13:00:56.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T13:00:56.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.641435+0000", 2024-05-15T13:00:56.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000109162 2024-05-15T13:00:56.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.655524+0000", 2024-05-15T13:00:56.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014089121 2024-05-15T13:00:56.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:00:56.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.662050+0000", 2024-05-15T13:00:56.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0065262009999999997 2024-05-15T13:00:56.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.860197+0000", 2024-05-15T13:00:56.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19814722300000001 2024-05-15T13:00:56.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:00:56.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.860258+0000", 2024-05-15T13:00:56.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0826999999999999e-05 2024-05-15T13:00:56.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:00:56.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.860309+0000", 2024-05-15T13:00:56.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0875999999999999e-05 2024-05-15T13:00:56.588 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.588 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:56.588 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.588 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:234137 3.6 3:64a7160a:::benchmark_data_smithi177_88104_object100515:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T13:00:56.588 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:44.636929+0000", 2024-05-15T13:00:56.588 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 251.85438491799999, 2024-05-15T13:00:56.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22352234700000001, 2024-05-15T13:00:56.589 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:56.589 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:56.589 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:00:56.589 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:00:56.589 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:00:56.589 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 234137 2024-05-15T13:00:56.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.589 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:56.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:56.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636929+0000", 2024-05-15T13:00:56.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:56.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636929+0000", 2024-05-15T13:00:56.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:56.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636930+0000", 2024-05-15T13:00:56.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.057e-06 2024-05-15T13:00:56.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:56.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636934+0000", 2024-05-15T13:00:56.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4750000000000004e-06 2024-05-15T13:00:56.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:56.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636945+0000", 2024-05-15T13:00:56.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.094e-05 2024-05-15T13:00:56.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:56.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636953+0000", 2024-05-15T13:00:56.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7400000000000004e-06 2024-05-15T13:00:56.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:00:56.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650374+0000", 2024-05-15T13:00:56.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013421535 2024-05-15T13:00:56.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:00:56.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650489+0000", 2024-05-15T13:00:56.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000114919 2024-05-15T13:00:56.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T13:00:56.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650628+0000", 2024-05-15T13:00:56.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013872799999999999 2024-05-15T13:00:56.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.657205+0000", 2024-05-15T13:00:56.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0065768520000000002 2024-05-15T13:00:56.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:00:56.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.666120+0000", 2024-05-15T13:00:56.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0089147159999999996 2024-05-15T13:00:56.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.860355+0000", 2024-05-15T13:00:56.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19423544000000001 2024-05-15T13:00:56.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:00:56.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.860409+0000", 2024-05-15T13:00:56.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4172999999999998e-05 2024-05-15T13:00:56.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:00:56.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.860451+0000", 2024-05-15T13:00:56.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1771999999999999e-05 2024-05-15T13:00:56.594 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.594 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:56.595 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.595 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:234139 3.6 3:67f4ccfa:::benchmark_data_smithi177_88104_object100517:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T13:00:56.595 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:44.642307+0000", 2024-05-15T13:00:56.595 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 251.84900683999999, 2024-05-15T13:00:56.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22900473900000001, 2024-05-15T13:00:56.595 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:56.595 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:56.595 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:00:56.595 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:00:56.595 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:00:56.595 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 234139 2024-05-15T13:00:56.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.596 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:56.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:56.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.642307+0000", 2024-05-15T13:00:56.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:56.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.642307+0000", 2024-05-15T13:00:56.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:56.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.642308+0000", 2024-05-15T13:00:56.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2100000000000001e-06 2024-05-15T13:00:56.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.597 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:56.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.642311+0000", 2024-05-15T13:00:56.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4630000000000001e-06 2024-05-15T13:00:56.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.597 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:56.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.642320+0000", 2024-05-15T13:00:56.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3119999999999994e-06 2024-05-15T13:00:56.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.597 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:56.598 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.642326+0000", 2024-05-15T13:00:56.598 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1709999999999999e-06 2024-05-15T13:00:56.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.598 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:00:56.598 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.657235+0000", 2024-05-15T13:00:56.598 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014908889999999999 2024-05-15T13:00:56.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.598 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:00:56.598 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.657392+0000", 2024-05-15T13:00:56.598 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015708900000000001 2024-05-15T13:00:56.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T13:00:56.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.657525+0000", 2024-05-15T13:00:56.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000132764 2024-05-15T13:00:56.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.661929+0000", 2024-05-15T13:00:56.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0044041469999999997 2024-05-15T13:00:56.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:00:56.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.666168+0000", 2024-05-15T13:00:56.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0042391920000000001 2024-05-15T13:00:56.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.871182+0000", 2024-05-15T13:00:56.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20501413499999999 2024-05-15T13:00:56.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:00:56.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.871250+0000", 2024-05-15T13:00:56.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8338999999999997e-05 2024-05-15T13:00:56.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:00:56.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.871311+0000", 2024-05-15T13:00:56.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1026999999999997e-05 2024-05-15T13:00:56.601 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.601 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:56.601 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.601 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:234142 3.6 3:6ff8b345:::benchmark_data_smithi177_88104_object100520:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T13:00:56.601 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:44.650217+0000", 2024-05-15T13:00:56.601 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 251.84109670300001, 2024-05-15T13:00:56.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22408456900000001, 2024-05-15T13:00:56.601 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:56.601 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:56.601 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:00:56.601 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:00:56.601 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:00:56.601 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 234142 2024-05-15T13:00:56.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.602 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:56.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.602 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:56.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650217+0000", 2024-05-15T13:00:56.602 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.602 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:56.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650217+0000", 2024-05-15T13:00:56.602 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.602 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:56.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650218+0000", 2024-05-15T13:00:56.603 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.316e-06 2024-05-15T13:00:56.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.603 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.603 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:56.603 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650222+0000", 2024-05-15T13:00:56.603 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5319999999999998e-06 2024-05-15T13:00:56.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.603 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.603 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:56.603 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650230+0000", 2024-05-15T13:00:56.603 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7090000000000008e-06 2024-05-15T13:00:56.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.603 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.603 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:56.604 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650236+0000", 2024-05-15T13:00:56.604 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6899999999999997e-06 2024-05-15T13:00:56.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.604 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.604 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:00:56.604 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.662077+0000", 2024-05-15T13:00:56.604 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.01184057 2024-05-15T13:00:56.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.604 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.604 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:00:56.604 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.662237+0000", 2024-05-15T13:00:56.604 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016033200000000001 2024-05-15T13:00:56.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.604 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.604 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T13:00:56.605 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.662332+0000", 2024-05-15T13:00:56.605 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5199999999999997e-05 2024-05-15T13:00:56.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.605 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:00:56.605 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.666359+0000", 2024-05-15T13:00:56.605 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.004027325 2024-05-15T13:00:56.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.605 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.605 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.668031+0000", 2024-05-15T13:00:56.605 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016717629999999999 2024-05-15T13:00:56.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.606 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.606 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.874188+0000", 2024-05-15T13:00:56.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20615639699999999 2024-05-15T13:00:56.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.606 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:00:56.606 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.874247+0000", 2024-05-15T13:00:56.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9775000000000003e-05 2024-05-15T13:00:56.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.606 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:00:56.606 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.874301+0000", 2024-05-15T13:00:56.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3959999999999998e-05 2024-05-15T13:00:56.606 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.607 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:56.607 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.607 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.607 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:234166 3.6 3:65e82527:::benchmark_data_smithi177_88104_object100544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T13:00:56.607 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:44.676834+0000", 2024-05-15T13:00:56.607 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 251.81447972699999, 2024-05-15T13:00:56.607 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.214610197, 2024-05-15T13:00:56.607 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:56.607 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:56.607 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:00:56.607 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:00:56.607 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:00:56.607 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 234166 2024-05-15T13:00:56.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.608 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:56.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:56.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.676834+0000", 2024-05-15T13:00:56.608 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:56.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.676834+0000", 2024-05-15T13:00:56.608 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:56.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.676836+0000", 2024-05-15T13:00:56.609 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0260000000000002e-06 2024-05-15T13:00:56.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.609 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.609 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:56.609 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.676852+0000", 2024-05-15T13:00:56.609 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.647e-05 2024-05-15T13:00:56.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.609 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.609 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:56.609 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.676869+0000", 2024-05-15T13:00:56.609 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6915999999999999e-05 2024-05-15T13:00:56.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.609 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.609 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:56.609 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.676888+0000", 2024-05-15T13:00:56.610 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8437999999999999e-05 2024-05-15T13:00:56.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.610 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.610 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:00:56.610 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.676954+0000", 2024-05-15T13:00:56.610 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6723000000000005e-05 2024-05-15T13:00:56.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.610 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.610 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:00:56.610 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.677089+0000", 2024-05-15T13:00:56.610 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013486300000000001 2024-05-15T13:00:56.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.610 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.611 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T13:00:56.611 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.677239+0000", 2024-05-15T13:00:56.611 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014941600000000001 2024-05-15T13:00:56.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.611 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.611 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.611 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.683263+0000", 2024-05-15T13:00:56.611 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0060247510000000001 2024-05-15T13:00:56.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.611 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.611 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:00:56.611 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.683276+0000", 2024-05-15T13:00:56.611 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2281e-05 2024-05-15T13:00:56.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.611 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.612 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.891351+0000", 2024-05-15T13:00:56.612 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20807553100000001 2024-05-15T13:00:56.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.612 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.612 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:00:56.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.891400+0000", 2024-05-15T13:00:56.612 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8640999999999997e-05 2024-05-15T13:00:56.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.612 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.612 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:00:56.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.891444+0000", 2024-05-15T13:00:56.612 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4141000000000003e-05 2024-05-15T13:00:56.613 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.613 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:56.613 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.613 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.613 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:234181 3.6 3:635178a9:::benchmark_data_smithi177_88104_object100559:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T13:00:56.613 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:44.691125+0000", 2024-05-15T13:00:56.613 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 251.800188139, 2024-05-15T13:00:56.613 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20043533499999999, 2024-05-15T13:00:56.613 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:00:56.613 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:00:56.613 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:00:56.613 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:00:56.613 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:00:56.613 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 234181 2024-05-15T13:00:56.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.614 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:00:56.614 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.614 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:00:56.614 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691125+0000", 2024-05-15T13:00:56.614 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.614 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.614 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:00:56.614 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691125+0000", 2024-05-15T13:00:56.614 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:00:56.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.614 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.614 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:00:56.615 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691127+0000", 2024-05-15T13:00:56.615 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.392e-06 2024-05-15T13:00:56.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.615 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:00:56.615 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691131+0000", 2024-05-15T13:00:56.615 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5020000000000003e-06 2024-05-15T13:00:56.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.615 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:00:56.615 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691142+0000", 2024-05-15T13:00:56.615 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0478e-05 2024-05-15T13:00:56.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.616 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:00:56.616 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691149+0000", 2024-05-15T13:00:56.616 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9689999999999997e-06 2024-05-15T13:00:56.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.616 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:00:56.616 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691205+0000", 2024-05-15T13:00:56.616 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6137999999999997e-05 2024-05-15T13:00:56.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.616 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:00:56.616 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691332+0000", 2024-05-15T13:00:56.616 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012724599999999999 2024-05-15T13:00:56.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.617 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T13:00:56.617 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691458+0000", 2024-05-15T13:00:56.617 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012549999999999999 2024-05-15T13:00:56.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.617 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.617 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.696662+0000", 2024-05-15T13:00:56.617 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0052043760000000001 2024-05-15T13:00:56.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.617 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:00:56.617 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.697207+0000", 2024-05-15T13:00:56.617 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00054491700000000001 2024-05-15T13:00:56.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.618 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.618 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:00:56.618 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.891478+0000", 2024-05-15T13:00:56.618 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19427066900000001 2024-05-15T13:00:56.618 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.618 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.618 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:00:56.618 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.891527+0000", 2024-05-15T13:00:56.618 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9308999999999998e-05 2024-05-15T13:00:56.618 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:00:56.618 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:00:56.618 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:00:56.618 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.891561+0000", 2024-05-15T13:00:56.618 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3838999999999999e-05 2024-05-15T13:00:56.619 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.619 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:56.619 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.619 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:00:56.619 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:00:56.619 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:00:56.619 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:00:56.653 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:56.677 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:00:56.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:56.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:00:56.785 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:00:56.837 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:56.867 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:00:56.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:56.870 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:00:56.971 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:00:57.037 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:58.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:58 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:00:58.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:58 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:00:58.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:58 smithi027 ceph-mon[39795]: osdmap e372: 12 total, 12 up, 11 in 2024-05-15T13:00:58.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:58 smithi027 ceph-mon[39795]: from='client.25300 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:00:58.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:58 smithi027 ceph-mon[39795]: pgmap v1011: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 561 op/s; 19060/216597 objects degraded (8.800%); 13026/216597 objects misplaced (6.014%); 8.7 MiB/s, 141 objects/s recovering 2024-05-15T13:00:58.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:58 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:00:58.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:58 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:00:58.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:58 smithi120 ceph-mon[43961]: osdmap e372: 12 total, 12 up, 11 in 2024-05-15T13:00:58.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:58 smithi120 ceph-mon[43961]: from='client.25300 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:00:58.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:58 smithi120 ceph-mon[43961]: pgmap v1011: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 561 op/s; 19060/216597 objects degraded (8.800%); 13026/216597 objects misplaced (6.014%); 8.7 MiB/s, 141 objects/s recovering 2024-05-15T13:00:58.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:58 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:00:58.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:58 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:00:58.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:58 smithi177 ceph-mon[44303]: osdmap e372: 12 total, 12 up, 11 in 2024-05-15T13:00:58.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:58 smithi177 ceph-mon[44303]: from='client.25300 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:00:58.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:58 smithi177 ceph-mon[44303]: pgmap v1011: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 561 op/s; 19060/216597 objects degraded (8.800%); 13026/216597 objects misplaced (6.014%); 8.7 MiB/s, 141 objects/s recovering 2024-05-15T13:00:59.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:59 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 19060/216597 objects degraded (8.800%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:59.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:59 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:00:59.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:59 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:00:59.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:59 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:00:59.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:00:59 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:00:59.428 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:00:59.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:59 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 19060/216597 objects degraded (8.800%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:59.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:59 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:00:59.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:59 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:00:59.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:59 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:00:59.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:00:59 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:00:59.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:59.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:00:59.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:59 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 19060/216597 objects degraded (8.800%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:00:59.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:59 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:00:59.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:59 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:00:59.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:59 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:00:59.444 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:00:59 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:00:59.540 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:00:59.569 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:00:59.570 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:00:59.588 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:59.636 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:00:59.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:59.640 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:00:59.741 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:00:59.810 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:00:59.897 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:00:59.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:00:59.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:01:00.023 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:01:00.070 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:00.338 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:00 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:01:00.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:00 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:01:00.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:00 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:01:00.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:00 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:01:00.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:00 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:01:00.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:00 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:01:00.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:00 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:01:00.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:00 smithi027 ceph-mon[39795]: pgmap v1012: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 713 op/s; 19059/211140 objects degraded (9.027%); 12359/211140 objects misplaced (5.853%); 14 MiB/s, 218 objects/s recovering 2024-05-15T13:01:00.353 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:01:00.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:00.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:01:00.389 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-05-15T13:01:00.460 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:01:00.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:00 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:01:00.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:00 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:01:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:00 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:01:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:00 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:01:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:00 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:01:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:00 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:01:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:00 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:01:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:00 smithi120 ceph-mon[43961]: pgmap v1012: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 713 op/s; 19059/211140 objects degraded (9.027%); 12359/211140 objects misplaced (5.853%); 14 MiB/s, 218 objects/s recovering 2024-05-15T13:01:00.562 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:00.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:00 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:01:00.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:00 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:01:00.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:00 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:01:00.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:00 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:01:00.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:00 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:01:00.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:00 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:01:00.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:00 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:01:00.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:00 smithi177 ceph-mon[44303]: pgmap v1012: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 713 op/s; 19059/211140 objects degraded (9.027%); 12359/211140 objects misplaced (5.853%); 14 MiB/s, 218 objects/s recovering 2024-05-15T13:01:00.710 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:01:00.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:00.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:01:00.756 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":1010,"stamp":"2024-05-15T13:00:57.026226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4744741432,"num_objects":72393,"num_object_clones":0,"num_object_copies":217179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19060,"num_objects_misplaced":13026,"num_objects_unfound":0,"num_objects_dirty":72393,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":167730,"num_write_kb":7680955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12270,"num_bytes_recovered":797736672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160788,"ondisk_log_size":160788,"up":51,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":18517572,"kb_used_data":15333892,"kb_used_omap":17,"kb_used_meta":3183278,"kb_avail":1012679100,"statfs":{"total":1055945392128,"available":1036983398400,"internally_reserved":0,"allocated":15701905408,"data_stored":15699110620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17551,"internal_metadata":3259677553},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,9,8,6,3,1,0,0,1,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-204275712,"num_objects":-3117,"num_object_clones":0,"num_object_copies":-9351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":-829,"num_objects_unfound":0,"num_objects_dirty":-3117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3117,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":833,"num_bytes_recovered":54067200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.151616"},"pg_stats":[{"pgid":"4.f","version":"371'10289","reported_seq":16786,"reported_epoch":371,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-05-15T13:00:56.031775+0000","last_change":"2024-05-15T12:59:12.836225+0000","last_active":"2024-05-15T13:00:56.031775+0000","last_peered":"2024-05-15T13:00:56.031775+0000","last_clean":"2024-05-15T12:59:10.259362+0000","last_became_active":"2024-05-15T12:59:12.819643+0000","last_became_peered":"2024-05-15T12:59:12.819643+0000","last_unstale":"2024-05-15T13:00:56.031775+0000","last_undegraded":"2024-05-15T12:59:12.358355+0000","last_fullsized":"2024-05-15T12:59:12.315304+0000","mapping_epoch":343,"log_start":"322'200","ondisk_log_start":"322'200","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288292864,"num_objects":4399,"num_object_clones":0,"num_object_copies":13197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":2978,"num_objects_unfound":0,"num_objects_dirty":4399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10289,"num_write_kb":470016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3189,"num_bytes_recovered":208994304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3],"avail_no_missing":["1","3","9"],"object_location_counts":[{"shards":"1,3","objects":1418},{"shards":"1,3,9","objects":2978}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"371'10540","reported_seq":10659,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-05-15T13:00:54.612892+0000","last_change":"2024-05-15T12:57:55.242710+0000","last_active":"2024-05-15T13:00:54.612892+0000","last_peered":"2024-05-15T13:00:54.612892+0000","last_clean":"2024-05-15T13:00:54.612892+0000","last_became_active":"2024-05-15T12:57:55.242367+0000","last_became_peered":"2024-05-15T12:57:55.242367+0000","last_unstale":"2024-05-15T13:00:54.612892+0000","last_undegraded":"2024-05-15T13:00:54.612892+0000","last_fullsized":"2024-05-15T13:00:54.612892+0000","mapping_epoch":317,"log_start":"322'500","ondisk_log_start":"322'500","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298844160,"num_objects":4560,"num_object_clones":0,"num_object_copies":13680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10540,"num_write_kb":483200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"371'10270","reported_seq":10390,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-05-15T13:00:54.725471+0000","last_change":"2024-05-15T12:57:55.155833+0000","last_active":"2024-05-15T13:00:54.725471+0000","last_peered":"2024-05-15T13:00:54.725471+0000","last_clean":"2024-05-15T13:00:54.725471+0000","last_became_active":"2024-05-15T12:57:55.155480+0000","last_became_peered":"2024-05-15T12:57:55.155480+0000","last_unstale":"2024-05-15T13:00:54.725471+0000","last_undegraded":"2024-05-15T13:00:54.725471+0000","last_fullsized":"2024-05-15T13:00:54.725471+0000","mapping_epoch":317,"log_start":"322'200","ondisk_log_start":"322'200","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294060056,"num_objects":4488,"num_object_clones":0,"num_object_copies":13464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4488,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10270,"num_write_kb":472193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"371'10503","reported_seq":10622,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-05-15T13:00:54.685370+0000","last_change":"2024-05-15T12:57:55.157244+0000","last_active":"2024-05-15T13:00:54.685370+0000","last_peered":"2024-05-15T13:00:54.685370+0000","last_clean":"2024-05-15T13:00:54.685370+0000","last_became_active":"2024-05-15T12:57:55.156859+0000","last_became_peered":"2024-05-15T12:57:55.156859+0000","last_unstale":"2024-05-15T13:00:54.685370+0000","last_undegraded":"2024-05-15T13:00:54.685370+0000","last_fullsized":"2024-05-15T13:00:54.685370+0000","mapping_epoch":317,"log_start":"322'500","ondisk_log_start":"322'500","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293404672,"num_objects":4477,"num_object_clones":0,"num_object_copies":13431,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10503,"num_write_kb":479360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"370'10435","reported_seq":10552,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-05-15T13:00:54.035233+0000","last_change":"2024-05-15T12:57:55.160034+0000","last_active":"2024-05-15T13:00:54.035233+0000","last_peered":"2024-05-15T13:00:54.035233+0000","last_clean":"2024-05-15T13:00:54.035233+0000","last_became_active":"2024-05-15T12:57:55.159699+0000","last_became_peered":"2024-05-15T12:57:55.159699+0000","last_unstale":"2024-05-15T13:00:54.035233+0000","last_undegraded":"2024-05-15T13:00:54.035233+0000","last_fullsized":"2024-05-15T13:00:54.035233+0000","mapping_epoch":317,"log_start":"322'400","ondisk_log_start":"322'400","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298123264,"num_objects":4549,"num_object_clones":0,"num_object_copies":13647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10435,"num_write_kb":479488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"370'10381","reported_seq":10498,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-05-15T13:00:53.486077+0000","last_change":"2024-05-15T12:57:55.242821+0000","last_active":"2024-05-15T13:00:53.486077+0000","last_peered":"2024-05-15T13:00:53.486077+0000","last_clean":"2024-05-15T13:00:53.486077+0000","last_became_active":"2024-05-15T12:57:55.242372+0000","last_became_peered":"2024-05-15T12:57:55.242372+0000","last_unstale":"2024-05-15T13:00:53.486077+0000","last_undegraded":"2024-05-15T13:00:53.486077+0000","last_fullsized":"2024-05-15T13:00:53.486077+0000","mapping_epoch":317,"log_start":"322'300","ondisk_log_start":"322'300","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301006848,"num_objects":4593,"num_object_clones":0,"num_object_copies":13779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10381,"num_write_kb":479168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"371'10546","reported_seq":10777,"reported_epoch":371,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-15T13:00:56.020437+0000","last_change":"2024-05-15T12:59:12.862170+0000","last_active":"2024-05-15T13:00:56.020437+0000","last_peered":"2024-05-15T13:00:56.020437+0000","last_clean":"2024-05-15T12:59:10.318134+0000","last_became_active":"2024-05-15T12:59:12.847627+0000","last_became_peered":"2024-05-15T12:59:12.847627+0000","last_unstale":"2024-05-15T13:00:56.020437+0000","last_undegraded":"2024-05-15T12:59:12.454722+0000","last_fullsized":"2024-05-15T12:59:12.315028+0000","mapping_epoch":343,"log_start":"322'500","ondisk_log_start":"322'500","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287309824,"num_objects":4384,"num_object_clones":0,"num_object_copies":13152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10546,"num_write_kb":477760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[1,3],"avail_no_missing":["1","3","6"],"object_location_counts":[{"shards":"1,3","objects":-1848},{"shards":"1,3,6","objects":3204}],"blocked_by":[],"up_primary":11,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"370'10704","reported_seq":10881,"reported_epoch":370,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-15T13:00:53.445160+0000","last_change":"2024-05-15T12:59:12.870715+0000","last_active":"2024-05-15T13:00:53.445160+0000","last_peered":"2024-05-15T13:00:53.445160+0000","last_clean":"2024-05-15T12:59:10.299125+0000","last_became_active":"2024-05-15T12:59:12.854810+0000","last_became_peered":"2024-05-15T12:59:12.854810+0000","last_unstale":"2024-05-15T13:00:53.445160+0000","last_undegraded":"2024-05-15T12:59:12.358625+0000","last_fullsized":"2024-05-15T12:59:12.315099+0000","mapping_epoch":343,"log_start":"322'700","ondisk_log_start":"322'700","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309067776,"num_objects":4716,"num_object_clones":0,"num_object_copies":14148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10704,"num_write_kb":493440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,9],"acting":[6,9],"avail_no_missing":["6","9","10"],"object_location_counts":[{"shards":"6,9","objects":-1760},{"shards":"6,9,10","objects":3511}],"blocked_by":[],"up_primary":11,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"371'10425","reported_seq":10543,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-05-15T13:00:56.011403+0000","last_change":"2024-05-15T12:57:55.242791+0000","last_active":"2024-05-15T13:00:56.011403+0000","last_peered":"2024-05-15T13:00:56.011403+0000","last_clean":"2024-05-15T13:00:56.011403+0000","last_became_active":"2024-05-15T12:57:55.242453+0000","last_became_peered":"2024-05-15T12:57:55.242453+0000","last_unstale":"2024-05-15T13:00:56.011403+0000","last_undegraded":"2024-05-15T13:00:56.011403+0000","last_fullsized":"2024-05-15T13:00:56.011403+0000","mapping_epoch":317,"log_start":"322'400","ondisk_log_start":"322'400","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291700736,"num_objects":4451,"num_object_clones":0,"num_object_copies":13353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10425,"num_write_kb":476032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"371'10559","reported_seq":10677,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-05-15T13:00:55.445743+0000","last_change":"2024-05-15T12:57:55.157512+0000","last_active":"2024-05-15T13:00:55.445743+0000","last_peered":"2024-05-15T13:00:55.445743+0000","last_clean":"2024-05-15T13:00:55.445743+0000","last_became_active":"2024-05-15T12:57:55.157131+0000","last_became_peered":"2024-05-15T12:57:55.157131+0000","last_unstale":"2024-05-15T13:00:55.445743+0000","last_undegraded":"2024-05-15T13:00:55.445743+0000","last_fullsized":"2024-05-15T13:00:55.445743+0000","mapping_epoch":317,"log_start":"322'500","ondisk_log_start":"322'500","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296943616,"num_objects":4531,"num_object_clones":0,"num_object_copies":13593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10559,"num_write_kb":482880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"371'10504","reported_seq":10623,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-05-15T13:00:55.995402+0000","last_change":"2024-05-15T12:57:55.158173+0000","last_active":"2024-05-15T13:00:55.995402+0000","last_peered":"2024-05-15T13:00:55.995402+0000","last_clean":"2024-05-15T13:00:55.995402+0000","last_became_active":"2024-05-15T12:57:55.157465+0000","last_became_peered":"2024-05-15T12:57:55.157465+0000","last_unstale":"2024-05-15T13:00:55.995402+0000","last_undegraded":"2024-05-15T13:00:55.995402+0000","last_fullsized":"2024-05-15T13:00:55.995402+0000","mapping_epoch":317,"log_start":"322'500","ondisk_log_start":"322'500","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290324480,"num_objects":4430,"num_object_clones":0,"num_object_copies":13290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10504,"num_write_kb":477888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"370'10390","reported_seq":18159,"reported_epoch":370,"state":"active+recovering+undersized+remapped","last_fresh":"2024-05-15T13:00:54.055521+0000","last_change":"2024-05-15T12:59:12.892718+0000","last_active":"2024-05-15T13:00:54.055521+0000","last_peered":"2024-05-15T13:00:54.055521+0000","last_clean":"2024-05-15T12:59:10.291321+0000","last_became_active":"2024-05-15T12:59:12.843364+0000","last_became_peered":"2024-05-15T12:59:12.843364+0000","last_unstale":"2024-05-15T13:00:54.055521+0000","last_undegraded":"2024-05-15T13:00:54.055521+0000","last_fullsized":"2024-05-15T12:59:12.316583+0000","mapping_epoch":343,"log_start":"322'300","ondisk_log_start":"322'300","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298844160,"num_objects":4560,"num_object_clones":0,"num_object_copies":13680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2425,"num_objects_unfound":0,"num_objects_dirty":4560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10390,"num_write_kb":478400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3825,"num_bytes_recovered":250675200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[0,1],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1","objects":2135},{"shards":"0,1,2","objects":2425}],"blocked_by":[],"up_primary":11,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"369'10414","reported_seq":10554,"reported_epoch":369,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-15T13:00:52.750688+0000","last_change":"2024-05-15T12:59:12.864423+0000","last_active":"2024-05-15T13:00:52.750688+0000","last_peered":"2024-05-15T13:00:52.750688+0000","last_clean":"2024-05-15T12:59:10.299975+0000","last_became_active":"2024-05-15T12:59:12.847592+0000","last_became_peered":"2024-05-15T12:59:12.847592+0000","last_unstale":"2024-05-15T13:00:52.750688+0000","last_undegraded":"2024-05-15T12:59:12.358581+0000","last_fullsized":"2024-05-15T12:59:12.320946+0000","mapping_epoch":343,"log_start":"322'400","ondisk_log_start":"322'400","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298975232,"num_objects":4562,"num_object_clones":0,"num_object_copies":13686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6349,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10414,"num_write_kb":479232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,11],"acting":[5,9],"avail_no_missing":["5","8","9"],"object_location_counts":[{"shards":"5,8,9","objects":3434},{"shards":"5,9","objects":-1787}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"371'10510","reported_seq":10790,"reported_epoch":371,"state":"active+recovery_wait+remapped","last_fresh":"2024-05-15T13:00:54.698480+0000","last_change":"2024-05-15T12:59:12.857109+0000","last_active":"2024-05-15T13:00:54.698480+0000","last_peered":"2024-05-15T13:00:54.698480+0000","last_clean":"2024-05-15T12:59:10.257764+0000","last_became_active":"2024-05-15T12:59:12.843041+0000","last_became_peered":"2024-05-15T12:59:12.843041+0000","last_unstale":"2024-05-15T13:00:54.698480+0000","last_undegraded":"2024-05-15T13:00:54.698480+0000","last_fullsized":"2024-05-15T13:00:54.698480+0000","mapping_epoch":343,"log_start":"322'500","ondisk_log_start":"322'500","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'5644","last_scrub_stamp":"2024-05-15T12:59:00.172752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:59:00.172752+0000","objects_scrubbed":5592,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301727744,"num_objects":4604,"num_object_clones":0,"num_object_copies":13812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6340,"num_objects_unfound":0,"num_objects_dirty":4604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10510,"num_write_kb":483648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[4,5,6],"avail_no_missing":["4","5","6"],"object_location_counts":[{"shards":"4,5,6","objects":-1736}],"blocked_by":[],"up_primary":11,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"371'10672","reported_seq":10791,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-05-15T13:00:56.563426+0000","last_change":"2024-05-15T12:57:55.160182+0000","last_active":"2024-05-15T13:00:56.563426+0000","last_peered":"2024-05-15T13:00:56.563426+0000","last_clean":"2024-05-15T13:00:56.563426+0000","last_became_active":"2024-05-15T12:57:55.159918+0000","last_became_peered":"2024-05-15T12:57:55.159918+0000","last_unstale":"2024-05-15T13:00:56.563426+0000","last_undegraded":"2024-05-15T13:00:56.563426+0000","last_fullsized":"2024-05-15T13:00:56.563426+0000","mapping_epoch":317,"log_start":"322'600","ondisk_log_start":"322'600","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292290560,"num_objects":4460,"num_object_clones":0,"num_object_copies":13380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10672,"num_write_kb":484224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"370'10407","reported_seq":20727,"reported_epoch":370,"state":"active+recovering+undersized+remapped","last_fresh":"2024-05-15T13:00:53.514088+0000","last_change":"2024-05-15T12:59:12.844855+0000","last_active":"2024-05-15T13:00:53.514088+0000","last_peered":"2024-05-15T13:00:53.514088+0000","last_clean":"2024-05-15T12:59:10.303422+0000","last_became_active":"2024-05-15T12:59:12.805581+0000","last_became_peered":"2024-05-15T12:59:12.805581+0000","last_unstale":"2024-05-15T13:00:53.514088+0000","last_undegraded":"2024-05-15T13:00:53.514088+0000","last_fullsized":"2024-05-15T12:59:12.315125+0000","mapping_epoch":343,"log_start":"322'400","ondisk_log_start":"322'400","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303235072,"num_objects":4627,"num_object_clones":0,"num_object_copies":13881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1283,"num_objects_unfound":0,"num_objects_dirty":4627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10407,"num_write_kb":481088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5100,"num_bytes_recovered":334233600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10],"avail_no_missing":["6","0","10"],"object_location_counts":[{"shards":"0,6,10","objects":1283},{"shards":"6,10","objects":3344}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":551,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-05-15T13:00:54.255848+0000","last_change":"2024-05-15T13:00:26.045887+0000","last_active":"2024-05-15T13:00:54.255848+0000","last_peered":"2024-05-15T13:00:54.255848+0000","last_clean":"2024-05-15T13:00:54.255848+0000","last_became_active":"2024-05-15T13:00:26.045412+0000","last_became_peered":"2024-05-15T13:00:26.045412+0000","last_unstale":"2024-05-15T13:00:54.255848+0000","last_undegraded":"2024-05-15T13:00:54.255848+0000","last_fullsized":"2024-05-15T13:00:54.255848+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T12:59:04.037279+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T12:59:04.037279+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":4744151064,"num_objects":72391,"num_object_clones":0,"num_object_copies":217173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19060,"num_objects_misplaced":13026,"num_objects_unfound":0,"num_objects_dirty":72391,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167549,"num_write_kb":7678017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12256,"num_bytes_recovered":794034176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15660888064,"data_stored":15660875848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160649,"ondisk_log_size":160649,"up":48,"acting":42,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056825,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":916888,"kb_used_data":737100,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92828264,"statfs":{"total":95995035648,"available":95056142336,"internally_reserved":0,"allocated":754790400,"data_stored":754543448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":184023494},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351166,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1814068,"kb_used_data":1566316,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":91931084,"statfs":{"total":95995035648,"available":94137430016,"internally_reserved":0,"allocated":1603907584,"data_stored":1603648514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":253688256},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514691,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1929524,"kb_used_data":1555432,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":91815628,"statfs":{"total":95995035648,"available":94019203072,"internally_reserved":0,"allocated":1592762368,"data_stored":1592502778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":383056320},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874441,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":910372,"kb_used_data":698712,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92834780,"statfs":{"total":95995035648,"available":95062814720,"internally_reserved":0,"allocated":715481088,"data_stored":715229898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":240518168780,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430417,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2500460,"kb_used_data":2180512,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":91244692,"statfs":{"total":95995035648,"available":93434564608,"internally_reserved":0,"allocated":2232844288,"data_stored":2232589546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":327612870},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575085,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2949116,"kb_used_data":2566512,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":90796036,"statfs":{"total":95995035648,"available":92975140864,"internally_reserved":0,"allocated":2628108288,"data_stored":2627845166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,1,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738612,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1915940,"kb_used_data":1577944,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":91829212,"statfs":{"total":95995035648,"available":94033113088,"internally_reserved":0,"allocated":1615814656,"data_stored":1615563466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411561,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1182872,"kb_used_data":993420,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":92562280,"statfs":{"total":95995035648,"available":94783774720,"internally_reserved":0,"allocated":1017262080,"data_stored":1017015148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149923,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1779820,"kb_used_data":1409892,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":91965332,"statfs":{"total":95995035648,"available":94172499968,"internally_reserved":0,"allocated":1443729408,"data_stored":1443473910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":378796487},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986398,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1785400,"kb_used_data":1452968,"kb_used_omap":1,"kb_used_meta":332414,"kb_avail":91959752,"statfs":{"total":95995035648,"available":94166786048,"internally_reserved":0,"allocated":1487839232,"data_stored":1487579662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":340392389},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692055,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":833112,"kb_used_data":595084,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":92912040,"statfs":{"total":95995035648,"available":95141928960,"internally_reserved":0,"allocated":609366016,"data_stored":609119084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1609498624,"data_stored":1609498624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2623016960,"data_stored":2623012888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1011875840,"data_stored":1011875840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1442119680,"data_stored":1442119680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1483542528,"data_stored":1483538456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":605618176,"data_stored":605618176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2226388992,"data_stored":2226388992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":712966144,"data_stored":712966144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1588989952,"data_stored":1588985880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1602355200,"data_stored":1602355200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":754515968,"data_stored":754515968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:01:00.757 INFO:tasks.thrashosds.thrasher:Setting 4.f to [1, 7, 3] 2024-05-15T13:01:00.757 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.f', '1', '7', '3'] 2024-05-15T13:01:00.758 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pg-upmap 4.f 1 7 3 2024-05-15T13:01:00.803 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:01:00.827 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:01:00.867 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:00.887 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:01:00.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:00.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:01:00.994 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:01:01.023 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:01.044 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:01:01.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:01.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:01:01.148 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:01:01.193 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:01.218 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:01:01.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:01.225 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:01:01.326 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:01:01.381 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:01.474 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:01:01.481 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:01:01.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:01.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:01:01.586 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:01:01.626 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:01.653 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:01:01.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:01.658 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:01:01.759 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:01:01.866 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:02.643 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:01:02.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:02.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:01:02.754 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:01:02.791 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:02.825 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:01:02.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:02.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:01:02.928 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:01:02.977 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:03.008 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:01:03.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:03.015 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:01:03.116 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:01:03.149 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:02 smithi120 ceph-mon[43961]: pgmap v1013: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 687 op/s; 19057/207615 objects degraded (9.179%); 12192/207615 objects misplaced (5.872%); 9.8 MiB/s, 158 objects/s recovering 2024-05-15T13:01:03.151 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:03.174 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:01:03.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:03.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:01:03.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:02 smithi027 ceph-mon[39795]: pgmap v1013: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 687 op/s; 19057/207615 objects degraded (9.179%); 12192/207615 objects misplaced (5.872%); 9.8 MiB/s, 158 objects/s recovering 2024-05-15T13:01:03.280 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:01:03.370 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:03.448 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:02 smithi177 ceph-mon[44303]: pgmap v1013: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 687 op/s; 19057/207615 objects degraded (9.179%); 12192/207615 objects misplaced (5.872%); 9.8 MiB/s, 158 objects/s recovering 2024-05-15T13:01:04.124 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:01:04.148 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:01:04.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:04.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:01:04.152 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:01:04.221 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:01:04.253 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:01:04.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:03 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 19057/207615 objects degraded (9.179%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:04.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:03 smithi027 ceph-mon[39795]: pgmap v1014: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 612 op/s; 19053/205218 objects degraded (9.284%); 11708/205218 objects misplaced (5.705%); 9.8 MiB/s, 158 objects/s recovering 2024-05-15T13:01:04.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:03 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 19057/207615 objects degraded (9.179%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:04.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:03 smithi120 ceph-mon[43961]: pgmap v1014: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 612 op/s; 19053/205218 objects degraded (9.284%); 11708/205218 objects misplaced (5.705%); 9.8 MiB/s, 158 objects/s recovering 2024-05-15T13:01:04.311 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:04.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:03 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 19057/207615 objects degraded (9.179%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:04.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:03 smithi177 ceph-mon[44303]: pgmap v1014: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 612 op/s; 19053/205218 objects degraded (9.284%); 11708/205218 objects misplaced (5.705%); 9.8 MiB/s, 158 objects/s recovering 2024-05-15T13:01:04.434 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:01:04.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:04.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:01:04.538 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:01:04.605 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:05.225 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:01:05.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:05.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:01:05.333 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:01:05.407 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:05.453 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:01:05.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:05.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:01:05.563 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:01:05.617 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:05.649 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:01:05.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:05.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:01:05.757 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:01:05.813 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:05.865 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:01:05.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:05.871 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:01:05.972 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:01:06.033 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:06.198 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:01:06.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:06.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:01:06.306 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:01:06.347 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:06.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:06 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:01:06.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:06 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:01:06.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:06 smithi027 ceph-mon[39795]: osdmap e373: 12 total, 12 up, 11 in 2024-05-15T13:01:06.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:06 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:01:06.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:06 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:01:06.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:06 smithi027 ceph-mon[39795]: pgmap v1016: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 607 op/s; 19053/201039 objects degraded (9.477%); 11477/201039 objects misplaced (5.709%); 11 MiB/s, 175 objects/s recovering 2024-05-15T13:01:06.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:06 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:01:06.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:06 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:01:06.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:06 smithi120 ceph-mon[43961]: osdmap e373: 12 total, 12 up, 11 in 2024-05-15T13:01:06.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:06 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:01:06.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:06 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:01:06.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:06 smithi120 ceph-mon[43961]: pgmap v1016: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 607 op/s; 19053/201039 objects degraded (9.477%); 11477/201039 objects misplaced (5.709%); 11 MiB/s, 175 objects/s recovering 2024-05-15T13:01:06.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:06 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:01:06.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:06 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:01:06.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:06 smithi177 ceph-mon[44303]: osdmap e373: 12 total, 12 up, 11 in 2024-05-15T13:01:06.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:06 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:01:06.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:06 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:01:06.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:06 smithi177 ceph-mon[44303]: pgmap v1016: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 607 op/s; 19053/201039 objects degraded (9.477%); 11477/201039 objects misplaced (5.709%); 11 MiB/s, 175 objects/s recovering 2024-05-15T13:01:07.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:07 smithi027 ceph-mon[39795]: osdmap e374: 12 total, 12 up, 11 in 2024-05-15T13:01:07.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:07 smithi120 ceph-mon[43961]: osdmap e374: 12 total, 12 up, 11 in 2024-05-15T13:01:07.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:07 smithi177 ceph-mon[44303]: osdmap e374: 12 total, 12 up, 11 in 2024-05-15T13:01:08.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:08 smithi027 ceph-mon[39795]: osdmap e375: 12 total, 12 up, 11 in 2024-05-15T13:01:08.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:08 smithi027 ceph-mon[39795]: pgmap v1019: 17 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 560 op/s; 16180/197517 objects degraded (8.192%); 14119/197517 objects misplaced (7.148%); 9.8 MiB/s, 158 objects/s recovering 2024-05-15T13:01:08.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:08 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 19053/201039 objects degraded (9.477%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:08.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:08 smithi120 ceph-mon[43961]: osdmap e375: 12 total, 12 up, 11 in 2024-05-15T13:01:08.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:08 smithi120 ceph-mon[43961]: pgmap v1019: 17 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 560 op/s; 16180/197517 objects degraded (8.192%); 14119/197517 objects misplaced (7.148%); 9.8 MiB/s, 158 objects/s recovering 2024-05-15T13:01:08.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:08 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 19053/201039 objects degraded (9.477%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:08.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:08 smithi177 ceph-mon[44303]: osdmap e375: 12 total, 12 up, 11 in 2024-05-15T13:01:08.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:08 smithi177 ceph-mon[44303]: pgmap v1019: 17 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 10 active+clean; 4.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 560 op/s; 16180/197517 objects degraded (8.192%); 14119/197517 objects misplaced (7.148%); 9.8 MiB/s, 158 objects/s recovering 2024-05-15T13:01:08.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:08 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 19053/201039 objects degraded (9.477%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:09.143 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:01:09.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:09.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:01:09.254 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:01:09.340 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:09.382 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:01:09.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:09.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:01:09.486 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:01:09.530 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:09.555 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:01:09.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:09.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:01:09.659 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:01:09.715 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:09 smithi027 ceph-mon[39795]: osdmap e376: 12 total, 12 up, 11 in 2024-05-15T13:01:09.720 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:09.760 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:01:09.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:09.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:01:09.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:09 smithi120 ceph-mon[43961]: osdmap e376: 12 total, 12 up, 11 in 2024-05-15T13:01:09.869 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:01:09.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:09 smithi177 ceph-mon[44303]: osdmap e376: 12 total, 12 up, 11 in 2024-05-15T13:01:09.962 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:10.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:10 smithi027 ceph-mon[39795]: pgmap v1021: 17 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 11 active+clean; 3.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 640 op/s; 16176/193038 objects degraded (8.380%); 13419/193038 objects misplaced (6.951%); 13 MiB/s, 209 objects/s recovering 2024-05-15T13:01:10.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:10 smithi120 ceph-mon[43961]: pgmap v1021: 17 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 11 active+clean; 3.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 640 op/s; 16176/193038 objects degraded (8.380%); 13419/193038 objects misplaced (6.951%); 13 MiB/s, 209 objects/s recovering 2024-05-15T13:01:10.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:10 smithi177 ceph-mon[44303]: pgmap v1021: 17 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 11 active+clean; 3.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 640 op/s; 16176/193038 objects degraded (8.380%); 13419/193038 objects misplaced (6.951%); 13 MiB/s, 209 objects/s recovering 2024-05-15T13:01:11.740 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:01:11.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:11.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:01:11.851 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:01:11.879 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:11.910 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:01:11.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:11.913 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:01:12.014 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:01:12.064 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:12.093 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:01:12.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:12.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:01:12.206 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:01:12.246 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:12.268 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:01:12.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:12.271 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:01:12.372 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:01:12.403 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:12.432 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:01:12.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:12.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:01:12.540 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:01:12.566 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:01:12.584 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:12.606 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:01:12.606 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:01:12.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:12.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:01:12.649 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:01:12.702 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:01:12.709 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:01:12.734 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:01:12.737 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:12.774 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:01:12.789 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:01:12.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:12.798 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:01:12.801 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:01:12.832 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:01:12.875 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:01:12.899 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:01:12.921 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:01:12.981 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:01:12.981 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:13.000 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:01:13.029 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:01:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:01:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:01:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:01:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:01:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:01:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:01:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:01:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:01:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:01:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:01:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:01:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:01:13.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:12 smithi177 ceph-mon[44303]: pgmap v1022: 17 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 11 active+clean; 3.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 606 op/s; 16173/190119 objects degraded (8.507%); 13134/190119 objects misplaced (6.908%); 13 MiB/s, 203 objects/s recovering 2024-05-15T13:01:13.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:12 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1474069769' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:01:13.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:12 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:01:13.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:12 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3737494121' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:01:13.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:12 smithi027 ceph-mon[39795]: pgmap v1022: 17 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 11 active+clean; 3.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 606 op/s; 16173/190119 objects degraded (8.507%); 13134/190119 objects misplaced (6.908%); 13 MiB/s, 203 objects/s recovering 2024-05-15T13:01:13.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:12 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1474069769' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:01:13.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:12 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:01:13.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:12 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3737494121' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:01:13.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:12 smithi120 ceph-mon[43961]: pgmap v1022: 17 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 11 active+clean; 3.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 606 op/s; 16173/190119 objects degraded (8.507%); 13134/190119 objects misplaced (6.908%); 13 MiB/s, 203 objects/s recovering 2024-05-15T13:01:13.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:12 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1474069769' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:01:13.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:12 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:01:13.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:12 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3737494121' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:01:14.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:13 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:01:14.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:13 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:01:14.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:13 smithi027 ceph-mon[39795]: osdmap e377: 12 total, 12 up, 11 in 2024-05-15T13:01:14.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:13 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 16173/190119 objects degraded (8.507%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:14.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:13 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/967648268' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.f", "id": [1, 7, 3]}]: dispatch 2024-05-15T13:01:14.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:13 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.f", "id": [1, 7, 3]}]: dispatch 2024-05-15T13:01:14.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:13 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:01:14.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:13 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:01:14.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:13 smithi120 ceph-mon[43961]: osdmap e377: 12 total, 12 up, 11 in 2024-05-15T13:01:14.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:13 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 16173/190119 objects degraded (8.507%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:14.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:13 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/967648268' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.f", "id": [1, 7, 3]}]: dispatch 2024-05-15T13:01:14.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:13 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.f", "id": [1, 7, 3]}]: dispatch 2024-05-15T13:01:14.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:13 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:01:14.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:13 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:01:14.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:13 smithi177 ceph-mon[44303]: osdmap e377: 12 total, 12 up, 11 in 2024-05-15T13:01:14.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:13 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 16173/190119 objects degraded (8.507%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:14.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:13 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/967648268' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.f", "id": [1, 7, 3]}]: dispatch 2024-05-15T13:01:14.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:13 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.f", "id": [1, 7, 3]}]: dispatch 2024-05-15T13:01:15.138 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:01:15.138 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:01:15.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:14 smithi027 ceph-mon[39795]: pgmap v1024: 17 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 11 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 525 op/s; 16167/187899 objects degraded (8.604%); 12945/187899 objects misplaced (6.889%); 12 MiB/s, 190 objects/s recovering 2024-05-15T13:01:15.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:14 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.f", "id": [1, 7, 3]}]': finished 2024-05-15T13:01:15.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:14 smithi027 ceph-mon[39795]: osdmap e378: 12 total, 12 up, 11 in 2024-05-15T13:01:15.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:14 smithi120 ceph-mon[43961]: pgmap v1024: 17 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 11 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 525 op/s; 16167/187899 objects degraded (8.604%); 12945/187899 objects misplaced (6.889%); 12 MiB/s, 190 objects/s recovering 2024-05-15T13:01:15.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:14 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.f", "id": [1, 7, 3]}]': finished 2024-05-15T13:01:15.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:14 smithi120 ceph-mon[43961]: osdmap e378: 12 total, 12 up, 11 in 2024-05-15T13:01:15.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:14 smithi177 ceph-mon[44303]: pgmap v1024: 17 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 11 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 525 op/s; 16167/187899 objects degraded (8.604%); 12945/187899 objects misplaced (6.889%); 12 MiB/s, 190 objects/s recovering 2024-05-15T13:01:15.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:14 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.f", "id": [1, 7, 3]}]': finished 2024-05-15T13:01:15.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:14 smithi177 ceph-mon[44303]: osdmap e378: 12 total, 12 up, 11 in 2024-05-15T13:01:15.593 INFO:teuthology.orchestra.run.smithi027.stderr:set 4.f pg_upmap mapping to [1,7,3] 2024-05-15T13:01:15.928 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:01:15.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:15.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:01:16.039 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:01:16.081 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:16.116 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:01:16.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:16.124 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:01:16.225 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:01:16.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:15 smithi027 ceph-mon[39795]: pgmap v1026: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 620 op/s; 16167/183717 objects degraded (8.800%); 12662/183717 objects misplaced (6.892%); 12 MiB/s, 198 objects/s recovering 2024-05-15T13:01:16.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:15 smithi120 ceph-mon[43961]: pgmap v1026: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 620 op/s; 16167/183717 objects degraded (8.800%); 12662/183717 objects misplaced (6.892%); 12 MiB/s, 198 objects/s recovering 2024-05-15T13:01:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:16.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:15 smithi177 ceph-mon[44303]: pgmap v1026: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 620 op/s; 16167/183717 objects degraded (8.800%); 12662/183717 objects misplaced (6.892%); 12 MiB/s, 198 objects/s recovering 2024-05-15T13:01:17.167 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:01:17.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:17.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:01:17.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:17 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:01:17.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:17 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:01:17.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:17 smithi027 ceph-mon[39795]: osdmap e379: 12 total, 12 up, 11 in 2024-05-15T13:01:17.276 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:01:17.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:17 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:01:17.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:17 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:01:17.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:17 smithi120 ceph-mon[43961]: osdmap e379: 12 total, 12 up, 11 in 2024-05-15T13:01:17.325 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:17.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:17 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:01:17.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:17 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:01:17.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:17 smithi177 ceph-mon[44303]: osdmap e379: 12 total, 12 up, 11 in 2024-05-15T13:01:17.597 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:01:17.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:17.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:01:17.630 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-05-15T13:01:17.715 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:01:17.745 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:17.774 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:01:17.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:17.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:01:17.878 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:01:17.914 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:17.962 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:01:17.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:17.975 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:01:17.995 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:01:18.009 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:18 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:01:18.009 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:18 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:01:18.009 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:18 smithi027 ceph-mon[39795]: osdmap e380: 12 total, 12 up, 11 in 2024-05-15T13:01:18.009 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:18 smithi027 ceph-mon[39795]: pgmap v1029: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 630 op/s; 16166/180309 objects degraded (8.966%); 12248/180309 objects misplaced (6.793%); 8.8 MiB/s, 142 objects/s recovering 2024-05-15T13:01:18.010 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:18 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:01:18.010 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:18 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:01:18.010 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:18 smithi120 ceph-mon[43961]: osdmap e380: 12 total, 12 up, 11 in 2024-05-15T13:01:18.010 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:18 smithi120 ceph-mon[43961]: pgmap v1029: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 630 op/s; 16166/180309 objects degraded (8.966%); 12248/180309 objects misplaced (6.793%); 8.8 MiB/s, 142 objects/s recovering 2024-05-15T13:01:18.076 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:01:18.160 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:18.177 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:01:18.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:18 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:01:18.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:18 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:01:18.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:18 smithi177 ceph-mon[44303]: osdmap e380: 12 total, 12 up, 11 in 2024-05-15T13:01:18.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:18 smithi177 ceph-mon[44303]: pgmap v1029: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 630 op/s; 16166/180309 objects degraded (8.966%); 12248/180309 objects misplaced (6.793%); 8.8 MiB/s, 142 objects/s recovering 2024-05-15T13:01:18.469 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:01:18.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:18.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:01:18.576 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:01:18.625 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:18.678 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:01:18.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:18.681 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:01:18.782 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:01:18.839 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:19.045 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:19 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 16166/180309 objects degraded (8.966%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:19.107 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:01:19.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:19.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:01:19.146 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:01:19.207 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:01:19.215 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:01:19.264 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:19.307 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:01:19.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:19.314 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:01:19.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:19 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 16166/180309 objects degraded (8.966%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:19.415 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:01:19.476 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:19.502 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:01:19.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:19.506 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:01:19.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:19 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 16166/180309 objects degraded (8.966%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:19.607 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:01:19.647 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:19.688 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:01:19.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:19.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:01:19.791 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:01:19.827 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:19.854 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:01:19.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:19.863 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:01:19.964 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:01:20.017 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:20.049 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:01:20.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:20.057 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:01:20.158 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:01:20.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:20 smithi177 ceph-mon[44303]: pgmap v1030: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 714 op/s; 16160/175023 objects degraded (9.233%); 11906/175023 objects misplaced (6.803%); 11 MiB/s, 174 objects/s recovering 2024-05-15T13:01:20.194 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:20.216 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:01:20.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:20.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:01:20.319 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:01:20.355 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:20 smithi120 ceph-mon[43961]: pgmap v1030: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 714 op/s; 16160/175023 objects degraded (9.233%); 11906/175023 objects misplaced (6.803%); 11 MiB/s, 174 objects/s recovering 2024-05-15T13:01:20.360 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:20.394 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:01:20.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:20.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:01:20.428 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:20 smithi027 ceph-mon[39795]: pgmap v1030: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 714 op/s; 16160/175023 objects degraded (9.233%); 11906/175023 objects misplaced (6.803%); 11 MiB/s, 174 objects/s recovering 2024-05-15T13:01:20.498 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:01:20.580 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:20.943 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:01:20.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:20.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:01:21.056 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:01:21.090 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:21.282 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:01:21.299 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:01:21.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:21.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:01:21.412 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:01:21.463 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:21.486 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:01:21.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:21.490 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:01:21.591 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:01:21.642 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:21.751 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:01:21.752 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:01:21.752 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:01:21.874 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:01:21.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:21.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:01:21.980 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:01:22.010 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:22.039 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:01:22.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:22.046 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:01:22.147 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:01:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:22.689 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:01:22.689 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:01:22.953 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:22 smithi027 ceph-mon[39795]: pgmap v1031: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 666 op/s; 16159/171396 objects degraded (9.428%); 11678/171396 objects misplaced (6.813%); 8.0 MiB/s, 130 objects/s recovering 2024-05-15T13:01:22.954 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:22 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2012719497' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:01:22.977 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:01:23.025 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:01:23.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:23.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:01:23.135 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:01:23.176 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:23.203 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:01:23.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:23.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:01:23.216 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:22 smithi120 ceph-mon[43961]: pgmap v1031: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 666 op/s; 16159/171396 objects degraded (9.428%); 11678/171396 objects misplaced (6.813%); 8.0 MiB/s, 130 objects/s recovering 2024-05-15T13:01:23.216 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:22 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2012719497' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:01:23.317 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:01:23.370 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:23.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:22 smithi177 ceph-mon[44303]: pgmap v1031: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 666 op/s; 16159/171396 objects degraded (9.428%); 11678/171396 objects misplaced (6.813%); 8.0 MiB/s, 130 objects/s recovering 2024-05-15T13:01:23.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:22 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2012719497' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:01:24.151 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:01:24.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:24.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:01:24.160 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-05-15T13:01:24.226 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:23 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:01:24.226 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:23 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2012719497' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:01:24.226 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:23 smithi027 ceph-mon[39795]: osdmap e381: 12 total, 12 up, 11 in 2024-05-15T13:01:24.226 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:23 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 16159/171396 objects degraded (9.428%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:24.256 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:01:24.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:23 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:01:24.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:23 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2012719497' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:01:24.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:23 smithi120 ceph-mon[43961]: osdmap e381: 12 total, 12 up, 11 in 2024-05-15T13:01:24.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:23 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 16159/171396 objects degraded (9.428%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:24.281 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:23 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:01:24.281 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:23 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2012719497' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:01:24.281 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:23 smithi177 ceph-mon[44303]: osdmap e381: 12 total, 12 up, 11 in 2024-05-15T13:01:24.281 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:23 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 16159/171396 objects degraded (9.428%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:24.283 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:24.303 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:01:24.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:24.306 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:01:24.408 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:01:24.435 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:24.458 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:01:24.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:24.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:01:24.563 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:01:24.596 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:24.624 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:01:24.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:24.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:01:24.731 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:01:24.794 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:24.951 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:24 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:01:24.951 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:24 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:01:25.070 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:01:25.130 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:01:25.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:24 smithi027 ceph-mon[39795]: pgmap v1033: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 624 op/s; 16177/168927 objects degraded (9.576%); 11485/168927 objects misplaced (6.799%); 9.1 MiB/s, 147 objects/s recovering 2024-05-15T13:01:25.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:24 smithi027 ceph-mon[39795]: osdmap e382: 12 total, 12 up, 11 in 2024-05-15T13:01:25.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:24 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:01:25.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:24 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:01:25.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:24 smithi120 ceph-mon[43961]: pgmap v1033: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 624 op/s; 16177/168927 objects degraded (9.576%); 11485/168927 objects misplaced (6.799%); 9.1 MiB/s, 147 objects/s recovering 2024-05-15T13:01:25.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:24 smithi120 ceph-mon[43961]: osdmap e382: 12 total, 12 up, 11 in 2024-05-15T13:01:25.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:24 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:01:25.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:24 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:01:25.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:24 smithi177 ceph-mon[44303]: pgmap v1033: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 624 op/s; 16177/168927 objects degraded (9.576%); 11485/168927 objects misplaced (6.799%); 9.1 MiB/s, 147 objects/s recovering 2024-05-15T13:01:25.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:24 smithi177 ceph-mon[44303]: osdmap e382: 12 total, 12 up, 11 in 2024-05-15T13:01:26.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:26 smithi027 ceph-mon[39795]: pgmap v1035: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 16177/164973 objects degraded (9.806%); 11140/164973 objects misplaced (6.753%); 8.5 MiB/s, 137 objects/s recovering 2024-05-15T13:01:26.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:26 smithi120 ceph-mon[43961]: pgmap v1035: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 16177/164973 objects degraded (9.806%); 11140/164973 objects misplaced (6.753%); 8.5 MiB/s, 137 objects/s recovering 2024-05-15T13:01:26.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:26 smithi177 ceph-mon[44303]: pgmap v1035: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 16177/164973 objects degraded (9.806%); 11140/164973 objects misplaced (6.753%); 8.5 MiB/s, 137 objects/s recovering 2024-05-15T13:01:26.624 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:01:27.973 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:01:27.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:27.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:01:28.078 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:01:28.195 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:28.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:28 smithi027 ceph-mon[39795]: pgmap v1036: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.3 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 555 op/s; 16174/161682 objects degraded (10.004%); 10663/161682 objects misplaced (6.595%); 9.6 MiB/s, 153 objects/s recovering 2024-05-15T13:01:28.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:28 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1950819910' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:01:28.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:28 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 16174/161682 objects degraded (10.004%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:29.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:28 smithi120 ceph-mon[43961]: pgmap v1036: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.3 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 555 op/s; 16174/161682 objects degraded (10.004%); 10663/161682 objects misplaced (6.595%); 9.6 MiB/s, 153 objects/s recovering 2024-05-15T13:01:29.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:28 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1950819910' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:01:29.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:28 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 16174/161682 objects degraded (10.004%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:29.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:28 smithi177 ceph-mon[44303]: pgmap v1036: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.3 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 555 op/s; 16174/161682 objects degraded (10.004%); 10663/161682 objects misplaced (6.595%); 9.6 MiB/s, 153 objects/s recovering 2024-05-15T13:01:29.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:28 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1950819910' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:01:29.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:28 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 16174/161682 objects degraded (10.004%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:30.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:30 smithi120 ceph-mon[43961]: pgmap v1037: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 635 op/s; 16173/156150 objects degraded (10.357%); 10293/156150 objects misplaced (6.592%); 11 MiB/s, 171 objects/s recovering 2024-05-15T13:01:30.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:30 smithi177 ceph-mon[44303]: pgmap v1037: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 635 op/s; 16173/156150 objects degraded (10.357%); 10293/156150 objects misplaced (6.592%); 11 MiB/s, 171 objects/s recovering 2024-05-15T13:01:30.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:30 smithi027 ceph-mon[39795]: pgmap v1037: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 635 op/s; 16173/156150 objects degraded (10.357%); 10293/156150 objects misplaced (6.592%); 11 MiB/s, 171 objects/s recovering 2024-05-15T13:01:32.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:32 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:01:32.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:32 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:01:32.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:32 smithi120 ceph-mon[43961]: osdmap e383: 12 total, 12 up, 11 in 2024-05-15T13:01:32.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:32 smithi120 ceph-mon[43961]: pgmap v1039: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 692 op/s; 16173/152310 objects degraded (10.618%); 10057/152310 objects misplaced (6.603%); 11 MiB/s, 178 objects/s recovering 2024-05-15T13:01:32.386 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:01:32.387 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:01:32.387 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:01:32.387 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:01:32.387 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:01:32.387 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:01:32.387 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:01:32.387 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:01:32.387 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:01:32.387 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:01:32.387 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:01:32.387 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:01:32.388 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:01:32.388 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:01:32.388 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:01:32.388 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:01:32.388 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:01:32.388 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:01:32.388 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:01:32.388 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:01:32.388 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:01:32.388 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:01:32.388 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:01:32.388 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:01:32.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:32 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:01:32.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:32 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:01:32.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:32 smithi177 ceph-mon[44303]: osdmap e383: 12 total, 12 up, 11 in 2024-05-15T13:01:32.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:32 smithi177 ceph-mon[44303]: pgmap v1039: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 692 op/s; 16173/152310 objects degraded (10.618%); 10057/152310 objects misplaced (6.603%); 11 MiB/s, 178 objects/s recovering 2024-05-15T13:01:32.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:32 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:01:32.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:32 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:01:32.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:32 smithi027 ceph-mon[39795]: osdmap e383: 12 total, 12 up, 11 in 2024-05-15T13:01:32.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:32 smithi027 ceph-mon[39795]: pgmap v1039: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 692 op/s; 16173/152310 objects degraded (10.618%); 10057/152310 objects misplaced (6.603%); 11 MiB/s, 178 objects/s recovering 2024-05-15T13:01:33.095 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:01:33.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:33.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:01:33.203 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:01:33.268 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:33.647 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:01:33.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:33.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:01:33.754 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:01:33.790 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:34.144 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:01:34.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:34.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:01:34.189 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:01:34.189 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T13:01:34.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T13:01:34.189 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:01:34.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.189 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:101362 4.f 4:fe841796:::benchmark_data_smithi177_99756_object101361:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-05-15T13:01:34.189 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:59:10.303295+0000", 2024-05-15T13:01:34.189 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 143.89196694200001, 2024-05-15T13:01:34.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5258583200000002, 2024-05-15T13:01:34.189 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:01:34.190 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:01:34.190 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:01:34.190 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:01:34.190 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:01:34.190 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 101362 2024-05-15T13:01:34.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.190 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:01:34.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:01:34.190 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.303295+0000", 2024-05-15T13:01:34.190 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.191 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:01:34.191 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.303295+0000", 2024-05-15T13:01:34.191 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.191 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:01:34.191 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.303296+0000", 2024-05-15T13:01:34.191 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5700000000000002e-07 2024-05-15T13:01:34.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.191 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:01:34.192 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.303322+0000", 2024-05-15T13:01:34.192 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6444e-05 2024-05-15T13:01:34.192 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.192 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.192 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:01:34.192 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.303355+0000", 2024-05-15T13:01:34.192 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2477000000000001e-05 2024-05-15T13:01:34.192 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.192 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.192 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:01:34.192 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.303361+0000", 2024-05-15T13:01:34.193 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5169999999999996e-06 2024-05-15T13:01:34.193 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.193 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.193 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.193 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.306874+0000", 2024-05-15T13:01:34.193 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035131110000000002 2024-05-15T13:01:34.193 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.193 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.193 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:01:34.193 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.307032+0000", 2024-05-15T13:01:34.193 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000157598 2024-05-15T13:01:34.193 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.193 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.194 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T13:01:34.194 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.307211+0000", 2024-05-15T13:01:34.194 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017897200000000001 2024-05-15T13:01:34.194 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.194 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.194 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.194 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.322151+0000", 2024-05-15T13:01:34.194 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014939737 2024-05-15T13:01:34.194 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.194 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.194 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:01:34.194 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.322160+0000", 2024-05-15T13:01:34.194 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7510000000000003e-06 2024-05-15T13:01:34.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.195 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.195 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:11.313271+0000", 2024-05-15T13:01:34.195 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99111032700000001 2024-05-15T13:01:34.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.195 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:01:34.195 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:11.313284+0000", 2024-05-15T13:01:34.195 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3580999999999999e-05 2024-05-15T13:01:34.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.195 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.196 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:12.829085+0000", 2024-05-15T13:01:34.196 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5158001489999999 2024-05-15T13:01:34.196 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.196 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.196 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:01:34.196 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:12.829154+0000", 2024-05-15T13:01:34.196 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9098999999999997e-05 2024-05-15T13:01:34.196 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.196 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:01:34.196 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.196 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.196 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.196 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:101364 4.f 4:fa04d9c8:::benchmark_data_smithi177_99756_object101363:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-05-15T13:01:34.196 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:59:10.303739+0000", 2024-05-15T13:01:34.196 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 143.891523363, 2024-05-15T13:01:34.197 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5255353270000001, 2024-05-15T13:01:34.197 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:01:34.197 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:01:34.197 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:01:34.197 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:01:34.197 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:01:34.197 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 101364 2024-05-15T13:01:34.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.197 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:01:34.197 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.197 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:01:34.197 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.303739+0000", 2024-05-15T13:01:34.197 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.198 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.198 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:01:34.198 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.303739+0000", 2024-05-15T13:01:34.198 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.198 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.198 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.198 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:01:34.198 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.303740+0000", 2024-05-15T13:01:34.198 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.23e-07 2024-05-15T13:01:34.198 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.198 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.198 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:01:34.198 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.303761+0000", 2024-05-15T13:01:34.199 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0962999999999999e-05 2024-05-15T13:01:34.199 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.199 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.199 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:01:34.199 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.303769+0000", 2024-05-15T13:01:34.199 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0870000000000007e-06 2024-05-15T13:01:34.199 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.199 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.199 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:01:34.199 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.303777+0000", 2024-05-15T13:01:34.199 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0749999999999998e-06 2024-05-15T13:01:34.199 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.199 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.199 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.200 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.310515+0000", 2024-05-15T13:01:34.200 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0067384200000000002 2024-05-15T13:01:34.200 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.200 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.200 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:01:34.200 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.310680+0000", 2024-05-15T13:01:34.200 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000165057 2024-05-15T13:01:34.200 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.200 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.200 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T13:01:34.200 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.310842+0000", 2024-05-15T13:01:34.200 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016158700000000001 2024-05-15T13:01:34.200 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.201 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.201 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.201 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.322192+0000", 2024-05-15T13:01:34.201 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011350179 2024-05-15T13:01:34.201 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.201 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.201 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:01:34.201 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.322200+0000", 2024-05-15T13:01:34.201 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.276e-06 2024-05-15T13:01:34.201 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.201 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.201 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.201 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:11.313329+0000", 2024-05-15T13:01:34.201 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99112866399999999 2024-05-15T13:01:34.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.202 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.202 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:01:34.202 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:11.313338+0000", 2024-05-15T13:01:34.202 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.55e-06 2024-05-15T13:01:34.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.202 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.202 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.202 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:12.829209+0000", 2024-05-15T13:01:34.202 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5158704589999998 2024-05-15T13:01:34.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.202 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.202 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:01:34.203 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:12.829274+0000", 2024-05-15T13:01:34.203 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5287e-05 2024-05-15T13:01:34.203 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.203 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:01:34.203 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.203 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.203 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.203 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:140795 4.9 4:9e5e757e:::benchmark_data_smithi177_99756_object20823:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e357)", 2024-05-15T13:01:34.203 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:11.337401+0000", 2024-05-15T13:01:34.203 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 82.857861181000004, 2024-05-15T13:01:34.203 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.256904681, 2024-05-15T13:01:34.203 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:01:34.204 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:01:34.204 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:01:34.204 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:01:34.204 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:01:34.204 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 140795 2024-05-15T13:01:34.204 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.204 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:01:34.204 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.204 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:01:34.204 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:11.337401+0000", 2024-05-15T13:01:34.204 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.204 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.204 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.205 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:01:34.205 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:11.337401+0000", 2024-05-15T13:01:34.205 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.205 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.205 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.205 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:01:34.205 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:11.337402+0000", 2024-05-15T13:01:34.205 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0800000000000005e-07 2024-05-15T13:01:34.205 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.205 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.205 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:01:34.205 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:11.337404+0000", 2024-05-15T13:01:34.205 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.323e-06 2024-05-15T13:01:34.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.206 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.206 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:01:34.206 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:11.337410+0000", 2024-05-15T13:01:34.206 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1099999999999999e-06 2024-05-15T13:01:34.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.206 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.206 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:01:34.206 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:11.337415+0000", 2024-05-15T13:01:34.206 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.904e-06 2024-05-15T13:01:34.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.206 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.206 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.206 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:11.337653+0000", 2024-05-15T13:01:34.207 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00023781900000000001 2024-05-15T13:01:34.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.207 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.207 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:01:34.207 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:11.337729+0000", 2024-05-15T13:01:34.207 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6136000000000004e-05 2024-05-15T13:01:34.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.207 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.207 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.207 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:14.594289+0000", 2024-05-15T13:01:34.207 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2565602500000002 2024-05-15T13:01:34.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.207 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.208 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:01:34.208 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:14.594306+0000", 2024-05-15T13:01:34.208 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6631000000000001e-05 2024-05-15T13:01:34.208 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.208 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:01:34.208 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.208 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.208 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.208 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:142000 4.9 4:9e3e0cd7:::benchmark_data_smithi177_99756_object22028:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-05-15T13:01:34.208 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:13.180376+0000", 2024-05-15T13:01:34.208 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 81.014886271999998, 2024-05-15T13:01:34.208 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.170883333, 2024-05-15T13:01:34.208 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:01:34.208 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:01:34.209 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:01:34.209 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:01:34.209 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:01:34.209 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 142000 2024-05-15T13:01:34.209 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.209 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:01:34.209 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.209 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:01:34.209 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.180376+0000", 2024-05-15T13:01:34.209 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.209 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.209 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.209 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:01:34.209 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.180376+0000", 2024-05-15T13:01:34.210 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.210 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.210 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.210 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:01:34.210 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.180377+0000", 2024-05-15T13:01:34.210 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.491e-06 2024-05-15T13:01:34.210 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.210 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.210 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:01:34.210 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.180385+0000", 2024-05-15T13:01:34.210 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.858e-06 2024-05-15T13:01:34.210 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.211 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.211 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:01:34.211 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.180399+0000", 2024-05-15T13:01:34.211 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3794e-05 2024-05-15T13:01:34.211 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.211 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.211 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:01:34.211 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.180407+0000", 2024-05-15T13:01:34.211 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1540000000000002e-06 2024-05-15T13:01:34.211 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.211 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.211 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.211 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.180482+0000", 2024-05-15T13:01:34.211 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4431999999999995e-05 2024-05-15T13:01:34.212 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.212 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.212 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:01:34.212 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.180691+0000", 2024-05-15T13:01:34.212 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00020957500000000001 2024-05-15T13:01:34.212 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.212 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.212 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,11", 2024-05-15T13:01:34.212 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.180883+0000", 2024-05-15T13:01:34.212 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000191929 2024-05-15T13:01:34.212 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.212 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.212 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:01:34.212 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.184559+0000", 2024-05-15T13:01:34.212 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003676141 2024-05-15T13:01:34.213 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.213 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.213 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:01:34.213 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.205170+0000", 2024-05-15T13:01:34.213 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.020610621999999999 2024-05-15T13:01:34.213 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.213 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.213 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:01:34.213 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346600+0000", 2024-05-15T13:01:34.213 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.14142964099999999 2024-05-15T13:01:34.213 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.213 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.213 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:01:34.213 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346671+0000", 2024-05-15T13:01:34.213 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1037000000000004e-05 2024-05-15T13:01:34.214 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.214 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.214 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:01:34.214 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.351259+0000", 2024-05-15T13:01:34.214 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0045886590000000001 2024-05-15T13:01:34.214 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.214 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:01:34.214 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.214 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.214 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.214 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:142022 4.f 4:fbe69b59:::benchmark_data_smithi177_99756_object22050:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-05-15T13:01:34.214 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:13.218096+0000", 2024-05-15T13:01:34.214 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 80.977166392000001, 2024-05-15T13:01:34.215 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.15891720200000001, 2024-05-15T13:01:34.215 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:01:34.215 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:01:34.215 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:01:34.215 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:01:34.215 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:01:34.215 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 142022 2024-05-15T13:01:34.215 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.215 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:01:34.215 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.215 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:01:34.215 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.218096+0000", 2024-05-15T13:01:34.215 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.215 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.216 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.216 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:01:34.216 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.218096+0000", 2024-05-15T13:01:34.216 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.216 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.216 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.216 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:01:34.216 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.218096+0000", 2024-05-15T13:01:34.216 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0499999999999999e-07 2024-05-15T13:01:34.216 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.216 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.216 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:01:34.216 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.218099+0000", 2024-05-15T13:01:34.216 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0359999999999998e-06 2024-05-15T13:01:34.216 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.217 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.217 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:01:34.217 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.218107+0000", 2024-05-15T13:01:34.217 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8699999999999992e-06 2024-05-15T13:01:34.217 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.217 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.217 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:01:34.217 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.218119+0000", 2024-05-15T13:01:34.217 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1513e-05 2024-05-15T13:01:34.217 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.217 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.217 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.217 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.223037+0000", 2024-05-15T13:01:34.217 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0049181119999999997 2024-05-15T13:01:34.218 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.218 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.218 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:01:34.218 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.223149+0000", 2024-05-15T13:01:34.218 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011206500000000001 2024-05-15T13:01:34.218 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.218 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.218 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,11", 2024-05-15T13:01:34.218 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.223247+0000", 2024-05-15T13:01:34.218 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7854000000000002e-05 2024-05-15T13:01:34.218 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.218 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.218 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:01:34.219 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.244786+0000", 2024-05-15T13:01:34.219 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.021538838000000001 2024-05-15T13:01:34.219 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.219 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.219 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:01:34.219 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.244900+0000", 2024-05-15T13:01:34.219 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000114337 2024-05-15T13:01:34.219 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.219 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.219 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:01:34.219 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.374306+0000", 2024-05-15T13:01:34.219 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12940575000000001 2024-05-15T13:01:34.219 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.220 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.220 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:01:34.220 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.374349+0000", 2024-05-15T13:01:34.220 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3062999999999997e-05 2024-05-15T13:01:34.220 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.220 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.220 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:01:34.220 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.377013+0000", 2024-05-15T13:01:34.220 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0026643589999999998 2024-05-15T13:01:34.220 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.220 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:01:34.220 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.220 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.220 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.220 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:142023 4.f 4:f4e2aa72:::benchmark_data_smithi177_99756_object22051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-05-15T13:01:34.221 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:13.218132+0000", 2024-05-15T13:01:34.221 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 80.977130637000002, 2024-05-15T13:01:34.221 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16180884500000001, 2024-05-15T13:01:34.221 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:01:34.221 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:01:34.221 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:01:34.221 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:01:34.221 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:01:34.221 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 142023 2024-05-15T13:01:34.221 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.221 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:01:34.221 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.221 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:01:34.221 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.218132+0000", 2024-05-15T13:01:34.221 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.222 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.222 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.222 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:01:34.222 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.218132+0000", 2024-05-15T13:01:34.222 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.222 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.222 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.222 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:01:34.222 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.218132+0000", 2024-05-15T13:01:34.222 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0499999999999999e-07 2024-05-15T13:01:34.222 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.222 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.222 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:01:34.223 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.218135+0000", 2024-05-15T13:01:34.223 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6460000000000002e-06 2024-05-15T13:01:34.223 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.223 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.223 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:01:34.223 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.218140+0000", 2024-05-15T13:01:34.223 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6659999999999996e-06 2024-05-15T13:01:34.223 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.223 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.223 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:01:34.223 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.218147+0000", 2024-05-15T13:01:34.223 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.691e-06 2024-05-15T13:01:34.223 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.223 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.224 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.224 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.226760+0000", 2024-05-15T13:01:34.224 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0086132550000000002 2024-05-15T13:01:34.224 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.224 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.224 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:01:34.224 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.226927+0000", 2024-05-15T13:01:34.224 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016701699999999999 2024-05-15T13:01:34.224 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.224 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.224 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,11", 2024-05-15T13:01:34.224 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.227089+0000", 2024-05-15T13:01:34.224 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000161903 2024-05-15T13:01:34.224 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.225 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.225 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:01:34.225 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.244791+0000", 2024-05-15T13:01:34.225 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.017701627000000001 2024-05-15T13:01:34.225 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.225 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.225 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:01:34.225 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.244940+0000", 2024-05-15T13:01:34.225 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014946499999999999 2024-05-15T13:01:34.225 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.225 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.225 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:01:34.225 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.377102+0000", 2024-05-15T13:01:34.225 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.132161797 2024-05-15T13:01:34.226 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.226 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.226 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:01:34.226 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.377179+0000", 2024-05-15T13:01:34.226 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6489000000000006e-05 2024-05-15T13:01:34.226 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.226 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.226 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:01:34.226 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.379940+0000", 2024-05-15T13:01:34.226 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002761884 2024-05-15T13:01:34.226 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.226 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:01:34.226 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.226 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.227 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.227 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:144471 4.9 4:980f9544:::benchmark_data_smithi177_99756_object24499:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e358)", 2024-05-15T13:01:34.227 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:17.176806+0000", 2024-05-15T13:01:34.227 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 77.018456431999994, 2024-05-15T13:01:34.227 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.56759795199999996, 2024-05-15T13:01:34.227 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:01:34.227 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:01:34.227 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:01:34.227 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:01:34.227 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:01:34.227 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 144471 2024-05-15T13:01:34.227 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.227 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:01:34.227 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.228 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:01:34.228 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.176806+0000", 2024-05-15T13:01:34.228 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.228 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.228 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.228 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:01:34.228 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.176806+0000", 2024-05-15T13:01:34.228 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.228 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.228 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.228 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:01:34.228 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.176807+0000", 2024-05-15T13:01:34.228 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1999999999999999e-06 2024-05-15T13:01:34.228 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.228 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.229 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:01:34.229 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.176812+0000", 2024-05-15T13:01:34.229 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8919999999999999e-06 2024-05-15T13:01:34.229 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.229 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.229 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:01:34.229 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.176820+0000", 2024-05-15T13:01:34.229 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5720000000000003e-06 2024-05-15T13:01:34.229 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.229 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.229 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:01:34.229 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.176827+0000", 2024-05-15T13:01:34.229 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.832e-06 2024-05-15T13:01:34.229 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.230 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.230 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.230 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.176947+0000", 2024-05-15T13:01:34.230 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000119704 2024-05-15T13:01:34.230 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.230 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.230 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:01:34.230 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.177031+0000", 2024-05-15T13:01:34.230 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3758999999999995e-05 2024-05-15T13:01:34.230 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.230 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.230 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.230 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.744391+0000", 2024-05-15T13:01:34.230 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.56736049499999996 2024-05-15T13:01:34.231 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.231 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.231 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:01:34.231 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.744404+0000", 2024-05-15T13:01:34.231 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2498e-05 2024-05-15T13:01:34.231 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.231 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:01:34.231 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.231 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.231 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.231 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:145134 4.9 4:9e0839ad:::benchmark_data_smithi177_99756_object25162:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e359)", 2024-05-15T13:01:34.231 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:18.574293+0000", 2024-05-15T13:01:34.231 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 75.620968754000003, 2024-05-15T13:01:34.231 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9424836380000001, 2024-05-15T13:01:34.232 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:01:34.232 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:01:34.232 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:01:34.232 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:01:34.232 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:01:34.232 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 145134 2024-05-15T13:01:34.232 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.232 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:01:34.232 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.232 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:01:34.232 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574293+0000", 2024-05-15T13:01:34.232 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.232 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.232 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.233 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:01:34.233 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574293+0000", 2024-05-15T13:01:34.233 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.233 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.233 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.233 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:01:34.233 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574295+0000", 2024-05-15T13:01:34.233 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7799999999999999e-06 2024-05-15T13:01:34.233 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.233 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.233 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:01:34.233 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574301+0000", 2024-05-15T13:01:34.233 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8610000000000003e-06 2024-05-15T13:01:34.233 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.234 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.234 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:01:34.234 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574312+0000", 2024-05-15T13:01:34.234 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0468999999999999e-05 2024-05-15T13:01:34.234 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.234 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.234 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:01:34.234 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574320+0000", 2024-05-15T13:01:34.234 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6230000000000003e-06 2024-05-15T13:01:34.234 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.234 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.234 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.234 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574539+0000", 2024-05-15T13:01:34.234 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00021846100000000001 2024-05-15T13:01:34.235 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.235 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.235 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:01:34.235 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574622+0000", 2024-05-15T13:01:34.235 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3628000000000006e-05 2024-05-15T13:01:34.235 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.235 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.235 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.235 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.516765+0000", 2024-05-15T13:01:34.235 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9421424389999999 2024-05-15T13:01:34.235 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.235 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.235 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:01:34.235 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.516777+0000", 2024-05-15T13:01:34.236 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2377e-05 2024-05-15T13:01:34.236 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.236 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:01:34.236 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.236 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.236 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.236 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:145137 4.9 4:909fd2a8:::benchmark_data_smithi177_99756_object25165:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e359)", 2024-05-15T13:01:34.236 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:18.574342+0000", 2024-05-15T13:01:34.236 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 75.620919921999999, 2024-05-15T13:01:34.236 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.951285672, 2024-05-15T13:01:34.236 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:01:34.236 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:01:34.236 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:01:34.236 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:01:34.237 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:01:34.237 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 145137 2024-05-15T13:01:34.237 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.237 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:01:34.237 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.237 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:01:34.237 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574342+0000", 2024-05-15T13:01:34.237 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.237 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.237 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.237 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:01:34.237 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574342+0000", 2024-05-15T13:01:34.237 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.237 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.237 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.238 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:01:34.238 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574343+0000", 2024-05-15T13:01:34.238 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4900000000000001e-07 2024-05-15T13:01:34.238 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.238 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.238 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:01:34.238 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574346+0000", 2024-05-15T13:01:34.238 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7980000000000001e-06 2024-05-15T13:01:34.238 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.238 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.238 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:01:34.238 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574352+0000", 2024-05-15T13:01:34.239 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.601e-06 2024-05-15T13:01:34.239 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.239 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.239 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:01:34.239 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574357+0000", 2024-05-15T13:01:34.239 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2120000000000002e-06 2024-05-15T13:01:34.239 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.239 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.239 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.239 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574652+0000", 2024-05-15T13:01:34.239 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00029439100000000002 2024-05-15T13:01:34.239 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.239 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.239 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:01:34.240 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574724+0000", 2024-05-15T13:01:34.240 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2488000000000001e-05 2024-05-15T13:01:34.240 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.240 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.240 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.240 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.525612+0000", 2024-05-15T13:01:34.240 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.950888199 2024-05-15T13:01:34.240 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.240 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.240 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:01:34.240 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.525628+0000", 2024-05-15T13:01:34.240 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5534000000000001e-05 2024-05-15T13:01:34.240 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.240 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:01:34.240 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.241 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.241 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:146277 4.9 4:9e2047c8:::benchmark_data_smithi177_99756_object26305:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e359)", 2024-05-15T13:01:34.241 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:20.495780+0000", 2024-05-15T13:01:34.241 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 73.699481977000005, 2024-05-15T13:01:34.241 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5112627170000001, 2024-05-15T13:01:34.241 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:01:34.241 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:01:34.241 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:01:34.241 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:01:34.241 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:01:34.241 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 146277 2024-05-15T13:01:34.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.241 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:01:34.242 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.242 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:01:34.242 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.495780+0000", 2024-05-15T13:01:34.242 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.242 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.242 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.242 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:01:34.242 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.495780+0000", 2024-05-15T13:01:34.242 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.242 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.242 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.242 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:01:34.242 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.495780+0000", 2024-05-15T13:01:34.242 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1400000000000001e-07 2024-05-15T13:01:34.243 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.243 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.243 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:01:34.243 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.495782+0000", 2024-05-15T13:01:34.243 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4219999999999999e-06 2024-05-15T13:01:34.243 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.243 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.243 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:01:34.243 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.495785+0000", 2024-05-15T13:01:34.243 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1690000000000001e-06 2024-05-15T13:01:34.243 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.243 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.243 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:01:34.243 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.495788+0000", 2024-05-15T13:01:34.243 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.723e-06 2024-05-15T13:01:34.244 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.244 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.244 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.244 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.496052+0000", 2024-05-15T13:01:34.244 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00026399799999999999 2024-05-15T13:01:34.244 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.244 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.244 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:01:34.244 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.496136+0000", 2024-05-15T13:01:34.244 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4623000000000006e-05 2024-05-15T13:01:34.244 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.244 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.244 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.244 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:23.007030+0000", 2024-05-15T13:01:34.244 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.510893636 2024-05-15T13:01:34.245 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.245 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.245 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:01:34.245 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:23.007043+0000", 2024-05-15T13:01:34.245 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2931999999999999e-05 2024-05-15T13:01:34.245 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.245 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:01:34.245 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.245 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.245 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.245 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:166038 4.9 4:9c1e7859:::benchmark_data_smithi177_99756_object46066:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e369)", 2024-05-15T13:01:34.245 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:52.276137+0000", 2024-05-15T13:01:34.245 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 41.919125518999998, 2024-05-15T13:01:34.245 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8569116879999998, 2024-05-15T13:01:34.245 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:01:34.246 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:01:34.246 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:01:34.246 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:01:34.246 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:01:34.246 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 166038 2024-05-15T13:01:34.246 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.246 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:01:34.246 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.246 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:01:34.246 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:52.276137+0000", 2024-05-15T13:01:34.246 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.246 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.246 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.246 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:01:34.247 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:52.276137+0000", 2024-05-15T13:01:34.247 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.247 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.247 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.247 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:01:34.247 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:52.276137+0000", 2024-05-15T13:01:34.247 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5600000000000001e-07 2024-05-15T13:01:34.247 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.247 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.247 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:01:34.247 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:52.276140+0000", 2024-05-15T13:01:34.247 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.134e-06 2024-05-15T13:01:34.247 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.247 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.247 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:01:34.248 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:52.276147+0000", 2024-05-15T13:01:34.248 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0670000000000003e-06 2024-05-15T13:01:34.248 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.248 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.248 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:01:34.248 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:52.276154+0000", 2024-05-15T13:01:34.248 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3990000000000001e-06 2024-05-15T13:01:34.248 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.248 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.248 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.248 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:52.279462+0000", 2024-05-15T13:01:34.248 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033081759999999999 2024-05-15T13:01:34.248 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.248 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.248 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:01:34.248 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:52.279526+0000", 2024-05-15T13:01:34.249 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4324000000000003e-05 2024-05-15T13:01:34.249 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.249 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.249 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.249 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:55.133020+0000", 2024-05-15T13:01:34.249 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8534936769999999 2024-05-15T13:01:34.249 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.249 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.249 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:01:34.249 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:55.133048+0000", 2024-05-15T13:01:34.249 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8455000000000001e-05 2024-05-15T13:01:34.249 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.249 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:01:34.249 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.250 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.250 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.250 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:167732 4.9 4:9df1642a:::benchmark_data_smithi177_99756_object47760:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e371)", 2024-05-15T13:01:34.250 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:55.093701+0000", 2024-05-15T13:01:34.250 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 39.101561017000002, 2024-05-15T13:01:34.250 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8566292610000001, 2024-05-15T13:01:34.250 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:01:34.250 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:01:34.250 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:01:34.250 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:01:34.250 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:01:34.250 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 167732 2024-05-15T13:01:34.250 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.250 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:01:34.251 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.251 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:01:34.251 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:55.093701+0000", 2024-05-15T13:01:34.251 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.251 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.251 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.251 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:01:34.251 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:55.093701+0000", 2024-05-15T13:01:34.251 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.251 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.251 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.251 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:01:34.251 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:55.093702+0000", 2024-05-15T13:01:34.251 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7800000000000002e-07 2024-05-15T13:01:34.251 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.252 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.252 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:01:34.252 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:55.093704+0000", 2024-05-15T13:01:34.252 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3140000000000002e-06 2024-05-15T13:01:34.252 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.252 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.252 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:01:34.252 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:55.093710+0000", 2024-05-15T13:01:34.252 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8669999999999999e-06 2024-05-15T13:01:34.252 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.252 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.252 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:01:34.252 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:55.093715+0000", 2024-05-15T13:01:34.252 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9640000000000002e-06 2024-05-15T13:01:34.252 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.252 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.253 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.253 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:55.093836+0000", 2024-05-15T13:01:34.253 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000121204 2024-05-15T13:01:34.253 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.253 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.253 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:01:34.253 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:55.093922+0000", 2024-05-15T13:01:34.253 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6473999999999999e-05 2024-05-15T13:01:34.253 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.253 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.253 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.253 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:56.950316+0000", 2024-05-15T13:01:34.253 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8563931060000001 2024-05-15T13:01:34.253 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.254 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.254 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:01:34.254 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:56.950330+0000", 2024-05-15T13:01:34.254 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4953999999999999e-05 2024-05-15T13:01:34.254 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.254 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:01:34.254 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.254 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.254 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.254 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:168914 4.9 4:975d04d4:::benchmark_data_smithi177_99756_object48942:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e371)", 2024-05-15T13:01:34.254 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:56.929264+0000", 2024-05-15T13:01:34.254 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 37.265998381999999, 2024-05-15T13:01:34.254 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.220342311, 2024-05-15T13:01:34.254 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:01:34.255 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:01:34.255 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:01:34.255 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:01:34.255 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:01:34.255 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 168914 2024-05-15T13:01:34.255 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.255 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:01:34.255 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.255 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:01:34.255 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:56.929264+0000", 2024-05-15T13:01:34.255 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.255 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.255 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.255 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:01:34.256 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:56.929264+0000", 2024-05-15T13:01:34.256 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.256 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.256 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.256 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:01:34.256 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:56.929264+0000", 2024-05-15T13:01:34.256 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5900000000000003e-07 2024-05-15T13:01:34.256 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.256 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.256 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:01:34.256 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:56.929266+0000", 2024-05-15T13:01:34.256 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9489999999999999e-06 2024-05-15T13:01:34.256 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.256 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.256 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:01:34.257 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:56.929270+0000", 2024-05-15T13:01:34.257 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2629999999999997e-06 2024-05-15T13:01:34.257 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.257 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.257 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:01:34.257 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:56.929274+0000", 2024-05-15T13:01:34.257 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3900000000000002e-06 2024-05-15T13:01:34.257 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.257 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.257 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.257 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:56.929488+0000", 2024-05-15T13:01:34.257 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00021410600000000001 2024-05-15T13:01:34.257 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.258 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.258 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:01:34.258 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:56.929539+0000", 2024-05-15T13:01:34.258 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1205999999999997e-05 2024-05-15T13:01:34.258 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.258 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.258 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.258 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:00.149591+0000", 2024-05-15T13:01:34.258 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2200518680000001 2024-05-15T13:01:34.258 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.258 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.258 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:01:34.258 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:00.149606+0000", 2024-05-15T13:01:34.259 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.517e-05 2024-05-15T13:01:34.259 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.259 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:01:34.259 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.259 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.259 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.259 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:172163 4.9 4:98451c33:::benchmark_data_smithi177_99756_object52191:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-05-15T13:01:34.259 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:02.164575+0000", 2024-05-15T13:01:34.259 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 32.030687409000002, 2024-05-15T13:01:34.259 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.236568898, 2024-05-15T13:01:34.259 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:01:34.259 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:01:34.259 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:01:34.259 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:01:34.260 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:01:34.260 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 172163 2024-05-15T13:01:34.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.260 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:01:34.260 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.260 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:01:34.260 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164575+0000", 2024-05-15T13:01:34.260 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.260 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.260 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:01:34.260 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164575+0000", 2024-05-15T13:01:34.260 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.261 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.261 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:01:34.261 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164575+0000", 2024-05-15T13:01:34.261 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6199999999999998e-07 2024-05-15T13:01:34.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.261 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.261 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:01:34.261 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164578+0000", 2024-05-15T13:01:34.261 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9160000000000001e-06 2024-05-15T13:01:34.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.261 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.261 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:01:34.261 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164585+0000", 2024-05-15T13:01:34.261 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6549999999999998e-06 2024-05-15T13:01:34.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.262 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.262 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:01:34.262 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164590+0000", 2024-05-15T13:01:34.262 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4739999999999996e-06 2024-05-15T13:01:34.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.262 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.262 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.262 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.168793+0000", 2024-05-15T13:01:34.262 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.004202842 2024-05-15T13:01:34.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.262 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.262 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:01:34.263 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.168879+0000", 2024-05-15T13:01:34.263 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5791000000000006e-05 2024-05-15T13:01:34.263 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.263 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.263 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.263 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.401130+0000", 2024-05-15T13:01:34.263 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.23225063000000001 2024-05-15T13:01:34.263 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.263 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.263 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:01:34.263 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.401144+0000", 2024-05-15T13:01:34.263 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4127999999999999e-05 2024-05-15T13:01:34.263 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.263 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:01:34.264 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.264 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.264 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.264 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:172167 4.9 4:9c3933af:::benchmark_data_smithi177_99756_object52195:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-05-15T13:01:34.264 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:02.164597+0000", 2024-05-15T13:01:34.264 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 32.030664966000003, 2024-05-15T13:01:34.264 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.24428116699999999, 2024-05-15T13:01:34.264 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:01:34.264 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:01:34.264 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:01:34.264 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:01:34.264 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:01:34.264 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 172167 2024-05-15T13:01:34.264 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.265 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:01:34.265 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.265 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:01:34.265 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164597+0000", 2024-05-15T13:01:34.265 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.265 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.265 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.265 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:01:34.265 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164597+0000", 2024-05-15T13:01:34.265 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.265 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.265 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.265 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:01:34.265 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164598+0000", 2024-05-15T13:01:34.266 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1e-07 2024-05-15T13:01:34.266 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.266 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.266 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:01:34.266 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164599+0000", 2024-05-15T13:01:34.266 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6500000000000001e-06 2024-05-15T13:01:34.266 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.266 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.266 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:01:34.266 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164603+0000", 2024-05-15T13:01:34.266 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9369999999999997e-06 2024-05-15T13:01:34.266 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.266 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.266 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:01:34.267 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164606+0000", 2024-05-15T13:01:34.267 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1140000000000002e-06 2024-05-15T13:01:34.267 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.267 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.267 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.267 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.168923+0000", 2024-05-15T13:01:34.267 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0043163259999999997 2024-05-15T13:01:34.267 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.267 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.267 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:01:34.267 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.169000+0000", 2024-05-15T13:01:34.267 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6975000000000001e-05 2024-05-15T13:01:34.267 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.267 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.267 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.268 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.408864+0000", 2024-05-15T13:01:34.268 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.23986439100000001 2024-05-15T13:01:34.268 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.268 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.268 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:01:34.268 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.408878+0000", 2024-05-15T13:01:34.268 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4464000000000001e-05 2024-05-15T13:01:34.268 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.268 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:01:34.268 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.268 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.268 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.268 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:172180 4.9 4:9a38ca0a:::benchmark_data_smithi177_99756_object52208:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-05-15T13:01:34.268 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:02.164610+0000", 2024-05-15T13:01:34.268 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 32.030651804999998, 2024-05-15T13:01:34.269 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.25200767200000002, 2024-05-15T13:01:34.269 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:01:34.269 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:01:34.269 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:01:34.269 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:01:34.269 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:01:34.269 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 172180 2024-05-15T13:01:34.269 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.269 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:01:34.269 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.269 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:01:34.269 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164610+0000", 2024-05-15T13:01:34.269 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.269 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.269 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.269 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:01:34.270 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164610+0000", 2024-05-15T13:01:34.270 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.270 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.270 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.270 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:01:34.270 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164611+0000", 2024-05-15T13:01:34.270 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2399999999999999e-07 2024-05-15T13:01:34.270 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.270 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.270 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:01:34.270 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164612+0000", 2024-05-15T13:01:34.270 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4300000000000001e-06 2024-05-15T13:01:34.270 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.271 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.271 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:01:34.271 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164615+0000", 2024-05-15T13:01:34.271 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3840000000000001e-06 2024-05-15T13:01:34.271 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.271 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.271 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:01:34.271 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164618+0000", 2024-05-15T13:01:34.271 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8609999999999998e-06 2024-05-15T13:01:34.271 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.271 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.271 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.271 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.169035+0000", 2024-05-15T13:01:34.271 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0044169819999999999 2024-05-15T13:01:34.271 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.272 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.272 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:01:34.272 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.169125+0000", 2024-05-15T13:01:34.272 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0130000000000006e-05 2024-05-15T13:01:34.272 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.272 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.272 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.272 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.416606+0000", 2024-05-15T13:01:34.272 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.247480643 2024-05-15T13:01:34.272 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.272 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.272 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:01:34.272 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.416618+0000", 2024-05-15T13:01:34.273 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2018000000000001e-05 2024-05-15T13:01:34.273 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.273 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:01:34.273 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.273 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.273 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.273 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:172183 4.9 4:90753368:::benchmark_data_smithi177_99756_object52211:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-05-15T13:01:34.273 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:02.164623+0000", 2024-05-15T13:01:34.273 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 32.030639591000003, 2024-05-15T13:01:34.273 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.25915582700000001, 2024-05-15T13:01:34.273 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:01:34.273 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:01:34.273 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:01:34.273 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:01:34.274 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:01:34.274 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 172183 2024-05-15T13:01:34.274 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.274 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:01:34.274 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.274 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:01:34.274 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164623+0000", 2024-05-15T13:01:34.274 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.274 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.274 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.274 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:01:34.274 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164623+0000", 2024-05-15T13:01:34.274 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.275 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.275 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:01:34.275 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164623+0000", 2024-05-15T13:01:34.275 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2000000000000001e-07 2024-05-15T13:01:34.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.275 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.275 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:01:34.275 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164625+0000", 2024-05-15T13:01:34.275 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1280000000000002e-06 2024-05-15T13:01:34.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.275 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.275 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:01:34.276 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164628+0000", 2024-05-15T13:01:34.276 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3900000000000002e-06 2024-05-15T13:01:34.276 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.276 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.276 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:01:34.276 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164631+0000", 2024-05-15T13:01:34.276 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8250000000000001e-06 2024-05-15T13:01:34.276 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.276 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.276 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.276 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.169155+0000", 2024-05-15T13:01:34.276 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0045237109999999997 2024-05-15T13:01:34.276 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.276 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.277 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:01:34.277 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.169230+0000", 2024-05-15T13:01:34.277 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5099999999999996e-05 2024-05-15T13:01:34.277 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.277 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.277 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.277 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.423763+0000", 2024-05-15T13:01:34.277 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.25453252399999998 2024-05-15T13:01:34.277 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.277 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.277 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:01:34.277 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.423778+0000", 2024-05-15T13:01:34.277 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5928999999999999e-05 2024-05-15T13:01:34.278 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.278 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:01:34.278 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.278 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.278 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.278 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:173038 4.4 4:2d62fb0b:::benchmark_data_smithi177_99756_object53066:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-05-15T13:01:34.278 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:03.398884+0000", 2024-05-15T13:01:34.278 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 30.796377757999998, 2024-05-15T13:01:34.278 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.23040877300000001, 2024-05-15T13:01:34.278 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:01:34.278 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:01:34.278 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:01:34.278 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:01:34.278 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:01:34.279 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 173038 2024-05-15T13:01:34.279 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.279 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:01:34.279 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.279 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:01:34.279 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.398884+0000", 2024-05-15T13:01:34.279 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.279 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.279 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.279 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:01:34.279 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.398884+0000", 2024-05-15T13:01:34.279 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.279 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.280 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.280 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:01:34.280 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.398885+0000", 2024-05-15T13:01:34.280 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5699999999999998e-07 2024-05-15T13:01:34.280 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.280 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.280 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:01:34.280 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.398888+0000", 2024-05-15T13:01:34.280 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1920000000000001e-06 2024-05-15T13:01:34.280 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.280 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.280 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:01:34.280 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.398895+0000", 2024-05-15T13:01:34.280 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9569999999999996e-06 2024-05-15T13:01:34.280 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.281 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.281 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:01:34.281 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.398901+0000", 2024-05-15T13:01:34.281 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2099999999999998e-06 2024-05-15T13:01:34.281 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.281 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.281 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.281 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.398963+0000", 2024-05-15T13:01:34.281 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1816999999999994e-05 2024-05-15T13:01:34.281 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.281 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.281 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:01:34.281 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.399056+0000", 2024-05-15T13:01:34.281 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2388999999999999e-05 2024-05-15T13:01:34.282 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.282 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.282 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2,3", 2024-05-15T13:01:34.282 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.399147+0000", 2024-05-15T13:01:34.282 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1587000000000003e-05 2024-05-15T13:01:34.282 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.282 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.282 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:01:34.282 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.405168+0000", 2024-05-15T13:01:34.282 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.006020498 2024-05-15T13:01:34.282 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.282 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.282 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:01:34.283 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.431511+0000", 2024-05-15T13:01:34.283 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.026343096 2024-05-15T13:01:34.283 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.283 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.283 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:01:34.283 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.629177+0000", 2024-05-15T13:01:34.283 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19766649 2024-05-15T13:01:34.283 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.283 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.283 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:01:34.283 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.629240+0000", 2024-05-15T13:01:34.283 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2558999999999995e-05 2024-05-15T13:01:34.283 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.283 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.284 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:01:34.284 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.629293+0000", 2024-05-15T13:01:34.284 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3520999999999998e-05 2024-05-15T13:01:34.284 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.284 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:01:34.284 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.284 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.284 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.284 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:173062 4.4 4:2e627ecb:::benchmark_data_smithi177_99756_object53090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-05-15T13:01:34.284 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:03.440922+0000", 2024-05-15T13:01:34.284 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 30.754340218999999, 2024-05-15T13:01:34.284 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.18849624100000001, 2024-05-15T13:01:34.284 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:01:34.284 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:01:34.285 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:01:34.285 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:01:34.285 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:01:34.285 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 173062 2024-05-15T13:01:34.285 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.285 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:01:34.285 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.285 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:01:34.285 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.440922+0000", 2024-05-15T13:01:34.285 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.285 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.285 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.285 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:01:34.285 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.440922+0000", 2024-05-15T13:01:34.286 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.286 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.286 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.286 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:01:34.286 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.440923+0000", 2024-05-15T13:01:34.286 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.209e-06 2024-05-15T13:01:34.286 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.286 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.286 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:01:34.286 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.440935+0000", 2024-05-15T13:01:34.286 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1583000000000001e-05 2024-05-15T13:01:34.286 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.286 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.287 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:01:34.287 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.440950+0000", 2024-05-15T13:01:34.287 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5153e-05 2024-05-15T13:01:34.287 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.287 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.287 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:01:34.287 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.440958+0000", 2024-05-15T13:01:34.287 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4740000000000005e-06 2024-05-15T13:01:34.287 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.287 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.287 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.287 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.441020+0000", 2024-05-15T13:01:34.287 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1550000000000005e-05 2024-05-15T13:01:34.287 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.288 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.288 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:01:34.288 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.441161+0000", 2024-05-15T13:01:34.288 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014142399999999999 2024-05-15T13:01:34.288 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.288 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.288 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2,3", 2024-05-15T13:01:34.288 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.441297+0000", 2024-05-15T13:01:34.288 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000135735 2024-05-15T13:01:34.288 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.288 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.288 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:01:34.288 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.446280+0000", 2024-05-15T13:01:34.288 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0049829449999999999 2024-05-15T13:01:34.289 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.289 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.289 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:01:34.289 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.446554+0000", 2024-05-15T13:01:34.289 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00027428400000000002 2024-05-15T13:01:34.289 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.289 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.289 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:01:34.289 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.629332+0000", 2024-05-15T13:01:34.289 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.18277717099999999 2024-05-15T13:01:34.289 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.289 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.289 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:01:34.289 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.629382+0000", 2024-05-15T13:01:34.290 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0683999999999998e-05 2024-05-15T13:01:34.290 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.290 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.290 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:01:34.290 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:03.629418+0000", 2024-05-15T13:01:34.290 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6028999999999997e-05 2024-05-15T13:01:34.290 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.290 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:01:34.290 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.290 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.290 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.290 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:175340 4.9 4:9f0a7fac:::benchmark_data_smithi177_99756_object55368:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e374)", 2024-05-15T13:01:34.290 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:06.982258+0000", 2024-05-15T13:01:34.291 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 27.213004126000001, 2024-05-15T13:01:34.291 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8210182489999998, 2024-05-15T13:01:34.291 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:01:34.291 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:01:34.291 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:01:34.291 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:01:34.291 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:01:34.291 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 175340 2024-05-15T13:01:34.291 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.291 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:01:34.291 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.291 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:01:34.291 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:06.982258+0000", 2024-05-15T13:01:34.291 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.292 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.292 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.292 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:01:34.292 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:06.982258+0000", 2024-05-15T13:01:34.292 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:34.292 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.292 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.292 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:01:34.292 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:06.982259+0000", 2024-05-15T13:01:34.292 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.89e-07 2024-05-15T13:01:34.292 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.292 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.292 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:01:34.292 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:06.982261+0000", 2024-05-15T13:01:34.293 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6970000000000002e-06 2024-05-15T13:01:34.293 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.293 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.293 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:01:34.293 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:06.982267+0000", 2024-05-15T13:01:34.293 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8919999999999997e-06 2024-05-15T13:01:34.293 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.293 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.293 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:01:34.293 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:06.982272+0000", 2024-05-15T13:01:34.293 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0359999999999997e-06 2024-05-15T13:01:34.293 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.293 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.293 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.294 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:06.985640+0000", 2024-05-15T13:01:34.294 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033680469999999999 2024-05-15T13:01:34.294 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.294 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.294 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:01:34.294 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:06.985687+0000", 2024-05-15T13:01:34.294 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7246000000000001e-05 2024-05-15T13:01:34.294 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.294 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.294 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:34.294 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:10.803260+0000", 2024-05-15T13:01:34.294 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8175724669999997 2024-05-15T13:01:34.294 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:34.295 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:34.295 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:01:34.295 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:10.803276+0000", 2024-05-15T13:01:34.295 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6475000000000001e-05 2024-05-15T13:01:34.295 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.295 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:01:34.295 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.295 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:34.295 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:01:34.295 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:01:34.295 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:01:34.343 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:34.371 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:01:34.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:34.377 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:01:34.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:34 smithi177 ceph-mon[44303]: osdmap e384: 12 total, 12 up, 11 in 2024-05-15T13:01:34.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:34 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 16173/152310 objects degraded (10.618%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:34.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:34 smithi177 ceph-mon[44303]: pgmap v1041: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 637 op/s; 16209/149673 objects degraded (10.830%); 9849/149673 objects misplaced (6.580%); 9.8 MiB/s, 157 objects/s recovering 2024-05-15T13:01:34.478 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:01:34.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:34 smithi027 ceph-mon[39795]: osdmap e384: 12 total, 12 up, 11 in 2024-05-15T13:01:34.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:34 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 16173/152310 objects degraded (10.618%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:34.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:34 smithi027 ceph-mon[39795]: pgmap v1041: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 637 op/s; 16209/149673 objects degraded (10.830%); 9849/149673 objects misplaced (6.580%); 9.8 MiB/s, 157 objects/s recovering 2024-05-15T13:01:34.527 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:34.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:34 smithi120 ceph-mon[43961]: osdmap e384: 12 total, 12 up, 11 in 2024-05-15T13:01:34.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:34 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 16173/152310 objects degraded (10.618%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:34.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:34 smithi120 ceph-mon[43961]: pgmap v1041: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 637 op/s; 16209/149673 objects degraded (10.830%); 9849/149673 objects misplaced (6.580%); 9.8 MiB/s, 157 objects/s recovering 2024-05-15T13:01:35.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:35 smithi177 ceph-mon[44303]: osdmap e385: 12 total, 12 up, 11 in 2024-05-15T13:01:35.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:35 smithi177 ceph-mon[44303]: from='client.25327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:01:35.494 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:01:35.494 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:01:35.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:35 smithi027 ceph-mon[39795]: osdmap e385: 12 total, 12 up, 11 in 2024-05-15T13:01:35.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:35 smithi027 ceph-mon[39795]: from='client.25327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:01:35.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:35 smithi120 ceph-mon[43961]: osdmap e385: 12 total, 12 up, 11 in 2024-05-15T13:01:35.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:35 smithi120 ceph-mon[43961]: from='client.25327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:01:35.667 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:01:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:01:36.427 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:01:36.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:36 smithi177 ceph-mon[44303]: pgmap v1043: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 546 op/s; 16209/146316 objects degraded (11.078%); 9846/146316 objects misplaced (6.729%); 4.2 MiB/s, 68 objects/s recovering 2024-05-15T13:01:36.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:36.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:01:36.430 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:36 smithi027 ceph-mon[39795]: pgmap v1043: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 546 op/s; 16209/146316 objects degraded (11.078%); 9846/146316 objects misplaced (6.729%); 4.2 MiB/s, 68 objects/s recovering 2024-05-15T13:01:36.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:36 smithi120 ceph-mon[43961]: pgmap v1043: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 546 op/s; 16209/146316 objects degraded (11.078%); 9846/146316 objects misplaced (6.729%); 4.2 MiB/s, 68 objects/s recovering 2024-05-15T13:01:36.530 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:01:36.581 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:37.559 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":1042,"stamp":"2024-05-15T13:01:34.137784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3257336376,"num_objects":49697,"num_object_clones":0,"num_object_copies":149091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16209,"num_objects_misplaced":9849,"num_objects_unfound":0,"num_objects_dirty":49697,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":190426,"num_write_kb":7680955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18298,"num_bytes_recovered":1189314272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160984,"ondisk_log_size":160984,"up":51,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":14539580,"kb_used_data":11195028,"kb_used_omap":17,"kb_used_meta":3344302,"kb_avail":1016657092,"statfs":{"total":1055945392128,"available":1041056862208,"internally_reserved":0,"allocated":11463708672,"data_stored":11460809208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17551,"internal_metadata":3424566129},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,7,6,16,7,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-275054592,"num_objects":-4197,"num_object_clones":0,"num_object_copies":-12591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35,"num_objects_misplaced":-814,"num_objects_unfound":0,"num_objects_dirty":-4197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4197,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":779,"num_bytes_recovered":50855936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.240482"},"pg_stats":[{"pgid":"4.f","version":"382'11608","reported_seq":21383,"reported_epoch":383,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-05-15T13:01:31.119180+0000","last_change":"2024-05-15T12:59:12.836225+0000","last_active":"2024-05-15T13:01:31.119180+0000","last_peered":"2024-05-15T13:01:31.119180+0000","last_clean":"2024-05-15T12:59:10.259362+0000","last_became_active":"2024-05-15T12:59:12.819643+0000","last_became_peered":"2024-05-15T12:59:12.819643+0000","last_unstale":"2024-05-15T13:01:31.119180+0000","last_undegraded":"2024-05-15T12:59:12.358355+0000","last_fullsized":"2024-05-15T12:59:12.315304+0000","mapping_epoch":343,"log_start":"323'1600","ondisk_log_start":"323'1600","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201850880,"num_objects":3080,"num_object_clones":0,"num_object_copies":9240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":1351,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11608,"num_write_kb":470016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4816,"num_bytes_recovered":315621376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3],"avail_no_missing":["1","3","9"],"object_location_counts":[{"shards":"1,3","objects":1726},{"shards":"1,3,9","objects":1351}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"382'11856","reported_seq":11997,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-05-15T13:01:29.706287+0000","last_change":"2024-05-15T12:57:55.242710+0000","last_active":"2024-05-15T13:01:29.706287+0000","last_peered":"2024-05-15T13:01:29.706287+0000","last_clean":"2024-05-15T13:01:29.706287+0000","last_became_active":"2024-05-15T12:57:55.242367+0000","last_became_peered":"2024-05-15T12:57:55.242367+0000","last_unstale":"2024-05-15T13:01:29.706287+0000","last_undegraded":"2024-05-15T13:01:29.706287+0000","last_fullsized":"2024-05-15T13:01:29.706287+0000","mapping_epoch":317,"log_start":"323'1800","ondisk_log_start":"323'1800","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212598784,"num_objects":3244,"num_object_clones":0,"num_object_copies":9732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11856,"num_write_kb":483200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"382'11621","reported_seq":11763,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-05-15T13:01:29.767068+0000","last_change":"2024-05-15T12:57:55.155833+0000","last_active":"2024-05-15T13:01:29.767068+0000","last_peered":"2024-05-15T13:01:29.767068+0000","last_clean":"2024-05-15T13:01:29.767068+0000","last_became_active":"2024-05-15T12:57:55.155480+0000","last_became_peered":"2024-05-15T12:57:55.155480+0000","last_unstale":"2024-05-15T13:01:29.767068+0000","last_undegraded":"2024-05-15T13:01:29.767068+0000","last_fullsized":"2024-05-15T13:01:29.767068+0000","mapping_epoch":317,"log_start":"323'1600","ondisk_log_start":"323'1600","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520920,"num_objects":3137,"num_object_clones":0,"num_object_copies":9411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3137,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11621,"num_write_kb":472193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"382'11937","reported_seq":12078,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-05-15T13:01:29.742852+0000","last_change":"2024-05-15T12:57:55.157244+0000","last_active":"2024-05-15T13:01:29.742852+0000","last_peered":"2024-05-15T13:01:29.742852+0000","last_clean":"2024-05-15T13:01:29.742852+0000","last_became_active":"2024-05-15T12:57:55.156859+0000","last_became_peered":"2024-05-15T12:57:55.156859+0000","last_unstale":"2024-05-15T13:01:29.742852+0000","last_undegraded":"2024-05-15T13:01:29.742852+0000","last_fullsized":"2024-05-15T13:01:29.742852+0000","mapping_epoch":317,"log_start":"323'1900","ondisk_log_start":"323'1900","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199426048,"num_objects":3043,"num_object_clones":0,"num_object_copies":9129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11937,"num_write_kb":479360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"384'12032","reported_seq":12177,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-05-15T13:01:33.865037+0000","last_change":"2024-05-15T12:57:55.160034+0000","last_active":"2024-05-15T13:01:33.865037+0000","last_peered":"2024-05-15T13:01:33.865037+0000","last_clean":"2024-05-15T13:01:33.865037+0000","last_became_active":"2024-05-15T12:57:55.159699+0000","last_became_peered":"2024-05-15T12:57:55.159699+0000","last_unstale":"2024-05-15T13:01:33.865037+0000","last_undegraded":"2024-05-15T13:01:33.865037+0000","last_fullsized":"2024-05-15T13:01:33.865037+0000","mapping_epoch":317,"log_start":"323'2000","ondisk_log_start":"323'2000","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193462272,"num_objects":2952,"num_object_clones":0,"num_object_copies":8856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12032,"num_write_kb":479488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"384'11942","reported_seq":12087,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-05-15T13:01:33.460015+0000","last_change":"2024-05-15T12:57:55.242821+0000","last_active":"2024-05-15T13:01:33.460015+0000","last_peered":"2024-05-15T13:01:33.460015+0000","last_clean":"2024-05-15T13:01:33.460015+0000","last_became_active":"2024-05-15T12:57:55.242372+0000","last_became_peered":"2024-05-15T12:57:55.242372+0000","last_unstale":"2024-05-15T13:01:33.460015+0000","last_undegraded":"2024-05-15T13:01:33.460015+0000","last_fullsized":"2024-05-15T13:01:33.460015+0000","mapping_epoch":317,"log_start":"323'1900","ondisk_log_start":"323'1900","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198705152,"num_objects":3032,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11942,"num_write_kb":479168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"383'11797","reported_seq":12080,"reported_epoch":383,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-15T13:01:31.078801+0000","last_change":"2024-05-15T12:59:12.862170+0000","last_active":"2024-05-15T13:01:31.078801+0000","last_peered":"2024-05-15T13:01:31.078801+0000","last_clean":"2024-05-15T12:59:10.318134+0000","last_became_active":"2024-05-15T12:59:12.847627+0000","last_became_peered":"2024-05-15T12:59:12.847627+0000","last_unstale":"2024-05-15T13:01:31.078801+0000","last_undegraded":"2024-05-15T12:59:12.454722+0000","last_fullsized":"2024-05-15T12:59:12.315028+0000","mapping_epoch":343,"log_start":"323'1700","ondisk_log_start":"323'1700","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205324288,"num_objects":3133,"num_object_clones":0,"num_object_copies":9399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11797,"num_write_kb":477760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[1,3],"avail_no_missing":["1","3","6"],"object_location_counts":[{"shards":"1,3","objects":-3084},{"shards":"1,3,6","objects":1953}],"blocked_by":[],"up_primary":11,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"384'12325","reported_seq":12572,"reported_epoch":384,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-15T13:01:33.424955+0000","last_change":"2024-05-15T12:59:12.870715+0000","last_active":"2024-05-15T13:01:33.424955+0000","last_peered":"2024-05-15T13:01:33.424955+0000","last_clean":"2024-05-15T12:59:10.299125+0000","last_became_active":"2024-05-15T12:59:12.854810+0000","last_became_peered":"2024-05-15T12:59:12.854810+0000","last_unstale":"2024-05-15T13:01:33.424955+0000","last_undegraded":"2024-05-15T12:59:12.358625+0000","last_fullsized":"2024-05-15T12:59:12.315099+0000","mapping_epoch":343,"log_start":"323'2300","ondisk_log_start":"323'2300","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202833920,"num_objects":3095,"num_object_clones":0,"num_object_copies":9285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6455,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12325,"num_write_kb":493440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,9],"acting":[6,9],"avail_no_missing":["6","9","10"],"object_location_counts":[{"shards":"6,9","objects":-3360},{"shards":"6,9,10","objects":1890}],"blocked_by":[],"up_primary":11,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"383'11790","reported_seq":11933,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-05-15T13:01:31.102754+0000","last_change":"2024-05-15T12:57:55.242791+0000","last_active":"2024-05-15T13:01:31.102754+0000","last_peered":"2024-05-15T13:01:31.102754+0000","last_clean":"2024-05-15T13:01:31.102754+0000","last_became_active":"2024-05-15T12:57:55.242453+0000","last_became_peered":"2024-05-15T12:57:55.242453+0000","last_unstale":"2024-05-15T13:01:31.102754+0000","last_undegraded":"2024-05-15T13:01:31.102754+0000","last_fullsized":"2024-05-15T13:01:31.102754+0000","mapping_epoch":317,"log_start":"323'1700","ondisk_log_start":"323'1700","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202244096,"num_objects":3086,"num_object_clones":0,"num_object_copies":9258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11790,"num_write_kb":476032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"382'11978","reported_seq":12119,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-05-15T13:01:30.473795+0000","last_change":"2024-05-15T12:57:55.157512+0000","last_active":"2024-05-15T13:01:30.473795+0000","last_peered":"2024-05-15T13:01:30.473795+0000","last_clean":"2024-05-15T13:01:30.473795+0000","last_became_active":"2024-05-15T12:57:55.157131+0000","last_became_peered":"2024-05-15T12:57:55.157131+0000","last_unstale":"2024-05-15T13:01:30.473795+0000","last_undegraded":"2024-05-15T13:01:30.473795+0000","last_fullsized":"2024-05-15T13:01:30.473795+0000","mapping_epoch":317,"log_start":"323'1900","ondisk_log_start":"323'1900","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203948032,"num_objects":3112,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11978,"num_write_kb":482880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"383'11888","reported_seq":12031,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-05-15T13:01:31.098735+0000","last_change":"2024-05-15T12:57:55.158173+0000","last_active":"2024-05-15T13:01:31.098735+0000","last_peered":"2024-05-15T13:01:31.098735+0000","last_clean":"2024-05-15T13:01:31.098735+0000","last_became_active":"2024-05-15T12:57:55.157465+0000","last_became_peered":"2024-05-15T12:57:55.157465+0000","last_unstale":"2024-05-15T13:01:31.098735+0000","last_undegraded":"2024-05-15T13:01:31.098735+0000","last_fullsized":"2024-05-15T13:01:31.098735+0000","mapping_epoch":317,"log_start":"323'1800","ondisk_log_start":"323'1800","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199622656,"num_objects":3046,"num_object_clones":0,"num_object_copies":9138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11888,"num_write_kb":477888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"382'11689","reported_seq":23731,"reported_epoch":382,"state":"active+recovering+undersized+remapped","last_fresh":"2024-05-15T13:01:29.075459+0000","last_change":"2024-05-15T12:59:12.892718+0000","last_active":"2024-05-15T13:01:29.075459+0000","last_peered":"2024-05-15T13:01:29.075459+0000","last_clean":"2024-05-15T12:59:10.291321+0000","last_became_active":"2024-05-15T12:59:12.843364+0000","last_became_peered":"2024-05-15T12:59:12.843364+0000","last_unstale":"2024-05-15T13:01:29.075459+0000","last_undegraded":"2024-05-15T13:01:29.075459+0000","last_fullsized":"2024-05-15T12:59:12.316583+0000","mapping_epoch":343,"log_start":"323'1600","ondisk_log_start":"323'1600","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213712896,"num_objects":3261,"num_object_clones":0,"num_object_copies":9783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":301,"num_objects_unfound":0,"num_objects_dirty":3261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11689,"num_write_kb":478400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5949,"num_bytes_recovered":389873664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[0,1],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1","objects":2960},{"shards":"0,1,2","objects":301}],"blocked_by":[],"up_primary":11,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"383'11974","reported_seq":14029,"reported_epoch":383,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-05-15T13:01:32.772116+0000","last_change":"2024-05-15T13:01:06.812888+0000","last_active":"2024-05-15T13:01:32.772116+0000","last_peered":"2024-05-15T13:01:32.772116+0000","last_clean":"2024-05-15T12:59:10.299975+0000","last_became_active":"2024-05-15T12:59:12.847592+0000","last_became_peered":"2024-05-15T12:59:12.847592+0000","last_unstale":"2024-05-15T13:01:32.772116+0000","last_undegraded":"2024-05-15T12:59:12.358581+0000","last_fullsized":"2024-05-15T12:59:12.320946+0000","mapping_epoch":343,"log_start":"323'1900","ondisk_log_start":"323'1900","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196739072,"num_objects":3002,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":1872,"num_objects_unfound":0,"num_objects_dirty":3002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11974,"num_write_kb":479232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":954,"num_bytes_recovered":61669376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,11],"acting":[5,9],"avail_no_missing":["5","8","9"],"object_location_counts":[{"shards":"5,8,9","objects":1872},{"shards":"5,9","objects":-2404}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"382'11880","reported_seq":12213,"reported_epoch":382,"state":"active+recovery_wait+remapped","last_fresh":"2024-05-15T13:01:29.740958+0000","last_change":"2024-05-15T12:59:12.857109+0000","last_active":"2024-05-15T13:01:29.740958+0000","last_peered":"2024-05-15T13:01:29.740958+0000","last_clean":"2024-05-15T12:59:10.257764+0000","last_became_active":"2024-05-15T12:59:12.843041+0000","last_became_peered":"2024-05-15T12:59:12.843041+0000","last_unstale":"2024-05-15T13:01:29.740958+0000","last_undegraded":"2024-05-15T13:01:29.740958+0000","last_fullsized":"2024-05-15T13:01:29.740958+0000","mapping_epoch":343,"log_start":"323'1800","ondisk_log_start":"323'1800","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'5644","last_scrub_stamp":"2024-05-15T12:59:00.172752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:59:00.172752+0000","objects_scrubbed":5592,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211943424,"num_objects":3234,"num_object_clones":0,"num_object_copies":9702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6325,"num_objects_unfound":0,"num_objects_dirty":3234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11880,"num_write_kb":483648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[4,5,6],"avail_no_missing":["4","5","6"],"object_location_counts":[{"shards":"4,5,6","objects":-3091}],"blocked_by":[],"up_primary":11,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"383'12007","reported_seq":12150,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-05-15T13:01:31.619624+0000","last_change":"2024-05-15T12:57:55.160182+0000","last_active":"2024-05-15T13:01:31.619624+0000","last_peered":"2024-05-15T13:01:31.619624+0000","last_clean":"2024-05-15T13:01:31.619624+0000","last_became_active":"2024-05-15T12:57:55.159918+0000","last_became_peered":"2024-05-15T12:57:55.159918+0000","last_unstale":"2024-05-15T13:01:31.619624+0000","last_undegraded":"2024-05-15T13:01:31.619624+0000","last_fullsized":"2024-05-15T13:01:31.619624+0000","mapping_epoch":317,"log_start":"323'2000","ondisk_log_start":"323'2000","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204800000,"num_objects":3125,"num_object_clones":0,"num_object_copies":9375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12007,"num_write_kb":484224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"384'11921","reported_seq":24842,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-05-15T13:01:33.516924+0000","last_change":"2024-05-15T13:01:08.498040+0000","last_active":"2024-05-15T13:01:33.516924+0000","last_peered":"2024-05-15T13:01:33.516924+0000","last_clean":"2024-05-15T13:01:33.516924+0000","last_became_active":"2024-05-15T13:01:08.475326+0000","last_became_peered":"2024-05-15T13:01:08.475326+0000","last_unstale":"2024-05-15T13:01:33.516924+0000","last_undegraded":"2024-05-15T13:01:33.516924+0000","last_fullsized":"2024-05-15T13:01:33.516924+0000","mapping_epoch":375,"log_start":"323'1900","ondisk_log_start":"323'1900","created":317,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:57:54.138788+0000","last_clean_scrub_stamp":"2024-05-15T12:57:54.138788+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204013568,"num_objects":3113,"num_object_clones":0,"num_object_copies":9339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11921,"num_write_kb":481088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6383,"num_bytes_recovered":418316288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":575,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-05-15T13:01:31.064838+0000","last_change":"2024-05-15T13:00:26.045887+0000","last_active":"2024-05-15T13:01:31.064838+0000","last_peered":"2024-05-15T13:01:31.064838+0000","last_clean":"2024-05-15T13:01:31.064838+0000","last_became_active":"2024-05-15T13:00:26.045412+0000","last_became_peered":"2024-05-15T13:00:26.045412+0000","last_unstale":"2024-05-15T13:01:31.064838+0000","last_undegraded":"2024-05-15T13:01:31.064838+0000","last_fullsized":"2024-05-15T13:01:31.064838+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T12:59:04.037279+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T12:59:04.037279+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":3256746008,"num_objects":49695,"num_object_clones":0,"num_object_copies":149085,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16209,"num_objects_misplaced":9849,"num_objects_unfound":0,"num_objects_dirty":49695,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190245,"num_write_kb":7678017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18284,"num_bytes_recovered":1185611776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11398098944,"data_stored":11398086728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160845,"ondisk_log_size":160845,"up":48,"acting":43,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056833,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1022728,"kb_used_data":811068,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92722424,"statfs":{"total":95995035648,"available":94947762176,"internally_reserved":0,"allocated":830533632,"data_stored":830270958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":216725958},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351173,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1486848,"kb_used_data":1234872,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":92258304,"statfs":{"total":95995035648,"available":94472503296,"internally_reserved":0,"allocated":1264508928,"data_stored":1264246956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":257948096},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514698,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1618940,"kb_used_data":1204596,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92126212,"statfs":{"total":95995035648,"available":94337241088,"internally_reserved":0,"allocated":1233506304,"data_stored":1233243812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":424278464},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":828728,"kb_used_data":612912,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":92916424,"statfs":{"total":95995035648,"available":95146418176,"internally_reserved":0,"allocated":627621888,"data_stored":627363488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":240518168787,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430425,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1903644,"kb_used_data":1579536,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":91841508,"statfs":{"total":95995035648,"available":94045704192,"internally_reserved":0,"allocated":1617444864,"data_stored":1617174354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":331872710},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575092,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2175172,"kb_used_data":1805244,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":91569980,"statfs":{"total":95995035648,"available":93767659520,"internally_reserved":0,"allocated":1848569856,"data_stored":1848303832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,2,0,5,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738620,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1120532,"kb_used_data":774216,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92624620,"statfs":{"total":95995035648,"available":94847610880,"internally_reserved":0,"allocated":792797184,"data_stored":792530226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411569,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1006148,"kb_used_data":786108,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":92739004,"statfs":{"total":95995035648,"available":94964740096,"internally_reserved":0,"allocated":804974592,"data_stored":804711938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149930,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1395012,"kb_used_data":980668,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92350140,"statfs":{"total":95995035648,"available":94566543360,"internally_reserved":0,"allocated":1004204032,"data_stored":1003941378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":424278471},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986405,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1336316,"kb_used_data":1012212,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92408836,"statfs":{"total":95995035648,"available":94626648064,"internally_reserved":0,"allocated":1036505088,"data_stored":1036242616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692063,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":645512,"kb_used_data":393596,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93099640,"statfs":{"total":95995035648,"available":95334031360,"internally_reserved":0,"allocated":403042304,"data_stored":402779650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":790822912,"data_stored":790822912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1826820096,"data_stored":1826816024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":800587776,"data_stored":800587776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1000734720,"data_stored":1000734720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1033375744,"data_stored":1033371672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":399572992,"data_stored":399572992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1611137024,"data_stored":1611137024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":625016832,"data_stored":625016832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1227296768,"data_stored":1227292696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1252261888,"data_stored":1252261888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":830472192,"data_stored":830472192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:01:37.560 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg force-backfill 4.9 2024-05-15T13:01:37.700 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-05-15T13:01:37.889 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:01:37.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:37.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:01:37.997 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:01:38.037 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:38.069 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:01:38.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:38.072 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:01:38.173 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:01:38.266 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:38.614 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:01:38.662 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:01:38.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:38.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:01:38.767 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:01:38.828 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:38.871 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:01:38.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:38.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:01:38.974 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:01:39.053 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:39.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:38 smithi027 ceph-mon[39795]: pgmap v1044: 17 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 654 op/s; 14359/141936 objects degraded (10.117%); 10764/141936 objects misplaced (7.584%); 12 MiB/s, 195 objects/s recovering 2024-05-15T13:01:39.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:38 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 14359/141936 objects degraded (10.117%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:39.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:38 smithi120 ceph-mon[43961]: pgmap v1044: 17 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 654 op/s; 14359/141936 objects degraded (10.117%); 10764/141936 objects misplaced (7.584%); 12 MiB/s, 195 objects/s recovering 2024-05-15T13:01:39.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:38 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 14359/141936 objects degraded (10.117%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:39.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:38 smithi177 ceph-mon[44303]: pgmap v1044: 17 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 654 op/s; 14359/141936 objects degraded (10.117%); 10764/141936 objects misplaced (7.584%); 12 MiB/s, 195 objects/s recovering 2024-05-15T13:01:39.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:38 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 14359/141936 objects degraded (10.117%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:40.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:39 smithi027 ceph-mon[39795]: 4.b scrub starts 2024-05-15T13:01:40.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:39 smithi120 ceph-mon[43961]: 4.b scrub starts 2024-05-15T13:01:40.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:39 smithi177 ceph-mon[44303]: 4.b scrub starts 2024-05-15T13:01:40.498 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:01:40.536 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:01:40.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:40.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:01:40.550 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:01:40.651 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:01:40.681 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:40.707 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:01:40.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:40.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:01:40.811 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:01:40.861 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:41.013 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:01:41.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:41.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:01:41.121 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:01:41.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:40 smithi027 ceph-mon[39795]: 4.b scrub ok 2024-05-15T13:01:41.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:40 smithi027 ceph-mon[39795]: pgmap v1045: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+clean; 2.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 627 op/s; 14359/137247 objects degraded (10.462%); 10755/137247 objects misplaced (7.836%); 8.6 MiB/s, 139 objects/s recovering 2024-05-15T13:01:41.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:40 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:01:41.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:40 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:01:41.171 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:41.200 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:01:41.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:41.207 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:01:41.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:40 smithi120 ceph-mon[43961]: 4.b scrub ok 2024-05-15T13:01:41.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:40 smithi120 ceph-mon[43961]: pgmap v1045: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+clean; 2.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 627 op/s; 14359/137247 objects degraded (10.462%); 10755/137247 objects misplaced (7.836%); 8.6 MiB/s, 139 objects/s recovering 2024-05-15T13:01:41.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:40 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:01:41.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:40 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:01:41.307 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:01:41.338 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:41.367 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:01:41.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:41.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:01:41.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:40 smithi177 ceph-mon[44303]: 4.b scrub ok 2024-05-15T13:01:41.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:40 smithi177 ceph-mon[44303]: pgmap v1045: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+clean; 2.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 627 op/s; 14359/137247 objects degraded (10.462%); 10755/137247 objects misplaced (7.836%); 8.6 MiB/s, 139 objects/s recovering 2024-05-15T13:01:41.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:40 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:01:41.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:40 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:01:41.471 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:01:41.548 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:41.660 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:01:41.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:41.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:01:41.765 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:01:41.803 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:41.836 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:01:41.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:41.839 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:01:41.940 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:01:41.969 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:41.993 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:01:41.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:41.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:01:42.004 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:42 smithi027 ceph-mon[39795]: osdmap e386: 12 total, 12 up, 11 in 2024-05-15T13:01:42.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:42 smithi027 ceph-mon[39795]: pgmap v1047: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+clean; 2.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 677 op/s; 14426/133410 objects degraded (10.813%); 10293/133410 objects misplaced (7.715%); 10 MiB/s, 167 objects/s recovering 2024-05-15T13:01:42.099 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:01:42.162 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:42 smithi177 ceph-mon[44303]: osdmap e386: 12 total, 12 up, 11 in 2024-05-15T13:01:42.162 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:42 smithi177 ceph-mon[44303]: pgmap v1047: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+clean; 2.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 677 op/s; 14426/133410 objects degraded (10.813%); 10293/133410 objects misplaced (7.715%); 10 MiB/s, 167 objects/s recovering 2024-05-15T13:01:42.168 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:42.200 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:01:42.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:42.203 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:01:42.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:42 smithi120 ceph-mon[43961]: osdmap e386: 12 total, 12 up, 11 in 2024-05-15T13:01:42.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:42 smithi120 ceph-mon[43961]: pgmap v1047: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+clean; 2.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 677 op/s; 14426/133410 objects degraded (10.813%); 10293/133410 objects misplaced (7.715%); 10 MiB/s, 167 objects/s recovering 2024-05-15T13:01:42.304 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:01:42.339 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:42.374 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:01:42.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:42.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:01:42.479 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:01:42.580 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:43.455 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:43 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 14426/133410 objects degraded (10.813%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:43.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:43 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 14426/133410 objects degraded (10.813%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:43.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:43 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 14426/133410 objects degraded (10.813%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:44.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:44 smithi027 ceph-mon[39795]: pgmap v1048: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+clean; 2.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 644 op/s; 14437/130206 objects degraded (11.088%); 10088/130206 objects misplaced (7.748%); 9.7 MiB/s, 157 objects/s recovering 2024-05-15T13:01:44.604 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:01:44.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:44.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:01:44.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:44 smithi177 ceph-mon[44303]: pgmap v1048: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+clean; 2.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 644 op/s; 14437/130206 objects degraded (11.088%); 10088/130206 objects misplaced (7.748%); 9.7 MiB/s, 157 objects/s recovering 2024-05-15T13:01:44.715 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:01:44.750 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:44 smithi120 ceph-mon[43961]: pgmap v1048: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+clean; 2.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 644 op/s; 14437/130206 objects degraded (11.088%); 10088/130206 objects misplaced (7.748%); 9.7 MiB/s, 157 objects/s recovering 2024-05-15T13:01:44.752 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:44.777 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:01:44.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:44.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:01:44.881 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:01:44.916 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:44.944 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:01:44.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:44.950 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:01:45.051 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:01:45.174 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:45.302 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:01:45.302 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:01:45.302 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:45.302 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:199305 4.6 e386/317 4:69d398ea:::benchmark_data_smithi177_99756_object79333:head v 386'12548, mlcod=386'12545)", 2024-05-15T13:01:45.302 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:45.307346+0000", 2024-05-15T13:01:45.303 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.0028778699999999998, 2024-05-15T13:01:45.303 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002901477, 2024-05-15T13:01:45.303 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:01:45.303 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:01:45.303 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:01:45.303 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:45.303 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:01:45.303 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.307346+0000", 2024-05-15T13:01:45.303 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:45.303 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:45.303 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:45.303 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:01:45.303 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.307346+0000", 2024-05-15T13:01:45.304 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:01:45.304 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:45.304 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:45.304 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:01:45.304 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.307347+0000", 2024-05-15T13:01:45.304 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.015e-06 2024-05-15T13:01:45.304 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:45.304 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:45.304 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:01:45.304 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.307355+0000", 2024-05-15T13:01:45.304 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4829999999999999e-06 2024-05-15T13:01:45.304 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:45.304 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:45.304 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:01:45.304 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.307366+0000", 2024-05-15T13:01:45.305 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1219e-05 2024-05-15T13:01:45.305 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:45.305 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:45.305 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:01:45.305 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.307374+0000", 2024-05-15T13:01:45.305 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6680000000000001e-06 2024-05-15T13:01:45.305 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:45.305 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:45.305 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:01:45.305 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.309206+0000", 2024-05-15T13:01:45.305 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0018316140000000001 2024-05-15T13:01:45.305 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:01:45.305 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:01:45.305 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:01:45.305 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.309221+0000", 2024-05-15T13:01:45.305 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5642999999999999e-05 2024-05-15T13:01:45.306 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:45.306 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:01:45.306 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:45.306 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:01:45.306 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:01:45.306 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 1 2024-05-15T13:01:45.306 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:01:45.332 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:01:45.472 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:01:45.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:45.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:01:45.477 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:45 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1031345529' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:01:45.578 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:01:45.605 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:45 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1031345529' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:01:45.608 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:45.631 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:01:45.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:45.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:01:45.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:45 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1031345529' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:01:45.735 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:01:45.776 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:45.832 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:01:45.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:45.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:01:45.938 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:01:45.982 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:46.017 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:01:46.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:46.024 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:01:46.125 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:01:46.156 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:46.181 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:01:46.190 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:01:46.190 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:01:46.190 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:01:46.190 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:01:46.190 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:01:46.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:46.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:01:46.201 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:01:46.240 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:01:46.271 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:01:46.292 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:01:46.327 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:46 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:01:46.327 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:46 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1031345529' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:01:46.328 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:46 smithi177 ceph-mon[44303]: osdmap e387: 12 total, 12 up, 11 in 2024-05-15T13:01:46.328 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:46 smithi177 ceph-mon[44303]: pgmap v1050: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 647 op/s; 14437/126387 objects degraded (11.423%); 10082/126387 objects misplaced (7.977%); 4.6 MiB/s, 75 objects/s recovering 2024-05-15T13:01:46.328 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:46 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/443168169' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:01:46.357 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:46.507 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:01:46.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:46 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:01:46.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:46 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1031345529' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:01:46.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:46 smithi120 ceph-mon[43961]: osdmap e387: 12 total, 12 up, 11 in 2024-05-15T13:01:46.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:46 smithi120 ceph-mon[43961]: pgmap v1050: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 647 op/s; 14437/126387 objects degraded (11.423%); 10082/126387 objects misplaced (7.977%); 4.6 MiB/s, 75 objects/s recovering 2024-05-15T13:01:46.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:46 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/443168169' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:01:46.549 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:01:46.588 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:01:46.624 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:01:46.634 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:01:46.634 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:01:46.635 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:01:46.635 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:01:46.635 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:01:46.635 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:01:46.635 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:01:46.635 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:01:46.635 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:01:46.635 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:01:46.635 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:01:46.635 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:01:46.666 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:46 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:01:46.667 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:46 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1031345529' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:01:46.667 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:46 smithi027 ceph-mon[39795]: osdmap e387: 12 total, 12 up, 11 in 2024-05-15T13:01:46.667 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:46 smithi027 ceph-mon[39795]: pgmap v1050: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 647 op/s; 14437/126387 objects degraded (11.423%); 10082/126387 objects misplaced (7.977%); 4.6 MiB/s, 75 objects/s recovering 2024-05-15T13:01:46.667 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:46 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/443168169' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:01:49.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:48 smithi027 ceph-mon[39795]: pgmap v1051: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.5 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 641 op/s; 14549/121836 objects degraded (11.941%); 9350/121836 objects misplaced (7.674%); 9.4 MiB/s, 151 objects/s recovering 2024-05-15T13:01:49.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:48 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 14549/121836 objects degraded (11.941%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:49.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:48 smithi120 ceph-mon[43961]: pgmap v1051: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.5 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 641 op/s; 14549/121836 objects degraded (11.941%); 9350/121836 objects misplaced (7.674%); 9.4 MiB/s, 151 objects/s recovering 2024-05-15T13:01:49.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:48 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 14549/121836 objects degraded (11.941%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:49.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:48 smithi177 ceph-mon[44303]: pgmap v1051: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.5 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 641 op/s; 14549/121836 objects degraded (11.941%); 9350/121836 objects misplaced (7.674%); 9.4 MiB/s, 151 objects/s recovering 2024-05-15T13:01:49.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:48 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 14549/121836 objects degraded (11.941%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:49.574 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-05-15T13:01:50.560 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:01:50.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:50.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:01:50.668 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:01:50.702 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:50.730 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:01:50.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:50.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:01:50.836 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:01:50.867 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:50.900 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:01:50.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:50.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:01:50.972 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:50 smithi027 ceph-mon[39795]: pgmap v1052: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 14548/116637 objects degraded (12.473%); 9344/116637 objects misplaced (8.011%); 8.5 MiB/s, 137 objects/s recovering 2024-05-15T13:01:51.007 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:01:51.007 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:01:51.099 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:51.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:50 smithi120 ceph-mon[43961]: pgmap v1052: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 14548/116637 objects degraded (12.473%); 9344/116637 objects misplaced (8.011%); 8.5 MiB/s, 137 objects/s recovering 2024-05-15T13:01:51.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:50 smithi177 ceph-mon[44303]: pgmap v1052: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 14548/116637 objects degraded (12.473%); 9344/116637 objects misplaced (8.011%); 8.5 MiB/s, 137 objects/s recovering 2024-05-15T13:01:51.674 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:01:51.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:51.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:01:51.708 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:01:51.783 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:01:51.829 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:51.847 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:01:51.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:51.854 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:01:51.955 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:01:51.990 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:52.019 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:01:52.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:52.024 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:01:52.126 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:01:52.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:52 smithi177 ceph-mon[44303]: pgmap v1053: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 687 op/s; 14657/112782 objects degraded (12.996%); 8754/112782 objects misplaced (7.762%); 8.1 MiB/s, 130 objects/s recovering 2024-05-15T13:01:52.206 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg(s) [4.9] on osd.1 to force-backfill; 2024-05-15T13:01:52.207 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-05-15T13:01:52.215 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:52.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:52 smithi027 ceph-mon[39795]: pgmap v1053: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 687 op/s; 14657/112782 objects degraded (12.996%); 8754/112782 objects misplaced (7.762%); 8.1 MiB/s, 130 objects/s recovering 2024-05-15T13:01:52.239 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:01:52.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:52.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:01:52.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:52 smithi120 ceph-mon[43961]: pgmap v1053: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 687 op/s; 14657/112782 objects degraded (12.996%); 8754/112782 objects misplaced (7.762%); 8.1 MiB/s, 130 objects/s recovering 2024-05-15T13:01:52.346 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:01:52.391 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:52.419 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:01:52.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:52.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:01:52.527 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:01:52.565 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:52.614 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:01:52.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:52.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:01:52.721 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:01:53.056 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:01:53.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:53 smithi177 ceph-mon[44303]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "pg force-backfill", "pgid": ["4.9"], "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T13:01:53.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:53 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:01:53.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:53 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:01:53.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:53 smithi177 ceph-mon[44303]: osdmap e388: 12 total, 12 up, 11 in 2024-05-15T13:01:53.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:53 smithi027 ceph-mon[39795]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "pg force-backfill", "pgid": ["4.9"], "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T13:01:53.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:53 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:01:53.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:53 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:01:53.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:53 smithi027 ceph-mon[39795]: osdmap e388: 12 total, 12 up, 11 in 2024-05-15T13:01:53.550 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:53 smithi120 ceph-mon[43961]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "pg force-backfill", "pgid": ["4.9"], "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T13:01:53.550 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:53 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:01:53.551 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:53 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:01:53.551 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:53 smithi120 ceph-mon[43961]: osdmap e388: 12 total, 12 up, 11 in 2024-05-15T13:01:53.597 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:01:54.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:54 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 14657/112782 objects degraded (12.996%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:54.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:54 smithi027 ceph-mon[39795]: pgmap v1055: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 673 op/s; 14663/109695 objects degraded (13.367%); 8534/109695 objects misplaced (7.780%); 9.5 MiB/s, 154 objects/s recovering 2024-05-15T13:01:54.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:54 smithi027 ceph-mon[39795]: osdmap e389: 12 total, 12 up, 11 in 2024-05-15T13:01:54.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:54 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 14657/112782 objects degraded (12.996%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:54.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:54 smithi177 ceph-mon[44303]: pgmap v1055: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 673 op/s; 14663/109695 objects degraded (13.367%); 8534/109695 objects misplaced (7.780%); 9.5 MiB/s, 154 objects/s recovering 2024-05-15T13:01:54.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:54 smithi177 ceph-mon[44303]: osdmap e389: 12 total, 12 up, 11 in 2024-05-15T13:01:54.761 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:54 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 14657/112782 objects degraded (12.996%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:54.761 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:54 smithi120 ceph-mon[43961]: pgmap v1055: 17 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 673 op/s; 14663/109695 objects degraded (13.367%); 8534/109695 objects misplaced (7.780%); 9.5 MiB/s, 154 objects/s recovering 2024-05-15T13:01:54.761 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:54 smithi120 ceph-mon[43961]: osdmap e389: 12 total, 12 up, 11 in 2024-05-15T13:01:55.193 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:55.439 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:01:55.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:55 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:01:55.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:55 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:01:55.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:55.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:01:55.577 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:01:55.577 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:01:55.607 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:55 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:01:55.607 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:55 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:01:55.653 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:01:55.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:55 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:01:55.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:55 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:01:55.924 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:55.954 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:01:55.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:55.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:01:56.060 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:01:56.061 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:01:56.111 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:56.153 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:01:56.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:56.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:01:56.277 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:01:56.329 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:56.356 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:01:56.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:56.360 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:01:56.461 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:01:56.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:56 smithi027 ceph-mon[39795]: osdmap e390: 12 total, 12 up, 11 in 2024-05-15T13:01:56.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:56 smithi027 ceph-mon[39795]: pgmap v1058: 17 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 582 op/s; 14663/106149 objects degraded (13.814%); 8432/106149 objects misplaced (7.944%); 8.1 MiB/s, 132 objects/s recovering 2024-05-15T13:01:56.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:56 smithi177 ceph-mon[44303]: osdmap e390: 12 total, 12 up, 11 in 2024-05-15T13:01:56.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:56 smithi177 ceph-mon[44303]: pgmap v1058: 17 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 582 op/s; 14663/106149 objects degraded (13.814%); 8432/106149 objects misplaced (7.944%); 8.1 MiB/s, 132 objects/s recovering 2024-05-15T13:01:56.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:56 smithi120 ceph-mon[43961]: osdmap e390: 12 total, 12 up, 11 in 2024-05-15T13:01:56.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:56 smithi120 ceph-mon[43961]: pgmap v1058: 17 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 582 op/s; 14663/106149 objects degraded (13.814%); 8432/106149 objects misplaced (7.944%); 8.1 MiB/s, 132 objects/s recovering 2024-05-15T13:01:57.107 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:57.168 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:01:57.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:57.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:01:57.273 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:01:57.329 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:57.356 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:01:57.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:57.359 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:01:57.461 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:01:57.491 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:57.519 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:01:57.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:01:57.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:01:57.624 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:01:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:01:58.082 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:01:58.082 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 0 1 2024-05-15T13:01:58.358 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-05-15T13:01:59.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:58 smithi027 ceph-mon[39795]: pgmap v1059: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 593 op/s; 14715/102090 objects degraded (14.414%); 7948/102090 objects misplaced (7.785%); 7.6 MiB/s, 124 objects/s recovering 2024-05-15T13:01:59.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:58 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 14715/102090 objects degraded (14.414%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:59.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:58 smithi120 ceph-mon[43961]: pgmap v1059: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 593 op/s; 14715/102090 objects degraded (14.414%); 7948/102090 objects misplaced (7.785%); 7.6 MiB/s, 124 objects/s recovering 2024-05-15T13:01:59.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:58 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 14715/102090 objects degraded (14.414%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:01:59.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:58 smithi177 ceph-mon[44303]: pgmap v1059: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 593 op/s; 14715/102090 objects degraded (14.414%); 7948/102090 objects misplaced (7.785%); 7.6 MiB/s, 124 objects/s recovering 2024-05-15T13:01:59.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:58 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 14715/102090 objects degraded (14.414%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:00.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:01:59 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:02:00.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:01:59 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:02:00.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:01:59 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:02:00.467 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:02:00.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:00.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:02:00.578 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:02:00.598 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:02:00.611 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:00.638 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:02:00.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:00.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:02:00.745 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:02:00.786 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:00.817 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:02:00.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:00.821 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:02:00.922 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:02:00.970 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:00.994 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:02:01.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:01.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:02:01.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:00 smithi027 ceph-mon[39795]: pgmap v1060: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.0 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 713 op/s; 14713/96807 objects degraded (15.198%); 7793/96807 objects misplaced (8.050%); 6.2 MiB/s, 101 objects/s recovering 2024-05-15T13:02:01.113 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:02:01.156 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:01.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:00 smithi177 ceph-mon[44303]: pgmap v1060: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.0 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 713 op/s; 14713/96807 objects degraded (15.198%); 7793/96807 objects misplaced (8.050%); 6.2 MiB/s, 101 objects/s recovering 2024-05-15T13:02:01.200 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:02:01.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:01.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:02:01.212 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:02:01.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:00 smithi120 ceph-mon[43961]: pgmap v1060: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.0 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 713 op/s; 14713/96807 objects degraded (15.198%); 7793/96807 objects misplaced (8.050%); 6.2 MiB/s, 101 objects/s recovering 2024-05-15T13:02:01.308 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:02:01.339 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:01.375 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:02:01.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:01.383 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:02:01.484 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:02:01.533 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:01.588 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:02:01.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:01.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:02:01.700 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:02:01.745 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:01.780 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:02:01.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:01.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:02:01.889 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:02:01.975 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:02.012 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:02:02.052 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:02:02.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:02.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:02:02.161 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:02:02.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:02 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2122816313' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:02:02.240 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:02.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:02 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2122816313' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:02:02.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:02 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2122816313' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:02:02.525 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:02:02.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:02.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:02:02.629 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:02:02.689 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:03.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:03 smithi027 ceph-mon[39795]: pgmap v1061: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.9 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 701 op/s; 14820/92838 objects degraded (15.963%); 7566/92838 objects misplaced (8.150%); 6.3 MiB/s, 101 objects/s recovering 2024-05-15T13:02:03.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:03 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:02:03.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:03 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2122816313' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:02:03.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:03 smithi027 ceph-mon[39795]: osdmap e391: 12 total, 12 up, 11 in 2024-05-15T13:02:03.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:03 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3082850160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:02:03.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:03 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:02:03.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:03 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:02:03.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:03 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:02:03.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:03 smithi120 ceph-mon[43961]: pgmap v1061: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.9 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 701 op/s; 14820/92838 objects degraded (15.963%); 7566/92838 objects misplaced (8.150%); 6.3 MiB/s, 101 objects/s recovering 2024-05-15T13:02:03.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:03 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:02:03.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:03 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2122816313' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:02:03.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:03 smithi120 ceph-mon[43961]: osdmap e391: 12 total, 12 up, 11 in 2024-05-15T13:02:03.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:03 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3082850160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:02:03.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:03 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:02:03.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:03 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:02:03.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:03 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:02:03.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:03 smithi177 ceph-mon[44303]: pgmap v1061: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.9 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 701 op/s; 14820/92838 objects degraded (15.963%); 7566/92838 objects misplaced (8.150%); 6.3 MiB/s, 101 objects/s recovering 2024-05-15T13:02:03.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:03 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:02:03.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:03 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2122816313' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:02:03.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:03 smithi177 ceph-mon[44303]: osdmap e391: 12 total, 12 up, 11 in 2024-05-15T13:02:03.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:03 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3082850160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:02:03.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:03 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:02:03.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:03 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:02:03.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:03 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:02:04.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:04 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 14820/92838 objects degraded (15.963%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:04.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:04 smithi120 ceph-mon[43961]: pgmap v1063: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.8 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 666 op/s; 14815/89565 objects degraded (16.541%); 7363/89565 objects misplaced (8.221%); 6.6 MiB/s, 106 objects/s recovering 2024-05-15T13:02:04.384 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:04 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 14820/92838 objects degraded (15.963%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:04.384 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:04 smithi027 ceph-mon[39795]: pgmap v1063: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.8 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 666 op/s; 14815/89565 objects degraded (16.541%); 7363/89565 objects misplaced (8.221%); 6.6 MiB/s, 106 objects/s recovering 2024-05-15T13:02:04.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:04 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 14820/92838 objects degraded (15.963%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:04.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:04 smithi177 ceph-mon[44303]: pgmap v1063: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.8 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 666 op/s; 14815/89565 objects degraded (16.541%); 7363/89565 objects misplaced (8.221%); 6.6 MiB/s, 106 objects/s recovering 2024-05-15T13:02:06.612 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:02:06.612 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:02:06.612 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:02:06.612 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:02:06.612 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:02:06.612 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:02:06.613 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:02:06.613 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:02:06.613 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:02:06.613 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:02:06.613 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:02:06.613 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:02:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:02:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:02:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:02:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:02:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:02:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:02:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:02:06.614 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:02:06.614 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:02:06.614 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:02:06.614 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:02:06.614 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:02:06.840 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:02:06.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:06.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:02:06.949 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:02:07.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:06 smithi027 ceph-mon[39795]: pgmap v1064: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.7 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 678 op/s; 14815/85761 objects degraded (17.275%); 7217/85761 objects misplaced (8.415%); 6.6 MiB/s, 106 objects/s recovering 2024-05-15T13:02:07.030 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:07.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:06 smithi120 ceph-mon[43961]: pgmap v1064: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.7 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 678 op/s; 14815/85761 objects degraded (17.275%); 7217/85761 objects misplaced (8.415%); 6.6 MiB/s, 106 objects/s recovering 2024-05-15T13:02:07.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:06 smithi177 ceph-mon[44303]: pgmap v1064: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.7 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 678 op/s; 14815/85761 objects degraded (17.275%); 7217/85761 objects misplaced (8.415%); 6.6 MiB/s, 106 objects/s recovering 2024-05-15T13:02:08.396 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:02:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T13:02:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T13:02:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:02:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:87307 4.6 e333/317 4:60dfecb0:::benchmark_data_smithi177_99756_object87306:head v 333'5518, mlcod=333'5508)", 2024-05-15T13:02:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:58:57.965996+0000", 2024-05-15T13:02:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 188.77312798899999, 2024-05-15T13:02:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.025241053999999999, 2024-05-15T13:02:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:08.397 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:08.397 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.397 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:08.397 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.965996+0000", 2024-05-15T13:02:08.397 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.397 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.397 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.397 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:08.397 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.965996+0000", 2024-05-15T13:02:08.397 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.397 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.397 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.397 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:08.398 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.965996+0000", 2024-05-15T13:02:08.398 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7800000000000001e-07 2024-05-15T13:02:08.398 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.398 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.398 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:08.398 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.966039+0000", 2024-05-15T13:02:08.398 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3136000000000002e-05 2024-05-15T13:02:08.398 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.398 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.398 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:08.398 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.966045+0000", 2024-05-15T13:02:08.398 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.964e-06 2024-05-15T13:02:08.398 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.399 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.399 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:08.399 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.966049+0000", 2024-05-15T13:02:08.399 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8519999999999997e-06 2024-05-15T13:02:08.399 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.399 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.399 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:08.399 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.976673+0000", 2024-05-15T13:02:08.399 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010624368 2024-05-15T13:02:08.399 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.399 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.399 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:08.399 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.976689+0000", 2024-05-15T13:02:08.399 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5290000000000001e-05 2024-05-15T13:02:08.400 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.400 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.400 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:08.400 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.991191+0000", 2024-05-15T13:02:08.400 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014502474 2024-05-15T13:02:08.400 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.400 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.400 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:08.400 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.991237+0000", 2024-05-15T13:02:08.400 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5791999999999998e-05 2024-05-15T13:02:08.400 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.400 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:08.400 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.400 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.401 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.401 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:87362 4.6 e333/317 4:604ff663:::benchmark_data_smithi177_99756_object87361:head v 333'5519, mlcod=333'5508)", 2024-05-15T13:02:08.401 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:58:57.966057+0000", 2024-05-15T13:02:08.401 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 188.77306692400001, 2024-05-15T13:02:08.401 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.025213592999999999, 2024-05-15T13:02:08.401 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:08.401 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:08.401 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:08.401 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.401 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:08.401 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.966057+0000", 2024-05-15T13:02:08.401 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.401 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.402 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.402 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:08.402 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.966057+0000", 2024-05-15T13:02:08.402 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.402 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.402 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.402 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:08.402 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.966057+0000", 2024-05-15T13:02:08.402 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8300000000000001e-07 2024-05-15T13:02:08.402 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.402 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.402 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:08.402 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.966135+0000", 2024-05-15T13:02:08.402 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7404999999999995e-05 2024-05-15T13:02:08.403 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.403 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.403 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:08.403 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.966141+0000", 2024-05-15T13:02:08.403 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7880000000000001e-06 2024-05-15T13:02:08.403 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.403 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.403 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:08.403 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.966145+0000", 2024-05-15T13:02:08.403 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.721e-06 2024-05-15T13:02:08.403 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.403 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.403 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:08.403 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.977794+0000", 2024-05-15T13:02:08.404 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011649325 2024-05-15T13:02:08.404 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.404 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.404 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:08.404 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.977808+0000", 2024-05-15T13:02:08.404 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3895e-05 2024-05-15T13:02:08.404 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.404 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.404 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:08.404 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.991244+0000", 2024-05-15T13:02:08.404 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013435647 2024-05-15T13:02:08.405 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.405 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.405 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:08.405 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.991271+0000", 2024-05-15T13:02:08.405 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6628999999999999e-05 2024-05-15T13:02:08.405 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.405 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:08.405 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.405 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.405 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.405 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:87363 4.6 e333/317 4:61640be0:::benchmark_data_smithi177_99756_object87362:head v 333'5520, mlcod=333'5508)", 2024-05-15T13:02:08.405 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:58:57.966151+0000", 2024-05-15T13:02:08.406 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 188.77297321099999, 2024-05-15T13:02:08.406 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.025145239999999999, 2024-05-15T13:02:08.406 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:08.406 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:08.406 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:08.407 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.407 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:08.407 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.966151+0000", 2024-05-15T13:02:08.407 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.407 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.407 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.407 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:08.407 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.966151+0000", 2024-05-15T13:02:08.407 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.407 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.407 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.408 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:08.408 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.966151+0000", 2024-05-15T13:02:08.408 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.28e-07 2024-05-15T13:02:08.408 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.408 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.408 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:08.408 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.966185+0000", 2024-05-15T13:02:08.408 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4174000000000002e-05 2024-05-15T13:02:08.408 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.408 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.408 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:08.409 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.966191+0000", 2024-05-15T13:02:08.409 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7139999999999998e-06 2024-05-15T13:02:08.409 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.409 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.409 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:08.409 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.966194+0000", 2024-05-15T13:02:08.409 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.467e-06 2024-05-15T13:02:08.409 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.409 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.409 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:08.409 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.979376+0000", 2024-05-15T13:02:08.409 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013181788999999999 2024-05-15T13:02:08.410 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.410 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.410 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:08.410 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.979396+0000", 2024-05-15T13:02:08.410 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9959999999999999e-05 2024-05-15T13:02:08.410 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.410 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.410 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:08.410 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.991272+0000", 2024-05-15T13:02:08.410 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011876453 2024-05-15T13:02:08.410 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.410 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.410 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:08.410 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.991296+0000", 2024-05-15T13:02:08.411 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3455000000000002e-05 2024-05-15T13:02:08.411 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.411 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:08.411 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.411 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.411 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.411 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:87371 4.6 e333/317 4:6a9e1afb:::benchmark_data_smithi177_99756_object87370:head v 333'5521, mlcod=333'5508)", 2024-05-15T13:02:08.411 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:58:57.966199+0000", 2024-05-15T13:02:08.411 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 188.77292497799999, 2024-05-15T13:02:08.411 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.025122401999999999, 2024-05-15T13:02:08.411 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:08.411 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:08.411 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:08.412 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.412 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:08.412 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.966199+0000", 2024-05-15T13:02:08.412 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.412 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.412 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.412 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:08.412 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.966199+0000", 2024-05-15T13:02:08.412 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.412 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.412 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.412 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:08.412 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.966199+0000", 2024-05-15T13:02:08.412 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4600000000000001e-07 2024-05-15T13:02:08.413 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.413 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.413 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:08.413 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.966260+0000", 2024-05-15T13:02:08.413 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0547000000000002e-05 2024-05-15T13:02:08.413 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.413 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.413 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:08.413 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.966266+0000", 2024-05-15T13:02:08.413 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1550000000000004e-06 2024-05-15T13:02:08.413 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.413 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.413 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:08.413 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.966269+0000", 2024-05-15T13:02:08.413 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5949999999999999e-06 2024-05-15T13:02:08.414 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.414 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.414 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:08.414 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.981149+0000", 2024-05-15T13:02:08.414 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014879949 2024-05-15T13:02:08.415 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.415 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.415 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:08.415 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.981169+0000", 2024-05-15T13:02:08.415 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9695999999999999e-05 2024-05-15T13:02:08.415 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.415 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.415 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:08.415 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.991298+0000", 2024-05-15T13:02:08.416 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010128731 2024-05-15T13:02:08.416 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.416 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.416 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:08.416 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:57.991321+0000", 2024-05-15T13:02:08.416 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3583000000000001e-05 2024-05-15T13:02:08.416 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.416 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:08.416 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.416 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.416 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.417 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:110120 4.4 e347/317 4:22ec66ff:::benchmark_data_smithi177_99756_object110119:head v 347'6888, mlcod=347'6885)", 2024-05-15T13:02:08.417 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:59:25.006949+0000", 2024-05-15T13:02:08.417 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 161.732175259, 2024-05-15T13:02:08.417 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.033335848000000001, 2024-05-15T13:02:08.417 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:08.417 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:08.417 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:08.417 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.417 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:08.417 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:25.006949+0000", 2024-05-15T13:02:08.417 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.417 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.417 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.417 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:08.418 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:25.006949+0000", 2024-05-15T13:02:08.418 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.418 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.418 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.418 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:08.418 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:25.006949+0000", 2024-05-15T13:02:08.418 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8800000000000001e-07 2024-05-15T13:02:08.418 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.418 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.418 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:08.418 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:25.006975+0000", 2024-05-15T13:02:08.418 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5752999999999999e-05 2024-05-15T13:02:08.418 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.419 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.419 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:08.419 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:25.006993+0000", 2024-05-15T13:02:08.419 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7754e-05 2024-05-15T13:02:08.419 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.419 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.419 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:08.419 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:25.007002+0000", 2024-05-15T13:02:08.419 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9830000000000002e-06 2024-05-15T13:02:08.419 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.419 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.419 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:08.419 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:25.007114+0000", 2024-05-15T13:02:08.420 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000112137 2024-05-15T13:02:08.420 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.420 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.420 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:08.420 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:25.007139+0000", 2024-05-15T13:02:08.420 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4623000000000001e-05 2024-05-15T13:02:08.420 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.420 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.420 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:08.420 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:25.040214+0000", 2024-05-15T13:02:08.420 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.033075498000000002 2024-05-15T13:02:08.420 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.420 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.420 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:08.420 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:25.040284+0000", 2024-05-15T13:02:08.421 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0211999999999998e-05 2024-05-15T13:02:08.421 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.421 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:08.421 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.421 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.421 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.421 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:110121 4.4 e347/317 4:2a2a63cb:::benchmark_data_smithi177_99756_object110120:head v 347'6889, mlcod=347'6885)", 2024-05-15T13:02:08.421 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:59:25.014268+0000", 2024-05-15T13:02:08.421 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 161.72485612599999, 2024-05-15T13:02:08.421 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.026053028999999998, 2024-05-15T13:02:08.421 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:08.421 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:08.421 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:08.421 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.421 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:08.422 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:25.014268+0000", 2024-05-15T13:02:08.422 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.422 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.422 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.422 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:08.422 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:25.014268+0000", 2024-05-15T13:02:08.422 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.422 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.422 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.422 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:08.422 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:25.014269+0000", 2024-05-15T13:02:08.422 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2440000000000001e-06 2024-05-15T13:02:08.422 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.423 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.423 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:08.423 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:25.014301+0000", 2024-05-15T13:02:08.423 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2014999999999998e-05 2024-05-15T13:02:08.423 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.423 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.423 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:08.423 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:25.014314+0000", 2024-05-15T13:02:08.423 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2860999999999999e-05 2024-05-15T13:02:08.423 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.424 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.424 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:08.424 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:25.014323+0000", 2024-05-15T13:02:08.424 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6020000000000006e-06 2024-05-15T13:02:08.424 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.424 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.424 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:08.424 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:25.014375+0000", 2024-05-15T13:02:08.424 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1956999999999997e-05 2024-05-15T13:02:08.424 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.424 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.424 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:08.424 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:25.014397+0000", 2024-05-15T13:02:08.424 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1458e-05 2024-05-15T13:02:08.425 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.425 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.425 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:08.425 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:25.040291+0000", 2024-05-15T13:02:08.425 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.025894093 2024-05-15T13:02:08.425 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.425 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.425 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:08.425 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:25.040321+0000", 2024-05-15T13:02:08.425 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9799e-05 2024-05-15T13:02:08.425 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.425 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:08.425 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.425 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.425 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.426 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:116334 4.6 e349/317 4:66040612:::benchmark_data_smithi177_99756_object116333:head v 349'7302, mlcod=349'7300)", 2024-05-15T13:02:08.426 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:59:33.959479+0000", 2024-05-15T13:02:08.426 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 152.779644494, 2024-05-15T13:02:08.426 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.025902396000000001, 2024-05-15T13:02:08.426 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:08.426 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:08.426 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:08.426 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.426 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:08.426 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:33.959479+0000", 2024-05-15T13:02:08.426 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.426 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.426 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.426 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:08.427 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:33.959479+0000", 2024-05-15T13:02:08.427 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.427 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.427 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.427 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:08.427 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:33.959481+0000", 2024-05-15T13:02:08.427 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.207e-06 2024-05-15T13:02:08.427 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.427 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.427 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:08.428 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:33.959565+0000", 2024-05-15T13:02:08.428 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4046000000000001e-05 2024-05-15T13:02:08.428 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.428 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.428 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:08.428 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:33.959583+0000", 2024-05-15T13:02:08.428 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8770000000000002e-05 2024-05-15T13:02:08.428 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.428 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.428 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:08.428 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:33.959593+0000", 2024-05-15T13:02:08.428 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3000000000000007e-06 2024-05-15T13:02:08.428 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.428 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.428 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:08.428 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:33.963106+0000", 2024-05-15T13:02:08.429 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035133199999999999 2024-05-15T13:02:08.429 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.429 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.429 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:08.429 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:33.963136+0000", 2024-05-15T13:02:08.429 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0006e-05 2024-05-15T13:02:08.429 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.429 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.429 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:08.429 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:33.985314+0000", 2024-05-15T13:02:08.429 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.022178127999999998 2024-05-15T13:02:08.429 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.429 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.429 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:08.430 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:33.985382+0000", 2024-05-15T13:02:08.430 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7619000000000004e-05 2024-05-15T13:02:08.430 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.430 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:08.430 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.430 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.430 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.430 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:119213 4.6 e349/317 4:6858605c:::benchmark_data_smithi177_99756_object119212:head v 349'7488, mlcod=349'7486)", 2024-05-15T13:02:08.430 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:59:37.942666+0000", 2024-05-15T13:02:08.430 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 148.796457825, 2024-05-15T13:02:08.430 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.029070321999999999, 2024-05-15T13:02:08.430 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:08.430 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:08.431 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:08.431 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.431 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:08.431 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:37.942666+0000", 2024-05-15T13:02:08.431 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.431 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.431 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.431 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:08.431 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:37.942666+0000", 2024-05-15T13:02:08.431 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.431 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.431 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.431 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:08.431 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:37.942667+0000", 2024-05-15T13:02:08.431 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1009999999999999e-06 2024-05-15T13:02:08.432 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.432 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.432 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:08.432 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:37.942700+0000", 2024-05-15T13:02:08.432 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2911999999999999e-05 2024-05-15T13:02:08.432 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.432 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.432 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:08.432 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:37.942720+0000", 2024-05-15T13:02:08.432 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0418e-05 2024-05-15T13:02:08.432 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.433 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.433 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:08.433 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:37.942730+0000", 2024-05-15T13:02:08.433 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0899999999999994e-06 2024-05-15T13:02:08.433 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.433 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.433 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:08.433 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:37.945343+0000", 2024-05-15T13:02:08.433 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002613724 2024-05-15T13:02:08.433 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.433 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.433 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:08.433 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:37.945370+0000", 2024-05-15T13:02:08.433 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6664000000000001e-05 2024-05-15T13:02:08.434 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.434 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.434 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:08.434 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:37.971684+0000", 2024-05-15T13:02:08.434 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.026313847000000001 2024-05-15T13:02:08.434 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.434 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.434 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:08.434 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:37.971736+0000", 2024-05-15T13:02:08.434 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2565999999999998e-05 2024-05-15T13:02:08.434 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.434 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:08.434 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.435 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.435 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.435 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:144744 4.6 e359/317 4:60d0cfa1:::benchmark_data_smithi177_99756_object24772:head v 359'9085, mlcod=359'9083)", 2024-05-15T13:02:08.435 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:17.816758+0000", 2024-05-15T13:02:08.435 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 108.922365865, 2024-05-15T13:02:08.435 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.035170905000000002, 2024-05-15T13:02:08.435 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:08.435 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:08.435 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:08.435 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.435 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:08.435 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.816758+0000", 2024-05-15T13:02:08.436 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.436 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.436 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.436 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:08.436 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.816758+0000", 2024-05-15T13:02:08.436 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.436 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.436 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.436 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:08.436 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.816758+0000", 2024-05-15T13:02:08.436 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.39e-07 2024-05-15T13:02:08.436 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.436 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.436 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:08.437 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.816763+0000", 2024-05-15T13:02:08.437 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7389999999999999e-06 2024-05-15T13:02:08.437 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.437 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.437 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:08.437 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.816771+0000", 2024-05-15T13:02:08.437 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9629999999999998e-06 2024-05-15T13:02:08.437 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.437 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.437 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:08.437 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.816777+0000", 2024-05-15T13:02:08.438 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7860000000000002e-06 2024-05-15T13:02:08.438 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.438 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.438 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:08.438 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.816861+0000", 2024-05-15T13:02:08.438 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4109000000000004e-05 2024-05-15T13:02:08.438 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.438 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.438 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:08.438 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.816882+0000", 2024-05-15T13:02:08.438 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0880999999999999e-05 2024-05-15T13:02:08.438 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.438 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.439 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:08.439 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.851882+0000", 2024-05-15T13:02:08.439 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.035000336 2024-05-15T13:02:08.439 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.439 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.439 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:08.439 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.851929+0000", 2024-05-15T13:02:08.439 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6652e-05 2024-05-15T13:02:08.439 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.439 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:08.439 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.439 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.439 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.439 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:144748 4.6 e359/317 4:67fda673:::benchmark_data_smithi177_99756_object24776:head v 359'9086, mlcod=359'9083)", 2024-05-15T13:02:08.440 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:17.821396+0000", 2024-05-15T13:02:08.440 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 108.917727782, 2024-05-15T13:02:08.440 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.030563415, 2024-05-15T13:02:08.440 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:08.440 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:08.440 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:08.440 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.440 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:08.440 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.821396+0000", 2024-05-15T13:02:08.440 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.440 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.440 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.440 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:08.440 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.821396+0000", 2024-05-15T13:02:08.441 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.441 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.441 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.441 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:08.441 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.821398+0000", 2024-05-15T13:02:08.441 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4479999999999999e-06 2024-05-15T13:02:08.441 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.441 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.441 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:08.441 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.821410+0000", 2024-05-15T13:02:08.441 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2261e-05 2024-05-15T13:02:08.441 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.441 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.441 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:08.441 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.821427+0000", 2024-05-15T13:02:08.442 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7700999999999999e-05 2024-05-15T13:02:08.442 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.442 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.442 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:08.442 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.821441+0000", 2024-05-15T13:02:08.442 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3677e-05 2024-05-15T13:02:08.442 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.442 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.442 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:08.442 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.821918+0000", 2024-05-15T13:02:08.442 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00047655000000000001 2024-05-15T13:02:08.442 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.442 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.443 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:08.443 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.821938+0000", 2024-05-15T13:02:08.443 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0015000000000001e-05 2024-05-15T13:02:08.443 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.443 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.443 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:08.443 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.851933+0000", 2024-05-15T13:02:08.443 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.029995432999999998 2024-05-15T13:02:08.443 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.443 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.443 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:08.443 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.851959+0000", 2024-05-15T13:02:08.443 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.633e-05 2024-05-15T13:02:08.443 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.443 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:08.444 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.444 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.444 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.444 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:146450 4.6 e359/317 4:6f026166:::benchmark_data_smithi177_99756_object26478:head v 359'9183, mlcod=359'9181)", 2024-05-15T13:02:08.444 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:20.772928+0000", 2024-05-15T13:02:08.444 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 105.966196242, 2024-05-15T13:02:08.444 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.033370865, 2024-05-15T13:02:08.444 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:08.444 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:08.444 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:08.444 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.444 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:08.444 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.772928+0000", 2024-05-15T13:02:08.444 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.445 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.445 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.445 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:08.445 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.772928+0000", 2024-05-15T13:02:08.445 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.445 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.445 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.445 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:08.445 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.772929+0000", 2024-05-15T13:02:08.445 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4899999999999999e-06 2024-05-15T13:02:08.445 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.445 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.445 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:08.445 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.772939+0000", 2024-05-15T13:02:08.446 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5279999999999992e-06 2024-05-15T13:02:08.446 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.446 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.446 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:08.446 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.772950+0000", 2024-05-15T13:02:08.446 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1392999999999999e-05 2024-05-15T13:02:08.446 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.446 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.446 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:08.446 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.772959+0000", 2024-05-15T13:02:08.446 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7669999999999992e-06 2024-05-15T13:02:08.446 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.446 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.446 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:08.447 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.776408+0000", 2024-05-15T13:02:08.447 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034495939999999998 2024-05-15T13:02:08.447 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.447 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.447 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:08.447 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.776434+0000", 2024-05-15T13:02:08.447 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5723000000000001e-05 2024-05-15T13:02:08.447 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.447 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.447 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:08.448 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.806241+0000", 2024-05-15T13:02:08.448 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.029806928 2024-05-15T13:02:08.448 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.448 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.448 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:08.448 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.806298+0000", 2024-05-15T13:02:08.448 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7442000000000003e-05 2024-05-15T13:02:08.448 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.449 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:08.449 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.449 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.449 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.449 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:146550 4.4 e359/317 4:2b320cd4:::benchmark_data_smithi177_99756_object26578:head v 359'9116, mlcod=359'9113)", 2024-05-15T13:02:08.449 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:20.932570+0000", 2024-05-15T13:02:08.450 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 105.80655391099999, 2024-05-15T13:02:08.450 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.029031403000000001, 2024-05-15T13:02:08.450 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:08.450 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:08.450 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:08.450 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.451 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:08.451 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.932570+0000", 2024-05-15T13:02:08.451 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.451 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.451 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.451 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:08.452 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.932570+0000", 2024-05-15T13:02:08.452 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.452 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.452 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.452 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:08.452 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.932570+0000", 2024-05-15T13:02:08.452 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3900000000000001e-07 2024-05-15T13:02:08.453 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.453 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.453 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:08.453 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.932574+0000", 2024-05-15T13:02:08.453 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.371e-06 2024-05-15T13:02:08.453 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.454 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.454 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:08.454 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.932577+0000", 2024-05-15T13:02:08.454 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7510000000000002e-06 2024-05-15T13:02:08.454 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.454 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.455 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:08.455 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.932580+0000", 2024-05-15T13:02:08.455 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9119999999999998e-06 2024-05-15T13:02:08.455 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.455 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.455 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:08.455 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.937747+0000", 2024-05-15T13:02:08.456 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0051663350000000002 2024-05-15T13:02:08.456 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.456 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.456 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:08.456 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.937775+0000", 2024-05-15T13:02:08.456 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8109000000000001e-05 2024-05-15T13:02:08.457 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.457 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.457 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:08.457 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.961545+0000", 2024-05-15T13:02:08.457 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.023770067999999998 2024-05-15T13:02:08.457 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.458 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.458 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:08.458 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.961601+0000", 2024-05-15T13:02:08.458 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6617999999999999e-05 2024-05-15T13:02:08.458 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.458 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:08.459 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.459 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.459 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.459 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:164320 4.6 e368/317 4:66ad1edd:::benchmark_data_smithi177_99756_object44348:head v 368'10327, mlcod=368'10321)", 2024-05-15T13:02:08.459 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:49.639844+0000", 2024-05-15T13:02:08.459 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 77.099279766999999, 2024-05-15T13:02:08.459 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.034496188999999997, 2024-05-15T13:02:08.459 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:08.459 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:08.460 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:08.460 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.460 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:08.460 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:49.639844+0000", 2024-05-15T13:02:08.460 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.460 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.461 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.461 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:08.461 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:49.639844+0000", 2024-05-15T13:02:08.461 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.461 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.461 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.461 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:08.462 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:49.639845+0000", 2024-05-15T13:02:08.462 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.175e-06 2024-05-15T13:02:08.462 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.462 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.462 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:08.462 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:49.639855+0000", 2024-05-15T13:02:08.463 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0045e-05 2024-05-15T13:02:08.463 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.463 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.463 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:08.463 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:49.639869+0000", 2024-05-15T13:02:08.463 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3883000000000001e-05 2024-05-15T13:02:08.464 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.464 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.464 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:08.464 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:49.639881+0000", 2024-05-15T13:02:08.464 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1816e-05 2024-05-15T13:02:08.465 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.465 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.465 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:08.465 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:49.643737+0000", 2024-05-15T13:02:08.465 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003855724 2024-05-15T13:02:08.465 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.466 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.466 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:08.466 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:49.643753+0000", 2024-05-15T13:02:08.466 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5835e-05 2024-05-15T13:02:08.466 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.467 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.467 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:08.467 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:49.674304+0000", 2024-05-15T13:02:08.467 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.030551433999999999 2024-05-15T13:02:08.467 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.467 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.468 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:08.468 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:49.674340+0000", 2024-05-15T13:02:08.468 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6276999999999998e-05 2024-05-15T13:02:08.468 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.468 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:08.468 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.469 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.469 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.469 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:165809 4.4 e368/317 4:25083c30:::benchmark_data_smithi177_99756_object45837:head v 368'10336, mlcod=368'10334)", 2024-05-15T13:02:08.469 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:51.922523+0000", 2024-05-15T13:02:08.469 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 74.816600608000002, 2024-05-15T13:02:08.469 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.028920524, 2024-05-15T13:02:08.470 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:08.470 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:08.470 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:08.470 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.470 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:08.471 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:51.922523+0000", 2024-05-15T13:02:08.471 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.471 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.471 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.471 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:08.471 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:51.922523+0000", 2024-05-15T13:02:08.472 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.472 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.472 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.472 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:08.472 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:51.922525+0000", 2024-05-15T13:02:08.472 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.322e-06 2024-05-15T13:02:08.473 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.473 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.473 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:08.473 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:51.922537+0000", 2024-05-15T13:02:08.473 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2585999999999999e-05 2024-05-15T13:02:08.473 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.474 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.474 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:08.474 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:51.922552+0000", 2024-05-15T13:02:08.474 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4987999999999999e-05 2024-05-15T13:02:08.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.474 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.475 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:08.475 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:51.922562+0000", 2024-05-15T13:02:08.475 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6830000000000003e-06 2024-05-15T13:02:08.475 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.475 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.475 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:08.476 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:51.922638+0000", 2024-05-15T13:02:08.476 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6314000000000002e-05 2024-05-15T13:02:08.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.476 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.476 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:08.476 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:51.922662+0000", 2024-05-15T13:02:08.477 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3643999999999999e-05 2024-05-15T13:02:08.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.477 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.477 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:08.477 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:51.951389+0000", 2024-05-15T13:02:08.478 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.028727505 2024-05-15T13:02:08.478 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.478 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:08.478 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:51.951444+0000", 2024-05-15T13:02:08.479 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4481999999999998e-05 2024-05-15T13:02:08.479 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.479 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:08.479 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.480 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:171259 4.4 e372/317 4:2d6d331d:::benchmark_data_smithi177_99756_object51287:head v 372'10696, mlcod=372'10690)", 2024-05-15T13:02:08.480 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:00.631420+0000", 2024-05-15T13:02:08.480 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 66.107703709999996, 2024-05-15T13:02:08.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.027309077000000001, 2024-05-15T13:02:08.480 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:08.480 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:08.481 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:08.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:08.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:00.631420+0000", 2024-05-15T13:02:08.481 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.482 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:08.482 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:00.631420+0000", 2024-05-15T13:02:08.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.482 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:08.482 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:00.631421+0000", 2024-05-15T13:02:08.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1820000000000001e-06 2024-05-15T13:02:08.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.483 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:08.483 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:00.631430+0000", 2024-05-15T13:02:08.483 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9150000000000002e-06 2024-05-15T13:02:08.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.484 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:08.484 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:00.631442+0000", 2024-05-15T13:02:08.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2211999999999999e-05 2024-05-15T13:02:08.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.484 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:08.484 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:00.631451+0000", 2024-05-15T13:02:08.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.259e-06 2024-05-15T13:02:08.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:08.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:00.635370+0000", 2024-05-15T13:02:08.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039193789999999997 2024-05-15T13:02:08.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:08.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:00.635400+0000", 2024-05-15T13:02:08.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9781000000000002e-05 2024-05-15T13:02:08.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:08.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:00.658680+0000", 2024-05-15T13:02:08.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.023280193000000001 2024-05-15T13:02:08.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:08.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:00.658729+0000", 2024-05-15T13:02:08.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9156000000000002e-05 2024-05-15T13:02:08.486 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.486 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:08.486 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.487 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:171988 4.6 e372/317 4:68c3b4d4:::benchmark_data_smithi177_99756_object52016:head v 372'10800, mlcod=372'10798)", 2024-05-15T13:02:08.487 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:01.843170+0000", 2024-05-15T13:02:08.487 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 64.895954017999998, 2024-05-15T13:02:08.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.032529491000000001, 2024-05-15T13:02:08.487 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:08.487 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:08.487 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:08.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:08.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:01.843170+0000", 2024-05-15T13:02:08.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:08.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:01.843170+0000", 2024-05-15T13:02:08.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:08.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:01.843171+0000", 2024-05-15T13:02:08.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0079999999999999e-06 2024-05-15T13:02:08.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:08.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:01.843179+0000", 2024-05-15T13:02:08.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1089999999999998e-06 2024-05-15T13:02:08.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:08.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:01.843200+0000", 2024-05-15T13:02:08.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0866e-05 2024-05-15T13:02:08.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:08.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:01.843211+0000", 2024-05-15T13:02:08.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1205e-05 2024-05-15T13:02:08.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:08.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:01.847325+0000", 2024-05-15T13:02:08.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041143129999999997 2024-05-15T13:02:08.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:08.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:01.847351+0000", 2024-05-15T13:02:08.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.605e-05 2024-05-15T13:02:08.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:08.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:01.875646+0000", 2024-05-15T13:02:08.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.028294369999999999 2024-05-15T13:02:08.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:08.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:01.875699+0000", 2024-05-15T13:02:08.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3569999999999997e-05 2024-05-15T13:02:08.491 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.491 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:08.491 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.494 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:171991 4.6 e372/317 4:60d59694:::benchmark_data_smithi177_99756_object52019:head v 372'10801, mlcod=372'10798)", 2024-05-15T13:02:08.494 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:01.844498+0000", 2024-05-15T13:02:08.494 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 64.894625740999999, 2024-05-15T13:02:08.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.031232314000000001, 2024-05-15T13:02:08.494 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:08.494 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:08.494 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:08.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:08.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:01.844498+0000", 2024-05-15T13:02:08.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:08.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:01.844498+0000", 2024-05-15T13:02:08.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:08.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:01.844499+0000", 2024-05-15T13:02:08.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8200000000000008e-07 2024-05-15T13:02:08.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:08.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:01.844507+0000", 2024-05-15T13:02:08.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2670000000000006e-06 2024-05-15T13:02:08.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:08.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:01.844526+0000", 2024-05-15T13:02:08.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8267e-05 2024-05-15T13:02:08.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:08.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:01.844534+0000", 2024-05-15T13:02:08.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7940000000000008e-06 2024-05-15T13:02:08.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:08.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:01.854776+0000", 2024-05-15T13:02:08.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.01024166 2024-05-15T13:02:08.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:08.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:01.854805+0000", 2024-05-15T13:02:08.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9377999999999999e-05 2024-05-15T13:02:08.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:08.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:01.875703+0000", 2024-05-15T13:02:08.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.020897402999999998 2024-05-15T13:02:08.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:08.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:01.875730+0000", 2024-05-15T13:02:08.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7563e-05 2024-05-15T13:02:08.498 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.498 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:08.498 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.498 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:193966 4.6 e385/317 4:6771ea98:::benchmark_data_smithi177_99756_object73994:head v 385'12232, mlcod=385'12230)", 2024-05-15T13:02:08.498 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:37.427669+0000", 2024-05-15T13:02:08.498 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 29.311455134999999, 2024-05-15T13:02:08.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.027656854000000002, 2024-05-15T13:02:08.499 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:08.499 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:08.499 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:08.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:08.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:37.427669+0000", 2024-05-15T13:02:08.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:08.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:37.427669+0000", 2024-05-15T13:02:08.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:08.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:37.427674+0000", 2024-05-15T13:02:08.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4859999999999997e-06 2024-05-15T13:02:08.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:08.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:37.427687+0000", 2024-05-15T13:02:08.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3151e-05 2024-05-15T13:02:08.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:08.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:37.427710+0000", 2024-05-15T13:02:08.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2979000000000001e-05 2024-05-15T13:02:08.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:08.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:37.427721+0000", 2024-05-15T13:02:08.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1138000000000001e-05 2024-05-15T13:02:08.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:08.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:37.427787+0000", 2024-05-15T13:02:08.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5302000000000006e-05 2024-05-15T13:02:08.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:08.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:37.427808+0000", 2024-05-15T13:02:08.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1611e-05 2024-05-15T13:02:08.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:08.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:37.455280+0000", 2024-05-15T13:02:08.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.027471643 2024-05-15T13:02:08.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:08.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:37.455326+0000", 2024-05-15T13:02:08.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5544000000000003e-05 2024-05-15T13:02:08.502 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.502 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:08.502 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.503 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:193971 4.6 e385/317 4:699748d5:::benchmark_data_smithi177_99756_object73999:head v 385'12233, mlcod=385'12230)", 2024-05-15T13:02:08.503 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:37.429713+0000", 2024-05-15T13:02:08.503 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 29.309410963000001, 2024-05-15T13:02:08.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.025633631, 2024-05-15T13:02:08.503 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:08.503 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:08.503 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:08.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:08.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:37.429713+0000", 2024-05-15T13:02:08.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:08.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:37.429713+0000", 2024-05-15T13:02:08.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:08.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:37.429714+0000", 2024-05-15T13:02:08.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7700000000000004e-07 2024-05-15T13:02:08.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:08.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:37.429723+0000", 2024-05-15T13:02:08.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8730000000000002e-06 2024-05-15T13:02:08.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:08.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:37.429734+0000", 2024-05-15T13:02:08.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0789e-05 2024-05-15T13:02:08.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:08.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:37.429741+0000", 2024-05-15T13:02:08.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.759e-06 2024-05-15T13:02:08.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:08.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:37.433014+0000", 2024-05-15T13:02:08.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032730490000000001 2024-05-15T13:02:08.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:08.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:37.433040+0000", 2024-05-15T13:02:08.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6159000000000001e-05 2024-05-15T13:02:08.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:08.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:37.455328+0000", 2024-05-15T13:02:08.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.022287466999999998 2024-05-15T13:02:08.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:08.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:37.455347+0000", 2024-05-15T13:02:08.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8858000000000001e-05 2024-05-15T13:02:08.507 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.507 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:08.508 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.508 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:199242 4.4 e386/317 4:2863b6bb:::benchmark_data_smithi177_99756_object79270:head v 386'12474, mlcod=386'12471)", 2024-05-15T13:02:08.508 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:45.199737+0000", 2024-05-15T13:02:08.508 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 21.539387323, 2024-05-15T13:02:08.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.026647741999999999, 2024-05-15T13:02:08.508 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:08.508 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:08.508 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:08.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.508 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:08.508 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.199737+0000", 2024-05-15T13:02:08.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:08.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.199737+0000", 2024-05-15T13:02:08.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:08.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:08.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.199737+0000", 2024-05-15T13:02:08.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2399999999999999e-07 2024-05-15T13:02:08.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:08.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.199741+0000", 2024-05-15T13:02:08.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5539999999999999e-06 2024-05-15T13:02:08.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:08.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.199748+0000", 2024-05-15T13:02:08.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6279999999999999e-06 2024-05-15T13:02:08.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:08.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.199753+0000", 2024-05-15T13:02:08.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1410000000000004e-06 2024-05-15T13:02:08.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:08.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.209959+0000", 2024-05-15T13:02:08.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010205907 2024-05-15T13:02:08.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:08.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.209981+0000", 2024-05-15T13:02:08.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2225000000000002e-05 2024-05-15T13:02:08.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:08.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.224132+0000", 2024-05-15T13:02:08.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014150881000000001 2024-05-15T13:02:08.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:08.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:08.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:08.512 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.226384+0000", 2024-05-15T13:02:08.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0022520819999999999 2024-05-15T13:02:08.512 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.512 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:08.512 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.512 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:08.512 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:08.512 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:02:08.607 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:02:08.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:08.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:02:08.712 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:02:08.781 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:08.826 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:02:08.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:08.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:02:08.930 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:02:08.965 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:09.001 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:02:09.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:09.002 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:02:09.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:09 smithi120 ceph-mon[43961]: pgmap v1065: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 1.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 699 op/s; 14921/81066 objects degraded (18.406%); 6773/81066 objects misplaced (8.355%); 6.0 MiB/s, 96 objects/s recovering 2024-05-15T13:02:09.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:09 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 14921/81066 objects degraded (18.406%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:09.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:09 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3970525414' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:02:09.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:09 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:02:09.105 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:02:09.156 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:09.180 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:02:09.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:09.184 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:02:09.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:09 smithi027 ceph-mon[39795]: pgmap v1065: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 1.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 699 op/s; 14921/81066 objects degraded (18.406%); 6773/81066 objects misplaced (8.355%); 6.0 MiB/s, 96 objects/s recovering 2024-05-15T13:02:09.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:09 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 14921/81066 objects degraded (18.406%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:09.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:09 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3970525414' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:02:09.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:09 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:02:09.285 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:02:09.289 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.0 to 1 (10000) 2024-05-15T13:02:09.314 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-05-15T13:02:09.321 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:09 smithi177 ceph-mon[44303]: pgmap v1065: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 1.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 699 op/s; 14921/81066 objects degraded (18.406%); 6773/81066 objects misplaced (8.355%); 6.0 MiB/s, 96 objects/s recovering 2024-05-15T13:02:09.321 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:09 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 14921/81066 objects degraded (18.406%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:09.321 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:09 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3970525414' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:02:09.321 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:09 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:02:09.330 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:09.366 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:02:09.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:09.373 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:02:09.474 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:02:09.518 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:09.559 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:02:09.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:09.564 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:02:09.665 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:02:09.729 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:09.944 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:02:09.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:09.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:02:10.048 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:02:10.099 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:10.145 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:02:10.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:10.156 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:02:10.211 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:02:10.256 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:02:10.290 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:10.329 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:02:10.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:10.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:02:10.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:10 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T13:02:10.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:10 smithi177 ceph-mon[44303]: osdmap e392: 12 total, 12 up, 11 in 2024-05-15T13:02:10.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:10 smithi177 ceph-mon[44303]: pgmap v1067: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.5 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 713 op/s; 14920/75696 objects degraded (19.710%); 6623/75696 objects misplaced (8.749%); 6.5 MiB/s, 105 objects/s recovering 2024-05-15T13:02:10.435 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:02:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:02:10.445 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 1 1 2024-05-15T13:02:10.470 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:10.502 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:02:10.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:10.507 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:02:10.527 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:10 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T13:02:10.527 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:10 smithi027 ceph-mon[39795]: osdmap e392: 12 total, 12 up, 11 in 2024-05-15T13:02:10.527 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:10 smithi027 ceph-mon[39795]: pgmap v1067: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.5 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 713 op/s; 14920/75696 objects degraded (19.710%); 6623/75696 objects misplaced (8.749%); 6.5 MiB/s, 105 objects/s recovering 2024-05-15T13:02:10.608 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:02:10.685 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:10.751 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:02:10.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:10.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:02:10.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:10 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T13:02:10.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:10 smithi120 ceph-mon[43961]: osdmap e392: 12 total, 12 up, 11 in 2024-05-15T13:02:10.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:10 smithi120 ceph-mon[43961]: pgmap v1067: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.5 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 713 op/s; 14920/75696 objects degraded (19.710%); 6623/75696 objects misplaced (8.749%); 6.5 MiB/s, 105 objects/s recovering 2024-05-15T13:02:10.859 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:02:10.900 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:10.924 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:02:10.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:10.928 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:02:11.029 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:02:11.068 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:11.113 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:02:11.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:11.116 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:02:11.217 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:02:11.261 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:11.298 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:02:11.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:11.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:02:11.402 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:02:11.452 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:11.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:11 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:02:11.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:11 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:02:11.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:11 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:02:11.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:11 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:02:11.701 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:02:11.745 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:11 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:02:11.745 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:11 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:02:12.516 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:12 smithi027 ceph-mon[39795]: osdmap e393: 12 total, 12 up, 11 in 2024-05-15T13:02:12.517 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:12 smithi027 ceph-mon[39795]: pgmap v1069: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.5 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 742 op/s; 15021/71721 objects degraded (20.944%); 6396/71721 objects misplaced (8.918%); 5.9 MiB/s, 95 objects/s recovering 2024-05-15T13:02:12.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:12 smithi177 ceph-mon[44303]: osdmap e393: 12 total, 12 up, 11 in 2024-05-15T13:02:12.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:12 smithi177 ceph-mon[44303]: pgmap v1069: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.5 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 742 op/s; 15021/71721 objects degraded (20.944%); 6396/71721 objects misplaced (8.918%); 5.9 MiB/s, 95 objects/s recovering 2024-05-15T13:02:12.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:12 smithi120 ceph-mon[43961]: osdmap e393: 12 total, 12 up, 11 in 2024-05-15T13:02:12.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:12 smithi120 ceph-mon[43961]: pgmap v1069: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.5 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 742 op/s; 15021/71721 objects degraded (20.944%); 6396/71721 objects misplaced (8.918%); 5.9 MiB/s, 95 objects/s recovering 2024-05-15T13:02:12.957 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:02:12.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:12.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:02:12.977 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:02:12.977 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:02:13.061 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:02:13.062 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:02:13.122 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:13.336 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:02:13.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:13.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:02:13.443 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:02:13.507 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:13.537 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:02:13.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:13.541 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:02:13.616 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:13 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 15021/71721 objects degraded (20.944%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:13.617 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:13 smithi027 ceph-mon[39795]: osdmap e394: 12 total, 12 up, 11 in 2024-05-15T13:02:13.641 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:02:13.668 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:13 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 15021/71721 objects degraded (20.944%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:13.668 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:13 smithi120 ceph-mon[43961]: osdmap e394: 12 total, 12 up, 11 in 2024-05-15T13:02:13.675 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:13.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:13 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 15021/71721 objects degraded (20.944%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:13.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:13 smithi177 ceph-mon[44303]: osdmap e394: 12 total, 12 up, 11 in 2024-05-15T13:02:13.706 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:02:13.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:13.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:02:13.812 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:02:13.848 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:13.883 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:02:13.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:13.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:02:13.987 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:02:14.026 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:14.057 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:02:14.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:14.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:02:14.163 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:02:14.255 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:14.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:14 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:02:14.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:14 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:02:14.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:14 smithi027 ceph-mon[39795]: pgmap v1071: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.4 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 705 op/s; 15014/68364 objects degraded (21.962%); 6160/68364 objects misplaced (9.011%); 5.3 MiB/s, 86 objects/s recovering 2024-05-15T13:02:14.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:14 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:02:14.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:14 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:02:14.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:14 smithi177 ceph-mon[44303]: pgmap v1071: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.4 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 705 op/s; 15014/68364 objects degraded (21.962%); 6160/68364 objects misplaced (9.011%); 5.3 MiB/s, 86 objects/s recovering 2024-05-15T13:02:14.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:14 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:02:14.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:14 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:02:14.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:14 smithi120 ceph-mon[43961]: pgmap v1071: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.4 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 705 op/s; 15014/68364 objects degraded (21.962%); 6160/68364 objects misplaced (9.011%); 5.3 MiB/s, 86 objects/s recovering 2024-05-15T13:02:15.126 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:02:15.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:15.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:02:15.234 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:02:15.304 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:15.456 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:02:15.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:15.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:02:15.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:15 smithi027 ceph-mon[39795]: osdmap e395: 12 total, 12 up, 11 in 2024-05-15T13:02:15.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:15 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:02:15.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:15 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:02:15.564 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:02:15.608 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:15.641 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:02:15.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:15.646 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:02:15.647 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:15 smithi177 ceph-mon[44303]: osdmap e395: 12 total, 12 up, 11 in 2024-05-15T13:02:15.647 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:15 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:02:15.647 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:15 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:02:15.747 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:02:15.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:15 smithi120 ceph-mon[43961]: osdmap e395: 12 total, 12 up, 11 in 2024-05-15T13:02:15.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:15 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:02:15.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:15 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:02:15.812 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:16.022 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:02:16.023 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T13:02:16.023 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-05-15T13:02:16.023 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:02:16.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:16 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:02:16.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:16 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:02:16.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:16 smithi177 ceph-mon[44303]: osdmap e396: 12 total, 12 up, 11 in 2024-05-15T13:02:16.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:16 smithi177 ceph-mon[44303]: pgmap v1074: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.3 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 575 op/s; 15014/64485 objects degraded (23.283%); 5997/64485 objects misplaced (9.300%); 5.4 MiB/s, 88 objects/s recovering 2024-05-15T13:02:16.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:16 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:02:16.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:16 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:02:16.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:16 smithi027 ceph-mon[39795]: osdmap e396: 12 total, 12 up, 11 in 2024-05-15T13:02:16.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:16 smithi027 ceph-mon[39795]: pgmap v1074: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.3 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 575 op/s; 15014/64485 objects degraded (23.283%); 5997/64485 objects misplaced (9.300%); 5.4 MiB/s, 88 objects/s recovering 2024-05-15T13:02:16.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:16 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:02:16.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:16 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:02:16.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:16 smithi120 ceph-mon[43961]: osdmap e396: 12 total, 12 up, 11 in 2024-05-15T13:02:16.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:16 smithi120 ceph-mon[43961]: pgmap v1074: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.3 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 575 op/s; 15014/64485 objects degraded (23.283%); 5997/64485 objects misplaced (9.300%); 5.4 MiB/s, 88 objects/s recovering 2024-05-15T13:02:17.702 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:02:17.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:17.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:02:17.815 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:02:17.854 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:17.885 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:02:17.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:17.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:02:17.989 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:02:18.030 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:18.059 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:02:18.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:18.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:02:18.165 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:02:18.258 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:18.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:18 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:02:18.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:18 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:02:18.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:18 smithi177 ceph-mon[44303]: osdmap e397: 12 total, 12 up, 11 in 2024-05-15T13:02:18.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:18 smithi177 ceph-mon[44303]: pgmap v1076: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 1.2 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 772 op/s; 14993/59730 objects degraded (25.101%); 5635/59730 objects misplaced (9.434%); 5.7 MiB/s, 119 objects/s recovering 2024-05-15T13:02:18.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:18 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 15014/64485 objects degraded (23.283%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:18.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:18 smithi177 ceph-mon[44303]: osdmap e398: 12 total, 12 up, 11 in 2024-05-15T13:02:19.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:18 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:02:19.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:18 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:02:19.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:18 smithi027 ceph-mon[39795]: osdmap e397: 12 total, 12 up, 11 in 2024-05-15T13:02:19.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:18 smithi027 ceph-mon[39795]: pgmap v1076: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 1.2 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 772 op/s; 14993/59730 objects degraded (25.101%); 5635/59730 objects misplaced (9.434%); 5.7 MiB/s, 119 objects/s recovering 2024-05-15T13:02:19.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:18 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 15014/64485 objects degraded (23.283%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:19.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:18 smithi027 ceph-mon[39795]: osdmap e398: 12 total, 12 up, 11 in 2024-05-15T13:02:19.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:18 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:02:19.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:18 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:02:19.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:18 smithi120 ceph-mon[43961]: osdmap e397: 12 total, 12 up, 11 in 2024-05-15T13:02:19.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:18 smithi120 ceph-mon[43961]: pgmap v1076: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 1.2 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 772 op/s; 14993/59730 objects degraded (25.101%); 5635/59730 objects misplaced (9.434%); 5.7 MiB/s, 119 objects/s recovering 2024-05-15T13:02:19.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:18 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 15014/64485 objects degraded (23.283%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:19.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:18 smithi120 ceph-mon[43961]: osdmap e398: 12 total, 12 up, 11 in 2024-05-15T13:02:19.919 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-05-15T13:02:20.160 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:02:20.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:20.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:02:20.269 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:02:20.394 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:02:20.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:20.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:02:21.010 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:02:21.085 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:02:21.116 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:21.146 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:02:21.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:21.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:02:21.228 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:02:21.251 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:02:21.269 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:02:21.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:21 smithi027 ceph-mon[39795]: pgmap v1078: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.1 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 806 op/s; 14992/54291 objects degraded (27.614%); 5476/54291 objects misplaced (10.086%); 5.6 MiB/s, 127 objects/s recovering 2024-05-15T13:02:21.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:21 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/711791825' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:02:21.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:21 smithi120 ceph-mon[43961]: pgmap v1078: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.1 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 806 op/s; 14992/54291 objects degraded (27.614%); 5476/54291 objects misplaced (10.086%); 5.6 MiB/s, 127 objects/s recovering 2024-05-15T13:02:21.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:21 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/711791825' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:02:21.281 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:21 smithi177 ceph-mon[44303]: pgmap v1078: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.1 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 806 op/s; 14992/54291 objects degraded (27.614%); 5476/54291 objects misplaced (10.086%); 5.6 MiB/s, 127 objects/s recovering 2024-05-15T13:02:21.281 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:21 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/711791825' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:02:21.286 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:21.306 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:02:21.320 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:02:21.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:21.325 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:02:21.356 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:02:21.389 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:02:21.426 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:02:21.433 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:02:21.469 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:02:21.473 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:21.500 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:02:21.500 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:02:21.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:21.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:02:21.533 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:02:21.574 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:02:21.604 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:02:21.614 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:02:21.663 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:02:21.673 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:21.688 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:02:21.688 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:02:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:02:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:02:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:02:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:02:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:02:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:02:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:02:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:02:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:02:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:02:22.012 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:22 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:02:22.012 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:22 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/711791825' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:02:22.012 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:22 smithi027 ceph-mon[39795]: osdmap e399: 12 total, 12 up, 11 in 2024-05-15T13:02:22.012 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:22 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2646911225' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:02:22.265 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:02:22.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:22.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:02:22.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:22 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:02:22.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:22 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/711791825' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:02:22.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:22 smithi120 ceph-mon[43961]: osdmap e399: 12 total, 12 up, 11 in 2024-05-15T13:02:22.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:22 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2646911225' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:02:22.370 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:02:22.411 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:22.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:22 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:02:22.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:22 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/711791825' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:02:22.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:22 smithi177 ceph-mon[44303]: osdmap e399: 12 total, 12 up, 11 in 2024-05-15T13:02:22.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:22 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2646911225' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:02:22.497 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:02:22.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:22.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:02:22.603 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:02:22.643 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:22.675 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:02:22.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:22.679 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:02:22.780 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:02:22.834 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:23.017 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:23 smithi027 ceph-mon[39795]: pgmap v1080: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.0 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 786 op/s; 14916/50322 objects degraded (29.641%); 5426/50322 objects misplaced (10.783%); 3.8 MiB/s, 111 objects/s recovering 2024-05-15T13:02:23.210 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:02:23.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:23.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:02:23.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:23 smithi120 ceph-mon[43961]: pgmap v1080: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.0 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 786 op/s; 14916/50322 objects degraded (29.641%); 5426/50322 objects misplaced (10.783%); 3.8 MiB/s, 111 objects/s recovering 2024-05-15T13:02:23.319 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:02:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:23.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:23 smithi177 ceph-mon[44303]: pgmap v1080: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.0 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 786 op/s; 14916/50322 objects degraded (29.641%); 5426/50322 objects misplaced (10.783%); 3.8 MiB/s, 111 objects/s recovering 2024-05-15T13:02:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:02:23.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:23.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:02:23.587 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:02:23.652 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:23.681 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:02:23.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:23.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:02:23.791 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:02:23.842 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:23.880 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:02:23.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:23.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:02:23.987 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:02:23.988 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:02:24.061 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:24.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:24 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 14916/50322 objects degraded (29.641%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:24.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:24 smithi177 ceph-mon[44303]: osdmap e400: 12 total, 12 up, 11 in 2024-05-15T13:02:24.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:24 smithi177 ceph-mon[44303]: pgmap v1082: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 977 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 714 op/s; 14634/46854 objects degraded (31.233%); 5426/46854 objects misplaced (11.581%); 1.1 MiB/s, 94 objects/s recovering 2024-05-15T13:02:24.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:24 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 14916/50322 objects degraded (29.641%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:24.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:24 smithi027 ceph-mon[39795]: osdmap e400: 12 total, 12 up, 11 in 2024-05-15T13:02:24.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:24 smithi027 ceph-mon[39795]: pgmap v1082: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 977 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 714 op/s; 14634/46854 objects degraded (31.233%); 5426/46854 objects misplaced (11.581%); 1.1 MiB/s, 94 objects/s recovering 2024-05-15T13:02:24.290 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:02:24.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:24.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:02:24.403 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:02:24.430 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:24 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 14916/50322 objects degraded (29.641%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:24.431 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:24 smithi120 ceph-mon[43961]: osdmap e400: 12 total, 12 up, 11 in 2024-05-15T13:02:24.431 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:24 smithi120 ceph-mon[43961]: pgmap v1082: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 977 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 714 op/s; 14634/46854 objects degraded (31.233%); 5426/46854 objects misplaced (11.581%); 1.1 MiB/s, 94 objects/s recovering 2024-05-15T13:02:24.433 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:24.464 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:02:24.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:24.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:02:24.573 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:02:24.627 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:24.904 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:02:24.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:24.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:02:25.009 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:02:25.070 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:25.077 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.1 to 1 (10000) 2024-05-15T13:02:25.122 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:02:25.170 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:25 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/834819346' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T13:02:25.170 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:25 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T13:02:25.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:25 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/834819346' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T13:02:25.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:25 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T13:02:25.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:25 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/834819346' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T13:02:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:25 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T13:02:25.882 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:02:26.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:26 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T13:02:26.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:26 smithi027 ceph-mon[39795]: osdmap e401: 12 total, 12 up, 11 in 2024-05-15T13:02:26.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:26 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:02:26.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:26 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:02:26.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:26 smithi027 ceph-mon[39795]: pgmap v1084: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 891 MiB data, 7.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 642 op/s; 14634/42729 objects degraded (34.248%); 5282/42729 objects misplaced (12.362%); 331 KiB/s, 91 objects/s recovering 2024-05-15T13:02:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:26 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T13:02:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:26 smithi177 ceph-mon[44303]: osdmap e401: 12 total, 12 up, 11 in 2024-05-15T13:02:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:26 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:02:26.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:26 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:02:26.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:26 smithi177 ceph-mon[44303]: pgmap v1084: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 891 MiB data, 7.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 642 op/s; 14634/42729 objects degraded (34.248%); 5282/42729 objects misplaced (12.362%); 331 KiB/s, 91 objects/s recovering 2024-05-15T13:02:26.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:26 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T13:02:26.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:26 smithi120 ceph-mon[43961]: osdmap e401: 12 total, 12 up, 11 in 2024-05-15T13:02:26.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:26 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:02:26.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:26 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:02:26.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:26 smithi120 ceph-mon[43961]: pgmap v1084: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 891 MiB data, 7.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 642 op/s; 14634/42729 objects degraded (34.248%); 5282/42729 objects misplaced (12.362%); 331 KiB/s, 91 objects/s recovering 2024-05-15T13:02:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:02:26.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:26.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:02:27.036 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:02:27.074 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:27.109 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:02:27.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:27.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:02:27.219 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:02:27.268 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:27.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:27 smithi027 ceph-mon[39795]: osdmap e402: 12 total, 12 up, 11 in 2024-05-15T13:02:27.291 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:02:27.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:27.294 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:02:27.395 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:02:27.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:27 smithi177 ceph-mon[44303]: osdmap e402: 12 total, 12 up, 11 in 2024-05-15T13:02:27.441 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 2 1 2024-05-15T13:02:27.513 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:27.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:27 smithi120 ceph-mon[43961]: osdmap e402: 12 total, 12 up, 11 in 2024-05-15T13:02:27.644 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:02:27.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:27.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:02:27.756 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:02:27.843 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:27.977 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:02:27.983 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:02:27.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:27.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:02:27.992 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:02:28.087 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:02:28.089 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:02:28.089 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:02:28.161 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:28.197 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:02:28.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:28.204 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:02:28.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:28 smithi027 ceph-mon[39795]: pgmap v1086: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 788 MiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 695 op/s; 14242/37803 objects degraded (37.674%); 5282/37803 objects misplaced (13.972%); 0 B/s, 136 objects/s recovering 2024-05-15T13:02:28.304 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:02:28.344 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:28.372 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:02:28.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:28.377 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:28 smithi177 ceph-mon[44303]: pgmap v1086: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 788 MiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 695 op/s; 14242/37803 objects degraded (37.674%); 5282/37803 objects misplaced (13.972%); 0 B/s, 136 objects/s recovering 2024-05-15T13:02:28.378 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:02:28.479 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:02:28.511 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:28 smithi120 ceph-mon[43961]: pgmap v1086: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 788 MiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 695 op/s; 14242/37803 objects degraded (37.674%); 5282/37803 objects misplaced (13.972%); 0 B/s, 136 objects/s recovering 2024-05-15T13:02:28.515 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:28.549 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:02:28.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:28.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:02:28.656 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:02:28.693 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:28.733 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:02:28.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:28.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:02:28.836 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:02:28.901 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:29.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:29 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 14242/37803 objects degraded (37.674%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:29.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:29 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 14242/37803 objects degraded (37.674%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:29.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:29 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 14242/37803 objects degraded (37.674%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:30.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:30 smithi177 ceph-mon[44303]: pgmap v1087: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 672 MiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 839 op/s; 14242/32256 objects degraded (44.153%); 5126/32256 objects misplaced (15.892%); 0 B/s, 104 objects/s recovering 2024-05-15T13:02:30.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:30 smithi027 ceph-mon[39795]: pgmap v1087: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 672 MiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 839 op/s; 14242/32256 objects degraded (44.153%); 5126/32256 objects misplaced (15.892%); 0 B/s, 104 objects/s recovering 2024-05-15T13:02:30.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:30 smithi120 ceph-mon[43961]: pgmap v1087: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 672 MiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 839 op/s; 14242/32256 objects degraded (44.153%); 5126/32256 objects misplaced (15.892%); 0 B/s, 104 objects/s recovering 2024-05-15T13:02:32.423 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-05-15T13:02:33.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:32 smithi027 ceph-mon[39795]: pgmap v1088: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 587 MiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 779 op/s; 14123/28137 objects degraded (50.194%); 5126/28137 objects misplaced (18.218%); 0 B/s, 101 objects/s recovering 2024-05-15T13:02:33.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:32 smithi120 ceph-mon[43961]: pgmap v1088: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 587 MiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 779 op/s; 14123/28137 objects degraded (50.194%); 5126/28137 objects misplaced (18.218%); 0 B/s, 101 objects/s recovering 2024-05-15T13:02:33.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:32 smithi177 ceph-mon[44303]: pgmap v1088: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 587 MiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 779 op/s; 14123/28137 objects degraded (50.194%); 5126/28137 objects misplaced (18.218%); 0 B/s, 101 objects/s recovering 2024-05-15T13:02:34.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:34 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 14123/28137 objects degraded (50.194%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:34.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:34 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/145289374' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:02:34.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:34 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 14123/28137 objects degraded (50.194%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:34.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:34 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/145289374' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:02:34.296 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:02:34.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:34.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:02:34.337 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:02:34.407 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:02:34.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:34 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 14123/28137 objects degraded (50.194%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:34.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:34 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/145289374' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:02:34.442 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:34.471 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:02:34.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:34.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:02:34.577 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:02:34.635 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:34.672 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:02:34.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:34.678 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:02:34.779 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:02:34.886 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:34.974 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:02:34.974 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:02:34.974 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:02:34.974 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:02:34.974 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:02:34.974 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:02:34.975 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:02:34.975 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:02:34.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:34.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:02:34.993 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:02:35.028 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:02:35.077 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:02:35.080 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:02:35.114 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:35.139 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:02:35.147 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:02:35.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:35.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:02:35.183 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:02:35.192 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:02:35.192 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:02:35.192 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:02:35.192 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:02:35.192 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:02:35.192 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:02:35.192 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:02:35.192 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:02:35.193 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:02:35.193 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:02:35.193 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:02:35.193 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:02:35.227 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:35 smithi027 ceph-mon[39795]: pgmap v1089: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 512 MiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 708 op/s; 13837/24537 objects degraded (56.392%); 5126/24537 objects misplaced (20.891%); 0 B/s, 107 objects/s recovering 2024-05-15T13:02:35.227 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:35 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:02:35.227 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:35 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/145289374' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:02:35.227 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:35 smithi027 ceph-mon[39795]: osdmap e403: 12 total, 12 up, 11 in 2024-05-15T13:02:35.227 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:35 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1560969135' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:02:35.258 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:02:35.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:35 smithi120 ceph-mon[43961]: pgmap v1089: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 512 MiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 708 op/s; 13837/24537 objects degraded (56.392%); 5126/24537 objects misplaced (20.891%); 0 B/s, 107 objects/s recovering 2024-05-15T13:02:35.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:35 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:02:35.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:35 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/145289374' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:02:35.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:35 smithi120 ceph-mon[43961]: osdmap e403: 12 total, 12 up, 11 in 2024-05-15T13:02:35.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:35 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1560969135' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:02:35.284 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:35 smithi177 ceph-mon[44303]: pgmap v1089: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 512 MiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 708 op/s; 13837/24537 objects degraded (56.392%); 5126/24537 objects misplaced (20.891%); 0 B/s, 107 objects/s recovering 2024-05-15T13:02:35.284 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:35 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:02:35.284 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:35 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/145289374' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:02:35.284 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:35 smithi177 ceph-mon[44303]: osdmap e403: 12 total, 12 up, 11 in 2024-05-15T13:02:35.284 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:35 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1560969135' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:02:35.286 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:35.323 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:02:35.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:35.330 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:02:35.431 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:02:35.494 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:35.534 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:02:35.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:35.540 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:02:35.641 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:02:35.688 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:36.055 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:02:36.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:36.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:02:36.136 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:02:36.167 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:02:36.197 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:02:36.234 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:02:36.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:36.242 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:02:36.343 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:02:36.387 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:36.412 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:02:36.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:36.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:02:36.525 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:02:36.581 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:36.622 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:02:36.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:36.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:02:36.731 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:02:36.782 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:36.814 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:02:36.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:36.815 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:02:36.916 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:02:36.969 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:37.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:37 smithi177 ceph-mon[44303]: pgmap v1091: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 426 MiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 753 op/s; 13837/20439 objects degraded (67.699%); 4969/20439 objects misplaced (24.311%); 0 B/s, 112 objects/s recovering 2024-05-15T13:02:37.241 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:02:37.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:37.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:02:37.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:37 smithi120 ceph-mon[43961]: pgmap v1091: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 426 MiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 753 op/s; 13837/20439 objects degraded (67.699%); 4969/20439 objects misplaced (24.311%); 0 B/s, 112 objects/s recovering 2024-05-15T13:02:37.350 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:02:37.382 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:37.409 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:02:37.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:37.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:02:37.519 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:02:37.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:37 smithi027 ceph-mon[39795]: pgmap v1091: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 426 MiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 753 op/s; 13837/20439 objects degraded (67.699%); 4969/20439 objects misplaced (24.311%); 0 B/s, 112 objects/s recovering 2024-05-15T13:02:37.556 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:37.592 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:02:37.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:37.604 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:02:37.705 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:02:37.722 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:02:37.764 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:38.098 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:02:38.435 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:38 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:02:38.435 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:38 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:02:38.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:38 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:02:38.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:38 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:02:38.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:38 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:02:38.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:38 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:02:39.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:39 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:02:39.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:39 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:02:39.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:39 smithi177 ceph-mon[44303]: pgmap v1092: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 327 MiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 736 op/s; 13436/15687 objects degraded (85.651%); 4969/15687 objects misplaced (31.676%); 0 B/s, 111 objects/s recovering 2024-05-15T13:02:39.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:39 smithi177 ceph-mon[44303]: osdmap e404: 12 total, 12 up, 11 in 2024-05-15T13:02:39.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:39 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 13436/15687 objects degraded (85.651%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:39.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:39 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:02:39.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:39 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:02:39.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:39 smithi027 ceph-mon[39795]: pgmap v1092: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 327 MiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 736 op/s; 13436/15687 objects degraded (85.651%); 4969/15687 objects misplaced (31.676%); 0 B/s, 111 objects/s recovering 2024-05-15T13:02:39.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:39 smithi027 ceph-mon[39795]: osdmap e404: 12 total, 12 up, 11 in 2024-05-15T13:02:39.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:39 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 13436/15687 objects degraded (85.651%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:39.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:39 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:02:39.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:39 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:02:39.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:39 smithi120 ceph-mon[43961]: pgmap v1092: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 327 MiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 736 op/s; 13436/15687 objects degraded (85.651%); 4969/15687 objects misplaced (31.676%); 0 B/s, 111 objects/s recovering 2024-05-15T13:02:39.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:39 smithi120 ceph-mon[43961]: osdmap e404: 12 total, 12 up, 11 in 2024-05-15T13:02:39.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:39 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 13436/15687 objects degraded (85.651%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:39.552 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:02:39.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:39.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:02:39.659 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:02:39.714 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:40.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:40 smithi027 ceph-mon[39795]: pgmap v1094: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 211 MiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 750 op/s; 13436/10104 objects degraded (132.977%); 4833/10104 objects misplaced (47.833%); 0 B/s, 122 objects/s recovering 2024-05-15T13:02:40.279 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:02:40.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:40.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:02:40.386 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:02:40.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:40 smithi177 ceph-mon[44303]: pgmap v1094: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 211 MiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 750 op/s; 13436/10104 objects degraded (132.977%); 4833/10104 objects misplaced (47.833%); 0 B/s, 122 objects/s recovering 2024-05-15T13:02:40.464 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:40.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:40 smithi120 ceph-mon[43961]: pgmap v1094: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 211 MiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 750 op/s; 13436/10104 objects degraded (132.977%); 4833/10104 objects misplaced (47.833%); 0 B/s, 122 objects/s recovering 2024-05-15T13:02:40.633 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:02:40.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:40.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:02:40.737 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:02:40.773 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:40.802 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:02:40.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:40.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:02:40.872 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:02:40.872 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T13:02:40.872 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T13:02:40.873 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:02:40.873 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.873 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:14047 3.b 3:d248903a:::benchmark_data_smithi177_88104_object14046:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-05-15T13:02:40.873 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:53:13.727192+0000", 2024-05-15T13:02:40.873 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 567.13974228699999, 2024-05-15T13:02:40.873 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.209732843, 2024-05-15T13:02:40.873 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:40.873 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:40.873 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:02:40.873 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:02:40.873 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:02:40.873 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 14047 2024-05-15T13:02:40.873 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.874 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:40.874 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.874 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:40.874 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.727192+0000", 2024-05-15T13:02:40.874 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.874 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.874 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.874 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:40.874 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.727192+0000", 2024-05-15T13:02:40.874 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.874 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.874 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.875 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:40.875 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.727192+0000", 2024-05-15T13:02:40.875 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1500000000000004e-07 2024-05-15T13:02:40.875 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.875 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.875 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:40.875 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.727238+0000", 2024-05-15T13:02:40.875 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6038999999999997e-05 2024-05-15T13:02:40.875 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.875 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.875 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:40.875 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.727248+0000", 2024-05-15T13:02:40.876 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3959999999999995e-06 2024-05-15T13:02:40.876 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.876 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.876 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:40.876 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.727259+0000", 2024-05-15T13:02:40.876 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1253000000000001e-05 2024-05-15T13:02:40.876 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.876 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.876 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:40.876 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.728460+0000", 2024-05-15T13:02:40.876 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0012010269999999999 2024-05-15T13:02:40.876 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.876 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.876 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:40.877 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.728575+0000", 2024-05-15T13:02:40.877 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011443299999999999 2024-05-15T13:02:40.877 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.877 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.877 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,4", 2024-05-15T13:02:40.877 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.728710+0000", 2024-05-15T13:02:40.877 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013532999999999999 2024-05-15T13:02:40.877 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.877 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.877 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:02:40.877 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.730388+0000", 2024-05-15T13:02:40.877 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016780510000000001 2024-05-15T13:02:40.878 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.878 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.878 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:02:40.878 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.737116+0000", 2024-05-15T13:02:40.878 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0067282949999999996 2024-05-15T13:02:40.878 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.878 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.878 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:02:40.878 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.936863+0000", 2024-05-15T13:02:40.878 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.199747175 2024-05-15T13:02:40.878 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.878 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.878 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:40.878 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.936899+0000", 2024-05-15T13:02:40.879 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5201999999999999e-05 2024-05-15T13:02:40.879 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.879 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.879 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:40.879 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.936925+0000", 2024-05-15T13:02:40.879 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6027e-05 2024-05-15T13:02:40.879 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.879 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:40.879 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.879 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.879 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.879 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:14048 3.b 3:d72aa540:::benchmark_data_smithi177_88104_object14047:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-05-15T13:02:40.879 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:53:13.727272+0000", 2024-05-15T13:02:40.880 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 567.13966200699997, 2024-05-15T13:02:40.880 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20974601400000001, 2024-05-15T13:02:40.880 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:40.880 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:40.880 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:02:40.880 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:02:40.880 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:02:40.880 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 14048 2024-05-15T13:02:40.880 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.880 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:40.880 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.880 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:40.880 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.727272+0000", 2024-05-15T13:02:40.881 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.881 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.881 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.881 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:40.881 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.727272+0000", 2024-05-15T13:02:40.881 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.881 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.881 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.881 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:40.881 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.727272+0000", 2024-05-15T13:02:40.881 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.89e-07 2024-05-15T13:02:40.881 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.881 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.882 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:40.882 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.727303+0000", 2024-05-15T13:02:40.882 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0088999999999999e-05 2024-05-15T13:02:40.882 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.882 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.882 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:40.882 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.727308+0000", 2024-05-15T13:02:40.882 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9150000000000001e-06 2024-05-15T13:02:40.882 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.882 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.882 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:40.882 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.727316+0000", 2024-05-15T13:02:40.883 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7519999999999996e-06 2024-05-15T13:02:40.883 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.883 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.883 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:40.883 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.729243+0000", 2024-05-15T13:02:40.883 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019272250000000001 2024-05-15T13:02:40.883 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.883 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.883 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:40.883 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.729364+0000", 2024-05-15T13:02:40.883 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120785 2024-05-15T13:02:40.883 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.883 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.883 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,4", 2024-05-15T13:02:40.884 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.729493+0000", 2024-05-15T13:02:40.884 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012843800000000001 2024-05-15T13:02:40.884 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.884 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.884 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:02:40.884 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.730395+0000", 2024-05-15T13:02:40.884 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00090209000000000003 2024-05-15T13:02:40.884 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.884 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.884 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:02:40.884 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.737151+0000", 2024-05-15T13:02:40.884 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0067564979999999997 2024-05-15T13:02:40.884 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.885 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.885 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:02:40.885 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.936959+0000", 2024-05-15T13:02:40.885 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.199807969 2024-05-15T13:02:40.885 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.885 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.885 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:40.885 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.936993+0000", 2024-05-15T13:02:40.885 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3810999999999999e-05 2024-05-15T13:02:40.885 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.885 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.885 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:40.885 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:53:13.937018+0000", 2024-05-15T13:02:40.885 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5052999999999999e-05 2024-05-15T13:02:40.886 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.886 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:40.886 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.886 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.886 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.886 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:198322 3.2 3:4aa13b34:::benchmark_data_smithi177_88104_object64700:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-05-15T13:02:40.886 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:55:57.184536+0000", 2024-05-15T13:02:40.886 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 403.68239825299997, 2024-05-15T13:02:40.886 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.23267259000000001, 2024-05-15T13:02:40.886 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:40.886 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:40.886 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:02:40.886 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:02:40.887 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:02:40.887 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 198322 2024-05-15T13:02:40.887 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.887 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:40.887 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.887 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:40.887 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:55:57.184536+0000", 2024-05-15T13:02:40.887 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.887 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.887 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.887 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:40.887 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:55:57.184536+0000", 2024-05-15T13:02:40.887 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.887 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.888 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.888 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:40.888 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:55:57.184538+0000", 2024-05-15T13:02:40.888 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0049999999999999e-06 2024-05-15T13:02:40.888 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.888 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.888 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:40.888 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:55:57.184546+0000", 2024-05-15T13:02:40.888 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7710000000000006e-06 2024-05-15T13:02:40.888 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.888 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.888 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:40.888 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:55:57.184569+0000", 2024-05-15T13:02:40.889 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3147999999999999e-05 2024-05-15T13:02:40.889 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.889 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.889 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:40.889 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:55:57.184580+0000", 2024-05-15T13:02:40.889 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1334e-05 2024-05-15T13:02:40.889 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.889 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.889 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:40.889 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:55:57.184649+0000", 2024-05-15T13:02:40.889 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8978000000000003e-05 2024-05-15T13:02:40.889 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.889 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.889 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:40.890 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:55:57.184764+0000", 2024-05-15T13:02:40.890 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011490899999999999 2024-05-15T13:02:40.890 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.890 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.890 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6,9", 2024-05-15T13:02:40.890 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:55:57.184876+0000", 2024-05-15T13:02:40.890 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000111576 2024-05-15T13:02:40.890 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.890 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.890 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:02:40.890 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:55:57.194446+0000", 2024-05-15T13:02:40.890 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0095709560000000003 2024-05-15T13:02:40.890 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.891 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.891 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:02:40.891 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:55:57.195477+0000", 2024-05-15T13:02:40.891 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010309150000000001 2024-05-15T13:02:40.891 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.891 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.891 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:02:40.891 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:55:57.417041+0000", 2024-05-15T13:02:40.891 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22156332400000001 2024-05-15T13:02:40.891 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.891 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.891 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:40.891 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:55:57.417154+0000", 2024-05-15T13:02:40.891 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000112921 2024-05-15T13:02:40.892 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.892 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.892 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:40.892 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:55:57.417208+0000", 2024-05-15T13:02:40.892 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4753000000000002e-05 2024-05-15T13:02:40.892 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.892 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:40.892 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.892 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.892 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.892 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:48851 4.0 4:0f538546:::benchmark_data_smithi177_99756_object48850:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-05-15T13:02:40.892 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:58:29.293070+0000", 2024-05-15T13:02:40.892 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 251.573864499, 2024-05-15T13:02:40.892 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.210520136, 2024-05-15T13:02:40.893 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:40.893 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:40.893 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:02:40.893 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:02:40.893 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:02:40.893 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 48851 2024-05-15T13:02:40.893 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.893 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:40.893 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.893 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:40.893 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.293070+0000", 2024-05-15T13:02:40.893 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.893 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.894 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.894 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:40.894 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.293070+0000", 2024-05-15T13:02:40.894 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.894 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.894 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.894 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:40.894 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.293071+0000", 2024-05-15T13:02:40.894 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2109999999999999e-06 2024-05-15T13:02:40.894 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.894 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.894 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:40.895 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.293152+0000", 2024-05-15T13:02:40.895 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1561000000000001e-05 2024-05-15T13:02:40.895 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.895 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.895 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:40.895 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.293163+0000", 2024-05-15T13:02:40.895 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1107000000000001e-05 2024-05-15T13:02:40.895 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.895 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.895 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:40.895 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.293170+0000", 2024-05-15T13:02:40.895 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3690000000000003e-06 2024-05-15T13:02:40.895 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.895 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.896 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:40.896 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.293224+0000", 2024-05-15T13:02:40.896 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3971999999999998e-05 2024-05-15T13:02:40.896 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.896 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.896 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:40.896 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.293328+0000", 2024-05-15T13:02:40.896 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010450600000000001 2024-05-15T13:02:40.896 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.896 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.896 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,8", 2024-05-15T13:02:40.896 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.293437+0000", 2024-05-15T13:02:40.896 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010872800000000001 2024-05-15T13:02:40.896 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.897 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.897 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:02:40.897 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.295197+0000", 2024-05-15T13:02:40.897 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0017604229999999999 2024-05-15T13:02:40.897 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.897 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.897 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:02:40.897 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.298482+0000", 2024-05-15T13:02:40.897 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032841150000000002 2024-05-15T13:02:40.897 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.897 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.897 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:02:40.897 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.503519+0000", 2024-05-15T13:02:40.897 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.205037107 2024-05-15T13:02:40.898 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.898 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.898 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:40.898 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.503558+0000", 2024-05-15T13:02:40.898 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9320000000000003e-05 2024-05-15T13:02:40.898 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.898 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.898 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:40.898 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.503590+0000", 2024-05-15T13:02:40.898 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1717000000000001e-05 2024-05-15T13:02:40.898 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.898 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:40.898 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.898 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.899 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.899 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:142000 4.9 e357/343 4:9e3e0cd7:::benchmark_data_smithi177_99756_object22028:head v 357'8894, mlcod=323'1360)", 2024-05-15T13:02:40.899 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:13.180988+0000", 2024-05-15T13:02:40.899 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 147.68594604, 2024-05-15T13:02:40.899 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16514794299999999, 2024-05-15T13:02:40.899 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:40.899 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:40.899 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:40.899 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.899 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:40.899 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.180988+0000", 2024-05-15T13:02:40.899 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.899 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.900 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.900 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:40.900 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.180988+0000", 2024-05-15T13:02:40.900 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.900 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.900 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.900 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:40.900 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.180989+0000", 2024-05-15T13:02:40.900 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3099999999999997e-07 2024-05-15T13:02:40.900 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.900 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.900 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:40.900 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.180996+0000", 2024-05-15T13:02:40.900 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5630000000000003e-06 2024-05-15T13:02:40.900 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.901 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.901 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:40.901 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.181007+0000", 2024-05-15T13:02:40.901 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0494e-05 2024-05-15T13:02:40.901 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.901 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.901 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:40.901 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.181015+0000", 2024-05-15T13:02:40.901 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3189999999999995e-06 2024-05-15T13:02:40.901 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.901 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.901 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:40.901 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.181066+0000", 2024-05-15T13:02:40.902 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0540000000000001e-05 2024-05-15T13:02:40.902 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.902 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.902 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:40.902 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.181091+0000", 2024-05-15T13:02:40.902 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5448000000000001e-05 2024-05-15T13:02:40.902 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.902 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.902 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:40.902 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.184878+0000", 2024-05-15T13:02:40.902 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0037873149999999999 2024-05-15T13:02:40.902 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.902 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.903 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:40.903 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346136+0000", 2024-05-15T13:02:40.903 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16125753300000001 2024-05-15T13:02:40.903 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.903 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:40.903 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.903 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.903 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.903 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:142002 4.c e357/317 4:39c01dc0:::benchmark_data_smithi177_99756_object22030:head v 357'8893, mlcod=357'8891)", 2024-05-15T13:02:40.903 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:13.183315+0000", 2024-05-15T13:02:40.903 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 147.68361881499999, 2024-05-15T13:02:40.903 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16287758499999999, 2024-05-15T13:02:40.903 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:40.903 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:40.904 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:40.904 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.904 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:40.904 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.183315+0000", 2024-05-15T13:02:40.904 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.904 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.904 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.904 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:40.904 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.183315+0000", 2024-05-15T13:02:40.904 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.904 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.904 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.904 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:40.905 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.183316+0000", 2024-05-15T13:02:40.905 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.54e-07 2024-05-15T13:02:40.905 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.905 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.905 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:40.905 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.183323+0000", 2024-05-15T13:02:40.905 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3030000000000002e-06 2024-05-15T13:02:40.905 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.905 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.905 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:40.905 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.183335+0000", 2024-05-15T13:02:40.905 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1935e-05 2024-05-15T13:02:40.905 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.905 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.906 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:40.906 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.183342+0000", 2024-05-15T13:02:40.906 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0870000000000001e-06 2024-05-15T13:02:40.906 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.906 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.906 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:40.906 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.183398+0000", 2024-05-15T13:02:40.906 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5373e-05 2024-05-15T13:02:40.906 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.906 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.906 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:40.906 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.183418+0000", 2024-05-15T13:02:40.906 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9882999999999999e-05 2024-05-15T13:02:40.907 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.907 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.907 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:40.907 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346152+0000", 2024-05-15T13:02:40.907 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16273420299999999 2024-05-15T13:02:40.907 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.907 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.907 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:40.907 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346193+0000", 2024-05-15T13:02:40.907 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0946999999999999e-05 2024-05-15T13:02:40.907 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.907 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:40.907 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.908 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.908 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.908 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:142007 4.0 4:006ed880:::benchmark_data_smithi177_99756_object22035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-05-15T13:02:40.908 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:13.189831+0000", 2024-05-15T13:02:40.908 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 147.67710329600001, 2024-05-15T13:02:40.908 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.171594948, 2024-05-15T13:02:40.908 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:40.908 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:40.908 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:02:40.908 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:02:40.908 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:02:40.908 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 142007 2024-05-15T13:02:40.908 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.908 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:40.909 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.909 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:40.909 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.189831+0000", 2024-05-15T13:02:40.909 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.909 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.909 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.909 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:40.909 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.189831+0000", 2024-05-15T13:02:40.909 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.909 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.909 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.909 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:40.909 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.189833+0000", 2024-05-15T13:02:40.909 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.826e-06 2024-05-15T13:02:40.910 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.910 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.910 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:40.910 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.189841+0000", 2024-05-15T13:02:40.910 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5290000000000008e-06 2024-05-15T13:02:40.910 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.910 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.910 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:40.910 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346127+0000", 2024-05-15T13:02:40.910 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.156286175 2024-05-15T13:02:40.910 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.910 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.910 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:40.911 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346142+0000", 2024-05-15T13:02:40.911 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4600000000000001e-05 2024-05-15T13:02:40.911 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.911 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.911 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:40.911 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346259+0000", 2024-05-15T13:02:40.911 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011670700000000001 2024-05-15T13:02:40.911 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.911 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.911 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:40.911 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346391+0000", 2024-05-15T13:02:40.911 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013222200000000001 2024-05-15T13:02:40.911 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.911 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.912 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,8", 2024-05-15T13:02:40.912 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346510+0000", 2024-05-15T13:02:40.912 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000118898 2024-05-15T13:02:40.912 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.912 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.912 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:02:40.912 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.355786+0000", 2024-05-15T13:02:40.912 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0092759000000000001 2024-05-15T13:02:40.912 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.912 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.912 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:02:40.912 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.360397+0000", 2024-05-15T13:02:40.912 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046116919999999997 2024-05-15T13:02:40.912 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.913 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.913 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:02:40.913 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.361325+0000", 2024-05-15T13:02:40.913 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00092745299999999996 2024-05-15T13:02:40.913 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.913 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.913 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:40.913 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.361381+0000", 2024-05-15T13:02:40.913 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6512999999999999e-05 2024-05-15T13:02:40.913 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.913 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.913 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:40.913 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.361426+0000", 2024-05-15T13:02:40.913 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4433e-05 2024-05-15T13:02:40.914 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.914 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:40.914 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.914 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.914 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.914 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:142021 4.f e357/343 4:ff930601:::benchmark_data_smithi177_99756_object22049:head v 357'8697, mlcod=323'1700)", 2024-05-15T13:02:40.914 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:13.218868+0000", 2024-05-15T13:02:40.914 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 147.648066231, 2024-05-15T13:02:40.914 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.152184717, 2024-05-15T13:02:40.914 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:40.914 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:40.914 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:40.914 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.915 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:40.915 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.218868+0000", 2024-05-15T13:02:40.915 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.915 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.915 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.915 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:40.915 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.218868+0000", 2024-05-15T13:02:40.915 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.915 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.915 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.915 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:40.915 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.218870+0000", 2024-05-15T13:02:40.916 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.747e-06 2024-05-15T13:02:40.916 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.916 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.916 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:40.916 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.218880+0000", 2024-05-15T13:02:40.916 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0275000000000001e-05 2024-05-15T13:02:40.916 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.916 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.916 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:40.916 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346117+0000", 2024-05-15T13:02:40.916 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12723722200000001 2024-05-15T13:02:40.916 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.916 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.917 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:40.917 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346133+0000", 2024-05-15T13:02:40.917 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5945999999999999e-05 2024-05-15T13:02:40.917 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.917 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.917 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:40.917 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346197+0000", 2024-05-15T13:02:40.917 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4321999999999998e-05 2024-05-15T13:02:40.917 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.917 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.917 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:40.917 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346228+0000", 2024-05-15T13:02:40.917 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0898999999999997e-05 2024-05-15T13:02:40.917 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.918 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.918 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:40.918 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.371017+0000", 2024-05-15T13:02:40.918 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.024788849000000002 2024-05-15T13:02:40.918 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.918 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.918 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:40.918 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.371053+0000", 2024-05-15T13:02:40.918 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5457000000000003e-05 2024-05-15T13:02:40.918 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.918 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:40.918 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.918 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.919 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.919 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:145947 4.0 4:02e2b03e:::benchmark_data_smithi177_99756_object25975:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-05-15T13:02:40.919 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:19.978855+0000", 2024-05-15T13:02:40.919 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 140.888079366, 2024-05-15T13:02:40.919 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.057148081000000003, 2024-05-15T13:02:40.919 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:40.919 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:40.919 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:02:40.919 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:02:40.919 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:02:40.919 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 145947 2024-05-15T13:02:40.919 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.919 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:40.920 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.920 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:40.920 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978855+0000", 2024-05-15T13:02:40.920 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.920 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.920 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.920 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:40.920 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978855+0000", 2024-05-15T13:02:40.920 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.920 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.920 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.920 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:40.920 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978857+0000", 2024-05-15T13:02:40.921 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0250000000000001e-06 2024-05-15T13:02:40.921 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.921 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.921 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:40.921 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978866+0000", 2024-05-15T13:02:40.921 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0790000000000007e-06 2024-05-15T13:02:40.921 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.921 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.921 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:40.921 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978892+0000", 2024-05-15T13:02:40.921 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6670999999999999e-05 2024-05-15T13:02:40.921 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.921 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.921 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:40.922 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978904+0000", 2024-05-15T13:02:40.922 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1214e-05 2024-05-15T13:02:40.922 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.922 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.922 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:40.922 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978971+0000", 2024-05-15T13:02:40.922 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7081000000000004e-05 2024-05-15T13:02:40.922 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.922 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.922 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:40.922 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.979127+0000", 2024-05-15T13:02:40.922 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015630900000000001 2024-05-15T13:02:40.922 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.923 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.923 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,8", 2024-05-15T13:02:40.923 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.979236+0000", 2024-05-15T13:02:40.923 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000109277 2024-05-15T13:02:40.923 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.923 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.923 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:02:40.923 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.993968+0000", 2024-05-15T13:02:40.923 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014731164 2024-05-15T13:02:40.923 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.923 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.923 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:02:40.924 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.993984+0000", 2024-05-15T13:02:40.924 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6492999999999999e-05 2024-05-15T13:02:40.924 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.924 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.924 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:02:40.924 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.035928+0000", 2024-05-15T13:02:40.924 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.041944082000000001 2024-05-15T13:02:40.924 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.924 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.924 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:40.924 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.035969+0000", 2024-05-15T13:02:40.924 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0572999999999999e-05 2024-05-15T13:02:40.924 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.924 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.924 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:40.925 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.036003+0000", 2024-05-15T13:02:40.925 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4112999999999997e-05 2024-05-15T13:02:40.925 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.925 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:40.925 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.925 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.925 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.925 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:145950 4.0 4:0acf3df5:::benchmark_data_smithi177_99756_object25978:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-05-15T13:02:40.925 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:19.978932+0000", 2024-05-15T13:02:40.925 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 140.888002075, 2024-05-15T13:02:40.925 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.057137765, 2024-05-15T13:02:40.925 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:40.925 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:40.926 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:02:40.926 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:02:40.926 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:02:40.926 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 145950 2024-05-15T13:02:40.926 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.926 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:40.926 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.926 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:40.926 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978932+0000", 2024-05-15T13:02:40.926 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.926 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.926 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.926 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:40.926 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978932+0000", 2024-05-15T13:02:40.927 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.927 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.927 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.927 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:40.927 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978933+0000", 2024-05-15T13:02:40.927 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4099999999999998e-07 2024-05-15T13:02:40.927 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.927 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.927 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:40.927 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978936+0000", 2024-05-15T13:02:40.927 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.642e-06 2024-05-15T13:02:40.927 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.927 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.928 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:40.928 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978945+0000", 2024-05-15T13:02:40.928 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2730000000000002e-06 2024-05-15T13:02:40.928 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.928 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.928 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:40.928 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978951+0000", 2024-05-15T13:02:40.928 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6490000000000002e-06 2024-05-15T13:02:40.928 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.928 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.928 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:40.928 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.983771+0000", 2024-05-15T13:02:40.928 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0048201249999999998 2024-05-15T13:02:40.928 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.929 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.929 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:40.929 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.983895+0000", 2024-05-15T13:02:40.929 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012362699999999999 2024-05-15T13:02:40.929 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.929 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.929 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,8", 2024-05-15T13:02:40.929 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.984007+0000", 2024-05-15T13:02:40.929 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011250499999999999 2024-05-15T13:02:40.929 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.929 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.929 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:02:40.929 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.993989+0000", 2024-05-15T13:02:40.929 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0099817480000000004 2024-05-15T13:02:40.929 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.930 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.930 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:02:40.930 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.000442+0000", 2024-05-15T13:02:40.930 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0064523130000000003 2024-05-15T13:02:40.930 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.930 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.930 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:02:40.930 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.036025+0000", 2024-05-15T13:02:40.930 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.035583468 2024-05-15T13:02:40.930 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.930 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.930 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:40.930 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.036050+0000", 2024-05-15T13:02:40.931 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5097e-05 2024-05-15T13:02:40.931 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.931 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.931 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:40.931 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.036070+0000", 2024-05-15T13:02:40.931 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9677000000000002e-05 2024-05-15T13:02:40.931 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.931 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:40.931 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.931 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.931 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.931 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:145951 4.0 4:089eb460:::benchmark_data_smithi177_99756_object25979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-05-15T13:02:40.931 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:19.978965+0000", 2024-05-15T13:02:40.931 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 140.88796886099999, 2024-05-15T13:02:40.932 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.063096423999999998, 2024-05-15T13:02:40.932 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:40.932 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:40.932 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:02:40.932 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:02:40.932 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:02:40.932 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 145951 2024-05-15T13:02:40.932 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.932 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:40.932 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.932 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:40.932 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978965+0000", 2024-05-15T13:02:40.932 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.932 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.933 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.933 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:40.933 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978965+0000", 2024-05-15T13:02:40.933 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.933 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.933 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.933 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:40.933 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978966+0000", 2024-05-15T13:02:40.933 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4600000000000005e-07 2024-05-15T13:02:40.933 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.933 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.933 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:40.933 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978969+0000", 2024-05-15T13:02:40.934 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4620000000000001e-06 2024-05-15T13:02:40.934 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.934 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.934 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:40.934 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978977+0000", 2024-05-15T13:02:40.934 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8019999999999992e-06 2024-05-15T13:02:40.934 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.934 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.934 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:40.934 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978983+0000", 2024-05-15T13:02:40.934 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4309999999999999e-06 2024-05-15T13:02:40.934 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.935 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.935 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:40.935 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.988763+0000", 2024-05-15T13:02:40.935 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0097792399999999998 2024-05-15T13:02:40.935 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.935 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.935 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:40.935 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.988898+0000", 2024-05-15T13:02:40.935 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013542400000000001 2024-05-15T13:02:40.935 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.935 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.935 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,8", 2024-05-15T13:02:40.935 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.989013+0000", 2024-05-15T13:02:40.936 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000115254 2024-05-15T13:02:40.936 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.936 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.936 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:02:40.936 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.994073+0000", 2024-05-15T13:02:40.936 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050596740000000001 2024-05-15T13:02:40.936 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.936 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.936 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:02:40.936 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.000489+0000", 2024-05-15T13:02:40.936 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0064156689999999997 2024-05-15T13:02:40.936 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.936 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.936 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:02:40.936 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.041979+0000", 2024-05-15T13:02:40.937 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.041490064 2024-05-15T13:02:40.937 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.937 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.937 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:40.937 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.042014+0000", 2024-05-15T13:02:40.937 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5497999999999998e-05 2024-05-15T13:02:40.937 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.937 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.937 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:40.937 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.042062+0000", 2024-05-15T13:02:40.937 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7360000000000001e-05 2024-05-15T13:02:40.937 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.937 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:40.937 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.938 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.938 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.938 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:167639 4.c e371/317 4:30d68d4c:::benchmark_data_smithi177_99756_object47667:head v 371'10508, mlcod=371'10505)", 2024-05-15T13:02:40.938 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:54.908023+0000", 2024-05-15T13:02:40.938 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 105.958911435, 2024-05-15T13:02:40.938 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.058411269000000002, 2024-05-15T13:02:40.938 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:40.938 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:40.938 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:40.938 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.938 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:40.938 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908023+0000", 2024-05-15T13:02:40.938 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.939 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.939 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.939 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:40.939 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908023+0000", 2024-05-15T13:02:40.939 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.939 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.939 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.939 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:40.939 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908023+0000", 2024-05-15T13:02:40.939 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7700000000000004e-07 2024-05-15T13:02:40.939 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.939 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.939 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:40.939 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908029+0000", 2024-05-15T13:02:40.940 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7420000000000003e-06 2024-05-15T13:02:40.940 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.940 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.940 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:40.940 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908036+0000", 2024-05-15T13:02:40.940 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7039999999999997e-06 2024-05-15T13:02:40.940 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.940 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.940 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:40.940 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908041+0000", 2024-05-15T13:02:40.940 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2970000000000003e-06 2024-05-15T13:02:40.940 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.940 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.940 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:40.941 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908089+0000", 2024-05-15T13:02:40.941 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8225e-05 2024-05-15T13:02:40.941 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.941 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.941 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:40.941 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908115+0000", 2024-05-15T13:02:40.941 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5627e-05 2024-05-15T13:02:40.941 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.941 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.941 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:40.941 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.966395+0000", 2024-05-15T13:02:40.941 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.058280294000000003 2024-05-15T13:02:40.941 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.941 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.942 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:40.942 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.966434+0000", 2024-05-15T13:02:40.942 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8702999999999999e-05 2024-05-15T13:02:40.942 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.942 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:40.942 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.942 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.942 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.942 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:167636 4.4 e371/317 4:2330c8ea:::benchmark_data_smithi177_99756_object47664:head v 371'10448, mlcod=371'10446)", 2024-05-15T13:02:40.942 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:54.908071+0000", 2024-05-15T13:02:40.942 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 105.95886333999999, 2024-05-15T13:02:40.942 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.064628550000000007, 2024-05-15T13:02:40.942 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:40.943 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:40.943 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:40.943 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.943 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:40.943 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908071+0000", 2024-05-15T13:02:40.943 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.943 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.943 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.943 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:40.943 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908071+0000", 2024-05-15T13:02:40.943 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.943 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.943 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.944 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:40.944 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908071+0000", 2024-05-15T13:02:40.944 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4000000000000002e-07 2024-05-15T13:02:40.944 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.944 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.944 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:40.944 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908075+0000", 2024-05-15T13:02:40.944 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4079999999999998e-06 2024-05-15T13:02:40.944 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.944 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.944 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:40.944 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908082+0000", 2024-05-15T13:02:40.944 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2609999999999996e-06 2024-05-15T13:02:40.944 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.945 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.945 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:40.945 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908087+0000", 2024-05-15T13:02:40.945 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1080000000000001e-06 2024-05-15T13:02:40.945 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.945 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.945 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:40.945 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908132+0000", 2024-05-15T13:02:40.945 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5312000000000003e-05 2024-05-15T13:02:40.945 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.945 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.945 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:40.945 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908151+0000", 2024-05-15T13:02:40.946 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8536999999999999e-05 2024-05-15T13:02:40.946 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.946 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.946 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:40.946 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.972658+0000", 2024-05-15T13:02:40.946 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.064507007000000005 2024-05-15T13:02:40.946 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.946 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.946 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:40.946 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.972699+0000", 2024-05-15T13:02:40.946 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1477000000000003e-05 2024-05-15T13:02:40.946 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.946 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:40.947 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.947 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.947 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.947 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:167640 4.c e371/317 4:3084d351:::benchmark_data_smithi177_99756_object47668:head v 371'10509, mlcod=371'10505)", 2024-05-15T13:02:40.947 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:54.909239+0000", 2024-05-15T13:02:40.947 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 105.957694925, 2024-05-15T13:02:40.947 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.065503817000000006, 2024-05-15T13:02:40.947 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:40.947 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:40.947 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:40.947 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.947 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:40.947 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.909239+0000", 2024-05-15T13:02:40.947 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.948 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.948 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:40.948 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.909239+0000", 2024-05-15T13:02:40.948 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.948 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.948 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:40.948 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.909240+0000", 2024-05-15T13:02:40.948 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8100000000000003e-07 2024-05-15T13:02:40.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.948 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.948 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:40.949 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.909245+0000", 2024-05-15T13:02:40.949 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1000000000000003e-06 2024-05-15T13:02:40.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.949 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.949 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:40.949 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.909256+0000", 2024-05-15T13:02:40.949 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1409e-05 2024-05-15T13:02:40.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.949 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.949 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:40.949 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.909263+0000", 2024-05-15T13:02:40.949 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5350000000000001e-06 2024-05-15T13:02:40.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.949 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.950 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:40.950 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.912707+0000", 2024-05-15T13:02:40.950 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003444145 2024-05-15T13:02:40.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.950 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.950 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:40.950 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.912727+0000", 2024-05-15T13:02:40.950 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0013e-05 2024-05-15T13:02:40.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.950 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.950 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:40.950 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.974702+0000", 2024-05-15T13:02:40.950 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.061974896000000002 2024-05-15T13:02:40.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.951 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.951 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:40.951 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.974743+0000", 2024-05-15T13:02:40.951 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1238000000000001e-05 2024-05-15T13:02:40.951 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.951 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:40.951 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.951 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.951 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.951 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:195002 4.0 4:0a6de0aa:::benchmark_data_smithi177_99756_object75030:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:02:40.951 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:39.022199+0000", 2024-05-15T13:02:40.951 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 61.844735292000003, 2024-05-15T13:02:40.951 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.061146316999999999, 2024-05-15T13:02:40.952 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:40.952 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:40.952 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:02:40.952 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:02:40.952 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:02:40.952 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 195002 2024-05-15T13:02:40.952 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.952 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:40.952 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.952 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:40.952 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.022199+0000", 2024-05-15T13:02:40.952 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.952 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.952 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.953 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:40.953 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.022199+0000", 2024-05-15T13:02:40.953 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.953 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.953 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.953 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:40.953 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.022199+0000", 2024-05-15T13:02:40.953 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2600000000000002e-07 2024-05-15T13:02:40.953 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.953 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.953 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:40.953 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.022203+0000", 2024-05-15T13:02:40.953 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2279999999999998e-06 2024-05-15T13:02:40.954 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.954 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.954 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:40.954 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.022212+0000", 2024-05-15T13:02:40.954 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4679999999999998e-06 2024-05-15T13:02:40.954 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.954 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.954 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:40.954 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.022219+0000", 2024-05-15T13:02:40.954 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2110000000000001e-06 2024-05-15T13:02:40.954 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.954 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.954 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:40.955 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.028574+0000", 2024-05-15T13:02:40.955 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0063544710000000004 2024-05-15T13:02:40.955 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.955 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.955 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:40.955 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.028665+0000", 2024-05-15T13:02:40.955 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1428000000000006e-05 2024-05-15T13:02:40.955 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.955 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.955 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,8", 2024-05-15T13:02:40.955 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.028762+0000", 2024-05-15T13:02:40.955 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7356999999999996e-05 2024-05-15T13:02:40.955 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.956 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.956 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:02:40.956 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.036978+0000", 2024-05-15T13:02:40.956 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0082155570000000001 2024-05-15T13:02:40.956 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.956 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.956 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:02:40.956 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.051023+0000", 2024-05-15T13:02:40.956 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014045095 2024-05-15T13:02:40.956 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.956 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.956 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:02:40.956 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.083238+0000", 2024-05-15T13:02:40.956 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.032215150999999997 2024-05-15T13:02:40.957 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.957 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.957 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:40.957 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.083297+0000", 2024-05-15T13:02:40.957 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8390000000000002e-05 2024-05-15T13:02:40.957 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.957 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.957 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:40.957 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.083345+0000", 2024-05-15T13:02:40.957 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8435e-05 2024-05-15T13:02:40.957 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.957 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:40.957 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.958 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.958 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.958 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:195093 4.f e385/343 4:f1629b82:::benchmark_data_smithi177_99756_object75121:head v 385'11904, mlcod=333'5297)", 2024-05-15T13:02:40.958 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:39.173239+0000", 2024-05-15T13:02:40.958 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 61.693694733999997, 2024-05-15T13:02:40.958 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.156115744, 2024-05-15T13:02:40.958 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:40.958 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:40.958 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:40.958 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.958 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:40.958 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.173239+0000", 2024-05-15T13:02:40.959 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.959 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.959 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:40.959 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.173239+0000", 2024-05-15T13:02:40.959 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.959 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.959 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:40.959 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.173240+0000", 2024-05-15T13:02:40.959 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5700000000000002e-07 2024-05-15T13:02:40.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.959 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.960 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:40.960 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.173246+0000", 2024-05-15T13:02:40.960 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.592e-06 2024-05-15T13:02:40.960 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.960 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.960 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:40.960 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.173253+0000", 2024-05-15T13:02:40.960 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0419999999999997e-06 2024-05-15T13:02:40.960 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.960 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.960 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:40.960 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.173258+0000", 2024-05-15T13:02:40.960 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3870000000000003e-06 2024-05-15T13:02:40.960 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.961 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.961 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:40.961 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.173307+0000", 2024-05-15T13:02:40.961 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9376000000000003e-05 2024-05-15T13:02:40.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.961 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.961 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:40.961 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.173326+0000", 2024-05-15T13:02:40.961 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9045000000000001e-05 2024-05-15T13:02:40.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.961 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.961 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:40.961 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329308+0000", 2024-05-15T13:02:40.961 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.15598142600000001 2024-05-15T13:02:40.962 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.962 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.962 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:40.962 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329355+0000", 2024-05-15T13:02:40.962 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7219000000000003e-05 2024-05-15T13:02:40.962 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.962 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:40.962 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.962 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.962 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.962 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:195062 4.9 e385/343 4:9dfccf82:::benchmark_data_smithi177_99756_object75090:head v 385'12098, mlcod=330'4626)", 2024-05-15T13:02:40.962 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:39.178226+0000", 2024-05-15T13:02:40.962 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 61.688708302999999, 2024-05-15T13:02:40.962 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16531588999999999, 2024-05-15T13:02:40.963 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:40.963 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:40.963 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:40.963 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.963 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:40.963 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.178226+0000", 2024-05-15T13:02:40.963 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.963 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.963 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.963 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:40.963 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.178226+0000", 2024-05-15T13:02:40.963 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.963 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.964 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.964 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:40.964 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.178227+0000", 2024-05-15T13:02:40.964 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7699999999999992e-07 2024-05-15T13:02:40.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.964 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.964 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:40.964 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.178236+0000", 2024-05-15T13:02:40.964 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0429999999999996e-06 2024-05-15T13:02:40.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.964 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.964 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:40.964 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.178248+0000", 2024-05-15T13:02:40.965 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2438e-05 2024-05-15T13:02:40.965 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.965 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.965 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:40.965 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.178257+0000", 2024-05-15T13:02:40.965 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0550000000000005e-06 2024-05-15T13:02:40.965 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.965 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.965 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:40.965 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.178314+0000", 2024-05-15T13:02:40.965 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6331e-05 2024-05-15T13:02:40.965 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.965 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.965 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:40.966 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.178342+0000", 2024-05-15T13:02:40.966 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8101000000000001e-05 2024-05-15T13:02:40.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.966 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.966 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:40.966 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.343498+0000", 2024-05-15T13:02:40.966 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16515676000000001 2024-05-15T13:02:40.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.966 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.966 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:40.966 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.343542+0000", 2024-05-15T13:02:40.966 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3185000000000001e-05 2024-05-15T13:02:40.966 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.967 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:40.967 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.967 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.967 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:195097 4.0 4:01df6f19:::benchmark_data_smithi177_99756_object75125:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:02:40.967 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:39.179776+0000", 2024-05-15T13:02:40.967 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 61.687158322000002, 2024-05-15T13:02:40.967 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16930087199999999, 2024-05-15T13:02:40.967 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:40.967 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:40.967 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:02:40.967 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:02:40.967 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:02:40.968 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 195097 2024-05-15T13:02:40.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.968 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:40.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.968 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:40.968 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.179776+0000", 2024-05-15T13:02:40.968 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.968 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:40.968 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.179776+0000", 2024-05-15T13:02:40.968 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.968 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:40.969 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.179777+0000", 2024-05-15T13:02:40.969 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5400000000000001e-06 2024-05-15T13:02:40.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.969 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.969 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:40.969 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.179783+0000", 2024-05-15T13:02:40.969 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.254e-06 2024-05-15T13:02:40.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.969 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.969 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:40.969 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329321+0000", 2024-05-15T13:02:40.969 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.14953818699999999 2024-05-15T13:02:40.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.969 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.970 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:40.970 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329331+0000", 2024-05-15T13:02:40.970 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0556e-05 2024-05-15T13:02:40.970 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.970 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.970 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:40.970 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329551+0000", 2024-05-15T13:02:40.970 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000219385 2024-05-15T13:02:40.970 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.970 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.970 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:40.970 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329679+0000", 2024-05-15T13:02:40.970 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000127961 2024-05-15T13:02:40.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.971 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.971 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,8", 2024-05-15T13:02:40.971 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329796+0000", 2024-05-15T13:02:40.971 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000117175 2024-05-15T13:02:40.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.971 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.971 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:02:40.971 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.344048+0000", 2024-05-15T13:02:40.971 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014252453999999999 2024-05-15T13:02:40.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.971 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.971 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:02:40.971 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.344176+0000", 2024-05-15T13:02:40.972 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012799999999999999 2024-05-15T13:02:40.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.972 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.972 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:02:40.972 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.348983+0000", 2024-05-15T13:02:40.972 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0048068659999999999 2024-05-15T13:02:40.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.972 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.972 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:40.972 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.349033+0000", 2024-05-15T13:02:40.972 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9848000000000001e-05 2024-05-15T13:02:40.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.972 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.972 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:40.973 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.349077+0000", 2024-05-15T13:02:40.973 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3646000000000002e-05 2024-05-15T13:02:40.973 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.973 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:40.973 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.973 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.973 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.973 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:195098 4.f e385/343 4:f78a4cdc:::benchmark_data_smithi177_99756_object75126:head v 385'11905, mlcod=333'5297)", 2024-05-15T13:02:40.973 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:39.180038+0000", 2024-05-15T13:02:40.973 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 61.686896288, 2024-05-15T13:02:40.973 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.158549575, 2024-05-15T13:02:40.973 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:40.973 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:40.974 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:40.974 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.974 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:40.974 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.180038+0000", 2024-05-15T13:02:40.974 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.974 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.974 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:40.974 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.180038+0000", 2024-05-15T13:02:40.974 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.974 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.975 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:40.975 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.180038+0000", 2024-05-15T13:02:40.975 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3099999999999997e-07 2024-05-15T13:02:40.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.975 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.975 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:40.975 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.180044+0000", 2024-05-15T13:02:40.975 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7130000000000002e-06 2024-05-15T13:02:40.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.975 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.975 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:40.975 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329299+0000", 2024-05-15T13:02:40.975 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.14925477400000001 2024-05-15T13:02:40.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.976 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.976 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:40.976 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329313+0000", 2024-05-15T13:02:40.976 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3796e-05 2024-05-15T13:02:40.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.976 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.976 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:40.976 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329383+0000", 2024-05-15T13:02:40.976 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0069999999999998e-05 2024-05-15T13:02:40.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.976 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.976 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:40.976 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329410+0000", 2024-05-15T13:02:40.977 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.694e-05 2024-05-15T13:02:40.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.977 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.977 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:40.977 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.338543+0000", 2024-05-15T13:02:40.977 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0091335180000000002 2024-05-15T13:02:40.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.977 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.977 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:40.977 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.338587+0000", 2024-05-15T13:02:40.977 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4132999999999999e-05 2024-05-15T13:02:40.977 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.977 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:40.978 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.978 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.978 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.978 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:199254 4.0 4:029e2200:::benchmark_data_smithi177_99756_object79282:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-05-15T13:02:40.978 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:45.214841+0000", 2024-05-15T13:02:40.978 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 55.652092600000003, 2024-05-15T13:02:40.978 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.054510022999999998, 2024-05-15T13:02:40.978 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:02:40.978 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:02:40.978 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:02:40.978 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:02:40.978 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:02:40.978 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 199254 2024-05-15T13:02:40.978 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.979 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:02:40.979 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.979 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:02:40.979 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.214841+0000", 2024-05-15T13:02:40.979 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.979 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.979 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.979 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:02:40.979 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.214841+0000", 2024-05-15T13:02:40.979 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:02:40.979 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.979 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.979 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:02:40.979 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.214842+0000", 2024-05-15T13:02:40.980 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0100000000000005e-07 2024-05-15T13:02:40.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.980 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.980 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:02:40.980 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.214845+0000", 2024-05-15T13:02:40.980 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.326e-06 2024-05-15T13:02:40.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.980 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.980 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:02:40.980 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.214852+0000", 2024-05-15T13:02:40.980 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8109999999999997e-06 2024-05-15T13:02:40.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.980 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.981 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:02:40.981 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.214858+0000", 2024-05-15T13:02:40.981 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5600000000000001e-06 2024-05-15T13:02:40.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.981 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.981 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:02:40.981 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.220146+0000", 2024-05-15T13:02:40.981 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0052882479999999997 2024-05-15T13:02:40.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.981 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.981 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:02:40.981 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.220279+0000", 2024-05-15T13:02:40.981 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000133526 2024-05-15T13:02:40.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.982 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.982 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,8", 2024-05-15T13:02:40.982 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.220403+0000", 2024-05-15T13:02:40.982 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012339800000000001 2024-05-15T13:02:40.982 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.982 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.982 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:02:40.982 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.237564+0000", 2024-05-15T13:02:40.982 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.017161372000000001 2024-05-15T13:02:40.982 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.982 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.982 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:02:40.983 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.237601+0000", 2024-05-15T13:02:40.983 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6820000000000003e-05 2024-05-15T13:02:40.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.983 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.983 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:02:40.983 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.269244+0000", 2024-05-15T13:02:40.983 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.031643114 2024-05-15T13:02:40.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.983 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.983 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:02:40.983 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.269302+0000", 2024-05-15T13:02:40.983 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7896000000000003e-05 2024-05-15T13:02:40.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:02:40.983 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:02:40.984 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:02:40.984 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.269351+0000", 2024-05-15T13:02:40.984 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9450999999999998e-05 2024-05-15T13:02:40.984 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.984 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:40.984 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.984 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:02:40.984 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:02:40.984 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:02:40.984 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:02:41.031 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:41.053 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:02:41.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:41.056 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:02:41.157 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:02:41.195 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:41.227 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:02:41.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:41.234 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:02:41.335 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:02:41.415 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:41.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:41 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/796186577' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:02:41.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:41 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:02:41.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:41 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/796186577' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:02:41.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:41 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:02:42.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:41 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/796186577' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:02:42.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:41 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:02:42.106 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:02:42.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:42.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:02:42.211 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:02:42.242 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:42.276 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:02:42.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:42.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:02:42.380 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:02:42.429 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:42.462 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:02:42.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:42.470 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:02:42.493 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Removed 119969 objects 2024-05-15T13:02:42.493 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Clean up completed and total clean up time :182.348 2024-05-15T13:02:42.535 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.2 to 1 (10000) 2024-05-15T13:02:42.574 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:02:42.601 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:42.621 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:02:42.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:42.624 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:02:42.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:42 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T13:02:42.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:42 smithi177 ceph-mon[44303]: osdmap e405: 12 total, 12 up, 11 in 2024-05-15T13:02:42.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:42 smithi177 ceph-mon[44303]: 4.b scrub starts 2024-05-15T13:02:42.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:42 smithi177 ceph-mon[44303]: 4.b scrub ok 2024-05-15T13:02:42.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:42 smithi177 ceph-mon[44303]: pgmap v1096: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 127 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 777 op/s; 13329/6057 objects degraded (220.059%); 4833/6057 objects misplaced (79.792%); 0 B/s, 105 objects/s recovering 2024-05-15T13:02:42.726 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:02:42.753 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:42.772 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:02:42.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:42.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:02:42.856 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:42 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T13:02:42.856 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:42 smithi027 ceph-mon[39795]: osdmap e405: 12 total, 12 up, 11 in 2024-05-15T13:02:42.856 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:42 smithi027 ceph-mon[39795]: 4.b scrub starts 2024-05-15T13:02:42.856 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:42 smithi027 ceph-mon[39795]: 4.b scrub ok 2024-05-15T13:02:42.856 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:42 smithi027 ceph-mon[39795]: pgmap v1096: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 127 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 777 op/s; 13329/6057 objects degraded (220.059%); 4833/6057 objects misplaced (79.792%); 0 B/s, 105 objects/s recovering 2024-05-15T13:02:42.876 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:02:42.901 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:42.923 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:02:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:42.926 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:02:43.027 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:02:43.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:42 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T13:02:43.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:42 smithi120 ceph-mon[43961]: osdmap e405: 12 total, 12 up, 11 in 2024-05-15T13:02:43.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:42 smithi120 ceph-mon[43961]: 4.b scrub starts 2024-05-15T13:02:43.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:42 smithi120 ceph-mon[43961]: 4.b scrub ok 2024-05-15T13:02:43.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:42 smithi120 ceph-mon[43961]: pgmap v1096: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 127 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 777 op/s; 13329/6057 objects degraded (220.059%); 4833/6057 objects misplaced (79.792%); 0 B/s, 105 objects/s recovering 2024-05-15T13:02:43.036 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-05-15T13:02:43.056 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:43.470 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:02:43.486 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:02:43.487 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 3 1 2024-05-15T13:02:43.506 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:02:43.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:43.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:02:43.611 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:02:43.631 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-05-15T13:02:43.631 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-05-15T13:02:43.640 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:43.740 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:02:43.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:43.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:02:43.744 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:43 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 13329/6057 objects degraded (220.059%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:43.744 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:43 smithi027 ceph-mon[39795]: osdmap e406: 12 total, 12 up, 11 in 2024-05-15T13:02:43.845 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:02:43.872 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:43 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 13329/6057 objects degraded (220.059%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:43.872 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:43 smithi177 ceph-mon[44303]: osdmap e406: 12 total, 12 up, 11 in 2024-05-15T13:02:43.874 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:43.896 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:02:43.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:43.900 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:02:44.001 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:02:44.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:43 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 13329/6057 objects degraded (220.059%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:44.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:43 smithi120 ceph-mon[43961]: osdmap e406: 12 total, 12 up, 11 in 2024-05-15T13:02:44.031 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:44.051 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:02:44.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:44.054 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:02:44.155 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:02:44.183 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:44.205 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:02:44.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:44.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:02:44.309 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:02:44.337 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:44.361 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:02:44.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:44.364 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:02:44.465 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:02:44.491 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:44.515 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:02:44.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:44.519 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:02:44.605 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:44 smithi027 ceph-mon[39795]: pgmap v1098: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 63 MiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 703 op/s; 12971/3021 objects degraded (429.361%); 4833/3021 objects misplaced (159.980%); 0 B/s, 100 objects/s recovering 2024-05-15T13:02:44.619 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:02:44.645 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:44.663 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:02:44.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:44.667 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:02:44.768 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:02:44.796 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:44.818 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:02:44.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:44.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:02:44.926 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:02:44.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:44 smithi177 ceph-mon[44303]: pgmap v1098: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 63 MiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 703 op/s; 12971/3021 objects degraded (429.361%); 4833/3021 objects misplaced (159.980%); 0 B/s, 100 objects/s recovering 2024-05-15T13:02:44.953 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:44 smithi120 ceph-mon[43961]: pgmap v1098: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 63 MiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 703 op/s; 12971/3021 objects degraded (429.361%); 4833/3021 objects misplaced (159.980%); 0 B/s, 100 objects/s recovering 2024-05-15T13:02:44.955 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:44.977 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:02:44.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:44.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:02:45.082 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:02:45.114 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:45.139 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:02:45.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:45.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:02:45.159 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:02:45.244 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:02:45.272 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:45.290 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:02:45.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:45.293 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:02:45.394 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:02:45.421 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:45.444 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:02:45.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:45.447 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:02:45.548 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:02:45.575 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:45.594 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:02:45.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:45.597 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:02:45.698 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:02:45.724 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:45.746 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:02:45.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:45.749 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:02:45.850 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:02:45.881 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:46.333 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:02:46.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:46.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:02:46.437 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:02:46.466 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:47.160 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:46 smithi027 ceph-mon[39795]: pgmap v1099: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 17 MiB data, 5.2 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 631 op/s; 12971/777 objects degraded (1669.369%); 4476/777 objects misplaced (576.062%); 0 B/s, 121 objects/s recovering 2024-05-15T13:02:47.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:46 smithi120 ceph-mon[43961]: pgmap v1099: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 17 MiB data, 5.2 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 631 op/s; 12971/777 objects degraded (1669.369%); 4476/777 objects misplaced (576.062%); 0 B/s, 121 objects/s recovering 2024-05-15T13:02:47.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:46 smithi177 ceph-mon[44303]: pgmap v1099: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 17 MiB data, 5.2 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 631 op/s; 12971/777 objects degraded (1669.369%); 4476/777 objects misplaced (576.062%); 0 B/s, 121 objects/s recovering 2024-05-15T13:02:47.720 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:02:47.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:47.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:02:47.824 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:02:47.850 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:47.870 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:02:47.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:47.872 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:02:47.973 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:02:47.991 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:02:48.000 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:48.019 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:02:48.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:48.022 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:02:48.123 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:02:48.154 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:48.174 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:02:48.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:48.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:02:48.279 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:02:48.306 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:48.328 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:02:48.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:48.331 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:02:48.432 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:02:48.458 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:48.477 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:02:48.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:48.480 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:02:48.581 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:02:48.611 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:48.710 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:02:48.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:48.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:02:48.807 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:02:48.814 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:02:48.842 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:49.042 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:02:49.045 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:48 smithi120 ceph-mon[43961]: pgmap v1100: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 420 op/s; 11130/6 objects degraded (185500.000%); 4476/6 objects misplaced (74600.000%); 0 B/s, 332 objects/s recovering 2024-05-15T13:02:49.045 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:48 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 11130/6 objects degraded (185500.000%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:49.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:49.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:02:49.147 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:02:49.174 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:49.192 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:02:49.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:49.199 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:02:49.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:48 smithi027 ceph-mon[39795]: pgmap v1100: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 420 op/s; 11130/6 objects degraded (185500.000%); 4476/6 objects misplaced (74600.000%); 0 B/s, 332 objects/s recovering 2024-05-15T13:02:49.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:48 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 11130/6 objects degraded (185500.000%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:49.300 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:02:49.332 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:49.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:48 smithi177 ceph-mon[44303]: pgmap v1100: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 420 op/s; 11130/6 objects degraded (185500.000%); 4476/6 objects misplaced (74600.000%); 0 B/s, 332 objects/s recovering 2024-05-15T13:02:49.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:48 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 11130/6 objects degraded (185500.000%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:50.583 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:02:50.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:50.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:02:50.688 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:02:50.719 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:50.741 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:02:50.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:50.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:02:50.846 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:02:50.874 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:50.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-15T13:02:50.917+0000 7fca2d568700 0 --2- 172.21.15.27:0/658430490 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca28107180 0x7fca2819e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-05-15T13:02:51.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:51 smithi027 ceph-mon[39795]: pgmap v1101: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 268 op/s; 11130/6 objects degraded (185500.000%); 4015/6 objects misplaced (66916.667%); 0 B/s, 361 objects/s recovering 2024-05-15T13:02:51.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:51 smithi027 ceph-mon[39795]: osdmap e407: 12 total, 12 up, 11 in 2024-05-15T13:02:51.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:51 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1280753260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:02:51.170 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:02:51.170 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:02:51.170 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:02:51.170 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:02:51.170 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:02:51.171 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:02:51.171 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:02:51.171 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:02:51.171 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:02:51.171 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:02:51.171 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:02:51.171 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:02:51.171 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:02:51.171 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:02:51.172 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:02:51.172 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:02:51.172 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:02:51.172 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:02:51.172 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:02:51.172 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:02:51.172 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:02:51.172 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:02:51.172 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:02:51.172 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:02:51.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:51 smithi120 ceph-mon[43961]: pgmap v1101: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 268 op/s; 11130/6 objects degraded (185500.000%); 4015/6 objects misplaced (66916.667%); 0 B/s, 361 objects/s recovering 2024-05-15T13:02:51.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:51 smithi120 ceph-mon[43961]: osdmap e407: 12 total, 12 up, 11 in 2024-05-15T13:02:51.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:51 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1280753260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:02:51.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:51 smithi177 ceph-mon[44303]: pgmap v1101: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 268 op/s; 11130/6 objects degraded (185500.000%); 4015/6 objects misplaced (66916.667%); 0 B/s, 361 objects/s recovering 2024-05-15T13:02:51.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:51 smithi177 ceph-mon[44303]: osdmap e407: 12 total, 12 up, 11 in 2024-05-15T13:02:51.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:51 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1280753260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:02:52.016 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:02:52.141 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:52 smithi027 ceph-mon[39795]: osdmap e408: 12 total, 12 up, 11 in 2024-05-15T13:02:52.141 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:52 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/658430490' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:02:52.227 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:02:52.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:52.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:02:52.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:52 smithi120 ceph-mon[43961]: osdmap e408: 12 total, 12 up, 11 in 2024-05-15T13:02:52.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:52 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/658430490' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:02:52.332 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:02:52.363 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:52.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:52 smithi177 ceph-mon[44303]: osdmap e408: 12 total, 12 up, 11 in 2024-05-15T13:02:52.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:52 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/658430490' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:02:52.466 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:02:52.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:52.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:02:52.570 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:02:52.596 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:52.616 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:02:52.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:52.619 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:02:52.720 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:02:52.746 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:52.767 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:02:52.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:52.770 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:02:52.871 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:02:52.902 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:53.149 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:53 smithi120 ceph-mon[43961]: pgmap v1104: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 125 op/s; 10631/6 objects degraded (177183.333%); 4015/6 objects misplaced (66916.667%); 0 B/s, 394 objects/s recovering 2024-05-15T13:02:53.150 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:53 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:02:53.150 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:53 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/658430490' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:02:53.150 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:53 smithi120 ceph-mon[43961]: osdmap e409: 12 total, 12 up, 11 in 2024-05-15T13:02:53.252 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:02:53.260 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:53 smithi027 ceph-mon[39795]: pgmap v1104: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 125 op/s; 10631/6 objects degraded (177183.333%); 4015/6 objects misplaced (66916.667%); 0 B/s, 394 objects/s recovering 2024-05-15T13:02:53.261 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:53 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:02:53.261 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:53 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/658430490' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:02:53.261 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:53 smithi027 ceph-mon[39795]: osdmap e409: 12 total, 12 up, 11 in 2024-05-15T13:02:53.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:53.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:02:53.363 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:02:53.391 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:53.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:53 smithi177 ceph-mon[44303]: pgmap v1104: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 125 op/s; 10631/6 objects degraded (177183.333%); 4015/6 objects misplaced (66916.667%); 0 B/s, 394 objects/s recovering 2024-05-15T13:02:53.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:53 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:02:53.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:53 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/658430490' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:02:53.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:53 smithi177 ceph-mon[44303]: osdmap e409: 12 total, 12 up, 11 in 2024-05-15T13:02:53.910 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:02:54.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:54 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:02:54.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:54 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:02:54.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:54 smithi027 ceph-mon[39795]: osdmap e410: 12 total, 12 up, 11 in 2024-05-15T13:02:54.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:54 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 10631/6 objects degraded (177183.333%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:54.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:54 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:02:54.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:54 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:02:54.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:54 smithi120 ceph-mon[43961]: osdmap e410: 12 total, 12 up, 11 in 2024-05-15T13:02:54.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:54 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 10631/6 objects degraded (177183.333%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:54.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:54 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:02:54.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:54 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:02:54.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:54 smithi177 ceph-mon[44303]: osdmap e410: 12 total, 12 up, 11 in 2024-05-15T13:02:54.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:54 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 10631/6 objects degraded (177183.333%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:54.687 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:02:54.687 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:02:54.933 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:02:55.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:55 smithi027 ceph-mon[39795]: pgmap v1107: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 14 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 9891/6 objects degraded (164850.000%); 4015/6 objects misplaced (66916.667%); 0 B/s, 309 objects/s recovering 2024-05-15T13:02:55.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:55 smithi120 ceph-mon[43961]: pgmap v1107: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 14 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 9891/6 objects degraded (164850.000%); 4015/6 objects misplaced (66916.667%); 0 B/s, 309 objects/s recovering 2024-05-15T13:02:55.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:55 smithi177 ceph-mon[44303]: pgmap v1107: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 14 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 9891/6 objects degraded (164850.000%); 4015/6 objects misplaced (66916.667%); 0 B/s, 309 objects/s recovering 2024-05-15T13:02:57.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:57 smithi027 ceph-mon[39795]: pgmap v1108: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 14 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 9891/6 objects degraded (164850.000%); 3314/6 objects misplaced (55233.333%); 0 B/s, 326 objects/s recovering 2024-05-15T13:02:57.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:57 smithi120 ceph-mon[43961]: pgmap v1108: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 14 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 9891/6 objects degraded (164850.000%); 3314/6 objects misplaced (55233.333%); 0 B/s, 326 objects/s recovering 2024-05-15T13:02:57.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:57 smithi177 ceph-mon[44303]: pgmap v1108: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 14 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 9891/6 objects degraded (164850.000%); 3314/6 objects misplaced (55233.333%); 0 B/s, 326 objects/s recovering 2024-05-15T13:02:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:02:58.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:02:58.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:02:58.999 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:02:59.025 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.3 to 1 (10000) 2024-05-15T13:02:59.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:59 smithi027 ceph-mon[39795]: pgmap v1109: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 14 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 9102/6 objects degraded (151700.000%); 3314/6 objects misplaced (55233.333%); 0 B/s, 393 objects/s recovering 2024-05-15T13:02:59.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:59 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 9102/6 objects degraded (151700.000%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:59.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:02:59 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3667538907' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:02:59.273 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:02:59.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:59 smithi120 ceph-mon[43961]: pgmap v1109: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 14 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 9102/6 objects degraded (151700.000%); 3314/6 objects misplaced (55233.333%); 0 B/s, 393 objects/s recovering 2024-05-15T13:02:59.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:59 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 9102/6 objects degraded (151700.000%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:59.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:02:59 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3667538907' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:02:59.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:59 smithi177 ceph-mon[44303]: pgmap v1109: 17 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 14 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 9102/6 objects degraded (151700.000%); 3314/6 objects misplaced (55233.333%); 0 B/s, 393 objects/s recovering 2024-05-15T13:02:59.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:59 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 9102/6 objects degraded (151700.000%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-05-15T13:02:59.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:02:59 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3667538907' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:03:00.039 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_2' removed 2024-05-15T13:03:00.419 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:03:00.420 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:03:00.420 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:00 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3667538907' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T13:03:00.420 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:00 smithi027 ceph-mon[39795]: osdmap e411: 12 total, 12 up, 11 in 2024-05-15T13:03:00.420 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:00 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3984319212' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T13:03:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:00 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3667538907' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T13:03:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:00 smithi177 ceph-mon[44303]: osdmap e411: 12 total, 12 up, 11 in 2024-05-15T13:03:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:00 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3984319212' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T13:03:00.462 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 4 1 2024-05-15T13:03:00.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:00 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3667538907' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T13:03:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:00 smithi120 ceph-mon[43961]: osdmap e411: 12 total, 12 up, 11 in 2024-05-15T13:03:00.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:00 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3984319212' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T13:03:01.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:01 smithi177 ceph-mon[44303]: pgmap v1111: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 14 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 8762/6 objects degraded (146033.333%); 2467/6 objects misplaced (41116.667%); 0 B/s, 431 objects/s recovering 2024-05-15T13:03:01.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:01 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3984319212' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T13:03:01.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:01 smithi177 ceph-mon[44303]: osdmap e412: 12 total, 12 up, 11 in 2024-05-15T13:03:01.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:01 smithi027 ceph-mon[39795]: pgmap v1111: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 14 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 8762/6 objects degraded (146033.333%); 2467/6 objects misplaced (41116.667%); 0 B/s, 431 objects/s recovering 2024-05-15T13:03:01.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:01 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3984319212' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T13:03:01.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:01 smithi027 ceph-mon[39795]: osdmap e412: 12 total, 12 up, 11 in 2024-05-15T13:03:01.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:01 smithi120 ceph-mon[43961]: pgmap v1111: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 14 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 8762/6 objects degraded (146033.333%); 2467/6 objects misplaced (41116.667%); 0 B/s, 431 objects/s recovering 2024-05-15T13:03:01.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:01 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3984319212' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T13:03:01.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:01 smithi120 ceph-mon[43961]: osdmap e412: 12 total, 12 up, 11 in 2024-05-15T13:03:01.649 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:03:01.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:01.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:03:01.753 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:03:01.785 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:01.807 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:03:01.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:01.810 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:03:01.911 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:03:01.940 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:02.224 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:02 smithi027 ceph-mon[39795]: pgmap v1113: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:02.248 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-05-15T13:03:02.248 INFO:tasks.radosbench:Beginning radosbench... 2024-05-15T13:03:02.249 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-05-15T13:03:02.249 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-05-15T13:03:02.368 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:02 smithi120 ceph-mon[43961]: pgmap v1113: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:02.370 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-05-15T13:03:02.391 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:03:02.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:02.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:03:02.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:02 smithi177 ceph-mon[44303]: pgmap v1113: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:02.495 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:03:02.527 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:02.549 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:03:02.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:02.552 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:03:02.653 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:03:02.683 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:02.704 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:03:02.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:02.707 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:03:02.809 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:03:02.837 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:03:02.838 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:03.087 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:03 smithi027 ceph-mon[39795]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8762/6 objects degraded (146033.333%), 2 pgs degraded, 2 pgs undersized) 2024-05-15T13:03:03.087 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:03 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:03:03.087 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:03 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:03:03.087 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:03 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:03:03.087 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:03 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:03:03.087 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:03 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:03:03.348 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:03:03.372 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:03:03.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:03.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:03:03.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:03 smithi177 ceph-mon[44303]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8762/6 objects degraded (146033.333%), 2 pgs degraded, 2 pgs undersized) 2024-05-15T13:03:03.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:03 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:03:03.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:03 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:03:03.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:03 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:03:03.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:03 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:03:03.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:03 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:03:03.480 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:03:03.507 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:03.526 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:03:03.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:03.529 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:03:03.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:03 smithi120 ceph-mon[43961]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8762/6 objects degraded (146033.333%), 2 pgs degraded, 2 pgs undersized) 2024-05-15T13:03:03.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:03 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:03:03.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:03 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:03:03.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:03 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:03:03.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:03 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:03:03.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:03 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:03:03.629 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:03:03.656 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:03.674 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:03:03.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:03.677 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:03:03.778 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:03:03.804 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:03.822 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:03:03.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:03.825 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:03:03.926 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:03:03.952 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:03.970 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:03:03.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:03.973 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:03:04.074 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:03:04.106 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:04.126 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:03:04.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:04.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:03:04.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:04 smithi177 ceph-mon[44303]: osdmap e413: 12 total, 12 up, 11 in 2024-05-15T13:03:04.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:04 smithi177 ceph-mon[44303]: osdmap e414: 12 total, 12 up, 11 in 2024-05-15T13:03:04.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:04 smithi177 ceph-mon[44303]: pgmap v1116: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:04.223 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:03:04.223 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:03:04.223 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:03:04.229 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:03:04.256 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:04.274 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:03:04.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:04.277 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:03:04.378 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:03:04.404 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:04.420 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:03:04.423 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:03:04.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:04.426 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:03:04.441 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:03:04.462 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:03:04.481 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:04 smithi027 ceph-mon[39795]: osdmap e413: 12 total, 12 up, 11 in 2024-05-15T13:03:04.482 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:04 smithi027 ceph-mon[39795]: osdmap e414: 12 total, 12 up, 11 in 2024-05-15T13:03:04.482 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:04 smithi027 ceph-mon[39795]: pgmap v1116: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:04.483 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:03:04.505 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:03:04.525 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:03:04.527 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:03:04.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:04 smithi120 ceph-mon[43961]: osdmap e413: 12 total, 12 up, 11 in 2024-05-15T13:03:04.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:04 smithi120 ceph-mon[43961]: osdmap e414: 12 total, 12 up, 11 in 2024-05-15T13:03:04.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:04 smithi120 ceph-mon[43961]: pgmap v1116: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:04.549 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:03:04.557 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:04.571 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:03:04.592 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:03:04.593 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:03:04.593 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:03:04.593 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:03:04.593 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:03:04.593 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:03:04.593 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:03:04.593 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:03:04.593 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:03:04.594 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:03:04.594 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:03:04.594 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:03:04.594 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:03:04.908 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:03:04.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:04.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:03:05.012 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:03:05.039 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:05.058 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:03:05.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:05.061 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:03:05.162 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:03:05.188 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:05.207 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:03:05.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:05.210 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:03:05.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:05 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3098190874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:03:05.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:05 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3746243741' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:03:05.290 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:03:05.311 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:03:05.336 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:05 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3098190874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:03:05.336 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:05 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3746243741' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:03:05.338 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:05.356 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:03:05.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:05.359 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:03:05.460 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:03:05.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:05 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3098190874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:03:05.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:05 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3746243741' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:03:05.494 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:05.516 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:03:05.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:05.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:03:05.621 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:03:05.647 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:05.666 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:03:05.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:05.669 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:03:05.770 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:03:05.796 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:05.815 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:03:05.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:05.818 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:03:05.919 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:03:05.952 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:05.974 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:03:05.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:05.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:03:06.029 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:03:06.077 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:03:06.109 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:06.130 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:03:06.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:06.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:03:06.234 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:03:06.260 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:06 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:03:06.260 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:06 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3746243741' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:03:06.260 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:06 smithi120 ceph-mon[43961]: osdmap e415: 12 total, 12 up, 11 in 2024-05-15T13:03:06.260 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:06 smithi120 ceph-mon[43961]: pgmap v1118: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:06.260 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:06 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:03:06.261 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:06 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3746243741' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:03:06.261 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:06 smithi027 ceph-mon[39795]: osdmap e415: 12 total, 12 up, 11 in 2024-05-15T13:03:06.261 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:06 smithi027 ceph-mon[39795]: pgmap v1118: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:06.262 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:06.300 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:03:06.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:06.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:03:06.404 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:03:06.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:06 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:03:06.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:06 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3746243741' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:03:06.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:06 smithi177 ceph-mon[44303]: osdmap e415: 12 total, 12 up, 11 in 2024-05-15T13:03:06.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:06 smithi177 ceph-mon[44303]: pgmap v1118: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:06.430 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:06.629 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:03:06.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:06.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:03:06.733 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:03:06.764 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:06.783 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:03:06.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:06.786 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:03:06.887 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:03:06.914 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:07.112 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:03:07.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:07.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:03:07.216 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:03:07.248 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:07.270 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:03:07.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:07.273 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:03:07.374 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:03:07.402 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:07.423 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:03:07.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:07.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:03:07.479 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:03:07.527 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:03:07.554 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:07.581 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:03:07.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:07.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:03:07.685 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:03:07.713 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:07.720 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:03:07.721 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T13:03:07.721 INFO:teuthology.orchestra.run.smithi120.stdout: "complaint_time": 30, 2024-05-15T13:03:07.721 INFO:teuthology.orchestra.run.smithi120.stdout: "num_blocked_ops": 0 2024-05-15T13:03:07.721 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:03:07.732 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:03:07.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:07.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:03:07.835 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:03:07.871 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:08.030 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:03:08.327 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-05-15T13:03:08.587 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:03:09.022 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:03:09.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:09.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:03:09.093 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:09 smithi120 ceph-mon[43961]: pgmap v1119: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:09.093 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:09 smithi120 ceph-mon[43961]: osdmap e416: 12 total, 12 up, 11 in 2024-05-15T13:03:09.093 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:09 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/829062597' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T13:03:09.125 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:03:09.155 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:09.227 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:03:09.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:09.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:03:09.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:09 smithi027 ceph-mon[39795]: pgmap v1119: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:09.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:09 smithi027 ceph-mon[39795]: osdmap e416: 12 total, 12 up, 11 in 2024-05-15T13:03:09.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:09 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/829062597' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T13:03:09.295 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.4 to 1 (10000) 2024-05-15T13:03:09.331 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:03:09.362 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:09.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:09 smithi177 ceph-mon[44303]: pgmap v1119: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:09.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:09 smithi177 ceph-mon[44303]: osdmap e416: 12 total, 12 up, 11 in 2024-05-15T13:03:09.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:09 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/829062597' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T13:03:09.551 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:03:09.555 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:03:09.811 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:03:09.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:09.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:03:09.916 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:03:09.948 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:09.969 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:03:09.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:09.972 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:03:10.073 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:03:10.104 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:10.115 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 5 1 2024-05-15T13:03:10.125 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:03:10.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:10.128 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:03:10.229 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:03:10.263 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:10.310 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_3' created 2024-05-15T13:03:10.364 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:03:10.364 INFO:teuthology.orchestra.run.smithi120.stdout: "size": 20, 2024-05-15T13:03:10.364 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 600, 2024-05-15T13:03:10.364 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [ 2024-05-15T13:03:10.364 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.364 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:101379 4.1 4:8bcb24a5:::benchmark_data_smithi177_99756_object101378:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e343)", 2024-05-15T13:03:10.364 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:59:11.312757+0000", 2024-05-15T13:03:10.364 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 239.05950212600001, 2024-05-15T13:03:10.364 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.550042221, 2024-05-15T13:03:10.364 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:03:10.364 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:03:10.365 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:03:10.365 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:03:10.365 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:03:10.365 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 101379 2024-05-15T13:03:10.365 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.365 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:03:10.365 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.365 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:03:10.365 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312757+0000", 2024-05-15T13:03:10.365 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.365 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.365 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.365 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:03:10.365 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312757+0000", 2024-05-15T13:03:10.366 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.366 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.366 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.366 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:03:10.366 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312759+0000", 2024-05-15T13:03:10.366 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3349999999999999e-06 2024-05-15T13:03:10.366 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.366 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.366 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:03:10.366 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312913+0000", 2024-05-15T13:03:10.367 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000154656 2024-05-15T13:03:10.367 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.367 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.367 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:03:10.367 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312926+0000", 2024-05-15T13:03:10.367 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3294e-05 2024-05-15T13:03:10.367 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.367 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.367 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:03:10.367 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312938+0000", 2024-05-15T13:03:10.367 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1751e-05 2024-05-15T13:03:10.367 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.367 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.368 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.368 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313019+0000", 2024-05-15T13:03:10.368 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1163999999999997e-05 2024-05-15T13:03:10.368 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.368 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.368 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:03:10.368 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313029+0000", 2024-05-15T13:03:10.368 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0047999999999999e-05 2024-05-15T13:03:10.368 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.368 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.368 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.368 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.851467+0000", 2024-05-15T13:03:10.368 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5384380449999999 2024-05-15T13:03:10.368 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.369 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.369 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:03:10.369 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.853365+0000", 2024-05-15T13:03:10.369 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001897302 2024-05-15T13:03:10.369 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.369 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.369 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 5,6,11", 2024-05-15T13:03:10.369 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.853497+0000", 2024-05-15T13:03:10.369 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000132058 2024-05-15T13:03:10.369 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.369 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.369 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:03:10.369 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.860348+0000", 2024-05-15T13:03:10.369 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0068511919999999999 2024-05-15T13:03:10.370 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.370 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.370 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:03:10.370 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.860430+0000", 2024-05-15T13:03:10.370 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1646999999999999e-05 2024-05-15T13:03:10.370 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.370 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.370 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:03:10.370 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.860447+0000", 2024-05-15T13:03:10.370 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7697e-05 2024-05-15T13:03:10.370 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.370 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.370 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:03:10.370 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.860501+0000", 2024-05-15T13:03:10.371 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3378999999999998e-05 2024-05-15T13:03:10.371 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.371 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.371 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:03:10.371 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.860575+0000", 2024-05-15T13:03:10.371 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3875000000000007e-05 2024-05-15T13:03:10.371 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.371 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.371 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:03:10.371 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.862799+0000", 2024-05-15T13:03:10.371 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0022247780000000002 2024-05-15T13:03:10.371 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.371 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:03:10.371 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.371 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.372 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.372 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:101386 4.1 4:83191a75:::benchmark_data_smithi177_99756_object101385:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e343)", 2024-05-15T13:03:10.372 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:59:11.312977+0000", 2024-05-15T13:03:10.372 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 239.05928194200001, 2024-05-15T13:03:10.372 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5704636750000001, 2024-05-15T13:03:10.372 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:03:10.372 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:03:10.372 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:03:10.372 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:03:10.372 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:03:10.372 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 101386 2024-05-15T13:03:10.372 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.372 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:03:10.372 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.372 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:03:10.373 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312977+0000", 2024-05-15T13:03:10.373 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.373 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.373 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.373 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:03:10.373 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312977+0000", 2024-05-15T13:03:10.373 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.373 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.373 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.373 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:03:10.373 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312978+0000", 2024-05-15T13:03:10.373 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.68e-07 2024-05-15T13:03:10.373 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.373 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.373 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:03:10.373 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313011+0000", 2024-05-15T13:03:10.374 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.2957999999999999e-05 2024-05-15T13:03:10.374 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.374 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.374 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:03:10.374 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313020+0000", 2024-05-15T13:03:10.374 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8950000000000004e-06 2024-05-15T13:03:10.374 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.374 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.374 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:03:10.374 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313027+0000", 2024-05-15T13:03:10.374 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7900000000000002e-06 2024-05-15T13:03:10.374 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.374 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.374 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.375 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313080+0000", 2024-05-15T13:03:10.375 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3122999999999999e-05 2024-05-15T13:03:10.375 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.375 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.375 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:03:10.375 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313096+0000", 2024-05-15T13:03:10.375 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5934e-05 2024-05-15T13:03:10.375 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.375 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.375 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.375 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.862932+0000", 2024-05-15T13:03:10.375 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5498364360000001 2024-05-15T13:03:10.375 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.375 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.375 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:03:10.375 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.863088+0000", 2024-05-15T13:03:10.376 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015589900000000001 2024-05-15T13:03:10.376 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.376 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.376 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 5,6,11", 2024-05-15T13:03:10.376 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.863214+0000", 2024-05-15T13:03:10.376 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012642600000000001 2024-05-15T13:03:10.376 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.376 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.376 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:03:10.376 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.869892+0000", 2024-05-15T13:03:10.376 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0066774360000000001 2024-05-15T13:03:10.376 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.376 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.376 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:03:10.376 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.869952+0000", 2024-05-15T13:03:10.377 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.0081000000000003e-05 2024-05-15T13:03:10.377 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.377 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.377 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:03:10.377 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.874357+0000", 2024-05-15T13:03:10.377 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0044051089999999999 2024-05-15T13:03:10.377 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.377 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.377 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:03:10.377 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.881092+0000", 2024-05-15T13:03:10.377 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.006734711 2024-05-15T13:03:10.377 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.377 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.377 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:03:10.378 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.881135+0000", 2024-05-15T13:03:10.378 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2799000000000001e-05 2024-05-15T13:03:10.378 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.378 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.378 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:03:10.378 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.883441+0000", 2024-05-15T13:03:10.378 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0023064100000000001 2024-05-15T13:03:10.378 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.378 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:03:10.378 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.378 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.378 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.378 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:101387 4.1 4:83e2b5a9:::benchmark_data_smithi177_99756_object101386:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e343)", 2024-05-15T13:03:10.378 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:59:11.313041+0000", 2024-05-15T13:03:10.379 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 239.059217872, 2024-05-15T13:03:10.379 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.578328618, 2024-05-15T13:03:10.379 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:03:10.379 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:03:10.379 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:03:10.379 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:03:10.379 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:03:10.379 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 101387 2024-05-15T13:03:10.379 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.379 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:03:10.380 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.380 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:03:10.380 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313041+0000", 2024-05-15T13:03:10.380 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.380 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.380 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.380 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:03:10.380 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313041+0000", 2024-05-15T13:03:10.380 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.380 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.380 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.380 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:03:10.380 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313042+0000", 2024-05-15T13:03:10.380 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.5799999999999999e-07 2024-05-15T13:03:10.381 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.381 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.381 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:03:10.381 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313149+0000", 2024-05-15T13:03:10.381 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00010729 2024-05-15T13:03:10.381 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.381 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.381 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:03:10.381 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313159+0000", 2024-05-15T13:03:10.381 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0047000000000001e-05 2024-05-15T13:03:10.381 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.381 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.381 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:03:10.381 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313167+0000", 2024-05-15T13:03:10.381 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4789999999999999e-06 2024-05-15T13:03:10.382 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.382 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.382 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.382 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313224+0000", 2024-05-15T13:03:10.382 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6832000000000001e-05 2024-05-15T13:03:10.382 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.382 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.382 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:03:10.382 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313235+0000", 2024-05-15T13:03:10.382 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1086e-05 2024-05-15T13:03:10.382 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.382 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.382 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.382 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.866156+0000", 2024-05-15T13:03:10.383 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.552920962 2024-05-15T13:03:10.383 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.383 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.383 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:03:10.383 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.866295+0000", 2024-05-15T13:03:10.383 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013931800000000001 2024-05-15T13:03:10.383 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.383 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.383 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 5,6,11", 2024-05-15T13:03:10.383 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.866413+0000", 2024-05-15T13:03:10.383 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000118336 2024-05-15T13:03:10.383 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.383 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.383 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:03:10.384 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.870229+0000", 2024-05-15T13:03:10.384 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0038157120000000002 2024-05-15T13:03:10.384 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.384 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.384 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:03:10.384 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.874291+0000", 2024-05-15T13:03:10.384 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0040620220000000002 2024-05-15T13:03:10.384 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.384 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.384 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:03:10.384 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.874400+0000", 2024-05-15T13:03:10.384 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000108929 2024-05-15T13:03:10.384 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.384 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.384 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:03:10.385 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.887902+0000", 2024-05-15T13:03:10.385 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.013501701 2024-05-15T13:03:10.385 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.385 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.385 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:03:10.385 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.887972+0000", 2024-05-15T13:03:10.385 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0091000000000004e-05 2024-05-15T13:03:10.385 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.385 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.385 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:03:10.385 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.891370+0000", 2024-05-15T13:03:10.385 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0033981549999999999 2024-05-15T13:03:10.385 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.385 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:03:10.385 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.386 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.386 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.386 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:140738 4.1 4:859c5c94:::benchmark_data_smithi177_99756_object20766:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e357)", 2024-05-15T13:03:10.386 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:00:11.249829+0000", 2024-05-15T13:03:10.386 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 179.12243007800001, 2024-05-15T13:03:10.386 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.770064275, 2024-05-15T13:03:10.386 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:03:10.386 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:03:10.386 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:03:10.386 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:03:10.386 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:03:10.386 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 140738 2024-05-15T13:03:10.386 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.386 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:03:10.387 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.387 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:03:10.387 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:11.249829+0000", 2024-05-15T13:03:10.387 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.387 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.387 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.387 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:03:10.387 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:11.249829+0000", 2024-05-15T13:03:10.387 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.387 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.387 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.387 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:03:10.387 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:11.249830+0000", 2024-05-15T13:03:10.387 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.3799999999999998e-07 2024-05-15T13:03:10.387 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.387 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.388 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:03:10.388 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:11.249831+0000", 2024-05-15T13:03:10.388 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.8619999999999999e-06 2024-05-15T13:03:10.388 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.388 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.388 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:03:10.388 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:11.249837+0000", 2024-05-15T13:03:10.388 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.4330000000000003e-06 2024-05-15T13:03:10.388 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.388 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.388 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:03:10.388 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:11.249841+0000", 2024-05-15T13:03:10.388 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.8319999999999999e-06 2024-05-15T13:03:10.388 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.388 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.389 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.389 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:11.250009+0000", 2024-05-15T13:03:10.389 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00016829200000000001 2024-05-15T13:03:10.389 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.389 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.389 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:03:10.389 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:11.250061+0000", 2024-05-15T13:03:10.389 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2185999999999998e-05 2024-05-15T13:03:10.389 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.389 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.389 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.389 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:13.019878+0000", 2024-05-15T13:03:10.389 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.769816678 2024-05-15T13:03:10.389 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.390 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.390 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:03:10.390 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:13.019893+0000", 2024-05-15T13:03:10.390 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5654e-05 2024-05-15T13:03:10.390 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.390 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:03:10.390 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.390 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.390 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.390 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:146672 4.1 4:89e32c44:::benchmark_data_smithi177_99756_object26700:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e359)", 2024-05-15T13:03:10.390 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:00:21.169464+0000", 2024-05-15T13:03:10.390 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 169.20279503500001, 2024-05-15T13:03:10.390 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.3409342730000002, 2024-05-15T13:03:10.391 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:03:10.391 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:03:10.391 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:03:10.391 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:03:10.391 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:03:10.391 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 146672 2024-05-15T13:03:10.391 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.391 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:03:10.391 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.391 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:03:10.391 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:21.169464+0000", 2024-05-15T13:03:10.391 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.391 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.391 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.391 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:03:10.391 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:21.169464+0000", 2024-05-15T13:03:10.392 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.392 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.392 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.392 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:03:10.392 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:21.169465+0000", 2024-05-15T13:03:10.392 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.51e-07 2024-05-15T13:03:10.392 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.392 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.392 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:03:10.392 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:21.169467+0000", 2024-05-15T13:03:10.392 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.7389999999999999e-06 2024-05-15T13:03:10.392 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.392 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.392 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:03:10.392 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:21.169475+0000", 2024-05-15T13:03:10.393 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9969999999999998e-06 2024-05-15T13:03:10.393 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.393 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.393 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:03:10.393 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:21.169481+0000", 2024-05-15T13:03:10.393 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6030000000000004e-06 2024-05-15T13:03:10.393 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.393 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.393 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.393 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:21.169520+0000", 2024-05-15T13:03:10.393 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.8849e-05 2024-05-15T13:03:10.393 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.393 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.393 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:03:10.393 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:21.169608+0000", 2024-05-15T13:03:10.394 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8508e-05 2024-05-15T13:03:10.394 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.394 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.394 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.394 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.510389+0000", 2024-05-15T13:03:10.394 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.3407809099999999 2024-05-15T13:03:10.394 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.394 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.394 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:03:10.394 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.510399+0000", 2024-05-15T13:03:10.394 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.2159999999999995e-06 2024-05-15T13:03:10.394 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.394 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:03:10.395 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.395 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.395 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.395 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:148132 4.1 4:81344e8a:::benchmark_data_smithi177_99756_object28160:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e361)", 2024-05-15T13:03:10.395 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:00:23.474291+0000", 2024-05-15T13:03:10.395 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 166.89796866399999, 2024-05-15T13:03:10.395 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0802568319999999, 2024-05-15T13:03:10.395 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:03:10.395 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:03:10.395 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:03:10.395 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:03:10.395 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:03:10.395 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 148132 2024-05-15T13:03:10.396 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.396 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:03:10.396 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.396 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:03:10.396 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474291+0000", 2024-05-15T13:03:10.396 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.396 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.396 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.396 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:03:10.396 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474291+0000", 2024-05-15T13:03:10.396 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.396 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.396 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.396 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:03:10.396 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474291+0000", 2024-05-15T13:03:10.397 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.7500000000000001e-07 2024-05-15T13:03:10.397 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.397 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.397 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:03:10.397 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474293+0000", 2024-05-15T13:03:10.397 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7999999999999999e-06 2024-05-15T13:03:10.397 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.397 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.397 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:03:10.397 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474298+0000", 2024-05-15T13:03:10.397 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5300000000000004e-06 2024-05-15T13:03:10.397 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.397 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.397 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:03:10.397 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474311+0000", 2024-05-15T13:03:10.397 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2347999999999999e-05 2024-05-15T13:03:10.398 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.398 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.398 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.398 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.477555+0000", 2024-05-15T13:03:10.398 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0032441499999999999 2024-05-15T13:03:10.398 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.398 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.398 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:03:10.398 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.477605+0000", 2024-05-15T13:03:10.398 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0139999999999998e-05 2024-05-15T13:03:10.398 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.398 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.398 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.399 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:27.554534+0000", 2024-05-15T13:03:10.399 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0769294489999997 2024-05-15T13:03:10.399 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.399 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.399 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:03:10.399 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:27.554547+0000", 2024-05-15T13:03:10.399 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.314e-05 2024-05-15T13:03:10.399 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.399 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:03:10.399 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.399 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.399 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.399 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:148138 4.1 4:8ed26e5a:::benchmark_data_smithi177_99756_object28166:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e361)", 2024-05-15T13:03:10.399 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:00:23.474317+0000", 2024-05-15T13:03:10.399 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 166.89794270300001, 2024-05-15T13:03:10.400 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.085580319, 2024-05-15T13:03:10.400 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:03:10.400 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:03:10.400 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:03:10.400 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:03:10.400 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:03:10.400 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 148138 2024-05-15T13:03:10.400 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.400 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:03:10.400 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.400 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:03:10.400 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474317+0000", 2024-05-15T13:03:10.400 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.400 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.400 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.400 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:03:10.401 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474317+0000", 2024-05-15T13:03:10.401 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.401 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.401 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.401 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:03:10.401 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474317+0000", 2024-05-15T13:03:10.401 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.9999999999999999e-07 2024-05-15T13:03:10.401 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.401 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.401 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:03:10.401 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474322+0000", 2024-05-15T13:03:10.401 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.4330000000000003e-06 2024-05-15T13:03:10.401 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.401 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.401 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:03:10.402 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474329+0000", 2024-05-15T13:03:10.402 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4699999999999999e-06 2024-05-15T13:03:10.402 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.402 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.402 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:03:10.402 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474333+0000", 2024-05-15T13:03:10.402 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2409999999999997e-06 2024-05-15T13:03:10.402 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.402 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.402 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.402 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.477641+0000", 2024-05-15T13:03:10.402 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0033075270000000002 2024-05-15T13:03:10.402 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.403 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.403 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:03:10.403 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.477720+0000", 2024-05-15T13:03:10.403 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.928e-05 2024-05-15T13:03:10.403 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.403 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.403 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.403 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:27.559884+0000", 2024-05-15T13:03:10.403 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0821640099999996 2024-05-15T13:03:10.403 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.403 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.403 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:03:10.403 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:27.559897+0000", 2024-05-15T13:03:10.403 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3057999999999999e-05 2024-05-15T13:03:10.404 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.404 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:03:10.404 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.404 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.404 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.404 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:148140 4.1 4:81937744:::benchmark_data_smithi177_99756_object28168:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e361)", 2024-05-15T13:03:10.404 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:00:23.474338+0000", 2024-05-15T13:03:10.404 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 166.89792089599999, 2024-05-15T13:03:10.404 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0889538669999999, 2024-05-15T13:03:10.404 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:03:10.404 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:03:10.404 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:03:10.404 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:03:10.404 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:03:10.405 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 148140 2024-05-15T13:03:10.405 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.405 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:03:10.405 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.405 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:03:10.405 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474338+0000", 2024-05-15T13:03:10.405 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.405 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.405 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.405 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:03:10.405 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474338+0000", 2024-05-15T13:03:10.405 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.405 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.405 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.405 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:03:10.406 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474339+0000", 2024-05-15T13:03:10.406 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.2099999999999998e-07 2024-05-15T13:03:10.406 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.406 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.406 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:03:10.406 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474341+0000", 2024-05-15T13:03:10.406 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9240000000000001e-06 2024-05-15T13:03:10.406 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.406 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.406 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:03:10.406 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474345+0000", 2024-05-15T13:03:10.406 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6349999999999997e-06 2024-05-15T13:03:10.406 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.406 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.407 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:03:10.407 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474349+0000", 2024-05-15T13:03:10.407 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0940000000000001e-06 2024-05-15T13:03:10.407 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.407 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.407 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.407 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.477766+0000", 2024-05-15T13:03:10.407 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00341625 2024-05-15T13:03:10.407 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.407 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.407 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:03:10.407 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.477861+0000", 2024-05-15T13:03:10.407 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.5136999999999994e-05 2024-05-15T13:03:10.407 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.407 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.407 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.408 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:27.563277+0000", 2024-05-15T13:03:10.408 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0854161339999999 2024-05-15T13:03:10.408 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.408 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.408 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:03:10.408 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:27.563292+0000", 2024-05-15T13:03:10.408 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5372000000000001e-05 2024-05-15T13:03:10.408 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.408 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:03:10.408 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.408 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.408 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.408 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:175091 4.1 4:8bb6ec4a:::benchmark_data_smithi177_99756_object55119:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e374)", 2024-05-15T13:03:10.408 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:06.598855+0000", 2024-05-15T13:03:10.408 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 123.773404278, 2024-05-15T13:03:10.409 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0032877979999999, 2024-05-15T13:03:10.409 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:03:10.409 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:03:10.409 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:03:10.409 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:03:10.409 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:03:10.409 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 175091 2024-05-15T13:03:10.409 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.409 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:03:10.409 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.409 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:03:10.409 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:06.598855+0000", 2024-05-15T13:03:10.409 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.409 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.409 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.410 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:03:10.410 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:06.598855+0000", 2024-05-15T13:03:10.410 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.410 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.410 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.410 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:03:10.410 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:06.598856+0000", 2024-05-15T13:03:10.410 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2309999999999999e-06 2024-05-15T13:03:10.410 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.410 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.410 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:03:10.410 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:06.598861+0000", 2024-05-15T13:03:10.410 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3669999999999999e-06 2024-05-15T13:03:10.411 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.411 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.411 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:03:10.411 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:06.598870+0000", 2024-05-15T13:03:10.411 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.3370000000000006e-06 2024-05-15T13:03:10.411 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.411 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.411 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:03:10.411 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:06.598878+0000", 2024-05-15T13:03:10.411 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1499999999999999e-06 2024-05-15T13:03:10.411 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.411 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.411 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.411 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:06.599921+0000", 2024-05-15T13:03:10.411 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001042935 2024-05-15T13:03:10.411 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.412 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.412 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:03:10.412 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:06.599989+0000", 2024-05-15T13:03:10.412 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7613999999999993e-05 2024-05-15T13:03:10.412 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.412 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.412 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.412 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.602128+0000", 2024-05-15T13:03:10.412 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.002138929 2024-05-15T13:03:10.412 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.412 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.412 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:03:10.412 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.602143+0000", 2024-05-15T13:03:10.412 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5235000000000001e-05 2024-05-15T13:03:10.412 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.413 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:03:10.413 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.413 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.413 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.413 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:175603 4.1 4:8bc99fd7:::benchmark_data_smithi177_99756_object55631:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e375)", 2024-05-15T13:03:10.413 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:07.585213+0000", 2024-05-15T13:03:10.413 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 122.787046171, 2024-05-15T13:03:10.413 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.6113388139999998, 2024-05-15T13:03:10.413 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:03:10.413 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:03:10.413 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:03:10.413 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:03:10.413 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:03:10.413 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 175603 2024-05-15T13:03:10.413 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.414 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:03:10.414 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.414 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:03:10.414 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.585213+0000", 2024-05-15T13:03:10.414 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.414 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.414 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.414 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:03:10.414 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.585213+0000", 2024-05-15T13:03:10.414 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.414 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.414 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.414 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:03:10.415 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.585214+0000", 2024-05-15T13:03:10.415 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4099999999999999e-07 2024-05-15T13:03:10.415 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.415 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.415 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:03:10.415 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.585216+0000", 2024-05-15T13:03:10.415 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.571e-06 2024-05-15T13:03:10.415 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.415 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.415 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:03:10.415 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.585227+0000", 2024-05-15T13:03:10.415 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0369e-05 2024-05-15T13:03:10.415 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.415 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.416 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:03:10.416 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.585231+0000", 2024-05-15T13:03:10.416 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.476e-06 2024-05-15T13:03:10.416 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.416 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.416 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.416 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.585442+0000", 2024-05-15T13:03:10.416 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00021099000000000001 2024-05-15T13:03:10.416 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.416 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.416 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:03:10.416 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.585496+0000", 2024-05-15T13:03:10.416 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3974999999999997e-05 2024-05-15T13:03:10.416 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.417 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.417 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.417 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:11.196538+0000", 2024-05-15T13:03:10.417 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.6110417510000001 2024-05-15T13:03:10.417 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.417 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.417 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:03:10.417 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:11.196552+0000", 2024-05-15T13:03:10.417 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4241e-05 2024-05-15T13:03:10.417 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.417 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:03:10.417 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.417 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.418 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.418 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:191878 4.1 4:84f87391:::benchmark_data_smithi177_99756_object71906:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:03:10.418 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:34.339639+0000", 2024-05-15T13:03:10.418 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 96.032620072, 2024-05-15T13:03:10.418 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.050434613, 2024-05-15T13:03:10.418 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:03:10.418 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:03:10.418 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:03:10.418 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:03:10.418 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:03:10.418 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 191878 2024-05-15T13:03:10.418 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.418 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:03:10.418 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.418 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:03:10.419 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339639+0000", 2024-05-15T13:03:10.419 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.419 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.419 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.419 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:03:10.419 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339639+0000", 2024-05-15T13:03:10.419 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.419 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.419 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.419 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:03:10.419 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339640+0000", 2024-05-15T13:03:10.419 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0599999999999999e-07 2024-05-15T13:03:10.419 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.419 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.420 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:03:10.420 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339644+0000", 2024-05-15T13:03:10.420 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7340000000000002e-06 2024-05-15T13:03:10.420 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.420 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.420 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:03:10.420 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339653+0000", 2024-05-15T13:03:10.420 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.9719999999999998e-06 2024-05-15T13:03:10.420 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.420 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.420 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:03:10.420 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339659+0000", 2024-05-15T13:03:10.420 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4670000000000001e-06 2024-05-15T13:03:10.421 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.421 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.421 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.421 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339742+0000", 2024-05-15T13:03:10.421 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2886000000000006e-05 2024-05-15T13:03:10.421 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.421 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.421 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:03:10.421 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339831+0000", 2024-05-15T13:03:10.421 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8708999999999994e-05 2024-05-15T13:03:10.421 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.421 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.421 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.421 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:35.390058+0000", 2024-05-15T13:03:10.421 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0502274650000001 2024-05-15T13:03:10.422 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.422 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.422 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:03:10.422 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:35.390074+0000", 2024-05-15T13:03:10.422 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5474000000000001e-05 2024-05-15T13:03:10.422 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.422 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:03:10.422 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.422 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.422 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.422 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:191882 4.1 4:8c08eda6:::benchmark_data_smithi177_99756_object71910:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:03:10.422 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:34.339674+0000", 2024-05-15T13:03:10.422 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 96.032584916000005, 2024-05-15T13:03:10.422 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.05286353, 2024-05-15T13:03:10.423 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:03:10.423 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:03:10.423 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:03:10.423 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:03:10.423 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:03:10.423 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 191882 2024-05-15T13:03:10.423 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.423 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:03:10.423 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.423 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:03:10.423 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339674+0000", 2024-05-15T13:03:10.423 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.423 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.423 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.424 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:03:10.424 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339674+0000", 2024-05-15T13:03:10.424 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.424 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.424 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.424 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:03:10.424 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339675+0000", 2024-05-15T13:03:10.424 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7199999999999999e-07 2024-05-15T13:03:10.424 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.424 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.424 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:03:10.424 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339678+0000", 2024-05-15T13:03:10.424 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.768e-06 2024-05-15T13:03:10.424 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.425 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.425 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:03:10.425 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339684+0000", 2024-05-15T13:03:10.425 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4049999999999997e-06 2024-05-15T13:03:10.425 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.425 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.425 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:03:10.425 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339688+0000", 2024-05-15T13:03:10.425 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5260000000000002e-06 2024-05-15T13:03:10.425 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.425 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.425 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.425 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339878+0000", 2024-05-15T13:03:10.425 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00019029800000000001 2024-05-15T13:03:10.426 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.426 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.426 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:03:10.426 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339937+0000", 2024-05-15T13:03:10.426 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9110000000000002e-05 2024-05-15T13:03:10.426 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.426 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.426 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.426 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:35.392531+0000", 2024-05-15T13:03:10.426 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0525935909999999 2024-05-15T13:03:10.426 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.426 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.426 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:03:10.427 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:35.392538+0000", 2024-05-15T13:03:10.427 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3599999999999998e-06 2024-05-15T13:03:10.427 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.427 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:03:10.427 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.427 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.427 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.427 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:191893 4.1 4:88f9917d:::benchmark_data_smithi177_99756_object71921:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:03:10.427 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:34.339696+0000", 2024-05-15T13:03:10.427 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 96.032563065999994, 2024-05-15T13:03:10.427 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.055227972, 2024-05-15T13:03:10.427 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:03:10.427 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:03:10.427 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:03:10.427 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:03:10.428 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:03:10.428 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 191893 2024-05-15T13:03:10.428 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.428 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:03:10.428 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.428 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:03:10.428 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339696+0000", 2024-05-15T13:03:10.428 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.428 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.428 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.428 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:03:10.428 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339696+0000", 2024-05-15T13:03:10.428 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.428 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.429 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.429 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:03:10.429 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339697+0000", 2024-05-15T13:03:10.429 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.6100000000000002e-07 2024-05-15T13:03:10.429 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.429 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.429 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:03:10.429 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339699+0000", 2024-05-15T13:03:10.429 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.5289999999999998e-06 2024-05-15T13:03:10.429 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.429 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.429 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:03:10.429 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339705+0000", 2024-05-15T13:03:10.430 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.0390000000000001e-06 2024-05-15T13:03:10.430 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.430 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.430 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:03:10.430 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339710+0000", 2024-05-15T13:03:10.430 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1930000000000001e-06 2024-05-15T13:03:10.430 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.430 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.430 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.430 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339975+0000", 2024-05-15T13:03:10.430 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000264443 2024-05-15T13:03:10.430 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.430 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.431 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:03:10.431 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.340057+0000", 2024-05-15T13:03:10.431 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1929999999999997e-05 2024-05-15T13:03:10.431 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.431 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.431 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.431 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:35.394916+0000", 2024-05-15T13:03:10.431 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0548597049999999 2024-05-15T13:03:10.431 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.431 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.431 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:03:10.431 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:35.394924+0000", 2024-05-15T13:03:10.431 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7719999999999994e-06 2024-05-15T13:03:10.431 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.431 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:03:10.432 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.432 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.432 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.432 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:191897 4.1 4:849b4824:::benchmark_data_smithi177_99756_object71925:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:03:10.432 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:34.339722+0000", 2024-05-15T13:03:10.432 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 96.032537062000003, 2024-05-15T13:03:10.432 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0569823270000001, 2024-05-15T13:03:10.432 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:03:10.432 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:03:10.432 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:03:10.432 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:03:10.432 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:03:10.432 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 191897 2024-05-15T13:03:10.432 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.432 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:03:10.432 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.433 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:03:10.433 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339722+0000", 2024-05-15T13:03:10.433 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.433 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.433 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.433 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:03:10.433 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339722+0000", 2024-05-15T13:03:10.433 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.433 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.433 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.433 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:03:10.433 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339723+0000", 2024-05-15T13:03:10.433 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.7599999999999998e-07 2024-05-15T13:03:10.433 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.433 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.433 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:03:10.434 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339724+0000", 2024-05-15T13:03:10.434 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4389999999999999e-06 2024-05-15T13:03:10.434 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.434 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.434 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:03:10.434 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339741+0000", 2024-05-15T13:03:10.434 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7385999999999999e-05 2024-05-15T13:03:10.434 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.434 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.434 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:03:10.434 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339745+0000", 2024-05-15T13:03:10.434 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.844e-06 2024-05-15T13:03:10.434 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.434 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.435 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.435 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.340088+0000", 2024-05-15T13:03:10.435 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00034268299999999999 2024-05-15T13:03:10.435 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.435 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.435 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:03:10.435 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.340143+0000", 2024-05-15T13:03:10.435 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5572000000000003e-05 2024-05-15T13:03:10.435 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.435 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.435 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.435 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:35.396697+0000", 2024-05-15T13:03:10.435 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0565533309999999 2024-05-15T13:03:10.435 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.436 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.436 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:03:10.436 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:35.396705+0000", 2024-05-15T13:03:10.436 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7959999999999996e-06 2024-05-15T13:03:10.436 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.436 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:03:10.436 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.436 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.436 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.436 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:191901 4.1 4:8bfc74de:::benchmark_data_smithi177_99756_object71929:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:03:10.436 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:34.339750+0000", 2024-05-15T13:03:10.436 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 96.032509094999995, 2024-05-15T13:03:10.436 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0593702620000001, 2024-05-15T13:03:10.436 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:03:10.437 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:03:10.437 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:03:10.437 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:03:10.437 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:03:10.437 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 191901 2024-05-15T13:03:10.437 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.437 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:03:10.437 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.437 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:03:10.437 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339750+0000", 2024-05-15T13:03:10.437 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.437 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.437 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.437 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:03:10.437 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339750+0000", 2024-05-15T13:03:10.438 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.438 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.438 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.438 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:03:10.438 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339750+0000", 2024-05-15T13:03:10.438 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.7700000000000001e-07 2024-05-15T13:03:10.438 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.438 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.438 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:03:10.438 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339755+0000", 2024-05-15T13:03:10.438 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3250000000000001e-06 2024-05-15T13:03:10.438 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.438 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.438 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:03:10.439 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339758+0000", 2024-05-15T13:03:10.439 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5980000000000001e-06 2024-05-15T13:03:10.439 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.439 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.439 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:03:10.439 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339761+0000", 2024-05-15T13:03:10.439 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.779e-06 2024-05-15T13:03:10.439 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.439 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.439 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.439 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.340180+0000", 2024-05-15T13:03:10.439 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00041860299999999998 2024-05-15T13:03:10.439 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.440 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.440 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:03:10.440 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.340272+0000", 2024-05-15T13:03:10.440 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1893999999999998e-05 2024-05-15T13:03:10.440 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.440 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.440 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.440 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:35.399110+0000", 2024-05-15T13:03:10.440 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0588386279999999 2024-05-15T13:03:10.440 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.440 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.440 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:03:10.440 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:35.399120+0000", 2024-05-15T13:03:10.440 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0158e-05 2024-05-15T13:03:10.441 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.441 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:03:10.441 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.441 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.441 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.441 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:193640 4.1 4:81b56918:::benchmark_data_smithi177_99756_object73668:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:03:10.441 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:37.012243+0000", 2024-05-15T13:03:10.441 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 93.360016282999993, 2024-05-15T13:03:10.441 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4679297899999999, 2024-05-15T13:03:10.441 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:03:10.441 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:03:10.441 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:03:10.441 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:03:10.441 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:03:10.441 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 193640 2024-05-15T13:03:10.442 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.442 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:03:10.442 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.442 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:03:10.442 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:37.012243+0000", 2024-05-15T13:03:10.442 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.442 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.442 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.442 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:03:10.442 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:37.012243+0000", 2024-05-15T13:03:10.442 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.442 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.442 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.442 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:03:10.442 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:37.012244+0000", 2024-05-15T13:03:10.443 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3549999999999999e-06 2024-05-15T13:03:10.443 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.443 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.443 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:03:10.443 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:37.012249+0000", 2024-05-15T13:03:10.443 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9250000000000003e-06 2024-05-15T13:03:10.443 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.443 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.443 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:03:10.443 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:37.012259+0000", 2024-05-15T13:03:10.443 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.5300000000000002e-06 2024-05-15T13:03:10.443 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.443 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.443 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:03:10.443 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:37.012266+0000", 2024-05-15T13:03:10.443 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.5839999999999997e-06 2024-05-15T13:03:10.444 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.444 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.444 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.444 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:37.013803+0000", 2024-05-15T13:03:10.444 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0015370850000000001 2024-05-15T13:03:10.444 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.444 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.444 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:03:10.444 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:37.013909+0000", 2024-05-15T13:03:10.444 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000105258 2024-05-15T13:03:10.444 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.444 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.444 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.444 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:38.480163+0000", 2024-05-15T13:03:10.445 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4662545730000001 2024-05-15T13:03:10.445 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.445 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.445 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:03:10.445 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:38.480173+0000", 2024-05-15T13:03:10.445 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.4800000000000007e-06 2024-05-15T13:03:10.445 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.445 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:03:10.445 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.445 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.445 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.445 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:195199 4.1 4:85c111cf:::benchmark_data_smithi177_99756_object75227:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:03:10.445 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:39.392643+0000", 2024-05-15T13:03:10.446 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 90.979616738999994, 2024-05-15T13:03:10.446 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2585088790000001, 2024-05-15T13:03:10.446 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:03:10.446 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:03:10.446 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:03:10.446 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:03:10.446 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:03:10.446 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 195199 2024-05-15T13:03:10.446 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.446 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:03:10.446 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.446 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:03:10.446 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.392643+0000", 2024-05-15T13:03:10.447 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.447 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.447 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.447 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:03:10.447 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.392643+0000", 2024-05-15T13:03:10.447 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.447 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.447 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.447 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:03:10.447 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.392643+0000", 2024-05-15T13:03:10.447 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2500000000000001e-07 2024-05-15T13:03:10.447 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.447 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.447 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:03:10.447 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.392645+0000", 2024-05-15T13:03:10.448 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1299999999999999e-06 2024-05-15T13:03:10.448 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.448 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.448 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:03:10.448 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.392652+0000", 2024-05-15T13:03:10.448 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9120000000000001e-06 2024-05-15T13:03:10.448 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.448 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.448 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:03:10.448 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.392657+0000", 2024-05-15T13:03:10.448 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7559999999999999e-06 2024-05-15T13:03:10.448 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.448 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.448 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.449 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.394772+0000", 2024-05-15T13:03:10.449 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0021149189999999998 2024-05-15T13:03:10.449 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.449 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.449 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:03:10.449 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.394853+0000", 2024-05-15T13:03:10.449 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1406e-05 2024-05-15T13:03:10.449 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.449 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.449 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.449 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.651138+0000", 2024-05-15T13:03:10.449 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2562850999999999 2024-05-15T13:03:10.449 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.449 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.449 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:03:10.450 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.651151+0000", 2024-05-15T13:03:10.450 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3230999999999999e-05 2024-05-15T13:03:10.450 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.450 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:03:10.450 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.450 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.450 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.450 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:195204 4.1 4:8d0019e2:::benchmark_data_smithi177_99756_object75232:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:03:10.450 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:39.392710+0000", 2024-05-15T13:03:10.450 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 90.979549691000003, 2024-05-15T13:03:10.450 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2609789259999999, 2024-05-15T13:03:10.450 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:03:10.450 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:03:10.451 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:03:10.451 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:03:10.451 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:03:10.451 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 195204 2024-05-15T13:03:10.451 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.451 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:03:10.451 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.451 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:03:10.451 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.392710+0000", 2024-05-15T13:03:10.451 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.451 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.451 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.451 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:03:10.451 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.392710+0000", 2024-05-15T13:03:10.451 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.452 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.452 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.452 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:03:10.452 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.392710+0000", 2024-05-15T13:03:10.452 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.1e-07 2024-05-15T13:03:10.452 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.452 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.452 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:03:10.452 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.392715+0000", 2024-05-15T13:03:10.452 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8439999999999997e-06 2024-05-15T13:03:10.452 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.452 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.452 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:03:10.453 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.392720+0000", 2024-05-15T13:03:10.453 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6030000000000004e-06 2024-05-15T13:03:10.453 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.453 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.453 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:03:10.453 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.392734+0000", 2024-05-15T13:03:10.453 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3867e-05 2024-05-15T13:03:10.453 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.453 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.453 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.453 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.394882+0000", 2024-05-15T13:03:10.453 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0021474319999999999 2024-05-15T13:03:10.453 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.454 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.454 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:03:10.454 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.394958+0000", 2024-05-15T13:03:10.454 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.6174999999999995e-05 2024-05-15T13:03:10.454 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.454 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.454 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.454 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.653674+0000", 2024-05-15T13:03:10.454 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2587164660000001 2024-05-15T13:03:10.454 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.454 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.454 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:03:10.454 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.653689+0000", 2024-05-15T13:03:10.454 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4229e-05 2024-05-15T13:03:10.455 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.455 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:03:10.455 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.455 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.455 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.455 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:196084 4.1 4:88a7e217:::benchmark_data_smithi177_99756_object76112:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:03:10.455 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:40.633254+0000", 2024-05-15T13:03:10.455 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 89.739005587999998, 2024-05-15T13:03:10.455 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4718218709999999, 2024-05-15T13:03:10.455 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:03:10.455 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:03:10.455 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:03:10.455 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:03:10.456 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:03:10.456 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 196084 2024-05-15T13:03:10.456 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.456 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:03:10.456 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.456 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:03:10.456 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633254+0000", 2024-05-15T13:03:10.456 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.456 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.456 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.456 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:03:10.456 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633254+0000", 2024-05-15T13:03:10.456 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.456 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.457 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.457 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:03:10.457 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633254+0000", 2024-05-15T13:03:10.457 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0699999999999998e-07 2024-05-15T13:03:10.457 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.457 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.457 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:03:10.457 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633256+0000", 2024-05-15T13:03:10.457 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2919999999999998e-06 2024-05-15T13:03:10.457 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.457 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.457 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:03:10.457 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633263+0000", 2024-05-15T13:03:10.457 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.472e-06 2024-05-15T13:03:10.458 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.458 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.458 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:03:10.458 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633269+0000", 2024-05-15T13:03:10.458 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9460000000000003e-06 2024-05-15T13:03:10.458 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.458 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.458 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.458 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633671+0000", 2024-05-15T13:03:10.458 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00040208399999999998 2024-05-15T13:03:10.458 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.458 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.458 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:03:10.458 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633762+0000", 2024-05-15T13:03:10.459 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1438000000000001e-05 2024-05-15T13:03:10.459 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.459 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.459 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.459 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:42.105066+0000", 2024-05-15T13:03:10.459 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4713031750000001 2024-05-15T13:03:10.459 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.459 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.459 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:03:10.459 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:42.105076+0000", 2024-05-15T13:03:10.459 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0057e-05 2024-05-15T13:03:10.459 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.459 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:03:10.460 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.460 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.460 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.460 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:196086 4.1 4:88127b73:::benchmark_data_smithi177_99756_object76114:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:03:10.460 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:40.633276+0000", 2024-05-15T13:03:10.460 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 89.738983078999993, 2024-05-15T13:03:10.460 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.479033697, 2024-05-15T13:03:10.460 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:03:10.460 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:03:10.460 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:03:10.460 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:03:10.460 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:03:10.460 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 196086 2024-05-15T13:03:10.460 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.461 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:03:10.461 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.461 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:03:10.461 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633276+0000", 2024-05-15T13:03:10.461 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.461 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.461 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.461 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:03:10.461 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633276+0000", 2024-05-15T13:03:10.461 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:03:10.461 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.461 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.461 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:03:10.461 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633277+0000", 2024-05-15T13:03:10.462 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7e-07 2024-05-15T13:03:10.462 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.462 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.462 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:03:10.462 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633279+0000", 2024-05-15T13:03:10.462 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.03e-06 2024-05-15T13:03:10.462 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.462 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.462 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:03:10.462 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633284+0000", 2024-05-15T13:03:10.462 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.553e-06 2024-05-15T13:03:10.462 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.462 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.463 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:03:10.463 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633289+0000", 2024-05-15T13:03:10.463 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0710000000000002e-06 2024-05-15T13:03:10.463 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.463 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.463 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.463 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633792+0000", 2024-05-15T13:03:10.463 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00050264299999999999 2024-05-15T13:03:10.463 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.463 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.463 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:03:10.463 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633873+0000", 2024-05-15T13:03:10.463 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0603000000000001e-05 2024-05-15T13:03:10.463 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.463 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.464 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:03:10.464 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:42.112301+0000", 2024-05-15T13:03:10.464 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.478428528 2024-05-15T13:03:10.464 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:03:10.464 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:03:10.464 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:03:10.464 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:42.112310+0000", 2024-05-15T13:03:10.464 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.799e-06 2024-05-15T13:03:10.464 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.464 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:03:10.464 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.464 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:03:10.464 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:03:10.464 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:03:10.465 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:03:10.465 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:10 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/829062597' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T13:03:10.465 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:10 smithi027 ceph-mon[39795]: osdmap e417: 12 total, 12 up, 11 in 2024-05-15T13:03:10.465 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:10 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:03:10.465 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:10 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:03:10.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:10 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1859703428' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-05-15T13:03:10.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:10 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-05-15T13:03:10.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:10 smithi027 ceph-mon[39795]: pgmap v1122: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:10.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:10 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:03:10.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:10 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:03:10.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:10.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:03:10.567 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:03:10.597 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:10.619 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:03:10.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:10.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:03:10.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:10 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/829062597' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T13:03:10.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:10 smithi177 ceph-mon[44303]: osdmap e417: 12 total, 12 up, 11 in 2024-05-15T13:03:10.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:10 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:03:10.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:10 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:03:10.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:10 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1859703428' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-05-15T13:03:10.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:10 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-05-15T13:03:10.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:10 smithi177 ceph-mon[44303]: pgmap v1122: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:10.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:10 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:03:10.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:10 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:03:10.722 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:03:10.753 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:10.757 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:10 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/829062597' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T13:03:10.757 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:10 smithi120 ceph-mon[43961]: osdmap e417: 12 total, 12 up, 11 in 2024-05-15T13:03:10.757 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:10 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:03:10.758 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:10 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:03:10.758 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:10 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1859703428' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-05-15T13:03:10.758 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:10 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-05-15T13:03:10.758 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:10 smithi120 ceph-mon[43961]: pgmap v1122: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:10.758 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:10 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:03:10.758 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:10 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:03:10.774 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:03:10.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:10.777 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:03:10.878 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:03:10.905 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:11.005 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:03:11.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:11.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:03:11.089 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-05-15T13:03:11.108 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:03:11.140 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:11.161 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:03:11.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:11.164 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:03:11.265 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:03:11.294 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:11.315 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:03:11.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:11.322 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:03:11.354 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:03:11.422 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:03:11.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:11 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-05-15T13:03:11.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:11 smithi177 ceph-mon[44303]: osdmap e418: 12 total, 12 up, 11 in 2024-05-15T13:03:11.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:11 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:03:11.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:11 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:03:11.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:11 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:03:11.451 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:11.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:11 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-05-15T13:03:11.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:11 smithi120 ceph-mon[43961]: osdmap e418: 12 total, 12 up, 11 in 2024-05-15T13:03:11.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:11 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:03:11.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:11 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:03:11.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:11 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:03:11.550 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:03:11.552 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:11 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-05-15T13:03:11.552 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:11 smithi027 ceph-mon[39795]: osdmap e418: 12 total, 12 up, 11 in 2024-05-15T13:03:11.552 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:11 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:03:11.552 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:11 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:03:11.552 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:11 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:03:11.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:11.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:03:11.654 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:03:11.682 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:11.702 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:03:11.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:11.705 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:03:11.806 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:03:11.834 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:11.907 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:03:11.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:11.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:03:12.011 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:03:12.038 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:12.549 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:03:12.570 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:03:12.592 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:03:12.612 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:03:12.633 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:03:12.654 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:03:12.655 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:12 smithi120 ceph-mon[43961]: osdmap e419: 12 total, 12 up, 11 in 2024-05-15T13:03:12.655 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:12 smithi120 ceph-mon[43961]: pgmap v1125: 17 pgs: 3 active+clean, 2 creating+peering, 12 unknown; 577 KiB data, 5.0 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:12.675 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:03:12.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:12 smithi177 ceph-mon[44303]: osdmap e419: 12 total, 12 up, 11 in 2024-05-15T13:03:12.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:12 smithi177 ceph-mon[44303]: pgmap v1125: 17 pgs: 3 active+clean, 2 creating+peering, 12 unknown; 577 KiB data, 5.0 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:12.687 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:03:12.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:12.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:03:12.696 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:03:12.717 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:03:12.738 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:03:12.759 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:03:12.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:12 smithi027 ceph-mon[39795]: osdmap e419: 12 total, 12 up, 11 in 2024-05-15T13:03:12.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:12 smithi027 ceph-mon[39795]: pgmap v1125: 17 pgs: 3 active+clean, 2 creating+peering, 12 unknown; 577 KiB data, 5.0 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:12.779 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:03:12.780 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:03:12.780 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:03:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:03:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:03:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:03:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:03:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:03:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:03:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:03:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:03:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:03:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:03:12.791 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:03:12.824 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:12.845 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:03:12.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:12.848 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:03:12.949 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:03:12.980 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:13.130 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:03:13.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:13.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:03:13.235 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:03:13.250 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:03:13.250 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:03:13.264 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:13.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:13 smithi027 ceph-mon[39795]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T13:03:13.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:13 smithi027 ceph-mon[39795]: osdmap e420: 12 total, 12 up, 11 in 2024-05-15T13:03:13.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:13 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3442234011' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:03:13.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:13 smithi120 ceph-mon[43961]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T13:03:13.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:13 smithi120 ceph-mon[43961]: osdmap e420: 12 total, 12 up, 11 in 2024-05-15T13:03:13.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:13 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3442234011' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:03:13.609 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-05-15T13:03:13.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:13 smithi177 ceph-mon[44303]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T13:03:13.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:13 smithi177 ceph-mon[44303]: osdmap e420: 12 total, 12 up, 11 in 2024-05-15T13:03:13.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:13 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3442234011' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:03:13.915 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:03:13.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:13.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:03:13.962 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-05-15T13:03:14.021 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:03:14.047 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:14.066 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:03:14.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:14.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:03:14.170 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:03:14.198 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:14.228 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:03:14.397 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:03:14.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:14.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:03:14.417 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:03:14.426 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:03:14.505 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:03:14.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:14 smithi027 ceph-mon[39795]: pgmap v1127: 17 pgs: 11 active+clean, 2 creating+peering, 4 unknown; 577 KiB data, 5.0 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:14.537 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:14.558 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:03:14.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:14.562 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:03:14.663 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:03:14.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:14 smithi177 ceph-mon[44303]: pgmap v1127: 17 pgs: 11 active+clean, 2 creating+peering, 4 unknown; 577 KiB data, 5.0 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:14.689 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:14 smithi120 ceph-mon[43961]: pgmap v1127: 17 pgs: 11 active+clean, 2 creating+peering, 4 unknown; 577 KiB data, 5.0 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:14.692 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:15.441 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:03:15.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:15.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:03:15.546 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:03:15.578 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:15.599 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:03:15.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:15.602 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:03:15.704 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:03:15.733 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:15.753 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:03:15.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:15.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:03:15.858 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:03:15.890 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:15.911 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:03:15.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:15.915 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:03:15.931 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:03:16.000 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:03:16.000 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:03:16.016 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:03:16.043 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:16.243 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:03:16.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:16.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:03:16.347 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:03:16.380 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:16.401 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:03:16.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:16.404 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:03:16.505 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:03:16.538 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:16.644 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:03:16.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:16.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:03:16.738 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-05-15T13:03:16.764 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:03:16.791 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:16.997 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:03:17.099 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:03:17.263 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:17 smithi120 ceph-mon[43961]: pgmap v1128: 17 pgs: 16 active+clean, 1 creating+peering; 577 KiB data, 5.0 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:17.264 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:17 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3121459893' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:03:17.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:17 smithi027 ceph-mon[39795]: pgmap v1128: 17 pgs: 16 active+clean, 1 creating+peering; 577 KiB data, 5.0 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:17.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:17 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3121459893' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:03:17.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:17 smithi177 ceph-mon[44303]: pgmap v1128: 17 pgs: 16 active+clean, 1 creating+peering; 577 KiB data, 5.0 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:17.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:17 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3121459893' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:03:17.941 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:03:17.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:17.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:03:18.049 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:03:18.079 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:18.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:18 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:03:18.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:18 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3121459893' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:03:18.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:18 smithi027 ceph-mon[39795]: osdmap e421: 12 total, 12 up, 11 in 2024-05-15T13:03:18.276 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:18 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:03:18.276 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:18 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3121459893' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:03:18.276 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:18 smithi120 ceph-mon[43961]: osdmap e421: 12 total, 12 up, 11 in 2024-05-15T13:03:18.277 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:03:18.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:18.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:03:18.383 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:03:18.413 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:18 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:03:18.413 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:18 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3121459893' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:03:18.413 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:18 smithi177 ceph-mon[44303]: osdmap e421: 12 total, 12 up, 11 in 2024-05-15T13:03:18.415 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:18.435 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:03:18.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:18.439 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:03:18.540 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:03:18.569 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:18.708 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:03:18.719 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:03:18.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:18.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:03:18.823 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:03:18.852 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:18.878 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:03:18.878 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:03:18.951 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:03:18.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:18.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:03:19.008 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.5 to 1 (10000) 2024-05-15T13:03:19.008 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-05-15T13:03:19.055 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:03:19.083 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:19.103 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:03:19.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:19.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:03:19.207 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:03:19.219 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:19 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:03:19.220 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:19 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:03:19.220 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:19 smithi120 ceph-mon[43961]: pgmap v1130: 17 pgs: 17 active+clean; 577 KiB data, 5.0 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:19.220 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:19 smithi120 ceph-mon[43961]: osdmap e422: 12 total, 12 up, 11 in 2024-05-15T13:03:19.220 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:19 smithi120 ceph-mon[43961]: from='client.15525 172.21.15.27:0/3290232959' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T13:03:19.220 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:19 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1722383910' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T13:03:19.239 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:19.261 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:03:19.263 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:19 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:03:19.263 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:19 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:03:19.263 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:19 smithi177 ceph-mon[44303]: pgmap v1130: 17 pgs: 17 active+clean; 577 KiB data, 5.0 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:19.263 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:19 smithi177 ceph-mon[44303]: osdmap e422: 12 total, 12 up, 11 in 2024-05-15T13:03:19.263 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:19 smithi177 ceph-mon[44303]: from='client.15525 172.21.15.27:0/3290232959' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T13:03:19.263 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:19 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1722383910' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T13:03:19.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:19.264 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:03:19.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:19 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:03:19.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:19 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:03:19.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:19 smithi027 ceph-mon[39795]: pgmap v1130: 17 pgs: 17 active+clean; 577 KiB data, 5.0 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:19.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:19 smithi027 ceph-mon[39795]: osdmap e422: 12 total, 12 up, 11 in 2024-05-15T13:03:19.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:19 smithi027 ceph-mon[39795]: from='client.15525 172.21.15.27:0/3290232959' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T13:03:19.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:19 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1722383910' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T13:03:19.365 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:03:19.395 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:19.414 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:03:19.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:19.418 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:03:19.519 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:03:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:19.567 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-05-15T13:03:19.826 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:03:19.897 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:03:19.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:19.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:03:20.002 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:03:20.032 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:20.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:20 smithi027 ceph-mon[39795]: from='client.15525 172.21.15.27:0/3290232959' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T13:03:20.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:20 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1722383910' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T13:03:20.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:20 smithi027 ceph-mon[39795]: osdmap e423: 12 total, 12 up, 11 in 2024-05-15T13:03:20.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:20 smithi120 ceph-mon[43961]: from='client.15525 172.21.15.27:0/3290232959' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T13:03:20.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:20 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1722383910' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T13:03:20.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:20 smithi120 ceph-mon[43961]: osdmap e423: 12 total, 12 up, 11 in 2024-05-15T13:03:20.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:20 smithi177 ceph-mon[44303]: from='client.15525 172.21.15.27:0/3290232959' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T13:03:20.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:20 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1722383910' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T13:03:20.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:20 smithi177 ceph-mon[44303]: osdmap e423: 12 total, 12 up, 11 in 2024-05-15T13:03:20.739 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:03:20.849 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 6 1 2024-05-15T13:03:20.892 DEBUG:teuthology.orchestra.run.smithi177:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-05-15T13:03:20.895 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-05-15T13:03:20.895 INFO:teuthology.orchestra.run:waiting for 3000 2024-05-15T13:03:20.949 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:hints = 1 2024-05-15T13:03:20.950 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-05-15T13:03:20.950 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Object prefix: benchmark_data_smithi177_109991 2024-05-15T13:03:20.950 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:03:20.950 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 0 0 0 0 0 0 - 0 2024-05-15T13:03:21.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:21 smithi120 ceph-mon[43961]: pgmap v1133: 17 pgs: 17 active+clean; 577 KiB data, 5.0 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:21.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:21 smithi120 ceph-mon[43961]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T13:03:21.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:21 smithi027 ceph-mon[39795]: pgmap v1133: 17 pgs: 17 active+clean; 577 KiB data, 5.0 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:21.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:21 smithi027 ceph-mon[39795]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T13:03:21.386 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:03:21.387 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:03:21.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:21.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:03:21.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:21 smithi177 ceph-mon[44303]: pgmap v1133: 17 pgs: 17 active+clean; 577 KiB data, 5.0 GiB used, 978 GiB / 983 GiB avail 2024-05-15T13:03:21.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:21 smithi177 ceph-mon[44303]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T13:03:21.446 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:03:21.493 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:03:21.543 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:21.574 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:03:21.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:21.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:03:21.680 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:03:21.716 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:21.746 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:03:21.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:21.750 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:03:21.851 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:03:21.894 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:21.922 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:03:21.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:21.925 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:03:21.937 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:03:21.937 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:03:21.950 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 1 16 3886 3870 241.862 241.875 0.00298337 0.0041224 2024-05-15T13:03:22.027 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:03:22.068 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:22.096 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:03:22.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:22.102 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:03:22.203 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:03:22.242 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:22.274 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:03:22.278 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:22 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T13:03:22.278 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:22 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T13:03:22.278 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:22 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:03:22.278 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:22 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:03:22.278 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:22 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:03:22.278 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:22 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:03:22.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:22.279 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:03:22.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:22 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T13:03:22.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:22 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T13:03:22.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:22 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:03:22.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:22 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:03:22.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:22 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:03:22.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:22 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:03:22.380 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:03:22.426 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:22 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T13:03:22.426 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:22 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T13:03:22.426 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:22 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:03:22.426 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:22 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:03:22.426 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:22 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:03:22.426 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:22 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:03:22.431 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:22.720 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-05-15T13:03:22.885 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:03:22.916 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:03:22.937 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:03:22.950 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 2 16 7758 7742 241.916 242 0.00138593 0.00404147 2024-05-15T13:03:22.958 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:03:22.979 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:03:23.000 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:03:23.021 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:03:23.041 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:03:23.046 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:03:23.061 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:03:23.081 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:03:23.107 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:03:23.150 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:03:23.156 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:03:23.156 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:03:23.156 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:03:23.156 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:03:23.156 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:03:23.156 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:03:23.156 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:03:23.156 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:03:23.157 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:03:23.157 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:03:23.157 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:03:23.157 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:03:23.188 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:23 smithi027 ceph-mon[39795]: pgmap v1134: 17 pgs: 17 active+clean; 19 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 3.1 MiB/s wr, 49 op/s 2024-05-15T13:03:23.188 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:23 smithi027 ceph-mon[39795]: osdmap e424: 12 total, 12 up, 11 in 2024-05-15T13:03:23.189 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:23 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/978645333' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:03:23.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:23 smithi120 ceph-mon[43961]: pgmap v1134: 17 pgs: 17 active+clean; 19 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 3.1 MiB/s wr, 49 op/s 2024-05-15T13:03:23.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:23 smithi120 ceph-mon[43961]: osdmap e424: 12 total, 12 up, 11 in 2024-05-15T13:03:23.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:23 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/978645333' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:03:23.429 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:03:23.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:23 smithi177 ceph-mon[44303]: pgmap v1134: 17 pgs: 17 active+clean; 19 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 3.1 MiB/s wr, 49 op/s 2024-05-15T13:03:23.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:23 smithi177 ceph-mon[44303]: osdmap e424: 12 total, 12 up, 11 in 2024-05-15T13:03:23.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:23 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/978645333' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:03:23.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:23.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:03:23.537 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:03:23.593 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:23.950 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 3 16 10873 10857 226.162 194.688 0.00516319 0.00441593 2024-05-15T13:03:24.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:24 smithi177 ceph-mon[44303]: pgmap v1136: 17 pgs: 17 active+clean; 248 MiB data, 5.2 GiB used, 978 GiB / 983 GiB avail; 41 MiB/s wr, 658 op/s 2024-05-15T13:03:24.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:24 smithi027 ceph-mon[39795]: pgmap v1136: 17 pgs: 17 active+clean; 248 MiB data, 5.2 GiB used, 978 GiB / 983 GiB avail; 41 MiB/s wr, 658 op/s 2024-05-15T13:03:24.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:24 smithi120 ceph-mon[43961]: pgmap v1136: 17 pgs: 17 active+clean; 248 MiB data, 5.2 GiB used, 978 GiB / 983 GiB avail; 41 MiB/s wr, 658 op/s 2024-05-15T13:03:24.951 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 4 16 14463 14447 225.708 224.375 0.00144982 0.00438116 2024-05-15T13:03:25.951 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 5 16 16443 16427 205.314 123.75 0.00159091 0.00479667 2024-05-15T13:03:26.951 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 6 16 18711 18695 194.718 141.75 0.00598112 0.00513076 2024-05-15T13:03:27.032 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:03:27.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:27.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:03:27.139 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:03:27.170 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:27.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:27 smithi027 ceph-mon[39795]: pgmap v1137: 17 pgs: 17 active+clean; 504 MiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-05-15T13:03:27.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:27 smithi120 ceph-mon[43961]: pgmap v1137: 17 pgs: 17 active+clean; 504 MiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-05-15T13:03:27.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:27 smithi177 ceph-mon[44303]: pgmap v1137: 17 pgs: 17 active+clean; 504 MiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-05-15T13:03:27.951 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 7 16 20935 20919 186.757 139 0.00607807 0.00534951 2024-05-15T13:03:28.030 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:03:28.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:28.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:03:28.134 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:03:28.161 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:28.179 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:03:28.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:28.182 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:03:28.283 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:03:28.328 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:28.371 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:03:28.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:28.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:03:28.474 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:03:28.515 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:28.539 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:03:28.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:28.542 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:03:28.643 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:03:28.684 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:28.714 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:03:28.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:28.717 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:03:28.818 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:03:28.852 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:28.880 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:03:28.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:28.883 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:03:28.951 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 8 15 23290 23275 181.817 147.25 0.00501334 0.00549514 2024-05-15T13:03:28.985 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:03:29.024 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:29.044 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:03:29.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:29.047 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:03:29.148 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:03:29.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:29 smithi177 ceph-mon[44303]: pgmap v1138: 17 pgs: 17 active+clean; 1015 MiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-05-15T13:03:29.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:29 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:03:29.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:29 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:03:29.197 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:29.233 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:03:29.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:29.237 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:03:29.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:29 smithi027 ceph-mon[39795]: pgmap v1138: 17 pgs: 17 active+clean; 1015 MiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-05-15T13:03:29.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:29 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:03:29.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:29 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:03:29.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:29 smithi120 ceph-mon[43961]: pgmap v1138: 17 pgs: 17 active+clean; 1015 MiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-05-15T13:03:29.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:29 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:03:29.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:29 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:03:29.337 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:03:29.382 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:29.412 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:03:29.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:29.415 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:03:29.516 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:03:29.566 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:29.598 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:03:29.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:29.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:03:29.704 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:03:29.727 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:03:29.751 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:29.778 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:03:29.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:29.785 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:03:29.886 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:03:29.951 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 9 15 25981 25966 180.301 168.188 0.00530981 0.00554146 2024-05-15T13:03:30.004 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:30.116 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.6 to 1 (10000) 2024-05-15T13:03:30.122 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:03:30.196 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:03:30.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:30 smithi120 ceph-mon[43961]: osdmap e425: 12 total, 12 up, 11 in 2024-05-15T13:03:30.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:30 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1197452727' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:03:30.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:30 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1728213180' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T13:03:30.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:30 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T13:03:30.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:30 smithi120 ceph-mon[43961]: pgmap v1140: 17 pgs: 17 active+clean; 1.3 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 159 MiB/s wr, 2.55k op/s 2024-05-15T13:03:30.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:30 smithi177 ceph-mon[44303]: osdmap e425: 12 total, 12 up, 11 in 2024-05-15T13:03:30.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:30 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1197452727' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:03:30.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:30 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1728213180' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T13:03:30.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:30 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T13:03:30.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:30 smithi177 ceph-mon[44303]: pgmap v1140: 17 pgs: 17 active+clean; 1.3 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 159 MiB/s wr, 2.55k op/s 2024-05-15T13:03:30.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:30 smithi027 ceph-mon[39795]: osdmap e425: 12 total, 12 up, 11 in 2024-05-15T13:03:30.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:30 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1197452727' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:03:30.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:30 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1728213180' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T13:03:30.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:30 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T13:03:30.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:30 smithi027 ceph-mon[39795]: pgmap v1140: 17 pgs: 17 active+clean; 1.3 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 159 MiB/s wr, 2.55k op/s 2024-05-15T13:03:30.951 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 10 16 28602 28586 178.645 163.75 0.00467136 0.0055934 2024-05-15T13:03:31.136 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:03:31.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:31.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:03:31.243 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:03:31.269 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:31.298 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:03:31.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:31.303 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:03:31.404 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:03:31.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:31 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:03:31.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:31 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T13:03:31.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:31 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1197452727' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:03:31.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:31 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T13:03:31.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:31 smithi177 ceph-mon[44303]: osdmap e426: 12 total, 12 up, 11 in 2024-05-15T13:03:31.442 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:31 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:03:31.442 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:31 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T13:03:31.442 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:31 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1197452727' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:03:31.442 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:31 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T13:03:31.442 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:31 smithi120 ceph-mon[43961]: osdmap e426: 12 total, 12 up, 11 in 2024-05-15T13:03:31.447 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:31.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:31 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:03:31.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:31 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T13:03:31.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:31 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1197452727' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:03:31.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:31 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T13:03:31.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:31 smithi027 ceph-mon[39795]: osdmap e426: 12 total, 12 up, 11 in 2024-05-15T13:03:31.951 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 11 16 31063 31047 176.386 153.812 0.00697262 0.00566452 2024-05-15T13:03:32.383 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:32 smithi027 ceph-mon[39795]: pgmap v1142: 17 pgs: 17 active+clean; 1.5 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-05-15T13:03:32.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:32 smithi177 ceph-mon[44303]: pgmap v1142: 17 pgs: 17 active+clean; 1.5 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-05-15T13:03:32.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:32 smithi120 ceph-mon[43961]: pgmap v1142: 17 pgs: 17 active+clean; 1.5 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-05-15T13:03:32.951 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 12 15 33465 33450 174.202 150.188 0.00259713 0.00573594 2024-05-15T13:03:33.209 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:03:33.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:33.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:03:33.313 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:03:33.342 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:33.698 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:03:33.699 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T13:03:33.699 INFO:teuthology.orchestra.run.smithi120.stdout: "complaint_time": 30, 2024-05-15T13:03:33.699 INFO:teuthology.orchestra.run.smithi120.stdout: "num_blocked_ops": 0 2024-05-15T13:03:33.699 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:03:33.951 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 13 16 35783 35767 171.94 144.812 0.0072655 0.00581158 2024-05-15T13:03:34.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:34 smithi177 ceph-mon[44303]: pgmap v1143: 17 pgs: 17 active+clean; 1.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 171 MiB/s wr, 2.73k op/s 2024-05-15T13:03:34.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:34 smithi027 ceph-mon[39795]: pgmap v1143: 17 pgs: 17 active+clean; 1.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 171 MiB/s wr, 2.73k op/s 2024-05-15T13:03:34.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:34 smithi120 ceph-mon[43961]: pgmap v1143: 17 pgs: 17 active+clean; 1.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 171 MiB/s wr, 2.73k op/s 2024-05-15T13:03:34.951 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 14 16 37578 37562 167.672 112.188 0.0110926 0.00595894 2024-05-15T13:03:34.999 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:03:35.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:35.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:03:35.112 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:03:35.159 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:35.493 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 8 1 2024-05-15T13:03:35.951 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 15 16 39489 39473 164.456 119.438 0.00209887 0.00600632 2024-05-15T13:03:36.023 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:03:36.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:36.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:03:36.034 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:03:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:03:36.084 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:03:36.119 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:03:36.128 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:03:36.167 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:36.197 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:03:36.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:36.201 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:03:36.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:36 smithi120 ceph-mon[43961]: pgmap v1144: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 136 MiB/s wr, 2.17k op/s 2024-05-15T13:03:36.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:36 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1718944210' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:03:36.302 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:03:36.322 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:36 smithi027 ceph-mon[39795]: pgmap v1144: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 136 MiB/s wr, 2.17k op/s 2024-05-15T13:03:36.322 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:36 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1718944210' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:03:36.367 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:36.369 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:03:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:03:36.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:36.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:03:36.399 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:03:36.423 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:03:36.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:36 smithi177 ceph-mon[44303]: pgmap v1144: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 136 MiB/s wr, 2.17k op/s 2024-05-15T13:03:36.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:36 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1718944210' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:03:36.448 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:03:36.474 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:03:36.496 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:03:36.498 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:03:36.500 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-05-15T13:03:36.517 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:03:36.549 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:03:36.585 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:03:36.587 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:36.589 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:03:36.589 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:03:36.590 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:03:36.590 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:03:36.590 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:03:36.590 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:03:36.590 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:03:36.590 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:03:36.590 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:03:36.590 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:03:36.590 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:03:36.590 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:03:36.615 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:03:36.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:36.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:03:36.721 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:03:36.769 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:36.802 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:03:36.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:36.807 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:03:36.908 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:03:36.914 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:03:36.952 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 16 16 41367 41351 161.513 117.375 0.00662152 0.00618688 2024-05-15T13:03:36.955 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:36.986 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:03:36.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:36.992 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:03:37.093 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:03:37.136 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:37.149 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:03:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:03:37.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:37.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:03:37.604 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:03:37.649 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:37.952 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 17 16 43322 43306 159.199 122.188 0.00311247 0.00627425 2024-05-15T13:03:38.617 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:03:38.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:38.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:03:38.724 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:03:38.766 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:38.802 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:03:38.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:38.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:03:38.880 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:38 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:03:38.880 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:38 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:03:38.880 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:38 smithi027 ceph-mon[39795]: osdmap e427: 12 total, 12 up, 11 in 2024-05-15T13:03:38.880 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:38 smithi027 ceph-mon[39795]: pgmap v1146: 17 pgs: 17 active+clean; 2.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-05-15T13:03:38.880 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:38 smithi027 ceph-mon[39795]: osdmap e428: 12 total, 12 up, 11 in 2024-05-15T13:03:38.914 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:03:38.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:38 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:03:38.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:38 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:03:38.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:38 smithi177 ceph-mon[44303]: osdmap e427: 12 total, 12 up, 11 in 2024-05-15T13:03:38.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:38 smithi177 ceph-mon[44303]: pgmap v1146: 17 pgs: 17 active+clean; 2.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-05-15T13:03:38.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:38 smithi177 ceph-mon[44303]: osdmap e428: 12 total, 12 up, 11 in 2024-05-15T13:03:38.949 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:38.951 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 18 16 45178 45162 156.798 116 0.00271571 0.00636281 2024-05-15T13:03:38.969 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:03:38.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:38.972 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:03:39.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:38 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:03:39.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:38 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:03:39.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:38 smithi120 ceph-mon[43961]: osdmap e427: 12 total, 12 up, 11 in 2024-05-15T13:03:39.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:38 smithi120 ceph-mon[43961]: pgmap v1146: 17 pgs: 17 active+clean; 2.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-05-15T13:03:39.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:38 smithi120 ceph-mon[43961]: osdmap e428: 12 total, 12 up, 11 in 2024-05-15T13:03:39.072 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:03:39.098 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:39.129 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:03:39.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:39.134 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:03:39.235 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:03:39.285 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:39.315 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:03:39.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:39.321 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:03:39.422 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:03:39.462 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:39.499 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:03:39.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:39.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:03:39.605 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:03:39.648 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:39.736 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:03:39.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:39.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:03:39.841 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:03:39.894 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:39.954 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 19 16 46997 46981 154.513 113.688 0.00802406 0.00646828 2024-05-15T13:03:40.053 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:03:40.053 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:03:40.458 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:03:40.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:40.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:03:40.564 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:03:40.609 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:40.637 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:03:40.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:40.640 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:03:40.691 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:40 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:03:40.691 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:40 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:03:40.691 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:40 smithi027 ceph-mon[39795]: osdmap e429: 12 total, 12 up, 11 in 2024-05-15T13:03:40.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:40 smithi027 ceph-mon[39795]: pgmap v1149: 17 pgs: 17 active+clean; 2.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 136 MiB/s wr, 2.18k op/s 2024-05-15T13:03:40.701 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:03:40.741 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:03:40.791 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:40.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:40 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:03:40.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:40 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:03:40.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:40 smithi177 ceph-mon[44303]: osdmap e429: 12 total, 12 up, 11 in 2024-05-15T13:03:40.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:40 smithi177 ceph-mon[44303]: pgmap v1149: 17 pgs: 17 active+clean; 2.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 136 MiB/s wr, 2.18k op/s 2024-05-15T13:03:40.954 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T13:03:40.965407+0000 min lat: 0.00112512 max lat: 0.211403 avg lat: 0.00652453 2024-05-15T13:03:40.954 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:03:40.954 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 20 16 48883 48867 152.681 117.875 0.00174037 0.00652453 2024-05-15T13:03:41.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:40 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:03:41.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:40 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:03:41.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:40 smithi120 ceph-mon[43961]: osdmap e429: 12 total, 12 up, 11 in 2024-05-15T13:03:41.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:40 smithi120 ceph-mon[43961]: pgmap v1149: 17 pgs: 17 active+clean; 2.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 136 MiB/s wr, 2.18k op/s 2024-05-15T13:03:41.348 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:03:41.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:41.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:03:41.456 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:03:41.510 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:41.645 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:41 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:03:41.645 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:41 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:03:41.745 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:41 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:03:41.746 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:41 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:03:41.787 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-05-15T13:03:41.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:41 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:03:41.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:41 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:03:41.954 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 21 16 50623 50607 150.589 108.75 0.00427246 0.0066367 2024-05-15T13:03:42.167 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:03:42.486 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:03:42.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:42.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:03:42.591 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:03:42.614 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.8 to 1 (10000) 2024-05-15T13:03:42.648 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:42.739 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:03:42.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:42 smithi027 ceph-mon[39795]: osdmap e430: 12 total, 12 up, 11 in 2024-05-15T13:03:42.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:42 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/678069934' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T13:03:42.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:42 smithi027 ceph-mon[39795]: pgmap v1151: 17 pgs: 17 active+clean; 2.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-05-15T13:03:42.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:42 smithi120 ceph-mon[43961]: osdmap e430: 12 total, 12 up, 11 in 2024-05-15T13:03:42.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:42 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/678069934' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T13:03:42.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:42 smithi120 ceph-mon[43961]: pgmap v1151: 17 pgs: 17 active+clean; 2.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-05-15T13:03:42.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:42 smithi177 ceph-mon[44303]: osdmap e430: 12 total, 12 up, 11 in 2024-05-15T13:03:42.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:42 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/678069934' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T13:03:42.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:42 smithi177 ceph-mon[44303]: pgmap v1151: 17 pgs: 17 active+clean; 2.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-05-15T13:03:42.954 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 22 16 52475 52459 149.005 115.75 0.0157931 0.00670668 2024-05-15T13:03:43.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:43 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/678069934' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T13:03:43.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:43 smithi027 ceph-mon[39795]: osdmap e431: 12 total, 12 up, 11 in 2024-05-15T13:03:43.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:43 smithi027 ceph-mon[39795]: osdmap e432: 12 total, 12 up, 11 in 2024-05-15T13:03:43.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:43 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/678069934' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T13:03:43.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:43 smithi177 ceph-mon[44303]: osdmap e431: 12 total, 12 up, 11 in 2024-05-15T13:03:43.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:43 smithi177 ceph-mon[44303]: osdmap e432: 12 total, 12 up, 11 in 2024-05-15T13:03:43.955 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 23 16 54466 54450 147.93 124.438 0.0118195 0.00675537 2024-05-15T13:03:44.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:43 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/678069934' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T13:03:44.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:43 smithi120 ceph-mon[43961]: osdmap e431: 12 total, 12 up, 11 in 2024-05-15T13:03:44.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:43 smithi120 ceph-mon[43961]: osdmap e432: 12 total, 12 up, 11 in 2024-05-15T13:03:44.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:44 smithi120 ceph-mon[43961]: pgmap v1154: 17 pgs: 17 active+clean; 3.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-05-15T13:03:44.924 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:03:44.924 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:03:44.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:44 smithi177 ceph-mon[44303]: pgmap v1154: 17 pgs: 17 active+clean; 3.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-05-15T13:03:44.955 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 24 16 55891 55875 145.477 89.0625 0.00235507 0.00684974 2024-05-15T13:03:45.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:44 smithi027 ceph-mon[39795]: pgmap v1154: 17 pgs: 17 active+clean; 3.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-05-15T13:03:45.428 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:03:45.431 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 9 1 2024-05-15T13:03:45.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:45.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:03:45.434 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:03:45.533 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:03:45.574 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:45.647 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:03:45.657 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:45 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4108229588' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:03:45.657 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:45 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:03:45.658 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:45 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:03:45.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:45 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4108229588' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:03:45.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:45 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:03:45.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:45 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:03:45.955 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 25 15 57054 57039 142.568 72.75 0.00261956 0.00697949 2024-05-15T13:03:46.025 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:03:46.028 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:45 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4108229588' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:03:46.028 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:45 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:03:46.028 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:45 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:03:46.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:46.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:03:46.130 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:03:46.177 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:46.647 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-05-15T13:03:46.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:46 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:03:46.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:46 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4108229588' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:03:46.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:46 smithi027 ceph-mon[39795]: osdmap e433: 12 total, 12 up, 11 in 2024-05-15T13:03:46.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:46 smithi027 ceph-mon[39795]: pgmap v1156: 17 pgs: 17 active+clean; 3.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 107 MiB/s wr, 1.72k op/s 2024-05-15T13:03:46.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:46 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:03:46.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:46 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4108229588' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:03:46.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:46 smithi177 ceph-mon[44303]: osdmap e433: 12 total, 12 up, 11 in 2024-05-15T13:03:46.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:46 smithi177 ceph-mon[44303]: pgmap v1156: 17 pgs: 17 active+clean; 3.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 107 MiB/s wr, 1.72k op/s 2024-05-15T13:03:46.955 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 26 15 58866 58851 141.44 113.25 0.00894779 0.00706584 2024-05-15T13:03:47.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:46 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:03:47.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:46 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4108229588' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:03:47.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:46 smithi120 ceph-mon[43961]: osdmap e433: 12 total, 12 up, 11 in 2024-05-15T13:03:47.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:46 smithi120 ceph-mon[43961]: pgmap v1156: 17 pgs: 17 active+clean; 3.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 107 MiB/s wr, 1.72k op/s 2024-05-15T13:03:47.089 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:03:47.346 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:03:47.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:47.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:03:47.450 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:03:47.486 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:47.880 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:03:47.955 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 27 16 60644 60628 140.314 111.062 0.0205741 0.00712299 2024-05-15T13:03:48.033 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:03:48.078 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:03:48.106 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:03:48.134 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:03:48.150 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:03:48.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:48.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:03:48.169 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:03:48.223 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:03:48.255 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:03:48.258 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:03:48.281 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:03:48.295 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:48.311 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:03:48.334 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:03:48.343 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:03:48.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:48.345 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:48 smithi120 ceph-mon[43961]: pgmap v1157: 17 pgs: 17 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-05-15T13:03:48.345 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:48 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2429059713' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:03:48.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:03:48.380 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:03:48.413 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:03:48.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:48 smithi177 ceph-mon[44303]: pgmap v1157: 17 pgs: 17 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-05-15T13:03:48.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:48 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2429059713' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:03:48.429 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:03:48.429 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:03:48.429 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:03:48.430 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:03:48.430 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:03:48.430 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:03:48.430 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:03:48.430 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:03:48.430 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:03:48.430 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:03:48.430 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:03:48.430 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:03:48.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:48 smithi027 ceph-mon[39795]: pgmap v1157: 17 pgs: 17 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-05-15T13:03:48.443 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:48 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2429059713' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:03:48.446 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:03:48.501 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:48.956 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 28 16 62365 62349 139.145 107.562 0.00742667 0.00718281 2024-05-15T13:03:49.376 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:03:49.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:49 smithi120 ceph-mon[43961]: osdmap e434: 12 total, 12 up, 11 in 2024-05-15T13:03:49.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:49 smithi177 ceph-mon[44303]: osdmap e434: 12 total, 12 up, 11 in 2024-05-15T13:03:49.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:49 smithi027 ceph-mon[39795]: osdmap e434: 12 total, 12 up, 11 in 2024-05-15T13:03:49.956 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 29 16 64174 64158 138.245 113.062 0.00378809 0.00723013 2024-05-15T13:03:50.068 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:03:50.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:50.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:03:50.175 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:03:50.226 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:50.338 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:03:50.343 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:50 smithi027 ceph-mon[39795]: pgmap v1159: 17 pgs: 17 active+clean; 3.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-05-15T13:03:50.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:50.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:03:50.446 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:03:50.491 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:50.521 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:03:50.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:50.525 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:03:50.552 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:03:50.552 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T13:03:50.552 INFO:teuthology.orchestra.run.smithi120.stdout: "complaint_time": 30, 2024-05-15T13:03:50.552 INFO:teuthology.orchestra.run.smithi120.stdout: "num_blocked_ops": 0 2024-05-15T13:03:50.553 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:03:50.580 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:50 smithi120 ceph-mon[43961]: pgmap v1159: 17 pgs: 17 active+clean; 3.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-05-15T13:03:50.625 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:03:50.670 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:50.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:50 smithi177 ceph-mon[44303]: pgmap v1159: 17 pgs: 17 active+clean; 3.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-05-15T13:03:50.956 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 30 16 65794 65778 137.012 101.25 0.00756871 0.00729505 2024-05-15T13:03:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:03:51.958 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 31 16 67257 67241 135.533 91.4375 0.00909409 0.00737288 2024-05-15T13:03:52.235 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:03:52.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:52.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:03:52.297 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-05-15T13:03:52.342 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:03:52.395 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:52.428 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:03:52.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:52.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:03:52.534 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:03:52.575 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:52.654 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:03:52.958 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 32 16 68724 68708 134.163 91.6875 0.00780186 0.00744933 2024-05-15T13:03:53.019 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:53 smithi027 ceph-mon[39795]: pgmap v1160: 17 pgs: 17 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-05-15T13:03:53.019 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:53 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/76337836' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:03:53.019 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:53 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:03:53.026 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.9 to 1 (10000) 2024-05-15T13:03:53.278 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:03:53.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:53 smithi120 ceph-mon[43961]: pgmap v1160: 17 pgs: 17 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-05-15T13:03:53.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:53 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/76337836' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:03:53.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:53 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:03:53.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:53 smithi177 ceph-mon[44303]: pgmap v1160: 17 pgs: 17 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-05-15T13:03:53.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:53 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/76337836' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:03:53.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:53 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:03:53.434 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:03:53.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:53.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:03:53.544 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:03:53.590 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:53.665 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:03:53.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:53.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:03:53.769 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:03:53.813 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:53.843 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:03:53.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:53.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:03:53.953 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:03:53.958 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 33 15 70102 70087 132.709 86.1875 0.00885692 0.00753124 2024-05-15T13:03:53.997 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:54.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:54 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T13:03:54.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:54 smithi027 ceph-mon[39795]: osdmap e435: 12 total, 12 up, 11 in 2024-05-15T13:03:54.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:54 smithi027 ceph-mon[39795]: osdmap e436: 12 total, 12 up, 11 in 2024-05-15T13:03:54.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:54 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T13:03:54.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:54 smithi120 ceph-mon[43961]: osdmap e435: 12 total, 12 up, 11 in 2024-05-15T13:03:54.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:54 smithi120 ceph-mon[43961]: osdmap e436: 12 total, 12 up, 11 in 2024-05-15T13:03:54.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:54 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T13:03:54.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:54 smithi177 ceph-mon[44303]: osdmap e435: 12 total, 12 up, 11 in 2024-05-15T13:03:54.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:54 smithi177 ceph-mon[44303]: osdmap e436: 12 total, 12 up, 11 in 2024-05-15T13:03:54.602 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:03:54.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:54.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:03:54.633 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 10 1 2024-05-15T13:03:54.711 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:03:54.754 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:54.961 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 34 16 71742 71726 131.806 102.438 0.00893039 0.00758318 2024-05-15T13:03:55.182 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:03:55.219 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:03:55.318 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:03:55.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:55.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:03:55.322 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:55 smithi120 ceph-mon[43961]: pgmap v1163: 17 pgs: 17 active+clean; 4.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-05-15T13:03:55.423 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:03:55.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:55 smithi177 ceph-mon[44303]: pgmap v1163: 17 pgs: 17 active+clean; 4.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-05-15T13:03:55.437 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:03:55.437 INFO:teuthology.orchestra.run.smithi120.stdout: "size": 20, 2024-05-15T13:03:55.437 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 600, 2024-05-15T13:03:55.437 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [] 2024-05-15T13:03:55.437 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:03:55.457 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:55.457 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:55 smithi027 ceph-mon[39795]: pgmap v1163: 17 pgs: 17 active+clean; 4.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-05-15T13:03:55.499 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:03:55.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:55.505 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:03:55.606 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:03:55.681 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:55.815 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:03:55.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:55.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:03:55.924 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:03:55.961 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 35 16 73169 73153 130.588 89.1875 0.00621465 0.00765287 2024-05-15T13:03:55.965 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:56.075 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:03:56.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:56.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:03:56.180 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:03:56.216 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:56.246 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:03:56.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:56.251 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:03:56.352 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:03:56.393 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:56.422 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:03:56.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:56.427 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:03:56.528 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:03:56.559 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:56.740 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-05-15T13:03:56.961 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 36 16 74696 74680 129.612 95.4375 0.00361504 0.00771137 2024-05-15T13:03:57.050 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:57 smithi177 ceph-mon[44303]: pgmap v1164: 17 pgs: 17 active+clean; 4.2 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-05-15T13:03:57.177 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:03:57.415 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:03:57.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:57.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:03:57.421 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:57 smithi027 ceph-mon[39795]: pgmap v1164: 17 pgs: 17 active+clean; 4.2 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-05-15T13:03:57.522 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:03:57.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:57 smithi120 ceph-mon[43961]: pgmap v1164: 17 pgs: 17 active+clean; 4.2 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-05-15T13:03:57.573 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:57.962 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 37 16 76206 76190 128.659 94.375 0.0049605 0.00776862 2024-05-15T13:03:58.936 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:03:58.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:58.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:03:58.962 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 38 15 77696 77681 127.725 93.1875 0.00728773 0.00782532 2024-05-15T13:03:59.043 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:03:59.095 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:59.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:03:59 smithi177 ceph-mon[44303]: pgmap v1165: 17 pgs: 17 active+clean; 4.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-05-15T13:03:59.227 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:03:59.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:59.231 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:03:59.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:03:59 smithi027 ceph-mon[39795]: pgmap v1165: 17 pgs: 17 active+clean; 4.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-05-15T13:03:59.278 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:03:59.278 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T13:03:59.278 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-05-15T13:03:59.278 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:03:59.332 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:03:59.372 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:59.528 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:03:59.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:03:59 smithi120 ceph-mon[43961]: pgmap v1165: 17 pgs: 17 active+clean; 4.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-05-15T13:03:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:03:59.532 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:03:59.633 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:03:59.676 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:03:59.962 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 39 16 78977 78961 126.501 80 0.00322245 0.00787736 2024-05-15T13:04:00.006 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-05-15T13:04:00.086 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:04:00.400 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:04:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:00 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2130774706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:04:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:00 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/640422304' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:04:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:00 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:04:00.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:00 smithi177 ceph-mon[44303]: pgmap v1166: 17 pgs: 17 active+clean; 4.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-05-15T13:04:00.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:00 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2130774706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:04:00.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:00 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/640422304' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:04:00.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:00 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:04:00.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:00 smithi027 ceph-mon[39795]: pgmap v1166: 17 pgs: 17 active+clean; 4.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-05-15T13:04:00.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:00 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2130774706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:04:00.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:00 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/640422304' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:04:00.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:00 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:04:00.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:00 smithi120 ceph-mon[43961]: pgmap v1166: 17 pgs: 17 active+clean; 4.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-05-15T13:04:00.636 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:04:00.636 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:04:00.636 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:04:00.636 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:04:00.636 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:04:00.636 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:04:00.637 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:04:00.637 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:04:00.637 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:04:00.637 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:04:00.637 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:04:00.637 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:04:00.637 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:04:00.637 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:04:00.637 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:04:00.637 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:04:00.637 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:04:00.637 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:04:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:04:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:04:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:04:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:04:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:04:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:04:00.848 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:04:00.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:00.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:04:00.955 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:04:00.962 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T13:04:00.973810+0000 min lat: 0.00112512 max lat: 0.21806 avg lat: 0.00798658 2024-05-15T13:04:00.962 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:04:00.962 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 40 16 80132 80116 125.143 72.1875 0.00700342 0.00798658 2024-05-15T13:04:01.016 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:01.071 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:01 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:04:01.071 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:01 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:04:01.071 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:01 smithi177 ceph-mon[44303]: osdmap e437: 12 total, 12 up, 11 in 2024-05-15T13:04:01.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:01 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:04:01.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:01 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:04:01.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:01 smithi027 ceph-mon[39795]: osdmap e437: 12 total, 12 up, 11 in 2024-05-15T13:04:01.377 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:04:01.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:01.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:04:01.482 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:04:01.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:01 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:04:01.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:01 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:04:01.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:01 smithi120 ceph-mon[43961]: osdmap e437: 12 total, 12 up, 11 in 2024-05-15T13:04:01.533 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:01.675 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:04:01.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:01.680 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:04:01.781 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:04:01.818 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:01.962 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 41 16 81621 81605 124.361 93.0625 0.00859305 0.00803704 2024-05-15T13:04:02.094 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:02 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:04:02.094 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:02 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:04:02.094 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:02 smithi120 ceph-mon[43961]: osdmap e438: 12 total, 12 up, 11 in 2024-05-15T13:04:02.094 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:02 smithi120 ceph-mon[43961]: pgmap v1169: 17 pgs: 17 active+clean; 4.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-05-15T13:04:02.132 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:04:02.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:02.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:04:02.297 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:04:02.331 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:02.357 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:04:02.357 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:04:02.375 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:04:02.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:02.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:04:02.390 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:02 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:04:02.390 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:02 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:04:02.391 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:02 smithi177 ceph-mon[44303]: osdmap e438: 12 total, 12 up, 11 in 2024-05-15T13:04:02.391 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:02 smithi177 ceph-mon[44303]: pgmap v1169: 17 pgs: 17 active+clean; 4.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-05-15T13:04:02.481 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:04:02.520 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:02.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:02 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:04:02.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:02 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:04:02.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:02 smithi027 ceph-mon[39795]: osdmap e438: 12 total, 12 up, 11 in 2024-05-15T13:04:02.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:02 smithi027 ceph-mon[39795]: pgmap v1169: 17 pgs: 17 active+clean; 4.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-05-15T13:04:02.553 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:04:02.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:02.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:04:02.659 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:04:02.720 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:02.966 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 42 16 83047 83031 123.512 89.125 0.0090722 0.00809248 2024-05-15T13:04:03.044 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:04:03.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:03.050 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:04:03.096 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-05-15T13:04:03.106 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.10 to 1 (10000) 2024-05-15T13:04:03.151 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:04:03.216 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:03.328 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:04:03.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:03.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:04:03.417 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:04:03.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:03 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2954030350' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T13:04:03.436 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:04:03.442 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:03 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2954030350' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T13:04:03.470 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:03 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2954030350' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T13:04:03.474 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:03.506 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:04:03.513 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:04:03.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:03.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:04:03.620 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:04:03.684 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:03.830 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:04:03.848 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:04:03.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:03.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:04:03.907 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 11 1 2024-05-15T13:04:03.930 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:04:03.953 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:04:03.966 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 43 16 84381 84365 122.579 83.375 0.0106301 0.00815383 2024-05-15T13:04:04.044 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:04.110 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:04 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2954030350' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T13:04:04.111 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:04 smithi027 ceph-mon[39795]: osdmap e439: 12 total, 12 up, 11 in 2024-05-15T13:04:04.111 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:04 smithi027 ceph-mon[39795]: osdmap e440: 12 total, 12 up, 11 in 2024-05-15T13:04:04.111 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:04 smithi027 ceph-mon[39795]: pgmap v1172: 17 pgs: 17 active+clean; 4.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-05-15T13:04:04.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:04 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2954030350' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T13:04:04.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:04 smithi177 ceph-mon[44303]: osdmap e439: 12 total, 12 up, 11 in 2024-05-15T13:04:04.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:04 smithi177 ceph-mon[44303]: osdmap e440: 12 total, 12 up, 11 in 2024-05-15T13:04:04.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:04 smithi177 ceph-mon[44303]: pgmap v1172: 17 pgs: 17 active+clean; 4.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-05-15T13:04:04.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:04 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2954030350' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T13:04:04.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:04 smithi120 ceph-mon[43961]: osdmap e439: 12 total, 12 up, 11 in 2024-05-15T13:04:04.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:04 smithi120 ceph-mon[43961]: osdmap e440: 12 total, 12 up, 11 in 2024-05-15T13:04:04.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:04 smithi120 ceph-mon[43961]: pgmap v1172: 17 pgs: 17 active+clean; 4.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-05-15T13:04:04.902 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:04:04.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:04.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:04:04.966 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 44 16 85532 85516 121.428 71.9375 0.0228811 0.00823079 2024-05-15T13:04:05.008 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:04:05.056 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:05.091 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:04:05.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:05.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:04:05.196 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:04:05.247 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:05.380 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:04:05.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:05.386 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:04:05.487 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:04:05.543 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:05.572 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:04:05.572 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:04:05.909 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:04:05.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:05.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:04:05.967 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 45 16 86891 86875 120.617 84.9375 0.008846 0.00828726 2024-05-15T13:04:06.013 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:04:06.056 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:06.351 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-05-15T13:04:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:04:06.516 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:04:06.521 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:04:06.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:06.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:04:06.628 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:04:06.662 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:06.691 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:04:06.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:06.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:04:06.757 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:04:06.797 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:04:06.843 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:06.966 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 46 16 88211 88195 119.789 82.5 0.00740636 0.00834394 2024-05-15T13:04:07.010 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:07 smithi120 ceph-mon[43961]: pgmap v1173: 17 pgs: 17 active+clean; 5.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-05-15T13:04:07.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:07 smithi027 ceph-mon[39795]: pgmap v1173: 17 pgs: 17 active+clean; 5.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-05-15T13:04:07.331 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:07 smithi177 ceph-mon[44303]: pgmap v1173: 17 pgs: 17 active+clean; 5.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-05-15T13:04:07.909 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:04:07.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:07.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:04:07.966 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 47 16 89569 89553 119.046 84.875 0.00950208 0.00839618 2024-05-15T13:04:08.017 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:04:08.080 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:08.338 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:04:08.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:08.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:04:08.443 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:04:08.498 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:08.525 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:04:08.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:08.534 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:04:08.635 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:04:08.674 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:08.739 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:04:08.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:08.744 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:04:08.811 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:04:08.811 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:04:08.845 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:04:08.894 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:08.966 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 48 16 90841 90825 118.222 79.5 0.013539 0.0084543 2024-05-15T13:04:08.981 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:04:08.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:08.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:04:09.023 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:09 smithi177 ceph-mon[44303]: pgmap v1174: 17 pgs: 17 active+clean; 5.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 95 MiB/s wr, 1.53k op/s 2024-05-15T13:04:09.023 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:09 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3877904226' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:04:09.088 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:04:09.118 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:04:09.118 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:04:09.118 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:04:09.118 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:04:09.119 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:04:09.119 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:04:09.119 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:04:09.119 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:04:09.119 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:04:09.123 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:04:09.135 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:09.153 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:04:09.177 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:04:09.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:09.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:04:09.190 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:04:09.201 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:04:09.201 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:04:09.201 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:04:09.201 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:04:09.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:04:09.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:04:09.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:04:09.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:04:09.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:04:09.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:04:09.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:04:09.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:04:09.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:09 smithi027 ceph-mon[39795]: pgmap v1174: 17 pgs: 17 active+clean; 5.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 95 MiB/s wr, 1.53k op/s 2024-05-15T13:04:09.276 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:09 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3877904226' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:04:09.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:09 smithi120 ceph-mon[43961]: pgmap v1174: 17 pgs: 17 active+clean; 5.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 95 MiB/s wr, 1.53k op/s 2024-05-15T13:04:09.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:09 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3877904226' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:04:09.284 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:04:09.327 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:09.361 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:04:09.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:09.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:04:09.465 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:04:09.522 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:09.574 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:04:09.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:09.578 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:04:09.584 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-05-15T13:04:09.679 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:04:09.718 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:09.744 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:04:09.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:09.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:04:09.850 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:04:09.886 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:09.914 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:04:09.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:09.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:04:09.967 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 49 16 92188 92172 117.526 84.1875 0.0126993 0.00850533 2024-05-15T13:04:10.018 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:04:10.049 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:04:10.054 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:10.093 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:04:10.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:10.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:04:10.197 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:04:10.249 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:10.334 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:04:10.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:10.339 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:04:10.440 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:04:10.490 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:10.515 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:04:10.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:10.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:04:10.623 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:04:10.670 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:10.967 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 50 16 93468 93452 116.775 80 0.0102912 0.00855995 2024-05-15T13:04:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:04:11.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:11.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:04:11.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:11 smithi027 ceph-mon[39795]: pgmap v1175: 17 pgs: 17 active+clean; 5.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-05-15T13:04:11.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:11 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:04:11.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:11 smithi120 ceph-mon[43961]: pgmap v1175: 17 pgs: 17 active+clean; 5.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-05-15T13:04:11.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:11 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:04:11.349 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:04:11.366 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:11 smithi177 ceph-mon[44303]: pgmap v1175: 17 pgs: 17 active+clean; 5.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-05-15T13:04:11.366 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:11 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:04:11.394 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:11.421 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:04:11.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:11.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:04:11.528 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:04:11.559 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:11.585 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:04:11.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:11.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:04:11.692 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:04:11.735 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:11.771 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:04:11.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:11.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:04:11.879 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:04:11.930 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:11.967 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 51 16 94652 94636 115.936 74 0.0170499 0.00862134 2024-05-15T13:04:11.998 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:04:12.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:12.001 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:04:12.070 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:04:12.070 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T13:04:12.070 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-05-15T13:04:12.071 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-05-15T13:04:12.071 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:04:12.101 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:04:12.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:12 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:04:12.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:12 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:04:12.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:12 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:04:12.135 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:12.157 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:04:12.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:12.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:04:12.262 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:04:12.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:12 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:04:12.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:12 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:04:12.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:12 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:04:12.307 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:12.338 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:04:12.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:12.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:04:12.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:12 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:04:12.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:12 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:04:12.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:12 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:04:12.441 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:04:12.491 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:12.521 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:04:12.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:12.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:04:12.645 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:04:12.709 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:12.820 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:04:12.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:12.824 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:04:12.851 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-05-15T13:04:12.925 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:04:12.960 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:12.967 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 52 16 95878 95862 115.181 76.625 0.0128228 0.00867698 2024-05-15T13:04:12.984 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:04:12.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:12.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:04:13.090 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:04:13.146 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:13 smithi177 ceph-mon[44303]: pgmap v1176: 17 pgs: 17 active+clean; 5.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-05-15T13:04:13.146 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:13 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3546719208' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:04:13.151 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:13.191 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:04:13.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:13.204 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:04:13.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:13 smithi120 ceph-mon[43961]: pgmap v1176: 17 pgs: 17 active+clean; 5.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-05-15T13:04:13.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:13 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3546719208' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:04:13.306 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:04:13.307 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:04:13.349 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:13 smithi027 ceph-mon[39795]: pgmap v1176: 17 pgs: 17 active+clean; 5.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-05-15T13:04:13.349 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:13 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3546719208' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:04:13.352 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:13.504 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:04:13.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:13.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:04:13.608 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:04:13.639 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:13.665 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:04:13.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:13.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:04:13.672 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:04:13.771 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:04:13.830 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:13.967 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 53 16 97051 97035 114.391 73.3125 0.00685815 0.00873831 2024-05-15T13:04:13.968 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:04:13.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:13.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:04:14.076 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:04:14.142 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:14.500 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:04:14.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:14.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:04:14.604 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:04:14.647 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:14 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:04:14.648 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:14 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3546719208' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:04:14.648 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:14 smithi027 ceph-mon[39795]: osdmap e441: 12 total, 12 up, 11 in 2024-05-15T13:04:14.648 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:14 smithi027 ceph-mon[39795]: pgmap v1178: 17 pgs: 17 active+clean; 5.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-05-15T13:04:14.648 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:14 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1778070610' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T13:04:14.648 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:14 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T13:04:14.651 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:14.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:14 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:04:14.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:14 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3546719208' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:04:14.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:14 smithi177 ceph-mon[44303]: osdmap e441: 12 total, 12 up, 11 in 2024-05-15T13:04:14.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:14 smithi177 ceph-mon[44303]: pgmap v1178: 17 pgs: 17 active+clean; 5.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-05-15T13:04:14.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:14 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1778070610' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T13:04:14.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:14 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T13:04:14.684 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:04:14.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:14.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:04:14.698 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.11 to 1 (10000) 2024-05-15T13:04:14.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:14 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:04:14.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:14 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3546719208' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:04:14.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:14 smithi120 ceph-mon[43961]: osdmap e441: 12 total, 12 up, 11 in 2024-05-15T13:04:14.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:14 smithi120 ceph-mon[43961]: pgmap v1178: 17 pgs: 17 active+clean; 5.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-05-15T13:04:14.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:14 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1778070610' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T13:04:14.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:14 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T13:04:14.788 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:04:14.825 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:14.852 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:04:14.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:14.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:04:14.956 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:04:14.967 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 54 16 98360 98344 113.787 81.8125 0.00772542 0.00878467 2024-05-15T13:04:14.994 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:15.023 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:04:15.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:15.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:04:15.129 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:04:15.185 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:15.222 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:04:15.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:15.226 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:04:15.327 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:04:15.368 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:15.467 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-05-15T13:04:15.467 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:04:15.481 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:04:15.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:15.484 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:04:15.549 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:04:15.549 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-05-15T13:04:15.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-05-15T13:04:15.549 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T13:04:15.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.549 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24832.0:236805 3.0 3:0e55670e:::benchmark_data_smithi177_88104_object103183:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T13:04:15.549 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:56:47.809759+0000", 2024-05-15T13:04:15.549 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 447.74676493499999, 2024-05-15T13:04:15.550 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.212380018, 2024-05-15T13:04:15.550 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:15.550 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:15.550 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:15.550 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24832", 2024-05-15T13:04:15.550 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:04:15.550 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 236805 2024-05-15T13:04:15.550 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.550 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:15.550 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.550 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:15.550 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.809759+0000", 2024-05-15T13:04:15.550 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:15.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.809759+0000", 2024-05-15T13:04:15.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:15.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.809760+0000", 2024-05-15T13:04:15.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3939999999999999e-06 2024-05-15T13:04:15.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:15.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.809765+0000", 2024-05-15T13:04:15.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3889999999999998e-06 2024-05-15T13:04:15.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:15.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.809776+0000", 2024-05-15T13:04:15.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.172e-05 2024-05-15T13:04:15.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:15.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.809789+0000", 2024-05-15T13:04:15.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2519e-05 2024-05-15T13:04:15.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:15.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.809857+0000", 2024-05-15T13:04:15.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8149000000000001e-05 2024-05-15T13:04:15.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:15.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.809969+0000", 2024-05-15T13:04:15.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000112417 2024-05-15T13:04:15.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.554 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:04:15.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.810049+0000", 2024-05-15T13:04:15.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9956000000000005e-05 2024-05-15T13:04:15.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.554 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:15.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.817842+0000", 2024-05-15T13:04:15.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0077925809999999998 2024-05-15T13:04:15.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.555 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.555 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.834327+0000", 2024-05-15T13:04:15.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.016484664 2024-05-15T13:04:15.555 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.555 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.555 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.022020+0000", 2024-05-15T13:04:15.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18769355500000001 2024-05-15T13:04:15.555 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:15.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.022086+0000", 2024-05-15T13:04:15.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5604000000000004e-05 2024-05-15T13:04:15.556 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.556 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:15.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.022139+0000", 2024-05-15T13:04:15.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3069999999999998e-05 2024-05-15T13:04:15.556 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.556 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:15.556 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.557 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.557 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.557 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24832.0:236808 3.0 3:049069b2:::benchmark_data_smithi177_88104_object103186:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T13:04:15.557 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:56:47.809817+0000", 2024-05-15T13:04:15.557 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 447.746706813, 2024-05-15T13:04:15.557 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21953202599999999, 2024-05-15T13:04:15.557 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:15.557 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:15.557 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:15.557 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24832", 2024-05-15T13:04:15.557 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:04:15.558 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 236808 2024-05-15T13:04:15.558 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.558 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:15.558 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.558 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:15.558 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.809817+0000", 2024-05-15T13:04:15.558 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.558 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.558 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.559 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:15.559 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.809817+0000", 2024-05-15T13:04:15.559 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.559 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.559 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.559 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:15.559 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.809818+0000", 2024-05-15T13:04:15.559 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0399999999999996e-07 2024-05-15T13:04:15.559 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.559 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.559 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:15.560 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.809821+0000", 2024-05-15T13:04:15.560 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.36e-06 2024-05-15T13:04:15.560 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.560 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.560 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:15.560 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.809829+0000", 2024-05-15T13:04:15.560 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8990000000000001e-06 2024-05-15T13:04:15.560 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.560 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.560 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:15.560 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.809835+0000", 2024-05-15T13:04:15.561 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6719999999999998e-06 2024-05-15T13:04:15.561 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.561 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.561 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:15.561 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.813704+0000", 2024-05-15T13:04:15.561 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0038686749999999998 2024-05-15T13:04:15.561 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.561 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.561 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:15.562 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.813791+0000", 2024-05-15T13:04:15.562 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6793999999999996e-05 2024-05-15T13:04:15.562 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.562 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.562 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:04:15.562 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.813881+0000", 2024-05-15T13:04:15.562 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0299999999999999e-05 2024-05-15T13:04:15.562 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.562 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.562 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:15.562 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.822884+0000", 2024-05-15T13:04:15.563 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.009003212 2024-05-15T13:04:15.563 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.563 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.563 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.563 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.837295+0000", 2024-05-15T13:04:15.563 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014410665 2024-05-15T13:04:15.563 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.563 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.563 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.563 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.029274+0000", 2024-05-15T13:04:15.564 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.191979343 2024-05-15T13:04:15.564 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.564 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.564 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:15.564 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.029314+0000", 2024-05-15T13:04:15.564 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9753999999999999e-05 2024-05-15T13:04:15.564 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.564 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.564 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:15.564 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.029349+0000", 2024-05-15T13:04:15.565 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4748e-05 2024-05-15T13:04:15.565 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.565 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:15.565 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.565 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.565 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.565 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24832.0:236819 3.0 3:0b495621:::benchmark_data_smithi177_88104_object103197:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T13:04:15.565 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:56:47.826780+0000", 2024-05-15T13:04:15.565 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 447.72974352900002, 2024-05-15T13:04:15.565 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.207007896, 2024-05-15T13:04:15.565 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:15.565 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:15.565 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:15.565 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24832", 2024-05-15T13:04:15.566 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:04:15.566 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 236819 2024-05-15T13:04:15.566 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.566 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:15.566 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.566 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:15.566 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.826780+0000", 2024-05-15T13:04:15.566 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.566 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.566 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.566 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:15.566 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.826780+0000", 2024-05-15T13:04:15.566 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.566 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.567 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.567 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:15.567 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.826782+0000", 2024-05-15T13:04:15.567 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.353e-06 2024-05-15T13:04:15.567 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.567 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.567 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:15.567 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.826788+0000", 2024-05-15T13:04:15.567 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5420000000000002e-06 2024-05-15T13:04:15.567 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.567 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.567 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:15.567 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.826800+0000", 2024-05-15T13:04:15.567 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1836e-05 2024-05-15T13:04:15.568 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.568 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.568 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:15.568 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.826808+0000", 2024-05-15T13:04:15.568 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.605e-06 2024-05-15T13:04:15.568 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.568 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.568 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:15.568 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.826852+0000", 2024-05-15T13:04:15.568 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4653000000000001e-05 2024-05-15T13:04:15.568 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.568 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.568 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:15.568 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.826979+0000", 2024-05-15T13:04:15.569 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012649199999999999 2024-05-15T13:04:15.569 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.569 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.569 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:04:15.569 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.827058+0000", 2024-05-15T13:04:15.569 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9741999999999996e-05 2024-05-15T13:04:15.569 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.569 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.569 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:15.569 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.834344+0000", 2024-05-15T13:04:15.569 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0072856320000000002 2024-05-15T13:04:15.569 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.569 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.569 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.569 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.837331+0000", 2024-05-15T13:04:15.570 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0029874390000000002 2024-05-15T13:04:15.570 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.570 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.570 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.570 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.033689+0000", 2024-05-15T13:04:15.570 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19635738699999999 2024-05-15T13:04:15.570 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.570 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.570 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:15.570 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.033750+0000", 2024-05-15T13:04:15.570 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1484999999999998e-05 2024-05-15T13:04:15.570 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.570 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.570 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:15.571 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.033788+0000", 2024-05-15T13:04:15.571 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.773e-05 2024-05-15T13:04:15.571 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.571 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:15.571 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.571 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.571 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.571 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24832.0:236821 3.0 3:0e3a4f10:::benchmark_data_smithi177_88104_object103199:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T13:04:15.571 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:56:47.827030+0000", 2024-05-15T13:04:15.571 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 447.72949394199998, 2024-05-15T13:04:15.571 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.206909863, 2024-05-15T13:04:15.571 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:15.571 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:15.572 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:15.572 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24832", 2024-05-15T13:04:15.572 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:04:15.572 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 236821 2024-05-15T13:04:15.572 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.572 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:15.572 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.572 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:15.572 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.827030+0000", 2024-05-15T13:04:15.572 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.572 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.572 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.572 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:15.572 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.827030+0000", 2024-05-15T13:04:15.573 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.573 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.573 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.573 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:15.573 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.827030+0000", 2024-05-15T13:04:15.573 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3300000000000002e-07 2024-05-15T13:04:15.573 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.573 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.573 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:15.573 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.827033+0000", 2024-05-15T13:04:15.573 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9960000000000002e-06 2024-05-15T13:04:15.573 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.573 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.573 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:15.574 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.827039+0000", 2024-05-15T13:04:15.574 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1689999999999997e-06 2024-05-15T13:04:15.574 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.574 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.574 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:15.574 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.827044+0000", 2024-05-15T13:04:15.574 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.391e-06 2024-05-15T13:04:15.574 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.574 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.574 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:15.574 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.830716+0000", 2024-05-15T13:04:15.574 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0036718969999999999 2024-05-15T13:04:15.574 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.575 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.575 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:15.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.830806+0000", 2024-05-15T13:04:15.575 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0531000000000004e-05 2024-05-15T13:04:15.575 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.575 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.575 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:04:15.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.830896+0000", 2024-05-15T13:04:15.575 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9944999999999993e-05 2024-05-15T13:04:15.575 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.575 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.575 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:15.575 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.834450+0000", 2024-05-15T13:04:15.575 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035542479999999999 2024-05-15T13:04:15.575 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.576 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.576 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.852071+0000", 2024-05-15T13:04:15.576 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017620713999999999 2024-05-15T13:04:15.576 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.576 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.576 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.033835+0000", 2024-05-15T13:04:15.576 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18176414900000001 2024-05-15T13:04:15.576 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.576 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.576 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:15.576 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.033897+0000", 2024-05-15T13:04:15.576 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1577999999999999e-05 2024-05-15T13:04:15.576 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.577 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.577 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:15.577 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.033940+0000", 2024-05-15T13:04:15.577 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2712e-05 2024-05-15T13:04:15.577 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.577 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:15.577 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.577 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.577 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.577 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24832.0:236828 3.0 3:0d4bab3f:::benchmark_data_smithi177_88104_object103206:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T13:04:15.577 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:56:47.838804+0000", 2024-05-15T13:04:15.577 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 447.71771925000002, 2024-05-15T13:04:15.577 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.201278189, 2024-05-15T13:04:15.577 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:15.577 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:15.578 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:15.578 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24832", 2024-05-15T13:04:15.578 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:04:15.578 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 236828 2024-05-15T13:04:15.578 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.578 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:15.578 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.578 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:15.578 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.838804+0000", 2024-05-15T13:04:15.578 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.578 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.578 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.578 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:15.579 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.838804+0000", 2024-05-15T13:04:15.579 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.579 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.579 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.579 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:15.579 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.838806+0000", 2024-05-15T13:04:15.579 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.238e-06 2024-05-15T13:04:15.579 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.579 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.579 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:15.579 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.838812+0000", 2024-05-15T13:04:15.579 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5200000000000003e-06 2024-05-15T13:04:15.579 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.579 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.579 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:15.580 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.838825+0000", 2024-05-15T13:04:15.580 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2695e-05 2024-05-15T13:04:15.580 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.580 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.580 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:15.580 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.838833+0000", 2024-05-15T13:04:15.580 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7649999999999993e-06 2024-05-15T13:04:15.580 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.580 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.580 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:15.580 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.838875+0000", 2024-05-15T13:04:15.580 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2673000000000003e-05 2024-05-15T13:04:15.580 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.580 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.581 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:15.581 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.838965+0000", 2024-05-15T13:04:15.581 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9804999999999998e-05 2024-05-15T13:04:15.581 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.581 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.581 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:04:15.581 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.839042+0000", 2024-05-15T13:04:15.581 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7097999999999999e-05 2024-05-15T13:04:15.581 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.581 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.581 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:15.581 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.842407+0000", 2024-05-15T13:04:15.581 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0033650870000000001 2024-05-15T13:04:15.581 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.581 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.582 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.582 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.852104+0000", 2024-05-15T13:04:15.582 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0096970719999999993 2024-05-15T13:04:15.582 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.582 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.582 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.582 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.039989+0000", 2024-05-15T13:04:15.582 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18788474999999999 2024-05-15T13:04:15.582 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.582 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.582 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:15.583 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.040041+0000", 2024-05-15T13:04:15.583 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2111000000000003e-05 2024-05-15T13:04:15.583 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.583 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.583 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:15.583 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.040083+0000", 2024-05-15T13:04:15.583 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1375000000000003e-05 2024-05-15T13:04:15.583 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.583 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:15.583 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.583 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.584 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.584 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24832.0:236833 3.0 3:0974be9e:::benchmark_data_smithi177_88104_object103211:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T13:04:15.584 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:56:47.845210+0000", 2024-05-15T13:04:15.584 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 447.71131422000002, 2024-05-15T13:04:15.584 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19971081700000001, 2024-05-15T13:04:15.584 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:15.584 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:15.584 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:15.584 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24832", 2024-05-15T13:04:15.584 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:04:15.584 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 236833 2024-05-15T13:04:15.584 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.584 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:15.585 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.585 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:15.585 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.845210+0000", 2024-05-15T13:04:15.585 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.585 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.585 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.585 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:15.585 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.845210+0000", 2024-05-15T13:04:15.585 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.586 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.586 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.586 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:15.586 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.845210+0000", 2024-05-15T13:04:15.586 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5799999999999998e-07 2024-05-15T13:04:15.586 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.586 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.586 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:15.586 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.845215+0000", 2024-05-15T13:04:15.586 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3409999999999996e-06 2024-05-15T13:04:15.586 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.586 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.586 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:15.587 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.845226+0000", 2024-05-15T13:04:15.587 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0927000000000001e-05 2024-05-15T13:04:15.587 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.587 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.587 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:15.587 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.845232+0000", 2024-05-15T13:04:15.587 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5230000000000001e-06 2024-05-15T13:04:15.587 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.587 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.588 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:15.588 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.845274+0000", 2024-05-15T13:04:15.588 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2336999999999998e-05 2024-05-15T13:04:15.588 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.588 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.588 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:15.588 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.845357+0000", 2024-05-15T13:04:15.588 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2600000000000002e-05 2024-05-15T13:04:15.588 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.588 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.588 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:04:15.588 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.845428+0000", 2024-05-15T13:04:15.588 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1030000000000003e-05 2024-05-15T13:04:15.589 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.589 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.589 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:15.589 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.848264+0000", 2024-05-15T13:04:15.589 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0028355939999999999 2024-05-15T13:04:15.589 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.589 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.589 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.589 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.852386+0000", 2024-05-15T13:04:15.589 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0041226880000000002 2024-05-15T13:04:15.589 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.589 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.590 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.590 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.044814+0000", 2024-05-15T13:04:15.590 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19242753300000001 2024-05-15T13:04:15.590 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.590 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.590 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:15.590 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.044871+0000", 2024-05-15T13:04:15.590 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7484000000000001e-05 2024-05-15T13:04:15.590 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.590 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.591 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:15.591 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.044920+0000", 2024-05-15T13:04:15.591 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9002000000000003e-05 2024-05-15T13:04:15.591 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.591 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:15.591 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.591 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.591 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.591 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24832.0:236859 3.0 3:09d5c0bc:::benchmark_data_smithi177_88104_object103237:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T13:04:15.591 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:56:47.881063+0000", 2024-05-15T13:04:15.591 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 447.67546103799998, 2024-05-15T13:04:15.591 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.181318805, 2024-05-15T13:04:15.591 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:15.591 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:15.592 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:15.592 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24832", 2024-05-15T13:04:15.592 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:04:15.592 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 236859 2024-05-15T13:04:15.592 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.592 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:15.592 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.592 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:15.592 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.881063+0000", 2024-05-15T13:04:15.592 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.592 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.592 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.592 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:15.592 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.881063+0000", 2024-05-15T13:04:15.593 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.593 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.593 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.593 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:15.593 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.881064+0000", 2024-05-15T13:04:15.593 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1820000000000001e-06 2024-05-15T13:04:15.593 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.593 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.593 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:15.593 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.881070+0000", 2024-05-15T13:04:15.593 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7069999999999997e-06 2024-05-15T13:04:15.593 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.593 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.593 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:15.594 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.881081+0000", 2024-05-15T13:04:15.594 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1772e-05 2024-05-15T13:04:15.594 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.594 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.594 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:15.594 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.881088+0000", 2024-05-15T13:04:15.594 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.545e-06 2024-05-15T13:04:15.594 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.594 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.594 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:15.594 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.881132+0000", 2024-05-15T13:04:15.594 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3782999999999997e-05 2024-05-15T13:04:15.594 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.595 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.595 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:15.595 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.881227+0000", 2024-05-15T13:04:15.595 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4953999999999999e-05 2024-05-15T13:04:15.595 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.595 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.595 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:04:15.595 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.881307+0000", 2024-05-15T13:04:15.595 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0218999999999999e-05 2024-05-15T13:04:15.595 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.595 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.595 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:15.596 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.884984+0000", 2024-05-15T13:04:15.596 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003676655 2024-05-15T13:04:15.596 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.596 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.596 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.596 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.895511+0000", 2024-05-15T13:04:15.596 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010527205 2024-05-15T13:04:15.596 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.596 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.596 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.596 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.062281+0000", 2024-05-15T13:04:15.596 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.166769996 2024-05-15T13:04:15.596 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.597 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.597 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:15.597 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.062335+0000", 2024-05-15T13:04:15.597 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4129000000000003e-05 2024-05-15T13:04:15.597 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.597 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.597 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:15.597 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.062381+0000", 2024-05-15T13:04:15.597 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6657999999999999e-05 2024-05-15T13:04:15.597 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.597 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:15.597 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.598 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.598 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.598 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24832.0:236875 3.0 3:0edee374:::benchmark_data_smithi177_88104_object103253:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T13:04:15.598 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:56:47.904492+0000", 2024-05-15T13:04:15.598 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 447.652031604, 2024-05-15T13:04:15.598 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.158026786, 2024-05-15T13:04:15.598 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:15.598 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:15.598 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:15.598 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24832", 2024-05-15T13:04:15.599 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:04:15.599 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 236875 2024-05-15T13:04:15.599 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.599 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:15.599 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.599 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:15.599 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.904492+0000", 2024-05-15T13:04:15.599 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.599 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.599 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.600 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:15.600 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.904492+0000", 2024-05-15T13:04:15.600 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.600 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.600 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.600 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:15.600 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.904494+0000", 2024-05-15T13:04:15.600 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6300000000000001e-06 2024-05-15T13:04:15.600 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.601 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.601 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:15.601 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.904499+0000", 2024-05-15T13:04:15.601 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6400000000000002e-06 2024-05-15T13:04:15.601 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.601 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.601 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:15.601 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.904511+0000", 2024-05-15T13:04:15.601 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1686e-05 2024-05-15T13:04:15.602 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.602 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.602 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:15.602 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.904518+0000", 2024-05-15T13:04:15.602 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5889999999999999e-06 2024-05-15T13:04:15.602 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.602 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.602 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:15.602 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.904577+0000", 2024-05-15T13:04:15.603 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9617000000000002e-05 2024-05-15T13:04:15.603 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.603 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.603 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:15.603 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.904672+0000", 2024-05-15T13:04:15.603 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4913000000000004e-05 2024-05-15T13:04:15.603 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.603 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.603 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:04:15.603 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.904752+0000", 2024-05-15T13:04:15.603 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0148000000000006e-05 2024-05-15T13:04:15.603 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.603 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.603 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:15.604 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.909138+0000", 2024-05-15T13:04:15.604 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0043853249999999998 2024-05-15T13:04:15.604 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.604 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.604 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.604 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.912165+0000", 2024-05-15T13:04:15.604 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0030273240000000001 2024-05-15T13:04:15.604 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.604 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.604 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.604 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.062426+0000", 2024-05-15T13:04:15.604 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.15026115500000001 2024-05-15T13:04:15.604 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.605 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.605 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:15.605 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.062477+0000", 2024-05-15T13:04:15.605 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1162000000000002e-05 2024-05-15T13:04:15.605 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.605 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.605 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:15.605 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.062519+0000", 2024-05-15T13:04:15.605 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1597000000000002e-05 2024-05-15T13:04:15.605 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.605 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:15.605 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.605 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.605 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.605 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.24832.0:236876 3.0 3:039a99ac:::benchmark_data_smithi177_88104_object103254:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T13:04:15.606 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:56:47.909097+0000", 2024-05-15T13:04:15.606 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 447.647426908, 2024-05-15T13:04:15.606 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16179138600000001, 2024-05-15T13:04:15.606 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:15.606 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:15.606 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:15.606 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.24832", 2024-05-15T13:04:15.606 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:04:15.606 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 236876 2024-05-15T13:04:15.606 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.606 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:15.606 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.606 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:15.607 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.909097+0000", 2024-05-15T13:04:15.607 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.607 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.607 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.607 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:15.607 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.909097+0000", 2024-05-15T13:04:15.607 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.607 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.607 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.607 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:15.607 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.909098+0000", 2024-05-15T13:04:15.607 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9400000000000004e-07 2024-05-15T13:04:15.607 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.607 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.608 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:15.608 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.909105+0000", 2024-05-15T13:04:15.608 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4760000000000001e-06 2024-05-15T13:04:15.608 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.608 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.608 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:15.608 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.909117+0000", 2024-05-15T13:04:15.608 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.147e-05 2024-05-15T13:04:15.608 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.608 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.608 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:15.608 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.909132+0000", 2024-05-15T13:04:15.608 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5628999999999999e-05 2024-05-15T13:04:15.608 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.609 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.609 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:15.609 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.909172+0000", 2024-05-15T13:04:15.609 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0312000000000003e-05 2024-05-15T13:04:15.609 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.609 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.609 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:15.609 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.909270+0000", 2024-05-15T13:04:15.609 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7972000000000003e-05 2024-05-15T13:04:15.609 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.609 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.609 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:04:15.609 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.909344+0000", 2024-05-15T13:04:15.609 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3541000000000006e-05 2024-05-15T13:04:15.610 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.610 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.610 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:15.610 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.912263+0000", 2024-05-15T13:04:15.610 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0029188830000000002 2024-05-15T13:04:15.610 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.610 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.610 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.610 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:47.914946+0000", 2024-05-15T13:04:15.610 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0026826459999999999 2024-05-15T13:04:15.610 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.610 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.610 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.611 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.070787+0000", 2024-05-15T13:04:15.611 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.155841019 2024-05-15T13:04:15.611 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.611 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.611 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:15.611 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.070845+0000", 2024-05-15T13:04:15.611 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7985999999999997e-05 2024-05-15T13:04:15.611 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.611 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.611 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:15.611 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:56:48.070888+0000", 2024-05-15T13:04:15.611 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3658000000000001e-05 2024-05-15T13:04:15.611 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.611 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:15.612 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.612 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.612 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.612 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25180.0:87189 4.6 4:639dc0c9:::benchmark_data_smithi177_99756_object87188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-05-15T13:04:15.612 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:58:57.760947+0000", 2024-05-15T13:04:15.612 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 317.79557703199998, 2024-05-15T13:04:15.612 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21072169900000001, 2024-05-15T13:04:15.612 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:15.612 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:15.612 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:15.612 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25180", 2024-05-15T13:04:15.612 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:04:15.612 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 87189 2024-05-15T13:04:15.612 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.612 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:15.613 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.613 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:15.613 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.760947+0000", 2024-05-15T13:04:15.613 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.613 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.613 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.613 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:15.613 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.760947+0000", 2024-05-15T13:04:15.613 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.613 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.613 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.613 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:15.613 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.760948+0000", 2024-05-15T13:04:15.613 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4619999999999999e-06 2024-05-15T13:04:15.614 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.614 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.614 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:15.614 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.760975+0000", 2024-05-15T13:04:15.614 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6735000000000001e-05 2024-05-15T13:04:15.614 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.614 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.614 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:15.614 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.760989+0000", 2024-05-15T13:04:15.614 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4127999999999999e-05 2024-05-15T13:04:15.614 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.614 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.614 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:15.615 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.760998+0000", 2024-05-15T13:04:15.615 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2250000000000006e-06 2024-05-15T13:04:15.615 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.615 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.615 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:15.615 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.761066+0000", 2024-05-15T13:04:15.615 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7608999999999996e-05 2024-05-15T13:04:15.615 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.615 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.615 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:15.615 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.761213+0000", 2024-05-15T13:04:15.615 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014743199999999999 2024-05-15T13:04:15.615 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.616 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.616 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,2", 2024-05-15T13:04:15.616 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.761345+0000", 2024-05-15T13:04:15.616 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000131476 2024-05-15T13:04:15.616 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.616 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.616 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:15.616 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.763778+0000", 2024-05-15T13:04:15.616 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024332680000000001 2024-05-15T13:04:15.616 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.616 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.616 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.617 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.767865+0000", 2024-05-15T13:04:15.617 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0040873430000000002 2024-05-15T13:04:15.617 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.617 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.617 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.617 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.971439+0000", 2024-05-15T13:04:15.617 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20357341300000001 2024-05-15T13:04:15.617 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.617 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.617 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:15.617 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.971534+0000", 2024-05-15T13:04:15.617 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5048000000000002e-05 2024-05-15T13:04:15.617 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.618 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.618 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:15.618 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.971668+0000", 2024-05-15T13:04:15.618 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013456000000000001 2024-05-15T13:04:15.618 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.618 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:15.618 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.618 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.618 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.618 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25180.0:87224 4.6 4:6f3c272e:::benchmark_data_smithi177_99756_object87223:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-05-15T13:04:15.618 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:58:57.789587+0000", 2024-05-15T13:04:15.618 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 317.76693696500001, 2024-05-15T13:04:15.618 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18748165899999999, 2024-05-15T13:04:15.619 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:15.619 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:15.619 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:15.619 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25180", 2024-05-15T13:04:15.619 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:04:15.619 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 87224 2024-05-15T13:04:15.619 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.619 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:15.619 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.619 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:15.619 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.789587+0000", 2024-05-15T13:04:15.619 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.619 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.620 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.620 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:15.620 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.789587+0000", 2024-05-15T13:04:15.620 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.620 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.620 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.620 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:15.620 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.789589+0000", 2024-05-15T13:04:15.620 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.232e-06 2024-05-15T13:04:15.620 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.620 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.620 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:15.620 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.789618+0000", 2024-05-15T13:04:15.620 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9354000000000001e-05 2024-05-15T13:04:15.620 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.621 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.621 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:15.621 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.789639+0000", 2024-05-15T13:04:15.621 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1072000000000001e-05 2024-05-15T13:04:15.621 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.621 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.621 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:15.621 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.789648+0000", 2024-05-15T13:04:15.621 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2779999999999993e-06 2024-05-15T13:04:15.621 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.621 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.621 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:15.621 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.789706+0000", 2024-05-15T13:04:15.622 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8799999999999999e-05 2024-05-15T13:04:15.622 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.622 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.622 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:15.622 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.789828+0000", 2024-05-15T13:04:15.622 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000121235 2024-05-15T13:04:15.622 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.622 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.622 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,2", 2024-05-15T13:04:15.622 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.789971+0000", 2024-05-15T13:04:15.622 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014303699999999999 2024-05-15T13:04:15.622 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.622 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.622 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:15.623 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.792400+0000", 2024-05-15T13:04:15.623 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024290689999999998 2024-05-15T13:04:15.623 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.623 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.623 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.623 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.795818+0000", 2024-05-15T13:04:15.623 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0034178709999999998 2024-05-15T13:04:15.623 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.623 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.623 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.623 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.976866+0000", 2024-05-15T13:04:15.623 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18104798899999999 2024-05-15T13:04:15.623 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.623 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.624 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:15.624 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.976955+0000", 2024-05-15T13:04:15.624 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9381000000000004e-05 2024-05-15T13:04:15.624 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.624 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.624 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:15.624 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.977068+0000", 2024-05-15T13:04:15.624 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000113341 2024-05-15T13:04:15.624 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.625 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:15.625 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.625 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.625 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.625 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25180.0:87241 4.6 4:629f316e:::benchmark_data_smithi177_99756_object87240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-05-15T13:04:15.625 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:58:57.800684+0000", 2024-05-15T13:04:15.625 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 317.75583997000001, 2024-05-15T13:04:15.625 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.176514161, 2024-05-15T13:04:15.625 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:15.625 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:15.625 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:15.625 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25180", 2024-05-15T13:04:15.625 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:04:15.626 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 87241 2024-05-15T13:04:15.626 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.626 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:15.626 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.626 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:15.626 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.800684+0000", 2024-05-15T13:04:15.626 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.626 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.626 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.626 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:15.626 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.800684+0000", 2024-05-15T13:04:15.627 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.627 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.627 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.627 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:15.627 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.800686+0000", 2024-05-15T13:04:15.627 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8330000000000001e-06 2024-05-15T13:04:15.627 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.627 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.627 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:15.627 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.800712+0000", 2024-05-15T13:04:15.627 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6075999999999999e-05 2024-05-15T13:04:15.627 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.627 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.628 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:15.628 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.800728+0000", 2024-05-15T13:04:15.628 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6311e-05 2024-05-15T13:04:15.628 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.628 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.628 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:15.628 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.800739+0000", 2024-05-15T13:04:15.628 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1014e-05 2024-05-15T13:04:15.628 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.628 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.628 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:15.628 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.800800+0000", 2024-05-15T13:04:15.629 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0534e-05 2024-05-15T13:04:15.629 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.629 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.629 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:15.629 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.800913+0000", 2024-05-15T13:04:15.629 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0001132 2024-05-15T13:04:15.629 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.629 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.629 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,2", 2024-05-15T13:04:15.630 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.801023+0000", 2024-05-15T13:04:15.630 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010997600000000001 2024-05-15T13:04:15.630 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.630 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.630 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:15.630 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.804118+0000", 2024-05-15T13:04:15.630 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003095184 2024-05-15T13:04:15.630 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.630 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.630 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.630 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.807555+0000", 2024-05-15T13:04:15.630 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0034369779999999998 2024-05-15T13:04:15.631 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.631 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.631 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.631 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.977128+0000", 2024-05-15T13:04:15.631 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.169573368 2024-05-15T13:04:15.631 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.631 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.631 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:15.631 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.977161+0000", 2024-05-15T13:04:15.631 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2277999999999998e-05 2024-05-15T13:04:15.631 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.632 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.632 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:15.632 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.977198+0000", 2024-05-15T13:04:15.632 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7409e-05 2024-05-15T13:04:15.632 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.632 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:15.632 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.632 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.632 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.632 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25180.0:87249 4.6 4:6ca3a1b8:::benchmark_data_smithi177_99756_object87248:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-05-15T13:04:15.632 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:58:57.808208+0000", 2024-05-15T13:04:15.633 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 317.74831551900002, 2024-05-15T13:04:15.633 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17109327199999999, 2024-05-15T13:04:15.633 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:15.633 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:15.633 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:15.633 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25180", 2024-05-15T13:04:15.633 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:04:15.633 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 87249 2024-05-15T13:04:15.633 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.633 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:15.633 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.633 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:15.633 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.808208+0000", 2024-05-15T13:04:15.634 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.634 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.634 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.634 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:15.634 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.808208+0000", 2024-05-15T13:04:15.634 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.634 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.634 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.634 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:15.635 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.808210+0000", 2024-05-15T13:04:15.635 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.311e-06 2024-05-15T13:04:15.635 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.635 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.635 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:15.635 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.808228+0000", 2024-05-15T13:04:15.635 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8422000000000001e-05 2024-05-15T13:04:15.635 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.635 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.635 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:15.636 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.808240+0000", 2024-05-15T13:04:15.636 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1725e-05 2024-05-15T13:04:15.636 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.636 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.636 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:15.636 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.808246+0000", 2024-05-15T13:04:15.636 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0209999999999996e-06 2024-05-15T13:04:15.636 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.636 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.637 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:15.637 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.808297+0000", 2024-05-15T13:04:15.637 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1773e-05 2024-05-15T13:04:15.637 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.637 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.637 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:15.637 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.808464+0000", 2024-05-15T13:04:15.637 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000166885 2024-05-15T13:04:15.637 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.637 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.637 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,2", 2024-05-15T13:04:15.638 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.808640+0000", 2024-05-15T13:04:15.638 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017578099999999999 2024-05-15T13:04:15.638 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.638 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.638 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:15.638 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.810995+0000", 2024-05-15T13:04:15.638 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002355191 2024-05-15T13:04:15.638 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.638 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.638 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.638 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.813140+0000", 2024-05-15T13:04:15.638 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002144679 2024-05-15T13:04:15.638 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.638 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.639 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.639 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.979238+0000", 2024-05-15T13:04:15.639 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16609758299999999 2024-05-15T13:04:15.639 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.639 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.639 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:15.639 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.979269+0000", 2024-05-15T13:04:15.639 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1642e-05 2024-05-15T13:04:15.639 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.639 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.639 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:15.639 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.979301+0000", 2024-05-15T13:04:15.639 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2258999999999997e-05 2024-05-15T13:04:15.639 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.640 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:15.640 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.640 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.640 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.640 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25180.0:87260 4.6 4:60e4d7fd:::benchmark_data_smithi177_99756_object87259:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-05-15T13:04:15.640 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T12:58:57.821335+0000", 2024-05-15T13:04:15.640 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 317.73518862700001, 2024-05-15T13:04:15.640 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.15813233300000001, 2024-05-15T13:04:15.640 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:15.640 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:15.640 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:15.640 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25180", 2024-05-15T13:04:15.640 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:04:15.640 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 87260 2024-05-15T13:04:15.640 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.641 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:15.641 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.641 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:15.641 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.821335+0000", 2024-05-15T13:04:15.641 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.641 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.641 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.641 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:15.641 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.821335+0000", 2024-05-15T13:04:15.641 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.641 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.641 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.642 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:15.642 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.821336+0000", 2024-05-15T13:04:15.642 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.026e-06 2024-05-15T13:04:15.642 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.642 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.642 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:15.642 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.821356+0000", 2024-05-15T13:04:15.642 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9669000000000001e-05 2024-05-15T13:04:15.642 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.642 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.642 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:15.642 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.821367+0000", 2024-05-15T13:04:15.642 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0725e-05 2024-05-15T13:04:15.643 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.643 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.643 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:15.643 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.821372+0000", 2024-05-15T13:04:15.643 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3959999999999997e-06 2024-05-15T13:04:15.643 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.643 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.643 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:15.643 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.821412+0000", 2024-05-15T13:04:15.643 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0263999999999999e-05 2024-05-15T13:04:15.643 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.643 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.643 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:15.643 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.821521+0000", 2024-05-15T13:04:15.644 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010870399999999999 2024-05-15T13:04:15.644 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.644 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.644 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,2", 2024-05-15T13:04:15.644 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.821628+0000", 2024-05-15T13:04:15.644 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000107537 2024-05-15T13:04:15.644 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.644 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.644 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:15.644 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.823949+0000", 2024-05-15T13:04:15.644 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023207010000000001 2024-05-15T13:04:15.644 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.644 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.644 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.645 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.826308+0000", 2024-05-15T13:04:15.645 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023585379999999999 2024-05-15T13:04:15.645 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.645 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.645 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.645 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.979355+0000", 2024-05-15T13:04:15.645 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.15304744000000001 2024-05-15T13:04:15.645 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.645 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.645 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:15.645 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.979411+0000", 2024-05-15T13:04:15.645 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5858e-05 2024-05-15T13:04:15.645 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.645 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.646 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:15.646 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T12:58:57.979467+0000", 2024-05-15T13:04:15.646 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6474999999999997e-05 2024-05-15T13:04:15.646 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.646 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:15.646 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.646 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.646 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.646 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25180.0:211405 4.6 4:6412caac:::benchmark_data_smithi177_99756_object91433:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-05-15T13:04:15.646 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:02:03.309414+0000", 2024-05-15T13:04:15.646 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 132.24710989600001, 2024-05-15T13:04:15.646 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.083073603999999995, 2024-05-15T13:04:15.646 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:15.647 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:15.647 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:15.647 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25180", 2024-05-15T13:04:15.647 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:04:15.647 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 211405 2024-05-15T13:04:15.647 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.647 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:15.647 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.647 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:15.647 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.309414+0000", 2024-05-15T13:04:15.647 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.647 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.647 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.647 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:15.647 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.309414+0000", 2024-05-15T13:04:15.648 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.648 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.648 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.648 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:15.648 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.309415+0000", 2024-05-15T13:04:15.648 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5460000000000001e-06 2024-05-15T13:04:15.648 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.648 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.648 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:15.648 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.309433+0000", 2024-05-15T13:04:15.648 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.804e-05 2024-05-15T13:04:15.648 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.648 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.649 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:15.649 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.309453+0000", 2024-05-15T13:04:15.649 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9738e-05 2024-05-15T13:04:15.649 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.649 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.649 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:15.649 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.309462+0000", 2024-05-15T13:04:15.649 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9749999999999996e-06 2024-05-15T13:04:15.649 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.649 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.649 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:15.649 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.309568+0000", 2024-05-15T13:04:15.649 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105794 2024-05-15T13:04:15.649 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.649 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.650 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:15.650 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.309702+0000", 2024-05-15T13:04:15.650 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013413499999999999 2024-05-15T13:04:15.650 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.650 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.650 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,2", 2024-05-15T13:04:15.650 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.309822+0000", 2024-05-15T13:04:15.650 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011945700000000001 2024-05-15T13:04:15.650 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.650 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.650 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:15.650 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.320051+0000", 2024-05-15T13:04:15.650 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010229145 2024-05-15T13:04:15.651 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.651 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.651 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.651 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.320288+0000", 2024-05-15T13:04:15.651 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023767699999999999 2024-05-15T13:04:15.651 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.651 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.651 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.651 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.392408+0000", 2024-05-15T13:04:15.651 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.072120039999999996 2024-05-15T13:04:15.651 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.651 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.651 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:15.651 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.392450+0000", 2024-05-15T13:04:15.652 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1176e-05 2024-05-15T13:04:15.652 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.652 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.652 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:15.652 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.392487+0000", 2024-05-15T13:04:15.652 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7880999999999999e-05 2024-05-15T13:04:15.652 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.652 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:15.652 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.652 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.652 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.652 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25180.0:211407 4.6 4:6f4c4d2d:::benchmark_data_smithi177_99756_object91435:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-05-15T13:04:15.652 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:02:03.309498+0000", 2024-05-15T13:04:15.652 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 132.247025941, 2024-05-15T13:04:15.652 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.083121438000000006, 2024-05-15T13:04:15.653 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:15.653 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:15.653 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:15.653 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25180", 2024-05-15T13:04:15.653 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:04:15.653 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 211407 2024-05-15T13:04:15.653 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.653 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:15.653 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.653 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:15.653 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.309498+0000", 2024-05-15T13:04:15.653 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.653 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.653 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.654 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:15.654 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.309498+0000", 2024-05-15T13:04:15.654 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.654 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.654 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.654 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:15.654 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.309498+0000", 2024-05-15T13:04:15.654 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4199999999999996e-07 2024-05-15T13:04:15.654 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.654 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.654 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:15.654 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.309502+0000", 2024-05-15T13:04:15.655 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2940000000000001e-06 2024-05-15T13:04:15.655 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.655 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.655 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:04:15.656 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:15.656 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.309513+0000", 2024-05-15T13:04:15.656 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1012999999999999e-05 2024-05-15T13:04:15.657 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.657 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.657 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:15.657 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.309519+0000", 2024-05-15T13:04:15.657 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5209999999999999e-06 2024-05-15T13:04:15.657 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.657 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.657 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:15.657 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.314845+0000", 2024-05-15T13:04:15.657 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0053259270000000003 2024-05-15T13:04:15.658 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.658 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.658 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:15.658 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.314975+0000", 2024-05-15T13:04:15.658 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012988600000000001 2024-05-15T13:04:15.658 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.658 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.658 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,2", 2024-05-15T13:04:15.658 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.315099+0000", 2024-05-15T13:04:15.658 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012406599999999999 2024-05-15T13:04:15.659 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.659 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.659 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:15.659 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.320167+0000", 2024-05-15T13:04:15.659 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0050675540000000002 2024-05-15T13:04:15.659 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.659 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.659 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.659 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.320693+0000", 2024-05-15T13:04:15.659 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000526462 2024-05-15T13:04:15.659 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.659 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.660 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.660 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.392528+0000", 2024-05-15T13:04:15.660 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.071834791999999995 2024-05-15T13:04:15.660 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.660 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.660 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:15.660 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.392586+0000", 2024-05-15T13:04:15.660 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8600000000000001e-05 2024-05-15T13:04:15.660 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.660 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.661 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:15.661 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:03.392619+0000", 2024-05-15T13:04:15.661 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2780999999999997e-05 2024-05-15T13:04:15.661 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.661 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:15.661 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.661 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.661 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.661 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25180.0:237538 4.6 4:6ab31870:::benchmark_data_smithi177_99756_object117566:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e404)", 2024-05-15T13:04:15.661 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:02:39.224514+0000", 2024-05-15T13:04:15.661 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 96.332009546999998, 2024-05-15T13:04:15.661 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.078582611999999996, 2024-05-15T13:04:15.661 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:15.662 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:15.662 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:15.662 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25180", 2024-05-15T13:04:15.662 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:04:15.662 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 237538 2024-05-15T13:04:15.662 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.662 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:15.662 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.662 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:15.663 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.224514+0000", 2024-05-15T13:04:15.663 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.663 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.663 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.663 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:15.663 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.224514+0000", 2024-05-15T13:04:15.663 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.663 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.663 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.663 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:15.663 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.224516+0000", 2024-05-15T13:04:15.663 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.531e-06 2024-05-15T13:04:15.664 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.664 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.664 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:15.664 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.224524+0000", 2024-05-15T13:04:15.664 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8709999999999997e-06 2024-05-15T13:04:15.664 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.664 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.664 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:15.664 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.224541+0000", 2024-05-15T13:04:15.664 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7618e-05 2024-05-15T13:04:15.664 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.665 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.665 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:15.665 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.224563+0000", 2024-05-15T13:04:15.665 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1569e-05 2024-05-15T13:04:15.665 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.665 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.665 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:15.665 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.224643+0000", 2024-05-15T13:04:15.665 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0710999999999993e-05 2024-05-15T13:04:15.665 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.665 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.665 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:15.665 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.224806+0000", 2024-05-15T13:04:15.666 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016213299999999999 2024-05-15T13:04:15.666 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.666 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.666 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,2", 2024-05-15T13:04:15.666 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.224933+0000", 2024-05-15T13:04:15.666 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012739899999999999 2024-05-15T13:04:15.666 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.666 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.666 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:15.666 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.233954+0000", 2024-05-15T13:04:15.666 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0090205619999999993 2024-05-15T13:04:15.666 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.666 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.667 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.667 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.234117+0000", 2024-05-15T13:04:15.667 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016306200000000001 2024-05-15T13:04:15.667 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.667 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.667 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.667 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.302966+0000", 2024-05-15T13:04:15.667 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.068849091000000001 2024-05-15T13:04:15.667 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.667 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.667 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:15.667 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.303044+0000", 2024-05-15T13:04:15.667 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8644000000000001e-05 2024-05-15T13:04:15.668 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.668 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.668 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:15.668 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.303097+0000", 2024-05-15T13:04:15.668 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2420999999999998e-05 2024-05-15T13:04:15.668 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.668 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:15.668 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.668 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.668 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.668 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25180.0:237545 4.6 4:60e1b730:::benchmark_data_smithi177_99756_object117573:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e404)", 2024-05-15T13:04:15.668 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:02:39.228894+0000", 2024-05-15T13:04:15.668 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 96.327630013000004, 2024-05-15T13:04:15.669 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.074384663000000004, 2024-05-15T13:04:15.669 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:15.669 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:15.669 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:15.669 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25180", 2024-05-15T13:04:15.669 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:04:15.669 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 237545 2024-05-15T13:04:15.669 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.669 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:15.669 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.669 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:15.669 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.228894+0000", 2024-05-15T13:04:15.669 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.669 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.670 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.670 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:15.670 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.228894+0000", 2024-05-15T13:04:15.670 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.670 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.670 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.670 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:15.670 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.228895+0000", 2024-05-15T13:04:15.670 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4759999999999999e-06 2024-05-15T13:04:15.670 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.670 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.671 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:15.671 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.228903+0000", 2024-05-15T13:04:15.671 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.452e-06 2024-05-15T13:04:15.671 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.671 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.671 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:15.671 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.228919+0000", 2024-05-15T13:04:15.671 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6008e-05 2024-05-15T13:04:15.671 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.671 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.671 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:15.671 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.228927+0000", 2024-05-15T13:04:15.671 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3809999999999999e-06 2024-05-15T13:04:15.671 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.672 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.672 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:15.672 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.229720+0000", 2024-05-15T13:04:15.672 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00079284799999999997 2024-05-15T13:04:15.672 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.672 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.672 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:15.672 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.229820+0000", 2024-05-15T13:04:15.672 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000100473 2024-05-15T13:04:15.672 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.672 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.672 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,2", 2024-05-15T13:04:15.673 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.229903+0000", 2024-05-15T13:04:15.673 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3117999999999999e-05 2024-05-15T13:04:15.673 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.673 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.673 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:15.673 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.234177+0000", 2024-05-15T13:04:15.673 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0042731920000000003 2024-05-15T13:04:15.673 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.673 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.673 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.673 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.235664+0000", 2024-05-15T13:04:15.673 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014870020000000001 2024-05-15T13:04:15.674 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.674 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.674 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.674 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.303153+0000", 2024-05-15T13:04:15.674 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.067489507000000004 2024-05-15T13:04:15.674 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.674 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.674 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:15.675 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.303214+0000", 2024-05-15T13:04:15.675 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1253000000000005e-05 2024-05-15T13:04:15.675 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.675 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.675 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:15.675 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:39.303278+0000", 2024-05-15T13:04:15.675 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3953000000000002e-05 2024-05-15T13:04:15.675 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.675 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:15.675 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.676 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.676 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.676 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:64212 5.e 5:7c8b4018:::benchmark_data_smithi177_109991_object64211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-05-15T13:04:15.676 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:49.990397+0000", 2024-05-15T13:04:15.676 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 25.566126783000001, 2024-05-15T13:04:15.676 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.081069499000000003, 2024-05-15T13:04:15.676 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:15.676 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:15.676 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:15.677 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:04:15.677 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:04:15.677 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 64212 2024-05-15T13:04:15.677 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.677 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:15.677 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.677 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:15.677 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:49.990397+0000", 2024-05-15T13:04:15.677 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.677 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.677 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.677 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:15.678 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:49.990397+0000", 2024-05-15T13:04:15.678 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.678 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.678 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.678 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:15.678 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:49.990399+0000", 2024-05-15T13:04:15.678 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.024e-06 2024-05-15T13:04:15.678 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.678 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.678 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:15.678 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:49.990420+0000", 2024-05-15T13:04:15.678 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0682999999999998e-05 2024-05-15T13:04:15.678 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.679 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.679 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:15.679 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:49.990433+0000", 2024-05-15T13:04:15.679 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3777e-05 2024-05-15T13:04:15.679 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.679 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.679 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:15.679 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:49.990441+0000", 2024-05-15T13:04:15.679 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.559e-06 2024-05-15T13:04:15.679 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.679 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.679 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:15.679 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:49.990493+0000", 2024-05-15T13:04:15.679 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1538e-05 2024-05-15T13:04:15.680 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.680 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.680 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:15.680 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:49.990635+0000", 2024-05-15T13:04:15.680 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014210199999999999 2024-05-15T13:04:15.680 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.680 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.680 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,8", 2024-05-15T13:04:15.680 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:49.990791+0000", 2024-05-15T13:04:15.680 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015622100000000001 2024-05-15T13:04:15.680 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.680 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.680 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:15.680 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:49.992916+0000", 2024-05-15T13:04:15.681 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021248529999999999 2024-05-15T13:04:15.681 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.681 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.681 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.681 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:49.995530+0000", 2024-05-15T13:04:15.681 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002614636 2024-05-15T13:04:15.681 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.681 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.681 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.681 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:50.071360+0000", 2024-05-15T13:04:15.681 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.075829296000000004 2024-05-15T13:04:15.681 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.681 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.681 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:15.682 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:50.071429+0000", 2024-05-15T13:04:15.682 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9014e-05 2024-05-15T13:04:15.682 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.682 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.682 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:15.682 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:50.071466+0000", 2024-05-15T13:04:15.682 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7796000000000002e-05 2024-05-15T13:04:15.682 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.682 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:15.682 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.682 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.682 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.682 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:64225 5.e 5:7c2ea5a6:::benchmark_data_smithi177_109991_object64224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-05-15T13:04:15.682 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:49.996668+0000", 2024-05-15T13:04:15.683 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 25.559855829, 2024-05-15T13:04:15.683 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.079789819999999997, 2024-05-15T13:04:15.683 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:15.683 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:15.683 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:15.683 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:04:15.683 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:04:15.683 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 64225 2024-05-15T13:04:15.683 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.683 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:15.683 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.683 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:15.683 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:49.996668+0000", 2024-05-15T13:04:15.683 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.684 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.684 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.684 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:15.684 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:49.996668+0000", 2024-05-15T13:04:15.684 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:15.684 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.684 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.684 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:15.684 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:49.996669+0000", 2024-05-15T13:04:15.684 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6020000000000001e-06 2024-05-15T13:04:15.684 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.684 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.684 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:15.685 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:49.996692+0000", 2024-05-15T13:04:15.685 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2277e-05 2024-05-15T13:04:15.685 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.685 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.685 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:15.685 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:49.996703+0000", 2024-05-15T13:04:15.685 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1623e-05 2024-05-15T13:04:15.685 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.685 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.685 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:15.685 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:49.996710+0000", 2024-05-15T13:04:15.685 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6139999999999997e-06 2024-05-15T13:04:15.685 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.685 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.685 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:15.686 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:49.996758+0000", 2024-05-15T13:04:15.686 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8355e-05 2024-05-15T13:04:15.686 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.686 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.686 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:15.686 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:49.996914+0000", 2024-05-15T13:04:15.686 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015549100000000001 2024-05-15T13:04:15.686 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.686 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.686 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,8", 2024-05-15T13:04:15.686 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:49.997090+0000", 2024-05-15T13:04:15.686 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017612800000000001 2024-05-15T13:04:15.686 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.686 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.687 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:15.687 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:49.999427+0000", 2024-05-15T13:04:15.687 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002336664 2024-05-15T13:04:15.687 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.687 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.687 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.687 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:50.001817+0000", 2024-05-15T13:04:15.687 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023908419999999998 2024-05-15T13:04:15.687 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.687 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.687 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:15.687 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:50.076383+0000", 2024-05-15T13:04:15.687 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.074565329 2024-05-15T13:04:15.687 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.688 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.688 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:15.688 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:50.076428+0000", 2024-05-15T13:04:15.688 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5569999999999999e-05 2024-05-15T13:04:15.688 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:15.688 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:15.688 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:15.688 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:50.076458+0000", 2024-05-15T13:04:15.688 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9325000000000001e-05 2024-05-15T13:04:15.688 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.688 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:15.688 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.688 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:15.688 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:15.689 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:04:15.689 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:04:15.690 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:15 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T13:04:15.690 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:15 smithi027 ceph-mon[39795]: osdmap e442: 12 total, 12 up, 11 in 2024-05-15T13:04:15.694 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:15.722 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:04:15.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:15.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:04:15.829 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:04:15.882 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:15.884 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:15 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T13:04:15.884 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:15 smithi177 ceph-mon[44303]: osdmap e442: 12 total, 12 up, 11 in 2024-05-15T13:04:15.925 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:04:15.963 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:04:15.967 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 55 16 99561 99545 113.083 75.0625 0.0239053 0.00883924 2024-05-15T13:04:15.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:15.975 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:04:16.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:15 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T13:04:16.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:15 smithi120 ceph-mon[43961]: osdmap e442: 12 total, 12 up, 11 in 2024-05-15T13:04:16.076 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:04:16.146 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:16.235 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:04:16.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:16.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:04:16.339 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:04:16.347 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-05-15T13:04:16.381 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:16.410 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:04:16.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:16.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:04:16.501 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:04:16.515 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:04:16.617 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:16.735 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:04:16.838 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:04:16.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:16 smithi027 ceph-mon[39795]: pgmap v1180: 17 pgs: 17 active+clean; 5.9 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-05-15T13:04:16.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:16.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:04:16.948 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:04:16.967 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 56 16 100730 100714 112.369 73.0625 0.00713246 0.00889527 2024-05-15T13:04:16.993 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:17.011 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:16 smithi177 ceph-mon[44303]: pgmap v1180: 17 pgs: 17 active+clean; 5.9 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-05-15T13:04:17.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:16 smithi120 ceph-mon[43961]: pgmap v1180: 17 pgs: 17 active+clean; 5.9 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-05-15T13:04:17.445 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:04:17.458 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:04:17.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:17.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:04:17.515 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:04:17.566 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:04:17.608 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:17.752 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:04:17.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:17.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:04:17.868 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:04:17.910 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:17.968 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 57 16 101864 101848 111.641 70.875 0.00883121 0.00895308 2024-05-15T13:04:18.014 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:04:18.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:18.022 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:04:18.123 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:04:18.180 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:18.407 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:04:18.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:18.415 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:04:18.516 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:04:18.582 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:18.637 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:04:18.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:18.643 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:04:18.715 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:04:18.715 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T13:04:18.715 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.715 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:102655 5.6 e442/418 5:6a93c110:::benchmark_data_smithi177_109991_object102654:head v 442'6421, mlcod=442'6417)", 2024-05-15T13:04:18.715 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:04:18.719066+0000", 2024-05-15T13:04:18.715 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0060016100000000001, 2024-05-15T13:04:18.716 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0060327039999999998, 2024-05-15T13:04:18.716 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:18.716 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T13:04:18.716 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:18.716 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.716 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:18.716 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.719066+0000", 2024-05-15T13:04:18.716 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:18.716 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.716 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.716 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:18.716 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.719066+0000", 2024-05-15T13:04:18.716 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:18.716 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.717 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.717 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:18.717 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.719067+0000", 2024-05-15T13:04:18.717 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6600000000000005e-07 2024-05-15T13:04:18.717 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.717 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.717 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:18.717 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.719124+0000", 2024-05-15T13:04:18.717 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6712000000000002e-05 2024-05-15T13:04:18.717 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.717 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.717 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:18.717 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.719134+0000", 2024-05-15T13:04:18.717 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0155e-05 2024-05-15T13:04:18.717 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.717 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.718 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:18.718 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.719140+0000", 2024-05-15T13:04:18.718 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5430000000000001e-06 2024-05-15T13:04:18.718 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.718 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.718 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:18.718 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.719901+0000", 2024-05-15T13:04:18.718 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00076141100000000003 2024-05-15T13:04:18.718 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.718 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.718 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:18.718 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.719925+0000", 2024-05-15T13:04:18.718 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3349999999999998e-05 2024-05-15T13:04:18.719 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:18.719 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:18.719 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:18.719 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.719 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.719 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:102657 5.6 e442/418 5:6c51de7e:::benchmark_data_smithi177_109991_object102656:head v 442'6422, mlcod=442'6417)", 2024-05-15T13:04:18.719 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:04:18.719974+0000", 2024-05-15T13:04:18.719 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0050937659999999996, 2024-05-15T13:04:18.719 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0051947349999999998, 2024-05-15T13:04:18.719 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:18.719 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T13:04:18.719 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:18.719 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.719 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:18.720 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.719974+0000", 2024-05-15T13:04:18.720 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:18.720 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.720 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.720 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:18.720 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.719974+0000", 2024-05-15T13:04:18.720 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:18.720 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.720 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.720 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:18.720 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.719975+0000", 2024-05-15T13:04:18.720 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0800000000000005e-07 2024-05-15T13:04:18.720 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.720 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.720 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:18.721 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.720053+0000", 2024-05-15T13:04:18.721 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8177999999999996e-05 2024-05-15T13:04:18.721 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.721 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.721 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:18.721 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.720065+0000", 2024-05-15T13:04:18.721 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.242e-05 2024-05-15T13:04:18.721 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.721 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.721 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:18.721 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.720072+0000", 2024-05-15T13:04:18.721 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3459999999999999e-06 2024-05-15T13:04:18.721 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.721 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.721 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:18.722 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.722957+0000", 2024-05-15T13:04:18.722 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0028857190000000001 2024-05-15T13:04:18.722 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.722 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.722 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:18.722 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.722980+0000", 2024-05-15T13:04:18.722 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2218999999999999e-05 2024-05-15T13:04:18.722 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:18.722 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:18.722 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:18.722 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.722 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.722 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:102672 5.a e442/418 5:5f12962c:::benchmark_data_smithi177_109991_object102671:head v 442'6407, mlcod=442'6405)", 2024-05-15T13:04:18.723 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:04:18.722296+0000", 2024-05-15T13:04:18.723 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0027717000000000002, 2024-05-15T13:04:18.723 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0029380669999999999, 2024-05-15T13:04:18.723 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:18.723 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T13:04:18.723 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:18.723 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.723 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:18.723 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.722296+0000", 2024-05-15T13:04:18.723 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:18.723 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.723 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.723 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:18.723 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.722296+0000", 2024-05-15T13:04:18.723 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:18.723 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.724 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.724 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:18.724 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.722297+0000", 2024-05-15T13:04:18.724 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9500000000000002e-07 2024-05-15T13:04:18.724 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.724 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.724 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:18.724 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.722354+0000", 2024-05-15T13:04:18.724 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6691000000000003e-05 2024-05-15T13:04:18.724 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.724 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.724 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:18.724 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.722364+0000", 2024-05-15T13:04:18.724 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8409999999999995e-06 2024-05-15T13:04:18.725 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.725 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.725 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:18.725 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.722369+0000", 2024-05-15T13:04:18.725 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3120000000000001e-06 2024-05-15T13:04:18.725 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.725 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.725 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:18.725 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.722423+0000", 2024-05-15T13:04:18.725 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4048000000000002e-05 2024-05-15T13:04:18.725 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.725 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.725 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:18.725 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.722451+0000", 2024-05-15T13:04:18.725 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7637999999999999e-05 2024-05-15T13:04:18.725 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:18.726 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:18.726 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:18.726 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.726 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.726 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:102664 5.6 e442/418)", 2024-05-15T13:04:18.726 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:04:18.723203+0000", 2024-05-15T13:04:18.726 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0018651620000000001, 2024-05-15T13:04:18.726 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020930670000000001, 2024-05-15T13:04:18.726 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:18.726 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "queued for pg", 2024-05-15T13:04:18.726 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:18.726 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.726 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:18.726 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.723203+0000", 2024-05-15T13:04:18.727 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:18.727 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.727 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.727 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:18.727 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.723203+0000", 2024-05-15T13:04:18.727 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:18.727 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.727 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.727 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:18.727 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.723204+0000", 2024-05-15T13:04:18.727 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7899999999999998e-07 2024-05-15T13:04:18.727 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.727 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.727 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:18.727 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.723278+0000", 2024-05-15T13:04:18.728 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4726000000000002e-05 2024-05-15T13:04:18.728 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.728 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.728 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:18.728 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.723289+0000", 2024-05-15T13:04:18.728 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1218e-05 2024-05-15T13:04:18.728 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:18.728 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:18.728 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:18.728 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:18.723296+0000", 2024-05-15T13:04:18.728 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5119999999999997e-06 2024-05-15T13:04:18.728 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:18.728 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:18.728 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:18.728 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:18.728 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-05-15T13:04:18.729 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 4 2024-05-15T13:04:18.729 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:04:18.744 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:04:18.779 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:18.808 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:04:18.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:18.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:04:18.916 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:04:18.967 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:18.972 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 58 16 102993 102977 110.925 70.5625 0.0150838 0.00901064 2024-05-15T13:04:19.028 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:19 smithi120 ceph-mon[43961]: 5.6 scrub starts 2024-05-15T13:04:19.028 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:19 smithi120 ceph-mon[43961]: pgmap v1181: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.1 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-05-15T13:04:19.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:19 smithi177 ceph-mon[44303]: 5.6 scrub starts 2024-05-15T13:04:19.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:19 smithi177 ceph-mon[44303]: pgmap v1181: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.1 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-05-15T13:04:19.444 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:04:19.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:19.448 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:04:19.478 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-05-15T13:04:19.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:19 smithi027 ceph-mon[39795]: 5.6 scrub starts 2024-05-15T13:04:19.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:19 smithi027 ceph-mon[39795]: pgmap v1181: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.1 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-05-15T13:04:19.549 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:04:19.604 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:19.927 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:04:19.972 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 59 16 104097 104081 110.214 69 0.015246 0.00906903 2024-05-15T13:04:20.313 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:20 smithi177 ceph-mon[44303]: 5.6 scrub ok 2024-05-15T13:04:20.314 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:20 smithi177 ceph-mon[44303]: 5.7 scrub starts 2024-05-15T13:04:20.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:20 smithi027 ceph-mon[39795]: 5.6 scrub ok 2024-05-15T13:04:20.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:20 smithi027 ceph-mon[39795]: 5.7 scrub starts 2024-05-15T13:04:20.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:20 smithi120 ceph-mon[43961]: 5.6 scrub ok 2024-05-15T13:04:20.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:20 smithi120 ceph-mon[43961]: 5.7 scrub starts 2024-05-15T13:04:20.583 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:04:20.592 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:04:20.972 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T13:04:20.983747+0000 min lat: 0.00112512 max lat: 0.21806 avg lat: 0.00913919 2024-05-15T13:04:20.972 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:04:20.972 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 60 16 104987 104971 109.305 55.625 0.104549 0.00913919 2024-05-15T13:04:21.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:21 smithi177 ceph-mon[44303]: 5.a scrub starts 2024-05-15T13:04:21.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:21 smithi177 ceph-mon[44303]: pgmap v1182: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-05-15T13:04:21.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:21 smithi177 ceph-mon[44303]: 5.5 scrub starts 2024-05-15T13:04:21.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:21 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:04:21.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:21 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:04:21.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:21 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:04:21.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:21 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:04:21.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:21 smithi177 ceph-mon[44303]: 5.7 scrub ok 2024-05-15T13:04:21.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:21 smithi177 ceph-mon[44303]: 5.4 scrub starts 2024-05-15T13:04:21.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:21 smithi027 ceph-mon[39795]: 5.a scrub starts 2024-05-15T13:04:21.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:21 smithi027 ceph-mon[39795]: pgmap v1182: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-05-15T13:04:21.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:21 smithi027 ceph-mon[39795]: 5.5 scrub starts 2024-05-15T13:04:21.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:21 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:04:21.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:21 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:04:21.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:21 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:04:21.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:21 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:04:21.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:21 smithi027 ceph-mon[39795]: 5.7 scrub ok 2024-05-15T13:04:21.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:21 smithi027 ceph-mon[39795]: 5.4 scrub starts 2024-05-15T13:04:21.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:21 smithi120 ceph-mon[43961]: 5.a scrub starts 2024-05-15T13:04:21.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:21 smithi120 ceph-mon[43961]: pgmap v1182: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-05-15T13:04:21.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:21 smithi120 ceph-mon[43961]: 5.5 scrub starts 2024-05-15T13:04:21.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:21 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:04:21.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:21 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:04:21.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:21 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:04:21.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:21 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:04:21.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:21 smithi120 ceph-mon[43961]: 5.7 scrub ok 2024-05-15T13:04:21.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:21 smithi120 ceph-mon[43961]: 5.4 scrub starts 2024-05-15T13:04:21.790 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:04:21.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:21.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:04:21.897 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:04:21.947 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:21.972 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 61 16 105977 105961 108.527 61.875 0.0166918 0.00921019 2024-05-15T13:04:21.999 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:04:22.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:22.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:04:22.050 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:22 smithi177 ceph-mon[44303]: from='client.25427 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:04:22.050 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:22 smithi177 ceph-mon[44303]: 5.a scrub ok 2024-05-15T13:04:22.050 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:22 smithi177 ceph-mon[44303]: 5.1 scrub starts 2024-05-15T13:04:22.050 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:22 smithi177 ceph-mon[44303]: osdmap e443: 12 total, 12 up, 11 in 2024-05-15T13:04:22.050 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:22 smithi177 ceph-mon[44303]: 5.3 scrub starts 2024-05-15T13:04:22.053 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:22 smithi027 ceph-mon[39795]: from='client.25427 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:04:22.053 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:22 smithi027 ceph-mon[39795]: 5.a scrub ok 2024-05-15T13:04:22.053 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:22 smithi027 ceph-mon[39795]: 5.1 scrub starts 2024-05-15T13:04:22.053 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:22 smithi027 ceph-mon[39795]: osdmap e443: 12 total, 12 up, 11 in 2024-05-15T13:04:22.053 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:22 smithi027 ceph-mon[39795]: 5.3 scrub starts 2024-05-15T13:04:22.103 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:04:22.157 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:22.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:22 smithi120 ceph-mon[43961]: from='client.25427 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:04:22.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:22 smithi120 ceph-mon[43961]: 5.a scrub ok 2024-05-15T13:04:22.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:22 smithi120 ceph-mon[43961]: 5.1 scrub starts 2024-05-15T13:04:22.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:22 smithi120 ceph-mon[43961]: osdmap e443: 12 total, 12 up, 11 in 2024-05-15T13:04:22.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:22 smithi120 ceph-mon[43961]: 5.3 scrub starts 2024-05-15T13:04:22.365 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:04:22.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:22.368 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:04:22.402 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:04:22.402 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T13:04:22.402 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-05-15T13:04:22.402 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-05-15T13:04:22.402 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:04:22.469 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:04:22.504 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:22.541 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:04:22.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:22.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:04:22.651 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:04:22.703 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:22.792 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:04:22.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:22.807 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:04:22.908 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:04:22.953 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:22.972 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 62 16 106998 106982 107.806 63.8125 0.00866669 0.00927102 2024-05-15T13:04:23.167 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:04:23.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:23.172 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:04:23.184 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-05-15T13:04:23.265 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:23 smithi177 ceph-mon[44303]: pgmap v1184: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.3 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-05-15T13:04:23.265 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:23 smithi177 ceph-mon[44303]: 5.4 scrub ok 2024-05-15T13:04:23.265 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:23 smithi177 ceph-mon[44303]: 5.5 scrub ok 2024-05-15T13:04:23.273 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:04:23.338 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:23.375 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:04:23.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:23.386 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:04:23.487 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:04:23.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:23 smithi027 ceph-mon[39795]: pgmap v1184: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.3 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-05-15T13:04:23.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:23 smithi027 ceph-mon[39795]: 5.4 scrub ok 2024-05-15T13:04:23.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:23 smithi027 ceph-mon[39795]: 5.5 scrub ok 2024-05-15T13:04:23.524 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:23 smithi120 ceph-mon[43961]: pgmap v1184: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.3 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-05-15T13:04:23.525 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:23 smithi120 ceph-mon[43961]: 5.4 scrub ok 2024-05-15T13:04:23.525 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:23 smithi120 ceph-mon[43961]: 5.5 scrub ok 2024-05-15T13:04:23.527 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:23.559 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:04:23.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:23.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:04:23.618 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:04:23.666 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:04:23.721 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:23.757 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:04:23.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:23.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:04:23.861 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:04:23.900 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:23.973 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 63 16 108123 108107 107.211 70.3125 0.00848685 0.00932329 2024-05-15T13:04:24.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:24 smithi177 ceph-mon[44303]: 5.1 scrub ok 2024-05-15T13:04:24.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:24 smithi177 ceph-mon[44303]: 5.3 scrub ok 2024-05-15T13:04:24.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:24 smithi177 ceph-mon[44303]: osdmap e444: 12 total, 12 up, 11 in 2024-05-15T13:04:24.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:24 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:04:24.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:24 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:04:24.509 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:04:24.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:24.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:24 smithi027 ceph-mon[39795]: 5.1 scrub ok 2024-05-15T13:04:24.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:24 smithi027 ceph-mon[39795]: 5.3 scrub ok 2024-05-15T13:04:24.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:24 smithi027 ceph-mon[39795]: osdmap e444: 12 total, 12 up, 11 in 2024-05-15T13:04:24.529 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:24 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:04:24.529 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:24 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:04:24.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:04:24.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:24 smithi120 ceph-mon[43961]: 5.1 scrub ok 2024-05-15T13:04:24.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:24 smithi120 ceph-mon[43961]: 5.3 scrub ok 2024-05-15T13:04:24.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:24 smithi120 ceph-mon[43961]: osdmap e444: 12 total, 12 up, 11 in 2024-05-15T13:04:24.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:24 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:04:24.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:24 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:04:24.558 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":1182,"stamp":"2024-05-15T13:04:19.976709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6664552992,"num_objects":101686,"num_object_clones":0,"num_object_copies":305058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101686,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":101865,"num_write_kb":6510714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101823,"ondisk_log_size":101823,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":24595360,"kb_used_data":20369336,"kb_used_omap":17,"kb_used_meta":4225902,"kb_avail":1006601312,"statfs":{"total":1055945392128,"available":1030759743488,"internally_reserved":0,"allocated":20858200064,"data_stored":20852383963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17551,"internal_metadata":4327324529},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,2,8,10,8,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":701366272,"num_objects":10702,"num_object_clones":0,"num_object_copies":32106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10702,"num_write_kb":684928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.012157"},"pg_stats":[{"pgid":"5.f","version":"442'6329","reported_seq":6387,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-05-15T13:04:17.859969+0000","last_change":"2024-05-15T13:03:11.329088+0000","last_active":"2024-05-15T13:04:17.859969+0000","last_peered":"2024-05-15T13:04:17.859969+0000","last_clean":"2024-05-15T13:04:17.859969+0000","last_became_active":"2024-05-15T13:03:11.328783+0000","last_became_peered":"2024-05-15T13:03:11.328783+0000","last_unstale":"2024-05-15T13:04:17.859969+0000","last_undegraded":"2024-05-15T13:04:17.859969+0000","last_fullsized":"2024-05-15T13:04:17.859969+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:03:10.302515+0000","objects_scrubbed":0,"log_size":6329,"log_dups_size":0,"ondisk_log_size":6329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:04:40.000374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414777344,"num_objects":6329,"num_object_clones":0,"num_object_copies":18987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6329,"num_write_kb":405056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"442'6267","reported_seq":6325,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-05-15T13:04:15.577945+0000","last_change":"2024-05-15T13:03:11.330562+0000","last_active":"2024-05-15T13:04:15.577945+0000","last_peered":"2024-05-15T13:04:15.577945+0000","last_clean":"2024-05-15T13:04:15.577945+0000","last_became_active":"2024-05-15T13:03:11.330241+0000","last_became_peered":"2024-05-15T13:03:11.330241+0000","last_unstale":"2024-05-15T13:04:15.577945+0000","last_undegraded":"2024-05-15T13:04:15.577945+0000","last_fullsized":"2024-05-15T13:04:15.577945+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:03:10.302515+0000","objects_scrubbed":0,"log_size":6267,"log_dups_size":0,"ondisk_log_size":6267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:04:24.862302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410714112,"num_objects":6267,"num_object_clones":0,"num_object_copies":18801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6267,"num_write_kb":401088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"442'6401","reported_seq":6459,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-05-15T13:04:17.487200+0000","last_change":"2024-05-15T13:03:11.331257+0000","last_active":"2024-05-15T13:04:17.487200+0000","last_peered":"2024-05-15T13:04:17.487200+0000","last_clean":"2024-05-15T13:04:17.487200+0000","last_became_active":"2024-05-15T13:03:11.330919+0000","last_became_peered":"2024-05-15T13:03:11.330919+0000","last_unstale":"2024-05-15T13:04:17.487200+0000","last_undegraded":"2024-05-15T13:04:17.487200+0000","last_fullsized":"2024-05-15T13:04:17.487200+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:03:10.302515+0000","objects_scrubbed":0,"log_size":6401,"log_dups_size":0,"ondisk_log_size":6401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:04:24.740516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419495936,"num_objects":6401,"num_object_clones":0,"num_object_copies":19203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6401,"num_write_kb":409664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"442'6321","reported_seq":6379,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-05-15T13:04:16.359238+0000","last_change":"2024-05-15T13:03:11.327685+0000","last_active":"2024-05-15T13:04:16.359238+0000","last_peered":"2024-05-15T13:04:16.359238+0000","last_clean":"2024-05-15T13:04:16.359238+0000","last_became_active":"2024-05-15T13:03:11.327387+0000","last_became_peered":"2024-05-15T13:03:11.327387+0000","last_unstale":"2024-05-15T13:04:16.359238+0000","last_undegraded":"2024-05-15T13:04:16.359238+0000","last_fullsized":"2024-05-15T13:04:16.359238+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:03:10.302515+0000","objects_scrubbed":0,"log_size":6321,"log_dups_size":0,"ondisk_log_size":6321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:04:29.235177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414253056,"num_objects":6321,"num_object_clones":0,"num_object_copies":18963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6321,"num_write_kb":404544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"442'6407","reported_seq":6465,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-05-15T13:04:19.101957+0000","last_change":"2024-05-15T13:03:11.328886+0000","last_active":"2024-05-15T13:04:19.101957+0000","last_peered":"2024-05-15T13:04:19.101957+0000","last_clean":"2024-05-15T13:04:19.101957+0000","last_became_active":"2024-05-15T13:03:11.328691+0000","last_became_peered":"2024-05-15T13:03:11.328691+0000","last_unstale":"2024-05-15T13:04:19.101957+0000","last_undegraded":"2024-05-15T13:04:19.101957+0000","last_fullsized":"2024-05-15T13:04:19.101957+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:03:10.302515+0000","objects_scrubbed":0,"log_size":6407,"log_dups_size":0,"ondisk_log_size":6407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419889152,"num_objects":6407,"num_object_clones":0,"num_object_copies":19221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6407,"num_write_kb":410048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,5],"acting":[0,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"442'6324","reported_seq":6381,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-05-15T13:04:17.561800+0000","last_change":"2024-05-15T13:03:11.330764+0000","last_active":"2024-05-15T13:04:17.561800+0000","last_peered":"2024-05-15T13:04:17.561800+0000","last_clean":"2024-05-15T13:04:17.561800+0000","last_became_active":"2024-05-15T13:03:11.330425+0000","last_became_peered":"2024-05-15T13:03:11.330425+0000","last_unstale":"2024-05-15T13:04:17.561800+0000","last_undegraded":"2024-05-15T13:04:17.561800+0000","last_fullsized":"2024-05-15T13:04:17.561800+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:03:10.302515+0000","objects_scrubbed":0,"log_size":6324,"log_dups_size":0,"ondisk_log_size":6324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414449664,"num_objects":6324,"num_object_clones":0,"num_object_copies":18972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6324,"num_write_kb":404736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"442'6347","reported_seq":6405,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-05-15T13:04:18.660302+0000","last_change":"2024-05-15T13:03:11.329537+0000","last_active":"2024-05-15T13:04:18.660302+0000","last_peered":"2024-05-15T13:04:18.660302+0000","last_clean":"2024-05-15T13:04:18.660302+0000","last_became_active":"2024-05-15T13:03:11.329210+0000","last_became_peered":"2024-05-15T13:03:11.329210+0000","last_unstale":"2024-05-15T13:04:18.660302+0000","last_undegraded":"2024-05-15T13:04:18.660302+0000","last_fullsized":"2024-05-15T13:04:18.660302+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:03:10.302515+0000","objects_scrubbed":0,"log_size":6347,"log_dups_size":0,"ondisk_log_size":6347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:04:25.829042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415956992,"num_objects":6347,"num_object_clones":0,"num_object_copies":19041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6347,"num_write_kb":406208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,9],"acting":[6,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"442'6219","reported_seq":6277,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-05-15T13:04:17.463029+0000","last_change":"2024-05-15T13:03:11.331442+0000","last_active":"2024-05-15T13:04:17.463029+0000","last_peered":"2024-05-15T13:04:17.463029+0000","last_clean":"2024-05-15T13:04:17.463029+0000","last_became_active":"2024-05-15T13:03:11.330983+0000","last_became_peered":"2024-05-15T13:03:11.330983+0000","last_unstale":"2024-05-15T13:04:17.463029+0000","last_undegraded":"2024-05-15T13:04:17.463029+0000","last_fullsized":"2024-05-15T13:04:17.463029+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:03:10.302515+0000","objects_scrubbed":0,"log_size":6219,"log_dups_size":0,"ondisk_log_size":6219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:04:23.964784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407568384,"num_objects":6219,"num_object_clones":0,"num_object_copies":18657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6219,"num_write_kb":398016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"442'6374","reported_seq":6432,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-05-15T13:04:17.872316+0000","last_change":"2024-05-15T13:03:11.329777+0000","last_active":"2024-05-15T13:04:17.872316+0000","last_peered":"2024-05-15T13:04:17.872316+0000","last_clean":"2024-05-15T13:04:17.872316+0000","last_became_active":"2024-05-15T13:03:11.329484+0000","last_became_peered":"2024-05-15T13:03:11.329484+0000","last_unstale":"2024-05-15T13:04:17.872316+0000","last_undegraded":"2024-05-15T13:04:17.872316+0000","last_fullsized":"2024-05-15T13:04:17.872316+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:03:10.302515+0000","objects_scrubbed":0,"log_size":6374,"log_dups_size":0,"ondisk_log_size":6374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417726464,"num_objects":6374,"num_object_clones":0,"num_object_copies":19122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6374,"num_write_kb":407936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"442'6341","reported_seq":6419,"reported_epoch":442,"state":"active+clean+scrubbing","last_fresh":"2024-05-15T13:04:17.552691+0000","last_change":"2024-05-15T13:04:17.180739+0000","last_active":"2024-05-15T13:04:17.552691+0000","last_peered":"2024-05-15T13:04:17.552691+0000","last_clean":"2024-05-15T13:04:17.552691+0000","last_became_active":"2024-05-15T13:03:11.332923+0000","last_became_peered":"2024-05-15T13:03:11.332923+0000","last_unstale":"2024-05-15T13:04:17.552691+0000","last_undegraded":"2024-05-15T13:04:17.552691+0000","last_fullsized":"2024-05-15T13:04:17.552691+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:03:10.302515+0000","objects_scrubbed":1699,"log_size":6341,"log_dups_size":0,"ondisk_log_size":6341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415563776,"num_objects":6341,"num_object_clones":0,"num_object_copies":19023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6341,"num_write_kb":405824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,10],"acting":[2,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"442'6389","reported_seq":6447,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-05-15T13:04:19.166728+0000","last_change":"2024-05-15T13:03:11.333665+0000","last_active":"2024-05-15T13:04:19.166728+0000","last_peered":"2024-05-15T13:04:19.166728+0000","last_clean":"2024-05-15T13:04:19.166728+0000","last_became_active":"2024-05-15T13:03:11.333336+0000","last_became_peered":"2024-05-15T13:03:11.333336+0000","last_unstale":"2024-05-15T13:04:19.166728+0000","last_undegraded":"2024-05-15T13:04:19.166728+0000","last_fullsized":"2024-05-15T13:04:19.166728+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:03:10.302515+0000","objects_scrubbed":0,"log_size":6389,"log_dups_size":0,"ondisk_log_size":6389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418709504,"num_objects":6389,"num_object_clones":0,"num_object_copies":19167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6389,"num_write_kb":408896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"442'6493","reported_seq":6551,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-05-15T13:04:19.150777+0000","last_change":"2024-05-15T13:03:11.328126+0000","last_active":"2024-05-15T13:04:19.150777+0000","last_peered":"2024-05-15T13:04:19.150777+0000","last_clean":"2024-05-15T13:04:19.150777+0000","last_became_active":"2024-05-15T13:03:11.327788+0000","last_became_peered":"2024-05-15T13:03:11.327788+0000","last_unstale":"2024-05-15T13:04:19.150777+0000","last_undegraded":"2024-05-15T13:04:19.150777+0000","last_fullsized":"2024-05-15T13:04:19.150777+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:03:10.302515+0000","objects_scrubbed":0,"log_size":6493,"log_dups_size":0,"ondisk_log_size":6493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425525248,"num_objects":6493,"num_object_clones":0,"num_object_copies":19479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6493,"num_write_kb":415552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"442'6402","reported_seq":6460,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-05-15T13:04:18.660700+0000","last_change":"2024-05-15T13:03:11.330588+0000","last_active":"2024-05-15T13:04:18.660700+0000","last_peered":"2024-05-15T13:04:18.660700+0000","last_clean":"2024-05-15T13:04:18.660700+0000","last_became_active":"2024-05-15T13:03:11.330316+0000","last_became_peered":"2024-05-15T13:03:11.330316+0000","last_unstale":"2024-05-15T13:04:18.660700+0000","last_undegraded":"2024-05-15T13:04:18.660700+0000","last_fullsized":"2024-05-15T13:04:18.660700+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:03:10.302515+0000","objects_scrubbed":0,"log_size":6402,"log_dups_size":0,"ondisk_log_size":6402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:04:23.682087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419561472,"num_objects":6402,"num_object_clones":0,"num_object_copies":19206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6402,"num_write_kb":409728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"442'6486","reported_seq":6544,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-05-15T13:04:19.873787+0000","last_change":"2024-05-15T13:03:11.431263+0000","last_active":"2024-05-15T13:04:19.873787+0000","last_peered":"2024-05-15T13:04:19.873787+0000","last_clean":"2024-05-15T13:04:19.873787+0000","last_became_active":"2024-05-15T13:03:11.430987+0000","last_became_peered":"2024-05-15T13:03:11.430987+0000","last_unstale":"2024-05-15T13:04:19.873787+0000","last_undegraded":"2024-05-15T13:04:19.873787+0000","last_fullsized":"2024-05-15T13:04:19.873787+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:03:10.302515+0000","objects_scrubbed":0,"log_size":6486,"log_dups_size":0,"ondisk_log_size":6486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425066496,"num_objects":6486,"num_object_clones":0,"num_object_copies":19458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6486,"num_write_kb":415104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"442'6295","reported_seq":6353,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-05-15T13:04:16.341539+0000","last_change":"2024-05-15T13:03:11.331154+0000","last_active":"2024-05-15T13:04:16.341539+0000","last_peered":"2024-05-15T13:04:16.341539+0000","last_clean":"2024-05-15T13:04:16.341539+0000","last_became_active":"2024-05-15T13:03:11.330479+0000","last_became_peered":"2024-05-15T13:03:11.330479+0000","last_unstale":"2024-05-15T13:04:16.341539+0000","last_undegraded":"2024-05-15T13:04:16.341539+0000","last_fullsized":"2024-05-15T13:04:16.341539+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:03:10.302515+0000","objects_scrubbed":0,"log_size":6295,"log_dups_size":0,"ondisk_log_size":6295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:04:31.595397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412549120,"num_objects":6295,"num_object_clones":0,"num_object_copies":18885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6295,"num_write_kb":402880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,4],"acting":[1,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"442'6289","reported_seq":6347,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-05-15T13:04:16.317981+0000","last_change":"2024-05-15T13:03:11.331588+0000","last_active":"2024-05-15T13:04:16.317981+0000","last_peered":"2024-05-15T13:04:16.317981+0000","last_clean":"2024-05-15T13:04:16.317981+0000","last_became_active":"2024-05-15T13:03:11.331291+0000","last_became_peered":"2024-05-15T13:03:11.331291+0000","last_unstale":"2024-05-15T13:04:16.317981+0000","last_undegraded":"2024-05-15T13:04:16.317981+0000","last_fullsized":"2024-05-15T13:04:16.317981+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:03:10.302515+0000","objects_scrubbed":0,"log_size":6289,"log_dups_size":0,"ondisk_log_size":6289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:04:19.774084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412155904,"num_objects":6289,"num_object_clones":0,"num_object_copies":18867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6289,"num_write_kb":402496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":702,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-05-15T13:04:14.689503+0000","last_change":"2024-05-15T13:03:21.248322+0000","last_active":"2024-05-15T13:04:14.689503+0000","last_peered":"2024-05-15T13:04:14.689503+0000","last_clean":"2024-05-15T13:04:14.689503+0000","last_became_active":"2024-05-15T13:00:26.045412+0000","last_became_peered":"2024-05-15T13:00:26.045412+0000","last_unstale":"2024-05-15T13:04:14.689503+0000","last_undegraded":"2024-05-15T13:04:14.689503+0000","last_fullsized":"2024-05-15T13:04:14.689503+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:03:21.248170+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:03:21.248170+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:04:38.165247+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":6663962624,"num_objects":101684,"num_object_clones":0,"num_object_copies":305052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101684,"num_write_kb":6507776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19938803712,"data_stored":19938803712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101684,"ondisk_log_size":101684,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056866,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1927948,"kb_used_data":1603844,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":91817204,"statfs":{"total":95995035648,"available":94020816896,"internally_reserved":0,"allocated":1642336256,"data_stored":1641808064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":331872710},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351206,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3043604,"kb_used_data":2647308,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":90701548,"statfs":{"total":95995035648,"available":92878385152,"internally_reserved":0,"allocated":2710843392,"data_stored":2710311648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":405797312},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514732,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2534548,"kb_used_data":2092420,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":91210604,"statfs":{"total":95995035648,"available":93399658496,"internally_reserved":0,"allocated":2142638080,"data_stored":2142109888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":452721088},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874481,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1300100,"kb_used_data":1034296,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":92445052,"statfs":{"total":95995035648,"available":94663733248,"internally_reserved":0,"allocated":1059119104,"data_stored":1058595439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":240518168820,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430458,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1847444,"kb_used_data":1473356,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":91897708,"statfs":{"total":95995035648,"available":94103252992,"internally_reserved":0,"allocated":1508716544,"data_stored":1508184820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":383056326},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575125,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2097752,"kb_used_data":1593164,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":91647400,"statfs":{"total":95995035648,"available":93846937600,"internally_reserved":0,"allocated":1631399936,"data_stored":1630868212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738654,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3741772,"kb_used_data":3299652,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":90003380,"statfs":{"total":95995035648,"available":92163461120,"internally_reserved":0,"allocated":3378843648,"data_stored":3378315476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":452721093},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411602,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1574668,"kb_used_data":1304708,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92170484,"statfs":{"total":95995035648,"available":94382575616,"internally_reserved":0,"allocated":1336020992,"data_stored":1335492820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149963,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":835212,"kb_used_data":402820,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":92909940,"statfs":{"total":95995035648,"available":95139778560,"internally_reserved":0,"allocated":412487680,"data_stored":411959508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":442759623},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986439,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2478284,"kb_used_data":2063940,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":91266868,"statfs":{"total":95995035648,"available":93457272832,"internally_reserved":0,"allocated":2113474560,"data_stored":2112946388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692096,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3214028,"kb_used_data":2853828,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":90531124,"statfs":{"total":95995035648,"available":92703870976,"internally_reserved":0,"allocated":2922319872,"data_stored":2921791700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3381067776,"data_stored":3381067776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1642201088,"data_stored":1642201088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1248198656,"data_stored":1248198656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":411172864,"data_stored":411172864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2131886080,"data_stored":2131886080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2922381312,"data_stored":2922381312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1258291200,"data_stored":1258291200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":811401216,"data_stored":811401216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2027094016,"data_stored":2027094016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2454061056,"data_stored":2454061056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1651048448,"data_stored":1651048448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:04:24.560 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:04:24.630 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:04:24.663 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:24.690 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:04:24.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:24.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:04:24.795 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:04:24.835 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:24.865 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:04:24.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:24.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:04:24.973 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 64 16 109066 109050 106.457 58.9375 0.0187956 0.00938913 2024-05-15T13:04:24.973 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:04:25.024 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:25.342 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:25 smithi177 ceph-mon[44303]: pgmap v1186: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.4 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-05-15T13:04:25.343 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:25 smithi177 ceph-mon[44303]: 5.b scrub starts 2024-05-15T13:04:25.343 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:25 smithi177 ceph-mon[44303]: osdmap e445: 12 total, 12 up, 11 in 2024-05-15T13:04:25.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:25 smithi027 ceph-mon[39795]: pgmap v1186: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.4 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-05-15T13:04:25.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:25 smithi027 ceph-mon[39795]: 5.b scrub starts 2024-05-15T13:04:25.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:25 smithi027 ceph-mon[39795]: osdmap e445: 12 total, 12 up, 11 in 2024-05-15T13:04:25.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:25 smithi120 ceph-mon[43961]: pgmap v1186: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.4 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-05-15T13:04:25.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:25 smithi120 ceph-mon[43961]: 5.b scrub starts 2024-05-15T13:04:25.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:25 smithi120 ceph-mon[43961]: osdmap e445: 12 total, 12 up, 11 in 2024-05-15T13:04:25.724 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:04:25.724 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-05-15T13:04:25.724 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-05-15T13:04:25.724 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T13:04:25.724 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.724 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:9156 5.4 5:2f29b624:::benchmark_data_smithi177_109991_object9155:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:04:25.725 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:23.466337+0000", 2024-05-15T13:04:25.725 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 62.265023116999998, 2024-05-15T13:04:25.725 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.060024363999999997, 2024-05-15T13:04:25.725 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:25.725 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:25.725 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:25.725 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:04:25.725 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:04:25.725 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 9156 2024-05-15T13:04:25.725 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.725 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:25.725 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.725 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:25.725 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:23.466337+0000", 2024-05-15T13:04:25.726 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.726 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.726 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.726 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:25.726 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:23.466337+0000", 2024-05-15T13:04:25.726 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.726 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.726 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.726 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:25.726 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:23.466338+0000", 2024-05-15T13:04:25.726 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5770000000000001e-06 2024-05-15T13:04:25.726 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.726 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.727 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:25.727 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:23.466359+0000", 2024-05-15T13:04:25.727 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0483999999999999e-05 2024-05-15T13:04:25.727 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.727 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.727 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:25.727 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:23.466372+0000", 2024-05-15T13:04:25.727 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3268000000000001e-05 2024-05-15T13:04:25.727 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.727 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.727 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:25.727 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:23.466381+0000", 2024-05-15T13:04:25.727 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4989999999999994e-06 2024-05-15T13:04:25.728 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.728 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.728 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:25.728 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:23.466445+0000", 2024-05-15T13:04:25.728 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4389999999999998e-05 2024-05-15T13:04:25.728 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.728 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.728 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:25.728 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:23.466593+0000", 2024-05-15T13:04:25.728 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000147823 2024-05-15T13:04:25.728 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.728 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.728 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:04:25.728 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:23.466786+0000", 2024-05-15T13:04:25.729 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00019276499999999999 2024-05-15T13:04:25.729 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.729 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.729 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:25.729 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:23.467434+0000", 2024-05-15T13:04:25.729 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00064865699999999997 2024-05-15T13:04:25.729 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.729 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.729 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.729 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:23.469032+0000", 2024-05-15T13:04:25.729 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015973859999999999 2024-05-15T13:04:25.729 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.729 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.729 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.730 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:23.526104+0000", 2024-05-15T13:04:25.730 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.057072622000000003 2024-05-15T13:04:25.730 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.730 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.730 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:25.730 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:23.526303+0000", 2024-05-15T13:04:25.730 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00019822199999999999 2024-05-15T13:04:25.730 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.730 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.730 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:25.730 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:23.526361+0000", 2024-05-15T13:04:25.730 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8671000000000001e-05 2024-05-15T13:04:25.731 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.731 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:25.731 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.731 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.731 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.731 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:19088 5.4 5:2a33d050:::benchmark_data_smithi177_109991_object19087:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:04:25.731 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:27.083242+0000", 2024-05-15T13:04:25.731 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 58.648118484999998, 2024-05-15T13:04:25.731 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.092245548999999996, 2024-05-15T13:04:25.731 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:25.731 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:25.731 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:25.731 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:04:25.731 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:04:25.732 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 19088 2024-05-15T13:04:25.732 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.732 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:25.732 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.732 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:25.732 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.083242+0000", 2024-05-15T13:04:25.732 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.732 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.733 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.733 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:25.733 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.083242+0000", 2024-05-15T13:04:25.733 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.733 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.733 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.733 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:25.733 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.083242+0000", 2024-05-15T13:04:25.733 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3e-07 2024-05-15T13:04:25.733 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.733 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.733 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:25.733 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.083270+0000", 2024-05-15T13:04:25.734 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7640999999999998e-05 2024-05-15T13:04:25.734 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.734 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.734 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:25.734 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.083279+0000", 2024-05-15T13:04:25.734 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8200000000000003e-06 2024-05-15T13:04:25.734 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.734 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.734 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:25.734 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.083283+0000", 2024-05-15T13:04:25.734 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.442e-06 2024-05-15T13:04:25.734 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.734 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.734 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:25.735 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.083332+0000", 2024-05-15T13:04:25.735 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9081e-05 2024-05-15T13:04:25.735 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.735 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.735 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:25.735 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.083459+0000", 2024-05-15T13:04:25.735 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012679400000000001 2024-05-15T13:04:25.735 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.735 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.735 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:04:25.735 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.083623+0000", 2024-05-15T13:04:25.735 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000163917 2024-05-15T13:04:25.735 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.736 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.736 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:25.736 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.085789+0000", 2024-05-15T13:04:25.736 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021660130000000001 2024-05-15T13:04:25.736 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.736 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.736 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.736 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.087340+0000", 2024-05-15T13:04:25.736 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015510140000000001 2024-05-15T13:04:25.736 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.736 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.736 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.736 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175382+0000", 2024-05-15T13:04:25.736 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.088042001999999994 2024-05-15T13:04:25.737 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.737 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.737 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:25.737 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175442+0000", 2024-05-15T13:04:25.737 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9800999999999999e-05 2024-05-15T13:04:25.737 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.737 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.737 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:25.737 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175487+0000", 2024-05-15T13:04:25.737 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5293999999999997e-05 2024-05-15T13:04:25.737 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.737 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:25.737 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.737 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.738 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.738 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:19089 5.4 5:2f920fea:::benchmark_data_smithi177_109991_object19088:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:04:25.738 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:27.084059+0000", 2024-05-15T13:04:25.738 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 58.647301165000002, 2024-05-15T13:04:25.738 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.091531791000000001, 2024-05-15T13:04:25.738 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:25.738 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:25.738 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:25.738 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:04:25.738 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:04:25.739 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 19089 2024-05-15T13:04:25.739 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.739 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:25.739 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.739 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:25.739 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084059+0000", 2024-05-15T13:04:25.739 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.739 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.739 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.739 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:25.739 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084059+0000", 2024-05-15T13:04:25.739 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.740 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.740 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.740 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:25.740 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084060+0000", 2024-05-15T13:04:25.740 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0999999999999998e-07 2024-05-15T13:04:25.740 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.740 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.740 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:25.740 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084079+0000", 2024-05-15T13:04:25.740 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9014000000000001e-05 2024-05-15T13:04:25.740 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.740 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.740 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:25.740 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084088+0000", 2024-05-15T13:04:25.741 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5950000000000005e-06 2024-05-15T13:04:25.741 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.741 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:25.741 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084094+0000", 2024-05-15T13:04:25.741 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9009999999999999e-06 2024-05-15T13:04:25.741 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.741 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:25.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084276+0000", 2024-05-15T13:04:25.742 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000181795 2024-05-15T13:04:25.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:25.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084402+0000", 2024-05-15T13:04:25.742 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012629699999999999 2024-05-15T13:04:25.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:04:25.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084542+0000", 2024-05-15T13:04:25.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013983999999999999 2024-05-15T13:04:25.743 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.743 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.743 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:25.743 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.085797+0000", 2024-05-15T13:04:25.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012547350000000001 2024-05-15T13:04:25.743 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.743 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.743 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.743 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.087410+0000", 2024-05-15T13:04:25.744 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016136309999999999 2024-05-15T13:04:25.744 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.744 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175516+0000", 2024-05-15T13:04:25.744 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.088105476000000002 2024-05-15T13:04:25.744 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:25.744 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175554+0000", 2024-05-15T13:04:25.744 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8308e-05 2024-05-15T13:04:25.744 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:25.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175591+0000", 2024-05-15T13:04:25.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6489000000000002e-05 2024-05-15T13:04:25.745 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.745 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:25.745 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.745 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:19091 5.4 5:23c909f8:::benchmark_data_smithi177_109991_object19090:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:04:25.745 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:27.084143+0000", 2024-05-15T13:04:25.745 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 58.647216780999997, 2024-05-15T13:04:25.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.091549106000000005, 2024-05-15T13:04:25.745 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:25.745 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:25.745 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:25.746 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:04:25.746 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:04:25.746 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 19091 2024-05-15T13:04:25.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.746 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:25.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:25.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084143+0000", 2024-05-15T13:04:25.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:25.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084143+0000", 2024-05-15T13:04:25.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.747 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.747 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:25.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084144+0000", 2024-05-15T13:04:25.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5599999999999995e-07 2024-05-15T13:04:25.747 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.747 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:25.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084162+0000", 2024-05-15T13:04:25.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8422000000000001e-05 2024-05-15T13:04:25.747 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:25.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084170+0000", 2024-05-15T13:04:25.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2770000000000005e-06 2024-05-15T13:04:25.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:25.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084176+0000", 2024-05-15T13:04:25.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4020000000000001e-06 2024-05-15T13:04:25.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:25.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.085231+0000", 2024-05-15T13:04:25.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001055432 2024-05-15T13:04:25.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:25.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.085311+0000", 2024-05-15T13:04:25.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9831000000000002e-05 2024-05-15T13:04:25.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:04:25.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.085390+0000", 2024-05-15T13:04:25.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8793000000000003e-05 2024-05-15T13:04:25.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:25.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.086012+0000", 2024-05-15T13:04:25.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00062231899999999998 2024-05-15T13:04:25.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.087445+0000", 2024-05-15T13:04:25.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001432675 2024-05-15T13:04:25.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175628+0000", 2024-05-15T13:04:25.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.088183239999999996 2024-05-15T13:04:25.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:25.751 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175658+0000", 2024-05-15T13:04:25.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9607e-05 2024-05-15T13:04:25.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.751 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:25.751 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175692+0000", 2024-05-15T13:04:25.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4551999999999997e-05 2024-05-15T13:04:25.751 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.751 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:25.751 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.751 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:19097 5.4 5:2620206e:::benchmark_data_smithi177_109991_object19096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:04:25.751 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:27.086354+0000", 2024-05-15T13:04:25.752 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 58.645005796, 2024-05-15T13:04:25.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.089473733, 2024-05-15T13:04:25.752 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:25.752 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:25.752 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:25.752 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:04:25.752 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:04:25.752 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 19097 2024-05-15T13:04:25.752 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.752 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:25.752 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.752 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:25.752 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.086354+0000", 2024-05-15T13:04:25.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.752 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:25.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.086354+0000", 2024-05-15T13:04:25.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:25.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.086355+0000", 2024-05-15T13:04:25.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.018e-06 2024-05-15T13:04:25.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:25.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.086377+0000", 2024-05-15T13:04:25.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1515999999999999e-05 2024-05-15T13:04:25.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:25.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.086390+0000", 2024-05-15T13:04:25.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2984e-05 2024-05-15T13:04:25.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:25.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.086398+0000", 2024-05-15T13:04:25.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2260000000000005e-06 2024-05-15T13:04:25.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.755 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:25.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.086460+0000", 2024-05-15T13:04:25.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1606999999999995e-05 2024-05-15T13:04:25.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.755 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:25.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.086603+0000", 2024-05-15T13:04:25.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014295099999999999 2024-05-15T13:04:25.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.756 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:04:25.756 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.086731+0000", 2024-05-15T13:04:25.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012876599999999999 2024-05-15T13:04:25.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.756 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:25.756 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.088423+0000", 2024-05-15T13:04:25.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001691407 2024-05-15T13:04:25.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.088547+0000", 2024-05-15T13:04:25.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000124175 2024-05-15T13:04:25.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175735+0000", 2024-05-15T13:04:25.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.087188584999999999 2024-05-15T13:04:25.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:25.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175791+0000", 2024-05-15T13:04:25.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5371000000000002e-05 2024-05-15T13:04:25.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:25.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175828+0000", 2024-05-15T13:04:25.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7126999999999999e-05 2024-05-15T13:04:25.758 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.758 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:25.758 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.758 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:19105 5.4 5:2c08e1d6:::benchmark_data_smithi177_109991_object19104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:04:25.758 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:27.090769+0000", 2024-05-15T13:04:25.758 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 58.640591161000003, 2024-05-15T13:04:25.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.085190499000000003, 2024-05-15T13:04:25.759 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:25.759 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:25.759 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:25.759 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:04:25.759 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:04:25.759 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 19105 2024-05-15T13:04:25.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.759 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:25.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.759 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:25.759 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.090769+0000", 2024-05-15T13:04:25.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:25.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.090769+0000", 2024-05-15T13:04:25.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:25.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.090770+0000", 2024-05-15T13:04:25.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4980000000000001e-06 2024-05-15T13:04:25.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:25.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.090793+0000", 2024-05-15T13:04:25.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2254000000000001e-05 2024-05-15T13:04:25.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:25.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.090804+0000", 2024-05-15T13:04:25.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0939e-05 2024-05-15T13:04:25.761 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:25.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.090820+0000", 2024-05-15T13:04:25.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6371e-05 2024-05-15T13:04:25.761 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:25.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.090864+0000", 2024-05-15T13:04:25.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4508000000000001e-05 2024-05-15T13:04:25.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.762 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:25.762 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.090964+0000", 2024-05-15T13:04:25.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9105999999999996e-05 2024-05-15T13:04:25.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.762 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:04:25.762 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.091051+0000", 2024-05-15T13:04:25.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7614999999999994e-05 2024-05-15T13:04:25.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.763 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:25.763 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.091803+0000", 2024-05-15T13:04:25.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00075175199999999995 2024-05-15T13:04:25.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.763 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.763 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.092303+0000", 2024-05-15T13:04:25.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00050048699999999998 2024-05-15T13:04:25.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.763 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.763 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175869+0000", 2024-05-15T13:04:25.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.083565523000000003 2024-05-15T13:04:25.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:25.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175922+0000", 2024-05-15T13:04:25.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2966000000000001e-05 2024-05-15T13:04:25.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:25.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175959+0000", 2024-05-15T13:04:25.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.748e-05 2024-05-15T13:04:25.765 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.765 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:25.765 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.765 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:20882 5.4 5:2045f33d:::benchmark_data_smithi177_109991_object20881:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:04:25.765 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:27.874815+0000", 2024-05-15T13:04:25.765 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 57.856544518, 2024-05-15T13:04:25.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.078067048, 2024-05-15T13:04:25.765 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:25.765 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:25.765 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:25.765 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:04:25.766 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:04:25.766 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 20882 2024-05-15T13:04:25.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.766 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:25.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:25.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.874815+0000", 2024-05-15T13:04:25.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:25.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.874815+0000", 2024-05-15T13:04:25.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:25.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.874817+0000", 2024-05-15T13:04:25.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.311e-06 2024-05-15T13:04:25.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:25.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.874842+0000", 2024-05-15T13:04:25.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4723e-05 2024-05-15T13:04:25.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:25.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.874853+0000", 2024-05-15T13:04:25.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1773999999999999e-05 2024-05-15T13:04:25.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:25.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.874861+0000", 2024-05-15T13:04:25.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0970000000000006e-06 2024-05-15T13:04:25.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:25.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.874913+0000", 2024-05-15T13:04:25.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1573000000000002e-05 2024-05-15T13:04:25.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:25.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875032+0000", 2024-05-15T13:04:25.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000118847 2024-05-15T13:04:25.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:04:25.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875161+0000", 2024-05-15T13:04:25.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012939099999999999 2024-05-15T13:04:25.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952687+0000", 2024-05-15T13:04:25.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.077526088000000007 2024-05-15T13:04:25.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:25.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952700+0000", 2024-05-15T13:04:25.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2442e-05 2024-05-15T13:04:25.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952731+0000", 2024-05-15T13:04:25.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1745999999999997e-05 2024-05-15T13:04:25.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:25.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952833+0000", 2024-05-15T13:04:25.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010121800000000001 2024-05-15T13:04:25.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.771 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:25.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952883+0000", 2024-05-15T13:04:25.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9838e-05 2024-05-15T13:04:25.771 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.771 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:25.771 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.771 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:20879 5.2 e424/418 5:45079427:::benchmark_data_smithi177_109991_object20878:head v 424'1264, mlcod=424'1262)", 2024-05-15T13:04:25.772 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:27.875224+0000", 2024-05-15T13:04:25.772 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 57.856136231000001, 2024-05-15T13:04:25.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.077611594000000006, 2024-05-15T13:04:25.772 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:25.772 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:25.772 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:25.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:25.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875224+0000", 2024-05-15T13:04:25.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:25.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875224+0000", 2024-05-15T13:04:25.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.773 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:25.773 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875224+0000", 2024-05-15T13:04:25.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9699999999999995e-07 2024-05-15T13:04:25.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.773 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:25.773 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875340+0000", 2024-05-15T13:04:25.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115752 2024-05-15T13:04:25.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.773 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:25.773 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875351+0000", 2024-05-15T13:04:25.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0864e-05 2024-05-15T13:04:25.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:25.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875358+0000", 2024-05-15T13:04:25.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7360000000000004e-06 2024-05-15T13:04:25.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:25.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875417+0000", 2024-05-15T13:04:25.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9159000000000001e-05 2024-05-15T13:04:25.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:25.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875439+0000", 2024-05-15T13:04:25.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2286999999999999e-05 2024-05-15T13:04:25.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:25.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952607+0000", 2024-05-15T13:04:25.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.077167432999999994 2024-05-15T13:04:25.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:25.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952835+0000", 2024-05-15T13:04:25.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022866600000000001 2024-05-15T13:04:25.776 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.776 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:25.776 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.776 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:20880 5.a e424/418 5:56abf9c6:::benchmark_data_smithi177_109991_object20879:head v 424'1300, mlcod=424'1297)", 2024-05-15T13:04:25.776 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:27.875370+0000", 2024-05-15T13:04:25.776 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 57.855989596000001, 2024-05-15T13:04:25.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.079926545000000002, 2024-05-15T13:04:25.776 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:25.777 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:25.777 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:25.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:25.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875370+0000", 2024-05-15T13:04:25.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:25.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875370+0000", 2024-05-15T13:04:25.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:25.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875371+0000", 2024-05-15T13:04:25.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8999999999999996e-07 2024-05-15T13:04:25.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:25.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875539+0000", 2024-05-15T13:04:25.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016774100000000001 2024-05-15T13:04:25.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:25.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875549+0000", 2024-05-15T13:04:25.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0366e-05 2024-05-15T13:04:25.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:25.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875556+0000", 2024-05-15T13:04:25.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8630000000000002e-06 2024-05-15T13:04:25.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:25.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875634+0000", 2024-05-15T13:04:25.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8335000000000001e-05 2024-05-15T13:04:25.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:25.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875654+0000", 2024-05-15T13:04:25.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9457e-05 2024-05-15T13:04:25.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:25.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.955250+0000", 2024-05-15T13:04:25.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.079595770999999996 2024-05-15T13:04:25.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:25.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.955297+0000", 2024-05-15T13:04:25.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7422000000000001e-05 2024-05-15T13:04:25.780 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.780 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:25.780 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.781 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.15540.0:20882 5.4 e424/418 ondisk, result = 0)", 2024-05-15T13:04:25.781 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:27.876548+0000", 2024-05-15T13:04:25.781 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 57.854811523999999, 2024-05-15T13:04:25.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.076339893000000006, 2024-05-15T13:04:25.781 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:25.781 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T13:04:25.781 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:25.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:25.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.876548+0000", 2024-05-15T13:04:25.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:25.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.876548+0000", 2024-05-15T13:04:25.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:25.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.876549+0000", 2024-05-15T13:04:25.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3099999999999997e-07 2024-05-15T13:04:25.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:25.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.876554+0000", 2024-05-15T13:04:25.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.741e-06 2024-05-15T13:04:25.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:25.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952640+0000", 2024-05-15T13:04:25.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.076086283000000005 2024-05-15T13:04:25.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:25.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952647+0000", 2024-05-15T13:04:25.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9160000000000004e-06 2024-05-15T13:04:25.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:25.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952721+0000", 2024-05-15T13:04:25.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3801999999999996e-05 2024-05-15T13:04:25.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:25.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952727+0000", 2024-05-15T13:04:25.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2049999999999999e-06 2024-05-15T13:04:25.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:25.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952888+0000", 2024-05-15T13:04:25.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016131500000000001 2024-05-15T13:04:25.785 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.785 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:25.785 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.785 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.15540.0:20882 5.4 e424/418 ondisk, result = 0)", 2024-05-15T13:04:25.785 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:27.877367+0000", 2024-05-15T13:04:25.785 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 57.853993131000003, 2024-05-15T13:04:25.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.075333856000000005, 2024-05-15T13:04:25.786 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:25.786 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T13:04:25.786 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:25.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:25.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.877367+0000", 2024-05-15T13:04:25.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:25.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.877367+0000", 2024-05-15T13:04:25.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:25.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.877368+0000", 2024-05-15T13:04:25.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6499999999999999e-07 2024-05-15T13:04:25.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:25.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.877373+0000", 2024-05-15T13:04:25.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2650000000000004e-06 2024-05-15T13:04:25.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:25.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952602+0000", 2024-05-15T13:04:25.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.075229277999999997 2024-05-15T13:04:25.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:25.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952615+0000", 2024-05-15T13:04:25.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2619000000000001e-05 2024-05-15T13:04:25.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:25.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952672+0000", 2024-05-15T13:04:25.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6830999999999998e-05 2024-05-15T13:04:25.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:25.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952681+0000", 2024-05-15T13:04:25.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7089999999999997e-06 2024-05-15T13:04:25.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:25.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952701+0000", 2024-05-15T13:04:25.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9488999999999999e-05 2024-05-15T13:04:25.789 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.789 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:25.789 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.789 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:20881 5.0 e424/418 5:0cebd2d6:::benchmark_data_smithi177_109991_object20880:head v 424'1281, mlcod=424'1278)", 2024-05-15T13:04:25.790 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:27.877430+0000", 2024-05-15T13:04:25.790 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 57.853930245000001, 2024-05-15T13:04:25.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.078308953000000001, 2024-05-15T13:04:25.790 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:25.790 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:25.790 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:25.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:25.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.877430+0000", 2024-05-15T13:04:25.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:25.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.877430+0000", 2024-05-15T13:04:25.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:25.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.877430+0000", 2024-05-15T13:04:25.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1099999999999995e-07 2024-05-15T13:04:25.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:25.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.877494+0000", 2024-05-15T13:04:25.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3373000000000005e-05 2024-05-15T13:04:25.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:25.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952607+0000", 2024-05-15T13:04:25.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.075113048000000002 2024-05-15T13:04:25.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:25.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952616+0000", 2024-05-15T13:04:25.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5540000000000003e-06 2024-05-15T13:04:25.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:25.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952671+0000", 2024-05-15T13:04:25.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4685999999999998e-05 2024-05-15T13:04:25.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:25.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952694+0000", 2024-05-15T13:04:25.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3263000000000001e-05 2024-05-15T13:04:25.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:25.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.955687+0000", 2024-05-15T13:04:25.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0029930960000000002 2024-05-15T13:04:25.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:25.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.955739+0000", 2024-05-15T13:04:25.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1221999999999998e-05 2024-05-15T13:04:25.794 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.794 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:25.794 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.794 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:23470 5.4 5:222094c6:::benchmark_data_smithi177_109991_object23469:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:04:25.794 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:29.032826+0000", 2024-05-15T13:04:25.794 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 56.698533714, 2024-05-15T13:04:25.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.070901916999999995, 2024-05-15T13:04:25.794 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:25.795 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:25.795 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:25.795 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:04:25.795 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:04:25.795 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 23470 2024-05-15T13:04:25.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.795 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:25.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:25.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.032826+0000", 2024-05-15T13:04:25.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:25.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.032826+0000", 2024-05-15T13:04:25.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:25.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.032828+0000", 2024-05-15T13:04:25.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3960000000000001e-06 2024-05-15T13:04:25.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:25.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.032847+0000", 2024-05-15T13:04:25.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9772e-05 2024-05-15T13:04:25.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:25.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.032860+0000", 2024-05-15T13:04:25.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2068e-05 2024-05-15T13:04:25.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:25.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.032865+0000", 2024-05-15T13:04:25.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7769999999999999e-06 2024-05-15T13:04:25.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:25.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.032916+0000", 2024-05-15T13:04:25.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0983999999999998e-05 2024-05-15T13:04:25.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:25.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.033027+0000", 2024-05-15T13:04:25.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111037 2024-05-15T13:04:25.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:04:25.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.033141+0000", 2024-05-15T13:04:25.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000113807 2024-05-15T13:04:25.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:25.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.034104+0000", 2024-05-15T13:04:25.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00096271699999999998 2024-05-15T13:04:25.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.034795+0000", 2024-05-15T13:04:25.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00069082200000000001 2024-05-15T13:04:25.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.103637+0000", 2024-05-15T13:04:25.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.068842293999999998 2024-05-15T13:04:25.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:25.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.103681+0000", 2024-05-15T13:04:25.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3886e-05 2024-05-15T13:04:25.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:25.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.103728+0000", 2024-05-15T13:04:25.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7357000000000001e-05 2024-05-15T13:04:25.801 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.801 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:25.801 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.801 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:23478 5.4 5:2fa10f30:::benchmark_data_smithi177_109991_object23477:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:04:25.801 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:29.037512+0000", 2024-05-15T13:04:25.801 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 56.693848166000002, 2024-05-15T13:04:25.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.066378486, 2024-05-15T13:04:25.802 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:25.802 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:25.802 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:25.802 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:04:25.802 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:04:25.802 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 23478 2024-05-15T13:04:25.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.802 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:25.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:25.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.037512+0000", 2024-05-15T13:04:25.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:25.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.037512+0000", 2024-05-15T13:04:25.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:25.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.037513+0000", 2024-05-15T13:04:25.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3030000000000001e-06 2024-05-15T13:04:25.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:25.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.037534+0000", 2024-05-15T13:04:25.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0939000000000001e-05 2024-05-15T13:04:25.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:25.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.037545+0000", 2024-05-15T13:04:25.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1175000000000001e-05 2024-05-15T13:04:25.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:25.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.037551+0000", 2024-05-15T13:04:25.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7660000000000004e-06 2024-05-15T13:04:25.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:25.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.037613+0000", 2024-05-15T13:04:25.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2362999999999999e-05 2024-05-15T13:04:25.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:25.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.037749+0000", 2024-05-15T13:04:25.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013542299999999999 2024-05-15T13:04:25.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:04:25.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.037943+0000", 2024-05-15T13:04:25.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00019399099999999999 2024-05-15T13:04:25.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:25.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.039000+0000", 2024-05-15T13:04:25.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010574720000000001 2024-05-15T13:04:25.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.039500+0000", 2024-05-15T13:04:25.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00049945100000000004 2024-05-15T13:04:25.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.103783+0000", 2024-05-15T13:04:25.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.064283062000000002 2024-05-15T13:04:25.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:25.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.103849+0000", 2024-05-15T13:04:25.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5798999999999998e-05 2024-05-15T13:04:25.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:25.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.103890+0000", 2024-05-15T13:04:25.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1742000000000001e-05 2024-05-15T13:04:25.810 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.810 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:25.810 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.811 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:36490 5.4 5:2efcba10:::benchmark_data_smithi177_109991_object36489:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-05-15T13:04:25.811 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:34.237835+0000", 2024-05-15T13:04:25.811 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 51.493525421000001, 2024-05-15T13:04:25.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.060179849000000001, 2024-05-15T13:04:25.811 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:25.811 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:25.811 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:25.811 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:04:25.811 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:04:25.811 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 36490 2024-05-15T13:04:25.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.811 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:25.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:25.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:34.237835+0000", 2024-05-15T13:04:25.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:25.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:34.237835+0000", 2024-05-15T13:04:25.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:25.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:34.237836+0000", 2024-05-15T13:04:25.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.029e-06 2024-05-15T13:04:25.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:25.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:34.237857+0000", 2024-05-15T13:04:25.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1212e-05 2024-05-15T13:04:25.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:25.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:34.237868+0000", 2024-05-15T13:04:25.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1438e-05 2024-05-15T13:04:25.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:25.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:34.237875+0000", 2024-05-15T13:04:25.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1080000000000004e-06 2024-05-15T13:04:25.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:25.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:34.238239+0000", 2024-05-15T13:04:25.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00036407099999999999 2024-05-15T13:04:25.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:25.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:34.238380+0000", 2024-05-15T13:04:25.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014063699999999999 2024-05-15T13:04:25.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:04:25.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:34.238517+0000", 2024-05-15T13:04:25.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013689799999999999 2024-05-15T13:04:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:34.239989+0000", 2024-05-15T13:04:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001471621 2024-05-15T13:04:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:34.242076+0000", 2024-05-15T13:04:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020869420000000001 2024-05-15T13:04:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:34.297901+0000", 2024-05-15T13:04:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.055825795999999997 2024-05-15T13:04:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:34.297970+0000", 2024-05-15T13:04:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8967999999999994e-05 2024-05-15T13:04:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:34.298014+0000", 2024-05-15T13:04:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4128999999999997e-05 2024-05-15T13:04:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:72586 5.4 5:2111a196:::benchmark_data_smithi177_109991_object72585:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-05-15T13:04:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:55.531803+0000", 2024-05-15T13:04:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 30.199556671, 2024-05-15T13:04:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.085809504999999994, 2024-05-15T13:04:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:04:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:04:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 72586 2024-05-15T13:04:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.531803+0000", 2024-05-15T13:04:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:25.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.531803+0000", 2024-05-15T13:04:25.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:25.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.531805+0000", 2024-05-15T13:04:25.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9700000000000002e-06 2024-05-15T13:04:25.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.531837+0000", 2024-05-15T13:04:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2079999999999998e-05 2024-05-15T13:04:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.531851+0000", 2024-05-15T13:04:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3149999999999999e-05 2024-05-15T13:04:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.531859+0000", 2024-05-15T13:04:25.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0139999999999998e-06 2024-05-15T13:04:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.531903+0000", 2024-05-15T13:04:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4453000000000003e-05 2024-05-15T13:04:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.531990+0000", 2024-05-15T13:04:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6623000000000001e-05 2024-05-15T13:04:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:04:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.532076+0000", 2024-05-15T13:04:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6738999999999997e-05 2024-05-15T13:04:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.535817+0000", 2024-05-15T13:04:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003740686 2024-05-15T13:04:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.537104+0000", 2024-05-15T13:04:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001287414 2024-05-15T13:04:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.617520+0000", 2024-05-15T13:04:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.080415259000000003 2024-05-15T13:04:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.617559+0000", 2024-05-15T13:04:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8946000000000003e-05 2024-05-15T13:04:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:25.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.617613+0000", 2024-05-15T13:04:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4171e-05 2024-05-15T13:04:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:72608 5.4 5:23d4fbbe:::benchmark_data_smithi177_109991_object72607:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-05-15T13:04:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:55.557961+0000", 2024-05-15T13:04:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 30.173398761000001, 2024-05-15T13:04:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.060076653000000001, 2024-05-15T13:04:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:25.824 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:04:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:04:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 72608 2024-05-15T13:04:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.557961+0000", 2024-05-15T13:04:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.557961+0000", 2024-05-15T13:04:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.557963+0000", 2024-05-15T13:04:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.362e-06 2024-05-15T13:04:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.557983+0000", 2024-05-15T13:04:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9924000000000001e-05 2024-05-15T13:04:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.557995+0000", 2024-05-15T13:04:25.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.269e-05 2024-05-15T13:04:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.558002+0000", 2024-05-15T13:04:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4439999999999996e-06 2024-05-15T13:04:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.558047+0000", 2024-05-15T13:04:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5596000000000002e-05 2024-05-15T13:04:25.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:25.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.558144+0000", 2024-05-15T13:04:25.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6848999999999994e-05 2024-05-15T13:04:25.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:04:25.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.558239+0000", 2024-05-15T13:04:25.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5020999999999997e-05 2024-05-15T13:04:25.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.560081+0000", 2024-05-15T13:04:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0018413730000000001 2024-05-15T13:04:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.560654+0000", 2024-05-15T13:04:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00057392100000000002 2024-05-15T13:04:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.617942+0000", 2024-05-15T13:04:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.057287877000000001 2024-05-15T13:04:25.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:25.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.617999+0000", 2024-05-15T13:04:25.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6255000000000003e-05 2024-05-15T13:04:25.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:25.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.618038+0000", 2024-05-15T13:04:25.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9341000000000002e-05 2024-05-15T13:04:25.830 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.830 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:104783 5.4 5:2abd1f52:::benchmark_data_smithi177_109991_object104782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-05-15T13:04:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:04:20.671108+0000", 2024-05-15T13:04:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.0602517819999999, 2024-05-15T13:04:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.074323771999999996, 2024-05-15T13:04:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:04:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:04:25.831 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 104783 2024-05-15T13:04:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.671108+0000", 2024-05-15T13:04:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.671108+0000", 2024-05-15T13:04:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:25.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.671110+0000", 2024-05-15T13:04:25.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7549999999999999e-06 2024-05-15T13:04:25.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:25.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.671132+0000", 2024-05-15T13:04:25.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1642999999999999e-05 2024-05-15T13:04:25.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:25.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.671147+0000", 2024-05-15T13:04:25.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5322e-05 2024-05-15T13:04:25.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.671156+0000", 2024-05-15T13:04:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8440000000000004e-06 2024-05-15T13:04:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.671244+0000", 2024-05-15T13:04:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7774000000000004e-05 2024-05-15T13:04:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.671363+0000", 2024-05-15T13:04:25.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000119726 2024-05-15T13:04:25.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:04:25.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.671489+0000", 2024-05-15T13:04:25.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000125831 2024-05-15T13:04:25.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:25.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.674533+0000", 2024-05-15T13:04:25.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0030441270000000002 2024-05-15T13:04:25.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.674718+0000", 2024-05-15T13:04:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000184945 2024-05-15T13:04:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.745334+0000", 2024-05-15T13:04:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.070615913000000002 2024-05-15T13:04:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.745373+0000", 2024-05-15T13:04:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8834000000000001e-05 2024-05-15T13:04:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:25.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.745432+0000", 2024-05-15T13:04:25.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9058000000000003e-05 2024-05-15T13:04:25.837 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.837 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:25.837 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.837 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:104792 5.4 5:2c256096:::benchmark_data_smithi177_109991_object104791:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-05-15T13:04:25.837 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:04:20.677560+0000", 2024-05-15T13:04:25.837 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.0538002730000002, 2024-05-15T13:04:25.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.068042884999999997, 2024-05-15T13:04:25.837 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:25.837 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:25.837 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:25.838 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:04:25.838 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:04:25.838 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 104792 2024-05-15T13:04:25.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.838 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:25.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:25.839 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.677560+0000", 2024-05-15T13:04:25.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.839 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:25.839 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.677560+0000", 2024-05-15T13:04:25.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.839 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:25.839 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.677561+0000", 2024-05-15T13:04:25.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6810000000000001e-06 2024-05-15T13:04:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.677590+0000", 2024-05-15T13:04:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8867999999999999e-05 2024-05-15T13:04:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.677603+0000", 2024-05-15T13:04:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2909e-05 2024-05-15T13:04:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:25.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.677612+0000", 2024-05-15T13:04:25.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6419999999999996e-06 2024-05-15T13:04:25.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:25.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.677655+0000", 2024-05-15T13:04:25.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3653999999999999e-05 2024-05-15T13:04:25.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:25.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.677754+0000", 2024-05-15T13:04:25.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8735999999999998e-05 2024-05-15T13:04:25.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:04:25.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.677866+0000", 2024-05-15T13:04:25.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111468 2024-05-15T13:04:25.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.681408+0000", 2024-05-15T13:04:25.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035424139999999998 2024-05-15T13:04:25.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.681666+0000", 2024-05-15T13:04:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025800299999999998 2024-05-15T13:04:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.745490+0000", 2024-05-15T13:04:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.063823823000000002 2024-05-15T13:04:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.745547+0000", 2024-05-15T13:04:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6758000000000001e-05 2024-05-15T13:04:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:25.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.745603+0000", 2024-05-15T13:04:25.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5928999999999999e-05 2024-05-15T13:04:25.844 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.844 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:25.844 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.844 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:104798 5.4 5:2256fe8f:::benchmark_data_smithi177_109991_object104797:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-05-15T13:04:25.844 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:04:20.683478+0000", 2024-05-15T13:04:25.844 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.0478820830000002, 2024-05-15T13:04:25.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.062245428999999998, 2024-05-15T13:04:25.844 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:04:25.845 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:04:25.845 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:04:25.845 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:04:25.845 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:04:25.845 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 104798 2024-05-15T13:04:25.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.845 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:04:25.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:04:25.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.683478+0000", 2024-05-15T13:04:25.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:04:25.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.683478+0000", 2024-05-15T13:04:25.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:04:25.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:04:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.683479+0000", 2024-05-15T13:04:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1799999999999999e-06 2024-05-15T13:04:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:04:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.683498+0000", 2024-05-15T13:04:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8413000000000001e-05 2024-05-15T13:04:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:04:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.683509+0000", 2024-05-15T13:04:25.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1354e-05 2024-05-15T13:04:25.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:04:25.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.683515+0000", 2024-05-15T13:04:25.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.015e-06 2024-05-15T13:04:25.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:04:25.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.683568+0000", 2024-05-15T13:04:25.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3576999999999999e-05 2024-05-15T13:04:25.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:04:25.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.683711+0000", 2024-05-15T13:04:25.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014238599999999999 2024-05-15T13:04:25.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:04:25.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.683877+0000", 2024-05-15T13:04:25.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000166205 2024-05-15T13:04:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:04:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.686404+0000", 2024-05-15T13:04:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025272300000000001 2024-05-15T13:04:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.686728+0000", 2024-05-15T13:04:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00032391100000000002 2024-05-15T13:04:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:04:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.745656+0000", 2024-05-15T13:04:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.058927946000000002 2024-05-15T13:04:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:04:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.745691+0000", 2024-05-15T13:04:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4428999999999999e-05 2024-05-15T13:04:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:04:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:04:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:04:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.745723+0000", 2024-05-15T13:04:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2783000000000001e-05 2024-05-15T13:04:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:25.851 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.852 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:04:25.852 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:04:25.852 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:04:25.973 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 65 16 110034 110018 105.75 60.5 0.0102405 0.00945267 2024-05-15T13:04:26.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:26 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:04:26.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:26 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:04:26.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:26 smithi177 ceph-mon[44303]: 5.2 scrub starts 2024-05-15T13:04:26.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:26 smithi177 ceph-mon[44303]: 5.e scrub starts 2024-05-15T13:04:26.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:26 smithi177 ceph-mon[44303]: osdmap e446: 12 total, 12 up, 11 in 2024-05-15T13:04:26.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:26 smithi177 ceph-mon[44303]: 5.b scrub ok 2024-05-15T13:04:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:26 smithi177 ceph-mon[44303]: pgmap v1189: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 6.5 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-05-15T13:04:26.506 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-05-15T13:04:26.595 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:04:26.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:26.604 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:26 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:04:26.604 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:26 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:04:26.605 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:26 smithi027 ceph-mon[39795]: 5.2 scrub starts 2024-05-15T13:04:26.605 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:26 smithi027 ceph-mon[39795]: 5.e scrub starts 2024-05-15T13:04:26.605 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:26 smithi027 ceph-mon[39795]: osdmap e446: 12 total, 12 up, 11 in 2024-05-15T13:04:26.605 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:26 smithi027 ceph-mon[39795]: 5.b scrub ok 2024-05-15T13:04:26.605 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:26 smithi027 ceph-mon[39795]: pgmap v1189: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 6.5 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-05-15T13:04:26.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:04:26.706 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:04:26.754 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:04:26.754 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:04:26.754 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:04:26.755 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:26.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:26 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:04:26.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:26 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:04:26.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:26 smithi120 ceph-mon[43961]: 5.2 scrub starts 2024-05-15T13:04:26.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:26 smithi120 ceph-mon[43961]: 5.e scrub starts 2024-05-15T13:04:26.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:26 smithi120 ceph-mon[43961]: osdmap e446: 12 total, 12 up, 11 in 2024-05-15T13:04:26.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:26 smithi120 ceph-mon[43961]: 5.b scrub ok 2024-05-15T13:04:26.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:26 smithi120 ceph-mon[43961]: pgmap v1189: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 6.5 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-05-15T13:04:26.801 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:04:26.848 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:04:26.878 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:04:26.910 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:04:26.951 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:04:26.963 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:04:26.976 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 66 16 111028 111012 105.087 62.125 0.0186882 0.00951184 2024-05-15T13:04:26.978 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:04:27.021 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:04:27.029 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:04:27.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:27.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:04:27.061 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:04:27.108 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:04:27.115 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:04:27.116 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:04:27.116 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:04:27.116 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:04:27.116 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:04:27.116 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:04:27.116 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:04:27.116 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:04:27.116 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:04:27.116 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:04:27.116 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:04:27.116 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:04:27.136 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:04:27.188 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:27.343 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:27 smithi027 ceph-mon[39795]: 5.9 scrub starts 2024-05-15T13:04:27.344 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:27 smithi027 ceph-mon[39795]: 5.2 scrub ok 2024-05-15T13:04:27.344 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:27 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2649909365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:04:27.344 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:27 smithi027 ceph-mon[39795]: 5.e scrub ok 2024-05-15T13:04:27.402 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:04:27.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:27.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:04:27.510 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:04:27.519 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:27 smithi177 ceph-mon[44303]: 5.9 scrub starts 2024-05-15T13:04:27.519 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:27 smithi177 ceph-mon[44303]: 5.2 scrub ok 2024-05-15T13:04:27.519 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:27 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2649909365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:04:27.520 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:27 smithi177 ceph-mon[44303]: 5.e scrub ok 2024-05-15T13:04:27.553 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:27.583 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:04:27.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:27.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:04:27.691 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:04:27.734 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:27 smithi120 ceph-mon[43961]: 5.9 scrub starts 2024-05-15T13:04:27.734 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:27 smithi120 ceph-mon[43961]: 5.2 scrub ok 2024-05-15T13:04:27.735 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:27 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2649909365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:04:27.735 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:27 smithi120 ceph-mon[43961]: 5.e scrub ok 2024-05-15T13:04:27.737 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:27.753 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:04:27.762 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:04:27.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:27.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:04:27.867 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:04:27.915 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:27.974 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 67 16 112059 112043 104.48 64.4375 0.020849 0.00956709 2024-05-15T13:04:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:04:28.672 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:28 smithi177 ceph-mon[44303]: 5.9 scrub ok 2024-05-15T13:04:28.672 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:28 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1365815837' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:04:28.672 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:28 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:04:28.673 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:28 smithi177 ceph-mon[44303]: pgmap v1190: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.7 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-05-15T13:04:28.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:28 smithi027 ceph-mon[39795]: 5.9 scrub ok 2024-05-15T13:04:28.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:28 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1365815837' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:04:28.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:28 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:04:28.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:28 smithi027 ceph-mon[39795]: pgmap v1190: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.7 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-05-15T13:04:28.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:28 smithi120 ceph-mon[43961]: 5.9 scrub ok 2024-05-15T13:04:28.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:28 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1365815837' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:04:28.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:28 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:04:28.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:28 smithi120 ceph-mon[43961]: pgmap v1190: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.7 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-05-15T13:04:28.974 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 68 16 113117 113101 103.916 66.125 0.0281207 0.00961828 2024-05-15T13:04:29.032 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:04:29.032 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:04:29.196 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:04:29.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:29.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:04:29.304 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:04:29.354 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:29.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:29 smithi177 ceph-mon[44303]: 5.8 scrub starts 2024-05-15T13:04:29.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:29 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:04:29.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:29 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T13:04:29.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:29 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:04:29.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:29 smithi177 ceph-mon[44303]: osdmap e447: 12 total, 12 up, 11 in 2024-05-15T13:04:29.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:29 smithi027 ceph-mon[39795]: 5.8 scrub starts 2024-05-15T13:04:29.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:29 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:04:29.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:29 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T13:04:29.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:29 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:04:29.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:29 smithi027 ceph-mon[39795]: osdmap e447: 12 total, 12 up, 11 in 2024-05-15T13:04:29.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:29 smithi120 ceph-mon[43961]: 5.8 scrub starts 2024-05-15T13:04:29.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:29 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:04:29.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:29 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T13:04:29.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:29 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:04:29.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:29 smithi120 ceph-mon[43961]: osdmap e447: 12 total, 12 up, 11 in 2024-05-15T13:04:29.811 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-05-15T13:04:29.825 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:04:29.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:29.828 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:04:29.929 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:04:29.974 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 69 16 114211 114195 103.401 68.375 0.0161311 0.00966783 2024-05-15T13:04:29.985 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:30.014 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:04:30.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:30.018 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:04:30.119 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:04:30.159 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:30.185 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:04:30.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:30.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:04:30.236 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:04:30.288 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:04:30.331 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:30.463 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:04:30.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:30.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:04:30.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:30 smithi120 ceph-mon[43961]: pgmap v1192: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.8 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-05-15T13:04:30.569 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:04:30.605 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:30.606 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:30 smithi027 ceph-mon[39795]: pgmap v1192: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.8 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-05-15T13:04:30.634 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:30 smithi177 ceph-mon[44303]: pgmap v1192: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.8 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-05-15T13:04:30.635 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:04:30.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:30.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:04:30.741 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:04:30.780 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:30.809 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:04:30.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:30.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:04:30.920 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:04:30.969 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:30.975 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 70 16 115320 115304 102.913 69.3125 0.00870501 0.00971283 2024-05-15T13:04:31.009 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:04:31.017 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:04:31.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:31.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:04:31.122 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:04:31.170 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:31.330 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:04:31.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:31.332 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:04:31.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:31 smithi177 ceph-mon[44303]: 5.8 scrub ok 2024-05-15T13:04:31.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:31 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:04:31.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:31 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:04:31.433 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:04:31.477 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:31.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:31 smithi027 ceph-mon[39795]: 5.8 scrub ok 2024-05-15T13:04:31.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:31 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:04:31.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:31 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:04:31.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:31 smithi120 ceph-mon[43961]: 5.8 scrub ok 2024-05-15T13:04:31.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:31 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:04:31.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:31 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:04:31.786 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:04:31.831 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:04:31.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:31.837 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:04:31.938 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:04:31.970 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:31.975 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 71 16 116326 116310 102.349 62.875 0.011201 0.00976658 2024-05-15T13:04:32.003 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:04:32.005 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:04:32.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:32.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:04:32.107 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:04:32.150 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:32.185 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:04:32.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:32.188 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:04:32.201 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:04:32.214 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:04:32.289 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:04:32.341 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:32.388 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:04:32.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:32.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:04:32.498 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:04:32.525 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:32.551 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:04:32.551 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:04:32.558 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:04:32.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:32.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:04:32.666 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:04:32.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:32 smithi177 ceph-mon[44303]: osdmap e448: 12 total, 12 up, 11 in 2024-05-15T13:04:32.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:32 smithi177 ceph-mon[44303]: pgmap v1194: 17 pgs: 17 active+clean; 6.9 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 60 MiB/s wr, 953 op/s 2024-05-15T13:04:32.698 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:32 smithi120 ceph-mon[43961]: osdmap e448: 12 total, 12 up, 11 in 2024-05-15T13:04:32.698 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:32 smithi120 ceph-mon[43961]: pgmap v1194: 17 pgs: 17 active+clean; 6.9 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 60 MiB/s wr, 953 op/s 2024-05-15T13:04:32.700 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:32.730 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:04:32.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:32.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:04:32.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:32 smithi027 ceph-mon[39795]: osdmap e448: 12 total, 12 up, 11 in 2024-05-15T13:04:32.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:32 smithi027 ceph-mon[39795]: pgmap v1194: 17 pgs: 17 active+clean; 6.9 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 60 MiB/s wr, 953 op/s 2024-05-15T13:04:32.834 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:04:32.884 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:32.948 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:04:32.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:32.958 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:04:32.979 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 72 16 117309 117293 101.779 61.4375 0.00776778 0.00982113 2024-05-15T13:04:33.059 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:04:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:33.284 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-05-15T13:04:33.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:33 smithi177 ceph-mon[44303]: 5.d scrub starts 2024-05-15T13:04:33.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:33 smithi177 ceph-mon[44303]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:04:33.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:33 smithi027 ceph-mon[39795]: 5.d scrub starts 2024-05-15T13:04:33.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:33 smithi027 ceph-mon[39795]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:04:33.774 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:04:33.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:33 smithi120 ceph-mon[43961]: 5.d scrub starts 2024-05-15T13:04:33.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:33 smithi120 ceph-mon[43961]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:04:33.977 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 73 16 118221 118205 101.165 57 0.00964923 0.00988093 2024-05-15T13:04:34.239 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":1194,"stamp":"2024-05-15T13:04:31.983644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7420183072,"num_objects":113216,"num_object_clones":0,"num_object_copies":339648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113216,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":113395,"num_write_kb":7248634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113353,"ondisk_log_size":113353,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":26935968,"kb_used_data":22580308,"kb_used_omap":17,"kb_used_meta":4354990,"kb_avail":1004260704,"statfs":{"total":1055945392128,"available":1028362960896,"internally_reserved":0,"allocated":23122235392,"data_stored":23116721531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17551,"internal_metadata":4459510641},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":415236096,"num_objects":6336,"num_object_clones":0,"num_object_copies":19008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6336,"num_write_kb":405504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.645257"},"pg_stats":[{"pgid":"5.f","version":"446'6967","reported_seq":7032,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-05-15T13:04:27.894477+0000","last_change":"2024-05-15T13:03:11.329088+0000","last_active":"2024-05-15T13:04:27.894477+0000","last_peered":"2024-05-15T13:04:27.894477+0000","last_clean":"2024-05-15T13:04:27.894477+0000","last_became_active":"2024-05-15T13:03:11.328783+0000","last_became_peered":"2024-05-15T13:03:11.328783+0000","last_unstale":"2024-05-15T13:04:27.894477+0000","last_undegraded":"2024-05-15T13:04:27.894477+0000","last_fullsized":"2024-05-15T13:04:27.894477+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:03:10.302515+0000","objects_scrubbed":0,"log_size":6967,"log_dups_size":0,"ondisk_log_size":6967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:04:40.000374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456589312,"num_objects":6967,"num_object_clones":0,"num_object_copies":20901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6967,"num_write_kb":445888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"447'7216","reported_seq":7362,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-05-15T13:04:30.544028+0000","last_change":"2024-05-15T13:04:26.638047+0000","last_active":"2024-05-15T13:04:30.544028+0000","last_peered":"2024-05-15T13:04:30.544028+0000","last_clean":"2024-05-15T13:04:30.544028+0000","last_became_active":"2024-05-15T13:03:11.330241+0000","last_became_peered":"2024-05-15T13:03:11.330241+0000","last_unstale":"2024-05-15T13:04:30.544028+0000","last_undegraded":"2024-05-15T13:04:30.544028+0000","last_fullsized":"2024-05-15T13:04:30.544028+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'6959","last_scrub_stamp":"2024-05-15T13:04:26.637942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:26.637942+0000","objects_scrubbed":6905,"log_size":7216,"log_dups_size":0,"ondisk_log_size":7216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:43.654277+0000","scrub_duration":1597,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472907776,"num_objects":7216,"num_object_clones":0,"num_object_copies":21648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7216,"num_write_kb":461824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"446'7042","reported_seq":7108,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-05-15T13:04:27.506065+0000","last_change":"2024-05-15T13:03:11.331257+0000","last_active":"2024-05-15T13:04:27.506065+0000","last_peered":"2024-05-15T13:04:27.506065+0000","last_clean":"2024-05-15T13:04:27.506065+0000","last_became_active":"2024-05-15T13:03:11.330919+0000","last_became_peered":"2024-05-15T13:03:11.330919+0000","last_unstale":"2024-05-15T13:04:27.506065+0000","last_undegraded":"2024-05-15T13:04:27.506065+0000","last_fullsized":"2024-05-15T13:04:27.506065+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:03:10.302515+0000","objects_scrubbed":0,"log_size":7042,"log_dups_size":0,"ondisk_log_size":7042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":461504512,"num_objects":7042,"num_object_clones":0,"num_object_copies":21126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7042,"num_write_kb":450688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"447'7272","reported_seq":7340,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-05-15T13:04:31.341820+0000","last_change":"2024-05-15T13:03:11.327685+0000","last_active":"2024-05-15T13:04:31.341820+0000","last_peered":"2024-05-15T13:04:31.341820+0000","last_clean":"2024-05-15T13:04:31.341820+0000","last_became_active":"2024-05-15T13:03:11.327387+0000","last_became_peered":"2024-05-15T13:03:11.327387+0000","last_unstale":"2024-05-15T13:04:31.341820+0000","last_undegraded":"2024-05-15T13:04:31.341820+0000","last_fullsized":"2024-05-15T13:04:31.341820+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:03:10.302515+0000","objects_scrubbed":0,"log_size":7272,"log_dups_size":0,"ondisk_log_size":7272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":476577792,"num_objects":7272,"num_object_clones":0,"num_object_copies":21816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7272,"num_write_kb":465408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"447'7000","reported_seq":7145,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-05-15T13:04:29.167264+0000","last_change":"2024-05-15T13:04:25.608723+0000","last_active":"2024-05-15T13:04:29.167264+0000","last_peered":"2024-05-15T13:04:29.167264+0000","last_clean":"2024-05-15T13:04:29.167264+0000","last_became_active":"2024-05-15T13:03:11.328691+0000","last_became_peered":"2024-05-15T13:03:11.328691+0000","last_unstale":"2024-05-15T13:04:29.167264+0000","last_undegraded":"2024-05-15T13:04:29.167264+0000","last_fullsized":"2024-05-15T13:04:29.167264+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'6796","last_scrub_stamp":"2024-05-15T13:04:25.608583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:25.608583+0000","objects_scrubbed":6750,"log_size":7000,"log_dups_size":0,"ondisk_log_size":7000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:45.331376+0000","scrub_duration":1526,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458752000,"num_objects":7000,"num_object_clones":0,"num_object_copies":21000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7000,"num_write_kb":448000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,5],"acting":[0,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"446'6970","reported_seq":7108,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-05-15T13:04:27.568516+0000","last_change":"2024-05-15T13:04:20.798543+0000","last_active":"2024-05-15T13:04:27.568516+0000","last_peered":"2024-05-15T13:04:27.568516+0000","last_clean":"2024-05-15T13:04:27.568516+0000","last_became_active":"2024-05-15T13:03:11.330425+0000","last_became_peered":"2024-05-15T13:03:11.330425+0000","last_unstale":"2024-05-15T13:04:27.568516+0000","last_undegraded":"2024-05-15T13:04:27.568516+0000","last_fullsized":"2024-05-15T13:04:27.568516+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'6531","last_scrub_stamp":"2024-05-15T13:04:20.798310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:20.798310+0000","objects_scrubbed":6490,"log_size":6970,"log_dups_size":0,"ondisk_log_size":6970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:28.156588+0000","scrub_duration":1689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456785920,"num_objects":6970,"num_object_clones":0,"num_object_copies":20910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6970,"num_write_kb":446080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"447'6977","reported_seq":7121,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-05-15T13:04:28.663463+0000","last_change":"2024-05-15T13:04:27.305971+0000","last_active":"2024-05-15T13:04:28.663463+0000","last_peered":"2024-05-15T13:04:28.663463+0000","last_clean":"2024-05-15T13:04:28.663463+0000","last_became_active":"2024-05-15T13:03:11.329210+0000","last_became_peered":"2024-05-15T13:03:11.329210+0000","last_unstale":"2024-05-15T13:04:28.663463+0000","last_undegraded":"2024-05-15T13:04:28.663463+0000","last_fullsized":"2024-05-15T13:04:28.663463+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'6893","last_scrub_stamp":"2024-05-15T13:04:27.305709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:27.305709+0000","objects_scrubbed":6847,"log_size":6977,"log_dups_size":0,"ondisk_log_size":6977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:39.171201+0000","scrub_duration":1459,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457244672,"num_objects":6977,"num_object_clones":0,"num_object_copies":20931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6977,"num_write_kb":446528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,9],"acting":[6,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"446'6886","reported_seq":6951,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-05-15T13:04:27.480055+0000","last_change":"2024-05-15T13:03:11.331442+0000","last_active":"2024-05-15T13:04:27.480055+0000","last_peered":"2024-05-15T13:04:27.480055+0000","last_clean":"2024-05-15T13:04:27.480055+0000","last_became_active":"2024-05-15T13:03:11.330983+0000","last_became_peered":"2024-05-15T13:03:11.330983+0000","last_unstale":"2024-05-15T13:04:27.480055+0000","last_undegraded":"2024-05-15T13:04:27.480055+0000","last_fullsized":"2024-05-15T13:04:27.480055+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:03:10.302515+0000","objects_scrubbed":0,"log_size":6886,"log_dups_size":0,"ondisk_log_size":6886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451280896,"num_objects":6886,"num_object_clones":0,"num_object_copies":20658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6886,"num_write_kb":440704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"446'7026","reported_seq":7165,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-05-15T13:04:27.884866+0000","last_change":"2024-05-15T13:04:20.177631+0000","last_active":"2024-05-15T13:04:27.884866+0000","last_peered":"2024-05-15T13:04:27.884866+0000","last_clean":"2024-05-15T13:04:27.884866+0000","last_became_active":"2024-05-15T13:03:11.329484+0000","last_became_peered":"2024-05-15T13:03:11.329484+0000","last_unstale":"2024-05-15T13:04:27.884866+0000","last_undegraded":"2024-05-15T13:04:27.884866+0000","last_fullsized":"2024-05-15T13:04:27.884866+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'6536","last_scrub_stamp":"2024-05-15T13:04:20.177485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:20.177485+0000","objects_scrubbed":6499,"log_size":7026,"log_dups_size":0,"ondisk_log_size":7026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:29.617242+0000","scrub_duration":1314,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460455936,"num_objects":7026,"num_object_clones":0,"num_object_copies":21078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7026,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"446'7011","reported_seq":7150,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-05-15T13:04:27.571840+0000","last_change":"2024-05-15T13:04:18.779479+0000","last_active":"2024-05-15T13:04:27.571840+0000","last_peered":"2024-05-15T13:04:27.571840+0000","last_clean":"2024-05-15T13:04:27.571840+0000","last_became_active":"2024-05-15T13:03:11.332923+0000","last_became_peered":"2024-05-15T13:03:11.332923+0000","last_unstale":"2024-05-15T13:04:27.571840+0000","last_undegraded":"2024-05-15T13:04:27.571840+0000","last_fullsized":"2024-05-15T13:04:27.571840+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'6425","last_scrub_stamp":"2024-05-15T13:04:18.779285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:18.779285+0000","objects_scrubbed":6366,"log_size":7011,"log_dups_size":0,"ondisk_log_size":7011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:48.641808+0000","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459472896,"num_objects":7011,"num_object_clones":0,"num_object_copies":21033,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7011,"num_write_kb":448704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,10],"acting":[2,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"447'7031","reported_seq":7172,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-05-15T13:04:29.186150+0000","last_change":"2024-05-15T13:04:22.066492+0000","last_active":"2024-05-15T13:04:29.186150+0000","last_peered":"2024-05-15T13:04:29.186150+0000","last_clean":"2024-05-15T13:04:29.186150+0000","last_became_active":"2024-05-15T13:03:11.333336+0000","last_became_peered":"2024-05-15T13:03:11.333336+0000","last_unstale":"2024-05-15T13:04:29.186150+0000","last_undegraded":"2024-05-15T13:04:29.186150+0000","last_fullsized":"2024-05-15T13:04:29.186150+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'6582","last_scrub_stamp":"2024-05-15T13:04:22.066294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:22.066294+0000","objects_scrubbed":6533,"log_size":7031,"log_dups_size":0,"ondisk_log_size":7031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:46.801279+0000","scrub_duration":1991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460783616,"num_objects":7031,"num_object_clones":0,"num_object_copies":21093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7031,"num_write_kb":449984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"447'7138","reported_seq":7280,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-05-15T13:04:29.167259+0000","last_change":"2024-05-15T13:04:22.690622+0000","last_active":"2024-05-15T13:04:29.167259+0000","last_peered":"2024-05-15T13:04:29.167259+0000","last_clean":"2024-05-15T13:04:29.167259+0000","last_became_active":"2024-05-15T13:03:11.327788+0000","last_became_peered":"2024-05-15T13:03:11.327788+0000","last_unstale":"2024-05-15T13:04:29.167259+0000","last_undegraded":"2024-05-15T13:04:29.167259+0000","last_fullsized":"2024-05-15T13:04:29.167259+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'6704","last_scrub_stamp":"2024-05-15T13:04:22.690346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:22.690346+0000","objects_scrubbed":6642,"log_size":7138,"log_dups_size":0,"ondisk_log_size":7138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:51.461018+0000","scrub_duration":1607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467795968,"num_objects":7138,"num_object_clones":0,"num_object_copies":21414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7138,"num_write_kb":456832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"447'7018","reported_seq":7162,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-05-15T13:04:28.654790+0000","last_change":"2024-05-15T13:04:26.196007+0000","last_active":"2024-05-15T13:04:28.654790+0000","last_peered":"2024-05-15T13:04:28.654790+0000","last_clean":"2024-05-15T13:04:28.654790+0000","last_became_active":"2024-05-15T13:03:11.330316+0000","last_became_peered":"2024-05-15T13:03:11.330316+0000","last_unstale":"2024-05-15T13:04:28.654790+0000","last_undegraded":"2024-05-15T13:04:28.654790+0000","last_fullsized":"2024-05-15T13:04:28.654790+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'6865","last_scrub_stamp":"2024-05-15T13:04:26.195895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:26.195895+0000","objects_scrubbed":6825,"log_size":7018,"log_dups_size":0,"ondisk_log_size":7018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:55.500383+0000","scrub_duration":1403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459931648,"num_objects":7018,"num_object_clones":0,"num_object_copies":21054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7018,"num_write_kb":449152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"447'7154","reported_seq":7296,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-05-15T13:04:29.852039+0000","last_change":"2024-05-15T13:04:22.294876+0000","last_active":"2024-05-15T13:04:29.852039+0000","last_peered":"2024-05-15T13:04:29.852039+0000","last_clean":"2024-05-15T13:04:29.852039+0000","last_became_active":"2024-05-15T13:03:11.430987+0000","last_became_peered":"2024-05-15T13:03:11.430987+0000","last_unstale":"2024-05-15T13:04:29.852039+0000","last_undegraded":"2024-05-15T13:04:29.852039+0000","last_fullsized":"2024-05-15T13:04:29.852039+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'6645","last_scrub_stamp":"2024-05-15T13:04:22.294671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:22.294671+0000","objects_scrubbed":6590,"log_size":7154,"log_dups_size":0,"ondisk_log_size":7154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:42.703798+0000","scrub_duration":1280,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":468844544,"num_objects":7154,"num_object_clones":0,"num_object_copies":21462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7154,"num_write_kb":457856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"447'7259","reported_seq":7327,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-05-15T13:04:31.362809+0000","last_change":"2024-05-15T13:03:11.331154+0000","last_active":"2024-05-15T13:04:31.362809+0000","last_peered":"2024-05-15T13:04:31.362809+0000","last_clean":"2024-05-15T13:04:31.362809+0000","last_became_active":"2024-05-15T13:03:11.330479+0000","last_became_peered":"2024-05-15T13:03:11.330479+0000","last_unstale":"2024-05-15T13:04:31.362809+0000","last_undegraded":"2024-05-15T13:04:31.362809+0000","last_fullsized":"2024-05-15T13:04:31.362809+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:03:10.302515+0000","objects_scrubbed":0,"log_size":7259,"log_dups_size":0,"ondisk_log_size":7259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:04:31.595397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475725824,"num_objects":7259,"num_object_clones":0,"num_object_copies":21777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7259,"num_write_kb":464576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,4],"acting":[1,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"447'7247","reported_seq":7390,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-05-15T13:04:31.351170+0000","last_change":"2024-05-15T13:04:21.985812+0000","last_active":"2024-05-15T13:04:31.351170+0000","last_peered":"2024-05-15T13:04:31.351170+0000","last_clean":"2024-05-15T13:04:31.351170+0000","last_became_active":"2024-05-15T13:03:11.331291+0000","last_became_peered":"2024-05-15T13:03:11.331291+0000","last_unstale":"2024-05-15T13:04:31.351170+0000","last_undegraded":"2024-05-15T13:04:31.351170+0000","last_fullsized":"2024-05-15T13:04:31.351170+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'6673","last_scrub_stamp":"2024-05-15T13:04:21.985599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:21.985599+0000","objects_scrubbed":6626,"log_size":7247,"log_dups_size":0,"ondisk_log_size":7247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:27.438473+0000","scrub_duration":1491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474939392,"num_objects":7247,"num_object_clones":0,"num_object_copies":21741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7247,"num_write_kb":463808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":713,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-05-15T13:04:28.379249+0000","last_change":"2024-05-15T13:03:21.248322+0000","last_active":"2024-05-15T13:04:28.379249+0000","last_peered":"2024-05-15T13:04:28.379249+0000","last_clean":"2024-05-15T13:04:28.379249+0000","last_became_active":"2024-05-15T13:00:26.045412+0000","last_became_peered":"2024-05-15T13:00:26.045412+0000","last_unstale":"2024-05-15T13:04:28.379249+0000","last_undegraded":"2024-05-15T13:04:28.379249+0000","last_fullsized":"2024-05-15T13:04:28.379249+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:03:21.248170+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:03:21.248170+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:04:38.165247+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":7419592704,"num_objects":113214,"num_object_clones":0,"num_object_copies":339642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113214,"num_write_kb":7245696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22345678848,"data_stored":22345678848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113214,"ondisk_log_size":113214,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056868,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2086012,"kb_used_data":1770228,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":91659140,"statfs":{"total":95995035648,"available":93858959360,"internally_reserved":0,"allocated":1812713472,"data_stored":1812167420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":323353030},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351209,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3397404,"kb_used_data":2996948,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":90347748,"statfs":{"total":95995035648,"available":92516093952,"internally_reserved":0,"allocated":3068874752,"data_stored":3068388495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":410057152},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514735,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2898380,"kb_used_data":2379724,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":90846772,"statfs":{"total":95995035648,"available":93027094528,"internally_reserved":0,"allocated":2436837376,"data_stored":2436354671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":530905536},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874484,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1428372,"kb_used_data":1136204,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":92316780,"statfs":{"total":95995035648,"available":94532382720,"internally_reserved":0,"allocated":1163472896,"data_stored":1162990211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":240518168823,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430460,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1957316,"kb_used_data":1574908,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":91787836,"statfs":{"total":95995035648,"available":93990744064,"internally_reserved":0,"allocated":1612705792,"data_stored":1612156208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":391576006},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575128,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2349404,"kb_used_data":1844820,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":91395748,"statfs":{"total":95995035648,"available":93589245952,"internally_reserved":0,"allocated":1889095680,"data_stored":1888609443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738656,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4060884,"kb_used_data":3618764,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":89684268,"statfs":{"total":95995035648,"available":91836690432,"internally_reserved":0,"allocated":3705614336,"data_stored":3705131651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":452721093},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411604,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1701824,"kb_used_data":1409652,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":92043328,"statfs":{"total":95995035648,"available":94252367872,"internally_reserved":0,"allocated":1443483648,"data_stored":1442937616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149966,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":902052,"kb_used_data":465316,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92843100,"statfs":{"total":95995035648,"available":95071334400,"internally_reserved":0,"allocated":476483584,"data_stored":476060677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":447019463},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986441,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2677708,"kb_used_data":2267340,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":91067444,"statfs":{"total":95995035648,"available":93253062656,"internally_reserved":0,"allocated":2321756160,"data_stored":2321273475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692098,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3476612,"kb_used_data":3116404,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":90268540,"statfs":{"total":95995035648,"available":92434984960,"internally_reserved":0,"allocated":3191197696,"data_stored":3190651664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3710124032,"data_stored":3710124032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1899888640,"data_stored":1899888640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1375666176,"data_stored":1375666176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":475594752,"data_stored":475594752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2340028416,"data_stored":2340028416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3215589376,"data_stored":3215589376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1382481920,"data_stored":1382481920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":938147840,"data_stored":938147840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2348613632,"data_stored":2348613632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2838888448,"data_stored":2838888448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1820655616,"data_stored":1820655616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:04:34.240 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-05-15T13:04:34.240 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:04:34.241 INFO:tasks.thrashosds.thrasher:inject_pause on osd.10 2024-05-15T13:04:34.241 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-15T13:04:34.241 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-15T13:04:34.241 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 config set filestore_inject_stall 3 2024-05-15T13:04:34.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:34 smithi027 ceph-mon[39795]: pgmap v1195: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.1 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-05-15T13:04:34.585 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:04:34.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:34.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:04:34.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:34 smithi177 ceph-mon[44303]: pgmap v1195: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.1 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-05-15T13:04:34.726 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:04:34.763 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:34.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:34 smithi120 ceph-mon[43961]: pgmap v1195: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.1 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-05-15T13:04:34.977 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 74 16 119086 119070 100.529 54.0625 0.0195659 0.00994353 2024-05-15T13:04:35.144 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:04:35.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:35.156 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:04:35.204 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:04:35.258 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:04:35.313 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:35.730 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:35 smithi177 ceph-mon[44303]: 5.c scrub starts 2024-05-15T13:04:35.731 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:35 smithi177 ceph-mon[44303]: 5.d scrub ok 2024-05-15T13:04:35.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:35 smithi027 ceph-mon[39795]: 5.c scrub starts 2024-05-15T13:04:35.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:35 smithi027 ceph-mon[39795]: 5.d scrub ok 2024-05-15T13:04:35.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:35 smithi120 ceph-mon[43961]: 5.c scrub starts 2024-05-15T13:04:35.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:35 smithi120 ceph-mon[43961]: 5.d scrub ok 2024-05-15T13:04:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:04:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:04:35.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:35.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:04:35.977 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 75 16 119881 119865 99.8513 49.6875 0.0149868 0.0100112 2024-05-15T13:04:35.994 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:04:35.994 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:04:36.049 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:04:36.087 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:36.123 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:04:36.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:36.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:04:36.230 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:04:36.283 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:36.327 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:04:36.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:36.330 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:04:36.431 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:04:36.489 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:36 smithi027 ceph-mon[39795]: 5.c scrub ok 2024-05-15T13:04:36.489 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:36 smithi027 ceph-mon[39795]: pgmap v1196: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.2 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 57 MiB/s wr, 916 op/s 2024-05-15T13:04:36.493 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:36.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:36 smithi177 ceph-mon[44303]: 5.c scrub ok 2024-05-15T13:04:36.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:36 smithi177 ceph-mon[44303]: pgmap v1196: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.2 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 57 MiB/s wr, 916 op/s 2024-05-15T13:04:36.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:36 smithi120 ceph-mon[43961]: 5.c scrub ok 2024-05-15T13:04:36.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:36 smithi120 ceph-mon[43961]: pgmap v1196: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.2 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 57 MiB/s wr, 916 op/s 2024-05-15T13:04:36.977 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 76 16 120955 120939 99.4207 67.125 0.00994398 0.0100546 2024-05-15T13:04:37.760 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:04:37.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:37.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:04:37.868 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:04:37.896 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:37.977 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 77 16 121859 121843 98.8634 56.5 0.00395507 0.0100925 2024-05-15T13:04:38.095 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:04:38.124 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:04:38.181 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:04:38.212 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:04:38.260 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:04:38.294 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:04:38.333 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:04:38.371 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:04:38.423 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:04:38.469 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:04:38.503 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:04:38.535 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:04:38.538 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:04:38.539 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:04:38.539 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:04:38.539 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:04:38.539 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:04:38.539 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:04:38.539 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:04:38.539 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:04:38.539 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:04:38.539 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:04:38.539 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:04:38.539 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:04:38.977 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 78 16 122869 122853 98.4052 63.125 0.0148236 0.0101583 2024-05-15T13:04:39.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:39 smithi027 ceph-mon[39795]: 5.0 scrub starts 2024-05-15T13:04:39.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:39 smithi027 ceph-mon[39795]: pgmap v1197: 17 pgs: 17 active+clean; 7.3 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-05-15T13:04:39.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:39 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/380396896' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:04:39.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:39 smithi177 ceph-mon[44303]: 5.0 scrub starts 2024-05-15T13:04:39.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:39 smithi177 ceph-mon[44303]: pgmap v1197: 17 pgs: 17 active+clean; 7.3 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-05-15T13:04:39.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:39 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/380396896' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:04:39.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:39 smithi120 ceph-mon[43961]: 5.0 scrub starts 2024-05-15T13:04:39.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:39 smithi120 ceph-mon[43961]: pgmap v1197: 17 pgs: 17 active+clean; 7.3 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-05-15T13:04:39.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:39 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/380396896' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:04:39.978 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 79 16 123884 123868 97.9627 63.4375 0.0100933 0.0102037 2024-05-15T13:04:40.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:40 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T13:04:40.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:40 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T13:04:40.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:40 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T13:04:40.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:40 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T13:04:40.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:40 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T13:04:40.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:40 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T13:04:40.978 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T13:04:40.989286+0000 min lat: 0.00112512 max lat: 0.227364 avg lat: 0.0102441 2024-05-15T13:04:40.979 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:04:40.979 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 80 16 124964 124948 97.5819 67.5 0.0129643 0.0102441 2024-05-15T13:04:41.085 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:04:41.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:41 smithi177 ceph-mon[44303]: 5.0 scrub ok 2024-05-15T13:04:41.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:41 smithi177 ceph-mon[44303]: pgmap v1198: 17 pgs: 17 active+clean; 7.4 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 61 MiB/s wr, 981 op/s 2024-05-15T13:04:41.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:41 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1753298874' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:04:41.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:41 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:04:41.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:41 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:04:41.483 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:04:41.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:41 smithi027 ceph-mon[39795]: 5.0 scrub ok 2024-05-15T13:04:41.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:41 smithi027 ceph-mon[39795]: pgmap v1198: 17 pgs: 17 active+clean; 7.4 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 61 MiB/s wr, 981 op/s 2024-05-15T13:04:41.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:41 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1753298874' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:04:41.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:41 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:04:41.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:41 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:04:41.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:41 smithi120 ceph-mon[43961]: 5.0 scrub ok 2024-05-15T13:04:41.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:41 smithi120 ceph-mon[43961]: pgmap v1198: 17 pgs: 17 active+clean; 7.4 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 61 MiB/s wr, 981 op/s 2024-05-15T13:04:41.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:41 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1753298874' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:04:41.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:41 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:04:41.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:41 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:04:41.960 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:04:41.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:41.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:04:41.978 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 81 16 125979 125963 97.1604 63.4375 0.0180071 0.0102883 2024-05-15T13:04:42.066 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:04:42.106 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:42.133 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:04:42.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:42.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:04:42.242 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:04:42.302 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:42.330 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:04:42.337 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:42 smithi120 ceph-mon[43961]: 5.f scrub starts 2024-05-15T13:04:42.337 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:42 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:04:42.337 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:42 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1753298874' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:04:42.337 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:42 smithi120 ceph-mon[43961]: osdmap e449: 12 total, 12 up, 11 in 2024-05-15T13:04:42.337 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:42 smithi120 ceph-mon[43961]: pgmap v1200: 17 pgs: 17 active+clean; 7.5 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 61 MiB/s wr, 977 op/s 2024-05-15T13:04:42.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:42.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:04:42.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:42 smithi177 ceph-mon[44303]: 5.f scrub starts 2024-05-15T13:04:42.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:42 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:04:42.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:42 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1753298874' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:04:42.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:42 smithi177 ceph-mon[44303]: osdmap e449: 12 total, 12 up, 11 in 2024-05-15T13:04:42.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:42 smithi177 ceph-mon[44303]: pgmap v1200: 17 pgs: 17 active+clean; 7.5 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 61 MiB/s wr, 977 op/s 2024-05-15T13:04:42.439 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:04:42.497 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:42.518 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:42 smithi027 ceph-mon[39795]: 5.f scrub starts 2024-05-15T13:04:42.518 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:42 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:04:42.518 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:42 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1753298874' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:04:42.518 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:42 smithi027 ceph-mon[39795]: osdmap e449: 12 total, 12 up, 11 in 2024-05-15T13:04:42.518 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:42 smithi027 ceph-mon[39795]: pgmap v1200: 17 pgs: 17 active+clean; 7.5 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 61 MiB/s wr, 977 op/s 2024-05-15T13:04:42.571 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:04:42.978 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 82 16 127029 127013 96.7758 65.625 0.00663662 0.0103245 2024-05-15T13:04:43.048 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-05-15T13:04:43.049 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:04:43.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:43.055 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:04:43.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:04:43.247 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:43.978 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 83 16 128046 128030 96.3757 63.5625 0.0123675 0.0103723 2024-05-15T13:04:44.130 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:04:44.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:44.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:04:44.235 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:04:44.294 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:44 smithi177 ceph-mon[44303]: pgmap v1201: 17 pgs: 17 active+clean; 7.7 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 61 MiB/s wr, 970 op/s 2024-05-15T13:04:44.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:44 smithi027 ceph-mon[39795]: pgmap v1201: 17 pgs: 17 active+clean; 7.7 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 61 MiB/s wr, 970 op/s 2024-05-15T13:04:44.560 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:04:44.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:44 smithi120 ceph-mon[43961]: pgmap v1201: 17 pgs: 17 active+clean; 7.7 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 61 MiB/s wr, 970 op/s 2024-05-15T13:04:44.978 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 84 16 129102 129086 96.0141 66 0.0174343 0.010411 2024-05-15T13:04:45.091 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:04:45.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:45.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:04:45.102 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:04:45.200 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:04:45.262 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:45.370 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:04:45.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:45.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:04:45.414 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:04:45.483 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:04:45.549 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:45.978 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 85 16 130113 130097 95.6279 63.1875 0.0180546 0.0104533 2024-05-15T13:04:46.119 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:04:46.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:46.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:04:46.224 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:04:46.285 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:46.286 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:04:46.286 INFO:teuthology.orchestra.run.smithi177.stdout: "success": "" 2024-05-15T13:04:46.286 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:04:46.315 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:04:46.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:46.322 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:04:46.423 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:04:46.488 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:46.516 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:04:46.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:46.520 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:04:46.621 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:04:46.669 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:46.696 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:04:46.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:46.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:04:46.802 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:04:46.847 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:46.978 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 86 16 131144 131128 95.2653 64.4375 0.013804 0.0104929 2024-05-15T13:04:47.123 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:04:47.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:47.135 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:04:47.237 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:04:47.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:47 smithi027 ceph-mon[39795]: pgmap v1202: 17 pgs: 17 active+clean; 7.8 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 61 MiB/s wr, 978 op/s 2024-05-15T13:04:47.271 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:47.296 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:04:47.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:47.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:04:47.400 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:04:47.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:47 smithi177 ceph-mon[44303]: pgmap v1202: 17 pgs: 17 active+clean; 7.8 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 61 MiB/s wr, 978 op/s 2024-05-15T13:04:47.453 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:47.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:47 smithi120 ceph-mon[43961]: pgmap v1202: 17 pgs: 17 active+clean; 7.8 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 61 MiB/s wr, 978 op/s 2024-05-15T13:04:47.575 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:04:47.620 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:04:47.652 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:04:47.690 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:04:47.721 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:04:47.754 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:04:47.785 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:04:47.819 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:04:47.819 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:04:47.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:47.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:04:47.852 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:04:47.896 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:04:47.926 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:04:47.940 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:04:47.978 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:04:47.980 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 87 16 132172 132156 94.9064 64.25 0.022297 0.0105327 2024-05-15T13:04:47.982 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:48.009 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:04:48.010 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:04:48.010 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:04:48.010 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:04:48.010 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:04:48.010 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:04:48.010 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:04:48.010 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:04:48.010 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:04:48.010 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:04:48.010 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:04:48.244 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:48 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4045074166' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:04:48.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:48 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4045074166' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:04:48.486 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:04:48.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:48.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:04:48.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:48 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4045074166' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:04:48.593 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:04:48.645 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:48.686 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:04:48.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:48.695 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:04:48.796 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:04:48.843 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:48.874 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:04:48.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:48.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:04:48.979 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:04:48.980 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 88 16 133170 133154 94.5367 62.375 0.0100198 0.0105736 2024-05-15T13:04:49.037 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:49.184 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:49 smithi027 ceph-mon[39795]: pgmap v1203: 17 pgs: 17 active+clean; 7.9 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-05-15T13:04:49.184 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:04:49.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:49.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:04:49.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:49 smithi120 ceph-mon[43961]: pgmap v1203: 17 pgs: 17 active+clean; 7.9 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-05-15T13:04:49.290 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:04:49.331 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:49.354 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:04:49.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:49.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:04:49.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:49 smithi177 ceph-mon[44303]: pgmap v1203: 17 pgs: 17 active+clean; 7.9 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-05-15T13:04:49.460 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:04:49.503 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:49.537 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:04:49.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:49.541 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:04:49.642 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:04:49.714 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:49.891 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:04:49.891 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:04:49.983 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 89 16 134186 134170 94.188 63.5 0.0120399 0.0106131 2024-05-15T13:04:50.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:50 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:04:50.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:50 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:04:50.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:50 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:04:50.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:50 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:04:50.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:50 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:04:50.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:50 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:04:50.934 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:04:50.981 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Total time run: 90.0095 2024-05-15T13:04:50.981 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Total writes made: 135188 2024-05-15T13:04:50.981 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Write size: 65536 2024-05-15T13:04:50.981 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Object size: 65536 2024-05-15T13:04:50.981 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Bandwidth (MB/sec): 93.8707 2024-05-15T13:04:50.981 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Stddev Bandwidth: 40.1424 2024-05-15T13:04:50.981 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Max bandwidth (MB/sec): 242 2024-05-15T13:04:50.981 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Min bandwidth (MB/sec): 49.6875 2024-05-15T13:04:50.982 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Average IOPS: 1501 2024-05-15T13:04:50.982 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Stddev IOPS: 642.278 2024-05-15T13:04:50.982 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Max IOPS: 3872 2024-05-15T13:04:50.982 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Min IOPS: 795 2024-05-15T13:04:50.982 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Average Latency(s): 0.0106497 2024-05-15T13:04:50.982 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Stddev Latency(s): 0.0126728 2024-05-15T13:04:50.982 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Max latency(s): 0.227364 2024-05-15T13:04:50.982 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Min latency(s): 0.00112512 2024-05-15T13:04:50.989 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Cleaning up (deleting benchmark objects) 2024-05-15T13:04:51.140 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:04:51.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:51 smithi027 ceph-mon[39795]: pgmap v1204: 17 pgs: 17 active+clean; 8.1 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-05-15T13:04:51.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:51 smithi027 ceph-mon[39795]: osdmap e450: 12 total, 12 up, 11 in 2024-05-15T13:04:51.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:51 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2724047484' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:04:51.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:51 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:04:51.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:51 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:04:51.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:51 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:04:51.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:51 smithi177 ceph-mon[44303]: pgmap v1204: 17 pgs: 17 active+clean; 8.1 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-05-15T13:04:51.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:51 smithi177 ceph-mon[44303]: osdmap e450: 12 total, 12 up, 11 in 2024-05-15T13:04:51.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:51 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2724047484' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:04:51.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:51 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:04:51.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:51 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:04:51.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:51 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:04:51.490 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:04:51.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:51.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:04:51.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:51 smithi120 ceph-mon[43961]: pgmap v1204: 17 pgs: 17 active+clean; 8.1 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-05-15T13:04:51.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:51 smithi120 ceph-mon[43961]: osdmap e450: 12 total, 12 up, 11 in 2024-05-15T13:04:51.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:51 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2724047484' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:04:51.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:51 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:04:51.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:51 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:04:51.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:51 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:04:51.595 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:04:51.634 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:51.672 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:04:51.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:51.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:04:51.778 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:04:51.820 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:52.016 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:04:52.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:52.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:04:52.065 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-05-15T13:04:52.121 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:04:52.157 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:04:52.157 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:04:52.158 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-05-15T13:04:52.158 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd dump -f json-pretty 2024-05-15T13:04:52.163 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:52.186 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:04:52.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:52.191 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:04:52.292 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:04:52.327 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:52.353 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:04:52.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:52.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:04:52.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:52 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:04:52.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:52 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:04:52.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:52 smithi177 ceph-mon[44303]: osdmap e451: 12 total, 12 up, 11 in 2024-05-15T13:04:52.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:52 smithi177 ceph-mon[44303]: pgmap v1207: 17 pgs: 17 active+clean; 8.1 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 59 MiB/s wr, 960 op/s 2024-05-15T13:04:52.457 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:04:52.487 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:52.502 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:52 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:04:52.502 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:52 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:04:52.502 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:52 smithi027 ceph-mon[39795]: osdmap e451: 12 total, 12 up, 11 in 2024-05-15T13:04:52.502 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:52 smithi027 ceph-mon[39795]: pgmap v1207: 17 pgs: 17 active+clean; 8.1 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 59 MiB/s wr, 960 op/s 2024-05-15T13:04:52.518 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:04:52.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:52.521 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:04:52.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:52 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:04:52.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:52 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:04:52.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:52 smithi120 ceph-mon[43961]: osdmap e451: 12 total, 12 up, 11 in 2024-05-15T13:04:52.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:52 smithi120 ceph-mon[43961]: pgmap v1207: 17 pgs: 17 active+clean; 8.1 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 59 MiB/s wr, 960 op/s 2024-05-15T13:04:52.580 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:04:52.586 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:04:52.622 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:04:52.635 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:04:52.651 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:52.674 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:04:52.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:52.684 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:04:52.785 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:04:52.830 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:52.901 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:04:52.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:52.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:04:53.016 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:04:53.089 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:53.136 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:04:53.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:53.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:04:53.244 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:04:53.274 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:53.307 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:04:53.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:53.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:04:53.412 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:04:53.444 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:53.472 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:04:53.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:53.475 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:04:53.576 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:04:53.622 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:53.648 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:04:53.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:53.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:04:53.758 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:04:53.808 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:53.844 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:04:53.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:53.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:04:53.950 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:04:54.001 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:54.051 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:04:54.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:54.054 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:04:54.155 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:04:54.201 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:54.226 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:04:54.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:54.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:04:54.337 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:04:54.378 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:54.409 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:04:54.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:54.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:04:54.517 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:04:54.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:54 smithi120 ceph-mon[43961]: pgmap v1208: 17 pgs: 17 active+clean; 8.2 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 127 B/s rd, 59 MiB/s wr, 1.07k op/s 2024-05-15T13:04:54.552 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:04:54.561 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:54 smithi027 ceph-mon[39795]: pgmap v1208: 17 pgs: 17 active+clean; 8.2 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 127 B/s rd, 59 MiB/s wr, 1.07k op/s 2024-05-15T13:04:54.567 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:54.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:54 smithi177 ceph-mon[44303]: pgmap v1208: 17 pgs: 17 active+clean; 8.2 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 127 B/s rd, 59 MiB/s wr, 1.07k op/s 2024-05-15T13:04:54.712 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:04:54.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:54.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:04:54.829 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:04:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:55.554 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:04:55.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:55.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:04:55.661 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:04:55.696 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:55.724 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:04:55.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:55.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:04:55.829 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:04:55.874 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:57.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:57 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:04:57.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:57 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:04:57.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:57 smithi027 ceph-mon[39795]: osdmap e452: 12 total, 12 up, 11 in 2024-05-15T13:04:57.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:57 smithi027 ceph-mon[39795]: pgmap v1210: 17 pgs: 17 active+clean; 8.1 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 170 B/s rd, 33 MiB/s wr, 886 op/s 2024-05-15T13:04:57.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:57 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:04:57.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:57 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:04:57.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:57 smithi120 ceph-mon[43961]: osdmap e452: 12 total, 12 up, 11 in 2024-05-15T13:04:57.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:57 smithi120 ceph-mon[43961]: pgmap v1210: 17 pgs: 17 active+clean; 8.1 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 170 B/s rd, 33 MiB/s wr, 886 op/s 2024-05-15T13:04:57.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:57 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:04:57.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:57 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:04:57.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:57 smithi177 ceph-mon[44303]: osdmap e452: 12 total, 12 up, 11 in 2024-05-15T13:04:57.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:57 smithi177 ceph-mon[44303]: pgmap v1210: 17 pgs: 17 active+clean; 8.1 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 170 B/s rd, 33 MiB/s wr, 886 op/s 2024-05-15T13:04:57.504 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:04:57.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:57.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:04:57.609 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:04:57.662 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:57.695 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:04:57.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:57.699 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:04:57.800 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:04:57.862 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:58.158 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:04:58.194 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:04:58.225 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:04:58.271 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:04:58.314 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:04:58.356 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:04:58.384 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:04:58.423 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:04:58.468 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:04:58.510 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:04:58.542 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:04:58.582 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:04:58.600 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:04:58.600 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:04:58.600 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:04:58.600 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:04:58.600 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:04:58.600 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:04:58.601 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:04:58.601 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:04:58.601 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:04:58.601 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:04:58.601 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:04:58.601 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:04:59.046 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:04:59.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:59.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:04:59.157 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:04:59.195 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:59.227 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:04:59.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:59.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:04:59.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:59 smithi120 ceph-mon[43961]: pgmap v1211: 17 pgs: 17 active+clean; 7.9 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 129 B/s rd, 25 MiB/s wr, 1.05k op/s 2024-05-15T13:04:59.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:04:59 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1919898112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:04:59.293 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:59 smithi027 ceph-mon[39795]: pgmap v1211: 17 pgs: 17 active+clean; 7.9 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 129 B/s rd, 25 MiB/s wr, 1.05k op/s 2024-05-15T13:04:59.293 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:04:59 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1919898112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:04:59.334 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:04:59.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:59 smithi177 ceph-mon[44303]: pgmap v1211: 17 pgs: 17 active+clean; 7.9 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 129 B/s rd, 25 MiB/s wr, 1.05k op/s 2024-05-15T13:04:59.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:04:59 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1919898112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:04:59.368 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:59.394 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:04:59.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:59.400 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:04:59.501 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:04:59.534 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:04:59.554 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:04:59.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:04:59.557 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:04:59.631 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:04:59.632 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T13:04:59.632 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-05-15T13:04:59.632 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-05-15T13:04:59.632 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:04:59.658 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:04:59.740 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:00.219 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:05:00.219 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:05:00.219 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 452, 2024-05-15T13:05:00.219 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "deaea04e-12b7-11ef-bc98-c7b262605968", 2024-05-15T13:05:00.219 INFO:teuthology.orchestra.run.smithi027.stdout: "created": "2024-05-15T12:38:21.244556+0000", 2024-05-15T13:05:00.219 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-05-15T13:04:55.812915+0000", 2024-05-15T13:05:00.219 INFO:teuthology.orchestra.run.smithi027.stdout: "last_up_change": "2024-05-15T12:45:50.876090+0000", 2024-05-15T13:05:00.219 INFO:teuthology.orchestra.run.smithi027.stdout: "last_in_change": "2024-05-15T12:59:10.294721+0000", 2024-05-15T13:05:00.219 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-15T13:05:00.220 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_num": 5806080, 2024-05-15T13:05:00.220 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_set": [ 2024-05-15T13:05:00.220 INFO:teuthology.orchestra.run.smithi027.stdout: "nodeep-scrub", 2024-05-15T13:05:00.220 INFO:teuthology.orchestra.run.smithi027.stdout: "noscrub", 2024-05-15T13:05:00.220 INFO:teuthology.orchestra.run.smithi027.stdout: "pglog_hardlimit", 2024-05-15T13:05:00.220 INFO:teuthology.orchestra.run.smithi027.stdout: "purged_snapdirs", 2024-05-15T13:05:00.220 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_deletes", 2024-05-15T13:05:00.220 INFO:teuthology.orchestra.run.smithi027.stdout: "sortbitwise" 2024-05-15T13:05:00.220 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:05:00.221 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_version": 26, 2024-05-15T13:05:00.221 INFO:teuthology.orchestra.run.smithi027.stdout: "full_ratio": 0.94999998807907104, 2024-05-15T13:05:00.221 INFO:teuthology.orchestra.run.smithi027.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-05-15T13:05:00.221 INFO:teuthology.orchestra.run.smithi027.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-15T13:05:00.221 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_snapshot": "", 2024-05-15T13:05:00.221 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_max": 5, 2024-05-15T13:05:00.221 INFO:teuthology.orchestra.run.smithi027.stdout: "max_osd": 12, 2024-05-15T13:05:00.222 INFO:teuthology.orchestra.run.smithi027.stdout: "require_min_compat_client": "luminous", 2024-05-15T13:05:00.222 INFO:teuthology.orchestra.run.smithi027.stdout: "min_compat_client": "luminous", 2024-05-15T13:05:00.222 INFO:teuthology.orchestra.run.smithi027.stdout: "require_osd_release": "squid", 2024-05-15T13:05:00.222 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_crimson": false, 2024-05-15T13:05:00.222 INFO:teuthology.orchestra.run.smithi027.stdout: "pools": [ 2024-05-15T13:05:00.222 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.223 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 1, 2024-05-15T13:05:00.223 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": ".mgr", 2024-05-15T13:05:00.223 INFO:teuthology.orchestra.run.smithi027.stdout: "create_time": "2024-05-15T12:41:57.948613+0000", 2024-05-15T13:05:00.223 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-15T13:05:00.223 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_names": "hashpspool", 2024-05-15T13:05:00.223 INFO:teuthology.orchestra.run.smithi027.stdout: "type": 1, 2024-05-15T13:05:00.224 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 3, 2024-05-15T13:05:00.224 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": 2, 2024-05-15T13:05:00.224 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_rule": 0, 2024-05-15T13:05:00.224 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_count": 0, 2024-05-15T13:05:00.224 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_target": 0, 2024-05-15T13:05:00.224 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T13:05:00.224 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T13:05:00.225 INFO:teuthology.orchestra.run.smithi027.stdout: "object_hash": 2, 2024-05-15T13:05:00.225 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscale_mode": "off", 2024-05-15T13:05:00.225 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": 1, 2024-05-15T13:05:00.225 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num": 1, 2024-05-15T13:05:00.225 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num_target": 1, 2024-05-15T13:05:00.225 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_target": 1, 2024-05-15T13:05:00.226 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_pending": 1, 2024-05-15T13:05:00.226 INFO:teuthology.orchestra.run.smithi027.stdout: "last_pg_merge_meta": { 2024-05-15T13:05:00.226 INFO:teuthology.orchestra.run.smithi027.stdout: "source_pgid": "0.0", 2024-05-15T13:05:00.226 INFO:teuthology.orchestra.run.smithi027.stdout: "ready_epoch": 0, 2024-05-15T13:05:00.226 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_started": 0, 2024-05-15T13:05:00.226 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_clean": 0, 2024-05-15T13:05:00.226 INFO:teuthology.orchestra.run.smithi027.stdout: "source_version": "0'0", 2024-05-15T13:05:00.227 INFO:teuthology.orchestra.run.smithi027.stdout: "target_version": "0'0" 2024-05-15T13:05:00.227 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.227 INFO:teuthology.orchestra.run.smithi027.stdout: "last_change": "21", 2024-05-15T13:05:00.227 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend": "0", 2024-05-15T13:05:00.227 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T13:05:00.227 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T13:05:00.228 INFO:teuthology.orchestra.run.smithi027.stdout: "auid": 0, 2024-05-15T13:05:00.228 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_mode": "selfmanaged", 2024-05-15T13:05:00.228 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_seq": 0, 2024-05-15T13:05:00.228 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_epoch": 0, 2024-05-15T13:05:00.228 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_snaps": [], 2024-05-15T13:05:00.228 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps": "[]", 2024-05-15T13:05:00.229 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_bytes": 0, 2024-05-15T13:05:00.229 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_objects": 0, 2024-05-15T13:05:00.229 INFO:teuthology.orchestra.run.smithi027.stdout: "tiers": [], 2024-05-15T13:05:00.229 INFO:teuthology.orchestra.run.smithi027.stdout: "tier_of": -1, 2024-05-15T13:05:00.229 INFO:teuthology.orchestra.run.smithi027.stdout: "read_tier": -1, 2024-05-15T13:05:00.229 INFO:teuthology.orchestra.run.smithi027.stdout: "write_tier": -1, 2024-05-15T13:05:00.230 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_mode": "none", 2024-05-15T13:05:00.230 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_bytes": 0, 2024-05-15T13:05:00.230 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_objects": 0, 2024-05-15T13:05:00.230 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T13:05:00.230 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T13:05:00.230 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T13:05:00.230 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_flush_age": 0, 2024-05-15T13:05:00.230 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_evict_age": 0, 2024-05-15T13:05:00.230 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profile": "", 2024-05-15T13:05:00.230 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_params": { 2024-05-15T13:05:00.230 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "none" 2024-05-15T13:05:00.231 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.231 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_period": 0, 2024-05-15T13:05:00.231 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_count": 0, 2024-05-15T13:05:00.231 INFO:teuthology.orchestra.run.smithi027.stdout: "use_gmt_hitset": true, 2024-05-15T13:05:00.231 INFO:teuthology.orchestra.run.smithi027.stdout: "min_read_recency_for_promote": 0, 2024-05-15T13:05:00.231 INFO:teuthology.orchestra.run.smithi027.stdout: "min_write_recency_for_promote": 0, 2024-05-15T13:05:00.231 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T13:05:00.231 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_search_last_n": 0, 2024-05-15T13:05:00.231 INFO:teuthology.orchestra.run.smithi027.stdout: "grade_table": [], 2024-05-15T13:05:00.231 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_width": 0, 2024-05-15T13:05:00.231 INFO:teuthology.orchestra.run.smithi027.stdout: "expected_num_objects": 0, 2024-05-15T13:05:00.231 INFO:teuthology.orchestra.run.smithi027.stdout: "fast_read": false, 2024-05-15T13:05:00.231 INFO:teuthology.orchestra.run.smithi027.stdout: "options": { 2024-05-15T13:05:00.231 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_max": 32, 2024-05-15T13:05:00.232 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_min": 1 2024-05-15T13:05:00.232 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.232 INFO:teuthology.orchestra.run.smithi027.stdout: "application_metadata": { 2024-05-15T13:05:00.232 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": {} 2024-05-15T13:05:00.232 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.232 INFO:teuthology.orchestra.run.smithi027.stdout: "read_balance": { 2024-05-15T13:05:00.232 INFO:teuthology.orchestra.run.smithi027.stdout: "score_type": "Fair distribution", 2024-05-15T13:05:00.232 INFO:teuthology.orchestra.run.smithi027.stdout: "score_acting": 12, 2024-05-15T13:05:00.232 INFO:teuthology.orchestra.run.smithi027.stdout: "score_stable": 12, 2024-05-15T13:05:00.232 INFO:teuthology.orchestra.run.smithi027.stdout: "optimal_score": 0.25, 2024-05-15T13:05:00.232 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_acting": 3, 2024-05-15T13:05:00.232 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_stable": 3, 2024-05-15T13:05:00.232 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity_weighted": 0.94652295112609863, 2024-05-15T13:05:00.233 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity": 0.94999998807907104, 2024-05-15T13:05:00.233 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-05-15T13:05:00.233 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.233 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.233 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.233 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 5, 2024-05-15T13:05:00.233 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": "unique_pool_3", 2024-05-15T13:05:00.233 INFO:teuthology.orchestra.run.smithi027.stdout: "create_time": "2024-05-15T13:03:10.005384+0000", 2024-05-15T13:05:00.233 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-15T13:05:00.233 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_names": "hashpspool", 2024-05-15T13:05:00.233 INFO:teuthology.orchestra.run.smithi027.stdout: "type": 1, 2024-05-15T13:05:00.233 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 3, 2024-05-15T13:05:00.233 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": 2, 2024-05-15T13:05:00.233 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_rule": 0, 2024-05-15T13:05:00.234 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_count": 0, 2024-05-15T13:05:00.234 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_target": 0, 2024-05-15T13:05:00.234 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T13:05:00.234 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T13:05:00.234 INFO:teuthology.orchestra.run.smithi027.stdout: "object_hash": 2, 2024-05-15T13:05:00.234 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscale_mode": "off", 2024-05-15T13:05:00.234 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": 16, 2024-05-15T13:05:00.234 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num": 16, 2024-05-15T13:05:00.234 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num_target": 16, 2024-05-15T13:05:00.234 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_target": 16, 2024-05-15T13:05:00.234 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_pending": 16, 2024-05-15T13:05:00.234 INFO:teuthology.orchestra.run.smithi027.stdout: "last_pg_merge_meta": { 2024-05-15T13:05:00.235 INFO:teuthology.orchestra.run.smithi027.stdout: "source_pgid": "0.0", 2024-05-15T13:05:00.235 INFO:teuthology.orchestra.run.smithi027.stdout: "ready_epoch": 0, 2024-05-15T13:05:00.235 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_started": 0, 2024-05-15T13:05:00.235 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_clean": 0, 2024-05-15T13:05:00.235 INFO:teuthology.orchestra.run.smithi027.stdout: "source_version": "0'0", 2024-05-15T13:05:00.235 INFO:teuthology.orchestra.run.smithi027.stdout: "target_version": "0'0" 2024-05-15T13:05:00.235 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.235 INFO:teuthology.orchestra.run.smithi027.stdout: "last_change": "423", 2024-05-15T13:05:00.235 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend": "0", 2024-05-15T13:05:00.235 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T13:05:00.236 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T13:05:00.236 INFO:teuthology.orchestra.run.smithi027.stdout: "auid": 0, 2024-05-15T13:05:00.236 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_mode": "selfmanaged", 2024-05-15T13:05:00.236 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_seq": 0, 2024-05-15T13:05:00.236 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_epoch": 0, 2024-05-15T13:05:00.236 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_snaps": [], 2024-05-15T13:05:00.236 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps": "[]", 2024-05-15T13:05:00.236 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_bytes": 0, 2024-05-15T13:05:00.236 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_objects": 0, 2024-05-15T13:05:00.236 INFO:teuthology.orchestra.run.smithi027.stdout: "tiers": [], 2024-05-15T13:05:00.236 INFO:teuthology.orchestra.run.smithi027.stdout: "tier_of": -1, 2024-05-15T13:05:00.236 INFO:teuthology.orchestra.run.smithi027.stdout: "read_tier": -1, 2024-05-15T13:05:00.236 INFO:teuthology.orchestra.run.smithi027.stdout: "write_tier": -1, 2024-05-15T13:05:00.236 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_mode": "none", 2024-05-15T13:05:00.237 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_bytes": 0, 2024-05-15T13:05:00.237 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_objects": 0, 2024-05-15T13:05:00.237 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T13:05:00.237 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T13:05:00.237 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T13:05:00.237 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_flush_age": 0, 2024-05-15T13:05:00.237 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_evict_age": 0, 2024-05-15T13:05:00.237 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profile": "", 2024-05-15T13:05:00.237 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_params": { 2024-05-15T13:05:00.237 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "none" 2024-05-15T13:05:00.238 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.238 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_period": 0, 2024-05-15T13:05:00.238 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_count": 0, 2024-05-15T13:05:00.238 INFO:teuthology.orchestra.run.smithi027.stdout: "use_gmt_hitset": true, 2024-05-15T13:05:00.238 INFO:teuthology.orchestra.run.smithi027.stdout: "min_read_recency_for_promote": 0, 2024-05-15T13:05:00.238 INFO:teuthology.orchestra.run.smithi027.stdout: "min_write_recency_for_promote": 0, 2024-05-15T13:05:00.238 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T13:05:00.238 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_search_last_n": 0, 2024-05-15T13:05:00.238 INFO:teuthology.orchestra.run.smithi027.stdout: "grade_table": [], 2024-05-15T13:05:00.238 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_width": 0, 2024-05-15T13:05:00.238 INFO:teuthology.orchestra.run.smithi027.stdout: "expected_num_objects": 0, 2024-05-15T13:05:00.238 INFO:teuthology.orchestra.run.smithi027.stdout: "fast_read": false, 2024-05-15T13:05:00.238 INFO:teuthology.orchestra.run.smithi027.stdout: "options": {}, 2024-05-15T13:05:00.239 INFO:teuthology.orchestra.run.smithi027.stdout: "application_metadata": { 2024-05-15T13:05:00.239 INFO:teuthology.orchestra.run.smithi027.stdout: "rados": {} 2024-05-15T13:05:00.239 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.239 INFO:teuthology.orchestra.run.smithi027.stdout: "read_balance": { 2024-05-15T13:05:00.239 INFO:teuthology.orchestra.run.smithi027.stdout: "score_type": "Fair distribution", 2024-05-15T13:05:00.239 INFO:teuthology.orchestra.run.smithi027.stdout: "score_acting": 2.2400000095367432, 2024-05-15T13:05:00.239 INFO:teuthology.orchestra.run.smithi027.stdout: "score_stable": 2.2400000095367432, 2024-05-15T13:05:00.239 INFO:teuthology.orchestra.run.smithi027.stdout: "optimal_score": 0.92000001668930054, 2024-05-15T13:05:00.239 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_acting": 2.059999942779541, 2024-05-15T13:05:00.239 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_stable": 2.059999942779541, 2024-05-15T13:05:00.239 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity_weighted": 0.94652295112609863, 2024-05-15T13:05:00.239 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity": 0.94999998807907104, 2024-05-15T13:05:00.239 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-05-15T13:05:00.239 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.240 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.240 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:05:00.240 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T13:05:00.240 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.240 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 0, 2024-05-15T13:05:00.240 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "64fb3ad2-08a7-4106-a07a-6c776c877faf", 2024-05-15T13:05:00.240 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:05:00.240 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:05:00.240 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:05:00.240 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:05:00.240 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:05:00.240 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:05:00.240 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 8, 2024-05-15T13:05:00.240 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 418, 2024-05-15T13:05:00.240 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:05:00.241 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:05:00.241 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:05:00.241 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.241 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.241 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.241 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6802", 2024-05-15T13:05:00.241 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:05:00.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.241 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.241 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.241 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6803", 2024-05-15T13:05:00.241 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:05:00.241 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.241 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.242 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.242 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:05:00.242 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.242 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.242 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.242 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6804", 2024-05-15T13:05:00.242 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:05:00.242 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.242 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.242 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.242 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6805", 2024-05-15T13:05:00.242 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:05:00.243 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.243 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.243 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.243 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:05:00.243 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.243 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.243 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.243 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6808", 2024-05-15T13:05:00.243 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:05:00.243 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.243 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.243 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.243 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6809", 2024-05-15T13:05:00.243 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:05:00.244 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.244 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.244 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.244 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:05:00.244 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.244 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.244 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.244 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6806", 2024-05-15T13:05:00.244 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:05:00.244 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.244 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.244 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.244 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6807", 2024-05-15T13:05:00.244 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:05:00.244 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.245 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.245 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.245 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6803/4239116935", 2024-05-15T13:05:00.245 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6805/4239116935", 2024-05-15T13:05:00.245 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6809/4239116935", 2024-05-15T13:05:00.245 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6807/4239116935", 2024-05-15T13:05:00.245 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:05:00.245 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:05:00.245 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:05:00.245 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.245 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.245 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.246 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 1, 2024-05-15T13:05:00.246 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "9c16dd90-9d51-48b8-a296-1c8879ba1bc8", 2024-05-15T13:05:00.246 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:05:00.246 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:05:00.246 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:05:00.246 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:05:00.246 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:05:00.246 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:05:00.246 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 13, 2024-05-15T13:05:00.246 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 418, 2024-05-15T13:05:00.247 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:05:00.247 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:05:00.247 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:05:00.247 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.247 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.247 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.247 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6810", 2024-05-15T13:05:00.247 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:05:00.247 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.247 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.247 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.247 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6811", 2024-05-15T13:05:00.247 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:05:00.247 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.248 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.248 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.248 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:05:00.248 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.248 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.248 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.248 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6812", 2024-05-15T13:05:00.248 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:05:00.248 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.248 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.248 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.248 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6813", 2024-05-15T13:05:00.248 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:05:00.249 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.249 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.249 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.249 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:05:00.249 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.249 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.249 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.249 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6816", 2024-05-15T13:05:00.249 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:05:00.249 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.249 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.249 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.249 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6817", 2024-05-15T13:05:00.249 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:05:00.250 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.250 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.250 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.250 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:05:00.250 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.250 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.250 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.250 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6814", 2024-05-15T13:05:00.250 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:05:00.250 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.250 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.251 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.251 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6815", 2024-05-15T13:05:00.251 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:05:00.251 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.251 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.251 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.251 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6811/1466270637", 2024-05-15T13:05:00.251 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6813/1466270637", 2024-05-15T13:05:00.251 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6817/1466270637", 2024-05-15T13:05:00.251 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6815/1466270637", 2024-05-15T13:05:00.251 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:05:00.251 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:05:00.251 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:05:00.252 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.252 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.252 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.252 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 2, 2024-05-15T13:05:00.252 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "cbe3ba13-3ea3-4891-8d2d-6b85581a732c", 2024-05-15T13:05:00.252 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:05:00.252 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:05:00.252 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:05:00.252 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:05:00.252 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:05:00.252 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:05:00.252 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 18, 2024-05-15T13:05:00.252 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 418, 2024-05-15T13:05:00.252 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:05:00.252 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:05:00.253 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:05:00.253 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.253 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.253 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.253 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6818", 2024-05-15T13:05:00.253 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:05:00.253 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.253 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.253 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.253 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6819", 2024-05-15T13:05:00.253 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:05:00.254 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.254 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.254 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.254 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:05:00.254 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.254 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.254 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.254 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6820", 2024-05-15T13:05:00.254 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:05:00.254 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.254 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.254 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.255 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6821", 2024-05-15T13:05:00.255 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:05:00.255 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.255 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.255 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.255 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:05:00.255 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.255 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.255 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.255 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6824", 2024-05-15T13:05:00.255 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:05:00.255 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.255 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.255 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.256 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6825", 2024-05-15T13:05:00.256 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:05:00.256 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.256 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.256 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.256 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:05:00.256 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.256 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.256 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.256 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6822", 2024-05-15T13:05:00.256 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:05:00.256 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.256 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.256 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.256 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6823", 2024-05-15T13:05:00.256 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:05:00.257 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.257 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.257 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.257 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6819/3285413897", 2024-05-15T13:05:00.257 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6821/3285413897", 2024-05-15T13:05:00.257 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6825/3285413897", 2024-05-15T13:05:00.257 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6823/3285413897", 2024-05-15T13:05:00.257 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:05:00.257 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:05:00.257 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:05:00.257 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.257 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.257 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.257 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 3, 2024-05-15T13:05:00.257 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "2e83735b-b944-4f87-8080-b994d4b96897", 2024-05-15T13:05:00.257 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:05:00.258 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:05:00.258 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:05:00.258 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:05:00.258 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:05:00.258 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:05:00.258 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 26, 2024-05-15T13:05:00.258 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 317, 2024-05-15T13:05:00.258 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:05:00.258 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:05:00.258 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:05:00.258 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.258 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.258 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.259 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6826", 2024-05-15T13:05:00.259 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:05:00.259 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.259 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.259 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.259 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6827", 2024-05-15T13:05:00.259 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:05:00.259 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.259 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.259 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.259 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:05:00.259 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.259 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.259 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.259 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6828", 2024-05-15T13:05:00.260 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:05:00.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.260 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.260 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.260 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6829", 2024-05-15T13:05:00.260 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:05:00.260 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.260 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.260 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:05:00.260 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.260 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.260 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.260 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6832", 2024-05-15T13:05:00.260 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:05:00.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.261 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.261 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.261 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6833", 2024-05-15T13:05:00.261 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:05:00.261 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.261 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.261 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:05:00.261 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.261 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.261 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.261 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6830", 2024-05-15T13:05:00.261 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:05:00.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.261 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.262 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.262 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6831", 2024-05-15T13:05:00.262 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:05:00.262 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.262 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.262 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6827/3008975244", 2024-05-15T13:05:00.262 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6829/3008975244", 2024-05-15T13:05:00.262 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6833/3008975244", 2024-05-15T13:05:00.262 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6831/3008975244", 2024-05-15T13:05:00.262 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:05:00.262 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:05:00.262 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:05:00.262 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.263 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.263 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.263 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 4, 2024-05-15T13:05:00.263 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "7c39c1dd-3867-46f7-8372-7688fa13b16a", 2024-05-15T13:05:00.263 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:05:00.263 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:05:00.263 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:05:00.263 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:05:00.263 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:05:00.263 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:05:00.263 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 31, 2024-05-15T13:05:00.263 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 418, 2024-05-15T13:05:00.263 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:05:00.264 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:05:00.264 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:05:00.264 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.264 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.264 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.264 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6800", 2024-05-15T13:05:00.264 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:05:00.264 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.264 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.264 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.264 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6801", 2024-05-15T13:05:00.264 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:05:00.264 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.265 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.265 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.265 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:05:00.265 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.265 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.265 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.265 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6802", 2024-05-15T13:05:00.265 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:05:00.265 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.265 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.265 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.265 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6803", 2024-05-15T13:05:00.265 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:05:00.265 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.266 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.266 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.266 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:05:00.266 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.266 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.266 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.266 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6806", 2024-05-15T13:05:00.266 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:05:00.266 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.266 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.266 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.266 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6807", 2024-05-15T13:05:00.266 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:05:00.266 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.267 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.267 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.267 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:05:00.267 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.267 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.267 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.267 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6804", 2024-05-15T13:05:00.267 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:05:00.267 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.267 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.267 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.267 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6805", 2024-05-15T13:05:00.267 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:05:00.267 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.267 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.268 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.268 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6801/3877842957", 2024-05-15T13:05:00.268 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6803/3877842957", 2024-05-15T13:05:00.268 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6807/3877842957", 2024-05-15T13:05:00.268 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6805/3877842957", 2024-05-15T13:05:00.268 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:05:00.268 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:05:00.268 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:05:00.268 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.268 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.268 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.268 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 5, 2024-05-15T13:05:00.268 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "a6dc220f-aec0-4454-9695-b44c0b9770e0", 2024-05-15T13:05:00.268 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:05:00.268 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:05:00.269 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:05:00.269 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:05:00.269 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:05:00.269 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:05:00.269 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 40, 2024-05-15T13:05:00.269 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 418, 2024-05-15T13:05:00.269 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:05:00.269 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:05:00.269 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:05:00.269 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.269 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.269 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.269 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6808", 2024-05-15T13:05:00.269 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:05:00.270 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.270 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.270 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.270 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6809", 2024-05-15T13:05:00.270 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:05:00.270 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.270 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.270 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.270 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:05:00.270 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.270 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.270 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.270 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6810", 2024-05-15T13:05:00.271 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:05:00.271 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.271 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.271 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.271 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6811", 2024-05-15T13:05:00.271 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:05:00.271 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.271 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.271 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.271 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:05:00.271 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.271 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.271 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.271 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6814", 2024-05-15T13:05:00.271 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:05:00.271 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.272 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.272 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.272 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6815", 2024-05-15T13:05:00.272 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:05:00.272 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.272 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.272 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.272 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:05:00.272 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.272 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.272 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.272 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6812", 2024-05-15T13:05:00.272 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:05:00.272 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.272 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.273 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.273 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6813", 2024-05-15T13:05:00.273 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:05:00.273 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.273 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.273 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.273 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6809/1600179183", 2024-05-15T13:05:00.273 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6811/1600179183", 2024-05-15T13:05:00.273 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6815/1600179183", 2024-05-15T13:05:00.273 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6813/1600179183", 2024-05-15T13:05:00.273 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:05:00.273 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:05:00.273 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:05:00.273 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.273 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.273 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.274 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 6, 2024-05-15T13:05:00.274 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b", 2024-05-15T13:05:00.274 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:05:00.274 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:05:00.274 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:05:00.274 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:05:00.274 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:05:00.274 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:05:00.274 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 48, 2024-05-15T13:05:00.274 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 418, 2024-05-15T13:05:00.274 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:05:00.274 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:05:00.274 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:05:00.275 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.275 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.275 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.275 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6816", 2024-05-15T13:05:00.275 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:05:00.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.275 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.275 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.275 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6817", 2024-05-15T13:05:00.275 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:05:00.275 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.275 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.275 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:05:00.275 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.275 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.276 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.276 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6818", 2024-05-15T13:05:00.276 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:05:00.276 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.276 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.276 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.276 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6819", 2024-05-15T13:05:00.276 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:05:00.276 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.276 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.276 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.276 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:05:00.276 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.276 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.277 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.277 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6822", 2024-05-15T13:05:00.277 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:05:00.277 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.277 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.277 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.277 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6823", 2024-05-15T13:05:00.277 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:05:00.277 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.277 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.277 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.277 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:05:00.277 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.277 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.277 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.277 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6820", 2024-05-15T13:05:00.278 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:05:00.278 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.278 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.278 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.278 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6821", 2024-05-15T13:05:00.278 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:05:00.278 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.278 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.278 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.278 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6817/3323706735", 2024-05-15T13:05:00.278 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6819/3323706735", 2024-05-15T13:05:00.279 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6823/3323706735", 2024-05-15T13:05:00.279 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6821/3323706735", 2024-05-15T13:05:00.279 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:05:00.279 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:05:00.279 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:05:00.279 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.279 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.279 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.279 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 7, 2024-05-15T13:05:00.279 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "6ef085d8-f59b-4033-8280-2c8f78693e30", 2024-05-15T13:05:00.279 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:05:00.279 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 0, 2024-05-15T13:05:00.279 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 0, 2024-05-15T13:05:00.279 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:05:00.280 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:05:00.280 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:05:00.280 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 56, 2024-05-15T13:05:00.280 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 219, 2024-05-15T13:05:00.280 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:05:00.280 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:05:00.280 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:05:00.280 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.280 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.280 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.280 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6824", 2024-05-15T13:05:00.280 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:05:00.280 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.280 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.280 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.280 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6825", 2024-05-15T13:05:00.281 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:05:00.281 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.281 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.281 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.281 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:05:00.281 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.281 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.281 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.281 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6826", 2024-05-15T13:05:00.281 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:05:00.281 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.281 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.281 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.281 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6827", 2024-05-15T13:05:00.281 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:05:00.282 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.282 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.282 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.282 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:05:00.282 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.282 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.282 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.282 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6830", 2024-05-15T13:05:00.282 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:05:00.282 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.282 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.282 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.282 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6831", 2024-05-15T13:05:00.282 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:05:00.283 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.283 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.283 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.283 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:05:00.283 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.283 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.283 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.283 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6828", 2024-05-15T13:05:00.283 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:05:00.283 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.283 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.283 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.283 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6829", 2024-05-15T13:05:00.283 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:05:00.283 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.284 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.284 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.284 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6825/2960616153", 2024-05-15T13:05:00.284 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6827/2960616153", 2024-05-15T13:05:00.284 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6831/2960616153", 2024-05-15T13:05:00.284 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6829/2960616153", 2024-05-15T13:05:00.284 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:05:00.284 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:05:00.284 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:05:00.284 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.284 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.284 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.284 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 8, 2024-05-15T13:05:00.284 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "26514b11-d02b-46e9-a4e8-3da193fcdbcc", 2024-05-15T13:05:00.284 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:05:00.285 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:05:00.285 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:05:00.285 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:05:00.285 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:05:00.285 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:05:00.285 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 65, 2024-05-15T13:05:00.285 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 292, 2024-05-15T13:05:00.285 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:05:00.285 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:05:00.285 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:05:00.285 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.285 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.285 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.285 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6800", 2024-05-15T13:05:00.285 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:05:00.286 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.286 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.286 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.286 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6801", 2024-05-15T13:05:00.286 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:05:00.286 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.286 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.286 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.286 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:05:00.286 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.286 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.286 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.286 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6802", 2024-05-15T13:05:00.287 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:05:00.287 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.287 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.287 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.287 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6803", 2024-05-15T13:05:00.287 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:05:00.287 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.287 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.287 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.287 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:05:00.287 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.287 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.287 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.287 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6806", 2024-05-15T13:05:00.287 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:05:00.288 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.288 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.288 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.288 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6807", 2024-05-15T13:05:00.288 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:05:00.288 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.288 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.288 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.288 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:05:00.288 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.288 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.288 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.288 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6804", 2024-05-15T13:05:00.288 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:05:00.289 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.289 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.289 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.289 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6805", 2024-05-15T13:05:00.289 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:05:00.289 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.289 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.289 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.289 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6801/1132459315", 2024-05-15T13:05:00.289 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6803/1132459315", 2024-05-15T13:05:00.289 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6807/1132459315", 2024-05-15T13:05:00.289 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6805/1132459315", 2024-05-15T13:05:00.289 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:05:00.289 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:05:00.290 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:05:00.290 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.290 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.290 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.290 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 9, 2024-05-15T13:05:00.290 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "6f9497bc-e962-409d-8a2c-0c634cc1124f", 2024-05-15T13:05:00.290 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:05:00.290 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:05:00.290 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:05:00.290 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 0.9062042236328125, 2024-05-15T13:05:00.290 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:05:00.290 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:05:00.290 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 76, 2024-05-15T13:05:00.291 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 418, 2024-05-15T13:05:00.291 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:05:00.291 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:05:00.291 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:05:00.291 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.291 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.291 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.291 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6808", 2024-05-15T13:05:00.291 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:05:00.291 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.291 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.291 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.291 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6809", 2024-05-15T13:05:00.291 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:05:00.292 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.292 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.292 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.292 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:05:00.292 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.292 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.292 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.292 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6810", 2024-05-15T13:05:00.292 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:05:00.292 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.292 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.292 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.292 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6811", 2024-05-15T13:05:00.293 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:05:00.293 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.293 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.293 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.293 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:05:00.293 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.293 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.293 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.293 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6814", 2024-05-15T13:05:00.293 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:05:00.293 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.293 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.293 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.293 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6815", 2024-05-15T13:05:00.294 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:05:00.294 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.294 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.294 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.294 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:05:00.294 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.294 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.294 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.294 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6812", 2024-05-15T13:05:00.294 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:05:00.294 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.294 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.294 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.295 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6813", 2024-05-15T13:05:00.295 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:05:00.295 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.295 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.295 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.295 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6809/2710457646", 2024-05-15T13:05:00.295 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6811/2710457646", 2024-05-15T13:05:00.295 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6815/2710457646", 2024-05-15T13:05:00.295 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6813/2710457646", 2024-05-15T13:05:00.295 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:05:00.295 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:05:00.295 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:05:00.296 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.296 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.296 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.296 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 10, 2024-05-15T13:05:00.296 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "f852876f-c7d1-4039-90c4-5ab09dbd3e32", 2024-05-15T13:05:00.296 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:05:00.296 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:05:00.296 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:05:00.296 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 0.4520721435546875, 2024-05-15T13:05:00.296 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:05:00.297 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:05:00.297 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 81, 2024-05-15T13:05:00.297 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 418, 2024-05-15T13:05:00.297 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:05:00.297 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:05:00.297 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:05:00.297 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.297 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.297 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.297 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6816", 2024-05-15T13:05:00.298 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:05:00.298 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.298 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.298 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.298 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6817", 2024-05-15T13:05:00.298 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:05:00.298 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.298 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.298 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.298 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:05:00.298 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.298 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.299 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.299 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6818", 2024-05-15T13:05:00.299 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:05:00.299 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.299 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.299 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.299 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6819", 2024-05-15T13:05:00.299 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:05:00.299 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.299 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.299 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.299 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:05:00.299 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.299 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.300 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.300 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6822", 2024-05-15T13:05:00.300 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:05:00.300 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.300 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.300 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.300 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6823", 2024-05-15T13:05:00.300 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:05:00.300 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.300 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.300 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.300 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:05:00.300 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.300 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.301 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.301 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6820", 2024-05-15T13:05:00.301 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:05:00.301 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.301 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.301 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.301 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6821", 2024-05-15T13:05:00.301 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:05:00.301 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.301 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.301 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.301 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6817/3236900263", 2024-05-15T13:05:00.301 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6819/3236900263", 2024-05-15T13:05:00.302 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6823/3236900263", 2024-05-15T13:05:00.302 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6821/3236900263", 2024-05-15T13:05:00.302 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:05:00.302 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:05:00.302 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:05:00.302 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.302 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.302 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.302 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 11, 2024-05-15T13:05:00.302 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "cccc53c5-8476-4d55-b160-4bdc5cf92902", 2024-05-15T13:05:00.302 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:05:00.302 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:05:00.302 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:05:00.303 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:05:00.303 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:05:00.303 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:05:00.303 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 90, 2024-05-15T13:05:00.303 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 418, 2024-05-15T13:05:00.303 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:05:00.303 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:05:00.303 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:05:00.303 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.303 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.303 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.303 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6824", 2024-05-15T13:05:00.304 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:05:00.304 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.304 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.304 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.304 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6825", 2024-05-15T13:05:00.304 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:05:00.304 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.304 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.304 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.304 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:05:00.304 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.304 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.304 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.304 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6826", 2024-05-15T13:05:00.305 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:05:00.305 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.305 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.305 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.305 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6827", 2024-05-15T13:05:00.305 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:05:00.305 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.305 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.305 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.305 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:05:00.305 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.305 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.305 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.305 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6830", 2024-05-15T13:05:00.306 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:05:00.306 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.306 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.306 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.306 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6831", 2024-05-15T13:05:00.306 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:05:00.306 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.306 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.306 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.306 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:05:00.306 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:05:00.306 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.306 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:05:00.307 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6828", 2024-05-15T13:05:00.307 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:05:00.307 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.307 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.307 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:05:00.307 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6829", 2024-05-15T13:05:00.307 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:05:00.307 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.307 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.307 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.307 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6825/681011628", 2024-05-15T13:05:00.307 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6827/681011628", 2024-05-15T13:05:00.308 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6831/681011628", 2024-05-15T13:05:00.308 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6829/681011628", 2024-05-15T13:05:00.308 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:05:00.308 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:05:00.308 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:05:00.308 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.308 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.308 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:05:00.308 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_xinfo": [ 2024-05-15T13:05:00.308 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.308 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 0, 2024-05-15T13:05:00.308 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:05:00.308 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:05:00.308 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:05:00.309 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:05:00.309 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:05:00.309 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:04:30.836275+0000", 2024-05-15T13:05:00.309 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:05:00.309 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.309 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.309 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 1, 2024-05-15T13:05:00.309 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:05:00.309 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:05:00.309 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:05:00.309 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:05:00.309 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:05:00.309 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:04:20.122087+0000", 2024-05-15T13:05:00.309 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:05:00.310 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.310 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.310 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 2, 2024-05-15T13:05:00.310 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:05:00.310 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:05:00.310 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:05:00.310 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:05:00.310 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:05:00.310 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:04:50.651769+0000", 2024-05-15T13:05:00.310 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:05:00.310 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.310 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.311 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 3, 2024-05-15T13:05:00.311 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:05:00.311 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:05:00.311 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:05:00.311 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:05:00.311 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:05:00.311 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:03:40.680176+0000", 2024-05-15T13:05:00.311 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:05:00.311 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.311 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.311 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 4, 2024-05-15T13:05:00.311 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:05:00.311 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:05:00.311 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:05:00.312 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:05:00.312 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:05:00.312 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:03:44.788460+0000", 2024-05-15T13:05:00.312 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:05:00.312 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.312 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.312 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 5, 2024-05-15T13:05:00.312 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:05:00.312 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:05:00.312 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:05:00.312 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:05:00.312 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:05:00.312 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:04:00.460801+0000", 2024-05-15T13:05:00.312 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:05:00.313 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.313 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.313 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 6, 2024-05-15T13:05:00.313 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:05:00.313 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:05:00.313 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:05:00.313 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:05:00.313 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:05:00.313 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:04:24.737473+0000", 2024-05-15T13:05:00.313 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:05:00.313 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.313 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.313 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 7, 2024-05-15T13:05:00.313 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:05:00.313 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:05:00.314 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:05:00.314 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:05:00.314 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 65536, 2024-05-15T13:05:00.314 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:04:40.695993+0000", 2024-05-15T13:05:00.314 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:05:00.314 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.314 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.314 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 8, 2024-05-15T13:05:00.314 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:05:00.314 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:05:00.314 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:05:00.314 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:05:00.315 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:05:00.315 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:04:23.489780+0000", 2024-05-15T13:05:00.315 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:05:00.315 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.315 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.315 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 9, 2024-05-15T13:05:00.315 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:05:00.315 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:05:00.315 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:05:00.315 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:05:00.315 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:05:00.316 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:04:20.144896+0000", 2024-05-15T13:05:00.316 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:05:00.316 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.316 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.316 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 10, 2024-05-15T13:05:00.316 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:05:00.316 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:05:00.316 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:05:00.316 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:05:00.316 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:05:00.316 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:04:49.434839+0000", 2024-05-15T13:05:00.316 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:05:00.316 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.317 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.317 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 11, 2024-05-15T13:05:00.317 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:05:00.317 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:05:00.317 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:05:00.317 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:05:00.317 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:05:00.317 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:04:55.752123+0000", 2024-05-15T13:05:00.317 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:05:00.317 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.317 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:05:00.318 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap": [ 2024-05-15T13:05:00.318 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:00.318 INFO:teuthology.orchestra.run.smithi027.stdout: "pgid": "1.0", 2024-05-15T13:05:00.318 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T13:05:00.318 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-05-15T13:05:00.318 INFO:teuthology.orchestra.run.smithi027.stdout: 6, 2024-05-15T13:05:00.318 INFO:teuthology.orchestra.run.smithi027.stdout: 10 2024-05-15T13:05:00.318 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:00.318 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.318 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:05:00.318 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap_items": [], 2024-05-15T13:05:00.318 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap_primaries": [], 2024-05-15T13:05:00.318 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_temp": [], 2024-05-15T13:05:00.319 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_temp": [], 2024-05-15T13:05:00.319 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklist": { 2024-05-15T13:05:00.319 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6801/2530617473": "2024-05-16T12:39:19.409195+0000", 2024-05-15T13:05:00.319 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6800/2530617473": "2024-05-16T12:39:19.409195+0000", 2024-05-15T13:05:00.319 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/1880261204": "2024-05-16T12:39:19.409195+0000", 2024-05-15T13:05:00.319 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6801/3441136471": "2024-05-16T12:38:50.931011+0000", 2024-05-15T13:05:00.319 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/4291446247": "2024-05-16T12:39:19.409195+0000", 2024-05-15T13:05:00.319 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/580128209": "2024-05-16T12:38:50.931011+0000", 2024-05-15T13:05:00.319 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6800/3441136471": "2024-05-16T12:38:50.931011+0000", 2024-05-15T13:05:00.319 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/3011229857": "2024-05-16T12:38:50.931011+0000", 2024-05-15T13:05:00.319 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/2109693913": "2024-05-16T12:38:50.931011+0000", 2024-05-15T13:05:00.319 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/2445663024": "2024-05-16T12:39:19.409195+0000" 2024-05-15T13:05:00.319 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.320 INFO:teuthology.orchestra.run.smithi027.stdout: "range_blocklist": {}, 2024-05-15T13:05:00.320 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profiles": { 2024-05-15T13:05:00.320 INFO:teuthology.orchestra.run.smithi027.stdout: "default": { 2024-05-15T13:05:00.320 INFO:teuthology.orchestra.run.smithi027.stdout: "crush-failure-domain": "osd", 2024-05-15T13:05:00.320 INFO:teuthology.orchestra.run.smithi027.stdout: "k": "2", 2024-05-15T13:05:00.320 INFO:teuthology.orchestra.run.smithi027.stdout: "m": "1", 2024-05-15T13:05:00.320 INFO:teuthology.orchestra.run.smithi027.stdout: "plugin": "jerasure", 2024-05-15T13:05:00.320 INFO:teuthology.orchestra.run.smithi027.stdout: "technique": "reed_sol_van" 2024-05-15T13:05:00.320 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.321 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:00.321 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps_queue": [], 2024-05-15T13:05:00.321 INFO:teuthology.orchestra.run.smithi027.stdout: "new_removed_snaps": [], 2024-05-15T13:05:00.321 INFO:teuthology.orchestra.run.smithi027.stdout: "new_purged_snaps": [], 2024-05-15T13:05:00.321 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_node_flags": {}, 2024-05-15T13:05:00.321 INFO:teuthology.orchestra.run.smithi027.stdout: "device_class_flags": {}, 2024-05-15T13:05:00.321 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode": { 2024-05-15T13:05:00.322 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode_enabled": false, 2024-05-15T13:05:00.322 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_bucket_count": 0, 2024-05-15T13:05:00.322 INFO:teuthology.orchestra.run.smithi027.stdout: "degraded_stretch_mode": 0, 2024-05-15T13:05:00.322 INFO:teuthology.orchestra.run.smithi027.stdout: "recovering_stretch_mode": 0, 2024-05-15T13:05:00.322 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode_bucket": 0 2024-05-15T13:05:00.323 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:00.323 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:05:01.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:01 smithi027 ceph-mon[39795]: pgmap v1212: 17 pgs: 17 active+clean; 7.8 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 115 B/s rd, 20 MiB/s wr, 1.12k op/s 2024-05-15T13:05:01.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:01 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1950629674' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T13:05:01.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:01 smithi177 ceph-mon[44303]: pgmap v1212: 17 pgs: 17 active+clean; 7.8 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 115 B/s rd, 20 MiB/s wr, 1.12k op/s 2024-05-15T13:05:01.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:01 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1950629674' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T13:05:01.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:01 smithi120 ceph-mon[43961]: pgmap v1212: 17 pgs: 17 active+clean; 7.8 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 115 B/s rd, 20 MiB/s wr, 1.12k op/s 2024-05-15T13:05:01.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:01 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1950629674' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T13:05:01.900 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-05-15T13:05:02.569 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:05:02.619 INFO:tasks.thrashosds.thrasher:j is {'epoch': 452, 'fsid': 'deaea04e-12b7-11ef-bc98-c7b262605968', 'created': '2024-05-15T12:38:21.244556+0000', 'modified': '2024-05-15T13:04:55.812915+0000', 'last_up_change': '2024-05-15T12:45:50.876090+0000', 'last_in_change': '2024-05-15T12:59:10.294721+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 5, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-15T12:41:57.948613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9465229511260986, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 5, 'pool_name': 'unique_pool_3', 'create_time': '2024-05-15T13:03:10.005384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '423', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.9465229511260986, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': '64fb3ad2-08a7-4106-a07a-6c776c877faf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 418, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6802', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6803', 'nonce': 4239116935}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6804', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6805', 'nonce': 4239116935}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6808', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6809', 'nonce': 4239116935}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6806', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6807', 'nonce': 4239116935}]}, 'public_addr': '172.21.15.27:6803/4239116935', 'cluster_addr': '172.21.15.27:6805/4239116935', 'heartbeat_back_addr': '172.21.15.27:6809/4239116935', 'heartbeat_front_addr': '172.21.15.27:6807/4239116935', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '9c16dd90-9d51-48b8-a296-1c8879ba1bc8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 418, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6810', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6811', 'nonce': 1466270637}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6812', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6813', 'nonce': 1466270637}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6816', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6817', 'nonce': 1466270637}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6814', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6815', 'nonce': 1466270637}]}, 'public_addr': '172.21.15.27:6811/1466270637', 'cluster_addr': '172.21.15.27:6813/1466270637', 'heartbeat_back_addr': '172.21.15.27:6817/1466270637', 'heartbeat_front_addr': '172.21.15.27:6815/1466270637', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'cbe3ba13-3ea3-4891-8d2d-6b85581a732c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 418, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6818', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6819', 'nonce': 3285413897}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6820', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6821', 'nonce': 3285413897}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6824', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6825', 'nonce': 3285413897}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6822', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6823', 'nonce': 3285413897}]}, 'public_addr': '172.21.15.27:6819/3285413897', 'cluster_addr': '172.21.15.27:6821/3285413897', 'heartbeat_back_addr': '172.21.15.27:6825/3285413897', 'heartbeat_front_addr': '172.21.15.27:6823/3285413897', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2e83735b-b944-4f87-8080-b994d4b96897', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 317, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6826', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6827', 'nonce': 3008975244}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6828', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6829', 'nonce': 3008975244}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6833', 'nonce': 3008975244}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6830', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6831', 'nonce': 3008975244}]}, 'public_addr': '172.21.15.27:6827/3008975244', 'cluster_addr': '172.21.15.27:6829/3008975244', 'heartbeat_back_addr': '172.21.15.27:6833/3008975244', 'heartbeat_front_addr': '172.21.15.27:6831/3008975244', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7c39c1dd-3867-46f7-8372-7688fa13b16a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 418, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6800', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6801', 'nonce': 3877842957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6802', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6803', 'nonce': 3877842957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6806', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6807', 'nonce': 3877842957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6804', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6805', 'nonce': 3877842957}]}, 'public_addr': '172.21.15.120:6801/3877842957', 'cluster_addr': '172.21.15.120:6803/3877842957', 'heartbeat_back_addr': '172.21.15.120:6807/3877842957', 'heartbeat_front_addr': '172.21.15.120:6805/3877842957', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a6dc220f-aec0-4454-9695-b44c0b9770e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 418, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6808', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6809', 'nonce': 1600179183}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6810', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6811', 'nonce': 1600179183}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6814', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6815', 'nonce': 1600179183}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6812', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6813', 'nonce': 1600179183}]}, 'public_addr': '172.21.15.120:6809/1600179183', 'cluster_addr': '172.21.15.120:6811/1600179183', 'heartbeat_back_addr': '172.21.15.120:6815/1600179183', 'heartbeat_front_addr': '172.21.15.120:6813/1600179183', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 418, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6816', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6817', 'nonce': 3323706735}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6818', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6819', 'nonce': 3323706735}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6822', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6823', 'nonce': 3323706735}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6820', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6821', 'nonce': 3323706735}]}, 'public_addr': '172.21.15.120:6817/3323706735', 'cluster_addr': '172.21.15.120:6819/3323706735', 'heartbeat_back_addr': '172.21.15.120:6823/3323706735', 'heartbeat_front_addr': '172.21.15.120:6821/3323706735', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6ef085d8-f59b-4033-8280-2c8f78693e30', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 219, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6824', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6825', 'nonce': 2960616153}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6826', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6827', 'nonce': 2960616153}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6830', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6831', 'nonce': 2960616153}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6828', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6829', 'nonce': 2960616153}]}, 'public_addr': '172.21.15.120:6825/2960616153', 'cluster_addr': '172.21.15.120:6827/2960616153', 'heartbeat_back_addr': '172.21.15.120:6831/2960616153', 'heartbeat_front_addr': '172.21.15.120:6829/2960616153', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '26514b11-d02b-46e9-a4e8-3da193fcdbcc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 292, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6800', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6801', 'nonce': 1132459315}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6802', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6803', 'nonce': 1132459315}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6806', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6807', 'nonce': 1132459315}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6804', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6805', 'nonce': 1132459315}]}, 'public_addr': '172.21.15.177:6801/1132459315', 'cluster_addr': '172.21.15.177:6803/1132459315', 'heartbeat_back_addr': '172.21.15.177:6807/1132459315', 'heartbeat_front_addr': '172.21.15.177:6805/1132459315', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '6f9497bc-e962-409d-8a2c-0c634cc1124f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9062042236328125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 418, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6808', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6809', 'nonce': 2710457646}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6810', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6811', 'nonce': 2710457646}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6814', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6815', 'nonce': 2710457646}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6812', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6813', 'nonce': 2710457646}]}, 'public_addr': '172.21.15.177:6809/2710457646', 'cluster_addr': '172.21.15.177:6811/2710457646', 'heartbeat_back_addr': '172.21.15.177:6815/2710457646', 'heartbeat_front_addr': '172.21.15.177:6813/2710457646', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'f852876f-c7d1-4039-90c4-5ab09dbd3e32', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.4520721435546875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 81, 'up_thru': 418, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6816', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6817', 'nonce': 3236900263}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6818', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6819', 'nonce': 3236900263}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6822', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6823', 'nonce': 3236900263}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6820', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6821', 'nonce': 3236900263}]}, 'public_addr': '172.21.15.177:6817/3236900263', 'cluster_addr': '172.21.15.177:6819/3236900263', 'heartbeat_back_addr': '172.21.15.177:6823/3236900263', 'heartbeat_front_addr': '172.21.15.177:6821/3236900263', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'cccc53c5-8476-4d55-b160-4bdc5cf92902', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 90, 'up_thru': 418, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6824', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6825', 'nonce': 681011628}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6826', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6827', 'nonce': 681011628}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6830', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6831', 'nonce': 681011628}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6828', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6829', 'nonce': 681011628}]}, 'public_addr': '172.21.15.177:6825/681011628', 'cluster_addr': '172.21.15.177:6827/681011628', 'heartbeat_back_addr': '172.21.15.177:6831/681011628', 'heartbeat_front_addr': '172.21.15.177:6829/681011628', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:04:30.836275+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:04:20.122087+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:04:50.651769+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:03:40.680176+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:03:44.788460+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:04:00.460801+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:04:24.737473+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-15T13:04:40.695993+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:04:23.489780+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:04:20.144896+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:04:49.434839+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:04:55.752123+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [1, 6, 10]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.27:6801/2530617473': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:6800/2530617473': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:0/1880261204': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:6801/3441136471': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/4291446247': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:0/580128209': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:6800/3441136471': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/3011229857': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/2109693913': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/2445663024': '2024-05-16T12:39:19.409195+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-05-15T13:05:02.620 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:05:02.860 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:05:02.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:02.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:05:02.966 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:05:03.052 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:03.175 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:05:03.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:03.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:05:03.180 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:03 smithi027 ceph-mon[39795]: pgmap v1213: 17 pgs: 17 active+clean; 7.7 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 102 B/s rd, 13 MiB/s wr, 1.03k op/s 2024-05-15T13:05:03.281 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:05:03.308 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:03 smithi120 ceph-mon[43961]: pgmap v1213: 17 pgs: 17 active+clean; 7.7 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 102 B/s rd, 13 MiB/s wr, 1.03k op/s 2024-05-15T13:05:03.311 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:03.339 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:05:03.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:03.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:05:03.397 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:05:03.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:03 smithi177 ceph-mon[44303]: pgmap v1213: 17 pgs: 17 active+clean; 7.7 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 102 B/s rd, 13 MiB/s wr, 1.03k op/s 2024-05-15T13:05:03.443 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:05:03.481 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:03.510 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:05:03.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:03.513 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:05:03.614 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:05:03.701 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:04.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:04 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:05:04.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:04 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:05:04.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:04 smithi027 ceph-mon[39795]: osdmap e453: 12 total, 12 up, 11 in 2024-05-15T13:05:04.281 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:05:04.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:04 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:05:04.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:04 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:05:04.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:04 smithi177 ceph-mon[44303]: osdmap e453: 12 total, 12 up, 11 in 2024-05-15T13:05:04.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:04 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:05:04.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:04 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:05:04.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:04 smithi120 ceph-mon[43961]: osdmap e453: 12 total, 12 up, 11 in 2024-05-15T13:05:05.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:05 smithi177 ceph-mon[44303]: pgmap v1215: 17 pgs: 17 active+clean; 7.6 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T13:05:05.438 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:05 smithi027 ceph-mon[39795]: pgmap v1215: 17 pgs: 17 active+clean; 7.6 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T13:05:05.439 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:05:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:05:05.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:05.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:05:05.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:05 smithi120 ceph-mon[43961]: pgmap v1215: 17 pgs: 17 active+clean; 7.6 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T13:05:05.572 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:05:05.612 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:05.641 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:05:05.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:05.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:05:05.747 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:05:05.800 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:05.935 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:05:05.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:05.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:05:06.040 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:05:06.086 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:06.300 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:05:06.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:06.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:05:06.414 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:05:06.454 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:06.499 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:05:06.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:06.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:05:06.603 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:05:06.645 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:06.674 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:05:06.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:06.681 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:05:06.782 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:05:06.818 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:06.852 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:05:06.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:06.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:05:06.960 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:05:06.999 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:07.027 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:05:07.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:07.030 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:05:07.131 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:05:07.180 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:07.214 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:05:07.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:07.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:05:07.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:07 smithi027 ceph-mon[39795]: pgmap v1216: 17 pgs: 17 active+clean; 7.5 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s 2024-05-15T13:05:07.318 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:05:07.386 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:07.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:07 smithi177 ceph-mon[44303]: pgmap v1216: 17 pgs: 17 active+clean; 7.5 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s 2024-05-15T13:05:07.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:07 smithi120 ceph-mon[43961]: pgmap v1216: 17 pgs: 17 active+clean; 7.5 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s 2024-05-15T13:05:08.055 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:05:08.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:08.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:05:08.110 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:05:08.160 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:05:08.214 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:08.414 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:08 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2168805446' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:05:08.414 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:08 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:05:08.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:08 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2168805446' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:05:08.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:08 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:05:08.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:08 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2168805446' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:05:08.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:08 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:05:09.175 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:05:09.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:09.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:05:09.285 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:05:09.324 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:09.346 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:05:09.355 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:05:09.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:09.360 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:05:09.361 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:09 smithi177 ceph-mon[44303]: pgmap v1217: 17 pgs: 17 active+clean; 7.3 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 998 op/s 2024-05-15T13:05:09.361 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:09 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:05:09.361 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:09 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:05:09.361 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:09 smithi177 ceph-mon[44303]: osdmap e454: 12 total, 12 up, 11 in 2024-05-15T13:05:09.461 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:05:09.517 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:09 smithi027 ceph-mon[39795]: pgmap v1217: 17 pgs: 17 active+clean; 7.3 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 998 op/s 2024-05-15T13:05:09.517 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:09 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:05:09.517 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:09 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:05:09.517 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:09 smithi027 ceph-mon[39795]: osdmap e454: 12 total, 12 up, 11 in 2024-05-15T13:05:09.520 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:09.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:09 smithi120 ceph-mon[43961]: pgmap v1217: 17 pgs: 17 active+clean; 7.3 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 998 op/s 2024-05-15T13:05:09.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:09 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:05:09.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:09 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:05:09.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:09 smithi120 ceph-mon[43961]: osdmap e454: 12 total, 12 up, 11 in 2024-05-15T13:05:09.633 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:05:09.633 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T13:05:09.633 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T13:05:09.634 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:05:09.634 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.634 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:234133 3.6 3:6bfad9ff:::benchmark_data_smithi177_88104_object100511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T13:05:09.634 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:44.636070+0000", 2024-05-15T13:05:09.634 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 505.00443951300002, 2024-05-15T13:05:09.634 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22423902700000001, 2024-05-15T13:05:09.634 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:09.634 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:05:09.634 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:09.634 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:05:09.634 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:05:09.634 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 234133 2024-05-15T13:05:09.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.635 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:09.635 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.635 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:09.635 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636070+0000", 2024-05-15T13:05:09.635 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.635 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.635 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:09.635 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636070+0000", 2024-05-15T13:05:09.635 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.635 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.635 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:09.636 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636071+0000", 2024-05-15T13:05:09.636 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3900000000000004e-07 2024-05-15T13:05:09.636 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.636 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.636 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:09.636 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636076+0000", 2024-05-15T13:05:09.636 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7040000000000002e-06 2024-05-15T13:05:09.636 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.636 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.636 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:09.636 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636087+0000", 2024-05-15T13:05:09.636 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1718e-05 2024-05-15T13:05:09.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.637 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.637 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:09.637 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636098+0000", 2024-05-15T13:05:09.637 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.084e-05 2024-05-15T13:05:09.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.637 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.637 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:09.637 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.641220+0000", 2024-05-15T13:05:09.637 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0051218169999999999 2024-05-15T13:05:09.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.637 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.638 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:09.638 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.641326+0000", 2024-05-15T13:05:09.638 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010569900000000001 2024-05-15T13:05:09.638 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.638 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.638 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T13:05:09.638 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.641435+0000", 2024-05-15T13:05:09.638 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000109162 2024-05-15T13:05:09.638 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.638 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.638 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.638 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.655524+0000", 2024-05-15T13:05:09.638 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014089121 2024-05-15T13:05:09.639 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.639 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.639 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:05:09.639 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.662050+0000", 2024-05-15T13:05:09.639 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0065262009999999997 2024-05-15T13:05:09.639 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.639 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.639 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.639 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.860197+0000", 2024-05-15T13:05:09.639 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19814722300000001 2024-05-15T13:05:09.639 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.639 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.639 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:05:09.639 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.860258+0000", 2024-05-15T13:05:09.639 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0826999999999999e-05 2024-05-15T13:05:09.640 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.640 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.640 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:09.640 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.860309+0000", 2024-05-15T13:05:09.640 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0875999999999999e-05 2024-05-15T13:05:09.640 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.640 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:09.640 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.640 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.640 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.640 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:234137 3.6 3:64a7160a:::benchmark_data_smithi177_88104_object100515:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T13:05:09.640 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:44.636929+0000", 2024-05-15T13:05:09.641 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 505.00358088000002, 2024-05-15T13:05:09.641 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22352234700000001, 2024-05-15T13:05:09.641 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:09.641 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:05:09.641 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:09.641 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:05:09.641 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:05:09.641 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 234137 2024-05-15T13:05:09.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.641 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:09.641 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.641 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:09.642 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636929+0000", 2024-05-15T13:05:09.642 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.642 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.642 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:09.642 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636929+0000", 2024-05-15T13:05:09.642 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.642 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.642 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:09.642 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636930+0000", 2024-05-15T13:05:09.642 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.057e-06 2024-05-15T13:05:09.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:09.643 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636934+0000", 2024-05-15T13:05:09.643 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4750000000000004e-06 2024-05-15T13:05:09.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:09.643 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636945+0000", 2024-05-15T13:05:09.643 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.094e-05 2024-05-15T13:05:09.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:09.643 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.636953+0000", 2024-05-15T13:05:09.644 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7400000000000004e-06 2024-05-15T13:05:09.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.644 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.644 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:09.644 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650374+0000", 2024-05-15T13:05:09.644 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013421535 2024-05-15T13:05:09.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.644 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.644 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:09.644 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650489+0000", 2024-05-15T13:05:09.644 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000114919 2024-05-15T13:05:09.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.644 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.644 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T13:05:09.644 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650628+0000", 2024-05-15T13:05:09.645 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013872799999999999 2024-05-15T13:05:09.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.645 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.645 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.645 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.657205+0000", 2024-05-15T13:05:09.645 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0065768520000000002 2024-05-15T13:05:09.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.645 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.645 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:05:09.645 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.666120+0000", 2024-05-15T13:05:09.645 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0089147159999999996 2024-05-15T13:05:09.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.645 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.645 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.646 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.860355+0000", 2024-05-15T13:05:09.646 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19423544000000001 2024-05-15T13:05:09.646 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.646 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.646 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:05:09.646 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.860409+0000", 2024-05-15T13:05:09.646 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4172999999999998e-05 2024-05-15T13:05:09.646 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.646 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.646 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:09.646 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.860451+0000", 2024-05-15T13:05:09.646 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1771999999999999e-05 2024-05-15T13:05:09.646 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.647 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:09.647 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.647 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.647 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:234139 3.6 3:67f4ccfa:::benchmark_data_smithi177_88104_object100517:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T13:05:09.647 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:44.642307+0000", 2024-05-15T13:05:09.647 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 504.99820280199998, 2024-05-15T13:05:09.647 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22900473900000001, 2024-05-15T13:05:09.647 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:09.647 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:05:09.647 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:09.647 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:05:09.647 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:05:09.647 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 234139 2024-05-15T13:05:09.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.648 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:09.648 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.648 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:09.648 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.642307+0000", 2024-05-15T13:05:09.648 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.648 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.648 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:09.648 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.642307+0000", 2024-05-15T13:05:09.648 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.648 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.648 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:09.649 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.642308+0000", 2024-05-15T13:05:09.649 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2100000000000001e-06 2024-05-15T13:05:09.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.649 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.649 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:09.649 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.642311+0000", 2024-05-15T13:05:09.649 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4630000000000001e-06 2024-05-15T13:05:09.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.649 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.649 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:09.649 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.642320+0000", 2024-05-15T13:05:09.649 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3119999999999994e-06 2024-05-15T13:05:09.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.649 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.650 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:09.650 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.642326+0000", 2024-05-15T13:05:09.650 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1709999999999999e-06 2024-05-15T13:05:09.650 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.650 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.650 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:09.650 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.657235+0000", 2024-05-15T13:05:09.650 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014908889999999999 2024-05-15T13:05:09.650 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.650 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.650 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:09.650 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.657392+0000", 2024-05-15T13:05:09.650 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015708900000000001 2024-05-15T13:05:09.650 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.651 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.651 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T13:05:09.651 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.657525+0000", 2024-05-15T13:05:09.651 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000132764 2024-05-15T13:05:09.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.651 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.651 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.651 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.661929+0000", 2024-05-15T13:05:09.652 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0044041469999999997 2024-05-15T13:05:09.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.652 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.652 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:05:09.652 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.666168+0000", 2024-05-15T13:05:09.652 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0042391920000000001 2024-05-15T13:05:09.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.652 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.652 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.652 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.871182+0000", 2024-05-15T13:05:09.652 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20501413499999999 2024-05-15T13:05:09.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.652 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.652 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:05:09.653 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.871250+0000", 2024-05-15T13:05:09.653 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8338999999999997e-05 2024-05-15T13:05:09.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.653 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.653 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:09.653 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.871311+0000", 2024-05-15T13:05:09.653 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1026999999999997e-05 2024-05-15T13:05:09.653 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.653 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:09.653 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.653 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.653 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:234142 3.6 3:6ff8b345:::benchmark_data_smithi177_88104_object100520:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T13:05:09.653 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:44.650217+0000", 2024-05-15T13:05:09.654 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 504.99029266500003, 2024-05-15T13:05:09.654 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22408456900000001, 2024-05-15T13:05:09.654 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:09.654 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:05:09.654 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:09.654 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:05:09.654 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:05:09.654 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 234142 2024-05-15T13:05:09.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.654 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:09.654 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.654 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:09.654 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650217+0000", 2024-05-15T13:05:09.654 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.655 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.655 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:09.655 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650217+0000", 2024-05-15T13:05:09.655 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.655 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.655 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:09.655 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650218+0000", 2024-05-15T13:05:09.655 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.316e-06 2024-05-15T13:05:09.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.655 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.655 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:09.655 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650222+0000", 2024-05-15T13:05:09.656 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5319999999999998e-06 2024-05-15T13:05:09.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.656 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.656 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:09.656 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650230+0000", 2024-05-15T13:05:09.656 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7090000000000008e-06 2024-05-15T13:05:09.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.656 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.656 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:09.656 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.650236+0000", 2024-05-15T13:05:09.656 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6899999999999997e-06 2024-05-15T13:05:09.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.656 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.656 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:09.656 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.662077+0000", 2024-05-15T13:05:09.657 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.01184057 2024-05-15T13:05:09.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.657 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.657 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:09.657 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.662237+0000", 2024-05-15T13:05:09.657 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016033200000000001 2024-05-15T13:05:09.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.657 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.657 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T13:05:09.657 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.662332+0000", 2024-05-15T13:05:09.657 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5199999999999997e-05 2024-05-15T13:05:09.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.657 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.658 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:05:09.658 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.666359+0000", 2024-05-15T13:05:09.658 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.004027325 2024-05-15T13:05:09.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.658 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.658 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.658 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.668031+0000", 2024-05-15T13:05:09.658 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016717629999999999 2024-05-15T13:05:09.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.658 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.658 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.658 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.874188+0000", 2024-05-15T13:05:09.658 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20615639699999999 2024-05-15T13:05:09.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.659 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.659 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:05:09.659 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.874247+0000", 2024-05-15T13:05:09.659 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9775000000000003e-05 2024-05-15T13:05:09.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.659 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.659 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:09.659 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.874301+0000", 2024-05-15T13:05:09.659 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3959999999999998e-05 2024-05-15T13:05:09.659 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.659 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:09.659 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.659 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.660 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:234166 3.6 3:65e82527:::benchmark_data_smithi177_88104_object100544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T13:05:09.660 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:44.676834+0000", 2024-05-15T13:05:09.660 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 504.96367568900001, 2024-05-15T13:05:09.660 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.214610197, 2024-05-15T13:05:09.660 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:09.660 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:05:09.660 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:09.660 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:05:09.660 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:05:09.660 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 234166 2024-05-15T13:05:09.660 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.660 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:09.660 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.660 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:09.661 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.676834+0000", 2024-05-15T13:05:09.661 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.661 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.661 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:09.661 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.676834+0000", 2024-05-15T13:05:09.661 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.661 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.661 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:09.661 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.676836+0000", 2024-05-15T13:05:09.661 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0260000000000002e-06 2024-05-15T13:05:09.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.662 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.662 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:09.662 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.676852+0000", 2024-05-15T13:05:09.662 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.647e-05 2024-05-15T13:05:09.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.662 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.662 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:09.662 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.676869+0000", 2024-05-15T13:05:09.662 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6915999999999999e-05 2024-05-15T13:05:09.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.662 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.663 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:09.663 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.676888+0000", 2024-05-15T13:05:09.663 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8437999999999999e-05 2024-05-15T13:05:09.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.663 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.663 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:09.663 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.676954+0000", 2024-05-15T13:05:09.663 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6723000000000005e-05 2024-05-15T13:05:09.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.663 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.663 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:09.663 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.677089+0000", 2024-05-15T13:05:09.663 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013486300000000001 2024-05-15T13:05:09.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.664 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.664 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T13:05:09.664 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.677239+0000", 2024-05-15T13:05:09.664 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014941600000000001 2024-05-15T13:05:09.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.664 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.664 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.664 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.683263+0000", 2024-05-15T13:05:09.664 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0060247510000000001 2024-05-15T13:05:09.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.664 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.664 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:05:09.664 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.683276+0000", 2024-05-15T13:05:09.665 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2281e-05 2024-05-15T13:05:09.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.665 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.665 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.665 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.891351+0000", 2024-05-15T13:05:09.665 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20807553100000001 2024-05-15T13:05:09.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.665 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.665 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:05:09.665 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.891400+0000", 2024-05-15T13:05:09.665 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8640999999999997e-05 2024-05-15T13:05:09.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.665 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.665 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:09.665 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.891444+0000", 2024-05-15T13:05:09.666 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4141000000000003e-05 2024-05-15T13:05:09.666 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.666 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:09.666 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.666 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.666 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.24832.0:234181 3.6 3:635178a9:::benchmark_data_smithi177_88104_object100559:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-05-15T13:05:09.666 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:56:44.691125+0000", 2024-05-15T13:05:09.666 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 504.94938410100002, 2024-05-15T13:05:09.666 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20043533499999999, 2024-05-15T13:05:09.666 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:09.666 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:05:09.667 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:09.667 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.24832", 2024-05-15T13:05:09.667 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:05:09.667 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 234181 2024-05-15T13:05:09.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.667 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:09.667 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.667 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:09.667 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691125+0000", 2024-05-15T13:05:09.667 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.667 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.667 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:09.667 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691125+0000", 2024-05-15T13:05:09.667 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.668 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.668 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:09.668 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691127+0000", 2024-05-15T13:05:09.668 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.392e-06 2024-05-15T13:05:09.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.668 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.668 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:09.668 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691131+0000", 2024-05-15T13:05:09.668 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5020000000000003e-06 2024-05-15T13:05:09.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.668 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.668 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:09.668 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691142+0000", 2024-05-15T13:05:09.669 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0478e-05 2024-05-15T13:05:09.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.669 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.669 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:09.669 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691149+0000", 2024-05-15T13:05:09.669 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9689999999999997e-06 2024-05-15T13:05:09.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.669 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.669 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:09.669 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691205+0000", 2024-05-15T13:05:09.669 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6137999999999997e-05 2024-05-15T13:05:09.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.669 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.669 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:09.669 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691332+0000", 2024-05-15T13:05:09.670 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012724599999999999 2024-05-15T13:05:09.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.670 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.670 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,4", 2024-05-15T13:05:09.670 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.691458+0000", 2024-05-15T13:05:09.670 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012549999999999999 2024-05-15T13:05:09.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.670 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.670 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.670 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.696662+0000", 2024-05-15T13:05:09.670 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0052043760000000001 2024-05-15T13:05:09.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.670 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.671 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:05:09.671 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.697207+0000", 2024-05-15T13:05:09.671 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00054491700000000001 2024-05-15T13:05:09.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.671 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.671 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.671 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.891478+0000", 2024-05-15T13:05:09.671 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19427066900000001 2024-05-15T13:05:09.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.671 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.671 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:05:09.671 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.891527+0000", 2024-05-15T13:05:09.671 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9308999999999998e-05 2024-05-15T13:05:09.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.672 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.672 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:09.672 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:56:44.891561+0000", 2024-05-15T13:05:09.672 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3838999999999999e-05 2024-05-15T13:05:09.672 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.672 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:09.672 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.672 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.672 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:21009 5.3 5:c0e5f930:::benchmark_data_smithi177_109991_object21008:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:05:09.672 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:27.986822+0000", 2024-05-15T13:05:09.673 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 101.653687858, 2024-05-15T13:05:09.673 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20485505300000001, 2024-05-15T13:05:09.673 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:09.673 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:05:09.673 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:09.673 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:05:09.673 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:05:09.673 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 21009 2024-05-15T13:05:09.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.673 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:09.673 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.673 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:09.673 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.986822+0000", 2024-05-15T13:05:09.673 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.674 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.674 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:09.674 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.986822+0000", 2024-05-15T13:05:09.674 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.674 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.674 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:09.674 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.986822+0000", 2024-05-15T13:05:09.674 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9500000000000001e-07 2024-05-15T13:05:09.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.674 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.674 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:09.675 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.986941+0000", 2024-05-15T13:05:09.675 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000118968 2024-05-15T13:05:09.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.675 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.675 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:09.675 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.986953+0000", 2024-05-15T13:05:09.675 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1151999999999999e-05 2024-05-15T13:05:09.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.675 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.675 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:09.675 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.986962+0000", 2024-05-15T13:05:09.675 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1710000000000008e-06 2024-05-15T13:05:09.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.675 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.675 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:09.676 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.987015+0000", 2024-05-15T13:05:09.676 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3193999999999999e-05 2024-05-15T13:05:09.676 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.676 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.676 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:09.676 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.987096+0000", 2024-05-15T13:05:09.676 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0620999999999999e-05 2024-05-15T13:05:09.676 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.676 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.676 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,6", 2024-05-15T13:05:09.676 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.987226+0000", 2024-05-15T13:05:09.676 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013094699999999999 2024-05-15T13:05:09.676 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.676 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.677 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:05:09.677 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.988235+0000", 2024-05-15T13:05:09.677 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010089330000000001 2024-05-15T13:05:09.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.677 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.677 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.990458+0000", 2024-05-15T13:05:09.677 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0022220970000000001 2024-05-15T13:05:09.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.677 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.677 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.191460+0000", 2024-05-15T13:05:09.677 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20100295000000001 2024-05-15T13:05:09.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.678 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:05:09.678 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.191582+0000", 2024-05-15T13:05:09.678 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000121118 2024-05-15T13:05:09.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.678 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:09.678 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.191677+0000", 2024-05-15T13:05:09.678 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5106999999999996e-05 2024-05-15T13:05:09.678 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.678 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:09.678 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.679 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:21031 5.b 5:d879a200:::benchmark_data_smithi177_109991_object21030:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:05:09.679 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:27.993052+0000", 2024-05-15T13:05:09.679 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 101.647457577, 2024-05-15T13:05:09.679 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19869366499999999, 2024-05-15T13:05:09.679 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:09.679 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:05:09.679 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:09.679 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:05:09.679 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:05:09.679 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 21031 2024-05-15T13:05:09.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.679 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:09.680 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.680 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:09.680 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.993052+0000", 2024-05-15T13:05:09.680 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.680 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.680 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.681 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:09.681 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.993052+0000", 2024-05-15T13:05:09.681 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.681 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.681 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.681 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:09.681 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.993053+0000", 2024-05-15T13:05:09.682 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3200000000000004e-07 2024-05-15T13:05:09.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.682 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.682 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:09.682 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.993246+0000", 2024-05-15T13:05:09.682 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00019335500000000001 2024-05-15T13:05:09.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.683 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.683 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:09.683 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.993258+0000", 2024-05-15T13:05:09.683 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1715999999999999e-05 2024-05-15T13:05:09.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.683 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.684 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:09.684 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.993267+0000", 2024-05-15T13:05:09.684 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9309999999999997e-06 2024-05-15T13:05:09.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.684 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.684 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:09.685 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.993308+0000", 2024-05-15T13:05:09.685 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0979999999999997e-05 2024-05-15T13:05:09.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.685 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.685 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:09.685 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.993406+0000", 2024-05-15T13:05:09.686 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8225000000000002e-05 2024-05-15T13:05:09.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.686 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.686 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:05:09.686 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.993558+0000", 2024-05-15T13:05:09.686 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015191400000000001 2024-05-15T13:05:09.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.687 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.687 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:05:09.687 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.995561+0000", 2024-05-15T13:05:09.687 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020029660000000001 2024-05-15T13:05:09.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.687 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.688 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.688 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.997225+0000", 2024-05-15T13:05:09.688 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001664659 2024-05-15T13:05:09.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.688 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.688 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.689 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.191650+0000", 2024-05-15T13:05:09.689 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19442416000000001 2024-05-15T13:05:09.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.689 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.689 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:05:09.689 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.191714+0000", 2024-05-15T13:05:09.689 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4258999999999996e-05 2024-05-15T13:05:09.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.689 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.689 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:09.690 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.191746+0000", 2024-05-15T13:05:09.690 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1668000000000002e-05 2024-05-15T13:05:09.690 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.690 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:09.690 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.691 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.691 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:21033 5.b 5:d5fde012:::benchmark_data_smithi177_109991_object21032:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:05:09.691 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:27.993518+0000", 2024-05-15T13:05:09.691 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 101.646991599, 2024-05-15T13:05:09.691 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19896936100000001, 2024-05-15T13:05:09.691 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:09.691 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:05:09.692 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:09.692 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:05:09.692 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:05:09.692 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 21033 2024-05-15T13:05:09.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.692 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:09.693 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.693 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:09.693 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.993518+0000", 2024-05-15T13:05:09.693 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.693 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.694 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:09.694 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.993518+0000", 2024-05-15T13:05:09.694 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.694 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.694 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:09.695 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.993519+0000", 2024-05-15T13:05:09.695 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2799999999999995e-07 2024-05-15T13:05:09.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.695 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.695 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:09.695 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.995961+0000", 2024-05-15T13:05:09.696 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002442165 2024-05-15T13:05:09.696 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.696 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.696 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:09.696 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.995972+0000", 2024-05-15T13:05:09.696 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1182000000000001e-05 2024-05-15T13:05:09.697 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.697 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.697 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:09.697 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.995978+0000", 2024-05-15T13:05:09.697 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8089999999999998e-06 2024-05-15T13:05:09.697 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.698 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.698 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:09.698 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.996031+0000", 2024-05-15T13:05:09.698 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3523999999999998e-05 2024-05-15T13:05:09.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.698 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.699 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:09.699 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.996176+0000", 2024-05-15T13:05:09.699 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014440899999999999 2024-05-15T13:05:09.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.699 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.699 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:05:09.700 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.996277+0000", 2024-05-15T13:05:09.700 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000101229 2024-05-15T13:05:09.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.700 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.700 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.700 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.997334+0000", 2024-05-15T13:05:09.700 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010571529999999999 2024-05-15T13:05:09.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.701 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.701 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:05:09.701 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.998416+0000", 2024-05-15T13:05:09.701 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010813190000000001 2024-05-15T13:05:09.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.702 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.702 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.702 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.192393+0000", 2024-05-15T13:05:09.702 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19397751499999999 2024-05-15T13:05:09.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.703 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.703 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:05:09.703 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.192447+0000", 2024-05-15T13:05:09.703 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3640999999999997e-05 2024-05-15T13:05:09.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.703 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.703 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:09.703 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.192487+0000", 2024-05-15T13:05:09.703 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0587000000000003e-05 2024-05-15T13:05:09.704 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.704 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:09.704 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.704 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.704 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:21038 5.b 5:d9bd3b12:::benchmark_data_smithi177_109991_object21037:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:05:09.704 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:27.997203+0000", 2024-05-15T13:05:09.704 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 101.643306477, 2024-05-15T13:05:09.704 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19545880400000001, 2024-05-15T13:05:09.705 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:09.705 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:05:09.705 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:09.705 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:05:09.705 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:05:09.705 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 21038 2024-05-15T13:05:09.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.705 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:09.705 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.706 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:09.706 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.997203+0000", 2024-05-15T13:05:09.706 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.706 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.706 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:09.707 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.997203+0000", 2024-05-15T13:05:09.707 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.707 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.707 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:09.707 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.997204+0000", 2024-05-15T13:05:09.707 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3300000000000002e-07 2024-05-15T13:05:09.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.707 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.707 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:09.707 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.997232+0000", 2024-05-15T13:05:09.707 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7906999999999999e-05 2024-05-15T13:05:09.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.707 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.708 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:09.708 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.997241+0000", 2024-05-15T13:05:09.708 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9170000000000001e-06 2024-05-15T13:05:09.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.708 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.708 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:09.708 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.997250+0000", 2024-05-15T13:05:09.708 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6819999999999992e-06 2024-05-15T13:05:09.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.708 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.708 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:09.708 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.997376+0000", 2024-05-15T13:05:09.708 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000125599 2024-05-15T13:05:09.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.709 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.709 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:09.709 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.997504+0000", 2024-05-15T13:05:09.709 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000128589 2024-05-15T13:05:09.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.709 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.709 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:05:09.709 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.997640+0000", 2024-05-15T13:05:09.709 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013561500000000001 2024-05-15T13:05:09.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.709 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.709 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:05:09.709 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.998936+0000", 2024-05-15T13:05:09.709 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0012962449999999999 2024-05-15T13:05:09.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.710 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.710 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.710 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.999819+0000", 2024-05-15T13:05:09.710 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00088295099999999998 2024-05-15T13:05:09.710 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.710 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.710 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.710 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.192540+0000", 2024-05-15T13:05:09.710 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.192720422 2024-05-15T13:05:09.710 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.710 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.710 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:05:09.710 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.192592+0000", 2024-05-15T13:05:09.711 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2114000000000003e-05 2024-05-15T13:05:09.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.711 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.711 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:09.711 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.192662+0000", 2024-05-15T13:05:09.711 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0129999999999994e-05 2024-05-15T13:05:09.711 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.711 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:09.711 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.711 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.711 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:21067 5.b 5:dd81d426:::benchmark_data_smithi177_109991_object21066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:05:09.711 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:28.004740+0000", 2024-05-15T13:05:09.711 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 101.635769053, 2024-05-15T13:05:09.712 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.18853851099999999, 2024-05-15T13:05:09.712 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:09.712 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:05:09.712 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:09.712 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:05:09.712 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:05:09.712 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 21067 2024-05-15T13:05:09.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.712 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:09.712 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.712 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:09.712 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.004740+0000", 2024-05-15T13:05:09.712 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.712 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.713 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:09.713 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.004740+0000", 2024-05-15T13:05:09.713 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.713 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.713 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:09.713 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.004741+0000", 2024-05-15T13:05:09.713 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8899999999999999e-07 2024-05-15T13:05:09.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.713 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.713 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:09.713 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.004884+0000", 2024-05-15T13:05:09.713 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014267800000000001 2024-05-15T13:05:09.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.713 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.714 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:09.714 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.004895+0000", 2024-05-15T13:05:09.714 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.117e-05 2024-05-15T13:05:09.714 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.714 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.714 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:09.714 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.004902+0000", 2024-05-15T13:05:09.714 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.576e-06 2024-05-15T13:05:09.714 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.714 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.714 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:09.714 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.005755+0000", 2024-05-15T13:05:09.714 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00085300100000000004 2024-05-15T13:05:09.715 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.715 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.715 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:09.715 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.005901+0000", 2024-05-15T13:05:09.715 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014568900000000001 2024-05-15T13:05:09.715 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.715 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.715 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:05:09.715 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.006047+0000", 2024-05-15T13:05:09.715 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014598500000000001 2024-05-15T13:05:09.715 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.715 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.715 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.715 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.008976+0000", 2024-05-15T13:05:09.716 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029288199999999999 2024-05-15T13:05:09.716 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.716 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.716 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:05:09.716 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.009199+0000", 2024-05-15T13:05:09.716 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000223353 2024-05-15T13:05:09.716 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.716 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.716 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.716 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.193167+0000", 2024-05-15T13:05:09.716 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.183967837 2024-05-15T13:05:09.716 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.716 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.716 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:05:09.716 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.193236+0000", 2024-05-15T13:05:09.717 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8378000000000002e-05 2024-05-15T13:05:09.717 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.717 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.717 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:09.717 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.193279+0000", 2024-05-15T13:05:09.717 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3435e-05 2024-05-15T13:05:09.717 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.717 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:09.717 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.717 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.717 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.717 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:21068 5.b 5:d17322b5:::benchmark_data_smithi177_109991_object21067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:05:09.717 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:28.004930+0000", 2024-05-15T13:05:09.718 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 101.63557972700001, 2024-05-15T13:05:09.718 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.18845712100000001, 2024-05-15T13:05:09.718 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:09.718 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:05:09.718 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:09.718 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:05:09.718 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:05:09.718 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 21068 2024-05-15T13:05:09.718 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.718 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:09.718 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.718 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:09.718 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.004930+0000", 2024-05-15T13:05:09.719 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.719 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.719 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.719 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:09.719 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.004930+0000", 2024-05-15T13:05:09.719 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.719 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.719 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.719 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:09.719 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.004930+0000", 2024-05-15T13:05:09.719 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9599999999999999e-07 2024-05-15T13:05:09.719 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.719 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.719 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:09.720 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.004967+0000", 2024-05-15T13:05:09.720 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6683000000000001e-05 2024-05-15T13:05:09.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.720 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.720 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:09.720 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.004976+0000", 2024-05-15T13:05:09.720 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3850000000000008e-06 2024-05-15T13:05:09.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.720 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.720 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:09.720 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.004983+0000", 2024-05-15T13:05:09.720 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4620000000000001e-06 2024-05-15T13:05:09.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.720 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.721 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:09.721 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.006841+0000", 2024-05-15T13:05:09.721 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001858246 2024-05-15T13:05:09.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.721 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.721 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:09.721 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.006982+0000", 2024-05-15T13:05:09.721 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014072100000000001 2024-05-15T13:05:09.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.721 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.721 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:05:09.721 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.007162+0000", 2024-05-15T13:05:09.721 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00018030100000000001 2024-05-15T13:05:09.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.722 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.722 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.722 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.009168+0000", 2024-05-15T13:05:09.722 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020057069999999998 2024-05-15T13:05:09.722 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.722 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.722 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:05:09.722 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.009205+0000", 2024-05-15T13:05:09.722 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6820000000000003e-05 2024-05-15T13:05:09.722 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.722 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.722 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.722 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.193311+0000", 2024-05-15T13:05:09.723 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.184106356 2024-05-15T13:05:09.723 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.723 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.723 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:05:09.723 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.193343+0000", 2024-05-15T13:05:09.723 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1921000000000002e-05 2024-05-15T13:05:09.723 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.723 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.723 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:09.723 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.193387+0000", 2024-05-15T13:05:09.723 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3922999999999999e-05 2024-05-15T13:05:09.723 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.723 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:09.723 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.723 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.724 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.724 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:21075 5.b 5:dd98a7f6:::benchmark_data_smithi177_109991_object21074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:05:09.724 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:28.006309+0000", 2024-05-15T13:05:09.724 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 101.63420034400001, 2024-05-15T13:05:09.724 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.18860990999999999, 2024-05-15T13:05:09.724 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:09.724 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:05:09.724 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:09.724 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:05:09.724 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:05:09.724 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 21075 2024-05-15T13:05:09.724 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.724 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:09.725 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.725 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:09.725 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.006309+0000", 2024-05-15T13:05:09.725 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.725 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.725 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:09.725 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.006309+0000", 2024-05-15T13:05:09.725 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.725 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.725 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:09.725 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.006310+0000", 2024-05-15T13:05:09.725 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.127e-06 2024-05-15T13:05:09.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.726 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.726 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:09.726 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.006538+0000", 2024-05-15T13:05:09.726 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00022818799999999999 2024-05-15T13:05:09.726 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.726 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.726 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:09.726 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.006549+0000", 2024-05-15T13:05:09.726 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0971999999999999e-05 2024-05-15T13:05:09.726 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.726 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.726 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:09.726 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.006557+0000", 2024-05-15T13:05:09.727 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3900000000000004e-06 2024-05-15T13:05:09.727 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.727 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.727 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:09.727 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.007946+0000", 2024-05-15T13:05:09.727 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0013886879999999999 2024-05-15T13:05:09.727 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.727 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.727 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:09.727 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.008076+0000", 2024-05-15T13:05:09.727 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013073300000000001 2024-05-15T13:05:09.727 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.727 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.727 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:05:09.727 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.008246+0000", 2024-05-15T13:05:09.728 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000169443 2024-05-15T13:05:09.728 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.728 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.728 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:05:09.728 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.009209+0000", 2024-05-15T13:05:09.728 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00096361099999999996 2024-05-15T13:05:09.728 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.728 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.728 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.728 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.009540+0000", 2024-05-15T13:05:09.728 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00033092400000000002 2024-05-15T13:05:09.728 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.728 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.728 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.729 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.194806+0000", 2024-05-15T13:05:09.729 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.185266029 2024-05-15T13:05:09.729 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.729 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.729 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:05:09.729 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.194876+0000", 2024-05-15T13:05:09.729 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9345999999999996e-05 2024-05-15T13:05:09.729 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.729 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.729 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:09.729 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.194919+0000", 2024-05-15T13:05:09.729 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3458999999999998e-05 2024-05-15T13:05:09.729 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.729 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:09.729 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.730 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.730 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.730 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:104940 5.3 5:c982b2f2:::benchmark_data_smithi177_109991_object104939:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-05-15T13:05:09.730 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:04:20.865652+0000", 2024-05-15T13:05:09.730 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 48.774857378999997, 2024-05-15T13:05:09.730 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.108498083, 2024-05-15T13:05:09.730 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:09.730 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:05:09.730 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:09.730 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:05:09.730 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:05:09.730 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 104940 2024-05-15T13:05:09.730 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.731 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:09.731 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.731 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:09.731 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.865652+0000", 2024-05-15T13:05:09.731 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.731 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.731 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.731 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:09.731 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.865652+0000", 2024-05-15T13:05:09.731 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.731 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.731 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.731 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:09.731 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.865653+0000", 2024-05-15T13:05:09.732 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2900000000000003e-07 2024-05-15T13:05:09.732 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.732 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.732 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:09.732 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.865680+0000", 2024-05-15T13:05:09.732 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6913000000000001e-05 2024-05-15T13:05:09.732 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.732 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.732 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:09.732 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.865689+0000", 2024-05-15T13:05:09.732 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9940000000000006e-06 2024-05-15T13:05:09.732 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.732 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.732 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:09.732 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.865696+0000", 2024-05-15T13:05:09.733 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0350000000000004e-06 2024-05-15T13:05:09.733 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.733 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.733 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:09.733 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.872159+0000", 2024-05-15T13:05:09.733 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.006463328 2024-05-15T13:05:09.733 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.733 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.733 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:09.733 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.872300+0000", 2024-05-15T13:05:09.733 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014115800000000001 2024-05-15T13:05:09.733 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.733 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.733 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,6", 2024-05-15T13:05:09.734 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.872436+0000", 2024-05-15T13:05:09.734 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013553999999999999 2024-05-15T13:05:09.734 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.734 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.734 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.734 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.877510+0000", 2024-05-15T13:05:09.734 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.005073971 2024-05-15T13:05:09.734 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.734 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.734 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.734 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.974037+0000", 2024-05-15T13:05:09.734 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.096527689 2024-05-15T13:05:09.734 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.735 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.735 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:05:09.735 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.974050+0000", 2024-05-15T13:05:09.735 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2551999999999999e-05 2024-05-15T13:05:09.735 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.735 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.735 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:05:09.735 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.974095+0000", 2024-05-15T13:05:09.735 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5447e-05 2024-05-15T13:05:09.735 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.735 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.735 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:09.735 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.974150+0000", 2024-05-15T13:05:09.735 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4827000000000002e-05 2024-05-15T13:05:09.735 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.736 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:09.736 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.736 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.736 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.736 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:104962 5.b 5:db7abce2:::benchmark_data_smithi177_109991_object104961:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-05-15T13:05:09.736 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:04:20.878373+0000", 2024-05-15T13:05:09.736 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 48.762136556000002, 2024-05-15T13:05:09.736 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.107531684, 2024-05-15T13:05:09.736 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:09.736 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:05:09.736 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:09.736 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:05:09.736 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:05:09.737 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 104962 2024-05-15T13:05:09.737 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.737 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:09.737 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.737 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:09.737 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.878373+0000", 2024-05-15T13:05:09.737 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.737 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.737 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.737 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:09.737 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.878373+0000", 2024-05-15T13:05:09.737 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.738 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.738 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.738 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:09.738 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.878374+0000", 2024-05-15T13:05:09.738 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.09e-07 2024-05-15T13:05:09.738 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.738 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.738 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:09.738 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.878424+0000", 2024-05-15T13:05:09.738 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0738000000000001e-05 2024-05-15T13:05:09.738 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.738 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.738 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:09.739 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.878435+0000", 2024-05-15T13:05:09.739 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0473e-05 2024-05-15T13:05:09.739 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.739 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.739 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:09.739 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.878442+0000", 2024-05-15T13:05:09.739 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1400000000000002e-06 2024-05-15T13:05:09.739 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.739 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.739 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:09.739 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.978050+0000", 2024-05-15T13:05:09.739 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.099607645999999994 2024-05-15T13:05:09.739 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.739 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.740 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:09.740 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.978218+0000", 2024-05-15T13:05:09.740 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016780899999999999 2024-05-15T13:05:09.740 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.740 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.740 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:05:09.740 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.978400+0000", 2024-05-15T13:05:09.740 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00018252700000000001 2024-05-15T13:05:09.740 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.740 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.740 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:05:09.740 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.982412+0000", 2024-05-15T13:05:09.740 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0040118940000000002 2024-05-15T13:05:09.740 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.741 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.741 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.741 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.985284+0000", 2024-05-15T13:05:09.741 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0028724430000000001 2024-05-15T13:05:09.741 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.741 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.741 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.741 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.985785+0000", 2024-05-15T13:05:09.741 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00050084999999999995 2024-05-15T13:05:09.741 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.741 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.741 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:05:09.741 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.985847+0000", 2024-05-15T13:05:09.741 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1926000000000003e-05 2024-05-15T13:05:09.741 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.742 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.742 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:09.742 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.985905+0000", 2024-05-15T13:05:09.742 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7429000000000002e-05 2024-05-15T13:05:09.742 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.742 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:09.742 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.742 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.742 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.742 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.15540.0:149520 5.2 e453/418 5:4497d6cf:::benchmark_data_smithi177_109991_object14329:head v 453'9341, mlcod=453'9339)", 2024-05-15T13:05:09.742 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:05:05.104256+0000", 2024-05-15T13:05:09.742 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.5362534639999996, 2024-05-15T13:05:09.742 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22089091799999999, 2024-05-15T13:05:09.743 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:09.743 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:05:09.743 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:09.743 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.743 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:09.743 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.104256+0000", 2024-05-15T13:05:09.743 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.743 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.743 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.743 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:09.743 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.104256+0000", 2024-05-15T13:05:09.743 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.743 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.743 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.743 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:09.744 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.104257+0000", 2024-05-15T13:05:09.744 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0500000000000003e-07 2024-05-15T13:05:09.744 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.744 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.744 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:09.744 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.104266+0000", 2024-05-15T13:05:09.744 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6350000000000001e-06 2024-05-15T13:05:09.744 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.744 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.744 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:09.744 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.104276+0000", 2024-05-15T13:05:09.744 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7499999999999998e-06 2024-05-15T13:05:09.744 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.744 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.744 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:09.745 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.104284+0000", 2024-05-15T13:05:09.745 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5730000000000002e-06 2024-05-15T13:05:09.745 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.745 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.745 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:09.745 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.110158+0000", 2024-05-15T13:05:09.745 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0058744640000000002 2024-05-15T13:05:09.745 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.745 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.745 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:09.745 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.110178+0000", 2024-05-15T13:05:09.745 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9381e-05 2024-05-15T13:05:09.745 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.746 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.746 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:05:09.746 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.325096+0000", 2024-05-15T13:05:09.746 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.21491801499999999 2024-05-15T13:05:09.746 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.746 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.746 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:09.746 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.325147+0000", 2024-05-15T13:05:09.746 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1394999999999998e-05 2024-05-15T13:05:09.746 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.746 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:09.746 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.746 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.746 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.747 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.15540.0:149526 5.2 e453/418 5:4a7b51f4:::benchmark_data_smithi177_109991_object14335:head v 453'9342, mlcod=453'9339)", 2024-05-15T13:05:09.747 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:05:05.108081+0000", 2024-05-15T13:05:09.747 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.5324286049999998, 2024-05-15T13:05:09.747 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22160389799999999, 2024-05-15T13:05:09.747 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:09.747 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:05:09.747 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:09.747 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.747 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:09.747 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108081+0000", 2024-05-15T13:05:09.747 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.747 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.747 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.747 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:09.748 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108081+0000", 2024-05-15T13:05:09.748 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.748 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.748 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.748 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:09.748 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108082+0000", 2024-05-15T13:05:09.748 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1100000000000004e-07 2024-05-15T13:05:09.748 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.748 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.748 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:09.748 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108092+0000", 2024-05-15T13:05:09.748 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0679999999999999e-05 2024-05-15T13:05:09.748 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.748 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.748 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:09.749 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108105+0000", 2024-05-15T13:05:09.749 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2918999999999999e-05 2024-05-15T13:05:09.749 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.749 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.749 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:09.749 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108120+0000", 2024-05-15T13:05:09.749 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4184e-05 2024-05-15T13:05:09.749 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.749 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.749 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:09.749 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.320605+0000", 2024-05-15T13:05:09.749 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.212485062 2024-05-15T13:05:09.749 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.750 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.750 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:09.750 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.320626+0000", 2024-05-15T13:05:09.750 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1594999999999999e-05 2024-05-15T13:05:09.750 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.750 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.750 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:05:09.750 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.329617+0000", 2024-05-15T13:05:09.750 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0089910159999999992 2024-05-15T13:05:09.750 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.750 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.750 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:09.750 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.329685+0000", 2024-05-15T13:05:09.751 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7531000000000001e-05 2024-05-15T13:05:09.751 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.751 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:09.751 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.751 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.751 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.751 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.15540.0:149521 5.d e453/418 5:bfc1d209:::benchmark_data_smithi177_109991_object14330:head v 453'9514, mlcod=453'9512)", 2024-05-15T13:05:09.751 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:05:05.108380+0000", 2024-05-15T13:05:09.751 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.532129705, 2024-05-15T13:05:09.751 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.2077521, 2024-05-15T13:05:09.751 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:09.751 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:05:09.751 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:09.751 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.751 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:09.752 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108380+0000", 2024-05-15T13:05:09.752 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.752 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.752 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.752 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:09.752 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108380+0000", 2024-05-15T13:05:09.752 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.752 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.752 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.752 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:09.752 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108380+0000", 2024-05-15T13:05:09.752 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0200000000000002e-07 2024-05-15T13:05:09.752 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.753 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.753 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:09.753 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108387+0000", 2024-05-15T13:05:09.753 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3269999999999997e-06 2024-05-15T13:05:09.753 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.753 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.753 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:09.753 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108397+0000", 2024-05-15T13:05:09.753 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0155e-05 2024-05-15T13:05:09.753 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.754 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.754 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:09.754 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108403+0000", 2024-05-15T13:05:09.754 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4989999999999999e-06 2024-05-15T13:05:09.754 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.754 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.754 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:09.754 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108513+0000", 2024-05-15T13:05:09.754 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010940199999999999 2024-05-15T13:05:09.754 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.754 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.754 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:09.754 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108526+0000", 2024-05-15T13:05:09.755 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3169e-05 2024-05-15T13:05:09.755 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.755 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.755 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:05:09.755 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.316073+0000", 2024-05-15T13:05:09.755 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20754674300000001 2024-05-15T13:05:09.755 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.755 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.755 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:09.755 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.316132+0000", 2024-05-15T13:05:09.755 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9203000000000002e-05 2024-05-15T13:05:09.755 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.755 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:09.755 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.755 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.756 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.756 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.15540.0:149532 5.e e453/418 5:7f4b648b:::benchmark_data_smithi177_109991_object14341:head v 453'9374, mlcod=453'9372)", 2024-05-15T13:05:09.756 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:05:05.112260+0000", 2024-05-15T13:05:09.756 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.5282499520000004, 2024-05-15T13:05:09.756 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.21301437000000001, 2024-05-15T13:05:09.756 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:09.756 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:05:09.756 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:09.756 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.756 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:09.756 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.112260+0000", 2024-05-15T13:05:09.756 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.756 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.756 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.757 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:09.757 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.112260+0000", 2024-05-15T13:05:09.757 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.757 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.757 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.757 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:09.757 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.112260+0000", 2024-05-15T13:05:09.757 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0500000000000003e-07 2024-05-15T13:05:09.757 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.757 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.757 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:09.757 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.112267+0000", 2024-05-15T13:05:09.757 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4200000000000004e-06 2024-05-15T13:05:09.757 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.757 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.758 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:09.758 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.316035+0000", 2024-05-15T13:05:09.758 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20376809400000001 2024-05-15T13:05:09.758 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.758 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.758 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:09.758 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.316051+0000", 2024-05-15T13:05:09.758 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6132000000000001e-05 2024-05-15T13:05:09.758 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.758 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.758 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:09.758 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.316125+0000", 2024-05-15T13:05:09.758 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3817000000000002e-05 2024-05-15T13:05:09.758 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.759 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.759 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:09.759 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.316160+0000", 2024-05-15T13:05:09.759 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5612999999999999e-05 2024-05-15T13:05:09.759 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.759 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.759 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:05:09.759 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.325231+0000", 2024-05-15T13:05:09.759 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090705000000000004 2024-05-15T13:05:09.759 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.759 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.759 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:09.759 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.325274+0000", 2024-05-15T13:05:09.759 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3089e-05 2024-05-15T13:05:09.759 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.760 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:09.760 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.760 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.760 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.760 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:149533 5.3 5:c8312cf3:::benchmark_data_smithi177_109991_object14342:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e453)", 2024-05-15T13:05:09.760 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:05:05.117039+0000", 2024-05-15T13:05:09.760 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.5234702059999998, 2024-05-15T13:05:09.760 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.21683490499999999, 2024-05-15T13:05:09.760 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:09.760 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:05:09.760 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:09.760 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:05:09.760 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:05:09.760 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 149533 2024-05-15T13:05:09.761 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.761 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:09.761 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.761 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:09.761 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.117039+0000", 2024-05-15T13:05:09.761 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.761 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.761 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.761 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:09.761 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.117039+0000", 2024-05-15T13:05:09.761 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:09.761 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.761 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.761 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:09.762 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.117041+0000", 2024-05-15T13:05:09.762 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.269e-06 2024-05-15T13:05:09.762 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.762 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.762 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:09.762 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.117047+0000", 2024-05-15T13:05:09.762 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1670000000000004e-06 2024-05-15T13:05:09.762 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.762 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.762 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:09.762 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.316070+0000", 2024-05-15T13:05:09.762 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19902281999999999 2024-05-15T13:05:09.762 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.762 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.763 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:09.763 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.316083+0000", 2024-05-15T13:05:09.763 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3030999999999999e-05 2024-05-15T13:05:09.763 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.763 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.763 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:09.763 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.316152+0000", 2024-05-15T13:05:09.763 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9429000000000002e-05 2024-05-15T13:05:09.763 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.763 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.763 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:09.763 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.316292+0000", 2024-05-15T13:05:09.763 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014007099999999999 2024-05-15T13:05:09.763 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.763 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.764 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,6", 2024-05-15T13:05:09.764 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.316405+0000", 2024-05-15T13:05:09.764 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000112821 2024-05-15T13:05:09.764 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.764 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.764 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.764 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.325489+0000", 2024-05-15T13:05:09.764 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090839400000000004 2024-05-15T13:05:09.764 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.764 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.764 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:05:09.764 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.325508+0000", 2024-05-15T13:05:09.764 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9635000000000001e-05 2024-05-15T13:05:09.764 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.765 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.765 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:09.765 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.333780+0000", 2024-05-15T13:05:09.765 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0082712500000000008 2024-05-15T13:05:09.765 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.765 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.765 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:05:09.765 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.333832+0000", 2024-05-15T13:05:09.765 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2488999999999998e-05 2024-05-15T13:05:09.765 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:09.765 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:09.765 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:09.765 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.333874+0000", 2024-05-15T13:05:09.765 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1983000000000001e-05 2024-05-15T13:05:09.766 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.766 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:09.766 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.766 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:09.766 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:09.766 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:05:09.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:09.767 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:05:09.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:05:09.868 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:05:09.917 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:09.954 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:05:09.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:09.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:05:10.058 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:05:10.101 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:10.134 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:05:10.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:10.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:05:10.238 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:05:10.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:10 smithi120 ceph-mon[43961]: pgmap v1219: 17 pgs: 17 active+clean; 7.2 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 1.07k op/s 2024-05-15T13:05:10.296 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:10.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:10 smithi177 ceph-mon[44303]: pgmap v1219: 17 pgs: 17 active+clean; 7.2 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 1.07k op/s 2024-05-15T13:05:10.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:10 smithi027 ceph-mon[39795]: pgmap v1219: 17 pgs: 17 active+clean; 7.2 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 1.07k op/s 2024-05-15T13:05:10.577 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:05:10.580 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:05:11.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:11 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:05:11.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:11 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:05:11.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:11 smithi177 ceph-mon[44303]: osdmap e455: 12 total, 12 up, 11 in 2024-05-15T13:05:11.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:11 smithi177 ceph-mon[44303]: from='client.25525 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:05:11.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:11 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:05:11.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:11 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:05:11.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:11 smithi027 ceph-mon[39795]: osdmap e455: 12 total, 12 up, 11 in 2024-05-15T13:05:11.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:11 smithi027 ceph-mon[39795]: from='client.25525 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:05:11.522 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:05:11.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:11.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:05:11.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:11 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:05:11.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:11 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:05:11.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:11 smithi120 ceph-mon[43961]: osdmap e455: 12 total, 12 up, 11 in 2024-05-15T13:05:11.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:11 smithi120 ceph-mon[43961]: from='client.25525 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:05:11.628 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:05:11.672 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:11.713 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:05:11.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:11.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:05:11.820 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:05:11.882 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:11.948 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:05:11.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:11.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:05:12.060 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:05:12.134 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:12.181 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:05:12.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:12.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:05:12.285 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:05:12.342 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:12.367 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:12 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:05:12.367 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:12 smithi027 ceph-mon[39795]: pgmap v1221: 17 pgs: 17 active+clean; 7.1 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 905 op/s 2024-05-15T13:05:12.370 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:05:12.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:12.377 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:05:12.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:12 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:05:12.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:12 smithi177 ceph-mon[44303]: pgmap v1221: 17 pgs: 17 active+clean; 7.1 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 905 op/s 2024-05-15T13:05:12.478 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:05:12.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:12 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:05:12.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:12 smithi120 ceph-mon[43961]: pgmap v1221: 17 pgs: 17 active+clean; 7.1 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 905 op/s 2024-05-15T13:05:12.542 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:12.574 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:05:12.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:12.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:05:12.684 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:05:12.729 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-05-15T13:05:12.770 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:13.118 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":1220,"stamp":"2024-05-15T13:05:10.130124+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7753630264,"num_objects":118305,"num_object_clones":0,"num_object_copies":354915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118305,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":152256,"num_write_kb":8654971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152214,"ondisk_log_size":152214,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":27807524,"kb_used_data":23306636,"kb_used_omap":17,"kb_used_meta":4500782,"kb_avail":1003389148,"statfs":{"total":1055945392128,"available":1027470487552,"internally_reserved":0,"allocated":23865995264,"data_stored":23862902976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17551,"internal_metadata":4608801649},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,3,8,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-435224576,"num_objects":-6641,"num_object_clones":0,"num_object_copies":-19923,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6641,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.055261"},"pg_stats":[{"pgid":"5.f","version":"453'9535","reported_seq":9628,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-05-15T13:05:07.846826+0000","last_change":"2024-05-15T13:04:41.104880+0000","last_active":"2024-05-15T13:05:07.846826+0000","last_peered":"2024-05-15T13:05:07.846826+0000","last_clean":"2024-05-15T13:05:07.846826+0000","last_became_active":"2024-05-15T13:03:11.328783+0000","last_became_peered":"2024-05-15T13:03:11.328783+0000","last_unstale":"2024-05-15T13:05:07.846826+0000","last_undegraded":"2024-05-15T13:05:07.846826+0000","last_fullsized":"2024-05-15T13:05:07.846826+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:03:10.302515+0000","objects_scrubbed":999,"log_size":9535,"log_dups_size":0,"ondisk_log_size":9535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:10.560896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484114432,"num_objects":7387,"num_object_clones":0,"num_object_copies":22161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9535,"num_write_kb":541504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"453'9392","reported_seq":9550,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-05-15T13:05:05.593687+0000","last_change":"2024-05-15T13:04:26.638047+0000","last_active":"2024-05-15T13:05:05.593687+0000","last_peered":"2024-05-15T13:05:05.593687+0000","last_clean":"2024-05-15T13:05:05.593687+0000","last_became_active":"2024-05-15T13:03:11.330241+0000","last_became_peered":"2024-05-15T13:03:11.330241+0000","last_unstale":"2024-05-15T13:05:05.593687+0000","last_undegraded":"2024-05-15T13:05:05.593687+0000","last_fullsized":"2024-05-15T13:05:05.593687+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'6959","last_scrub_stamp":"2024-05-15T13:04:26.637942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:26.637942+0000","objects_scrubbed":6905,"log_size":9392,"log_dups_size":0,"ondisk_log_size":9392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:43.654277+0000","scrub_duration":1597,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":490602496,"num_objects":7486,"num_object_clones":0,"num_object_copies":22458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9392,"num_write_kb":540096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"453'9665","reported_seq":9829,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-05-15T13:05:07.552116+0000","last_change":"2024-05-15T13:04:34.347782+0000","last_active":"2024-05-15T13:05:07.552116+0000","last_peered":"2024-05-15T13:05:07.552116+0000","last_clean":"2024-05-15T13:05:07.552116+0000","last_became_active":"2024-05-15T13:03:11.330919+0000","last_became_peered":"2024-05-15T13:03:11.330919+0000","last_unstale":"2024-05-15T13:05:07.552116+0000","last_undegraded":"2024-05-15T13:05:07.552116+0000","last_fullsized":"2024-05-15T13:05:07.552116+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'7491","last_scrub_stamp":"2024-05-15T13:04:34.347495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:34.347495+0000","objects_scrubbed":7414,"log_size":9665,"log_dups_size":0,"ondisk_log_size":9665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:36.777946+0000","scrub_duration":2380,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":494665752,"num_objects":7549,"num_object_clones":0,"num_object_copies":22647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7549,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9665,"num_write_kb":550785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"453'9437","reported_seq":9600,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-05-15T13:05:06.449906+0000","last_change":"2024-05-15T13:04:35.286472+0000","last_active":"2024-05-15T13:05:06.449906+0000","last_peered":"2024-05-15T13:05:06.449906+0000","last_clean":"2024-05-15T13:05:06.449906+0000","last_became_active":"2024-05-15T13:03:11.327387+0000","last_became_peered":"2024-05-15T13:03:11.327387+0000","last_unstale":"2024-05-15T13:05:06.449906+0000","last_undegraded":"2024-05-15T13:05:06.449906+0000","last_fullsized":"2024-05-15T13:05:06.449906+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'7507","last_scrub_stamp":"2024-05-15T13:04:35.285734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:35.285734+0000","objects_scrubbed":7450,"log_size":9437,"log_dups_size":0,"ondisk_log_size":9437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:57.795544+0000","scrub_duration":1571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":488964096,"num_objects":7461,"num_object_clones":0,"num_object_copies":22383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9437,"num_write_kb":540736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"454'9544","reported_seq":9703,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-05-15T13:05:09.216939+0000","last_change":"2024-05-15T13:04:25.608723+0000","last_active":"2024-05-15T13:05:09.216939+0000","last_peered":"2024-05-15T13:05:09.216939+0000","last_clean":"2024-05-15T13:05:09.216939+0000","last_became_active":"2024-05-15T13:03:11.328691+0000","last_became_peered":"2024-05-15T13:03:11.328691+0000","last_unstale":"2024-05-15T13:05:09.216939+0000","last_undegraded":"2024-05-15T13:05:09.216939+0000","last_fullsized":"2024-05-15T13:05:09.216939+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'6796","last_scrub_stamp":"2024-05-15T13:04:25.608583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:25.608583+0000","objects_scrubbed":6750,"log_size":9544,"log_dups_size":0,"ondisk_log_size":9544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:45.331376+0000","scrub_duration":1526,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475136000,"num_objects":7250,"num_object_clones":0,"num_object_copies":21750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9544,"num_write_kb":537408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,5],"acting":[0,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"453'9482","reported_seq":9635,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-05-15T13:05:07.541795+0000","last_change":"2024-05-15T13:04:20.798543+0000","last_active":"2024-05-15T13:05:07.541795+0000","last_peered":"2024-05-15T13:05:07.541795+0000","last_clean":"2024-05-15T13:05:07.541795+0000","last_became_active":"2024-05-15T13:03:11.330425+0000","last_became_peered":"2024-05-15T13:03:11.330425+0000","last_unstale":"2024-05-15T13:05:07.541795+0000","last_undegraded":"2024-05-15T13:05:07.541795+0000","last_fullsized":"2024-05-15T13:05:07.541795+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'6531","last_scrub_stamp":"2024-05-15T13:04:20.798310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:20.798310+0000","objects_scrubbed":6490,"log_size":9482,"log_dups_size":0,"ondisk_log_size":9482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:28.156588+0000","scrub_duration":1689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485097472,"num_objects":7402,"num_object_clones":0,"num_object_copies":22206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9482,"num_write_kb":540288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"454'9451","reported_seq":9609,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-05-15T13:05:08.691486+0000","last_change":"2024-05-15T13:04:27.305971+0000","last_active":"2024-05-15T13:05:08.691486+0000","last_peered":"2024-05-15T13:05:08.691486+0000","last_clean":"2024-05-15T13:05:08.691486+0000","last_became_active":"2024-05-15T13:03:11.329210+0000","last_became_peered":"2024-05-15T13:03:11.329210+0000","last_unstale":"2024-05-15T13:05:08.691486+0000","last_undegraded":"2024-05-15T13:05:08.691486+0000","last_fullsized":"2024-05-15T13:05:08.691486+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'6893","last_scrub_stamp":"2024-05-15T13:04:27.305709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:27.305709+0000","objects_scrubbed":6847,"log_size":9451,"log_dups_size":0,"ondisk_log_size":9451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:39.171201+0000","scrub_duration":1459,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478347264,"num_objects":7299,"num_object_clones":0,"num_object_copies":21897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9451,"num_write_kb":536000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,9],"acting":[6,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"453'9328","reported_seq":9486,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-05-15T13:05:07.533187+0000","last_change":"2024-05-15T13:04:29.718135+0000","last_active":"2024-05-15T13:05:07.533187+0000","last_peered":"2024-05-15T13:05:07.533187+0000","last_clean":"2024-05-15T13:05:07.533187+0000","last_became_active":"2024-05-15T13:03:11.330983+0000","last_became_peered":"2024-05-15T13:03:11.330983+0000","last_unstale":"2024-05-15T13:05:07.533187+0000","last_undegraded":"2024-05-15T13:05:07.533187+0000","last_fullsized":"2024-05-15T13:05:07.533187+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"447'7023","last_scrub_stamp":"2024-05-15T13:04:29.717870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:29.717870+0000","objects_scrubbed":6961,"log_size":9328,"log_dups_size":0,"ondisk_log_size":9328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:47.890055+0000","scrub_duration":1740,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481689600,"num_objects":7350,"num_object_clones":0,"num_object_copies":22050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9328,"num_write_kb":533696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"453'9535","reported_seq":9688,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-05-15T13:05:07.883596+0000","last_change":"2024-05-15T13:04:20.177631+0000","last_active":"2024-05-15T13:05:07.883596+0000","last_peered":"2024-05-15T13:05:07.883596+0000","last_clean":"2024-05-15T13:05:07.883596+0000","last_became_active":"2024-05-15T13:03:11.329484+0000","last_became_peered":"2024-05-15T13:03:11.329484+0000","last_unstale":"2024-05-15T13:05:07.883596+0000","last_undegraded":"2024-05-15T13:05:07.883596+0000","last_fullsized":"2024-05-15T13:05:07.883596+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'6536","last_scrub_stamp":"2024-05-15T13:04:20.177485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:20.177485+0000","objects_scrubbed":6499,"log_size":9535,"log_dups_size":0,"ondisk_log_size":9535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:29.617242+0000","scrub_duration":1314,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487260160,"num_objects":7435,"num_object_clones":0,"num_object_copies":22305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9535,"num_write_kb":543040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"453'9452","reported_seq":9605,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-05-15T13:05:07.531490+0000","last_change":"2024-05-15T13:04:18.779479+0000","last_active":"2024-05-15T13:05:07.531490+0000","last_peered":"2024-05-15T13:05:07.531490+0000","last_clean":"2024-05-15T13:05:07.531490+0000","last_became_active":"2024-05-15T13:03:11.332923+0000","last_became_peered":"2024-05-15T13:03:11.332923+0000","last_unstale":"2024-05-15T13:05:07.531490+0000","last_undegraded":"2024-05-15T13:05:07.531490+0000","last_fullsized":"2024-05-15T13:05:07.531490+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'6425","last_scrub_stamp":"2024-05-15T13:04:18.779285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:18.779285+0000","objects_scrubbed":6366,"log_size":9452,"log_dups_size":0,"ondisk_log_size":9452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:48.641808+0000","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482476032,"num_objects":7362,"num_object_clones":0,"num_object_copies":22086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9452,"num_write_kb":538048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,10],"acting":[2,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"454'9520","reported_seq":9675,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-05-15T13:05:09.214199+0000","last_change":"2024-05-15T13:04:22.066492+0000","last_active":"2024-05-15T13:05:09.214199+0000","last_peered":"2024-05-15T13:05:09.214199+0000","last_clean":"2024-05-15T13:05:09.214199+0000","last_became_active":"2024-05-15T13:03:11.333336+0000","last_became_peered":"2024-05-15T13:03:11.333336+0000","last_unstale":"2024-05-15T13:05:09.214199+0000","last_undegraded":"2024-05-15T13:05:09.214199+0000","last_fullsized":"2024-05-15T13:05:09.214199+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'6582","last_scrub_stamp":"2024-05-15T13:04:22.066294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:22.066294+0000","objects_scrubbed":6533,"log_size":9520,"log_dups_size":0,"ondisk_log_size":9520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:46.801279+0000","scrub_duration":1991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477626368,"num_objects":7288,"num_object_clones":0,"num_object_copies":21864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9520,"num_write_kb":537856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"454'9715","reported_seq":9871,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-05-15T13:05:09.236364+0000","last_change":"2024-05-15T13:04:22.690622+0000","last_active":"2024-05-15T13:05:09.236364+0000","last_peered":"2024-05-15T13:05:09.236364+0000","last_clean":"2024-05-15T13:05:09.236364+0000","last_became_active":"2024-05-15T13:03:11.327788+0000","last_became_peered":"2024-05-15T13:03:11.327788+0000","last_unstale":"2024-05-15T13:05:09.236364+0000","last_undegraded":"2024-05-15T13:05:09.236364+0000","last_fullsized":"2024-05-15T13:05:09.236364+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'6704","last_scrub_stamp":"2024-05-15T13:04:22.690346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:22.690346+0000","objects_scrubbed":6642,"log_size":9715,"log_dups_size":0,"ondisk_log_size":9715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:51.461018+0000","scrub_duration":1607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":486735872,"num_objects":7427,"num_object_clones":0,"num_object_copies":22281,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9715,"num_write_kb":548544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"454'9526","reported_seq":9684,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-05-15T13:05:08.680947+0000","last_change":"2024-05-15T13:04:26.196007+0000","last_active":"2024-05-15T13:05:08.680947+0000","last_peered":"2024-05-15T13:05:08.680947+0000","last_clean":"2024-05-15T13:05:08.680947+0000","last_became_active":"2024-05-15T13:03:11.330316+0000","last_became_peered":"2024-05-15T13:03:11.330316+0000","last_unstale":"2024-05-15T13:05:08.680947+0000","last_undegraded":"2024-05-15T13:05:08.680947+0000","last_fullsized":"2024-05-15T13:05:08.680947+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'6865","last_scrub_stamp":"2024-05-15T13:04:26.195895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:26.195895+0000","objects_scrubbed":6825,"log_size":9526,"log_dups_size":0,"ondisk_log_size":9526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:55.500383+0000","scrub_duration":1403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482607104,"num_objects":7364,"num_object_clones":0,"num_object_copies":22092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9526,"num_write_kb":540480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"454'9669","reported_seq":9825,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-05-15T13:05:09.897071+0000","last_change":"2024-05-15T13:04:22.294876+0000","last_active":"2024-05-15T13:05:09.897071+0000","last_peered":"2024-05-15T13:05:09.897071+0000","last_clean":"2024-05-15T13:05:09.897071+0000","last_became_active":"2024-05-15T13:03:11.430987+0000","last_became_peered":"2024-05-15T13:03:11.430987+0000","last_unstale":"2024-05-15T13:05:09.897071+0000","last_undegraded":"2024-05-15T13:05:09.897071+0000","last_fullsized":"2024-05-15T13:05:09.897071+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'6645","last_scrub_stamp":"2024-05-15T13:04:22.294671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:22.294671+0000","objects_scrubbed":6590,"log_size":9669,"log_dups_size":0,"ondisk_log_size":9669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:42.703798+0000","scrub_duration":1280,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477560832,"num_objects":7287,"num_object_clones":0,"num_object_copies":21861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9669,"num_write_kb":542592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"453'9365","reported_seq":9531,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-05-15T13:05:06.407312+0000","last_change":"2024-05-15T13:04:39.845227+0000","last_active":"2024-05-15T13:05:06.407312+0000","last_peered":"2024-05-15T13:05:06.407312+0000","last_clean":"2024-05-15T13:05:06.407312+0000","last_became_active":"2024-05-15T13:03:11.330479+0000","last_became_peered":"2024-05-15T13:03:11.330479+0000","last_unstale":"2024-05-15T13:05:06.407312+0000","last_undegraded":"2024-05-15T13:05:06.407312+0000","last_fullsized":"2024-05-15T13:05:06.407312+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'7705","last_scrub_stamp":"2024-05-15T13:04:39.844914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:39.844914+0000","objects_scrubbed":7656,"log_size":9365,"log_dups_size":0,"ondisk_log_size":9365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:06:06.170367+0000","scrub_duration":2147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489226240,"num_objects":7465,"num_object_clones":0,"num_object_copies":22395,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9365,"num_write_kb":538560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,4],"acting":[1,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"453'9459","reported_seq":9614,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-05-15T13:05:06.372697+0000","last_change":"2024-05-15T13:04:21.985812+0000","last_active":"2024-05-15T13:05:06.372697+0000","last_peered":"2024-05-15T13:05:06.372697+0000","last_clean":"2024-05-15T13:05:06.372697+0000","last_became_active":"2024-05-15T13:03:11.331291+0000","last_became_peered":"2024-05-15T13:03:11.331291+0000","last_unstale":"2024-05-15T13:05:06.372697+0000","last_undegraded":"2024-05-15T13:05:06.372697+0000","last_fullsized":"2024-05-15T13:05:06.372697+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'6673","last_scrub_stamp":"2024-05-15T13:04:21.985599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:21.985599+0000","objects_scrubbed":6626,"log_size":9459,"log_dups_size":0,"ondisk_log_size":9459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:27.438473+0000","scrub_duration":1491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":490930176,"num_objects":7491,"num_object_clones":0,"num_object_copies":22473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9459,"num_write_kb":542400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":733,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-05-15T13:05:03.080466+0000","last_change":"2024-05-15T13:04:38.891844+0000","last_active":"2024-05-15T13:05:03.080466+0000","last_peered":"2024-05-15T13:05:03.080466+0000","last_clean":"2024-05-15T13:05:03.080466+0000","last_became_active":"2024-05-15T13:00:26.045412+0000","last_became_peered":"2024-05-15T13:00:26.045412+0000","last_unstale":"2024-05-15T13:05:03.080466+0000","last_undegraded":"2024-05-15T13:05:03.080466+0000","last_fullsized":"2024-05-15T13:05:03.080466+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:04:38.891702+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:04:38.891702+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:49.376370+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":7753039896,"num_objects":118303,"num_object_clones":0,"num_object_copies":354909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118303,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152075,"num_write_kb":8652033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23263457280,"data_stored":23263445064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152075,"ondisk_log_size":152075,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056876,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2297560,"kb_used_data":1905424,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":91447592,"statfs":{"total":95995035648,"available":93642334208,"internally_reserved":0,"allocated":1951154176,"data_stored":1950872283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":401537478},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351216,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3468124,"kb_used_data":3053780,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":90277028,"statfs":{"total":95995035648,"available":92443676672,"internally_reserved":0,"allocated":3127070720,"data_stored":3126789347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":424278464},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514742,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2967252,"kb_used_data":2444620,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":90777900,"statfs":{"total":95995035648,"available":92956569600,"internally_reserved":0,"allocated":2503290880,"data_stored":2503013059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":535165376},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874492,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1362720,"kb_used_data":1070552,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":92382432,"statfs":{"total":95995035648,"available":94599610368,"internally_reserved":0,"allocated":1096245248,"data_stored":1095962932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":240518168830,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430468,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1959068,"kb_used_data":1580820,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":91786084,"statfs":{"total":95995035648,"available":93988950016,"internally_reserved":0,"allocated":1618759680,"data_stored":1618478327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":387316166},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575135,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2423452,"kb_used_data":1923028,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":91321700,"statfs":{"total":95995035648,"available":93513420800,"internally_reserved":0,"allocated":1969180672,"data_stored":1968899319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738664,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4235364,"kb_used_data":3757148,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":89509788,"statfs":{"total":95995035648,"available":91658022912,"internally_reserved":0,"allocated":3847319552,"data_stored":3847033164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":489683397},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411612,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1711892,"kb_used_data":1423884,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":92033260,"statfs":{"total":95995035648,"available":94242058240,"internally_reserved":0,"allocated":1458057216,"data_stored":1457779415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149973,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":912020,"kb_used_data":483788,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":92833132,"statfs":{"total":95995035648,"available":95061127168,"internally_reserved":0,"allocated":495398912,"data_stored":495121111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":438499783},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986449,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2755104,"kb_used_data":2326872,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":90990048,"statfs":{"total":95995035648,"available":93173809152,"internally_reserved":0,"allocated":2382716928,"data_stored":2382434612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692106,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3714968,"kb_used_data":3336720,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":90030184,"statfs":{"total":95995035648,"available":92190908416,"internally_reserved":0,"allocated":3416801280,"data_stored":3416519407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3836678144,"data_stored":3836674072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1960902656,"data_stored":1960902656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1453391872,"data_stored":1453391872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":490668032,"data_stored":490668032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2378563584,"data_stored":2378563584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3402960896,"data_stored":3402956824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1450049536,"data_stored":1450049536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":947978240,"data_stored":947978240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2463629312,"data_stored":2463629312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2934374400,"data_stored":2934374400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1944260608,"data_stored":1944256536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:05:13.120 INFO:tasks.thrashosds.thrasher:Setting 5.b to [1, 0, 7] 2024-05-15T13:05:13.120 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '5.b', '1', '0', '7'] 2024-05-15T13:05:13.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pg-upmap 5.b 1 0 7 2024-05-15T13:05:13.141 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:05:13.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:13.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:05:13.254 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:05:13.319 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:13.345 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:05:13.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:13.349 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:05:13.353 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:05:13.413 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:13 smithi027 ceph-mon[39795]: 5.f scrub starts 2024-05-15T13:05:13.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:13 smithi177 ceph-mon[44303]: 5.f scrub starts 2024-05-15T13:05:13.450 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:05:13.477 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:13.508 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:05:13.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:13.515 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:05:13.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:13 smithi120 ceph-mon[43961]: 5.f scrub starts 2024-05-15T13:05:13.616 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:05:13.652 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:13.696 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:05:13.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:13.699 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:05:13.800 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:05:13.846 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:13.908 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:05:13.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:13.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:05:13.986 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:05:13.991 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:05:14.014 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:05:14.046 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:14.069 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:05:14.071 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:05:14.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:14.076 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:05:14.177 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:05:14.226 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:14.364 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:14 smithi177 ceph-mon[44303]: 5.f scrub ok 2024-05-15T13:05:14.365 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:14 smithi177 ceph-mon[44303]: pgmap v1222: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.0 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-05-15T13:05:14.365 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:05:14.365 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:05:14.436 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:05:14.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:14.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:05:14.473 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:14 smithi027 ceph-mon[39795]: 5.f scrub ok 2024-05-15T13:05:14.473 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:14 smithi027 ceph-mon[39795]: pgmap v1222: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.0 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-05-15T13:05:14.473 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:05:14.474 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:05:14.542 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:05:14.592 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:14.617 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:05:14.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:14.620 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:05:14.723 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:05:14.766 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:14 smithi120 ceph-mon[43961]: 5.f scrub ok 2024-05-15T13:05:14.766 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:14 smithi120 ceph-mon[43961]: pgmap v1222: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.0 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-05-15T13:05:14.766 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:05:14.766 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:05:14.769 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:14.807 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:05:14.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:14.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:05:14.914 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:05:14.979 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:15.430 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:15 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:05:15.430 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:15 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:05:15.430 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:15 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:05:15.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:15 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:05:15.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:15 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:05:15.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:15 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:05:15.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:15 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:05:15.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:15 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:05:15.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:15 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:05:16.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:16 smithi027 ceph-mon[39795]: pgmap v1223: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.9 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 843 op/s 2024-05-15T13:05:16.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:16 smithi177 ceph-mon[44303]: pgmap v1223: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.9 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 843 op/s 2024-05-15T13:05:16.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:16 smithi120 ceph-mon[43961]: pgmap v1223: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.9 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 843 op/s 2024-05-15T13:05:19.074 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:05:19.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:19.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:05:19.183 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:05:19.202 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:19 smithi027 ceph-mon[39795]: pgmap v1224: 17 pgs: 17 active+clean; 6.8 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 928 op/s 2024-05-15T13:05:19.202 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:19 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3643999222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:05:19.267 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:19.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:19 smithi177 ceph-mon[44303]: pgmap v1224: 17 pgs: 17 active+clean; 6.8 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 928 op/s 2024-05-15T13:05:19.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:19 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3643999222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:05:19.442 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:05:19.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:19.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:05:19.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:19 smithi120 ceph-mon[43961]: pgmap v1224: 17 pgs: 17 active+clean; 6.8 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 928 op/s 2024-05-15T13:05:19.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:19 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3643999222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:05:19.548 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:05:19.580 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:19.614 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:05:19.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:19.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:05:19.725 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:05:19.801 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:19.869 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:05:19.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:19.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:05:19.972 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:05:20.061 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:21.089 INFO:teuthology.orchestra.run.smithi027.stderr:set 5.b pg_upmap mapping to [1,0,7] 2024-05-15T13:05:21.094 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:05:21.398 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:21 smithi027 ceph-mon[39795]: pgmap v1225: 17 pgs: 17 active+clean; 6.7 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 910 op/s 2024-05-15T13:05:21.398 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:21 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2186566953' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:05:21.398 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:21 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:05:21.398 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:21 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3441341429' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.b", "id": [1, 0, 7]}]: dispatch 2024-05-15T13:05:21.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:21 smithi177 ceph-mon[44303]: pgmap v1225: 17 pgs: 17 active+clean; 6.7 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 910 op/s 2024-05-15T13:05:21.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:21 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2186566953' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:05:21.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:21 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:05:21.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:21 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3441341429' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.b", "id": [1, 0, 7]}]: dispatch 2024-05-15T13:05:21.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:21 smithi120 ceph-mon[43961]: pgmap v1225: 17 pgs: 17 active+clean; 6.7 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 910 op/s 2024-05-15T13:05:21.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:21 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2186566953' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:05:21.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:21 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:05:21.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:21 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3441341429' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.b", "id": [1, 0, 7]}]: dispatch 2024-05-15T13:05:21.928 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:05:21.928 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:05:21.928 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:05:21.928 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:05:21.929 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:05:21.929 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:05:21.929 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:05:21.929 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:05:21.929 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:05:21.929 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:05:21.929 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:05:21.929 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:05:21.929 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:05:21.929 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:05:21.930 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:05:21.930 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:05:21.930 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:05:21.930 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:05:21.930 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:05:21.930 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:05:21.930 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:05:21.930 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:05:21.930 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:05:21.930 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:05:21.965 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:05:21.965 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:05:22.165 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:05:22.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:22.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:05:22.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:22 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:05:22.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:22 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T13:05:22.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:22 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:05:22.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:22 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3441341429' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.b", "id": [1, 0, 7]}]': finished 2024-05-15T13:05:22.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:22 smithi027 ceph-mon[39795]: osdmap e456: 12 total, 12 up, 11 in 2024-05-15T13:05:22.274 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:05:22.322 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:22.363 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:05:22.367 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:22 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:05:22.367 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:22 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T13:05:22.368 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:22 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:05:22.368 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:22 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3441341429' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.b", "id": [1, 0, 7]}]': finished 2024-05-15T13:05:22.368 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:22 smithi120 ceph-mon[43961]: osdmap e456: 12 total, 12 up, 11 in 2024-05-15T13:05:22.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:22.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:05:22.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:22 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:05:22.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:22 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T13:05:22.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:22 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:05:22.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:22 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3441341429' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.b", "id": [1, 0, 7]}]': finished 2024-05-15T13:05:22.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:22 smithi177 ceph-mon[44303]: osdmap e456: 12 total, 12 up, 11 in 2024-05-15T13:05:22.470 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:05:22.520 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:22.547 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:05:22.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:22.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:05:22.654 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:05:22.718 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:23.001 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:05:23.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:23.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:05:23.112 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:05:23.152 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:23.187 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:05:23.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:23.193 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:05:23.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:23 smithi027 ceph-mon[39795]: pgmap v1227: 17 pgs: 17 active+clean; 6.6 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 899 op/s 2024-05-15T13:05:23.293 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:05:23.344 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:23.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:23 smithi177 ceph-mon[44303]: pgmap v1227: 17 pgs: 17 active+clean; 6.6 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 899 op/s 2024-05-15T13:05:23.436 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:05:23.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:23.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:05:23.466 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-05-15T13:05:23.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:23 smithi120 ceph-mon[43961]: pgmap v1227: 17 pgs: 17 active+clean; 6.6 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 899 op/s 2024-05-15T13:05:23.540 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:05:23.614 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:23.697 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:05:23.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:23.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:05:23.804 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:05:23.835 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:23.864 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:05:23.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:23.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:05:23.970 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:05:24.020 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:24.047 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:05:24.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:24.052 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:05:24.153 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:05:24.184 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:24.222 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:05:24.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:24.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:05:24.328 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:05:24.369 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:24.401 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:05:24.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:24.409 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:05:24.510 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:05:24.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:24 smithi027 ceph-mon[39795]: pgmap v1228: 17 pgs: 17 active+clean; 6.5 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 894 op/s 2024-05-15T13:05:24.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:24 smithi120 ceph-mon[43961]: pgmap v1228: 17 pgs: 17 active+clean; 6.5 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 894 op/s 2024-05-15T13:05:24.557 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:24.590 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:05:24.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:24.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:05:24.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:24 smithi177 ceph-mon[44303]: pgmap v1228: 17 pgs: 17 active+clean; 6.5 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 894 op/s 2024-05-15T13:05:24.697 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:05:24.705 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:05:24.730 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:24.754 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:05:24.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:24.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:05:24.859 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:05:24.901 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:24.934 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:05:24.938 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:05:24.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:24.943 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:05:25.044 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:05:25.082 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:25.113 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:05:25.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:25.118 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:05:25.219 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:05:25.261 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:25.296 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:05:25.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:25.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:05:25.400 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:05:25.431 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:25.457 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:05:25.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:25.464 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:05:25.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=634536324,config=1059521453,health=3897446751,kv=3235169738}) 2024-05-15T13:05:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=2430553457,logm=3327244219}) 2024-05-15T13:05:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=538410766}) 2024-05-15T13:05:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=188826085}) 2024-05-15T13:05:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=772883439}) 2024-05-15T13:05:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3614826148}) 2024-05-15T13:05:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3773886210}) 2024-05-15T13:05:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2451772996}) 2024-05-15T13:05:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1597541375}) 2024-05-15T13:05:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3119014430}) 2024-05-15T13:05:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=425805924}) 2024-05-15T13:05:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=690354095}) 2024-05-15T13:05:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=659643006}) 2024-05-15T13:05:25.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487252965}) 2024-05-15T13:05:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2782855003}) 2024-05-15T13:05:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2568277089}) 2024-05-15T13:05:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3484924554}) 2024-05-15T13:05:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3695821502}) 2024-05-15T13:05:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2419615486}) 2024-05-15T13:05:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1333808243}) 2024-05-15T13:05:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3340063315}) 2024-05-15T13:05:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2995622711}) 2024-05-15T13:05:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=402628725}) 2024-05-15T13:05:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2475282545}) 2024-05-15T13:05:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1949093649}) 2024-05-15T13:05:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=254380693}) 2024-05-15T13:05:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2644436125}) 2024-05-15T13:05:25.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1441203711}) 2024-05-15T13:05:25.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4121392414}) 2024-05-15T13:05:25.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=638346149}) 2024-05-15T13:05:25.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1272015075}) 2024-05-15T13:05:25.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2517240245}) 2024-05-15T13:05:25.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1406105385}) 2024-05-15T13:05:25.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1440272964}) 2024-05-15T13:05:25.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214883712}) 2024-05-15T13:05:25.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2642697010}) 2024-05-15T13:05:25.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4189148566}) 2024-05-15T13:05:25.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2079120647}) 2024-05-15T13:05:25.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4042722344}) 2024-05-15T13:05:25.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3583411424}) 2024-05-15T13:05:25.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2532057694}) 2024-05-15T13:05:25.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2467254877}) 2024-05-15T13:05:25.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2498512077}) 2024-05-15T13:05:25.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=17,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=45} crc {logm=4143295141,mdsmap=3984425268,mgr=76093979,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=1185698319,mon_config_key=3542150020}) 2024-05-15T13:05:25.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2233563460}) 2024-05-15T13:05:25.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1431521387}) 2024-05-15T13:05:25.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=30,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=52} crc {mon_config_key=23789456,monmap=2435304556,osd_metadata=1212035326,osd_pg_creating=3316468889,osdmap=1792139593}) 2024-05-15T13:05:25.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=53} crc {osdmap=3774903397}) 2024-05-15T13:05:25.564 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:05:25.604 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=634536324,config=1059521453,health=3897446751,kv=3235169738}) 2024-05-15T13:05:25.605 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=2430553457,logm=3327244219}) 2024-05-15T13:05:25.605 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=538410766}) 2024-05-15T13:05:25.605 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=188826085}) 2024-05-15T13:05:25.605 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=772883439}) 2024-05-15T13:05:25.605 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3614826148}) 2024-05-15T13:05:25.605 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3773886210}) 2024-05-15T13:05:25.605 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2451772996}) 2024-05-15T13:05:25.605 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1597541375}) 2024-05-15T13:05:25.605 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3119014430}) 2024-05-15T13:05:25.606 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=425805924}) 2024-05-15T13:05:25.606 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=690354095}) 2024-05-15T13:05:25.606 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=659643006}) 2024-05-15T13:05:25.606 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487252965}) 2024-05-15T13:05:25.606 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2782855003}) 2024-05-15T13:05:25.606 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2568277089}) 2024-05-15T13:05:25.606 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3484924554}) 2024-05-15T13:05:25.606 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3695821502}) 2024-05-15T13:05:25.606 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2419615486}) 2024-05-15T13:05:25.606 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1333808243}) 2024-05-15T13:05:25.606 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3340063315}) 2024-05-15T13:05:25.606 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2995622711}) 2024-05-15T13:05:25.607 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=402628725}) 2024-05-15T13:05:25.607 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2475282545}) 2024-05-15T13:05:25.607 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1949093649}) 2024-05-15T13:05:25.607 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=254380693}) 2024-05-15T13:05:25.607 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2644436125}) 2024-05-15T13:05:25.607 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1441203711}) 2024-05-15T13:05:25.607 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4121392414}) 2024-05-15T13:05:25.607 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=638346149}) 2024-05-15T13:05:25.607 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1272015075}) 2024-05-15T13:05:25.607 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2517240245}) 2024-05-15T13:05:25.607 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1406105385}) 2024-05-15T13:05:25.607 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1440272964}) 2024-05-15T13:05:25.607 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214883712}) 2024-05-15T13:05:25.607 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2642697010}) 2024-05-15T13:05:25.607 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4189148566}) 2024-05-15T13:05:25.608 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2079120647}) 2024-05-15T13:05:25.608 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4042722344}) 2024-05-15T13:05:25.608 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3583411424}) 2024-05-15T13:05:25.608 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2532057694}) 2024-05-15T13:05:25.608 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2467254877}) 2024-05-15T13:05:25.608 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2498512077}) 2024-05-15T13:05:25.608 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=17,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=45} crc {logm=4143295141,mdsmap=3984425268,mgr=76093979,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=1185698319,mon_config_key=3542150020}) 2024-05-15T13:05:25.608 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2233563460}) 2024-05-15T13:05:25.608 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1431521387}) 2024-05-15T13:05:25.608 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=30,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=52} crc {mon_config_key=23789456,monmap=2435304556,osd_metadata=1212035326,osd_pg_creating=3316468889,osdmap=1792139593}) 2024-05-15T13:05:25.608 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=53} crc {osdmap=3774903397}) 2024-05-15T13:05:25.609 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:25.661 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:05:25.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:25.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:05:25.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=634536324,config=1059521453,health=3897446751,kv=3235169738}) 2024-05-15T13:05:25.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=2430553457,logm=3327244219}) 2024-05-15T13:05:25.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=538410766}) 2024-05-15T13:05:25.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=188826085}) 2024-05-15T13:05:25.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=772883439}) 2024-05-15T13:05:25.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3614826148}) 2024-05-15T13:05:25.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3773886210}) 2024-05-15T13:05:25.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2451772996}) 2024-05-15T13:05:25.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1597541375}) 2024-05-15T13:05:25.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3119014430}) 2024-05-15T13:05:25.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=425805924}) 2024-05-15T13:05:25.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=690354095}) 2024-05-15T13:05:25.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=659643006}) 2024-05-15T13:05:25.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487252965}) 2024-05-15T13:05:25.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2782855003}) 2024-05-15T13:05:25.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2568277089}) 2024-05-15T13:05:25.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3484924554}) 2024-05-15T13:05:25.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3695821502}) 2024-05-15T13:05:25.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2419615486}) 2024-05-15T13:05:25.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1333808243}) 2024-05-15T13:05:25.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3340063315}) 2024-05-15T13:05:25.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2995622711}) 2024-05-15T13:05:25.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=402628725}) 2024-05-15T13:05:25.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2475282545}) 2024-05-15T13:05:25.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1949093649}) 2024-05-15T13:05:25.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=254380693}) 2024-05-15T13:05:25.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2644436125}) 2024-05-15T13:05:25.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1441203711}) 2024-05-15T13:05:25.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4121392414}) 2024-05-15T13:05:25.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=638346149}) 2024-05-15T13:05:25.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1272015075}) 2024-05-15T13:05:25.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2517240245}) 2024-05-15T13:05:25.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1406105385}) 2024-05-15T13:05:25.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1440272964}) 2024-05-15T13:05:25.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214883712}) 2024-05-15T13:05:25.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2642697010}) 2024-05-15T13:05:25.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4189148566}) 2024-05-15T13:05:25.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2079120647}) 2024-05-15T13:05:25.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4042722344}) 2024-05-15T13:05:25.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3583411424}) 2024-05-15T13:05:25.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2532057694}) 2024-05-15T13:05:25.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2467254877}) 2024-05-15T13:05:25.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2498512077}) 2024-05-15T13:05:25.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=17,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=45} crc {logm=4143295141,mdsmap=3984425268,mgr=76093979,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=1185698319,mon_config_key=3542150020}) 2024-05-15T13:05:25.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2233563460}) 2024-05-15T13:05:25.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1431521387}) 2024-05-15T13:05:25.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=30,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=52} crc {mon_config_key=23789456,monmap=2435304556,osd_metadata=1212035326,osd_pg_creating=3316468889,osdmap=1792139593}) 2024-05-15T13:05:25.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=53} crc {osdmap=3774903397}) 2024-05-15T13:05:25.766 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:05:25.797 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:25.836 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:05:25.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:25.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:05:25.944 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:05:25.989 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:26.151 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:05:26.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:26.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:05:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:05:26.260 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:05:26.298 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:26.325 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:05:26.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:26.332 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:05:26.433 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:05:26.471 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:26.506 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:05:26.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:26.511 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:05:26.612 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:05:26.656 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:26 smithi027 ceph-mon[39795]: pgmap v1229: 17 pgs: 17 active+clean; 6.4 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 896 op/s 2024-05-15T13:05:26.661 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:26.677 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:05:26.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:26 smithi177 ceph-mon[44303]: pgmap v1229: 17 pgs: 17 active+clean; 6.4 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 896 op/s 2024-05-15T13:05:26.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:26 smithi120 ceph-mon[43961]: pgmap v1229: 17 pgs: 17 active+clean; 6.4 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 896 op/s 2024-05-15T13:05:26.800 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:05:26.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:26.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:05:26.914 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:05:26.952 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:26.977 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:05:26.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:26.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:05:27.084 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:05:27.108 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:05:27.109 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:05:27.131 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:27.179 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:05:27.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:27.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:05:27.287 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:05:27.330 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:29.221 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:05:29.222 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:05:29.222 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-05-15T13:05:29.222 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-15T13:05:29.222 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-15T13:05:29.222 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-05-15T13:05:29.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:29 smithi177 ceph-mon[44303]: pgmap v1230: 17 pgs: 17 active+clean; 6.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 916 op/s 2024-05-15T13:05:29.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:29 smithi027 ceph-mon[39795]: pgmap v1230: 17 pgs: 17 active+clean; 6.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 916 op/s 2024-05-15T13:05:29.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:29 smithi120 ceph-mon[43961]: pgmap v1230: 17 pgs: 17 active+clean; 6.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 916 op/s 2024-05-15T13:05:29.602 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:05:29.646 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:05:29.646 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-05-15T13:05:30.114 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:05:30.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:30.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:05:30.212 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:30 smithi120 ceph-mon[43961]: 5.a scrub starts 2024-05-15T13:05:30.212 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:30 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:05:30.212 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:30 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:05:30.212 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:30 smithi120 ceph-mon[43961]: osdmap e457: 12 total, 12 up, 11 in 2024-05-15T13:05:30.225 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:05:30.262 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:30.374 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:30 smithi027 ceph-mon[39795]: 5.a scrub starts 2024-05-15T13:05:30.374 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:30 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:05:30.374 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:30 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:05:30.375 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:30 smithi027 ceph-mon[39795]: osdmap e457: 12 total, 12 up, 11 in 2024-05-15T13:05:30.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:30 smithi177 ceph-mon[44303]: 5.a scrub starts 2024-05-15T13:05:30.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:30 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:05:30.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:30 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:05:30.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:30 smithi177 ceph-mon[44303]: osdmap e457: 12 total, 12 up, 11 in 2024-05-15T13:05:30.531 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:05:30.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:30.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:05:30.636 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:05:30.688 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:31.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:31 smithi177 ceph-mon[44303]: 5.a scrub ok 2024-05-15T13:05:31.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:31 smithi177 ceph-mon[44303]: pgmap v1232: 17 pgs: 17 active+clean; 6.1 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 993 op/s 2024-05-15T13:05:31.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:31 smithi027 ceph-mon[39795]: 5.a scrub ok 2024-05-15T13:05:31.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:31 smithi027 ceph-mon[39795]: pgmap v1232: 17 pgs: 17 active+clean; 6.1 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 993 op/s 2024-05-15T13:05:31.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:31 smithi120 ceph-mon[43961]: 5.a scrub ok 2024-05-15T13:05:31.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:31 smithi120 ceph-mon[43961]: pgmap v1232: 17 pgs: 17 active+clean; 6.1 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 993 op/s 2024-05-15T13:05:31.747 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:05:31.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:31.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:05:31.856 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:05:31.889 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:05:31.889 INFO:teuthology.orchestra.run.smithi120.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-05-15T13:05:31.889 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:05:31.913 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:31.945 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:05:31.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:31.950 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:05:32.051 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:05:32.092 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:32.123 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:05:32.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:32.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:05:32.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:32 smithi177 ceph-mon[44303]: 5.4 scrub starts 2024-05-15T13:05:32.232 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:05:32.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:32 smithi120 ceph-mon[43961]: 5.4 scrub starts 2024-05-15T13:05:32.287 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:32 smithi027 ceph-mon[39795]: 5.4 scrub starts 2024-05-15T13:05:32.291 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:32.324 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:05:32.471 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:05:32.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:32.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:05:32.579 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:05:32.633 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:32.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:32.669 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:05:32.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:05:32.762 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:05:32.770 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:05:32.804 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:32.814 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:05:32.840 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:05:32.875 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:05:32.896 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:05:32.918 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:05:32.936 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:05:32.962 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:05:32.988 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:05:33.020 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:05:33.051 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:05:33.095 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:05:33.115 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:05:33.115 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:05:33.115 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:05:33.116 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:05:33.116 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:05:33.116 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:05:33.116 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:05:33.116 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:05:33.116 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:05:33.116 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:05:33.116 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:05:33.116 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:05:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:05:33.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:33 smithi027 ceph-mon[39795]: pgmap v1233: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.0 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 917 op/s 2024-05-15T13:05:33.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:33 smithi027 ceph-mon[39795]: 5.4 scrub ok 2024-05-15T13:05:33.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:33 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/56060033' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:05:33.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:33 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1245549888' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:05:33.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:33 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:05:33.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:33 smithi177 ceph-mon[44303]: pgmap v1233: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.0 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 917 op/s 2024-05-15T13:05:33.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:33 smithi177 ceph-mon[44303]: 5.4 scrub ok 2024-05-15T13:05:33.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:33 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/56060033' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:05:33.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:33 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1245549888' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:05:33.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:33 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:05:33.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:33 smithi120 ceph-mon[43961]: pgmap v1233: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.0 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 917 op/s 2024-05-15T13:05:33.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:33 smithi120 ceph-mon[43961]: 5.4 scrub ok 2024-05-15T13:05:33.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:33 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/56060033' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:05:33.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:33 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1245549888' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:05:33.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:33 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:05:34.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:34 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:05:34.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:34 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:05:34.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:34 smithi177 ceph-mon[44303]: osdmap e458: 12 total, 12 up, 11 in 2024-05-15T13:05:34.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:34 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:05:34.448 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:34 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:05:34.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:34 smithi027 ceph-mon[39795]: osdmap e458: 12 total, 12 up, 11 in 2024-05-15T13:05:34.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:34 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:05:34.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:34 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:05:34.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:34 smithi120 ceph-mon[43961]: osdmap e458: 12 total, 12 up, 11 in 2024-05-15T13:05:35.338 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:05:35.339 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T13:05:35.339 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T13:05:35.339 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:05:35.339 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.339 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:101362 4.f 4:fe841796:::benchmark_data_smithi177_99756_object101361:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-05-15T13:05:35.339 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:59:10.303295+0000", 2024-05-15T13:05:35.339 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 385.04300204200001, 2024-05-15T13:05:35.339 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5258583200000002, 2024-05-15T13:05:35.339 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:35.339 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:05:35.339 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:35.339 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:05:35.339 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:05:35.340 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 101362 2024-05-15T13:05:35.340 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.340 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:35.340 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.340 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:35.340 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.303295+0000", 2024-05-15T13:05:35.340 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.340 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.340 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.340 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:35.340 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.303295+0000", 2024-05-15T13:05:35.341 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.341 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.341 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.341 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:35.341 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.303296+0000", 2024-05-15T13:05:35.341 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5700000000000002e-07 2024-05-15T13:05:35.341 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.341 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.341 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:35.341 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.303322+0000", 2024-05-15T13:05:35.341 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6444e-05 2024-05-15T13:05:35.341 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.341 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.341 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:35.342 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.303355+0000", 2024-05-15T13:05:35.342 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2477000000000001e-05 2024-05-15T13:05:35.342 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.342 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.342 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:35.342 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.303361+0000", 2024-05-15T13:05:35.342 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5169999999999996e-06 2024-05-15T13:05:35.342 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.342 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.342 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.342 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.306874+0000", 2024-05-15T13:05:35.342 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035131110000000002 2024-05-15T13:05:35.343 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.343 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.343 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:35.343 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.307032+0000", 2024-05-15T13:05:35.343 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000157598 2024-05-15T13:05:35.343 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.343 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.343 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T13:05:35.343 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.307211+0000", 2024-05-15T13:05:35.343 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017897200000000001 2024-05-15T13:05:35.343 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.343 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.343 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.343 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.322151+0000", 2024-05-15T13:05:35.343 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014939737 2024-05-15T13:05:35.344 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.344 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.344 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:05:35.344 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.322160+0000", 2024-05-15T13:05:35.344 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7510000000000003e-06 2024-05-15T13:05:35.344 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.344 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.344 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.344 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:11.313271+0000", 2024-05-15T13:05:35.344 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99111032700000001 2024-05-15T13:05:35.344 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.344 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.344 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:05:35.345 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:11.313284+0000", 2024-05-15T13:05:35.345 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3580999999999999e-05 2024-05-15T13:05:35.345 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.345 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.345 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.345 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:12.829085+0000", 2024-05-15T13:05:35.345 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5158001489999999 2024-05-15T13:05:35.345 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.345 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.345 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:35.345 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:12.829154+0000", 2024-05-15T13:05:35.345 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9098999999999997e-05 2024-05-15T13:05:35.345 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.345 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:35.345 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.346 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.346 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.346 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:101364 4.f 4:fa04d9c8:::benchmark_data_smithi177_99756_object101363:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-05-15T13:05:35.346 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:59:10.303739+0000", 2024-05-15T13:05:35.346 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 385.04255846299998, 2024-05-15T13:05:35.346 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5255353270000001, 2024-05-15T13:05:35.346 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:35.346 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:05:35.346 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:35.346 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:05:35.346 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:05:35.346 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 101364 2024-05-15T13:05:35.347 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.347 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:35.347 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.347 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:35.347 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.303739+0000", 2024-05-15T13:05:35.347 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.347 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.347 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.347 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:35.347 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.303739+0000", 2024-05-15T13:05:35.347 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.347 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.347 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.347 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:35.347 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.303740+0000", 2024-05-15T13:05:35.348 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.23e-07 2024-05-15T13:05:35.348 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.348 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.348 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:35.348 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.303761+0000", 2024-05-15T13:05:35.348 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0962999999999999e-05 2024-05-15T13:05:35.348 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.348 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.348 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:35.348 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.303769+0000", 2024-05-15T13:05:35.348 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0870000000000007e-06 2024-05-15T13:05:35.348 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.348 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.348 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:35.349 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.303777+0000", 2024-05-15T13:05:35.349 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0749999999999998e-06 2024-05-15T13:05:35.349 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.349 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.349 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.349 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.310515+0000", 2024-05-15T13:05:35.349 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0067384200000000002 2024-05-15T13:05:35.349 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.349 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.349 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:35.349 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.310680+0000", 2024-05-15T13:05:35.349 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000165057 2024-05-15T13:05:35.349 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.349 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.350 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,9", 2024-05-15T13:05:35.350 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.310842+0000", 2024-05-15T13:05:35.350 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016158700000000001 2024-05-15T13:05:35.350 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.350 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.350 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.350 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.322192+0000", 2024-05-15T13:05:35.350 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011350179 2024-05-15T13:05:35.350 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.350 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.350 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:05:35.350 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:10.322200+0000", 2024-05-15T13:05:35.350 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.276e-06 2024-05-15T13:05:35.350 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.350 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.351 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.351 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:11.313329+0000", 2024-05-15T13:05:35.351 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99112866399999999 2024-05-15T13:05:35.351 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.351 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.351 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:05:35.351 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:11.313338+0000", 2024-05-15T13:05:35.351 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.55e-06 2024-05-15T13:05:35.351 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.351 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.351 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.351 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:12.829209+0000", 2024-05-15T13:05:35.351 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5158704589999998 2024-05-15T13:05:35.352 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.352 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.352 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:35.352 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:59:12.829274+0000", 2024-05-15T13:05:35.352 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5287e-05 2024-05-15T13:05:35.352 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.352 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:35.352 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.352 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.352 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.352 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:140795 4.9 4:9e5e757e:::benchmark_data_smithi177_99756_object20823:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e357)", 2024-05-15T13:05:35.352 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:11.337401+0000", 2024-05-15T13:05:35.352 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 324.00889628099998, 2024-05-15T13:05:35.353 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.256904681, 2024-05-15T13:05:35.353 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:35.353 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:05:35.353 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:35.353 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:05:35.353 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:05:35.353 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 140795 2024-05-15T13:05:35.353 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.353 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:35.353 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.353 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:35.353 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:11.337401+0000", 2024-05-15T13:05:35.353 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.353 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.353 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.354 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:35.354 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:11.337401+0000", 2024-05-15T13:05:35.354 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.354 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.354 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.354 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:35.354 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:11.337402+0000", 2024-05-15T13:05:35.354 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0800000000000005e-07 2024-05-15T13:05:35.354 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.354 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.354 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:35.354 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:11.337404+0000", 2024-05-15T13:05:35.354 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.323e-06 2024-05-15T13:05:35.354 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.354 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.355 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:35.355 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:11.337410+0000", 2024-05-15T13:05:35.355 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1099999999999999e-06 2024-05-15T13:05:35.355 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.355 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.355 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:35.355 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:11.337415+0000", 2024-05-15T13:05:35.355 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.904e-06 2024-05-15T13:05:35.355 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.355 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.355 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.355 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:11.337653+0000", 2024-05-15T13:05:35.356 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00023781900000000001 2024-05-15T13:05:35.356 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.356 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.356 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:05:35.356 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:11.337729+0000", 2024-05-15T13:05:35.356 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6136000000000004e-05 2024-05-15T13:05:35.356 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.356 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.356 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.356 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:14.594289+0000", 2024-05-15T13:05:35.356 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2565602500000002 2024-05-15T13:05:35.356 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.356 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.356 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:35.357 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:14.594306+0000", 2024-05-15T13:05:35.357 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6631000000000001e-05 2024-05-15T13:05:35.357 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.357 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:35.357 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.357 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.357 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.357 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:144471 4.9 4:980f9544:::benchmark_data_smithi177_99756_object24499:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e358)", 2024-05-15T13:05:35.357 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:17.176806+0000", 2024-05-15T13:05:35.357 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 318.169491532, 2024-05-15T13:05:35.357 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.56759795199999996, 2024-05-15T13:05:35.357 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:35.357 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:05:35.358 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:35.358 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:05:35.358 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:05:35.358 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 144471 2024-05-15T13:05:35.358 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.358 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:35.358 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.358 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:35.358 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.176806+0000", 2024-05-15T13:05:35.358 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.358 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.358 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.358 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:35.359 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.176806+0000", 2024-05-15T13:05:35.359 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.359 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.359 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.359 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:35.359 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.176807+0000", 2024-05-15T13:05:35.359 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1999999999999999e-06 2024-05-15T13:05:35.359 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.359 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.359 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:35.359 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.176812+0000", 2024-05-15T13:05:35.359 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8919999999999999e-06 2024-05-15T13:05:35.359 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.359 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.360 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:35.360 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.176820+0000", 2024-05-15T13:05:35.360 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5720000000000003e-06 2024-05-15T13:05:35.360 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.360 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.360 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:35.360 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.176827+0000", 2024-05-15T13:05:35.360 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.832e-06 2024-05-15T13:05:35.360 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.360 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.360 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.360 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.176947+0000", 2024-05-15T13:05:35.360 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000119704 2024-05-15T13:05:35.360 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.360 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.361 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:05:35.361 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.177031+0000", 2024-05-15T13:05:35.361 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3758999999999995e-05 2024-05-15T13:05:35.361 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.361 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.361 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.361 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.744391+0000", 2024-05-15T13:05:35.361 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.56736049499999996 2024-05-15T13:05:35.361 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.361 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.361 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:35.361 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:17.744404+0000", 2024-05-15T13:05:35.361 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2498e-05 2024-05-15T13:05:35.361 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.361 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:35.361 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.362 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.362 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.362 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:145134 4.9 4:9e0839ad:::benchmark_data_smithi177_99756_object25162:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e359)", 2024-05-15T13:05:35.362 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:18.574293+0000", 2024-05-15T13:05:35.362 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 316.77200385399999, 2024-05-15T13:05:35.362 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9424836380000001, 2024-05-15T13:05:35.362 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:35.362 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:05:35.362 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:35.362 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:05:35.362 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:05:35.362 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 145134 2024-05-15T13:05:35.362 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.362 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:35.363 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.363 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:35.363 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574293+0000", 2024-05-15T13:05:35.363 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.363 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.363 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.363 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:35.363 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574293+0000", 2024-05-15T13:05:35.363 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.363 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.363 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.363 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:35.363 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574295+0000", 2024-05-15T13:05:35.363 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7799999999999999e-06 2024-05-15T13:05:35.364 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.364 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.364 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:35.364 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574301+0000", 2024-05-15T13:05:35.364 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8610000000000003e-06 2024-05-15T13:05:35.364 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.364 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.364 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:35.364 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574312+0000", 2024-05-15T13:05:35.364 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0468999999999999e-05 2024-05-15T13:05:35.364 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.364 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.364 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:35.364 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574320+0000", 2024-05-15T13:05:35.364 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6230000000000003e-06 2024-05-15T13:05:35.364 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.365 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.365 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.365 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574539+0000", 2024-05-15T13:05:35.365 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00021846100000000001 2024-05-15T13:05:35.365 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.365 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.365 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:05:35.365 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574622+0000", 2024-05-15T13:05:35.365 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3628000000000006e-05 2024-05-15T13:05:35.365 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.365 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.365 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.365 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.516765+0000", 2024-05-15T13:05:35.366 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9421424389999999 2024-05-15T13:05:35.366 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.366 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.366 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:35.366 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.516777+0000", 2024-05-15T13:05:35.366 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2377e-05 2024-05-15T13:05:35.366 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.366 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:35.366 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.366 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.366 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.366 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:145137 4.9 4:909fd2a8:::benchmark_data_smithi177_99756_object25165:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e359)", 2024-05-15T13:05:35.367 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:18.574342+0000", 2024-05-15T13:05:35.367 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 316.77195502199999, 2024-05-15T13:05:35.367 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.951285672, 2024-05-15T13:05:35.367 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:35.367 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:05:35.367 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:35.367 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:05:35.368 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:05:35.368 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 145137 2024-05-15T13:05:35.368 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.368 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:35.368 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.368 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:35.368 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574342+0000", 2024-05-15T13:05:35.368 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.368 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.368 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.368 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:35.368 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574342+0000", 2024-05-15T13:05:35.368 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.369 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.369 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.369 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:35.369 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574343+0000", 2024-05-15T13:05:35.369 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4900000000000001e-07 2024-05-15T13:05:35.369 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.369 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.369 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:35.369 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574346+0000", 2024-05-15T13:05:35.369 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7980000000000001e-06 2024-05-15T13:05:35.369 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.369 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.369 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:35.369 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574352+0000", 2024-05-15T13:05:35.369 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.601e-06 2024-05-15T13:05:35.370 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.370 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.370 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:35.370 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574357+0000", 2024-05-15T13:05:35.370 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2120000000000002e-06 2024-05-15T13:05:35.370 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.370 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.370 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.370 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574652+0000", 2024-05-15T13:05:35.370 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00029439100000000002 2024-05-15T13:05:35.370 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.370 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.371 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:05:35.371 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:18.574724+0000", 2024-05-15T13:05:35.371 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2488000000000001e-05 2024-05-15T13:05:35.371 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.371 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.371 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.371 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.525612+0000", 2024-05-15T13:05:35.371 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.950888199 2024-05-15T13:05:35.371 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.371 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.371 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:35.371 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.525628+0000", 2024-05-15T13:05:35.371 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5534000000000001e-05 2024-05-15T13:05:35.371 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.372 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:35.372 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.372 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.372 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.372 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:146277 4.9 4:9e2047c8:::benchmark_data_smithi177_99756_object26305:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e359)", 2024-05-15T13:05:35.372 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:20.495780+0000", 2024-05-15T13:05:35.372 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 314.85051707700001, 2024-05-15T13:05:35.372 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5112627170000001, 2024-05-15T13:05:35.372 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:35.372 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:05:35.372 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:35.372 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:05:35.372 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:05:35.373 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 146277 2024-05-15T13:05:35.373 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.373 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:35.373 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.373 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:35.373 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.495780+0000", 2024-05-15T13:05:35.373 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.373 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.373 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.373 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:35.373 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.495780+0000", 2024-05-15T13:05:35.373 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.373 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.373 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.373 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:35.374 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.495780+0000", 2024-05-15T13:05:35.374 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1400000000000001e-07 2024-05-15T13:05:35.374 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.374 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.374 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:35.374 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.495782+0000", 2024-05-15T13:05:35.374 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4219999999999999e-06 2024-05-15T13:05:35.374 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.374 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.374 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:35.374 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.495785+0000", 2024-05-15T13:05:35.374 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1690000000000001e-06 2024-05-15T13:05:35.374 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.375 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.375 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:35.375 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.495788+0000", 2024-05-15T13:05:35.375 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.723e-06 2024-05-15T13:05:35.375 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.375 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.375 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.375 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.496052+0000", 2024-05-15T13:05:35.375 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00026399799999999999 2024-05-15T13:05:35.375 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.375 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.375 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:05:35.375 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.496136+0000", 2024-05-15T13:05:35.375 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4623000000000006e-05 2024-05-15T13:05:35.377 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.377 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.377 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.377 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:23.007030+0000", 2024-05-15T13:05:35.377 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.510893636 2024-05-15T13:05:35.377 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.377 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.377 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:35.377 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:23.007043+0000", 2024-05-15T13:05:35.377 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2931999999999999e-05 2024-05-15T13:05:35.378 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.378 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:35.378 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.378 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.378 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.378 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:166038 4.9 4:9c1e7859:::benchmark_data_smithi177_99756_object46066:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e369)", 2024-05-15T13:05:35.378 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:52.276137+0000", 2024-05-15T13:05:35.378 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 283.07016061899998, 2024-05-15T13:05:35.378 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8569116879999998, 2024-05-15T13:05:35.378 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:35.378 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:05:35.378 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:35.379 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:05:35.379 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:05:35.379 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 166038 2024-05-15T13:05:35.379 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.379 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:35.379 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.379 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:35.379 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:52.276137+0000", 2024-05-15T13:05:35.379 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.379 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.379 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.379 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:35.379 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:52.276137+0000", 2024-05-15T13:05:35.379 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.379 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.380 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.380 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:35.380 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:52.276137+0000", 2024-05-15T13:05:35.380 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5600000000000001e-07 2024-05-15T13:05:35.380 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.380 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.380 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:35.380 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:52.276140+0000", 2024-05-15T13:05:35.380 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.134e-06 2024-05-15T13:05:35.380 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.380 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.380 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:35.380 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:52.276147+0000", 2024-05-15T13:05:35.380 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0670000000000003e-06 2024-05-15T13:05:35.380 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.380 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.381 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:35.381 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:52.276154+0000", 2024-05-15T13:05:35.381 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3990000000000001e-06 2024-05-15T13:05:35.381 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.381 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.381 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.381 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:52.279462+0000", 2024-05-15T13:05:35.381 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033081759999999999 2024-05-15T13:05:35.381 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.381 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.381 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:05:35.381 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:52.279526+0000", 2024-05-15T13:05:35.381 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4324000000000003e-05 2024-05-15T13:05:35.381 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.381 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.382 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.382 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:55.133020+0000", 2024-05-15T13:05:35.382 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8534936769999999 2024-05-15T13:05:35.382 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.382 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.382 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:35.382 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:55.133048+0000", 2024-05-15T13:05:35.382 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8455000000000001e-05 2024-05-15T13:05:35.382 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.382 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:35.382 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.382 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.382 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.382 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:167732 4.9 4:9df1642a:::benchmark_data_smithi177_99756_object47760:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e371)", 2024-05-15T13:05:35.383 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:55.093701+0000", 2024-05-15T13:05:35.383 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 280.252596117, 2024-05-15T13:05:35.383 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8566292610000001, 2024-05-15T13:05:35.383 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:35.383 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:05:35.383 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:35.383 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:05:35.383 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:05:35.383 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 167732 2024-05-15T13:05:35.383 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.383 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:35.383 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.383 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:35.383 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:55.093701+0000", 2024-05-15T13:05:35.383 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.383 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.384 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.384 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:35.384 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:55.093701+0000", 2024-05-15T13:05:35.384 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.384 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.384 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.384 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:35.384 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:55.093702+0000", 2024-05-15T13:05:35.384 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7800000000000002e-07 2024-05-15T13:05:35.384 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.384 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.384 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:35.384 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:55.093704+0000", 2024-05-15T13:05:35.384 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3140000000000002e-06 2024-05-15T13:05:35.384 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.385 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.385 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:35.385 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:55.093710+0000", 2024-05-15T13:05:35.385 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8669999999999999e-06 2024-05-15T13:05:35.385 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.385 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.385 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:35.385 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:55.093715+0000", 2024-05-15T13:05:35.385 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9640000000000002e-06 2024-05-15T13:05:35.385 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.385 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.385 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.385 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:55.093836+0000", 2024-05-15T13:05:35.386 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000121204 2024-05-15T13:05:35.386 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.386 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.386 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:05:35.386 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:55.093922+0000", 2024-05-15T13:05:35.386 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6473999999999999e-05 2024-05-15T13:05:35.386 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.386 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.386 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.386 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:56.950316+0000", 2024-05-15T13:05:35.386 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8563931060000001 2024-05-15T13:05:35.386 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.386 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.387 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:35.387 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:56.950330+0000", 2024-05-15T13:05:35.387 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4953999999999999e-05 2024-05-15T13:05:35.387 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.387 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:35.387 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.387 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.387 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.387 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:168914 4.9 4:975d04d4:::benchmark_data_smithi177_99756_object48942:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e371)", 2024-05-15T13:05:35.387 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:56.929264+0000", 2024-05-15T13:05:35.387 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 278.41703348200002, 2024-05-15T13:05:35.387 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.220342311, 2024-05-15T13:05:35.387 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:35.387 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:05:35.388 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:35.388 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:05:35.388 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:05:35.388 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 168914 2024-05-15T13:05:35.388 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.388 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:35.388 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.388 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:35.388 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:56.929264+0000", 2024-05-15T13:05:35.388 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.388 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.388 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.388 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:35.388 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:56.929264+0000", 2024-05-15T13:05:35.389 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.389 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.389 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.389 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:35.389 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:56.929264+0000", 2024-05-15T13:05:35.389 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5900000000000003e-07 2024-05-15T13:05:35.389 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.389 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.389 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:35.389 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:56.929266+0000", 2024-05-15T13:05:35.389 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9489999999999999e-06 2024-05-15T13:05:35.389 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.389 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.389 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:35.389 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:56.929270+0000", 2024-05-15T13:05:35.390 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2629999999999997e-06 2024-05-15T13:05:35.390 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.390 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.390 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:35.390 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:56.929274+0000", 2024-05-15T13:05:35.390 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3900000000000002e-06 2024-05-15T13:05:35.390 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.390 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.390 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.390 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:56.929488+0000", 2024-05-15T13:05:35.390 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00021410600000000001 2024-05-15T13:05:35.390 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.390 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.391 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:05:35.391 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:56.929539+0000", 2024-05-15T13:05:35.391 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1205999999999997e-05 2024-05-15T13:05:35.391 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.391 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.391 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.391 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:00.149591+0000", 2024-05-15T13:05:35.391 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2200518680000001 2024-05-15T13:05:35.391 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.391 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.391 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:35.391 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:00.149606+0000", 2024-05-15T13:05:35.391 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.517e-05 2024-05-15T13:05:35.391 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.392 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:35.392 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.392 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.392 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.392 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:172167 4.9 4:9c3933af:::benchmark_data_smithi177_99756_object52195:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-05-15T13:05:35.392 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:02.164597+0000", 2024-05-15T13:05:35.392 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 273.18170006600002, 2024-05-15T13:05:35.392 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.24428116699999999, 2024-05-15T13:05:35.392 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:35.392 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:05:35.392 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:35.392 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:05:35.392 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:05:35.392 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 172167 2024-05-15T13:05:35.393 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.393 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:35.393 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.393 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:35.393 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164597+0000", 2024-05-15T13:05:35.393 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.393 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.393 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.393 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:35.393 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164597+0000", 2024-05-15T13:05:35.393 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.393 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.393 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.393 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:35.393 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164598+0000", 2024-05-15T13:05:35.394 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1e-07 2024-05-15T13:05:35.394 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.394 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.394 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:35.394 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164599+0000", 2024-05-15T13:05:35.394 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6500000000000001e-06 2024-05-15T13:05:35.394 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.394 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.394 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:35.394 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164603+0000", 2024-05-15T13:05:35.394 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9369999999999997e-06 2024-05-15T13:05:35.394 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.394 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.394 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:35.395 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164606+0000", 2024-05-15T13:05:35.395 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1140000000000002e-06 2024-05-15T13:05:35.395 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.395 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.395 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.395 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.168923+0000", 2024-05-15T13:05:35.395 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0043163259999999997 2024-05-15T13:05:35.395 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.395 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.395 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:05:35.395 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.169000+0000", 2024-05-15T13:05:35.395 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6975000000000001e-05 2024-05-15T13:05:35.395 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.395 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.396 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.396 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.408864+0000", 2024-05-15T13:05:35.396 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.23986439100000001 2024-05-15T13:05:35.396 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.396 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.396 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:35.396 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.408878+0000", 2024-05-15T13:05:35.396 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4464000000000001e-05 2024-05-15T13:05:35.396 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.396 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:35.396 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.396 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.396 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.397 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:172180 4.9 4:9a38ca0a:::benchmark_data_smithi177_99756_object52208:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-05-15T13:05:35.397 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:02.164610+0000", 2024-05-15T13:05:35.397 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 273.18168690499999, 2024-05-15T13:05:35.397 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.25200767200000002, 2024-05-15T13:05:35.397 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:35.397 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:05:35.397 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:35.397 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:05:35.397 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:05:35.397 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 172180 2024-05-15T13:05:35.397 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.397 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:35.398 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.398 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:35.398 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164610+0000", 2024-05-15T13:05:35.398 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.398 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.398 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.398 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:35.398 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164610+0000", 2024-05-15T13:05:35.398 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.398 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.398 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.398 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:35.398 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164611+0000", 2024-05-15T13:05:35.399 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2399999999999999e-07 2024-05-15T13:05:35.399 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.399 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.399 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:35.399 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164612+0000", 2024-05-15T13:05:35.399 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4300000000000001e-06 2024-05-15T13:05:35.399 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.399 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.399 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:35.399 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164615+0000", 2024-05-15T13:05:35.399 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3840000000000001e-06 2024-05-15T13:05:35.399 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.399 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.399 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:35.400 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164618+0000", 2024-05-15T13:05:35.400 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8609999999999998e-06 2024-05-15T13:05:35.400 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.400 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.400 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.400 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.169035+0000", 2024-05-15T13:05:35.400 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0044169819999999999 2024-05-15T13:05:35.400 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.400 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.400 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:05:35.400 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.169125+0000", 2024-05-15T13:05:35.400 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0130000000000006e-05 2024-05-15T13:05:35.400 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.400 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.401 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.401 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.416606+0000", 2024-05-15T13:05:35.401 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.247480643 2024-05-15T13:05:35.401 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.401 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.401 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:35.401 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.416618+0000", 2024-05-15T13:05:35.401 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2018000000000001e-05 2024-05-15T13:05:35.401 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.401 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:35.401 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.401 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.401 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.401 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:172183 4.9 4:90753368:::benchmark_data_smithi177_99756_object52211:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-05-15T13:05:35.401 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:02.164623+0000", 2024-05-15T13:05:35.402 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 273.18167469100001, 2024-05-15T13:05:35.402 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.25915582700000001, 2024-05-15T13:05:35.402 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:35.402 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:05:35.402 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:35.402 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:05:35.402 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:05:35.402 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 172183 2024-05-15T13:05:35.402 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.402 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:35.402 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.402 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:35.402 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164623+0000", 2024-05-15T13:05:35.402 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.403 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.403 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.403 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:35.403 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164623+0000", 2024-05-15T13:05:35.403 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.403 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.403 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.403 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:35.403 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164623+0000", 2024-05-15T13:05:35.403 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2000000000000001e-07 2024-05-15T13:05:35.403 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.403 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.403 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:35.403 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164625+0000", 2024-05-15T13:05:35.403 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1280000000000002e-06 2024-05-15T13:05:35.404 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.404 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.404 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:35.404 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164628+0000", 2024-05-15T13:05:35.404 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3900000000000002e-06 2024-05-15T13:05:35.404 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.404 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.404 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:35.404 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.164631+0000", 2024-05-15T13:05:35.404 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8250000000000001e-06 2024-05-15T13:05:35.404 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.404 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.404 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.404 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.169155+0000", 2024-05-15T13:05:35.404 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0045237109999999997 2024-05-15T13:05:35.404 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.405 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.405 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:05:35.405 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.169230+0000", 2024-05-15T13:05:35.405 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5099999999999996e-05 2024-05-15T13:05:35.405 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.405 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.405 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.405 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.423763+0000", 2024-05-15T13:05:35.405 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.25453252399999998 2024-05-15T13:05:35.405 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.405 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.405 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:35.405 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:02.423778+0000", 2024-05-15T13:05:35.406 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5928999999999999e-05 2024-05-15T13:05:35.406 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.406 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:35.406 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.406 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.406 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.406 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:175340 4.9 4:9f0a7fac:::benchmark_data_smithi177_99756_object55368:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e374)", 2024-05-15T13:05:35.406 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:06.982258+0000", 2024-05-15T13:05:35.406 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 268.36403922599999, 2024-05-15T13:05:35.406 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8210182489999998, 2024-05-15T13:05:35.406 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:35.406 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:05:35.406 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:35.407 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:05:35.407 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:05:35.407 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 175340 2024-05-15T13:05:35.407 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.407 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:35.407 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.407 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:35.407 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:06.982258+0000", 2024-05-15T13:05:35.407 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.407 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.407 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.407 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:35.407 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:06.982258+0000", 2024-05-15T13:05:35.408 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.408 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.408 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.408 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:35.408 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:06.982259+0000", 2024-05-15T13:05:35.408 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.89e-07 2024-05-15T13:05:35.408 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.408 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.408 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:35.408 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:06.982261+0000", 2024-05-15T13:05:35.408 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6970000000000002e-06 2024-05-15T13:05:35.408 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.408 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.408 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:35.408 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:06.982267+0000", 2024-05-15T13:05:35.409 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8919999999999997e-06 2024-05-15T13:05:35.409 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.409 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.409 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:35.409 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:06.982272+0000", 2024-05-15T13:05:35.409 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0359999999999997e-06 2024-05-15T13:05:35.409 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.409 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.409 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.409 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:06.985640+0000", 2024-05-15T13:05:35.409 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033680469999999999 2024-05-15T13:05:35.409 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.410 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.410 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:05:35.410 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:06.985687+0000", 2024-05-15T13:05:35.410 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7246000000000001e-05 2024-05-15T13:05:35.410 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.410 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.410 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.410 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:10.803260+0000", 2024-05-15T13:05:35.410 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8175724669999997 2024-05-15T13:05:35.410 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.410 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.410 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:35.410 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:10.803276+0000", 2024-05-15T13:05:35.411 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6475000000000001e-05 2024-05-15T13:05:35.411 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.411 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:35.411 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.411 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.411 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.411 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:214339 4.9 4:92a8a31e:::benchmark_data_smithi177_99756_object94367:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e391)", 2024-05-15T13:05:35.411 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:02:07.330795+0000", 2024-05-15T13:05:35.411 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 208.01550235900001, 2024-05-15T13:05:35.411 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34146168799999999, 2024-05-15T13:05:35.411 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:35.411 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:05:35.411 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:35.411 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:05:35.412 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:05:35.412 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 214339 2024-05-15T13:05:35.412 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.412 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:35.412 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.412 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:35.412 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.330795+0000", 2024-05-15T13:05:35.412 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.412 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.412 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.412 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:35.412 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.330795+0000", 2024-05-15T13:05:35.412 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.412 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.412 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.413 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:35.413 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.330795+0000", 2024-05-15T13:05:35.413 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6400000000000003e-07 2024-05-15T13:05:35.413 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.413 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.413 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:35.413 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.330798+0000", 2024-05-15T13:05:35.413 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.667e-06 2024-05-15T13:05:35.413 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.413 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.413 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:35.413 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.330806+0000", 2024-05-15T13:05:35.413 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4379999999999998e-06 2024-05-15T13:05:35.413 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.413 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.414 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:35.414 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.330811+0000", 2024-05-15T13:05:35.414 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1889999999999998e-06 2024-05-15T13:05:35.414 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.414 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.414 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.414 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.332626+0000", 2024-05-15T13:05:35.414 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001814757 2024-05-15T13:05:35.414 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.414 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.414 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:05:35.414 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.332710+0000", 2024-05-15T13:05:35.414 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4862000000000002e-05 2024-05-15T13:05:35.414 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.414 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.415 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.415 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.672247+0000", 2024-05-15T13:05:35.415 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.33953666599999999 2024-05-15T13:05:35.415 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.415 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.415 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:35.415 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.672257+0000", 2024-05-15T13:05:35.415 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.645e-06 2024-05-15T13:05:35.415 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.415 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:35.415 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.415 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.415 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.415 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:214340 4.9 4:9536964c:::benchmark_data_smithi177_99756_object94368:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e391)", 2024-05-15T13:05:35.416 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:02:07.330817+0000", 2024-05-15T13:05:35.416 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 208.01548010100001, 2024-05-15T13:05:35.416 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34488550200000001, 2024-05-15T13:05:35.416 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:35.416 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:05:35.416 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:35.416 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:05:35.416 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:05:35.416 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 214340 2024-05-15T13:05:35.416 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.416 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:35.416 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.416 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:35.416 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.330817+0000", 2024-05-15T13:05:35.416 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.417 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.417 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.417 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:35.417 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.330817+0000", 2024-05-15T13:05:35.417 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.417 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.417 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.417 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:35.417 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.330818+0000", 2024-05-15T13:05:35.417 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4499999999999998e-07 2024-05-15T13:05:35.417 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.417 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.417 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:35.418 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.330820+0000", 2024-05-15T13:05:35.418 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.99e-06 2024-05-15T13:05:35.418 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.418 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.418 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:35.418 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.330826+0000", 2024-05-15T13:05:35.418 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6379999999999998e-06 2024-05-15T13:05:35.418 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.418 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.418 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:35.418 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.330831+0000", 2024-05-15T13:05:35.418 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7489999999999998e-06 2024-05-15T13:05:35.418 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.419 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.419 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.419 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.332742+0000", 2024-05-15T13:05:35.419 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001910548 2024-05-15T13:05:35.419 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.419 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.419 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:05:35.419 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.332822+0000", 2024-05-15T13:05:35.419 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0861999999999999e-05 2024-05-15T13:05:35.419 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.419 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.419 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.419 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.675685+0000", 2024-05-15T13:05:35.419 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34286307599999999 2024-05-15T13:05:35.419 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.420 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.420 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:35.420 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.675703+0000", 2024-05-15T13:05:35.420 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7294000000000001e-05 2024-05-15T13:05:35.420 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.420 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:35.420 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.420 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.420 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.420 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:214346 4.9 4:901c5a94:::benchmark_data_smithi177_99756_object94374:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e391)", 2024-05-15T13:05:35.420 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:02:07.330856+0000", 2024-05-15T13:05:35.420 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 208.01544159100001, 2024-05-15T13:05:35.421 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.347773679, 2024-05-15T13:05:35.421 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:35.421 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:05:35.421 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:35.421 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:05:35.421 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:05:35.421 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 214346 2024-05-15T13:05:35.421 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.421 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:35.421 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.421 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:35.421 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.330856+0000", 2024-05-15T13:05:35.421 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.421 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.421 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.422 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:35.422 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.330856+0000", 2024-05-15T13:05:35.422 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.422 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.422 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.422 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:35.422 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.330856+0000", 2024-05-15T13:05:35.422 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9299999999999999e-07 2024-05-15T13:05:35.422 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.422 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.422 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:35.422 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.330864+0000", 2024-05-15T13:05:35.422 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2779999999999993e-06 2024-05-15T13:05:35.423 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.423 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.423 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:35.423 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.330877+0000", 2024-05-15T13:05:35.423 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3030999999999999e-05 2024-05-15T13:05:35.423 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.423 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.423 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:35.423 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.330884+0000", 2024-05-15T13:05:35.423 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7179999999999999e-06 2024-05-15T13:05:35.423 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.423 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.423 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.423 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.332852+0000", 2024-05-15T13:05:35.423 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019681540000000002 2024-05-15T13:05:35.424 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.424 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.424 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:05:35.424 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.332934+0000", 2024-05-15T13:05:35.424 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1371000000000004e-05 2024-05-15T13:05:35.424 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.424 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.424 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.424 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.678619+0000", 2024-05-15T13:05:35.424 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34568542600000002 2024-05-15T13:05:35.424 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.424 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.424 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:35.424 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.678629+0000", 2024-05-15T13:05:35.424 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0407999999999999e-05 2024-05-15T13:05:35.425 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.425 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:35.425 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.425 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.425 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.425 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:214349 4.9 4:9f3e031f:::benchmark_data_smithi177_99756_object94377:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e391)", 2024-05-15T13:05:35.425 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:02:07.330891+0000", 2024-05-15T13:05:35.425 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 208.01540651600001, 2024-05-15T13:05:35.425 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35060067, 2024-05-15T13:05:35.425 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:35.426 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:05:35.426 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:35.426 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:05:35.426 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:05:35.426 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 214349 2024-05-15T13:05:35.426 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.426 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:35.426 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.426 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:35.426 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.330891+0000", 2024-05-15T13:05:35.426 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.426 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.426 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.427 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:35.427 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.330891+0000", 2024-05-15T13:05:35.427 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.427 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.427 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.427 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:35.427 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.330891+0000", 2024-05-15T13:05:35.427 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6399999999999998e-07 2024-05-15T13:05:35.427 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.427 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.427 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:35.427 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.330893+0000", 2024-05-15T13:05:35.427 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2050000000000001e-06 2024-05-15T13:05:35.427 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.428 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.428 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:35.428 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.330899+0000", 2024-05-15T13:05:35.428 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4229999999999996e-06 2024-05-15T13:05:35.428 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.428 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.428 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:35.428 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.330904+0000", 2024-05-15T13:05:35.428 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7809999999999996e-06 2024-05-15T13:05:35.428 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.428 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.428 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.428 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.332966+0000", 2024-05-15T13:05:35.428 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020626920000000001 2024-05-15T13:05:35.428 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.429 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.429 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:05:35.429 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.333048+0000", 2024-05-15T13:05:35.429 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2101000000000006e-05 2024-05-15T13:05:35.429 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.429 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.429 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.429 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.681479+0000", 2024-05-15T13:05:35.429 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34843012600000001 2024-05-15T13:05:35.429 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.429 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.429 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:35.429 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:07.681492+0000", 2024-05-15T13:05:35.429 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2978e-05 2024-05-15T13:05:35.430 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.430 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:35.430 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.430 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.430 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.430 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:220834 4.9 4:957c8fe1:::benchmark_data_smithi177_99756_object100862:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e396)", 2024-05-15T13:05:35.430 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:02:16.593726+0000", 2024-05-15T13:05:35.430 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 198.75257169, 2024-05-15T13:05:35.430 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.26265953800000003, 2024-05-15T13:05:35.430 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:35.430 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:05:35.430 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:35.430 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:05:35.430 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:05:35.430 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 220834 2024-05-15T13:05:35.431 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.431 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:35.431 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.431 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:35.431 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:16.593726+0000", 2024-05-15T13:05:35.431 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.431 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.431 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.431 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:35.431 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:16.593726+0000", 2024-05-15T13:05:35.431 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.431 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.431 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.431 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:35.431 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:16.593726+0000", 2024-05-15T13:05:35.432 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6400000000000002e-07 2024-05-15T13:05:35.432 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.432 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.432 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:35.432 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:16.593729+0000", 2024-05-15T13:05:35.432 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8140000000000001e-06 2024-05-15T13:05:35.432 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.432 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.432 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:35.432 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:16.593737+0000", 2024-05-15T13:05:35.432 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5800000000000003e-06 2024-05-15T13:05:35.432 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.432 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.432 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:35.433 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:16.593742+0000", 2024-05-15T13:05:35.433 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6439999999999997e-06 2024-05-15T13:05:35.433 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.433 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.433 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.433 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:16.595564+0000", 2024-05-15T13:05:35.433 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00182125 2024-05-15T13:05:35.433 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.433 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.433 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:05:35.433 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:16.595645+0000", 2024-05-15T13:05:35.433 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1203000000000002e-05 2024-05-15T13:05:35.433 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.433 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.433 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.434 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:16.856361+0000", 2024-05-15T13:05:35.434 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.26071675300000002 2024-05-15T13:05:35.434 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.434 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.434 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:35.434 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:16.856385+0000", 2024-05-15T13:05:35.434 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3730000000000001e-05 2024-05-15T13:05:35.434 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.434 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:35.434 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.434 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.434 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.434 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:220836 4.9 4:96004f92:::benchmark_data_smithi177_99756_object100864:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e396)", 2024-05-15T13:05:35.434 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:02:16.593749+0000", 2024-05-15T13:05:35.435 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 198.75254792000001, 2024-05-15T13:05:35.435 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.26472731199999999, 2024-05-15T13:05:35.435 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:35.435 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:05:35.435 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:35.435 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:05:35.435 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:05:35.435 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 220836 2024-05-15T13:05:35.435 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.435 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:35.435 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.435 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:35.435 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:16.593749+0000", 2024-05-15T13:05:35.435 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.435 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.435 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.436 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:35.436 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:16.593749+0000", 2024-05-15T13:05:35.436 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:35.436 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.436 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.436 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:35.436 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:16.593750+0000", 2024-05-15T13:05:35.436 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4299999999999999e-07 2024-05-15T13:05:35.436 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.436 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.436 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:35.436 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:16.593752+0000", 2024-05-15T13:05:35.436 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.131e-06 2024-05-15T13:05:35.436 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.436 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.437 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:35.437 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:16.593758+0000", 2024-05-15T13:05:35.437 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4860000000000002e-06 2024-05-15T13:05:35.437 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.437 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.437 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:35.437 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:16.593763+0000", 2024-05-15T13:05:35.437 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0610000000000003e-06 2024-05-15T13:05:35.437 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.437 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.437 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.437 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:16.595695+0000", 2024-05-15T13:05:35.437 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019315020000000001 2024-05-15T13:05:35.437 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.437 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.437 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:05:35.438 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:16.595785+0000", 2024-05-15T13:05:35.438 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0209000000000003e-05 2024-05-15T13:05:35.438 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.438 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.438 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:35.438 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:16.858466+0000", 2024-05-15T13:05:35.438 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.26268115399999997 2024-05-15T13:05:35.438 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:35.438 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:35.438 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:05:35.438 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:02:16.858477+0000", 2024-05-15T13:05:35.438 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0426e-05 2024-05-15T13:05:35.438 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.438 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:35.439 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.439 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:35.439 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:35.439 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:05:35.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:35 smithi177 ceph-mon[44303]: pgmap v1235: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 956 op/s 2024-05-15T13:05:35.440 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:35 smithi027 ceph-mon[39795]: pgmap v1235: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 956 op/s 2024-05-15T13:05:35.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:35 smithi120 ceph-mon[43961]: pgmap v1235: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 956 op/s 2024-05-15T13:05:36.422 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:36 smithi120 ceph-mon[43961]: pgmap v1236: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 801 op/s 2024-05-15T13:05:36.423 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:05:36.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:36.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:05:36.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:36 smithi177 ceph-mon[44303]: pgmap v1236: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 801 op/s 2024-05-15T13:05:36.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:36 smithi027 ceph-mon[39795]: pgmap v1236: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 801 op/s 2024-05-15T13:05:36.528 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:05:36.575 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:36.595 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-05-15T13:05:36.609 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:05:36.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:36.610 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:05:36.711 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:05:36.771 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:36.895 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:05:36.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:36.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:05:37.001 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:05:37.031 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:37.063 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:05:37.068 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:05:37.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:37.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:05:37.172 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:05:37.211 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:37.242 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:05:37.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:37.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:05:37.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:37 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:05:37.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:37 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:05:37.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:37 smithi027 ceph-mon[39795]: osdmap e459: 12 total, 12 up, 11 in 2024-05-15T13:05:37.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:37 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:05:37.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:37 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:05:37.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:37 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:05:37.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:37 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:05:37.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:37 smithi120 ceph-mon[43961]: osdmap e459: 12 total, 12 up, 11 in 2024-05-15T13:05:37.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:37 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:05:37.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:37 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:05:37.348 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:05:37.421 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:37.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:37 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:05:37.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:37 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:05:37.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:37 smithi177 ceph-mon[44303]: osdmap e459: 12 total, 12 up, 11 in 2024-05-15T13:05:37.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:37 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:05:37.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:37 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:05:37.458 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:05:37.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:37.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:05:37.567 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:05:37.605 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:37.635 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:05:37.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:37.644 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:05:37.745 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:05:37.746 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:05:37.781 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:37.816 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:05:37.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:37.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:05:37.921 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:05:37.996 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:38.057 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:05:38.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:38.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:05:38.168 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:05:38.209 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:38.243 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:05:38.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:38.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:05:38.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:38 smithi027 ceph-mon[39795]: osdmap e460: 12 total, 12 up, 11 in 2024-05-15T13:05:38.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:38 smithi027 ceph-mon[39795]: pgmap v1239: 17 pgs: 17 active+clean; 5.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-05-15T13:05:38.353 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:05:38.395 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:38.435 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:38 smithi177 ceph-mon[44303]: osdmap e460: 12 total, 12 up, 11 in 2024-05-15T13:05:38.435 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:38 smithi177 ceph-mon[44303]: pgmap v1239: 17 pgs: 17 active+clean; 5.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-05-15T13:05:38.504 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:05:38.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:38.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:05:38.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:38 smithi120 ceph-mon[43961]: osdmap e460: 12 total, 12 up, 11 in 2024-05-15T13:05:38.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:38 smithi120 ceph-mon[43961]: pgmap v1239: 17 pgs: 17 active+clean; 5.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-05-15T13:05:38.618 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:05:38.649 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:38.677 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:05:38.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:38.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:05:38.788 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:05:38.837 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:38.866 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:05:38.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:38.873 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:05:38.892 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:05:38.946 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:05:38.973 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:05:39.062 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:40.044 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:05:40.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:40.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:05:40.084 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:05:40.084 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:05:40.084 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.084 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:181331 5.a 5:5d1e17b1:::benchmark_data_smithi177_109991_object46140:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e460)", 2024-05-15T13:05:40.084 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:05:40.090037+0000", 2024-05-15T13:05:40.084 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.0011066909999999999, 2024-05-15T13:05:40.084 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001131115, 2024-05-15T13:05:40.084 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:40.084 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:05:40.084 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:40.084 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:05:40.085 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:05:40.085 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 181331 2024-05-15T13:05:40.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.085 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:40.085 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.085 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:40.085 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.090037+0000", 2024-05-15T13:05:40.085 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:40.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.085 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.085 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:40.085 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.090037+0000", 2024-05-15T13:05:40.085 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:40.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.085 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.086 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:40.086 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.090038+0000", 2024-05-15T13:05:40.086 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1999999999999998e-07 2024-05-15T13:05:40.086 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.086 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.086 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:40.086 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.090045+0000", 2024-05-15T13:05:40.086 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9189999999999999e-06 2024-05-15T13:05:40.086 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.086 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.086 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:40.086 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.090060+0000", 2024-05-15T13:05:40.087 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4173e-05 2024-05-15T13:05:40.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.087 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:40.087 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.090068+0000", 2024-05-15T13:05:40.087 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.816e-06 2024-05-15T13:05:40.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.087 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:40.087 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.090128+0000", 2024-05-15T13:05:40.087 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9707999999999998e-05 2024-05-15T13:05:40.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.087 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:40.087 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.090239+0000", 2024-05-15T13:05:40.088 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011076199999999999 2024-05-15T13:05:40.088 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.088 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.088 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,10", 2024-05-15T13:05:40.088 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.090346+0000", 2024-05-15T13:05:40.088 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000107588 2024-05-15T13:05:40.088 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:40.088 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:40.088 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:40.088 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.088 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.088 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:181318 5.6 5:6caba75c:::benchmark_data_smithi177_109991_object46127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e460)", 2024-05-15T13:05:40.088 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:05:40.074584+0000", 2024-05-15T13:05:40.088 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.016558960000000001, 2024-05-15T13:05:40.088 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016636722, 2024-05-15T13:05:40.089 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:40.089 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:05:40.089 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:40.089 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:05:40.089 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:05:40.089 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 181318 2024-05-15T13:05:40.089 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.089 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:40.089 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.089 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:40.089 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.074584+0000", 2024-05-15T13:05:40.089 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:40.089 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.089 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.090 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:40.090 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.074584+0000", 2024-05-15T13:05:40.090 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:40.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.090 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.090 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:40.090 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.074586+0000", 2024-05-15T13:05:40.090 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0389999999999999e-06 2024-05-15T13:05:40.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.090 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.090 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:40.091 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.074595+0000", 2024-05-15T13:05:40.091 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4919999999999993e-06 2024-05-15T13:05:40.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.091 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:40.091 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.074616+0000", 2024-05-15T13:05:40.091 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.139e-05 2024-05-15T13:05:40.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.091 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:40.091 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.074626+0000", 2024-05-15T13:05:40.091 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7990000000000005e-06 2024-05-15T13:05:40.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.092 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:40.092 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.074698+0000", 2024-05-15T13:05:40.092 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2297000000000003e-05 2024-05-15T13:05:40.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.092 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:40.092 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.074840+0000", 2024-05-15T13:05:40.092 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014177999999999999 2024-05-15T13:05:40.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.092 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,10", 2024-05-15T13:05:40.093 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.074972+0000", 2024-05-15T13:05:40.093 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000131669 2024-05-15T13:05:40.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.093 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:40.093 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.079583+0000", 2024-05-15T13:05:40.093 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046116719999999998 2024-05-15T13:05:40.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.093 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:05:40.093 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.081824+0000", 2024-05-15T13:05:40.093 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002240644 2024-05-15T13:05:40.093 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:40.093 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:40.094 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:40.094 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.094 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.094 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:181320 5.a 5:5b06d3b7:::benchmark_data_smithi177_109991_object46129:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e460)", 2024-05-15T13:05:40.094 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:05:40.081749+0000", 2024-05-15T13:05:40.094 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.0093938259999999992, 2024-05-15T13:05:40.094 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0095229259999999993, 2024-05-15T13:05:40.094 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:40.094 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:05:40.094 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:40.094 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:05:40.094 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:05:40.094 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 181320 2024-05-15T13:05:40.095 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.095 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:40.095 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.095 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:40.095 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.081749+0000", 2024-05-15T13:05:40.095 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:40.095 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.095 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.095 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:40.095 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.081749+0000", 2024-05-15T13:05:40.095 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:40.095 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.095 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.095 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:40.095 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.081751+0000", 2024-05-15T13:05:40.096 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.866e-06 2024-05-15T13:05:40.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.096 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:40.096 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.081763+0000", 2024-05-15T13:05:40.096 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1184e-05 2024-05-15T13:05:40.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.096 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:40.096 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.081777+0000", 2024-05-15T13:05:40.096 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4892000000000001e-05 2024-05-15T13:05:40.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.096 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:40.096 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.081787+0000", 2024-05-15T13:05:40.097 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6770000000000007e-06 2024-05-15T13:05:40.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.097 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:40.097 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.081843+0000", 2024-05-15T13:05:40.097 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5809e-05 2024-05-15T13:05:40.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.097 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:40.097 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.081944+0000", 2024-05-15T13:05:40.097 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000101227 2024-05-15T13:05:40.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.097 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,10", 2024-05-15T13:05:40.097 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.082026+0000", 2024-05-15T13:05:40.098 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2330999999999995e-05 2024-05-15T13:05:40.098 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:40.098 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:40.098 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:40.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.098 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.098 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:181329 5.6 5:60fec5ba:::benchmark_data_smithi177_109991_object46138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e460)", 2024-05-15T13:05:40.098 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:05:40.089159+0000", 2024-05-15T13:05:40.098 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.0019845380000000001, 2024-05-15T13:05:40.098 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0021576099999999999, 2024-05-15T13:05:40.098 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:05:40.098 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:05:40.098 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:05:40.098 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:05:40.099 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:05:40.099 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 181329 2024-05-15T13:05:40.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.099 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:05:40.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.099 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:05:40.099 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.089159+0000", 2024-05-15T13:05:40.099 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:40.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.099 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:05:40.099 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.089159+0000", 2024-05-15T13:05:40.099 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:05:40.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.099 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:05:40.100 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.089160+0000", 2024-05-15T13:05:40.100 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7150000000000001e-06 2024-05-15T13:05:40.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.100 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.100 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:05:40.100 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.089168+0000", 2024-05-15T13:05:40.100 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6899999999999992e-06 2024-05-15T13:05:40.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.100 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.100 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:05:40.100 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.089185+0000", 2024-05-15T13:05:40.100 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6382e-05 2024-05-15T13:05:40.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.100 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.100 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:05:40.100 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.089194+0000", 2024-05-15T13:05:40.101 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8509999999999994e-06 2024-05-15T13:05:40.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.101 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:05:40.101 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.089261+0000", 2024-05-15T13:05:40.101 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6403999999999996e-05 2024-05-15T13:05:40.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.101 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:05:40.101 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.089400+0000", 2024-05-15T13:05:40.101 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013922199999999999 2024-05-15T13:05:40.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:05:40.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:05:40.101 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,10", 2024-05-15T13:05:40.101 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:40.089533+0000", 2024-05-15T13:05:40.101 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013284399999999999 2024-05-15T13:05:40.102 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:40.102 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:05:40.102 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:40.102 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:05:40.102 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:05:40.102 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 4 2024-05-15T13:05:40.102 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:05:40.148 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:05:40.198 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:40.243 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:05:40.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:40.250 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:05:40.351 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:05:40.398 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:40.435 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:05:40.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:40.438 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:05:40.539 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:05:40.580 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:40.614 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:05:40.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:40.626 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:05:40.727 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:05:40.775 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:40.803 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:05:40.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:40.808 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:05:40.908 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:05:40.951 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:40.979 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:05:40.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:40.986 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:05:41.087 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:05:41.136 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:41.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:41 smithi177 ceph-mon[44303]: pgmap v1240: 17 pgs: 17 active+clean; 5.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 907 op/s 2024-05-15T13:05:41.464 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:05:41.465 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:05:41.465 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_3 2024-05-15T13:05:41.465 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:05:41.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:41 smithi027 ceph-mon[39795]: pgmap v1240: 17 pgs: 17 active+clean; 5.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 907 op/s 2024-05-15T13:05:41.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:41 smithi120 ceph-mon[43961]: pgmap v1240: 17 pgs: 17 active+clean; 5.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 907 op/s 2024-05-15T13:05:41.930 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:05:41.968 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-05-15T13:05:43.108 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:05:43.113 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:05:43.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:43.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:05:43.223 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:05:43.403 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:43 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:05:43.403 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:43 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:05:43.403 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:43 smithi120 ceph-mon[43961]: pgmap v1241: 17 pgs: 17 active+clean; 5.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 856 op/s 2024-05-15T13:05:43.403 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:43 smithi120 ceph-mon[43961]: osdmap e461: 12 total, 12 up, 11 in 2024-05-15T13:05:43.404 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:43.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:43 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:05:43.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:43 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:05:43.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:43 smithi177 ceph-mon[44303]: pgmap v1241: 17 pgs: 17 active+clean; 5.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 856 op/s 2024-05-15T13:05:43.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:43 smithi177 ceph-mon[44303]: osdmap e461: 12 total, 12 up, 11 in 2024-05-15T13:05:43.436 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:05:43.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:43.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:05:43.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:43 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:05:43.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:43 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:05:43.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:43 smithi027 ceph-mon[39795]: pgmap v1241: 17 pgs: 17 active+clean; 5.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 856 op/s 2024-05-15T13:05:43.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:43 smithi027 ceph-mon[39795]: osdmap e461: 12 total, 12 up, 11 in 2024-05-15T13:05:43.538 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:05:43.581 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:44 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:05:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:44 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:05:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:44 smithi177 ceph-mon[44303]: osdmap e462: 12 total, 12 up, 11 in 2024-05-15T13:05:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:44 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1775666021' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:05:44.436 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:05:44.437 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:05:44.437 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:05:44.437 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:05:44.437 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:05:44.437 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:05:44.437 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:05:44.437 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:05:44.437 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:05:44.437 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:05:44.437 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:05:44.437 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:05:44.437 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:05:44.438 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:05:44.438 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:05:44.438 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:05:44.438 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:05:44.438 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:05:44.438 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:05:44.438 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:05:44.438 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:05:44.438 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:05:44.438 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:05:44.439 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:05:44.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:44 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:05:44.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:44 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:05:44.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:44 smithi027 ceph-mon[39795]: osdmap e462: 12 total, 12 up, 11 in 2024-05-15T13:05:44.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:44 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1775666021' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:05:44.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:44 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:05:44.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:44 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:05:44.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:44 smithi120 ceph-mon[43961]: osdmap e462: 12 total, 12 up, 11 in 2024-05-15T13:05:44.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:44 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1775666021' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:05:45.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:45 smithi177 ceph-mon[44303]: pgmap v1244: 17 pgs: 17 active+clean; 5.4 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 988 op/s 2024-05-15T13:05:45.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:45 smithi027 ceph-mon[39795]: pgmap v1244: 17 pgs: 17 active+clean; 5.4 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 988 op/s 2024-05-15T13:05:45.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:45 smithi120 ceph-mon[43961]: pgmap v1244: 17 pgs: 17 active+clean; 5.4 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 988 op/s 2024-05-15T13:05:46.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:46 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:05:46.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:46 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:05:46.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:46 smithi177 ceph-mon[44303]: pgmap v1245: 17 pgs: 17 active+clean; 5.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 807 op/s 2024-05-15T13:05:46.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:46 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:05:46.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:46 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:05:46.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:46 smithi027 ceph-mon[39795]: pgmap v1245: 17 pgs: 17 active+clean; 5.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 807 op/s 2024-05-15T13:05:46.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:46 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:05:46.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:46 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:05:46.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:46 smithi120 ceph-mon[43961]: pgmap v1245: 17 pgs: 17 active+clean; 5.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 807 op/s 2024-05-15T13:05:46.658 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:05:46.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:46.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:05:46.766 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:05:46.814 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:46.884 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:05:46.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:46.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:05:46.991 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:05:47.047 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:47.181 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:47 smithi027 ceph-mon[39795]: osdmap e463: 12 total, 12 up, 11 in 2024-05-15T13:05:47.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:47 smithi177 ceph-mon[44303]: osdmap e463: 12 total, 12 up, 11 in 2024-05-15T13:05:47.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:47 smithi120 ceph-mon[43961]: osdmap e463: 12 total, 12 up, 11 in 2024-05-15T13:05:47.725 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:05:47.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:47.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:05:47.837 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:05:47.877 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:47.907 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:05:47.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:47.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:05:48.015 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:05:48.062 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:48.098 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:05:48.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:48.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:05:48.148 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:48 smithi027 ceph-mon[39795]: pgmap v1247: 17 pgs: 17 active+clean; 5.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s 2024-05-15T13:05:48.204 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:05:48.240 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:48.280 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:05:48.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:48.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:05:48.387 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:05:48.430 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:48.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:48 smithi177 ceph-mon[44303]: pgmap v1247: 17 pgs: 17 active+clean; 5.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s 2024-05-15T13:05:48.465 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:05:48.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:48.468 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:05:48.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:48 smithi120 ceph-mon[43961]: pgmap v1247: 17 pgs: 17 active+clean; 5.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s 2024-05-15T13:05:48.570 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:05:48.622 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:48.663 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:05:48.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:48.670 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:05:48.771 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:05:48.826 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:48.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:48.861 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:05:48.861 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:05:48.962 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:05:49.003 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:49.039 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:05:49.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:49.051 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:05:49.109 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:05:49.151 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:05:49.156 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:49 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1064770562' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:05:49.191 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:05:49.230 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:49.410 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:05:49.414 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:49 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1064770562' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:05:49.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:49.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:05:49.517 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:05:49.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:49 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1064770562' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:05:49.576 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:50.385 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:05:50.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:50 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:05:50.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:50 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1064770562' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:05:50.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:50 smithi177 ceph-mon[44303]: osdmap e464: 12 total, 12 up, 11 in 2024-05-15T13:05:50.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:50 smithi177 ceph-mon[44303]: pgmap v1249: 17 pgs: 17 active+clean; 5.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 923 op/s 2024-05-15T13:05:50.438 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:50 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:05:50.438 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:50 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1064770562' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:05:50.438 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:50 smithi027 ceph-mon[39795]: osdmap e464: 12 total, 12 up, 11 in 2024-05-15T13:05:50.438 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:50 smithi027 ceph-mon[39795]: pgmap v1249: 17 pgs: 17 active+clean; 5.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 923 op/s 2024-05-15T13:05:50.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:50 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:05:50.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:50 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1064770562' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:05:50.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:50 smithi120 ceph-mon[43961]: osdmap e464: 12 total, 12 up, 11 in 2024-05-15T13:05:50.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:50 smithi120 ceph-mon[43961]: pgmap v1249: 17 pgs: 17 active+clean; 5.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 923 op/s 2024-05-15T13:05:51.225 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:05:51.225 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:05:51.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:51 smithi177 ceph-mon[44303]: from='client.25520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:05:51.459 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:05:51.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:51.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:05:51.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:51 smithi027 ceph-mon[39795]: from='client.25520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:05:51.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:51 smithi120 ceph-mon[43961]: from='client.25520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:05:51.565 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:05:51.606 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:51.978 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:05:51.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:51.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:05:52.002 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":1249,"stamp":"2024-05-15T13:05:50.026565+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5363008056,"num_objects":81827,"num_object_clones":0,"num_object_copies":245481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81827,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":188734,"num_write_kb":8654971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160992,"ondisk_log_size":160992,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":20715892,"kb_used_data":16098308,"kb_used_omap":17,"kb_used_meta":4617390,"kb_avail":1010480780,"statfs":{"total":1055945392128,"available":1034732318720,"internally_reserved":0,"allocated":16484667392,"data_stored":16481588706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17551,"internal_metadata":4728208241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-418906112,"num_objects":-6392,"num_object_clones":0,"num_object_copies":-19176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6392,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.920667"},"pg_stats":[{"pgid":"5.f","version":"463'11790","reported_seq":11983,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T13:05:47.912661+0000","last_change":"2024-05-15T13:05:13.243193+0000","last_active":"2024-05-15T13:05:47.912661+0000","last_peered":"2024-05-15T13:05:47.912661+0000","last_clean":"2024-05-15T13:05:47.912661+0000","last_became_active":"2024-05-15T13:03:11.328783+0000","last_became_peered":"2024-05-15T13:03:11.328783+0000","last_unstale":"2024-05-15T13:05:47.912661+0000","last_undegraded":"2024-05-15T13:05:47.912661+0000","last_fullsized":"2024-05-15T13:05:47.912661+0000","mapping_epoch":418,"log_start":"425'1700","ondisk_log_start":"425'1700","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'9853","last_scrub_stamp":"2024-05-15T13:05:13.242947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:05:13.242947+0000","objects_scrubbed":7125,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:06:15.122366+0000","scrub_duration":1769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336330752,"num_objects":5132,"num_object_clones":0,"num_object_copies":15396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11790,"num_write_kb":541504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"462'11749","reported_seq":11925,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-05-15T13:05:45.623774+0000","last_change":"2024-05-15T13:04:26.638047+0000","last_active":"2024-05-15T13:05:45.623774+0000","last_peered":"2024-05-15T13:05:45.623774+0000","last_clean":"2024-05-15T13:05:45.623774+0000","last_became_active":"2024-05-15T13:03:11.330241+0000","last_became_peered":"2024-05-15T13:03:11.330241+0000","last_unstale":"2024-05-15T13:05:45.623774+0000","last_undegraded":"2024-05-15T13:05:45.623774+0000","last_fullsized":"2024-05-15T13:05:45.623774+0000","mapping_epoch":418,"log_start":"425'1700","ondisk_log_start":"425'1700","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'6959","last_scrub_stamp":"2024-05-15T13:04:26.637942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:26.637942+0000","objects_scrubbed":6905,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1597,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336134144,"num_objects":5129,"num_object_clones":0,"num_object_copies":15387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11749,"num_write_kb":540096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"463'11957","reported_seq":12141,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T13:05:47.570897+0000","last_change":"2024-05-15T13:04:34.347782+0000","last_active":"2024-05-15T13:05:47.570897+0000","last_peered":"2024-05-15T13:05:47.570897+0000","last_clean":"2024-05-15T13:05:47.570897+0000","last_became_active":"2024-05-15T13:03:11.330919+0000","last_became_peered":"2024-05-15T13:03:11.330919+0000","last_unstale":"2024-05-15T13:05:47.570897+0000","last_undegraded":"2024-05-15T13:05:47.570897+0000","last_fullsized":"2024-05-15T13:05:47.570897+0000","mapping_epoch":418,"log_start":"426'1900","ondisk_log_start":"426'1900","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'7491","last_scrub_stamp":"2024-05-15T13:04:34.347495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:34.347495+0000","objects_scrubbed":7414,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:49.420446+0000","scrub_duration":2380,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344457240,"num_objects":5257,"num_object_clones":0,"num_object_copies":15771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5257,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11957,"num_write_kb":550785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"463'11691","reported_seq":11875,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T13:05:46.443344+0000","last_change":"2024-05-15T13:04:35.286472+0000","last_active":"2024-05-15T13:05:46.443344+0000","last_peered":"2024-05-15T13:05:46.443344+0000","last_clean":"2024-05-15T13:05:46.443344+0000","last_became_active":"2024-05-15T13:03:11.327387+0000","last_became_peered":"2024-05-15T13:03:11.327387+0000","last_unstale":"2024-05-15T13:05:46.443344+0000","last_undegraded":"2024-05-15T13:05:46.443344+0000","last_fullsized":"2024-05-15T13:05:46.443344+0000","mapping_epoch":418,"log_start":"425'1600","ondisk_log_start":"425'1600","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'7507","last_scrub_stamp":"2024-05-15T13:04:35.285734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:35.285734+0000","objects_scrubbed":7450,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:57.795544+0000","scrub_duration":1571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341245952,"num_objects":5207,"num_object_clones":0,"num_object_copies":15621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11691,"num_write_kb":540736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"464'11750","reported_seq":11929,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-05-15T13:05:49.321172+0000","last_change":"2024-05-15T13:04:25.608723+0000","last_active":"2024-05-15T13:05:49.321172+0000","last_peered":"2024-05-15T13:05:49.321172+0000","last_clean":"2024-05-15T13:05:49.321172+0000","last_became_active":"2024-05-15T13:03:11.328691+0000","last_became_peered":"2024-05-15T13:03:11.328691+0000","last_unstale":"2024-05-15T13:05:49.321172+0000","last_undegraded":"2024-05-15T13:05:49.321172+0000","last_fullsized":"2024-05-15T13:05:49.321172+0000","mapping_epoch":418,"log_start":"426'1700","ondisk_log_start":"426'1700","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'6796","last_scrub_stamp":"2024-05-15T13:04:25.608583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:25.608583+0000","objects_scrubbed":6750,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:50.861823+0000","scrub_duration":1526,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330563584,"num_objects":5044,"num_object_clones":0,"num_object_copies":15132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11750,"num_write_kb":537408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,5],"acting":[0,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"463'11715","reported_seq":11960,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T13:05:47.614695+0000","last_change":"2024-05-15T13:05:29.969040+0000","last_active":"2024-05-15T13:05:47.614695+0000","last_peered":"2024-05-15T13:05:47.614695+0000","last_clean":"2024-05-15T13:05:47.614695+0000","last_became_active":"2024-05-15T13:03:11.330425+0000","last_became_peered":"2024-05-15T13:03:11.330425+0000","last_unstale":"2024-05-15T13:05:47.614695+0000","last_undegraded":"2024-05-15T13:05:47.614695+0000","last_fullsized":"2024-05-15T13:05:47.614695+0000","mapping_epoch":418,"log_start":"426'1700","ondisk_log_start":"426'1700","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'10689","last_scrub_stamp":"2024-05-15T13:05:29.968791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:05:29.968791+0000","objects_scrubbed":6257,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:06:45.596029+0000","scrub_duration":1767,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338755584,"num_objects":5169,"num_object_clones":0,"num_object_copies":15507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11715,"num_write_kb":540288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"463'11737","reported_seq":11914,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T13:05:48.663461+0000","last_change":"2024-05-15T13:04:27.305971+0000","last_active":"2024-05-15T13:05:48.663461+0000","last_peered":"2024-05-15T13:05:48.663461+0000","last_clean":"2024-05-15T13:05:48.663461+0000","last_became_active":"2024-05-15T13:03:11.329210+0000","last_became_peered":"2024-05-15T13:03:11.329210+0000","last_unstale":"2024-05-15T13:05:48.663461+0000","last_undegraded":"2024-05-15T13:05:48.663461+0000","last_fullsized":"2024-05-15T13:05:48.663461+0000","mapping_epoch":418,"log_start":"426'1700","ondisk_log_start":"426'1700","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'6893","last_scrub_stamp":"2024-05-15T13:04:27.305709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:27.305709+0000","objects_scrubbed":6847,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1459,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328531968,"num_objects":5013,"num_object_clones":0,"num_object_copies":15039,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11737,"num_write_kb":536000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,9],"acting":[6,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"463'11563","reported_seq":11742,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T13:05:47.590940+0000","last_change":"2024-05-15T13:04:29.718135+0000","last_active":"2024-05-15T13:05:47.590940+0000","last_peered":"2024-05-15T13:05:47.590940+0000","last_clean":"2024-05-15T13:05:47.590940+0000","last_became_active":"2024-05-15T13:03:11.330983+0000","last_became_peered":"2024-05-15T13:03:11.330983+0000","last_unstale":"2024-05-15T13:05:47.590940+0000","last_undegraded":"2024-05-15T13:05:47.590940+0000","last_fullsized":"2024-05-15T13:05:47.590940+0000","mapping_epoch":418,"log_start":"425'1500","ondisk_log_start":"425'1500","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"447'7023","last_scrub_stamp":"2024-05-15T13:04:29.717870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:29.717870+0000","objects_scrubbed":6961,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:47.890055+0000","scrub_duration":1740,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335216640,"num_objects":5115,"num_object_clones":0,"num_object_copies":15345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11563,"num_write_kb":533696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"463'11917","reported_seq":12091,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T13:05:47.880136+0000","last_change":"2024-05-15T13:04:20.177631+0000","last_active":"2024-05-15T13:05:47.880136+0000","last_peered":"2024-05-15T13:05:47.880136+0000","last_clean":"2024-05-15T13:05:47.880136+0000","last_became_active":"2024-05-15T13:03:11.329484+0000","last_became_peered":"2024-05-15T13:03:11.329484+0000","last_unstale":"2024-05-15T13:05:47.880136+0000","last_undegraded":"2024-05-15T13:05:47.880136+0000","last_fullsized":"2024-05-15T13:05:47.880136+0000","mapping_epoch":418,"log_start":"426'1900","ondisk_log_start":"426'1900","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'6536","last_scrub_stamp":"2024-05-15T13:04:20.177485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:20.177485+0000","objects_scrubbed":6499,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1314,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331153408,"num_objects":5053,"num_object_clones":0,"num_object_copies":15159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11917,"num_write_kb":543040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"463'11788","reported_seq":11961,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T13:05:47.601300+0000","last_change":"2024-05-15T13:04:18.779479+0000","last_active":"2024-05-15T13:05:47.601300+0000","last_peered":"2024-05-15T13:05:47.601300+0000","last_clean":"2024-05-15T13:05:47.601300+0000","last_became_active":"2024-05-15T13:03:11.332923+0000","last_became_peered":"2024-05-15T13:03:11.332923+0000","last_unstale":"2024-05-15T13:05:47.601300+0000","last_undegraded":"2024-05-15T13:05:47.601300+0000","last_fullsized":"2024-05-15T13:05:47.601300+0000","mapping_epoch":418,"log_start":"426'1700","ondisk_log_start":"426'1700","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'6425","last_scrub_stamp":"2024-05-15T13:04:18.779285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:18.779285+0000","objects_scrubbed":6366,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:48.641808+0000","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329383936,"num_objects":5026,"num_object_clones":0,"num_object_copies":15078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11788,"num_write_kb":538048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,10],"acting":[2,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"464'11796","reported_seq":11970,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-05-15T13:05:49.331252+0000","last_change":"2024-05-15T13:04:22.066492+0000","last_active":"2024-05-15T13:05:49.331252+0000","last_peered":"2024-05-15T13:05:49.331252+0000","last_clean":"2024-05-15T13:05:49.331252+0000","last_became_active":"2024-05-15T13:03:11.333336+0000","last_became_peered":"2024-05-15T13:03:11.333336+0000","last_unstale":"2024-05-15T13:05:49.331252+0000","last_undegraded":"2024-05-15T13:05:49.331252+0000","last_fullsized":"2024-05-15T13:05:49.331252+0000","mapping_epoch":418,"log_start":"426'1700","ondisk_log_start":"426'1700","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'6582","last_scrub_stamp":"2024-05-15T13:04:22.066294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:22.066294+0000","objects_scrubbed":6533,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:52.896549+0000","scrub_duration":1991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328466432,"num_objects":5012,"num_object_clones":0,"num_object_copies":15036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11796,"num_write_kb":537856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"464'12002","reported_seq":12179,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-05-15T13:05:49.252262+0000","last_change":"2024-05-15T13:04:22.690622+0000","last_active":"2024-05-15T13:05:49.252262+0000","last_peered":"2024-05-15T13:05:49.252262+0000","last_clean":"2024-05-15T13:05:49.252262+0000","last_became_active":"2024-05-15T13:03:11.327788+0000","last_became_peered":"2024-05-15T13:03:11.327788+0000","last_unstale":"2024-05-15T13:05:49.252262+0000","last_undegraded":"2024-05-15T13:05:49.252262+0000","last_fullsized":"2024-05-15T13:05:49.252262+0000","mapping_epoch":418,"log_start":"426'2000","ondisk_log_start":"426'2000","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'6704","last_scrub_stamp":"2024-05-15T13:04:22.690346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:22.690346+0000","objects_scrubbed":6642,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:51.461018+0000","scrub_duration":1607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336855040,"num_objects":5140,"num_object_clones":0,"num_object_copies":15420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12002,"num_write_kb":548544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"463'11789","reported_seq":11966,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T13:05:48.697081+0000","last_change":"2024-05-15T13:04:26.196007+0000","last_active":"2024-05-15T13:05:48.697081+0000","last_peered":"2024-05-15T13:05:48.697081+0000","last_clean":"2024-05-15T13:05:48.697081+0000","last_became_active":"2024-05-15T13:03:11.330316+0000","last_became_peered":"2024-05-15T13:03:11.330316+0000","last_unstale":"2024-05-15T13:05:48.697081+0000","last_undegraded":"2024-05-15T13:05:48.697081+0000","last_fullsized":"2024-05-15T13:05:48.697081+0000","mapping_epoch":418,"log_start":"426'1700","ondisk_log_start":"426'1700","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'6865","last_scrub_stamp":"2024-05-15T13:04:26.195895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:26.195895+0000","objects_scrubbed":6825,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:55.500383+0000","scrub_duration":1403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334299136,"num_objects":5101,"num_object_clones":0,"num_object_copies":15303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11789,"num_write_kb":540480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"464'11941","reported_seq":12118,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-05-15T13:05:49.950312+0000","last_change":"2024-05-15T13:04:22.294876+0000","last_active":"2024-05-15T13:05:49.950312+0000","last_peered":"2024-05-15T13:05:49.950312+0000","last_clean":"2024-05-15T13:05:49.950312+0000","last_became_active":"2024-05-15T13:03:11.430987+0000","last_became_peered":"2024-05-15T13:03:11.430987+0000","last_unstale":"2024-05-15T13:05:49.950312+0000","last_undegraded":"2024-05-15T13:05:49.950312+0000","last_fullsized":"2024-05-15T13:05:49.950312+0000","mapping_epoch":418,"log_start":"426'1900","ondisk_log_start":"426'1900","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'6645","last_scrub_stamp":"2024-05-15T13:04:22.294671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:22.294671+0000","objects_scrubbed":6590,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:53.848469+0000","scrub_duration":1280,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328663040,"num_objects":5015,"num_object_clones":0,"num_object_copies":15045,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11941,"num_write_kb":542592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"463'11625","reported_seq":11812,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T13:05:46.424162+0000","last_change":"2024-05-15T13:04:39.845227+0000","last_active":"2024-05-15T13:05:46.424162+0000","last_peered":"2024-05-15T13:05:46.424162+0000","last_clean":"2024-05-15T13:05:46.424162+0000","last_became_active":"2024-05-15T13:03:11.330479+0000","last_became_peered":"2024-05-15T13:03:11.330479+0000","last_unstale":"2024-05-15T13:05:46.424162+0000","last_undegraded":"2024-05-15T13:05:46.424162+0000","last_fullsized":"2024-05-15T13:05:46.424162+0000","mapping_epoch":418,"log_start":"425'1600","ondisk_log_start":"425'1600","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'7705","last_scrub_stamp":"2024-05-15T13:04:39.844914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:04:39.844914+0000","objects_scrubbed":7656,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:06:06.170367+0000","scrub_duration":2147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341114880,"num_objects":5205,"num_object_clones":0,"num_object_copies":15615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11625,"num_write_kb":538560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,4],"acting":[1,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"463'11743","reported_seq":11986,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T13:05:46.443432+0000","last_change":"2024-05-15T13:05:32.580562+0000","last_active":"2024-05-15T13:05:46.443432+0000","last_peered":"2024-05-15T13:05:46.443432+0000","last_clean":"2024-05-15T13:05:46.443432+0000","last_became_active":"2024-05-15T13:03:11.331291+0000","last_became_peered":"2024-05-15T13:03:11.331291+0000","last_unstale":"2024-05-15T13:05:46.443432+0000","last_undegraded":"2024-05-15T13:05:46.443432+0000","last_fullsized":"2024-05-15T13:05:46.443432+0000","mapping_epoch":418,"log_start":"426'1700","ondisk_log_start":"426'1700","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'10936","last_scrub_stamp":"2024-05-15T13:05:32.580374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:05:32.580374+0000","objects_scrubbed":6054,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:07:01.319356+0000","scrub_duration":1468,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341245952,"num_objects":5207,"num_object_clones":0,"num_object_copies":15621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11743,"num_write_kb":542400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":753,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T13:05:46.150079+0000","last_change":"2024-05-15T13:04:38.891844+0000","last_active":"2024-05-15T13:05:46.150079+0000","last_peered":"2024-05-15T13:05:46.150079+0000","last_clean":"2024-05-15T13:05:46.150079+0000","last_became_active":"2024-05-15T13:00:26.045412+0000","last_became_peered":"2024-05-15T13:00:26.045412+0000","last_unstale":"2024-05-15T13:05:46.150079+0000","last_undegraded":"2024-05-15T13:05:46.150079+0000","last_fullsized":"2024-05-15T13:05:46.150079+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:04:38.891702+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:04:38.891702+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:05:49.376370+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":5362417688,"num_objects":81825,"num_object_clones":0,"num_object_copies":245475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81825,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188553,"num_write_kb":8652033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16128421888,"data_stored":16128409672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160853,"ondisk_log_size":160853,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056884,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1734044,"kb_used_data":1323792,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92011108,"statfs":{"total":95995035648,"available":94219374592,"internally_reserved":0,"allocated":1355563008,"data_stored":1355279916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":420018630},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351224,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2513744,"kb_used_data":2099400,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":91231408,"statfs":{"total":95995035648,"available":93420961792,"internally_reserved":0,"allocated":2149785600,"data_stored":2149507460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":424278464},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514750,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2204488,"kb_used_data":1686016,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":91540664,"statfs":{"total":95995035648,"available":93737639936,"internally_reserved":0,"allocated":1726480384,"data_stored":1726205796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":530905536},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874499,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1042188,"kb_used_data":750016,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":92702964,"statfs":{"total":95995035648,"available":94927835136,"internally_reserved":0,"allocated":768016384,"data_stored":767741816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":240518168838,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430476,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1460128,"kb_used_data":1067988,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92285024,"statfs":{"total":95995035648,"available":94499864576,"internally_reserved":0,"allocated":1093619712,"data_stored":1093337160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":401537478},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575143,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1850208,"kb_used_data":1331732,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":91894944,"statfs":{"total":95995035648,"available":94100422656,"internally_reserved":0,"allocated":1363693568,"data_stored":1363411016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738672,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3123160,"kb_used_data":2600528,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":90621992,"statfs":{"total":95995035648,"available":92796919808,"internally_reserved":0,"allocated":2662940672,"data_stored":2662657600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411620,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1269012,"kb_used_data":985164,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":92476140,"statfs":{"total":95995035648,"available":94695567360,"internally_reserved":0,"allocated":1008807936,"data_stored":1008528936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149981,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":772872,"kb_used_data":336320,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92972280,"statfs":{"total":95995035648,"available":95203614720,"internally_reserved":0,"allocated":344391680,"data_stored":344117112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":447019463},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986457,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2039652,"kb_used_data":1603096,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":91705500,"statfs":{"total":95995035648,"available":93906432000,"internally_reserved":0,"allocated":1641570304,"data_stored":1641286822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692114,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2706396,"kb_used_data":2314256,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":91038756,"statfs":{"total":95995035648,"available":93223686144,"internally_reserved":0,"allocated":2369798144,"data_stored":2369515072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2648117248,"data_stored":2648113176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1359347712,"data_stored":1359347712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1005191168,"data_stored":1005191168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":339476480,"data_stored":339476480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1638465536,"data_stored":1638465536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2352484352,"data_stored":2352480280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1001652224,"data_stored":1001652224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":686620672,"data_stored":686620672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1706098688,"data_stored":1706098688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2040266752,"data_stored":2040266752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1350701056,"data_stored":1350696984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:05:52.003 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool set unique_pool_3 pgp_num 16 2024-05-15T13:05:52.081 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:05:52.136 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:52.167 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:05:52.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:52.170 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:05:52.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:52 smithi027 ceph-mon[39795]: pgmap v1250: 17 pgs: 17 active+clean; 4.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 879 op/s 2024-05-15T13:05:52.271 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:05:52.315 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:52.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:52 smithi177 ceph-mon[44303]: pgmap v1250: 17 pgs: 17 active+clean; 4.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 879 op/s 2024-05-15T13:05:52.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:52 smithi120 ceph-mon[43961]: pgmap v1250: 17 pgs: 17 active+clean; 4.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 879 op/s 2024-05-15T13:05:53.177 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:05:53.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:53.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:05:53.214 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:05:53.231 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:05:53.285 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:05:53.329 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:53.363 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:05:53.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:53.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:05:53.480 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:05:53.536 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:54.459 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:05:54.459 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:05:54.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:54 smithi177 ceph-mon[44303]: pgmap v1251: 17 pgs: 17 active+clean; 4.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 987 op/s 2024-05-15T13:05:54.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:54 smithi027 ceph-mon[39795]: pgmap v1251: 17 pgs: 17 active+clean; 4.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 987 op/s 2024-05-15T13:05:54.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:54 smithi120 ceph-mon[43961]: pgmap v1251: 17 pgs: 17 active+clean; 4.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 987 op/s 2024-05-15T13:05:54.860 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:05:54.902 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:05:54.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:54.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:05:55.016 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:05:55.090 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:56.687 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:05:56.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:56.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:05:56.794 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:05:56.854 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:56.896 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:05:56.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:56.901 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:05:57.002 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:05:57.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:56 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:05:57.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:56 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:05:57.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:56 smithi027 ceph-mon[39795]: osdmap e465: 12 total, 12 up, 11 in 2024-05-15T13:05:57.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:56 smithi027 ceph-mon[39795]: pgmap v1253: 17 pgs: 17 active+clean; 4.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 840 op/s 2024-05-15T13:05:57.055 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:57.086 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:05:57.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:57.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:05:57.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:56 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:05:57.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:56 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:05:57.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:56 smithi177 ceph-mon[44303]: osdmap e465: 12 total, 12 up, 11 in 2024-05-15T13:05:57.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:56 smithi177 ceph-mon[44303]: pgmap v1253: 17 pgs: 17 active+clean; 4.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 840 op/s 2024-05-15T13:05:57.193 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:05:57.248 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:57.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:56 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:05:57.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:56 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:05:57.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:56 smithi120 ceph-mon[43961]: osdmap e465: 12 total, 12 up, 11 in 2024-05-15T13:05:57.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:56 smithi120 ceph-mon[43961]: pgmap v1253: 17 pgs: 17 active+clean; 4.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 840 op/s 2024-05-15T13:05:57.285 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:05:57.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:57.291 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:05:57.392 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:05:57.436 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:57.808 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:05:57.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:57.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:05:57.857 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-05-15T13:05:57.918 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:05:57.954 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:57.984 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:05:57.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:57.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:05:58.090 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:05:58.141 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:58.162 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:05:58.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:58.170 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:05:58.271 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:05:58.316 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:58.337 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:05:58.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:58.341 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:05:58.442 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:05:58.468 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:58.487 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:05:58.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:58.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:05:58.591 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:05:58.625 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:58.651 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:05:58.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:58.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:05:58.664 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:05:58.754 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:05:58.799 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:58.840 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:05:58.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:58.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:05:58.945 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:05:58.992 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:59.020 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:05:59.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:59.023 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:05:59.104 INFO:teuthology.orchestra.run.smithi027.stderr:set pool 5 pgp_num to 16 2024-05-15T13:05:59.124 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:05:59.161 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:59.185 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:05:59.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:59.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:05:59.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:59 smithi027 ceph-mon[39795]: pgmap v1254: 17 pgs: 17 active+clean; 4.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 988 op/s 2024-05-15T13:05:59.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:59 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1303062652' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-05-15T13:05:59.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:05:59 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-05-15T13:05:59.300 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:05:59.361 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:59.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:59 smithi177 ceph-mon[44303]: pgmap v1254: 17 pgs: 17 active+clean; 4.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 988 op/s 2024-05-15T13:05:59.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:59 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1303062652' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-05-15T13:05:59.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:05:59 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-05-15T13:05:59.429 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:05:59.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:05:59.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:05:59.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:59 smithi120 ceph-mon[43961]: pgmap v1254: 17 pgs: 17 active+clean; 4.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 988 op/s 2024-05-15T13:05:59.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:59 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1303062652' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-05-15T13:05:59.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:05:59 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-05-15T13:05:59.536 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:05:59.578 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:05:59.788 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:05:59.825 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:05:59.854 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:05:59.893 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:05:59.938 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:05:59.967 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:06:00.010 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:06:00.053 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:06:00.098 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:06:00.146 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:06:00.174 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:06:00.210 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:06:00.218 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:06:00.218 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:06:00.218 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:06:00.219 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:06:00.219 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:06:00.219 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:06:00.219 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:06:00.219 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:06:00.219 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:06:00.219 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:06:00.219 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:06:00.219 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:06:00.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:00 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-05-15T13:06:00.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:00 smithi177 ceph-mon[44303]: osdmap e466: 12 total, 12 up, 11 in 2024-05-15T13:06:00.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:00 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1509139331' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:06:00.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:00 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-05-15T13:06:00.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:00 smithi027 ceph-mon[39795]: osdmap e466: 12 total, 12 up, 11 in 2024-05-15T13:06:00.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:00 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1509139331' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:06:00.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:00 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-05-15T13:06:00.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:00 smithi120 ceph-mon[43961]: osdmap e466: 12 total, 12 up, 11 in 2024-05-15T13:06:00.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:00 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1509139331' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:06:01.119 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:06:01.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:01 smithi027 ceph-mon[39795]: pgmap v1256: 17 pgs: 17 active+clean; 4.4 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-05-15T13:06:01.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:01 smithi177 ceph-mon[44303]: pgmap v1256: 17 pgs: 17 active+clean; 4.4 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-05-15T13:06:01.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:01 smithi120 ceph-mon[43961]: pgmap v1256: 17 pgs: 17 active+clean; 4.4 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-05-15T13:06:02.877 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:06:02.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:02.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:06:02.986 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:06:03.036 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:03.067 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:06:03.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:03.070 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:06:03.154 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:03 smithi027 ceph-mon[39795]: pgmap v1257: 17 pgs: 17 active+clean; 4.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 866 op/s 2024-05-15T13:06:03.171 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:06:03.221 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:03.242 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:06:03.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:03.251 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:06:03.352 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:06:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:03.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:03 smithi177 ceph-mon[44303]: pgmap v1257: 17 pgs: 17 active+clean; 4.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 866 op/s 2024-05-15T13:06:03.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:03 smithi120 ceph-mon[43961]: pgmap v1257: 17 pgs: 17 active+clean; 4.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 866 op/s 2024-05-15T13:06:03.554 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:06:03.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:03.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:06:03.662 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:06:03.750 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:03.899 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:06:03.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:03.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:06:04.004 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:06:04.045 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:04.085 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:06:04.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:04.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:06:04.194 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:06:04.229 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:04.257 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:06:04.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:04.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:06:04.361 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:06:04.395 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:04.411 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:06:04.419 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:06:04.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:04.424 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:06:04.525 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:06:04.577 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:04.643 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:06:04.666 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:06:04.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:04.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:06:04.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:04 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2871750781' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:06:04.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:04 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:06:04.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:04 smithi027 ceph-mon[39795]: pgmap v1258: 17 pgs: 17 active+clean; 4.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 939 op/s 2024-05-15T13:06:04.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:04 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2871750781' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:06:04.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:04 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:06:04.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:04 smithi177 ceph-mon[44303]: pgmap v1258: 17 pgs: 17 active+clean; 4.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 939 op/s 2024-05-15T13:06:04.778 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:06:04.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:04 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2871750781' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:06:04.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:04 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:06:04.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:04 smithi120 ceph-mon[43961]: pgmap v1258: 17 pgs: 17 active+clean; 4.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 939 op/s 2024-05-15T13:06:04.811 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:04.835 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:06:04.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:04.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:06:04.943 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:06:04.974 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:05.008 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:06:05.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:05.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:06:05.115 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:06:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:05.600 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:05 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:06:05.600 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:05 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:06:05.600 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:05 smithi027 ceph-mon[39795]: osdmap e467: 12 total, 12 up, 11 in 2024-05-15T13:06:05.651 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:06:05.651 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T13:06:05.651 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T13:06:05.652 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:06:05.652 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.652 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:17231 5.6 5:6074def8:::benchmark_data_smithi177_109991_object17230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:06:05.652 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:26.276983+0000", 2024-05-15T13:06:05.652 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 159.039892708, 2024-05-15T13:06:05.652 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.094001816000000002, 2024-05-15T13:06:05.652 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:05.652 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:05.652 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:06:05.652 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:06:05.652 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:06:05.652 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 17231 2024-05-15T13:06:05.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.653 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:05.653 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.653 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:05.653 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.276983+0000", 2024-05-15T13:06:05.653 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.653 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.653 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:05.653 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.276983+0000", 2024-05-15T13:06:05.653 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.653 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.653 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:05.654 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.276985+0000", 2024-05-15T13:06:05.654 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4020000000000001e-06 2024-05-15T13:06:05.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.654 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.654 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:05.654 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.277043+0000", 2024-05-15T13:06:05.654 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8356000000000002e-05 2024-05-15T13:06:05.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.654 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.654 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:05.654 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.277055+0000", 2024-05-15T13:06:05.654 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2051e-05 2024-05-15T13:06:05.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.655 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.655 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:05.655 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.277061+0000", 2024-05-15T13:06:05.655 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7019999999999998e-06 2024-05-15T13:06:05.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.655 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.655 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:05.655 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.277204+0000", 2024-05-15T13:06:05.655 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014371499999999999 2024-05-15T13:06:05.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.655 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.655 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:05.655 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.277328+0000", 2024-05-15T13:06:05.655 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012367199999999999 2024-05-15T13:06:05.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.656 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.656 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,10", 2024-05-15T13:06:05.656 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.277467+0000", 2024-05-15T13:06:05.656 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000138632 2024-05-15T13:06:05.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.656 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.656 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:06:05.656 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.279118+0000", 2024-05-15T13:06:05.656 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001651065 2024-05-15T13:06:05.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.656 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.656 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.656 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.279942+0000", 2024-05-15T13:06:05.657 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00082398300000000003 2024-05-15T13:06:05.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.657 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.657 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.657 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280627+0000", 2024-05-15T13:06:05.657 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00068536200000000002 2024-05-15T13:06:05.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.657 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.657 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:05.657 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.370934+0000", 2024-05-15T13:06:05.657 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.090306484000000006 2024-05-15T13:06:05.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.657 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.657 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:05.658 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.370985+0000", 2024-05-15T13:06:05.658 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1391999999999999e-05 2024-05-15T13:06:05.658 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.658 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:05.658 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.658 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.658 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:17234 5.6 5:66a9cf93:::benchmark_data_smithi177_109991_object17233:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:06:05.658 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:26.278295+0000", 2024-05-15T13:06:05.658 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 159.038580368, 2024-05-15T13:06:05.658 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.092853842000000006, 2024-05-15T13:06:05.658 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:05.659 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:05.659 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:06:05.659 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:06:05.659 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:06:05.659 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 17234 2024-05-15T13:06:05.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.659 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:05.659 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.659 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:05.659 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.278295+0000", 2024-05-15T13:06:05.659 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.659 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.659 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:05.659 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.278295+0000", 2024-05-15T13:06:05.660 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.660 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.660 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.660 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:05.660 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.278296+0000", 2024-05-15T13:06:05.660 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.569999999999999e-07 2024-05-15T13:06:05.660 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.660 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.660 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:05.660 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.278329+0000", 2024-05-15T13:06:05.660 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2928000000000001e-05 2024-05-15T13:06:05.660 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.660 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.660 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:05.661 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.278341+0000", 2024-05-15T13:06:05.661 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1450999999999999e-05 2024-05-15T13:06:05.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.661 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.661 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:05.661 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.278348+0000", 2024-05-15T13:06:05.661 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0709999999999997e-06 2024-05-15T13:06:05.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.661 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.661 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:05.661 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.278398+0000", 2024-05-15T13:06:05.661 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.99e-05 2024-05-15T13:06:05.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.661 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.661 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:05.662 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.278494+0000", 2024-05-15T13:06:05.662 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5989000000000005e-05 2024-05-15T13:06:05.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.662 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.662 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,10", 2024-05-15T13:06:05.662 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.278590+0000", 2024-05-15T13:06:05.662 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6032000000000005e-05 2024-05-15T13:06:05.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.662 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.662 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:06:05.662 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.279256+0000", 2024-05-15T13:06:05.663 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00066654099999999999 2024-05-15T13:06:05.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.663 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.663 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.663 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280063+0000", 2024-05-15T13:06:05.663 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00080693799999999999 2024-05-15T13:06:05.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.663 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.663 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.663 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.371051+0000", 2024-05-15T13:06:05.663 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.090988187999999998 2024-05-15T13:06:05.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.663 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.664 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:05.664 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.371102+0000", 2024-05-15T13:06:05.664 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0543e-05 2024-05-15T13:06:05.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.664 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.664 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:05.664 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.371149+0000", 2024-05-15T13:06:05.664 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7304e-05 2024-05-15T13:06:05.664 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.664 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:05.664 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.664 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.664 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:17242 5.a 5:5f5524b5:::benchmark_data_smithi177_109991_object17241:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:06:05.664 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:26.280331+0000", 2024-05-15T13:06:05.665 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 159.03654500600001, 2024-05-15T13:06:05.665 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.093852726999999997, 2024-05-15T13:06:05.665 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:05.665 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:05.665 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:06:05.665 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:06:05.665 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:06:05.665 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 17242 2024-05-15T13:06:05.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.665 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:05.665 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.665 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:05.665 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280331+0000", 2024-05-15T13:06:05.666 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.666 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.666 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:05.666 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280331+0000", 2024-05-15T13:06:05.666 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.666 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.666 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:05.666 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280332+0000", 2024-05-15T13:06:05.666 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4e-07 2024-05-15T13:06:05.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.666 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.666 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:05.667 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280406+0000", 2024-05-15T13:06:05.667 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4497000000000002e-05 2024-05-15T13:06:05.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.667 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.667 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:05.667 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280416+0000", 2024-05-15T13:06:05.667 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3570000000000004e-06 2024-05-15T13:06:05.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.667 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.667 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:05.667 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280421+0000", 2024-05-15T13:06:05.668 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.66e-06 2024-05-15T13:06:05.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.668 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.668 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:05.668 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280471+0000", 2024-05-15T13:06:05.668 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0083000000000001e-05 2024-05-15T13:06:05.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.668 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.668 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:05.668 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280585+0000", 2024-05-15T13:06:05.668 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000114175 2024-05-15T13:06:05.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.669 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.669 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,10", 2024-05-15T13:06:05.669 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.371615+0000", 2024-05-15T13:06:05.669 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.091029502999999998 2024-05-15T13:06:05.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.669 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.669 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.669 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.373727+0000", 2024-05-15T13:06:05.669 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0021119440000000001 2024-05-15T13:06:05.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.669 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.669 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:06:05.670 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.373738+0000", 2024-05-15T13:06:05.670 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1596999999999999e-05 2024-05-15T13:06:05.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.670 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.670 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.670 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.374061+0000", 2024-05-15T13:06:05.670 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000322584 2024-05-15T13:06:05.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.670 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.670 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:05.670 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.374134+0000", 2024-05-15T13:06:05.670 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2997999999999995e-05 2024-05-15T13:06:05.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.671 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.671 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:05.671 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.374184+0000", 2024-05-15T13:06:05.671 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9489e-05 2024-05-15T13:06:05.671 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.671 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:05.671 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.671 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.671 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.15540.0:17231 5.6 e424/418 ondisk, result = 0)", 2024-05-15T13:06:05.672 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:26.280558+0000", 2024-05-15T13:06:05.672 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 159.03631775100001, 2024-05-15T13:06:05.672 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.090434608, 2024-05-15T13:06:05.672 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:05.672 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:06:05.672 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:05.672 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.672 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:05.672 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280558+0000", 2024-05-15T13:06:05.672 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.672 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.672 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:05.673 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280558+0000", 2024-05-15T13:06:05.673 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.673 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.673 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:05.673 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280559+0000", 2024-05-15T13:06:05.673 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4899999999999995e-07 2024-05-15T13:06:05.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.673 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.673 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:05.673 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280563+0000", 2024-05-15T13:06:05.674 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4469999999999999e-06 2024-05-15T13:06:05.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.674 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.674 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:05.674 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280573+0000", 2024-05-15T13:06:05.674 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8279999999999998e-06 2024-05-15T13:06:05.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.674 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.674 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:05.674 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280579+0000", 2024-05-15T13:06:05.674 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.392e-06 2024-05-15T13:06:05.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.675 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.675 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:05.675 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280619+0000", 2024-05-15T13:06:05.675 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9517000000000001e-05 2024-05-15T13:06:05.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.675 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.675 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:05.675 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280624+0000", 2024-05-15T13:06:05.675 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0309999999999998e-06 2024-05-15T13:06:05.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.675 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.676 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:05.676 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.370993+0000", 2024-05-15T13:06:05.676 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.090368744000000001 2024-05-15T13:06:05.676 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.676 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:05.676 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.676 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.676 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.676 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.15540.0:17234 5.6 e424/418 ondisk, result = 0)", 2024-05-15T13:06:05.676 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:26.280775+0000", 2024-05-15T13:06:05.676 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 159.036101089, 2024-05-15T13:06:05.676 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.090397524000000007, 2024-05-15T13:06:05.677 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:05.677 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:06:05.677 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:05.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.677 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:05.677 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280775+0000", 2024-05-15T13:06:05.677 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.677 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:05.677 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280775+0000", 2024-05-15T13:06:05.677 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.678 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:05.678 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280775+0000", 2024-05-15T13:06:05.678 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3200000000000005e-07 2024-05-15T13:06:05.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.678 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:05.678 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280779+0000", 2024-05-15T13:06:05.678 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5769999999999998e-06 2024-05-15T13:06:05.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.679 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:05.679 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.370780+0000", 2024-05-15T13:06:05.679 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.090000822999999994 2024-05-15T13:06:05.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.679 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.679 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:05.679 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.370793+0000", 2024-05-15T13:06:05.679 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.376e-05 2024-05-15T13:06:05.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.679 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.679 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:05.679 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.371037+0000", 2024-05-15T13:06:05.679 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000243382 2024-05-15T13:06:05.680 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.680 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.680 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:05.680 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.371045+0000", 2024-05-15T13:06:05.680 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6519999999999995e-06 2024-05-15T13:06:05.680 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.680 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.680 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:05.680 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.371172+0000", 2024-05-15T13:06:05.680 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012679799999999999 2024-05-15T13:06:05.680 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.680 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:05.681 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.681 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.681 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.681 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:19090 5.6 5:6e7c4261:::benchmark_data_smithi177_109991_object19089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:06:05.681 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:27.084229+0000", 2024-05-15T13:06:05.681 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 158.23264645, 2024-05-15T13:06:05.681 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.090169716999999996, 2024-05-15T13:06:05.681 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:05.681 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:05.681 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:06:05.681 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:06:05.681 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:06:05.682 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 19090 2024-05-15T13:06:05.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.682 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:05.682 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.682 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:05.682 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.084229+0000", 2024-05-15T13:06:05.682 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.682 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.682 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:05.682 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.084229+0000", 2024-05-15T13:06:05.683 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.683 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.683 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:05.683 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.084231+0000", 2024-05-15T13:06:05.683 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.471e-06 2024-05-15T13:06:05.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.683 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.683 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:05.683 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.084296+0000", 2024-05-15T13:06:05.683 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5009e-05 2024-05-15T13:06:05.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.684 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.684 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:05.684 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.084308+0000", 2024-05-15T13:06:05.684 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1809e-05 2024-05-15T13:06:05.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.684 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.684 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:05.684 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.084315+0000", 2024-05-15T13:06:05.684 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3300000000000001e-06 2024-05-15T13:06:05.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.684 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.685 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:05.685 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.084375+0000", 2024-05-15T13:06:05.685 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0340000000000002e-05 2024-05-15T13:06:05.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.685 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.685 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:05.685 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.084512+0000", 2024-05-15T13:06:05.685 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013636900000000001 2024-05-15T13:06:05.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.685 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.685 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,10", 2024-05-15T13:06:05.685 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.084657+0000", 2024-05-15T13:06:05.686 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014542 2024-05-15T13:06:05.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.686 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.686 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.686 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.086161+0000", 2024-05-15T13:06:05.686 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001504357 2024-05-15T13:06:05.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.686 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.686 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:06:05.686 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.087460+0000", 2024-05-15T13:06:05.686 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001298319 2024-05-15T13:06:05.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.687 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.687 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.687 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.174296+0000", 2024-05-15T13:06:05.687 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.086836216999999993 2024-05-15T13:06:05.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.687 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.687 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:05.687 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.174365+0000", 2024-05-15T13:06:05.687 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8515000000000004e-05 2024-05-15T13:06:05.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.687 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.687 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:05.688 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.174399+0000", 2024-05-15T13:06:05.688 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4561000000000003e-05 2024-05-15T13:06:05.688 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.688 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:05.688 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.688 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.688 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:20880 5.a 5:56abf9c6:::benchmark_data_smithi177_109991_object20879:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:06:05.688 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:27.874888+0000", 2024-05-15T13:06:05.688 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 157.441987907, 2024-05-15T13:06:05.689 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.080734568000000007, 2024-05-15T13:06:05.689 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:05.689 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:05.689 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:06:05.689 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:06:05.689 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:06:05.689 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 20880 2024-05-15T13:06:05.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.689 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:05.689 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.689 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:05.690 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.874888+0000", 2024-05-15T13:06:05.690 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.690 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.690 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:05.690 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.874888+0000", 2024-05-15T13:06:05.690 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.690 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.690 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:05.690 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.874889+0000", 2024-05-15T13:06:05.691 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2419999999999999e-06 2024-05-15T13:06:05.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.691 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.691 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:05.691 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.874956+0000", 2024-05-15T13:06:05.691 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7033999999999995e-05 2024-05-15T13:06:05.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.691 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.691 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:05.691 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.874967+0000", 2024-05-15T13:06:05.691 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0878e-05 2024-05-15T13:06:05.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.692 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:05.692 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.874993+0000", 2024-05-15T13:06:05.692 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5463999999999999e-05 2024-05-15T13:06:05.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.692 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:05.692 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.875056+0000", 2024-05-15T13:06:05.692 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3145999999999995e-05 2024-05-15T13:06:05.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.693 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:05.693 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.875186+0000", 2024-05-15T13:06:05.693 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000129927 2024-05-15T13:06:05.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.693 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.693 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,10", 2024-05-15T13:06:05.693 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.875328+0000", 2024-05-15T13:06:05.693 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014283899999999999 2024-05-15T13:06:05.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.693 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.693 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:06:05.693 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.876264+0000", 2024-05-15T13:06:05.694 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00093588600000000003 2024-05-15T13:06:05.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.694 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.694 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.694 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.878169+0000", 2024-05-15T13:06:05.694 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019042359999999999 2024-05-15T13:06:05.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.694 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.694 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.694 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.955559+0000", 2024-05-15T13:06:05.694 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.077390027 2024-05-15T13:06:05.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.695 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.695 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:05.695 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.955593+0000", 2024-05-15T13:06:05.695 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4446999999999997e-05 2024-05-15T13:06:05.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.695 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.695 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:05.695 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.955622+0000", 2024-05-15T13:06:05.695 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9442e-05 2024-05-15T13:06:05.695 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.695 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:05.696 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.696 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.696 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.696 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:20887 5.a 5:529f2edf:::benchmark_data_smithi177_109991_object20886:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:06:05.696 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:27.877575+0000", 2024-05-15T13:06:05.696 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 157.43930099799999, 2024-05-15T13:06:05.696 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.078160082000000006, 2024-05-15T13:06:05.696 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:05.696 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:05.696 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:06:05.696 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:06:05.696 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:06:05.697 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 20887 2024-05-15T13:06:05.697 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.697 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:05.697 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.697 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:05.697 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.877575+0000", 2024-05-15T13:06:05.697 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.697 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.697 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.697 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:05.697 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.877575+0000", 2024-05-15T13:06:05.697 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.698 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.698 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:05.698 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.877576+0000", 2024-05-15T13:06:05.698 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1960000000000001e-06 2024-05-15T13:06:05.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.698 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.698 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:05.698 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.877718+0000", 2024-05-15T13:06:05.698 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014181099999999999 2024-05-15T13:06:05.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.699 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.699 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:05.699 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.877731+0000", 2024-05-15T13:06:05.699 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3078000000000001e-05 2024-05-15T13:06:05.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.699 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.699 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:05.699 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.877738+0000", 2024-05-15T13:06:05.699 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.379e-06 2024-05-15T13:06:05.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.699 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.699 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:05.700 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.878214+0000", 2024-05-15T13:06:05.700 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00047557699999999998 2024-05-15T13:06:05.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.700 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.700 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:05.700 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.878341+0000", 2024-05-15T13:06:05.700 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012668999999999999 2024-05-15T13:06:05.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.700 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.700 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,10", 2024-05-15T13:06:05.700 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.878458+0000", 2024-05-15T13:06:05.700 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011706599999999999 2024-05-15T13:06:05.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.701 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.701 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.701 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.879897+0000", 2024-05-15T13:06:05.701 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001439259 2024-05-15T13:06:05.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.701 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.701 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:06:05.701 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.881324+0000", 2024-05-15T13:06:05.701 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001427411 2024-05-15T13:06:05.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.701 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.701 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.701 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.955674+0000", 2024-05-15T13:06:05.702 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.074349354000000006 2024-05-15T13:06:05.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.702 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.702 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:05.702 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.955706+0000", 2024-05-15T13:06:05.702 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2456000000000002e-05 2024-05-15T13:06:05.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.702 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.702 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:05.702 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.955735+0000", 2024-05-15T13:06:05.702 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8804999999999999e-05 2024-05-15T13:06:05.703 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.703 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:05.703 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.703 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.703 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:20889 5.6 5:63a806d2:::benchmark_data_smithi177_109991_object20888:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:06:05.703 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:27.878383+0000", 2024-05-15T13:06:05.703 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 157.43849313800001, 2024-05-15T13:06:05.703 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.076953523999999995, 2024-05-15T13:06:05.703 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:05.703 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:05.703 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:06:05.703 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:06:05.704 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:06:05.704 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 20889 2024-05-15T13:06:05.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.704 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:05.704 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.704 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:05.704 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.878383+0000", 2024-05-15T13:06:05.704 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.704 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.705 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:05.705 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.878383+0000", 2024-05-15T13:06:05.705 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.705 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.705 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:05.705 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.878383+0000", 2024-05-15T13:06:05.705 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.16e-07 2024-05-15T13:06:05.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.705 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.705 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:05.706 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.878403+0000", 2024-05-15T13:06:05.706 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9655000000000001e-05 2024-05-15T13:06:05.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.706 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.706 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:05.706 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.878412+0000", 2024-05-15T13:06:05.706 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2690000000000005e-06 2024-05-15T13:06:05.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.706 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.706 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:05.706 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.878418+0000", 2024-05-15T13:06:05.706 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6579999999999999e-06 2024-05-15T13:06:05.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.707 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.707 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:05.707 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.878470+0000", 2024-05-15T13:06:05.707 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2379000000000001e-05 2024-05-15T13:06:05.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.707 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.707 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:05.707 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.878605+0000", 2024-05-15T13:06:05.707 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000134917 2024-05-15T13:06:05.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.707 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.708 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,10", 2024-05-15T13:06:05.708 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.878748+0000", 2024-05-15T13:06:05.708 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014219499999999999 2024-05-15T13:06:05.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.708 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.708 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.708 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.881180+0000", 2024-05-15T13:06:05.708 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002432554 2024-05-15T13:06:05.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.708 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.708 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:06:05.708 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.881218+0000", 2024-05-15T13:06:05.708 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7537000000000003e-05 2024-05-15T13:06:05.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.709 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.709 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.709 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.955219+0000", 2024-05-15T13:06:05.709 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.074001258 2024-05-15T13:06:05.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.709 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.709 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:05.709 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.955281+0000", 2024-05-15T13:06:05.709 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2417999999999998e-05 2024-05-15T13:06:05.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.709 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.709 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:05.710 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.955336+0000", 2024-05-15T13:06:05.710 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4867999999999997e-05 2024-05-15T13:06:05.710 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.710 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:05.710 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.710 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.710 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.710 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:20896 5.a 5:5071910b:::benchmark_data_smithi177_109991_object20895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:06:05.710 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:27.879997+0000", 2024-05-15T13:06:05.710 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 157.43687847999999, 2024-05-15T13:06:05.710 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.075883105000000006, 2024-05-15T13:06:05.710 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:05.711 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:05.711 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:06:05.711 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:06:05.711 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:06:05.711 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 20896 2024-05-15T13:06:05.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.711 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:05.711 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.711 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:05.711 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.879997+0000", 2024-05-15T13:06:05.711 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.712 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.712 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:05.712 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.879997+0000", 2024-05-15T13:06:05.712 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.712 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.712 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:05.712 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.879998+0000", 2024-05-15T13:06:05.712 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.125e-06 2024-05-15T13:06:05.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.712 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.712 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:05.713 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.880098+0000", 2024-05-15T13:06:05.713 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9130999999999997e-05 2024-05-15T13:06:05.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.713 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.713 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:05.713 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.880133+0000", 2024-05-15T13:06:05.713 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5620000000000001e-05 2024-05-15T13:06:05.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.713 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.713 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:05.713 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.880142+0000", 2024-05-15T13:06:05.714 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3750000000000003e-06 2024-05-15T13:06:05.714 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.714 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.714 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:05.714 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.880199+0000", 2024-05-15T13:06:05.714 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7015000000000002e-05 2024-05-15T13:06:05.714 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.714 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.714 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:05.714 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.880328+0000", 2024-05-15T13:06:05.714 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012898099999999999 2024-05-15T13:06:05.714 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.715 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.715 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,10", 2024-05-15T13:06:05.715 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.880472+0000", 2024-05-15T13:06:05.715 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000144795 2024-05-15T13:06:05.715 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.715 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.715 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:06:05.715 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.881543+0000", 2024-05-15T13:06:05.715 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010708810000000001 2024-05-15T13:06:05.715 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.715 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.715 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.716 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.881790+0000", 2024-05-15T13:06:05.716 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000246708 2024-05-15T13:06:05.716 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.716 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.716 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.716 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.955781+0000", 2024-05-15T13:06:05.716 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.073990768999999998 2024-05-15T13:06:05.716 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.716 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.716 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:05.716 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.955841+0000", 2024-05-15T13:06:05.716 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9827999999999997e-05 2024-05-15T13:06:05.717 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.717 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.717 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:05.717 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.955880+0000", 2024-05-15T13:06:05.717 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9876999999999998e-05 2024-05-15T13:06:05.717 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.717 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:05.717 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.717 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.717 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.717 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:20905 5.a 5:5ea86e0b:::benchmark_data_smithi177_109991_object20904:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:06:05.717 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:27.883848+0000", 2024-05-15T13:06:05.718 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 157.43302830499999, 2024-05-15T13:06:05.718 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.074158858999999994, 2024-05-15T13:06:05.718 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:05.718 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:05.718 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:06:05.718 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:06:05.718 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:06:05.718 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 20905 2024-05-15T13:06:05.718 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.718 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:05.719 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.719 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:05.719 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.883848+0000", 2024-05-15T13:06:05.719 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.719 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.719 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.719 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:05.719 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.883848+0000", 2024-05-15T13:06:05.719 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.719 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.719 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.719 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:05.720 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.883848+0000", 2024-05-15T13:06:05.720 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7699999999999992e-07 2024-05-15T13:06:05.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.720 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.720 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:05.720 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.883940+0000", 2024-05-15T13:06:05.720 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1566999999999999e-05 2024-05-15T13:06:05.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.720 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.720 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:05.720 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.883954+0000", 2024-05-15T13:06:05.720 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4066999999999999e-05 2024-05-15T13:06:05.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.721 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.721 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:05.721 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.883962+0000", 2024-05-15T13:06:05.721 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9119999999999998e-06 2024-05-15T13:06:05.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.721 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.721 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:05.721 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.884017+0000", 2024-05-15T13:06:05.721 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.482e-05 2024-05-15T13:06:05.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.722 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.722 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:05.722 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.884166+0000", 2024-05-15T13:06:05.722 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000148894 2024-05-15T13:06:05.722 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.722 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.722 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,10", 2024-05-15T13:06:05.722 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.884274+0000", 2024-05-15T13:06:05.722 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010799800000000001 2024-05-15T13:06:05.722 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.722 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.722 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:06:05.723 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.885131+0000", 2024-05-15T13:06:05.723 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00085765799999999997 2024-05-15T13:06:05.723 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.723 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.723 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.723 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.885353+0000", 2024-05-15T13:06:05.723 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00022131200000000001 2024-05-15T13:06:05.723 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.723 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.723 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.723 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.957901+0000", 2024-05-15T13:06:05.723 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.072548084999999998 2024-05-15T13:06:05.724 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.724 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.724 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:05.724 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.957960+0000", 2024-05-15T13:06:05.724 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9679000000000002e-05 2024-05-15T13:06:05.724 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.724 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.724 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:05.724 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.958006+0000", 2024-05-15T13:06:05.724 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5890000000000003e-05 2024-05-15T13:06:05.724 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.724 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:05.725 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.725 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.725 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:20908 5.6 5:61a826ed:::benchmark_data_smithi177_109991_object20907:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:06:05.725 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:27.885969+0000", 2024-05-15T13:06:05.725 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 157.430906349, 2024-05-15T13:06:05.725 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.069468366000000004, 2024-05-15T13:06:05.725 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:05.725 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:05.725 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:06:05.725 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:06:05.725 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:06:05.725 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 20908 2024-05-15T13:06:05.726 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.726 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:05.726 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.726 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:05.726 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.885969+0000", 2024-05-15T13:06:05.726 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.726 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.727 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.727 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:05.727 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.885969+0000", 2024-05-15T13:06:05.727 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.727 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.727 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.727 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:05.727 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.885970+0000", 2024-05-15T13:06:05.727 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9999999999999996e-07 2024-05-15T13:06:05.727 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.727 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.727 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:05.728 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.886074+0000", 2024-05-15T13:06:05.728 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000103315 2024-05-15T13:06:05.728 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.728 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.728 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:05.728 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.886089+0000", 2024-05-15T13:06:05.728 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5004999999999999e-05 2024-05-15T13:06:05.728 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.728 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.728 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:05.728 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.886094+0000", 2024-05-15T13:06:05.728 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6860000000000003e-06 2024-05-15T13:06:05.729 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.729 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.729 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:05.729 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.886157+0000", 2024-05-15T13:06:05.729 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2902000000000002e-05 2024-05-15T13:06:05.729 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.729 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.729 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:05.729 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.886316+0000", 2024-05-15T13:06:05.730 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015842800000000001 2024-05-15T13:06:05.730 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.730 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.730 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,10", 2024-05-15T13:06:05.730 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.886479+0000", 2024-05-15T13:06:05.730 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016364099999999999 2024-05-15T13:06:05.730 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.730 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.730 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:06:05.730 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.887282+0000", 2024-05-15T13:06:05.731 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00080301299999999999 2024-05-15T13:06:05.731 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.731 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.731 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.731 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.887581+0000", 2024-05-15T13:06:05.731 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00029853600000000002 2024-05-15T13:06:05.731 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.731 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.731 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.731 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.955375+0000", 2024-05-15T13:06:05.732 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.067794120999999999 2024-05-15T13:06:05.732 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.732 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.732 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:05.732 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.955411+0000", 2024-05-15T13:06:05.732 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5734e-05 2024-05-15T13:06:05.732 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.732 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.732 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:05.732 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.955438+0000", 2024-05-15T13:06:05.732 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7185000000000001e-05 2024-05-15T13:06:05.733 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.733 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:05.733 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.733 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.733 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.733 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:23458 5.6 5:60320e8d:::benchmark_data_smithi177_109991_object23457:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:06:05.733 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:29.026615+0000", 2024-05-15T13:06:05.733 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 156.29026046600001, 2024-05-15T13:06:05.733 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.077272093, 2024-05-15T13:06:05.733 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:05.733 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:05.734 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:06:05.734 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:06:05.734 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:06:05.734 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 23458 2024-05-15T13:06:05.734 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.734 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:05.734 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.734 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:05.734 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.026615+0000", 2024-05-15T13:06:05.734 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.734 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.735 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.735 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:05.735 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.026615+0000", 2024-05-15T13:06:05.735 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.735 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.735 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.735 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:05.735 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.026616+0000", 2024-05-15T13:06:05.735 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5499999999999999e-07 2024-05-15T13:06:05.735 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.735 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.736 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:05.736 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.026636+0000", 2024-05-15T13:06:05.736 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0038999999999999e-05 2024-05-15T13:06:05.736 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.736 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.736 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:05.736 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.026640+0000", 2024-05-15T13:06:05.736 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5859999999999998e-06 2024-05-15T13:06:05.736 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.736 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.736 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:05.736 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.026644+0000", 2024-05-15T13:06:05.737 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.985e-06 2024-05-15T13:06:05.737 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.737 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.737 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:05.737 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.026700+0000", 2024-05-15T13:06:05.737 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5954e-05 2024-05-15T13:06:05.737 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.737 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.737 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:05.737 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.026832+0000", 2024-05-15T13:06:05.737 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013141699999999999 2024-05-15T13:06:05.737 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.738 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.738 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,10", 2024-05-15T13:06:05.738 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.026978+0000", 2024-05-15T13:06:05.738 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000146117 2024-05-15T13:06:05.738 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.738 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.738 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:06:05.738 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.029045+0000", 2024-05-15T13:06:05.738 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020671729999999998 2024-05-15T13:06:05.738 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.738 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.739 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.739 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.030593+0000", 2024-05-15T13:06:05.739 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001547742 2024-05-15T13:06:05.739 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.739 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.739 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.739 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.103776+0000", 2024-05-15T13:06:05.739 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.073183317999999997 2024-05-15T13:06:05.739 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.739 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.739 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:05.739 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.103840+0000", 2024-05-15T13:06:05.739 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4053999999999994e-05 2024-05-15T13:06:05.740 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.740 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.740 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:05.740 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.103887+0000", 2024-05-15T13:06:05.740 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7352999999999999e-05 2024-05-15T13:06:05.740 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.740 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:05.740 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.740 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.740 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.740 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:23469 5.6 5:62180c3d:::benchmark_data_smithi177_109991_object23468:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:06:05.740 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:29.033145+0000", 2024-05-15T13:06:05.740 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 156.28373125900001, 2024-05-15T13:06:05.741 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.070897488999999994, 2024-05-15T13:06:05.741 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:05.741 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:05.741 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:06:05.741 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:06:05.741 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:06:05.741 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 23469 2024-05-15T13:06:05.741 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.741 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:05.741 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.741 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:05.741 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.033145+0000", 2024-05-15T13:06:05.741 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.742 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.742 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.742 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:05.742 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.033145+0000", 2024-05-15T13:06:05.742 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.742 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.742 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.742 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:05.742 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.033146+0000", 2024-05-15T13:06:05.742 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3149999999999999e-06 2024-05-15T13:06:05.742 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.742 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.743 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:05.743 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.033181+0000", 2024-05-15T13:06:05.743 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5493000000000001e-05 2024-05-15T13:06:05.743 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.743 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.743 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:05.743 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.033198+0000", 2024-05-15T13:06:05.743 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6605000000000001e-05 2024-05-15T13:06:05.743 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.743 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.743 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:05.744 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.033207+0000", 2024-05-15T13:06:05.744 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9590000000000001e-06 2024-05-15T13:06:05.744 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.744 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.744 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:05.744 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.033259+0000", 2024-05-15T13:06:05.744 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1919999999999998e-05 2024-05-15T13:06:05.744 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.744 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.744 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:05.744 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.033385+0000", 2024-05-15T13:06:05.744 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012605799999999999 2024-05-15T13:06:05.744 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.745 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.745 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,10", 2024-05-15T13:06:05.745 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.033566+0000", 2024-05-15T13:06:05.745 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000180719 2024-05-15T13:06:05.745 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.745 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.745 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:06:05.745 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.034746+0000", 2024-05-15T13:06:05.745 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0011801540000000001 2024-05-15T13:06:05.745 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.745 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.746 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.746 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.034940+0000", 2024-05-15T13:06:05.746 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000194666 2024-05-15T13:06:05.746 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.746 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.746 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.746 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.103944+0000", 2024-05-15T13:06:05.746 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.069003661999999993 2024-05-15T13:06:05.746 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.746 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.747 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:05.747 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.104003+0000", 2024-05-15T13:06:05.747 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9058000000000003e-05 2024-05-15T13:06:05.747 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.747 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.747 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:05.747 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:29.104042+0000", 2024-05-15T13:06:05.747 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.888e-05 2024-05-15T13:06:05.747 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.747 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:05.747 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.748 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.748 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.748 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:31155 5.6 5:6e92a747:::benchmark_data_smithi177_109991_object31154:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-05-15T13:06:05.748 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:31.996055+0000", 2024-05-15T13:06:05.748 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 153.320820615, 2024-05-15T13:06:05.748 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.096537703000000002, 2024-05-15T13:06:05.748 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:05.748 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:05.748 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:06:05.748 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:06:05.748 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:06:05.749 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 31155 2024-05-15T13:06:05.749 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.749 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:05.749 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.749 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:05.749 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:31.996055+0000", 2024-05-15T13:06:05.749 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.749 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.749 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.749 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:05.749 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:31.996055+0000", 2024-05-15T13:06:05.750 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.750 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.750 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.750 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:05.750 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:31.996057+0000", 2024-05-15T13:06:05.750 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8419999999999999e-06 2024-05-15T13:06:05.750 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.750 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.750 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:05.750 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:31.996407+0000", 2024-05-15T13:06:05.751 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000349907 2024-05-15T13:06:05.751 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.751 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.751 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:05.751 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:31.996427+0000", 2024-05-15T13:06:05.751 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0168000000000001e-05 2024-05-15T13:06:05.751 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.751 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.751 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:05.751 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:31.996437+0000", 2024-05-15T13:06:05.751 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6800000000000005e-06 2024-05-15T13:06:05.752 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.752 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.752 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:05.752 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:31.996505+0000", 2024-05-15T13:06:05.752 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8137000000000002e-05 2024-05-15T13:06:05.752 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.752 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.752 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:05.752 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:31.996637+0000", 2024-05-15T13:06:05.752 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013156299999999999 2024-05-15T13:06:05.752 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.753 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.753 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,10", 2024-05-15T13:06:05.753 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:31.996776+0000", 2024-05-15T13:06:05.753 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013938700000000001 2024-05-15T13:06:05.753 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.753 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.753 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:06:05.753 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:31.998297+0000", 2024-05-15T13:06:05.753 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001521281 2024-05-15T13:06:05.753 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.753 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.754 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.754 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:31.998483+0000", 2024-05-15T13:06:05.754 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00018571399999999999 2024-05-15T13:06:05.754 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.754 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.754 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.754 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:32.092457+0000", 2024-05-15T13:06:05.754 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.093974577000000004 2024-05-15T13:06:05.754 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.754 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.754 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:05.755 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:32.092548+0000", 2024-05-15T13:06:05.755 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0687999999999996e-05 2024-05-15T13:06:05.755 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.755 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.755 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:05.755 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:32.092593+0000", 2024-05-15T13:06:05.755 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4759000000000003e-05 2024-05-15T13:06:05.755 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.755 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:05.755 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.755 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.756 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.756 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:72592 5.6 5:69cebc8e:::benchmark_data_smithi177_109991_object72591:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-05-15T13:06:05.756 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:55.537670+0000", 2024-05-15T13:06:05.756 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 129.77920539199999, 2024-05-15T13:06:05.756 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.078217196000000003, 2024-05-15T13:06:05.756 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:05.756 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:05.756 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:06:05.756 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:06:05.756 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:06:05.756 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 72592 2024-05-15T13:06:05.756 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.757 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:05.757 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.757 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:05.757 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:55.537670+0000", 2024-05-15T13:06:05.757 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.757 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.757 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.757 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:05.757 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:55.537670+0000", 2024-05-15T13:06:05.757 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.758 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.758 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.758 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:05.758 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:55.537672+0000", 2024-05-15T13:06:05.758 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.486e-06 2024-05-15T13:06:05.758 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.758 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.758 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:05.758 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:55.537729+0000", 2024-05-15T13:06:05.758 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7339000000000001e-05 2024-05-15T13:06:05.758 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.759 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.759 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:05.759 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:55.537741+0000", 2024-05-15T13:06:05.759 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2227e-05 2024-05-15T13:06:05.759 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.759 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.759 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:05.759 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:55.537748+0000", 2024-05-15T13:06:05.759 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.455e-06 2024-05-15T13:06:05.759 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.759 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.759 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:05.759 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:55.537819+0000", 2024-05-15T13:06:05.760 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1433000000000005e-05 2024-05-15T13:06:05.760 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.760 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.760 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:05.760 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:55.537984+0000", 2024-05-15T13:06:05.760 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000164926 2024-05-15T13:06:05.760 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.760 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.760 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,10", 2024-05-15T13:06:05.760 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:55.538172+0000", 2024-05-15T13:06:05.760 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000188137 2024-05-15T13:06:05.760 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.761 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.761 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:06:05.761 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:55.540952+0000", 2024-05-15T13:06:05.761 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0027794930000000001 2024-05-15T13:06:05.761 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.761 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.761 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.761 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:55.541218+0000", 2024-05-15T13:06:05.761 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00026625900000000003 2024-05-15T13:06:05.761 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.761 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.761 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.762 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:55.615806+0000", 2024-05-15T13:06:05.762 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.074587340000000002 2024-05-15T13:06:05.762 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.762 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.762 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:05.762 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:55.615851+0000", 2024-05-15T13:06:05.762 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5278999999999998e-05 2024-05-15T13:06:05.762 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.762 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.762 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:05.762 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:55.615888+0000", 2024-05-15T13:06:05.763 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6822000000000001e-05 2024-05-15T13:06:05.763 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.763 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:05.763 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.763 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.763 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.763 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:104770 5.a 5:5190c780:::benchmark_data_smithi177_109991_object104769:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-05-15T13:06:05.763 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:04:20.656964+0000", 2024-05-15T13:06:05.763 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 104.659912017, 2024-05-15T13:06:05.763 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.071740845999999997, 2024-05-15T13:06:05.763 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:05.763 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:05.764 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:06:05.764 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:06:05.764 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:06:05.764 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 104770 2024-05-15T13:06:05.764 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.764 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:05.764 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.764 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:05.764 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.656964+0000", 2024-05-15T13:06:05.764 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.765 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.765 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.765 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:05.765 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.656964+0000", 2024-05-15T13:06:05.765 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.765 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.765 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.765 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:05.765 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.656964+0000", 2024-05-15T13:06:05.765 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4600000000000005e-07 2024-05-15T13:06:05.765 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.765 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.766 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:05.766 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.656995+0000", 2024-05-15T13:06:05.766 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0691999999999997e-05 2024-05-15T13:06:05.766 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.766 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.766 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:05.766 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.657003+0000", 2024-05-15T13:06:05.766 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1130000000000001e-06 2024-05-15T13:06:05.766 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.766 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.766 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:05.767 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.657010+0000", 2024-05-15T13:06:05.767 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0790000000000003e-06 2024-05-15T13:06:05.767 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.767 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.767 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:05.767 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.657064+0000", 2024-05-15T13:06:05.767 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3402999999999997e-05 2024-05-15T13:06:05.767 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.767 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.767 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:05.767 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.657222+0000", 2024-05-15T13:06:05.767 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015835299999999999 2024-05-15T13:06:05.768 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.768 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.768 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,10", 2024-05-15T13:06:05.768 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.657377+0000", 2024-05-15T13:06:05.768 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000155449 2024-05-15T13:06:05.768 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.768 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.768 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:06:05.768 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.660710+0000", 2024-05-15T13:06:05.768 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033329520000000001 2024-05-15T13:06:05.768 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.768 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.769 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.769 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.728548+0000", 2024-05-15T13:06:05.769 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.067837880000000003 2024-05-15T13:06:05.769 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.769 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.769 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.769 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.728588+0000", 2024-05-15T13:06:05.769 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9981000000000002e-05 2024-05-15T13:06:05.769 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.769 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.769 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:05.769 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.728645+0000", 2024-05-15T13:06:05.770 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6725999999999999e-05 2024-05-15T13:06:05.770 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.770 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.770 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:05.770 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.728705+0000", 2024-05-15T13:06:05.770 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9672000000000001e-05 2024-05-15T13:06:05.770 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.770 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:05.770 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.770 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.770 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.771 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.15540.0:104770 5.a e442/418 ondisk, result = 0)", 2024-05-15T13:06:05.771 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:04:20.663198+0000", 2024-05-15T13:06:05.771 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 104.65367799800001, 2024-05-15T13:06:05.771 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.065357837000000002, 2024-05-15T13:06:05.771 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:05.771 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:06:05.771 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:05.771 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.771 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:05.771 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.663198+0000", 2024-05-15T13:06:05.771 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.771 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.772 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.772 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:05.772 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.663198+0000", 2024-05-15T13:06:05.772 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.772 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.772 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.772 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:05.772 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.663199+0000", 2024-05-15T13:06:05.772 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4600000000000003e-07 2024-05-15T13:06:05.772 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.772 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.773 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:05.773 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.663205+0000", 2024-05-15T13:06:05.773 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5100000000000004e-06 2024-05-15T13:06:05.773 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.773 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.773 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:05.773 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.663216+0000", 2024-05-15T13:06:05.773 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0712e-05 2024-05-15T13:06:05.773 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.773 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.773 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:05.773 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.663225+0000", 2024-05-15T13:06:05.774 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9959999999999999e-06 2024-05-15T13:06:05.774 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.774 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.774 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:05.774 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.728535+0000", 2024-05-15T13:06:05.774 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.065309876000000003 2024-05-15T13:06:05.774 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.774 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.774 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:05.774 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.728542+0000", 2024-05-15T13:06:05.774 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6569999999999997e-06 2024-05-15T13:06:05.775 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.775 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.775 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:05.775 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.728556+0000", 2024-05-15T13:06:05.775 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.324e-05 2024-05-15T13:06:05.775 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.775 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:05.775 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.775 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.775 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.775 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.15540.0:104783 5.4 e442/418 5:2abd1f52:::benchmark_data_smithi177_109991_object104782:head v 442'6594, mlcod=442'6592)", 2024-05-15T13:06:05.775 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:04:20.674786+0000", 2024-05-15T13:06:05.775 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 104.64208949099999, 2024-05-15T13:06:05.776 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.063273033000000006, 2024-05-15T13:06:05.776 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:05.776 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:05.776 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:05.776 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.776 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:05.776 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.674786+0000", 2024-05-15T13:06:05.776 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.776 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.776 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.776 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:05.777 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.674786+0000", 2024-05-15T13:06:05.777 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.777 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.777 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.777 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:05.777 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.674787+0000", 2024-05-15T13:06:05.777 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0330000000000001e-06 2024-05-15T13:06:05.777 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.777 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.777 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:05.777 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.674830+0000", 2024-05-15T13:06:05.777 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2925e-05 2024-05-15T13:06:05.777 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.778 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.778 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:05.778 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.727919+0000", 2024-05-15T13:06:05.778 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.053088528000000003 2024-05-15T13:06:05.778 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.778 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.778 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:05.778 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.727934+0000", 2024-05-15T13:06:05.778 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5659e-05 2024-05-15T13:06:05.778 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.778 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.779 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:05.779 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.728410+0000", 2024-05-15T13:06:05.779 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00047527799999999999 2024-05-15T13:06:05.779 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.779 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.779 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:05.779 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.728429+0000", 2024-05-15T13:06:05.779 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9000000000000001e-05 2024-05-15T13:06:05.779 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.779 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.779 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:05.780 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.737976+0000", 2024-05-15T13:06:05.780 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0095475950000000007 2024-05-15T13:06:05.780 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.780 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.780 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:05.780 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.738059+0000", 2024-05-15T13:06:05.780 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3015000000000004e-05 2024-05-15T13:06:05.780 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.780 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:05.780 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.780 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.780 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.780 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:104790 5.a 5:5e18630e:::benchmark_data_smithi177_109991_object104789:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-05-15T13:06:05.781 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:04:20.675629+0000", 2024-05-15T13:06:05.781 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 104.641247185, 2024-05-15T13:06:05.781 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.070053560000000001, 2024-05-15T13:06:05.781 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:05.781 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:05.781 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:06:05.781 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:06:05.781 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:06:05.781 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 104790 2024-05-15T13:06:05.781 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.781 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:05.781 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.781 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:05.782 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.675629+0000", 2024-05-15T13:06:05.782 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.782 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.782 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.782 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:05.782 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.675629+0000", 2024-05-15T13:06:05.782 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:05.782 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.782 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.782 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:05.782 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.675630+0000", 2024-05-15T13:06:05.782 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6929999999999999e-06 2024-05-15T13:06:05.783 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.783 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.783 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:05.783 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.675669+0000", 2024-05-15T13:06:05.783 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8869999999999999e-05 2024-05-15T13:06:05.783 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.783 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.783 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:05.783 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.727982+0000", 2024-05-15T13:06:05.783 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.052312874000000002 2024-05-15T13:06:05.783 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.783 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.783 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:05.784 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.727997+0000", 2024-05-15T13:06:05.784 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5142e-05 2024-05-15T13:06:05.784 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.784 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.784 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:05.784 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.728735+0000", 2024-05-15T13:06:05.784 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00073777999999999995 2024-05-15T13:06:05.784 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.784 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.784 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:05.784 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.728867+0000", 2024-05-15T13:06:05.784 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013232599999999999 2024-05-15T13:06:05.785 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.785 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.785 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,10", 2024-05-15T13:06:05.785 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.729015+0000", 2024-05-15T13:06:05.785 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014746799999999999 2024-05-15T13:06:05.785 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.785 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.785 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:06:05.785 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.738966+0000", 2024-05-15T13:06:05.785 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0099514849999999995 2024-05-15T13:06:05.785 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.785 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.786 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.786 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.741255+0000", 2024-05-15T13:06:05.786 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0022884379999999998 2024-05-15T13:06:05.786 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.786 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.786 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:05.786 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.745568+0000", 2024-05-15T13:06:05.786 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0043128239999999998 2024-05-15T13:06:05.786 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.786 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.786 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:05.786 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.745618+0000", 2024-05-15T13:06:05.787 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0148999999999997e-05 2024-05-15T13:06:05.787 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:05.787 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:05.787 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:05.787 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:20.745682+0000", 2024-05-15T13:06:05.787 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4511000000000006e-05 2024-05-15T13:06:05.787 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.787 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:05.787 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.787 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:05.787 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:05.787 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:06:05.788 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:05 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:06:05.788 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:05 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:06:05.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:05 smithi120 ceph-mon[43961]: osdmap e467: 12 total, 12 up, 11 in 2024-05-15T13:06:05.789 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:05 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:06:05.789 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:05 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:06:05.789 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:05 smithi177 ceph-mon[44303]: osdmap e467: 12 total, 12 up, 11 in 2024-05-15T13:06:05.942 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:06:05.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:05.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:06:06.055 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:06:06.098 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:06.129 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:06:06.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:06.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:06:06.234 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:06:06.286 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:06.328 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:06:06.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:06.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:06:06.406 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:06:06.434 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:06:06.458 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-05-15T13:06:06.491 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:06.534 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:06:06.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:06.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:06:06.641 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:06:06.696 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:06 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:06:06.696 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:06 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:06:06.696 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:06 smithi027 ceph-mon[39795]: osdmap e468: 12 total, 12 up, 11 in 2024-05-15T13:06:06.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:06 smithi027 ceph-mon[39795]: 5.1 scrub starts 2024-05-15T13:06:06.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:06 smithi027 ceph-mon[39795]: pgmap v1261: 17 pgs: 17 active+clean; 4.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 865 op/s 2024-05-15T13:06:06.697 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:06 smithi027 ceph-mon[39795]: 5.7 scrub starts 2024-05-15T13:06:06.697 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:06.744 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:06:06.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:06.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:06:06.802 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:06 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:06:06.802 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:06 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:06:06.802 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:06 smithi120 ceph-mon[43961]: osdmap e468: 12 total, 12 up, 11 in 2024-05-15T13:06:06.803 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:06 smithi120 ceph-mon[43961]: 5.1 scrub starts 2024-05-15T13:06:06.803 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:06 smithi120 ceph-mon[43961]: pgmap v1261: 17 pgs: 17 active+clean; 4.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 865 op/s 2024-05-15T13:06:06.803 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:06 smithi120 ceph-mon[43961]: 5.7 scrub starts 2024-05-15T13:06:06.902 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:06:06.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:06 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:06:06.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:06 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:06:06.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:06 smithi177 ceph-mon[44303]: osdmap e468: 12 total, 12 up, 11 in 2024-05-15T13:06:06.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:06 smithi177 ceph-mon[44303]: 5.1 scrub starts 2024-05-15T13:06:06.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:06 smithi177 ceph-mon[44303]: pgmap v1261: 17 pgs: 17 active+clean; 4.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 865 op/s 2024-05-15T13:06:06.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:06 smithi177 ceph-mon[44303]: 5.7 scrub starts 2024-05-15T13:06:06.963 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:07.006 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:06:07.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:07.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:06:07.111 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:06:07.148 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:07.179 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:06:07.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:07.182 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:06:07.283 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:06:07.326 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:07.360 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:06:07.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:07.365 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:06:07.466 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:06:07.505 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:07.539 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:06:07.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:07.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:06:07.553 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:06:07.553 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:06:07.554 INFO:tasks.thrashosds.thrasher:inject_pause on osd.8 2024-05-15T13:06:07.554 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-15T13:06:07.554 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-15T13:06:07.554 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 config set filestore_inject_stall 3 2024-05-15T13:06:07.602 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:06:07.650 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:06:07.715 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:07 smithi027 ceph-mon[39795]: 5.1 scrub ok 2024-05-15T13:06:07.725 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:07.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:07 smithi120 ceph-mon[43961]: 5.1 scrub ok 2024-05-15T13:06:07.795 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:06:07.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:07.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:06:07.902 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:06:07.913 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:06:07.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:07 smithi177 ceph-mon[44303]: 5.1 scrub ok 2024-05-15T13:06:07.949 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:07.955 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:06:07.979 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:06:07.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:07.986 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:06:08.088 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:06:08.136 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:08.258 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:06:08.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:08.264 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:06:08.365 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:06:08.412 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:08.442 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:06:08.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:08.447 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:06:08.548 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:06:08.603 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:08 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T13:06:08.603 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:08 smithi177 ceph-mon[44303]: 5.8 scrub starts 2024-05-15T13:06:08.603 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:08 smithi177 ceph-mon[44303]: 5.7 scrub ok 2024-05-15T13:06:08.603 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:08 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T13:06:08.604 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:08 smithi177 ceph-mon[44303]: 5.5 scrub starts 2024-05-15T13:06:08.604 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:08 smithi177 ceph-mon[44303]: pgmap v1262: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 925 op/s 2024-05-15T13:06:08.625 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:08.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:08 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T13:06:08.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:08 smithi027 ceph-mon[39795]: 5.8 scrub starts 2024-05-15T13:06:08.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:08 smithi027 ceph-mon[39795]: 5.7 scrub ok 2024-05-15T13:06:08.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:08 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T13:06:08.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:08 smithi027 ceph-mon[39795]: 5.5 scrub starts 2024-05-15T13:06:08.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:08 smithi027 ceph-mon[39795]: pgmap v1262: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 925 op/s 2024-05-15T13:06:08.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:08 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T13:06:08.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:08 smithi120 ceph-mon[43961]: 5.8 scrub starts 2024-05-15T13:06:08.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:08 smithi120 ceph-mon[43961]: 5.7 scrub ok 2024-05-15T13:06:08.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:08 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T13:06:08.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:08 smithi120 ceph-mon[43961]: 5.5 scrub starts 2024-05-15T13:06:08.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:08 smithi120 ceph-mon[43961]: pgmap v1262: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 925 op/s 2024-05-15T13:06:09.105 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:06:09.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:09.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:06:09.213 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:06:09.263 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:09.299 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:06:09.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:09.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:06:09.404 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:06:09.449 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:09.473 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:09 smithi027 ceph-mon[39795]: 5.9 scrub starts 2024-05-15T13:06:09.473 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:09 smithi027 ceph-mon[39795]: 5.6 scrub starts 2024-05-15T13:06:09.473 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:09 smithi027 ceph-mon[39795]: 5.e scrub starts 2024-05-15T13:06:09.473 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:09 smithi027 ceph-mon[39795]: 5.c scrub starts 2024-05-15T13:06:09.473 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:09 smithi027 ceph-mon[39795]: 5.9 scrub ok 2024-05-15T13:06:09.473 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:09 smithi027 ceph-mon[39795]: 5.5 scrub ok 2024-05-15T13:06:09.484 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:06:09.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:09.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:06:09.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:09 smithi120 ceph-mon[43961]: 5.9 scrub starts 2024-05-15T13:06:09.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:09 smithi120 ceph-mon[43961]: 5.6 scrub starts 2024-05-15T13:06:09.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:09 smithi120 ceph-mon[43961]: 5.e scrub starts 2024-05-15T13:06:09.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:09 smithi120 ceph-mon[43961]: 5.c scrub starts 2024-05-15T13:06:09.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:09 smithi120 ceph-mon[43961]: 5.9 scrub ok 2024-05-15T13:06:09.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:09 smithi120 ceph-mon[43961]: 5.5 scrub ok 2024-05-15T13:06:09.591 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:06:09.636 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:09.803 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:06:09.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:09.809 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:06:09.809 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:09 smithi177 ceph-mon[44303]: 5.9 scrub starts 2024-05-15T13:06:09.809 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:09 smithi177 ceph-mon[44303]: 5.6 scrub starts 2024-05-15T13:06:09.809 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:09 smithi177 ceph-mon[44303]: 5.e scrub starts 2024-05-15T13:06:09.809 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:09 smithi177 ceph-mon[44303]: 5.c scrub starts 2024-05-15T13:06:09.809 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:09 smithi177 ceph-mon[44303]: 5.9 scrub ok 2024-05-15T13:06:09.809 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:09 smithi177 ceph-mon[44303]: 5.5 scrub ok 2024-05-15T13:06:09.909 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:06:09.958 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:09.985 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:06:09.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:09.990 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:06:10.091 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:06:10.136 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:10.165 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:06:10.165 INFO:teuthology.orchestra.run.smithi177.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-05-15T13:06:10.165 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:06:10.167 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:06:10.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:10.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:06:10.272 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:06:10.324 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:10.739 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:06:10.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:10 smithi027 ceph-mon[39795]: 5.8 scrub ok 2024-05-15T13:06:10.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:10 smithi027 ceph-mon[39795]: 5.6 scrub ok 2024-05-15T13:06:10.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:10 smithi027 ceph-mon[39795]: 5.e scrub ok 2024-05-15T13:06:10.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:10 smithi027 ceph-mon[39795]: pgmap v1263: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 977 op/s 2024-05-15T13:06:10.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:10 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:06:10.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:10 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:06:10.780 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:06:10.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:10 smithi120 ceph-mon[43961]: 5.8 scrub ok 2024-05-15T13:06:10.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:10 smithi120 ceph-mon[43961]: 5.6 scrub ok 2024-05-15T13:06:10.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:10 smithi120 ceph-mon[43961]: 5.e scrub ok 2024-05-15T13:06:10.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:10 smithi120 ceph-mon[43961]: pgmap v1263: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 977 op/s 2024-05-15T13:06:10.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:10 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:06:10.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:10 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:06:10.815 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:06:10.841 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:06:10.893 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:10 smithi177 ceph-mon[44303]: 5.8 scrub ok 2024-05-15T13:06:10.893 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:10 smithi177 ceph-mon[44303]: 5.6 scrub ok 2024-05-15T13:06:10.893 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:10 smithi177 ceph-mon[44303]: 5.e scrub ok 2024-05-15T13:06:10.893 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:10 smithi177 ceph-mon[44303]: pgmap v1263: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 977 op/s 2024-05-15T13:06:10.893 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:10 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:06:10.893 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:10 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:06:10.906 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:06:10.906 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:06:10.926 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:06:10.957 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:06:11.001 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:06:11.049 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:06:11.095 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:06:11.140 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:06:11.149 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:06:11.150 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:06:11.150 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:06:11.150 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:06:11.150 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:06:11.150 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:06:11.150 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:06:11.150 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:06:11.150 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:06:11.150 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:06:11.150 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:06:11.150 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:06:11.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:11 smithi027 ceph-mon[39795]: 5.2 scrub starts 2024-05-15T13:06:11.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:11 smithi027 ceph-mon[39795]: 5.c scrub ok 2024-05-15T13:06:11.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:11 smithi027 ceph-mon[39795]: osdmap e469: 12 total, 12 up, 11 in 2024-05-15T13:06:11.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:11 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3274500248' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:06:11.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:11 smithi027 ceph-mon[39795]: 5.2 scrub ok 2024-05-15T13:06:11.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:11 smithi120 ceph-mon[43961]: 5.2 scrub starts 2024-05-15T13:06:11.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:11 smithi120 ceph-mon[43961]: 5.c scrub ok 2024-05-15T13:06:11.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:11 smithi120 ceph-mon[43961]: osdmap e469: 12 total, 12 up, 11 in 2024-05-15T13:06:11.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:11 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3274500248' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:06:11.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:11 smithi120 ceph-mon[43961]: 5.2 scrub ok 2024-05-15T13:06:11.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:11 smithi177 ceph-mon[44303]: 5.2 scrub starts 2024-05-15T13:06:11.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:11 smithi177 ceph-mon[44303]: 5.c scrub ok 2024-05-15T13:06:11.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:11 smithi177 ceph-mon[44303]: osdmap e469: 12 total, 12 up, 11 in 2024-05-15T13:06:11.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:11 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3274500248' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:06:11.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:11 smithi177 ceph-mon[44303]: 5.2 scrub ok 2024-05-15T13:06:12.106 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:06:12.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:12.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:06:12.218 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:06:12.274 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:12.304 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:06:12.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:12.311 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:06:12.321 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:06:12.411 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:06:12.461 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:12.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:12 smithi177 ceph-mon[44303]: 5.0 scrub starts 2024-05-15T13:06:12.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:12 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:06:12.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:12 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:06:12.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:12 smithi177 ceph-mon[44303]: pgmap v1265: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 754 op/s 2024-05-15T13:06:12.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:12 smithi027 ceph-mon[39795]: 5.0 scrub starts 2024-05-15T13:06:12.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:12 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:06:12.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:12 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:06:12.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:12 smithi027 ceph-mon[39795]: pgmap v1265: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 754 op/s 2024-05-15T13:06:12.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:12 smithi120 ceph-mon[43961]: 5.0 scrub starts 2024-05-15T13:06:12.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:12 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:06:12.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:12 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:06:12.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:12 smithi120 ceph-mon[43961]: pgmap v1265: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 754 op/s 2024-05-15T13:06:13.510 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:13 smithi027 ceph-mon[39795]: 5.d scrub starts 2024-05-15T13:06:13.511 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:13 smithi027 ceph-mon[39795]: 5.0 scrub ok 2024-05-15T13:06:13.511 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:13 smithi027 ceph-mon[39795]: osdmap e470: 12 total, 12 up, 11 in 2024-05-15T13:06:13.511 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:13 smithi027 ceph-mon[39795]: 5.3 scrub starts 2024-05-15T13:06:13.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:13 smithi120 ceph-mon[43961]: 5.d scrub starts 2024-05-15T13:06:13.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:13 smithi120 ceph-mon[43961]: 5.0 scrub ok 2024-05-15T13:06:13.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:13 smithi120 ceph-mon[43961]: osdmap e470: 12 total, 12 up, 11 in 2024-05-15T13:06:13.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:13 smithi120 ceph-mon[43961]: 5.3 scrub starts 2024-05-15T13:06:13.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:13 smithi177 ceph-mon[44303]: 5.d scrub starts 2024-05-15T13:06:13.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:13 smithi177 ceph-mon[44303]: 5.0 scrub ok 2024-05-15T13:06:13.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:13 smithi177 ceph-mon[44303]: osdmap e470: 12 total, 12 up, 11 in 2024-05-15T13:06:13.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:13 smithi177 ceph-mon[44303]: 5.3 scrub starts 2024-05-15T13:06:13.964 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:06:13.965 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:06:14.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:14 smithi027 ceph-mon[39795]: 5.d scrub ok 2024-05-15T13:06:14.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:14 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:06:14.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:14 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:06:14.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:14 smithi027 ceph-mon[39795]: 5.3 scrub ok 2024-05-15T13:06:14.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:14 smithi027 ceph-mon[39795]: pgmap v1267: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 3.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 898 op/s 2024-05-15T13:06:14.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:14 smithi027 ceph-mon[39795]: osdmap e471: 12 total, 12 up, 11 in 2024-05-15T13:06:14.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:14 smithi120 ceph-mon[43961]: 5.d scrub ok 2024-05-15T13:06:14.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:14 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:06:14.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:14 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:06:14.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:14 smithi120 ceph-mon[43961]: 5.3 scrub ok 2024-05-15T13:06:14.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:14 smithi120 ceph-mon[43961]: pgmap v1267: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 3.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 898 op/s 2024-05-15T13:06:14.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:14 smithi120 ceph-mon[43961]: osdmap e471: 12 total, 12 up, 11 in 2024-05-15T13:06:14.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:14 smithi177 ceph-mon[44303]: 5.d scrub ok 2024-05-15T13:06:14.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:14 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:06:14.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:14 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:06:14.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:14 smithi177 ceph-mon[44303]: 5.3 scrub ok 2024-05-15T13:06:14.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:14 smithi177 ceph-mon[44303]: pgmap v1267: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 3.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 898 op/s 2024-05-15T13:06:14.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:14 smithi177 ceph-mon[44303]: osdmap e471: 12 total, 12 up, 11 in 2024-05-15T13:06:15.038 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:06:15.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:15.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:06:15.144 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:06:15.200 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:15.230 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:06:15.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:15.239 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:06:15.340 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:06:15.415 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:15.459 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:06:15.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:15.472 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:06:15.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:15 smithi027 ceph-mon[39795]: 5.b scrub starts 2024-05-15T13:06:15.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:15 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:06:15.573 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:06:15.621 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:15.655 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:06:15.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:15.663 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:06:15.764 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:06:15.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:15 smithi120 ceph-mon[43961]: 5.b scrub starts 2024-05-15T13:06:15.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:15 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:06:15.804 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:15 smithi177 ceph-mon[44303]: 5.b scrub starts 2024-05-15T13:06:15.804 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:15 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:06:15.804 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:15.828 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:06:15.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:15.831 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:06:15.932 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:06:15.936 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:06:15.936 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:06:15.937 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-05-15T13:06:15.937 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:06:15.971 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:15.991 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:06:15.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:15.995 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:06:16.096 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:06:16.128 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:16.155 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:06:16.163 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-05-15T13:06:16.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:16.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:06:16.265 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:06:16.331 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:16.377 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:06:16.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:16.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:06:16.430 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:06:16.498 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:06:16.528 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:16.560 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:06:16.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:16.571 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:06:16.673 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:06:16.764 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:16 smithi027 ceph-mon[39795]: 5.b scrub ok 2024-05-15T13:06:16.765 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:16 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:06:16.765 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:16 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:06:16.765 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:16 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:06:16.765 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:16 smithi027 ceph-mon[39795]: pgmap v1269: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 3.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 746 op/s 2024-05-15T13:06:16.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:16 smithi120 ceph-mon[43961]: 5.b scrub ok 2024-05-15T13:06:16.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:16 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:06:16.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:16 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:06:16.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:16 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:06:16.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:16 smithi120 ceph-mon[43961]: pgmap v1269: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 3.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 746 op/s 2024-05-15T13:06:16.787 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:16.821 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:06:16.830 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:06:16.843 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:06:16.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:16.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:06:16.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:16 smithi177 ceph-mon[44303]: 5.b scrub ok 2024-05-15T13:06:16.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:16 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:06:16.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:16 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:06:16.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:16 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:06:16.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:16 smithi177 ceph-mon[44303]: pgmap v1269: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 3.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 746 op/s 2024-05-15T13:06:16.947 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:06:16.952 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:06:16.979 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:16.998 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:06:17.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:17.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:06:17.102 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:06:17.142 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:17.171 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:06:17.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:17.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:06:17.277 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:06:17.325 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:17.357 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:06:17.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:17.359 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:06:17.460 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:06:17.513 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:17.974 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:06:18.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:17 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:06:18.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:17 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:06:18.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:17 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2430595583' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:06:18.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:17 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:06:18.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:17 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:06:18.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:17 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:06:18.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:17 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2430595583' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:06:18.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:17 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:06:18.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:17 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:06:18.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:17 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:06:18.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:17 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2430595583' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:06:18.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:17 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:06:19.116 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:18 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:06:19.117 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:18 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T13:06:19.117 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:18 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:06:19.117 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:18 smithi027 ceph-mon[39795]: osdmap e472: 12 total, 12 up, 11 in 2024-05-15T13:06:19.117 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:18 smithi027 ceph-mon[39795]: pgmap v1271: 17 pgs: 17 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 978 op/s 2024-05-15T13:06:19.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:18 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:06:19.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:18 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T13:06:19.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:18 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:06:19.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:18 smithi120 ceph-mon[43961]: osdmap e472: 12 total, 12 up, 11 in 2024-05-15T13:06:19.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:18 smithi120 ceph-mon[43961]: pgmap v1271: 17 pgs: 17 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 978 op/s 2024-05-15T13:06:19.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:18 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:06:19.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:18 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T13:06:19.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:18 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:06:19.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:18 smithi177 ceph-mon[44303]: osdmap e472: 12 total, 12 up, 11 in 2024-05-15T13:06:19.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:18 smithi177 ceph-mon[44303]: pgmap v1271: 17 pgs: 17 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 978 op/s 2024-05-15T13:06:21.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:21 smithi177 ceph-mon[44303]: pgmap v1272: 17 pgs: 17 active+clean; 3.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 995 op/s 2024-05-15T13:06:21.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:21 smithi027 ceph-mon[39795]: pgmap v1272: 17 pgs: 17 active+clean; 3.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 995 op/s 2024-05-15T13:06:21.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:21 smithi120 ceph-mon[43961]: pgmap v1272: 17 pgs: 17 active+clean; 3.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 995 op/s 2024-05-15T13:06:21.833 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:06:21.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:21.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:06:21.940 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:06:21.977 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:22.007 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:06:22.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:22.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:06:22.117 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:06:22.172 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:06:22.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:22.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:06:22.316 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:06:22.365 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:22.393 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:06:22.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:22.397 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:06:22.498 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:06:22.544 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:22.568 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:06:22.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:22.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:06:22.675 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:06:22.740 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:22.962 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:06:22.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:22.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:06:23.068 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:06:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:23.407 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:23 smithi027 ceph-mon[39795]: pgmap v1273: 17 pgs: 17 active+clean; 3.3 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 823 op/s 2024-05-15T13:06:23.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:23 smithi177 ceph-mon[44303]: pgmap v1273: 17 pgs: 17 active+clean; 3.3 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 823 op/s 2024-05-15T13:06:23.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:23 smithi120 ceph-mon[43961]: pgmap v1273: 17 pgs: 17 active+clean; 3.3 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 823 op/s 2024-05-15T13:06:24.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:24 smithi177 ceph-mon[44303]: pgmap v1274: 17 pgs: 17 active+clean; 3.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 927 op/s 2024-05-15T13:06:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:06:24.690 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:24 smithi027 ceph-mon[39795]: pgmap v1274: 17 pgs: 17 active+clean; 3.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 927 op/s 2024-05-15T13:06:24.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:24.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:06:24.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:24 smithi120 ceph-mon[43961]: pgmap v1274: 17 pgs: 17 active+clean; 3.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 927 op/s 2024-05-15T13:06:24.786 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:06:24.795 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:06:24.825 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:06:24.863 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:24.864 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:06:24.901 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:06:24.907 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:06:24.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:24.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:06:24.937 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:06:24.978 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:06:25.012 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:06:25.017 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:06:25.047 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:06:25.061 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:25.083 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:06:25.094 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:06:25.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:25.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:06:25.111 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:06:25.161 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:06:25.198 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:06:25.205 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:06:25.205 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:06:25.205 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:06:25.205 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:06:25.206 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:06:25.206 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:06:25.206 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:06:25.206 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:06:25.206 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:06:25.206 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:06:25.206 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:06:25.206 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:06:25.206 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:06:25.224 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:06:25.224 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T13:06:25.224 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-05-15T13:06:25.224 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-05-15T13:06:25.224 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:06:25.233 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:25.263 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:06:25.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:25.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:06:25.372 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:06:25.394 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:25 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1158815962' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:06:25.405 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:25.434 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:06:25.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:25.439 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:06:25.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:25 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1158815962' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:06:25.540 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:06:25.581 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:25.618 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:06:25.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:25.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:06:25.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:25 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1158815962' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:06:25.724 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:06:25.774 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:25.811 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:06:25.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:25.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:06:25.918 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:06:25.959 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:25.998 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:06:26.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:26.002 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:06:26.049 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:06:26.103 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:06:26.172 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:26.390 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:26 smithi027 ceph-mon[39795]: 5.f deep-scrub starts 2024-05-15T13:06:26.397 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:26 smithi177 ceph-mon[44303]: 5.f deep-scrub starts 2024-05-15T13:06:26.397 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:26 smithi177 ceph-mon[44303]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:06:26.397 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:26 smithi177 ceph-mon[44303]: pgmap v1275: 17 pgs: 17 active+clean; 3.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 901 op/s 2024-05-15T13:06:26.735 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:06:26.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:26.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:06:26.743 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:26 smithi027 ceph-mon[39795]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:06:26.743 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:26 smithi027 ceph-mon[39795]: pgmap v1275: 17 pgs: 17 active+clean; 3.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 901 op/s 2024-05-15T13:06:26.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:26 smithi120 ceph-mon[43961]: 5.f deep-scrub starts 2024-05-15T13:06:26.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:26 smithi120 ceph-mon[43961]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:06:26.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:26 smithi120 ceph-mon[43961]: pgmap v1275: 17 pgs: 17 active+clean; 3.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 901 op/s 2024-05-15T13:06:26.844 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:06:26.893 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:27.399 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:06:27.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:27.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:06:27.507 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:06:27.508 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:06:27.508 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:06:27.544 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:28.097 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:06:28.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:28.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:06:28.110 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-05-15T13:06:28.207 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:06:28.256 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:28.283 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:06:28.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:28.285 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:06:28.386 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:06:28.441 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:28.528 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:06:28.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:28.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:06:28.582 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":1274,"stamp":"2024-05-15T13:06:24.040119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3377791544,"num_objects":51535,"num_object_clones":0,"num_object_copies":154605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51535,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":219026,"num_write_kb":8654971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160884,"ondisk_log_size":160884,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":14882560,"kb_used_data":10110820,"kb_used_omap":17,"kb_used_meta":4771502,"kb_avail":1016314112,"statfs":{"total":1055945392128,"available":1040705650688,"internally_reserved":0,"allocated":10353479680,"data_stored":10350451812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17551,"internal_metadata":4886018929},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-587661312,"num_objects":-8967,"num_object_clones":0,"num_object_copies":-26901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8967,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.672433"},"pg_stats":[{"pgid":"5.f","version":"472'13731","reported_seq":13943,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T13:06:22.965768+0000","last_change":"2024-05-15T13:05:13.243193+0000","last_active":"2024-05-15T13:06:22.965768+0000","last_peered":"2024-05-15T13:06:22.965768+0000","last_clean":"2024-05-15T13:06:22.965768+0000","last_became_active":"2024-05-15T13:03:11.328783+0000","last_became_peered":"2024-05-15T13:03:11.328783+0000","last_unstale":"2024-05-15T13:06:22.965768+0000","last_undegraded":"2024-05-15T13:06:22.965768+0000","last_fullsized":"2024-05-15T13:06:22.965768+0000","mapping_epoch":418,"log_start":"433'3700","ondisk_log_start":"433'3700","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'9853","last_scrub_stamp":"2024-05-15T13:05:13.242947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:05:13.242947+0000","objects_scrubbed":7125,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:06:24.462839+0000","scrub_duration":1769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209125376,"num_objects":3191,"num_object_clones":0,"num_object_copies":9573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13731,"num_write_kb":541504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"472'13697","reported_seq":13938,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T13:06:20.706145+0000","last_change":"2024-05-15T13:06:09.341696+0000","last_active":"2024-05-15T13:06:20.706145+0000","last_peered":"2024-05-15T13:06:20.706145+0000","last_clean":"2024-05-15T13:06:20.706145+0000","last_became_active":"2024-05-15T13:03:11.330241+0000","last_became_peered":"2024-05-15T13:03:11.330241+0000","last_unstale":"2024-05-15T13:06:20.706145+0000","last_undegraded":"2024-05-15T13:06:20.706145+0000","last_fullsized":"2024-05-15T13:06:20.706145+0000","mapping_epoch":418,"log_start":"432'3600","ondisk_log_start":"432'3600","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'13097","last_scrub_stamp":"2024-05-15T13:06:09.341563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:06:09.341563+0000","objects_scrubbed":3801,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:07:27.196776+0000","scrub_duration":853,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208470016,"num_objects":3181,"num_object_clones":0,"num_object_copies":9543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13697,"num_write_kb":540096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"472'13927","reported_seq":14178,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T13:06:22.632659+0000","last_change":"2024-05-15T13:06:12.652886+0000","last_active":"2024-05-15T13:06:22.632659+0000","last_peered":"2024-05-15T13:06:22.632659+0000","last_clean":"2024-05-15T13:06:22.632659+0000","last_became_active":"2024-05-15T13:03:11.330919+0000","last_became_peered":"2024-05-15T13:03:11.330919+0000","last_unstale":"2024-05-15T13:06:22.632659+0000","last_undegraded":"2024-05-15T13:06:22.632659+0000","last_fullsized":"2024-05-15T13:06:22.632659+0000","mapping_epoch":418,"log_start":"434'3900","ondisk_log_start":"434'3900","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'13360","last_scrub_stamp":"2024-05-15T13:06:12.652697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:06:12.652697+0000","objects_scrubbed":3875,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:07:39.686553+0000","scrub_duration":1077,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215351320,"num_objects":3287,"num_object_clones":0,"num_object_copies":9861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3287,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13927,"num_write_kb":550785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"472'13739","reported_seq":13990,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T13:06:21.541943+0000","last_change":"2024-05-15T13:06:09.800468+0000","last_active":"2024-05-15T13:06:21.541943+0000","last_peered":"2024-05-15T13:06:21.541943+0000","last_clean":"2024-05-15T13:06:21.541943+0000","last_became_active":"2024-05-15T13:03:11.327387+0000","last_became_peered":"2024-05-15T13:03:11.327387+0000","last_unstale":"2024-05-15T13:06:21.541943+0000","last_undegraded":"2024-05-15T13:06:21.541943+0000","last_fullsized":"2024-05-15T13:06:21.541943+0000","mapping_epoch":418,"log_start":"433'3700","ondisk_log_start":"433'3700","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'13068","last_scrub_stamp":"2024-05-15T13:06:09.800243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:06:09.800243+0000","objects_scrubbed":3878,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:07:23.879055+0000","scrub_duration":1307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207028224,"num_objects":3159,"num_object_clones":0,"num_object_copies":9477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13739,"num_write_kb":540736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"472'13434","reported_seq":13675,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T13:06:19.306378+0000","last_change":"2024-05-15T13:06:15.909883+0000","last_active":"2024-05-15T13:06:19.306378+0000","last_peered":"2024-05-15T13:06:19.306378+0000","last_clean":"2024-05-15T13:06:19.306378+0000","last_became_active":"2024-05-15T13:03:11.328691+0000","last_became_peered":"2024-05-15T13:03:11.328691+0000","last_unstale":"2024-05-15T13:06:19.306378+0000","last_undegraded":"2024-05-15T13:06:19.306378+0000","last_fullsized":"2024-05-15T13:06:19.306378+0000","mapping_epoch":418,"log_start":"432'3400","ondisk_log_start":"432'3400","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"471'13240","last_scrub_stamp":"2024-05-15T13:06:15.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:06:15.909706+0000","objects_scrubbed":3581,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:07:34.493734+0000","scrub_duration":1168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220200960,"num_objects":3360,"num_object_clones":0,"num_object_copies":10080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13434,"num_write_kb":537408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,5],"acting":[0,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"472'13674","reported_seq":13938,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T13:06:22.599823+0000","last_change":"2024-05-15T13:05:29.969040+0000","last_active":"2024-05-15T13:06:22.599823+0000","last_peered":"2024-05-15T13:06:22.599823+0000","last_clean":"2024-05-15T13:06:22.599823+0000","last_became_active":"2024-05-15T13:03:11.330425+0000","last_became_peered":"2024-05-15T13:03:11.330425+0000","last_unstale":"2024-05-15T13:06:22.599823+0000","last_undegraded":"2024-05-15T13:06:22.599823+0000","last_fullsized":"2024-05-15T13:06:22.599823+0000","mapping_epoch":418,"log_start":"433'3600","ondisk_log_start":"433'3600","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'10689","last_scrub_stamp":"2024-05-15T13:05:29.968791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:05:29.968791+0000","objects_scrubbed":6257,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:06:53.094334+0000","scrub_duration":1767,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210370560,"num_objects":3210,"num_object_clones":0,"num_object_copies":9630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13674,"num_write_kb":540288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"472'13639","reported_seq":13883,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T13:06:23.685596+0000","last_change":"2024-05-15T13:06:08.688185+0000","last_active":"2024-05-15T13:06:23.685596+0000","last_peered":"2024-05-15T13:06:23.685596+0000","last_clean":"2024-05-15T13:06:23.685596+0000","last_became_active":"2024-05-15T13:03:11.329210+0000","last_became_peered":"2024-05-15T13:03:11.329210+0000","last_unstale":"2024-05-15T13:06:23.685596+0000","last_undegraded":"2024-05-15T13:06:23.685596+0000","last_fullsized":"2024-05-15T13:06:23.685596+0000","mapping_epoch":418,"log_start":"433'3600","ondisk_log_start":"433'3600","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'12836","last_scrub_stamp":"2024-05-15T13:06:08.688065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:06:08.688065+0000","objects_scrubbed":3932,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:07:17.404565+0000","scrub_duration":880,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203882496,"num_objects":3111,"num_object_clones":0,"num_object_copies":9333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13639,"num_write_kb":536000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,9],"acting":[6,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"472'13511","reported_seq":13757,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T13:06:22.614574+0000","last_change":"2024-05-15T13:06:08.701773+0000","last_active":"2024-05-15T13:06:22.614574+0000","last_peered":"2024-05-15T13:06:22.614574+0000","last_clean":"2024-05-15T13:06:22.614574+0000","last_became_active":"2024-05-15T13:03:11.330983+0000","last_became_peered":"2024-05-15T13:03:11.330983+0000","last_unstale":"2024-05-15T13:06:22.614574+0000","last_undegraded":"2024-05-15T13:06:22.614574+0000","last_fullsized":"2024-05-15T13:06:22.614574+0000","mapping_epoch":418,"log_start":"433'3500","ondisk_log_start":"433'3500","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'12727","last_scrub_stamp":"2024-05-15T13:06:08.701412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:06:08.701412+0000","objects_scrubbed":3977,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:07:26.983604+0000","scrub_duration":1219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207552512,"num_objects":3167,"num_object_clones":0,"num_object_copies":9501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13511,"num_write_kb":533696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"472'13846","reported_seq":14087,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T13:06:22.973868+0000","last_change":"2024-05-15T13:06:07.504147+0000","last_active":"2024-05-15T13:06:22.973868+0000","last_peered":"2024-05-15T13:06:22.973868+0000","last_clean":"2024-05-15T13:06:22.973868+0000","last_became_active":"2024-05-15T13:03:11.329484+0000","last_became_peered":"2024-05-15T13:03:11.329484+0000","last_unstale":"2024-05-15T13:06:22.973868+0000","last_undegraded":"2024-05-15T13:06:22.973868+0000","last_fullsized":"2024-05-15T13:06:22.973868+0000","mapping_epoch":418,"log_start":"433'3800","ondisk_log_start":"433'3800","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'13020","last_scrub_stamp":"2024-05-15T13:06:07.503900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:06:07.503900+0000","objects_scrubbed":3987,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:07:21.930524+0000","scrub_duration":1282,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204734464,"num_objects":3124,"num_object_clones":0,"num_object_copies":9372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13846,"num_write_kb":543040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"472'13731","reported_seq":13970,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T13:06:22.642597+0000","last_change":"2024-05-15T13:06:09.202863+0000","last_active":"2024-05-15T13:06:22.642597+0000","last_peered":"2024-05-15T13:06:22.642597+0000","last_clean":"2024-05-15T13:06:22.642597+0000","last_became_active":"2024-05-15T13:03:11.332923+0000","last_became_peered":"2024-05-15T13:03:11.332923+0000","last_unstale":"2024-05-15T13:06:22.642597+0000","last_undegraded":"2024-05-15T13:06:22.642597+0000","last_fullsized":"2024-05-15T13:06:22.642597+0000","mapping_epoch":418,"log_start":"433'3700","ondisk_log_start":"433'3700","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'12948","last_scrub_stamp":"2024-05-15T13:06:09.202656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:06:09.202656+0000","objects_scrubbed":3885,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:07:12.656870+0000","scrub_duration":986,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202047488,"num_objects":3083,"num_object_clones":0,"num_object_copies":9249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13731,"num_write_kb":538048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,10],"acting":[2,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"472'13466","reported_seq":13707,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T13:06:19.312332+0000","last_change":"2024-05-15T13:06:08.743422+0000","last_active":"2024-05-15T13:06:19.312332+0000","last_peered":"2024-05-15T13:06:19.312332+0000","last_clean":"2024-05-15T13:06:19.312332+0000","last_became_active":"2024-05-15T13:03:11.333336+0000","last_became_peered":"2024-05-15T13:03:11.333336+0000","last_unstale":"2024-05-15T13:06:19.312332+0000","last_undegraded":"2024-05-15T13:06:19.312332+0000","last_fullsized":"2024-05-15T13:06:19.312332+0000","mapping_epoch":418,"log_start":"432'3400","ondisk_log_start":"432'3400","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'12843","last_scrub_stamp":"2024-05-15T13:06:08.743221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:06:08.743221+0000","objects_scrubbed":3977,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:07:29.200669+0000","scrub_duration":1057,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219021312,"num_objects":3342,"num_object_clones":0,"num_object_copies":10026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13466,"num_write_kb":537856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"472'13690","reported_seq":13929,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T13:06:19.356325+0000","last_change":"2024-05-15T13:06:13.847410+0000","last_active":"2024-05-15T13:06:19.356325+0000","last_peered":"2024-05-15T13:06:19.356325+0000","last_clean":"2024-05-15T13:06:19.356325+0000","last_became_active":"2024-05-15T13:03:11.327788+0000","last_became_peered":"2024-05-15T13:03:11.327788+0000","last_unstale":"2024-05-15T13:06:19.356325+0000","last_undegraded":"2024-05-15T13:06:19.356325+0000","last_fullsized":"2024-05-15T13:06:19.356325+0000","mapping_epoch":418,"log_start":"433'3600","ondisk_log_start":"433'3600","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'13369","last_scrub_stamp":"2024-05-15T13:06:13.846987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:06:13.846987+0000","objects_scrubbed":3798,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:07:23.520381+0000","scrub_duration":1108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226230272,"num_objects":3452,"num_object_clones":0,"num_object_copies":10356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13690,"num_write_kb":548544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"472'13770","reported_seq":14012,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T13:06:23.705628+0000","last_change":"2024-05-15T13:06:10.694844+0000","last_active":"2024-05-15T13:06:23.705628+0000","last_peered":"2024-05-15T13:06:23.705628+0000","last_clean":"2024-05-15T13:06:23.705628+0000","last_became_active":"2024-05-15T13:03:11.330316+0000","last_became_peered":"2024-05-15T13:03:11.330316+0000","last_unstale":"2024-05-15T13:06:23.705628+0000","last_undegraded":"2024-05-15T13:06:23.705628+0000","last_fullsized":"2024-05-15T13:06:23.705628+0000","mapping_epoch":418,"log_start":"433'3700","ondisk_log_start":"433'3700","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'13043","last_scrub_stamp":"2024-05-15T13:06:10.694597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:06:10.694597+0000","objects_scrubbed":3875,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:07:13.467238+0000","scrub_duration":900,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204472320,"num_objects":3120,"num_object_clones":0,"num_object_copies":9360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13770,"num_write_kb":540480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"472'13650","reported_seq":13893,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T13:06:19.973482+0000","last_change":"2024-05-15T13:06:06.808513+0000","last_active":"2024-05-15T13:06:19.973482+0000","last_peered":"2024-05-15T13:06:19.973482+0000","last_clean":"2024-05-15T13:06:19.973482+0000","last_became_active":"2024-05-15T13:03:11.430987+0000","last_became_peered":"2024-05-15T13:03:11.430987+0000","last_unstale":"2024-05-15T13:06:19.973482+0000","last_undegraded":"2024-05-15T13:06:19.973482+0000","last_fullsized":"2024-05-15T13:06:19.973482+0000","mapping_epoch":418,"log_start":"433'3600","ondisk_log_start":"433'3600","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'12920","last_scrub_stamp":"2024-05-15T13:06:06.808420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:06:06.808420+0000","objects_scrubbed":4065,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:07:17.606250+0000","scrub_duration":999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216662016,"num_objects":3306,"num_object_clones":0,"num_object_copies":9918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13650,"num_write_kb":542592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"472'13630","reported_seq":13880,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T13:06:21.557355+0000","last_change":"2024-05-15T13:06:12.446484+0000","last_active":"2024-05-15T13:06:21.557355+0000","last_peered":"2024-05-15T13:06:21.557355+0000","last_clean":"2024-05-15T13:06:21.557355+0000","last_became_active":"2024-05-15T13:03:11.330479+0000","last_became_peered":"2024-05-15T13:03:11.330479+0000","last_unstale":"2024-05-15T13:06:21.557355+0000","last_undegraded":"2024-05-15T13:06:21.557355+0000","last_fullsized":"2024-05-15T13:06:21.557355+0000","mapping_epoch":418,"log_start":"433'3600","ondisk_log_start":"433'3600","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'13106","last_scrub_stamp":"2024-05-15T13:06:12.446234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:06:12.446234+0000","objects_scrubbed":3744,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:07:35.800592+0000","scrub_duration":924,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209715200,"num_objects":3200,"num_object_clones":0,"num_object_copies":9600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13630,"num_write_kb":538560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,4],"acting":[1,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"472'13710","reported_seq":13973,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T13:06:21.480699+0000","last_change":"2024-05-15T13:05:32.580562+0000","last_active":"2024-05-15T13:06:21.480699+0000","last_peered":"2024-05-15T13:06:21.480699+0000","last_clean":"2024-05-15T13:06:21.480699+0000","last_became_active":"2024-05-15T13:03:11.331291+0000","last_became_peered":"2024-05-15T13:03:11.331291+0000","last_unstale":"2024-05-15T13:06:21.480699+0000","last_undegraded":"2024-05-15T13:06:21.480699+0000","last_fullsized":"2024-05-15T13:06:21.480699+0000","mapping_epoch":418,"log_start":"433'3700","ondisk_log_start":"433'3700","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'10936","last_scrub_stamp":"2024-05-15T13:05:32.580374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:03:10.302515+0000","last_clean_scrub_stamp":"2024-05-15T13:05:32.580374+0000","objects_scrubbed":6054,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:06:53.823617+0000","scrub_duration":1468,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212336640,"num_objects":3240,"num_object_clones":0,"num_object_copies":9720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13710,"num_write_kb":542400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":779,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T13:06:17.974979+0000","last_change":"2024-05-15T13:06:07.613894+0000","last_active":"2024-05-15T13:06:17.974979+0000","last_peered":"2024-05-15T13:06:17.974979+0000","last_clean":"2024-05-15T13:06:17.974979+0000","last_became_active":"2024-05-15T13:00:26.045412+0000","last_became_peered":"2024-05-15T13:00:26.045412+0000","last_unstale":"2024-05-15T13:06:17.974979+0000","last_undegraded":"2024-05-15T13:06:17.974979+0000","last_fullsized":"2024-05-15T13:06:17.974979+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:06:07.613802+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:06:07.613802+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:07:32.703344+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":3377201176,"num_objects":51533,"num_object_clones":0,"num_object_copies":154599,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51533,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":218845,"num_write_kb":8652033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10190860288,"data_stored":10190848072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160745,"ondisk_log_size":160745,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056891,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1235212,"kb_used_data":825028,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92509940,"statfs":{"total":95995035648,"available":94730178560,"internally_reserved":0,"allocated":844828672,"data_stored":844551404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":420018630},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351231,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1728400,"kb_used_data":1282120,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":92016752,"statfs":{"total":95995035648,"available":94225154048,"internally_reserved":0,"allocated":1312890880,"data_stored":1312614132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":456980928},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514757,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1608136,"kb_used_data":1049344,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92137016,"statfs":{"total":95995035648,"available":94348304384,"internally_reserved":0,"allocated":1074528256,"data_stored":1074255060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":572127680},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874506,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":726024,"kb_used_data":433856,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":93019128,"statfs":{"total":95995035648,"available":95251587072,"internally_reserved":0,"allocated":444268544,"data_stored":443995368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":240518168846,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430483,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1024720,"kb_used_data":628424,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92720432,"statfs":{"total":95995035648,"available":94945722368,"internally_reserved":0,"allocated":643506176,"data_stored":643229448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":405797318},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575150,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1367696,"kb_used_data":835336,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":92377456,"statfs":{"total":95995035648,"available":94594514944,"internally_reserved":0,"allocated":855384064,"data_stored":855107336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738678,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2259276,"kb_used_data":1740804,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":91485876,"statfs":{"total":95995035648,"available":93681537024,"internally_reserved":0,"allocated":1782583296,"data_stored":1782306048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411627,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896520,"kb_used_data":608512,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":92848632,"statfs":{"total":95995035648,"available":95076999168,"internally_reserved":0,"allocated":623116288,"data_stored":622843112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149988,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":637192,"kb_used_data":208960,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93107960,"statfs":{"total":95995035648,"available":95342551040,"internally_reserved":0,"allocated":213975040,"data_stored":213701864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":438499783},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986463,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1571532,"kb_used_data":1062784,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":92173620,"statfs":{"total":95995035648,"available":94385786880,"internally_reserved":0,"allocated":1088290816,"data_stored":1088017640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692121,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1827852,"kb_used_data":1435652,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":91917300,"statfs":{"total":95995035648,"available":94123315200,"internally_reserved":0,"allocated":1470107648,"data_stored":1469830400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1761415168,"data_stored":1761411096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":838664192,"data_stored":838664192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":619184128,"data_stored":619184128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":209518592,"data_stored":209518592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1082392576,"data_stored":1082392576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1450840064,"data_stored":1450835992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":616562688,"data_stored":616562688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":432275456,"data_stored":432275456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1068957696,"data_stored":1068957696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1269104640,"data_stored":1269104640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":841945088,"data_stored":841941016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:06:28.584 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-05-15T13:06:28.584 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-05-15T13:06:28.634 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:06:28.641 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:06:28.685 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:28.702 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:06:28.945 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:06:28.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:28.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:06:29.055 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:06:29.088 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:29.126 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:06:29.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:29.129 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:06:29.205 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:29 smithi027 ceph-mon[39795]: pgmap v1276: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 948 op/s 2024-05-15T13:06:29.229 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:06:29.276 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:29.306 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:06:29.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:29.309 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:06:29.410 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:06:29.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:29 smithi177 ceph-mon[44303]: pgmap v1276: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 948 op/s 2024-05-15T13:06:29.439 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:06:29.446 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:29 smithi120 ceph-mon[43961]: pgmap v1276: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 948 op/s 2024-05-15T13:06:29.448 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:29.488 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:06:29.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:29.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:06:29.591 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:06:29.635 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:29.668 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:06:29.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:29.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:06:29.778 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:06:29.817 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:29.846 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:06:29.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:29.853 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:06:29.954 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:06:29.988 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:30.030 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:06:30.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:30.033 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:06:30.134 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:06:30.170 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:30.202 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:06:30.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:30.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:06:30.307 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:06:30.340 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:30.372 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:06:30.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:30.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:06:30.476 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:06:30.536 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:31.435 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:31 smithi177 ceph-mon[44303]: pgmap v1277: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 867 op/s 2024-05-15T13:06:31.482 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:06:31.482 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T13:06:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T13:06:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:06:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:48851 4.0 4:0f538546:::benchmark_data_smithi177_99756_object48850:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-05-15T13:06:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T12:58:29.293070+0000", 2024-05-15T13:06:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 482.195287033, 2024-05-15T13:06:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.210520136, 2024-05-15T13:06:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:06:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:06:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 48851 2024-05-15T13:06:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.293070+0000", 2024-05-15T13:06:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.293070+0000", 2024-05-15T13:06:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.293071+0000", 2024-05-15T13:06:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2109999999999999e-06 2024-05-15T13:06:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.293152+0000", 2024-05-15T13:06:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1561000000000001e-05 2024-05-15T13:06:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.293163+0000", 2024-05-15T13:06:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1107000000000001e-05 2024-05-15T13:06:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.293170+0000", 2024-05-15T13:06:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3690000000000003e-06 2024-05-15T13:06:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.293224+0000", 2024-05-15T13:06:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3971999999999998e-05 2024-05-15T13:06:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.293328+0000", 2024-05-15T13:06:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010450600000000001 2024-05-15T13:06:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,8", 2024-05-15T13:06:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.293437+0000", 2024-05-15T13:06:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010872800000000001 2024-05-15T13:06:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:06:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.295197+0000", 2024-05-15T13:06:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0017604229999999999 2024-05-15T13:06:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.298482+0000", 2024-05-15T13:06:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032841150000000002 2024-05-15T13:06:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.503519+0000", 2024-05-15T13:06:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.205037107 2024-05-15T13:06:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.503558+0000", 2024-05-15T13:06:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9320000000000003e-05 2024-05-15T13:06:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T12:58:29.503590+0000", 2024-05-15T13:06:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1717000000000001e-05 2024-05-15T13:06:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:142000 4.9 e357/343 4:9e3e0cd7:::benchmark_data_smithi177_99756_object22028:head v 357'8894, mlcod=323'1360)", 2024-05-15T13:06:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:13.180988+0000", 2024-05-15T13:06:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 378.30736857400001, 2024-05-15T13:06:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16514794299999999, 2024-05-15T13:06:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.180988+0000", 2024-05-15T13:06:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.180988+0000", 2024-05-15T13:06:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.180989+0000", 2024-05-15T13:06:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3099999999999997e-07 2024-05-15T13:06:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.180996+0000", 2024-05-15T13:06:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5630000000000003e-06 2024-05-15T13:06:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.181007+0000", 2024-05-15T13:06:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0494e-05 2024-05-15T13:06:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.181015+0000", 2024-05-15T13:06:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3189999999999995e-06 2024-05-15T13:06:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.181066+0000", 2024-05-15T13:06:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0540000000000001e-05 2024-05-15T13:06:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.181091+0000", 2024-05-15T13:06:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5448000000000001e-05 2024-05-15T13:06:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.184878+0000", 2024-05-15T13:06:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0037873149999999999 2024-05-15T13:06:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346136+0000", 2024-05-15T13:06:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16125753300000001 2024-05-15T13:06:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:142002 4.c e357/317 4:39c01dc0:::benchmark_data_smithi177_99756_object22030:head v 357'8893, mlcod=357'8891)", 2024-05-15T13:06:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:13.183315+0000", 2024-05-15T13:06:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 378.30504134900002, 2024-05-15T13:06:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16287758499999999, 2024-05-15T13:06:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.183315+0000", 2024-05-15T13:06:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.183315+0000", 2024-05-15T13:06:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.183316+0000", 2024-05-15T13:06:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.54e-07 2024-05-15T13:06:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.183323+0000", 2024-05-15T13:06:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3030000000000002e-06 2024-05-15T13:06:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.183335+0000", 2024-05-15T13:06:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1935e-05 2024-05-15T13:06:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.183342+0000", 2024-05-15T13:06:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0870000000000001e-06 2024-05-15T13:06:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.183398+0000", 2024-05-15T13:06:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5373e-05 2024-05-15T13:06:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.183418+0000", 2024-05-15T13:06:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9882999999999999e-05 2024-05-15T13:06:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346152+0000", 2024-05-15T13:06:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16273420299999999 2024-05-15T13:06:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346193+0000", 2024-05-15T13:06:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0946999999999999e-05 2024-05-15T13:06:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:142007 4.0 4:006ed880:::benchmark_data_smithi177_99756_object22035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-05-15T13:06:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:13.189831+0000", 2024-05-15T13:06:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 378.29852583000002, 2024-05-15T13:06:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.171594948, 2024-05-15T13:06:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:06:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:06:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 142007 2024-05-15T13:06:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.189831+0000", 2024-05-15T13:06:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.189831+0000", 2024-05-15T13:06:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.189833+0000", 2024-05-15T13:06:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.826e-06 2024-05-15T13:06:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.189841+0000", 2024-05-15T13:06:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5290000000000008e-06 2024-05-15T13:06:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346127+0000", 2024-05-15T13:06:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.156286175 2024-05-15T13:06:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346142+0000", 2024-05-15T13:06:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4600000000000001e-05 2024-05-15T13:06:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346259+0000", 2024-05-15T13:06:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011670700000000001 2024-05-15T13:06:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346391+0000", 2024-05-15T13:06:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013222200000000001 2024-05-15T13:06:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,8", 2024-05-15T13:06:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346510+0000", 2024-05-15T13:06:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000118898 2024-05-15T13:06:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:06:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.355786+0000", 2024-05-15T13:06:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0092759000000000001 2024-05-15T13:06:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.360397+0000", 2024-05-15T13:06:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046116919999999997 2024-05-15T13:06:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.361325+0000", 2024-05-15T13:06:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00092745299999999996 2024-05-15T13:06:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.361381+0000", 2024-05-15T13:06:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6512999999999999e-05 2024-05-15T13:06:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.361426+0000", 2024-05-15T13:06:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4433e-05 2024-05-15T13:06:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:142021 4.f e357/343 4:ff930601:::benchmark_data_smithi177_99756_object22049:head v 357'8697, mlcod=323'1700)", 2024-05-15T13:06:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:13.218868+0000", 2024-05-15T13:06:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 378.26948876500001, 2024-05-15T13:06:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.152184717, 2024-05-15T13:06:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.218868+0000", 2024-05-15T13:06:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.218868+0000", 2024-05-15T13:06:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.218870+0000", 2024-05-15T13:06:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.747e-06 2024-05-15T13:06:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.218880+0000", 2024-05-15T13:06:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0275000000000001e-05 2024-05-15T13:06:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346117+0000", 2024-05-15T13:06:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12723722200000001 2024-05-15T13:06:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346133+0000", 2024-05-15T13:06:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5945999999999999e-05 2024-05-15T13:06:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346197+0000", 2024-05-15T13:06:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4321999999999998e-05 2024-05-15T13:06:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.346228+0000", 2024-05-15T13:06:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0898999999999997e-05 2024-05-15T13:06:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.371017+0000", 2024-05-15T13:06:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.024788849000000002 2024-05-15T13:06:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:13.371053+0000", 2024-05-15T13:06:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5457000000000003e-05 2024-05-15T13:06:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:145947 4.0 4:02e2b03e:::benchmark_data_smithi177_99756_object25975:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-05-15T13:06:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:19.978855+0000", 2024-05-15T13:06:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 371.50950189999998, 2024-05-15T13:06:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.057148081000000003, 2024-05-15T13:06:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:06:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:06:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 145947 2024-05-15T13:06:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978855+0000", 2024-05-15T13:06:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978855+0000", 2024-05-15T13:06:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978857+0000", 2024-05-15T13:06:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0250000000000001e-06 2024-05-15T13:06:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978866+0000", 2024-05-15T13:06:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0790000000000007e-06 2024-05-15T13:06:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978892+0000", 2024-05-15T13:06:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6670999999999999e-05 2024-05-15T13:06:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978904+0000", 2024-05-15T13:06:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1214e-05 2024-05-15T13:06:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978971+0000", 2024-05-15T13:06:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7081000000000004e-05 2024-05-15T13:06:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.979127+0000", 2024-05-15T13:06:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015630900000000001 2024-05-15T13:06:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,8", 2024-05-15T13:06:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.979236+0000", 2024-05-15T13:06:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000109277 2024-05-15T13:06:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.993968+0000", 2024-05-15T13:06:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014731164 2024-05-15T13:06:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:06:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.993984+0000", 2024-05-15T13:06:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6492999999999999e-05 2024-05-15T13:06:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.035928+0000", 2024-05-15T13:06:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.041944082000000001 2024-05-15T13:06:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.035969+0000", 2024-05-15T13:06:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0572999999999999e-05 2024-05-15T13:06:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.036003+0000", 2024-05-15T13:06:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4112999999999997e-05 2024-05-15T13:06:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:145950 4.0 4:0acf3df5:::benchmark_data_smithi177_99756_object25978:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-05-15T13:06:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:19.978932+0000", 2024-05-15T13:06:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 371.50942460900001, 2024-05-15T13:06:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.057137765, 2024-05-15T13:06:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:06:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:06:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 145950 2024-05-15T13:06:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978932+0000", 2024-05-15T13:06:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978932+0000", 2024-05-15T13:06:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978933+0000", 2024-05-15T13:06:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4099999999999998e-07 2024-05-15T13:06:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978936+0000", 2024-05-15T13:06:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.642e-06 2024-05-15T13:06:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978945+0000", 2024-05-15T13:06:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2730000000000002e-06 2024-05-15T13:06:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978951+0000", 2024-05-15T13:06:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6490000000000002e-06 2024-05-15T13:06:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.983771+0000", 2024-05-15T13:06:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0048201249999999998 2024-05-15T13:06:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.983895+0000", 2024-05-15T13:06:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012362699999999999 2024-05-15T13:06:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,8", 2024-05-15T13:06:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.984007+0000", 2024-05-15T13:06:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011250499999999999 2024-05-15T13:06:31.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:06:31.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.993989+0000", 2024-05-15T13:06:31.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0099817480000000004 2024-05-15T13:06:31.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:31.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.000442+0000", 2024-05-15T13:06:31.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0064523130000000003 2024-05-15T13:06:31.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:31.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.036025+0000", 2024-05-15T13:06:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.035583468 2024-05-15T13:06:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.036050+0000", 2024-05-15T13:06:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5097e-05 2024-05-15T13:06:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.036070+0000", 2024-05-15T13:06:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9677000000000002e-05 2024-05-15T13:06:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:145951 4.0 4:089eb460:::benchmark_data_smithi177_99756_object25979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-05-15T13:06:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:19.978965+0000", 2024-05-15T13:06:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 371.50939139500002, 2024-05-15T13:06:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.063096423999999998, 2024-05-15T13:06:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:06:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:06:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 145951 2024-05-15T13:06:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978965+0000", 2024-05-15T13:06:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978965+0000", 2024-05-15T13:06:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978966+0000", 2024-05-15T13:06:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4600000000000005e-07 2024-05-15T13:06:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.522 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.522 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:31.522 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978969+0000", 2024-05-15T13:06:31.522 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4620000000000001e-06 2024-05-15T13:06:31.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.522 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.522 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:31.522 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978977+0000", 2024-05-15T13:06:31.522 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8019999999999992e-06 2024-05-15T13:06:31.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.522 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.522 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:31.522 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.978983+0000", 2024-05-15T13:06:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4309999999999999e-06 2024-05-15T13:06:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.988763+0000", 2024-05-15T13:06:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0097792399999999998 2024-05-15T13:06:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.988898+0000", 2024-05-15T13:06:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013542400000000001 2024-05-15T13:06:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,8", 2024-05-15T13:06:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.989013+0000", 2024-05-15T13:06:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000115254 2024-05-15T13:06:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:06:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:19.994073+0000", 2024-05-15T13:06:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050596740000000001 2024-05-15T13:06:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.000489+0000", 2024-05-15T13:06:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0064156689999999997 2024-05-15T13:06:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.041979+0000", 2024-05-15T13:06:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.041490064 2024-05-15T13:06:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.042014+0000", 2024-05-15T13:06:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5497999999999998e-05 2024-05-15T13:06:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:20.042062+0000", 2024-05-15T13:06:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7360000000000001e-05 2024-05-15T13:06:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:167639 4.c e371/317 4:30d68d4c:::benchmark_data_smithi177_99756_object47667:head v 371'10508, mlcod=371'10505)", 2024-05-15T13:06:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:54.908023+0000", 2024-05-15T13:06:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 336.58033396899998, 2024-05-15T13:06:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.058411269000000002, 2024-05-15T13:06:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908023+0000", 2024-05-15T13:06:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908023+0000", 2024-05-15T13:06:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908023+0000", 2024-05-15T13:06:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7700000000000004e-07 2024-05-15T13:06:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908029+0000", 2024-05-15T13:06:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7420000000000003e-06 2024-05-15T13:06:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908036+0000", 2024-05-15T13:06:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7039999999999997e-06 2024-05-15T13:06:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908041+0000", 2024-05-15T13:06:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2970000000000003e-06 2024-05-15T13:06:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908089+0000", 2024-05-15T13:06:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8225e-05 2024-05-15T13:06:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908115+0000", 2024-05-15T13:06:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5627e-05 2024-05-15T13:06:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.966395+0000", 2024-05-15T13:06:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.058280294000000003 2024-05-15T13:06:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.966434+0000", 2024-05-15T13:06:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8702999999999999e-05 2024-05-15T13:06:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:167636 4.4 e371/317 4:2330c8ea:::benchmark_data_smithi177_99756_object47664:head v 371'10448, mlcod=371'10446)", 2024-05-15T13:06:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:54.908071+0000", 2024-05-15T13:06:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 336.58028587400003, 2024-05-15T13:06:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.064628550000000007, 2024-05-15T13:06:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908071+0000", 2024-05-15T13:06:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908071+0000", 2024-05-15T13:06:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908071+0000", 2024-05-15T13:06:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4000000000000002e-07 2024-05-15T13:06:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908075+0000", 2024-05-15T13:06:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4079999999999998e-06 2024-05-15T13:06:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908082+0000", 2024-05-15T13:06:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2609999999999996e-06 2024-05-15T13:06:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908087+0000", 2024-05-15T13:06:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1080000000000001e-06 2024-05-15T13:06:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908132+0000", 2024-05-15T13:06:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5312000000000003e-05 2024-05-15T13:06:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.908151+0000", 2024-05-15T13:06:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8536999999999999e-05 2024-05-15T13:06:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.972658+0000", 2024-05-15T13:06:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.064507007000000005 2024-05-15T13:06:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.972699+0000", 2024-05-15T13:06:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1477000000000003e-05 2024-05-15T13:06:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:167640 4.c e371/317 4:3084d351:::benchmark_data_smithi177_99756_object47668:head v 371'10509, mlcod=371'10505)", 2024-05-15T13:06:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:00:54.909239+0000", 2024-05-15T13:06:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 336.57911745899997, 2024-05-15T13:06:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.065503817000000006, 2024-05-15T13:06:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.909239+0000", 2024-05-15T13:06:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.909239+0000", 2024-05-15T13:06:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.909240+0000", 2024-05-15T13:06:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8100000000000003e-07 2024-05-15T13:06:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.909245+0000", 2024-05-15T13:06:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1000000000000003e-06 2024-05-15T13:06:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.909256+0000", 2024-05-15T13:06:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1409e-05 2024-05-15T13:06:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.909263+0000", 2024-05-15T13:06:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5350000000000001e-06 2024-05-15T13:06:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.912707+0000", 2024-05-15T13:06:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003444145 2024-05-15T13:06:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.912727+0000", 2024-05-15T13:06:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0013e-05 2024-05-15T13:06:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.974702+0000", 2024-05-15T13:06:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.061974896000000002 2024-05-15T13:06:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:00:54.974743+0000", 2024-05-15T13:06:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1238000000000001e-05 2024-05-15T13:06:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:195002 4.0 4:0a6de0aa:::benchmark_data_smithi177_99756_object75030:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:06:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:39.022199+0000", 2024-05-15T13:06:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 292.46615782600003, 2024-05-15T13:06:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.061146316999999999, 2024-05-15T13:06:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:06:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:06:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 195002 2024-05-15T13:06:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.022199+0000", 2024-05-15T13:06:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.022199+0000", 2024-05-15T13:06:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.022199+0000", 2024-05-15T13:06:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2600000000000002e-07 2024-05-15T13:06:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.022203+0000", 2024-05-15T13:06:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2279999999999998e-06 2024-05-15T13:06:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.022212+0000", 2024-05-15T13:06:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4679999999999998e-06 2024-05-15T13:06:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.022219+0000", 2024-05-15T13:06:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2110000000000001e-06 2024-05-15T13:06:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.028574+0000", 2024-05-15T13:06:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0063544710000000004 2024-05-15T13:06:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.028665+0000", 2024-05-15T13:06:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1428000000000006e-05 2024-05-15T13:06:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,8", 2024-05-15T13:06:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.028762+0000", 2024-05-15T13:06:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7356999999999996e-05 2024-05-15T13:06:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:06:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.036978+0000", 2024-05-15T13:06:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0082155570000000001 2024-05-15T13:06:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.051023+0000", 2024-05-15T13:06:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014045095 2024-05-15T13:06:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.083238+0000", 2024-05-15T13:06:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.032215150999999997 2024-05-15T13:06:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.083297+0000", 2024-05-15T13:06:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8390000000000002e-05 2024-05-15T13:06:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.083345+0000", 2024-05-15T13:06:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8435e-05 2024-05-15T13:06:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:195093 4.f e385/343 4:f1629b82:::benchmark_data_smithi177_99756_object75121:head v 385'11904, mlcod=333'5297)", 2024-05-15T13:06:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:39.173239+0000", 2024-05-15T13:06:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 292.31511726799999, 2024-05-15T13:06:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.156115744, 2024-05-15T13:06:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.173239+0000", 2024-05-15T13:06:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.173239+0000", 2024-05-15T13:06:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.173240+0000", 2024-05-15T13:06:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5700000000000002e-07 2024-05-15T13:06:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.173246+0000", 2024-05-15T13:06:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.592e-06 2024-05-15T13:06:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.173253+0000", 2024-05-15T13:06:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0419999999999997e-06 2024-05-15T13:06:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.173258+0000", 2024-05-15T13:06:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3870000000000003e-06 2024-05-15T13:06:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.173307+0000", 2024-05-15T13:06:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9376000000000003e-05 2024-05-15T13:06:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.173326+0000", 2024-05-15T13:06:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9045000000000001e-05 2024-05-15T13:06:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329308+0000", 2024-05-15T13:06:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.15598142600000001 2024-05-15T13:06:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329355+0000", 2024-05-15T13:06:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7219000000000003e-05 2024-05-15T13:06:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:195062 4.9 e385/343 4:9dfccf82:::benchmark_data_smithi177_99756_object75090:head v 385'12098, mlcod=330'4626)", 2024-05-15T13:06:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:39.178226+0000", 2024-05-15T13:06:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 292.31013083699997, 2024-05-15T13:06:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16531588999999999, 2024-05-15T13:06:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.178226+0000", 2024-05-15T13:06:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.178226+0000", 2024-05-15T13:06:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.178227+0000", 2024-05-15T13:06:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7699999999999992e-07 2024-05-15T13:06:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.178236+0000", 2024-05-15T13:06:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0429999999999996e-06 2024-05-15T13:06:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.178248+0000", 2024-05-15T13:06:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2438e-05 2024-05-15T13:06:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.178257+0000", 2024-05-15T13:06:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0550000000000005e-06 2024-05-15T13:06:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.178314+0000", 2024-05-15T13:06:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6331e-05 2024-05-15T13:06:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.178342+0000", 2024-05-15T13:06:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8101000000000001e-05 2024-05-15T13:06:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.343498+0000", 2024-05-15T13:06:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16515676000000001 2024-05-15T13:06:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.343542+0000", 2024-05-15T13:06:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3185000000000001e-05 2024-05-15T13:06:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:195097 4.0 4:01df6f19:::benchmark_data_smithi177_99756_object75125:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:06:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:39.179776+0000", 2024-05-15T13:06:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 292.30858085599999, 2024-05-15T13:06:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16930087199999999, 2024-05-15T13:06:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:06:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:06:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 195097 2024-05-15T13:06:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.179776+0000", 2024-05-15T13:06:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.179776+0000", 2024-05-15T13:06:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.179777+0000", 2024-05-15T13:06:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5400000000000001e-06 2024-05-15T13:06:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.179783+0000", 2024-05-15T13:06:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.254e-06 2024-05-15T13:06:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329321+0000", 2024-05-15T13:06:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.14953818699999999 2024-05-15T13:06:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329331+0000", 2024-05-15T13:06:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0556e-05 2024-05-15T13:06:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329551+0000", 2024-05-15T13:06:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000219385 2024-05-15T13:06:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329679+0000", 2024-05-15T13:06:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000127961 2024-05-15T13:06:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,8", 2024-05-15T13:06:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329796+0000", 2024-05-15T13:06:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000117175 2024-05-15T13:06:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.344048+0000", 2024-05-15T13:06:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014252453999999999 2024-05-15T13:06:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.344176+0000", 2024-05-15T13:06:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012799999999999999 2024-05-15T13:06:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:06:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.348983+0000", 2024-05-15T13:06:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0048068659999999999 2024-05-15T13:06:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.349033+0000", 2024-05-15T13:06:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9848000000000001e-05 2024-05-15T13:06:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.349077+0000", 2024-05-15T13:06:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3646000000000002e-05 2024-05-15T13:06:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:195098 4.f e385/343 4:f78a4cdc:::benchmark_data_smithi177_99756_object75126:head v 385'11905, mlcod=333'5297)", 2024-05-15T13:06:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:39.180038+0000", 2024-05-15T13:06:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 292.30831882199999, 2024-05-15T13:06:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.158549575, 2024-05-15T13:06:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.180038+0000", 2024-05-15T13:06:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.180038+0000", 2024-05-15T13:06:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.180038+0000", 2024-05-15T13:06:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3099999999999997e-07 2024-05-15T13:06:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.180044+0000", 2024-05-15T13:06:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7130000000000002e-06 2024-05-15T13:06:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329299+0000", 2024-05-15T13:06:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.14925477400000001 2024-05-15T13:06:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329313+0000", 2024-05-15T13:06:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3796e-05 2024-05-15T13:06:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329383+0000", 2024-05-15T13:06:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0069999999999998e-05 2024-05-15T13:06:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329410+0000", 2024-05-15T13:06:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.694e-05 2024-05-15T13:06:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.338543+0000", 2024-05-15T13:06:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0091335180000000002 2024-05-15T13:06:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.338587+0000", 2024-05-15T13:06:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4132999999999999e-05 2024-05-15T13:06:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:199254 4.0 4:029e2200:::benchmark_data_smithi177_99756_object79282:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-05-15T13:06:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:45.214841+0000", 2024-05-15T13:06:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 286.27351513399998, 2024-05-15T13:06:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.054510022999999998, 2024-05-15T13:06:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:06:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:06:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 199254 2024-05-15T13:06:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.214841+0000", 2024-05-15T13:06:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.214841+0000", 2024-05-15T13:06:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.214842+0000", 2024-05-15T13:06:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0100000000000005e-07 2024-05-15T13:06:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.214845+0000", 2024-05-15T13:06:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.326e-06 2024-05-15T13:06:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.214852+0000", 2024-05-15T13:06:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8109999999999997e-06 2024-05-15T13:06:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.214858+0000", 2024-05-15T13:06:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5600000000000001e-06 2024-05-15T13:06:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.220146+0000", 2024-05-15T13:06:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0052882479999999997 2024-05-15T13:06:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.220279+0000", 2024-05-15T13:06:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000133526 2024-05-15T13:06:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,8", 2024-05-15T13:06:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.220403+0000", 2024-05-15T13:06:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012339800000000001 2024-05-15T13:06:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:06:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.237564+0000", 2024-05-15T13:06:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.017161372000000001 2024-05-15T13:06:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.237601+0000", 2024-05-15T13:06:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6820000000000003e-05 2024-05-15T13:06:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.269244+0000", 2024-05-15T13:06:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.031643114 2024-05-15T13:06:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.269302+0000", 2024-05-15T13:06:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7896000000000003e-05 2024-05-15T13:06:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:45.269351+0000", 2024-05-15T13:06:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9450999999999998e-05 2024-05-15T13:06:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.15540.0:123457 5.1 e448/418 5:8c71fcbf:::benchmark_data_smithi177_109991_object123456:head v 448'7711, mlcod=448'7700)", 2024-05-15T13:06:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:04:39.633713+0000", 2024-05-15T13:06:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 111.854643223, 2024-05-15T13:06:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.040274206, 2024-05-15T13:06:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:39.633713+0000", 2024-05-15T13:06:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:39.633713+0000", 2024-05-15T13:06:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:39.633714+0000", 2024-05-15T13:06:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1400000000000001e-07 2024-05-15T13:06:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:39.633790+0000", 2024-05-15T13:06:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6822000000000004e-05 2024-05-15T13:06:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:39.633797+0000", 2024-05-15T13:06:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.285e-06 2024-05-15T13:06:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:39.633800+0000", 2024-05-15T13:06:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4249999999999998e-06 2024-05-15T13:06:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:39.663249+0000", 2024-05-15T13:06:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.029448888999999999 2024-05-15T13:06:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:39.663269+0000", 2024-05-15T13:06:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9667999999999999e-05 2024-05-15T13:06:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:39.673927+0000", 2024-05-15T13:06:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010658590000000001 2024-05-15T13:06:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:39.673988+0000", 2024-05-15T13:06:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0312999999999997e-05 2024-05-15T13:06:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.15540.0:123496 5.1 e448/418 5:899d129b:::benchmark_data_smithi177_109991_object123495:head v 448'7713, mlcod=448'7700)", 2024-05-15T13:06:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:04:39.633925+0000", 2024-05-15T13:06:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 111.854431581, 2024-05-15T13:06:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.043988995000000003, 2024-05-15T13:06:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:39.633925+0000", 2024-05-15T13:06:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:39.633925+0000", 2024-05-15T13:06:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:39.633925+0000", 2024-05-15T13:06:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8700000000000002e-07 2024-05-15T13:06:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:39.634045+0000", 2024-05-15T13:06:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000119361 2024-05-15T13:06:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:39.634055+0000", 2024-05-15T13:06:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9310000000000003e-06 2024-05-15T13:06:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:39.634062+0000", 2024-05-15T13:06:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5000000000000002e-06 2024-05-15T13:06:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:39.670264+0000", 2024-05-15T13:06:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.036201891 2024-05-15T13:06:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:39.670307+0000", 2024-05-15T13:06:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2755e-05 2024-05-15T13:06:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:39.677865+0000", 2024-05-15T13:06:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0075582940000000001 2024-05-15T13:06:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:04:39.677914+0000", 2024-05-15T13:06:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8976e-05 2024-05-15T13:06:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.15540.0:205586 5.1 e468/418 5:895502a5:::benchmark_data_smithi177_109991_object70395:head v 468'12904, mlcod=468'12899)", 2024-05-15T13:06:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:06:06.422213+0000", 2024-05-15T13:06:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 25.066143786000001, 2024-05-15T13:06:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.034486656999999997, 2024-05-15T13:06:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:06:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:06:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:06:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:06.422213+0000", 2024-05-15T13:06:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:06:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:06.422213+0000", 2024-05-15T13:06:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:06:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:06:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:06.422215+0000", 2024-05-15T13:06:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9769999999999999e-06 2024-05-15T13:06:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:06:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:06.422230+0000", 2024-05-15T13:06:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5328999999999998e-05 2024-05-15T13:06:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:06:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:06.422257+0000", 2024-05-15T13:06:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6659e-05 2024-05-15T13:06:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:06:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:06.422270+0000", 2024-05-15T13:06:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3023999999999999e-05 2024-05-15T13:06:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:06:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:06.431353+0000", 2024-05-15T13:06:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090836170000000004 2024-05-15T13:06:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:06:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:06.431379+0000", 2024-05-15T13:06:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5904000000000001e-05 2024-05-15T13:06:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:06:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:06.456668+0000", 2024-05-15T13:06:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.025288386999999999 2024-05-15T13:06:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:06:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:06:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:06:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:06.456699+0000", 2024-05-15T13:06:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1760000000000001e-05 2024-05-15T13:06:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.584 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:31.584 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.584 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:06:31.584 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:06:31.584 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:06:31.585 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:31 smithi120 ceph-mon[43961]: pgmap v1277: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 867 op/s 2024-05-15T13:06:31.585 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:31 smithi027 ceph-mon[39795]: pgmap v1277: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 867 op/s 2024-05-15T13:06:32.212 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:06:32.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:32.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:06:32.320 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:06:32.371 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:33.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:33 smithi177 ceph-mon[44303]: pgmap v1278: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.8 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 802 op/s 2024-05-15T13:06:33.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:33 smithi027 ceph-mon[39795]: pgmap v1278: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.8 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 802 op/s 2024-05-15T13:06:33.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:33 smithi120 ceph-mon[43961]: pgmap v1278: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.8 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 802 op/s 2024-05-15T13:06:33.731 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-05-15T13:06:33.830 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:06:33.831 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:06:33.835 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:06:33.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:33.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:06:33.942 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:06:34.007 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:34.041 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:06:34.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:34.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:06:34.112 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:06:34.148 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:06:34.177 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:34.383 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:06:34.416 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:34 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2350649329' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:06:34.416 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:34 smithi027 ceph-mon[39795]: pgmap v1279: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.7 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 842 op/s 2024-05-15T13:06:34.456 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:06:34.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:34.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:06:34.560 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:06:34.607 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:34.648 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:06:34.653 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:34 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2350649329' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:06:34.653 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:34 smithi177 ceph-mon[44303]: pgmap v1279: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.7 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 842 op/s 2024-05-15T13:06:34.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:34.654 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:06:34.755 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:06:34.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:34 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2350649329' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:06:34.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:34 smithi120 ceph-mon[43961]: pgmap v1279: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.7 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 842 op/s 2024-05-15T13:06:34.820 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:34.852 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:06:34.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:34.859 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:06:34.960 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:06:35.002 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:35.038 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:06:35.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:35.040 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:06:35.141 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:06:35.190 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:35.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:35 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:06:35.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:35 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2350649329' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:06:35.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:35 smithi177 ceph-mon[44303]: osdmap e473: 12 total, 12 up, 11 in 2024-05-15T13:06:35.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:35 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:06:35.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:35 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2350649329' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:06:35.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:35 smithi120 ceph-mon[43961]: osdmap e473: 12 total, 12 up, 11 in 2024-05-15T13:06:35.753 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:06:35.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:35.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:06:35.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:35 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:06:35.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:35 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2350649329' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:06:35.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:35 smithi027 ceph-mon[39795]: osdmap e473: 12 total, 12 up, 11 in 2024-05-15T13:06:35.861 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:06:35.894 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:36.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:36 smithi027 ceph-mon[39795]: pgmap v1281: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 730 op/s 2024-05-15T13:06:36.535 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:06:36.535 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [ 2024-05-15T13:06:36.535 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.535 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(osd.5.0:115567 5.f e473/418 5:f8bce070:::benchmark_data_smithi177_109991_object126984:head v 473'15922, mlcod=473'15899)", 2024-05-15T13:06:36.535 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:06:36.517615+0000", 2024-05-15T13:06:36.535 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.023922783, 2024-05-15T13:06:36.535 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.023950262999999999, 2024-05-15T13:06:36.535 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:36.536 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "started", 2024-05-15T13:06:36.536 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:36.536 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.536 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:36.536 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.517615+0000", 2024-05-15T13:06:36.536 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:36.536 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.536 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.536 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:36.536 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.517615+0000", 2024-05-15T13:06:36.536 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:36.536 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.536 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.536 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:36.536 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.517616+0000", 2024-05-15T13:06:36.537 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.1900000000000002e-07 2024-05-15T13:06:36.537 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.537 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.537 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:36.537 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.517625+0000", 2024-05-15T13:06:36.537 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.4640000000000006e-06 2024-05-15T13:06:36.537 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.537 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.537 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:36.537 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.517637+0000", 2024-05-15T13:06:36.537 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2330000000000001e-05 2024-05-15T13:06:36.537 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.537 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.537 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:36.538 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.517644+0000", 2024-05-15T13:06:36.538 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3660000000000003e-06 2024-05-15T13:06:36.538 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.538 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.538 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:36.538 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.525495+0000", 2024-05-15T13:06:36.538 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0078508850000000002 2024-05-15T13:06:36.538 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.538 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.538 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:06:36.538 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.525518+0000", 2024-05-15T13:06:36.538 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.3308000000000001e-05 2024-05-15T13:06:36.538 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:36.538 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:36.539 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:36.539 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.539 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.539 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(osd.5.0:115568 5.f e473/418 5:f8bcecca:::benchmark_data_smithi177_109991_object116225:head v 473'15923, mlcod=473'15899)", 2024-05-15T13:06:36.539 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:06:36.522033+0000", 2024-05-15T13:06:36.539 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.019504984, 2024-05-15T13:06:36.539 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.019599335999999998, 2024-05-15T13:06:36.539 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:36.539 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "started", 2024-05-15T13:06:36.539 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:36.539 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.539 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:36.539 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.522033+0000", 2024-05-15T13:06:36.539 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:36.540 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.540 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.540 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:36.540 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.522033+0000", 2024-05-15T13:06:36.540 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:36.540 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.540 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.540 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:36.540 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.522034+0000", 2024-05-15T13:06:36.540 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.5000000000000001e-07 2024-05-15T13:06:36.540 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.540 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.540 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:36.540 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.522042+0000", 2024-05-15T13:06:36.541 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.174e-06 2024-05-15T13:06:36.541 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.541 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.541 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:36.541 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.522053+0000", 2024-05-15T13:06:36.541 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1051e-05 2024-05-15T13:06:36.541 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.541 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.541 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:36.541 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.522061+0000", 2024-05-15T13:06:36.541 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0690000000000002e-06 2024-05-15T13:06:36.541 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.541 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.541 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:36.542 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.529640+0000", 2024-05-15T13:06:36.542 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.007578263 2024-05-15T13:06:36.542 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.542 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.542 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:06:36.542 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.529652+0000", 2024-05-15T13:06:36.542 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1934e-05 2024-05-15T13:06:36.542 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:36.542 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:36.542 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:36.542 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.542 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.543 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(osd.5.0:115569 5.f e473/418 5:f8bd0964:::benchmark_data_smithi177_109991_object104250:head v 473'15924, mlcod=473'15899)", 2024-05-15T13:06:36.543 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:06:36.526515+0000", 2024-05-15T13:06:36.543 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.015022977999999999, 2024-05-15T13:06:36.543 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.015184929999999999, 2024-05-15T13:06:36.543 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:36.543 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "started", 2024-05-15T13:06:36.543 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:36.543 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.543 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:36.543 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.526515+0000", 2024-05-15T13:06:36.543 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:36.543 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.543 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.543 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:36.544 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.526515+0000", 2024-05-15T13:06:36.544 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:36.544 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.544 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.544 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:36.544 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.526516+0000", 2024-05-15T13:06:36.544 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8999999999999995e-07 2024-05-15T13:06:36.544 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.544 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.544 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:36.544 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.526525+0000", 2024-05-15T13:06:36.544 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.5699999999999993e-06 2024-05-15T13:06:36.544 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.544 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.545 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:36.545 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.526539+0000", 2024-05-15T13:06:36.545 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4068e-05 2024-05-15T13:06:36.545 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.545 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.545 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:36.545 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.526546+0000", 2024-05-15T13:06:36.545 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7190000000000001e-06 2024-05-15T13:06:36.545 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.545 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.545 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:36.545 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.534158+0000", 2024-05-15T13:06:36.545 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0076116159999999999 2024-05-15T13:06:36.545 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.546 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.546 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:06:36.546 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.534178+0000", 2024-05-15T13:06:36.546 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0299999999999999e-05 2024-05-15T13:06:36.546 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:36.546 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:36.546 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:36.546 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.546 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.546 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(osd.5.0:115570 5.f e473/418 5:f8c0a63c:::benchmark_data_smithi177_109991_object114008:head v 473'15925, mlcod=473'15899)", 2024-05-15T13:06:36.546 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:06:36.530529+0000", 2024-05-15T13:06:36.546 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.01100946, 2024-05-15T13:06:36.546 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.011221493000000001, 2024-05-15T13:06:36.547 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:36.547 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "started", 2024-05-15T13:06:36.547 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:36.547 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.547 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:36.547 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.530529+0000", 2024-05-15T13:06:36.547 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:36.547 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.547 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.547 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:36.547 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.530529+0000", 2024-05-15T13:06:36.547 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:36.547 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.548 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.548 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:36.548 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.530529+0000", 2024-05-15T13:06:36.548 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2699999999999999e-07 2024-05-15T13:06:36.548 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.548 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.548 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:36.548 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.530536+0000", 2024-05-15T13:06:36.548 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.161e-06 2024-05-15T13:06:36.548 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.548 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.548 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:36.548 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.530548+0000", 2024-05-15T13:06:36.549 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2819999999999999e-05 2024-05-15T13:06:36.549 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.549 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.549 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:36.549 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.530555+0000", 2024-05-15T13:06:36.549 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8090000000000004e-06 2024-05-15T13:06:36.549 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.549 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.549 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:36.549 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.538786+0000", 2024-05-15T13:06:36.549 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0082309670000000005 2024-05-15T13:06:36.549 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.549 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.549 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:06:36.549 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.538802+0000", 2024-05-15T13:06:36.550 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5616000000000001e-05 2024-05-15T13:06:36.550 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:36.550 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:36.550 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:36.550 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.550 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.550 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.15540.0:229794 5.f e473/418)", 2024-05-15T13:06:36.550 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:06:36.536185+0000", 2024-05-15T13:06:36.550 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.0053529069999999996, 2024-05-15T13:06:36.550 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0056090480000000002, 2024-05-15T13:06:36.550 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:36.550 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "queued for pg", 2024-05-15T13:06:36.550 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:36.551 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.551 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:36.551 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.536185+0000", 2024-05-15T13:06:36.551 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:36.551 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.551 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.551 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:36.551 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.536185+0000", 2024-05-15T13:06:36.551 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:36.551 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.551 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.551 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:36.551 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.536186+0000", 2024-05-15T13:06:36.551 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.5199999999999996e-07 2024-05-15T13:06:36.551 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.552 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.552 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:36.552 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.536194+0000", 2024-05-15T13:06:36.552 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8299999999999996e-06 2024-05-15T13:06:36.552 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.552 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.552 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:36.552 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.536204+0000", 2024-05-15T13:06:36.552 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0549e-05 2024-05-15T13:06:36.552 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.552 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.552 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:36.552 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.536212+0000", 2024-05-15T13:06:36.552 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9419999999999995e-06 2024-05-15T13:06:36.553 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:36.553 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:36.553 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:36.553 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.553 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.553 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.15540.0:229870 5.c e473/418 5:3e123a0f:::benchmark_data_smithi177_109991_object94679:head v 473'14415, mlcod=473'14413)", 2024-05-15T13:06:36.553 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:06:36.539534+0000", 2024-05-15T13:06:36.553 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.0020036759999999998, 2024-05-15T13:06:36.553 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0022883700000000001, 2024-05-15T13:06:36.553 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:36.553 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "started", 2024-05-15T13:06:36.553 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:36.553 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.553 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:36.553 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.539534+0000", 2024-05-15T13:06:36.554 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:36.554 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.554 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.554 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:36.554 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.539534+0000", 2024-05-15T13:06:36.554 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:36.554 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.554 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.554 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:36.554 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.539535+0000", 2024-05-15T13:06:36.554 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.64e-07 2024-05-15T13:06:36.554 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.554 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.554 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:36.555 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.539543+0000", 2024-05-15T13:06:36.555 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0709999999999997e-06 2024-05-15T13:06:36.555 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.555 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.555 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:36.555 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.539557+0000", 2024-05-15T13:06:36.555 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4236999999999999e-05 2024-05-15T13:06:36.555 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.555 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.555 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:36.555 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.539565+0000", 2024-05-15T13:06:36.555 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2190000000000004e-06 2024-05-15T13:06:36.555 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.555 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.555 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:36.556 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.539624+0000", 2024-05-15T13:06:36.556 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8638000000000003e-05 2024-05-15T13:06:36.556 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:36.556 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:36.556 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:06:36.556 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:06:36.539645+0000", 2024-05-15T13:06:36.556 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1693999999999999e-05 2024-05-15T13:06:36.556 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:36.556 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:36.556 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:36.556 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:36.556 INFO:teuthology.orchestra.run.smithi120.stdout: ], 2024-05-15T13:06:36.556 INFO:teuthology.orchestra.run.smithi120.stdout: "num_ops": 6 2024-05-15T13:06:36.556 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:06:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:36 smithi177 ceph-mon[44303]: pgmap v1281: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 730 op/s 2024-05-15T13:06:36.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:36 smithi120 ceph-mon[43961]: pgmap v1281: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 730 op/s 2024-05-15T13:06:37.278 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:06:37.367 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-05-15T13:06:37.673 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:06:37.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:37.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:06:37.772 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:06:37.778 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:06:37.827 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:38.085 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:37 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1831845102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:06:38.108 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:06:38.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:38.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:06:38.219 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:06:38.250 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:06:38.250 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:06:38.250 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:06:38.250 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:06:38.262 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:38.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:37 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1831845102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:06:38.273 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:06:38.299 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:06:38.353 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:06:38.358 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:06:38.393 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:06:38.427 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:06:38.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:37 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1831845102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:06:38.457 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:06:38.487 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:06:38.488 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:06:38.488 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:06:38.488 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:06:38.488 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:06:38.488 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:06:38.488 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:06:38.488 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:06:38.488 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:06:38.489 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:06:38.489 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:06:38.489 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:06:38.489 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:06:38.600 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:06:38.729 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:06:38.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:38.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:06:38.834 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:06:38.880 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:38.985 INFO:teuthology.orchestra.run.smithi027.stderr:set pool 1 pg_num to 11 2024-05-15T13:06:39.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:38 smithi027 ceph-mon[39795]: pgmap v1282: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.5 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 643 op/s 2024-05-15T13:06:39.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:38 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3123964888' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-05-15T13:06:39.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:38 smithi120 ceph-mon[43961]: pgmap v1282: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.5 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 643 op/s 2024-05-15T13:06:39.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:38 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3123964888' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-05-15T13:06:39.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:38 smithi177 ceph-mon[44303]: pgmap v1282: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.5 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 643 op/s 2024-05-15T13:06:39.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:38 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3123964888' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-05-15T13:06:39.990 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:40 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3123964888' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-05-15T13:06:39.990 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:40 smithi027 ceph-mon[39795]: osdmap e474: 12 total, 12 up, 11 in 2024-05-15T13:06:40.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:40 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3123964888' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-05-15T13:06:40.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:40 smithi120 ceph-mon[43961]: osdmap e474: 12 total, 12 up, 11 in 2024-05-15T13:06:40.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:40 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3123964888' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-05-15T13:06:40.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:40 smithi177 ceph-mon[44303]: osdmap e474: 12 total, 12 up, 11 in 2024-05-15T13:06:41.247 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:06:41.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:41.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:06:41.251 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:41 smithi120 ceph-mon[43961]: pgmap v1284: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.4 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 748 op/s 2024-05-15T13:06:41.251 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:41 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-05-15T13:06:41.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:41 smithi027 ceph-mon[39795]: pgmap v1284: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.4 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 748 op/s 2024-05-15T13:06:41.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:41 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-05-15T13:06:41.352 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:06:41.394 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:41.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:41 smithi177 ceph-mon[44303]: pgmap v1284: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.4 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 748 op/s 2024-05-15T13:06:41.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:41 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-05-15T13:06:42.001 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:42 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-05-15T13:06:42.001 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:42 smithi027 ceph-mon[39795]: osdmap e475: 12 total, 12 up, 11 in 2024-05-15T13:06:42.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:42 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-05-15T13:06:42.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:42 smithi120 ceph-mon[43961]: osdmap e475: 12 total, 12 up, 11 in 2024-05-15T13:06:42.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:42 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-05-15T13:06:42.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:42 smithi177 ceph-mon[44303]: osdmap e475: 12 total, 12 up, 11 in 2024-05-15T13:06:42.676 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:06:43.035 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:06:43.062 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:06:43.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:43 smithi027 ceph-mon[39795]: osdmap e476: 12 total, 12 up, 11 in 2024-05-15T13:06:43.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:43 smithi027 ceph-mon[39795]: pgmap v1287: 27 pgs: 1 peering, 10 unknown, 1 active+clean+scrubbing+deep, 15 active+clean; 2.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 693 op/s 2024-05-15T13:06:43.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:43 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-05-15T13:06:43.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:43 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3248201149' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:06:43.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:43 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:06:43.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:43 smithi120 ceph-mon[43961]: osdmap e476: 12 total, 12 up, 11 in 2024-05-15T13:06:43.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:43 smithi120 ceph-mon[43961]: pgmap v1287: 27 pgs: 1 peering, 10 unknown, 1 active+clean+scrubbing+deep, 15 active+clean; 2.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 693 op/s 2024-05-15T13:06:43.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:43 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-05-15T13:06:43.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:43 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3248201149' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:06:43.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:43 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:06:43.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:43 smithi177 ceph-mon[44303]: osdmap e476: 12 total, 12 up, 11 in 2024-05-15T13:06:43.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:43 smithi177 ceph-mon[44303]: pgmap v1287: 27 pgs: 1 peering, 10 unknown, 1 active+clean+scrubbing+deep, 15 active+clean; 2.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 693 op/s 2024-05-15T13:06:43.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:43 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-05-15T13:06:43.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:43 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3248201149' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:06:43.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:43 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:06:43.756 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:06:43.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:43.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:06:43.865 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:06:43.940 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:44.024 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:44 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:06:44.024 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:44 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-05-15T13:06:44.024 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:44 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:06:44.024 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:44 smithi027 ceph-mon[39795]: osdmap e477: 12 total, 12 up, 11 in 2024-05-15T13:06:44.025 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:44 smithi027 ceph-mon[39795]: osdmap e478: 12 total, 12 up, 11 in 2024-05-15T13:06:44.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:44 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:06:44.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:44 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-05-15T13:06:44.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:44 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:06:44.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:44 smithi120 ceph-mon[43961]: osdmap e477: 12 total, 12 up, 11 in 2024-05-15T13:06:44.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:44 smithi120 ceph-mon[43961]: osdmap e478: 12 total, 12 up, 11 in 2024-05-15T13:06:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:44 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:06:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:44 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-05-15T13:06:44.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:44 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:06:44.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:44 smithi177 ceph-mon[44303]: osdmap e477: 12 total, 12 up, 11 in 2024-05-15T13:06:44.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:44 smithi177 ceph-mon[44303]: osdmap e478: 12 total, 12 up, 11 in 2024-05-15T13:06:44.578 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:06:44.578 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:06:45.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:45 smithi177 ceph-mon[44303]: pgmap v1290: 27 pgs: 1 remapped+peering, 1 peering, 9 unknown, 16 active+clean; 2.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 678 op/s 2024-05-15T13:06:45.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:45 smithi177 ceph-mon[44303]: osdmap e479: 12 total, 12 up, 11 in 2024-05-15T13:06:45.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:45 smithi027 ceph-mon[39795]: pgmap v1290: 27 pgs: 1 remapped+peering, 1 peering, 9 unknown, 16 active+clean; 2.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 678 op/s 2024-05-15T13:06:45.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:45 smithi027 ceph-mon[39795]: osdmap e479: 12 total, 12 up, 11 in 2024-05-15T13:06:45.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:45 smithi120 ceph-mon[43961]: pgmap v1290: 27 pgs: 1 remapped+peering, 1 peering, 9 unknown, 16 active+clean; 2.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 678 op/s 2024-05-15T13:06:45.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:45 smithi120 ceph-mon[43961]: osdmap e479: 12 total, 12 up, 11 in 2024-05-15T13:06:45.630 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:06:45.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:45.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:06:45.676 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:06:45.677 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 0 1 2024-05-15T13:06:45.678 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-05-15T13:06:45.740 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:06:45.776 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:46.059 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:06:46.261 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:06:46.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:46.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:06:46.303 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:06:46.362 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:46 smithi120 ceph-mon[43961]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-05-15T13:06:46.362 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:46 smithi120 ceph-mon[43961]: osdmap e480: 12 total, 12 up, 11 in 2024-05-15T13:06:46.369 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:06:46.413 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:46 smithi027 ceph-mon[39795]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-05-15T13:06:46.413 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:46 smithi027 ceph-mon[39795]: osdmap e480: 12 total, 12 up, 11 in 2024-05-15T13:06:46.416 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:46.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:46 smithi177 ceph-mon[44303]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-05-15T13:06:46.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:46 smithi177 ceph-mon[44303]: osdmap e480: 12 total, 12 up, 11 in 2024-05-15T13:06:46.589 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:06:46.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:46.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:06:46.693 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:06:46.740 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:46.774 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:06:46.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:46.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:06:46.878 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:06:46.912 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:46.947 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:06:46.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:46.956 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:06:47.053 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:47 smithi177 ceph-mon[44303]: pgmap v1293: 27 pgs: 1 remapped+peering, 4 peering, 5 unknown, 17 active+clean; 2.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 12 KiB/s rd, 334 KiB/s wr, 799 op/s; 423 KiB/s, 1 objects/s recovering 2024-05-15T13:06:47.054 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:47 smithi177 ceph-mon[44303]: osdmap e481: 12 total, 12 up, 11 in 2024-05-15T13:06:47.057 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:06:47.104 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:47.135 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:06:47.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:47.143 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:06:47.244 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:06:47.255 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:06:47.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:47 smithi027 ceph-mon[39795]: pgmap v1293: 27 pgs: 1 remapped+peering, 4 peering, 5 unknown, 17 active+clean; 2.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 12 KiB/s rd, 334 KiB/s wr, 799 op/s; 423 KiB/s, 1 objects/s recovering 2024-05-15T13:06:47.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:47 smithi027 ceph-mon[39795]: osdmap e481: 12 total, 12 up, 11 in 2024-05-15T13:06:47.288 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:47.342 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:06:47.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:47.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:06:47.449 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:06:47.494 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:47.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:47 smithi120 ceph-mon[43961]: pgmap v1293: 27 pgs: 1 remapped+peering, 4 peering, 5 unknown, 17 active+clean; 2.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 12 KiB/s rd, 334 KiB/s wr, 799 op/s; 423 KiB/s, 1 objects/s recovering 2024-05-15T13:06:47.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:47 smithi120 ceph-mon[43961]: osdmap e481: 12 total, 12 up, 11 in 2024-05-15T13:06:47.530 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:06:47.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:47.532 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:06:47.633 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:06:47.697 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:47.763 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:06:47.795 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:06:47.829 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:06:47.889 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:06:47.911 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:06:47.951 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:06:47.992 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:06:48.032 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:06:48.071 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:06:48.106 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:06:48.142 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:06:48.173 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:06:48.190 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:06:48.190 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:06:48.190 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:06:48.191 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:06:48.191 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:06:48.191 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:06:48.191 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:06:48.191 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:06:48.191 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:06:48.191 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:06:48.191 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:06:48.191 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:06:48.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:48 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/221002037' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:06:48.356 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:06:48.356 INFO:teuthology.orchestra.run.smithi120.stdout: "size": 20, 2024-05-15T13:06:48.356 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 600, 2024-05-15T13:06:48.356 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [ 2024-05-15T13:06:48.356 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.356 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:101379 4.1 4:8bcb24a5:::benchmark_data_smithi177_99756_object101378:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e343)", 2024-05-15T13:06:48.356 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:59:11.312757+0000", 2024-05-15T13:06:48.357 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 457.052348924, 2024-05-15T13:06:48.357 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.550042221, 2024-05-15T13:06:48.357 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:48.357 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:48.357 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:48.357 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:06:48.357 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:48.357 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 101379 2024-05-15T13:06:48.357 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.357 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:48.357 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.357 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:48.357 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312757+0000", 2024-05-15T13:06:48.358 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.358 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.358 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.358 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:48.358 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312757+0000", 2024-05-15T13:06:48.358 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.358 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.358 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.358 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:48.358 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312759+0000", 2024-05-15T13:06:48.358 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3349999999999999e-06 2024-05-15T13:06:48.359 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.359 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.359 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:48.359 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312913+0000", 2024-05-15T13:06:48.359 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000154656 2024-05-15T13:06:48.359 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.359 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.359 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:48.359 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312926+0000", 2024-05-15T13:06:48.359 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3294e-05 2024-05-15T13:06:48.359 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.359 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.359 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:48.359 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312938+0000", 2024-05-15T13:06:48.360 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1751e-05 2024-05-15T13:06:48.360 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.360 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.360 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.360 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313019+0000", 2024-05-15T13:06:48.360 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1163999999999997e-05 2024-05-15T13:06:48.360 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.360 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.360 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:06:48.360 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313029+0000", 2024-05-15T13:06:48.360 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0047999999999999e-05 2024-05-15T13:06:48.360 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.360 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.360 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.361 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.851467+0000", 2024-05-15T13:06:48.361 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5384380449999999 2024-05-15T13:06:48.361 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.361 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.361 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:06:48.361 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.853365+0000", 2024-05-15T13:06:48.361 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001897302 2024-05-15T13:06:48.361 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.361 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.361 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 5,6,11", 2024-05-15T13:06:48.361 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.853497+0000", 2024-05-15T13:06:48.361 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000132058 2024-05-15T13:06:48.361 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.361 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.362 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:48.362 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.860348+0000", 2024-05-15T13:06:48.362 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0068511919999999999 2024-05-15T13:06:48.362 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.362 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.362 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:48.362 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.860430+0000", 2024-05-15T13:06:48.362 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1646999999999999e-05 2024-05-15T13:06:48.362 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.362 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.362 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:06:48.362 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.860447+0000", 2024-05-15T13:06:48.362 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7697e-05 2024-05-15T13:06:48.363 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.363 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.363 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:48.363 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.860501+0000", 2024-05-15T13:06:48.363 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3378999999999998e-05 2024-05-15T13:06:48.363 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.363 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.363 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:06:48.363 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.860575+0000", 2024-05-15T13:06:48.363 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3875000000000007e-05 2024-05-15T13:06:48.363 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.363 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.363 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:48.363 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.862799+0000", 2024-05-15T13:06:48.364 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0022247780000000002 2024-05-15T13:06:48.364 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.364 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:48.364 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.364 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.364 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.364 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:101386 4.1 4:83191a75:::benchmark_data_smithi177_99756_object101385:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e343)", 2024-05-15T13:06:48.364 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:59:11.312977+0000", 2024-05-15T13:06:48.364 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 457.05212874, 2024-05-15T13:06:48.364 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5704636750000001, 2024-05-15T13:06:48.364 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:48.364 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:48.364 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:48.364 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:06:48.364 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:48.365 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 101386 2024-05-15T13:06:48.365 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.365 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:48.365 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.365 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:48.365 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312977+0000", 2024-05-15T13:06:48.365 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.365 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.365 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.365 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:48.365 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312977+0000", 2024-05-15T13:06:48.365 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.365 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.365 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.365 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:48.366 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312978+0000", 2024-05-15T13:06:48.366 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.68e-07 2024-05-15T13:06:48.366 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.366 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.366 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:48.366 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313011+0000", 2024-05-15T13:06:48.366 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.2957999999999999e-05 2024-05-15T13:06:48.366 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.366 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.366 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:48.366 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313020+0000", 2024-05-15T13:06:48.366 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8950000000000004e-06 2024-05-15T13:06:48.366 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.367 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.367 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:48.367 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313027+0000", 2024-05-15T13:06:48.367 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7900000000000002e-06 2024-05-15T13:06:48.367 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.367 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.367 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.367 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313080+0000", 2024-05-15T13:06:48.367 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3122999999999999e-05 2024-05-15T13:06:48.367 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.367 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.367 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:06:48.367 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313096+0000", 2024-05-15T13:06:48.367 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5934e-05 2024-05-15T13:06:48.367 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.368 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.368 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.368 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.862932+0000", 2024-05-15T13:06:48.368 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5498364360000001 2024-05-15T13:06:48.368 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.368 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.368 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:06:48.368 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.863088+0000", 2024-05-15T13:06:48.368 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015589900000000001 2024-05-15T13:06:48.368 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.368 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.368 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 5,6,11", 2024-05-15T13:06:48.369 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.863214+0000", 2024-05-15T13:06:48.369 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012642600000000001 2024-05-15T13:06:48.369 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.369 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.369 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:06:48.369 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.869892+0000", 2024-05-15T13:06:48.369 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0066774360000000001 2024-05-15T13:06:48.369 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.369 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.369 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:48.369 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.869952+0000", 2024-05-15T13:06:48.369 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.0081000000000003e-05 2024-05-15T13:06:48.369 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.369 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.370 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:48.370 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.874357+0000", 2024-05-15T13:06:48.370 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0044051089999999999 2024-05-15T13:06:48.370 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.370 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.370 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:48.370 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.881092+0000", 2024-05-15T13:06:48.370 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.006734711 2024-05-15T13:06:48.370 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.370 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.370 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:06:48.371 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.881135+0000", 2024-05-15T13:06:48.371 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2799000000000001e-05 2024-05-15T13:06:48.371 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.371 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.371 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:48.371 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.883441+0000", 2024-05-15T13:06:48.371 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0023064100000000001 2024-05-15T13:06:48.371 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.371 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:48.371 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.371 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.371 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.371 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:101387 4.1 4:83e2b5a9:::benchmark_data_smithi177_99756_object101386:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e343)", 2024-05-15T13:06:48.371 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:59:11.313041+0000", 2024-05-15T13:06:48.372 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 457.05206466999999, 2024-05-15T13:06:48.372 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.578328618, 2024-05-15T13:06:48.372 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:48.372 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:48.372 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:48.372 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:06:48.372 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:48.372 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 101387 2024-05-15T13:06:48.372 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.372 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:48.372 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.372 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:48.373 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313041+0000", 2024-05-15T13:06:48.373 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.373 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.373 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.373 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:48.373 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313041+0000", 2024-05-15T13:06:48.373 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.373 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.373 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.373 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:48.373 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313042+0000", 2024-05-15T13:06:48.373 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.5799999999999999e-07 2024-05-15T13:06:48.373 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.373 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.374 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:48.374 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313149+0000", 2024-05-15T13:06:48.374 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00010729 2024-05-15T13:06:48.374 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.374 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.374 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:48.374 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313159+0000", 2024-05-15T13:06:48.374 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0047000000000001e-05 2024-05-15T13:06:48.374 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.374 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.374 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:48.374 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313167+0000", 2024-05-15T13:06:48.374 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4789999999999999e-06 2024-05-15T13:06:48.375 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.375 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.375 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.375 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313224+0000", 2024-05-15T13:06:48.375 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6832000000000001e-05 2024-05-15T13:06:48.375 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.375 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.375 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:06:48.375 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313235+0000", 2024-05-15T13:06:48.375 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1086e-05 2024-05-15T13:06:48.375 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.375 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.376 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.376 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.866156+0000", 2024-05-15T13:06:48.376 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.552920962 2024-05-15T13:06:48.376 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.376 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.376 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:06:48.376 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.866295+0000", 2024-05-15T13:06:48.376 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013931800000000001 2024-05-15T13:06:48.376 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.376 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.376 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 5,6,11", 2024-05-15T13:06:48.376 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.866413+0000", 2024-05-15T13:06:48.377 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000118336 2024-05-15T13:06:48.377 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.377 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.377 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:06:48.377 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.870229+0000", 2024-05-15T13:06:48.377 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0038157120000000002 2024-05-15T13:06:48.377 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.377 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.377 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:48.377 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.874291+0000", 2024-05-15T13:06:48.377 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0040620220000000002 2024-05-15T13:06:48.377 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.377 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.378 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:48.378 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.874400+0000", 2024-05-15T13:06:48.378 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000108929 2024-05-15T13:06:48.378 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.378 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.378 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:48.378 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.887902+0000", 2024-05-15T13:06:48.378 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.013501701 2024-05-15T13:06:48.378 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.378 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.378 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:06:48.378 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.887972+0000", 2024-05-15T13:06:48.378 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0091000000000004e-05 2024-05-15T13:06:48.379 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.379 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.379 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:48.379 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.891370+0000", 2024-05-15T13:06:48.379 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0033981549999999999 2024-05-15T13:06:48.379 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.379 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:48.379 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.379 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.379 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.379 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:140738 4.1 4:859c5c94:::benchmark_data_smithi177_99756_object20766:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e357)", 2024-05-15T13:06:48.379 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:00:11.249829+0000", 2024-05-15T13:06:48.379 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 397.115276876, 2024-05-15T13:06:48.379 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.770064275, 2024-05-15T13:06:48.379 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:48.380 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:48.380 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:48.380 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:06:48.380 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:48.380 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 140738 2024-05-15T13:06:48.380 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.380 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:48.380 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.380 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:48.380 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:11.249829+0000", 2024-05-15T13:06:48.380 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.380 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.380 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.380 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:48.380 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:11.249829+0000", 2024-05-15T13:06:48.381 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.381 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.381 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.381 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:48.381 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:11.249830+0000", 2024-05-15T13:06:48.381 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.3799999999999998e-07 2024-05-15T13:06:48.381 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.381 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.381 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:48.381 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:11.249831+0000", 2024-05-15T13:06:48.381 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.8619999999999999e-06 2024-05-15T13:06:48.381 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.381 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.381 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:48.382 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:11.249837+0000", 2024-05-15T13:06:48.382 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.4330000000000003e-06 2024-05-15T13:06:48.382 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.382 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.382 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:48.382 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:11.249841+0000", 2024-05-15T13:06:48.382 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.8319999999999999e-06 2024-05-15T13:06:48.382 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.382 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.383 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.383 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:11.250009+0000", 2024-05-15T13:06:48.383 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00016829200000000001 2024-05-15T13:06:48.383 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.383 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.383 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:06:48.383 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:11.250061+0000", 2024-05-15T13:06:48.383 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2185999999999998e-05 2024-05-15T13:06:48.383 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.383 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.383 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.383 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:13.019878+0000", 2024-05-15T13:06:48.383 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.769816678 2024-05-15T13:06:48.383 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.384 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.384 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:48.384 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:13.019893+0000", 2024-05-15T13:06:48.384 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5654e-05 2024-05-15T13:06:48.384 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.384 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:48.384 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.384 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.384 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.384 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:146672 4.1 4:89e32c44:::benchmark_data_smithi177_99756_object26700:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e359)", 2024-05-15T13:06:48.384 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:00:21.169464+0000", 2024-05-15T13:06:48.384 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 387.19564183300002, 2024-05-15T13:06:48.384 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.3409342730000002, 2024-05-15T13:06:48.384 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:48.385 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:48.385 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:48.385 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:06:48.385 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:48.385 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 146672 2024-05-15T13:06:48.385 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.385 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:48.385 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.385 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:48.385 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:21.169464+0000", 2024-05-15T13:06:48.385 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.385 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.385 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.385 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:48.385 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:21.169464+0000", 2024-05-15T13:06:48.385 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.386 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.386 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.386 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:48.386 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:21.169465+0000", 2024-05-15T13:06:48.386 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.51e-07 2024-05-15T13:06:48.386 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.386 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.386 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:48.386 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:21.169467+0000", 2024-05-15T13:06:48.386 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.7389999999999999e-06 2024-05-15T13:06:48.386 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.386 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.387 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:48.387 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:21.169475+0000", 2024-05-15T13:06:48.387 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9969999999999998e-06 2024-05-15T13:06:48.387 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.387 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.387 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:48.387 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:21.169481+0000", 2024-05-15T13:06:48.387 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6030000000000004e-06 2024-05-15T13:06:48.387 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.387 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.387 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.387 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:21.169520+0000", 2024-05-15T13:06:48.387 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.8849e-05 2024-05-15T13:06:48.387 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.387 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.388 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:06:48.388 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:21.169608+0000", 2024-05-15T13:06:48.388 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8508e-05 2024-05-15T13:06:48.388 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.388 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.388 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.388 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.510389+0000", 2024-05-15T13:06:48.388 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.3407809099999999 2024-05-15T13:06:48.388 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.388 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.388 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:48.388 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.510399+0000", 2024-05-15T13:06:48.389 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.2159999999999995e-06 2024-05-15T13:06:48.389 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.389 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:48.389 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.389 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.389 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.389 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:148132 4.1 4:81344e8a:::benchmark_data_smithi177_99756_object28160:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e361)", 2024-05-15T13:06:48.389 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:00:23.474291+0000", 2024-05-15T13:06:48.389 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 384.89081546199998, 2024-05-15T13:06:48.389 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0802568319999999, 2024-05-15T13:06:48.389 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:48.389 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:48.390 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:48.390 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:06:48.390 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:48.390 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 148132 2024-05-15T13:06:48.390 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.390 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:48.390 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.390 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:48.390 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474291+0000", 2024-05-15T13:06:48.390 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.390 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.390 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.391 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:48.391 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474291+0000", 2024-05-15T13:06:48.391 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.391 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.391 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.391 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:48.391 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474291+0000", 2024-05-15T13:06:48.391 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.7500000000000001e-07 2024-05-15T13:06:48.391 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.391 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.391 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:48.391 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474293+0000", 2024-05-15T13:06:48.391 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7999999999999999e-06 2024-05-15T13:06:48.391 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.392 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.392 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:48.392 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474298+0000", 2024-05-15T13:06:48.392 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5300000000000004e-06 2024-05-15T13:06:48.392 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.392 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.392 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:48.392 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474311+0000", 2024-05-15T13:06:48.392 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2347999999999999e-05 2024-05-15T13:06:48.392 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.392 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.392 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.392 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.477555+0000", 2024-05-15T13:06:48.392 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0032441499999999999 2024-05-15T13:06:48.393 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.393 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.393 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:06:48.393 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.477605+0000", 2024-05-15T13:06:48.393 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0139999999999998e-05 2024-05-15T13:06:48.393 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.393 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.393 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.393 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:27.554534+0000", 2024-05-15T13:06:48.393 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0769294489999997 2024-05-15T13:06:48.393 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.393 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.393 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:48.394 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:27.554547+0000", 2024-05-15T13:06:48.394 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.314e-05 2024-05-15T13:06:48.394 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.394 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:48.394 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.394 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.394 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.394 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:148138 4.1 4:8ed26e5a:::benchmark_data_smithi177_99756_object28166:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e361)", 2024-05-15T13:06:48.394 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:00:23.474317+0000", 2024-05-15T13:06:48.394 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 384.89078950099997, 2024-05-15T13:06:48.394 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.085580319, 2024-05-15T13:06:48.394 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:48.394 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:48.395 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:48.395 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:06:48.395 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:48.395 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 148138 2024-05-15T13:06:48.395 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.395 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:48.395 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.395 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:48.395 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474317+0000", 2024-05-15T13:06:48.395 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.395 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.395 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.395 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:48.395 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474317+0000", 2024-05-15T13:06:48.396 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.396 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.396 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.396 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:48.396 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474317+0000", 2024-05-15T13:06:48.396 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.9999999999999999e-07 2024-05-15T13:06:48.396 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.396 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.396 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:48.396 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474322+0000", 2024-05-15T13:06:48.396 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.4330000000000003e-06 2024-05-15T13:06:48.396 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.396 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.396 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:48.396 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474329+0000", 2024-05-15T13:06:48.396 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4699999999999999e-06 2024-05-15T13:06:48.397 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.397 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.397 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:48.397 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474333+0000", 2024-05-15T13:06:48.397 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2409999999999997e-06 2024-05-15T13:06:48.397 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.397 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.397 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.397 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.477641+0000", 2024-05-15T13:06:48.397 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0033075270000000002 2024-05-15T13:06:48.397 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.397 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.397 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:06:48.397 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.477720+0000", 2024-05-15T13:06:48.398 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.928e-05 2024-05-15T13:06:48.398 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.398 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.398 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.398 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:27.559884+0000", 2024-05-15T13:06:48.398 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0821640099999996 2024-05-15T13:06:48.398 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.398 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.398 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:48.398 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:27.559897+0000", 2024-05-15T13:06:48.398 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3057999999999999e-05 2024-05-15T13:06:48.398 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.398 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:48.399 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.399 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.399 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.399 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:148140 4.1 4:81937744:::benchmark_data_smithi177_99756_object28168:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e361)", 2024-05-15T13:06:48.399 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:00:23.474338+0000", 2024-05-15T13:06:48.399 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 384.89076769399998, 2024-05-15T13:06:48.399 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0889538669999999, 2024-05-15T13:06:48.399 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:48.399 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:48.399 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:48.399 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:06:48.399 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:48.399 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 148140 2024-05-15T13:06:48.400 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.400 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:48.400 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.400 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:48.400 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474338+0000", 2024-05-15T13:06:48.400 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.400 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.400 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.400 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:48.400 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474338+0000", 2024-05-15T13:06:48.400 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.400 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.400 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.400 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:48.401 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474339+0000", 2024-05-15T13:06:48.401 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.2099999999999998e-07 2024-05-15T13:06:48.401 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.401 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.401 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:48.401 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474341+0000", 2024-05-15T13:06:48.401 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9240000000000001e-06 2024-05-15T13:06:48.401 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.401 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.401 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:48.401 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474345+0000", 2024-05-15T13:06:48.401 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6349999999999997e-06 2024-05-15T13:06:48.402 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.402 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.402 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:48.402 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.474349+0000", 2024-05-15T13:06:48.402 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0940000000000001e-06 2024-05-15T13:06:48.402 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.402 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.402 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.402 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.477766+0000", 2024-05-15T13:06:48.402 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00341625 2024-05-15T13:06:48.402 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.402 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.403 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:06:48.403 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:23.477861+0000", 2024-05-15T13:06:48.403 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.5136999999999994e-05 2024-05-15T13:06:48.403 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.403 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.403 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.403 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:27.563277+0000", 2024-05-15T13:06:48.403 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0854161339999999 2024-05-15T13:06:48.403 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.403 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.403 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:48.403 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:27.563292+0000", 2024-05-15T13:06:48.403 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5372000000000001e-05 2024-05-15T13:06:48.403 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.404 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:48.404 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.404 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.404 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.404 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:175091 4.1 4:8bb6ec4a:::benchmark_data_smithi177_99756_object55119:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e374)", 2024-05-15T13:06:48.404 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:06.598855+0000", 2024-05-15T13:06:48.404 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 341.766251076, 2024-05-15T13:06:48.404 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0032877979999999, 2024-05-15T13:06:48.404 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:48.404 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:48.404 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:48.404 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:06:48.404 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:48.405 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 175091 2024-05-15T13:06:48.405 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.405 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:48.405 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.405 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:48.405 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:06.598855+0000", 2024-05-15T13:06:48.405 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.405 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.405 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.405 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:48.405 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:06.598855+0000", 2024-05-15T13:06:48.405 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.405 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.405 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.406 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:48.406 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:06.598856+0000", 2024-05-15T13:06:48.406 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2309999999999999e-06 2024-05-15T13:06:48.406 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.406 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.406 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:48.406 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:06.598861+0000", 2024-05-15T13:06:48.406 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3669999999999999e-06 2024-05-15T13:06:48.406 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.406 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.406 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:48.406 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:06.598870+0000", 2024-05-15T13:06:48.406 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.3370000000000006e-06 2024-05-15T13:06:48.407 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.407 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.407 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:48.407 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:06.598878+0000", 2024-05-15T13:06:48.407 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1499999999999999e-06 2024-05-15T13:06:48.407 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.407 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.407 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.407 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:06.599921+0000", 2024-05-15T13:06:48.407 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001042935 2024-05-15T13:06:48.407 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.407 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.407 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:06:48.408 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:06.599989+0000", 2024-05-15T13:06:48.408 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7613999999999993e-05 2024-05-15T13:06:48.408 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.408 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.408 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.408 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.602128+0000", 2024-05-15T13:06:48.408 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.002138929 2024-05-15T13:06:48.408 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.408 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.408 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:48.408 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.602143+0000", 2024-05-15T13:06:48.408 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5235000000000001e-05 2024-05-15T13:06:48.408 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.408 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:48.409 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.409 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.409 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.409 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:175603 4.1 4:8bc99fd7:::benchmark_data_smithi177_99756_object55631:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e375)", 2024-05-15T13:06:48.409 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:07.585213+0000", 2024-05-15T13:06:48.409 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 340.779892969, 2024-05-15T13:06:48.409 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.6113388139999998, 2024-05-15T13:06:48.409 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:48.409 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:48.409 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:48.409 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:06:48.409 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:48.410 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 175603 2024-05-15T13:06:48.410 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.410 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:48.410 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.410 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:48.410 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.585213+0000", 2024-05-15T13:06:48.410 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.410 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.410 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.410 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:48.410 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.585213+0000", 2024-05-15T13:06:48.410 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.410 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.410 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.411 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:48.411 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.585214+0000", 2024-05-15T13:06:48.411 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4099999999999999e-07 2024-05-15T13:06:48.411 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.411 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.411 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:48.411 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.585216+0000", 2024-05-15T13:06:48.411 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.571e-06 2024-05-15T13:06:48.411 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.411 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.411 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:48.411 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.585227+0000", 2024-05-15T13:06:48.411 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0369e-05 2024-05-15T13:06:48.412 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.412 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.412 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:48.412 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.585231+0000", 2024-05-15T13:06:48.412 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.476e-06 2024-05-15T13:06:48.412 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.412 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.412 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.412 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.585442+0000", 2024-05-15T13:06:48.412 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00021099000000000001 2024-05-15T13:06:48.412 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.412 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.412 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:06:48.412 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.585496+0000", 2024-05-15T13:06:48.413 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3974999999999997e-05 2024-05-15T13:06:48.413 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.413 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.413 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.413 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:11.196538+0000", 2024-05-15T13:06:48.413 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.6110417510000001 2024-05-15T13:06:48.413 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.413 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.413 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:48.413 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:11.196552+0000", 2024-05-15T13:06:48.413 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4241e-05 2024-05-15T13:06:48.413 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.413 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:48.414 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.414 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.414 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.414 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:191878 4.1 4:84f87391:::benchmark_data_smithi177_99756_object71906:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:06:48.414 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:34.339639+0000", 2024-05-15T13:06:48.414 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 314.02546687, 2024-05-15T13:06:48.414 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.050434613, 2024-05-15T13:06:48.414 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:48.414 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:48.414 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:48.414 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:06:48.414 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:48.414 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 191878 2024-05-15T13:06:48.415 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.415 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:48.415 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.415 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:48.415 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339639+0000", 2024-05-15T13:06:48.415 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.415 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.415 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.415 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:48.415 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339639+0000", 2024-05-15T13:06:48.415 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.415 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.417 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.417 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:48.417 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339640+0000", 2024-05-15T13:06:48.417 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0599999999999999e-07 2024-05-15T13:06:48.417 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.417 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.417 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:48.417 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339644+0000", 2024-05-15T13:06:48.417 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7340000000000002e-06 2024-05-15T13:06:48.417 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.417 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.417 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:48.418 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339653+0000", 2024-05-15T13:06:48.418 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.9719999999999998e-06 2024-05-15T13:06:48.418 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.418 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.418 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:48.418 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339659+0000", 2024-05-15T13:06:48.418 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4670000000000001e-06 2024-05-15T13:06:48.418 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.418 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.418 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.418 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339742+0000", 2024-05-15T13:06:48.418 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2886000000000006e-05 2024-05-15T13:06:48.419 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.419 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.419 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:06:48.419 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339831+0000", 2024-05-15T13:06:48.419 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8708999999999994e-05 2024-05-15T13:06:48.419 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.419 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.419 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.419 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:35.390058+0000", 2024-05-15T13:06:48.419 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0502274650000001 2024-05-15T13:06:48.419 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.419 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.419 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:48.420 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:35.390074+0000", 2024-05-15T13:06:48.420 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5474000000000001e-05 2024-05-15T13:06:48.420 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.420 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:48.420 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.420 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.420 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.420 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:191882 4.1 4:8c08eda6:::benchmark_data_smithi177_99756_object71910:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:06:48.420 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:34.339674+0000", 2024-05-15T13:06:48.420 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 314.02543171399998, 2024-05-15T13:06:48.420 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.05286353, 2024-05-15T13:06:48.420 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:48.420 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:48.420 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:48.421 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:06:48.421 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:48.421 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 191882 2024-05-15T13:06:48.421 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.421 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:48.421 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.421 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:48.421 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339674+0000", 2024-05-15T13:06:48.421 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.421 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.421 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.421 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:48.421 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339674+0000", 2024-05-15T13:06:48.422 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.422 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.422 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.422 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:48.422 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339675+0000", 2024-05-15T13:06:48.422 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7199999999999999e-07 2024-05-15T13:06:48.422 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.422 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.422 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:48.422 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339678+0000", 2024-05-15T13:06:48.422 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.768e-06 2024-05-15T13:06:48.422 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.422 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.423 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:48.423 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339684+0000", 2024-05-15T13:06:48.423 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4049999999999997e-06 2024-05-15T13:06:48.423 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.423 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.423 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:48.423 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339688+0000", 2024-05-15T13:06:48.423 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5260000000000002e-06 2024-05-15T13:06:48.423 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.423 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.423 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.423 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339878+0000", 2024-05-15T13:06:48.423 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00019029800000000001 2024-05-15T13:06:48.424 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.424 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.424 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:06:48.424 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339937+0000", 2024-05-15T13:06:48.424 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9110000000000002e-05 2024-05-15T13:06:48.424 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.424 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.424 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.424 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:35.392531+0000", 2024-05-15T13:06:48.424 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0525935909999999 2024-05-15T13:06:48.424 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.424 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.424 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:48.424 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:35.392538+0000", 2024-05-15T13:06:48.425 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3599999999999998e-06 2024-05-15T13:06:48.425 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.425 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:48.425 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.425 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.425 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.425 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:191893 4.1 4:88f9917d:::benchmark_data_smithi177_99756_object71921:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:06:48.425 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:34.339696+0000", 2024-05-15T13:06:48.425 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 314.02540986399998, 2024-05-15T13:06:48.425 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.055227972, 2024-05-15T13:06:48.425 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:48.425 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:48.425 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:48.425 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:06:48.425 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:48.426 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 191893 2024-05-15T13:06:48.426 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.426 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:48.426 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.426 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:48.426 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339696+0000", 2024-05-15T13:06:48.426 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.426 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.426 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.426 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:48.426 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339696+0000", 2024-05-15T13:06:48.426 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.426 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.427 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.427 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:48.427 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339697+0000", 2024-05-15T13:06:48.427 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.6100000000000002e-07 2024-05-15T13:06:48.427 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.427 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.427 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:48.427 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339699+0000", 2024-05-15T13:06:48.427 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.5289999999999998e-06 2024-05-15T13:06:48.427 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.427 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.427 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:48.427 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339705+0000", 2024-05-15T13:06:48.427 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.0390000000000001e-06 2024-05-15T13:06:48.428 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.428 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.428 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:48.428 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339710+0000", 2024-05-15T13:06:48.428 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1930000000000001e-06 2024-05-15T13:06:48.428 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.428 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.428 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.428 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339975+0000", 2024-05-15T13:06:48.428 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000264443 2024-05-15T13:06:48.428 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.428 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.428 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:06:48.429 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.340057+0000", 2024-05-15T13:06:48.429 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1929999999999997e-05 2024-05-15T13:06:48.429 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.429 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.429 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.429 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:35.394916+0000", 2024-05-15T13:06:48.429 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0548597049999999 2024-05-15T13:06:48.429 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.429 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.429 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:48.429 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:35.394924+0000", 2024-05-15T13:06:48.429 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7719999999999994e-06 2024-05-15T13:06:48.429 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.430 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:48.430 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.430 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.430 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.430 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:191897 4.1 4:849b4824:::benchmark_data_smithi177_99756_object71925:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:06:48.430 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:34.339722+0000", 2024-05-15T13:06:48.430 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 314.02538385999998, 2024-05-15T13:06:48.430 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0569823270000001, 2024-05-15T13:06:48.430 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:48.430 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:48.430 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:48.430 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:06:48.430 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:48.431 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 191897 2024-05-15T13:06:48.431 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.431 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:48.431 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.431 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:48.431 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339722+0000", 2024-05-15T13:06:48.431 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.431 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.431 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.431 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:48.431 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339722+0000", 2024-05-15T13:06:48.431 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.432 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.432 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.432 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:48.432 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339723+0000", 2024-05-15T13:06:48.432 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.7599999999999998e-07 2024-05-15T13:06:48.432 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.432 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.432 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:48.432 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339724+0000", 2024-05-15T13:06:48.432 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4389999999999999e-06 2024-05-15T13:06:48.432 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.432 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.432 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:48.432 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339741+0000", 2024-05-15T13:06:48.432 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7385999999999999e-05 2024-05-15T13:06:48.433 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.433 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.433 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:48.433 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339745+0000", 2024-05-15T13:06:48.433 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.844e-06 2024-05-15T13:06:48.433 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.433 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.433 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.433 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.340088+0000", 2024-05-15T13:06:48.433 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00034268299999999999 2024-05-15T13:06:48.433 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.433 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.433 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:06:48.433 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.340143+0000", 2024-05-15T13:06:48.434 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5572000000000003e-05 2024-05-15T13:06:48.434 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.434 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.434 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.434 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:35.396697+0000", 2024-05-15T13:06:48.434 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0565533309999999 2024-05-15T13:06:48.434 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.434 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.434 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:48.434 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:35.396705+0000", 2024-05-15T13:06:48.434 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7959999999999996e-06 2024-05-15T13:06:48.434 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.434 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:48.435 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.435 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.435 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.435 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:191901 4.1 4:8bfc74de:::benchmark_data_smithi177_99756_object71929:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:06:48.435 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:34.339750+0000", 2024-05-15T13:06:48.435 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 314.02535589299998, 2024-05-15T13:06:48.435 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0593702620000001, 2024-05-15T13:06:48.435 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:48.435 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:48.435 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:48.435 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:06:48.435 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:48.435 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 191901 2024-05-15T13:06:48.436 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.436 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:48.436 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.436 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:48.436 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339750+0000", 2024-05-15T13:06:48.436 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.436 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.436 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.436 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:48.436 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339750+0000", 2024-05-15T13:06:48.436 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.436 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.436 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.437 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:48.437 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339750+0000", 2024-05-15T13:06:48.437 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.7700000000000001e-07 2024-05-15T13:06:48.437 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.437 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.437 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:48.437 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339755+0000", 2024-05-15T13:06:48.437 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3250000000000001e-06 2024-05-15T13:06:48.437 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.437 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.437 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:48.437 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339758+0000", 2024-05-15T13:06:48.437 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5980000000000001e-06 2024-05-15T13:06:48.437 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.438 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.438 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:48.438 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.339761+0000", 2024-05-15T13:06:48.438 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.779e-06 2024-05-15T13:06:48.438 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.438 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.438 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.438 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.340180+0000", 2024-05-15T13:06:48.438 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00041860299999999998 2024-05-15T13:06:48.438 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.438 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.438 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:06:48.438 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:34.340272+0000", 2024-05-15T13:06:48.438 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1893999999999998e-05 2024-05-15T13:06:48.439 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.439 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.439 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.439 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:35.399110+0000", 2024-05-15T13:06:48.439 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0588386279999999 2024-05-15T13:06:48.439 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.439 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.439 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:48.439 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:35.399120+0000", 2024-05-15T13:06:48.439 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0158e-05 2024-05-15T13:06:48.439 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.439 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:48.439 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.439 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.440 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.440 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:193640 4.1 4:81b56918:::benchmark_data_smithi177_99756_object73668:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:06:48.440 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:37.012243+0000", 2024-05-15T13:06:48.440 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 311.35286308100001, 2024-05-15T13:06:48.440 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4679297899999999, 2024-05-15T13:06:48.440 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:48.440 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:48.440 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:48.440 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:06:48.440 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:48.440 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 193640 2024-05-15T13:06:48.440 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.440 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:48.440 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.441 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:48.441 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:37.012243+0000", 2024-05-15T13:06:48.441 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.441 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.441 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.441 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:48.441 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:37.012243+0000", 2024-05-15T13:06:48.441 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.441 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.441 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.441 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:48.441 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:37.012244+0000", 2024-05-15T13:06:48.441 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3549999999999999e-06 2024-05-15T13:06:48.441 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.442 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.442 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:48.442 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:37.012249+0000", 2024-05-15T13:06:48.442 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9250000000000003e-06 2024-05-15T13:06:48.442 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.442 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.442 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:48.442 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:37.012259+0000", 2024-05-15T13:06:48.442 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.5300000000000002e-06 2024-05-15T13:06:48.442 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.442 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.442 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:48.442 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:37.012266+0000", 2024-05-15T13:06:48.443 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.5839999999999997e-06 2024-05-15T13:06:48.443 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.443 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.443 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.443 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:37.013803+0000", 2024-05-15T13:06:48.443 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0015370850000000001 2024-05-15T13:06:48.443 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.443 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.443 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:06:48.443 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:37.013909+0000", 2024-05-15T13:06:48.443 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000105258 2024-05-15T13:06:48.443 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.443 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.444 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.444 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:38.480163+0000", 2024-05-15T13:06:48.444 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4662545730000001 2024-05-15T13:06:48.444 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.444 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.444 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:48.444 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:38.480173+0000", 2024-05-15T13:06:48.444 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.4800000000000007e-06 2024-05-15T13:06:48.444 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.444 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:48.444 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.444 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.444 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.444 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:195199 4.1 4:85c111cf:::benchmark_data_smithi177_99756_object75227:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:06:48.445 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:39.392643+0000", 2024-05-15T13:06:48.445 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 308.97246353700001, 2024-05-15T13:06:48.445 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2585088790000001, 2024-05-15T13:06:48.445 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:48.445 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:48.445 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:48.445 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:06:48.445 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:48.445 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 195199 2024-05-15T13:06:48.445 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.445 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:48.445 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.445 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:48.445 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.392643+0000", 2024-05-15T13:06:48.446 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.446 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.446 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.446 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:48.446 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.392643+0000", 2024-05-15T13:06:48.446 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.446 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.446 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.446 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:48.446 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.392643+0000", 2024-05-15T13:06:48.446 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2500000000000001e-07 2024-05-15T13:06:48.446 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.446 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.447 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:48.447 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.392645+0000", 2024-05-15T13:06:48.447 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1299999999999999e-06 2024-05-15T13:06:48.447 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.447 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.447 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:48.447 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.392652+0000", 2024-05-15T13:06:48.447 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9120000000000001e-06 2024-05-15T13:06:48.447 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.447 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.447 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:48.447 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.392657+0000", 2024-05-15T13:06:48.447 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7559999999999999e-06 2024-05-15T13:06:48.448 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.448 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.448 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.448 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.394772+0000", 2024-05-15T13:06:48.448 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0021149189999999998 2024-05-15T13:06:48.448 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.448 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.448 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:06:48.448 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.394853+0000", 2024-05-15T13:06:48.448 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1406e-05 2024-05-15T13:06:48.448 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.448 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.448 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.448 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.651138+0000", 2024-05-15T13:06:48.449 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2562850999999999 2024-05-15T13:06:48.449 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.449 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.449 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:48.449 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.651151+0000", 2024-05-15T13:06:48.449 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3230999999999999e-05 2024-05-15T13:06:48.449 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.449 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:48.449 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.449 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.449 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.449 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:195204 4.1 4:8d0019e2:::benchmark_data_smithi177_99756_object75232:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:06:48.449 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:39.392710+0000", 2024-05-15T13:06:48.450 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 308.972396489, 2024-05-15T13:06:48.450 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2609789259999999, 2024-05-15T13:06:48.450 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:48.450 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:48.450 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:48.450 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:06:48.450 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:48.450 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 195204 2024-05-15T13:06:48.450 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.450 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:48.450 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.450 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:48.450 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.392710+0000", 2024-05-15T13:06:48.451 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.451 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.451 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.451 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:48.451 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.392710+0000", 2024-05-15T13:06:48.451 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.451 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.451 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.451 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:48.451 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.392710+0000", 2024-05-15T13:06:48.451 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.1e-07 2024-05-15T13:06:48.451 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.451 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.451 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:48.452 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.392715+0000", 2024-05-15T13:06:48.452 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8439999999999997e-06 2024-05-15T13:06:48.452 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.452 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.452 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:48.452 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.392720+0000", 2024-05-15T13:06:48.452 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6030000000000004e-06 2024-05-15T13:06:48.452 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.452 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.452 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:48.452 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.392734+0000", 2024-05-15T13:06:48.452 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3867e-05 2024-05-15T13:06:48.452 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.452 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.453 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.453 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.394882+0000", 2024-05-15T13:06:48.453 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0021474319999999999 2024-05-15T13:06:48.453 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.453 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.453 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:06:48.453 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:39.394958+0000", 2024-05-15T13:06:48.453 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.6174999999999995e-05 2024-05-15T13:06:48.453 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.453 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.453 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.453 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.653674+0000", 2024-05-15T13:06:48.453 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2587164660000001 2024-05-15T13:06:48.454 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.454 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.454 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:48.454 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.653689+0000", 2024-05-15T13:06:48.454 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4229e-05 2024-05-15T13:06:48.454 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.454 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:48.454 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.454 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.454 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.454 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:196084 4.1 4:88a7e217:::benchmark_data_smithi177_99756_object76112:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:06:48.454 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:40.633254+0000", 2024-05-15T13:06:48.454 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 307.73185238600001, 2024-05-15T13:06:48.455 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4718218709999999, 2024-05-15T13:06:48.455 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:48.455 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:48.455 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:48.455 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:06:48.455 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:48.455 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 196084 2024-05-15T13:06:48.455 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.455 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:48.455 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.455 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:48.455 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633254+0000", 2024-05-15T13:06:48.455 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.455 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.456 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.456 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:48.456 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633254+0000", 2024-05-15T13:06:48.456 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.456 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.456 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.456 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:48.456 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633254+0000", 2024-05-15T13:06:48.456 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0699999999999998e-07 2024-05-15T13:06:48.456 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.456 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.456 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:48.456 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633256+0000", 2024-05-15T13:06:48.456 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2919999999999998e-06 2024-05-15T13:06:48.457 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.457 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.457 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:48.457 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633263+0000", 2024-05-15T13:06:48.457 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.472e-06 2024-05-15T13:06:48.457 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.457 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.457 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:48.457 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633269+0000", 2024-05-15T13:06:48.457 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9460000000000003e-06 2024-05-15T13:06:48.457 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.457 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.457 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.458 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633671+0000", 2024-05-15T13:06:48.458 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00040208399999999998 2024-05-15T13:06:48.458 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.458 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.458 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:06:48.458 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633762+0000", 2024-05-15T13:06:48.458 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1438000000000001e-05 2024-05-15T13:06:48.458 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.458 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.458 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.458 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:42.105066+0000", 2024-05-15T13:06:48.458 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4713031750000001 2024-05-15T13:06:48.458 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.459 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.459 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:48.459 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:42.105076+0000", 2024-05-15T13:06:48.459 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0057e-05 2024-05-15T13:06:48.459 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.459 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:48.459 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.459 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.459 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.459 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:196086 4.1 4:88127b73:::benchmark_data_smithi177_99756_object76114:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:06:48.459 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:40.633276+0000", 2024-05-15T13:06:48.459 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 307.731829877, 2024-05-15T13:06:48.459 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.479033697, 2024-05-15T13:06:48.459 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:48.460 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:48.460 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:48.460 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:06:48.460 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:48.460 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 196086 2024-05-15T13:06:48.460 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.460 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:48.460 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.460 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:48.460 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633276+0000", 2024-05-15T13:06:48.460 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.460 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.460 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.461 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:48.461 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633276+0000", 2024-05-15T13:06:48.461 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:48.461 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.461 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.461 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:48.461 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633277+0000", 2024-05-15T13:06:48.461 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7e-07 2024-05-15T13:06:48.461 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.461 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.461 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:48.461 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633279+0000", 2024-05-15T13:06:48.461 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.03e-06 2024-05-15T13:06:48.462 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.462 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.462 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:48.462 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633284+0000", 2024-05-15T13:06:48.462 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.553e-06 2024-05-15T13:06:48.462 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.462 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.462 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:48.462 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633289+0000", 2024-05-15T13:06:48.462 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0710000000000002e-06 2024-05-15T13:06:48.462 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.462 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.462 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.462 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633792+0000", 2024-05-15T13:06:48.463 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00050264299999999999 2024-05-15T13:06:48.463 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.463 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.463 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:06:48.463 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:40.633873+0000", 2024-05-15T13:06:48.463 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0603000000000001e-05 2024-05-15T13:06:48.463 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.463 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.463 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:48.463 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:42.112301+0000", 2024-05-15T13:06:48.463 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.478428528 2024-05-15T13:06:48.463 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:48.463 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:48.463 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:48.464 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:42.112310+0000", 2024-05-15T13:06:48.464 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.799e-06 2024-05-15T13:06:48.464 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.464 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:48.464 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.464 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:48.464 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:48.464 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:06:48.465 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:48 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/221002037' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:06:48.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:48.466 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:06:48.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:06:48.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:48 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/221002037' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:06:48.567 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:06:48.605 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:48.647 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:06:48.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:48.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:06:48.751 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:06:48.798 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:49.160 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-05-15T13:06:49.168 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:06:49.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:49.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:06:49.274 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:06:49.328 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:49 smithi120 ceph-mon[43961]: pgmap v1295: 27 pgs: 1 remapped+peering, 3 peering, 2 unknown, 21 active+clean; 2.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 17 KiB/s rd, 459 KiB/s wr, 808 op/s; 643 KiB/s, 2 objects/s recovering 2024-05-15T13:06:49.328 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:49 smithi120 ceph-mon[43961]: osdmap e482: 12 total, 12 up, 11 in 2024-05-15T13:06:49.329 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:49.380 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:06:49.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:49.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:06:49.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:49 smithi177 ceph-mon[44303]: pgmap v1295: 27 pgs: 1 remapped+peering, 3 peering, 2 unknown, 21 active+clean; 2.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 17 KiB/s rd, 459 KiB/s wr, 808 op/s; 643 KiB/s, 2 objects/s recovering 2024-05-15T13:06:49.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:49 smithi177 ceph-mon[44303]: osdmap e482: 12 total, 12 up, 11 in 2024-05-15T13:06:49.489 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:06:49.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:49 smithi027 ceph-mon[39795]: pgmap v1295: 27 pgs: 1 remapped+peering, 3 peering, 2 unknown, 21 active+clean; 2.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 17 KiB/s rd, 459 KiB/s wr, 808 op/s; 643 KiB/s, 2 objects/s recovering 2024-05-15T13:06:49.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:49 smithi027 ceph-mon[39795]: osdmap e482: 12 total, 12 up, 11 in 2024-05-15T13:06:49.534 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:49.567 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:06:49.573 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:06:49.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:49.578 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:06:49.678 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:06:49.718 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:49.750 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:06:49.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:49.762 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:06:49.863 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:06:49.903 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:49.938 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:06:49.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:49.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:06:50.043 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:06:50.089 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:50.121 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:06:50.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:50.128 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:06:50.229 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:06:50.259 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:50.274 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:06:50.276 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:50 smithi027 ceph-mon[39795]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-05-15T13:06:50.283 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:06:50.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:50.289 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:06:50.390 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:06:50.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:50 smithi177 ceph-mon[44303]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-05-15T13:06:50.443 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:50.471 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:06:50.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:50.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:06:50.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:50 smithi120 ceph-mon[43961]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-05-15T13:06:50.586 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:06:50.632 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:51.092 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:06:51.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:51.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:06:51.143 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.0 to 1 (10000) 2024-05-15T13:06:51.198 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:06:51.258 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:51.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:51 smithi120 ceph-mon[43961]: pgmap v1297: 27 pgs: 3 peering, 24 active+clean; 1.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 16 KiB/s rd, 425 KiB/s wr, 982 op/s; 614 KiB/s, 2 objects/s recovering 2024-05-15T13:06:51.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:51 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/583000300' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:06:51.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:51 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:06:51.306 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:06:51.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:51.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:06:51.418 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:06:51.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:51 smithi177 ceph-mon[44303]: pgmap v1297: 27 pgs: 3 peering, 24 active+clean; 1.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 16 KiB/s rd, 425 KiB/s wr, 982 op/s; 614 KiB/s, 2 objects/s recovering 2024-05-15T13:06:51.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:51 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/583000300' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:06:51.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:51 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:06:51.461 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:51.495 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:06:51.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:51.499 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:06:51.519 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:51 smithi027 ceph-mon[39795]: pgmap v1297: 27 pgs: 3 peering, 24 active+clean; 1.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 16 KiB/s rd, 425 KiB/s wr, 982 op/s; 614 KiB/s, 2 objects/s recovering 2024-05-15T13:06:51.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:51 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/583000300' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:06:51.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:51 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:06:51.599 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:06:51.642 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:51.725 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:06:51.761 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:06:51.762 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T13:06:51.762 INFO:teuthology.orchestra.run.smithi120.stdout: "num_ops": 0 2024-05-15T13:06:51.762 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:06:51.905 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:06:51.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:51.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:06:52.010 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:06:52.075 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:52.163 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:52 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T13:06:52.163 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:52 smithi120 ceph-mon[43961]: osdmap e483: 12 total, 12 up, 11 in 2024-05-15T13:06:52.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:52 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T13:06:52.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:52 smithi177 ceph-mon[44303]: osdmap e483: 12 total, 12 up, 11 in 2024-05-15T13:06:52.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:52 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T13:06:52.452 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:52 smithi027 ceph-mon[39795]: osdmap e483: 12 total, 12 up, 11 in 2024-05-15T13:06:52.488 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 1 1 2024-05-15T13:06:52.549 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-05-15T13:06:52.742 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:06:52.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:52.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:06:52.847 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:06:52.869 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:06:52.882 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:52.915 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:06:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:52.921 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:06:53.022 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:06:53.035 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:06:53.062 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:06:53.063 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:53.092 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:06:53.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:53.099 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:06:53.200 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:06:53.237 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:53.289 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:06:53.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:53.294 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:06:53.395 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:06:53.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:53 smithi177 ceph-mon[44303]: pgmap v1299: 27 pgs: 27 active+clean; 1.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 8.3 KiB/s rd, 222 KiB/s wr, 887 op/s; 360 KiB/s, 1 objects/s recovering 2024-05-15T13:06:53.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:53 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:06:53.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:53 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:06:53.436 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:53.469 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:06:53.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:53.477 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:06:53.485 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:53 smithi120 ceph-mon[43961]: pgmap v1299: 27 pgs: 27 active+clean; 1.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 8.3 KiB/s rd, 222 KiB/s wr, 887 op/s; 360 KiB/s, 1 objects/s recovering 2024-05-15T13:06:53.485 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:53 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:06:53.485 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:53 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:06:53.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:53 smithi027 ceph-mon[39795]: pgmap v1299: 27 pgs: 27 active+clean; 1.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 8.3 KiB/s rd, 222 KiB/s wr, 887 op/s; 360 KiB/s, 1 objects/s recovering 2024-05-15T13:06:53.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:53 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:06:53.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:53 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:06:53.578 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:06:53.636 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:53.898 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:06:53.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:53.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:06:54.005 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:06:54.046 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:54.342 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:54 smithi027 ceph-mon[39795]: osdmap e484: 12 total, 12 up, 11 in 2024-05-15T13:06:54.342 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:54 smithi027 ceph-mon[39795]: osdmap e485: 12 total, 12 up, 11 in 2024-05-15T13:06:54.342 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:54 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1191584151' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:06:54.343 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:54 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:06:54.427 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:06:54.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:54 smithi177 ceph-mon[44303]: osdmap e484: 12 total, 12 up, 11 in 2024-05-15T13:06:54.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:54 smithi177 ceph-mon[44303]: osdmap e485: 12 total, 12 up, 11 in 2024-05-15T13:06:54.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:54 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1191584151' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:06:54.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:54 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:06:54.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:54 smithi120 ceph-mon[43961]: osdmap e484: 12 total, 12 up, 11 in 2024-05-15T13:06:54.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:54 smithi120 ceph-mon[43961]: osdmap e485: 12 total, 12 up, 11 in 2024-05-15T13:06:54.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:54 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1191584151' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:06:54.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:54 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:06:55.249 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:06:55.249 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T13:06:55.249 INFO:teuthology.orchestra.run.smithi120.stdout: "complaint_time": 30, 2024-05-15T13:06:55.249 INFO:teuthology.orchestra.run.smithi120.stdout: "num_blocked_ops": 0 2024-05-15T13:06:55.249 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:06:55.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:55 smithi177 ceph-mon[44303]: pgmap v1302: 27 pgs: 27 active+clean; 1.7 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 1.8 KiB/s rd, 47 KiB/s wr, 877 op/s; 58 KiB/s, 0 objects/s recovering 2024-05-15T13:06:55.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:55 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:06:55.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:55 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:06:55.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:55 smithi177 ceph-mon[44303]: osdmap e486: 12 total, 12 up, 11 in 2024-05-15T13:06:55.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:55 smithi027 ceph-mon[39795]: pgmap v1302: 27 pgs: 27 active+clean; 1.7 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 1.8 KiB/s rd, 47 KiB/s wr, 877 op/s; 58 KiB/s, 0 objects/s recovering 2024-05-15T13:06:55.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:55 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:06:55.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:55 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:06:55.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:55 smithi027 ceph-mon[39795]: osdmap e486: 12 total, 12 up, 11 in 2024-05-15T13:06:55.525 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:06:55.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:55.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:06:55.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:55 smithi120 ceph-mon[43961]: pgmap v1302: 27 pgs: 27 active+clean; 1.7 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 1.8 KiB/s rd, 47 KiB/s wr, 877 op/s; 58 KiB/s, 0 objects/s recovering 2024-05-15T13:06:55.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:55 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:06:55.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:55 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:06:55.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:55 smithi120 ceph-mon[43961]: osdmap e486: 12 total, 12 up, 11 in 2024-05-15T13:06:55.631 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:06:55.680 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:56.006 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-05-15T13:06:56.038 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:06:56.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:56.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:06:56.143 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:06:56.186 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:56.218 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:06:56.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:56.221 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:06:56.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:56 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:06:56.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:56 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:06:56.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:56 smithi120 ceph-mon[43961]: osdmap e487: 12 total, 12 up, 11 in 2024-05-15T13:06:56.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:56 smithi120 ceph-mon[43961]: 1.a scrub starts 2024-05-15T13:06:56.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:56 smithi120 ceph-mon[43961]: 1.4 scrub starts 2024-05-15T13:06:56.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:56 smithi120 ceph-mon[43961]: 1.a scrub ok 2024-05-15T13:06:56.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:56 smithi120 ceph-mon[43961]: pgmap v1305: 27 pgs: 27 active+clean; 1.6 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 865 op/s 2024-05-15T13:06:56.321 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:06:56.376 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:56.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:56.415 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:06:56.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:06:56.417 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:06:56.515 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:06:56.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:56 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:06:56.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:56 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:06:56.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:56 smithi027 ceph-mon[39795]: osdmap e487: 12 total, 12 up, 11 in 2024-05-15T13:06:56.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:56 smithi027 ceph-mon[39795]: 1.a scrub starts 2024-05-15T13:06:56.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:56 smithi027 ceph-mon[39795]: 1.4 scrub starts 2024-05-15T13:06:56.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:56 smithi027 ceph-mon[39795]: 1.a scrub ok 2024-05-15T13:06:56.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:56 smithi027 ceph-mon[39795]: pgmap v1305: 27 pgs: 27 active+clean; 1.6 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 865 op/s 2024-05-15T13:06:56.545 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:56.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:56 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:06:56.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:56 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:06:56.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:56 smithi177 ceph-mon[44303]: osdmap e487: 12 total, 12 up, 11 in 2024-05-15T13:06:56.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:56 smithi177 ceph-mon[44303]: 1.a scrub starts 2024-05-15T13:06:56.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:56 smithi177 ceph-mon[44303]: 1.4 scrub starts 2024-05-15T13:06:56.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:56 smithi177 ceph-mon[44303]: 1.a scrub ok 2024-05-15T13:06:56.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:56 smithi177 ceph-mon[44303]: pgmap v1305: 27 pgs: 27 active+clean; 1.6 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 865 op/s 2024-05-15T13:06:56.715 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:06:56.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:56.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:06:56.828 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:06:56.867 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:56.899 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:06:56.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:56.905 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:06:57.006 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:06:57.055 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:06:57.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:57.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:06:57.312 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:06:57.351 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:57.393 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:06:57.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:57.397 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:06:57.498 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:06:57.554 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:57.654 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:57 smithi027 ceph-mon[39795]: 1.4 scrub ok 2024-05-15T13:06:57.655 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:57 smithi027 ceph-mon[39795]: 1.6 scrub starts 2024-05-15T13:06:57.655 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:57 smithi027 ceph-mon[39795]: 1.6 scrub ok 2024-05-15T13:06:57.655 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:57 smithi027 ceph-mon[39795]: 1.3 scrub starts 2024-05-15T13:06:57.655 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:57 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:06:57.655 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:57 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:06:57.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:57 smithi177 ceph-mon[44303]: 1.4 scrub ok 2024-05-15T13:06:57.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:57 smithi177 ceph-mon[44303]: 1.6 scrub starts 2024-05-15T13:06:57.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:57 smithi177 ceph-mon[44303]: 1.6 scrub ok 2024-05-15T13:06:57.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:57 smithi177 ceph-mon[44303]: 1.3 scrub starts 2024-05-15T13:06:57.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:57 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:06:57.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:57 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:06:57.764 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:06:57.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:57.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:06:57.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:57 smithi120 ceph-mon[43961]: 1.4 scrub ok 2024-05-15T13:06:57.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:57 smithi120 ceph-mon[43961]: 1.6 scrub starts 2024-05-15T13:06:57.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:57 smithi120 ceph-mon[43961]: 1.6 scrub ok 2024-05-15T13:06:57.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:57 smithi120 ceph-mon[43961]: 1.3 scrub starts 2024-05-15T13:06:57.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:57 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:06:57.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:57 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:06:57.870 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:06:57.907 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:57.957 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:06:57.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:57.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:06:58.061 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:06:58.118 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:58.147 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:06:58.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:58.154 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:06:58.255 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:06:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:58.663 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:06:58.677 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:06:58.690 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:06:58.696 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:06:58.696 INFO:teuthology.orchestra.run.smithi120.stdout: "size": 20, 2024-05-15T13:06:58.696 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 600, 2024-05-15T13:06:58.697 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [ 2024-05-15T13:06:58.697 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.697 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:258309 3.d 3:b44df3e2:::benchmark_data_smithi177_88104_object124687:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-05-15T13:06:58.697 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:57:13.127941+0000", 2024-05-15T13:06:58.697 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 585.57701937299998, 2024-05-15T13:06:58.697 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.053544627, 2024-05-15T13:06:58.697 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:58.697 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:58.697 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:58.697 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T13:06:58.697 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:06:58.697 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 258309 2024-05-15T13:06:58.697 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.698 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:58.698 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.698 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:58.698 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.127941+0000", 2024-05-15T13:06:58.698 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.698 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.698 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.698 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:58.698 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.127941+0000", 2024-05-15T13:06:58.698 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.698 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.698 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.698 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:58.699 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.127942+0000", 2024-05-15T13:06:58.699 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.136e-06 2024-05-15T13:06:58.699 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.699 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.699 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:58.699 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.127946+0000", 2024-05-15T13:06:58.699 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2180000000000001e-06 2024-05-15T13:06:58.699 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.699 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.699 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:58.699 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.127955+0000", 2024-05-15T13:06:58.699 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8100000000000004e-06 2024-05-15T13:06:58.699 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.700 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.700 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:58.700 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.127961+0000", 2024-05-15T13:06:58.700 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7610000000000004e-06 2024-05-15T13:06:58.700 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.700 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.700 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.700 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.128016+0000", 2024-05-15T13:06:58.700 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.4756999999999998e-05 2024-05-15T13:06:58.700 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.700 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.700 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:06:58.700 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.128167+0000", 2024-05-15T13:06:58.700 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000151769 2024-05-15T13:06:58.700 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.701 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.701 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 6,7,9", 2024-05-15T13:06:58.701 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.128305+0000", 2024-05-15T13:06:58.701 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000138059 2024-05-15T13:06:58.701 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.701 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.701 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:58.701 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.133890+0000", 2024-05-15T13:06:58.701 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0055847939999999997 2024-05-15T13:06:58.701 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.701 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.701 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:06:58.701 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.133903+0000", 2024-05-15T13:06:58.702 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3138e-05 2024-05-15T13:06:58.702 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.702 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.702 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.702 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.145262+0000", 2024-05-15T13:06:58.702 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.011358432 2024-05-15T13:06:58.702 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.702 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.702 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:06:58.702 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.145272+0000", 2024-05-15T13:06:58.702 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0236e-05 2024-05-15T13:06:58.702 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.702 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.703 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.703 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.181477+0000", 2024-05-15T13:06:58.703 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0362050229999999 2024-05-15T13:06:58.703 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.703 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.703 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:58.703 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.181486+0000", 2024-05-15T13:06:58.703 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.4940000000000003e-06 2024-05-15T13:06:58.703 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.703 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:58.703 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.703 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.703 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.703 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:258335 3.d 3.4cad7e8d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e304)", 2024-05-15T13:06:58.704 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:57:13.156757+0000", 2024-05-15T13:06:58.704 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 585.54820344999996, 2024-05-15T13:06:58.704 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.025115043, 2024-05-15T13:06:58.704 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:58.704 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:58.704 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:58.704 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T13:06:58.704 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:06:58.704 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 258335 2024-05-15T13:06:58.704 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.704 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:58.704 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.704 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:58.704 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.156757+0000", 2024-05-15T13:06:58.704 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.705 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.705 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.705 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:58.705 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.156757+0000", 2024-05-15T13:06:58.705 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.705 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.705 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.705 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:58.705 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.156758+0000", 2024-05-15T13:06:58.705 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0750000000000001e-06 2024-05-15T13:06:58.705 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.705 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.705 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:58.705 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.156761+0000", 2024-05-15T13:06:58.706 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.23e-06 2024-05-15T13:06:58.706 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.706 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.706 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:58.706 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.156775+0000", 2024-05-15T13:06:58.706 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3855e-05 2024-05-15T13:06:58.706 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.706 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.706 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:58.706 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.156783+0000", 2024-05-15T13:06:58.706 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2009999999999999e-06 2024-05-15T13:06:58.706 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.706 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.706 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.707 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.156829+0000", 2024-05-15T13:06:58.707 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6137000000000002e-05 2024-05-15T13:06:58.707 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.707 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.707 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:06:58.707 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.156839+0000", 2024-05-15T13:06:58.707 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1130000000000007e-06 2024-05-15T13:06:58.707 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.707 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.707 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.707 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.181861+0000", 2024-05-15T13:06:58.707 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0250228960000001 2024-05-15T13:06:58.707 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.707 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.708 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:58.708 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.181872+0000", 2024-05-15T13:06:58.708 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0536e-05 2024-05-15T13:06:58.708 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.708 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:58.708 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.708 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.708 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.708 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:258360 3.d 3.e406dcd (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e304)", 2024-05-15T13:06:58.708 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:57:13.186921+0000", 2024-05-15T13:06:58.708 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 585.51803935800001, 2024-05-15T13:06:58.708 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.99500018899999998, 2024-05-15T13:06:58.708 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:58.708 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:58.709 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:58.709 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T13:06:58.709 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:06:58.709 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 258360 2024-05-15T13:06:58.709 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.709 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:58.709 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.709 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:58.709 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.186921+0000", 2024-05-15T13:06:58.709 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.709 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.709 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.709 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:58.710 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.186921+0000", 2024-05-15T13:06:58.710 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.710 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.710 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.710 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:58.710 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.186922+0000", 2024-05-15T13:06:58.710 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.494e-06 2024-05-15T13:06:58.710 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.710 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.710 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:58.710 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.186929+0000", 2024-05-15T13:06:58.710 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1689999999999997e-06 2024-05-15T13:06:58.710 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.710 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.710 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:58.711 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.186941+0000", 2024-05-15T13:06:58.711 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2037e-05 2024-05-15T13:06:58.711 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.711 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.711 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:58.711 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.186949+0000", 2024-05-15T13:06:58.711 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.6940000000000001e-06 2024-05-15T13:06:58.711 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.711 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.711 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.711 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.187020+0000", 2024-05-15T13:06:58.711 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0717999999999996e-05 2024-05-15T13:06:58.711 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.712 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.712 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:06:58.712 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.187030+0000", 2024-05-15T13:06:58.712 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0277e-05 2024-05-15T13:06:58.712 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.712 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.712 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.712 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.181910+0000", 2024-05-15T13:06:58.712 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.99487993200000002 2024-05-15T13:06:58.712 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.712 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.713 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:58.713 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.181921+0000", 2024-05-15T13:06:58.713 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0868e-05 2024-05-15T13:06:58.713 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.713 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:58.713 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.713 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.713 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.713 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:258383 3.d 3.a6f112dd (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e304)", 2024-05-15T13:06:58.713 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:57:13.208763+0000", 2024-05-15T13:06:58.713 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 585.496197424, 2024-05-15T13:06:58.713 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.973186141, 2024-05-15T13:06:58.713 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:58.713 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:58.714 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:58.714 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T13:06:58.714 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:06:58.714 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 258383 2024-05-15T13:06:58.714 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.714 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:58.714 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.714 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:58.714 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.208763+0000", 2024-05-15T13:06:58.714 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.714 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.714 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.714 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:58.715 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.208763+0000", 2024-05-15T13:06:58.715 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.715 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.715 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.715 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:58.715 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.208765+0000", 2024-05-15T13:06:58.715 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.592e-06 2024-05-15T13:06:58.715 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.715 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.715 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:58.715 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.208770+0000", 2024-05-15T13:06:58.715 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8889999999999999e-06 2024-05-15T13:06:58.715 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.716 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.716 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:58.716 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.208785+0000", 2024-05-15T13:06:58.716 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4450999999999999e-05 2024-05-15T13:06:58.716 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.716 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.716 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:58.716 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.208793+0000", 2024-05-15T13:06:58.716 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0059999999999992e-06 2024-05-15T13:06:58.716 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.716 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.716 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.716 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.208848+0000", 2024-05-15T13:06:58.716 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5247999999999997e-05 2024-05-15T13:06:58.717 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.717 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.717 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:06:58.717 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.208857+0000", 2024-05-15T13:06:58.717 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0759999999999992e-06 2024-05-15T13:06:58.717 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.717 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.717 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.717 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.181940+0000", 2024-05-15T13:06:58.717 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.97308276000000005 2024-05-15T13:06:58.717 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.717 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.717 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:58.718 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.181949+0000", 2024-05-15T13:06:58.718 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1190000000000003e-06 2024-05-15T13:06:58.718 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.718 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:58.718 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.718 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.718 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.718 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:258386 3.d 3.aca22b8d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e304)", 2024-05-15T13:06:58.718 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:57:13.214819+0000", 2024-05-15T13:06:58.718 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 585.49014128700003, 2024-05-15T13:06:58.718 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.96716916799999997, 2024-05-15T13:06:58.718 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:58.718 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:58.719 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:58.719 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T13:06:58.719 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:06:58.719 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 258386 2024-05-15T13:06:58.719 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.719 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:58.719 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.719 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:58.719 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.214819+0000", 2024-05-15T13:06:58.719 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.719 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.719 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.719 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:58.719 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.214819+0000", 2024-05-15T13:06:58.720 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.720 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.720 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.720 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:58.720 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.214820+0000", 2024-05-15T13:06:58.720 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.5600000000000004e-07 2024-05-15T13:06:58.720 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.720 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.720 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:58.720 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.214825+0000", 2024-05-15T13:06:58.720 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3020000000000002e-06 2024-05-15T13:06:58.720 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.720 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.720 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:58.721 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.214836+0000", 2024-05-15T13:06:58.721 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1041e-05 2024-05-15T13:06:58.721 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.721 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.721 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:58.721 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.214842+0000", 2024-05-15T13:06:58.721 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6489999999999996e-06 2024-05-15T13:06:58.721 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.721 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.721 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.721 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.214877+0000", 2024-05-15T13:06:58.721 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5039000000000001e-05 2024-05-15T13:06:58.721 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.721 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.721 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:06:58.722 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.214883+0000", 2024-05-15T13:06:58.722 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7899999999999996e-06 2024-05-15T13:06:58.722 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.722 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.722 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.722 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.181980+0000", 2024-05-15T13:06:58.722 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.96709679999999998 2024-05-15T13:06:58.722 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.722 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.722 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:58.722 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.181988+0000", 2024-05-15T13:06:58.722 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.6910000000000003e-06 2024-05-15T13:06:58.722 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.722 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:58.723 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.723 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.723 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.723 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:258389 3.d 3.f5e18f4d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e304)", 2024-05-15T13:06:58.723 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:57:13.217011+0000", 2024-05-15T13:06:58.723 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 585.48794909200001, 2024-05-15T13:06:58.723 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.96500432400000002, 2024-05-15T13:06:58.723 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:58.723 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:58.723 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:58.723 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T13:06:58.723 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:06:58.723 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 258389 2024-05-15T13:06:58.723 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.724 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:58.724 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.724 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:58.724 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.217011+0000", 2024-05-15T13:06:58.724 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.724 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.724 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.724 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:58.724 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.217011+0000", 2024-05-15T13:06:58.724 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.724 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.724 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.724 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:58.724 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.217013+0000", 2024-05-15T13:06:58.725 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.747e-06 2024-05-15T13:06:58.725 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.725 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.725 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:58.725 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.217020+0000", 2024-05-15T13:06:58.725 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9299999999999997e-06 2024-05-15T13:06:58.725 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.725 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.725 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:58.725 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.217042+0000", 2024-05-15T13:06:58.725 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2172e-05 2024-05-15T13:06:58.725 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.725 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.725 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:58.726 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.217053+0000", 2024-05-15T13:06:58.726 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1003e-05 2024-05-15T13:06:58.726 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.726 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.726 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.726 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.217112+0000", 2024-05-15T13:06:58.726 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8792000000000002e-05 2024-05-15T13:06:58.726 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.726 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.726 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:06:58.726 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.217120+0000", 2024-05-15T13:06:58.726 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7300000000000005e-06 2024-05-15T13:06:58.726 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.726 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.727 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.727 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.182011+0000", 2024-05-15T13:06:58.727 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.96489112099999996 2024-05-15T13:06:58.727 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.727 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.727 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:58.727 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.182016+0000", 2024-05-15T13:06:58.727 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8289999999999998e-06 2024-05-15T13:06:58.727 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.727 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:58.727 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.727 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.727 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.727 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:258397 3.d 3.4ef9e18d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e304)", 2024-05-15T13:06:58.728 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:57:13.225342+0000", 2024-05-15T13:06:58.728 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 585.479617856, 2024-05-15T13:06:58.728 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.95670922000000003, 2024-05-15T13:06:58.728 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:58.728 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:58.728 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:58.728 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T13:06:58.728 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:06:58.728 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 258397 2024-05-15T13:06:58.728 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.728 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:58.728 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.728 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:58.728 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.225342+0000", 2024-05-15T13:06:58.729 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.729 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.729 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.729 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:58.729 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.225342+0000", 2024-05-15T13:06:58.729 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.729 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.729 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.729 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:58.729 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.225344+0000", 2024-05-15T13:06:58.729 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6330000000000001e-06 2024-05-15T13:06:58.729 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.729 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.730 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:58.730 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.225352+0000", 2024-05-15T13:06:58.730 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8970000000000008e-06 2024-05-15T13:06:58.730 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.730 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.730 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:58.730 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.225369+0000", 2024-05-15T13:06:58.730 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7119999999999999e-05 2024-05-15T13:06:58.730 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.730 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.730 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:58.730 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.225379+0000", 2024-05-15T13:06:58.730 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.5840000000000001e-06 2024-05-15T13:06:58.730 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.731 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.731 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.731 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.225444+0000", 2024-05-15T13:06:58.731 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.5560000000000002e-05 2024-05-15T13:06:58.731 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.731 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.731 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:06:58.731 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.225456+0000", 2024-05-15T13:06:58.731 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1494e-05 2024-05-15T13:06:58.731 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.731 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.731 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.731 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.182042+0000", 2024-05-15T13:06:58.731 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.95658666400000003 2024-05-15T13:06:58.732 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.732 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.732 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:58.732 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.182052+0000", 2024-05-15T13:06:58.732 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.268e-06 2024-05-15T13:06:58.732 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.732 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:58.732 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.732 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.732 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.732 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:258419 3.d 3.e22d3a4d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e304)", 2024-05-15T13:06:58.732 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:57:13.251710+0000", 2024-05-15T13:06:58.732 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 585.45325009400005, 2024-05-15T13:06:58.732 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.93040936299999999, 2024-05-15T13:06:58.733 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:58.733 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:58.733 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:58.733 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T13:06:58.733 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:06:58.733 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 258419 2024-05-15T13:06:58.733 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.733 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:58.733 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.733 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:58.733 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.251710+0000", 2024-05-15T13:06:58.733 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.733 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.733 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.734 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:58.734 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.251710+0000", 2024-05-15T13:06:58.734 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.734 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.734 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.734 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:58.734 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.251711+0000", 2024-05-15T13:06:58.734 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.018e-06 2024-05-15T13:06:58.734 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.734 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.734 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:58.734 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.251716+0000", 2024-05-15T13:06:58.734 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4950000000000002e-06 2024-05-15T13:06:58.735 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.735 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.735 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:58.735 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.251732+0000", 2024-05-15T13:06:58.735 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6208000000000001e-05 2024-05-15T13:06:58.735 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.735 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.735 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:58.735 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.251741+0000", 2024-05-15T13:06:58.735 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.9570000000000008e-06 2024-05-15T13:06:58.735 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.735 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.735 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.735 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.251790+0000", 2024-05-15T13:06:58.736 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8708999999999997e-05 2024-05-15T13:06:58.736 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.736 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.736 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:06:58.736 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.251801+0000", 2024-05-15T13:06:58.736 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1671999999999999e-05 2024-05-15T13:06:58.736 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.736 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.736 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.736 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.182064+0000", 2024-05-15T13:06:58.736 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.93026244000000002 2024-05-15T13:06:58.736 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.736 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.736 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:58.737 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.182120+0000", 2024-05-15T13:06:58.737 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5863999999999999e-05 2024-05-15T13:06:58.737 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.737 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:58.737 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.737 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.737 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.737 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:258386 3.d 3:b1d44535:::benchmark_data_smithi177_88104_object124764:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e304)", 2024-05-15T13:06:58.737 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:57:13.292590+0000", 2024-05-15T13:06:58.737 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 585.41236986800004, 2024-05-15T13:06:58.737 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.91481164999999998, 2024-05-15T13:06:58.737 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:58.737 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:58.737 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:58.738 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T13:06:58.738 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:06:58.738 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 258386 2024-05-15T13:06:58.738 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.738 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:58.738 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.738 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:58.738 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292590+0000", 2024-05-15T13:06:58.738 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.738 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.738 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.738 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:58.738 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292590+0000", 2024-05-15T13:06:58.739 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.739 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.739 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.739 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:58.739 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292591+0000", 2024-05-15T13:06:58.739 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.9900000000000001e-07 2024-05-15T13:06:58.739 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.739 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.739 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:58.739 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292597+0000", 2024-05-15T13:06:58.739 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.2140000000000001e-06 2024-05-15T13:06:58.739 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.739 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.739 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:58.740 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292604+0000", 2024-05-15T13:06:58.740 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.939e-06 2024-05-15T13:06:58.740 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.740 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.740 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:58.740 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292609+0000", 2024-05-15T13:06:58.740 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2449999999999998e-06 2024-05-15T13:06:58.740 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.740 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.740 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.740 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292635+0000", 2024-05-15T13:06:58.740 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.5689000000000001e-05 2024-05-15T13:06:58.740 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.740 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.741 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:06:58.741 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292640+0000", 2024-05-15T13:06:58.741 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2809999999999999e-06 2024-05-15T13:06:58.741 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.741 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.741 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.741 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.196784+0000", 2024-05-15T13:06:58.741 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.90414376200000002 2024-05-15T13:06:58.741 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.741 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.741 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:06:58.741 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.196863+0000", 2024-05-15T13:06:58.741 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8934e-05 2024-05-15T13:06:58.742 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.742 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.742 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 6,9", 2024-05-15T13:06:58.742 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.196935+0000", 2024-05-15T13:06:58.742 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.1735000000000004e-05 2024-05-15T13:06:58.742 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.742 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.742 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:58.742 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.207200+0000", 2024-05-15T13:06:58.742 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.010265724 2024-05-15T13:06:58.742 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.742 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.742 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:58.742 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.207252+0000", 2024-05-15T13:06:58.743 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1504e-05 2024-05-15T13:06:58.743 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.743 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.743 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:06:58.743 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.207320+0000", 2024-05-15T13:06:58.743 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7912999999999998e-05 2024-05-15T13:06:58.743 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.743 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.743 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:06:58.743 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.207371+0000", 2024-05-15T13:06:58.743 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1001999999999997e-05 2024-05-15T13:06:58.743 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.743 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.744 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:58.744 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.207402+0000", 2024-05-15T13:06:58.744 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.1309000000000001e-05 2024-05-15T13:06:58.744 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.744 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:58.744 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.744 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.744 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.744 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:258389 3.d 3:b2f187af:::benchmark_data_smithi177_88104_object124767:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e304)", 2024-05-15T13:06:58.744 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:57:13.292617+0000", 2024-05-15T13:06:58.744 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 585.41234352000004, 2024-05-15T13:06:58.744 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.92016158599999998, 2024-05-15T13:06:58.744 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:58.745 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:58.745 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:58.745 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T13:06:58.745 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:06:58.745 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 258389 2024-05-15T13:06:58.745 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.745 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:58.745 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.745 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:58.745 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292617+0000", 2024-05-15T13:06:58.745 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.745 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.745 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.745 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:58.746 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292617+0000", 2024-05-15T13:06:58.746 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.746 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.746 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.746 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:58.746 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292617+0000", 2024-05-15T13:06:58.746 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.34e-07 2024-05-15T13:06:58.746 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.746 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.746 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:58.746 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292619+0000", 2024-05-15T13:06:58.746 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2759999999999999e-06 2024-05-15T13:06:58.746 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.746 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.747 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:58.747 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292625+0000", 2024-05-15T13:06:58.747 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.4160000000000003e-06 2024-05-15T13:06:58.747 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.747 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.747 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:58.747 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292630+0000", 2024-05-15T13:06:58.747 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9910000000000002e-06 2024-05-15T13:06:58.747 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.747 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.747 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.747 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292665+0000", 2024-05-15T13:06:58.747 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5077000000000003e-05 2024-05-15T13:06:58.747 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.748 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.748 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:06:58.748 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292672+0000", 2024-05-15T13:06:58.748 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2250000000000003e-06 2024-05-15T13:06:58.748 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.748 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.748 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.748 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.201988+0000", 2024-05-15T13:06:58.748 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.90931629199999997 2024-05-15T13:06:58.748 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.748 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.748 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:06:58.748 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.202121+0000", 2024-05-15T13:06:58.748 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013233099999999999 2024-05-15T13:06:58.748 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.749 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.749 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 6,9", 2024-05-15T13:06:58.749 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.202237+0000", 2024-05-15T13:06:58.749 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000115944 2024-05-15T13:06:58.749 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.749 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.749 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:58.749 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.207307+0000", 2024-05-15T13:06:58.749 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0050699109999999999 2024-05-15T13:06:58.749 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.749 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.749 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:58.749 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.207456+0000", 2024-05-15T13:06:58.749 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000149875 2024-05-15T13:06:58.749 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.749 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.750 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:06:58.750 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.212670+0000", 2024-05-15T13:06:58.750 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0052136839999999997 2024-05-15T13:06:58.750 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.750 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.750 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:06:58.750 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.212724+0000", 2024-05-15T13:06:58.750 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3831e-05 2024-05-15T13:06:58.750 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.750 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.750 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:58.750 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.212778+0000", 2024-05-15T13:06:58.750 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.4398999999999999e-05 2024-05-15T13:06:58.750 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.751 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:58.751 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.751 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.751 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.751 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:258397 3.d 3:b1879f72:::benchmark_data_smithi177_88104_object124775:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e304)", 2024-05-15T13:06:58.751 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:57:13.292637+0000", 2024-05-15T13:06:58.751 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 585.41232359399999, 2024-05-15T13:06:58.751 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.93426573800000001, 2024-05-15T13:06:58.751 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:58.751 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:58.751 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:58.751 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T13:06:58.751 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:06:58.751 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 258397 2024-05-15T13:06:58.751 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.752 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:58.752 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.752 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:58.752 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292637+0000", 2024-05-15T13:06:58.752 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.752 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.752 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.752 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:58.752 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292637+0000", 2024-05-15T13:06:58.752 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.752 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.752 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.752 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:58.752 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292637+0000", 2024-05-15T13:06:58.752 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.8299999999999998e-07 2024-05-15T13:06:58.753 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.753 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.753 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:58.753 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292639+0000", 2024-05-15T13:06:58.753 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7579999999999999e-06 2024-05-15T13:06:58.753 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.753 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.753 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:58.753 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292644+0000", 2024-05-15T13:06:58.753 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3329999999999996e-06 2024-05-15T13:06:58.753 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.753 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.753 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:58.753 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292649+0000", 2024-05-15T13:06:58.753 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.5750000000000002e-06 2024-05-15T13:06:58.754 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.754 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.754 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.754 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292704+0000", 2024-05-15T13:06:58.754 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5163e-05 2024-05-15T13:06:58.754 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.754 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.754 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:06:58.754 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292710+0000", 2024-05-15T13:06:58.754 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8309999999999997e-06 2024-05-15T13:06:58.754 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.754 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.754 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.754 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.207489+0000", 2024-05-15T13:06:58.755 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.91477916500000001 2024-05-15T13:06:58.755 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.755 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.755 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:06:58.755 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.207611+0000", 2024-05-15T13:06:58.755 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012237500000000001 2024-05-15T13:06:58.755 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.755 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.755 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 6,9", 2024-05-15T13:06:58.755 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.207745+0000", 2024-05-15T13:06:58.755 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013344999999999999 2024-05-15T13:06:58.755 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.755 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.755 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:58.756 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.217147+0000", 2024-05-15T13:06:58.756 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0094022719999999997 2024-05-15T13:06:58.756 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.756 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.756 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:58.756 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.217201+0000", 2024-05-15T13:06:58.756 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.4336999999999998e-05 2024-05-15T13:06:58.756 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.756 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.756 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:06:58.756 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.226815+0000", 2024-05-15T13:06:58.756 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0096133869999999993 2024-05-15T13:06:58.756 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.756 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.757 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:06:58.757 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.226861+0000", 2024-05-15T13:06:58.757 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6217999999999997e-05 2024-05-15T13:06:58.757 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.757 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.757 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:58.757 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.226902+0000", 2024-05-15T13:06:58.757 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.1591000000000002e-05 2024-05-15T13:06:58.757 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.757 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:58.757 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.757 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.757 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.757 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:258419 3.d 3:b25cb447:::benchmark_data_smithi177_88104_object124797:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e304)", 2024-05-15T13:06:58.758 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:57:13.292654+0000", 2024-05-15T13:06:58.758 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 585.412306403, 2024-05-15T13:06:58.758 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.93433320200000003, 2024-05-15T13:06:58.758 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:58.758 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:58.758 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:58.758 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T13:06:58.758 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:06:58.758 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 258419 2024-05-15T13:06:58.758 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.758 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:58.758 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.758 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:58.759 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292654+0000", 2024-05-15T13:06:58.759 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.759 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.759 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.759 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:58.759 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292654+0000", 2024-05-15T13:06:58.759 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.759 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.759 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.759 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:58.759 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292654+0000", 2024-05-15T13:06:58.759 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.6800000000000002e-07 2024-05-15T13:06:58.759 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.759 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.760 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:58.760 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292656+0000", 2024-05-15T13:06:58.760 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.753e-06 2024-05-15T13:06:58.760 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.760 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.760 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:58.760 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292661+0000", 2024-05-15T13:06:58.760 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2499999999999997e-06 2024-05-15T13:06:58.760 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.760 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.760 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:58.760 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292666+0000", 2024-05-15T13:06:58.760 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.5580000000000002e-06 2024-05-15T13:06:58.760 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.761 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.761 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.761 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292743+0000", 2024-05-15T13:06:58.761 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.6967999999999999e-05 2024-05-15T13:06:58.761 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.761 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.761 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:06:58.761 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292748+0000", 2024-05-15T13:06:58.761 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5990000000000001e-06 2024-05-15T13:06:58.761 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.761 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.761 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.761 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.212840+0000", 2024-05-15T13:06:58.761 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.92009147099999999 2024-05-15T13:06:58.761 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.762 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.762 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:06:58.762 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.212933+0000", 2024-05-15T13:06:58.762 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.3525999999999999e-05 2024-05-15T13:06:58.762 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.762 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.762 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 6,9", 2024-05-15T13:06:58.762 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.213007+0000", 2024-05-15T13:06:58.762 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3671000000000007e-05 2024-05-15T13:06:58.762 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.762 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.762 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:58.762 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.226687+0000", 2024-05-15T13:06:58.762 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.013680318 2024-05-15T13:06:58.763 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.763 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.763 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:58.763 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.226802+0000", 2024-05-15T13:06:58.763 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00011473599999999999 2024-05-15T13:06:58.763 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.763 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.763 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:06:58.763 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.226910+0000", 2024-05-15T13:06:58.763 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000107897 2024-05-15T13:06:58.763 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.763 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.763 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:06:58.763 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.226952+0000", 2024-05-15T13:06:58.763 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.1798000000000002e-05 2024-05-15T13:06:58.764 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.764 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.764 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:58.764 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.226987+0000", 2024-05-15T13:06:58.764 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5389000000000003e-05 2024-05-15T13:06:58.764 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.764 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:58.764 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.764 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.764 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.764 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:258444 3.d 3:be0314d8:::benchmark_data_smithi177_88104_object124822:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e304)", 2024-05-15T13:06:58.764 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:57:13.292712+0000", 2024-05-15T13:06:58.764 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 585.41224790399997, 2024-05-15T13:06:58.764 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.93443452100000002, 2024-05-15T13:06:58.764 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:58.765 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:58.765 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:58.765 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T13:06:58.765 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:06:58.765 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 258444 2024-05-15T13:06:58.765 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.765 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:58.765 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.765 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:58.765 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292712+0000", 2024-05-15T13:06:58.765 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.765 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.765 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.765 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:58.765 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292712+0000", 2024-05-15T13:06:58.766 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.766 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.766 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.766 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:58.766 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292713+0000", 2024-05-15T13:06:58.766 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.91e-07 2024-05-15T13:06:58.766 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.766 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.766 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:58.766 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292715+0000", 2024-05-15T13:06:58.766 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.8819999999999999e-06 2024-05-15T13:06:58.766 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.767 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.767 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:58.767 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292720+0000", 2024-05-15T13:06:58.767 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.4160000000000003e-06 2024-05-15T13:06:58.767 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.767 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.767 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:58.767 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292734+0000", 2024-05-15T13:06:58.767 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3648e-05 2024-05-15T13:06:58.767 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.767 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.767 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.767 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292770+0000", 2024-05-15T13:06:58.767 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.6810999999999997e-05 2024-05-15T13:06:58.768 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.768 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.768 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:06:58.768 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.292776+0000", 2024-05-15T13:06:58.768 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9819999999999996e-06 2024-05-15T13:06:58.768 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.768 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.768 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.768 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.217236+0000", 2024-05-15T13:06:58.768 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.92445995199999997 2024-05-15T13:06:58.768 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.768 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.768 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:06:58.769 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.217361+0000", 2024-05-15T13:06:58.769 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000124249 2024-05-15T13:06:58.769 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.769 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.769 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 6,9", 2024-05-15T13:06:58.769 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.217507+0000", 2024-05-15T13:06:58.769 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000146327 2024-05-15T13:06:58.769 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.769 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.769 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:58.769 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.226752+0000", 2024-05-15T13:06:58.769 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0092453919999999998 2024-05-15T13:06:58.769 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.769 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.770 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:06:58.770 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.226994+0000", 2024-05-15T13:06:58.770 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000241742 2024-05-15T13:06:58.770 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.770 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.770 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:58.770 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.227048+0000", 2024-05-15T13:06:58.770 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3699999999999997e-05 2024-05-15T13:06:58.770 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.770 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.770 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:06:58.770 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.227103+0000", 2024-05-15T13:06:58.770 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5461000000000003e-05 2024-05-15T13:06:58.770 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.771 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.771 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:58.771 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.227147+0000", 2024-05-15T13:06:58.771 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3668000000000003e-05 2024-05-15T13:06:58.771 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.771 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:58.771 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.771 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.771 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.771 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:258457 3.d 3:b04bc6c2:::benchmark_data_smithi177_88104_object124835:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e304)", 2024-05-15T13:06:58.771 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:57:13.306144+0000", 2024-05-15T13:06:58.771 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 585.39881683900001, 2024-05-15T13:06:58.771 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.93120055099999999, 2024-05-15T13:06:58.772 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:58.772 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:58.772 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:58.772 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T13:06:58.772 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:06:58.772 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 258457 2024-05-15T13:06:58.772 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.772 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:58.772 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.772 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:58.772 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.306144+0000", 2024-05-15T13:06:58.772 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.772 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.772 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.773 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:58.773 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.306144+0000", 2024-05-15T13:06:58.773 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.773 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.773 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.773 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:58.773 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.306144+0000", 2024-05-15T13:06:58.773 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.8599999999999996e-07 2024-05-15T13:06:58.773 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.773 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.773 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:58.773 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.306149+0000", 2024-05-15T13:06:58.773 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.843e-06 2024-05-15T13:06:58.773 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.774 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.774 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:58.774 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.306159+0000", 2024-05-15T13:06:58.774 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.4919999999999999e-06 2024-05-15T13:06:58.774 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.774 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.774 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:58.774 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.306164+0000", 2024-05-15T13:06:58.774 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2639999999999999e-06 2024-05-15T13:06:58.774 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.774 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.774 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.775 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.306206+0000", 2024-05-15T13:06:58.775 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2351000000000002e-05 2024-05-15T13:06:58.775 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.775 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.775 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:06:58.775 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.306213+0000", 2024-05-15T13:06:58.775 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8800000000000002e-06 2024-05-15T13:06:58.775 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.775 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.775 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.775 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.222522+0000", 2024-05-15T13:06:58.775 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.91630846799999999 2024-05-15T13:06:58.775 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.775 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.776 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:06:58.776 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.222621+0000", 2024-05-15T13:06:58.776 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.9597000000000002e-05 2024-05-15T13:06:58.776 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.776 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.776 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 6,9", 2024-05-15T13:06:58.776 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.222733+0000", 2024-05-15T13:06:58.776 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000111383 2024-05-15T13:06:58.776 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.776 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.776 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:06:58.776 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.237176+0000", 2024-05-15T13:06:58.776 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.014442873 2024-05-15T13:06:58.776 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.776 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.777 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:58.777 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.237214+0000", 2024-05-15T13:06:58.777 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.8726000000000002e-05 2024-05-15T13:06:58.777 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.777 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.777 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:58.777 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.237256+0000", 2024-05-15T13:06:58.777 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.1355e-05 2024-05-15T13:06:58.777 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.777 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.777 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:06:58.777 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.237304+0000", 2024-05-15T13:06:58.777 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7799000000000001e-05 2024-05-15T13:06:58.777 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.778 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.778 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:58.778 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.237344+0000", 2024-05-15T13:06:58.778 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0534000000000002e-05 2024-05-15T13:06:58.778 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.778 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:58.778 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.778 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.778 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.778 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:258479 3.d 3:b7df521a:::benchmark_data_smithi177_88104_object124857:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e304)", 2024-05-15T13:06:58.778 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:57:13.335198+0000", 2024-05-15T13:06:58.778 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 585.36976256599996, 2024-05-15T13:06:58.778 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.91377813900000004, 2024-05-15T13:06:58.779 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:58.779 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:58.779 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:58.779 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T13:06:58.779 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:06:58.779 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 258479 2024-05-15T13:06:58.779 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.779 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:58.779 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.779 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:58.779 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.335198+0000", 2024-05-15T13:06:58.779 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.779 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.779 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.779 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:58.780 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.335198+0000", 2024-05-15T13:06:58.780 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.780 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.780 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.780 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:58.780 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.335199+0000", 2024-05-15T13:06:58.780 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3209999999999999e-06 2024-05-15T13:06:58.780 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.780 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.780 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:58.780 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.335204+0000", 2024-05-15T13:06:58.780 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7720000000000002e-06 2024-05-15T13:06:58.780 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.781 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.781 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:58.781 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.335216+0000", 2024-05-15T13:06:58.781 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2615e-05 2024-05-15T13:06:58.781 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.781 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.781 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:58.781 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.335223+0000", 2024-05-15T13:06:58.781 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.545e-06 2024-05-15T13:06:58.781 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.781 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.781 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.781 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.335274+0000", 2024-05-15T13:06:58.781 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0735000000000001e-05 2024-05-15T13:06:58.782 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.782 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.782 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:06:58.782 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.335286+0000", 2024-05-15T13:06:58.782 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1772e-05 2024-05-15T13:06:58.782 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.782 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.782 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.782 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.227190+0000", 2024-05-15T13:06:58.782 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.89190473299999995 2024-05-15T13:06:58.782 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.782 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.782 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:06:58.783 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.227285+0000", 2024-05-15T13:06:58.783 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.4418000000000003e-05 2024-05-15T13:06:58.783 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.783 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.783 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 6,9", 2024-05-15T13:06:58.783 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.227398+0000", 2024-05-15T13:06:58.783 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000113169 2024-05-15T13:06:58.783 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.783 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.783 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:06:58.783 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.237225+0000", 2024-05-15T13:06:58.783 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0098273500000000003 2024-05-15T13:06:58.783 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.783 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.783 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:58.784 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.237379+0000", 2024-05-15T13:06:58.784 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015382300000000001 2024-05-15T13:06:58.784 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.784 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.784 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:58.784 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.248877+0000", 2024-05-15T13:06:58.784 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.011497848 2024-05-15T13:06:58.784 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.784 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.784 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:06:58.784 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.248931+0000", 2024-05-15T13:06:58.784 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3748000000000001e-05 2024-05-15T13:06:58.784 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.784 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.785 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:58.785 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.248976+0000", 2024-05-15T13:06:58.785 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.5290000000000002e-05 2024-05-15T13:06:58.785 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.785 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:58.785 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.785 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.785 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.785 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:258481 3.d 3:b7e79888:::benchmark_data_smithi177_88104_object124859:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e304)", 2024-05-15T13:06:58.785 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:57:13.340670+0000", 2024-05-15T13:06:58.785 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 585.36429036499999, 2024-05-15T13:06:58.785 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.91865587400000004, 2024-05-15T13:06:58.785 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:58.785 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:58.786 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:58.786 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T13:06:58.786 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:06:58.786 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 258481 2024-05-15T13:06:58.786 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.786 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:58.786 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.786 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:58.786 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.340670+0000", 2024-05-15T13:06:58.786 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.786 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.786 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.786 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:58.787 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.340670+0000", 2024-05-15T13:06:58.787 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.787 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.787 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.787 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:58.787 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.340671+0000", 2024-05-15T13:06:58.787 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2279999999999999e-06 2024-05-15T13:06:58.787 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.787 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.787 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:58.787 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.340678+0000", 2024-05-15T13:06:58.787 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3679999999999998e-06 2024-05-15T13:06:58.787 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.787 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.787 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:58.788 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.340692+0000", 2024-05-15T13:06:58.788 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.433e-05 2024-05-15T13:06:58.788 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.788 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.788 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:58.788 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.340701+0000", 2024-05-15T13:06:58.788 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.6340000000000007e-06 2024-05-15T13:06:58.788 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.788 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.788 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.788 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.340761+0000", 2024-05-15T13:06:58.788 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.0643999999999997e-05 2024-05-15T13:06:58.788 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.788 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.789 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:06:58.789 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.340770+0000", 2024-05-15T13:06:58.789 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1789999999999997e-06 2024-05-15T13:06:58.789 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.789 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.789 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.789 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.232176+0000", 2024-05-15T13:06:58.789 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.89140557399999998 2024-05-15T13:06:58.789 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.789 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.789 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:06:58.789 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.232270+0000", 2024-05-15T13:06:58.789 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.4537999999999995e-05 2024-05-15T13:06:58.789 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.790 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.790 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 6,9", 2024-05-15T13:06:58.790 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.232387+0000", 2024-05-15T13:06:58.790 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000116457 2024-05-15T13:06:58.790 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.790 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.790 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:06:58.790 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.237389+0000", 2024-05-15T13:06:58.790 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0050019310000000003 2024-05-15T13:06:58.790 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.790 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.790 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:58.790 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.237420+0000", 2024-05-15T13:06:58.790 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.0800000000000003e-05 2024-05-15T13:06:58.791 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.791 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.791 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:58.791 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.259232+0000", 2024-05-15T13:06:58.791 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.021811939999999998 2024-05-15T13:06:58.791 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.791 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.791 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:06:58.791 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.259285+0000", 2024-05-15T13:06:58.791 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3619000000000003e-05 2024-05-15T13:06:58.791 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.791 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.791 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:58.791 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.259326+0000", 2024-05-15T13:06:58.792 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0632e-05 2024-05-15T13:06:58.792 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.792 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:58.792 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.792 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.792 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.792 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.24832.0:258489 3.d 3:b8cc8abc:::benchmark_data_smithi177_88104_object124867:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e304)", 2024-05-15T13:06:58.792 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:57:13.351093+0000", 2024-05-15T13:06:58.792 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 585.35386774000006, 2024-05-15T13:06:58.792 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.908396655, 2024-05-15T13:06:58.792 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:58.792 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:06:58.792 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:58.792 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.24832", 2024-05-15T13:06:58.792 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/1345222897", 2024-05-15T13:06:58.793 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 258489 2024-05-15T13:06:58.793 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.793 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:58.793 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.793 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:58.793 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.351093+0000", 2024-05-15T13:06:58.793 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.793 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.793 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.793 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:58.793 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.351093+0000", 2024-05-15T13:06:58.793 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.793 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.794 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.794 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:58.794 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.351093+0000", 2024-05-15T13:06:58.794 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.9100000000000002e-07 2024-05-15T13:06:58.794 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.794 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.794 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:58.794 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.351098+0000", 2024-05-15T13:06:58.794 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.5240000000000002e-06 2024-05-15T13:06:58.794 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.794 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.794 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:58.794 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.351109+0000", 2024-05-15T13:06:58.794 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1077000000000001e-05 2024-05-15T13:06:58.795 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.795 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.795 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:58.795 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.351117+0000", 2024-05-15T13:06:58.795 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9049999999999997e-06 2024-05-15T13:06:58.795 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.795 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.795 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.795 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.351169+0000", 2024-05-15T13:06:58.795 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1582999999999997e-05 2024-05-15T13:06:58.795 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.795 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.795 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:06:58.795 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:13.351178+0000", 2024-05-15T13:06:58.795 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1859999999999998e-06 2024-05-15T13:06:58.796 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.796 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.796 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.796 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.237447+0000", 2024-05-15T13:06:58.796 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.88626885899999996 2024-05-15T13:06:58.796 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.796 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.796 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:06:58.796 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.237557+0000", 2024-05-15T13:06:58.796 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00011013799999999999 2024-05-15T13:06:58.796 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.796 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.796 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 6,9", 2024-05-15T13:06:58.796 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.237669+0000", 2024-05-15T13:06:58.796 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000111978 2024-05-15T13:06:58.797 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.797 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.797 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:06:58.797 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.248986+0000", 2024-05-15T13:06:58.797 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.011317196 2024-05-15T13:06:58.797 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.797 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.797 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:58.797 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.259367+0000", 2024-05-15T13:06:58.797 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.010381082 2024-05-15T13:06:58.797 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.797 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.797 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:06:58.797 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.259409+0000", 2024-05-15T13:06:58.798 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.1486999999999998e-05 2024-05-15T13:06:58.798 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.798 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.798 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:06:58.798 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.259457+0000", 2024-05-15T13:06:58.798 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8925999999999999e-05 2024-05-15T13:06:58.798 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.798 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.798 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:58.798 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:57:14.259489+0000", 2024-05-15T13:06:58.798 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.1822999999999997e-05 2024-05-15T13:06:58.798 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.798 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:58.799 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.799 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.799 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.799 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:140270 4.2 4:497730ee:::benchmark_data_smithi177_99756_object20298:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e356)", 2024-05-15T13:06:58.799 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:00:10.504723+0000", 2024-05-15T13:06:58.799 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 408.200237808, 2024-05-15T13:06:58.799 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8829001480000009, 2024-05-15T13:06:58.799 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:58.799 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:58.799 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:58.799 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:06:58.799 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:58.799 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 140270 2024-05-15T13:06:58.800 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.800 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:58.800 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.800 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:58.800 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:10.504723+0000", 2024-05-15T13:06:58.800 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.800 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.800 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.800 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:58.800 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:10.504723+0000", 2024-05-15T13:06:58.800 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.800 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.800 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.800 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:58.801 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:10.504724+0000", 2024-05-15T13:06:58.801 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2160000000000001e-06 2024-05-15T13:06:58.801 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.801 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.801 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:58.801 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:10.504732+0000", 2024-05-15T13:06:58.801 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.6070000000000008e-06 2024-05-15T13:06:58.801 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.801 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.801 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:58.801 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:10.504741+0000", 2024-05-15T13:06:58.801 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.3769999999999996e-06 2024-05-15T13:06:58.801 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.801 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.801 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:58.802 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:10.504747+0000", 2024-05-15T13:06:58.802 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7560000000000002e-06 2024-05-15T13:06:58.802 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.802 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.802 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.802 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:10.504809+0000", 2024-05-15T13:06:58.802 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1198000000000006e-05 2024-05-15T13:06:58.802 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.802 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.802 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:06:58.802 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:10.504918+0000", 2024-05-15T13:06:58.802 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000109792 2024-05-15T13:06:58.802 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.803 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.803 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.803 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:19.387607+0000", 2024-05-15T13:06:58.803 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8826883139999993 2024-05-15T13:06:58.803 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.803 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.803 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:58.803 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:19.387623+0000", 2024-05-15T13:06:58.803 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5888000000000001e-05 2024-05-15T13:06:58.803 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.803 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:58.803 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.803 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.803 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.804 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:166906 4.2 4:4263f3a5:::benchmark_data_smithi177_99756_object46934:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e370)", 2024-05-15T13:06:58.804 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:00:53.753115+0000", 2024-05-15T13:06:58.804 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 364.95184546399997, 2024-05-15T13:06:58.804 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.73727652, 2024-05-15T13:06:58.804 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:58.804 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:58.804 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:58.804 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:06:58.804 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:58.804 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 166906 2024-05-15T13:06:58.804 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.804 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:58.804 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.805 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:58.805 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:53.753115+0000", 2024-05-15T13:06:58.805 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.805 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.805 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.805 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:58.805 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:53.753115+0000", 2024-05-15T13:06:58.805 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.805 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.805 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.805 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:58.805 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:53.753116+0000", 2024-05-15T13:06:58.805 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1850000000000001e-06 2024-05-15T13:06:58.805 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.806 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.806 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:58.806 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:53.753120+0000", 2024-05-15T13:06:58.806 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.189e-06 2024-05-15T13:06:58.806 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.806 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.806 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:58.806 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:53.753128+0000", 2024-05-15T13:06:58.806 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1820000000000006e-06 2024-05-15T13:06:58.806 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.806 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.806 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:58.806 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:53.753135+0000", 2024-05-15T13:06:58.807 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4200000000000004e-06 2024-05-15T13:06:58.807 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.807 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.807 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.807 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:53.754963+0000", 2024-05-15T13:06:58.807 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001827815 2024-05-15T13:06:58.807 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.807 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.807 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:06:58.807 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:53.755057+0000", 2024-05-15T13:06:58.807 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.4029000000000004e-05 2024-05-15T13:06:58.807 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.807 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.807 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.807 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:01.490381+0000", 2024-05-15T13:06:58.808 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.735324243 2024-05-15T13:06:58.808 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.808 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.808 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:58.808 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:01.490391+0000", 2024-05-15T13:06:58.808 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0457e-05 2024-05-15T13:06:58.808 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.808 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:58.808 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.808 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.808 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.808 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:172275 4.2 4:41199a6c:::benchmark_data_smithi177_99756_object52303:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-05-15T13:06:58.808 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:02.283620+0000", 2024-05-15T13:06:58.808 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 356.42134009099999, 2024-05-15T13:06:58.808 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.4264981429999999, 2024-05-15T13:06:58.809 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:06:58.809 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:06:58.809 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:06:58.809 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:06:58.809 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:06:58.809 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 172275 2024-05-15T13:06:58.809 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.809 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:06:58.809 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.809 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:06:58.809 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:02.283620+0000", 2024-05-15T13:06:58.809 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.809 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.809 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.809 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:06:58.809 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:02.283620+0000", 2024-05-15T13:06:58.810 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:06:58.810 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.810 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.810 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:06:58.810 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:02.283621+0000", 2024-05-15T13:06:58.810 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9400000000000005e-07 2024-05-15T13:06:58.810 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.810 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.810 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:06:58.810 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:02.283624+0000", 2024-05-15T13:06:58.810 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.0929999999999999e-06 2024-05-15T13:06:58.810 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.810 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.810 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:06:58.811 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:02.283631+0000", 2024-05-15T13:06:58.811 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9959999999999996e-06 2024-05-15T13:06:58.811 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.811 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.811 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:06:58.811 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:02.283638+0000", 2024-05-15T13:06:58.811 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7100000000000001e-06 2024-05-15T13:06:58.811 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.811 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.811 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.811 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:02.283678+0000", 2024-05-15T13:06:58.811 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0380000000000003e-05 2024-05-15T13:06:58.811 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.811 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.811 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:06:58.811 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:02.283749+0000", 2024-05-15T13:06:58.812 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.1150999999999997e-05 2024-05-15T13:06:58.812 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.812 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.812 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:06:58.812 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:04.710098+0000", 2024-05-15T13:06:58.812 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.4263489410000001 2024-05-15T13:06:58.812 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:06:58.812 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:06:58.812 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:06:58.812 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:04.710118+0000", 2024-05-15T13:06:58.812 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0177999999999999e-05 2024-05-15T13:06:58.812 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.812 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:58.812 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.812 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:06:58.813 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:06:58.813 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:06:58.814 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:58 smithi120 ceph-mon[43961]: 1.3 scrub ok 2024-05-15T13:06:58.814 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:58 smithi120 ceph-mon[43961]: 5.4 scrub starts 2024-05-15T13:06:58.814 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:58 smithi120 ceph-mon[43961]: osdmap e488: 12 total, 12 up, 11 in 2024-05-15T13:06:58.814 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:58 smithi120 ceph-mon[43961]: 1.9 scrub starts 2024-05-15T13:06:58.814 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:58 smithi120 ceph-mon[43961]: 5.4 scrub ok 2024-05-15T13:06:58.814 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:58 smithi120 ceph-mon[43961]: 1.9 scrub ok 2024-05-15T13:06:58.814 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:58 smithi120 ceph-mon[43961]: pgmap v1307: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.5 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 797 op/s 2024-05-15T13:06:58.814 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:58 smithi120 ceph-mon[43961]: osdmap e489: 12 total, 12 up, 11 in 2024-05-15T13:06:58.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:58.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:06:58.815 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:06:58.815 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:58 smithi027 ceph-mon[39795]: 1.3 scrub ok 2024-05-15T13:06:58.815 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:58 smithi027 ceph-mon[39795]: 5.4 scrub starts 2024-05-15T13:06:58.815 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:58 smithi027 ceph-mon[39795]: osdmap e488: 12 total, 12 up, 11 in 2024-05-15T13:06:58.816 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:58 smithi027 ceph-mon[39795]: 1.9 scrub starts 2024-05-15T13:06:58.816 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:58 smithi027 ceph-mon[39795]: 5.4 scrub ok 2024-05-15T13:06:58.816 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:58 smithi027 ceph-mon[39795]: 1.9 scrub ok 2024-05-15T13:06:58.816 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:58 smithi027 ceph-mon[39795]: pgmap v1307: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.5 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 797 op/s 2024-05-15T13:06:58.816 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:58 smithi027 ceph-mon[39795]: osdmap e489: 12 total, 12 up, 11 in 2024-05-15T13:06:58.816 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:06:58.816 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:06:58.820 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:06:58.879 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:06:58.916 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:06:58.929 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:06:58.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:58 smithi177 ceph-mon[44303]: 1.3 scrub ok 2024-05-15T13:06:58.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:58 smithi177 ceph-mon[44303]: 5.4 scrub starts 2024-05-15T13:06:58.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:58 smithi177 ceph-mon[44303]: osdmap e488: 12 total, 12 up, 11 in 2024-05-15T13:06:58.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:58 smithi177 ceph-mon[44303]: 1.9 scrub starts 2024-05-15T13:06:58.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:58 smithi177 ceph-mon[44303]: 5.4 scrub ok 2024-05-15T13:06:58.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:58 smithi177 ceph-mon[44303]: 1.9 scrub ok 2024-05-15T13:06:58.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:58 smithi177 ceph-mon[44303]: pgmap v1307: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.5 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 797 op/s 2024-05-15T13:06:58.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:58 smithi177 ceph-mon[44303]: osdmap e489: 12 total, 12 up, 11 in 2024-05-15T13:06:58.967 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:06:59.004 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:59.053 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:06:59.055 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:06:59.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:59.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:06:59.103 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:06:59.148 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:06:59.159 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:06:59.185 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:06:59.186 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:06:59.186 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:06:59.186 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:06:59.187 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:06:59.187 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:06:59.187 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:06:59.187 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:06:59.187 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:06:59.187 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:06:59.187 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:06:59.187 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:06:59.187 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:06:59.200 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:59.214 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:06:59.476 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-05-15T13:06:59.479 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:06:59.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:59.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:06:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.1 to 1 (10000) 2024-05-15T13:06:59.582 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:06:59.611 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:06:59.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:06:59.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:06:59.766 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:06:59.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:59 smithi027 ceph-mon[39795]: 1.8 scrub starts 2024-05-15T13:06:59.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:59 smithi027 ceph-mon[39795]: 1.8 scrub ok 2024-05-15T13:06:59.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:59 smithi027 ceph-mon[39795]: 1.1 scrub starts 2024-05-15T13:06:59.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:59 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3946115862' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:06:59.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:59 smithi027 ceph-mon[39795]: 1.1 scrub ok 2024-05-15T13:06:59.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:59 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4013619159' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T13:06:59.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:06:59 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T13:06:59.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:59 smithi120 ceph-mon[43961]: 1.8 scrub starts 2024-05-15T13:06:59.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:59 smithi120 ceph-mon[43961]: 1.8 scrub ok 2024-05-15T13:06:59.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:59 smithi120 ceph-mon[43961]: 1.1 scrub starts 2024-05-15T13:06:59.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:59 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3946115862' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:06:59.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:59 smithi120 ceph-mon[43961]: 1.1 scrub ok 2024-05-15T13:06:59.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:59 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4013619159' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T13:06:59.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:06:59 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T13:06:59.791 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:06:59.853 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:06:59.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:59 smithi177 ceph-mon[44303]: 1.8 scrub starts 2024-05-15T13:06:59.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:59 smithi177 ceph-mon[44303]: 1.8 scrub ok 2024-05-15T13:06:59.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:59 smithi177 ceph-mon[44303]: 1.1 scrub starts 2024-05-15T13:06:59.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:59 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3946115862' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:06:59.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:59 smithi177 ceph-mon[44303]: 1.1 scrub ok 2024-05-15T13:06:59.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:59 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4013619159' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T13:06:59.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:06:59 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T13:07:00.064 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:07:00.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:00.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:07:00.168 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:07:00.214 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:00.248 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:07:00.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:00.251 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:07:00.352 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:07:00.399 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:00.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:00 smithi027 ceph-mon[39795]: 1.2 scrub starts 2024-05-15T13:07:00.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:00 smithi027 ceph-mon[39795]: 1.2 scrub ok 2024-05-15T13:07:00.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:00 smithi027 ceph-mon[39795]: 1.7 scrub starts 2024-05-15T13:07:00.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:00 smithi027 ceph-mon[39795]: 1.7 scrub ok 2024-05-15T13:07:00.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:00 smithi027 ceph-mon[39795]: 1.5 scrub starts 2024-05-15T13:07:00.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:00 smithi027 ceph-mon[39795]: 1.5 scrub ok 2024-05-15T13:07:00.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:00 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T13:07:00.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:00 smithi027 ceph-mon[39795]: osdmap e490: 12 total, 12 up, 11 in 2024-05-15T13:07:00.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:00 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T13:07:00.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:00 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T13:07:00.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:00 smithi027 ceph-mon[39795]: pgmap v1310: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.4 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 926 op/s 2024-05-15T13:07:00.755 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:07:00.757 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:00 smithi120 ceph-mon[43961]: 1.2 scrub starts 2024-05-15T13:07:00.757 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:00 smithi120 ceph-mon[43961]: 1.2 scrub ok 2024-05-15T13:07:00.757 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:00 smithi120 ceph-mon[43961]: 1.7 scrub starts 2024-05-15T13:07:00.757 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:00 smithi120 ceph-mon[43961]: 1.7 scrub ok 2024-05-15T13:07:00.757 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:00 smithi120 ceph-mon[43961]: 1.5 scrub starts 2024-05-15T13:07:00.758 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:00 smithi120 ceph-mon[43961]: 1.5 scrub ok 2024-05-15T13:07:00.758 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:00 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T13:07:00.758 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:00 smithi120 ceph-mon[43961]: osdmap e490: 12 total, 12 up, 11 in 2024-05-15T13:07:00.758 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:00 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T13:07:00.758 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:00 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T13:07:00.758 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:00 smithi120 ceph-mon[43961]: pgmap v1310: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.4 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 926 op/s 2024-05-15T13:07:00.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:00.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:07:00.860 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:07:00.893 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:00.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:00 smithi177 ceph-mon[44303]: 1.2 scrub starts 2024-05-15T13:07:00.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:00 smithi177 ceph-mon[44303]: 1.2 scrub ok 2024-05-15T13:07:00.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:00 smithi177 ceph-mon[44303]: 1.7 scrub starts 2024-05-15T13:07:00.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:00 smithi177 ceph-mon[44303]: 1.7 scrub ok 2024-05-15T13:07:00.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:00 smithi177 ceph-mon[44303]: 1.5 scrub starts 2024-05-15T13:07:00.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:00 smithi177 ceph-mon[44303]: 1.5 scrub ok 2024-05-15T13:07:00.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:00 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T13:07:00.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:00 smithi177 ceph-mon[44303]: osdmap e490: 12 total, 12 up, 11 in 2024-05-15T13:07:00.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:00 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T13:07:00.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:00 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T13:07:00.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:00 smithi177 ceph-mon[44303]: pgmap v1310: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.4 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 926 op/s 2024-05-15T13:07:01.246 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:07:01.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:01.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:07:01.356 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:07:01.406 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:01.486 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:07:01.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:01.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:07:01.594 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:07:01.649 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:01.673 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:07:01.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:01.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:07:01.777 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:07:01.852 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:01.929 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:07:01.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:01.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:07:02.041 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:07:02.066 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:07:02.066 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:07:02.087 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:02.115 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:07:02.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:02.119 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:07:02.220 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:07:02.262 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:02.301 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:07:02.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:02.310 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:07:02.411 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:07:02.468 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:02.508 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 2 1 2024-05-15T13:07:02.867 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-05-15T13:07:03.256 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:07:03.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:03.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:07:03.261 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:07:03.360 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:07:03.385 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:03 smithi177 ceph-mon[44303]: 5.a scrub starts 2024-05-15T13:07:03.385 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:03 smithi177 ceph-mon[44303]: 5.a scrub ok 2024-05-15T13:07:03.385 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:03 smithi177 ceph-mon[44303]: pgmap v1311: 27 pgs: 27 active+clean; 1.3 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 822 op/s 2024-05-15T13:07:03.385 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:03 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:07:03.385 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:03 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:07:03.387 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:03.405 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:07:03.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:03.408 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:07:03.509 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:07:03.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:03 smithi027 ceph-mon[39795]: 5.a scrub starts 2024-05-15T13:07:03.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:03 smithi027 ceph-mon[39795]: 5.a scrub ok 2024-05-15T13:07:03.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:03 smithi027 ceph-mon[39795]: pgmap v1311: 27 pgs: 27 active+clean; 1.3 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 822 op/s 2024-05-15T13:07:03.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:03 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:07:03.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:03 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:07:03.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:03 smithi120 ceph-mon[43961]: 5.a scrub starts 2024-05-15T13:07:03.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:03 smithi120 ceph-mon[43961]: 5.a scrub ok 2024-05-15T13:07:03.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:03 smithi120 ceph-mon[43961]: pgmap v1311: 27 pgs: 27 active+clean; 1.3 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 822 op/s 2024-05-15T13:07:03.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:03 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:07:03.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:03 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:07:03.547 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:03.575 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:07:03.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:03.577 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:07:03.589 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:07:03.678 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:07:03.719 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:03.751 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:07:03.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:03.758 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:07:03.859 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:07:03.926 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:03.963 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:07:03.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:03.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:07:04.069 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:07:04.108 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:04.129 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:04 smithi027 ceph-mon[39795]: 5.f scrub starts 2024-05-15T13:07:04.129 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:04 smithi027 ceph-mon[39795]: 5.f scrub ok 2024-05-15T13:07:04.129 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:04 smithi027 ceph-mon[39795]: osdmap e491: 12 total, 12 up, 11 in 2024-05-15T13:07:04.129 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:04 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:07:04.129 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:04 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:07:04.129 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:04 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2414940729' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:07:04.129 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:04 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:07:04.134 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:07:04.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:04.141 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:07:04.242 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:07:04.277 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:04 smithi120 ceph-mon[43961]: 5.f scrub starts 2024-05-15T13:07:04.278 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:04 smithi120 ceph-mon[43961]: 5.f scrub ok 2024-05-15T13:07:04.278 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:04 smithi120 ceph-mon[43961]: osdmap e491: 12 total, 12 up, 11 in 2024-05-15T13:07:04.278 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:04 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:07:04.278 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:04 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:07:04.278 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:04 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2414940729' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:07:04.278 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:04 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:07:04.286 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:04.426 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:07:04.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:04 smithi177 ceph-mon[44303]: 5.f scrub starts 2024-05-15T13:07:04.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:04 smithi177 ceph-mon[44303]: 5.f scrub ok 2024-05-15T13:07:04.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:04 smithi177 ceph-mon[44303]: osdmap e491: 12 total, 12 up, 11 in 2024-05-15T13:07:04.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:04 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:07:04.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:04 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:07:04.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:04 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2414940729' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:07:04.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:04 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:07:04.857 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:07:04.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:04.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:07:04.962 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:07:05.021 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:05.058 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:07:05.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:05.064 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:07:05.165 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:07:05.188 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:07:05.234 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:05.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:05 smithi177 ceph-mon[44303]: pgmap v1313: 27 pgs: 27 active+clean; 1.2 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 849 op/s 2024-05-15T13:07:05.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:05 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:07:05.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:05 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T13:07:05.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:05 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:07:05.435 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:05 smithi177 ceph-mon[44303]: osdmap e492: 12 total, 12 up, 11 in 2024-05-15T13:07:05.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:05 smithi027 ceph-mon[39795]: pgmap v1313: 27 pgs: 27 active+clean; 1.2 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 849 op/s 2024-05-15T13:07:05.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:05 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:07:05.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:05 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T13:07:05.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:05 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:07:05.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:05 smithi027 ceph-mon[39795]: osdmap e492: 12 total, 12 up, 11 in 2024-05-15T13:07:05.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:05 smithi120 ceph-mon[43961]: pgmap v1313: 27 pgs: 27 active+clean; 1.2 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 849 op/s 2024-05-15T13:07:05.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:05 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:07:05.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:05 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T13:07:05.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:05 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:07:05.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:05 smithi120 ceph-mon[43961]: osdmap e492: 12 total, 12 up, 11 in 2024-05-15T13:07:05.612 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:07:05.612 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:07:06.384 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-05-15T13:07:06.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:06 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:07:06.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:06 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:07:06.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:06 smithi177 ceph-mon[44303]: osdmap e493: 12 total, 12 up, 11 in 2024-05-15T13:07:06.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:06 smithi177 ceph-mon[44303]: pgmap v1316: 27 pgs: 27 active+clean; 1.1 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 724 op/s 2024-05-15T13:07:06.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:06 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:07:06.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:06 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:07:06.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:06 smithi027 ceph-mon[39795]: osdmap e493: 12 total, 12 up, 11 in 2024-05-15T13:07:06.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:06 smithi027 ceph-mon[39795]: pgmap v1316: 27 pgs: 27 active+clean; 1.1 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 724 op/s 2024-05-15T13:07:06.774 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:07:06.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:06 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:07:06.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:06 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:07:06.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:06 smithi120 ceph-mon[43961]: osdmap e493: 12 total, 12 up, 11 in 2024-05-15T13:07:06.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:06 smithi120 ceph-mon[43961]: pgmap v1316: 27 pgs: 27 active+clean; 1.1 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 724 op/s 2024-05-15T13:07:06.943 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:07:06.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:06.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:07:07.056 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:07:07.105 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:07.128 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:07:07.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:07.130 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:07:07.231 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:07:07.299 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:07.407 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:07:07.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:07.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:07:07.512 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:07:07.574 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:07.615 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:07:07.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:07.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:07:07.682 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:07:07.730 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:07:07.778 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:07.811 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:07:07.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:07.814 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:07:07.915 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:07:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:08.037 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:07:08.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:08.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:07:08.146 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:07:08.181 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:08.987 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:07:09.155 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.2 to 1 (10000) 2024-05-15T13:07:09.188 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:09 smithi027 ceph-mon[39795]: pgmap v1317: 27 pgs: 27 active+clean; 1015 MiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 896 op/s 2024-05-15T13:07:09.188 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:09 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/448649896' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:07:09.188 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:09 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:07:09.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:09 smithi177 ceph-mon[44303]: pgmap v1317: 27 pgs: 27 active+clean; 1015 MiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 896 op/s 2024-05-15T13:07:09.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:09 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/448649896' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:07:09.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:09 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:07:09.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:09 smithi120 ceph-mon[43961]: pgmap v1317: 27 pgs: 27 active+clean; 1015 MiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 896 op/s 2024-05-15T13:07:09.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:09 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/448649896' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:07:09.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:09 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:07:10.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 3 1 2024-05-15T13:07:10.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:10 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:07:10.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:10 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:07:10.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:10 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T13:07:10.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:10 smithi027 ceph-mon[39795]: osdmap e494: 12 total, 12 up, 11 in 2024-05-15T13:07:10.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:10 smithi027 ceph-mon[39795]: osdmap e495: 12 total, 12 up, 11 in 2024-05-15T13:07:10.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:10 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:07:10.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:10 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:07:10.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:10 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T13:07:10.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:10 smithi177 ceph-mon[44303]: osdmap e494: 12 total, 12 up, 11 in 2024-05-15T13:07:10.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:10 smithi177 ceph-mon[44303]: osdmap e495: 12 total, 12 up, 11 in 2024-05-15T13:07:10.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:10 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:07:10.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:10 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:07:10.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:10 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T13:07:10.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:10 smithi120 ceph-mon[43961]: osdmap e494: 12 total, 12 up, 11 in 2024-05-15T13:07:10.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:10 smithi120 ceph-mon[43961]: osdmap e495: 12 total, 12 up, 11 in 2024-05-15T13:07:10.763 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:07:10.834 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:07:10.962 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:07:10.998 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:07:11.040 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:07:11.074 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:07:11.105 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:07:11.148 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:07:11.179 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:07:11.211 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:07:11.240 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:07:11.267 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:07:11.290 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:07:11.321 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:07:11.331 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:07:11.331 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:07:11.331 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:07:11.331 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:07:11.331 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:07:11.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:07:11.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:07:11.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:07:11.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:07:11.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:07:11.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:07:11.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:07:11.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:11 smithi177 ceph-mon[44303]: pgmap v1319: 27 pgs: 27 active+clean; 876 MiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 922 op/s 2024-05-15T13:07:11.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:11 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2340014259' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:07:11.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:11 smithi027 ceph-mon[39795]: pgmap v1319: 27 pgs: 27 active+clean; 876 MiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 922 op/s 2024-05-15T13:07:11.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:11 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2340014259' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:07:11.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:11 smithi120 ceph-mon[43961]: pgmap v1319: 27 pgs: 27 active+clean; 876 MiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 922 op/s 2024-05-15T13:07:11.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:11 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2340014259' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:07:11.661 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:07:11.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:11.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:07:11.765 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:07:11.809 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:11.837 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:07:11.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:11.844 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:07:11.945 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:07:12.003 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:12.046 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:07:12.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:12.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:07:12.150 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:07:12.197 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:12.404 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:07:12.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:12.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:07:12.511 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:07:12.513 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:07:12.513 INFO:teuthology.orchestra.run.smithi120.stdout: "size": 20, 2024-05-15T13:07:12.513 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 600, 2024-05-15T13:07:12.513 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [ 2024-05-15T13:07:12.514 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.514 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:101368 4.5 4:abf3cbb1:::benchmark_data_smithi177_99756_object101367:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-05-15T13:07:12.514 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:59:10.304149+0000", 2024-05-15T13:07:12.514 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 482.21228241199998, 2024-05-15T13:07:12.514 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.5083956729999999, 2024-05-15T13:07:12.514 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:07:12.514 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:07:12.514 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:07:12.514 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:07:12.514 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:12.514 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 101368 2024-05-15T13:07:12.514 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.515 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:07:12.515 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.515 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:07:12.515 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.304149+0000", 2024-05-15T13:07:12.515 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.515 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.515 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.515 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:07:12.515 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.304149+0000", 2024-05-15T13:07:12.515 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.515 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.515 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.515 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:07:12.516 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.304150+0000", 2024-05-15T13:07:12.516 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2099999999999997e-07 2024-05-15T13:07:12.516 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.516 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.516 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:07:12.516 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.304218+0000", 2024-05-15T13:07:12.516 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8374999999999995e-05 2024-05-15T13:07:12.516 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.516 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.516 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:07:12.516 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.304226+0000", 2024-05-15T13:07:12.516 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3590000000000002e-06 2024-05-15T13:07:12.517 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.517 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.517 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:07:12.517 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.304232+0000", 2024-05-15T13:07:12.517 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1659999999999999e-06 2024-05-15T13:07:12.517 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.517 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.517 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.517 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.304280+0000", 2024-05-15T13:07:12.517 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7682000000000002e-05 2024-05-15T13:07:12.517 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.517 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.517 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:07:12.517 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.304384+0000", 2024-05-15T13:07:12.518 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00010478499999999999 2024-05-15T13:07:12.518 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.518 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.518 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,10", 2024-05-15T13:07:12.518 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.304491+0000", 2024-05-15T13:07:12.518 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000106982 2024-05-15T13:07:12.518 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.518 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.518 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:07:12.518 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.307821+0000", 2024-05-15T13:07:12.518 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0033297019999999999 2024-05-15T13:07:12.518 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.519 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.519 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:12.519 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.308306+0000", 2024-05-15T13:07:12.519 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00048475299999999999 2024-05-15T13:07:12.519 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.519 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.519 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.519 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.315153+0000", 2024-05-15T13:07:12.519 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.006847517 2024-05-15T13:07:12.519 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.519 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.519 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:07:12.519 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:10.315161+0000", 2024-05-15T13:07:12.519 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.255e-06 2024-05-15T13:07:12.520 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.520 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.520 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.520 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312207+0000", 2024-05-15T13:07:12.520 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.99704610400000004 2024-05-15T13:07:12.520 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.520 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.520 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:07:12.520 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312223+0000", 2024-05-15T13:07:12.520 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6019999999999999e-05 2024-05-15T13:07:12.520 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.520 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.520 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.521 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.812476+0000", 2024-05-15T13:07:12.521 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.500253813 2024-05-15T13:07:12.521 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.521 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.521 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:07:12.521 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.812545+0000", 2024-05-15T13:07:12.521 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8639000000000005e-05 2024-05-15T13:07:12.521 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.521 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:07:12.521 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.521 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.521 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.521 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:101391 4.8 4:1d63c299:::benchmark_data_smithi177_99756_object101390:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e343)", 2024-05-15T13:07:12.521 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T12:59:11.312996+0000", 2024-05-15T13:07:12.522 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 481.20343561200002, 2024-05-15T13:07:12.522 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.572125896, 2024-05-15T13:07:12.522 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:07:12.522 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:12.522 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:07:12.522 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:07:12.522 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:12.522 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 101391 2024-05-15T13:07:12.522 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.522 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:07:12.522 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.522 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:07:12.522 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312996+0000", 2024-05-15T13:07:12.523 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.523 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.523 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.523 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:07:12.523 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312996+0000", 2024-05-15T13:07:12.523 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.523 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.523 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.523 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:07:12.523 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.312998+0000", 2024-05-15T13:07:12.523 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6750000000000001e-06 2024-05-15T13:07:12.523 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.523 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.523 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:07:12.524 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313176+0000", 2024-05-15T13:07:12.524 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00017834599999999999 2024-05-15T13:07:12.524 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.524 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.524 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:07:12.524 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313191+0000", 2024-05-15T13:07:12.524 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4741e-05 2024-05-15T13:07:12.524 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.524 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.524 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:07:12.524 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313201+0000", 2024-05-15T13:07:12.524 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.5370000000000003e-06 2024-05-15T13:07:12.524 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.525 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.525 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.525 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313254+0000", 2024-05-15T13:07:12.525 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3526999999999998e-05 2024-05-15T13:07:12.525 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.525 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.525 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:07:12.525 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:11.313260+0000", 2024-05-15T13:07:12.525 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8590000000000001e-06 2024-05-15T13:07:12.525 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.525 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.525 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.525 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.867433+0000", 2024-05-15T13:07:12.525 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5541727939999999 2024-05-15T13:07:12.526 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.526 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.526 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:07:12.526 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.867550+0000", 2024-05-15T13:07:12.526 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00011683 2024-05-15T13:07:12.526 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.526 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.526 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 9,11", 2024-05-15T13:07:12.526 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.867653+0000", 2024-05-15T13:07:12.526 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000103139 2024-05-15T13:07:12.526 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.526 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.527 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:12.527 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.878453+0000", 2024-05-15T13:07:12.527 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.010800149 2024-05-15T13:07:12.527 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.527 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.527 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:07:12.527 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.880516+0000", 2024-05-15T13:07:12.527 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0020627269999999999 2024-05-15T13:07:12.527 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.527 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.527 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:12.527 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.880623+0000", 2024-05-15T13:07:12.527 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000107155 2024-05-15T13:07:12.527 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.528 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.528 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:07:12.528 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.880670+0000", 2024-05-15T13:07:12.528 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6801999999999997e-05 2024-05-15T13:07:12.528 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.528 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.528 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:07:12.528 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T12:59:12.885122+0000", 2024-05-15T13:07:12.528 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.004452615 2024-05-15T13:07:12.528 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.528 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:07:12.528 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.528 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.528 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.529 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:139676 4.8 4:1fd785be:::benchmark_data_smithi177_99756_object19704:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e356)", 2024-05-15T13:07:12.529 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:00:09.558323+0000", 2024-05-15T13:07:12.529 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 422.95810896199998, 2024-05-15T13:07:12.529 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6374820859999999, 2024-05-15T13:07:12.529 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:07:12.529 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:07:12.529 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:07:12.529 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:07:12.529 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:12.529 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 139676 2024-05-15T13:07:12.529 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.529 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:07:12.529 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.529 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:07:12.530 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:09.558323+0000", 2024-05-15T13:07:12.530 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.530 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.530 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.530 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:07:12.530 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:09.558323+0000", 2024-05-15T13:07:12.530 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.530 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.530 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.530 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:07:12.530 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:09.558323+0000", 2024-05-15T13:07:12.530 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.8799999999999998e-07 2024-05-15T13:07:12.530 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.531 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.531 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:07:12.531 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:09.558325+0000", 2024-05-15T13:07:12.531 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2680000000000001e-06 2024-05-15T13:07:12.531 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.531 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.531 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:07:12.531 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:09.558337+0000", 2024-05-15T13:07:12.531 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1755e-05 2024-05-15T13:07:12.531 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.531 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.531 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:07:12.531 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:09.558348+0000", 2024-05-15T13:07:12.532 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0939e-05 2024-05-15T13:07:12.532 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.532 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.532 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.532 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:09.561275+0000", 2024-05-15T13:07:12.532 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.002926409 2024-05-15T13:07:12.532 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.532 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.532 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:07:12.532 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:09.561328+0000", 2024-05-15T13:07:12.532 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3238000000000001e-05 2024-05-15T13:07:12.532 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.532 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.532 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.532 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:11.195792+0000", 2024-05-15T13:07:12.533 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6344638379999998 2024-05-15T13:07:12.533 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.533 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.533 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:07:12.533 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:11.195805+0000", 2024-05-15T13:07:12.533 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3351e-05 2024-05-15T13:07:12.533 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.533 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:07:12.533 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.533 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.533 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.533 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:143378 4.8 4:1ea41d0a:::benchmark_data_smithi177_99756_object23406:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e357)", 2024-05-15T13:07:12.533 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:00:15.352051+0000", 2024-05-15T13:07:12.533 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 417.16438041599997, 2024-05-15T13:07:12.534 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3467720810000001, 2024-05-15T13:07:12.534 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:07:12.534 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:07:12.534 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:07:12.534 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:07:12.534 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:12.534 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 143378 2024-05-15T13:07:12.534 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.534 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:07:12.534 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.534 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:07:12.534 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352051+0000", 2024-05-15T13:07:12.534 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.535 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.535 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.535 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:07:12.535 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352051+0000", 2024-05-15T13:07:12.535 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.535 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.535 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.535 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:07:12.535 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352052+0000", 2024-05-15T13:07:12.535 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9400000000000005e-07 2024-05-15T13:07:12.535 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.535 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.535 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:07:12.536 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352055+0000", 2024-05-15T13:07:12.536 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.066e-06 2024-05-15T13:07:12.536 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.536 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.536 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:07:12.536 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352063+0000", 2024-05-15T13:07:12.536 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9389999999999997e-06 2024-05-15T13:07:12.536 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.536 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.536 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:07:12.536 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352070+0000", 2024-05-15T13:07:12.536 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.5080000000000002e-06 2024-05-15T13:07:12.536 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.536 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.536 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.537 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352120+0000", 2024-05-15T13:07:12.537 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0432000000000001e-05 2024-05-15T13:07:12.537 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.537 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.537 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:07:12.537 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352414+0000", 2024-05-15T13:07:12.537 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00029437899999999999 2024-05-15T13:07:12.537 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.537 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.537 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.537 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:19.698800+0000", 2024-05-15T13:07:12.537 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3463857209999999 2024-05-15T13:07:12.537 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.537 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.538 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:07:12.538 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:19.698824+0000", 2024-05-15T13:07:12.538 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.3442e-05 2024-05-15T13:07:12.538 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.538 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:07:12.538 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.538 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.538 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.538 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:143379 4.8 4:1ab9a300:::benchmark_data_smithi177_99756_object23407:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e357)", 2024-05-15T13:07:12.538 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:00:15.352081+0000", 2024-05-15T13:07:12.538 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 417.16435117200001, 2024-05-15T13:07:12.538 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3569042859999998, 2024-05-15T13:07:12.538 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:07:12.539 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:07:12.539 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:07:12.539 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:07:12.539 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:12.539 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 143379 2024-05-15T13:07:12.539 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.539 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:07:12.539 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.539 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:07:12.539 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352081+0000", 2024-05-15T13:07:12.539 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.539 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.539 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.539 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:07:12.539 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352081+0000", 2024-05-15T13:07:12.540 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.540 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.540 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.540 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:07:12.540 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352081+0000", 2024-05-15T13:07:12.540 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7500000000000001e-07 2024-05-15T13:07:12.540 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.540 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.540 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:07:12.540 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352083+0000", 2024-05-15T13:07:12.540 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2680000000000001e-06 2024-05-15T13:07:12.540 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.540 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.540 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:07:12.541 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352090+0000", 2024-05-15T13:07:12.541 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9e-06 2024-05-15T13:07:12.541 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.541 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.541 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:07:12.541 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352096+0000", 2024-05-15T13:07:12.541 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5589999999999996e-06 2024-05-15T13:07:12.541 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.541 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.541 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.541 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352452+0000", 2024-05-15T13:07:12.541 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00035633099999999999 2024-05-15T13:07:12.541 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.541 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.542 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:07:12.542 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352530+0000", 2024-05-15T13:07:12.542 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7948999999999995e-05 2024-05-15T13:07:12.542 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.542 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.542 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.542 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:19.708972+0000", 2024-05-15T13:07:12.542 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3564422279999997 2024-05-15T13:07:12.542 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.542 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.542 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:07:12.542 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:19.708985+0000", 2024-05-15T13:07:12.543 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2676e-05 2024-05-15T13:07:12.543 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.543 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:07:12.543 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.543 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.543 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.543 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:143381 4.8 4:1163b2cb:::benchmark_data_smithi177_99756_object23409:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e357)", 2024-05-15T13:07:12.543 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:00:15.352103+0000", 2024-05-15T13:07:12.543 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 417.16432887100001, 2024-05-15T13:07:12.543 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3658745979999996, 2024-05-15T13:07:12.543 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:07:12.543 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:07:12.543 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:07:12.544 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:07:12.544 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:12.544 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 143381 2024-05-15T13:07:12.544 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.544 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:07:12.544 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.544 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:07:12.544 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352103+0000", 2024-05-15T13:07:12.544 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.544 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.544 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.544 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:07:12.544 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352103+0000", 2024-05-15T13:07:12.545 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.545 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.545 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.545 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:07:12.545 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352103+0000", 2024-05-15T13:07:12.545 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5999999999999999e-07 2024-05-15T13:07:12.545 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.545 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.545 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:07:12.545 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352106+0000", 2024-05-15T13:07:12.545 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.182e-06 2024-05-15T13:07:12.545 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.545 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.545 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:07:12.545 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352111+0000", 2024-05-15T13:07:12.546 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6380000000000001e-06 2024-05-15T13:07:12.546 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.546 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.546 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:07:12.546 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352116+0000", 2024-05-15T13:07:12.546 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9119999999999997e-06 2024-05-15T13:07:12.546 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.546 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.546 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.546 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352559+0000", 2024-05-15T13:07:12.546 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00044265899999999998 2024-05-15T13:07:12.546 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.546 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.547 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:07:12.547 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352608+0000", 2024-05-15T13:07:12.547 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9379000000000003e-05 2024-05-15T13:07:12.547 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.547 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.547 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.547 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:19.717968+0000", 2024-05-15T13:07:12.547 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3653598359999997 2024-05-15T13:07:12.547 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.547 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.547 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:07:12.547 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:19.717978+0000", 2024-05-15T13:07:12.547 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.6320000000000003e-06 2024-05-15T13:07:12.547 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.548 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:07:12.548 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.548 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.548 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.548 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:143382 4.8 4:13e3467a:::benchmark_data_smithi177_99756_object23410:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e357)", 2024-05-15T13:07:12.548 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:00:15.352122+0000", 2024-05-15T13:07:12.548 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 417.16430977900001, 2024-05-15T13:07:12.548 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3737501510000003, 2024-05-15T13:07:12.548 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:07:12.548 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:07:12.548 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:07:12.548 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:07:12.548 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:12.548 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 143382 2024-05-15T13:07:12.549 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.549 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:07:12.549 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.549 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:07:12.549 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352122+0000", 2024-05-15T13:07:12.549 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.549 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.549 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.549 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:07:12.549 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352122+0000", 2024-05-15T13:07:12.549 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.549 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.549 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.549 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:07:12.549 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352122+0000", 2024-05-15T13:07:12.549 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.2899999999999999e-07 2024-05-15T13:07:12.550 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.550 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.550 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:07:12.550 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352125+0000", 2024-05-15T13:07:12.550 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.402e-06 2024-05-15T13:07:12.550 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.550 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.550 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:07:12.550 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352135+0000", 2024-05-15T13:07:12.550 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0107e-05 2024-05-15T13:07:12.550 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.550 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.550 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:07:12.551 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352141+0000", 2024-05-15T13:07:12.551 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1210000000000003e-06 2024-05-15T13:07:12.551 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.551 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.551 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.551 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352632+0000", 2024-05-15T13:07:12.551 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00049144599999999998 2024-05-15T13:07:12.551 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.551 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.551 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:07:12.551 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352678+0000", 2024-05-15T13:07:12.551 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.5819999999999998e-05 2024-05-15T13:07:12.551 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.551 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.552 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.552 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:19.725861+0000", 2024-05-15T13:07:12.552 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.373183085 2024-05-15T13:07:12.552 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.552 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.552 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:07:12.552 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:19.725872+0000", 2024-05-15T13:07:12.552 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0841e-05 2024-05-15T13:07:12.552 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.552 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:07:12.552 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.552 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.552 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.552 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:143384 4.8 4:15b4278a:::benchmark_data_smithi177_99756_object23412:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e357)", 2024-05-15T13:07:12.553 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:00:15.352148+0000", 2024-05-15T13:07:12.553 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 417.16428421099999, 2024-05-15T13:07:12.553 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3796533020000004, 2024-05-15T13:07:12.553 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:07:12.553 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:07:12.553 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:07:12.553 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:07:12.553 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:12.553 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 143384 2024-05-15T13:07:12.553 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.553 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:07:12.553 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.553 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:07:12.553 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352148+0000", 2024-05-15T13:07:12.553 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.554 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.554 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.554 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:07:12.554 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352148+0000", 2024-05-15T13:07:12.554 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.554 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.554 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.554 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:07:12.554 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352148+0000", 2024-05-15T13:07:12.554 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.9299999999999999e-07 2024-05-15T13:07:12.554 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.554 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.554 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:07:12.554 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352154+0000", 2024-05-15T13:07:12.555 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5600000000000001e-06 2024-05-15T13:07:12.555 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.555 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.555 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:07:12.555 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352352+0000", 2024-05-15T13:07:12.555 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00019874000000000001 2024-05-15T13:07:12.555 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.555 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.555 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:07:12.555 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352358+0000", 2024-05-15T13:07:12.555 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8209999999999998e-06 2024-05-15T13:07:12.555 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.555 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.555 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.555 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352700+0000", 2024-05-15T13:07:12.555 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00034227599999999997 2024-05-15T13:07:12.556 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.556 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.556 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:07:12.556 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:15.352749+0000", 2024-05-15T13:07:12.556 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8597000000000002e-05 2024-05-15T13:07:12.556 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.556 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.556 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.556 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:19.731793+0000", 2024-05-15T13:07:12.556 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3790435270000003 2024-05-15T13:07:12.556 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.556 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.556 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:07:12.556 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:19.731801+0000", 2024-05-15T13:07:12.557 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.388e-06 2024-05-15T13:07:12.557 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.557 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:07:12.557 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.557 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.557 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.557 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:166378 4.8 4:1446be75:::benchmark_data_smithi177_99756_object46406:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e369)", 2024-05-15T13:07:12.557 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:00:52.776699+0000", 2024-05-15T13:07:12.557 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 379.73973269099997, 2024-05-15T13:07:12.557 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2407082950000001, 2024-05-15T13:07:12.557 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:07:12.557 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:07:12.557 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:07:12.557 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:07:12.557 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:12.558 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 166378 2024-05-15T13:07:12.558 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.558 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:07:12.558 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.558 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:07:12.558 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:52.776699+0000", 2024-05-15T13:07:12.558 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.558 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.558 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.558 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:07:12.558 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:52.776699+0000", 2024-05-15T13:07:12.558 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.559 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.559 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.559 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:07:12.559 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:52.776700+0000", 2024-05-15T13:07:12.559 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.4999999999999998e-07 2024-05-15T13:07:12.559 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.559 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.559 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:07:12.559 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:52.776702+0000", 2024-05-15T13:07:12.559 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1100000000000001e-06 2024-05-15T13:07:12.559 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.559 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.559 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:07:12.559 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:52.776707+0000", 2024-05-15T13:07:12.560 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7939999999999999e-06 2024-05-15T13:07:12.560 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.560 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.560 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:07:12.560 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:52.776713+0000", 2024-05-15T13:07:12.560 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1959999999999999e-06 2024-05-15T13:07:12.560 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.560 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.560 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.560 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:52.776759+0000", 2024-05-15T13:07:12.560 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6876999999999999e-05 2024-05-15T13:07:12.560 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.560 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.561 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:07:12.561 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:52.776859+0000", 2024-05-15T13:07:12.561 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.9444000000000005e-05 2024-05-15T13:07:12.561 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.561 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.561 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.561 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:55.017393+0000", 2024-05-15T13:07:12.561 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.240534416 2024-05-15T13:07:12.561 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.561 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.561 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:07:12.561 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:00:55.017407+0000", 2024-05-15T13:07:12.561 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4107999999999999e-05 2024-05-15T13:07:12.561 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.562 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:07:12.562 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.562 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.562 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.562 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:174036 4.8 4:1764670f:::benchmark_data_smithi177_99756_object54064:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-05-15T13:07:12.562 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:04.994119+0000", 2024-05-15T13:07:12.562 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 367.522312408, 2024-05-15T13:07:12.562 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.350619649, 2024-05-15T13:07:12.562 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:07:12.562 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:07:12.562 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:07:12.562 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:07:12.562 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:12.563 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 174036 2024-05-15T13:07:12.563 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.563 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:07:12.563 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.563 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:07:12.563 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:04.994119+0000", 2024-05-15T13:07:12.563 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.563 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.563 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.563 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:07:12.563 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:04.994119+0000", 2024-05-15T13:07:12.563 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.563 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.563 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.564 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:07:12.564 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:04.994121+0000", 2024-05-15T13:07:12.564 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2139999999999999e-06 2024-05-15T13:07:12.564 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.564 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.564 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:07:12.564 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:04.994125+0000", 2024-05-15T13:07:12.564 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0980000000000004e-06 2024-05-15T13:07:12.564 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.564 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.564 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:07:12.564 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:04.994136+0000", 2024-05-15T13:07:12.564 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1497e-05 2024-05-15T13:07:12.565 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.565 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.565 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:07:12.565 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:04.994143+0000", 2024-05-15T13:07:12.565 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.1999999999999997e-06 2024-05-15T13:07:12.565 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.565 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.565 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.565 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:04.994194+0000", 2024-05-15T13:07:12.565 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0419e-05 2024-05-15T13:07:12.565 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.565 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.565 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:07:12.565 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:04.994285+0000", 2024-05-15T13:07:12.566 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0651999999999999e-05 2024-05-15T13:07:12.566 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.566 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.566 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.566 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:06.344720+0000", 2024-05-15T13:07:12.566 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3504358380000001 2024-05-15T13:07:12.566 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.566 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.566 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:07:12.566 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:06.344739+0000", 2024-05-15T13:07:12.566 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.8731000000000001e-05 2024-05-15T13:07:12.566 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.566 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:07:12.567 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.567 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.567 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.567 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:174042 4.8 4:12777e54:::benchmark_data_smithi177_99756_object54070:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-05-15T13:07:12.567 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:04.994160+0000", 2024-05-15T13:07:12.567 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 367.52227151800003, 2024-05-15T13:07:12.567 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.355482965, 2024-05-15T13:07:12.567 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:07:12.567 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:07:12.567 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:07:12.567 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:07:12.567 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:12.567 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 174042 2024-05-15T13:07:12.567 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.567 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:07:12.568 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.568 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:07:12.568 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:04.994160+0000", 2024-05-15T13:07:12.568 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.568 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.568 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.568 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:07:12.568 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:04.994160+0000", 2024-05-15T13:07:12.568 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.568 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.568 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.568 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:07:12.568 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:04.994161+0000", 2024-05-15T13:07:12.568 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8100000000000003e-07 2024-05-15T13:07:12.569 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.569 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.569 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:07:12.569 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:04.994164+0000", 2024-05-15T13:07:12.569 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.5940000000000001e-06 2024-05-15T13:07:12.569 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.569 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.569 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:07:12.569 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:04.994171+0000", 2024-05-15T13:07:12.569 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.858e-06 2024-05-15T13:07:12.569 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.569 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.569 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:07:12.569 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:04.994177+0000", 2024-05-15T13:07:12.570 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.0179999999999998e-06 2024-05-15T13:07:12.570 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.570 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.570 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.570 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:04.994311+0000", 2024-05-15T13:07:12.570 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013318600000000001 2024-05-15T13:07:12.570 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.570 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.570 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:07:12.570 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:04.994368+0000", 2024-05-15T13:07:12.570 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7878999999999999e-05 2024-05-15T13:07:12.570 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.570 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.570 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.571 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:06.349635+0000", 2024-05-15T13:07:12.571 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3552661420000001 2024-05-15T13:07:12.571 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.571 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.571 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:07:12.571 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:06.349643+0000", 2024-05-15T13:07:12.571 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7069999999999998e-06 2024-05-15T13:07:12.571 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.571 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:07:12.571 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.571 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.571 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.571 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:175539 4.5 4:a0c85544:::benchmark_data_smithi177_99756_object55567:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-05-15T13:07:12.572 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:07.459595+0000", 2024-05-15T13:07:12.572 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 365.05683731099998, 2024-05-15T13:07:12.572 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.046056232, 2024-05-15T13:07:12.572 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:07:12.572 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:12.572 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:07:12.572 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:07:12.572 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:12.572 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 175539 2024-05-15T13:07:12.572 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.572 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:07:12.572 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.572 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:07:12.573 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.459595+0000", 2024-05-15T13:07:12.573 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.573 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.573 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.573 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:07:12.573 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.459595+0000", 2024-05-15T13:07:12.573 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.573 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.573 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.573 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:07:12.573 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.459596+0000", 2024-05-15T13:07:12.573 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.186e-06 2024-05-15T13:07:12.573 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.573 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.574 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:07:12.574 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.459601+0000", 2024-05-15T13:07:12.574 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8080000000000004e-06 2024-05-15T13:07:12.574 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.574 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.574 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:07:12.574 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.459612+0000", 2024-05-15T13:07:12.574 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1800999999999999e-05 2024-05-15T13:07:12.574 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.574 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.574 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:07:12.574 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.459620+0000", 2024-05-15T13:07:12.575 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9410000000000007e-06 2024-05-15T13:07:12.575 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.575 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.575 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.575 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.459692+0000", 2024-05-15T13:07:12.575 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.1662999999999995e-05 2024-05-15T13:07:12.575 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.575 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.575 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:07:12.575 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.459707+0000", 2024-05-15T13:07:12.575 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4758e-05 2024-05-15T13:07:12.575 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.575 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.575 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.576 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.493914+0000", 2024-05-15T13:07:12.576 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.034206865 2024-05-15T13:07:12.576 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.576 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.576 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:07:12.576 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.494039+0000", 2024-05-15T13:07:12.576 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000124988 2024-05-15T13:07:12.576 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.576 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.576 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 10,11", 2024-05-15T13:07:12.576 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.494128+0000", 2024-05-15T13:07:12.576 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.9240999999999995e-05 2024-05-15T13:07:12.576 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.577 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.577 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:12.577 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.498182+0000", 2024-05-15T13:07:12.577 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0040541539999999999 2024-05-15T13:07:12.577 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.577 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.577 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:07:12.577 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.501838+0000", 2024-05-15T13:07:12.577 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0036556549999999998 2024-05-15T13:07:12.577 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.577 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.577 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:12.577 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.505589+0000", 2024-05-15T13:07:12.577 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.003751786 2024-05-15T13:07:12.578 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.578 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.578 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:07:12.578 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.505624+0000", 2024-05-15T13:07:12.578 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.4122000000000003e-05 2024-05-15T13:07:12.578 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.578 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.578 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:07:12.578 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.505651+0000", 2024-05-15T13:07:12.578 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.7263999999999998e-05 2024-05-15T13:07:12.578 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.578 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:07:12.578 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.579 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.579 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.579 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:175542 4.5 4:a1ecc3a9:::benchmark_data_smithi177_99756_object55570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-05-15T13:07:12.579 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:07.470181+0000", 2024-05-15T13:07:12.579 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 365.046250886, 2024-05-15T13:07:12.579 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.042000327, 2024-05-15T13:07:12.579 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:07:12.579 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:12.579 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:07:12.579 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:07:12.579 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:12.579 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 175542 2024-05-15T13:07:12.579 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.579 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:07:12.580 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.580 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:07:12.580 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.470181+0000", 2024-05-15T13:07:12.580 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.580 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.580 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.580 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:07:12.580 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.470181+0000", 2024-05-15T13:07:12.580 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.580 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.580 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.580 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:07:12.580 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.470182+0000", 2024-05-15T13:07:12.581 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5230000000000001e-06 2024-05-15T13:07:12.581 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.581 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.581 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:07:12.581 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.470188+0000", 2024-05-15T13:07:12.581 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.0059999999999998e-06 2024-05-15T13:07:12.581 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.581 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.581 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:07:12.581 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.470200+0000", 2024-05-15T13:07:12.581 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1805e-05 2024-05-15T13:07:12.581 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.581 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.581 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:07:12.582 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.470210+0000", 2024-05-15T13:07:12.582 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.4509999999999998e-06 2024-05-15T13:07:12.582 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.582 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.582 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.582 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.470275+0000", 2024-05-15T13:07:12.582 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.5592000000000005e-05 2024-05-15T13:07:12.582 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.582 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.582 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:07:12.582 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.470285+0000", 2024-05-15T13:07:12.582 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0087e-05 2024-05-15T13:07:12.582 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.583 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.583 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.583 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.498213+0000", 2024-05-15T13:07:12.583 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.027927225 2024-05-15T13:07:12.583 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.583 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.583 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:07:12.583 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.498301+0000", 2024-05-15T13:07:12.583 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.878e-05 2024-05-15T13:07:12.583 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.583 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.583 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 10,11", 2024-05-15T13:07:12.583 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.498376+0000", 2024-05-15T13:07:12.584 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4169000000000001e-05 2024-05-15T13:07:12.584 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.584 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.584 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:12.584 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.505489+0000", 2024-05-15T13:07:12.584 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0071138649999999996 2024-05-15T13:07:12.584 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.584 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.584 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:07:12.584 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.505694+0000", 2024-05-15T13:07:12.584 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00020458499999999999 2024-05-15T13:07:12.584 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.584 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.584 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:12.585 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.512104+0000", 2024-05-15T13:07:12.585 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0064102719999999998 2024-05-15T13:07:12.585 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.585 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.585 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:07:12.585 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.512139+0000", 2024-05-15T13:07:12.585 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.4390000000000001e-05 2024-05-15T13:07:12.585 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.585 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.585 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:07:12.585 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.512181+0000", 2024-05-15T13:07:12.585 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2577000000000002e-05 2024-05-15T13:07:12.585 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.585 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:07:12.586 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.586 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.586 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.586 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:175552 4.5 4:ac77fef5:::benchmark_data_smithi177_99756_object55580:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-05-15T13:07:12.586 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:07.487069+0000", 2024-05-15T13:07:12.586 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 365.02936258699998, 2024-05-15T13:07:12.586 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.025238256, 2024-05-15T13:07:12.586 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:07:12.586 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:12.586 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:07:12.586 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:07:12.586 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:12.587 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 175552 2024-05-15T13:07:12.587 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.587 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:07:12.587 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.587 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:07:12.587 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.487069+0000", 2024-05-15T13:07:12.587 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.587 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.587 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.587 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:07:12.587 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.487069+0000", 2024-05-15T13:07:12.587 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.587 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.587 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.588 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:07:12.588 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.487071+0000", 2024-05-15T13:07:12.588 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3519999999999999e-06 2024-05-15T13:07:12.588 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.588 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.588 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:07:12.588 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.487077+0000", 2024-05-15T13:07:12.588 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1500000000000004e-06 2024-05-15T13:07:12.588 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.588 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.588 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:07:12.588 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.487091+0000", 2024-05-15T13:07:12.588 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4557e-05 2024-05-15T13:07:12.588 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.589 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.589 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:07:12.589 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.487099+0000", 2024-05-15T13:07:12.589 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1009999999999992e-06 2024-05-15T13:07:12.589 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.589 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.589 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.589 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.487179+0000", 2024-05-15T13:07:12.589 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9079999999999995e-05 2024-05-15T13:07:12.589 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.589 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.589 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:07:12.589 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.487192+0000", 2024-05-15T13:07:12.589 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3281e-05 2024-05-15T13:07:12.590 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.590 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.590 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.590 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.501883+0000", 2024-05-15T13:07:12.590 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.014690987 2024-05-15T13:07:12.590 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.590 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.590 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:07:12.590 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.501988+0000", 2024-05-15T13:07:12.590 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000105251 2024-05-15T13:07:12.590 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.590 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.591 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 10,11", 2024-05-15T13:07:12.591 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.502088+0000", 2024-05-15T13:07:12.591 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000100312 2024-05-15T13:07:12.591 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.591 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.591 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:12.591 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.505686+0000", 2024-05-15T13:07:12.591 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0035975260000000002 2024-05-15T13:07:12.591 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.591 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.591 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:12.591 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.512240+0000", 2024-05-15T13:07:12.591 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0065540319999999996 2024-05-15T13:07:12.591 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.592 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.592 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:07:12.592 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.512250+0000", 2024-05-15T13:07:12.592 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0447e-05 2024-05-15T13:07:12.592 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.592 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.592 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:07:12.592 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.512281+0000", 2024-05-15T13:07:12.592 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.0803000000000003e-05 2024-05-15T13:07:12.592 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.592 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.592 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:07:12.592 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.512308+0000", 2024-05-15T13:07:12.592 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.6376999999999998e-05 2024-05-15T13:07:12.592 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.593 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:07:12.593 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.593 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.593 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.593 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:175579 4.5 4:aa0e2654:::benchmark_data_smithi177_99756_object55607:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-05-15T13:07:12.593 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:07.527098+0000", 2024-05-15T13:07:12.593 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 364.98933346500002, 2024-05-15T13:07:12.593 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.99362199900000003, 2024-05-15T13:07:12.593 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:07:12.593 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:12.593 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:07:12.593 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:07:12.593 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:12.593 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 175579 2024-05-15T13:07:12.594 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.594 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:07:12.594 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.594 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:07:12.594 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.527098+0000", 2024-05-15T13:07:12.594 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.594 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.594 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.594 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:07:12.594 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.527098+0000", 2024-05-15T13:07:12.594 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.594 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.594 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.595 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:07:12.595 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.527099+0000", 2024-05-15T13:07:12.595 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6499999999999999e-07 2024-05-15T13:07:12.595 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.595 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.595 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:07:12.595 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.527102+0000", 2024-05-15T13:07:12.595 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.8279999999999999e-06 2024-05-15T13:07:12.595 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.595 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.595 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:07:12.595 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.527108+0000", 2024-05-15T13:07:12.595 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.669e-06 2024-05-15T13:07:12.596 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.596 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.596 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:07:12.596 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.527113+0000", 2024-05-15T13:07:12.596 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9609999999999996e-06 2024-05-15T13:07:12.596 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.596 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.596 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.596 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.527175+0000", 2024-05-15T13:07:12.596 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1599000000000004e-05 2024-05-15T13:07:12.596 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.596 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.596 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:07:12.596 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.527187+0000", 2024-05-15T13:07:12.597 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2208000000000001e-05 2024-05-15T13:07:12.597 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.597 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.597 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.597 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.505717+0000", 2024-05-15T13:07:12.597 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.97853029899999999 2024-05-15T13:07:12.597 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.597 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.597 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:07:12.597 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.505800+0000", 2024-05-15T13:07:12.597 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2455999999999998e-05 2024-05-15T13:07:12.597 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.597 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.597 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 10,11", 2024-05-15T13:07:12.598 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.505870+0000", 2024-05-15T13:07:12.598 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9973000000000002e-05 2024-05-15T13:07:12.598 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.598 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.598 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:07:12.598 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.520504+0000", 2024-05-15T13:07:12.598 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.014634232 2024-05-15T13:07:12.598 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.598 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.598 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:12.598 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.520566+0000", 2024-05-15T13:07:12.598 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.2248000000000004e-05 2024-05-15T13:07:12.598 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.599 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.599 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:12.599 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.520618+0000", 2024-05-15T13:07:12.599 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1829000000000001e-05 2024-05-15T13:07:12.599 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.599 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.599 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:07:12.599 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.520675+0000", 2024-05-15T13:07:12.599 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6563e-05 2024-05-15T13:07:12.599 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.599 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.599 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:07:12.599 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.520720+0000", 2024-05-15T13:07:12.599 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.5668999999999999e-05 2024-05-15T13:07:12.600 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.600 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:07:12.600 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.600 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.600 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.600 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:175584 4.5 4:a00c11ae:::benchmark_data_smithi177_99756_object55612:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-05-15T13:07:12.600 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:07.533835+0000", 2024-05-15T13:07:12.600 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 364.98259720800002, 2024-05-15T13:07:12.600 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.997202056, 2024-05-15T13:07:12.600 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:07:12.600 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:12.600 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:07:12.600 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:07:12.600 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:12.600 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 175584 2024-05-15T13:07:12.601 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.601 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:07:12.601 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.601 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:07:12.601 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.533835+0000", 2024-05-15T13:07:12.601 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.601 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.601 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.601 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:07:12.601 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.533835+0000", 2024-05-15T13:07:12.601 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.601 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.601 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.601 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:07:12.602 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.533836+0000", 2024-05-15T13:07:12.602 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3769999999999999e-06 2024-05-15T13:07:12.602 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.602 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.602 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:07:12.602 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.533841+0000", 2024-05-15T13:07:12.602 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.831e-06 2024-05-15T13:07:12.602 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.602 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.602 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:07:12.602 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.533852+0000", 2024-05-15T13:07:12.602 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0712e-05 2024-05-15T13:07:12.602 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.603 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.603 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:07:12.603 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.533858+0000", 2024-05-15T13:07:12.603 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9479999999999996e-06 2024-05-15T13:07:12.603 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.603 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.603 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.603 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.533920+0000", 2024-05-15T13:07:12.603 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.2586e-05 2024-05-15T13:07:12.603 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.603 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.603 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:07:12.604 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.533935+0000", 2024-05-15T13:07:12.604 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.453e-05 2024-05-15T13:07:12.604 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.604 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.604 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.604 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.509070+0000", 2024-05-15T13:07:12.604 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.97513506599999999 2024-05-15T13:07:12.604 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.604 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.604 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:07:12.604 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.509154+0000", 2024-05-15T13:07:12.604 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.4221999999999994e-05 2024-05-15T13:07:12.604 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.604 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.605 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 10,11", 2024-05-15T13:07:12.605 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.509248+0000", 2024-05-15T13:07:12.605 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.4166000000000006e-05 2024-05-15T13:07:12.605 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.605 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.605 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:07:12.605 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.520511+0000", 2024-05-15T13:07:12.605 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.011263132 2024-05-15T13:07:12.605 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.605 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.605 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:12.605 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.520785+0000", 2024-05-15T13:07:12.605 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000273756 2024-05-15T13:07:12.605 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.606 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.606 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:12.606 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.530938+0000", 2024-05-15T13:07:12.606 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.010153519999999999 2024-05-15T13:07:12.606 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.606 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.606 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:07:12.606 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.530989+0000", 2024-05-15T13:07:12.606 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0720999999999998e-05 2024-05-15T13:07:12.606 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.606 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.606 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:07:12.606 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.531037+0000", 2024-05-15T13:07:12.607 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7488999999999999e-05 2024-05-15T13:07:12.607 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.607 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:07:12.607 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.607 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.607 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.607 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:175607 4.5 4:a9ddec70:::benchmark_data_smithi177_99756_object55635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-05-15T13:07:12.607 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:01:07.592799+0000", 2024-05-15T13:07:12.607 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 364.923632774, 2024-05-15T13:07:12.607 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.93841144399999998, 2024-05-15T13:07:12.607 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:07:12.607 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:12.607 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:07:12.608 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:07:12.608 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:12.608 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 175607 2024-05-15T13:07:12.608 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.608 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:07:12.608 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.608 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:07:12.608 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.592799+0000", 2024-05-15T13:07:12.608 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.608 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.608 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.609 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:07:12.609 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.592799+0000", 2024-05-15T13:07:12.609 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.609 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.609 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.609 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:07:12.609 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.592801+0000", 2024-05-15T13:07:12.609 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5549999999999999e-06 2024-05-15T13:07:12.609 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.609 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.609 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:07:12.609 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.592807+0000", 2024-05-15T13:07:12.610 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6830000000000002e-06 2024-05-15T13:07:12.610 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.610 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.610 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:07:12.610 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.592819+0000", 2024-05-15T13:07:12.610 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2026e-05 2024-05-15T13:07:12.610 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.610 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.610 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:07:12.610 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.592828+0000", 2024-05-15T13:07:12.610 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7150000000000004e-06 2024-05-15T13:07:12.610 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.610 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.611 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.611 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.592889+0000", 2024-05-15T13:07:12.611 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1131999999999997e-05 2024-05-15T13:07:12.611 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.611 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.611 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:07:12.611 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:07.592899+0000", 2024-05-15T13:07:12.611 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0172e-05 2024-05-15T13:07:12.611 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.611 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.611 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.611 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.512329+0000", 2024-05-15T13:07:12.611 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.91942933699999996 2024-05-15T13:07:12.611 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.611 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.612 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:07:12.612 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.512396+0000", 2024-05-15T13:07:12.612 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7292000000000006e-05 2024-05-15T13:07:12.612 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.612 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.612 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 10,11", 2024-05-15T13:07:12.612 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.512476+0000", 2024-05-15T13:07:12.612 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0253999999999994e-05 2024-05-15T13:07:12.612 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.612 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.612 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:07:12.612 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.520745+0000", 2024-05-15T13:07:12.612 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0082687030000000005 2024-05-15T13:07:12.613 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.613 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.613 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:12.613 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.531089+0000", 2024-05-15T13:07:12.613 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.01034382 2024-05-15T13:07:12.613 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.613 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.613 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:12.613 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.531127+0000", 2024-05-15T13:07:12.613 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7839999999999997e-05 2024-05-15T13:07:12.613 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.613 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.613 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:07:12.613 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.531174+0000", 2024-05-15T13:07:12.613 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7497000000000003e-05 2024-05-15T13:07:12.614 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.614 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.614 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:07:12.614 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:01:08.531211+0000", 2024-05-15T13:07:12.614 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.6418000000000003e-05 2024-05-15T13:07:12.614 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.614 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:07:12.614 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.614 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.614 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.614 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:211766 4.8 4:13b32146:::benchmark_data_smithi177_99756_object91794:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e391)", 2024-05-15T13:07:12.614 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:02:03.810591+0000", 2024-05-15T13:07:12.614 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 308.70584065899999, 2024-05-15T13:07:12.615 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7269911609999999, 2024-05-15T13:07:12.615 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:07:12.615 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:07:12.615 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:07:12.615 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:07:12.615 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:12.615 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 211766 2024-05-15T13:07:12.615 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.615 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:07:12.615 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.615 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:07:12.615 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:03.810591+0000", 2024-05-15T13:07:12.615 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.615 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.616 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.616 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:07:12.616 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:03.810591+0000", 2024-05-15T13:07:12.616 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.616 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.616 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.616 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:07:12.616 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:03.810592+0000", 2024-05-15T13:07:12.616 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5400000000000002e-07 2024-05-15T13:07:12.616 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.616 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.616 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:07:12.616 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:03.810594+0000", 2024-05-15T13:07:12.616 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.396e-06 2024-05-15T13:07:12.617 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.617 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.617 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:07:12.617 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:03.810600+0000", 2024-05-15T13:07:12.617 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4420000000000003e-06 2024-05-15T13:07:12.617 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.617 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.617 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:07:12.617 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:03.810605+0000", 2024-05-15T13:07:12.617 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8969999999999999e-06 2024-05-15T13:07:12.617 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.617 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.617 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.617 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:03.812501+0000", 2024-05-15T13:07:12.618 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0018954169999999999 2024-05-15T13:07:12.618 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.618 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.618 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:07:12.618 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:03.812581+0000", 2024-05-15T13:07:12.618 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9919999999999994e-05 2024-05-15T13:07:12.618 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.618 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.618 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.618 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:07.537558+0000", 2024-05-15T13:07:12.618 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7249777659999999 2024-05-15T13:07:12.618 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.619 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.619 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:07:12.619 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:07.537582+0000", 2024-05-15T13:07:12.619 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.3969e-05 2024-05-15T13:07:12.619 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.619 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:07:12.619 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.619 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.619 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.619 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:216547 4.8 4:16e1c84e:::benchmark_data_smithi177_99756_object96575:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e392)", 2024-05-15T13:07:12.619 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:02:10.474631+0000", 2024-05-15T13:07:12.619 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 302.04180083199998, 2024-05-15T13:07:12.619 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.107790772, 2024-05-15T13:07:12.619 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:07:12.620 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:07:12.620 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:07:12.620 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:07:12.620 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:12.620 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 216547 2024-05-15T13:07:12.620 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.620 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:07:12.620 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.620 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:07:12.620 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:10.474631+0000", 2024-05-15T13:07:12.620 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.620 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.620 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.621 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:07:12.621 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:10.474631+0000", 2024-05-15T13:07:12.621 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.621 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.621 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.621 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:07:12.621 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:10.474631+0000", 2024-05-15T13:07:12.621 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.3799999999999998e-07 2024-05-15T13:07:12.621 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.621 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.621 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:07:12.621 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:10.474633+0000", 2024-05-15T13:07:12.621 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3480000000000001e-06 2024-05-15T13:07:12.621 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.622 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.622 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:07:12.622 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:10.474636+0000", 2024-05-15T13:07:12.622 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.2320000000000001e-06 2024-05-15T13:07:12.622 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.622 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.622 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:07:12.622 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:10.474639+0000", 2024-05-15T13:07:12.622 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.27e-06 2024-05-15T13:07:12.622 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.622 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.622 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.622 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:10.475046+0000", 2024-05-15T13:07:12.623 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00040690499999999999 2024-05-15T13:07:12.623 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.623 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.623 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:07:12.623 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:10.475097+0000", 2024-05-15T13:07:12.623 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0458999999999999e-05 2024-05-15T13:07:12.623 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.623 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.623 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.623 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:11.582410+0000", 2024-05-15T13:07:12.623 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1073134250000001 2024-05-15T13:07:12.623 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.623 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.623 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:07:12.624 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:11.582422+0000", 2024-05-15T13:07:12.624 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1795e-05 2024-05-15T13:07:12.624 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.624 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:07:12.624 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.624 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.624 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.624 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:217740 4.8 4:1e14c616:::benchmark_data_smithi177_99756_object97768:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e393)", 2024-05-15T13:07:12.624 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:02:12.257738+0000", 2024-05-15T13:07:12.624 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 300.25869356499999, 2024-05-15T13:07:12.624 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.166419063, 2024-05-15T13:07:12.624 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:07:12.624 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:07:12.624 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:07:12.624 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:07:12.625 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:12.625 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 217740 2024-05-15T13:07:12.625 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.625 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:07:12.625 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.625 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:07:12.625 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:12.257738+0000", 2024-05-15T13:07:12.625 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.625 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.625 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.625 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:07:12.625 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:12.257738+0000", 2024-05-15T13:07:12.625 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:07:12.626 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.626 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.626 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:07:12.626 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:12.257739+0000", 2024-05-15T13:07:12.626 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.4399999999999999e-07 2024-05-15T13:07:12.626 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.626 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.626 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:07:12.626 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:12.257743+0000", 2024-05-15T13:07:12.626 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2080000000000002e-06 2024-05-15T13:07:12.626 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.626 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.626 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:07:12.626 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:12.257755+0000", 2024-05-15T13:07:12.627 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1493e-05 2024-05-15T13:07:12.627 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.627 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.627 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:07:12.627 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:12.257762+0000", 2024-05-15T13:07:12.627 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4379999999999998e-06 2024-05-15T13:07:12.627 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.627 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.627 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.627 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:12.259588+0000", 2024-05-15T13:07:12.627 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001826184 2024-05-15T13:07:12.627 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.627 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.627 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:07:12.628 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:12.259687+0000", 2024-05-15T13:07:12.628 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.8612999999999999e-05 2024-05-15T13:07:12.628 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.628 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.628 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:07:12.628 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:13.424141+0000", 2024-05-15T13:07:12.628 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.164454423 2024-05-15T13:07:12.628 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:07:12.628 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:07:12.628 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:07:12.628 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:13.424157+0000", 2024-05-15T13:07:12.628 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5860000000000001e-05 2024-05-15T13:07:12.628 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.628 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:07:12.629 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.629 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:07:12.629 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:07:12.629 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:07:12.686 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:12.712 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:07:12.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:12.719 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:07:12.820 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:07:12.864 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:12.973 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:07:12.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:12.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:07:13.080 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:07:13.126 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:13.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:13 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:07:13.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:13 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:07:13.317 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:07:13.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:13.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:07:13.355 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-05-15T13:07:13.425 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:07:13.450 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:13 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:07:13.450 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:13 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:07:13.450 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:13 smithi120 ceph-mon[43961]: pgmap v1321: 27 pgs: 27 active+clean; 807 MiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 827 op/s 2024-05-15T13:07:13.450 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:13 smithi120 ceph-mon[43961]: osdmap e496: 12 total, 12 up, 11 in 2024-05-15T13:07:13.468 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:13 smithi177 ceph-mon[44303]: pgmap v1321: 27 pgs: 27 active+clean; 807 MiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 827 op/s 2024-05-15T13:07:13.469 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:13 smithi177 ceph-mon[44303]: osdmap e496: 12 total, 12 up, 11 in 2024-05-15T13:07:13.470 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:13.501 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:07:13.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:13.506 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:07:13.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:13 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:07:13.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:13 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:07:13.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:13 smithi027 ceph-mon[39795]: pgmap v1321: 27 pgs: 27 active+clean; 807 MiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 827 op/s 2024-05-15T13:07:13.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:13 smithi027 ceph-mon[39795]: osdmap e496: 12 total, 12 up, 11 in 2024-05-15T13:07:13.607 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:07:13.647 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:13.791 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:07:14.387 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:14 smithi120 ceph-mon[43961]: osdmap e497: 12 total, 12 up, 11 in 2024-05-15T13:07:14.387 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:14 smithi120 ceph-mon[43961]: pgmap v1324: 27 pgs: 27 active+clean; 662 MiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 923 op/s 2024-05-15T13:07:14.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:14 smithi027 ceph-mon[39795]: osdmap e497: 12 total, 12 up, 11 in 2024-05-15T13:07:14.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:14 smithi027 ceph-mon[39795]: pgmap v1324: 27 pgs: 27 active+clean; 662 MiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 923 op/s 2024-05-15T13:07:14.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:14 smithi177 ceph-mon[44303]: osdmap e497: 12 total, 12 up, 11 in 2024-05-15T13:07:14.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:14 smithi177 ceph-mon[44303]: pgmap v1324: 27 pgs: 27 active+clean; 662 MiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 923 op/s 2024-05-15T13:07:14.761 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:07:14.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:14.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:07:14.889 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:07:14.926 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:14.961 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:07:14.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:14.972 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:07:15.073 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:07:15.118 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:15.157 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:07:15.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:15.165 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:07:15.266 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:07:15.307 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:15.341 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:07:15.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:15.346 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:07:15.447 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:07:15.496 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:16.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:15 smithi120 ceph-mon[43961]: 5.2 scrub starts 2024-05-15T13:07:16.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:15 smithi120 ceph-mon[43961]: 5.2 scrub ok 2024-05-15T13:07:16.106 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:07:16.106 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T13:07:16.106 INFO:teuthology.orchestra.run.smithi120.stdout: "num_ops": 0 2024-05-15T13:07:16.106 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:07:16.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:15 smithi177 ceph-mon[44303]: 5.2 scrub starts 2024-05-15T13:07:16.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:15 smithi177 ceph-mon[44303]: 5.2 scrub ok 2024-05-15T13:07:16.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:15 smithi027 ceph-mon[39795]: 5.2 scrub starts 2024-05-15T13:07:16.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:15 smithi027 ceph-mon[39795]: 5.2 scrub ok 2024-05-15T13:07:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:07:16.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:16.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:07:16.484 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:07:16.587 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:07:16.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:16.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:07:16.721 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:07:16.808 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:07:16.845 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:16.853 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.3 to 1 (10000) 2024-05-15T13:07:16.863 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:07:16.886 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-05-15T13:07:17.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:16 smithi027 ceph-mon[39795]: 5.6 scrub starts 2024-05-15T13:07:17.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:16 smithi027 ceph-mon[39795]: 5.6 scrub ok 2024-05-15T13:07:17.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:16 smithi027 ceph-mon[39795]: pgmap v1325: 27 pgs: 27 active+clean; 572 MiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 809 op/s 2024-05-15T13:07:17.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:16 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:07:17.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:16 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2228432920' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:07:17.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:16 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:07:17.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:16 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1006628563' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:07:17.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:16 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:07:17.051 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:07:17.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:17.055 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:07:17.157 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:07:17.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:16 smithi177 ceph-mon[44303]: 5.6 scrub starts 2024-05-15T13:07:17.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:16 smithi177 ceph-mon[44303]: 5.6 scrub ok 2024-05-15T13:07:17.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:16 smithi177 ceph-mon[44303]: pgmap v1325: 27 pgs: 27 active+clean; 572 MiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 809 op/s 2024-05-15T13:07:17.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:16 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:07:17.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:16 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2228432920' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:07:17.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:16 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:07:17.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:16 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1006628563' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:07:17.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:16 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:07:17.189 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:17.240 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:07:17.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:16 smithi120 ceph-mon[43961]: 5.6 scrub starts 2024-05-15T13:07:17.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:16 smithi120 ceph-mon[43961]: 5.6 scrub ok 2024-05-15T13:07:17.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:16 smithi120 ceph-mon[43961]: pgmap v1325: 27 pgs: 27 active+clean; 572 MiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 809 op/s 2024-05-15T13:07:17.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:16 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:07:17.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:16 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2228432920' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:07:17.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:16 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:07:17.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:16 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1006628563' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:07:17.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:16 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:07:17.360 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:07:17.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:17.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:07:17.466 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:07:17.530 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:17.583 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:07:17.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:17.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:07:17.691 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:07:17.750 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:17.795 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:07:17.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:17.801 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:07:17.902 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:07:17.942 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:18.153 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:07:18.159 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:17 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:07:18.159 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:17 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:07:18.159 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:17 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T13:07:18.159 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:17 smithi120 ceph-mon[43961]: osdmap e498: 12 total, 12 up, 11 in 2024-05-15T13:07:18.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:18.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:07:18.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:17 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:07:18.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:17 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:07:18.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:17 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T13:07:18.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:17 smithi177 ceph-mon[44303]: osdmap e498: 12 total, 12 up, 11 in 2024-05-15T13:07:18.261 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:07:18.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:17 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:07:18.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:17 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:07:18.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:17 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T13:07:18.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:17 smithi027 ceph-mon[39795]: osdmap e498: 12 total, 12 up, 11 in 2024-05-15T13:07:18.306 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:18.457 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:07:18.665 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:07:18.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:18.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:07:18.769 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:07:18.795 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:18.818 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:07:18.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:18.823 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:07:18.924 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:07:18.974 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:19.000 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:07:19.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:19.004 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:07:19.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:07:19.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:07:19.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:18 smithi027 ceph-mon[39795]: pgmap v1327: 27 pgs: 27 active+clean; 454 MiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 942 op/s 2024-05-15T13:07:19.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:18 smithi027 ceph-mon[39795]: osdmap e499: 12 total, 12 up, 11 in 2024-05-15T13:07:19.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:07:19.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:07:19.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:07:19.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:07:19.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:07:19.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:07:19.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:07:19.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:07:19.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:07:19.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:07:19.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:07:19.105 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:07:19.146 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:19.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:07:19.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:07:19.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:18 smithi177 ceph-mon[44303]: pgmap v1327: 27 pgs: 27 active+clean; 454 MiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 942 op/s 2024-05-15T13:07:19.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:18 smithi177 ceph-mon[44303]: osdmap e499: 12 total, 12 up, 11 in 2024-05-15T13:07:19.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:07:19.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:07:19.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:07:19.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:07:19.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:07:19.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:07:19.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:07:19.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:07:19.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:07:19.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:07:19.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:07:19.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:07:19.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:07:19.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:18 smithi120 ceph-mon[43961]: pgmap v1327: 27 pgs: 27 active+clean; 454 MiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 942 op/s 2024-05-15T13:07:19.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:18 smithi120 ceph-mon[43961]: osdmap e499: 12 total, 12 up, 11 in 2024-05-15T13:07:19.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:07:19.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:07:19.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:07:19.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:07:19.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:07:19.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:07:19.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:07:19.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:07:19.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:07:19.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:07:19.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:07:19.350 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:07:19.351 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T13:07:19.351 INFO:teuthology.orchestra.run.smithi120.stdout: "complaint_time": 30, 2024-05-15T13:07:19.351 INFO:teuthology.orchestra.run.smithi120.stdout: "num_blocked_ops": 0 2024-05-15T13:07:19.351 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:07:19.418 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:07:19.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:19.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:07:19.429 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 4 1 2024-05-15T13:07:19.528 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:07:19.575 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:19.612 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:07:19.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:19.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:07:19.718 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:07:19.780 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:20.131 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-05-15T13:07:20.240 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:07:20.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:20.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:07:20.354 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:07:20.398 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:20.425 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:07:20.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:20.430 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:07:20.495 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:07:20.524 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:07:20.530 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:07:20.568 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:20.600 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:07:20.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:20.603 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:07:20.705 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:07:20.752 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:20.783 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:07:20.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:20.787 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:07:20.888 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:07:20.944 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:21.014 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:07:21.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:21.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:07:21.037 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:07:21.060 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:20 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:07:21.060 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:20 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:07:21.061 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:20 smithi120 ceph-mon[43961]: osdmap e500: 12 total, 12 up, 11 in 2024-05-15T13:07:21.061 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:20 smithi120 ceph-mon[43961]: pgmap v1330: 27 pgs: 27 active+clean; 322 MiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 905 op/s 2024-05-15T13:07:21.095 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:07:21.125 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:07:21.158 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:20 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:07:21.158 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:20 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:07:21.158 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:20 smithi177 ceph-mon[44303]: osdmap e500: 12 total, 12 up, 11 in 2024-05-15T13:07:21.158 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:20 smithi177 ceph-mon[44303]: pgmap v1330: 27 pgs: 27 active+clean; 322 MiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 905 op/s 2024-05-15T13:07:21.160 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:21.195 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:07:21.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:21.198 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:07:21.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:20 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:07:21.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:20 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:07:21.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:20 smithi027 ceph-mon[39795]: osdmap e500: 12 total, 12 up, 11 in 2024-05-15T13:07:21.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:20 smithi027 ceph-mon[39795]: pgmap v1330: 27 pgs: 27 active+clean; 322 MiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 905 op/s 2024-05-15T13:07:21.299 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:07:21.337 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:21.369 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:07:21.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:21.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:07:21.474 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:07:21.512 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:21.546 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:07:21.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:21.549 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:07:21.558 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:07:21.594 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:07:21.627 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:07:21.650 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:07:21.665 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:07:21.702 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:07:21.712 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:21.742 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:07:21.773 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:07:21.816 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:07:21.863 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:07:21.893 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:07:21.930 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:07:21.969 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:07:21.970 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:07:21.970 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:07:21.971 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:07:21.971 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:07:21.971 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:07:21.971 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:07:21.971 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:07:21.971 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:07:21.971 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:07:21.971 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:07:21.971 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:07:21.971 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:07:22.206 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:21 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4145296173' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:07:22.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:21 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4145296173' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:07:22.312 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:07:22.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:22.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:07:22.421 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:07:22.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:21 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4145296173' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:07:22.467 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:22.510 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:07:22.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:22.514 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:07:22.601 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:07:22.602 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:07:22.615 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:07:22.658 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:22.686 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:07:22.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:22.690 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:07:22.791 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:07:22.843 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:22.876 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:07:22.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:22.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:07:22.983 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:07:23.034 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:23.263 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:23 smithi027 ceph-mon[39795]: pgmap v1331: 27 pgs: 27 active+clean; 257 MiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 839 op/s 2024-05-15T13:07:23.263 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:23 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:07:23.263 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:23 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:07:23.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:23 smithi120 ceph-mon[43961]: pgmap v1331: 27 pgs: 27 active+clean; 257 MiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 839 op/s 2024-05-15T13:07:23.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:23 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:07:23.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:23 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:07:23.401 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:07:23.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:23.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:07:23.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:23 smithi177 ceph-mon[44303]: pgmap v1331: 27 pgs: 27 active+clean; 257 MiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 839 op/s 2024-05-15T13:07:23.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:23 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:07:23.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:23 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:07:23.440 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-05-15T13:07:23.509 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:07:23.553 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:23.660 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:07:23.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:23.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:07:23.764 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:07:23.818 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:23.864 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:07:23.866 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:07:23.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:23.876 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:07:23.977 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:07:24.013 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:24.103 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Removed 135188 objects 2024-05-15T13:07:24.104 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Clean up completed and total clean up time :153.115 2024-05-15T13:07:24.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:24 smithi177 ceph-mon[44303]: osdmap e501: 12 total, 12 up, 11 in 2024-05-15T13:07:24.168 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:07:24.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:24.171 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:07:24.272 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:07:24.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:24 smithi120 ceph-mon[43961]: osdmap e501: 12 total, 12 up, 11 in 2024-05-15T13:07:24.298 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:24.317 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:07:24.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:24.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:07:24.421 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:07:24.447 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:24.457 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:24 smithi027 ceph-mon[39795]: osdmap e501: 12 total, 12 up, 11 in 2024-05-15T13:07:24.465 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:07:24.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:24.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:07:24.569 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:07:24.599 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:24.849 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:07:24.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:24.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:07:24.953 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:07:24.982 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:25.072 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-05-15T13:07:25.072 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-05-15T13:07:25.099 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:25 smithi177 ceph-mon[44303]: pgmap v1333: 27 pgs: 27 active+clean; 124 MiB data, 5.2 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 880 op/s 2024-05-15T13:07:25.103 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:25 smithi027 ceph-mon[39795]: pgmap v1333: 27 pgs: 27 active+clean; 124 MiB data, 5.2 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 880 op/s 2024-05-15T13:07:25.105 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:07:25.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:25.108 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:07:25.209 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:07:25.241 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:25.263 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:07:25.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:25.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:07:25.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:25 smithi120 ceph-mon[43961]: pgmap v1333: 27 pgs: 27 active+clean; 124 MiB data, 5.2 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 880 op/s 2024-05-15T13:07:25.300 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:07:25.366 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:07:25.397 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:25.474 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:07:25.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:25.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:07:25.578 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:07:25.611 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:25.632 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:07:25.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:25.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:07:25.652 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:07:25.652 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:07:25.736 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:07:25.766 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:26.214 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:07:26.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:26.218 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:07:26.239 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-05-15T13:07:26.319 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:07:26.351 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:26.372 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:07:26.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:26.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:07:26.476 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:07:26.505 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:26.509 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:07:26.511 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:07:26.526 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:07:26.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:26.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:07:26.538 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:07:26.630 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:07:26.658 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:26.757 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:07:26.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:26.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:07:26.862 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:07:26.894 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:26.915 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:07:26.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:26.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:07:27.020 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:07:27.037 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:07:27.039 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.4 to 1 (10000) 2024-05-15T13:07:27.059 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:27.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:27 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:07:27.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:27 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:07:27.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:27 smithi027 ceph-mon[39795]: osdmap e502: 12 total, 12 up, 11 in 2024-05-15T13:07:27.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:27 smithi027 ceph-mon[39795]: pgmap v1335: 27 pgs: 27 active+clean; 52 MiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 751 op/s 2024-05-15T13:07:27.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:27 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2902343051' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:07:27.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:27 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2122685295' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T13:07:27.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:27 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:07:27.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:27 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:07:27.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:27 smithi120 ceph-mon[43961]: osdmap e502: 12 total, 12 up, 11 in 2024-05-15T13:07:27.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:27 smithi120 ceph-mon[43961]: pgmap v1335: 27 pgs: 27 active+clean; 52 MiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 751 op/s 2024-05-15T13:07:27.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:27 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2902343051' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:07:27.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:27 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2122685295' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T13:07:27.391 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:27 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:07:27.391 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:27 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:07:27.391 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:27 smithi177 ceph-mon[44303]: osdmap e502: 12 total, 12 up, 11 in 2024-05-15T13:07:27.391 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:27 smithi177 ceph-mon[44303]: pgmap v1335: 27 pgs: 27 active+clean; 52 MiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 751 op/s 2024-05-15T13:07:27.391 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:27 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2902343051' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:07:27.391 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:27 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2122685295' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T13:07:28.307 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:07:28.307 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:07:28.329 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:28 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:07:28.330 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:28 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2902343051' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:07:28.330 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:28 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2122685295' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T13:07:28.330 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:28 smithi177 ceph-mon[44303]: osdmap e503: 12 total, 12 up, 11 in 2024-05-15T13:07:28.382 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:28 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:07:28.382 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:28 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2902343051' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:07:28.382 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:28 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2122685295' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T13:07:28.382 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:28 smithi027 ceph-mon[39795]: osdmap e503: 12 total, 12 up, 11 in 2024-05-15T13:07:28.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:28 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:07:28.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:28 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2902343051' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:07:28.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:28 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2122685295' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T13:07:28.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:28 smithi120 ceph-mon[43961]: osdmap e503: 12 total, 12 up, 11 in 2024-05-15T13:07:28.972 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-05-15T13:07:29.233 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:07:29.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:29 smithi027 ceph-mon[39795]: pgmap v1337: 27 pgs: 27 active+clean; 9.2 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 661 op/s 2024-05-15T13:07:29.314 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:07:29.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:29.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:07:29.419 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:07:29.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:29 smithi177 ceph-mon[44303]: pgmap v1337: 27 pgs: 27 active+clean; 9.2 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 661 op/s 2024-05-15T13:07:29.452 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:29.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:29 smithi120 ceph-mon[43961]: pgmap v1337: 27 pgs: 27 active+clean; 9.2 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 661 op/s 2024-05-15T13:07:30.329 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:30 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/536700432' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:07:30.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:30 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/536700432' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:07:30.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:30 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/536700432' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:07:30.762 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 5 1 2024-05-15T13:07:30.873 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:07:30.873 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:07:30.873 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:07:30.873 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:07:30.873 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:07:30.873 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:07:30.873 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:07:30.873 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:07:30.873 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:07:30.873 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:07:30.873 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:07:30.873 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:07:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:07:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:07:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:07:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:07:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:07:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:07:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:07:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:07:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:07:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:07:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:07:30.874 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:07:30.899 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:07:31.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:31 smithi177 ceph-mon[44303]: pgmap v1338: 27 pgs: 27 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 431 op/s 2024-05-15T13:07:31.457 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:31 smithi027 ceph-mon[39795]: pgmap v1338: 27 pgs: 27 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 431 op/s 2024-05-15T13:07:31.497 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:07:31.504 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:07:31.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:31.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:07:31.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:31 smithi120 ceph-mon[43961]: pgmap v1338: 27 pgs: 27 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 431 op/s 2024-05-15T13:07:31.608 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:07:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:32.488 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:07:32.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:32.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:07:32.592 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:07:32.623 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:32.974 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:07:32.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:32.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:07:32.978 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:07:33.079 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:07:33.107 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:33.157 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_3' removed 2024-05-15T13:07:33.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:33 smithi027 ceph-mon[39795]: pgmap v1339: 27 pgs: 27 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 245 op/s 2024-05-15T13:07:33.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:33 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3297018071' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T13:07:33.347 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:33 smithi177 ceph-mon[44303]: pgmap v1339: 27 pgs: 27 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 245 op/s 2024-05-15T13:07:33.347 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:33 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3297018071' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T13:07:33.490 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:07:33.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:33 smithi120 ceph-mon[43961]: pgmap v1339: 27 pgs: 27 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 245 op/s 2024-05-15T13:07:33.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:33 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3297018071' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T13:07:33.666 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:07:33.666 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-05-15T13:07:33.667 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-05-15T13:07:33.667 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T13:07:33.667 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.667 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25180.0:146849 4.0 e359/317 4:003e45a0:::benchmark_data_smithi177_99756_object26877:head v 359'9275, mlcod=359'9272)", 2024-05-15T13:07:33.667 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:00:21.451275+0000", 2024-05-15T13:07:33.667 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 432.22371748, 2024-05-15T13:07:33.668 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.026767085999999999, 2024-05-15T13:07:33.668 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:33.668 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:33.668 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:33.668 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.668 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:33.668 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:21.451275+0000", 2024-05-15T13:07:33.669 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.669 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.669 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.669 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:33.669 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:21.451275+0000", 2024-05-15T13:07:33.669 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.670 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.670 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.670 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:33.670 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:21.451277+0000", 2024-05-15T13:07:33.670 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6670000000000001e-06 2024-05-15T13:07:33.670 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.671 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.671 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:33.671 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:21.451286+0000", 2024-05-15T13:07:33.671 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1279999999999997e-06 2024-05-15T13:07:33.671 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.671 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.672 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:33.672 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:21.451308+0000", 2024-05-15T13:07:33.672 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1903999999999999e-05 2024-05-15T13:07:33.672 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.672 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.672 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:33.673 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:21.451318+0000", 2024-05-15T13:07:33.673 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5149999999999995e-06 2024-05-15T13:07:33.673 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.673 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.673 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:33.673 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:21.454627+0000", 2024-05-15T13:07:33.674 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00330942 2024-05-15T13:07:33.674 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.674 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.674 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:33.674 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:21.454653+0000", 2024-05-15T13:07:33.674 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6097000000000001e-05 2024-05-15T13:07:33.675 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.675 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.675 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:33.675 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:21.477989+0000", 2024-05-15T13:07:33.675 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.023336242 2024-05-15T13:07:33.675 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.676 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.676 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:33.676 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:00:21.478042+0000", 2024-05-15T13:07:33.676 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3112999999999998e-05 2024-05-15T13:07:33.676 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.676 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:33.677 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.677 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.677 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.677 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25180.0:219299 4.0 e395/317 4:0670f711:::benchmark_data_smithi177_99756_object99327:head v 395'13849, mlcod=395'13845)", 2024-05-15T13:07:33.677 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:02:14.469673+0000", 2024-05-15T13:07:33.678 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 319.20532021999998, 2024-05-15T13:07:33.678 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028436128000000001, 2024-05-15T13:07:33.678 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:33.678 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:33.678 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:33.679 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.679 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:33.679 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:14.469673+0000", 2024-05-15T13:07:33.679 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.679 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.679 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.680 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:33.680 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:14.469673+0000", 2024-05-15T13:07:33.680 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.680 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.680 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.680 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:33.681 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:14.469673+0000", 2024-05-15T13:07:33.681 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9999999999999998e-07 2024-05-15T13:07:33.681 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.681 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.681 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:33.682 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:14.469679+0000", 2024-05-15T13:07:33.682 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0449999999999997e-06 2024-05-15T13:07:33.682 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.682 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.682 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:33.683 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:14.469689+0000", 2024-05-15T13:07:33.683 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4609999999999997e-06 2024-05-15T13:07:33.683 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.683 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.683 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:33.683 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:14.469696+0000", 2024-05-15T13:07:33.684 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.25e-06 2024-05-15T13:07:33.684 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.684 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.684 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:33.684 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:14.473575+0000", 2024-05-15T13:07:33.684 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0038787919999999998 2024-05-15T13:07:33.684 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.685 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.685 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:33.685 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:14.473603+0000", 2024-05-15T13:07:33.685 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8027000000000001e-05 2024-05-15T13:07:33.685 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.685 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.686 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:33.686 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:14.498064+0000", 2024-05-15T13:07:33.686 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.024461846999999998 2024-05-15T13:07:33.686 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.686 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.686 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:33.686 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:14.498109+0000", 2024-05-15T13:07:33.687 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4206000000000003e-05 2024-05-15T13:07:33.687 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.687 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:33.687 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.687 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.687 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.688 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25180.0:219302 4.0 e395/317 4:0fd3cce0:::benchmark_data_smithi177_99756_object99330:head v 395'13850, mlcod=395'13845)", 2024-05-15T13:07:33.688 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:02:14.470677+0000", 2024-05-15T13:07:33.688 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 319.20431610200001, 2024-05-15T13:07:33.688 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.027461328, 2024-05-15T13:07:33.688 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:33.688 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:33.689 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:33.689 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.689 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:33.689 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:14.470677+0000", 2024-05-15T13:07:33.689 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.689 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.690 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.690 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:33.690 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:14.470677+0000", 2024-05-15T13:07:33.690 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.690 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.691 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.691 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:33.691 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:14.470677+0000", 2024-05-15T13:07:33.691 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3e-07 2024-05-15T13:07:33.691 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.691 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.692 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:33.692 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:14.470684+0000", 2024-05-15T13:07:33.692 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7619999999999998e-06 2024-05-15T13:07:33.692 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.692 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.692 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:33.692 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:14.470696+0000", 2024-05-15T13:07:33.693 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2067e-05 2024-05-15T13:07:33.693 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.693 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.693 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:33.693 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:14.470703+0000", 2024-05-15T13:07:33.693 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9110000000000004e-06 2024-05-15T13:07:33.694 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.694 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.694 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:33.694 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:14.479091+0000", 2024-05-15T13:07:33.694 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0083883129999999997 2024-05-15T13:07:33.695 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.695 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.695 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:33.695 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:14.479115+0000", 2024-05-15T13:07:33.695 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3694e-05 2024-05-15T13:07:33.695 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.696 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.696 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:33.696 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:14.498112+0000", 2024-05-15T13:07:33.696 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.018996751999999999 2024-05-15T13:07:33.696 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.696 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.697 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:33.697 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:14.498138+0000", 2024-05-15T13:07:33.697 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6098999999999999e-05 2024-05-15T13:07:33.697 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.697 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:33.697 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.698 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.698 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.698 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:98863 5.a e442/418 5:547e905d:::benchmark_data_smithi177_109991_object98862:head v 442'6176, mlcod=442'6172)", 2024-05-15T13:07:33.698 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:04:15.386768+0000", 2024-05-15T13:07:33.698 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 198.28822450499999, 2024-05-15T13:07:33.699 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028421753000000001, 2024-05-15T13:07:33.699 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:33.699 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:33.699 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:33.699 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.699 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:33.700 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:15.386768+0000", 2024-05-15T13:07:33.700 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.700 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.700 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.700 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:33.700 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:15.386768+0000", 2024-05-15T13:07:33.701 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.701 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.701 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.701 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:33.701 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:15.386770+0000", 2024-05-15T13:07:33.701 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6950000000000001e-06 2024-05-15T13:07:33.702 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.702 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.702 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:33.702 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:15.386918+0000", 2024-05-15T13:07:33.702 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000148302 2024-05-15T13:07:33.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.703 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.703 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:33.703 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:15.386966+0000", 2024-05-15T13:07:33.703 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7367999999999998e-05 2024-05-15T13:07:33.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.703 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.704 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:33.704 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:15.386976+0000", 2024-05-15T13:07:33.704 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0504e-05 2024-05-15T13:07:33.704 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.704 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.704 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:33.704 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:15.387082+0000", 2024-05-15T13:07:33.704 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000106074 2024-05-15T13:07:33.704 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.704 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.705 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:33.705 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:15.387108+0000", 2024-05-15T13:07:33.705 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5321e-05 2024-05-15T13:07:33.705 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.705 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.705 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:33.705 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:15.415129+0000", 2024-05-15T13:07:33.705 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.028021198000000001 2024-05-15T13:07:33.705 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.706 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.706 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:33.706 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:15.415190+0000", 2024-05-15T13:07:33.706 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1291000000000007e-05 2024-05-15T13:07:33.706 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.706 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:33.706 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.706 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.706 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.707 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:229664 5.a e473/418 5:5d40bfbe:::benchmark_data_smithi177_109991_object94473:head v 473'14315, mlcod=473'14313)", 2024-05-15T13:07:33.707 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:06:36.201072+0000", 2024-05-15T13:07:33.707 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 57.473920941999999, 2024-05-15T13:07:33.707 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.022269072000000001, 2024-05-15T13:07:33.707 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:33.707 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:33.707 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:33.707 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.707 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:33.707 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:36.201072+0000", 2024-05-15T13:07:33.708 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.708 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.708 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.708 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:33.708 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:36.201072+0000", 2024-05-15T13:07:33.708 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.708 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.708 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.708 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:33.709 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:36.201073+0000", 2024-05-15T13:07:33.709 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1999999999999998e-07 2024-05-15T13:07:33.709 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.709 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.709 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:33.709 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:36.201079+0000", 2024-05-15T13:07:33.709 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1940000000000003e-06 2024-05-15T13:07:33.709 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.709 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.709 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:33.709 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:36.201090+0000", 2024-05-15T13:07:33.710 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1590000000000001e-05 2024-05-15T13:07:33.710 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.710 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.710 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:33.710 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:36.201096+0000", 2024-05-15T13:07:33.710 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5999999999999997e-06 2024-05-15T13:07:33.710 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.710 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.710 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:33.711 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:36.201627+0000", 2024-05-15T13:07:33.711 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00053139599999999995 2024-05-15T13:07:33.711 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.711 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.711 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:33.711 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:36.201645+0000", 2024-05-15T13:07:33.711 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7822999999999999e-05 2024-05-15T13:07:33.711 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.712 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.712 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:33.712 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:36.223301+0000", 2024-05-15T13:07:33.712 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021655363 2024-05-15T13:07:33.712 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.712 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.712 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:33.712 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:36.223341+0000", 2024-05-15T13:07:33.712 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0286e-05 2024-05-15T13:07:33.712 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.713 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:33.713 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.713 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.713 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.713 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:229681 5.a e473/418 5:5094afef:::benchmark_data_smithi177_109991_object94490:head v 473'14319, mlcod=473'14313)", 2024-05-15T13:07:33.713 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:06:36.223152+0000", 2024-05-15T13:07:33.713 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 57.451840318000002, 2024-05-15T13:07:33.713 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.023518706, 2024-05-15T13:07:33.714 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:33.714 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:33.714 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:33.714 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.714 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:33.714 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:36.223152+0000", 2024-05-15T13:07:33.714 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.714 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.715 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.715 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:33.715 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:36.223152+0000", 2024-05-15T13:07:33.715 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.715 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.715 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.716 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:33.716 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:36.223153+0000", 2024-05-15T13:07:33.716 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7100000000000011e-07 2024-05-15T13:07:33.716 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.716 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.716 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:33.717 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:36.223160+0000", 2024-05-15T13:07:33.717 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6440000000000003e-06 2024-05-15T13:07:33.717 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.717 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.717 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:33.718 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:36.223171+0000", 2024-05-15T13:07:33.718 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1066999999999999e-05 2024-05-15T13:07:33.718 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.718 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.718 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:33.718 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:36.223178+0000", 2024-05-15T13:07:33.718 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9269999999999999e-06 2024-05-15T13:07:33.719 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.719 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.719 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:33.719 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:36.228114+0000", 2024-05-15T13:07:33.719 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0049361250000000004 2024-05-15T13:07:33.719 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.720 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.720 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:33.720 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:36.228140+0000", 2024-05-15T13:07:33.720 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5497999999999999e-05 2024-05-15T13:07:33.720 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.720 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.721 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:33.721 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:36.246633+0000", 2024-05-15T13:07:33.721 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.018493473999999999 2024-05-15T13:07:33.721 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.721 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.721 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:33.722 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:36.246671+0000", 2024-05-15T13:07:33.722 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8000000000000002e-05 2024-05-15T13:07:33.722 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.722 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:33.722 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.722 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.723 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.723 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:239415 5.e e482/418 5:792e3166:::benchmark_data_smithi177_109991_object104224:head v 482'15001, mlcod=481'14999)", 2024-05-15T13:07:33.723 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:06:48.408988+0000", 2024-05-15T13:07:33.723 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 45.266004334000002, 2024-05-15T13:07:33.723 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.030538717, 2024-05-15T13:07:33.723 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:33.724 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:33.724 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:33.724 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.724 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:33.724 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:48.408988+0000", 2024-05-15T13:07:33.724 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.724 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.724 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.724 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:33.724 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:48.408988+0000", 2024-05-15T13:07:33.724 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.724 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.724 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.725 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:33.725 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:48.408989+0000", 2024-05-15T13:07:33.725 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9799999999999999e-07 2024-05-15T13:07:33.725 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.725 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.725 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:33.725 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:48.408994+0000", 2024-05-15T13:07:33.725 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1989999999999997e-06 2024-05-15T13:07:33.725 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.725 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.725 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:33.725 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:48.409007+0000", 2024-05-15T13:07:33.725 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3312999999999999e-05 2024-05-15T13:07:33.726 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.726 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.726 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:33.726 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:48.409015+0000", 2024-05-15T13:07:33.726 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8099999999999998e-06 2024-05-15T13:07:33.726 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.726 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.726 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:33.726 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:48.415627+0000", 2024-05-15T13:07:33.726 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0066119600000000001 2024-05-15T13:07:33.726 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.726 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.726 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:33.727 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:48.415648+0000", 2024-05-15T13:07:33.727 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0543e-05 2024-05-15T13:07:33.727 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.727 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.727 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:33.727 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:48.439478+0000", 2024-05-15T13:07:33.727 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.023830232999999999 2024-05-15T13:07:33.727 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.727 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.727 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:33.727 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:48.439527+0000", 2024-05-15T13:07:33.727 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9360999999999997e-05 2024-05-15T13:07:33.727 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.727 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:33.727 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.728 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.728 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.728 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:239420 5.e e482/418 5:7a666b81:::benchmark_data_smithi177_109991_object104229:head v 482'15002, mlcod=481'14999)", 2024-05-15T13:07:33.728 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:06:48.417895+0000", 2024-05-15T13:07:33.728 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 45.257097332000001, 2024-05-15T13:07:33.728 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021662407000000002, 2024-05-15T13:07:33.728 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:33.728 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:33.728 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:33.728 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.728 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:33.728 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:48.417895+0000", 2024-05-15T13:07:33.728 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.728 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.729 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.729 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:33.729 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:48.417895+0000", 2024-05-15T13:07:33.729 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.729 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.729 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.729 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:33.729 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:48.417896+0000", 2024-05-15T13:07:33.729 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7199999999999999e-07 2024-05-15T13:07:33.729 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.729 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.729 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:33.729 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:48.417903+0000", 2024-05-15T13:07:33.730 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6540000000000002e-06 2024-05-15T13:07:33.730 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.730 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.730 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:33.730 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:48.417915+0000", 2024-05-15T13:07:33.730 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2762000000000001e-05 2024-05-15T13:07:33.730 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.730 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.730 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:33.730 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:48.417922+0000", 2024-05-15T13:07:33.730 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9800000000000001e-06 2024-05-15T13:07:33.730 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.730 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.731 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:33.731 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:48.420493+0000", 2024-05-15T13:07:33.731 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025704159999999998 2024-05-15T13:07:33.731 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.731 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.731 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:33.731 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:48.420507+0000", 2024-05-15T13:07:33.731 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3794e-05 2024-05-15T13:07:33.734 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.734 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.734 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:33.734 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:48.439531+0000", 2024-05-15T13:07:33.734 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.019024692999999999 2024-05-15T13:07:33.734 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.734 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.734 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:33.734 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:48.439558+0000", 2024-05-15T13:07:33.734 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6536000000000001e-05 2024-05-15T13:07:33.734 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.734 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:33.734 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.734 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.735 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.735 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:240792 5.a e482/418 5:5a5f8c78:::benchmark_data_smithi177_109991_object105601:head v 482'15018, mlcod=482'15015)", 2024-05-15T13:07:33.735 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:06:49.902542+0000", 2024-05-15T13:07:33.735 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 43.772450620999997, 2024-05-15T13:07:33.735 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021796247000000001, 2024-05-15T13:07:33.735 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:33.735 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:33.735 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:33.735 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.735 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:33.735 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:49.902542+0000", 2024-05-15T13:07:33.735 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.735 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.735 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.736 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:33.736 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:49.902542+0000", 2024-05-15T13:07:33.736 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.736 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.736 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.736 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:33.736 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:49.902543+0000", 2024-05-15T13:07:33.736 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.052e-06 2024-05-15T13:07:33.736 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.736 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.737 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:33.737 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:49.902551+0000", 2024-05-15T13:07:33.737 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7770000000000002e-06 2024-05-15T13:07:33.737 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.737 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.737 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:33.737 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:49.902563+0000", 2024-05-15T13:07:33.737 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1953000000000001e-05 2024-05-15T13:07:33.738 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.738 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.738 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:33.738 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:49.902575+0000", 2024-05-15T13:07:33.738 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2108e-05 2024-05-15T13:07:33.738 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.738 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.738 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:33.738 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:49.905609+0000", 2024-05-15T13:07:33.738 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0030338499999999998 2024-05-15T13:07:33.738 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.738 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.738 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:33.738 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:49.905632+0000", 2024-05-15T13:07:33.739 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2751e-05 2024-05-15T13:07:33.739 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.739 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.739 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:33.739 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:49.924305+0000", 2024-05-15T13:07:33.739 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.018673794000000001 2024-05-15T13:07:33.739 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.739 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.739 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:33.739 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:49.924338+0000", 2024-05-15T13:07:33.739 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2962000000000001e-05 2024-05-15T13:07:33.739 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.739 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:33.739 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.740 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.740 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.740 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:241918 5.e e482/418 5:7e10fcf6:::benchmark_data_smithi177_109991_object106727:head v 482'15167, mlcod=482'15165)", 2024-05-15T13:07:33.740 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:06:51.150665+0000", 2024-05-15T13:07:33.740 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 42.524327546999999, 2024-05-15T13:07:33.740 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.022802811999999999, 2024-05-15T13:07:33.740 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:33.740 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:33.740 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:33.740 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.740 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:33.740 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.150665+0000", 2024-05-15T13:07:33.740 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.740 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.741 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:33.741 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.150665+0000", 2024-05-15T13:07:33.741 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.741 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.741 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:33.741 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.150666+0000", 2024-05-15T13:07:33.741 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5000000000000001e-07 2024-05-15T13:07:33.741 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.741 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:33.741 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.150676+0000", 2024-05-15T13:07:33.741 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0129000000000001e-05 2024-05-15T13:07:33.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:33.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.150691+0000", 2024-05-15T13:07:33.742 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4248e-05 2024-05-15T13:07:33.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:33.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.150700+0000", 2024-05-15T13:07:33.742 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8309999999999996e-06 2024-05-15T13:07:33.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:33.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.155487+0000", 2024-05-15T13:07:33.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0047864639999999998 2024-05-15T13:07:33.743 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.743 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.743 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:33.743 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.155509+0000", 2024-05-15T13:07:33.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2150999999999998e-05 2024-05-15T13:07:33.743 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.743 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.743 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:33.743 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.173415+0000", 2024-05-15T13:07:33.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017905827999999999 2024-05-15T13:07:33.743 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.743 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.743 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:33.743 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.173468+0000", 2024-05-15T13:07:33.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3211000000000003e-05 2024-05-15T13:07:33.744 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.744 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:33.744 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.744 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.744 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:242023 5.e e483/418 5:79efb97f:::benchmark_data_smithi177_109991_object106832:head v 483'15172, mlcod=483'15170)", 2024-05-15T13:07:33.744 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:06:51.284040+0000", 2024-05-15T13:07:33.744 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 42.390952691000003, 2024-05-15T13:07:33.744 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.022535196, 2024-05-15T13:07:33.744 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:33.744 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:33.744 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:33.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:33.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.284040+0000", 2024-05-15T13:07:33.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.745 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:33.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.284040+0000", 2024-05-15T13:07:33.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.745 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:33.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.284041+0000", 2024-05-15T13:07:33.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3200000000000003e-07 2024-05-15T13:07:33.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:33.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.284050+0000", 2024-05-15T13:07:33.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9269999999999994e-06 2024-05-15T13:07:33.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:33.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.284067+0000", 2024-05-15T13:07:33.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6728e-05 2024-05-15T13:07:33.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:33.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.284075+0000", 2024-05-15T13:07:33.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2479999999999996e-06 2024-05-15T13:07:33.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.747 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:33.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.284192+0000", 2024-05-15T13:07:33.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117533 2024-05-15T13:07:33.747 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.747 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:33.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.284212+0000", 2024-05-15T13:07:33.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9786e-05 2024-05-15T13:07:33.747 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.747 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:33.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.306526+0000", 2024-05-15T13:07:33.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.022314008 2024-05-15T13:07:33.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:33.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.306575+0000", 2024-05-15T13:07:33.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9033999999999998e-05 2024-05-15T13:07:33.748 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.748 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:33.748 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.748 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:242028 5.e e483/418 5:7758bf8b:::benchmark_data_smithi177_109991_object106837:head v 483'15173, mlcod=483'15170)", 2024-05-15T13:07:33.748 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:06:51.284096+0000", 2024-05-15T13:07:33.748 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 42.390896644000001, 2024-05-15T13:07:33.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.022507237999999999, 2024-05-15T13:07:33.749 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:33.749 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:33.749 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:33.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:33.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.284096+0000", 2024-05-15T13:07:33.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:33.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.284096+0000", 2024-05-15T13:07:33.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:33.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.284096+0000", 2024-05-15T13:07:33.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.79e-07 2024-05-15T13:07:33.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:33.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.284102+0000", 2024-05-15T13:07:33.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4090000000000002e-06 2024-05-15T13:07:33.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.751 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:33.751 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.284108+0000", 2024-05-15T13:07:33.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.772e-06 2024-05-15T13:07:33.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.751 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:33.751 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.284112+0000", 2024-05-15T13:07:33.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5020000000000003e-06 2024-05-15T13:07:33.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.751 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:33.751 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.288947+0000", 2024-05-15T13:07:33.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0048348310000000004 2024-05-15T13:07:33.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.752 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.752 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:33.752 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.288968+0000", 2024-05-15T13:07:33.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1007999999999999e-05 2024-05-15T13:07:33.752 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.752 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.752 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:33.752 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.306579+0000", 2024-05-15T13:07:33.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017610665000000001 2024-05-15T13:07:33.752 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.752 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.752 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:33.752 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.306603+0000", 2024-05-15T13:07:33.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4771999999999999e-05 2024-05-15T13:07:33.753 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.753 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:33.753 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.753 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:242030 5.e e483/418 5:76fd9fbd:::benchmark_data_smithi177_109991_object106839:head v 483'15174, mlcod=483'15170)", 2024-05-15T13:07:33.753 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:06:51.284119+0000", 2024-05-15T13:07:33.753 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 42.390873974999998, 2024-05-15T13:07:33.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.022510915999999999, 2024-05-15T13:07:33.753 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:33.753 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:33.753 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:33.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:33.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.284119+0000", 2024-05-15T13:07:33.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:33.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.284119+0000", 2024-05-15T13:07:33.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:33.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.284119+0000", 2024-05-15T13:07:33.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.03e-07 2024-05-15T13:07:33.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:33.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.284126+0000", 2024-05-15T13:07:33.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4580000000000004e-06 2024-05-15T13:07:33.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.755 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:33.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.284132+0000", 2024-05-15T13:07:33.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1370000000000001e-06 2024-05-15T13:07:33.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.755 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:33.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.284136+0000", 2024-05-15T13:07:33.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9729999999999999e-06 2024-05-15T13:07:33.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.756 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:33.756 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.293583+0000", 2024-05-15T13:07:33.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0094475280000000002 2024-05-15T13:07:33.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.756 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:33.756 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.293607+0000", 2024-05-15T13:07:33.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3612e-05 2024-05-15T13:07:33.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.756 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:33.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.306608+0000", 2024-05-15T13:07:33.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013000957 2024-05-15T13:07:33.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:33.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:51.306630+0000", 2024-05-15T13:07:33.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2048e-05 2024-05-15T13:07:33.757 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.757 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:33.757 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.757 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:246230 5.e e487/418 5:72f2b15d:::benchmark_data_smithi177_109991_object111039:head v 487'15427, mlcod=487'15422)", 2024-05-15T13:07:33.757 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:06:56.176083+0000", 2024-05-15T13:07:33.757 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 37.498910016000004, 2024-05-15T13:07:33.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.025252625000000001, 2024-05-15T13:07:33.758 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:33.758 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:33.758 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:33.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:33.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:56.176083+0000", 2024-05-15T13:07:33.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:33.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:56.176083+0000", 2024-05-15T13:07:33.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.759 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:33.759 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:56.176083+0000", 2024-05-15T13:07:33.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7099999999999998e-07 2024-05-15T13:07:33.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.759 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:33.759 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:56.176090+0000", 2024-05-15T13:07:33.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8329999999999996e-06 2024-05-15T13:07:33.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.759 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:33.759 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:56.181589+0000", 2024-05-15T13:07:33.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0054996239999999998 2024-05-15T13:07:33.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:33.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:56.181600+0000", 2024-05-15T13:07:33.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0485e-05 2024-05-15T13:07:33.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:33.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:56.185733+0000", 2024-05-15T13:07:33.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0041327170000000002 2024-05-15T13:07:33.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:33.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:56.185753+0000", 2024-05-15T13:07:33.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9854e-05 2024-05-15T13:07:33.761 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:33.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:56.201279+0000", 2024-05-15T13:07:33.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015526958 2024-05-15T13:07:33.761 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:33.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:56.201335+0000", 2024-05-15T13:07:33.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5883e-05 2024-05-15T13:07:33.761 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.761 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:33.762 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.762 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:260444 5.e e496/418 5:754b7b32:::benchmark_data_smithi177_109991_object125253:head v 496'16262, mlcod=496'16259)", 2024-05-15T13:07:33.762 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:07:12.580380+0000", 2024-05-15T13:07:33.762 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.094612335000001, 2024-05-15T13:07:33.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.022024018999999999, 2024-05-15T13:07:33.762 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:33.762 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:33.762 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:33.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.762 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:33.762 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.580380+0000", 2024-05-15T13:07:33.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.763 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:33.763 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.580380+0000", 2024-05-15T13:07:33.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.763 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:33.763 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.580381+0000", 2024-05-15T13:07:33.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6700000000000003e-07 2024-05-15T13:07:33.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.763 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:33.763 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.580389+0000", 2024-05-15T13:07:33.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8159999999999994e-06 2024-05-15T13:07:33.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:33.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.580402+0000", 2024-05-15T13:07:33.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2673e-05 2024-05-15T13:07:33.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:33.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.580409+0000", 2024-05-15T13:07:33.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2840000000000001e-06 2024-05-15T13:07:33.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:33.765 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.580509+0000", 2024-05-15T13:07:33.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010043 2024-05-15T13:07:33.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:33.765 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.580528+0000", 2024-05-15T13:07:33.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.878e-05 2024-05-15T13:07:33.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:33.765 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.602374+0000", 2024-05-15T13:07:33.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021846029999999999 2024-05-15T13:07:33.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:33.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.602404+0000", 2024-05-15T13:07:33.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0338999999999999e-05 2024-05-15T13:07:33.766 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.766 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:33.766 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.766 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:260449 5.e e496/418 5:73cd3433:::benchmark_data_smithi177_109991_object125258:head v 496'16263, mlcod=496'16259)", 2024-05-15T13:07:33.766 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:07:12.580430+0000", 2024-05-15T13:07:33.766 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.094562795000002, 2024-05-15T13:07:33.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.022003782, 2024-05-15T13:07:33.766 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:33.767 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:33.767 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:33.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:33.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.580430+0000", 2024-05-15T13:07:33.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:33.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.580430+0000", 2024-05-15T13:07:33.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:33.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.580430+0000", 2024-05-15T13:07:33.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7099999999999998e-07 2024-05-15T13:07:33.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:33.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.580434+0000", 2024-05-15T13:07:33.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8600000000000003e-06 2024-05-15T13:07:33.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:33.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.580442+0000", 2024-05-15T13:07:33.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0919999999999998e-06 2024-05-15T13:07:33.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:33.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.580447+0000", 2024-05-15T13:07:33.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3159999999999998e-06 2024-05-15T13:07:33.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:33.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.584925+0000", 2024-05-15T13:07:33.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0044786560000000001 2024-05-15T13:07:33.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:33.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.584947+0000", 2024-05-15T13:07:33.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1506999999999999e-05 2024-05-15T13:07:33.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:33.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.602409+0000", 2024-05-15T13:07:33.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017462546999999998 2024-05-15T13:07:33.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:33.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.602434+0000", 2024-05-15T13:07:33.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4533e-05 2024-05-15T13:07:33.770 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.770 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:33.770 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.771 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:260451 5.e e496/418 5:72107395:::benchmark_data_smithi177_109991_object125260:head v 496'16265, mlcod=496'16259)", 2024-05-15T13:07:33.771 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:07:12.586618+0000", 2024-05-15T13:07:33.771 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 21.088374948999999, 2024-05-15T13:07:33.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.022604598, 2024-05-15T13:07:33.771 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:33.771 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:33.771 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:33.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.771 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:33.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.586618+0000", 2024-05-15T13:07:33.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:33.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.586618+0000", 2024-05-15T13:07:33.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:33.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.586619+0000", 2024-05-15T13:07:33.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6599999999999995e-07 2024-05-15T13:07:33.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:33.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.586626+0000", 2024-05-15T13:07:33.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.447e-06 2024-05-15T13:07:33.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.773 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:33.773 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.586641+0000", 2024-05-15T13:07:33.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4589999999999999e-05 2024-05-15T13:07:33.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.773 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:33.773 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.586648+0000", 2024-05-15T13:07:33.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4529999999999996e-06 2024-05-15T13:07:33.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.773 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:33.773 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.593218+0000", 2024-05-15T13:07:33.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0065696790000000001 2024-05-15T13:07:33.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:33.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.593238+0000", 2024-05-15T13:07:33.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0220999999999998e-05 2024-05-15T13:07:33.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:33.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.609172+0000", 2024-05-15T13:07:33.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015934455 2024-05-15T13:07:33.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:33.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:12.609222+0000", 2024-05-15T13:07:33.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9987000000000001e-05 2024-05-15T13:07:33.775 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.775 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:33.775 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.775 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:261846 5.e e497/418 5:7d39be3f:::benchmark_data_smithi177_109991_object126655:head v 497'16353, mlcod=497'16351)", 2024-05-15T13:07:33.775 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:07:14.108456+0000", 2024-05-15T13:07:33.775 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 19.566536667000001, 2024-05-15T13:07:33.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.023906093999999999, 2024-05-15T13:07:33.775 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:33.775 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:33.775 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:33.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:33.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:14.108456+0000", 2024-05-15T13:07:33.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:33.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:14.108456+0000", 2024-05-15T13:07:33.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:33.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:14.108457+0000", 2024-05-15T13:07:33.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8599999999999997e-07 2024-05-15T13:07:33.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:33.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:14.108465+0000", 2024-05-15T13:07:33.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4230000000000005e-06 2024-05-15T13:07:33.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:33.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:14.108480+0000", 2024-05-15T13:07:33.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4640999999999999e-05 2024-05-15T13:07:33.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:33.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:14.108488+0000", 2024-05-15T13:07:33.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7729999999999999e-06 2024-05-15T13:07:33.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:33.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:14.108573+0000", 2024-05-15T13:07:33.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5352999999999994e-05 2024-05-15T13:07:33.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:33.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:14.108595+0000", 2024-05-15T13:07:33.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1509e-05 2024-05-15T13:07:33.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:33.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:14.132327+0000", 2024-05-15T13:07:33.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.023732795000000001 2024-05-15T13:07:33.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:33.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:14.132362+0000", 2024-05-15T13:07:33.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4814000000000002e-05 2024-05-15T13:07:33.779 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.779 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:33.779 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.779 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:264093 5.a e497/418 5:57ef9094:::benchmark_data_smithi177_109991_object128902:head v 497'16486, mlcod=497'16481)", 2024-05-15T13:07:33.779 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:07:16.648674+0000", 2024-05-15T13:07:33.779 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.026318796000002, 2024-05-15T13:07:33.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021085929999999999, 2024-05-15T13:07:33.779 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:33.779 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:33.780 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:33.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:33.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:16.648674+0000", 2024-05-15T13:07:33.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:33.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:16.648674+0000", 2024-05-15T13:07:33.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:33.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:16.648676+0000", 2024-05-15T13:07:33.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.736e-06 2024-05-15T13:07:33.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:33.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:16.648686+0000", 2024-05-15T13:07:33.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8700000000000004e-06 2024-05-15T13:07:33.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:33.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:16.648697+0000", 2024-05-15T13:07:33.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1433e-05 2024-05-15T13:07:33.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:33.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:16.648707+0000", 2024-05-15T13:07:33.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7419999999999992e-06 2024-05-15T13:07:33.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:33.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:16.652452+0000", 2024-05-15T13:07:33.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037457010000000002 2024-05-15T13:07:33.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:33.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:16.652474+0000", 2024-05-15T13:07:33.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1613000000000001e-05 2024-05-15T13:07:33.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:33.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:16.669723+0000", 2024-05-15T13:07:33.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017249269000000001 2024-05-15T13:07:33.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:33.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:16.669760+0000", 2024-05-15T13:07:33.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6566000000000002e-05 2024-05-15T13:07:33.783 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.783 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:33.783 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.783 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:264480 5.a e498/418 5:5c17f147:::benchmark_data_smithi177_109991_object129289:head v 498'16510, mlcod=498'16507)", 2024-05-15T13:07:33.783 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:07:17.247454+0000", 2024-05-15T13:07:33.783 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.427538800000001, 2024-05-15T13:07:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.026191023000000001, 2024-05-15T13:07:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:17.247454+0000", 2024-05-15T13:07:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:17.247454+0000", 2024-05-15T13:07:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:17.247455+0000", 2024-05-15T13:07:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.540000000000001e-07 2024-05-15T13:07:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:17.247464+0000", 2024-05-15T13:07:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9609999999999994e-06 2024-05-15T13:07:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:17.247478+0000", 2024-05-15T13:07:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4435e-05 2024-05-15T13:07:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:17.247486+0000", 2024-05-15T13:07:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9179999999999994e-06 2024-05-15T13:07:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:17.247539+0000", 2024-05-15T13:07:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2692999999999998e-05 2024-05-15T13:07:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:17.247554+0000", 2024-05-15T13:07:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5417000000000002e-05 2024-05-15T13:07:33.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:17.273571+0000", 2024-05-15T13:07:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.026016943000000001 2024-05-15T13:07:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:07:17.273645+0000", 2024-05-15T13:07:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3701999999999993e-05 2024-05-15T13:07:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:33.787 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:33.787 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:07:33.959 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:07:33.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:33.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:07:33.983 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-05-15T13:07:33.984 INFO:tasks.radosbench:Beginning radosbench... 2024-05-15T13:07:33.984 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-05-15T13:07:33.985 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-05-15T13:07:34.063 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:07:34.094 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:34.231 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:34 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3297018071' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T13:07:34.232 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:34 smithi027 ceph-mon[39795]: osdmap e504: 12 total, 12 up, 11 in 2024-05-15T13:07:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:07:34.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:34.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:07:34.293 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:34 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3297018071' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T13:07:34.294 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:34 smithi177 ceph-mon[44303]: osdmap e504: 12 total, 12 up, 11 in 2024-05-15T13:07:34.319 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-05-15T13:07:34.347 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:07:34.377 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:34.397 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:07:34.399 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:34 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3297018071' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T13:07:34.399 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:34 smithi120 ceph-mon[43961]: osdmap e504: 12 total, 12 up, 11 in 2024-05-15T13:07:34.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:34.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:07:34.501 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:07:34.531 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:34.553 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:07:34.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:34.557 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:07:34.587 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:07:34.658 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:07:34.686 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:34.706 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:07:34.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:34.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:07:34.810 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:07:34.840 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:35.042 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:07:35.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:35.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:07:35.147 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:07:35.175 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:35.412 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.5 to 1 (10000) 2024-05-15T13:07:35.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:35 smithi177 ceph-mon[44303]: pgmap v1341: 11 pgs: 11 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:35.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:35 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3692150031' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T13:07:35.438 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:35 smithi027 ceph-mon[39795]: pgmap v1341: 11 pgs: 11 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:35.438 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:35 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3692150031' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T13:07:35.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:35 smithi120 ceph-mon[43961]: pgmap v1341: 11 pgs: 11 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:35.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:35 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3692150031' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T13:07:35.784 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:07:35.785 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:07:36.123 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 6 1 2024-05-15T13:07:36.310 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:07:36.310 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T13:07:36.310 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-05-15T13:07:36.310 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:07:36.659 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:36 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3692150031' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T13:07:36.660 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:36 smithi177 ceph-mon[44303]: osdmap e505: 12 total, 12 up, 11 in 2024-05-15T13:07:36.660 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:36 smithi177 ceph-mon[44303]: pgmap v1343: 11 pgs: 11 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:36.660 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:36 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:07:36.660 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:36 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:07:36.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:36 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3692150031' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T13:07:36.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:36 smithi027 ceph-mon[39795]: osdmap e505: 12 total, 12 up, 11 in 2024-05-15T13:07:36.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:36 smithi027 ceph-mon[39795]: pgmap v1343: 11 pgs: 11 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:36.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:36 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:07:36.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:36 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:07:36.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:36 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3692150031' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T13:07:36.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:36 smithi120 ceph-mon[43961]: osdmap e505: 12 total, 12 up, 11 in 2024-05-15T13:07:36.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:36 smithi120 ceph-mon[43961]: pgmap v1343: 11 pgs: 11 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:36.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:36 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:07:36.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:36 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:07:36.999 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-05-15T13:07:37.126 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:07:37.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:37.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:07:37.231 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:07:37.264 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:37.285 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:07:37.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:37.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:07:37.301 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:07:37.389 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:07:37.416 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:37.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:37 smithi120 ceph-mon[43961]: osdmap e506: 12 total, 12 up, 11 in 2024-05-15T13:07:37.542 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:07:37.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:37.545 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:07:37.646 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:07:37.679 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:37.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:37 smithi177 ceph-mon[44303]: osdmap e506: 12 total, 12 up, 11 in 2024-05-15T13:07:37.700 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:07:37.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:37.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:07:37.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:37 smithi027 ceph-mon[39795]: osdmap e506: 12 total, 12 up, 11 in 2024-05-15T13:07:37.804 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:07:37.834 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:37.854 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:07:37.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:37.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:07:37.957 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:07:37.986 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:38.083 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:07:38.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:38.086 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:07:38.187 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:07:38.219 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:38.344 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:07:38.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:38.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:07:38.373 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:07:38.429 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:07:38.449 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:07:38.479 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:38.500 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:07:38.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:38.503 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:07:38.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:38 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2651625949' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:07:38.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:38 smithi027 ceph-mon[39795]: pgmap v1345: 11 pgs: 11 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:38.604 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:07:38.630 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:38.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:38 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2651625949' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:07:38.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:38 smithi177 ceph-mon[44303]: pgmap v1345: 11 pgs: 11 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:38.778 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:07:38.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:38 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2651625949' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:07:38.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:38 smithi120 ceph-mon[43961]: pgmap v1345: 11 pgs: 11 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:38.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:38.781 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:07:38.882 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:07:38.911 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:39.044 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:07:39.044 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T13:07:39.044 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-05-15T13:07:39.044 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-05-15T13:07:39.044 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:07:39.162 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:07:39.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:39.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:07:39.266 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:07:39.300 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:39.321 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:07:39.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:39.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:07:39.426 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:07:39.456 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:39.530 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:07:39.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:39.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:07:39.635 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:07:39.665 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:07:39.665 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:39.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:39 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:07:39.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:39 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2651625949' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:07:39.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:39 smithi177 ceph-mon[44303]: osdmap e507: 12 total, 12 up, 11 in 2024-05-15T13:07:39.686 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:07:39.706 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:07:39.726 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:07:39.729 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:39 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:07:39.730 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:39 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2651625949' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:07:39.730 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:39 smithi027 ceph-mon[39795]: osdmap e507: 12 total, 12 up, 11 in 2024-05-15T13:07:39.742 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:07:39.749 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:07:39.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:39.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:07:39.770 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:07:39.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:39 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:07:39.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:39 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2651625949' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:07:39.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:39 smithi120 ceph-mon[43961]: osdmap e507: 12 total, 12 up, 11 in 2024-05-15T13:07:39.791 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:07:39.797 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-05-15T13:07:39.812 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:07:39.832 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:07:39.852 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:07:39.862 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:07:39.874 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:07:39.889 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:39.895 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:07:39.896 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:07:39.896 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:07:39.896 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:07:39.896 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:07:39.896 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:07:39.896 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:07:39.896 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:07:39.897 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:07:39.897 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:07:39.897 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:07:39.897 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:07:39.897 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:07:39.909 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:07:39.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:39.912 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:07:40.013 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:07:40.040 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:40.060 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:07:40.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:40.065 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:07:40.070 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:07:40.166 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:07:40.192 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:40.290 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:07:40.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:40.293 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:07:40.394 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:07:40.426 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:40.448 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:07:40.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:40.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:07:40.553 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:07:40.581 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:40 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1119805748' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:07:40.581 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:40 smithi177 ceph-mon[44303]: pgmap v1347: 11 pgs: 11 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:40.583 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:40.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:40 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1119805748' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:07:40.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:40 smithi027 ceph-mon[39795]: pgmap v1347: 11 pgs: 11 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:40.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:40 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1119805748' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:07:40.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:40 smithi120 ceph-mon[43961]: pgmap v1347: 11 pgs: 11 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:40.932 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:07:40.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:40.935 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:07:41.036 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:07:41.067 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:41.266 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:07:41.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:41.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:07:41.370 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:07:41.403 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:41.425 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:07:41.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:41.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:07:41.529 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:07:41.558 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:41.578 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:07:41.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:41.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:07:41.682 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:07:41.710 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:41.729 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:07:41.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:41.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:07:41.833 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:07:41.860 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:41.883 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:07:41.885 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:41 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3566792791' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T13:07:41.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:41.887 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:07:41.913 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:07:41.914 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:07:41.976 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.6 to 1 (10000) 2024-05-15T13:07:41.988 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:07:42.015 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:41 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3566792791' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T13:07:42.018 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:42.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:41 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3566792791' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T13:07:42.269 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:07:42.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:42.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:07:42.375 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:07:42.404 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:42.568 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:07:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:42.572 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:07:42.587 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-05-15T13:07:42.673 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:07:42.699 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:42.718 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:07:42.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:42.721 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:07:42.822 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:07:42.834 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:42 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3566792791' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T13:07:42.834 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:42 smithi177 ceph-mon[44303]: osdmap e508: 12 total, 12 up, 11 in 2024-05-15T13:07:42.834 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:42 smithi177 ceph-mon[44303]: pgmap v1349: 11 pgs: 11 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:42.834 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:42 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1925024312' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-05-15T13:07:42.839 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_4' created 2024-05-15T13:07:42.859 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:42.890 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:07:42.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:42 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3566792791' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T13:07:42.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:42 smithi027 ceph-mon[39795]: osdmap e508: 12 total, 12 up, 11 in 2024-05-15T13:07:42.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:42 smithi027 ceph-mon[39795]: pgmap v1349: 11 pgs: 11 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:42.969 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:42 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1925024312' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-05-15T13:07:43.068 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:07:43.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:42 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3566792791' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T13:07:43.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:42 smithi120 ceph-mon[43961]: osdmap e508: 12 total, 12 up, 11 in 2024-05-15T13:07:43.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:42 smithi120 ceph-mon[43961]: pgmap v1349: 11 pgs: 11 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:43.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:42 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1925024312' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-05-15T13:07:43.786 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 8 1 2024-05-15T13:07:43.809 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:07:43.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:43.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:07:43.913 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:07:43.943 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:44.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:43 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1925024312' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-05-15T13:07:44.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:43 smithi027 ceph-mon[39795]: osdmap e509: 12 total, 12 up, 11 in 2024-05-15T13:07:44.049 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:43 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1925024312' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-05-15T13:07:44.069 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:43 smithi177 ceph-mon[44303]: osdmap e509: 12 total, 12 up, 11 in 2024-05-15T13:07:44.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:44.070 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:07:44.070 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:07:44.173 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:07:44.203 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:43 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1925024312' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-05-15T13:07:44.203 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:43 smithi120 ceph-mon[43961]: osdmap e509: 12 total, 12 up, 11 in 2024-05-15T13:07:44.205 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:44.226 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:07:44.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:44.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:07:44.305 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-05-15T13:07:44.308 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:07:44.330 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:07:44.361 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:44.380 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:07:44.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:44.383 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:07:44.484 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:07:44.516 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:44.537 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:07:44.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:44.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:07:44.615 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:07:44.616 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:07:44.641 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:07:44.670 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:44.691 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:07:44.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:44.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:07:44.720 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:07:44.794 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:07:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:44.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:44 smithi177 ceph-mon[44303]: osdmap e510: 12 total, 12 up, 11 in 2024-05-15T13:07:44.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:44 smithi177 ceph-mon[44303]: pgmap v1352: 27 pgs: 5 creating+peering, 11 unknown, 11 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:45.132 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:07:45.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:45.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:07:45.139 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:44 smithi027 ceph-mon[39795]: osdmap e510: 12 total, 12 up, 11 in 2024-05-15T13:07:45.139 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:44 smithi027 ceph-mon[39795]: pgmap v1352: 27 pgs: 5 creating+peering, 11 unknown, 11 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:45.239 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:07:45.269 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:45.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:44 smithi120 ceph-mon[43961]: osdmap e510: 12 total, 12 up, 11 in 2024-05-15T13:07:45.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:44 smithi120 ceph-mon[43961]: pgmap v1352: 27 pgs: 5 creating+peering, 11 unknown, 11 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:45.298 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:07:45.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:45.301 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:07:45.326 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-05-15T13:07:45.402 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:07:45.434 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:45.456 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:07:45.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:45.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:07:45.560 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:07:45.588 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:45.601 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:07:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:07:45.609 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:07:45.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:45.612 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:07:45.628 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:07:45.713 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:07:45.739 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:45.830 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:07:45.861 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:07:45.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:45.864 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:07:45.965 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:07:45.992 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:46.089 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:07:46.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:46.092 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:07:46.099 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:45 smithi027 ceph-mon[39795]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T13:07:46.099 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:45 smithi027 ceph-mon[39795]: osdmap e511: 12 total, 12 up, 11 in 2024-05-15T13:07:46.123 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:45 smithi177 ceph-mon[44303]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T13:07:46.123 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:45 smithi177 ceph-mon[44303]: osdmap e511: 12 total, 12 up, 11 in 2024-05-15T13:07:46.194 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:07:46.222 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:46.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:45 smithi120 ceph-mon[43961]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T13:07:46.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:45 smithi120 ceph-mon[43961]: osdmap e511: 12 total, 12 up, 11 in 2024-05-15T13:07:46.573 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:07:46.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:46.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:07:46.677 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:07:46.707 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:47.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:46 smithi027 ceph-mon[39795]: pgmap v1354: 27 pgs: 5 creating+peering, 6 unknown, 16 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:47.096 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:46 smithi177 ceph-mon[44303]: pgmap v1354: 27 pgs: 5 creating+peering, 6 unknown, 16 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:47.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:46 smithi120 ceph-mon[43961]: pgmap v1354: 27 pgs: 5 creating+peering, 6 unknown, 16 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:47.386 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:07:47.386 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:07:48.032 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-05-15T13:07:48.332 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:07:48.360 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:07:48.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:48.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:07:48.478 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:07:48.511 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:48.531 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:07:48.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:48.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:07:48.636 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:07:48.665 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:48.684 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:07:48.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:48.687 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:07:48.788 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:07:48.817 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:49.317 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:49 smithi027 ceph-mon[39795]: pgmap v1355: 27 pgs: 5 creating+peering, 22 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:49.317 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:49 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1246065775' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:07:49.402 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:07:49.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:49 smithi177 ceph-mon[44303]: pgmap v1355: 27 pgs: 5 creating+peering, 22 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:49.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:49 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1246065775' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:07:49.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:49 smithi120 ceph-mon[43961]: pgmap v1355: 27 pgs: 5 creating+peering, 22 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:49.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:49 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1246065775' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:07:50.096 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:07:50.096 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:07:50.141 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:50 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:07:50.141 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:50 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1246065775' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:07:50.141 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:50 smithi027 ceph-mon[39795]: osdmap e512: 12 total, 12 up, 11 in 2024-05-15T13:07:50.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:50 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:07:50.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:50 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1246065775' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:07:50.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:50 smithi177 ceph-mon[44303]: osdmap e512: 12 total, 12 up, 11 in 2024-05-15T13:07:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:07:50.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:50.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:07:50.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:50 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:07:50.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:50 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1246065775' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:07:50.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:50 smithi120 ceph-mon[43961]: osdmap e512: 12 total, 12 up, 11 in 2024-05-15T13:07:50.603 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:07:50.635 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:50.657 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:07:50.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:50.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:07:50.749 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:07:50.760 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:07:50.768 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:07:50.788 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:07:50.790 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:50.793 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-05-15T13:07:50.807 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:07:50.811 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:07:50.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:50.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:07:50.828 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:07:50.847 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:07:50.868 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:07:50.888 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:07:50.908 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:07:50.915 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:07:50.931 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:07:50.946 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:50.953 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:07:50.978 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:07:50.979 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:07:50.979 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:07:50.979 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:07:50.979 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:07:50.979 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:07:50.980 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:07:50.980 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:07:50.980 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:07:50.980 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:07:50.980 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:07:50.980 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:07:50.980 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:07:51.056 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:07:51.120 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.8 to 1 (10000) 2024-05-15T13:07:51.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:51 smithi027 ceph-mon[39795]: pgmap v1357: 27 pgs: 27 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:51.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:51 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/584960555' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T13:07:51.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:51 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1898370830' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:07:51.299 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:07:51.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:51.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:07:51.405 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:07:51.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:51 smithi177 ceph-mon[44303]: pgmap v1357: 27 pgs: 27 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:51.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:51 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/584960555' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T13:07:51.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:51 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1898370830' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:07:51.433 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:51.507 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:07:51.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:51.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:07:51.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:51 smithi120 ceph-mon[43961]: pgmap v1357: 27 pgs: 27 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:51.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:51 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/584960555' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T13:07:51.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:51 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1898370830' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:07:51.611 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:07:51.639 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:52.070 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 9 1 2024-05-15T13:07:52.127 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-05-15T13:07:52.217 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:52 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/584960555' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T13:07:52.217 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:52 smithi177 ceph-mon[44303]: osdmap e513: 12 total, 12 up, 11 in 2024-05-15T13:07:52.217 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:52 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4048837216' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T13:07:52.504 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:07:52.519 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:52 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/584960555' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T13:07:52.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:52 smithi027 ceph-mon[39795]: osdmap e513: 12 total, 12 up, 11 in 2024-05-15T13:07:52.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:52 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4048837216' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T13:07:52.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:52 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/584960555' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T13:07:52.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:52 smithi120 ceph-mon[43961]: osdmap e513: 12 total, 12 up, 11 in 2024-05-15T13:07:52.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:52 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4048837216' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T13:07:52.852 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:07:52.853 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T13:07:52.853 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-05-15T13:07:52.853 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:07:53.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:53 smithi177 ceph-mon[44303]: pgmap v1359: 27 pgs: 27 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:53.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:53 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4048837216' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T13:07:53.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:53 smithi177 ceph-mon[44303]: osdmap e514: 12 total, 12 up, 11 in 2024-05-15T13:07:53.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:53 smithi027 ceph-mon[39795]: pgmap v1359: 27 pgs: 27 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:53.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:53 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4048837216' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T13:07:53.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:53 smithi027 ceph-mon[39795]: osdmap e514: 12 total, 12 up, 11 in 2024-05-15T13:07:53.519 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:07:53.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:53.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:07:53.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:53 smithi120 ceph-mon[43961]: pgmap v1359: 27 pgs: 27 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:53.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:53 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4048837216' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T13:07:53.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:53 smithi120 ceph-mon[43961]: osdmap e514: 12 total, 12 up, 11 in 2024-05-15T13:07:53.555 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:07:53.583 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-05-15T13:07:53.624 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:07:53.652 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:07:53.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:53.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:07:53.844 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:07:53.880 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:07:53.907 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:54.154 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:07:54.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:54.158 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:07:54.259 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:07:54.274 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:07:54.287 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:54.339 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:07:54.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:54.342 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:07:54.444 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:07:54.477 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:54.480 DEBUG:teuthology.orchestra.run.smithi177:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-05-15T13:07:54.483 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-05-15T13:07:54.483 INFO:teuthology.orchestra.run:waiting for 3000 2024-05-15T13:07:54.499 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:07:54.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:54.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:07:54.530 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:hints = 1 2024-05-15T13:07:54.530 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-05-15T13:07:54.530 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Object prefix: benchmark_data_smithi177_122278 2024-05-15T13:07:54.531 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:07:54.531 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 0 0 0 0 0 0 - 0 2024-05-15T13:07:54.604 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:07:54.645 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:54.672 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:07:54.675 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:54 smithi120 ceph-mon[43961]: pgmap v1361: 27 pgs: 27 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:54.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:54.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:07:54.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:54 smithi177 ceph-mon[44303]: pgmap v1361: 27 pgs: 27 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:54.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:54 smithi027 ceph-mon[39795]: pgmap v1361: 27 pgs: 27 active+clean; 577 KiB data, 4.7 GiB used, 979 GiB / 983 GiB avail 2024-05-15T13:07:54.777 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:07:54.817 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:54.992 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:07:55.173 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:07:55.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:55.177 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:07:55.278 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:07:55.332 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:55.360 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:07:55.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:55.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:07:55.466 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:07:55.493 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:55.531 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 1 16 3359 3343 208.917 208.938 0.00238221 0.00393488 2024-05-15T13:07:55.596 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:55 smithi027 ceph-mon[39795]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T13:07:55.596 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:55 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T13:07:55.618 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:07:55.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:55.622 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:07:55.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:55 smithi177 ceph-mon[44303]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T13:07:55.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:55 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T13:07:55.723 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:07:55.775 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:55.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:55 smithi120 ceph-mon[43961]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T13:07:55.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:55 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T13:07:55.797 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:07:55.797 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T13:07:55.797 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-05-15T13:07:55.797 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-05-15T13:07:55.797 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:07:56.135 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:07:56.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:56.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:07:56.243 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:07:56.282 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:56.425 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:56 smithi027 ceph-mon[39795]: pgmap v1362: 27 pgs: 27 active+clean; 34 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 5.1 MiB/s wr, 81 op/s 2024-05-15T13:07:56.531 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 2 16 6590 6574 205.417 201.938 0.00293844 0.00485996 2024-05-15T13:07:56.539 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:07:56.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:56.543 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:07:56.553 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-05-15T13:07:56.644 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:07:56.677 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:56 smithi120 ceph-mon[43961]: pgmap v1362: 27 pgs: 27 active+clean; 34 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 5.1 MiB/s wr, 81 op/s 2024-05-15T13:07:56.677 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:56.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:56 smithi177 ceph-mon[44303]: pgmap v1362: 27 pgs: 27 active+clean; 34 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 5.1 MiB/s wr, 81 op/s 2024-05-15T13:07:56.702 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:07:56.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:56.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:07:56.807 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:07:56.844 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:56.868 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:07:56.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:56.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:07:56.923 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:07:56.965 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:07:56.971 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:07:57.007 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:57.034 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:07:57.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:57.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:07:57.138 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:07:57.175 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:57.222 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:07:57.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:57.226 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:07:57.327 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:07:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:57.443 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.9 to 1 (10000) 2024-05-15T13:07:57.445 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:07:57.476 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:57 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2400126' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:07:57.476 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:57 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:07:57.477 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:57 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/178598286' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:07:57.532 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 3 16 9835 9819 204.543 202.812 0.00497403 0.00488255 2024-05-15T13:07:57.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:57 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2400126' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:07:57.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:57 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:07:57.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:57 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/178598286' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:07:57.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:57 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2400126' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:07:57.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:57 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:07:57.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:57 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/178598286' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:07:58.471 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:07:58.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:58.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:07:58.531 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 4 16 13297 13281 207.497 216.375 0.00631026 0.00481306 2024-05-15T13:07:58.577 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:07:58.612 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:58 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:07:58.613 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:58 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T13:07:58.613 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:58 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/178598286' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:07:58.613 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:58 smithi177 ceph-mon[44303]: osdmap e515: 12 total, 12 up, 11 in 2024-05-15T13:07:58.613 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:07:58 smithi177 ceph-mon[44303]: pgmap v1364: 27 pgs: 27 active+clean; 232 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 33 MiB/s wr, 533 op/s 2024-05-15T13:07:58.616 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:58.651 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:07:58.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:58.654 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:07:58.755 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:07:58.769 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:58 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:07:58.769 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:58 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T13:07:58.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:58 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/178598286' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:07:58.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:58 smithi027 ceph-mon[39795]: osdmap e515: 12 total, 12 up, 11 in 2024-05-15T13:07:58.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:07:58 smithi027 ceph-mon[39795]: pgmap v1364: 27 pgs: 27 active+clean; 232 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 33 MiB/s wr, 533 op/s 2024-05-15T13:07:58.779 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:58 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:07:58.779 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:58 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T13:07:58.779 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:58 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/178598286' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:07:58.779 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:58 smithi120 ceph-mon[43961]: osdmap e515: 12 total, 12 up, 11 in 2024-05-15T13:07:58.779 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:07:58 smithi120 ceph-mon[43961]: pgmap v1364: 27 pgs: 27 active+clean; 232 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 33 MiB/s wr, 533 op/s 2024-05-15T13:07:58.780 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:58.805 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:07:58.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:58.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:07:58.916 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:07:58.961 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:58.988 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:07:58.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:58.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:07:59.094 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:07:59.136 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:59.162 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:07:59.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:59.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:07:59.268 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:07:59.300 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:59.325 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:07:59.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:59.329 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:07:59.342 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:07:59.342 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-05-15T13:07:59.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-05-15T13:07:59.342 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T13:07:59.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.342 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25180.0:191483 4.3 4:cfbf05f9:::benchmark_data_smithi177_99756_object71511:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e384)", 2024-05-15T13:07:59.342 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:01:33.137202+0000", 2024-05-15T13:07:59.342 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 386.21255786400002, 2024-05-15T13:07:59.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.030570733, 2024-05-15T13:07:59.343 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:59.343 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-05-15T13:07:59.343 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:07:59.343 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25180", 2024-05-15T13:07:59.343 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:59.343 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 191483 2024-05-15T13:07:59.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.343 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:59.343 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.343 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:59.343 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.137202+0000", 2024-05-15T13:07:59.343 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.343 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.344 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:59.344 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.137202+0000", 2024-05-15T13:07:59.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.344 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:59.344 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.137204+0000", 2024-05-15T13:07:59.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9489999999999999e-06 2024-05-15T13:07:59.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.344 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:59.344 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.137222+0000", 2024-05-15T13:07:59.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8201000000000001e-05 2024-05-15T13:07:59.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.345 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.345 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:59.345 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.137238+0000", 2024-05-15T13:07:59.345 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6140000000000001e-05 2024-05-15T13:07:59.345 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.345 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.345 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:59.345 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.137248+0000", 2024-05-15T13:07:59.345 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2860000000000005e-06 2024-05-15T13:07:59.345 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.345 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.345 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.345 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.137309+0000", 2024-05-15T13:07:59.345 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1622000000000001e-05 2024-05-15T13:07:59.346 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.346 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.346 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op must wait for map", 2024-05-15T13:07:59.346 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.137319+0000", 2024-05-15T13:07:59.346 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9660000000000008e-06 2024-05-15T13:07:59.346 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.346 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.346 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.346 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.141244+0000", 2024-05-15T13:07:59.346 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0039251499999999996 2024-05-15T13:07:59.346 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.346 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.346 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:07:59.347 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.141251+0000", 2024-05-15T13:07:59.347 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7490000000000001e-06 2024-05-15T13:07:59.347 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.347 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.347 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.347 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.167685+0000", 2024-05-15T13:07:59.347 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0264337029999999 2024-05-15T13:07:59.347 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.347 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.347 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:59.347 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.167773+0000", 2024-05-15T13:07:59.347 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7967000000000007e-05 2024-05-15T13:07:59.347 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.347 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:59.348 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.348 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.348 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.348 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25180.0:191499 4.3 4:c651d98a:::benchmark_data_smithi177_99756_object71527:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-05-15T13:07:59.348 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:01:33.147802+0000", 2024-05-15T13:07:59.348 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 386.20195796500002, 2024-05-15T13:07:59.348 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0478178950000001, 2024-05-15T13:07:59.348 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:59.348 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:59.348 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:07:59.348 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25180", 2024-05-15T13:07:59.348 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:59.348 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 191499 2024-05-15T13:07:59.348 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.348 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:59.349 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.349 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:59.349 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.147802+0000", 2024-05-15T13:07:59.349 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.349 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.349 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.349 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:59.349 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.147802+0000", 2024-05-15T13:07:59.349 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.349 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.349 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.349 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:59.349 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.147803+0000", 2024-05-15T13:07:59.349 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1680000000000001e-06 2024-05-15T13:07:59.350 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.350 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:59.350 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.147807+0000", 2024-05-15T13:07:59.350 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4039999999999997e-06 2024-05-15T13:07:59.350 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.350 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:59.350 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.147819+0000", 2024-05-15T13:07:59.351 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2045000000000001e-05 2024-05-15T13:07:59.351 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.351 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.351 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:59.351 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.147826+0000", 2024-05-15T13:07:59.351 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8210000000000004e-06 2024-05-15T13:07:59.351 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.351 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.351 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.351 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.147869+0000", 2024-05-15T13:07:59.351 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2491999999999999e-05 2024-05-15T13:07:59.351 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.351 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.351 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:07:59.351 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.147876+0000", 2024-05-15T13:07:59.352 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8580000000000002e-06 2024-05-15T13:07:59.352 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.352 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.352 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.352 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.168265+0000", 2024-05-15T13:07:59.352 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0203895869999999 2024-05-15T13:07:59.352 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.352 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.352 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:59.352 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.168384+0000", 2024-05-15T13:07:59.352 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000118575 2024-05-15T13:07:59.352 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.352 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.352 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:07:59.353 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.168507+0000", 2024-05-15T13:07:59.353 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012308100000000001 2024-05-15T13:07:59.353 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.353 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.353 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:07:59.353 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.180220+0000", 2024-05-15T13:07:59.353 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.01171355 2024-05-15T13:07:59.353 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.353 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.353 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.353 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.195479+0000", 2024-05-15T13:07:59.353 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015258595999999999 2024-05-15T13:07:59.353 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.354 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.354 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.354 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.195521+0000", 2024-05-15T13:07:59.354 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2222000000000003e-05 2024-05-15T13:07:59.354 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.354 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.354 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:59.354 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.195559+0000", 2024-05-15T13:07:59.354 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7472000000000003e-05 2024-05-15T13:07:59.354 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.354 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.354 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:59.354 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.195620+0000", 2024-05-15T13:07:59.354 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1023999999999997e-05 2024-05-15T13:07:59.355 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.355 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:59.355 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.355 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.355 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.355 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25180.0:191527 4.3 4:c961a6b6:::benchmark_data_smithi177_99756_object71555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-05-15T13:07:59.355 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:01:33.193863+0000", 2024-05-15T13:07:59.355 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 386.15589728800001, 2024-05-15T13:07:59.355 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.023267578, 2024-05-15T13:07:59.355 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:59.355 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:59.355 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:07:59.355 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25180", 2024-05-15T13:07:59.355 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:59.355 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 191527 2024-05-15T13:07:59.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.356 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:59.356 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.356 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:59.356 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.193863+0000", 2024-05-15T13:07:59.356 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.356 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.356 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:59.356 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.193863+0000", 2024-05-15T13:07:59.356 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.356 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.356 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:59.356 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.193864+0000", 2024-05-15T13:07:59.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.838e-06 2024-05-15T13:07:59.357 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.357 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.357 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:59.357 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.193872+0000", 2024-05-15T13:07:59.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9510000000000006e-06 2024-05-15T13:07:59.357 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.357 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.357 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:59.357 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.193887+0000", 2024-05-15T13:07:59.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4228e-05 2024-05-15T13:07:59.357 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.357 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.358 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:59.358 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.193898+0000", 2024-05-15T13:07:59.358 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0983e-05 2024-05-15T13:07:59.358 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.358 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.358 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.358 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.193966+0000", 2024-05-15T13:07:59.358 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8465000000000003e-05 2024-05-15T13:07:59.358 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.358 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.358 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:07:59.358 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.193980+0000", 2024-05-15T13:07:59.359 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3991999999999999e-05 2024-05-15T13:07:59.359 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.359 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.359 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.359 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.171846+0000", 2024-05-15T13:07:59.359 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.97786605199999999 2024-05-15T13:07:59.359 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.359 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.359 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:59.359 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.171969+0000", 2024-05-15T13:07:59.359 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012274900000000001 2024-05-15T13:07:59.359 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.359 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.359 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:07:59.359 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.172090+0000", 2024-05-15T13:07:59.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012167699999999999 2024-05-15T13:07:59.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.360 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.360 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:07:59.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.187884+0000", 2024-05-15T13:07:59.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015793604999999999 2024-05-15T13:07:59.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.360 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.360 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.195664+0000", 2024-05-15T13:07:59.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0077797370000000001 2024-05-15T13:07:59.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.360 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.360 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.217060+0000", 2024-05-15T13:07:59.361 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021396681000000001 2024-05-15T13:07:59.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:59.361 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.217095+0000", 2024-05-15T13:07:59.361 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4187000000000003e-05 2024-05-15T13:07:59.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:59.361 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.217130+0000", 2024-05-15T13:07:59.361 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5432999999999998e-05 2024-05-15T13:07:59.361 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.361 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:59.362 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.362 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.362 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.362 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25180.0:191546 4.3 4:c01d7af0:::benchmark_data_smithi177_99756_object71574:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-05-15T13:07:59.362 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:01:33.221700+0000", 2024-05-15T13:07:59.362 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 386.12806013199997, 2024-05-15T13:07:59.362 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.99551631399999996, 2024-05-15T13:07:59.362 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:59.362 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:59.362 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:07:59.362 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25180", 2024-05-15T13:07:59.362 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:59.362 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 191546 2024-05-15T13:07:59.362 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.363 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:59.363 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.363 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:59.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.221700+0000", 2024-05-15T13:07:59.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.363 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.363 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.363 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:59.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.221700+0000", 2024-05-15T13:07:59.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.363 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.363 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.363 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:59.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.221701+0000", 2024-05-15T13:07:59.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.745e-06 2024-05-15T13:07:59.364 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:59.364 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.221708+0000", 2024-05-15T13:07:59.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3960000000000003e-06 2024-05-15T13:07:59.364 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:59.364 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.221721+0000", 2024-05-15T13:07:59.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2809999999999999e-05 2024-05-15T13:07:59.364 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:59.365 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.221729+0000", 2024-05-15T13:07:59.365 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7620000000000001e-06 2024-05-15T13:07:59.365 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.365 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.365 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.365 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.221796+0000", 2024-05-15T13:07:59.365 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6426000000000004e-05 2024-05-15T13:07:59.365 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.365 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.365 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:07:59.365 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.221804+0000", 2024-05-15T13:07:59.365 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9799999999999998e-06 2024-05-15T13:07:59.365 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.365 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.365 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.366 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.176077+0000", 2024-05-15T13:07:59.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95427309999999999 2024-05-15T13:07:59.366 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.366 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:59.366 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.176159+0000", 2024-05-15T13:07:59.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1849999999999997e-05 2024-05-15T13:07:59.366 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.366 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:07:59.366 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.176236+0000", 2024-05-15T13:07:59.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6865000000000004e-05 2024-05-15T13:07:59.366 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.366 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.367 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:07:59.367 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.192255+0000", 2024-05-15T13:07:59.367 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.016019881999999999 2024-05-15T13:07:59.367 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.367 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.367 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.367 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.195705+0000", 2024-05-15T13:07:59.367 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0034498839999999998 2024-05-15T13:07:59.367 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.367 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.367 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.367 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.217161+0000", 2024-05-15T13:07:59.367 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021455825000000001 2024-05-15T13:07:59.367 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.367 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.368 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:59.368 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.217191+0000", 2024-05-15T13:07:59.368 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9940000000000001e-05 2024-05-15T13:07:59.368 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.368 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.368 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:59.368 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.217216+0000", 2024-05-15T13:07:59.368 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4848999999999999e-05 2024-05-15T13:07:59.368 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.368 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:59.368 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.368 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.368 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.368 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25180.0:191637 4.3 4:c5eaa889:::benchmark_data_smithi177_99756_object71665:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-05-15T13:07:59.369 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:01:33.364248+0000", 2024-05-15T13:07:59.369 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 385.98551192399998, 2024-05-15T13:07:59.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.866422523, 2024-05-15T13:07:59.369 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:59.369 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:59.369 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:07:59.369 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25180", 2024-05-15T13:07:59.369 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:59.369 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 191637 2024-05-15T13:07:59.369 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.369 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:59.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.369 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:59.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.364248+0000", 2024-05-15T13:07:59.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:59.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.364248+0000", 2024-05-15T13:07:59.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:59.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.364250+0000", 2024-05-15T13:07:59.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.795e-06 2024-05-15T13:07:59.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:59.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.364255+0000", 2024-05-15T13:07:59.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7509999999999997e-06 2024-05-15T13:07:59.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.371 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:59.371 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.364270+0000", 2024-05-15T13:07:59.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4270999999999999e-05 2024-05-15T13:07:59.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.371 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:59.371 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.364278+0000", 2024-05-15T13:07:59.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0069999999999997e-06 2024-05-15T13:07:59.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.371 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.371 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.364345+0000", 2024-05-15T13:07:59.372 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7058999999999996e-05 2024-05-15T13:07:59.372 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.372 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.372 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:07:59.372 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.364358+0000", 2024-05-15T13:07:59.372 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3708e-05 2024-05-15T13:07:59.372 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.372 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.372 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.372 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.180264+0000", 2024-05-15T13:07:59.372 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81590599600000002 2024-05-15T13:07:59.372 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.372 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.372 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:59.373 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.180377+0000", 2024-05-15T13:07:59.373 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000112849 2024-05-15T13:07:59.373 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.373 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.373 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:07:59.373 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.180504+0000", 2024-05-15T13:07:59.373 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000126415 2024-05-15T13:07:59.373 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.373 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.373 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:07:59.373 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.192260+0000", 2024-05-15T13:07:59.373 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011756244000000001 2024-05-15T13:07:59.373 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.373 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.373 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.374 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.195755+0000", 2024-05-15T13:07:59.374 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0034952059999999998 2024-05-15T13:07:59.374 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.374 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.374 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.374 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.230602+0000", 2024-05-15T13:07:59.374 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.034847068000000002 2024-05-15T13:07:59.374 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.374 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.374 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:59.374 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.230636+0000", 2024-05-15T13:07:59.374 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3778000000000001e-05 2024-05-15T13:07:59.374 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.375 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.375 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:59.375 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.230670+0000", 2024-05-15T13:07:59.375 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4375999999999997e-05 2024-05-15T13:07:59.375 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.375 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:59.375 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.375 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.375 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.375 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25180.0:191639 4.3 4:c6972d4a:::benchmark_data_smithi177_99756_object71667:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-05-15T13:07:59.375 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:01:33.370660+0000", 2024-05-15T13:07:59.375 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 385.979099905, 2024-05-15T13:07:59.375 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.86011850000000001, 2024-05-15T13:07:59.376 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:59.376 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:59.376 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:07:59.376 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25180", 2024-05-15T13:07:59.376 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:59.376 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 191639 2024-05-15T13:07:59.376 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.376 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:59.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.376 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:59.376 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.370660+0000", 2024-05-15T13:07:59.376 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.376 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:59.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.370660+0000", 2024-05-15T13:07:59.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:59.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.370662+0000", 2024-05-15T13:07:59.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.767e-06 2024-05-15T13:07:59.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:59.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.370668+0000", 2024-05-15T13:07:59.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9510000000000003e-06 2024-05-15T13:07:59.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.378 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:59.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.370684+0000", 2024-05-15T13:07:59.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5891000000000001e-05 2024-05-15T13:07:59.378 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.378 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:59.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.370692+0000", 2024-05-15T13:07:59.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6379999999999993e-06 2024-05-15T13:07:59.378 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.378 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.370759+0000", 2024-05-15T13:07:59.379 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6606000000000006e-05 2024-05-15T13:07:59.379 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.379 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:07:59.379 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.370770+0000", 2024-05-15T13:07:59.379 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1725999999999999e-05 2024-05-15T13:07:59.379 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.379 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.379 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.184695+0000", 2024-05-15T13:07:59.379 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81392453099999995 2024-05-15T13:07:59.379 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.380 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:59.380 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.184771+0000", 2024-05-15T13:07:59.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6255999999999996e-05 2024-05-15T13:07:59.380 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.380 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.380 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:07:59.380 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.184847+0000", 2024-05-15T13:07:59.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5269000000000001e-05 2024-05-15T13:07:59.380 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.380 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.380 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:07:59.380 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.195629+0000", 2024-05-15T13:07:59.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010782323999999999 2024-05-15T13:07:59.380 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.381 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.381 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.381 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.211228+0000", 2024-05-15T13:07:59.381 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015599233000000001 2024-05-15T13:07:59.381 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.381 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.381 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.381 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.230717+0000", 2024-05-15T13:07:59.381 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.019488512999999999 2024-05-15T13:07:59.381 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.381 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.381 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:59.381 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.230746+0000", 2024-05-15T13:07:59.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9437999999999998e-05 2024-05-15T13:07:59.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.382 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:59.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.230778+0000", 2024-05-15T13:07:59.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2357000000000002e-05 2024-05-15T13:07:59.382 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.382 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:59.382 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.382 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.382 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25180.0:191645 4.3 4:c18e037e:::benchmark_data_smithi177_99756_object71673:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-05-15T13:07:59.382 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:01:33.382460+0000", 2024-05-15T13:07:59.383 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 385.967300206, 2024-05-15T13:07:59.383 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.84840918300000001, 2024-05-15T13:07:59.383 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:59.383 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:59.383 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:07:59.383 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25180", 2024-05-15T13:07:59.383 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:59.383 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 191645 2024-05-15T13:07:59.383 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.383 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:59.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.383 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:59.383 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.382460+0000", 2024-05-15T13:07:59.383 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.383 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:59.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.382460+0000", 2024-05-15T13:07:59.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.384 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:59.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.382461+0000", 2024-05-15T13:07:59.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.328e-06 2024-05-15T13:07:59.384 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:59.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.382474+0000", 2024-05-15T13:07:59.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3098000000000001e-05 2024-05-15T13:07:59.384 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.385 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.385 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:59.385 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.382492+0000", 2024-05-15T13:07:59.385 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7518000000000001e-05 2024-05-15T13:07:59.385 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.385 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.385 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:59.385 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.382507+0000", 2024-05-15T13:07:59.385 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5523999999999999e-05 2024-05-15T13:07:59.385 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.385 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.385 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.385 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.382600+0000", 2024-05-15T13:07:59.385 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2707999999999994e-05 2024-05-15T13:07:59.385 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.386 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.386 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:07:59.386 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.382611+0000", 2024-05-15T13:07:59.386 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1168e-05 2024-05-15T13:07:59.386 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.386 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.386 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.386 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.187959+0000", 2024-05-15T13:07:59.386 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.80534787100000005 2024-05-15T13:07:59.386 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.386 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.386 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:59.386 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.188065+0000", 2024-05-15T13:07:59.386 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000106278 2024-05-15T13:07:59.387 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.387 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.387 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:07:59.387 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.188178+0000", 2024-05-15T13:07:59.387 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011287599999999999 2024-05-15T13:07:59.387 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.387 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.387 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:07:59.387 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.195719+0000", 2024-05-15T13:07:59.387 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0075406270000000003 2024-05-15T13:07:59.387 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.387 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.387 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.387 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.211277+0000", 2024-05-15T13:07:59.387 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.01555865 2024-05-15T13:07:59.387 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.388 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.388 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.388 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.230814+0000", 2024-05-15T13:07:59.388 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.019536886999999999 2024-05-15T13:07:59.388 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.388 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.388 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:59.388 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.230845+0000", 2024-05-15T13:07:59.388 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0726999999999999e-05 2024-05-15T13:07:59.388 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.388 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.388 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:59.388 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.230869+0000", 2024-05-15T13:07:59.388 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3923000000000001e-05 2024-05-15T13:07:59.388 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.389 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:59.389 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.389 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.389 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.389 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25180.0:191668 4.3 4:ce2535d8:::benchmark_data_smithi177_99756_object71696:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-05-15T13:07:59.389 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:01:33.415598+0000", 2024-05-15T13:07:59.389 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 385.93416187000003, 2024-05-15T13:07:59.389 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.81536553000000001, 2024-05-15T13:07:59.389 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:59.389 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:59.389 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:07:59.389 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25180", 2024-05-15T13:07:59.389 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:59.389 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 191668 2024-05-15T13:07:59.389 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.390 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:59.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.390 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:59.390 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.415598+0000", 2024-05-15T13:07:59.390 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.390 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:59.390 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.415598+0000", 2024-05-15T13:07:59.390 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.390 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:59.390 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.415600+0000", 2024-05-15T13:07:59.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7090000000000001e-06 2024-05-15T13:07:59.391 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.391 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:59.391 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.415623+0000", 2024-05-15T13:07:59.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3414999999999999e-05 2024-05-15T13:07:59.391 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.391 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:59.391 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.415637+0000", 2024-05-15T13:07:59.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4346e-05 2024-05-15T13:07:59.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.392 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.392 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:59.392 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.415647+0000", 2024-05-15T13:07:59.392 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1689999999999998e-06 2024-05-15T13:07:59.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.392 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.392 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.392 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.415714+0000", 2024-05-15T13:07:59.392 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7411999999999998e-05 2024-05-15T13:07:59.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.392 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.392 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:07:59.393 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.415725+0000", 2024-05-15T13:07:59.393 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1374e-05 2024-05-15T13:07:59.393 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.393 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.393 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.393 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.192295+0000", 2024-05-15T13:07:59.393 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.77657010000000004 2024-05-15T13:07:59.393 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.393 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.393 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:59.393 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.192378+0000", 2024-05-15T13:07:59.393 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2516999999999996e-05 2024-05-15T13:07:59.393 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.394 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:07:59.394 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.192456+0000", 2024-05-15T13:07:59.394 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.839e-05 2024-05-15T13:07:59.394 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.394 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:07:59.394 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.195724+0000", 2024-05-15T13:07:59.394 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003267935 2024-05-15T13:07:59.394 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.394 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.394 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.211315+0000", 2024-05-15T13:07:59.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015590915 2024-05-15T13:07:59.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.230905+0000", 2024-05-15T13:07:59.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.019589557 2024-05-15T13:07:59.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:59.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.230937+0000", 2024-05-15T13:07:59.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1965999999999999e-05 2024-05-15T13:07:59.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:59.396 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.230963+0000", 2024-05-15T13:07:59.396 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6724999999999999e-05 2024-05-15T13:07:59.396 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.396 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:59.396 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.396 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25180.0:191680 4.3 4:c499681a:::benchmark_data_smithi177_99756_object71708:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-05-15T13:07:59.396 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:01:33.442827+0000", 2024-05-15T13:07:59.396 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 385.90693253000001, 2024-05-15T13:07:59.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.78827002000000002, 2024-05-15T13:07:59.397 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:59.397 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:59.397 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:07:59.397 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25180", 2024-05-15T13:07:59.397 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:59.397 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 191680 2024-05-15T13:07:59.397 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.397 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:59.397 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.397 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:59.397 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.442827+0000", 2024-05-15T13:07:59.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.398 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.398 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.398 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:59.398 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.442827+0000", 2024-05-15T13:07:59.398 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.398 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.398 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.398 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:59.398 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.442829+0000", 2024-05-15T13:07:59.398 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4249999999999999e-06 2024-05-15T13:07:59.398 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.398 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.398 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:59.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.442835+0000", 2024-05-15T13:07:59.399 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1469999999999998e-06 2024-05-15T13:07:59.399 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:59.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.442848+0000", 2024-05-15T13:07:59.399 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3360000000000001e-05 2024-05-15T13:07:59.399 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:59.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.442856+0000", 2024-05-15T13:07:59.399 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0690000000000002e-06 2024-05-15T13:07:59.399 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.442926+0000", 2024-05-15T13:07:59.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9956000000000006e-05 2024-05-15T13:07:59.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:07:59.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.442940+0000", 2024-05-15T13:07:59.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4134000000000001e-05 2024-05-15T13:07:59.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.195778+0000", 2024-05-15T13:07:59.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.75283778400000001 2024-05-15T13:07:59.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.401 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.401 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:59.401 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.195853+0000", 2024-05-15T13:07:59.401 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4773999999999999e-05 2024-05-15T13:07:59.401 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.401 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.401 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:07:59.401 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.195925+0000", 2024-05-15T13:07:59.401 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2076000000000006e-05 2024-05-15T13:07:59.401 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.401 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.401 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:07:59.401 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.211178+0000", 2024-05-15T13:07:59.401 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015253040000000001 2024-05-15T13:07:59.402 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.215964+0000", 2024-05-15T13:07:59.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0047862979999999996 2024-05-15T13:07:59.402 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.231017+0000", 2024-05-15T13:07:59.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015052482000000001 2024-05-15T13:07:59.402 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:59.403 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.231050+0000", 2024-05-15T13:07:59.403 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3155999999999999e-05 2024-05-15T13:07:59.403 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.403 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.403 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:59.403 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.231097+0000", 2024-05-15T13:07:59.403 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7318999999999999e-05 2024-05-15T13:07:59.403 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.403 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:59.403 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.403 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.403 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.403 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25180.0:191706 4.3 4:c046394b:::benchmark_data_smithi177_99756_object71734:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-05-15T13:07:59.403 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:01:33.526122+0000", 2024-05-15T13:07:59.404 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 385.82363767200002, 2024-05-15T13:07:59.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.71792786099999994, 2024-05-15T13:07:59.404 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:59.404 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:59.404 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:07:59.404 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25180", 2024-05-15T13:07:59.404 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:59.404 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 191706 2024-05-15T13:07:59.404 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.404 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:59.404 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.404 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:59.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.526122+0000", 2024-05-15T13:07:59.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.405 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.405 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:59.405 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.526122+0000", 2024-05-15T13:07:59.405 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.405 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.405 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:59.405 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.526124+0000", 2024-05-15T13:07:59.405 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.607e-06 2024-05-15T13:07:59.405 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:59.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.526129+0000", 2024-05-15T13:07:59.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6289999999999998e-06 2024-05-15T13:07:59.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:59.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.526143+0000", 2024-05-15T13:07:59.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3402e-05 2024-05-15T13:07:59.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:59.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.526151+0000", 2024-05-15T13:07:59.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8599999999999993e-06 2024-05-15T13:07:59.407 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.407 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.526211+0000", 2024-05-15T13:07:59.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0025999999999998e-05 2024-05-15T13:07:59.407 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.407 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:07:59.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.526222+0000", 2024-05-15T13:07:59.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1578e-05 2024-05-15T13:07:59.407 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.407 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.408 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.198988+0000", 2024-05-15T13:07:59.408 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.67276610100000001 2024-05-15T13:07:59.408 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.408 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.408 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:59.408 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.199069+0000", 2024-05-15T13:07:59.408 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0276000000000002e-05 2024-05-15T13:07:59.408 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.408 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.408 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:07:59.408 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.199152+0000", 2024-05-15T13:07:59.408 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3308999999999998e-05 2024-05-15T13:07:59.408 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.408 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.409 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:07:59.409 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.211184+0000", 2024-05-15T13:07:59.409 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012032391999999999 2024-05-15T13:07:59.409 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.409 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.409 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.409 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.222294+0000", 2024-05-15T13:07:59.409 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.01110921 2024-05-15T13:07:59.409 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.409 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.409 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.409 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.243931+0000", 2024-05-15T13:07:59.409 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021637863 2024-05-15T13:07:59.409 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.410 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.410 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:59.410 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.243986+0000", 2024-05-15T13:07:59.410 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4483e-05 2024-05-15T13:07:59.410 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.410 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.410 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:59.410 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.244050+0000", 2024-05-15T13:07:59.410 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4125e-05 2024-05-15T13:07:59.410 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.410 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:59.410 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.410 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.410 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.411 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25180.0:191724 4.3 4:c12e820d:::benchmark_data_smithi177_99756_object71752:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-05-15T13:07:59.411 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:01:33.665539+0000", 2024-05-15T13:07:59.411 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 385.68422109599999, 2024-05-15T13:07:59.411 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.57863392199999997, 2024-05-15T13:07:59.411 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:59.411 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:59.411 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:07:59.411 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25180", 2024-05-15T13:07:59.411 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:59.411 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 191724 2024-05-15T13:07:59.411 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.411 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:59.411 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.412 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:59.412 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.665539+0000", 2024-05-15T13:07:59.412 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.412 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.412 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.412 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:59.412 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.665539+0000", 2024-05-15T13:07:59.412 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.412 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.412 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.412 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:59.412 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.665540+0000", 2024-05-15T13:07:59.412 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.483e-06 2024-05-15T13:07:59.412 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.413 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:59.413 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.665545+0000", 2024-05-15T13:07:59.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4279999999999998e-06 2024-05-15T13:07:59.413 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.413 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:59.413 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.665555+0000", 2024-05-15T13:07:59.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0027000000000001e-05 2024-05-15T13:07:59.413 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.413 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:59.413 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.665560+0000", 2024-05-15T13:07:59.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5629999999999999e-06 2024-05-15T13:07:59.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.414 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.665635+0000", 2024-05-15T13:07:59.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5016000000000001e-05 2024-05-15T13:07:59.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:07:59.414 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.665648+0000", 2024-05-15T13:07:59.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.293e-05 2024-05-15T13:07:59.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.201788+0000", 2024-05-15T13:07:59.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.53613936900000003 2024-05-15T13:07:59.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:59.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.201864+0000", 2024-05-15T13:07:59.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6598000000000001e-05 2024-05-15T13:07:59.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:07:59.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.201936+0000", 2024-05-15T13:07:59.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2106000000000004e-05 2024-05-15T13:07:59.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:07:59.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.211289+0000", 2024-05-15T13:07:59.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0093532049999999999 2024-05-15T13:07:59.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.235772+0000", 2024-05-15T13:07:59.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.024482729000000002 2024-05-15T13:07:59.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.417 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.244086+0000", 2024-05-15T13:07:59.417 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0083134539999999996 2024-05-15T13:07:59.417 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:59.417 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.244128+0000", 2024-05-15T13:07:59.417 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2506999999999998e-05 2024-05-15T13:07:59.417 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:59.417 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.244173+0000", 2024-05-15T13:07:59.417 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4506999999999999e-05 2024-05-15T13:07:59.417 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.417 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:59.418 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.418 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.418 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.418 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25180.0:191729 4.3 4:cf9f6678:::benchmark_data_smithi177_99756_object71757:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-05-15T13:07:59.418 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:01:33.672509+0000", 2024-05-15T13:07:59.418 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 385.67725039599998, 2024-05-15T13:07:59.418 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.57177415099999995, 2024-05-15T13:07:59.418 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:59.418 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:59.418 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:07:59.418 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25180", 2024-05-15T13:07:59.418 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:59.418 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 191729 2024-05-15T13:07:59.419 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.419 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:59.419 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.419 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:59.419 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.672509+0000", 2024-05-15T13:07:59.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.419 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.419 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.419 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:59.419 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.672509+0000", 2024-05-15T13:07:59.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.419 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.419 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.419 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:59.419 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.672511+0000", 2024-05-15T13:07:59.420 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4300000000000001e-06 2024-05-15T13:07:59.420 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.420 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.420 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:59.420 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.672516+0000", 2024-05-15T13:07:59.420 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8760000000000004e-06 2024-05-15T13:07:59.420 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.420 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.420 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:59.420 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.672531+0000", 2024-05-15T13:07:59.420 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5004000000000001e-05 2024-05-15T13:07:59.420 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.420 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.420 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:59.420 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.672539+0000", 2024-05-15T13:07:59.421 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6260000000000001e-06 2024-05-15T13:07:59.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.421 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.421 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.421 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.672614+0000", 2024-05-15T13:07:59.421 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.47e-05 2024-05-15T13:07:59.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.421 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.421 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:07:59.421 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.672626+0000", 2024-05-15T13:07:59.421 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2092e-05 2024-05-15T13:07:59.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.421 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.421 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.422 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.204245+0000", 2024-05-15T13:07:59.422 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.53161868599999995 2024-05-15T13:07:59.422 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.422 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.422 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:59.422 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.204339+0000", 2024-05-15T13:07:59.422 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3954000000000002e-05 2024-05-15T13:07:59.422 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.422 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.422 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:07:59.422 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.204424+0000", 2024-05-15T13:07:59.422 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5147000000000003e-05 2024-05-15T13:07:59.422 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.423 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.423 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:07:59.423 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.211296+0000", 2024-05-15T13:07:59.423 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0068724490000000001 2024-05-15T13:07:59.423 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.423 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.423 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.423 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.238639+0000", 2024-05-15T13:07:59.423 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.027342984000000001 2024-05-15T13:07:59.423 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.423 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.423 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.423 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.244204+0000", 2024-05-15T13:07:59.423 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.005564332 2024-05-15T13:07:59.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.424 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:59.424 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.244244+0000", 2024-05-15T13:07:59.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0021999999999997e-05 2024-05-15T13:07:59.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.424 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:59.424 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.244284+0000", 2024-05-15T13:07:59.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9848999999999997e-05 2024-05-15T13:07:59.424 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.424 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:59.424 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.425 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25180.0:191747 4.3 4:c09056a2:::benchmark_data_smithi177_99756_object71775:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-05-15T13:07:59.425 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:01:33.727911+0000", 2024-05-15T13:07:59.425 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 385.621848822, 2024-05-15T13:07:59.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.51648255200000004, 2024-05-15T13:07:59.425 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:59.425 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:59.425 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:07:59.425 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25180", 2024-05-15T13:07:59.425 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:59.425 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 191747 2024-05-15T13:07:59.425 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.425 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:59.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.425 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:59.426 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.727911+0000", 2024-05-15T13:07:59.426 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.426 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.426 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.426 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:59.426 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.727911+0000", 2024-05-15T13:07:59.426 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.426 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.426 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.426 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:59.426 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.727912+0000", 2024-05-15T13:07:59.427 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2559999999999999e-06 2024-05-15T13:07:59.427 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.427 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.427 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:59.427 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.727917+0000", 2024-05-15T13:07:59.427 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6140000000000002e-06 2024-05-15T13:07:59.427 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.427 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.427 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:59.427 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.727930+0000", 2024-05-15T13:07:59.427 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3416e-05 2024-05-15T13:07:59.427 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.428 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.428 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:59.428 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.727937+0000", 2024-05-15T13:07:59.428 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5089999999999999e-06 2024-05-15T13:07:59.428 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.428 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.428 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.428 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.728002+0000", 2024-05-15T13:07:59.428 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.567e-05 2024-05-15T13:07:59.428 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.428 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.428 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:07:59.428 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.728015+0000", 2024-05-15T13:07:59.428 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2364999999999999e-05 2024-05-15T13:07:59.429 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.429 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.429 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.429 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.207050+0000", 2024-05-15T13:07:59.429 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.47903485899999998 2024-05-15T13:07:59.429 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.429 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.429 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:59.429 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.207133+0000", 2024-05-15T13:07:59.429 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2867999999999993e-05 2024-05-15T13:07:59.429 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.429 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.429 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:07:59.429 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.207220+0000", 2024-05-15T13:07:59.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7571000000000006e-05 2024-05-15T13:07:59.430 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.430 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.430 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:07:59.430 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.213948+0000", 2024-05-15T13:07:59.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0067279389999999996 2024-05-15T13:07:59.430 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.430 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.430 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.430 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.238686+0000", 2024-05-15T13:07:59.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.024738095000000002 2024-05-15T13:07:59.430 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.430 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.431 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.431 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.244314+0000", 2024-05-15T13:07:59.431 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0056276490000000002 2024-05-15T13:07:59.431 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.431 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.431 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:59.431 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.244354+0000", 2024-05-15T13:07:59.431 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.032e-05 2024-05-15T13:07:59.431 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.431 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.431 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:59.431 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.244394+0000", 2024-05-15T13:07:59.431 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9421000000000001e-05 2024-05-15T13:07:59.431 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.431 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:59.432 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.432 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.432 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.432 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25180.0:191758 4.3 4:ce536d99:::benchmark_data_smithi177_99756_object71786:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-05-15T13:07:59.432 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:01:33.770369+0000", 2024-05-15T13:07:59.432 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 385.57939036900001, 2024-05-15T13:07:59.432 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.474131988, 2024-05-15T13:07:59.432 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:59.432 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:59.432 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:07:59.432 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25180", 2024-05-15T13:07:59.432 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:59.432 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 191758 2024-05-15T13:07:59.432 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.433 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:59.433 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.433 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:59.433 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.770369+0000", 2024-05-15T13:07:59.433 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.433 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.433 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.433 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:59.433 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.770369+0000", 2024-05-15T13:07:59.433 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.433 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.433 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.433 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:59.434 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.770371+0000", 2024-05-15T13:07:59.434 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3400000000000001e-06 2024-05-15T13:07:59.434 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.434 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.434 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:59.434 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.770376+0000", 2024-05-15T13:07:59.434 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3920000000000002e-06 2024-05-15T13:07:59.434 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.434 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.434 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:59.434 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.770388+0000", 2024-05-15T13:07:59.434 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1664e-05 2024-05-15T13:07:59.434 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.435 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.435 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:59.435 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.770395+0000", 2024-05-15T13:07:59.435 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0949999999999998e-06 2024-05-15T13:07:59.435 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.435 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.435 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.435 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.770450+0000", 2024-05-15T13:07:59.435 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4592000000000002e-05 2024-05-15T13:07:59.435 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.435 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.435 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:07:59.435 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.770458+0000", 2024-05-15T13:07:59.435 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1340000000000004e-06 2024-05-15T13:07:59.436 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.436 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.436 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.436 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.211350+0000", 2024-05-15T13:07:59.436 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.44089187099999999 2024-05-15T13:07:59.436 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.436 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.436 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:59.436 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.211457+0000", 2024-05-15T13:07:59.436 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010721999999999999 2024-05-15T13:07:59.436 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.436 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.436 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:07:59.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.211587+0000", 2024-05-15T13:07:59.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012989300000000001 2024-05-15T13:07:59.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.437 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:07:59.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.215906+0000", 2024-05-15T13:07:59.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0043192880000000001 2024-05-15T13:07:59.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.437 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.238722+0000", 2024-05-15T13:07:59.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.022815760000000001 2024-05-15T13:07:59.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.244423+0000", 2024-05-15T13:07:59.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0057017400000000003 2024-05-15T13:07:59.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.438 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:59.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.244463+0000", 2024-05-15T13:07:59.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9558000000000003e-05 2024-05-15T13:07:59.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.438 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:59.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.244501+0000", 2024-05-15T13:07:59.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8441e-05 2024-05-15T13:07:59.439 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.439 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:59.439 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.439 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.439 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.439 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25180.0:191784 4.3 4:c945da09:::benchmark_data_smithi177_99756_object71812:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-05-15T13:07:59.439 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:01:33.880328+0000", 2024-05-15T13:07:59.439 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 385.469431751, 2024-05-15T13:07:59.439 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36429404599999998, 2024-05-15T13:07:59.439 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:59.439 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:59.439 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:07:59.439 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25180", 2024-05-15T13:07:59.440 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:07:59.440 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 191784 2024-05-15T13:07:59.440 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.440 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:59.440 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.440 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:59.440 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.880328+0000", 2024-05-15T13:07:59.440 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.440 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.440 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.440 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:59.440 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.880328+0000", 2024-05-15T13:07:59.440 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.441 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.441 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.441 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:59.441 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.880330+0000", 2024-05-15T13:07:59.441 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6789999999999999e-06 2024-05-15T13:07:59.441 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.441 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.441 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:59.441 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.880335+0000", 2024-05-15T13:07:59.441 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9960000000000001e-06 2024-05-15T13:07:59.441 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.441 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.442 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:59.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.880348+0000", 2024-05-15T13:07:59.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2782000000000001e-05 2024-05-15T13:07:59.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.442 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.442 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:59.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.880354+0000", 2024-05-15T13:07:59.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0009999999999998e-06 2024-05-15T13:07:59.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.442 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.442 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.880404+0000", 2024-05-15T13:07:59.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0442999999999998e-05 2024-05-15T13:07:59.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.443 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:07:59.443 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:33.880412+0000", 2024-05-15T13:07:59.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4950000000000008e-06 2024-05-15T13:07:59.443 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.443 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.443 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.213966+0000", 2024-05-15T13:07:59.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33355379200000002 2024-05-15T13:07:59.443 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.443 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:59.443 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.214035+0000", 2024-05-15T13:07:59.444 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8609999999999995e-05 2024-05-15T13:07:59.444 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.444 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.444 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:07:59.444 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.214103+0000", 2024-05-15T13:07:59.444 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8229e-05 2024-05-15T13:07:59.444 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.444 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.444 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:07:59.444 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.216084+0000", 2024-05-15T13:07:59.444 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001980416 2024-05-15T13:07:59.444 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.444 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.238759+0000", 2024-05-15T13:07:59.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.022675250000000001 2024-05-15T13:07:59.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.244532+0000", 2024-05-15T13:07:59.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0057728850000000002 2024-05-15T13:07:59.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:59.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.244582+0000", 2024-05-15T13:07:59.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0238000000000002e-05 2024-05-15T13:07:59.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:59.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:01:34.244622+0000", 2024-05-15T13:07:59.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0229999999999999e-05 2024-05-15T13:07:59.446 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.446 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:59.446 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.446 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDelete(4.1 e398,343 4:8d2708be:::benchmark_data_smithi177_99756_object908:head,350'7608)", 2024-05-15T13:07:59.446 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:02:18.650642+0000", 2024-05-15T13:07:59.446 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 340.699117742, 2024-05-15T13:07:59.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.122462879, 2024-05-15T13:07:59.446 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:59.447 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T13:07:59.447 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:59.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:59.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:18.650642+0000", 2024-05-15T13:07:59.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:59.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:18.650642+0000", 2024-05-15T13:07:59.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:59.448 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:18.650643+0000", 2024-05-15T13:07:59.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9199999999999999e-07 2024-05-15T13:07:59.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.448 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:59.448 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:18.650649+0000", 2024-05-15T13:07:59.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1199999999999999e-06 2024-05-15T13:07:59.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.448 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:59.448 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:18.650661+0000", 2024-05-15T13:07:59.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1824000000000001e-05 2024-05-15T13:07:59.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:59.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:18.650670+0000", 2024-05-15T13:07:59.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3190000000000001e-06 2024-05-15T13:07:59.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:18.756656+0000", 2024-05-15T13:07:59.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.105985818 2024-05-15T13:07:59.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:59.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:18.756666+0000", 2024-05-15T13:07:59.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9010000000000006e-06 2024-05-15T13:07:59.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:59.450 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:18.773105+0000", 2024-05-15T13:07:59.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.016439005 2024-05-15T13:07:59.450 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.450 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:59.450 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.450 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGRecoveryDelete(4.9 e403,343 4:974ec453:::benchmark_data_smithi177_99756_object7108:head,352'7910)", 2024-05-15T13:07:59.450 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:02:36.560674+0000", 2024-05-15T13:07:59.450 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 322.78908606099998, 2024-05-15T13:07:59.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.160653819, 2024-05-15T13:07:59.450 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:59.451 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T13:07:59.451 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:59.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.451 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:59.451 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:36.560674+0000", 2024-05-15T13:07:59.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.451 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:59.451 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:36.560674+0000", 2024-05-15T13:07:59.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.451 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:59.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:36.560674+0000", 2024-05-15T13:07:59.452 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8100000000000003e-07 2024-05-15T13:07:59.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:59.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:36.560679+0000", 2024-05-15T13:07:59.452 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8450000000000002e-06 2024-05-15T13:07:59.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:59.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:36.560688+0000", 2024-05-15T13:07:59.452 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4339999999999992e-06 2024-05-15T13:07:59.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.453 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:59.453 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:36.560693+0000", 2024-05-15T13:07:59.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8340000000000004e-06 2024-05-15T13:07:59.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.453 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.453 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.453 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:36.721198+0000", 2024-05-15T13:07:59.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16050508999999999 2024-05-15T13:07:59.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.453 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.453 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:59.453 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:36.721206+0000", 2024-05-15T13:07:59.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8739999999999995e-06 2024-05-15T13:07:59.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.454 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.454 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:59.454 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:02:36.721328+0000", 2024-05-15T13:07:59.454 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000121261 2024-05-15T13:07:59.454 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.454 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:59.454 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.454 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.454 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.454 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:149521 5.d 5:bfc1d209:::benchmark_data_smithi177_109991_object14330:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e453)", 2024-05-15T13:07:59.454 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:05:05.103859+0000", 2024-05-15T13:07:59.454 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 174.245900834, 2024-05-15T13:07:59.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21243480200000001, 2024-05-15T13:07:59.455 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:59.455 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:59.455 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:07:59.455 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:07:59.455 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:07:59.455 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 149521 2024-05-15T13:07:59.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.455 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:59.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.455 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:59.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.103859+0000", 2024-05-15T13:07:59.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:59.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.103859+0000", 2024-05-15T13:07:59.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:59.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.103860+0000", 2024-05-15T13:07:59.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.353e-06 2024-05-15T13:07:59.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:59.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.103865+0000", 2024-05-15T13:07:59.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.045e-06 2024-05-15T13:07:59.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.457 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:59.457 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.103881+0000", 2024-05-15T13:07:59.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5330999999999999e-05 2024-05-15T13:07:59.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.457 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:59.457 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.103890+0000", 2024-05-15T13:07:59.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2949999999999999e-06 2024-05-15T13:07:59.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.457 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.457 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.103952+0000", 2024-05-15T13:07:59.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1870000000000002e-05 2024-05-15T13:07:59.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:59.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.104079+0000", 2024-05-15T13:07:59.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000126668 2024-05-15T13:07:59.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,5", 2024-05-15T13:07:59.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.104187+0000", 2024-05-15T13:07:59.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108096 2024-05-15T13:07:59.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:07:59.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.111866+0000", 2024-05-15T13:07:59.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0076792060000000001 2024-05-15T13:07:59.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.117054+0000", 2024-05-15T13:07:59.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0051881169999999999 2024-05-15T13:07:59.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.316221+0000", 2024-05-15T13:07:59.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.199167028 2024-05-15T13:07:59.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:59.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.316260+0000", 2024-05-15T13:07:59.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9378999999999997e-05 2024-05-15T13:07:59.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:59.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.316294+0000", 2024-05-15T13:07:59.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3414000000000002e-05 2024-05-15T13:07:59.460 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.460 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:59.460 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.460 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:149552 5.d 5:b71c1ad6:::benchmark_data_smithi177_109991_object14361:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e453)", 2024-05-15T13:07:59.461 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:05:05.135101+0000", 2024-05-15T13:07:59.461 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 174.21465924899999, 2024-05-15T13:07:59.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19906881500000001, 2024-05-15T13:07:59.461 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:59.461 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:59.461 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:07:59.461 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:07:59.461 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:07:59.461 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 149552 2024-05-15T13:07:59.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.461 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:59.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:59.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.135101+0000", 2024-05-15T13:07:59.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:59.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.135101+0000", 2024-05-15T13:07:59.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:59.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.135102+0000", 2024-05-15T13:07:59.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1990000000000001e-06 2024-05-15T13:07:59.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:59.463 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.135107+0000", 2024-05-15T13:07:59.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8890000000000001e-06 2024-05-15T13:07:59.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:59.463 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.135118+0000", 2024-05-15T13:07:59.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1076e-05 2024-05-15T13:07:59.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:59.463 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.135124+0000", 2024-05-15T13:07:59.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8730000000000004e-06 2024-05-15T13:07:59.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.135166+0000", 2024-05-15T13:07:59.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2867000000000001e-05 2024-05-15T13:07:59.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:59.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.135258+0000", 2024-05-15T13:07:59.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2027999999999994e-05 2024-05-15T13:07:59.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,5", 2024-05-15T13:07:59.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.135331+0000", 2024-05-15T13:07:59.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2219999999999996e-05 2024-05-15T13:07:59.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.139763+0000", 2024-05-15T13:07:59.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0044323410000000002 2024-05-15T13:07:59.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:07:59.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.139851+0000", 2024-05-15T13:07:59.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7987999999999999e-05 2024-05-15T13:07:59.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.334066+0000", 2024-05-15T13:07:59.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19421459499999999 2024-05-15T13:07:59.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.466 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:59.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.334122+0000", 2024-05-15T13:07:59.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6569e-05 2024-05-15T13:07:59.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.466 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:59.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.334169+0000", 2024-05-15T13:07:59.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7169999999999997e-05 2024-05-15T13:07:59.466 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.466 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:59.466 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.467 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:149559 5.8 5:1eebcc77:::benchmark_data_smithi177_109991_object14368:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e453)", 2024-05-15T13:07:59.467 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:05:05.155099+0000", 2024-05-15T13:07:59.467 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 174.194660333, 2024-05-15T13:07:59.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16962189599999999, 2024-05-15T13:07:59.467 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:07:59.467 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:07:59.467 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:07:59.467 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:07:59.467 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:07:59.467 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 149559 2024-05-15T13:07:59.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.467 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:07:59.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.467 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:07:59.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.155099+0000", 2024-05-15T13:07:59.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:07:59.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.155099+0000", 2024-05-15T13:07:59.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:07:59.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:07:59.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.155101+0000", 2024-05-15T13:07:59.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7150000000000001e-06 2024-05-15T13:07:59.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:07:59.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.155107+0000", 2024-05-15T13:07:59.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4199999999999998e-06 2024-05-15T13:07:59.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:07:59.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.155121+0000", 2024-05-15T13:07:59.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4301000000000001e-05 2024-05-15T13:07:59.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:07:59.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.155129+0000", 2024-05-15T13:07:59.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.259e-06 2024-05-15T13:07:59.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:07:59.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.155173+0000", 2024-05-15T13:07:59.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4196000000000002e-05 2024-05-15T13:07:59.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:07:59.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.155265+0000", 2024-05-15T13:07:59.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1904999999999995e-05 2024-05-15T13:07:59.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,10", 2024-05-15T13:07:59.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.155338+0000", 2024-05-15T13:07:59.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2578999999999997e-05 2024-05-15T13:07:59.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:07:59.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.158716+0000", 2024-05-15T13:07:59.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0033776909999999999 2024-05-15T13:07:59.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.159119+0000", 2024-05-15T13:07:59.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0004037 2024-05-15T13:07:59.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.472 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:07:59.472 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.324639+0000", 2024-05-15T13:07:59.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.165519793 2024-05-15T13:07:59.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.472 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:07:59.472 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.324675+0000", 2024-05-15T13:07:59.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5652999999999999e-05 2024-05-15T13:07:59.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:07:59.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:07:59.472 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:07:59.472 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:05.324721+0000", 2024-05-15T13:07:59.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6684000000000002e-05 2024-05-15T13:07:59.473 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.473 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:59.473 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.473 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:07:59.473 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:07:59.473 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:07:59.473 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:07:59.505 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:07:59.524 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:07:59.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:07:59.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:07:59.531 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 5 16 15673 15657 195.695 148.5 0.00130793 0.00505673 2024-05-15T13:07:59.628 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:07:59.655 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:00.081 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-05-15T13:08:00.113 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:08:00.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:00.116 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:08:00.217 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:08:00.253 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:00.278 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:08:00.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:00.281 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:08:00.383 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:08:00.423 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:00.449 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:08:00.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:00.453 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:08:00.531 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 6 16 18204 18188 189.441 158.188 0.0084128 0.00527324 2024-05-15T13:08:00.554 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:08:00.587 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:08:00.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:00.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:08:00.802 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:08:00.841 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:00.869 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:08:00.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:00.873 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:08:00.974 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:08:01.003 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:01.023 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:08:01.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:01.026 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:08:01.117 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:01 smithi177 ceph-mon[44303]: pgmap v1365: 27 pgs: 27 active+clean; 716 MiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-05-15T13:08:01.127 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:08:01.185 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:01.225 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:08:01.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:01.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:08:01.330 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:08:01.365 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:01.390 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:08:01.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:01.394 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:08:01.496 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:08:01.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:01 smithi120 ceph-mon[43961]: pgmap v1365: 27 pgs: 27 active+clean; 716 MiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-05-15T13:08:01.531 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 7 16 20761 20745 185.207 159.812 0.0120547 0.00539343 2024-05-15T13:08:01.533 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:01.563 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:08:01.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:01.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:08:01.671 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:08:01.715 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:01.745 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:08:01.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:01.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:08:01.850 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:08:01.876 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:01.896 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:08:01.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:01.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:08:02.000 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:08:02.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:01 smithi027 ceph-mon[39795]: pgmap v1365: 27 pgs: 27 active+clean; 716 MiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-05-15T13:08:02.054 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:02.294 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:08:02.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:02.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:08:02.401 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:08:02.449 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:02.531 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 8 16 23111 23095 180.414 146.875 0.0034968 0.00553633 2024-05-15T13:08:02.587 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 10 1 2024-05-15T13:08:03.080 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:08:03.095 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:08:03.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:03.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:08:03.203 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:08:03.238 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:03.264 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:08:03.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:03.268 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:08:03.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:03 smithi027 ceph-mon[39795]: pgmap v1366: 27 pgs: 27 active+clean; 991 MiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-05-15T13:08:03.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:03 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:08:03.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:03 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:08:03.369 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:08:03.398 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:03 smithi177 ceph-mon[44303]: pgmap v1366: 27 pgs: 27 active+clean; 991 MiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-05-15T13:08:03.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:03 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:08:03.399 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:03 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:08:03.401 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:03.429 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:08:03.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:03.431 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:08:03.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:03 smithi120 ceph-mon[43961]: pgmap v1366: 27 pgs: 27 active+clean; 991 MiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-05-15T13:08:03.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:03 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:08:03.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:03 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:08:03.531 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 9 16 25605 25589 177.686 155.875 0.00249616 0.00562281 2024-05-15T13:08:03.532 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:08:03.574 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:03.604 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:08:03.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:03.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:08:03.709 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:08:03.746 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:03.769 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:08:03.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:03.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:08:03.873 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:08:03.907 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:03.931 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:08:03.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:03.934 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:08:04.035 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:08:04.086 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:04.137 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:04 smithi177 ceph-mon[44303]: osdmap e516: 12 total, 12 up, 11 in 2024-05-15T13:08:04.137 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:04 smithi027 ceph-mon[39795]: osdmap e516: 12 total, 12 up, 11 in 2024-05-15T13:08:04.325 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:08:04.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:04 smithi120 ceph-mon[43961]: osdmap e516: 12 total, 12 up, 11 in 2024-05-15T13:08:04.531 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 10 16 28166 28150 175.922 160.062 0.0122953 0.00567946 2024-05-15T13:08:04.576 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:08:05.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:05 smithi177 ceph-mon[44303]: pgmap v1368: 27 pgs: 27 active+clean; 1.3 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-05-15T13:08:05.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:05 smithi027 ceph-mon[39795]: pgmap v1368: 27 pgs: 27 active+clean; 1.3 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-05-15T13:08:05.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:05 smithi120 ceph-mon[43961]: pgmap v1368: 27 pgs: 27 active+clean; 1.3 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-05-15T13:08:05.531 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 11 16 30608 30592 173.803 152.625 0.00547321 0.0057492 2024-05-15T13:08:05.879 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:08:05.907 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:08:05.947 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:08:05.986 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:08:06.025 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:08:06.067 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:08:06.104 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:08:06.149 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:08:06.190 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:08:06.307 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:08:06.309 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:08:06.309 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:08:06.310 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:08:06.310 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:08:06.310 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:08:06.310 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:08:06.310 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:08:06.310 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:08:06.310 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:08:06.310 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:08:06.310 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:08:06.310 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:08:06.310 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:08:06.310 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:08:06.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:06 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1245845562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:08:06.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:06 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1245845562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:08:06.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:06 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1245845562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:08:06.531 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 12 16 32473 32457 169.032 116.562 0.00194118 0.0058357 2024-05-15T13:08:07.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:07 smithi177 ceph-mon[44303]: pgmap v1369: 27 pgs: 27 active+clean; 1.5 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 166 MiB/s wr, 2.65k op/s 2024-05-15T13:08:07.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:07 smithi027 ceph-mon[39795]: pgmap v1369: 27 pgs: 27 active+clean; 1.5 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 166 MiB/s wr, 2.65k op/s 2024-05-15T13:08:07.529 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:07 smithi120 ceph-mon[43961]: pgmap v1369: 27 pgs: 27 active+clean; 1.5 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 166 MiB/s wr, 2.65k op/s 2024-05-15T13:08:07.531 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 13 16 34145 34129 164.067 104.5 0.00244407 0.00608527 2024-05-15T13:08:08.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:08 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:08:08.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:08 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:08:08.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:08 smithi027 ceph-mon[39795]: osdmap e517: 12 total, 12 up, 11 in 2024-05-15T13:08:08.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:08 smithi027 ceph-mon[39795]: pgmap v1371: 27 pgs: 27 active+clean; 1.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-05-15T13:08:08.529 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:08 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:08:08.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:08 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:08:08.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:08 smithi120 ceph-mon[43961]: osdmap e517: 12 total, 12 up, 11 in 2024-05-15T13:08:08.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:08 smithi120 ceph-mon[43961]: pgmap v1371: 27 pgs: 27 active+clean; 1.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-05-15T13:08:08.532 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 14 16 35382 35366 157.87 77.3125 0.00716519 0.00633064 2024-05-15T13:08:08.667 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:08:08.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:08.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:08:08.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:08 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:08:08.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:08 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:08:08.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:08 smithi177 ceph-mon[44303]: osdmap e517: 12 total, 12 up, 11 in 2024-05-15T13:08:08.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:08 smithi177 ceph-mon[44303]: pgmap v1371: 27 pgs: 27 active+clean; 1.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-05-15T13:08:08.775 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:08:08.815 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:08.842 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:08:08.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:08.846 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:08:08.947 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:08:08.980 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:09.007 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:08:09.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:09.010 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:08:09.111 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:08:09.175 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:09.533 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 15 16 37634 37618 156.72 140.75 0.0095878 0.00637611 2024-05-15T13:08:10.533 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 16 16 39578 39562 154.518 121.5 0.00299446 0.00646865 2024-05-15T13:08:10.743 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:08:10.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:10.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:08:10.853 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:08:10.899 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:10.931 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:08:10.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:10.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:08:11.037 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:08:11.072 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:11.093 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:08:11.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:11.096 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:08:11.197 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:08:11.232 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:11 smithi120 ceph-mon[43961]: pgmap v1372: 27 pgs: 27 active+clean; 2.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-05-15T13:08:11.237 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:11.263 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:08:11.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:11.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:08:11.326 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:11 smithi027 ceph-mon[39795]: pgmap v1372: 27 pgs: 27 active+clean; 2.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-05-15T13:08:11.367 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:08:11.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:11 smithi177 ceph-mon[44303]: pgmap v1372: 27 pgs: 27 active+clean; 2.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-05-15T13:08:11.404 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:11.431 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:08:11.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:11.435 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:08:11.533 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 17 16 41697 41681 153.219 132.438 0.00331538 0.00652274 2024-05-15T13:08:11.536 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:08:11.569 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:11.594 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:08:11.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:11.597 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:08:11.698 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:08:11.760 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:11.835 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:08:11.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:11.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:08:11.944 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:08:11.977 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:11.995 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:08:11.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:11.998 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:08:12.099 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:08:12.134 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:12.154 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:08:12.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:12.157 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:08:12.258 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:08:12.284 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:12.302 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:08:12.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:12.305 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:08:12.406 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:08:12.452 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:12.489 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:08:12.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:12.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:08:12.533 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 18 16 43117 43101 149.637 88.75 0.00550603 0.00667872 2024-05-15T13:08:12.585 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:08:12.585 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:08:12.596 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:08:12.637 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:12.666 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:08:12.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:12.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:08:12.773 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:08:12.786 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:08:12.837 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:13.216 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:08:13.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:13.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:08:13.325 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:08:13.362 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:13.399 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:13 smithi027 ceph-mon[39795]: pgmap v1373: 27 pgs: 27 active+clean; 2.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 143 MiB/s wr, 2.28k op/s 2024-05-15T13:08:13.399 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:13 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4086107148' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T13:08:13.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:13 smithi177 ceph-mon[44303]: pgmap v1373: 27 pgs: 27 active+clean; 2.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 143 MiB/s wr, 2.28k op/s 2024-05-15T13:08:13.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:13 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4086107148' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T13:08:13.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:13 smithi120 ceph-mon[43961]: pgmap v1373: 27 pgs: 27 active+clean; 2.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 143 MiB/s wr, 2.28k op/s 2024-05-15T13:08:13.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:13 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4086107148' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T13:08:13.534 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 19 15 44872 44857 147.53 109.75 0.0142161 0.0067737 2024-05-15T13:08:13.842 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:08:13.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:13.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:08:13.890 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-05-15T13:08:13.952 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:08:13.984 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:13.991 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.10 to 1 (10000) 2024-05-15T13:08:14.008 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:08:14.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:14.012 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:08:14.113 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:08:14.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:14 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4086107148' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T13:08:14.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:14 smithi027 ceph-mon[39795]: osdmap e518: 12 total, 12 up, 11 in 2024-05-15T13:08:14.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:14 smithi027 ceph-mon[39795]: osdmap e519: 12 total, 12 up, 11 in 2024-05-15T13:08:14.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:14 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:08:14.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:14 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:08:14.175 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:14.212 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:08:14.212 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:08:14.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:14.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:08:14.320 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:08:14.355 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:14 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4086107148' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T13:08:14.355 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:14 smithi177 ceph-mon[44303]: osdmap e518: 12 total, 12 up, 11 in 2024-05-15T13:08:14.355 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:14 smithi177 ceph-mon[44303]: osdmap e519: 12 total, 12 up, 11 in 2024-05-15T13:08:14.355 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:14 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:08:14.355 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:14 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:08:14.357 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:14.382 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:08:14.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:14.385 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:08:14.486 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:08:14.521 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:14.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:14 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4086107148' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T13:08:14.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:14 smithi120 ceph-mon[43961]: osdmap e518: 12 total, 12 up, 11 in 2024-05-15T13:08:14.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:14 smithi120 ceph-mon[43961]: osdmap e519: 12 total, 12 up, 11 in 2024-05-15T13:08:14.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:14 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:08:14.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:14 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:08:14.534 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T13:08:14.546174+0000 min lat: 0.00114056 max lat: 0.299471 avg lat: 0.00683852 2024-05-15T13:08:14.534 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:08:14.534 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 20 16 46789 46773 146.139 119.75 0.0100145 0.00683852 2024-05-15T13:08:14.546 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:08:14.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:14.549 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:08:14.650 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:08:14.683 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:14.704 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 11 1 2024-05-15T13:08:14.705 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:08:14.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:14.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:08:14.716 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:08:14.808 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:08:14.878 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:14.950 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:08:14.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:14.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:08:15.054 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:08:15.092 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:15.122 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:08:15.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:15.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:08:15.227 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:08:15.260 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:15.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:15 smithi027 ceph-mon[39795]: pgmap v1376: 27 pgs: 27 active+clean; 2.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-05-15T13:08:15.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:15 smithi027 ceph-mon[39795]: osdmap e520: 12 total, 12 up, 11 in 2024-05-15T13:08:15.287 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:08:15.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:15.291 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:08:15.392 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:08:15.421 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:15.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:15 smithi177 ceph-mon[44303]: pgmap v1376: 27 pgs: 27 active+clean; 2.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-05-15T13:08:15.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:15 smithi177 ceph-mon[44303]: osdmap e520: 12 total, 12 up, 11 in 2024-05-15T13:08:15.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:15 smithi120 ceph-mon[43961]: pgmap v1376: 27 pgs: 27 active+clean; 2.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-05-15T13:08:15.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:15 smithi120 ceph-mon[43961]: osdmap e520: 12 total, 12 up, 11 in 2024-05-15T13:08:15.534 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 21 16 48312 48296 143.713 95.1875 0.00699717 0.0069541 2024-05-15T13:08:16.534 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 22 16 49887 49871 141.654 98.4375 0.00358294 0.00705503 2024-05-15T13:08:17.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:17 smithi177 ceph-mon[44303]: pgmap v1378: 27 pgs: 27 active+clean; 2.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-05-15T13:08:17.504 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:08:17.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:17.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:08:17.510 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:17 smithi027 ceph-mon[39795]: pgmap v1378: 27 pgs: 27 active+clean; 2.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-05-15T13:08:17.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:17 smithi120 ceph-mon[43961]: pgmap v1378: 27 pgs: 27 active+clean; 2.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-05-15T13:08:17.534 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 23 16 51484 51468 139.835 99.8125 0.0100938 0.00714605 2024-05-15T13:08:17.611 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:08:17.646 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:17.673 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:08:17.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:17.676 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:08:17.777 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:08:17.818 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:17.843 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:08:17.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:17.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:08:17.876 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:08:17.947 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:08:17.992 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:18.014 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:08:18.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:18.017 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:08:18.118 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:08:18.163 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:18.186 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:08:18.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:18.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:08:18.292 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:08:18.349 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:18.438 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:08:18.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:18.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:08:18.535 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 24 15 53267 53252 138.654 111.5 0.00668589 0.00718631 2024-05-15T13:08:18.548 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:08:18.582 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:18.610 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:08:18.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:18.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:08:18.716 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:08:18.761 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:18.789 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:08:18.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:18.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:08:18.855 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:08:18.893 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:08:18.920 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:18.938 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:08:18.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:18.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:08:19.042 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:08:19.076 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:19.105 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:08:19.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:19.109 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:08:19.210 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:08:19.252 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:19 smithi120 ceph-mon[43961]: pgmap v1379: 27 pgs: 27 active+clean; 2.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-05-15T13:08:19.252 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:19 smithi120 ceph-mon[43961]: osdmap e521: 12 total, 12 up, 11 in 2024-05-15T13:08:19.252 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:19 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:08:19.254 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:19.279 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:08:19.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:19.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:08:19.385 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:08:19.408 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:19 smithi177 ceph-mon[44303]: pgmap v1379: 27 pgs: 27 active+clean; 2.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-05-15T13:08:19.409 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:19 smithi177 ceph-mon[44303]: osdmap e521: 12 total, 12 up, 11 in 2024-05-15T13:08:19.409 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:19 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:08:19.410 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:19.413 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:19 smithi027 ceph-mon[39795]: pgmap v1379: 27 pgs: 27 active+clean; 2.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-05-15T13:08:19.413 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:19 smithi027 ceph-mon[39795]: osdmap e521: 12 total, 12 up, 11 in 2024-05-15T13:08:19.413 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:19 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:08:19.429 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:08:19.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:19.432 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:08:19.533 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:08:19.536 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 25 16 54449 54433 136.06 73.8125 0.0135489 0.00734519 2024-05-15T13:08:19.584 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:19.618 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:08:19.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:19.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:08:19.724 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:08:19.798 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:20.169 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:20 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:08:20.170 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:20 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:08:20.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:20 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:08:20.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:20 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:08:20.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:20 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:08:20.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:20 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:08:20.535 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 26 16 56103 56087 134.803 103.375 0.0637552 0.00740372 2024-05-15T13:08:21.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:21 smithi177 ceph-mon[44303]: pgmap v1381: 27 pgs: 27 active+clean; 3.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-05-15T13:08:21.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:21 smithi177 ceph-mon[44303]: osdmap e522: 12 total, 12 up, 11 in 2024-05-15T13:08:21.519 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:21 smithi027 ceph-mon[39795]: pgmap v1381: 27 pgs: 27 active+clean; 3.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-05-15T13:08:21.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:21 smithi027 ceph-mon[39795]: osdmap e522: 12 total, 12 up, 11 in 2024-05-15T13:08:21.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:21 smithi120 ceph-mon[43961]: pgmap v1381: 27 pgs: 27 active+clean; 3.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-05-15T13:08:21.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:21 smithi120 ceph-mon[43961]: osdmap e522: 12 total, 12 up, 11 in 2024-05-15T13:08:21.535 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 27 16 57721 57705 133.555 101.125 0.0104413 0.00748311 2024-05-15T13:08:22.536 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 28 16 59545 59529 132.851 114 0.00920443 0.00752289 2024-05-15T13:08:22.771 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:08:22.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:22.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:08:22.883 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:08:22.928 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:22.953 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:08:22.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:22.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:08:23.057 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:08:23.107 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:23.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:23 smithi027 ceph-mon[39795]: pgmap v1383: 27 pgs: 27 active+clean; 3.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-05-15T13:08:23.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:23 smithi120 ceph-mon[43961]: pgmap v1383: 27 pgs: 27 active+clean; 3.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-05-15T13:08:23.536 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 29 16 61104 61088 131.63 97.4375 0.00792314 0.00759227 2024-05-15T13:08:23.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:23 smithi177 ceph-mon[44303]: pgmap v1383: 27 pgs: 27 active+clean; 3.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-05-15T13:08:24.077 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:08:24.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:24.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:08:24.183 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:08:24.210 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:24.230 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:08:24.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:24.233 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:08:24.334 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:08:24.384 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:24.412 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:08:24.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:24.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:08:24.518 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:08:24.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:24 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:08:24.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:24 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:08:24.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:24 smithi027 ceph-mon[39795]: osdmap e523: 12 total, 12 up, 11 in 2024-05-15T13:08:24.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:24 smithi027 ceph-mon[39795]: osdmap e524: 12 total, 12 up, 11 in 2024-05-15T13:08:24.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:24 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:08:24.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:24 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:08:24.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:24 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:08:24.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:24 smithi027 ceph-mon[39795]: pgmap v1386: 27 pgs: 27 active+clean; 3.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-05-15T13:08:24.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:24 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:08:24.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:24 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:08:24.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:24 smithi120 ceph-mon[43961]: osdmap e523: 12 total, 12 up, 11 in 2024-05-15T13:08:24.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:24 smithi120 ceph-mon[43961]: osdmap e524: 12 total, 12 up, 11 in 2024-05-15T13:08:24.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:08:24.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:08:24.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:08:24.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:24 smithi120 ceph-mon[43961]: pgmap v1386: 27 pgs: 27 active+clean; 3.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-05-15T13:08:24.536 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 30 16 62605 62589 130.369 93.8125 0.013383 0.00766526 2024-05-15T13:08:24.553 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:24.579 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:08:24.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:24.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:08:24.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:24 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:08:24.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:24 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:08:24.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:24 smithi177 ceph-mon[44303]: osdmap e523: 12 total, 12 up, 11 in 2024-05-15T13:08:24.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:24 smithi177 ceph-mon[44303]: osdmap e524: 12 total, 12 up, 11 in 2024-05-15T13:08:24.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:24 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:08:24.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:24 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:08:24.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:24 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:08:24.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:24 smithi177 ceph-mon[44303]: pgmap v1386: 27 pgs: 27 active+clean; 3.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-05-15T13:08:24.685 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:08:24.720 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:24.743 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:08:24.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:24.748 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:08:24.849 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:08:25.063 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.11 to 1 (10000) 2024-05-15T13:08:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:08:25.536 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 31 16 63909 63893 128.793 81.5 0.00300831 0.00775908 2024-05-15T13:08:25.758 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:08:25.758 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:08:25.758 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:08:25.758 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:08:25.758 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:08:25.758 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:08:25.758 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:08:25.758 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:08:25.758 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:08:25.759 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:08:25.759 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:08:25.759 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:08:25.759 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:08:25.759 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:08:25.759 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:08:25.759 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:08:25.759 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:08:25.759 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:08:25.759 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:08:25.759 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:08:25.760 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:08:25.760 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:08:25.760 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:08:25.760 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:08:25.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:25 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:08:25.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:25 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:08:25.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:25 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/354615438' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:08:25.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:25 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:08:25.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:25 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:08:25.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:25 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1792881600' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T13:08:25.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:25 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4260194277' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:08:25.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:25 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:08:25.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:25 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:08:25.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:25 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:08:25.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:25 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/354615438' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:08:25.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:25 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:08:25.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:25 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:08:25.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:25 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1792881600' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T13:08:25.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:25 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4260194277' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:08:25.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:25 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:08:25.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:25 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:08:25.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:25 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:08:25.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:25 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/354615438' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:08:25.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:25 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:08:25.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:25 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:08:25.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:25 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1792881600' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T13:08:25.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:25 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4260194277' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:08:25.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:25 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:08:26.329 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:08:26.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:26.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:08:26.435 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:08:26.470 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:26.501 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:08:26.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:26.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:08:26.536 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 32 16 65457 65441 127.791 96.75 0.00444667 0.0078213 2024-05-15T13:08:26.609 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:08:26.653 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:26.685 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:08:26.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:26.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:08:26.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:26 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:08:26.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:26 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1792881600' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T13:08:26.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:26 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:08:26.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:26 smithi027 ceph-mon[39795]: osdmap e525: 12 total, 12 up, 11 in 2024-05-15T13:08:26.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:26 smithi027 ceph-mon[39795]: pgmap v1388: 27 pgs: 27 active+clean; 3.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-05-15T13:08:26.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:26 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:08:26.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:26 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1792881600' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T13:08:26.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:26 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:08:26.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:26 smithi120 ceph-mon[43961]: osdmap e525: 12 total, 12 up, 11 in 2024-05-15T13:08:26.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:26 smithi120 ceph-mon[43961]: pgmap v1388: 27 pgs: 27 active+clean; 3.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-05-15T13:08:26.792 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:08:26.840 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:26.869 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:08:26.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:26.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:08:26.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:26 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:08:26.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:26 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1792881600' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T13:08:26.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:26 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:08:26.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:26 smithi177 ceph-mon[44303]: osdmap e525: 12 total, 12 up, 11 in 2024-05-15T13:08:26.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:26 smithi177 ceph-mon[44303]: pgmap v1388: 27 pgs: 27 active+clean; 3.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-05-15T13:08:26.974 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:08:27.031 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:27.536 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 33 16 67074 67058 126.981 101.062 0.00780161 0.00787096 2024-05-15T13:08:27.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:27 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:08:27.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:27 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:08:27.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:27 smithi177 ceph-mon[44303]: osdmap e526: 12 total, 12 up, 11 in 2024-05-15T13:08:28.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:27 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:08:28.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:27 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:08:28.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:27 smithi027 ceph-mon[39795]: osdmap e526: 12 total, 12 up, 11 in 2024-05-15T13:08:28.029 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:27 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:08:28.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:27 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:08:28.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:27 smithi120 ceph-mon[43961]: osdmap e526: 12 total, 12 up, 11 in 2024-05-15T13:08:28.031 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:08:28.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:28.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:08:28.144 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:08:28.170 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-05-15T13:08:28.170 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:08:28.178 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:28.204 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:08:28.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:28.208 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:08:28.309 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:08:28.354 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:28.385 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:08:28.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:28.392 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:08:28.493 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:08:28.523 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:28.536 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 34 16 68501 68485 125.869 89.1875 0.0083516 0.0079409 2024-05-15T13:08:28.545 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:08:28.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:28.549 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:08:28.650 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:08:28.681 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:28.706 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:08:28.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:28.710 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:08:28.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:28 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:08:28.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:28 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:08:28.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:28 smithi027 ceph-mon[39795]: osdmap e527: 12 total, 12 up, 11 in 2024-05-15T13:08:28.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:28 smithi027 ceph-mon[39795]: pgmap v1391: 27 pgs: 27 active+clean; 3.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 85 MiB/s wr, 1.37k op/s 2024-05-15T13:08:28.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:28 smithi027 ceph-mon[39795]: osdmap e528: 12 total, 12 up, 11 in 2024-05-15T13:08:28.811 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:08:28.872 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:28.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:28 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:08:28.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:28 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:08:28.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:28 smithi177 ceph-mon[44303]: osdmap e527: 12 total, 12 up, 11 in 2024-05-15T13:08:28.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:28 smithi177 ceph-mon[44303]: pgmap v1391: 27 pgs: 27 active+clean; 3.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 85 MiB/s wr, 1.37k op/s 2024-05-15T13:08:28.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:28 smithi177 ceph-mon[44303]: osdmap e528: 12 total, 12 up, 11 in 2024-05-15T13:08:29.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:28 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:08:29.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:28 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:08:29.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:28 smithi120 ceph-mon[43961]: osdmap e527: 12 total, 12 up, 11 in 2024-05-15T13:08:29.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:28 smithi120 ceph-mon[43961]: pgmap v1391: 27 pgs: 27 active+clean; 3.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 85 MiB/s wr, 1.37k op/s 2024-05-15T13:08:29.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:28 smithi120 ceph-mon[43961]: osdmap e528: 12 total, 12 up, 11 in 2024-05-15T13:08:29.412 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:08:29.412 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T13:08:29.412 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-05-15T13:08:29.412 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-05-15T13:08:29.412 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:08:29.537 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 35 16 69766 69750 124.532 79.0625 0.0150251 0.00802571 2024-05-15T13:08:29.625 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:08:29.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:29.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:08:29.740 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:08:29.783 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:29.811 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:08:29.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:29.818 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:08:29.919 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:08:29.962 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:30.045 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:08:30.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:30.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:08:30.149 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:08:30.185 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:30.223 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:08:30.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:30.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:08:30.329 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:08:30.360 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:30.387 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:08:30.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:30.389 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:08:30.491 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:08:30.537 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 36 16 71308 71292 123.75 96.375 0.0175289 0.00807718 2024-05-15T13:08:30.545 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:30.586 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:08:30.590 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-05-15T13:08:30.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:30.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:08:30.600 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:08:30.692 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:08:30.726 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:30.753 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:08:30.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:30.755 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:08:30.849 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:08:30.857 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:08:30.885 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:30.908 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:08:30.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:30.913 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:08:31.014 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:08:31.044 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:31.068 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:08:31.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:31.072 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:08:31.174 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:08:31.175 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:31 smithi027 ceph-mon[39795]: pgmap v1393: 27 pgs: 27 active+clean; 4.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-05-15T13:08:31.206 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:31.230 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:08:31.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:31.234 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:08:31.336 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:08:31.365 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:31 smithi177 ceph-mon[44303]: pgmap v1393: 27 pgs: 27 active+clean; 4.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-05-15T13:08:31.367 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:31.399 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:08:31.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:31.404 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:08:31.505 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:08:31.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:31 smithi120 ceph-mon[43961]: pgmap v1393: 27 pgs: 27 active+clean; 4.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-05-15T13:08:31.537 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 37 16 72621 72605 122.623 82.0625 0.00851849 0.00815134 2024-05-15T13:08:31.546 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:31.574 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:08:31.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:31.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:08:31.680 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:08:31.721 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:31.757 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:08:31.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:31.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:08:31.864 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:08:31.893 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:31.920 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:08:31.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:31.922 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:08:32.023 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:08:32.086 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:32.102 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:08:32.258 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:08:32.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:32.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:08:32.371 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:08:32.431 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:32.537 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 38 16 74005 73989 121.672 86.5 0.0057317 0.0082146 2024-05-15T13:08:32.863 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:08:33.414 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:08:33.419 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:33 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:08:33.419 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:33 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:08:33.419 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:33 smithi027 ceph-mon[39795]: pgmap v1394: 27 pgs: 27 active+clean; 4.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-05-15T13:08:33.419 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:33 smithi027 ceph-mon[39795]: osdmap e529: 12 total, 12 up, 11 in 2024-05-15T13:08:33.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:33.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:08:33.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:33 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:08:33.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:33 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:08:33.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:33 smithi177 ceph-mon[44303]: pgmap v1394: 27 pgs: 27 active+clean; 4.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-05-15T13:08:33.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:33 smithi177 ceph-mon[44303]: osdmap e529: 12 total, 12 up, 11 in 2024-05-15T13:08:33.521 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:08:33.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:33 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:08:33.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:33 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:08:33.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:33 smithi120 ceph-mon[43961]: pgmap v1394: 27 pgs: 27 active+clean; 4.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-05-15T13:08:33.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:33 smithi120 ceph-mon[43961]: osdmap e529: 12 total, 12 up, 11 in 2024-05-15T13:08:33.537 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 39 16 75391 75375 120.773 86.625 0.00599738 0.00827584 2024-05-15T13:08:33.583 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:34.537 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T13:08:34.549020+0000 min lat: 0.00114056 max lat: 0.299471 avg lat: 0.0083493 2024-05-15T13:08:34.538 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:08:34.538 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 40 16 76639 76623 119.704 78 0.00783356 0.0083493 2024-05-15T13:08:34.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:34 smithi177 ceph-mon[44303]: osdmap e530: 12 total, 12 up, 11 in 2024-05-15T13:08:34.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:34 smithi177 ceph-mon[44303]: pgmap v1397: 27 pgs: 27 active+clean; 4.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-05-15T13:08:34.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:34 smithi027 ceph-mon[39795]: osdmap e530: 12 total, 12 up, 11 in 2024-05-15T13:08:34.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:34 smithi027 ceph-mon[39795]: pgmap v1397: 27 pgs: 27 active+clean; 4.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-05-15T13:08:34.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:34 smithi120 ceph-mon[43961]: osdmap e530: 12 total, 12 up, 11 in 2024-05-15T13:08:34.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:34 smithi120 ceph-mon[43961]: pgmap v1397: 27 pgs: 27 active+clean; 4.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-05-15T13:08:34.881 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:08:34.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:34.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:08:34.986 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:08:35.006 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:08:35.017 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:35.048 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:08:35.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:35.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:08:35.153 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:08:35.184 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:08:35.185 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:35.193 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:08:35.210 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:08:35.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:35.213 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:08:35.315 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:08:35.354 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:35.384 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:08:35.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:35.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:08:35.490 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:08:35.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:35 smithi027 ceph-mon[39795]: from='client.15873 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:08:35.537 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 41 16 77932 77916 118.755 80.8125 0.00628151 0.00841664 2024-05-15T13:08:35.549 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:35.593 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:08:35.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:35.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:08:35.603 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:08:35.699 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:08:35.726 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:08:35.726 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T13:08:35.726 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T13:08:35.726 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:08:35.726 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.726 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:21009 5.3 5:c0e5f930:::benchmark_data_smithi177_109991_object21008:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:08:35.726 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:27.986822+0000", 2024-05-15T13:08:35.727 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 307.74795206599998, 2024-05-15T13:08:35.727 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20485505300000001, 2024-05-15T13:08:35.727 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:08:35.727 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:08:35.727 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:08:35.727 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:08:35.727 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:08:35.727 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 21009 2024-05-15T13:08:35.727 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.727 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:08:35.727 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.727 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:08:35.727 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.986822+0000", 2024-05-15T13:08:35.728 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.728 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.728 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.728 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:08:35.728 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.986822+0000", 2024-05-15T13:08:35.728 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.728 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.728 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.728 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:08:35.728 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.986822+0000", 2024-05-15T13:08:35.728 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9500000000000001e-07 2024-05-15T13:08:35.728 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.728 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.729 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:08:35.729 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.986941+0000", 2024-05-15T13:08:35.729 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000118968 2024-05-15T13:08:35.729 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.729 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.729 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:08:35.729 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.986953+0000", 2024-05-15T13:08:35.729 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1151999999999999e-05 2024-05-15T13:08:35.729 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.729 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.729 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:08:35.729 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.986962+0000", 2024-05-15T13:08:35.729 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1710000000000008e-06 2024-05-15T13:08:35.729 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.730 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.730 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:08:35.730 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.987015+0000", 2024-05-15T13:08:35.730 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3193999999999999e-05 2024-05-15T13:08:35.730 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.730 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.730 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:08:35.730 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.987096+0000", 2024-05-15T13:08:35.730 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0620999999999999e-05 2024-05-15T13:08:35.730 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.730 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.730 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,6", 2024-05-15T13:08:35.730 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.987226+0000", 2024-05-15T13:08:35.731 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013094699999999999 2024-05-15T13:08:35.731 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.731 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.731 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:08:35.731 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.988235+0000", 2024-05-15T13:08:35.731 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010089330000000001 2024-05-15T13:08:35.731 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.731 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.731 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.731 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.990458+0000", 2024-05-15T13:08:35.731 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0022220970000000001 2024-05-15T13:08:35.731 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.731 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.732 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.732 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.191460+0000", 2024-05-15T13:08:35.732 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20100295000000001 2024-05-15T13:08:35.732 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.732 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.732 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:08:35.732 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.191582+0000", 2024-05-15T13:08:35.732 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000121118 2024-05-15T13:08:35.732 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.732 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.732 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:08:35.732 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:28.191677+0000", 2024-05-15T13:08:35.732 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5106999999999996e-05 2024-05-15T13:08:35.732 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.733 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:08:35.733 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.733 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.733 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.733 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.15540.0:149520 5.2 e453/418 5:4497d6cf:::benchmark_data_smithi177_109991_object14329:head v 453'9341, mlcod=453'9339)", 2024-05-15T13:08:35.733 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:05:05.104256+0000", 2024-05-15T13:08:35.733 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 210.630517672, 2024-05-15T13:08:35.733 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22089091799999999, 2024-05-15T13:08:35.733 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:08:35.733 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:08:35.733 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:08:35.733 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.733 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:08:35.733 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.104256+0000", 2024-05-15T13:08:35.734 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.734 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.734 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.734 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:08:35.734 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.104256+0000", 2024-05-15T13:08:35.734 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.734 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.734 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.734 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:08:35.734 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.104257+0000", 2024-05-15T13:08:35.734 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0500000000000003e-07 2024-05-15T13:08:35.734 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.734 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.735 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:08:35.735 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.104266+0000", 2024-05-15T13:08:35.735 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6350000000000001e-06 2024-05-15T13:08:35.735 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.735 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.735 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:08:35.735 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.104276+0000", 2024-05-15T13:08:35.735 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7499999999999998e-06 2024-05-15T13:08:35.735 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.735 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.735 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:08:35.735 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.104284+0000", 2024-05-15T13:08:35.735 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5730000000000002e-06 2024-05-15T13:08:35.736 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.736 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.736 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:08:35.736 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.110158+0000", 2024-05-15T13:08:35.736 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0058744640000000002 2024-05-15T13:08:35.736 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.736 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.736 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:08:35.736 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.110178+0000", 2024-05-15T13:08:35.736 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9381e-05 2024-05-15T13:08:35.736 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.736 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.736 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:08:35.736 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.325096+0000", 2024-05-15T13:08:35.736 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.21491801499999999 2024-05-15T13:08:35.737 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.737 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.737 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:08:35.737 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.325147+0000", 2024-05-15T13:08:35.737 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1394999999999998e-05 2024-05-15T13:08:35.737 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.737 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:08:35.737 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.737 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.737 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.737 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.15540.0:149526 5.2 e453/418 5:4a7b51f4:::benchmark_data_smithi177_109991_object14335:head v 453'9342, mlcod=453'9339)", 2024-05-15T13:08:35.737 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:05:05.108081+0000", 2024-05-15T13:08:35.737 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 210.62669281300001, 2024-05-15T13:08:35.737 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22160389799999999, 2024-05-15T13:08:35.738 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:08:35.738 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:08:35.738 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:08:35.738 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.738 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:08:35.738 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108081+0000", 2024-05-15T13:08:35.738 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.738 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.738 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.738 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:08:35.738 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108081+0000", 2024-05-15T13:08:35.738 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.738 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.739 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.739 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:08:35.739 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108082+0000", 2024-05-15T13:08:35.739 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1100000000000004e-07 2024-05-15T13:08:35.739 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.739 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.739 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:08:35.739 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108092+0000", 2024-05-15T13:08:35.739 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0679999999999999e-05 2024-05-15T13:08:35.739 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.739 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.739 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:08:35.739 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108105+0000", 2024-05-15T13:08:35.739 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2918999999999999e-05 2024-05-15T13:08:35.739 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.740 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.740 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:08:35.740 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108120+0000", 2024-05-15T13:08:35.740 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4184e-05 2024-05-15T13:08:35.740 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.740 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.740 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:08:35.740 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.320605+0000", 2024-05-15T13:08:35.740 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.212485062 2024-05-15T13:08:35.740 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.740 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.740 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:08:35.740 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.320626+0000", 2024-05-15T13:08:35.740 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1594999999999999e-05 2024-05-15T13:08:35.741 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.741 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.741 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:08:35.741 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.329617+0000", 2024-05-15T13:08:35.741 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0089910159999999992 2024-05-15T13:08:35.741 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.741 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.741 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:08:35.741 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.329685+0000", 2024-05-15T13:08:35.741 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7531000000000001e-05 2024-05-15T13:08:35.741 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.741 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:08:35.741 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.741 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.742 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.742 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.15540.0:149521 5.d e453/418 5:bfc1d209:::benchmark_data_smithi177_109991_object14330:head v 453'9514, mlcod=453'9512)", 2024-05-15T13:08:35.742 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:05:05.108380+0000", 2024-05-15T13:08:35.742 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 210.62639391299999, 2024-05-15T13:08:35.742 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.2077521, 2024-05-15T13:08:35.742 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:08:35.742 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:08:35.742 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:08:35.742 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.742 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:08:35.742 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108380+0000", 2024-05-15T13:08:35.742 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.742 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.743 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.743 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:08:35.743 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108380+0000", 2024-05-15T13:08:35.743 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.743 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.743 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.743 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:08:35.743 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108380+0000", 2024-05-15T13:08:35.743 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0200000000000002e-07 2024-05-15T13:08:35.743 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.743 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.743 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:08:35.743 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108387+0000", 2024-05-15T13:08:35.744 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3269999999999997e-06 2024-05-15T13:08:35.744 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.744 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.744 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:08:35.744 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108397+0000", 2024-05-15T13:08:35.744 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0155e-05 2024-05-15T13:08:35.744 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.744 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.744 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:08:35.744 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108403+0000", 2024-05-15T13:08:35.744 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4989999999999999e-06 2024-05-15T13:08:35.744 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.744 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.744 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:08:35.745 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108513+0000", 2024-05-15T13:08:35.745 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010940199999999999 2024-05-15T13:08:35.745 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.745 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.745 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:08:35.745 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.108526+0000", 2024-05-15T13:08:35.745 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3169e-05 2024-05-15T13:08:35.745 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.745 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.745 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:08:35.745 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.316073+0000", 2024-05-15T13:08:35.745 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20754674300000001 2024-05-15T13:08:35.745 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.745 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.746 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:08:35.746 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.316132+0000", 2024-05-15T13:08:35.746 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9203000000000002e-05 2024-05-15T13:08:35.746 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.746 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:08:35.746 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.746 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.746 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.746 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.15540.0:149532 5.e e453/418 5:7f4b648b:::benchmark_data_smithi177_109991_object14341:head v 453'9374, mlcod=453'9372)", 2024-05-15T13:08:35.746 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:05:05.112260+0000", 2024-05-15T13:08:35.746 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 210.62251416000001, 2024-05-15T13:08:35.746 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.21301437000000001, 2024-05-15T13:08:35.746 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:08:35.747 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:08:35.747 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:08:35.747 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.747 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:08:35.747 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.112260+0000", 2024-05-15T13:08:35.747 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.747 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.747 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.747 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:08:35.747 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.112260+0000", 2024-05-15T13:08:35.747 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.747 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.747 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.747 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:08:35.748 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.112260+0000", 2024-05-15T13:08:35.748 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0500000000000003e-07 2024-05-15T13:08:35.748 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.748 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.748 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:08:35.748 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.112267+0000", 2024-05-15T13:08:35.748 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4200000000000004e-06 2024-05-15T13:08:35.748 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.748 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.748 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:08:35.748 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.316035+0000", 2024-05-15T13:08:35.748 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20376809400000001 2024-05-15T13:08:35.748 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.749 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.749 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:08:35.749 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.316051+0000", 2024-05-15T13:08:35.749 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6132000000000001e-05 2024-05-15T13:08:35.749 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.749 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.749 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:08:35.749 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.316125+0000", 2024-05-15T13:08:35.749 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3817000000000002e-05 2024-05-15T13:08:35.749 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.749 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.749 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:08:35.749 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.316160+0000", 2024-05-15T13:08:35.749 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5612999999999999e-05 2024-05-15T13:08:35.749 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.750 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.750 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:08:35.750 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.325231+0000", 2024-05-15T13:08:35.750 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090705000000000004 2024-05-15T13:08:35.750 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.750 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.750 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:08:35.750 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.325274+0000", 2024-05-15T13:08:35.750 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3089e-05 2024-05-15T13:08:35.750 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.750 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:08:35.750 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.750 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.750 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.751 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:149533 5.3 5:c8312cf3:::benchmark_data_smithi177_109991_object14342:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e453)", 2024-05-15T13:08:35.751 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:05:05.117039+0000", 2024-05-15T13:08:35.751 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 210.61773441400001, 2024-05-15T13:08:35.751 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.21683490499999999, 2024-05-15T13:08:35.751 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:08:35.751 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:08:35.751 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:08:35.751 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:08:35.751 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:08:35.751 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 149533 2024-05-15T13:08:35.751 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.751 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:08:35.751 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.751 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:08:35.752 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.117039+0000", 2024-05-15T13:08:35.752 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.752 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.752 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.752 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:08:35.752 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.117039+0000", 2024-05-15T13:08:35.752 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.752 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.752 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.752 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:08:35.752 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.117041+0000", 2024-05-15T13:08:35.752 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.269e-06 2024-05-15T13:08:35.752 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.752 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.753 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:08:35.753 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.117047+0000", 2024-05-15T13:08:35.753 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1670000000000004e-06 2024-05-15T13:08:35.753 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.753 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.753 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:08:35.753 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.316070+0000", 2024-05-15T13:08:35.753 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19902281999999999 2024-05-15T13:08:35.753 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.753 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.753 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:08:35.753 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.316083+0000", 2024-05-15T13:08:35.753 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3030999999999999e-05 2024-05-15T13:08:35.753 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.754 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.754 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:08:35.754 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.316152+0000", 2024-05-15T13:08:35.754 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9429000000000002e-05 2024-05-15T13:08:35.754 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.754 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.754 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:08:35.754 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.316292+0000", 2024-05-15T13:08:35.754 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014007099999999999 2024-05-15T13:08:35.754 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.754 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.754 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,6", 2024-05-15T13:08:35.755 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.316405+0000", 2024-05-15T13:08:35.755 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000112821 2024-05-15T13:08:35.755 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.755 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.755 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.755 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.325489+0000", 2024-05-15T13:08:35.755 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090839400000000004 2024-05-15T13:08:35.755 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.755 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.755 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:08:35.755 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.325508+0000", 2024-05-15T13:08:35.755 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9635000000000001e-05 2024-05-15T13:08:35.755 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.755 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.755 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.756 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.333780+0000", 2024-05-15T13:08:35.756 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0082712500000000008 2024-05-15T13:08:35.756 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.756 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.756 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:08:35.756 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.333832+0000", 2024-05-15T13:08:35.756 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2488999999999998e-05 2024-05-15T13:08:35.756 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.756 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.756 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:08:35.756 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:05.333874+0000", 2024-05-15T13:08:35.756 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1983000000000001e-05 2024-05-15T13:08:35.756 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.756 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:08:35.757 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.757 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.757 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.757 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:159818 5.3 5:c58ab5c4:::benchmark_data_smithi177_109991_object24627:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-05-15T13:08:35.757 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:05:16.256134+0000", 2024-05-15T13:08:35.757 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 199.47863933900001, 2024-05-15T13:08:35.757 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22852134499999999, 2024-05-15T13:08:35.757 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:08:35.757 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:08:35.757 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:08:35.757 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:08:35.758 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:08:35.758 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 159818 2024-05-15T13:08:35.758 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.758 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:08:35.758 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.758 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:08:35.758 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.256134+0000", 2024-05-15T13:08:35.758 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.758 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.758 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.758 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:08:35.758 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.256134+0000", 2024-05-15T13:08:35.758 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.759 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.759 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.759 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:08:35.759 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.256136+0000", 2024-05-15T13:08:35.759 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.561e-06 2024-05-15T13:08:35.759 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.759 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.759 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:08:35.759 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.256142+0000", 2024-05-15T13:08:35.759 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.404e-06 2024-05-15T13:08:35.759 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.759 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.759 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:08:35.759 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.256155+0000", 2024-05-15T13:08:35.760 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2519e-05 2024-05-15T13:08:35.760 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.760 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.760 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:08:35.760 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.256165+0000", 2024-05-15T13:08:35.760 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0105000000000001e-05 2024-05-15T13:08:35.760 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.760 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.760 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:08:35.760 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.256230+0000", 2024-05-15T13:08:35.760 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5029000000000004e-05 2024-05-15T13:08:35.760 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.760 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.760 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:08:35.761 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.256373+0000", 2024-05-15T13:08:35.761 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000142628 2024-05-15T13:08:35.761 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.761 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.761 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,6", 2024-05-15T13:08:35.761 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.257159+0000", 2024-05-15T13:08:35.761 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00078688100000000002 2024-05-15T13:08:35.761 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.761 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.761 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.761 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.263723+0000", 2024-05-15T13:08:35.761 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0065636059999999996 2024-05-15T13:08:35.761 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.762 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.762 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:08:35.762 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.263776+0000", 2024-05-15T13:08:35.762 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2667000000000002e-05 2024-05-15T13:08:35.762 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.762 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.762 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.762 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.484589+0000", 2024-05-15T13:08:35.762 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22081320900000001 2024-05-15T13:08:35.762 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.762 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.762 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:08:35.762 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.484625+0000", 2024-05-15T13:08:35.762 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6086e-05 2024-05-15T13:08:35.763 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.763 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.763 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:08:35.763 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.484656+0000", 2024-05-15T13:08:35.763 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.065e-05 2024-05-15T13:08:35.763 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.763 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:08:35.763 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.763 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.763 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.763 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:159828 5.b 5:da0c93dc:::benchmark_data_smithi177_109991_object24637:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-05-15T13:08:35.764 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:05:16.269198+0000", 2024-05-15T13:08:35.764 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 199.465575204, 2024-05-15T13:08:35.764 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.209967765, 2024-05-15T13:08:35.764 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:08:35.764 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:08:35.764 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:08:35.764 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:08:35.764 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:08:35.764 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 159828 2024-05-15T13:08:35.764 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.764 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:08:35.765 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.765 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:08:35.765 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.269198+0000", 2024-05-15T13:08:35.765 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.765 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.765 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.765 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:08:35.765 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.269198+0000", 2024-05-15T13:08:35.765 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.765 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.765 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.765 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:08:35.765 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.269200+0000", 2024-05-15T13:08:35.766 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9149999999999999e-06 2024-05-15T13:08:35.766 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.766 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.766 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:08:35.766 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.269211+0000", 2024-05-15T13:08:35.766 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1004e-05 2024-05-15T13:08:35.766 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.766 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.766 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:08:35.766 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.269223+0000", 2024-05-15T13:08:35.767 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2087e-05 2024-05-15T13:08:35.767 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.767 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.767 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:08:35.767 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.269232+0000", 2024-05-15T13:08:35.767 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.568e-06 2024-05-15T13:08:35.767 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.767 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.767 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:08:35.767 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.270491+0000", 2024-05-15T13:08:35.767 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001258784 2024-05-15T13:08:35.767 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.767 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.768 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:08:35.768 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.270611+0000", 2024-05-15T13:08:35.768 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120034 2024-05-15T13:08:35.768 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.768 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.768 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:08:35.768 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.270711+0000", 2024-05-15T13:08:35.768 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010027 2024-05-15T13:08:35.768 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.768 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.768 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:08:35.768 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.283613+0000", 2024-05-15T13:08:35.768 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012902238 2024-05-15T13:08:35.768 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.769 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.769 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.769 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.284947+0000", 2024-05-15T13:08:35.769 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0013337589999999999 2024-05-15T13:08:35.769 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.769 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.769 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.769 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.479065+0000", 2024-05-15T13:08:35.769 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.194117703 2024-05-15T13:08:35.769 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.769 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.769 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:08:35.769 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.479106+0000", 2024-05-15T13:08:35.769 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0874999999999997e-05 2024-05-15T13:08:35.770 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.770 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.770 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:08:35.770 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.479166+0000", 2024-05-15T13:08:35.770 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0528000000000001e-05 2024-05-15T13:08:35.770 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.770 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:08:35.770 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.770 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.770 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.770 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:159831 5.b 5:d09fa99a:::benchmark_data_smithi177_109991_object24640:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-05-15T13:08:35.770 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:05:16.269756+0000", 2024-05-15T13:08:35.770 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 199.46501739000001, 2024-05-15T13:08:35.771 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.21775823, 2024-05-15T13:08:35.771 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:08:35.771 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:08:35.771 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:08:35.771 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:08:35.771 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:08:35.771 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 159831 2024-05-15T13:08:35.771 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.771 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:08:35.771 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.771 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:08:35.771 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.269756+0000", 2024-05-15T13:08:35.771 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.771 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.771 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.772 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:08:35.772 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.269756+0000", 2024-05-15T13:08:35.772 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.772 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.772 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.772 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:08:35.772 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.269757+0000", 2024-05-15T13:08:35.772 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8000000000000005e-07 2024-05-15T13:08:35.772 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.772 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.772 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:08:35.772 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.269761+0000", 2024-05-15T13:08:35.772 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1659999999999996e-06 2024-05-15T13:08:35.772 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.773 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.773 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:08:35.773 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.269771+0000", 2024-05-15T13:08:35.773 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4669999999999993e-06 2024-05-15T13:08:35.773 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.773 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.773 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:08:35.773 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.269777+0000", 2024-05-15T13:08:35.773 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6950000000000003e-06 2024-05-15T13:08:35.773 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.773 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.773 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:08:35.773 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.275136+0000", 2024-05-15T13:08:35.773 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.005358163 2024-05-15T13:08:35.774 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.774 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.774 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:08:35.774 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.275246+0000", 2024-05-15T13:08:35.774 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011052699999999999 2024-05-15T13:08:35.774 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.774 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.774 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:08:35.774 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.275350+0000", 2024-05-15T13:08:35.774 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000104287 2024-05-15T13:08:35.774 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.774 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.774 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:08:35.774 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.283618+0000", 2024-05-15T13:08:35.775 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0082679369999999995 2024-05-15T13:08:35.775 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.775 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.775 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.775 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.284982+0000", 2024-05-15T13:08:35.775 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001363759 2024-05-15T13:08:35.775 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.775 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.775 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.775 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.487408+0000", 2024-05-15T13:08:35.775 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.202426047 2024-05-15T13:08:35.775 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.775 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.775 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:08:35.776 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.487467+0000", 2024-05-15T13:08:35.776 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8964e-05 2024-05-15T13:08:35.776 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.776 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.776 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:08:35.776 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.487515+0000", 2024-05-15T13:08:35.776 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7438000000000002e-05 2024-05-15T13:08:35.776 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.776 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:08:35.776 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.776 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.776 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.776 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:160103 5.b 5:d13eec40:::benchmark_data_smithi177_109991_object24912:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-05-15T13:08:35.777 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:05:16.676296+0000", 2024-05-15T13:08:35.777 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 199.05847724399999, 2024-05-15T13:08:35.777 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.21115857399999999, 2024-05-15T13:08:35.777 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:08:35.777 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:08:35.777 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:08:35.777 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:08:35.777 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:08:35.777 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 160103 2024-05-15T13:08:35.777 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.777 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:08:35.777 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.778 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:08:35.778 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.676296+0000", 2024-05-15T13:08:35.778 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.778 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.778 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.778 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:08:35.778 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.676296+0000", 2024-05-15T13:08:35.778 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.778 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.778 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.778 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:08:35.778 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.676298+0000", 2024-05-15T13:08:35.778 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4109999999999999e-06 2024-05-15T13:08:35.778 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.779 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.779 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:08:35.779 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.676307+0000", 2024-05-15T13:08:35.779 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6899999999999998e-06 2024-05-15T13:08:35.779 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.779 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.779 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:08:35.779 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.676318+0000", 2024-05-15T13:08:35.779 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1073999999999999e-05 2024-05-15T13:08:35.779 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.779 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.779 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:08:35.779 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.676329+0000", 2024-05-15T13:08:35.779 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0937e-05 2024-05-15T13:08:35.780 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.780 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.780 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:08:35.780 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.676397+0000", 2024-05-15T13:08:35.780 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8607000000000002e-05 2024-05-15T13:08:35.780 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.780 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.780 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:08:35.780 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.676503+0000", 2024-05-15T13:08:35.780 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010546499999999999 2024-05-15T13:08:35.780 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.780 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.780 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:08:35.780 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.676571+0000", 2024-05-15T13:08:35.781 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8742e-05 2024-05-15T13:08:35.781 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.781 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.781 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:08:35.781 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.685914+0000", 2024-05-15T13:08:35.781 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0093426780000000001 2024-05-15T13:08:35.781 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.781 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.781 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.781 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.688304+0000", 2024-05-15T13:08:35.781 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002390367 2024-05-15T13:08:35.781 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.781 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.782 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.782 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.887338+0000", 2024-05-15T13:08:35.782 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19903394699999999 2024-05-15T13:08:35.782 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.782 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.782 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:08:35.782 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.887423+0000", 2024-05-15T13:08:35.782 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4240999999999995e-05 2024-05-15T13:08:35.782 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.782 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.782 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:08:35.782 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.887455+0000", 2024-05-15T13:08:35.782 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2415000000000001e-05 2024-05-15T13:08:35.782 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.783 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:08:35.783 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.783 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.783 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.783 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:160105 5.b 5:db37bac3:::benchmark_data_smithi177_109991_object24914:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-05-15T13:08:35.783 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:05:16.676355+0000", 2024-05-15T13:08:35.783 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 199.05841917999999, 2024-05-15T13:08:35.783 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.21326201, 2024-05-15T13:08:35.783 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:08:35.783 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:08:35.783 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:08:35.783 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:08:35.783 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:08:35.784 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 160105 2024-05-15T13:08:35.784 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.784 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:08:35.784 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.784 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:08:35.784 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.676355+0000", 2024-05-15T13:08:35.784 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.784 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.784 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.784 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:08:35.784 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.676355+0000", 2024-05-15T13:08:35.784 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.784 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.785 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.785 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:08:35.785 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.676355+0000", 2024-05-15T13:08:35.785 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1399999999999997e-07 2024-05-15T13:08:35.785 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.785 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.785 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:08:35.785 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.676359+0000", 2024-05-15T13:08:35.785 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.619e-06 2024-05-15T13:08:35.785 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.785 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.785 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:08:35.785 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.676366+0000", 2024-05-15T13:08:35.786 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7769999999999997e-06 2024-05-15T13:08:35.786 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.786 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.786 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:08:35.786 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.676372+0000", 2024-05-15T13:08:35.786 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.353e-06 2024-05-15T13:08:35.786 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.786 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.786 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:08:35.786 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.681096+0000", 2024-05-15T13:08:35.786 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0047244619999999996 2024-05-15T13:08:35.786 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.787 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.787 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:08:35.787 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.681221+0000", 2024-05-15T13:08:35.787 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000124752 2024-05-15T13:08:35.787 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.787 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.787 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:08:35.787 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.681327+0000", 2024-05-15T13:08:35.787 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000105999 2024-05-15T13:08:35.787 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.787 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.787 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:08:35.787 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.686061+0000", 2024-05-15T13:08:35.788 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0047340139999999999 2024-05-15T13:08:35.788 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.788 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.788 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.788 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.693609+0000", 2024-05-15T13:08:35.788 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0075476969999999999 2024-05-15T13:08:35.788 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.788 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.788 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.788 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.889556+0000", 2024-05-15T13:08:35.788 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.195947393 2024-05-15T13:08:35.788 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.788 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.789 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:08:35.789 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.889590+0000", 2024-05-15T13:08:35.789 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3875999999999999e-05 2024-05-15T13:08:35.789 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.789 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.789 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:08:35.790 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.889617+0000", 2024-05-15T13:08:35.790 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6454000000000001e-05 2024-05-15T13:08:35.790 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.790 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:08:35.790 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.790 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.790 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.790 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:160171 5.3 5:c80d6d64:::benchmark_data_smithi177_109991_object24980:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-05-15T13:08:35.790 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:05:16.745688+0000", 2024-05-15T13:08:35.790 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 198.98908558700001, 2024-05-15T13:08:35.790 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19949794900000001, 2024-05-15T13:08:35.790 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:08:35.791 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:08:35.791 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:08:35.791 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:08:35.791 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:08:35.791 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 160171 2024-05-15T13:08:35.791 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.791 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:08:35.791 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.791 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:08:35.791 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.745688+0000", 2024-05-15T13:08:35.791 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.791 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.791 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.792 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:08:35.792 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.745688+0000", 2024-05-15T13:08:35.792 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.792 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.792 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.792 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:08:35.792 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.745689+0000", 2024-05-15T13:08:35.792 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8999999999999996e-07 2024-05-15T13:08:35.792 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.792 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.792 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:08:35.792 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.745692+0000", 2024-05-15T13:08:35.792 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3569999999999998e-06 2024-05-15T13:08:35.792 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.793 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.793 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:08:35.793 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.745703+0000", 2024-05-15T13:08:35.793 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1265e-05 2024-05-15T13:08:35.793 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.793 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.793 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:08:35.793 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.745710+0000", 2024-05-15T13:08:35.793 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6950000000000003e-06 2024-05-15T13:08:35.793 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.793 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.794 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:08:35.794 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.748954+0000", 2024-05-15T13:08:35.794 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032438889999999998 2024-05-15T13:08:35.794 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.794 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.794 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:08:35.794 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.749038+0000", 2024-05-15T13:08:35.794 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4071000000000002e-05 2024-05-15T13:08:35.794 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.794 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.794 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,6", 2024-05-15T13:08:35.795 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.749124+0000", 2024-05-15T13:08:35.795 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6287999999999998e-05 2024-05-15T13:08:35.795 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.795 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.795 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.795 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.752736+0000", 2024-05-15T13:08:35.795 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0036120990000000001 2024-05-15T13:08:35.795 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.795 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.795 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:08:35.795 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.752779+0000", 2024-05-15T13:08:35.795 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2889000000000002e-05 2024-05-15T13:08:35.795 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.795 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.796 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.796 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.945094+0000", 2024-05-15T13:08:35.796 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19231424499999999 2024-05-15T13:08:35.796 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.796 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.796 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:08:35.796 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.945144+0000", 2024-05-15T13:08:35.796 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0198000000000003e-05 2024-05-15T13:08:35.796 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.796 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.796 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:08:35.796 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:05:16.945186+0000", 2024-05-15T13:08:35.796 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2262999999999998e-05 2024-05-15T13:08:35.796 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.797 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:08:35.797 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.797 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.797 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.797 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:243097 5.b 5:d170f94c:::benchmark_data_smithi177_109991_object107906:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e483)", 2024-05-15T13:08:35.797 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:06:52.572889+0000", 2024-05-15T13:08:35.797 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 103.161884578, 2024-05-15T13:08:35.797 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22338834399999999, 2024-05-15T13:08:35.797 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:08:35.797 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:08:35.797 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:08:35.797 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:08:35.797 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:08:35.797 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 243097 2024-05-15T13:08:35.798 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.798 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:08:35.798 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.798 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:08:35.798 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.572889+0000", 2024-05-15T13:08:35.798 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.798 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.798 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.798 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:08:35.798 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.572889+0000", 2024-05-15T13:08:35.798 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.798 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.798 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.799 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:08:35.799 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.572889+0000", 2024-05-15T13:08:35.799 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3099999999999999e-07 2024-05-15T13:08:35.799 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.799 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.799 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:08:35.799 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.572891+0000", 2024-05-15T13:08:35.799 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.747e-06 2024-05-15T13:08:35.799 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.799 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.799 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:08:35.799 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.572895+0000", 2024-05-15T13:08:35.800 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9249999999999997e-06 2024-05-15T13:08:35.800 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.800 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.800 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:08:35.800 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.572898+0000", 2024-05-15T13:08:35.800 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9390000000000002e-06 2024-05-15T13:08:35.800 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.800 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.800 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:08:35.800 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.578451+0000", 2024-05-15T13:08:35.800 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0055527509999999999 2024-05-15T13:08:35.800 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.800 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.801 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:08:35.801 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.578570+0000", 2024-05-15T13:08:35.801 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000119023 2024-05-15T13:08:35.801 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.801 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.801 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:08:35.801 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.578683+0000", 2024-05-15T13:08:35.801 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011298699999999999 2024-05-15T13:08:35.801 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.801 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.801 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.801 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.589625+0000", 2024-05-15T13:08:35.801 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010942386 2024-05-15T13:08:35.802 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.802 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.802 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:08:35.802 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.595284+0000", 2024-05-15T13:08:35.802 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0056585450000000001 2024-05-15T13:08:35.802 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.802 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.802 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.802 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.795938+0000", 2024-05-15T13:08:35.802 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20065452 2024-05-15T13:08:35.802 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.802 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.802 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:08:35.802 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.796210+0000", 2024-05-15T13:08:35.803 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00027190999999999999 2024-05-15T13:08:35.803 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.803 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.803 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:08:35.803 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.796277+0000", 2024-05-15T13:08:35.803 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7280000000000006e-05 2024-05-15T13:08:35.803 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.803 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:08:35.803 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.803 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.803 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.803 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:243100 5.b 5:d2a8b606:::benchmark_data_smithi177_109991_object107909:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e483)", 2024-05-15T13:08:35.804 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:06:52.576628+0000", 2024-05-15T13:08:35.804 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 103.15814560299999, 2024-05-15T13:08:35.804 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22816081799999999, 2024-05-15T13:08:35.804 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:08:35.804 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:08:35.804 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:08:35.804 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:08:35.804 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:08:35.804 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 243100 2024-05-15T13:08:35.804 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.804 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:08:35.804 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.804 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:08:35.804 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.576628+0000", 2024-05-15T13:08:35.805 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.805 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.805 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.805 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:08:35.805 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.576628+0000", 2024-05-15T13:08:35.805 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.805 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.805 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.805 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:08:35.805 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.576629+0000", 2024-05-15T13:08:35.805 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.006e-06 2024-05-15T13:08:35.805 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.805 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.806 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:08:35.806 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.576632+0000", 2024-05-15T13:08:35.806 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3000000000000002e-06 2024-05-15T13:08:35.806 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.806 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.806 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:08:35.806 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.576643+0000", 2024-05-15T13:08:35.806 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.022e-05 2024-05-15T13:08:35.806 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.806 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.806 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:08:35.806 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.576648+0000", 2024-05-15T13:08:35.806 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0939999999999998e-06 2024-05-15T13:08:35.807 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.807 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.807 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:08:35.807 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.584495+0000", 2024-05-15T13:08:35.807 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0078467929999999995 2024-05-15T13:08:35.807 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.807 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.807 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:08:35.807 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.584602+0000", 2024-05-15T13:08:35.807 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000107017 2024-05-15T13:08:35.807 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.807 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.807 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:08:35.807 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.584710+0000", 2024-05-15T13:08:35.808 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000108421 2024-05-15T13:08:35.808 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.808 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.808 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.808 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.594598+0000", 2024-05-15T13:08:35.808 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0098875669999999999 2024-05-15T13:08:35.808 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.808 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.808 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:08:35.808 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.595294+0000", 2024-05-15T13:08:35.808 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00069630799999999997 2024-05-15T13:08:35.808 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.808 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.808 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.809 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.804676+0000", 2024-05-15T13:08:35.809 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20938241699999999 2024-05-15T13:08:35.809 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.809 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.809 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:08:35.809 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.804736+0000", 2024-05-15T13:08:35.809 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9518999999999997e-05 2024-05-15T13:08:35.809 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.809 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.809 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:08:35.809 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.804789+0000", 2024-05-15T13:08:35.809 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3155999999999997e-05 2024-05-15T13:08:35.810 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.810 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:08:35.810 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.810 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.810 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.810 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:243103 5.3 5:c005d816:::benchmark_data_smithi177_109991_object107912:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e483)", 2024-05-15T13:08:35.810 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:06:52.581144+0000", 2024-05-15T13:08:35.810 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 103.153629331, 2024-05-15T13:08:35.810 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.215133095, 2024-05-15T13:08:35.810 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:08:35.810 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:08:35.810 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:08:35.810 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:08:35.811 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:08:35.811 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 243103 2024-05-15T13:08:35.811 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.811 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:08:35.811 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.811 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:08:35.811 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.581144+0000", 2024-05-15T13:08:35.811 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.811 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.811 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.811 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:08:35.811 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.581144+0000", 2024-05-15T13:08:35.811 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.811 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.812 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.812 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:08:35.812 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.581146+0000", 2024-05-15T13:08:35.812 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3689999999999999e-06 2024-05-15T13:08:35.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.812 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.812 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:08:35.812 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.581152+0000", 2024-05-15T13:08:35.812 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.947e-06 2024-05-15T13:08:35.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.812 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.812 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:08:35.812 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.581165+0000", 2024-05-15T13:08:35.813 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3161e-05 2024-05-15T13:08:35.813 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.813 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.813 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:08:35.813 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.581173+0000", 2024-05-15T13:08:35.813 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0330000000000008e-06 2024-05-15T13:08:35.813 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.813 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.813 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:08:35.813 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.584479+0000", 2024-05-15T13:08:35.813 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033057619999999998 2024-05-15T13:08:35.813 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.813 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.814 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:08:35.814 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.584588+0000", 2024-05-15T13:08:35.814 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010894499999999999 2024-05-15T13:08:35.814 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.814 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.814 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,6", 2024-05-15T13:08:35.814 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.584700+0000", 2024-05-15T13:08:35.814 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011227599999999999 2024-05-15T13:08:35.814 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.814 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.814 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.814 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.594622+0000", 2024-05-15T13:08:35.815 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0099217809999999993 2024-05-15T13:08:35.815 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.815 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.815 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:08:35.815 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.595300+0000", 2024-05-15T13:08:35.815 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00067790999999999999 2024-05-15T13:08:35.815 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.815 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.815 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.815 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.795947+0000", 2024-05-15T13:08:35.815 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.200647138 2024-05-15T13:08:35.815 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.815 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.815 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:08:35.816 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.796206+0000", 2024-05-15T13:08:35.816 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00025885699999999998 2024-05-15T13:08:35.816 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.816 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.816 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:08:35.816 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.796277+0000", 2024-05-15T13:08:35.816 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1915999999999994e-05 2024-05-15T13:08:35.816 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.816 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:08:35.817 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.817 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.817 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.817 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:243110 5.b 5:d9ab042b:::benchmark_data_smithi177_109991_object107919:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e483)", 2024-05-15T13:08:35.817 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:06:52.587202+0000", 2024-05-15T13:08:35.817 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 103.14757140099999, 2024-05-15T13:08:35.817 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22631024199999999, 2024-05-15T13:08:35.817 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:08:35.817 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:08:35.817 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:08:35.817 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:08:35.817 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:08:35.817 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 243110 2024-05-15T13:08:35.817 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.818 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:08:35.818 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.818 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:08:35.818 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.587202+0000", 2024-05-15T13:08:35.818 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.818 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.818 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.818 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:08:35.818 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.587202+0000", 2024-05-15T13:08:35.818 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.818 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.818 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.818 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:08:35.818 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.587204+0000", 2024-05-15T13:08:35.819 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2899999999999999e-06 2024-05-15T13:08:35.819 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.819 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.819 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:08:35.819 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.587208+0000", 2024-05-15T13:08:35.819 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2780000000000004e-06 2024-05-15T13:08:35.819 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.819 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.819 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:08:35.819 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.587217+0000", 2024-05-15T13:08:35.819 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7609999999999996e-06 2024-05-15T13:08:35.819 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.819 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.819 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:08:35.820 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.587223+0000", 2024-05-15T13:08:35.820 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.879e-06 2024-05-15T13:08:35.820 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.820 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.820 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:08:35.820 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.589655+0000", 2024-05-15T13:08:35.820 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0024328050000000001 2024-05-15T13:08:35.820 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.820 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.820 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:08:35.820 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.589760+0000", 2024-05-15T13:08:35.820 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000104756 2024-05-15T13:08:35.820 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.820 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.821 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:08:35.821 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.589866+0000", 2024-05-15T13:08:35.821 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000106377 2024-05-15T13:08:35.821 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.821 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.821 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:08:35.821 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.595304+0000", 2024-05-15T13:08:35.821 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0054378179999999996 2024-05-15T13:08:35.821 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.821 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.821 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.821 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.595385+0000", 2024-05-15T13:08:35.821 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0290999999999994e-05 2024-05-15T13:08:35.822 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.822 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.822 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.822 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.813438+0000", 2024-05-15T13:08:35.822 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.21805297800000001 2024-05-15T13:08:35.822 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.822 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.822 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:08:35.822 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.813481+0000", 2024-05-15T13:08:35.822 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3380999999999997e-05 2024-05-15T13:08:35.822 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.823 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.823 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:08:35.823 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.813513+0000", 2024-05-15T13:08:35.823 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1628000000000003e-05 2024-05-15T13:08:35.823 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.823 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:08:35.823 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.823 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.823 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.823 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:243120 5.3 5:c463b00d:::benchmark_data_smithi177_109991_object107929:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e483)", 2024-05-15T13:08:35.823 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:06:52.606097+0000", 2024-05-15T13:08:35.824 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 103.1286771, 2024-05-15T13:08:35.824 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20255478399999999, 2024-05-15T13:08:35.824 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:08:35.824 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:08:35.824 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:08:35.824 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:08:35.824 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:08:35.824 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 243120 2024-05-15T13:08:35.824 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.824 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:08:35.824 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.824 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:08:35.824 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.606097+0000", 2024-05-15T13:08:35.825 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.825 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.825 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.825 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:08:35.825 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.606097+0000", 2024-05-15T13:08:35.825 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.825 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.825 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.825 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:08:35.825 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.606099+0000", 2024-05-15T13:08:35.825 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3159999999999999e-06 2024-05-15T13:08:35.825 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.825 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.826 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:08:35.826 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.606105+0000", 2024-05-15T13:08:35.826 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4869999999999999e-06 2024-05-15T13:08:35.826 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.826 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.826 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:08:35.826 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.606134+0000", 2024-05-15T13:08:35.826 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8308000000000001e-05 2024-05-15T13:08:35.826 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.826 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.826 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:08:35.826 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.606145+0000", 2024-05-15T13:08:35.827 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0849000000000001e-05 2024-05-15T13:08:35.827 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.827 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.827 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:08:35.827 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.606218+0000", 2024-05-15T13:08:35.827 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3059999999999995e-05 2024-05-15T13:08:35.827 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.827 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.827 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:08:35.827 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.606306+0000", 2024-05-15T13:08:35.827 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8787000000000002e-05 2024-05-15T13:08:35.827 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.827 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.827 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,6", 2024-05-15T13:08:35.828 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.606376+0000", 2024-05-15T13:08:35.828 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9890999999999999e-05 2024-05-15T13:08:35.828 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.829 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.829 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:08:35.829 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.613159+0000", 2024-05-15T13:08:35.829 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0067827210000000002 2024-05-15T13:08:35.829 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.829 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.829 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.829 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.620600+0000", 2024-05-15T13:08:35.829 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0074412000000000002 2024-05-15T13:08:35.829 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.829 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.830 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.830 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.808527+0000", 2024-05-15T13:08:35.830 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.18792634899999999 2024-05-15T13:08:35.830 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.830 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.830 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:08:35.830 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.808582+0000", 2024-05-15T13:08:35.831 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5816000000000002e-05 2024-05-15T13:08:35.831 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.831 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.831 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:08:35.831 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.808651+0000", 2024-05-15T13:08:35.831 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8999999999999997e-05 2024-05-15T13:08:35.831 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.831 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:08:35.832 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.832 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.832 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.832 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:243124 5.3 5:c8757277:::benchmark_data_smithi177_109991_object107933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e483)", 2024-05-15T13:08:35.832 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:06:52.614521+0000", 2024-05-15T13:08:35.832 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 103.12025250799999, 2024-05-15T13:08:35.832 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20479784200000001, 2024-05-15T13:08:35.832 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:08:35.832 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:08:35.832 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:08:35.832 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:08:35.832 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:08:35.832 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 243124 2024-05-15T13:08:35.833 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.833 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:08:35.833 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.833 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:08:35.833 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.614521+0000", 2024-05-15T13:08:35.833 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.833 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.833 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.833 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:08:35.833 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.614521+0000", 2024-05-15T13:08:35.834 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.834 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.834 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.834 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:08:35.834 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.614522+0000", 2024-05-15T13:08:35.835 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4199999999999996e-07 2024-05-15T13:08:35.835 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.835 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.835 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:08:35.835 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.614526+0000", 2024-05-15T13:08:35.835 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8979999999999998e-06 2024-05-15T13:08:35.835 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.835 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.835 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:08:35.835 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.614534+0000", 2024-05-15T13:08:35.835 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.909e-06 2024-05-15T13:08:35.835 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.835 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.835 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:08:35.836 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.614541+0000", 2024-05-15T13:08:35.836 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9300000000000003e-06 2024-05-15T13:08:35.836 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.836 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.836 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:08:35.836 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.614716+0000", 2024-05-15T13:08:35.836 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017407999999999999 2024-05-15T13:08:35.836 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.836 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.836 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:08:35.836 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.614860+0000", 2024-05-15T13:08:35.836 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000144874 2024-05-15T13:08:35.836 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.836 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.837 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,6", 2024-05-15T13:08:35.837 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.615003+0000", 2024-05-15T13:08:35.837 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014271800000000001 2024-05-15T13:08:35.837 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.837 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.837 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:08:35.837 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.624680+0000", 2024-05-15T13:08:35.837 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.009676423 2024-05-15T13:08:35.837 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.837 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.837 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.837 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.624774+0000", 2024-05-15T13:08:35.837 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4578000000000001e-05 2024-05-15T13:08:35.837 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.838 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.838 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.838 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.819201+0000", 2024-05-15T13:08:35.838 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19442732800000001 2024-05-15T13:08:35.838 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.838 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.838 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:08:35.838 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.819268+0000", 2024-05-15T13:08:35.838 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6261999999999997e-05 2024-05-15T13:08:35.838 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.838 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.838 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:08:35.838 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.819319+0000", 2024-05-15T13:08:35.838 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.13e-05 2024-05-15T13:08:35.839 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.839 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:08:35.839 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.839 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.839 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.839 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:243130 5.b 5:d0d7ff10:::benchmark_data_smithi177_109991_object107939:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e483)", 2024-05-15T13:08:35.839 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:06:52.618901+0000", 2024-05-15T13:08:35.839 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 103.11587292500001, 2024-05-15T13:08:35.839 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.203074106, 2024-05-15T13:08:35.839 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:08:35.839 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:08:35.839 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:08:35.839 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:08:35.839 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:08:35.840 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 243130 2024-05-15T13:08:35.840 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.840 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:08:35.840 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.840 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:08:35.840 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.618901+0000", 2024-05-15T13:08:35.840 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.840 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.840 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.840 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:08:35.840 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.618901+0000", 2024-05-15T13:08:35.840 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.840 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.840 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.840 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:08:35.841 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.618902+0000", 2024-05-15T13:08:35.841 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.077e-06 2024-05-15T13:08:35.841 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.841 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.841 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:08:35.841 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.618906+0000", 2024-05-15T13:08:35.841 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.703e-06 2024-05-15T13:08:35.841 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.841 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.841 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:08:35.841 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.618916+0000", 2024-05-15T13:08:35.841 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0397999999999999e-05 2024-05-15T13:08:35.841 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.841 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.842 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:08:35.842 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.618921+0000", 2024-05-15T13:08:35.842 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2259999999999996e-06 2024-05-15T13:08:35.842 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.842 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.842 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:08:35.842 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.620577+0000", 2024-05-15T13:08:35.842 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016563249999999999 2024-05-15T13:08:35.842 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.842 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.842 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:08:35.842 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.620728+0000", 2024-05-15T13:08:35.842 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015090599999999999 2024-05-15T13:08:35.842 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.843 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.843 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:08:35.843 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.620816+0000", 2024-05-15T13:08:35.843 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7677999999999996e-05 2024-05-15T13:08:35.843 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.843 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.843 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:08:35.843 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.624837+0000", 2024-05-15T13:08:35.843 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0040204660000000003 2024-05-15T13:08:35.843 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.843 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.843 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.843 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.630010+0000", 2024-05-15T13:08:35.843 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.005173413 2024-05-15T13:08:35.844 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.844 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.844 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.844 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.821862+0000", 2024-05-15T13:08:35.844 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.191852247 2024-05-15T13:08:35.844 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.844 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.844 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:08:35.844 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.821923+0000", 2024-05-15T13:08:35.844 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0872999999999998e-05 2024-05-15T13:08:35.844 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.844 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.844 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:08:35.844 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:06:52.821975+0000", 2024-05-15T13:08:35.845 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1793999999999999e-05 2024-05-15T13:08:35.845 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.845 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:08:35.845 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.845 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.845 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.845 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:15525 6.9 6:93b47e9d:::benchmark_data_smithi177_122278_object15524:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e515)", 2024-05-15T13:08:35.845 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:07:59.461441+0000", 2024-05-15T13:08:35.845 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 36.273332492000002, 2024-05-15T13:08:35.845 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20655172499999999, 2024-05-15T13:08:35.845 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:08:35.845 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:08:35.845 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:08:35.845 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:08:35.846 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:08:35.846 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 15525 2024-05-15T13:08:35.846 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.846 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:08:35.846 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.846 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:08:35.846 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:07:59.461441+0000", 2024-05-15T13:08:35.846 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.846 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.846 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.846 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:08:35.846 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:07:59.461441+0000", 2024-05-15T13:08:35.846 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:08:35.846 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.847 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.847 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:08:35.847 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:07:59.461446+0000", 2024-05-15T13:08:35.847 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7910000000000004e-06 2024-05-15T13:08:35.847 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.847 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.847 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:08:35.847 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:07:59.461515+0000", 2024-05-15T13:08:35.847 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9039000000000001e-05 2024-05-15T13:08:35.847 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.847 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.847 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:08:35.847 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:07:59.461529+0000", 2024-05-15T13:08:35.847 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3784999999999999e-05 2024-05-15T13:08:35.848 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.848 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.848 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:08:35.848 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:07:59.461538+0000", 2024-05-15T13:08:35.848 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7050000000000005e-06 2024-05-15T13:08:35.848 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.848 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.848 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:08:35.848 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:07:59.461589+0000", 2024-05-15T13:08:35.848 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1752999999999997e-05 2024-05-15T13:08:35.848 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.848 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.848 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:08:35.848 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:07:59.461690+0000", 2024-05-15T13:08:35.849 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000100802 2024-05-15T13:08:35.849 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.849 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.849 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,9", 2024-05-15T13:08:35.849 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:07:59.461775+0000", 2024-05-15T13:08:35.849 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4887000000000002e-05 2024-05-15T13:08:35.849 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.849 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.849 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:08:35.849 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:07:59.462786+0000", 2024-05-15T13:08:35.849 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010112089999999999 2024-05-15T13:08:35.849 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.849 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.849 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.850 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:07:59.463614+0000", 2024-05-15T13:08:35.850 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00082830700000000002 2024-05-15T13:08:35.850 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.850 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.850 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:08:35.850 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:07:59.667920+0000", 2024-05-15T13:08:35.850 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.204305601 2024-05-15T13:08:35.850 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.850 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.850 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:08:35.850 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:07:59.667958+0000", 2024-05-15T13:08:35.850 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8052000000000001e-05 2024-05-15T13:08:35.850 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:08:35.850 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:08:35.851 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:08:35.851 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:07:59.667993+0000", 2024-05-15T13:08:35.851 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4793999999999999e-05 2024-05-15T13:08:35.851 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.851 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:08:35.851 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.851 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:08:35.851 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:08:35.851 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:08:35.852 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:35 smithi120 ceph-mon[43961]: from='client.15873 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:08:35.852 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:35.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:35 smithi177 ceph-mon[44303]: from='client.15873 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:08:36.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:36 smithi027 ceph-mon[39795]: pgmap v1398: 27 pgs: 27 active+clean; 4.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-05-15T13:08:36.537 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 42 16 79005 78989 117.524 67.0625 0.0125813 0.00850482 2024-05-15T13:08:36.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:36 smithi120 ceph-mon[43961]: pgmap v1398: 27 pgs: 27 active+clean; 4.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-05-15T13:08:36.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:36 smithi177 ceph-mon[44303]: pgmap v1398: 27 pgs: 27 active+clean; 4.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-05-15T13:08:37.537 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 43 16 80435 80419 116.87 89.375 0.00692224 0.00855212 2024-05-15T13:08:37.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:37 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:08:37.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:37 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:08:37.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:37 smithi027 ceph-mon[39795]: osdmap e531: 12 total, 12 up, 11 in 2024-05-15T13:08:37.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:37 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:08:37.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:37 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:08:37.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:37 smithi120 ceph-mon[43961]: osdmap e531: 12 total, 12 up, 11 in 2024-05-15T13:08:37.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:37 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:08:37.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:37 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:08:37.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:37 smithi177 ceph-mon[44303]: osdmap e531: 12 total, 12 up, 11 in 2024-05-15T13:08:37.994 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:08:38.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:38.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:08:38.112 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:08:38.143 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:38.168 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:08:38.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:38.172 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:08:38.273 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:08:38.306 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:38.331 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:08:38.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:38.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:08:38.435 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:08:38.468 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:38.494 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:08:38.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:38.499 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:08:38.537 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 44 16 81825 81809 116.188 86.875 0.0131146 0.00860277 2024-05-15T13:08:38.600 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:08:38.629 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:38.653 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:08:38.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:38.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:08:38.757 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:08:38.762 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:38 smithi027 ceph-mon[39795]: pgmap v1400: 27 pgs: 27 active+clean; 4.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-05-15T13:08:38.762 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:38 smithi027 ceph-mon[39795]: osdmap e532: 12 total, 12 up, 11 in 2024-05-15T13:08:38.779 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:38 smithi120 ceph-mon[43961]: pgmap v1400: 27 pgs: 27 active+clean; 4.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-05-15T13:08:38.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:38 smithi120 ceph-mon[43961]: osdmap e532: 12 total, 12 up, 11 in 2024-05-15T13:08:38.784 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:38 smithi177 ceph-mon[44303]: pgmap v1400: 27 pgs: 27 active+clean; 4.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-05-15T13:08:38.784 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:38 smithi177 ceph-mon[44303]: osdmap e532: 12 total, 12 up, 11 in 2024-05-15T13:08:38.785 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:38.787 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-05-15T13:08:38.809 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:08:38.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:38.815 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:08:38.916 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:08:38.948 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:38.971 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:08:38.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:38.976 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:08:39.077 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:08:39.106 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:39.132 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:08:39.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:39.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:08:39.238 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:08:39.240 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":1397,"stamp":"2024-05-15T13:08:34.086468+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4788978208,"num_objects":73067,"num_object_clones":0,"num_object_copies":219201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73065,"num_write_kb":4676160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74594,"ondisk_log_size":74594,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":19199848,"kb_used_data":13871540,"kb_used_omap":17,"kb_used_meta":5328174,"kb_avail":1011996824,"statfs":{"total":1055945392128,"available":1036284747776,"internally_reserved":0,"allocated":14204456960,"data_stored":14200474480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17551,"internal_metadata":5456051057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,12,8,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":614006784,"num_objects":9369,"num_object_clones":0,"num_object_copies":28107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9369,"num_write_kb":599616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001654"},"pg_stats":[{"pgid":"6.c","version":"528'4677","reported_seq":4725,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-05-15T13:08:31.849011+0000","last_change":"2024-05-15T13:07:43.861146+0000","last_active":"2024-05-15T13:08:31.849011+0000","last_peered":"2024-05-15T13:08:31.849011+0000","last_clean":"2024-05-15T13:08:31.849011+0000","last_became_active":"2024-05-15T13:07:43.860763+0000","last_became_peered":"2024-05-15T13:07:43.860763+0000","last_unstale":"2024-05-15T13:08:31.849011+0000","last_undegraded":"2024-05-15T13:08:31.849011+0000","last_fullsized":"2024-05-15T13:08:31.849011+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":4677,"log_dups_size":0,"ondisk_log_size":4677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:44.283079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306511872,"num_objects":4677,"num_object_clones":0,"num_object_copies":14031,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4677,"num_write_kb":299328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"528'4612","reported_seq":4660,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-05-15T13:08:31.810490+0000","last_change":"2024-05-15T13:07:43.864203+0000","last_active":"2024-05-15T13:08:31.810490+0000","last_peered":"2024-05-15T13:08:31.810490+0000","last_clean":"2024-05-15T13:08:31.810490+0000","last_became_active":"2024-05-15T13:07:43.863906+0000","last_became_peered":"2024-05-15T13:07:43.863906+0000","last_unstale":"2024-05-15T13:08:31.810490+0000","last_undegraded":"2024-05-15T13:08:31.810490+0000","last_fullsized":"2024-05-15T13:08:31.810490+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":4612,"log_dups_size":0,"ondisk_log_size":4612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:46.965522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302252032,"num_objects":4612,"num_object_clones":0,"num_object_copies":13836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4612,"num_write_kb":295168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"529'4641","reported_seq":4691,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-05-15T13:08:33.155520+0000","last_change":"2024-05-15T13:07:43.856272+0000","last_active":"2024-05-15T13:08:33.155520+0000","last_peered":"2024-05-15T13:08:33.155520+0000","last_clean":"2024-05-15T13:08:33.155520+0000","last_became_active":"2024-05-15T13:07:43.855799+0000","last_became_peered":"2024-05-15T13:07:43.855799+0000","last_unstale":"2024-05-15T13:08:33.155520+0000","last_undegraded":"2024-05-15T13:08:33.155520+0000","last_fullsized":"2024-05-15T13:08:33.155520+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":4641,"log_dups_size":0,"ondisk_log_size":4641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:55.927298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304152576,"num_objects":4641,"num_object_clones":0,"num_object_copies":13923,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4641,"num_write_kb":297024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"528'4285","reported_seq":4332,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-05-15T13:08:29.781471+0000","last_change":"2024-05-15T13:07:44.416156+0000","last_active":"2024-05-15T13:08:29.781471+0000","last_peered":"2024-05-15T13:08:29.781471+0000","last_clean":"2024-05-15T13:08:29.781471+0000","last_became_active":"2024-05-15T13:07:44.415859+0000","last_became_peered":"2024-05-15T13:07:44.415859+0000","last_unstale":"2024-05-15T13:08:29.781471+0000","last_undegraded":"2024-05-15T13:08:29.781471+0000","last_fullsized":"2024-05-15T13:08:29.781471+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":4285,"log_dups_size":0,"ondisk_log_size":4285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:44.704574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280821760,"num_objects":4285,"num_object_clones":0,"num_object_copies":12855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4285,"num_write_kb":274240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"529'4591","reported_seq":4641,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-05-15T13:08:32.718405+0000","last_change":"2024-05-15T13:07:43.858333+0000","last_active":"2024-05-15T13:08:32.718405+0000","last_peered":"2024-05-15T13:08:32.718405+0000","last_clean":"2024-05-15T13:08:32.718405+0000","last_became_active":"2024-05-15T13:07:43.858034+0000","last_became_peered":"2024-05-15T13:07:43.858034+0000","last_unstale":"2024-05-15T13:08:32.718405+0000","last_undegraded":"2024-05-15T13:08:32.718405+0000","last_fullsized":"2024-05-15T13:08:32.718405+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":4591,"log_dups_size":0,"ondisk_log_size":4591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:57.970835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300875776,"num_objects":4591,"num_object_clones":0,"num_object_copies":13773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4591,"num_write_kb":293824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,6],"acting":[11,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.d","version":"529'4690","reported_seq":4740,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-05-15T13:08:33.143267+0000","last_change":"2024-05-15T13:07:44.092209+0000","last_active":"2024-05-15T13:08:33.143267+0000","last_peered":"2024-05-15T13:08:33.143267+0000","last_clean":"2024-05-15T13:08:33.143267+0000","last_became_active":"2024-05-15T13:07:44.091933+0000","last_became_peered":"2024-05-15T13:07:44.091933+0000","last_unstale":"2024-05-15T13:08:33.143267+0000","last_undegraded":"2024-05-15T13:08:33.143267+0000","last_fullsized":"2024-05-15T13:08:33.143267+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:55.177111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307363840,"num_objects":4690,"num_object_clones":0,"num_object_copies":14070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4690,"num_write_kb":300160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"94'139","reported_seq":904,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-05-15T13:08:28.437886+0000","last_change":"2024-05-15T13:06:55.617237+0000","last_active":"2024-05-15T13:08:28.437886+0000","last_peered":"2024-05-15T13:08:28.437886+0000","last_clean":"2024-05-15T13:08:28.437886+0000","last_became_active":"2024-05-15T13:06:45.407469+0000","last_became_peered":"2024-05-15T13:06:45.407469+0000","last_unstale":"2024-05-15T13:08:28.437886+0000","last_undegraded":"2024-05-15T13:08:28.437886+0000","last_fullsized":"2024-05-15T13:08:28.437886+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:06:55.617099+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:06:55.617099+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:29.867761+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.e","version":"528'4440","reported_seq":4488,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-05-15T13:08:30.015864+0000","last_change":"2024-05-15T13:07:43.864202+0000","last_active":"2024-05-15T13:08:30.015864+0000","last_peered":"2024-05-15T13:08:30.015864+0000","last_clean":"2024-05-15T13:08:30.015864+0000","last_became_active":"2024-05-15T13:07:43.863923+0000","last_became_peered":"2024-05-15T13:07:43.863923+0000","last_unstale":"2024-05-15T13:08:30.015864+0000","last_undegraded":"2024-05-15T13:08:30.015864+0000","last_fullsized":"2024-05-15T13:08:30.015864+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":4440,"log_dups_size":0,"ondisk_log_size":4440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:59.591350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290979840,"num_objects":4440,"num_object_clones":0,"num_object_copies":13320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4440,"num_write_kb":284160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"94'139","reported_seq":902,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-05-15T13:08:28.439658+0000","last_change":"2024-05-15T13:06:57.664860+0000","last_active":"2024-05-15T13:08:28.439658+0000","last_peered":"2024-05-15T13:08:28.439658+0000","last_clean":"2024-05-15T13:08:28.439658+0000","last_became_active":"2024-05-15T13:06:46.424667+0000","last_became_peered":"2024-05-15T13:06:46.424667+0000","last_unstale":"2024-05-15T13:08:28.439658+0000","last_undegraded":"2024-05-15T13:08:28.439658+0000","last_fullsized":"2024-05-15T13:08:28.439658+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:06:57.664687+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:06:57.664687+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.f","version":"529'4652","reported_seq":4702,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-05-15T13:08:32.729004+0000","last_change":"2024-05-15T13:07:44.416852+0000","last_active":"2024-05-15T13:08:32.729004+0000","last_peered":"2024-05-15T13:08:32.729004+0000","last_clean":"2024-05-15T13:08:32.729004+0000","last_became_active":"2024-05-15T13:07:44.416528+0000","last_became_peered":"2024-05-15T13:07:44.416528+0000","last_unstale":"2024-05-15T13:08:32.729004+0000","last_undegraded":"2024-05-15T13:08:32.729004+0000","last_fullsized":"2024-05-15T13:08:32.729004+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":4652,"log_dups_size":0,"ondisk_log_size":4652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:09:04.289381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304873472,"num_objects":4652,"num_object_clones":0,"num_object_copies":13956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4652,"num_write_kb":297728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"94'139","reported_seq":904,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-05-15T13:08:32.165028+0000","last_change":"2024-05-15T13:06:58.489453+0000","last_active":"2024-05-15T13:08:32.165028+0000","last_peered":"2024-05-15T13:08:32.165028+0000","last_clean":"2024-05-15T13:08:32.165028+0000","last_became_active":"2024-05-15T13:06:46.426908+0000","last_became_peered":"2024-05-15T13:06:46.426908+0000","last_unstale":"2024-05-15T13:08:32.165028+0000","last_undegraded":"2024-05-15T13:08:32.165028+0000","last_fullsized":"2024-05-15T13:08:32.165028+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:06:58.489366+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:06:58.489366+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:34.725888+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"528'4509","reported_seq":4557,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-05-15T13:08:31.767118+0000","last_change":"2024-05-15T13:07:43.865627+0000","last_active":"2024-05-15T13:08:31.767118+0000","last_peered":"2024-05-15T13:08:31.767118+0000","last_clean":"2024-05-15T13:08:31.767118+0000","last_became_active":"2024-05-15T13:07:43.865339+0000","last_became_peered":"2024-05-15T13:07:43.865339+0000","last_unstale":"2024-05-15T13:08:31.767118+0000","last_undegraded":"2024-05-15T13:08:31.767118+0000","last_fullsized":"2024-05-15T13:08:31.767118+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":4509,"log_dups_size":0,"ondisk_log_size":4509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:55.567826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295501824,"num_objects":4509,"num_object_clones":0,"num_object_copies":13527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4509,"num_write_kb":288576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"94'139","reported_seq":909,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-05-15T13:08:28.437206+0000","last_change":"2024-05-15T13:06:56.791460+0000","last_active":"2024-05-15T13:08:28.437206+0000","last_peered":"2024-05-15T13:08:28.437206+0000","last_clean":"2024-05-15T13:08:28.437206+0000","last_became_active":"2024-05-15T13:06:45.402594+0000","last_became_peered":"2024-05-15T13:06:45.402594+0000","last_unstale":"2024-05-15T13:08:28.437206+0000","last_undegraded":"2024-05-15T13:08:28.437206+0000","last_fullsized":"2024-05-15T13:08:28.437206+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:06:56.791343+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:06:56.791343+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:33.436905+0000","scrub_duration":170,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"530'4820","reported_seq":4872,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-05-15T13:08:33.717973+0000","last_change":"2024-05-15T13:07:44.416267+0000","last_active":"2024-05-15T13:08:33.717973+0000","last_peered":"2024-05-15T13:08:33.717973+0000","last_clean":"2024-05-15T13:08:33.717973+0000","last_became_active":"2024-05-15T13:07:44.415987+0000","last_became_peered":"2024-05-15T13:07:44.415987+0000","last_unstale":"2024-05-15T13:08:33.717973+0000","last_undegraded":"2024-05-15T13:08:33.717973+0000","last_fullsized":"2024-05-15T13:08:33.717973+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:09:06.262336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315883520,"num_objects":4820,"num_object_clones":0,"num_object_copies":14460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4820,"num_write_kb":308480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"94'139","reported_seq":901,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-05-15T13:08:28.429906+0000","last_change":"2024-05-15T13:06:58.688595+0000","last_active":"2024-05-15T13:08:28.429906+0000","last_peered":"2024-05-15T13:08:28.429906+0000","last_clean":"2024-05-15T13:08:28.429906+0000","last_became_active":"2024-05-15T13:06:46.420025+0000","last_became_peered":"2024-05-15T13:06:46.420025+0000","last_unstale":"2024-05-15T13:08:28.429906+0000","last_undegraded":"2024-05-15T13:08:28.429906+0000","last_fullsized":"2024-05-15T13:08:28.429906+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:06:58.688446+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:06:58.688446+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:31.733322+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,8],"acting":[0,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"528'4558","reported_seq":4606,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-05-15T13:08:31.845277+0000","last_change":"2024-05-15T13:07:43.860330+0000","last_active":"2024-05-15T13:08:31.845277+0000","last_peered":"2024-05-15T13:08:31.845277+0000","last_clean":"2024-05-15T13:08:31.845277+0000","last_became_active":"2024-05-15T13:07:43.860023+0000","last_became_peered":"2024-05-15T13:07:43.860023+0000","last_unstale":"2024-05-15T13:08:31.845277+0000","last_undegraded":"2024-05-15T13:08:31.845277+0000","last_fullsized":"2024-05-15T13:08:31.845277+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":4558,"log_dups_size":0,"ondisk_log_size":4558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:43.450397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298713088,"num_objects":4558,"num_object_clones":0,"num_object_copies":13674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4558,"num_write_kb":291712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"94'139","reported_seq":902,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-05-15T13:08:28.439539+0000","last_change":"2024-05-15T13:06:58.629278+0000","last_active":"2024-05-15T13:08:28.439539+0000","last_peered":"2024-05-15T13:08:28.439539+0000","last_clean":"2024-05-15T13:08:28.439539+0000","last_became_active":"2024-05-15T13:06:46.445091+0000","last_became_peered":"2024-05-15T13:06:46.445091+0000","last_unstale":"2024-05-15T13:08:28.439539+0000","last_undegraded":"2024-05-15T13:08:28.439539+0000","last_fullsized":"2024-05-15T13:08:28.439539+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:06:58.629152+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:06:58.629152+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,0],"acting":[9,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.0","version":"528'4373","reported_seq":4421,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-05-15T13:08:29.757152+0000","last_change":"2024-05-15T13:07:43.859654+0000","last_active":"2024-05-15T13:08:29.757152+0000","last_peered":"2024-05-15T13:08:29.757152+0000","last_clean":"2024-05-15T13:08:29.757152+0000","last_became_active":"2024-05-15T13:07:43.859339+0000","last_became_peered":"2024-05-15T13:07:43.859339+0000","last_unstale":"2024-05-15T13:08:29.757152+0000","last_undegraded":"2024-05-15T13:08:29.757152+0000","last_fullsized":"2024-05-15T13:08:29.757152+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":4373,"log_dups_size":0,"ondisk_log_size":4373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:09:04.767546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286588928,"num_objects":4373,"num_object_clones":0,"num_object_copies":13119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4373,"num_write_kb":279872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"94'139","reported_seq":665,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-05-15T13:08:33.434278+0000","last_change":"2024-05-15T13:06:59.028927+0000","last_active":"2024-05-15T13:08:33.434278+0000","last_peered":"2024-05-15T13:08:33.434278+0000","last_clean":"2024-05-15T13:08:33.434278+0000","last_became_active":"2024-05-15T13:06:46.425852+0000","last_became_peered":"2024-05-15T13:06:46.425852+0000","last_unstale":"2024-05-15T13:08:33.434278+0000","last_undegraded":"2024-05-15T13:08:33.434278+0000","last_fullsized":"2024-05-15T13:08:33.434278+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:06:59.028839+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:06:59.028839+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"528'4415","reported_seq":4463,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-05-15T13:08:30.081846+0000","last_change":"2024-05-15T13:07:44.416346+0000","last_active":"2024-05-15T13:08:30.081846+0000","last_peered":"2024-05-15T13:08:30.081846+0000","last_clean":"2024-05-15T13:08:30.081846+0000","last_became_active":"2024-05-15T13:07:44.416072+0000","last_became_peered":"2024-05-15T13:07:44.416072+0000","last_unstale":"2024-05-15T13:08:30.081846+0000","last_undegraded":"2024-05-15T13:08:30.081846+0000","last_fullsized":"2024-05-15T13:08:30.081846+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":4415,"log_dups_size":0,"ondisk_log_size":4415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:53.591543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289341440,"num_objects":4415,"num_object_clones":0,"num_object_copies":13245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4415,"num_write_kb":282560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"94'139","reported_seq":903,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-05-15T13:08:28.437928+0000","last_change":"2024-05-15T13:06:56.613879+0000","last_active":"2024-05-15T13:08:28.437928+0000","last_peered":"2024-05-15T13:08:28.437928+0000","last_clean":"2024-05-15T13:08:28.437928+0000","last_became_active":"2024-05-15T13:06:46.419250+0000","last_became_peered":"2024-05-15T13:06:46.419250+0000","last_unstale":"2024-05-15T13:08:28.437928+0000","last_undegraded":"2024-05-15T13:08:28.437928+0000","last_fullsized":"2024-05-15T13:08:28.437928+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:06:56.613748+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:06:56.613748+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.2","version":"528'4392","reported_seq":4440,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-05-15T13:08:29.995407+0000","last_change":"2024-05-15T13:07:43.858548+0000","last_active":"2024-05-15T13:08:29.995407+0000","last_peered":"2024-05-15T13:08:29.995407+0000","last_clean":"2024-05-15T13:08:29.995407+0000","last_became_active":"2024-05-15T13:07:43.858107+0000","last_became_peered":"2024-05-15T13:07:43.858107+0000","last_unstale":"2024-05-15T13:08:29.995407+0000","last_undegraded":"2024-05-15T13:08:29.995407+0000","last_fullsized":"2024-05-15T13:08:29.995407+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":4392,"log_dups_size":0,"ondisk_log_size":4392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:56.120553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287834112,"num_objects":4392,"num_object_clones":0,"num_object_copies":13176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4392,"num_write_kb":281088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"94'139","reported_seq":647,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-05-15T13:08:28.431365+0000","last_change":"2024-05-15T13:06:59.242408+0000","last_active":"2024-05-15T13:08:28.431365+0000","last_peered":"2024-05-15T13:08:28.431365+0000","last_clean":"2024-05-15T13:08:28.431365+0000","last_became_active":"2024-05-15T13:06:46.420188+0000","last_became_peered":"2024-05-15T13:06:46.420188+0000","last_unstale":"2024-05-15T13:08:28.431365+0000","last_undegraded":"2024-05-15T13:08:28.431365+0000","last_fullsized":"2024-05-15T13:08:28.431365+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:06:59.242294+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:06:59.242294+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.3","version":"530'4700","reported_seq":4752,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-05-15T13:08:33.737934+0000","last_change":"2024-05-15T13:07:43.855924+0000","last_active":"2024-05-15T13:08:33.737934+0000","last_peered":"2024-05-15T13:08:33.737934+0000","last_clean":"2024-05-15T13:08:33.737934+0000","last_became_active":"2024-05-15T13:07:43.855524+0000","last_became_peered":"2024-05-15T13:07:43.855524+0000","last_unstale":"2024-05-15T13:08:33.737934+0000","last_undegraded":"2024-05-15T13:08:33.737934+0000","last_fullsized":"2024-05-15T13:08:33.737934+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":4700,"log_dups_size":0,"ondisk_log_size":4700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:09:10.856367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308019200,"num_objects":4700,"num_object_clones":0,"num_object_copies":14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4700,"num_write_kb":300800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,8],"acting":[6,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"94'139","reported_seq":904,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-05-15T13:08:32.174974+0000","last_change":"2024-05-15T13:06:55.589049+0000","last_active":"2024-05-15T13:08:32.174974+0000","last_peered":"2024-05-15T13:08:32.174974+0000","last_clean":"2024-05-15T13:08:32.174974+0000","last_became_active":"2024-05-15T13:06:45.409132+0000","last_became_peered":"2024-05-15T13:06:45.409132+0000","last_unstale":"2024-05-15T13:08:32.174974+0000","last_undegraded":"2024-05-15T13:08:32.174974+0000","last_fullsized":"2024-05-15T13:08:32.174974+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:06:55.588949+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:06:55.588949+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:34.309096+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.7","version":"529'4710","reported_seq":4759,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-05-15T13:08:33.154887+0000","last_change":"2024-05-15T13:07:43.861203+0000","last_active":"2024-05-15T13:08:33.154887+0000","last_peered":"2024-05-15T13:08:33.154887+0000","last_clean":"2024-05-15T13:08:33.154887+0000","last_became_active":"2024-05-15T13:07:43.860859+0000","last_became_peered":"2024-05-15T13:07:43.860859+0000","last_unstale":"2024-05-15T13:08:33.154887+0000","last_undegraded":"2024-05-15T13:08:33.154887+0000","last_fullsized":"2024-05-15T13:08:33.154887+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":4710,"log_dups_size":0,"ondisk_log_size":4710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:09:09.485083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308674560,"num_objects":4710,"num_object_clones":0,"num_object_copies":14130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4710,"num_write_kb":301440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":910,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-05-15T13:08:28.437151+0000","last_change":"2024-05-15T13:06:59.662493+0000","last_active":"2024-05-15T13:08:28.437151+0000","last_peered":"2024-05-15T13:08:28.437151+0000","last_clean":"2024-05-15T13:08:28.437151+0000","last_became_active":"2024-05-15T13:06:42.028286+0000","last_became_peered":"2024-05-15T13:06:42.028286+0000","last_unstale":"2024-05-15T13:08:28.437151+0000","last_undegraded":"2024-05-15T13:08:28.437151+0000","last_fullsized":"2024-05-15T13:08:28.437151+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:06:59.662380+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:06:59.662380+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:29.025046+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":4788387840,"num_objects":73065,"num_object_clones":0,"num_object_copies":219195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73065,"num_write_kb":4676160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14374469632,"data_stored":14374469632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73065,"ondisk_log_size":73065,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056918,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1043776,"kb_used_data":585012,"kb_used_omap":1,"kb_used_meta":458750,"kb_avail":92701376,"statfs":{"total":95995035648,"available":94926209024,"internally_reserved":0,"allocated":599052288,"data_stored":598687960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":469760454},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351257,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1331248,"kb_used_data":858600,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":92413904,"statfs":{"total":95995035648,"available":94631837696,"internally_reserved":0,"allocated":879206400,"data_stored":878846537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":483981760},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514783,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1703480,"kb_used_data":1126696,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92041672,"statfs":{"total":95995035648,"available":94250672128,"internally_reserved":0,"allocated":1153736704,"data_stored":1153376841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":590608832},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874532,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1130928,"kb_used_data":824872,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":92614224,"statfs":{"total":95995035648,"available":94836965376,"internally_reserved":0,"allocated":844668928,"data_stored":844309073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":240518168872,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430510,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2568192,"kb_used_data":2091380,"kb_used_omap":1,"kb_used_meta":476798,"kb_avail":91176960,"statfs":{"total":95995035648,"available":93365207040,"internally_reserved":0,"allocated":2141573120,"data_stored":2141208800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":488241606},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575176,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1743920,"kb_used_data":1162984,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":92001232,"statfs":{"total":95995035648,"available":94209261568,"internally_reserved":0,"allocated":1190895616,"data_stored":1190535761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738704,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1390256,"kb_used_data":831528,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92354896,"statfs":{"total":95995035648,"available":94571413504,"internally_reserved":0,"allocated":851484672,"data_stored":851124817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411653,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1807672,"kb_used_data":1473840,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":91937480,"statfs":{"total":95995035648,"available":94143979520,"internally_reserved":0,"allocated":1509212160,"data_stored":1508848753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150014,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2834744,"kb_used_data":2302384,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":90910408,"statfs":{"total":95995035648,"available":93092257792,"internally_reserved":0,"allocated":2357641216,"data_stored":2357277809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":545126855},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986489,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1642936,"kb_used_data":1124456,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":92102216,"statfs":{"total":95995035648,"available":94312669184,"internally_reserved":0,"allocated":1151442944,"data_stored":1151083089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692148,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2002696,"kb_used_data":1489788,"kb_used_omap":1,"kb_used_meta":512894,"kb_avail":91742456,"statfs":{"total":95995035648,"available":93944274944,"internally_reserved":0,"allocated":1525542912,"data_stored":1525175040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":525203909},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":852819968,"data_stored":852819968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1198456832,"data_stored":1198456832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1527709696,"data_stored":1527709696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2397503488,"data_stored":2397503488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1159200768,"data_stored":1159200768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1535770624,"data_stored":1535770624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2176057344,"data_stored":2176057344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":859439104,"data_stored":859439104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1169752064,"data_stored":1169752064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":894435328,"data_stored":894435328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":603324416,"data_stored":603324416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:08:39.242 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:08:39.268 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:39.290 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:08:39.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:39.295 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:08:39.395 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:08:39.428 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:39.454 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:08:39.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:39.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:08:39.538 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 45 16 83109 83093 115.389 80.25 0.00381127 0.00866061 2024-05-15T13:08:39.563 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:08:39.604 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:39.629 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:08:39.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:39.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:08:39.738 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:08:39.786 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:39.818 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:08:39.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:39.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:08:39.931 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:08:39.979 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:40.010 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:08:40.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:40.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:08:40.115 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:08:40.176 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:40.404 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:08:40.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:40.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:08:40.507 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:08:40.537 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 46 16 84390 84374 114.621 80.0625 0.00453375 0.00872034 2024-05-15T13:08:40.557 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:40.696 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:08:40.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:40.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:08:40.802 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:08:40.839 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:40.875 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:08:40.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:40.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:08:40.980 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:08:41.020 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:41.053 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:08:41.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:41.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:08:41.161 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:08:41.189 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:41.211 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:08:41.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:41.217 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:08:41.318 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:08:41.358 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:41.391 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:08:41.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:41.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:08:41.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:41 smithi177 ceph-mon[44303]: pgmap v1402: 27 pgs: 27 active+clean; 5.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-05-15T13:08:41.499 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:08:41.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:41 smithi027 ceph-mon[39795]: pgmap v1402: 27 pgs: 27 active+clean; 5.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-05-15T13:08:41.529 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:41 smithi120 ceph-mon[43961]: pgmap v1402: 27 pgs: 27 active+clean; 5.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-05-15T13:08:41.537 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 47 16 85762 85746 114.007 85.75 0.00641614 0.00876793 2024-05-15T13:08:41.541 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:41.574 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:08:41.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:41.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:08:41.682 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:08:41.726 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:41.759 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:08:41.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:41.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:08:41.866 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:08:41.911 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:41.937 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:08:41.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:41.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:08:42.042 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:08:42.080 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:42.109 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:08:42.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:42.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:08:42.218 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:08:42.272 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:42.538 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 48 16 87043 87027 113.299 80.0625 0.0139155 0.00882226 2024-05-15T13:08:42.754 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:08:42.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:42.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:08:42.783 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:08:42.783 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:08:42.867 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:08:42.899 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:42.928 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:08:42.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:42.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:08:43.036 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:08:43.071 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:43.103 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:08:43.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:43.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:08:43.189 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:08:43.211 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:08:43.285 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:43.333 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:43 smithi027 ceph-mon[39795]: pgmap v1403: 27 pgs: 27 active+clean; 5.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-05-15T13:08:43.333 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:43 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2360716656' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:08:43.333 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:43 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:08:43.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:43 smithi177 ceph-mon[44303]: pgmap v1403: 27 pgs: 27 active+clean; 5.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-05-15T13:08:43.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:43 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2360716656' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:08:43.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:43 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:08:43.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:43 smithi120 ceph-mon[43961]: pgmap v1403: 27 pgs: 27 active+clean; 5.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-05-15T13:08:43.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:43 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2360716656' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:08:43.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:43 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:08:43.538 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 49 16 88291 88275 112.579 78 0.00970997 0.00887756 2024-05-15T13:08:43.762 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:08:43.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:43.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:08:43.870 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:08:43.935 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:44 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:08:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:44 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:08:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:44 smithi177 ceph-mon[44303]: osdmap e533: 12 total, 12 up, 11 in 2024-05-15T13:08:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:44 smithi177 ceph-mon[44303]: osdmap e534: 12 total, 12 up, 11 in 2024-05-15T13:08:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:44 smithi177 ceph-mon[44303]: 1.3 scrub starts 2024-05-15T13:08:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:44 smithi177 ceph-mon[44303]: 1.3 scrub ok 2024-05-15T13:08:44.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:44 smithi177 ceph-mon[44303]: 1.7 scrub starts 2024-05-15T13:08:44.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:44 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:08:44.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:44 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:08:44.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:44 smithi027 ceph-mon[39795]: osdmap e533: 12 total, 12 up, 11 in 2024-05-15T13:08:44.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:44 smithi027 ceph-mon[39795]: osdmap e534: 12 total, 12 up, 11 in 2024-05-15T13:08:44.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:44 smithi027 ceph-mon[39795]: 1.3 scrub starts 2024-05-15T13:08:44.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:44 smithi027 ceph-mon[39795]: 1.3 scrub ok 2024-05-15T13:08:44.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:44 smithi027 ceph-mon[39795]: 1.7 scrub starts 2024-05-15T13:08:44.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:44 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:08:44.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:44 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:08:44.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:44 smithi120 ceph-mon[43961]: osdmap e533: 12 total, 12 up, 11 in 2024-05-15T13:08:44.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:44 smithi120 ceph-mon[43961]: osdmap e534: 12 total, 12 up, 11 in 2024-05-15T13:08:44.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:44 smithi120 ceph-mon[43961]: 1.3 scrub starts 2024-05-15T13:08:44.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:44 smithi120 ceph-mon[43961]: 1.3 scrub ok 2024-05-15T13:08:44.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:44 smithi120 ceph-mon[43961]: 1.7 scrub starts 2024-05-15T13:08:44.538 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 50 16 89443 89427 111.767 72 0.0101306 0.00894304 2024-05-15T13:08:45.020 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:08:45.020 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:08:45.020 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:08:45.020 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:08:45.020 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:08:45.020 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:08:45.021 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:08:45.021 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:08:45.021 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:08:45.021 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:08:45.021 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:08:45.021 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:08:45.021 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:08:45.021 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:08:45.021 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:08:45.021 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:08:45.021 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:08:45.022 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:08:45.022 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:08:45.022 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:08:45.022 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:08:45.022 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:08:45.022 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:08:45.022 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:08:45.177 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:45 smithi027 ceph-mon[39795]: 1.7 scrub ok 2024-05-15T13:08:45.177 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:45 smithi027 ceph-mon[39795]: pgmap v1406: 27 pgs: 27 active+clean; 5.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-05-15T13:08:45.177 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:45 smithi027 ceph-mon[39795]: 1.4 scrub starts 2024-05-15T13:08:45.178 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:45 smithi027 ceph-mon[39795]: 1.4 scrub ok 2024-05-15T13:08:45.178 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:45 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1955120312' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:08:45.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:45 smithi177 ceph-mon[44303]: 1.7 scrub ok 2024-05-15T13:08:45.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:45 smithi177 ceph-mon[44303]: pgmap v1406: 27 pgs: 27 active+clean; 5.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-05-15T13:08:45.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:45 smithi177 ceph-mon[44303]: 1.4 scrub starts 2024-05-15T13:08:45.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:45 smithi177 ceph-mon[44303]: 1.4 scrub ok 2024-05-15T13:08:45.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:45 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1955120312' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:08:45.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:45 smithi120 ceph-mon[43961]: 1.7 scrub ok 2024-05-15T13:08:45.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:45 smithi120 ceph-mon[43961]: pgmap v1406: 27 pgs: 27 active+clean; 5.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-05-15T13:08:45.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:45 smithi120 ceph-mon[43961]: 1.4 scrub starts 2024-05-15T13:08:45.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:45 smithi120 ceph-mon[43961]: 1.4 scrub ok 2024-05-15T13:08:45.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:45 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1955120312' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:08:45.538 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 51 16 90661 90645 111.068 76.125 0.0117863 0.00899968 2024-05-15T13:08:46.519 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:46 smithi027 ceph-mon[39795]: 6.6 scrub starts 2024-05-15T13:08:46.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:46 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T13:08:46.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:46 smithi120 ceph-mon[43961]: 6.6 scrub starts 2024-05-15T13:08:46.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:46 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T13:08:46.538 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 52 16 91783 91767 110.281 70.125 0.0100696 0.00906388 2024-05-15T13:08:46.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:46 smithi177 ceph-mon[44303]: 6.6 scrub starts 2024-05-15T13:08:46.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:46 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T13:08:47.190 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:47 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T13:08:47.190 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:47 smithi027 ceph-mon[39795]: 6.c scrub starts 2024-05-15T13:08:47.191 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:47 smithi027 ceph-mon[39795]: pgmap v1407: 27 pgs: 27 active+clean; 5.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-05-15T13:08:47.191 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:47 smithi027 ceph-mon[39795]: 6.6 scrub ok 2024-05-15T13:08:47.191 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:47 smithi027 ceph-mon[39795]: 1.2 scrub starts 2024-05-15T13:08:47.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:47 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T13:08:47.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:47 smithi120 ceph-mon[43961]: 6.c scrub starts 2024-05-15T13:08:47.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:47 smithi120 ceph-mon[43961]: pgmap v1407: 27 pgs: 27 active+clean; 5.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-05-15T13:08:47.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:47 smithi120 ceph-mon[43961]: 6.6 scrub ok 2024-05-15T13:08:47.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:47 smithi120 ceph-mon[43961]: 1.2 scrub starts 2024-05-15T13:08:47.538 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 53 16 92953 92937 109.58 73.125 0.0063648 0.00912105 2024-05-15T13:08:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:08:47.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:47.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:08:47.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:47 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T13:08:47.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:47 smithi177 ceph-mon[44303]: 6.c scrub starts 2024-05-15T13:08:47.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:47 smithi177 ceph-mon[44303]: pgmap v1407: 27 pgs: 27 active+clean; 5.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-05-15T13:08:47.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:47 smithi177 ceph-mon[44303]: 6.6 scrub ok 2024-05-15T13:08:47.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:47 smithi177 ceph-mon[44303]: 1.2 scrub starts 2024-05-15T13:08:47.742 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:08:47.772 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:47.797 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:08:47.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:47.801 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:08:47.902 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:08:47.936 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:47.958 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:08:47.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:47.962 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:08:48.063 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:08:48.102 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:48.132 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:08:48.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:48.137 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:08:48.238 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:08:48.292 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:48.299 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:08:48.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:48 smithi177 ceph-mon[44303]: 1.2 scrub ok 2024-05-15T13:08:48.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:48 smithi177 ceph-mon[44303]: 6.c scrub ok 2024-05-15T13:08:48.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:48 smithi027 ceph-mon[39795]: 1.2 scrub ok 2024-05-15T13:08:48.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:48 smithi027 ceph-mon[39795]: 6.c scrub ok 2024-05-15T13:08:48.529 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:48 smithi120 ceph-mon[43961]: 1.2 scrub ok 2024-05-15T13:08:48.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:48 smithi120 ceph-mon[43961]: 6.c scrub ok 2024-05-15T13:08:48.538 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 54 16 94073 94057 108.847 70 0.0104594 0.0091836 2024-05-15T13:08:49.356 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:49 smithi027 ceph-mon[39795]: 6.b scrub starts 2024-05-15T13:08:49.356 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:49 smithi027 ceph-mon[39795]: 1.6 scrub starts 2024-05-15T13:08:49.356 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:49 smithi027 ceph-mon[39795]: pgmap v1408: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-05-15T13:08:49.356 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:49 smithi027 ceph-mon[39795]: 1.6 scrub ok 2024-05-15T13:08:49.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:49 smithi120 ceph-mon[43961]: 6.b scrub starts 2024-05-15T13:08:49.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:49 smithi120 ceph-mon[43961]: 1.6 scrub starts 2024-05-15T13:08:49.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:49 smithi120 ceph-mon[43961]: pgmap v1408: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-05-15T13:08:49.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:49 smithi120 ceph-mon[43961]: 1.6 scrub ok 2024-05-15T13:08:49.538 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 55 16 95129 95113 108.068 66 0.00361454 0.00924147 2024-05-15T13:08:49.626 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:08:49.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:49 smithi177 ceph-mon[44303]: 6.b scrub starts 2024-05-15T13:08:49.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:49 smithi177 ceph-mon[44303]: 1.6 scrub starts 2024-05-15T13:08:49.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:49 smithi177 ceph-mon[44303]: pgmap v1408: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-05-15T13:08:49.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:49 smithi177 ceph-mon[44303]: 1.6 scrub ok 2024-05-15T13:08:49.789 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:08:49.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:49.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:08:49.904 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:08:49.940 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:49.964 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:08:49.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:49.967 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:08:50.047 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:08:50.047 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:08:50.068 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:08:50.115 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:50.140 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:08:50.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:50.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:08:50.226 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:50 smithi177 ceph-mon[44303]: 1.8 scrub starts 2024-05-15T13:08:50.227 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:50 smithi177 ceph-mon[44303]: 1.8 scrub ok 2024-05-15T13:08:50.227 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:50 smithi177 ceph-mon[44303]: 1.9 scrub starts 2024-05-15T13:08:50.227 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:50 smithi177 ceph-mon[44303]: 1.9 scrub ok 2024-05-15T13:08:50.227 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:50 smithi177 ceph-mon[44303]: 6.b scrub ok 2024-05-15T13:08:50.244 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:08:50.320 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:50.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:50 smithi027 ceph-mon[39795]: 1.8 scrub starts 2024-05-15T13:08:50.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:50 smithi027 ceph-mon[39795]: 1.8 scrub ok 2024-05-15T13:08:50.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:50 smithi027 ceph-mon[39795]: 1.9 scrub starts 2024-05-15T13:08:50.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:50 smithi027 ceph-mon[39795]: 1.9 scrub ok 2024-05-15T13:08:50.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:50 smithi027 ceph-mon[39795]: 6.b scrub ok 2024-05-15T13:08:50.529 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:50 smithi120 ceph-mon[43961]: 1.8 scrub starts 2024-05-15T13:08:50.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:50 smithi120 ceph-mon[43961]: 1.8 scrub ok 2024-05-15T13:08:50.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:50 smithi120 ceph-mon[43961]: 1.9 scrub starts 2024-05-15T13:08:50.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:50 smithi120 ceph-mon[43961]: 1.9 scrub ok 2024-05-15T13:08:50.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:50 smithi120 ceph-mon[43961]: 6.b scrub ok 2024-05-15T13:08:50.538 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 56 16 96186 96170 107.317 66.0625 0.00962023 0.00931423 2024-05-15T13:08:50.789 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:08:50.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:50.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:08:50.899 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:08:50.928 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:50.959 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:08:50.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:50.963 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:08:51.064 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:08:51.101 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:51.124 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:08:51.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:51.130 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:08:51.231 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:08:51.240 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:08:51.277 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-05-15T13:08:51.292 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:08:51.314 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:51.526 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:08:51.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:51 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:08:51.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:51 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:08:51.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:51 smithi027 ceph-mon[39795]: pgmap v1409: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-05-15T13:08:51.529 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:51 smithi027 ceph-mon[39795]: 1.1 scrub starts 2024-05-15T13:08:51.529 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:51 smithi027 ceph-mon[39795]: osdmap e535: 12 total, 12 up, 11 in 2024-05-15T13:08:51.529 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:51 smithi027 ceph-mon[39795]: 1.1 scrub ok 2024-05-15T13:08:51.529 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:51 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:08:51.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:51 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:08:51.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:51 smithi120 ceph-mon[43961]: pgmap v1409: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-05-15T13:08:51.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:51 smithi120 ceph-mon[43961]: 1.1 scrub starts 2024-05-15T13:08:51.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:51 smithi120 ceph-mon[43961]: osdmap e535: 12 total, 12 up, 11 in 2024-05-15T13:08:51.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:51 smithi120 ceph-mon[43961]: 1.1 scrub ok 2024-05-15T13:08:51.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:51.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:08:51.540 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 57 16 97329 97313 106.688 71.4375 0.0176311 0.00936858 2024-05-15T13:08:51.634 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:08:51.662 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:51 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:08:51.662 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:51 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:08:51.662 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:51 smithi177 ceph-mon[44303]: pgmap v1409: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-05-15T13:08:51.662 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:51 smithi177 ceph-mon[44303]: 1.1 scrub starts 2024-05-15T13:08:51.662 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:51 smithi177 ceph-mon[44303]: osdmap e535: 12 total, 12 up, 11 in 2024-05-15T13:08:51.662 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:51 smithi177 ceph-mon[44303]: 1.1 scrub ok 2024-05-15T13:08:51.664 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:51.687 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:08:51.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:51.691 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:08:51.793 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:08:51.829 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:51.855 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:08:51.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:51.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:08:51.962 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:08:51.989 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:52.010 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:08:52.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:52.013 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:08:52.114 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:08:52.148 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:52.221 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:08:52.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:52.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:08:52.245 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:52 smithi177 ceph-mon[44303]: 1.a scrub starts 2024-05-15T13:08:52.245 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:52 smithi177 ceph-mon[44303]: 1.a scrub ok 2024-05-15T13:08:52.246 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:52 smithi120 ceph-mon[43961]: 1.a scrub starts 2024-05-15T13:08:52.246 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:52 smithi120 ceph-mon[43961]: 1.a scrub ok 2024-05-15T13:08:52.329 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:08:52.366 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:52.393 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:08:52.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:52.399 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:08:52.500 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:08:52.519 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:52 smithi027 ceph-mon[39795]: 1.a scrub starts 2024-05-15T13:08:52.519 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:52 smithi027 ceph-mon[39795]: 1.a scrub ok 2024-05-15T13:08:52.539 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 58 16 98361 98345 105.96 64.5 0.00727702 0.00943283 2024-05-15T13:08:52.545 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:52.787 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:08:52.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:52.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:08:52.841 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:08:52.856 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:08:52.900 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:08:52.943 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:52.969 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:08:52.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:52.977 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:08:53.078 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:08:53.115 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:53.143 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:08:53.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:53.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:08:53.248 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:08:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:53.344 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:08:53.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:53.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:08:53.452 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:08:53.498 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:53 smithi027 ceph-mon[39795]: pgmap v1411: 27 pgs: 27 active+clean; 5.8 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-05-15T13:08:53.498 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:53 smithi027 ceph-mon[39795]: 1.5 scrub starts 2024-05-15T13:08:53.498 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:53 smithi027 ceph-mon[39795]: 1.5 scrub ok 2024-05-15T13:08:53.499 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:53 smithi027 ceph-mon[39795]: from='client.15879 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:08:53.500 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:53.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:53 smithi120 ceph-mon[43961]: pgmap v1411: 27 pgs: 27 active+clean; 5.8 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-05-15T13:08:53.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:53 smithi120 ceph-mon[43961]: 1.5 scrub starts 2024-05-15T13:08:53.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:53 smithi120 ceph-mon[43961]: 1.5 scrub ok 2024-05-15T13:08:53.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:53 smithi120 ceph-mon[43961]: from='client.15879 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:08:53.539 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 59 16 99510 99494 105.382 71.8125 0.0112289 0.00948502 2024-05-15T13:08:53.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:53 smithi177 ceph-mon[44303]: pgmap v1411: 27 pgs: 27 active+clean; 5.8 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-05-15T13:08:53.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:53 smithi177 ceph-mon[44303]: 1.5 scrub starts 2024-05-15T13:08:53.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:53 smithi177 ceph-mon[44303]: 1.5 scrub ok 2024-05-15T13:08:53.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:53 smithi177 ceph-mon[44303]: from='client.15879 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:08:53.946 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:08:53.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:53.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:08:54.052 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:08:54.110 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:54.183 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:08:54.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:54.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:08:54.292 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:08:54.325 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:54.365 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:08:54.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:54.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:08:54.471 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:08:54.499 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:54.539 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T13:08:54.550739+0000 min lat: 0.00114056 max lat: 0.299471 avg lat: 0.0095375 2024-05-15T13:08:54.539 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:08:54.539 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 60 16 100644 100628 104.806 70.875 0.0119476 0.0095375 2024-05-15T13:08:54.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:54 smithi027 ceph-mon[39795]: osdmap e536: 12 total, 12 up, 11 in 2024-05-15T13:08:54.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:54 smithi027 ceph-mon[39795]: pgmap v1413: 27 pgs: 27 active+clean; 6.0 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-05-15T13:08:54.779 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:54 smithi120 ceph-mon[43961]: osdmap e536: 12 total, 12 up, 11 in 2024-05-15T13:08:54.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:54 smithi120 ceph-mon[43961]: pgmap v1413: 27 pgs: 27 active+clean; 6.0 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-05-15T13:08:54.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:54 smithi177 ceph-mon[44303]: osdmap e536: 12 total, 12 up, 11 in 2024-05-15T13:08:54.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:54 smithi177 ceph-mon[44303]: pgmap v1413: 27 pgs: 27 active+clean; 6.0 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-05-15T13:08:55.462 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:55 smithi027 ceph-mon[39795]: 6.1 scrub starts 2024-05-15T13:08:55.539 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 61 16 101856 101840 104.33 75.75 0.0264337 0.00957887 2024-05-15T13:08:55.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:55 smithi120 ceph-mon[43961]: 6.1 scrub starts 2024-05-15T13:08:55.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:55 smithi177 ceph-mon[44303]: 6.1 scrub starts 2024-05-15T13:08:56.539 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 62 16 103084 103068 103.885 76.75 0.00708752 0.00962188 2024-05-15T13:08:56.667 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:08:56.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:56.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:08:56.772 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:08:56.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:56 smithi027 ceph-mon[39795]: 6.1 scrub ok 2024-05-15T13:08:56.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:56 smithi027 ceph-mon[39795]: pgmap v1414: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 6.1 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-05-15T13:08:56.779 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:56 smithi120 ceph-mon[43961]: 6.1 scrub ok 2024-05-15T13:08:56.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:56 smithi120 ceph-mon[43961]: pgmap v1414: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 6.1 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-05-15T13:08:56.804 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:56 smithi177 ceph-mon[44303]: 6.1 scrub ok 2024-05-15T13:08:56.805 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:56 smithi177 ceph-mon[44303]: pgmap v1414: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 6.1 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-05-15T13:08:56.809 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:56.832 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:08:56.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:56.836 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:08:56.937 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:08:57.005 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:57.254 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:08:57.300 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:08:57.333 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:08:57.362 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:08:57.389 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:08:57.418 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:08:57.441 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:08:57.462 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:08:57.483 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:08:57.503 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:08:57.523 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:08:57.539 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 63 16 104067 104051 103.211 61.4375 0.200936 0.00966327 2024-05-15T13:08:57.553 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:08:57.560 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:08:57.561 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:08:57.561 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:08:57.561 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:08:57.561 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:08:57.561 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:08:57.561 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:08:57.561 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:08:57.561 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:08:57.561 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:08:57.561 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:08:57.561 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:08:57.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:57 smithi027 ceph-mon[39795]: 6.d scrub starts 2024-05-15T13:08:57.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:57 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1264197826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:08:57.779 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:57 smithi120 ceph-mon[43961]: 6.d scrub starts 2024-05-15T13:08:57.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:57 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1264197826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:08:57.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:57 smithi177 ceph-mon[44303]: 6.d scrub starts 2024-05-15T13:08:57.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:57 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1264197826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:08:58.495 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:58 smithi027 ceph-mon[39795]: 6.d scrub ok 2024-05-15T13:08:58.495 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:58 smithi027 ceph-mon[39795]: pgmap v1415: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 6.2 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-05-15T13:08:58.540 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 64 16 105226 105210 102.729 72.4375 0.00588639 0.00973058 2024-05-15T13:08:58.601 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":1411,"stamp":"2024-05-15T13:08:52.103431+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6243156512,"num_objects":95256,"num_object_clones":0,"num_object_copies":285768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95254,"num_write_kb":6096256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96783,"ondisk_log_size":96783,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":23768936,"kb_used_data":18258764,"kb_used_omap":17,"kb_used_meta":5510062,"kb_avail":1007427736,"statfs":{"total":1055945392128,"available":1031606001664,"internally_reserved":0,"allocated":18696974336,"data_stored":18692694966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17551,"internal_metadata":5642304369},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":649986048,"num_objects":9918,"num_object_clones":0,"num_object_copies":29754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9918,"num_write_kb":634752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.661159"},"pg_stats":[{"pgid":"6.c","version":"535'6161","reported_seq":6291,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-05-15T13:08:51.921642+0000","last_change":"2024-05-15T13:08:47.669502+0000","last_active":"2024-05-15T13:08:51.921642+0000","last_peered":"2024-05-15T13:08:51.921642+0000","last_clean":"2024-05-15T13:08:51.921642+0000","last_became_active":"2024-05-15T13:07:43.860763+0000","last_became_peered":"2024-05-15T13:07:43.860763+0000","last_unstale":"2024-05-15T13:08:51.921642+0000","last_undegraded":"2024-05-15T13:08:51.921642+0000","last_fullsized":"2024-05-15T13:08:51.921642+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"534'5905","last_scrub_stamp":"2024-05-15T13:08:47.669339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:08:47.669339+0000","objects_scrubbed":5848,"log_size":6161,"log_dups_size":0,"ondisk_log_size":6161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:10:16.955750+0000","scrub_duration":1726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403767296,"num_objects":6161,"num_object_clones":0,"num_object_copies":18483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6161,"num_write_kb":394304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"535'6208","reported_seq":6337,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-05-15T13:08:51.857520+0000","last_change":"2024-05-15T13:08:49.610032+0000","last_active":"2024-05-15T13:08:51.857520+0000","last_peered":"2024-05-15T13:08:51.857520+0000","last_clean":"2024-05-15T13:08:51.857520+0000","last_became_active":"2024-05-15T13:07:43.863906+0000","last_became_peered":"2024-05-15T13:07:43.863906+0000","last_unstale":"2024-05-15T13:08:51.857520+0000","last_undegraded":"2024-05-15T13:08:51.857520+0000","last_fullsized":"2024-05-15T13:08:51.857520+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"534'6048","last_scrub_stamp":"2024-05-15T13:08:49.609807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:08:49.609807+0000","objects_scrubbed":5994,"log_size":6208,"log_dups_size":0,"ondisk_log_size":6208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:10:10.476067+0000","scrub_duration":1697,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406847488,"num_objects":6208,"num_object_clones":0,"num_object_copies":18624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6208,"num_write_kb":397312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"534'5807","reported_seq":5866,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-05-15T13:08:48.183782+0000","last_change":"2024-05-15T13:07:43.856272+0000","last_active":"2024-05-15T13:08:48.183782+0000","last_peered":"2024-05-15T13:08:48.183782+0000","last_clean":"2024-05-15T13:08:48.183782+0000","last_became_active":"2024-05-15T13:07:43.855799+0000","last_became_peered":"2024-05-15T13:07:43.855799+0000","last_unstale":"2024-05-15T13:08:48.183782+0000","last_undegraded":"2024-05-15T13:08:48.183782+0000","last_fullsized":"2024-05-15T13:08:48.183782+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":5807,"log_dups_size":0,"ondisk_log_size":5807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:55.927298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380567552,"num_objects":5807,"num_object_clones":0,"num_object_copies":17421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5807,"num_write_kb":371648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"534'5844","reported_seq":5904,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-05-15T13:08:49.788749+0000","last_change":"2024-05-15T13:07:44.416156+0000","last_active":"2024-05-15T13:08:49.788749+0000","last_peered":"2024-05-15T13:08:49.788749+0000","last_clean":"2024-05-15T13:08:49.788749+0000","last_became_active":"2024-05-15T13:07:44.415859+0000","last_became_peered":"2024-05-15T13:07:44.415859+0000","last_unstale":"2024-05-15T13:08:49.788749+0000","last_undegraded":"2024-05-15T13:08:49.788749+0000","last_fullsized":"2024-05-15T13:08:49.788749+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":5844,"log_dups_size":0,"ondisk_log_size":5844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382992384,"num_objects":5844,"num_object_clones":0,"num_object_copies":17532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5844,"num_write_kb":374016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"534'5782","reported_seq":5842,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-05-15T13:08:47.729948+0000","last_change":"2024-05-15T13:07:43.858333+0000","last_active":"2024-05-15T13:08:47.729948+0000","last_peered":"2024-05-15T13:08:47.729948+0000","last_clean":"2024-05-15T13:08:47.729948+0000","last_became_active":"2024-05-15T13:07:43.858034+0000","last_became_peered":"2024-05-15T13:07:43.858034+0000","last_unstale":"2024-05-15T13:08:47.729948+0000","last_undegraded":"2024-05-15T13:08:47.729948+0000","last_fullsized":"2024-05-15T13:08:47.729948+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":5782,"log_dups_size":0,"ondisk_log_size":5782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:57.970835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378929152,"num_objects":5782,"num_object_clones":0,"num_object_copies":17346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5782,"num_write_kb":370048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,6],"acting":[11,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.d","version":"534'5900","reported_seq":5960,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-05-15T13:08:48.178661+0000","last_change":"2024-05-15T13:07:44.092209+0000","last_active":"2024-05-15T13:08:48.178661+0000","last_peered":"2024-05-15T13:08:48.178661+0000","last_clean":"2024-05-15T13:08:48.178661+0000","last_became_active":"2024-05-15T13:07:44.091933+0000","last_became_peered":"2024-05-15T13:07:44.091933+0000","last_unstale":"2024-05-15T13:08:48.178661+0000","last_undegraded":"2024-05-15T13:08:48.178661+0000","last_fullsized":"2024-05-15T13:08:48.178661+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":5900,"log_dups_size":0,"ondisk_log_size":5900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:55.177111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386662400,"num_objects":5900,"num_object_clones":0,"num_object_copies":17700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5900,"num_write_kb":377600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"94'139","reported_seq":918,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-05-15T13:08:50.245781+0000","last_change":"2024-05-15T13:06:55.617237+0000","last_active":"2024-05-15T13:08:50.245781+0000","last_peered":"2024-05-15T13:08:50.245781+0000","last_clean":"2024-05-15T13:08:50.245781+0000","last_became_active":"2024-05-15T13:06:45.407469+0000","last_became_peered":"2024-05-15T13:06:45.407469+0000","last_unstale":"2024-05-15T13:08:50.245781+0000","last_undegraded":"2024-05-15T13:08:50.245781+0000","last_fullsized":"2024-05-15T13:08:50.245781+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:06:55.617099+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:06:55.617099+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.e","version":"534'5987","reported_seq":6047,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-05-15T13:08:50.002845+0000","last_change":"2024-05-15T13:07:43.864202+0000","last_active":"2024-05-15T13:08:50.002845+0000","last_peered":"2024-05-15T13:08:50.002845+0000","last_clean":"2024-05-15T13:08:50.002845+0000","last_became_active":"2024-05-15T13:07:43.863923+0000","last_became_peered":"2024-05-15T13:07:43.863923+0000","last_unstale":"2024-05-15T13:08:50.002845+0000","last_undegraded":"2024-05-15T13:08:50.002845+0000","last_fullsized":"2024-05-15T13:08:50.002845+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":5987,"log_dups_size":0,"ondisk_log_size":5987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:59.591350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392364032,"num_objects":5987,"num_object_clones":0,"num_object_copies":17961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5987,"num_write_kb":383168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"94'139","reported_seq":924,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-05-15T13:08:50.245798+0000","last_change":"2024-05-15T13:08:49.259919+0000","last_active":"2024-05-15T13:08:50.245798+0000","last_peered":"2024-05-15T13:08:50.245798+0000","last_clean":"2024-05-15T13:08:50.245798+0000","last_became_active":"2024-05-15T13:06:46.424667+0000","last_became_peered":"2024-05-15T13:06:46.424667+0000","last_unstale":"2024-05-15T13:08:50.245798+0000","last_undegraded":"2024-05-15T13:08:50.245798+0000","last_fullsized":"2024-05-15T13:08:50.245798+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:08:49.259818+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:08:49.259818+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:10:14.278382+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.f","version":"534'5848","reported_seq":5908,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-05-15T13:08:47.745331+0000","last_change":"2024-05-15T13:07:44.416852+0000","last_active":"2024-05-15T13:08:47.745331+0000","last_peered":"2024-05-15T13:08:47.745331+0000","last_clean":"2024-05-15T13:08:47.745331+0000","last_became_active":"2024-05-15T13:07:44.416528+0000","last_became_peered":"2024-05-15T13:07:44.416528+0000","last_unstale":"2024-05-15T13:08:47.745331+0000","last_undegraded":"2024-05-15T13:08:47.745331+0000","last_fullsized":"2024-05-15T13:08:47.745331+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":5848,"log_dups_size":0,"ondisk_log_size":5848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:09:04.289381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383254528,"num_objects":5848,"num_object_clones":0,"num_object_copies":17544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5848,"num_write_kb":374272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"94'139","reported_seq":914,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-05-15T13:08:43.449596+0000","last_change":"2024-05-15T13:06:58.489453+0000","last_active":"2024-05-15T13:08:43.449596+0000","last_peered":"2024-05-15T13:08:43.449596+0000","last_clean":"2024-05-15T13:08:43.449596+0000","last_became_active":"2024-05-15T13:06:46.426908+0000","last_became_peered":"2024-05-15T13:06:46.426908+0000","last_unstale":"2024-05-15T13:08:43.449596+0000","last_undegraded":"2024-05-15T13:08:43.449596+0000","last_fullsized":"2024-05-15T13:08:43.449596+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:06:58.489366+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:06:58.489366+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:47.664975+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"535'5958","reported_seq":6019,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-05-15T13:08:51.802902+0000","last_change":"2024-05-15T13:07:43.865627+0000","last_active":"2024-05-15T13:08:51.802902+0000","last_peered":"2024-05-15T13:08:51.802902+0000","last_clean":"2024-05-15T13:08:51.802902+0000","last_became_active":"2024-05-15T13:07:43.865339+0000","last_became_peered":"2024-05-15T13:07:43.865339+0000","last_unstale":"2024-05-15T13:08:51.802902+0000","last_undegraded":"2024-05-15T13:08:51.802902+0000","last_fullsized":"2024-05-15T13:08:51.802902+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":5958,"log_dups_size":0,"ondisk_log_size":5958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:55.567826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390463488,"num_objects":5958,"num_object_clones":0,"num_object_copies":17874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5958,"num_write_kb":381312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"94'139","reported_seq":932,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-05-15T13:08:50.233784+0000","last_change":"2024-05-15T13:08:43.693960+0000","last_active":"2024-05-15T13:08:50.233784+0000","last_peered":"2024-05-15T13:08:50.233784+0000","last_clean":"2024-05-15T13:08:50.233784+0000","last_became_active":"2024-05-15T13:06:45.402594+0000","last_became_peered":"2024-05-15T13:06:45.402594+0000","last_unstale":"2024-05-15T13:08:50.233784+0000","last_undegraded":"2024-05-15T13:08:50.233784+0000","last_fullsized":"2024-05-15T13:08:50.233784+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:08:43.693843+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:08:43.693843+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:10:07.564342+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"534'5973","reported_seq":6033,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-05-15T13:08:48.757481+0000","last_change":"2024-05-15T13:07:44.416267+0000","last_active":"2024-05-15T13:08:48.757481+0000","last_peered":"2024-05-15T13:08:48.757481+0000","last_clean":"2024-05-15T13:08:48.757481+0000","last_became_active":"2024-05-15T13:07:44.415987+0000","last_became_peered":"2024-05-15T13:07:44.415987+0000","last_unstale":"2024-05-15T13:08:48.757481+0000","last_undegraded":"2024-05-15T13:08:48.757481+0000","last_fullsized":"2024-05-15T13:08:48.757481+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":5973,"log_dups_size":0,"ondisk_log_size":5973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:09:06.262336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391446528,"num_objects":5973,"num_object_clones":0,"num_object_copies":17919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5973,"num_write_kb":382272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"94'139","reported_seq":921,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-05-15T13:08:46.849115+0000","last_change":"2024-05-15T13:08:46.849155+0000","last_active":"2024-05-15T13:08:46.849115+0000","last_peered":"2024-05-15T13:08:46.849115+0000","last_clean":"2024-05-15T13:08:46.849115+0000","last_became_active":"2024-05-15T13:06:46.420025+0000","last_became_peered":"2024-05-15T13:06:46.420025+0000","last_unstale":"2024-05-15T13:08:46.849115+0000","last_undegraded":"2024-05-15T13:08:46.849115+0000","last_fullsized":"2024-05-15T13:08:46.849115+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:08:46.849024+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:08:46.849024+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:09:47.075424+0000","scrub_duration":196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,8],"acting":[0,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"535'6129","reported_seq":6255,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-05-15T13:08:51.901857+0000","last_change":"2024-05-15T13:08:46.534942+0000","last_active":"2024-05-15T13:08:51.901857+0000","last_peered":"2024-05-15T13:08:51.901857+0000","last_clean":"2024-05-15T13:08:51.901857+0000","last_became_active":"2024-05-15T13:07:43.860023+0000","last_became_peered":"2024-05-15T13:07:43.860023+0000","last_unstale":"2024-05-15T13:08:51.901857+0000","last_undegraded":"2024-05-15T13:08:51.901857+0000","last_fullsized":"2024-05-15T13:08:51.901857+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"534'5731","last_scrub_stamp":"2024-05-15T13:08:46.534625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:08:46.534625+0000","objects_scrubbed":5687,"log_size":6129,"log_dups_size":0,"ondisk_log_size":6129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:10:07.102842+0000","scrub_duration":1562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401670144,"num_objects":6129,"num_object_clones":0,"num_object_copies":18387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6129,"num_write_kb":392256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"94'139","reported_seq":925,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-05-15T13:08:50.256631+0000","last_change":"2024-05-15T13:08:50.256631+0000","last_active":"2024-05-15T13:08:50.256631+0000","last_peered":"2024-05-15T13:08:50.256631+0000","last_clean":"2024-05-15T13:08:50.256631+0000","last_became_active":"2024-05-15T13:06:46.445091+0000","last_became_peered":"2024-05-15T13:06:46.445091+0000","last_unstale":"2024-05-15T13:08:50.256631+0000","last_undegraded":"2024-05-15T13:08:50.256631+0000","last_fullsized":"2024-05-15T13:08:50.256631+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:08:50.256485+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:08:50.256485+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:09:52.766807+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,0],"acting":[9,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.0","version":"534'5991","reported_seq":6051,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-05-15T13:08:49.750365+0000","last_change":"2024-05-15T13:07:43.859654+0000","last_active":"2024-05-15T13:08:49.750365+0000","last_peered":"2024-05-15T13:08:49.750365+0000","last_clean":"2024-05-15T13:08:49.750365+0000","last_became_active":"2024-05-15T13:07:43.859339+0000","last_became_peered":"2024-05-15T13:07:43.859339+0000","last_unstale":"2024-05-15T13:08:49.750365+0000","last_undegraded":"2024-05-15T13:08:49.750365+0000","last_fullsized":"2024-05-15T13:08:49.750365+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":5991,"log_dups_size":0,"ondisk_log_size":5991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:09:04.767546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392626176,"num_objects":5991,"num_object_clones":0,"num_object_copies":17973,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5991,"num_write_kb":383424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"94'139","reported_seq":681,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-05-15T13:08:43.919163+0000","last_change":"2024-05-15T13:08:43.919202+0000","last_active":"2024-05-15T13:08:43.919163+0000","last_peered":"2024-05-15T13:08:43.919163+0000","last_clean":"2024-05-15T13:08:43.919163+0000","last_became_active":"2024-05-15T13:06:46.425852+0000","last_became_peered":"2024-05-15T13:06:46.425852+0000","last_unstale":"2024-05-15T13:08:43.919163+0000","last_undegraded":"2024-05-15T13:08:43.919163+0000","last_fullsized":"2024-05-15T13:08:43.919163+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:08:43.919077+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:08:43.919077+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:10:08.292862+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"534'6000","reported_seq":6058,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-05-15T13:08:50.099337+0000","last_change":"2024-05-15T13:07:44.416346+0000","last_active":"2024-05-15T13:08:50.099337+0000","last_peered":"2024-05-15T13:08:50.099337+0000","last_clean":"2024-05-15T13:08:50.099337+0000","last_became_active":"2024-05-15T13:07:44.416072+0000","last_became_peered":"2024-05-15T13:07:44.416072+0000","last_unstale":"2024-05-15T13:08:50.099337+0000","last_undegraded":"2024-05-15T13:08:50.099337+0000","last_fullsized":"2024-05-15T13:08:50.099337+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":6000,"log_dups_size":0,"ondisk_log_size":6000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:53.591543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216000,"num_objects":6000,"num_object_clones":0,"num_object_copies":18000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6000,"num_write_kb":384000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"94'139","reported_seq":925,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-05-15T13:08:50.245787+0000","last_change":"2024-05-15T13:08:48.185495+0000","last_active":"2024-05-15T13:08:50.245787+0000","last_peered":"2024-05-15T13:08:50.245787+0000","last_clean":"2024-05-15T13:08:50.245787+0000","last_became_active":"2024-05-15T13:06:46.419250+0000","last_became_peered":"2024-05-15T13:06:46.419250+0000","last_unstale":"2024-05-15T13:08:50.245787+0000","last_undegraded":"2024-05-15T13:08:50.245787+0000","last_fullsized":"2024-05-15T13:08:50.245787+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:08:48.185367+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:08:48.185367+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:10:17.614857+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.2","version":"534'5942","reported_seq":6002,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-05-15T13:08:50.047997+0000","last_change":"2024-05-15T13:07:43.858548+0000","last_active":"2024-05-15T13:08:50.047997+0000","last_peered":"2024-05-15T13:08:50.047997+0000","last_clean":"2024-05-15T13:08:50.047997+0000","last_became_active":"2024-05-15T13:07:43.858107+0000","last_became_peered":"2024-05-15T13:07:43.858107+0000","last_unstale":"2024-05-15T13:08:50.047997+0000","last_undegraded":"2024-05-15T13:08:50.047997+0000","last_fullsized":"2024-05-15T13:08:50.047997+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":5942,"log_dups_size":0,"ondisk_log_size":5942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:08:56.120553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389414912,"num_objects":5942,"num_object_clones":0,"num_object_copies":17826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5942,"num_write_kb":380288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"94'139","reported_seq":662,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-05-15T13:08:50.250124+0000","last_change":"2024-05-15T13:06:59.242408+0000","last_active":"2024-05-15T13:08:50.250124+0000","last_peered":"2024-05-15T13:08:50.250124+0000","last_clean":"2024-05-15T13:08:50.250124+0000","last_became_active":"2024-05-15T13:06:46.420188+0000","last_became_peered":"2024-05-15T13:06:46.420188+0000","last_unstale":"2024-05-15T13:08:50.250124+0000","last_undegraded":"2024-05-15T13:08:50.250124+0000","last_fullsized":"2024-05-15T13:08:50.250124+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:06:59.242294+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:06:59.242294+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.3","version":"534'5849","reported_seq":5906,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-05-15T13:08:48.737856+0000","last_change":"2024-05-15T13:07:43.855924+0000","last_active":"2024-05-15T13:08:48.737856+0000","last_peered":"2024-05-15T13:08:48.737856+0000","last_clean":"2024-05-15T13:08:48.737856+0000","last_became_active":"2024-05-15T13:07:43.855524+0000","last_became_peered":"2024-05-15T13:07:43.855524+0000","last_unstale":"2024-05-15T13:08:48.737856+0000","last_undegraded":"2024-05-15T13:08:48.737856+0000","last_fullsized":"2024-05-15T13:08:48.737856+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":5849,"log_dups_size":0,"ondisk_log_size":5849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:09:10.856367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383320064,"num_objects":5849,"num_object_clones":0,"num_object_copies":17547,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5849,"num_write_kb":374336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,8],"acting":[6,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"94'139","reported_seq":923,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-05-15T13:08:44.269095+0000","last_change":"2024-05-15T13:08:44.269095+0000","last_active":"2024-05-15T13:08:44.269095+0000","last_peered":"2024-05-15T13:08:44.269095+0000","last_clean":"2024-05-15T13:08:44.269095+0000","last_became_active":"2024-05-15T13:06:45.409132+0000","last_became_peered":"2024-05-15T13:06:45.409132+0000","last_unstale":"2024-05-15T13:08:44.269095+0000","last_undegraded":"2024-05-15T13:08:44.269095+0000","last_fullsized":"2024-05-15T13:08:44.269095+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:08:44.269001+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:08:44.269001+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:09:53.896065+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.7","version":"534'5875","reported_seq":5935,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-05-15T13:08:48.183677+0000","last_change":"2024-05-15T13:07:43.861203+0000","last_active":"2024-05-15T13:08:48.183677+0000","last_peered":"2024-05-15T13:08:48.183677+0000","last_clean":"2024-05-15T13:08:48.183677+0000","last_became_active":"2024-05-15T13:07:43.860859+0000","last_became_peered":"2024-05-15T13:07:43.860859+0000","last_unstale":"2024-05-15T13:08:48.183677+0000","last_undegraded":"2024-05-15T13:08:48.183677+0000","last_fullsized":"2024-05-15T13:08:48.183677+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":509,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:07:42.834678+0000","last_clean_scrub_stamp":"2024-05-15T13:07:42.834678+0000","objects_scrubbed":0,"log_size":5875,"log_dups_size":0,"ondisk_log_size":5875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:09:09.485083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385024000,"num_objects":5875,"num_object_clones":0,"num_object_copies":17625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5875,"num_write_kb":376000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":932,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-05-15T13:08:50.233655+0000","last_change":"2024-05-15T13:08:45.779271+0000","last_active":"2024-05-15T13:08:50.233655+0000","last_peered":"2024-05-15T13:08:50.233655+0000","last_clean":"2024-05-15T13:08:50.233655+0000","last_became_active":"2024-05-15T13:06:42.028286+0000","last_became_peered":"2024-05-15T13:06:42.028286+0000","last_unstale":"2024-05-15T13:08:50.233655+0000","last_undegraded":"2024-05-15T13:08:50.233655+0000","last_fullsized":"2024-05-15T13:08:50.233655+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:08:45.779150+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:08:45.779150+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:10:06.683237+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":6242566144,"num_objects":95254,"num_object_clones":0,"num_object_copies":285762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95254,"num_write_kb":6096256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18784583680,"data_stored":18784583680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95254,"ondisk_log_size":95254,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056921,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1187320,"kb_used_data":736880,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":92557832,"statfs":{"total":95995035648,"available":94779219968,"internally_reserved":0,"allocated":754565120,"data_stored":754178267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":461240774},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351261,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1651912,"kb_used_data":1173692,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":92093240,"statfs":{"total":95995035648,"available":94303477760,"internally_reserved":0,"allocated":1201860608,"data_stored":1201469290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":489683392},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514787,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2136888,"kb_used_data":1533744,"kb_used_omap":1,"kb_used_meta":603134,"kb_avail":91608264,"statfs":{"total":95995035648,"available":93806862336,"internally_reserved":0,"allocated":1570553856,"data_stored":1570167003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":617609664},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874536,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1456248,"kb_used_data":1132144,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92288904,"statfs":{"total":95995035648,"available":94503837696,"internally_reserved":0,"allocated":1159315456,"data_stored":1158928611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":240518168876,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430513,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3149432,"kb_used_data":2622640,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":90595720,"statfs":{"total":95995035648,"available":92770017280,"internally_reserved":0,"allocated":2685583360,"data_stored":2685196515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":539425222},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575180,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2161028,"kb_used_data":1557884,"kb_used_omap":1,"kb_used_meta":603134,"kb_avail":91584124,"statfs":{"total":95995035648,"available":93782142976,"internally_reserved":0,"allocated":1595273216,"data_stored":1594881906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":617609669},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738708,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1693176,"kb_used_data":1134448,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92051976,"statfs":{"total":95995035648,"available":94261223424,"internally_reserved":0,"allocated":1161674752,"data_stored":1161287907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411656,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2211456,"kb_used_data":1855416,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":91533696,"statfs":{"total":95995035648,"available":93730504704,"internally_reserved":0,"allocated":1899945984,"data_stored":1899555587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150018,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3678796,"kb_used_data":3128388,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":90066356,"statfs":{"total":95995035648,"available":92227948544,"internally_reserved":0,"allocated":3203469312,"data_stored":3203074450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":563608007},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986493,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2063032,"kb_used_data":1522352,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":91682120,"statfs":{"total":95995035648,"available":93882490880,"internally_reserved":0,"allocated":1558888448,"data_stored":1558501603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692151,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2379648,"kb_used_data":1861176,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":91365504,"statfs":{"total":95995035648,"available":93558276096,"internally_reserved":0,"allocated":1905844224,"data_stored":1905453827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1162936320,"data_stored":1162936320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1598947328,"data_stored":1598947328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1914306560,"data_stored":1914306560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3203006464,"data_stored":3203006464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1566441472,"data_stored":1566441472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1916272640,"data_stored":1916272640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2711814144,"data_stored":2711814144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1167982592,"data_stored":1167982592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1582497792,"data_stored":1582497792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1201537024,"data_stored":1201537024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":758841344,"data_stored":758841344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:08:58.603 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-05-15T13:08:58.603 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:08:58.604 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-05-15T13:08:58.604 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-15T13:08:58.604 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-15T13:08:58.604 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-05-15T13:08:58.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:58 smithi120 ceph-mon[43961]: 6.d scrub ok 2024-05-15T13:08:58.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:58 smithi120 ceph-mon[43961]: pgmap v1415: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 6.2 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-05-15T13:08:58.885 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:08:58.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:58.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:08:58.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:58 smithi177 ceph-mon[44303]: 6.d scrub ok 2024-05-15T13:08:58.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:58 smithi177 ceph-mon[44303]: pgmap v1415: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 6.2 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-05-15T13:08:58.974 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:08:58.990 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:08:59.022 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:59.052 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:08:59.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:59.055 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:08:59.156 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:08:59.190 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:08:59.194 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:59.218 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:08:59.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:59.221 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:08:59.322 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:08:59.358 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:59.378 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:08:59.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:59.381 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:08:59.482 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:08:59.516 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:08:59.537 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:08:59.539 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 65 16 106184 106168 102.07 59.875 0.00995265 0.00979277 2024-05-15T13:08:59.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:08:59.543 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:08:59.644 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:08:59.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:59 smithi177 ceph-mon[44303]: 6.8 scrub starts 2024-05-15T13:08:59.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:08:59 smithi177 ceph-mon[44303]: 6.4 scrub starts 2024-05-15T13:08:59.708 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:59 smithi027 ceph-mon[39795]: 6.8 scrub starts 2024-05-15T13:08:59.708 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:08:59 smithi027 ceph-mon[39795]: 6.4 scrub starts 2024-05-15T13:08:59.711 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:59 smithi120 ceph-mon[43961]: 6.8 scrub starts 2024-05-15T13:08:59.711 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:08:59 smithi120 ceph-mon[43961]: 6.4 scrub starts 2024-05-15T13:08:59.714 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:00.007 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:09:00.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:00.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:09:00.119 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:09:00.200 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:00.540 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 66 16 107173 107157 101.46 61.8125 0.00644027 0.00985146 2024-05-15T13:09:00.552 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:09:00.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:00 smithi027 ceph-mon[39795]: 6.2 scrub starts 2024-05-15T13:09:00.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:00 smithi027 ceph-mon[39795]: 6.8 scrub ok 2024-05-15T13:09:00.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:00 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3812266916' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:09:00.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:00 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:09:00.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:00 smithi027 ceph-mon[39795]: pgmap v1416: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 6.4 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-05-15T13:09:00.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:00 smithi120 ceph-mon[43961]: 6.2 scrub starts 2024-05-15T13:09:00.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:00 smithi120 ceph-mon[43961]: 6.8 scrub ok 2024-05-15T13:09:00.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:00 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3812266916' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:09:00.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:00 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:09:00.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:00 smithi120 ceph-mon[43961]: pgmap v1416: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 6.4 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-05-15T13:09:00.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:00 smithi177 ceph-mon[44303]: 6.2 scrub starts 2024-05-15T13:09:00.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:00 smithi177 ceph-mon[44303]: 6.8 scrub ok 2024-05-15T13:09:00.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:00 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3812266916' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:09:00.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:00 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:09:00.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:00 smithi177 ceph-mon[44303]: pgmap v1416: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 6.4 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-05-15T13:09:01.073 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:09:01.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:01.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:09:01.178 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:09:01.222 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:01.270 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:09:01.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:01.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:09:01.374 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:09:01.402 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:01.434 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:09:01.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:01.439 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:09:01.455 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:09:01.455 INFO:teuthology.orchestra.run.smithi120.stdout: "success": "" 2024-05-15T13:09:01.455 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:09:01.540 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:09:01.541 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 67 16 108013 107997 100.729 52.5 0.015898 0.0099235 2024-05-15T13:09:01.594 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:01.632 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:09:01.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:01.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:09:01.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:01 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:09:01.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:01 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T13:09:01.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:01 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:09:01.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:01 smithi177 ceph-mon[44303]: osdmap e537: 12 total, 12 up, 11 in 2024-05-15T13:09:01.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:01 smithi177 ceph-mon[44303]: 6.9 scrub starts 2024-05-15T13:09:01.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:01 smithi177 ceph-mon[44303]: 6.a scrub starts 2024-05-15T13:09:01.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:01 smithi177 ceph-mon[44303]: 6.2 scrub ok 2024-05-15T13:09:01.739 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:09:01.769 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:01.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:01 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:09:01.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:01 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T13:09:01.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:01 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:09:01.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:01 smithi027 ceph-mon[39795]: osdmap e537: 12 total, 12 up, 11 in 2024-05-15T13:09:01.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:01 smithi027 ceph-mon[39795]: 6.9 scrub starts 2024-05-15T13:09:01.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:01 smithi027 ceph-mon[39795]: 6.a scrub starts 2024-05-15T13:09:01.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:01 smithi027 ceph-mon[39795]: 6.2 scrub ok 2024-05-15T13:09:01.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:01 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:09:01.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:01 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T13:09:01.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:01 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:09:01.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:01 smithi120 ceph-mon[43961]: osdmap e537: 12 total, 12 up, 11 in 2024-05-15T13:09:01.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:01 smithi120 ceph-mon[43961]: 6.9 scrub starts 2024-05-15T13:09:01.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:01 smithi120 ceph-mon[43961]: 6.a scrub starts 2024-05-15T13:09:01.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:01 smithi120 ceph-mon[43961]: 6.2 scrub ok 2024-05-15T13:09:01.796 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:09:01.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:01.801 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:09:01.902 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:09:01.931 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:01.960 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:09:01.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:01.963 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:09:02.049 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:09:02.064 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:09:02.092 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:02.113 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:09:02.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:02.118 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:09:02.219 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:09:02.247 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:02.281 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:09:02.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:02.284 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:09:02.385 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:09:02.418 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:02.540 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 68 16 109026 109010 100.179 63.3125 0.0110521 0.00997758 2024-05-15T13:09:02.672 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:09:02.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:02.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:09:02.780 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:09:02.806 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:02 smithi120 ceph-mon[43961]: 6.4 scrub ok 2024-05-15T13:09:02.806 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:02 smithi120 ceph-mon[43961]: 6.a scrub ok 2024-05-15T13:09:02.806 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:02 smithi120 ceph-mon[43961]: pgmap v1418: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 6.5 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-05-15T13:09:02.808 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:02.827 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:09:02.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:02.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:09:02.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:02 smithi177 ceph-mon[44303]: 6.4 scrub ok 2024-05-15T13:09:02.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:02 smithi177 ceph-mon[44303]: 6.a scrub ok 2024-05-15T13:09:02.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:02 smithi177 ceph-mon[44303]: pgmap v1418: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 6.5 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-05-15T13:09:02.931 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:09:03.002 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:02 smithi027 ceph-mon[39795]: 6.4 scrub ok 2024-05-15T13:09:03.002 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:02 smithi027 ceph-mon[39795]: 6.a scrub ok 2024-05-15T13:09:03.002 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:02 smithi027 ceph-mon[39795]: pgmap v1418: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 6.5 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-05-15T13:09:03.002 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:03.540 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 69 16 110052 110036 99.6565 64.125 0.00655454 0.0100303 2024-05-15T13:09:03.661 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:09:03.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:03 smithi177 ceph-mon[44303]: 6.9 scrub ok 2024-05-15T13:09:04.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:03 smithi027 ceph-mon[39795]: 6.9 scrub ok 2024-05-15T13:09:04.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:03 smithi120 ceph-mon[43961]: 6.9 scrub ok 2024-05-15T13:09:04.275 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:09:04.278 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:09:04.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:04.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:09:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:09:04.386 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:09:04.448 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:04.540 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 70 16 111014 110998 99.0917 60.125 0.0286548 0.0100879 2024-05-15T13:09:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:09:04.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:04.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:09:04.658 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:09:04.690 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:04.716 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:09:04.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:04.721 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:09:04.822 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:09:04.864 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:04 smithi120 ceph-mon[43961]: 6.e scrub starts 2024-05-15T13:09:04.865 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:04 smithi120 ceph-mon[43961]: pgmap v1419: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 6.6 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-05-15T13:09:04.869 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:04.899 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:09:04.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:04.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:09:04.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:04 smithi177 ceph-mon[44303]: 6.e scrub starts 2024-05-15T13:09:04.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:04 smithi177 ceph-mon[44303]: pgmap v1419: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 6.6 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-05-15T13:09:04.939 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:04 smithi027 ceph-mon[39795]: 6.e scrub starts 2024-05-15T13:09:04.939 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:04 smithi027 ceph-mon[39795]: pgmap v1419: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 6.6 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-05-15T13:09:05.004 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:09:05.056 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:05.533 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:09:05.540 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 71 16 112040 112024 98.5992 64.125 0.00523921 0.0101372 2024-05-15T13:09:05.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:05.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:09:05.642 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:09:05.689 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:05.715 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:09:05.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:05.719 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:09:05.819 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:09:05.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:05 smithi027 ceph-mon[39795]: 6.e scrub ok 2024-05-15T13:09:05.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:05 smithi027 ceph-mon[39795]: 6.f scrub starts 2024-05-15T13:09:05.851 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:05 smithi120 ceph-mon[43961]: 6.e scrub ok 2024-05-15T13:09:05.851 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:05 smithi120 ceph-mon[43961]: 6.f scrub starts 2024-05-15T13:09:05.857 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:05.878 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:09:05.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:05.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:09:05.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:05 smithi177 ceph-mon[44303]: 6.e scrub ok 2024-05-15T13:09:05.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:05 smithi177 ceph-mon[44303]: 6.f scrub starts 2024-05-15T13:09:05.985 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:09:06.017 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:06.049 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:09:06.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:06.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:09:06.153 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:09:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:06.540 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 72 16 112990 112974 98.0544 59.375 0.0090953 0.0101948 2024-05-15T13:09:06.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:06 smithi177 ceph-mon[44303]: 6.0 scrub starts 2024-05-15T13:09:06.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:06 smithi177 ceph-mon[44303]: pgmap v1420: 27 pgs: 27 active+clean; 6.7 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-05-15T13:09:06.934 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:09:06.938 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:06 smithi027 ceph-mon[39795]: 6.0 scrub starts 2024-05-15T13:09:06.938 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:06 smithi027 ceph-mon[39795]: pgmap v1420: 27 pgs: 27 active+clean; 6.7 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-05-15T13:09:06.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:06.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:09:07.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:06 smithi120 ceph-mon[43961]: 6.0 scrub starts 2024-05-15T13:09:07.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:06 smithi120 ceph-mon[43961]: pgmap v1420: 27 pgs: 27 active+clean; 6.7 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-05-15T13:09:07.040 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:09:07.077 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:07.105 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:09:07.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:07.108 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:09:07.193 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:09:07.193 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T13:09:07.193 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-05-15T13:09:07.193 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-05-15T13:09:07.193 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:09:07.209 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:09:07.270 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:07.541 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 73 16 113852 113836 97.4491 53.875 0.0127437 0.0102572 2024-05-15T13:09:07.722 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:09:07.722 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:09:07.722 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-05-15T13:09:07.723 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-05-15T13:09:07.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:07 smithi177 ceph-mon[44303]: 6.f scrub ok 2024-05-15T13:09:08.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:07 smithi027 ceph-mon[39795]: 6.f scrub ok 2024-05-15T13:09:08.029 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:07 smithi120 ceph-mon[43961]: 6.f scrub ok 2024-05-15T13:09:08.174 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:09:08.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:08.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:09:08.282 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:09:08.328 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:08.361 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:09:08.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:08.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:09:08.468 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:09:08.534 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:08.540 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 74 16 114727 114711 96.8713 54.6875 0.0105584 0.0103186 2024-05-15T13:09:08.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:08 smithi027 ceph-mon[39795]: pgmap v1421: 27 pgs: 27 active+clean; 6.8 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-05-15T13:09:08.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:08 smithi177 ceph-mon[44303]: pgmap v1421: 27 pgs: 27 active+clean; 6.8 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-05-15T13:09:09.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:08 smithi120 ceph-mon[43961]: pgmap v1421: 27 pgs: 27 active+clean; 6.8 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-05-15T13:09:09.540 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 75 16 115758 115742 96.4387 64.4375 0.00672713 0.0103657 2024-05-15T13:09:09.759 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:09:09.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:09.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:09:09.871 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:09:09.906 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:09 smithi027 ceph-mon[39795]: 6.0 scrub ok 2024-05-15T13:09:09.928 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:09 smithi120 ceph-mon[43961]: 6.0 scrub ok 2024-05-15T13:09:09.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:09 smithi177 ceph-mon[44303]: 6.0 scrub ok 2024-05-15T13:09:09.931 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:09.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:09.988 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:09:09.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:09:10.090 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:09:10.116 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:10.135 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:09:10.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:10.138 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:09:10.239 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:09:10.295 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:10.336 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:09:10.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:10.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:09:10.441 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:09:10.477 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:10.531 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:09:10.540 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 76 16 116661 116645 95.9124 56.4375 0.0118805 0.010422 2024-05-15T13:09:10.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:10.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:09:10.644 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:09:10.682 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:10.709 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:09:10.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:10.713 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:09:10.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:10 smithi027 ceph-mon[39795]: 6.5 scrub starts 2024-05-15T13:09:10.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:10 smithi027 ceph-mon[39795]: 6.7 scrub starts 2024-05-15T13:09:10.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:10 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:09:10.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:10 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:09:10.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:10 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:09:10.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:10 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:09:10.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:10 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:09:10.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:10 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:09:10.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:10 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:09:10.814 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:09:10.848 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:10.879 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:09:10.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:10.882 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:09:10.882 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:10 smithi177 ceph-mon[44303]: 6.5 scrub starts 2024-05-15T13:09:10.882 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:10 smithi177 ceph-mon[44303]: 6.7 scrub starts 2024-05-15T13:09:10.883 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:10 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:09:10.883 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:10 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:09:10.883 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:10 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:09:10.883 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:10 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:09:10.883 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:10 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:09:10.883 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:10 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:09:10.883 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:10 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:09:10.983 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:09:11.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:10 smithi120 ceph-mon[43961]: 6.5 scrub starts 2024-05-15T13:09:11.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:10 smithi120 ceph-mon[43961]: 6.7 scrub starts 2024-05-15T13:09:11.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:10 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:09:11.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:10 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:09:11.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:10 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:09:11.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:10 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:09:11.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:10 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:09:11.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:10 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:09:11.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:10 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:09:11.055 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:11.090 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-05-15T13:09:11.154 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:09:11.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:11.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:09:11.265 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:09:11.305 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:11.431 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:09:11.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:11.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:09:11.536 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:09:11.540 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 77 16 117548 117532 95.3867 55.4375 0.00465702 0.0104676 2024-05-15T13:09:11.573 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:11.617 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:09:11.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:11.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:09:11.723 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:09:11.768 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:11.813 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:09:11.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:11.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:09:11.916 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:09:11.948 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:11.970 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:09:11.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:11.976 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:09:12.077 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:09:12.139 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:11 smithi027 ceph-mon[39795]: 6.5 scrub ok 2024-05-15T13:09:12.139 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:11 smithi027 ceph-mon[39795]: 6.3 scrub starts 2024-05-15T13:09:12.151 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:12.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:11 smithi177 ceph-mon[44303]: 6.5 scrub ok 2024-05-15T13:09:12.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:11 smithi177 ceph-mon[44303]: 6.3 scrub starts 2024-05-15T13:09:12.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:11 smithi120 ceph-mon[43961]: 6.5 scrub ok 2024-05-15T13:09:12.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:11 smithi120 ceph-mon[43961]: 6.3 scrub starts 2024-05-15T13:09:12.541 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 78 16 118466 118450 94.8993 57.375 0.0071575 0.0105328 2024-05-15T13:09:12.852 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:12 smithi027 ceph-mon[39795]: 6.7 scrub ok 2024-05-15T13:09:12.852 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:12 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2472663549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:09:12.853 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:12 smithi027 ceph-mon[39795]: pgmap v1423: 27 pgs: 27 active+clean; 7.1 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 58 MiB/s wr, 928 op/s 2024-05-15T13:09:12.853 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:12 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/266847856' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:09:12.853 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:12 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:09:12.853 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:12 smithi027 ceph-mon[39795]: 6.3 scrub ok 2024-05-15T13:09:12.885 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:09:12.888 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:09:12.888 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:09:12.888 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:09:12.889 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:09:12.889 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:09:12.889 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:09:12.889 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:09:12.889 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:09:12.889 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:09:12.889 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:09:12.889 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:09:12.889 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:09:12.889 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:09:12.889 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:09:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:09:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:09:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:09:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:09:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:09:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:09:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:09:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:09:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:09:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:09:13.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:12 smithi177 ceph-mon[44303]: 6.7 scrub ok 2024-05-15T13:09:13.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:12 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2472663549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:09:13.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:12 smithi177 ceph-mon[44303]: pgmap v1423: 27 pgs: 27 active+clean; 7.1 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 58 MiB/s wr, 928 op/s 2024-05-15T13:09:13.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:12 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/266847856' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:09:13.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:12 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:09:13.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:12 smithi177 ceph-mon[44303]: 6.3 scrub ok 2024-05-15T13:09:13.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:12 smithi120 ceph-mon[43961]: 6.7 scrub ok 2024-05-15T13:09:13.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:12 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2472663549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:09:13.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:12 smithi120 ceph-mon[43961]: pgmap v1423: 27 pgs: 27 active+clean; 7.1 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 58 MiB/s wr, 928 op/s 2024-05-15T13:09:13.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:12 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/266847856' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:09:13.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:12 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:09:13.282 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:12 smithi120 ceph-mon[43961]: 6.3 scrub ok 2024-05-15T13:09:13.541 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 79 16 119461 119445 94.4852 62.1875 0.00425543 0.0105643 2024-05-15T13:09:14.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:13 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:09:14.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:13 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:09:14.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:13 smithi177 ceph-mon[44303]: osdmap e538: 12 total, 12 up, 11 in 2024-05-15T13:09:14.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:13 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:09:14.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:13 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:09:14.200 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:13 smithi027 ceph-mon[39795]: osdmap e538: 12 total, 12 up, 11 in 2024-05-15T13:09:14.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:13 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:09:14.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:13 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:09:14.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:13 smithi120 ceph-mon[43961]: osdmap e538: 12 total, 12 up, 11 in 2024-05-15T13:09:14.317 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:09:14.323 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:09:14.367 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:09:14.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:14.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:09:14.471 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:09:14.541 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T13:09:14.552993+0000 min lat: 0.00114056 max lat: 0.299471 avg lat: 0.0106262 2024-05-15T13:09:14.541 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:09:14.541 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 80 16 120439 120423 94.0682 61.125 0.0118557 0.0106262 2024-05-15T13:09:14.558 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:14.856 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:14 smithi027 ceph-mon[39795]: pgmap v1425: 27 pgs: 27 active+clean; 7.2 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 60 MiB/s wr, 956 op/s 2024-05-15T13:09:15.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:14 smithi177 ceph-mon[44303]: pgmap v1425: 27 pgs: 27 active+clean; 7.2 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 60 MiB/s wr, 956 op/s 2024-05-15T13:09:15.280 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:14 smithi120 ceph-mon[43961]: pgmap v1425: 27 pgs: 27 active+clean; 7.2 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 60 MiB/s wr, 956 op/s 2024-05-15T13:09:15.541 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 81 16 121438 121422 93.6776 62.4375 0.0038419 0.0106583 2024-05-15T13:09:15.831 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:09:15.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:15.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:09:15.943 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:09:15.977 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:16.003 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:09:16.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:16.010 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:09:16.111 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:09:16.180 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:16.242 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:09:16.367 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:09:16.409 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:09:16.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:16.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:09:16.516 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:09:16.541 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 82 16 122442 122426 93.3004 62.75 0.0342742 0.0107132 2024-05-15T13:09:16.545 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:16.568 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:09:16.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:16.573 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:09:16.674 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:09:16.714 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:16.751 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:09:16.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:16.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:09:16.855 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:09:16.885 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:16.919 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:09:16.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:16.924 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:09:17.025 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:09:17.073 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:17.178 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:17 smithi027 ceph-mon[39795]: pgmap v1426: 27 pgs: 27 active+clean; 7.3 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 59 MiB/s wr, 948 op/s 2024-05-15T13:09:17.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:17 smithi177 ceph-mon[44303]: pgmap v1426: 27 pgs: 27 active+clean; 7.3 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 59 MiB/s wr, 948 op/s 2024-05-15T13:09:17.229 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:09:17.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:17.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:09:17.336 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:09:17.375 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:17.397 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:09:17.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:17.400 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:09:17.501 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:09:17.528 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:17 smithi120 ceph-mon[43961]: pgmap v1426: 27 pgs: 27 active+clean; 7.3 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 59 MiB/s wr, 948 op/s 2024-05-15T13:09:17.530 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:17.541 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 83 16 123407 123391 92.9029 60.3125 0.00865509 0.0107601 2024-05-15T13:09:17.555 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:09:17.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:17.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:09:17.665 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:09:17.710 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:17.731 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:09:17.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:17.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:09:17.842 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:09:17.872 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:17.902 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:09:17.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:17.909 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:09:18.010 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:09:18.047 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:18.067 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:09:18.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:18.070 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:09:18.170 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:09:18.210 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:18.240 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:09:18.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:18.245 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:09:18.347 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:09:18.379 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:18.414 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:09:18.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:18.419 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:09:18.520 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:09:18.541 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 84 16 124442 124426 92.567 64.6875 0.00411741 0.0107919 2024-05-15T13:09:18.546 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:18.569 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:09:18.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:18.574 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:09:18.675 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:09:18.721 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:19.487 INFO:teuthology.orchestra.run.smithi027.stderr:no valid command found; 10 closest matches: 2024-05-15T13:09:19.487 INFO:teuthology.orchestra.run.smithi027.stderr:0 2024-05-15T13:09:19.487 INFO:teuthology.orchestra.run.smithi027.stderr:1 2024-05-15T13:09:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:2 2024-05-15T13:09:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:abort 2024-05-15T13:09:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:assert 2024-05-15T13:09:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:bench [] [] [] [] 2024-05-15T13:09:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs debug_inject_read_zeros 2024-05-15T13:09:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs files list 2024-05-15T13:09:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs stats 2024-05-15T13:09:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:bluestore allocator dump block 2024-05-15T13:09:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: invalid command 2024-05-15T13:09:19.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:19 smithi027 ceph-mon[39795]: pgmap v1427: 27 pgs: 27 active+clean; 7.4 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 59 MiB/s wr, 948 op/s 2024-05-15T13:09:19.529 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:19 smithi120 ceph-mon[43961]: pgmap v1427: 27 pgs: 27 active+clean; 7.4 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 59 MiB/s wr, 948 op/s 2024-05-15T13:09:19.534 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:09:19.534 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:09:19.541 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 85 16 125469 125453 92.2331 64.1875 0.0229664 0.010838 2024-05-15T13:09:19.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:19 smithi177 ceph-mon[44303]: pgmap v1427: 27 pgs: 27 active+clean; 7.4 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 59 MiB/s wr, 948 op/s 2024-05-15T13:09:19.729 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:09:20.542 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 86 16 126473 126457 91.8902 62.75 0.00558039 0.0108784 2024-05-15T13:09:20.962 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T13:09:20.962 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T13:09:20.962 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-05-15T13:09:21.210 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:09:21.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:21.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:09:21.218 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:09:21.318 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:09:21.354 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:21.375 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:09:21.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:21.379 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:09:21.480 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:09:21.510 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:21 smithi177 ceph-mon[44303]: pgmap v1428: 27 pgs: 27 active+clean; 7.6 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 60 MiB/s wr, 960 op/s 2024-05-15T13:09:21.512 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:21.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:21 smithi027 ceph-mon[39795]: pgmap v1428: 27 pgs: 27 active+clean; 7.6 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 60 MiB/s wr, 960 op/s 2024-05-15T13:09:21.529 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:21 smithi120 ceph-mon[43961]: pgmap v1428: 27 pgs: 27 active+clean; 7.6 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 60 MiB/s wr, 960 op/s 2024-05-15T13:09:21.534 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:09:21.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:21.537 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:09:21.541 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 87 16 127442 127426 91.5301 60.5625 0.0219531 0.0109207 2024-05-15T13:09:21.638 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:09:21.677 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:21.712 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:09:21.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:21.717 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:09:21.817 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:09:21.877 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:21.902 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:09:21.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:21.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:09:22.006 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:09:22.040 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:22.073 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:09:22.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:22.075 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:09:22.176 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:09:22.211 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:22.239 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:09:22.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:22.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:09:22.346 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:09:22.399 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:22.542 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 88 16 128453 128437 91.2079 63.1875 0.017308 0.0109605 2024-05-15T13:09:22.671 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:09:22.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:22.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:09:22.710 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-05-15T13:09:22.779 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:09:22.813 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:22.837 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:09:22.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:22.841 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:09:22.942 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:09:23.005 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:23.145 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:09:23.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:23.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:09:23.255 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:09:23.271 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:23 smithi027 ceph-mon[39795]: pgmap v1429: 27 pgs: 27 active+clean; 7.7 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 61 MiB/s wr, 976 op/s 2024-05-15T13:09:23.299 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:23.319 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:09:23.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:23.322 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:09:23.423 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:09:23.462 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:23.487 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:09:23.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:23 smithi120 ceph-mon[43961]: pgmap v1429: 27 pgs: 27 active+clean; 7.7 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 61 MiB/s wr, 976 op/s 2024-05-15T13:09:23.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:23.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:09:23.542 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 89 16 129411 129395 90.8558 59.875 0.0180683 0.0110023 2024-05-15T13:09:23.591 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:09:23.642 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:23.669 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:09:23.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:23.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:09:23.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:23 smithi177 ceph-mon[44303]: pgmap v1429: 27 pgs: 27 active+clean; 7.7 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 61 MiB/s wr, 976 op/s 2024-05-15T13:09:23.774 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:09:23.811 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:23.835 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:09:23.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:23.838 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:09:23.864 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:09:23.939 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:09:24.006 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:24.542 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Total time run: 90.0102 2024-05-15T13:09:24.542 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Total writes made: 130368 2024-05-15T13:09:24.542 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Write size: 65536 2024-05-15T13:09:24.542 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Object size: 65536 2024-05-15T13:09:24.543 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Bandwidth (MB/sec): 90.5231 2024-05-15T13:09:24.543 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Stddev Bandwidth: 37.539 2024-05-15T13:09:24.543 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Max bandwidth (MB/sec): 216.375 2024-05-15T13:09:24.543 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Min bandwidth (MB/sec): 52.5 2024-05-15T13:09:24.543 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Average IOPS: 1448 2024-05-15T13:09:24.543 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Stddev IOPS: 600.623 2024-05-15T13:09:24.543 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Max IOPS: 3462 2024-05-15T13:09:24.543 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Min IOPS: 840 2024-05-15T13:09:24.543 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Average Latency(s): 0.0110434 2024-05-15T13:09:24.543 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Stddev Latency(s): 0.0133014 2024-05-15T13:09:24.543 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Max latency(s): 0.299471 2024-05-15T13:09:24.543 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Min latency(s): 0.00114056 2024-05-15T13:09:24.549 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Cleaning up (deleting benchmark objects) 2024-05-15T13:09:24.809 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:09:24.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:24.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:09:24.920 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:09:24.952 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:24.983 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:09:24.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:24.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:09:25.087 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:09:25.161 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:25.238 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:09:25.519 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:25 smithi027 ceph-mon[39795]: pgmap v1430: 27 pgs: 27 active+clean; 7.8 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-05-15T13:09:25.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:25 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2255106628' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:09:25.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:25 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:09:25.529 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:25 smithi120 ceph-mon[43961]: pgmap v1430: 27 pgs: 27 active+clean; 7.8 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-05-15T13:09:25.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:25 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2255106628' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:09:25.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:25 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:09:25.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:25 smithi177 ceph-mon[44303]: pgmap v1430: 27 pgs: 27 active+clean; 7.8 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-05-15T13:09:25.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:25 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2255106628' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:09:25.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:25 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:09:25.842 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:09:25.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:25.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:09:25.949 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:09:26.015 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:26.294 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:09:26.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:26.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:09:26.400 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:09:26.437 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:26.459 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:09:26.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:26.463 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:09:26.494 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:26 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:09:26.494 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:26 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:09:26.494 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:26 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:09:26.494 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:26 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:09:26.495 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:26 smithi027 ceph-mon[39795]: osdmap e539: 12 total, 12 up, 11 in 2024-05-15T13:09:26.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:26 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:09:26.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:26 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:09:26.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:26 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:09:26.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:26 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:09:26.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:26 smithi120 ceph-mon[43961]: osdmap e539: 12 total, 12 up, 11 in 2024-05-15T13:09:26.547 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:09:26.564 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:09:26.601 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:26.642 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:09:26.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:26.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:09:26.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:26 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:09:26.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:26 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:09:26.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:26 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:09:26.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:26 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:09:26.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:26 smithi177 ceph-mon[44303]: osdmap e539: 12 total, 12 up, 11 in 2024-05-15T13:09:26.746 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:09:26.827 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:09:26.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:26.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:09:26.939 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:09:26.973 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:09:27.002 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:09:27.021 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:09:27.040 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:27.071 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:09:27.073 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:09:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:27.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:09:27.118 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:09:27.169 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:09:27.176 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:09:27.207 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:27.213 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:09:27.229 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:09:27.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:27.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:09:27.247 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:09:27.289 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:09:27.331 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:09:27.336 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:09:27.367 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:27.368 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:09:27.390 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:09:27.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:27.393 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:09:27.406 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:09:27.418 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:09:27.418 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:09:27.418 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:09:27.418 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:09:27.418 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:09:27.418 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:09:27.418 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:09:27.419 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:09:27.419 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:09:27.419 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:09:27.419 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:09:27.419 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:09:27.493 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:09:27.519 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:27 smithi027 ceph-mon[39795]: pgmap v1432: 27 pgs: 27 active+clean; 7.9 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 61 MiB/s wr, 986 op/s 2024-05-15T13:09:27.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:27 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1629053708' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:09:27.529 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:27 smithi120 ceph-mon[43961]: pgmap v1432: 27 pgs: 27 active+clean; 7.9 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 61 MiB/s wr, 986 op/s 2024-05-15T13:09:27.529 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:27 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1629053708' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:09:27.545 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:27.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:27 smithi177 ceph-mon[44303]: pgmap v1432: 27 pgs: 27 active+clean; 7.9 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 61 MiB/s wr, 986 op/s 2024-05-15T13:09:27.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:27 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1629053708' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:09:28.587 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:09:29.254 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:29 smithi027 ceph-mon[39795]: pgmap v1433: 27 pgs: 27 active+clean; 7.9 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 55 MiB/s wr, 993 op/s 2024-05-15T13:09:29.254 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:29 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:09:29.254 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:29 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:09:29.529 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:29 smithi120 ceph-mon[43961]: pgmap v1433: 27 pgs: 27 active+clean; 7.9 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 55 MiB/s wr, 993 op/s 2024-05-15T13:09:29.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:29 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:09:29.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:29 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:09:29.566 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:09:29.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:29.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:09:29.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:29 smithi177 ceph-mon[44303]: pgmap v1433: 27 pgs: 27 active+clean; 7.9 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 55 MiB/s wr, 993 op/s 2024-05-15T13:09:29.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:29 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:09:29.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:29 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:09:29.679 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:09:29.714 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:29.747 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:09:29.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:29.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:09:29.851 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:09:29.893 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:29.922 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:09:29.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:29.926 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:09:30.027 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:09:30.087 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:30.234 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:09:30.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:30.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:09:30.342 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:09:30.355 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:30 smithi027 ceph-mon[39795]: osdmap e540: 12 total, 12 up, 11 in 2024-05-15T13:09:30.371 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:30.392 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:09:30.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:30.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:09:30.497 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:09:30.523 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:30 smithi177 ceph-mon[44303]: osdmap e540: 12 total, 12 up, 11 in 2024-05-15T13:09:30.525 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:30.529 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:30 smithi120 ceph-mon[43961]: osdmap e540: 12 total, 12 up, 11 in 2024-05-15T13:09:30.548 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:09:30.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:30.553 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:09:30.654 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:09:30.698 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:30.723 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:09:30.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:30.726 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:09:30.827 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:09:30.902 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:31.232 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:09:31.232 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:09:31.232 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.232 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25735.0:136682 6.0 e540/509 6:0d2a4010:::benchmark_data_smithi177_122278_object6311:head v 540'8496, mlcod=540'8492)", 2024-05-15T13:09:31.232 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:09:31.198138+0000", 2024-05-15T13:09:31.232 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.0046051240000000004, 2024-05-15T13:09:31.232 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046350219999999999, 2024-05-15T13:09:31.232 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:09:31.232 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:09:31.233 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:09:31.233 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.233 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:09:31.233 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:31.198138+0000", 2024-05-15T13:09:31.233 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:09:31.233 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:09:31.233 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.233 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:09:31.233 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:31.198138+0000", 2024-05-15T13:09:31.233 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:09:31.233 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:09:31.233 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.233 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:09:31.234 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:31.198140+0000", 2024-05-15T13:09:31.234 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.933e-06 2024-05-15T13:09:31.234 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:09:31.234 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.234 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:09:31.234 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:31.198153+0000", 2024-05-15T13:09:31.234 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2675e-05 2024-05-15T13:09:31.234 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:09:31.234 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.234 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:09:31.234 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:31.198178+0000", 2024-05-15T13:09:31.234 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5321e-05 2024-05-15T13:09:31.234 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:09:31.235 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.235 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:09:31.235 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:31.198189+0000", 2024-05-15T13:09:31.235 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0509000000000001e-05 2024-05-15T13:09:31.235 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:09:31.235 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.235 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:09:31.235 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:31.198254+0000", 2024-05-15T13:09:31.235 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5761999999999998e-05 2024-05-15T13:09:31.235 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:09:31.235 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.235 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:09:31.235 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:31.198278+0000", 2024-05-15T13:09:31.235 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3904e-05 2024-05-15T13:09:31.236 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:09:31.236 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:09:31.236 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:09:31.236 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:09:31.236 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.236 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25735.0:136686 6.0 e540/509)", 2024-05-15T13:09:31.236 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:09:31.198315+0000", 2024-05-15T13:09:31.236 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.0044285150000000001, 2024-05-15T13:09:31.236 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0045235739999999998, 2024-05-15T13:09:31.236 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:09:31.236 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "queued for pg", 2024-05-15T13:09:31.236 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:09:31.236 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.236 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:09:31.237 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:31.198315+0000", 2024-05-15T13:09:31.237 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:09:31.237 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:09:31.237 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.237 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:09:31.237 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:31.198315+0000", 2024-05-15T13:09:31.237 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:09:31.238 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:09:31.238 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.238 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:09:31.238 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:31.198315+0000", 2024-05-15T13:09:31.238 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3900000000000005e-07 2024-05-15T13:09:31.238 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:09:31.238 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.238 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:09:31.238 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:31.198321+0000", 2024-05-15T13:09:31.238 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8109999999999999e-06 2024-05-15T13:09:31.238 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:09:31.239 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.239 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:09:31.239 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:31.198330+0000", 2024-05-15T13:09:31.239 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7490000000000004e-06 2024-05-15T13:09:31.239 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:09:31.239 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.239 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:09:31.239 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:31.198337+0000", 2024-05-15T13:09:31.239 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7569999999999998e-06 2024-05-15T13:09:31.239 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:09:31.239 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:09:31.239 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:09:31.239 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:09:31.240 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.240 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25735.0:136688 6.2 e540/509 6:4575e734:::benchmark_data_smithi177_122278_object6317:head v 540'8451, mlcod=540'8449)", 2024-05-15T13:09:31.240 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:09:31.202664+0000", 2024-05-15T13:09:31.240 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 7.9136999999999998e-05, 2024-05-15T13:09:31.240 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00022766999999999999, 2024-05-15T13:09:31.240 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:09:31.240 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:09:31.240 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:09:31.240 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.240 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:09:31.240 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:31.202664+0000", 2024-05-15T13:09:31.240 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:09:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:09:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:09:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:31.202664+0000", 2024-05-15T13:09:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:09:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:09:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:09:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:31.202665+0000", 2024-05-15T13:09:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3069999999999999e-06 2024-05-15T13:09:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:09:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.242 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:09:31.242 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:31.202676+0000", 2024-05-15T13:09:31.242 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0643e-05 2024-05-15T13:09:31.242 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:09:31.242 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.242 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:09:31.242 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:31.202691+0000", 2024-05-15T13:09:31.242 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4908e-05 2024-05-15T13:09:31.242 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:09:31.242 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.242 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:09:31.242 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:31.202700+0000", 2024-05-15T13:09:31.242 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2380000000000003e-06 2024-05-15T13:09:31.242 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:09:31.243 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.243 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:09:31.243 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:31.202788+0000", 2024-05-15T13:09:31.243 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7869999999999997e-05 2024-05-15T13:09:31.243 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:09:31.243 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:09:31.243 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:09:31.243 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:31.202810+0000", 2024-05-15T13:09:31.243 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2169999999999999e-05 2024-05-15T13:09:31.243 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:09:31.243 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:09:31.244 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:09:31.244 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:09:31.244 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:09:31.244 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 3 2024-05-15T13:09:31.244 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:09:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:09:31.502 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:31 smithi027 ceph-mon[39795]: pgmap v1435: 27 pgs: 27 active+clean; 7.7 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 127 B/s rd, 37 MiB/s wr, 1.03k op/s 2024-05-15T13:09:31.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:31.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:09:31.604 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:09:31.630 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:31 smithi177 ceph-mon[44303]: pgmap v1435: 27 pgs: 27 active+clean; 7.7 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 127 B/s rd, 37 MiB/s wr, 1.03k op/s 2024-05-15T13:09:31.633 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:31.662 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:09:31.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:31.670 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:09:31.770 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:09:31.779 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:31 smithi120 ceph-mon[43961]: pgmap v1435: 27 pgs: 27 active+clean; 7.7 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 127 B/s rd, 37 MiB/s wr, 1.03k op/s 2024-05-15T13:09:31.803 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:31.822 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:09:31.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:31.826 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:09:31.927 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:09:31.967 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:31.989 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:09:31.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:31.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:09:32.096 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:09:32.154 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:32.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:09:32.175 INFO:teuthology.orchestra.run.smithi027.stderr:no valid command found; 10 closest matches: 2024-05-15T13:09:32.175 INFO:teuthology.orchestra.run.smithi027.stderr:0 2024-05-15T13:09:32.175 INFO:teuthology.orchestra.run.smithi027.stderr:1 2024-05-15T13:09:32.175 INFO:teuthology.orchestra.run.smithi027.stderr:2 2024-05-15T13:09:32.175 INFO:teuthology.orchestra.run.smithi027.stderr:abort 2024-05-15T13:09:32.175 INFO:teuthology.orchestra.run.smithi027.stderr:assert 2024-05-15T13:09:32.175 INFO:teuthology.orchestra.run.smithi027.stderr:bench [] [] [] [] 2024-05-15T13:09:32.175 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs debug_inject_read_zeros 2024-05-15T13:09:32.175 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs files list 2024-05-15T13:09:32.175 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs stats 2024-05-15T13:09:32.175 INFO:teuthology.orchestra.run.smithi027.stderr:bluestore allocator dump block 2024-05-15T13:09:32.176 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: invalid command 2024-05-15T13:09:32.482 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:32 smithi027 ceph-mon[39795]: pgmap v1436: 27 pgs: 27 active+clean; 7.7 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 127 B/s rd, 20 MiB/s wr, 949 op/s 2024-05-15T13:09:32.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:32 smithi177 ceph-mon[44303]: pgmap v1436: 27 pgs: 27 active+clean; 7.7 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 127 B/s rd, 20 MiB/s wr, 949 op/s 2024-05-15T13:09:32.729 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:09:32.746 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:09:32.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:32.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:09:32.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:32 smithi120 ceph-mon[43961]: pgmap v1436: 27 pgs: 27 active+clean; 7.7 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 127 B/s rd, 20 MiB/s wr, 949 op/s 2024-05-15T13:09:32.850 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:09:32.894 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:32.922 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:09:32.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:32.925 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:09:33.026 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:09:33.068 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:33.098 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:09:33.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:33.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:09:33.207 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:09:33.250 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:33.276 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:09:33.276 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T13:09:33.277 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T13:09:33.277 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-05-15T13:09:33.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:33.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:09:33.380 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:09:33.423 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:33.451 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:09:33.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:33.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:09:33.476 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-05-15T13:09:33.555 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:09:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:33.722 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:09:33.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:33.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:09:33.831 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:09:33.927 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:34.241 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:09:34.243 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:09:34.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:34.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:09:34.308 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:09:34.357 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:09:34.391 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:34.423 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:09:34.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:34.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:09:34.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:34 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:09:34.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:34 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:09:34.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:34 smithi027 ceph-mon[39795]: osdmap e541: 12 total, 12 up, 11 in 2024-05-15T13:09:34.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:34 smithi027 ceph-mon[39795]: pgmap v1438: 27 pgs: 27 active+clean; 7.5 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 127 B/s rd, 9.5 MiB/s wr, 1.01k op/s 2024-05-15T13:09:34.530 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:09:34.574 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:34.597 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:09:34.602 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:34 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:09:34.602 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:34 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:09:34.602 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:34 smithi120 ceph-mon[43961]: osdmap e541: 12 total, 12 up, 11 in 2024-05-15T13:09:34.602 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:34 smithi120 ceph-mon[43961]: pgmap v1438: 27 pgs: 27 active+clean; 7.5 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 127 B/s rd, 9.5 MiB/s wr, 1.01k op/s 2024-05-15T13:09:34.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:34.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:09:34.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:34 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:09:34.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:34 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:09:34.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:34 smithi177 ceph-mon[44303]: osdmap e541: 12 total, 12 up, 11 in 2024-05-15T13:09:34.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:34 smithi177 ceph-mon[44303]: pgmap v1438: 27 pgs: 27 active+clean; 7.5 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 127 B/s rd, 9.5 MiB/s wr, 1.01k op/s 2024-05-15T13:09:34.704 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:09:34.747 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:34.773 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:09:34.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:34.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:09:34.877 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:09:34.915 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:34.947 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:09:34.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:34.950 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:09:35.051 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:09:35.091 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:35.114 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:09:35.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:35.117 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:09:35.219 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:09:35.296 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:36.586 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:09:36.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:36.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:09:36.691 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:09:36.752 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:36 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:09:36.752 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:36 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:09:36.752 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:36 smithi027 ceph-mon[39795]: osdmap e542: 12 total, 12 up, 11 in 2024-05-15T13:09:36.752 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:36 smithi027 ceph-mon[39795]: pgmap v1440: 27 pgs: 27 active+clean; 7.4 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 895 op/s 2024-05-15T13:09:36.757 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:36.779 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:36 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:09:36.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:36 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:09:36.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:36 smithi120 ceph-mon[43961]: osdmap e542: 12 total, 12 up, 11 in 2024-05-15T13:09:36.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:36 smithi120 ceph-mon[43961]: pgmap v1440: 27 pgs: 27 active+clean; 7.4 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 895 op/s 2024-05-15T13:09:36.894 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:09:36.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:36.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:09:36.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:36 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:09:36.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:36 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:09:36.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:36 smithi177 ceph-mon[44303]: osdmap e542: 12 total, 12 up, 11 in 2024-05-15T13:09:36.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:36 smithi177 ceph-mon[44303]: pgmap v1440: 27 pgs: 27 active+clean; 7.4 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 895 op/s 2024-05-15T13:09:36.998 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:09:37.037 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:37.062 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:09:37.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:37.065 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:09:37.166 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:09:37.204 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:37.237 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:09:37.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:37.240 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:09:37.341 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:09:37.381 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:38.045 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:09:38.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:38.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:09:38.153 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:09:38.195 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:38.233 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:09:38.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:38.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:09:38.337 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:09:38.368 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:38.390 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:09:38.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:38.396 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:09:38.497 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:09:38.575 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:38.658 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:09:38.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:38.664 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:09:38.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:09:38.766 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:09:38.801 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:38.829 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:09:38.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:38.833 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:09:38.934 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:09:38.968 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:38.993 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:09:38.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:38.998 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:09:39.099 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:09:39.162 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:39.288 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:09:39.292 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:39 smithi027 ceph-mon[39795]: pgmap v1441: 27 pgs: 27 active+clean; 7.3 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 833 op/s 2024-05-15T13:09:39.293 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:39 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:09:39.293 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:39 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:09:39.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:39.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:09:39.395 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:09:39.436 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:39.487 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:09:39.495 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:39 smithi120 ceph-mon[43961]: pgmap v1441: 27 pgs: 27 active+clean; 7.3 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 833 op/s 2024-05-15T13:09:39.495 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:39 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:09:39.495 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:39 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:09:39.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:39.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:09:39.597 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:09:39.630 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:39 smithi177 ceph-mon[44303]: pgmap v1441: 27 pgs: 27 active+clean; 7.3 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 833 op/s 2024-05-15T13:09:39.631 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:39 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:09:39.631 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:39 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:09:39.631 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:39.656 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:09:39.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:39.663 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:09:39.763 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:09:39.800 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:39.834 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:09:39.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:39.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:09:39.939 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:09:39.968 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:39.992 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:09:39.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:39.996 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:09:40.097 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:09:40.195 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:40.475 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:40 smithi027 ceph-mon[39795]: osdmap e543: 12 total, 12 up, 11 in 2024-05-15T13:09:40.475 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:40 smithi027 ceph-mon[39795]: pgmap v1443: 27 pgs: 27 active+clean; 7.2 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 930 op/s 2024-05-15T13:09:40.475 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:40 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3607728730' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:09:40.475 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:40 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:09:40.516 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:09:40.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:40 smithi120 ceph-mon[43961]: osdmap e543: 12 total, 12 up, 11 in 2024-05-15T13:09:40.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:40 smithi120 ceph-mon[43961]: pgmap v1443: 27 pgs: 27 active+clean; 7.2 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 930 op/s 2024-05-15T13:09:40.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:40 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3607728730' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:09:40.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:40 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:09:40.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:40 smithi177 ceph-mon[44303]: osdmap e543: 12 total, 12 up, 11 in 2024-05-15T13:09:40.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:40 smithi177 ceph-mon[44303]: pgmap v1443: 27 pgs: 27 active+clean; 7.2 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 930 op/s 2024-05-15T13:09:40.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:40 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3607728730' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:09:40.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:40 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:09:41.307 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:09:41.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:41.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:09:41.415 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:09:41.437 INFO:teuthology.orchestra.run.smithi027.stderr:no valid command found; 10 closest matches: 2024-05-15T13:09:41.437 INFO:teuthology.orchestra.run.smithi027.stderr:0 2024-05-15T13:09:41.437 INFO:teuthology.orchestra.run.smithi027.stderr:1 2024-05-15T13:09:41.437 INFO:teuthology.orchestra.run.smithi027.stderr:2 2024-05-15T13:09:41.437 INFO:teuthology.orchestra.run.smithi027.stderr:abort 2024-05-15T13:09:41.437 INFO:teuthology.orchestra.run.smithi027.stderr:assert 2024-05-15T13:09:41.437 INFO:teuthology.orchestra.run.smithi027.stderr:bench [] [] [] [] 2024-05-15T13:09:41.437 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs debug_inject_read_zeros 2024-05-15T13:09:41.437 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs files list 2024-05-15T13:09:41.437 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs stats 2024-05-15T13:09:41.437 INFO:teuthology.orchestra.run.smithi027.stderr:bluestore allocator dump block 2024-05-15T13:09:41.437 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: invalid command 2024-05-15T13:09:41.486 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:41.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:41 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:09:41.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:41 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:09:41.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:41 smithi027 ceph-mon[39795]: osdmap e544: 12 total, 12 up, 11 in 2024-05-15T13:09:41.779 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:41 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:09:41.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:41 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:09:41.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:41 smithi120 ceph-mon[43961]: osdmap e544: 12 total, 12 up, 11 in 2024-05-15T13:09:41.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:41 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:09:41.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:41 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:09:41.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:41 smithi177 ceph-mon[44303]: osdmap e544: 12 total, 12 up, 11 in 2024-05-15T13:09:42.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:42 smithi027 ceph-mon[39795]: pgmap v1445: 27 pgs: 27 active+clean; 7.1 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 858 op/s 2024-05-15T13:09:42.779 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:42 smithi120 ceph-mon[43961]: pgmap v1445: 27 pgs: 27 active+clean; 7.1 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 858 op/s 2024-05-15T13:09:42.909 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:09:42.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:42.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:09:42.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:42 smithi177 ceph-mon[44303]: pgmap v1445: 27 pgs: 27 active+clean; 7.1 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 858 op/s 2024-05-15T13:09:43.028 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:09:43.065 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:43.098 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:09:43.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:43.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:09:43.207 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:09:43.256 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:43.525 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:09:43.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:43.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:09:43.575 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T13:09:43.576 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T13:09:43.576 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-05-15T13:09:43.643 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:09:43.739 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:43.774 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:09:43.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:43.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:09:43.887 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:09:43.951 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:44.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:44 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:09:44.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:44 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:09:44.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:44 smithi120 ceph-mon[43961]: osdmap e545: 12 total, 12 up, 11 in 2024-05-15T13:09:44.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:44 smithi120 ceph-mon[43961]: pgmap v1447: 27 pgs: 27 active+clean; 7.0 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-05-15T13:09:44.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:44 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:09:44.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:44 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:09:44.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:44 smithi177 ceph-mon[44303]: osdmap e545: 12 total, 12 up, 11 in 2024-05-15T13:09:44.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:44 smithi177 ceph-mon[44303]: pgmap v1447: 27 pgs: 27 active+clean; 7.0 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-05-15T13:09:45.004 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:44 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:09:45.004 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:44 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:09:45.004 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:44 smithi027 ceph-mon[39795]: osdmap e545: 12 total, 12 up, 11 in 2024-05-15T13:09:45.004 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:44 smithi027 ceph-mon[39795]: pgmap v1447: 27 pgs: 27 active+clean; 7.0 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-05-15T13:09:45.525 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:09:45.525 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:09:45.525 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:09:45.525 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:09:45.525 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:09:45.525 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:09:45.525 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:09:45.525 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:09:45.525 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:09:45.525 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:09:45.525 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:09:45.526 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:09:45.526 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:09:45.526 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:09:45.526 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:09:45.526 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:09:45.526 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:09:45.526 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:09:45.526 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:09:45.526 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:09:45.526 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:09:45.526 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:09:45.527 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:09:45.527 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:09:45.887 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:45 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2127184266' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:09:45.920 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:09:45.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:45.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:09:45.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:45 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2127184266' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:09:46.026 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:09:46.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:45 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2127184266' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:09:46.064 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:46.090 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:09:46.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:46.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:09:46.195 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:09:46.249 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:46.922 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:09:46.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:46 smithi177 ceph-mon[44303]: pgmap v1448: 27 pgs: 27 active+clean; 6.9 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 907 op/s 2024-05-15T13:09:46.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:46.931 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:46 smithi027 ceph-mon[39795]: pgmap v1448: 27 pgs: 27 active+clean; 6.9 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 907 op/s 2024-05-15T13:09:46.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:09:47.029 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:46 smithi120 ceph-mon[43961]: pgmap v1448: 27 pgs: 27 active+clean; 6.9 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 907 op/s 2024-05-15T13:09:47.033 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:09:47.083 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:47.109 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:09:47.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:47.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:09:47.213 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:09:47.240 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:47.262 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:09:47.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:47.268 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:09:47.294 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:09:47.369 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:09:47.426 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:09:47.426 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:09:47.461 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:47.495 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:09:47.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:47.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:09:47.604 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:09:47.631 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:47.662 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:09:47.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:47.664 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:09:47.765 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:09:47.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:47 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:09:47.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:47 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:09:47.801 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:47.823 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:09:47.826 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:47 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:09:47.826 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:47 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:09:47.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:47.827 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:09:47.928 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:09:47.956 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:47 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:09:47.956 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:47 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:09:47.958 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:47.986 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:09:47.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:47.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:09:48.090 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:09:48.142 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:48.161 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:09:48.172 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:09:48.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:48.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:09:48.279 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:09:48.353 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:48.709 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:09:48.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:48 smithi177 ceph-mon[44303]: osdmap e546: 12 total, 12 up, 11 in 2024-05-15T13:09:48.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:48 smithi177 ceph-mon[44303]: pgmap v1450: 27 pgs: 27 active+clean; 6.8 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 829 op/s 2024-05-15T13:09:49.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:48 smithi027 ceph-mon[39795]: osdmap e546: 12 total, 12 up, 11 in 2024-05-15T13:09:49.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:48 smithi027 ceph-mon[39795]: pgmap v1450: 27 pgs: 27 active+clean; 6.8 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 829 op/s 2024-05-15T13:09:49.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:48 smithi120 ceph-mon[43961]: osdmap e546: 12 total, 12 up, 11 in 2024-05-15T13:09:49.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:48 smithi120 ceph-mon[43961]: pgmap v1450: 27 pgs: 27 active+clean; 6.8 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 829 op/s 2024-05-15T13:09:49.135 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:09:49.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:49.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:09:49.242 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:09:49.272 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:49.296 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:09:49.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:49.301 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:09:49.402 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:09:49.435 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:49.480 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:09:49.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:49.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:09:49.585 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:09:49.617 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:49.646 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:09:49.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:49.649 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:09:49.750 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:09:49.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:49 smithi120 ceph-mon[43961]: 1.2 scrub starts 2024-05-15T13:09:49.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:49 smithi120 ceph-mon[43961]: 1.2 scrub ok 2024-05-15T13:09:49.782 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:49.808 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:09:49.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:49.811 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:09:49.912 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:09:49.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:49 smithi177 ceph-mon[44303]: 1.2 scrub starts 2024-05-15T13:09:49.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:49 smithi177 ceph-mon[44303]: 1.2 scrub ok 2024-05-15T13:09:49.947 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:49.968 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:09:49.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:49.971 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:09:50.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:49 smithi027 ceph-mon[39795]: 1.2 scrub starts 2024-05-15T13:09:50.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:49 smithi027 ceph-mon[39795]: 1.2 scrub ok 2024-05-15T13:09:50.071 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:09:50.106 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:50.140 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:09:50.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:50.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:09:50.248 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:09:50.275 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:50.296 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:09:50.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:50.298 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:09:50.313 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-05-15T13:09:50.399 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:09:50.476 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:50.513 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:09:50.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:50.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:09:50.619 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:09:50.667 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:50.709 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:09:50.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:50.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:09:50.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:50 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:09:50.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:50 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:09:50.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:50 smithi027 ceph-mon[39795]: pgmap v1451: 27 pgs: 27 active+clean; 6.6 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 1.00k op/s 2024-05-15T13:09:50.817 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:09:50.908 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:09:50.912 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:09:50.917 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:50.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:50 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:09:50.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:50 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:09:50.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:50 smithi177 ceph-mon[44303]: pgmap v1451: 27 pgs: 27 active+clean; 6.6 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 1.00k op/s 2024-05-15T13:09:50.990 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:09:51.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:50 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:09:51.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:50 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:09:51.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:50 smithi120 ceph-mon[43961]: pgmap v1451: 27 pgs: 27 active+clean; 6.6 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 1.00k op/s 2024-05-15T13:09:51.050 INFO:teuthology.orchestra.run.smithi027.stderr:no valid command found; 10 closest matches: 2024-05-15T13:09:51.051 INFO:teuthology.orchestra.run.smithi027.stderr:0 2024-05-15T13:09:51.051 INFO:teuthology.orchestra.run.smithi027.stderr:1 2024-05-15T13:09:51.051 INFO:teuthology.orchestra.run.smithi027.stderr:2 2024-05-15T13:09:51.051 INFO:teuthology.orchestra.run.smithi027.stderr:abort 2024-05-15T13:09:51.051 INFO:teuthology.orchestra.run.smithi027.stderr:assert 2024-05-15T13:09:51.051 INFO:teuthology.orchestra.run.smithi027.stderr:bench [] [] [] [] 2024-05-15T13:09:51.051 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs debug_inject_read_zeros 2024-05-15T13:09:51.051 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs files list 2024-05-15T13:09:51.051 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs stats 2024-05-15T13:09:51.051 INFO:teuthology.orchestra.run.smithi027.stderr:bluestore allocator dump block 2024-05-15T13:09:51.051 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: invalid command 2024-05-15T13:09:51.319 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:09:51.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:51.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:09:51.425 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:09:51.482 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:51.507 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:09:51.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:51.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:09:51.612 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:09:51.629 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:51 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:09:51.629 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:51 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:09:51.629 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:51 smithi027 ceph-mon[39795]: osdmap e547: 12 total, 12 up, 11 in 2024-05-15T13:09:51.670 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:51.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:51 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:09:51.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:51 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:09:51.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:51 smithi120 ceph-mon[43961]: osdmap e547: 12 total, 12 up, 11 in 2024-05-15T13:09:51.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:51 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:09:51.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:51 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:09:51.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:51 smithi177 ceph-mon[44303]: osdmap e547: 12 total, 12 up, 11 in 2024-05-15T13:09:52.478 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:09:52.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:52.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:09:52.588 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:09:52.657 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:52.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:52 smithi027 ceph-mon[39795]: pgmap v1453: 27 pgs: 27 active+clean; 6.5 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 875 op/s 2024-05-15T13:09:52.833 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:09:52.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:52.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:09:52.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:52 smithi177 ceph-mon[44303]: pgmap v1453: 27 pgs: 27 active+clean; 6.5 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 875 op/s 2024-05-15T13:09:52.939 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:09:52.996 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:53.030 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:52 smithi120 ceph-mon[43961]: pgmap v1453: 27 pgs: 27 active+clean; 6.5 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 875 op/s 2024-05-15T13:09:53.080 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:09:53.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:53.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:09:53.186 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:09:53.240 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:53.614 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:09:53.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:53.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:09:53.723 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:09:53.776 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:53.967 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:09:53.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:53.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:09:54.076 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:09:54.109 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:54.133 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:09:54.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:54.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:09:54.236 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:09:54.276 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:54.317 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:09:54.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:54.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:09:54.423 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:09:54.484 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:54.540 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:09:54.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:54.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:09:54.648 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:09:54.677 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:54.699 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:09:54.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:54.705 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:09:54.806 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:09:54.862 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:54.946 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:09:54.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:54.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:09:55.054 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:09:55.089 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:55.116 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:09:55.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:55.123 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:09:55.211 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:55 smithi027 ceph-mon[39795]: pgmap v1454: 27 pgs: 27 active+clean; 6.4 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 941 op/s 2024-05-15T13:09:55.223 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:09:55.317 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:55.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:55 smithi177 ceph-mon[44303]: pgmap v1454: 27 pgs: 27 active+clean; 6.4 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 941 op/s 2024-05-15T13:09:55.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:55 smithi120 ceph-mon[43961]: pgmap v1454: 27 pgs: 27 active+clean; 6.4 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 941 op/s 2024-05-15T13:09:56.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:56 smithi027 ceph-mon[39795]: 1.8 scrub starts 2024-05-15T13:09:56.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:56 smithi027 ceph-mon[39795]: 1.4 scrub starts 2024-05-15T13:09:56.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:56 smithi027 ceph-mon[39795]: 1.8 scrub ok 2024-05-15T13:09:56.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:56 smithi027 ceph-mon[39795]: 1.4 scrub ok 2024-05-15T13:09:56.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:56 smithi027 ceph-mon[39795]: 1.1 scrub starts 2024-05-15T13:09:56.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:56 smithi027 ceph-mon[39795]: 1.1 scrub ok 2024-05-15T13:09:56.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:56 smithi120 ceph-mon[43961]: 1.8 scrub starts 2024-05-15T13:09:56.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:56 smithi120 ceph-mon[43961]: 1.4 scrub starts 2024-05-15T13:09:56.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:56 smithi120 ceph-mon[43961]: 1.8 scrub ok 2024-05-15T13:09:56.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:56 smithi120 ceph-mon[43961]: 1.4 scrub ok 2024-05-15T13:09:56.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:56 smithi120 ceph-mon[43961]: 1.1 scrub starts 2024-05-15T13:09:56.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:56 smithi120 ceph-mon[43961]: 1.1 scrub ok 2024-05-15T13:09:56.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:56 smithi177 ceph-mon[44303]: 1.8 scrub starts 2024-05-15T13:09:56.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:56 smithi177 ceph-mon[44303]: 1.4 scrub starts 2024-05-15T13:09:56.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:56 smithi177 ceph-mon[44303]: 1.8 scrub ok 2024-05-15T13:09:56.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:56 smithi177 ceph-mon[44303]: 1.4 scrub ok 2024-05-15T13:09:56.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:56 smithi177 ceph-mon[44303]: 1.1 scrub starts 2024-05-15T13:09:56.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:56 smithi177 ceph-mon[44303]: 1.1 scrub ok 2024-05-15T13:09:56.922 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:09:56.965 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:09:57.014 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:09:57.070 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:09:57.105 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:09:57.138 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:09:57.190 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:09:57.231 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:09:57.255 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:09:57.268 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:09:57.311 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:09:57.324 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:09:57.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:57.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:09:57.371 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:09:57.418 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:09:57.448 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:09:57.449 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:09:57.449 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:09:57.449 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:09:57.449 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:09:57.449 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:09:57.449 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:09:57.449 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:09:57.449 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:09:57.449 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:09:57.449 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:09:57.449 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:09:57.450 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:09:57.499 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:57 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:09:57.500 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:57 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:09:57.500 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:57 smithi120 ceph-mon[43961]: pgmap v1455: 27 pgs: 27 active+clean; 6.3 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 896 op/s 2024-05-15T13:09:57.500 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:57 smithi120 ceph-mon[43961]: osdmap e548: 12 total, 12 up, 11 in 2024-05-15T13:09:57.500 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:57 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1128540286' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:09:57.500 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:57 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3564816138' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:09:57.500 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:57 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:09:57.503 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:57.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:57 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:09:57.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:57 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:09:57.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:57 smithi027 ceph-mon[39795]: pgmap v1455: 27 pgs: 27 active+clean; 6.3 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 896 op/s 2024-05-15T13:09:57.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:57 smithi027 ceph-mon[39795]: osdmap e548: 12 total, 12 up, 11 in 2024-05-15T13:09:57.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:57 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1128540286' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:09:57.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:57 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3564816138' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:09:57.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:57 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:09:57.527 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:09:57.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:57.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:09:57.633 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:09:57.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:57 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:09:57.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:57 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:09:57.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:57 smithi177 ceph-mon[44303]: pgmap v1455: 27 pgs: 27 active+clean; 6.3 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 896 op/s 2024-05-15T13:09:57.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:57 smithi177 ceph-mon[44303]: osdmap e548: 12 total, 12 up, 11 in 2024-05-15T13:09:57.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:57 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1128540286' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:09:57.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:57 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3564816138' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:09:57.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:57 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:09:57.680 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:58.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:58 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:09:58.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:58 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T13:09:58.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:58 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:09:58.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:58 smithi027 ceph-mon[39795]: osdmap e549: 12 total, 12 up, 11 in 2024-05-15T13:09:58.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:58 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:09:58.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:58 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T13:09:58.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:58 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:09:58.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:58 smithi120 ceph-mon[43961]: osdmap e549: 12 total, 12 up, 11 in 2024-05-15T13:09:58.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:58 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:09:58.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:58 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T13:09:58.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:58 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:09:58.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:58 smithi177 ceph-mon[44303]: osdmap e549: 12 total, 12 up, 11 in 2024-05-15T13:09:59.268 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:09:59.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:59.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:09:59.379 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:09:59.413 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:59.458 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:09:59.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:59.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:09:59.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:09:59 smithi027 ceph-mon[39795]: pgmap v1458: 27 pgs: 27 active+clean; 6.2 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 792 op/s 2024-05-15T13:09:59.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:09:59 smithi120 ceph-mon[43961]: pgmap v1458: 27 pgs: 27 active+clean; 6.2 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 792 op/s 2024-05-15T13:09:59.562 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:09:59.639 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:59.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:09:59 smithi177 ceph-mon[44303]: pgmap v1458: 27 pgs: 27 active+clean; 6.2 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 792 op/s 2024-05-15T13:09:59.807 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:09:59.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:59.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:09:59.917 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:09:59.946 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:09:59.968 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:09:59.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:09:59.971 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:10:00.072 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:10:00.102 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:00.112 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:10:00.112 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T13:10:00.113 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T13:10:00.113 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:10:00.113 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.113 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:17231 5.6 5:6074def8:::benchmark_data_smithi177_109991_object17230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:10:00.113 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:26.276983+0000", 2024-05-15T13:10:00.113 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 393.84397882100001, 2024-05-15T13:10:00.113 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.094001816000000002, 2024-05-15T13:10:00.113 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:00.113 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:00.113 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:00.113 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:10:00.113 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:10:00.113 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 17231 2024-05-15T13:10:00.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.114 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:00.114 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.114 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:00.114 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.276983+0000", 2024-05-15T13:10:00.114 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.114 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.114 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:00.114 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.276983+0000", 2024-05-15T13:10:00.114 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.114 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.115 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:00.115 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.276985+0000", 2024-05-15T13:10:00.115 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4020000000000001e-06 2024-05-15T13:10:00.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.115 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:00.115 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.277043+0000", 2024-05-15T13:10:00.115 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8356000000000002e-05 2024-05-15T13:10:00.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.115 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:00.115 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.277055+0000", 2024-05-15T13:10:00.115 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2051e-05 2024-05-15T13:10:00.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.116 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.116 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:00.116 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.277061+0000", 2024-05-15T13:10:00.116 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7019999999999998e-06 2024-05-15T13:10:00.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.116 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.116 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:00.116 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.277204+0000", 2024-05-15T13:10:00.116 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014371499999999999 2024-05-15T13:10:00.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.116 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.116 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:00.116 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.277328+0000", 2024-05-15T13:10:00.117 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012367199999999999 2024-05-15T13:10:00.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.117 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,10", 2024-05-15T13:10:00.117 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.277467+0000", 2024-05-15T13:10:00.117 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000138632 2024-05-15T13:10:00.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.117 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:00.117 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.279118+0000", 2024-05-15T13:10:00.117 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001651065 2024-05-15T13:10:00.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.118 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:00.118 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.279942+0000", 2024-05-15T13:10:00.118 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00082398300000000003 2024-05-15T13:10:00.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.118 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.118 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:00.118 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280627+0000", 2024-05-15T13:10:00.118 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00068536200000000002 2024-05-15T13:10:00.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.118 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.118 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:00.118 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.370934+0000", 2024-05-15T13:10:00.119 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.090306484000000006 2024-05-15T13:10:00.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.119 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.119 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:00.119 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.370985+0000", 2024-05-15T13:10:00.119 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1391999999999999e-05 2024-05-15T13:10:00.119 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.119 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:00.119 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.119 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.119 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:17234 5.6 5:66a9cf93:::benchmark_data_smithi177_109991_object17233:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:10:00.119 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:26.278295+0000", 2024-05-15T13:10:00.120 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 393.84266648099998, 2024-05-15T13:10:00.120 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.092853842000000006, 2024-05-15T13:10:00.120 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:00.120 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:00.120 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:00.120 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:10:00.120 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:10:00.120 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 17234 2024-05-15T13:10:00.120 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.120 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:00.120 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.120 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:00.120 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.278295+0000", 2024-05-15T13:10:00.120 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.121 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.121 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:00.121 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.278295+0000", 2024-05-15T13:10:00.121 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.121 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.121 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:00.121 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.278296+0000", 2024-05-15T13:10:00.121 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.569999999999999e-07 2024-05-15T13:10:00.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.121 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.121 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:00.121 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.278329+0000", 2024-05-15T13:10:00.122 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2928000000000001e-05 2024-05-15T13:10:00.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.122 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.122 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:00.122 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.278341+0000", 2024-05-15T13:10:00.122 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1450999999999999e-05 2024-05-15T13:10:00.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.122 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.122 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:00.122 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.278348+0000", 2024-05-15T13:10:00.122 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0709999999999997e-06 2024-05-15T13:10:00.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.122 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.122 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:00.123 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.278398+0000", 2024-05-15T13:10:00.123 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.99e-05 2024-05-15T13:10:00.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.123 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.123 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:00.123 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.278494+0000", 2024-05-15T13:10:00.123 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5989000000000005e-05 2024-05-15T13:10:00.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.123 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.123 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,10", 2024-05-15T13:10:00.123 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.278590+0000", 2024-05-15T13:10:00.123 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6032000000000005e-05 2024-05-15T13:10:00.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.123 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.123 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:00.124 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.279256+0000", 2024-05-15T13:10:00.124 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00066654099999999999 2024-05-15T13:10:00.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.124 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.124 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:00.124 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280063+0000", 2024-05-15T13:10:00.124 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00080693799999999999 2024-05-15T13:10:00.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.124 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.124 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:00.124 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.371051+0000", 2024-05-15T13:10:00.124 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.090988187999999998 2024-05-15T13:10:00.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.124 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.125 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:00.125 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.371102+0000", 2024-05-15T13:10:00.125 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0543e-05 2024-05-15T13:10:00.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.125 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:00.125 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.371149+0000", 2024-05-15T13:10:00.125 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7304e-05 2024-05-15T13:10:00.125 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.125 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:00.125 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.125 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:17242 5.a 5:5f5524b5:::benchmark_data_smithi177_109991_object17241:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:10:00.126 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:26.280331+0000", 2024-05-15T13:10:00.126 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 393.84063111900002, 2024-05-15T13:10:00.126 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.093852726999999997, 2024-05-15T13:10:00.126 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:00.126 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:00.126 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:00.126 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:10:00.126 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:10:00.126 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 17242 2024-05-15T13:10:00.126 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.126 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:00.126 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.127 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:00.127 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280331+0000", 2024-05-15T13:10:00.127 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.127 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.127 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:00.127 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280331+0000", 2024-05-15T13:10:00.127 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.127 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.127 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:00.127 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280332+0000", 2024-05-15T13:10:00.127 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4e-07 2024-05-15T13:10:00.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.128 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.128 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:00.128 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280406+0000", 2024-05-15T13:10:00.128 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4497000000000002e-05 2024-05-15T13:10:00.128 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.128 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.128 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:00.128 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280416+0000", 2024-05-15T13:10:00.128 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3570000000000004e-06 2024-05-15T13:10:00.128 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.128 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.128 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:00.128 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280421+0000", 2024-05-15T13:10:00.128 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.66e-06 2024-05-15T13:10:00.128 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.129 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.129 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:00.129 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280471+0000", 2024-05-15T13:10:00.129 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0083000000000001e-05 2024-05-15T13:10:00.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.129 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.129 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:00.129 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280585+0000", 2024-05-15T13:10:00.129 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000114175 2024-05-15T13:10:00.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.129 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.129 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,10", 2024-05-15T13:10:00.129 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.371615+0000", 2024-05-15T13:10:00.129 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.091029502999999998 2024-05-15T13:10:00.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.130 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.130 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:00.130 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.373727+0000", 2024-05-15T13:10:00.130 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0021119440000000001 2024-05-15T13:10:00.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.130 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.130 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:00.130 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.373738+0000", 2024-05-15T13:10:00.130 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1596999999999999e-05 2024-05-15T13:10:00.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.130 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.130 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:00.130 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.374061+0000", 2024-05-15T13:10:00.131 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000322584 2024-05-15T13:10:00.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.131 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.131 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:00.131 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.374134+0000", 2024-05-15T13:10:00.131 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2997999999999995e-05 2024-05-15T13:10:00.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.131 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.131 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:00.131 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.374184+0000", 2024-05-15T13:10:00.131 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9489e-05 2024-05-15T13:10:00.131 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.131 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:00.131 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.132 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.132 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.15540.0:17231 5.6 e424/418 ondisk, result = 0)", 2024-05-15T13:10:00.132 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:26.280558+0000", 2024-05-15T13:10:00.132 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 393.840403864, 2024-05-15T13:10:00.132 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.090434608, 2024-05-15T13:10:00.132 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:00.132 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:10:00.132 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:00.132 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.132 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:00.132 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280558+0000", 2024-05-15T13:10:00.132 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.132 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.132 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.133 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:00.133 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280558+0000", 2024-05-15T13:10:00.133 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.133 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.133 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.133 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:00.133 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280559+0000", 2024-05-15T13:10:00.133 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4899999999999995e-07 2024-05-15T13:10:00.133 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.133 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.133 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:00.133 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280563+0000", 2024-05-15T13:10:00.133 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4469999999999999e-06 2024-05-15T13:10:00.133 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.134 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.134 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:00.134 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280573+0000", 2024-05-15T13:10:00.134 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8279999999999998e-06 2024-05-15T13:10:00.134 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.134 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.134 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:00.134 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280579+0000", 2024-05-15T13:10:00.134 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.392e-06 2024-05-15T13:10:00.134 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.134 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.134 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:00.134 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280619+0000", 2024-05-15T13:10:00.134 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9517000000000001e-05 2024-05-15T13:10:00.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.135 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:00.135 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280624+0000", 2024-05-15T13:10:00.135 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0309999999999998e-06 2024-05-15T13:10:00.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.135 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:00.135 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.370993+0000", 2024-05-15T13:10:00.135 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.090368744000000001 2024-05-15T13:10:00.135 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.135 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:00.135 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.136 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.15540.0:17234 5.6 e424/418 ondisk, result = 0)", 2024-05-15T13:10:00.136 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:26.280775+0000", 2024-05-15T13:10:00.136 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 393.84018720199998, 2024-05-15T13:10:00.136 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.090397524000000007, 2024-05-15T13:10:00.136 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:00.136 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:10:00.136 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:00.136 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.136 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:00.136 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280775+0000", 2024-05-15T13:10:00.136 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.136 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.136 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.136 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:00.137 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280775+0000", 2024-05-15T13:10:00.137 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.137 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.137 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.137 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:00.137 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280775+0000", 2024-05-15T13:10:00.137 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3200000000000005e-07 2024-05-15T13:10:00.137 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.137 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.137 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:00.137 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.280779+0000", 2024-05-15T13:10:00.137 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5769999999999998e-06 2024-05-15T13:10:00.137 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.137 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.137 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:00.138 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.370780+0000", 2024-05-15T13:10:00.138 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.090000822999999994 2024-05-15T13:10:00.138 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.138 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.138 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:00.138 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.370793+0000", 2024-05-15T13:10:00.138 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.376e-05 2024-05-15T13:10:00.138 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.138 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.138 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:00.139 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.371037+0000", 2024-05-15T13:10:00.139 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000243382 2024-05-15T13:10:00.139 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.139 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.139 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:00.139 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.371045+0000", 2024-05-15T13:10:00.139 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6519999999999995e-06 2024-05-15T13:10:00.139 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.139 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.139 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:00.139 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:26.371172+0000", 2024-05-15T13:10:00.139 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012679799999999999 2024-05-15T13:10:00.139 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.139 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:00.139 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.140 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.140 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.140 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:19090 5.6 5:6e7c4261:::benchmark_data_smithi177_109991_object19089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:10:00.140 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:27.084229+0000", 2024-05-15T13:10:00.140 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 393.03673256299999, 2024-05-15T13:10:00.140 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.090169716999999996, 2024-05-15T13:10:00.140 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:00.140 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:00.140 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:00.140 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:10:00.140 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:10:00.140 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 19090 2024-05-15T13:10:00.140 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.141 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:00.141 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.141 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:00.141 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.084229+0000", 2024-05-15T13:10:00.141 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.141 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.141 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.141 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:00.141 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.084229+0000", 2024-05-15T13:10:00.141 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.141 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.141 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.141 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:00.141 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.084231+0000", 2024-05-15T13:10:00.142 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.471e-06 2024-05-15T13:10:00.142 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.142 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.142 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:00.142 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.084296+0000", 2024-05-15T13:10:00.142 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5009e-05 2024-05-15T13:10:00.142 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.142 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.142 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:00.142 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.084308+0000", 2024-05-15T13:10:00.142 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1809e-05 2024-05-15T13:10:00.142 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.142 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.142 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:00.143 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.084315+0000", 2024-05-15T13:10:00.143 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3300000000000001e-06 2024-05-15T13:10:00.143 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.143 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.143 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:00.143 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.084375+0000", 2024-05-15T13:10:00.143 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0340000000000002e-05 2024-05-15T13:10:00.143 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.143 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.143 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:00.143 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.084512+0000", 2024-05-15T13:10:00.143 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013636900000000001 2024-05-15T13:10:00.143 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.143 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.143 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,10", 2024-05-15T13:10:00.144 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.084657+0000", 2024-05-15T13:10:00.144 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014542 2024-05-15T13:10:00.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.144 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.144 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:00.144 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.086161+0000", 2024-05-15T13:10:00.144 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001504357 2024-05-15T13:10:00.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.144 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.144 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:00.144 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.087460+0000", 2024-05-15T13:10:00.144 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001298319 2024-05-15T13:10:00.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.144 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.145 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:00.145 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.174296+0000", 2024-05-15T13:10:00.145 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.086836216999999993 2024-05-15T13:10:00.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.145 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.145 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:00.145 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.174365+0000", 2024-05-15T13:10:00.145 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8515000000000004e-05 2024-05-15T13:10:00.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.145 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.145 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:00.145 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.174399+0000", 2024-05-15T13:10:00.145 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4561000000000003e-05 2024-05-15T13:10:00.145 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.145 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:00.146 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.146 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.146 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.146 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:20880 5.a 5:56abf9c6:::benchmark_data_smithi177_109991_object20879:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:10:00.146 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:27.874888+0000", 2024-05-15T13:10:00.146 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 392.24607401999998, 2024-05-15T13:10:00.146 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.080734568000000007, 2024-05-15T13:10:00.146 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:00.146 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:00.146 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:00.146 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:10:00.146 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:10:00.146 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 20880 2024-05-15T13:10:00.147 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.147 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:00.147 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.147 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:00.147 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.874888+0000", 2024-05-15T13:10:00.147 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.147 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.147 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.147 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:00.147 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.874888+0000", 2024-05-15T13:10:00.147 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.147 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.147 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.147 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:00.148 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.874889+0000", 2024-05-15T13:10:00.148 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2419999999999999e-06 2024-05-15T13:10:00.148 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.148 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.148 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:00.148 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.874956+0000", 2024-05-15T13:10:00.148 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7033999999999995e-05 2024-05-15T13:10:00.148 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.148 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.148 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:00.148 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.874967+0000", 2024-05-15T13:10:00.148 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0878e-05 2024-05-15T13:10:00.148 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.149 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.149 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:00.149 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.874993+0000", 2024-05-15T13:10:00.149 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5463999999999999e-05 2024-05-15T13:10:00.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.149 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.149 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:00.149 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.875056+0000", 2024-05-15T13:10:00.149 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3145999999999995e-05 2024-05-15T13:10:00.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.149 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.149 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:00.149 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.875186+0000", 2024-05-15T13:10:00.149 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000129927 2024-05-15T13:10:00.150 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.150 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.150 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,10", 2024-05-15T13:10:00.150 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.875328+0000", 2024-05-15T13:10:00.150 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014283899999999999 2024-05-15T13:10:00.150 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.150 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.150 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:00.150 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.876264+0000", 2024-05-15T13:10:00.150 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00093588600000000003 2024-05-15T13:10:00.150 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.150 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.150 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:00.150 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.878169+0000", 2024-05-15T13:10:00.151 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019042359999999999 2024-05-15T13:10:00.151 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.151 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.151 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:00.151 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.955559+0000", 2024-05-15T13:10:00.151 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.077390027 2024-05-15T13:10:00.151 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.151 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.151 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:00.151 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.955593+0000", 2024-05-15T13:10:00.151 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4446999999999997e-05 2024-05-15T13:10:00.151 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.152 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.152 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:00.152 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:27.955622+0000", 2024-05-15T13:10:00.152 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9442e-05 2024-05-15T13:10:00.152 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.152 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:00.152 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.152 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.152 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.152 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.15540.0:31155 5.6 5:6e92a747:::benchmark_data_smithi177_109991_object31154:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-05-15T13:10:00.152 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:03:31.996055+0000", 2024-05-15T13:10:00.152 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 388.12490672799998, 2024-05-15T13:10:00.152 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.096537703000000002, 2024-05-15T13:10:00.152 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:00.152 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:00.153 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:00.153 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.15540", 2024-05-15T13:10:00.153 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:10:00.153 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 31155 2024-05-15T13:10:00.153 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.153 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:00.153 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.153 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:00.153 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:31.996055+0000", 2024-05-15T13:10:00.153 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.153 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.153 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.153 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:00.153 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:31.996055+0000", 2024-05-15T13:10:00.154 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.154 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.154 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.154 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:00.154 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:31.996057+0000", 2024-05-15T13:10:00.154 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8419999999999999e-06 2024-05-15T13:10:00.154 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.154 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.154 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:00.154 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:31.996407+0000", 2024-05-15T13:10:00.154 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000349907 2024-05-15T13:10:00.154 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.154 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.155 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:00.155 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:31.996427+0000", 2024-05-15T13:10:00.155 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0168000000000001e-05 2024-05-15T13:10:00.155 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.155 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.155 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:00.155 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:31.996437+0000", 2024-05-15T13:10:00.155 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6800000000000005e-06 2024-05-15T13:10:00.155 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.155 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.155 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:00.155 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:31.996505+0000", 2024-05-15T13:10:00.155 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8137000000000002e-05 2024-05-15T13:10:00.155 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.156 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.156 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:00.156 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:31.996637+0000", 2024-05-15T13:10:00.156 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013156299999999999 2024-05-15T13:10:00.156 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.156 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.156 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,10", 2024-05-15T13:10:00.156 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:31.996776+0000", 2024-05-15T13:10:00.156 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013938700000000001 2024-05-15T13:10:00.156 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.156 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.156 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:00.156 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:31.998297+0000", 2024-05-15T13:10:00.156 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001521281 2024-05-15T13:10:00.157 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.157 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.157 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:00.157 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:31.998483+0000", 2024-05-15T13:10:00.157 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00018571399999999999 2024-05-15T13:10:00.157 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.157 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.157 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:00.157 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:32.092457+0000", 2024-05-15T13:10:00.157 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.093974577000000004 2024-05-15T13:10:00.157 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.157 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.157 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:00.157 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:32.092548+0000", 2024-05-15T13:10:00.157 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0687999999999996e-05 2024-05-15T13:10:00.158 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.158 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.158 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:00.158 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:03:32.092593+0000", 2024-05-15T13:10:00.158 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4759000000000003e-05 2024-05-15T13:10:00.158 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.158 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:00.158 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.158 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.158 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.158 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25735.0:63767 6.e e524/509 6:72497558:::benchmark_data_smithi177_122278_object63766:head v 524'3990, mlcod=524'3988)", 2024-05-15T13:10:00.158 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:08:25.329016+0000", 2024-05-15T13:10:00.158 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 94.791945455999993, 2024-05-15T13:10:00.158 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.15607338000000001, 2024-05-15T13:10:00.159 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:00.159 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:00.159 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:00.159 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.159 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:00.159 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.329016+0000", 2024-05-15T13:10:00.159 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.159 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.159 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:00.159 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.329016+0000", 2024-05-15T13:10:00.159 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.159 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.160 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:00.160 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.329017+0000", 2024-05-15T13:10:00.160 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9100000000000003e-07 2024-05-15T13:10:00.160 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.160 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.160 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:00.160 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.329068+0000", 2024-05-15T13:10:00.160 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1140000000000002e-05 2024-05-15T13:10:00.160 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.160 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.160 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:00.160 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.329077+0000", 2024-05-15T13:10:00.160 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2130000000000008e-06 2024-05-15T13:10:00.160 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.160 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.161 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:00.161 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.329082+0000", 2024-05-15T13:10:00.161 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3199999999999999e-06 2024-05-15T13:10:00.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.161 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.161 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:00.161 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.329161+0000", 2024-05-15T13:10:00.161 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8702000000000006e-05 2024-05-15T13:10:00.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.161 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.161 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:00.161 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.329183+0000", 2024-05-15T13:10:00.161 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2019999999999999e-05 2024-05-15T13:10:00.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.162 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:00.162 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.333290+0000", 2024-05-15T13:10:00.162 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041073439999999998 2024-05-15T13:10:00.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.162 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:00.162 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.485090+0000", 2024-05-15T13:10:00.162 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.15179994999999999 2024-05-15T13:10:00.162 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.162 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:00.162 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.163 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.163 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25735.0:63768 6.5 e524/509 6:a6cb4d37:::benchmark_data_smithi177_122278_object63767:head v 524'4054, mlcod=524'4052)", 2024-05-15T13:10:00.163 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:08:25.331163+0000", 2024-05-15T13:10:00.163 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 94.789798754000003, 2024-05-15T13:10:00.163 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.15393645, 2024-05-15T13:10:00.163 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:00.163 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:00.163 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:00.163 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.163 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:00.163 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.331163+0000", 2024-05-15T13:10:00.163 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.163 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.163 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.164 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:00.164 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.331163+0000", 2024-05-15T13:10:00.164 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.164 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.164 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:00.164 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.331164+0000", 2024-05-15T13:10:00.164 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2900000000000002e-07 2024-05-15T13:10:00.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.164 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.164 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:00.164 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.331206+0000", 2024-05-15T13:10:00.164 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2379999999999997e-05 2024-05-15T13:10:00.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.165 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.165 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:00.165 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.331216+0000", 2024-05-15T13:10:00.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9690000000000006e-06 2024-05-15T13:10:00.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.165 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.165 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:00.165 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.331224+0000", 2024-05-15T13:10:00.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3180000000000001e-06 2024-05-15T13:10:00.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.165 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.165 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:00.165 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.331386+0000", 2024-05-15T13:10:00.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016237599999999999 2024-05-15T13:10:00.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.166 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.166 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:00.166 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.331406+0000", 2024-05-15T13:10:00.166 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0299999999999999e-05 2024-05-15T13:10:00.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.166 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.166 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:00.166 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.485049+0000", 2024-05-15T13:10:00.166 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.153642999 2024-05-15T13:10:00.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.166 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.166 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:00.167 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.485100+0000", 2024-05-15T13:10:00.167 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0278999999999998e-05 2024-05-15T13:10:00.167 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.167 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:00.167 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.167 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.167 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.167 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25735.0:63769 6.0 e524/509 6:09ab85c3:::benchmark_data_smithi177_122278_object63768:head v 524'3975, mlcod=524'3973)", 2024-05-15T13:10:00.167 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:08:25.333103+0000", 2024-05-15T13:10:00.167 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 94.787859312999998, 2024-05-15T13:10:00.167 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.15779794, 2024-05-15T13:10:00.167 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:00.167 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:00.167 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:00.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.168 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:00.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.333103+0000", 2024-05-15T13:10:00.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.168 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:00.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.333103+0000", 2024-05-15T13:10:00.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.168 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:00.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.333104+0000", 2024-05-15T13:10:00.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0470000000000001e-06 2024-05-15T13:10:00.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:00.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.333130+0000", 2024-05-15T13:10:00.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6529999999999998e-05 2024-05-15T13:10:00.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:00.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.333138+0000", 2024-05-15T13:10:00.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0029999999999994e-06 2024-05-15T13:10:00.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:00.170 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.333143+0000", 2024-05-15T13:10:00.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0529999999999997e-06 2024-05-15T13:10:00.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:00.170 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.333189+0000", 2024-05-15T13:10:00.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5717999999999998e-05 2024-05-15T13:10:00.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:00.170 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.333210+0000", 2024-05-15T13:10:00.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0554e-05 2024-05-15T13:10:00.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.171 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:00.171 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.490856+0000", 2024-05-15T13:10:00.171 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.15764613699999999 2024-05-15T13:10:00.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.171 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.171 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:00.171 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.490901+0000", 2024-05-15T13:10:00.171 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4898000000000003e-05 2024-05-15T13:10:00.171 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.171 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:00.171 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.171 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.171 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:63773 6.f 6:ff07ed7d:::benchmark_data_smithi177_122278_object63772:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-05-15T13:10:00.171 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:08:25.334876+0000", 2024-05-15T13:10:00.172 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 94.786086363999999, 2024-05-15T13:10:00.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.159621402, 2024-05-15T13:10:00.172 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:00.172 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:00.172 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:00.172 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:10:00.172 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:10:00.172 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 63773 2024-05-15T13:10:00.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.172 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:00.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.172 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:00.172 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.334876+0000", 2024-05-15T13:10:00.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.173 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:00.173 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.334876+0000", 2024-05-15T13:10:00.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.173 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:00.173 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.334885+0000", 2024-05-15T13:10:00.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5670000000000001e-06 2024-05-15T13:10:00.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.173 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:00.173 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.334921+0000", 2024-05-15T13:10:00.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5475000000000002e-05 2024-05-15T13:10:00.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.174 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:00.174 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.485105+0000", 2024-05-15T13:10:00.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.15018420099999999 2024-05-15T13:10:00.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.174 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:00.174 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.485124+0000", 2024-05-15T13:10:00.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8859e-05 2024-05-15T13:10:00.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:00.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.485385+0000", 2024-05-15T13:10:00.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00026160300000000001 2024-05-15T13:10:00.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:00.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.485546+0000", 2024-05-15T13:10:00.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016093499999999999 2024-05-15T13:10:00.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:10:00.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.485702+0000", 2024-05-15T13:10:00.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015576199999999999 2024-05-15T13:10:00.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:00.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.491838+0000", 2024-05-15T13:10:00.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0061361580000000001 2024-05-15T13:10:00.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:00.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.494310+0000", 2024-05-15T13:10:00.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0024715119999999999 2024-05-15T13:10:00.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:00.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.494402+0000", 2024-05-15T13:10:00.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2398999999999994e-05 2024-05-15T13:10:00.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.177 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:00.177 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.494451+0000", 2024-05-15T13:10:00.177 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9452e-05 2024-05-15T13:10:00.177 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.177 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.177 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:00.177 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.494497+0000", 2024-05-15T13:10:00.177 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5479000000000003e-05 2024-05-15T13:10:00.177 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.177 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:00.177 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.177 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.177 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.177 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25735.0:63786 6.0 e524/509 6:0d41e0ee:::benchmark_data_smithi177_122278_object63785:head v 524'3976, mlcod=524'3973)", 2024-05-15T13:10:00.177 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:08:25.346581+0000", 2024-05-15T13:10:00.178 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 94.774380742000005, 2024-05-15T13:10:00.178 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.14434434199999999, 2024-05-15T13:10:00.178 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:00.178 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:00.178 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:00.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.178 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:00.178 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.346581+0000", 2024-05-15T13:10:00.178 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.178 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:00.178 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.346581+0000", 2024-05-15T13:10:00.178 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.179 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:00.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.346583+0000", 2024-05-15T13:10:00.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6819999999999999e-06 2024-05-15T13:10:00.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.179 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:00.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.346611+0000", 2024-05-15T13:10:00.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7719e-05 2024-05-15T13:10:00.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.179 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:00.180 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.485118+0000", 2024-05-15T13:10:00.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.138507086 2024-05-15T13:10:00.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:00.180 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.485125+0000", 2024-05-15T13:10:00.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3420000000000002e-06 2024-05-15T13:10:00.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:00.180 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.487181+0000", 2024-05-15T13:10:00.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020555299999999999 2024-05-15T13:10:00.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:00.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.487202+0000", 2024-05-15T13:10:00.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1688e-05 2024-05-15T13:10:00.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:00.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.490903+0000", 2024-05-15T13:10:00.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0037009959999999998 2024-05-15T13:10:00.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:00.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.490926+0000", 2024-05-15T13:10:00.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2299000000000001e-05 2024-05-15T13:10:00.182 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.182 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:00.182 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.182 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25735.0:63788 6.e e524/509 6:70fd4442:::benchmark_data_smithi177_122278_object63787:head v 524'3991, mlcod=524'3988)", 2024-05-15T13:10:00.182 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:08:25.349136+0000", 2024-05-15T13:10:00.182 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 94.771825965999994, 2024-05-15T13:10:00.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.14322212200000001, 2024-05-15T13:10:00.182 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:00.182 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:00.182 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:00.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.182 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:00.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.349136+0000", 2024-05-15T13:10:00.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:00.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.349136+0000", 2024-05-15T13:10:00.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:00.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.349137+0000", 2024-05-15T13:10:00.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.111e-06 2024-05-15T13:10:00.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.184 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:00.184 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.349198+0000", 2024-05-15T13:10:00.184 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1155000000000007e-05 2024-05-15T13:10:00.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.184 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.184 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:00.184 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.485051+0000", 2024-05-15T13:10:00.184 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.135852416 2024-05-15T13:10:00.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.184 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.184 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:00.184 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.485063+0000", 2024-05-15T13:10:00.184 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2038999999999999e-05 2024-05-15T13:10:00.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.185 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.185 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:00.185 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.485136+0000", 2024-05-15T13:10:00.185 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3583000000000003e-05 2024-05-15T13:10:00.185 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.185 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.185 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:00.185 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.485161+0000", 2024-05-15T13:10:00.185 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4528e-05 2024-05-15T13:10:00.185 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.185 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.185 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:00.185 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.492307+0000", 2024-05-15T13:10:00.185 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0071459879999999998 2024-05-15T13:10:00.185 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.186 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.186 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:00.186 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:25.492358+0000", 2024-05-15T13:10:00.186 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1301999999999998e-05 2024-05-15T13:10:00.186 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.186 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:00.186 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.186 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.186 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:113373 6.f 6:feeb6092:::benchmark_data_smithi177_122278_object113372:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e537)", 2024-05-15T13:10:00.186 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:09:06.977126+0000", 2024-05-15T13:10:00.186 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 53.143835826, 2024-05-15T13:10:00.186 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.089767329000000007, 2024-05-15T13:10:00.186 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:00.187 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:00.187 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:00.187 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:10:00.187 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:10:00.187 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 113373 2024-05-15T13:10:00.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.187 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:00.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.187 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:00.187 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:06.977126+0000", 2024-05-15T13:10:00.187 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.187 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:00.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:06.977126+0000", 2024-05-15T13:10:00.188 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:00.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:06.977128+0000", 2024-05-15T13:10:00.188 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.466e-06 2024-05-15T13:10:00.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:00.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:06.977175+0000", 2024-05-15T13:10:00.188 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6934999999999997e-05 2024-05-15T13:10:00.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:00.189 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:07.037593+0000", 2024-05-15T13:10:00.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.060418938999999998 2024-05-15T13:10:00.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.189 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:00.189 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:07.037607+0000", 2024-05-15T13:10:00.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3638999999999999e-05 2024-05-15T13:10:00.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.189 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:00.189 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:07.046451+0000", 2024-05-15T13:10:00.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.008843867 2024-05-15T13:10:00.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:00.190 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:07.046606+0000", 2024-05-15T13:10:00.190 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000155348 2024-05-15T13:10:00.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:10:00.190 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:07.046776+0000", 2024-05-15T13:10:00.190 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000169624 2024-05-15T13:10:00.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:00.190 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:07.058944+0000", 2024-05-15T13:10:00.190 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012168554999999999 2024-05-15T13:10:00.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.191 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:00.191 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:07.059785+0000", 2024-05-15T13:10:00.191 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00084074900000000001 2024-05-15T13:10:00.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.191 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:00.191 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:07.066758+0000", 2024-05-15T13:10:00.191 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0069726850000000002 2024-05-15T13:10:00.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.191 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:00.191 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:07.066819+0000", 2024-05-15T13:10:00.191 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0909000000000002e-05 2024-05-15T13:10:00.192 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.192 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.192 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:00.192 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:07.066893+0000", 2024-05-15T13:10:00.192 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4612999999999999e-05 2024-05-15T13:10:00.192 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.192 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:00.192 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.192 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.192 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.192 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:113481 6.f 6:fdd6e4fa:::benchmark_data_smithi177_122278_object113480:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e537)", 2024-05-15T13:10:00.192 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:09:07.134803+0000", 2024-05-15T13:10:00.192 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 52.986158736, 2024-05-15T13:10:00.192 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.090959023999999999, 2024-05-15T13:10:00.193 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:00.193 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:00.193 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:00.193 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:10:00.193 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:10:00.193 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 113481 2024-05-15T13:10:00.193 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.193 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:00.193 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.193 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:00.193 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:07.134803+0000", 2024-05-15T13:10:00.193 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.193 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.193 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.193 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:00.194 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:07.134803+0000", 2024-05-15T13:10:00.194 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.194 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.194 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.194 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:00.194 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:07.134805+0000", 2024-05-15T13:10:00.194 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7489999999999999e-06 2024-05-15T13:10:00.194 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.194 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.194 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:00.194 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:07.134870+0000", 2024-05-15T13:10:00.194 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4982999999999998e-05 2024-05-15T13:10:00.194 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.194 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.195 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:00.195 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:07.134883+0000", 2024-05-15T13:10:00.195 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2870999999999999e-05 2024-05-15T13:10:00.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.195 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:00.195 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:07.134891+0000", 2024-05-15T13:10:00.195 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3899999999999993e-06 2024-05-15T13:10:00.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.195 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:00.195 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:07.143947+0000", 2024-05-15T13:10:00.195 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090560160000000001 2024-05-15T13:10:00.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.196 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:00.196 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:07.144036+0000", 2024-05-15T13:10:00.196 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8613e-05 2024-05-15T13:10:00.196 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.196 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.196 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:10:00.196 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:07.144143+0000", 2024-05-15T13:10:00.196 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010704699999999999 2024-05-15T13:10:00.196 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.196 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.196 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:00.196 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:07.210978+0000", 2024-05-15T13:10:00.196 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.066835614000000002 2024-05-15T13:10:00.196 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.197 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.197 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:00.197 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:07.211021+0000", 2024-05-15T13:10:00.197 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2561000000000001e-05 2024-05-15T13:10:00.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.197 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.197 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:00.197 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:07.225640+0000", 2024-05-15T13:10:00.197 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014619096999999999 2024-05-15T13:10:00.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.197 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.197 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:00.197 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:07.225696+0000", 2024-05-15T13:10:00.197 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6081000000000001e-05 2024-05-15T13:10:00.198 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.198 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.198 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:00.198 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:07.225762+0000", 2024-05-15T13:10:00.198 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6001999999999996e-05 2024-05-15T13:10:00.198 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.198 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:00.198 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.198 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.198 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.198 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25735.0:115237 6.e e537/509 6:7e9c9295:::benchmark_data_smithi177_122278_object115236:head v 537'7179, mlcod=537'7173)", 2024-05-15T13:10:00.198 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:09:09.044326+0000", 2024-05-15T13:10:00.198 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 51.076636233999999, 2024-05-15T13:10:00.199 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.085015289999999993, 2024-05-15T13:10:00.199 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:00.199 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:00.199 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:00.199 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.199 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:00.199 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.044326+0000", 2024-05-15T13:10:00.199 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.199 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.199 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.199 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:00.199 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.044326+0000", 2024-05-15T13:10:00.199 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.199 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.200 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.200 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:00.200 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.044327+0000", 2024-05-15T13:10:00.200 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1100000000000004e-07 2024-05-15T13:10:00.200 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.200 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.200 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:00.200 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.044452+0000", 2024-05-15T13:10:00.200 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012558400000000001 2024-05-15T13:10:00.200 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.200 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.200 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:00.200 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.044461+0000", 2024-05-15T13:10:00.200 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1730000000000001e-06 2024-05-15T13:10:00.201 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.201 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.201 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:00.201 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.044468+0000", 2024-05-15T13:10:00.201 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4479999999999999e-06 2024-05-15T13:10:00.201 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.201 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.201 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:00.201 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.048240+0000", 2024-05-15T13:10:00.201 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003772601 2024-05-15T13:10:00.201 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.201 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.201 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:00.202 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.048259+0000", 2024-05-15T13:10:00.202 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8219e-05 2024-05-15T13:10:00.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.202 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.202 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:00.202 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.051743+0000", 2024-05-15T13:10:00.202 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034842200000000001 2024-05-15T13:10:00.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.202 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.202 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:00.202 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.129341+0000", 2024-05-15T13:10:00.202 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.077598133999999999 2024-05-15T13:10:00.202 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.202 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:00.203 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.203 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.203 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.203 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25735.0:115246 6.e e537/509 6:79dff326:::benchmark_data_smithi177_122278_object115245:head v 537'7180, mlcod=537'7173)", 2024-05-15T13:10:00.203 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:09:09.050771+0000", 2024-05-15T13:10:00.203 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 51.070191065000003, 2024-05-15T13:10:00.203 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.083264510999999999, 2024-05-15T13:10:00.203 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:00.203 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:00.203 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:00.203 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.203 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:00.203 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.050771+0000", 2024-05-15T13:10:00.203 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.204 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.204 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.204 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:00.204 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.050771+0000", 2024-05-15T13:10:00.204 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.204 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.204 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.204 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:00.204 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.050772+0000", 2024-05-15T13:10:00.204 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4700000000000001e-07 2024-05-15T13:10:00.204 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.204 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.204 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:00.204 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.050834+0000", 2024-05-15T13:10:00.204 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2510999999999998e-05 2024-05-15T13:10:00.205 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.205 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.205 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:00.205 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.050843+0000", 2024-05-15T13:10:00.205 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7809999999999994e-06 2024-05-15T13:10:00.205 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.205 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.205 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:00.205 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.050848+0000", 2024-05-15T13:10:00.205 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1080000000000001e-06 2024-05-15T13:10:00.205 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.205 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.205 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:00.206 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.129448+0000", 2024-05-15T13:10:00.206 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.078600041999999995 2024-05-15T13:10:00.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.206 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.206 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:00.206 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.129468+0000", 2024-05-15T13:10:00.206 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.961e-05 2024-05-15T13:10:00.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.206 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.206 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:00.206 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.133978+0000", 2024-05-15T13:10:00.206 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0045107130000000004 2024-05-15T13:10:00.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.206 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.207 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:00.207 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.134035+0000", 2024-05-15T13:10:00.207 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6999000000000001e-05 2024-05-15T13:10:00.207 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.207 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:00.207 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.207 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.207 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25735.0:115245 6.5 e537/509 6:ac3d69b4:::benchmark_data_smithi177_122278_object115244:head v 537'7277, mlcod=537'7275)", 2024-05-15T13:10:00.207 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:09:09.051725+0000", 2024-05-15T13:10:00.207 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 51.069236646, 2024-05-15T13:10:00.207 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.088390498999999997, 2024-05-15T13:10:00.207 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:00.207 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:00.207 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:00.208 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.208 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:00.208 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.051725+0000", 2024-05-15T13:10:00.208 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.208 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.208 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.208 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:00.208 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.051725+0000", 2024-05-15T13:10:00.208 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.208 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.208 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.208 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:00.209 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.051726+0000", 2024-05-15T13:10:00.209 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2699999999999999e-07 2024-05-15T13:10:00.209 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.209 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.209 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:00.209 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.051761+0000", 2024-05-15T13:10:00.209 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5407999999999997e-05 2024-05-15T13:10:00.209 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.209 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.209 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:00.209 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.129303+0000", 2024-05-15T13:10:00.209 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.077542036999999994 2024-05-15T13:10:00.209 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.209 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.210 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:00.210 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.129321+0000", 2024-05-15T13:10:00.210 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7456e-05 2024-05-15T13:10:00.210 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.210 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.210 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:00.210 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.129387+0000", 2024-05-15T13:10:00.210 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6019999999999995e-05 2024-05-15T13:10:00.210 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.210 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.210 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:00.210 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.129408+0000", 2024-05-15T13:10:00.210 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0585e-05 2024-05-15T13:10:00.211 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.211 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.211 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:00.211 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.140062+0000", 2024-05-15T13:10:00.211 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010654248 2024-05-15T13:10:00.211 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.211 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.211 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:00.211 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.140116+0000", 2024-05-15T13:10:00.211 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4017999999999997e-05 2024-05-15T13:10:00.211 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.211 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:00.211 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.211 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.211 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.212 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:115257 6.f 6:f7572c1d:::benchmark_data_smithi177_122278_object115256:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e537)", 2024-05-15T13:10:00.212 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:09:09.058678+0000", 2024-05-15T13:10:00.212 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 51.062283583999999, 2024-05-15T13:10:00.212 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.082020284999999998, 2024-05-15T13:10:00.212 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:00.212 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:00.212 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:00.212 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:10:00.212 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:10:00.212 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 115257 2024-05-15T13:10:00.212 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.212 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:00.212 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.213 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:00.213 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.058678+0000", 2024-05-15T13:10:00.213 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.213 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.213 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.213 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:00.213 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.058678+0000", 2024-05-15T13:10:00.213 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:00.213 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.213 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.213 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:00.213 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.058680+0000", 2024-05-15T13:10:00.213 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7659999999999999e-06 2024-05-15T13:10:00.213 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.214 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.214 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:00.214 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.058714+0000", 2024-05-15T13:10:00.214 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3973999999999997e-05 2024-05-15T13:10:00.214 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.214 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.214 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:00.214 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.129489+0000", 2024-05-15T13:10:00.214 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.070775014999999997 2024-05-15T13:10:00.214 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.214 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.214 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:00.214 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.129501+0000", 2024-05-15T13:10:00.215 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2357e-05 2024-05-15T13:10:00.215 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.215 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.215 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:00.215 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.129563+0000", 2024-05-15T13:10:00.215 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1699000000000007e-05 2024-05-15T13:10:00.215 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.215 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.215 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:00.215 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.129717+0000", 2024-05-15T13:10:00.215 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015374099999999999 2024-05-15T13:10:00.215 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.215 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.215 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:10:00.215 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.129865+0000", 2024-05-15T13:10:00.216 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014830400000000001 2024-05-15T13:10:00.216 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.216 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.216 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:00.216 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.140531+0000", 2024-05-15T13:10:00.216 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010665857000000001 2024-05-15T13:10:00.216 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.216 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.216 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:00.216 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.140544+0000", 2024-05-15T13:10:00.216 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2795e-05 2024-05-15T13:10:00.216 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.216 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.216 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:00.216 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.140595+0000", 2024-05-15T13:10:00.217 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0905999999999997e-05 2024-05-15T13:10:00.217 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.217 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.217 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:00.217 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.140655+0000", 2024-05-15T13:10:00.217 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9740000000000001e-05 2024-05-15T13:10:00.217 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:00.217 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:00.217 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:00.217 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:09.140699+0000", 2024-05-15T13:10:00.217 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4131000000000001e-05 2024-05-15T13:10:00.217 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.217 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:00.217 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.218 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:00.218 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:00.218 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:10:00.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:00.219 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:10:00.219 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:10:00.320 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:10:00.376 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:00.402 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:10:00.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:00.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:10:00.512 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:10:00.520 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:00 smithi027 ceph-mon[39795]: overall HEALTH_OK 2024-05-15T13:10:00.540 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:00 smithi177 ceph-mon[44303]: overall HEALTH_OK 2024-05-15T13:10:00.550 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:00.566 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:10:00.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:00.572 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:10:00.673 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:10:00.705 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:00.737 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:10:00.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:00.744 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:10:00.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:00 smithi120 ceph-mon[43961]: overall HEALTH_OK 2024-05-15T13:10:00.845 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:10:00.934 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:01.029 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:10:01.039 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:10:01.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:01.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:10:01.083 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-05-15T13:10:01.153 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:10:01.191 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:01.209 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T13:10:01.209 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T13:10:01.210 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-05-15T13:10:01.216 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:10:01.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:01.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:10:01.322 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:10:01.353 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:01.374 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:10:01.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:01.381 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:10:01.482 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:10:01.512 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:01.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:01 smithi027 ceph-mon[39795]: pgmap v1459: 27 pgs: 27 active+clean; 6.1 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 937 op/s 2024-05-15T13:10:01.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:01 smithi120 ceph-mon[43961]: pgmap v1459: 27 pgs: 27 active+clean; 6.1 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 937 op/s 2024-05-15T13:10:01.533 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:10:01.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:01.536 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:10:01.600 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:10:01.637 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:10:01.661 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:01 smithi177 ceph-mon[44303]: pgmap v1459: 27 pgs: 27 active+clean; 6.1 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 937 op/s 2024-05-15T13:10:01.663 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:01.682 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:10:01.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:01.685 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:10:01.786 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:10:01.823 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:01.862 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:10:01.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:01.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:10:01.978 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:10:02.022 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:02.051 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:10:02.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:02.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:10:02.160 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:10:02.189 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:02.216 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:10:02.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:02.219 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:10:02.320 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:10:02.336 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:10:02.352 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:02.378 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:10:02.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:02.382 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:10:02.483 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:10:02.589 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:02 smithi027 ceph-mon[39795]: pgmap v1460: 27 pgs: 27 active+clean; 6.0 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 830 op/s 2024-05-15T13:10:02.589 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:10:02.654 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:10:02.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:02.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:10:02.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:02 smithi177 ceph-mon[44303]: pgmap v1460: 27 pgs: 27 active+clean; 6.0 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 830 op/s 2024-05-15T13:10:02.703 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:10:02.765 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:10:02.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:02 smithi120 ceph-mon[43961]: pgmap v1460: 27 pgs: 27 active+clean; 6.0 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 830 op/s 2024-05-15T13:10:02.800 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:02.823 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:10:02.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:02.826 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:10:02.927 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:10:03.013 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:04.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:04 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:10:04.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:04 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:10:04.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:04 smithi177 ceph-mon[44303]: osdmap e550: 12 total, 12 up, 11 in 2024-05-15T13:10:04.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:04 smithi177 ceph-mon[44303]: 6.d scrub starts 2024-05-15T13:10:04.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:04 smithi177 ceph-mon[44303]: pgmap v1462: 27 pgs: 27 active+clean; 5.9 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 902 op/s 2024-05-15T13:10:04.688 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:04 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:10:04.688 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:04 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:10:04.688 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:04 smithi027 ceph-mon[39795]: osdmap e550: 12 total, 12 up, 11 in 2024-05-15T13:10:04.688 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:04 smithi027 ceph-mon[39795]: 6.d scrub starts 2024-05-15T13:10:04.688 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:04 smithi027 ceph-mon[39795]: pgmap v1462: 27 pgs: 27 active+clean; 5.9 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 902 op/s 2024-05-15T13:10:04.706 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:10:04.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:04.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:10:04.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:04 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:10:04.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:04 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:10:04.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:04 smithi120 ceph-mon[43961]: osdmap e550: 12 total, 12 up, 11 in 2024-05-15T13:10:04.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:04 smithi120 ceph-mon[43961]: 6.d scrub starts 2024-05-15T13:10:04.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:04 smithi120 ceph-mon[43961]: pgmap v1462: 27 pgs: 27 active+clean; 5.9 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 902 op/s 2024-05-15T13:10:04.812 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:10:04.869 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:04.938 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:10:04.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:04.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:10:05.042 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:10:05.079 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:05.106 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:10:05.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:05.110 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:10:05.211 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:10:05.257 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:05.276 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:10:05.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:05.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:10:05.383 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:10:05.440 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:05.495 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:05 smithi027 ceph-mon[39795]: 6.d scrub ok 2024-05-15T13:10:05.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:05 smithi120 ceph-mon[43961]: 6.d scrub ok 2024-05-15T13:10:05.719 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:10:05.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:05.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:10:05.826 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:10:05.913 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:05.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:05 smithi177 ceph-mon[44303]: 6.d scrub ok 2024-05-15T13:10:05.956 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:10:05.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:05.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:10:06.073 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:10:06.102 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:06.122 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:10:06.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:06.127 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:10:06.228 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:10:06.277 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:06.298 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:10:06.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:06.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:10:06.402 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:10:06.449 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:10:06.453 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:06.478 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:10:06.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:06.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:10:06.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:06 smithi027 ceph-mon[39795]: 1.5 scrub starts 2024-05-15T13:10:06.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:06 smithi027 ceph-mon[39795]: 1.5 scrub ok 2024-05-15T13:10:06.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:06 smithi027 ceph-mon[39795]: pgmap v1463: 27 pgs: 27 active+clean; 5.8 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 820 op/s 2024-05-15T13:10:06.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:06 smithi120 ceph-mon[43961]: 1.5 scrub starts 2024-05-15T13:10:06.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:06 smithi120 ceph-mon[43961]: 1.5 scrub ok 2024-05-15T13:10:06.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:06 smithi120 ceph-mon[43961]: pgmap v1463: 27 pgs: 27 active+clean; 5.8 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 820 op/s 2024-05-15T13:10:06.584 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:10:06.665 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:06.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:06 smithi177 ceph-mon[44303]: 1.5 scrub starts 2024-05-15T13:10:06.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:06 smithi177 ceph-mon[44303]: 1.5 scrub ok 2024-05-15T13:10:06.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:06 smithi177 ceph-mon[44303]: pgmap v1463: 27 pgs: 27 active+clean; 5.8 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 820 op/s 2024-05-15T13:10:07.565 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:10:07.745 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:07 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1124389094' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:10:07.746 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:07 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:10:07.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:07 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1124389094' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:10:07.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:07 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:10:07.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:07 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1124389094' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:10:07.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:07 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:10:08.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:08 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:10:08.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:08 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:10:08.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:08 smithi027 ceph-mon[39795]: osdmap e551: 12 total, 12 up, 11 in 2024-05-15T13:10:08.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:08 smithi027 ceph-mon[39795]: pgmap v1465: 27 pgs: 27 active+clean; 5.7 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 721 op/s 2024-05-15T13:10:08.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:08 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:10:08.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:08 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:10:08.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:08 smithi120 ceph-mon[43961]: osdmap e551: 12 total, 12 up, 11 in 2024-05-15T13:10:08.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:08 smithi120 ceph-mon[43961]: pgmap v1465: 27 pgs: 27 active+clean; 5.7 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 721 op/s 2024-05-15T13:10:08.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:08 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:10:08.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:08 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:10:08.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:08 smithi177 ceph-mon[44303]: osdmap e551: 12 total, 12 up, 11 in 2024-05-15T13:10:08.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:08 smithi177 ceph-mon[44303]: pgmap v1465: 27 pgs: 27 active+clean; 5.7 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 721 op/s 2024-05-15T13:10:09.151 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:10:09.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:09.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:10:09.260 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:10:09.325 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:10:09.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:09.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:10:09.496 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:10:09.553 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:10.277 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:10:10.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:10.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:10:10.388 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:10:10.421 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:no valid command found; 10 closest matches: 2024-05-15T13:10:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:0 2024-05-15T13:10:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:1 2024-05-15T13:10:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2 2024-05-15T13:10:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:abort 2024-05-15T13:10:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:assert 2024-05-15T13:10:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:bench [] [] [] [] 2024-05-15T13:10:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs debug_inject_read_zeros 2024-05-15T13:10:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs files list 2024-05-15T13:10:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs stats 2024-05-15T13:10:10.424 INFO:teuthology.orchestra.run.smithi027.stderr:bluestore allocator dump block 2024-05-15T13:10:10.424 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: invalid command 2024-05-15T13:10:10.443 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:10:10.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:10.446 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:10:10.547 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:10:10.578 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:10.616 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:10:10.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:10.623 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:10:10.723 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:10:10.762 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:10.787 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:10:10.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:10.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:10:10.893 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:10:10.974 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:11.077 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:10:11.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:11.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:10:11.137 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T13:10:11.138 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T13:10:11.139 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-05-15T13:10:11.182 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:10:11.191 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:10:11.246 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:11.280 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:10:11.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:11.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:10:11.391 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:10:11.449 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:11 smithi027 ceph-mon[39795]: pgmap v1466: 27 pgs: 27 active+clean; 5.6 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 915 op/s 2024-05-15T13:10:11.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:11 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:10:11.492 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:11.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:11 smithi120 ceph-mon[43961]: pgmap v1466: 27 pgs: 27 active+clean; 5.6 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 915 op/s 2024-05-15T13:10:11.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:11 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:10:11.565 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:10:11.566 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:10:11.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.566 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25735.0:170892 6.4 e551/509 6:239eaa90:::benchmark_data_smithi177_122278_object40521:head v 551'10555, mlcod=551'10549)", 2024-05-15T13:10:11.566 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:10:10.013978+0000", 2024-05-15T13:10:11.566 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.016528582, 2024-05-15T13:10:11.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016576832, 2024-05-15T13:10:11.566 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:11.566 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:10:11.566 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:11.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.013978+0000", 2024-05-15T13:10:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.013978+0000", 2024-05-15T13:10:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.013978+0000", 2024-05-15T13:10:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.68e-07 2024-05-15T13:10:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.014000+0000", 2024-05-15T13:10:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1271999999999999e-05 2024-05-15T13:10:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.014012+0000", 2024-05-15T13:10:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2734e-05 2024-05-15T13:10:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.014025+0000", 2024-05-15T13:10:11.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2675e-05 2024-05-15T13:10:11.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:11.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.014074+0000", 2024-05-15T13:10:11.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8884999999999997e-05 2024-05-15T13:10:11.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:11.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.014092+0000", 2024-05-15T13:10:11.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7941e-05 2024-05-15T13:10:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25735.0:170894 6.4 e551/509 6:2c5e9c89:::benchmark_data_smithi177_122278_object40523:head v 551'10556, mlcod=551'10549)", 2024-05-15T13:10:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:10:10.019103+0000", 2024-05-15T13:10:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.011403286, 2024-05-15T13:10:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011536113000000001, 2024-05-15T13:10:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:10:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.019103+0000", 2024-05-15T13:10:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.019103+0000", 2024-05-15T13:10:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.019103+0000", 2024-05-15T13:10:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4400000000000001e-07 2024-05-15T13:10:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.019113+0000", 2024-05-15T13:10:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8879999999999992e-06 2024-05-15T13:10:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.019122+0000", 2024-05-15T13:10:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9819999999999997e-06 2024-05-15T13:10:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.019128+0000", 2024-05-15T13:10:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7450000000000001e-06 2024-05-15T13:10:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.019383+0000", 2024-05-15T13:10:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00025453800000000001 2024-05-15T13:10:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.019407+0000", 2024-05-15T13:10:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4504000000000001e-05 2024-05-15T13:10:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25735.0:170900 6.7 e551/509 6:ea500c1d:::benchmark_data_smithi177_122278_object40529:head v 551'10773, mlcod=551'10771)", 2024-05-15T13:10:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:10:10.019648+0000", 2024-05-15T13:10:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.010858736000000001, 2024-05-15T13:10:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011064586, 2024-05-15T13:10:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:10:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.019648+0000", 2024-05-15T13:10:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.019648+0000", 2024-05-15T13:10:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.019648+0000", 2024-05-15T13:10:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5000000000000002e-07 2024-05-15T13:10:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.019654+0000", 2024-05-15T13:10:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1789999999999996e-06 2024-05-15T13:10:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.019666+0000", 2024-05-15T13:10:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1499e-05 2024-05-15T13:10:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.019673+0000", 2024-05-15T13:10:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8229999999999997e-06 2024-05-15T13:10:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.019739+0000", 2024-05-15T13:10:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6543000000000003e-05 2024-05-15T13:10:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.019762+0000", 2024-05-15T13:10:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2311999999999999e-05 2024-05-15T13:10:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25735.0:170895 6.4 e551/509 6:288ceb1f:::benchmark_data_smithi177_122278_object40524:head v 551'10557, mlcod=551'10553)", 2024-05-15T13:10:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:10:10.019803+0000", 2024-05-15T13:10:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.010703638, 2024-05-15T13:10:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010980688000000001, 2024-05-15T13:10:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:10:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.019803+0000", 2024-05-15T13:10:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.019803+0000", 2024-05-15T13:10:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.019803+0000", 2024-05-15T13:10:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0800000000000004e-07 2024-05-15T13:10:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.019809+0000", 2024-05-15T13:10:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5720000000000002e-06 2024-05-15T13:10:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:11.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.019818+0000", 2024-05-15T13:10:11.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2860000000000005e-06 2024-05-15T13:10:11.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:11.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.019824+0000", 2024-05-15T13:10:11.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2779999999999999e-06 2024-05-15T13:10:11.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:11.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.024250+0000", 2024-05-15T13:10:11.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0044251459999999996 2024-05-15T13:10:11.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:11.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.024270+0000", 2024-05-15T13:10:11.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0585999999999999e-05 2024-05-15T13:10:11.581 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:11.581 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:11.581 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:11.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.581 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25735.0:170902 6.8 e551/509 6:1b095cac:::benchmark_data_smithi177_122278_object40531:head v 551'10625, mlcod=551'10623)", 2024-05-15T13:10:11.581 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:10:10.020161+0000", 2024-05-15T13:10:11.582 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.010345182, 2024-05-15T13:10:11.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010695124, 2024-05-15T13:10:11.582 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:11.582 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:10:11.582 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:11.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:11.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.020161+0000", 2024-05-15T13:10:11.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:11.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:11.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.020161+0000", 2024-05-15T13:10:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.020162+0000", 2024-05-15T13:10:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9299999999999998e-07 2024-05-15T13:10:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.020167+0000", 2024-05-15T13:10:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7019999999999998e-06 2024-05-15T13:10:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:11.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.020176+0000", 2024-05-15T13:10:11.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2179999999999999e-06 2024-05-15T13:10:11.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.584 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:11.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.020181+0000", 2024-05-15T13:10:11.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8499999999999999e-06 2024-05-15T13:10:11.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:11.585 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.020268+0000", 2024-05-15T13:10:11.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6123000000000002e-05 2024-05-15T13:10:11.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:11.585 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.020286+0000", 2024-05-15T13:10:11.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8142e-05 2024-05-15T13:10:11.585 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:11.585 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:11.585 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:11.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25735.0:170904 6.8 e551/509 6:10f6da0e:::benchmark_data_smithi177_122278_object40533:head v 551'10626, mlcod=551'10623)", 2024-05-15T13:10:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:10:10.024259+0000", 2024-05-15T13:10:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.0062471870000000004, 2024-05-15T13:10:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0066673770000000004, 2024-05-15T13:10:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:10:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.024259+0000", 2024-05-15T13:10:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.024259+0000", 2024-05-15T13:10:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.024260+0000", 2024-05-15T13:10:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1650000000000001e-06 2024-05-15T13:10:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.024267+0000", 2024-05-15T13:10:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9099999999999999e-06 2024-05-15T13:10:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.024276+0000", 2024-05-15T13:10:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0929999999999992e-06 2024-05-15T13:10:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.024284+0000", 2024-05-15T13:10:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6650000000000003e-06 2024-05-15T13:10:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.025126+0000", 2024-05-15T13:10:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00084216100000000002 2024-05-15T13:10:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.025145+0000", 2024-05-15T13:10:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9409999999999999e-05 2024-05-15T13:10:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25735.0:170905 6.2 e551/509 6:4b3e6e6e:::benchmark_data_smithi177_122278_object40534:head v 551'10562, mlcod=551'10560)", 2024-05-15T13:10:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:10:10.024454+0000", 2024-05-15T13:10:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.0060519440000000001, 2024-05-15T13:10:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.006542712, 2024-05-15T13:10:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:10:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.024454+0000", 2024-05-15T13:10:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.024454+0000", 2024-05-15T13:10:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:11.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.024455+0000", 2024-05-15T13:10:11.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9100000000000003e-07 2024-05-15T13:10:11.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:11.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.024461+0000", 2024-05-15T13:10:11.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2589999999999997e-06 2024-05-15T13:10:11.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:11.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.024470+0000", 2024-05-15T13:10:11.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1530000000000003e-06 2024-05-15T13:10:11.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:11.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.024476+0000", 2024-05-15T13:10:11.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7130000000000002e-06 2024-05-15T13:10:11.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:11.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.025197+0000", 2024-05-15T13:10:11.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000720982 2024-05-15T13:10:11.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:11.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.025215+0000", 2024-05-15T13:10:11.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7931000000000002e-05 2024-05-15T13:10:11.593 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:11.593 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:11.593 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:11.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.593 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25735.0:170899 6.0 e551/509 6:0415fba3:::benchmark_data_smithi177_122278_object40528:head v 551'10625, mlcod=551'10623)", 2024-05-15T13:10:11.593 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:10:10.029775+0000", 2024-05-15T13:10:11.593 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.00073100800000000001, 2024-05-15T13:10:11.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0012935290000000001, 2024-05-15T13:10:11.594 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:11.594 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:10:11.594 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:11.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:11.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.029775+0000", 2024-05-15T13:10:11.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:11.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:11.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.029775+0000", 2024-05-15T13:10:11.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:11.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:11.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.029776+0000", 2024-05-15T13:10:11.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6099999999999999e-07 2024-05-15T13:10:11.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:11.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.029785+0000", 2024-05-15T13:10:11.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8729999999999996e-06 2024-05-15T13:10:11.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:11.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.029795+0000", 2024-05-15T13:10:11.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0108e-05 2024-05-15T13:10:11.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:11.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.029803+0000", 2024-05-15T13:10:11.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7840000000000004e-06 2024-05-15T13:10:11.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.597 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:11.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.029846+0000", 2024-05-15T13:10:11.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3285000000000003e-05 2024-05-15T13:10:11.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.597 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:11.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.029861+0000", 2024-05-15T13:10:11.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4961e-05 2024-05-15T13:10:11.597 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:11.597 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:11.597 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:11.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.598 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25735.0:170907 6.8 e551/509 6:15d3035e:::benchmark_data_smithi177_122278_object40536:head v 551'10627, mlcod=551'10623)", 2024-05-15T13:10:11.598 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:10:10.030340+0000", 2024-05-15T13:10:11.598 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.00016658800000000001, 2024-05-15T13:10:11.598 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00080972399999999995, 2024-05-15T13:10:11.598 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:11.598 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:10:11.598 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:11.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.598 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:11.598 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.030340+0000", 2024-05-15T13:10:11.598 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:11.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:11.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.030340+0000", 2024-05-15T13:10:11.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:11.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:11.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.030340+0000", 2024-05-15T13:10:11.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1299999999999999e-07 2024-05-15T13:10:11.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:11.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.030347+0000", 2024-05-15T13:10:11.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8240000000000002e-06 2024-05-15T13:10:11.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:11.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.030355+0000", 2024-05-15T13:10:11.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1340000000000004e-06 2024-05-15T13:10:11.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:11.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.030362+0000", 2024-05-15T13:10:11.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5880000000000003e-06 2024-05-15T13:10:11.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:11.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.030412+0000", 2024-05-15T13:10:11.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9580999999999998e-05 2024-05-15T13:10:11.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:11.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:11.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:11.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:10.030430+0000", 2024-05-15T13:10:11.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7989e-05 2024-05-15T13:10:11.601 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:11.601 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:11.601 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:11.601 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:11.601 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:10:11.602 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 9 2024-05-15T13:10:11.602 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:10:11.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:11 smithi177 ceph-mon[44303]: pgmap v1466: 27 pgs: 27 active+clean; 5.6 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 915 op/s 2024-05-15T13:10:11.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:11 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:10:12.017 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:10:12.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:12.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:10:12.146 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:10:12.181 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:12.209 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:10:12.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:12.211 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:10:12.312 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:10:12.339 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:12.365 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:10:12.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:12.369 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:10:12.466 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-05-15T13:10:12.470 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:10:12.545 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:12.602 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:10:12.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:12.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:10:12.718 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:10:12.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:12 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:10:12.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:12 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:10:12.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:12 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:10:12.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:12 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:10:12.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:12 smithi027 ceph-mon[39795]: pgmap v1467: 27 pgs: 27 active+clean; 5.5 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 808 op/s 2024-05-15T13:10:12.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:12 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:10:12.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:12 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:10:12.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:12 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:10:12.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:12 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:10:12.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:12 smithi120 ceph-mon[43961]: pgmap v1467: 27 pgs: 27 active+clean; 5.5 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 808 op/s 2024-05-15T13:10:12.814 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:12.859 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:10:12.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:12.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:10:12.907 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:10:12.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:12 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:10:12.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:12 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:10:12.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:12 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:10:12.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:12 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:10:12.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:12 smithi177 ceph-mon[44303]: pgmap v1467: 27 pgs: 27 active+clean; 5.5 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 808 op/s 2024-05-15T13:10:12.961 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:10:12.967 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:10:12.981 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:10:13.009 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:13.031 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:10:13.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:13.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:10:13.103 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:10:13.136 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:10:13.151 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:10:13.169 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:13.189 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:10:13.206 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:10:13.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:13.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:10:13.229 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:10:13.278 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:10:13.313 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:10:13.318 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:10:13.348 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:10:13.396 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:13.396 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:10:13.456 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:10:13.488 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:10:13.519 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:10:13.565 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:10:13.585 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:10:13.585 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:10:13.585 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:10:13.585 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:10:13.585 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:10:13.585 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:10:13.585 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:10:13.585 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:10:13.586 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:10:13.586 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:10:13.586 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:10:13.586 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:10:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:10:13.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:13.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:10:13.784 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:10:13.813 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:13.834 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:10:13.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:13.837 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:10:13.938 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:10:13.998 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:14.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:13 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:10:14.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:13 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:10:14.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:13 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:10:14.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:13 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:10:14.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:13 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:10:14.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:13 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/301312913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:10:14.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:13 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:10:14.272 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:13 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:10:14.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:13 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:10:14.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:13 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:10:14.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:13 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:10:14.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:13 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/301312913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:10:14.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:13 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:10:14.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:13 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:10:14.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:13 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:10:14.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:13 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:10:14.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:13 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:10:14.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:13 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/301312913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:10:15.020 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:14 smithi027 ceph-mon[39795]: pgmap v1468: 27 pgs: 27 active+clean; 5.4 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 827 op/s 2024-05-15T13:10:15.072 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:10:15.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:15.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:10:15.178 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:10:15.213 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:15.244 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:10:15.249 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:14 smithi120 ceph-mon[43961]: pgmap v1468: 27 pgs: 27 active+clean; 5.4 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 827 op/s 2024-05-15T13:10:15.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:15.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:10:15.352 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:10:15.387 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:15.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:14 smithi177 ceph-mon[44303]: pgmap v1468: 27 pgs: 27 active+clean; 5.4 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 827 op/s 2024-05-15T13:10:17.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:17 smithi027 ceph-mon[39795]: pgmap v1469: 27 pgs: 27 active+clean; 5.3 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 826 op/s 2024-05-15T13:10:17.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:17 smithi120 ceph-mon[43961]: pgmap v1469: 27 pgs: 27 active+clean; 5.3 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 826 op/s 2024-05-15T13:10:17.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:17 smithi177 ceph-mon[44303]: pgmap v1469: 27 pgs: 27 active+clean; 5.3 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 826 op/s 2024-05-15T13:10:18.228 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:10:18.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:18.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:10:18.336 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:10:18.376 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:18.402 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:10:18.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:18.408 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:10:18.509 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:10:18.542 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:18.578 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:10:18.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:18.581 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:10:18.681 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:10:18.751 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:18.844 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:10:18.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:18.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:10:18.950 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:10:18.998 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:19.025 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:10:19.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:19.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:10:19.130 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:10:19.168 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:19.193 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:10:19.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:19.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:10:19.297 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:10:19.346 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:19.377 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:10:19.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:19.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:10:19.481 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:10:19.516 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:19.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:19 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:10:19.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:19 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:10:19.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:19 smithi027 ceph-mon[39795]: pgmap v1470: 27 pgs: 27 active+clean; 5.2 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 828 op/s 2024-05-15T13:10:19.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:19 smithi027 ceph-mon[39795]: osdmap e552: 12 total, 12 up, 11 in 2024-05-15T13:10:19.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:19 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:10:19.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:19 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:10:19.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:19 smithi120 ceph-mon[43961]: pgmap v1470: 27 pgs: 27 active+clean; 5.2 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 828 op/s 2024-05-15T13:10:19.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:19 smithi120 ceph-mon[43961]: osdmap e552: 12 total, 12 up, 11 in 2024-05-15T13:10:19.540 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:10:19.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:19.543 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:10:19.548 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:19 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:10:19.548 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:19 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:10:19.548 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:19 smithi177 ceph-mon[44303]: pgmap v1470: 27 pgs: 27 active+clean; 5.2 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 828 op/s 2024-05-15T13:10:19.548 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:19 smithi177 ceph-mon[44303]: osdmap e552: 12 total, 12 up, 11 in 2024-05-15T13:10:19.644 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:10:19.676 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:19.695 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:10:19.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:19.698 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:10:19.799 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:10:19.827 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:19.851 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:10:19.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:19.853 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:10:19.954 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:10:19.996 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:20.034 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:10:20.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:20.037 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:10:20.138 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:10:20.172 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:20.203 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:10:20.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:20.212 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:10:20.313 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:10:20.405 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:20.520 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:10:20.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:20.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:10:20.626 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:10:20.638 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:10:20.714 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:20.831 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:10:20.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:20.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:10:20.937 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:10:21.066 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:21.339 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:10:21.356 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:21 smithi027 ceph-mon[39795]: pgmap v1472: 27 pgs: 27 active+clean; 5.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 842 op/s 2024-05-15T13:10:21.356 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:21 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2979544956' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:10:21.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:21 smithi177 ceph-mon[44303]: pgmap v1472: 27 pgs: 27 active+clean; 5.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 842 op/s 2024-05-15T13:10:21.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:21 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2979544956' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:10:21.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:21 smithi120 ceph-mon[43961]: pgmap v1472: 27 pgs: 27 active+clean; 5.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 842 op/s 2024-05-15T13:10:21.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:21 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2979544956' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:10:22.161 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:10:22.162 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:10:22.460 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:10:22.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:22.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:10:22.572 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:10:22.599 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:22 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:10:22.599 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:22 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2979544956' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:10:22.599 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:22 smithi177 ceph-mon[44303]: osdmap e553: 12 total, 12 up, 11 in 2024-05-15T13:10:22.601 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:22.626 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:10:22.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:22.629 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:10:22.730 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:10:22.774 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:22 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:10:22.775 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:22 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2979544956' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:10:22.775 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:22 smithi120 ceph-mon[43961]: osdmap e553: 12 total, 12 up, 11 in 2024-05-15T13:10:22.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:22 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:10:22.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:22 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2979544956' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:10:22.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:22 smithi027 ceph-mon[39795]: osdmap e553: 12 total, 12 up, 11 in 2024-05-15T13:10:22.781 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:22.817 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:10:22.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:22.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:10:22.922 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:10:22.962 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:22.990 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:10:22.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:22.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:10:23.100 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:10:23.139 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:23.166 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:10:23.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:23.167 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:10:23.268 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:10:23.332 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:23.399 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:10:23.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:23.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:10:23.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:23 smithi177 ceph-mon[44303]: pgmap v1474: 27 pgs: 27 active+clean; 5.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 833 op/s 2024-05-15T13:10:23.507 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:10:23.514 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:23 smithi027 ceph-mon[39795]: pgmap v1474: 27 pgs: 27 active+clean; 5.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 833 op/s 2024-05-15T13:10:23.546 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:23.578 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:10:23.581 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:23 smithi120 ceph-mon[43961]: pgmap v1474: 27 pgs: 27 active+clean; 5.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 833 op/s 2024-05-15T13:10:23.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:23.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:10:23.683 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:10:23.751 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:24.120 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:10:24.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:24.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:10:24.141 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-05-15T13:10:24.227 INFO:teuthology.orchestra.run.smithi027.stderr:no valid command found; 10 closest matches: 2024-05-15T13:10:24.227 INFO:teuthology.orchestra.run.smithi027.stderr:0 2024-05-15T13:10:24.227 INFO:teuthology.orchestra.run.smithi027.stderr:1 2024-05-15T13:10:24.227 INFO:teuthology.orchestra.run.smithi027.stderr:2 2024-05-15T13:10:24.227 INFO:teuthology.orchestra.run.smithi027.stderr:abort 2024-05-15T13:10:24.227 INFO:teuthology.orchestra.run.smithi027.stderr:assert 2024-05-15T13:10:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:bench [] [] [] [] 2024-05-15T13:10:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs debug_inject_read_zeros 2024-05-15T13:10:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs files list 2024-05-15T13:10:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs stats 2024-05-15T13:10:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:bluestore allocator dump block 2024-05-15T13:10:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: invalid command 2024-05-15T13:10:24.235 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:10:24.263 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:24.284 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:10:24.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:24.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:10:24.388 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:10:24.428 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:24.457 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:10:24.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:24.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:10:24.519 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:24 smithi120 ceph-mon[43961]: pgmap v1475: 27 pgs: 27 active+clean; 4.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 906 op/s 2024-05-15T13:10:24.564 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:10:24.603 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:24.646 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:10:24.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:24.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:10:24.755 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:10:24.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:24 smithi027 ceph-mon[39795]: pgmap v1475: 27 pgs: 27 active+clean; 4.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 906 op/s 2024-05-15T13:10:24.813 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:24.881 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:10:24.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:24.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:10:24.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:24 smithi177 ceph-mon[44303]: pgmap v1475: 27 pgs: 27 active+clean; 4.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 906 op/s 2024-05-15T13:10:24.938 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T13:10:24.938 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T13:10:24.938 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-05-15T13:10:24.952 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:10:24.986 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:10:25.024 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:25.054 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:10:25.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:25.060 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:10:25.161 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:10:25.224 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:25.279 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:10:25.304 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:10:25.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:25.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:10:25.412 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:10:25.440 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:25.466 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:10:25.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:25.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:10:25.582 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:10:25.616 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:25.641 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:10:25.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:25.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:10:25.745 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:10:25.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {auth=34,config=2,health=12,kv=52} crc {auth=2811709443,config=1059521453,health=179246142,kv=3441365707}) 2024-05-15T13:10:25.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {kv=9,logm=91} crc {kv=2828365697,logm=4234667530}) 2024-05-15T13:10:25.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3822726467}) 2024-05-15T13:10:25.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3933148739}) 2024-05-15T13:10:25.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1209363843}) 2024-05-15T13:10:25.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3584702550}) 2024-05-15T13:10:25.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2536323320}) 2024-05-15T13:10:25.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2714131814}) 2024-05-15T13:10:25.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3242969588}) 2024-05-15T13:10:25.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3438031892}) 2024-05-15T13:10:25.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1876350762}) 2024-05-15T13:10:25.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2579330522}) 2024-05-15T13:10:25.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3291359001}) 2024-05-15T13:10:25.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4111381431}) 2024-05-15T13:10:25.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1537842641}) 2024-05-15T13:10:25.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4236582740}) 2024-05-15T13:10:25.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2148582861}) 2024-05-15T13:10:25.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=636403271}) 2024-05-15T13:10:25.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1846778652}) 2024-05-15T13:10:25.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1737682994}) 2024-05-15T13:10:25.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2638735323}) 2024-05-15T13:10:25.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1612242689}) 2024-05-15T13:10:25.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3610818495}) 2024-05-15T13:10:25.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2525202000}) 2024-05-15T13:10:25.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3963360223}) 2024-05-15T13:10:25.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=546533944}) 2024-05-15T13:10:25.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=380014908}) 2024-05-15T13:10:25.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3786309596}) 2024-05-15T13:10:25.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=241465486}) 2024-05-15T13:10:25.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=612648757}) 2024-05-15T13:10:25.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=467787}) 2024-05-15T13:10:25.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2614841238}) 2024-05-15T13:10:25.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2136639456}) 2024-05-15T13:10:25.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1879324941}) 2024-05-15T13:10:25.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3416581068}) 2024-05-15T13:10:25.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=407544943}) 2024-05-15T13:10:25.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3885191761}) 2024-05-15T13:10:25.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3642040247}) 2024-05-15T13:10:25.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3292655211}) 2024-05-15T13:10:25.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=486947534}) 2024-05-15T13:10:25.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=225636447}) 2024-05-15T13:10:25.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1658622968}) 2024-05-15T13:10:25.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3894227190}) 2024-05-15T13:10:25.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3085356793}) 2024-05-15T13:10:25.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1825586763}) 2024-05-15T13:10:25.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=526274757}) 2024-05-15T13:10:25.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1620290169}) 2024-05-15T13:10:25.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=300070480}) 2024-05-15T13:10:25.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2016903167}) 2024-05-15T13:10:25.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3383423557}) 2024-05-15T13:10:25.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=42,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=23} crc {logm=2606990220,mdsmap=3984425268,mgr=76093979,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=2846738870,mon_config_key=1630071851}) 2024-05-15T13:10:25.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=6104833}) 2024-05-15T13:10:25.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3633622200}) 2024-05-15T13:10:25.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=52,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=30} crc {mon_config_key=3618531236,monmap=2435304556,osd_metadata=1212035326,osd_pg_creating=198460560,osdmap=2508217190}) 2024-05-15T13:10:25.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=89} crc {osdmap=4070890934}) 2024-05-15T13:10:25.785 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:25.812 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:10:25.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:25.816 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:10:25.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {auth=34,config=2,health=12,kv=52} crc {auth=2811709443,config=1059521453,health=179246142,kv=3441365707}) 2024-05-15T13:10:25.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {kv=9,logm=91} crc {kv=2828365697,logm=4234667530}) 2024-05-15T13:10:25.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3822726467}) 2024-05-15T13:10:25.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3933148739}) 2024-05-15T13:10:25.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1209363843}) 2024-05-15T13:10:25.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3584702550}) 2024-05-15T13:10:25.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2536323320}) 2024-05-15T13:10:25.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2714131814}) 2024-05-15T13:10:25.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3242969588}) 2024-05-15T13:10:25.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3438031892}) 2024-05-15T13:10:25.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1876350762}) 2024-05-15T13:10:25.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2579330522}) 2024-05-15T13:10:25.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3291359001}) 2024-05-15T13:10:25.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4111381431}) 2024-05-15T13:10:25.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1537842641}) 2024-05-15T13:10:25.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4236582740}) 2024-05-15T13:10:25.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2148582861}) 2024-05-15T13:10:25.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=636403271}) 2024-05-15T13:10:25.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1846778652}) 2024-05-15T13:10:25.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1737682994}) 2024-05-15T13:10:25.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2638735323}) 2024-05-15T13:10:25.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1612242689}) 2024-05-15T13:10:25.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3610818495}) 2024-05-15T13:10:25.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2525202000}) 2024-05-15T13:10:25.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3963360223}) 2024-05-15T13:10:25.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=546533944}) 2024-05-15T13:10:25.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=380014908}) 2024-05-15T13:10:25.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3786309596}) 2024-05-15T13:10:25.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=241465486}) 2024-05-15T13:10:25.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=612648757}) 2024-05-15T13:10:25.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=467787}) 2024-05-15T13:10:25.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2614841238}) 2024-05-15T13:10:25.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2136639456}) 2024-05-15T13:10:25.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1879324941}) 2024-05-15T13:10:25.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3416581068}) 2024-05-15T13:10:25.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=407544943}) 2024-05-15T13:10:25.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3885191761}) 2024-05-15T13:10:25.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3642040247}) 2024-05-15T13:10:25.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3292655211}) 2024-05-15T13:10:25.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=486947534}) 2024-05-15T13:10:25.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=225636447}) 2024-05-15T13:10:25.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1658622968}) 2024-05-15T13:10:25.820 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3894227190}) 2024-05-15T13:10:25.820 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3085356793}) 2024-05-15T13:10:25.820 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1825586763}) 2024-05-15T13:10:25.820 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=526274757}) 2024-05-15T13:10:25.820 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1620290169}) 2024-05-15T13:10:25.820 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=300070480}) 2024-05-15T13:10:25.820 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2016903167}) 2024-05-15T13:10:25.820 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3383423557}) 2024-05-15T13:10:25.820 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=42,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=23} crc {logm=2606990220,mdsmap=3984425268,mgr=76093979,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=2846738870,mon_config_key=1630071851}) 2024-05-15T13:10:25.820 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=6104833}) 2024-05-15T13:10:25.820 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3633622200}) 2024-05-15T13:10:25.820 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=52,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=30} crc {mon_config_key=3618531236,monmap=2435304556,osd_metadata=1212035326,osd_pg_creating=198460560,osdmap=2508217190}) 2024-05-15T13:10:25.820 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=89} crc {osdmap=4070890934}) 2024-05-15T13:10:25.917 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:10:25.963 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {auth=34,config=2,health=12,kv=52} crc {auth=2811709443,config=1059521453,health=179246142,kv=3441365707}) 2024-05-15T13:10:25.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {kv=9,logm=91} crc {kv=2828365697,logm=4234667530}) 2024-05-15T13:10:25.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3822726467}) 2024-05-15T13:10:25.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3933148739}) 2024-05-15T13:10:25.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1209363843}) 2024-05-15T13:10:25.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3584702550}) 2024-05-15T13:10:25.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2536323320}) 2024-05-15T13:10:25.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2714131814}) 2024-05-15T13:10:25.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3242969588}) 2024-05-15T13:10:25.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3438031892}) 2024-05-15T13:10:25.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1876350762}) 2024-05-15T13:10:25.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2579330522}) 2024-05-15T13:10:25.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3291359001}) 2024-05-15T13:10:25.964 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4111381431}) 2024-05-15T13:10:25.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1537842641}) 2024-05-15T13:10:25.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4236582740}) 2024-05-15T13:10:25.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2148582861}) 2024-05-15T13:10:25.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=636403271}) 2024-05-15T13:10:25.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1846778652}) 2024-05-15T13:10:25.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1737682994}) 2024-05-15T13:10:25.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2638735323}) 2024-05-15T13:10:25.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1612242689}) 2024-05-15T13:10:25.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3610818495}) 2024-05-15T13:10:25.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2525202000}) 2024-05-15T13:10:25.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3963360223}) 2024-05-15T13:10:25.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=546533944}) 2024-05-15T13:10:25.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=380014908}) 2024-05-15T13:10:25.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3786309596}) 2024-05-15T13:10:25.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=241465486}) 2024-05-15T13:10:25.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=612648757}) 2024-05-15T13:10:25.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=467787}) 2024-05-15T13:10:25.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2614841238}) 2024-05-15T13:10:25.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2136639456}) 2024-05-15T13:10:25.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1879324941}) 2024-05-15T13:10:25.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3416581068}) 2024-05-15T13:10:25.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=407544943}) 2024-05-15T13:10:25.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3885191761}) 2024-05-15T13:10:25.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3642040247}) 2024-05-15T13:10:25.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3292655211}) 2024-05-15T13:10:25.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=486947534}) 2024-05-15T13:10:25.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=225636447}) 2024-05-15T13:10:25.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1658622968}) 2024-05-15T13:10:25.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3894227190}) 2024-05-15T13:10:25.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3085356793}) 2024-05-15T13:10:25.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1825586763}) 2024-05-15T13:10:25.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=526274757}) 2024-05-15T13:10:25.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1620290169}) 2024-05-15T13:10:25.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=300070480}) 2024-05-15T13:10:25.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2016903167}) 2024-05-15T13:10:25.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3383423557}) 2024-05-15T13:10:25.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=42,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=23} crc {logm=2606990220,mdsmap=3984425268,mgr=76093979,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=2846738870,mon_config_key=1630071851}) 2024-05-15T13:10:25.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=6104833}) 2024-05-15T13:10:25.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3633622200}) 2024-05-15T13:10:25.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=52,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=30} crc {mon_config_key=3618531236,monmap=2435304556,osd_metadata=1212035326,osd_pg_creating=198460560,osdmap=2508217190}) 2024-05-15T13:10:25.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=89} crc {osdmap=4070890934}) 2024-05-15T13:10:25.971 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:26.059 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:10:26.061 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:10:26.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:26.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:10:26.143 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:10:26.170 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:10:26.204 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:26.229 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:10:26.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:26.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:10:26.334 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:10:26.393 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:26.429 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:10:26.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:26.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:10:26.537 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:10:26.583 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:26.625 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:10:26.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:26.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:10:26.730 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:10:26.766 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:26.792 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:10:26.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:26.799 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:10:26.900 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:10:26.927 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:26 smithi120 ceph-mon[43961]: pgmap v1476: 27 pgs: 27 active+clean; 4.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 867 op/s 2024-05-15T13:10:26.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:26 smithi177 ceph-mon[44303]: pgmap v1476: 27 pgs: 27 active+clean; 4.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 867 op/s 2024-05-15T13:10:26.934 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:26.955 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:10:26.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:26.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:10:27.015 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:26 smithi027 ceph-mon[39795]: pgmap v1476: 27 pgs: 27 active+clean; 4.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 867 op/s 2024-05-15T13:10:27.063 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:10:27.135 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:29.386 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:10:29.386 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:10:29.386 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:10:29.386 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:10:29.387 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:10:29.387 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:10:29.387 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:10:29.387 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:10:29.387 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:10:29.387 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:10:29.387 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:10:29.387 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:10:29.387 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:10:29.387 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:10:29.387 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:10:29.388 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:10:29.388 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:10:29.388 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:10:29.388 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:10:29.388 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:10:29.388 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:10:29.388 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:10:29.388 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:10:29.388 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:10:29.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:29 smithi027 ceph-mon[39795]: pgmap v1477: 27 pgs: 27 active+clean; 4.7 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 773 op/s 2024-05-15T13:10:29.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:29 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3660760693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:10:29.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:29 smithi120 ceph-mon[43961]: pgmap v1477: 27 pgs: 27 active+clean; 4.7 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 773 op/s 2024-05-15T13:10:29.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:29 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3660760693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:10:29.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:29 smithi177 ceph-mon[44303]: pgmap v1477: 27 pgs: 27 active+clean; 4.7 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 773 op/s 2024-05-15T13:10:29.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:29 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3660760693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:10:29.817 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:10:29.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:29.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:10:29.927 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:10:29.956 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:29.982 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:10:29.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:29.987 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:10:30.088 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:10:30.157 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:31.041 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:10:31.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:31.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:10:31.152 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:10:31.192 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:31.214 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:10:31.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:31.219 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:10:31.252 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:31 smithi027 ceph-mon[39795]: pgmap v1478: 27 pgs: 27 active+clean; 4.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 815 op/s 2024-05-15T13:10:31.320 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:10:31.383 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:31.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:31 smithi120 ceph-mon[43961]: pgmap v1478: 27 pgs: 27 active+clean; 4.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 815 op/s 2024-05-15T13:10:31.541 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:10:31.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:31.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:10:31.648 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:10:31.678 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:31.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:31 smithi177 ceph-mon[44303]: pgmap v1478: 27 pgs: 27 active+clean; 4.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 815 op/s 2024-05-15T13:10:31.716 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:10:31.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:31.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:10:31.820 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:10:31.850 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:31.871 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:10:31.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:31.874 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:10:31.975 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:10:32.038 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:10:32.051 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:32.149 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:10:32.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:32.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:10:32.258 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:10:32.318 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:32.435 INFO:teuthology.orchestra.run.smithi027.stderr:no valid command found; 10 closest matches: 2024-05-15T13:10:32.436 INFO:teuthology.orchestra.run.smithi027.stderr:0 2024-05-15T13:10:32.436 INFO:teuthology.orchestra.run.smithi027.stderr:1 2024-05-15T13:10:32.436 INFO:teuthology.orchestra.run.smithi027.stderr:2 2024-05-15T13:10:32.436 INFO:teuthology.orchestra.run.smithi027.stderr:abort 2024-05-15T13:10:32.436 INFO:teuthology.orchestra.run.smithi027.stderr:assert 2024-05-15T13:10:32.436 INFO:teuthology.orchestra.run.smithi027.stderr:bench [] [] [] [] 2024-05-15T13:10:32.436 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs debug_inject_read_zeros 2024-05-15T13:10:32.436 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs files list 2024-05-15T13:10:32.436 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs stats 2024-05-15T13:10:32.436 INFO:teuthology.orchestra.run.smithi027.stderr:bluestore allocator dump block 2024-05-15T13:10:32.436 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: invalid command 2024-05-15T13:10:33.320 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:33 smithi027 ceph-mon[39795]: pgmap v1479: 27 pgs: 27 active+clean; 4.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 773 op/s 2024-05-15T13:10:33.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:33 smithi177 ceph-mon[44303]: pgmap v1479: 27 pgs: 27 active+clean; 4.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 773 op/s 2024-05-15T13:10:33.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:33 smithi120 ceph-mon[43961]: pgmap v1479: 27 pgs: 27 active+clean; 4.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 773 op/s 2024-05-15T13:10:33.816 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:10:33.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:33.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:10:33.927 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:10:33.985 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:34.012 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:10:34.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:34.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:10:34.069 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:10:34.117 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:10:34.157 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:34.180 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:10:34.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:34.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:10:34.286 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:10:34.326 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:34.363 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:10:34.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:34.371 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:10:34.472 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:10:34.511 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:34.511 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:34 smithi027 ceph-mon[39795]: pgmap v1480: 27 pgs: 27 active+clean; 4.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 832 op/s 2024-05-15T13:10:34.544 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:10:34.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:34.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:10:34.648 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:10:34.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:34 smithi177 ceph-mon[44303]: pgmap v1480: 27 pgs: 27 active+clean; 4.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 832 op/s 2024-05-15T13:10:34.688 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:34.718 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:10:34.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:34.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:10:34.780 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:34 smithi120 ceph-mon[43961]: pgmap v1480: 27 pgs: 27 active+clean; 4.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 832 op/s 2024-05-15T13:10:34.823 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:10:34.871 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:34.899 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:10:34.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:34.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:10:34.946 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:10:34.946 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T13:10:34.946 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T13:10:34.947 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:10:34.947 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.947 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:195093 4.f e385/343 4:f1629b82:::benchmark_data_smithi177_99756_object75121:head v 385'11904, mlcod=333'5297)", 2024-05-15T13:10:34.947 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:39.173239+0000", 2024-05-15T13:10:34.947 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 534.51633646000005, 2024-05-15T13:10:34.947 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.156115744, 2024-05-15T13:10:34.947 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:34.947 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:34.947 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:34.947 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.947 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:34.948 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.173239+0000", 2024-05-15T13:10:34.948 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:34.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.948 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.948 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:34.948 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.173239+0000", 2024-05-15T13:10:34.948 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:34.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.948 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.948 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:34.948 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.173240+0000", 2024-05-15T13:10:34.948 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5700000000000002e-07 2024-05-15T13:10:34.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.949 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.949 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:34.949 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.173246+0000", 2024-05-15T13:10:34.949 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.592e-06 2024-05-15T13:10:34.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.949 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.949 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:34.949 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.173253+0000", 2024-05-15T13:10:34.949 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0419999999999997e-06 2024-05-15T13:10:34.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.949 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.949 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:34.949 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.173258+0000", 2024-05-15T13:10:34.950 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3870000000000003e-06 2024-05-15T13:10:34.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.950 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.950 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:34.950 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.173307+0000", 2024-05-15T13:10:34.950 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9376000000000003e-05 2024-05-15T13:10:34.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.950 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.950 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:34.950 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.173326+0000", 2024-05-15T13:10:34.950 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9045000000000001e-05 2024-05-15T13:10:34.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.950 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.950 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:34.951 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329308+0000", 2024-05-15T13:10:34.951 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.15598142600000001 2024-05-15T13:10:34.951 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.951 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.951 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:34.951 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329355+0000", 2024-05-15T13:10:34.951 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7219000000000003e-05 2024-05-15T13:10:34.951 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:34.951 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:34.951 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:34.951 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.951 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.951 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:195062 4.9 e385/343 4:9dfccf82:::benchmark_data_smithi177_99756_object75090:head v 385'12098, mlcod=330'4626)", 2024-05-15T13:10:34.951 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:39.178226+0000", 2024-05-15T13:10:34.952 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 534.51135002900003, 2024-05-15T13:10:34.952 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16531588999999999, 2024-05-15T13:10:34.952 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:34.952 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:34.952 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:34.952 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.952 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:34.952 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.178226+0000", 2024-05-15T13:10:34.952 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:34.952 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.952 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.952 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:34.952 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.178226+0000", 2024-05-15T13:10:34.953 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:34.953 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.953 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.953 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:34.953 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.178227+0000", 2024-05-15T13:10:34.953 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7699999999999992e-07 2024-05-15T13:10:34.953 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.953 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.953 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:34.953 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.178236+0000", 2024-05-15T13:10:34.953 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0429999999999996e-06 2024-05-15T13:10:34.953 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.953 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.953 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:34.954 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.178248+0000", 2024-05-15T13:10:34.954 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2438e-05 2024-05-15T13:10:34.954 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.954 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.954 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:34.954 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.178257+0000", 2024-05-15T13:10:34.954 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0550000000000005e-06 2024-05-15T13:10:34.954 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.954 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.954 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:34.954 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.178314+0000", 2024-05-15T13:10:34.954 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6331e-05 2024-05-15T13:10:34.954 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.954 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.955 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:34.955 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.178342+0000", 2024-05-15T13:10:34.955 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8101000000000001e-05 2024-05-15T13:10:34.955 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.955 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.955 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:34.955 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.343498+0000", 2024-05-15T13:10:34.955 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16515676000000001 2024-05-15T13:10:34.955 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.955 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.955 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:34.955 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.343542+0000", 2024-05-15T13:10:34.955 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3185000000000001e-05 2024-05-15T13:10:34.956 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:34.956 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:34.956 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:34.956 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.956 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.956 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25180.0:195097 4.0 4:01df6f19:::benchmark_data_smithi177_99756_object75125:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-05-15T13:10:34.956 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:39.179776+0000", 2024-05-15T13:10:34.956 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 534.50980004799999, 2024-05-15T13:10:34.956 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16930087199999999, 2024-05-15T13:10:34.956 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:34.957 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:34.957 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:34.957 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25180", 2024-05-15T13:10:34.957 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:10:34.957 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 195097 2024-05-15T13:10:34.957 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.957 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:34.957 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.957 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:34.957 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.179776+0000", 2024-05-15T13:10:34.957 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:34.957 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.957 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.958 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:34.958 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.179776+0000", 2024-05-15T13:10:34.958 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:34.958 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.958 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.958 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:34.958 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.179777+0000", 2024-05-15T13:10:34.958 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5400000000000001e-06 2024-05-15T13:10:34.958 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.958 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.958 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:34.958 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.179783+0000", 2024-05-15T13:10:34.958 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.254e-06 2024-05-15T13:10:34.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.959 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.959 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:34.959 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329321+0000", 2024-05-15T13:10:34.959 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.14953818699999999 2024-05-15T13:10:34.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.959 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.959 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:34.959 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329331+0000", 2024-05-15T13:10:34.959 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0556e-05 2024-05-15T13:10:34.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.959 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.959 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:34.959 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329551+0000", 2024-05-15T13:10:34.959 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000219385 2024-05-15T13:10:34.960 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.960 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.960 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:34.960 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329679+0000", 2024-05-15T13:10:34.960 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000127961 2024-05-15T13:10:34.960 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.960 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.960 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,8", 2024-05-15T13:10:34.960 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329796+0000", 2024-05-15T13:10:34.960 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000117175 2024-05-15T13:10:34.960 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.960 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.960 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:34.960 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.344048+0000", 2024-05-15T13:10:34.961 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014252453999999999 2024-05-15T13:10:34.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.961 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.961 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:34.961 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.344176+0000", 2024-05-15T13:10:34.961 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012799999999999999 2024-05-15T13:10:34.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.961 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.961 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:34.961 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.348983+0000", 2024-05-15T13:10:34.961 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0048068659999999999 2024-05-15T13:10:34.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.961 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.961 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:34.962 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.349033+0000", 2024-05-15T13:10:34.962 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9848000000000001e-05 2024-05-15T13:10:34.962 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.962 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.962 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:34.962 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.349077+0000", 2024-05-15T13:10:34.962 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3646000000000002e-05 2024-05-15T13:10:34.962 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:34.962 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:34.962 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:34.962 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.962 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.962 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.25180.0:195098 4.f e385/343 4:f78a4cdc:::benchmark_data_smithi177_99756_object75126:head v 385'11905, mlcod=333'5297)", 2024-05-15T13:10:34.963 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:01:39.180038+0000", 2024-05-15T13:10:34.963 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 534.50953801399999, 2024-05-15T13:10:34.963 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.158549575, 2024-05-15T13:10:34.963 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:34.963 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:34.963 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:34.963 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.963 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:34.963 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.180038+0000", 2024-05-15T13:10:34.963 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:34.963 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.963 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.963 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:34.963 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.180038+0000", 2024-05-15T13:10:34.963 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:34.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.964 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.964 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:34.964 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.180038+0000", 2024-05-15T13:10:34.964 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3099999999999997e-07 2024-05-15T13:10:34.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.964 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.964 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:34.964 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.180044+0000", 2024-05-15T13:10:34.964 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7130000000000002e-06 2024-05-15T13:10:34.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.964 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.964 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:34.964 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329299+0000", 2024-05-15T13:10:34.964 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.14925477400000001 2024-05-15T13:10:34.965 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.965 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.965 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:34.965 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329313+0000", 2024-05-15T13:10:34.965 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3796e-05 2024-05-15T13:10:34.965 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.965 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.965 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:34.965 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329383+0000", 2024-05-15T13:10:34.965 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0069999999999998e-05 2024-05-15T13:10:34.965 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.965 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.965 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:34.965 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.329410+0000", 2024-05-15T13:10:34.965 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.694e-05 2024-05-15T13:10:34.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.966 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.966 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:34.966 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.338543+0000", 2024-05-15T13:10:34.966 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0091335180000000002 2024-05-15T13:10:34.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.966 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.966 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:34.966 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:01:39.338587+0000", 2024-05-15T13:10:34.966 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4132999999999999e-05 2024-05-15T13:10:34.966 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:34.966 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:34.966 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:34.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.967 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.967 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:68992 6.b 6:dd227e64:::benchmark_data_smithi177_122278_object68991:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e528)", 2024-05-15T13:10:34.967 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:08:28.936175+0000", 2024-05-15T13:10:34.967 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 124.75340031499999, 2024-05-15T13:10:34.967 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.15873723100000001, 2024-05-15T13:10:34.967 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:34.967 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:34.967 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:34.967 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:10:34.967 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:10:34.967 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 68992 2024-05-15T13:10:34.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.967 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:34.967 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.968 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:34.968 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.936175+0000", 2024-05-15T13:10:34.968 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:34.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.968 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:34.968 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.936175+0000", 2024-05-15T13:10:34.968 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:34.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.968 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:34.968 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.936177+0000", 2024-05-15T13:10:34.968 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.339e-06 2024-05-15T13:10:34.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.969 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:34.969 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.936208+0000", 2024-05-15T13:10:34.969 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1188e-05 2024-05-15T13:10:34.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.969 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.969 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:34.969 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.936218+0000", 2024-05-15T13:10:34.969 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0412e-05 2024-05-15T13:10:34.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.969 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.969 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:34.969 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.936228+0000", 2024-05-15T13:10:34.969 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1910000000000006e-06 2024-05-15T13:10:34.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.970 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.970 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:34.970 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.936321+0000", 2024-05-15T13:10:34.970 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3057e-05 2024-05-15T13:10:34.970 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.970 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.970 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:34.970 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.936505+0000", 2024-05-15T13:10:34.970 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00018385499999999999 2024-05-15T13:10:34.970 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.970 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.970 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:10:34.970 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.936637+0000", 2024-05-15T13:10:34.970 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013250999999999999 2024-05-15T13:10:34.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.971 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.971 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:34.971 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.941257+0000", 2024-05-15T13:10:34.971 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046198569999999998 2024-05-15T13:10:34.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.971 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.971 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:34.971 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.946303+0000", 2024-05-15T13:10:34.971 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.005046076 2024-05-15T13:10:34.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.971 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.971 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:34.971 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:29.094826+0000", 2024-05-15T13:10:34.971 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.14852270300000001 2024-05-15T13:10:34.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.972 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.972 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:34.972 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:29.094864+0000", 2024-05-15T13:10:34.972 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8609000000000003e-05 2024-05-15T13:10:34.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.972 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.972 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:34.972 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:29.094913+0000", 2024-05-15T13:10:34.972 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8433999999999997e-05 2024-05-15T13:10:34.972 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:34.972 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:34.972 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:34.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.972 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.973 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:68999 6.b 6:de47c2d8:::benchmark_data_smithi177_122278_object68998:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e528)", 2024-05-15T13:10:34.973 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:08:28.941297+0000", 2024-05-15T13:10:34.973 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 124.748278996, 2024-05-15T13:10:34.973 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.15370994700000001, 2024-05-15T13:10:34.973 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:34.973 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:34.973 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:34.973 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:10:34.973 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:10:34.973 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 68999 2024-05-15T13:10:34.973 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.973 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:34.973 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.973 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:34.973 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.941297+0000", 2024-05-15T13:10:34.974 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:34.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.974 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.974 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:34.974 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.941297+0000", 2024-05-15T13:10:34.974 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:34.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.974 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.974 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:34.974 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.941297+0000", 2024-05-15T13:10:34.974 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4900000000000001e-07 2024-05-15T13:10:34.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.974 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.975 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:34.975 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.941325+0000", 2024-05-15T13:10:34.975 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8039e-05 2024-05-15T13:10:34.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.975 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.975 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:34.975 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.941333+0000", 2024-05-15T13:10:34.975 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7300000000000005e-06 2024-05-15T13:10:34.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.975 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.975 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:34.975 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.941339+0000", 2024-05-15T13:10:34.975 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6049999999999997e-06 2024-05-15T13:10:34.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.976 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.976 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:34.976 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.941444+0000", 2024-05-15T13:10:34.976 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000105535 2024-05-15T13:10:34.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.976 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.976 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:34.976 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.941571+0000", 2024-05-15T13:10:34.976 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000127007 2024-05-15T13:10:34.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.976 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.976 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:10:34.976 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.941675+0000", 2024-05-15T13:10:34.976 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010398799999999999 2024-05-15T13:10:34.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.977 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.977 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:34.977 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.946310+0000", 2024-05-15T13:10:34.977 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046345309999999999 2024-05-15T13:10:34.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.977 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.977 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:34.977 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.946604+0000", 2024-05-15T13:10:34.977 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000294648 2024-05-15T13:10:34.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.977 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.977 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:34.977 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:29.094940+0000", 2024-05-15T13:10:34.977 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.14833553799999999 2024-05-15T13:10:34.978 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.978 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.978 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:34.978 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:29.094972+0000", 2024-05-15T13:10:34.978 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2607000000000001e-05 2024-05-15T13:10:34.978 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.978 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.978 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:34.978 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:29.095007+0000", 2024-05-15T13:10:34.978 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4270000000000002e-05 2024-05-15T13:10:34.978 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:34.978 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:34.978 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:34.978 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.979 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.979 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:69005 6.b 6:d240df61:::benchmark_data_smithi177_122278_object69004:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e528)", 2024-05-15T13:10:34.979 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:08:28.943682+0000", 2024-05-15T13:10:34.979 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 124.745893749, 2024-05-15T13:10:34.979 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.15661480799999999, 2024-05-15T13:10:34.979 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:34.979 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:34.979 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:34.979 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:10:34.979 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:10:34.979 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 69005 2024-05-15T13:10:34.979 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.979 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:34.979 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.980 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:34.980 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.943682+0000", 2024-05-15T13:10:34.980 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:34.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.980 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.980 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:34.980 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.943682+0000", 2024-05-15T13:10:34.980 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:34.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.980 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.980 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:34.980 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.943682+0000", 2024-05-15T13:10:34.980 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0400000000000002e-07 2024-05-15T13:10:34.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.981 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.981 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:34.981 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.943715+0000", 2024-05-15T13:10:34.981 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2539000000000001e-05 2024-05-15T13:10:34.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.981 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.981 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:34.981 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.943724+0000", 2024-05-15T13:10:34.981 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8859999999999993e-06 2024-05-15T13:10:34.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.981 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.981 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:34.981 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.943728+0000", 2024-05-15T13:10:34.981 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.578e-06 2024-05-15T13:10:34.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.982 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.982 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:34.982 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.943796+0000", 2024-05-15T13:10:34.982 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7163999999999996e-05 2024-05-15T13:10:34.982 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.982 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.982 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:34.982 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.943975+0000", 2024-05-15T13:10:34.982 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017943899999999999 2024-05-15T13:10:34.982 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.982 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.982 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:10:34.982 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.944139+0000", 2024-05-15T13:10:34.982 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000163654 2024-05-15T13:10:34.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.983 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.983 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:34.983 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.948673+0000", 2024-05-15T13:10:34.983 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0045340129999999999 2024-05-15T13:10:34.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.983 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.983 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:34.983 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.949251+0000", 2024-05-15T13:10:34.983 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00057852700000000001 2024-05-15T13:10:34.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.983 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.983 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:34.983 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:29.100199+0000", 2024-05-15T13:10:34.984 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.150948214 2024-05-15T13:10:34.984 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.984 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.984 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:34.984 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:29.100255+0000", 2024-05-15T13:10:34.984 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5612999999999998e-05 2024-05-15T13:10:34.984 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.984 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.984 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:34.984 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:29.100297+0000", 2024-05-15T13:10:34.984 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1777000000000003e-05 2024-05-15T13:10:34.984 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:34.984 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:34.984 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:34.984 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.985 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.985 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:69015 6.b 6:dd233641:::benchmark_data_smithi177_122278_object69014:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e528)", 2024-05-15T13:10:34.985 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:08:28.949365+0000", 2024-05-15T13:10:34.985 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 124.74021099799999, 2024-05-15T13:10:34.985 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.15109656899999999, 2024-05-15T13:10:34.985 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:34.985 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:34.985 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:34.985 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:10:34.985 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:10:34.985 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 69015 2024-05-15T13:10:34.985 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.985 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:34.985 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.985 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:34.986 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.949365+0000", 2024-05-15T13:10:34.986 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:34.986 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.986 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.986 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:34.986 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.949365+0000", 2024-05-15T13:10:34.986 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:34.986 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.986 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.986 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:34.986 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.949369+0000", 2024-05-15T13:10:34.986 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9559999999999999e-06 2024-05-15T13:10:34.986 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.986 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.986 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:34.987 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.949428+0000", 2024-05-15T13:10:34.987 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9635000000000001e-05 2024-05-15T13:10:34.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.987 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.987 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:34.987 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.949441+0000", 2024-05-15T13:10:34.987 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.234e-05 2024-05-15T13:10:34.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.987 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.987 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:34.987 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.949449+0000", 2024-05-15T13:10:34.987 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9969999999999998e-06 2024-05-15T13:10:34.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.987 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.987 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:34.987 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.949497+0000", 2024-05-15T13:10:34.988 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7994000000000002e-05 2024-05-15T13:10:34.988 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.988 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.988 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:34.988 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.949650+0000", 2024-05-15T13:10:34.988 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015303100000000001 2024-05-15T13:10:34.988 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.988 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.988 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:10:34.988 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.949808+0000", 2024-05-15T13:10:34.988 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015869699999999999 2024-05-15T13:10:34.988 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.988 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.988 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:34.988 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.952004+0000", 2024-05-15T13:10:34.989 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0021950849999999998 2024-05-15T13:10:34.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.989 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.989 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:34.989 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:28.956385+0000", 2024-05-15T13:10:34.989 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.004381284 2024-05-15T13:10:34.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.989 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.989 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:34.989 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:29.100378+0000", 2024-05-15T13:10:34.989 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.14399356199999999 2024-05-15T13:10:34.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.989 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.989 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:34.990 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:29.100427+0000", 2024-05-15T13:10:34.990 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8751000000000001e-05 2024-05-15T13:10:34.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.990 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.990 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:34.990 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:29.100461+0000", 2024-05-15T13:10:34.990 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4236999999999998e-05 2024-05-15T13:10:34.990 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:34.990 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:34.990 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:34.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.990 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.991 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:88229 6.b 6:df8c22be:::benchmark_data_smithi177_122278_object88228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e534)", 2024-05-15T13:10:34.991 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:08:43.502833+0000", 2024-05-15T13:10:34.991 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 110.186742405, 2024-05-15T13:10:34.991 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.18556212999999999, 2024-05-15T13:10:34.991 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:34.991 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:34.991 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:34.991 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:10:34.991 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:10:34.991 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 88229 2024-05-15T13:10:34.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.991 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:34.991 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.991 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:34.991 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.502833+0000", 2024-05-15T13:10:34.992 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:34.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.992 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.992 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:34.992 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.502833+0000", 2024-05-15T13:10:34.992 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:34.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.992 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.992 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:34.992 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.502834+0000", 2024-05-15T13:10:34.992 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.088e-06 2024-05-15T13:10:34.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.992 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.993 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:34.993 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.502955+0000", 2024-05-15T13:10:34.993 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012085 2024-05-15T13:10:34.993 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.993 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.993 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:34.993 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.502962+0000", 2024-05-15T13:10:34.993 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5370000000000003e-06 2024-05-15T13:10:34.993 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.993 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.993 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:34.993 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.502966+0000", 2024-05-15T13:10:34.993 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5850000000000001e-06 2024-05-15T13:10:34.993 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.993 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.994 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:34.994 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.503072+0000", 2024-05-15T13:10:34.994 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010581699999999999 2024-05-15T13:10:34.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.994 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.994 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:34.994 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.503195+0000", 2024-05-15T13:10:34.994 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000123149 2024-05-15T13:10:34.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.994 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.994 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:10:34.994 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.503359+0000", 2024-05-15T13:10:34.995 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016405400000000001 2024-05-15T13:10:34.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.995 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.995 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:34.995 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.506139+0000", 2024-05-15T13:10:34.995 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002779152 2024-05-15T13:10:34.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.995 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.995 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:34.995 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.508583+0000", 2024-05-15T13:10:34.995 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0024448220000000001 2024-05-15T13:10:34.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.995 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.995 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:34.995 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.688269+0000", 2024-05-15T13:10:34.996 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.17968595300000001 2024-05-15T13:10:34.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.996 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.996 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:34.996 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.688341+0000", 2024-05-15T13:10:34.996 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1548e-05 2024-05-15T13:10:34.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.996 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.996 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:34.996 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.688396+0000", 2024-05-15T13:10:34.996 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4574999999999998e-05 2024-05-15T13:10:34.996 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:34.996 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:34.996 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:34.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.997 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.997 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:88260 6.b 6:d42636cb:::benchmark_data_smithi177_122278_object88259:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e534)", 2024-05-15T13:10:34.997 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:08:43.525097+0000", 2024-05-15T13:10:34.997 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 110.164478346, 2024-05-15T13:10:34.997 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.163398549, 2024-05-15T13:10:34.997 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:34.997 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:34.997 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:34.997 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:10:34.997 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:10:34.997 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 88260 2024-05-15T13:10:34.997 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.997 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:34.997 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.997 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:34.998 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.525097+0000", 2024-05-15T13:10:34.998 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:34.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.998 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.998 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:34.998 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.525097+0000", 2024-05-15T13:10:34.998 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:34.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.998 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.998 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:34.998 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.525099+0000", 2024-05-15T13:10:34.998 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1620000000000001e-06 2024-05-15T13:10:34.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.999 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.999 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:34.999 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.525152+0000", 2024-05-15T13:10:34.999 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3559e-05 2024-05-15T13:10:34.999 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.999 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.999 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:34.999 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.525176+0000", 2024-05-15T13:10:34.999 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3657000000000001e-05 2024-05-15T13:10:34.999 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.999 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:34.999 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:34.999 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.525184+0000", 2024-05-15T13:10:34.999 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3420000000000008e-06 2024-05-15T13:10:34.999 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:34.999 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.000 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:35.000 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.525314+0000", 2024-05-15T13:10:35.000 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000129562 2024-05-15T13:10:35.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.000 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.000 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:35.000 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.525437+0000", 2024-05-15T13:10:35.000 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012304199999999999 2024-05-15T13:10:35.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.000 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.000 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:10:35.000 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.525590+0000", 2024-05-15T13:10:35.000 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015316200000000001 2024-05-15T13:10:35.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.001 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:35.001 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.535591+0000", 2024-05-15T13:10:35.001 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010000902000000001 2024-05-15T13:10:35.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.001 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:35.001 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.536133+0000", 2024-05-15T13:10:35.001 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00054196099999999998 2024-05-15T13:10:35.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.001 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:35.001 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.688430+0000", 2024-05-15T13:10:35.001 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.15229759800000001 2024-05-15T13:10:35.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.002 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.002 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:35.002 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.688466+0000", 2024-05-15T13:10:35.002 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5636999999999998e-05 2024-05-15T13:10:35.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.002 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.002 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:35.002 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.688496+0000", 2024-05-15T13:10:35.002 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9964999999999999e-05 2024-05-15T13:10:35.002 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:35.002 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:35.002 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:35.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.003 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.003 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:88262 6.b 6:d6ec44ff:::benchmark_data_smithi177_122278_object88261:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e534)", 2024-05-15T13:10:35.003 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:08:43.525210+0000", 2024-05-15T13:10:35.003 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 110.16436614, 2024-05-15T13:10:35.003 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16344135600000001, 2024-05-15T13:10:35.003 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:35.003 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:35.003 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:35.003 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:10:35.003 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:10:35.003 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 88262 2024-05-15T13:10:35.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.003 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:35.004 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.004 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:35.004 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.525210+0000", 2024-05-15T13:10:35.004 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:35.004 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.004 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.004 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:35.004 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.525210+0000", 2024-05-15T13:10:35.004 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:35.004 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.004 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.004 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:35.004 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.525210+0000", 2024-05-15T13:10:35.004 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2900000000000003e-07 2024-05-15T13:10:35.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.005 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.005 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:35.005 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.525242+0000", 2024-05-15T13:10:35.005 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1891999999999999e-05 2024-05-15T13:10:35.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.005 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.005 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:35.005 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.525253+0000", 2024-05-15T13:10:35.005 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0869e-05 2024-05-15T13:10:35.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.005 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.006 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:35.006 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.525259+0000", 2024-05-15T13:10:35.006 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.348e-06 2024-05-15T13:10:35.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.006 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.006 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:35.006 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.528646+0000", 2024-05-15T13:10:35.006 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033870269999999999 2024-05-15T13:10:35.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.006 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.006 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:35.006 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.528815+0000", 2024-05-15T13:10:35.006 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016902200000000001 2024-05-15T13:10:35.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.007 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.007 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:10:35.007 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.528961+0000", 2024-05-15T13:10:35.007 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014510700000000001 2024-05-15T13:10:35.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.007 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.007 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:35.007 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.536174+0000", 2024-05-15T13:10:35.007 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0072134499999999997 2024-05-15T13:10:35.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.007 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.007 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:35.007 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.536305+0000", 2024-05-15T13:10:35.008 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013077100000000001 2024-05-15T13:10:35.008 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.008 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.008 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:35.008 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.688548+0000", 2024-05-15T13:10:35.008 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.15224372999999999 2024-05-15T13:10:35.008 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.008 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.008 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:35.008 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.688609+0000", 2024-05-15T13:10:35.008 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0353999999999998e-05 2024-05-15T13:10:35.008 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.008 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.009 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:35.009 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.688651+0000", 2024-05-15T13:10:35.009 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2157000000000003e-05 2024-05-15T13:10:35.009 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:35.009 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:35.009 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:35.009 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.009 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.009 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:88292 6.b 6:d37c0e17:::benchmark_data_smithi177_122278_object88291:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e534)", 2024-05-15T13:10:35.009 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:08:43.550055+0000", 2024-05-15T13:10:35.009 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 110.139521137, 2024-05-15T13:10:35.009 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.13873607800000001, 2024-05-15T13:10:35.009 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:35.009 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:35.010 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:35.010 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:10:35.010 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:10:35.010 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 88292 2024-05-15T13:10:35.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.010 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:35.010 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.010 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:35.010 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.550055+0000", 2024-05-15T13:10:35.010 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:35.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.010 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.011 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:35.011 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.550055+0000", 2024-05-15T13:10:35.011 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:35.011 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.011 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.011 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:35.011 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.550056+0000", 2024-05-15T13:10:35.011 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.029e-06 2024-05-15T13:10:35.011 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.011 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.011 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:35.011 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.550118+0000", 2024-05-15T13:10:35.011 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1892999999999998e-05 2024-05-15T13:10:35.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.012 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.012 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:35.012 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.550129+0000", 2024-05-15T13:10:35.012 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1576000000000001e-05 2024-05-15T13:10:35.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.012 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.012 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:35.012 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.550137+0000", 2024-05-15T13:10:35.012 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9969999999999998e-06 2024-05-15T13:10:35.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.012 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.012 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:35.012 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.550215+0000", 2024-05-15T13:10:35.012 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7675000000000004e-05 2024-05-15T13:10:35.013 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.013 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.013 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:35.013 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.550353+0000", 2024-05-15T13:10:35.013 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013771200000000001 2024-05-15T13:10:35.013 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.013 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.013 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:10:35.013 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.550488+0000", 2024-05-15T13:10:35.013 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013527299999999999 2024-05-15T13:10:35.013 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.013 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.013 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:35.014 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.555384+0000", 2024-05-15T13:10:35.014 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0048959390000000002 2024-05-15T13:10:35.014 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.014 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.014 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:35.014 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.557282+0000", 2024-05-15T13:10:35.014 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0018979229999999999 2024-05-15T13:10:35.014 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.014 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.014 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:35.014 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.688707+0000", 2024-05-15T13:10:35.014 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.13142551499999999 2024-05-15T13:10:35.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.015 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.015 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:35.015 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.688764+0000", 2024-05-15T13:10:35.015 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7265000000000001e-05 2024-05-15T13:10:35.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.015 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.015 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:35.015 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.688791+0000", 2024-05-15T13:10:35.015 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6281000000000001e-05 2024-05-15T13:10:35.015 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:35.015 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:35.015 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:35.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.015 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.016 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:88309 6.b 6:db3f05ee:::benchmark_data_smithi177_122278_object88308:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e534)", 2024-05-15T13:10:35.016 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:08:43.565228+0000", 2024-05-15T13:10:35.016 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 110.124348004, 2024-05-15T13:10:35.016 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.123676245, 2024-05-15T13:10:35.016 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:35.016 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:35.016 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:35.016 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:10:35.016 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:10:35.016 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 88309 2024-05-15T13:10:35.016 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.016 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:35.016 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.016 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:35.016 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.565228+0000", 2024-05-15T13:10:35.017 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:35.017 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.017 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.017 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:35.017 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.565228+0000", 2024-05-15T13:10:35.017 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:35.017 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.017 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.017 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:35.017 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.565229+0000", 2024-05-15T13:10:35.017 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.068e-06 2024-05-15T13:10:35.017 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.017 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.017 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:35.018 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.565273+0000", 2024-05-15T13:10:35.018 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4499999999999997e-05 2024-05-15T13:10:35.018 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.018 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.018 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:35.018 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.565289+0000", 2024-05-15T13:10:35.018 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5682999999999999e-05 2024-05-15T13:10:35.018 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.018 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.018 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:35.018 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.565297+0000", 2024-05-15T13:10:35.018 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5979999999999999e-06 2024-05-15T13:10:35.018 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.018 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.019 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:35.019 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.565353+0000", 2024-05-15T13:10:35.019 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6489000000000001e-05 2024-05-15T13:10:35.019 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.019 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.019 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:35.019 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.565493+0000", 2024-05-15T13:10:35.019 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000140195 2024-05-15T13:10:35.019 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.019 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.019 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:10:35.019 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.565675+0000", 2024-05-15T13:10:35.019 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000181903 2024-05-15T13:10:35.019 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.019 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.020 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:35.020 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.568614+0000", 2024-05-15T13:10:35.020 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029386550000000001 2024-05-15T13:10:35.020 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.020 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.020 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:35.020 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.568640+0000", 2024-05-15T13:10:35.020 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5752e-05 2024-05-15T13:10:35.020 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.020 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.020 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:35.020 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.688842+0000", 2024-05-15T13:10:35.020 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12020226000000001 2024-05-15T13:10:35.020 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.020 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.020 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:35.021 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.688879+0000", 2024-05-15T13:10:35.021 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7049999999999999e-05 2024-05-15T13:10:35.021 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.021 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.021 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:35.021 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:08:43.688904+0000", 2024-05-15T13:10:35.021 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5092e-05 2024-05-15T13:10:35.021 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:35.021 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:35.021 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:35.021 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.021 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.021 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:117474 6.6 6:6d00ee65:::benchmark_data_smithi177_122278_object117473:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e537)", 2024-05-15T13:10:35.021 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:09:11.411695+0000", 2024-05-15T13:10:35.021 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 82.277880472999996, 2024-05-15T13:10:35.022 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.156304263, 2024-05-15T13:10:35.022 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:35.022 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:35.022 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:35.022 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:10:35.022 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:10:35.022 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 117474 2024-05-15T13:10:35.022 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.022 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:35.022 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.022 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:35.022 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.411695+0000", 2024-05-15T13:10:35.022 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:35.022 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.023 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.023 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:35.023 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.411695+0000", 2024-05-15T13:10:35.023 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:35.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.023 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.023 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:35.023 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.411697+0000", 2024-05-15T13:10:35.023 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4109999999999999e-06 2024-05-15T13:10:35.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.023 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.023 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:35.023 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.411842+0000", 2024-05-15T13:10:35.023 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014515200000000001 2024-05-15T13:10:35.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.024 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.024 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:35.024 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.411859+0000", 2024-05-15T13:10:35.024 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6739e-05 2024-05-15T13:10:35.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.024 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.024 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:35.024 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.411876+0000", 2024-05-15T13:10:35.024 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7212999999999999e-05 2024-05-15T13:10:35.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.024 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.024 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:35.024 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.411950+0000", 2024-05-15T13:10:35.024 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4501999999999999e-05 2024-05-15T13:10:35.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.025 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.025 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:35.025 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.412088+0000", 2024-05-15T13:10:35.025 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013733300000000001 2024-05-15T13:10:35.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.025 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.025 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 4,6", 2024-05-15T13:10:35.025 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.412244+0000", 2024-05-15T13:10:35.025 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015671000000000001 2024-05-15T13:10:35.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.025 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.025 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:35.025 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.418261+0000", 2024-05-15T13:10:35.025 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0060165469999999997 2024-05-15T13:10:35.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.025 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.026 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:35.026 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.423489+0000", 2024-05-15T13:10:35.026 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.005227733 2024-05-15T13:10:35.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.026 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.026 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:35.026 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.567896+0000", 2024-05-15T13:10:35.026 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.144407652 2024-05-15T13:10:35.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.026 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.026 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:35.026 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.567947+0000", 2024-05-15T13:10:35.026 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0243e-05 2024-05-15T13:10:35.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.027 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.027 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:35.027 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.568000+0000", 2024-05-15T13:10:35.027 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3028000000000001e-05 2024-05-15T13:10:35.027 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:35.027 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:35.027 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:35.027 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.027 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.027 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:117499 6.6 6:6e580f81:::benchmark_data_smithi177_122278_object117498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e537)", 2024-05-15T13:10:35.027 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:09:11.435666+0000", 2024-05-15T13:10:35.027 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 82.253909528999998, 2024-05-15T13:10:35.027 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.133559714, 2024-05-15T13:10:35.027 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:35.027 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:35.028 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:35.028 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:10:35.028 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:10:35.028 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 117499 2024-05-15T13:10:35.028 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.028 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:35.028 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.028 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:35.028 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.435666+0000", 2024-05-15T13:10:35.028 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:35.028 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.028 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.028 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:35.028 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.435666+0000", 2024-05-15T13:10:35.028 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:35.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.029 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.029 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:35.029 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.435667+0000", 2024-05-15T13:10:35.029 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0400000000000005e-07 2024-05-15T13:10:35.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.029 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.029 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:35.029 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.435718+0000", 2024-05-15T13:10:35.029 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1419999999999999e-05 2024-05-15T13:10:35.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.029 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.029 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:35.029 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.435734+0000", 2024-05-15T13:10:35.029 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5413e-05 2024-05-15T13:10:35.030 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.030 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.030 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:35.030 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.435745+0000", 2024-05-15T13:10:35.030 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1121000000000001e-05 2024-05-15T13:10:35.030 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.030 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.030 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:35.030 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.435801+0000", 2024-05-15T13:10:35.030 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5809e-05 2024-05-15T13:10:35.030 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.030 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.030 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:35.030 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.435951+0000", 2024-05-15T13:10:35.031 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000150063 2024-05-15T13:10:35.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.031 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.031 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 4,6", 2024-05-15T13:10:35.031 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.436119+0000", 2024-05-15T13:10:35.031 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000168219 2024-05-15T13:10:35.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.031 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.031 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:35.031 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.440334+0000", 2024-05-15T13:10:35.031 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0042146179999999998 2024-05-15T13:10:35.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.031 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.031 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:35.031 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.449209+0000", 2024-05-15T13:10:35.032 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0088749229999999998 2024-05-15T13:10:35.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.032 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:35.032 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.569126+0000", 2024-05-15T13:10:35.032 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.11991734599999999 2024-05-15T13:10:35.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.032 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:35.032 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.569182+0000", 2024-05-15T13:10:35.032 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6215999999999998e-05 2024-05-15T13:10:35.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.032 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:35.032 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.569226+0000", 2024-05-15T13:10:35.033 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3761999999999999e-05 2024-05-15T13:10:35.033 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:35.033 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:35.033 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:35.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.033 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.033 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:117510 6.6 6:6acf24ab:::benchmark_data_smithi177_122278_object117509:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e537)", 2024-05-15T13:10:35.033 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:09:11.454343+0000", 2024-05-15T13:10:35.033 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 82.235233018000002, 2024-05-15T13:10:35.033 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.119638415, 2024-05-15T13:10:35.033 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:35.033 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:35.033 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:35.033 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:10:35.033 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:10:35.034 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 117510 2024-05-15T13:10:35.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.034 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:35.034 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.034 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:35.034 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.454343+0000", 2024-05-15T13:10:35.034 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:35.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.034 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.034 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:35.034 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.454343+0000", 2024-05-15T13:10:35.034 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:35.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.035 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.035 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:35.035 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.454344+0000", 2024-05-15T13:10:35.035 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.291e-06 2024-05-15T13:10:35.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.035 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.035 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:35.035 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.454417+0000", 2024-05-15T13:10:35.035 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3104999999999999e-05 2024-05-15T13:10:35.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.035 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.035 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:35.035 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.454436+0000", 2024-05-15T13:10:35.035 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9238000000000001e-05 2024-05-15T13:10:35.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.035 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.036 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:35.036 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.454445+0000", 2024-05-15T13:10:35.036 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0040000000000005e-06 2024-05-15T13:10:35.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.036 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.036 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:35.036 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.454503+0000", 2024-05-15T13:10:35.036 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7734e-05 2024-05-15T13:10:35.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.036 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.036 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:35.036 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.454632+0000", 2024-05-15T13:10:35.036 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012930099999999999 2024-05-15T13:10:35.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.036 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.037 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 4,6", 2024-05-15T13:10:35.037 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.454728+0000", 2024-05-15T13:10:35.037 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5907000000000002e-05 2024-05-15T13:10:35.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.037 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.037 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:35.037 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.460565+0000", 2024-05-15T13:10:35.037 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0058369770000000001 2024-05-15T13:10:35.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.037 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.037 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:35.037 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.463663+0000", 2024-05-15T13:10:35.037 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0030974219999999998 2024-05-15T13:10:35.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.037 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.038 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:35.038 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.573876+0000", 2024-05-15T13:10:35.038 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.11021307900000001 2024-05-15T13:10:35.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.038 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:35.038 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.573938+0000", 2024-05-15T13:10:35.038 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2534999999999997e-05 2024-05-15T13:10:35.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.038 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:35.038 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:09:11.573981+0000", 2024-05-15T13:10:35.038 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2821999999999998e-05 2024-05-15T13:10:35.038 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:35.038 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:35.039 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:35.039 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.039 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.039 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:190161 6.c 6:311901fe:::benchmark_data_smithi177_122278_object59790:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-05-15T13:10:35.039 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:10:33.283244+0000", 2024-05-15T13:10:35.039 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.40633149800000001, 2024-05-15T13:10:35.039 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.060146052999999998, 2024-05-15T13:10:35.039 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:35.039 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:35.039 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:35.039 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:10:35.039 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:10:35.039 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 190161 2024-05-15T13:10:35.039 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.040 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:35.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.040 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:35.040 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.283244+0000", 2024-05-15T13:10:35.040 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:35.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.040 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:35.040 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.283244+0000", 2024-05-15T13:10:35.040 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:35.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.040 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:35.040 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.283245+0000", 2024-05-15T13:10:35.041 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3099999999999998e-07 2024-05-15T13:10:35.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.041 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:35.041 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.283248+0000", 2024-05-15T13:10:35.041 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8009999999999999e-06 2024-05-15T13:10:35.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.041 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:35.041 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.283255+0000", 2024-05-15T13:10:35.041 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0269999999999998e-06 2024-05-15T13:10:35.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.041 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:35.041 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.283261+0000", 2024-05-15T13:10:35.042 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8569999999999997e-06 2024-05-15T13:10:35.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.042 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.042 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:35.042 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.309875+0000", 2024-05-15T13:10:35.042 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.026613102 2024-05-15T13:10:35.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.042 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.042 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:35.042 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.309998+0000", 2024-05-15T13:10:35.042 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000123857 2024-05-15T13:10:35.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.042 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.043 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,6", 2024-05-15T13:10:35.043 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.310124+0000", 2024-05-15T13:10:35.043 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012598500000000001 2024-05-15T13:10:35.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.043 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.043 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:35.043 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.324403+0000", 2024-05-15T13:10:35.043 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014278766 2024-05-15T13:10:35.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.043 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.043 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:35.043 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.328933+0000", 2024-05-15T13:10:35.043 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0045298439999999999 2024-05-15T13:10:35.044 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.044 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.044 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:35.044 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.343316+0000", 2024-05-15T13:10:35.044 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.01438262 2024-05-15T13:10:35.044 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.044 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.044 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:35.044 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.343358+0000", 2024-05-15T13:10:35.044 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2104999999999998e-05 2024-05-15T13:10:35.044 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.044 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.044 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:35.044 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.343390+0000", 2024-05-15T13:10:35.045 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2657999999999998e-05 2024-05-15T13:10:35.045 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:35.045 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:35.045 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:35.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.045 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:190324 6.b 6:d477b35d:::benchmark_data_smithi177_122278_object59953:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-05-15T13:10:35.045 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:10:33.489843+0000", 2024-05-15T13:10:35.045 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.19973228500000001, 2024-05-15T13:10:35.045 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.072243412000000007, 2024-05-15T13:10:35.045 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:35.045 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:35.045 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:35.045 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:10:35.046 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:10:35.046 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 190324 2024-05-15T13:10:35.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.046 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:35.046 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.046 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:35.046 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.489843+0000", 2024-05-15T13:10:35.046 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:35.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.046 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.046 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:35.046 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.489843+0000", 2024-05-15T13:10:35.046 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:35.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.047 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.047 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:35.047 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.489844+0000", 2024-05-15T13:10:35.047 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7899999999999999e-07 2024-05-15T13:10:35.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.047 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.047 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:35.047 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.489846+0000", 2024-05-15T13:10:35.047 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3700000000000002e-06 2024-05-15T13:10:35.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.047 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.047 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:35.047 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.489852+0000", 2024-05-15T13:10:35.047 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9349999999999999e-06 2024-05-15T13:10:35.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.048 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.048 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:35.048 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.489857+0000", 2024-05-15T13:10:35.048 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9439999999999996e-06 2024-05-15T13:10:35.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.048 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.048 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:35.048 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.495988+0000", 2024-05-15T13:10:35.048 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00613074 2024-05-15T13:10:35.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.048 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.048 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:35.048 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.496133+0000", 2024-05-15T13:10:35.048 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014508500000000001 2024-05-15T13:10:35.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.049 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.049 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:10:35.049 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.496235+0000", 2024-05-15T13:10:35.049 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000102433 2024-05-15T13:10:35.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.049 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.049 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:35.049 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.508641+0000", 2024-05-15T13:10:35.049 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012405803999999999 2024-05-15T13:10:35.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.049 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.049 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:35.049 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.523185+0000", 2024-05-15T13:10:35.050 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014543395000000001 2024-05-15T13:10:35.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.050 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:35.050 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.561968+0000", 2024-05-15T13:10:35.050 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.038783249999999998 2024-05-15T13:10:35.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.050 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:35.050 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.562004+0000", 2024-05-15T13:10:35.050 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6288999999999998e-05 2024-05-15T13:10:35.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.050 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:35.051 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.562087+0000", 2024-05-15T13:10:35.051 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2787999999999994e-05 2024-05-15T13:10:35.051 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:35.051 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:35.051 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:35.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.051 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:190326 6.b 6:d02e5563:::benchmark_data_smithi177_122278_object59955:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-05-15T13:10:35.051 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:10:33.492687+0000", 2024-05-15T13:10:35.051 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.196889012, 2024-05-15T13:10:35.051 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.069573121000000002, 2024-05-15T13:10:35.051 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:35.051 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:35.052 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:35.052 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:10:35.052 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:10:35.052 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 190326 2024-05-15T13:10:35.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.052 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:35.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.052 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:35.052 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.492687+0000", 2024-05-15T13:10:35.052 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:35.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.052 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:35.052 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.492687+0000", 2024-05-15T13:10:35.052 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:35.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.053 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:35.053 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.492688+0000", 2024-05-15T13:10:35.053 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.141e-06 2024-05-15T13:10:35.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.053 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:35.053 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.492696+0000", 2024-05-15T13:10:35.053 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9719999999999992e-06 2024-05-15T13:10:35.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.053 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:35.053 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.492721+0000", 2024-05-15T13:10:35.054 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5324999999999999e-05 2024-05-15T13:10:35.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.054 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.054 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:35.054 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.492730+0000", 2024-05-15T13:10:35.054 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4030000000000007e-06 2024-05-15T13:10:35.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.054 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.054 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:35.054 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.500230+0000", 2024-05-15T13:10:35.054 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0075000570000000001 2024-05-15T13:10:35.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.054 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.055 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:35.055 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.500372+0000", 2024-05-15T13:10:35.055 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000142554 2024-05-15T13:10:35.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.055 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:10:35.055 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.500496+0000", 2024-05-15T13:10:35.055 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012334800000000001 2024-05-15T13:10:35.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.055 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:35.055 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.508681+0000", 2024-05-15T13:10:35.055 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0081854189999999993 2024-05-15T13:10:35.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.056 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:35.056 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.523190+0000", 2024-05-15T13:10:35.056 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014508716 2024-05-15T13:10:35.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.056 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:35.056 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.562148+0000", 2024-05-15T13:10:35.056 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.038958287000000001 2024-05-15T13:10:35.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.056 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:35.056 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.562208+0000", 2024-05-15T13:10:35.057 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0402999999999997e-05 2024-05-15T13:10:35.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.057 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:35.057 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.562260+0000", 2024-05-15T13:10:35.057 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1496000000000003e-05 2024-05-15T13:10:35.057 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:35.057 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:35.057 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:35.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.057 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:190329 6.b 6:db466b2e:::benchmark_data_smithi177_122278_object59958:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-05-15T13:10:35.057 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:10:33.493200+0000", 2024-05-15T13:10:35.057 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.19637576000000001, 2024-05-15T13:10:35.058 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.069218136, 2024-05-15T13:10:35.058 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:10:35.058 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:10:35.058 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:10:35.058 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:10:35.058 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:10:35.058 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 190329 2024-05-15T13:10:35.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.058 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:10:35.058 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.058 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:10:35.058 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.493200+0000", 2024-05-15T13:10:35.058 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:35.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.059 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:10:35.059 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.493200+0000", 2024-05-15T13:10:35.059 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:10:35.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.059 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:10:35.059 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.493201+0000", 2024-05-15T13:10:35.059 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9500000000000001e-07 2024-05-15T13:10:35.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.059 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:10:35.059 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.493205+0000", 2024-05-15T13:10:35.059 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2930000000000002e-06 2024-05-15T13:10:35.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.060 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:10:35.060 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.493216+0000", 2024-05-15T13:10:35.060 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0696999999999999e-05 2024-05-15T13:10:35.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.060 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:10:35.060 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.493223+0000", 2024-05-15T13:10:35.060 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.311e-06 2024-05-15T13:10:35.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.060 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:10:35.060 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.504639+0000", 2024-05-15T13:10:35.060 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011415532000000001 2024-05-15T13:10:35.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.061 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:10:35.061 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.504746+0000", 2024-05-15T13:10:35.061 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000106903 2024-05-15T13:10:35.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.061 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.061 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:10:35.061 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.504831+0000", 2024-05-15T13:10:35.061 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5024000000000005e-05 2024-05-15T13:10:35.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.061 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.061 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:10:35.061 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.523195+0000", 2024-05-15T13:10:35.061 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.018364012999999998 2024-05-15T13:10:35.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.061 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.062 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:35.062 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.523241+0000", 2024-05-15T13:10:35.062 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6270000000000003e-05 2024-05-15T13:10:35.062 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.062 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.062 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:10:35.062 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.562342+0000", 2024-05-15T13:10:35.062 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.039101359000000002 2024-05-15T13:10:35.062 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.062 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.062 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:10:35.062 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.562378+0000", 2024-05-15T13:10:35.062 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6239999999999999e-05 2024-05-15T13:10:35.062 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:10:35.062 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:10:35.063 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:10:35.063 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:33.562418+0000", 2024-05-15T13:10:35.063 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9699000000000001e-05 2024-05-15T13:10:35.063 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:35.063 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:35.063 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:35.063 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:10:35.063 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:10:35.063 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:10:35.063 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:10:35.114 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:35.152 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:10:35.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:35.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:10:35.257 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:10:35.313 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:36.823 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:10:36.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:36.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:10:36.848 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T13:10:36.848 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T13:10:36.849 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.8 'trim stale osdmaps' 2024-05-15T13:10:36.933 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:10:36.960 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:36.970 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-05-15T13:10:36.984 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:10:36.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:36.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:10:37.090 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:10:37.126 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:37.148 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:10:37.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:37.159 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:10:37.260 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:10:37.291 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:37.311 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:10:37.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:37.324 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:10:37.337 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:10:37.425 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:10:37.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:37 smithi177 ceph-mon[44303]: pgmap v1481: 27 pgs: 27 active+clean; 4.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 792 op/s 2024-05-15T13:10:37.456 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:37.477 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:10:37.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:37.483 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:10:37.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:37 smithi027 ceph-mon[39795]: pgmap v1481: 27 pgs: 27 active+clean; 4.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 792 op/s 2024-05-15T13:10:37.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:37 smithi120 ceph-mon[43961]: pgmap v1481: 27 pgs: 27 active+clean; 4.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 792 op/s 2024-05-15T13:10:37.583 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:10:37.624 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:37.647 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:10:37.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:37.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:10:37.686 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:10:37.753 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:10:37.758 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:10:37.816 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:37.878 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:10:37.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:37.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:10:37.984 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:10:38.016 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:38.037 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:10:38.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:38.039 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:10:38.140 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:10:38.166 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:38.193 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:10:38.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:38.195 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:10:38.296 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:10:38.345 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:38.400 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:10:38.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:38.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:10:38.506 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:10:38.563 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:38.740 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:10:38.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:38.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:10:38.847 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:10:38.878 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:38.900 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:10:38.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:38.906 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:10:39.007 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:10:39.048 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:39.074 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:10:39.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:39.080 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:10:39.180 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:10:39.243 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:39.275 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:10:39.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:39.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:10:39.318 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:10:39.389 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:10:39.424 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:39.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:39 smithi027 ceph-mon[39795]: pgmap v1482: 27 pgs: 27 active+clean; 4.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 823 op/s 2024-05-15T13:10:39.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:39 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/106187388' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:10:39.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:39 smithi120 ceph-mon[43961]: pgmap v1482: 27 pgs: 27 active+clean; 4.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 823 op/s 2024-05-15T13:10:39.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:39 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/106187388' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:10:39.635 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:10:39.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:39.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:10:39.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:39 smithi177 ceph-mon[44303]: pgmap v1482: 27 pgs: 27 active+clean; 4.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 823 op/s 2024-05-15T13:10:39.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:39 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/106187388' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:10:39.732 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:10:39.732 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T13:10:39.732 INFO:teuthology.orchestra.run.smithi120.stdout: "num_ops": 0 2024-05-15T13:10:39.732 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:10:39.742 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:10:39.805 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:39.915 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:10:39.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:39.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:10:40.023 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:10:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:40.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:40 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:10:40.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:40 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/106187388' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:10:40.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:40 smithi027 ceph-mon[39795]: osdmap e554: 12 total, 12 up, 11 in 2024-05-15T13:10:40.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:40 smithi027 ceph-mon[39795]: 6.9 scrub starts 2024-05-15T13:10:40.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:40 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:10:40.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:40 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/106187388' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:10:40.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:40 smithi120 ceph-mon[43961]: osdmap e554: 12 total, 12 up, 11 in 2024-05-15T13:10:40.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:40 smithi120 ceph-mon[43961]: 6.9 scrub starts 2024-05-15T13:10:40.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:40 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:10:40.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:40 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/106187388' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:10:40.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:40 smithi177 ceph-mon[44303]: osdmap e554: 12 total, 12 up, 11 in 2024-05-15T13:10:40.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:40 smithi177 ceph-mon[44303]: 6.9 scrub starts 2024-05-15T13:10:40.795 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-05-15T13:10:41.172 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:10:41.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:41 smithi027 ceph-mon[39795]: 6.2 scrub starts 2024-05-15T13:10:41.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:41 smithi027 ceph-mon[39795]: 6.7 scrub starts 2024-05-15T13:10:41.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:41 smithi027 ceph-mon[39795]: pgmap v1484: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 4.0 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 838 op/s 2024-05-15T13:10:41.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:41 smithi027 ceph-mon[39795]: 6.6 scrub starts 2024-05-15T13:10:41.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:41 smithi120 ceph-mon[43961]: 6.2 scrub starts 2024-05-15T13:10:41.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:41 smithi120 ceph-mon[43961]: 6.7 scrub starts 2024-05-15T13:10:41.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:41 smithi120 ceph-mon[43961]: pgmap v1484: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 4.0 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 838 op/s 2024-05-15T13:10:41.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:41 smithi120 ceph-mon[43961]: 6.6 scrub starts 2024-05-15T13:10:41.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:41 smithi177 ceph-mon[44303]: 6.2 scrub starts 2024-05-15T13:10:41.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:41 smithi177 ceph-mon[44303]: 6.7 scrub starts 2024-05-15T13:10:41.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:41 smithi177 ceph-mon[44303]: pgmap v1484: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 4.0 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 838 op/s 2024-05-15T13:10:41.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:41 smithi177 ceph-mon[44303]: 6.6 scrub starts 2024-05-15T13:10:42.515 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:10:42.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:42.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:10:42.554 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:42 smithi027 ceph-mon[39795]: 6.2 scrub ok 2024-05-15T13:10:42.555 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:42 smithi027 ceph-mon[39795]: 6.f scrub starts 2024-05-15T13:10:42.555 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:42 smithi027 ceph-mon[39795]: 6.9 scrub ok 2024-05-15T13:10:42.555 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:42 smithi027 ceph-mon[39795]: 6.7 scrub ok 2024-05-15T13:10:42.555 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:42 smithi027 ceph-mon[39795]: 1.9 scrub starts 2024-05-15T13:10:42.555 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:42 smithi027 ceph-mon[39795]: 1.9 scrub ok 2024-05-15T13:10:42.555 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:42 smithi027 ceph-mon[39795]: 6.6 scrub ok 2024-05-15T13:10:42.555 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:42 smithi027 ceph-mon[39795]: 6.0 scrub starts 2024-05-15T13:10:42.627 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:10:42.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:42 smithi177 ceph-mon[44303]: 6.2 scrub ok 2024-05-15T13:10:42.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:42 smithi177 ceph-mon[44303]: 6.f scrub starts 2024-05-15T13:10:42.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:42 smithi177 ceph-mon[44303]: 6.9 scrub ok 2024-05-15T13:10:42.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:42 smithi177 ceph-mon[44303]: 6.7 scrub ok 2024-05-15T13:10:42.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:42 smithi177 ceph-mon[44303]: 1.9 scrub starts 2024-05-15T13:10:42.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:42 smithi177 ceph-mon[44303]: 1.9 scrub ok 2024-05-15T13:10:42.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:42 smithi177 ceph-mon[44303]: 6.6 scrub ok 2024-05-15T13:10:42.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:42 smithi177 ceph-mon[44303]: 6.0 scrub starts 2024-05-15T13:10:42.720 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:42.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:42 smithi120 ceph-mon[43961]: 6.2 scrub ok 2024-05-15T13:10:42.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:42 smithi120 ceph-mon[43961]: 6.f scrub starts 2024-05-15T13:10:42.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:42 smithi120 ceph-mon[43961]: 6.9 scrub ok 2024-05-15T13:10:42.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:42 smithi120 ceph-mon[43961]: 6.7 scrub ok 2024-05-15T13:10:42.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:42 smithi120 ceph-mon[43961]: 1.9 scrub starts 2024-05-15T13:10:42.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:42 smithi120 ceph-mon[43961]: 1.9 scrub ok 2024-05-15T13:10:42.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:42 smithi120 ceph-mon[43961]: 6.6 scrub ok 2024-05-15T13:10:42.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:42 smithi120 ceph-mon[43961]: 6.0 scrub starts 2024-05-15T13:10:42.943 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:10:42.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:42.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:10:43.056 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:10:43.112 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:43.398 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:43 smithi120 ceph-mon[43961]: 6.f scrub ok 2024-05-15T13:10:43.398 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:43 smithi120 ceph-mon[43961]: 1.a scrub starts 2024-05-15T13:10:43.398 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:43 smithi120 ceph-mon[43961]: pgmap v1485: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 4.0 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 836 op/s 2024-05-15T13:10:43.398 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:43 smithi120 ceph-mon[43961]: 1.a scrub ok 2024-05-15T13:10:43.398 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:43 smithi120 ceph-mon[43961]: 6.8 scrub starts 2024-05-15T13:10:43.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:43 smithi177 ceph-mon[44303]: 6.f scrub ok 2024-05-15T13:10:43.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:43 smithi177 ceph-mon[44303]: 1.a scrub starts 2024-05-15T13:10:43.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:43 smithi177 ceph-mon[44303]: pgmap v1485: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 4.0 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 836 op/s 2024-05-15T13:10:43.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:43 smithi177 ceph-mon[44303]: 1.a scrub ok 2024-05-15T13:10:43.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:43 smithi177 ceph-mon[44303]: 6.8 scrub starts 2024-05-15T13:10:43.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:43 smithi027 ceph-mon[39795]: 6.f scrub ok 2024-05-15T13:10:43.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:43 smithi027 ceph-mon[39795]: 1.a scrub starts 2024-05-15T13:10:43.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:43 smithi027 ceph-mon[39795]: pgmap v1485: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 4.0 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 836 op/s 2024-05-15T13:10:43.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:43 smithi027 ceph-mon[39795]: 1.a scrub ok 2024-05-15T13:10:43.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:43 smithi027 ceph-mon[39795]: 6.8 scrub starts 2024-05-15T13:10:44.029 INFO:teuthology.orchestra.run.smithi027.stderr:no valid command found; 10 closest matches: 2024-05-15T13:10:44.029 INFO:teuthology.orchestra.run.smithi027.stderr:0 2024-05-15T13:10:44.029 INFO:teuthology.orchestra.run.smithi027.stderr:1 2024-05-15T13:10:44.029 INFO:teuthology.orchestra.run.smithi027.stderr:2 2024-05-15T13:10:44.030 INFO:teuthology.orchestra.run.smithi027.stderr:abort 2024-05-15T13:10:44.030 INFO:teuthology.orchestra.run.smithi027.stderr:assert 2024-05-15T13:10:44.030 INFO:teuthology.orchestra.run.smithi027.stderr:bench [] [] [] [] 2024-05-15T13:10:44.030 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs debug_inject_read_zeros 2024-05-15T13:10:44.030 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs files list 2024-05-15T13:10:44.030 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs stats 2024-05-15T13:10:44.030 INFO:teuthology.orchestra.run.smithi027.stderr:bluestore allocator dump block 2024-05-15T13:10:44.030 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: invalid command 2024-05-15T13:10:44.038 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:10:44.140 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:10:44.140 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:10:44.184 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:10:44.230 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:10:44.273 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:10:44.302 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:10:44.343 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:10:44.360 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:44 smithi027 ceph-mon[39795]: 6.0 scrub ok 2024-05-15T13:10:44.360 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:44 smithi027 ceph-mon[39795]: 6.1 scrub starts 2024-05-15T13:10:44.360 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:44 smithi027 ceph-mon[39795]: 6.8 scrub ok 2024-05-15T13:10:44.360 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:44 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T13:10:44.360 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:44 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T13:10:44.360 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:44 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2409102192' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:10:44.380 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:10:44.438 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:10:44.494 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:10:44.547 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:10:44.557 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:10:44.576 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:10:44.576 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:10:44.576 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:10:44.576 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:10:44.576 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:10:44.576 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:10:44.576 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:10:44.577 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:10:44.577 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:10:44.577 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:10:44.577 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:10:44.577 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:10:44.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:44 smithi177 ceph-mon[44303]: 6.0 scrub ok 2024-05-15T13:10:44.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:44 smithi177 ceph-mon[44303]: 6.1 scrub starts 2024-05-15T13:10:44.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:44 smithi177 ceph-mon[44303]: 6.8 scrub ok 2024-05-15T13:10:44.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:44 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T13:10:44.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:44 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T13:10:44.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:44 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2409102192' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:10:44.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:44 smithi120 ceph-mon[43961]: 6.0 scrub ok 2024-05-15T13:10:44.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:44 smithi120 ceph-mon[43961]: 6.1 scrub starts 2024-05-15T13:10:44.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:44 smithi120 ceph-mon[43961]: 6.8 scrub ok 2024-05-15T13:10:44.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:44 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T13:10:44.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:44 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T13:10:44.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:44 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2409102192' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:10:45.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:45 smithi177 ceph-mon[44303]: 1.7 scrub starts 2024-05-15T13:10:45.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:45 smithi177 ceph-mon[44303]: 1.7 scrub ok 2024-05-15T13:10:45.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:45 smithi177 ceph-mon[44303]: 6.1 scrub ok 2024-05-15T13:10:45.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:45 smithi177 ceph-mon[44303]: pgmap v1486: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 3.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 816 op/s 2024-05-15T13:10:45.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:45 smithi027 ceph-mon[39795]: 1.7 scrub starts 2024-05-15T13:10:45.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:45 smithi027 ceph-mon[39795]: 1.7 scrub ok 2024-05-15T13:10:45.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:45 smithi027 ceph-mon[39795]: 6.1 scrub ok 2024-05-15T13:10:45.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:45 smithi027 ceph-mon[39795]: pgmap v1486: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 3.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 816 op/s 2024-05-15T13:10:45.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:45 smithi120 ceph-mon[43961]: 1.7 scrub starts 2024-05-15T13:10:45.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:45 smithi120 ceph-mon[43961]: 1.7 scrub ok 2024-05-15T13:10:45.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:45 smithi120 ceph-mon[43961]: 6.1 scrub ok 2024-05-15T13:10:45.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:45 smithi120 ceph-mon[43961]: pgmap v1486: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 3.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 816 op/s 2024-05-15T13:10:46.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:46 smithi177 ceph-mon[44303]: 6.3 scrub starts 2024-05-15T13:10:46.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:46 smithi177 ceph-mon[44303]: 6.3 scrub ok 2024-05-15T13:10:46.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:46 smithi177 ceph-mon[44303]: pgmap v1487: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 3.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 797 op/s 2024-05-15T13:10:46.741 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:10:46.747 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:46 smithi120 ceph-mon[43961]: 6.3 scrub starts 2024-05-15T13:10:46.747 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:46 smithi120 ceph-mon[43961]: 6.3 scrub ok 2024-05-15T13:10:46.747 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:46 smithi120 ceph-mon[43961]: pgmap v1487: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 3.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 797 op/s 2024-05-15T13:10:46.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:46.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:10:46.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:46 smithi027 ceph-mon[39795]: 6.3 scrub starts 2024-05-15T13:10:46.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:46 smithi027 ceph-mon[39795]: 6.3 scrub ok 2024-05-15T13:10:46.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:46 smithi027 ceph-mon[39795]: pgmap v1487: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 3.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 797 op/s 2024-05-15T13:10:46.851 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:10:46.882 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:46.906 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:10:46.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:46.913 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:10:46.938 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:10:46.977 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T13:10:46.978 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T13:10:46.978 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.9 'trim stale osdmaps' 2024-05-15T13:10:47.013 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:10:47.091 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:47.168 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:10:47.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:47.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:10:47.274 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:10:47.308 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:47.341 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:10:47.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:47.344 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:10:47.445 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:10:47.472 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:47.491 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:10:47.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:47.494 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:10:47.594 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:10:47.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:47 smithi027 ceph-mon[39795]: 6.b scrub starts 2024-05-15T13:10:47.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:47 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:10:47.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:47 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:10:47.654 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:47.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:47 smithi177 ceph-mon[44303]: 6.b scrub starts 2024-05-15T13:10:47.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:47 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:10:47.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:47 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:10:47.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:47 smithi120 ceph-mon[43961]: 6.b scrub starts 2024-05-15T13:10:47.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:47 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:10:47.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:47 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:10:48.080 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:10:48.080 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:10:48.426 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:10:48.581 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:48 smithi120 ceph-mon[43961]: 6.5 scrub starts 2024-05-15T13:10:48.581 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:48 smithi120 ceph-mon[43961]: 1.6 scrub starts 2024-05-15T13:10:48.581 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:48 smithi120 ceph-mon[43961]: 1.6 scrub ok 2024-05-15T13:10:48.581 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:48 smithi120 ceph-mon[43961]: 6.c scrub starts 2024-05-15T13:10:48.581 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:48 smithi120 ceph-mon[43961]: 1.3 scrub starts 2024-05-15T13:10:48.581 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:48 smithi120 ceph-mon[43961]: osdmap e555: 12 total, 12 up, 11 in 2024-05-15T13:10:48.581 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:48 smithi120 ceph-mon[43961]: 1.3 scrub ok 2024-05-15T13:10:48.581 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:48 smithi120 ceph-mon[43961]: 6.5 scrub ok 2024-05-15T13:10:48.581 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:48 smithi120 ceph-mon[43961]: 6.a scrub starts 2024-05-15T13:10:48.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:48 smithi027 ceph-mon[39795]: 6.5 scrub starts 2024-05-15T13:10:48.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:48 smithi027 ceph-mon[39795]: 1.6 scrub starts 2024-05-15T13:10:48.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:48 smithi027 ceph-mon[39795]: 1.6 scrub ok 2024-05-15T13:10:48.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:48 smithi027 ceph-mon[39795]: 6.c scrub starts 2024-05-15T13:10:48.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:48 smithi027 ceph-mon[39795]: 1.3 scrub starts 2024-05-15T13:10:48.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:48 smithi027 ceph-mon[39795]: osdmap e555: 12 total, 12 up, 11 in 2024-05-15T13:10:48.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:48 smithi027 ceph-mon[39795]: 1.3 scrub ok 2024-05-15T13:10:48.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:48 smithi027 ceph-mon[39795]: 6.5 scrub ok 2024-05-15T13:10:48.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:48 smithi027 ceph-mon[39795]: 6.a scrub starts 2024-05-15T13:10:48.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:48 smithi027 ceph-mon[39795]: pgmap v1489: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 3.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 734 op/s 2024-05-15T13:10:48.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:48 smithi177 ceph-mon[44303]: 6.5 scrub starts 2024-05-15T13:10:48.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:48 smithi177 ceph-mon[44303]: 1.6 scrub starts 2024-05-15T13:10:48.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:48 smithi177 ceph-mon[44303]: 1.6 scrub ok 2024-05-15T13:10:48.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:48 smithi177 ceph-mon[44303]: 6.c scrub starts 2024-05-15T13:10:48.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:48 smithi177 ceph-mon[44303]: 1.3 scrub starts 2024-05-15T13:10:48.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:48 smithi177 ceph-mon[44303]: osdmap e555: 12 total, 12 up, 11 in 2024-05-15T13:10:48.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:48 smithi177 ceph-mon[44303]: 1.3 scrub ok 2024-05-15T13:10:48.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:48 smithi177 ceph-mon[44303]: 6.5 scrub ok 2024-05-15T13:10:48.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:48 smithi177 ceph-mon[44303]: 6.a scrub starts 2024-05-15T13:10:48.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:48 smithi177 ceph-mon[44303]: pgmap v1489: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 3.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 734 op/s 2024-05-15T13:10:49.124 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:10:49.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:49.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:10:49.148 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:10:49.232 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:10:49.269 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:49.294 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:10:49.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:49.297 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:10:49.398 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:10:49.458 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:49.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:49 smithi177 ceph-mon[44303]: 6.b scrub ok 2024-05-15T13:10:49.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:49 smithi177 ceph-mon[44303]: 6.4 scrub starts 2024-05-15T13:10:49.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:49 smithi177 ceph-mon[44303]: 6.c scrub ok 2024-05-15T13:10:49.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:49 smithi177 ceph-mon[44303]: 6.a scrub ok 2024-05-15T13:10:49.690 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:10:49.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:49.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:10:49.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:49 smithi027 ceph-mon[39795]: 6.b scrub ok 2024-05-15T13:10:49.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:49 smithi027 ceph-mon[39795]: 6.4 scrub starts 2024-05-15T13:10:49.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:49 smithi027 ceph-mon[39795]: 6.c scrub ok 2024-05-15T13:10:49.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:49 smithi027 ceph-mon[39795]: 6.a scrub ok 2024-05-15T13:10:49.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:48 smithi120 ceph-mon[43961]: pgmap v1489: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 3.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 734 op/s 2024-05-15T13:10:49.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:49 smithi120 ceph-mon[43961]: 6.b scrub ok 2024-05-15T13:10:49.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:49 smithi120 ceph-mon[43961]: 6.4 scrub starts 2024-05-15T13:10:49.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:49 smithi120 ceph-mon[43961]: 6.c scrub ok 2024-05-15T13:10:49.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:49 smithi120 ceph-mon[43961]: 6.a scrub ok 2024-05-15T13:10:49.801 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:10:49.876 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:49.937 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-05-15T13:10:49.988 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:10:49.990 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:10:49.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:49.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:10:50.093 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:10:50.184 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:50.210 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:10:50.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:50.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:10:50.316 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:10:50.345 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:10:50.399 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:50.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:50 smithi027 ceph-mon[39795]: pgmap v1490: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 3.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 764 op/s 2024-05-15T13:10:50.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:50 smithi120 ceph-mon[43961]: pgmap v1490: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 3.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 764 op/s 2024-05-15T13:10:50.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:50 smithi177 ceph-mon[44303]: pgmap v1490: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 3.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 764 op/s 2024-05-15T13:10:51.761 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:51 smithi120 ceph-mon[43961]: 6.4 scrub ok 2024-05-15T13:10:51.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:51 smithi177 ceph-mon[44303]: 6.4 scrub ok 2024-05-15T13:10:52.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:51 smithi027 ceph-mon[39795]: 6.4 scrub ok 2024-05-15T13:10:52.583 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:10:52.603 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:10:52.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:52.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:10:52.624 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:10:52.625 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:10:52.709 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:10:52.739 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:52.761 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:10:52.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:52.767 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:10:52.868 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:10:52.895 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:52 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2290007938' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:10:52.895 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:52 smithi027 ceph-mon[39795]: pgmap v1491: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 3.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 764 op/s 2024-05-15T13:10:52.898 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:52 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2290007938' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:10:52.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:52 smithi177 ceph-mon[44303]: pgmap v1491: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 3.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 764 op/s 2024-05-15T13:10:52.900 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:52.922 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:10:52.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:52.925 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:10:53.026 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:10:53.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:52 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2290007938' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:10:53.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:52 smithi120 ceph-mon[43961]: pgmap v1491: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 3.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 764 op/s 2024-05-15T13:10:53.053 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:53.074 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:10:53.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:53.080 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:10:53.181 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:10:53.220 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:53.241 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:10:53.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:53.247 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:10:53.348 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:10:53.396 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-05-15T13:10:53.412 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:53.561 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:53 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:10:53.561 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:53 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T13:10:53.561 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:53 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2290007938' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:10:53.561 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:53 smithi027 ceph-mon[39795]: osdmap e556: 12 total, 12 up, 11 in 2024-05-15T13:10:53.562 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:53 smithi027 ceph-mon[39795]: 6.e scrub starts 2024-05-15T13:10:53.635 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:10:53.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:53.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:10:53.742 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:10:53.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:53 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:10:53.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:53 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T13:10:53.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:53 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2290007938' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:10:53.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:53 smithi120 ceph-mon[43961]: osdmap e556: 12 total, 12 up, 11 in 2024-05-15T13:10:53.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:53 smithi120 ceph-mon[43961]: 6.e scrub starts 2024-05-15T13:10:53.798 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:53.893 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:10:53.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:53 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:10:53.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:53 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T13:10:53.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:53 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2290007938' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:10:53.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:53 smithi177 ceph-mon[44303]: osdmap e556: 12 total, 12 up, 11 in 2024-05-15T13:10:53.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:53 smithi177 ceph-mon[44303]: 6.e scrub starts 2024-05-15T13:10:54.271 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:10:54.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:54.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:10:54.379 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:10:54.456 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:54.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:54 smithi177 ceph-mon[44303]: 6.e scrub ok 2024-05-15T13:10:54.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:54 smithi177 ceph-mon[44303]: pgmap v1493: 27 pgs: 27 active+clean; 3.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 827 op/s 2024-05-15T13:10:54.940 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:54 smithi027 ceph-mon[39795]: 6.e scrub ok 2024-05-15T13:10:54.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:54 smithi027 ceph-mon[39795]: pgmap v1493: 27 pgs: 27 active+clean; 3.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 827 op/s 2024-05-15T13:10:55.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:54 smithi120 ceph-mon[43961]: 6.e scrub ok 2024-05-15T13:10:55.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:54 smithi120 ceph-mon[43961]: pgmap v1493: 27 pgs: 27 active+clean; 3.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 827 op/s 2024-05-15T13:10:55.432 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:10:55.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:55.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:10:55.538 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:10:55.591 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:55.632 INFO:teuthology.orchestra.run.smithi027.stderr:no valid command found; 10 closest matches: 2024-05-15T13:10:55.632 INFO:teuthology.orchestra.run.smithi027.stderr:0 2024-05-15T13:10:55.632 INFO:teuthology.orchestra.run.smithi027.stderr:1 2024-05-15T13:10:55.633 INFO:teuthology.orchestra.run.smithi027.stderr:2 2024-05-15T13:10:55.633 INFO:teuthology.orchestra.run.smithi027.stderr:abort 2024-05-15T13:10:55.633 INFO:teuthology.orchestra.run.smithi027.stderr:assert 2024-05-15T13:10:55.633 INFO:teuthology.orchestra.run.smithi027.stderr:bench [] [] [] [] 2024-05-15T13:10:55.633 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs debug_inject_read_zeros 2024-05-15T13:10:55.633 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs files list 2024-05-15T13:10:55.633 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs stats 2024-05-15T13:10:55.633 INFO:teuthology.orchestra.run.smithi027.stderr:bluestore allocator dump block 2024-05-15T13:10:55.633 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: invalid command 2024-05-15T13:10:56.162 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:10:56.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:56.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:10:56.272 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:10:56.326 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:56.372 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:10:56.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:56.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:10:56.472 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T13:10:56.473 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T13:10:56.473 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.10 'trim stale osdmaps' 2024-05-15T13:10:56.482 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:10:56.575 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:56.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:56 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:10:56.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:56 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:10:56.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:56 smithi027 ceph-mon[39795]: osdmap e557: 12 total, 12 up, 11 in 2024-05-15T13:10:56.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:56 smithi027 ceph-mon[39795]: pgmap v1495: 27 pgs: 27 active+clean; 3.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 803 op/s 2024-05-15T13:10:56.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:56 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:10:56.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:56 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:10:56.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:56 smithi120 ceph-mon[43961]: osdmap e557: 12 total, 12 up, 11 in 2024-05-15T13:10:56.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:56 smithi120 ceph-mon[43961]: pgmap v1495: 27 pgs: 27 active+clean; 3.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 803 op/s 2024-05-15T13:10:56.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:56 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:10:56.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:56 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:10:56.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:56 smithi177 ceph-mon[44303]: osdmap e557: 12 total, 12 up, 11 in 2024-05-15T13:10:56.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:56 smithi177 ceph-mon[44303]: pgmap v1495: 27 pgs: 27 active+clean; 3.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 803 op/s 2024-05-15T13:10:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:10:56.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:56.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:10:57.056 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:10:57.121 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:57.207 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:10:57.305 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:10:57.305 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:10:57.306 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:10:57.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:57.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:10:57.412 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:10:57.447 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:57.475 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:10:57.482 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:10:57.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:57.485 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:10:57.586 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:10:57.693 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:57.808 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:10:57.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:57.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:10:57.912 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:10:57.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:57 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:10:57.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:57 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:10:57.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:57 smithi177 ceph-mon[44303]: 1.2 scrub starts 2024-05-15T13:10:57.949 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:57 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:10:57.949 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:57 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:10:57.949 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:57 smithi120 ceph-mon[43961]: 1.2 scrub starts 2024-05-15T13:10:57.956 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:58.019 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:57 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:10:58.019 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:57 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:10:58.019 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:57 smithi027 ceph-mon[39795]: 1.2 scrub starts 2024-05-15T13:10:58.050 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:10:58.107 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:10:58.151 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:10:58.190 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:10:58.230 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:10:58.280 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:10:58.318 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:10:58.352 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:10:58.425 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:10:58.437 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:10:58.486 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:10:58.527 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:10:58.545 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:10:58.546 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:10:58.547 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:10:58.547 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:10:58.547 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:10:58.547 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:10:58.547 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:10:58.547 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:10:58.547 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:10:58.547 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:10:58.547 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:10:58.547 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:10:58.547 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:10:58.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:58.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:10:58.570 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-05-15T13:10:58.653 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:10:58.684 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:58.708 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:10:58.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:58.711 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:10:58.812 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:10:58.870 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:10:58.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:58 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:10:58.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:58 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:10:58.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:58 smithi177 ceph-mon[44303]: 1.2 scrub ok 2024-05-15T13:10:58.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:58 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:10:58.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:58 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:10:58.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:58 smithi177 ceph-mon[44303]: osdmap e558: 12 total, 12 up, 11 in 2024-05-15T13:10:58.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:58 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1540613141' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:10:58.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:10:58 smithi177 ceph-mon[44303]: pgmap v1497: 27 pgs: 27 active+clean; 3.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 888 op/s 2024-05-15T13:10:58.995 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:10:59.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:58 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:10:59.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:58 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:10:59.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:58 smithi027 ceph-mon[39795]: 1.2 scrub ok 2024-05-15T13:10:59.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:58 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:10:59.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:58 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:10:59.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:58 smithi027 ceph-mon[39795]: osdmap e558: 12 total, 12 up, 11 in 2024-05-15T13:10:59.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:58 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1540613141' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:10:59.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:10:58 smithi027 ceph-mon[39795]: pgmap v1497: 27 pgs: 27 active+clean; 3.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 888 op/s 2024-05-15T13:10:59.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:58 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:10:59.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:58 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:10:59.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:58 smithi120 ceph-mon[43961]: 1.2 scrub ok 2024-05-15T13:10:59.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:58 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:10:59.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:58 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:10:59.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:58 smithi120 ceph-mon[43961]: osdmap e558: 12 total, 12 up, 11 in 2024-05-15T13:10:59.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:58 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1540613141' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:10:59.031 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:10:58 smithi120 ceph-mon[43961]: pgmap v1497: 27 pgs: 27 active+clean; 3.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 888 op/s 2024-05-15T13:10:59.858 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:10:59.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:10:59.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:10:59.967 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:11:00.005 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:00.030 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:11:00.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:00.033 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:11:00.134 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:11:00.163 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:00.188 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:11:00.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:00.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:11:00.292 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:11:00.367 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:01.151 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:11:01.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:01.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:11:01.259 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:11:01.290 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:01.313 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:11:01.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:01.316 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:11:01.418 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:11:01.468 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:01 smithi027 ceph-mon[39795]: pgmap v1498: 27 pgs: 27 active+clean; 3.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 927 op/s 2024-05-15T13:11:01.468 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:01 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:11:01.468 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:01 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:11:01.471 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:01.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:01 smithi120 ceph-mon[43961]: pgmap v1498: 27 pgs: 27 active+clean; 3.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 927 op/s 2024-05-15T13:11:01.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:01 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:11:01.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:01 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:11:01.547 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:11:01.547 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:11:01.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:01.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:11:01.661 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:11:01.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:01 smithi177 ceph-mon[44303]: pgmap v1498: 27 pgs: 27 active+clean; 3.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 927 op/s 2024-05-15T13:11:01.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:01 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:11:01.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:01 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:11:01.721 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:01.882 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:11:01.882 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T13:11:01.882 INFO:teuthology.orchestra.run.smithi120.stdout: "complaint_time": 30, 2024-05-15T13:11:01.882 INFO:teuthology.orchestra.run.smithi120.stdout: "num_blocked_ops": 0 2024-05-15T13:11:01.882 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:11:01.946 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:11:01.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:01.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:11:02.011 INFO:teuthology.orchestra.run.smithi027.stderr:no valid command found; 10 closest matches: 2024-05-15T13:11:02.012 INFO:teuthology.orchestra.run.smithi027.stderr:0 2024-05-15T13:11:02.012 INFO:teuthology.orchestra.run.smithi027.stderr:1 2024-05-15T13:11:02.012 INFO:teuthology.orchestra.run.smithi027.stderr:2 2024-05-15T13:11:02.012 INFO:teuthology.orchestra.run.smithi027.stderr:abort 2024-05-15T13:11:02.012 INFO:teuthology.orchestra.run.smithi027.stderr:assert 2024-05-15T13:11:02.012 INFO:teuthology.orchestra.run.smithi027.stderr:bench [] [] [] [] 2024-05-15T13:11:02.012 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs debug_inject_read_zeros 2024-05-15T13:11:02.012 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs files list 2024-05-15T13:11:02.012 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs stats 2024-05-15T13:11:02.012 INFO:teuthology.orchestra.run.smithi027.stderr:bluestore allocator dump block 2024-05-15T13:11:02.012 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: invalid command 2024-05-15T13:11:02.053 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:11:02.094 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:02.134 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:11:02.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:02.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:11:02.239 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:11:02.342 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:02.456 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:11:02.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:02.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:11:02.563 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:11:02.592 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:02 smithi177 ceph-mon[44303]: osdmap e559: 12 total, 12 up, 11 in 2024-05-15T13:11:02.592 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:02.611 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:11:02.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:02.615 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:11:02.683 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:02 smithi120 ceph-mon[43961]: osdmap e559: 12 total, 12 up, 11 in 2024-05-15T13:11:02.716 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:11:02.719 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-05-15T13:11:02.766 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:02 smithi027 ceph-mon[39795]: osdmap e559: 12 total, 12 up, 11 in 2024-05-15T13:11:02.767 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:02.803 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:11:02.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:02.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:11:02.913 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:11:02.941 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:02.966 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:11:02.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:02.974 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:11:03.075 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:11:03.128 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:11:03.133 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:03.153 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:11:03.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:03 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:11:03.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:03 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:11:03.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:03 smithi027 ceph-mon[39795]: pgmap v1500: 27 pgs: 27 active+clean; 3.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 642 op/s 2024-05-15T13:11:03.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:03 smithi027 ceph-mon[39795]: osdmap e560: 12 total, 12 up, 11 in 2024-05-15T13:11:03.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:03 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:11:03.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:03 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:11:03.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:03 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:11:03.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:03 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:11:03.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:03 smithi177 ceph-mon[44303]: pgmap v1500: 27 pgs: 27 active+clean; 3.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 642 op/s 2024-05-15T13:11:03.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:03 smithi177 ceph-mon[44303]: osdmap e560: 12 total, 12 up, 11 in 2024-05-15T13:11:03.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:03 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:11:03.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:03 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:11:03.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:03 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:11:03.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:03 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:11:03.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:03 smithi120 ceph-mon[43961]: pgmap v1500: 27 pgs: 27 active+clean; 3.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 642 op/s 2024-05-15T13:11:03.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:03 smithi120 ceph-mon[43961]: osdmap e560: 12 total, 12 up, 11 in 2024-05-15T13:11:03.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:03 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:11:03.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:03 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:11:04.516 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:04 smithi120 ceph-mon[43961]: osdmap e561: 12 total, 12 up, 11 in 2024-05-15T13:11:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:11:04.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:04.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:11:04.591 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T13:11:04.591 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T13:11:04.591 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.11 'trim stale osdmaps' 2024-05-15T13:11:04.654 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:11:04.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:04 smithi177 ceph-mon[44303]: osdmap e561: 12 total, 12 up, 11 in 2024-05-15T13:11:04.690 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:04.699 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:04 smithi027 ceph-mon[39795]: osdmap e561: 12 total, 12 up, 11 in 2024-05-15T13:11:04.871 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:11:04.894 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:11:04.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:04.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:11:05.000 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:11:05.052 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:05.192 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:11:05.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:05.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:11:05.283 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:11:05.283 INFO:teuthology.orchestra.run.smithi120.stdout: "size": 20, 2024-05-15T13:11:05.284 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 600, 2024-05-15T13:11:05.284 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [ 2024-05-15T13:11:05.284 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.284 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:69011 6.d 6:bd873bb1:::benchmark_data_smithi177_122278_object69010:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e528)", 2024-05-15T13:11:05.284 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:08:28.948289+0000", 2024-05-15T13:11:05.284 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 156.34156109899999, 2024-05-15T13:11:05.284 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.14924116600000001, 2024-05-15T13:11:05.284 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:05.284 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:05.284 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:05.284 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:05.284 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:05.284 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 69011 2024-05-15T13:11:05.285 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.285 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:05.285 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.285 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:05.285 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.948289+0000", 2024-05-15T13:11:05.285 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.285 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.285 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.285 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:05.285 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.948289+0000", 2024-05-15T13:11:05.285 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.285 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.285 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.285 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:05.286 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.948290+0000", 2024-05-15T13:11:05.286 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2e-07 2024-05-15T13:11:05.286 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.286 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.286 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:05.286 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.948312+0000", 2024-05-15T13:11:05.286 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2699000000000001e-05 2024-05-15T13:11:05.286 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.286 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.286 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:05.286 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.948317+0000", 2024-05-15T13:11:05.286 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6140000000000002e-06 2024-05-15T13:11:05.287 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.287 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.287 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:05.287 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.948321+0000", 2024-05-15T13:11:05.287 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5360000000000001e-06 2024-05-15T13:11:05.287 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.287 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.287 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:05.287 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.948370+0000", 2024-05-15T13:11:05.287 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9529e-05 2024-05-15T13:11:05.287 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.287 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.287 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:05.287 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.948481+0000", 2024-05-15T13:11:05.287 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000110418 2024-05-15T13:11:05.288 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.288 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.288 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,10", 2024-05-15T13:11:05.288 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.948581+0000", 2024-05-15T13:11:05.288 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000100009 2024-05-15T13:11:05.288 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.288 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.288 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:05.288 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.953053+0000", 2024-05-15T13:11:05.288 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0044724409999999997 2024-05-15T13:11:05.288 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.288 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.288 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.288 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.953379+0000", 2024-05-15T13:11:05.288 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00032635500000000002 2024-05-15T13:11:05.289 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.289 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.289 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.289 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:29.097425+0000", 2024-05-15T13:11:05.289 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.14404602599999999 2024-05-15T13:11:05.289 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.289 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.289 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:05.289 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:29.097488+0000", 2024-05-15T13:11:05.289 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.2512000000000001e-05 2024-05-15T13:11:05.290 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.290 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.290 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:05.290 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:29.097531+0000", 2024-05-15T13:11:05.290 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2607000000000001e-05 2024-05-15T13:11:05.290 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.290 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:05.290 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.290 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.290 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.290 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:69013 6.d 6:b142e70c:::benchmark_data_smithi177_122278_object69012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e528)", 2024-05-15T13:11:05.290 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:08:28.949091+0000", 2024-05-15T13:11:05.290 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 156.34075972799999, 2024-05-15T13:11:05.290 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.14854789700000001, 2024-05-15T13:11:05.291 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:05.291 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:05.291 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:05.291 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:05.291 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:05.291 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 69013 2024-05-15T13:11:05.291 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.291 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:05.291 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.291 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:05.291 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.949091+0000", 2024-05-15T13:11:05.291 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.291 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.291 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.291 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:05.292 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.949091+0000", 2024-05-15T13:11:05.292 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.292 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.292 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.292 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:05.292 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.949092+0000", 2024-05-15T13:11:05.292 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8400000000000003e-07 2024-05-15T13:11:05.292 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.292 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.292 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:05.292 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.949153+0000", 2024-05-15T13:11:05.292 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1929999999999998e-05 2024-05-15T13:11:05.292 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.292 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.292 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:05.293 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.949165+0000", 2024-05-15T13:11:05.293 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1144999999999999e-05 2024-05-15T13:11:05.293 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.293 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.293 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:05.293 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.949172+0000", 2024-05-15T13:11:05.293 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4019999999999996e-06 2024-05-15T13:11:05.293 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.293 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.293 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:05.293 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.950913+0000", 2024-05-15T13:11:05.293 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001740828 2024-05-15T13:11:05.293 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.293 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.293 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:05.294 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.951046+0000", 2024-05-15T13:11:05.294 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013318499999999999 2024-05-15T13:11:05.294 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.294 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.294 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,10", 2024-05-15T13:11:05.294 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.951230+0000", 2024-05-15T13:11:05.294 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00018363599999999999 2024-05-15T13:11:05.294 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.294 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.294 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:05.294 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.953157+0000", 2024-05-15T13:11:05.294 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0019269739999999999 2024-05-15T13:11:05.294 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.295 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.295 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.295 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.956000+0000", 2024-05-15T13:11:05.295 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0028437670000000001 2024-05-15T13:11:05.295 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.295 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.295 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.295 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:29.097576+0000", 2024-05-15T13:11:05.295 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.141575276 2024-05-15T13:11:05.295 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.295 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.295 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:05.295 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:29.097612+0000", 2024-05-15T13:11:05.295 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.6350000000000003e-05 2024-05-15T13:11:05.296 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.296 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.296 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:05.296 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:29.097639+0000", 2024-05-15T13:11:05.296 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.6619999999999999e-05 2024-05-15T13:11:05.296 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.296 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:05.296 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.296 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.296 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.296 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:69031 6.d 6:bfa11285:::benchmark_data_smithi177_122278_object69030:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e528)", 2024-05-15T13:11:05.296 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:08:28.964307+0000", 2024-05-15T13:11:05.296 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 156.32554320700001, 2024-05-15T13:11:05.296 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.13347766999999999, 2024-05-15T13:11:05.297 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:05.297 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:05.297 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:05.297 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:05.297 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:05.297 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 69031 2024-05-15T13:11:05.297 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.297 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:05.297 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.297 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:05.297 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.964307+0000", 2024-05-15T13:11:05.297 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.297 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.298 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.298 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:05.298 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.964307+0000", 2024-05-15T13:11:05.298 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.298 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.298 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.298 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:05.298 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.964308+0000", 2024-05-15T13:11:05.298 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.2399999999999996e-07 2024-05-15T13:11:05.298 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.298 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.298 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:05.298 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.964395+0000", 2024-05-15T13:11:05.299 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7077999999999995e-05 2024-05-15T13:11:05.299 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.299 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.299 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:05.299 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.964411+0000", 2024-05-15T13:11:05.299 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5362e-05 2024-05-15T13:11:05.299 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.299 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.299 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:05.299 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.964420+0000", 2024-05-15T13:11:05.299 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.9970000000000004e-06 2024-05-15T13:11:05.300 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.300 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.300 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:05.300 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.964476+0000", 2024-05-15T13:11:05.300 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6558999999999998e-05 2024-05-15T13:11:05.300 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.300 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.300 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:05.300 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.964604+0000", 2024-05-15T13:11:05.300 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012801199999999999 2024-05-15T13:11:05.300 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.300 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.300 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,10", 2024-05-15T13:11:05.300 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.964745+0000", 2024-05-15T13:11:05.301 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000140662 2024-05-15T13:11:05.301 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.301 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.301 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:05.301 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.966871+0000", 2024-05-15T13:11:05.301 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0021260329999999998 2024-05-15T13:11:05.301 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.301 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.301 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.301 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.967339+0000", 2024-05-15T13:11:05.301 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00046783099999999999 2024-05-15T13:11:05.301 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.301 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.301 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.302 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:29.097676+0000", 2024-05-15T13:11:05.302 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.130336958 2024-05-15T13:11:05.302 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.302 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.302 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:05.302 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:29.097734+0000", 2024-05-15T13:11:05.302 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8452000000000002e-05 2024-05-15T13:11:05.302 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.302 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.302 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:05.302 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:29.097785+0000", 2024-05-15T13:11:05.302 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0801999999999999e-05 2024-05-15T13:11:05.302 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.303 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:05.303 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.303 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.303 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.303 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:69039 6.d 6:bae456ba:::benchmark_data_smithi177_122278_object69038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e528)", 2024-05-15T13:11:05.303 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:08:28.981320+0000", 2024-05-15T13:11:05.303 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 156.308530695, 2024-05-15T13:11:05.303 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.118233565, 2024-05-15T13:11:05.303 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:05.303 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:05.303 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:05.303 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:05.303 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:05.303 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 69039 2024-05-15T13:11:05.303 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.304 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:05.304 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.304 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:05.304 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.981320+0000", 2024-05-15T13:11:05.304 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.304 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.304 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.304 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:05.304 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.981320+0000", 2024-05-15T13:11:05.304 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.304 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.304 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.304 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:05.304 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.981321+0000", 2024-05-15T13:11:05.304 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.271e-06 2024-05-15T13:11:05.305 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.305 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.305 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:05.305 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.981377+0000", 2024-05-15T13:11:05.305 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5711999999999998e-05 2024-05-15T13:11:05.305 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.305 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.305 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:05.305 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.981390+0000", 2024-05-15T13:11:05.305 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3711e-05 2024-05-15T13:11:05.305 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.305 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.305 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:05.305 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.981397+0000", 2024-05-15T13:11:05.305 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7390000000000002e-06 2024-05-15T13:11:05.306 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.306 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.306 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:05.306 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.981443+0000", 2024-05-15T13:11:05.306 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.5550000000000003e-05 2024-05-15T13:11:05.306 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.306 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.306 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:05.306 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.981539+0000", 2024-05-15T13:11:05.306 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.6565999999999997e-05 2024-05-15T13:11:05.306 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.306 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.306 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,10", 2024-05-15T13:11:05.306 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.981647+0000", 2024-05-15T13:11:05.307 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000108139 2024-05-15T13:11:05.307 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.307 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.307 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.307 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.983585+0000", 2024-05-15T13:11:05.307 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0019371779999999999 2024-05-15T13:11:05.307 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.307 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.307 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:05.307 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:28.983639+0000", 2024-05-15T13:11:05.307 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.4329000000000001e-05 2024-05-15T13:11:05.307 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.307 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.307 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.307 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:29.099461+0000", 2024-05-15T13:11:05.307 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.115822067 2024-05-15T13:11:05.308 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.308 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.308 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:05.308 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:29.099512+0000", 2024-05-15T13:11:05.308 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0523999999999999e-05 2024-05-15T13:11:05.308 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.308 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.308 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:05.308 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:29.099553+0000", 2024-05-15T13:11:05.308 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.1779000000000001e-05 2024-05-15T13:11:05.308 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.308 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:05.308 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.308 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.308 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.309 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:104059 6.a 6:5eae2ef6:::benchmark_data_smithi177_122278_object104058:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e536)", 2024-05-15T13:11:05.309 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:08:57.434658+0000", 2024-05-15T13:11:05.309 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 127.85519222400001, 2024-05-15T13:11:05.309 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.13181632900000001, 2024-05-15T13:11:05.309 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:05.309 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:05.309 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:05.309 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:05.309 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:05.309 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 104059 2024-05-15T13:11:05.309 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.309 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:05.309 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.309 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:05.309 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.434658+0000", 2024-05-15T13:11:05.310 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.310 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.310 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.310 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:05.310 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.434658+0000", 2024-05-15T13:11:05.310 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.310 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.310 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.310 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:05.311 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.434660+0000", 2024-05-15T13:11:05.311 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3239999999999999e-06 2024-05-15T13:11:05.311 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.311 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.311 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:05.311 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.434704+0000", 2024-05-15T13:11:05.311 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.422e-05 2024-05-15T13:11:05.311 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.311 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.311 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:05.311 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.434719+0000", 2024-05-15T13:11:05.311 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5569999999999998e-05 2024-05-15T13:11:05.311 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.311 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.312 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:05.312 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.434738+0000", 2024-05-15T13:11:05.312 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.8844000000000001e-05 2024-05-15T13:11:05.312 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.312 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.312 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:05.312 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.434784+0000", 2024-05-15T13:11:05.312 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.5908999999999997e-05 2024-05-15T13:11:05.312 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.312 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.312 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:05.312 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.434878+0000", 2024-05-15T13:11:05.312 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.3653000000000006e-05 2024-05-15T13:11:05.312 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.312 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.313 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,6", 2024-05-15T13:11:05.313 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.434967+0000", 2024-05-15T13:11:05.313 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.9577e-05 2024-05-15T13:11:05.313 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.313 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.313 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:05.313 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.438389+0000", 2024-05-15T13:11:05.313 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0034221289999999999 2024-05-15T13:11:05.313 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.313 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.313 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.313 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.438635+0000", 2024-05-15T13:11:05.313 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00024603400000000001 2024-05-15T13:11:05.313 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.313 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.314 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.314 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.566364+0000", 2024-05-15T13:11:05.314 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.12772823699999999 2024-05-15T13:11:05.314 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.314 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.314 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:05.314 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.566434+0000", 2024-05-15T13:11:05.314 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9851000000000006e-05 2024-05-15T13:11:05.314 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.314 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.314 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:05.314 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.566475+0000", 2024-05-15T13:11:05.314 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0980999999999999e-05 2024-05-15T13:11:05.314 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.315 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:05.315 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.315 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.315 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.315 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:104066 6.a 6:510fd49e:::benchmark_data_smithi177_122278_object104065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e536)", 2024-05-15T13:11:05.315 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:08:57.458776+0000", 2024-05-15T13:11:05.315 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 127.83107456800001, 2024-05-15T13:11:05.315 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.10909133999999999, 2024-05-15T13:11:05.315 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:05.315 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:05.315 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:05.315 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:05.315 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:05.315 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 104066 2024-05-15T13:11:05.315 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.316 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:05.316 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.316 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:05.316 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.458776+0000", 2024-05-15T13:11:05.316 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.316 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.316 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.316 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:05.316 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.458776+0000", 2024-05-15T13:11:05.316 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.316 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.316 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.316 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:05.316 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.458777+0000", 2024-05-15T13:11:05.316 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5510000000000001e-06 2024-05-15T13:11:05.317 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.317 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.317 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:05.317 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.458836+0000", 2024-05-15T13:11:05.317 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8255999999999999e-05 2024-05-15T13:11:05.317 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.317 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.317 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:05.317 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.458847+0000", 2024-05-15T13:11:05.317 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0963e-05 2024-05-15T13:11:05.317 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.317 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.317 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:05.318 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.458853+0000", 2024-05-15T13:11:05.318 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4289999999999998e-06 2024-05-15T13:11:05.318 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.318 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.318 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:05.318 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.458907+0000", 2024-05-15T13:11:05.318 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.4191999999999999e-05 2024-05-15T13:11:05.318 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.318 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.318 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:05.318 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.459043+0000", 2024-05-15T13:11:05.318 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013549300000000001 2024-05-15T13:11:05.319 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.319 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.319 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,6", 2024-05-15T13:11:05.319 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.459194+0000", 2024-05-15T13:11:05.319 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015136899999999999 2024-05-15T13:11:05.319 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.319 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.319 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:05.319 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.462147+0000", 2024-05-15T13:11:05.319 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0029530820000000001 2024-05-15T13:11:05.319 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.319 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.319 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.319 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.462198+0000", 2024-05-15T13:11:05.319 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0664000000000001e-05 2024-05-15T13:11:05.320 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.320 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.320 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.320 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.567778+0000", 2024-05-15T13:11:05.320 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.10558023599999999 2024-05-15T13:11:05.320 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.320 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.320 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:05.320 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.567829+0000", 2024-05-15T13:11:05.320 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1110999999999999e-05 2024-05-15T13:11:05.320 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.320 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.321 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:05.321 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:08:57.567867+0000", 2024-05-15T13:11:05.321 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7994000000000002e-05 2024-05-15T13:11:05.321 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.321 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:05.321 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.321 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.321 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.321 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:113886 6.a 6:5ce6445f:::benchmark_data_smithi177_122278_object113885:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e537)", 2024-05-15T13:11:05.321 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:09:07.581901+0000", 2024-05-15T13:11:05.321 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 117.707949832, 2024-05-15T13:11:05.321 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.14446249, 2024-05-15T13:11:05.321 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:05.321 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:05.321 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:05.322 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:05.322 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:05.322 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 113886 2024-05-15T13:11:05.322 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.322 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:05.322 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.322 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:05.322 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.581901+0000", 2024-05-15T13:11:05.322 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.322 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.322 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.322 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:05.322 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.581901+0000", 2024-05-15T13:11:05.323 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.323 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.323 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.323 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:05.323 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.581902+0000", 2024-05-15T13:11:05.323 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.322e-06 2024-05-15T13:11:05.323 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.323 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.323 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:05.323 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.582002+0000", 2024-05-15T13:11:05.323 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000100065 2024-05-15T13:11:05.323 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.323 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.323 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:05.323 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.582014+0000", 2024-05-15T13:11:05.323 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1929e-05 2024-05-15T13:11:05.324 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.324 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.324 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:05.324 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.582021+0000", 2024-05-15T13:11:05.324 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8689999999999998e-06 2024-05-15T13:11:05.324 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.324 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.324 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:05.324 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.582071+0000", 2024-05-15T13:11:05.324 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0253999999999997e-05 2024-05-15T13:11:05.324 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.324 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.324 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:05.324 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.582209+0000", 2024-05-15T13:11:05.324 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013803700000000001 2024-05-15T13:11:05.325 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.325 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.325 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,6", 2024-05-15T13:11:05.325 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.582359+0000", 2024-05-15T13:11:05.325 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014985399999999999 2024-05-15T13:11:05.325 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.325 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.325 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.325 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.590839+0000", 2024-05-15T13:11:05.325 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0084800780000000003 2024-05-15T13:11:05.325 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.325 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.325 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:05.326 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.590854+0000", 2024-05-15T13:11:05.326 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4909000000000001e-05 2024-05-15T13:11:05.326 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.326 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.326 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.326 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.726264+0000", 2024-05-15T13:11:05.326 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.13540988500000001 2024-05-15T13:11:05.326 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.326 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.326 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:05.326 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.726304+0000", 2024-05-15T13:11:05.326 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.9867000000000003e-05 2024-05-15T13:11:05.326 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.327 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.327 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:05.327 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.726363+0000", 2024-05-15T13:11:05.327 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9420999999999999e-05 2024-05-15T13:11:05.327 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.327 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:05.327 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.327 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.327 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.327 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:113888 6.a 6:5bb9053f:::benchmark_data_smithi177_122278_object113887:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e537)", 2024-05-15T13:11:05.327 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:09:07.582036+0000", 2024-05-15T13:11:05.327 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 117.707814293, 2024-05-15T13:11:05.327 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.144495545, 2024-05-15T13:11:05.327 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:05.328 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:05.328 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:05.328 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:05.328 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:05.328 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 113888 2024-05-15T13:11:05.328 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.328 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:05.328 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.328 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:05.328 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.582036+0000", 2024-05-15T13:11:05.328 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.328 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.328 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.328 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:05.328 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.582036+0000", 2024-05-15T13:11:05.328 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.329 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.329 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.329 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:05.329 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.582037+0000", 2024-05-15T13:11:05.329 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1999999999999999e-07 2024-05-15T13:11:05.329 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.329 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.329 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:05.329 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.582078+0000", 2024-05-15T13:11:05.329 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0865000000000003e-05 2024-05-15T13:11:05.329 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.329 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.329 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:05.329 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.582088+0000", 2024-05-15T13:11:05.330 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0284e-05 2024-05-15T13:11:05.330 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.330 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.330 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:05.330 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.582092+0000", 2024-05-15T13:11:05.330 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.844e-06 2024-05-15T13:11:05.330 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.330 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.330 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:05.330 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.587151+0000", 2024-05-15T13:11:05.330 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0050594619999999998 2024-05-15T13:11:05.331 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.331 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.331 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:05.331 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.587293+0000", 2024-05-15T13:11:05.331 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014159900000000001 2024-05-15T13:11:05.331 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.331 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.331 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,6", 2024-05-15T13:11:05.331 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.587377+0000", 2024-05-15T13:11:05.331 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.3763000000000004e-05 2024-05-15T13:11:05.331 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.331 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.332 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:05.332 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.593899+0000", 2024-05-15T13:11:05.332 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0065219780000000003 2024-05-15T13:11:05.332 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.332 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.332 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.332 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.594151+0000", 2024-05-15T13:11:05.332 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00025212999999999999 2024-05-15T13:11:05.332 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.332 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.332 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.332 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.726428+0000", 2024-05-15T13:11:05.332 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.13227705200000001 2024-05-15T13:11:05.332 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.333 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.333 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:05.333 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.726487+0000", 2024-05-15T13:11:05.333 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8786999999999998e-05 2024-05-15T13:11:05.333 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.333 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.333 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:05.333 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.726532+0000", 2024-05-15T13:11:05.333 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.5160999999999997e-05 2024-05-15T13:11:05.333 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.333 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:05.333 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.333 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.333 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.334 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:113930 6.a 6:5e83a33a:::benchmark_data_smithi177_122278_object113929:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e537)", 2024-05-15T13:11:05.334 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:09:07.630261+0000", 2024-05-15T13:11:05.334 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 117.659589806, 2024-05-15T13:11:05.334 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.096387044000000005, 2024-05-15T13:11:05.334 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:05.334 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:05.334 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:05.334 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:05.334 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:05.334 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 113930 2024-05-15T13:11:05.334 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.334 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:05.334 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.334 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:05.334 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.630261+0000", 2024-05-15T13:11:05.335 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.335 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.335 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.335 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:05.335 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.630261+0000", 2024-05-15T13:11:05.335 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.335 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.335 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.335 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:05.335 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.630262+0000", 2024-05-15T13:11:05.335 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0589999999999999e-06 2024-05-15T13:11:05.335 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.335 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.335 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:05.336 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.630467+0000", 2024-05-15T13:11:05.336 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000204895 2024-05-15T13:11:05.336 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.336 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.336 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:05.336 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.630477+0000", 2024-05-15T13:11:05.336 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.024e-05 2024-05-15T13:11:05.336 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.336 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.336 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:05.336 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.630482+0000", 2024-05-15T13:11:05.336 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5260000000000001e-06 2024-05-15T13:11:05.336 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.337 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.337 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:05.337 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.630528+0000", 2024-05-15T13:11:05.337 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6094000000000003e-05 2024-05-15T13:11:05.337 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.337 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.337 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:05.337 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.630639+0000", 2024-05-15T13:11:05.337 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000110275 2024-05-15T13:11:05.337 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.337 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.337 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,6", 2024-05-15T13:11:05.337 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.630751+0000", 2024-05-15T13:11:05.337 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00011178100000000001 2024-05-15T13:11:05.338 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.338 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.338 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:05.338 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.634450+0000", 2024-05-15T13:11:05.338 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0036992029999999999 2024-05-15T13:11:05.338 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.338 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.338 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.338 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.638085+0000", 2024-05-15T13:11:05.338 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0036357059999999998 2024-05-15T13:11:05.338 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.338 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.338 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.338 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.726575+0000", 2024-05-15T13:11:05.338 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.088489569000000004 2024-05-15T13:11:05.339 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.339 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.339 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:05.339 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.726608+0000", 2024-05-15T13:11:05.339 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.2826e-05 2024-05-15T13:11:05.339 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.339 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.339 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:05.339 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:07.726648+0000", 2024-05-15T13:11:05.339 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.9870000000000003e-05 2024-05-15T13:11:05.339 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.339 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:05.339 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.339 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.339 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.340 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:142544 6.a 6:5faf890c:::benchmark_data_smithi177_122278_object12173:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e542)", 2024-05-15T13:11:05.340 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:09:37.373528+0000", 2024-05-15T13:11:05.340 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 87.916322020999999, 2024-05-15T13:11:05.340 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.18855435400000001, 2024-05-15T13:11:05.340 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:05.340 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:05.340 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:05.340 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:05.340 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:05.340 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 142544 2024-05-15T13:11:05.340 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.340 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:05.340 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.340 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:05.340 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.373528+0000", 2024-05-15T13:11:05.340 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.341 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.341 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.341 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:05.341 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.373528+0000", 2024-05-15T13:11:05.341 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.341 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.341 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.341 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:05.342 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.373530+0000", 2024-05-15T13:11:05.342 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1030000000000001e-06 2024-05-15T13:11:05.342 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.342 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.342 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:05.342 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.373535+0000", 2024-05-15T13:11:05.342 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0429999999999998e-06 2024-05-15T13:11:05.342 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.342 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.342 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:05.342 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.373546+0000", 2024-05-15T13:11:05.342 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.112e-05 2024-05-15T13:11:05.342 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.343 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.343 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:05.343 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.373552+0000", 2024-05-15T13:11:05.343 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.5769999999999999e-06 2024-05-15T13:11:05.343 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.343 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.343 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:05.343 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.373598+0000", 2024-05-15T13:11:05.343 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.5541999999999999e-05 2024-05-15T13:11:05.343 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.343 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.343 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:05.343 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.373679+0000", 2024-05-15T13:11:05.343 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1527999999999996e-05 2024-05-15T13:11:05.344 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.344 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.344 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,6", 2024-05-15T13:11:05.344 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.373754+0000", 2024-05-15T13:11:05.344 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4635000000000006e-05 2024-05-15T13:11:05.344 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.344 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.344 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:05.344 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.381508+0000", 2024-05-15T13:11:05.344 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0077543869999999997 2024-05-15T13:11:05.344 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.344 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.344 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.344 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.381580+0000", 2024-05-15T13:11:05.345 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.1662999999999995e-05 2024-05-15T13:11:05.345 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.345 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.345 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.345 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.562016+0000", 2024-05-15T13:11:05.345 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.18043641999999999 2024-05-15T13:11:05.345 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.345 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.345 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:05.345 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.562051+0000", 2024-05-15T13:11:05.345 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.4106000000000002e-05 2024-05-15T13:11:05.345 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.345 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.345 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:05.345 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.562083+0000", 2024-05-15T13:11:05.345 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.2230000000000001e-05 2024-05-15T13:11:05.346 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.346 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:05.346 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.346 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.346 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.346 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:142547 6.a 6:51e7d83c:::benchmark_data_smithi177_122278_object12176:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e542)", 2024-05-15T13:11:05.346 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:09:37.375179+0000", 2024-05-15T13:11:05.346 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 87.914671928000004, 2024-05-15T13:11:05.346 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.19132473899999999, 2024-05-15T13:11:05.346 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:05.346 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:05.346 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:05.346 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:05.346 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:05.347 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 142547 2024-05-15T13:11:05.347 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.347 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:05.347 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.347 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:05.347 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.375179+0000", 2024-05-15T13:11:05.347 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.347 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.347 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.347 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:05.347 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.375179+0000", 2024-05-15T13:11:05.347 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.347 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.347 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.347 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:05.347 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.375179+0000", 2024-05-15T13:11:05.348 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.3500000000000005e-07 2024-05-15T13:11:05.348 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.348 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.348 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:05.348 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.375184+0000", 2024-05-15T13:11:05.348 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8319999999999996e-06 2024-05-15T13:11:05.348 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.348 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.348 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:05.348 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.375199+0000", 2024-05-15T13:11:05.348 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4426e-05 2024-05-15T13:11:05.348 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.348 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.348 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:05.348 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.375206+0000", 2024-05-15T13:11:05.348 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4959999999999999e-06 2024-05-15T13:11:05.349 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.349 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.349 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:05.349 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.377081+0000", 2024-05-15T13:11:05.349 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001874541 2024-05-15T13:11:05.349 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.349 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.349 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:05.349 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.377194+0000", 2024-05-15T13:11:05.349 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000113187 2024-05-15T13:11:05.349 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.349 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.349 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,6", 2024-05-15T13:11:05.349 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.377302+0000", 2024-05-15T13:11:05.349 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00010804899999999999 2024-05-15T13:11:05.350 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.350 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.350 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.350 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.381632+0000", 2024-05-15T13:11:05.350 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0043297309999999999 2024-05-15T13:11:05.350 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.350 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.350 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:05.350 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.381717+0000", 2024-05-15T13:11:05.350 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.5001999999999997e-05 2024-05-15T13:11:05.350 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.350 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.350 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.350 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.566385+0000", 2024-05-15T13:11:05.351 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.18466834200000001 2024-05-15T13:11:05.351 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.351 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.351 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:05.351 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.566445+0000", 2024-05-15T13:11:05.351 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9718e-05 2024-05-15T13:11:05.351 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.351 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.351 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:05.351 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.566503+0000", 2024-05-15T13:11:05.351 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8579999999999998e-05 2024-05-15T13:11:05.351 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.351 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:05.351 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.351 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.352 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.352 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:142557 6.a 6:56725581:::benchmark_data_smithi177_122278_object12186:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e542)", 2024-05-15T13:11:05.352 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:09:37.392615+0000", 2024-05-15T13:11:05.352 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 87.897235899999998, 2024-05-15T13:11:05.352 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.17402535899999999, 2024-05-15T13:11:05.352 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:05.352 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:05.352 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:05.352 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:05.352 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:05.352 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 142557 2024-05-15T13:11:05.353 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.353 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:05.353 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.353 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:05.353 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.392615+0000", 2024-05-15T13:11:05.353 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.353 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.353 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.353 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:05.353 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.392615+0000", 2024-05-15T13:11:05.353 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.353 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.353 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.354 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:05.354 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.392616+0000", 2024-05-15T13:11:05.354 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.184e-06 2024-05-15T13:11:05.354 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.354 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.354 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:05.354 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.392621+0000", 2024-05-15T13:11:05.354 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1100000000000002e-06 2024-05-15T13:11:05.354 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.354 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.355 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:05.355 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.392634+0000", 2024-05-15T13:11:05.355 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3611999999999999e-05 2024-05-15T13:11:05.355 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.355 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.355 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:05.355 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.392642+0000", 2024-05-15T13:11:05.355 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0899999999999999e-06 2024-05-15T13:11:05.355 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.355 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.355 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:05.355 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.392688+0000", 2024-05-15T13:11:05.355 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6409999999999998e-05 2024-05-15T13:11:05.355 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.355 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.356 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:05.356 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.392784+0000", 2024-05-15T13:11:05.356 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.6037000000000002e-05 2024-05-15T13:11:05.356 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.356 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.356 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,6", 2024-05-15T13:11:05.356 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.392854+0000", 2024-05-15T13:11:05.356 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9610000000000006e-05 2024-05-15T13:11:05.356 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.356 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.356 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:05.356 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.396283+0000", 2024-05-15T13:11:05.356 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0034293940000000001 2024-05-15T13:11:05.356 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.357 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.357 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.357 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.396444+0000", 2024-05-15T13:11:05.357 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00016054700000000001 2024-05-15T13:11:05.357 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.357 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.357 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.357 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.566550+0000", 2024-05-15T13:11:05.357 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.17010645899999999 2024-05-15T13:11:05.357 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.357 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.357 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:05.357 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.566600+0000", 2024-05-15T13:11:05.357 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9968e-05 2024-05-15T13:11:05.358 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.358 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.358 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:05.358 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:37.566640+0000", 2024-05-15T13:11:05.358 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.9938000000000003e-05 2024-05-15T13:11:05.358 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.358 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:05.358 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.358 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.358 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.358 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:190871 6.d 6:b36a8c8b:::benchmark_data_smithi177_122278_object60500:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-05-15T13:11:05.358 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:10:34.151419+0000", 2024-05-15T13:11:05.359 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 31.138431339, 2024-05-15T13:11:05.359 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.105936819, 2024-05-15T13:11:05.359 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:05.359 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:05.359 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:05.359 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:05.359 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:05.359 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 190871 2024-05-15T13:11:05.359 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.359 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:05.359 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.359 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:05.359 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.151419+0000", 2024-05-15T13:11:05.359 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.359 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.360 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.360 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:05.360 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.151419+0000", 2024-05-15T13:11:05.360 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.360 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.360 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.360 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:05.360 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.151420+0000", 2024-05-15T13:11:05.360 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2099999999999996e-07 2024-05-15T13:11:05.360 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.360 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.360 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:05.360 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.151424+0000", 2024-05-15T13:11:05.360 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4649999999999996e-06 2024-05-15T13:11:05.360 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.361 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.361 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:05.361 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.151435+0000", 2024-05-15T13:11:05.361 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0603e-05 2024-05-15T13:11:05.361 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.361 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.361 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:05.361 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.151441+0000", 2024-05-15T13:11:05.361 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8270000000000003e-06 2024-05-15T13:11:05.361 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.361 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.361 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:05.361 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.154387+0000", 2024-05-15T13:11:05.362 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0029462329999999999 2024-05-15T13:11:05.362 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.362 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.362 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:05.362 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.154503+0000", 2024-05-15T13:11:05.362 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000115841 2024-05-15T13:11:05.362 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.362 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.363 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,10", 2024-05-15T13:11:05.363 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.154578+0000", 2024-05-15T13:11:05.363 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4863999999999993e-05 2024-05-15T13:11:05.363 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.363 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.363 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:05.363 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.164028+0000", 2024-05-15T13:11:05.363 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0094503499999999997 2024-05-15T13:11:05.363 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.363 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.363 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.363 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.164166+0000", 2024-05-15T13:11:05.363 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013775899999999999 2024-05-15T13:11:05.363 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.364 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.364 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.364 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.257289+0000", 2024-05-15T13:11:05.364 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.093122785 2024-05-15T13:11:05.364 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.364 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.364 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:05.364 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.257327+0000", 2024-05-15T13:11:05.364 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.8300000000000003e-05 2024-05-15T13:11:05.364 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.364 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.364 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:05.364 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.257356+0000", 2024-05-15T13:11:05.364 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.9071e-05 2024-05-15T13:11:05.365 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.365 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:05.365 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.365 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.365 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.365 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:190872 6.d 6:bf8830a8:::benchmark_data_smithi177_122278_object60501:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-05-15T13:11:05.365 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:10:34.151455+0000", 2024-05-15T13:11:05.365 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 31.138395383999999, 2024-05-15T13:11:05.365 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.121407371, 2024-05-15T13:11:05.365 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:05.365 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:05.365 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:05.365 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:05.366 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:05.366 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 190872 2024-05-15T13:11:05.366 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.366 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:05.366 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.366 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:05.366 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.151455+0000", 2024-05-15T13:11:05.366 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.366 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.366 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.366 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:05.366 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.151455+0000", 2024-05-15T13:11:05.366 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.366 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.367 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.367 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:05.367 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.151456+0000", 2024-05-15T13:11:05.367 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7399999999999998e-07 2024-05-15T13:11:05.367 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.367 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.367 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:05.367 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.151458+0000", 2024-05-15T13:11:05.367 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.7989999999999998e-06 2024-05-15T13:11:05.367 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.367 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.367 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:05.367 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.151465+0000", 2024-05-15T13:11:05.367 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6540000000000002e-06 2024-05-15T13:11:05.367 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.367 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.368 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:05.368 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.151470+0000", 2024-05-15T13:11:05.368 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.4070000000000001e-06 2024-05-15T13:11:05.368 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.368 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.368 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:05.368 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.159338+0000", 2024-05-15T13:11:05.368 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0078674399999999999 2024-05-15T13:11:05.368 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.368 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.368 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:05.368 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.159459+0000", 2024-05-15T13:11:05.368 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012152 2024-05-15T13:11:05.368 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.369 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.369 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,10", 2024-05-15T13:11:05.369 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.159576+0000", 2024-05-15T13:11:05.369 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000116647 2024-05-15T13:11:05.369 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.369 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.369 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.369 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.168815+0000", 2024-05-15T13:11:05.369 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0092392050000000003 2024-05-15T13:11:05.369 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.369 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.369 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:05.369 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.174005+0000", 2024-05-15T13:11:05.369 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0051902429999999998 2024-05-15T13:11:05.369 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.370 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.370 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.370 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.272800+0000", 2024-05-15T13:11:05.370 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.098794227999999998 2024-05-15T13:11:05.370 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.370 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.370 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:05.370 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.272833+0000", 2024-05-15T13:11:05.370 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.3206e-05 2024-05-15T13:11:05.370 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.370 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.370 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:05.370 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.272862+0000", 2024-05-15T13:11:05.370 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.9547999999999999e-05 2024-05-15T13:11:05.371 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.371 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:05.371 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.371 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.371 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.371 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:190880 6.d 6:b86b4027:::benchmark_data_smithi177_122278_object60509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-05-15T13:11:05.371 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:10:34.155699+0000", 2024-05-15T13:11:05.371 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 31.134151215999999, 2024-05-15T13:11:05.371 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.117302689, 2024-05-15T13:11:05.371 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:05.371 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:05.371 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:05.371 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:05.371 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:05.372 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 190880 2024-05-15T13:11:05.372 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.372 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:05.372 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.372 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:05.372 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.155699+0000", 2024-05-15T13:11:05.372 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.372 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.372 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.372 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:05.372 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.155699+0000", 2024-05-15T13:11:05.372 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.373 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.373 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.373 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:05.373 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.155700+0000", 2024-05-15T13:11:05.373 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.77e-07 2024-05-15T13:11:05.373 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.373 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.373 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:05.373 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.155702+0000", 2024-05-15T13:11:05.373 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.3319999999999999e-06 2024-05-15T13:11:05.374 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.374 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.374 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:05.374 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.155708+0000", 2024-05-15T13:11:05.374 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9529999999999996e-06 2024-05-15T13:11:05.374 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.374 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.374 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:05.374 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.155713+0000", 2024-05-15T13:11:05.374 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0340000000000004e-06 2024-05-15T13:11:05.374 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.374 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.374 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:05.374 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.164298+0000", 2024-05-15T13:11:05.374 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0085855370000000007 2024-05-15T13:11:05.375 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.375 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.375 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:05.375 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.164372+0000", 2024-05-15T13:11:05.375 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3359e-05 2024-05-15T13:11:05.375 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.375 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.375 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,10", 2024-05-15T13:11:05.375 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.164442+0000", 2024-05-15T13:11:05.375 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0650999999999998e-05 2024-05-15T13:11:05.375 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.375 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.375 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.375 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.173951+0000", 2024-05-15T13:11:05.375 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0095084200000000001 2024-05-15T13:11:05.375 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.376 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.376 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:05.376 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.174016+0000", 2024-05-15T13:11:05.376 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4958999999999999e-05 2024-05-15T13:11:05.376 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.376 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.376 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.376 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.272904+0000", 2024-05-15T13:11:05.376 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.098888113 2024-05-15T13:11:05.376 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.376 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.376 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:05.376 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.272958+0000", 2024-05-15T13:11:05.376 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3730000000000002e-05 2024-05-15T13:11:05.376 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.377 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.377 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:05.377 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.273002+0000", 2024-05-15T13:11:05.377 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4224000000000002e-05 2024-05-15T13:11:05.377 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.377 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:05.377 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.377 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.377 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.377 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:190887 6.d 6:b184c57b:::benchmark_data_smithi177_122278_object60516:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-05-15T13:11:05.377 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:10:34.167737+0000", 2024-05-15T13:11:05.378 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 31.122113628000001, 2024-05-15T13:11:05.378 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.115727253, 2024-05-15T13:11:05.378 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:05.378 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:05.378 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:05.378 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:05.378 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:05.378 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 190887 2024-05-15T13:11:05.378 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.378 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:05.378 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.378 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:05.378 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.167737+0000", 2024-05-15T13:11:05.378 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.379 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.379 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.379 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:05.379 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.167737+0000", 2024-05-15T13:11:05.379 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.379 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.379 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.379 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:05.379 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.167738+0000", 2024-05-15T13:11:05.379 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6190000000000001e-06 2024-05-15T13:11:05.379 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.379 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.379 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:05.379 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.167745+0000", 2024-05-15T13:11:05.380 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9290000000000001e-06 2024-05-15T13:11:05.380 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.380 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.380 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:05.380 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.167760+0000", 2024-05-15T13:11:05.380 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4541e-05 2024-05-15T13:11:05.380 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.380 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.380 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:05.380 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.167769+0000", 2024-05-15T13:11:05.380 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7579999999999998e-06 2024-05-15T13:11:05.380 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.380 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.380 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:05.380 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.168856+0000", 2024-05-15T13:11:05.380 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001087293 2024-05-15T13:11:05.381 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.381 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.381 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:05.381 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.168982+0000", 2024-05-15T13:11:05.381 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012606600000000001 2024-05-15T13:11:05.381 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.381 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.381 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,10", 2024-05-15T13:11:05.381 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.169059+0000", 2024-05-15T13:11:05.381 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.6953999999999995e-05 2024-05-15T13:11:05.381 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.381 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.381 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.381 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.173994+0000", 2024-05-15T13:11:05.381 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0049346329999999999 2024-05-15T13:11:05.382 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.382 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.382 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:05.382 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.174226+0000", 2024-05-15T13:11:05.382 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000232678 2024-05-15T13:11:05.382 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.382 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.382 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.382 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.283358+0000", 2024-05-15T13:11:05.382 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.109131333 2024-05-15T13:11:05.382 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.382 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.382 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:05.382 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.283415+0000", 2024-05-15T13:11:05.383 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7501999999999999e-05 2024-05-15T13:11:05.383 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.383 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.383 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:05.383 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:34.283464+0000", 2024-05-15T13:11:05.383 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8946999999999997e-05 2024-05-15T13:11:05.383 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.383 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:05.383 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.383 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.383 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.384 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:214235 6.a 6:5ffec632:::benchmark_data_smithi177_122278_object83864:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e561)", 2024-05-15T13:11:05.384 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:03.404629+0000", 2024-05-15T13:11:05.384 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 1.8852211589999999, 2024-05-15T13:11:05.384 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.169448981, 2024-05-15T13:11:05.384 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:05.384 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:05.384 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:05.384 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:05.384 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:05.384 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 214235 2024-05-15T13:11:05.384 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.384 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:05.384 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.384 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:05.384 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.404629+0000", 2024-05-15T13:11:05.385 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.385 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.385 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.385 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:05.385 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.404629+0000", 2024-05-15T13:11:05.385 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.385 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.385 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.385 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:05.385 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.404631+0000", 2024-05-15T13:11:05.385 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5060000000000001e-06 2024-05-15T13:11:05.385 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.385 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.385 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:05.385 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.404638+0000", 2024-05-15T13:11:05.386 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9630000000000001e-06 2024-05-15T13:11:05.386 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.386 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.386 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:05.386 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.404653+0000", 2024-05-15T13:11:05.386 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4919000000000001e-05 2024-05-15T13:11:05.386 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.386 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.386 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:05.386 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.404662+0000", 2024-05-15T13:11:05.386 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.7319999999999993e-06 2024-05-15T13:11:05.386 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.386 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.386 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:05.387 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.404718+0000", 2024-05-15T13:11:05.387 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5328000000000003e-05 2024-05-15T13:11:05.387 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.387 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.387 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:05.387 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.404858+0000", 2024-05-15T13:11:05.387 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014056600000000001 2024-05-15T13:11:05.387 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.387 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.387 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,6", 2024-05-15T13:11:05.387 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.404976+0000", 2024-05-15T13:11:05.387 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000117625 2024-05-15T13:11:05.387 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.387 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.387 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:05.388 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.409847+0000", 2024-05-15T13:11:05.388 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0048713560000000003 2024-05-15T13:11:05.388 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.388 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.388 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.388 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.417667+0000", 2024-05-15T13:11:05.388 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0078196710000000003 2024-05-15T13:11:05.388 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.388 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.388 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.388 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.574016+0000", 2024-05-15T13:11:05.388 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.15634878199999999 2024-05-15T13:11:05.388 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.388 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.389 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:05.389 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.574049+0000", 2024-05-15T13:11:05.389 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.3237e-05 2024-05-15T13:11:05.389 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.389 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.389 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:05.389 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.574078+0000", 2024-05-15T13:11:05.389 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.9295999999999999e-05 2024-05-15T13:11:05.389 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.389 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:05.389 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.389 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.389 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.390 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:214251 6.a 6:580f9d0a:::benchmark_data_smithi177_122278_object83880:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e561)", 2024-05-15T13:11:05.390 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:03.427033+0000", 2024-05-15T13:11:05.390 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 1.862817808, 2024-05-15T13:11:05.390 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.14714851700000001, 2024-05-15T13:11:05.390 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:05.390 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:05.390 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:05.390 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:05.390 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:05.390 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 214251 2024-05-15T13:11:05.390 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.390 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:05.390 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.390 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:05.390 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.427033+0000", 2024-05-15T13:11:05.391 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.391 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.391 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.391 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:05.391 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.427033+0000", 2024-05-15T13:11:05.391 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.391 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.391 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.391 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:05.391 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.427034+0000", 2024-05-15T13:11:05.391 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2130000000000001e-06 2024-05-15T13:11:05.391 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.391 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.391 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:05.391 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.427038+0000", 2024-05-15T13:11:05.392 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.861e-06 2024-05-15T13:11:05.392 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.392 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.392 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:05.392 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.427048+0000", 2024-05-15T13:11:05.392 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0287e-05 2024-05-15T13:11:05.392 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.392 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.392 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:05.392 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.427054+0000", 2024-05-15T13:11:05.392 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3810000000000004e-06 2024-05-15T13:11:05.392 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.392 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.392 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:05.392 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.427096+0000", 2024-05-15T13:11:05.393 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.1949000000000001e-05 2024-05-15T13:11:05.393 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.393 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.393 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:05.393 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.427212+0000", 2024-05-15T13:11:05.393 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00011570600000000001 2024-05-15T13:11:05.393 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.393 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.393 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,6", 2024-05-15T13:11:05.393 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.427325+0000", 2024-05-15T13:11:05.393 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000113174 2024-05-15T13:11:05.393 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.393 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.393 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:05.393 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.436371+0000", 2024-05-15T13:11:05.394 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0090452929999999994 2024-05-15T13:11:05.394 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.394 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.394 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.394 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.438987+0000", 2024-05-15T13:11:05.394 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0026168989999999998 2024-05-15T13:11:05.394 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.394 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.394 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.395 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.574113+0000", 2024-05-15T13:11:05.395 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.13512554600000001 2024-05-15T13:11:05.395 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.395 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.395 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:05.395 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.574145+0000", 2024-05-15T13:11:05.395 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.2249999999999998e-05 2024-05-15T13:11:05.395 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.395 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.395 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:05.395 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.574181+0000", 2024-05-15T13:11:05.395 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5957999999999997e-05 2024-05-15T13:11:05.395 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.395 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:05.395 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.396 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.396 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.396 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:214265 6.a 6:540adee5:::benchmark_data_smithi177_122278_object83894:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e561)", 2024-05-15T13:11:05.396 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:03.447764+0000", 2024-05-15T13:11:05.396 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 1.8420864240000001, 2024-05-15T13:11:05.396 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.13552996, 2024-05-15T13:11:05.396 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:05.396 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:05.396 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:05.396 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:05.396 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:05.396 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 214265 2024-05-15T13:11:05.396 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.396 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:05.397 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.397 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:05.397 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.447764+0000", 2024-05-15T13:11:05.397 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.397 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.397 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.397 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:05.397 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.447764+0000", 2024-05-15T13:11:05.397 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.397 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.397 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.397 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:05.397 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.447766+0000", 2024-05-15T13:11:05.397 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2029999999999999e-06 2024-05-15T13:11:05.397 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.398 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.398 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:05.398 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.447774+0000", 2024-05-15T13:11:05.398 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7789999999999996e-06 2024-05-15T13:11:05.398 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.398 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.398 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:05.398 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.447794+0000", 2024-05-15T13:11:05.398 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.016e-05 2024-05-15T13:11:05.398 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.398 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.398 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:05.398 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.447806+0000", 2024-05-15T13:11:05.398 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1593999999999999e-05 2024-05-15T13:11:05.399 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.399 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.399 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:05.399 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.447880+0000", 2024-05-15T13:11:05.399 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3763999999999994e-05 2024-05-15T13:11:05.399 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.399 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.399 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:05.399 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.448006+0000", 2024-05-15T13:11:05.399 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012608500000000001 2024-05-15T13:11:05.399 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.399 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.399 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,6", 2024-05-15T13:11:05.399 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.448126+0000", 2024-05-15T13:11:05.399 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000120567 2024-05-15T13:11:05.400 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.400 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.400 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:05.400 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.457950+0000", 2024-05-15T13:11:05.400 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0098243110000000005 2024-05-15T13:11:05.400 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.400 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.400 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.400 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.460983+0000", 2024-05-15T13:11:05.400 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.003032529 2024-05-15T13:11:05.400 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.400 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.400 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.400 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.583187+0000", 2024-05-15T13:11:05.400 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.12220428699999999 2024-05-15T13:11:05.401 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.401 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.401 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:05.401 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.583245+0000", 2024-05-15T13:11:05.401 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7491000000000002e-05 2024-05-15T13:11:05.401 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.401 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.401 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:05.401 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.583294+0000", 2024-05-15T13:11:05.401 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9190000000000002e-05 2024-05-15T13:11:05.401 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.401 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:05.401 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.401 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.401 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.401 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:214286 6.a 6:59fe10de:::benchmark_data_smithi177_122278_object83915:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e561)", 2024-05-15T13:11:05.402 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:03.481438+0000", 2024-05-15T13:11:05.402 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 1.808412498, 2024-05-15T13:11:05.402 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.101958966, 2024-05-15T13:11:05.402 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:05.402 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:05.402 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:05.402 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:05.402 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:05.402 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 214286 2024-05-15T13:11:05.402 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.402 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:05.402 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.402 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:05.402 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.481438+0000", 2024-05-15T13:11:05.402 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.403 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.403 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.403 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:05.403 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.481438+0000", 2024-05-15T13:11:05.403 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:05.403 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.403 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.403 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:05.403 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.481439+0000", 2024-05-15T13:11:05.403 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.2200000000000004e-07 2024-05-15T13:11:05.403 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.403 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.403 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:05.403 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.481442+0000", 2024-05-15T13:11:05.403 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.0589999999999998e-06 2024-05-15T13:11:05.403 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.404 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.404 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:05.404 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.481451+0000", 2024-05-15T13:11:05.404 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.2439999999999999e-06 2024-05-15T13:11:05.404 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.404 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.404 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:05.404 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.481455+0000", 2024-05-15T13:11:05.404 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.5669999999999996e-06 2024-05-15T13:11:05.404 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.404 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.404 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:05.405 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.481512+0000", 2024-05-15T13:11:05.405 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6147000000000003e-05 2024-05-15T13:11:05.405 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.405 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.405 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:05.405 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.481629+0000", 2024-05-15T13:11:05.405 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000117188 2024-05-15T13:11:05.405 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.405 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.405 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,6", 2024-05-15T13:11:05.405 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.481755+0000", 2024-05-15T13:11:05.405 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012594799999999999 2024-05-15T13:11:05.405 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.405 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.405 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.406 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.486420+0000", 2024-05-15T13:11:05.406 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0046656090000000002 2024-05-15T13:11:05.406 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.406 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.406 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:05.406 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.486771+0000", 2024-05-15T13:11:05.406 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00035090799999999998 2024-05-15T13:11:05.406 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.406 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.406 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:05.406 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.583339+0000", 2024-05-15T13:11:05.406 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.096567358000000006 2024-05-15T13:11:05.406 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.406 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.406 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:05.407 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.583371+0000", 2024-05-15T13:11:05.407 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.2632999999999997e-05 2024-05-15T13:11:05.407 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:05.407 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:05.407 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:05.407 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:03.583397+0000", 2024-05-15T13:11:05.407 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.5683000000000001e-05 2024-05-15T13:11:05.407 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.407 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:05.407 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.407 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:05.407 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:05.407 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:11:05.408 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:11:05.463 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:05.668 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:05 smithi027 ceph-mon[39795]: pgmap v1503: 27 pgs: 27 active+clean; 2.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 877 op/s 2024-05-15T13:11:05.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:05 smithi177 ceph-mon[44303]: pgmap v1503: 27 pgs: 27 active+clean; 2.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 877 op/s 2024-05-15T13:11:05.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:05 smithi120 ceph-mon[43961]: pgmap v1503: 27 pgs: 27 active+clean; 2.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 877 op/s 2024-05-15T13:11:06.328 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:11:06.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:06.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:11:06.437 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:11:06.484 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:06.534 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:11:06.545 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:11:06.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:06 smithi120 ceph-mon[43961]: 1.4 deep-scrub starts 2024-05-15T13:11:06.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:06 smithi120 ceph-mon[43961]: 1.4 deep-scrub ok 2024-05-15T13:11:06.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:06 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:11:06.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:06 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:11:06.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:06 smithi120 ceph-mon[43961]: osdmap e562: 12 total, 12 up, 11 in 2024-05-15T13:11:06.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:06 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3755250045' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:11:06.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:06 smithi120 ceph-mon[43961]: pgmap v1505: 27 pgs: 27 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 831 op/s 2024-05-15T13:11:06.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:06 smithi177 ceph-mon[44303]: 1.4 deep-scrub starts 2024-05-15T13:11:06.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:06 smithi177 ceph-mon[44303]: 1.4 deep-scrub ok 2024-05-15T13:11:06.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:06 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:11:06.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:06 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:11:06.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:06 smithi177 ceph-mon[44303]: osdmap e562: 12 total, 12 up, 11 in 2024-05-15T13:11:06.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:06 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3755250045' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:11:06.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:06 smithi177 ceph-mon[44303]: pgmap v1505: 27 pgs: 27 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 831 op/s 2024-05-15T13:11:07.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:06 smithi027 ceph-mon[39795]: 1.4 deep-scrub starts 2024-05-15T13:11:07.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:06 smithi027 ceph-mon[39795]: 1.4 deep-scrub ok 2024-05-15T13:11:07.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:06 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:11:07.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:06 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:11:07.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:06 smithi027 ceph-mon[39795]: osdmap e562: 12 total, 12 up, 11 in 2024-05-15T13:11:07.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:06 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3755250045' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:11:07.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:06 smithi027 ceph-mon[39795]: pgmap v1505: 27 pgs: 27 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 831 op/s 2024-05-15T13:11:07.249 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:11:07.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:07.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:11:07.353 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:11:07.396 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:07.441 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:11:07.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:07.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:11:07.461 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-05-15T13:11:07.548 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:11:07.578 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:07.613 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:11:07.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:07.619 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:11:07.720 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:11:07.763 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:07 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:11:07.764 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:07 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3755250045' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:11:07.764 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:07 smithi120 ceph-mon[43961]: osdmap e563: 12 total, 12 up, 11 in 2024-05-15T13:11:07.765 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:07.805 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:11:07.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:07.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:11:07.905 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:11:07.911 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:11:07.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:07 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:11:07.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:07 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3755250045' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:11:07.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:07 smithi177 ceph-mon[44303]: osdmap e563: 12 total, 12 up, 11 in 2024-05-15T13:11:07.980 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:08.025 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:07 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:11:08.025 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:07 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3755250045' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:11:08.025 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:07 smithi027 ceph-mon[39795]: osdmap e563: 12 total, 12 up, 11 in 2024-05-15T13:11:08.192 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:11:08.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:08.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:11:08.298 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:11:08.332 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:08.361 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:11:08.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:08.364 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:11:08.465 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:11:08.528 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:08.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:08 smithi177 ceph-mon[44303]: pgmap v1507: 27 pgs: 27 active+clean; 2.7 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 797 op/s 2024-05-15T13:11:08.704 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:11:08.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:08.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:11:08.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:08 smithi027 ceph-mon[39795]: pgmap v1507: 27 pgs: 27 active+clean; 2.7 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 797 op/s 2024-05-15T13:11:08.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:08 smithi120 ceph-mon[43961]: pgmap v1507: 27 pgs: 27 active+clean; 2.7 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 797 op/s 2024-05-15T13:11:08.808 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:11:08.839 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:08.864 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:11:08.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:08.869 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:11:08.870 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:11:08.901 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:11:08.952 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:11:08.970 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:11:08.999 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:11:09.021 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:09.048 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:11:09.065 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:11:09.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:09.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:11:09.079 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:11:09.116 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:11:09.155 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:11:09.171 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:11:09.201 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:11:09.216 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:09.232 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:11:09.268 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:11:09.302 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:11:09.315 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:11:09.315 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:11:09.315 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:11:09.315 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:11:09.315 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:11:09.315 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:11:09.315 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:11:09.315 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:11:09.315 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:11:09.315 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:11:09.315 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:11:09.315 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:11:09.390 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:11:09.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:09.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:11:09.496 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:11:09.535 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:09.563 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:11:09.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:09.566 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:11:09.667 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:11:09.698 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:09 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2923281207' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:11:09.703 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:09.785 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:11:09.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:09.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:11:09.890 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:11:09.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:09 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2923281207' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:11:09.931 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:09.963 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:11:09.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:09.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:11:10.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:09 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2923281207' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:11:10.070 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:11:10.126 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:10.166 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:11:10.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:10.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:11:10.173 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:11:10.271 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:11:10.294 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:11:10.294 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:11:10.411 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:10.592 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:10 smithi120 ceph-mon[43961]: pgmap v1508: 27 pgs: 27 active+clean; 2.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 879 op/s 2024-05-15T13:11:10.640 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:11:10.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:10.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:11:10.656 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:10 smithi027 ceph-mon[39795]: pgmap v1508: 27 pgs: 27 active+clean; 2.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 879 op/s 2024-05-15T13:11:10.751 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:11:10.799 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:10.827 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:11:10.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:10.834 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:11:10.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:10 smithi177 ceph-mon[44303]: pgmap v1508: 27 pgs: 27 active+clean; 2.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 879 op/s 2024-05-15T13:11:10.935 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:11:10.980 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:11.014 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:11:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:no valid command found; 10 closest matches: 2024-05-15T13:11:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:0 2024-05-15T13:11:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:1 2024-05-15T13:11:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:2 2024-05-15T13:11:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:abort 2024-05-15T13:11:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:assert 2024-05-15T13:11:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:bench [] [] [] [] 2024-05-15T13:11:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs debug_inject_read_zeros 2024-05-15T13:11:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs files list 2024-05-15T13:11:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs stats 2024-05-15T13:11:11.042 INFO:teuthology.orchestra.run.smithi027.stderr:bluestore allocator dump block 2024-05-15T13:11:11.042 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: invalid command 2024-05-15T13:11:11.549 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:11:11.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:11.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:11:11.586 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-05-15T13:11:11.654 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:11:11.739 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:11.799 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:11:11.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:11.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:11:11.904 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:11:11.954 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:12.039 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:11:12.115 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:11:13.028 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:11:13.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:13.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:11:13.139 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:11:13.181 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:13.270 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:11:13.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:13.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:11:13.285 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:13 smithi027 ceph-mon[39795]: pgmap v1509: 27 pgs: 27 active+clean; 2.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 721 op/s 2024-05-15T13:11:13.381 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:11:13.452 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:13.531 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:13 smithi120 ceph-mon[43961]: pgmap v1509: 27 pgs: 27 active+clean; 2.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 721 op/s 2024-05-15T13:11:13.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:13 smithi177 ceph-mon[44303]: pgmap v1509: 27 pgs: 27 active+clean; 2.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 721 op/s 2024-05-15T13:11:14.085 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:11:14.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:14.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:11:14.142 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T13:11:14.142 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T13:11:14.199 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:11:14.235 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:14.262 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:11:14.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:14.265 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:11:14.366 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:11:14.429 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:11:14.429 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:11:14.429 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:11:14.429 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:11:14.429 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:11:14.429 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:11:14.429 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:11:14.437 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:14.669 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:11:14.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:11:14.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:11:14.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:11:14.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:11:14.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:11:14.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:11:14.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:11:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:11:14.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:14.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:11:14.775 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:11:14.775 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:11:14.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:11:14.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:11:14.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:11:14.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:11:14.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:11:14.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:11:14.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:11:14.791 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:11:14.824 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:14.845 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:11:14.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:14.850 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:11:14.951 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:11:15.036 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:15.269 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:11:15.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:15.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:11:15.374 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:11:15.414 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:15.662 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:15 smithi120 ceph-mon[43961]: pgmap v1510: 27 pgs: 27 active+clean; 2.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 823 op/s 2024-05-15T13:11:15.662 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:15 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:11:15.662 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:15 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:11:15.662 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:15 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:11:15.662 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:15 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:11:15.662 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:15 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:11:15.662 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:15 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:11:15.672 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:11:15.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:15.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:11:15.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:15 smithi177 ceph-mon[44303]: pgmap v1510: 27 pgs: 27 active+clean; 2.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 823 op/s 2024-05-15T13:11:15.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:15 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:11:15.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:15 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:11:15.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:15 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:11:15.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:15 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:11:15.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:15 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:11:15.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:15 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:11:15.700 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-05-15T13:11:15.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:15 smithi027 ceph-mon[39795]: pgmap v1510: 27 pgs: 27 active+clean; 2.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 823 op/s 2024-05-15T13:11:15.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:15 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:11:15.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:15 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:11:15.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:15 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:11:15.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:15 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:11:15.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:15 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:11:15.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:15 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:11:15.778 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:11:15.823 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:16.120 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:11:16.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:16 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:11:16.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:16 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:11:16.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:16 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:11:16.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:16 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:11:16.591 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:11:16.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:16.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:11:16.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:16 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:11:16.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:16 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:11:16.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:16 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:11:16.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:16 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:11:16.698 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:11:16.709 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:16 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:11:16.710 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:16 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:11:16.710 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:16 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:11:16.710 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:16 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:11:16.763 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:16.982 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:11:16.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:16.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:11:17.088 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:11:17.119 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:17.144 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:11:17.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:17.148 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:11:17.249 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:11:17.285 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:11:17.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:17.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:11:17.319 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:11:17.338 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:11:17.358 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:11:17.417 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:11:17.437 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:11:17.494 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:17.631 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:11:17.645 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:17 smithi120 ceph-mon[43961]: pgmap v1511: 27 pgs: 27 active+clean; 2.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 837 op/s 2024-05-15T13:11:17.645 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:17 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/326336444' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:11:17.646 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:17 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:11:17.646 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:17 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3115778816' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:11:17.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:17 smithi177 ceph-mon[44303]: pgmap v1511: 27 pgs: 27 active+clean; 2.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 837 op/s 2024-05-15T13:11:17.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:17 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/326336444' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:11:17.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:17 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:11:17.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:17 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3115778816' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:11:17.720 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:11:17.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:17.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:11:17.737 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:17 smithi027 ceph-mon[39795]: pgmap v1511: 27 pgs: 27 active+clean; 2.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 837 op/s 2024-05-15T13:11:17.737 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:17 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/326336444' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:11:17.737 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:17 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:11:17.737 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:17 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3115778816' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:11:17.829 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:11:17.857 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:17.876 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:11:17.881 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:11:17.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:17.887 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:11:17.930 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:11:17.975 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:11:17.987 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:11:18.026 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:18.033 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:11:18.064 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:11:18.065 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:11:18.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:18.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:11:18.116 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:11:18.156 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:11:18.168 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:11:18.192 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:11:18.196 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:18.206 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:11:18.206 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:11:18.206 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:11:18.206 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:11:18.206 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:11:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:11:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:11:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:11:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:11:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:11:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:11:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:11:18.232 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:11:18.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:18.238 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:11:18.339 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:11:18.381 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:18.402 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:11:18.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:18.405 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:11:18.420 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:11:18.420 INFO:teuthology.orchestra.run.smithi120.stdout: "size": 20, 2024-05-15T13:11:18.420 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 600, 2024-05-15T13:11:18.420 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [ 2024-05-15T13:11:18.420 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.421 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:211766 4.8 4:13b32146:::benchmark_data_smithi177_99756_object91794:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e391)", 2024-05-15T13:11:18.421 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:02:03.810591+0000", 2024-05-15T13:11:18.421 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 554.61649245800004, 2024-05-15T13:11:18.421 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7269911609999999, 2024-05-15T13:11:18.421 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:18.421 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:11:18.421 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:18.421 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:11:18.421 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:11:18.421 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 211766 2024-05-15T13:11:18.421 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.421 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:18.421 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.422 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:18.422 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:03.810591+0000", 2024-05-15T13:11:18.422 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.422 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.422 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.422 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:18.422 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:03.810591+0000", 2024-05-15T13:11:18.422 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.422 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.422 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.422 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:18.422 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:03.810592+0000", 2024-05-15T13:11:18.423 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5400000000000002e-07 2024-05-15T13:11:18.423 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.423 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.423 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:18.423 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:03.810594+0000", 2024-05-15T13:11:18.423 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.396e-06 2024-05-15T13:11:18.423 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.423 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.423 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:18.423 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:03.810600+0000", 2024-05-15T13:11:18.423 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4420000000000003e-06 2024-05-15T13:11:18.423 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.423 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.423 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:18.424 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:03.810605+0000", 2024-05-15T13:11:18.424 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8969999999999999e-06 2024-05-15T13:11:18.424 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.424 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.424 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:18.424 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:03.812501+0000", 2024-05-15T13:11:18.424 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0018954169999999999 2024-05-15T13:11:18.424 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.424 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.424 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:11:18.424 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:03.812581+0000", 2024-05-15T13:11:18.424 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9919999999999994e-05 2024-05-15T13:11:18.425 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.425 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.425 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:18.425 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:07.537558+0000", 2024-05-15T13:11:18.425 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7249777659999999 2024-05-15T13:11:18.425 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.425 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.425 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:18.425 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:07.537582+0000", 2024-05-15T13:11:18.425 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.3969e-05 2024-05-15T13:11:18.425 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.425 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:18.425 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.425 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.426 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.426 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:216547 4.8 4:16e1c84e:::benchmark_data_smithi177_99756_object96575:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e392)", 2024-05-15T13:11:18.426 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:02:10.474631+0000", 2024-05-15T13:11:18.426 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 547.95245263100003, 2024-05-15T13:11:18.426 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.107790772, 2024-05-15T13:11:18.426 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:18.426 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:11:18.426 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:18.426 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:11:18.426 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:11:18.426 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 216547 2024-05-15T13:11:18.426 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.426 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:18.426 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.427 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:18.427 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:10.474631+0000", 2024-05-15T13:11:18.427 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.427 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.427 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.427 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:18.427 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:10.474631+0000", 2024-05-15T13:11:18.427 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.427 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.427 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.427 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:18.427 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:10.474631+0000", 2024-05-15T13:11:18.427 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.3799999999999998e-07 2024-05-15T13:11:18.428 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.428 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.428 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:18.428 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:10.474633+0000", 2024-05-15T13:11:18.428 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3480000000000001e-06 2024-05-15T13:11:18.428 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.428 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.428 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:18.428 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:10.474636+0000", 2024-05-15T13:11:18.428 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.2320000000000001e-06 2024-05-15T13:11:18.428 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.428 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.428 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:18.428 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:10.474639+0000", 2024-05-15T13:11:18.429 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.27e-06 2024-05-15T13:11:18.429 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.429 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.429 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:18.429 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:10.475046+0000", 2024-05-15T13:11:18.429 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00040690499999999999 2024-05-15T13:11:18.429 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.429 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.429 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:11:18.429 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:10.475097+0000", 2024-05-15T13:11:18.429 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0458999999999999e-05 2024-05-15T13:11:18.429 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.429 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.429 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:18.429 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:11.582410+0000", 2024-05-15T13:11:18.430 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1073134250000001 2024-05-15T13:11:18.430 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.430 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.430 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:18.430 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:11.582422+0000", 2024-05-15T13:11:18.430 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1795e-05 2024-05-15T13:11:18.430 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.430 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:18.430 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.430 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.433 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.434 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25180.0:217740 4.8 4:1e14c616:::benchmark_data_smithi177_99756_object97768:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e393)", 2024-05-15T13:11:18.434 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:02:12.257738+0000", 2024-05-15T13:11:18.434 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 546.16934536400004, 2024-05-15T13:11:18.434 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.166419063, 2024-05-15T13:11:18.434 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:18.434 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:11:18.435 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:18.435 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25180", 2024-05-15T13:11:18.435 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3705305284", 2024-05-15T13:11:18.435 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 217740 2024-05-15T13:11:18.435 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.435 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:18.435 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.435 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:18.435 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:12.257738+0000", 2024-05-15T13:11:18.435 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.435 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.435 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.435 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:18.435 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:12.257738+0000", 2024-05-15T13:11:18.436 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.436 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.436 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.436 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:18.436 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:12.257739+0000", 2024-05-15T13:11:18.436 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.4399999999999999e-07 2024-05-15T13:11:18.436 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.436 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.436 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:18.436 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:12.257743+0000", 2024-05-15T13:11:18.436 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2080000000000002e-06 2024-05-15T13:11:18.437 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.437 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.437 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:18.437 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:12.257755+0000", 2024-05-15T13:11:18.437 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1493e-05 2024-05-15T13:11:18.437 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.437 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.437 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:18.437 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:12.257762+0000", 2024-05-15T13:11:18.437 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4379999999999998e-06 2024-05-15T13:11:18.438 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.438 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.438 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:18.438 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:12.259588+0000", 2024-05-15T13:11:18.438 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001826184 2024-05-15T13:11:18.438 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.438 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.438 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:11:18.438 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:12.259687+0000", 2024-05-15T13:11:18.438 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.8612999999999999e-05 2024-05-15T13:11:18.438 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.438 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.438 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:18.439 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:13.424141+0000", 2024-05-15T13:11:18.439 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.164454423 2024-05-15T13:11:18.439 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.439 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.439 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:18.439 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:02:13.424157+0000", 2024-05-15T13:11:18.439 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5860000000000001e-05 2024-05-15T13:11:18.439 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.439 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:18.439 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.439 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.439 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.439 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:144245 6.5 6:aa3fadbf:::benchmark_data_smithi177_122278_object13874:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e542)", 2024-05-15T13:11:18.440 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:09:39.352010+0000", 2024-05-15T13:11:18.440 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 99.075073943000007, 2024-05-15T13:11:18.440 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.043760332999999998, 2024-05-15T13:11:18.440 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:18.440 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:18.440 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:18.440 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:18.440 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:18.440 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 144245 2024-05-15T13:11:18.440 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.440 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:18.440 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.441 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:18.441 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.352010+0000", 2024-05-15T13:11:18.441 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.441 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.441 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.441 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:18.441 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.352010+0000", 2024-05-15T13:11:18.441 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.441 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.441 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.442 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:18.442 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.352010+0000", 2024-05-15T13:11:18.442 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0200000000000001e-07 2024-05-15T13:11:18.442 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.442 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.442 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:18.443 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.352015+0000", 2024-05-15T13:11:18.443 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2919999999999997e-06 2024-05-15T13:11:18.443 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.443 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.443 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:18.444 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.352024+0000", 2024-05-15T13:11:18.444 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8210000000000008e-06 2024-05-15T13:11:18.444 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.444 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.444 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:18.444 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.352031+0000", 2024-05-15T13:11:18.445 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9949999999999999e-06 2024-05-15T13:11:18.445 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.445 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.445 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:18.445 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.360311+0000", 2024-05-15T13:11:18.445 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0082808259999999998 2024-05-15T13:11:18.446 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.446 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.446 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:18.446 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.360435+0000", 2024-05-15T13:11:18.446 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012327599999999999 2024-05-15T13:11:18.447 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.447 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.447 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,9", 2024-05-15T13:11:18.447 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.360548+0000", 2024-05-15T13:11:18.447 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00011327 2024-05-15T13:11:18.447 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.447 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.447 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:18.447 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.364747+0000", 2024-05-15T13:11:18.448 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0041992610000000001 2024-05-15T13:11:18.448 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.448 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.448 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.448 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.368442+0000", 2024-05-15T13:11:18.448 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0036951100000000001 2024-05-15T13:11:18.448 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.448 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.448 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.448 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.395694+0000", 2024-05-15T13:11:18.448 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.027251398 2024-05-15T13:11:18.448 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.449 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.449 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:18.449 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.395738+0000", 2024-05-15T13:11:18.449 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3927000000000001e-05 2024-05-15T13:11:18.449 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.449 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.449 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:18.449 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.395770+0000", 2024-05-15T13:11:18.449 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.2455e-05 2024-05-15T13:11:18.449 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.449 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:18.449 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.449 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.449 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.450 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:144575 6.3 6:c2276d46:::benchmark_data_smithi177_122278_object14204:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e543)", 2024-05-15T13:11:18.450 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:09:39.709043+0000", 2024-05-15T13:11:18.450 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 98.718040166999998, 2024-05-15T13:11:18.450 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.061221679000000001, 2024-05-15T13:11:18.450 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:18.450 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:18.450 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:18.450 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:18.450 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:18.450 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 144575 2024-05-15T13:11:18.450 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.450 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:18.451 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.451 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:18.451 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.709043+0000", 2024-05-15T13:11:18.451 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.451 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.451 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.451 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:18.451 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.709043+0000", 2024-05-15T13:11:18.451 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.451 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.451 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.451 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:18.452 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.709044+0000", 2024-05-15T13:11:18.452 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1999999999999998e-07 2024-05-15T13:11:18.452 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.452 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.452 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:18.452 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.709048+0000", 2024-05-15T13:11:18.452 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.799e-06 2024-05-15T13:11:18.452 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.452 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.452 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:18.452 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.709057+0000", 2024-05-15T13:11:18.452 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1419999999999999e-06 2024-05-15T13:11:18.452 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.452 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.453 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:18.453 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.709065+0000", 2024-05-15T13:11:18.453 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.6120000000000001e-06 2024-05-15T13:11:18.453 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.453 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.453 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:18.453 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.709099+0000", 2024-05-15T13:11:18.453 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.3976999999999997e-05 2024-05-15T13:11:18.453 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.453 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.453 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:18.453 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.709185+0000", 2024-05-15T13:11:18.453 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.6311999999999996e-05 2024-05-15T13:11:18.453 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.454 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.454 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 8,11", 2024-05-15T13:11:18.454 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.709257+0000", 2024-05-15T13:11:18.454 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2044000000000003e-05 2024-05-15T13:11:18.454 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.454 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.454 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:18.454 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.712779+0000", 2024-05-15T13:11:18.454 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0035220329999999999 2024-05-15T13:11:18.454 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.454 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.454 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.455 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.713474+0000", 2024-05-15T13:11:18.455 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00069443700000000001 2024-05-15T13:11:18.455 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.455 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.455 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.455 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.770166+0000", 2024-05-15T13:11:18.455 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.056691843999999998 2024-05-15T13:11:18.455 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.455 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.455 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:18.455 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.770214+0000", 2024-05-15T13:11:18.455 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8065999999999997e-05 2024-05-15T13:11:18.455 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.456 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.456 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:18.456 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:09:39.770265+0000", 2024-05-15T13:11:18.456 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1493000000000003e-05 2024-05-15T13:11:18.456 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.456 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:18.456 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.456 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.456 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.456 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:187774 6.3 6:c2d8055a:::benchmark_data_smithi177_122278_object57403:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-05-15T13:11:18.456 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:10:30.512212+0000", 2024-05-15T13:11:18.456 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 47.914872037999999, 2024-05-15T13:11:18.456 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.044718750000000002, 2024-05-15T13:11:18.456 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:18.456 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:18.457 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:18.457 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:18.457 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:18.457 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 187774 2024-05-15T13:11:18.457 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.457 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:18.457 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.457 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:18.457 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:30.512212+0000", 2024-05-15T13:11:18.457 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.458 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.458 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.458 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:18.458 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:30.512212+0000", 2024-05-15T13:11:18.458 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.458 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.458 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.458 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:18.458 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:30.512212+0000", 2024-05-15T13:11:18.458 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.5000000000000001e-07 2024-05-15T13:11:18.458 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.459 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.459 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:18.459 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:30.512217+0000", 2024-05-15T13:11:18.459 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4360000000000004e-06 2024-05-15T13:11:18.459 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.459 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.459 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:18.459 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:30.512227+0000", 2024-05-15T13:11:18.459 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0091e-05 2024-05-15T13:11:18.459 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.459 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.459 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:18.460 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:30.512235+0000", 2024-05-15T13:11:18.460 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7130000000000005e-06 2024-05-15T13:11:18.460 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.460 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.460 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:18.460 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:30.542622+0000", 2024-05-15T13:11:18.460 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.030387396000000001 2024-05-15T13:11:18.460 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.460 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.460 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:18.460 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:30.542758+0000", 2024-05-15T13:11:18.460 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000136088 2024-05-15T13:11:18.460 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.460 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.461 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 8,11", 2024-05-15T13:11:18.461 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:30.542875+0000", 2024-05-15T13:11:18.461 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000116484 2024-05-15T13:11:18.461 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.461 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.461 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:18.461 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:30.556748+0000", 2024-05-15T13:11:18.461 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.013873735999999999 2024-05-15T13:11:18.461 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.461 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.461 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.461 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:30.556785+0000", 2024-05-15T13:11:18.462 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.6295999999999999e-05 2024-05-15T13:11:18.462 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.462 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.462 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.462 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:30.556831+0000", 2024-05-15T13:11:18.462 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6746999999999998e-05 2024-05-15T13:11:18.462 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.462 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.462 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:18.462 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:30.556886+0000", 2024-05-15T13:11:18.462 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.4382000000000002e-05 2024-05-15T13:11:18.462 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.462 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.463 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:18.463 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:30.556930+0000", 2024-05-15T13:11:18.463 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4530999999999998e-05 2024-05-15T13:11:18.463 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.463 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:18.463 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.463 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.463 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.463 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:195253 6.5 6:a33e017c:::benchmark_data_smithi177_122278_object64882:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e554)", 2024-05-15T13:11:18.463 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:10:39.392352+0000", 2024-05-15T13:11:18.463 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 39.034731342000001, 2024-05-15T13:11:18.463 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.053298208, 2024-05-15T13:11:18.463 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:18.464 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:18.464 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:18.464 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:18.464 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:18.464 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 195253 2024-05-15T13:11:18.464 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.464 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:18.464 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.464 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:18.464 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:39.392352+0000", 2024-05-15T13:11:18.464 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.464 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.464 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.465 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:18.465 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:39.392352+0000", 2024-05-15T13:11:18.465 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.465 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.465 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.465 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:18.465 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:39.392353+0000", 2024-05-15T13:11:18.465 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8899999999999998e-07 2024-05-15T13:11:18.465 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.465 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.465 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:18.465 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:39.392357+0000", 2024-05-15T13:11:18.465 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3680000000000004e-06 2024-05-15T13:11:18.465 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.466 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.466 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:18.466 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:39.392368+0000", 2024-05-15T13:11:18.466 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0557000000000001e-05 2024-05-15T13:11:18.466 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.466 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.466 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:18.466 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:39.392374+0000", 2024-05-15T13:11:18.466 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.2079999999999997e-06 2024-05-15T13:11:18.466 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.467 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.467 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:18.467 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:39.395008+0000", 2024-05-15T13:11:18.467 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0026334879999999998 2024-05-15T13:11:18.467 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.467 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.467 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:18.467 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:39.395135+0000", 2024-05-15T13:11:18.467 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012689999999999999 2024-05-15T13:11:18.467 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.467 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.467 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,9", 2024-05-15T13:11:18.467 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:39.395215+0000", 2024-05-15T13:11:18.467 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0655999999999995e-05 2024-05-15T13:11:18.468 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.468 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.468 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.468 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:39.398749+0000", 2024-05-15T13:11:18.468 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0035338029999999999 2024-05-15T13:11:18.468 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.468 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.468 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:18.468 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:39.398789+0000", 2024-05-15T13:11:18.468 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.9663999999999998e-05 2024-05-15T13:11:18.469 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.469 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.469 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.469 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:39.445533+0000", 2024-05-15T13:11:18.469 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.046744657000000002 2024-05-15T13:11:18.469 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.469 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.470 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:18.470 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:39.445599+0000", 2024-05-15T13:11:18.470 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.5407999999999994e-05 2024-05-15T13:11:18.470 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.470 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.470 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:18.470 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:39.445651+0000", 2024-05-15T13:11:18.470 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1709999999999998e-05 2024-05-15T13:11:18.470 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.470 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:18.470 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.471 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.471 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.471 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:196513 6.5 6:a7f6010f:::benchmark_data_smithi177_122278_object66142:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e554)", 2024-05-15T13:11:18.471 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:10:41.044634+0000", 2024-05-15T13:11:18.471 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 37.382449627, 2024-05-15T13:11:18.471 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.121536344, 2024-05-15T13:11:18.471 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:18.471 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:18.471 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:18.471 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:18.471 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:18.471 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 196513 2024-05-15T13:11:18.471 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.471 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:18.472 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.472 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:18.472 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.044634+0000", 2024-05-15T13:11:18.472 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.472 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.472 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.472 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:18.472 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.044634+0000", 2024-05-15T13:11:18.472 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.472 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.472 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.472 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:18.472 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.044636+0000", 2024-05-15T13:11:18.473 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.669e-06 2024-05-15T13:11:18.473 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.473 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.473 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:18.473 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.044643+0000", 2024-05-15T13:11:18.473 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8539999999999999e-06 2024-05-15T13:11:18.473 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.473 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.473 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:18.473 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.044658+0000", 2024-05-15T13:11:18.473 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5885000000000001e-05 2024-05-15T13:11:18.474 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.474 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.474 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:18.474 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.044667+0000", 2024-05-15T13:11:18.474 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1210000000000007e-06 2024-05-15T13:11:18.474 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.474 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.474 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:18.474 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.046779+0000", 2024-05-15T13:11:18.474 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0021125480000000001 2024-05-15T13:11:18.474 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.474 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.474 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:18.475 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.046920+0000", 2024-05-15T13:11:18.475 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014129400000000001 2024-05-15T13:11:18.475 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.475 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.475 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,9", 2024-05-15T13:11:18.475 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.047046+0000", 2024-05-15T13:11:18.475 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012540399999999999 2024-05-15T13:11:18.475 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.475 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.475 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.475 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.052791+0000", 2024-05-15T13:11:18.475 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0057451710000000003 2024-05-15T13:11:18.476 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.476 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.476 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:18.476 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.052798+0000", 2024-05-15T13:11:18.476 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.1400000000000002e-06 2024-05-15T13:11:18.476 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.476 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.476 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.476 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.166099+0000", 2024-05-15T13:11:18.476 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.113301033 2024-05-15T13:11:18.476 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.476 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.476 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:18.477 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.166140+0000", 2024-05-15T13:11:18.477 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0991000000000001e-05 2024-05-15T13:11:18.477 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.477 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.477 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:18.477 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.166170+0000", 2024-05-15T13:11:18.477 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.0233999999999999e-05 2024-05-15T13:11:18.477 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.478 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:18.478 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.478 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.478 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.478 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:196524 6.5 6:a31c859f:::benchmark_data_smithi177_122278_object66153:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e554)", 2024-05-15T13:11:18.478 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:10:41.061357+0000", 2024-05-15T13:11:18.478 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 37.365726563999999, 2024-05-15T13:11:18.478 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.10946439299999999, 2024-05-15T13:11:18.478 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:18.478 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:18.478 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:18.478 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:18.479 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:18.479 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 196524 2024-05-15T13:11:18.479 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.479 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:18.479 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.479 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:18.479 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.061357+0000", 2024-05-15T13:11:18.479 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.479 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.479 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.479 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:18.479 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.061357+0000", 2024-05-15T13:11:18.480 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.480 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.480 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.480 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:18.480 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.061359+0000", 2024-05-15T13:11:18.480 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5820000000000001e-06 2024-05-15T13:11:18.480 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.480 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.480 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:18.480 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.061364+0000", 2024-05-15T13:11:18.480 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3279999999999996e-06 2024-05-15T13:11:18.480 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.480 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.480 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:18.481 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.061376+0000", 2024-05-15T13:11:18.481 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2272e-05 2024-05-15T13:11:18.481 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.481 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.481 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:18.481 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.061383+0000", 2024-05-15T13:11:18.481 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9940000000000003e-06 2024-05-15T13:11:18.481 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.481 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.482 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:18.482 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.061435+0000", 2024-05-15T13:11:18.482 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1354999999999999e-05 2024-05-15T13:11:18.482 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.482 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.482 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:18.482 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.061542+0000", 2024-05-15T13:11:18.482 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000107404 2024-05-15T13:11:18.482 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.482 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.482 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,9", 2024-05-15T13:11:18.483 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.061647+0000", 2024-05-15T13:11:18.483 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000104961 2024-05-15T13:11:18.483 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.483 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.483 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.483 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.068785+0000", 2024-05-15T13:11:18.483 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0071377430000000002 2024-05-15T13:11:18.483 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.483 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.483 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:18.483 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.071718+0000", 2024-05-15T13:11:18.484 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0029331299999999999 2024-05-15T13:11:18.484 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.484 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.484 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.484 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.170710+0000", 2024-05-15T13:11:18.484 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.098992245000000006 2024-05-15T13:11:18.484 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.484 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.484 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:18.484 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.170770+0000", 2024-05-15T13:11:18.484 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9695999999999999e-05 2024-05-15T13:11:18.484 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.485 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.485 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:18.485 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.170821+0000", 2024-05-15T13:11:18.485 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1683e-05 2024-05-15T13:11:18.485 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.485 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:18.485 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.485 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.485 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.485 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:196543 6.5 6:aaa0ee9a:::benchmark_data_smithi177_122278_object66172:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e554)", 2024-05-15T13:11:18.486 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:10:41.095102+0000", 2024-05-15T13:11:18.486 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 37.331982027999999, 2024-05-15T13:11:18.486 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.076412287999999995, 2024-05-15T13:11:18.486 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:18.486 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:18.486 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:18.487 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:18.487 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:18.487 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 196543 2024-05-15T13:11:18.487 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.487 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:18.487 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.487 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:18.487 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.095102+0000", 2024-05-15T13:11:18.487 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.487 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.488 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.488 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:18.488 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.095102+0000", 2024-05-15T13:11:18.488 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.488 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.488 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.488 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:18.488 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.095103+0000", 2024-05-15T13:11:18.488 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.641e-06 2024-05-15T13:11:18.488 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.488 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.488 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:18.488 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.095111+0000", 2024-05-15T13:11:18.488 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.5009999999999998e-06 2024-05-15T13:11:18.489 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.489 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.489 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:18.489 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.095127+0000", 2024-05-15T13:11:18.489 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6427999999999999e-05 2024-05-15T13:11:18.489 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.489 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.489 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:18.489 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.095136+0000", 2024-05-15T13:11:18.490 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.6319999999999997e-06 2024-05-15T13:11:18.490 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.490 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.490 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:18.490 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.095200+0000", 2024-05-15T13:11:18.491 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3758999999999997e-05 2024-05-15T13:11:18.491 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.491 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.491 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:18.491 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.095355+0000", 2024-05-15T13:11:18.491 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015589900000000001 2024-05-15T13:11:18.491 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.491 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.491 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,9", 2024-05-15T13:11:18.491 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.095500+0000", 2024-05-15T13:11:18.491 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014405399999999999 2024-05-15T13:11:18.491 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.491 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.491 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.492 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.100584+0000", 2024-05-15T13:11:18.492 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0050840649999999996 2024-05-15T13:11:18.492 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.492 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.492 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:18.492 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.100609+0000", 2024-05-15T13:11:18.492 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.5561000000000001e-05 2024-05-15T13:11:18.492 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.492 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.492 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.492 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.171407+0000", 2024-05-15T13:11:18.492 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.070798041000000006 2024-05-15T13:11:18.492 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.492 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.493 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:18.493 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.171468+0000", 2024-05-15T13:11:18.493 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1069999999999996e-05 2024-05-15T13:11:18.493 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.493 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.493 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:18.493 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.171514+0000", 2024-05-15T13:11:18.493 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.5636999999999997e-05 2024-05-15T13:11:18.493 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.493 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:18.493 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.493 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.493 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.494 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:210472 6.5 6:aba39d58:::benchmark_data_smithi177_122278_object80101:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e558)", 2024-05-15T13:11:18.494 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:10:58.864812+0000", 2024-05-15T13:11:18.494 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 19.562272010000001, 2024-05-15T13:11:18.494 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.051266964999999998, 2024-05-15T13:11:18.494 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:18.494 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:18.494 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:18.494 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:18.494 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:18.494 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 210472 2024-05-15T13:11:18.494 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.494 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:18.495 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.495 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:18.495 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:58.864812+0000", 2024-05-15T13:11:18.495 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.495 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.495 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.495 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:18.495 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:58.864812+0000", 2024-05-15T13:11:18.495 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.495 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.495 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.495 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:18.495 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:58.864813+0000", 2024-05-15T13:11:18.496 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.9000000000000005e-07 2024-05-15T13:11:18.496 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.496 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.496 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:18.496 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:58.864817+0000", 2024-05-15T13:11:18.496 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0749999999999999e-06 2024-05-15T13:11:18.496 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.496 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.496 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:18.496 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:58.864827+0000", 2024-05-15T13:11:18.496 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0013999999999999e-05 2024-05-15T13:11:18.497 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.497 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.497 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:18.497 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:58.864833+0000", 2024-05-15T13:11:18.497 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3049999999999998e-06 2024-05-15T13:11:18.497 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.497 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.497 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:18.497 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:58.868006+0000", 2024-05-15T13:11:18.497 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0031726940000000002 2024-05-15T13:11:18.497 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.497 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.497 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:18.497 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:58.868143+0000", 2024-05-15T13:11:18.497 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013698000000000001 2024-05-15T13:11:18.498 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.498 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.498 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,9", 2024-05-15T13:11:18.498 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:58.868214+0000", 2024-05-15T13:11:18.498 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.1204000000000005e-05 2024-05-15T13:11:18.498 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.498 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.498 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.498 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:58.872493+0000", 2024-05-15T13:11:18.498 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0042786279999999996 2024-05-15T13:11:18.498 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.498 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.498 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:18.498 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:58.877604+0000", 2024-05-15T13:11:18.499 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0051114919999999996 2024-05-15T13:11:18.499 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.499 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.499 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.499 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:58.915984+0000", 2024-05-15T13:11:18.499 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.038380086000000001 2024-05-15T13:11:18.499 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.499 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.499 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:18.499 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:58.916043+0000", 2024-05-15T13:11:18.499 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9210999999999999e-05 2024-05-15T13:11:18.499 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.499 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.499 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:18.499 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:58.916079+0000", 2024-05-15T13:11:18.500 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5286000000000001e-05 2024-05-15T13:11:18.500 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.500 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:18.500 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.500 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.500 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.500 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.25735.0:211325 6.b e558/509 6:d5dafdd0:::benchmark_data_smithi177_122278_object80954:head v 558'13430, mlcod=558'13427)", 2024-05-15T13:11:18.500 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:10:59.917544+0000", 2024-05-15T13:11:18.500 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 18.509539904, 2024-05-15T13:11:18.500 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.045325538999999998, 2024-05-15T13:11:18.500 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:18.500 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:18.500 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:18.500 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.501 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:18.501 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:59.917544+0000", 2024-05-15T13:11:18.501 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.501 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.501 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.501 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:18.501 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:59.917544+0000", 2024-05-15T13:11:18.501 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.501 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.501 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.501 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:18.501 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:59.917544+0000", 2024-05-15T13:11:18.501 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2200000000000004e-07 2024-05-15T13:11:18.501 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.501 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.502 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:18.502 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:59.917549+0000", 2024-05-15T13:11:18.502 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.5170000000000001e-06 2024-05-15T13:11:18.502 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.502 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.502 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:18.502 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:59.917555+0000", 2024-05-15T13:11:18.502 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.5089999999999999e-06 2024-05-15T13:11:18.502 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.502 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.502 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:18.503 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:59.917560+0000", 2024-05-15T13:11:18.503 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3370000000000001e-06 2024-05-15T13:11:18.503 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.503 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.503 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:18.503 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:59.917592+0000", 2024-05-15T13:11:18.504 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.2682000000000003e-05 2024-05-15T13:11:18.504 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.504 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.504 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:18.504 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:59.917608+0000", 2024-05-15T13:11:18.504 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6178999999999999e-05 2024-05-15T13:11:18.505 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.505 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.505 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:18.505 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:59.962825+0000", 2024-05-15T13:11:18.505 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.045216435999999999 2024-05-15T13:11:18.505 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.505 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.505 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:18.506 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:59.962869+0000", 2024-05-15T13:11:18.506 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4357000000000002e-05 2024-05-15T13:11:18.506 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.506 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:18.506 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.507 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.507 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.507 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:216060 6.5 6:a8ddad33:::benchmark_data_smithi177_122278_object85689:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e562)", 2024-05-15T13:11:18.507 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:05.671592+0000", 2024-05-15T13:11:18.507 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 12.755491216999999, 2024-05-15T13:11:18.507 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.032625102000000003, 2024-05-15T13:11:18.507 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:18.507 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:18.507 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:18.507 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:18.508 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:18.508 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 216060 2024-05-15T13:11:18.508 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.508 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:18.508 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.508 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:18.508 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.671592+0000", 2024-05-15T13:11:18.508 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.508 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.508 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.508 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:18.508 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.671592+0000", 2024-05-15T13:11:18.508 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.508 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.509 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.509 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:18.509 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.671594+0000", 2024-05-15T13:11:18.509 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.297e-06 2024-05-15T13:11:18.509 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.509 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.509 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:18.509 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.671599+0000", 2024-05-15T13:11:18.509 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3569999999999997e-06 2024-05-15T13:11:18.509 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.509 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.509 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:18.509 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.671611+0000", 2024-05-15T13:11:18.509 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2138e-05 2024-05-15T13:11:18.509 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.510 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.510 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:18.510 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.671619+0000", 2024-05-15T13:11:18.510 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.6890000000000004e-06 2024-05-15T13:11:18.510 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.510 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.510 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:18.510 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.671668+0000", 2024-05-15T13:11:18.510 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8998000000000001e-05 2024-05-15T13:11:18.510 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.510 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.510 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:18.510 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.671807+0000", 2024-05-15T13:11:18.511 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013927000000000001 2024-05-15T13:11:18.511 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.511 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.511 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,9", 2024-05-15T13:11:18.511 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.671927+0000", 2024-05-15T13:11:18.511 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000119401 2024-05-15T13:11:18.511 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.511 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.511 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.511 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.676966+0000", 2024-05-15T13:11:18.511 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0050389880000000003 2024-05-15T13:11:18.511 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.511 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.511 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:18.512 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.686351+0000", 2024-05-15T13:11:18.512 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0093852789999999998 2024-05-15T13:11:18.512 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.512 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.512 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.512 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.704123+0000", 2024-05-15T13:11:18.512 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.01777248 2024-05-15T13:11:18.512 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.512 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.512 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:18.512 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.704175+0000", 2024-05-15T13:11:18.512 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1379999999999999e-05 2024-05-15T13:11:18.512 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.512 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.513 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:18.513 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.704218+0000", 2024-05-15T13:11:18.513 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2824999999999997e-05 2024-05-15T13:11:18.513 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.513 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:18.513 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.513 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.513 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.513 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:216065 6.5 6:a5b81ac0:::benchmark_data_smithi177_122278_object85694:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e562)", 2024-05-15T13:11:18.513 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:05.673242+0000", 2024-05-15T13:11:18.513 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 12.753841264, 2024-05-15T13:11:18.513 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.031256460999999999, 2024-05-15T13:11:18.513 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:18.513 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:18.513 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:18.514 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:18.514 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:18.514 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 216065 2024-05-15T13:11:18.514 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.514 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:18.514 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.514 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:18.514 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.673242+0000", 2024-05-15T13:11:18.514 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.514 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.514 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.514 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:18.515 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.673242+0000", 2024-05-15T13:11:18.515 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.515 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.515 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.515 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:18.515 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.673243+0000", 2024-05-15T13:11:18.515 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0070000000000001e-06 2024-05-15T13:11:18.515 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.515 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.515 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:18.515 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.673248+0000", 2024-05-15T13:11:18.515 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2470000000000002e-06 2024-05-15T13:11:18.516 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.516 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.516 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:18.516 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.673261+0000", 2024-05-15T13:11:18.516 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3175e-05 2024-05-15T13:11:18.516 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.516 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.516 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:18.516 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.673268+0000", 2024-05-15T13:11:18.516 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9190000000000002e-06 2024-05-15T13:11:18.516 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.516 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.516 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:18.516 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.677004+0000", 2024-05-15T13:11:18.517 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0037360000000000002 2024-05-15T13:11:18.517 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.517 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.517 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:18.517 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.677137+0000", 2024-05-15T13:11:18.517 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000132953 2024-05-15T13:11:18.517 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.517 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.517 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,9", 2024-05-15T13:11:18.517 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.677254+0000", 2024-05-15T13:11:18.517 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00011743600000000001 2024-05-15T13:11:18.517 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.517 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.517 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.518 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.686337+0000", 2024-05-15T13:11:18.518 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0090827640000000001 2024-05-15T13:11:18.518 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.518 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.518 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:18.518 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.690655+0000", 2024-05-15T13:11:18.518 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0043179530000000002 2024-05-15T13:11:18.518 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.518 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.518 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.518 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.704397+0000", 2024-05-15T13:11:18.518 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.013741975999999999 2024-05-15T13:11:18.519 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.519 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.519 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:18.519 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.704453+0000", 2024-05-15T13:11:18.519 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5924000000000002e-05 2024-05-15T13:11:18.519 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.519 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.519 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:18.519 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:05.704499+0000", 2024-05-15T13:11:18.519 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6106999999999997e-05 2024-05-15T13:11:18.519 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.519 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:18.519 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.519 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.519 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.519 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:217669 6.3 6:c4ca276f:::benchmark_data_smithi177_122278_object87298:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e563)", 2024-05-15T13:11:18.520 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:07.538027+0000", 2024-05-15T13:11:18.520 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 10.889056513, 2024-05-15T13:11:18.520 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.034272805000000003, 2024-05-15T13:11:18.520 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:18.520 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:18.520 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:18.520 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:18.520 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:18.520 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 217669 2024-05-15T13:11:18.520 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.520 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:18.520 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.520 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:18.520 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.538027+0000", 2024-05-15T13:11:18.521 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.521 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.521 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.521 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:18.521 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.538027+0000", 2024-05-15T13:11:18.521 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.521 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.521 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.521 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:18.521 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.538028+0000", 2024-05-15T13:11:18.521 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.085e-06 2024-05-15T13:11:18.521 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.521 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.521 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:18.522 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.538034+0000", 2024-05-15T13:11:18.522 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3589999999999999e-06 2024-05-15T13:11:18.522 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.522 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.522 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:18.522 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.538045+0000", 2024-05-15T13:11:18.522 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.183e-05 2024-05-15T13:11:18.522 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.522 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.522 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:18.522 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.538053+0000", 2024-05-15T13:11:18.522 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0490000000000004e-06 2024-05-15T13:11:18.523 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.523 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.523 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:18.523 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.541039+0000", 2024-05-15T13:11:18.523 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0029857350000000002 2024-05-15T13:11:18.523 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.523 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.523 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:18.523 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.541154+0000", 2024-05-15T13:11:18.523 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000114411 2024-05-15T13:11:18.523 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.523 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.523 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 8,11", 2024-05-15T13:11:18.523 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.541250+0000", 2024-05-15T13:11:18.523 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.6213999999999997e-05 2024-05-15T13:11:18.524 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.524 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.524 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:18.524 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.554420+0000", 2024-05-15T13:11:18.524 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.013170387 2024-05-15T13:11:18.524 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.524 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.524 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.524 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.568215+0000", 2024-05-15T13:11:18.524 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.013795178 2024-05-15T13:11:18.524 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.524 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.524 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.524 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.572225+0000", 2024-05-15T13:11:18.524 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0040093799999999999 2024-05-15T13:11:18.525 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.525 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.525 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:18.525 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.572271+0000", 2024-05-15T13:11:18.525 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6063000000000003e-05 2024-05-15T13:11:18.525 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.525 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.525 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:18.525 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.572300+0000", 2024-05-15T13:11:18.525 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.9113999999999999e-05 2024-05-15T13:11:18.525 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.525 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:18.525 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.526 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.526 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.526 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:217675 6.3 6:ce8a5cd7:::benchmark_data_smithi177_122278_object87304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e563)", 2024-05-15T13:11:18.526 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:07.542438+0000", 2024-05-15T13:11:18.526 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 10.884645705000001, 2024-05-15T13:11:18.526 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.030027116999999999, 2024-05-15T13:11:18.526 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:18.526 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:18.526 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:18.526 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:18.526 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:18.526 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 217675 2024-05-15T13:11:18.527 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.527 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:18.527 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.527 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:18.527 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.542438+0000", 2024-05-15T13:11:18.527 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.527 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.527 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.527 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:18.527 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.542438+0000", 2024-05-15T13:11:18.527 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.527 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.527 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.527 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:18.527 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.542440+0000", 2024-05-15T13:11:18.528 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9300000000000002e-06 2024-05-15T13:11:18.528 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.528 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.528 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:18.528 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.542447+0000", 2024-05-15T13:11:18.528 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.187e-06 2024-05-15T13:11:18.528 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.528 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.528 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:18.528 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.542462+0000", 2024-05-15T13:11:18.528 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5183e-05 2024-05-15T13:11:18.528 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.528 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.528 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:18.528 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.542473+0000", 2024-05-15T13:11:18.529 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0665e-05 2024-05-15T13:11:18.529 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.529 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.529 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:18.529 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.545493+0000", 2024-05-15T13:11:18.529 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0030203090000000001 2024-05-15T13:11:18.529 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.529 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.529 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:18.529 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.545633+0000", 2024-05-15T13:11:18.529 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000140233 2024-05-15T13:11:18.529 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.529 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.529 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 8,11", 2024-05-15T13:11:18.529 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.545774+0000", 2024-05-15T13:11:18.530 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014012900000000001 2024-05-15T13:11:18.530 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.530 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.530 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:18.530 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.554623+0000", 2024-05-15T13:11:18.530 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0088495229999999998 2024-05-15T13:11:18.530 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.530 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.530 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.530 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.568238+0000", 2024-05-15T13:11:18.530 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.013614571000000001 2024-05-15T13:11:18.530 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.530 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.531 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.531 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.572390+0000", 2024-05-15T13:11:18.531 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0041526269999999999 2024-05-15T13:11:18.531 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.531 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.531 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:18.531 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.572430+0000", 2024-05-15T13:11:18.531 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.9888000000000002e-05 2024-05-15T13:11:18.531 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.531 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.531 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:18.531 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.572465+0000", 2024-05-15T13:11:18.531 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.4872000000000001e-05 2024-05-15T13:11:18.531 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.532 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:18.532 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.532 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.532 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.532 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.25735.0:218013 6.6 e563/509 6:6c439344:::benchmark_data_smithi177_122278_object87642:head v 563'13612, mlcod=563'13608)", 2024-05-15T13:11:18.532 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:07.945660+0000", 2024-05-15T13:11:18.532 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 10.481423190999999, 2024-05-15T13:11:18.532 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.027774119, 2024-05-15T13:11:18.532 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:18.532 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:18.532 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:18.532 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.532 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:18.532 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.945660+0000", 2024-05-15T13:11:18.533 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.533 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.533 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.533 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:18.533 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.945660+0000", 2024-05-15T13:11:18.533 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.533 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.533 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.533 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:18.533 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.945661+0000", 2024-05-15T13:11:18.533 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.8700000000000004e-07 2024-05-15T13:11:18.533 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.533 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.533 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:18.533 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.945670+0000", 2024-05-15T13:11:18.534 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.5210000000000002e-06 2024-05-15T13:11:18.534 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.534 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.534 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:18.534 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.945682+0000", 2024-05-15T13:11:18.534 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2500999999999999e-05 2024-05-15T13:11:18.534 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.534 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.534 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:18.534 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.945690+0000", 2024-05-15T13:11:18.534 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.6520000000000006e-06 2024-05-15T13:11:18.534 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.535 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.535 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:18.535 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.951184+0000", 2024-05-15T13:11:18.535 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0054934449999999996 2024-05-15T13:11:18.535 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.535 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.535 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:18.535 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.951203+0000", 2024-05-15T13:11:18.535 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9698999999999999e-05 2024-05-15T13:11:18.535 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.536 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.536 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:18.536 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.973379+0000", 2024-05-15T13:11:18.536 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.022175929 2024-05-15T13:11:18.536 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.536 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.536 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:18.536 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.973435+0000", 2024-05-15T13:11:18.536 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5384999999999999e-05 2024-05-15T13:11:18.536 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.536 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:18.536 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.536 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.537 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.537 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.25735.0:218019 6.6 e563/509 6:6f0bbf7c:::benchmark_data_smithi177_122278_object87648:head v 563'13613, mlcod=563'13608)", 2024-05-15T13:11:18.537 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:07.945709+0000", 2024-05-15T13:11:18.537 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 10.481374215000001, 2024-05-15T13:11:18.537 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.027755156, 2024-05-15T13:11:18.537 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:18.537 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:18.537 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:18.537 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.537 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:18.537 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.945709+0000", 2024-05-15T13:11:18.537 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.537 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.537 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.538 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:18.538 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.945709+0000", 2024-05-15T13:11:18.538 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.538 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.538 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.538 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:18.538 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.945710+0000", 2024-05-15T13:11:18.538 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.3999999999999998e-07 2024-05-15T13:11:18.538 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.539 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.539 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:18.539 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.945714+0000", 2024-05-15T13:11:18.539 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.8809999999999998e-06 2024-05-15T13:11:18.539 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.539 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.539 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:18.539 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.945719+0000", 2024-05-15T13:11:18.539 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6289999999999998e-06 2024-05-15T13:11:18.539 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.539 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.539 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:18.539 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.945723+0000", 2024-05-15T13:11:18.540 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.189e-06 2024-05-15T13:11:18.540 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.540 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.540 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:18.540 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.955597+0000", 2024-05-15T13:11:18.540 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0098736090000000002 2024-05-15T13:11:18.540 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.540 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.540 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:18.540 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.955616+0000", 2024-05-15T13:11:18.540 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9349999999999999e-05 2024-05-15T13:11:18.540 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.540 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.540 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:18.540 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.973440+0000", 2024-05-15T13:11:18.541 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.017823551999999999 2024-05-15T13:11:18.541 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.541 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.541 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:18.541 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.973465+0000", 2024-05-15T13:11:18.541 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.4706e-05 2024-05-15T13:11:18.541 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.541 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:18.541 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.541 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.541 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.541 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.25735.0:218017 6.8 e563/509 6:103432d5:::benchmark_data_smithi177_122278_object87646:head v 563'13552, mlcod=563'13548)", 2024-05-15T13:11:18.541 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:07.945744+0000", 2024-05-15T13:11:18.541 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 10.481339368, 2024-05-15T13:11:18.542 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.029797889000000001, 2024-05-15T13:11:18.542 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:18.542 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:18.542 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:18.542 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.542 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:18.542 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.945744+0000", 2024-05-15T13:11:18.542 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.542 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.542 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.542 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:18.542 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.945744+0000", 2024-05-15T13:11:18.542 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.543 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.543 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.543 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:18.543 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.945745+0000", 2024-05-15T13:11:18.543 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.9099999999999999e-07 2024-05-15T13:11:18.543 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.543 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.543 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:18.543 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.945749+0000", 2024-05-15T13:11:18.543 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.1420000000000003e-06 2024-05-15T13:11:18.543 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.543 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.543 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:18.543 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.945755+0000", 2024-05-15T13:11:18.544 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9100000000000002e-06 2024-05-15T13:11:18.544 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.544 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.544 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:18.544 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.945759+0000", 2024-05-15T13:11:18.544 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4700000000000004e-06 2024-05-15T13:11:18.544 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.544 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.544 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:18.544 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.949297+0000", 2024-05-15T13:11:18.544 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0035381689999999999 2024-05-15T13:11:18.544 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.544 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.544 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:18.545 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.949313+0000", 2024-05-15T13:11:18.545 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5359999999999999e-05 2024-05-15T13:11:18.545 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.545 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.545 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:18.545 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.975497+0000", 2024-05-15T13:11:18.545 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.026183879 2024-05-15T13:11:18.545 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.545 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.545 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:18.545 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:07.975542+0000", 2024-05-15T13:11:18.545 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.5568000000000001e-05 2024-05-15T13:11:18.545 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.545 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:18.545 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.546 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.546 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.546 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:218464 6.5 6:a0335414:::benchmark_data_smithi177_122278_object88093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e563)", 2024-05-15T13:11:18.546 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:08.460495+0000", 2024-05-15T13:11:18.546 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 9.9665890810000004, 2024-05-15T13:11:18.546 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.031240463999999999, 2024-05-15T13:11:18.546 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:11:18.546 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:18.546 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:11:18.546 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:11:18.546 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:18.546 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 218464 2024-05-15T13:11:18.546 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.546 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:11:18.547 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.547 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:11:18.547 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:08.460495+0000", 2024-05-15T13:11:18.547 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.547 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.547 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.547 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:11:18.547 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:08.460495+0000", 2024-05-15T13:11:18.547 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:11:18.547 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.547 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.547 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:11:18.547 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:08.460496+0000", 2024-05-15T13:11:18.547 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.331e-06 2024-05-15T13:11:18.547 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.548 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.548 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:11:18.548 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:08.460502+0000", 2024-05-15T13:11:18.548 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.2949999999999999e-06 2024-05-15T13:11:18.548 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.548 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.548 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:11:18.548 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:08.460520+0000", 2024-05-15T13:11:18.548 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.8172000000000002e-05 2024-05-15T13:11:18.548 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.548 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.548 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:11:18.548 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:08.460530+0000", 2024-05-15T13:11:18.548 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.8549999999999997e-06 2024-05-15T13:11:18.548 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.548 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.549 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:11:18.549 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:08.467614+0000", 2024-05-15T13:11:18.549 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0070840110000000003 2024-05-15T13:11:18.549 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.549 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.549 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:11:18.549 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:08.467703+0000", 2024-05-15T13:11:18.549 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8925999999999995e-05 2024-05-15T13:11:18.549 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.549 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.549 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,9", 2024-05-15T13:11:18.549 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:08.467785+0000", 2024-05-15T13:11:18.549 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2070999999999994e-05 2024-05-15T13:11:18.550 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.550 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.550 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.550 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:08.475210+0000", 2024-05-15T13:11:18.550 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0074243479999999999 2024-05-15T13:11:18.550 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.550 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.550 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:11:18.550 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:08.475376+0000", 2024-05-15T13:11:18.550 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000166 2024-05-15T13:11:18.550 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.550 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.550 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:18.550 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:08.491632+0000", 2024-05-15T13:11:18.551 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.016256165999999999 2024-05-15T13:11:18.551 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.551 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.551 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:11:18.551 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:08.491689+0000", 2024-05-15T13:11:18.551 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7272000000000003e-05 2024-05-15T13:11:18.551 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:11:18.551 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:11:18.551 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:11:18.551 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:08.491735+0000", 2024-05-15T13:11:18.551 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6016999999999997e-05 2024-05-15T13:11:18.552 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.552 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:18.552 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.552 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:11:18.552 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:11:18.552 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:11:18.552 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:11:18.553 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:18 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:11:18.553 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:18 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:11:18.553 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:18 smithi027 ceph-mon[39795]: osdmap e564: 12 total, 12 up, 11 in 2024-05-15T13:11:18.553 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:11:18.553 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:11:18.553 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:11:18.554 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:11:18.554 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:11:18.554 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:11:18.554 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:18 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:11:18.600 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:18.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:18 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:11:18.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:18 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:11:18.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:18 smithi177 ceph-mon[44303]: osdmap e564: 12 total, 12 up, 11 in 2024-05-15T13:11:18.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:11:18.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:11:18.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:11:18.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:11:18.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:11:18.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:11:18.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:18 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:11:18.701 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:11:18.705 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:18 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:11:18.706 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:18 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:11:18.706 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:18 smithi120 ceph-mon[43961]: osdmap e564: 12 total, 12 up, 11 in 2024-05-15T13:11:18.706 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:11:18.706 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:11:18.706 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:11:18.706 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:11:18.706 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:11:18.706 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:11:18.706 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:18 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:11:18.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:18.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:11:18.808 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:11:18.883 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:18.935 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:11:18.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:18.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:11:19.046 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:11:19.117 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:19.142 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:11:19.144 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:11:19.144 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:11:19.144 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-05-15T13:11:19.144 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 1.000000 2024-05-15T13:11:19.144 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd primary-affinity 3 1 2024-05-15T13:11:19.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:19.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:11:19.246 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:11:19.279 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:19.299 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:11:19.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:19.305 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:11:19.313 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-05-15T13:11:19.406 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:11:19.413 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:19 smithi027 ceph-mon[39795]: pgmap v1513: 27 pgs: 27 active+clean; 2.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 817 op/s 2024-05-15T13:11:19.458 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:19.490 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:11:19.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:19.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:11:19.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:19 smithi120 ceph-mon[43961]: pgmap v1513: 27 pgs: 27 active+clean; 2.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 817 op/s 2024-05-15T13:11:19.599 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:11:19.614 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:11:19.669 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:19.680 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:11:19.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:19 smithi177 ceph-mon[44303]: pgmap v1513: 27 pgs: 27 active+clean; 2.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 817 op/s 2024-05-15T13:11:19.910 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:11:19.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:19.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:11:20.017 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:11:20.059 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:20.254 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:11:20.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:20 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:11:20.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:20 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:11:20.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:20 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:11:20.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:20 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:11:20.692 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:20 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:11:20.692 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:20 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:11:20.982 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:11:21.137 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:11:21.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:21.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:11:21.243 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:11:21.320 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:21.386 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:11:21.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:21.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:11:21.499 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:11:21.560 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:21.562 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:21 smithi027 ceph-mon[39795]: pgmap v1514: 27 pgs: 27 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 812 op/s 2024-05-15T13:11:21.562 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:21 smithi027 ceph-mon[39795]: osdmap e565: 12 total, 12 up, 11 in 2024-05-15T13:11:21.562 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:21 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:11:21.562 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:21 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:11:21.604 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:11:21.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:21.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:11:21.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:21 smithi177 ceph-mon[44303]: pgmap v1514: 27 pgs: 27 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 812 op/s 2024-05-15T13:11:21.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:21 smithi177 ceph-mon[44303]: osdmap e565: 12 total, 12 up, 11 in 2024-05-15T13:11:21.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:21 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:11:21.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:21 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:11:21.711 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:11:21.738 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:21.759 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:11:21.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:21.761 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:11:21.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:21 smithi120 ceph-mon[43961]: pgmap v1514: 27 pgs: 27 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 812 op/s 2024-05-15T13:11:21.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:21 smithi120 ceph-mon[43961]: osdmap e565: 12 total, 12 up, 11 in 2024-05-15T13:11:21.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:21 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:11:21.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:21 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:11:21.862 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:11:21.866 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:11:21.866 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T13:11:21.866 INFO:teuthology.orchestra.run.smithi120.stdout: "num_ops": 0 2024-05-15T13:11:21.866 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:11:21.895 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:21.900 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:11:21.920 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:11:21.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:21.922 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:11:21.996 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:11:22.023 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:11:22.093 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:11:22.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:22.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:11:22.355 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:11:22.390 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:22.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:22 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:11:22.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:22 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:11:22.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:22 smithi120 ceph-mon[43961]: osdmap e566: 12 total, 12 up, 11 in 2024-05-15T13:11:22.650 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:11:22.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:22.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:11:22.681 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-05-15T13:11:22.756 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:11:22.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:22 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:11:22.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:22 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:11:22.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:22 smithi027 ceph-mon[39795]: osdmap e566: 12 total, 12 up, 11 in 2024-05-15T13:11:22.814 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:22.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:22 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:11:22.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:22 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:11:22.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:22 smithi177 ceph-mon[44303]: osdmap e566: 12 total, 12 up, 11 in 2024-05-15T13:11:23.107 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:11:23.502 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-05-15T13:11:23.716 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:23 smithi120 ceph-mon[43961]: pgmap v1517: 27 pgs: 27 active+clean; 2.1 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 702 op/s 2024-05-15T13:11:23.716 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:23 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3741365657' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:11:23.716 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:23 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:11:23.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:23 smithi027 ceph-mon[39795]: pgmap v1517: 27 pgs: 27 active+clean; 2.1 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 702 op/s 2024-05-15T13:11:23.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:23 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3741365657' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:11:23.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:23 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:11:23.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:23 smithi177 ceph-mon[44303]: pgmap v1517: 27 pgs: 27 active+clean; 2.1 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 702 op/s 2024-05-15T13:11:23.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:23 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3741365657' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:11:23.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:23 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:11:24.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:24 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-05-15T13:11:24.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:24 smithi027 ceph-mon[39795]: osdmap e567: 12 total, 12 up, 11 in 2024-05-15T13:11:24.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:24 smithi027 ceph-mon[39795]: pgmap v1519: 27 pgs: 27 active+clean; 1.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 887 op/s 2024-05-15T13:11:24.781 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:24 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-05-15T13:11:24.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:24 smithi120 ceph-mon[43961]: osdmap e567: 12 total, 12 up, 11 in 2024-05-15T13:11:24.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:24 smithi120 ceph-mon[43961]: pgmap v1519: 27 pgs: 27 active+clean; 1.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 887 op/s 2024-05-15T13:11:24.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:24 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-05-15T13:11:24.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:24 smithi177 ceph-mon[44303]: osdmap e567: 12 total, 12 up, 11 in 2024-05-15T13:11:24.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:24 smithi177 ceph-mon[44303]: pgmap v1519: 27 pgs: 27 active+clean; 1.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 887 op/s 2024-05-15T13:11:25.119 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:11:25.119 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T13:11:25.119 INFO:teuthology.orchestra.run.smithi120.stdout: "complaint_time": 30, 2024-05-15T13:11:25.119 INFO:teuthology.orchestra.run.smithi120.stdout: "num_blocked_ops": 0 2024-05-15T13:11:25.119 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:11:25.502 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:11:25.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:25.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:11:25.610 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:11:25.640 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:25.662 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:11:25.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:25.671 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:11:25.772 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:11:25.818 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:27.476 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:11:27.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:27.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:11:27.581 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:11:27.647 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:27 smithi120 ceph-mon[43961]: pgmap v1520: 27 pgs: 27 active+clean; 1.8 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 664 op/s 2024-05-15T13:11:27.673 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:27 smithi027 ceph-mon[39795]: pgmap v1520: 27 pgs: 27 active+clean; 1.8 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 664 op/s 2024-05-15T13:11:27.676 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:27.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:27 smithi177 ceph-mon[44303]: pgmap v1520: 27 pgs: 27 active+clean; 1.8 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 664 op/s 2024-05-15T13:11:27.852 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:11:27.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:27.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:11:27.963 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:11:27.992 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:28.013 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:11:28.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:28.015 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:11:28.116 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:11:28.123 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-05-15T13:11:28.168 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:28.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:28 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3935733926' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:11:28.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:28 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:11:28.335 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:11:28.374 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:11:28.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:28.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:11:28.485 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:11:28.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:28 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3935733926' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:11:28.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:28 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:11:28.538 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:11:28.547 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:28.586 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:11:28.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:28.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:11:28.633 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:28 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3935733926' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:11:28.633 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:28 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:11:28.690 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:11:28.725 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:28.752 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:11:28.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:28.756 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:11:28.856 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:11:28.888 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:29.013 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:11:29.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:29.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:11:29.122 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:11:29.150 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:29.166 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:11:29.170 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:11:29.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:29.173 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:11:29.209 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:11:29.251 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:11:29.273 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:11:29.290 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:11:29.311 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:29.316 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:11:29.332 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:11:29.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:29.335 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:11:29.358 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:11:29.414 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:11:29.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:29 smithi177 ceph-mon[44303]: pgmap v1521: 27 pgs: 27 active+clean; 1.7 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 703 op/s 2024-05-15T13:11:29.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:29 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:11:29.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:29 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:11:29.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:29 smithi177 ceph-mon[44303]: osdmap e568: 12 total, 12 up, 11 in 2024-05-15T13:11:29.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:29 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3004277046' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:11:29.436 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:11:29.463 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:11:29.474 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:29.509 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:11:29.514 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:11:29.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:29.520 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:29 smithi120 ceph-mon[43961]: pgmap v1521: 27 pgs: 27 active+clean; 1.7 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 703 op/s 2024-05-15T13:11:29.520 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:29 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:11:29.520 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:29 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:11:29.520 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:29 smithi120 ceph-mon[43961]: osdmap e568: 12 total, 12 up, 11 in 2024-05-15T13:11:29.520 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:29 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3004277046' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:11:29.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:11:29.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:29 smithi027 ceph-mon[39795]: pgmap v1521: 27 pgs: 27 active+clean; 1.7 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 703 op/s 2024-05-15T13:11:29.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:29 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:11:29.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:29 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:11:29.527 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:29 smithi027 ceph-mon[39795]: osdmap e568: 12 total, 12 up, 11 in 2024-05-15T13:11:29.527 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:29 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3004277046' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:11:29.569 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:11:29.610 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:11:29.621 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:11:29.658 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:29.661 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:11:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:11:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:11:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:11:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:11:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:11:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:11:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:11:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:11:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:11:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:11:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:11:29.687 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:11:29.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:29.691 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:11:29.792 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:11:29.821 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:29.852 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:11:29.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:29.855 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:11:29.956 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:11:29.995 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:30.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:30 smithi027 ceph-mon[39795]: 1.5 scrub starts 2024-05-15T13:11:30.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:30 smithi027 ceph-mon[39795]: 1.5 scrub ok 2024-05-15T13:11:30.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:30 smithi120 ceph-mon[43961]: 1.5 scrub starts 2024-05-15T13:11:30.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:30 smithi120 ceph-mon[43961]: 1.5 scrub ok 2024-05-15T13:11:30.670 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:11:30.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:30.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:11:30.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:30 smithi177 ceph-mon[44303]: 1.5 scrub starts 2024-05-15T13:11:30.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:30 smithi177 ceph-mon[44303]: 1.5 scrub ok 2024-05-15T13:11:30.782 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:11:30.849 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:30.889 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:11:30.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:30.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:11:30.993 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:11:31.058 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:31.109 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:11:31.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:31.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:11:31.215 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:11:31.254 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:31.355 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:11:31.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:31.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:11:31.459 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:11:31.471 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:11:31.471 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:11:31.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:31 smithi027 ceph-mon[39795]: 1.1 scrub starts 2024-05-15T13:11:31.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:31 smithi027 ceph-mon[39795]: pgmap v1523: 27 pgs: 27 active+clean; 1.6 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 947 op/s 2024-05-15T13:11:31.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:31 smithi027 ceph-mon[39795]: 1.1 scrub ok 2024-05-15T13:11:31.525 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:31.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:31 smithi177 ceph-mon[44303]: 1.1 scrub starts 2024-05-15T13:11:31.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:31 smithi177 ceph-mon[44303]: pgmap v1523: 27 pgs: 27 active+clean; 1.6 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 947 op/s 2024-05-15T13:11:31.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:31 smithi177 ceph-mon[44303]: 1.1 scrub ok 2024-05-15T13:11:31.716 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:11:31.761 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:11:31.766 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:31 smithi120 ceph-mon[43961]: 1.1 scrub starts 2024-05-15T13:11:31.766 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:31 smithi120 ceph-mon[43961]: pgmap v1523: 27 pgs: 27 active+clean; 1.6 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 947 op/s 2024-05-15T13:11:31.766 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:31 smithi120 ceph-mon[43961]: 1.1 scrub ok 2024-05-15T13:11:31.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:31.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:11:31.872 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:11:31.904 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:31.924 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:11:31.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:31.927 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:11:32.028 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:11:32.096 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:32.140 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:11:32.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:32.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:11:32.171 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:11:32.232 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-05-15T13:11:32.257 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:11:32.339 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:32.364 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:11:32.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:32.366 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:11:32.467 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:11:32.545 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:32.563 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:11:32.585 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:11:32.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:32.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:11:32.616 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:11:32.694 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:11:32.783 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:32.840 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:11:32.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:32.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:11:32.945 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:11:33.011 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:33.418 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:11:33.418 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:11:33.418 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11] 2024-05-15T13:11:33.418 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd out 4 2024-05-15T13:11:33.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:33 smithi177 ceph-mon[44303]: pgmap v1524: 27 pgs: 27 active+clean; 1.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 663 op/s 2024-05-15T13:11:33.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:33 smithi177 ceph-mon[44303]: 6.d scrub starts 2024-05-15T13:11:33.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:33 smithi177 ceph-mon[44303]: 6.d scrub ok 2024-05-15T13:11:33.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:33 smithi027 ceph-mon[39795]: pgmap v1524: 27 pgs: 27 active+clean; 1.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 663 op/s 2024-05-15T13:11:33.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:33 smithi027 ceph-mon[39795]: 6.d scrub starts 2024-05-15T13:11:33.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:33 smithi027 ceph-mon[39795]: 6.d scrub ok 2024-05-15T13:11:33.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:33 smithi120 ceph-mon[43961]: pgmap v1524: 27 pgs: 27 active+clean; 1.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 663 op/s 2024-05-15T13:11:33.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:33 smithi120 ceph-mon[43961]: 6.d scrub starts 2024-05-15T13:11:33.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:33 smithi120 ceph-mon[43961]: 6.d scrub ok 2024-05-15T13:11:33.993 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:11:33.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:34.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:11:34.101 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:11:34.144 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:34.184 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:11:34.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:34.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:11:34.293 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:11:34.355 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:34.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:34 smithi027 ceph-mon[39795]: pgmap v1525: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.4 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 777 op/s 2024-05-15T13:11:34.832 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:11:34.832 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:11:34.857 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:34 smithi177 ceph-mon[44303]: pgmap v1525: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.4 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 777 op/s 2024-05-15T13:11:34.940 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:11:34.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:34.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:11:34.983 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:11:35.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:34 smithi120 ceph-mon[43961]: pgmap v1525: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.4 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 777 op/s 2024-05-15T13:11:35.034 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:11:35.050 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:11:35.078 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:35.104 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:11:35.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:35.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:11:35.212 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:11:35.249 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:35.564 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-05-15T13:11:35.600 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:11:35.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:35.606 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:11:35.707 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:11:35.748 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:35.775 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:11:35.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:35.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:11:35.879 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:11:35.926 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:35.953 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:11:37.221 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:11:37.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:37.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:11:37.311 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:11:37.332 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:11:37.369 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:37.397 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:11:37.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:37.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:11:37.414 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:37 smithi177 ceph-mon[44303]: 1.8 scrub starts 2024-05-15T13:11:37.414 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:37 smithi177 ceph-mon[44303]: 1.8 scrub ok 2024-05-15T13:11:37.414 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:37 smithi177 ceph-mon[44303]: pgmap v1526: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.4 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 789 op/s 2024-05-15T13:11:37.414 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:37 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4005333868' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:11:37.503 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:11:37.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:37 smithi027 ceph-mon[39795]: 1.8 scrub starts 2024-05-15T13:11:37.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:37 smithi027 ceph-mon[39795]: 1.8 scrub ok 2024-05-15T13:11:37.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:37 smithi027 ceph-mon[39795]: pgmap v1526: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.4 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 789 op/s 2024-05-15T13:11:37.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:37 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4005333868' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:11:37.553 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:37.752 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:11:37.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:37.755 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:11:37.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:37 smithi120 ceph-mon[43961]: 1.8 scrub starts 2024-05-15T13:11:37.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:37 smithi120 ceph-mon[43961]: 1.8 scrub ok 2024-05-15T13:11:37.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:37 smithi120 ceph-mon[43961]: pgmap v1526: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.4 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 789 op/s 2024-05-15T13:11:37.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:37 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4005333868' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:11:37.856 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:11:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:38.038 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:11:38.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:38.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:11:38.057 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:11:38.057 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:11:38.147 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:11:38.200 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:38.237 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:11:38.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:38.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:11:38.308 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:38 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:11:38.309 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:38 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T13:11:38.309 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:38 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4005333868' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:11:38.309 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:38 smithi027 ceph-mon[39795]: osdmap e569: 12 total, 12 up, 11 in 2024-05-15T13:11:38.341 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:11:38.380 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:38 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:11:38.381 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:38 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T13:11:38.381 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:38 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4005333868' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:11:38.381 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:38 smithi177 ceph-mon[44303]: osdmap e569: 12 total, 12 up, 11 in 2024-05-15T13:11:38.382 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:38.425 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:11:38.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:38.435 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:11:38.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:38 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:11:38.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:38 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T13:11:38.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:38 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4005333868' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:11:38.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:38 smithi120 ceph-mon[43961]: osdmap e569: 12 total, 12 up, 11 in 2024-05-15T13:11:38.535 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:11:38.602 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:38.828 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-05-15T13:11:39.178 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:11:39.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:39 smithi177 ceph-mon[44303]: pgmap v1528: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.3 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 733 op/s 2024-05-15T13:11:39.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:39 smithi027 ceph-mon[39795]: pgmap v1528: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.3 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 733 op/s 2024-05-15T13:11:39.683 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:11:39.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:39.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:11:39.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:39 smithi120 ceph-mon[43961]: pgmap v1528: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.3 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 733 op/s 2024-05-15T13:11:39.801 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:11:39.880 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:40.407 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:11:40.407 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:11:40.407 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:11:40.408 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:11:40.408 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:11:40.408 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:11:40.408 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:11:40.408 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:11:40.408 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:11:40.408 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:11:40.408 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:11:40.408 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:11:40.408 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:11:40.408 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:11:40.408 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:11:40.408 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:11:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:11:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:11:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:11:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:11:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:11:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:11:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:11:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:11:40.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:40 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3390957854' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:11:40.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:40 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3390957854' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:11:40.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:40 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3390957854' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:11:41.147 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:11:41.147 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-05-15T13:11:41.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-05-15T13:11:41.147 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T13:11:41.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.148 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:144574 6.1 6:8cb1ff2e:::benchmark_data_smithi177_122278_object14203:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e543)", 2024-05-15T13:11:41.148 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:09:39.708514+0000", 2024-05-15T13:11:41.148 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 121.444600631, 2024-05-15T13:11:41.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.054474545999999999, 2024-05-15T13:11:41.148 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:11:41.148 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:41.148 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:11:41.148 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:11:41.148 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:41.148 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 144574 2024-05-15T13:11:41.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.148 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:11:41.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:11:41.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.708514+0000", 2024-05-15T13:11:41.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:11:41.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.708514+0000", 2024-05-15T13:11:41.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:11:41.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.708515+0000", 2024-05-15T13:11:41.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5510000000000001e-06 2024-05-15T13:11:41.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:11:41.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.708522+0000", 2024-05-15T13:11:41.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8809999999999998e-06 2024-05-15T13:11:41.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:11:41.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.708536+0000", 2024-05-15T13:11:41.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3957999999999999e-05 2024-05-15T13:11:41.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:11:41.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.708545+0000", 2024-05-15T13:11:41.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5699999999999993e-06 2024-05-15T13:11:41.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:11:41.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.708612+0000", 2024-05-15T13:11:41.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7130000000000003e-05 2024-05-15T13:11:41.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:11:41.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.708739+0000", 2024-05-15T13:11:41.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000127112 2024-05-15T13:11:41.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:11:41.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.708846+0000", 2024-05-15T13:11:41.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010668 2024-05-15T13:11:41.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.758843+0000", 2024-05-15T13:11:41.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.049997819999999998 2024-05-15T13:11:41.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.758882+0000", 2024-05-15T13:11:41.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8494000000000001e-05 2024-05-15T13:11:41.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:11:41.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.762882+0000", 2024-05-15T13:11:41.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0040004439999999997 2024-05-15T13:11:41.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:11:41.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.762940+0000", 2024-05-15T13:11:41.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7457000000000002e-05 2024-05-15T13:11:41.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:11:41.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.762988+0000", 2024-05-15T13:11:41.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8449000000000003e-05 2024-05-15T13:11:41.153 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.154 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:11:41.154 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.154 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25735.0:144575 6.3 e543/509 6:c2276d46:::benchmark_data_smithi177_122278_object14204:head v 543'9067, mlcod=543'9065)", 2024-05-15T13:11:41.154 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:09:39.709188+0000", 2024-05-15T13:11:41.154 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 121.443926526, 2024-05-15T13:11:41.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.060702253999999997, 2024-05-15T13:11:41.154 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:11:41.154 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:41.154 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:11:41.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:11:41.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.709188+0000", 2024-05-15T13:11:41.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:11:41.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.709188+0000", 2024-05-15T13:11:41.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:11:41.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.709189+0000", 2024-05-15T13:11:41.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6899999999999996e-07 2024-05-15T13:11:41.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:11:41.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.709207+0000", 2024-05-15T13:11:41.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8646000000000001e-05 2024-05-15T13:11:41.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:11:41.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.709217+0000", 2024-05-15T13:11:41.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8719999999999997e-06 2024-05-15T13:11:41.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:11:41.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.709224+0000", 2024-05-15T13:11:41.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4019999999999999e-06 2024-05-15T13:11:41.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:11:41.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.758600+0000", 2024-05-15T13:11:41.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.049376301999999997 2024-05-15T13:11:41.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:11:41.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.758627+0000", 2024-05-15T13:11:41.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7265000000000001e-05 2024-05-15T13:11:41.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:11:41.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.769848+0000", 2024-05-15T13:11:41.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011220340000000001 2024-05-15T13:11:41.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:11:41.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.769890+0000", 2024-05-15T13:11:41.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2558000000000001e-05 2024-05-15T13:11:41.158 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.158 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:11:41.158 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.158 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.25735.0:144573 6.9 e543/509 6:9abe964c:::benchmark_data_smithi177_122278_object14202:head v 543'8882, mlcod=543'8880)", 2024-05-15T13:11:41.158 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:09:39.711433+0000", 2024-05-15T13:11:41.158 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 121.441681285, 2024-05-15T13:11:41.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.062102923999999997, 2024-05-15T13:11:41.158 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:11:41.158 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:41.158 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:11:41.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:11:41.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.711433+0000", 2024-05-15T13:11:41.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:11:41.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.711433+0000", 2024-05-15T13:11:41.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:11:41.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.711434+0000", 2024-05-15T13:11:41.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1100000000000005e-07 2024-05-15T13:11:41.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:11:41.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.711442+0000", 2024-05-15T13:11:41.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4719999999999995e-06 2024-05-15T13:11:41.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:11:41.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.758557+0000", 2024-05-15T13:11:41.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.047114459999999997 2024-05-15T13:11:41.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:11:41.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.758574+0000", 2024-05-15T13:11:41.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.715e-05 2024-05-15T13:11:41.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:11:41.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.758648+0000", 2024-05-15T13:11:41.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4067999999999996e-05 2024-05-15T13:11:41.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:11:41.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.758668+0000", 2024-05-15T13:11:41.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9967999999999999e-05 2024-05-15T13:11:41.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:11:41.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.773504+0000", 2024-05-15T13:11:41.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014835666000000001 2024-05-15T13:11:41.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:11:41.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.773536+0000", 2024-05-15T13:11:41.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2329000000000002e-05 2024-05-15T13:11:41.162 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.162 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:11:41.162 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.162 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:144576 6.1 6:8183c504:::benchmark_data_smithi177_122278_object14205:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e543)", 2024-05-15T13:11:41.162 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:09:39.713691+0000", 2024-05-15T13:11:41.162 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 121.439423487, 2024-05-15T13:11:41.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.067610067999999995, 2024-05-15T13:11:41.163 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:11:41.163 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:41.163 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:11:41.163 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:11:41.163 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:41.163 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 144576 2024-05-15T13:11:41.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.163 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:11:41.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:11:41.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.713691+0000", 2024-05-15T13:11:41.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:11:41.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.713691+0000", 2024-05-15T13:11:41.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:11:41.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.713692+0000", 2024-05-15T13:11:41.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3769999999999999e-06 2024-05-15T13:11:41.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:11:41.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.713699+0000", 2024-05-15T13:11:41.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5549999999999999e-06 2024-05-15T13:11:41.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:11:41.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.758980+0000", 2024-05-15T13:11:41.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.045281165999999998 2024-05-15T13:11:41.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:11:41.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.758993+0000", 2024-05-15T13:11:41.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2727e-05 2024-05-15T13:11:41.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:11:41.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.776322+0000", 2024-05-15T13:11:41.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017329252999999999 2024-05-15T13:11:41.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:11:41.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.776421+0000", 2024-05-15T13:11:41.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9538999999999997e-05 2024-05-15T13:11:41.166 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.166 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:11:41.166 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.776536+0000", 2024-05-15T13:11:41.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114318 2024-05-15T13:11:41.166 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.166 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:11:41.166 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.780680+0000", 2024-05-15T13:11:41.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0041442179999999999 2024-05-15T13:11:41.166 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.166 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.166 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.781079+0000", 2024-05-15T13:11:41.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000398608 2024-05-15T13:11:41.167 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.167 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.167 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.167 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.781197+0000", 2024-05-15T13:11:41.167 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011812899999999999 2024-05-15T13:11:41.167 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.167 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.167 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:11:41.167 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.781252+0000", 2024-05-15T13:11:41.167 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5730999999999999e-05 2024-05-15T13:11:41.167 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.167 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.167 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:11:41.167 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.781301+0000", 2024-05-15T13:11:41.167 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8446999999999999e-05 2024-05-15T13:11:41.167 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.168 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:11:41.168 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.168 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.168 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:171719 6.1 6:894701a7:::benchmark_data_smithi177_122278_object41348:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-05-15T13:11:41.168 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:10:11.074106+0000", 2024-05-15T13:11:41.168 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 90.079008348000002, 2024-05-15T13:11:41.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.22356810699999999, 2024-05-15T13:11:41.168 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:11:41.168 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:41.168 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:11:41.168 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:11:41.168 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:41.168 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 171719 2024-05-15T13:11:41.168 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.168 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:11:41.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.169 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:11:41.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.074106+0000", 2024-05-15T13:11:41.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.169 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:11:41.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.074106+0000", 2024-05-15T13:11:41.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.169 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:11:41.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.074107+0000", 2024-05-15T13:11:41.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3030000000000001e-06 2024-05-15T13:11:41.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:11:41.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.074113+0000", 2024-05-15T13:11:41.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0870000000000003e-06 2024-05-15T13:11:41.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:11:41.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.074129+0000", 2024-05-15T13:11:41.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5164000000000001e-05 2024-05-15T13:11:41.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:11:41.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.074137+0000", 2024-05-15T13:11:41.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6640000000000004e-06 2024-05-15T13:11:41.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.171 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:11:41.171 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.074188+0000", 2024-05-15T13:11:41.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0835999999999999e-05 2024-05-15T13:11:41.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.171 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:11:41.171 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.074321+0000", 2024-05-15T13:11:41.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013313499999999999 2024-05-15T13:11:41.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.171 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:11:41.171 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.074447+0000", 2024-05-15T13:11:41.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012615099999999999 2024-05-15T13:11:41.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:11:41.172 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.078599+0000", 2024-05-15T13:11:41.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0041520710000000002 2024-05-15T13:11:41.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.172 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.078921+0000", 2024-05-15T13:11:41.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00032142800000000001 2024-05-15T13:11:41.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.172 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.297598+0000", 2024-05-15T13:11:41.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21867695500000001 2024-05-15T13:11:41.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:11:41.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.297638+0000", 2024-05-15T13:11:41.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9703000000000003e-05 2024-05-15T13:11:41.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:11:41.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.297674+0000", 2024-05-15T13:11:41.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6609999999999997e-05 2024-05-15T13:11:41.173 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.173 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:11:41.173 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.173 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:171726 6.1 6:8a0d7b87:::benchmark_data_smithi177_122278_object41355:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-05-15T13:11:41.173 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:10:11.079031+0000", 2024-05-15T13:11:41.174 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 90.074083017999996, 2024-05-15T13:11:41.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.224700821, 2024-05-15T13:11:41.174 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:11:41.174 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:41.174 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:11:41.174 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:11:41.174 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:41.174 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 171726 2024-05-15T13:11:41.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.174 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:11:41.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:11:41.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.079031+0000", 2024-05-15T13:11:41.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.175 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:11:41.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.079031+0000", 2024-05-15T13:11:41.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.175 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:11:41.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.079032+0000", 2024-05-15T13:11:41.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.04e-06 2024-05-15T13:11:41.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.175 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:11:41.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.079038+0000", 2024-05-15T13:11:41.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3809999999999998e-06 2024-05-15T13:11:41.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.176 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.176 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:11:41.176 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.079047+0000", 2024-05-15T13:11:41.176 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5300000000000002e-06 2024-05-15T13:11:41.176 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.176 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.176 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:11:41.176 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.079053+0000", 2024-05-15T13:11:41.176 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4979999999999997e-06 2024-05-15T13:11:41.176 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.176 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.176 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:11:41.176 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.079090+0000", 2024-05-15T13:11:41.176 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7360999999999997e-05 2024-05-15T13:11:41.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.177 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:11:41.177 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.079177+0000", 2024-05-15T13:11:41.177 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7177999999999998e-05 2024-05-15T13:11:41.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.177 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:11:41.177 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.079251+0000", 2024-05-15T13:11:41.177 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3866000000000001e-05 2024-05-15T13:11:41.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.177 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.177 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:11:41.177 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.088147+0000", 2024-05-15T13:11:41.177 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0088959680000000006 2024-05-15T13:11:41.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.088273+0000", 2024-05-15T13:11:41.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000125816 2024-05-15T13:11:41.178 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.303606+0000", 2024-05-15T13:11:41.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21533274999999999 2024-05-15T13:11:41.178 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.178 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.178 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:11:41.178 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.303672+0000", 2024-05-15T13:11:41.178 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6210999999999993e-05 2024-05-15T13:11:41.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.179 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:11:41.179 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.303732+0000", 2024-05-15T13:11:41.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0222e-05 2024-05-15T13:11:41.179 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.179 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:11:41.179 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.179 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.179 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:171728 6.1 6:88889c72:::benchmark_data_smithi177_122278_object41357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-05-15T13:11:41.179 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:10:11.081031+0000", 2024-05-15T13:11:41.179 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 90.072083250999995, 2024-05-15T13:11:41.179 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.22285792900000001, 2024-05-15T13:11:41.179 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:11:41.179 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:41.180 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:11:41.180 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:11:41.180 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:41.180 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 171728 2024-05-15T13:11:41.180 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.180 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:11:41.180 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.180 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:11:41.180 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.081031+0000", 2024-05-15T13:11:41.180 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.180 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.180 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.180 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:11:41.180 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.081031+0000", 2024-05-15T13:11:41.180 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.180 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.181 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.181 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:11:41.181 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.081033+0000", 2024-05-15T13:11:41.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4750000000000001e-06 2024-05-15T13:11:41.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.181 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.181 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:11:41.181 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.081038+0000", 2024-05-15T13:11:41.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0919999999999997e-06 2024-05-15T13:11:41.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.181 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.181 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:11:41.181 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.081050+0000", 2024-05-15T13:11:41.181 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2520999999999999e-05 2024-05-15T13:11:41.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.181 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.182 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:11:41.182 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.081058+0000", 2024-05-15T13:11:41.182 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6159999999999996e-06 2024-05-15T13:11:41.182 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.182 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.182 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:11:41.182 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.083212+0000", 2024-05-15T13:11:41.182 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021537510000000002 2024-05-15T13:11:41.182 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.182 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.182 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:11:41.182 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.083338+0000", 2024-05-15T13:11:41.182 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000126539 2024-05-15T13:11:41.182 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.183 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.183 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:11:41.183 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.083452+0000", 2024-05-15T13:11:41.183 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000113853 2024-05-15T13:11:41.183 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.183 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.183 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:11:41.183 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.088290+0000", 2024-05-15T13:11:41.183 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0048382759999999999 2024-05-15T13:11:41.183 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.183 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.183 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.183 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.088420+0000", 2024-05-15T13:11:41.183 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013007200000000001 2024-05-15T13:11:41.183 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.183 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.184 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.184 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.303784+0000", 2024-05-15T13:11:41.184 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21536370799999999 2024-05-15T13:11:41.184 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.184 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.184 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:11:41.184 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.303845+0000", 2024-05-15T13:11:41.184 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1014999999999998e-05 2024-05-15T13:11:41.184 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.184 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.184 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:11:41.184 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.303889+0000", 2024-05-15T13:11:41.184 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4011000000000003e-05 2024-05-15T13:11:41.184 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.184 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:11:41.184 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.185 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.185 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:171736 6.1 6:8b94f17a:::benchmark_data_smithi177_122278_object41365:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-05-15T13:11:41.185 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:10:11.093648+0000", 2024-05-15T13:11:41.185 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 90.059466477000001, 2024-05-15T13:11:41.185 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21039311999999999, 2024-05-15T13:11:41.185 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:11:41.185 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:41.185 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:11:41.185 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:11:41.185 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:41.185 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 171736 2024-05-15T13:11:41.185 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.185 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:11:41.185 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:11:41.186 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.093648+0000", 2024-05-15T13:11:41.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:11:41.186 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.093648+0000", 2024-05-15T13:11:41.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:11:41.186 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.093649+0000", 2024-05-15T13:11:41.186 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.567e-06 2024-05-15T13:11:41.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.186 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.186 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:11:41.187 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.093657+0000", 2024-05-15T13:11:41.187 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.153e-06 2024-05-15T13:11:41.187 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.187 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.187 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:11:41.187 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.093671+0000", 2024-05-15T13:11:41.187 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4059999999999999e-05 2024-05-15T13:11:41.187 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.187 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.187 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:11:41.187 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.093678+0000", 2024-05-15T13:11:41.187 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6340000000000001e-06 2024-05-15T13:11:41.187 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.187 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.187 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:11:41.188 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.093731+0000", 2024-05-15T13:11:41.188 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2839e-05 2024-05-15T13:11:41.188 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.188 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.188 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:11:41.188 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.093839+0000", 2024-05-15T13:11:41.188 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108297 2024-05-15T13:11:41.188 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.188 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.188 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:11:41.188 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.093922+0000", 2024-05-15T13:11:41.188 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2224000000000004e-05 2024-05-15T13:11:41.188 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.188 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.188 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:11:41.189 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.097930+0000", 2024-05-15T13:11:41.189 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0040084600000000001 2024-05-15T13:11:41.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.189 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.189 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.189 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.098147+0000", 2024-05-15T13:11:41.189 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000217006 2024-05-15T13:11:41.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.189 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.189 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.189 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.303940+0000", 2024-05-15T13:11:41.189 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20579251000000001 2024-05-15T13:11:41.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.189 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.189 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:11:41.189 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.303997+0000", 2024-05-15T13:11:41.190 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.77e-05 2024-05-15T13:11:41.190 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.190 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.190 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:11:41.190 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.304041+0000", 2024-05-15T13:11:41.190 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.367e-05 2024-05-15T13:11:41.190 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.190 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:11:41.190 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.190 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.190 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.190 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:171761 6.1 6:88b94fba:::benchmark_data_smithi177_122278_object41390:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-05-15T13:11:41.190 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:10:11.133770+0000", 2024-05-15T13:11:41.190 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 90.019344841999995, 2024-05-15T13:11:41.190 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17041051800000001, 2024-05-15T13:11:41.191 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:11:41.191 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:41.191 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:11:41.191 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:11:41.191 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:41.191 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 171761 2024-05-15T13:11:41.191 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.191 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:11:41.191 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.191 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:11:41.191 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.133770+0000", 2024-05-15T13:11:41.191 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.191 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.191 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.191 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:11:41.192 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.133770+0000", 2024-05-15T13:11:41.192 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.192 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.192 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.192 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:11:41.192 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.133771+0000", 2024-05-15T13:11:41.192 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6050000000000001e-06 2024-05-15T13:11:41.192 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.192 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.192 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:11:41.192 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.133779+0000", 2024-05-15T13:11:41.192 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7519999999999996e-06 2024-05-15T13:11:41.192 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.192 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.192 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:11:41.193 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.133794+0000", 2024-05-15T13:11:41.193 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5077e-05 2024-05-15T13:11:41.193 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.193 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.193 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:11:41.193 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.133803+0000", 2024-05-15T13:11:41.193 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7520000000000002e-06 2024-05-15T13:11:41.193 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.193 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.193 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:11:41.193 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.133862+0000", 2024-05-15T13:11:41.193 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9218000000000001e-05 2024-05-15T13:11:41.193 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.193 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.193 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:11:41.193 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.133956+0000", 2024-05-15T13:11:41.194 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3911000000000003e-05 2024-05-15T13:11:41.194 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.194 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.194 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:11:41.194 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.134037+0000", 2024-05-15T13:11:41.194 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0840000000000005e-05 2024-05-15T13:11:41.194 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.194 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.194 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:11:41.194 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.143046+0000", 2024-05-15T13:11:41.194 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0090092090000000007 2024-05-15T13:11:41.194 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.194 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.194 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.194 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.143096+0000", 2024-05-15T13:11:41.195 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9804e-05 2024-05-15T13:11:41.195 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.195 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.195 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.195 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.304085+0000", 2024-05-15T13:11:41.195 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.160989717 2024-05-15T13:11:41.195 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.195 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.195 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:11:41.195 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.304140+0000", 2024-05-15T13:11:41.195 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4203999999999998e-05 2024-05-15T13:11:41.195 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.195 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.195 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:11:41.195 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.304180+0000", 2024-05-15T13:11:41.195 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0429000000000002e-05 2024-05-15T13:11:41.195 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.196 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:11:41.196 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.196 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.196 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.196 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:171764 6.1 6:84abda3d:::benchmark_data_smithi177_122278_object41393:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-05-15T13:11:41.196 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:10:11.136029+0000", 2024-05-15T13:11:41.196 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 90.017085846000001, 2024-05-15T13:11:41.196 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17323471300000001, 2024-05-15T13:11:41.196 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:11:41.196 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:41.196 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:11:41.196 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:11:41.196 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:41.196 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 171764 2024-05-15T13:11:41.196 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.196 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:11:41.197 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.197 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:11:41.197 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.136029+0000", 2024-05-15T13:11:41.197 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.197 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.197 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.197 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:11:41.197 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.136029+0000", 2024-05-15T13:11:41.197 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.197 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.197 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.197 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:11:41.197 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.136030+0000", 2024-05-15T13:11:41.197 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.406e-06 2024-05-15T13:11:41.197 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.197 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.198 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:11:41.198 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.136036+0000", 2024-05-15T13:11:41.198 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0990000000000004e-06 2024-05-15T13:11:41.198 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.198 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.198 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:11:41.198 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.136050+0000", 2024-05-15T13:11:41.198 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4229999999999999e-05 2024-05-15T13:11:41.198 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.198 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.198 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:11:41.198 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.136058+0000", 2024-05-15T13:11:41.198 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9619999999999993e-06 2024-05-15T13:11:41.199 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.199 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.199 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:11:41.199 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.138039+0000", 2024-05-15T13:11:41.199 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001980735 2024-05-15T13:11:41.199 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.199 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.199 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:11:41.199 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.138174+0000", 2024-05-15T13:11:41.199 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013523100000000001 2024-05-15T13:11:41.199 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.199 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.199 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:11:41.199 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.138278+0000", 2024-05-15T13:11:41.199 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010347 2024-05-15T13:11:41.199 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.200 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.200 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:11:41.200 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.143192+0000", 2024-05-15T13:11:41.200 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004914351 2024-05-15T13:11:41.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.200 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.200 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.200 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.143505+0000", 2024-05-15T13:11:41.200 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031253200000000002 2024-05-15T13:11:41.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.200 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.200 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.200 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.309147+0000", 2024-05-15T13:11:41.200 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16564235799999999 2024-05-15T13:11:41.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.201 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.201 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:11:41.201 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.309210+0000", 2024-05-15T13:11:41.201 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2899999999999997e-05 2024-05-15T13:11:41.201 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.201 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.201 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:11:41.201 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.309263+0000", 2024-05-15T13:11:41.201 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3439000000000001e-05 2024-05-15T13:11:41.201 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.201 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:11:41.201 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.201 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.201 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.201 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:171774 6.1 6:8b1363f6:::benchmark_data_smithi177_122278_object41403:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-05-15T13:11:41.201 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:10:11.150933+0000", 2024-05-15T13:11:41.202 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 90.002180910000007, 2024-05-15T13:11:41.202 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.15845246199999999, 2024-05-15T13:11:41.202 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:11:41.202 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:41.202 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:11:41.202 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:11:41.202 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:41.202 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 171774 2024-05-15T13:11:41.202 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.202 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:11:41.202 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.202 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:11:41.202 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.150933+0000", 2024-05-15T13:11:41.202 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.202 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.203 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.203 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:11:41.203 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.150933+0000", 2024-05-15T13:11:41.203 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.203 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.203 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.203 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:11:41.203 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.150935+0000", 2024-05-15T13:11:41.203 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3689999999999999e-06 2024-05-15T13:11:41.203 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.203 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.203 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:11:41.203 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.150941+0000", 2024-05-15T13:11:41.203 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8069999999999996e-06 2024-05-15T13:11:41.203 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.203 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.204 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:11:41.204 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.150952+0000", 2024-05-15T13:11:41.204 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1691e-05 2024-05-15T13:11:41.204 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.204 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.204 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:11:41.204 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.150959+0000", 2024-05-15T13:11:41.204 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6680000000000004e-06 2024-05-15T13:11:41.204 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.204 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.204 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:11:41.204 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.151013+0000", 2024-05-15T13:11:41.204 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4255999999999997e-05 2024-05-15T13:11:41.204 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.204 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.205 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:11:41.205 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.151108+0000", 2024-05-15T13:11:41.205 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4263000000000002e-05 2024-05-15T13:11:41.205 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.205 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.205 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:11:41.205 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.151187+0000", 2024-05-15T13:11:41.205 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9808999999999994e-05 2024-05-15T13:11:41.205 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.205 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.205 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:11:41.205 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.155475+0000", 2024-05-15T13:11:41.205 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0042873579999999998 2024-05-15T13:11:41.205 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.205 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.206 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.206 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.156445+0000", 2024-05-15T13:11:41.206 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00097013500000000001 2024-05-15T13:11:41.206 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.206 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.206 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.206 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.309309+0000", 2024-05-15T13:11:41.206 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.152863792 2024-05-15T13:11:41.206 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.206 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.206 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:11:41.206 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.309347+0000", 2024-05-15T13:11:41.206 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8738000000000001e-05 2024-05-15T13:11:41.206 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.206 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.207 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:11:41.207 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.309386+0000", 2024-05-15T13:11:41.207 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8575999999999998e-05 2024-05-15T13:11:41.207 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.207 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:11:41.207 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.207 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.207 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.207 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:171783 6.1 6:84ab70c0:::benchmark_data_smithi177_122278_object41412:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-05-15T13:11:41.207 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:10:11.173970+0000", 2024-05-15T13:11:41.207 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 89.979144012000006, 2024-05-15T13:11:41.207 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14187416999999999, 2024-05-15T13:11:41.207 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:11:41.207 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:41.207 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:11:41.208 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:11:41.208 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:41.208 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 171783 2024-05-15T13:11:41.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.208 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:11:41.208 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.208 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:11:41.208 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.173970+0000", 2024-05-15T13:11:41.208 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.208 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.208 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:11:41.208 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.173970+0000", 2024-05-15T13:11:41.208 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.209 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.209 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:11:41.209 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.173972+0000", 2024-05-15T13:11:41.209 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8640000000000001e-06 2024-05-15T13:11:41.209 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.209 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.209 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:11:41.209 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.173990+0000", 2024-05-15T13:11:41.209 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7759999999999999e-05 2024-05-15T13:11:41.209 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.209 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.209 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:11:41.209 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.174008+0000", 2024-05-15T13:11:41.209 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7907999999999999e-05 2024-05-15T13:11:41.209 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.210 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.210 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:11:41.210 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.174017+0000", 2024-05-15T13:11:41.210 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4259999999999992e-06 2024-05-15T13:11:41.210 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.210 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.210 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:11:41.210 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.174091+0000", 2024-05-15T13:11:41.210 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3650000000000001e-05 2024-05-15T13:11:41.210 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.210 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.210 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:11:41.210 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.174230+0000", 2024-05-15T13:11:41.211 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013892099999999999 2024-05-15T13:11:41.211 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.211 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.211 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:11:41.211 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.174370+0000", 2024-05-15T13:11:41.211 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013983499999999999 2024-05-15T13:11:41.211 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.211 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.211 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:11:41.211 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.178682+0000", 2024-05-15T13:11:41.211 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004312186 2024-05-15T13:11:41.211 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.211 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.211 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.211 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.179926+0000", 2024-05-15T13:11:41.212 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012439510000000001 2024-05-15T13:11:41.212 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.212 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.212 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.212 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.315737+0000", 2024-05-15T13:11:41.212 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.135810762 2024-05-15T13:11:41.212 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.212 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.212 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:11:41.212 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.315796+0000", 2024-05-15T13:11:41.212 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9058000000000003e-05 2024-05-15T13:11:41.212 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.212 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.212 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:11:41.212 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.315845+0000", 2024-05-15T13:11:41.213 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8848999999999999e-05 2024-05-15T13:11:41.213 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.213 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:11:41.213 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.213 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.213 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.213 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:171799 6.1 6:8748cec7:::benchmark_data_smithi177_122278_object41428:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-05-15T13:11:41.213 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:10:11.191685+0000", 2024-05-15T13:11:41.213 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 89.961428888, 2024-05-15T13:11:41.213 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13299766199999999, 2024-05-15T13:11:41.213 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:11:41.213 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:41.213 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:11:41.213 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:11:41.213 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:41.213 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 171799 2024-05-15T13:11:41.214 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.214 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:11:41.214 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.214 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:11:41.214 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.191685+0000", 2024-05-15T13:11:41.214 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.214 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.214 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.214 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:11:41.214 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.191685+0000", 2024-05-15T13:11:41.214 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.214 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.214 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.214 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:11:41.214 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.191687+0000", 2024-05-15T13:11:41.214 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.536e-06 2024-05-15T13:11:41.215 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.215 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.215 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:11:41.215 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.191694+0000", 2024-05-15T13:11:41.215 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9410000000000001e-06 2024-05-15T13:11:41.215 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.215 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.215 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:11:41.215 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.191707+0000", 2024-05-15T13:11:41.215 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3098999999999999e-05 2024-05-15T13:11:41.215 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.215 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.215 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:11:41.215 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.191716+0000", 2024-05-15T13:11:41.216 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.473e-06 2024-05-15T13:11:41.216 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.216 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.216 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:11:41.216 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.191767+0000", 2024-05-15T13:11:41.216 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1854000000000002e-05 2024-05-15T13:11:41.216 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.216 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.216 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:11:41.216 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.191865+0000", 2024-05-15T13:11:41.216 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7361000000000005e-05 2024-05-15T13:11:41.216 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.216 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.216 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:11:41.216 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.191948+0000", 2024-05-15T13:11:41.216 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3493999999999997e-05 2024-05-15T13:11:41.217 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.217 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.217 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.217 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.201040+0000", 2024-05-15T13:11:41.217 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0090918500000000003 2024-05-15T13:11:41.217 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.217 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.217 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:11:41.217 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.201063+0000", 2024-05-15T13:11:41.217 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3062e-05 2024-05-15T13:11:41.217 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.217 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.217 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.217 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.324561+0000", 2024-05-15T13:11:41.218 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.123497541 2024-05-15T13:11:41.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.218 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.218 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:11:41.218 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.324629+0000", 2024-05-15T13:11:41.218 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8758999999999997e-05 2024-05-15T13:11:41.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.218 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.218 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:11:41.218 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.324683+0000", 2024-05-15T13:11:41.218 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3692e-05 2024-05-15T13:11:41.218 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.218 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:11:41.218 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.218 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.219 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:171800 6.1 6:85f76e9a:::benchmark_data_smithi177_122278_object41429:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-05-15T13:11:41.219 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:10:11.194118+0000", 2024-05-15T13:11:41.219 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 89.958996529999993, 2024-05-15T13:11:41.219 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13074761400000001, 2024-05-15T13:11:41.219 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:11:41.219 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:41.219 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:11:41.219 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:11:41.219 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:41.219 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 171800 2024-05-15T13:11:41.219 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.219 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:11:41.219 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.219 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:11:41.220 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.194118+0000", 2024-05-15T13:11:41.220 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.220 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.220 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:11:41.220 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.194118+0000", 2024-05-15T13:11:41.220 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.220 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.220 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:11:41.220 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.194119+0000", 2024-05-15T13:11:41.220 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2529999999999999e-06 2024-05-15T13:11:41.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.220 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.220 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:11:41.221 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.194125+0000", 2024-05-15T13:11:41.221 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4609999999999999e-06 2024-05-15T13:11:41.221 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.221 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.221 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:11:41.221 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.194139+0000", 2024-05-15T13:11:41.221 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.486e-05 2024-05-15T13:11:41.221 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.221 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.221 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:11:41.221 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.194148+0000", 2024-05-15T13:11:41.221 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7150000000000004e-06 2024-05-15T13:11:41.221 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.221 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.221 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:11:41.221 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.195764+0000", 2024-05-15T13:11:41.222 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016162889999999999 2024-05-15T13:11:41.222 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.222 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.222 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:11:41.222 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.195917+0000", 2024-05-15T13:11:41.222 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015290200000000001 2024-05-15T13:11:41.222 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.222 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.222 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:11:41.222 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.196068+0000", 2024-05-15T13:11:41.222 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000150302 2024-05-15T13:11:41.222 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.222 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.222 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.223 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.201099+0000", 2024-05-15T13:11:41.223 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0050315999999999998 2024-05-15T13:11:41.223 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.223 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.223 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:11:41.223 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.201114+0000", 2024-05-15T13:11:41.223 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4622e-05 2024-05-15T13:11:41.223 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.223 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.223 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.223 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.324761+0000", 2024-05-15T13:11:41.223 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.123646848 2024-05-15T13:11:41.223 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.223 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.224 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:11:41.224 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.324819+0000", 2024-05-15T13:11:41.224 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8244e-05 2024-05-15T13:11:41.224 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.224 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.224 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:11:41.224 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.324865+0000", 2024-05-15T13:11:41.224 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6517999999999997e-05 2024-05-15T13:11:41.224 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.224 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:11:41.224 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.224 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.224 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.224 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:171811 6.1 6:82ad73b0:::benchmark_data_smithi177_122278_object41440:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-05-15T13:11:41.224 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:10:11.218180+0000", 2024-05-15T13:11:41.225 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 89.934934671999997, 2024-05-15T13:11:41.225 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.106833724, 2024-05-15T13:11:41.225 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:11:41.225 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:41.225 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:11:41.225 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:11:41.225 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:41.225 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 171811 2024-05-15T13:11:41.225 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.225 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:11:41.225 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.225 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:11:41.225 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.218180+0000", 2024-05-15T13:11:41.225 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.225 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.226 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.226 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:11:41.226 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.218180+0000", 2024-05-15T13:11:41.226 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.226 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.226 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.226 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:11:41.226 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.218181+0000", 2024-05-15T13:11:41.226 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2759999999999999e-06 2024-05-15T13:11:41.226 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.226 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.226 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:11:41.226 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.218186+0000", 2024-05-15T13:11:41.226 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2320000000000001e-06 2024-05-15T13:11:41.227 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.227 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.227 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:11:41.227 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.218199+0000", 2024-05-15T13:11:41.227 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2754e-05 2024-05-15T13:11:41.227 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.227 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.227 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:11:41.227 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.218207+0000", 2024-05-15T13:11:41.227 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0870000000000007e-06 2024-05-15T13:11:41.227 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.227 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.227 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:11:41.227 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.218258+0000", 2024-05-15T13:11:41.228 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1384999999999997e-05 2024-05-15T13:11:41.228 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.228 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.228 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:11:41.228 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.218344+0000", 2024-05-15T13:11:41.228 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5752000000000002e-05 2024-05-15T13:11:41.228 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.228 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.228 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:11:41.228 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.218420+0000", 2024-05-15T13:11:41.228 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5887e-05 2024-05-15T13:11:41.228 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.228 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.228 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:11:41.228 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.222486+0000", 2024-05-15T13:11:41.229 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0040663449999999999 2024-05-15T13:11:41.229 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.229 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.229 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.229 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.222681+0000", 2024-05-15T13:11:41.229 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00019489 2024-05-15T13:11:41.229 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.229 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.229 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.229 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.324924+0000", 2024-05-15T13:11:41.229 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.102242609 2024-05-15T13:11:41.229 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.229 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.229 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:11:41.229 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.324974+0000", 2024-05-15T13:11:41.230 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0529999999999999e-05 2024-05-15T13:11:41.230 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.230 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.230 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:11:41.230 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.325013+0000", 2024-05-15T13:11:41.230 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8977000000000003e-05 2024-05-15T13:11:41.230 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.230 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:11:41.230 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.230 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.230 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.230 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:171832 6.1 6:896c6d66:::benchmark_data_smithi177_122278_object41461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-05-15T13:11:41.230 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:10:11.268612+0000", 2024-05-15T13:11:41.230 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 89.884501978000003, 2024-05-15T13:11:41.230 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.056551711999999997, 2024-05-15T13:11:41.230 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:11:41.231 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:41.231 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:11:41.231 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:11:41.231 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:41.231 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 171832 2024-05-15T13:11:41.231 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.231 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:11:41.231 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.231 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:11:41.231 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.268612+0000", 2024-05-15T13:11:41.231 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.231 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.231 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.231 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:11:41.232 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.268612+0000", 2024-05-15T13:11:41.232 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.232 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.232 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.232 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:11:41.232 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.268614+0000", 2024-05-15T13:11:41.232 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4619999999999999e-06 2024-05-15T13:11:41.232 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.232 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.232 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:11:41.232 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.268622+0000", 2024-05-15T13:11:41.232 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8120000000000008e-06 2024-05-15T13:11:41.232 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.232 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.232 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:11:41.233 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.268638+0000", 2024-05-15T13:11:41.233 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6469000000000001e-05 2024-05-15T13:11:41.233 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.233 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.233 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:11:41.233 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.268646+0000", 2024-05-15T13:11:41.233 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2609999999999993e-06 2024-05-15T13:11:41.233 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.233 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.233 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:11:41.233 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.268709+0000", 2024-05-15T13:11:41.233 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2843999999999997e-05 2024-05-15T13:11:41.233 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.233 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.234 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:11:41.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.268806+0000", 2024-05-15T13:11:41.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6359000000000004e-05 2024-05-15T13:11:41.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.234 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:11:41.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.268884+0000", 2024-05-15T13:11:41.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7928000000000003e-05 2024-05-15T13:11:41.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.234 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:11:41.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.272849+0000", 2024-05-15T13:11:41.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0039650450000000004 2024-05-15T13:11:41.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.235 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.273998+0000", 2024-05-15T13:11:41.235 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00114993 2024-05-15T13:11:41.235 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.235 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.235 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.325070+0000", 2024-05-15T13:11:41.235 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.051071749 2024-05-15T13:11:41.235 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.235 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.235 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:11:41.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.325123+0000", 2024-05-15T13:11:41.235 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2984e-05 2024-05-15T13:11:41.235 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.235 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.236 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:11:41.236 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.325164+0000", 2024-05-15T13:11:41.236 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0868999999999998e-05 2024-05-15T13:11:41.236 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.236 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:11:41.236 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.236 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.236 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.236 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:229827 6.1 6:8e46af48:::benchmark_data_smithi177_122278_object99456:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e566)", 2024-05-15T13:11:41.236 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:11:22.634187+0000", 2024-05-15T13:11:41.236 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 18.518927467000001, 2024-05-15T13:11:41.236 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.045864489000000001, 2024-05-15T13:11:41.236 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:11:41.236 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:41.236 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:11:41.237 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:11:41.237 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:41.237 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 229827 2024-05-15T13:11:41.237 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.237 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:11:41.237 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.237 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:11:41.237 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.634187+0000", 2024-05-15T13:11:41.237 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.237 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.237 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.237 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:11:41.237 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.634187+0000", 2024-05-15T13:11:41.237 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.237 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.238 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.238 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:11:41.238 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.634187+0000", 2024-05-15T13:11:41.238 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.22e-07 2024-05-15T13:11:41.238 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.238 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.238 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:11:41.238 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.634188+0000", 2024-05-15T13:11:41.238 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3349999999999999e-06 2024-05-15T13:11:41.238 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.238 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.238 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:11:41.238 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.634192+0000", 2024-05-15T13:11:41.238 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7270000000000001e-06 2024-05-15T13:11:41.238 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.239 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.239 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:11:41.239 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.634195+0000", 2024-05-15T13:11:41.239 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9050000000000001e-06 2024-05-15T13:11:41.239 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.239 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.239 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:11:41.239 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.664838+0000", 2024-05-15T13:11:41.239 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.030642519999999999 2024-05-15T13:11:41.239 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.239 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.239 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:11:41.239 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.664956+0000", 2024-05-15T13:11:41.239 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000118082 2024-05-15T13:11:41.239 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.240 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.240 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:11:41.240 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.665075+0000", 2024-05-15T13:11:41.240 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011938 2024-05-15T13:11:41.240 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.240 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.240 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:11:41.240 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.679538+0000", 2024-05-15T13:11:41.240 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014462558 2024-05-15T13:11:41.240 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.240 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.240 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.240 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.679777+0000", 2024-05-15T13:11:41.240 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023982300000000001 2024-05-15T13:11:41.241 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.241 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.241 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.241 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.679961+0000", 2024-05-15T13:11:41.241 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000183871 2024-05-15T13:11:41.241 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.241 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.241 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:11:41.241 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.680012+0000", 2024-05-15T13:11:41.241 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1013000000000001e-05 2024-05-15T13:11:41.241 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.241 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.241 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:11:41.241 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.680051+0000", 2024-05-15T13:11:41.241 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9053e-05 2024-05-15T13:11:41.241 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.242 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:11:41.242 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.242 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.242 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.242 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:229837 6.1 6:83b15264:::benchmark_data_smithi177_122278_object99466:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e566)", 2024-05-15T13:11:41.242 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:11:22.634199+0000", 2024-05-15T13:11:41.242 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 18.518915513, 2024-05-15T13:11:41.242 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.050069184000000003, 2024-05-15T13:11:41.242 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:11:41.242 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:41.242 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:11:41.242 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:11:41.242 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:41.242 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 229837 2024-05-15T13:11:41.242 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.243 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:11:41.243 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.243 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:11:41.243 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.634199+0000", 2024-05-15T13:11:41.243 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.243 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.243 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.243 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:11:41.243 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.634199+0000", 2024-05-15T13:11:41.243 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.243 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.243 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.243 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:11:41.243 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.634199+0000", 2024-05-15T13:11:41.243 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.05e-07 2024-05-15T13:11:41.243 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.244 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.244 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:11:41.244 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.634200+0000", 2024-05-15T13:11:41.244 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2419999999999999e-06 2024-05-15T13:11:41.244 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.244 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.244 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:11:41.244 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.634203+0000", 2024-05-15T13:11:41.244 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.948e-06 2024-05-15T13:11:41.244 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.244 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.244 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:11:41.244 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.634206+0000", 2024-05-15T13:11:41.244 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8949999999999998e-06 2024-05-15T13:11:41.244 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.245 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.245 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:11:41.245 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.670018+0000", 2024-05-15T13:11:41.245 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.035811484999999997 2024-05-15T13:11:41.245 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.245 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.245 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:11:41.245 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.670147+0000", 2024-05-15T13:11:41.245 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012915600000000001 2024-05-15T13:11:41.245 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.245 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.245 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:11:41.245 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.670268+0000", 2024-05-15T13:11:41.245 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012104699999999999 2024-05-15T13:11:41.245 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.245 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.246 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:11:41.246 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.679734+0000", 2024-05-15T13:11:41.246 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0094665510000000001 2024-05-15T13:11:41.246 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.246 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.246 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.246 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.680103+0000", 2024-05-15T13:11:41.246 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00036822599999999998 2024-05-15T13:11:41.246 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.246 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.246 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.246 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.684165+0000", 2024-05-15T13:11:41.246 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0040622920000000003 2024-05-15T13:11:41.246 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.246 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.247 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:11:41.247 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.684223+0000", 2024-05-15T13:11:41.247 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7698000000000002e-05 2024-05-15T13:11:41.247 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.247 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.247 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:11:41.247 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.684268+0000", 2024-05-15T13:11:41.247 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5439000000000003e-05 2024-05-15T13:11:41.247 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.247 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:11:41.247 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.247 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.247 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.247 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:229866 6.1 6:8da575b0:::benchmark_data_smithi177_122278_object99495:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e566)", 2024-05-15T13:11:41.247 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:11:22.634209+0000", 2024-05-15T13:11:41.247 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 18.518905198999999, 2024-05-15T13:11:41.248 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.051152383000000003, 2024-05-15T13:11:41.248 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:11:41.248 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:41.248 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:11:41.248 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:11:41.248 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:41.248 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 229866 2024-05-15T13:11:41.248 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.248 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:11:41.248 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.248 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:11:41.248 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.634209+0000", 2024-05-15T13:11:41.248 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.248 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.248 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.249 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:11:41.249 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.634209+0000", 2024-05-15T13:11:41.249 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.249 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.249 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.249 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:11:41.249 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.634209+0000", 2024-05-15T13:11:41.249 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2399999999999999e-07 2024-05-15T13:11:41.249 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.249 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.249 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:11:41.249 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.634211+0000", 2024-05-15T13:11:41.249 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2979999999999999e-06 2024-05-15T13:11:41.249 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.249 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.250 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:11:41.250 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.634214+0000", 2024-05-15T13:11:41.250 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8729999999999999e-06 2024-05-15T13:11:41.250 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.250 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.250 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:11:41.250 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.634217+0000", 2024-05-15T13:11:41.250 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6739999999999999e-06 2024-05-15T13:11:41.250 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.250 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.250 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:11:41.250 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.674748+0000", 2024-05-15T13:11:41.250 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040531101 2024-05-15T13:11:41.250 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.250 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.250 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:11:41.251 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.674861+0000", 2024-05-15T13:11:41.251 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000112219 2024-05-15T13:11:41.251 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.251 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.251 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:11:41.251 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.674993+0000", 2024-05-15T13:11:41.251 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013282499999999999 2024-05-15T13:11:41.251 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.251 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.251 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:11:41.251 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.679925+0000", 2024-05-15T13:11:41.251 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0049314909999999997 2024-05-15T13:11:41.251 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.251 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.251 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.252 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.680164+0000", 2024-05-15T13:11:41.252 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000238879 2024-05-15T13:11:41.252 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.252 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.252 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.252 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.685306+0000", 2024-05-15T13:11:41.252 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00514227 2024-05-15T13:11:41.252 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.252 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.252 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:11:41.252 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.685337+0000", 2024-05-15T13:11:41.252 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1015e-05 2024-05-15T13:11:41.252 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.252 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.252 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:11:41.253 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.685362+0000", 2024-05-15T13:11:41.253 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4513999999999999e-05 2024-05-15T13:11:41.253 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.253 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:11:41.253 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.253 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.253 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.253 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:229934 6.1 6:8f41027c:::benchmark_data_smithi177_122278_object99563:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e566)", 2024-05-15T13:11:41.253 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:11:22.637013+0000", 2024-05-15T13:11:41.253 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 18.516101386999999, 2024-05-15T13:11:41.253 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.050739488999999999, 2024-05-15T13:11:41.253 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:11:41.253 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:11:41.253 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:11:41.253 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:11:41.253 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:11:41.254 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 229934 2024-05-15T13:11:41.254 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.254 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:11:41.254 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.254 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:11:41.254 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.637013+0000", 2024-05-15T13:11:41.254 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.254 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.254 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.254 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:11:41.254 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.637013+0000", 2024-05-15T13:11:41.254 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:11:41.254 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.254 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.254 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:11:41.255 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.637014+0000", 2024-05-15T13:11:41.255 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1909999999999999e-06 2024-05-15T13:11:41.255 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.255 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.255 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:11:41.255 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.637020+0000", 2024-05-15T13:11:41.255 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5589999999999996e-06 2024-05-15T13:11:41.255 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.255 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.255 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:11:41.255 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.637033+0000", 2024-05-15T13:11:41.255 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3015e-05 2024-05-15T13:11:41.255 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.255 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.255 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:11:41.255 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.637040+0000", 2024-05-15T13:11:41.256 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4900000000000003e-06 2024-05-15T13:11:41.256 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.256 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.256 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:11:41.256 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.680214+0000", 2024-05-15T13:11:41.256 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.043174168999999998 2024-05-15T13:11:41.256 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.256 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.256 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:11:41.256 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.680338+0000", 2024-05-15T13:11:41.256 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012374400000000001 2024-05-15T13:11:41.256 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.256 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.256 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:11:41.256 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.680457+0000", 2024-05-15T13:11:41.257 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000119132 2024-05-15T13:11:41.257 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.257 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.257 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:11:41.257 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.684288+0000", 2024-05-15T13:11:41.257 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0038306519999999999 2024-05-15T13:11:41.257 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.257 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.257 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.257 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.685393+0000", 2024-05-15T13:11:41.257 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011053860000000001 2024-05-15T13:11:41.257 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.257 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.257 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:11:41.257 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.687695+0000", 2024-05-15T13:11:41.258 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023015959999999999 2024-05-15T13:11:41.258 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.258 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.258 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:11:41.258 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.687726+0000", 2024-05-15T13:11:41.258 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1306999999999997e-05 2024-05-15T13:11:41.258 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:11:41.258 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:11:41.258 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:11:41.258 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:22.687752+0000", 2024-05-15T13:11:41.258 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6248e-05 2024-05-15T13:11:41.258 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.258 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:11:41.258 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.258 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:11:41.259 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:11:41.259 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:11:41.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:41 smithi177 ceph-mon[44303]: pgmap v1529: 27 pgs: 27 active+clean; 1.1 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 832 op/s 2024-05-15T13:11:41.593 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:41 smithi027 ceph-mon[39795]: pgmap v1529: 27 pgs: 27 active+clean; 1.1 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 832 op/s 2024-05-15T13:11:41.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:41 smithi120 ceph-mon[43961]: pgmap v1529: 27 pgs: 27 active+clean; 1.1 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 832 op/s 2024-05-15T13:11:41.860 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-05-15T13:11:42.220 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:11:42.276 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:11:42.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:42.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:11:42.383 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:11:42.451 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:42.631 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:11:42.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:42.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:11:42.740 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:11:42.774 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:43.374 INFO:teuthology.orchestra.run.smithi027.stderr:marked out osd.4. 2024-05-15T13:11:43.398 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:43 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-a[39772]: 2024-05-15T13:11:43.143+0000 7f8daffd2700 -1 mon.a@0(leader).osd e569 definitely_dead 0 2024-05-15T13:11:43.398 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:43 smithi027 ceph-mon[39795]: pgmap v1530: 27 pgs: 27 active+clean; 1.1 GiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 834 op/s 2024-05-15T13:11:43.398 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:43 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1855163369' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-05-15T13:11:43.398 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:43 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-05-15T13:11:43.398 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:43 smithi027 ceph-mon[39795]: Client client.admin marked osd.4 out, while it was still marked up 2024-05-15T13:11:43.505 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:43 smithi177 ceph-mon[44303]: pgmap v1530: 27 pgs: 27 active+clean; 1.1 GiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 834 op/s 2024-05-15T13:11:43.505 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:43 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1855163369' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-05-15T13:11:43.505 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:43 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-05-15T13:11:43.506 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:43 smithi177 ceph-mon[44303]: Client client.admin marked osd.4 out, while it was still marked up 2024-05-15T13:11:43.541 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:11:43.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:43.545 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:11:43.554 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:11:43.645 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:11:43.672 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:43.770 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:11:43.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:43.773 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:11:43.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:43 smithi120 ceph-mon[43961]: pgmap v1530: 27 pgs: 27 active+clean; 1.1 GiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 834 op/s 2024-05-15T13:11:43.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:43 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1855163369' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-05-15T13:11:43.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:43 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-05-15T13:11:43.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:43 smithi120 ceph-mon[43961]: Client client.admin marked osd.4 out, while it was still marked up 2024-05-15T13:11:43.874 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:11:43.902 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:43.969 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:11:44.066 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:11:44.066 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T13:11:44.066 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-05-15T13:11:44.066 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:11:44.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:44 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-05-15T13:11:44.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:44 smithi177 ceph-mon[44303]: osdmap e570: 12 total, 12 up, 10 in 2024-05-15T13:11:44.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:44 smithi177 ceph-mon[44303]: osdmap e571: 12 total, 12 up, 10 in 2024-05-15T13:11:44.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:44 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-05-15T13:11:44.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:44 smithi027 ceph-mon[39795]: osdmap e570: 12 total, 12 up, 10 in 2024-05-15T13:11:44.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:44 smithi027 ceph-mon[39795]: osdmap e571: 12 total, 12 up, 10 in 2024-05-15T13:11:44.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:44 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-05-15T13:11:44.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:44 smithi120 ceph-mon[43961]: osdmap e570: 12 total, 12 up, 10 in 2024-05-15T13:11:44.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:44 smithi120 ceph-mon[43961]: osdmap e571: 12 total, 12 up, 10 in 2024-05-15T13:11:44.953 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:11:44.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:44.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:11:45.036 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-05-15T13:11:45.068 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:11:45.070 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:11:45.077 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:11:45.187 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:45.379 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:45 smithi177 ceph-mon[44303]: pgmap v1533: 27 pgs: 27 active+clean; 938 MiB data, 7.6 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 989 op/s 2024-05-15T13:11:45.381 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:45 smithi177 ceph-mon[44303]: 6.7 starting backfill to osd.1 from (0'0,0'0] MAX to 569'15509 2024-05-15T13:11:45.381 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:45 smithi177 ceph-mon[44303]: 6.6 starting backfill to osd.5 from (0'0,0'0] MAX to 569'15349 2024-05-15T13:11:45.381 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:45 smithi177 ceph-mon[44303]: osdmap e572: 12 total, 12 up, 10 in 2024-05-15T13:11:45.381 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:45 smithi177 ceph-mon[44303]: 6.e starting backfill to osd.5 from (0'0,0'0] MAX to 569'15370 2024-05-15T13:11:45.381 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:45 smithi177 ceph-mon[44303]: 6.2 starting backfill to osd.11 from (0'0,0'0] MAX to 569'15328 2024-05-15T13:11:45.382 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:11:45.624 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:45 smithi027 ceph-mon[39795]: pgmap v1533: 27 pgs: 27 active+clean; 938 MiB data, 7.6 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 989 op/s 2024-05-15T13:11:45.624 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:45 smithi027 ceph-mon[39795]: 6.7 starting backfill to osd.1 from (0'0,0'0] MAX to 569'15509 2024-05-15T13:11:45.624 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:45 smithi027 ceph-mon[39795]: 6.6 starting backfill to osd.5 from (0'0,0'0] MAX to 569'15349 2024-05-15T13:11:45.624 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:45 smithi027 ceph-mon[39795]: osdmap e572: 12 total, 12 up, 10 in 2024-05-15T13:11:45.624 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:45 smithi027 ceph-mon[39795]: 6.e starting backfill to osd.5 from (0'0,0'0] MAX to 569'15370 2024-05-15T13:11:45.624 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:45 smithi027 ceph-mon[39795]: 6.2 starting backfill to osd.11 from (0'0,0'0] MAX to 569'15328 2024-05-15T13:11:45.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:45 smithi120 ceph-mon[43961]: pgmap v1533: 27 pgs: 27 active+clean; 938 MiB data, 7.6 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 989 op/s 2024-05-15T13:11:45.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:45 smithi120 ceph-mon[43961]: 6.7 starting backfill to osd.1 from (0'0,0'0] MAX to 569'15509 2024-05-15T13:11:45.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:45 smithi120 ceph-mon[43961]: 6.6 starting backfill to osd.5 from (0'0,0'0] MAX to 569'15349 2024-05-15T13:11:45.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:45 smithi120 ceph-mon[43961]: osdmap e572: 12 total, 12 up, 10 in 2024-05-15T13:11:45.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:45 smithi120 ceph-mon[43961]: 6.e starting backfill to osd.5 from (0'0,0'0] MAX to 569'15370 2024-05-15T13:11:45.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:45 smithi120 ceph-mon[43961]: 6.2 starting backfill to osd.11 from (0'0,0'0] MAX to 569'15328 2024-05-15T13:11:46.767 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:46 smithi177 ceph-mon[44303]: osdmap e573: 12 total, 12 up, 10 in 2024-05-15T13:11:46.768 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:46 smithi177 ceph-mon[44303]: pgmap v1536: 27 pgs: 1 peering, 1 active+undersized+remapped, 25 active+clean; 902 MiB data, 7.5 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 792 op/s 2024-05-15T13:11:47.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:46 smithi027 ceph-mon[39795]: osdmap e573: 12 total, 12 up, 10 in 2024-05-15T13:11:47.024 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:46 smithi027 ceph-mon[39795]: pgmap v1536: 27 pgs: 1 peering, 1 active+undersized+remapped, 25 active+clean; 902 MiB data, 7.5 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 792 op/s 2024-05-15T13:11:47.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:46 smithi120 ceph-mon[43961]: osdmap e573: 12 total, 12 up, 10 in 2024-05-15T13:11:47.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:46 smithi120 ceph-mon[43961]: pgmap v1536: 27 pgs: 1 peering, 1 active+undersized+remapped, 25 active+clean; 902 MiB data, 7.5 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 792 op/s 2024-05-15T13:11:47.065 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:11:47.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:47.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:11:47.174 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:11:47.218 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:47.258 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:11:47.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:47.261 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:11:47.361 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:11:47.413 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:47.433 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:11:47.433 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T13:11:47.433 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-05-15T13:11:47.433 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-05-15T13:11:47.433 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:11:47.446 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:11:47.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:47.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:11:47.551 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:11:47.597 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:47.624 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:11:47.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:47.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:11:47.732 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:11:47.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:47 smithi027 ceph-mon[39795]: osdmap e574: 12 total, 12 up, 10 in 2024-05-15T13:11:47.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:47 smithi120 ceph-mon[43961]: osdmap e574: 12 total, 12 up, 10 in 2024-05-15T13:11:47.797 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:47.808 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:47 smithi177 ceph-mon[44303]: osdmap e574: 12 total, 12 up, 10 in 2024-05-15T13:11:47.937 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:11:47.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:47.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:11:48.051 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:11:48.103 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:48.135 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:11:48.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:48.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:11:48.169 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-05-15T13:11:48.249 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:11:48.320 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:48.374 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:11:48.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:48.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:11:48.419 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:11:48.465 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:11:48.467 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:11:48.479 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:11:48.507 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:11:48.513 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:48.548 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:11:48.591 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:11:48.633 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:48 smithi027 ceph-mon[39795]: 1.9 scrub starts 2024-05-15T13:11:48.633 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:48 smithi027 ceph-mon[39795]: 1.9 scrub ok 2024-05-15T13:11:48.633 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:48 smithi027 ceph-mon[39795]: pgmap v1538: 27 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 peering, 1 active+undersized+remapped, 22 active+clean; 805 MiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 553 op/s; 1694/38601 objects misplaced (4.388%); 7.2 MiB/s, 115 objects/s recovering 2024-05-15T13:11:48.634 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:48 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1914866120' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:11:48.771 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:48 smithi177 ceph-mon[44303]: 1.9 scrub starts 2024-05-15T13:11:48.771 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:48 smithi177 ceph-mon[44303]: 1.9 scrub ok 2024-05-15T13:11:48.771 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:48 smithi177 ceph-mon[44303]: pgmap v1538: 27 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 peering, 1 active+undersized+remapped, 22 active+clean; 805 MiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 553 op/s; 1694/38601 objects misplaced (4.388%); 7.2 MiB/s, 115 objects/s recovering 2024-05-15T13:11:48.771 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:48 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1914866120' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:11:48.778 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:11:48.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:48.782 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:11:48.839 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:11:48.873 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:11:48.883 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:11:48.912 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:11:48.912 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:48.950 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:11:48.988 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:11:48.991 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:11:48.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:48.995 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:11:49.028 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:11:49.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:48 smithi120 ceph-mon[43961]: 1.9 scrub starts 2024-05-15T13:11:49.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:48 smithi120 ceph-mon[43961]: 1.9 scrub ok 2024-05-15T13:11:49.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:48 smithi120 ceph-mon[43961]: pgmap v1538: 27 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 peering, 1 active+undersized+remapped, 22 active+clean; 805 MiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 553 op/s; 1694/38601 objects misplaced (4.388%); 7.2 MiB/s, 115 objects/s recovering 2024-05-15T13:11:49.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:48 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1914866120' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:11:49.095 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:11:49.097 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:11:49.144 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:11:49.144 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:11:49.144 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:11:49.144 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:11:49.144 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:11:49.145 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:11:49.145 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:11:49.145 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:11:49.145 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:11:49.145 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:11:49.145 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:11:49.145 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:11:49.151 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:49.387 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:11:49.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:49.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:11:49.493 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:11:49.535 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:49.565 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:11:49.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:49.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:11:49.663 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:11:49.678 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:11:49.758 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:49.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:49 smithi120 ceph-mon[43961]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T13:11:49.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:49 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4228800793' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:11:49.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:49 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:11:49.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:49 smithi177 ceph-mon[44303]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T13:11:49.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:49 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4228800793' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:11:49.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:49 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:11:49.977 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:11:49.980 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:49 smithi027 ceph-mon[39795]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T13:11:49.980 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:49 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4228800793' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:11:49.980 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:49 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:11:49.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:49.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:11:50.008 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 8, 9, 10, 11] out_osds: [7, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:11:50.008 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:11:50.008 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-05-15T13:11:50.008 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-05-15T13:11:50.087 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:11:50.176 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:50.280 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:11:50.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:50.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:11:50.395 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:11:50.445 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:50.478 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:11:50.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:50.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:11:50.583 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:11:50.620 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:50.787 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:50 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:11:50.787 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:50 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:11:50.787 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:50 smithi120 ceph-mon[43961]: osdmap e575: 12 total, 12 up, 10 in 2024-05-15T13:11:50.787 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:50 smithi120 ceph-mon[43961]: pgmap v1540: 27 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 peering, 22 active+clean; 708 MiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 646 op/s; 2044/33984 objects misplaced (6.015%); 10 MiB/s, 166 objects/s recovering 2024-05-15T13:11:50.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:50 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:11:50.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:50 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:11:50.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:50 smithi177 ceph-mon[44303]: osdmap e575: 12 total, 12 up, 10 in 2024-05-15T13:11:50.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:50 smithi177 ceph-mon[44303]: pgmap v1540: 27 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 peering, 22 active+clean; 708 MiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 646 op/s; 2044/33984 objects misplaced (6.015%); 10 MiB/s, 166 objects/s recovering 2024-05-15T13:11:50.970 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:11:50.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:50.974 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:11:51.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:50 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:11:51.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:50 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:11:51.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:50 smithi027 ceph-mon[39795]: osdmap e575: 12 total, 12 up, 10 in 2024-05-15T13:11:51.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:50 smithi027 ceph-mon[39795]: pgmap v1540: 27 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 peering, 22 active+clean; 708 MiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 646 op/s; 2044/33984 objects misplaced (6.015%); 10 MiB/s, 166 objects/s recovering 2024-05-15T13:11:51.075 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:11:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:11:51.153 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:51.366 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:11:51.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:51.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:11:51.470 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:11:51.511 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:51.530 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:11:51.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:51.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:11:51.634 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:11:51.663 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:52.019 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:11:52.215 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:11:52.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:52.218 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:11:52.319 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:11:52.347 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:52.512 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:11:52.512 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:11:52.596 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:11:52.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:52.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:11:52.629 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:11:52.699 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:11:52.730 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:52.754 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:11:52.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:52.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:11:52.861 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:52 smithi027 ceph-mon[39795]: osdmap e576: 12 total, 12 up, 10 in 2024-05-15T13:11:52.861 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:52 smithi027 ceph-mon[39795]: pgmap v1542: 27 pgs: 1 peering, 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 23 active+clean; 681 MiB data, 7.1 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 590 op/s; 1532/32661 objects misplaced (4.691%); 14 MiB/s, 230 objects/s recovering 2024-05-15T13:11:52.864 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:11:52.893 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:52 smithi177 ceph-mon[44303]: osdmap e576: 12 total, 12 up, 10 in 2024-05-15T13:11:52.893 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:52 smithi177 ceph-mon[44303]: pgmap v1542: 27 pgs: 1 peering, 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 23 active+clean; 681 MiB data, 7.1 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 590 op/s; 1532/32661 objects misplaced (4.691%); 14 MiB/s, 230 objects/s recovering 2024-05-15T13:11:52.901 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:11:52.902 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:53.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:52 smithi120 ceph-mon[43961]: osdmap e576: 12 total, 12 up, 10 in 2024-05-15T13:11:53.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:52 smithi120 ceph-mon[43961]: pgmap v1542: 27 pgs: 1 peering, 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 23 active+clean; 681 MiB data, 7.1 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 590 op/s; 1532/32661 objects misplaced (4.691%); 14 MiB/s, 230 objects/s recovering 2024-05-15T13:11:53.249 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-05-15T13:11:53.259 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:11:53.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:53.262 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:11:53.362 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:11:53.390 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:53.409 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:11:53.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:53.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:11:53.504 INFO:teuthology.orchestra.run.smithi027.stdout:moved 0 / 81 (0%) 2024-05-15T13:11:53.504 INFO:teuthology.orchestra.run.smithi027.stdout:avg 8.1 2024-05-15T13:11:53.504 INFO:teuthology.orchestra.run.smithi027.stdout:stddev 2.3 -> 2.3 (expected baseline 2.7) 2024-05-15T13:11:53.504 INFO:teuthology.orchestra.run.smithi027.stdout:min osd.11 with 4 -> 4 pgs (0.493827 -> 0.493827 * mean) 2024-05-15T13:11:53.504 INFO:teuthology.orchestra.run.smithi027.stdout:max osd.6 with 13 -> 13 pgs (1.60494 -> 1.60494 * mean) 2024-05-15T13:11:53.504 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:11:53.504 INFO:teuthology.orchestra.run.smithi027.stdout:oload 1000 2024-05-15T13:11:53.504 INFO:teuthology.orchestra.run.smithi027.stdout:max_change 0.05 2024-05-15T13:11:53.504 INFO:teuthology.orchestra.run.smithi027.stdout:max_change_osds 4 2024-05-15T13:11:53.504 INFO:teuthology.orchestra.run.smithi027.stdout:average_utilization 0.0078 2024-05-15T13:11:53.505 INFO:teuthology.orchestra.run.smithi027.stdout:overload_utilization 0.0782 2024-05-15T13:11:53.506 INFO:teuthology.orchestra.run.smithi027.stderr:no change 2024-05-15T13:11:53.513 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:11:53.515 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:11:53.544 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:54.025 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:53 smithi027 ceph-mon[39795]: osdmap e577: 12 total, 12 up, 10 in 2024-05-15T13:11:54.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:53 smithi120 ceph-mon[43961]: osdmap e577: 12 total, 12 up, 10 in 2024-05-15T13:11:54.042 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:53 smithi177 ceph-mon[44303]: osdmap e577: 12 total, 12 up, 10 in 2024-05-15T13:11:54.527 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-05-15T13:11:54.527 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight-by-utilization 110 1.0 2024-05-15T13:11:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:11:54.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:54.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:11:54.644 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:11:54.673 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:54.706 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:11:54.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:54.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:11:54.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:54 smithi027 ceph-mon[39795]: from='client.16026 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T13:11:54.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:54 smithi027 ceph-mon[39795]: osdmap e578: 12 total, 12 up, 10 in 2024-05-15T13:11:54.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:54 smithi027 ceph-mon[39795]: pgmap v1545: 27 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 peering, 24 active+clean; 550 MiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 696 op/s; 496/26376 objects misplaced (1.880%); 16 MiB/s, 256 objects/s recovering 2024-05-15T13:11:54.811 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:11:54.840 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:54 smithi177 ceph-mon[44303]: from='client.16026 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T13:11:54.840 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:54 smithi177 ceph-mon[44303]: osdmap e578: 12 total, 12 up, 10 in 2024-05-15T13:11:54.841 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:54 smithi177 ceph-mon[44303]: pgmap v1545: 27 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 peering, 24 active+clean; 550 MiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 696 op/s; 496/26376 objects misplaced (1.880%); 16 MiB/s, 256 objects/s recovering 2024-05-15T13:11:54.844 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:54.870 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:11:54.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:54.873 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:11:54.974 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:11:55.016 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:54 smithi120 ceph-mon[43961]: from='client.16026 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T13:11:55.016 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:54 smithi120 ceph-mon[43961]: osdmap e578: 12 total, 12 up, 10 in 2024-05-15T13:11:55.016 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:54 smithi120 ceph-mon[43961]: pgmap v1545: 27 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 peering, 24 active+clean; 550 MiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 696 op/s; 496/26376 objects misplaced (1.880%); 16 MiB/s, 256 objects/s recovering 2024-05-15T13:11:55.018 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:55.041 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:11:55.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:55.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:11:55.084 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:11:55.130 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:11:55.146 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:11:55.172 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:55.200 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:11:55.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:55.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:11:55.308 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:11:55.341 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:55.366 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:11:55.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:55.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:11:55.470 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:11:55.512 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:55.536 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:11:55.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:55.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:11:55.570 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:11:55.571 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:11:55.640 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:11:55.714 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:56.301 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:56 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/925480215' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:11:56.334 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-05-15T13:11:56.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:56 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/925480215' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:11:56.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:56 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/925480215' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:11:56.692 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:11:56.735 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:11:56.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:56.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:11:56.843 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:11:56.879 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:56.938 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:11:56.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:56.944 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:11:57.045 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:11:57.092 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:57.175 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:11:57.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:57.184 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:11:57.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:57 smithi177 ceph-mon[44303]: pgmap v1546: 27 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 peering, 24 active+clean; 519 MiB data, 6.7 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 504 op/s; 496/24894 objects misplaced (1.992%); 12 MiB/s, 195 objects/s recovering 2024-05-15T13:11:57.276 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:57 smithi027 ceph-mon[39795]: pgmap v1546: 27 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 peering, 24 active+clean; 519 MiB data, 6.7 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 504 op/s; 496/24894 objects misplaced (1.992%); 12 MiB/s, 195 objects/s recovering 2024-05-15T13:11:57.284 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:11:57.323 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:57.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:57 smithi120 ceph-mon[43961]: pgmap v1546: 27 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 peering, 24 active+clean; 519 MiB data, 6.7 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 504 op/s; 496/24894 objects misplaced (1.992%); 12 MiB/s, 195 objects/s recovering 2024-05-15T13:11:57.533 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:11:57.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:57.535 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:11:57.636 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:11:57.756 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:57.883 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:11:57.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:57.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:11:57.989 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:11:58.012 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:11:58.012 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:11:58.012 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:11:58.012 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:11:58.012 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:11:58.012 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:11:58.012 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:11:58.012 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:11:58.012 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:11:58.012 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:11:58.012 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:11:58.013 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:11:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:11:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:11:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:11:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:11:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:11:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:11:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:11:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:11:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:11:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:11:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:11:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:11:58.054 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:11:58.150 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:11:58.394 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:58 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/816871370' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:11:58.394 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:58 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:11:58.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:58 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/816871370' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:11:58.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:58 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:11:58.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:58 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/816871370' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:11:58.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:58 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:11:58.730 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:11:58.730 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T13:11:58.730 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-05-15T13:11:58.730 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-05-15T13:11:58.730 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:11:59.160 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:59 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:11:59.160 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:59 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:11:59.160 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:59 smithi027 ceph-mon[39795]: osdmap e579: 12 total, 12 up, 10 in 2024-05-15T13:11:59.160 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:11:59 smithi027 ceph-mon[39795]: pgmap v1548: 27 pgs: 1 active+remapped+backfill_toofull, 26 active+clean; 422 MiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 690 op/s; 327/20232 objects misplaced (1.616%); 7.6 MiB/s, 122 objects/s recovering 2024-05-15T13:11:59.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:59 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:11:59.404 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:59 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:11:59.404 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:59 smithi177 ceph-mon[44303]: osdmap e579: 12 total, 12 up, 10 in 2024-05-15T13:11:59.404 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:11:59 smithi177 ceph-mon[44303]: pgmap v1548: 27 pgs: 1 active+remapped+backfill_toofull, 26 active+clean; 422 MiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 690 op/s; 327/20232 objects misplaced (1.616%); 7.6 MiB/s, 122 objects/s recovering 2024-05-15T13:11:59.437 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-05-15T13:11:59.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:59 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:11:59.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:59 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:11:59.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:59 smithi120 ceph-mon[43961]: osdmap e579: 12 total, 12 up, 10 in 2024-05-15T13:11:59.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:11:59 smithi120 ceph-mon[43961]: pgmap v1548: 27 pgs: 1 active+remapped+backfill_toofull, 26 active+clean; 422 MiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 690 op/s; 327/20232 objects misplaced (1.616%); 7.6 MiB/s, 122 objects/s recovering 2024-05-15T13:11:59.786 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:11:59.869 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:11:59.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:11:59.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:11:59.974 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:12:00.009 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:00.094 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:12:00.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:00.098 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:12:00.199 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:12:00.228 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:00.251 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:12:00.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:00.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:12:00.362 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:12:00.399 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:00.421 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:12:00.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:00.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:12:00.526 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:12:00.611 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:12:00.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:00.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:12:00.774 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:12:00.826 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:00.850 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:12:00.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:00.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:12:00.914 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:12:00.957 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:12:00.990 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:01.107 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:12:01.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:01.111 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:12:01.212 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:12:01.239 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:01.306 INFO:teuthology.orchestra.run.smithi027.stdout:moved 8 / 81 (9.87654%) 2024-05-15T13:12:01.307 INFO:teuthology.orchestra.run.smithi027.stdout:avg 8.1 2024-05-15T13:12:01.307 INFO:teuthology.orchestra.run.smithi027.stdout:stddev 2.3 -> 2.3 (expected baseline 2.7) 2024-05-15T13:12:01.307 INFO:teuthology.orchestra.run.smithi027.stdout:min osd.11 with 4 -> 6 pgs (0.493827 -> 0.740741 * mean) 2024-05-15T13:12:01.307 INFO:teuthology.orchestra.run.smithi027.stdout:max osd.6 with 13 -> 11 pgs (1.60494 -> 1.35802 * mean) 2024-05-15T13:12:01.307 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:12:01.307 INFO:teuthology.orchestra.run.smithi027.stdout:oload 110 2024-05-15T13:12:01.307 INFO:teuthology.orchestra.run.smithi027.stdout:max_change 1 2024-05-15T13:12:01.307 INFO:teuthology.orchestra.run.smithi027.stdout:max_change_osds 4 2024-05-15T13:12:01.307 INFO:teuthology.orchestra.run.smithi027.stdout:average_utilization 0.0069 2024-05-15T13:12:01.307 INFO:teuthology.orchestra.run.smithi027.stdout:overload_utilization 0.0076 2024-05-15T13:12:01.307 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9 weight 1.0000 -> 0.8410 2024-05-15T13:12:01.307 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 weight 1.0000 -> 0.7753 2024-05-15T13:12:01.307 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 weight 1.0000 -> 0.8302 2024-05-15T13:12:01.307 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6 weight 1.0000 -> 0.9071 2024-05-15T13:12:01.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:01 smithi177 ceph-mon[44303]: pgmap v1549: 27 pgs: 1 active+remapped+backfill_toofull, 26 active+clean; 297 MiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 709 op/s; 327/14256 objects misplaced (2.294%); 2.5 MiB/s, 39 objects/s recovering 2024-05-15T13:12:01.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:01 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'54407','3':'50808','6':'59447','9':'55119'}"}]: dispatch 2024-05-15T13:12:01.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:01 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'54407','3':'50808','6':'59447','9':'55119'}"}]: dispatch 2024-05-15T13:12:01.444 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:12:01.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:01.447 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:12:01.548 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:12:01.549 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:12:01.575 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:01 smithi027 ceph-mon[39795]: pgmap v1549: 27 pgs: 1 active+remapped+backfill_toofull, 26 active+clean; 297 MiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 709 op/s; 327/14256 objects misplaced (2.294%); 2.5 MiB/s, 39 objects/s recovering 2024-05-15T13:12:01.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:01 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'54407','3':'50808','6':'59447','9':'55119'}"}]: dispatch 2024-05-15T13:12:01.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:01 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'54407','3':'50808','6':'59447','9':'55119'}"}]: dispatch 2024-05-15T13:12:01.578 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:01.692 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:12:01.692 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-05-15T13:12:01.692 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-05-15T13:12:01.692 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T13:12:01.692 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.692 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:19088 5.4 5:2a33d050:::benchmark_data_smithi177_109991_object19087:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:12:01.692 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:27.083242+0000", 2024-05-15T13:12:01.692 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 514.61483272500004, 2024-05-15T13:12:01.692 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.092245548999999996, 2024-05-15T13:12:01.693 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:12:01.693 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:12:01.693 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:12:01.693 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:12:01.693 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:12:01.693 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 19088 2024-05-15T13:12:01.693 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.693 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:12:01.693 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.693 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:12:01.693 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.083242+0000", 2024-05-15T13:12:01.693 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.694 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.694 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.694 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:12:01.694 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.083242+0000", 2024-05-15T13:12:01.694 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.694 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.694 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.694 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:12:01.694 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.083242+0000", 2024-05-15T13:12:01.694 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3e-07 2024-05-15T13:12:01.694 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.694 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.695 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:12:01.695 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.083270+0000", 2024-05-15T13:12:01.695 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7640999999999998e-05 2024-05-15T13:12:01.695 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.695 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.695 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:12:01.695 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.083279+0000", 2024-05-15T13:12:01.695 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8200000000000003e-06 2024-05-15T13:12:01.695 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.695 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.695 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:12:01.695 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.083283+0000", 2024-05-15T13:12:01.695 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.442e-06 2024-05-15T13:12:01.695 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.696 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.696 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:12:01.696 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.083332+0000", 2024-05-15T13:12:01.696 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9081e-05 2024-05-15T13:12:01.696 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.696 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.696 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:12:01.696 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.083459+0000", 2024-05-15T13:12:01.696 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012679400000000001 2024-05-15T13:12:01.696 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.696 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.696 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:12:01.696 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.083623+0000", 2024-05-15T13:12:01.696 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000163917 2024-05-15T13:12:01.697 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.697 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.697 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:12:01.697 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.085789+0000", 2024-05-15T13:12:01.697 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021660130000000001 2024-05-15T13:12:01.697 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.697 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.697 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.697 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.087340+0000", 2024-05-15T13:12:01.697 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015510140000000001 2024-05-15T13:12:01.697 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.697 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.697 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.697 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175382+0000", 2024-05-15T13:12:01.698 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.088042001999999994 2024-05-15T13:12:01.698 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.698 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.698 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:12:01.698 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175442+0000", 2024-05-15T13:12:01.698 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9800999999999999e-05 2024-05-15T13:12:01.698 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.698 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.698 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:12:01.698 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175487+0000", 2024-05-15T13:12:01.698 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5293999999999997e-05 2024-05-15T13:12:01.698 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.698 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:12:01.699 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.699 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.699 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.699 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:19089 5.4 5:2f920fea:::benchmark_data_smithi177_109991_object19088:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:12:01.699 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:27.084059+0000", 2024-05-15T13:12:01.699 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 514.61401540500003, 2024-05-15T13:12:01.699 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.091531791000000001, 2024-05-15T13:12:01.699 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:12:01.699 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:12:01.699 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:12:01.699 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:12:01.699 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:12:01.699 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 19089 2024-05-15T13:12:01.699 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.699 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:12:01.700 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.700 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:12:01.700 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084059+0000", 2024-05-15T13:12:01.700 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.700 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.700 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.700 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:12:01.700 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084059+0000", 2024-05-15T13:12:01.700 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.700 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.700 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.700 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:12:01.700 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084060+0000", 2024-05-15T13:12:01.701 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0999999999999998e-07 2024-05-15T13:12:01.701 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.701 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.701 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:12:01.701 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084079+0000", 2024-05-15T13:12:01.701 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9014000000000001e-05 2024-05-15T13:12:01.701 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.701 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.701 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:12:01.701 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084088+0000", 2024-05-15T13:12:01.701 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5950000000000005e-06 2024-05-15T13:12:01.702 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.702 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.702 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:12:01.702 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084094+0000", 2024-05-15T13:12:01.702 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9009999999999999e-06 2024-05-15T13:12:01.702 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.702 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.702 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:12:01.702 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084276+0000", 2024-05-15T13:12:01.702 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000181795 2024-05-15T13:12:01.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.703 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.703 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:12:01.703 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084402+0000", 2024-05-15T13:12:01.703 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012629699999999999 2024-05-15T13:12:01.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.703 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.703 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:12:01.703 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084542+0000", 2024-05-15T13:12:01.703 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013983999999999999 2024-05-15T13:12:01.704 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.704 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.704 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:12:01.704 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.085797+0000", 2024-05-15T13:12:01.704 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012547350000000001 2024-05-15T13:12:01.704 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.704 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.704 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.704 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.087410+0000", 2024-05-15T13:12:01.704 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016136309999999999 2024-05-15T13:12:01.704 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.704 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.704 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.704 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175516+0000", 2024-05-15T13:12:01.704 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.088105476000000002 2024-05-15T13:12:01.704 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.705 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.705 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:12:01.705 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175554+0000", 2024-05-15T13:12:01.705 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8308e-05 2024-05-15T13:12:01.705 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.705 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.705 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:12:01.705 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175591+0000", 2024-05-15T13:12:01.705 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6489000000000002e-05 2024-05-15T13:12:01.705 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.705 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:12:01.705 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.705 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.705 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.706 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:19091 5.4 5:23c909f8:::benchmark_data_smithi177_109991_object19090:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:12:01.706 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:27.084143+0000", 2024-05-15T13:12:01.706 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 514.61393102099998, 2024-05-15T13:12:01.706 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.091549106000000005, 2024-05-15T13:12:01.706 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:12:01.706 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:12:01.706 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:12:01.706 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:12:01.706 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:12:01.706 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 19091 2024-05-15T13:12:01.706 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.706 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:12:01.706 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.706 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:12:01.707 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084143+0000", 2024-05-15T13:12:01.707 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.707 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.707 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.707 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:12:01.707 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084143+0000", 2024-05-15T13:12:01.707 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.707 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.707 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.707 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:12:01.707 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084144+0000", 2024-05-15T13:12:01.707 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5599999999999995e-07 2024-05-15T13:12:01.707 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.707 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.708 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:12:01.708 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084162+0000", 2024-05-15T13:12:01.708 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8422000000000001e-05 2024-05-15T13:12:01.708 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.708 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.708 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:12:01.709 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084170+0000", 2024-05-15T13:12:01.709 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2770000000000005e-06 2024-05-15T13:12:01.709 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.709 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.709 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:12:01.710 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.084176+0000", 2024-05-15T13:12:01.710 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4020000000000001e-06 2024-05-15T13:12:01.710 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.710 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.710 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:12:01.710 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.085231+0000", 2024-05-15T13:12:01.711 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001055432 2024-05-15T13:12:01.711 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.711 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.711 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:12:01.711 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.085311+0000", 2024-05-15T13:12:01.712 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9831000000000002e-05 2024-05-15T13:12:01.712 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.712 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.712 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:12:01.712 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.085390+0000", 2024-05-15T13:12:01.713 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8793000000000003e-05 2024-05-15T13:12:01.713 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.713 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.713 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:12:01.713 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.086012+0000", 2024-05-15T13:12:01.713 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00062231899999999998 2024-05-15T13:12:01.714 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.714 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.714 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.714 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.087445+0000", 2024-05-15T13:12:01.714 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001432675 2024-05-15T13:12:01.715 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.715 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.715 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.715 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175628+0000", 2024-05-15T13:12:01.715 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.088183239999999996 2024-05-15T13:12:01.715 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.717 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.717 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:12:01.717 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175658+0000", 2024-05-15T13:12:01.717 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9607e-05 2024-05-15T13:12:01.718 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.718 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.718 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:12:01.718 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175692+0000", 2024-05-15T13:12:01.718 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4551999999999997e-05 2024-05-15T13:12:01.719 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.719 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:12:01.719 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.719 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.719 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.719 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:19097 5.4 5:2620206e:::benchmark_data_smithi177_109991_object19096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:12:01.720 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:27.086354+0000", 2024-05-15T13:12:01.720 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 514.61172003599995, 2024-05-15T13:12:01.720 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.089473733, 2024-05-15T13:12:01.720 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:12:01.720 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:12:01.720 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:12:01.721 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:12:01.721 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:12:01.721 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 19097 2024-05-15T13:12:01.721 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.721 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:12:01.722 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.722 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:12:01.722 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.086354+0000", 2024-05-15T13:12:01.722 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.722 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.722 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.723 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:12:01.723 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.086354+0000", 2024-05-15T13:12:01.723 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.723 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.723 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.723 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:12:01.724 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.086355+0000", 2024-05-15T13:12:01.724 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.018e-06 2024-05-15T13:12:01.724 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.724 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.724 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:12:01.724 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.086377+0000", 2024-05-15T13:12:01.725 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1515999999999999e-05 2024-05-15T13:12:01.725 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.725 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.725 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:12:01.725 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.086390+0000", 2024-05-15T13:12:01.726 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2984e-05 2024-05-15T13:12:01.726 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.726 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.726 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:12:01.726 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.086398+0000", 2024-05-15T13:12:01.726 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2260000000000005e-06 2024-05-15T13:12:01.727 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.727 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.727 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:12:01.727 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.086460+0000", 2024-05-15T13:12:01.727 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1606999999999995e-05 2024-05-15T13:12:01.727 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.727 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.728 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:12:01.728 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.086603+0000", 2024-05-15T13:12:01.728 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014295099999999999 2024-05-15T13:12:01.728 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.728 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.728 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:12:01.729 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.086731+0000", 2024-05-15T13:12:01.729 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012876599999999999 2024-05-15T13:12:01.729 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.729 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.729 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:12:01.729 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.088423+0000", 2024-05-15T13:12:01.730 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001691407 2024-05-15T13:12:01.730 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.730 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.730 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.730 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.088547+0000", 2024-05-15T13:12:01.731 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000124175 2024-05-15T13:12:01.731 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.731 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.731 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.731 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175735+0000", 2024-05-15T13:12:01.731 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.087188584999999999 2024-05-15T13:12:01.732 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.732 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.732 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:12:01.732 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175791+0000", 2024-05-15T13:12:01.732 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5371000000000002e-05 2024-05-15T13:12:01.732 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.732 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.733 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:12:01.733 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175828+0000", 2024-05-15T13:12:01.733 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7126999999999999e-05 2024-05-15T13:12:01.733 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.733 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:12:01.733 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.734 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.734 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.734 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:19105 5.4 5:2c08e1d6:::benchmark_data_smithi177_109991_object19104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:12:01.734 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:27.090769+0000", 2024-05-15T13:12:01.734 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 514.60730540099996, 2024-05-15T13:12:01.735 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.085190499000000003, 2024-05-15T13:12:01.735 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:12:01.735 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:12:01.735 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:12:01.735 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:12:01.735 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:12:01.736 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 19105 2024-05-15T13:12:01.736 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.736 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:12:01.736 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.736 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:12:01.736 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.090769+0000", 2024-05-15T13:12:01.737 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.737 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.737 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.737 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:12:01.737 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.090769+0000", 2024-05-15T13:12:01.737 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.738 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.738 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.738 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:12:01.738 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.090770+0000", 2024-05-15T13:12:01.738 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4980000000000001e-06 2024-05-15T13:12:01.739 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.739 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.739 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:12:01.739 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.090793+0000", 2024-05-15T13:12:01.739 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2254000000000001e-05 2024-05-15T13:12:01.739 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.740 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.740 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:12:01.740 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.090804+0000", 2024-05-15T13:12:01.740 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0939e-05 2024-05-15T13:12:01.740 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.741 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:12:01.741 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.090820+0000", 2024-05-15T13:12:01.741 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6371e-05 2024-05-15T13:12:01.741 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:12:01.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.090864+0000", 2024-05-15T13:12:01.742 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4508000000000001e-05 2024-05-15T13:12:01.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:12:01.743 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.090964+0000", 2024-05-15T13:12:01.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9105999999999996e-05 2024-05-15T13:12:01.743 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.743 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.743 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:12:01.743 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.091051+0000", 2024-05-15T13:12:01.744 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7614999999999994e-05 2024-05-15T13:12:01.744 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:12:01.744 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.091803+0000", 2024-05-15T13:12:01.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00075175199999999995 2024-05-15T13:12:01.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.745 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.092303+0000", 2024-05-15T13:12:01.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00050048699999999998 2024-05-15T13:12:01.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175869+0000", 2024-05-15T13:12:01.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.083565523000000003 2024-05-15T13:12:01.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.747 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:12:01.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175922+0000", 2024-05-15T13:12:01.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2966000000000001e-05 2024-05-15T13:12:01.747 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:12:01.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.175959+0000", 2024-05-15T13:12:01.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.748e-05 2024-05-15T13:12:01.748 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.748 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:12:01.749 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.749 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:20882 5.4 5:2045f33d:::benchmark_data_smithi177_109991_object20881:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:12:01.749 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:27.874815+0000", 2024-05-15T13:12:01.749 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 513.82325875799995, 2024-05-15T13:12:01.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.078067048, 2024-05-15T13:12:01.750 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:12:01.750 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:12:01.750 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:12:01.750 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:12:01.751 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:12:01.751 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 20882 2024-05-15T13:12:01.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.751 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:12:01.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.752 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:12:01.752 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.874815+0000", 2024-05-15T13:12:01.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.752 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.752 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.752 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:12:01.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.874815+0000", 2024-05-15T13:12:01.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:12:01.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.874817+0000", 2024-05-15T13:12:01.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.311e-06 2024-05-15T13:12:01.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:12:01.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.874842+0000", 2024-05-15T13:12:01.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4723e-05 2024-05-15T13:12:01.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.755 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:12:01.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.874853+0000", 2024-05-15T13:12:01.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1773999999999999e-05 2024-05-15T13:12:01.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.756 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:12:01.756 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.874861+0000", 2024-05-15T13:12:01.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0970000000000006e-06 2024-05-15T13:12:01.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:12:01.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.874913+0000", 2024-05-15T13:12:01.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1573000000000002e-05 2024-05-15T13:12:01.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:12:01.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875032+0000", 2024-05-15T13:12:01.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000118847 2024-05-15T13:12:01.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.759 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:12:01.759 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875161+0000", 2024-05-15T13:12:01.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012939099999999999 2024-05-15T13:12:01.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952687+0000", 2024-05-15T13:12:01.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.077526088000000007 2024-05-15T13:12:01.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:12:01.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952700+0000", 2024-05-15T13:12:01.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2442e-05 2024-05-15T13:12:01.761 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.762 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952731+0000", 2024-05-15T13:12:01.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1745999999999997e-05 2024-05-15T13:12:01.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.762 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:12:01.763 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952833+0000", 2024-05-15T13:12:01.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010121800000000001 2024-05-15T13:12:01.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.763 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:12:01.763 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952883+0000", 2024-05-15T13:12:01.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9838e-05 2024-05-15T13:12:01.764 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.764 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:12:01.764 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.765 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:20879 5.2 e424/418 5:45079427:::benchmark_data_smithi177_109991_object20878:head v 424'1264, mlcod=424'1262)", 2024-05-15T13:12:01.765 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:27.875224+0000", 2024-05-15T13:12:01.765 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 513.82285047100004, 2024-05-15T13:12:01.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.077611594000000006, 2024-05-15T13:12:01.765 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:12:01.765 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:12:01.766 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:12:01.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:12:01.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875224+0000", 2024-05-15T13:12:01.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:12:01.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875224+0000", 2024-05-15T13:12:01.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:12:01.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875224+0000", 2024-05-15T13:12:01.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9699999999999995e-07 2024-05-15T13:12:01.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:12:01.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875340+0000", 2024-05-15T13:12:01.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115752 2024-05-15T13:12:01.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:12:01.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875351+0000", 2024-05-15T13:12:01.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0864e-05 2024-05-15T13:12:01.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:12:01.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875358+0000", 2024-05-15T13:12:01.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7360000000000004e-06 2024-05-15T13:12:01.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.771 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:12:01.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875417+0000", 2024-05-15T13:12:01.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9159000000000001e-05 2024-05-15T13:12:01.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:12:01.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875439+0000", 2024-05-15T13:12:01.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2286999999999999e-05 2024-05-15T13:12:01.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.773 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:12:01.773 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952607+0000", 2024-05-15T13:12:01.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.077167432999999994 2024-05-15T13:12:01.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:12:01.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952835+0000", 2024-05-15T13:12:01.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022866600000000001 2024-05-15T13:12:01.774 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.774 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:12:01.775 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.775 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:20880 5.a e424/418 5:56abf9c6:::benchmark_data_smithi177_109991_object20879:head v 424'1300, mlcod=424'1297)", 2024-05-15T13:12:01.775 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:27.875370+0000", 2024-05-15T13:12:01.775 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 513.82270383599996, 2024-05-15T13:12:01.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.079926545000000002, 2024-05-15T13:12:01.776 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:12:01.776 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:12:01.776 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:12:01.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:12:01.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875370+0000", 2024-05-15T13:12:01.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:12:01.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875370+0000", 2024-05-15T13:12:01.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:12:01.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875371+0000", 2024-05-15T13:12:01.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8999999999999996e-07 2024-05-15T13:12:01.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:12:01.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875539+0000", 2024-05-15T13:12:01.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016774100000000001 2024-05-15T13:12:01.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:12:01.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875549+0000", 2024-05-15T13:12:01.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0366e-05 2024-05-15T13:12:01.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:12:01.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875556+0000", 2024-05-15T13:12:01.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8630000000000002e-06 2024-05-15T13:12:01.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:12:01.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875634+0000", 2024-05-15T13:12:01.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8335000000000001e-05 2024-05-15T13:12:01.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:12:01.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.875654+0000", 2024-05-15T13:12:01.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9457e-05 2024-05-15T13:12:01.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:12:01.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.955250+0000", 2024-05-15T13:12:01.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.079595770999999996 2024-05-15T13:12:01.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:12:01.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.955297+0000", 2024-05-15T13:12:01.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7422000000000001e-05 2024-05-15T13:12:01.784 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.784 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:12:01.784 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.784 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.15540.0:20882 5.4 e424/418 ondisk, result = 0)", 2024-05-15T13:12:01.784 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:27.876548+0000", 2024-05-15T13:12:01.784 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 513.82152576399994, 2024-05-15T13:12:01.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.076339893000000006, 2024-05-15T13:12:01.785 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:12:01.785 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T13:12:01.785 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:12:01.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:12:01.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.876548+0000", 2024-05-15T13:12:01.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:12:01.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.876548+0000", 2024-05-15T13:12:01.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:12:01.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.876549+0000", 2024-05-15T13:12:01.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3099999999999997e-07 2024-05-15T13:12:01.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:12:01.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.876554+0000", 2024-05-15T13:12:01.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.741e-06 2024-05-15T13:12:01.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:12:01.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952640+0000", 2024-05-15T13:12:01.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.076086283000000005 2024-05-15T13:12:01.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:12:01.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952647+0000", 2024-05-15T13:12:01.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9160000000000004e-06 2024-05-15T13:12:01.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:12:01.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952721+0000", 2024-05-15T13:12:01.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3801999999999996e-05 2024-05-15T13:12:01.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:12:01.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952727+0000", 2024-05-15T13:12:01.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2049999999999999e-06 2024-05-15T13:12:01.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:12:01.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952888+0000", 2024-05-15T13:12:01.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016131500000000001 2024-05-15T13:12:01.788 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.788 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:12:01.788 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.788 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.15540.0:20882 5.4 e424/418 ondisk, result = 0)", 2024-05-15T13:12:01.788 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:27.877367+0000", 2024-05-15T13:12:01.789 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 513.82070737100003, 2024-05-15T13:12:01.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.075333856000000005, 2024-05-15T13:12:01.789 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:12:01.789 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T13:12:01.789 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:12:01.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:12:01.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.877367+0000", 2024-05-15T13:12:01.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:12:01.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.877367+0000", 2024-05-15T13:12:01.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:12:01.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.877368+0000", 2024-05-15T13:12:01.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6499999999999999e-07 2024-05-15T13:12:01.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:12:01.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.877373+0000", 2024-05-15T13:12:01.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2650000000000004e-06 2024-05-15T13:12:01.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:12:01.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952602+0000", 2024-05-15T13:12:01.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.075229277999999997 2024-05-15T13:12:01.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:12:01.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952615+0000", 2024-05-15T13:12:01.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2619000000000001e-05 2024-05-15T13:12:01.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:12:01.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952672+0000", 2024-05-15T13:12:01.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6830999999999998e-05 2024-05-15T13:12:01.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:12:01.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952681+0000", 2024-05-15T13:12:01.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7089999999999997e-06 2024-05-15T13:12:01.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:12:01.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952701+0000", 2024-05-15T13:12:01.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9488999999999999e-05 2024-05-15T13:12:01.792 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.792 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:12:01.792 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.792 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:20881 5.0 e424/418 5:0cebd2d6:::benchmark_data_smithi177_109991_object20880:head v 424'1281, mlcod=424'1278)", 2024-05-15T13:12:01.793 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:27.877430+0000", 2024-05-15T13:12:01.793 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 513.820644485, 2024-05-15T13:12:01.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.078308953000000001, 2024-05-15T13:12:01.793 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:12:01.793 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:12:01.793 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:12:01.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:12:01.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.877430+0000", 2024-05-15T13:12:01.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:12:01.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.877430+0000", 2024-05-15T13:12:01.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:12:01.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.877430+0000", 2024-05-15T13:12:01.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1099999999999995e-07 2024-05-15T13:12:01.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:12:01.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.877494+0000", 2024-05-15T13:12:01.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3373000000000005e-05 2024-05-15T13:12:01.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:12:01.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952607+0000", 2024-05-15T13:12:01.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.075113048000000002 2024-05-15T13:12:01.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:12:01.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952616+0000", 2024-05-15T13:12:01.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5540000000000003e-06 2024-05-15T13:12:01.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:12:01.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952671+0000", 2024-05-15T13:12:01.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4685999999999998e-05 2024-05-15T13:12:01.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:12:01.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.952694+0000", 2024-05-15T13:12:01.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3263000000000001e-05 2024-05-15T13:12:01.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:12:01.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.955687+0000", 2024-05-15T13:12:01.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0029930960000000002 2024-05-15T13:12:01.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:12:01.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:27.955739+0000", 2024-05-15T13:12:01.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1221999999999998e-05 2024-05-15T13:12:01.798 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.798 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:12:01.798 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.798 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:23470 5.4 5:222094c6:::benchmark_data_smithi177_109991_object23469:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-05-15T13:12:01.798 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:29.032826+0000", 2024-05-15T13:12:01.798 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 512.66524795400005, 2024-05-15T13:12:01.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.070901916999999995, 2024-05-15T13:12:01.798 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:12:01.798 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:12:01.798 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:12:01.798 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:12:01.799 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:12:01.799 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 23470 2024-05-15T13:12:01.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.799 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:12:01.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:12:01.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.032826+0000", 2024-05-15T13:12:01.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:12:01.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.032826+0000", 2024-05-15T13:12:01.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:12:01.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.032828+0000", 2024-05-15T13:12:01.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3960000000000001e-06 2024-05-15T13:12:01.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:12:01.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.032847+0000", 2024-05-15T13:12:01.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9772e-05 2024-05-15T13:12:01.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:12:01.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.032860+0000", 2024-05-15T13:12:01.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2068e-05 2024-05-15T13:12:01.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:12:01.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.032865+0000", 2024-05-15T13:12:01.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7769999999999999e-06 2024-05-15T13:12:01.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:12:01.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.032916+0000", 2024-05-15T13:12:01.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0983999999999998e-05 2024-05-15T13:12:01.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:12:01.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.033027+0000", 2024-05-15T13:12:01.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111037 2024-05-15T13:12:01.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:12:01.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.033141+0000", 2024-05-15T13:12:01.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000113807 2024-05-15T13:12:01.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:12:01.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.034104+0000", 2024-05-15T13:12:01.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00096271699999999998 2024-05-15T13:12:01.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.034795+0000", 2024-05-15T13:12:01.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00069082200000000001 2024-05-15T13:12:01.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.103637+0000", 2024-05-15T13:12:01.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.068842293999999998 2024-05-15T13:12:01.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:12:01.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.103681+0000", 2024-05-15T13:12:01.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3886e-05 2024-05-15T13:12:01.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:12:01.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:29.103728+0000", 2024-05-15T13:12:01.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7357000000000001e-05 2024-05-15T13:12:01.804 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.804 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:12:01.804 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.804 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:72586 5.4 5:2111a196:::benchmark_data_smithi177_109991_object72585:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-05-15T13:12:01.804 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:03:55.531803+0000", 2024-05-15T13:12:01.804 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 486.16627091100003, 2024-05-15T13:12:01.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.085809504999999994, 2024-05-15T13:12:01.804 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:12:01.804 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:12:01.804 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:12:01.804 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:12:01.804 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:12:01.805 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 72586 2024-05-15T13:12:01.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.805 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:12:01.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:12:01.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.531803+0000", 2024-05-15T13:12:01.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:12:01.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.531803+0000", 2024-05-15T13:12:01.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:12:01.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.531805+0000", 2024-05-15T13:12:01.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9700000000000002e-06 2024-05-15T13:12:01.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:12:01.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.531837+0000", 2024-05-15T13:12:01.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2079999999999998e-05 2024-05-15T13:12:01.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:12:01.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.531851+0000", 2024-05-15T13:12:01.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3149999999999999e-05 2024-05-15T13:12:01.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:12:01.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.531859+0000", 2024-05-15T13:12:01.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0139999999999998e-06 2024-05-15T13:12:01.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:12:01.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.531903+0000", 2024-05-15T13:12:01.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4453000000000003e-05 2024-05-15T13:12:01.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:12:01.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.531990+0000", 2024-05-15T13:12:01.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6623000000000001e-05 2024-05-15T13:12:01.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:12:01.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.532076+0000", 2024-05-15T13:12:01.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6738999999999997e-05 2024-05-15T13:12:01.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:12:01.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.535817+0000", 2024-05-15T13:12:01.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003740686 2024-05-15T13:12:01.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.537104+0000", 2024-05-15T13:12:01.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001287414 2024-05-15T13:12:01.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.617520+0000", 2024-05-15T13:12:01.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.080415259000000003 2024-05-15T13:12:01.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:12:01.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.617559+0000", 2024-05-15T13:12:01.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8946000000000003e-05 2024-05-15T13:12:01.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:12:01.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:03:55.617613+0000", 2024-05-15T13:12:01.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4171e-05 2024-05-15T13:12:01.809 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.809 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:12:01.809 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.810 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:104783 5.4 5:2abd1f52:::benchmark_data_smithi177_109991_object104782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-05-15T13:12:01.810 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:04:20.671108+0000", 2024-05-15T13:12:01.810 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 461.02696602200001, 2024-05-15T13:12:01.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.074323771999999996, 2024-05-15T13:12:01.810 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:12:01.810 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:12:01.810 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:12:01.810 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:12:01.810 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:12:01.810 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 104783 2024-05-15T13:12:01.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.811 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:12:01.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:12:01.811 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.671108+0000", 2024-05-15T13:12:01.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:12:01.811 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.671108+0000", 2024-05-15T13:12:01.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:12:01.811 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.671110+0000", 2024-05-15T13:12:01.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7549999999999999e-06 2024-05-15T13:12:01.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:12:01.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.671132+0000", 2024-05-15T13:12:01.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1642999999999999e-05 2024-05-15T13:12:01.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:12:01.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.671147+0000", 2024-05-15T13:12:01.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5322e-05 2024-05-15T13:12:01.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:12:01.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.671156+0000", 2024-05-15T13:12:01.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8440000000000004e-06 2024-05-15T13:12:01.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:12:01.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.671244+0000", 2024-05-15T13:12:01.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7774000000000004e-05 2024-05-15T13:12:01.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:12:01.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.671363+0000", 2024-05-15T13:12:01.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000119726 2024-05-15T13:12:01.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:12:01.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.671489+0000", 2024-05-15T13:12:01.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000125831 2024-05-15T13:12:01.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:12:01.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.674533+0000", 2024-05-15T13:12:01.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0030441270000000002 2024-05-15T13:12:01.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.674718+0000", 2024-05-15T13:12:01.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000184945 2024-05-15T13:12:01.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.745334+0000", 2024-05-15T13:12:01.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.070615913000000002 2024-05-15T13:12:01.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:12:01.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.745373+0000", 2024-05-15T13:12:01.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8834000000000001e-05 2024-05-15T13:12:01.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:12:01.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.745432+0000", 2024-05-15T13:12:01.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9058000000000003e-05 2024-05-15T13:12:01.815 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.816 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:12:01.816 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.816 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:104792 5.4 5:2c256096:::benchmark_data_smithi177_109991_object104791:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-05-15T13:12:01.816 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:04:20.677560+0000", 2024-05-15T13:12:01.816 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 461.02051451300002, 2024-05-15T13:12:01.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.068042884999999997, 2024-05-15T13:12:01.816 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:12:01.816 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:12:01.816 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:12:01.816 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:12:01.816 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:12:01.816 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 104792 2024-05-15T13:12:01.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.817 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:12:01.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:12:01.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.677560+0000", 2024-05-15T13:12:01.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:12:01.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.677560+0000", 2024-05-15T13:12:01.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:12:01.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.677561+0000", 2024-05-15T13:12:01.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6810000000000001e-06 2024-05-15T13:12:01.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:12:01.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.677590+0000", 2024-05-15T13:12:01.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8867999999999999e-05 2024-05-15T13:12:01.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:12:01.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.677603+0000", 2024-05-15T13:12:01.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2909e-05 2024-05-15T13:12:01.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:12:01.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.677612+0000", 2024-05-15T13:12:01.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6419999999999996e-06 2024-05-15T13:12:01.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:12:01.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.677655+0000", 2024-05-15T13:12:01.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3653999999999999e-05 2024-05-15T13:12:01.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:12:01.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.677754+0000", 2024-05-15T13:12:01.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8735999999999998e-05 2024-05-15T13:12:01.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:12:01.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.677866+0000", 2024-05-15T13:12:01.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111468 2024-05-15T13:12:01.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.681408+0000", 2024-05-15T13:12:01.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035424139999999998 2024-05-15T13:12:01.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:12:01.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.681666+0000", 2024-05-15T13:12:01.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025800299999999998 2024-05-15T13:12:01.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.745490+0000", 2024-05-15T13:12:01.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.063823823000000002 2024-05-15T13:12:01.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:12:01.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.745547+0000", 2024-05-15T13:12:01.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6758000000000001e-05 2024-05-15T13:12:01.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:12:01.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:04:20.745603+0000", 2024-05-15T13:12:01.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5928999999999999e-05 2024-05-15T13:12:01.822 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.822 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:12:01.822 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.822 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:161212 5.4 5:2e5baf90:::benchmark_data_smithi177_109991_object26021:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-05-15T13:12:01.822 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:05:17.983286+0000", 2024-05-15T13:12:01.822 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 403.71478777800002, 2024-05-15T13:12:01.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.066825691000000007, 2024-05-15T13:12:01.823 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:12:01.823 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:12:01.823 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:12:01.823 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:12:01.823 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:12:01.823 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 161212 2024-05-15T13:12:01.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.823 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:12:01.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:12:01.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:17.983286+0000", 2024-05-15T13:12:01.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:12:01.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:17.983286+0000", 2024-05-15T13:12:01.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:12:01.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:17.983287+0000", 2024-05-15T13:12:01.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.34e-07 2024-05-15T13:12:01.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:12:01.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:17.983288+0000", 2024-05-15T13:12:01.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.187e-06 2024-05-15T13:12:01.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:12:01.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:17.983291+0000", 2024-05-15T13:12:01.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.022e-06 2024-05-15T13:12:01.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:12:01.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:17.983294+0000", 2024-05-15T13:12:01.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7700000000000002e-06 2024-05-15T13:12:01.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:12:01.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:18.032812+0000", 2024-05-15T13:12:01.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.049518262 2024-05-15T13:12:01.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:12:01.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:18.032942+0000", 2024-05-15T13:12:01.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012960099999999999 2024-05-15T13:12:01.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:12:01.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:18.033070+0000", 2024-05-15T13:12:01.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012830900000000001 2024-05-15T13:12:01.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:18.037918+0000", 2024-05-15T13:12:01.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0048480199999999998 2024-05-15T13:12:01.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:12:01.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:18.042168+0000", 2024-05-15T13:12:01.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0042497120000000001 2024-05-15T13:12:01.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:18.050064+0000", 2024-05-15T13:12:01.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0078963569999999997 2024-05-15T13:12:01.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:12:01.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:18.050092+0000", 2024-05-15T13:12:01.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8425e-05 2024-05-15T13:12:01.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:12:01.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:18.050112+0000", 2024-05-15T13:12:01.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9792e-05 2024-05-15T13:12:01.828 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.828 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:12:01.828 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.828 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:161221 5.4 5:247625a2:::benchmark_data_smithi177_109991_object26030:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-05-15T13:12:01.828 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:05:17.983297+0000", 2024-05-15T13:12:01.828 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 403.714777696, 2024-05-15T13:12:01.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.066900526000000002, 2024-05-15T13:12:01.829 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:12:01.829 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:12:01.829 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:12:01.829 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:12:01.829 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:12:01.829 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 161221 2024-05-15T13:12:01.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.829 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:12:01.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:12:01.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:17.983297+0000", 2024-05-15T13:12:01.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:12:01.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:17.983297+0000", 2024-05-15T13:12:01.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:12:01.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:17.983297+0000", 2024-05-15T13:12:01.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2000000000000001e-07 2024-05-15T13:12:01.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:12:01.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:17.983298+0000", 2024-05-15T13:12:01.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1599999999999999e-06 2024-05-15T13:12:01.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:12:01.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:17.983301+0000", 2024-05-15T13:12:01.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9299999999999999e-06 2024-05-15T13:12:01.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:12:01.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:17.983304+0000", 2024-05-15T13:12:01.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0800000000000002e-06 2024-05-15T13:12:01.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:12:01.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:18.037952+0000", 2024-05-15T13:12:01.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.054647948000000002 2024-05-15T13:12:01.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:12:01.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:18.038076+0000", 2024-05-15T13:12:01.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012420500000000001 2024-05-15T13:12:01.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:12:01.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:18.038191+0000", 2024-05-15T13:12:01.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114794 2024-05-15T13:12:01.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:12:01.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:18.047189+0000", 2024-05-15T13:12:01.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0089978109999999997 2024-05-15T13:12:01.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:18.047281+0000", 2024-05-15T13:12:01.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2229000000000001e-05 2024-05-15T13:12:01.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:18.050137+0000", 2024-05-15T13:12:01.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0028557579999999999 2024-05-15T13:12:01.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:12:01.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:18.050165+0000", 2024-05-15T13:12:01.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8072000000000002e-05 2024-05-15T13:12:01.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:12:01.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:05:18.050197+0000", 2024-05-15T13:12:01.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2319e-05 2024-05-15T13:12:01.834 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.834 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:12:01.834 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.834 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.15540.0:247102 5.4 5:21cf26b7:::benchmark_data_smithi177_109991_object111911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e487)", 2024-05-15T13:12:01.834 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:06:57.180481+0000", 2024-05-15T13:12:01.834 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 304.51759274900002, 2024-05-15T13:12:01.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.073158055999999999, 2024-05-15T13:12:01.835 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:12:01.835 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:12:01.835 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:12:01.835 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.15540", 2024-05-15T13:12:01.835 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/584614718", 2024-05-15T13:12:01.835 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 247102 2024-05-15T13:12:01.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.835 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:12:01.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:12:01.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.180481+0000", 2024-05-15T13:12:01.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:12:01.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.180481+0000", 2024-05-15T13:12:01.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:12:01.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.180483+0000", 2024-05-15T13:12:01.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5540000000000001e-06 2024-05-15T13:12:01.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:12:01.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.180488+0000", 2024-05-15T13:12:01.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3630000000000002e-06 2024-05-15T13:12:01.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:12:01.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.180500+0000", 2024-05-15T13:12:01.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1839e-05 2024-05-15T13:12:01.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:12:01.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.180509+0000", 2024-05-15T13:12:01.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4069999999999993e-06 2024-05-15T13:12:01.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:12:01.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.185510+0000", 2024-05-15T13:12:01.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0050014880000000001 2024-05-15T13:12:01.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:12:01.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.185642+0000", 2024-05-15T13:12:01.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013209699999999999 2024-05-15T13:12:01.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,5", 2024-05-15T13:12:01.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.186286+0000", 2024-05-15T13:12:01.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00064361300000000002 2024-05-15T13:12:01.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:12:01.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.192876+0000", 2024-05-15T13:12:01.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0065902249999999999 2024-05-15T13:12:01.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.839 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.839 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.193841+0000", 2024-05-15T13:12:01.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00096457300000000004 2024-05-15T13:12:01.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.839 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:12:01.839 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.253525+0000", 2024-05-15T13:12:01.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.059683893000000002 2024-05-15T13:12:01.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:12:01.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.253563+0000", 2024-05-15T13:12:01.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8334000000000003e-05 2024-05-15T13:12:01.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:12:01.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.253640+0000", 2024-05-15T13:12:01.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6669999999999996e-05 2024-05-15T13:12:01.840 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.840 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:12:01.840 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.840 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:247103 5.2 e487/418 5:480a38a9:::benchmark_data_smithi177_109991_object111912:head v 487'15413, mlcod=487'15411)", 2024-05-15T13:12:01.840 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:06:57.187750+0000", 2024-05-15T13:12:01.841 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 304.51032467599998, 2024-05-15T13:12:01.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.074077830999999997, 2024-05-15T13:12:01.841 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:12:01.841 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:12:01.841 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:12:01.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:12:01.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.187750+0000", 2024-05-15T13:12:01.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:12:01.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.187750+0000", 2024-05-15T13:12:01.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:12:01.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.187750+0000", 2024-05-15T13:12:01.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5400000000000002e-07 2024-05-15T13:12:01.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:12:01.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.187754+0000", 2024-05-15T13:12:01.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9859999999999996e-06 2024-05-15T13:12:01.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:12:01.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.187759+0000", 2024-05-15T13:12:01.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.57e-06 2024-05-15T13:12:01.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:12:01.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.187763+0000", 2024-05-15T13:12:01.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6509999999999999e-06 2024-05-15T13:12:01.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:12:01.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.189620+0000", 2024-05-15T13:12:01.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0018572969999999999 2024-05-15T13:12:01.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:12:01.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.189639+0000", 2024-05-15T13:12:01.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8156e-05 2024-05-15T13:12:01.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:12:01.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.261796+0000", 2024-05-15T13:12:01.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.072157733000000002 2024-05-15T13:12:01.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:12:01.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.261827+0000", 2024-05-15T13:12:01.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1084000000000003e-05 2024-05-15T13:12:01.844 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.844 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:12:01.844 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.845 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.15540.0:247112 5.2 e487/418 5:4c05e289:::benchmark_data_smithi177_109991_object111921:head v 487'15414, mlcod=487'15411)", 2024-05-15T13:12:01.845 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:06:57.194131+0000", 2024-05-15T13:12:01.845 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 304.503942897, 2024-05-15T13:12:01.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.072447461000000005, 2024-05-15T13:12:01.845 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:12:01.845 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:12:01.845 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:12:01.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:12:01.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.194131+0000", 2024-05-15T13:12:01.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:12:01.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.194131+0000", 2024-05-15T13:12:01.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:12:01.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:12:01.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.194132+0000", 2024-05-15T13:12:01.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1900000000000002e-07 2024-05-15T13:12:01.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:12:01.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.194139+0000", 2024-05-15T13:12:01.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2910000000000002e-06 2024-05-15T13:12:01.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:12:01.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.194150+0000", 2024-05-15T13:12:01.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0149000000000001e-05 2024-05-15T13:12:01.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:12:01.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.194156+0000", 2024-05-15T13:12:01.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5509999999999996e-06 2024-05-15T13:12:01.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:12:01.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.194904+0000", 2024-05-15T13:12:01.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000747479 2024-05-15T13:12:01.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:12:01.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.194916+0000", 2024-05-15T13:12:01.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.222e-05 2024-05-15T13:12:01.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:12:01.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.266533+0000", 2024-05-15T13:12:01.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.071617060999999996 2024-05-15T13:12:01.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:12:01.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:12:01.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:12:01.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:06:57.266579+0000", 2024-05-15T13:12:01.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5991000000000001e-05 2024-05-15T13:12:01.849 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.849 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:12:01.849 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.849 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:12:01.849 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:12:01.849 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:12:01.850 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:01 smithi120 ceph-mon[43961]: pgmap v1549: 27 pgs: 1 active+remapped+backfill_toofull, 26 active+clean; 297 MiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 709 op/s; 327/14256 objects misplaced (2.294%); 2.5 MiB/s, 39 objects/s recovering 2024-05-15T13:12:01.850 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:01 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'54407','3':'50808','6':'59447','9':'55119'}"}]: dispatch 2024-05-15T13:12:01.850 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:01 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'54407','3':'50808','6':'59447','9':'55119'}"}]: dispatch 2024-05-15T13:12:01.892 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:12:02.129 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:12:02.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:02.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:12:02.166 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-05-15T13:12:02.166 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight-by-utilization 110 1.0 2024-05-15T13:12:02.205 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:12:02.233 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:12:02.264 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:02.284 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:12:02.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:02.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:12:02.366 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:02 smithi177 ceph-mon[44303]: from='client.25943 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T13:12:02.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:02 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'54407','3':'50808','6':'59447','9':'55119'}"}]': finished 2024-05-15T13:12:02.367 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:02 smithi177 ceph-mon[44303]: osdmap e580: 12 total, 12 up, 10 in 2024-05-15T13:12:02.388 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:12:02.393 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-05-15T13:12:02.416 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:02.456 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:12:02.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:02.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:12:02.488 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:02 smithi027 ceph-mon[39795]: from='client.25943 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T13:12:02.488 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:02 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'54407','3':'50808','6':'59447','9':'55119'}"}]': finished 2024-05-15T13:12:02.488 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:02 smithi027 ceph-mon[39795]: osdmap e580: 12 total, 12 up, 10 in 2024-05-15T13:12:02.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:02 smithi120 ceph-mon[43961]: from='client.25943 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T13:12:02.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:02 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'54407','3':'50808','6':'59447','9':'55119'}"}]': finished 2024-05-15T13:12:02.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:02 smithi120 ceph-mon[43961]: osdmap e580: 12 total, 12 up, 10 in 2024-05-15T13:12:02.561 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:12:02.590 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:02.610 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:12:02.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:02.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:12:02.652 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:12:02.714 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:12:02.741 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:02.918 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:12:02.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:02.920 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:12:03.021 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:12:03.052 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:03.150 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:12:03.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:03.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:12:03.254 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:12:03.282 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:03.326 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:03 smithi027 ceph-mon[39795]: pgmap v1551: 27 pgs: 1 peering, 1 active+remapped+backfill_toofull, 25 active+clean; 297 MiB data, 6.0 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 504 op/s; 327/14256 objects misplaced (2.294%) 2024-05-15T13:12:03.326 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:03 smithi027 ceph-mon[39795]: osdmap e581: 12 total, 12 up, 10 in 2024-05-15T13:12:03.430 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:12:03.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:03 smithi177 ceph-mon[44303]: pgmap v1551: 27 pgs: 1 peering, 1 active+remapped+backfill_toofull, 25 active+clean; 297 MiB data, 6.0 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 504 op/s; 327/14256 objects misplaced (2.294%) 2024-05-15T13:12:03.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:03 smithi177 ceph-mon[44303]: osdmap e581: 12 total, 12 up, 10 in 2024-05-15T13:12:03.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:03.432 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:12:03.533 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:12:03.589 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:03.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:03 smithi120 ceph-mon[43961]: pgmap v1551: 27 pgs: 1 peering, 1 active+remapped+backfill_toofull, 25 active+clean; 297 MiB data, 6.0 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 504 op/s; 327/14256 objects misplaced (2.294%) 2024-05-15T13:12:03.782 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:03 smithi120 ceph-mon[43961]: osdmap e581: 12 total, 12 up, 10 in 2024-05-15T13:12:03.950 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:12:03.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:03.954 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:12:04.055 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:12:04.097 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:04.131 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:12:04.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:04.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:12:04.240 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:12:04.285 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:04.309 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:12:04.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:04.312 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:12:04.401 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:12:04.413 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:12:04.450 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:04.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:04 smithi120 ceph-mon[43961]: 6.d starting backfill to osd.0 from (0'0,0'0] MAX to 579'16197 2024-05-15T13:12:04.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:04 smithi120 ceph-mon[43961]: 6.e starting backfill to osd.11 from (0'0,0'0] MAX to 579'16076 2024-05-15T13:12:04.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:04 smithi120 ceph-mon[43961]: 6.7 starting backfill to osd.10 from (0'0,0'0] MAX to 579'16267 2024-05-15T13:12:04.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:04 smithi120 ceph-mon[43961]: 6.4 starting backfill to osd.9 from (0'0,0'0] MAX to 579'15955 2024-05-15T13:12:04.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:04 smithi120 ceph-mon[43961]: osdmap e582: 12 total, 12 up, 10 in 2024-05-15T13:12:04.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:04 smithi120 ceph-mon[43961]: 6.6 starting backfill to osd.11 from (0'0,0'0] MAX to 579'16099 2024-05-15T13:12:04.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:04 smithi120 ceph-mon[43961]: osdmap e583: 12 total, 12 up, 10 in 2024-05-15T13:12:04.560 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:12:04.563 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:04 smithi177 ceph-mon[44303]: 6.d starting backfill to osd.0 from (0'0,0'0] MAX to 579'16197 2024-05-15T13:12:04.563 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:04 smithi177 ceph-mon[44303]: 6.e starting backfill to osd.11 from (0'0,0'0] MAX to 579'16076 2024-05-15T13:12:04.563 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:04 smithi177 ceph-mon[44303]: 6.7 starting backfill to osd.10 from (0'0,0'0] MAX to 579'16267 2024-05-15T13:12:04.563 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:04 smithi177 ceph-mon[44303]: 6.4 starting backfill to osd.9 from (0'0,0'0] MAX to 579'15955 2024-05-15T13:12:04.564 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:04 smithi177 ceph-mon[44303]: osdmap e582: 12 total, 12 up, 10 in 2024-05-15T13:12:04.564 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:04 smithi177 ceph-mon[44303]: 6.6 starting backfill to osd.11 from (0'0,0'0] MAX to 579'16099 2024-05-15T13:12:04.564 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:04 smithi177 ceph-mon[44303]: osdmap e583: 12 total, 12 up, 10 in 2024-05-15T13:12:04.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:04.565 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:12:04.629 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:12:04.629 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T13:12:04.629 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-05-15T13:12:04.629 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:12:04.666 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:12:04.693 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:04.710 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:12:04.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:04.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:12:04.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:04 smithi027 ceph-mon[39795]: 6.d starting backfill to osd.0 from (0'0,0'0] MAX to 579'16197 2024-05-15T13:12:04.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:04 smithi027 ceph-mon[39795]: 6.e starting backfill to osd.11 from (0'0,0'0] MAX to 579'16076 2024-05-15T13:12:04.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:04 smithi027 ceph-mon[39795]: 6.7 starting backfill to osd.10 from (0'0,0'0] MAX to 579'16267 2024-05-15T13:12:04.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:04 smithi027 ceph-mon[39795]: 6.4 starting backfill to osd.9 from (0'0,0'0] MAX to 579'15955 2024-05-15T13:12:04.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:04 smithi027 ceph-mon[39795]: osdmap e582: 12 total, 12 up, 10 in 2024-05-15T13:12:04.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:04 smithi027 ceph-mon[39795]: 6.6 starting backfill to osd.11 from (0'0,0'0] MAX to 579'16099 2024-05-15T13:12:04.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:04 smithi027 ceph-mon[39795]: osdmap e583: 12 total, 12 up, 10 in 2024-05-15T13:12:04.814 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:12:04.840 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:04.858 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:12:04.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:04.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:12:04.962 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:12:04.989 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:05.007 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:12:05.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:05.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:12:05.111 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:12:05.139 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:05.179 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:12:05.198 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:12:05.218 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:12:05.237 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:12:05.257 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:12:05.277 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:12:05.297 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:12:05.313 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-05-15T13:12:05.320 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:12:05.336 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:12:05.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:05.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:12:05.341 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:12:05.361 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:12:05.381 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:12:05.401 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:12:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:12:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:12:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:12:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:12:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:12:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:12:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:12:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:12:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:12:05.403 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:12:05.403 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:12:05.403 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:12:05.441 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:12:05.468 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:05.486 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:12:05.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:05.489 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:12:05.590 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:12:05.607 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:12:05.622 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:05.623 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:05 smithi177 ceph-mon[44303]: pgmap v1555: 27 pgs: 3 activating+remapped, 2 remapped+peering, 1 active+remapped+backfill_toofull, 2 peering, 19 active+clean; 155 MiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 571 op/s; 545/7398 objects misplaced (7.367%) 2024-05-15T13:12:05.623 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:05 smithi177 ceph-mon[44303]: osdmap e584: 12 total, 12 up, 10 in 2024-05-15T13:12:05.623 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:05 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2852232220' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:12:05.650 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:12:05.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:05.654 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:12:05.684 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:05 smithi027 ceph-mon[39795]: pgmap v1555: 27 pgs: 3 activating+remapped, 2 remapped+peering, 1 active+remapped+backfill_toofull, 2 peering, 19 active+clean; 155 MiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 571 op/s; 545/7398 objects misplaced (7.367%) 2024-05-15T13:12:05.684 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:05 smithi027 ceph-mon[39795]: osdmap e584: 12 total, 12 up, 10 in 2024-05-15T13:12:05.684 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:05 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2852232220' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:12:05.755 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:12:05.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:05 smithi120 ceph-mon[43961]: pgmap v1555: 27 pgs: 3 activating+remapped, 2 remapped+peering, 1 active+remapped+backfill_toofull, 2 peering, 19 active+clean; 155 MiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 571 op/s; 545/7398 objects misplaced (7.367%) 2024-05-15T13:12:05.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:05 smithi120 ceph-mon[43961]: osdmap e584: 12 total, 12 up, 10 in 2024-05-15T13:12:05.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:05 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2852232220' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:12:05.842 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:06.660 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:12:06.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:06.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:12:06.770 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:12:06.847 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:06 smithi027 ceph-mon[39795]: osdmap e585: 12 total, 12 up, 10 in 2024-05-15T13:12:06.847 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:06 smithi027 ceph-mon[39795]: pgmap v1558: 27 pgs: 3 activating+remapped, 2 remapped+peering, 1 active+remapped+backfill_toofull, 2 peering, 19 active+clean; 133 MiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 2.6 KiB/s rd, 68 KiB/s wr, 680 op/s; 545/6351 objects misplaced (8.581%); 84 KiB/s, 0 objects/s recovering 2024-05-15T13:12:06.850 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:06.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:06 smithi177 ceph-mon[44303]: osdmap e585: 12 total, 12 up, 10 in 2024-05-15T13:12:06.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:06 smithi177 ceph-mon[44303]: pgmap v1558: 27 pgs: 3 activating+remapped, 2 remapped+peering, 1 active+remapped+backfill_toofull, 2 peering, 19 active+clean; 133 MiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 2.6 KiB/s rd, 68 KiB/s wr, 680 op/s; 545/6351 objects misplaced (8.581%); 84 KiB/s, 0 objects/s recovering 2024-05-15T13:12:07.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:06 smithi120 ceph-mon[43961]: osdmap e585: 12 total, 12 up, 10 in 2024-05-15T13:12:07.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:06 smithi120 ceph-mon[43961]: pgmap v1558: 27 pgs: 3 activating+remapped, 2 remapped+peering, 1 active+remapped+backfill_toofull, 2 peering, 19 active+clean; 133 MiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 2.6 KiB/s rd, 68 KiB/s wr, 680 op/s; 545/6351 objects misplaced (8.581%); 84 KiB/s, 0 objects/s recovering 2024-05-15T13:12:07.477 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Removed 130368 objects 2024-05-15T13:12:07.478 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Clean up completed and total clean up time :162.925 2024-05-15T13:12:07.573 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:12:07.573 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:12:07.589 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:12:07.755 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:07 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2377260627' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:12:07.756 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:07 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:12:07.911 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:07 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2377260627' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:12:07.911 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:07 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:12:08.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:07 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2377260627' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:12:08.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:07 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:12:08.217 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-05-15T13:12:08.477 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:12:08.521 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:12:08.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:08.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:12:08.625 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:12:08.651 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:08.704 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:12:08.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:08.706 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:12:08.727 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:08 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:12:08.727 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:08 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:12:08.727 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:08 smithi027 ceph-mon[39795]: osdmap e586: 12 total, 12 up, 10 in 2024-05-15T13:12:08.727 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:08 smithi027 ceph-mon[39795]: pgmap v1560: 27 pgs: 2 active+remapped+backfill_toofull, 3 activating+remapped, 1 peering, 21 active+clean; 70 MiB data, 5.4 GiB used, 889 GiB / 894 GiB avail; 2.1 KiB/s rd, 57 KiB/s wr, 357 op/s; 415/3354 objects misplaced (12.373%); 1.5 MiB/s, 23 objects/s recovering 2024-05-15T13:12:08.807 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:12:08.832 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:08 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:12:08.832 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:08 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:12:08.832 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:08 smithi177 ceph-mon[44303]: osdmap e586: 12 total, 12 up, 10 in 2024-05-15T13:12:08.832 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:08 smithi177 ceph-mon[44303]: pgmap v1560: 27 pgs: 2 active+remapped+backfill_toofull, 3 activating+remapped, 1 peering, 21 active+clean; 70 MiB data, 5.4 GiB used, 889 GiB / 894 GiB avail; 2.1 KiB/s rd, 57 KiB/s wr, 357 op/s; 415/3354 objects misplaced (12.373%); 1.5 MiB/s, 23 objects/s recovering 2024-05-15T13:12:08.834 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:08.933 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:12:08.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:08.936 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:12:09.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:08 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:12:09.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:08 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:12:09.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:08 smithi120 ceph-mon[43961]: osdmap e586: 12 total, 12 up, 10 in 2024-05-15T13:12:09.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:08 smithi120 ceph-mon[43961]: pgmap v1560: 27 pgs: 2 active+remapped+backfill_toofull, 3 activating+remapped, 1 peering, 21 active+clean; 70 MiB data, 5.4 GiB used, 889 GiB / 894 GiB avail; 2.1 KiB/s rd, 57 KiB/s wr, 357 op/s; 415/3354 objects misplaced (12.373%); 1.5 MiB/s, 23 objects/s recovering 2024-05-15T13:12:09.036 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:12:09.066 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:09.089 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:12:09.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:09.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:12:09.193 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:12:09.221 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:09.240 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:12:09.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:09.243 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:12:09.344 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:12:09.370 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:09.591 INFO:teuthology.orchestra.run.smithi027.stdout:moved 5 / 81 (6.17284%) 2024-05-15T13:12:09.591 INFO:teuthology.orchestra.run.smithi027.stdout:avg 8.1 2024-05-15T13:12:09.591 INFO:teuthology.orchestra.run.smithi027.stdout:stddev 2.3 -> 2.2561 (expected baseline 2.7) 2024-05-15T13:12:09.591 INFO:teuthology.orchestra.run.smithi027.stdout:min osd.1 with 3 -> 3 pgs (0.37037 -> 0.37037 * mean) 2024-05-15T13:12:09.591 INFO:teuthology.orchestra.run.smithi027.stdout:max osd.6 with 11 -> 8 pgs (1.35802 -> 0.987654 * mean) 2024-05-15T13:12:09.591 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:12:09.591 INFO:teuthology.orchestra.run.smithi027.stdout:oload 110 2024-05-15T13:12:09.591 INFO:teuthology.orchestra.run.smithi027.stdout:max_change 1 2024-05-15T13:12:09.591 INFO:teuthology.orchestra.run.smithi027.stdout:max_change_osds 4 2024-05-15T13:12:09.591 INFO:teuthology.orchestra.run.smithi027.stdout:average_utilization 0.0061 2024-05-15T13:12:09.591 INFO:teuthology.orchestra.run.smithi027.stdout:overload_utilization 0.0067 2024-05-15T13:12:09.591 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9 weight 0.8410 -> 0.6992 2024-05-15T13:12:09.591 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 weight 0.8302 -> 0.7194 2024-05-15T13:12:09.592 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6 weight 0.9071 -> 0.7996 2024-05-15T13:12:09.592 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 weight 0.7753 -> 0.6948 2024-05-15T13:12:09.722 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:12:09.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:09.726 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:12:09.782 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:12:09.826 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:12:09.856 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:09 smithi120 ceph-mon[43961]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T13:12:09.856 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:09 smithi120 ceph-mon[43961]: 6.9 scrub starts 2024-05-15T13:12:09.856 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:09 smithi120 ceph-mon[43961]: 6.9 scrub ok 2024-05-15T13:12:09.856 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:09 smithi120 ceph-mon[43961]: from='client.16050 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T13:12:09.856 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:09 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'47147','3':'45536','6':'52403','9':'45825'}"}]: dispatch 2024-05-15T13:12:09.856 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:09 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'47147','3':'45536','6':'52403','9':'45825'}"}]: dispatch 2024-05-15T13:12:09.858 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:09.879 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:12:09.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:09.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:12:09.928 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:09 smithi027 ceph-mon[39795]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T13:12:09.928 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:09 smithi027 ceph-mon[39795]: 6.9 scrub starts 2024-05-15T13:12:09.929 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:09 smithi027 ceph-mon[39795]: 6.9 scrub ok 2024-05-15T13:12:09.929 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:09 smithi027 ceph-mon[39795]: from='client.16050 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T13:12:09.929 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:09 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'47147','3':'45536','6':'52403','9':'45825'}"}]: dispatch 2024-05-15T13:12:09.929 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:09 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'47147','3':'45536','6':'52403','9':'45825'}"}]: dispatch 2024-05-15T13:12:09.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:09 smithi177 ceph-mon[44303]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T13:12:09.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:09 smithi177 ceph-mon[44303]: 6.9 scrub starts 2024-05-15T13:12:09.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:09 smithi177 ceph-mon[44303]: 6.9 scrub ok 2024-05-15T13:12:09.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:09 smithi177 ceph-mon[44303]: from='client.16050 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T13:12:09.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:09 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'47147','3':'45536','6':'52403','9':'45825'}"}]: dispatch 2024-05-15T13:12:09.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:09 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'47147','3':'45536','6':'52403','9':'45825'}"}]: dispatch 2024-05-15T13:12:09.983 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:12:10.011 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:10.031 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:12:10.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:10.034 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:12:10.135 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:12:10.163 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:10.238 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:12:10.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:10.241 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:12:10.304 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:12:10.304 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:12:10.343 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:12:10.373 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-05-15T13:12:10.373 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight-by-utilization 1000 1.0 2024-05-15T13:12:10.374 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:10.393 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:12:10.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:10.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:12:10.475 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:12:10.497 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:12:10.505 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:12:10.525 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:10.543 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:12:10.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:10.547 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:12:10.648 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:12:10.680 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:10.701 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:12:10.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:10.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:12:10.737 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-05-15T13:12:10.738 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-05-15T13:12:10.805 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:12:10.832 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:10.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:10 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:12:10.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:10 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:12:10.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:10 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'47147','3':'45536','6':'52403','9':'45825'}"}]': finished 2024-05-15T13:12:10.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:10 smithi177 ceph-mon[44303]: osdmap e587: 12 total, 12 up, 10 in 2024-05-15T13:12:10.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:10 smithi177 ceph-mon[44303]: 1.1 scrub starts 2024-05-15T13:12:10.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:10 smithi177 ceph-mon[44303]: 1.1 scrub ok 2024-05-15T13:12:10.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:10 smithi177 ceph-mon[44303]: pgmap v1562: 27 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 24 active+clean; 1.6 MiB data, 5.3 GiB used, 889 GiB / 894 GiB avail; 1.8 KiB/s rd, 47 KiB/s wr, 436 op/s; 8/54 objects misplaced (14.815%); 5.5 MiB/s, 88 objects/s recovering 2024-05-15T13:12:10.980 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:12:10.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:10.984 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:12:10.993 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:10 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:12:10.993 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:10 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:12:10.993 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:10 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'47147','3':'45536','6':'52403','9':'45825'}"}]': finished 2024-05-15T13:12:10.994 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:10 smithi027 ceph-mon[39795]: osdmap e587: 12 total, 12 up, 10 in 2024-05-15T13:12:10.994 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:10 smithi027 ceph-mon[39795]: 1.1 scrub starts 2024-05-15T13:12:10.994 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:10 smithi027 ceph-mon[39795]: 1.1 scrub ok 2024-05-15T13:12:10.994 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:10 smithi027 ceph-mon[39795]: pgmap v1562: 27 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 24 active+clean; 1.6 MiB data, 5.3 GiB used, 889 GiB / 894 GiB avail; 1.8 KiB/s rd, 47 KiB/s wr, 436 op/s; 8/54 objects misplaced (14.815%); 5.5 MiB/s, 88 objects/s recovering 2024-05-15T13:12:11.007 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-05-15T13:12:11.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:10 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:12:11.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:10 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:12:11.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:10 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'47147','3':'45536','6':'52403','9':'45825'}"}]': finished 2024-05-15T13:12:11.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:10 smithi120 ceph-mon[43961]: osdmap e587: 12 total, 12 up, 10 in 2024-05-15T13:12:11.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:10 smithi120 ceph-mon[43961]: 1.1 scrub starts 2024-05-15T13:12:11.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:10 smithi120 ceph-mon[43961]: 1.1 scrub ok 2024-05-15T13:12:11.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:10 smithi120 ceph-mon[43961]: pgmap v1562: 27 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 24 active+clean; 1.6 MiB data, 5.3 GiB used, 889 GiB / 894 GiB avail; 1.8 KiB/s rd, 47 KiB/s wr, 436 op/s; 8/54 objects misplaced (14.815%); 5.5 MiB/s, 88 objects/s recovering 2024-05-15T13:12:11.085 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:12:11.114 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:11.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:11 smithi177 ceph-mon[44303]: 1.6 scrub starts 2024-05-15T13:12:11.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:11 smithi177 ceph-mon[44303]: 1.6 scrub ok 2024-05-15T13:12:11.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:11 smithi177 ceph-mon[44303]: osdmap e588: 12 total, 12 up, 10 in 2024-05-15T13:12:11.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:11 smithi177 ceph-mon[44303]: 6.1 scrub starts 2024-05-15T13:12:11.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:11 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:12:11.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:11 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:12:11.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:11 smithi177 ceph-mon[44303]: 6.f scrub starts 2024-05-15T13:12:11.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:11 smithi177 ceph-mon[44303]: 6.f scrub ok 2024-05-15T13:12:12.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:11 smithi027 ceph-mon[39795]: 1.6 scrub starts 2024-05-15T13:12:12.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:11 smithi027 ceph-mon[39795]: 1.6 scrub ok 2024-05-15T13:12:12.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:11 smithi027 ceph-mon[39795]: osdmap e588: 12 total, 12 up, 10 in 2024-05-15T13:12:12.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:11 smithi027 ceph-mon[39795]: 6.1 scrub starts 2024-05-15T13:12:12.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:11 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:12:12.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:11 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:12:12.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:11 smithi027 ceph-mon[39795]: 6.f scrub starts 2024-05-15T13:12:12.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:11 smithi027 ceph-mon[39795]: 6.f scrub ok 2024-05-15T13:12:12.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:11 smithi120 ceph-mon[43961]: 1.6 scrub starts 2024-05-15T13:12:12.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:11 smithi120 ceph-mon[43961]: 1.6 scrub ok 2024-05-15T13:12:12.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:11 smithi120 ceph-mon[43961]: osdmap e588: 12 total, 12 up, 10 in 2024-05-15T13:12:12.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:11 smithi120 ceph-mon[43961]: 6.1 scrub starts 2024-05-15T13:12:12.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:11 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:12:12.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:11 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:12:12.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:11 smithi120 ceph-mon[43961]: 6.f scrub starts 2024-05-15T13:12:12.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:11 smithi120 ceph-mon[43961]: 6.f scrub ok 2024-05-15T13:12:12.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:12 smithi177 ceph-mon[44303]: 6.1 scrub ok 2024-05-15T13:12:12.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:12 smithi177 ceph-mon[44303]: 6.8 scrub starts 2024-05-15T13:12:12.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:12 smithi177 ceph-mon[44303]: 6.8 scrub ok 2024-05-15T13:12:12.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:12 smithi177 ceph-mon[44303]: 1.7 scrub starts 2024-05-15T13:12:12.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:12 smithi177 ceph-mon[44303]: 1.7 scrub ok 2024-05-15T13:12:12.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:12 smithi177 ceph-mon[44303]: osdmap e589: 12 total, 12 up, 10 in 2024-05-15T13:12:12.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:12 smithi177 ceph-mon[44303]: 6.2 starting backfill to osd.8 from (0'0,0'0] MAX to 585'16132 2024-05-15T13:12:12.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:12 smithi177 ceph-mon[44303]: 6.5 starting backfill to osd.5 from (0'0,0'0] MAX to 585'16460 2024-05-15T13:12:12.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:12 smithi177 ceph-mon[44303]: 6.c starting backfill to osd.8 from (0'0,0'0] MAX to 585'16330 2024-05-15T13:12:12.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:12 smithi177 ceph-mon[44303]: 6.2 continuing backfill to osd.11 from (535'6100,585'16132] MIN to 585'16132 2024-05-15T13:12:12.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:12 smithi177 ceph-mon[44303]: pgmap v1565: 27 pgs: 2 remapped+peering, 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 22 active+clean; 1.6 MiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 331 op/s; 8/54 objects misplaced (14.815%); 6.7 MiB/s, 107 objects/s recovering 2024-05-15T13:12:12.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:12 smithi177 ceph-mon[44303]: 6.e scrub starts 2024-05-15T13:12:12.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:12 smithi177 ceph-mon[44303]: 6.e scrub ok 2024-05-15T13:12:13.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:12 smithi027 ceph-mon[39795]: 6.1 scrub ok 2024-05-15T13:12:13.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:12 smithi027 ceph-mon[39795]: 6.8 scrub starts 2024-05-15T13:12:13.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:12 smithi027 ceph-mon[39795]: 6.8 scrub ok 2024-05-15T13:12:13.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:12 smithi027 ceph-mon[39795]: 1.7 scrub starts 2024-05-15T13:12:13.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:12 smithi027 ceph-mon[39795]: 1.7 scrub ok 2024-05-15T13:12:13.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:12 smithi027 ceph-mon[39795]: osdmap e589: 12 total, 12 up, 10 in 2024-05-15T13:12:13.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:12 smithi027 ceph-mon[39795]: 6.2 starting backfill to osd.8 from (0'0,0'0] MAX to 585'16132 2024-05-15T13:12:13.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:12 smithi027 ceph-mon[39795]: 6.5 starting backfill to osd.5 from (0'0,0'0] MAX to 585'16460 2024-05-15T13:12:13.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:12 smithi027 ceph-mon[39795]: 6.c starting backfill to osd.8 from (0'0,0'0] MAX to 585'16330 2024-05-15T13:12:13.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:12 smithi027 ceph-mon[39795]: 6.2 continuing backfill to osd.11 from (535'6100,585'16132] MIN to 585'16132 2024-05-15T13:12:13.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:12 smithi027 ceph-mon[39795]: pgmap v1565: 27 pgs: 2 remapped+peering, 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 22 active+clean; 1.6 MiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 331 op/s; 8/54 objects misplaced (14.815%); 6.7 MiB/s, 107 objects/s recovering 2024-05-15T13:12:13.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:12 smithi027 ceph-mon[39795]: 6.e scrub starts 2024-05-15T13:12:13.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:12 smithi027 ceph-mon[39795]: 6.e scrub ok 2024-05-15T13:12:13.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:12 smithi120 ceph-mon[43961]: 6.1 scrub ok 2024-05-15T13:12:13.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:12 smithi120 ceph-mon[43961]: 6.8 scrub starts 2024-05-15T13:12:13.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:12 smithi120 ceph-mon[43961]: 6.8 scrub ok 2024-05-15T13:12:13.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:12 smithi120 ceph-mon[43961]: 1.7 scrub starts 2024-05-15T13:12:13.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:12 smithi120 ceph-mon[43961]: 1.7 scrub ok 2024-05-15T13:12:13.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:12 smithi120 ceph-mon[43961]: osdmap e589: 12 total, 12 up, 10 in 2024-05-15T13:12:13.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:12 smithi120 ceph-mon[43961]: 6.2 starting backfill to osd.8 from (0'0,0'0] MAX to 585'16132 2024-05-15T13:12:13.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:12 smithi120 ceph-mon[43961]: 6.5 starting backfill to osd.5 from (0'0,0'0] MAX to 585'16460 2024-05-15T13:12:13.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:12 smithi120 ceph-mon[43961]: 6.c starting backfill to osd.8 from (0'0,0'0] MAX to 585'16330 2024-05-15T13:12:13.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:12 smithi120 ceph-mon[43961]: 6.2 continuing backfill to osd.11 from (535'6100,585'16132] MIN to 585'16132 2024-05-15T13:12:13.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:12 smithi120 ceph-mon[43961]: pgmap v1565: 27 pgs: 2 remapped+peering, 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 22 active+clean; 1.6 MiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 331 op/s; 8/54 objects misplaced (14.815%); 6.7 MiB/s, 107 objects/s recovering 2024-05-15T13:12:13.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:12 smithi120 ceph-mon[43961]: 6.e scrub starts 2024-05-15T13:12:13.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:12 smithi120 ceph-mon[43961]: 6.e scrub ok 2024-05-15T13:12:13.269 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:12:13.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:13.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:12:13.375 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:12:13.408 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:13.429 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:12:13.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:13.432 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:12:13.533 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:12:13.565 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:13.586 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:12:13.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:13.589 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:12:13.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:13 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:12:13.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:13 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:12:13.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:13 smithi177 ceph-mon[44303]: osdmap e590: 12 total, 12 up, 10 in 2024-05-15T13:12:13.690 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:12:13.718 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:13.739 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:12:13.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:13.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:12:13.843 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:12:13.873 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:13 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:12:13.874 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:13 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:12:13.874 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:13 smithi027 ceph-mon[39795]: osdmap e590: 12 total, 12 up, 10 in 2024-05-15T13:12:13.876 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:14.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:13 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:12:14.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:13 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:12:14.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:13 smithi120 ceph-mon[43961]: osdmap e590: 12 total, 12 up, 10 in 2024-05-15T13:12:14.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:14 smithi177 ceph-mon[44303]: osdmap e591: 12 total, 12 up, 10 in 2024-05-15T13:12:14.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:14 smithi177 ceph-mon[44303]: 1.2 scrub starts 2024-05-15T13:12:14.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:14 smithi177 ceph-mon[44303]: 1.2 scrub ok 2024-05-15T13:12:14.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:14 smithi177 ceph-mon[44303]: 6.b scrub starts 2024-05-15T13:12:14.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:14 smithi177 ceph-mon[44303]: 6.b scrub ok 2024-05-15T13:12:14.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:14 smithi177 ceph-mon[44303]: pgmap v1568: 27 pgs: 2 remapped+peering, 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 21 active+clean; 681 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 2.5 KiB/s rd, 67 KiB/s wr, 10 op/s; 94 KiB/s, 0 objects/s recovering 2024-05-15T13:12:15.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:14 smithi027 ceph-mon[39795]: osdmap e591: 12 total, 12 up, 10 in 2024-05-15T13:12:15.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:14 smithi027 ceph-mon[39795]: 1.2 scrub starts 2024-05-15T13:12:15.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:14 smithi027 ceph-mon[39795]: 1.2 scrub ok 2024-05-15T13:12:15.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:14 smithi027 ceph-mon[39795]: 6.b scrub starts 2024-05-15T13:12:15.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:14 smithi027 ceph-mon[39795]: 6.b scrub ok 2024-05-15T13:12:15.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:14 smithi027 ceph-mon[39795]: pgmap v1568: 27 pgs: 2 remapped+peering, 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 21 active+clean; 681 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 2.5 KiB/s rd, 67 KiB/s wr, 10 op/s; 94 KiB/s, 0 objects/s recovering 2024-05-15T13:12:15.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:14 smithi120 ceph-mon[43961]: osdmap e591: 12 total, 12 up, 10 in 2024-05-15T13:12:15.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:14 smithi120 ceph-mon[43961]: 1.2 scrub starts 2024-05-15T13:12:15.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:14 smithi120 ceph-mon[43961]: 1.2 scrub ok 2024-05-15T13:12:15.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:14 smithi120 ceph-mon[43961]: 6.b scrub starts 2024-05-15T13:12:15.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:14 smithi120 ceph-mon[43961]: 6.b scrub ok 2024-05-15T13:12:15.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:14 smithi120 ceph-mon[43961]: pgmap v1568: 27 pgs: 2 remapped+peering, 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 21 active+clean; 681 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 2.5 KiB/s rd, 67 KiB/s wr, 10 op/s; 94 KiB/s, 0 objects/s recovering 2024-05-15T13:12:15.689 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:12:15.692 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:12:15.731 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:12:15.740 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:12:15.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:15.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:12:15.756 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:12:15.842 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:12:15.875 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:15.895 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:12:15.897 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:15 smithi177 ceph-mon[44303]: 1.a scrub starts 2024-05-15T13:12:15.897 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:15 smithi177 ceph-mon[44303]: 1.a scrub ok 2024-05-15T13:12:15.897 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:15 smithi177 ceph-mon[44303]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T13:12:15.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:15.898 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:12:15.977 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:15 smithi027 ceph-mon[39795]: 1.a scrub starts 2024-05-15T13:12:15.978 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:15 smithi027 ceph-mon[39795]: 1.a scrub ok 2024-05-15T13:12:15.978 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:15 smithi027 ceph-mon[39795]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T13:12:15.999 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:12:16.029 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:16.032 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:15 smithi120 ceph-mon[43961]: 1.a scrub starts 2024-05-15T13:12:16.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:15 smithi120 ceph-mon[43961]: 1.a scrub ok 2024-05-15T13:12:16.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:15 smithi120 ceph-mon[43961]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T13:12:16.662 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:12:16.681 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:12:16.701 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:12:16.722 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:12:16.736 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:16 smithi027 ceph-mon[39795]: 6.3 scrub starts 2024-05-15T13:12:16.736 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:16 smithi027 ceph-mon[39795]: 6.3 scrub ok 2024-05-15T13:12:16.736 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:16 smithi027 ceph-mon[39795]: pgmap v1569: 27 pgs: 1 remapped+peering, 4 active+remapped+backfill_toofull, 22 active+clean; 629 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 2 op/s; 8.3 KiB/s, 0 objects/s recovering 2024-05-15T13:12:16.743 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:12:16.763 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:12:16.785 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:12:16.807 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:12:16.828 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:12:16.850 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:12:16.871 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:12:16.891 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:12:16.893 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:12:16.893 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:12:16.893 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:12:16.893 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:12:16.893 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:12:16.893 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:12:16.893 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:12:16.893 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:12:16.893 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:12:16.893 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:12:16.893 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:12:16.893 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:12:16.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:16 smithi177 ceph-mon[44303]: 6.3 scrub starts 2024-05-15T13:12:16.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:16 smithi177 ceph-mon[44303]: 6.3 scrub ok 2024-05-15T13:12:16.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:16 smithi177 ceph-mon[44303]: pgmap v1569: 27 pgs: 1 remapped+peering, 4 active+remapped+backfill_toofull, 22 active+clean; 629 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 2 op/s; 8.3 KiB/s, 0 objects/s recovering 2024-05-15T13:12:17.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:16 smithi120 ceph-mon[43961]: 6.3 scrub starts 2024-05-15T13:12:17.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:16 smithi120 ceph-mon[43961]: 6.3 scrub ok 2024-05-15T13:12:17.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:16 smithi120 ceph-mon[43961]: pgmap v1569: 27 pgs: 1 remapped+peering, 4 active+remapped+backfill_toofull, 22 active+clean; 629 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 2 op/s; 8.3 KiB/s, 0 objects/s recovering 2024-05-15T13:12:17.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:17 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:12:17.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:17 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:12:17.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:17 smithi177 ceph-mon[44303]: osdmap e592: 12 total, 12 up, 10 in 2024-05-15T13:12:17.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:17 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3536640240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:12:17.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:17 smithi177 ceph-mon[44303]: 6.0 scrub starts 2024-05-15T13:12:17.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:17 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:12:17.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:17 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:12:17.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:17 smithi177 ceph-mon[44303]: 6.0 scrub ok 2024-05-15T13:12:17.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:17 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:12:17.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:17 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:12:17.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:17 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:12:17.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:17 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:12:17.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:17 smithi027 ceph-mon[39795]: osdmap e592: 12 total, 12 up, 10 in 2024-05-15T13:12:17.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:17 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3536640240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:12:17.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:17 smithi027 ceph-mon[39795]: 6.0 scrub starts 2024-05-15T13:12:17.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:17 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:12:17.967 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:17 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:12:17.968 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:17 smithi027 ceph-mon[39795]: 6.0 scrub ok 2024-05-15T13:12:17.968 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:17 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:12:17.968 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:17 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:12:18.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:17 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:12:18.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:17 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:12:18.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:17 smithi120 ceph-mon[43961]: osdmap e592: 12 total, 12 up, 10 in 2024-05-15T13:12:18.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:17 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3536640240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:12:18.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:17 smithi120 ceph-mon[43961]: 6.0 scrub starts 2024-05-15T13:12:18.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:17 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:12:18.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:17 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:12:18.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:17 smithi120 ceph-mon[43961]: 6.0 scrub ok 2024-05-15T13:12:18.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:17 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:12:18.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:17 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:12:18.080 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:12:18.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:18.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:12:18.187 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:12:18.218 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:18.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:18 smithi177 ceph-mon[44303]: 1.3 scrub starts 2024-05-15T13:12:18.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:18 smithi177 ceph-mon[44303]: 1.3 scrub ok 2024-05-15T13:12:18.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:18 smithi177 ceph-mon[44303]: osdmap e593: 12 total, 12 up, 10 in 2024-05-15T13:12:18.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:18 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:12:18.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:18 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:12:18.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:18 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:12:18.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:18 smithi177 ceph-mon[44303]: 6.7 scrub starts 2024-05-15T13:12:18.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:18 smithi177 ceph-mon[44303]: 6.7 scrub ok 2024-05-15T13:12:18.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:18 smithi177 ceph-mon[44303]: pgmap v1572: 27 pgs: 4 active+remapped+backfill_toofull, 23 active+clean; 629 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 0 op/s; 8.3 KiB/s, 0 objects/s recovering 2024-05-15T13:12:19.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:18 smithi027 ceph-mon[39795]: 1.3 scrub starts 2024-05-15T13:12:19.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:18 smithi027 ceph-mon[39795]: 1.3 scrub ok 2024-05-15T13:12:19.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:18 smithi027 ceph-mon[39795]: osdmap e593: 12 total, 12 up, 10 in 2024-05-15T13:12:19.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:18 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:12:19.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:18 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:12:19.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:18 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:12:19.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:18 smithi027 ceph-mon[39795]: 6.7 scrub starts 2024-05-15T13:12:19.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:18 smithi027 ceph-mon[39795]: 6.7 scrub ok 2024-05-15T13:12:19.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:18 smithi027 ceph-mon[39795]: pgmap v1572: 27 pgs: 4 active+remapped+backfill_toofull, 23 active+clean; 629 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 0 op/s; 8.3 KiB/s, 0 objects/s recovering 2024-05-15T13:12:19.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:18 smithi120 ceph-mon[43961]: 1.3 scrub starts 2024-05-15T13:12:19.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:18 smithi120 ceph-mon[43961]: 1.3 scrub ok 2024-05-15T13:12:19.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:18 smithi120 ceph-mon[43961]: osdmap e593: 12 total, 12 up, 10 in 2024-05-15T13:12:19.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:18 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:12:19.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:18 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:12:19.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:18 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:12:19.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:18 smithi120 ceph-mon[43961]: 6.7 scrub starts 2024-05-15T13:12:19.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:18 smithi120 ceph-mon[43961]: 6.7 scrub ok 2024-05-15T13:12:19.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:18 smithi120 ceph-mon[43961]: pgmap v1572: 27 pgs: 4 active+remapped+backfill_toofull, 23 active+clean; 629 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 0 op/s; 8.3 KiB/s, 0 objects/s recovering 2024-05-15T13:12:19.066 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:12:19.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:19.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:12:19.171 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:12:19.204 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:19.226 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:12:19.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:19.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:12:19.331 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:12:19.359 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:19.925 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:12:19.931 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:19 smithi027 ceph-mon[39795]: osdmap e594: 12 total, 12 up, 10 in 2024-05-15T13:12:19.931 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:19 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:12:19.931 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:19 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:12:19.931 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:19 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:12:19.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:19 smithi177 ceph-mon[44303]: osdmap e594: 12 total, 12 up, 10 in 2024-05-15T13:12:19.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:19 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:12:19.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:19 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:12:19.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:19 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:12:19.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:19.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:12:20.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:19 smithi120 ceph-mon[43961]: osdmap e594: 12 total, 12 up, 10 in 2024-05-15T13:12:20.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:19 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:12:20.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:19 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:12:20.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:19 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:12:20.038 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:12:20.069 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:20.091 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:12:20.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:20.094 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:12:20.195 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:12:20.227 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:20.249 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:12:20.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:20.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:12:20.353 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:12:20.383 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:20.403 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:12:20.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:20.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:12:20.508 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:12:20.541 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:20.562 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:12:20.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:20.565 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:12:20.666 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:12:20.698 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:20.718 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:12:20.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:20.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:12:20.823 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:12:20.852 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:20.873 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:12:20.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:20.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:12:20.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:20 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T13:12:20.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:20 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T13:12:20.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:20 smithi177 ceph-mon[44303]: pgmap v1574: 27 pgs: 4 active+remapped+backfill_toofull, 23 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:20.976 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:12:21.003 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:20 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T13:12:21.003 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:20 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T13:12:21.003 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:20 smithi027 ceph-mon[39795]: pgmap v1574: 27 pgs: 4 active+remapped+backfill_toofull, 23 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:21.006 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:21.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:20 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T13:12:21.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:20 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T13:12:21.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:20 smithi120 ceph-mon[43961]: pgmap v1574: 27 pgs: 4 active+remapped+backfill_toofull, 23 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:21.033 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:12:21.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:21.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:12:21.090 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:12:21.091 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T13:12:21.091 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-05-15T13:12:21.091 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:12:21.138 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:12:21.168 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:21.188 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:12:21.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:21.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:12:21.238 INFO:teuthology.orchestra.run.smithi027.stdout:moved 0 / 81 (0%) 2024-05-15T13:12:21.238 INFO:teuthology.orchestra.run.smithi027.stdout:avg 8.1 2024-05-15T13:12:21.238 INFO:teuthology.orchestra.run.smithi027.stdout:stddev 2.2561 -> 2.2561 (expected baseline 2.7) 2024-05-15T13:12:21.238 INFO:teuthology.orchestra.run.smithi027.stdout:min osd.1 with 3 -> 3 pgs (0.37037 -> 0.37037 * mean) 2024-05-15T13:12:21.238 INFO:teuthology.orchestra.run.smithi027.stdout:max osd.2 with 11 -> 11 pgs (1.35802 -> 1.35802 * mean) 2024-05-15T13:12:21.238 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:12:21.239 INFO:teuthology.orchestra.run.smithi027.stdout:oload 1000 2024-05-15T13:12:21.239 INFO:teuthology.orchestra.run.smithi027.stdout:max_change 1 2024-05-15T13:12:21.239 INFO:teuthology.orchestra.run.smithi027.stdout:max_change_osds 4 2024-05-15T13:12:21.239 INFO:teuthology.orchestra.run.smithi027.stdout:average_utilization 0.0058 2024-05-15T13:12:21.239 INFO:teuthology.orchestra.run.smithi027.stdout:overload_utilization 0.0583 2024-05-15T13:12:21.239 INFO:teuthology.orchestra.run.smithi027.stderr:no change 2024-05-15T13:12:21.292 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:12:21.321 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:21.675 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_4' removed 2024-05-15T13:12:21.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:21 smithi177 ceph-mon[44303]: from='client.16062 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T13:12:21.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:21 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2243347809' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T13:12:22.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:21 smithi027 ceph-mon[39795]: from='client.16062 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T13:12:22.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:21 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2243347809' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T13:12:22.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:21 smithi120 ceph-mon[43961]: from='client.16062 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T13:12:22.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:21 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2243347809' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T13:12:22.123 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-05-15T13:12:22.123 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd test-reweight-by-utilization 110 0.05 2024-05-15T13:12:22.150 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:12:22.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:22.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:12:22.254 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:12:22.280 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:22.298 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:12:22.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:22.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:12:22.403 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:12:22.429 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:22.447 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:12:22.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:22.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:12:22.552 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:12:22.579 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:22.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:22 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2243347809' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T13:12:22.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:22 smithi177 ceph-mon[44303]: osdmap e595: 12 total, 12 up, 10 in 2024-05-15T13:12:22.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:22 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:12:22.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:22 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:12:22.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:22 smithi177 ceph-mon[44303]: pgmap v1576: 11 pgs: 11 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:23.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:22 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2243347809' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T13:12:23.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:22 smithi027 ceph-mon[39795]: osdmap e595: 12 total, 12 up, 10 in 2024-05-15T13:12:23.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:22 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:12:23.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:22 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:12:23.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:22 smithi027 ceph-mon[39795]: pgmap v1576: 11 pgs: 11 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:23.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:22 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2243347809' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T13:12:23.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:22 smithi120 ceph-mon[43961]: osdmap e595: 12 total, 12 up, 10 in 2024-05-15T13:12:23.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:22 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:12:23.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:22 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:12:23.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:22 smithi120 ceph-mon[43961]: pgmap v1576: 11 pgs: 11 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:23.761 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:23 smithi027 ceph-mon[39795]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull) 2024-05-15T13:12:23.762 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:23 smithi027 ceph-mon[39795]: osdmap e596: 12 total, 12 up, 10 in 2024-05-15T13:12:23.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:23 smithi177 ceph-mon[44303]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull) 2024-05-15T13:12:23.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:23 smithi177 ceph-mon[44303]: osdmap e596: 12 total, 12 up, 10 in 2024-05-15T13:12:24.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:23 smithi120 ceph-mon[43961]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull) 2024-05-15T13:12:24.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:23 smithi120 ceph-mon[43961]: osdmap e596: 12 total, 12 up, 10 in 2024-05-15T13:12:24.093 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-05-15T13:12:24.094 INFO:tasks.radosbench:Beginning radosbench... 2024-05-15T13:12:24.094 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-05-15T13:12:24.094 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool create unique_pool_5 16 2024-05-15T13:12:24.330 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:12:24.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:24.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:12:24.438 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:12:24.466 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:24.957 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:24 smithi027 ceph-mon[39795]: pgmap v1578: 11 pgs: 11 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:24.981 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-05-15T13:12:25.016 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:12:25.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:25.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:12:25.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:24 smithi120 ceph-mon[43961]: pgmap v1578: 11 pgs: 11 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:25.111 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:12:25.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:12:25.130 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:12:25.133 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:12:25.166 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:25.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:24 smithi177 ceph-mon[44303]: pgmap v1578: 11 pgs: 11 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:25.424 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:12:25.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:25.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:12:25.529 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:12:25.562 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:25.582 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:12:25.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:25.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:12:25.687 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:12:25.691 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:12:25.716 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:25.736 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:12:25.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:25.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:12:25.841 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:12:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:25.993 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:12:25.995 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:25 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:12:25.995 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:25 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:12:25.995 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:25 smithi027 ceph-mon[39795]: osdmap e597: 12 total, 12 up, 10 in 2024-05-15T13:12:25.995 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:25 smithi027 ceph-mon[39795]: from='client.16074 172.21.15.27:0/2438346244' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:12:25.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:25.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:12:26.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:25 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:12:26.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:25 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:12:26.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:25 smithi120 ceph-mon[43961]: osdmap e597: 12 total, 12 up, 10 in 2024-05-15T13:12:26.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:25 smithi120 ceph-mon[43961]: from='client.16074 172.21.15.27:0/2438346244' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:12:26.098 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:12:26.130 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:26.152 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:12:26.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:26.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:12:26.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:25 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:12:26.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:25 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:12:26.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:25 smithi177 ceph-mon[44303]: osdmap e597: 12 total, 12 up, 10 in 2024-05-15T13:12:26.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:25 smithi177 ceph-mon[44303]: from='client.16074 172.21.15.27:0/2438346244' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:12:26.256 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:12:26.285 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:26.306 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:12:26.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:26.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:12:26.410 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:12:26.438 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:26.458 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:12:26.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:26.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:12:26.562 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:12:26.592 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:26.666 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:12:26.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:26.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:12:26.770 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:12:26.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:26 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:12:26.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:26 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T13:12:26.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:26 smithi120 ceph-mon[43961]: from='client.16074 172.21.15.27:0/2438346244' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:12:26.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:26 smithi120 ceph-mon[43961]: osdmap e598: 12 total, 12 up, 10 in 2024-05-15T13:12:26.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:26 smithi120 ceph-mon[43961]: pgmap v1581: 11 pgs: 11 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:26.798 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:26.897 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:12:26.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:26.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:12:26.930 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:26 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:12:26.930 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:26 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T13:12:26.930 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:26 smithi027 ceph-mon[39795]: from='client.16074 172.21.15.27:0/2438346244' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:12:26.931 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:26 smithi027 ceph-mon[39795]: osdmap e598: 12 total, 12 up, 10 in 2024-05-15T13:12:26.931 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:26 smithi027 ceph-mon[39795]: pgmap v1581: 11 pgs: 11 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:27.002 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:12:27.030 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:27.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:26 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:12:27.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:26 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T13:12:27.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:26 smithi177 ceph-mon[44303]: from='client.16074 172.21.15.27:0/2438346244' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:12:27.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:26 smithi177 ceph-mon[44303]: osdmap e598: 12 total, 12 up, 10 in 2024-05-15T13:12:27.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:26 smithi177 ceph-mon[44303]: pgmap v1581: 11 pgs: 11 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:27.228 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:12:27.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:27.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:12:27.333 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:12:27.365 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:27.386 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:12:27.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:27.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:12:27.490 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:12:27.520 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:27.540 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:12:27.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:27.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:12:27.656 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:12:27.684 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:27.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:27 smithi027 ceph-mon[39795]: 1.4 scrub starts 2024-05-15T13:12:27.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:27 smithi027 ceph-mon[39795]: 1.4 scrub ok 2024-05-15T13:12:28.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:27 smithi120 ceph-mon[43961]: 1.4 scrub starts 2024-05-15T13:12:28.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:27 smithi120 ceph-mon[43961]: 1.4 scrub ok 2024-05-15T13:12:28.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:27 smithi177 ceph-mon[44303]: 1.4 scrub starts 2024-05-15T13:12:28.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:27 smithi177 ceph-mon[44303]: 1.4 scrub ok 2024-05-15T13:12:28.234 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:12:28.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:28.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:12:28.338 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:12:28.370 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:28.391 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:12:28.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:28.395 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:12:28.496 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:12:28.527 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:28.549 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:12:28.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:28.552 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:12:28.653 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:12:28.683 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:28.703 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:12:28.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:28.707 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:12:28.707 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:12:28.708 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:12:28.808 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:12:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:28.907 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_5' created 2024-05-15T13:12:28.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:28 smithi177 ceph-mon[44303]: pgmap v1582: 11 pgs: 11 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:28.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:28 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/423670871' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-05-15T13:12:28.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:28 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/423670871' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-05-15T13:12:29.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:28 smithi027 ceph-mon[39795]: pgmap v1582: 11 pgs: 11 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:29.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:28 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/423670871' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-05-15T13:12:29.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:28 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/423670871' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-05-15T13:12:29.213 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:12:29.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:29.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:12:29.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:28 smithi120 ceph-mon[43961]: pgmap v1582: 11 pgs: 11 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:29.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:28 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/423670871' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-05-15T13:12:29.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:28 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/423670871' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-05-15T13:12:29.318 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:12:29.348 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:29.402 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:12:29.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:29.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:12:29.507 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:12:29.536 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:29.984 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:12:30.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:29 smithi177 ceph-mon[44303]: osdmap e599: 12 total, 12 up, 10 in 2024-05-15T13:12:30.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:29 smithi177 ceph-mon[44303]: osdmap e600: 12 total, 12 up, 10 in 2024-05-15T13:12:30.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:29 smithi027 ceph-mon[39795]: osdmap e599: 12 total, 12 up, 10 in 2024-05-15T13:12:30.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:29 smithi027 ceph-mon[39795]: osdmap e600: 12 total, 12 up, 10 in 2024-05-15T13:12:30.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:29 smithi120 ceph-mon[43961]: osdmap e599: 12 total, 12 up, 10 in 2024-05-15T13:12:30.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:29 smithi120 ceph-mon[43961]: osdmap e600: 12 total, 12 up, 10 in 2024-05-15T13:12:30.767 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-05-15T13:12:30.997 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:30 smithi027 ceph-mon[39795]: pgmap v1585: 27 pgs: 1 creating+peering, 10 unknown, 16 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:30.997 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:30 smithi027 ceph-mon[39795]: osdmap e601: 12 total, 12 up, 10 in 2024-05-15T13:12:31.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:30 smithi177 ceph-mon[44303]: pgmap v1585: 27 pgs: 1 creating+peering, 10 unknown, 16 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:31.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:30 smithi177 ceph-mon[44303]: osdmap e601: 12 total, 12 up, 10 in 2024-05-15T13:12:31.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:30 smithi120 ceph-mon[43961]: pgmap v1585: 27 pgs: 1 creating+peering, 10 unknown, 16 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:31.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:30 smithi120 ceph-mon[43961]: osdmap e601: 12 total, 12 up, 10 in 2024-05-15T13:12:31.686 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:12:31.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:31.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:12:31.790 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:12:31.823 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:31.840 INFO:teuthology.orchestra.run.smithi027.stdout:moved 1 / 81 (1.23457%) 2024-05-15T13:12:31.840 INFO:teuthology.orchestra.run.smithi027.stdout:avg 8.1 2024-05-15T13:12:31.840 INFO:teuthology.orchestra.run.smithi027.stdout:stddev 2.50799 -> 2.80891 (expected baseline 2.7) 2024-05-15T13:12:31.840 INFO:teuthology.orchestra.run.smithi027.stdout:min osd.6 with 5 -> 5 pgs (0.617284 -> 0.617284 * mean) 2024-05-15T13:12:31.840 INFO:teuthology.orchestra.run.smithi027.stdout:max osd.10 with 13 -> 14 pgs (1.60494 -> 1.72839 * mean) 2024-05-15T13:12:31.840 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:12:31.840 INFO:teuthology.orchestra.run.smithi027.stdout:oload 110 2024-05-15T13:12:31.840 INFO:teuthology.orchestra.run.smithi027.stdout:max_change 0.05 2024-05-15T13:12:31.840 INFO:teuthology.orchestra.run.smithi027.stdout:max_change_osds 4 2024-05-15T13:12:31.840 INFO:teuthology.orchestra.run.smithi027.stdout:average_utilization 0.0059 2024-05-15T13:12:31.840 INFO:teuthology.orchestra.run.smithi027.stdout:overload_utilization 0.0065 2024-05-15T13:12:31.841 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9 weight 0.6992 -> 0.6492 2024-05-15T13:12:31.841 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 weight 0.7194 -> 0.6694 2024-05-15T13:12:31.841 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 weight 0.6948 -> 0.6448 2024-05-15T13:12:31.842 INFO:teuthology.orchestra.run.smithi027.stderr:no change 2024-05-15T13:12:31.844 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:12:31.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:31.847 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:12:31.948 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:12:31.977 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:31.997 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:12:32.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:32.001 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:12:32.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:31 smithi027 ceph-mon[39795]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T13:12:32.023 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 13:12:31 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T13:12:31.846+0000 7f6a7fdb4700 -1 mgr.server reply reply (3) No such process no change 2024-05-15T13:12:32.101 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:12:32.134 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:32.155 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:12:32.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:32.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:12:32.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:31 smithi177 ceph-mon[44303]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T13:12:32.259 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:12:32.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:31 smithi120 ceph-mon[43961]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T13:12:32.290 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:32.311 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:12:32.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:32.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:12:32.339 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:12:32.368 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:12:32.387 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:12:32.407 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:12:32.415 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:12:32.426 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:12:32.446 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:32.447 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:12:32.469 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:12:32.488 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:12:32.509 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:12:32.529 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:12:32.550 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:12:32.570 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:12:32.591 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:12:32.592 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:12:32.592 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:12:32.592 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:12:32.592 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:12:32.593 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:12:32.593 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:12:32.593 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:12:32.593 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:12:32.593 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:12:32.593 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:12:32.593 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:12:32.593 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:12:33.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:32 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:12:33.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:32 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:12:33.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:32 smithi177 ceph-mon[44303]: from='client.16086 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T13:12:33.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:32 smithi177 ceph-mon[44303]: osdmap e602: 12 total, 12 up, 10 in 2024-05-15T13:12:33.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:32 smithi177 ceph-mon[44303]: pgmap v1588: 27 pgs: 1 creating+peering, 5 unknown, 21 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:33.183 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:32 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1408118649' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:12:33.183 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:32 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:12:33.183 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:32 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:12:33.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:32 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:12:33.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:32 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:12:33.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:32 smithi027 ceph-mon[39795]: from='client.16086 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T13:12:33.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:32 smithi027 ceph-mon[39795]: osdmap e602: 12 total, 12 up, 10 in 2024-05-15T13:12:33.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:32 smithi027 ceph-mon[39795]: pgmap v1588: 27 pgs: 1 creating+peering, 5 unknown, 21 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:33.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:32 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1408118649' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:12:33.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:32 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:12:33.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:32 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:12:33.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:32 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:12:33.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:32 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:12:33.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:32 smithi120 ceph-mon[43961]: from='client.16086 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T13:12:33.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:32 smithi120 ceph-mon[43961]: osdmap e602: 12 total, 12 up, 10 in 2024-05-15T13:12:33.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:32 smithi120 ceph-mon[43961]: pgmap v1588: 27 pgs: 1 creating+peering, 5 unknown, 21 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:33.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:32 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1408118649' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:12:33.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:32 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:12:33.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:32 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:12:33.504 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:12:34.196 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:12:34.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:33 smithi027 ceph-mon[39795]: osdmap e603: 12 total, 12 up, 10 in 2024-05-15T13:12:34.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:34.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:12:34.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:33 smithi120 ceph-mon[43961]: osdmap e603: 12 total, 12 up, 10 in 2024-05-15T13:12:34.301 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:12:34.331 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:33 smithi177 ceph-mon[44303]: osdmap e603: 12 total, 12 up, 10 in 2024-05-15T13:12:34.333 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:34.354 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:12:34.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:34.357 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:12:34.458 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:12:34.487 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:34.507 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:12:34.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:34.511 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:12:34.612 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:12:34.642 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:34.741 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:12:34.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:34.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:12:34.845 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:12:34.878 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:34.899 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:12:34.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:34.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:12:35.003 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:12:35.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:34 smithi027 ceph-mon[39795]: pgmap v1590: 27 pgs: 27 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:35.035 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:35.056 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:12:35.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:35.059 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:12:35.154 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:12:35.154 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:12:35.160 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:12:35.192 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:35.213 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:12:35.215 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:34 smithi120 ceph-mon[43961]: pgmap v1590: 27 pgs: 27 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:35.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:35.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:12:35.317 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:12:35.348 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:35.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:34 smithi177 ceph-mon[44303]: pgmap v1590: 27 pgs: 27 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:35.496 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:12:35.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:35.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:12:35.600 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:12:35.629 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:35.964 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:12:36.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:35 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:12:36.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:35 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:12:36.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:35 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:12:36.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:35 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:12:36.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:35 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:12:36.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:35 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:12:36.779 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:12:36.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:36.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:12:36.814 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-05-15T13:12:36.885 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:12:36.927 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:37.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:36 smithi027 ceph-mon[39795]: osdmap e604: 12 total, 12 up, 10 in 2024-05-15T13:12:37.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:36 smithi027 ceph-mon[39795]: pgmap v1592: 27 pgs: 27 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:37.247 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:12:37.254 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:12:37.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:36 smithi120 ceph-mon[43961]: osdmap e604: 12 total, 12 up, 10 in 2024-05-15T13:12:37.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:36 smithi120 ceph-mon[43961]: pgmap v1592: 27 pgs: 27 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:37.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:36 smithi177 ceph-mon[44303]: osdmap e604: 12 total, 12 up, 10 in 2024-05-15T13:12:37.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:36 smithi177 ceph-mon[44303]: pgmap v1592: 27 pgs: 27 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:37.677 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:12:37.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:37.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:12:37.782 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:12:37.814 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:37.836 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:12:37.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:37.839 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:12:37.898 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 8, 9, 10, 11] out_osds: [7, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:12:37.898 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:12:37.940 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:12:37.972 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:37.993 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:12:37.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:37.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:12:38.098 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:12:38.130 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:38.151 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:12:38.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:38.154 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:12:38.183 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:38 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1892434278' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:12:38.183 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:12:38.254 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:12:38.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:38 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1892434278' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:12:38.286 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:38.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:38 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1892434278' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:12:38.935 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:12:38.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:38.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:12:39.045 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:12:39.078 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:39.099 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:12:39.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:39.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:12:39.203 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:12:39.232 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:39.252 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:12:39.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:39.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:12:39.356 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:12:39.388 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:39.409 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:12:39.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:39.415 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:12:39.516 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:12:39.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:39 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:12:39.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:39 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1892434278' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:12:39.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:39 smithi027 ceph-mon[39795]: osdmap e605: 12 total, 12 up, 10 in 2024-05-15T13:12:39.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:39 smithi027 ceph-mon[39795]: pgmap v1594: 27 pgs: 27 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:39.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:39 smithi027 ceph-mon[39795]: osdmap e606: 12 total, 12 up, 10 in 2024-05-15T13:12:39.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:39 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:12:39.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:39 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1892434278' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:12:39.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:39 smithi120 ceph-mon[43961]: osdmap e605: 12 total, 12 up, 10 in 2024-05-15T13:12:39.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:39 smithi120 ceph-mon[43961]: pgmap v1594: 27 pgs: 27 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:39.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:39 smithi120 ceph-mon[43961]: osdmap e606: 12 total, 12 up, 10 in 2024-05-15T13:12:39.543 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:39.563 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:12:39.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:39.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:12:39.666 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:12:39.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:39 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:12:39.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:39 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1892434278' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:12:39.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:39 smithi177 ceph-mon[44303]: osdmap e605: 12 total, 12 up, 10 in 2024-05-15T13:12:39.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:39 smithi177 ceph-mon[44303]: pgmap v1594: 27 pgs: 27 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:39.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:39 smithi177 ceph-mon[44303]: osdmap e606: 12 total, 12 up, 10 in 2024-05-15T13:12:39.697 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:40.146 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:12:40.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:40.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:12:40.199 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-05-15T13:12:40.251 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:12:40.282 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:40.303 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:12:40.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:40.306 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:12:40.407 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:12:40.436 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:40.456 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:12:40.458 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:40 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1110037078' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T13:12:40.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:40.459 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:12:40.523 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:40 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1110037078' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T13:12:40.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:40 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1110037078' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T13:12:40.560 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:12:40.591 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:40.611 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:12:40.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:40.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:12:40.721 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:12:40.752 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:40.773 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:12:40.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:40.776 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:12:40.877 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:12:40.906 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:41.004 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:12:41.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:41.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:12:41.109 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:12:41.141 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:41.162 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:12:41.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:41.165 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:12:41.266 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:12:41.298 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:41.320 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:12:41.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:41.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:12:41.424 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:12:41.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:41 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1110037078' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T13:12:41.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:41 smithi177 ceph-mon[44303]: osdmap e607: 12 total, 12 up, 10 in 2024-05-15T13:12:41.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:41 smithi177 ceph-mon[44303]: pgmap v1597: 27 pgs: 27 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:41.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:41 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1110037078' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T13:12:41.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:41 smithi027 ceph-mon[39795]: osdmap e607: 12 total, 12 up, 10 in 2024-05-15T13:12:41.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:41 smithi027 ceph-mon[39795]: pgmap v1597: 27 pgs: 27 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:41.452 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:41.505 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:12:41.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:41.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:12:41.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:41 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1110037078' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T13:12:41.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:41 smithi120 ceph-mon[43961]: osdmap e607: 12 total, 12 up, 10 in 2024-05-15T13:12:41.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:41 smithi120 ceph-mon[43961]: pgmap v1597: 27 pgs: 27 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:41.613 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:12:41.644 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:41.667 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:12:41.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:41.669 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:12:41.770 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:12:41.802 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:41.823 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:12:41.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:41.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:12:41.927 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:12:41.957 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:41.977 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:12:41.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:41.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:12:42.081 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:12:42.110 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:42.461 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:12:42.463 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:42 smithi027 ceph-mon[39795]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T13:12:42.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:42.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:12:42.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:42 smithi120 ceph-mon[43961]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T13:12:42.565 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:12:42.596 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:42.598 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:12:42.617 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:12:42.618 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:12:42.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:42.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:12:42.638 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:12:42.657 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:12:42.674 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:12:42.674 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:12:42.678 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:12:42.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:42 smithi177 ceph-mon[44303]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T13:12:42.698 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:12:42.719 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:12:42.721 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:12:42.739 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:12:42.753 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:42.759 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:12:42.775 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:12:42.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:42.778 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:12:42.779 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:12:42.800 DEBUG:teuthology.orchestra.run.smithi177:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-05-15T13:12:42.801 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-05-15T13:12:42.801 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:12:42.801 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:12:42.820 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:12:42.821 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-05-15T13:12:42.821 INFO:teuthology.orchestra.run:waiting for 3000 2024-05-15T13:12:42.821 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:12:42.822 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:12:42.822 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:12:42.822 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:12:42.822 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:12:42.822 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:12:42.822 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:12:42.822 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:12:42.822 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:12:42.822 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:12:42.822 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:12:42.822 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:12:42.871 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:hints = 1 2024-05-15T13:12:42.872 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-05-15T13:12:42.872 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Object prefix: benchmark_data_smithi177_134484 2024-05-15T13:12:42.872 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:12:42.872 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 0 0 0 0 0 0 - 0 2024-05-15T13:12:42.879 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:12:42.923 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:42.948 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:12:42.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:42.953 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:12:43.054 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:12:43.080 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:43.099 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:12:43.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:43.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:12:43.204 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:12:43.209 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:43 smithi027 ceph-mon[39795]: pgmap v1598: 27 pgs: 27 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:43.209 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:43 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1302658026' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:12:43.234 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:43.255 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:12:43.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:43.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:12:43.359 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:12:43.361 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:12:43.399 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:43.429 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:12:43.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:43.433 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:12:43.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:43 smithi120 ceph-mon[43961]: pgmap v1598: 27 pgs: 27 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:43.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:43 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1302658026' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:12:43.534 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:12:43.569 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:43 smithi177 ceph-mon[44303]: pgmap v1598: 27 pgs: 27 active+clean; 577 KiB data, 5.2 GiB used, 889 GiB / 894 GiB avail 2024-05-15T13:12:43.570 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:43 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1302658026' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:12:43.574 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:43.607 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:12:43.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:43.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:12:43.712 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:12:43.720 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-05-15T13:12:43.769 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:43.871 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 1 15 2976 2961 185.052 185.062 0.00999861 0.00537419 2024-05-15T13:12:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:12:44.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:44.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:12:44.242 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:12:44.242 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:12:44.243 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:12:44.281 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:12:44.322 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:44.352 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:12:44.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:44.359 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:12:44.460 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:12:44.506 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:44.725 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:12:44.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:44.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:12:44.835 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:12:44.871 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 2 16 5578 5562 173.778 162.562 0.00680396 0.0057387 2024-05-15T13:12:44.878 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:44.911 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:12:44.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:44.915 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:12:45.016 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:12:45.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:44 smithi027 ceph-mon[39795]: osdmap e608: 12 total, 12 up, 10 in 2024-05-15T13:12:45.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:44 smithi027 ceph-mon[39795]: pgmap v1600: 27 pgs: 27 active+clean; 26 MiB data, 5.3 GiB used, 889 GiB / 894 GiB avail; 4.3 MiB/s wr, 68 op/s 2024-05-15T13:12:45.041 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:45.060 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:12:45.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:45.062 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:12:45.163 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:12:45.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:44 smithi177 ceph-mon[44303]: osdmap e608: 12 total, 12 up, 10 in 2024-05-15T13:12:45.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:44 smithi177 ceph-mon[44303]: pgmap v1600: 27 pgs: 27 active+clean; 26 MiB data, 5.3 GiB used, 889 GiB / 894 GiB avail; 4.3 MiB/s wr, 68 op/s 2024-05-15T13:12:45.190 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:45.208 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:12:45.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:45.211 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:12:45.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:44 smithi120 ceph-mon[43961]: osdmap e608: 12 total, 12 up, 10 in 2024-05-15T13:12:45.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:44 smithi120 ceph-mon[43961]: pgmap v1600: 27 pgs: 27 active+clean; 26 MiB data, 5.3 GiB used, 889 GiB / 894 GiB avail; 4.3 MiB/s wr, 68 op/s 2024-05-15T13:12:45.311 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:12:45.338 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:45.357 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:12:45.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:45.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:12:45.461 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:12:45.495 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:45.871 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 3 16 7104 7088 147.643 95.375 0.00441742 0.0067631 2024-05-15T13:12:46.159 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:12:46.650 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:12:46.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:46.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:12:46.758 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:12:46.803 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:46.830 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:12:46.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:46.833 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:12:46.871 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 4 16 10267 10251 160.149 197.688 0.00480399 0.00623628 2024-05-15T13:12:46.934 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:12:46.989 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:47.024 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:12:47.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:47.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:12:47.130 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:12:47.170 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:47.201 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:12:47.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:47.205 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:12:47.306 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:12:47.349 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:47.377 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:12:47.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:47.381 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:12:47.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:47 smithi177 ceph-mon[44303]: pgmap v1601: 27 pgs: 27 active+clean; 172 MiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 23 MiB/s wr, 372 op/s 2024-05-15T13:12:47.482 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:12:47.525 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:47.554 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:12:47.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:47.558 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:12:47.659 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:12:47.686 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:47 smithi120 ceph-mon[43961]: pgmap v1601: 27 pgs: 27 active+clean; 172 MiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 23 MiB/s wr, 372 op/s 2024-05-15T13:12:47.688 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:47.709 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:12:47.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:47.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:12:47.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:47 smithi027 ceph-mon[39795]: pgmap v1601: 27 pgs: 27 active+clean; 172 MiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 23 MiB/s wr, 372 op/s 2024-05-15T13:12:47.813 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:12:47.863 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:47.871 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 5 16 13333 13317 166.438 191.625 0.0070334 0.00599979 2024-05-15T13:12:47.902 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:12:47.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:47.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:12:48.008 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:12:48.034 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:48.052 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:12:48.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:48.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:12:48.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:12:48.202 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:48.413 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:12:48.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:48.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:12:48.526 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:12:48.556 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:48.577 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:12:48.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:48.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:12:48.681 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:12:48.716 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:48.736 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:12:48.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:48.739 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:12:48.840 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:12:48.869 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:48.872 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 6 16 15556 15540 161.838 138.938 0.00423878 0.00616983 2024-05-15T13:12:48.889 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:12:48.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:48.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:12:48.993 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:12:49.020 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:49.038 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:12:49.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:49.041 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:12:49.142 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:12:49.173 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:49.217 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:12:49.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:49.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:12:49.323 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:12:49.365 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:49.393 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:12:49.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:49.397 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:12:49.498 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:12:49.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:49 smithi027 ceph-mon[39795]: pgmap v1602: 27 pgs: 27 active+clean; 469 MiB data, 6.6 GiB used, 887 GiB / 894 GiB avail; 58 MiB/s wr, 932 op/s 2024-05-15T13:12:49.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:49 smithi027 ceph-mon[39795]: osdmap e609: 12 total, 12 up, 10 in 2024-05-15T13:12:49.544 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:49.665 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:12:49.677 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:12:49.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:49 smithi177 ceph-mon[44303]: pgmap v1602: 27 pgs: 27 active+clean; 469 MiB data, 6.6 GiB used, 887 GiB / 894 GiB avail; 58 MiB/s wr, 932 op/s 2024-05-15T13:12:49.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:49 smithi177 ceph-mon[44303]: osdmap e609: 12 total, 12 up, 10 in 2024-05-15T13:12:49.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:49 smithi120 ceph-mon[43961]: pgmap v1602: 27 pgs: 27 active+clean; 469 MiB data, 6.6 GiB used, 887 GiB / 894 GiB avail; 58 MiB/s wr, 932 op/s 2024-05-15T13:12:49.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:49 smithi120 ceph-mon[43961]: osdmap e609: 12 total, 12 up, 10 in 2024-05-15T13:12:49.872 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 7 15 17552 17537 156.547 124.812 0.00592209 0.00638175 2024-05-15T13:12:50.872 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 8 16 20171 20155 157.43 163.625 0.00472501 0.00634139 2024-05-15T13:12:51.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:51 smithi177 ceph-mon[44303]: from='client.16125 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:12:51.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:51 smithi177 ceph-mon[44303]: pgmap v1604: 27 pgs: 27 active+clean; 862 MiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-05-15T13:12:51.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:51 smithi027 ceph-mon[39795]: from='client.16125 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:12:51.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:51 smithi027 ceph-mon[39795]: pgmap v1604: 27 pgs: 27 active+clean; 862 MiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-05-15T13:12:51.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:51 smithi120 ceph-mon[43961]: from='client.16125 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:12:51.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:51 smithi120 ceph-mon[43961]: pgmap v1604: 27 pgs: 27 active+clean; 862 MiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-05-15T13:12:51.872 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 9 16 22122 22106 153.485 121.938 0.0026465 0.00650427 2024-05-15T13:12:52.542 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:12:52.542 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:12:52.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:12:52.542 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:23602 7.d 7:b260f9d4:::benchmark_data_smithi177_134484_object23601:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e609)", 2024-05-15T13:12:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:12:52.541988+0000", 2024-05-15T13:12:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.0022315730000000002, 2024-05-15T13:12:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0022677980000000001, 2024-05-15T13:12:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:12:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:12:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:12:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:12:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:12:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 23602 2024-05-15T13:12:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:12:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:12:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:12:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:12:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:52.541988+0000", 2024-05-15T13:12:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:12:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:12:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:12:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:12:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:52.541988+0000", 2024-05-15T13:12:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:12:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:12:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:12:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:12:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:52.541989+0000", 2024-05-15T13:12:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.536e-06 2024-05-15T13:12:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:12:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:12:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:12:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:52.542019+0000", 2024-05-15T13:12:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0068000000000001e-05 2024-05-15T13:12:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:12:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:12:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:12:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:52.542037+0000", 2024-05-15T13:12:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7156e-05 2024-05-15T13:12:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:12:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:12:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:12:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:52.542044+0000", 2024-05-15T13:12:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0199999999999997e-06 2024-05-15T13:12:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:12:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:12:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:12:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:52.542159+0000", 2024-05-15T13:12:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000115361 2024-05-15T13:12:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:12:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:12:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:12:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:52.542321+0000", 2024-05-15T13:12:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016175299999999999 2024-05-15T13:12:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:12:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:12:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,3", 2024-05-15T13:12:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:52.542448+0000", 2024-05-15T13:12:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012710999999999999 2024-05-15T13:12:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:12:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:12:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:12:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:52.544078+0000", 2024-05-15T13:12:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001629845 2024-05-15T13:12:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:12:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:12:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:12:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:12:52.548 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:12:52.548 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 1 2024-05-15T13:12:52.548 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:12:52.873 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 10 16 24317 24301 151.854 137.188 0.00191021 0.00654851 2024-05-15T13:12:53.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:53 smithi177 ceph-mon[44303]: pgmap v1605: 27 pgs: 27 active+clean; 1.1 GiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-05-15T13:12:53.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:53 smithi027 ceph-mon[39795]: pgmap v1605: 27 pgs: 27 active+clean; 1.1 GiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-05-15T13:12:53.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:53 smithi120 ceph-mon[43961]: pgmap v1605: 27 pgs: 27 active+clean; 1.1 GiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-05-15T13:12:53.873 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 11 16 26484 26468 150.36 135.438 0.0365829 0.00663429 2024-05-15T13:12:54.873 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 12 16 28749 28733 149.626 141.562 0.00417413 0.00667788 2024-05-15T13:12:55.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:54 smithi027 ceph-mon[39795]: osdmap e610: 12 total, 12 up, 10 in 2024-05-15T13:12:55.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:54 smithi027 ceph-mon[39795]: pgmap v1607: 27 pgs: 27 active+clean; 1.3 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 149 MiB/s wr, 2.38k op/s 2024-05-15T13:12:55.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:54 smithi177 ceph-mon[44303]: osdmap e610: 12 total, 12 up, 10 in 2024-05-15T13:12:55.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:54 smithi177 ceph-mon[44303]: pgmap v1607: 27 pgs: 27 active+clean; 1.3 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 149 MiB/s wr, 2.38k op/s 2024-05-15T13:12:55.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:54 smithi120 ceph-mon[43961]: osdmap e610: 12 total, 12 up, 10 in 2024-05-15T13:12:55.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:54 smithi120 ceph-mon[43961]: pgmap v1607: 27 pgs: 27 active+clean; 1.3 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 149 MiB/s wr, 2.38k op/s 2024-05-15T13:12:55.873 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 13 16 30728 30712 147.63 123.688 0.00361153 0.00676967 2024-05-15T13:12:56.130 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-05-15T13:12:56.577 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:12:56.610 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":1603,"stamp":"2024-05-15T13:12:48.874158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600965664,"num_objects":9163,"num_object_clones":0,"num_object_copies":27489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9161,"num_write_kb":586304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10690,"ondisk_log_size":10690,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":7213140,"kb_used_data":1661884,"kb_used_omap":15,"kb_used_meta":5551088,"kb_avail":930238380,"statfs":{"total":959950356480,"available":952564101120,"internally_reserved":0,"allocated":1701769216,"data_stored":1698449705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15956,"internal_metadata":5684314540},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,1,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":600375296,"num_objects":9161,"num_object_clones":0,"num_object_copies":27483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9161,"num_write_kb":586304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.624897"},"pg_stats":[{"pgid":"7.d","version":"608'629","reported_seq":657,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:47.005407+0000","last_change":"2024-05-15T13:12:29.918499+0000","last_active":"2024-05-15T13:12:47.005407+0000","last_peered":"2024-05-15T13:12:47.005407+0000","last_clean":"2024-05-15T13:12:47.005407+0000","last_became_active":"2024-05-15T13:12:29.918181+0000","last_became_peered":"2024-05-15T13:12:29.918181+0000","last_unstale":"2024-05-15T13:12:47.005407+0000","last_undegraded":"2024-05-15T13:12:47.005407+0000","last_fullsized":"2024-05-15T13:12:47.005407+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:34.292215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41222144,"num_objects":629,"num_object_clones":0,"num_object_copies":1887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":629,"num_write_kb":40256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.b","version":"608'928","reported_seq":956,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:48.364968+0000","last_change":"2024-05-15T13:12:29.922254+0000","last_active":"2024-05-15T13:12:48.364968+0000","last_peered":"2024-05-15T13:12:48.364968+0000","last_clean":"2024-05-15T13:12:48.364968+0000","last_became_active":"2024-05-15T13:12:29.921964+0000","last_became_peered":"2024-05-15T13:12:29.921964+0000","last_unstale":"2024-05-15T13:12:48.364968+0000","last_undegraded":"2024-05-15T13:12:48.364968+0000","last_fullsized":"2024-05-15T13:12:48.364968+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":928,"log_dups_size":0,"ondisk_log_size":928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:36.969896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60817408,"num_objects":928,"num_object_clones":0,"num_object_copies":2784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":928,"num_write_kb":59392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,10],"acting":[5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"608'473","reported_seq":500,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:45.846764+0000","last_change":"2024-05-15T13:12:29.923682+0000","last_active":"2024-05-15T13:12:45.846764+0000","last_peered":"2024-05-15T13:12:45.846764+0000","last_clean":"2024-05-15T13:12:45.846764+0000","last_became_active":"2024-05-15T13:12:29.923300+0000","last_became_peered":"2024-05-15T13:12:29.923300+0000","last_unstale":"2024-05-15T13:12:45.846764+0000","last_undegraded":"2024-05-15T13:12:45.846764+0000","last_fullsized":"2024-05-15T13:12:45.846764+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:43.794094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30998528,"num_objects":473,"num_object_clones":0,"num_object_copies":1419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":473,"num_write_kb":30272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"608'371","reported_seq":399,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:44.992724+0000","last_change":"2024-05-15T13:12:29.921096+0000","last_active":"2024-05-15T13:12:44.992724+0000","last_peered":"2024-05-15T13:12:44.992724+0000","last_clean":"2024-05-15T13:12:44.992724+0000","last_became_active":"2024-05-15T13:12:29.920800+0000","last_became_peered":"2024-05-15T13:12:29.920800+0000","last_unstale":"2024-05-15T13:12:44.992724+0000","last_undegraded":"2024-05-15T13:12:44.992724+0000","last_fullsized":"2024-05-15T13:12:44.992724+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:29.769159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24313856,"num_objects":371,"num_object_clones":0,"num_object_copies":1113,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":371,"num_write_kb":23744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"608'834","reported_seq":861,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:47.961328+0000","last_change":"2024-05-15T13:12:29.920145+0000","last_active":"2024-05-15T13:12:47.961328+0000","last_peered":"2024-05-15T13:12:47.961328+0000","last_clean":"2024-05-15T13:12:47.961328+0000","last_became_active":"2024-05-15T13:12:29.919792+0000","last_became_peered":"2024-05-15T13:12:29.919792+0000","last_unstale":"2024-05-15T13:12:47.961328+0000","last_undegraded":"2024-05-15T13:12:47.961328+0000","last_fullsized":"2024-05-15T13:12:47.961328+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":834,"log_dups_size":0,"ondisk_log_size":834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:31.112240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54657024,"num_objects":834,"num_object_clones":0,"num_object_copies":2502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":834,"num_write_kb":53376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,1],"acting":[2,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.c","version":"608'218","reported_seq":245,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:43.999588+0000","last_change":"2024-05-15T13:12:29.921700+0000","last_active":"2024-05-15T13:12:43.999588+0000","last_peered":"2024-05-15T13:12:43.999588+0000","last_clean":"2024-05-15T13:12:43.999588+0000","last_became_active":"2024-05-15T13:12:29.921345+0000","last_became_peered":"2024-05-15T13:12:29.921345+0000","last_unstale":"2024-05-15T13:12:43.999588+0000","last_undegraded":"2024-05-15T13:12:43.999588+0000","last_fullsized":"2024-05-15T13:12:43.999588+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":218,"log_dups_size":0,"ondisk_log_size":218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:52.926517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14286848,"num_objects":218,"num_object_clones":0,"num_object_copies":654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":218,"num_write_kb":13952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,2],"acting":[6,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"94'139","reported_seq":601,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:43.875311+0000","last_change":"2024-05-15T13:12:14.568672+0000","last_active":"2024-05-15T13:12:43.875311+0000","last_peered":"2024-05-15T13:12:43.875311+0000","last_clean":"2024-05-15T13:12:43.875311+0000","last_became_active":"2024-05-15T13:12:13.628470+0000","last_became_peered":"2024-05-15T13:12:13.628470+0000","last_unstale":"2024-05-15T13:12:43.875311+0000","last_undegraded":"2024-05-15T13:12:43.875311+0000","last_fullsized":"2024-05-15T13:12:43.875311+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:12:14.568565+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:12:14.568565+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:38.236713+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.f","version":"608'366","reported_seq":394,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:44.988285+0000","last_change":"2024-05-15T13:12:29.925178+0000","last_active":"2024-05-15T13:12:44.988285+0000","last_peered":"2024-05-15T13:12:44.988285+0000","last_clean":"2024-05-15T13:12:44.988285+0000","last_became_active":"2024-05-15T13:12:29.924959+0000","last_became_peered":"2024-05-15T13:12:29.924959+0000","last_unstale":"2024-05-15T13:12:44.988285+0000","last_undegraded":"2024-05-15T13:12:44.988285+0000","last_fullsized":"2024-05-15T13:12:44.988285+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:54.714294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23986176,"num_objects":366,"num_object_clones":0,"num_object_copies":1098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":366,"num_write_kb":23424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"94'139","reported_seq":1088,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:43.878423+0000","last_change":"2024-05-15T13:11:47.340004+0000","last_active":"2024-05-15T13:12:43.878423+0000","last_peered":"2024-05-15T13:12:43.878423+0000","last_clean":"2024-05-15T13:12:43.878423+0000","last_became_active":"2024-05-15T13:06:46.424667+0000","last_became_peered":"2024-05-15T13:06:46.424667+0000","last_unstale":"2024-05-15T13:12:43.878423+0000","last_undegraded":"2024-05-15T13:12:43.878423+0000","last_fullsized":"2024-05-15T13:12:43.878423+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:11:47.339883+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:11:47.339883+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:03.923594+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.e","version":"608'793","reported_seq":820,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:47.952863+0000","last_change":"2024-05-15T13:12:29.926687+0000","last_active":"2024-05-15T13:12:47.952863+0000","last_peered":"2024-05-15T13:12:47.952863+0000","last_clean":"2024-05-15T13:12:47.952863+0000","last_became_active":"2024-05-15T13:12:29.926405+0000","last_became_peered":"2024-05-15T13:12:29.926405+0000","last_unstale":"2024-05-15T13:12:47.952863+0000","last_undegraded":"2024-05-15T13:12:47.952863+0000","last_fullsized":"2024-05-15T13:12:47.952863+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":793,"log_dups_size":0,"ondisk_log_size":793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:37.010570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51970048,"num_objects":793,"num_object_clones":0,"num_object_copies":2379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":793,"num_write_kb":50752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"94'139","reported_seq":1088,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:43.872660+0000","last_change":"2024-05-15T13:11:36.133693+0000","last_active":"2024-05-15T13:12:43.872660+0000","last_peered":"2024-05-15T13:12:43.872660+0000","last_clean":"2024-05-15T13:12:43.872660+0000","last_became_active":"2024-05-15T13:06:46.426908+0000","last_became_peered":"2024-05-15T13:06:46.426908+0000","last_unstale":"2024-05-15T13:12:43.872660+0000","last_undegraded":"2024-05-15T13:12:43.872660+0000","last_fullsized":"2024-05-15T13:12:43.872660+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:11:36.133565+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:11:36.133565+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:12:46.082478+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.5","version":"608'835","reported_seq":863,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:47.799350+0000","last_change":"2024-05-15T13:12:29.920954+0000","last_active":"2024-05-15T13:12:47.799350+0000","last_peered":"2024-05-15T13:12:47.799350+0000","last_clean":"2024-05-15T13:12:47.799350+0000","last_became_active":"2024-05-15T13:12:29.920552+0000","last_became_peered":"2024-05-15T13:12:29.920552+0000","last_unstale":"2024-05-15T13:12:47.799350+0000","last_undegraded":"2024-05-15T13:12:47.799350+0000","last_fullsized":"2024-05-15T13:12:47.799350+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":835,"log_dups_size":0,"ondisk_log_size":835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:50.079431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54722560,"num_objects":835,"num_object_clones":0,"num_object_copies":2505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":835,"num_write_kb":53440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"94'139","reported_seq":1113,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:43.874660+0000","last_change":"2024-05-15T13:12:17.296031+0000","last_active":"2024-05-15T13:12:43.874660+0000","last_peered":"2024-05-15T13:12:43.874660+0000","last_clean":"2024-05-15T13:12:43.874660+0000","last_became_active":"2024-05-15T13:12:04.588061+0000","last_became_peered":"2024-05-15T13:12:04.588061+0000","last_unstale":"2024-05-15T13:12:43.874660+0000","last_undegraded":"2024-05-15T13:12:43.874660+0000","last_fullsized":"2024-05-15T13:12:43.874660+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:12:17.295928+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:12:17.295928+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:42.464963+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,10],"acting":[1,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.4","version":"608'382","reported_seq":410,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:44.980717+0000","last_change":"2024-05-15T13:12:29.924549+0000","last_active":"2024-05-15T13:12:44.980717+0000","last_peered":"2024-05-15T13:12:44.980717+0000","last_clean":"2024-05-15T13:12:44.980717+0000","last_became_active":"2024-05-15T13:12:29.924271+0000","last_became_peered":"2024-05-15T13:12:29.924271+0000","last_unstale":"2024-05-15T13:12:44.980717+0000","last_undegraded":"2024-05-15T13:12:44.980717+0000","last_fullsized":"2024-05-15T13:12:44.980717+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":382,"log_dups_size":0,"ondisk_log_size":382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:56.477293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25034752,"num_objects":382,"num_object_clones":0,"num_object_copies":1146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":382,"num_write_kb":24448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,10],"acting":[0,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"94'139","reported_seq":1104,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:43.881096+0000","last_change":"2024-05-15T13:12:13.946263+0000","last_active":"2024-05-15T13:12:43.881096+0000","last_peered":"2024-05-15T13:12:43.881096+0000","last_clean":"2024-05-15T13:12:43.881096+0000","last_became_active":"2024-05-15T13:11:46.608493+0000","last_became_peered":"2024-05-15T13:11:46.608493+0000","last_unstale":"2024-05-15T13:12:43.881096+0000","last_undegraded":"2024-05-15T13:12:43.881096+0000","last_fullsized":"2024-05-15T13:12:43.881096+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":574,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:12:13.946140+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:12:13.946140+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:19.513480+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"608'355","reported_seq":383,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:44.996256+0000","last_change":"2024-05-15T13:12:29.921320+0000","last_active":"2024-05-15T13:12:44.996256+0000","last_peered":"2024-05-15T13:12:44.996256+0000","last_clean":"2024-05-15T13:12:44.996256+0000","last_became_active":"2024-05-15T13:12:29.920958+0000","last_became_peered":"2024-05-15T13:12:29.920958+0000","last_unstale":"2024-05-15T13:12:44.996256+0000","last_undegraded":"2024-05-15T13:12:44.996256+0000","last_fullsized":"2024-05-15T13:12:44.996256+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:39.498804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23265280,"num_objects":355,"num_object_clones":0,"num_object_copies":1065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":355,"num_write_kb":22720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,1],"acting":[8,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1","version":"94'139","reported_seq":864,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:43.881147+0000","last_change":"2024-05-15T13:12:10.009921+0000","last_active":"2024-05-15T13:12:43.881147+0000","last_peered":"2024-05-15T13:12:43.881147+0000","last_clean":"2024-05-15T13:12:43.881147+0000","last_became_active":"2024-05-15T13:12:04.585333+0000","last_became_peered":"2024-05-15T13:12:04.585333+0000","last_unstale":"2024-05-15T13:12:43.881147+0000","last_undegraded":"2024-05-15T13:12:43.881147+0000","last_fullsized":"2024-05-15T13:12:43.881147+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:12:10.009817+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:12:10.009817+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:10.107106+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,2],"acting":[0,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"608'591","reported_seq":618,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:46.588245+0000","last_change":"2024-05-15T13:12:29.920812+0000","last_active":"2024-05-15T13:12:46.588245+0000","last_peered":"2024-05-15T13:12:46.588245+0000","last_clean":"2024-05-15T13:12:46.588245+0000","last_became_active":"2024-05-15T13:12:29.920382+0000","last_became_peered":"2024-05-15T13:12:29.920382+0000","last_unstale":"2024-05-15T13:12:46.588245+0000","last_undegraded":"2024-05-15T13:12:46.588245+0000","last_fullsized":"2024-05-15T13:12:46.588245+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":591,"log_dups_size":0,"ondisk_log_size":591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:41.165370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38731776,"num_objects":591,"num_object_clones":0,"num_object_copies":1773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":591,"num_write_kb":37824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"94'139","reported_seq":855,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:43.875359+0000","last_change":"2024-05-15T13:12:11.602057+0000","last_active":"2024-05-15T13:12:43.875359+0000","last_peered":"2024-05-15T13:12:43.875359+0000","last_clean":"2024-05-15T13:12:43.875359+0000","last_became_active":"2024-05-15T13:12:04.577717+0000","last_became_peered":"2024-05-15T13:12:04.577717+0000","last_unstale":"2024-05-15T13:12:43.875359+0000","last_undegraded":"2024-05-15T13:12:43.875359+0000","last_fullsized":"2024-05-15T13:12:43.875359+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:12:11.601952+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:12:11.601952+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:22.675401+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"608'897","reported_seq":925,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:48.364542+0000","last_change":"2024-05-15T13:12:29.924331+0000","last_active":"2024-05-15T13:12:48.364542+0000","last_peered":"2024-05-15T13:12:48.364542+0000","last_clean":"2024-05-15T13:12:48.364542+0000","last_became_active":"2024-05-15T13:12:29.924027+0000","last_became_peered":"2024-05-15T13:12:29.924027+0000","last_unstale":"2024-05-15T13:12:48.364542+0000","last_undegraded":"2024-05-15T13:12:48.364542+0000","last_fullsized":"2024-05-15T13:12:48.364542+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":897,"log_dups_size":0,"ondisk_log_size":897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:55.335388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58785792,"num_objects":897,"num_object_clones":0,"num_object_copies":2691,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":897,"num_write_kb":57408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"94'139","reported_seq":1089,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:43.878541+0000","last_change":"2024-05-15T13:12:10.235434+0000","last_active":"2024-05-15T13:12:43.878541+0000","last_peered":"2024-05-15T13:12:43.878541+0000","last_clean":"2024-05-15T13:12:43.878541+0000","last_became_active":"2024-05-15T13:06:46.419250+0000","last_became_peered":"2024-05-15T13:06:46.419250+0000","last_unstale":"2024-05-15T13:12:43.878541+0000","last_undegraded":"2024-05-15T13:12:43.878541+0000","last_fullsized":"2024-05-15T13:12:43.878541+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:12:10.235320+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:12:10.235320+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:21.647539+0000","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.3","version":"608'516","reported_seq":544,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:46.495423+0000","last_change":"2024-05-15T13:12:29.922209+0000","last_active":"2024-05-15T13:12:46.495423+0000","last_peered":"2024-05-15T13:12:46.495423+0000","last_clean":"2024-05-15T13:12:46.495423+0000","last_became_active":"2024-05-15T13:12:29.921897+0000","last_became_peered":"2024-05-15T13:12:29.921897+0000","last_unstale":"2024-05-15T13:12:46.495423+0000","last_undegraded":"2024-05-15T13:12:46.495423+0000","last_fullsized":"2024-05-15T13:12:46.495423+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":516,"log_dups_size":0,"ondisk_log_size":516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:31.731347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33816576,"num_objects":516,"num_object_clones":0,"num_object_copies":1548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":516,"num_write_kb":33024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"94'139","reported_seq":841,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:43.875521+0000","last_change":"2024-05-15T13:12:13.623031+0000","last_active":"2024-05-15T13:12:43.875521+0000","last_peered":"2024-05-15T13:12:43.875521+0000","last_clean":"2024-05-15T13:12:43.875521+0000","last_became_active":"2024-05-15T13:12:13.622690+0000","last_became_peered":"2024-05-15T13:12:13.622690+0000","last_unstale":"2024-05-15T13:12:43.875521+0000","last_undegraded":"2024-05-15T13:12:43.875521+0000","last_fullsized":"2024-05-15T13:12:43.875521+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:11:28.773526+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:11:28.773526+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:12:46.050173+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.2","version":"608'584","reported_seq":612,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:46.519218+0000","last_change":"2024-05-15T13:12:29.919180+0000","last_active":"2024-05-15T13:12:46.519218+0000","last_peered":"2024-05-15T13:12:46.519218+0000","last_clean":"2024-05-15T13:12:46.519218+0000","last_became_active":"2024-05-15T13:12:29.918894+0000","last_became_peered":"2024-05-15T13:12:29.918894+0000","last_unstale":"2024-05-15T13:12:46.519218+0000","last_undegraded":"2024-05-15T13:12:46.519218+0000","last_fullsized":"2024-05-15T13:12:46.519218+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":584,"log_dups_size":0,"ondisk_log_size":584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:47.808066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38273024,"num_objects":584,"num_object_clones":0,"num_object_copies":1752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":584,"num_write_kb":37376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2],"acting":[3,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"94'139","reported_seq":1096,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:43.875757+0000","last_change":"2024-05-15T13:12:27.182191+0000","last_active":"2024-05-15T13:12:43.875757+0000","last_peered":"2024-05-15T13:12:43.875757+0000","last_clean":"2024-05-15T13:12:43.875757+0000","last_became_active":"2024-05-15T13:06:45.409132+0000","last_became_peered":"2024-05-15T13:06:45.409132+0000","last_unstale":"2024-05-15T13:12:43.875757+0000","last_undegraded":"2024-05-15T13:12:43.875757+0000","last_fullsized":"2024-05-15T13:12:43.875757+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:12:27.182062+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:12:27.182062+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:28.155289+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.6","version":"608'389","reported_seq":417,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:44.998168+0000","last_change":"2024-05-15T13:12:29.926933+0000","last_active":"2024-05-15T13:12:44.998168+0000","last_peered":"2024-05-15T13:12:44.998168+0000","last_clean":"2024-05-15T13:12:44.998168+0000","last_became_active":"2024-05-15T13:12:29.926581+0000","last_became_peered":"2024-05-15T13:12:29.926581+0000","last_unstale":"2024-05-15T13:12:44.998168+0000","last_undegraded":"2024-05-15T13:12:44.998168+0000","last_fullsized":"2024-05-15T13:12:44.998168+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:58.362844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25493504,"num_objects":389,"num_object_clones":0,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":389,"num_write_kb":24896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,5],"acting":[0,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":1096,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-05-15T13:12:43.876058+0000","last_change":"2024-05-15T13:12:19.293340+0000","last_active":"2024-05-15T13:12:43.876058+0000","last_peered":"2024-05-15T13:12:43.876058+0000","last_clean":"2024-05-15T13:12:43.876058+0000","last_became_active":"2024-05-15T13:06:42.028286+0000","last_became_peered":"2024-05-15T13:06:42.028286+0000","last_unstale":"2024-05-15T13:12:43.876058+0000","last_undegraded":"2024-05-15T13:12:43.876058+0000","last_fullsized":"2024-05-15T13:12:43.876058+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:12:19.293237+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:12:19.293237+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:13:35.954577+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":600375296,"num_objects":9161,"num_object_clones":0,"num_object_copies":27483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9161,"num_write_kb":586304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1957167104,"data_stored":1957167104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9161,"ondisk_log_size":9161,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056969,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":807200,"kb_used_data":294296,"kb_used_omap":1,"kb_used_meta":512894,"kb_avail":92937952,"statfs":{"total":95995035648,"available":95168462848,"internally_reserved":0,"allocated":301359104,"data_stored":301027767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":525203910},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351308,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":742624,"kb_used_data":224152,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":93002528,"statfs":{"total":95995035648,"available":95234588672,"internally_reserved":0,"allocated":229531648,"data_stored":229200311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":530905536},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514834,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":785888,"kb_used_data":141080,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":92959264,"statfs":{"total":95995035648,"available":95190286336,"internally_reserved":0,"allocated":144465920,"data_stored":144134583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":660273600},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874583,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":431968,"kb_used_data":57880,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":93313184,"statfs":{"total":95995035648,"available":95552700416,"internally_reserved":0,"allocated":59269120,"data_stored":58937783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":240518168923,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430560,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":595604,"kb_used_data":9100,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93149548,"statfs":{"total":95995035648,"available":95385137152,"internally_reserved":0,"allocated":9318400,"data_stored":8991578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":600570310},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575227,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":772004,"kb_used_data":127128,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":92973148,"statfs":{"total":95995035648,"available":95204503552,"internally_reserved":0,"allocated":130179072,"data_stored":129847735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738755,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":755044,"kb_used_data":138008,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":92990108,"statfs":{"total":95995035648,"available":95221870592,"internally_reserved":0,"allocated":141320192,"data_stored":140988855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411704,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":695976,"kb_used_data":281632,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93049176,"statfs":{"total":95995035648,"available":95282356224,"internally_reserved":0,"allocated":288391168,"data_stored":288056279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150065,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":798888,"kb_used_data":159648,"kb_used_omap":1,"kb_used_meta":639230,"kb_avail":92946264,"statfs":{"total":95995035648,"available":95176974336,"internally_reserved":0,"allocated":163479552,"data_stored":163144663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":654571975},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986541,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692200,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":827944,"kb_used_data":228960,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":92917208,"statfs":{"total":95995035648,"available":95147220992,"internally_reserved":0,"allocated":234455040,"data_stored":234120151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":148242432,"data_stored":148242432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":171704320,"data_stored":171704320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":330891264,"data_stored":330891264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":181207040,"data_stored":181207040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":242548736,"data_stored":242548736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":14286848,"data_stored":14286848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":71565312,"data_stored":71565312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":144703488,"data_stored":144703488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":270663680,"data_stored":270663680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":381353984,"data_stored":381353984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:12:56.612 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-05-15T13:12:56.613 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool set .mgr pg_num 21 2024-05-15T13:12:56.630 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:12:56.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:56.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:12:56.735 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:12:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:56.873 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 14 15 32362 32347 144.384 102.188 0.0082548 0.00691979 2024-05-15T13:12:56.874 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:12:56.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:56.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:12:56.987 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:12:57.016 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:57.036 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:12:57.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:12:57.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:12:57.140 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:12:57.200 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:12:57.345 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:12:57.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:57 smithi027 ceph-mon[39795]: pgmap v1608: 27 pgs: 27 active+clean; 1.6 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-05-15T13:12:57.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:57 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/963373588' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:12:57.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:57 smithi177 ceph-mon[44303]: pgmap v1608: 27 pgs: 27 active+clean; 1.6 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-05-15T13:12:57.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:57 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/963373588' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:12:57.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:57 smithi120 ceph-mon[43961]: pgmap v1608: 27 pgs: 27 active+clean; 1.6 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-05-15T13:12:57.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:57 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/963373588' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:12:57.873 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 15 16 34257 34241 142.65 118.375 0.00500711 0.00700558 2024-05-15T13:12:58.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:58 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:12:58.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:58 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/963373588' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:12:58.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:58 smithi177 ceph-mon[44303]: osdmap e611: 12 total, 12 up, 10 in 2024-05-15T13:12:58.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:58 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:12:58.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:58 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/963373588' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:12:58.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:58 smithi027 ceph-mon[39795]: osdmap e611: 12 total, 12 up, 10 in 2024-05-15T13:12:58.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:58 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:12:58.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:58 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/963373588' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:12:58.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:58 smithi120 ceph-mon[43961]: osdmap e611: 12 total, 12 up, 10 in 2024-05-15T13:12:58.873 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 16 16 35931 35915 140.272 104.625 0.00529646 0.00712504 2024-05-15T13:12:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:12:59.672 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:12:59 smithi027 ceph-mon[39795]: pgmap v1610: 27 pgs: 27 active+clean; 1.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-05-15T13:12:59.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:12:59 smithi177 ceph-mon[44303]: pgmap v1610: 27 pgs: 27 active+clean; 1.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-05-15T13:12:59.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:12:59 smithi120 ceph-mon[43961]: pgmap v1610: 27 pgs: 27 active+clean; 1.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-05-15T13:12:59.873 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 17 16 37870 37854 139.149 121.188 0.00646054 0.00718159 2024-05-15T13:13:00.115 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:13:00.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:00.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:13:00.229 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:13:00.278 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:00.372 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:13:00.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:00.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:13:00.479 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:13:00.506 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:00.525 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:13:00.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:00.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:13:00.630 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:13:00.658 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:00.678 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:13:00.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:00.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:13:00.783 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:13:00.819 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:13:00.824 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:00.855 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:13:00.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:00.859 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:13:00.874 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 18 16 39805 39789 138.136 120.938 0.0120596 0.00723375 2024-05-15T13:13:00.960 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:13:01.010 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:01.276 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:13:01.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:01.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:13:01.383 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:13:01.411 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:01.430 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:13:01.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:01.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:13:01.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:01 smithi027 ceph-mon[39795]: pgmap v1611: 27 pgs: 27 active+clean; 2.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 135 MiB/s wr, 2.15k op/s 2024-05-15T13:13:01.570 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:13:01.604 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:01 smithi177 ceph-mon[44303]: pgmap v1611: 27 pgs: 27 active+clean; 2.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 135 MiB/s wr, 2.15k op/s 2024-05-15T13:13:01.607 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:01.647 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:13:01.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:01.650 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:13:01.751 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:13:01.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:01 smithi120 ceph-mon[43961]: pgmap v1611: 27 pgs: 27 active+clean; 2.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 135 MiB/s wr, 2.15k op/s 2024-05-15T13:13:01.799 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:01.831 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:13:01.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:01.837 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:13:01.873 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 19 16 41586 41570 136.724 111.312 0.00426016 0.00730546 2024-05-15T13:13:01.938 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:13:01.988 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:02.017 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:13:02.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:02.022 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:13:02.123 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:13:02.149 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:13:02.180 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:02.218 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:13:02.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:02.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:13:02.325 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:13:02.372 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:02.400 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:13:02.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:02.407 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:13:02.508 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:13:02.554 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:02.874 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T13:13:02.881989+0000 min lat: 0.00113694 max lat: 0.603428 avg lat: 0.00739456 2024-05-15T13:13:02.874 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:13:02.874 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 20 16 43272 43256 135.156 105.375 0.00723127 0.00739456 2024-05-15T13:13:03.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:03 smithi177 ceph-mon[44303]: pgmap v1612: 27 pgs: 27 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-05-15T13:13:03.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:03 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:13:03.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:03 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:13:03.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:03 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:13:03.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:03 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:13:03.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:03 smithi027 ceph-mon[39795]: pgmap v1612: 27 pgs: 27 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-05-15T13:13:03.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:03 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:13:03.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:03 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:13:03.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:03 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:13:03.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:03 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:13:03.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:03 smithi120 ceph-mon[43961]: pgmap v1612: 27 pgs: 27 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-05-15T13:13:03.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:03 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:13:03.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:03 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:13:03.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:03 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:13:03.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:03 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:13:03.874 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 21 16 44502 44486 132.38 76.875 0.013099 0.00754917 2024-05-15T13:13:04.324 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:13:04.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:04.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:13:04.431 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:13:04.459 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:04.479 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:13:04.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:04.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:13:04.583 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:13:04.645 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:04.874 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 22 16 46216 46200 131.232 107.125 0.00955123 0.00761545 2024-05-15T13:13:04.914 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:04 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1981184693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:13:04.914 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:04 smithi027 ceph-mon[39795]: pgmap v1613: 27 pgs: 27 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-05-15T13:13:05.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:04 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1981184693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:13:05.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:04 smithi177 ceph-mon[44303]: pgmap v1613: 27 pgs: 27 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-05-15T13:13:05.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:04 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1981184693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:13:05.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:04 smithi120 ceph-mon[43961]: pgmap v1613: 27 pgs: 27 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-05-15T13:13:05.381 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:13:05.382 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:13:05.382 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:13:05.382 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:13:05.382 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:13:05.382 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:13:05.382 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:13:05.382 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:13:05.382 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:13:05.382 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:13:05.382 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:13:05.382 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:13:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:13:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:13:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:13:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:13:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:13:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:13:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:13:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:13:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:13:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:13:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:13:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:13:05.607 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:13:05.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:05.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:13:05.719 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:13:05.746 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:05.766 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:13:05.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:05.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:13:05.870 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:13:05.874 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 23 16 47844 47828 129.95 101.75 0.00279107 0.00766308 2024-05-15T13:13:05.896 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:05.915 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:13:05.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:05.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:13:06.019 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:13:06.065 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:06.093 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:13:06.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:06.099 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:13:06.200 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:13:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:06.850 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:13:06.850 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:13:06.874 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 24 16 49371 49355 128.51 95.4375 0.00932464 0.00777779 2024-05-15T13:13:07.047 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:13:07.061 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:13:07.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:07.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:13:07.166 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:13:07.195 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:07.216 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:13:07.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:07.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:13:07.320 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:13:07.384 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:07.436 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:13:07.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:07.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:13:07.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:07 smithi120 ceph-mon[43961]: pgmap v1614: 27 pgs: 27 active+clean; 2.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-05-15T13:13:07.542 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:13:07.591 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:07 smithi027 ceph-mon[39795]: pgmap v1614: 27 pgs: 27 active+clean; 2.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-05-15T13:13:07.593 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:07.685 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:07 smithi177 ceph-mon[44303]: pgmap v1614: 27 pgs: 27 active+clean; 2.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-05-15T13:13:07.874 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 25 16 51004 50988 127.452 102.062 0.0150655 0.00784105 2024-05-15T13:13:08.355 INFO:teuthology.orchestra.run.smithi027.stderr:set pool 1 pg_num to 21 2024-05-15T13:13:08.464 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:13:08.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:08.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:13:08.572 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:13:08.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:08 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3244086785' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-05-15T13:13:08.618 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:08 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-05-15T13:13:08.626 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:08.661 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:13:08.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:08.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:13:08.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:08 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3244086785' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-05-15T13:13:08.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:08 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-05-15T13:13:08.771 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:13:08.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:08 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3244086785' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-05-15T13:13:08.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:08 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-05-15T13:13:08.799 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:08.819 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:13:08.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:08.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:13:08.876 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 26 16 52601 52585 126.38 99.8125 0.00560609 0.00790754 2024-05-15T13:13:08.922 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:13:08.969 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:09.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:09 smithi177 ceph-mon[44303]: pgmap v1615: 27 pgs: 27 active+clean; 2.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-05-15T13:13:09.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:09 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-05-15T13:13:09.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:09 smithi177 ceph-mon[44303]: osdmap e612: 12 total, 12 up, 10 in 2024-05-15T13:13:09.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:09 smithi177 ceph-mon[44303]: osdmap e613: 12 total, 12 up, 10 in 2024-05-15T13:13:09.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:09 smithi027 ceph-mon[39795]: pgmap v1615: 27 pgs: 27 active+clean; 2.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-05-15T13:13:09.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:09 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-05-15T13:13:09.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:09 smithi027 ceph-mon[39795]: osdmap e612: 12 total, 12 up, 10 in 2024-05-15T13:13:09.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:09 smithi027 ceph-mon[39795]: osdmap e613: 12 total, 12 up, 10 in 2024-05-15T13:13:09.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:09 smithi120 ceph-mon[43961]: pgmap v1615: 27 pgs: 27 active+clean; 2.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-05-15T13:13:09.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:09 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-05-15T13:13:09.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:09 smithi120 ceph-mon[43961]: osdmap e612: 12 total, 12 up, 10 in 2024-05-15T13:13:09.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:09 smithi120 ceph-mon[43961]: osdmap e613: 12 total, 12 up, 10 in 2024-05-15T13:13:09.876 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 27 16 54276 54260 125.576 104.688 0.0137951 0.00795918 2024-05-15T13:13:10.031 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:13:10.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:10.047 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:13:10.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:13:10.149 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:13:10.210 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:10.349 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:10 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-05-15T13:13:10.361 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:13:10.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:10.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:13:10.468 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:13:10.516 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:10.554 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:13:10.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:10.559 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:13:10.660 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:13:10.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:10 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-05-15T13:13:10.716 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:10.742 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:13:10.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:10.748 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:13:10.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:10 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-05-15T13:13:10.848 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:13:10.876 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:10.876 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 28 16 55681 55665 124.227 87.8125 0.0029193 0.00802556 2024-05-15T13:13:11.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:11 smithi177 ceph-mon[44303]: pgmap v1618: 27 pgs: 27 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 107 MiB/s wr, 1.72k op/s 2024-05-15T13:13:11.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:11 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-05-15T13:13:11.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:11 smithi177 ceph-mon[44303]: osdmap e614: 12 total, 12 up, 10 in 2024-05-15T13:13:11.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:11 smithi027 ceph-mon[39795]: pgmap v1618: 27 pgs: 27 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 107 MiB/s wr, 1.72k op/s 2024-05-15T13:13:11.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:11 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-05-15T13:13:11.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:11 smithi027 ceph-mon[39795]: osdmap e614: 12 total, 12 up, 10 in 2024-05-15T13:13:11.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:11 smithi120 ceph-mon[43961]: pgmap v1618: 27 pgs: 27 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 107 MiB/s wr, 1.72k op/s 2024-05-15T13:13:11.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:11 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-05-15T13:13:11.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:11 smithi120 ceph-mon[43961]: osdmap e614: 12 total, 12 up, 10 in 2024-05-15T13:13:11.877 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 29 16 57135 57119 123.077 90.875 0.00496057 0.008121 2024-05-15T13:13:12.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:12 smithi177 ceph-mon[44303]: osdmap e615: 12 total, 12 up, 10 in 2024-05-15T13:13:12.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:12 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-05-15T13:13:12.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:12 smithi027 ceph-mon[39795]: osdmap e615: 12 total, 12 up, 10 in 2024-05-15T13:13:12.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:12 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-05-15T13:13:12.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:12 smithi120 ceph-mon[43961]: osdmap e615: 12 total, 12 up, 10 in 2024-05-15T13:13:12.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:12 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-05-15T13:13:12.877 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 30 16 58713 58697 122.262 98.625 0.0171824 0.00817432 2024-05-15T13:13:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:13:13.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:13.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:13:13.140 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:13:13.202 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:13.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:13 smithi027 ceph-mon[39795]: pgmap v1621: 37 pgs: 1 peering, 7 unknown, 29 active+clean; 3.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-05-15T13:13:13.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:13 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-05-15T13:13:13.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:13 smithi027 ceph-mon[39795]: osdmap e616: 12 total, 12 up, 10 in 2024-05-15T13:13:13.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:13 smithi177 ceph-mon[44303]: pgmap v1621: 37 pgs: 1 peering, 7 unknown, 29 active+clean; 3.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-05-15T13:13:13.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:13 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-05-15T13:13:13.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:13 smithi177 ceph-mon[44303]: osdmap e616: 12 total, 12 up, 10 in 2024-05-15T13:13:13.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:13 smithi120 ceph-mon[43961]: pgmap v1621: 37 pgs: 1 peering, 7 unknown, 29 active+clean; 3.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-05-15T13:13:13.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:13 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-05-15T13:13:13.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:13 smithi120 ceph-mon[43961]: osdmap e616: 12 total, 12 up, 10 in 2024-05-15T13:13:13.877 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 31 16 60242 60226 121.4 95.5625 0.00716365 0.00823263 2024-05-15T13:13:14.316 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 8, 9, 10, 11] out_osds: [7, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:13:14.316 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 0 1 2024-05-15T13:13:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:13:14.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:14.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:13:14.599 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:13:14.626 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:14.645 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:13:14.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:14.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:13:14.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:14 smithi177 ceph-mon[44303]: osdmap e617: 12 total, 12 up, 10 in 2024-05-15T13:13:14.726 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:14 smithi027 ceph-mon[39795]: osdmap e617: 12 total, 12 up, 10 in 2024-05-15T13:13:14.743 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-05-15T13:13:14.751 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:13:14.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:14 smithi120 ceph-mon[43961]: osdmap e617: 12 total, 12 up, 10 in 2024-05-15T13:13:14.877 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 32 16 61535 61519 120.132 80.8125 0.0130961 0.00831898 2024-05-15T13:13:14.878 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:14.969 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:13:14.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:14.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:13:15.008 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:13:15.066 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:13:15.074 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:13:15.101 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:15.119 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:13:15.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:15.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:13:15.223 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:13:15.270 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:15.438 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:13:15.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:15 smithi027 ceph-mon[39795]: pgmap v1624: 37 pgs: 3 remapped+peering, 1 peering, 5 unknown, 28 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 1.7 KiB/s rd, 84 MiB/s wr, 1.35k op/s; 63 KiB/s, 0 objects/s recovering 2024-05-15T13:13:15.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:15 smithi027 ceph-mon[39795]: osdmap e618: 12 total, 12 up, 10 in 2024-05-15T13:13:15.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:15 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3155839393' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:13:15.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:15 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:13:15.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:15 smithi027 ceph-mon[39795]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-05-15T13:13:15.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:15 smithi177 ceph-mon[44303]: pgmap v1624: 37 pgs: 3 remapped+peering, 1 peering, 5 unknown, 28 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 1.7 KiB/s rd, 84 MiB/s wr, 1.35k op/s; 63 KiB/s, 0 objects/s recovering 2024-05-15T13:13:15.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:15 smithi177 ceph-mon[44303]: osdmap e618: 12 total, 12 up, 10 in 2024-05-15T13:13:15.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:15 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3155839393' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:13:15.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:15 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:13:15.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:15 smithi177 ceph-mon[44303]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-05-15T13:13:15.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:15 smithi120 ceph-mon[43961]: pgmap v1624: 37 pgs: 3 remapped+peering, 1 peering, 5 unknown, 28 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 1.7 KiB/s rd, 84 MiB/s wr, 1.35k op/s; 63 KiB/s, 0 objects/s recovering 2024-05-15T13:13:15.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:15 smithi120 ceph-mon[43961]: osdmap e618: 12 total, 12 up, 10 in 2024-05-15T13:13:15.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:15 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3155839393' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:13:15.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:15 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:13:15.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:15 smithi120 ceph-mon[43961]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-05-15T13:13:15.877 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 33 16 62840 62824 118.963 81.5625 0.00914136 0.00840185 2024-05-15T13:13:16.038 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:13:16.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:16.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:13:16.144 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:13:16.194 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:16.233 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:13:16.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:16.239 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:13:16.340 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:13:16.397 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:16.442 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:13:16.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:16.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:13:16.551 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:13:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:16.613 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:13:16.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:16.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:13:16.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:16 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:13:16.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:16 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:13:16.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:16 smithi177 ceph-mon[44303]: osdmap e619: 12 total, 12 up, 10 in 2024-05-15T13:13:16.724 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:13:16.766 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:16.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:16 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:13:16.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:16 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:13:16.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:16 smithi027 ceph-mon[39795]: osdmap e619: 12 total, 12 up, 10 in 2024-05-15T13:13:16.783 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:16 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:13:16.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:16 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:13:16.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:16 smithi120 ceph-mon[43961]: osdmap e619: 12 total, 12 up, 10 in 2024-05-15T13:13:16.795 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:13:16.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:16.800 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:13:16.877 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 34 16 64056 64040 117.699 76 0.00772257 0.00849217 2024-05-15T13:13:16.901 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:13:16.927 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:16.948 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:13:16.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:16.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:13:17.052 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:13:17.119 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:17.158 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:13:17.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:17.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:13:17.262 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:13:17.297 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:17.351 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:13:17.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:17.359 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:13:17.460 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:13:17.470 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:17 smithi027 ceph-mon[39795]: pgmap v1627: 37 pgs: 3 active+undersized+remapped, 3 remapped+peering, 2 unknown, 29 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 3.0 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 104 KiB/s, 0 objects/s recovering 2024-05-15T13:13:17.497 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:17.528 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:13:17.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:17.536 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:13:17.637 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:13:17.664 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:17.684 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:13:17.685 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:17 smithi177 ceph-mon[44303]: pgmap v1627: 37 pgs: 3 active+undersized+remapped, 3 remapped+peering, 2 unknown, 29 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 3.0 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 104 KiB/s, 0 objects/s recovering 2024-05-15T13:13:17.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:17 smithi177 ceph-mon[44303]: osdmap e620: 12 total, 12 up, 10 in 2024-05-15T13:13:17.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:17 smithi177 ceph-mon[44303]: 1.f scrub starts 2024-05-15T13:13:17.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:17 smithi177 ceph-mon[44303]: 1.f scrub ok 2024-05-15T13:13:17.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:17 smithi177 ceph-mon[44303]: 1.11 scrub starts 2024-05-15T13:13:17.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:17 smithi177 ceph-mon[44303]: 1.c scrub starts 2024-05-15T13:13:17.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:17 smithi177 ceph-mon[44303]: 1.10 scrub starts 2024-05-15T13:13:17.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:17 smithi177 ceph-mon[44303]: 1.10 scrub ok 2024-05-15T13:13:17.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:17 smithi177 ceph-mon[44303]: 1.c scrub ok 2024-05-15T13:13:17.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:17.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:13:17.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:17 smithi027 ceph-mon[39795]: osdmap e620: 12 total, 12 up, 10 in 2024-05-15T13:13:17.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:17 smithi027 ceph-mon[39795]: 1.f scrub starts 2024-05-15T13:13:17.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:17 smithi027 ceph-mon[39795]: 1.f scrub ok 2024-05-15T13:13:17.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:17 smithi027 ceph-mon[39795]: 1.11 scrub starts 2024-05-15T13:13:17.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:17 smithi027 ceph-mon[39795]: 1.c scrub starts 2024-05-15T13:13:17.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:17 smithi027 ceph-mon[39795]: 1.10 scrub starts 2024-05-15T13:13:17.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:17 smithi027 ceph-mon[39795]: 1.10 scrub ok 2024-05-15T13:13:17.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:17 smithi027 ceph-mon[39795]: 1.c scrub ok 2024-05-15T13:13:17.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:17 smithi120 ceph-mon[43961]: pgmap v1627: 37 pgs: 3 active+undersized+remapped, 3 remapped+peering, 2 unknown, 29 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 3.0 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 104 KiB/s, 0 objects/s recovering 2024-05-15T13:13:17.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:17 smithi120 ceph-mon[43961]: osdmap e620: 12 total, 12 up, 10 in 2024-05-15T13:13:17.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:17 smithi120 ceph-mon[43961]: 1.f scrub starts 2024-05-15T13:13:17.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:17 smithi120 ceph-mon[43961]: 1.f scrub ok 2024-05-15T13:13:17.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:17 smithi120 ceph-mon[43961]: 1.11 scrub starts 2024-05-15T13:13:17.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:17 smithi120 ceph-mon[43961]: 1.c scrub starts 2024-05-15T13:13:17.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:17 smithi120 ceph-mon[43961]: 1.10 scrub starts 2024-05-15T13:13:17.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:17 smithi120 ceph-mon[43961]: 1.10 scrub ok 2024-05-15T13:13:17.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:17 smithi120 ceph-mon[43961]: 1.c scrub ok 2024-05-15T13:13:17.788 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:13:17.815 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:17.837 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:13:17.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:17.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:13:17.877 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 35 15 65238 65223 116.449 73.9375 0.0147453 0.00858302 2024-05-15T13:13:17.941 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:13:17.999 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:18.036 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:13:18.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:18.041 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:13:18.142 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:13:18.213 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:18.491 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:18 smithi027 ceph-mon[39795]: 1.b scrub starts 2024-05-15T13:13:18.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:18 smithi027 ceph-mon[39795]: 1.b scrub ok 2024-05-15T13:13:18.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:18 smithi027 ceph-mon[39795]: 1.11 scrub ok 2024-05-15T13:13:18.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:18 smithi027 ceph-mon[39795]: 1.d scrub starts 2024-05-15T13:13:18.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:18 smithi027 ceph-mon[39795]: 1.14 scrub starts 2024-05-15T13:13:18.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:18 smithi027 ceph-mon[39795]: 1.d scrub ok 2024-05-15T13:13:18.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:18 smithi027 ceph-mon[39795]: 1.14 scrub ok 2024-05-15T13:13:18.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:18 smithi027 ceph-mon[39795]: 1.7 scrub starts 2024-05-15T13:13:18.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:18 smithi027 ceph-mon[39795]: 1.6 scrub starts 2024-05-15T13:13:18.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:18 smithi027 ceph-mon[39795]: 1.7 scrub ok 2024-05-15T13:13:18.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:18 smithi027 ceph-mon[39795]: 1.6 scrub ok 2024-05-15T13:13:18.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:18 smithi027 ceph-mon[39795]: 1.e scrub starts 2024-05-15T13:13:18.493 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:18 smithi027 ceph-mon[39795]: 1.12 scrub starts 2024-05-15T13:13:18.493 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:18 smithi027 ceph-mon[39795]: 1.12 scrub ok 2024-05-15T13:13:18.493 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:18 smithi027 ceph-mon[39795]: 1.e scrub ok 2024-05-15T13:13:18.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:18 smithi120 ceph-mon[43961]: 1.b scrub starts 2024-05-15T13:13:18.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:18 smithi120 ceph-mon[43961]: 1.b scrub ok 2024-05-15T13:13:18.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:18 smithi120 ceph-mon[43961]: 1.11 scrub ok 2024-05-15T13:13:18.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:18 smithi120 ceph-mon[43961]: 1.d scrub starts 2024-05-15T13:13:18.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:18 smithi120 ceph-mon[43961]: 1.14 scrub starts 2024-05-15T13:13:18.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:18 smithi120 ceph-mon[43961]: 1.d scrub ok 2024-05-15T13:13:18.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:18 smithi120 ceph-mon[43961]: 1.14 scrub ok 2024-05-15T13:13:18.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:18 smithi120 ceph-mon[43961]: 1.7 scrub starts 2024-05-15T13:13:18.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:18 smithi120 ceph-mon[43961]: 1.6 scrub starts 2024-05-15T13:13:18.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:18 smithi120 ceph-mon[43961]: 1.7 scrub ok 2024-05-15T13:13:18.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:18 smithi120 ceph-mon[43961]: 1.6 scrub ok 2024-05-15T13:13:18.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:18 smithi120 ceph-mon[43961]: 1.e scrub starts 2024-05-15T13:13:18.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:18 smithi120 ceph-mon[43961]: 1.12 scrub starts 2024-05-15T13:13:18.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:18 smithi120 ceph-mon[43961]: 1.12 scrub ok 2024-05-15T13:13:18.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:18 smithi120 ceph-mon[43961]: 1.e scrub ok 2024-05-15T13:13:18.878 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 36 16 66488 66472 115.383 78.0625 0.0102527 0.00866239 2024-05-15T13:13:18.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:18 smithi177 ceph-mon[44303]: 1.b scrub starts 2024-05-15T13:13:18.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:18 smithi177 ceph-mon[44303]: 1.b scrub ok 2024-05-15T13:13:18.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:18 smithi177 ceph-mon[44303]: 1.11 scrub ok 2024-05-15T13:13:18.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:18 smithi177 ceph-mon[44303]: 1.d scrub starts 2024-05-15T13:13:18.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:18 smithi177 ceph-mon[44303]: 1.14 scrub starts 2024-05-15T13:13:18.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:18 smithi177 ceph-mon[44303]: 1.d scrub ok 2024-05-15T13:13:18.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:18 smithi177 ceph-mon[44303]: 1.14 scrub ok 2024-05-15T13:13:18.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:18 smithi177 ceph-mon[44303]: 1.7 scrub starts 2024-05-15T13:13:18.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:18 smithi177 ceph-mon[44303]: 1.6 scrub starts 2024-05-15T13:13:18.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:18 smithi177 ceph-mon[44303]: 1.7 scrub ok 2024-05-15T13:13:18.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:18 smithi177 ceph-mon[44303]: 1.6 scrub ok 2024-05-15T13:13:18.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:18 smithi177 ceph-mon[44303]: 1.e scrub starts 2024-05-15T13:13:18.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:18 smithi177 ceph-mon[44303]: 1.12 scrub starts 2024-05-15T13:13:18.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:18 smithi177 ceph-mon[44303]: 1.12 scrub ok 2024-05-15T13:13:18.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:18 smithi177 ceph-mon[44303]: 1.e scrub ok 2024-05-15T13:13:19.478 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:13:19.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:19.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:13:19.585 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:13:19.629 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:19.657 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:13:19.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:19.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:13:19.763 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:13:19.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:19 smithi027 ceph-mon[39795]: 1.4 scrub starts 2024-05-15T13:13:19.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:19 smithi027 ceph-mon[39795]: 1.4 scrub ok 2024-05-15T13:13:19.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:19 smithi027 ceph-mon[39795]: 1.5 scrub starts 2024-05-15T13:13:19.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:19 smithi027 ceph-mon[39795]: 1.5 scrub ok 2024-05-15T13:13:19.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:19 smithi027 ceph-mon[39795]: 1.3 scrub starts 2024-05-15T13:13:19.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:19 smithi027 ceph-mon[39795]: pgmap v1629: 37 pgs: 2 active+undersized+remapped, 1 remapped+peering, 1 unknown, 33 active+clean; 3.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 2.9 KiB/s rd, 79 MiB/s wr, 1.26k op/s; 93 KiB/s, 0 objects/s recovering 2024-05-15T13:13:19.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:19 smithi027 ceph-mon[39795]: 1.3 scrub ok 2024-05-15T13:13:19.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:19 smithi027 ceph-mon[39795]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-05-15T13:13:19.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:19 smithi027 ceph-mon[39795]: 1.2 scrub starts 2024-05-15T13:13:19.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:19 smithi027 ceph-mon[39795]: osdmap e621: 12 total, 12 up, 10 in 2024-05-15T13:13:19.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:19 smithi027 ceph-mon[39795]: 1.2 scrub ok 2024-05-15T13:13:19.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:19 smithi120 ceph-mon[43961]: 1.4 scrub starts 2024-05-15T13:13:19.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:19 smithi120 ceph-mon[43961]: 1.4 scrub ok 2024-05-15T13:13:19.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:19 smithi120 ceph-mon[43961]: 1.5 scrub starts 2024-05-15T13:13:19.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:19 smithi120 ceph-mon[43961]: 1.5 scrub ok 2024-05-15T13:13:19.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:19 smithi120 ceph-mon[43961]: 1.3 scrub starts 2024-05-15T13:13:19.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:19 smithi120 ceph-mon[43961]: pgmap v1629: 37 pgs: 2 active+undersized+remapped, 1 remapped+peering, 1 unknown, 33 active+clean; 3.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 2.9 KiB/s rd, 79 MiB/s wr, 1.26k op/s; 93 KiB/s, 0 objects/s recovering 2024-05-15T13:13:19.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:19 smithi120 ceph-mon[43961]: 1.3 scrub ok 2024-05-15T13:13:19.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:19 smithi120 ceph-mon[43961]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-05-15T13:13:19.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:19 smithi120 ceph-mon[43961]: 1.2 scrub starts 2024-05-15T13:13:19.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:19 smithi120 ceph-mon[43961]: osdmap e621: 12 total, 12 up, 10 in 2024-05-15T13:13:19.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:19 smithi120 ceph-mon[43961]: 1.2 scrub ok 2024-05-15T13:13:19.791 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:19.811 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:13:19.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:19.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:13:19.877 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 37 16 67806 67790 114.49 82.375 0.00818072 0.00872925 2024-05-15T13:13:19.916 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:13:19.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:19 smithi177 ceph-mon[44303]: 1.4 scrub starts 2024-05-15T13:13:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:19 smithi177 ceph-mon[44303]: 1.4 scrub ok 2024-05-15T13:13:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:19 smithi177 ceph-mon[44303]: 1.5 scrub starts 2024-05-15T13:13:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:19 smithi177 ceph-mon[44303]: 1.5 scrub ok 2024-05-15T13:13:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:19 smithi177 ceph-mon[44303]: 1.3 scrub starts 2024-05-15T13:13:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:19 smithi177 ceph-mon[44303]: pgmap v1629: 37 pgs: 2 active+undersized+remapped, 1 remapped+peering, 1 unknown, 33 active+clean; 3.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 2.9 KiB/s rd, 79 MiB/s wr, 1.26k op/s; 93 KiB/s, 0 objects/s recovering 2024-05-15T13:13:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:19 smithi177 ceph-mon[44303]: 1.3 scrub ok 2024-05-15T13:13:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:19 smithi177 ceph-mon[44303]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-05-15T13:13:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:19 smithi177 ceph-mon[44303]: 1.2 scrub starts 2024-05-15T13:13:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:19 smithi177 ceph-mon[44303]: osdmap e621: 12 total, 12 up, 10 in 2024-05-15T13:13:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:19 smithi177 ceph-mon[44303]: 1.2 scrub ok 2024-05-15T13:13:19.942 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:19.962 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:13:19.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:19.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:13:20.065 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:13:20.104 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:13:20.116 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:20.145 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:13:20.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:20.148 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:13:20.249 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:13:20.281 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:20.308 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:13:20.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:20.311 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:13:20.412 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:13:20.440 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:20.459 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:13:20.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:20.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:13:20.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:20 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T13:13:20.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:20 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T13:13:20.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:20 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:13:20.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:20 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:13:20.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:20 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:13:20.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:20 smithi027 ceph-mon[39795]: osdmap e622: 12 total, 12 up, 10 in 2024-05-15T13:13:20.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:20 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:13:20.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:20 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:13:20.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:20 smithi027 ceph-mon[39795]: 1.1 scrub starts 2024-05-15T13:13:20.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:20 smithi027 ceph-mon[39795]: 1.1 scrub ok 2024-05-15T13:13:20.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:20 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3819203650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:13:20.506 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:20 smithi027 ceph-mon[39795]: pgmap v1632: 37 pgs: 37 active+clean; 4.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 423 B/s rd, 85 MiB/s wr, 1.36k op/s; 8.1 KiB/s, 0 objects/s recovering 2024-05-15T13:13:20.563 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:13:20.609 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:20.641 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:13:20.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:20.645 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:13:20.746 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:13:20.771 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:20 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T13:13:20.771 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:20 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T13:13:20.771 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:20 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:13:20.771 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:20 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:13:20.771 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:20 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:13:20.771 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:20 smithi120 ceph-mon[43961]: osdmap e622: 12 total, 12 up, 10 in 2024-05-15T13:13:20.771 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:20 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:13:20.771 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:20 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:13:20.772 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:20 smithi120 ceph-mon[43961]: 1.1 scrub starts 2024-05-15T13:13:20.772 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:20 smithi120 ceph-mon[43961]: 1.1 scrub ok 2024-05-15T13:13:20.772 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:20 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3819203650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:13:20.772 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:20 smithi120 ceph-mon[43961]: pgmap v1632: 37 pgs: 37 active+clean; 4.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 423 B/s rd, 85 MiB/s wr, 1.36k op/s; 8.1 KiB/s, 0 objects/s recovering 2024-05-15T13:13:20.772 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:20.794 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:13:20.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:20.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:13:20.880 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 38 16 69005 68989 113.444 74.9375 0.0101191 0.00881056 2024-05-15T13:13:20.897 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:13:20.925 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:20.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:20 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T13:13:20.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:20 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T13:13:20.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:20 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:13:20.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:20 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:13:20.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:20 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:13:20.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:20 smithi177 ceph-mon[44303]: osdmap e622: 12 total, 12 up, 10 in 2024-05-15T13:13:20.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:20 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:13:20.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:20 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:13:20.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:20 smithi177 ceph-mon[44303]: 1.1 scrub starts 2024-05-15T13:13:20.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:20 smithi177 ceph-mon[44303]: 1.1 scrub ok 2024-05-15T13:13:20.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:20 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3819203650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:13:20.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:20 smithi177 ceph-mon[44303]: pgmap v1632: 37 pgs: 37 active+clean; 4.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 423 B/s rd, 85 MiB/s wr, 1.36k op/s; 8.1 KiB/s, 0 objects/s recovering 2024-05-15T13:13:20.943 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:13:20.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:20.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:13:21.047 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:13:21.097 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:21.124 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:13:21.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:21.128 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:13:21.229 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:13:21.287 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:21.880 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 39 16 70327 70311 112.654 82.625 0.0132342 0.00887235 2024-05-15T13:13:22.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:21 smithi177 ceph-mon[44303]: 1.13 scrub starts 2024-05-15T13:13:22.183 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:21 smithi177 ceph-mon[44303]: 1.13 scrub ok 2024-05-15T13:13:22.183 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:21 smithi177 ceph-mon[44303]: 1.9 scrub starts 2024-05-15T13:13:22.183 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:21 smithi177 ceph-mon[44303]: 1.8 scrub starts 2024-05-15T13:13:22.183 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:21 smithi177 ceph-mon[44303]: 1.8 scrub ok 2024-05-15T13:13:22.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:21 smithi027 ceph-mon[39795]: 1.13 scrub starts 2024-05-15T13:13:22.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:21 smithi027 ceph-mon[39795]: 1.13 scrub ok 2024-05-15T13:13:22.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:21 smithi027 ceph-mon[39795]: 1.9 scrub starts 2024-05-15T13:13:22.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:21 smithi027 ceph-mon[39795]: 1.8 scrub starts 2024-05-15T13:13:22.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:21 smithi027 ceph-mon[39795]: 1.8 scrub ok 2024-05-15T13:13:22.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:21 smithi120 ceph-mon[43961]: 1.13 scrub starts 2024-05-15T13:13:22.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:21 smithi120 ceph-mon[43961]: 1.13 scrub ok 2024-05-15T13:13:22.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:21 smithi120 ceph-mon[43961]: 1.9 scrub starts 2024-05-15T13:13:22.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:21 smithi120 ceph-mon[43961]: 1.8 scrub starts 2024-05-15T13:13:22.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:21 smithi120 ceph-mon[43961]: 1.8 scrub ok 2024-05-15T13:13:22.865 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:13:22.865 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:13:22.865 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:13:22.865 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:13:22.865 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:13:22.865 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:13:22.865 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:13:22.865 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:13:22.866 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:13:22.866 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:13:22.866 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:13:22.866 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:13:22.866 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:13:22.866 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:13:22.866 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:13:22.866 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:13:22.866 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:13:22.866 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:13:22.867 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:13:22.867 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:13:22.867 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:13:22.867 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:13:22.867 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:13:22.867 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:13:22.883 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T13:13:22.889584+0000 min lat: 0.00113694 max lat: 0.603428 avg lat: 0.00892688 2024-05-15T13:13:22.883 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:13:22.883 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 40 16 71695 71679 111.969 85.5 0.0110577 0.00892688 2024-05-15T13:13:23.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:22 smithi177 ceph-mon[44303]: 1.9 scrub ok 2024-05-15T13:13:23.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:22 smithi177 ceph-mon[44303]: pgmap v1633: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-05-15T13:13:23.273 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:22 smithi027 ceph-mon[39795]: 1.9 scrub ok 2024-05-15T13:13:23.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:22 smithi027 ceph-mon[39795]: pgmap v1633: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-05-15T13:13:23.283 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:22 smithi120 ceph-mon[43961]: 1.9 scrub ok 2024-05-15T13:13:23.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:22 smithi120 ceph-mon[43961]: pgmap v1633: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-05-15T13:13:23.356 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:13:23.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:23.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:13:23.372 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:13:23.465 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:13:23.519 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:23.791 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:13:23.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:23.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:13:23.883 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 41 16 72951 72935 111.15 78.5 0.0126883 0.00899272 2024-05-15T13:13:23.898 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:13:23.950 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:24.424 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:13:24.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:24.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:13:24.529 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:13:24.557 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:24.577 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:13:24.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:24.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:13:24.681 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:13:24.708 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:24.727 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:13:24.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:24.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:13:24.831 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:13:24.875 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:24.883 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 42 16 74220 74204 110.392 79.3125 0.00716978 0.00905346 2024-05-15T13:13:24.923 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:13:24.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:24.929 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:13:25.000 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:24 smithi027 ceph-mon[39795]: osdmap e623: 12 total, 12 up, 10 in 2024-05-15T13:13:25.001 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:24 smithi027 ceph-mon[39795]: pgmap v1635: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-05-15T13:13:25.001 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:24 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:13:25.001 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:24 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:13:25.001 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:24 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:13:25.001 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:24 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:13:25.001 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:24 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:13:25.001 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:24 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:13:25.030 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:13:25.107 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:25.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:24 smithi177 ceph-mon[44303]: osdmap e623: 12 total, 12 up, 10 in 2024-05-15T13:13:25.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:24 smithi177 ceph-mon[44303]: pgmap v1635: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-05-15T13:13:25.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:24 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:13:25.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:24 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:13:25.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:24 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:13:25.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:24 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:13:25.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:24 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:13:25.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:24 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:13:25.221 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:13:25.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:25.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:13:25.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:24 smithi120 ceph-mon[43961]: osdmap e623: 12 total, 12 up, 10 in 2024-05-15T13:13:25.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:24 smithi120 ceph-mon[43961]: pgmap v1635: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-05-15T13:13:25.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:24 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:13:25.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:24 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:13:25.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:13:25.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:13:25.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:24 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:13:25.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:24 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:13:25.330 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:13:25.379 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:25.883 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 43 16 75323 75307 109.428 68.9375 0.00700838 0.00913441 2024-05-15T13:13:25.922 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.0 to 1 (10000) 2024-05-15T13:13:26.088 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:13:26.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:25 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/487907436' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:13:26.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:25 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:13:26.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:25 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/487907436' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:13:26.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:25 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:13:26.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:25 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/487907436' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:13:26.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:25 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:13:26.883 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 44 16 76564 76548 108.704 77.5625 0.0113228 0.00919485 2024-05-15T13:13:27.146 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:13:27.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:27.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:13:27.182 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:26 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T13:13:27.183 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:26 smithi177 ceph-mon[44303]: osdmap e624: 12 total, 12 up, 10 in 2024-05-15T13:13:27.183 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:26 smithi177 ceph-mon[44303]: pgmap v1637: 37 pgs: 37 active+clean; 4.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-05-15T13:13:27.252 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:13:27.263 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:26 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T13:13:27.263 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:26 smithi027 ceph-mon[39795]: osdmap e624: 12 total, 12 up, 10 in 2024-05-15T13:13:27.263 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:26 smithi027 ceph-mon[39795]: pgmap v1637: 37 pgs: 37 active+clean; 4.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-05-15T13:13:27.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:26 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T13:13:27.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:26 smithi120 ceph-mon[43961]: osdmap e624: 12 total, 12 up, 10 in 2024-05-15T13:13:27.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:26 smithi120 ceph-mon[43961]: pgmap v1637: 37 pgs: 37 active+clean; 4.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-05-15T13:13:27.301 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:27.701 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:13:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:13:27.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:27.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:13:27.767 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 1 1 2024-05-15T13:13:27.816 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:13:27.844 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:27.867 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:13:27.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:27.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:13:27.883 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 45 15 77750 77735 107.937 74.1875 0.00474069 0.00926033 2024-05-15T13:13:27.971 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:13:27.997 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:28.018 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:13:28.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:28.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:13:28.024 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:27 smithi027 ceph-mon[39795]: 1.a scrub starts 2024-05-15T13:13:28.024 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:27 smithi027 ceph-mon[39795]: 1.a scrub ok 2024-05-15T13:13:28.123 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:13:28.168 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:28.195 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:13:28.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:28.200 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:13:28.200 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:27 smithi177 ceph-mon[44303]: 1.a scrub starts 2024-05-15T13:13:28.200 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:27 smithi177 ceph-mon[44303]: 1.a scrub ok 2024-05-15T13:13:28.243 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:13:28.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:27 smithi120 ceph-mon[43961]: 1.a scrub starts 2024-05-15T13:13:28.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:27 smithi120 ceph-mon[43961]: 1.a scrub ok 2024-05-15T13:13:28.300 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:13:28.392 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:28.436 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:13:28.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:28.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:13:28.541 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:13:28.578 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:28.611 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:13:28.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:28.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:13:28.716 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:13:28.744 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:28.766 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:13:28.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:28.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:13:28.870 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:13:28.888 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 46 16 78891 78875 107.138 71.25 0.00791797 0.00932975 2024-05-15T13:13:28.916 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:28.948 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:13:28.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:28.952 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:13:28.953 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:28 smithi027 ceph-mon[39795]: pgmap v1638: 37 pgs: 37 active+clean; 4.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-05-15T13:13:28.953 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:28 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:13:28.953 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:28 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:13:28.953 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:28 smithi027 ceph-mon[39795]: osdmap e625: 12 total, 12 up, 10 in 2024-05-15T13:13:29.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:28 smithi120 ceph-mon[43961]: pgmap v1638: 37 pgs: 37 active+clean; 4.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-05-15T13:13:29.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:28 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:13:29.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:28 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:13:29.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:28 smithi120 ceph-mon[43961]: osdmap e625: 12 total, 12 up, 10 in 2024-05-15T13:13:29.053 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:13:29.097 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:29.133 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:13:29.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:29.140 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:13:29.183 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:28 smithi177 ceph-mon[44303]: pgmap v1638: 37 pgs: 37 active+clean; 4.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-05-15T13:13:29.183 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:28 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:13:29.183 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:28 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:13:29.183 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:28 smithi177 ceph-mon[44303]: osdmap e625: 12 total, 12 up, 10 in 2024-05-15T13:13:29.242 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:13:29.268 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:29.291 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:13:29.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:29.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:13:29.396 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:13:29.453 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:29.886 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 47 16 80112 80096 106.482 76.3125 0.0195184 0.00938705 2024-05-15T13:13:30.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:30 smithi027 ceph-mon[39795]: 7.9 scrub starts 2024-05-15T13:13:30.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:30 smithi120 ceph-mon[43961]: 7.9 scrub starts 2024-05-15T13:13:30.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:30 smithi177 ceph-mon[44303]: 7.9 scrub starts 2024-05-15T13:13:30.884 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 48 15 81150 81135 105.617 64.9375 0.0260914 0.0094637 2024-05-15T13:13:31.325 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:13:31.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:31.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:13:31.434 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:13:31.477 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:13:31.478 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T13:13:31.478 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T13:13:31.478 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:13:31.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.478 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:190881 6.4 6:24cd1d36:::benchmark_data_smithi177_122278_object60510:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-05-15T13:13:31.478 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:10:34.160711+0000", 2024-05-15T13:13:31.478 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 177.14659305000001, 2024-05-15T13:13:31.478 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.122674115, 2024-05-15T13:13:31.478 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:13:31.478 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:13:31.478 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:13:31.478 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:13:31.479 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:13:31.479 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 190881 2024-05-15T13:13:31.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.479 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:13:31.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.479 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:13:31.479 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.160711+0000", 2024-05-15T13:13:31.479 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.479 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:13:31.479 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.160711+0000", 2024-05-15T13:13:31.479 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.480 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:13:31.480 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.160712+0000", 2024-05-15T13:13:31.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2920000000000001e-06 2024-05-15T13:13:31.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.480 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.480 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:13:31.480 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.160715+0000", 2024-05-15T13:13:31.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3469999999999999e-06 2024-05-15T13:13:31.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.480 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.480 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:13:31.480 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.160725+0000", 2024-05-15T13:13:31.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0629999999999995e-06 2024-05-15T13:13:31.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.480 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:13:31.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.160731+0000", 2024-05-15T13:13:31.481 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1149999999999999e-06 2024-05-15T13:13:31.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.161121+0000", 2024-05-15T13:13:31.481 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00039080000000000001 2024-05-15T13:13:31.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:13:31.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.161244+0000", 2024-05-15T13:13:31.481 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000122874 2024-05-15T13:13:31.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.482 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,5", 2024-05-15T13:13:31.482 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.161363+0000", 2024-05-15T13:13:31.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000118921 2024-05-15T13:13:31.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.482 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.482 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.266851+0000", 2024-05-15T13:13:31.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.105488184 2024-05-15T13:13:31.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.482 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.266898+0000", 2024-05-15T13:13:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6856e-05 2024-05-15T13:13:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:13:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.283281+0000", 2024-05-15T13:13:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016382376000000001 2024-05-15T13:13:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:13:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.283333+0000", 2024-05-15T13:13:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2590999999999998e-05 2024-05-15T13:13:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.483 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:13:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.283385+0000", 2024-05-15T13:13:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1696000000000001e-05 2024-05-15T13:13:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:13:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:190882 6.4 6:2cb94f63:::benchmark_data_smithi177_122278_object60511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-05-15T13:13:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:10:34.160750+0000", 2024-05-15T13:13:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 177.14655396000001, 2024-05-15T13:13:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.122719414, 2024-05-15T13:13:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:13:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:13:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:13:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:13:31.484 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:13:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 190882 2024-05-15T13:13:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:13:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:13:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.160750+0000", 2024-05-15T13:13:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:13:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.160750+0000", 2024-05-15T13:13:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:13:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.160750+0000", 2024-05-15T13:13:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9200000000000002e-07 2024-05-15T13:13:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:13:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.160753+0000", 2024-05-15T13:13:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4509999999999999e-06 2024-05-15T13:13:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:13:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.160759+0000", 2024-05-15T13:13:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4280000000000001e-06 2024-05-15T13:13:31.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:13:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.160764+0000", 2024-05-15T13:13:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9749999999999998e-06 2024-05-15T13:13:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.261826+0000", 2024-05-15T13:13:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.101062203 2024-05-15T13:13:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:13:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.261967+0000", 2024-05-15T13:13:31.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014109699999999999 2024-05-15T13:13:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,5", 2024-05-15T13:13:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.262100+0000", 2024-05-15T13:13:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013259200000000001 2024-05-15T13:13:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.266949+0000", 2024-05-15T13:13:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0048492370000000002 2024-05-15T13:13:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.283230+0000", 2024-05-15T13:13:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016280414999999999 2024-05-15T13:13:31.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:13:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.283394+0000", 2024-05-15T13:13:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016393399999999999 2024-05-15T13:13:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:13:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.283431+0000", 2024-05-15T13:13:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7509999999999998e-05 2024-05-15T13:13:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:13:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:10:34.283469+0000", 2024-05-15T13:13:31.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8179999999999997e-05 2024-05-15T13:13:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:13:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:233354 6.4 6:24faf307:::benchmark_data_smithi177_122278_object102983:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-05-15T13:13:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:27.166151+0000", 2024-05-15T13:13:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 124.141153004, 2024-05-15T13:13:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.120483377, 2024-05-15T13:13:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:13:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:13:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:13:31.490 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:13:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:13:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 233354 2024-05-15T13:13:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:13:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:13:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:27.166151+0000", 2024-05-15T13:13:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:13:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:27.166151+0000", 2024-05-15T13:13:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:13:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:27.166151+0000", 2024-05-15T13:13:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2099999999999997e-07 2024-05-15T13:13:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:13:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:27.166154+0000", 2024-05-15T13:13:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8710000000000001e-06 2024-05-15T13:13:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:13:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:27.166163+0000", 2024-05-15T13:13:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4570000000000005e-06 2024-05-15T13:13:31.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:13:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:27.166169+0000", 2024-05-15T13:13:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8119999999999996e-06 2024-05-15T13:13:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:27.173516+0000", 2024-05-15T13:13:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0073476779999999998 2024-05-15T13:13:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:13:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:27.173599+0000", 2024-05-15T13:13:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2676000000000006e-05 2024-05-15T13:13:31.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,5", 2024-05-15T13:13:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:27.173680+0000", 2024-05-15T13:13:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0776e-05 2024-05-15T13:13:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:13:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:27.178947+0000", 2024-05-15T13:13:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.005267407 2024-05-15T13:13:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:27.183593+0000", 2024-05-15T13:13:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046463490000000001 2024-05-15T13:13:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:27.286524+0000", 2024-05-15T13:13:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.102930411 2024-05-15T13:13:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:13:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:27.286584+0000", 2024-05-15T13:13:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9970999999999999e-05 2024-05-15T13:13:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:13:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:27.286634+0000", 2024-05-15T13:13:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0448000000000002e-05 2024-05-15T13:13:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:13:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:246900 6.e 6:73b61a50:::benchmark_data_smithi177_122278_object116529:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e570)", 2024-05-15T13:13:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:43.366772+0000", 2024-05-15T13:13:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 107.940531838, 2024-05-15T13:13:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.538920976, 2024-05-15T13:13:31.496 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:13:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:13:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:13:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:13:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:13:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 246900 2024-05-15T13:13:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:13:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:13:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.366772+0000", 2024-05-15T13:13:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:13:31.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.366772+0000", 2024-05-15T13:13:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:13:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.366773+0000", 2024-05-15T13:13:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9900000000000009e-07 2024-05-15T13:13:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:13:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.366779+0000", 2024-05-15T13:13:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5840000000000002e-06 2024-05-15T13:13:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:13:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.366791+0000", 2024-05-15T13:13:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2177999999999999e-05 2024-05-15T13:13:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:13:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.366800+0000", 2024-05-15T13:13:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4040000000000001e-06 2024-05-15T13:13:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.366869+0000", 2024-05-15T13:13:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9011999999999996e-05 2024-05-15T13:13:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:13:31.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.366881+0000", 2024-05-15T13:13:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1473e-05 2024-05-15T13:13:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.546365+0000", 2024-05-15T13:13:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.179484528 2024-05-15T13:13:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:13:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.546382+0000", 2024-05-15T13:13:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6494000000000002e-05 2024-05-15T13:13:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.848411+0000", 2024-05-15T13:13:31.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.302029082 2024-05-15T13:13:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:13:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.848548+0000", 2024-05-15T13:13:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013769499999999999 2024-05-15T13:13:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.905605+0000", 2024-05-15T13:13:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.057056253000000001 2024-05-15T13:13:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:13:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.905693+0000", 2024-05-15T13:13:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8274000000000002e-05 2024-05-15T13:13:31.501 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:13:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:246910 6.e 6:7cc3dc78:::benchmark_data_smithi177_122278_object116539:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e570)", 2024-05-15T13:13:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:43.367687+0000", 2024-05-15T13:13:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 107.939616761, 2024-05-15T13:13:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4979548970000001, 2024-05-15T13:13:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:13:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:13:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:13:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:13:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:13:31.502 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 246910 2024-05-15T13:13:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:13:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:13:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.367687+0000", 2024-05-15T13:13:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:13:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.367687+0000", 2024-05-15T13:13:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:13:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.367688+0000", 2024-05-15T13:13:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.097e-06 2024-05-15T13:13:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:13:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.367693+0000", 2024-05-15T13:13:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5460000000000002e-06 2024-05-15T13:13:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:13:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.367708+0000", 2024-05-15T13:13:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5247e-05 2024-05-15T13:13:31.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:13:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.367716+0000", 2024-05-15T13:13:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3129999999999999e-06 2024-05-15T13:13:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.373013+0000", 2024-05-15T13:13:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0052970550000000002 2024-05-15T13:13:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:13:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.373020+0000", 2024-05-15T13:13:31.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.409e-06 2024-05-15T13:13:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.546430+0000", 2024-05-15T13:13:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.17340994400000001 2024-05-15T13:13:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:13:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.546440+0000", 2024-05-15T13:13:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0169e-05 2024-05-15T13:13:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.848909+0000", 2024-05-15T13:13:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.302468959 2024-05-15T13:13:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:13:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.849064+0000", 2024-05-15T13:13:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015553 2024-05-15T13:13:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2,4,5", 2024-05-15T13:13:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.849241+0000", 2024-05-15T13:13:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017630099999999999 2024-05-15T13:13:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.865161+0000", 2024-05-15T13:13:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.015920804 2024-05-15T13:13:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:13:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.865178+0000", 2024-05-15T13:13:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6810999999999999e-05 2024-05-15T13:13:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.865309+0000", 2024-05-15T13:13:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013034599999999999 2024-05-15T13:13:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.865449+0000", 2024-05-15T13:13:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014012700000000001 2024-05-15T13:13:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:13:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.865518+0000", 2024-05-15T13:13:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9197999999999997e-05 2024-05-15T13:13:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:13:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.865642+0000", 2024-05-15T13:13:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000124041 2024-05-15T13:13:31.509 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:13:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:246916 6.e 6:7f5c1cc2:::benchmark_data_smithi177_122278_object116545:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e570)", 2024-05-15T13:13:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:43.382150+0000", 2024-05-15T13:13:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 107.925153447, 2024-05-15T13:13:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.483702957, 2024-05-15T13:13:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:13:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:13:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:13:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:13:31.510 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:13:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 246916 2024-05-15T13:13:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:13:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:13:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.382150+0000", 2024-05-15T13:13:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:13:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.382150+0000", 2024-05-15T13:13:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:13:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.382152+0000", 2024-05-15T13:13:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.829e-06 2024-05-15T13:13:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:13:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.382158+0000", 2024-05-15T13:13:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2400000000000004e-06 2024-05-15T13:13:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:13:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.382170+0000", 2024-05-15T13:13:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1785e-05 2024-05-15T13:13:31.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:13:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.382179+0000", 2024-05-15T13:13:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7649999999999999e-06 2024-05-15T13:13:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.382424+0000", 2024-05-15T13:13:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00024450399999999999 2024-05-15T13:13:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:13:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.382430+0000", 2024-05-15T13:13:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8430000000000004e-06 2024-05-15T13:13:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.546481+0000", 2024-05-15T13:13:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.164050684 2024-05-15T13:13:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:13:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.546490+0000", 2024-05-15T13:13:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.363e-06 2024-05-15T13:13:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.854597+0000", 2024-05-15T13:13:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3081070180000001 2024-05-15T13:13:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:13:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.854761+0000", 2024-05-15T13:13:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016333499999999999 2024-05-15T13:13:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2,4,5", 2024-05-15T13:13:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.854922+0000", 2024-05-15T13:13:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016075200000000001 2024-05-15T13:13:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:13:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.865186+0000", 2024-05-15T13:13:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010264323000000001 2024-05-15T13:13:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.865356+0000", 2024-05-15T13:13:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017057500000000001 2024-05-15T13:13:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.865402+0000", 2024-05-15T13:13:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5581999999999998e-05 2024-05-15T13:13:31.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.865689+0000", 2024-05-15T13:13:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00028735000000000002 2024-05-15T13:13:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:13:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.865739+0000", 2024-05-15T13:13:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9490000000000002e-05 2024-05-15T13:13:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:13:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.865853+0000", 2024-05-15T13:13:31.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011451900000000001 2024-05-15T13:13:31.518 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.518 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:13:31.518 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.518 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:246919 6.e 6:718991f2:::benchmark_data_smithi177_122278_object116548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e570)", 2024-05-15T13:13:31.518 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:43.383877+0000", 2024-05-15T13:13:31.518 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 107.923426675, 2024-05-15T13:13:31.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5034714729999998, 2024-05-15T13:13:31.518 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:13:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:13:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:13:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:13:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:13:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 246919 2024-05-15T13:13:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:13:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:13:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.383877+0000", 2024-05-15T13:13:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:13:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.383877+0000", 2024-05-15T13:13:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:13:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.383878+0000", 2024-05-15T13:13:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2580000000000001e-06 2024-05-15T13:13:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:13:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.383884+0000", 2024-05-15T13:13:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5319999999999997e-06 2024-05-15T13:13:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:13:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.383896+0000", 2024-05-15T13:13:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2179e-05 2024-05-15T13:13:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:13:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.383903+0000", 2024-05-15T13:13:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.238e-06 2024-05-15T13:13:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.383959+0000", 2024-05-15T13:13:31.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5439000000000002e-05 2024-05-15T13:13:31.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.522 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.522 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:13:31.522 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.383968+0000", 2024-05-15T13:13:31.522 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2010000000000005e-06 2024-05-15T13:13:31.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.522 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.522 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.546527+0000", 2024-05-15T13:13:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.162559235 2024-05-15T13:13:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:13:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.546538+0000", 2024-05-15T13:13:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0876999999999999e-05 2024-05-15T13:13:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.865887+0000", 2024-05-15T13:13:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.319348854 2024-05-15T13:13:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:13:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.866024+0000", 2024-05-15T13:13:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000136568 2024-05-15T13:13:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2,4,5", 2024-05-15T13:13:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.866179+0000", 2024-05-15T13:13:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015536099999999999 2024-05-15T13:13:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.875740+0000", 2024-05-15T13:13:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0095607460000000002 2024-05-15T13:13:31.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:13:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.875876+0000", 2024-05-15T13:13:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013675800000000001 2024-05-15T13:13:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.875920+0000", 2024-05-15T13:13:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3578999999999997e-05 2024-05-15T13:13:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.887138+0000", 2024-05-15T13:13:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011217990000000001 2024-05-15T13:13:31.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:13:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.887199+0000", 2024-05-15T13:13:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1316999999999996e-05 2024-05-15T13:13:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:13:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.887349+0000", 2024-05-15T13:13:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014934099999999999 2024-05-15T13:13:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:13:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:246922 6.e 6:76c0ef34:::benchmark_data_smithi177_122278_object116551:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e570)", 2024-05-15T13:13:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:43.387038+0000", 2024-05-15T13:13:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 107.92026622100001, 2024-05-15T13:13:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5033219920000001, 2024-05-15T13:13:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:13:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:13:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:13:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:13:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:13:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 246922 2024-05-15T13:13:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.527 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:13:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:13:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.387038+0000", 2024-05-15T13:13:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:13:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.387038+0000", 2024-05-15T13:13:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:13:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.387039+0000", 2024-05-15T13:13:31.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.331e-06 2024-05-15T13:13:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:13:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.387045+0000", 2024-05-15T13:13:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.806e-06 2024-05-15T13:13:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:13:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.387060+0000", 2024-05-15T13:13:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5254e-05 2024-05-15T13:13:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:13:31.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.387070+0000", 2024-05-15T13:13:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6320000000000003e-06 2024-05-15T13:13:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.387172+0000", 2024-05-15T13:13:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000102132 2024-05-15T13:13:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:13:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.387181+0000", 2024-05-15T13:13:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6989999999999992e-06 2024-05-15T13:13:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.546565+0000", 2024-05-15T13:13:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.159384319 2024-05-15T13:13:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:13:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.546569+0000", 2024-05-15T13:13:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5600000000000004e-06 2024-05-15T13:13:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.871481+0000", 2024-05-15T13:13:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3249118229999999 2024-05-15T13:13:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:13:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.871563+0000", 2024-05-15T13:13:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2286000000000005e-05 2024-05-15T13:13:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2,4,5", 2024-05-15T13:13:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.871641+0000", 2024-05-15T13:13:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7655000000000001e-05 2024-05-15T13:13:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:13:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.887081+0000", 2024-05-15T13:13:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.015439908 2024-05-15T13:13:31.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.887413+0000", 2024-05-15T13:13:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00033151600000000002 2024-05-15T13:13:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.887459+0000", 2024-05-15T13:13:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6873999999999999e-05 2024-05-15T13:13:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.890153+0000", 2024-05-15T13:13:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0026940139999999998 2024-05-15T13:13:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:13:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.890218+0000", 2024-05-15T13:13:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4288000000000005e-05 2024-05-15T13:13:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:13:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.890360+0000", 2024-05-15T13:13:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014189500000000001 2024-05-15T13:13:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:13:31.534 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:246923 6.e 6:7b56eff6:::benchmark_data_smithi177_122278_object116552:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e570)", 2024-05-15T13:13:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:43.387093+0000", 2024-05-15T13:13:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 107.9202111, 2024-05-15T13:13:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.509533826, 2024-05-15T13:13:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:13:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:13:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:13:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:13:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:13:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 246923 2024-05-15T13:13:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.535 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:13:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:13:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.387093+0000", 2024-05-15T13:13:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:13:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.387093+0000", 2024-05-15T13:13:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:13:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.387093+0000", 2024-05-15T13:13:31.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6100000000000002e-07 2024-05-15T13:13:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:13:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.387096+0000", 2024-05-15T13:13:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.435e-06 2024-05-15T13:13:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:13:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.387102+0000", 2024-05-15T13:13:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1809999999999998e-06 2024-05-15T13:13:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:13:31.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.387106+0000", 2024-05-15T13:13:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5410000000000002e-06 2024-05-15T13:13:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.387197+0000", 2024-05-15T13:13:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0438000000000004e-05 2024-05-15T13:13:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:13:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.387201+0000", 2024-05-15T13:13:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0600000000000001e-06 2024-05-15T13:13:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.546596+0000", 2024-05-15T13:13:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.15939481999999999 2024-05-15T13:13:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:13:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.546605+0000", 2024-05-15T13:13:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1870000000000003e-06 2024-05-15T13:13:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.875962+0000", 2024-05-15T13:13:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3293576149999999 2024-05-15T13:13:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:13:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.876177+0000", 2024-05-15T13:13:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00021449899999999999 2024-05-15T13:13:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2,4,5", 2024-05-15T13:13:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.876317+0000", 2024-05-15T13:13:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000140035 2024-05-15T13:13:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.887551+0000", 2024-05-15T13:13:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011234277000000001 2024-05-15T13:13:31.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:13:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.887905+0000", 2024-05-15T13:13:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00035386899999999998 2024-05-15T13:13:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.891580+0000", 2024-05-15T13:13:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0036749650000000001 2024-05-15T13:13:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.896406+0000", 2024-05-15T13:13:31.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0048259189999999997 2024-05-15T13:13:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:13:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.896508+0000", 2024-05-15T13:13:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000101649 2024-05-15T13:13:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:13:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.896627+0000", 2024-05-15T13:13:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011897500000000001 2024-05-15T13:13:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:13:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:248328 6.e 6:7f01531a:::benchmark_data_smithi177_122278_object117957:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e574)", 2024-05-15T13:13:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:46.713549+0000", 2024-05-15T13:13:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 104.593754793, 2024-05-15T13:13:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20530549100000001, 2024-05-15T13:13:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:13:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:13:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:13:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:13:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:13:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 248328 2024-05-15T13:13:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:13:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:13:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.713549+0000", 2024-05-15T13:13:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:13:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.713549+0000", 2024-05-15T13:13:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:13:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.713551+0000", 2024-05-15T13:13:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9400000000000001e-06 2024-05-15T13:13:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:13:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.713560+0000", 2024-05-15T13:13:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2699999999999993e-06 2024-05-15T13:13:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:13:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.713573+0000", 2024-05-15T13:13:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3159999999999999e-05 2024-05-15T13:13:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:13:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.713580+0000", 2024-05-15T13:13:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2709999999999998e-06 2024-05-15T13:13:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.713634+0000", 2024-05-15T13:13:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4648000000000003e-05 2024-05-15T13:13:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:13:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.713727+0000", 2024-05-15T13:13:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2675000000000003e-05 2024-05-15T13:13:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2,4,5", 2024-05-15T13:13:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.713807+0000", 2024-05-15T13:13:31.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9803999999999997e-05 2024-05-15T13:13:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:13:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.717864+0000", 2024-05-15T13:13:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0040570830000000004 2024-05-15T13:13:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.718798+0000", 2024-05-15T13:13:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00093369000000000004 2024-05-15T13:13:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.726578+0000", 2024-05-15T13:13:31.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0077801320000000004 2024-05-15T13:13:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.918644+0000", 2024-05-15T13:13:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.192065825 2024-05-15T13:13:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:13:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.918712+0000", 2024-05-15T13:13:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8171999999999997e-05 2024-05-15T13:13:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:13:31.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.918855+0000", 2024-05-15T13:13:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014282100000000001 2024-05-15T13:13:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:13:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:248331 6.e 6:7ecac96b:::benchmark_data_smithi177_122278_object117960:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e574)", 2024-05-15T13:13:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:46.721362+0000", 2024-05-15T13:13:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 104.585941808, 2024-05-15T13:13:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20578916999999999, 2024-05-15T13:13:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:13:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:13:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:13:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:13:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:13:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 248331 2024-05-15T13:13:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:13:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:13:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.721362+0000", 2024-05-15T13:13:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:13:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.721362+0000", 2024-05-15T13:13:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:13:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.721363+0000", 2024-05-15T13:13:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.15e-06 2024-05-15T13:13:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:13:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.721368+0000", 2024-05-15T13:13:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1259999999999997e-06 2024-05-15T13:13:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:13:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.721380+0000", 2024-05-15T13:13:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1742999999999999e-05 2024-05-15T13:13:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:13:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.721388+0000", 2024-05-15T13:13:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1130000000000001e-06 2024-05-15T13:13:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.721449+0000", 2024-05-15T13:13:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1140000000000001e-05 2024-05-15T13:13:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:13:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.721591+0000", 2024-05-15T13:13:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014162300000000001 2024-05-15T13:13:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2,4,5", 2024-05-15T13:13:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.721749+0000", 2024-05-15T13:13:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015766200000000001 2024-05-15T13:13:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:13:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.727900+0000", 2024-05-15T13:13:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0061513540000000004 2024-05-15T13:13:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.727940+0000", 2024-05-15T13:13:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9545000000000002e-05 2024-05-15T13:13:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.730447+0000", 2024-05-15T13:13:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025069979999999999 2024-05-15T13:13:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.926922+0000", 2024-05-15T13:13:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19647516500000001 2024-05-15T13:13:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:13:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.926995+0000", 2024-05-15T13:13:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3597999999999996e-05 2024-05-15T13:13:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:13:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.927151+0000", 2024-05-15T13:13:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015595299999999999 2024-05-15T13:13:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:13:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:248360 6.e 6:7cd3bbba:::benchmark_data_smithi177_122278_object117989:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e574)", 2024-05-15T13:13:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:46.772574+0000", 2024-05-15T13:13:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 104.534729594, 2024-05-15T13:13:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.160330097, 2024-05-15T13:13:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:13:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:13:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:13:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:13:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:13:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 248360 2024-05-15T13:13:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:13:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:13:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.772574+0000", 2024-05-15T13:13:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:13:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.772574+0000", 2024-05-15T13:13:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:13:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.772576+0000", 2024-05-15T13:13:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.407e-06 2024-05-15T13:13:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:13:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.772582+0000", 2024-05-15T13:13:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3520000000000003e-06 2024-05-15T13:13:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:13:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.772603+0000", 2024-05-15T13:13:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0761999999999999e-05 2024-05-15T13:13:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:13:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.772612+0000", 2024-05-15T13:13:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.628e-06 2024-05-15T13:13:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.773156+0000", 2024-05-15T13:13:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000543718 2024-05-15T13:13:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:13:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.773245+0000", 2024-05-15T13:13:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8941000000000001e-05 2024-05-15T13:13:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2,4,5", 2024-05-15T13:13:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.773334+0000", 2024-05-15T13:13:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9000999999999997e-05 2024-05-15T13:13:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.777901+0000", 2024-05-15T13:13:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0045665269999999999 2024-05-15T13:13:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:13:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.777919+0000", 2024-05-15T13:13:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.808e-05 2024-05-15T13:13:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.790011+0000", 2024-05-15T13:13:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012091925 2024-05-15T13:13:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.932671+0000", 2024-05-15T13:13:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.142660591 2024-05-15T13:13:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:13:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.932770+0000", 2024-05-15T13:13:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8956999999999995e-05 2024-05-15T13:13:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:13:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.932904+0000", 2024-05-15T13:13:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000134208 2024-05-15T13:13:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:13:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:248364 6.e 6:7fb34270:::benchmark_data_smithi177_122278_object117993:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e574)", 2024-05-15T13:13:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:46.783331+0000", 2024-05-15T13:13:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 104.52397325699999, 2024-05-15T13:13:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.14982255, 2024-05-15T13:13:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:13:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:13:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:13:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:13:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:13:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 248364 2024-05-15T13:13:31.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:13:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:13:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.783331+0000", 2024-05-15T13:13:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:13:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.783331+0000", 2024-05-15T13:13:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:13:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.783332+0000", 2024-05-15T13:13:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4929999999999999e-06 2024-05-15T13:13:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:13:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.783337+0000", 2024-05-15T13:13:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2780000000000001e-06 2024-05-15T13:13:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:13:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.783349+0000", 2024-05-15T13:13:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1236999999999999e-05 2024-05-15T13:13:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:13:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.783355+0000", 2024-05-15T13:13:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7530000000000004e-06 2024-05-15T13:13:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.783416+0000", 2024-05-15T13:13:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0297999999999998e-05 2024-05-15T13:13:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:13:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.783572+0000", 2024-05-15T13:13:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000156186 2024-05-15T13:13:31.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2,4,5", 2024-05-15T13:13:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.783722+0000", 2024-05-15T13:13:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014970900000000001 2024-05-15T13:13:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.788733+0000", 2024-05-15T13:13:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050116739999999998 2024-05-15T13:13:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:13:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.788798+0000", 2024-05-15T13:13:31.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4344999999999995e-05 2024-05-15T13:13:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.790056+0000", 2024-05-15T13:13:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001258339 2024-05-15T13:13:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.932956+0000", 2024-05-15T13:13:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.14289997600000001 2024-05-15T13:13:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:13:31.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.933016+0000", 2024-05-15T13:13:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0229999999999998e-05 2024-05-15T13:13:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:13:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.933153+0000", 2024-05-15T13:13:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013703200000000001 2024-05-15T13:13:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:13:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:248376 6.e 6:7183a27b:::benchmark_data_smithi177_122278_object118005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e574)", 2024-05-15T13:13:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:46.815643+0000", 2024-05-15T13:13:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 104.49166059300001, 2024-05-15T13:13:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.126976856, 2024-05-15T13:13:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:13:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:13:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:13:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:13:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:13:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 248376 2024-05-15T13:13:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:13:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:13:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.815643+0000", 2024-05-15T13:13:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:13:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.815643+0000", 2024-05-15T13:13:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:13:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.815644+0000", 2024-05-15T13:13:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.102e-06 2024-05-15T13:13:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:13:31.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.815649+0000", 2024-05-15T13:13:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9389999999999996e-06 2024-05-15T13:13:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:13:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.815661+0000", 2024-05-15T13:13:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1319e-05 2024-05-15T13:13:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:13:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.815666+0000", 2024-05-15T13:13:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.977e-06 2024-05-15T13:13:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.815705+0000", 2024-05-15T13:13:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9511000000000002e-05 2024-05-15T13:13:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:13:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.815800+0000", 2024-05-15T13:13:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4412000000000003e-05 2024-05-15T13:13:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2,4,5", 2024-05-15T13:13:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.815933+0000", 2024-05-15T13:13:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013340100000000001 2024-05-15T13:13:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.820276+0000", 2024-05-15T13:13:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0043427120000000003 2024-05-15T13:13:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.820363+0000", 2024-05-15T13:13:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6877000000000002e-05 2024-05-15T13:13:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:13:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.820385+0000", 2024-05-15T13:13:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2412000000000002e-05 2024-05-15T13:13:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.942428+0000", 2024-05-15T13:13:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12204337799999999 2024-05-15T13:13:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:13:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.942493+0000", 2024-05-15T13:13:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4906000000000005e-05 2024-05-15T13:13:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:13:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:46.942620+0000", 2024-05-15T13:13:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012690999999999999 2024-05-15T13:13:31.574 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:13:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "rep_scrubmap(1.0 e475 from shard 10)", 2024-05-15T13:13:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:12:19.219320+0000", 2024-05-15T13:13:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 72.087984113000005, 2024-05-15T13:13:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.073614337000000002, 2024-05-15T13:13:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:13:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:13:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:13:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:13:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:19.219320+0000", 2024-05-15T13:13:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:13:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:19.219320+0000", 2024-05-15T13:13:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:13:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:19.219321+0000", 2024-05-15T13:13:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.316e-06 2024-05-15T13:13:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:13:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:19.219328+0000", 2024-05-15T13:13:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7830000000000001e-06 2024-05-15T13:13:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:13:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:19.219343+0000", 2024-05-15T13:13:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4992e-05 2024-05-15T13:13:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:13:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:19.219354+0000", 2024-05-15T13:13:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0824e-05 2024-05-15T13:13:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:19.292894+0000", 2024-05-15T13:13:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.073540052999999994 2024-05-15T13:13:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:13:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:19.292909+0000", 2024-05-15T13:13:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5081e-05 2024-05-15T13:13:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:13:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:19.292934+0000", 2024-05-15T13:13:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5287999999999999e-05 2024-05-15T13:13:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.578 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:13:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:13526 7.d 7:b2b696f5:::benchmark_data_smithi177_134484_object13525:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-05-15T13:13:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:12:47.961510+0000", 2024-05-15T13:13:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 43.345793942, 2024-05-15T13:13:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10356096100000001, 2024-05-15T13:13:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:13:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:13:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:13:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:13:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:13:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 13526 2024-05-15T13:13:31.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:13:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:13:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:47.961510+0000", 2024-05-15T13:13:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:13:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:47.961510+0000", 2024-05-15T13:13:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:13:31.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:47.961511+0000", 2024-05-15T13:13:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.336e-06 2024-05-15T13:13:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:13:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:47.961569+0000", 2024-05-15T13:13:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7836e-05 2024-05-15T13:13:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:13:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:47.961581+0000", 2024-05-15T13:13:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2106000000000001e-05 2024-05-15T13:13:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:13:31.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:47.961590+0000", 2024-05-15T13:13:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6060000000000003e-06 2024-05-15T13:13:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:47.961640+0000", 2024-05-15T13:13:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9802000000000002e-05 2024-05-15T13:13:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:13:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:47.961732+0000", 2024-05-15T13:13:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2443999999999998e-05 2024-05-15T13:13:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,3", 2024-05-15T13:13:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:47.961853+0000", 2024-05-15T13:13:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120493 2024-05-15T13:13:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:13:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:47.962692+0000", 2024-05-15T13:13:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00083958000000000004 2024-05-15T13:13:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:47.963598+0000", 2024-05-15T13:13:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00090542699999999999 2024-05-15T13:13:31.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.584 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:48.064961+0000", 2024-05-15T13:13:31.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.101363853 2024-05-15T13:13:31.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.584 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:13:31.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:48.065026+0000", 2024-05-15T13:13:31.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4533e-05 2024-05-15T13:13:31.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.584 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:13:31.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:48.065071+0000", 2024-05-15T13:13:31.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4944999999999997e-05 2024-05-15T13:13:31.584 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.585 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:13:31.585 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.585 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:43984 7.d 7:ba4362e9:::benchmark_data_smithi177_134484_object43983:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e611)", 2024-05-15T13:13:31.585 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:13:03.400522+0000", 2024-05-15T13:13:31.585 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 27.906781771999999, 2024-05-15T13:13:31.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.069967426999999999, 2024-05-15T13:13:31.585 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:13:31.585 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:13:31.585 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:13:31.585 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:13:31.585 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:13:31.585 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 43984 2024-05-15T13:13:31.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.586 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:13:31.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:13:31.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:03.400522+0000", 2024-05-15T13:13:31.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:13:31.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:03.400522+0000", 2024-05-15T13:13:31.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:13:31.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:03.400523+0000", 2024-05-15T13:13:31.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2399999999999997e-07 2024-05-15T13:13:31.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:13:31.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:03.400602+0000", 2024-05-15T13:13:31.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9276999999999993e-05 2024-05-15T13:13:31.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:13:31.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:03.400615+0000", 2024-05-15T13:13:31.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2369e-05 2024-05-15T13:13:31.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:13:31.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:03.400622+0000", 2024-05-15T13:13:31.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8909999999999995e-06 2024-05-15T13:13:31.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:03.402676+0000", 2024-05-15T13:13:31.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020537490000000001 2024-05-15T13:13:31.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:13:31.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:03.402806+0000", 2024-05-15T13:13:31.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012954700000000001 2024-05-15T13:13:31.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,3", 2024-05-15T13:13:31.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:03.402909+0000", 2024-05-15T13:13:31.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000102811 2024-05-15T13:13:31.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:13:31.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:03.406658+0000", 2024-05-15T13:13:31.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0037498969999999999 2024-05-15T13:13:31.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:03.470259+0000", 2024-05-15T13:13:31.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.063600655000000006 2024-05-15T13:13:31.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:03.470385+0000", 2024-05-15T13:13:31.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000126248 2024-05-15T13:13:31.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:13:31.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:03.470445+0000", 2024-05-15T13:13:31.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9471e-05 2024-05-15T13:13:31.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:13:31.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:03.470490+0000", 2024-05-15T13:13:31.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4688000000000003e-05 2024-05-15T13:13:31.591 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.591 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:13:31.591 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.591 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:71805 7.d 7:b1d9c9be:::benchmark_data_smithi177_134484_object71804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e622)", 2024-05-15T13:13:31.591 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:13:22.986761+0000", 2024-05-15T13:13:31.591 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 8.3205427949999997, 2024-05-15T13:13:31.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.095718945, 2024-05-15T13:13:31.591 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:13:31.591 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:13:31.591 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:13:31.592 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:13:31.592 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:13:31.592 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 71805 2024-05-15T13:13:31.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.592 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:13:31.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:13:31.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:22.986761+0000", 2024-05-15T13:13:31.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:13:31.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:22.986761+0000", 2024-05-15T13:13:31.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:13:31.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:22.986770+0000", 2024-05-15T13:13:31.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.217e-06 2024-05-15T13:13:31.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:13:31.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:22.986896+0000", 2024-05-15T13:13:31.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012599900000000001 2024-05-15T13:13:31.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:13:31.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:22.986916+0000", 2024-05-15T13:13:31.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9616e-05 2024-05-15T13:13:31.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:13:31.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:22.986924+0000", 2024-05-15T13:13:31.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7409999999999992e-06 2024-05-15T13:13:31.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:22.986977+0000", 2024-05-15T13:13:31.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3332999999999999e-05 2024-05-15T13:13:31.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:13:31.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:22.987097+0000", 2024-05-15T13:13:31.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012018099999999999 2024-05-15T13:13:31.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,3", 2024-05-15T13:13:31.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:22.987285+0000", 2024-05-15T13:13:31.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00018824799999999999 2024-05-15T13:13:31.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:13:31.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:22.989636+0000", 2024-05-15T13:13:31.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002350946 2024-05-15T13:13:31.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:22.990028+0000", 2024-05-15T13:13:31.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00039119099999999999 2024-05-15T13:13:31.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:23.082370+0000", 2024-05-15T13:13:31.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.092342386999999998 2024-05-15T13:13:31.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:13:31.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:23.082432+0000", 2024-05-15T13:13:31.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2487e-05 2024-05-15T13:13:31.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.597 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:13:31.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:23.082480+0000", 2024-05-15T13:13:31.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7599000000000003e-05 2024-05-15T13:13:31.597 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.597 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:13:31.597 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.597 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:71824 7.d 7:bc197824:::benchmark_data_smithi177_134484_object71823:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e622)", 2024-05-15T13:13:31.597 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:13:23.003277+0000", 2024-05-15T13:13:31.597 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 8.3040266169999999, 2024-05-15T13:13:31.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.085335556000000007, 2024-05-15T13:13:31.597 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:13:31.598 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:13:31.598 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:13:31.598 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:13:31.598 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:13:31.598 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 71824 2024-05-15T13:13:31.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.598 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:13:31.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.598 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:13:31.598 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:23.003277+0000", 2024-05-15T13:13:31.598 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.598 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:13:31.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:23.003277+0000", 2024-05-15T13:13:31.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:13:31.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:23.003279+0000", 2024-05-15T13:13:31.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4640000000000001e-06 2024-05-15T13:13:31.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:13:31.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:23.003342+0000", 2024-05-15T13:13:31.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3608999999999994e-05 2024-05-15T13:13:31.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:13:31.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:23.003356+0000", 2024-05-15T13:13:31.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3283e-05 2024-05-15T13:13:31.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:13:31.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:23.003363+0000", 2024-05-15T13:13:31.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0949999999999998e-06 2024-05-15T13:13:31.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:23.003433+0000", 2024-05-15T13:13:31.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0109000000000002e-05 2024-05-15T13:13:31.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:13:31.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:23.003537+0000", 2024-05-15T13:13:31.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010425300000000001 2024-05-15T13:13:31.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,3", 2024-05-15T13:13:31.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:23.003629+0000", 2024-05-15T13:13:31.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2268000000000005e-05 2024-05-15T13:13:31.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:23.006257+0000", 2024-05-15T13:13:31.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0026276870000000001 2024-05-15T13:13:31.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.602 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:13:31.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:23.006867+0000", 2024-05-15T13:13:31.602 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00061006099999999996 2024-05-15T13:13:31.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.602 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:23.088499+0000", 2024-05-15T13:13:31.602 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.081632061000000006 2024-05-15T13:13:31.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.602 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:13:31.603 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:23.088561+0000", 2024-05-15T13:13:31.603 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2298000000000006e-05 2024-05-15T13:13:31.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.603 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.603 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:13:31.603 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:23.088613+0000", 2024-05-15T13:13:31.603 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1368e-05 2024-05-15T13:13:31.603 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.603 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:13:31.603 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.603 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.603 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:81065 7.d 7:b11a1c5e:::benchmark_data_smithi177_134484_object81064:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e625)", 2024-05-15T13:13:31.604 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:13:30.733710+0000", 2024-05-15T13:13:31.604 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.57359397599999995, 2024-05-15T13:13:31.604 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.102857063, 2024-05-15T13:13:31.604 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:13:31.604 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:13:31.604 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:13:31.604 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:13:31.604 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:13:31.604 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 81065 2024-05-15T13:13:31.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.604 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:13:31.604 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.604 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:13:31.604 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:30.733710+0000", 2024-05-15T13:13:31.604 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.605 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:13:31.605 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:30.733710+0000", 2024-05-15T13:13:31.605 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:13:31.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.605 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:13:31.605 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:30.733712+0000", 2024-05-15T13:13:31.605 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6589999999999999e-06 2024-05-15T13:13:31.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.605 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:13:31.605 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:30.733775+0000", 2024-05-15T13:13:31.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3156000000000003e-05 2024-05-15T13:13:31.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.606 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:13:31.606 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:30.733788+0000", 2024-05-15T13:13:31.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2842e-05 2024-05-15T13:13:31.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.606 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:13:31.606 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:30.733795+0000", 2024-05-15T13:13:31.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9689999999999997e-06 2024-05-15T13:13:31.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.607 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:13:31.607 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:30.733853+0000", 2024-05-15T13:13:31.607 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.893e-05 2024-05-15T13:13:31.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.607 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.607 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:13:31.607 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:30.733971+0000", 2024-05-15T13:13:31.607 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000117308 2024-05-15T13:13:31.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.607 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.607 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 0,3", 2024-05-15T13:13:31.607 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:30.734054+0000", 2024-05-15T13:13:31.607 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3706000000000001e-05 2024-05-15T13:13:31.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:13:31.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:30.736324+0000", 2024-05-15T13:13:31.608 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0022695250000000001 2024-05-15T13:13:31.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:30.736590+0000", 2024-05-15T13:13:31.608 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00026641700000000002 2024-05-15T13:13:31.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:13:31.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:30.836450+0000", 2024-05-15T13:13:31.609 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.099859753999999995 2024-05-15T13:13:31.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.609 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.609 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:13:31.609 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:30.836519+0000", 2024-05-15T13:13:31.609 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9058000000000002e-05 2024-05-15T13:13:31.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:13:31.609 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:13:31.609 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:13:31.609 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:13:30.836567+0000", 2024-05-15T13:13:31.609 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7738999999999998e-05 2024-05-15T13:13:31.609 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.609 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:13:31.609 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.610 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:13:31.610 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:13:31.610 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:13:31.611 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:31 smithi120 ceph-mon[43961]: pgmap v1640: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.8 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-05-15T13:13:31.611 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:31 smithi177 ceph-mon[44303]: pgmap v1640: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.8 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-05-15T13:13:31.611 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:31 smithi027 ceph-mon[39795]: pgmap v1640: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.8 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-05-15T13:13:31.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:31.612 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:31.612 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:13:31.612 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:13:31.714 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:13:31.759 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:31.796 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:13:31.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:31.800 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:13:31.884 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 49 16 82316 82300 104.947 72.8125 0.0115104 0.00952406 2024-05-15T13:13:31.900 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:13:31.946 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:32.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:32 smithi027 ceph-mon[39795]: 7.9 scrub ok 2024-05-15T13:13:32.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:32 smithi120 ceph-mon[43961]: 7.9 scrub ok 2024-05-15T13:13:32.685 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:32 smithi177 ceph-mon[44303]: 7.9 scrub ok 2024-05-15T13:13:32.785 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:13:32.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:32.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:13:32.884 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 50 16 83510 83494 104.341 74.625 0.0231283 0.00957958 2024-05-15T13:13:32.890 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:13:32.917 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:32.937 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:13:32.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:32.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:13:33.041 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:13:33.119 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:13:33.119 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:33.335 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:13:33.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:33.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:13:33.358 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:33 smithi027 ceph-mon[39795]: pgmap v1641: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-05-15T13:13:33.439 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:13:33.496 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:33.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:33 smithi120 ceph-mon[43961]: pgmap v1641: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-05-15T13:13:33.685 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:33 smithi177 ceph-mon[44303]: pgmap v1641: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-05-15T13:13:33.884 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 51 16 84641 84625 103.681 70.6875 0.00998044 0.00963972 2024-05-15T13:13:34.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:34 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:13:34.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:34 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:13:34.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:34 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1677932512' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:13:34.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:34 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:13:34.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:34 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:13:34.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:34 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1677932512' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:13:34.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:34 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:13:34.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:34 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:13:34.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:34 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1677932512' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:13:34.884 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 52 16 85583 85567 102.819 58.875 0.0111576 0.0097224 2024-05-15T13:13:35.393 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:13:35.394 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:13:35.394 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:13:35.394 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:13:35.394 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:13:35.394 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:13:35.394 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:13:35.394 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:13:35.394 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:13:35.394 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:13:35.394 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:13:35.394 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:13:35.395 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:13:35.395 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:13:35.395 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:13:35.395 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:13:35.395 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:13:35.395 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:13:35.395 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:13:35.395 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:13:35.395 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:13:35.395 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:13:35.395 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:13:35.395 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:13:35.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:35 smithi027 ceph-mon[39795]: 7.8 scrub starts 2024-05-15T13:13:35.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:35 smithi027 ceph-mon[39795]: 7.3 scrub starts 2024-05-15T13:13:35.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:35 smithi027 ceph-mon[39795]: osdmap e626: 12 total, 12 up, 10 in 2024-05-15T13:13:35.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:35 smithi027 ceph-mon[39795]: pgmap v1643: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-05-15T13:13:35.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:35 smithi120 ceph-mon[43961]: 7.8 scrub starts 2024-05-15T13:13:35.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:35 smithi120 ceph-mon[43961]: 7.3 scrub starts 2024-05-15T13:13:35.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:35 smithi120 ceph-mon[43961]: osdmap e626: 12 total, 12 up, 10 in 2024-05-15T13:13:35.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:35 smithi120 ceph-mon[43961]: pgmap v1643: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-05-15T13:13:35.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:35 smithi177 ceph-mon[44303]: 7.8 scrub starts 2024-05-15T13:13:35.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:35 smithi177 ceph-mon[44303]: 7.3 scrub starts 2024-05-15T13:13:35.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:35 smithi177 ceph-mon[44303]: osdmap e626: 12 total, 12 up, 10 in 2024-05-15T13:13:35.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:35 smithi177 ceph-mon[44303]: pgmap v1643: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-05-15T13:13:35.887 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 53 16 86652 86636 102.135 66.8125 0.0140499 0.00978639 2024-05-15T13:13:36.490 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:13:36.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:36.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:13:36.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:36 smithi027 ceph-mon[39795]: 7.8 scrub ok 2024-05-15T13:13:36.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:36 smithi120 ceph-mon[43961]: 7.8 scrub ok 2024-05-15T13:13:36.599 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:13:36.626 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:36.647 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:13:36.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:36.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:13:36.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:36 smithi177 ceph-mon[44303]: 7.8 scrub ok 2024-05-15T13:13:36.750 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:13:36.777 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:36.797 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:13:36.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:36.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:13:36.887 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 54 16 87779 87763 101.548 70.4375 0.00783422 0.00984303 2024-05-15T13:13:36.902 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:13:36.953 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:36.987 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:13:36.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:36.992 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:13:37.093 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:13:37.140 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:37.173 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-05-15T13:13:37.284 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:37 smithi177 ceph-mon[44303]: 7.3 scrub ok 2024-05-15T13:13:37.284 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:37 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:13:37.285 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:37 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:13:37.285 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:37 smithi177 ceph-mon[44303]: osdmap e627: 12 total, 12 up, 10 in 2024-05-15T13:13:37.285 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:37 smithi177 ceph-mon[44303]: pgmap v1645: 37 pgs: 37 active+clean; 5.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-05-15T13:13:37.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:37 smithi027 ceph-mon[39795]: 7.3 scrub ok 2024-05-15T13:13:37.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:37 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:13:37.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:37 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:13:37.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:37 smithi027 ceph-mon[39795]: osdmap e627: 12 total, 12 up, 10 in 2024-05-15T13:13:37.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:37 smithi027 ceph-mon[39795]: pgmap v1645: 37 pgs: 37 active+clean; 5.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-05-15T13:13:37.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:37 smithi120 ceph-mon[43961]: 7.3 scrub ok 2024-05-15T13:13:37.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:37 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:13:37.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:37 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:13:37.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:37 smithi120 ceph-mon[43961]: osdmap e627: 12 total, 12 up, 10 in 2024-05-15T13:13:37.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:37 smithi120 ceph-mon[43961]: pgmap v1645: 37 pgs: 37 active+clean; 5.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-05-15T13:13:37.887 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 55 16 88781 88765 100.841 62.625 0.00682002 0.00991242 2024-05-15T13:13:38.887 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 56 16 89709 89693 100.076 58 0.0079209 0.00998787 2024-05-15T13:13:39.109 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:13:39.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:39.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:13:39.220 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:13:39.247 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:39.267 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:13:39.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:39.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:13:39.371 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:13:39.393 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:39 smithi027 ceph-mon[39795]: 7.e scrub starts 2024-05-15T13:13:39.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:39 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:13:39.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:39 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:13:39.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:39 smithi027 ceph-mon[39795]: pgmap v1646: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-05-15T13:13:39.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:39 smithi027 ceph-mon[39795]: osdmap e628: 12 total, 12 up, 10 in 2024-05-15T13:13:39.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:39 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:13:39.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:39 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:13:39.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:39 smithi027 ceph-mon[39795]: osdmap e629: 12 total, 12 up, 10 in 2024-05-15T13:13:39.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:39 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2142982260' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:13:39.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:39 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:13:39.398 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:39.418 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:13:39.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:39.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:13:39.522 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:13:39.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:39 smithi120 ceph-mon[43961]: 7.e scrub starts 2024-05-15T13:13:39.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:39 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:13:39.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:39 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:13:39.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:39 smithi120 ceph-mon[43961]: pgmap v1646: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-05-15T13:13:39.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:39 smithi120 ceph-mon[43961]: osdmap e628: 12 total, 12 up, 10 in 2024-05-15T13:13:39.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:39 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:13:39.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:39 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:13:39.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:39 smithi120 ceph-mon[43961]: osdmap e629: 12 total, 12 up, 10 in 2024-05-15T13:13:39.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:39 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2142982260' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:13:39.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:39 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:13:39.584 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:39.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:39 smithi177 ceph-mon[44303]: 7.e scrub starts 2024-05-15T13:13:39.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:39 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:13:39.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:39 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:13:39.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:39 smithi177 ceph-mon[44303]: pgmap v1646: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-05-15T13:13:39.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:39 smithi177 ceph-mon[44303]: osdmap e628: 12 total, 12 up, 10 in 2024-05-15T13:13:39.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:39 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:13:39.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:39 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:13:39.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:39 smithi177 ceph-mon[44303]: osdmap e629: 12 total, 12 up, 10 in 2024-05-15T13:13:39.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:39 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2142982260' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:13:39.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:39 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:13:39.887 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 57 16 90655 90639 99.3574 59.125 0.032996 0.0100604 2024-05-15T13:13:39.917 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:13:40.224 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:13:40.685 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:13:40.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:40 smithi177 ceph-mon[44303]: 7.e scrub ok 2024-05-15T13:13:40.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:40 smithi177 ceph-mon[44303]: 7.d scrub starts 2024-05-15T13:13:40.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:40 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:13:40.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:40 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:13:40.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:40 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:13:40.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:40 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T13:13:40.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:40 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:13:40.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:40 smithi177 ceph-mon[44303]: osdmap e630: 12 total, 12 up, 10 in 2024-05-15T13:13:40.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:40.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:13:40.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:40 smithi027 ceph-mon[39795]: 7.e scrub ok 2024-05-15T13:13:40.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:40 smithi027 ceph-mon[39795]: 7.d scrub starts 2024-05-15T13:13:40.694 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:40 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:13:40.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:40 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:13:40.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:40 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:13:40.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:40 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T13:13:40.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:40 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:13:40.695 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:40 smithi027 ceph-mon[39795]: osdmap e630: 12 total, 12 up, 10 in 2024-05-15T13:13:40.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:40 smithi120 ceph-mon[43961]: 7.e scrub ok 2024-05-15T13:13:40.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:40 smithi120 ceph-mon[43961]: 7.d scrub starts 2024-05-15T13:13:40.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:40 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:13:40.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:40 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:13:40.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:40 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:13:40.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:40 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T13:13:40.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:40 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:13:40.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:40 smithi120 ceph-mon[43961]: osdmap e630: 12 total, 12 up, 10 in 2024-05-15T13:13:40.794 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:13:40.822 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:40.842 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:13:40.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:40.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:13:40.887 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 58 16 91705 91689 98.7759 65.625 0.0108939 0.0101198 2024-05-15T13:13:40.946 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:13:41.006 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:41.078 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:13:41.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:41.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:13:41.129 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:13:41.180 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:13:41.222 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:41.260 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:13:41.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:41.263 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:13:41.365 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:13:41.398 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:41.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:41 smithi177 ceph-mon[44303]: pgmap v1650: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-05-15T13:13:41.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:41 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:13:41.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:41 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:13:41.685 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:41 smithi177 ceph-mon[44303]: 7.d scrub ok 2024-05-15T13:13:41.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:41 smithi027 ceph-mon[39795]: pgmap v1650: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-05-15T13:13:41.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:41 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:13:41.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:41 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:13:41.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:41 smithi027 ceph-mon[39795]: 7.d scrub ok 2024-05-15T13:13:41.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:41 smithi120 ceph-mon[43961]: pgmap v1650: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-05-15T13:13:41.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:41 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:13:41.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:41 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:13:41.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:41 smithi120 ceph-mon[43961]: 7.d scrub ok 2024-05-15T13:13:41.887 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 59 16 92746 92730 98.2044 65.0625 0.0138315 0.0101768 2024-05-15T13:13:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:13:42.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:42.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:13:42.170 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:13:42.271 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:42.325 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:42 smithi027 ceph-mon[39795]: osdmap e631: 12 total, 12 up, 10 in 2024-05-15T13:13:42.325 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:42 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:13:42.325 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:42 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:13:42.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:42 smithi177 ceph-mon[44303]: osdmap e631: 12 total, 12 up, 10 in 2024-05-15T13:13:42.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:42 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:13:42.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:42 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:13:42.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:42 smithi120 ceph-mon[43961]: osdmap e631: 12 total, 12 up, 10 in 2024-05-15T13:13:42.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:42 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:13:42.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:42 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:13:42.890 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T13:13:42.897694+0000 min lat: 0.00113694 max lat: 0.603428 avg lat: 0.0102364 2024-05-15T13:13:42.890 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:13:42.890 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 60 16 93781 93765 97.6417 64.6875 0.0116392 0.0102364 2024-05-15T13:13:43.040 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:13:43.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:43.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:13:43.144 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:13:43.196 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:43.539 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:13:43.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:43.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:13:43.651 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:13:43.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:43 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:13:43.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:43 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:13:43.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:43 smithi177 ceph-mon[44303]: pgmap v1652: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.5 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-05-15T13:13:43.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:43 smithi177 ceph-mon[44303]: osdmap e632: 12 total, 12 up, 10 in 2024-05-15T13:13:43.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:43 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:13:43.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:43 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:13:43.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:43 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:13:43.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:43 smithi177 ceph-mon[44303]: 7.7 scrub starts 2024-05-15T13:13:43.691 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:43 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:13:43.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:43 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:13:43.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:43 smithi027 ceph-mon[39795]: pgmap v1652: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.5 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-05-15T13:13:43.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:43 smithi027 ceph-mon[39795]: osdmap e632: 12 total, 12 up, 10 in 2024-05-15T13:13:43.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:43 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:13:43.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:43 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:13:43.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:43 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:13:43.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:43 smithi027 ceph-mon[39795]: 7.7 scrub starts 2024-05-15T13:13:43.694 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:43.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:43 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:13:43.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:43 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:13:43.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:43 smithi120 ceph-mon[43961]: pgmap v1652: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.5 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-05-15T13:13:43.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:43 smithi120 ceph-mon[43961]: osdmap e632: 12 total, 12 up, 10 in 2024-05-15T13:13:43.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:43 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:13:43.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:43 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:13:43.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:43 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:13:43.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:43 smithi120 ceph-mon[43961]: 7.7 scrub starts 2024-05-15T13:13:43.891 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 61 16 94700 94684 96.9815 57.4375 0.0130648 0.0103067 2024-05-15T13:13:43.909 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:13:43.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:43.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:13:44.022 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:13:44.072 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:44.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:44 smithi177 ceph-mon[44303]: 7.1 scrub starts 2024-05-15T13:13:44.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:44 smithi177 ceph-mon[44303]: osdmap e633: 12 total, 12 up, 10 in 2024-05-15T13:13:44.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:44 smithi027 ceph-mon[39795]: 7.1 scrub starts 2024-05-15T13:13:44.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:44 smithi027 ceph-mon[39795]: osdmap e633: 12 total, 12 up, 10 in 2024-05-15T13:13:44.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:44 smithi120 ceph-mon[43961]: 7.1 scrub starts 2024-05-15T13:13:44.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:44 smithi120 ceph-mon[43961]: osdmap e633: 12 total, 12 up, 10 in 2024-05-15T13:13:44.891 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 62 16 95624 95608 96.3487 57.75 0.0176537 0.0103743 2024-05-15T13:13:45.439 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:13:45.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:45.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:13:45.471 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:13:45.542 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:13:45.607 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:45 smithi177 ceph-mon[44303]: 7.b deep-scrub starts 2024-05-15T13:13:45.607 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:45 smithi177 ceph-mon[44303]: 7.7 scrub ok 2024-05-15T13:13:45.607 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:45 smithi177 ceph-mon[44303]: 7.1 scrub ok 2024-05-15T13:13:45.607 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:45 smithi177 ceph-mon[44303]: pgmap v1655: 37 pgs: 37 active+clean; 5.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-05-15T13:13:45.609 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:45.643 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:13:45.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:45.650 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:13:45.751 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:13:45.759 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:45 smithi027 ceph-mon[39795]: 7.b deep-scrub starts 2024-05-15T13:13:45.760 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:45 smithi027 ceph-mon[39795]: 7.7 scrub ok 2024-05-15T13:13:45.760 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:45 smithi027 ceph-mon[39795]: 7.1 scrub ok 2024-05-15T13:13:45.760 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:45 smithi027 ceph-mon[39795]: pgmap v1655: 37 pgs: 37 active+clean; 5.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-05-15T13:13:45.776 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:45 smithi120 ceph-mon[43961]: 7.b deep-scrub starts 2024-05-15T13:13:45.776 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:45 smithi120 ceph-mon[43961]: 7.7 scrub ok 2024-05-15T13:13:45.777 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:45 smithi120 ceph-mon[43961]: 7.1 scrub ok 2024-05-15T13:13:45.777 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:45 smithi120 ceph-mon[43961]: pgmap v1655: 37 pgs: 37 active+clean; 5.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-05-15T13:13:45.778 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:45.801 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:13:45.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:45.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:13:45.891 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 63 16 96533 96517 95.7213 56.8125 0.0169506 0.0104417 2024-05-15T13:13:45.904 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:13:45.967 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:45.999 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:13:46.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:46.001 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:13:46.102 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:13:46.154 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:46.352 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:46 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:13:46.353 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:46 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:13:46.353 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:46 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/428170479' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T13:13:46.353 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:46 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T13:13:46.367 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.1 to 1 (10000) 2024-05-15T13:13:46.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:46 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:13:46.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:46 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:13:46.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:46 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/428170479' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T13:13:46.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:46 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T13:13:46.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:46 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:13:46.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:46 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:13:46.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:46 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/428170479' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T13:13:46.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:46 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T13:13:46.814 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:13:46.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:46.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:13:46.891 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 64 16 97029 97013 94.7102 31 0.00357437 0.0104724 2024-05-15T13:13:46.919 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:13:46.947 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:46.966 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:13:46.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:46.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:13:47.071 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:13:47.097 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:47.116 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:13:47.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:47.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:13:47.220 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:13:47.247 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:47.268 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 2 1 2024-05-15T13:13:47.269 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:13:47.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:47.272 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:13:47.345 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:13:47.373 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:13:47.418 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:47.457 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:13:47.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:47.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:13:47.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:47 smithi027 ceph-mon[39795]: pgmap v1656: 37 pgs: 37 active+clean; 5.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 56 MiB/s wr, 893 op/s 2024-05-15T13:13:47.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:47 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T13:13:47.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:47 smithi027 ceph-mon[39795]: osdmap e634: 12 total, 12 up, 10 in 2024-05-15T13:13:47.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:47 smithi027 ceph-mon[39795]: 7.a scrub starts 2024-05-15T13:13:47.562 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:13:47.588 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:47 smithi177 ceph-mon[44303]: pgmap v1656: 37 pgs: 37 active+clean; 5.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 56 MiB/s wr, 893 op/s 2024-05-15T13:13:47.588 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:47 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T13:13:47.588 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:47 smithi177 ceph-mon[44303]: osdmap e634: 12 total, 12 up, 10 in 2024-05-15T13:13:47.588 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:47 smithi177 ceph-mon[44303]: 7.a scrub starts 2024-05-15T13:13:47.590 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:47.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:47.614 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:13:47.614 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:13:47.715 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:13:47.744 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:47.783 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:13:47.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:47 smithi120 ceph-mon[43961]: pgmap v1656: 37 pgs: 37 active+clean; 5.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 56 MiB/s wr, 893 op/s 2024-05-15T13:13:47.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:47 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T13:13:47.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:47 smithi120 ceph-mon[43961]: osdmap e634: 12 total, 12 up, 10 in 2024-05-15T13:13:47.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:47 smithi120 ceph-mon[43961]: 7.a scrub starts 2024-05-15T13:13:47.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:47.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:13:47.887 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:13:47.891 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 65 16 97029 97013 93.2533 0 - 0.0104724 2024-05-15T13:13:47.899 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:13:47.899 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:13:47.915 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:48.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:48 smithi027 ceph-mon[39795]: osdmap e635: 12 total, 12 up, 10 in 2024-05-15T13:13:48.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:48 smithi027 ceph-mon[39795]: 7.a scrub ok 2024-05-15T13:13:48.505 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:48 smithi027 ceph-mon[39795]: osdmap e636: 12 total, 12 up, 10 in 2024-05-15T13:13:48.548 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-05-15T13:13:48.563 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:13:48.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:48.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:13:48.669 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:13:48.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:48 smithi177 ceph-mon[44303]: osdmap e635: 12 total, 12 up, 10 in 2024-05-15T13:13:48.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:48 smithi177 ceph-mon[44303]: 7.a scrub ok 2024-05-15T13:13:48.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:48 smithi177 ceph-mon[44303]: osdmap e636: 12 total, 12 up, 10 in 2024-05-15T13:13:48.695 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:48 smithi120 ceph-mon[43961]: osdmap e635: 12 total, 12 up, 10 in 2024-05-15T13:13:48.695 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:48 smithi120 ceph-mon[43961]: 7.a scrub ok 2024-05-15T13:13:48.695 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:48 smithi120 ceph-mon[43961]: osdmap e636: 12 total, 12 up, 10 in 2024-05-15T13:13:48.697 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:48.717 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:13:48.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:48.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:13:48.821 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:13:48.847 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:48.868 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:13:48.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:48.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:13:48.891 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 66 16 97201 97185 92.0035 5.375 0.056918 0.0108645 2024-05-15T13:13:48.972 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:13:49.064 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:49.115 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:13:49.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:49.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:13:49.220 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:13:49.304 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:13:49.304 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:13:49.323 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:49.425 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:13:49.470 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:13:49.519 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:13:49.578 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:13:49.602 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:13:49.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:49.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:13:49.614 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:49 smithi027 ceph-mon[39795]: pgmap v1659: 37 pgs: 1 unknown, 1 peering, 35 active+clean; 5.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail 2024-05-15T13:13:49.614 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:49 smithi027 ceph-mon[39795]: osdmap e637: 12 total, 12 up, 10 in 2024-05-15T13:13:49.614 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:49 smithi027 ceph-mon[39795]: 7.2 deep-scrub starts 2024-05-15T13:13:49.614 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:13:49.663 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:13:49.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:49 smithi177 ceph-mon[44303]: pgmap v1659: 37 pgs: 1 unknown, 1 peering, 35 active+clean; 5.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail 2024-05-15T13:13:49.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:49 smithi177 ceph-mon[44303]: osdmap e637: 12 total, 12 up, 10 in 2024-05-15T13:13:49.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:49 smithi177 ceph-mon[44303]: 7.2 deep-scrub starts 2024-05-15T13:13:49.693 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:13:49.715 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:13:49.731 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:13:49.746 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:49.773 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:13:49.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:49.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:13:49.781 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:13:49.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:49 smithi120 ceph-mon[43961]: pgmap v1659: 37 pgs: 1 unknown, 1 peering, 35 active+clean; 5.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail 2024-05-15T13:13:49.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:49 smithi120 ceph-mon[43961]: osdmap e637: 12 total, 12 up, 10 in 2024-05-15T13:13:49.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:49 smithi120 ceph-mon[43961]: 7.2 deep-scrub starts 2024-05-15T13:13:49.837 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:13:49.878 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:13:49.887 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:13:49.891 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 67 16 97778 97762 91.1687 36.0625 0.00952077 0.0109633 2024-05-15T13:13:49.944 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:13:49.959 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:49.969 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:13:49.970 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:13:49.970 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:13:49.970 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:13:49.970 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:13:49.970 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:13:49.970 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:13:49.970 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:13:49.971 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:13:49.971 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:13:49.971 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:13:49.971 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:13:50.640 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:13:50.651 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:13:50.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:50.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:13:50.657 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:50 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2499920246' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:13:50.657 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:50 smithi027 ceph-mon[39795]: osdmap e638: 12 total, 12 up, 10 in 2024-05-15T13:13:50.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:50 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2499920246' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:13:50.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:50 smithi177 ceph-mon[44303]: osdmap e638: 12 total, 12 up, 10 in 2024-05-15T13:13:50.757 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:13:50.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:50 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2499920246' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:13:50.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:50 smithi120 ceph-mon[43961]: osdmap e638: 12 total, 12 up, 10 in 2024-05-15T13:13:50.830 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:50.892 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 68 16 98391 98375 90.3915 38.3125 0.0240031 0.0110558 2024-05-15T13:13:50.972 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:13:50.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:50.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:13:51.082 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:13:51.117 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:51.148 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:13:51.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:51.154 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:13:51.255 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:13:51.291 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:51.332 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:13:51.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:51.337 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:13:51.414 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:51 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:13:51.414 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:51 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:13:51.414 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:51 smithi177 ceph-mon[44303]: pgmap v1663: 37 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+undersized+remapped, 1 peering, 33 active+clean; 5.9 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 45 MiB/s wr, 717 op/s; 6118/291861 objects degraded (2.096%); 6096/291861 objects misplaced (2.089%); 359 KiB/s, 5 objects/s recovering 2024-05-15T13:13:51.414 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:51 smithi177 ceph-mon[44303]: Health check failed: Degraded data redundancy: 6118/291861 objects degraded (2.096%), 1 pg degraded (PG_DEGRADED) 2024-05-15T13:13:51.415 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:51 smithi177 ceph-mon[44303]: osdmap e639: 12 total, 12 up, 10 in 2024-05-15T13:13:51.438 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:13:51.465 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:51.483 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:13:51.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:51.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:13:51.589 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:13:51.676 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:51 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:13:51.676 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:51 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:13:51.676 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:51 smithi027 ceph-mon[39795]: pgmap v1663: 37 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+undersized+remapped, 1 peering, 33 active+clean; 5.9 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 45 MiB/s wr, 717 op/s; 6118/291861 objects degraded (2.096%); 6096/291861 objects misplaced (2.089%); 359 KiB/s, 5 objects/s recovering 2024-05-15T13:13:51.676 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:51 smithi027 ceph-mon[39795]: Health check failed: Degraded data redundancy: 6118/291861 objects degraded (2.096%), 1 pg degraded (PG_DEGRADED) 2024-05-15T13:13:51.676 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:51 smithi027 ceph-mon[39795]: osdmap e639: 12 total, 12 up, 10 in 2024-05-15T13:13:51.681 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:51.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:51 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:13:51.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:51 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:13:51.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:51 smithi120 ceph-mon[43961]: pgmap v1663: 37 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+undersized+remapped, 1 peering, 33 active+clean; 5.9 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 45 MiB/s wr, 717 op/s; 6118/291861 objects degraded (2.096%); 6096/291861 objects misplaced (2.089%); 359 KiB/s, 5 objects/s recovering 2024-05-15T13:13:51.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:51 smithi120 ceph-mon[43961]: Health check failed: Degraded data redundancy: 6118/291861 objects degraded (2.096%), 1 pg degraded (PG_DEGRADED) 2024-05-15T13:13:51.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:51 smithi120 ceph-mon[43961]: osdmap e639: 12 total, 12 up, 10 in 2024-05-15T13:13:51.892 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 69 16 99041 99025 89.6703 40.625 0.0129823 0.0111461 2024-05-15T13:13:52.159 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:13:52.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:52.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:13:52.268 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:13:52.302 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:52.327 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:13:52.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:52.328 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:13:52.429 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:13:52.457 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:52.477 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:13:52.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:52.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:13:52.581 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:13:52.710 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:52.774 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:13:52.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:52.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:13:52.892 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 70 16 99622 99606 88.9082 36.3125 0.0099953 0.0112391 2024-05-15T13:13:52.914 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:13:52.952 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:52.987 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:13:52.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:52.990 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:13:53.091 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:13:53.117 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:53.138 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:13:53.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:53.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:13:53.243 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:13:53.270 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:53.291 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:13:53.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:53.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:13:53.397 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:13:53.450 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:53.467 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.2 to 1 (10000) 2024-05-15T13:13:53.477 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:13:53.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:53.480 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:13:53.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:53 smithi027 ceph-mon[39795]: pgmap v1665: 37 pgs: 2 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 1 active+undersized+remapped, 32 active+clean; 6.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 120 MiB/s wr, 1.91k op/s; 6118/293640 objects degraded (2.084%); 12021/293640 objects misplaced (4.094%); 1.7 MiB/s, 27 objects/s recovering 2024-05-15T13:13:53.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:53 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1035538272' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:13:53.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:53 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:13:53.580 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:13:53.607 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:53.627 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:13:53.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:53.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:13:53.731 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:13:53.766 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:53 smithi177 ceph-mon[44303]: pgmap v1665: 37 pgs: 2 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 1 active+undersized+remapped, 32 active+clean; 6.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 120 MiB/s wr, 1.91k op/s; 6118/293640 objects degraded (2.084%); 12021/293640 objects misplaced (4.094%); 1.7 MiB/s, 27 objects/s recovering 2024-05-15T13:13:53.766 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:53 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1035538272' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:13:53.766 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:53 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:13:53.770 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:53.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:53 smithi120 ceph-mon[43961]: pgmap v1665: 37 pgs: 2 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 1 active+undersized+remapped, 32 active+clean; 6.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 120 MiB/s wr, 1.91k op/s; 6118/293640 objects degraded (2.084%); 12021/293640 objects misplaced (4.094%); 1.7 MiB/s, 27 objects/s recovering 2024-05-15T13:13:53.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:53 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1035538272' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:13:53.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:53 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:13:53.806 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:13:53.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:53.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:13:53.892 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 71 16 100237 100221 88.1974 38.4375 0.0141335 0.0113331 2024-05-15T13:13:53.911 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:13:53.998 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:13:53.999 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:54.274 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:13:54.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:54.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:13:54.381 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:13:54.524 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:54.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:54 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T13:13:54.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:54 smithi027 ceph-mon[39795]: osdmap e640: 12 total, 12 up, 10 in 2024-05-15T13:13:54.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:54 smithi027 ceph-mon[39795]: 7.c scrub starts 2024-05-15T13:13:54.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:54 smithi027 ceph-mon[39795]: osdmap e641: 12 total, 12 up, 10 in 2024-05-15T13:13:54.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:54 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T13:13:54.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:54 smithi120 ceph-mon[43961]: osdmap e640: 12 total, 12 up, 10 in 2024-05-15T13:13:54.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:54 smithi120 ceph-mon[43961]: 7.c scrub starts 2024-05-15T13:13:54.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:54 smithi120 ceph-mon[43961]: osdmap e641: 12 total, 12 up, 10 in 2024-05-15T13:13:54.892 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 72 16 100791 100775 87.4535 34.625 0.0255047 0.0114269 2024-05-15T13:13:54.909 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 3 1 2024-05-15T13:13:54.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:54 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T13:13:54.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:54 smithi177 ceph-mon[44303]: osdmap e640: 12 total, 12 up, 10 in 2024-05-15T13:13:54.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:54 smithi177 ceph-mon[44303]: 7.c scrub starts 2024-05-15T13:13:54.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:54 smithi177 ceph-mon[44303]: osdmap e641: 12 total, 12 up, 10 in 2024-05-15T13:13:55.109 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:13:55.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:55.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:13:55.225 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:13:55.252 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:55.278 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:13:55.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:55.281 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:13:55.382 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:13:55.411 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:55.452 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:13:55.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:55.454 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:13:55.479 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:55 smithi027 ceph-mon[39795]: pgmap v1668: 37 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 32 active+clean; 6.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 33 MiB/s wr, 520 op/s; 17992/296799 objects misplaced (6.062%); 3.5 MiB/s, 56 objects/s recovering 2024-05-15T13:13:55.479 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:55 smithi027 ceph-mon[39795]: 7.c scrub ok 2024-05-15T13:13:55.479 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:55 smithi027 ceph-mon[39795]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6118/293640 objects degraded (2.084%), 1 pg degraded) 2024-05-15T13:13:55.479 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:55 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T13:13:55.555 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:13:55.659 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:55.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:55 smithi177 ceph-mon[44303]: pgmap v1668: 37 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 32 active+clean; 6.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 33 MiB/s wr, 520 op/s; 17992/296799 objects misplaced (6.062%); 3.5 MiB/s, 56 objects/s recovering 2024-05-15T13:13:55.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:55 smithi177 ceph-mon[44303]: 7.c scrub ok 2024-05-15T13:13:55.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:55 smithi177 ceph-mon[44303]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6118/293640 objects degraded (2.084%), 1 pg degraded) 2024-05-15T13:13:55.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:55 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T13:13:55.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:55 smithi120 ceph-mon[43961]: pgmap v1668: 37 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 32 active+clean; 6.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 33 MiB/s wr, 520 op/s; 17992/296799 objects misplaced (6.062%); 3.5 MiB/s, 56 objects/s recovering 2024-05-15T13:13:55.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:55 smithi120 ceph-mon[43961]: 7.c scrub ok 2024-05-15T13:13:55.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:55 smithi120 ceph-mon[43961]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6118/293640 objects degraded (2.084%), 1 pg degraded) 2024-05-15T13:13:55.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:55 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T13:13:55.892 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 73 16 101372 101356 86.7531 36.3125 0.0242839 0.0115203 2024-05-15T13:13:56.541 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:13:56.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:56 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1708913502' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:13:56.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:56 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:13:56.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:56 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1708913502' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:13:56.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:56 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:13:56.892 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 74 16 102004 101988 86.1146 39.5 0.0185023 0.0116057 2024-05-15T13:13:56.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:56 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1708913502' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:13:56.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:56 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:13:57.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:57 smithi027 ceph-mon[39795]: pgmap v1669: 37 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 32 active+clean; 6.1 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 29 MiB/s wr, 462 op/s; 17895/300192 objects misplaced (5.961%); 3.3 MiB/s, 53 objects/s recovering 2024-05-15T13:13:57.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:57 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:13:57.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:57 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:13:57.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:57 smithi027 ceph-mon[39795]: osdmap e642: 12 total, 12 up, 10 in 2024-05-15T13:13:57.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:57 smithi120 ceph-mon[43961]: pgmap v1669: 37 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 32 active+clean; 6.1 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 29 MiB/s wr, 462 op/s; 17895/300192 objects misplaced (5.961%); 3.3 MiB/s, 53 objects/s recovering 2024-05-15T13:13:57.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:57 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:13:57.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:57 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:13:57.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:57 smithi120 ceph-mon[43961]: osdmap e642: 12 total, 12 up, 10 in 2024-05-15T13:13:57.892 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 75 16 102621 102605 85.4806 38.5625 0.0194951 0.0116942 2024-05-15T13:13:57.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:57 smithi177 ceph-mon[44303]: pgmap v1669: 37 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 32 active+clean; 6.1 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 29 MiB/s wr, 462 op/s; 17895/300192 objects misplaced (5.961%); 3.3 MiB/s, 53 objects/s recovering 2024-05-15T13:13:57.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:57 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:13:57.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:57 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:13:57.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:57 smithi177 ceph-mon[44303]: osdmap e642: 12 total, 12 up, 10 in 2024-05-15T13:13:58.893 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 76 16 103237 103221 84.8625 38.5 0.0152676 0.0117766 2024-05-15T13:13:59.526 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:13:59.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:59.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:13:59.635 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:13:59.663 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:59.683 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:13:59.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:59.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:13:59.779 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:59 smithi027 ceph-mon[39795]: pgmap v1671: 37 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 32 active+clean; 6.2 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 36 MiB/s wr, 575 op/s; 17762/304026 objects misplaced (5.842%); 3.8 MiB/s, 61 objects/s recovering 2024-05-15T13:13:59.779 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:13:59 smithi027 ceph-mon[39795]: osdmap e643: 12 total, 12 up, 10 in 2024-05-15T13:13:59.789 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:13:59.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:59 smithi177 ceph-mon[44303]: pgmap v1671: 37 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 32 active+clean; 6.2 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 36 MiB/s wr, 575 op/s; 17762/304026 objects misplaced (5.842%); 3.8 MiB/s, 61 objects/s recovering 2024-05-15T13:13:59.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:13:59 smithi177 ceph-mon[44303]: osdmap e643: 12 total, 12 up, 10 in 2024-05-15T13:13:59.819 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:13:59.850 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:13:59.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:13:59.853 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:13:59.893 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 77 16 103826 103810 84.2386 36.8125 0.0206617 0.0118649 2024-05-15T13:13:59.954 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:13:59.979 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:59 smithi120 ceph-mon[43961]: pgmap v1671: 37 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 32 active+clean; 6.2 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 36 MiB/s wr, 575 op/s; 17762/304026 objects misplaced (5.842%); 3.8 MiB/s, 61 objects/s recovering 2024-05-15T13:13:59.979 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:13:59 smithi120 ceph-mon[43961]: osdmap e643: 12 total, 12 up, 10 in 2024-05-15T13:13:59.981 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:00.003 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:14:00.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:00.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:14:00.106 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:14:00.136 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:00.157 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:14:00.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:00.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:14:00.260 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:14:00.294 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:00.325 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:14:00.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:00.331 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:14:00.421 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:14:00.431 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:14:00.460 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:00.484 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:14:00.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:00.488 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:14:00.589 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:14:00.622 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:00.653 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:14:00.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:00.664 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:14:00.765 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:14:00.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:00 smithi027 ceph-mon[39795]: pgmap v1673: 37 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 33 active+clean; 6.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 41 MiB/s wr, 651 op/s; 17507/308679 objects misplaced (5.672%); 4.7 MiB/s, 75 objects/s recovering 2024-05-15T13:14:00.804 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:00.836 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:14:00.840 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:00 smithi177 ceph-mon[44303]: pgmap v1673: 37 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 33 active+clean; 6.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 41 MiB/s wr, 651 op/s; 17507/308679 objects misplaced (5.672%); 4.7 MiB/s, 75 objects/s recovering 2024-05-15T13:14:00.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:00.842 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:14:00.893 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 78 16 104393 104377 83.613 35.4375 0.0124015 0.011955 2024-05-15T13:14:00.942 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:14:01.034 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:00 smithi120 ceph-mon[43961]: pgmap v1673: 37 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 33 active+clean; 6.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 41 MiB/s wr, 651 op/s; 17507/308679 objects misplaced (5.672%); 4.7 MiB/s, 75 objects/s recovering 2024-05-15T13:14:01.049 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:01.088 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:14:01.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:01.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:14:01.096 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:14:01.196 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:14:01.246 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:01.316 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:14:01.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:01.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:14:01.330 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:14:01.330 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T13:14:01.330 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-05-15T13:14:01.330 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-05-15T13:14:01.330 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:14:01.426 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:14:01.453 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:01.476 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:14:01.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:01.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:14:01.581 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:14:01.682 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:01.893 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 79 16 104914 104898 82.9668 32.5625 0.0269 0.0120405 2024-05-15T13:14:02.480 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:14:02.893 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T13:14:02.900277+0000 min lat: 0.00113694 max lat: 2.31193 avg lat: 0.0121418 2024-05-15T13:14:02.893 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:14:02.893 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 80 15 105408 105393 82.3166 30.9375 0.0906403 0.0121418 2024-05-15T13:14:03.272 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:14:03.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:03.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:14:03.378 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:14:03.450 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:03.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:03 smithi177 ceph-mon[44303]: pgmap v1674: 37 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 33 active+clean; 6.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 38 MiB/s wr, 607 op/s; 17351/311406 objects misplaced (5.572%); 5.0 MiB/s, 80 objects/s recovering 2024-05-15T13:14:03.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:03 smithi027 ceph-mon[39795]: pgmap v1674: 37 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 33 active+clean; 6.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 38 MiB/s wr, 607 op/s; 17351/311406 objects misplaced (5.572%); 5.0 MiB/s, 80 objects/s recovering 2024-05-15T13:14:03.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:03 smithi120 ceph-mon[43961]: pgmap v1674: 37 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 33 active+clean; 6.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 38 MiB/s wr, 607 op/s; 17351/311406 objects misplaced (5.572%); 5.0 MiB/s, 80 objects/s recovering 2024-05-15T13:14:03.893 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 81 16 106004 105988 81.7595 37.1875 0.0123657 0.0122251 2024-05-15T13:14:04.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:04 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:14:04.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:04 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:14:04.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:04 smithi177 ceph-mon[44303]: osdmap e644: 12 total, 12 up, 10 in 2024-05-15T13:14:04.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:04 smithi177 ceph-mon[44303]: osdmap e645: 12 total, 12 up, 10 in 2024-05-15T13:14:04.710 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:04 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:14:04.711 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:04 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:14:04.711 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:04 smithi027 ceph-mon[39795]: osdmap e644: 12 total, 12 up, 10 in 2024-05-15T13:14:04.711 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:04 smithi027 ceph-mon[39795]: osdmap e645: 12 total, 12 up, 10 in 2024-05-15T13:14:04.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:04 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:14:04.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:04 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:14:04.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:04 smithi120 ceph-mon[43961]: osdmap e644: 12 total, 12 up, 10 in 2024-05-15T13:14:04.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:04 smithi120 ceph-mon[43961]: osdmap e645: 12 total, 12 up, 10 in 2024-05-15T13:14:04.882 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-05-15T13:14:04.893 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 82 16 106576 106560 81.1984 35.75 0.0199846 0.0123066 2024-05-15T13:14:05.165 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:14:05.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:05.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:14:05.277 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:14:05.378 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:05.502 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:14:05.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:05.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:14:05.514 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:05 smithi027 ceph-mon[39795]: pgmap v1677: 37 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 33 active+clean; 6.4 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 37 MiB/s wr, 589 op/s; 17200/314640 objects misplaced (5.467%); 5.9 MiB/s, 93 objects/s recovering 2024-05-15T13:14:05.609 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:14:05.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:05 smithi177 ceph-mon[44303]: pgmap v1677: 37 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 33 active+clean; 6.4 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 37 MiB/s wr, 589 op/s; 17200/314640 objects misplaced (5.467%); 5.9 MiB/s, 93 objects/s recovering 2024-05-15T13:14:05.700 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:05.764 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:14:05.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:05.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:14:05.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:05 smithi120 ceph-mon[43961]: pgmap v1677: 37 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 33 active+clean; 6.4 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 37 MiB/s wr, 589 op/s; 17200/314640 objects misplaced (5.467%); 5.9 MiB/s, 93 objects/s recovering 2024-05-15T13:14:05.880 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:14:05.893 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 83 16 107125 107109 80.6336 34.3125 0.0124763 0.0123872 2024-05-15T13:14:05.913 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:05.942 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:14:05.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:05.948 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:14:06.049 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:14:06.075 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:06.097 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:14:06.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:06.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:14:06.201 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:14:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:06.373 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:14:06.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:06.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:14:06.480 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:14:06.542 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:06.723 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:14:06.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:06.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:14:06.744 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:14:06.818 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:14:06.827 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:14:06.841 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:14:06.885 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:14:06.893 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 84 16 107639 107623 80.0562 32.125 0.055525 0.0124831 2024-05-15T13:14:06.898 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:06.956 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:14:07.027 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:14:07.046 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:14:07.069 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:14:07.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:07.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:14:07.125 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:14:07.175 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:14:07.208 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:14:07.212 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:14:07.252 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:14:07.297 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:07.297 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:14:07.361 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:14:07.422 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:14:07.485 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.3 to 1 (10000) 2024-05-15T13:14:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:14:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:14:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:14:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:14:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:14:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:14:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:14:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:14:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:14:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:14:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:14:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:14:07.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:07 smithi027 ceph-mon[39795]: pgmap v1678: 37 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 33 active+clean; 6.5 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 35 MiB/s wr, 558 op/s; 17102/317631 objects misplaced (5.384%); 4.3 MiB/s, 68 objects/s recovering 2024-05-15T13:14:07.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:07 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3170485944' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:14:07.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:07 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1738200798' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:14:07.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:07 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:14:07.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:07 smithi120 ceph-mon[43961]: pgmap v1678: 37 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 33 active+clean; 6.5 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 35 MiB/s wr, 558 op/s; 17102/317631 objects misplaced (5.384%); 4.3 MiB/s, 68 objects/s recovering 2024-05-15T13:14:07.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:07 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3170485944' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:14:07.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:07 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1738200798' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:14:07.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:07 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:14:07.893 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 85 16 107949 107933 79.3425 19.375 0.00475587 0.0125389 2024-05-15T13:14:07.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:07 smithi177 ceph-mon[44303]: pgmap v1678: 37 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 33 active+clean; 6.5 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 35 MiB/s wr, 558 op/s; 17102/317631 objects misplaced (5.384%); 4.3 MiB/s, 68 objects/s recovering 2024-05-15T13:14:07.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:07 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3170485944' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:14:07.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:07 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1738200798' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:14:07.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:07 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:14:08.486 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:14:08.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:08.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:14:08.491 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:08 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T13:14:08.491 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:08 smithi027 ceph-mon[39795]: osdmap e646: 12 total, 12 up, 10 in 2024-05-15T13:14:08.593 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:14:08.619 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:08.638 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:14:08.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:08.642 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:14:08.742 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:14:08.766 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:08 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T13:14:08.766 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:08 smithi177 ceph-mon[44303]: osdmap e646: 12 total, 12 up, 10 in 2024-05-15T13:14:08.768 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:08.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:08 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T13:14:08.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:08 smithi120 ceph-mon[43961]: osdmap e646: 12 total, 12 up, 10 in 2024-05-15T13:14:08.787 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:14:08.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:08.790 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:14:08.892 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:14:08.894 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 86 16 107949 107933 78.42 0 - 0.0125389 2024-05-15T13:14:08.920 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:08.938 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:14:08.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:08.941 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:14:09.042 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:14:09.069 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:09.089 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:14:09.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:09.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:14:09.192 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:14:09.225 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:09.458 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:14:09.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:09.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:14:09.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:09 smithi027 ceph-mon[39795]: pgmap v1680: 37 pgs: 1 peering, 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 32 active+clean; 6.5 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 34 MiB/s wr, 538 op/s; 16957/321111 objects misplaced (5.281%); 4.1 MiB/s, 65 objects/s recovering 2024-05-15T13:14:09.522 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:09 smithi027 ceph-mon[39795]: osdmap e647: 12 total, 12 up, 10 in 2024-05-15T13:14:09.565 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:14:09.592 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:09.612 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:14:09.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:09.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:14:09.716 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:14:09.741 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:09 smithi120 ceph-mon[43961]: pgmap v1680: 37 pgs: 1 peering, 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 32 active+clean; 6.5 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 34 MiB/s wr, 538 op/s; 16957/321111 objects misplaced (5.281%); 4.1 MiB/s, 65 objects/s recovering 2024-05-15T13:14:09.741 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:09 smithi120 ceph-mon[43961]: osdmap e647: 12 total, 12 up, 10 in 2024-05-15T13:14:09.743 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:09.762 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:14:09.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:09.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:14:09.867 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:14:09.894 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 87 16 107949 107933 77.5187 0 - 0.0125389 2024-05-15T13:14:09.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:09 smithi177 ceph-mon[44303]: pgmap v1680: 37 pgs: 1 peering, 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 32 active+clean; 6.5 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 34 MiB/s wr, 538 op/s; 16957/321111 objects misplaced (5.281%); 4.1 MiB/s, 65 objects/s recovering 2024-05-15T13:14:09.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:09 smithi177 ceph-mon[44303]: osdmap e647: 12 total, 12 up, 10 in 2024-05-15T13:14:09.935 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:10.123 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:14:10.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:10.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:14:10.245 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:14:10.271 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:10.291 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:14:10.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:10.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:14:10.396 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:14:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:10.534 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:10 smithi120 ceph-mon[43961]: osdmap e648: 12 total, 12 up, 10 in 2024-05-15T13:14:10.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:10 smithi027 ceph-mon[39795]: osdmap e648: 12 total, 12 up, 10 in 2024-05-15T13:14:10.894 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 88 16 108283 108267 76.8751 6.95833 0.0989609 0.0129975 2024-05-15T13:14:10.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:10 smithi177 ceph-mon[44303]: osdmap e648: 12 total, 12 up, 10 in 2024-05-15T13:14:11.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:11 smithi027 ceph-mon[39795]: pgmap v1683: 37 pgs: 2 remapped+peering, 1 peering, 1 active+clean+scrubbing+deep, 4 active+recovering+undersized+remapped, 29 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 31 MiB/s wr, 493 op/s; 23273/323526 objects misplaced (7.194%); 6.5 MiB/s, 103 objects/s recovering 2024-05-15T13:14:11.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:11 smithi027 ceph-mon[39795]: osdmap e649: 12 total, 12 up, 10 in 2024-05-15T13:14:11.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:11 smithi120 ceph-mon[43961]: pgmap v1683: 37 pgs: 2 remapped+peering, 1 peering, 1 active+clean+scrubbing+deep, 4 active+recovering+undersized+remapped, 29 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 31 MiB/s wr, 493 op/s; 23273/323526 objects misplaced (7.194%); 6.5 MiB/s, 103 objects/s recovering 2024-05-15T13:14:11.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:11 smithi120 ceph-mon[43961]: osdmap e649: 12 total, 12 up, 10 in 2024-05-15T13:14:11.894 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 89 16 108649 108633 76.2684 22.875 0.0216982 0.013099 2024-05-15T13:14:11.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:11 smithi177 ceph-mon[44303]: pgmap v1683: 37 pgs: 2 remapped+peering, 1 peering, 1 active+clean+scrubbing+deep, 4 active+recovering+undersized+remapped, 29 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 31 MiB/s wr, 493 op/s; 23273/323526 objects misplaced (7.194%); 6.5 MiB/s, 103 objects/s recovering 2024-05-15T13:14:11.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:11 smithi177 ceph-mon[44303]: osdmap e649: 12 total, 12 up, 10 in 2024-05-15T13:14:12.774 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:12 smithi027 ceph-mon[39795]: osdmap e650: 12 total, 12 up, 10 in 2024-05-15T13:14:12.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:12 smithi120 ceph-mon[43961]: osdmap e650: 12 total, 12 up, 10 in 2024-05-15T13:14:12.860 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:14:12.894 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 90 9 109062 109053 75.7128 26.25 0.0253252 0.0131992 2024-05-15T13:14:12.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:12 smithi177 ceph-mon[44303]: osdmap e650: 12 total, 12 up, 10 in 2024-05-15T13:14:13.519 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:13 smithi027 ceph-mon[39795]: pgmap v1686: 37 pgs: 2 remapped+peering, 1 peering, 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 29 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 18 MiB/s wr, 290 op/s; 12190/324603 objects degraded (3.755%); 17782/324603 objects misplaced (5.478%); 6.2 MiB/s, 98 objects/s recovering 2024-05-15T13:14:13.519 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:13 smithi027 ceph-mon[39795]: Health check failed: Degraded data redundancy: 12190/324603 objects degraded (3.755%), 1 pg degraded (PG_DEGRADED) 2024-05-15T13:14:13.689 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:14:13.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:13.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:14:13.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:13 smithi120 ceph-mon[43961]: pgmap v1686: 37 pgs: 2 remapped+peering, 1 peering, 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 29 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 18 MiB/s wr, 290 op/s; 12190/324603 objects degraded (3.755%); 17782/324603 objects misplaced (5.478%); 6.2 MiB/s, 98 objects/s recovering 2024-05-15T13:14:13.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:13 smithi120 ceph-mon[43961]: Health check failed: Degraded data redundancy: 12190/324603 objects degraded (3.755%), 1 pg degraded (PG_DEGRADED) 2024-05-15T13:14:13.796 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:14:13.823 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:13.845 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:14:13.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:13.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:14:13.894 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Total time run: 90.1664 2024-05-15T13:14:13.894 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Total writes made: 109062 2024-05-15T13:14:13.894 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Write size: 65536 2024-05-15T13:14:13.894 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Object size: 65536 2024-05-15T13:14:13.894 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Bandwidth (MB/sec): 75.5977 2024-05-15T13:14:13.895 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Stddev Bandwidth: 42.6615 2024-05-15T13:14:13.895 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Max bandwidth (MB/sec): 197.688 2024-05-15T13:14:13.895 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Min bandwidth (MB/sec): 0 2024-05-15T13:14:13.895 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Average IOPS: 1209 2024-05-15T13:14:13.895 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Stddev IOPS: 682.59 2024-05-15T13:14:13.895 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Max IOPS: 3163 2024-05-15T13:14:13.895 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Min IOPS: 0 2024-05-15T13:14:13.895 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Average Latency(s): 0.0132072 2024-05-15T13:14:13.895 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Stddev Latency(s): 0.0442865 2024-05-15T13:14:13.895 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Max latency(s): 2.58951 2024-05-15T13:14:13.895 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Min latency(s): 0.00113694 2024-05-15T13:14:13.914 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Cleaning up (deleting benchmark objects) 2024-05-15T13:14:13.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:13 smithi177 ceph-mon[44303]: pgmap v1686: 37 pgs: 2 remapped+peering, 1 peering, 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 29 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 18 MiB/s wr, 290 op/s; 12190/324603 objects degraded (3.755%); 17782/324603 objects misplaced (5.478%); 6.2 MiB/s, 98 objects/s recovering 2024-05-15T13:14:13.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:13 smithi177 ceph-mon[44303]: Health check failed: Degraded data redundancy: 12190/324603 objects degraded (3.755%), 1 pg degraded (PG_DEGRADED) 2024-05-15T13:14:13.948 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:14:13.974 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:13.994 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:14:13.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:13.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:14:14.099 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:14:14.128 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:14.152 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:14:14.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:14.155 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:14:14.256 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:14:14.283 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:14.306 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:14:14.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:14.310 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:14:14.331 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 5 1 2024-05-15T13:14:14.412 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:14:14.441 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:14.462 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:14:14.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:14.465 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:14:14.566 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:14:14.594 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:14.616 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:14:14.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:14.619 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:14:14.720 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:14:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:14.948 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:14:14.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:14.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:14:15.057 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:14:15.159 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:15.183 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:14 smithi177 ceph-mon[44303]: pgmap v1687: 37 pgs: 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 30 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 17 MiB/s wr, 270 op/s; 12190/325821 objects degraded (3.741%); 30990/325821 objects misplaced (9.511%); 8.7 MiB/s, 139 objects/s recovering 2024-05-15T13:14:15.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:14 smithi027 ceph-mon[39795]: pgmap v1687: 37 pgs: 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 30 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 17 MiB/s wr, 270 op/s; 12190/325821 objects degraded (3.741%); 30990/325821 objects misplaced (9.511%); 8.7 MiB/s, 139 objects/s recovering 2024-05-15T13:14:15.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:14 smithi120 ceph-mon[43961]: pgmap v1687: 37 pgs: 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 30 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 17 MiB/s wr, 270 op/s; 12190/325821 objects degraded (3.741%); 30990/325821 objects misplaced (9.511%); 8.7 MiB/s, 139 objects/s recovering 2024-05-15T13:14:16.276 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:14:16.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:16.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:14:16.389 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:14:16.416 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:16.436 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:14:16.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:16.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:14:16.541 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:14:16.568 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:16.592 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:14:16.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:16.596 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:14:16.697 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:14:16.822 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:16.894 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:14:16.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:16.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:14:17.001 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:14:17.027 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:17.049 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:14:17.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:17.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:14:17.110 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:14:17.152 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:14:17.182 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:17.201 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:14:17.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:17.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:14:17.305 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:14:17.399 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:17.484 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:14:17.626 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:14:17.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:17.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:14:17.662 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:14:17.663 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:14:17.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:17 smithi177 ceph-mon[44303]: pgmap v1688: 37 pgs: 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 30 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 14 MiB/s wr, 257 op/s; 12190/326250 objects degraded (3.736%); 30752/326250 objects misplaced (9.426%); 7.8 MiB/s, 124 objects/s recovering 2024-05-15T13:14:17.704 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:17 smithi027 ceph-mon[39795]: pgmap v1688: 37 pgs: 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 30 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 14 MiB/s wr, 257 op/s; 12190/326250 objects degraded (3.736%); 30752/326250 objects misplaced (9.426%); 7.8 MiB/s, 124 objects/s recovering 2024-05-15T13:14:17.739 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:14:17.764 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:17 smithi120 ceph-mon[43961]: pgmap v1688: 37 pgs: 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 30 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 14 MiB/s wr, 257 op/s; 12190/326250 objects degraded (3.736%); 30752/326250 objects misplaced (9.426%); 7.8 MiB/s, 124 objects/s recovering 2024-05-15T13:14:17.766 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:17.785 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:14:17.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:17.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:14:17.889 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:14:17.918 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:17.939 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:14:17.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:17.942 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:14:18.043 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:14:18.143 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:18.229 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:14:18.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:18.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:14:18.343 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:14:18.436 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:14:18.477 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:18.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:18 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3887466761' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:14:18.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:18 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:14:18.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:18 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3887466761' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:14:18.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:18 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:14:18.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:18 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3887466761' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:14:18.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:18 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:14:19.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:19 smithi027 ceph-mon[39795]: pgmap v1689: 37 pgs: 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 30 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 127 B/s rd, 9.5 MiB/s wr, 258 op/s; 12190/324651 objects degraded (3.755%); 30656/324651 objects misplaced (9.443%); 6.0 MiB/s, 96 objects/s recovering 2024-05-15T13:14:19.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:19 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:14:19.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:19 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 12190/324651 objects degraded (3.755%), 1 pg degraded (PG_DEGRADED) 2024-05-15T13:14:19.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:19 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:14:19.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:19 smithi027 ceph-mon[39795]: osdmap e651: 12 total, 12 up, 10 in 2024-05-15T13:14:19.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:19 smithi120 ceph-mon[43961]: pgmap v1689: 37 pgs: 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 30 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 127 B/s rd, 9.5 MiB/s wr, 258 op/s; 12190/324651 objects degraded (3.755%); 30656/324651 objects misplaced (9.443%); 6.0 MiB/s, 96 objects/s recovering 2024-05-15T13:14:19.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:19 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:14:19.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:19 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 12190/324651 objects degraded (3.755%), 1 pg degraded (PG_DEGRADED) 2024-05-15T13:14:19.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:19 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:14:19.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:19 smithi120 ceph-mon[43961]: osdmap e651: 12 total, 12 up, 10 in 2024-05-15T13:14:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:19 smithi177 ceph-mon[44303]: pgmap v1689: 37 pgs: 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 30 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 127 B/s rd, 9.5 MiB/s wr, 258 op/s; 12190/324651 objects degraded (3.755%); 30656/324651 objects misplaced (9.443%); 6.0 MiB/s, 96 objects/s recovering 2024-05-15T13:14:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:19 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:14:19.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:19 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 12190/324651 objects degraded (3.755%), 1 pg degraded (PG_DEGRADED) 2024-05-15T13:14:19.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:19 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:14:19.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:19 smithi177 ceph-mon[44303]: osdmap e651: 12 total, 12 up, 10 in 2024-05-15T13:14:21.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:21 smithi027 ceph-mon[39795]: pgmap v1691: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 30 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 116 B/s rd, 8.2 MiB/s wr, 281 op/s; 12204/323217 objects degraded (3.776%); 30357/323217 objects misplaced (9.392%); 7.5 MiB/s, 119 objects/s recovering 2024-05-15T13:14:21.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:21 smithi120 ceph-mon[43961]: pgmap v1691: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 30 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 116 B/s rd, 8.2 MiB/s wr, 281 op/s; 12204/323217 objects degraded (3.776%); 30357/323217 objects misplaced (9.392%); 7.5 MiB/s, 119 objects/s recovering 2024-05-15T13:14:21.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:21 smithi177 ceph-mon[44303]: pgmap v1691: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 30 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 116 B/s rd, 8.2 MiB/s wr, 281 op/s; 12204/323217 objects degraded (3.776%); 30357/323217 objects misplaced (9.392%); 7.5 MiB/s, 119 objects/s recovering 2024-05-15T13:14:22.434 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:14:22.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:22.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:14:22.541 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:14:22.569 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:22.590 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:14:22.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:22.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:14:22.694 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:14:22.762 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:23.246 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:14:23.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:23.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:14:23.371 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:14:23.399 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:23.420 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:14:23.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:23.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:14:23.524 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:14:23.525 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:23 smithi027 ceph-mon[39795]: pgmap v1692: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 31 active+clean; 6.5 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 102 B/s rd, 5.4 MiB/s wr, 346 op/s; 12204/319374 objects degraded (3.821%); 30190/319374 objects misplaced (9.453%); 7.5 MiB/s, 120 objects/s recovering 2024-05-15T13:14:23.552 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:23.571 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:14:23.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:23.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:14:23.676 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:14:23.702 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:23 smithi120 ceph-mon[43961]: pgmap v1692: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 31 active+clean; 6.5 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 102 B/s rd, 5.4 MiB/s wr, 346 op/s; 12204/319374 objects degraded (3.821%); 30190/319374 objects misplaced (9.453%); 7.5 MiB/s, 120 objects/s recovering 2024-05-15T13:14:23.704 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:23.725 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:14:23.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:23.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:14:23.830 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:14:23.844 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-05-15T13:14:23.857 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:23.878 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:14:23.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:23.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:14:23.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:23 smithi177 ceph-mon[44303]: pgmap v1692: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 31 active+clean; 6.5 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 102 B/s rd, 5.4 MiB/s wr, 346 op/s; 12204/319374 objects degraded (3.821%); 30190/319374 objects misplaced (9.453%); 7.5 MiB/s, 120 objects/s recovering 2024-05-15T13:14:23.982 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:14:24.085 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:24.473 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:24 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 12204/319374 objects degraded (3.821%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T13:14:24.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:24 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 12204/319374 objects degraded (3.821%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T13:14:24.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:24 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 12204/319374 objects degraded (3.821%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T13:14:25.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:25 smithi027 ceph-mon[39795]: pgmap v1693: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 31 active+clean; 6.4 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 102 B/s rd, 2.8 MiB/s wr, 399 op/s; 12246/316530 objects degraded (3.869%); 29749/316530 objects misplaced (9.398%); 7.4 MiB/s, 118 objects/s recovering 2024-05-15T13:14:25.784 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:25 smithi120 ceph-mon[43961]: pgmap v1693: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 31 active+clean; 6.4 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 102 B/s rd, 2.8 MiB/s wr, 399 op/s; 12246/316530 objects degraded (3.869%); 29749/316530 objects misplaced (9.398%); 7.4 MiB/s, 118 objects/s recovering 2024-05-15T13:14:25.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:25 smithi177 ceph-mon[44303]: pgmap v1693: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 31 active+clean; 6.4 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 102 B/s rd, 2.8 MiB/s wr, 399 op/s; 12246/316530 objects degraded (3.869%); 29749/316530 objects misplaced (9.398%); 7.4 MiB/s, 118 objects/s recovering 2024-05-15T13:14:26.360 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:14:26.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:26.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:14:26.479 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:14:26.573 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:26.623 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:14:26.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:26.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:14:26.731 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:14:26.759 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:26.779 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:14:26.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:26.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:14:26.883 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:14:26.982 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:27.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:27 smithi027 ceph-mon[39795]: pgmap v1694: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 6.4 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 102 B/s rd, 633 KiB/s wr, 436 op/s; 12315/313761 objects degraded (3.925%); 29499/313761 objects misplaced (9.402%); 7.0 MiB/s, 112 objects/s recovering 2024-05-15T13:14:27.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:27 smithi120 ceph-mon[43961]: pgmap v1694: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 6.4 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 102 B/s rd, 633 KiB/s wr, 436 op/s; 12315/313761 objects degraded (3.925%); 29499/313761 objects misplaced (9.402%); 7.0 MiB/s, 112 objects/s recovering 2024-05-15T13:14:27.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:27 smithi177 ceph-mon[44303]: pgmap v1694: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 6.4 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 102 B/s rd, 633 KiB/s wr, 436 op/s; 12315/313761 objects degraded (3.925%); 29499/313761 objects misplaced (9.402%); 7.0 MiB/s, 112 objects/s recovering 2024-05-15T13:14:28.409 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:14:29.285 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:14:29.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:29.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:14:29.401 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:14:29.430 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:29.449 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:14:29.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:29.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:14:29.554 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:14:29.558 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:14:29.575 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:14:29.582 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:29.601 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:14:29.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:29.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:14:29.614 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:29 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:14:29.614 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:29 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:14:29.615 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:29 smithi027 ceph-mon[39795]: pgmap v1695: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 6.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 460 op/s; 12381/310815 objects degraded (3.983%); 29343/310815 objects misplaced (9.441%); 7.0 MiB/s, 112 objects/s recovering 2024-05-15T13:14:29.615 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:29 smithi027 ceph-mon[39795]: osdmap e652: 12 total, 12 up, 10 in 2024-05-15T13:14:29.615 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:29 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 12381/310815 objects degraded (3.983%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:14:29.615 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:29 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2939686859' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T13:14:29.615 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:29 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T13:14:29.620 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.5 to 1 (10000) 2024-05-15T13:14:29.706 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:14:29.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:29 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:14:29.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:29 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:14:29.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:29 smithi120 ceph-mon[43961]: pgmap v1695: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 6.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 460 op/s; 12381/310815 objects degraded (3.983%); 29343/310815 objects misplaced (9.441%); 7.0 MiB/s, 112 objects/s recovering 2024-05-15T13:14:29.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:29 smithi120 ceph-mon[43961]: osdmap e652: 12 total, 12 up, 10 in 2024-05-15T13:14:29.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:29 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 12381/310815 objects degraded (3.983%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:14:29.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:29 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2939686859' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T13:14:29.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:29 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T13:14:29.803 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:29.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:29 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:14:29.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:29 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:14:29.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:29 smithi177 ceph-mon[44303]: pgmap v1695: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 6.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 460 op/s; 12381/310815 objects degraded (3.983%); 29343/310815 objects misplaced (9.441%); 7.0 MiB/s, 112 objects/s recovering 2024-05-15T13:14:29.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:29 smithi177 ceph-mon[44303]: osdmap e652: 12 total, 12 up, 10 in 2024-05-15T13:14:29.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:29 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 12381/310815 objects degraded (3.983%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:14:29.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:29 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2939686859' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T13:14:29.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:29 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T13:14:30.626 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:30 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T13:14:30.626 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:30 smithi027 ceph-mon[39795]: osdmap e653: 12 total, 12 up, 10 in 2024-05-15T13:14:30.627 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:30 smithi027 ceph-mon[39795]: pgmap v1698: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 6.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 431 op/s; 12400/309015 objects degraded (4.013%); 29015/309015 objects misplaced (9.390%); 7.6 MiB/s, 122 objects/s recovering 2024-05-15T13:14:30.782 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:14:30.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:30.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:14:30.889 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:14:30.914 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:30 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T13:14:30.914 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:30 smithi177 ceph-mon[44303]: osdmap e653: 12 total, 12 up, 10 in 2024-05-15T13:14:30.914 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:30 smithi177 ceph-mon[44303]: pgmap v1698: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 6.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 431 op/s; 12400/309015 objects degraded (4.013%); 29015/309015 objects misplaced (9.390%); 7.6 MiB/s, 122 objects/s recovering 2024-05-15T13:14:30.916 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:30.935 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:14:30.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:30.941 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:14:31.035 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:30 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T13:14:31.035 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:30 smithi120 ceph-mon[43961]: osdmap e653: 12 total, 12 up, 10 in 2024-05-15T13:14:31.035 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:30 smithi120 ceph-mon[43961]: pgmap v1698: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 6.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 431 op/s; 12400/309015 objects degraded (4.013%); 29015/309015 objects misplaced (9.390%); 7.6 MiB/s, 122 objects/s recovering 2024-05-15T13:14:31.042 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:14:31.069 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:31.092 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:14:31.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:31.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:14:31.196 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:14:31.225 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:31.249 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:14:31.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:31.253 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:14:31.354 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:14:31.447 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:31.683 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:14:31.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:31.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:14:31.800 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:14:31.829 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:31.848 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:14:31.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:31.851 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:14:31.951 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:14:31.979 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:32.000 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:14:32.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:32.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:14:32.104 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:14:32.131 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:32.153 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:14:32.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:32.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:14:32.257 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:14:32.288 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:32.312 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:14:32.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:32.315 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:14:32.417 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:14:32.538 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:33.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:33 smithi177 ceph-mon[44303]: pgmap v1699: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 6.2 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 473 op/s; 12467/305160 objects degraded (4.085%); 28755/305160 objects misplaced (9.423%); 6.0 MiB/s, 96 objects/s recovering 2024-05-15T13:14:33.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:33 smithi027 ceph-mon[39795]: pgmap v1699: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 6.2 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 473 op/s; 12467/305160 objects degraded (4.085%); 28755/305160 objects misplaced (9.423%); 6.0 MiB/s, 96 objects/s recovering 2024-05-15T13:14:33.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:33 smithi120 ceph-mon[43961]: pgmap v1699: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 6.2 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 473 op/s; 12467/305160 objects degraded (4.085%); 28755/305160 objects misplaced (9.423%); 6.0 MiB/s, 96 objects/s recovering 2024-05-15T13:14:33.837 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:14:33.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:33.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:14:33.942 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:14:34.031 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:34.088 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:14:34.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:34.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:14:34.225 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:14:34.254 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:34.273 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:14:34.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:34.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:14:34.377 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:14:34.409 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:34.431 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:14:34.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:34.437 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:14:34.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:34 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 12467/305160 objects degraded (4.085%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:14:34.537 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:14:34.564 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:34.586 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:14:34.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:34.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:14:34.673 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:34 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 12467/305160 objects degraded (4.085%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:14:34.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:34 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 12467/305160 objects degraded (4.085%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:14:34.688 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:14:34.715 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:34.735 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:14:34.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:34.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:14:34.839 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:14:34.913 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:35.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:35 smithi177 ceph-mon[44303]: pgmap v1700: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 6.2 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 470 op/s; 12539/302460 objects degraded (4.146%); 28273/302460 objects misplaced (9.348%); 7.8 MiB/s, 125 objects/s recovering 2024-05-15T13:14:35.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:35 smithi027 ceph-mon[39795]: pgmap v1700: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 6.2 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 470 op/s; 12539/302460 objects degraded (4.146%); 28273/302460 objects misplaced (9.348%); 7.8 MiB/s, 125 objects/s recovering 2024-05-15T13:14:35.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:35 smithi120 ceph-mon[43961]: pgmap v1700: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 6.2 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 470 op/s; 12539/302460 objects degraded (4.146%); 28273/302460 objects misplaced (9.348%); 7.8 MiB/s, 125 objects/s recovering 2024-05-15T13:14:35.906 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 6 1 2024-05-15T13:14:36.476 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:14:36.481 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:14:36.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:36.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:14:36.535 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:14:36.584 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:14:36.593 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:14:36.626 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:36 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2660653641' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:14:36.650 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:14:36.688 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:14:36.694 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:36.749 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:14:36.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:36 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2660653641' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:14:36.801 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:14:36.900 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:14:36.902 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:14:36.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:36 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2660653641' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:14:36.950 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:14:37.004 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:14:37.043 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:14:37.090 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:14:37.090 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:14:37.090 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:14:37.090 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:14:37.090 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:14:37.090 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:14:37.090 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:14:37.090 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:14:37.091 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:14:37.091 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:14:37.091 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:14:37.091 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:14:37.779 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:37 smithi027 ceph-mon[39795]: pgmap v1701: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 6.1 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 462 op/s; 12599/299703 objects degraded (4.204%); 28027/299703 objects misplaced (9.352%); 8.5 MiB/s, 137 objects/s recovering 2024-05-15T13:14:37.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:37 smithi120 ceph-mon[43961]: pgmap v1701: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 6.1 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 462 op/s; 12599/299703 objects degraded (4.204%); 28027/299703 objects misplaced (9.352%); 8.5 MiB/s, 137 objects/s recovering 2024-05-15T13:14:37.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:37 smithi177 ceph-mon[44303]: pgmap v1701: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 6.1 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 462 op/s; 12599/299703 objects degraded (4.204%); 28027/299703 objects misplaced (9.352%); 8.5 MiB/s, 137 objects/s recovering 2024-05-15T13:14:38.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:38 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:14:38.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:38 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:14:38.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:38 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1715017691' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:14:38.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:38 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:14:38.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:38 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:14:38.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:38 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:14:38.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:38 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1715017691' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:14:38.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:38 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:14:38.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:38 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:14:38.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:38 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:14:38.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:38 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1715017691' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:14:38.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:38 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:14:39.167 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:14:39.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:39.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:14:39.277 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:14:39.303 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:39.324 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:14:39.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:39.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:14:39.428 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:14:39.455 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:39.480 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:14:39.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:39.484 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:14:39.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:39 smithi027 ceph-mon[39795]: pgmap v1702: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 6.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 477 op/s; 12385/296778 objects degraded (4.173%); 27904/296778 objects misplaced (9.402%); 8.3 MiB/s, 133 objects/s recovering 2024-05-15T13:14:39.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:39 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:14:39.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:39 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:14:39.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:39 smithi027 ceph-mon[39795]: osdmap e654: 12 total, 12 up, 10 in 2024-05-15T13:14:39.524 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:39 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 12385/296778 objects degraded (4.173%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:14:39.585 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:14:39.671 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:39.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:39 smithi120 ceph-mon[43961]: pgmap v1702: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 6.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 477 op/s; 12385/296778 objects degraded (4.173%); 27904/296778 objects misplaced (9.402%); 8.3 MiB/s, 133 objects/s recovering 2024-05-15T13:14:39.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:39 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:14:39.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:39 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:14:39.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:39 smithi120 ceph-mon[43961]: osdmap e654: 12 total, 12 up, 10 in 2024-05-15T13:14:39.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:39 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 12385/296778 objects degraded (4.173%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:14:39.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:39 smithi177 ceph-mon[44303]: pgmap v1702: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 6.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 477 op/s; 12385/296778 objects degraded (4.173%); 27904/296778 objects misplaced (9.402%); 8.3 MiB/s, 133 objects/s recovering 2024-05-15T13:14:39.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:39 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:14:39.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:39 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:14:39.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:39 smithi177 ceph-mon[44303]: osdmap e654: 12 total, 12 up, 10 in 2024-05-15T13:14:39.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:39 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 12385/296778 objects degraded (4.173%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:14:40.030 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:14:40.256 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:14:40.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:40.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:14:40.366 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:14:40.393 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:40.415 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:14:40.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:40.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:14:40.519 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:14:40.601 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:40.778 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:14:40.779 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:14:40.779 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.779 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.26044.0:120799 7.6 e654/647 7:61ad5511:::benchmark_data_smithi177_134484_object11734:head v 654'7522, mlcod=608'754)", 2024-05-15T13:14:40.779 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:14:39.550227+0000", 2024-05-15T13:14:40.779 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.031993331999999999, 2024-05-15T13:14:40.779 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.032023296999999999, 2024-05-15T13:14:40.779 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:14:40.779 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:14:40.779 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:14:40.779 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.779 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:14:40.780 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.550227+0000", 2024-05-15T13:14:40.780 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:14:40.780 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.780 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.780 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:14:40.780 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.550227+0000", 2024-05-15T13:14:40.780 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:14:40.780 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.780 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.780 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:14:40.780 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.550228+0000", 2024-05-15T13:14:40.780 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.65e-07 2024-05-15T13:14:40.781 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.781 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.781 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:14:40.781 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.550232+0000", 2024-05-15T13:14:40.781 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3030000000000001e-06 2024-05-15T13:14:40.781 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.781 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.781 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:14:40.781 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.550238+0000", 2024-05-15T13:14:40.781 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5799999999999999e-06 2024-05-15T13:14:40.781 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.781 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.781 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:14:40.782 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.550242+0000", 2024-05-15T13:14:40.782 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7430000000000002e-06 2024-05-15T13:14:40.782 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.782 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.782 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:14:40.782 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.578316+0000", 2024-05-15T13:14:40.782 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.028073411999999999 2024-05-15T13:14:40.782 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.782 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.782 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:14:40.782 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.578344+0000", 2024-05-15T13:14:40.782 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8501999999999999e-05 2024-05-15T13:14:40.783 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:14:40.783 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:14:40.783 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:14:40.783 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.783 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.783 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:120808 7.3 7:cd57a53b:::benchmark_data_smithi177_134484_object11743:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e654)", 2024-05-15T13:14:40.783 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:14:39.566149+0000", 2024-05-15T13:14:40.783 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.016071707000000001, 2024-05-15T13:14:40.783 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016173719999999999, 2024-05-15T13:14:40.783 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:14:40.783 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:14:40.783 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:14:40.784 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:14:40.784 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:14:40.784 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 120808 2024-05-15T13:14:40.784 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.784 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:14:40.784 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.784 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:14:40.784 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.566149+0000", 2024-05-15T13:14:40.784 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:14:40.784 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.784 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.784 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:14:40.785 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.566149+0000", 2024-05-15T13:14:40.785 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:14:40.785 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.785 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.785 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:14:40.785 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.566151+0000", 2024-05-15T13:14:40.785 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9080000000000002e-06 2024-05-15T13:14:40.785 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.785 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.785 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:14:40.785 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.566160+0000", 2024-05-15T13:14:40.785 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.833e-06 2024-05-15T13:14:40.785 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.786 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.786 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:14:40.786 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.566176+0000", 2024-05-15T13:14:40.786 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6155999999999999e-05 2024-05-15T13:14:40.786 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.786 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.786 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:14:40.786 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.566187+0000", 2024-05-15T13:14:40.786 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0577000000000001e-05 2024-05-15T13:14:40.786 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.786 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.786 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:14:40.786 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.566279+0000", 2024-05-15T13:14:40.786 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1928999999999993e-05 2024-05-15T13:14:40.787 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.787 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.787 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:14:40.787 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.566405+0000", 2024-05-15T13:14:40.787 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012687700000000001 2024-05-15T13:14:40.787 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.787 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.787 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2,11", 2024-05-15T13:14:40.787 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.566521+0000", 2024-05-15T13:14:40.787 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011595300000000001 2024-05-15T13:14:40.787 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.787 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.787 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:14:40.787 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.581397+0000", 2024-05-15T13:14:40.788 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014875746 2024-05-15T13:14:40.788 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:14:40.788 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:14:40.788 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:14:40.788 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.788 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.788 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(7.6 654/647 [PushOp(7:60f0b7ab:::benchmark_data_smithi177_134484_object11756:head, version: 608'758, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:60f0b7ab:::benchmark_data_smithi177_134484_object11756:head@608'758, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-05-15T13:14:40.788 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:14:39.568149+0000", 2024-05-15T13:14:40.788 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.014072206, 2024-05-15T13:14:40.788 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014303120000000001, 2024-05-15T13:14:40.788 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:14:40.788 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "queued for pg", 2024-05-15T13:14:40.788 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:14:40.788 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.788 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:14:40.789 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.568149+0000", 2024-05-15T13:14:40.789 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:14:40.789 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.789 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.789 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:14:40.789 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.568149+0000", 2024-05-15T13:14:40.789 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:14:40.789 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.789 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.789 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:14:40.789 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.568150+0000", 2024-05-15T13:14:40.789 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2929999999999999e-06 2024-05-15T13:14:40.789 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.789 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.789 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:14:40.789 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.568186+0000", 2024-05-15T13:14:40.790 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5914000000000002e-05 2024-05-15T13:14:40.790 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.790 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.790 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:14:40.790 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.568207+0000", 2024-05-15T13:14:40.790 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1219000000000001e-05 2024-05-15T13:14:40.790 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.790 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.790 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:14:40.790 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.568223+0000", 2024-05-15T13:14:40.790 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6220000000000001e-05 2024-05-15T13:14:40.790 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:14:40.790 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:14:40.791 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:14:40.791 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.791 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.791 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.26044.0:120780 7.8 e654/647 7:1511a8c1:::benchmark_data_smithi177_134484_object11715:head v 654'7547, mlcod=608'726)", 2024-05-15T13:14:40.791 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:14:39.571123+0000", 2024-05-15T13:14:40.791 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.011098291, 2024-05-15T13:14:40.791 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011379210000000001, 2024-05-15T13:14:40.791 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:14:40.791 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:14:40.791 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:14:40.791 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.791 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:14:40.791 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.571123+0000", 2024-05-15T13:14:40.791 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:14:40.791 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.791 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.792 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:14:40.792 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.571123+0000", 2024-05-15T13:14:40.792 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:14:40.792 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.792 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.792 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:14:40.792 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.571123+0000", 2024-05-15T13:14:40.792 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8299999999999995e-07 2024-05-15T13:14:40.792 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.792 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.792 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:14:40.792 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.571131+0000", 2024-05-15T13:14:40.792 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0819999999999999e-06 2024-05-15T13:14:40.792 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.793 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.793 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:14:40.793 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.571142+0000", 2024-05-15T13:14:40.793 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0868e-05 2024-05-15T13:14:40.793 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.793 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.793 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:14:40.793 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.571150+0000", 2024-05-15T13:14:40.793 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9279999999999993e-06 2024-05-15T13:14:40.793 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.793 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.793 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:14:40.793 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.571222+0000", 2024-05-15T13:14:40.794 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1289000000000001e-05 2024-05-15T13:14:40.794 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.794 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.794 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:14:40.794 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.571248+0000", 2024-05-15T13:14:40.794 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5987999999999999e-05 2024-05-15T13:14:40.794 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:14:40.794 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:14:40.794 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:14:40.794 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.794 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.794 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:120809 7.3 7:ca40b3a7:::benchmark_data_smithi177_134484_object11744:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e654)", 2024-05-15T13:14:40.794 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:14:39.573641+0000", 2024-05-15T13:14:40.795 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.0085793540000000008, 2024-05-15T13:14:40.795 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0089260569999999994, 2024-05-15T13:14:40.795 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:14:40.795 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:14:40.795 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:14:40.795 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:14:40.795 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:14:40.795 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 120809 2024-05-15T13:14:40.795 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.795 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:14:40.795 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.795 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:14:40.795 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.573641+0000", 2024-05-15T13:14:40.795 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:14:40.795 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.796 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.796 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:14:40.796 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.573641+0000", 2024-05-15T13:14:40.796 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:14:40.796 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.796 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.796 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:14:40.796 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.573643+0000", 2024-05-15T13:14:40.796 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.584e-06 2024-05-15T13:14:40.796 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.796 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.796 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:14:40.796 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.573650+0000", 2024-05-15T13:14:40.797 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9650000000000002e-06 2024-05-15T13:14:40.797 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.797 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.797 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:14:40.797 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.573669+0000", 2024-05-15T13:14:40.797 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8646000000000001e-05 2024-05-15T13:14:40.797 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.797 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.797 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:14:40.797 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.573678+0000", 2024-05-15T13:14:40.797 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7999999999999993e-06 2024-05-15T13:14:40.797 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.797 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.798 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:14:40.798 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.573774+0000", 2024-05-15T13:14:40.798 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5141000000000003e-05 2024-05-15T13:14:40.798 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.798 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.798 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:14:40.798 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.573902+0000", 2024-05-15T13:14:40.798 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012841400000000001 2024-05-15T13:14:40.798 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.798 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.798 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2,11", 2024-05-15T13:14:40.798 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.574015+0000", 2024-05-15T13:14:40.799 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000112638 2024-05-15T13:14:40.799 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:14:40.799 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:14:40.799 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:14:40.799 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.799 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.799 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.26044.0:120807 7.d e654/599 7:b83a7817:::benchmark_data_smithi177_134484_object11742:head v 654'7361, mlcod=654'7358)", 2024-05-15T13:14:40.799 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:14:39.581355+0000", 2024-05-15T13:14:40.799 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.00086602300000000005, 2024-05-15T13:14:40.799 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0012887339999999999, 2024-05-15T13:14:40.799 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:14:40.799 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:14:40.799 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:14:40.799 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.799 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:14:40.800 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.581355+0000", 2024-05-15T13:14:40.800 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:14:40.800 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.800 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.800 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:14:40.800 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.581355+0000", 2024-05-15T13:14:40.800 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:14:40.800 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.800 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.800 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:14:40.800 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.581356+0000", 2024-05-15T13:14:40.800 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.04e-06 2024-05-15T13:14:40.800 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.800 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.800 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:14:40.801 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.581365+0000", 2024-05-15T13:14:40.801 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6630000000000005e-06 2024-05-15T13:14:40.801 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.801 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.801 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:14:40.801 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.581377+0000", 2024-05-15T13:14:40.801 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1831000000000001e-05 2024-05-15T13:14:40.801 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.801 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.801 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:14:40.801 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.581385+0000", 2024-05-15T13:14:40.801 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1760000000000004e-06 2024-05-15T13:14:40.801 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.801 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.801 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:14:40.801 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.581442+0000", 2024-05-15T13:14:40.802 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7397e-05 2024-05-15T13:14:40.802 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.802 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.802 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:14:40.802 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.581466+0000", 2024-05-15T13:14:40.802 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4476000000000001e-05 2024-05-15T13:14:40.802 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:14:40.802 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:14:40.802 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:14:40.802 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.802 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.802 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.26044.0:120811 7.c e654/647 7:3952623e:::benchmark_data_smithi177_134484_object11746:head v 654'7732, mlcod=608'849)", 2024-05-15T13:14:40.802 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:14:39.581417+0000", 2024-05-15T13:14:40.803 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.00080367399999999999, 2024-05-15T13:14:40.803 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001287744, 2024-05-15T13:14:40.803 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:14:40.803 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:14:40.803 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:14:40.803 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.803 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:14:40.803 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.581417+0000", 2024-05-15T13:14:40.803 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:14:40.803 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.803 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.803 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:14:40.803 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.581417+0000", 2024-05-15T13:14:40.804 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:14:40.804 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.804 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.804 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:14:40.804 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.581418+0000", 2024-05-15T13:14:40.804 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8500000000000002e-07 2024-05-15T13:14:40.804 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.804 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.804 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:14:40.804 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.581422+0000", 2024-05-15T13:14:40.804 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8180000000000003e-06 2024-05-15T13:14:40.804 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.804 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.804 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:14:40.805 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.581430+0000", 2024-05-15T13:14:40.805 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1169999999999998e-06 2024-05-15T13:14:40.805 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.805 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.805 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:14:40.805 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.581435+0000", 2024-05-15T13:14:40.805 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2249999999999999e-06 2024-05-15T13:14:40.805 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.805 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.805 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:14:40.805 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.581543+0000", 2024-05-15T13:14:40.805 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000108548 2024-05-15T13:14:40.805 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.805 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.806 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:14:40.806 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.581580+0000", 2024-05-15T13:14:40.806 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7018999999999999e-05 2024-05-15T13:14:40.806 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:14:40.806 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:14:40.806 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:14:40.806 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.806 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.806 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.26044.0:120788 7.6 e654/647 7:6c1d50cb:::benchmark_data_smithi177_134484_object11723:head v 654'7520, mlcod=608'754)", 2024-05-15T13:14:40.806 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:14:39.536695+0000", 2024-05-15T13:14:40.806 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.045525616999999997, 2024-05-15T13:14:40.806 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.046073153999999998, 2024-05-15T13:14:40.807 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:14:40.807 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:14:40.807 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:14:40.807 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.807 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:14:40.807 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.536695+0000", 2024-05-15T13:14:40.807 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:14:40.807 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.807 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.807 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:14:40.807 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.536695+0000", 2024-05-15T13:14:40.807 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:14:40.807 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.808 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.808 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:14:40.808 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.536697+0000", 2024-05-15T13:14:40.808 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4479999999999999e-06 2024-05-15T13:14:40.808 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.808 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.808 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:14:40.808 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.536703+0000", 2024-05-15T13:14:40.808 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3169999999999998e-06 2024-05-15T13:14:40.808 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.808 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.808 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:14:40.808 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.536711+0000", 2024-05-15T13:14:40.809 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1030000000000002e-06 2024-05-15T13:14:40.809 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.809 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.809 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:14:40.809 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.536718+0000", 2024-05-15T13:14:40.809 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3170000000000004e-06 2024-05-15T13:14:40.809 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.809 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.809 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:14:40.809 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.549165+0000", 2024-05-15T13:14:40.809 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012446654999999999 2024-05-15T13:14:40.809 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.809 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.810 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:14:40.810 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.549195+0000", 2024-05-15T13:14:40.810 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9479999999999999e-05 2024-05-15T13:14:40.810 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:14:40.810 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:14:40.810 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:14:40.810 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.810 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.810 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.26044.0:120800 7.8 e654/647 7:16ce11a4:::benchmark_data_smithi177_134484_object11735:head v 654'7546, mlcod=608'726)", 2024-05-15T13:14:40.810 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:14:39.545032+0000", 2024-05-15T13:14:40.810 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.037188855, 2024-05-15T13:14:40.810 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.037797297000000001, 2024-05-15T13:14:40.810 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:14:40.811 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:14:40.811 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:14:40.811 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.811 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:14:40.811 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.545032+0000", 2024-05-15T13:14:40.811 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:14:40.811 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.811 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.811 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:14:40.811 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.545032+0000", 2024-05-15T13:14:40.811 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:14:40.811 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.811 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.811 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:14:40.811 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.545033+0000", 2024-05-15T13:14:40.812 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2699999999999999e-07 2024-05-15T13:14:40.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.812 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.812 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:14:40.812 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.545039+0000", 2024-05-15T13:14:40.812 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9830000000000001e-06 2024-05-15T13:14:40.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.812 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.812 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:14:40.812 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.545048+0000", 2024-05-15T13:14:40.812 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6749999999999997e-06 2024-05-15T13:14:40.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.812 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.812 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:14:40.812 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.545055+0000", 2024-05-15T13:14:40.813 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4150000000000004e-06 2024-05-15T13:14:40.813 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.813 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.813 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:14:40.813 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.562081+0000", 2024-05-15T13:14:40.813 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.017026025 2024-05-15T13:14:40.813 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.813 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.813 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:14:40.813 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.562115+0000", 2024-05-15T13:14:40.813 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4468999999999998e-05 2024-05-15T13:14:40.813 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:14:40.813 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:14:40.813 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:14:40.814 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.814 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.814 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.26044.0:120796 7.6 e654/647 7:6c5b2c4c:::benchmark_data_smithi177_134484_object11731:head v 654'7521, mlcod=608'754)", 2024-05-15T13:14:40.814 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:14:39.550164+0000", 2024-05-15T13:14:40.814 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.032056336999999997, 2024-05-15T13:14:40.814 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.032725729000000002, 2024-05-15T13:14:40.814 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:14:40.814 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:14:40.814 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:14:40.814 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.814 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:14:40.814 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.550164+0000", 2024-05-15T13:14:40.814 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:14:40.814 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.815 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.815 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:14:40.815 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.550164+0000", 2024-05-15T13:14:40.815 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:14:40.815 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.815 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.815 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:14:40.815 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.550166+0000", 2024-05-15T13:14:40.815 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.477e-06 2024-05-15T13:14:40.815 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.815 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.815 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:14:40.815 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.550175+0000", 2024-05-15T13:14:40.815 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2890000000000003e-06 2024-05-15T13:14:40.816 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.816 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.816 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:14:40.816 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.550189+0000", 2024-05-15T13:14:40.816 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4024e-05 2024-05-15T13:14:40.816 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.816 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.816 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:14:40.816 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.550199+0000", 2024-05-15T13:14:40.816 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8300000000000008e-06 2024-05-15T13:14:40.816 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.816 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.816 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:14:40.816 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.558546+0000", 2024-05-15T13:14:40.817 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0083473079999999995 2024-05-15T13:14:40.817 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:14:40.817 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:14:40.817 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:14:40.817 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:39.558573+0000", 2024-05-15T13:14:40.817 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6526999999999999e-05 2024-05-15T13:14:40.817 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:14:40.817 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:14:40.817 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:14:40.817 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:14:40.817 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:14:40.817 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 10 2024-05-15T13:14:40.818 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:14:41.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:41 smithi027 ceph-mon[39795]: pgmap v1704: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 6.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 467 op/s; 12401/294996 objects degraded (4.204%); 27587/294996 objects misplaced (9.352%); 8.1 MiB/s, 130 objects/s recovering 2024-05-15T13:14:41.908 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:14:41.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:41.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:14:41.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:41 smithi177 ceph-mon[44303]: pgmap v1704: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 6.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 467 op/s; 12401/294996 objects degraded (4.204%); 27587/294996 objects misplaced (9.352%); 8.1 MiB/s, 130 objects/s recovering 2024-05-15T13:14:42.029 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:14:42.035 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:41 smithi120 ceph-mon[43961]: pgmap v1704: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 6.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 467 op/s; 12401/294996 objects degraded (4.204%); 27587/294996 objects misplaced (9.352%); 8.1 MiB/s, 130 objects/s recovering 2024-05-15T13:14:42.060 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:42.082 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:14:42.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:42.085 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:14:42.186 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:14:42.220 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:42.240 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:14:42.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:42.243 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:14:42.344 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:14:42.371 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:42.390 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:14:42.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:42.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:14:42.495 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:14:42.522 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:42.543 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:14:42.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:42.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:14:42.646 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:14:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:43.023 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:14:43.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:43.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:14:43.130 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:14:43.156 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:43.178 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:14:43.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:43.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:14:43.283 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:14:43.310 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:43.329 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:14:43.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:43.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:14:43.433 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:14:43.527 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:43.531 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:14:43.540 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-05-15T13:14:43.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:43 smithi027 ceph-mon[39795]: pgmap v1705: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 5.9 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 467 op/s; 12149/291126 objects degraded (4.173%); 27346/291126 objects misplaced (9.393%); 9.0 MiB/s, 144 objects/s recovering 2024-05-15T13:14:43.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:43 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:14:43.874 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:14:43.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:43.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:14:43.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:43 smithi177 ceph-mon[44303]: pgmap v1705: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 5.9 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 467 op/s; 12149/291126 objects degraded (4.173%); 27346/291126 objects misplaced (9.393%); 9.0 MiB/s, 144 objects/s recovering 2024-05-15T13:14:43.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:43 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:14:43.964 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:14:43.990 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:14:44.014 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:43 smithi120 ceph-mon[43961]: pgmap v1705: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 5.9 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 467 op/s; 12149/291126 objects degraded (4.173%); 27346/291126 objects misplaced (9.393%); 9.0 MiB/s, 144 objects/s recovering 2024-05-15T13:14:44.015 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:43 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:14:44.017 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:44.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:44 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 12149/291126 objects degraded (4.173%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:14:44.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:44 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 12149/291126 objects degraded (4.173%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:14:44.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:44 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 12149/291126 objects degraded (4.173%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:14:44.967 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:14:44.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:44.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:14:45.076 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:14:45.081 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:14:45.108 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:45.211 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:14:45.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:45.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:14:45.316 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:14:45.345 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:45.694 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:14:45.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:45.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:14:45.798 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:14:45.827 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:45.853 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:14:45.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:45.862 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:14:45.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:45 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:14:45.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:45 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:14:45.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:45 smithi177 ceph-mon[44303]: pgmap v1706: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 5.9 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 461 op/s; 12213/288609 objects degraded (4.232%); 26890/288609 objects misplaced (9.317%); 8.9 MiB/s, 142 objects/s recovering 2024-05-15T13:14:45.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:45 smithi177 ceph-mon[44303]: osdmap e655: 12 total, 12 up, 10 in 2024-05-15T13:14:45.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:45 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:14:45.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:45 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:14:45.963 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:14:45.989 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:45 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:14:45.989 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:45 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:14:45.989 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:45 smithi120 ceph-mon[43961]: pgmap v1706: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 5.9 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 461 op/s; 12213/288609 objects degraded (4.232%); 26890/288609 objects misplaced (9.317%); 8.9 MiB/s, 142 objects/s recovering 2024-05-15T13:14:45.989 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:45 smithi120 ceph-mon[43961]: osdmap e655: 12 total, 12 up, 10 in 2024-05-15T13:14:45.989 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:45 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:14:45.990 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:45 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:14:45.992 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:46.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:45 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:14:46.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:45 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:14:46.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:45 smithi027 ceph-mon[39795]: pgmap v1706: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 5.9 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 461 op/s; 12213/288609 objects degraded (4.232%); 26890/288609 objects misplaced (9.317%); 8.9 MiB/s, 142 objects/s recovering 2024-05-15T13:14:46.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:45 smithi027 ceph-mon[39795]: osdmap e655: 12 total, 12 up, 10 in 2024-05-15T13:14:46.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:45 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:14:46.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:45 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:14:46.342 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:14:46.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:46.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:14:46.446 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:14:46.518 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:46.820 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:14:46.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:46.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:14:46.871 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:14:46.922 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:14:46.932 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:14:47.039 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:47.042 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:47 smithi027 ceph-mon[39795]: osdmap e656: 12 total, 12 up, 10 in 2024-05-15T13:14:47.042 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:47 smithi027 ceph-mon[39795]: pgmap v1709: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 5.8 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 455 op/s; 12264/286095 objects degraded (4.287%); 26669/286095 objects misplaced (9.322%); 9.6 MiB/s, 153 objects/s recovering 2024-05-15T13:14:47.166 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:14:47.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:47.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:14:47.278 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:14:47.285 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:47 smithi120 ceph-mon[43961]: osdmap e656: 12 total, 12 up, 10 in 2024-05-15T13:14:47.285 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:47 smithi120 ceph-mon[43961]: pgmap v1709: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 5.8 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 455 op/s; 12264/286095 objects degraded (4.287%); 26669/286095 objects misplaced (9.322%); 9.6 MiB/s, 153 objects/s recovering 2024-05-15T13:14:47.304 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:47 smithi177 ceph-mon[44303]: osdmap e656: 12 total, 12 up, 10 in 2024-05-15T13:14:47.305 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:47 smithi177 ceph-mon[44303]: pgmap v1709: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 5.8 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 455 op/s; 12264/286095 objects degraded (4.287%); 26669/286095 objects misplaced (9.322%); 9.6 MiB/s, 153 objects/s recovering 2024-05-15T13:14:47.306 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:47.326 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:14:47.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:47.328 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:14:47.429 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:14:47.465 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:47.489 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:14:47.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:47.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:14:47.593 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:14:47.621 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:47.641 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:14:47.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:47.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:14:47.745 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:14:47.773 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:47.795 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:14:47.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:47.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:14:47.898 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:14:47.933 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:47.961 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:14:47.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:47.966 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:14:47.990 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:14:48.067 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:14:48.099 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:48.124 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:14:48.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:48.126 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:14:48.227 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:14:48.386 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:49.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:49 smithi177 ceph-mon[44303]: pgmap v1710: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 5.8 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 489 op/s; 11939/283236 objects degraded (4.215%); 26545/283236 objects misplaced (9.372%); 9.1 MiB/s, 145 objects/s recovering 2024-05-15T13:14:49.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:49 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 11939/283236 objects degraded (4.215%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:14:49.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:49 smithi027 ceph-mon[39795]: pgmap v1710: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 5.8 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 489 op/s; 11939/283236 objects degraded (4.215%); 26545/283236 objects misplaced (9.372%); 9.1 MiB/s, 145 objects/s recovering 2024-05-15T13:14:49.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:49 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 11939/283236 objects degraded (4.215%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:14:49.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:49 smithi120 ceph-mon[43961]: pgmap v1710: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 5.8 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 489 op/s; 11939/283236 objects degraded (4.215%); 26545/283236 objects misplaced (9.372%); 9.1 MiB/s, 145 objects/s recovering 2024-05-15T13:14:49.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:49 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 11939/283236 objects degraded (4.215%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:14:49.881 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:14:49.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:49.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:14:49.987 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:14:50.016 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:50.464 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.6 to 1 (10000) 2024-05-15T13:14:50.503 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:50 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:14:50.503 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:50 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:14:50.503 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:50 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2745403447' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T13:14:50.503 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:50 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T13:14:50.666 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:14:50.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:50.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:14:50.770 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:14:50.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:50 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:14:50.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:50 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:14:50.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:50 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2745403447' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T13:14:50.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:50 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T13:14:50.798 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:50.853 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:14:50.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:50.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:14:50.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:50 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:14:50.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:50 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:14:50.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:50 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2745403447' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T13:14:50.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:50 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T13:14:50.961 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:14:50.988 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:51.006 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:14:51.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:51.009 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:14:51.109 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:14:51.139 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:51.239 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:14:51.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:51.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:14:51.343 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:14:51.370 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:51.388 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:14:51.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:51.391 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:14:51.460 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:51 smithi027 ceph-mon[39795]: pgmap v1711: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 5.7 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 399 op/s; 11957/281538 objects degraded (4.247%); 26221/281538 objects misplaced (9.313%); 8.9 MiB/s, 142 objects/s recovering 2024-05-15T13:14:51.460 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:51 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T13:14:51.460 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:51 smithi027 ceph-mon[39795]: osdmap e657: 12 total, 12 up, 10 in 2024-05-15T13:14:51.460 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:51 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:14:51.460 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:51 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:14:51.460 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:51 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:14:51.460 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:51 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:14:51.460 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:51 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:14:51.492 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:14:51.519 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:51.537 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:14:51.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:51.540 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:14:51.641 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:14:51.670 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:51.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:51 smithi177 ceph-mon[44303]: pgmap v1711: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 5.7 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 399 op/s; 11957/281538 objects degraded (4.247%); 26221/281538 objects misplaced (9.313%); 8.9 MiB/s, 142 objects/s recovering 2024-05-15T13:14:51.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:51 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T13:14:51.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:51 smithi177 ceph-mon[44303]: osdmap e657: 12 total, 12 up, 10 in 2024-05-15T13:14:51.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:51 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:14:51.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:51 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:14:51.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:51 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:14:51.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:51 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:14:51.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:51 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:14:51.690 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:14:51.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:51.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:14:51.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:51 smithi120 ceph-mon[43961]: pgmap v1711: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 5.7 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 399 op/s; 11957/281538 objects degraded (4.247%); 26221/281538 objects misplaced (9.313%); 8.9 MiB/s, 142 objects/s recovering 2024-05-15T13:14:51.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:51 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T13:14:51.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:51 smithi120 ceph-mon[43961]: osdmap e657: 12 total, 12 up, 10 in 2024-05-15T13:14:51.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:51 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:14:51.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:51 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:14:51.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:51 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:14:51.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:51 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:14:51.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:51 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:14:51.794 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:14:51.821 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:51.840 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:14:51.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:51.843 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:14:51.944 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:14:52.010 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:52.670 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:14:52.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:52.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:14:52.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:52 smithi027 ceph-mon[39795]: osdmap e658: 12 total, 12 up, 10 in 2024-05-15T13:14:52.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:52 smithi027 ceph-mon[39795]: osdmap e659: 12 total, 12 up, 10 in 2024-05-15T13:14:52.776 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:14:52.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:52 smithi120 ceph-mon[43961]: osdmap e658: 12 total, 12 up, 10 in 2024-05-15T13:14:52.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:52 smithi120 ceph-mon[43961]: osdmap e659: 12 total, 12 up, 10 in 2024-05-15T13:14:52.801 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:52 smithi177 ceph-mon[44303]: osdmap e658: 12 total, 12 up, 10 in 2024-05-15T13:14:52.801 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:52 smithi177 ceph-mon[44303]: osdmap e659: 12 total, 12 up, 10 in 2024-05-15T13:14:52.802 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:52.821 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:14:52.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:52.824 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:14:52.925 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:14:52.954 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:52.977 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:14:52.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:52.980 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:14:53.081 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:14:53.221 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:53.320 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:14:53.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:53.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:14:53.379 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 8 1 2024-05-15T13:14:53.439 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:14:53.467 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:53.506 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:14:53.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:53.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:14:53.566 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:14:53.567 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T13:14:53.567 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-05-15T13:14:53.567 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-05-15T13:14:53.567 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:14:53.610 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:14:53.638 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:53.658 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:14:53.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:53.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:14:53.763 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:14:53.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:53 smithi027 ceph-mon[39795]: pgmap v1714: 37 pgs: 4 remapped+peering, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 5.7 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 419 op/s; 749/278481 objects degraded (0.269%); 25984/278481 objects misplaced (9.331%); 8.4 MiB/s, 134 objects/s recovering 2024-05-15T13:14:53.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:53 smithi120 ceph-mon[43961]: pgmap v1714: 37 pgs: 4 remapped+peering, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 5.7 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 419 op/s; 749/278481 objects degraded (0.269%); 25984/278481 objects misplaced (9.331%); 8.4 MiB/s, 134 objects/s recovering 2024-05-15T13:14:53.792 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:53.816 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:14:53.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:53.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:14:53.921 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:14:53.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:53 smithi177 ceph-mon[44303]: pgmap v1714: 37 pgs: 4 remapped+peering, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 5.7 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 419 op/s; 749/278481 objects degraded (0.269%); 25984/278481 objects misplaced (9.331%); 8.4 MiB/s, 134 objects/s recovering 2024-05-15T13:14:53.948 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:53.982 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:14:53.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:53.984 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:14:54.085 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:14:54.114 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:54.140 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:14:54.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:54.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:14:54.244 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:14:54.273 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:54.291 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:14:54.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:54.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:14:54.397 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:14:54.426 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:54.449 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:14:54.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:54.452 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:14:54.552 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:14:54.584 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:54.608 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:14:54.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:54.612 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:14:54.685 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:54 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:14:54.685 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:54 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:14:54.685 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:54 smithi177 ceph-mon[44303]: osdmap e660: 12 total, 12 up, 10 in 2024-05-15T13:14:54.685 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:54 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 749/278481 objects degraded (0.269%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-05-15T13:14:54.713 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:14:54.746 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:54.772 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:14:54.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:14:54.776 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:14:54.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:54 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:14:54.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:54 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:14:54.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:54 smithi027 ceph-mon[39795]: osdmap e660: 12 total, 12 up, 10 in 2024-05-15T13:14:54.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:54 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 749/278481 objects degraded (0.269%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-05-15T13:14:54.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:54 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:14:54.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:54 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:14:54.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:54 smithi120 ceph-mon[43961]: osdmap e660: 12 total, 12 up, 10 in 2024-05-15T13:14:54.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:54 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 749/278481 objects degraded (0.269%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-05-15T13:14:54.877 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:14:54.949 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:14:55.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:55 smithi177 ceph-mon[44303]: pgmap v1717: 37 pgs: 4 remapped+peering, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 5.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 370 op/s; 771/277083 objects degraded (0.278%); 25342/277083 objects misplaced (9.146%); 14 MiB/s, 225 objects/s recovering 2024-05-15T13:14:55.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:55 smithi177 ceph-mon[44303]: osdmap e661: 12 total, 12 up, 10 in 2024-05-15T13:14:56.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:55 smithi027 ceph-mon[39795]: pgmap v1717: 37 pgs: 4 remapped+peering, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 5.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 370 op/s; 771/277083 objects degraded (0.278%); 25342/277083 objects misplaced (9.146%); 14 MiB/s, 225 objects/s recovering 2024-05-15T13:14:56.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:55 smithi027 ceph-mon[39795]: osdmap e661: 12 total, 12 up, 10 in 2024-05-15T13:14:56.035 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:55 smithi120 ceph-mon[43961]: pgmap v1717: 37 pgs: 4 remapped+peering, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 5.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 370 op/s; 771/277083 objects degraded (0.278%); 25342/277083 objects misplaced (9.146%); 14 MiB/s, 225 objects/s recovering 2024-05-15T13:14:56.035 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:55 smithi120 ceph-mon[43961]: osdmap e661: 12 total, 12 up, 10 in 2024-05-15T13:14:57.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:57 smithi177 ceph-mon[44303]: pgmap v1719: 37 pgs: 4 remapped+peering, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 27 active+clean; 5.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 285 op/s; 791/275283 objects degraded (0.287%); 30611/275283 objects misplaced (11.120%); 14 MiB/s, 226 objects/s recovering 2024-05-15T13:14:58.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:57 smithi027 ceph-mon[39795]: pgmap v1719: 37 pgs: 4 remapped+peering, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 27 active+clean; 5.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 285 op/s; 791/275283 objects degraded (0.287%); 30611/275283 objects misplaced (11.120%); 14 MiB/s, 226 objects/s recovering 2024-05-15T13:14:58.035 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:57 smithi120 ceph-mon[43961]: pgmap v1719: 37 pgs: 4 remapped+peering, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 27 active+clean; 5.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 285 op/s; 791/275283 objects degraded (0.287%); 30611/275283 objects misplaced (11.120%); 14 MiB/s, 226 objects/s recovering 2024-05-15T13:14:59.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:59 smithi177 ceph-mon[44303]: pgmap v1720: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 327 op/s; 13523/272571 objects degraded (4.961%); 37429/272571 objects misplaced (13.732%); 13 MiB/s, 201 objects/s recovering 2024-05-15T13:14:59.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:14:59 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 13523/272571 objects degraded (4.961%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:00.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:59 smithi027 ceph-mon[39795]: pgmap v1720: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 327 op/s; 13523/272571 objects degraded (4.961%); 37429/272571 objects misplaced (13.732%); 13 MiB/s, 201 objects/s recovering 2024-05-15T13:15:00.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:14:59 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 13523/272571 objects degraded (4.961%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:00.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:59 smithi120 ceph-mon[43961]: pgmap v1720: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 327 op/s; 13523/272571 objects degraded (4.961%); 37429/272571 objects misplaced (13.732%); 13 MiB/s, 201 objects/s recovering 2024-05-15T13:15:00.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:14:59 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 13523/272571 objects degraded (4.961%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:00.254 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:15:00.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:00.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:15:00.377 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:15:00.471 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:00.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:00 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:15:00.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:00 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:15:00.936 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:00 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:15:00.937 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:00 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:15:01.035 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:00 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:15:01.035 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:00 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:15:01.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:01 smithi027 ceph-mon[39795]: pgmap v1721: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 303 op/s; 13536/271275 objects degraded (4.990%); 37078/271275 objects misplaced (13.668%); 12 MiB/s, 196 objects/s recovering 2024-05-15T13:15:01.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:01 smithi027 ceph-mon[39795]: osdmap e662: 12 total, 12 up, 10 in 2024-05-15T13:15:01.887 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:15:01.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:01.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:15:01.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:01 smithi177 ceph-mon[44303]: pgmap v1721: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 303 op/s; 13536/271275 objects degraded (4.990%); 37078/271275 objects misplaced (13.668%); 12 MiB/s, 196 objects/s recovering 2024-05-15T13:15:01.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:01 smithi177 ceph-mon[44303]: osdmap e662: 12 total, 12 up, 10 in 2024-05-15T13:15:01.993 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:15:02.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:01 smithi120 ceph-mon[43961]: pgmap v1721: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 303 op/s; 13536/271275 objects degraded (4.990%); 37078/271275 objects misplaced (13.668%); 12 MiB/s, 196 objects/s recovering 2024-05-15T13:15:02.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:01 smithi120 ceph-mon[43961]: osdmap e662: 12 total, 12 up, 10 in 2024-05-15T13:15:02.101 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:02.318 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:15:02.359 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:15:02.425 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:15:02.467 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:15:02.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:02.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:15:02.492 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:15:02.534 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:15:02.544 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-05-15T13:15:02.552 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:15:02.575 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:15:02.614 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:15:02.663 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:15:02.699 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:02.705 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:15:02.711 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:15:02.738 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:02 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1497011085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:15:02.738 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:02 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3492673247' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:15:02.738 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:02 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:15:02.752 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:15:02.790 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:15:02.842 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:15:02.903 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:15:02.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:02 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1497011085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:15:02.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:02 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3492673247' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:15:02.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:02 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:15:02.940 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:15:02.941 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:15:02.941 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:15:02.941 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:15:02.941 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:15:02.941 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:15:02.941 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:15:02.941 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:15:02.941 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:15:02.941 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:15:02.941 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:15:02.941 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:15:03.035 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:02 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1497011085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:15:03.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:02 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3492673247' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:15:03.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:02 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:15:03.348 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:15:03.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:03.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:15:03.460 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:15:03.491 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:03.531 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:15:03.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:03.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:15:03.637 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:15:03.720 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:03.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:03 smithi120 ceph-mon[43961]: pgmap v1723: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 380 op/s; 13246/267924 objects degraded (4.944%); 36750/267924 objects misplaced (13.717%); 11 MiB/s, 173 objects/s recovering 2024-05-15T13:15:03.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:03 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:15:03.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:03 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:15:03.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:03 smithi120 ceph-mon[43961]: osdmap e663: 12 total, 12 up, 10 in 2024-05-15T13:15:04.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:03 smithi027 ceph-mon[39795]: pgmap v1723: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 380 op/s; 13246/267924 objects degraded (4.944%); 36750/267924 objects misplaced (13.717%); 11 MiB/s, 173 objects/s recovering 2024-05-15T13:15:04.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:03 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:15:04.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:03 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:15:04.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:03 smithi027 ceph-mon[39795]: osdmap e663: 12 total, 12 up, 10 in 2024-05-15T13:15:04.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:03 smithi177 ceph-mon[44303]: pgmap v1723: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 380 op/s; 13246/267924 objects degraded (4.944%); 36750/267924 objects misplaced (13.717%); 11 MiB/s, 173 objects/s recovering 2024-05-15T13:15:04.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:03 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:15:04.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:03 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:15:04.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:03 smithi177 ceph-mon[44303]: osdmap e663: 12 total, 12 up, 10 in 2024-05-15T13:15:05.025 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:04 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 13246/267924 objects degraded (4.944%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:05.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:04 smithi027 ceph-mon[39795]: osdmap e664: 12 total, 12 up, 10 in 2024-05-15T13:15:05.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:04 smithi027 ceph-mon[39795]: pgmap v1726: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.4 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 439 op/s; 13263/264648 objects degraded (5.012%); 36338/264648 objects misplaced (13.731%); 12 MiB/s, 199 objects/s recovering 2024-05-15T13:15:05.035 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:04 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 13246/267924 objects degraded (4.944%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:05.035 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:04 smithi120 ceph-mon[43961]: osdmap e664: 12 total, 12 up, 10 in 2024-05-15T13:15:05.035 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:04 smithi120 ceph-mon[43961]: pgmap v1726: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.4 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 439 op/s; 13263/264648 objects degraded (5.012%); 36338/264648 objects misplaced (13.731%); 12 MiB/s, 199 objects/s recovering 2024-05-15T13:15:05.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:04 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 13246/267924 objects degraded (4.944%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:05.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:04 smithi177 ceph-mon[44303]: osdmap e664: 12 total, 12 up, 10 in 2024-05-15T13:15:05.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:04 smithi177 ceph-mon[44303]: pgmap v1726: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.4 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 439 op/s; 13263/264648 objects degraded (5.012%); 36338/264648 objects misplaced (13.731%); 12 MiB/s, 199 objects/s recovering 2024-05-15T13:15:06.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:06 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:15:06.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:06 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:15:06.285 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:06 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:15:06.285 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:06 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:15:06.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:06 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:15:06.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:06 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:15:07.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:07 smithi177 ceph-mon[44303]: osdmap e665: 12 total, 12 up, 10 in 2024-05-15T13:15:07.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:07 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:15:07.435 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:07 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:15:07.435 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:07 smithi177 ceph-mon[44303]: pgmap v1728: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 521 op/s; 13287/262323 objects degraded (5.065%); 35988/262323 objects misplaced (13.719%); 13 MiB/s, 207 objects/s recovering 2024-05-15T13:15:07.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:07 smithi027 ceph-mon[39795]: osdmap e665: 12 total, 12 up, 10 in 2024-05-15T13:15:07.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:07 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:15:07.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:07 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:15:07.527 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:07 smithi027 ceph-mon[39795]: pgmap v1728: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 521 op/s; 13287/262323 objects degraded (5.065%); 35988/262323 objects misplaced (13.719%); 13 MiB/s, 207 objects/s recovering 2024-05-15T13:15:07.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:07 smithi120 ceph-mon[43961]: osdmap e665: 12 total, 12 up, 10 in 2024-05-15T13:15:07.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:07 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:15:07.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:07 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:15:07.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:07 smithi120 ceph-mon[43961]: pgmap v1728: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 521 op/s; 13287/262323 objects degraded (5.065%); 35988/262323 objects misplaced (13.719%); 13 MiB/s, 207 objects/s recovering 2024-05-15T13:15:08.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:08 smithi177 ceph-mon[44303]: osdmap e666: 12 total, 12 up, 10 in 2024-05-15T13:15:08.527 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:08 smithi027 ceph-mon[39795]: osdmap e666: 12 total, 12 up, 10 in 2024-05-15T13:15:08.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:08 smithi120 ceph-mon[43961]: osdmap e666: 12 total, 12 up, 10 in 2024-05-15T13:15:09.356 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:15:09.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:09.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:15:09.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:09 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:15:09.373 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:09 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:15:09.373 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:09 smithi027 ceph-mon[39795]: osdmap e667: 12 total, 12 up, 10 in 2024-05-15T13:15:09.373 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:09 smithi027 ceph-mon[39795]: pgmap v1731: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 393 op/s; 13093/259842 objects degraded (5.039%); 35857/259842 objects misplaced (13.800%); 9.8 MiB/s, 158 objects/s recovering 2024-05-15T13:15:09.373 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:09 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 13287/262323 objects degraded (5.065%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:09.373 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:09 smithi027 ceph-mon[39795]: osdmap e668: 12 total, 12 up, 10 in 2024-05-15T13:15:09.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:09 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:15:09.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:09 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:15:09.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:09 smithi177 ceph-mon[44303]: osdmap e667: 12 total, 12 up, 10 in 2024-05-15T13:15:09.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:09 smithi177 ceph-mon[44303]: pgmap v1731: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 393 op/s; 13093/259842 objects degraded (5.039%); 35857/259842 objects misplaced (13.800%); 9.8 MiB/s, 158 objects/s recovering 2024-05-15T13:15:09.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:09 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 13287/262323 objects degraded (5.065%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:09.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:09 smithi177 ceph-mon[44303]: osdmap e668: 12 total, 12 up, 10 in 2024-05-15T13:15:09.471 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:15:09.500 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:09.523 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:15:09.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:09.529 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:15:09.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:09 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:15:09.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:09 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:15:09.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:09 smithi120 ceph-mon[43961]: osdmap e667: 12 total, 12 up, 10 in 2024-05-15T13:15:09.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:09 smithi120 ceph-mon[43961]: pgmap v1731: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 393 op/s; 13093/259842 objects degraded (5.039%); 35857/259842 objects misplaced (13.800%); 9.8 MiB/s, 158 objects/s recovering 2024-05-15T13:15:09.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:09 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 13287/262323 objects degraded (5.065%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:09.536 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:09 smithi120 ceph-mon[43961]: osdmap e668: 12 total, 12 up, 10 in 2024-05-15T13:15:09.629 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:15:09.758 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:10.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:10 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:15:10.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:10 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:15:10.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:10 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:15:10.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:10 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:15:10.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:10 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:15:10.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:10 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:15:11.430 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:15:11.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:11 smithi177 ceph-mon[44303]: pgmap v1733: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 278 op/s; 13101/258702 objects degraded (5.064%); 35463/258702 objects misplaced (13.708%); 8.8 MiB/s, 140 objects/s recovering 2024-05-15T13:15:11.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:11 smithi027 ceph-mon[39795]: pgmap v1733: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 278 op/s; 13101/258702 objects degraded (5.064%); 35463/258702 objects misplaced (13.708%); 8.8 MiB/s, 140 objects/s recovering 2024-05-15T13:15:11.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:11 smithi120 ceph-mon[43961]: pgmap v1733: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 278 op/s; 13101/258702 objects degraded (5.064%); 35463/258702 objects misplaced (13.708%); 8.8 MiB/s, 140 objects/s recovering 2024-05-15T13:15:13.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:13 smithi120 ceph-mon[43961]: pgmap v1734: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.2 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 374 op/s; 12871/255579 objects degraded (5.036%); 35077/255579 objects misplaced (13.725%); 11 MiB/s, 183 objects/s recovering 2024-05-15T13:15:13.797 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:13 smithi027 ceph-mon[39795]: pgmap v1734: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.2 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 374 op/s; 12871/255579 objects degraded (5.036%); 35077/255579 objects misplaced (13.725%); 11 MiB/s, 183 objects/s recovering 2024-05-15T13:15:13.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:13 smithi177 ceph-mon[44303]: pgmap v1734: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.2 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 374 op/s; 12871/255579 objects degraded (5.036%); 35077/255579 objects misplaced (13.725%); 11 MiB/s, 183 objects/s recovering 2024-05-15T13:15:13.986 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:15:13.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:13.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:15:14.085 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:15:14.090 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:15:14.120 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:14.166 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:15:14.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:14.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:15:14.270 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:15:14.301 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:14.322 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:15:14.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:14.326 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:15:14.427 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:15:14.455 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:14.479 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:15:14.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:14.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:15:14.583 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:15:14.610 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:14.631 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:15:14.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:14.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:15:14.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:14 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:15:14.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:14 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:15:14.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:14 smithi177 ceph-mon[44303]: osdmap e669: 12 total, 12 up, 10 in 2024-05-15T13:15:14.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:14 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 12871/255579 objects degraded (5.036%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:14.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:14 smithi177 ceph-mon[44303]: osdmap e670: 12 total, 12 up, 10 in 2024-05-15T13:15:14.736 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:15:14.765 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:14 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:15:14.765 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:14 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:15:14.765 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:14 smithi120 ceph-mon[43961]: osdmap e669: 12 total, 12 up, 10 in 2024-05-15T13:15:14.765 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:14 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 12871/255579 objects degraded (5.036%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:14.765 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:14 smithi120 ceph-mon[43961]: osdmap e670: 12 total, 12 up, 10 in 2024-05-15T13:15:14.767 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:14.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:14 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:15:14.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:14 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:15:14.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:14 smithi027 ceph-mon[39795]: osdmap e669: 12 total, 12 up, 10 in 2024-05-15T13:15:14.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:14 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 12871/255579 objects degraded (5.036%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:14.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:14 smithi027 ceph-mon[39795]: osdmap e670: 12 total, 12 up, 10 in 2024-05-15T13:15:14.790 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:15:14.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:14.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:15:14.895 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:15:15.011 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:15.060 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:15:15.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:15.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:15:15.167 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:15:15.197 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:15.218 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:15:15.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:15.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:15:15.322 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:15:15.349 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:15.369 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:15:15.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:15.372 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:15:15.472 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:15:15.579 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:15.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:15 smithi177 ceph-mon[44303]: pgmap v1737: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 422 op/s; 12895/252237 objects degraded (5.112%); 34611/252237 objects misplaced (13.722%); 14 MiB/s, 219 objects/s recovering 2024-05-15T13:15:15.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:15 smithi027 ceph-mon[39795]: pgmap v1737: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 422 op/s; 12895/252237 objects degraded (5.112%); 34611/252237 objects misplaced (13.722%); 14 MiB/s, 219 objects/s recovering 2024-05-15T13:15:15.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:15 smithi120 ceph-mon[43961]: pgmap v1737: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 422 op/s; 12895/252237 objects degraded (5.112%); 34611/252237 objects misplaced (13.722%); 14 MiB/s, 219 objects/s recovering 2024-05-15T13:15:16.039 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:15:16.534 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:16 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1702297488' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T13:15:16.534 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:16 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T13:15:16.577 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.8 to 1 (10000) 2024-05-15T13:15:16.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:16 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1702297488' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T13:15:16.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:16 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T13:15:16.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:16 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1702297488' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T13:15:16.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:16 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T13:15:17.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:17 smithi177 ceph-mon[44303]: pgmap v1738: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 423 op/s; 12953/249669 objects degraded (5.188%); 34244/249669 objects misplaced (13.716%); 12 MiB/s, 191 objects/s recovering 2024-05-15T13:15:17.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:17 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T13:15:17.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:17 smithi177 ceph-mon[44303]: osdmap e671: 12 total, 12 up, 10 in 2024-05-15T13:15:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:15:17.999 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:17 smithi027 ceph-mon[39795]: pgmap v1738: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 423 op/s; 12953/249669 objects degraded (5.188%); 34244/249669 objects misplaced (13.716%); 12 MiB/s, 191 objects/s recovering 2024-05-15T13:15:17.999 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:17 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T13:15:17.999 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:17 smithi027 ceph-mon[39795]: osdmap e671: 12 total, 12 up, 10 in 2024-05-15T13:15:18.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:18.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:15:18.035 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:17 smithi120 ceph-mon[43961]: pgmap v1738: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 423 op/s; 12953/249669 objects degraded (5.188%); 34244/249669 objects misplaced (13.716%); 12 MiB/s, 191 objects/s recovering 2024-05-15T13:15:18.035 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:17 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T13:15:18.035 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:17 smithi120 ceph-mon[43961]: osdmap e671: 12 total, 12 up, 10 in 2024-05-15T13:15:18.101 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:15:18.127 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:18.150 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:15:18.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:18.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:15:18.255 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:15:18.281 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:18.304 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:15:18.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:18.311 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:15:18.412 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:15:18.443 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:18.468 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:15:18.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:18.471 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:15:18.572 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:15:18.604 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:18.626 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:15:18.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:18.632 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:15:18.733 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:15:18.764 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:18.790 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:15:18.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:18.794 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:15:18.895 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:15:18.926 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:18.953 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:15:18.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:18.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:15:19.057 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:15:19.063 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 9 1 2024-05-15T13:15:19.088 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:19.108 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:15:19.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:19.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:15:19.212 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:15:19.240 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:19.263 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:15:19.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:19.266 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:15:19.367 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:15:19.394 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:19.416 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:15:19.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:19.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:15:19.524 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:15:19.613 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:19.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:19 smithi120 ceph-mon[43961]: pgmap v1740: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.0 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 480 op/s; 12739/246918 objects degraded (5.159%); 34095/246918 objects misplaced (13.808%); 10 MiB/s, 164 objects/s recovering 2024-05-15T13:15:19.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:19 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 12739/246918 objects degraded (5.159%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:19.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:19 smithi120 ceph-mon[43961]: osdmap e672: 12 total, 12 up, 10 in 2024-05-15T13:15:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:19 smithi177 ceph-mon[44303]: pgmap v1740: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.0 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 480 op/s; 12739/246918 objects degraded (5.159%); 34095/246918 objects misplaced (13.808%); 10 MiB/s, 164 objects/s recovering 2024-05-15T13:15:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:19 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 12739/246918 objects degraded (5.159%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:19 smithi177 ceph-mon[44303]: osdmap e672: 12 total, 12 up, 10 in 2024-05-15T13:15:20.025 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:19 smithi027 ceph-mon[39795]: pgmap v1740: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.0 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 480 op/s; 12739/246918 objects degraded (5.159%); 34095/246918 objects misplaced (13.808%); 10 MiB/s, 164 objects/s recovering 2024-05-15T13:15:20.025 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:19 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 12739/246918 objects degraded (5.159%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:20.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:19 smithi027 ceph-mon[39795]: osdmap e672: 12 total, 12 up, 10 in 2024-05-15T13:15:20.154 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:15:20.274 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:15:20.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:20.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:15:20.325 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:15:20.420 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:15:20.534 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:21.891 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:21 smithi027 ceph-mon[39795]: pgmap v1742: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.0 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 358 op/s; 12747/245790 objects degraded (5.186%); 33747/245790 objects misplaced (13.730%); 10 MiB/s, 161 objects/s recovering 2024-05-15T13:15:21.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:21 smithi177 ceph-mon[44303]: pgmap v1742: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.0 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 358 op/s; 12747/245790 objects degraded (5.186%); 33747/245790 objects misplaced (13.730%); 10 MiB/s, 161 objects/s recovering 2024-05-15T13:15:22.035 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:21 smithi120 ceph-mon[43961]: pgmap v1742: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 5.0 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 358 op/s; 12747/245790 objects degraded (5.186%); 33747/245790 objects misplaced (13.730%); 10 MiB/s, 161 objects/s recovering 2024-05-15T13:15:22.141 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:15:22.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:22.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:15:22.252 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:15:22.281 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:22.308 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:15:22.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:22.315 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:15:22.416 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:15:22.445 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:22.467 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:15:22.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:22.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:15:22.571 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:15:22.600 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:22.852 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:15:22.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:22.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:15:22.956 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:15:22.985 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:23.014 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:15:23.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:23.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:15:23.119 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:15:23.148 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:23.169 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:15:23.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:23.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:15:23.276 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:15:23.306 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:23.329 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:15:23.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:23.333 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:15:23.434 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:15:23.464 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:23.599 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:15:23.615 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:23 smithi027 ceph-mon[39795]: pgmap v1743: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 403 op/s; 12549/242541 objects degraded (5.174%); 33366/242541 objects misplaced (13.757%); 10 MiB/s, 166 objects/s recovering 2024-05-15T13:15:23.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:23.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:15:23.734 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:15:23.762 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:23.782 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:15:23.785 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:23 smithi120 ceph-mon[43961]: pgmap v1743: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 403 op/s; 12549/242541 objects degraded (5.174%); 33366/242541 objects misplaced (13.757%); 10 MiB/s, 166 objects/s recovering 2024-05-15T13:15:23.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:23.788 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:15:23.889 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:15:23.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:23 smithi177 ceph-mon[44303]: pgmap v1743: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 403 op/s; 12549/242541 objects degraded (5.174%); 33366/242541 objects misplaced (13.757%); 10 MiB/s, 166 objects/s recovering 2024-05-15T13:15:24.024 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:24.181 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:15:24.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:24.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:15:24.297 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:15:24.334 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:24.359 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:15:24.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:24.366 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:15:24.467 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:15:24.494 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:24.519 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:15:24.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:24.522 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:15:24.579 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:15:24.579 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:15:24.623 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:15:24.713 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:24.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:24 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 12549/242541 objects degraded (5.174%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:25.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:24 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 12549/242541 objects degraded (5.174%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:25.035 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:24 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 12549/242541 objects degraded (5.174%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:25.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: pgmap v1744: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 436 op/s; 12614/239178 objects degraded (5.274%); 32884/239178 objects misplaced (13.749%); 11 MiB/s, 179 objects/s recovering 2024-05-15T13:15:25.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1547205395,config=1059521453,health=2753466768,kv=59954984}) 2024-05-15T13:15:25.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=1693278236,logm=4215210336}) 2024-05-15T13:15:25.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3159705245}) 2024-05-15T13:15:25.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=255535117}) 2024-05-15T13:15:25.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2648028208}) 2024-05-15T13:15:25.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3218575439}) 2024-05-15T13:15:25.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3382096174}) 2024-05-15T13:15:25.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1859588163}) 2024-05-15T13:15:25.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1246582793}) 2024-05-15T13:15:25.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1262813217}) 2024-05-15T13:15:25.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4218156220}) 2024-05-15T13:15:25.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=201948211}) 2024-05-15T13:15:25.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2701958954}) 2024-05-15T13:15:25.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3047449873}) 2024-05-15T13:15:25.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=852569492}) 2024-05-15T13:15:25.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=459935131}) 2024-05-15T13:15:25.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=142986623}) 2024-05-15T13:15:25.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1502378867}) 2024-05-15T13:15:25.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=571194719}) 2024-05-15T13:15:25.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1668714925}) 2024-05-15T13:15:25.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3042107131}) 2024-05-15T13:15:25.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4171981229}) 2024-05-15T13:15:25.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=231550649}) 2024-05-15T13:15:25.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3031840709}) 2024-05-15T13:15:25.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=607370027}) 2024-05-15T13:15:25.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2090087793}) 2024-05-15T13:15:25.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1761511101}) 2024-05-15T13:15:25.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1158998506}) 2024-05-15T13:15:25.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:25 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=622716235}) 2024-05-15T13:15:26.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: pgmap v1744: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 436 op/s; 12614/239178 objects degraded (5.274%); 32884/239178 objects misplaced (13.749%); 11 MiB/s, 179 objects/s recovering 2024-05-15T13:15:26.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1547205395,config=1059521453,health=2753466768,kv=59954984}) 2024-05-15T13:15:26.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=1693278236,logm=4215210336}) 2024-05-15T13:15:26.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3159705245}) 2024-05-15T13:15:26.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=255535117}) 2024-05-15T13:15:26.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2648028208}) 2024-05-15T13:15:26.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3218575439}) 2024-05-15T13:15:26.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3382096174}) 2024-05-15T13:15:26.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1859588163}) 2024-05-15T13:15:26.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1246582793}) 2024-05-15T13:15:26.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1262813217}) 2024-05-15T13:15:26.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4218156220}) 2024-05-15T13:15:26.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=201948211}) 2024-05-15T13:15:26.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2701958954}) 2024-05-15T13:15:26.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3047449873}) 2024-05-15T13:15:26.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=852569492}) 2024-05-15T13:15:26.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=459935131}) 2024-05-15T13:15:26.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=142986623}) 2024-05-15T13:15:26.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1502378867}) 2024-05-15T13:15:26.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=571194719}) 2024-05-15T13:15:26.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1668714925}) 2024-05-15T13:15:26.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3042107131}) 2024-05-15T13:15:26.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4171981229}) 2024-05-15T13:15:26.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=231550649}) 2024-05-15T13:15:26.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3031840709}) 2024-05-15T13:15:26.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=607370027}) 2024-05-15T13:15:26.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2090087793}) 2024-05-15T13:15:26.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1761511101}) 2024-05-15T13:15:26.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1158998506}) 2024-05-15T13:15:26.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:25 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=622716235}) 2024-05-15T13:15:26.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: pgmap v1744: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 436 op/s; 12614/239178 objects degraded (5.274%); 32884/239178 objects misplaced (13.749%); 11 MiB/s, 179 objects/s recovering 2024-05-15T13:15:26.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1547205395,config=1059521453,health=2753466768,kv=59954984}) 2024-05-15T13:15:26.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=1693278236,logm=4215210336}) 2024-05-15T13:15:26.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3159705245}) 2024-05-15T13:15:26.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=255535117}) 2024-05-15T13:15:26.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2648028208}) 2024-05-15T13:15:26.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3218575439}) 2024-05-15T13:15:26.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3382096174}) 2024-05-15T13:15:26.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1859588163}) 2024-05-15T13:15:26.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1246582793}) 2024-05-15T13:15:26.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1262813217}) 2024-05-15T13:15:26.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4218156220}) 2024-05-15T13:15:26.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=201948211}) 2024-05-15T13:15:26.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2701958954}) 2024-05-15T13:15:26.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3047449873}) 2024-05-15T13:15:26.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=852569492}) 2024-05-15T13:15:26.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=459935131}) 2024-05-15T13:15:26.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=142986623}) 2024-05-15T13:15:26.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1502378867}) 2024-05-15T13:15:26.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=571194719}) 2024-05-15T13:15:26.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1668714925}) 2024-05-15T13:15:26.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3042107131}) 2024-05-15T13:15:26.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4171981229}) 2024-05-15T13:15:26.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=231550649}) 2024-05-15T13:15:26.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3031840709}) 2024-05-15T13:15:26.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=607370027}) 2024-05-15T13:15:26.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2090087793}) 2024-05-15T13:15:26.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1761511101}) 2024-05-15T13:15:26.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1158998506}) 2024-05-15T13:15:26.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:25 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=622716235}) 2024-05-15T13:15:26.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2532681713}) 2024-05-15T13:15:26.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2830594480}) 2024-05-15T13:15:26.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=594055213}) 2024-05-15T13:15:26.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3395024785}) 2024-05-15T13:15:26.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1187101102}) 2024-05-15T13:15:26.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2371213681}) 2024-05-15T13:15:26.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2896228506}) 2024-05-15T13:15:26.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=853743287}) 2024-05-15T13:15:26.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3334711985}) 2024-05-15T13:15:26.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3079743830}) 2024-05-15T13:15:26.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1349659752}) 2024-05-15T13:15:26.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=344722878}) 2024-05-15T13:15:26.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3364063405}) 2024-05-15T13:15:26.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2396882627}) 2024-05-15T13:15:26.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1843070962}) 2024-05-15T13:15:26.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3063428497}) 2024-05-15T13:15:26.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4046481287}) 2024-05-15T13:15:26.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2512279341}) 2024-05-15T13:15:26.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1675997396}) 2024-05-15T13:15:26.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2346173207}) 2024-05-15T13:15:26.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2687792659}) 2024-05-15T13:15:26.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=494441577}) 2024-05-15T13:15:26.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=758076824}) 2024-05-15T13:15:26.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1003978382}) 2024-05-15T13:15:26.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2859731434}) 2024-05-15T13:15:26.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2610308981}) 2024-05-15T13:15:26.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2615141302}) 2024-05-15T13:15:26.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2422610664}) 2024-05-15T13:15:26.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=506861828}) 2024-05-15T13:15:26.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=69,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=7} crc {logm=2540430720,mdsmap=3984425268,mgr=76093979,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=2945871377}) 2024-05-15T13:15:26.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=2,mon_config_key=98} crc {mgrstat=3768403019,mon_config_key=165462807}) 2024-05-15T13:15:26.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1154562908}) 2024-05-15T13:15:26.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=77,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=5} crc {mon_config_key=3842245807,monmap=2435304556,osd_metadata=1212035326,osd_pg_creating=1299638267,osdmap=1508204059}) 2024-05-15T13:15:26.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1826552157}) 2024-05-15T13:15:26.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=39} crc {osdmap=3653685950}) 2024-05-15T13:15:27.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2532681713}) 2024-05-15T13:15:27.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2830594480}) 2024-05-15T13:15:27.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=594055213}) 2024-05-15T13:15:27.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3395024785}) 2024-05-15T13:15:27.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1187101102}) 2024-05-15T13:15:27.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2371213681}) 2024-05-15T13:15:27.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2896228506}) 2024-05-15T13:15:27.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=853743287}) 2024-05-15T13:15:27.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3334711985}) 2024-05-15T13:15:27.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3079743830}) 2024-05-15T13:15:27.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1349659752}) 2024-05-15T13:15:27.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=344722878}) 2024-05-15T13:15:27.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3364063405}) 2024-05-15T13:15:27.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2396882627}) 2024-05-15T13:15:27.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1843070962}) 2024-05-15T13:15:27.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3063428497}) 2024-05-15T13:15:27.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4046481287}) 2024-05-15T13:15:27.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2512279341}) 2024-05-15T13:15:27.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1675997396}) 2024-05-15T13:15:27.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2346173207}) 2024-05-15T13:15:27.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2687792659}) 2024-05-15T13:15:27.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=494441577}) 2024-05-15T13:15:27.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=758076824}) 2024-05-15T13:15:27.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1003978382}) 2024-05-15T13:15:27.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2859731434}) 2024-05-15T13:15:27.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2610308981}) 2024-05-15T13:15:27.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2615141302}) 2024-05-15T13:15:27.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2422610664}) 2024-05-15T13:15:27.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=506861828}) 2024-05-15T13:15:27.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=69,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=7} crc {logm=2540430720,mdsmap=3984425268,mgr=76093979,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=2945871377}) 2024-05-15T13:15:27.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=2,mon_config_key=98} crc {mgrstat=3768403019,mon_config_key=165462807}) 2024-05-15T13:15:27.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1154562908}) 2024-05-15T13:15:27.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=77,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=5} crc {mon_config_key=3842245807,monmap=2435304556,osd_metadata=1212035326,osd_pg_creating=1299638267,osdmap=1508204059}) 2024-05-15T13:15:27.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1826552157}) 2024-05-15T13:15:27.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=39} crc {osdmap=3653685950}) 2024-05-15T13:15:27.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2532681713}) 2024-05-15T13:15:27.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2830594480}) 2024-05-15T13:15:27.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=594055213}) 2024-05-15T13:15:27.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3395024785}) 2024-05-15T13:15:27.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1187101102}) 2024-05-15T13:15:27.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2371213681}) 2024-05-15T13:15:27.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2896228506}) 2024-05-15T13:15:27.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=853743287}) 2024-05-15T13:15:27.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3334711985}) 2024-05-15T13:15:27.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3079743830}) 2024-05-15T13:15:27.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1349659752}) 2024-05-15T13:15:27.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=344722878}) 2024-05-15T13:15:27.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3364063405}) 2024-05-15T13:15:27.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2396882627}) 2024-05-15T13:15:27.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1843070962}) 2024-05-15T13:15:27.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3063428497}) 2024-05-15T13:15:27.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4046481287}) 2024-05-15T13:15:27.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2512279341}) 2024-05-15T13:15:27.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1675997396}) 2024-05-15T13:15:27.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2346173207}) 2024-05-15T13:15:27.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2687792659}) 2024-05-15T13:15:27.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=494441577}) 2024-05-15T13:15:27.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=758076824}) 2024-05-15T13:15:27.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1003978382}) 2024-05-15T13:15:27.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2859731434}) 2024-05-15T13:15:27.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2610308981}) 2024-05-15T13:15:27.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2615141302}) 2024-05-15T13:15:27.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2422610664}) 2024-05-15T13:15:27.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=506861828}) 2024-05-15T13:15:27.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=69,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=7} crc {logm=2540430720,mdsmap=3984425268,mgr=76093979,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=2945871377}) 2024-05-15T13:15:27.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=2,mon_config_key=98} crc {mgrstat=3768403019,mon_config_key=165462807}) 2024-05-15T13:15:27.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1154562908}) 2024-05-15T13:15:27.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=77,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=5} crc {mon_config_key=3842245807,monmap=2435304556,osd_metadata=1212035326,osd_pg_creating=1299638267,osdmap=1508204059}) 2024-05-15T13:15:27.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1826552157}) 2024-05-15T13:15:27.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=39} crc {osdmap=3653685950}) 2024-05-15T13:15:27.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:27 smithi177 ceph-mon[44303]: pgmap v1745: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.8 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 439 op/s; 12661/236649 objects degraded (5.350%); 32521/236649 objects misplaced (13.742%); 11 MiB/s, 177 objects/s recovering 2024-05-15T13:15:28.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:27 smithi027 ceph-mon[39795]: pgmap v1745: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.8 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 439 op/s; 12661/236649 objects degraded (5.350%); 32521/236649 objects misplaced (13.742%); 11 MiB/s, 177 objects/s recovering 2024-05-15T13:15:28.035 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:27 smithi120 ceph-mon[43961]: pgmap v1745: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.8 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 439 op/s; 12661/236649 objects degraded (5.350%); 32521/236649 objects misplaced (13.742%); 11 MiB/s, 177 objects/s recovering 2024-05-15T13:15:28.866 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:15:28.871 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:15:28.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:28.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:15:28.983 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:15:29.085 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:29.151 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:15:29.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:29.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:15:29.266 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:15:29.293 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:29.314 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:15:29.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:29.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:15:29.419 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:15:29.557 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:30.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:29 smithi027 ceph-mon[39795]: pgmap v1746: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 437 op/s; 12461/233772 objects degraded (5.330%); 32368/233772 objects misplaced (13.846%); 11 MiB/s, 174 objects/s recovering 2024-05-15T13:15:30.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:29 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 12461/233772 objects degraded (5.330%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:30.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:29 smithi120 ceph-mon[43961]: pgmap v1746: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 437 op/s; 12461/233772 objects degraded (5.330%); 32368/233772 objects misplaced (13.846%); 11 MiB/s, 174 objects/s recovering 2024-05-15T13:15:30.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:29 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 12461/233772 objects degraded (5.330%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:30.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:29 smithi177 ceph-mon[44303]: pgmap v1746: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 437 op/s; 12461/233772 objects degraded (5.330%); 32368/233772 objects misplaced (13.846%); 11 MiB/s, 174 objects/s recovering 2024-05-15T13:15:30.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:29 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 12461/233772 objects degraded (5.330%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:30.773 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:15:30.773 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:15:30.773 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:15:30.773 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:15:30.773 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:15:30.773 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:15:30.773 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:15:30.773 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:15:30.773 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:15:30.774 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:15:30.774 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:15:30.774 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:15:30.774 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:15:30.774 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:15:30.774 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:15:30.774 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:15:30.774 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:15:30.774 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:15:30.774 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:15:30.774 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:15:30.775 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:15:30.775 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:15:30.775 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:15:30.775 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:15:30.775 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:15:30.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:30.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:15:30.888 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:15:30.915 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:30.941 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:15:30.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:30.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:15:31.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:30 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/442768282' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:15:31.035 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:30 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/442768282' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:15:31.047 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:15:31.076 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:30 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/442768282' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:15:31.077 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:31.103 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:15:31.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:31.105 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:15:31.206 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:15:31.311 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:32.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:31 smithi027 ceph-mon[39795]: pgmap v1747: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 402 op/s; 12470/232467 objects degraded (5.364%); 32020/232467 objects misplaced (13.774%); 10 MiB/s, 166 objects/s recovering 2024-05-15T13:15:32.035 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:31 smithi120 ceph-mon[43961]: pgmap v1747: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 402 op/s; 12470/232467 objects degraded (5.364%); 32020/232467 objects misplaced (13.774%); 10 MiB/s, 166 objects/s recovering 2024-05-15T13:15:32.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:31 smithi177 ceph-mon[44303]: pgmap v1747: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 402 op/s; 12470/232467 objects degraded (5.364%); 32020/232467 objects misplaced (13.774%); 10 MiB/s, 166 objects/s recovering 2024-05-15T13:15:33.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:32 smithi027 ceph-mon[39795]: pgmap v1748: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 468 op/s; 12279/228891 objects degraded (5.365%); 31610/228891 objects misplaced (13.810%); 11 MiB/s, 183 objects/s recovering 2024-05-15T13:15:33.035 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:32 smithi120 ceph-mon[43961]: pgmap v1748: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 468 op/s; 12279/228891 objects degraded (5.365%); 31610/228891 objects misplaced (13.810%); 11 MiB/s, 183 objects/s recovering 2024-05-15T13:15:33.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:32 smithi177 ceph-mon[44303]: pgmap v1748: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 468 op/s; 12279/228891 objects degraded (5.365%); 31610/228891 objects misplaced (13.810%); 11 MiB/s, 183 objects/s recovering 2024-05-15T13:15:34.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:33 smithi177 ceph-mon[44303]: from='client.16188 172.21.15.27:0/665992486' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:15:34.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:33 smithi177 ceph-mon[44303]: from='client.16188 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:15:34.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:33 smithi027 ceph-mon[39795]: from='client.16188 172.21.15.27:0/665992486' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:15:34.284 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:33 smithi027 ceph-mon[39795]: from='client.16188 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:15:34.285 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:33 smithi120 ceph-mon[43961]: from='client.16188 172.21.15.27:0/665992486' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:15:34.286 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:33 smithi120 ceph-mon[43961]: from='client.16188 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:15:34.346 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:15:34.934 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:15:34.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:34.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:15:35.044 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:15:35.072 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:35.102 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:15:35.103 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:34 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:15:35.104 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:34 smithi177 ceph-mon[44303]: from='client.16188 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:15:35.104 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:34 smithi177 ceph-mon[44303]: osdmap e673: 12 total, 12 up, 10 in 2024-05-15T13:15:35.104 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:34 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 12279/228891 objects degraded (5.365%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:35.104 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:34 smithi177 ceph-mon[44303]: osdmap e674: 12 total, 12 up, 10 in 2024-05-15T13:15:35.104 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:34 smithi177 ceph-mon[44303]: pgmap v1751: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 452 op/s; 12316/225774 objects degraded (5.455%); 31118/225774 objects misplaced (13.783%); 11 MiB/s, 184 objects/s recovering 2024-05-15T13:15:35.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:35.105 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:15:35.206 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:15:35.232 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:34 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:15:35.232 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:34 smithi120 ceph-mon[43961]: from='client.16188 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:15:35.232 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:34 smithi120 ceph-mon[43961]: osdmap e673: 12 total, 12 up, 10 in 2024-05-15T13:15:35.232 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:34 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 12279/228891 objects degraded (5.365%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:35.232 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:34 smithi120 ceph-mon[43961]: osdmap e674: 12 total, 12 up, 10 in 2024-05-15T13:15:35.232 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:34 smithi120 ceph-mon[43961]: pgmap v1751: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 452 op/s; 12316/225774 objects degraded (5.455%); 31118/225774 objects misplaced (13.783%); 11 MiB/s, 184 objects/s recovering 2024-05-15T13:15:35.234 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:35.257 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:15:35.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:35.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:15:35.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:34 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:15:35.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:34 smithi027 ceph-mon[39795]: from='client.16188 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:15:35.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:34 smithi027 ceph-mon[39795]: osdmap e673: 12 total, 12 up, 10 in 2024-05-15T13:15:35.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:34 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 12279/228891 objects degraded (5.365%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:35.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:34 smithi027 ceph-mon[39795]: osdmap e674: 12 total, 12 up, 10 in 2024-05-15T13:15:35.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:34 smithi027 ceph-mon[39795]: pgmap v1751: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 452 op/s; 12316/225774 objects degraded (5.455%); 31118/225774 objects misplaced (13.783%); 11 MiB/s, 184 objects/s recovering 2024-05-15T13:15:35.362 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:15:35.474 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:35.697 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-05-15T13:15:36.032 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:15:36.062 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:15:36.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:36.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:15:36.175 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:15:36.209 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:36.234 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:15:36.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:36.238 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:15:36.339 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:15:36.377 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:36.399 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:15:36.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:36.404 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:15:36.506 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:15:36.569 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:37.069 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:15:37.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:37.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:15:37.174 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:15:37.306 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:37.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:37 smithi027 ceph-mon[39795]: pgmap v1752: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 433 op/s; 12371/223347 objects degraded (5.539%); 30722/223347 objects misplaced (13.755%); 12 MiB/s, 198 objects/s recovering 2024-05-15T13:15:37.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:37 smithi120 ceph-mon[43961]: pgmap v1752: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 433 op/s; 12371/223347 objects degraded (5.539%); 30722/223347 objects misplaced (13.755%); 12 MiB/s, 198 objects/s recovering 2024-05-15T13:15:37.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:37 smithi177 ceph-mon[44303]: pgmap v1752: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 433 op/s; 12371/223347 objects degraded (5.539%); 30722/223347 objects misplaced (13.755%); 12 MiB/s, 198 objects/s recovering 2024-05-15T13:15:38.143 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:15:38.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:38.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:15:38.250 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:15:38.283 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:38.311 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:15:38.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:38.315 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:15:38.416 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:15:38.437 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:15:38.438 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T13:15:38.438 INFO:teuthology.orchestra.run.smithi120.stdout: "num_ops": 0 2024-05-15T13:15:38.438 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:15:38.450 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:38.473 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:15:38.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:38.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:15:38.577 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:15:38.603 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:38.625 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:15:38.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:38.629 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:15:38.730 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:15:38.761 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:38.783 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:15:38.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:38.788 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:15:38.889 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:15:38.920 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:39.074 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:15:39.388 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-05-15T13:15:39.388 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:15:39.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:39.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:15:39.492 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:15:39.512 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:39 smithi027 ceph-mon[39795]: pgmap v1753: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 460 op/s; 12128/221409 objects degraded (5.478%); 30722/221409 objects misplaced (13.876%); 11 MiB/s, 171 objects/s recovering 2024-05-15T13:15:39.512 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:39 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 12128/221409 objects degraded (5.478%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:39.512 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:39 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4052674603' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:15:39.535 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.9 to 1 (10000) 2024-05-15T13:15:39.608 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:39.705 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:15:39.735 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:15:39.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:39 smithi120 ceph-mon[43961]: pgmap v1753: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 460 op/s; 12128/221409 objects degraded (5.478%); 30722/221409 objects misplaced (13.876%); 11 MiB/s, 171 objects/s recovering 2024-05-15T13:15:39.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:39 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 12128/221409 objects degraded (5.478%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:39.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:39 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4052674603' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:15:39.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:39.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:15:39.910 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:15:39.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:39 smithi177 ceph-mon[44303]: pgmap v1753: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 460 op/s; 12128/221409 objects degraded (5.478%); 30722/221409 objects misplaced (13.876%); 11 MiB/s, 171 objects/s recovering 2024-05-15T13:15:39.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:39 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 12128/221409 objects degraded (5.478%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:39.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:39 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4052674603' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:15:39.971 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:40.159 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:15:40.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:40 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4052674603' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T13:15:40.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:40 smithi027 ceph-mon[39795]: osdmap e675: 12 total, 12 up, 10 in 2024-05-15T13:15:40.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:40 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4052674603' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T13:15:40.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:40 smithi120 ceph-mon[43961]: osdmap e675: 12 total, 12 up, 10 in 2024-05-15T13:15:40.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:40 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4052674603' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T13:15:40.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:40 smithi177 ceph-mon[44303]: osdmap e675: 12 total, 12 up, 10 in 2024-05-15T13:15:41.683 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:15:41.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:41.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:15:41.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:41 smithi120 ceph-mon[43961]: pgmap v1755: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 342 op/s; 12172/219426 objects degraded (5.547%); 30210/219426 objects misplaced (13.768%); 10 MiB/s, 164 objects/s recovering 2024-05-15T13:15:41.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:41 smithi120 ceph-mon[43961]: osdmap e676: 12 total, 12 up, 10 in 2024-05-15T13:15:41.788 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:15:41.817 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:41.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:41 smithi177 ceph-mon[44303]: pgmap v1755: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 342 op/s; 12172/219426 objects degraded (5.547%); 30210/219426 objects misplaced (13.768%); 10 MiB/s, 164 objects/s recovering 2024-05-15T13:15:41.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:41 smithi177 ceph-mon[44303]: osdmap e676: 12 total, 12 up, 10 in 2024-05-15T13:15:42.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:41 smithi027 ceph-mon[39795]: pgmap v1755: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 4.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 342 op/s; 12172/219426 objects degraded (5.547%); 30210/219426 objects misplaced (13.768%); 10 MiB/s, 164 objects/s recovering 2024-05-15T13:15:42.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:41 smithi027 ceph-mon[39795]: osdmap e676: 12 total, 12 up, 10 in 2024-05-15T13:15:42.867 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:15:42.869 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:42 smithi120 ceph-mon[43961]: osdmap e677: 12 total, 12 up, 10 in 2024-05-15T13:15:42.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:42.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:15:42.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:42 smithi177 ceph-mon[44303]: osdmap e677: 12 total, 12 up, 10 in 2024-05-15T13:15:42.971 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:15:43.002 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:43.025 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:42 smithi027 ceph-mon[39795]: osdmap e677: 12 total, 12 up, 10 in 2024-05-15T13:15:43.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:43 smithi177 ceph-mon[44303]: pgmap v1758: 37 pgs: 1 active+recovery_wait+remapped, 2 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 27 active+clean; 4.4 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 349 op/s; 12037/217059 objects degraded (5.545%); 29724/217059 objects misplaced (13.694%); 12 MiB/s, 187 objects/s recovering 2024-05-15T13:15:43.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:43 smithi177 ceph-mon[44303]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-05-15T13:15:43.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:43 smithi177 ceph-mon[44303]: osdmap e678: 12 total, 12 up, 10 in 2024-05-15T13:15:43.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:43 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:15:43.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:43 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:15:43.998 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:43 smithi120 ceph-mon[43961]: pgmap v1758: 37 pgs: 1 active+recovery_wait+remapped, 2 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 27 active+clean; 4.4 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 349 op/s; 12037/217059 objects degraded (5.545%); 29724/217059 objects misplaced (13.694%); 12 MiB/s, 187 objects/s recovering 2024-05-15T13:15:43.998 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:43 smithi120 ceph-mon[43961]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-05-15T13:15:43.998 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:43 smithi120 ceph-mon[43961]: osdmap e678: 12 total, 12 up, 10 in 2024-05-15T13:15:43.998 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:43 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:15:43.998 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:43 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:15:44.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:43 smithi027 ceph-mon[39795]: pgmap v1758: 37 pgs: 1 active+recovery_wait+remapped, 2 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 27 active+clean; 4.4 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 349 op/s; 12037/217059 objects degraded (5.545%); 29724/217059 objects misplaced (13.694%); 12 MiB/s, 187 objects/s recovering 2024-05-15T13:15:44.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:43 smithi027 ceph-mon[39795]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-05-15T13:15:44.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:43 smithi027 ceph-mon[39795]: osdmap e678: 12 total, 12 up, 10 in 2024-05-15T13:15:44.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:43 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:15:44.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:43 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:15:44.389 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 10 1 2024-05-15T13:15:44.418 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:15:44.506 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:15:44.745 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:44 smithi027 ceph-mon[39795]: osdmap e679: 12 total, 12 up, 10 in 2024-05-15T13:15:44.746 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:44 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 12037/217059 objects degraded (5.545%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:44.746 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:44 smithi027 ceph-mon[39795]: osdmap e680: 12 total, 12 up, 10 in 2024-05-15T13:15:44.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:44 smithi177 ceph-mon[44303]: osdmap e679: 12 total, 12 up, 10 in 2024-05-15T13:15:44.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:44 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 12037/217059 objects degraded (5.545%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:44.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:44 smithi177 ceph-mon[44303]: osdmap e680: 12 total, 12 up, 10 in 2024-05-15T13:15:45.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:44 smithi120 ceph-mon[43961]: osdmap e679: 12 total, 12 up, 10 in 2024-05-15T13:15:45.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:44 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 12037/217059 objects degraded (5.545%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:45.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:44 smithi120 ceph-mon[43961]: osdmap e680: 12 total, 12 up, 10 in 2024-05-15T13:15:45.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:45 smithi177 ceph-mon[44303]: pgmap v1762: 37 pgs: 1 active+recovery_wait+remapped, 2 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 26 active+clean; 4.4 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 371 op/s; 12072/215100 objects degraded (5.612%); 33479/215100 objects misplaced (15.564%); 21 MiB/s, 333 objects/s recovering 2024-05-15T13:15:46.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:45 smithi027 ceph-mon[39795]: pgmap v1762: 37 pgs: 1 active+recovery_wait+remapped, 2 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 26 active+clean; 4.4 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 371 op/s; 12072/215100 objects degraded (5.612%); 33479/215100 objects misplaced (15.564%); 21 MiB/s, 333 objects/s recovering 2024-05-15T13:15:46.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:45 smithi120 ceph-mon[43961]: pgmap v1762: 37 pgs: 1 active+recovery_wait+remapped, 2 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 26 active+clean; 4.4 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 371 op/s; 12072/215100 objects degraded (5.612%); 33479/215100 objects misplaced (15.564%); 21 MiB/s, 333 objects/s recovering 2024-05-15T13:15:46.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:46 smithi177 ceph-mon[44303]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-05-15T13:15:47.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:46 smithi027 ceph-mon[39795]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-05-15T13:15:47.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:46 smithi120 ceph-mon[43961]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-05-15T13:15:47.455 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:15:47.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:47.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:15:47.560 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:15:47.625 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:47.670 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:15:47.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:47 smithi027 ceph-mon[39795]: pgmap v1763: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 4.3 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 311 op/s; 12072/212994 objects degraded (5.668%); 33107/212994 objects misplaced (15.544%); 16 MiB/s, 248 objects/s recovering 2024-05-15T13:15:47.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:47.686 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:15:47.787 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:15:47.902 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:47.907 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:47 smithi177 ceph-mon[44303]: pgmap v1763: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 4.3 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 311 op/s; 12072/212994 objects degraded (5.668%); 33107/212994 objects misplaced (15.544%); 16 MiB/s, 248 objects/s recovering 2024-05-15T13:15:47.929 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:15:47.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:47.934 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:15:48.035 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:15:48.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:47 smithi120 ceph-mon[43961]: pgmap v1763: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 4.3 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 311 op/s; 12072/212994 objects degraded (5.668%); 33107/212994 objects misplaced (15.544%); 16 MiB/s, 248 objects/s recovering 2024-05-15T13:15:48.075 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:48.102 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:15:48.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:48.104 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:15:48.206 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:15:48.239 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:48.265 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:15:48.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:48.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:15:48.369 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:15:48.402 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:48.419 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:15:48.420 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T13:15:48.420 INFO:teuthology.orchestra.run.smithi120.stdout: "complaint_time": 30, 2024-05-15T13:15:48.420 INFO:teuthology.orchestra.run.smithi120.stdout: "num_blocked_ops": 0 2024-05-15T13:15:48.420 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:15:48.431 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:15:48.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:48.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:15:48.536 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:15:48.573 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:48.596 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:15:48.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:48.607 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:15:48.708 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:15:48.843 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:15:48.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:48.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:15:49.017 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:15:49.048 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:49.060 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:15:49.070 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:15:49.150 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:15:49.210 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:15:49.278 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:15:49.298 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:15:49.299 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-05-15T13:15:49.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:49.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:15:49.363 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:15:49.388 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:15:49.403 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:15:49.434 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:49.455 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:15:49.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:49.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:15:49.560 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:15:49.641 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:15:49.649 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:49.654 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:15:49.699 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:15:49.761 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:15:49.808 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:15:49.870 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:15:49.902 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:15:49.920 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:15:49.920 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:15:49.920 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:15:49.920 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:15:49.920 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:15:49.921 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:15:49.921 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:15:49.921 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:15:49.921 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:15:49.921 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:15:49.921 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:15:49.921 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:15:49.957 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:49 smithi027 ceph-mon[39795]: pgmap v1764: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 4.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 344 op/s; 11805/210849 objects degraded (5.599%); 32892/210849 objects misplaced (15.600%); 15 MiB/s, 245 objects/s recovering 2024-05-15T13:15:49.957 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:49 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1449878209' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:15:49.957 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:49 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 11805/210849 objects degraded (5.599%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:49.957 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:49 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/997198595' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:15:49.983 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:49 smithi120 ceph-mon[43961]: pgmap v1764: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 4.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 344 op/s; 11805/210849 objects degraded (5.599%); 32892/210849 objects misplaced (15.600%); 15 MiB/s, 245 objects/s recovering 2024-05-15T13:15:49.984 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:49 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1449878209' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:15:49.984 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:49 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 11805/210849 objects degraded (5.599%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:49.984 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:49 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/997198595' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:15:50.087 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:15:50.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:49 smithi177 ceph-mon[44303]: pgmap v1764: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 4.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 344 op/s; 11805/210849 objects degraded (5.599%); 32892/210849 objects misplaced (15.600%); 15 MiB/s, 245 objects/s recovering 2024-05-15T13:15:50.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:49 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1449878209' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:15:50.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:49 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 11805/210849 objects degraded (5.599%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:50.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:49 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/997198595' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:15:50.483 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:15:50.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:50.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:15:50.594 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:15:50.689 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:50.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:50 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:15:50.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:50 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/997198595' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:15:50.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:50 smithi027 ceph-mon[39795]: osdmap e681: 12 total, 12 up, 10 in 2024-05-15T13:15:51.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:50 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:15:51.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:50 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/997198595' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:15:51.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:50 smithi120 ceph-mon[43961]: osdmap e681: 12 total, 12 up, 10 in 2024-05-15T13:15:51.299 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:50 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:15:51.300 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:50 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/997198595' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:15:51.300 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:50 smithi177 ceph-mon[44303]: osdmap e681: 12 total, 12 up, 10 in 2024-05-15T13:15:51.753 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:15:51.753 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:15:52.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:51 smithi027 ceph-mon[39795]: pgmap v1766: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 4.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 340 op/s; 11825/208836 objects degraded (5.662%); 32413/208836 objects misplaced (15.521%); 16 MiB/s, 253 objects/s recovering 2024-05-15T13:15:52.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:51 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:15:52.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:51 smithi120 ceph-mon[43961]: pgmap v1766: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 4.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 340 op/s; 11825/208836 objects degraded (5.662%); 32413/208836 objects misplaced (15.521%); 16 MiB/s, 253 objects/s recovering 2024-05-15T13:15:52.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:51 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:15:52.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:51 smithi177 ceph-mon[44303]: pgmap v1766: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 4.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 340 op/s; 11825/208836 objects degraded (5.662%); 32413/208836 objects misplaced (15.521%); 16 MiB/s, 253 objects/s recovering 2024-05-15T13:15:52.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:51 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:15:52.701 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:15:52.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:15:52.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:15:52.808 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:15:52.898 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:15:54.019 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:53 smithi120 ceph-mon[43961]: pgmap v1767: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 4.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 391 op/s; 11615/205455 objects degraded (5.653%); 32066/205455 objects misplaced (15.607%); 14 MiB/s, 221 objects/s recovering 2024-05-15T13:15:54.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:53 smithi027 ceph-mon[39795]: pgmap v1767: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 4.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 391 op/s; 11615/205455 objects degraded (5.653%); 32066/205455 objects misplaced (15.607%); 14 MiB/s, 221 objects/s recovering 2024-05-15T13:15:54.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:53 smithi177 ceph-mon[44303]: pgmap v1767: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 4.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 391 op/s; 11615/205455 objects degraded (5.653%); 32066/205455 objects misplaced (15.607%); 14 MiB/s, 221 objects/s recovering 2024-05-15T13:15:54.944 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-05-15T13:15:55.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:54 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 11615/205455 objects degraded (5.653%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:55.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:54 smithi027 ceph-mon[39795]: osdmap e682: 12 total, 12 up, 10 in 2024-05-15T13:15:55.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:54 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:15:55.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:54 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:15:55.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:54 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 11615/205455 objects degraded (5.653%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:55.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:54 smithi120 ceph-mon[43961]: osdmap e682: 12 total, 12 up, 10 in 2024-05-15T13:15:55.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:54 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:15:55.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:54 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:15:55.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:54 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 11615/205455 objects degraded (5.653%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:15:55.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:54 smithi177 ceph-mon[44303]: osdmap e682: 12 total, 12 up, 10 in 2024-05-15T13:15:55.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:54 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:15:55.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:54 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:15:55.898 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:15:56.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:55 smithi027 ceph-mon[39795]: pgmap v1769: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 4.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 440 op/s; 11632/202386 objects degraded (5.747%); 31296/202386 objects misplaced (15.464%); 16 MiB/s, 249 objects/s recovering 2024-05-15T13:15:56.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:55 smithi027 ceph-mon[39795]: osdmap e683: 12 total, 12 up, 10 in 2024-05-15T13:15:56.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:55 smithi120 ceph-mon[43961]: pgmap v1769: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 4.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 440 op/s; 11632/202386 objects degraded (5.747%); 31296/202386 objects misplaced (15.464%); 16 MiB/s, 249 objects/s recovering 2024-05-15T13:15:56.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:55 smithi120 ceph-mon[43961]: osdmap e683: 12 total, 12 up, 10 in 2024-05-15T13:15:56.137 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:55 smithi177 ceph-mon[44303]: pgmap v1769: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 4.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 440 op/s; 11632/202386 objects degraded (5.747%); 31296/202386 objects misplaced (15.464%); 16 MiB/s, 249 objects/s recovering 2024-05-15T13:15:56.138 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:55 smithi177 ceph-mon[44303]: osdmap e683: 12 total, 12 up, 10 in 2024-05-15T13:15:57.611 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:57 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:15:57.611 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:57 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:15:57.611 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:57 smithi027 ceph-mon[39795]: pgmap v1771: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 4.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 358 op/s; 11632/201987 objects degraded (5.759%); 31296/201987 objects misplaced (15.494%); 12 MiB/s, 187 objects/s recovering 2024-05-15T13:15:57.611 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:57 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:15:57.612 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:57 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:15:57.612 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:57 smithi027 ceph-mon[39795]: osdmap e684: 12 total, 12 up, 10 in 2024-05-15T13:15:57.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:57 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:15:57.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:57 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:15:57.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:57 smithi120 ceph-mon[43961]: pgmap v1771: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 4.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 358 op/s; 11632/201987 objects degraded (5.759%); 31296/201987 objects misplaced (15.494%); 12 MiB/s, 187 objects/s recovering 2024-05-15T13:15:57.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:57 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:15:57.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:57 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:15:57.787 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:57 smithi120 ceph-mon[43961]: osdmap e684: 12 total, 12 up, 10 in 2024-05-15T13:15:57.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:57 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:15:57.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:57 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:15:57.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:57 smithi177 ceph-mon[44303]: pgmap v1771: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 4.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 358 op/s; 11632/201987 objects degraded (5.759%); 31296/201987 objects misplaced (15.494%); 12 MiB/s, 187 objects/s recovering 2024-05-15T13:15:57.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:57 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:15:57.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:57 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:15:57.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:57 smithi177 ceph-mon[44303]: osdmap e684: 12 total, 12 up, 10 in 2024-05-15T13:15:58.178 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:15:58.178 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:15:58.940 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-05-15T13:15:59.251 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:16:00.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:59 smithi120 ceph-mon[43961]: pgmap v1773: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 4.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 412 op/s; 11399/198024 objects degraded (5.756%); 30676/198024 objects misplaced (15.491%); 15 MiB/s, 247 objects/s recovering 2024-05-15T13:16:00.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:15:59 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 11399/198024 objects degraded (5.756%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:00.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:59 smithi177 ceph-mon[44303]: pgmap v1773: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 4.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 412 op/s; 11399/198024 objects degraded (5.756%); 30676/198024 objects misplaced (15.491%); 15 MiB/s, 247 objects/s recovering 2024-05-15T13:16:00.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:15:59 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 11399/198024 objects degraded (5.756%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:00.279 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:59 smithi027 ceph-mon[39795]: pgmap v1773: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 4.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 412 op/s; 11399/198024 objects degraded (5.756%); 30676/198024 objects misplaced (15.491%); 15 MiB/s, 247 objects/s recovering 2024-05-15T13:16:00.279 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:15:59 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 11399/198024 objects degraded (5.756%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:00.922 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:16:00.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:00.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:16:00.964 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:00 smithi120 ceph-mon[43961]: pgmap v1774: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 4.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 352 op/s; 11415/196062 objects degraded (5.822%); 30083/196062 objects misplaced (15.344%); 14 MiB/s, 229 objects/s recovering 2024-05-15T13:16:01.032 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:16:01.106 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:00 smithi027 ceph-mon[39795]: pgmap v1774: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 4.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 352 op/s; 11415/196062 objects degraded (5.822%); 30083/196062 objects misplaced (15.344%); 14 MiB/s, 229 objects/s recovering 2024-05-15T13:16:01.156 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:01.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:00 smithi177 ceph-mon[44303]: pgmap v1774: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 4.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 352 op/s; 11415/196062 objects degraded (5.822%); 30083/196062 objects misplaced (15.344%); 14 MiB/s, 229 objects/s recovering 2024-05-15T13:16:01.292 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:16:01.292 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:16:01.420 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:16:01.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:01.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:16:01.534 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:16:02.139 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-05-15T13:16:02.414 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:02.448 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:16:02.923 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:16:02.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:02.926 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:16:02.933 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:16:03.027 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:16:03.108 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:16:03.133 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:03.206 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:16:03.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:03.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:16:03.318 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:16:03.470 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:03.512 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:16:03.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:03.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:16:03.588 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:16:03.631 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:16:03.661 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:03.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:03 smithi027 ceph-mon[39795]: pgmap v1775: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 395 op/s; 11136/192897 objects degraded (5.773%); 29692/192897 objects misplaced (15.393%); 14 MiB/s, 229 objects/s recovering 2024-05-15T13:16:03.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:03 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:16:03.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:03 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:16:03.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:03 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:16:03.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:03 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:16:03.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:03 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3933236269' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T13:16:03.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:03 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T13:16:03.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:03 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:16:03.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:03 smithi120 ceph-mon[43961]: pgmap v1775: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 395 op/s; 11136/192897 objects degraded (5.773%); 29692/192897 objects misplaced (15.393%); 14 MiB/s, 229 objects/s recovering 2024-05-15T13:16:03.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:03 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:16:03.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:03 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:16:03.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:03 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:16:03.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:03 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:16:03.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:03 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3933236269' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T13:16:03.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:03 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T13:16:03.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:03 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:16:03.927 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:16:03.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:03 smithi177 ceph-mon[44303]: pgmap v1775: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 395 op/s; 11136/192897 objects degraded (5.773%); 29692/192897 objects misplaced (15.393%); 14 MiB/s, 229 objects/s recovering 2024-05-15T13:16:03.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:03 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:16:03.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:03 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:16:03.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:03 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:16:03.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:03 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:16:03.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:03 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3933236269' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T13:16:03.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:03 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T13:16:03.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:03 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:16:04.062 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.10 to 1 (10000) 2024-05-15T13:16:05.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:05 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T13:16:05.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:05 smithi027 ceph-mon[39795]: osdmap e685: 12 total, 12 up, 10 in 2024-05-15T13:16:05.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:05 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 11136/192897 objects degraded (5.773%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:05.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:05 smithi027 ceph-mon[39795]: osdmap e686: 12 total, 12 up, 10 in 2024-05-15T13:16:05.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:05 smithi027 ceph-mon[39795]: pgmap v1778: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 442 op/s; 11165/189873 objects degraded (5.880%); 28909/189873 objects misplaced (15.225%); 18 MiB/s, 288 objects/s recovering 2024-05-15T13:16:05.286 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:05 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T13:16:05.286 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:05 smithi120 ceph-mon[43961]: osdmap e685: 12 total, 12 up, 10 in 2024-05-15T13:16:05.286 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:05 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 11136/192897 objects degraded (5.773%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:05.286 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:05 smithi120 ceph-mon[43961]: osdmap e686: 12 total, 12 up, 10 in 2024-05-15T13:16:05.286 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:05 smithi120 ceph-mon[43961]: pgmap v1778: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 442 op/s; 11165/189873 objects degraded (5.880%); 28909/189873 objects misplaced (15.225%); 18 MiB/s, 288 objects/s recovering 2024-05-15T13:16:05.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:05 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T13:16:05.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:05 smithi177 ceph-mon[44303]: osdmap e685: 12 total, 12 up, 10 in 2024-05-15T13:16:05.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:05 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 11136/192897 objects degraded (5.773%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:05.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:05 smithi177 ceph-mon[44303]: osdmap e686: 12 total, 12 up, 10 in 2024-05-15T13:16:05.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:05 smithi177 ceph-mon[44303]: pgmap v1778: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 442 op/s; 11165/189873 objects degraded (5.880%); 28909/189873 objects misplaced (15.225%); 18 MiB/s, 288 objects/s recovering 2024-05-15T13:16:06.158 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:16:06.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:06.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:16:06.264 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:16:06.337 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:06.420 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:16:06.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:06.423 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:16:06.524 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:16:06.562 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:06.589 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:16:06.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:06.596 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:16:06.697 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:16:06.814 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:07.748 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:07 smithi027 ceph-mon[39795]: pgmap v1779: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 354 op/s; 11165/189495 objects degraded (5.892%); 28909/189495 objects misplaced (15.256%); 14 MiB/s, 231 objects/s recovering 2024-05-15T13:16:07.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:07 smithi120 ceph-mon[43961]: pgmap v1779: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 354 op/s; 11165/189495 objects degraded (5.892%); 28909/189495 objects misplaced (15.256%); 14 MiB/s, 231 objects/s recovering 2024-05-15T13:16:07.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:07 smithi177 ceph-mon[44303]: pgmap v1779: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 354 op/s; 11165/189495 objects degraded (5.892%); 28909/189495 objects misplaced (15.256%); 14 MiB/s, 231 objects/s recovering 2024-05-15T13:16:08.044 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:16:08.044 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:16:08.422 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:16:08.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:08.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:16:08.532 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:16:08.581 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:08.648 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:16:08.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:08.652 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:16:08.753 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:16:08.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:08 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:16:08.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:08 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:16:08.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:08 smithi027 ceph-mon[39795]: osdmap e687: 12 total, 12 up, 10 in 2024-05-15T13:16:08.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:08 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:16:08.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:08 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:16:08.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:08 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:16:08.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:08 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:16:08.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:08 smithi120 ceph-mon[43961]: osdmap e687: 12 total, 12 up, 10 in 2024-05-15T13:16:08.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:08 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:16:08.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:08 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:16:08.788 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:08.828 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:16:08.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:08.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:16:08.832 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:08 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:16:08.832 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:08 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:16:08.832 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:08 smithi177 ceph-mon[44303]: osdmap e687: 12 total, 12 up, 10 in 2024-05-15T13:16:08.833 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:08 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:16:08.833 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:08 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:16:08.854 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 11 1 2024-05-15T13:16:08.933 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:16:08.956 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-05-15T13:16:08.965 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:08.990 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:16:08.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:08.993 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:16:09.095 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:16:09.232 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:09.252 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:16:09.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:09.321 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:16:09.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:16:09.422 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:16:09.454 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:09.509 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:16:09.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:09.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:16:09.613 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:16:09.645 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:09.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:09 smithi120 ceph-mon[43961]: pgmap v1781: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 413 op/s; 10958/185442 objects degraded (5.909%); 28257/185442 objects misplaced (15.238%); 15 MiB/s, 247 objects/s recovering 2024-05-15T13:16:09.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:09 smithi120 ceph-mon[43961]: osdmap e688: 12 total, 12 up, 10 in 2024-05-15T13:16:09.787 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:09 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 10958/185442 objects degraded (5.909%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:09.794 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:16:09.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:09.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:16:09.898 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:16:09.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:09 smithi177 ceph-mon[44303]: pgmap v1781: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 413 op/s; 10958/185442 objects degraded (5.909%); 28257/185442 objects misplaced (15.238%); 15 MiB/s, 247 objects/s recovering 2024-05-15T13:16:09.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:09 smithi177 ceph-mon[44303]: osdmap e688: 12 total, 12 up, 10 in 2024-05-15T13:16:09.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:09 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 10958/185442 objects degraded (5.909%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:10.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:09 smithi027 ceph-mon[39795]: pgmap v1781: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 413 op/s; 10958/185442 objects degraded (5.909%); 28257/185442 objects misplaced (15.238%); 15 MiB/s, 247 objects/s recovering 2024-05-15T13:16:10.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:09 smithi027 ceph-mon[39795]: osdmap e688: 12 total, 12 up, 10 in 2024-05-15T13:16:10.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:09 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 10958/185442 objects degraded (5.909%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:10.043 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:16:10.043 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:10.095 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:16:10.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:10.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:16:10.131 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:16:10.208 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:16:10.238 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:10.288 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:16:10.288 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:16:10.310 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:16:10.365 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:16:10.399 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:16:10.432 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:16:10.554 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:16:10.554 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:16:10.569 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:16:10.609 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:16:10.621 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:16:10.684 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:16:10.688 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:16:10.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:10.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:16:10.731 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:16:10.732 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:16:10.732 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:16:10.732 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:16:10.732 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:16:10.732 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:16:10.732 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:16:10.732 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:16:10.732 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:16:10.732 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:16:10.732 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:16:10.732 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:16:10.792 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:16:10.882 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:10 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:16:10.882 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:10 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:16:10.882 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:10 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3019622645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:16:10.882 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:10 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3997202466' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:16:10.882 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:10 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:16:10.887 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:10.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:10 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:16:10.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:10 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:16:10.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:10 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3019622645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:16:10.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:10 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3997202466' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:16:10.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:10 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:16:11.028 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:10 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:16:11.028 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:10 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:16:11.028 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:10 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3019622645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:16:11.028 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:10 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3997202466' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:16:11.028 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:10 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:16:11.328 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:16:11.328 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T13:16:11.329 INFO:teuthology.orchestra.run.smithi120.stdout: "num_ops": 0 2024-05-15T13:16:11.329 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:16:11.755 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:11 smithi120 ceph-mon[43961]: pgmap v1783: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.7 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 356 op/s; 11004/183390 objects degraded (6.000%); 27685/183390 objects misplaced (15.096%); 14 MiB/s, 222 objects/s recovering 2024-05-15T13:16:11.755 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:11 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:16:11.755 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:11 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:16:11.755 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:11 smithi120 ceph-mon[43961]: osdmap e689: 12 total, 12 up, 10 in 2024-05-15T13:16:11.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:11 smithi177 ceph-mon[44303]: pgmap v1783: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.7 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 356 op/s; 11004/183390 objects degraded (6.000%); 27685/183390 objects misplaced (15.096%); 14 MiB/s, 222 objects/s recovering 2024-05-15T13:16:11.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:11 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:16:11.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:11 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:16:11.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:11 smithi177 ceph-mon[44303]: osdmap e689: 12 total, 12 up, 10 in 2024-05-15T13:16:12.036 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:11 smithi027 ceph-mon[39795]: pgmap v1783: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.7 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 356 op/s; 11004/183390 objects degraded (6.000%); 27685/183390 objects misplaced (15.096%); 14 MiB/s, 222 objects/s recovering 2024-05-15T13:16:12.036 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:11 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:16:12.036 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:11 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:16:12.036 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:11 smithi027 ceph-mon[39795]: osdmap e689: 12 total, 12 up, 10 in 2024-05-15T13:16:12.137 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-05-15T13:16:12.442 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:16:12.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:12 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:16:12.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:12 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:16:13.024 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:12 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:16:13.024 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:12 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:16:13.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:12 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:16:13.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:12 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:16:13.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:13 smithi120 ceph-mon[43961]: pgmap v1785: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.7 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 515 op/s; 10758/180204 objects degraded (5.970%); 27290/180204 objects misplaced (15.144%); 18 MiB/s, 293 objects/s recovering 2024-05-15T13:16:13.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:13 smithi120 ceph-mon[43961]: osdmap e690: 12 total, 12 up, 10 in 2024-05-15T13:16:13.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:13 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:16:13.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:13 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:16:13.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:13 smithi177 ceph-mon[44303]: pgmap v1785: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.7 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 515 op/s; 10758/180204 objects degraded (5.970%); 27290/180204 objects misplaced (15.144%); 18 MiB/s, 293 objects/s recovering 2024-05-15T13:16:13.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:13 smithi177 ceph-mon[44303]: osdmap e690: 12 total, 12 up, 10 in 2024-05-15T13:16:13.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:13 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:16:13.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:13 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:16:14.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:13 smithi027 ceph-mon[39795]: pgmap v1785: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.7 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 515 op/s; 10758/180204 objects degraded (5.970%); 27290/180204 objects misplaced (15.144%); 18 MiB/s, 293 objects/s recovering 2024-05-15T13:16:14.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:13 smithi027 ceph-mon[39795]: osdmap e690: 12 total, 12 up, 10 in 2024-05-15T13:16:14.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:13 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:16:14.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:13 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:16:14.626 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:16:14.626 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T13:16:14.626 INFO:teuthology.orchestra.run.smithi120.stdout: "complaint_time": 30, 2024-05-15T13:16:14.626 INFO:teuthology.orchestra.run.smithi120.stdout: "num_blocked_ops": 0 2024-05-15T13:16:14.626 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:16:14.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:14 smithi177 ceph-mon[44303]: osdmap e691: 12 total, 12 up, 10 in 2024-05-15T13:16:14.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:14 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 10758/180204 objects degraded (5.970%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:14.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:14 smithi177 ceph-mon[44303]: osdmap e692: 12 total, 12 up, 10 in 2024-05-15T13:16:15.021 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:14 smithi120 ceph-mon[43961]: osdmap e691: 12 total, 12 up, 10 in 2024-05-15T13:16:15.022 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:14 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 10758/180204 objects degraded (5.970%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:15.022 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:14 smithi120 ceph-mon[43961]: osdmap e692: 12 total, 12 up, 10 in 2024-05-15T13:16:15.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:14 smithi027 ceph-mon[39795]: osdmap e691: 12 total, 12 up, 10 in 2024-05-15T13:16:15.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:14 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 10758/180204 objects degraded (5.970%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:15.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:14 smithi027 ceph-mon[39795]: osdmap e692: 12 total, 12 up, 10 in 2024-05-15T13:16:15.489 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-05-15T13:16:15.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:15 smithi120 ceph-mon[43961]: pgmap v1789: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 534 op/s; 10700/176979 objects degraded (6.046%); 26547/176979 objects misplaced (15.000%); 19 MiB/s, 324 objects/s recovering 2024-05-15T13:16:15.794 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:16:15.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:15 smithi177 ceph-mon[44303]: pgmap v1789: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 534 op/s; 10700/176979 objects degraded (6.046%); 26547/176979 objects misplaced (15.000%); 19 MiB/s, 324 objects/s recovering 2024-05-15T13:16:15.950 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:15 smithi027 ceph-mon[39795]: pgmap v1789: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 534 op/s; 10700/176979 objects degraded (6.046%); 26547/176979 objects misplaced (15.000%); 19 MiB/s, 324 objects/s recovering 2024-05-15T13:16:15.950 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:16:15.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:15.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:16:16.075 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:16:16.114 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:16.142 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:16:16.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:16.149 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:16:16.250 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:16:16.282 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:16.311 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:16:16.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:16.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:16:16.415 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:16:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:16.583 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:16:16.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:16.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:16:16.690 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:16:16.749 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:17.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:17 smithi177 ceph-mon[44303]: pgmap v1790: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 386 op/s; 10700/176556 objects degraded (6.060%); 26547/176556 objects misplaced (15.036%); 13 MiB/s, 220 objects/s recovering 2024-05-15T13:16:17.988 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:16:17.988 INFO:teuthology.orchestra.run.smithi120.stdout: "size": 20, 2024-05-15T13:16:17.988 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 600, 2024-05-15T13:16:17.988 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [ 2024-05-15T13:16:17.988 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.988 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:196513 6.5 6:a7f6010f:::benchmark_data_smithi177_122278_object66142:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e554)", 2024-05-15T13:16:17.988 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:10:41.044634+0000", 2024-05-15T13:16:17.989 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 336.94558726999998, 2024-05-15T13:16:17.989 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.121536344, 2024-05-15T13:16:17.989 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:16:17.989 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:17.989 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:16:17.989 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:16:17.989 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:16:17.989 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 196513 2024-05-15T13:16:17.989 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.989 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:16:17.989 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.989 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:16:17.989 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.044634+0000", 2024-05-15T13:16:17.990 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:17.990 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.990 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.990 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:16:17.990 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.044634+0000", 2024-05-15T13:16:17.990 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:17.990 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.990 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.990 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:16:17.990 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.044636+0000", 2024-05-15T13:16:17.990 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.669e-06 2024-05-15T13:16:17.991 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.991 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.991 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:16:17.991 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.044643+0000", 2024-05-15T13:16:17.991 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8539999999999999e-06 2024-05-15T13:16:17.991 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.991 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.991 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:16:17.991 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.044658+0000", 2024-05-15T13:16:17.991 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5885000000000001e-05 2024-05-15T13:16:17.991 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.991 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.991 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:16:17.992 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.044667+0000", 2024-05-15T13:16:17.992 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1210000000000007e-06 2024-05-15T13:16:17.992 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.992 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.992 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:17.992 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.046779+0000", 2024-05-15T13:16:17.992 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0021125480000000001 2024-05-15T13:16:17.992 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.992 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.992 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:16:17.992 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.046920+0000", 2024-05-15T13:16:17.992 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014129400000000001 2024-05-15T13:16:17.992 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.993 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.993 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,9", 2024-05-15T13:16:17.993 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.047046+0000", 2024-05-15T13:16:17.993 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012540399999999999 2024-05-15T13:16:17.993 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.993 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.993 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:17.993 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.052791+0000", 2024-05-15T13:16:17.993 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0057451710000000003 2024-05-15T13:16:17.993 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.993 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.993 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:16:17.993 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.052798+0000", 2024-05-15T13:16:17.994 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.1400000000000002e-06 2024-05-15T13:16:17.994 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.994 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.994 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:17.994 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.166099+0000", 2024-05-15T13:16:17.994 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.113301033 2024-05-15T13:16:17.994 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.994 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.994 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:16:17.994 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.166140+0000", 2024-05-15T13:16:17.994 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0991000000000001e-05 2024-05-15T13:16:17.994 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.994 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.995 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:16:17.995 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.166170+0000", 2024-05-15T13:16:17.995 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.0233999999999999e-05 2024-05-15T13:16:17.995 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:17.995 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:16:17.995 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:17.995 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.995 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.995 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:196524 6.5 6:a31c859f:::benchmark_data_smithi177_122278_object66153:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e554)", 2024-05-15T13:16:17.995 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:10:41.061357+0000", 2024-05-15T13:16:17.995 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 336.928864207, 2024-05-15T13:16:17.995 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.10946439299999999, 2024-05-15T13:16:17.995 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:16:17.995 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:17.996 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:16:17.996 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:16:17.996 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:16:17.996 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 196524 2024-05-15T13:16:17.996 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.996 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:16:17.996 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.996 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:16:17.996 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.061357+0000", 2024-05-15T13:16:17.996 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:17.996 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.996 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.996 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:16:17.997 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.061357+0000", 2024-05-15T13:16:17.997 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:17.997 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.997 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.997 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:16:17.997 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.061359+0000", 2024-05-15T13:16:17.997 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5820000000000001e-06 2024-05-15T13:16:17.997 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.997 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.997 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:16:17.997 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.061364+0000", 2024-05-15T13:16:17.997 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3279999999999996e-06 2024-05-15T13:16:17.997 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.998 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.998 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:16:17.998 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.061376+0000", 2024-05-15T13:16:17.998 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2272e-05 2024-05-15T13:16:17.998 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.998 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.998 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:16:17.998 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.061383+0000", 2024-05-15T13:16:17.998 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9940000000000003e-06 2024-05-15T13:16:17.998 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.998 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.998 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:17.998 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.061435+0000", 2024-05-15T13:16:17.998 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1354999999999999e-05 2024-05-15T13:16:17.999 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.999 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.999 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:16:17.999 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.061542+0000", 2024-05-15T13:16:17.999 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000107404 2024-05-15T13:16:17.999 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.999 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.999 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,9", 2024-05-15T13:16:17.999 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.061647+0000", 2024-05-15T13:16:17.999 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000104961 2024-05-15T13:16:17.999 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:17.999 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:17.999 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:18.000 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.068785+0000", 2024-05-15T13:16:18.000 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0071377430000000002 2024-05-15T13:16:18.000 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.000 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.000 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:16:18.000 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.071718+0000", 2024-05-15T13:16:18.000 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0029331299999999999 2024-05-15T13:16:18.000 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.000 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.000 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:18.000 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.170710+0000", 2024-05-15T13:16:18.000 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.098992245000000006 2024-05-15T13:16:18.000 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.000 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.001 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:16:18.001 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.170770+0000", 2024-05-15T13:16:18.001 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9695999999999999e-05 2024-05-15T13:16:18.001 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.001 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.001 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:16:18.001 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.170821+0000", 2024-05-15T13:16:18.001 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1683e-05 2024-05-15T13:16:18.001 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.001 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:16:18.001 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.001 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.001 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.001 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.25735.0:259106 6.b e584/509 6:d8f2a530:::benchmark_data_smithi177_122278_object128735:head v 584'16505, mlcod=583'16503)", 2024-05-15T13:16:18.002 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:12:04.595237+0000", 2024-05-15T13:16:18.002 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 253.39498432100001, 2024-05-15T13:16:18.002 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.096121385000000004, 2024-05-15T13:16:18.002 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:16:18.002 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:18.002 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:16:18.002 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.002 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:16:18.002 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:12:04.595237+0000", 2024-05-15T13:16:18.002 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.002 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.002 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.002 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:16:18.003 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:12:04.595237+0000", 2024-05-15T13:16:18.003 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.003 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.003 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.003 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:16:18.003 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:12:04.595238+0000", 2024-05-15T13:16:18.003 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.3399999999999997e-07 2024-05-15T13:16:18.003 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.003 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.003 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:16:18.003 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:12:04.595244+0000", 2024-05-15T13:16:18.003 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1990000000000003e-06 2024-05-15T13:16:18.003 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.004 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.004 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:16:18.004 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:12:04.595253+0000", 2024-05-15T13:16:18.004 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.2580000000000001e-06 2024-05-15T13:16:18.004 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.004 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.004 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:16:18.004 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:12:04.595260+0000", 2024-05-15T13:16:18.004 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.5309999999999998e-06 2024-05-15T13:16:18.004 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.004 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.004 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.004 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:12:04.595327+0000", 2024-05-15T13:16:18.004 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6806999999999999e-05 2024-05-15T13:16:18.005 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.005 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.005 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:16:18.005 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:12:04.595340+0000", 2024-05-15T13:16:18.005 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3628000000000001e-05 2024-05-15T13:16:18.005 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.005 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.005 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:16:18.005 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:12:04.691316+0000", 2024-05-15T13:16:18.005 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.095975817000000005 2024-05-15T13:16:18.005 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.005 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.005 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:16:18.006 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:12:04.691358+0000", 2024-05-15T13:16:18.006 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2211e-05 2024-05-15T13:16:18.006 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.006 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:16:18.006 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.006 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.006 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.006 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.25735.0:259115 6.b e584/509 6:ddb8c34d:::benchmark_data_smithi177_122278_object128744:head v 584'16506, mlcod=583'16503)", 2024-05-15T13:16:18.006 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:12:04.600690+0000", 2024-05-15T13:16:18.006 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 253.38953135, 2024-05-15T13:16:18.006 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.090687093999999996, 2024-05-15T13:16:18.006 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:16:18.006 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:18.007 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:16:18.007 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.007 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:16:18.007 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:12:04.600690+0000", 2024-05-15T13:16:18.007 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.007 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.007 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.007 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:16:18.007 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:12:04.600690+0000", 2024-05-15T13:16:18.007 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.007 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.007 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.007 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:16:18.007 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:12:04.600690+0000", 2024-05-15T13:16:18.008 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.53e-07 2024-05-15T13:16:18.008 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.008 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.008 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:16:18.008 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:12:04.600693+0000", 2024-05-15T13:16:18.008 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.7269999999999999e-06 2024-05-15T13:16:18.008 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.008 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.008 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:16:18.008 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:12:04.600699+0000", 2024-05-15T13:16:18.008 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5010000000000004e-06 2024-05-15T13:16:18.008 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.008 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.008 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:16:18.009 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:12:04.600702+0000", 2024-05-15T13:16:18.009 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7560000000000001e-06 2024-05-15T13:16:18.009 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.009 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.009 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.009 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:12:04.600752+0000", 2024-05-15T13:16:18.009 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0099999999999998e-05 2024-05-15T13:16:18.009 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.009 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.009 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:16:18.009 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:12:04.600772+0000", 2024-05-15T13:16:18.009 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9343000000000001e-05 2024-05-15T13:16:18.009 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.010 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.010 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:16:18.010 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:12:04.691362+0000", 2024-05-15T13:16:18.010 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.090590052000000004 2024-05-15T13:16:18.010 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.010 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.010 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:16:18.010 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:12:04.691377+0000", 2024-05-15T13:16:18.010 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5262000000000001e-05 2024-05-15T13:16:18.010 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.010 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:16:18.010 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.010 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.011 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.011 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:43851 7.c 7:3704fd17:::benchmark_data_smithi177_134484_object43850:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e611)", 2024-05-15T13:16:18.011 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:13:03.240645+0000", 2024-05-15T13:16:18.011 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 194.749576338, 2024-05-15T13:16:18.011 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.11324569499999999, 2024-05-15T13:16:18.011 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:16:18.011 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:18.011 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:16:18.011 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:16:18.011 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:16:18.011 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 43851 2024-05-15T13:16:18.011 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.011 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:16:18.011 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.012 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:16:18.012 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.240645+0000", 2024-05-15T13:16:18.012 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.012 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.012 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.012 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:16:18.012 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.240645+0000", 2024-05-15T13:16:18.012 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.012 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.012 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.012 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:16:18.012 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.240647+0000", 2024-05-15T13:16:18.012 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.317e-06 2024-05-15T13:16:18.012 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.013 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.013 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:16:18.013 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.240699+0000", 2024-05-15T13:16:18.013 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1545999999999997e-05 2024-05-15T13:16:18.013 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.013 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.013 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:16:18.013 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.240746+0000", 2024-05-15T13:16:18.013 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7039000000000001e-05 2024-05-15T13:16:18.013 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.013 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.013 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:16:18.013 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.240762+0000", 2024-05-15T13:16:18.013 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6336000000000001e-05 2024-05-15T13:16:18.013 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.014 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.014 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.014 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.240923+0000", 2024-05-15T13:16:18.014 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00016124299999999999 2024-05-15T13:16:18.014 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.014 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.014 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:16:18.014 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.241026+0000", 2024-05-15T13:16:18.014 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00010267099999999999 2024-05-15T13:16:18.014 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.014 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.014 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,11", 2024-05-15T13:16:18.014 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.241117+0000", 2024-05-15T13:16:18.014 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0686999999999994e-05 2024-05-15T13:16:18.015 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.015 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.015 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:16:18.015 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.242975+0000", 2024-05-15T13:16:18.015 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0018577260000000001 2024-05-15T13:16:18.015 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.015 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.015 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:18.015 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.246022+0000", 2024-05-15T13:16:18.015 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.003047045 2024-05-15T13:16:18.015 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.015 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.015 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:18.015 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.353798+0000", 2024-05-15T13:16:18.016 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.10777653 2024-05-15T13:16:18.016 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.016 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.016 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:16:18.016 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.353850+0000", 2024-05-15T13:16:18.016 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1975999999999998e-05 2024-05-15T13:16:18.016 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.016 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.016 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:16:18.016 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.353891+0000", 2024-05-15T13:16:18.016 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0578999999999999e-05 2024-05-15T13:16:18.016 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.016 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:16:18.016 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.017 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.017 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.017 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:43861 7.c 7:376f22d6:::benchmark_data_smithi177_134484_object43860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e611)", 2024-05-15T13:16:18.017 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:13:03.241012+0000", 2024-05-15T13:16:18.017 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 194.74920933600001, 2024-05-15T13:16:18.017 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.11298029900000001, 2024-05-15T13:16:18.017 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:16:18.017 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:18.017 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:16:18.017 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:16:18.017 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:16:18.017 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 43861 2024-05-15T13:16:18.017 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.018 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:16:18.018 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.018 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:16:18.018 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.241012+0000", 2024-05-15T13:16:18.018 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.018 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.018 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.018 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:16:18.018 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.241012+0000", 2024-05-15T13:16:18.018 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.018 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.018 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.018 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:16:18.019 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.241012+0000", 2024-05-15T13:16:18.019 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2200000000000004e-07 2024-05-15T13:16:18.019 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.019 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.019 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:16:18.019 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.241288+0000", 2024-05-15T13:16:18.019 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000275071 2024-05-15T13:16:18.019 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.019 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.019 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:16:18.019 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.241295+0000", 2024-05-15T13:16:18.019 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4420000000000001e-06 2024-05-15T13:16:18.019 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.019 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.020 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:16:18.020 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.241299+0000", 2024-05-15T13:16:18.020 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3490000000000002e-06 2024-05-15T13:16:18.020 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.020 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.020 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.020 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.242189+0000", 2024-05-15T13:16:18.020 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00088952599999999997 2024-05-15T13:16:18.020 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.020 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.020 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:16:18.020 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.242274+0000", 2024-05-15T13:16:18.020 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.5599000000000005e-05 2024-05-15T13:16:18.020 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.021 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.021 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,11", 2024-05-15T13:16:18.021 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.242365+0000", 2024-05-15T13:16:18.021 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0540999999999999e-05 2024-05-15T13:16:18.021 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.021 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.021 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:16:18.021 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.243074+0000", 2024-05-15T13:16:18.021 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00070889 2024-05-15T13:16:18.021 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.021 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.021 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:18.022 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.246066+0000", 2024-05-15T13:16:18.022 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0029917379999999999 2024-05-15T13:16:18.022 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.022 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.022 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:18.022 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.353932+0000", 2024-05-15T13:16:18.022 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.107866372 2024-05-15T13:16:18.022 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.022 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.022 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:16:18.022 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.353966+0000", 2024-05-15T13:16:18.022 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.4239999999999997e-05 2024-05-15T13:16:18.022 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.023 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.023 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:16:18.023 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:13:03.353992+0000", 2024-05-15T13:16:18.023 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.6009000000000001e-05 2024-05-15T13:16:18.023 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.023 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:16:18.023 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.023 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.023 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.023 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:123473 7.c 7:3abc2c57:::benchmark_data_smithi177_134484_object14408:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e655)", 2024-05-15T13:16:18.023 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:14:45.545521+0000", 2024-05-15T13:16:18.023 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 92.444700237000006, 2024-05-15T13:16:18.023 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.108908611, 2024-05-15T13:16:18.023 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:16:18.023 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:18.024 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:16:18.024 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:16:18.024 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:16:18.024 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 123473 2024-05-15T13:16:18.024 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.024 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:16:18.024 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.024 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:16:18.024 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.545521+0000", 2024-05-15T13:16:18.024 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.024 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.024 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.024 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:16:18.024 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.545521+0000", 2024-05-15T13:16:18.025 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.025 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.025 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.025 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:16:18.025 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.545522+0000", 2024-05-15T13:16:18.025 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0299999999999997e-07 2024-05-15T13:16:18.025 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.025 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.025 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:16:18.025 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.545526+0000", 2024-05-15T13:16:18.025 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.8020000000000002e-06 2024-05-15T13:16:18.025 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.025 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.025 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:16:18.026 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.545537+0000", 2024-05-15T13:16:18.026 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0838e-05 2024-05-15T13:16:18.026 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.026 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.026 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:16:18.026 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.545543+0000", 2024-05-15T13:16:18.026 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.0870000000000003e-06 2024-05-15T13:16:18.026 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.026 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.026 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.026 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.554556+0000", 2024-05-15T13:16:18.026 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0090130249999999992 2024-05-15T13:16:18.026 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.026 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.027 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:16:18.027 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.554662+0000", 2024-05-15T13:16:18.027 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00010603300000000001 2024-05-15T13:16:18.027 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.027 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.027 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 3,11", 2024-05-15T13:16:18.027 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.554757+0000", 2024-05-15T13:16:18.027 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.5206999999999998e-05 2024-05-15T13:16:18.027 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.027 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.027 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:16:18.027 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.563672+0000", 2024-05-15T13:16:18.027 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0089147329999999993 2024-05-15T13:16:18.027 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.028 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.028 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:18.028 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.563711+0000", 2024-05-15T13:16:18.028 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.9103999999999997e-05 2024-05-15T13:16:18.028 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.028 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.028 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:18.028 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.651823+0000", 2024-05-15T13:16:18.028 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.088112180999999998 2024-05-15T13:16:18.028 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.028 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.028 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:16:18.028 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.651867+0000", 2024-05-15T13:16:18.029 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4270000000000001e-05 2024-05-15T13:16:18.029 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.029 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.029 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:16:18.029 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.654430+0000", 2024-05-15T13:16:18.029 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0025624279999999998 2024-05-15T13:16:18.029 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.029 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:16:18.029 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.029 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.029 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.029 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:123476 7.c 7:3195af09:::benchmark_data_smithi177_134484_object14411:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e655)", 2024-05-15T13:16:18.029 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:14:45.549068+0000", 2024-05-15T13:16:18.029 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 92.441152837999994, 2024-05-15T13:16:18.030 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.10671456999999999, 2024-05-15T13:16:18.030 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:16:18.030 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:18.030 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:16:18.030 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:16:18.030 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:16:18.030 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 123476 2024-05-15T13:16:18.030 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.030 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:16:18.030 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.030 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:16:18.030 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.549068+0000", 2024-05-15T13:16:18.031 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.031 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.031 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.031 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:16:18.031 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.549068+0000", 2024-05-15T13:16:18.031 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.031 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.031 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.031 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:16:18.031 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.549069+0000", 2024-05-15T13:16:18.031 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.4900000000000004e-07 2024-05-15T13:16:18.031 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.031 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.031 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:16:18.031 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.549074+0000", 2024-05-15T13:16:18.032 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2080000000000002e-06 2024-05-15T13:16:18.032 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.032 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.032 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:16:18.032 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.549085+0000", 2024-05-15T13:16:18.032 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1423000000000001e-05 2024-05-15T13:16:18.032 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.032 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.032 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:16:18.032 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.549091+0000", 2024-05-15T13:16:18.032 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.806e-06 2024-05-15T13:16:18.032 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.032 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.032 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.033 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.560426+0000", 2024-05-15T13:16:18.033 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.01133501 2024-05-15T13:16:18.033 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.033 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.033 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:16:18.033 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.560574+0000", 2024-05-15T13:16:18.033 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000148298 2024-05-15T13:16:18.033 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.033 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.033 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 3,11", 2024-05-15T13:16:18.033 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.560724+0000", 2024-05-15T13:16:18.033 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014948500000000001 2024-05-15T13:16:18.033 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.033 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.034 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:18.034 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.563893+0000", 2024-05-15T13:16:18.034 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0031697259999999999 2024-05-15T13:16:18.034 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.034 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.034 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:16:18.034 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.563910+0000", 2024-05-15T13:16:18.034 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7042999999999999e-05 2024-05-15T13:16:18.034 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.034 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.034 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:18.034 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.654479+0000", 2024-05-15T13:16:18.034 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.090568698000000003 2024-05-15T13:16:18.034 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.035 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.035 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:16:18.035 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.654515+0000", 2024-05-15T13:16:18.035 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5981e-05 2024-05-15T13:16:18.035 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.035 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.035 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:16:18.035 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:45.655783+0000", 2024-05-15T13:16:18.035 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0012679430000000001 2024-05-15T13:16:18.035 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.035 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:16:18.035 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.035 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.035 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.036 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:125708 7.7 7.7b696437 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e657)", 2024-05-15T13:16:18.036 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:14:50.450084+0000", 2024-05-15T13:16:18.036 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 87.540137232999996, 2024-05-15T13:16:18.036 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.004634933, 2024-05-15T13:16:18.036 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:16:18.036 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:16:18.036 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:16:18.036 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:16:18.036 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:16:18.036 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 125708 2024-05-15T13:16:18.036 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.036 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:16:18.036 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.037 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:16:18.037 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.450084+0000", 2024-05-15T13:16:18.037 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.037 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.037 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.037 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:16:18.037 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.450084+0000", 2024-05-15T13:16:18.037 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.037 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.037 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.037 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:16:18.037 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.450085+0000", 2024-05-15T13:16:18.037 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.393e-06 2024-05-15T13:16:18.038 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.038 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.038 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:16:18.038 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.450092+0000", 2024-05-15T13:16:18.038 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6059999999999999e-06 2024-05-15T13:16:18.038 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.038 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.038 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:16:18.038 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.450108+0000", 2024-05-15T13:16:18.038 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5543999999999999e-05 2024-05-15T13:16:18.038 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.038 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.038 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:16:18.039 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.450116+0000", 2024-05-15T13:16:18.039 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8650000000000007e-06 2024-05-15T13:16:18.039 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.039 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.039 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.039 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.456666+0000", 2024-05-15T13:16:18.039 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0065495869999999999 2024-05-15T13:16:18.039 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.039 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.039 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:16:18.039 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.456675+0000", 2024-05-15T13:16:18.039 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.054e-06 2024-05-15T13:16:18.039 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.040 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.040 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.040 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454707+0000", 2024-05-15T13:16:18.040 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.99803161399999996 2024-05-15T13:16:18.040 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.040 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.040 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:16:18.040 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454719+0000", 2024-05-15T13:16:18.040 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2269999999999999e-05 2024-05-15T13:16:18.040 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.040 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:16:18.040 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.040 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.040 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.041 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:125717 7.d 7.75ad9b4d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-05-15T13:16:18.041 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:14:50.461218+0000", 2024-05-15T13:16:18.041 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 87.529003302999996, 2024-05-15T13:16:18.041 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.99365158099999995, 2024-05-15T13:16:18.041 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:16:18.041 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:16:18.041 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:16:18.041 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:16:18.041 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:16:18.041 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 125717 2024-05-15T13:16:18.041 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.041 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:16:18.041 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.042 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:16:18.042 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.461218+0000", 2024-05-15T13:16:18.042 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.042 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.042 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.042 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:16:18.042 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.461218+0000", 2024-05-15T13:16:18.042 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.042 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.043 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.043 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:16:18.043 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.461219+0000", 2024-05-15T13:16:18.043 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.449e-06 2024-05-15T13:16:18.043 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.043 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.043 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:16:18.043 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.461225+0000", 2024-05-15T13:16:18.043 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7919999999999998e-06 2024-05-15T13:16:18.043 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.043 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.043 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:16:18.043 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.461237+0000", 2024-05-15T13:16:18.044 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2187e-05 2024-05-15T13:16:18.044 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.044 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.044 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:16:18.044 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.461260+0000", 2024-05-15T13:16:18.044 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2277999999999999e-05 2024-05-15T13:16:18.044 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.044 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.044 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.044 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.463106+0000", 2024-05-15T13:16:18.044 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001846721 2024-05-15T13:16:18.044 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.044 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.045 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:16:18.045 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.463117+0000", 2024-05-15T13:16:18.045 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0231000000000001e-05 2024-05-15T13:16:18.045 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.045 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.045 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.045 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454858+0000", 2024-05-15T13:16:18.045 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.99174162499999996 2024-05-15T13:16:18.045 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.045 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.045 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:16:18.045 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454870+0000", 2024-05-15T13:16:18.045 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1297999999999999e-05 2024-05-15T13:16:18.046 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.046 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:16:18.046 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.046 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.046 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.046 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:125719 7.7 7.fd392cc7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-05-15T13:16:18.046 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:14:50.470608+0000", 2024-05-15T13:16:18.046 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 87.519613720999999, 2024-05-15T13:16:18.046 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.98416746300000002, 2024-05-15T13:16:18.046 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:16:18.046 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:16:18.046 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:16:18.047 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:16:18.047 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:16:18.047 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 125719 2024-05-15T13:16:18.047 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.047 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:16:18.047 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.047 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:16:18.047 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.470608+0000", 2024-05-15T13:16:18.047 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.047 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.047 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.047 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:16:18.047 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.470608+0000", 2024-05-15T13:16:18.048 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.048 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.048 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.048 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:16:18.048 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.470609+0000", 2024-05-15T13:16:18.048 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.361e-06 2024-05-15T13:16:18.048 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.048 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.048 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:16:18.048 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.470616+0000", 2024-05-15T13:16:18.048 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.52e-06 2024-05-15T13:16:18.048 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.048 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.048 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:16:18.049 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.470630+0000", 2024-05-15T13:16:18.049 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3389e-05 2024-05-15T13:16:18.049 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.049 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.049 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:16:18.049 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.470641+0000", 2024-05-15T13:16:18.049 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1181e-05 2024-05-15T13:16:18.049 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.049 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.049 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.049 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.470699+0000", 2024-05-15T13:16:18.049 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7704000000000002e-05 2024-05-15T13:16:18.049 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.050 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.050 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:16:18.050 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.470709+0000", 2024-05-15T13:16:18.050 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.825e-06 2024-05-15T13:16:18.050 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.050 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.050 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.050 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454764+0000", 2024-05-15T13:16:18.050 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.98405541100000005 2024-05-15T13:16:18.050 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.050 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.050 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:16:18.051 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454775+0000", 2024-05-15T13:16:18.051 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1072e-05 2024-05-15T13:16:18.051 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.051 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:16:18.051 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.051 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.051 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.051 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:125735 7.d 7.cdea4a0d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-05-15T13:16:18.051 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:14:50.508723+0000", 2024-05-15T13:16:18.051 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 87.481498332000001, 2024-05-15T13:16:18.051 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.94617932000000005, 2024-05-15T13:16:18.051 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:16:18.051 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:16:18.051 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:16:18.052 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:16:18.052 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:16:18.052 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 125735 2024-05-15T13:16:18.052 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.052 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:16:18.052 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.052 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:16:18.052 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.508723+0000", 2024-05-15T13:16:18.052 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.052 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.052 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.052 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:16:18.052 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.508723+0000", 2024-05-15T13:16:18.053 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.053 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.053 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.053 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:16:18.053 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.508731+0000", 2024-05-15T13:16:18.053 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7500000000000003e-06 2024-05-15T13:16:18.053 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.053 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.053 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:16:18.053 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.508737+0000", 2024-05-15T13:16:18.053 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8259999999999998e-06 2024-05-15T13:16:18.053 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.053 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.053 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:16:18.054 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.508749+0000", 2024-05-15T13:16:18.054 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2456999999999999e-05 2024-05-15T13:16:18.054 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.054 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.054 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:16:18.054 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.508757+0000", 2024-05-15T13:16:18.054 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9570000000000002e-06 2024-05-15T13:16:18.054 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.054 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.054 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.054 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.508805+0000", 2024-05-15T13:16:18.054 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8238999999999997e-05 2024-05-15T13:16:18.054 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.055 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.055 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:16:18.055 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.508814+0000", 2024-05-15T13:16:18.055 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1989999999999995e-06 2024-05-15T13:16:18.055 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.055 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.055 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.055 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454896+0000", 2024-05-15T13:16:18.055 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.94608192899999999 2024-05-15T13:16:18.055 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.055 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.055 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:16:18.055 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454902+0000", 2024-05-15T13:16:18.055 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9630000000000003e-06 2024-05-15T13:16:18.056 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.056 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:16:18.056 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.056 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.056 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.056 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:125762 7.7 7.81691657 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-05-15T13:16:18.056 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:14:50.586614+0000", 2024-05-15T13:16:18.056 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 87.403607574999995, 2024-05-15T13:16:18.056 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.86822153999999996, 2024-05-15T13:16:18.056 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:16:18.056 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:16:18.056 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:16:18.056 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:16:18.057 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:16:18.057 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 125762 2024-05-15T13:16:18.057 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.057 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:16:18.057 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.057 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:16:18.057 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.586614+0000", 2024-05-15T13:16:18.057 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.057 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.057 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.057 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:16:18.057 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.586614+0000", 2024-05-15T13:16:18.057 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.057 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.058 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.058 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:16:18.058 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.586615+0000", 2024-05-15T13:16:18.058 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.466e-06 2024-05-15T13:16:18.058 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.058 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.058 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:16:18.058 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.586621+0000", 2024-05-15T13:16:18.058 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9150000000000001e-06 2024-05-15T13:16:18.058 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.058 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.058 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:16:18.058 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.586637+0000", 2024-05-15T13:16:18.058 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6158999999999999e-05 2024-05-15T13:16:18.059 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.059 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.059 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:16:18.059 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.586660+0000", 2024-05-15T13:16:18.059 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2296000000000001e-05 2024-05-15T13:16:18.059 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.059 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.059 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.059 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.586741+0000", 2024-05-15T13:16:18.059 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1347000000000006e-05 2024-05-15T13:16:18.059 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.059 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.059 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:16:18.059 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.586752+0000", 2024-05-15T13:16:18.059 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0893000000000001e-05 2024-05-15T13:16:18.060 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.060 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.060 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.060 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454813+0000", 2024-05-15T13:16:18.060 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.86806092599999995 2024-05-15T13:16:18.060 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.060 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.060 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:16:18.060 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454835+0000", 2024-05-15T13:16:18.060 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2538e-05 2024-05-15T13:16:18.060 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.060 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:16:18.060 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.061 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.061 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.061 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:125785 7.d 7.270a8e6d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-05-15T13:16:18.061 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:14:50.672780+0000", 2024-05-15T13:16:18.061 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 87.317441312, 2024-05-15T13:16:18.061 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.78215774000000005, 2024-05-15T13:16:18.061 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:16:18.061 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:16:18.061 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:16:18.061 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:16:18.061 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:16:18.061 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 125785 2024-05-15T13:16:18.061 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.062 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:16:18.062 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.062 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:16:18.062 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.672780+0000", 2024-05-15T13:16:18.062 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.062 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.062 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.062 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:16:18.062 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.672780+0000", 2024-05-15T13:16:18.062 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.062 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.062 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.062 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:16:18.063 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.672782+0000", 2024-05-15T13:16:18.063 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.635e-06 2024-05-15T13:16:18.063 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.063 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.063 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:16:18.063 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.672789+0000", 2024-05-15T13:16:18.063 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2649999999999999e-06 2024-05-15T13:16:18.063 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.063 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.063 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:16:18.063 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.672804+0000", 2024-05-15T13:16:18.063 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5328999999999998e-05 2024-05-15T13:16:18.063 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.064 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.064 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:16:18.064 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.672812+0000", 2024-05-15T13:16:18.064 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1699999999999997e-06 2024-05-15T13:16:18.064 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.064 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.064 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.064 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.672869+0000", 2024-05-15T13:16:18.064 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6779e-05 2024-05-15T13:16:18.064 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.064 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.064 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:16:18.064 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.672878+0000", 2024-05-15T13:16:18.064 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1670000000000005e-06 2024-05-15T13:16:18.065 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.065 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.065 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.065 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454929+0000", 2024-05-15T13:16:18.065 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.78205035000000001 2024-05-15T13:16:18.065 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.065 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.065 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:16:18.065 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454938+0000", 2024-05-15T13:16:18.065 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0450000000000006e-06 2024-05-15T13:16:18.065 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.065 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:16:18.065 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.065 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.065 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.066 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:125787 7.7 7.a38c5a27 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-05-15T13:16:18.066 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:14:50.678532+0000", 2024-05-15T13:16:18.066 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 87.311689134999995, 2024-05-15T13:16:18.066 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.77632293600000002, 2024-05-15T13:16:18.066 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:16:18.066 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:16:18.066 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:16:18.066 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:16:18.066 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:16:18.066 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 125787 2024-05-15T13:16:18.066 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.066 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:16:18.066 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.066 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:16:18.067 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.678532+0000", 2024-05-15T13:16:18.067 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.067 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.067 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.067 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:16:18.067 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.678532+0000", 2024-05-15T13:16:18.067 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.067 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.067 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.067 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:16:18.067 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.678533+0000", 2024-05-15T13:16:18.067 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.215e-06 2024-05-15T13:16:18.067 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.067 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.068 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:16:18.068 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.678541+0000", 2024-05-15T13:16:18.068 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2250000000000003e-06 2024-05-15T13:16:18.068 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.068 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.068 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:16:18.068 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.678558+0000", 2024-05-15T13:16:18.068 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7462e-05 2024-05-15T13:16:18.068 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.068 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.068 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:16:18.068 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.678565+0000", 2024-05-15T13:16:18.068 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.1400000000000002e-06 2024-05-15T13:16:18.068 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.069 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.069 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.069 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.678614+0000", 2024-05-15T13:16:18.069 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8331000000000002e-05 2024-05-15T13:16:18.069 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.069 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.069 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:16:18.069 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.678619+0000", 2024-05-15T13:16:18.069 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9370000000000001e-06 2024-05-15T13:16:18.069 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.069 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.069 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.069 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454844+0000", 2024-05-15T13:16:18.069 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.77622497099999999 2024-05-15T13:16:18.069 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.070 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.070 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:16:18.070 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454855+0000", 2024-05-15T13:16:18.070 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0655e-05 2024-05-15T13:16:18.070 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.070 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:16:18.070 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.070 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.070 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.070 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:125796 7.d 7.e7ec442d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-05-15T13:16:18.070 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:14:50.716834+0000", 2024-05-15T13:16:18.070 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 87.273387016000001, 2024-05-15T13:16:18.070 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.73814268500000002, 2024-05-15T13:16:18.071 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:16:18.071 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:16:18.071 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:16:18.071 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:16:18.071 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:16:18.071 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 125796 2024-05-15T13:16:18.071 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.071 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:16:18.071 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.071 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:16:18.071 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.716834+0000", 2024-05-15T13:16:18.071 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.071 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.071 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.072 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:16:18.072 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.716834+0000", 2024-05-15T13:16:18.072 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.072 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.072 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.072 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:16:18.072 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.716836+0000", 2024-05-15T13:16:18.072 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.311e-06 2024-05-15T13:16:18.072 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.072 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.072 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:16:18.072 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.716842+0000", 2024-05-15T13:16:18.072 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.0279999999999997e-06 2024-05-15T13:16:18.072 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.073 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.073 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:16:18.073 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.716854+0000", 2024-05-15T13:16:18.073 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2130000000000001e-05 2024-05-15T13:16:18.073 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.073 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.073 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:16:18.073 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.716860+0000", 2024-05-15T13:16:18.073 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4500000000000001e-06 2024-05-15T13:16:18.073 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.073 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.073 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.073 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.716905+0000", 2024-05-15T13:16:18.073 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4802000000000002e-05 2024-05-15T13:16:18.073 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.074 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.074 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:16:18.074 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.716912+0000", 2024-05-15T13:16:18.074 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4880000000000001e-06 2024-05-15T13:16:18.074 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.074 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.074 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.074 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454969+0000", 2024-05-15T13:16:18.074 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.73805667200000002 2024-05-15T13:16:18.074 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.074 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.074 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:16:18.074 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454977+0000", 2024-05-15T13:16:18.075 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8040000000000002e-06 2024-05-15T13:16:18.075 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.075 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:16:18.075 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.075 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.075 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.075 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:125797 7.7 7.a8e9b387 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-05-15T13:16:18.075 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:14:50.718728+0000", 2024-05-15T13:16:18.075 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 87.271493254000006, 2024-05-15T13:16:18.075 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.736156488, 2024-05-15T13:16:18.075 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:16:18.075 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:16:18.075 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:16:18.075 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:16:18.076 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:16:18.076 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 125797 2024-05-15T13:16:18.076 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.076 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:16:18.076 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.076 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:16:18.076 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.718728+0000", 2024-05-15T13:16:18.076 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.076 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.076 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.076 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:16:18.076 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.718728+0000", 2024-05-15T13:16:18.076 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.076 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.076 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.077 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:16:18.077 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.718729+0000", 2024-05-15T13:16:18.077 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.95e-07 2024-05-15T13:16:18.077 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.077 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.077 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:16:18.077 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.718733+0000", 2024-05-15T13:16:18.077 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0749999999999999e-06 2024-05-15T13:16:18.077 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.077 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.077 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:16:18.077 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.718742+0000", 2024-05-15T13:16:18.077 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.9490000000000002e-06 2024-05-15T13:16:18.077 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.078 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.078 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:16:18.078 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.718748+0000", 2024-05-15T13:16:18.078 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5670000000000002e-06 2024-05-15T13:16:18.078 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.078 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.078 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.078 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.718789+0000", 2024-05-15T13:16:18.078 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0921999999999999e-05 2024-05-15T13:16:18.078 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.078 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.078 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:16:18.078 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.718795+0000", 2024-05-15T13:16:18.079 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4679999999999997e-06 2024-05-15T13:16:18.079 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.079 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.079 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.079 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454876+0000", 2024-05-15T13:16:18.079 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.736080717 2024-05-15T13:16:18.079 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.079 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.079 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:16:18.079 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454885+0000", 2024-05-15T13:16:18.079 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7949999999999996e-06 2024-05-15T13:16:18.079 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.079 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:16:18.079 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.079 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.080 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.080 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:125800 7.d 7.b5c022dd (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-05-15T13:16:18.080 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:14:50.762810+0000", 2024-05-15T13:16:18.080 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 87.227411118999996, 2024-05-15T13:16:18.080 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.69218428099999996, 2024-05-15T13:16:18.080 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:16:18.080 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:16:18.080 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:16:18.080 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:16:18.080 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:16:18.080 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 125800 2024-05-15T13:16:18.080 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.080 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:16:18.080 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.081 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:16:18.081 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.762810+0000", 2024-05-15T13:16:18.081 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.081 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.081 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.081 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:16:18.081 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.762810+0000", 2024-05-15T13:16:18.081 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.081 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.081 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.081 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:16:18.081 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.762812+0000", 2024-05-15T13:16:18.081 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4869999999999999e-06 2024-05-15T13:16:18.081 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.082 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.082 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:16:18.082 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.762818+0000", 2024-05-15T13:16:18.082 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6250000000000001e-06 2024-05-15T13:16:18.082 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.082 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.082 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:16:18.082 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.762832+0000", 2024-05-15T13:16:18.082 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3767999999999999e-05 2024-05-15T13:16:18.082 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.082 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.082 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:16:18.082 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.762840+0000", 2024-05-15T13:16:18.083 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8159999999999994e-06 2024-05-15T13:16:18.083 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.083 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.083 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.083 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.762896+0000", 2024-05-15T13:16:18.083 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6159999999999998e-05 2024-05-15T13:16:18.083 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.083 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.083 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:16:18.083 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.762904+0000", 2024-05-15T13:16:18.083 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2900000000000002e-06 2024-05-15T13:16:18.083 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.083 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.083 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.084 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454990+0000", 2024-05-15T13:16:18.084 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.69208525799999998 2024-05-15T13:16:18.084 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.084 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.084 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:16:18.084 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454994+0000", 2024-05-15T13:16:18.084 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8770000000000001e-06 2024-05-15T13:16:18.084 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.084 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:16:18.084 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.084 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.084 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.084 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:127612 7.c 7:38a1016b:::benchmark_data_smithi177_134484_object18547:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e661)", 2024-05-15T13:16:18.085 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:14:56.874194+0000", 2024-05-15T13:16:18.085 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 81.116027762000002, 2024-05-15T13:16:18.085 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.090187319000000002, 2024-05-15T13:16:18.085 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:16:18.085 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:18.085 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:16:18.085 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:16:18.085 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:16:18.085 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 127612 2024-05-15T13:16:18.085 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.085 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:16:18.085 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.085 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:16:18.085 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.874194+0000", 2024-05-15T13:16:18.086 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.086 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.086 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.086 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:16:18.086 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.874194+0000", 2024-05-15T13:16:18.086 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.086 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.086 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.086 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:16:18.086 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.874194+0000", 2024-05-15T13:16:18.086 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7999999999999995e-07 2024-05-15T13:16:18.086 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.086 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.087 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:16:18.087 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.874198+0000", 2024-05-15T13:16:18.087 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.8530000000000002e-06 2024-05-15T13:16:18.087 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.087 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.087 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:16:18.087 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.874206+0000", 2024-05-15T13:16:18.087 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.5889999999999997e-06 2024-05-15T13:16:18.087 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.087 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.087 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:16:18.087 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.874217+0000", 2024-05-15T13:16:18.087 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1656e-05 2024-05-15T13:16:18.087 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.088 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.088 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.088 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.878184+0000", 2024-05-15T13:16:18.088 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0039666900000000001 2024-05-15T13:16:18.088 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.088 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.088 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:16:18.088 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.878338+0000", 2024-05-15T13:16:18.088 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000154139 2024-05-15T13:16:18.088 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.088 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.088 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 3,11", 2024-05-15T13:16:18.088 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.878427+0000", 2024-05-15T13:16:18.088 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8504999999999994e-05 2024-05-15T13:16:18.089 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.089 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.089 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:16:18.089 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.888090+0000", 2024-05-15T13:16:18.089 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0096629550000000008 2024-05-15T13:16:18.089 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.089 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.089 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:18.089 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.888117+0000", 2024-05-15T13:16:18.089 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.7746000000000002e-05 2024-05-15T13:16:18.089 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.089 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.089 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:18.090 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.961943+0000", 2024-05-15T13:16:18.090 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.073825687000000001 2024-05-15T13:16:18.090 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.090 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.090 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:16:18.090 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.961981+0000", 2024-05-15T13:16:18.090 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7663000000000002e-05 2024-05-15T13:16:18.090 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.090 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.090 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:16:18.090 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.964381+0000", 2024-05-15T13:16:18.090 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0024001560000000001 2024-05-15T13:16:18.090 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.091 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:16:18.091 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.091 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.091 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.091 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:127613 7.c 7:367527b6:::benchmark_data_smithi177_134484_object18548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e661)", 2024-05-15T13:16:18.091 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:14:56.874237+0000", 2024-05-15T13:16:18.091 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 81.115984366000006, 2024-05-15T13:16:18.091 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.092414070000000001, 2024-05-15T13:16:18.091 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:16:18.091 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:18.091 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:16:18.091 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:16:18.091 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:16:18.092 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 127613 2024-05-15T13:16:18.092 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.092 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:16:18.092 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.092 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:16:18.092 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.874237+0000", 2024-05-15T13:16:18.092 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.092 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.092 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.092 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:16:18.092 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.874237+0000", 2024-05-15T13:16:18.092 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:16:18.092 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.092 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.093 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:16:18.093 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.874237+0000", 2024-05-15T13:16:18.093 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.75e-07 2024-05-15T13:16:18.093 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.093 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.093 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:16:18.093 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.874240+0000", 2024-05-15T13:16:18.093 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.9610000000000001e-06 2024-05-15T13:16:18.093 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.093 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.093 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:16:18.093 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.874247+0000", 2024-05-15T13:16:18.093 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.5649999999999998e-06 2024-05-15T13:16:18.093 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.094 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.094 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:16:18.094 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.874253+0000", 2024-05-15T13:16:18.094 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9429999999999997e-06 2024-05-15T13:16:18.094 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.094 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.094 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:16:18.094 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.881605+0000", 2024-05-15T13:16:18.094 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.007351982 2024-05-15T13:16:18.094 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.094 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.094 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:16:18.094 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.881710+0000", 2024-05-15T13:16:18.094 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000104817 2024-05-15T13:16:18.095 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.095 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.095 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 3,11", 2024-05-15T13:16:18.095 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.881800+0000", 2024-05-15T13:16:18.095 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0377000000000006e-05 2024-05-15T13:16:18.095 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.095 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.095 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:16:18.095 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.890651+0000", 2024-05-15T13:16:18.095 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0088505059999999993 2024-05-15T13:16:18.095 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.095 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.095 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:18.095 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.892318+0000", 2024-05-15T13:16:18.096 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001667899 2024-05-15T13:16:18.096 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.096 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.096 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:18.096 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.964439+0000", 2024-05-15T13:16:18.096 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.072120670999999997 2024-05-15T13:16:18.096 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.096 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.096 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:16:18.096 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.964482+0000", 2024-05-15T13:16:18.096 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2429000000000003e-05 2024-05-15T13:16:18.096 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:16:18.096 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:16:18.096 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:16:18.097 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:56.966651+0000", 2024-05-15T13:16:18.097 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0021694449999999999 2024-05-15T13:16:18.097 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.097 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:16:18.097 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.097 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:16:18.097 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:16:18.097 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:16:18.098 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:17 smithi027 ceph-mon[39795]: pgmap v1790: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 386 op/s; 10700/176556 objects degraded (6.060%); 26547/176556 objects misplaced (15.036%); 13 MiB/s, 220 objects/s recovering 2024-05-15T13:16:18.098 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:17 smithi120 ceph-mon[43961]: pgmap v1790: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 386 op/s; 10700/176556 objects degraded (6.060%); 26547/176556 objects misplaced (15.036%); 13 MiB/s, 220 objects/s recovering 2024-05-15T13:16:18.487 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:16:18.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:18 smithi027 ceph-mon[39795]: osdmap e693: 12 total, 12 up, 10 in 2024-05-15T13:16:18.836 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-05-15T13:16:18.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:18 smithi177 ceph-mon[44303]: osdmap e693: 12 total, 12 up, 10 in 2024-05-15T13:16:19.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:18 smithi120 ceph-mon[43961]: osdmap e693: 12 total, 12 up, 10 in 2024-05-15T13:16:19.155 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:16:19.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:19 smithi177 ceph-mon[44303]: pgmap v1792: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 405 op/s; 10464/172314 objects degraded (6.073%); 25891/172314 objects misplaced (15.025%); 16 MiB/s, 267 objects/s recovering 2024-05-15T13:16:19.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:19 smithi177 ceph-mon[44303]: osdmap e694: 12 total, 12 up, 10 in 2024-05-15T13:16:19.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:19 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 10464/172314 objects degraded (6.073%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:20.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:19 smithi027 ceph-mon[39795]: pgmap v1792: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 405 op/s; 10464/172314 objects degraded (6.073%); 25891/172314 objects misplaced (15.025%); 16 MiB/s, 267 objects/s recovering 2024-05-15T13:16:20.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:19 smithi027 ceph-mon[39795]: osdmap e694: 12 total, 12 up, 10 in 2024-05-15T13:16:20.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:19 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 10464/172314 objects degraded (6.073%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:20.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:19 smithi120 ceph-mon[43961]: pgmap v1792: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 405 op/s; 10464/172314 objects degraded (6.073%); 25891/172314 objects misplaced (15.025%); 16 MiB/s, 267 objects/s recovering 2024-05-15T13:16:20.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:19 smithi120 ceph-mon[43961]: osdmap e694: 12 total, 12 up, 10 in 2024-05-15T13:16:20.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:19 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 10464/172314 objects degraded (6.073%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:20.171 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:16:20.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:20.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:16:20.284 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:16:20.352 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:21.571 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:16:21.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:21.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:16:21.734 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:16:21.767 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:22.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:21 smithi027 ceph-mon[39795]: pgmap v1794: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 342 op/s; 10477/170787 objects degraded (6.135%); 25528/170787 objects misplaced (14.947%); 13 MiB/s, 200 objects/s recovering 2024-05-15T13:16:22.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:21 smithi120 ceph-mon[43961]: pgmap v1794: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 342 op/s; 10477/170787 objects degraded (6.135%); 25528/170787 objects misplaced (14.947%); 13 MiB/s, 200 objects/s recovering 2024-05-15T13:16:22.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:21 smithi177 ceph-mon[44303]: pgmap v1794: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 342 op/s; 10477/170787 objects degraded (6.135%); 25528/170787 objects misplaced (14.947%); 13 MiB/s, 200 objects/s recovering 2024-05-15T13:16:22.343 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:16:23.547 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:16:23.748 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:23 smithi027 ceph-mon[39795]: pgmap v1795: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 3.4 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 413 op/s; 9996/167019 objects degraded (5.985%); 25041/167019 objects misplaced (14.993%); 13 MiB/s, 244 objects/s recovering 2024-05-15T13:16:23.824 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:16:23.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:23.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:16:23.929 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:16:23.956 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:24.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:23 smithi120 ceph-mon[43961]: pgmap v1795: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 3.4 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 413 op/s; 9996/167019 objects degraded (5.985%); 25041/167019 objects misplaced (14.993%); 13 MiB/s, 244 objects/s recovering 2024-05-15T13:16:24.109 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:16:24.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:24.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:16:24.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:23 smithi177 ceph-mon[44303]: pgmap v1795: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 3.4 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 413 op/s; 9996/167019 objects degraded (5.985%); 25041/167019 objects misplaced (14.993%); 13 MiB/s, 244 objects/s recovering 2024-05-15T13:16:24.213 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:16:24.363 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:24.452 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:16:24.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:24.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:16:24.472 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:16:24.558 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:16:24.558 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T13:16:24.558 INFO:teuthology.orchestra.run.smithi120.stdout: "num_ops": 0 2024-05-15T13:16:24.558 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:16:24.561 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:16:24.603 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:24.629 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:16:24.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:24.635 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:16:24.736 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:16:24.770 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:24.798 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:16:24.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:24.805 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:16:24.906 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:16:24.934 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:24 smithi120 ceph-mon[43961]: 7.0 scrub starts 2024-05-15T13:16:24.934 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:24 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 9996/167019 objects degraded (5.985%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:24.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:24 smithi177 ceph-mon[44303]: 7.0 scrub starts 2024-05-15T13:16:24.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:24 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 9996/167019 objects degraded (5.985%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:24.938 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:25.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:24 smithi027 ceph-mon[39795]: 7.0 scrub starts 2024-05-15T13:16:25.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:24 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 9996/167019 objects degraded (5.985%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:25.037 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:16:25.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:25.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:16:25.147 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:16:25.267 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:25.479 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-05-15T13:16:25.768 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:16:25.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:25.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:16:25.802 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:16:25.876 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:16:25.982 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:26.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:25 smithi027 ceph-mon[39795]: 7.0 scrub ok 2024-05-15T13:16:26.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:25 smithi027 ceph-mon[39795]: pgmap v1796: 37 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 510 op/s; 9996/164286 objects degraded (6.085%); 24444/164286 objects misplaced (14.879%); 18 MiB/s, 318 objects/s recovering 2024-05-15T13:16:26.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:25 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3461047703' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T13:16:26.033 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:16:26.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:25 smithi120 ceph-mon[43961]: 7.0 scrub ok 2024-05-15T13:16:26.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:25 smithi120 ceph-mon[43961]: pgmap v1796: 37 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 510 op/s; 9996/164286 objects degraded (6.085%); 24444/164286 objects misplaced (14.879%); 18 MiB/s, 318 objects/s recovering 2024-05-15T13:16:26.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:25 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3461047703' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T13:16:26.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:26.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:16:26.141 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:16:26.171 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:26.183 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.11 to 1 (10000) 2024-05-15T13:16:26.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:25 smithi177 ceph-mon[44303]: 7.0 scrub ok 2024-05-15T13:16:26.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:25 smithi177 ceph-mon[44303]: pgmap v1796: 37 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 510 op/s; 9996/164286 objects degraded (6.085%); 24444/164286 objects misplaced (14.879%); 18 MiB/s, 318 objects/s recovering 2024-05-15T13:16:26.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:25 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3461047703' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T13:16:26.194 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:16:26.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:26.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:16:26.298 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:16:26.432 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:16:26.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:26.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:16:26.700 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:16:26.844 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:26.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:26 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:16:26.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:26 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:16:26.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:26 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:16:26.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:26 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:16:26.860 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:26 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3461047703' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T13:16:26.860 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:26 smithi027 ceph-mon[39795]: osdmap e695: 12 total, 12 up, 10 in 2024-05-15T13:16:26.969 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:16:26.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:26.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:16:27.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:26 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:16:27.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:26 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:16:27.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:26 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:16:27.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:26 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:16:27.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:26 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3461047703' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T13:16:27.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:26 smithi120 ceph-mon[43961]: osdmap e695: 12 total, 12 up, 10 in 2024-05-15T13:16:27.080 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:16:27.105 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:16:27.122 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:26 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:16:27.122 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:26 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:16:27.122 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:26 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:16:27.122 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:26 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:16:27.122 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:26 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3461047703' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T13:16:27.122 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:26 smithi177 ceph-mon[44303]: osdmap e695: 12 total, 12 up, 10 in 2024-05-15T13:16:27.124 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:27.160 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:16:27.180 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:16:27.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:27.184 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:16:27.219 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:16:27.285 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:16:27.288 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:16:27.354 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:16:27.383 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:27.397 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:16:27.455 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:16:27.459 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-05-15T13:16:27.459 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:16:27.513 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:16:27.561 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:16:27.616 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:16:27.678 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:16:27.732 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:16:27.757 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:16:27.757 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:16:27.757 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:16:27.757 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:16:27.757 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:16:27.757 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:16:27.758 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:16:27.758 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:16:27.758 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:16:27.758 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:16:27.758 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:16:27.758 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:16:27.768 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:16:27.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:27.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:16:27.881 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:16:27.914 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:27.946 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:16:27.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:27.950 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:16:28.015 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:16:28.015 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:16:28.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:27 smithi120 ceph-mon[43961]: pgmap v1798: 37 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 350 op/s; 9996/163887 objects degraded (6.099%); 24444/163887 objects misplaced (14.915%); 12 MiB/s, 223 objects/s recovering 2024-05-15T13:16:28.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:27 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/216134612' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:16:28.051 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:16:28.087 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:27 smithi177 ceph-mon[44303]: pgmap v1798: 37 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 350 op/s; 9996/163887 objects degraded (6.099%); 24444/163887 objects misplaced (14.915%); 12 MiB/s, 223 objects/s recovering 2024-05-15T13:16:28.087 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:27 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/216134612' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:16:28.091 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:28.120 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:16:28.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:28.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:16:28.226 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:16:28.257 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:28.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:27 smithi027 ceph-mon[39795]: pgmap v1798: 37 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 350 op/s; 9996/163887 objects degraded (6.099%); 24444/163887 objects misplaced (14.915%); 12 MiB/s, 223 objects/s recovering 2024-05-15T13:16:28.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:27 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/216134612' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:16:28.285 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:16:28.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:28.291 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:16:28.393 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:16:28.424 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:28.829 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-05-15T13:16:28.875 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:16:28.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:28.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:16:28.979 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:16:29.035 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:28 smithi027 ceph-mon[39795]: pgmap v1799: 37 pgs: 1 active+recovery_wait+remapped, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 400 op/s; 9722/159420 objects degraded (6.098%); 23762/159420 objects misplaced (14.905%); 14 MiB/s, 252 objects/s recovering 2024-05-15T13:16:29.042 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:29.168 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:16:29.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:29.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:16:29.177 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:16:29.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:28 smithi177 ceph-mon[44303]: pgmap v1799: 37 pgs: 1 active+recovery_wait+remapped, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 400 op/s; 9722/159420 objects degraded (6.098%); 23762/159420 objects misplaced (14.905%); 14 MiB/s, 252 objects/s recovering 2024-05-15T13:16:29.276 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:16:29.286 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:28 smithi120 ceph-mon[43961]: pgmap v1799: 37 pgs: 1 active+recovery_wait+remapped, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 3.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 400 op/s; 9722/159420 objects degraded (6.098%); 23762/159420 objects misplaced (14.905%); 14 MiB/s, 252 objects/s recovering 2024-05-15T13:16:29.384 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:29.565 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:16:29.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:29.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:16:29.669 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:16:29.699 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:29.814 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:16:29.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:29 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 9722/159420 objects degraded (6.098%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:29.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:29 smithi027 ceph-mon[39795]: osdmap e696: 12 total, 12 up, 10 in 2024-05-15T13:16:30.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:29 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 9722/159420 objects degraded (6.098%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:30.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:29 smithi120 ceph-mon[43961]: osdmap e696: 12 total, 12 up, 10 in 2024-05-15T13:16:30.049 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:16:30.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:30.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:16:30.153 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:16:30.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:29 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 9722/159420 objects degraded (6.098%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:30.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:29 smithi177 ceph-mon[44303]: osdmap e696: 12 total, 12 up, 10 in 2024-05-15T13:16:30.245 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:30.327 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:16:30.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:30.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:16:30.437 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:16:30.470 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:30.504 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:16:30.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:30.512 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:16:30.613 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:16:30.727 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:16:30.879 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:30 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2292242477' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:16:30.879 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:30 smithi027 ceph-mon[39795]: pgmap v1801: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 3.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 383 op/s; 9750/157806 objects degraded (6.178%); 23451/157806 objects misplaced (14.861%); 13 MiB/s, 212 objects/s recovering 2024-05-15T13:16:30.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:30.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:16:30.928 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:16:30.966 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:30 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2292242477' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:16:30.966 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:30 smithi120 ceph-mon[43961]: pgmap v1801: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 3.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 383 op/s; 9750/157806 objects degraded (6.178%); 23451/157806 objects misplaced (14.861%); 13 MiB/s, 212 objects/s recovering 2024-05-15T13:16:30.989 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:16:31.024 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:31.092 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:16:31.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:30 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2292242477' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:16:31.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:30 smithi177 ceph-mon[44303]: pgmap v1801: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 3.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 383 op/s; 9750/157806 objects degraded (6.178%); 23451/157806 objects misplaced (14.861%); 13 MiB/s, 212 objects/s recovering 2024-05-15T13:16:31.270 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:16:31.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:31.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:16:31.374 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:16:31.381 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:16:31.381 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:16:31.407 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:31.433 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:16:31.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:31.439 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:16:31.540 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:16:31.570 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:31.921 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:16:31.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:31.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:16:32.025 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:16:32.054 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:32.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:31 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:16:32.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:31 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2292242477' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:16:32.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:31 smithi177 ceph-mon[44303]: osdmap e697: 12 total, 12 up, 10 in 2024-05-15T13:16:32.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:31 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:16:32.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:31 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:16:32.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:31 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:16:32.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:31 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2292242477' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:16:32.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:31 smithi027 ceph-mon[39795]: osdmap e697: 12 total, 12 up, 10 in 2024-05-15T13:16:32.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:31 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:16:32.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:31 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:16:32.286 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:31 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:16:32.286 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:31 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2292242477' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:16:32.286 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:31 smithi120 ceph-mon[43961]: osdmap e697: 12 total, 12 up, 10 in 2024-05-15T13:16:32.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:31 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:16:32.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:31 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:16:33.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:32 smithi177 ceph-mon[44303]: osdmap e698: 12 total, 12 up, 10 in 2024-05-15T13:16:33.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:32 smithi177 ceph-mon[44303]: pgmap v1804: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 3.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 543 op/s; 9563/154086 objects degraded (6.206%); 23015/154086 objects misplaced (14.936%); 17 MiB/s, 267 objects/s recovering 2024-05-15T13:16:33.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:32 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:16:33.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:32 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:16:33.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:32 smithi027 ceph-mon[39795]: osdmap e698: 12 total, 12 up, 10 in 2024-05-15T13:16:33.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:32 smithi027 ceph-mon[39795]: pgmap v1804: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 3.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 543 op/s; 9563/154086 objects degraded (6.206%); 23015/154086 objects misplaced (14.936%); 17 MiB/s, 267 objects/s recovering 2024-05-15T13:16:33.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:32 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:16:33.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:32 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:16:33.286 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:32 smithi120 ceph-mon[43961]: osdmap e698: 12 total, 12 up, 10 in 2024-05-15T13:16:33.286 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:32 smithi120 ceph-mon[43961]: pgmap v1804: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 3.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 543 op/s; 9563/154086 objects degraded (6.206%); 23015/154086 objects misplaced (14.936%); 17 MiB/s, 267 objects/s recovering 2024-05-15T13:16:33.286 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:32 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:16:33.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:32 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:16:34.203 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:33 smithi120 ceph-mon[43961]: osdmap e699: 12 total, 12 up, 10 in 2024-05-15T13:16:34.203 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:33 smithi120 ceph-mon[43961]: 1.10 scrub starts 2024-05-15T13:16:34.203 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:33 smithi120 ceph-mon[43961]: 1.10 scrub ok 2024-05-15T13:16:34.231 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:16:34.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:34.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:16:34.238 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-05-15T13:16:34.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:33 smithi027 ceph-mon[39795]: osdmap e699: 12 total, 12 up, 10 in 2024-05-15T13:16:34.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:33 smithi027 ceph-mon[39795]: 1.10 scrub starts 2024-05-15T13:16:34.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:33 smithi027 ceph-mon[39795]: 1.10 scrub ok 2024-05-15T13:16:34.282 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:33 smithi177 ceph-mon[44303]: osdmap e699: 12 total, 12 up, 10 in 2024-05-15T13:16:34.282 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:33 smithi177 ceph-mon[44303]: 1.10 scrub starts 2024-05-15T13:16:34.282 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:33 smithi177 ceph-mon[44303]: 1.10 scrub ok 2024-05-15T13:16:34.339 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:16:34.385 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:34.415 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:16:34.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:34.437 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:16:34.538 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:16:34.583 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:34.625 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:16:34.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:34.628 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:16:34.729 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:16:34.762 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:34.789 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:16:34.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:34.791 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:16:34.892 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:16:34.921 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:34.946 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:16:34.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:34.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:16:35.049 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:16:35.115 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:35.180 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:16:35.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:35.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:16:35.279 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:34 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 9563/154086 objects degraded (6.206%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:35.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:34 smithi027 ceph-mon[39795]: osdmap e700: 12 total, 12 up, 10 in 2024-05-15T13:16:35.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:34 smithi027 ceph-mon[39795]: pgmap v1807: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 3.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 565 op/s; 9600/151014 objects degraded (6.357%); 22362/151014 objects misplaced (14.808%); 17 MiB/s, 280 objects/s recovering 2024-05-15T13:16:35.286 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:16:35.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:34 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 9563/154086 objects degraded (6.206%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:35.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:34 smithi120 ceph-mon[43961]: osdmap e700: 12 total, 12 up, 10 in 2024-05-15T13:16:35.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:34 smithi120 ceph-mon[43961]: pgmap v1807: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 3.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 565 op/s; 9600/151014 objects degraded (6.357%); 22362/151014 objects misplaced (14.808%); 17 MiB/s, 280 objects/s recovering 2024-05-15T13:16:35.329 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:34 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 9563/154086 objects degraded (6.206%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:35.329 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:34 smithi177 ceph-mon[44303]: osdmap e700: 12 total, 12 up, 10 in 2024-05-15T13:16:35.329 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:34 smithi177 ceph-mon[44303]: pgmap v1807: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 3.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 565 op/s; 9600/151014 objects degraded (6.357%); 22362/151014 objects misplaced (14.808%); 17 MiB/s, 280 objects/s recovering 2024-05-15T13:16:35.332 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:35.370 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:16:35.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:35.373 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:16:35.474 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:16:35.581 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:36.219 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:16:36.685 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:36 smithi177 ceph-mon[44303]: 1.8 scrub starts 2024-05-15T13:16:36.685 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:36 smithi177 ceph-mon[44303]: 1.8 scrub ok 2024-05-15T13:16:36.785 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:36 smithi027 ceph-mon[39795]: 1.8 scrub starts 2024-05-15T13:16:36.833 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:36 smithi027 ceph-mon[39795]: 1.8 scrub ok 2024-05-15T13:16:36.833 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:36 smithi120 ceph-mon[43961]: 1.8 scrub starts 2024-05-15T13:16:36.833 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:36 smithi120 ceph-mon[43961]: 1.8 scrub ok 2024-05-15T13:16:37.671 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:37 smithi177 ceph-mon[44303]: pgmap v1808: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 3.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 430 op/s; 9600/150600 objects degraded (6.375%); 22362/150600 objects misplaced (14.849%); 13 MiB/s, 200 objects/s recovering 2024-05-15T13:16:37.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:37 smithi027 ceph-mon[39795]: pgmap v1808: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 3.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 430 op/s; 9600/150600 objects degraded (6.375%); 22362/150600 objects misplaced (14.849%); 13 MiB/s, 200 objects/s recovering 2024-05-15T13:16:37.786 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:37 smithi120 ceph-mon[43961]: pgmap v1808: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 3.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 430 op/s; 9600/150600 objects degraded (6.375%); 22362/150600 objects misplaced (14.849%); 13 MiB/s, 200 objects/s recovering 2024-05-15T13:16:38.812 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:16:38.812 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:16:39.662 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-05-15T13:16:39.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:39 smithi177 ceph-mon[44303]: pgmap v1809: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 3.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 465 op/s; 9387/145755 objects degraded (6.440%); 21585/145755 objects misplaced (14.809%); 14 MiB/s, 225 objects/s recovering 2024-05-15T13:16:39.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:39 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 9387/145755 objects degraded (6.440%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:40.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:39 smithi027 ceph-mon[39795]: pgmap v1809: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 3.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 465 op/s; 9387/145755 objects degraded (6.440%); 21585/145755 objects misplaced (14.809%); 14 MiB/s, 225 objects/s recovering 2024-05-15T13:16:40.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:39 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 9387/145755 objects degraded (6.440%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:40.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:39 smithi120 ceph-mon[43961]: pgmap v1809: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 3.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 465 op/s; 9387/145755 objects degraded (6.440%); 21585/145755 objects misplaced (14.809%); 14 MiB/s, 225 objects/s recovering 2024-05-15T13:16:40.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:39 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 9387/145755 objects degraded (6.440%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:40.058 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:16:40.106 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:16:40.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:40.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:16:40.222 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:16:40.263 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:40.340 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:16:40.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:40.347 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:16:40.448 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:16:40.476 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:40.497 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:16:40.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:40.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:16:40.601 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:16:40.651 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:41.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:41 smithi177 ceph-mon[44303]: pgmap v1810: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 420 op/s; 9385/143997 objects degraded (6.517%); 21236/143997 objects misplaced (14.748%); 14 MiB/s, 228 objects/s recovering 2024-05-15T13:16:42.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:41 smithi027 ceph-mon[39795]: pgmap v1810: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 420 op/s; 9385/143997 objects degraded (6.517%); 21236/143997 objects misplaced (14.748%); 14 MiB/s, 228 objects/s recovering 2024-05-15T13:16:42.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:41 smithi120 ceph-mon[43961]: pgmap v1810: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 420 op/s; 9385/143997 objects degraded (6.517%); 21236/143997 objects misplaced (14.748%); 14 MiB/s, 228 objects/s recovering 2024-05-15T13:16:42.132 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:16:42.695 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:16:42.913 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:16:42.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:42.916 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:16:43.017 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:16:43.044 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:43.065 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:16:43.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:43.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:16:43.169 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:16:43.271 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:43.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:43 smithi177 ceph-mon[44303]: pgmap v1811: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 448 op/s; 9143/140346 objects degraded (6.515%); 20690/140346 objects misplaced (14.742%); 16 MiB/s, 259 objects/s recovering 2024-05-15T13:16:44.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:43 smithi027 ceph-mon[39795]: pgmap v1811: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 448 op/s; 9143/140346 objects degraded (6.515%); 20690/140346 objects misplaced (14.742%); 16 MiB/s, 259 objects/s recovering 2024-05-15T13:16:44.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:43 smithi120 ceph-mon[43961]: pgmap v1811: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 448 op/s; 9143/140346 objects degraded (6.515%); 20690/140346 objects misplaced (14.742%); 16 MiB/s, 259 objects/s recovering 2024-05-15T13:16:44.484 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:16:44.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:44.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:16:44.598 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:16:44.626 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:44.645 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:16:44.646 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:16:44.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:44.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:16:44.683 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:16:44.716 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:44 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 9143/140346 objects degraded (6.515%), 5 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:44.716 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:44 smithi027 ceph-mon[39795]: 1.12 scrub starts 2024-05-15T13:16:44.716 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:44 smithi027 ceph-mon[39795]: 1.12 scrub ok 2024-05-15T13:16:44.723 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:16:44.723 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:16:44.750 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:16:44.787 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:44.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:44 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 9143/140346 objects degraded (6.515%), 5 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:44.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:44 smithi177 ceph-mon[44303]: 1.12 scrub starts 2024-05-15T13:16:44.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:44 smithi177 ceph-mon[44303]: 1.12 scrub ok 2024-05-15T13:16:45.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:44 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 9143/140346 objects degraded (6.515%), 5 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:45.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:44 smithi120 ceph-mon[43961]: 1.12 scrub starts 2024-05-15T13:16:45.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:44 smithi120 ceph-mon[43961]: 1.12 scrub ok 2024-05-15T13:16:45.346 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:16:45.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:45.349 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:16:45.450 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:16:45.492 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:45.628 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:45 smithi177 ceph-mon[44303]: pgmap v1812: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 450 op/s; 9181/137397 objects degraded (6.682%); 19965/137397 objects misplaced (14.531%); 16 MiB/s, 263 objects/s recovering 2024-05-15T13:16:45.628 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:45 smithi177 ceph-mon[44303]: from='client.26177 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:16:45.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:45 smithi027 ceph-mon[39795]: pgmap v1812: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 450 op/s; 9181/137397 objects degraded (6.682%); 19965/137397 objects misplaced (14.531%); 16 MiB/s, 263 objects/s recovering 2024-05-15T13:16:45.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:45 smithi027 ceph-mon[39795]: from='client.26177 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:16:45.959 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:16:45.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:45.967 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:16:45.980 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-05-15T13:16:46.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:45 smithi120 ceph-mon[43961]: pgmap v1812: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 450 op/s; 9181/137397 objects degraded (6.682%); 19965/137397 objects misplaced (14.531%); 16 MiB/s, 263 objects/s recovering 2024-05-15T13:16:46.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:45 smithi120 ceph-mon[43961]: from='client.26177 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:16:46.068 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:16:46.099 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:46.130 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:16:46.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:46.135 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:16:46.236 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:16:46.331 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:46.409 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:16:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:16:46.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:46.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:16:46.567 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:16:46.595 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:46.614 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:16:46.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:46.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:16:46.641 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:16:46.680 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":1812,"stamp":"2024-05-15T13:16:44.492661+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3001877048,"num_objects":45799,"num_object_clones":0,"num_object_copies":137397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9181,"num_objects_misplaced":19965,"num_objects_unfound":0,"num_objects_dirty":45799,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172329,"num_write_kb":6979969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31817,"num_bytes_recovered":2058158080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163833,"ondisk_log_size":163833,"up":111,"acting":104,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":17868876,"kb_used_data":11201084,"kb_used_omap":15,"kb_used_meta":6667312,"kb_avail":919582644,"statfs":{"total":959950356480,"available":941652627456,"internally_reserved":0,"allocated":11469910016,"data_stored":11465129624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15956,"internal_metadata":6827327916},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-307167232,"num_objects":-4687,"num_object_clones":0,"num_object_copies":-14061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-419,"num_objects_misplaced":-2583,"num_objects_unfound":0,"num_objects_dirty":-4687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4687,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2738,"num_bytes_recovered":178585600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.407878"},"pg_stats":[{"pgid":"1.14","version":"94'139","reported_seq":723,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:34.096425+0000","last_change":"2024-05-15T13:13:17.477074+0000","last_active":"2024-05-15T13:16:34.096425+0000","last_peered":"2024-05-15T13:16:34.096425+0000","last_clean":"2024-05-15T13:16:34.096425+0000","last_became_active":"2024-05-15T13:13:16.473818+0000","last_became_peered":"2024-05-15T13:13:16.473818+0000","last_unstale":"2024-05-15T13:16:34.096425+0000","last_undegraded":"2024-05-15T13:16:34.096425+0000","last_fullsized":"2024-05-15T13:16:34.096425+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.476961+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.476961+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1],"acting":[2,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":1298,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:34.084279+0000","last_change":"2024-05-15T13:13:19.224641+0000","last_active":"2024-05-15T13:16:34.084279+0000","last_peered":"2024-05-15T13:16:34.084279+0000","last_clean":"2024-05-15T13:16:34.084279+0000","last_became_active":"2024-05-15T13:13:11.361366+0000","last_became_peered":"2024-05-15T13:13:11.361366+0000","last_unstale":"2024-05-15T13:16:34.084279+0000","last_undegraded":"2024-05-15T13:16:34.084279+0000","last_fullsized":"2024-05-15T13:16:34.084279+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:19.224509+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:19.224509+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"700'10687","reported_seq":16864,"reported_epoch":700,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-05-15T13:16:41.911490+0000","last_change":"2024-05-15T13:14:21.034561+0000","last_active":"2024-05-15T13:16:41.911490+0000","last_peered":"2024-05-15T13:16:41.911490+0000","last_clean":"2024-05-15T13:14:07.430624+0000","last_became_active":"2024-05-15T13:14:09.931862+0000","last_became_peered":"2024-05-15T13:14:09.931862+0000","last_unstale":"2024-05-15T13:16:41.911490+0000","last_undegraded":"2024-05-15T13:14:21.029697+0000","last_fullsized":"2024-05-15T13:14:09.472302+0000","mapping_epoch":647,"log_start":"608'600","ondisk_log_start":"608'600","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186318848,"num_objects":2843,"num_object_clones":0,"num_object_copies":8529,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":951,"num_objects_misplaced":2764,"num_objects_unfound":0,"num_objects_dirty":2843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10687,"num_write_kb":432960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2980,"num_bytes_recovered":192610304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,10],"acting":[0,10],"avail_no_missing":["0","5","10"],"object_location_counts":[{"shards":"0,5,10","objects":2764},{"shards":"0,10","objects":-872}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"94'139","reported_seq":1298,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:34.081784+0000","last_change":"2024-05-15T13:13:17.949012+0000","last_active":"2024-05-15T13:16:34.081784+0000","last_peered":"2024-05-15T13:16:34.081784+0000","last_clean":"2024-05-15T13:16:34.081784+0000","last_became_active":"2024-05-15T13:13:11.371643+0000","last_became_peered":"2024-05-15T13:13:11.371643+0000","last_unstale":"2024-05-15T13:16:34.081784+0000","last_undegraded":"2024-05-15T13:16:34.081784+0000","last_fullsized":"2024-05-15T13:16:34.081784+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.948900+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.948900+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.2","version":"700'14319","reported_seq":14711,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:42.618444+0000","last_change":"2024-05-15T13:14:18.564785+0000","last_active":"2024-05-15T13:16:42.618444+0000","last_peered":"2024-05-15T13:16:42.618444+0000","last_clean":"2024-05-15T13:16:42.618444+0000","last_became_active":"2024-05-15T13:12:29.918894+0000","last_became_peered":"2024-05-15T13:12:29.918894+0000","last_unstale":"2024-05-15T13:16:42.618444+0000","last_undegraded":"2024-05-15T13:16:42.618444+0000","last_fullsized":"2024-05-15T13:16:42.618444+0000","mapping_epoch":599,"log_start":"621'4301","ondisk_log_start":"621'4301","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":4374,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184025088,"num_objects":2808,"num_object_clones":0,"num_object_copies":8424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10862,"num_write_kb":437440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2],"acting":[3,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"94'139","reported_seq":1052,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:34.097676+0000","last_change":"2024-05-15T13:14:54.577804+0000","last_active":"2024-05-15T13:16:34.097676+0000","last_peered":"2024-05-15T13:16:34.097676+0000","last_clean":"2024-05-15T13:16:34.097676+0000","last_became_active":"2024-05-15T13:14:54.577458+0000","last_became_peered":"2024-05-15T13:14:54.577458+0000","last_unstale":"2024-05-15T13:16:34.097676+0000","last_undegraded":"2024-05-15T13:16:34.097676+0000","last_fullsized":"2024-05-15T13:16:34.097676+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:18.049722+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:18.049722+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.3","version":"700'10601","reported_seq":10873,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:42.641853+0000","last_change":"2024-05-15T13:13:35.644249+0000","last_active":"2024-05-15T13:16:42.641853+0000","last_peered":"2024-05-15T13:16:42.641853+0000","last_clean":"2024-05-15T13:16:42.641853+0000","last_became_active":"2024-05-15T13:12:29.921897+0000","last_became_peered":"2024-05-15T13:12:29.921897+0000","last_unstale":"2024-05-15T13:16:42.641853+0000","last_undegraded":"2024-05-15T13:16:42.641853+0000","last_fullsized":"2024-05-15T13:16:42.641853+0000","mapping_epoch":599,"log_start":"608'600","ondisk_log_start":"608'600","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'5271","last_scrub_stamp":"2024-05-15T13:13:35.643996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:35.643996+0000","objects_scrubbed":5237,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187236352,"num_objects":2857,"num_object_clones":0,"num_object_copies":8571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10601,"num_write_kb":430656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"94'139","reported_seq":1291,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:34.080707+0000","last_change":"2024-05-15T13:13:17.875187+0000","last_active":"2024-05-15T13:16:34.080707+0000","last_peered":"2024-05-15T13:16:34.080707+0000","last_clean":"2024-05-15T13:16:34.080707+0000","last_became_active":"2024-05-15T13:13:11.373251+0000","last_became_peered":"2024-05-15T13:13:11.373251+0000","last_unstale":"2024-05-15T13:16:34.080707+0000","last_undegraded":"2024-05-15T13:16:34.080707+0000","last_fullsized":"2024-05-15T13:16:34.080707+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.875054+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.875054+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.0","version":"700'10799","reported_seq":11054,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:43.474272+0000","last_change":"2024-05-15T13:16:24.441683+0000","last_active":"2024-05-15T13:16:43.474272+0000","last_peered":"2024-05-15T13:16:43.474272+0000","last_clean":"2024-05-15T13:16:43.474272+0000","last_became_active":"2024-05-15T13:12:29.924027+0000","last_became_peered":"2024-05-15T13:12:29.924027+0000","last_unstale":"2024-05-15T13:16:43.474272+0000","last_undegraded":"2024-05-15T13:16:43.474272+0000","last_fullsized":"2024-05-15T13:16:43.474272+0000","mapping_epoch":599,"log_start":"608'700","ondisk_log_start":"608'700","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"694'10259","last_scrub_stamp":"2024-05-15T13:16:24.441404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:16:24.441404+0000","objects_scrubbed":3481,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:17:37.997733+0000","scrub_duration":1000,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191692800,"num_objects":2925,"num_object_clones":0,"num_object_copies":8775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10799,"num_write_kb":439168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"94'139","reported_seq":1066,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:34.083739+0000","last_change":"2024-05-15T13:14:11.522798+0000","last_active":"2024-05-15T13:16:34.083739+0000","last_peered":"2024-05-15T13:16:34.083739+0000","last_clean":"2024-05-15T13:16:34.083739+0000","last_became_active":"2024-05-15T13:14:11.522490+0000","last_became_peered":"2024-05-15T13:14:11.522490+0000","last_unstale":"2024-05-15T13:16:34.083739+0000","last_undegraded":"2024-05-15T13:16:34.083739+0000","last_fullsized":"2024-05-15T13:16:34.083739+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.847198+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.847198+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"700'10683","reported_seq":22084,"reported_epoch":700,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-15T13:16:41.849759+0000","last_change":"2024-05-15T13:14:52.829919+0000","last_active":"2024-05-15T13:16:41.849759+0000","last_peered":"2024-05-15T13:16:41.849759+0000","last_clean":"2024-05-15T13:13:46.353513+0000","last_became_active":"2024-05-15T13:14:52.812683+0000","last_became_peered":"2024-05-15T13:14:52.812683+0000","last_unstale":"2024-05-15T13:16:41.849759+0000","last_undegraded":"2024-05-15T13:14:52.601890+0000","last_fullsized":"2024-05-15T13:14:52.453432+0000","mapping_epoch":658,"log_start":"608'600","ondisk_log_start":"608'600","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"633'5877","last_scrub_stamp":"2024-05-15T13:13:44.171370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:44.171370+0000","objects_scrubbed":5827,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1393,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191430656,"num_objects":2921,"num_object_clones":0,"num_object_copies":8763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10683,"num_write_kb":435328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3883,"num_bytes_recovered":254345216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3],"avail_no_missing":["10","5","11"],"object_location_counts":[{"shards":"1,3,10","objects":86},{"shards":"1,10,11","objects":704},{"shards":"5,10,11","objects":2131}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"94'139","reported_seq":1069,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:34.080572+0000","last_change":"2024-05-15T13:15:43.615336+0000","last_active":"2024-05-15T13:16:34.080572+0000","last_peered":"2024-05-15T13:16:34.080572+0000","last_clean":"2024-05-15T13:16:34.080572+0000","last_became_active":"2024-05-15T13:15:43.614970+0000","last_became_peered":"2024-05-15T13:15:43.614970+0000","last_unstale":"2024-05-15T13:16:34.080572+0000","last_undegraded":"2024-05-15T13:16:34.080572+0000","last_fullsized":"2024-05-15T13:16:34.080572+0000","mapping_epoch":678,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:20.062928+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:20.062928+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,0],"acting":[9,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.7","version":"700'10900","reported_seq":11221,"reported_epoch":700,"state":"active+recovery_wait+remapped","last_fresh":"2024-05-15T13:16:42.089990+0000","last_change":"2024-05-15T13:14:52.781625+0000","last_active":"2024-05-15T13:16:42.089990+0000","last_peered":"2024-05-15T13:16:42.089990+0000","last_clean":"2024-05-15T13:14:50.323746+0000","last_became_active":"2024-05-15T13:14:52.762068+0000","last_became_peered":"2024-05-15T13:14:52.762068+0000","last_unstale":"2024-05-15T13:16:42.089990+0000","last_undegraded":"2024-05-15T13:16:42.089990+0000","last_fullsized":"2024-05-15T13:16:42.089990+0000","mapping_epoch":658,"log_start":"608'800","ondisk_log_start":"608'800","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"633'6037","last_scrub_stamp":"2024-05-15T13:13:44.144120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:44.144120+0000","objects_scrubbed":5996,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1443,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191889408,"num_objects":2928,"num_object_clones":0,"num_object_copies":8784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6895,"num_objects_unfound":0,"num_objects_dirty":2928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10900,"num_write_kb":442496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,3],"acting":[1,8,3],"avail_no_missing":["1","3","8"],"object_location_counts":[{"shards":"1,3,8","objects":-2925}],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"94'139","reported_seq":1306,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:34.089249+0000","last_change":"2024-05-15T13:13:18.973897+0000","last_active":"2024-05-15T13:16:34.089249+0000","last_peered":"2024-05-15T13:16:34.089249+0000","last_clean":"2024-05-15T13:16:34.089249+0000","last_became_active":"2024-05-15T13:13:11.373548+0000","last_became_peered":"2024-05-15T13:13:11.373548+0000","last_unstale":"2024-05-15T13:16:34.089249+0000","last_undegraded":"2024-05-15T13:16:34.089249+0000","last_fullsized":"2024-05-15T13:16:34.089249+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:18.973644+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:18.973644+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"700'10848","reported_seq":19287,"reported_epoch":700,"state":"active+recovering+undersized+remapped","last_fresh":"2024-05-15T13:16:41.903998+0000","last_change":"2024-05-15T13:13:48.759714+0000","last_active":"2024-05-15T13:16:41.903998+0000","last_peered":"2024-05-15T13:16:41.903998+0000","last_clean":"2024-05-15T13:13:46.349397+0000","last_became_active":"2024-05-15T13:13:48.627785+0000","last_became_peered":"2024-05-15T13:13:48.627785+0000","last_unstale":"2024-05-15T13:16:41.903998+0000","last_undegraded":"2024-05-15T13:16:41.903998+0000","last_fullsized":"2024-05-15T13:13:48.369601+0000","mapping_epoch":635,"log_start":"608'800","ondisk_log_start":"608'800","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189267968,"num_objects":2888,"num_object_clones":0,"num_object_copies":8664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2006,"num_objects_unfound":0,"num_objects_dirty":2888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10848,"num_write_kb":439552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4112,"num_bytes_recovered":269484032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,9],"avail_no_missing":["0","9","10"],"object_location_counts":[{"shards":"0,9","objects":882},{"shards":"0,9,10","objects":2006}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"94'139","reported_seq":1324,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:34.087707+0000","last_change":"2024-05-15T13:14:54.570374+0000","last_active":"2024-05-15T13:16:34.087707+0000","last_peered":"2024-05-15T13:16:34.087707+0000","last_clean":"2024-05-15T13:16:34.087707+0000","last_became_active":"2024-05-15T13:14:54.570004+0000","last_became_peered":"2024-05-15T13:14:54.570004+0000","last_unstale":"2024-05-15T13:16:34.087707+0000","last_undegraded":"2024-05-15T13:16:34.087707+0000","last_fullsized":"2024-05-15T13:16:34.087707+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:18.468889+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:18.468889+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":220,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"700'10820","reported_seq":17524,"reported_epoch":700,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-05-15T13:16:42.975910+0000","last_change":"2024-05-15T13:15:41.847709+0000","last_active":"2024-05-15T13:16:42.975910+0000","last_peered":"2024-05-15T13:16:42.975910+0000","last_clean":"2024-05-15T13:15:39.478703+0000","last_became_active":"2024-05-15T13:15:41.782308+0000","last_became_peered":"2024-05-15T13:15:41.782308+0000","last_unstale":"2024-05-15T13:16:42.975910+0000","last_undegraded":"2024-05-15T13:15:41.568837+0000","last_fullsized":"2024-05-15T13:15:41.534990+0000","mapping_epoch":676,"log_start":"608'800","ondisk_log_start":"608'800","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187695104,"num_objects":2864,"num_object_clones":0,"num_object_copies":8592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22,"num_objects_misplaced":1283,"num_objects_unfound":0,"num_objects_dirty":2864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10820,"num_write_kb":437888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3243,"num_bytes_recovered":212533248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,0],"acting":[11,0],"avail_no_missing":["11","0","1"],"object_location_counts":[{"shards":"0,1,11","objects":1283},{"shards":"0,11","objects":1559}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"94'139","reported_seq":1293,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:36.359754+0000","last_change":"2024-05-15T13:16:36.359811+0000","last_active":"2024-05-15T13:16:36.359754+0000","last_peered":"2024-05-15T13:16:36.359754+0000","last_clean":"2024-05-15T13:16:36.359754+0000","last_became_active":"2024-05-15T13:06:46.426908+0000","last_became_peered":"2024-05-15T13:06:46.426908+0000","last_unstale":"2024-05-15T13:16:36.359754+0000","last_undegraded":"2024-05-15T13:16:36.359754+0000","last_fullsized":"2024-05-15T13:16:36.359754+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:16:36.359645+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:16:36.359645+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:17:52.241397+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"700'10438","reported_seq":10714,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:43.257462+0000","last_change":"2024-05-15T13:13:39.097208+0000","last_active":"2024-05-15T13:16:43.257462+0000","last_peered":"2024-05-15T13:16:43.257462+0000","last_clean":"2024-05-15T13:16:43.257462+0000","last_became_active":"2024-05-15T13:12:29.926405+0000","last_became_peered":"2024-05-15T13:12:29.926405+0000","last_unstale":"2024-05-15T13:16:43.257462+0000","last_undegraded":"2024-05-15T13:16:43.257462+0000","last_fullsized":"2024-05-15T13:16:43.257462+0000","mapping_epoch":599,"log_start":"608'400","ondisk_log_start":"608'400","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"629'5457","last_scrub_stamp":"2024-05-15T13:13:39.097042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:39.097042+0000","objects_scrubbed":5417,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1513,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181403648,"num_objects":2768,"num_object_clones":0,"num_object_copies":8304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10438,"num_write_kb":422592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"94'139","reported_seq":1285,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:34.080678+0000","last_change":"2024-05-15T13:13:20.984754+0000","last_active":"2024-05-15T13:16:34.080678+0000","last_peered":"2024-05-15T13:16:34.080678+0000","last_clean":"2024-05-15T13:16:34.080678+0000","last_became_active":"2024-05-15T13:06:46.424667+0000","last_became_peered":"2024-05-15T13:06:46.424667+0000","last_unstale":"2024-05-15T13:16:34.080678+0000","last_undegraded":"2024-05-15T13:16:34.080678+0000","last_fullsized":"2024-05-15T13:16:34.080678+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:20.984642+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:20.984642+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.f","version":"700'10678","reported_seq":10891,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:41.900201+0000","last_change":"2024-05-15T13:12:29.925178+0000","last_active":"2024-05-15T13:16:41.900201+0000","last_peered":"2024-05-15T13:16:41.900201+0000","last_clean":"2024-05-15T13:16:41.900201+0000","last_became_active":"2024-05-15T13:12:29.924959+0000","last_became_peered":"2024-05-15T13:12:29.924959+0000","last_unstale":"2024-05-15T13:16:41.900201+0000","last_undegraded":"2024-05-15T13:16:41.900201+0000","last_fullsized":"2024-05-15T13:16:41.900201+0000","mapping_epoch":599,"log_start":"608'600","ondisk_log_start":"608'600","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185204736,"num_objects":2826,"num_object_clones":0,"num_object_copies":8478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10678,"num_write_kb":432128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"94'139","reported_seq":1064,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:34.080640+0000","last_change":"2024-05-15T13:15:43.612283+0000","last_active":"2024-05-15T13:16:34.080640+0000","last_peered":"2024-05-15T13:16:34.080640+0000","last_clean":"2024-05-15T13:16:34.080640+0000","last_became_active":"2024-05-15T13:15:43.611913+0000","last_became_peered":"2024-05-15T13:15:43.611913+0000","last_unstale":"2024-05-15T13:16:34.080640+0000","last_undegraded":"2024-05-15T13:16:34.080640+0000","last_fullsized":"2024-05-15T13:16:34.080640+0000","mapping_epoch":678,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:27.843747+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:27.843747+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.c","version":"700'11086","reported_seq":21595,"reported_epoch":700,"state":"active+recovering+undersized+remapped","last_fresh":"2024-05-15T13:16:44.291537+0000","last_change":"2024-05-15T13:14:09.989528+0000","last_active":"2024-05-15T13:16:44.291537+0000","last_peered":"2024-05-15T13:16:44.291537+0000","last_clean":"2024-05-15T13:14:07.437104+0000","last_became_active":"2024-05-15T13:14:09.928863+0000","last_became_peered":"2024-05-15T13:14:09.928863+0000","last_unstale":"2024-05-15T13:16:44.291537+0000","last_undegraded":"2024-05-15T13:16:44.291537+0000","last_fullsized":"2024-05-15T13:14:09.468866+0000","mapping_epoch":647,"log_start":"608'1000","ondisk_log_start":"608'1000","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"641'6420","last_scrub_stamp":"2024-05-15T13:13:54.703460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:54.703460+0000","objects_scrubbed":6408,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":989,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":2816,"num_object_clones":0,"num_object_copies":8448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1774,"num_objects_unfound":0,"num_objects_dirty":2816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11086,"num_write_kb":444864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5110,"num_bytes_recovered":334888960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,11],"acting":[6,11],"avail_no_missing":["6","2","11"],"object_location_counts":[{"shards":"2,6,11","objects":1774},{"shards":"6,11","objects":1042}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"700'10784","reported_seq":17606,"reported_epoch":700,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-05-15T13:16:43.524612+0000","last_change":"2024-05-15T13:14:20.686326+0000","last_active":"2024-05-15T13:16:43.524612+0000","last_peered":"2024-05-15T13:16:43.524612+0000","last_clean":"2024-05-15T13:14:07.355352+0000","last_became_active":"2024-05-15T13:14:09.930774+0000","last_became_peered":"2024-05-15T13:14:09.930774+0000","last_unstale":"2024-05-15T13:16:43.524612+0000","last_undegraded":"2024-05-15T13:14:20.671875+0000","last_fullsized":"2024-05-15T13:14:09.468986+0000","mapping_epoch":647,"log_start":"608'700","ondisk_log_start":"608'700","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'5339","last_scrub_stamp":"2024-05-15T13:13:35.290438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:35.290438+0000","objects_scrubbed":5292,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186646528,"num_objects":2848,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":695,"num_objects_misplaced":2776,"num_objects_unfound":0,"num_objects_dirty":2848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10784,"num_write_kb":436224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3273,"num_bytes_recovered":214171648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,10],"acting":[2,10],"avail_no_missing":["2","1","10"],"object_location_counts":[{"shards":"1,2,10","objects":2776},{"shards":"2,10","objects":-623}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"94'139","reported_seq":972,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:34.099291+0000","last_change":"2024-05-15T13:13:18.190667+0000","last_active":"2024-05-15T13:16:34.099291+0000","last_peered":"2024-05-15T13:16:34.099291+0000","last_clean":"2024-05-15T13:16:34.099291+0000","last_became_active":"2024-05-15T13:13:16.477771+0000","last_became_peered":"2024-05-15T13:13:16.477771+0000","last_unstale":"2024-05-15T13:16:34.099291+0000","last_undegraded":"2024-05-15T13:16:34.099291+0000","last_fullsized":"2024-05-15T13:16:34.099291+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:18.190529+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:18.190529+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,1],"acting":[3,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"700'10834","reported_seq":11106,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:41.911862+0000","last_change":"2024-05-15T13:13:31.369534+0000","last_active":"2024-05-15T13:16:41.911862+0000","last_peered":"2024-05-15T13:16:41.911862+0000","last_clean":"2024-05-15T13:16:41.911862+0000","last_became_active":"2024-05-15T13:12:29.920800+0000","last_became_peered":"2024-05-15T13:12:29.920800+0000","last_unstale":"2024-05-15T13:16:41.911862+0000","last_undegraded":"2024-05-15T13:16:41.911862+0000","last_fullsized":"2024-05-15T13:16:41.911862+0000","mapping_epoch":599,"log_start":"608'800","ondisk_log_start":"608'800","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'5126","last_scrub_stamp":"2024-05-15T13:13:31.369265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:31.369265+0000","objects_scrubbed":5076,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1450,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192675840,"num_objects":2940,"num_object_clones":0,"num_object_copies":8820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10834,"num_write_kb":440768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"94'139","reported_seq":1065,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:34.083877+0000","last_change":"2024-05-15T13:13:16.831084+0000","last_active":"2024-05-15T13:16:34.083877+0000","last_peered":"2024-05-15T13:16:34.083877+0000","last_clean":"2024-05-15T13:16:34.083877+0000","last_became_active":"2024-05-15T13:13:16.472318+0000","last_became_peered":"2024-05-15T13:13:16.472318+0000","last_unstale":"2024-05-15T13:16:34.083877+0000","last_undegraded":"2024-05-15T13:16:34.083877+0000","last_fullsized":"2024-05-15T13:16:34.083877+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:16.830998+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:16.830998+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"700'10856","reported_seq":11138,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:40.989043+0000","last_change":"2024-05-15T13:13:47.375374+0000","last_active":"2024-05-15T13:16:40.989043+0000","last_peered":"2024-05-15T13:16:40.989043+0000","last_clean":"2024-05-15T13:16:40.989043+0000","last_became_active":"2024-05-15T13:12:29.923300+0000","last_became_peered":"2024-05-15T13:12:29.923300+0000","last_unstale":"2024-05-15T13:16:40.989043+0000","last_undegraded":"2024-05-15T13:16:40.989043+0000","last_fullsized":"2024-05-15T13:16:40.989043+0000","mapping_epoch":599,"log_start":"608'800","ondisk_log_start":"608'800","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"634'6127","last_scrub_stamp":"2024-05-15T13:13:47.375250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:47.375250+0000","objects_scrubbed":6127,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":586,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187039744,"num_objects":2854,"num_object_clones":0,"num_object_copies":8562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10856,"num_write_kb":438720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.c","version":"94'139","reported_seq":724,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:34.098819+0000","last_change":"2024-05-15T13:13:17.140519+0000","last_active":"2024-05-15T13:16:34.098819+0000","last_peered":"2024-05-15T13:16:34.098819+0000","last_clean":"2024-05-15T13:16:34.098819+0000","last_became_active":"2024-05-15T13:13:16.476972+0000","last_became_peered":"2024-05-15T13:13:16.476972+0000","last_unstale":"2024-05-15T13:16:34.098819+0000","last_undegraded":"2024-05-15T13:16:34.098819+0000","last_fullsized":"2024-05-15T13:16:34.098819+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.140398+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.140398+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"700'11359","reported_seq":23823,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:42.306499+0000","last_change":"2024-05-15T13:16:18.785284+0000","last_active":"2024-05-15T13:16:42.306499+0000","last_peered":"2024-05-15T13:16:42.306499+0000","last_clean":"2024-05-15T13:16:42.306499+0000","last_became_active":"2024-05-15T13:16:18.691546+0000","last_became_peered":"2024-05-15T13:16:18.691546+0000","last_unstale":"2024-05-15T13:16:42.306499+0000","last_undegraded":"2024-05-15T13:16:42.306499+0000","last_fullsized":"2024-05-15T13:16:42.306499+0000","mapping_epoch":693,"log_start":"609'1300","ondisk_log_start":"609'1300","created":599,"last_epoch_clean":694,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":624,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190447616,"num_objects":2906,"num_object_clones":0,"num_object_copies":8718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10930,"num_write_kb":442752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6108,"num_bytes_recovered":378404864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"94'139","reported_seq":712,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:34.080458+0000","last_change":"2024-05-15T13:13:17.391026+0000","last_active":"2024-05-15T13:16:34.080458+0000","last_peered":"2024-05-15T13:16:34.080458+0000","last_clean":"2024-05-15T13:16:34.080458+0000","last_became_active":"2024-05-15T13:13:16.478031+0000","last_became_peered":"2024-05-15T13:13:16.478031+0000","last_unstale":"2024-05-15T13:16:34.080458+0000","last_undegraded":"2024-05-15T13:16:34.080458+0000","last_fullsized":"2024-05-15T13:16:34.080458+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.390903+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.390903+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,11],"acting":[1,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"700'10523","reported_seq":16982,"reported_epoch":700,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-05-15T13:16:41.928537+0000","last_change":"2024-05-15T13:14:52.877732+0000","last_active":"2024-05-15T13:16:41.928537+0000","last_peered":"2024-05-15T13:16:41.928537+0000","last_clean":"2024-05-15T13:14:50.332505+0000","last_became_active":"2024-05-15T13:14:52.801033+0000","last_became_peered":"2024-05-15T13:14:52.801033+0000","last_unstale":"2024-05-15T13:16:41.928537+0000","last_undegraded":"2024-05-15T13:14:52.518661+0000","last_fullsized":"2024-05-15T13:14:52.462597+0000","mapping_epoch":658,"log_start":"608'500","ondisk_log_start":"608'500","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"630'5615","last_scrub_stamp":"2024-05-15T13:13:40.897467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:40.897467+0000","objects_scrubbed":5565,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1781,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183762968,"num_objects":2805,"num_object_clones":0,"num_object_copies":8415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":118,"num_objects_misplaced":2467,"num_objects_unfound":0,"num_objects_dirty":2805,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10523,"num_write_kb":426433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3089,"num_bytes_recovered":201719808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[0,1],"avail_no_missing":["0","1","3"],"object_location_counts":[{"shards":"0,1","objects":220},{"shards":"0,1,3","objects":2467}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.b","version":"94'139","reported_seq":1317,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:34.081810+0000","last_change":"2024-05-15T13:13:17.088630+0000","last_active":"2024-05-15T13:16:34.081810+0000","last_peered":"2024-05-15T13:16:34.081810+0000","last_clean":"2024-05-15T13:16:34.081810+0000","last_became_active":"2024-05-15T13:13:16.484252+0000","last_became_peered":"2024-05-15T13:13:16.484252+0000","last_unstale":"2024-05-15T13:16:34.081810+0000","last_undegraded":"2024-05-15T13:16:34.081810+0000","last_fullsized":"2024-05-15T13:16:34.081810+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.088496+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.088496+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"94'139","reported_seq":1299,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:34.103504+0000","last_change":"2024-05-15T13:16:33.288194+0000","last_active":"2024-05-15T13:16:34.103504+0000","last_peered":"2024-05-15T13:16:34.103504+0000","last_clean":"2024-05-15T13:16:34.103504+0000","last_became_active":"2024-05-15T13:13:16.472800+0000","last_became_peered":"2024-05-15T13:13:16.472800+0000","last_unstale":"2024-05-15T13:16:34.103504+0000","last_undegraded":"2024-05-15T13:16:34.103504+0000","last_fullsized":"2024-05-15T13:16:34.103504+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:16:33.288090+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:16:33.288090+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:17:34.458556+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,8],"acting":[5,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"94'139","reported_seq":1073,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:34.088844+0000","last_change":"2024-05-15T13:13:17.150999+0000","last_active":"2024-05-15T13:16:34.088844+0000","last_peered":"2024-05-15T13:16:34.088844+0000","last_clean":"2024-05-15T13:16:34.088844+0000","last_became_active":"2024-05-15T13:13:16.476744+0000","last_became_peered":"2024-05-15T13:13:16.476744+0000","last_unstale":"2024-05-15T13:16:34.088844+0000","last_undegraded":"2024-05-15T13:16:34.088844+0000","last_fullsized":"2024-05-15T13:16:34.088844+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.150897+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.150897+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":232,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"94'139","reported_seq":1315,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:34.103675+0000","last_change":"2024-05-15T13:13:49.927086+0000","last_active":"2024-05-15T13:16:34.103675+0000","last_peered":"2024-05-15T13:16:34.103675+0000","last_clean":"2024-05-15T13:16:34.103675+0000","last_became_active":"2024-05-15T13:13:49.926687+0000","last_became_peered":"2024-05-15T13:13:49.926687+0000","last_unstale":"2024-05-15T13:16:34.103675+0000","last_undegraded":"2024-05-15T13:16:34.103675+0000","last_fullsized":"2024-05-15T13:16:34.103675+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:18.145708+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:18.145708+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"94'139","reported_seq":1323,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-05-15T13:16:34.087610+0000","last_change":"2024-05-15T13:13:20.455813+0000","last_active":"2024-05-15T13:16:34.087610+0000","last_peered":"2024-05-15T13:16:34.087610+0000","last_clean":"2024-05-15T13:16:34.087610+0000","last_became_active":"2024-05-15T13:13:16.478488+0000","last_became_peered":"2024-05-15T13:13:16.478488+0000","last_unstale":"2024-05-15T13:16:34.087610+0000","last_undegraded":"2024-05-15T13:16:34.087610+0000","last_fullsized":"2024-05-15T13:16:34.087610+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:20.455652+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:20.455652+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3001286680,"num_objects":45797,"num_object_clones":0,"num_object_copies":137391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9181,"num_objects_misplaced":19965,"num_objects_unfound":0,"num_objects_dirty":45797,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172329,"num_write_kb":6979969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31817,"num_bytes_recovered":2058158080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11422937088,"data_stored":11422924872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160914,"ondisk_log_size":160914,"up":48,"acting":41,"num_store_stats":10},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057016,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2122308,"kb_used_data":1541372,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":91622844,"statfs":{"total":95995035648,"available":93821792256,"internally_reserved":0,"allocated":1578364928,"data_stored":1577889176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":594868678},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351356,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2327172,"kb_used_data":1655996,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":91417980,"statfs":{"total":95995035648,"available":93612011520,"internally_reserved":0,"allocated":1695739904,"data_stored":1695264152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":687274432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514881,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1808260,"kb_used_data":1023228,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":91936892,"statfs":{"total":95995035648,"available":94143377408,"internally_reserved":0,"allocated":1047785472,"data_stored":1047309720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":803862976},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874630,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":982340,"kb_used_data":567868,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92762812,"statfs":{"total":95995035648,"available":94989119488,"internally_reserved":0,"allocated":581496832,"data_stored":581021080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":240518168970,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430609,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":872580,"kb_used_data":205564,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":92872572,"statfs":{"total":95995035648,"available":95101513728,"internally_reserved":0,"allocated":210497536,"data_stored":210021784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":683014598},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575274,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2213064,"kb_used_data":1383552,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":91532088,"statfs":{"total":95995035648,"available":93728858112,"internally_reserved":0,"allocated":1416757248,"data_stored":1416277424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":849344965},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738802,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1840968,"kb_used_data":1105920,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":91904184,"statfs":{"total":95995035648,"available":94109884416,"internally_reserved":0,"allocated":1132462080,"data_stored":1131982256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":752679365},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411751,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1878668,"kb_used_data":1346244,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":91866484,"statfs":{"total":95995035648,"available":94071279616,"internally_reserved":0,"allocated":1378553856,"data_stored":1378074552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150112,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2006416,"kb_used_data":1185160,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":91738736,"statfs":{"total":95995035648,"available":93940465664,"internally_reserved":0,"allocated":1213603840,"data_stored":1213120464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":840825287},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986588,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692247,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1817100,"kb_used_data":1186180,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":91928052,"statfs":{"total":95995035648,"available":94134325248,"internally_reserved":0,"allocated":1214648320,"data_stored":1214169016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1126305792,"data_stored":1126301720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1415188480,"data_stored":1415184408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1369767936,"data_stored":1369767936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1210454016,"data_stored":1210449944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1212153856,"data_stored":1212153856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":208207872,"data_stored":208207872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":578945024,"data_stored":578945024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1042415616,"data_stored":1042415616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1693843456,"data_stored":1693843456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1565655040,"data_stored":1565655040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:16:46.682 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:16:46.718 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:16:46.744 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:46.764 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:16:46.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:46.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:16:46.869 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:16:46.904 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:47.576 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:16:47.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:47.585 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:16:47.686 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:16:47.746 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:16:47.800 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:47.811 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:16:47.851 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:16:47.892 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:16:47.915 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:47 smithi177 ceph-mon[44303]: pgmap v1813: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 389 op/s; 9181/136974 objects degraded (6.703%); 19965/136974 objects misplaced (14.576%); 13 MiB/s, 212 objects/s recovering 2024-05-15T13:16:47.921 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:16:47.964 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:16:48.005 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:16:48.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:47 smithi027 ceph-mon[39795]: pgmap v1813: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 389 op/s; 9181/136974 objects degraded (6.703%); 19965/136974 objects misplaced (14.576%); 13 MiB/s, 212 objects/s recovering 2024-05-15T13:16:48.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:47 smithi120 ceph-mon[43961]: pgmap v1813: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 389 op/s; 9181/136974 objects degraded (6.703%); 19965/136974 objects misplaced (14.576%); 13 MiB/s, 212 objects/s recovering 2024-05-15T13:16:48.072 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:16:48.125 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:16:48.189 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:16:48.239 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:16:48.284 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:16:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:16:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:16:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:16:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:16:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:16:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:16:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:16:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:16:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:16:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:16:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:16:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:16:48.677 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:48 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1432455554' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:16:48.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:48 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1432455554' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:16:49.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:48 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1432455554' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:16:49.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:49 smithi177 ceph-mon[44303]: pgmap v1814: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 510 op/s; 8893/132183 objects degraded (6.728%); 19148/132183 objects misplaced (14.486%); 18 MiB/s, 291 objects/s recovering 2024-05-15T13:16:49.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:49 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 8893/132183 objects degraded (6.728%), 5 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:50.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:49 smithi027 ceph-mon[39795]: pgmap v1814: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 510 op/s; 8893/132183 objects degraded (6.728%); 19148/132183 objects misplaced (14.486%); 18 MiB/s, 291 objects/s recovering 2024-05-15T13:16:50.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:49 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 8893/132183 objects degraded (6.728%), 5 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:50.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:49 smithi120 ceph-mon[43961]: pgmap v1814: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 510 op/s; 8893/132183 objects degraded (6.728%); 19148/132183 objects misplaced (14.486%); 18 MiB/s, 291 objects/s recovering 2024-05-15T13:16:50.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:49 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 8893/132183 objects degraded (6.728%), 5 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:50.760 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:16:50.760 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-05-15T13:16:50.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-05-15T13:16:50.760 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T13:16:50.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.761 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:144576 6.1 6:8183c504:::benchmark_data_smithi177_122278_object14205:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e543)", 2024-05-15T13:16:50.761 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:09:39.713691+0000", 2024-05-15T13:16:50.761 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 431.04773201900002, 2024-05-15T13:16:50.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.067610067999999995, 2024-05-15T13:16:50.761 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:16:50.761 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:50.761 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:16:50.761 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:16:50.761 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:16:50.761 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 144576 2024-05-15T13:16:50.761 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.761 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:16:50.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.762 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:16:50.762 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.713691+0000", 2024-05-15T13:16:50.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.762 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:16:50.762 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.713691+0000", 2024-05-15T13:16:50.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.762 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:16:50.762 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.713692+0000", 2024-05-15T13:16:50.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3769999999999999e-06 2024-05-15T13:16:50.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.763 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:16:50.763 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.713699+0000", 2024-05-15T13:16:50.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5549999999999999e-06 2024-05-15T13:16:50.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.763 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:16:50.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.758980+0000", 2024-05-15T13:16:50.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.045281165999999998 2024-05-15T13:16:50.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:16:50.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.758993+0000", 2024-05-15T13:16:50.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2727e-05 2024-05-15T13:16:50.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:16:50.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.776322+0000", 2024-05-15T13:16:50.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017329252999999999 2024-05-15T13:16:50.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:16:50.765 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.776421+0000", 2024-05-15T13:16:50.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9538999999999997e-05 2024-05-15T13:16:50.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:16:50.765 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.776536+0000", 2024-05-15T13:16:50.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114318 2024-05-15T13:16:50.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:16:50.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.780680+0000", 2024-05-15T13:16:50.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0041442179999999999 2024-05-15T13:16:50.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.781079+0000", 2024-05-15T13:16:50.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000398608 2024-05-15T13:16:50.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.781197+0000", 2024-05-15T13:16:50.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011812899999999999 2024-05-15T13:16:50.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:16:50.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.781252+0000", 2024-05-15T13:16:50.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5730999999999999e-05 2024-05-15T13:16:50.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:16:50.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:09:39.781301+0000", 2024-05-15T13:16:50.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8446999999999999e-05 2024-05-15T13:16:50.767 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.767 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:16:50.767 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.768 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:171719 6.1 6:894701a7:::benchmark_data_smithi177_122278_object41348:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-05-15T13:16:50.768 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:10:11.074106+0000", 2024-05-15T13:16:50.768 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 399.68731688000003, 2024-05-15T13:16:50.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.22356810699999999, 2024-05-15T13:16:50.768 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:16:50.768 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:50.768 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:16:50.768 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:16:50.768 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:16:50.768 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 171719 2024-05-15T13:16:50.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.768 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:16:50.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:16:50.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.074106+0000", 2024-05-15T13:16:50.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:16:50.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.074106+0000", 2024-05-15T13:16:50.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:16:50.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.074107+0000", 2024-05-15T13:16:50.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3030000000000001e-06 2024-05-15T13:16:50.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:16:50.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.074113+0000", 2024-05-15T13:16:50.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0870000000000003e-06 2024-05-15T13:16:50.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:16:50.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.074129+0000", 2024-05-15T13:16:50.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5164000000000001e-05 2024-05-15T13:16:50.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:16:50.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.074137+0000", 2024-05-15T13:16:50.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6640000000000004e-06 2024-05-15T13:16:50.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.771 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:16:50.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.074188+0000", 2024-05-15T13:16:50.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0835999999999999e-05 2024-05-15T13:16:50.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.771 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:16:50.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.074321+0000", 2024-05-15T13:16:50.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013313499999999999 2024-05-15T13:16:50.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:16:50.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.074447+0000", 2024-05-15T13:16:50.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012615099999999999 2024-05-15T13:16:50.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:16:50.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.078599+0000", 2024-05-15T13:16:50.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0041520710000000002 2024-05-15T13:16:50.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.078921+0000", 2024-05-15T13:16:50.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00032142800000000001 2024-05-15T13:16:50.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.773 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.773 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.297598+0000", 2024-05-15T13:16:50.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21867695500000001 2024-05-15T13:16:50.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.773 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:16:50.773 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.297638+0000", 2024-05-15T13:16:50.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9703000000000003e-05 2024-05-15T13:16:50.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.773 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:16:50.773 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.297674+0000", 2024-05-15T13:16:50.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6609999999999997e-05 2024-05-15T13:16:50.773 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.774 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:16:50.774 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.774 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:171726 6.1 6:8a0d7b87:::benchmark_data_smithi177_122278_object41355:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-05-15T13:16:50.774 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:10:11.079031+0000", 2024-05-15T13:16:50.775 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 399.68239154999998, 2024-05-15T13:16:50.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.224700821, 2024-05-15T13:16:50.775 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:16:50.775 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:50.775 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:16:50.775 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:16:50.775 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:16:50.775 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 171726 2024-05-15T13:16:50.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.775 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:16:50.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:16:50.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.079031+0000", 2024-05-15T13:16:50.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:16:50.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.079031+0000", 2024-05-15T13:16:50.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:16:50.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.079032+0000", 2024-05-15T13:16:50.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.04e-06 2024-05-15T13:16:50.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:16:50.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.079038+0000", 2024-05-15T13:16:50.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3809999999999998e-06 2024-05-15T13:16:50.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:16:50.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.079047+0000", 2024-05-15T13:16:50.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5300000000000002e-06 2024-05-15T13:16:50.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:16:50.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.079053+0000", 2024-05-15T13:16:50.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4979999999999997e-06 2024-05-15T13:16:50.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:16:50.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.079090+0000", 2024-05-15T13:16:50.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7360999999999997e-05 2024-05-15T13:16:50.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:16:50.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.079177+0000", 2024-05-15T13:16:50.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7177999999999998e-05 2024-05-15T13:16:50.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:16:50.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.079251+0000", 2024-05-15T13:16:50.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3866000000000001e-05 2024-05-15T13:16:50.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:16:50.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.088147+0000", 2024-05-15T13:16:50.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0088959680000000006 2024-05-15T13:16:50.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.088273+0000", 2024-05-15T13:16:50.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000125816 2024-05-15T13:16:50.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.303606+0000", 2024-05-15T13:16:50.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21533274999999999 2024-05-15T13:16:50.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:16:50.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.303672+0000", 2024-05-15T13:16:50.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6210999999999993e-05 2024-05-15T13:16:50.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:16:50.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.303732+0000", 2024-05-15T13:16:50.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0222e-05 2024-05-15T13:16:50.780 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.780 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:16:50.780 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.780 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:171728 6.1 6:88889c72:::benchmark_data_smithi177_122278_object41357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-05-15T13:16:50.780 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:10:11.081031+0000", 2024-05-15T13:16:50.780 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 399.680391783, 2024-05-15T13:16:50.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.22285792900000001, 2024-05-15T13:16:50.781 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:16:50.781 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:50.781 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:16:50.781 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:16:50.781 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:16:50.781 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 171728 2024-05-15T13:16:50.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.781 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:16:50.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:16:50.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.081031+0000", 2024-05-15T13:16:50.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:16:50.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.081031+0000", 2024-05-15T13:16:50.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:16:50.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.081033+0000", 2024-05-15T13:16:50.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4750000000000001e-06 2024-05-15T13:16:50.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:16:50.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.081038+0000", 2024-05-15T13:16:50.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0919999999999997e-06 2024-05-15T13:16:50.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:16:50.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.081050+0000", 2024-05-15T13:16:50.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2520999999999999e-05 2024-05-15T13:16:50.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:16:50.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.081058+0000", 2024-05-15T13:16:50.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6159999999999996e-06 2024-05-15T13:16:50.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:16:50.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.083212+0000", 2024-05-15T13:16:50.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021537510000000002 2024-05-15T13:16:50.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:16:50.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.083338+0000", 2024-05-15T13:16:50.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000126539 2024-05-15T13:16:50.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:16:50.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.083452+0000", 2024-05-15T13:16:50.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000113853 2024-05-15T13:16:50.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:16:50.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.088290+0000", 2024-05-15T13:16:50.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0048382759999999999 2024-05-15T13:16:50.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.088420+0000", 2024-05-15T13:16:50.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013007200000000001 2024-05-15T13:16:50.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.303784+0000", 2024-05-15T13:16:50.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21536370799999999 2024-05-15T13:16:50.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:16:50.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.303845+0000", 2024-05-15T13:16:50.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1014999999999998e-05 2024-05-15T13:16:50.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:16:50.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.303889+0000", 2024-05-15T13:16:50.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4011000000000003e-05 2024-05-15T13:16:50.786 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.786 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:16:50.787 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.787 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:171736 6.1 6:8b94f17a:::benchmark_data_smithi177_122278_object41365:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-05-15T13:16:50.787 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:10:11.093648+0000", 2024-05-15T13:16:50.787 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 399.66777500900002, 2024-05-15T13:16:50.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21039311999999999, 2024-05-15T13:16:50.787 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:16:50.787 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:50.787 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:16:50.787 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:16:50.787 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:16:50.787 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 171736 2024-05-15T13:16:50.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.787 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:16:50.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:16:50.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.093648+0000", 2024-05-15T13:16:50.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:16:50.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.093648+0000", 2024-05-15T13:16:50.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:16:50.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.093649+0000", 2024-05-15T13:16:50.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.567e-06 2024-05-15T13:16:50.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:16:50.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.093657+0000", 2024-05-15T13:16:50.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.153e-06 2024-05-15T13:16:50.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:16:50.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.093671+0000", 2024-05-15T13:16:50.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4059999999999999e-05 2024-05-15T13:16:50.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:16:50.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.093678+0000", 2024-05-15T13:16:50.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6340000000000001e-06 2024-05-15T13:16:50.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:16:50.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.093731+0000", 2024-05-15T13:16:50.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2839e-05 2024-05-15T13:16:50.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:16:50.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.093839+0000", 2024-05-15T13:16:50.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108297 2024-05-15T13:16:50.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:16:50.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.093922+0000", 2024-05-15T13:16:50.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2224000000000004e-05 2024-05-15T13:16:50.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:16:50.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.097930+0000", 2024-05-15T13:16:50.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0040084600000000001 2024-05-15T13:16:50.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.098147+0000", 2024-05-15T13:16:50.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000217006 2024-05-15T13:16:50.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.303940+0000", 2024-05-15T13:16:50.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20579251000000001 2024-05-15T13:16:50.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:16:50.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.303997+0000", 2024-05-15T13:16:50.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.77e-05 2024-05-15T13:16:50.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:16:50.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.304041+0000", 2024-05-15T13:16:50.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.367e-05 2024-05-15T13:16:50.792 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.792 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:16:50.792 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.792 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:171761 6.1 6:88b94fba:::benchmark_data_smithi177_122278_object41390:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-05-15T13:16:50.793 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:10:11.133770+0000", 2024-05-15T13:16:50.793 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 399.62765337399998, 2024-05-15T13:16:50.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17041051800000001, 2024-05-15T13:16:50.793 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:16:50.793 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:50.793 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:16:50.793 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:16:50.793 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:16:50.793 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 171761 2024-05-15T13:16:50.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.793 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:16:50.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:16:50.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.133770+0000", 2024-05-15T13:16:50.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:16:50.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.133770+0000", 2024-05-15T13:16:50.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:16:50.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.133771+0000", 2024-05-15T13:16:50.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6050000000000001e-06 2024-05-15T13:16:50.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:16:50.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.133779+0000", 2024-05-15T13:16:50.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7519999999999996e-06 2024-05-15T13:16:50.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:16:50.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.133794+0000", 2024-05-15T13:16:50.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5077e-05 2024-05-15T13:16:50.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:16:50.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.133803+0000", 2024-05-15T13:16:50.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7520000000000002e-06 2024-05-15T13:16:50.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:16:50.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.133862+0000", 2024-05-15T13:16:50.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9218000000000001e-05 2024-05-15T13:16:50.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:16:50.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.133956+0000", 2024-05-15T13:16:50.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3911000000000003e-05 2024-05-15T13:16:50.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:16:50.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.134037+0000", 2024-05-15T13:16:50.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0840000000000005e-05 2024-05-15T13:16:50.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:16:50.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.143046+0000", 2024-05-15T13:16:50.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0090092090000000007 2024-05-15T13:16:50.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.143096+0000", 2024-05-15T13:16:50.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9804e-05 2024-05-15T13:16:50.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.304085+0000", 2024-05-15T13:16:50.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.160989717 2024-05-15T13:16:50.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:16:50.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.304140+0000", 2024-05-15T13:16:50.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4203999999999998e-05 2024-05-15T13:16:50.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:16:50.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.304180+0000", 2024-05-15T13:16:50.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0429000000000002e-05 2024-05-15T13:16:50.799 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.799 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:16:50.799 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.799 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:171764 6.1 6:84abda3d:::benchmark_data_smithi177_122278_object41393:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-05-15T13:16:50.799 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:10:11.136029+0000", 2024-05-15T13:16:50.799 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 399.62539437800001, 2024-05-15T13:16:50.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17323471300000001, 2024-05-15T13:16:50.799 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:16:50.799 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:50.799 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:16:50.799 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:16:50.799 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:16:50.799 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 171764 2024-05-15T13:16:50.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.800 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:16:50.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:16:50.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.136029+0000", 2024-05-15T13:16:50.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:16:50.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.136029+0000", 2024-05-15T13:16:50.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:16:50.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.136030+0000", 2024-05-15T13:16:50.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.406e-06 2024-05-15T13:16:50.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:16:50.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.136036+0000", 2024-05-15T13:16:50.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0990000000000004e-06 2024-05-15T13:16:50.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:16:50.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.136050+0000", 2024-05-15T13:16:50.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4229999999999999e-05 2024-05-15T13:16:50.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:16:50.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.136058+0000", 2024-05-15T13:16:50.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9619999999999993e-06 2024-05-15T13:16:50.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:16:50.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.138039+0000", 2024-05-15T13:16:50.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001980735 2024-05-15T13:16:50.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:16:50.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.138174+0000", 2024-05-15T13:16:50.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013523100000000001 2024-05-15T13:16:50.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:16:50.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.138278+0000", 2024-05-15T13:16:50.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010347 2024-05-15T13:16:50.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:16:50.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.143192+0000", 2024-05-15T13:16:50.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004914351 2024-05-15T13:16:50.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.143505+0000", 2024-05-15T13:16:50.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031253200000000002 2024-05-15T13:16:50.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.309147+0000", 2024-05-15T13:16:50.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16564235799999999 2024-05-15T13:16:50.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:16:50.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.309210+0000", 2024-05-15T13:16:50.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2899999999999997e-05 2024-05-15T13:16:50.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:16:50.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.309263+0000", 2024-05-15T13:16:50.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3439000000000001e-05 2024-05-15T13:16:50.805 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.805 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:16:50.805 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.805 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:171774 6.1 6:8b1363f6:::benchmark_data_smithi177_122278_object41403:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-05-15T13:16:50.805 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:10:11.150933+0000", 2024-05-15T13:16:50.805 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 399.61048944200002, 2024-05-15T13:16:50.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.15845246199999999, 2024-05-15T13:16:50.805 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:16:50.805 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:50.806 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:16:50.806 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:16:50.806 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:16:50.806 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 171774 2024-05-15T13:16:50.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.806 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:16:50.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:16:50.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.150933+0000", 2024-05-15T13:16:50.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:16:50.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.150933+0000", 2024-05-15T13:16:50.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:16:50.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.150935+0000", 2024-05-15T13:16:50.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3689999999999999e-06 2024-05-15T13:16:50.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:16:50.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.150941+0000", 2024-05-15T13:16:50.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8069999999999996e-06 2024-05-15T13:16:50.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:16:50.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.150952+0000", 2024-05-15T13:16:50.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1691e-05 2024-05-15T13:16:50.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:16:50.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.150959+0000", 2024-05-15T13:16:50.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6680000000000004e-06 2024-05-15T13:16:50.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:16:50.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.151013+0000", 2024-05-15T13:16:50.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4255999999999997e-05 2024-05-15T13:16:50.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:16:50.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.151108+0000", 2024-05-15T13:16:50.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4263000000000002e-05 2024-05-15T13:16:50.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:16:50.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.151187+0000", 2024-05-15T13:16:50.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9808999999999994e-05 2024-05-15T13:16:50.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:16:50.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.155475+0000", 2024-05-15T13:16:50.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0042873579999999998 2024-05-15T13:16:50.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.156445+0000", 2024-05-15T13:16:50.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00097013500000000001 2024-05-15T13:16:50.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.309309+0000", 2024-05-15T13:16:50.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.152863792 2024-05-15T13:16:50.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:16:50.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.309347+0000", 2024-05-15T13:16:50.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8738000000000001e-05 2024-05-15T13:16:50.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:16:50.811 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.309386+0000", 2024-05-15T13:16:50.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8575999999999998e-05 2024-05-15T13:16:50.811 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.811 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:16:50.811 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.811 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:171783 6.1 6:84ab70c0:::benchmark_data_smithi177_122278_object41412:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-05-15T13:16:50.811 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:10:11.173970+0000", 2024-05-15T13:16:50.811 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 399.58745254399997, 2024-05-15T13:16:50.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14187416999999999, 2024-05-15T13:16:50.811 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:16:50.811 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:50.812 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:16:50.812 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:16:50.812 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:16:50.812 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 171783 2024-05-15T13:16:50.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.812 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:16:50.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:16:50.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.173970+0000", 2024-05-15T13:16:50.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:16:50.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.173970+0000", 2024-05-15T13:16:50.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:16:50.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.173972+0000", 2024-05-15T13:16:50.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8640000000000001e-06 2024-05-15T13:16:50.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:16:50.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.173990+0000", 2024-05-15T13:16:50.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7759999999999999e-05 2024-05-15T13:16:50.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:16:50.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.174008+0000", 2024-05-15T13:16:50.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7907999999999999e-05 2024-05-15T13:16:50.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:16:50.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.174017+0000", 2024-05-15T13:16:50.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4259999999999992e-06 2024-05-15T13:16:50.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:16:50.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.174091+0000", 2024-05-15T13:16:50.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3650000000000001e-05 2024-05-15T13:16:50.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:16:50.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.174230+0000", 2024-05-15T13:16:50.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013892099999999999 2024-05-15T13:16:50.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:16:50.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.174370+0000", 2024-05-15T13:16:50.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013983499999999999 2024-05-15T13:16:50.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:16:50.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.178682+0000", 2024-05-15T13:16:50.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004312186 2024-05-15T13:16:50.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.179926+0000", 2024-05-15T13:16:50.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012439510000000001 2024-05-15T13:16:50.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.315737+0000", 2024-05-15T13:16:50.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.135810762 2024-05-15T13:16:50.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:16:50.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.315796+0000", 2024-05-15T13:16:50.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9058000000000003e-05 2024-05-15T13:16:50.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:16:50.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.315845+0000", 2024-05-15T13:16:50.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8848999999999999e-05 2024-05-15T13:16:50.817 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.817 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:16:50.817 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.817 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:171799 6.1 6:8748cec7:::benchmark_data_smithi177_122278_object41428:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-05-15T13:16:50.817 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:10:11.191685+0000", 2024-05-15T13:16:50.817 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 399.56973742000002, 2024-05-15T13:16:50.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13299766199999999, 2024-05-15T13:16:50.817 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:16:50.817 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:50.817 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:16:50.817 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:16:50.818 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:16:50.818 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 171799 2024-05-15T13:16:50.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.818 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:16:50.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:16:50.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.191685+0000", 2024-05-15T13:16:50.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:16:50.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.191685+0000", 2024-05-15T13:16:50.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:16:50.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.191687+0000", 2024-05-15T13:16:50.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.536e-06 2024-05-15T13:16:50.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:16:50.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.191694+0000", 2024-05-15T13:16:50.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9410000000000001e-06 2024-05-15T13:16:50.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:16:50.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.191707+0000", 2024-05-15T13:16:50.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3098999999999999e-05 2024-05-15T13:16:50.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:16:50.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.191716+0000", 2024-05-15T13:16:50.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.473e-06 2024-05-15T13:16:50.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:16:50.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.191767+0000", 2024-05-15T13:16:50.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1854000000000002e-05 2024-05-15T13:16:50.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:16:50.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.191865+0000", 2024-05-15T13:16:50.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7361000000000005e-05 2024-05-15T13:16:50.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:16:50.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.191948+0000", 2024-05-15T13:16:50.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3493999999999997e-05 2024-05-15T13:16:50.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.201040+0000", 2024-05-15T13:16:50.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0090918500000000003 2024-05-15T13:16:50.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:16:50.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.201063+0000", 2024-05-15T13:16:50.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3062e-05 2024-05-15T13:16:50.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.324561+0000", 2024-05-15T13:16:50.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.123497541 2024-05-15T13:16:50.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:16:50.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.324629+0000", 2024-05-15T13:16:50.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8758999999999997e-05 2024-05-15T13:16:50.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:16:50.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.324683+0000", 2024-05-15T13:16:50.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3692e-05 2024-05-15T13:16:50.823 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.823 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:16:50.823 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.823 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:171800 6.1 6:85f76e9a:::benchmark_data_smithi177_122278_object41429:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-05-15T13:16:50.823 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:10:11.194118+0000", 2024-05-15T13:16:50.823 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 399.567305062, 2024-05-15T13:16:50.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13074761400000001, 2024-05-15T13:16:50.823 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:16:50.823 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:50.824 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:16:50.824 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:16:50.824 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:16:50.824 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 171800 2024-05-15T13:16:50.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.824 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:16:50.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:16:50.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.194118+0000", 2024-05-15T13:16:50.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:16:50.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.194118+0000", 2024-05-15T13:16:50.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:16:50.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.194119+0000", 2024-05-15T13:16:50.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2529999999999999e-06 2024-05-15T13:16:50.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:16:50.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.194125+0000", 2024-05-15T13:16:50.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4609999999999999e-06 2024-05-15T13:16:50.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:16:50.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.194139+0000", 2024-05-15T13:16:50.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.486e-05 2024-05-15T13:16:50.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:16:50.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.194148+0000", 2024-05-15T13:16:50.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7150000000000004e-06 2024-05-15T13:16:50.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:16:50.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.195764+0000", 2024-05-15T13:16:50.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016162889999999999 2024-05-15T13:16:50.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:16:50.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.195917+0000", 2024-05-15T13:16:50.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015290200000000001 2024-05-15T13:16:50.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:16:50.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.196068+0000", 2024-05-15T13:16:50.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000150302 2024-05-15T13:16:50.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.201099+0000", 2024-05-15T13:16:50.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0050315999999999998 2024-05-15T13:16:50.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:16:50.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.201114+0000", 2024-05-15T13:16:50.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4622e-05 2024-05-15T13:16:50.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.324761+0000", 2024-05-15T13:16:50.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.123646848 2024-05-15T13:16:50.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:16:50.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.324819+0000", 2024-05-15T13:16:50.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8244e-05 2024-05-15T13:16:50.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:16:50.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.324865+0000", 2024-05-15T13:16:50.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6517999999999997e-05 2024-05-15T13:16:50.832 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.833 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:16:50.833 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.833 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:171811 6.1 6:82ad73b0:::benchmark_data_smithi177_122278_object41440:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-05-15T13:16:50.833 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:10:11.218180+0000", 2024-05-15T13:16:50.833 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 399.54324320400002, 2024-05-15T13:16:50.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.106833724, 2024-05-15T13:16:50.833 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:16:50.833 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:50.833 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:16:50.833 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:16:50.833 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:16:50.833 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 171811 2024-05-15T13:16:50.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.834 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:16:50.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:16:50.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.218180+0000", 2024-05-15T13:16:50.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:16:50.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.218180+0000", 2024-05-15T13:16:50.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:16:50.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.218181+0000", 2024-05-15T13:16:50.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2759999999999999e-06 2024-05-15T13:16:50.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:16:50.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.218186+0000", 2024-05-15T13:16:50.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2320000000000001e-06 2024-05-15T13:16:50.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:16:50.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.218199+0000", 2024-05-15T13:16:50.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2754e-05 2024-05-15T13:16:50.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:16:50.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.218207+0000", 2024-05-15T13:16:50.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0870000000000007e-06 2024-05-15T13:16:50.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:16:50.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.218258+0000", 2024-05-15T13:16:50.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1384999999999997e-05 2024-05-15T13:16:50.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:16:50.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.218344+0000", 2024-05-15T13:16:50.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5752000000000002e-05 2024-05-15T13:16:50.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:16:50.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.218420+0000", 2024-05-15T13:16:50.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5887e-05 2024-05-15T13:16:50.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:16:50.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.222486+0000", 2024-05-15T13:16:50.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0040663449999999999 2024-05-15T13:16:50.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.222681+0000", 2024-05-15T13:16:50.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00019489 2024-05-15T13:16:50.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.324924+0000", 2024-05-15T13:16:50.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.102242609 2024-05-15T13:16:50.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:16:50.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.324974+0000", 2024-05-15T13:16:50.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0529999999999999e-05 2024-05-15T13:16:50.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:16:50.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:10:11.325013+0000", 2024-05-15T13:16:50.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8977000000000003e-05 2024-05-15T13:16:50.839 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.839 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:16:50.839 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.839 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:248595 6.1 6:844450dc:::benchmark_data_smithi177_122278_object118224:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e574)", 2024-05-15T13:16:50.839 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:11:47.163753+0000", 2024-05-15T13:16:50.839 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 303.59767027200002, 2024-05-15T13:16:50.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.089202697999999997, 2024-05-15T13:16:50.839 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:16:50.839 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:50.839 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:16:50.839 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:16:50.840 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:16:50.840 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 248595 2024-05-15T13:16:50.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.840 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:16:50.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:16:50.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.163753+0000", 2024-05-15T13:16:50.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:16:50.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.163753+0000", 2024-05-15T13:16:50.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:16:50.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.163754+0000", 2024-05-15T13:16:50.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.364e-06 2024-05-15T13:16:50.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:16:50.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.163760+0000", 2024-05-15T13:16:50.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8459999999999996e-06 2024-05-15T13:16:50.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:16:50.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.163772+0000", 2024-05-15T13:16:50.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.202e-05 2024-05-15T13:16:50.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:16:50.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.163780+0000", 2024-05-15T13:16:50.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2810000000000008e-06 2024-05-15T13:16:50.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:16:50.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.163822+0000", 2024-05-15T13:16:50.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1937000000000002e-05 2024-05-15T13:16:50.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:16:50.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.163946+0000", 2024-05-15T13:16:50.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000124222 2024-05-15T13:16:50.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:16:50.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.164068+0000", 2024-05-15T13:16:50.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000121925 2024-05-15T13:16:50.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:16:50.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.183927+0000", 2024-05-15T13:16:50.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.019858402000000001 2024-05-15T13:16:50.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.183992+0000", 2024-05-15T13:16:50.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4998000000000003e-05 2024-05-15T13:16:50.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.252834+0000", 2024-05-15T13:16:50.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.068842108999999999 2024-05-15T13:16:50.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:16:50.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.252897+0000", 2024-05-15T13:16:50.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2786999999999994e-05 2024-05-15T13:16:50.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:16:50.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.252955+0000", 2024-05-15T13:16:50.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8807000000000001e-05 2024-05-15T13:16:50.845 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.845 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:16:50.846 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.846 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:248598 6.1 6:8913e8cb:::benchmark_data_smithi177_122278_object118227:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e574)", 2024-05-15T13:16:50.846 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:11:47.164974+0000", 2024-05-15T13:16:50.846 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 303.596449001, 2024-05-15T13:16:50.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.088173935999999994, 2024-05-15T13:16:50.847 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:16:50.847 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:50.847 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:16:50.847 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:16:50.847 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:16:50.847 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 248598 2024-05-15T13:16:50.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.847 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:16:50.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:16:50.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.164974+0000", 2024-05-15T13:16:50.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:16:50.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.164974+0000", 2024-05-15T13:16:50.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:16:50.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.164975+0000", 2024-05-15T13:16:50.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7400000000000002e-07 2024-05-15T13:16:50.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:16:50.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.164980+0000", 2024-05-15T13:16:50.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6420000000000004e-06 2024-05-15T13:16:50.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:16:50.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.165010+0000", 2024-05-15T13:16:50.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9668000000000001e-05 2024-05-15T13:16:50.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:16:50.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.165018+0000", 2024-05-15T13:16:50.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1769999999999998e-06 2024-05-15T13:16:50.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:16:50.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.168856+0000", 2024-05-15T13:16:50.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0038372749999999998 2024-05-15T13:16:50.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:16:50.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.168987+0000", 2024-05-15T13:16:50.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000131338 2024-05-15T13:16:50.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:16:50.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.169109+0000", 2024-05-15T13:16:50.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000121818 2024-05-15T13:16:50.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:16:50.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.183933+0000", 2024-05-15T13:16:50.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014824578999999999 2024-05-15T13:16:50.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.184040+0000", 2024-05-15T13:16:50.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000106502 2024-05-15T13:16:50.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.253056+0000", 2024-05-15T13:16:50.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.069016050999999995 2024-05-15T13:16:50.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:16:50.852 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.253108+0000", 2024-05-15T13:16:50.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2240000000000001e-05 2024-05-15T13:16:50.852 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.852 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:16:50.852 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.253148+0000", 2024-05-15T13:16:50.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9771999999999998e-05 2024-05-15T13:16:50.852 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.852 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:16:50.852 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.852 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.852 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:248599 6.1 6:84a4b21e:::benchmark_data_smithi177_122278_object118228:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e574)", 2024-05-15T13:16:50.852 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:11:47.165031+0000", 2024-05-15T13:16:50.853 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 303.59639209800002, 2024-05-15T13:16:50.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.088239523, 2024-05-15T13:16:50.853 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:16:50.853 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:50.853 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:16:50.853 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:16:50.853 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:16:50.853 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 248599 2024-05-15T13:16:50.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.853 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:16:50.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.853 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:16:50.853 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.165031+0000", 2024-05-15T13:16:50.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:16:50.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.165031+0000", 2024-05-15T13:16:50.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:16:50.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.165031+0000", 2024-05-15T13:16:50.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1099999999999996e-07 2024-05-15T13:16:50.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:16:50.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.165034+0000", 2024-05-15T13:16:50.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5289999999999998e-06 2024-05-15T13:16:50.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:16:50.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.165043+0000", 2024-05-15T13:16:50.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4660000000000005e-06 2024-05-15T13:16:50.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:16:50.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.165049+0000", 2024-05-15T13:16:50.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3140000000000003e-06 2024-05-15T13:16:50.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:16:50.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.173821+0000", 2024-05-15T13:16:50.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.008772317 2024-05-15T13:16:50.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:16:50.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.173938+0000", 2024-05-15T13:16:50.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117348 2024-05-15T13:16:50.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:16:50.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.174052+0000", 2024-05-15T13:16:50.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000113783 2024-05-15T13:16:50.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:16:50.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.184051+0000", 2024-05-15T13:16:50.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.009999338 2024-05-15T13:16:50.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.194242+0000", 2024-05-15T13:16:50.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010190172000000001 2024-05-15T13:16:50.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.253195+0000", 2024-05-15T13:16:50.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.058953859999999997 2024-05-15T13:16:50.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:16:50.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.253230+0000", 2024-05-15T13:16:50.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4449999999999997e-05 2024-05-15T13:16:50.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:16:50.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.253270+0000", 2024-05-15T13:16:50.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0435000000000002e-05 2024-05-15T13:16:50.858 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.858 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:16:50.858 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.858 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:248601 6.1 6:8b43545e:::benchmark_data_smithi177_122278_object118230:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e574)", 2024-05-15T13:16:50.859 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:11:47.165458+0000", 2024-05-15T13:16:50.859 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 303.595964901, 2024-05-15T13:16:50.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.087959865999999998, 2024-05-15T13:16:50.859 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:16:50.859 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:50.859 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:16:50.859 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:16:50.859 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:16:50.859 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 248601 2024-05-15T13:16:50.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.859 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:16:50.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:16:50.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.165458+0000", 2024-05-15T13:16:50.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:16:50.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.165458+0000", 2024-05-15T13:16:50.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:16:50.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.165459+0000", 2024-05-15T13:16:50.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.009e-06 2024-05-15T13:16:50.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:16:50.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.165464+0000", 2024-05-15T13:16:50.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2650000000000004e-06 2024-05-15T13:16:50.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:16:50.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.165477+0000", 2024-05-15T13:16:50.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3106e-05 2024-05-15T13:16:50.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:16:50.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.165486+0000", 2024-05-15T13:16:50.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8079999999999994e-06 2024-05-15T13:16:50.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:16:50.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.178823+0000", 2024-05-15T13:16:50.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013336316000000001 2024-05-15T13:16:50.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:16:50.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.178943+0000", 2024-05-15T13:16:50.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012039 2024-05-15T13:16:50.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:16:50.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.179059+0000", 2024-05-15T13:16:50.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000116593 2024-05-15T13:16:50.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.194297+0000", 2024-05-15T13:16:50.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015237992000000001 2024-05-15T13:16:50.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:16:50.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.194356+0000", 2024-05-15T13:16:50.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8208999999999998e-05 2024-05-15T13:16:50.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.253322+0000", 2024-05-15T13:16:50.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.058966138000000001 2024-05-15T13:16:50.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:16:50.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.253378+0000", 2024-05-15T13:16:50.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6539000000000002e-05 2024-05-15T13:16:50.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:16:50.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.253418+0000", 2024-05-15T13:16:50.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9501e-05 2024-05-15T13:16:50.864 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.864 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:16:50.864 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.864 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:248609 6.1 6:87d5b074:::benchmark_data_smithi177_122278_object118238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e574)", 2024-05-15T13:16:50.864 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:11:47.184854+0000", 2024-05-15T13:16:50.864 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 303.57656869900001, 2024-05-15T13:16:50.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.068690599000000005, 2024-05-15T13:16:50.865 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:16:50.865 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:50.865 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:16:50.865 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:16:50.865 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:16:50.865 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 248609 2024-05-15T13:16:50.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.865 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:16:50.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:16:50.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.184854+0000", 2024-05-15T13:16:50.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:16:50.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.184854+0000", 2024-05-15T13:16:50.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:16:50.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.184856+0000", 2024-05-15T13:16:50.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8059999999999999e-06 2024-05-15T13:16:50.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:16:50.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.184863+0000", 2024-05-15T13:16:50.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1840000000000002e-06 2024-05-15T13:16:50.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:16:50.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.184875+0000", 2024-05-15T13:16:50.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.208e-05 2024-05-15T13:16:50.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:16:50.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.184884+0000", 2024-05-15T13:16:50.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.969e-06 2024-05-15T13:16:50.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:16:50.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.184928+0000", 2024-05-15T13:16:50.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4165000000000002e-05 2024-05-15T13:16:50.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:16:50.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.185038+0000", 2024-05-15T13:16:50.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109492 2024-05-15T13:16:50.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:16:50.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.185114+0000", 2024-05-15T13:16:50.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6576999999999995e-05 2024-05-15T13:16:50.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.194344+0000", 2024-05-15T13:16:50.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0092297569999999999 2024-05-15T13:16:50.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:16:50.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.194361+0000", 2024-05-15T13:16:50.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6339999999999999e-05 2024-05-15T13:16:50.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.253460+0000", 2024-05-15T13:16:50.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.059099504999999997 2024-05-15T13:16:50.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:16:50.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.253510+0000", 2024-05-15T13:16:50.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9565999999999999e-05 2024-05-15T13:16:50.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:16:50.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.253545+0000", 2024-05-15T13:16:50.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5157999999999998e-05 2024-05-15T13:16:50.870 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.870 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:16:50.870 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.870 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:248612 6.1 6:8b2a89d1:::benchmark_data_smithi177_122278_object118241:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e574)", 2024-05-15T13:16:50.870 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:11:47.185088+0000", 2024-05-15T13:16:50.870 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 303.57633467699998, 2024-05-15T13:16:50.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.068600603999999996, 2024-05-15T13:16:50.870 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:16:50.870 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:50.870 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:16:50.871 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:16:50.871 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:16:50.871 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 248612 2024-05-15T13:16:50.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.871 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:16:50.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.871 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:16:50.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.185088+0000", 2024-05-15T13:16:50.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.871 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:16:50.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.185088+0000", 2024-05-15T13:16:50.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:16:50.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.185089+0000", 2024-05-15T13:16:50.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4300000000000001e-07 2024-05-15T13:16:50.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:16:50.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.185095+0000", 2024-05-15T13:16:50.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4349999999999996e-06 2024-05-15T13:16:50.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:16:50.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.185106+0000", 2024-05-15T13:16:50.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0976e-05 2024-05-15T13:16:50.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:16:50.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.185113+0000", 2024-05-15T13:16:50.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.503e-06 2024-05-15T13:16:50.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:16:50.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.189741+0000", 2024-05-15T13:16:50.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0046276490000000002 2024-05-15T13:16:50.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:16:50.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.189871+0000", 2024-05-15T13:16:50.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012995699999999999 2024-05-15T13:16:50.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:16:50.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.189996+0000", 2024-05-15T13:16:50.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012558 2024-05-15T13:16:50.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:16:50.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.199337+0000", 2024-05-15T13:16:50.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0093405490000000001 2024-05-15T13:16:50.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.203044+0000", 2024-05-15T13:16:50.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037072229999999999 2024-05-15T13:16:50.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.253599+0000", 2024-05-15T13:16:50.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.050555296 2024-05-15T13:16:50.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:16:50.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.253651+0000", 2024-05-15T13:16:50.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1214999999999997e-05 2024-05-15T13:16:50.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:16:50.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:11:47.253689+0000", 2024-05-15T13:16:50.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8278000000000002e-05 2024-05-15T13:16:50.876 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.876 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:16:50.876 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.876 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.25735.0:259122 6.1 6:8ee717db:::benchmark_data_smithi177_122278_object128751:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e584)", 2024-05-15T13:16:50.876 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:12:04.619815+0000", 2024-05-15T13:16:50.876 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 286.14160816200001, 2024-05-15T13:16:50.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.104438745, 2024-05-15T13:16:50.876 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:16:50.876 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:50.876 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:16:50.876 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.25735", 2024-05-15T13:16:50.876 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:16:50.876 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 259122 2024-05-15T13:16:50.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.877 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:16:50.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:16:50.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:04.619815+0000", 2024-05-15T13:16:50.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:16:50.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:04.619815+0000", 2024-05-15T13:16:50.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:16:50.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:04.619817+0000", 2024-05-15T13:16:50.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0870000000000001e-06 2024-05-15T13:16:50.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:16:50.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:04.619825+0000", 2024-05-15T13:16:50.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6719999999999993e-06 2024-05-15T13:16:50.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:16:50.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:04.619844+0000", 2024-05-15T13:16:50.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8235000000000001e-05 2024-05-15T13:16:50.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:16:50.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:04.619853+0000", 2024-05-15T13:16:50.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5910000000000002e-06 2024-05-15T13:16:50.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:16:50.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:04.619927+0000", 2024-05-15T13:16:50.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3207000000000006e-05 2024-05-15T13:16:50.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:16:50.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:04.620092+0000", 2024-05-15T13:16:50.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016579 2024-05-15T13:16:50.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:16:50.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:04.620247+0000", 2024-05-15T13:16:50.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015457700000000001 2024-05-15T13:16:50.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:04.625596+0000", 2024-05-15T13:16:50.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0053489649999999998 2024-05-15T13:16:50.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:16:50.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:04.625663+0000", 2024-05-15T13:16:50.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.737e-05 2024-05-15T13:16:50.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:04.724145+0000", 2024-05-15T13:16:50.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.098481289999999999 2024-05-15T13:16:50.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:16:50.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:04.724202+0000", 2024-05-15T13:16:50.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7575999999999999e-05 2024-05-15T13:16:50.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:16:50.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:04.724253+0000", 2024-05-15T13:16:50.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1384999999999997e-05 2024-05-15T13:16:50.882 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.882 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:16:50.882 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.882 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.882 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:24313 7.7 7:eae26020:::benchmark_data_smithi177_134484_object24312:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e609)", 2024-05-15T13:16:50.882 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:12:52.831669+0000", 2024-05-15T13:16:50.882 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 237.92975427799999, 2024-05-15T13:16:50.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.063530526000000004, 2024-05-15T13:16:50.882 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:16:50.882 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:16:50.882 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:16:50.882 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:16:50.882 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:16:50.883 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 24313 2024-05-15T13:16:50.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.883 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:16:50.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:16:50.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:52.831669+0000", 2024-05-15T13:16:50.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:16:50.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:52.831669+0000", 2024-05-15T13:16:50.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:50.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.884 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:16:50.884 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:52.831669+0000", 2024-05-15T13:16:50.884 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1399999999999997e-07 2024-05-15T13:16:50.884 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.884 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.884 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:16:50.884 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:52.831690+0000", 2024-05-15T13:16:50.884 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1263999999999998e-05 2024-05-15T13:16:50.884 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.884 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.884 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:16:50.884 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:52.831699+0000", 2024-05-15T13:16:50.884 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3680000000000002e-06 2024-05-15T13:16:50.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:16:50.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:52.831711+0000", 2024-05-15T13:16:50.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2150000000000001e-05 2024-05-15T13:16:50.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:16:50.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:52.832750+0000", 2024-05-15T13:16:50.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010386569999999999 2024-05-15T13:16:50.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:16:50.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:52.832884+0000", 2024-05-15T13:16:50.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000134194 2024-05-15T13:16:50.886 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.886 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,3", 2024-05-15T13:16:50.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:52.832987+0000", 2024-05-15T13:16:50.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010359199999999999 2024-05-15T13:16:50.886 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.886 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:16:50.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:52.833824+0000", 2024-05-15T13:16:50.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00083618199999999998 2024-05-15T13:16:50.886 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.886 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:52.834037+0000", 2024-05-15T13:16:50.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000213224 2024-05-15T13:16:50.886 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.887 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.887 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:16:50.887 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:52.895105+0000", 2024-05-15T13:16:50.887 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.061068182999999998 2024-05-15T13:16:50.887 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.887 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.887 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:16:50.887 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:52.895161+0000", 2024-05-15T13:16:50.887 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5569000000000003e-05 2024-05-15T13:16:50.887 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:50.887 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:50.887 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:16:50.887 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:52.895199+0000", 2024-05-15T13:16:50.887 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8529000000000003e-05 2024-05-15T13:16:50.887 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.887 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:16:50.888 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.888 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:50.888 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:16:50.888 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:16:51.855 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:16:51.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:51.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:16:51.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:51 smithi177 ceph-mon[44303]: pgmap v1815: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 426 op/s; 8921/130395 objects degraded (6.842%); 18804/130395 objects misplaced (14.421%); 15 MiB/s, 246 objects/s recovering 2024-05-15T13:16:51.961 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:16:51.977 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:51 smithi027 ceph-mon[39795]: pgmap v1815: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 426 op/s; 8921/130395 objects degraded (6.842%); 18804/130395 objects misplaced (14.421%); 15 MiB/s, 246 objects/s recovering 2024-05-15T13:16:52.022 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:52.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:51 smithi120 ceph-mon[43961]: pgmap v1815: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 426 op/s; 8921/130395 objects degraded (6.842%); 18804/130395 objects misplaced (14.421%); 15 MiB/s, 246 objects/s recovering 2024-05-15T13:16:52.108 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:16:52.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:52.113 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:16:52.142 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-05-15T13:16:52.214 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:16:52.242 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:52.263 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:16:52.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:52.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:16:52.369 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:16:52.397 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:52.430 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:16:52.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:52.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:16:52.439 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:16:52.534 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:16:52.570 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:52.576 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:16:52.599 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:16:52.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:52.612 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:16:52.713 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:16:52.817 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:53.040 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:16:53.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:53.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:16:53.150 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:16:53.207 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:54.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:53 smithi027 ceph-mon[39795]: pgmap v1816: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 484 op/s; 8678/126534 objects degraded (6.858%); 18304/126534 objects misplaced (14.466%); 17 MiB/s, 267 objects/s recovering 2024-05-15T13:16:54.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:53 smithi120 ceph-mon[43961]: pgmap v1816: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 484 op/s; 8678/126534 objects degraded (6.858%); 18304/126534 objects misplaced (14.466%); 17 MiB/s, 267 objects/s recovering 2024-05-15T13:16:54.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:53 smithi177 ceph-mon[44303]: pgmap v1816: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 484 op/s; 8678/126534 objects degraded (6.858%); 18304/126534 objects misplaced (14.466%); 17 MiB/s, 267 objects/s recovering 2024-05-15T13:16:54.266 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:16:54.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:54.270 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:16:54.371 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:16:54.464 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:55.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:54 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 8678/126534 objects degraded (6.858%), 5 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:55.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:54 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 8678/126534 objects degraded (6.858%), 5 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:55.076 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:16:55.157 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:54 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 8678/126534 objects degraded (6.858%), 5 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:16:55.455 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:16:55.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:55.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:16:55.564 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:16:55.602 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:56.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:55 smithi027 ceph-mon[39795]: pgmap v1817: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 470 op/s; 8687/123399 objects degraded (7.040%); 17540/123399 objects misplaced (14.214%); 17 MiB/s, 276 objects/s recovering 2024-05-15T13:16:56.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:55 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3643049981' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:16:56.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:55 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:16:56.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:55 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:16:56.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:55 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:16:56.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:55 smithi120 ceph-mon[43961]: pgmap v1817: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 470 op/s; 8687/123399 objects degraded (7.040%); 17540/123399 objects misplaced (14.214%); 17 MiB/s, 276 objects/s recovering 2024-05-15T13:16:56.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:55 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3643049981' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:16:56.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:55 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:16:56.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:55 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:16:56.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:55 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:16:56.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:55 smithi177 ceph-mon[44303]: pgmap v1817: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 470 op/s; 8687/123399 objects degraded (7.040%); 17540/123399 objects misplaced (14.214%); 17 MiB/s, 276 objects/s recovering 2024-05-15T13:16:56.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:55 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3643049981' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:16:56.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:55 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:16:56.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:55 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:16:56.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:55 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:16:56.264 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:16:56.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:56.267 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:16:56.368 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:16:56.474 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:56.581 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:16:56.581 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T13:16:56.582 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:56.582 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.26044.0:178862 7.4 e701/635 7:2444e783:::benchmark_data_smithi177_134484_object69797:head v 701'11270, mlcod=622'4588)", 2024-05-15T13:16:56.582 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:16:56.581024+0000", 2024-05-15T13:16:56.582 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0047082000000000001, 2024-05-15T13:16:56.582 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00473829, 2024-05-15T13:16:56.582 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:16:56.582 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T13:16:56.582 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:16:56.582 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:56.582 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:16:56.582 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:56.581024+0000", 2024-05-15T13:16:56.582 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:56.582 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:56.582 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:56.583 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:16:56.583 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:56.581024+0000", 2024-05-15T13:16:56.583 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:16:56.583 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:56.583 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:56.583 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:16:56.583 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:56.581025+0000", 2024-05-15T13:16:56.583 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3500000000000005e-07 2024-05-15T13:16:56.583 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:56.583 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:56.583 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:16:56.583 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:56.581032+0000", 2024-05-15T13:16:56.583 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1899999999999998e-06 2024-05-15T13:16:56.583 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:56.583 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:56.584 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:16:56.584 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:56.581046+0000", 2024-05-15T13:16:56.584 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3348e-05 2024-05-15T13:16:56.584 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:56.584 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:56.584 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:16:56.584 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:56.581054+0000", 2024-05-15T13:16:56.584 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1859999999999992e-06 2024-05-15T13:16:56.584 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:56.584 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:56.584 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:16:56.584 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:56.581103+0000", 2024-05-15T13:16:56.584 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9084e-05 2024-05-15T13:16:56.584 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:16:56.585 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:16:56.585 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:16:56.585 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:56.581124+0000", 2024-05-15T13:16:56.585 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0954e-05 2024-05-15T13:16:56.585 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:56.585 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:16:56.585 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:56.585 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:16:56.585 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-05-15T13:16:56.585 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 1 2024-05-15T13:16:56.585 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:16:56.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:56 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:16:56.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:56 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:16:56.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:56 smithi027 ceph-mon[39795]: osdmap e701: 12 total, 12 up, 10 in 2024-05-15T13:16:56.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:56 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:16:56.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:56 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:16:56.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:56 smithi177 ceph-mon[44303]: osdmap e701: 12 total, 12 up, 10 in 2024-05-15T13:16:57.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:56 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:16:57.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:56 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:16:57.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:56 smithi120 ceph-mon[43961]: osdmap e701: 12 total, 12 up, 10 in 2024-05-15T13:16:57.931 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:16:57.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:57.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:16:58.043 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:16:58.071 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:57 smithi120 ceph-mon[43961]: pgmap v1819: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 466 op/s; 8687/122979 objects degraded (7.064%); 17540/122979 objects misplaced (14.263%); 16 MiB/s, 263 objects/s recovering 2024-05-15T13:16:58.074 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:58.097 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:16:58.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:58.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:16:58.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:57 smithi177 ceph-mon[44303]: pgmap v1819: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 466 op/s; 8687/122979 objects degraded (7.064%); 17540/122979 objects misplaced (14.263%); 16 MiB/s, 263 objects/s recovering 2024-05-15T13:16:58.200 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:16:58.202 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:16:58.227 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:58.250 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:16:58.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:58.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:16:58.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:57 smithi027 ceph-mon[39795]: pgmap v1819: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 466 op/s; 8687/122979 objects degraded (7.064%); 17540/122979 objects misplaced (14.263%); 16 MiB/s, 263 objects/s recovering 2024-05-15T13:16:58.354 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:16:58.381 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:58.391 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-05-15T13:16:58.401 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:16:58.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:58.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:16:58.506 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:16:58.577 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:58.794 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:16:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:16:59.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:59.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:16:59.144 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:16:59.155 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:58 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:16:59.156 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:58 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:16:59.189 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:59.217 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:16:59.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:59.225 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:16:59.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:58 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:16:59.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:58 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:16:59.286 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:58 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:16:59.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:58 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:16:59.326 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:16:59.355 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:59.377 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:16:59.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:16:59.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:16:59.481 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:16:59.530 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:16:59.832 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:17:00.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:59 smithi177 ceph-mon[44303]: pgmap v1820: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 490 op/s; 8478/117444 objects degraded (7.219%); 16473/117444 objects misplaced (14.026%); 18 MiB/s, 282 objects/s recovering 2024-05-15T13:17:00.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:59 smithi177 ceph-mon[44303]: osdmap e702: 12 total, 12 up, 10 in 2024-05-15T13:17:00.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:16:59 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 8478/117444 objects degraded (7.219%), 5 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:00.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:59 smithi027 ceph-mon[39795]: pgmap v1820: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 490 op/s; 8478/117444 objects degraded (7.219%); 16473/117444 objects misplaced (14.026%); 18 MiB/s, 282 objects/s recovering 2024-05-15T13:17:00.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:59 smithi027 ceph-mon[39795]: osdmap e702: 12 total, 12 up, 10 in 2024-05-15T13:17:00.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:16:59 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 8478/117444 objects degraded (7.219%), 5 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:00.286 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:59 smithi120 ceph-mon[43961]: pgmap v1820: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 490 op/s; 8478/117444 objects degraded (7.219%); 16473/117444 objects misplaced (14.026%); 18 MiB/s, 282 objects/s recovering 2024-05-15T13:17:00.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:59 smithi120 ceph-mon[43961]: osdmap e702: 12 total, 12 up, 10 in 2024-05-15T13:17:00.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:16:59 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 8478/117444 objects degraded (7.219%), 5 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:00.295 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:17:00.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:00.299 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:17:00.399 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:17:00.427 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:00.449 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:17:00.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:00.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:17:00.553 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:17:00.580 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:00.600 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:17:00.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:00.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:17:00.704 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:17:00.706 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:17:00.731 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:17:00.732 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:00.752 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:17:00.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:00.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:17:00.858 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:17:00.868 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:17:00.868 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T13:17:00.868 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-05-15T13:17:00.868 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-05-15T13:17:00.868 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:17:00.884 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:00.894 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:00 smithi177 ceph-mon[44303]: pgmap v1822: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 415 op/s; 8478/116541 objects degraded (7.275%); 16234/116541 objects misplaced (13.930%); 17 MiB/s, 266 objects/s recovering 2024-05-15T13:17:00.895 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:00 smithi177 ceph-mon[44303]: from='client.26185 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:17:00.908 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:17:00.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:00.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:17:01.014 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:17:01.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:00 smithi027 ceph-mon[39795]: pgmap v1822: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 415 op/s; 8478/116541 objects degraded (7.275%); 16234/116541 objects misplaced (13.930%); 17 MiB/s, 266 objects/s recovering 2024-05-15T13:17:01.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:00 smithi027 ceph-mon[39795]: from='client.26185 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:17:01.036 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:00 smithi120 ceph-mon[43961]: pgmap v1822: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 415 op/s; 8478/116541 objects degraded (7.275%); 16234/116541 objects misplaced (13.930%); 17 MiB/s, 266 objects/s recovering 2024-05-15T13:17:01.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:00 smithi120 ceph-mon[43961]: from='client.26185 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:17:01.042 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:01.062 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:17:01.063 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:17:01.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:01.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:17:01.167 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:17:01.194 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:01.213 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:17:01.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:01.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:17:01.317 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:17:01.414 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:02.919 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:17:02.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:02.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:17:03.039 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:17:03.135 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:03.209 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-05-15T13:17:03.591 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:17:03.879 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:03 smithi177 ceph-mon[44303]: pgmap v1823: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 446 op/s; 8215/112677 objects degraded (7.291%); 15719/112677 objects misplaced (13.950%); 16 MiB/s, 252 objects/s recovering 2024-05-15T13:17:03.880 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:03 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:17:03.927 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:17:03.927 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:03 smithi027 ceph-mon[39795]: pgmap v1823: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 446 op/s; 8215/112677 objects degraded (7.291%); 15719/112677 objects misplaced (13.950%); 16 MiB/s, 252 objects/s recovering 2024-05-15T13:17:03.927 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:03 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:17:03.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:03.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:17:04.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:03 smithi120 ceph-mon[43961]: pgmap v1823: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 446 op/s; 8215/112677 objects degraded (7.291%); 15719/112677 objects misplaced (13.950%); 16 MiB/s, 252 objects/s recovering 2024-05-15T13:17:04.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:03 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:17:04.042 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:17:04.070 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:04.090 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:17:04.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:04.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:17:04.193 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:17:04.221 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:04.243 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:17:04.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:04.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:17:04.347 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:17:04.384 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:04.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:04 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 8215/112677 objects degraded (7.291%), 5 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:04.883 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:04 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 8215/112677 objects degraded (7.291%), 5 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:05.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:04 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 8215/112677 objects degraded (7.291%), 5 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:05.153 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:17:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:05.158 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:17:05.259 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:17:05.286 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:05.306 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:17:05.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:05.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:17:05.410 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:17:05.444 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:05.465 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":1822,"stamp":"2024-05-15T13:17:00.518196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2546270776,"num_objects":38847,"num_object_clones":0,"num_object_copies":116541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8478,"num_objects_misplaced":16234,"num_objects_unfound":0,"num_objects_dirty":38847,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179281,"num_write_kb":6979969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35834,"num_bytes_recovered":2320629760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163885,"ondisk_log_size":163885,"up":111,"acting":104,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":16819300,"kb_used_data":10111616,"kb_used_omap":15,"kb_used_meta":6707568,"kb_avail":920632220,"statfs":{"total":959950356480,"available":942727393280,"internally_reserved":0,"allocated":10354294784,"data_stored":10349467873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15956,"internal_metadata":6868550060},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,7,4,8,1,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-218300416,"num_objects":-3331,"num_object_clones":0,"num_object_copies":-9993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-200,"num_objects_misplaced":-2070,"num_objects_unfound":0,"num_objects_dirty":-3331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3331,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2134,"num_bytes_recovered":139460608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.010688"},"pg_stats":[{"pgid":"1.14","version":"94'139","reported_seq":725,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:55.771978+0000","last_change":"2024-05-15T13:13:17.477074+0000","last_active":"2024-05-15T13:16:55.771978+0000","last_peered":"2024-05-15T13:16:55.771978+0000","last_clean":"2024-05-15T13:16:55.771978+0000","last_became_active":"2024-05-15T13:13:16.473818+0000","last_became_peered":"2024-05-15T13:13:16.473818+0000","last_unstale":"2024-05-15T13:16:55.771978+0000","last_undegraded":"2024-05-15T13:16:55.771978+0000","last_fullsized":"2024-05-15T13:16:55.771978+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.476961+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.476961+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1],"acting":[2,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":1300,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:55.765379+0000","last_change":"2024-05-15T13:13:19.224641+0000","last_active":"2024-05-15T13:16:55.765379+0000","last_peered":"2024-05-15T13:16:55.765379+0000","last_clean":"2024-05-15T13:16:55.765379+0000","last_became_active":"2024-05-15T13:13:11.361366+0000","last_became_peered":"2024-05-15T13:13:11.361366+0000","last_unstale":"2024-05-15T13:16:55.765379+0000","last_undegraded":"2024-05-15T13:16:55.765379+0000","last_fullsized":"2024-05-15T13:16:55.765379+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:19.224509+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:19.224509+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"701'11110","reported_seq":17986,"reported_epoch":701,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-05-15T13:16:56.994571+0000","last_change":"2024-05-15T13:14:21.034561+0000","last_active":"2024-05-15T13:16:56.994571+0000","last_peered":"2024-05-15T13:16:56.994571+0000","last_clean":"2024-05-15T13:14:07.430624+0000","last_became_active":"2024-05-15T13:14:09.931862+0000","last_became_peered":"2024-05-15T13:14:09.931862+0000","last_unstale":"2024-05-15T13:16:56.994571+0000","last_undegraded":"2024-05-15T13:14:21.029697+0000","last_fullsized":"2024-05-15T13:14:09.472302+0000","mapping_epoch":647,"log_start":"609'1100","ondisk_log_start":"609'1100","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158597120,"num_objects":2420,"num_object_clones":0,"num_object_copies":7260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1017,"num_objects_misplaced":2350,"num_objects_unfound":0,"num_objects_dirty":2420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11110,"num_write_kb":432960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3328,"num_bytes_recovered":215154688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,10],"acting":[0,10],"avail_no_missing":["0","5","10"],"object_location_counts":[{"shards":"0,5,10","objects":2350},{"shards":"0,10","objects":-947}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"94'139","reported_seq":1300,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:55.758668+0000","last_change":"2024-05-15T13:13:17.949012+0000","last_active":"2024-05-15T13:16:55.758668+0000","last_peered":"2024-05-15T13:16:55.758668+0000","last_clean":"2024-05-15T13:16:55.758668+0000","last_became_active":"2024-05-15T13:13:11.371643+0000","last_became_peered":"2024-05-15T13:13:11.371643+0000","last_unstale":"2024-05-15T13:16:55.758668+0000","last_undegraded":"2024-05-15T13:16:55.758668+0000","last_fullsized":"2024-05-15T13:16:55.758668+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.948900+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.948900+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.2","version":"701'14748","reported_seq":15143,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:57.654579+0000","last_change":"2024-05-15T13:14:18.564785+0000","last_active":"2024-05-15T13:16:57.654579+0000","last_peered":"2024-05-15T13:16:57.654579+0000","last_clean":"2024-05-15T13:16:57.654579+0000","last_became_active":"2024-05-15T13:12:29.918894+0000","last_became_peered":"2024-05-15T13:12:29.918894+0000","last_unstale":"2024-05-15T13:16:57.654579+0000","last_undegraded":"2024-05-15T13:16:57.654579+0000","last_fullsized":"2024-05-15T13:16:57.654579+0000","mapping_epoch":599,"log_start":"623'4701","ondisk_log_start":"623'4701","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":4374,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155910144,"num_objects":2379,"num_object_clones":0,"num_object_copies":7137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11291,"num_write_kb":437440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2],"acting":[3,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"94'139","reported_seq":1054,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:55.753690+0000","last_change":"2024-05-15T13:14:54.577804+0000","last_active":"2024-05-15T13:16:55.753690+0000","last_peered":"2024-05-15T13:16:55.753690+0000","last_clean":"2024-05-15T13:16:55.753690+0000","last_became_active":"2024-05-15T13:14:54.577458+0000","last_became_peered":"2024-05-15T13:14:54.577458+0000","last_unstale":"2024-05-15T13:16:55.753690+0000","last_undegraded":"2024-05-15T13:16:55.753690+0000","last_fullsized":"2024-05-15T13:16:55.753690+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:18.049722+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:18.049722+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.3","version":"701'11046","reported_seq":11322,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:57.607948+0000","last_change":"2024-05-15T13:13:35.644249+0000","last_active":"2024-05-15T13:16:57.607948+0000","last_peered":"2024-05-15T13:16:57.607948+0000","last_clean":"2024-05-15T13:16:57.607948+0000","last_became_active":"2024-05-15T13:12:29.921897+0000","last_became_peered":"2024-05-15T13:12:29.921897+0000","last_unstale":"2024-05-15T13:16:57.607948+0000","last_undegraded":"2024-05-15T13:16:57.607948+0000","last_fullsized":"2024-05-15T13:16:57.607948+0000","mapping_epoch":599,"log_start":"609'1000","ondisk_log_start":"609'1000","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'5271","last_scrub_stamp":"2024-05-15T13:13:35.643996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:35.643996+0000","objects_scrubbed":5237,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158072832,"num_objects":2412,"num_object_clones":0,"num_object_copies":7236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11046,"num_write_kb":430656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"94'139","reported_seq":1293,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:55.860413+0000","last_change":"2024-05-15T13:13:17.875187+0000","last_active":"2024-05-15T13:16:55.860413+0000","last_peered":"2024-05-15T13:16:55.860413+0000","last_clean":"2024-05-15T13:16:55.860413+0000","last_became_active":"2024-05-15T13:13:11.373251+0000","last_became_peered":"2024-05-15T13:13:11.373251+0000","last_unstale":"2024-05-15T13:16:55.860413+0000","last_undegraded":"2024-05-15T13:16:55.860413+0000","last_fullsized":"2024-05-15T13:16:55.860413+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.875054+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.875054+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.0","version":"701'11269","reported_seq":11527,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:58.609035+0000","last_change":"2024-05-15T13:16:24.441683+0000","last_active":"2024-05-15T13:16:58.609035+0000","last_peered":"2024-05-15T13:16:58.609035+0000","last_clean":"2024-05-15T13:16:58.609035+0000","last_became_active":"2024-05-15T13:12:29.924027+0000","last_became_peered":"2024-05-15T13:12:29.924027+0000","last_unstale":"2024-05-15T13:16:58.609035+0000","last_undegraded":"2024-05-15T13:16:58.609035+0000","last_fullsized":"2024-05-15T13:16:58.609035+0000","mapping_epoch":599,"log_start":"609'1200","ondisk_log_start":"609'1200","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"694'10259","last_scrub_stamp":"2024-05-15T13:16:24.441404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:16:24.441404+0000","objects_scrubbed":3481,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:17:37.997733+0000","scrub_duration":1000,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160890880,"num_objects":2455,"num_object_clones":0,"num_object_copies":7365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11269,"num_write_kb":439168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"94'139","reported_seq":1070,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-05-15T13:16:58.875622+0000","last_change":"2024-05-15T13:14:11.522798+0000","last_active":"2024-05-15T13:16:58.875622+0000","last_peered":"2024-05-15T13:16:58.875622+0000","last_clean":"2024-05-15T13:16:58.875622+0000","last_became_active":"2024-05-15T13:14:11.522490+0000","last_became_peered":"2024-05-15T13:14:11.522490+0000","last_unstale":"2024-05-15T13:16:58.875622+0000","last_undegraded":"2024-05-15T13:16:58.875622+0000","last_fullsized":"2024-05-15T13:16:58.875622+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.847198+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.847198+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"701'11100","reported_seq":23754,"reported_epoch":701,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-15T13:16:56.839122+0000","last_change":"2024-05-15T13:14:52.829919+0000","last_active":"2024-05-15T13:16:56.839122+0000","last_peered":"2024-05-15T13:16:56.839122+0000","last_clean":"2024-05-15T13:13:46.353513+0000","last_became_active":"2024-05-15T13:14:52.812683+0000","last_became_peered":"2024-05-15T13:14:52.812683+0000","last_unstale":"2024-05-15T13:16:56.839122+0000","last_undegraded":"2024-05-15T13:14:52.601890+0000","last_fullsized":"2024-05-15T13:14:52.453432+0000","mapping_epoch":658,"log_start":"608'1000","ondisk_log_start":"608'1000","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"633'5877","last_scrub_stamp":"2024-05-15T13:13:44.171370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:44.171370+0000","objects_scrubbed":5827,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1393,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164102144,"num_objects":2504,"num_object_clones":0,"num_object_copies":7512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11100,"num_write_kb":435328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4300,"num_bytes_recovered":281673728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3],"avail_no_missing":["10","5","11"],"object_location_counts":[{"shards":"1,3,10","objects":86},{"shards":"1,10,11","objects":704},{"shards":"5,10,11","objects":1714}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"94'139","reported_seq":1071,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:55.860064+0000","last_change":"2024-05-15T13:15:43.615336+0000","last_active":"2024-05-15T13:16:55.860064+0000","last_peered":"2024-05-15T13:16:55.860064+0000","last_clean":"2024-05-15T13:16:55.860064+0000","last_became_active":"2024-05-15T13:15:43.614970+0000","last_became_peered":"2024-05-15T13:15:43.614970+0000","last_unstale":"2024-05-15T13:16:55.860064+0000","last_undegraded":"2024-05-15T13:16:55.860064+0000","last_fullsized":"2024-05-15T13:16:55.860064+0000","mapping_epoch":678,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:20.062928+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:20.062928+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,0],"acting":[9,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.7","version":"701'11347","reported_seq":11684,"reported_epoch":701,"state":"active+recovery_wait+remapped","last_fresh":"2024-05-15T13:16:57.359872+0000","last_change":"2024-05-15T13:14:52.781625+0000","last_active":"2024-05-15T13:16:57.359872+0000","last_peered":"2024-05-15T13:16:57.359872+0000","last_clean":"2024-05-15T13:14:50.323746+0000","last_became_active":"2024-05-15T13:14:52.762068+0000","last_became_peered":"2024-05-15T13:14:52.762068+0000","last_unstale":"2024-05-15T13:16:57.359872+0000","last_undegraded":"2024-05-15T13:16:57.359872+0000","last_fullsized":"2024-05-15T13:16:57.359872+0000","mapping_epoch":658,"log_start":"609'1300","ondisk_log_start":"609'1300","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"633'6037","last_scrub_stamp":"2024-05-15T13:13:44.144120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:44.144120+0000","objects_scrubbed":5996,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1443,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162594816,"num_objects":2481,"num_object_clones":0,"num_object_copies":7443,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6888,"num_objects_unfound":0,"num_objects_dirty":2481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11347,"num_write_kb":442496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,3],"acting":[1,8,3],"avail_no_missing":["1","3","8"],"object_location_counts":[{"shards":"1,3,8","objects":-3365}],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"94'139","reported_seq":1308,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:55.753006+0000","last_change":"2024-05-15T13:13:18.973897+0000","last_active":"2024-05-15T13:16:55.753006+0000","last_peered":"2024-05-15T13:16:55.753006+0000","last_clean":"2024-05-15T13:16:55.753006+0000","last_became_active":"2024-05-15T13:13:11.373548+0000","last_became_peered":"2024-05-15T13:13:11.373548+0000","last_unstale":"2024-05-15T13:16:55.753006+0000","last_undegraded":"2024-05-15T13:16:55.753006+0000","last_fullsized":"2024-05-15T13:16:55.753006+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:18.973644+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:18.973644+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"701'11277","reported_seq":20704,"reported_epoch":701,"state":"active+recovering+undersized+remapped","last_fresh":"2024-05-15T13:16:57.009232+0000","last_change":"2024-05-15T13:13:48.759714+0000","last_active":"2024-05-15T13:16:57.009232+0000","last_peered":"2024-05-15T13:16:57.009232+0000","last_clean":"2024-05-15T13:13:46.349397+0000","last_became_active":"2024-05-15T13:13:48.627785+0000","last_became_peered":"2024-05-15T13:13:48.627785+0000","last_unstale":"2024-05-15T13:16:57.009232+0000","last_undegraded":"2024-05-15T13:16:57.009232+0000","last_fullsized":"2024-05-15T13:13:48.369601+0000","mapping_epoch":635,"log_start":"609'1200","ondisk_log_start":"609'1200","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161153024,"num_objects":2459,"num_object_clones":0,"num_object_copies":7377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1514,"num_objects_unfound":0,"num_objects_dirty":2459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11277,"num_write_kb":439552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4604,"num_bytes_recovered":301727744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,9],"avail_no_missing":["0","9","10"],"object_location_counts":[{"shards":"0,9","objects":945},{"shards":"0,9,10","objects":1514}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"94'139","reported_seq":1326,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:55.762178+0000","last_change":"2024-05-15T13:14:54.570374+0000","last_active":"2024-05-15T13:16:55.762178+0000","last_peered":"2024-05-15T13:16:55.762178+0000","last_clean":"2024-05-15T13:16:55.762178+0000","last_became_active":"2024-05-15T13:14:54.570004+0000","last_became_peered":"2024-05-15T13:14:54.570004+0000","last_unstale":"2024-05-15T13:16:55.762178+0000","last_undegraded":"2024-05-15T13:16:55.762178+0000","last_fullsized":"2024-05-15T13:16:55.762178+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:18.468889+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:18.468889+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":220,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"701'11260","reported_seq":20326,"reported_epoch":701,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-05-15T13:16:57.994562+0000","last_change":"2024-05-15T13:15:41.847709+0000","last_active":"2024-05-15T13:16:57.994562+0000","last_peered":"2024-05-15T13:16:57.994562+0000","last_clean":"2024-05-15T13:15:39.478703+0000","last_became_active":"2024-05-15T13:15:41.782308+0000","last_became_peered":"2024-05-15T13:15:41.782308+0000","last_unstale":"2024-05-15T13:16:57.994562+0000","last_undegraded":"2024-05-15T13:15:41.568837+0000","last_fullsized":"2024-05-15T13:15:41.534990+0000","mapping_epoch":676,"log_start":"609'1200","ondisk_log_start":"609'1200","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158859264,"num_objects":2424,"num_object_clones":0,"num_object_copies":7272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22,"num_objects_misplaced":103,"num_objects_unfound":0,"num_objects_dirty":2424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11260,"num_write_kb":437888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4423,"num_bytes_recovered":289865728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,0],"acting":[11,0],"avail_no_missing":["11","0","1"],"object_location_counts":[{"shards":"0,1,11","objects":103},{"shards":"0,11","objects":2299}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"94'139","reported_seq":1295,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:55.754544+0000","last_change":"2024-05-15T13:16:36.359811+0000","last_active":"2024-05-15T13:16:55.754544+0000","last_peered":"2024-05-15T13:16:55.754544+0000","last_clean":"2024-05-15T13:16:55.754544+0000","last_became_active":"2024-05-15T13:06:46.426908+0000","last_became_peered":"2024-05-15T13:06:46.426908+0000","last_unstale":"2024-05-15T13:16:55.754544+0000","last_undegraded":"2024-05-15T13:16:55.754544+0000","last_fullsized":"2024-05-15T13:16:55.754544+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:16:36.359645+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:16:36.359645+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:17:52.241397+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"701'10878","reported_seq":11156,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:57.509806+0000","last_change":"2024-05-15T13:13:39.097208+0000","last_active":"2024-05-15T13:16:57.509806+0000","last_peered":"2024-05-15T13:16:57.509806+0000","last_clean":"2024-05-15T13:16:57.509806+0000","last_became_active":"2024-05-15T13:12:29.926405+0000","last_became_peered":"2024-05-15T13:12:29.926405+0000","last_unstale":"2024-05-15T13:16:57.509806+0000","last_undegraded":"2024-05-15T13:16:57.509806+0000","last_fullsized":"2024-05-15T13:16:57.509806+0000","mapping_epoch":599,"log_start":"608'800","ondisk_log_start":"608'800","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"629'5457","last_scrub_stamp":"2024-05-15T13:13:39.097042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:39.097042+0000","objects_scrubbed":5417,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1513,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152567808,"num_objects":2328,"num_object_clones":0,"num_object_copies":6984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10878,"num_write_kb":422592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"94'139","reported_seq":1287,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:55.860227+0000","last_change":"2024-05-15T13:13:20.984754+0000","last_active":"2024-05-15T13:16:55.860227+0000","last_peered":"2024-05-15T13:16:55.860227+0000","last_clean":"2024-05-15T13:16:55.860227+0000","last_became_active":"2024-05-15T13:06:46.424667+0000","last_became_peered":"2024-05-15T13:06:46.424667+0000","last_unstale":"2024-05-15T13:16:55.860227+0000","last_undegraded":"2024-05-15T13:16:55.860227+0000","last_fullsized":"2024-05-15T13:16:55.860227+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:20.984642+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:20.984642+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.f","version":"701'11105","reported_seq":11320,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:56.865957+0000","last_change":"2024-05-15T13:12:29.925178+0000","last_active":"2024-05-15T13:16:56.865957+0000","last_peered":"2024-05-15T13:16:56.865957+0000","last_clean":"2024-05-15T13:16:56.865957+0000","last_became_active":"2024-05-15T13:12:29.924959+0000","last_became_peered":"2024-05-15T13:12:29.924959+0000","last_unstale":"2024-05-15T13:16:56.865957+0000","last_undegraded":"2024-05-15T13:16:56.865957+0000","last_fullsized":"2024-05-15T13:16:56.865957+0000","mapping_epoch":599,"log_start":"609'1100","ondisk_log_start":"609'1100","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157220864,"num_objects":2399,"num_object_clones":0,"num_object_copies":7197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11105,"num_write_kb":432128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"94'139","reported_seq":1066,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:55.860247+0000","last_change":"2024-05-15T13:15:43.612283+0000","last_active":"2024-05-15T13:16:55.860247+0000","last_peered":"2024-05-15T13:16:55.860247+0000","last_clean":"2024-05-15T13:16:55.860247+0000","last_became_active":"2024-05-15T13:15:43.611913+0000","last_became_peered":"2024-05-15T13:15:43.611913+0000","last_unstale":"2024-05-15T13:16:55.860247+0000","last_undegraded":"2024-05-15T13:16:55.860247+0000","last_fullsized":"2024-05-15T13:16:55.860247+0000","mapping_epoch":678,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:27.843747+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:27.843747+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.c","version":"702'11511","reported_seq":23358,"reported_epoch":702,"state":"active+recovering+undersized+remapped","last_fresh":"2024-05-15T13:16:59.317613+0000","last_change":"2024-05-15T13:14:09.989528+0000","last_active":"2024-05-15T13:16:59.317613+0000","last_peered":"2024-05-15T13:16:59.317613+0000","last_clean":"2024-05-15T13:14:07.437104+0000","last_became_active":"2024-05-15T13:14:09.928863+0000","last_became_peered":"2024-05-15T13:14:09.928863+0000","last_unstale":"2024-05-15T13:16:59.317613+0000","last_undegraded":"2024-05-15T13:16:59.317613+0000","last_fullsized":"2024-05-15T13:14:09.468866+0000","mapping_epoch":647,"log_start":"609'1500","ondisk_log_start":"609'1500","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"641'6420","last_scrub_stamp":"2024-05-15T13:13:54.703460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:54.703460+0000","objects_scrubbed":6408,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":989,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156696576,"num_objects":2391,"num_object_clones":0,"num_object_copies":7173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1107,"num_objects_unfound":0,"num_objects_dirty":2391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11511,"num_write_kb":444864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5777,"num_bytes_recovered":378601472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,11],"acting":[6,11],"avail_no_missing":["6","2","11"],"object_location_counts":[{"shards":"2,6,11","objects":1107},{"shards":"6,11","objects":1284}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"701'11188","reported_seq":18696,"reported_epoch":701,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-05-15T13:16:57.555089+0000","last_change":"2024-05-15T13:14:20.686326+0000","last_active":"2024-05-15T13:16:57.555089+0000","last_peered":"2024-05-15T13:16:57.555089+0000","last_clean":"2024-05-15T13:14:07.355352+0000","last_became_active":"2024-05-15T13:14:09.930774+0000","last_became_peered":"2024-05-15T13:14:09.930774+0000","last_unstale":"2024-05-15T13:16:57.555089+0000","last_undegraded":"2024-05-15T13:14:20.671875+0000","last_fullsized":"2024-05-15T13:14:09.468986+0000","mapping_epoch":647,"log_start":"609'1100","ondisk_log_start":"609'1100","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'5339","last_scrub_stamp":"2024-05-15T13:13:35.290438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:35.290438+0000","objects_scrubbed":5292,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160169984,"num_objects":2444,"num_object_clones":0,"num_object_copies":7332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":760,"num_objects_misplaced":2370,"num_objects_unfound":0,"num_objects_dirty":2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11188,"num_write_kb":436224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3614,"num_bytes_recovered":236453888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,10],"acting":[2,10],"avail_no_missing":["2","1","10"],"object_location_counts":[{"shards":"1,2,10","objects":2370},{"shards":"2,10","objects":-686}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"94'139","reported_seq":974,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:55.756828+0000","last_change":"2024-05-15T13:13:18.190667+0000","last_active":"2024-05-15T13:16:55.756828+0000","last_peered":"2024-05-15T13:16:55.756828+0000","last_clean":"2024-05-15T13:16:55.756828+0000","last_became_active":"2024-05-15T13:13:16.477771+0000","last_became_peered":"2024-05-15T13:13:16.477771+0000","last_unstale":"2024-05-15T13:16:55.756828+0000","last_undegraded":"2024-05-15T13:16:55.756828+0000","last_fullsized":"2024-05-15T13:16:55.756828+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:18.190529+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:18.190529+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,1],"acting":[3,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"701'11260","reported_seq":11534,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:56.964727+0000","last_change":"2024-05-15T13:13:31.369534+0000","last_active":"2024-05-15T13:16:56.964727+0000","last_peered":"2024-05-15T13:16:56.964727+0000","last_clean":"2024-05-15T13:16:56.964727+0000","last_became_active":"2024-05-15T13:12:29.920800+0000","last_became_peered":"2024-05-15T13:12:29.920800+0000","last_unstale":"2024-05-15T13:16:56.964727+0000","last_undegraded":"2024-05-15T13:16:56.964727+0000","last_fullsized":"2024-05-15T13:16:56.964727+0000","mapping_epoch":599,"log_start":"609'1200","ondisk_log_start":"609'1200","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'5126","last_scrub_stamp":"2024-05-15T13:13:31.369265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:31.369265+0000","objects_scrubbed":5076,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1450,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164757504,"num_objects":2514,"num_object_clones":0,"num_object_copies":7542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11260,"num_write_kb":440768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"94'139","reported_seq":1069,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-05-15T13:16:58.875797+0000","last_change":"2024-05-15T13:13:16.831084+0000","last_active":"2024-05-15T13:16:58.875797+0000","last_peered":"2024-05-15T13:16:58.875797+0000","last_clean":"2024-05-15T13:16:58.875797+0000","last_became_active":"2024-05-15T13:13:16.472318+0000","last_became_peered":"2024-05-15T13:13:16.472318+0000","last_unstale":"2024-05-15T13:16:58.875797+0000","last_undegraded":"2024-05-15T13:16:58.875797+0000","last_fullsized":"2024-05-15T13:16:58.875797+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:16.830998+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:16.830998+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"701'11298","reported_seq":11583,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:55.993843+0000","last_change":"2024-05-15T13:13:47.375374+0000","last_active":"2024-05-15T13:16:55.993843+0000","last_peered":"2024-05-15T13:16:55.993843+0000","last_clean":"2024-05-15T13:16:55.993843+0000","last_became_active":"2024-05-15T13:12:29.923300+0000","last_became_peered":"2024-05-15T13:12:29.923300+0000","last_unstale":"2024-05-15T13:16:55.993843+0000","last_undegraded":"2024-05-15T13:16:55.993843+0000","last_fullsized":"2024-05-15T13:16:55.993843+0000","mapping_epoch":599,"log_start":"609'1200","ondisk_log_start":"609'1200","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"634'6127","last_scrub_stamp":"2024-05-15T13:13:47.375250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:47.375250+0000","objects_scrubbed":6127,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":586,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158072832,"num_objects":2412,"num_object_clones":0,"num_object_copies":7236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11298,"num_write_kb":438720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.c","version":"94'139","reported_seq":726,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:55.756516+0000","last_change":"2024-05-15T13:13:17.140519+0000","last_active":"2024-05-15T13:16:55.756516+0000","last_peered":"2024-05-15T13:16:55.756516+0000","last_clean":"2024-05-15T13:16:55.756516+0000","last_became_active":"2024-05-15T13:13:16.476972+0000","last_became_peered":"2024-05-15T13:13:16.476972+0000","last_unstale":"2024-05-15T13:16:55.756516+0000","last_undegraded":"2024-05-15T13:16:55.756516+0000","last_fullsized":"2024-05-15T13:16:55.756516+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.140398+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.140398+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"701'11800","reported_seq":24266,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:57.322508+0000","last_change":"2024-05-15T13:16:18.785284+0000","last_active":"2024-05-15T13:16:57.322508+0000","last_peered":"2024-05-15T13:16:57.322508+0000","last_clean":"2024-05-15T13:16:57.322508+0000","last_became_active":"2024-05-15T13:16:18.691546+0000","last_became_peered":"2024-05-15T13:16:18.691546+0000","last_unstale":"2024-05-15T13:16:57.322508+0000","last_undegraded":"2024-05-15T13:16:57.322508+0000","last_fullsized":"2024-05-15T13:16:57.322508+0000","mapping_epoch":693,"log_start":"610'1700","ondisk_log_start":"610'1700","created":599,"last_epoch_clean":694,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":624,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161546240,"num_objects":2465,"num_object_clones":0,"num_object_copies":7395,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11371,"num_write_kb":442752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6108,"num_bytes_recovered":378404864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"94'139","reported_seq":714,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:55.759073+0000","last_change":"2024-05-15T13:13:17.391026+0000","last_active":"2024-05-15T13:16:55.759073+0000","last_peered":"2024-05-15T13:16:55.759073+0000","last_clean":"2024-05-15T13:16:55.759073+0000","last_became_active":"2024-05-15T13:13:16.478031+0000","last_became_peered":"2024-05-15T13:13:16.478031+0000","last_unstale":"2024-05-15T13:16:55.759073+0000","last_undegraded":"2024-05-15T13:16:55.759073+0000","last_fullsized":"2024-05-15T13:16:55.759073+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.390903+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.390903+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,11],"acting":[1,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"701'10970","reported_seq":18558,"reported_epoch":701,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-05-15T13:16:57.016271+0000","last_change":"2024-05-15T13:14:52.877732+0000","last_active":"2024-05-15T13:16:57.016271+0000","last_peered":"2024-05-15T13:16:57.016271+0000","last_clean":"2024-05-15T13:14:50.332505+0000","last_became_active":"2024-05-15T13:14:52.801033+0000","last_became_peered":"2024-05-15T13:14:52.801033+0000","last_unstale":"2024-05-15T13:16:57.016271+0000","last_undegraded":"2024-05-15T13:14:52.518661+0000","last_fullsized":"2024-05-15T13:14:52.462597+0000","mapping_epoch":658,"log_start":"608'900","ondisk_log_start":"608'900","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"630'5615","last_scrub_stamp":"2024-05-15T13:13:40.897467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:40.897467+0000","objects_scrubbed":5565,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1781,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154468376,"num_objects":2358,"num_object_clones":0,"num_object_copies":7074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":118,"num_objects_misplaced":1902,"num_objects_unfound":0,"num_objects_dirty":2358,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10970,"num_write_kb":426433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3654,"num_bytes_recovered":238747648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[0,1],"avail_no_missing":["0","1","3"],"object_location_counts":[{"shards":"0,1","objects":338},{"shards":"0,1,3","objects":1902}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.b","version":"94'139","reported_seq":1319,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:55.758617+0000","last_change":"2024-05-15T13:13:17.088630+0000","last_active":"2024-05-15T13:16:55.758617+0000","last_peered":"2024-05-15T13:16:55.758617+0000","last_clean":"2024-05-15T13:16:55.758617+0000","last_became_active":"2024-05-15T13:13:16.484252+0000","last_became_peered":"2024-05-15T13:13:16.484252+0000","last_unstale":"2024-05-15T13:16:55.758617+0000","last_undegraded":"2024-05-15T13:16:55.758617+0000","last_fullsized":"2024-05-15T13:16:55.758617+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.088496+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.088496+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"94'139","reported_seq":1301,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:55.754532+0000","last_change":"2024-05-15T13:16:33.288194+0000","last_active":"2024-05-15T13:16:55.754532+0000","last_peered":"2024-05-15T13:16:55.754532+0000","last_clean":"2024-05-15T13:16:55.754532+0000","last_became_active":"2024-05-15T13:13:16.472800+0000","last_became_peered":"2024-05-15T13:13:16.472800+0000","last_unstale":"2024-05-15T13:16:55.754532+0000","last_undegraded":"2024-05-15T13:16:55.754532+0000","last_fullsized":"2024-05-15T13:16:55.754532+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:16:33.288090+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:16:33.288090+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:17:34.458556+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,8],"acting":[5,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"94'139","reported_seq":1075,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:55.753299+0000","last_change":"2024-05-15T13:13:17.150999+0000","last_active":"2024-05-15T13:16:55.753299+0000","last_peered":"2024-05-15T13:16:55.753299+0000","last_clean":"2024-05-15T13:16:55.753299+0000","last_became_active":"2024-05-15T13:13:16.476744+0000","last_became_peered":"2024-05-15T13:13:16.476744+0000","last_unstale":"2024-05-15T13:16:55.753299+0000","last_undegraded":"2024-05-15T13:16:55.753299+0000","last_fullsized":"2024-05-15T13:16:55.753299+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.150897+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.150897+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":232,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"94'139","reported_seq":1325,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:55.754584+0000","last_change":"2024-05-15T13:16:44.320180+0000","last_active":"2024-05-15T13:16:55.754584+0000","last_peered":"2024-05-15T13:16:55.754584+0000","last_clean":"2024-05-15T13:16:55.754584+0000","last_became_active":"2024-05-15T13:13:49.926687+0000","last_became_peered":"2024-05-15T13:13:49.926687+0000","last_unstale":"2024-05-15T13:16:55.754584+0000","last_undegraded":"2024-05-15T13:16:55.754584+0000","last_fullsized":"2024-05-15T13:16:55.754584+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:16:44.320095+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:16:44.320095+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:05.122264+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"94'139","reported_seq":1325,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-05-15T13:16:55.762074+0000","last_change":"2024-05-15T13:13:20.455813+0000","last_active":"2024-05-15T13:16:55.762074+0000","last_peered":"2024-05-15T13:16:55.762074+0000","last_clean":"2024-05-15T13:16:55.762074+0000","last_became_active":"2024-05-15T13:13:16.478488+0000","last_became_peered":"2024-05-15T13:13:16.478488+0000","last_unstale":"2024-05-15T13:16:55.762074+0000","last_undegraded":"2024-05-15T13:16:55.762074+0000","last_fullsized":"2024-05-15T13:16:55.762074+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:20.455652+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:20.455652+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2545680408,"num_objects":38845,"num_object_clones":0,"num_object_copies":116535,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8478,"num_objects_misplaced":16234,"num_objects_unfound":0,"num_objects_dirty":38845,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179281,"num_write_kb":6979969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35834,"num_bytes_recovered":2320629760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10299912192,"data_stored":10299899976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160966,"ondisk_log_size":160966,"up":48,"acting":41,"num_store_stats":10},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057019,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1949520,"kb_used_data":1372744,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":91795632,"statfs":{"total":95995035648,"available":93998727168,"internally_reserved":0,"allocated":1405689856,"data_stored":1405209875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":590608838},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351359,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2153556,"kb_used_data":1490696,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":91591596,"statfs":{"total":95995035648,"available":93789794304,"internally_reserved":0,"allocated":1526472704,"data_stored":1525992723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":678754752},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514884,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1713092,"kb_used_data":928060,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":92032060,"statfs":{"total":95995035648,"available":94240829440,"internally_reserved":0,"allocated":950333440,"data_stored":949857688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":803862976},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874634,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":864736,"kb_used_data":458708,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92880416,"statfs":{"total":95995035648,"available":95109545984,"internally_reserved":0,"allocated":469716992,"data_stored":469232782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":240518168973,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430613,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":849120,"kb_used_data":186260,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":92896032,"statfs":{"total":95995035648,"available":95125536768,"internally_reserved":0,"allocated":190730240,"data_stored":190246030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":678754758},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575277,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2128408,"kb_used_data":1307276,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":91616744,"statfs":{"total":95995035648,"available":93815545856,"internally_reserved":0,"allocated":1338650624,"data_stored":1338166571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":840825285},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,0,2,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738805,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1682644,"kb_used_data":939276,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":92062508,"statfs":{"total":95995035648,"available":94272008192,"internally_reserved":0,"allocated":961818624,"data_stored":961334571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,0,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411754,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1755484,"kb_used_data":1214800,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":91989668,"statfs":{"total":95995035648,"available":94197420032,"internally_reserved":0,"allocated":1243955200,"data_stored":1243471667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150115,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1958748,"kb_used_data":1084884,"kb_used_omap":1,"kb_used_meta":873854,"kb_avail":91786404,"statfs":{"total":95995035648,"available":93989277696,"internally_reserved":0,"allocated":1110921216,"data_stored":1110433611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":894826951},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986592,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692250,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1763992,"kb_used_data":1128912,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":91981160,"statfs":{"total":95995035648,"available":94188707840,"internally_reserved":0,"allocated":1156005888,"data_stored":1155522355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":956829696,"data_stored":956825624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1331564544,"data_stored":1331560472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1233780736,"data_stored":1233780736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1108021248,"data_stored":1108017176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1153040384,"data_stored":1153040384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":188809216,"data_stored":188809216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":465567744,"data_stored":465567744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":946012160,"data_stored":946012160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1523777536,"data_stored":1523777536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1392508928,"data_stored":1392508928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:17:05.467 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph status --format=json 2024-05-15T13:17:05.748 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:05 smithi177 ceph-mon[44303]: pgmap v1824: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 521 op/s; 8225/109674 objects degraded (7.499%); 15250/109674 objects misplaced (13.905%); 18 MiB/s, 329 objects/s recovering 2024-05-15T13:17:05.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:05 smithi027 ceph-mon[39795]: pgmap v1824: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 521 op/s; 8225/109674 objects degraded (7.499%); 15250/109674 objects misplaced (13.905%); 18 MiB/s, 329 objects/s recovering 2024-05-15T13:17:05.803 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:17:05.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:05.808 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:17:05.873 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:17:05.909 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:17:05.933 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:05 smithi120 ceph-mon[43961]: pgmap v1824: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 521 op/s; 8225/109674 objects degraded (7.499%); 15250/109674 objects misplaced (13.905%); 18 MiB/s, 329 objects/s recovering 2024-05-15T13:17:05.935 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:05.955 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:17:05.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:05.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:17:06.059 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:17:06.087 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:06.109 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:17:06.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:06.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:17:06.213 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:17:06.240 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:06.259 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:17:06.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:06.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:17:06.363 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:17:06.390 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:06.410 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:17:06.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:06.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:17:06.443 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:17:06.444 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-05-15T13:17:06.444 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-05-15T13:17:06.444 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T13:17:06.444 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.444 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:3261 7.a 7:503f7f7f:::benchmark_data_smithi177_134484_object3260:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-05-15T13:17:06.444 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:12:43.968609+0000", 2024-05-15T13:17:06.444 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 262.476458675, 2024-05-15T13:17:06.444 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20631449099999999, 2024-05-15T13:17:06.444 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:06.444 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:06.444 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:06.444 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:06.445 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:06.445 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 3261 2024-05-15T13:17:06.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.445 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:06.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:06.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.968609+0000", 2024-05-15T13:17:06.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:06.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.968609+0000", 2024-05-15T13:17:06.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:06.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.968610+0000", 2024-05-15T13:17:06.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4699999999999999e-06 2024-05-15T13:17:06.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:06.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.968635+0000", 2024-05-15T13:17:06.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4726999999999999e-05 2024-05-15T13:17:06.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:06.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.968649+0000", 2024-05-15T13:17:06.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.393e-05 2024-05-15T13:17:06.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:06.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.968657+0000", 2024-05-15T13:17:06.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5369999999999997e-06 2024-05-15T13:17:06.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:06.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.968744+0000", 2024-05-15T13:17:06.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6853000000000003e-05 2024-05-15T13:17:06.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.448 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:06.448 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.968931+0000", 2024-05-15T13:17:06.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018700100000000001 2024-05-15T13:17:06.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.448 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,11", 2024-05-15T13:17:06.448 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.969087+0000", 2024-05-15T13:17:06.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015548999999999999 2024-05-15T13:17:06.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.448 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:06.448 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.969791+0000", 2024-05-15T13:17:06.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00070438 2024-05-15T13:17:06.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.970461+0000", 2024-05-15T13:17:06.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000669668 2024-05-15T13:17:06.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.174845+0000", 2024-05-15T13:17:06.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.204383913 2024-05-15T13:17:06.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:06.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.174890+0000", 2024-05-15T13:17:06.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5713000000000001e-05 2024-05-15T13:17:06.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:06.450 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.174923+0000", 2024-05-15T13:17:06.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2808999999999997e-05 2024-05-15T13:17:06.450 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.450 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:06.450 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.450 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:3279 7.a 7:5e3a7c61:::benchmark_data_smithi177_134484_object3278:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-05-15T13:17:06.450 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:12:43.972885+0000", 2024-05-15T13:17:06.450 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 262.472182433, 2024-05-15T13:17:06.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.204547545, 2024-05-15T13:17:06.451 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:06.451 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:06.451 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:06.451 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:06.451 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:06.451 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 3279 2024-05-15T13:17:06.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.451 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:06.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.451 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:06.451 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.972885+0000", 2024-05-15T13:17:06.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:06.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.972885+0000", 2024-05-15T13:17:06.452 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:06.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.972886+0000", 2024-05-15T13:17:06.452 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.621e-06 2024-05-15T13:17:06.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:06.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.972909+0000", 2024-05-15T13:17:06.452 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2696000000000001e-05 2024-05-15T13:17:06.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.453 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.453 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:06.453 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.972922+0000", 2024-05-15T13:17:06.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3056e-05 2024-05-15T13:17:06.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.453 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.453 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:06.453 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.972931+0000", 2024-05-15T13:17:06.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4460000000000001e-06 2024-05-15T13:17:06.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.453 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.453 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:06.453 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.972992+0000", 2024-05-15T13:17:06.454 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1451000000000005e-05 2024-05-15T13:17:06.454 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.454 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.454 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:06.454 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.973171+0000", 2024-05-15T13:17:06.454 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017866099999999999 2024-05-15T13:17:06.454 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.454 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.454 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,11", 2024-05-15T13:17:06.454 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.973333+0000", 2024-05-15T13:17:06.454 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016246899999999999 2024-05-15T13:17:06.454 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.454 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.454 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:06.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.974660+0000", 2024-05-15T13:17:06.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001326841 2024-05-15T13:17:06.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.455 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.974990+0000", 2024-05-15T13:17:06.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000329742 2024-05-15T13:17:06.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.455 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.177330+0000", 2024-05-15T13:17:06.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20233974599999999 2024-05-15T13:17:06.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:06.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.177386+0000", 2024-05-15T13:17:06.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6563e-05 2024-05-15T13:17:06.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:06.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.177432+0000", 2024-05-15T13:17:06.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6252999999999999e-05 2024-05-15T13:17:06.456 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.456 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:06.456 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.456 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:3283 7.a 7:50243748:::benchmark_data_smithi177_134484_object3282:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-05-15T13:17:06.457 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:12:43.973580+0000", 2024-05-15T13:17:06.457 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 262.47148754800003, 2024-05-15T13:17:06.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.204022496, 2024-05-15T13:17:06.457 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:06.457 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:06.457 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:06.457 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:06.457 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:06.457 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 3283 2024-05-15T13:17:06.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.457 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:06.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.457 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:06.457 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.973580+0000", 2024-05-15T13:17:06.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:06.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.973580+0000", 2024-05-15T13:17:06.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:06.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.973580+0000", 2024-05-15T13:17:06.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9599999999999999e-07 2024-05-15T13:17:06.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:06.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.973594+0000", 2024-05-15T13:17:06.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3893999999999999e-05 2024-05-15T13:17:06.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:06.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.973607+0000", 2024-05-15T13:17:06.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2860999999999999e-05 2024-05-15T13:17:06.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:06.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.973615+0000", 2024-05-15T13:17:06.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1110000000000008e-06 2024-05-15T13:17:06.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:06.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.973763+0000", 2024-05-15T13:17:06.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014752600000000001 2024-05-15T13:17:06.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:06.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.973884+0000", 2024-05-15T13:17:06.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000121444 2024-05-15T13:17:06.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,11", 2024-05-15T13:17:06.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.974027+0000", 2024-05-15T13:17:06.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000143165 2024-05-15T13:17:06.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.975040+0000", 2024-05-15T13:17:06.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010125690000000001 2024-05-15T13:17:06.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:06.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.975268+0000", 2024-05-15T13:17:06.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000228562 2024-05-15T13:17:06.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.177496+0000", 2024-05-15T13:17:06.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20222785700000001 2024-05-15T13:17:06.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:06.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.177556+0000", 2024-05-15T13:17:06.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9670000000000003e-05 2024-05-15T13:17:06.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:06.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.177602+0000", 2024-05-15T13:17:06.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6140999999999998e-05 2024-05-15T13:17:06.462 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.462 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:06.462 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.463 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:3309 7.a 7:56b64e3f:::benchmark_data_smithi177_134484_object3308:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-05-15T13:17:06.463 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:12:43.982912+0000", 2024-05-15T13:17:06.463 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 262.46215535900001, 2024-05-15T13:17:06.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.194812401, 2024-05-15T13:17:06.463 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:06.463 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:06.463 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:06.463 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:06.463 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:06.463 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 3309 2024-05-15T13:17:06.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.463 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:06.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:06.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.982912+0000", 2024-05-15T13:17:06.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:06.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.982912+0000", 2024-05-15T13:17:06.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:06.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.982913+0000", 2024-05-15T13:17:06.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.22e-06 2024-05-15T13:17:06.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:06.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.982936+0000", 2024-05-15T13:17:06.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3136e-05 2024-05-15T13:17:06.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:06.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.982951+0000", 2024-05-15T13:17:06.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4866e-05 2024-05-15T13:17:06.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:06.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.982960+0000", 2024-05-15T13:17:06.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7679999999999997e-06 2024-05-15T13:17:06.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:06.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.983011+0000", 2024-05-15T13:17:06.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1124e-05 2024-05-15T13:17:06.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.466 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:06.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.983191+0000", 2024-05-15T13:17:06.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000180025 2024-05-15T13:17:06.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.466 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,11", 2024-05-15T13:17:06.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.983345+0000", 2024-05-15T13:17:06.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015431700000000001 2024-05-15T13:17:06.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.466 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:06.467 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.983852+0000", 2024-05-15T13:17:06.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00050665300000000003 2024-05-15T13:17:06.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.467 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.467 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.984057+0000", 2024-05-15T13:17:06.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020497399999999999 2024-05-15T13:17:06.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.467 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.467 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.177645+0000", 2024-05-15T13:17:06.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19358819999999999 2024-05-15T13:17:06.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:06.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.177688+0000", 2024-05-15T13:17:06.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2762000000000002e-05 2024-05-15T13:17:06.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:06.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.177724+0000", 2024-05-15T13:17:06.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6356000000000002e-05 2024-05-15T13:17:06.468 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.468 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:06.468 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.468 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:3320 7.a 7:53147a9d:::benchmark_data_smithi177_134484_object3319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-05-15T13:17:06.469 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:12:43.987658+0000", 2024-05-15T13:17:06.469 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 262.457409375, 2024-05-15T13:17:06.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.190206495, 2024-05-15T13:17:06.469 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:06.469 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:06.469 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:06.469 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:06.469 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:06.469 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 3320 2024-05-15T13:17:06.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.469 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:06.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:06.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.987658+0000", 2024-05-15T13:17:06.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:06.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.987658+0000", 2024-05-15T13:17:06.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:06.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.987659+0000", 2024-05-15T13:17:06.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4249999999999999e-06 2024-05-15T13:17:06.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:06.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.987682+0000", 2024-05-15T13:17:06.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2541999999999999e-05 2024-05-15T13:17:06.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:06.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.987692+0000", 2024-05-15T13:17:06.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0145e-05 2024-05-15T13:17:06.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:06.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.987699+0000", 2024-05-15T13:17:06.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4830000000000002e-06 2024-05-15T13:17:06.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:06.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.987754+0000", 2024-05-15T13:17:06.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4975000000000001e-05 2024-05-15T13:17:06.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.472 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:06.472 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.987889+0000", 2024-05-15T13:17:06.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013494900000000001 2024-05-15T13:17:06.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.472 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,11", 2024-05-15T13:17:06.472 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.988034+0000", 2024-05-15T13:17:06.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014455800000000001 2024-05-15T13:17:06.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.472 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:06.472 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.988638+0000", 2024-05-15T13:17:06.473 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00060430499999999997 2024-05-15T13:17:06.473 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.473 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.473 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.989245+0000", 2024-05-15T13:17:06.473 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00060691300000000005 2024-05-15T13:17:06.473 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.473 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.473 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.177774+0000", 2024-05-15T13:17:06.473 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18852857200000001 2024-05-15T13:17:06.473 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.473 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:06.473 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.177827+0000", 2024-05-15T13:17:06.474 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3331999999999997e-05 2024-05-15T13:17:06.474 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.474 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.474 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:06.474 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.177864+0000", 2024-05-15T13:17:06.474 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7295999999999997e-05 2024-05-15T13:17:06.474 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.474 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:06.474 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.474 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.474 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.474 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:3343 7.a 7:584225b1:::benchmark_data_smithi177_134484_object3342:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-05-15T13:17:06.474 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:12:43.996884+0000", 2024-05-15T13:17:06.474 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 262.44818281400001, 2024-05-15T13:17:06.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18111996, 2024-05-15T13:17:06.475 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:06.475 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:06.475 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:06.475 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:06.475 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:06.475 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 3343 2024-05-15T13:17:06.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.475 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:06.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:06.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.996884+0000", 2024-05-15T13:17:06.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:06.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.996884+0000", 2024-05-15T13:17:06.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:06.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.996886+0000", 2024-05-15T13:17:06.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.331e-06 2024-05-15T13:17:06.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:06.477 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.996907+0000", 2024-05-15T13:17:06.477 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1526999999999999e-05 2024-05-15T13:17:06.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.477 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:06.477 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.996922+0000", 2024-05-15T13:17:06.477 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4902999999999999e-05 2024-05-15T13:17:06.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.477 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:06.477 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.996930+0000", 2024-05-15T13:17:06.477 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8059999999999995e-06 2024-05-15T13:17:06.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.478 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.478 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:06.478 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.996992+0000", 2024-05-15T13:17:06.478 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1791000000000005e-05 2024-05-15T13:17:06.478 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.478 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.478 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:06.478 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.997454+0000", 2024-05-15T13:17:06.478 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00046179800000000003 2024-05-15T13:17:06.478 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.478 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.478 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,11", 2024-05-15T13:17:06.478 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.997593+0000", 2024-05-15T13:17:06.478 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013967299999999999 2024-05-15T13:17:06.479 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.479 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.479 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:06.479 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.998081+0000", 2024-05-15T13:17:06.479 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00048803099999999999 2024-05-15T13:17:06.479 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.479 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.479 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.479 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.998422+0000", 2024-05-15T13:17:06.479 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00034115000000000003 2024-05-15T13:17:06.479 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.479 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.479 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.479 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.177919+0000", 2024-05-15T13:17:06.479 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.179496356 2024-05-15T13:17:06.480 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.480 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.480 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:06.480 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.177968+0000", 2024-05-15T13:17:06.480 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9369000000000001e-05 2024-05-15T13:17:06.480 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.480 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.480 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:06.480 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.178004+0000", 2024-05-15T13:17:06.480 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6225e-05 2024-05-15T13:17:06.480 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.480 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:06.480 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.480 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.481 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:3348 7.a 7:5acf1600:::benchmark_data_smithi177_134484_object3347:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-05-15T13:17:06.481 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:12:43.998827+0000", 2024-05-15T13:17:06.481 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 262.44624039399997, 2024-05-15T13:17:06.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17930471000000001, 2024-05-15T13:17:06.481 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:06.481 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:06.481 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:06.481 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:06.481 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:06.481 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 3348 2024-05-15T13:17:06.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.481 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:06.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:06.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.998827+0000", 2024-05-15T13:17:06.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:06.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.998827+0000", 2024-05-15T13:17:06.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:06.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.998828+0000", 2024-05-15T13:17:06.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2500000000000004e-07 2024-05-15T13:17:06.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.483 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:06.483 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.998845+0000", 2024-05-15T13:17:06.483 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7448e-05 2024-05-15T13:17:06.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.483 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:06.483 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.998857+0000", 2024-05-15T13:17:06.483 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1445e-05 2024-05-15T13:17:06.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.483 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:06.483 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.998862+0000", 2024-05-15T13:17:06.484 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6799999999999998e-06 2024-05-15T13:17:06.484 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.484 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.484 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:06.484 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.998912+0000", 2024-05-15T13:17:06.484 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9990000000000001e-05 2024-05-15T13:17:06.484 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.484 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.484 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:06.484 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.999005+0000", 2024-05-15T13:17:06.484 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3016999999999994e-05 2024-05-15T13:17:06.484 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.484 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.484 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,11", 2024-05-15T13:17:06.485 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.999092+0000", 2024-05-15T13:17:06.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7142e-05 2024-05-15T13:17:06.485 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.485 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:06.485 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.999486+0000", 2024-05-15T13:17:06.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00039321100000000003 2024-05-15T13:17:06.485 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.485 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.485 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:43.999797+0000", 2024-05-15T13:17:06.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000311711 2024-05-15T13:17:06.485 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.485 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.178044+0000", 2024-05-15T13:17:06.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17824627800000001 2024-05-15T13:17:06.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:06.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.178096+0000", 2024-05-15T13:17:06.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2142999999999999e-05 2024-05-15T13:17:06.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:06.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.178132+0000", 2024-05-15T13:17:06.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5719999999999997e-05 2024-05-15T13:17:06.486 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.487 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:06.487 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.487 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:3354 7.a 7:5bd7c7ef:::benchmark_data_smithi177_134484_object3353:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-05-15T13:17:06.487 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:12:44.002227+0000", 2024-05-15T13:17:06.487 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 262.44284059500001, 2024-05-15T13:17:06.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17603614300000001, 2024-05-15T13:17:06.487 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:06.487 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:06.487 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:06.487 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:06.487 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:06.487 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 3354 2024-05-15T13:17:06.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.488 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:06.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.488 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:06.488 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.002227+0000", 2024-05-15T13:17:06.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.488 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.488 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:06.488 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.002227+0000", 2024-05-15T13:17:06.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.488 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.488 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:06.488 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.002228+0000", 2024-05-15T13:17:06.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0690000000000001e-06 2024-05-15T13:17:06.488 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:06.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.002247+0000", 2024-05-15T13:17:06.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8953e-05 2024-05-15T13:17:06.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:06.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.002259+0000", 2024-05-15T13:17:06.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2558000000000001e-05 2024-05-15T13:17:06.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:06.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.002266+0000", 2024-05-15T13:17:06.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7170000000000002e-06 2024-05-15T13:17:06.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:06.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.002321+0000", 2024-05-15T13:17:06.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4768000000000001e-05 2024-05-15T13:17:06.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:06.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.002455+0000", 2024-05-15T13:17:06.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013411 2024-05-15T13:17:06.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,11", 2024-05-15T13:17:06.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.002606+0000", 2024-05-15T13:17:06.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000151075 2024-05-15T13:17:06.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:06.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.003020+0000", 2024-05-15T13:17:06.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00041459999999999999 2024-05-15T13:17:06.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.003313+0000", 2024-05-15T13:17:06.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00029278099999999999 2024-05-15T13:17:06.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.492 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.492 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.178174+0000", 2024-05-15T13:17:06.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17486027600000001 2024-05-15T13:17:06.492 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.492 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:06.492 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.178229+0000", 2024-05-15T13:17:06.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5588999999999999e-05 2024-05-15T13:17:06.492 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.492 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:06.492 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:12:44.178263+0000", 2024-05-15T13:17:06.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3646999999999998e-05 2024-05-15T13:17:06.492 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.493 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:06.493 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.493 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.493 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.493 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:43854 7.a 7:513e1fdf:::benchmark_data_smithi177_134484_object43853:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e611)", 2024-05-15T13:17:06.493 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:13:03.240470+0000", 2024-05-15T13:17:06.493 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 243.20459770799999, 2024-05-15T13:17:06.493 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11072269, 2024-05-15T13:17:06.493 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:06.493 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:06.493 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:06.493 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:06.493 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:06.493 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 43854 2024-05-15T13:17:06.493 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.493 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:06.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:06.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:13:03.240470+0000", 2024-05-15T13:17:06.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:06.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:13:03.240470+0000", 2024-05-15T13:17:06.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:06.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:13:03.240471+0000", 2024-05-15T13:17:06.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8279999999999999e-06 2024-05-15T13:17:06.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:06.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:13:03.240493+0000", 2024-05-15T13:17:06.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1600999999999999e-05 2024-05-15T13:17:06.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:06.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:13:03.240511+0000", 2024-05-15T13:17:06.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8059000000000001e-05 2024-05-15T13:17:06.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:06.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:13:03.240520+0000", 2024-05-15T13:17:06.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0110000000000006e-06 2024-05-15T13:17:06.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:06.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:13:03.240593+0000", 2024-05-15T13:17:06.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3302999999999999e-05 2024-05-15T13:17:06.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:06.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:13:03.240733+0000", 2024-05-15T13:17:06.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000139577 2024-05-15T13:17:06.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,11", 2024-05-15T13:17:06.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:13:03.240947+0000", 2024-05-15T13:17:06.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000214324 2024-05-15T13:17:06.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:13:03.244418+0000", 2024-05-15T13:17:06.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0034704879999999999 2024-05-15T13:17:06.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:06.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:13:03.246112+0000", 2024-05-15T13:17:06.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001694463 2024-05-15T13:17:06.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:13:03.351075+0000", 2024-05-15T13:17:06.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.104963195 2024-05-15T13:17:06.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:06.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:13:03.351140+0000", 2024-05-15T13:17:06.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4481999999999997e-05 2024-05-15T13:17:06.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.498 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:06.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:13:03.351192+0000", 2024-05-15T13:17:06.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2358999999999998e-05 2024-05-15T13:17:06.498 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.498 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:06.498 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.498 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:107728 7.a 7:5e9b1d65:::benchmark_data_smithi177_134484_object107727:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e645)", 2024-05-15T13:17:06.499 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:14:07.078702+0000", 2024-05-15T13:17:06.499 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 179.366365118, 2024-05-15T13:17:06.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.143727349, 2024-05-15T13:17:06.499 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:06.499 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:06.499 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:06.499 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:06.499 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:06.499 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 107728 2024-05-15T13:17:06.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.499 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:06.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:06.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:07.078702+0000", 2024-05-15T13:17:06.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:06.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:07.078702+0000", 2024-05-15T13:17:06.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:06.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:07.078704+0000", 2024-05-15T13:17:06.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.781e-06 2024-05-15T13:17:06.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:06.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:07.078731+0000", 2024-05-15T13:17:06.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6967000000000001e-05 2024-05-15T13:17:06.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:06.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:07.078749+0000", 2024-05-15T13:17:06.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8482e-05 2024-05-15T13:17:06.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:06.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:07.078759+0000", 2024-05-15T13:17:06.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4369999999999996e-06 2024-05-15T13:17:06.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:06.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:07.078824+0000", 2024-05-15T13:17:06.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5186999999999998e-05 2024-05-15T13:17:06.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:06.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:07.079012+0000", 2024-05-15T13:17:06.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018823599999999999 2024-05-15T13:17:06.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,11", 2024-05-15T13:17:06.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:07.079163+0000", 2024-05-15T13:17:06.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015054900000000001 2024-05-15T13:17:06.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:06.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:07.082790+0000", 2024-05-15T13:17:06.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0036275890000000001 2024-05-15T13:17:06.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:07.083057+0000", 2024-05-15T13:17:06.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026703600000000001 2024-05-15T13:17:06.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:07.222310+0000", 2024-05-15T13:17:06.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13925221700000001 2024-05-15T13:17:06.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:06.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:07.222376+0000", 2024-05-15T13:17:06.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6161999999999994e-05 2024-05-15T13:17:06.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:06.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:07.222429+0000", 2024-05-15T13:17:06.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3705999999999997e-05 2024-05-15T13:17:06.504 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.504 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:06.504 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.505 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:143291 7.a 7:5365d091:::benchmark_data_smithi177_134484_object34226:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e672)", 2024-05-15T13:17:06.505 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:33.117878+0000", 2024-05-15T13:17:06.505 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 93.327189371000003, 2024-05-15T13:17:06.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21058098, 2024-05-15T13:17:06.505 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:06.505 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:06.505 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:06.505 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:06.505 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:06.505 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 143291 2024-05-15T13:17:06.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.505 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:06.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:06.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.117878+0000", 2024-05-15T13:17:06.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.506 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:06.506 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.117878+0000", 2024-05-15T13:17:06.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.506 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:06.506 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.117880+0000", 2024-05-15T13:17:06.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.702e-06 2024-05-15T13:17:06.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.506 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:06.506 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.117887+0000", 2024-05-15T13:17:06.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6070000000000002e-06 2024-05-15T13:17:06.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:06.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.117902+0000", 2024-05-15T13:17:06.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5211999999999999e-05 2024-05-15T13:17:06.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:06.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.117911+0000", 2024-05-15T13:17:06.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9900000000000003e-06 2024-05-15T13:17:06.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:06.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.117970+0000", 2024-05-15T13:17:06.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8576999999999998e-05 2024-05-15T13:17:06.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:06.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.118102+0000", 2024-05-15T13:17:06.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000132046 2024-05-15T13:17:06.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,11", 2024-05-15T13:17:06.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.118223+0000", 2024-05-15T13:17:06.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120726 2024-05-15T13:17:06.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:06.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.121922+0000", 2024-05-15T13:17:06.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003699529 2024-05-15T13:17:06.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.130040+0000", 2024-05-15T13:17:06.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0081178489999999999 2024-05-15T13:17:06.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.328385+0000", 2024-05-15T13:17:06.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19834512300000001 2024-05-15T13:17:06.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:06.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.328423+0000", 2024-05-15T13:17:06.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8148999999999997e-05 2024-05-15T13:17:06.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:06.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.328459+0000", 2024-05-15T13:17:06.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5469999999999998e-05 2024-05-15T13:17:06.510 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.510 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:06.510 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.510 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:143293 7.a 7:59b1bca6:::benchmark_data_smithi177_134484_object34228:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e672)", 2024-05-15T13:17:06.510 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:33.123026+0000", 2024-05-15T13:17:06.510 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 93.322040939999994, 2024-05-15T13:17:06.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21120455499999999, 2024-05-15T13:17:06.510 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:06.510 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:06.511 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:06.511 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:06.511 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:06.511 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 143293 2024-05-15T13:17:06.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.511 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:06.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:06.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.123026+0000", 2024-05-15T13:17:06.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:06.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.123026+0000", 2024-05-15T13:17:06.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.512 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:06.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.123040+0000", 2024-05-15T13:17:06.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4173e-05 2024-05-15T13:17:06.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.512 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:06.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.123047+0000", 2024-05-15T13:17:06.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.229e-06 2024-05-15T13:17:06.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.512 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:06.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.123062+0000", 2024-05-15T13:17:06.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5783000000000001e-05 2024-05-15T13:17:06.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:06.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.123071+0000", 2024-05-15T13:17:06.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7479999999999999e-06 2024-05-15T13:17:06.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:06.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.123129+0000", 2024-05-15T13:17:06.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7324999999999997e-05 2024-05-15T13:17:06.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:06.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.123258+0000", 2024-05-15T13:17:06.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012974199999999999 2024-05-15T13:17:06.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,11", 2024-05-15T13:17:06.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.123379+0000", 2024-05-15T13:17:06.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120835 2024-05-15T13:17:06.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:06.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.127696+0000", 2024-05-15T13:17:06.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0043172130000000003 2024-05-15T13:17:06.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.130347+0000", 2024-05-15T13:17:06.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0026510650000000002 2024-05-15T13:17:06.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.334144+0000", 2024-05-15T13:17:06.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20379657400000001 2024-05-15T13:17:06.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:06.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.334197+0000", 2024-05-15T13:17:06.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3368000000000001e-05 2024-05-15T13:17:06.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:06.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.334231+0000", 2024-05-15T13:17:06.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3500000000000001e-05 2024-05-15T13:17:06.515 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.515 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:06.516 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.516 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:143310 7.a 7:5d34e7c3:::benchmark_data_smithi177_134484_object34245:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e672)", 2024-05-15T13:17:06.516 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:33.169435+0000", 2024-05-15T13:17:06.516 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 93.275632341999994, 2024-05-15T13:17:06.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16491346800000001, 2024-05-15T13:17:06.516 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:06.516 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:06.516 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:06.516 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:06.516 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:06.516 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 143310 2024-05-15T13:17:06.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.517 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:06.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:06.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.169435+0000", 2024-05-15T13:17:06.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:06.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.169435+0000", 2024-05-15T13:17:06.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:06.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.169436+0000", 2024-05-15T13:17:06.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5400000000000001e-06 2024-05-15T13:17:06.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:06.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.169443+0000", 2024-05-15T13:17:06.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4239999999999998e-06 2024-05-15T13:17:06.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:06.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.169457+0000", 2024-05-15T13:17:06.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3916e-05 2024-05-15T13:17:06.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:06.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.169465+0000", 2024-05-15T13:17:06.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7829999999999999e-06 2024-05-15T13:17:06.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:06.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.169522+0000", 2024-05-15T13:17:06.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7908000000000002e-05 2024-05-15T13:17:06.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:06.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.169668+0000", 2024-05-15T13:17:06.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000145131 2024-05-15T13:17:06.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,11", 2024-05-15T13:17:06.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.169794+0000", 2024-05-15T13:17:06.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000125957 2024-05-15T13:17:06.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:06.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.173870+0000", 2024-05-15T13:17:06.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0040766600000000002 2024-05-15T13:17:06.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.174478+0000", 2024-05-15T13:17:06.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00060747899999999996 2024-05-15T13:17:06.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.334270+0000", 2024-05-15T13:17:06.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.15979273699999999 2024-05-15T13:17:06.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:06.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.334305+0000", 2024-05-15T13:17:06.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4659000000000001e-05 2024-05-15T13:17:06.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:06.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.334348+0000", 2024-05-15T13:17:06.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3273999999999999e-05 2024-05-15T13:17:06.521 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.522 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:06.522 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.522 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:143326 7.a 7:52d9af0a:::benchmark_data_smithi177_134484_object34261:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e672)", 2024-05-15T13:17:06.522 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:33.216605+0000", 2024-05-15T13:17:06.522 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 93.228462292000003, 2024-05-15T13:17:06.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11787494499999999, 2024-05-15T13:17:06.522 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:06.522 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:06.522 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:06.522 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:06.522 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:06.522 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 143326 2024-05-15T13:17:06.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.523 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:06.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:06.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.216605+0000", 2024-05-15T13:17:06.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:06.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.216605+0000", 2024-05-15T13:17:06.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:06.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.216607+0000", 2024-05-15T13:17:06.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8309999999999999e-06 2024-05-15T13:17:06.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:06.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.216617+0000", 2024-05-15T13:17:06.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0538e-05 2024-05-15T13:17:06.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:06.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.216637+0000", 2024-05-15T13:17:06.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9962999999999998e-05 2024-05-15T13:17:06.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:06.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.216649+0000", 2024-05-15T13:17:06.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1294000000000001e-05 2024-05-15T13:17:06.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:06.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.216721+0000", 2024-05-15T13:17:06.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2811999999999993e-05 2024-05-15T13:17:06.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:06.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.216856+0000", 2024-05-15T13:17:06.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013511900000000001 2024-05-15T13:17:06.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,11", 2024-05-15T13:17:06.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.216935+0000", 2024-05-15T13:17:06.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8091999999999997e-05 2024-05-15T13:17:06.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:06.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.220524+0000", 2024-05-15T13:17:06.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035893219999999998 2024-05-15T13:17:06.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.224498+0000", 2024-05-15T13:17:06.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0039738500000000001 2024-05-15T13:17:06.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.334403+0000", 2024-05-15T13:17:06.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.10990514999999999 2024-05-15T13:17:06.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.527 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:06.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.334444+0000", 2024-05-15T13:17:06.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1458000000000002e-05 2024-05-15T13:17:06.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.527 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:06.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.334480+0000", 2024-05-15T13:17:06.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5515999999999997e-05 2024-05-15T13:17:06.527 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.527 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:06.527 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.528 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:178472 7.a 7:58791292:::benchmark_data_smithi177_134484_object69407:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e700)", 2024-05-15T13:17:06.528 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:16:55.742739+0000", 2024-05-15T13:17:06.528 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.702328501, 2024-05-15T13:17:06.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11233568200000001, 2024-05-15T13:17:06.528 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:06.528 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:06.528 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:06.528 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:06.528 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:06.528 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 178472 2024-05-15T13:17:06.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.528 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:06.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:06.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.742739+0000", 2024-05-15T13:17:06.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:06.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.742739+0000", 2024-05-15T13:17:06.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:06.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.742740+0000", 2024-05-15T13:17:06.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6390000000000001e-06 2024-05-15T13:17:06.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:06.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.742747+0000", 2024-05-15T13:17:06.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1280000000000002e-06 2024-05-15T13:17:06.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:06.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.742761+0000", 2024-05-15T13:17:06.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3519e-05 2024-05-15T13:17:06.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:06.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.742770+0000", 2024-05-15T13:17:06.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.602e-06 2024-05-15T13:17:06.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:06.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.742829+0000", 2024-05-15T13:17:06.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9086000000000003e-05 2024-05-15T13:17:06.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:06.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.742922+0000", 2024-05-15T13:17:06.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3349000000000004e-05 2024-05-15T13:17:06.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,11", 2024-05-15T13:17:06.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.743000+0000", 2024-05-15T13:17:06.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7626000000000005e-05 2024-05-15T13:17:06.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.532 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.532 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.854427+0000", 2024-05-15T13:17:06.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11142741 2024-05-15T13:17:06.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.532 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:06.532 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.854440+0000", 2024-05-15T13:17:06.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2857000000000001e-05 2024-05-15T13:17:06.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.532 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.532 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.854975+0000", 2024-05-15T13:17:06.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00053462799999999999 2024-05-15T13:17:06.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:06.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.855023+0000", 2024-05-15T13:17:06.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8909000000000002e-05 2024-05-15T13:17:06.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:06.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.855074+0000", 2024-05-15T13:17:06.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0929e-05 2024-05-15T13:17:06.533 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.533 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:06.533 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.534 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.26044.0:178475 7.0 e700/599 7:0980e1c4:::benchmark_data_smithi177_134484_object69410:head v 700'11170, mlcod=700'11168)", 2024-05-15T13:17:06.534 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:16:55.743101+0000", 2024-05-15T13:17:06.534 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.701966561000001, 2024-05-15T13:17:06.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.111321397, 2024-05-15T13:17:06.534 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:06.534 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:06.534 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:06.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:06.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.743101+0000", 2024-05-15T13:17:06.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:06.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.743101+0000", 2024-05-15T13:17:06.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:06.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.743101+0000", 2024-05-15T13:17:06.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2699999999999999e-07 2024-05-15T13:17:06.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:06.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.743107+0000", 2024-05-15T13:17:06.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9490000000000001e-06 2024-05-15T13:17:06.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:06.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.743116+0000", 2024-05-15T13:17:06.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9409999999999996e-06 2024-05-15T13:17:06.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:06.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.743122+0000", 2024-05-15T13:17:06.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9290000000000003e-06 2024-05-15T13:17:06.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:06.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.745521+0000", 2024-05-15T13:17:06.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023986519999999998 2024-05-15T13:17:06.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:06.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.745541+0000", 2024-05-15T13:17:06.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0171000000000001e-05 2024-05-15T13:17:06.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:06.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.854385+0000", 2024-05-15T13:17:06.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.108844342 2024-05-15T13:17:06.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:06.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.854422+0000", 2024-05-15T13:17:06.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6786000000000003e-05 2024-05-15T13:17:06.538 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.538 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:06.538 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.538 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPush(7.5 700/676 [PushOp(7:a5436f65:::benchmark_data_smithi177_134484_object104138:head, version: 643'6538, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:a5436f65:::benchmark_data_smithi177_134484_object104138:head@643'6538, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-05-15T13:17:06.538 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:16:55.744202+0000", 2024-05-15T13:17:06.538 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.700865411000001, 2024-05-15T13:17:06.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11013100000000001, 2024-05-15T13:17:06.538 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:06.538 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T13:17:06.538 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:06.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:06.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.744202+0000", 2024-05-15T13:17:06.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:06.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.744202+0000", 2024-05-15T13:17:06.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:06.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.744202+0000", 2024-05-15T13:17:06.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7199999999999999e-07 2024-05-15T13:17:06.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:06.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.744234+0000", 2024-05-15T13:17:06.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1358e-05 2024-05-15T13:17:06.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:06.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.744252+0000", 2024-05-15T13:17:06.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8297000000000001e-05 2024-05-15T13:17:06.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:06.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.744268+0000", 2024-05-15T13:17:06.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5565000000000001e-05 2024-05-15T13:17:06.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:06.541 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.744375+0000", 2024-05-15T13:17:06.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000107471 2024-05-15T13:17:06.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:06.541 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.744388+0000", 2024-05-15T13:17:06.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3084999999999999e-05 2024-05-15T13:17:06.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:06.541 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.854333+0000", 2024-05-15T13:17:06.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.109944652 2024-05-15T13:17:06.541 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.542 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:06.542 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.542 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.26044.0:178476 7.0 e700/599 7:01c5112e:::benchmark_data_smithi177_134484_object69411:head v 700'11171, mlcod=700'11168)", 2024-05-15T13:17:06.542 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:16:55.747333+0000", 2024-05-15T13:17:06.542 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.697733741, 2024-05-15T13:17:06.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.126726387, 2024-05-15T13:17:06.542 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:06.542 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:06.542 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:06.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:06.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.747333+0000", 2024-05-15T13:17:06.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:06.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.747333+0000", 2024-05-15T13:17:06.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:06.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.747334+0000", 2024-05-15T13:17:06.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8899999999999998e-07 2024-05-15T13:17:06.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:06.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.747343+0000", 2024-05-15T13:17:06.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7260000000000008e-06 2024-05-15T13:17:06.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:06.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.854358+0000", 2024-05-15T13:17:06.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.10701437599999999 2024-05-15T13:17:06.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:06.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.854366+0000", 2024-05-15T13:17:06.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1550000000000007e-06 2024-05-15T13:17:06.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:06.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.854490+0000", 2024-05-15T13:17:06.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012472000000000001 2024-05-15T13:17:06.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.545 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:06.545 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.854508+0000", 2024-05-15T13:17:06.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7365999999999999e-05 2024-05-15T13:17:06.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.545 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:06.545 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.873997+0000", 2024-05-15T13:17:06.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.019489197 2024-05-15T13:17:06.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.545 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:06.545 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.874060+0000", 2024-05-15T13:17:06.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2858e-05 2024-05-15T13:17:06.545 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.546 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:06.546 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.546 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.26044.0:178472 7.a e700/599 ondisk, result = 0)", 2024-05-15T13:17:06.546 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:16:55.749477+0000", 2024-05-15T13:17:06.546 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.695590612, 2024-05-15T13:17:06.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.10496461999999999, 2024-05-15T13:17:06.546 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:06.546 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T13:17:06.546 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:06.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:06.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.749477+0000", 2024-05-15T13:17:06.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:06.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.749477+0000", 2024-05-15T13:17:06.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:06.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.749477+0000", 2024-05-15T13:17:06.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3099999999999997e-07 2024-05-15T13:17:06.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:06.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.749483+0000", 2024-05-15T13:17:06.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0719999999999996e-06 2024-05-15T13:17:06.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:06.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.854313+0000", 2024-05-15T13:17:06.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.104829319 2024-05-15T13:17:06.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:06.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.854333+0000", 2024-05-15T13:17:06.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9998000000000001e-05 2024-05-15T13:17:06.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:06.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.854413+0000", 2024-05-15T13:17:06.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0512000000000004e-05 2024-05-15T13:17:06.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:06.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.854421+0000", 2024-05-15T13:17:06.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8299999999999996e-06 2024-05-15T13:17:06.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:06.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:16:55.854441+0000", 2024-05-15T13:17:06.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0157999999999999e-05 2024-05-15T13:17:06.549 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.549 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:06.549 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.550 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:182421 7.a 7:5ec178cb:::benchmark_data_smithi177_134484_object73356:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e702)", 2024-05-15T13:17:06.550 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:17:04.179804+0000", 2024-05-15T13:17:06.550 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 2.2652636479999999, 2024-05-15T13:17:06.550 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.053667167000000002, 2024-05-15T13:17:06.550 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:06.550 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:06.550 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:06.550 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:06.550 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:06.550 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 182421 2024-05-15T13:17:06.550 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.550 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:06.550 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:06.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:17:04.179804+0000", 2024-05-15T13:17:06.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:06.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:17:04.179804+0000", 2024-05-15T13:17:06.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:06.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:06.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:17:04.179804+0000", 2024-05-15T13:17:06.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7100000000000001e-07 2024-05-15T13:17:06.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:06.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:17:04.179809+0000", 2024-05-15T13:17:06.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5170000000000001e-06 2024-05-15T13:17:06.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:06.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:17:04.179817+0000", 2024-05-15T13:17:06.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4230000000000005e-06 2024-05-15T13:17:06.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:06.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:17:04.179825+0000", 2024-05-15T13:17:06.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4599999999999997e-06 2024-05-15T13:17:06.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:06.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:17:04.184529+0000", 2024-05-15T13:17:06.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0047042309999999997 2024-05-15T13:17:06.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:06.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:17:04.184664+0000", 2024-05-15T13:17:06.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000134645 2024-05-15T13:17:06.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,11", 2024-05-15T13:17:06.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:17:04.184787+0000", 2024-05-15T13:17:06.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012370799999999999 2024-05-15T13:17:06.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:06.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:17:04.191637+0000", 2024-05-15T13:17:06.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0068494280000000003 2024-05-15T13:17:06.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.554 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:17:04.205401+0000", 2024-05-15T13:17:06.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0137643 2024-05-15T13:17:06.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.554 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:06.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:17:04.233383+0000", 2024-05-15T13:17:06.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.027982327000000001 2024-05-15T13:17:06.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.554 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:06.555 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:17:04.233431+0000", 2024-05-15T13:17:06.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7845e-05 2024-05-15T13:17:06.555 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:06.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:06.555 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:06.555 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:17:04.233471+0000", 2024-05-15T13:17:06.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9511999999999997e-05 2024-05-15T13:17:06.555 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.555 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:06.555 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.555 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:06.555 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:06.555 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:17:06.556 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:17:06.594 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:06.707 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:17:06.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:06.712 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:17:06.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:06 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:17:06.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:06 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:17:06.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:06 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:17:06.813 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:17:06.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:06 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:17:06.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:06 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:17:06.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:06 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:17:07.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:06 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:17:07.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:06 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:17:07.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:06 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:17:07.054 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:07.268 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-05-15T13:17:07.651 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:17:08.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:07 smithi027 ceph-mon[39795]: pgmap v1825: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 458 op/s; 8225/109212 objects degraded (7.531%); 15250/109212 objects misplaced (13.964%); 15 MiB/s, 288 objects/s recovering 2024-05-15T13:17:08.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:07 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2190861371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:17:08.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:07 smithi120 ceph-mon[43961]: pgmap v1825: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 458 op/s; 8225/109212 objects degraded (7.531%); 15250/109212 objects misplaced (13.964%); 15 MiB/s, 288 objects/s recovering 2024-05-15T13:17:08.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:07 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2190861371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:17:08.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:07 smithi177 ceph-mon[44303]: pgmap v1825: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 458 op/s; 8225/109212 objects degraded (7.531%); 15250/109212 objects misplaced (13.964%); 15 MiB/s, 288 objects/s recovering 2024-05-15T13:17:08.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:07 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2190861371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:17:09.609 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:17:09.609 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:17:09.609 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:17:09.609 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:17:09.609 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:17:09.609 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:17:09.609 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:17:09.609 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:17:09.609 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:17:09.610 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:17:09.610 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:17:09.610 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:17:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:17:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:17:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:17:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:17:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:17:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:17:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:17:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:17:09.611 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:17:09.611 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:17:09.611 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:17:09.611 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:17:09.861 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:17:09.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:09.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:17:09.971 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:17:10.007 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:09 smithi177 ceph-mon[44303]: pgmap v1826: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 459 op/s; 7924/103641 objects degraded (7.646%); 14580/103641 objects misplaced (14.068%); 13 MiB/s, 309 objects/s recovering 2024-05-15T13:17:10.008 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:09 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 7924/103641 objects degraded (7.646%), 5 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:10.011 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:10.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:09 smithi027 ceph-mon[39795]: pgmap v1826: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 459 op/s; 7924/103641 objects degraded (7.646%); 14580/103641 objects misplaced (14.068%); 13 MiB/s, 309 objects/s recovering 2024-05-15T13:17:10.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:09 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 7924/103641 objects degraded (7.646%), 5 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:10.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:09 smithi120 ceph-mon[43961]: pgmap v1826: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 459 op/s; 7924/103641 objects degraded (7.646%); 14580/103641 objects misplaced (14.068%); 13 MiB/s, 309 objects/s recovering 2024-05-15T13:17:10.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:09 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 7924/103641 objects degraded (7.646%), 5 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:10.873 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:17:10.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:10.881 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:17:10.919 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:17:10.982 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:17:11.011 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:11.030 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:17:11.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:11.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:17:11.135 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:17:11.164 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:11.183 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:17:11.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:11.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:17:11.287 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:17:11.391 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:11.453 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:17:11.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:11.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:17:11.500 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:17:11.500 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:17:11.561 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:17:11.602 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:11.744 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:17:11.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:11.750 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:17:11.851 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:17:11.853 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:11 smithi177 ceph-mon[44303]: pgmap v1827: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 435 op/s; 7926/101712 objects degraded (7.793%); 14150/101712 objects misplaced (13.912%); 14 MiB/s, 302 objects/s recovering 2024-05-15T13:17:11.922 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:12.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:11 smithi027 ceph-mon[39795]: pgmap v1827: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 435 op/s; 7926/101712 objects degraded (7.793%); 14150/101712 objects misplaced (13.912%); 14 MiB/s, 302 objects/s recovering 2024-05-15T13:17:12.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:11 smithi120 ceph-mon[43961]: pgmap v1827: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 435 op/s; 7926/101712 objects degraded (7.793%); 14150/101712 objects misplaced (13.912%); 14 MiB/s, 302 objects/s recovering 2024-05-15T13:17:12.212 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:17:12.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:12.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:17:12.239 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-05-15T13:17:12.325 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:17:12.362 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:12.398 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:17:12.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:12.401 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:17:12.502 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:17:12.534 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:12.575 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:17:12.805 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:12 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3096791754' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:17:12.805 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:12 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:17:12.806 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:17:12.977 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:17:13.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:12 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3096791754' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:17:13.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:12 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:17:13.126 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:12 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3096791754' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:17:13.126 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:12 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:17:13.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:13 smithi177 ceph-mon[44303]: pgmap v1828: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 465 op/s; 7604/99780 objects degraded (7.621%); 14150/99780 objects misplaced (14.181%); 13 MiB/s, 287 objects/s recovering 2024-05-15T13:17:13.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:13 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:17:13.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:13 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:17:13.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:13 smithi177 ceph-mon[44303]: osdmap e703: 12 total, 12 up, 10 in 2024-05-15T13:17:14.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:13 smithi120 ceph-mon[43961]: pgmap v1828: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 465 op/s; 7604/99780 objects degraded (7.621%); 14150/99780 objects misplaced (14.181%); 13 MiB/s, 287 objects/s recovering 2024-05-15T13:17:14.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:13 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:17:14.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:13 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:17:14.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:13 smithi120 ceph-mon[43961]: osdmap e703: 12 total, 12 up, 10 in 2024-05-15T13:17:14.037 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:13 smithi027 ceph-mon[39795]: pgmap v1828: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 465 op/s; 7604/99780 objects degraded (7.621%); 14150/99780 objects misplaced (14.181%); 13 MiB/s, 287 objects/s recovering 2024-05-15T13:17:14.038 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:13 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:17:14.038 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:13 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:17:14.038 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:13 smithi027 ceph-mon[39795]: osdmap e703: 12 total, 12 up, 10 in 2024-05-15T13:17:14.562 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:17:14.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:14.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:17:14.661 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:17:14.661 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:17:14.671 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:17:14.699 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:14.717 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:17:14.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:14.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:17:14.821 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:17:14.861 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:14.908 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:17:14.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:14.915 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:17:15.016 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:17:15.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:14 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:17:15.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:14 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:17:15.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:14 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 7604/99780 objects degraded (7.621%), 5 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:15.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:14 smithi120 ceph-mon[43961]: osdmap e704: 12 total, 12 up, 10 in 2024-05-15T13:17:15.037 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:14 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:17:15.038 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:14 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:17:15.038 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:14 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 7604/99780 objects degraded (7.621%), 5 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:15.038 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:14 smithi027 ceph-mon[39795]: osdmap e704: 12 total, 12 up, 10 in 2024-05-15T13:17:15.049 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:14 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:17:15.049 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:14 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:17:15.049 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:14 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 7604/99780 objects degraded (7.621%), 5 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:15.049 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:14 smithi177 ceph-mon[44303]: osdmap e704: 12 total, 12 up, 10 in 2024-05-15T13:17:15.055 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:15.082 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:17:15.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:15.093 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:17:15.194 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:17:15.222 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:15.243 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:17:15.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:15.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:17:15.347 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:17:15.375 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:15.396 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:17:15.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:15.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:17:15.407 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-05-15T13:17:15.500 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:17:15.531 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:15.576 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:17:15.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:15.580 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:17:15.681 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:17:15.742 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:15.773 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:17:15.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:15.781 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:17:15.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:15 smithi027 ceph-mon[39795]: pgmap v1831: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 1.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 615 op/s; 7660/94425 objects degraded (8.112%); 13223/94425 objects misplaced (14.004%); 18 MiB/s, 433 objects/s recovering 2024-05-15T13:17:15.796 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:17:15.882 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:17:15.925 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:15.986 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:17:15.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:15 smithi177 ceph-mon[44303]: pgmap v1831: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 1.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 615 op/s; 7660/94425 objects degraded (8.112%); 13223/94425 objects misplaced (14.004%); 18 MiB/s, 433 objects/s recovering 2024-05-15T13:17:15.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:15.992 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:17:16.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:15 smithi120 ceph-mon[43961]: pgmap v1831: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 1.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 615 op/s; 7660/94425 objects degraded (8.112%); 13223/94425 objects misplaced (14.004%); 18 MiB/s, 433 objects/s recovering 2024-05-15T13:17:16.093 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:17:16.127 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:16.155 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:17:16.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:16.158 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:17:16.260 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:17:16.338 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:16.433 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:17:16.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:16.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:17:16.507 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:17:16.541 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:17:16.645 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:17.072 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:17:17.072 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"deaea04e-12b7-11ef-bc98-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7660/94017 objects degraded (8.147%), 5 pgs degraded, 7 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2212,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":704,"num_osds":12,"num_up_osds":12,"osd_up_since":1715777150,"num_in_osds":10,"osd_in_since":1715778703,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":29},{"state_name":"active+recovering+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+remapped","count":2},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait+remapped","count":1}],"num_pgs":37,"num_pools":2,"num_objects":31339,"data_bytes":2054226488,"bytes_used":16016138240,"bytes_avail":943934218240,"bytes_total":959950356480,"degraded_objects":7660,"degraded_total":94017,"degraded_ratio":0.081474626929172386,"misplaced_objects":13223,"misplaced_total":94017,"misplaced_ratio":0.14064477700841338,"recovering_objects_per_sec":265,"recovering_bytes_per_sec":11941412,"recovering_keys_per_sec":0,"num_objects_recovered":2126,"num_bytes_recovered":95682560,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":400},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.27:8443/"}},"servicemap":{"epoch":47,"modified":"2024-05-15T13:16:12.455588+0000","services":{"osd":{"daemons":{"summary":"","6":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-05-15T13:17:17.859 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:17:17.859 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:17:17.871 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:17:18.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:17 smithi177 ceph-mon[44303]: pgmap v1832: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 1.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 400 op/s; 7660/94017 objects degraded (8.147%); 13223/94017 objects misplaced (14.064%); 11 MiB/s, 265 objects/s recovering 2024-05-15T13:17:18.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:17 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1906090480' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-05-15T13:17:18.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:17 smithi027 ceph-mon[39795]: pgmap v1832: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 1.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 400 op/s; 7660/94017 objects degraded (8.147%); 13223/94017 objects misplaced (14.064%); 11 MiB/s, 265 objects/s recovering 2024-05-15T13:17:18.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:17 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1906090480' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-05-15T13:17:18.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:17 smithi120 ceph-mon[43961]: pgmap v1832: 37 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 1.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 400 op/s; 7660/94017 objects degraded (8.147%); 13223/94017 objects misplaced (14.064%); 11 MiB/s, 265 objects/s recovering 2024-05-15T13:17:18.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:17 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1906090480' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-05-15T13:17:18.590 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-05-15T13:17:18.862 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:17:18.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:18 smithi177 ceph-mon[44303]: osdmap e705: 12 total, 12 up, 10 in 2024-05-15T13:17:19.273 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:17:19.279 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:18 smithi027 ceph-mon[39795]: osdmap e705: 12 total, 12 up, 10 in 2024-05-15T13:17:19.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:19.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:17:19.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:18 smithi120 ceph-mon[43961]: osdmap e705: 12 total, 12 up, 10 in 2024-05-15T13:17:19.383 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:17:19.410 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:19.432 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:17:19.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:19.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:17:19.538 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:17:19.567 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:19.591 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:17:19.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:19.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:17:19.695 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:17:19.772 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:19.817 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:17:19.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:19.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:17:19.923 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:17:19.950 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:19.968 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:17:19.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:19.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:17:20.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:19 smithi027 ceph-mon[39795]: pgmap v1834: 37 pgs: 1 peering, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 1.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 516 op/s; 7379/90474 objects degraded (8.156%); 13062/90474 objects misplaced (14.437%); 12 MiB/s, 418 objects/s recovering 2024-05-15T13:17:20.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:19 smithi027 ceph-mon[39795]: osdmap e706: 12 total, 12 up, 10 in 2024-05-15T13:17:20.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:19 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 7379/90474 objects degraded (8.156%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:20.072 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:17:20.125 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:19 smithi177 ceph-mon[44303]: pgmap v1834: 37 pgs: 1 peering, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 1.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 516 op/s; 7379/90474 objects degraded (8.156%); 13062/90474 objects misplaced (14.437%); 12 MiB/s, 418 objects/s recovering 2024-05-15T13:17:20.125 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:19 smithi177 ceph-mon[44303]: osdmap e706: 12 total, 12 up, 10 in 2024-05-15T13:17:20.126 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:19 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 7379/90474 objects degraded (8.156%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:20.160 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:20.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:19 smithi120 ceph-mon[43961]: pgmap v1834: 37 pgs: 1 peering, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 1.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 516 op/s; 7379/90474 objects degraded (8.156%); 13062/90474 objects misplaced (14.437%); 12 MiB/s, 418 objects/s recovering 2024-05-15T13:17:20.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:19 smithi120 ceph-mon[43961]: osdmap e706: 12 total, 12 up, 10 in 2024-05-15T13:17:20.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:19 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 7379/90474 objects degraded (8.156%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:20.779 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:17:20.779 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:17:21.144 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:20 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:17:21.144 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:20 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:17:21.144 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:20 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:17:21.144 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:20 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:17:21.144 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:20 smithi177 ceph-mon[44303]: osdmap e707: 12 total, 12 up, 10 in 2024-05-15T13:17:21.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:20 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:17:21.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:20 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:17:21.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:20 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:17:21.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:20 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:17:21.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:20 smithi027 ceph-mon[39795]: osdmap e707: 12 total, 12 up, 10 in 2024-05-15T13:17:21.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:20 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:17:21.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:20 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:17:21.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:20 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:17:21.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:20 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:17:21.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:20 smithi120 ceph-mon[43961]: osdmap e707: 12 total, 12 up, 10 in 2024-05-15T13:17:21.497 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-05-15T13:17:21.811 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:17:22.078 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:21 smithi177 ceph-mon[44303]: pgmap v1837: 37 pgs: 1 peering, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 1.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 368 op/s; 7388/87795 objects degraded (8.415%); 11974/87795 objects misplaced (13.639%); 14 MiB/s, 342 objects/s recovering 2024-05-15T13:17:22.078 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:21 smithi177 ceph-mon[44303]: osdmap e708: 12 total, 12 up, 10 in 2024-05-15T13:17:22.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:21 smithi027 ceph-mon[39795]: pgmap v1837: 37 pgs: 1 peering, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 1.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 368 op/s; 7388/87795 objects degraded (8.415%); 11974/87795 objects misplaced (13.639%); 14 MiB/s, 342 objects/s recovering 2024-05-15T13:17:22.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:21 smithi027 ceph-mon[39795]: osdmap e708: 12 total, 12 up, 10 in 2024-05-15T13:17:22.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:21 smithi120 ceph-mon[43961]: pgmap v1837: 37 pgs: 1 peering, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 1.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 368 op/s; 7388/87795 objects degraded (8.415%); 11974/87795 objects misplaced (13.639%); 14 MiB/s, 342 objects/s recovering 2024-05-15T13:17:22.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:21 smithi120 ceph-mon[43961]: osdmap e708: 12 total, 12 up, 10 in 2024-05-15T13:17:22.872 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:17:22.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:22.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:17:22.907 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-05-15T13:17:22.907 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:17:22.978 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:17:22.995 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:17:23.105 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:23.228 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:17:23.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:23.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:17:23.339 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:17:23.352 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:23 smithi027 ceph-mon[39795]: osdmap e709: 12 total, 12 up, 10 in 2024-05-15T13:17:23.352 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:23 smithi027 ceph-mon[39795]: pgmap v1840: 37 pgs: 1 active+recovering+remapped, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 1.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 341 op/s; 7158/86379 objects degraded (8.287%); 10854/86379 objects misplaced (12.566%); 20 MiB/s, 317 objects/s recovering 2024-05-15T13:17:23.366 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:23.386 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:17:23.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:23.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:17:23.393 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:23 smithi177 ceph-mon[44303]: osdmap e709: 12 total, 12 up, 10 in 2024-05-15T13:17:23.393 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:23 smithi177 ceph-mon[44303]: pgmap v1840: 37 pgs: 1 active+recovering+remapped, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 1.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 341 op/s; 7158/86379 objects degraded (8.287%); 10854/86379 objects misplaced (12.566%); 20 MiB/s, 317 objects/s recovering 2024-05-15T13:17:23.490 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:17:23.517 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:23 smithi120 ceph-mon[43961]: osdmap e709: 12 total, 12 up, 10 in 2024-05-15T13:17:23.517 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:23 smithi120 ceph-mon[43961]: pgmap v1840: 37 pgs: 1 active+recovering+remapped, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 1.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 341 op/s; 7158/86379 objects degraded (8.287%); 10854/86379 objects misplaced (12.566%); 20 MiB/s, 317 objects/s recovering 2024-05-15T13:17:23.519 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:23.537 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:17:23.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:23.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:17:23.642 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:17:23.671 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:23.683 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:17:23.683 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:17:23.691 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:17:23.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:23.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:17:23.795 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:17:23.821 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:23.845 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:17:23.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:23.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:17:23.950 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:17:23.993 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:24.839 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:17:24.890 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:17:24.954 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:17:24.988 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:17:25.035 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:17:25.085 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:17:25.143 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:17:25.178 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:17:25.264 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:17:25.322 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:17:25.379 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:17:25.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:25 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 7158/86379 objects degraded (8.287%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:25.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:25 smithi177 ceph-mon[44303]: osdmap e710: 12 total, 12 up, 10 in 2024-05-15T13:17:25.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:25 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:17:25.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:25 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:17:25.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:25 smithi177 ceph-mon[44303]: pgmap v1842: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 31 active+clean; 1.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 459 op/s; 7155/81714 objects degraded (8.756%); 9897/81714 objects misplaced (12.112%); 16 MiB/s, 261 objects/s recovering 2024-05-15T13:17:25.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:25 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2910865498' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:17:25.437 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:17:25.457 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:17:25.457 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:17:25.457 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:17:25.457 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:17:25.457 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:17:25.457 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:17:25.457 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:17:25.457 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:17:25.457 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:17:25.457 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:17:25.458 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:17:25.458 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:17:25.499 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:25 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 7158/86379 objects degraded (8.287%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:25.499 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:25 smithi027 ceph-mon[39795]: osdmap e710: 12 total, 12 up, 10 in 2024-05-15T13:17:25.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:25 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:17:25.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:25 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:17:25.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:25 smithi027 ceph-mon[39795]: pgmap v1842: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 31 active+clean; 1.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 459 op/s; 7155/81714 objects degraded (8.756%); 9897/81714 objects misplaced (12.112%); 16 MiB/s, 261 objects/s recovering 2024-05-15T13:17:25.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:25 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2910865498' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:17:25.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:25 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 7158/86379 objects degraded (8.287%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:25.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:25 smithi120 ceph-mon[43961]: osdmap e710: 12 total, 12 up, 10 in 2024-05-15T13:17:25.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:25 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:17:25.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:25 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:17:25.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:25 smithi120 ceph-mon[43961]: pgmap v1842: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 31 active+clean; 1.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 459 op/s; 7155/81714 objects degraded (8.756%); 9897/81714 objects misplaced (12.112%); 16 MiB/s, 261 objects/s recovering 2024-05-15T13:17:25.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:25 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2910865498' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:17:26.527 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:26 smithi027 ceph-mon[39795]: osdmap e711: 12 total, 12 up, 10 in 2024-05-15T13:17:26.527 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:26 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:17:26.527 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:26 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:17:26.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:26 smithi120 ceph-mon[43961]: osdmap e711: 12 total, 12 up, 10 in 2024-05-15T13:17:26.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:26 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:17:26.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:26 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:17:26.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:26 smithi177 ceph-mon[44303]: osdmap e711: 12 total, 12 up, 10 in 2024-05-15T13:17:26.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:26 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:17:26.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:26 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:17:26.696 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:17:26.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:26.702 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:17:26.716 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-05-15T13:17:26.804 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:17:26.832 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:26.852 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:17:26.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:26.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:17:26.956 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:17:27.014 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:27.045 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:17:27.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:27.052 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:17:27.122 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:17:27.153 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:17:27.181 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:27.389 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:17:27.393 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:27 smithi177 ceph-mon[44303]: osdmap e712: 12 total, 12 up, 10 in 2024-05-15T13:17:27.393 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:27 smithi177 ceph-mon[44303]: pgmap v1845: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 31 active+clean; 1.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 440 op/s; 7155/81300 objects degraded (8.801%); 9897/81300 objects misplaced (12.173%); 15 MiB/s, 237 objects/s recovering 2024-05-15T13:17:27.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:27.394 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:17:27.496 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:17:27.527 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:27 smithi027 ceph-mon[39795]: osdmap e712: 12 total, 12 up, 10 in 2024-05-15T13:17:27.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:27 smithi027 ceph-mon[39795]: pgmap v1845: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 31 active+clean; 1.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 440 op/s; 7155/81300 objects degraded (8.801%); 9897/81300 objects misplaced (12.173%); 15 MiB/s, 237 objects/s recovering 2024-05-15T13:17:27.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:27 smithi120 ceph-mon[43961]: osdmap e712: 12 total, 12 up, 10 in 2024-05-15T13:17:27.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:27 smithi120 ceph-mon[43961]: pgmap v1845: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 31 active+clean; 1.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 440 op/s; 7155/81300 objects degraded (8.801%); 9897/81300 objects misplaced (12.173%); 15 MiB/s, 237 objects/s recovering 2024-05-15T13:17:27.540 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:27.568 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:17:27.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:27.576 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:17:27.677 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:17:27.704 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:27.723 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:17:27.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:27.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:17:27.827 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:17:27.915 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:28.165 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:17:28.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:28.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:17:28.276 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:17:28.305 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:28.328 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:17:28.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:28.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:17:28.432 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:17:28.497 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:29.179 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:17:29.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:29.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:17:29.271 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:17:29.271 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-05-15T13:17:29.272 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-05-15T13:17:29.272 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T13:17:29.272 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.272 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:104900 7.5 7:a95fde01:::benchmark_data_smithi177_134484_object104899:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e643)", 2024-05-15T13:17:29.272 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:14:01.834913+0000", 2024-05-15T13:17:29.272 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 207.435439882, 2024-05-15T13:17:29.272 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23812360599999999, 2024-05-15T13:17:29.272 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:29.272 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:29.272 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:29.272 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:29.272 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:29.273 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 104900 2024-05-15T13:17:29.273 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.273 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:29.273 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.273 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:29.273 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.834913+0000", 2024-05-15T13:17:29.273 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.273 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.273 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.273 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:29.273 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.834913+0000", 2024-05-15T13:17:29.273 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.274 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.274 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.274 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:29.274 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.834915+0000", 2024-05-15T13:17:29.274 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.432e-06 2024-05-15T13:17:29.274 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.274 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.274 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:29.274 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.834935+0000", 2024-05-15T13:17:29.274 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0846999999999999e-05 2024-05-15T13:17:29.274 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.274 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.275 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:29.275 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.834945+0000", 2024-05-15T13:17:29.275 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1600000000000004e-06 2024-05-15T13:17:29.275 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.275 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.275 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:29.275 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.834952+0000", 2024-05-15T13:17:29.275 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3130000000000001e-06 2024-05-15T13:17:29.275 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.275 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.275 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.276 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.835012+0000", 2024-05-15T13:17:29.276 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9978000000000001e-05 2024-05-15T13:17:29.276 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.276 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.276 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:29.276 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.835181+0000", 2024-05-15T13:17:29.276 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000169438 2024-05-15T13:17:29.276 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.276 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.277 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:17:29.277 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.835311+0000", 2024-05-15T13:17:29.277 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013021300000000001 2024-05-15T13:17:29.277 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.277 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.277 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:29.277 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.844897+0000", 2024-05-15T13:17:29.277 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0095854789999999992 2024-05-15T13:17:29.277 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.278 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.278 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.278 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.844958+0000", 2024-05-15T13:17:29.278 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1267999999999997e-05 2024-05-15T13:17:29.278 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.278 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.278 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.278 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:02.072901+0000", 2024-05-15T13:17:29.278 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.22794307899999999 2024-05-15T13:17:29.278 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.278 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.278 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:29.279 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:02.072979+0000", 2024-05-15T13:17:29.279 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7502000000000004e-05 2024-05-15T13:17:29.279 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.279 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.279 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:29.279 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:02.073037+0000", 2024-05-15T13:17:29.279 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7896999999999998e-05 2024-05-15T13:17:29.279 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.279 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:29.279 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.279 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.279 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.279 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:104905 7.5 7:ae617ca7:::benchmark_data_smithi177_134484_object104904:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e643)", 2024-05-15T13:17:29.279 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:14:01.838459+0000", 2024-05-15T13:17:29.280 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 207.43189391799999, 2024-05-15T13:17:29.280 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.238458432, 2024-05-15T13:17:29.280 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:29.280 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:29.280 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:29.280 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:29.280 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:29.280 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 104905 2024-05-15T13:17:29.280 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.280 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:29.280 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.280 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:29.280 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.838459+0000", 2024-05-15T13:17:29.281 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.281 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.281 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.281 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:29.281 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.838459+0000", 2024-05-15T13:17:29.281 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.281 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.281 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.281 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:29.281 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.838460+0000", 2024-05-15T13:17:29.281 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.291e-06 2024-05-15T13:17:29.281 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.281 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.281 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:29.281 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.838482+0000", 2024-05-15T13:17:29.282 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1212999999999998e-05 2024-05-15T13:17:29.282 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.282 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.282 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:29.282 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.838495+0000", 2024-05-15T13:17:29.282 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3254e-05 2024-05-15T13:17:29.282 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.282 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.282 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:29.282 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.838502+0000", 2024-05-15T13:17:29.282 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.046e-06 2024-05-15T13:17:29.282 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.282 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.283 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.283 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.838746+0000", 2024-05-15T13:17:29.283 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024436899999999998 2024-05-15T13:17:29.283 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.283 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.283 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:29.283 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.838903+0000", 2024-05-15T13:17:29.283 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000157217 2024-05-15T13:17:29.283 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.283 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.283 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:17:29.283 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.839047+0000", 2024-05-15T13:17:29.283 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014342700000000001 2024-05-15T13:17:29.284 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.284 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.284 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:29.284 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.847905+0000", 2024-05-15T13:17:29.284 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.008858253 2024-05-15T13:17:29.284 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.284 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.284 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.284 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.847963+0000", 2024-05-15T13:17:29.284 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7717000000000003e-05 2024-05-15T13:17:29.284 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.284 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.284 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.284 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:02.076845+0000", 2024-05-15T13:17:29.285 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.22888188200000001 2024-05-15T13:17:29.285 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.285 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.285 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:29.285 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:02.076882+0000", 2024-05-15T13:17:29.285 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7580000000000003e-05 2024-05-15T13:17:29.285 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.285 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.285 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:29.285 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:02.076918+0000", 2024-05-15T13:17:29.285 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5182999999999998e-05 2024-05-15T13:17:29.285 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.285 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:29.285 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.286 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.286 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.286 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:104906 7.5 7:a8d1313b:::benchmark_data_smithi177_134484_object104905:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e643)", 2024-05-15T13:17:29.286 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:14:01.841773+0000", 2024-05-15T13:17:29.286 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 207.42858034, 2024-05-15T13:17:29.286 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.235256458, 2024-05-15T13:17:29.286 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:29.286 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:29.286 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:29.286 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:29.286 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:29.286 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 104906 2024-05-15T13:17:29.286 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.287 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:29.287 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.287 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:29.287 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.841773+0000", 2024-05-15T13:17:29.287 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.287 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.287 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.287 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:29.287 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.841773+0000", 2024-05-15T13:17:29.287 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.287 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.287 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.287 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:29.287 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.841774+0000", 2024-05-15T13:17:29.288 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7430000000000001e-06 2024-05-15T13:17:29.288 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.288 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.288 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:29.288 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.841807+0000", 2024-05-15T13:17:29.288 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2298999999999997e-05 2024-05-15T13:17:29.288 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.288 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.288 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:29.288 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.841827+0000", 2024-05-15T13:17:29.288 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9930000000000001e-05 2024-05-15T13:17:29.288 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.289 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.289 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:29.289 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.841839+0000", 2024-05-15T13:17:29.289 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2393e-05 2024-05-15T13:17:29.289 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.289 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.289 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.289 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.842017+0000", 2024-05-15T13:17:29.289 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000178476 2024-05-15T13:17:29.289 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.289 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.289 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:29.289 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.842119+0000", 2024-05-15T13:17:29.289 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010194799999999999 2024-05-15T13:17:29.290 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.290 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.290 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:17:29.290 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.842221+0000", 2024-05-15T13:17:29.290 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000101537 2024-05-15T13:17:29.290 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.290 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.290 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.290 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.848017+0000", 2024-05-15T13:17:29.290 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0057962550000000002 2024-05-15T13:17:29.290 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.290 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.290 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:29.291 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.848025+0000", 2024-05-15T13:17:29.291 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9270000000000005e-06 2024-05-15T13:17:29.291 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.291 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.291 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.291 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:02.076959+0000", 2024-05-15T13:17:29.291 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.228933358 2024-05-15T13:17:29.291 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.291 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.291 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:29.291 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:02.076994+0000", 2024-05-15T13:17:29.291 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5751999999999999e-05 2024-05-15T13:17:29.291 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.291 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.291 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:29.292 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:02.077029+0000", 2024-05-15T13:17:29.292 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4839999999999998e-05 2024-05-15T13:17:29.292 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.292 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:29.292 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.292 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.292 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.292 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:104907 7.5 7:a8b7f01f:::benchmark_data_smithi177_134484_object104906:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e643)", 2024-05-15T13:17:29.292 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:14:01.841871+0000", 2024-05-15T13:17:29.292 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 207.42848165199999, 2024-05-15T13:17:29.292 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23529304400000001, 2024-05-15T13:17:29.292 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:29.292 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:29.293 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:29.293 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:29.293 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:29.293 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 104907 2024-05-15T13:17:29.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.293 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:29.293 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.293 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:29.293 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.841871+0000", 2024-05-15T13:17:29.293 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.293 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.294 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:29.294 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.841871+0000", 2024-05-15T13:17:29.294 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.294 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.294 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.294 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:29.294 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.841872+0000", 2024-05-15T13:17:29.294 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.44e-07 2024-05-15T13:17:29.294 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.295 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.295 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:29.295 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.841890+0000", 2024-05-15T13:17:29.295 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7620000000000001e-05 2024-05-15T13:17:29.295 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.295 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.295 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:29.295 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.841896+0000", 2024-05-15T13:17:29.295 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.331e-06 2024-05-15T13:17:29.295 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.295 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.296 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:29.296 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.841902+0000", 2024-05-15T13:17:29.296 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9680000000000003e-06 2024-05-15T13:17:29.296 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.296 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.296 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.296 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.844995+0000", 2024-05-15T13:17:29.296 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0030927020000000001 2024-05-15T13:17:29.296 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.296 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.296 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:29.296 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.845136+0000", 2024-05-15T13:17:29.296 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014147500000000001 2024-05-15T13:17:29.296 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.297 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.297 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:17:29.297 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.845281+0000", 2024-05-15T13:17:29.297 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014524699999999999 2024-05-15T13:17:29.297 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.297 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.297 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:29.297 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.848030+0000", 2024-05-15T13:17:29.297 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0027486490000000001 2024-05-15T13:17:29.297 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.297 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.297 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.297 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.848807+0000", 2024-05-15T13:17:29.297 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00077746200000000003 2024-05-15T13:17:29.298 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.298 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.298 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.298 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:02.077078+0000", 2024-05-15T13:17:29.298 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.228270907 2024-05-15T13:17:29.298 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.298 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.298 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:29.298 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:02.077131+0000", 2024-05-15T13:17:29.298 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2957000000000001e-05 2024-05-15T13:17:29.298 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.298 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.298 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:29.298 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:02.077164+0000", 2024-05-15T13:17:29.299 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3081999999999999e-05 2024-05-15T13:17:29.299 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.299 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:29.299 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.299 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.299 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.299 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:145960 7.5 7:a1515312:::benchmark_data_smithi177_134484_object36895:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:17:29.299 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.569536+0000", 2024-05-15T13:17:29.299 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 109.700816909, 2024-05-15T13:17:29.299 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2435298179999998, 2024-05-15T13:17:29.299 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:29.299 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:29.299 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:29.300 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:29.300 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:29.300 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 145960 2024-05-15T13:17:29.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.300 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:29.300 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.300 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:29.300 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569536+0000", 2024-05-15T13:17:29.300 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.300 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.300 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:29.300 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569536+0000", 2024-05-15T13:17:29.300 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.301 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.301 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:29.301 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569538+0000", 2024-05-15T13:17:29.301 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4789999999999999e-06 2024-05-15T13:17:29.301 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.301 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.301 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:17:29.303 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:29.303 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569545+0000", 2024-05-15T13:17:29.303 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5299999999999999e-06 2024-05-15T13:17:29.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:29.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569561+0000", 2024-05-15T13:17:29.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5747e-05 2024-05-15T13:17:29.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:29.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569575+0000", 2024-05-15T13:17:29.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3915e-05 2024-05-15T13:17:29.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.305 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569651+0000", 2024-05-15T13:17:29.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6126999999999998e-05 2024-05-15T13:17:29.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.305 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.305 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569660+0000", 2024-05-15T13:17:29.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5480000000000007e-06 2024-05-15T13:17:29.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.305 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.305 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530145+0000", 2024-05-15T13:17:29.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96048494399999995 2024-05-15T13:17:29.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.306 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.306 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530173+0000", 2024-05-15T13:17:29.306 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7702999999999999e-05 2024-05-15T13:17:29.306 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.306 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.306 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.306 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.800476+0000", 2024-05-15T13:17:29.306 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2703030390000001 2024-05-15T13:17:29.306 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.306 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.306 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:29.306 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.800594+0000", 2024-05-15T13:17:29.306 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117899 2024-05-15T13:17:29.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.307 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:17:29.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.800698+0000", 2024-05-15T13:17:29.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104308 2024-05-15T13:17:29.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.307 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:29.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.810577+0000", 2024-05-15T13:17:29.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0098783619999999999 2024-05-15T13:17:29.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.307 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.810611+0000", 2024-05-15T13:17:29.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4394000000000003e-05 2024-05-15T13:17:29.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.308 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.810670+0000", 2024-05-15T13:17:29.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9113999999999997e-05 2024-05-15T13:17:29.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:29.308 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.810718+0000", 2024-05-15T13:17:29.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7692999999999999e-05 2024-05-15T13:17:29.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:29.308 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.813066+0000", 2024-05-15T13:17:29.309 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023480160000000001 2024-05-15T13:17:29.309 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.309 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:29.309 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.309 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.309 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.309 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:145962 7.5 7:a16081f3:::benchmark_data_smithi177_134484_object36897:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:17:29.309 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.569605+0000", 2024-05-15T13:17:29.309 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 109.700747611, 2024-05-15T13:17:29.309 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2554589759999999, 2024-05-15T13:17:29.309 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:29.309 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:29.309 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:29.310 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:29.310 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:29.310 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 145962 2024-05-15T13:17:29.310 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.310 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:29.310 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.310 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:29.310 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569605+0000", 2024-05-15T13:17:29.310 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.310 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.310 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.310 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:29.310 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569605+0000", 2024-05-15T13:17:29.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:29.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569606+0000", 2024-05-15T13:17:29.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8899999999999999e-07 2024-05-15T13:17:29.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:29.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569609+0000", 2024-05-15T13:17:29.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3390000000000001e-06 2024-05-15T13:17:29.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:29.312 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569617+0000", 2024-05-15T13:17:29.312 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3019999999999997e-06 2024-05-15T13:17:29.312 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.312 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.312 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:29.312 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569623+0000", 2024-05-15T13:17:29.312 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9370000000000001e-06 2024-05-15T13:17:29.312 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.312 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.312 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.312 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569685+0000", 2024-05-15T13:17:29.312 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2867000000000006e-05 2024-05-15T13:17:29.312 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.312 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.312 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.313 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569691+0000", 2024-05-15T13:17:29.313 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8000000000000004e-06 2024-05-15T13:17:29.313 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.313 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.313 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.313 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530285+0000", 2024-05-15T13:17:29.313 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96059413500000002 2024-05-15T13:17:29.313 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.313 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.313 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.313 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530298+0000", 2024-05-15T13:17:29.313 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2278e-05 2024-05-15T13:17:29.313 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.313 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.314 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.314 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.806841+0000", 2024-05-15T13:17:29.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2765429770000001 2024-05-15T13:17:29.314 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.314 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.314 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:29.314 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.806953+0000", 2024-05-15T13:17:29.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011249200000000001 2024-05-15T13:17:29.314 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.314 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.314 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:17:29.314 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.807060+0000", 2024-05-15T13:17:29.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000107354 2024-05-15T13:17:29.315 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.315 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.315 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:29.315 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.822037+0000", 2024-05-15T13:17:29.315 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014976635 2024-05-15T13:17:29.315 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.315 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.315 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.315 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.822097+0000", 2024-05-15T13:17:29.315 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9883999999999998e-05 2024-05-15T13:17:29.315 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.315 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.315 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.315 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.822147+0000", 2024-05-15T13:17:29.316 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0435000000000001e-05 2024-05-15T13:17:29.316 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.316 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.316 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:29.316 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.822189+0000", 2024-05-15T13:17:29.316 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1446999999999998e-05 2024-05-15T13:17:29.316 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.316 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.316 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:29.316 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.825064+0000", 2024-05-15T13:17:29.316 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002875505 2024-05-15T13:17:29.316 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.317 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:29.317 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.317 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.317 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.317 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:145966 7.5 7:aee1f7e1:::benchmark_data_smithi177_134484_object36901:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:17:29.317 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.588210+0000", 2024-05-15T13:17:29.317 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 109.682142954, 2024-05-15T13:17:29.317 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2397243819999999, 2024-05-15T13:17:29.317 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:29.317 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:29.317 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:29.317 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:29.317 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:29.318 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 145966 2024-05-15T13:17:29.318 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.318 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:29.318 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.318 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:29.318 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.588210+0000", 2024-05-15T13:17:29.318 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.318 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.318 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.318 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:29.318 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.588210+0000", 2024-05-15T13:17:29.318 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.318 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.318 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.319 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:29.319 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.588212+0000", 2024-05-15T13:17:29.319 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8470000000000001e-06 2024-05-15T13:17:29.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.319 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.319 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:29.319 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.588219+0000", 2024-05-15T13:17:29.319 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1239999999999999e-06 2024-05-15T13:17:29.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.319 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.319 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:29.319 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.588232+0000", 2024-05-15T13:17:29.319 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2988e-05 2024-05-15T13:17:29.320 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.320 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.320 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:29.320 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.588240+0000", 2024-05-15T13:17:29.320 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1110000000000008e-06 2024-05-15T13:17:29.320 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.320 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.320 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.320 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.588295+0000", 2024-05-15T13:17:29.320 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5294000000000003e-05 2024-05-15T13:17:29.320 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.320 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.320 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.320 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.588303+0000", 2024-05-15T13:17:29.321 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7810000000000005e-06 2024-05-15T13:17:29.321 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.321 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.321 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.321 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530361+0000", 2024-05-15T13:17:29.321 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.94205747900000003 2024-05-15T13:17:29.321 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.321 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.321 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.321 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530373+0000", 2024-05-15T13:17:29.321 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2500000000000001e-05 2024-05-15T13:17:29.321 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.321 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.321 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.322 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.813130+0000", 2024-05-15T13:17:29.322 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2827566130000001 2024-05-15T13:17:29.322 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.322 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.322 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:29.322 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.813244+0000", 2024-05-15T13:17:29.322 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114652 2024-05-15T13:17:29.322 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.322 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.322 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:17:29.322 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.813419+0000", 2024-05-15T13:17:29.322 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017432600000000001 2024-05-15T13:17:29.322 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.322 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.323 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:29.323 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.825083+0000", 2024-05-15T13:17:29.323 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011664091999999999 2024-05-15T13:17:29.323 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.323 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.323 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.323 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.825153+0000", 2024-05-15T13:17:29.323 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0055999999999995e-05 2024-05-15T13:17:29.323 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.323 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.323 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.323 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.825293+0000", 2024-05-15T13:17:29.323 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014042699999999999 2024-05-15T13:17:29.323 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.324 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.324 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:29.324 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.825361+0000", 2024-05-15T13:17:29.324 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7619000000000004e-05 2024-05-15T13:17:29.324 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.324 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.324 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:29.324 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.827934+0000", 2024-05-15T13:17:29.324 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025734730000000002 2024-05-15T13:17:29.324 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.324 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:29.324 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.324 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.325 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.325 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:145976 7.5 7:ab177066:::benchmark_data_smithi177_134484_object36911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:17:29.325 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.611889+0000", 2024-05-15T13:17:29.325 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 109.658463971, 2024-05-15T13:17:29.325 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2317611080000002, 2024-05-15T13:17:29.325 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:29.325 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:29.325 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:29.325 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:29.325 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:29.325 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 145976 2024-05-15T13:17:29.325 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.325 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:29.325 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.326 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:29.326 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.611889+0000", 2024-05-15T13:17:29.326 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.326 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.326 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.326 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:29.326 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.611889+0000", 2024-05-15T13:17:29.326 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.326 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.326 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.326 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:29.326 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.611890+0000", 2024-05-15T13:17:29.326 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2810000000000001e-06 2024-05-15T13:17:29.326 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.327 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.327 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:29.327 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.611897+0000", 2024-05-15T13:17:29.327 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.81e-06 2024-05-15T13:17:29.327 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.327 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.327 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:29.327 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.611910+0000", 2024-05-15T13:17:29.327 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2934e-05 2024-05-15T13:17:29.327 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.327 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.327 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:29.327 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.611917+0000", 2024-05-15T13:17:29.327 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7179999999999999e-06 2024-05-15T13:17:29.328 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.328 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.328 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.328 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.611990+0000", 2024-05-15T13:17:29.328 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3441000000000004e-05 2024-05-15T13:17:29.328 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.328 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.328 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.328 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.612002+0000", 2024-05-15T13:17:29.328 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1928e-05 2024-05-15T13:17:29.328 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.328 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.328 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.328 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530420+0000", 2024-05-15T13:17:29.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.91841766199999997 2024-05-15T13:17:29.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.329 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.329 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530430+0000", 2024-05-15T13:17:29.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9159999999999996e-06 2024-05-15T13:17:29.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.329 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.329 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.818055+0000", 2024-05-15T13:17:29.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.287625435 2024-05-15T13:17:29.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.329 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:29.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.818242+0000", 2024-05-15T13:17:29.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018715100000000001 2024-05-15T13:17:29.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:17:29.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.818409+0000", 2024-05-15T13:17:29.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016631600000000001 2024-05-15T13:17:29.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:29.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.825240+0000", 2024-05-15T13:17:29.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0068310080000000004 2024-05-15T13:17:29.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.828014+0000", 2024-05-15T13:17:29.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00277392 2024-05-15T13:17:29.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.840375+0000", 2024-05-15T13:17:29.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012361158000000001 2024-05-15T13:17:29.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.332 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.332 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:29.332 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.840445+0000", 2024-05-15T13:17:29.332 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9921999999999999e-05 2024-05-15T13:17:29.332 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.332 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.332 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:29.332 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.843650+0000", 2024-05-15T13:17:29.332 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0032055080000000001 2024-05-15T13:17:29.332 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.332 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:29.332 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.332 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.332 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.333 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:145984 7.5 7:a3dbbd39:::benchmark_data_smithi177_134484_object36919:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:17:29.333 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.636999+0000", 2024-05-15T13:17:29.333 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 109.63335407700001, 2024-05-15T13:17:29.333 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2198527179999998, 2024-05-15T13:17:29.333 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:29.333 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:29.333 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:29.333 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:29.333 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:29.333 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 145984 2024-05-15T13:17:29.333 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.333 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:29.333 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.334 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:29.334 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.636999+0000", 2024-05-15T13:17:29.334 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.334 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.334 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.334 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:29.334 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.636999+0000", 2024-05-15T13:17:29.334 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.334 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.334 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.334 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:29.334 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.637000+0000", 2024-05-15T13:17:29.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4219999999999999e-06 2024-05-15T13:17:29.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.335 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.335 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:29.335 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.637008+0000", 2024-05-15T13:17:29.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7479999999999993e-06 2024-05-15T13:17:29.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.335 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.335 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:29.335 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.637025+0000", 2024-05-15T13:17:29.335 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7059999999999999e-05 2024-05-15T13:17:29.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.335 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.335 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:29.336 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.637033+0000", 2024-05-15T13:17:29.336 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9640000000000003e-06 2024-05-15T13:17:29.336 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.336 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.336 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.336 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.637108+0000", 2024-05-15T13:17:29.336 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4603000000000004e-05 2024-05-15T13:17:29.336 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.336 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.336 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.336 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.637121+0000", 2024-05-15T13:17:29.336 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3145e-05 2024-05-15T13:17:29.336 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.336 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.337 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.337 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530492+0000", 2024-05-15T13:17:29.337 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.89337067400000003 2024-05-15T13:17:29.337 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.337 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.337 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530500+0000", 2024-05-15T13:17:29.337 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4330000000000004e-06 2024-05-15T13:17:29.337 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.337 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.337 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.337 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.828066+0000", 2024-05-15T13:17:29.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2975655580000001 2024-05-15T13:17:29.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.338 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.338 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:29.338 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.828240+0000", 2024-05-15T13:17:29.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017429500000000001 2024-05-15T13:17:29.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.338 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.338 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:17:29.338 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.828413+0000", 2024-05-15T13:17:29.338 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000173327 2024-05-15T13:17:29.338 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.338 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.339 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:29.339 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.843665+0000", 2024-05-15T13:17:29.339 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015251951999999999 2024-05-15T13:17:29.339 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.339 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.339 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.339 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.854202+0000", 2024-05-15T13:17:29.339 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010536634 2024-05-15T13:17:29.339 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.339 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.339 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.339 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.854318+0000", 2024-05-15T13:17:29.339 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011631199999999999 2024-05-15T13:17:29.340 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.340 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.340 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:29.340 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.854366+0000", 2024-05-15T13:17:29.340 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7944e-05 2024-05-15T13:17:29.340 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.340 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.340 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:29.340 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.856852+0000", 2024-05-15T13:17:29.340 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024856470000000001 2024-05-15T13:17:29.340 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.340 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:29.340 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.340 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.340 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.341 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:146006 7.5 7:a497d408:::benchmark_data_smithi177_134484_object36941:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:17:29.341 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.688862+0000", 2024-05-15T13:17:29.341 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 109.58149136, 2024-05-15T13:17:29.341 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.18289022, 2024-05-15T13:17:29.341 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:29.341 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:29.341 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:29.341 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:29.341 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:29.341 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 146006 2024-05-15T13:17:29.341 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.341 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:29.341 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.342 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:29.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.688862+0000", 2024-05-15T13:17:29.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.342 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.342 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:29.342 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.688862+0000", 2024-05-15T13:17:29.342 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.342 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.342 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.343 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:29.343 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.688863+0000", 2024-05-15T13:17:29.343 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2300000000000001e-06 2024-05-15T13:17:29.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.343 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.343 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:29.343 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.688868+0000", 2024-05-15T13:17:29.343 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5249999999999996e-06 2024-05-15T13:17:29.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.343 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.343 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:29.343 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.688880+0000", 2024-05-15T13:17:29.343 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1637999999999999e-05 2024-05-15T13:17:29.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.344 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:29.344 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.688888+0000", 2024-05-15T13:17:29.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0949999999999996e-06 2024-05-15T13:17:29.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.344 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.344 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.688951+0000", 2024-05-15T13:17:29.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3332999999999998e-05 2024-05-15T13:17:29.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.344 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.344 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.344 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.688963+0000", 2024-05-15T13:17:29.344 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.172e-05 2024-05-15T13:17:29.345 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.345 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.345 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.345 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530549+0000", 2024-05-15T13:17:29.345 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.84158606400000002 2024-05-15T13:17:29.345 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.345 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.345 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.345 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530562+0000", 2024-05-15T13:17:29.345 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3243e-05 2024-05-15T13:17:29.345 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.345 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.345 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.345 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.832724+0000", 2024-05-15T13:17:29.346 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3021619609999999 2024-05-15T13:17:29.346 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.346 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.346 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:29.346 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.832845+0000", 2024-05-15T13:17:29.346 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120609 2024-05-15T13:17:29.346 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.346 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.346 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:17:29.346 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.832953+0000", 2024-05-15T13:17:29.347 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108044 2024-05-15T13:17:29.347 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.347 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.347 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:29.347 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.850232+0000", 2024-05-15T13:17:29.347 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017278853 2024-05-15T13:17:29.347 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.347 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.347 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.347 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.854263+0000", 2024-05-15T13:17:29.347 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0040310479999999997 2024-05-15T13:17:29.347 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.347 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.347 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.348 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.868854+0000", 2024-05-15T13:17:29.348 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014591425 2024-05-15T13:17:29.348 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.348 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.348 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:29.348 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.868931+0000", 2024-05-15T13:17:29.348 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6642000000000002e-05 2024-05-15T13:17:29.348 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.348 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.348 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:29.348 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.871752+0000", 2024-05-15T13:17:29.348 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0028207900000000001 2024-05-15T13:17:29.348 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.348 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:29.349 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.349 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.349 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.349 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:146038 7.5 7:a8212863:::benchmark_data_smithi177_134484_object36973:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:17:29.349 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.762883+0000", 2024-05-15T13:17:29.349 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 109.50747037399999, 2024-05-15T13:17:29.349 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1114912019999998, 2024-05-15T13:17:29.349 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:29.349 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:29.349 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:29.349 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:29.349 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:29.349 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 146038 2024-05-15T13:17:29.350 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.350 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:29.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.350 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:29.350 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.762883+0000", 2024-05-15T13:17:29.350 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.350 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.350 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:29.350 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.762883+0000", 2024-05-15T13:17:29.350 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.350 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.350 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.351 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:29.351 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.762884+0000", 2024-05-15T13:17:29.351 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.469e-06 2024-05-15T13:17:29.351 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.351 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.351 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:29.351 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.762890+0000", 2024-05-15T13:17:29.351 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.117e-06 2024-05-15T13:17:29.351 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.351 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.351 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:29.351 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.762904+0000", 2024-05-15T13:17:29.351 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3906e-05 2024-05-15T13:17:29.351 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.352 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.352 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:29.352 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.762913+0000", 2024-05-15T13:17:29.352 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5269999999999998e-06 2024-05-15T13:17:29.352 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.352 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.352 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.352 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.762973+0000", 2024-05-15T13:17:29.352 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0609000000000002e-05 2024-05-15T13:17:29.352 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.352 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.353 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.353 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.762986+0000", 2024-05-15T13:17:29.353 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2736e-05 2024-05-15T13:17:29.353 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.353 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.353 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.353 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530624+0000", 2024-05-15T13:17:29.353 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.76763827900000003 2024-05-15T13:17:29.353 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.353 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.353 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.353 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530633+0000", 2024-05-15T13:17:29.353 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6969999999999999e-06 2024-05-15T13:17:29.353 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.354 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.354 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.354 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.836480+0000", 2024-05-15T13:17:29.354 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3058468300000001 2024-05-15T13:17:29.354 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.354 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.354 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:29.354 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.836597+0000", 2024-05-15T13:17:29.354 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117296 2024-05-15T13:17:29.354 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.354 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.354 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:17:29.355 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.836717+0000", 2024-05-15T13:17:29.355 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000119752 2024-05-15T13:17:29.355 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.355 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.355 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:29.355 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.850239+0000", 2024-05-15T13:17:29.355 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013522008 2024-05-15T13:17:29.355 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.355 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.356 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.356 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.856916+0000", 2024-05-15T13:17:29.356 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0066768829999999998 2024-05-15T13:17:29.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.356 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.356 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.356 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.871813+0000", 2024-05-15T13:17:29.356 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.01489701 2024-05-15T13:17:29.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.356 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.356 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:29.356 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.871858+0000", 2024-05-15T13:17:29.356 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5318999999999998e-05 2024-05-15T13:17:29.356 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.357 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.357 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:29.357 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.874374+0000", 2024-05-15T13:17:29.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025157640000000002 2024-05-15T13:17:29.357 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.357 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:29.357 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.357 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.357 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.357 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:146045 7.5 7:a967a4c6:::benchmark_data_smithi177_134484_object36980:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:17:29.357 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.777741+0000", 2024-05-15T13:17:29.357 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 109.49261240200001, 2024-05-15T13:17:29.357 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1091810830000002, 2024-05-15T13:17:29.357 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:29.357 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:29.358 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:29.358 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:29.358 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:29.358 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 146045 2024-05-15T13:17:29.358 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.358 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:29.358 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.358 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:29.358 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.777741+0000", 2024-05-15T13:17:29.358 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.358 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.358 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.358 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:29.359 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.777741+0000", 2024-05-15T13:17:29.359 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.359 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.359 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.359 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:29.359 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.777742+0000", 2024-05-15T13:17:29.359 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.288e-06 2024-05-15T13:17:29.359 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.359 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.359 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:29.359 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.777748+0000", 2024-05-15T13:17:29.359 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0009999999999998e-06 2024-05-15T13:17:29.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.360 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.360 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:29.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.777760+0000", 2024-05-15T13:17:29.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1987999999999999e-05 2024-05-15T13:17:29.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.360 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.360 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:29.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.777768+0000", 2024-05-15T13:17:29.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2360000000000004e-06 2024-05-15T13:17:29.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.360 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.360 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.360 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.777828+0000", 2024-05-15T13:17:29.360 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9814e-05 2024-05-15T13:17:29.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.361 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.777838+0000", 2024-05-15T13:17:29.361 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7350000000000008e-06 2024-05-15T13:17:29.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.361 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530677+0000", 2024-05-15T13:17:29.361 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.75283949299999997 2024-05-15T13:17:29.361 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.361 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.361 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.362 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530689+0000", 2024-05-15T13:17:29.362 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2292e-05 2024-05-15T13:17:29.362 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.362 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.362 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.362 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.843709+0000", 2024-05-15T13:17:29.362 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.313019911 2024-05-15T13:17:29.362 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.362 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.362 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:29.362 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.843836+0000", 2024-05-15T13:17:29.362 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012651599999999999 2024-05-15T13:17:29.362 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.362 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.363 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:17:29.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.843950+0000", 2024-05-15T13:17:29.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011428499999999999 2024-05-15T13:17:29.363 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.363 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.363 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:29.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.854275+0000", 2024-05-15T13:17:29.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010324453000000001 2024-05-15T13:17:29.363 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.363 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.363 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.363 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.874462+0000", 2024-05-15T13:17:29.363 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.020187159 2024-05-15T13:17:29.363 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.364 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.883398+0000", 2024-05-15T13:17:29.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0089358379999999998 2024-05-15T13:17:29.364 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:29.364 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.883455+0000", 2024-05-15T13:17:29.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7015000000000002e-05 2024-05-15T13:17:29.364 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.364 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.364 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:29.364 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.886922+0000", 2024-05-15T13:17:29.364 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0034670590000000002 2024-05-15T13:17:29.365 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.365 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:29.365 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.365 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.365 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.365 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:146047 7.5 7:af883097:::benchmark_data_smithi177_134484_object36982:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:17:29.365 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.783367+0000", 2024-05-15T13:17:29.365 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 109.48698603299999, 2024-05-15T13:17:29.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1068296379999998, 2024-05-15T13:17:29.366 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:29.366 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:29.366 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:29.366 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:29.366 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:29.367 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 146047 2024-05-15T13:17:29.367 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.367 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:29.367 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.367 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:29.367 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.783367+0000", 2024-05-15T13:17:29.367 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.367 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.368 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.368 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:29.368 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.783367+0000", 2024-05-15T13:17:29.368 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.368 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.368 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.368 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:29.368 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.783369+0000", 2024-05-15T13:17:29.368 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6029999999999999e-06 2024-05-15T13:17:29.368 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.368 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.368 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:29.368 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.783374+0000", 2024-05-15T13:17:29.368 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8100000000000003e-06 2024-05-15T13:17:29.369 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.369 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:29.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.783387+0000", 2024-05-15T13:17:29.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2296999999999999e-05 2024-05-15T13:17:29.369 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.369 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:29.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.783395+0000", 2024-05-15T13:17:29.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1689999999999992e-06 2024-05-15T13:17:29.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.783459+0000", 2024-05-15T13:17:29.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3909000000000001e-05 2024-05-15T13:17:29.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.783465+0000", 2024-05-15T13:17:29.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7279999999999998e-06 2024-05-15T13:17:29.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.371 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530725+0000", 2024-05-15T13:17:29.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.74725946799999998 2024-05-15T13:17:29.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.371 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.371 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530730+0000", 2024-05-15T13:17:29.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0000000000000004e-06 2024-05-15T13:17:29.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.371 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.372 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.850301+0000", 2024-05-15T13:17:29.372 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.319570774 2024-05-15T13:17:29.372 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.372 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.372 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:29.372 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.850451+0000", 2024-05-15T13:17:29.372 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000150266 2024-05-15T13:17:29.372 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.372 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.372 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:17:29.372 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.850597+0000", 2024-05-15T13:17:29.372 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014563600000000001 2024-05-15T13:17:29.372 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.372 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.373 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:29.373 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.864951+0000", 2024-05-15T13:17:29.373 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014353899999999999 2024-05-15T13:17:29.373 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.373 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.373 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.373 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.883335+0000", 2024-05-15T13:17:29.373 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.018384939999999999 2024-05-15T13:17:29.373 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.373 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.374 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.374 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.887036+0000", 2024-05-15T13:17:29.374 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037008459999999998 2024-05-15T13:17:29.374 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.374 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.374 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:29.374 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.887084+0000", 2024-05-15T13:17:29.374 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7394000000000001e-05 2024-05-15T13:17:29.374 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.374 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.374 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:29.374 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.890197+0000", 2024-05-15T13:17:29.375 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031128979999999998 2024-05-15T13:17:29.375 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.375 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:29.375 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.375 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.375 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.375 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:146049 7.5 7:ac75f5ad:::benchmark_data_smithi177_134484_object36984:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:17:29.375 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.791535+0000", 2024-05-15T13:17:29.375 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 109.478818072, 2024-05-15T13:17:29.375 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1017435240000002, 2024-05-15T13:17:29.375 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:29.375 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:29.375 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:29.375 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:29.376 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:29.376 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 146049 2024-05-15T13:17:29.376 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.376 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:29.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.376 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:29.376 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.791535+0000", 2024-05-15T13:17:29.376 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.376 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.376 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:29.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.791535+0000", 2024-05-15T13:17:29.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:29.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.791536+0000", 2024-05-15T13:17:29.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0529999999999999e-06 2024-05-15T13:17:29.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:29.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.791541+0000", 2024-05-15T13:17:29.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9300000000000002e-06 2024-05-15T13:17:29.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:29.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.791552+0000", 2024-05-15T13:17:29.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1117e-05 2024-05-15T13:17:29.378 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.378 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:29.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.791558+0000", 2024-05-15T13:17:29.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8209999999999998e-06 2024-05-15T13:17:29.378 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.378 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.791613+0000", 2024-05-15T13:17:29.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4722000000000002e-05 2024-05-15T13:17:29.379 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.379 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.379 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.791621+0000", 2024-05-15T13:17:29.379 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2989999999999997e-06 2024-05-15T13:17:29.379 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.379 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.379 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530771+0000", 2024-05-15T13:17:29.379 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.73915060200000005 2024-05-15T13:17:29.379 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.379 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.379 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530783+0000", 2024-05-15T13:17:29.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1513e-05 2024-05-15T13:17:29.380 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.380 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.380 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.380 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.856950+0000", 2024-05-15T13:17:29.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.326167332 2024-05-15T13:17:29.380 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.380 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.380 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:29.381 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.857060+0000", 2024-05-15T13:17:29.381 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010977899999999999 2024-05-15T13:17:29.381 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.381 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.381 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:17:29.381 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.857169+0000", 2024-05-15T13:17:29.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010908699999999999 2024-05-15T13:17:29.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.382 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:29.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.871772+0000", 2024-05-15T13:17:29.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014602364 2024-05-15T13:17:29.383 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.383 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.383 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.886979+0000", 2024-05-15T13:17:29.383 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015207001 2024-05-15T13:17:29.383 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.890279+0000", 2024-05-15T13:17:29.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0033007190000000001 2024-05-15T13:17:29.384 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:29.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.890345+0000", 2024-05-15T13:17:29.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5369000000000004e-05 2024-05-15T13:17:29.384 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:29.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.893278+0000", 2024-05-15T13:17:29.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0029338160000000001 2024-05-15T13:17:29.385 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.385 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:29.385 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.385 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.385 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.385 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:146056 7.5 7:a6b3a042:::benchmark_data_smithi177_134484_object36991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:17:29.385 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.815388+0000", 2024-05-15T13:17:29.385 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 109.45496477899999, 2024-05-15T13:17:29.385 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0813361119999998, 2024-05-15T13:17:29.385 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:29.385 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:29.385 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:29.385 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:29.385 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:29.386 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 146056 2024-05-15T13:17:29.386 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.386 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:29.386 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.386 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:29.386 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.815388+0000", 2024-05-15T13:17:29.386 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.386 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.386 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.386 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:29.386 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.815388+0000", 2024-05-15T13:17:29.387 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.387 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.387 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.387 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:29.387 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.815390+0000", 2024-05-15T13:17:29.387 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9230000000000001e-06 2024-05-15T13:17:29.387 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.387 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.387 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:29.387 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.815397+0000", 2024-05-15T13:17:29.387 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6880000000000002e-06 2024-05-15T13:17:29.387 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.387 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.388 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:29.388 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.815412+0000", 2024-05-15T13:17:29.388 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5184999999999999e-05 2024-05-15T13:17:29.388 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.388 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.388 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:29.388 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.815421+0000", 2024-05-15T13:17:29.388 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7360000000000007e-06 2024-05-15T13:17:29.388 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.388 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.388 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.388 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.815486+0000", 2024-05-15T13:17:29.388 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5191000000000007e-05 2024-05-15T13:17:29.388 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.389 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.389 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.389 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.815495+0000", 2024-05-15T13:17:29.389 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0089999999999996e-06 2024-05-15T13:17:29.389 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.389 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.389 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.389 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530838+0000", 2024-05-15T13:17:29.389 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.71534338099999994 2024-05-15T13:17:29.389 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.389 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.389 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.389 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530850+0000", 2024-05-15T13:17:29.389 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1725999999999999e-05 2024-05-15T13:17:29.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.390 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.390 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.861247+0000", 2024-05-15T13:17:29.390 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3303964509999999 2024-05-15T13:17:29.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.390 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:29.390 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.861408+0000", 2024-05-15T13:17:29.390 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016124800000000001 2024-05-15T13:17:29.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.390 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:17:29.390 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.861518+0000", 2024-05-15T13:17:29.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000110092 2024-05-15T13:17:29.391 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.391 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:29.391 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.874481+0000", 2024-05-15T13:17:29.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012963595 2024-05-15T13:17:29.391 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.391 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.391 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.893333+0000", 2024-05-15T13:17:29.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.018851402999999999 2024-05-15T13:17:29.391 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.391 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.392 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.893362+0000", 2024-05-15T13:17:29.392 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9371999999999999e-05 2024-05-15T13:17:29.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.392 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.392 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:29.392 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.893408+0000", 2024-05-15T13:17:29.392 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5445000000000003e-05 2024-05-15T13:17:29.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.392 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.392 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:29.392 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.896724+0000", 2024-05-15T13:17:29.392 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0033166670000000001 2024-05-15T13:17:29.392 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.393 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:29.393 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.393 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.393 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.393 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:146091 7.5 7:aeed9694:::benchmark_data_smithi177_134484_object37026:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:17:29.393 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.950203+0000", 2024-05-15T13:17:29.393 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 109.320149763, 2024-05-15T13:17:29.393 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9582113840000002, 2024-05-15T13:17:29.393 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:29.393 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:29.393 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:29.393 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:29.393 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:29.393 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 146091 2024-05-15T13:17:29.394 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.394 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:29.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.394 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:29.394 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.950203+0000", 2024-05-15T13:17:29.394 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.394 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.394 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:29.394 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.950203+0000", 2024-05-15T13:17:29.394 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.394 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:29.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.950204+0000", 2024-05-15T13:17:29.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.175e-06 2024-05-15T13:17:29.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:29.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.950211+0000", 2024-05-15T13:17:29.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7989999999999996e-06 2024-05-15T13:17:29.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:29.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.950225+0000", 2024-05-15T13:17:29.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3940999999999999e-05 2024-05-15T13:17:29.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:29.396 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.950232+0000", 2024-05-15T13:17:29.396 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.489e-06 2024-05-15T13:17:29.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.396 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.950293+0000", 2024-05-15T13:17:29.396 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1647000000000001e-05 2024-05-15T13:17:29.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.396 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.950301+0000", 2024-05-15T13:17:29.396 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6109999999999996e-06 2024-05-15T13:17:29.397 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.397 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.397 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.397 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530901+0000", 2024-05-15T13:17:29.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.580600373 2024-05-15T13:17:29.397 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.397 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.397 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.397 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530913+0000", 2024-05-15T13:17:29.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1718e-05 2024-05-15T13:17:29.397 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.397 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.398 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.398 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.865005+0000", 2024-05-15T13:17:29.398 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3340925129999999 2024-05-15T13:17:29.398 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.398 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.398 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:29.398 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.865116+0000", 2024-05-15T13:17:29.398 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011082300000000001 2024-05-15T13:17:29.398 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.398 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.398 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:17:29.398 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.865238+0000", 2024-05-15T13:17:29.398 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000121303 2024-05-15T13:17:29.399 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:29.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.874491+0000", 2024-05-15T13:17:29.399 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0092530589999999992 2024-05-15T13:17:29.399 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.896804+0000", 2024-05-15T13:17:29.399 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.022313625 2024-05-15T13:17:29.399 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.905865+0000", 2024-05-15T13:17:29.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0090606160000000005 2024-05-15T13:17:29.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:29.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.905905+0000", 2024-05-15T13:17:29.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9866000000000001e-05 2024-05-15T13:17:29.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:29.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.908415+0000", 2024-05-15T13:17:29.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025098260000000002 2024-05-15T13:17:29.400 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.400 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:29.400 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.401 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.401 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.401 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:146097 7.5 7:a626cefc:::benchmark_data_smithi177_134484_object37032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:17:29.401 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.982750+0000", 2024-05-15T13:17:29.401 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 109.28760334, 2024-05-15T13:17:29.401 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9281458680000001, 2024-05-15T13:17:29.401 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:29.401 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:29.401 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:29.401 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:29.401 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:29.401 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 146097 2024-05-15T13:17:29.401 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.401 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:29.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:29.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.982750+0000", 2024-05-15T13:17:29.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.402 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:29.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.982750+0000", 2024-05-15T13:17:29.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.402 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:29.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.982751+0000", 2024-05-15T13:17:29.403 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.339e-06 2024-05-15T13:17:29.403 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.403 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.403 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:29.403 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.982757+0000", 2024-05-15T13:17:29.403 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6350000000000003e-06 2024-05-15T13:17:29.403 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.404 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.404 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:29.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.982769+0000", 2024-05-15T13:17:29.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2581e-05 2024-05-15T13:17:29.404 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.404 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.404 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:29.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.982776+0000", 2024-05-15T13:17:29.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3729999999999998e-06 2024-05-15T13:17:29.404 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.404 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.404 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.982828+0000", 2024-05-15T13:17:29.405 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2868000000000003e-05 2024-05-15T13:17:29.405 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.405 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.405 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.982838+0000", 2024-05-15T13:17:29.405 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3270000000000007e-06 2024-05-15T13:17:29.405 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.405 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.405 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530958+0000", 2024-05-15T13:17:29.405 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.548119837 2024-05-15T13:17:29.405 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.405 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530969+0000", 2024-05-15T13:17:29.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1474000000000001e-05 2024-05-15T13:17:29.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.874519+0000", 2024-05-15T13:17:29.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.343550003 2024-05-15T13:17:29.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:29.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.874697+0000", 2024-05-15T13:17:29.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017830999999999999 2024-05-15T13:17:29.407 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.407 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:17:29.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.874814+0000", 2024-05-15T13:17:29.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000116564 2024-05-15T13:17:29.407 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.407 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:29.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.890221+0000", 2024-05-15T13:17:29.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015406716000000001 2024-05-15T13:17:29.407 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.408 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.408 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.896866+0000", 2024-05-15T13:17:29.408 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0066452270000000001 2024-05-15T13:17:29.408 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.408 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.408 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.408 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.908480+0000", 2024-05-15T13:17:29.409 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0116144 2024-05-15T13:17:29.409 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.409 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.409 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:29.409 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.908524+0000", 2024-05-15T13:17:29.409 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4098999999999999e-05 2024-05-15T13:17:29.409 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.409 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.410 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:29.410 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.910896+0000", 2024-05-15T13:17:29.410 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002371115 2024-05-15T13:17:29.410 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.410 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:29.410 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.410 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.410 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.411 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:146098 7.5 7:a23e1871:::benchmark_data_smithi177_134484_object37033:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:17:29.411 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.990742+0000", 2024-05-15T13:17:29.411 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 109.27961068099999, 2024-05-15T13:17:29.411 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9231258900000001, 2024-05-15T13:17:29.411 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:29.411 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:29.411 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:29.411 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:29.412 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:29.412 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 146098 2024-05-15T13:17:29.412 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.412 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:29.412 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.412 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:29.413 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.990742+0000", 2024-05-15T13:17:29.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.413 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.413 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:29.413 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.990742+0000", 2024-05-15T13:17:29.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:29.414 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.990744+0000", 2024-05-15T13:17:29.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4079999999999999e-06 2024-05-15T13:17:29.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:29.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.990749+0000", 2024-05-15T13:17:29.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5140000000000001e-06 2024-05-15T13:17:29.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:29.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.990761+0000", 2024-05-15T13:17:29.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2225000000000001e-05 2024-05-15T13:17:29.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:29.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.990768+0000", 2024-05-15T13:17:29.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0339999999999999e-06 2024-05-15T13:17:29.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.417 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.990823+0000", 2024-05-15T13:17:29.417 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4178000000000002e-05 2024-05-15T13:17:29.417 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.418 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.418 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.990830+0000", 2024-05-15T13:17:29.418 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5290000000000003e-06 2024-05-15T13:17:29.418 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.418 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.418 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.419 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.531010+0000", 2024-05-15T13:17:29.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.54018028200000001 2024-05-15T13:17:29.419 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.419 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.419 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.419 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.531017+0000", 2024-05-15T13:17:29.420 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9789999999999996e-06 2024-05-15T13:17:29.420 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.420 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.420 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.420 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.878483+0000", 2024-05-15T13:17:29.420 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3474650509999999 2024-05-15T13:17:29.420 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.420 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.420 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:29.421 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.878650+0000", 2024-05-15T13:17:29.421 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000167897 2024-05-15T13:17:29.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.421 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.421 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:17:29.421 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.878805+0000", 2024-05-15T13:17:29.421 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015463400000000001 2024-05-15T13:17:29.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.421 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.421 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:29.421 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.893292+0000", 2024-05-15T13:17:29.421 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014487458 2024-05-15T13:17:29.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.422 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.422 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.422 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.896924+0000", 2024-05-15T13:17:29.422 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0036316970000000001 2024-05-15T13:17:29.422 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.422 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.422 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.422 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.910956+0000", 2024-05-15T13:17:29.422 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014032205000000001 2024-05-15T13:17:29.422 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.422 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.422 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:29.422 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.911017+0000", 2024-05-15T13:17:29.423 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0924000000000001e-05 2024-05-15T13:17:29.423 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.423 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.423 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:29.423 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.913868+0000", 2024-05-15T13:17:29.423 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0028508750000000001 2024-05-15T13:17:29.423 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.423 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:29.423 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.423 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.423 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.423 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:146107 7.5 7:a81c3c36:::benchmark_data_smithi177_134484_object37042:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:17:29.423 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:40.061892+0000", 2024-05-15T13:17:29.423 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 109.208461098, 2024-05-15T13:17:29.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8648366159999998, 2024-05-15T13:17:29.424 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:29.424 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:29.424 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:29.424 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:29.424 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:29.424 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 146107 2024-05-15T13:17:29.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.424 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:29.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.424 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:29.424 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.061892+0000", 2024-05-15T13:17:29.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.425 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:29.425 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.061892+0000", 2024-05-15T13:17:29.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.425 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.425 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:29.425 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.061893+0000", 2024-05-15T13:17:29.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.57e-06 2024-05-15T13:17:29.425 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.425 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:29.425 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.061899+0000", 2024-05-15T13:17:29.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2780000000000001e-06 2024-05-15T13:17:29.426 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.426 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.426 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:29.426 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.061912+0000", 2024-05-15T13:17:29.426 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2865e-05 2024-05-15T13:17:29.426 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.426 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.426 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:29.426 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.061918+0000", 2024-05-15T13:17:29.426 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5400000000000001e-06 2024-05-15T13:17:29.426 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.426 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.426 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.427 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.061970+0000", 2024-05-15T13:17:29.427 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1683e-05 2024-05-15T13:17:29.427 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.427 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.427 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.427 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.061978+0000", 2024-05-15T13:17:29.427 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8140000000000001e-06 2024-05-15T13:17:29.427 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.427 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.427 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.427 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.531056+0000", 2024-05-15T13:17:29.427 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46907885199999999 2024-05-15T13:17:29.427 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.427 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.428 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.428 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.531066+0000", 2024-05-15T13:17:29.428 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6730000000000004e-06 2024-05-15T13:17:29.428 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.428 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.428 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.428 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.896961+0000", 2024-05-15T13:17:29.428 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3658945419999999 2024-05-15T13:17:29.428 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.428 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.429 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:29.429 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.897084+0000", 2024-05-15T13:17:29.429 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000123349 2024-05-15T13:17:29.429 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.429 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.429 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:17:29.429 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.897193+0000", 2024-05-15T13:17:29.429 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108665 2024-05-15T13:17:29.429 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.429 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.429 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:29.430 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.910909+0000", 2024-05-15T13:17:29.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013716425000000001 2024-05-15T13:17:29.430 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.430 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.430 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.430 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.913942+0000", 2024-05-15T13:17:29.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003032603 2024-05-15T13:17:29.430 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.430 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.430 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.430 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.923664+0000", 2024-05-15T13:17:29.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0097224900000000003 2024-05-15T13:17:29.431 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.431 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.431 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:29.431 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.923730+0000", 2024-05-15T13:17:29.431 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5875000000000002e-05 2024-05-15T13:17:29.431 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.431 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.431 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:29.431 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.926729+0000", 2024-05-15T13:17:29.431 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0029983919999999999 2024-05-15T13:17:29.431 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.431 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:29.431 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.431 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.431 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.432 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:146109 7.5 7:a65de73c:::benchmark_data_smithi177_134484_object37044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:17:29.432 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:40.073893+0000", 2024-05-15T13:17:29.432 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 109.196460114, 2024-05-15T13:17:29.432 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.856148006, 2024-05-15T13:17:29.432 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:17:29.432 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:17:29.432 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:17:29.432 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:17:29.432 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:29.432 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 146109 2024-05-15T13:17:29.432 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.432 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:17:29.432 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.432 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:17:29.433 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.073893+0000", 2024-05-15T13:17:29.433 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.433 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.433 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.433 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:17:29.433 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.073893+0000", 2024-05-15T13:17:29.433 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:17:29.433 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.433 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.433 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:17:29.433 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.073894+0000", 2024-05-15T13:17:29.433 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3680000000000001e-06 2024-05-15T13:17:29.433 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.433 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.433 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:17:29.434 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.073899+0000", 2024-05-15T13:17:29.434 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9910000000000002e-06 2024-05-15T13:17:29.434 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.434 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.434 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:17:29.434 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.073911+0000", 2024-05-15T13:17:29.434 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1781e-05 2024-05-15T13:17:29.434 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.434 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.434 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:17:29.434 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.073919+0000", 2024-05-15T13:17:29.434 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1559999999999995e-06 2024-05-15T13:17:29.434 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.435 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.435 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.435 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.074002+0000", 2024-05-15T13:17:29.435 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2902e-05 2024-05-15T13:17:29.435 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.435 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.435 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.435 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.074010+0000", 2024-05-15T13:17:29.435 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4200000000000007e-06 2024-05-15T13:17:29.435 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.435 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.435 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.435 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.531086+0000", 2024-05-15T13:17:29.436 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.45707555799999999 2024-05-15T13:17:29.436 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.436 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.436 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:17:29.436 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.531096+0000", 2024-05-15T13:17:29.436 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0284e-05 2024-05-15T13:17:29.436 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.436 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.436 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:17:29.436 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.900952+0000", 2024-05-15T13:17:29.436 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.369855265 2024-05-15T13:17:29.436 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.436 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.436 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:17:29.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.901099+0000", 2024-05-15T13:17:29.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014783600000000001 2024-05-15T13:17:29.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.437 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:17:29.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.901255+0000", 2024-05-15T13:17:29.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015536400000000001 2024-05-15T13:17:29.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.437 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:17:29.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.913959+0000", 2024-05-15T13:17:29.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012703743 2024-05-15T13:17:29.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.914003+0000", 2024-05-15T13:17:29.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4048999999999998e-05 2024-05-15T13:17:29.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.438 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:29.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.926805+0000", 2024-05-15T13:17:29.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012802167999999999 2024-05-15T13:17:29.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.438 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:17:29.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.926847+0000", 2024-05-15T13:17:29.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2052999999999999e-05 2024-05-15T13:17:29.439 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:17:29.439 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:17:29.439 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:17:29.439 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.930041+0000", 2024-05-15T13:17:29.439 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003194068 2024-05-15T13:17:29.439 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.439 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:29.439 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.439 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:17:29.439 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:17:29.439 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:17:29.440 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:29.875 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:29 smithi027 ceph-mon[39795]: pgmap v1846: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 31 active+clean; 1.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 495 op/s; 6883/77451 objects degraded (8.887%); 9609/77451 objects misplaced (12.407%); 14 MiB/s, 230 objects/s recovering 2024-05-15T13:17:29.875 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:29 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 6883/77451 objects degraded (8.887%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:29.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:29 smithi177 ceph-mon[44303]: pgmap v1846: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 31 active+clean; 1.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 495 op/s; 6883/77451 objects degraded (8.887%); 9609/77451 objects misplaced (12.407%); 14 MiB/s, 230 objects/s recovering 2024-05-15T13:17:29.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:29 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 6883/77451 objects degraded (8.887%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:30.003 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-05-15T13:17:30.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:29 smithi120 ceph-mon[43961]: pgmap v1846: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 31 active+clean; 1.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 495 op/s; 6883/77451 objects degraded (8.887%); 9609/77451 objects misplaced (12.407%); 14 MiB/s, 230 objects/s recovering 2024-05-15T13:17:30.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:29 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 6883/77451 objects degraded (8.887%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:30.329 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:17:30.370 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:17:30.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:30.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:17:30.491 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:17:30.525 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:30.555 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:17:30.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:30.563 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:17:30.664 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:17:30.692 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:30.719 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:17:30.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:30.723 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:17:30.825 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:17:30.852 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:30.873 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:17:30.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:30.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:17:30.979 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:17:31.035 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:31.869 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:17:31.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:31 smithi177 ceph-mon[44303]: pgmap v1847: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 530 op/s; 6883/74049 objects degraded (9.295%); 9044/74049 objects misplaced (12.214%); 17 MiB/s, 267 objects/s recovering 2024-05-15T13:17:31.955 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:31 smithi027 ceph-mon[39795]: pgmap v1847: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 530 op/s; 6883/74049 objects degraded (9.295%); 9044/74049 objects misplaced (12.214%); 17 MiB/s, 267 objects/s recovering 2024-05-15T13:17:32.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:31 smithi120 ceph-mon[43961]: pgmap v1847: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 530 op/s; 6883/74049 objects degraded (9.295%); 9044/74049 objects misplaced (12.214%); 17 MiB/s, 267 objects/s recovering 2024-05-15T13:17:32.053 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:17:32.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:32.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:17:32.172 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:17:32.200 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:32.219 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:17:32.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:32.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:17:32.324 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:17:32.353 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:32.378 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:17:32.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:32.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:17:32.483 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:17:32.509 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:32.533 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:17:32.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:32.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:17:32.637 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:17:32.673 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:32.710 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:17:32.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:32.713 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:17:32.814 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:17:32.892 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:17:33.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:33.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:17:33.136 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:17:33.320 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:33.343 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:17:33.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:33.346 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:17:33.447 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:17:33.533 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:33.660 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:33 smithi177 ceph-mon[44303]: pgmap v1848: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 361 op/s; 6521/73026 objects degraded (8.930%); 9044/73026 objects misplaced (12.385%); 9.1 MiB/s, 146 objects/s recovering 2024-05-15T13:17:33.920 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:33 smithi027 ceph-mon[39795]: pgmap v1848: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 361 op/s; 6521/73026 objects degraded (8.930%); 9044/73026 objects misplaced (12.385%); 9.1 MiB/s, 146 objects/s recovering 2024-05-15T13:17:33.925 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:17:33.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:33.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:17:34.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:33 smithi120 ceph-mon[43961]: pgmap v1848: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 361 op/s; 6521/73026 objects degraded (8.930%); 9044/73026 objects misplaced (12.385%); 9.1 MiB/s, 146 objects/s recovering 2024-05-15T13:17:34.053 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:17:34.079 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:34.098 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:17:34.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:34.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:17:34.203 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:17:34.239 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:34.266 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:17:34.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:34.271 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:17:34.372 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:17:34.399 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:34.423 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:17:34.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:34.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:17:34.527 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:17:34.641 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:34.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:34 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 6521/73026 objects degraded (8.930%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:35.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:34 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 6521/73026 objects degraded (8.930%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:35.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:34 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 6521/73026 objects degraded (8.930%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:35.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:35 smithi177 ceph-mon[44303]: pgmap v1849: 37 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 557 op/s; 6521/66042 objects degraded (9.874%); 8166/66042 objects misplaced (12.365%); 14 MiB/s, 218 objects/s recovering 2024-05-15T13:17:35.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:35 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2525042853' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:17:35.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:35 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:17:36.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:35 smithi027 ceph-mon[39795]: pgmap v1849: 37 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 557 op/s; 6521/66042 objects degraded (9.874%); 8166/66042 objects misplaced (12.365%); 14 MiB/s, 218 objects/s recovering 2024-05-15T13:17:36.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:35 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2525042853' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:17:36.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:35 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:17:36.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:35 smithi120 ceph-mon[43961]: pgmap v1849: 37 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 557 op/s; 6521/66042 objects degraded (9.874%); 8166/66042 objects misplaced (12.365%); 14 MiB/s, 218 objects/s recovering 2024-05-15T13:17:36.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:35 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2525042853' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:17:36.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:35 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:17:37.021 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:17:37.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:36 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:17:37.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:36 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:17:37.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:36 smithi027 ceph-mon[39795]: osdmap e713: 12 total, 12 up, 10 in 2024-05-15T13:17:37.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:36 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:17:37.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:36 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:17:37.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:36 smithi120 ceph-mon[43961]: osdmap e713: 12 total, 12 up, 10 in 2024-05-15T13:17:37.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:36 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:17:37.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:36 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:17:37.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:36 smithi177 ceph-mon[44303]: osdmap e713: 12 total, 12 up, 10 in 2024-05-15T13:17:38.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:37 smithi027 ceph-mon[39795]: pgmap v1851: 37 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 522 op/s; 6521/65610 objects degraded (9.939%); 8166/65610 objects misplaced (12.446%); 13 MiB/s, 204 objects/s recovering 2024-05-15T13:17:38.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:37 smithi027 ceph-mon[39795]: osdmap e714: 12 total, 12 up, 10 in 2024-05-15T13:17:38.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:37 smithi120 ceph-mon[43961]: pgmap v1851: 37 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 522 op/s; 6521/65610 objects degraded (9.939%); 8166/65610 objects misplaced (12.446%); 13 MiB/s, 204 objects/s recovering 2024-05-15T13:17:38.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:37 smithi120 ceph-mon[43961]: osdmap e714: 12 total, 12 up, 10 in 2024-05-15T13:17:38.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:37 smithi177 ceph-mon[44303]: pgmap v1851: 37 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 522 op/s; 6521/65610 objects degraded (9.939%); 8166/65610 objects misplaced (12.446%); 13 MiB/s, 204 objects/s recovering 2024-05-15T13:17:38.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:37 smithi177 ceph-mon[44303]: osdmap e714: 12 total, 12 up, 10 in 2024-05-15T13:17:38.893 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:17:38.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:38.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:17:39.004 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:17:39.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:38 smithi027 ceph-mon[39795]: osdmap e715: 12 total, 12 up, 10 in 2024-05-15T13:17:39.029 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:38 smithi120 ceph-mon[43961]: osdmap e715: 12 total, 12 up, 10 in 2024-05-15T13:17:39.031 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:39.049 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:17:39.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:39.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:17:39.154 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:17:39.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:38 smithi177 ceph-mon[44303]: osdmap e715: 12 total, 12 up, 10 in 2024-05-15T13:17:39.255 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:40.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:39 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:17:40.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:39 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:17:40.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:39 smithi027 ceph-mon[39795]: pgmap v1854: 37 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 610 op/s; 6197/62031 objects degraded (9.990%); 7809/62031 objects misplaced (12.589%); 15 MiB/s, 232 objects/s recovering 2024-05-15T13:17:40.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:39 smithi027 ceph-mon[39795]: 1.6 scrub starts 2024-05-15T13:17:40.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:39 smithi027 ceph-mon[39795]: 1.6 scrub ok 2024-05-15T13:17:40.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:39 smithi027 ceph-mon[39795]: osdmap e716: 12 total, 12 up, 10 in 2024-05-15T13:17:40.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:39 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 6197/62031 objects degraded (9.990%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:40.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:39 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:17:40.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:39 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:17:40.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:39 smithi120 ceph-mon[43961]: pgmap v1854: 37 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 610 op/s; 6197/62031 objects degraded (9.990%); 7809/62031 objects misplaced (12.589%); 15 MiB/s, 232 objects/s recovering 2024-05-15T13:17:40.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:39 smithi120 ceph-mon[43961]: 1.6 scrub starts 2024-05-15T13:17:40.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:39 smithi120 ceph-mon[43961]: 1.6 scrub ok 2024-05-15T13:17:40.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:39 smithi120 ceph-mon[43961]: osdmap e716: 12 total, 12 up, 10 in 2024-05-15T13:17:40.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:39 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 6197/62031 objects degraded (9.990%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:40.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:39 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:17:40.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:39 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:17:40.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:39 smithi177 ceph-mon[44303]: pgmap v1854: 37 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 610 op/s; 6197/62031 objects degraded (9.990%); 7809/62031 objects misplaced (12.589%); 15 MiB/s, 232 objects/s recovering 2024-05-15T13:17:40.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:39 smithi177 ceph-mon[44303]: 1.6 scrub starts 2024-05-15T13:17:40.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:39 smithi177 ceph-mon[44303]: 1.6 scrub ok 2024-05-15T13:17:40.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:39 smithi177 ceph-mon[44303]: osdmap e716: 12 total, 12 up, 10 in 2024-05-15T13:17:40.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:39 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 6197/62031 objects degraded (9.990%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:41.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:40 smithi027 ceph-mon[39795]: 1.10 scrub starts 2024-05-15T13:17:41.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:40 smithi027 ceph-mon[39795]: 1.10 scrub ok 2024-05-15T13:17:41.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:40 smithi120 ceph-mon[43961]: 1.10 scrub starts 2024-05-15T13:17:41.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:40 smithi120 ceph-mon[43961]: 1.10 scrub ok 2024-05-15T13:17:41.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:40 smithi177 ceph-mon[44303]: 1.10 scrub starts 2024-05-15T13:17:41.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:40 smithi177 ceph-mon[44303]: 1.10 scrub ok 2024-05-15T13:17:41.203 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:17:42.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:41 smithi120 ceph-mon[43961]: pgmap v1856: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 1.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 488 op/s; 6214/58917 objects degraded (10.547%); 7501/58917 objects misplaced (12.731%); 10 MiB/s, 256 objects/s recovering 2024-05-15T13:17:42.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:41 smithi120 ceph-mon[43961]: 7.5 scrub starts 2024-05-15T13:17:42.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:41 smithi120 ceph-mon[43961]: 7.5 scrub ok 2024-05-15T13:17:42.055 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:41 smithi027 ceph-mon[39795]: pgmap v1856: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 1.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 488 op/s; 6214/58917 objects degraded (10.547%); 7501/58917 objects misplaced (12.731%); 10 MiB/s, 256 objects/s recovering 2024-05-15T13:17:42.055 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:41 smithi027 ceph-mon[39795]: 7.5 scrub starts 2024-05-15T13:17:42.055 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:41 smithi027 ceph-mon[39795]: 7.5 scrub ok 2024-05-15T13:17:42.080 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:17:42.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:42.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:17:42.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:41 smithi177 ceph-mon[44303]: pgmap v1856: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 1.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 488 op/s; 6214/58917 objects degraded (10.547%); 7501/58917 objects misplaced (12.731%); 10 MiB/s, 256 objects/s recovering 2024-05-15T13:17:42.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:41 smithi177 ceph-mon[44303]: 7.5 scrub starts 2024-05-15T13:17:42.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:41 smithi177 ceph-mon[44303]: 7.5 scrub ok 2024-05-15T13:17:42.189 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:17:42.300 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:42.352 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:17:42.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:42.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:17:42.457 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:17:42.567 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:42.853 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:17:42.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:42.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:17:42.966 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:17:43.054 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:43.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:42 smithi177 ceph-mon[44303]: 1.1 scrub starts 2024-05-15T13:17:43.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:42 smithi177 ceph-mon[44303]: 1.1 scrub ok 2024-05-15T13:17:43.254 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:17:43.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:42 smithi027 ceph-mon[39795]: 1.1 scrub starts 2024-05-15T13:17:43.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:42 smithi027 ceph-mon[39795]: 1.1 scrub ok 2024-05-15T13:17:43.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:42 smithi120 ceph-mon[43961]: 1.1 scrub starts 2024-05-15T13:17:43.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:42 smithi120 ceph-mon[43961]: 1.1 scrub ok 2024-05-15T13:17:43.409 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:17:43.446 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:17:44.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:43 smithi177 ceph-mon[44303]: pgmap v1857: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 1.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 423 op/s; 5908/57984 objects degraded (10.189%); 7501/57984 objects misplaced (12.936%); 10 MiB/s, 233 objects/s recovering 2024-05-15T13:17:44.279 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:43 smithi027 ceph-mon[39795]: pgmap v1857: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 1.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 423 op/s; 5908/57984 objects degraded (10.189%); 7501/57984 objects misplaced (12.936%); 10 MiB/s, 233 objects/s recovering 2024-05-15T13:17:44.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:43 smithi120 ceph-mon[43961]: pgmap v1857: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 1.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 423 op/s; 5908/57984 objects degraded (10.189%); 7501/57984 objects misplaced (12.936%); 10 MiB/s, 233 objects/s recovering 2024-05-15T13:17:45.152 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:17:45.180 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:17:45.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:44 smithi177 ceph-mon[44303]: from='client.26219 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:17:45.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:44 smithi177 ceph-mon[44303]: 1.b scrub starts 2024-05-15T13:17:45.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:44 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 5908/57984 objects degraded (10.189%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:45.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:44 smithi177 ceph-mon[44303]: 1.b scrub ok 2024-05-15T13:17:45.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:45.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:17:45.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:44 smithi027 ceph-mon[39795]: from='client.26219 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:17:45.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:44 smithi027 ceph-mon[39795]: 1.b scrub starts 2024-05-15T13:17:45.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:44 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 5908/57984 objects degraded (10.189%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:45.196 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:44 smithi027 ceph-mon[39795]: 1.b scrub ok 2024-05-15T13:17:45.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:44 smithi120 ceph-mon[43961]: from='client.26219 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:17:45.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:44 smithi120 ceph-mon[43961]: 1.b scrub starts 2024-05-15T13:17:45.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:44 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 5908/57984 objects degraded (10.189%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:45.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:44 smithi120 ceph-mon[43961]: 1.b scrub ok 2024-05-15T13:17:45.298 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:17:45.325 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:45.346 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:17:45.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:45.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:17:45.452 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:17:45.554 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:45.733 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:17:45.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:45.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:17:45.841 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:17:45.847 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:45 smithi027 ceph-mon[39795]: 1.9 scrub starts 2024-05-15T13:17:45.868 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:45.889 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:17:45.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:45.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:17:45.995 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:17:46.107 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:45 smithi027 ceph-mon[39795]: pgmap v1858: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 1.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 627 op/s; 5908/50853 objects degraded (11.618%); 7017/50853 objects misplaced (13.799%); 12 MiB/s, 296 objects/s recovering 2024-05-15T13:17:46.107 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:45 smithi027 ceph-mon[39795]: 1.9 scrub ok 2024-05-15T13:17:46.107 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:45 smithi027 ceph-mon[39795]: 1.4 scrub starts 2024-05-15T13:17:46.107 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:45 smithi027 ceph-mon[39795]: 1.4 scrub ok 2024-05-15T13:17:46.120 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:46.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:45 smithi177 ceph-mon[44303]: 1.9 scrub starts 2024-05-15T13:17:46.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:45 smithi177 ceph-mon[44303]: pgmap v1858: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 1.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 627 op/s; 5908/50853 objects degraded (11.618%); 7017/50853 objects misplaced (13.799%); 12 MiB/s, 296 objects/s recovering 2024-05-15T13:17:46.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:45 smithi177 ceph-mon[44303]: 1.9 scrub ok 2024-05-15T13:17:46.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:45 smithi177 ceph-mon[44303]: 1.4 scrub starts 2024-05-15T13:17:46.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:45 smithi177 ceph-mon[44303]: 1.4 scrub ok 2024-05-15T13:17:46.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:45 smithi120 ceph-mon[43961]: 1.9 scrub starts 2024-05-15T13:17:46.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:45 smithi120 ceph-mon[43961]: pgmap v1858: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 1.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 627 op/s; 5908/50853 objects degraded (11.618%); 7017/50853 objects misplaced (13.799%); 12 MiB/s, 296 objects/s recovering 2024-05-15T13:17:46.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:45 smithi120 ceph-mon[43961]: 1.9 scrub ok 2024-05-15T13:17:46.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:45 smithi120 ceph-mon[43961]: 1.4 scrub starts 2024-05-15T13:17:46.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:45 smithi120 ceph-mon[43961]: 1.4 scrub ok 2024-05-15T13:17:46.815 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":1857,"stamp":"2024-05-15T13:17:42.550899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1267073592,"num_objects":19328,"num_object_clones":0,"num_object_copies":57984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5908,"num_objects_misplaced":7501,"num_objects_unfound":0,"num_objects_dirty":19328,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198800,"num_write_kb":6979969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47476,"num_bytes_recovered":2902523928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163604,"ondisk_log_size":163604,"up":111,"acting":107,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":121,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":13392864,"kb_used_data":6550356,"kb_used_omap":15,"kb_used_meta":6842224,"kb_avail":924058656,"statfs":{"total":959950356480,"available":946236063744,"internally_reserved":0,"allocated":6707564544,"data_stored":6702132560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15956,"internal_metadata":7006437804},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,3,5,2,1,1,0,15,10],"upper_bound":2048},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-166592512,"num_objects":-2542,"num_object_clones":0,"num_object_copies":-7626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-613,"num_objects_misplaced":-665,"num_objects_unfound":0,"num_objects_dirty":-2542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2542,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1400,"num_bytes_recovered":63045656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002321"},"pg_stats":[{"pgid":"1.14","version":"94'139","reported_seq":751,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-05-15T13:17:36.749376+0000","last_change":"2024-05-15T13:13:17.477074+0000","last_active":"2024-05-15T13:17:36.749376+0000","last_peered":"2024-05-15T13:17:36.749376+0000","last_clean":"2024-05-15T13:17:36.749376+0000","last_became_active":"2024-05-15T13:13:16.473818+0000","last_became_peered":"2024-05-15T13:13:16.473818+0000","last_unstale":"2024-05-15T13:17:36.749376+0000","last_undegraded":"2024-05-15T13:17:36.749376+0000","last_fullsized":"2024-05-15T13:17:36.749376+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.476961+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.476961+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1],"acting":[2,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":1326,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-05-15T13:17:36.720514+0000","last_change":"2024-05-15T13:13:19.224641+0000","last_active":"2024-05-15T13:17:36.720514+0000","last_peered":"2024-05-15T13:17:36.720514+0000","last_clean":"2024-05-15T13:17:36.720514+0000","last_became_active":"2024-05-15T13:13:11.361366+0000","last_became_peered":"2024-05-15T13:13:11.361366+0000","last_unstale":"2024-05-15T13:17:36.720514+0000","last_undegraded":"2024-05-15T13:17:36.720514+0000","last_fullsized":"2024-05-15T13:17:36.720514+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:19.224509+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:19.224509+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"716'12307","reported_seq":21525,"reported_epoch":716,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-05-15T13:17:39.298616+0000","last_change":"2024-05-15T13:14:21.034561+0000","last_active":"2024-05-15T13:17:39.298616+0000","last_peered":"2024-05-15T13:17:39.298616+0000","last_clean":"2024-05-15T13:14:07.430624+0000","last_became_active":"2024-05-15T13:14:09.931862+0000","last_became_peered":"2024-05-15T13:14:09.931862+0000","last_unstale":"2024-05-15T13:17:39.298616+0000","last_undegraded":"2024-05-15T13:14:21.029697+0000","last_fullsized":"2024-05-15T13:14:09.472302+0000","mapping_epoch":647,"log_start":"611'2300","ondisk_log_start":"611'2300","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80150528,"num_objects":1223,"num_object_clones":0,"num_object_copies":3669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1120,"num_objects_misplaced":1091,"num_objects_unfound":0,"num_objects_dirty":1223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12307,"num_write_kb":432960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4484,"num_bytes_recovered":290914304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,10],"acting":[0,10],"avail_no_missing":["0","5","10"],"object_location_counts":[{"shards":"0,5,10","objects":1091},{"shards":"0,10","objects":-988}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"94'139","reported_seq":1329,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-05-15T13:17:37.839304+0000","last_change":"2024-05-15T13:13:17.949012+0000","last_active":"2024-05-15T13:17:37.839304+0000","last_peered":"2024-05-15T13:17:37.839304+0000","last_clean":"2024-05-15T13:17:37.839304+0000","last_became_active":"2024-05-15T13:13:11.371643+0000","last_became_peered":"2024-05-15T13:13:11.371643+0000","last_unstale":"2024-05-15T13:17:37.839304+0000","last_undegraded":"2024-05-15T13:17:37.839304+0000","last_fullsized":"2024-05-15T13:17:37.839304+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.948900+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.948900+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:17:37.966879+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.2","version":"714'15915","reported_seq":16338,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-05-15T13:17:37.732573+0000","last_change":"2024-05-15T13:14:18.564785+0000","last_active":"2024-05-15T13:17:37.732573+0000","last_peered":"2024-05-15T13:17:37.732573+0000","last_clean":"2024-05-15T13:17:37.732573+0000","last_became_active":"2024-05-15T13:12:29.918894+0000","last_became_peered":"2024-05-15T13:12:29.918894+0000","last_unstale":"2024-05-15T13:17:37.732573+0000","last_undegraded":"2024-05-15T13:17:37.732573+0000","last_fullsized":"2024-05-15T13:17:37.732573+0000","mapping_epoch":599,"log_start":"632'5901","ondisk_log_start":"632'5901","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":4374,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79429632,"num_objects":1212,"num_object_clones":0,"num_object_copies":3636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12458,"num_write_kb":437440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2],"acting":[3,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"94'139","reported_seq":1085,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-05-15T13:17:38.755504+0000","last_change":"2024-05-15T13:14:54.577804+0000","last_active":"2024-05-15T13:17:38.755504+0000","last_peered":"2024-05-15T13:17:38.755504+0000","last_clean":"2024-05-15T13:17:38.755504+0000","last_became_active":"2024-05-15T13:14:54.577458+0000","last_became_peered":"2024-05-15T13:14:54.577458+0000","last_unstale":"2024-05-15T13:17:38.755504+0000","last_undegraded":"2024-05-15T13:17:38.755504+0000","last_fullsized":"2024-05-15T13:17:38.755504+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:18.049722+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:18.049722+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.3","version":"714'12176","reported_seq":12480,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-05-15T13:17:37.732658+0000","last_change":"2024-05-15T13:13:35.644249+0000","last_active":"2024-05-15T13:17:37.732658+0000","last_peered":"2024-05-15T13:17:37.732658+0000","last_clean":"2024-05-15T13:17:37.732658+0000","last_became_active":"2024-05-15T13:12:29.921897+0000","last_became_peered":"2024-05-15T13:12:29.921897+0000","last_unstale":"2024-05-15T13:17:37.732658+0000","last_undegraded":"2024-05-15T13:17:37.732658+0000","last_fullsized":"2024-05-15T13:17:37.732658+0000","mapping_epoch":599,"log_start":"611'2100","ondisk_log_start":"611'2100","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'5271","last_scrub_stamp":"2024-05-15T13:13:35.643996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:35.643996+0000","objects_scrubbed":5237,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84017152,"num_objects":1282,"num_object_clones":0,"num_object_copies":3846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12176,"num_write_kb":430656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"94'139","reported_seq":1332,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-05-15T13:17:38.754850+0000","last_change":"2024-05-15T13:17:38.671246+0000","last_active":"2024-05-15T13:17:38.754850+0000","last_peered":"2024-05-15T13:17:38.754850+0000","last_clean":"2024-05-15T13:17:38.754850+0000","last_became_active":"2024-05-15T13:13:11.373251+0000","last_became_peered":"2024-05-15T13:13:11.373251+0000","last_unstale":"2024-05-15T13:17:38.754850+0000","last_undegraded":"2024-05-15T13:17:38.754850+0000","last_fullsized":"2024-05-15T13:17:38.754850+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:38.671141+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:17:38.671141+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:51.578496+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.0","version":"715'12499","reported_seq":12785,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-05-15T13:17:38.582581+0000","last_change":"2024-05-15T13:16:24.441683+0000","last_active":"2024-05-15T13:17:38.582581+0000","last_peered":"2024-05-15T13:17:38.582581+0000","last_clean":"2024-05-15T13:17:38.582581+0000","last_became_active":"2024-05-15T13:12:29.924027+0000","last_became_peered":"2024-05-15T13:12:29.924027+0000","last_unstale":"2024-05-15T13:17:38.582581+0000","last_undegraded":"2024-05-15T13:17:38.582581+0000","last_fullsized":"2024-05-15T13:17:38.582581+0000","mapping_epoch":599,"log_start":"611'2400","ondisk_log_start":"611'2400","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"694'10259","last_scrub_stamp":"2024-05-15T13:16:24.441404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:16:24.441404+0000","objects_scrubbed":3481,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1000,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80281600,"num_objects":1225,"num_object_clones":0,"num_object_copies":3675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12499,"num_write_kb":439168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"94'139","reported_seq":1099,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-05-15T13:17:38.748187+0000","last_change":"2024-05-15T13:14:11.522798+0000","last_active":"2024-05-15T13:17:38.748187+0000","last_peered":"2024-05-15T13:17:38.748187+0000","last_clean":"2024-05-15T13:17:38.748187+0000","last_became_active":"2024-05-15T13:14:11.522490+0000","last_became_peered":"2024-05-15T13:14:11.522490+0000","last_unstale":"2024-05-15T13:17:38.748187+0000","last_undegraded":"2024-05-15T13:17:38.748187+0000","last_fullsized":"2024-05-15T13:17:38.748187+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.847198+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.847198+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"716'12450","reported_seq":29183,"reported_epoch":716,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-15T13:17:41.899645+0000","last_change":"2024-05-15T13:14:52.829919+0000","last_active":"2024-05-15T13:17:41.899645+0000","last_peered":"2024-05-15T13:17:41.899645+0000","last_clean":"2024-05-15T13:13:46.353513+0000","last_became_active":"2024-05-15T13:14:52.812683+0000","last_became_peered":"2024-05-15T13:14:52.812683+0000","last_unstale":"2024-05-15T13:17:41.899645+0000","last_undegraded":"2024-05-15T13:14:52.601890+0000","last_fullsized":"2024-05-15T13:14:52.453432+0000","mapping_epoch":658,"log_start":"611'2400","ondisk_log_start":"611'2400","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"633'5877","last_scrub_stamp":"2024-05-15T13:13:44.171370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:44.171370+0000","objects_scrubbed":5827,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1393,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75628544,"num_objects":1154,"num_object_clones":0,"num_object_copies":3462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12450,"num_write_kb":435328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5650,"num_bytes_recovered":370147328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3],"avail_no_missing":["10","5","11"],"object_location_counts":[{"shards":"1,3,10","objects":86},{"shards":"1,10,11","objects":704},{"shards":"5,10,11","objects":364}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"94'139","reported_seq":1101,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-05-15T13:17:38.754751+0000","last_change":"2024-05-15T13:15:43.615336+0000","last_active":"2024-05-15T13:17:38.754751+0000","last_peered":"2024-05-15T13:17:38.754751+0000","last_clean":"2024-05-15T13:17:38.754751+0000","last_became_active":"2024-05-15T13:15:43.614970+0000","last_became_peered":"2024-05-15T13:15:43.614970+0000","last_unstale":"2024-05-15T13:17:38.754751+0000","last_undegraded":"2024-05-15T13:17:38.754751+0000","last_fullsized":"2024-05-15T13:17:38.754751+0000","mapping_epoch":678,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:20.062928+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:20.062928+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,0],"acting":[9,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.7","version":"714'12592","reported_seq":13997,"reported_epoch":714,"state":"active+recovering+remapped","last_fresh":"2024-05-15T13:17:37.532435+0000","last_change":"2024-05-15T13:17:20.364816+0000","last_active":"2024-05-15T13:17:37.532435+0000","last_peered":"2024-05-15T13:17:37.532435+0000","last_clean":"2024-05-15T13:14:50.323746+0000","last_became_active":"2024-05-15T13:14:52.762068+0000","last_became_peered":"2024-05-15T13:14:52.762068+0000","last_unstale":"2024-05-15T13:17:37.532435+0000","last_undegraded":"2024-05-15T13:17:37.532435+0000","last_fullsized":"2024-05-15T13:17:37.532435+0000","mapping_epoch":658,"log_start":"611'2500","ondisk_log_start":"611'2500","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"633'6037","last_scrub_stamp":"2024-05-15T13:13:44.144120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:44.144120+0000","objects_scrubbed":5996,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1443,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81002496,"num_objects":1236,"num_object_clones":0,"num_object_copies":3708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5326,"num_objects_unfound":0,"num_objects_dirty":1236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12592,"num_write_kb":442496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":546,"num_bytes_recovered":33685504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,3],"acting":[1,8,3],"avail_no_missing":["1","3","8"],"object_location_counts":[{"shards":"1,3,8","objects":-4090}],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"94'139","reported_seq":1339,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-05-15T13:17:38.753931+0000","last_change":"2024-05-15T13:13:18.973897+0000","last_active":"2024-05-15T13:17:38.753931+0000","last_peered":"2024-05-15T13:17:38.753931+0000","last_clean":"2024-05-15T13:17:38.753931+0000","last_became_active":"2024-05-15T13:13:11.373548+0000","last_became_peered":"2024-05-15T13:13:11.373548+0000","last_unstale":"2024-05-15T13:17:38.753931+0000","last_undegraded":"2024-05-15T13:17:38.753931+0000","last_fullsized":"2024-05-15T13:17:38.753931+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:18.973644+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:18.973644+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"716'12536","reported_seq":25034,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-05-15T13:17:39.237638+0000","last_change":"2024-05-15T13:17:37.903772+0000","last_active":"2024-05-15T13:17:39.237638+0000","last_peered":"2024-05-15T13:17:39.237638+0000","last_clean":"2024-05-15T13:17:39.237638+0000","last_became_active":"2024-05-15T13:17:37.742339+0000","last_became_peered":"2024-05-15T13:17:37.742339+0000","last_unstale":"2024-05-15T13:17:39.237638+0000","last_undegraded":"2024-05-15T13:17:39.237638+0000","last_fullsized":"2024-05-15T13:17:39.237638+0000","mapping_epoch":714,"log_start":"611'2500","ondisk_log_start":"611'2500","created":599,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78643200,"num_objects":1200,"num_object_clones":0,"num_object_copies":3600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12536,"num_write_kb":439552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6119,"num_bytes_recovered":400949248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"94'139","reported_seq":1353,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-05-15T13:17:36.729437+0000","last_change":"2024-05-15T13:14:54.570374+0000","last_active":"2024-05-15T13:17:36.729437+0000","last_peered":"2024-05-15T13:17:36.729437+0000","last_clean":"2024-05-15T13:17:36.729437+0000","last_became_active":"2024-05-15T13:14:54.570004+0000","last_became_peered":"2024-05-15T13:14:54.570004+0000","last_unstale":"2024-05-15T13:17:36.729437+0000","last_undegraded":"2024-05-15T13:17:36.729437+0000","last_fullsized":"2024-05-15T13:17:36.729437+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:18.468889+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:18.468889+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":220,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"715'12439","reported_seq":26378,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-05-15T13:17:37.896301+0000","last_change":"2024-05-15T13:17:18.852934+0000","last_active":"2024-05-15T13:17:37.896301+0000","last_peered":"2024-05-15T13:17:37.896301+0000","last_clean":"2024-05-15T13:17:37.896301+0000","last_became_active":"2024-05-15T13:17:18.819383+0000","last_became_peered":"2024-05-15T13:17:18.819383+0000","last_unstale":"2024-05-15T13:17:37.896301+0000","last_undegraded":"2024-05-15T13:17:37.896301+0000","last_fullsized":"2024-05-15T13:17:37.896301+0000","mapping_epoch":705,"log_start":"611'2400","ondisk_log_start":"611'2400","created":599,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:17:37.966806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81592320,"num_objects":1245,"num_object_clones":0,"num_object_copies":3735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12439,"num_write_kb":437888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6842,"num_bytes_recovered":296615936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,0],"acting":[11,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"94'139","reported_seq":1324,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-05-15T13:17:37.858395+0000","last_change":"2024-05-15T13:16:36.359811+0000","last_active":"2024-05-15T13:17:37.858395+0000","last_peered":"2024-05-15T13:17:37.858395+0000","last_clean":"2024-05-15T13:17:37.858395+0000","last_became_active":"2024-05-15T13:06:46.426908+0000","last_became_peered":"2024-05-15T13:06:46.426908+0000","last_unstale":"2024-05-15T13:17:37.858395+0000","last_undegraded":"2024-05-15T13:17:37.858395+0000","last_fullsized":"2024-05-15T13:17:37.858395+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:16:36.359645+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:16:36.359645+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:17:52.241397+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"714'12011","reported_seq":12315,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-05-15T13:17:37.087502+0000","last_change":"2024-05-15T13:13:39.097208+0000","last_active":"2024-05-15T13:17:37.087502+0000","last_peered":"2024-05-15T13:17:37.087502+0000","last_clean":"2024-05-15T13:17:37.087502+0000","last_became_active":"2024-05-15T13:12:29.926405+0000","last_became_peered":"2024-05-15T13:12:29.926405+0000","last_unstale":"2024-05-15T13:17:37.087502+0000","last_undegraded":"2024-05-15T13:17:37.087502+0000","last_fullsized":"2024-05-15T13:17:37.087502+0000","mapping_epoch":599,"log_start":"611'2000","ondisk_log_start":"611'2000","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"629'5457","last_scrub_stamp":"2024-05-15T13:13:39.097042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:39.097042+0000","objects_scrubbed":5417,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1513,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78315520,"num_objects":1195,"num_object_clones":0,"num_object_copies":3585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12011,"num_write_kb":422592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"94'139","reported_seq":1318,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-05-15T13:17:38.754821+0000","last_change":"2024-05-15T13:13:20.984754+0000","last_active":"2024-05-15T13:17:38.754821+0000","last_peered":"2024-05-15T13:17:38.754821+0000","last_clean":"2024-05-15T13:17:38.754821+0000","last_became_active":"2024-05-15T13:06:46.424667+0000","last_became_peered":"2024-05-15T13:06:46.424667+0000","last_unstale":"2024-05-15T13:17:38.754821+0000","last_undegraded":"2024-05-15T13:17:38.754821+0000","last_fullsized":"2024-05-15T13:17:38.754821+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:20.984642+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:20.984642+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.f","version":"716'12339","reported_seq":12585,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-05-15T13:17:39.204368+0000","last_change":"2024-05-15T13:12:29.925178+0000","last_active":"2024-05-15T13:17:39.204368+0000","last_peered":"2024-05-15T13:17:39.204368+0000","last_clean":"2024-05-15T13:17:39.204368+0000","last_became_active":"2024-05-15T13:12:29.924959+0000","last_became_peered":"2024-05-15T13:12:29.924959+0000","last_unstale":"2024-05-15T13:17:39.204368+0000","last_undegraded":"2024-05-15T13:17:39.204368+0000","last_fullsized":"2024-05-15T13:17:39.204368+0000","mapping_epoch":599,"log_start":"611'2300","ondisk_log_start":"611'2300","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76349440,"num_objects":1165,"num_object_clones":0,"num_object_copies":3495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12339,"num_write_kb":432128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"94'139","reported_seq":1096,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-05-15T13:17:38.754790+0000","last_change":"2024-05-15T13:15:43.612283+0000","last_active":"2024-05-15T13:17:38.754790+0000","last_peered":"2024-05-15T13:17:38.754790+0000","last_clean":"2024-05-15T13:17:38.754790+0000","last_became_active":"2024-05-15T13:15:43.611913+0000","last_became_peered":"2024-05-15T13:15:43.611913+0000","last_unstale":"2024-05-15T13:17:38.754790+0000","last_undegraded":"2024-05-15T13:17:38.754790+0000","last_fullsized":"2024-05-15T13:17:38.754790+0000","mapping_epoch":678,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:27.843747+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:27.843747+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.c","version":"716'12704","reported_seq":26800,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-05-15T13:17:39.254936+0000","last_change":"2024-05-15T13:17:22.205596+0000","last_active":"2024-05-15T13:17:39.254936+0000","last_peered":"2024-05-15T13:17:39.254936+0000","last_clean":"2024-05-15T13:17:39.254936+0000","last_became_active":"2024-05-15T13:17:22.175461+0000","last_became_peered":"2024-05-15T13:17:22.175461+0000","last_unstale":"2024-05-15T13:17:39.254936+0000","last_undegraded":"2024-05-15T13:17:39.254936+0000","last_fullsized":"2024-05-15T13:17:39.254936+0000","mapping_epoch":708,"log_start":"611'2700","ondisk_log_start":"611'2700","created":599,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"641'6420","last_scrub_stamp":"2024-05-15T13:13:54.703460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:54.703460+0000","objects_scrubbed":6408,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":989,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78512128,"num_objects":1198,"num_object_clones":0,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12704,"num_write_kb":444864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6884,"num_bytes_recovered":451149824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,11],"acting":[6,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"714'12356","reported_seq":22364,"reported_epoch":714,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-05-15T13:17:37.595657+0000","last_change":"2024-05-15T13:14:20.686326+0000","last_active":"2024-05-15T13:17:37.595657+0000","last_peered":"2024-05-15T13:17:37.595657+0000","last_clean":"2024-05-15T13:14:07.355352+0000","last_became_active":"2024-05-15T13:14:09.930774+0000","last_became_peered":"2024-05-15T13:14:09.930774+0000","last_unstale":"2024-05-15T13:17:37.595657+0000","last_undegraded":"2024-05-15T13:14:20.671875+0000","last_fullsized":"2024-05-15T13:14:09.468986+0000","mapping_epoch":647,"log_start":"611'2300","ondisk_log_start":"611'2300","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'5339","last_scrub_stamp":"2024-05-15T13:13:35.290438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:35.290438+0000","objects_scrubbed":5292,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83623936,"num_objects":1276,"num_object_clones":0,"num_object_copies":3828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":809,"num_objects_misplaced":1084,"num_objects_unfound":0,"num_objects_dirty":1276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12356,"num_write_kb":436224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4851,"num_bytes_recovered":317325312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,10],"acting":[2,10],"avail_no_missing":["2","1","10"],"object_location_counts":[{"shards":"1,2,10","objects":1084},{"shards":"2,10","objects":-617}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"94'139","reported_seq":1002,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-05-15T13:17:37.732737+0000","last_change":"2024-05-15T13:13:18.190667+0000","last_active":"2024-05-15T13:17:37.732737+0000","last_peered":"2024-05-15T13:17:37.732737+0000","last_clean":"2024-05-15T13:17:37.732737+0000","last_became_active":"2024-05-15T13:13:16.477771+0000","last_became_peered":"2024-05-15T13:13:16.477771+0000","last_unstale":"2024-05-15T13:17:37.732737+0000","last_undegraded":"2024-05-15T13:17:37.732737+0000","last_fullsized":"2024-05-15T13:17:37.732737+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:18.190529+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:18.190529+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,1],"acting":[3,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"716'12560","reported_seq":12864,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-05-15T13:17:39.298094+0000","last_change":"2024-05-15T13:13:31.369534+0000","last_active":"2024-05-15T13:17:39.298094+0000","last_peered":"2024-05-15T13:17:39.298094+0000","last_clean":"2024-05-15T13:17:39.298094+0000","last_became_active":"2024-05-15T13:12:29.920800+0000","last_became_peered":"2024-05-15T13:12:29.920800+0000","last_unstale":"2024-05-15T13:17:39.298094+0000","last_undegraded":"2024-05-15T13:17:39.298094+0000","last_fullsized":"2024-05-15T13:17:39.298094+0000","mapping_epoch":599,"log_start":"611'2500","ondisk_log_start":"611'2500","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'5126","last_scrub_stamp":"2024-05-15T13:13:31.369265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:31.369265+0000","objects_scrubbed":5076,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1450,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79560704,"num_objects":1214,"num_object_clones":0,"num_object_copies":3642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12560,"num_write_kb":440768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"94'139","reported_seq":1097,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-05-15T13:17:38.748230+0000","last_change":"2024-05-15T13:13:16.831084+0000","last_active":"2024-05-15T13:17:38.748230+0000","last_peered":"2024-05-15T13:17:38.748230+0000","last_clean":"2024-05-15T13:17:38.748230+0000","last_became_active":"2024-05-15T13:13:16.472318+0000","last_became_peered":"2024-05-15T13:13:16.472318+0000","last_unstale":"2024-05-15T13:17:38.748230+0000","last_undegraded":"2024-05-15T13:17:38.748230+0000","last_fullsized":"2024-05-15T13:17:38.748230+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:16.830998+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:16.830998+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"716'12602","reported_seq":12917,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-05-15T13:17:41.090022+0000","last_change":"2024-05-15T13:13:47.375374+0000","last_active":"2024-05-15T13:17:41.090022+0000","last_peered":"2024-05-15T13:17:41.090022+0000","last_clean":"2024-05-15T13:17:41.090022+0000","last_became_active":"2024-05-15T13:12:29.923300+0000","last_became_peered":"2024-05-15T13:12:29.923300+0000","last_unstale":"2024-05-15T13:17:41.090022+0000","last_undegraded":"2024-05-15T13:17:41.090022+0000","last_fullsized":"2024-05-15T13:17:41.090022+0000","mapping_epoch":599,"log_start":"611'2600","ondisk_log_start":"611'2600","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"634'6127","last_scrub_stamp":"2024-05-15T13:13:47.375250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:47.375250+0000","objects_scrubbed":6127,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":586,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72613888,"num_objects":1108,"num_object_clones":0,"num_object_copies":3324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12602,"num_write_kb":438720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.c","version":"94'139","reported_seq":754,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-05-15T13:17:37.732775+0000","last_change":"2024-05-15T13:13:17.140519+0000","last_active":"2024-05-15T13:17:37.732775+0000","last_peered":"2024-05-15T13:17:37.732775+0000","last_clean":"2024-05-15T13:17:37.732775+0000","last_became_active":"2024-05-15T13:13:16.476972+0000","last_became_peered":"2024-05-15T13:13:16.476972+0000","last_unstale":"2024-05-15T13:17:37.732775+0000","last_undegraded":"2024-05-15T13:17:37.732775+0000","last_fullsized":"2024-05-15T13:17:37.732775+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.140398+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.140398+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"714'13014","reported_seq":25506,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-05-15T13:17:37.219248+0000","last_change":"2024-05-15T13:16:18.785284+0000","last_active":"2024-05-15T13:17:37.219248+0000","last_peered":"2024-05-15T13:17:37.219248+0000","last_clean":"2024-05-15T13:17:37.219248+0000","last_became_active":"2024-05-15T13:16:18.691546+0000","last_became_peered":"2024-05-15T13:16:18.691546+0000","last_unstale":"2024-05-15T13:17:37.219248+0000","last_undegraded":"2024-05-15T13:17:37.219248+0000","last_fullsized":"2024-05-15T13:17:37.219248+0000","mapping_epoch":693,"log_start":"611'3000","ondisk_log_start":"611'3000","created":599,"last_epoch_clean":694,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":624,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81985536,"num_objects":1251,"num_object_clones":0,"num_object_copies":3753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12585,"num_write_kb":442752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6108,"num_bytes_recovered":378404864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"94'139","reported_seq":740,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-05-15T13:17:36.720911+0000","last_change":"2024-05-15T13:13:17.391026+0000","last_active":"2024-05-15T13:17:36.720911+0000","last_peered":"2024-05-15T13:17:36.720911+0000","last_clean":"2024-05-15T13:17:36.720911+0000","last_became_active":"2024-05-15T13:13:16.478031+0000","last_became_peered":"2024-05-15T13:13:16.478031+0000","last_unstale":"2024-05-15T13:17:36.720911+0000","last_undegraded":"2024-05-15T13:17:36.720911+0000","last_fullsized":"2024-05-15T13:17:36.720911+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.390903+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.390903+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,11],"acting":[1,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"716'12186","reported_seq":24484,"reported_epoch":716,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-05-15T13:17:39.324074+0000","last_change":"2024-05-15T13:14:52.877732+0000","last_active":"2024-05-15T13:17:39.324074+0000","last_peered":"2024-05-15T13:17:39.324074+0000","last_clean":"2024-05-15T13:14:50.332505+0000","last_became_active":"2024-05-15T13:14:52.801033+0000","last_became_peered":"2024-05-15T13:14:52.801033+0000","last_unstale":"2024-05-15T13:17:39.324074+0000","last_undegraded":"2024-05-15T13:14:52.518661+0000","last_fullsized":"2024-05-15T13:14:52.462597+0000","mapping_epoch":658,"log_start":"611'2100","ondisk_log_start":"611'2100","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"630'5615","last_scrub_stamp":"2024-05-15T13:13:40.897467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:40.897467+0000","objects_scrubbed":5565,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1781,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74776600,"num_objects":1142,"num_object_clones":0,"num_object_copies":3426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1142,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12186,"num_write_kb":426433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5992,"num_bytes_recovered":363331608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[0,1],"avail_no_missing":["0","1","3"],"object_location_counts":[{"shards":"0,1","objects":1024}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.b","version":"94'139","reported_seq":1347,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-05-15T13:17:37.839615+0000","last_change":"2024-05-15T13:13:17.088630+0000","last_active":"2024-05-15T13:17:37.839615+0000","last_peered":"2024-05-15T13:17:37.839615+0000","last_clean":"2024-05-15T13:17:37.839615+0000","last_became_active":"2024-05-15T13:13:16.484252+0000","last_became_peered":"2024-05-15T13:13:16.484252+0000","last_unstale":"2024-05-15T13:17:37.839615+0000","last_undegraded":"2024-05-15T13:17:37.839615+0000","last_fullsized":"2024-05-15T13:17:37.839615+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.088496+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.088496+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:17:37.966829+0000","scrub_duration":194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"94'139","reported_seq":1329,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-05-15T13:17:37.858337+0000","last_change":"2024-05-15T13:16:33.288194+0000","last_active":"2024-05-15T13:17:37.858337+0000","last_peered":"2024-05-15T13:17:37.858337+0000","last_clean":"2024-05-15T13:17:37.858337+0000","last_became_active":"2024-05-15T13:13:16.472800+0000","last_became_peered":"2024-05-15T13:13:16.472800+0000","last_unstale":"2024-05-15T13:17:37.858337+0000","last_undegraded":"2024-05-15T13:17:37.858337+0000","last_fullsized":"2024-05-15T13:17:37.858337+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:16:33.288090+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:16:33.288090+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,8],"acting":[5,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"94'139","reported_seq":1105,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-05-15T13:17:38.755797+0000","last_change":"2024-05-15T13:13:17.150999+0000","last_active":"2024-05-15T13:17:38.755797+0000","last_peered":"2024-05-15T13:17:38.755797+0000","last_clean":"2024-05-15T13:17:38.755797+0000","last_became_active":"2024-05-15T13:13:16.476744+0000","last_became_peered":"2024-05-15T13:13:16.476744+0000","last_unstale":"2024-05-15T13:17:38.755797+0000","last_undegraded":"2024-05-15T13:17:38.755797+0000","last_fullsized":"2024-05-15T13:17:38.755797+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.150897+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.150897+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":232,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"94'139","reported_seq":1353,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-05-15T13:17:37.858393+0000","last_change":"2024-05-15T13:16:44.320180+0000","last_active":"2024-05-15T13:17:37.858393+0000","last_peered":"2024-05-15T13:17:37.858393+0000","last_clean":"2024-05-15T13:17:37.858393+0000","last_became_active":"2024-05-15T13:13:49.926687+0000","last_became_peered":"2024-05-15T13:13:49.926687+0000","last_unstale":"2024-05-15T13:17:37.858393+0000","last_undegraded":"2024-05-15T13:17:37.858393+0000","last_fullsized":"2024-05-15T13:17:37.858393+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:16:44.320095+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:16:44.320095+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:05.122264+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"94'139","reported_seq":1351,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-05-15T13:17:36.729341+0000","last_change":"2024-05-15T13:13:20.455813+0000","last_active":"2024-05-15T13:17:36.729341+0000","last_peered":"2024-05-15T13:17:36.729341+0000","last_clean":"2024-05-15T13:17:36.729341+0000","last_became_active":"2024-05-15T13:13:16.478488+0000","last_became_peered":"2024-05-15T13:13:16.478488+0000","last_unstale":"2024-05-15T13:17:36.729341+0000","last_undegraded":"2024-05-15T13:17:36.729341+0000","last_fullsized":"2024-05-15T13:17:36.729341+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:20.455652+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:20.455652+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1266483224,"num_objects":19326,"num_object_clones":0,"num_object_copies":57978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5908,"num_objects_misplaced":7501,"num_objects_unfound":0,"num_objects_dirty":19326,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198800,"num_write_kb":6979969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47476,"num_bytes_recovered":2902523928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6662078464,"data_stored":6662062176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160685,"ondisk_log_size":160685,"up":48,"acting":44,"num_store_stats":10},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057027,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1501760,"kb_used_data":911088,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":92243392,"statfs":{"total":95995035648,"available":94457233408,"internally_reserved":0,"allocated":932954112,"data_stored":932414027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":604830150},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351368,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1649412,"kb_used_data":974076,"kb_used_omap":1,"kb_used_meta":675326,"kb_avail":92095740,"statfs":{"total":95995035648,"available":94306037760,"internally_reserved":0,"allocated":997453824,"data_stored":996909423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":691534272},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514893,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1241096,"kb_used_data":438012,"kb_used_omap":1,"kb_used_meta":803070,"kb_avail":92504056,"statfs":{"total":95995035648,"available":94724153344,"internally_reserved":0,"allocated":448524288,"data_stored":447979887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":822344128},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874642,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":646796,"kb_used_data":228284,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93098356,"statfs":{"total":95995035648,"available":95332716544,"internally_reserved":0,"allocated":233762816,"data_stored":233218415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":240518168982,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430621,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":859028,"kb_used_data":155904,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":92886124,"statfs":{"total":95995035648,"available":95115390976,"internally_reserved":0,"allocated":159645696,"data_stored":159097223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":719976902},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575285,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1631528,"kb_used_data":802076,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":92113624,"statfs":{"total":95995035648,"available":94324350976,"internally_reserved":0,"allocated":821325824,"data_stored":820790362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":849344965},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738815,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1212692,"kb_used_data":459456,"kb_used_omap":1,"kb_used_meta":753086,"kb_avail":92532460,"statfs":{"total":95995035648,"available":94753239040,"internally_reserved":0,"allocated":470482944,"data_stored":469934471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":771160517},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1,0,15,10],"upper_bound":2048},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411762,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1354860,"kb_used_data":814176,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":92390292,"statfs":{"total":95995035648,"available":94607659008,"internally_reserved":0,"allocated":833716224,"data_stored":833181282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150123,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1690248,"kb_used_data":792764,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":92054904,"statfs":{"total":95995035648,"available":94264221696,"internally_reserved":0,"allocated":811790336,"data_stored":811242627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":919009735},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986600,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692258,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1605444,"kb_used_data":974520,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92139708,"statfs":{"total":95995035648,"available":94351060992,"internally_reserved":0,"allocated":997908480,"data_stored":997364843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":468127744,"data_stored":468123672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":817434624,"data_stored":817430552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":825556992,"data_stored":825556992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":809701376,"data_stored":809697304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":993394688,"data_stored":993394688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":156831744,"data_stored":156827672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":227082240,"data_stored":227082240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":439418880,"data_stored":439418880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":993591296,"data_stored":993591296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":930938880,"data_stored":930938880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:17:46.820 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:17:46.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:46.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:17:46.929 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:17:46.963 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:47.008 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:17:47.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:47.022 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:17:47.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:46 smithi027 ceph-mon[39795]: 1.a scrub starts 2024-05-15T13:17:47.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:46 smithi027 ceph-mon[39795]: 1.a scrub ok 2024-05-15T13:17:47.123 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:17:47.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:46 smithi120 ceph-mon[43961]: 1.a scrub starts 2024-05-15T13:17:47.148 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:46 smithi120 ceph-mon[43961]: 1.a scrub ok 2024-05-15T13:17:47.149 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:47.171 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:17:47.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:47.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:17:47.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:46 smithi177 ceph-mon[44303]: 1.a scrub starts 2024-05-15T13:17:47.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:46 smithi177 ceph-mon[44303]: 1.a scrub ok 2024-05-15T13:17:47.275 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:17:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:48.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:47 smithi177 ceph-mon[44303]: pgmap v1859: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 1.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 488 op/s; 5908/50388 objects degraded (11.725%); 7017/50388 objects misplaced (13.926%); 6.6 MiB/s, 204 objects/s recovering 2024-05-15T13:17:48.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:47 smithi027 ceph-mon[39795]: pgmap v1859: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 1.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 488 op/s; 5908/50388 objects degraded (11.725%); 7017/50388 objects misplaced (13.926%); 6.6 MiB/s, 204 objects/s recovering 2024-05-15T13:17:48.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:47 smithi120 ceph-mon[43961]: pgmap v1859: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 1.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 488 op/s; 5908/50388 objects degraded (11.725%); 7017/50388 objects misplaced (13.926%); 6.6 MiB/s, 204 objects/s recovering 2024-05-15T13:17:48.980 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:17:48.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:48.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:17:48.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:48 smithi027 ceph-mon[39795]: osdmap e717: 12 total, 12 up, 10 in 2024-05-15T13:17:48.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:48 smithi027 ceph-mon[39795]: 7.0 scrub starts 2024-05-15T13:17:48.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:48 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:17:48.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:48 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:17:48.985 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:48 smithi027 ceph-mon[39795]: 7.0 scrub ok 2024-05-15T13:17:48.985 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:48 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:17:48.985 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:48 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:17:48.985 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:48 smithi027 ceph-mon[39795]: osdmap e718: 12 total, 12 up, 10 in 2024-05-15T13:17:49.083 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:17:49.110 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:49.132 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:17:49.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:49.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:17:49.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:48 smithi177 ceph-mon[44303]: osdmap e717: 12 total, 12 up, 10 in 2024-05-15T13:17:49.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:48 smithi177 ceph-mon[44303]: 7.0 scrub starts 2024-05-15T13:17:49.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:48 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:17:49.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:48 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:17:49.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:48 smithi177 ceph-mon[44303]: 7.0 scrub ok 2024-05-15T13:17:49.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:48 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:17:49.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:48 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:17:49.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:48 smithi177 ceph-mon[44303]: osdmap e718: 12 total, 12 up, 10 in 2024-05-15T13:17:49.237 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:17:49.263 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:48 smithi120 ceph-mon[43961]: osdmap e717: 12 total, 12 up, 10 in 2024-05-15T13:17:49.263 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:48 smithi120 ceph-mon[43961]: 7.0 scrub starts 2024-05-15T13:17:49.263 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:48 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:17:49.263 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:48 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:17:49.263 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:48 smithi120 ceph-mon[43961]: 7.0 scrub ok 2024-05-15T13:17:49.263 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:48 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:17:49.263 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:48 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:17:49.263 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:48 smithi120 ceph-mon[43961]: osdmap e718: 12 total, 12 up, 10 in 2024-05-15T13:17:49.265 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:49.286 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:17:49.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:49.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:17:49.389 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:17:49.500 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:49.521 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:17:49.521 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:17:49.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.521 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:204353 7.6 7:6123e62a:::benchmark_data_smithi177_134484_object95288:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e718)", 2024-05-15T13:17:49.521 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:17:49.464249+0000", 2024-05-15T13:17:49.522 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.039479095999999998, 2024-05-15T13:17:49.522 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.039523975000000003, 2024-05-15T13:17:49.522 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:17:49.522 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:17:49.522 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:17:49.522 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:17:49.522 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:49.522 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 204353 2024-05-15T13:17:49.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.522 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:17:49.522 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.522 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:17:49.522 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.464249+0000", 2024-05-15T13:17:49.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:17:49.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:17:49.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.464249+0000", 2024-05-15T13:17:49.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:17:49.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:17:49.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.464251+0000", 2024-05-15T13:17:49.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.652e-06 2024-05-15T13:17:49.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:17:49.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.464258+0000", 2024-05-15T13:17:49.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1969999999999999e-06 2024-05-15T13:17:49.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:17:49.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.464272+0000", 2024-05-15T13:17:49.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3952e-05 2024-05-15T13:17:49.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:17:49.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.464282+0000", 2024-05-15T13:17:49.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9629999999999993e-06 2024-05-15T13:17:49.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:17:49.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.466299+0000", 2024-05-15T13:17:49.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020172250000000001 2024-05-15T13:17:49.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:17:49.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.466450+0000", 2024-05-15T13:17:49.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000150414 2024-05-15T13:17:49.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,10", 2024-05-15T13:17:49.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.466604+0000", 2024-05-15T13:17:49.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015406299999999999 2024-05-15T13:17:49.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:17:49.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.472183+0000", 2024-05-15T13:17:49.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0055792760000000002 2024-05-15T13:17:49.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:49.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.476737+0000", 2024-05-15T13:17:49.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0045540429999999998 2024-05-15T13:17:49.526 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:17:49.526 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:17:49.526 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:17:49.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.527 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:204362 7.9 7:9fb2db98:::benchmark_data_smithi177_134484_object95297:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e718)", 2024-05-15T13:17:49.527 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:17:49.489879+0000", 2024-05-15T13:17:49.527 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.013849267, 2024-05-15T13:17:49.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014013283, 2024-05-15T13:17:49.527 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:17:49.527 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:17:49.527 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:17:49.527 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:17:49.527 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:49.527 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 204362 2024-05-15T13:17:49.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.527 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:17:49.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:17:49.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.489879+0000", 2024-05-15T13:17:49.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:17:49.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:17:49.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.489879+0000", 2024-05-15T13:17:49.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:17:49.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:17:49.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.489880+0000", 2024-05-15T13:17:49.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.147e-06 2024-05-15T13:17:49.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:17:49.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.489889+0000", 2024-05-15T13:17:49.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2810000000000008e-06 2024-05-15T13:17:49.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:17:49.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.489903+0000", 2024-05-15T13:17:49.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4219999999999999e-05 2024-05-15T13:17:49.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:17:49.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.489913+0000", 2024-05-15T13:17:49.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6420000000000002e-06 2024-05-15T13:17:49.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:17:49.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.495079+0000", 2024-05-15T13:17:49.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0051662829999999998 2024-05-15T13:17:49.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:17:49.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.495220+0000", 2024-05-15T13:17:49.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000141111 2024-05-15T13:17:49.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,11", 2024-05-15T13:17:49.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.495348+0000", 2024-05-15T13:17:49.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000128033 2024-05-15T13:17:49.531 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:17:49.531 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:17:49.531 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:17:49.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.531 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:204363 7.6 7:64639278:::benchmark_data_smithi177_134484_object95298:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e718)", 2024-05-15T13:17:49.531 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:17:49.489928+0000", 2024-05-15T13:17:49.531 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.013800516000000001, 2024-05-15T13:17:49.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014047179999999999, 2024-05-15T13:17:49.531 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:17:49.531 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:17:49.531 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:17:49.531 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:17:49.531 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:49.532 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 204363 2024-05-15T13:17:49.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.532 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:17:49.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:17:49.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.489928+0000", 2024-05-15T13:17:49.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:17:49.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:17:49.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.489928+0000", 2024-05-15T13:17:49.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:17:49.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:17:49.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.489929+0000", 2024-05-15T13:17:49.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0999999999999998e-07 2024-05-15T13:17:49.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:17:49.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.489933+0000", 2024-05-15T13:17:49.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5199999999999999e-06 2024-05-15T13:17:49.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:17:49.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.489941+0000", 2024-05-15T13:17:49.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1640000000000001e-06 2024-05-15T13:17:49.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:17:49.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.489951+0000", 2024-05-15T13:17:49.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1719999999999996e-06 2024-05-15T13:17:49.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:17:49.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.490179+0000", 2024-05-15T13:17:49.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000228898 2024-05-15T13:17:49.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:17:49.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.490312+0000", 2024-05-15T13:17:49.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013236199999999999 2024-05-15T13:17:49.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,10", 2024-05-15T13:17:49.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.490452+0000", 2024-05-15T13:17:49.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000140025 2024-05-15T13:17:49.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:17:49.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.500988+0000", 2024-05-15T13:17:49.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010536685 2024-05-15T13:17:49.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:17:49.536 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.501005+0000", 2024-05-15T13:17:49.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6300999999999999e-05 2024-05-15T13:17:49.536 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:17:49.536 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:17:49.536 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:17:49.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.536 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:204364 7.9 7:9d5e6e7a:::benchmark_data_smithi177_134484_object95299:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e718)", 2024-05-15T13:17:49.536 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:17:49.489976+0000", 2024-05-15T13:17:49.536 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.013752442, 2024-05-15T13:17:49.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014090465, 2024-05-15T13:17:49.536 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:17:49.536 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:17:49.537 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:17:49.537 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:17:49.537 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:17:49.537 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 204364 2024-05-15T13:17:49.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.537 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:17:49.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:17:49.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.489976+0000", 2024-05-15T13:17:49.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:17:49.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:17:49.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.489976+0000", 2024-05-15T13:17:49.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:17:49.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:17:49.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.489977+0000", 2024-05-15T13:17:49.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1999999999999999e-07 2024-05-15T13:17:49.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:17:49.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.489980+0000", 2024-05-15T13:17:49.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4089999999999999e-06 2024-05-15T13:17:49.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:17:49.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.489990+0000", 2024-05-15T13:17:49.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0142e-05 2024-05-15T13:17:49.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:17:49.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.489997+0000", 2024-05-15T13:17:49.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5019999999999998e-06 2024-05-15T13:17:49.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:17:49.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.500541+0000", 2024-05-15T13:17:49.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010543848 2024-05-15T13:17:49.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:17:49.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.500666+0000", 2024-05-15T13:17:49.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012563599999999999 2024-05-15T13:17:49.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:17:49.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:17:49.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 8,11", 2024-05-15T13:17:49.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:49.500793+0000", 2024-05-15T13:17:49.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000126919 2024-05-15T13:17:49.540 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:17:49.540 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:17:49.540 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:17:49.540 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:17:49.540 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:17:49.540 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 4 2024-05-15T13:17:49.541 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:17:49.543 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:17:49.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:49.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:17:49.653 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:17:49.679 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:49.716 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:17:49.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:49.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:17:49.818 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:17:49.820 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:17:49.849 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:49.875 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:17:49.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:49.879 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:17:49.980 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:17:50.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:49 smithi120 ceph-mon[43961]: pgmap v1861: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 964 MiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 518 op/s; 5566/46275 objects degraded (12.028%); 6717/46275 objects misplaced (14.515%); 8.9 MiB/s, 232 objects/s recovering 2024-05-15T13:17:50.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:49 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:17:50.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:49 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:17:50.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:49 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 5566/46275 objects degraded (12.028%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:50.090 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:50.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:49 smithi177 ceph-mon[44303]: pgmap v1861: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 964 MiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 518 op/s; 5566/46275 objects degraded (12.028%); 6717/46275 objects misplaced (14.515%); 8.9 MiB/s, 232 objects/s recovering 2024-05-15T13:17:50.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:49 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:17:50.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:49 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:17:50.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:49 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 5566/46275 objects degraded (12.028%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:50.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:49 smithi027 ceph-mon[39795]: pgmap v1861: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 964 MiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 518 op/s; 5566/46275 objects degraded (12.028%); 6717/46275 objects misplaced (14.515%); 8.9 MiB/s, 232 objects/s recovering 2024-05-15T13:17:50.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:49 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:17:50.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:49 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:17:50.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:49 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 5566/46275 objects degraded (12.028%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:50.324 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:17:50.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:50.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:17:50.357 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:17:50.430 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:17:50.459 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:50.482 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:17:50.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:50.486 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:17:50.586 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:17:50.614 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:50.634 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:17:50.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:50.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:17:50.737 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:17:50.819 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:50.861 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-05-15T13:17:50.931 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:17:50.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:50.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:17:51.042 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:17:51.069 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:51.072 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:50 smithi027 ceph-mon[39795]: 7.a scrub starts 2024-05-15T13:17:51.072 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:50 smithi027 ceph-mon[39795]: 7.a scrub ok 2024-05-15T13:17:51.090 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:17:51.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:51.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:17:51.195 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:17:51.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:50 smithi120 ceph-mon[43961]: 7.a scrub starts 2024-05-15T13:17:51.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:50 smithi120 ceph-mon[43961]: 7.a scrub ok 2024-05-15T13:17:51.350 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:51.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:50 smithi177 ceph-mon[44303]: 7.a scrub starts 2024-05-15T13:17:51.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:50 smithi177 ceph-mon[44303]: 7.a scrub ok 2024-05-15T13:17:51.506 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:17:51.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:51.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:17:51.613 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:17:51.640 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:51.661 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:17:51.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:51.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:17:51.765 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:17:51.793 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:51.812 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:17:51.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:51.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:17:51.916 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:17:51.948 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:51.961 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:51 smithi027 ceph-mon[39795]: pgmap v1863: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 893 MiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 630 op/s; 5453/42849 objects degraded (12.726%); 6514/42849 objects misplaced (15.202%); 9.0 MiB/s, 227 objects/s recovering 2024-05-15T13:17:51.978 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:17:51.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:51.982 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:17:52.083 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:17:52.189 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:52.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:51 smithi120 ceph-mon[43961]: pgmap v1863: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 893 MiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 630 op/s; 5453/42849 objects degraded (12.726%); 6514/42849 objects misplaced (15.202%); 9.0 MiB/s, 227 objects/s recovering 2024-05-15T13:17:52.412 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:17:52.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:52.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:17:52.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:51 smithi177 ceph-mon[44303]: pgmap v1863: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 893 MiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 630 op/s; 5453/42849 objects degraded (12.726%); 6514/42849 objects misplaced (15.202%); 9.0 MiB/s, 227 objects/s recovering 2024-05-15T13:17:52.524 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:17:52.551 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:52.573 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:17:52.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:52.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:17:52.680 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:17:52.778 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:54.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:54 smithi027 ceph-mon[39795]: pgmap v1864: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 872 MiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 375 op/s; 5126/41829 objects degraded (12.255%); 6514/41829 objects misplaced (15.573%); 6.5 MiB/s, 133 objects/s recovering 2024-05-15T13:17:54.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:54 smithi120 ceph-mon[43961]: pgmap v1864: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 872 MiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 375 op/s; 5126/41829 objects degraded (12.255%); 6514/41829 objects misplaced (15.573%); 6.5 MiB/s, 133 objects/s recovering 2024-05-15T13:17:54.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:54 smithi177 ceph-mon[44303]: pgmap v1864: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 872 MiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 375 op/s; 5126/41829 objects degraded (12.255%); 6514/41829 objects misplaced (15.573%); 6.5 MiB/s, 133 objects/s recovering 2024-05-15T13:17:54.475 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:17:54.753 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:17:54.806 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:17:54.853 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:17:54.904 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:17:54.942 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:17:54.977 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:17:55.060 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:17:55.078 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:17:55.145 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:17:55.209 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:17:55.259 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:17:55.274 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:17:55.274 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:17:55.274 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:17:55.274 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:17:55.274 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:17:55.274 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:17:55.274 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:17:55.274 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:17:55.274 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:17:55.274 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:17:55.274 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:17:55.275 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:17:55.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:55 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:17:55.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:55 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:17:55.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:55 smithi027 ceph-mon[39795]: osdmap e719: 12 total, 12 up, 10 in 2024-05-15T13:17:55.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:55 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 5126/41829 objects degraded (12.255%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:55.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:55 smithi027 ceph-mon[39795]: osdmap e720: 12 total, 12 up, 10 in 2024-05-15T13:17:55.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:55 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3858657750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:17:55.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:55 smithi027 ceph-mon[39795]: pgmap v1867: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 730 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 624 op/s; 5126/35019 objects degraded (14.638%); 5979/35019 objects misplaced (17.074%); 9.3 MiB/s, 188 objects/s recovering 2024-05-15T13:17:55.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:55 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:17:55.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:55 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:17:55.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:55 smithi120 ceph-mon[43961]: osdmap e719: 12 total, 12 up, 10 in 2024-05-15T13:17:55.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:55 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 5126/41829 objects degraded (12.255%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:55.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:55 smithi120 ceph-mon[43961]: osdmap e720: 12 total, 12 up, 10 in 2024-05-15T13:17:55.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:55 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3858657750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:17:55.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:55 smithi120 ceph-mon[43961]: pgmap v1867: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 730 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 624 op/s; 5126/35019 objects degraded (14.638%); 5979/35019 objects misplaced (17.074%); 9.3 MiB/s, 188 objects/s recovering 2024-05-15T13:17:55.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:55 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:17:55.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:55 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:17:55.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:55 smithi177 ceph-mon[44303]: osdmap e719: 12 total, 12 up, 10 in 2024-05-15T13:17:55.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:55 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 5126/41829 objects degraded (12.255%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:55.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:55 smithi177 ceph-mon[44303]: osdmap e720: 12 total, 12 up, 10 in 2024-05-15T13:17:55.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:55 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3858657750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:17:55.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:55 smithi177 ceph-mon[44303]: pgmap v1867: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 730 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 624 op/s; 5126/35019 objects degraded (14.638%); 5979/35019 objects misplaced (17.074%); 9.3 MiB/s, 188 objects/s recovering 2024-05-15T13:17:56.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:56 smithi177 ceph-mon[44303]: 1.8 scrub starts 2024-05-15T13:17:56.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:56 smithi177 ceph-mon[44303]: 1.8 scrub ok 2024-05-15T13:17:56.527 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:56 smithi027 ceph-mon[39795]: 1.8 scrub starts 2024-05-15T13:17:56.527 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:56 smithi027 ceph-mon[39795]: 1.8 scrub ok 2024-05-15T13:17:56.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:56 smithi120 ceph-mon[43961]: 1.8 scrub starts 2024-05-15T13:17:56.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:56 smithi120 ceph-mon[43961]: 1.8 scrub ok 2024-05-15T13:17:57.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:57 smithi177 ceph-mon[44303]: pgmap v1868: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 719 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 487 op/s; 5126/34497 objects degraded (14.859%); 5979/34497 objects misplaced (17.332%); 7.3 MiB/s, 147 objects/s recovering 2024-05-15T13:17:57.527 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:57 smithi027 ceph-mon[39795]: pgmap v1868: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 719 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 487 op/s; 5126/34497 objects degraded (14.859%); 5979/34497 objects misplaced (17.332%); 7.3 MiB/s, 147 objects/s recovering 2024-05-15T13:17:57.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:57 smithi120 ceph-mon[43961]: pgmap v1868: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 719 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 487 op/s; 5126/34497 objects degraded (14.859%); 5979/34497 objects misplaced (17.332%); 7.3 MiB/s, 147 objects/s recovering 2024-05-15T13:17:58.151 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:17:58.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:58.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:17:58.195 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:17:58.229 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:17:58.259 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:17:58.288 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:58.307 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:17:58.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:58.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:17:58.411 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:17:58.437 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:58.457 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:17:58.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:58.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:17:58.561 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:17:58.609 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:58.637 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:17:58.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:58.643 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:17:58.744 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:17:58.770 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:58.791 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:17:58.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:58.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:17:58.896 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:17:59.002 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:59.300 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:17:59.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:59.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:17:59.408 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:17:59.444 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:59.470 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:17:59.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:59.478 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:17:59.579 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:17:59.622 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:59.642 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:17:59.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:17:59.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:17:59.746 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:17:59.802 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:17:59.820 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:17:59.897 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:59 smithi027 ceph-mon[39795]: pgmap v1869: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 624 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 537 op/s; 4721/29952 objects degraded (15.762%); 5651/29952 objects misplaced (18.867%); 9.6 MiB/s, 153 objects/s recovering 2024-05-15T13:17:59.897 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:17:59 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 4721/29952 objects degraded (15.762%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:17:59.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:59 smithi177 ceph-mon[44303]: pgmap v1869: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 624 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 537 op/s; 4721/29952 objects degraded (15.762%); 5651/29952 objects misplaced (18.867%); 9.6 MiB/s, 153 objects/s recovering 2024-05-15T13:17:59.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:17:59 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 4721/29952 objects degraded (15.762%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:18:00.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:59 smithi120 ceph-mon[43961]: pgmap v1869: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 624 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 537 op/s; 4721/29952 objects degraded (15.762%); 5651/29952 objects misplaced (18.867%); 9.6 MiB/s, 153 objects/s recovering 2024-05-15T13:18:00.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:17:59 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 4721/29952 objects degraded (15.762%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:18:01.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:01 smithi177 ceph-mon[44303]: pgmap v1870: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 541 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 661 op/s; 4731/25944 objects degraded (18.235%); 5489/25944 objects misplaced (21.157%); 9.5 MiB/s, 152 objects/s recovering 2024-05-15T13:18:02.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:01 smithi027 ceph-mon[39795]: pgmap v1870: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 541 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 661 op/s; 4731/25944 objects degraded (18.235%); 5489/25944 objects misplaced (21.157%); 9.5 MiB/s, 152 objects/s recovering 2024-05-15T13:18:02.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:01 smithi120 ceph-mon[43961]: pgmap v1870: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 541 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 661 op/s; 4731/25944 objects degraded (18.235%); 5489/25944 objects misplaced (21.157%); 9.5 MiB/s, 152 objects/s recovering 2024-05-15T13:18:02.754 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:18:02.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:02.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:18:02.864 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:18:02.902 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:02.945 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:18:02.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:02.952 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:18:03.053 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:18:03.135 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:03.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:03 smithi177 ceph-mon[44303]: pgmap v1871: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 516 MiB data, 11 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 523 op/s; 4333/24747 objects degraded (17.509%); 5489/24747 objects misplaced (22.180%); 7.5 MiB/s, 120 objects/s recovering 2024-05-15T13:18:04.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:03 smithi027 ceph-mon[39795]: pgmap v1871: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 516 MiB data, 11 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 523 op/s; 4333/24747 objects degraded (17.509%); 5489/24747 objects misplaced (22.180%); 7.5 MiB/s, 120 objects/s recovering 2024-05-15T13:18:04.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:03 smithi120 ceph-mon[43961]: pgmap v1871: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 516 MiB data, 11 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 523 op/s; 4333/24747 objects degraded (17.509%); 5489/24747 objects misplaced (22.180%); 7.5 MiB/s, 120 objects/s recovering 2024-05-15T13:18:04.720 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:18:04.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:04 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2365226864' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:18:04.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:04 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:18:04.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:04 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 4333/24747 objects degraded (17.509%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:18:05.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:04 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2365226864' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:18:05.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:04 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:18:05.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:04 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 4333/24747 objects degraded (17.509%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:18:05.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:04 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2365226864' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:18:05.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:04 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:18:05.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:04 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 4333/24747 objects degraded (17.509%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:18:05.163 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:18:05.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:05.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:18:05.275 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:18:05.313 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:05.345 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:18:05.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:05.348 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:18:05.449 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:18:05.539 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:05.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:05 smithi027 ceph-mon[39795]: pgmap v1872: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 354 MiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 678 op/s; 4377/16980 objects degraded (25.777%); 4944/16980 objects misplaced (29.117%); 9.2 MiB/s, 146 objects/s recovering 2024-05-15T13:18:05.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:05 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:18:05.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:05 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:18:05.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:05 smithi027 ceph-mon[39795]: osdmap e721: 12 total, 12 up, 10 in 2024-05-15T13:18:05.938 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:18:05.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:05.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:18:06.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:05 smithi120 ceph-mon[43961]: pgmap v1872: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 354 MiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 678 op/s; 4377/16980 objects degraded (25.777%); 4944/16980 objects misplaced (29.117%); 9.2 MiB/s, 146 objects/s recovering 2024-05-15T13:18:06.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:05 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:18:06.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:05 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:18:06.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:05 smithi120 ceph-mon[43961]: osdmap e721: 12 total, 12 up, 10 in 2024-05-15T13:18:06.053 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:18:06.142 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:06.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:05 smithi177 ceph-mon[44303]: pgmap v1872: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 354 MiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 678 op/s; 4377/16980 objects degraded (25.777%); 4944/16980 objects misplaced (29.117%); 9.2 MiB/s, 146 objects/s recovering 2024-05-15T13:18:06.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:05 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:18:06.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:05 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:18:06.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:05 smithi177 ceph-mon[44303]: osdmap e721: 12 total, 12 up, 10 in 2024-05-15T13:18:06.522 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:18:06.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:06.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:18:06.637 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:18:06.663 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:18:06.681 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:06.824 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:06 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:18:06.824 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:06 smithi027 ceph-mon[39795]: 1.12 deep-scrub starts 2024-05-15T13:18:06.824 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:06 smithi027 ceph-mon[39795]: 1.12 deep-scrub ok 2024-05-15T13:18:06.886 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:18:06.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:06.893 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:18:06.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:06 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:18:06.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:06 smithi177 ceph-mon[44303]: 1.12 deep-scrub starts 2024-05-15T13:18:06.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:06 smithi177 ceph-mon[44303]: 1.12 deep-scrub ok 2024-05-15T13:18:06.994 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:18:07.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:06 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:18:07.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:06 smithi120 ceph-mon[43961]: 1.12 deep-scrub starts 2024-05-15T13:18:07.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:06 smithi120 ceph-mon[43961]: 1.12 deep-scrub ok 2024-05-15T13:18:07.121 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:07.366 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:18:07.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:07.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:18:07.476 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:18:07.516 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:07.547 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:18:07.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:07.563 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:18:07.665 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:18:07.692 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:07.712 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:18:07.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:07.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:18:07.816 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:18:07.843 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:07.864 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:18:07.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:07.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:18:07.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:07 smithi177 ceph-mon[44303]: pgmap v1874: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 343 MiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 601 op/s; 4377/16455 objects degraded (26.600%); 4944/16455 objects misplaced (30.046%); 8.6 MiB/s, 138 objects/s recovering 2024-05-15T13:18:07.967 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:18:07.970 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:18:07.993 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:18:08.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:07 smithi027 ceph-mon[39795]: pgmap v1874: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 343 MiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 601 op/s; 4377/16455 objects degraded (26.600%); 4944/16455 objects misplaced (30.046%); 8.6 MiB/s, 138 objects/s recovering 2024-05-15T13:18:08.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:07 smithi120 ceph-mon[43961]: pgmap v1874: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 343 MiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 601 op/s; 4377/16455 objects degraded (26.600%); 4944/16455 objects misplaced (30.046%); 8.6 MiB/s, 138 objects/s recovering 2024-05-15T13:18:08.061 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:08.125 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:18:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:08.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:18:08.231 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:18:08.259 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:08.281 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:18:08.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:08.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:18:08.387 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:18:08.427 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:08.469 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:18:08.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:08.474 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:18:08.575 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:18:08.579 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:18:08.603 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:08.625 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:18:08.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:08.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:18:08.673 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:18:08.673 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:18:08.733 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:18:08.761 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:08.793 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:18:08.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:08.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:18:08.819 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:08 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:18:08.819 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:08 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:18:08.819 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:08 smithi027 ceph-mon[39795]: from='client.16263 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:18:08.819 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:08 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:18:08.819 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:08 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:18:08.819 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:08 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:18:08.897 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:18:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:09.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:08 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:18:09.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:08 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:18:09.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:08 smithi120 ceph-mon[43961]: from='client.16263 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:18:09.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:08 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:18:09.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:08 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:18:09.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:08 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:18:09.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:08 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:18:09.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:08 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:18:09.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:08 smithi177 ceph-mon[44303]: from='client.16263 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:18:09.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:08 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:18:09.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:08 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:18:09.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:08 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:18:10.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:09 smithi177 ceph-mon[44303]: pgmap v1875: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 253 MiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 592 op/s; 4021/12147 objects degraded (33.103%); 4616/12147 objects misplaced (38.001%); 8.5 MiB/s, 135 objects/s recovering 2024-05-15T13:18:10.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:09 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 4377/16455 objects degraded (26.600%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:18:10.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:09 smithi177 ceph-mon[44303]: osdmap e722: 12 total, 12 up, 10 in 2024-05-15T13:18:10.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:09 smithi027 ceph-mon[39795]: pgmap v1875: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 253 MiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 592 op/s; 4021/12147 objects degraded (33.103%); 4616/12147 objects misplaced (38.001%); 8.5 MiB/s, 135 objects/s recovering 2024-05-15T13:18:10.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:09 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 4377/16455 objects degraded (26.600%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:18:10.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:09 smithi027 ceph-mon[39795]: osdmap e722: 12 total, 12 up, 10 in 2024-05-15T13:18:10.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:09 smithi120 ceph-mon[43961]: pgmap v1875: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 253 MiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 592 op/s; 4021/12147 objects degraded (33.103%); 4616/12147 objects misplaced (38.001%); 8.5 MiB/s, 135 objects/s recovering 2024-05-15T13:18:10.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:09 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 4377/16455 objects degraded (26.600%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:18:10.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:09 smithi120 ceph-mon[43961]: osdmap e722: 12 total, 12 up, 10 in 2024-05-15T13:18:12.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:11 smithi177 ceph-mon[44303]: pgmap v1877: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 177 MiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 676 op/s; 4025/8493 objects degraded (47.392%); 4444/8493 objects misplaced (52.325%); 9.2 MiB/s, 146 objects/s recovering 2024-05-15T13:18:12.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:11 smithi027 ceph-mon[39795]: pgmap v1877: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 177 MiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 676 op/s; 4025/8493 objects degraded (47.392%); 4444/8493 objects misplaced (52.325%); 9.2 MiB/s, 146 objects/s recovering 2024-05-15T13:18:12.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:11 smithi120 ceph-mon[43961]: pgmap v1877: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 177 MiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 676 op/s; 4025/8493 objects degraded (47.392%); 4444/8493 objects misplaced (52.325%); 9.2 MiB/s, 146 objects/s recovering 2024-05-15T13:18:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:18:12.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:12.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:18:12.582 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:18:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:12.957 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Removed 109062 objects 2024-05-15T13:18:12.957 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Clean up completed and total clean up time :239.04 2024-05-15T13:18:13.175 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:18:13.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:13.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:18:13.214 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":1874,"stamp":"2024-05-15T13:18:06.562998+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":359858744,"num_objects":5485,"num_object_clones":0,"num_object_copies":16455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4377,"num_objects_misplaced":4944,"num_objects_unfound":0,"num_objects_dirty":5485,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":212643,"num_write_kb":6979969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51382,"num_bytes_recovered":3114467352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163747,"ondisk_log_size":163747,"up":111,"acting":108,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":120,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":10553784,"kb_used_data":3661224,"kb_used_omap":15,"kb_used_meta":6892208,"kb_avail":926897736,"statfs":{"total":959950356480,"available":949143281664,"internally_reserved":0,"allocated":3749093376,"data_stored":3743448669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15956,"internal_metadata":7057621420},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-394133504,"num_objects":-6014,"num_object_clones":0,"num_object_copies":-18042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-749,"num_objects_misplaced":-1035,"num_objects_unfound":0,"num_objects_dirty":-6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6014,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1383,"num_bytes_recovered":90636288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005523"},"pg_stats":[{"pgid":"1.14","version":"94'139","reported_seq":764,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:17:54.144094+0000","last_change":"2024-05-15T13:13:17.477074+0000","last_active":"2024-05-15T13:17:54.144094+0000","last_peered":"2024-05-15T13:17:54.144094+0000","last_clean":"2024-05-15T13:17:54.144094+0000","last_became_active":"2024-05-15T13:13:16.473818+0000","last_became_peered":"2024-05-15T13:13:16.473818+0000","last_unstale":"2024-05-15T13:17:54.144094+0000","last_undegraded":"2024-05-15T13:17:54.144094+0000","last_fullsized":"2024-05-15T13:17:54.144094+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.476961+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.476961+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1],"acting":[2,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":1338,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:17:54.144103+0000","last_change":"2024-05-15T13:13:19.224641+0000","last_active":"2024-05-15T13:17:54.144103+0000","last_peered":"2024-05-15T13:17:54.144103+0000","last_clean":"2024-05-15T13:17:54.144103+0000","last_became_active":"2024-05-15T13:13:11.361366+0000","last_became_peered":"2024-05-15T13:13:11.361366+0000","last_unstale":"2024-05-15T13:17:54.144103+0000","last_undegraded":"2024-05-15T13:17:54.144103+0000","last_fullsized":"2024-05-15T13:17:54.144103+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:19.224509+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:19.224509+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"720'13208","reported_seq":24028,"reported_epoch":720,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-05-15T13:18:04.387120+0000","last_change":"2024-05-15T13:14:21.034561+0000","last_active":"2024-05-15T13:18:04.387120+0000","last_peered":"2024-05-15T13:18:04.387120+0000","last_clean":"2024-05-15T13:14:07.430624+0000","last_became_active":"2024-05-15T13:14:09.931862+0000","last_became_peered":"2024-05-15T13:14:09.931862+0000","last_unstale":"2024-05-15T13:18:04.387120+0000","last_undegraded":"2024-05-15T13:14:21.029697+0000","last_fullsized":"2024-05-15T13:14:09.472302+0000","mapping_epoch":647,"log_start":"611'3200","ondisk_log_start":"611'3200","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21102592,"num_objects":322,"num_object_clones":0,"num_object_copies":966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1179,"num_objects_misplaced":235,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13208,"num_write_kb":432960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5281,"num_bytes_recovered":343146496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,10],"acting":[0,10],"avail_no_missing":["0","5","10"],"object_location_counts":[{"shards":"0,5,10","objects":235},{"shards":"0,10","objects":-1092}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"94'139","reported_seq":1347,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:17:54.142051+0000","last_change":"2024-05-15T13:17:45.047784+0000","last_active":"2024-05-15T13:17:54.142051+0000","last_peered":"2024-05-15T13:17:54.142051+0000","last_clean":"2024-05-15T13:17:54.142051+0000","last_became_active":"2024-05-15T13:13:11.371643+0000","last_became_peered":"2024-05-15T13:13:11.371643+0000","last_unstale":"2024-05-15T13:17:54.142051+0000","last_undegraded":"2024-05-15T13:17:54.142051+0000","last_fullsized":"2024-05-15T13:17:54.142051+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:45.047653+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:17:45.047653+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:02.727762+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.2","version":"720'16751","reported_seq":17184,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:18:02.835716+0000","last_change":"2024-05-15T13:14:18.564785+0000","last_active":"2024-05-15T13:18:02.835716+0000","last_peered":"2024-05-15T13:18:02.835716+0000","last_clean":"2024-05-15T13:18:02.835716+0000","last_became_active":"2024-05-15T13:12:29.918894+0000","last_became_peered":"2024-05-15T13:12:29.918894+0000","last_unstale":"2024-05-15T13:18:02.835716+0000","last_undegraded":"2024-05-15T13:18:02.835716+0000","last_fullsized":"2024-05-15T13:18:02.835716+0000","mapping_epoch":599,"log_start":"639'6701","ondisk_log_start":"639'6701","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":4374,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24641536,"num_objects":376,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13294,"num_write_kb":437440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2],"acting":[3,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"94'139","reported_seq":1093,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:17:54.148206+0000","last_change":"2024-05-15T13:14:54.577804+0000","last_active":"2024-05-15T13:17:54.148206+0000","last_peered":"2024-05-15T13:17:54.148206+0000","last_clean":"2024-05-15T13:17:54.148206+0000","last_became_active":"2024-05-15T13:14:54.577458+0000","last_became_peered":"2024-05-15T13:14:54.577458+0000","last_unstale":"2024-05-15T13:17:54.148206+0000","last_undegraded":"2024-05-15T13:17:54.148206+0000","last_fullsized":"2024-05-15T13:17:54.148206+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:18.049722+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:18.049722+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.3","version":"720'13049","reported_seq":13363,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:18:02.794556+0000","last_change":"2024-05-15T13:13:35.644249+0000","last_active":"2024-05-15T13:18:02.794556+0000","last_peered":"2024-05-15T13:18:02.794556+0000","last_clean":"2024-05-15T13:18:02.794556+0000","last_became_active":"2024-05-15T13:12:29.921897+0000","last_became_peered":"2024-05-15T13:12:29.921897+0000","last_unstale":"2024-05-15T13:18:02.794556+0000","last_undegraded":"2024-05-15T13:18:02.794556+0000","last_fullsized":"2024-05-15T13:18:02.794556+0000","mapping_epoch":599,"log_start":"611'3000","ondisk_log_start":"611'3000","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'5271","last_scrub_stamp":"2024-05-15T13:13:35.643996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:35.643996+0000","objects_scrubbed":5237,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26804224,"num_objects":409,"num_object_clones":0,"num_object_copies":1227,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13049,"num_write_kb":430656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"94'139","reported_seq":1342,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-05-15T13:18:04.673491+0000","last_change":"2024-05-15T13:17:38.671246+0000","last_active":"2024-05-15T13:18:04.673491+0000","last_peered":"2024-05-15T13:18:04.673491+0000","last_clean":"2024-05-15T13:18:04.673491+0000","last_became_active":"2024-05-15T13:13:11.373251+0000","last_became_peered":"2024-05-15T13:13:11.373251+0000","last_unstale":"2024-05-15T13:18:04.673491+0000","last_undegraded":"2024-05-15T13:18:04.673491+0000","last_fullsized":"2024-05-15T13:18:04.673491+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:38.671141+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:17:38.671141+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:51.578496+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.0","version":"720'13396","reported_seq":13708,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:18:03.562634+0000","last_change":"2024-05-15T13:17:48.266919+0000","last_active":"2024-05-15T13:18:03.562634+0000","last_peered":"2024-05-15T13:18:03.562634+0000","last_clean":"2024-05-15T13:18:03.562634+0000","last_became_active":"2024-05-15T13:12:29.924027+0000","last_became_peered":"2024-05-15T13:12:29.924027+0000","last_unstale":"2024-05-15T13:18:03.562634+0000","last_undegraded":"2024-05-15T13:18:03.562634+0000","last_fullsized":"2024-05-15T13:18:03.562634+0000","mapping_epoch":599,"log_start":"613'3300","ondisk_log_start":"613'3300","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"717'12845","last_scrub_stamp":"2024-05-15T13:17:48.266618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:17:48.266618+0000","objects_scrubbed":884,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:54.566295+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21495808,"num_objects":328,"num_object_clones":0,"num_object_copies":984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13396,"num_write_kb":439168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"94'139","reported_seq":1107,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:17:54.140362+0000","last_change":"2024-05-15T13:14:11.522798+0000","last_active":"2024-05-15T13:17:54.140362+0000","last_peered":"2024-05-15T13:17:54.140362+0000","last_clean":"2024-05-15T13:17:54.140362+0000","last_became_active":"2024-05-15T13:14:11.522490+0000","last_became_peered":"2024-05-15T13:14:11.522490+0000","last_unstale":"2024-05-15T13:17:54.140362+0000","last_undegraded":"2024-05-15T13:17:54.140362+0000","last_fullsized":"2024-05-15T13:17:54.140362+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.847198+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.847198+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"720'13186","reported_seq":32136,"reported_epoch":720,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-15T13:18:01.908324+0000","last_change":"2024-05-15T13:14:52.829919+0000","last_active":"2024-05-15T13:18:01.908324+0000","last_peered":"2024-05-15T13:18:01.908324+0000","last_clean":"2024-05-15T13:13:46.353513+0000","last_became_active":"2024-05-15T13:14:52.812683+0000","last_became_peered":"2024-05-15T13:14:52.812683+0000","last_unstale":"2024-05-15T13:18:01.908324+0000","last_undegraded":"2024-05-15T13:14:52.601890+0000","last_fullsized":"2024-05-15T13:14:52.453432+0000","mapping_epoch":658,"log_start":"611'3100","ondisk_log_start":"611'3100","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"633'5877","last_scrub_stamp":"2024-05-15T13:13:44.171370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:44.171370+0000","objects_scrubbed":5827,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1393,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27394048,"num_objects":418,"num_object_clones":0,"num_object_copies":1254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13186,"num_write_kb":435328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6386,"num_bytes_recovered":418381824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3],"avail_no_missing":["10","5","11"],"object_location_counts":[{"shards":"1,3,10","objects":86},{"shards":"1,10,11","objects":332}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"94'139","reported_seq":1120,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-05-15T13:18:04.673289+0000","last_change":"2024-05-15T13:17:41.721650+0000","last_active":"2024-05-15T13:18:04.673289+0000","last_peered":"2024-05-15T13:18:04.673289+0000","last_clean":"2024-05-15T13:18:04.673289+0000","last_became_active":"2024-05-15T13:15:43.614970+0000","last_became_peered":"2024-05-15T13:15:43.614970+0000","last_unstale":"2024-05-15T13:18:04.673289+0000","last_undegraded":"2024-05-15T13:18:04.673289+0000","last_fullsized":"2024-05-15T13:18:04.673289+0000","mapping_epoch":678,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:41.721516+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:17:41.721516+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:59.128918+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,0],"acting":[9,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.7","version":"720'13450","reported_seq":16561,"reported_epoch":720,"state":"active+recovering+remapped","last_fresh":"2024-05-15T13:18:02.636190+0000","last_change":"2024-05-15T13:17:20.364816+0000","last_active":"2024-05-15T13:18:02.636190+0000","last_peered":"2024-05-15T13:18:02.636190+0000","last_clean":"2024-05-15T13:14:50.323746+0000","last_became_active":"2024-05-15T13:14:52.762068+0000","last_became_peered":"2024-05-15T13:14:52.762068+0000","last_unstale":"2024-05-15T13:18:02.636190+0000","last_undegraded":"2024-05-15T13:18:02.636190+0000","last_fullsized":"2024-05-15T13:18:02.636190+0000","mapping_epoch":658,"log_start":"613'3400","ondisk_log_start":"613'3400","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"633'6037","last_scrub_stamp":"2024-05-15T13:13:44.144120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:44.144120+0000","objects_scrubbed":5996,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1443,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24772608,"num_objects":378,"num_object_clones":0,"num_object_copies":1134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4479,"num_objects_unfound":0,"num_objects_dirty":378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13450,"num_write_kb":442496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1393,"num_bytes_recovered":89194496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,3],"acting":[1,8,3],"avail_no_missing":["1","3","8"],"object_location_counts":[{"shards":"1,3,8","objects":-4101}],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"94'139","reported_seq":1347,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:17:54.149793+0000","last_change":"2024-05-15T13:13:18.973897+0000","last_active":"2024-05-15T13:17:54.149793+0000","last_peered":"2024-05-15T13:17:54.149793+0000","last_clean":"2024-05-15T13:17:54.149793+0000","last_became_active":"2024-05-15T13:13:11.373548+0000","last_became_peered":"2024-05-15T13:13:11.373548+0000","last_unstale":"2024-05-15T13:17:54.149793+0000","last_undegraded":"2024-05-15T13:17:54.149793+0000","last_fullsized":"2024-05-15T13:17:54.149793+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:18.973644+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:18.973644+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"720'13410","reported_seq":25916,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:18:04.334386+0000","last_change":"2024-05-15T13:17:37.903772+0000","last_active":"2024-05-15T13:18:04.334386+0000","last_peered":"2024-05-15T13:18:04.334386+0000","last_clean":"2024-05-15T13:18:04.334386+0000","last_became_active":"2024-05-15T13:17:37.742339+0000","last_became_peered":"2024-05-15T13:17:37.742339+0000","last_unstale":"2024-05-15T13:18:04.334386+0000","last_undegraded":"2024-05-15T13:18:04.334386+0000","last_fullsized":"2024-05-15T13:18:04.334386+0000","mapping_epoch":714,"log_start":"613'3400","ondisk_log_start":"613'3400","created":599,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21364736,"num_objects":326,"num_object_clones":0,"num_object_copies":978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13410,"num_write_kb":439552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6119,"num_bytes_recovered":400949248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"94'139","reported_seq":1365,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:17:54.145054+0000","last_change":"2024-05-15T13:14:54.570374+0000","last_active":"2024-05-15T13:17:54.145054+0000","last_peered":"2024-05-15T13:17:54.145054+0000","last_clean":"2024-05-15T13:17:54.145054+0000","last_became_active":"2024-05-15T13:14:54.570004+0000","last_became_peered":"2024-05-15T13:14:54.570004+0000","last_unstale":"2024-05-15T13:17:54.145054+0000","last_undegraded":"2024-05-15T13:17:54.145054+0000","last_fullsized":"2024-05-15T13:17:54.145054+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:18.468889+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:18.468889+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":220,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"720'13332","reported_seq":27300,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:18:03.017517+0000","last_change":"2024-05-15T13:17:41.296650+0000","last_active":"2024-05-15T13:18:03.017517+0000","last_peered":"2024-05-15T13:18:03.017517+0000","last_clean":"2024-05-15T13:18:03.017517+0000","last_became_active":"2024-05-15T13:17:18.819383+0000","last_became_peered":"2024-05-15T13:17:18.819383+0000","last_unstale":"2024-05-15T13:18:03.017517+0000","last_undegraded":"2024-05-15T13:18:03.017517+0000","last_fullsized":"2024-05-15T13:18:03.017517+0000","mapping_epoch":705,"log_start":"613'3300","ondisk_log_start":"613'3300","created":599,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":0,"last_scrub":"716'12560","last_scrub_stamp":"2024-05-15T13:17:41.296484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:17:41.296484+0000","objects_scrubbed":1128,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:44.135405+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23068672,"num_objects":352,"num_object_clones":0,"num_object_copies":1056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13332,"num_write_kb":437888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6842,"num_bytes_recovered":296615936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,0],"acting":[11,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"94'139","reported_seq":1342,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:17:55.115920+0000","last_change":"2024-05-15T13:17:55.115951+0000","last_active":"2024-05-15T13:17:55.115920+0000","last_peered":"2024-05-15T13:17:55.115920+0000","last_clean":"2024-05-15T13:17:55.115920+0000","last_became_active":"2024-05-15T13:06:46.426908+0000","last_became_peered":"2024-05-15T13:06:46.426908+0000","last_unstale":"2024-05-15T13:17:55.115920+0000","last_undegraded":"2024-05-15T13:17:55.115920+0000","last_fullsized":"2024-05-15T13:17:55.115920+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:55.115854+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:17:55.115854+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:04.858229+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"720'12850","reported_seq":13166,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:18:02.557189+0000","last_change":"2024-05-15T13:13:39.097208+0000","last_active":"2024-05-15T13:18:02.557189+0000","last_peered":"2024-05-15T13:18:02.557189+0000","last_clean":"2024-05-15T13:18:02.557189+0000","last_became_active":"2024-05-15T13:12:29.926405+0000","last_became_peered":"2024-05-15T13:12:29.926405+0000","last_unstale":"2024-05-15T13:18:02.557189+0000","last_undegraded":"2024-05-15T13:18:02.557189+0000","last_fullsized":"2024-05-15T13:18:02.557189+0000","mapping_epoch":599,"log_start":"611'2800","ondisk_log_start":"611'2800","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"629'5457","last_scrub_stamp":"2024-05-15T13:13:39.097042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:39.097042+0000","objects_scrubbed":5417,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1513,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23330816,"num_objects":356,"num_object_clones":0,"num_object_copies":1068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12850,"num_write_kb":422592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"94'139","reported_seq":1336,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-05-15T13:18:04.673360+0000","last_change":"2024-05-15T13:17:44.642745+0000","last_active":"2024-05-15T13:18:04.673360+0000","last_peered":"2024-05-15T13:18:04.673360+0000","last_clean":"2024-05-15T13:18:04.673360+0000","last_became_active":"2024-05-15T13:06:46.424667+0000","last_became_peered":"2024-05-15T13:06:46.424667+0000","last_unstale":"2024-05-15T13:18:04.673360+0000","last_undegraded":"2024-05-15T13:18:04.673360+0000","last_fullsized":"2024-05-15T13:18:04.673360+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:44.642652+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:17:44.642652+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:46.761399+0000","scrub_duration":205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.f","version":"720'13208","reported_seq":13461,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:18:04.412144+0000","last_change":"2024-05-15T13:12:29.925178+0000","last_active":"2024-05-15T13:18:04.412144+0000","last_peered":"2024-05-15T13:18:04.412144+0000","last_clean":"2024-05-15T13:18:04.412144+0000","last_became_active":"2024-05-15T13:12:29.924959+0000","last_became_peered":"2024-05-15T13:12:29.924959+0000","last_unstale":"2024-05-15T13:18:04.412144+0000","last_undegraded":"2024-05-15T13:18:04.412144+0000","last_fullsized":"2024-05-15T13:18:04.412144+0000","mapping_epoch":599,"log_start":"611'3200","ondisk_log_start":"611'3200","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19398656,"num_objects":296,"num_object_clones":0,"num_object_copies":888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13208,"num_write_kb":432128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"94'139","reported_seq":1115,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-05-15T13:18:04.673347+0000","last_change":"2024-05-15T13:17:45.527703+0000","last_active":"2024-05-15T13:18:04.673347+0000","last_peered":"2024-05-15T13:18:04.673347+0000","last_clean":"2024-05-15T13:18:04.673347+0000","last_became_active":"2024-05-15T13:15:43.611913+0000","last_became_peered":"2024-05-15T13:15:43.611913+0000","last_unstale":"2024-05-15T13:18:04.673347+0000","last_undegraded":"2024-05-15T13:18:04.673347+0000","last_fullsized":"2024-05-15T13:18:04.673347+0000","mapping_epoch":678,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:45.527612+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:17:45.527612+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:58.076902+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.c","version":"720'13584","reported_seq":27688,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:18:04.271472+0000","last_change":"2024-05-15T13:17:22.205596+0000","last_active":"2024-05-15T13:18:04.271472+0000","last_peered":"2024-05-15T13:18:04.271472+0000","last_clean":"2024-05-15T13:18:04.271472+0000","last_became_active":"2024-05-15T13:17:22.175461+0000","last_became_peered":"2024-05-15T13:17:22.175461+0000","last_unstale":"2024-05-15T13:18:04.271472+0000","last_undegraded":"2024-05-15T13:18:04.271472+0000","last_fullsized":"2024-05-15T13:18:04.271472+0000","mapping_epoch":708,"log_start":"613'3500","ondisk_log_start":"613'3500","created":599,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":0,"last_scrub":"641'6420","last_scrub_stamp":"2024-05-15T13:13:54.703460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:54.703460+0000","objects_scrubbed":6408,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":989,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20840448,"num_objects":318,"num_object_clones":0,"num_object_copies":954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13584,"num_write_kb":444864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6884,"num_bytes_recovered":451149824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,11],"acting":[6,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"720'13242","reported_seq":24966,"reported_epoch":720,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-05-15T13:18:02.634452+0000","last_change":"2024-05-15T13:14:20.686326+0000","last_active":"2024-05-15T13:18:02.634452+0000","last_peered":"2024-05-15T13:18:02.634452+0000","last_clean":"2024-05-15T13:14:07.355352+0000","last_became_active":"2024-05-15T13:14:09.930774+0000","last_became_peered":"2024-05-15T13:14:09.930774+0000","last_unstale":"2024-05-15T13:18:02.634452+0000","last_undegraded":"2024-05-15T13:14:20.671875+0000","last_fullsized":"2024-05-15T13:14:09.468986+0000","mapping_epoch":647,"log_start":"612'3200","ondisk_log_start":"612'3200","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'5339","last_scrub_stamp":"2024-05-15T13:13:35.290438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:35.290438+0000","objects_scrubbed":5292,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25559040,"num_objects":390,"num_object_clones":0,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":809,"num_objects_misplaced":230,"num_objects_unfound":0,"num_objects_dirty":390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13242,"num_write_kb":436224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5705,"num_bytes_recovered":373293056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,10],"acting":[2,10],"avail_no_missing":["2","1","10"],"object_location_counts":[{"shards":"1,2,10","objects":230},{"shards":"2,10","objects":-649}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"94'139","reported_seq":1013,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:17:54.163659+0000","last_change":"2024-05-15T13:13:18.190667+0000","last_active":"2024-05-15T13:17:54.163659+0000","last_peered":"2024-05-15T13:17:54.163659+0000","last_clean":"2024-05-15T13:17:54.163659+0000","last_became_active":"2024-05-15T13:13:16.477771+0000","last_became_peered":"2024-05-15T13:13:16.477771+0000","last_unstale":"2024-05-15T13:17:54.163659+0000","last_undegraded":"2024-05-15T13:17:54.163659+0000","last_fullsized":"2024-05-15T13:17:54.163659+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:18.190529+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:18.190529+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,1],"acting":[3,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"720'13482","reported_seq":13795,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:18:04.389895+0000","last_change":"2024-05-15T13:13:31.369534+0000","last_active":"2024-05-15T13:18:04.389895+0000","last_peered":"2024-05-15T13:18:04.389895+0000","last_clean":"2024-05-15T13:18:04.389895+0000","last_became_active":"2024-05-15T13:12:29.920800+0000","last_became_peered":"2024-05-15T13:12:29.920800+0000","last_unstale":"2024-05-15T13:18:04.389895+0000","last_undegraded":"2024-05-15T13:18:04.389895+0000","last_fullsized":"2024-05-15T13:18:04.389895+0000","mapping_epoch":599,"log_start":"613'3400","ondisk_log_start":"613'3400","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'5126","last_scrub_stamp":"2024-05-15T13:13:31.369265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:31.369265+0000","objects_scrubbed":5076,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1450,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19136512,"num_objects":292,"num_object_clones":0,"num_object_copies":876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13482,"num_write_kb":440768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"94'139","reported_seq":1106,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:17:54.140461+0000","last_change":"2024-05-15T13:13:16.831084+0000","last_active":"2024-05-15T13:17:54.140461+0000","last_peered":"2024-05-15T13:17:54.140461+0000","last_clean":"2024-05-15T13:17:54.140461+0000","last_became_active":"2024-05-15T13:13:16.472318+0000","last_became_peered":"2024-05-15T13:13:16.472318+0000","last_unstale":"2024-05-15T13:17:54.140461+0000","last_undegraded":"2024-05-15T13:17:54.140461+0000","last_fullsized":"2024-05-15T13:17:54.140461+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:16.830998+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:16.830998+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"721'13482","reported_seq":13823,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-05-15T13:18:05.978980+0000","last_change":"2024-05-15T13:17:49.782037+0000","last_active":"2024-05-15T13:18:05.978980+0000","last_peered":"2024-05-15T13:18:05.978980+0000","last_clean":"2024-05-15T13:18:05.978980+0000","last_became_active":"2024-05-15T13:12:29.923300+0000","last_became_peered":"2024-05-15T13:12:29.923300+0000","last_unstale":"2024-05-15T13:18:05.978980+0000","last_undegraded":"2024-05-15T13:18:05.978980+0000","last_fullsized":"2024-05-15T13:18:05.978980+0000","mapping_epoch":599,"log_start":"613'3400","ondisk_log_start":"613'3400","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"718'12890","last_scrub_stamp":"2024-05-15T13:17:49.781904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:17:49.781904+0000","objects_scrubbed":829,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:14.843498+0000","scrub_duration":343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14942208,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13482,"num_write_kb":438720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.c","version":"94'139","reported_seq":765,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:17:54.160232+0000","last_change":"2024-05-15T13:13:17.140519+0000","last_active":"2024-05-15T13:17:54.160232+0000","last_peered":"2024-05-15T13:17:54.160232+0000","last_clean":"2024-05-15T13:17:54.160232+0000","last_became_active":"2024-05-15T13:13:16.476972+0000","last_became_peered":"2024-05-15T13:13:16.476972+0000","last_unstale":"2024-05-15T13:17:54.160232+0000","last_undegraded":"2024-05-15T13:17:54.160232+0000","last_fullsized":"2024-05-15T13:17:54.160232+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.140398+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.140398+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"720'13868","reported_seq":26372,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:18:02.545578+0000","last_change":"2024-05-15T13:16:18.785284+0000","last_active":"2024-05-15T13:18:02.545578+0000","last_peered":"2024-05-15T13:18:02.545578+0000","last_clean":"2024-05-15T13:18:02.545578+0000","last_became_active":"2024-05-15T13:16:18.691546+0000","last_became_peered":"2024-05-15T13:16:18.691546+0000","last_unstale":"2024-05-15T13:18:02.545578+0000","last_undegraded":"2024-05-15T13:18:02.545578+0000","last_fullsized":"2024-05-15T13:18:02.545578+0000","mapping_epoch":693,"log_start":"617'3800","ondisk_log_start":"617'3800","created":599,"last_epoch_clean":694,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:12:28.908905+0000","objects_scrubbed":624,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26017792,"num_objects":397,"num_object_clones":0,"num_object_copies":1191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13439,"num_write_kb":442752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6108,"num_bytes_recovered":378404864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"94'139","reported_seq":753,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:17:54.144571+0000","last_change":"2024-05-15T13:13:17.391026+0000","last_active":"2024-05-15T13:17:54.144571+0000","last_peered":"2024-05-15T13:17:54.144571+0000","last_clean":"2024-05-15T13:17:54.144571+0000","last_became_active":"2024-05-15T13:13:16.478031+0000","last_became_peered":"2024-05-15T13:13:16.478031+0000","last_unstale":"2024-05-15T13:17:54.144571+0000","last_undegraded":"2024-05-15T13:17:54.144571+0000","last_fullsized":"2024-05-15T13:17:54.144571+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.390903+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.390903+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,11],"acting":[1,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"720'13031","reported_seq":26689,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:18:04.278526+0000","last_change":"2024-05-15T13:17:48.918440+0000","last_active":"2024-05-15T13:18:04.278526+0000","last_peered":"2024-05-15T13:18:04.278526+0000","last_clean":"2024-05-15T13:18:04.278526+0000","last_became_active":"2024-05-15T13:17:48.884759+0000","last_became_peered":"2024-05-15T13:17:48.884759+0000","last_unstale":"2024-05-15T13:18:04.278526+0000","last_undegraded":"2024-05-15T13:18:04.278526+0000","last_fullsized":"2024-05-15T13:18:04.278526+0000","mapping_epoch":717,"log_start":"611'3000","ondisk_log_start":"611'3000","created":599,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"630'5615","last_scrub_stamp":"2024-05-15T13:13:40.897467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:12:28.908905+0000","last_clean_scrub_stamp":"2024-05-15T13:13:40.897467+0000","objects_scrubbed":5565,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1781,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19398680,"num_objects":297,"num_object_clones":0,"num_object_copies":891,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":297,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13031,"num_write_kb":426433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6664,"num_bytes_recovered":363331608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"94'139","reported_seq":1366,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:17:54.141645+0000","last_change":"2024-05-15T13:17:44.264911+0000","last_active":"2024-05-15T13:17:54.141645+0000","last_peered":"2024-05-15T13:17:54.141645+0000","last_clean":"2024-05-15T13:17:54.141645+0000","last_became_active":"2024-05-15T13:13:16.484252+0000","last_became_peered":"2024-05-15T13:13:16.484252+0000","last_unstale":"2024-05-15T13:17:54.141645+0000","last_undegraded":"2024-05-15T13:17:54.141645+0000","last_fullsized":"2024-05-15T13:17:54.141645+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:44.264755+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:17:44.264755+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:50.708865+0000","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"94'139","reported_seq":1348,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:17:54.135850+0000","last_change":"2024-05-15T13:17:40.143307+0000","last_active":"2024-05-15T13:17:54.135850+0000","last_peered":"2024-05-15T13:17:54.135850+0000","last_clean":"2024-05-15T13:17:54.135850+0000","last_became_active":"2024-05-15T13:13:16.472800+0000","last_became_peered":"2024-05-15T13:13:16.472800+0000","last_unstale":"2024-05-15T13:17:54.135850+0000","last_undegraded":"2024-05-15T13:17:54.135850+0000","last_fullsized":"2024-05-15T13:17:54.135850+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:40.143179+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:17:40.143179+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:00.376173+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,8],"acting":[5,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"94'139","reported_seq":1114,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:17:54.150361+0000","last_change":"2024-05-15T13:13:17.150999+0000","last_active":"2024-05-15T13:17:54.150361+0000","last_peered":"2024-05-15T13:17:54.150361+0000","last_clean":"2024-05-15T13:17:54.150361+0000","last_became_active":"2024-05-15T13:13:16.476744+0000","last_became_peered":"2024-05-15T13:13:16.476744+0000","last_unstale":"2024-05-15T13:17:54.150361+0000","last_undegraded":"2024-05-15T13:17:54.150361+0000","last_fullsized":"2024-05-15T13:17:54.150361+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.150897+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.150897+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":232,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"94'139","reported_seq":1364,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:17:54.135832+0000","last_change":"2024-05-15T13:16:44.320180+0000","last_active":"2024-05-15T13:17:54.135832+0000","last_peered":"2024-05-15T13:17:54.135832+0000","last_clean":"2024-05-15T13:17:54.135832+0000","last_became_active":"2024-05-15T13:13:49.926687+0000","last_became_peered":"2024-05-15T13:13:49.926687+0000","last_unstale":"2024-05-15T13:17:54.135832+0000","last_undegraded":"2024-05-15T13:17:54.135832+0000","last_fullsized":"2024-05-15T13:17:54.135832+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:16:44.320095+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:16:44.320095+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:05.122264+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"94'139","reported_seq":1364,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-05-15T13:17:54.144973+0000","last_change":"2024-05-15T13:13:20.455813+0000","last_active":"2024-05-15T13:17:54.144973+0000","last_peered":"2024-05-15T13:17:54.144973+0000","last_clean":"2024-05-15T13:17:54.144973+0000","last_became_active":"2024-05-15T13:13:16.478488+0000","last_became_peered":"2024-05-15T13:13:16.478488+0000","last_unstale":"2024-05-15T13:17:54.144973+0000","last_undegraded":"2024-05-15T13:17:54.144973+0000","last_fullsized":"2024-05-15T13:17:54.144973+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:20.455652+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:20.455652+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":359268376,"num_objects":5483,"num_object_clones":0,"num_object_copies":16449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4377,"num_objects_misplaced":4944,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":212643,"num_write_kb":6979969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51382,"num_bytes_recovered":3114467352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3687067648,"data_stored":3687055432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160828,"ondisk_log_size":160828,"up":48,"acting":45,"num_store_stats":10},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057032,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1165876,"kb_used_data":571052,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":92579276,"statfs":{"total":95995035648,"available":94801178624,"internally_reserved":0,"allocated":584757248,"data_stored":584195504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":609089990},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351372,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1342772,"kb_used_data":671596,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":92402380,"statfs":{"total":95995035648,"available":94620037120,"internally_reserved":0,"allocated":687714304,"data_stored":687152560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":687274432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514898,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":912064,"kb_used_data":104824,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92833088,"statfs":{"total":95995035648,"available":95061082112,"internally_reserved":0,"allocated":107339776,"data_stored":106773637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":826603968},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874647,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":472180,"kb_used_data":57836,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93272972,"statfs":{"total":95995035648,"available":95511523328,"internally_reserved":0,"allocated":59224064,"data_stored":58662320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":240518168987,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430626,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":739512,"kb_used_data":44720,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93005640,"statfs":{"total":95995035648,"available":95237775360,"internally_reserved":0,"allocated":45793280,"data_stored":45227464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":711457222},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575290,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1446840,"kb_used_data":589552,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":92298312,"statfs":{"total":95995035648,"available":94513471488,"internally_reserved":0,"allocated":603701248,"data_stored":603135432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":877787589},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738820,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":873848,"kb_used_data":120752,"kb_used_omap":1,"kb_used_meta":753086,"kb_avail":92871304,"statfs":{"total":95995035648,"available":95100215296,"internally_reserved":0,"allocated":123650048,"data_stored":123084232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":771160517},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411767,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1054652,"kb_used_data":495796,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92690500,"statfs":{"total":95995035648,"available":94915072000,"internally_reserved":0,"allocated":507695104,"data_stored":507129808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150128,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1049724,"kb_used_data":143860,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":92695428,"statfs":{"total":95995035648,"available":94920118272,"internally_reserved":0,"allocated":147312640,"data_stored":146747344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":927529415},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986605,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692263,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1496316,"kb_used_data":861236,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92248836,"statfs":{"total":95995035648,"available":94462808064,"internally_reserved":0,"allocated":881905664,"data_stored":881340368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":120262656,"data_stored":120258584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":595136512,"data_stored":595132440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":494731264,"data_stored":494731264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":145031168,"data_stored":145031168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":876609536,"data_stored":876609536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40964096,"data_stored":40960024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":52494336,"data_stored":52494336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":94437376,"data_stored":94437376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":685113344,"data_stored":685113344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":582287360,"data_stored":582287360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:18:13.216 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph status --format=json 2024-05-15T13:18:13.283 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:18:13.333 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:13.416 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:18:13.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:13.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:18:13.525 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:18:13.554 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:13.573 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:18:13.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:13.577 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:18:13.678 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:18:13.711 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:14.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:13 smithi027 ceph-mon[39795]: pgmap v1878: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 156 MiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 394 op/s; 3717/7491 objects degraded (49.620%); 4444/7491 objects misplaced (59.325%); 6.5 MiB/s, 103 objects/s recovering 2024-05-15T13:18:14.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:13 smithi177 ceph-mon[44303]: pgmap v1878: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 156 MiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 394 op/s; 3717/7491 objects degraded (49.620%); 4444/7491 objects misplaced (59.325%); 6.5 MiB/s, 103 objects/s recovering 2024-05-15T13:18:14.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:13 smithi120 ceph-mon[43961]: pgmap v1878: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 156 MiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 394 op/s; 3717/7491 objects degraded (49.620%); 4444/7491 objects misplaced (59.325%); 6.5 MiB/s, 103 objects/s recovering 2024-05-15T13:18:15.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:14 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 3717/7491 objects degraded (49.620%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:18:15.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:14 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 3717/7491 objects degraded (49.620%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:18:15.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:14 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 3717/7491 objects degraded (49.620%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:18:16.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:15 smithi177 ceph-mon[44303]: pgmap v1879: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 13 MiB data, 9.2 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 551 op/s; 3325/585 objects degraded (568.376%); 4139/585 objects misplaced (707.521%); 7.2 MiB/s, 154 objects/s recovering 2024-05-15T13:18:16.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:15 smithi027 ceph-mon[39795]: pgmap v1879: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 13 MiB data, 9.2 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 551 op/s; 3325/585 objects degraded (568.376%); 4139/585 objects misplaced (707.521%); 7.2 MiB/s, 154 objects/s recovering 2024-05-15T13:18:16.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:15 smithi120 ceph-mon[43961]: pgmap v1879: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 13 MiB data, 9.2 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 551 op/s; 3325/585 objects degraded (568.376%); 4139/585 objects misplaced (707.521%); 7.2 MiB/s, 154 objects/s recovering 2024-05-15T13:18:16.678 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:18:16.680 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-05-15T13:18:16.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:16.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:18:16.782 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:18:16.809 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:16.830 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:18:16.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:16.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:18:16.933 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:18:16.964 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:16.986 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:18:16.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:16.989 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:18:17.090 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:18:17.118 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:17.137 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:18:17.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:17.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:18:17.241 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:18:17.267 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:17.287 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:18:17.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:17.290 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:18:17.391 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:18:17.416 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:17.438 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:18:17.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:17.441 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:18:17.542 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:18:17.568 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:17.588 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:18:17.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:17.591 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:18:17.692 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:18:17.718 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:17.737 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:18:17.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:17.740 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:18:17.776 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:18:17.776 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:18:17.776 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:18:17.776 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:18:17.776 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:18:17.777 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:18:17.777 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:18:17.777 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:18:17.777 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:18:17.777 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:18:17.777 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:18:17.777 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:18:17.777 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:18:17.777 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:18:17.777 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:18:17.777 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:18:17.777 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:18:17.777 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:18:17.777 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:18:17.778 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:18:17.778 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:18:17.778 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:18:17.778 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:18:17.778 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:18:17.841 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:18:17.868 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:17.890 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:18:17.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:17.893 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:18:17.994 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:18:18.022 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:18.041 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:18:18.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:18.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:18:18.145 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:18:18.152 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:18:18.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:17 smithi120 ceph-mon[43961]: pgmap v1880: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 7.2 MiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 537 op/s; 3325/324 objects degraded (1026.235%); 4139/324 objects misplaced (1277.469%); 7.1 MiB/s, 152 objects/s recovering 2024-05-15T13:18:18.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:17 smithi120 ceph-mon[43961]: osdmap e723: 12 total, 12 up, 10 in 2024-05-15T13:18:18.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:17 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3987592990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:18:18.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:17 smithi120 ceph-mon[43961]: 1.14 scrub starts 2024-05-15T13:18:18.171 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:18.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:17 smithi177 ceph-mon[44303]: pgmap v1880: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 7.2 MiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 537 op/s; 3325/324 objects degraded (1026.235%); 4139/324 objects misplaced (1277.469%); 7.1 MiB/s, 152 objects/s recovering 2024-05-15T13:18:18.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:17 smithi177 ceph-mon[44303]: osdmap e723: 12 total, 12 up, 10 in 2024-05-15T13:18:18.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:17 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3987592990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:18:18.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:17 smithi177 ceph-mon[44303]: 1.14 scrub starts 2024-05-15T13:18:18.191 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:18:18.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:18.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:18:18.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:17 smithi027 ceph-mon[39795]: pgmap v1880: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 7.2 MiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 537 op/s; 3325/324 objects degraded (1026.235%); 4139/324 objects misplaced (1277.469%); 7.1 MiB/s, 152 objects/s recovering 2024-05-15T13:18:18.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:17 smithi027 ceph-mon[39795]: osdmap e723: 12 total, 12 up, 10 in 2024-05-15T13:18:18.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:17 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3987592990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:18:18.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:17 smithi027 ceph-mon[39795]: 1.14 scrub starts 2024-05-15T13:18:18.295 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:18:18.321 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:18.339 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:18:18.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:18.343 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:18:18.401 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-05-15T13:18:18.401 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-05-15T13:18:18.444 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:18:18.470 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:18.489 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:18:18.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:18.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:18:18.593 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:18:18.619 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:18.638 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:18:18.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:18.641 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:18:18.742 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:18:18.777 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:19.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:18 smithi027 ceph-mon[39795]: 1.14 scrub ok 2024-05-15T13:18:19.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:18 smithi027 ceph-mon[39795]: osdmap e724: 12 total, 12 up, 10 in 2024-05-15T13:18:19.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:18 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:18:19.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:18 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:18:19.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:18 smithi177 ceph-mon[44303]: 1.14 scrub ok 2024-05-15T13:18:19.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:18 smithi177 ceph-mon[44303]: osdmap e724: 12 total, 12 up, 10 in 2024-05-15T13:18:19.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:18 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:18:19.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:18 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:18:19.287 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:18 smithi120 ceph-mon[43961]: 1.14 scrub ok 2024-05-15T13:18:19.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:18 smithi120 ceph-mon[43961]: osdmap e724: 12 total, 12 up, 10 in 2024-05-15T13:18:19.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:18 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:18:19.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:18 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:18:20.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:19 smithi027 ceph-mon[39795]: pgmap v1883: 37 pgs: 1 peering, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 577 KiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 353 op/s; 989/6 objects degraded (16483.333%); 4135/6 objects misplaced (68916.667%); 3.6 MiB/s, 251 objects/s recovering 2024-05-15T13:18:20.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:19 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:18:20.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:19 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:18:20.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:19 smithi027 ceph-mon[39795]: osdmap e725: 12 total, 12 up, 10 in 2024-05-15T13:18:20.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:19 smithi027 ceph-mon[39795]: 7.8 scrub starts 2024-05-15T13:18:20.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:19 smithi027 ceph-mon[39795]: 7.8 scrub ok 2024-05-15T13:18:20.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:19 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 989/6 objects degraded (16483.333%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-05-15T13:18:20.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:19 smithi027 ceph-mon[39795]: osdmap e726: 12 total, 12 up, 10 in 2024-05-15T13:18:20.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:19 smithi027 ceph-mon[39795]: 7.f scrub starts 2024-05-15T13:18:20.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:19 smithi027 ceph-mon[39795]: 7.f scrub ok 2024-05-15T13:18:20.151 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:18:20.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:19 smithi177 ceph-mon[44303]: pgmap v1883: 37 pgs: 1 peering, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 577 KiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 353 op/s; 989/6 objects degraded (16483.333%); 4135/6 objects misplaced (68916.667%); 3.6 MiB/s, 251 objects/s recovering 2024-05-15T13:18:20.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:19 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:18:20.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:19 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:18:20.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:19 smithi177 ceph-mon[44303]: osdmap e725: 12 total, 12 up, 10 in 2024-05-15T13:18:20.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:19 smithi177 ceph-mon[44303]: 7.8 scrub starts 2024-05-15T13:18:20.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:19 smithi177 ceph-mon[44303]: 7.8 scrub ok 2024-05-15T13:18:20.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:19 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 989/6 objects degraded (16483.333%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-05-15T13:18:20.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:19 smithi177 ceph-mon[44303]: osdmap e726: 12 total, 12 up, 10 in 2024-05-15T13:18:20.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:19 smithi177 ceph-mon[44303]: 7.f scrub starts 2024-05-15T13:18:20.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:19 smithi177 ceph-mon[44303]: 7.f scrub ok 2024-05-15T13:18:20.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:19 smithi120 ceph-mon[43961]: pgmap v1883: 37 pgs: 1 peering, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 577 KiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 353 op/s; 989/6 objects degraded (16483.333%); 4135/6 objects misplaced (68916.667%); 3.6 MiB/s, 251 objects/s recovering 2024-05-15T13:18:20.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:19 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:18:20.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:19 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:18:20.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:19 smithi120 ceph-mon[43961]: osdmap e725: 12 total, 12 up, 10 in 2024-05-15T13:18:20.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:19 smithi120 ceph-mon[43961]: 7.8 scrub starts 2024-05-15T13:18:20.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:19 smithi120 ceph-mon[43961]: 7.8 scrub ok 2024-05-15T13:18:20.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:19 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 989/6 objects degraded (16483.333%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-05-15T13:18:20.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:19 smithi120 ceph-mon[43961]: osdmap e726: 12 total, 12 up, 10 in 2024-05-15T13:18:20.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:19 smithi120 ceph-mon[43961]: 7.f scrub starts 2024-05-15T13:18:20.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:19 smithi120 ceph-mon[43961]: 7.f scrub ok 2024-05-15T13:18:21.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:21 smithi177 ceph-mon[44303]: osdmap e727: 12 total, 12 up, 10 in 2024-05-15T13:18:21.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:21 smithi177 ceph-mon[44303]: 7.4 deep-scrub starts 2024-05-15T13:18:21.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:21 smithi177 ceph-mon[44303]: 7.4 deep-scrub ok 2024-05-15T13:18:21.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:21 smithi177 ceph-mon[44303]: pgmap v1887: 37 pgs: 1 peering, 1 active+recovering+remapped, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 33 active+clean; 577 KiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 28 op/s; 86/6 objects degraded (1433.333%); 4135/6 objects misplaced (68916.667%); 69 KiB/s, 560 objects/s recovering 2024-05-15T13:18:21.512 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:21 smithi027 ceph-mon[39795]: osdmap e727: 12 total, 12 up, 10 in 2024-05-15T13:18:21.512 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:21 smithi027 ceph-mon[39795]: 7.4 deep-scrub starts 2024-05-15T13:18:21.512 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:21 smithi027 ceph-mon[39795]: 7.4 deep-scrub ok 2024-05-15T13:18:21.512 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:21 smithi027 ceph-mon[39795]: pgmap v1887: 37 pgs: 1 peering, 1 active+recovering+remapped, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 33 active+clean; 577 KiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 28 op/s; 86/6 objects degraded (1433.333%); 4135/6 objects misplaced (68916.667%); 69 KiB/s, 560 objects/s recovering 2024-05-15T13:18:21.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:21 smithi120 ceph-mon[43961]: osdmap e727: 12 total, 12 up, 10 in 2024-05-15T13:18:21.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:21 smithi120 ceph-mon[43961]: 7.4 deep-scrub starts 2024-05-15T13:18:21.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:21 smithi120 ceph-mon[43961]: 7.4 deep-scrub ok 2024-05-15T13:18:21.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:21 smithi120 ceph-mon[43961]: pgmap v1887: 37 pgs: 1 peering, 1 active+recovering+remapped, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 33 active+clean; 577 KiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 28 op/s; 86/6 objects degraded (1433.333%); 4135/6 objects misplaced (68916.667%); 69 KiB/s, 560 objects/s recovering 2024-05-15T13:18:22.190 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:18:22.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:22 smithi177 ceph-mon[44303]: osdmap e728: 12 total, 12 up, 10 in 2024-05-15T13:18:22.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:22 smithi177 ceph-mon[44303]: 7.e scrub starts 2024-05-15T13:18:22.527 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:22 smithi027 ceph-mon[39795]: osdmap e728: 12 total, 12 up, 10 in 2024-05-15T13:18:22.527 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:22 smithi027 ceph-mon[39795]: 7.e scrub starts 2024-05-15T13:18:22.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:22 smithi120 ceph-mon[43961]: osdmap e728: 12 total, 12 up, 10 in 2024-05-15T13:18:22.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:22 smithi120 ceph-mon[43961]: 7.e scrub starts 2024-05-15T13:18:23.312 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:18:23.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:23.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:18:23.417 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:18:23.426 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:23 smithi027 ceph-mon[39795]: 7.e scrub ok 2024-05-15T13:18:23.426 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:23 smithi027 ceph-mon[39795]: osdmap e729: 12 total, 12 up, 10 in 2024-05-15T13:18:23.426 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:23 smithi027 ceph-mon[39795]: 7.6 scrub starts 2024-05-15T13:18:23.426 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:23 smithi027 ceph-mon[39795]: 7.6 scrub ok 2024-05-15T13:18:23.426 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:23 smithi027 ceph-mon[39795]: pgmap v1890: 37 pgs: 1 peering, 1 active+recovering+remapped, 2 active+undersized+remapped, 33 active+clean; 577 KiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 4135/6 objects misplaced (68916.667%); 0 B/s, 561 objects/s recovering 2024-05-15T13:18:23.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:23 smithi177 ceph-mon[44303]: 7.e scrub ok 2024-05-15T13:18:23.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:23 smithi177 ceph-mon[44303]: osdmap e729: 12 total, 12 up, 10 in 2024-05-15T13:18:23.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:23 smithi177 ceph-mon[44303]: 7.6 scrub starts 2024-05-15T13:18:23.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:23 smithi177 ceph-mon[44303]: 7.6 scrub ok 2024-05-15T13:18:23.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:23 smithi177 ceph-mon[44303]: pgmap v1890: 37 pgs: 1 peering, 1 active+recovering+remapped, 2 active+undersized+remapped, 33 active+clean; 577 KiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 4135/6 objects misplaced (68916.667%); 0 B/s, 561 objects/s recovering 2024-05-15T13:18:23.447 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:23.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:23 smithi120 ceph-mon[43961]: 7.e scrub ok 2024-05-15T13:18:23.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:23 smithi120 ceph-mon[43961]: osdmap e729: 12 total, 12 up, 10 in 2024-05-15T13:18:23.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:23 smithi120 ceph-mon[43961]: 7.6 scrub starts 2024-05-15T13:18:23.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:23 smithi120 ceph-mon[43961]: 7.6 scrub ok 2024-05-15T13:18:23.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:23 smithi120 ceph-mon[43961]: pgmap v1890: 37 pgs: 1 peering, 1 active+recovering+remapped, 2 active+undersized+remapped, 33 active+clean; 577 KiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 4135/6 objects misplaced (68916.667%); 0 B/s, 561 objects/s recovering 2024-05-15T13:18:24.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:24 smithi177 ceph-mon[44303]: osdmap e730: 12 total, 12 up, 10 in 2024-05-15T13:18:24.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:24 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-05-15T13:18:24.527 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:24 smithi027 ceph-mon[39795]: osdmap e730: 12 total, 12 up, 10 in 2024-05-15T13:18:24.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:24 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-05-15T13:18:24.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:24 smithi120 ceph-mon[43961]: osdmap e730: 12 total, 12 up, 10 in 2024-05-15T13:18:24.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:24 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-05-15T13:18:25.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:25 smithi177 ceph-mon[44303]: pgmap v1892: 37 pgs: 1 active+recovering+remapped, 1 active+undersized+remapped, 35 active+clean; 577 KiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 4043/6 objects misplaced (67383.333%); 0 B/s, 420 objects/s recovering 2024-05-15T13:18:25.527 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:25 smithi027 ceph-mon[39795]: pgmap v1892: 37 pgs: 1 active+recovering+remapped, 1 active+undersized+remapped, 35 active+clean; 577 KiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 4043/6 objects misplaced (67383.333%); 0 B/s, 420 objects/s recovering 2024-05-15T13:18:25.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:25 smithi120 ceph-mon[43961]: pgmap v1892: 37 pgs: 1 active+recovering+remapped, 1 active+undersized+remapped, 35 active+clean; 577 KiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 4043/6 objects misplaced (67383.333%); 0 B/s, 420 objects/s recovering 2024-05-15T13:18:26.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:26 smithi177 ceph-mon[44303]: 1.5 deep-scrub starts 2024-05-15T13:18:26.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:26 smithi177 ceph-mon[44303]: 1.5 deep-scrub ok 2024-05-15T13:18:26.480 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:18:26.483 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:18:26.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:26 smithi027 ceph-mon[39795]: 1.5 deep-scrub starts 2024-05-15T13:18:26.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:26 smithi027 ceph-mon[39795]: 1.5 deep-scrub ok 2024-05-15T13:18:26.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:26.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:18:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:18:26.527 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:18:26.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:26 smithi120 ceph-mon[43961]: 1.5 deep-scrub starts 2024-05-15T13:18:26.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:26 smithi120 ceph-mon[43961]: 1.5 deep-scrub ok 2024-05-15T13:18:26.586 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:18:26.618 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:26.640 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:18:26.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:26.643 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:18:26.744 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:18:26.770 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:26.790 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:18:26.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:26.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:18:26.894 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:18:26.925 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:26.949 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:18:26.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:26.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:18:27.054 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:18:27.086 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:27.108 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:18:27.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:27.111 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:18:27.212 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:18:27.238 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:27.256 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:18:27.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:27.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:18:27.360 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:18:27.390 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:27 smithi177 ceph-mon[44303]: pgmap v1893: 37 pgs: 1 active+recovering+remapped, 1 active+undersized+remapped, 35 active+clean; 577 KiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 4043/6 objects misplaced (67383.333%); 0 B/s, 309 objects/s recovering 2024-05-15T13:18:27.392 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:27.414 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:18:27.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:27.417 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:18:27.485 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:27 smithi027 ceph-mon[39795]: pgmap v1893: 37 pgs: 1 active+recovering+remapped, 1 active+undersized+remapped, 35 active+clean; 577 KiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 4043/6 objects misplaced (67383.333%); 0 B/s, 309 objects/s recovering 2024-05-15T13:18:27.518 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:18:27.537 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:27 smithi120 ceph-mon[43961]: pgmap v1893: 37 pgs: 1 active+recovering+remapped, 1 active+undersized+remapped, 35 active+clean; 577 KiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 4043/6 objects misplaced (67383.333%); 0 B/s, 309 objects/s recovering 2024-05-15T13:18:27.545 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:27.564 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:18:27.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:27.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:18:27.668 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:18:27.696 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:28.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:28 smithi177 ceph-mon[44303]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-05-15T13:18:28.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:28 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T13:18:29.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:28 smithi027 ceph-mon[39795]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-05-15T13:18:29.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:28 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T13:18:29.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:28 smithi120 ceph-mon[43961]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-05-15T13:18:29.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:28 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T13:18:29.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:29 smithi027 ceph-mon[39795]: pgmap v1894: 37 pgs: 1 active+recovering+remapped, 36 active+clean; 577 KiB data, 8.6 GiB used, 885 GiB / 894 GiB avail; 3340/6 objects misplaced (55666.667%); 0 B/s, 345 objects/s recovering 2024-05-15T13:18:29.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:29 smithi177 ceph-mon[44303]: pgmap v1894: 37 pgs: 1 active+recovering+remapped, 36 active+clean; 577 KiB data, 8.6 GiB used, 885 GiB / 894 GiB avail; 3340/6 objects misplaced (55666.667%); 0 B/s, 345 objects/s recovering 2024-05-15T13:18:30.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:29 smithi120 ceph-mon[43961]: pgmap v1894: 37 pgs: 1 active+recovering+remapped, 36 active+clean; 577 KiB data, 8.6 GiB used, 885 GiB / 894 GiB avail; 3340/6 objects misplaced (55666.667%); 0 B/s, 345 objects/s recovering 2024-05-15T13:18:30.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:30 smithi177 ceph-mon[44303]: 7.1 scrub starts 2024-05-15T13:18:30.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:30 smithi177 ceph-mon[44303]: 7.1 scrub ok 2024-05-15T13:18:31.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:30 smithi027 ceph-mon[39795]: 7.1 scrub starts 2024-05-15T13:18:31.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:30 smithi027 ceph-mon[39795]: 7.1 scrub ok 2024-05-15T13:18:31.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:30 smithi120 ceph-mon[43961]: 7.1 scrub starts 2024-05-15T13:18:31.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:30 smithi120 ceph-mon[43961]: 7.1 scrub ok 2024-05-15T13:18:31.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:31 smithi177 ceph-mon[44303]: 1.11 scrub starts 2024-05-15T13:18:31.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:31 smithi177 ceph-mon[44303]: 1.11 scrub ok 2024-05-15T13:18:31.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:31 smithi177 ceph-mon[44303]: pgmap v1895: 37 pgs: 1 active+recovering+remapped, 36 active+clean; 577 KiB data, 8.5 GiB used, 885 GiB / 894 GiB avail; 3340/6 objects misplaced (55666.667%); 0 B/s, 167 objects/s recovering 2024-05-15T13:18:31.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:31 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:18:31.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:31 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:18:32.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:31 smithi027 ceph-mon[39795]: 1.11 scrub starts 2024-05-15T13:18:32.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:31 smithi027 ceph-mon[39795]: 1.11 scrub ok 2024-05-15T13:18:32.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:31 smithi027 ceph-mon[39795]: pgmap v1895: 37 pgs: 1 active+recovering+remapped, 36 active+clean; 577 KiB data, 8.5 GiB used, 885 GiB / 894 GiB avail; 3340/6 objects misplaced (55666.667%); 0 B/s, 167 objects/s recovering 2024-05-15T13:18:32.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:31 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:18:32.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:31 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:18:32.037 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:31 smithi120 ceph-mon[43961]: 1.11 scrub starts 2024-05-15T13:18:32.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:31 smithi120 ceph-mon[43961]: 1.11 scrub ok 2024-05-15T13:18:32.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:31 smithi120 ceph-mon[43961]: pgmap v1895: 37 pgs: 1 active+recovering+remapped, 36 active+clean; 577 KiB data, 8.5 GiB used, 885 GiB / 894 GiB avail; 3340/6 objects misplaced (55666.667%); 0 B/s, 167 objects/s recovering 2024-05-15T13:18:32.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:31 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:18:32.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:31 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:18:32.950 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:18:32.953 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:32 smithi027 ceph-mon[39795]: osdmap e731: 12 total, 12 up, 10 in 2024-05-15T13:18:32.953 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:32 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2590517436' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:18:32.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:32.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:18:33.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:32 smithi120 ceph-mon[43961]: osdmap e731: 12 total, 12 up, 10 in 2024-05-15T13:18:33.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:32 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2590517436' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:18:33.055 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:18:33.082 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:33.101 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:18:33.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:33.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:18:33.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:32 smithi177 ceph-mon[44303]: osdmap e731: 12 total, 12 up, 10 in 2024-05-15T13:18:33.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:32 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2590517436' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:18:33.205 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:18:33.236 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:34.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:33 smithi027 ceph-mon[39795]: pgmap v1897: 37 pgs: 1 active+recovering+remapped, 36 active+clean; 577 KiB data, 8.5 GiB used, 885 GiB / 894 GiB avail; 3340/6 objects misplaced (55666.667%); 0 B/s, 74 objects/s recovering 2024-05-15T13:18:34.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:33 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:18:34.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:33 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2590517436' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:18:34.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:33 smithi027 ceph-mon[39795]: osdmap e732: 12 total, 12 up, 10 in 2024-05-15T13:18:34.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:33 smithi120 ceph-mon[43961]: pgmap v1897: 37 pgs: 1 active+recovering+remapped, 36 active+clean; 577 KiB data, 8.5 GiB used, 885 GiB / 894 GiB avail; 3340/6 objects misplaced (55666.667%); 0 B/s, 74 objects/s recovering 2024-05-15T13:18:34.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:33 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:18:34.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:33 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2590517436' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:18:34.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:33 smithi120 ceph-mon[43961]: osdmap e732: 12 total, 12 up, 10 in 2024-05-15T13:18:34.088 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:18:34.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:34.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:18:34.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:33 smithi177 ceph-mon[44303]: pgmap v1897: 37 pgs: 1 active+recovering+remapped, 36 active+clean; 577 KiB data, 8.5 GiB used, 885 GiB / 894 GiB avail; 3340/6 objects misplaced (55666.667%); 0 B/s, 74 objects/s recovering 2024-05-15T13:18:34.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:33 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:18:34.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:33 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2590517436' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:18:34.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:33 smithi177 ceph-mon[44303]: osdmap e732: 12 total, 12 up, 10 in 2024-05-15T13:18:34.191 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:18:34.223 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:34.245 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:18:34.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:34.249 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:18:34.350 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:18:34.379 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:34.400 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:18:34.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:34.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:18:34.505 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:18:34.535 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:34.556 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:18:34.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:34.560 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:18:34.661 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:18:34.688 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:34.707 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:18:34.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:34.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:18:34.810 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:18:34.837 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:34 smithi177 ceph-mon[44303]: 7.9 deep-scrub starts 2024-05-15T13:18:34.837 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:34 smithi177 ceph-mon[44303]: 7.9 deep-scrub ok 2024-05-15T13:18:34.839 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:34.859 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:18:34.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:34.863 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:18:34.963 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:18:34.991 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:34 smithi027 ceph-mon[39795]: 7.9 deep-scrub starts 2024-05-15T13:18:34.991 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:34 smithi027 ceph-mon[39795]: 7.9 deep-scrub ok 2024-05-15T13:18:34.994 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:35.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:34 smithi120 ceph-mon[43961]: 7.9 deep-scrub starts 2024-05-15T13:18:35.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:34 smithi120 ceph-mon[43961]: 7.9 deep-scrub ok 2024-05-15T13:18:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:18:35.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:35.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:18:35.198 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:18:35.225 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:35.244 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:18:35.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:35.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:18:35.348 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:18:35.374 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:35.394 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:18:35.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:35.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:18:35.438 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:18:35.497 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:18:35.529 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:36.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:35 smithi027 ceph-mon[39795]: 1.2 scrub starts 2024-05-15T13:18:36.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:35 smithi027 ceph-mon[39795]: pgmap v1899: 37 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 35 active+clean; 577 KiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 2434/6 objects misplaced (40566.667%); 0 B/s, 201 objects/s recovering 2024-05-15T13:18:36.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:35 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:18:36.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:35 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:18:36.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:35 smithi120 ceph-mon[43961]: 1.2 scrub starts 2024-05-15T13:18:36.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:35 smithi120 ceph-mon[43961]: pgmap v1899: 37 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 35 active+clean; 577 KiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 2434/6 objects misplaced (40566.667%); 0 B/s, 201 objects/s recovering 2024-05-15T13:18:36.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:35 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:18:36.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:35 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:18:36.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:35 smithi177 ceph-mon[44303]: 1.2 scrub starts 2024-05-15T13:18:36.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:35 smithi177 ceph-mon[44303]: pgmap v1899: 37 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 35 active+clean; 577 KiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 2434/6 objects misplaced (40566.667%); 0 B/s, 201 objects/s recovering 2024-05-15T13:18:36.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:35 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:18:36.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:35 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:18:36.683 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:18:36.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:36.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:18:36.792 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:18:36.822 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:37.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:36 smithi027 ceph-mon[39795]: osdmap e733: 12 total, 12 up, 10 in 2024-05-15T13:18:37.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:36 smithi120 ceph-mon[43961]: osdmap e733: 12 total, 12 up, 10 in 2024-05-15T13:18:37.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:36 smithi177 ceph-mon[44303]: osdmap e733: 12 total, 12 up, 10 in 2024-05-15T13:18:37.605 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:18:37.605 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T13:18:37.605 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T13:18:37.605 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:18:37.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.605 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:125717 7.d 7:b2d9b5ae:::benchmark_data_smithi177_134484_object16652:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e658)", 2024-05-15T13:18:37.605 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:14:51.454869+0000", 2024-05-15T13:18:37.605 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 226.15048699499999, 2024-05-15T13:18:37.605 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.406739046, 2024-05-15T13:18:37.606 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:18:37.606 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:18:37.606 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:18:37.606 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:18:37.606 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:18:37.606 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 125717 2024-05-15T13:18:37.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.606 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:18:37.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.606 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:18:37.606 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.454869+0000", 2024-05-15T13:18:37.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.607 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.607 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:18:37.607 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.454869+0000", 2024-05-15T13:18:37.607 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.607 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.607 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:18:37.607 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.454871+0000", 2024-05-15T13:18:37.607 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.635e-06 2024-05-15T13:18:37.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:18:37.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.454877+0000", 2024-05-15T13:18:37.608 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6749999999999999e-06 2024-05-15T13:18:37.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:18:37.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.454889+0000", 2024-05-15T13:18:37.608 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1970000000000001e-05 2024-05-15T13:18:37.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:18:37.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.454897+0000", 2024-05-15T13:18:37.609 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.089e-06 2024-05-15T13:18:37.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.609 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.609 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.609 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.458033+0000", 2024-05-15T13:18:37.609 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031358279999999998 2024-05-15T13:18:37.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.609 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.609 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:18:37.609 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.458047+0000", 2024-05-15T13:18:37.609 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4738e-05 2024-05-15T13:18:37.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.609 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.610 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.610 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.821644+0000", 2024-05-15T13:18:37.610 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.36359644 2024-05-15T13:18:37.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.610 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.610 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:18:37.610 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.824379+0000", 2024-05-15T13:18:37.610 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0027352140000000001 2024-05-15T13:18:37.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.610 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.610 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:18:37.610 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.824485+0000", 2024-05-15T13:18:37.610 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000106455 2024-05-15T13:18:37.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.611 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.611 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:18:37.611 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.835853+0000", 2024-05-15T13:18:37.611 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011367752 2024-05-15T13:18:37.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.611 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.611 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:18:37.611 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.839563+0000", 2024-05-15T13:18:37.611 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0037097570000000002 2024-05-15T13:18:37.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.611 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.611 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:18:37.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.858107+0000", 2024-05-15T13:18:37.612 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.018543922000000001 2024-05-15T13:18:37.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.612 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.612 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:18:37.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.858230+0000", 2024-05-15T13:18:37.612 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000122891 2024-05-15T13:18:37.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.612 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.612 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:18:37.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.861608+0000", 2024-05-15T13:18:37.612 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033786799999999998 2024-05-15T13:18:37.612 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.612 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:18:37.613 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.613 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.613 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:125735 7.d 7:b05257b3:::benchmark_data_smithi177_134484_object16670:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e658)", 2024-05-15T13:18:37.613 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:14:51.456192+0000", 2024-05-15T13:18:37.613 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 226.149164762, 2024-05-15T13:18:37.613 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4168959249999999, 2024-05-15T13:18:37.613 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:18:37.613 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:18:37.613 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:18:37.613 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:18:37.613 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:18:37.613 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 125735 2024-05-15T13:18:37.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.613 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:18:37.614 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.614 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:18:37.614 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.456192+0000", 2024-05-15T13:18:37.614 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.614 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.614 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:18:37.614 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.456192+0000", 2024-05-15T13:18:37.614 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.614 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.614 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:18:37.614 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.456193+0000", 2024-05-15T13:18:37.614 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.048e-06 2024-05-15T13:18:37.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.615 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:18:37.615 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.456197+0000", 2024-05-15T13:18:37.615 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8180000000000003e-06 2024-05-15T13:18:37.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.615 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:18:37.615 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.456211+0000", 2024-05-15T13:18:37.615 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3712e-05 2024-05-15T13:18:37.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.615 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:18:37.616 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.456221+0000", 2024-05-15T13:18:37.616 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.448e-06 2024-05-15T13:18:37.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.616 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.616 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.458136+0000", 2024-05-15T13:18:37.616 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001915284 2024-05-15T13:18:37.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.616 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:18:37.616 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.458141+0000", 2024-05-15T13:18:37.616 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6739999999999996e-06 2024-05-15T13:18:37.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.617 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.617 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.831711+0000", 2024-05-15T13:18:37.617 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3735702889999999 2024-05-15T13:18:37.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.617 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:18:37.617 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.831893+0000", 2024-05-15T13:18:37.617 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00018173399999999999 2024-05-15T13:18:37.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.617 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:18:37.617 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.832062+0000", 2024-05-15T13:18:37.617 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016907599999999999 2024-05-15T13:18:37.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.618 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:18:37.618 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.849770+0000", 2024-05-15T13:18:37.618 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.017708420999999998 2024-05-15T13:18:37.618 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.618 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.618 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:18:37.618 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.849790+0000", 2024-05-15T13:18:37.618 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9528999999999999e-05 2024-05-15T13:18:37.618 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.618 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.618 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:18:37.618 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.869256+0000", 2024-05-15T13:18:37.618 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.019466781999999998 2024-05-15T13:18:37.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.619 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.619 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:18:37.619 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.869307+0000", 2024-05-15T13:18:37.619 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0469000000000001e-05 2024-05-15T13:18:37.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.619 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.619 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:18:37.619 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.873088+0000", 2024-05-15T13:18:37.619 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0037806409999999999 2024-05-15T13:18:37.619 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.619 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:18:37.619 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.620 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.620 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:125785 7.d 7:b67150e4:::benchmark_data_smithi177_134484_object16720:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e658)", 2024-05-15T13:18:37.620 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:14:51.456238+0000", 2024-05-15T13:18:37.620 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 226.14911870099999, 2024-05-15T13:18:37.620 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4205718940000001, 2024-05-15T13:18:37.620 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:18:37.620 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:18:37.620 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:18:37.620 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:18:37.620 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:18:37.620 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 125785 2024-05-15T13:18:37.620 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.620 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:18:37.620 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.621 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:18:37.621 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.456238+0000", 2024-05-15T13:18:37.621 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.621 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.621 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:18:37.621 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.456238+0000", 2024-05-15T13:18:37.621 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.621 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.621 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:18:37.621 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.456238+0000", 2024-05-15T13:18:37.621 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.46e-07 2024-05-15T13:18:37.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.622 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.622 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:18:37.622 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.456241+0000", 2024-05-15T13:18:37.622 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6989999999999999e-06 2024-05-15T13:18:37.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.622 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.622 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:18:37.622 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.456247+0000", 2024-05-15T13:18:37.622 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4280000000000001e-06 2024-05-15T13:18:37.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.622 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.622 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:18:37.623 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.456252+0000", 2024-05-15T13:18:37.623 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9320000000000004e-06 2024-05-15T13:18:37.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.623 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.623 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.623 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.458156+0000", 2024-05-15T13:18:37.623 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019042130000000001 2024-05-15T13:18:37.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.623 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.623 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:18:37.623 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.458160+0000", 2024-05-15T13:18:37.623 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4999999999999999e-06 2024-05-15T13:18:37.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.623 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.624 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.624 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.839678+0000", 2024-05-15T13:18:37.624 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.381518359 2024-05-15T13:18:37.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.624 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.624 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:18:37.624 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.839833+0000", 2024-05-15T13:18:37.624 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015481499999999999 2024-05-15T13:18:37.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.624 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.624 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:18:37.624 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.839990+0000", 2024-05-15T13:18:37.624 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015724200000000001 2024-05-15T13:18:37.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.625 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.625 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:18:37.625 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.849841+0000", 2024-05-15T13:18:37.625 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0098508620000000002 2024-05-15T13:18:37.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.625 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.625 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:18:37.625 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.858006+0000", 2024-05-15T13:18:37.625 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0081649289999999996 2024-05-15T13:18:37.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.625 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.625 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:18:37.625 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.873208+0000", 2024-05-15T13:18:37.625 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.015201469 2024-05-15T13:18:37.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.626 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.626 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:18:37.626 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.873263+0000", 2024-05-15T13:18:37.626 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5220000000000003e-05 2024-05-15T13:18:37.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.626 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.626 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:18:37.626 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.876810+0000", 2024-05-15T13:18:37.626 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035467799999999998 2024-05-15T13:18:37.626 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.626 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:18:37.626 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.627 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.627 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:125796 7.d 7:b42237e7:::benchmark_data_smithi177_134484_object16731:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e658)", 2024-05-15T13:18:37.627 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:14:51.456258+0000", 2024-05-15T13:18:37.627 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 226.14909812499999, 2024-05-15T13:18:37.627 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.429812388, 2024-05-15T13:18:37.627 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:18:37.627 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:18:37.627 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:18:37.627 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:18:37.627 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:18:37.627 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 125796 2024-05-15T13:18:37.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.627 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:18:37.628 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.628 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:18:37.628 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.456258+0000", 2024-05-15T13:18:37.628 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.628 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.628 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:18:37.628 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.456258+0000", 2024-05-15T13:18:37.628 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.628 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.628 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:18:37.628 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.456259+0000", 2024-05-15T13:18:37.628 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0100000000000001e-07 2024-05-15T13:18:37.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.629 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.629 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:18:37.629 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.456261+0000", 2024-05-15T13:18:37.629 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0700000000000001e-06 2024-05-15T13:18:37.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.629 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.629 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:18:37.629 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.456266+0000", 2024-05-15T13:18:37.629 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2279999999999998e-06 2024-05-15T13:18:37.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.629 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.629 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:18:37.629 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.456270+0000", 2024-05-15T13:18:37.629 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3429999999999998e-06 2024-05-15T13:18:37.630 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.630 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.630 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.630 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.458183+0000", 2024-05-15T13:18:37.630 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019131910000000001 2024-05-15T13:18:37.630 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.630 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.630 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:18:37.630 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.458189+0000", 2024-05-15T13:18:37.631 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8799999999999996e-06 2024-05-15T13:18:37.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.631 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.631 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.631 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.844493+0000", 2024-05-15T13:18:37.631 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.386303504 2024-05-15T13:18:37.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.631 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.631 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:18:37.631 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.844646+0000", 2024-05-15T13:18:37.631 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000153547 2024-05-15T13:18:37.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.631 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.631 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:18:37.631 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.844791+0000", 2024-05-15T13:18:37.632 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014504000000000001 2024-05-15T13:18:37.632 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.632 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.632 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:18:37.632 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.853827+0000", 2024-05-15T13:18:37.632 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.009035342 2024-05-15T13:18:37.632 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.632 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.632 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:18:37.632 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.877561+0000", 2024-05-15T13:18:37.632 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.023734723999999999 2024-05-15T13:18:37.632 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.632 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.632 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:18:37.632 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.881640+0000", 2024-05-15T13:18:37.633 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0040785480000000004 2024-05-15T13:18:37.633 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.633 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.633 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:18:37.633 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.881705+0000", 2024-05-15T13:18:37.633 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5147999999999994e-05 2024-05-15T13:18:37.633 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.633 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.633 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:18:37.633 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.886071+0000", 2024-05-15T13:18:37.633 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0043655220000000002 2024-05-15T13:18:37.633 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.633 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:18:37.634 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.634 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.634 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.634 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:125800 7.d 7:bb4403ad:::benchmark_data_smithi177_134484_object16735:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e658)", 2024-05-15T13:18:37.634 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:14:51.456282+0000", 2024-05-15T13:18:37.634 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 226.14907432800001, 2024-05-15T13:18:37.634 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4330441679999999, 2024-05-15T13:18:37.634 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:18:37.634 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:18:37.634 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:18:37.634 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:18:37.634 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:18:37.634 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 125800 2024-05-15T13:18:37.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.635 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:18:37.635 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.635 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:18:37.635 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.456282+0000", 2024-05-15T13:18:37.635 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.635 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.635 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:18:37.635 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.456282+0000", 2024-05-15T13:18:37.635 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.635 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.635 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:18:37.636 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.456282+0000", 2024-05-15T13:18:37.636 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3599999999999999e-07 2024-05-15T13:18:37.636 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.636 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.636 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:18:37.636 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.456285+0000", 2024-05-15T13:18:37.636 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8439999999999998e-06 2024-05-15T13:18:37.636 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.636 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.636 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:18:37.636 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.456291+0000", 2024-05-15T13:18:37.636 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6450000000000002e-06 2024-05-15T13:18:37.636 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.636 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.637 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:18:37.637 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.456296+0000", 2024-05-15T13:18:37.637 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.831e-06 2024-05-15T13:18:37.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.637 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.637 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.637 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.458219+0000", 2024-05-15T13:18:37.637 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019230370000000001 2024-05-15T13:18:37.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.637 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.637 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:18:37.637 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:51.458225+0000", 2024-05-15T13:18:37.637 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4230000000000002e-06 2024-05-15T13:18:37.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.637 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.638 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.638 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.850340+0000", 2024-05-15T13:18:37.638 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.392114614 2024-05-15T13:18:37.638 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.638 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.638 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:18:37.638 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.850481+0000", 2024-05-15T13:18:37.638 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000141621 2024-05-15T13:18:37.638 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.638 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.638 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:18:37.638 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.850620+0000", 2024-05-15T13:18:37.638 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013860500000000001 2024-05-15T13:18:37.639 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.639 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.639 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:18:37.639 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.857886+0000", 2024-05-15T13:18:37.639 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0072659059999999999 2024-05-15T13:18:37.639 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.639 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.639 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:18:37.639 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.877633+0000", 2024-05-15T13:18:37.639 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.019746914000000001 2024-05-15T13:18:37.639 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.639 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.639 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:18:37.639 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.886173+0000", 2024-05-15T13:18:37.640 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0085405129999999996 2024-05-15T13:18:37.640 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.640 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.640 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:18:37.640 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.886234+0000", 2024-05-15T13:18:37.640 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0699999999999998e-05 2024-05-15T13:18:37.640 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.640 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.640 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:18:37.640 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:52.889326+0000", 2024-05-15T13:18:37.640 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003092179 2024-05-15T13:18:37.640 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.640 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:18:37.640 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.641 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.641 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:143314 7.d 7:b0b7e7e5:::benchmark_data_smithi177_134484_object34249:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e672)", 2024-05-15T13:18:37.641 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:15:33.204214+0000", 2024-05-15T13:18:37.641 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 184.401142462, 2024-05-15T13:18:37.641 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5849619740000001, 2024-05-15T13:18:37.641 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:18:37.641 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:18:37.641 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:18:37.641 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:18:37.641 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:18:37.641 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 143314 2024-05-15T13:18:37.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.641 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:18:37.642 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.642 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:18:37.642 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:33.204214+0000", 2024-05-15T13:18:37.642 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.642 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.642 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:18:37.642 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:33.204214+0000", 2024-05-15T13:18:37.642 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.642 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.642 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:18:37.642 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:33.204214+0000", 2024-05-15T13:18:37.642 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2300000000000002e-07 2024-05-15T13:18:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:18:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:33.204218+0000", 2024-05-15T13:18:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2940000000000001e-06 2024-05-15T13:18:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:18:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:33.204224+0000", 2024-05-15T13:18:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1929999999999998e-06 2024-05-15T13:18:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:18:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:33.204231+0000", 2024-05-15T13:18:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7469999999999999e-06 2024-05-15T13:18:37.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.644 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.644 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.644 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:33.223212+0000", 2024-05-15T13:18:37.644 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.018980961000000001 2024-05-15T13:18:37.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.644 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.644 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:18:37.644 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:33.223299+0000", 2024-05-15T13:18:37.644 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7614000000000005e-05 2024-05-15T13:18:37.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.644 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.644 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:34.789164+0000", 2024-05-15T13:18:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.565864945 2024-05-15T13:18:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:18:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:34.789176+0000", 2024-05-15T13:18:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1797000000000001e-05 2024-05-15T13:18:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:18:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:143945 7.6 7:6ea5385a:::benchmark_data_smithi177_134484_object34880:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e674)", 2024-05-15T13:18:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:15:34.747228+0000", 2024-05-15T13:18:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 182.85812860499999, 2024-05-15T13:18:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.321169957, 2024-05-15T13:18:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:18:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:18:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:18:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:18:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:18:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 143945 2024-05-15T13:18:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:18:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:18:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:34.747228+0000", 2024-05-15T13:18:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.647 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.647 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:18:37.647 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:34.747228+0000", 2024-05-15T13:18:37.647 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.647 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.647 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:18:37.647 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:34.747228+0000", 2024-05-15T13:18:37.647 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7e-07 2024-05-15T13:18:37.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.647 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.647 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:18:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:34.747230+0000", 2024-05-15T13:18:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2800000000000002e-06 2024-05-15T13:18:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:18:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:34.747236+0000", 2024-05-15T13:18:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6949999999999997e-06 2024-05-15T13:18:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:18:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:34.747241+0000", 2024-05-15T13:18:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9370000000000003e-06 2024-05-15T13:18:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:34.747479+0000", 2024-05-15T13:18:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000238225 2024-05-15T13:18:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:18:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:34.747565+0000", 2024-05-15T13:18:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5215000000000003e-05 2024-05-15T13:18:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:36.068385+0000", 2024-05-15T13:18:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3208209319999999 2024-05-15T13:18:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.650 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.650 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:18:37.650 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:36.068398+0000", 2024-05-15T13:18:37.650 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2303000000000001e-05 2024-05-15T13:18:37.650 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.650 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:18:37.650 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.650 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.650 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.650 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:197998 7.4 7:29af6335:::benchmark_data_smithi177_134484_object88933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e713)", 2024-05-15T13:18:37.650 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:17:36.732280+0000", 2024-05-15T13:18:37.650 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 60.873076009000002, 2024-05-15T13:18:37.650 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0520236970000001, 2024-05-15T13:18:37.650 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:18:37.651 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:18:37.651 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:18:37.651 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:18:37.651 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:18:37.651 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 197998 2024-05-15T13:18:37.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.651 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:18:37.651 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.651 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:18:37.651 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.732280+0000", 2024-05-15T13:18:37.651 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.651 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.651 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:18:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.732280+0000", 2024-05-15T13:18:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:18:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.732281+0000", 2024-05-15T13:18:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9499999999999998e-07 2024-05-15T13:18:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:18:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.732283+0000", 2024-05-15T13:18:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5450000000000002e-06 2024-05-15T13:18:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:18:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.732290+0000", 2024-05-15T13:18:37.653 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4729999999999997e-06 2024-05-15T13:18:37.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.653 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.653 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:18:37.653 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.732295+0000", 2024-05-15T13:18:37.653 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6779999999999997e-06 2024-05-15T13:18:37.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.653 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.653 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.653 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.732356+0000", 2024-05-15T13:18:37.653 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0906999999999998e-05 2024-05-15T13:18:37.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.653 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.653 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:18:37.654 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.732505+0000", 2024-05-15T13:18:37.654 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014865700000000001 2024-05-15T13:18:37.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.654 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.654 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,9", 2024-05-15T13:18:37.654 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.732634+0000", 2024-05-15T13:18:37.654 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012876000000000001 2024-05-15T13:18:37.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.654 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.654 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.654 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.743037+0000", 2024-05-15T13:18:37.654 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010403333000000001 2024-05-15T13:18:37.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.654 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:18:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.743050+0000", 2024-05-15T13:18:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3050999999999999e-05 2024-05-15T13:18:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.784291+0000", 2024-05-15T13:18:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.041241208 2024-05-15T13:18:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:18:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.784304+0000", 2024-05-15T13:18:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.269e-05 2024-05-15T13:18:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:18:37.656 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.656 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.656 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:198002 7.4 7.e8c35024 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e713)", 2024-05-15T13:18:37.656 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:17:36.738153+0000", 2024-05-15T13:18:37.656 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 60.867203285000002, 2024-05-15T13:18:37.656 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0563994759999999, 2024-05-15T13:18:37.656 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:18:37.656 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:18:37.656 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:18:37.656 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:18:37.656 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:18:37.656 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 198002 2024-05-15T13:18:37.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.656 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:18:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:18:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.738153+0000", 2024-05-15T13:18:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:18:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.738153+0000", 2024-05-15T13:18:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:18:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.738154+0000", 2024-05-15T13:18:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3769999999999999e-06 2024-05-15T13:18:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.658 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.658 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:18:37.658 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.738159+0000", 2024-05-15T13:18:37.658 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.651e-06 2024-05-15T13:18:37.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.658 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.658 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:18:37.658 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.738171+0000", 2024-05-15T13:18:37.658 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1792e-05 2024-05-15T13:18:37.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.658 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.658 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:18:37.658 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.738180+0000", 2024-05-15T13:18:37.658 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7299999999999994e-06 2024-05-15T13:18:37.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.659 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.659 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.659 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.743241+0000", 2024-05-15T13:18:37.659 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00506122 2024-05-15T13:18:37.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.659 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.659 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:18:37.659 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.743249+0000", 2024-05-15T13:18:37.659 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3359999999999995e-06 2024-05-15T13:18:37.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.659 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.659 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.659 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.794543+0000", 2024-05-15T13:18:37.660 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0512938540000001 2024-05-15T13:18:37.660 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.660 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.660 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:18:37.660 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.794553+0000", 2024-05-15T13:18:37.660 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.516e-06 2024-05-15T13:18:37.660 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.660 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:18:37.660 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.660 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.661 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.661 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:198003 7.4 7.a1de35c4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e713)", 2024-05-15T13:18:37.661 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:17:36.738195+0000", 2024-05-15T13:18:37.661 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 60.867161654, 2024-05-15T13:18:37.661 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.056436962, 2024-05-15T13:18:37.661 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:18:37.661 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:18:37.661 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:18:37.661 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:18:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:18:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 198003 2024-05-15T13:18:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:18:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:18:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.738195+0000", 2024-05-15T13:18:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:18:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.738195+0000", 2024-05-15T13:18:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:18:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.738195+0000", 2024-05-15T13:18:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2599999999999998e-07 2024-05-15T13:18:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:18:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.738198+0000", 2024-05-15T13:18:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4229999999999999e-06 2024-05-15T13:18:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:18:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.738204+0000", 2024-05-15T13:18:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3550000000000001e-06 2024-05-15T13:18:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:18:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.738209+0000", 2024-05-15T13:18:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5319999999999997e-06 2024-05-15T13:18:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.743845+0000", 2024-05-15T13:18:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0056355650000000004 2024-05-15T13:18:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:18:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.743854+0000", 2024-05-15T13:18:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3789999999999995e-06 2024-05-15T13:18:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.794625+0000", 2024-05-15T13:18:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.050770083 2024-05-15T13:18:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:18:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.794632+0000", 2024-05-15T13:18:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.199e-06 2024-05-15T13:18:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:18:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:198013 7.4 7.776a96f4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e714)", 2024-05-15T13:18:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:17:36.758925+0000", 2024-05-15T13:18:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 60.846431695, 2024-05-15T13:18:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.035730035, 2024-05-15T13:18:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:18:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:18:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:18:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:18:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:18:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 198013 2024-05-15T13:18:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:18:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:18:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.758925+0000", 2024-05-15T13:18:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:18:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.758925+0000", 2024-05-15T13:18:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:18:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.758926+0000", 2024-05-15T13:18:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2640000000000001e-06 2024-05-15T13:18:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:18:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.758932+0000", 2024-05-15T13:18:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6080000000000003e-06 2024-05-15T13:18:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:18:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.758941+0000", 2024-05-15T13:18:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1800000000000002e-06 2024-05-15T13:18:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:18:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.758958+0000", 2024-05-15T13:18:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7374e-05 2024-05-15T13:18:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.759464+0000", 2024-05-15T13:18:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00050630800000000002 2024-05-15T13:18:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:18:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.759474+0000", 2024-05-15T13:18:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8069999999999994e-06 2024-05-15T13:18:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.794648+0000", 2024-05-15T13:18:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0351739069999999 2024-05-15T13:18:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:18:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.794655+0000", 2024-05-15T13:18:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5869999999999998e-06 2024-05-15T13:18:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:18:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:198021 7.4 7.e4a21094 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e714)", 2024-05-15T13:18:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:17:36.769084+0000", 2024-05-15T13:18:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 60.836272379, 2024-05-15T13:18:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.025592815, 2024-05-15T13:18:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:18:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:18:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:18:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:18:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:18:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 198021 2024-05-15T13:18:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:18:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:18:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.769084+0000", 2024-05-15T13:18:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:18:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.769084+0000", 2024-05-15T13:18:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:18:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.769085+0000", 2024-05-15T13:18:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0299999999999997e-07 2024-05-15T13:18:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:18:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.769089+0000", 2024-05-15T13:18:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3710000000000002e-06 2024-05-15T13:18:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:18:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.769100+0000", 2024-05-15T13:18:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0913e-05 2024-05-15T13:18:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:18:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.769107+0000", 2024-05-15T13:18:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6919999999999996e-06 2024-05-15T13:18:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.674 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.674 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.674 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.773105+0000", 2024-05-15T13:18:37.674 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039984110000000003 2024-05-15T13:18:37.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.674 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.674 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:18:37.674 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.773120+0000", 2024-05-15T13:18:37.674 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5073e-05 2024-05-15T13:18:37.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.674 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.674 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.674 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.794671+0000", 2024-05-15T13:18:37.674 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0215501279999999 2024-05-15T13:18:37.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.675 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.675 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:18:37.675 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.794677+0000", 2024-05-15T13:18:37.675 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3239999999999999e-06 2024-05-15T13:18:37.675 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.675 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:18:37.675 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.675 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.675 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:198037 7.4 7.62cc1764 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e714)", 2024-05-15T13:18:37.675 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:17:36.817142+0000", 2024-05-15T13:18:37.676 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 60.788214807999999, 2024-05-15T13:18:37.676 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.977557495, 2024-05-15T13:18:37.676 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:18:37.676 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:18:37.676 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:18:37.676 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:18:37.676 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:18:37.676 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 198037 2024-05-15T13:18:37.676 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.676 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:18:37.676 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.676 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:18:37.676 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.817142+0000", 2024-05-15T13:18:37.677 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.677 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:18:37.677 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.817142+0000", 2024-05-15T13:18:37.677 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.677 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:18:37.677 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.817143+0000", 2024-05-15T13:18:37.677 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7179999999999999e-06 2024-05-15T13:18:37.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:18:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.817151+0000", 2024-05-15T13:18:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7610000000000007e-06 2024-05-15T13:18:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:18:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.817164+0000", 2024-05-15T13:18:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3134e-05 2024-05-15T13:18:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:18:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.817173+0000", 2024-05-15T13:18:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1509999999999993e-06 2024-05-15T13:18:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.833923+0000", 2024-05-15T13:18:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016749995 2024-05-15T13:18:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:18:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.833936+0000", 2024-05-15T13:18:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2291e-05 2024-05-15T13:18:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.680 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.794693+0000", 2024-05-15T13:18:37.680 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.96075725599999995 2024-05-15T13:18:37.680 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.680 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.680 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:18:37.680 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.794699+0000", 2024-05-15T13:18:37.680 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1890000000000004e-06 2024-05-15T13:18:37.680 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.680 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:18:37.680 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.680 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.680 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.680 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:198040 7.4 7.492e7144 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e714)", 2024-05-15T13:18:37.680 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:17:36.824697+0000", 2024-05-15T13:18:37.681 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 60.780659065000002, 2024-05-15T13:18:37.681 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.97002408600000001, 2024-05-15T13:18:37.681 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:18:37.681 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:18:37.681 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:18:37.681 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:18:37.681 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:18:37.681 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 198040 2024-05-15T13:18:37.681 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.681 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:18:37.681 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.681 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:18:37.681 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.824697+0000", 2024-05-15T13:18:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:18:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.824697+0000", 2024-05-15T13:18:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:18:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.824699+0000", 2024-05-15T13:18:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.539e-06 2024-05-15T13:18:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:18:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.824706+0000", 2024-05-15T13:18:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3309999999999998e-06 2024-05-15T13:18:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:18:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.824719+0000", 2024-05-15T13:18:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2536e-05 2024-05-15T13:18:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:18:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.824727+0000", 2024-05-15T13:18:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5730000000000008e-06 2024-05-15T13:18:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.833970+0000", 2024-05-15T13:18:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0092426060000000004 2024-05-15T13:18:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:18:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.833980+0000", 2024-05-15T13:18:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7149999999999993e-06 2024-05-15T13:18:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.794715+0000", 2024-05-15T13:18:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.96073496000000003 2024-05-15T13:18:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:18:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.794721+0000", 2024-05-15T13:18:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8260000000000004e-06 2024-05-15T13:18:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:18:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:198057 7.4 7.df786fa4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e714)", 2024-05-15T13:18:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:17:36.858248+0000", 2024-05-15T13:18:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 60.747108746000002, 2024-05-15T13:18:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.93649453400000005, 2024-05-15T13:18:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:18:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:18:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:18:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:18:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:18:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 198057 2024-05-15T13:18:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:18:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:18:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.858248+0000", 2024-05-15T13:18:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:18:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.858248+0000", 2024-05-15T13:18:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:18:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.858249+0000", 2024-05-15T13:18:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2139999999999999e-06 2024-05-15T13:18:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:18:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.858254+0000", 2024-05-15T13:18:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3449999999999996e-06 2024-05-15T13:18:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:18:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.858266+0000", 2024-05-15T13:18:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.169e-05 2024-05-15T13:18:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:18:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.858273+0000", 2024-05-15T13:18:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8979999999999999e-06 2024-05-15T13:18:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.860286+0000", 2024-05-15T13:18:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020130640000000002 2024-05-15T13:18:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:18:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.860297+0000", 2024-05-15T13:18:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1348e-05 2024-05-15T13:18:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.794737+0000", 2024-05-15T13:18:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.93443936400000005 2024-05-15T13:18:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:18:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.794742+0000", 2024-05-15T13:18:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6110000000000001e-06 2024-05-15T13:18:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:18:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:198037 7.4 7:26e83346:::benchmark_data_smithi177_134484_object88972:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e714)", 2024-05-15T13:18:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:17:36.957822+0000", 2024-05-15T13:18:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 60.647534862000001, 2024-05-15T13:18:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.90117990699999995, 2024-05-15T13:18:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:18:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:18:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:18:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:18:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:18:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 198037 2024-05-15T13:18:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:18:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:18:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957822+0000", 2024-05-15T13:18:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:18:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957822+0000", 2024-05-15T13:18:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:18:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957822+0000", 2024-05-15T13:18:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.46e-07 2024-05-15T13:18:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:18:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957824+0000", 2024-05-15T13:18:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.88e-06 2024-05-15T13:18:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:18:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957829+0000", 2024-05-15T13:18:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3140000000000003e-06 2024-05-15T13:18:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:18:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957836+0000", 2024-05-15T13:18:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8589999999999999e-06 2024-05-15T13:18:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.693 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.693 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.693 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.958148+0000", 2024-05-15T13:18:37.693 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000312324 2024-05-15T13:18:37.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.693 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.693 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:18:37.693 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.958155+0000", 2024-05-15T13:18:37.693 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6479999999999997e-06 2024-05-15T13:18:37.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.693 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.693 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.693 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.837907+0000", 2024-05-15T13:18:37.693 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.879752384 2024-05-15T13:18:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:18:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.838060+0000", 2024-05-15T13:18:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015273500000000001 2024-05-15T13:18:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,9", 2024-05-15T13:18:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.838227+0000", 2024-05-15T13:18:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016722300000000001 2024-05-15T13:18:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:18:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.843473+0000", 2024-05-15T13:18:37.695 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0052460639999999999 2024-05-15T13:18:37.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.695 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.695 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:18:37.695 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.843780+0000", 2024-05-15T13:18:37.695 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000306543 2024-05-15T13:18:37.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.695 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.695 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:18:37.695 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.858852+0000", 2024-05-15T13:18:37.695 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.01507231 2024-05-15T13:18:37.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.695 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.695 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:18:37.696 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.858908+0000", 2024-05-15T13:18:37.696 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6091000000000002e-05 2024-05-15T13:18:37.696 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.696 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.696 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:18:37.696 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.859001+0000", 2024-05-15T13:18:37.696 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3185999999999999e-05 2024-05-15T13:18:37.696 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.696 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:18:37.696 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.696 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.696 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.696 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:198040 7.4 7:228e7492:::benchmark_data_smithi177_134484_object88975:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e714)", 2024-05-15T13:18:37.696 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:17:36.957841+0000", 2024-05-15T13:18:37.696 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 60.647514948999998, 2024-05-15T13:18:37.697 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.92288215299999998, 2024-05-15T13:18:37.697 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:18:37.697 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:18:37.697 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:18:37.697 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:18:37.697 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:18:37.697 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 198040 2024-05-15T13:18:37.697 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.697 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:18:37.697 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.697 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:18:37.697 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957841+0000", 2024-05-15T13:18:37.697 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.697 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.697 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.698 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:18:37.698 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957841+0000", 2024-05-15T13:18:37.698 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.698 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.698 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:18:37.698 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957842+0000", 2024-05-15T13:18:37.698 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8999999999999998e-07 2024-05-15T13:18:37.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.698 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.698 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:18:37.698 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957844+0000", 2024-05-15T13:18:37.698 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.007e-06 2024-05-15T13:18:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:18:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957850+0000", 2024-05-15T13:18:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.516e-06 2024-05-15T13:18:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:18:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957855+0000", 2024-05-15T13:18:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0209999999999999e-06 2024-05-15T13:18:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.958180+0000", 2024-05-15T13:18:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000324891 2024-05-15T13:18:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:18:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.958187+0000", 2024-05-15T13:18:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6320000000000002e-06 2024-05-15T13:18:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.843819+0000", 2024-05-15T13:18:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.88563178300000001 2024-05-15T13:18:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:18:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.843924+0000", 2024-05-15T13:18:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000105052 2024-05-15T13:18:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,9", 2024-05-15T13:18:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.844035+0000", 2024-05-15T13:18:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000111 2024-05-15T13:18:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:18:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.859042+0000", 2024-05-15T13:18:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.015007761 2024-05-15T13:18:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:18:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.869878+0000", 2024-05-15T13:18:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010836083 2024-05-15T13:18:37.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.702 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.702 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:18:37.702 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.880571+0000", 2024-05-15T13:18:37.702 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010692434000000001 2024-05-15T13:18:37.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.702 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.702 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:18:37.702 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.880627+0000", 2024-05-15T13:18:37.702 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6070999999999999e-05 2024-05-15T13:18:37.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.702 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.702 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:18:37.702 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.880724+0000", 2024-05-15T13:18:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6612000000000003e-05 2024-05-15T13:18:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:18:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:198057 7.4 7:25f61efb:::benchmark_data_smithi177_134484_object88992:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e714)", 2024-05-15T13:18:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:17:36.957861+0000", 2024-05-15T13:18:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 60.647495534000001, 2024-05-15T13:18:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.92303684100000005, 2024-05-15T13:18:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:18:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:18:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:18:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:18:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:18:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 198057 2024-05-15T13:18:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:18:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:18:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957861+0000", 2024-05-15T13:18:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:18:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957861+0000", 2024-05-15T13:18:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:18:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957861+0000", 2024-05-15T13:18:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8999999999999998e-07 2024-05-15T13:18:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:18:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957863+0000", 2024-05-15T13:18:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.877e-06 2024-05-15T13:18:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:18:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957869+0000", 2024-05-15T13:18:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4609999999999996e-06 2024-05-15T13:18:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:18:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957874+0000", 2024-05-15T13:18:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8389999999999997e-06 2024-05-15T13:18:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.958215+0000", 2024-05-15T13:18:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00034055400000000002 2024-05-15T13:18:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:18:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.958222+0000", 2024-05-15T13:18:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.63e-06 2024-05-15T13:18:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.848922+0000", 2024-05-15T13:18:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.89070034099999995 2024-05-15T13:18:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:18:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.849067+0000", 2024-05-15T13:18:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014552599999999999 2024-05-15T13:18:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,9", 2024-05-15T13:18:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.849210+0000", 2024-05-15T13:18:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000142275 2024-05-15T13:18:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:18:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.859082+0000", 2024-05-15T13:18:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0098719099999999994 2024-05-15T13:18:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:18:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.874931+0000", 2024-05-15T13:18:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.015849510000000001 2024-05-15T13:18:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:18:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.880769+0000", 2024-05-15T13:18:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0058382970000000001 2024-05-15T13:18:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:18:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.880815+0000", 2024-05-15T13:18:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5550999999999998e-05 2024-05-15T13:18:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:18:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.880898+0000", 2024-05-15T13:18:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2780000000000004e-05 2024-05-15T13:18:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.710 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:18:37.710 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.710 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.710 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.710 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:198080 7.4 7:26285b82:::benchmark_data_smithi177_134484_object89015:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e714)", 2024-05-15T13:18:37.710 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:17:36.957880+0000", 2024-05-15T13:18:37.710 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 60.647476689000001, 2024-05-15T13:18:37.710 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.923195871, 2024-05-15T13:18:37.710 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:18:37.710 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:18:37.710 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:18:37.710 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:18:37.710 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:18:37.710 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 198080 2024-05-15T13:18:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:18:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:18:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957880+0000", 2024-05-15T13:18:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:18:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957880+0000", 2024-05-15T13:18:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:18:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957880+0000", 2024-05-15T13:18:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1300000000000001e-07 2024-05-15T13:18:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:18:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957882+0000", 2024-05-15T13:18:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.792e-06 2024-05-15T13:18:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:18:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957888+0000", 2024-05-15T13:18:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4790000000000001e-06 2024-05-15T13:18:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:18:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957893+0000", 2024-05-15T13:18:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6160000000000004e-06 2024-05-15T13:18:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.958243+0000", 2024-05-15T13:18:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00035042599999999998 2024-05-15T13:18:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:18:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.958250+0000", 2024-05-15T13:18:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8260000000000004e-06 2024-05-15T13:18:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.854018+0000", 2024-05-15T13:18:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.89576776400000002 2024-05-15T13:18:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:18:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.854169+0000", 2024-05-15T13:18:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015109800000000001 2024-05-15T13:18:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,9", 2024-05-15T13:18:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.854305+0000", 2024-05-15T13:18:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000135773 2024-05-15T13:18:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:18:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.869863+0000", 2024-05-15T13:18:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.015557764 2024-05-15T13:18:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:18:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.874937+0000", 2024-05-15T13:18:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050748099999999999 2024-05-15T13:18:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:18:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.880947+0000", 2024-05-15T13:18:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0060099159999999997 2024-05-15T13:18:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:18:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.880993+0000", 2024-05-15T13:18:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5920000000000001e-05 2024-05-15T13:18:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:18:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.881076+0000", 2024-05-15T13:18:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2373999999999994e-05 2024-05-15T13:18:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:18:37.717 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.717 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.717 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.717 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:198096 7.4 7:20d87625:::benchmark_data_smithi177_134484_object89031:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e714)", 2024-05-15T13:18:37.717 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:17:36.957927+0000", 2024-05-15T13:18:37.717 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 60.647429441, 2024-05-15T13:18:37.717 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.92332141300000004, 2024-05-15T13:18:37.717 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:18:37.717 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:18:37.717 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:18:37.717 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:18:37.717 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:18:37.717 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 198096 2024-05-15T13:18:37.717 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.718 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:18:37.718 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.718 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:18:37.718 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957927+0000", 2024-05-15T13:18:37.718 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.718 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.718 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.718 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:18:37.718 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957927+0000", 2024-05-15T13:18:37.718 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:18:37.718 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.718 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.718 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:18:37.718 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957927+0000", 2024-05-15T13:18:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5499999999999999e-07 2024-05-15T13:18:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:18:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.957930+0000", 2024-05-15T13:18:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.261e-06 2024-05-15T13:18:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:18:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.958067+0000", 2024-05-15T13:18:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013748100000000001 2024-05-15T13:18:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:18:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.958074+0000", 2024-05-15T13:18:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0809999999999996e-06 2024-05-15T13:18:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.958373+0000", 2024-05-15T13:18:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00029887300000000001 2024-05-15T13:18:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:18:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.958381+0000", 2024-05-15T13:18:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4290000000000001e-06 2024-05-15T13:18:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:18:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.859119+0000", 2024-05-15T13:18:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.90073717600000003 2024-05-15T13:18:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:18:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.859227+0000", 2024-05-15T13:18:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010818400000000001 2024-05-15T13:18:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,9", 2024-05-15T13:18:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.859500+0000", 2024-05-15T13:18:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00027309600000000002 2024-05-15T13:18:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:18:37.722 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.869920+0000", 2024-05-15T13:18:37.722 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010420211 2024-05-15T13:18:37.722 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.722 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.722 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:18:37.722 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.874943+0000", 2024-05-15T13:18:37.722 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050225699999999996 2024-05-15T13:18:37.722 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.722 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.722 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:18:37.722 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.881120+0000", 2024-05-15T13:18:37.722 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0061772279999999999 2024-05-15T13:18:37.722 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.723 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.723 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:18:37.723 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.881166+0000", 2024-05-15T13:18:37.723 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6346e-05 2024-05-15T13:18:37.723 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:18:37.723 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:18:37.723 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:18:37.723 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.881248+0000", 2024-05-15T13:18:37.723 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2121999999999997e-05 2024-05-15T13:18:37.723 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.723 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:18:37.723 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.723 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:18:37.723 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:18:37.723 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:18:38.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:37 smithi027 ceph-mon[39795]: pgmap v1901: 37 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 35 active+clean; 577 KiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 2434/6 objects misplaced (40566.667%); 0 B/s, 150 objects/s recovering 2024-05-15T13:18:38.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:37 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:18:38.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:37 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:18:38.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:37 smithi027 ceph-mon[39795]: osdmap e734: 12 total, 12 up, 10 in 2024-05-15T13:18:38.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:37 smithi120 ceph-mon[43961]: pgmap v1901: 37 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 35 active+clean; 577 KiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 2434/6 objects misplaced (40566.667%); 0 B/s, 150 objects/s recovering 2024-05-15T13:18:38.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:37 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:18:38.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:37 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:18:38.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:37 smithi120 ceph-mon[43961]: osdmap e734: 12 total, 12 up, 10 in 2024-05-15T13:18:38.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:37 smithi177 ceph-mon[44303]: pgmap v1901: 37 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 35 active+clean; 577 KiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 2434/6 objects misplaced (40566.667%); 0 B/s, 150 objects/s recovering 2024-05-15T13:18:38.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:37 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:18:38.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:37 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:18:38.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:37 smithi177 ceph-mon[44303]: osdmap e734: 12 total, 12 up, 10 in 2024-05-15T13:18:39.806 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:18:39.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:39.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:18:39.910 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:18:39.942 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:39.964 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:18:39.966 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:39 smithi177 ceph-mon[44303]: pgmap v1903: 37 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 35 active+clean; 577 KiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 1085/6 objects misplaced (18083.333%); 0 B/s, 375 objects/s recovering 2024-05-15T13:18:39.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:39.967 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:18:40.008 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:39 smithi027 ceph-mon[39795]: pgmap v1903: 37 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 35 active+clean; 577 KiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 1085/6 objects misplaced (18083.333%); 0 B/s, 375 objects/s recovering 2024-05-15T13:18:40.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:39 smithi120 ceph-mon[43961]: pgmap v1903: 37 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 35 active+clean; 577 KiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 1085/6 objects misplaced (18083.333%); 0 B/s, 375 objects/s recovering 2024-05-15T13:18:40.068 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:18:40.094 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:40.114 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:18:40.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:40.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:18:40.218 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:18:40.250 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:40.271 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:18:40.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:40.274 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:18:40.375 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:18:40.401 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:40.421 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:18:40.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:40.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:18:40.525 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:18:40.552 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:40.572 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:18:40.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:40.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:18:40.676 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:18:40.708 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:40.731 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:18:40.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:40.736 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:18:40.737 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_5' removed 2024-05-15T13:18:40.837 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:18:40.863 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:40.881 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:18:40.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:40.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:18:40.984 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:18:41.021 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:40 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3611086116' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T13:18:41.023 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:41.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:40 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3611086116' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T13:18:41.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:40 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3611086116' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T13:18:41.043 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:18:41.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:41.046 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:18:41.147 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:18:41.174 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:41.192 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:18:41.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:41.195 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:18:41.296 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:18:41.322 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:41.341 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:18:41.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:41.343 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:18:41.444 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:18:41.477 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:41.577 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:18:41.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:41.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:18:41.599 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-05-15T13:18:41.681 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:18:41.714 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:41.730 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:41 smithi027 ceph-mon[39795]: pgmap v1904: 37 pgs: 1 active+recovering+remapped, 36 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 1085/6 objects misplaced (18083.333%); 0 B/s, 286 objects/s recovering 2024-05-15T13:18:41.730 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:41 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3611086116' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T13:18:41.730 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:41 smithi027 ceph-mon[39795]: osdmap e735: 12 total, 12 up, 10 in 2024-05-15T13:18:41.730 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:41 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:18:41.730 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:41 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:18:41.736 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:18:41.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:41.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:18:41.822 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:18:41.840 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:18:41.893 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:41.950 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:18:41.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:41.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:18:42.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:41 smithi120 ceph-mon[43961]: pgmap v1904: 37 pgs: 1 active+recovering+remapped, 36 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 1085/6 objects misplaced (18083.333%); 0 B/s, 286 objects/s recovering 2024-05-15T13:18:42.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:41 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3611086116' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T13:18:42.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:41 smithi120 ceph-mon[43961]: osdmap e735: 12 total, 12 up, 10 in 2024-05-15T13:18:42.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:41 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:18:42.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:41 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:18:42.053 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:18:42.085 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:42.107 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:18:42.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:42.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:18:42.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:41 smithi177 ceph-mon[44303]: pgmap v1904: 37 pgs: 1 active+recovering+remapped, 36 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 1085/6 objects misplaced (18083.333%); 0 B/s, 286 objects/s recovering 2024-05-15T13:18:42.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:41 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3611086116' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T13:18:42.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:41 smithi177 ceph-mon[44303]: osdmap e735: 12 total, 12 up, 10 in 2024-05-15T13:18:42.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:41 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:18:42.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:41 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:18:42.211 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:18:42.240 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:42.260 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:18:42.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:42.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:18:42.364 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:18:42.397 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:42.418 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:18:42.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:42.422 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:18:42.455 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:18:42.455 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"deaea04e-12b7-11ef-bc98-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2297,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":736,"num_osds":12,"num_up_osds":12,"osd_up_since":1715777150,"num_in_osds":10,"osd_in_since":1715778703,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":36},{"state_name":"active+recovering+remapped","count":1}],"num_pgs":37,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":9058459648,"bytes_avail":950891896832,"bytes_total":959950356480,"misplaced_objects":1085,"misplaced_total":6,"misplaced_ratio":180.83333333333337,"recovering_objects_per_sec":286,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2255,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.27:8443/"}},"servicemap":{"epoch":52,"modified":"2024-05-15T13:18:24.572800+0000","services":{"osd":{"daemons":{"summary":"","9":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-05-15T13:18:42.523 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:18:42.552 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:43.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:42 smithi027 ceph-mon[39795]: osdmap e736: 12 total, 12 up, 10 in 2024-05-15T13:18:43.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:42 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3002028140' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-05-15T13:18:43.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:42 smithi120 ceph-mon[43961]: osdmap e736: 12 total, 12 up, 10 in 2024-05-15T13:18:43.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:42 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3002028140' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-05-15T13:18:43.187 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-05-15T13:18:43.188 INFO:tasks.radosbench:Beginning radosbench... 2024-05-15T13:18:43.188 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_6 2024-05-15T13:18:43.188 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool create unique_pool_6 16 2024-05-15T13:18:43.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:42 smithi177 ceph-mon[44303]: osdmap e736: 12 total, 12 up, 10 in 2024-05-15T13:18:43.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:42 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3002028140' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-05-15T13:18:43.407 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:18:43.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:43.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:18:43.514 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:18:43.546 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:43.568 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:18:43.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:43.571 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:18:43.672 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:18:43.702 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:43.722 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:18:43.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:43.725 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:18:43.826 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:18:43.859 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:43.881 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:18:43.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:43.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:18:43.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:43 smithi177 ceph-mon[44303]: pgmap v1907: 21 pgs: 21 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:18:43.964 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-05-15T13:18:43.965 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:18:43.984 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:18:43.993 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:43 smithi027 ceph-mon[39795]: pgmap v1907: 21 pgs: 21 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:18:44.015 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:44.036 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:18:44.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:43 smithi120 ceph-mon[43961]: pgmap v1907: 21 pgs: 21 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:18:44.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:44.039 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:18:44.068 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:18:44.068 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:18:44.068 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:18:44.068 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:18:44.068 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:18:44.069 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:18:44.069 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:18:44.069 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:18:44.069 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:18:44.069 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:18:44.069 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:18:44.069 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:18:44.069 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:18:44.069 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:18:44.069 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:18:44.069 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:18:44.070 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:18:44.070 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:18:44.070 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:18:44.070 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:18:44.070 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:18:44.070 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:18:44.070 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:18:44.070 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:18:44.140 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:18:44.173 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:44.194 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:18:44.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:44.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:18:44.299 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:18:44.327 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:44.347 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:18:44.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:44.350 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:18:44.451 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:18:44.478 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:44.496 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:18:44.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:44.500 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:18:44.601 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:18:44.631 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:44.670 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:18:44.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:44.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:18:44.770 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:18:44.774 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:18:44.806 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:44.819 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:18:44.820 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:18:44.827 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:18:44.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:44.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:18:44.931 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:18:44.958 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:44.976 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:18:44.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:44.979 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:18:45.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:44 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4026894454' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:18:45.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:44 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4026894454' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:18:45.052 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:18:45.080 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:18:45.110 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:45.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:44 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4026894454' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:18:46.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:45 smithi027 ceph-mon[39795]: pgmap v1908: 21 pgs: 21 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:18:46.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:45 smithi120 ceph-mon[43961]: pgmap v1908: 21 pgs: 21 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:18:46.061 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:18:46.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:46.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:18:46.165 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:18:46.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:45 smithi177 ceph-mon[44303]: pgmap v1908: 21 pgs: 21 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:18:46.198 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:46.219 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:18:46.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:46.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:18:46.323 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:18:46.355 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:46.376 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:18:46.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:46.380 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:18:46.481 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:18:46.512 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:46.533 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:18:46.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:46.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:18:46.637 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:18:46.666 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:46.696 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:18:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:18:46.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:46.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:18:46.894 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:18:46.927 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:46.948 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:18:46.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:46.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:18:47.053 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:18:47.081 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:47.102 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:18:47.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:47.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:18:47.207 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:18:47.236 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:47.266 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:18:47.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:47.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:18:47.370 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:18:47.400 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:47.420 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:18:47.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:47.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:18:47.525 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:18:47.553 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:47.965 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:47 smithi027 ceph-mon[39795]: pgmap v1909: 21 pgs: 21 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:18:48.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:47 smithi120 ceph-mon[43961]: pgmap v1909: 21 pgs: 21 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:18:48.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:47 smithi177 ceph-mon[44303]: pgmap v1909: 21 pgs: 21 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:18:48.402 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:18:48.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:48.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:18:48.506 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:18:48.541 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:48.993 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:18:48.995 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:48 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:18:48.995 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:48 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:18:48.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:48.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:18:49.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:48 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:18:49.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:48 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:18:49.097 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:18:49.130 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:49.152 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:18:49.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:49.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:18:49.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:48 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:18:49.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:48 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:18:49.256 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:18:49.286 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:49.340 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:18:49.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:49.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:18:49.428 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:18:49.428 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:18:49.443 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:18:49.471 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:49.595 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:18:49.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:49.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:18:49.700 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:18:49.732 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:49.754 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:18:49.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:49.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:18:49.859 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:18:49.874 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:49 smithi027 ceph-mon[39795]: pgmap v1910: 21 pgs: 21 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:18:49.874 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:49 smithi027 ceph-mon[39795]: osdmap e737: 12 total, 12 up, 10 in 2024-05-15T13:18:49.874 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:49 smithi027 ceph-mon[39795]: from='client.16299 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:18:49.874 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:49 smithi027 ceph-mon[39795]: osdmap e738: 12 total, 12 up, 10 in 2024-05-15T13:18:49.888 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:49.909 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:18:49.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:49.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:18:50.013 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:18:50.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:49 smithi120 ceph-mon[43961]: pgmap v1910: 21 pgs: 21 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:18:50.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:49 smithi120 ceph-mon[43961]: osdmap e737: 12 total, 12 up, 10 in 2024-05-15T13:18:50.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:49 smithi120 ceph-mon[43961]: from='client.16299 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:18:50.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:49 smithi120 ceph-mon[43961]: osdmap e738: 12 total, 12 up, 10 in 2024-05-15T13:18:50.041 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:50.060 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:18:50.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:50.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:18:50.164 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:18:50.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:49 smithi177 ceph-mon[44303]: pgmap v1910: 21 pgs: 21 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:18:50.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:49 smithi177 ceph-mon[44303]: osdmap e737: 12 total, 12 up, 10 in 2024-05-15T13:18:50.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:49 smithi177 ceph-mon[44303]: from='client.16299 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:18:50.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:49 smithi177 ceph-mon[44303]: osdmap e738: 12 total, 12 up, 10 in 2024-05-15T13:18:50.197 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:50.217 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:18:50.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:50.220 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:18:50.224 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:18:50.224 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:18:50.321 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:18:50.349 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:50.369 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:18:50.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:50.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:18:50.473 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:18:50.507 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:50.757 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_6' created 2024-05-15T13:18:50.767 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:50 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3362131650' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-05-15T13:18:50.796 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":1911,"stamp":"2024-05-15T13:18:48.745381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":63,"acting":63,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":75,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":8771948,"kb_used_data":1761592,"kb_used_omap":15,"kb_used_meta":7010224,"kb_avail":928679572,"statfs":{"total":959950356480,"available":950967881728,"internally_reserved":0,"allocated":1803870208,"data_stored":1797488762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15956,"internal_metadata":7178469804},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":50,"apply_latency_ms":50,"commit_latency_ns":50000000,"apply_latency_ns":50000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.022255"},"pg_stats":[{"pgid":"1.14","version":"94'139","reported_seq":804,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-05-15T13:18:41.882546+0000","last_change":"2024-05-15T13:18:17.100933+0000","last_active":"2024-05-15T13:18:41.882546+0000","last_peered":"2024-05-15T13:18:41.882546+0000","last_clean":"2024-05-15T13:18:41.882546+0000","last_became_active":"2024-05-15T13:13:16.473818+0000","last_became_peered":"2024-05-15T13:13:16.473818+0000","last_unstale":"2024-05-15T13:18:41.882546+0000","last_undegraded":"2024-05-15T13:18:41.882546+0000","last_fullsized":"2024-05-15T13:18:41.882546+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:18:17.100768+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:18:17.100768+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:25.805853+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1],"acting":[2,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":1371,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-05-15T13:18:41.728773+0000","last_change":"2024-05-15T13:13:19.224641+0000","last_active":"2024-05-15T13:18:41.728773+0000","last_peered":"2024-05-15T13:18:41.728773+0000","last_clean":"2024-05-15T13:18:41.728773+0000","last_became_active":"2024-05-15T13:13:11.361366+0000","last_became_peered":"2024-05-15T13:13:11.361366+0000","last_unstale":"2024-05-15T13:18:41.728773+0000","last_undegraded":"2024-05-15T13:18:41.728773+0000","last_fullsized":"2024-05-15T13:18:41.728773+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:19.224509+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:19.224509+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"94'139","reported_seq":1379,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-05-15T13:18:41.882424+0000","last_change":"2024-05-15T13:17:45.047784+0000","last_active":"2024-05-15T13:18:41.882424+0000","last_peered":"2024-05-15T13:18:41.882424+0000","last_clean":"2024-05-15T13:18:41.882424+0000","last_became_active":"2024-05-15T13:13:11.371643+0000","last_became_peered":"2024-05-15T13:13:11.371643+0000","last_unstale":"2024-05-15T13:18:41.882424+0000","last_undegraded":"2024-05-15T13:18:41.882424+0000","last_fullsized":"2024-05-15T13:18:41.882424+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:45.047653+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:17:45.047653+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:02.727762+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"94'139","reported_seq":1133,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-05-15T13:18:41.882473+0000","last_change":"2024-05-15T13:18:24.970432+0000","last_active":"2024-05-15T13:18:41.882473+0000","last_peered":"2024-05-15T13:18:41.882473+0000","last_clean":"2024-05-15T13:18:41.882473+0000","last_became_active":"2024-05-15T13:14:54.577458+0000","last_became_peered":"2024-05-15T13:14:54.577458+0000","last_unstale":"2024-05-15T13:18:41.882473+0000","last_undegraded":"2024-05-15T13:18:41.882473+0000","last_fullsized":"2024-05-15T13:18:41.882473+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:18:24.970285+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:18:24.970285+0000","last_clean_scrub_stamp":"2024-05-15T13:18:24.970285+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:53.226976+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"94'139","reported_seq":1372,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-05-15T13:18:41.882424+0000","last_change":"2024-05-15T13:17:38.671246+0000","last_active":"2024-05-15T13:18:41.882424+0000","last_peered":"2024-05-15T13:18:41.882424+0000","last_clean":"2024-05-15T13:18:41.882424+0000","last_became_active":"2024-05-15T13:13:11.373251+0000","last_became_peered":"2024-05-15T13:13:11.373251+0000","last_unstale":"2024-05-15T13:18:41.882424+0000","last_undegraded":"2024-05-15T13:18:41.882424+0000","last_fullsized":"2024-05-15T13:18:41.882424+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:38.671141+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:17:38.671141+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:51.578496+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"94'139","reported_seq":1139,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-05-15T13:18:41.882465+0000","last_change":"2024-05-15T13:14:11.522798+0000","last_active":"2024-05-15T13:18:41.882465+0000","last_peered":"2024-05-15T13:18:41.882465+0000","last_clean":"2024-05-15T13:18:41.882465+0000","last_became_active":"2024-05-15T13:14:11.522490+0000","last_became_peered":"2024-05-15T13:14:11.522490+0000","last_unstale":"2024-05-15T13:18:41.882465+0000","last_undegraded":"2024-05-15T13:18:41.882465+0000","last_fullsized":"2024-05-15T13:18:41.882465+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.847198+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.847198+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"94'139","reported_seq":1150,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-05-15T13:18:41.882663+0000","last_change":"2024-05-15T13:17:41.721650+0000","last_active":"2024-05-15T13:18:41.882663+0000","last_peered":"2024-05-15T13:18:41.882663+0000","last_clean":"2024-05-15T13:18:41.882663+0000","last_became_active":"2024-05-15T13:15:43.614970+0000","last_became_peered":"2024-05-15T13:15:43.614970+0000","last_unstale":"2024-05-15T13:18:41.882663+0000","last_undegraded":"2024-05-15T13:18:41.882663+0000","last_fullsized":"2024-05-15T13:18:41.882663+0000","mapping_epoch":678,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:41.721516+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:17:41.721516+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:59.128918+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,0],"acting":[9,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.2","version":"94'139","reported_seq":1382,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-05-15T13:18:41.882275+0000","last_change":"2024-05-15T13:18:35.981730+0000","last_active":"2024-05-15T13:18:41.882275+0000","last_peered":"2024-05-15T13:18:41.882275+0000","last_clean":"2024-05-15T13:18:41.882275+0000","last_became_active":"2024-05-15T13:13:11.373548+0000","last_became_peered":"2024-05-15T13:13:11.373548+0000","last_unstale":"2024-05-15T13:18:41.882275+0000","last_undegraded":"2024-05-15T13:18:41.882275+0000","last_fullsized":"2024-05-15T13:18:41.882275+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:18.973644+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:18.973644+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:45.218223+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"94'139","reported_seq":1397,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-05-15T13:18:41.728803+0000","last_change":"2024-05-15T13:14:54.570374+0000","last_active":"2024-05-15T13:18:41.728803+0000","last_peered":"2024-05-15T13:18:41.728803+0000","last_clean":"2024-05-15T13:18:41.728803+0000","last_became_active":"2024-05-15T13:14:54.570004+0000","last_became_peered":"2024-05-15T13:14:54.570004+0000","last_unstale":"2024-05-15T13:18:41.728803+0000","last_undegraded":"2024-05-15T13:18:41.728803+0000","last_fullsized":"2024-05-15T13:18:41.728803+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:18.468889+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:18.468889+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":220,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"94'139","reported_seq":1374,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-05-15T13:18:41.729515+0000","last_change":"2024-05-15T13:17:55.115951+0000","last_active":"2024-05-15T13:18:41.729515+0000","last_peered":"2024-05-15T13:18:41.729515+0000","last_clean":"2024-05-15T13:18:41.729515+0000","last_became_active":"2024-05-15T13:06:46.426908+0000","last_became_peered":"2024-05-15T13:06:46.426908+0000","last_unstale":"2024-05-15T13:18:41.729515+0000","last_undegraded":"2024-05-15T13:18:41.729515+0000","last_fullsized":"2024-05-15T13:18:41.729515+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:55.115854+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:17:55.115854+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:04.858229+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"94'139","reported_seq":1366,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-05-15T13:18:41.882514+0000","last_change":"2024-05-15T13:17:44.642745+0000","last_active":"2024-05-15T13:18:41.882514+0000","last_peered":"2024-05-15T13:18:41.882514+0000","last_clean":"2024-05-15T13:18:41.882514+0000","last_became_active":"2024-05-15T13:06:46.424667+0000","last_became_peered":"2024-05-15T13:06:46.424667+0000","last_unstale":"2024-05-15T13:18:41.882514+0000","last_undegraded":"2024-05-15T13:18:41.882514+0000","last_fullsized":"2024-05-15T13:18:41.882514+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:44.642652+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:17:44.642652+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:46.761399+0000","scrub_duration":205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"94'139","reported_seq":1145,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-05-15T13:18:41.882700+0000","last_change":"2024-05-15T13:17:45.527703+0000","last_active":"2024-05-15T13:18:41.882700+0000","last_peered":"2024-05-15T13:18:41.882700+0000","last_clean":"2024-05-15T13:18:41.882700+0000","last_became_active":"2024-05-15T13:15:43.611913+0000","last_became_peered":"2024-05-15T13:15:43.611913+0000","last_unstale":"2024-05-15T13:18:41.882700+0000","last_undegraded":"2024-05-15T13:18:41.882700+0000","last_fullsized":"2024-05-15T13:18:41.882700+0000","mapping_epoch":678,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:45.527612+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:17:45.527612+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:58.076902+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.e","version":"94'139","reported_seq":1045,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-05-15T13:18:41.882211+0000","last_change":"2024-05-15T13:13:18.190667+0000","last_active":"2024-05-15T13:18:41.882211+0000","last_peered":"2024-05-15T13:18:41.882211+0000","last_clean":"2024-05-15T13:18:41.882211+0000","last_became_active":"2024-05-15T13:13:16.477771+0000","last_became_peered":"2024-05-15T13:13:16.477771+0000","last_unstale":"2024-05-15T13:18:41.882211+0000","last_undegraded":"2024-05-15T13:18:41.882211+0000","last_fullsized":"2024-05-15T13:18:41.882211+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:18.190529+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:18.190529+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:46.101910+0000","scrub_duration":228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,1],"acting":[3,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"94'139","reported_seq":1138,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-05-15T13:18:41.882490+0000","last_change":"2024-05-15T13:13:16.831084+0000","last_active":"2024-05-15T13:18:41.882490+0000","last_peered":"2024-05-15T13:18:41.882490+0000","last_clean":"2024-05-15T13:18:41.882490+0000","last_became_active":"2024-05-15T13:13:16.472318+0000","last_became_peered":"2024-05-15T13:13:16.472318+0000","last_unstale":"2024-05-15T13:18:41.882490+0000","last_undegraded":"2024-05-15T13:18:41.882490+0000","last_fullsized":"2024-05-15T13:18:41.882490+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:16.830998+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:16.830998+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"94'139","reported_seq":797,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-05-15T13:18:41.882064+0000","last_change":"2024-05-15T13:13:17.140519+0000","last_active":"2024-05-15T13:18:41.882064+0000","last_peered":"2024-05-15T13:18:41.882064+0000","last_clean":"2024-05-15T13:18:41.882064+0000","last_became_active":"2024-05-15T13:13:16.476972+0000","last_became_peered":"2024-05-15T13:13:16.476972+0000","last_unstale":"2024-05-15T13:18:41.882064+0000","last_undegraded":"2024-05-15T13:18:41.882064+0000","last_fullsized":"2024-05-15T13:18:41.882064+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.140398+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.140398+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:46.101930+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"94'139","reported_seq":785,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-05-15T13:18:41.728930+0000","last_change":"2024-05-15T13:13:17.391026+0000","last_active":"2024-05-15T13:18:41.728930+0000","last_peered":"2024-05-15T13:18:41.728930+0000","last_clean":"2024-05-15T13:18:41.728930+0000","last_became_active":"2024-05-15T13:13:16.478031+0000","last_became_peered":"2024-05-15T13:13:16.478031+0000","last_unstale":"2024-05-15T13:18:41.728930+0000","last_undegraded":"2024-05-15T13:18:41.728930+0000","last_fullsized":"2024-05-15T13:18:41.728930+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.390903+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.390903+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,11],"acting":[1,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"94'139","reported_seq":1398,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-05-15T13:18:41.882472+0000","last_change":"2024-05-15T13:17:44.264911+0000","last_active":"2024-05-15T13:18:41.882472+0000","last_peered":"2024-05-15T13:18:41.882472+0000","last_clean":"2024-05-15T13:18:41.882472+0000","last_became_active":"2024-05-15T13:13:16.484252+0000","last_became_peered":"2024-05-15T13:13:16.484252+0000","last_unstale":"2024-05-15T13:18:41.882472+0000","last_undegraded":"2024-05-15T13:18:41.882472+0000","last_fullsized":"2024-05-15T13:18:41.882472+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:44.264755+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:17:44.264755+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:50.708865+0000","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"94'139","reported_seq":1380,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-05-15T13:18:41.729511+0000","last_change":"2024-05-15T13:17:40.143307+0000","last_active":"2024-05-15T13:18:41.729511+0000","last_peered":"2024-05-15T13:18:41.729511+0000","last_clean":"2024-05-15T13:18:41.729511+0000","last_became_active":"2024-05-15T13:13:16.472800+0000","last_became_peered":"2024-05-15T13:13:16.472800+0000","last_unstale":"2024-05-15T13:18:41.729511+0000","last_undegraded":"2024-05-15T13:18:41.729511+0000","last_fullsized":"2024-05-15T13:18:41.729511+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:40.143179+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:17:40.143179+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:00.376173+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,8],"acting":[5,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"94'139","reported_seq":1154,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-05-15T13:18:41.882069+0000","last_change":"2024-05-15T13:18:30.438040+0000","last_active":"2024-05-15T13:18:41.882069+0000","last_peered":"2024-05-15T13:18:41.882069+0000","last_clean":"2024-05-15T13:18:41.882069+0000","last_became_active":"2024-05-15T13:13:16.476744+0000","last_became_peered":"2024-05-15T13:13:16.476744+0000","last_unstale":"2024-05-15T13:18:41.882069+0000","last_undegraded":"2024-05-15T13:18:41.882069+0000","last_fullsized":"2024-05-15T13:18:41.882069+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:18:30.437917+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:18:30.437917+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:31.780510+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"94'139","reported_seq":1404,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-05-15T13:18:41.729547+0000","last_change":"2024-05-15T13:18:06.341818+0000","last_active":"2024-05-15T13:18:41.729547+0000","last_peered":"2024-05-15T13:18:41.729547+0000","last_clean":"2024-05-15T13:18:41.729547+0000","last_became_active":"2024-05-15T13:13:49.926687+0000","last_became_peered":"2024-05-15T13:13:49.926687+0000","last_unstale":"2024-05-15T13:18:41.729547+0000","last_undegraded":"2024-05-15T13:18:41.729547+0000","last_fullsized":"2024-05-15T13:18:41.729547+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:18:06.340555+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:18:06.340555+0000","last_clean_scrub_stamp":"2024-05-15T13:18:06.340555+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:14.400006+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"94'139","reported_seq":1396,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-05-15T13:18:41.728793+0000","last_change":"2024-05-15T13:13:20.455813+0000","last_active":"2024-05-15T13:18:41.728793+0000","last_peered":"2024-05-15T13:18:41.728793+0000","last_clean":"2024-05-15T13:18:41.728793+0000","last_became_active":"2024-05-15T13:13:16.478488+0000","last_became_peered":"2024-05-15T13:13:16.478488+0000","last_unstale":"2024-05-15T13:18:41.728793+0000","last_undegraded":"2024-05-15T13:18:41.728793+0000","last_fullsized":"2024-05-15T13:18:41.728793+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:20.455652+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:20.455652+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057041,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":970168,"kb_used_data":383664,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":92774984,"statfs":{"total":95995035648,"available":95001583616,"internally_reserved":0,"allocated":392871936,"data_stored":392234857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":600570310},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351381,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1052408,"kb_used_data":371504,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":92692744,"statfs":{"total":95995035648,"available":94917369856,"internally_reserved":0,"allocated":380420096,"data_stored":379783017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":697235904},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514906,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":817276,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":92927876,"statfs":{"total":95995035648,"available":95158145024,"internally_reserved":0,"allocated":1753088,"data_stored":1116009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":835123648},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874655,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":456320,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":93288832,"statfs":{"total":95995035648,"available":95527763968,"internally_reserved":0,"allocated":1753088,"data_stored":1116009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":7,"up_from":0,"seq":240518168995,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430634,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":741248,"kb_used_data":2032,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":93003904,"statfs":{"total":95995035648,"available":95235997696,"internally_reserved":0,"allocated":2080768,"data_stored":1443689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":756939206},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575299,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":867260,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":92877892,"statfs":{"total":95995035648,"available":95106961408,"internally_reserved":0,"allocated":1753088,"data_stored":1116009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":886307269},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738828,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":759296,"kb_used_data":2032,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92985856,"statfs":{"total":95995035648,"available":95217516544,"internally_reserved":0,"allocated":2080768,"data_stored":1443689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":775420357},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411776,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":829824,"kb_used_data":279416,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":92915328,"statfs":{"total":95995035648,"available":95145295872,"internally_reserved":0,"allocated":286121984,"data_stored":285481353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150137,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":939972,"kb_used_data":2232,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92805180,"statfs":{"total":95995035648,"available":95032504320,"internally_reserved":0,"allocated":2285568,"data_stored":1644937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":960231879},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986613,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692272,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1338176,"kb_used_data":715576,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":92406976,"statfs":{"total":95995035648,"available":94624743424,"internally_reserved":0,"allocated":732749824,"data_stored":732109193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:18:50.956 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:18:50.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:50.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:18:51.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:50 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3362131650' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-05-15T13:18:51.061 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:18:51.094 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:51.115 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:18:51.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:51.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:18:51.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:50 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3362131650' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-05-15T13:18:51.218 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:18:51.250 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:51.271 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:18:51.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:51.274 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:18:51.375 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:18:51.407 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:51.429 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:18:51.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:51.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:18:51.533 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:18:51.563 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:51.763 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:51 smithi027 ceph-mon[39795]: pgmap v1913: 21 pgs: 21 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:18:51.763 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:51 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3362131650' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-05-15T13:18:51.763 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:51 smithi027 ceph-mon[39795]: osdmap e739: 12 total, 12 up, 10 in 2024-05-15T13:18:51.799 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:18:51.812 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:18:51.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:51.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:18:51.840 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-05-15T13:18:51.917 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:18:51.951 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:52.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:51 smithi120 ceph-mon[43961]: pgmap v1913: 21 pgs: 21 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:18:52.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:51 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3362131650' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-05-15T13:18:52.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:51 smithi120 ceph-mon[43961]: osdmap e739: 12 total, 12 up, 10 in 2024-05-15T13:18:52.050 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:18:52.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:52.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:18:52.156 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:18:52.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:51 smithi177 ceph-mon[44303]: pgmap v1913: 21 pgs: 21 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:18:52.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:51 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3362131650' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-05-15T13:18:52.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:51 smithi177 ceph-mon[44303]: osdmap e739: 12 total, 12 up, 10 in 2024-05-15T13:18:52.189 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:52.210 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:18:52.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:52.213 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:18:52.314 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:18:52.344 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:52.364 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:18:52.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:52.367 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:18:52.469 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:18:52.497 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:52.768 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:18:53.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:52 smithi027 ceph-mon[39795]: osdmap e740: 12 total, 12 up, 10 in 2024-05-15T13:18:53.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:52 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3456651885' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:18:53.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:52 smithi120 ceph-mon[43961]: osdmap e740: 12 total, 12 up, 10 in 2024-05-15T13:18:53.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:52 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3456651885' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:18:53.147 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:18:53.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:53.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:18:53.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:52 smithi177 ceph-mon[44303]: osdmap e740: 12 total, 12 up, 10 in 2024-05-15T13:18:53.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:52 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3456651885' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:18:53.263 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:18:53.293 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:53.800 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:18:53.946 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:18:53.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:53.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:18:54.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:53 smithi027 ceph-mon[39795]: pgmap v1916: 37 pgs: 15 unknown, 22 active+clean; 577 KiB data, 8.3 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:18:54.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:53 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:18:54.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:53 smithi027 ceph-mon[39795]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T13:18:54.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:53 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3456651885' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:18:54.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:53 smithi027 ceph-mon[39795]: osdmap e741: 12 total, 12 up, 10 in 2024-05-15T13:18:54.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:53 smithi120 ceph-mon[43961]: pgmap v1916: 37 pgs: 15 unknown, 22 active+clean; 577 KiB data, 8.3 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:18:54.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:53 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:18:54.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:53 smithi120 ceph-mon[43961]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T13:18:54.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:53 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3456651885' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:18:54.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:53 smithi120 ceph-mon[43961]: osdmap e741: 12 total, 12 up, 10 in 2024-05-15T13:18:54.051 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:18:54.078 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:54.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:53 smithi177 ceph-mon[44303]: pgmap v1916: 37 pgs: 15 unknown, 22 active+clean; 577 KiB data, 8.3 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:18:54.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:53 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:18:54.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:53 smithi177 ceph-mon[44303]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T13:18:54.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:53 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3456651885' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:18:54.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:53 smithi177 ceph-mon[44303]: osdmap e741: 12 total, 12 up, 10 in 2024-05-15T13:18:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:18:54.492 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool application enable unique_pool_6 rados --yes-i-really-mean-it || true 2024-05-15T13:18:54.558 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:18:54.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:54.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:18:54.664 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:18:54.696 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:54.717 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:18:54.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:54.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:18:54.821 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:18:54.855 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:54.877 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:18:54.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:54.881 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:18:54.982 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:18:55.013 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:55.033 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:18:55.035 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:54 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:18:55.035 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:54 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:18:55.035 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:54 smithi177 ceph-mon[44303]: osdmap e742: 12 total, 12 up, 10 in 2024-05-15T13:18:55.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:55.037 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:18:55.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:54 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:18:55.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:54 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:18:55.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:54 smithi120 ceph-mon[43961]: osdmap e742: 12 total, 12 up, 10 in 2024-05-15T13:18:55.058 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:54 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:18:55.058 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:54 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:18:55.058 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:54 smithi027 ceph-mon[39795]: osdmap e742: 12 total, 12 up, 10 in 2024-05-15T13:18:55.137 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:18:55.167 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:55.245 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:18:55.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:55.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:18:55.349 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:18:55.381 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:55.402 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:18:55.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:55.405 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:18:55.506 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:18:55.535 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:55.544 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:18:55.573 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:18:55.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:55.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:18:55.669 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:18:55.669 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:18:55.669 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:18:55.669 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:18:55.669 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:18:55.669 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:18:55.669 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:18:55.670 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:18:55.670 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:18:55.670 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:18:55.670 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:18:55.670 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:18:55.671 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:18:55.671 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:18:55.671 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:18:55.671 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:18:55.671 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:18:55.671 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:18:55.671 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:18:55.671 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:18:55.671 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:18:55.671 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:18:55.671 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:18:55.671 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:18:55.677 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:18:55.704 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:55.878 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:18:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:18:55.908 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:55 smithi027 ceph-mon[39795]: pgmap v1919: 37 pgs: 3 unknown, 34 active+clean; 577 KiB data, 8.3 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:18:55.909 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:55 smithi027 ceph-mon[39795]: from='client.16317 172.21.15.27:0/3522302632' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:18:55.909 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:55 smithi027 ceph-mon[39795]: from='client.16317 172.21.15.27:0/3522302632' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:18:55.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:55.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:18:56.011 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:18:56.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:55 smithi120 ceph-mon[43961]: pgmap v1919: 37 pgs: 3 unknown, 34 active+clean; 577 KiB data, 8.3 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:18:56.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:55 smithi120 ceph-mon[43961]: from='client.16317 172.21.15.27:0/3522302632' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:18:56.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:55 smithi120 ceph-mon[43961]: from='client.16317 172.21.15.27:0/3522302632' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:18:56.040 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:56.137 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:18:56.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:56.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:18:56.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:55 smithi177 ceph-mon[44303]: pgmap v1919: 37 pgs: 3 unknown, 34 active+clean; 577 KiB data, 8.3 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:18:56.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:55 smithi177 ceph-mon[44303]: from='client.16317 172.21.15.27:0/3522302632' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:18:56.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:55 smithi177 ceph-mon[44303]: from='client.16317 172.21.15.27:0/3522302632' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:18:56.242 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:18:56.274 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:56.295 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:18:56.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:56.298 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:18:56.399 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:18:56.429 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:56.450 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:18:56.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:56.453 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:18:56.554 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:18:56.583 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:56.604 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:18:56.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:56.607 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:18:56.708 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:18:56.740 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:56.761 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:18:56.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:56.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:18:56.865 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:18:56.894 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:56.914 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:18:56.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:56.917 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:18:57.018 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:18:57.049 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:57.104 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:18:57.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:57.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:18:57.209 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:18:57.238 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:57.686 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:18:57.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:57.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:18:57.791 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:18:57.822 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:57.842 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:18:57.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:57.845 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:18:57.946 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:18:57.975 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:57.995 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:18:57.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:57.998 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:18:57.998 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:57 smithi027 ceph-mon[39795]: pgmap v1920: 37 pgs: 37 active+clean; 577 KiB data, 8.3 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:18:57.999 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:18:58.000 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:18:58.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:57 smithi120 ceph-mon[43961]: pgmap v1920: 37 pgs: 37 active+clean; 577 KiB data, 8.3 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:18:58.099 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:18:58.130 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:58.151 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:18:58.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:58.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:18:58.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:57 smithi177 ceph-mon[44303]: pgmap v1920: 37 pgs: 37 active+clean; 577 KiB data, 8.3 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:18:58.256 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:18:58.286 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:58.307 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:18:58.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:58.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:18:58.411 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:18:58.440 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:58.459 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:18:58.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:58.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:18:58.563 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:18:58.596 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:58.616 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:18:58.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:58.620 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:18:58.721 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:18:58.750 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:58.770 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:18:58.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:58.774 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:18:58.802 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:18:58.874 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:18:58.903 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:58.924 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:18:58.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:58.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:18:59.028 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:18:59.055 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:18:59.908 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:18:59.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:18:59.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:00.014 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:19:00.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:18:59 smithi120 ceph-mon[43961]: pgmap v1921: 37 pgs: 37 active+clean; 577 KiB data, 8.3 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:19:00.043 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:18:59 smithi177 ceph-mon[44303]: pgmap v1921: 37 pgs: 37 active+clean; 577 KiB data, 8.3 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:19:00.046 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:00.067 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:19:00.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:00.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:19:00.172 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:00.242 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:18:59 smithi027 ceph-mon[39795]: pgmap v1921: 37 pgs: 37 active+clean; 577 KiB data, 8.3 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:19:00.245 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:00.276 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:00.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:00.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:00.331 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:19:00.333 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:19:00.379 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:19:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:00.450 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:19:00.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:00.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:19:00.554 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:19:00.586 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:00.608 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:19:00.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:00.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:19:00.712 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:19:00.740 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:00.797 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:19:00.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:00 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2276159893' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:19:01.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:00 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2276159893' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:19:01.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:00 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2276159893' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:19:02.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:01 smithi027 ceph-mon[39795]: from='client.16323 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:19:02.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:01 smithi027 ceph-mon[39795]: pgmap v1922: 37 pgs: 37 active+clean; 577 KiB data, 8.3 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:19:02.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:01 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:19:02.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:01 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2276159893' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:19:02.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:01 smithi027 ceph-mon[39795]: osdmap e743: 12 total, 12 up, 10 in 2024-05-15T13:19:02.063 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":1921,"stamp":"2024-05-15T13:18:58.579540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":8678576,"kb_used_data":1668196,"kb_used_omap":15,"kb_used_meta":7010224,"kb_avail":928772944,"statfs":{"total":959950356480,"available":951063494656,"internally_reserved":0,"allocated":1708232704,"data_stored":1701594325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15956,"internal_metadata":7178469804},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.821403"},"pg_stats":[{"pgid":"1.14","version":"94'139","reported_seq":816,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:53.781732+0000","last_change":"2024-05-15T13:18:17.100933+0000","last_active":"2024-05-15T13:18:53.781732+0000","last_peered":"2024-05-15T13:18:53.781732+0000","last_clean":"2024-05-15T13:18:53.781732+0000","last_became_active":"2024-05-15T13:13:16.473818+0000","last_became_peered":"2024-05-15T13:13:16.473818+0000","last_unstale":"2024-05-15T13:18:53.781732+0000","last_undegraded":"2024-05-15T13:18:53.781732+0000","last_fullsized":"2024-05-15T13:18:53.781732+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:18:17.100768+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:18:17.100768+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:25.805853+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1],"acting":[2,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":13,"reported_epoch":741,"state":"active+clean","last_fresh":"2024-05-15T13:18:53.385229+0000","last_change":"2024-05-15T13:18:51.769954+0000","last_active":"2024-05-15T13:18:53.385229+0000","last_peered":"2024-05-15T13:18:53.385229+0000","last_clean":"2024-05-15T13:18:53.385229+0000","last_became_active":"2024-05-15T13:18:51.769627+0000","last_became_peered":"2024-05-15T13:18:51.769627+0000","last_unstale":"2024-05-15T13:18:53.385229+0000","last_undegraded":"2024-05-15T13:18:53.385229+0000","last_fullsized":"2024-05-15T13:18:53.385229+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:10.580711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,0],"acting":[5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":1383,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:53.780315+0000","last_change":"2024-05-15T13:13:19.224641+0000","last_active":"2024-05-15T13:18:53.780315+0000","last_peered":"2024-05-15T13:18:53.780315+0000","last_clean":"2024-05-15T13:18:53.780315+0000","last_became_active":"2024-05-15T13:13:11.361366+0000","last_became_peered":"2024-05-15T13:13:11.361366+0000","last_unstale":"2024-05-15T13:18:53.780315+0000","last_undegraded":"2024-05-15T13:18:53.780315+0000","last_fullsized":"2024-05-15T13:18:53.780315+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:19.224509+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:19.224509+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:56.716971+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":15,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:54.137252+0000","last_change":"2024-05-15T13:18:51.823482+0000","last_active":"2024-05-15T13:18:54.137252+0000","last_peered":"2024-05-15T13:18:54.137252+0000","last_clean":"2024-05-15T13:18:54.137252+0000","last_became_active":"2024-05-15T13:18:51.823045+0000","last_became_peered":"2024-05-15T13:18:51.823045+0000","last_unstale":"2024-05-15T13:18:54.137252+0000","last_undegraded":"2024-05-15T13:18:54.137252+0000","last_fullsized":"2024-05-15T13:18:54.137252+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:54.582844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"94'139","reported_seq":1391,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:54.137051+0000","last_change":"2024-05-15T13:17:45.047784+0000","last_active":"2024-05-15T13:18:54.137051+0000","last_peered":"2024-05-15T13:18:54.137051+0000","last_clean":"2024-05-15T13:18:54.137051+0000","last_became_active":"2024-05-15T13:13:11.371643+0000","last_became_peered":"2024-05-15T13:13:11.371643+0000","last_unstale":"2024-05-15T13:18:54.137051+0000","last_undegraded":"2024-05-15T13:18:54.137051+0000","last_fullsized":"2024-05-15T13:18:54.137051+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:45.047653+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:17:45.047653+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:02.727762+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":15,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:54.137579+0000","last_change":"2024-05-15T13:18:51.771623+0000","last_active":"2024-05-15T13:18:54.137579+0000","last_peered":"2024-05-15T13:18:54.137579+0000","last_clean":"2024-05-15T13:18:54.137579+0000","last_became_active":"2024-05-15T13:18:51.771276+0000","last_became_peered":"2024-05-15T13:18:51.771276+0000","last_unstale":"2024-05-15T13:18:54.137579+0000","last_undegraded":"2024-05-15T13:18:54.137579+0000","last_fullsized":"2024-05-15T13:18:54.137579+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:59.671271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"94'139","reported_seq":1145,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:54.136979+0000","last_change":"2024-05-15T13:18:24.970432+0000","last_active":"2024-05-15T13:18:54.136979+0000","last_peered":"2024-05-15T13:18:54.136979+0000","last_clean":"2024-05-15T13:18:54.136979+0000","last_became_active":"2024-05-15T13:14:54.577458+0000","last_became_peered":"2024-05-15T13:14:54.577458+0000","last_unstale":"2024-05-15T13:18:54.136979+0000","last_undegraded":"2024-05-15T13:18:54.136979+0000","last_fullsized":"2024-05-15T13:18:54.136979+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:18:24.970285+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:18:24.970285+0000","last_clean_scrub_stamp":"2024-05-15T13:18:24.970285+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:53.226976+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":15,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:54.137676+0000","last_change":"2024-05-15T13:18:51.825618+0000","last_active":"2024-05-15T13:18:54.137676+0000","last_peered":"2024-05-15T13:18:54.137676+0000","last_clean":"2024-05-15T13:18:54.137676+0000","last_became_active":"2024-05-15T13:18:51.825302+0000","last_became_peered":"2024-05-15T13:18:51.825302+0000","last_unstale":"2024-05-15T13:18:54.137676+0000","last_undegraded":"2024-05-15T13:18:54.137676+0000","last_fullsized":"2024-05-15T13:18:54.137676+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:53.590875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"94'139","reported_seq":1384,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:54.137259+0000","last_change":"2024-05-15T13:17:38.671246+0000","last_active":"2024-05-15T13:18:54.137259+0000","last_peered":"2024-05-15T13:18:54.137259+0000","last_clean":"2024-05-15T13:18:54.137259+0000","last_became_active":"2024-05-15T13:13:11.373251+0000","last_became_peered":"2024-05-15T13:13:11.373251+0000","last_unstale":"2024-05-15T13:18:54.137259+0000","last_undegraded":"2024-05-15T13:18:54.137259+0000","last_fullsized":"2024-05-15T13:18:54.137259+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:38.671141+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:17:38.671141+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:58.453362+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":15,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:53.781147+0000","last_change":"2024-05-15T13:18:51.823811+0000","last_active":"2024-05-15T13:18:53.781147+0000","last_peered":"2024-05-15T13:18:53.781147+0000","last_clean":"2024-05-15T13:18:53.781147+0000","last_became_active":"2024-05-15T13:18:51.823490+0000","last_became_peered":"2024-05-15T13:18:51.823490+0000","last_unstale":"2024-05-15T13:18:53.781147+0000","last_undegraded":"2024-05-15T13:18:53.781147+0000","last_fullsized":"2024-05-15T13:18:53.781147+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:05.401343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,11],"acting":[0,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"94'139","reported_seq":1151,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:54.136957+0000","last_change":"2024-05-15T13:14:11.522798+0000","last_active":"2024-05-15T13:18:54.136957+0000","last_peered":"2024-05-15T13:18:54.136957+0000","last_clean":"2024-05-15T13:18:54.136957+0000","last_became_active":"2024-05-15T13:14:11.522490+0000","last_became_peered":"2024-05-15T13:14:11.522490+0000","last_unstale":"2024-05-15T13:18:54.136957+0000","last_undegraded":"2024-05-15T13:18:54.136957+0000","last_fullsized":"2024-05-15T13:18:54.136957+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.847198+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.847198+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":13,"reported_epoch":741,"state":"active+clean","last_fresh":"2024-05-15T13:18:53.385167+0000","last_change":"2024-05-15T13:18:51.770676+0000","last_active":"2024-05-15T13:18:53.385167+0000","last_peered":"2024-05-15T13:18:53.385167+0000","last_clean":"2024-05-15T13:18:53.385167+0000","last_became_active":"2024-05-15T13:18:51.770286+0000","last_became_peered":"2024-05-15T13:18:51.770286+0000","last_unstale":"2024-05-15T13:18:53.385167+0000","last_undegraded":"2024-05-15T13:18:53.385167+0000","last_fullsized":"2024-05-15T13:18:53.385167+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:53.234785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"94'139","reported_seq":1162,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:54.136985+0000","last_change":"2024-05-15T13:17:41.721650+0000","last_active":"2024-05-15T13:18:54.136985+0000","last_peered":"2024-05-15T13:18:54.136985+0000","last_clean":"2024-05-15T13:18:54.136985+0000","last_became_active":"2024-05-15T13:15:43.614970+0000","last_became_peered":"2024-05-15T13:15:43.614970+0000","last_unstale":"2024-05-15T13:18:54.136985+0000","last_undegraded":"2024-05-15T13:18:54.136985+0000","last_fullsized":"2024-05-15T13:18:54.136985+0000","mapping_epoch":678,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:41.721516+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:17:41.721516+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:59.128918+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,0],"acting":[9,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":15,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:54.137208+0000","last_change":"2024-05-15T13:18:51.768048+0000","last_active":"2024-05-15T13:18:54.137208+0000","last_peered":"2024-05-15T13:18:54.137208+0000","last_clean":"2024-05-15T13:18:54.137208+0000","last_became_active":"2024-05-15T13:18:51.767604+0000","last_became_peered":"2024-05-15T13:18:51.767604+0000","last_unstale":"2024-05-15T13:18:54.137208+0000","last_undegraded":"2024-05-15T13:18:54.137208+0000","last_fullsized":"2024-05-15T13:18:54.137208+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:58.793815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"94'139","reported_seq":1394,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:53.781076+0000","last_change":"2024-05-15T13:18:35.981730+0000","last_active":"2024-05-15T13:18:53.781076+0000","last_peered":"2024-05-15T13:18:53.781076+0000","last_clean":"2024-05-15T13:18:53.781076+0000","last_became_active":"2024-05-15T13:13:11.373548+0000","last_became_peered":"2024-05-15T13:13:11.373548+0000","last_unstale":"2024-05-15T13:18:53.781076+0000","last_undegraded":"2024-05-15T13:18:53.781076+0000","last_fullsized":"2024-05-15T13:18:53.781076+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:18.973644+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:18.973644+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:58.239223+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":15,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:54.137422+0000","last_change":"2024-05-15T13:18:51.823543+0000","last_active":"2024-05-15T13:18:54.137422+0000","last_peered":"2024-05-15T13:18:54.137422+0000","last_clean":"2024-05-15T13:18:54.137422+0000","last_became_active":"2024-05-15T13:18:51.823137+0000","last_became_peered":"2024-05-15T13:18:51.823137+0000","last_unstale":"2024-05-15T13:18:54.137422+0000","last_undegraded":"2024-05-15T13:18:54.137422+0000","last_fullsized":"2024-05-15T13:18:54.137422+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:12.075938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"94'139","reported_seq":1409,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:53.780746+0000","last_change":"2024-05-15T13:14:54.570374+0000","last_active":"2024-05-15T13:18:53.780746+0000","last_peered":"2024-05-15T13:18:53.780746+0000","last_clean":"2024-05-15T13:18:53.780746+0000","last_became_active":"2024-05-15T13:14:54.570004+0000","last_became_peered":"2024-05-15T13:14:54.570004+0000","last_unstale":"2024-05-15T13:18:53.780746+0000","last_undegraded":"2024-05-15T13:18:53.780746+0000","last_fullsized":"2024-05-15T13:18:53.780746+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:18.468889+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:18.468889+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:56.717087+0000","scrub_duration":220,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":15,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:53.780315+0000","last_change":"2024-05-15T13:18:51.824173+0000","last_active":"2024-05-15T13:18:53.780315+0000","last_peered":"2024-05-15T13:18:53.780315+0000","last_clean":"2024-05-15T13:18:53.780315+0000","last_became_active":"2024-05-15T13:18:51.823777+0000","last_became_peered":"2024-05-15T13:18:51.823777+0000","last_unstale":"2024-05-15T13:18:53.780315+0000","last_undegraded":"2024-05-15T13:18:53.780315+0000","last_fullsized":"2024-05-15T13:18:53.780315+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:15.555994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"94'139","reported_seq":1384,"reported_epoch":741,"state":"active+clean","last_fresh":"2024-05-15T13:18:53.385338+0000","last_change":"2024-05-15T13:17:55.115951+0000","last_active":"2024-05-15T13:18:53.385338+0000","last_peered":"2024-05-15T13:18:53.385338+0000","last_clean":"2024-05-15T13:18:53.385338+0000","last_became_active":"2024-05-15T13:06:46.426908+0000","last_became_peered":"2024-05-15T13:06:46.426908+0000","last_unstale":"2024-05-15T13:18:53.385338+0000","last_undegraded":"2024-05-15T13:18:53.385338+0000","last_fullsized":"2024-05-15T13:18:53.385338+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:55.115854+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:17:55.115854+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:04.858229+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":15,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:53.992504+0000","last_change":"2024-05-15T13:18:51.770583+0000","last_active":"2024-05-15T13:18:53.992504+0000","last_peered":"2024-05-15T13:18:53.992504+0000","last_clean":"2024-05-15T13:18:53.992504+0000","last_became_active":"2024-05-15T13:18:51.770274+0000","last_became_peered":"2024-05-15T13:18:51.770274+0000","last_unstale":"2024-05-15T13:18:53.992504+0000","last_undegraded":"2024-05-15T13:18:53.992504+0000","last_fullsized":"2024-05-15T13:18:53.992504+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:55.634576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"94'139","reported_seq":1378,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:54.137126+0000","last_change":"2024-05-15T13:17:44.642745+0000","last_active":"2024-05-15T13:18:54.137126+0000","last_peered":"2024-05-15T13:18:54.137126+0000","last_clean":"2024-05-15T13:18:54.137126+0000","last_became_active":"2024-05-15T13:06:46.424667+0000","last_became_peered":"2024-05-15T13:06:46.424667+0000","last_unstale":"2024-05-15T13:18:54.137126+0000","last_undegraded":"2024-05-15T13:18:54.137126+0000","last_fullsized":"2024-05-15T13:18:54.137126+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:44.642652+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:17:44.642652+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":15,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:54.137799+0000","last_change":"2024-05-15T13:18:51.769282+0000","last_active":"2024-05-15T13:18:54.137799+0000","last_peered":"2024-05-15T13:18:54.137799+0000","last_clean":"2024-05-15T13:18:54.137799+0000","last_became_active":"2024-05-15T13:18:51.768837+0000","last_became_peered":"2024-05-15T13:18:51.768837+0000","last_unstale":"2024-05-15T13:18:54.137799+0000","last_undegraded":"2024-05-15T13:18:54.137799+0000","last_fullsized":"2024-05-15T13:18:54.137799+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:00.065411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"94'139","reported_seq":1157,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:54.137451+0000","last_change":"2024-05-15T13:17:45.527703+0000","last_active":"2024-05-15T13:18:54.137451+0000","last_peered":"2024-05-15T13:18:54.137451+0000","last_clean":"2024-05-15T13:18:54.137451+0000","last_became_active":"2024-05-15T13:15:43.611913+0000","last_became_peered":"2024-05-15T13:15:43.611913+0000","last_unstale":"2024-05-15T13:18:54.137451+0000","last_undegraded":"2024-05-15T13:18:54.137451+0000","last_fullsized":"2024-05-15T13:18:54.137451+0000","mapping_epoch":678,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:45.527612+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:17:45.527612+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:58.076902+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":15,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:53.780498+0000","last_change":"2024-05-15T13:18:51.767665+0000","last_active":"2024-05-15T13:18:53.780498+0000","last_peered":"2024-05-15T13:18:53.780498+0000","last_clean":"2024-05-15T13:18:53.780498+0000","last_became_active":"2024-05-15T13:18:51.767342+0000","last_became_peered":"2024-05-15T13:18:51.767342+0000","last_unstale":"2024-05-15T13:18:53.780498+0000","last_undegraded":"2024-05-15T13:18:53.780498+0000","last_fullsized":"2024-05-15T13:18:53.780498+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:20.375039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"94'139","reported_seq":1057,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:53.992560+0000","last_change":"2024-05-15T13:13:18.190667+0000","last_active":"2024-05-15T13:18:53.992560+0000","last_peered":"2024-05-15T13:18:53.992560+0000","last_clean":"2024-05-15T13:18:53.992560+0000","last_became_active":"2024-05-15T13:13:16.477771+0000","last_became_peered":"2024-05-15T13:13:16.477771+0000","last_unstale":"2024-05-15T13:18:53.992560+0000","last_undegraded":"2024-05-15T13:18:53.992560+0000","last_fullsized":"2024-05-15T13:18:53.992560+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:18.190529+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:18.190529+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:55.851837+0000","scrub_duration":228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,1],"acting":[3,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":15,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:54.137492+0000","last_change":"2024-05-15T13:18:51.823957+0000","last_active":"2024-05-15T13:18:54.137492+0000","last_peered":"2024-05-15T13:18:54.137492+0000","last_clean":"2024-05-15T13:18:54.137492+0000","last_became_active":"2024-05-15T13:18:51.823632+0000","last_became_peered":"2024-05-15T13:18:51.823632+0000","last_unstale":"2024-05-15T13:18:54.137492+0000","last_undegraded":"2024-05-15T13:18:54.137492+0000","last_fullsized":"2024-05-15T13:18:54.137492+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:14.911295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,2],"acting":[11,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"94'139","reported_seq":1150,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:54.137064+0000","last_change":"2024-05-15T13:13:16.831084+0000","last_active":"2024-05-15T13:18:54.137064+0000","last_peered":"2024-05-15T13:18:54.137064+0000","last_clean":"2024-05-15T13:18:54.137064+0000","last_became_active":"2024-05-15T13:13:16.472318+0000","last_became_peered":"2024-05-15T13:13:16.472318+0000","last_unstale":"2024-05-15T13:18:54.137064+0000","last_undegraded":"2024-05-15T13:18:54.137064+0000","last_fullsized":"2024-05-15T13:18:54.137064+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:16.830998+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:16.830998+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":15,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:53.780450+0000","last_change":"2024-05-15T13:18:51.767865+0000","last_active":"2024-05-15T13:18:53.780450+0000","last_peered":"2024-05-15T13:18:53.780450+0000","last_clean":"2024-05-15T13:18:53.780450+0000","last_became_active":"2024-05-15T13:18:51.767454+0000","last_became_peered":"2024-05-15T13:18:51.767454+0000","last_unstale":"2024-05-15T13:18:53.780450+0000","last_undegraded":"2024-05-15T13:18:53.780450+0000","last_fullsized":"2024-05-15T13:18:53.780450+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:51.600857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,11],"acting":[1,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"94'139","reported_seq":809,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:53.992553+0000","last_change":"2024-05-15T13:13:17.140519+0000","last_active":"2024-05-15T13:18:53.992553+0000","last_peered":"2024-05-15T13:18:53.992553+0000","last_clean":"2024-05-15T13:18:53.992553+0000","last_became_active":"2024-05-15T13:13:16.476972+0000","last_became_peered":"2024-05-15T13:13:16.476972+0000","last_unstale":"2024-05-15T13:18:53.992553+0000","last_undegraded":"2024-05-15T13:18:53.992553+0000","last_fullsized":"2024-05-15T13:18:53.992553+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.140398+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.140398+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:55.851857+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":13,"reported_epoch":741,"state":"active+clean","last_fresh":"2024-05-15T13:18:53.385402+0000","last_change":"2024-05-15T13:18:51.769457+0000","last_active":"2024-05-15T13:18:53.385402+0000","last_peered":"2024-05-15T13:18:53.385402+0000","last_clean":"2024-05-15T13:18:53.385402+0000","last_became_active":"2024-05-15T13:18:51.769098+0000","last_became_peered":"2024-05-15T13:18:51.769098+0000","last_unstale":"2024-05-15T13:18:53.385402+0000","last_undegraded":"2024-05-15T13:18:53.385402+0000","last_fullsized":"2024-05-15T13:18:53.385402+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:03.335364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"94'139","reported_seq":797,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:53.780583+0000","last_change":"2024-05-15T13:13:17.391026+0000","last_active":"2024-05-15T13:18:53.780583+0000","last_peered":"2024-05-15T13:18:53.780583+0000","last_clean":"2024-05-15T13:18:53.780583+0000","last_became_active":"2024-05-15T13:13:16.478031+0000","last_became_peered":"2024-05-15T13:13:16.478031+0000","last_unstale":"2024-05-15T13:18:53.780583+0000","last_undegraded":"2024-05-15T13:18:53.780583+0000","last_fullsized":"2024-05-15T13:18:53.780583+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.390903+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.390903+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:56.717010+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,11],"acting":[1,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":15,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:54.136887+0000","last_change":"2024-05-15T13:18:51.770698+0000","last_active":"2024-05-15T13:18:54.136887+0000","last_peered":"2024-05-15T13:18:54.136887+0000","last_clean":"2024-05-15T13:18:54.136887+0000","last_became_active":"2024-05-15T13:18:51.770335+0000","last_became_peered":"2024-05-15T13:18:51.770335+0000","last_unstale":"2024-05-15T13:18:54.136887+0000","last_undegraded":"2024-05-15T13:18:54.136887+0000","last_fullsized":"2024-05-15T13:18:54.136887+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:54.525803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,9],"acting":[10,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"94'139","reported_seq":1410,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:54.137729+0000","last_change":"2024-05-15T13:17:44.264911+0000","last_active":"2024-05-15T13:18:54.137729+0000","last_peered":"2024-05-15T13:18:54.137729+0000","last_clean":"2024-05-15T13:18:54.137729+0000","last_became_active":"2024-05-15T13:13:16.484252+0000","last_became_peered":"2024-05-15T13:13:16.484252+0000","last_unstale":"2024-05-15T13:18:54.137729+0000","last_undegraded":"2024-05-15T13:18:54.137729+0000","last_fullsized":"2024-05-15T13:18:54.137729+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:44.264755+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:17:44.264755+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:56.001100+0000","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"94'139","reported_seq":1390,"reported_epoch":741,"state":"active+clean","last_fresh":"2024-05-15T13:18:53.385125+0000","last_change":"2024-05-15T13:17:40.143307+0000","last_active":"2024-05-15T13:18:53.385125+0000","last_peered":"2024-05-15T13:18:53.385125+0000","last_clean":"2024-05-15T13:18:53.385125+0000","last_became_active":"2024-05-15T13:13:16.472800+0000","last_became_peered":"2024-05-15T13:13:16.472800+0000","last_unstale":"2024-05-15T13:18:53.385125+0000","last_undegraded":"2024-05-15T13:18:53.385125+0000","last_fullsized":"2024-05-15T13:18:53.385125+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:40.143179+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:17:40.143179+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:00.376173+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,8],"acting":[5,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"94'139","reported_seq":1166,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:53.780957+0000","last_change":"2024-05-15T13:18:30.438040+0000","last_active":"2024-05-15T13:18:53.780957+0000","last_peered":"2024-05-15T13:18:53.780957+0000","last_clean":"2024-05-15T13:18:53.780957+0000","last_became_active":"2024-05-15T13:13:16.476744+0000","last_became_peered":"2024-05-15T13:13:16.476744+0000","last_unstale":"2024-05-15T13:18:53.780957+0000","last_undegraded":"2024-05-15T13:18:53.780957+0000","last_fullsized":"2024-05-15T13:18:53.780957+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:18:30.437917+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:18:30.437917+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:31.780510+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"94'139","reported_seq":1414,"reported_epoch":741,"state":"active+clean","last_fresh":"2024-05-15T13:18:53.385153+0000","last_change":"2024-05-15T13:18:06.341818+0000","last_active":"2024-05-15T13:18:53.385153+0000","last_peered":"2024-05-15T13:18:53.385153+0000","last_clean":"2024-05-15T13:18:53.385153+0000","last_became_active":"2024-05-15T13:13:49.926687+0000","last_became_peered":"2024-05-15T13:13:49.926687+0000","last_unstale":"2024-05-15T13:18:53.385153+0000","last_undegraded":"2024-05-15T13:18:53.385153+0000","last_fullsized":"2024-05-15T13:18:53.385153+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:18:06.340555+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:18:06.340555+0000","last_clean_scrub_stamp":"2024-05-15T13:18:06.340555+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:14.400006+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"94'139","reported_seq":1408,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T13:18:53.780664+0000","last_change":"2024-05-15T13:13:20.455813+0000","last_active":"2024-05-15T13:18:53.780664+0000","last_peered":"2024-05-15T13:18:53.780664+0000","last_clean":"2024-05-15T13:18:53.780664+0000","last_became_active":"2024-05-15T13:13:16.478488+0000","last_became_peered":"2024-05-15T13:13:16.478488+0000","last_unstale":"2024-05-15T13:18:53.780664+0000","last_undegraded":"2024-05-15T13:18:53.780664+0000","last_fullsized":"2024-05-15T13:18:53.780664+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:20.455652+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:20.455652+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:18:56.717051+0000","scrub_duration":238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":9},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057043,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":951040,"kb_used_data":364536,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":92794112,"statfs":{"total":95995035648,"available":95021170688,"internally_reserved":0,"allocated":373284864,"data_stored":372620776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":600570310},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351383,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1014080,"kb_used_data":333176,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":92731072,"statfs":{"total":95995035648,"available":94956617728,"internally_reserved":0,"allocated":341172224,"data_stored":340508136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":697235904},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514908,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":817352,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":92927800,"statfs":{"total":95995035648,"available":95158067200,"internally_reserved":0,"allocated":1826816,"data_stored":1162728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":835123648},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874657,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":456392,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":93288760,"statfs":{"total":95995035648,"available":95527690240,"internally_reserved":0,"allocated":1826816,"data_stored":1162728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":240518168997,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430636,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":741308,"kb_used_data":2092,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":93003844,"statfs":{"total":95995035648,"available":95235936256,"internally_reserved":0,"allocated":2142208,"data_stored":1482598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":756939206},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575301,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":867332,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":92877820,"statfs":{"total":95995035648,"available":95106887680,"internally_reserved":0,"allocated":1826816,"data_stored":1162728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":886307269},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738830,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":759368,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92985784,"statfs":{"total":95995035648,"available":95217442816,"internally_reserved":0,"allocated":2154496,"data_stored":1490408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":775420357},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411778,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":812620,"kb_used_data":262208,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":92932532,"statfs":{"total":95995035648,"available":95162912768,"internally_reserved":0,"allocated":268500992,"data_stored":267833352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150139,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":940048,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92805104,"statfs":{"total":95995035648,"available":95032426496,"internally_reserved":0,"allocated":2359296,"data_stored":1691656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":960231879},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986615,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692273,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1319036,"kb_used_data":696424,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":92426116,"statfs":{"total":95995035648,"available":94644342784,"internally_reserved":0,"allocated":713138176,"data_stored":712479215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:19:02.065 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-05-15T13:19:02.066 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:19:02.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:01 smithi177 ceph-mon[44303]: from='client.16323 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:19:02.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:01 smithi177 ceph-mon[44303]: pgmap v1922: 37 pgs: 37 active+clean; 577 KiB data, 8.3 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:19:02.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:01 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:19:02.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:01 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2276159893' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:19:02.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:01 smithi177 ceph-mon[44303]: osdmap e743: 12 total, 12 up, 10 in 2024-05-15T13:19:02.247 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:19:02.248 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T13:19:02.248 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-05-15T13:19:02.248 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-05-15T13:19:02.248 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:19:02.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:01 smithi120 ceph-mon[43961]: from='client.16323 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:19:02.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:01 smithi120 ceph-mon[43961]: pgmap v1922: 37 pgs: 37 active+clean; 577 KiB data, 8.3 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:19:02.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:01 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:19:02.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:01 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2276159893' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:19:02.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:01 smithi120 ceph-mon[43961]: osdmap e743: 12 total, 12 up, 10 in 2024-05-15T13:19:02.840 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:19:02.910 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:02 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:19:02.910 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:02 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:19:02.910 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:02 smithi027 ceph-mon[39795]: 1.9 scrub starts 2024-05-15T13:19:02.910 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:02 smithi027 ceph-mon[39795]: 1.9 scrub ok 2024-05-15T13:19:02.911 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:02 smithi027 ceph-mon[39795]: osdmap e744: 12 total, 12 up, 10 in 2024-05-15T13:19:03.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:02 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:19:03.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:02 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:19:03.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:02 smithi177 ceph-mon[44303]: 1.9 scrub starts 2024-05-15T13:19:03.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:02 smithi177 ceph-mon[44303]: 1.9 scrub ok 2024-05-15T13:19:03.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:02 smithi177 ceph-mon[44303]: osdmap e744: 12 total, 12 up, 10 in 2024-05-15T13:19:03.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:02 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:19:03.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:02 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:19:03.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:02 smithi120 ceph-mon[43961]: 1.9 scrub starts 2024-05-15T13:19:03.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:02 smithi120 ceph-mon[43961]: 1.9 scrub ok 2024-05-15T13:19:03.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:02 smithi120 ceph-mon[43961]: osdmap e744: 12 total, 12 up, 10 in 2024-05-15T13:19:03.293 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:19:03.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:03.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:19:03.303 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-05-15T13:19:03.397 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:19:03.429 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:03.451 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:19:03.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:03.454 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:19:03.555 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:19:03.585 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:03.605 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:19:03.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:03.609 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:19:03.692 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:19:03.710 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:19:03.763 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:03.791 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:19:03.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:03.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:19:03.856 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:03 smithi027 ceph-mon[39795]: pgmap v1925: 37 pgs: 37 active+clean; 577 KiB data, 8.2 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:19:03.896 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:03.927 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:04.054 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:04.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:04.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:04.158 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:19:04.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:03 smithi177 ceph-mon[44303]: pgmap v1925: 37 pgs: 37 active+clean; 577 KiB data, 8.2 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:19:04.190 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:04.211 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:19:04.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:04.215 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:19:04.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:03 smithi120 ceph-mon[43961]: pgmap v1925: 37 pgs: 37 active+clean; 577 KiB data, 8.2 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:19:04.316 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:19:04.347 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:04.368 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:19:04.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:04.371 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:19:04.472 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:19:04.500 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:04.519 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:19:04.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:04.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:19:04.623 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:19:04.652 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:19:04.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:04.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:19:04.864 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:19:04.954 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:19:04.983 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:05.003 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:19:05.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:05.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:19:05.107 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:19:05.133 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:04 smithi120 ceph-mon[43961]: 1.2 scrub starts 2024-05-15T13:19:05.133 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:04 smithi120 ceph-mon[43961]: 1.2 scrub ok 2024-05-15T13:19:05.134 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:04 smithi120 ceph-mon[43961]: 1.f scrub starts 2024-05-15T13:19:05.134 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:04 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2289404765' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T13:19:05.134 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:04 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T13:19:05.134 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:04 smithi027 ceph-mon[39795]: 1.2 scrub starts 2024-05-15T13:19:05.134 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:04 smithi027 ceph-mon[39795]: 1.2 scrub ok 2024-05-15T13:19:05.134 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:04 smithi027 ceph-mon[39795]: 1.f scrub starts 2024-05-15T13:19:05.134 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:04 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2289404765' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T13:19:05.134 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:04 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T13:19:05.135 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:05.154 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:19:05.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:05.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:19:05.166 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rados' on pool 'unique_pool_6' 2024-05-15T13:19:05.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:04 smithi177 ceph-mon[44303]: 1.2 scrub starts 2024-05-15T13:19:05.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:04 smithi177 ceph-mon[44303]: 1.2 scrub ok 2024-05-15T13:19:05.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:04 smithi177 ceph-mon[44303]: 1.f scrub starts 2024-05-15T13:19:05.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:04 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2289404765' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T13:19:05.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:04 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T13:19:05.258 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:19:05.290 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:05.312 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:19:05.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:05.315 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:19:05.416 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:19:05.446 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:05.466 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:19:05.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:05.469 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:19:05.570 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:19:05.602 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:05.804 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:19:05.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:05.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:19:05.908 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:19:05.942 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:05.965 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:19:05.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:05.969 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:19:06.070 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:19:06.095 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:05 smithi120 ceph-mon[43961]: 1.f scrub ok 2024-05-15T13:19:06.095 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:05 smithi120 ceph-mon[43961]: pgmap v1926: 37 pgs: 37 active+clean; 577 KiB data, 8.2 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:19:06.095 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:05 smithi120 ceph-mon[43961]: 1.6 scrub starts 2024-05-15T13:19:06.095 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:05 smithi120 ceph-mon[43961]: 1.6 scrub ok 2024-05-15T13:19:06.095 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:05 smithi120 ceph-mon[43961]: 1.e scrub starts 2024-05-15T13:19:06.095 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:05 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T13:19:06.095 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:05 smithi120 ceph-mon[43961]: osdmap e745: 12 total, 12 up, 10 in 2024-05-15T13:19:06.096 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:06.115 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:19:06.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:06.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:19:06.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:05 smithi177 ceph-mon[44303]: 1.f scrub ok 2024-05-15T13:19:06.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:05 smithi177 ceph-mon[44303]: pgmap v1926: 37 pgs: 37 active+clean; 577 KiB data, 8.2 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:19:06.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:05 smithi177 ceph-mon[44303]: 1.6 scrub starts 2024-05-15T13:19:06.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:05 smithi177 ceph-mon[44303]: 1.6 scrub ok 2024-05-15T13:19:06.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:05 smithi177 ceph-mon[44303]: 1.e scrub starts 2024-05-15T13:19:06.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:05 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T13:19:06.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:05 smithi177 ceph-mon[44303]: osdmap e745: 12 total, 12 up, 10 in 2024-05-15T13:19:06.219 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:19:06.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:05 smithi027 ceph-mon[39795]: 1.f scrub ok 2024-05-15T13:19:06.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:05 smithi027 ceph-mon[39795]: pgmap v1926: 37 pgs: 37 active+clean; 577 KiB data, 8.2 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:19:06.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:05 smithi027 ceph-mon[39795]: 1.6 scrub starts 2024-05-15T13:19:06.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:05 smithi027 ceph-mon[39795]: 1.6 scrub ok 2024-05-15T13:19:06.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:05 smithi027 ceph-mon[39795]: 1.e scrub starts 2024-05-15T13:19:06.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:05 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T13:19:06.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:05 smithi027 ceph-mon[39795]: osdmap e745: 12 total, 12 up, 10 in 2024-05-15T13:19:06.246 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:06.264 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:19:06.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:06.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:19:06.368 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:19:06.394 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:06.413 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:19:06.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:06.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:19:06.517 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:19:06.543 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:06.562 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:19:06.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:06.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:19:06.625 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:19:06.666 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:06.694 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:19:06.696 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:06.715 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:19:06.734 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:19:06.754 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:19:06.777 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:19:06.798 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:19:06.819 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:19:06.839 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:19:06.863 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:19:06.890 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:19:06.909 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:19:06.930 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:19:06.931 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:19:06.931 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:19:06.931 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:19:06.931 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:19:06.931 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:19:06.931 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:19:06.932 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:19:06.932 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:19:06.932 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:19:06.932 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:19:06.932 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:19:06.932 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:19:06.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:06 smithi027 ceph-mon[39795]: 1.e scrub ok 2024-05-15T13:19:06.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:06 smithi027 ceph-mon[39795]: 1.7 scrub starts 2024-05-15T13:19:06.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:06 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3619683620' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:19:07.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:06 smithi177 ceph-mon[44303]: 1.e scrub ok 2024-05-15T13:19:07.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:06 smithi177 ceph-mon[44303]: 1.7 scrub starts 2024-05-15T13:19:07.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:06 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3619683620' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:19:07.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:06 smithi120 ceph-mon[43961]: 1.e scrub ok 2024-05-15T13:19:07.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:06 smithi120 ceph-mon[43961]: 1.7 scrub starts 2024-05-15T13:19:07.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:06 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3619683620' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:19:07.518 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:19:07.518 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:19:07.846 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:07.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:07.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:07.953 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:19:07.983 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:08.055 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:19:08.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:08.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:19:08.086 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:07 smithi027 ceph-mon[39795]: 1.7 scrub ok 2024-05-15T13:19:08.087 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:07 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:19:08.087 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:07 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:19:08.087 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:07 smithi027 ceph-mon[39795]: pgmap v1928: 37 pgs: 37 active+clean; 577 KiB data, 8.2 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:19:08.087 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:07 smithi027 ceph-mon[39795]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T13:19:08.087 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:07 smithi027 ceph-mon[39795]: osdmap e746: 12 total, 12 up, 10 in 2024-05-15T13:19:08.087 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:07 smithi027 ceph-mon[39795]: 1.10 scrub starts 2024-05-15T13:19:08.163 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:19:08.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:07 smithi177 ceph-mon[44303]: 1.7 scrub ok 2024-05-15T13:19:08.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:07 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:19:08.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:07 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:19:08.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:07 smithi177 ceph-mon[44303]: pgmap v1928: 37 pgs: 37 active+clean; 577 KiB data, 8.2 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:19:08.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:07 smithi177 ceph-mon[44303]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T13:19:08.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:07 smithi177 ceph-mon[44303]: osdmap e746: 12 total, 12 up, 10 in 2024-05-15T13:19:08.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:07 smithi177 ceph-mon[44303]: 1.10 scrub starts 2024-05-15T13:19:08.195 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:08.216 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:19:08.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:08.220 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:19:08.289 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:07 smithi120 ceph-mon[43961]: 1.7 scrub ok 2024-05-15T13:19:08.289 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:07 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:19:08.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:07 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:19:08.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:07 smithi120 ceph-mon[43961]: pgmap v1928: 37 pgs: 37 active+clean; 577 KiB data, 8.2 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:19:08.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:07 smithi120 ceph-mon[43961]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T13:19:08.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:07 smithi120 ceph-mon[43961]: osdmap e746: 12 total, 12 up, 10 in 2024-05-15T13:19:08.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:07 smithi120 ceph-mon[43961]: 1.10 scrub starts 2024-05-15T13:19:08.320 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:19:08.353 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:08.374 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:19:08.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:08.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:19:08.478 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:19:08.506 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:08.525 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:19:08.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:08.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:19:08.629 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:08.658 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:08.782 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:08.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:08.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:08.875 DEBUG:teuthology.orchestra.run.smithi177:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_6 bench 90 write' 2024-05-15T13:19:08.875 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-05-15T13:19:08.875 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:19:08.877 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-05-15T13:19:08.878 INFO:teuthology.orchestra.run:waiting for 3000 2024-05-15T13:19:08.894 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:19:08.921 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:08.940 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:19:08.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:08.944 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:19:08.982 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:hints = 1 2024-05-15T13:19:08.982 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-05-15T13:19:08.982 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Object prefix: benchmark_data_smithi177_149673 2024-05-15T13:19:08.982 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:19:08.982 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 0 0 0 0 0 0 - 0 2024-05-15T13:19:09.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:08 smithi027 ceph-mon[39795]: 1.10 scrub ok 2024-05-15T13:19:09.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:08 smithi027 ceph-mon[39795]: 1.8 scrub starts 2024-05-15T13:19:09.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:08 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:19:09.045 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:19:09.098 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:08 smithi177 ceph-mon[44303]: 1.10 scrub ok 2024-05-15T13:19:09.098 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:08 smithi177 ceph-mon[44303]: 1.8 scrub starts 2024-05-15T13:19:09.098 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:08 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:19:09.098 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:09.136 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:19:09.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:09.140 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:19:09.241 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:19:09.269 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-05-15T13:19:09.271 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:09.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:08 smithi120 ceph-mon[43961]: 1.10 scrub ok 2024-05-15T13:19:09.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:08 smithi120 ceph-mon[43961]: 1.8 scrub starts 2024-05-15T13:19:09.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:08 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:19:09.291 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:19:09.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:09.294 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:19:09.395 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:19:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:09.982 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 1 16 722 706 44.1215 44.125 0.0014074 0.00484404 2024-05-15T13:19:10.088 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:09 smithi027 ceph-mon[39795]: 1.8 scrub ok 2024-05-15T13:19:10.088 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:09 smithi027 ceph-mon[39795]: pgmap v1930: 37 pgs: 37 active+clean; 577 KiB data, 8.2 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:19:10.088 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:09 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:19:10.088 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:09 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:19:10.089 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:09 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:19:10.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:09 smithi177 ceph-mon[44303]: 1.8 scrub ok 2024-05-15T13:19:10.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:09 smithi177 ceph-mon[44303]: pgmap v1930: 37 pgs: 37 active+clean; 577 KiB data, 8.2 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:19:10.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:09 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:19:10.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:09 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:19:10.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:09 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:19:10.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:09 smithi120 ceph-mon[43961]: 1.8 scrub ok 2024-05-15T13:19:10.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:09 smithi120 ceph-mon[43961]: pgmap v1930: 37 pgs: 37 active+clean; 577 KiB data, 8.2 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:19:10.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:09 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:19:10.289 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:09 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:19:10.289 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:09 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:19:10.982 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 2 16 722 706 22.0607 0 - 0.00484404 2024-05-15T13:19:11.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:10 smithi177 ceph-mon[44303]: 1.b scrub starts 2024-05-15T13:19:11.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:10 smithi177 ceph-mon[44303]: 1.b scrub ok 2024-05-15T13:19:11.269 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:19:11.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:10 smithi027 ceph-mon[39795]: 1.b scrub starts 2024-05-15T13:19:11.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:10 smithi027 ceph-mon[39795]: 1.b scrub ok 2024-05-15T13:19:11.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:10 smithi120 ceph-mon[43961]: 1.b scrub starts 2024-05-15T13:19:11.288 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:10 smithi120 ceph-mon[43961]: 1.b scrub ok 2024-05-15T13:19:11.304 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:19:11.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:11.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:19:11.315 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:19:11.409 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:19:11.441 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:11.463 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:19:11.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:11.466 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:19:11.567 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:19:11.596 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:11.615 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:19:11.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:11.618 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:19:11.719 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:19:11.750 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:11.804 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:19:11.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:11.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:19:11.834 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:19:11.909 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:11.939 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:11.982 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 3 16 722 706 14.7069 0 - 0.00484404 2024-05-15T13:19:12.124 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:19:12.140 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:12.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:11 smithi027 ceph-mon[39795]: pgmap v1931: 37 pgs: 37 active+clean; 9.4 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 1.1 MiB/s wr, 17 op/s 2024-05-15T13:19:12.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:11 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:19:12.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:11 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:19:12.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:11 smithi027 ceph-mon[39795]: osdmap e747: 12 total, 12 up, 10 in 2024-05-15T13:19:12.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:11 smithi027 ceph-mon[39795]: 1.4 scrub starts 2024-05-15T13:19:12.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:11 smithi027 ceph-mon[39795]: 1.4 scrub ok 2024-05-15T13:19:12.144 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:11 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T13:19:12.144 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:11 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3254595321' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:19:12.144 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:11 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T13:19:12.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:12.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:12.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:11 smithi177 ceph-mon[44303]: pgmap v1931: 37 pgs: 37 active+clean; 9.4 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 1.1 MiB/s wr, 17 op/s 2024-05-15T13:19:12.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:11 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:19:12.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:11 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:19:12.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:11 smithi177 ceph-mon[44303]: osdmap e747: 12 total, 12 up, 10 in 2024-05-15T13:19:12.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:11 smithi177 ceph-mon[44303]: 1.4 scrub starts 2024-05-15T13:19:12.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:11 smithi177 ceph-mon[44303]: 1.4 scrub ok 2024-05-15T13:19:12.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:11 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T13:19:12.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:11 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3254595321' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:19:12.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:11 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T13:19:12.245 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:19:12.275 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:11 smithi120 ceph-mon[43961]: pgmap v1931: 37 pgs: 37 active+clean; 9.4 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 1.1 MiB/s wr, 17 op/s 2024-05-15T13:19:12.275 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:11 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:19:12.276 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:11 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:19:12.276 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:11 smithi120 ceph-mon[43961]: osdmap e747: 12 total, 12 up, 10 in 2024-05-15T13:19:12.276 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:11 smithi120 ceph-mon[43961]: 1.4 scrub starts 2024-05-15T13:19:12.276 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:11 smithi120 ceph-mon[43961]: 1.4 scrub ok 2024-05-15T13:19:12.276 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:11 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T13:19:12.276 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:11 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3254595321' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:19:12.276 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:11 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T13:19:12.278 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:12.299 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:19:12.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:12.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:19:12.404 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:19:12.433 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:12.454 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:19:12.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:12.457 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:19:12.558 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:19:12.591 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:12.612 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:19:12.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:12.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:19:12.717 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:19:12.746 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:12.768 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:19:12.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:12.771 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:19:12.872 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:19:12.898 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:12.917 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:19:12.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:12.920 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:19:12.982 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 4 16 722 706 11.0302 0 - 0.00484404 2024-05-15T13:19:13.021 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:19:13.045 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:12 smithi177 ceph-mon[44303]: 1.a scrub starts 2024-05-15T13:19:13.045 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:12 smithi177 ceph-mon[44303]: 1.a scrub ok 2024-05-15T13:19:13.045 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:12 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:19:13.045 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:12 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T13:19:13.046 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:12 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3254595321' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:19:13.046 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:12 smithi177 ceph-mon[44303]: osdmap e748: 12 total, 12 up, 10 in 2024-05-15T13:19:13.046 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:12 smithi177 ceph-mon[44303]: 1.c scrub starts 2024-05-15T13:19:13.046 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:12 smithi177 ceph-mon[44303]: 1.c scrub ok 2024-05-15T13:19:13.047 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:13.066 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:19:13.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:13.069 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:19:13.169 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:19:13.173 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:12 smithi027 ceph-mon[39795]: 1.a scrub starts 2024-05-15T13:19:13.174 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:12 smithi027 ceph-mon[39795]: 1.a scrub ok 2024-05-15T13:19:13.174 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:12 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:19:13.174 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:12 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T13:19:13.174 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:12 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3254595321' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:19:13.174 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:12 smithi027 ceph-mon[39795]: osdmap e748: 12 total, 12 up, 10 in 2024-05-15T13:19:13.174 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:12 smithi027 ceph-mon[39795]: 1.c scrub starts 2024-05-15T13:19:13.174 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:12 smithi027 ceph-mon[39795]: 1.c scrub ok 2024-05-15T13:19:13.199 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:12 smithi120 ceph-mon[43961]: 1.a scrub starts 2024-05-15T13:19:13.199 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:12 smithi120 ceph-mon[43961]: 1.a scrub ok 2024-05-15T13:19:13.199 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:12 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:19:13.200 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:12 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T13:19:13.200 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:12 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3254595321' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:19:13.200 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:12 smithi120 ceph-mon[43961]: osdmap e748: 12 total, 12 up, 10 in 2024-05-15T13:19:13.200 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:12 smithi120 ceph-mon[43961]: 1.c scrub starts 2024-05-15T13:19:13.200 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:12 smithi120 ceph-mon[43961]: 1.c scrub ok 2024-05-15T13:19:13.201 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:13.223 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:19:13.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:13.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:19:13.327 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:19:13.354 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:13.373 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:19:13.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:13.376 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:19:13.477 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:19:13.508 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:13.530 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:19:13.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:13.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:19:13.633 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:13.663 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:13.737 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:13.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:13.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:13.841 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:19:13.871 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:13.892 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:19:13.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:13.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:19:13.982 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 5 16 722 706 8.82406 0 - 0.00484404 2024-05-15T13:19:13.996 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:14.024 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:14.151 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:14.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:13 smithi027 ceph-mon[39795]: pgmap v1934: 37 pgs: 37 active+clean; 15 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 2.4 MiB/s wr, 37 op/s 2024-05-15T13:19:14.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:13 smithi027 ceph-mon[39795]: 1.1 scrub starts 2024-05-15T13:19:14.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:13 smithi027 ceph-mon[39795]: 1.1 scrub ok 2024-05-15T13:19:14.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:14.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:14.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:13 smithi177 ceph-mon[44303]: pgmap v1934: 37 pgs: 37 active+clean; 15 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 2.4 MiB/s wr, 37 op/s 2024-05-15T13:19:14.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:13 smithi177 ceph-mon[44303]: 1.1 scrub starts 2024-05-15T13:19:14.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:13 smithi177 ceph-mon[44303]: 1.1 scrub ok 2024-05-15T13:19:14.255 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:19:14.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:13 smithi120 ceph-mon[43961]: pgmap v1934: 37 pgs: 37 active+clean; 15 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 2.4 MiB/s wr, 37 op/s 2024-05-15T13:19:14.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:13 smithi120 ceph-mon[43961]: 1.1 scrub starts 2024-05-15T13:19:14.284 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:13 smithi120 ceph-mon[43961]: 1.1 scrub ok 2024-05-15T13:19:14.286 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:14.307 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:19:14.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:14.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:19:14.411 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:14.439 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:14.875 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:19:14.875 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:19:14.983 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 6 16 722 706 7.35334 0 - 0.00484404 2024-05-15T13:19:15.255 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:19:15.257 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:19:15.290 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:15.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:15.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:15.395 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:19:15.424 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:15 smithi027 ceph-mon[39795]: 1.d scrub starts 2024-05-15T13:19:15.424 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:15 smithi027 ceph-mon[39795]: pgmap v1935: 37 pgs: 37 active+clean; 46 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 5.8 MiB/s wr, 92 op/s 2024-05-15T13:19:15.427 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:15.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:15 smithi177 ceph-mon[44303]: 1.d scrub starts 2024-05-15T13:19:15.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:15 smithi177 ceph-mon[44303]: pgmap v1935: 37 pgs: 37 active+clean; 46 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 5.8 MiB/s wr, 92 op/s 2024-05-15T13:19:15.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:15 smithi120 ceph-mon[43961]: 1.d scrub starts 2024-05-15T13:19:15.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:15 smithi120 ceph-mon[43961]: pgmap v1935: 37 pgs: 37 active+clean; 46 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 5.8 MiB/s wr, 92 op/s 2024-05-15T13:19:15.541 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:19:15.983 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 7 16 722 706 6.30279 0 - 0.00484404 2024-05-15T13:19:15.988 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-05-15T13:19:16.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:16 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:19:16.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:16 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:19:16.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:16 smithi027 ceph-mon[39795]: 1.d scrub ok 2024-05-15T13:19:16.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:16 smithi027 ceph-mon[39795]: osdmap e749: 12 total, 12 up, 10 in 2024-05-15T13:19:16.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:16 smithi027 ceph-mon[39795]: from='client.16362 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:19:16.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:16 smithi027 ceph-mon[39795]: 1.13 scrub starts 2024-05-15T13:19:16.418 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":1936,"stamp":"2024-05-15T13:19:15.168811+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47907360,"num_objects":724,"num_object_clones":0,"num_object_copies":2172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":722,"num_write_kb":46208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3641,"ondisk_log_size":3641,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":117,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":8622924,"kb_used_data":1612560,"kb_used_omap":15,"kb_used_meta":7010224,"kb_avail":928828596,"statfs":{"total":959950356480,"available":951120482304,"internally_reserved":0,"allocated":1651261440,"data_stored":1644370584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15956,"internal_metadata":7178469804},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":47316992,"num_objects":722,"num_object_clones":0,"num_object_copies":2166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":722,"num_write_kb":46208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.587090"},"pg_stats":[{"pgid":"1.14","version":"94'139","reported_seq":828,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:11.836360+0000","last_change":"2024-05-15T13:18:17.100933+0000","last_active":"2024-05-15T13:19:11.836360+0000","last_peered":"2024-05-15T13:19:11.836360+0000","last_clean":"2024-05-15T13:19:11.836360+0000","last_became_active":"2024-05-15T13:13:16.473818+0000","last_became_peered":"2024-05-15T13:13:16.473818+0000","last_unstale":"2024-05-15T13:19:11.836360+0000","last_undegraded":"2024-05-15T13:19:11.836360+0000","last_fullsized":"2024-05-15T13:19:11.836360+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:18:17.100768+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:18:17.100768+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:25.805853+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1],"acting":[2,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"746'39","reported_seq":51,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:12.091774+0000","last_change":"2024-05-15T13:18:51.769954+0000","last_active":"2024-05-15T13:19:12.091774+0000","last_peered":"2024-05-15T13:19:12.091774+0000","last_clean":"2024-05-15T13:19:12.091774+0000","last_became_active":"2024-05-15T13:18:51.769627+0000","last_became_peered":"2024-05-15T13:18:51.769627+0000","last_unstale":"2024-05-15T13:19:12.091774+0000","last_undegraded":"2024-05-15T13:19:12.091774+0000","last_fullsized":"2024-05-15T13:19:12.091774+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:11.531828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2555904,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":2496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,0],"acting":[5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":1403,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:11.836913+0000","last_change":"2024-05-15T13:19:11.513679+0000","last_active":"2024-05-15T13:19:11.836913+0000","last_peered":"2024-05-15T13:19:11.836913+0000","last_clean":"2024-05-15T13:19:11.836913+0000","last_became_active":"2024-05-15T13:13:11.361366+0000","last_became_peered":"2024-05-15T13:13:11.361366+0000","last_unstale":"2024-05-15T13:19:11.836913+0000","last_undegraded":"2024-05-15T13:19:11.836913+0000","last_fullsized":"2024-05-15T13:19:11.836913+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:11.513579+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:11.513579+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:35.021808+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.d","version":"746'57","reported_seq":85,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:11.834685+0000","last_change":"2024-05-15T13:18:51.823482+0000","last_active":"2024-05-15T13:19:11.834685+0000","last_peered":"2024-05-15T13:19:11.834685+0000","last_clean":"2024-05-15T13:19:11.834685+0000","last_became_active":"2024-05-15T13:18:51.823045+0000","last_became_peered":"2024-05-15T13:18:51.823045+0000","last_unstale":"2024-05-15T13:19:11.834685+0000","last_undegraded":"2024-05-15T13:19:11.834685+0000","last_fullsized":"2024-05-15T13:19:11.834685+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:07.422810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3735552,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"94'139","reported_seq":1412,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:11.834716+0000","last_change":"2024-05-15T13:19:11.008069+0000","last_active":"2024-05-15T13:19:11.834716+0000","last_peered":"2024-05-15T13:19:11.834716+0000","last_clean":"2024-05-15T13:19:11.834716+0000","last_became_active":"2024-05-15T13:13:11.371643+0000","last_became_peered":"2024-05-15T13:13:11.371643+0000","last_unstale":"2024-05-15T13:19:11.834716+0000","last_undegraded":"2024-05-15T13:19:11.834716+0000","last_fullsized":"2024-05-15T13:19:11.834716+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:11.007920+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:19:11.007920+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:38.435892+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"746'41","reported_seq":65,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-05-15T13:19:09.250052+0000","last_change":"2024-05-15T13:18:51.771623+0000","last_active":"2024-05-15T13:19:09.250052+0000","last_peered":"2024-05-15T13:19:09.250052+0000","last_clean":"2024-05-15T13:19:09.250052+0000","last_became_active":"2024-05-15T13:18:51.771276+0000","last_became_peered":"2024-05-15T13:18:51.771276+0000","last_unstale":"2024-05-15T13:19:09.250052+0000","last_undegraded":"2024-05-15T13:19:09.250052+0000","last_fullsized":"2024-05-15T13:19:09.250052+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:16.197016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2686976,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41,"num_write_kb":2624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"94'139","reported_seq":1155,"reported_epoch":747,"state":"active+clean","last_fresh":"2024-05-15T13:19:11.376175+0000","last_change":"2024-05-15T13:18:24.970432+0000","last_active":"2024-05-15T13:19:11.376175+0000","last_peered":"2024-05-15T13:19:11.376175+0000","last_clean":"2024-05-15T13:19:11.376175+0000","last_became_active":"2024-05-15T13:14:54.577458+0000","last_became_peered":"2024-05-15T13:14:54.577458+0000","last_unstale":"2024-05-15T13:19:11.376175+0000","last_undegraded":"2024-05-15T13:19:11.376175+0000","last_fullsized":"2024-05-15T13:19:11.376175+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:18:24.970285+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:18:24.970285+0000","last_clean_scrub_stamp":"2024-05-15T13:18:24.970285+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:53.226976+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"746'46","reported_seq":74,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:11.834763+0000","last_change":"2024-05-15T13:18:51.825618+0000","last_active":"2024-05-15T13:19:11.834763+0000","last_peered":"2024-05-15T13:19:11.834763+0000","last_clean":"2024-05-15T13:19:11.834763+0000","last_became_active":"2024-05-15T13:18:51.825302+0000","last_became_peered":"2024-05-15T13:18:51.825302+0000","last_unstale":"2024-05-15T13:19:11.834763+0000","last_undegraded":"2024-05-15T13:19:11.834763+0000","last_fullsized":"2024-05-15T13:19:11.834763+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:07.838359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3014656,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":2944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"94'139","reported_seq":1400,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-05-15T13:19:06.891198+0000","last_change":"2024-05-15T13:19:04.701370+0000","last_active":"2024-05-15T13:19:06.891198+0000","last_peered":"2024-05-15T13:19:06.891198+0000","last_clean":"2024-05-15T13:19:06.891198+0000","last_became_active":"2024-05-15T13:13:11.373251+0000","last_became_peered":"2024-05-15T13:13:11.373251+0000","last_unstale":"2024-05-15T13:19:06.891198+0000","last_undegraded":"2024-05-15T13:19:06.891198+0000","last_fullsized":"2024-05-15T13:19:06.891198+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:04.701253+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:04.701253+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:14.045749+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.e","version":"746'49","reported_seq":77,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:12.609554+0000","last_change":"2024-05-15T13:18:51.823811+0000","last_active":"2024-05-15T13:19:12.609554+0000","last_peered":"2024-05-15T13:19:12.609554+0000","last_clean":"2024-05-15T13:19:12.609554+0000","last_became_active":"2024-05-15T13:18:51.823490+0000","last_became_peered":"2024-05-15T13:18:51.823490+0000","last_unstale":"2024-05-15T13:19:12.609554+0000","last_undegraded":"2024-05-15T13:19:12.609554+0000","last_fullsized":"2024-05-15T13:19:12.609554+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:58.173536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3211264,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":3136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,11],"acting":[0,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"94'139","reported_seq":1171,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:12.397052+0000","last_change":"2024-05-15T13:19:06.027430+0000","last_active":"2024-05-15T13:19:12.397052+0000","last_peered":"2024-05-15T13:19:12.397052+0000","last_clean":"2024-05-15T13:19:12.397052+0000","last_became_active":"2024-05-15T13:14:11.522490+0000","last_became_peered":"2024-05-15T13:14:11.522490+0000","last_unstale":"2024-05-15T13:19:12.397052+0000","last_undegraded":"2024-05-15T13:19:12.397052+0000","last_fullsized":"2024-05-15T13:19:12.397052+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:06.027331+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:06.027331+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:24.071024+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"746'46","reported_seq":74,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:12.091718+0000","last_change":"2024-05-15T13:18:51.770676+0000","last_active":"2024-05-15T13:19:12.091718+0000","last_peered":"2024-05-15T13:19:12.091718+0000","last_clean":"2024-05-15T13:19:12.091718+0000","last_became_active":"2024-05-15T13:18:51.770286+0000","last_became_peered":"2024-05-15T13:18:51.770286+0000","last_unstale":"2024-05-15T13:19:12.091718+0000","last_undegraded":"2024-05-15T13:19:12.091718+0000","last_fullsized":"2024-05-15T13:19:12.091718+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:05.255233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3014656,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":2944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"94'139","reported_seq":1170,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-05-15T13:19:06.890883+0000","last_change":"2024-05-15T13:17:41.721650+0000","last_active":"2024-05-15T13:19:06.890883+0000","last_peered":"2024-05-15T13:19:06.890883+0000","last_clean":"2024-05-15T13:19:06.890883+0000","last_became_active":"2024-05-15T13:15:43.614970+0000","last_became_peered":"2024-05-15T13:15:43.614970+0000","last_unstale":"2024-05-15T13:19:06.890883+0000","last_undegraded":"2024-05-15T13:19:06.890883+0000","last_fullsized":"2024-05-15T13:19:06.890883+0000","mapping_epoch":678,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:41.721516+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:17:41.721516+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,0],"acting":[9,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.b","version":"746'53","reported_seq":77,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-05-15T13:19:09.233438+0000","last_change":"2024-05-15T13:18:51.768048+0000","last_active":"2024-05-15T13:19:09.233438+0000","last_peered":"2024-05-15T13:19:09.233438+0000","last_clean":"2024-05-15T13:19:09.233438+0000","last_became_active":"2024-05-15T13:18:51.767604+0000","last_became_peered":"2024-05-15T13:18:51.767604+0000","last_unstale":"2024-05-15T13:19:09.233438+0000","last_undegraded":"2024-05-15T13:19:09.233438+0000","last_fullsized":"2024-05-15T13:19:09.233438+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:01.870097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3473408,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53,"num_write_kb":3392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"94'139","reported_seq":1414,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:12.609601+0000","last_change":"2024-05-15T13:19:03.412805+0000","last_active":"2024-05-15T13:19:12.609601+0000","last_peered":"2024-05-15T13:19:12.609601+0000","last_clean":"2024-05-15T13:19:12.609601+0000","last_became_active":"2024-05-15T13:13:11.373548+0000","last_became_peered":"2024-05-15T13:13:11.373548+0000","last_unstale":"2024-05-15T13:19:12.609601+0000","last_undegraded":"2024-05-15T13:19:12.609601+0000","last_fullsized":"2024-05-15T13:19:12.609601+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:03.412703+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:03.412703+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:31.550085+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"746'39","reported_seq":63,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-05-15T13:19:09.247143+0000","last_change":"2024-05-15T13:18:51.823543+0000","last_active":"2024-05-15T13:19:09.247143+0000","last_peered":"2024-05-15T13:19:09.247143+0000","last_clean":"2024-05-15T13:19:09.247143+0000","last_became_active":"2024-05-15T13:18:51.823137+0000","last_became_peered":"2024-05-15T13:18:51.823137+0000","last_unstale":"2024-05-15T13:19:09.247143+0000","last_undegraded":"2024-05-15T13:19:09.247143+0000","last_fullsized":"2024-05-15T13:19:09.247143+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:12.114852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2555904,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":2496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"94'139","reported_seq":1421,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:11.837307+0000","last_change":"2024-05-15T13:14:54.570374+0000","last_active":"2024-05-15T13:19:11.837307+0000","last_peered":"2024-05-15T13:19:11.837307+0000","last_clean":"2024-05-15T13:19:11.837307+0000","last_became_active":"2024-05-15T13:14:54.570004+0000","last_became_peered":"2024-05-15T13:14:54.570004+0000","last_unstale":"2024-05-15T13:19:11.837307+0000","last_undegraded":"2024-05-15T13:19:11.837307+0000","last_fullsized":"2024-05-15T13:19:11.837307+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:18.468889+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:18.468889+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":220,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"746'46","reported_seq":74,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:11.836925+0000","last_change":"2024-05-15T13:18:51.824173+0000","last_active":"2024-05-15T13:19:11.836925+0000","last_peered":"2024-05-15T13:19:11.836925+0000","last_clean":"2024-05-15T13:19:11.836925+0000","last_became_active":"2024-05-15T13:18:51.823777+0000","last_became_peered":"2024-05-15T13:18:51.823777+0000","last_unstale":"2024-05-15T13:19:11.836925+0000","last_undegraded":"2024-05-15T13:19:11.836925+0000","last_fullsized":"2024-05-15T13:19:11.836925+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:20.118351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3014656,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":2944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"94'139","reported_seq":1406,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:12.091903+0000","last_change":"2024-05-15T13:19:08.450681+0000","last_active":"2024-05-15T13:19:12.091903+0000","last_peered":"2024-05-15T13:19:12.091903+0000","last_clean":"2024-05-15T13:19:12.091903+0000","last_became_active":"2024-05-15T13:06:46.426908+0000","last_became_peered":"2024-05-15T13:06:46.426908+0000","last_unstale":"2024-05-15T13:19:12.091903+0000","last_undegraded":"2024-05-15T13:19:12.091903+0000","last_fullsized":"2024-05-15T13:19:12.091903+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:08.450567+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:08.450567+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:11.781968+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.0","version":"746'51","reported_seq":79,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:12.595849+0000","last_change":"2024-05-15T13:18:51.770583+0000","last_active":"2024-05-15T13:19:12.595849+0000","last_peered":"2024-05-15T13:19:12.595849+0000","last_clean":"2024-05-15T13:19:12.595849+0000","last_became_active":"2024-05-15T13:18:51.770274+0000","last_became_peered":"2024-05-15T13:18:51.770274+0000","last_unstale":"2024-05-15T13:19:12.595849+0000","last_undegraded":"2024-05-15T13:19:12.595849+0000","last_fullsized":"2024-05-15T13:19:12.595849+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:11.283287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3342336,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51,"num_write_kb":3264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"94'139","reported_seq":1394,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-05-15T13:19:06.891146+0000","last_change":"2024-05-15T13:19:01.735324+0000","last_active":"2024-05-15T13:19:06.891146+0000","last_peered":"2024-05-15T13:19:06.891146+0000","last_clean":"2024-05-15T13:19:06.891146+0000","last_became_active":"2024-05-15T13:06:46.424667+0000","last_became_peered":"2024-05-15T13:06:46.424667+0000","last_unstale":"2024-05-15T13:19:06.891146+0000","last_undegraded":"2024-05-15T13:19:06.891146+0000","last_fullsized":"2024-05-15T13:19:06.891146+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:01.735216+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:01.735216+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:27.546887+0000","scrub_duration":181,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.3","version":"746'41","reported_seq":69,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:11.834682+0000","last_change":"2024-05-15T13:18:51.769282+0000","last_active":"2024-05-15T13:19:11.834682+0000","last_peered":"2024-05-15T13:19:11.834682+0000","last_clean":"2024-05-15T13:19:11.834682+0000","last_became_active":"2024-05-15T13:18:51.768837+0000","last_became_peered":"2024-05-15T13:18:51.768837+0000","last_unstale":"2024-05-15T13:19:11.834682+0000","last_undegraded":"2024-05-15T13:19:11.834682+0000","last_fullsized":"2024-05-15T13:19:11.834682+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:14.139438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2686976,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41,"num_write_kb":2624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"94'139","reported_seq":1165,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-05-15T13:19:06.891146+0000","last_change":"2024-05-15T13:17:45.527703+0000","last_active":"2024-05-15T13:19:06.891146+0000","last_peered":"2024-05-15T13:19:06.891146+0000","last_clean":"2024-05-15T13:19:06.891146+0000","last_became_active":"2024-05-15T13:15:43.611913+0000","last_became_peered":"2024-05-15T13:15:43.611913+0000","last_unstale":"2024-05-15T13:19:06.891146+0000","last_undegraded":"2024-05-15T13:19:06.891146+0000","last_fullsized":"2024-05-15T13:19:06.891146+0000","mapping_epoch":678,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:17:45.527612+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:17:45.527612+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.7","version":"746'39","reported_seq":67,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:11.837151+0000","last_change":"2024-05-15T13:18:51.767665+0000","last_active":"2024-05-15T13:19:11.837151+0000","last_peered":"2024-05-15T13:19:11.837151+0000","last_clean":"2024-05-15T13:19:11.837151+0000","last_became_active":"2024-05-15T13:18:51.767342+0000","last_became_peered":"2024-05-15T13:18:51.767342+0000","last_unstale":"2024-05-15T13:19:11.837151+0000","last_undegraded":"2024-05-15T13:19:11.837151+0000","last_fullsized":"2024-05-15T13:19:11.837151+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:53.885208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2555904,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":2496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"94'139","reported_seq":1077,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:12.595885+0000","last_change":"2024-05-15T13:19:05.011225+0000","last_active":"2024-05-15T13:19:12.595885+0000","last_peered":"2024-05-15T13:19:12.595885+0000","last_clean":"2024-05-15T13:19:12.595885+0000","last_became_active":"2024-05-15T13:13:16.477771+0000","last_became_peered":"2024-05-15T13:13:16.477771+0000","last_unstale":"2024-05-15T13:19:12.595885+0000","last_undegraded":"2024-05-15T13:19:12.595885+0000","last_fullsized":"2024-05-15T13:19:12.595885+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:05.011120+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:05.011120+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:23.470999+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,1],"acting":[3,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"746'39","reported_seq":67,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:11.834807+0000","last_change":"2024-05-15T13:18:51.823957+0000","last_active":"2024-05-15T13:19:11.834807+0000","last_peered":"2024-05-15T13:19:11.834807+0000","last_clean":"2024-05-15T13:19:11.834807+0000","last_became_active":"2024-05-15T13:18:51.823632+0000","last_became_peered":"2024-05-15T13:18:51.823632+0000","last_unstale":"2024-05-15T13:19:11.834807+0000","last_undegraded":"2024-05-15T13:19:11.834807+0000","last_fullsized":"2024-05-15T13:19:11.834807+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:04.211611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2555904,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":2496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,2],"acting":[11,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"94'139","reported_seq":1170,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:12.397150+0000","last_change":"2024-05-15T13:19:03.984125+0000","last_active":"2024-05-15T13:19:12.397150+0000","last_peered":"2024-05-15T13:19:12.397150+0000","last_clean":"2024-05-15T13:19:12.397150+0000","last_became_active":"2024-05-15T13:13:16.472318+0000","last_became_peered":"2024-05-15T13:13:16.472318+0000","last_unstale":"2024-05-15T13:19:12.397150+0000","last_undegraded":"2024-05-15T13:19:12.397150+0000","last_fullsized":"2024-05-15T13:19:12.397150+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:03.983993+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:03.983993+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:21.133720+0000","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"746'48","reported_seq":76,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:11.837030+0000","last_change":"2024-05-15T13:18:51.767865+0000","last_active":"2024-05-15T13:19:11.837030+0000","last_peered":"2024-05-15T13:19:11.837030+0000","last_clean":"2024-05-15T13:19:11.837030+0000","last_became_active":"2024-05-15T13:18:51.767454+0000","last_became_peered":"2024-05-15T13:18:51.767454+0000","last_unstale":"2024-05-15T13:19:11.837030+0000","last_undegraded":"2024-05-15T13:19:11.837030+0000","last_fullsized":"2024-05-15T13:19:11.837030+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:11.694078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,11],"acting":[1,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"94'139","reported_seq":829,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:12.595890+0000","last_change":"2024-05-15T13:19:11.928809+0000","last_active":"2024-05-15T13:19:12.595890+0000","last_peered":"2024-05-15T13:19:12.595890+0000","last_clean":"2024-05-15T13:19:12.595890+0000","last_became_active":"2024-05-15T13:13:16.476972+0000","last_became_peered":"2024-05-15T13:13:16.476972+0000","last_unstale":"2024-05-15T13:19:12.595890+0000","last_undegraded":"2024-05-15T13:19:12.595890+0000","last_fullsized":"2024-05-15T13:19:12.595890+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:11.928703+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:19:11.928703+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:37.054546+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"746'44","reported_seq":72,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:12.091990+0000","last_change":"2024-05-15T13:18:51.769457+0000","last_active":"2024-05-15T13:19:12.091990+0000","last_peered":"2024-05-15T13:19:12.091990+0000","last_clean":"2024-05-15T13:19:12.091990+0000","last_became_active":"2024-05-15T13:18:51.769098+0000","last_became_peered":"2024-05-15T13:18:51.769098+0000","last_unstale":"2024-05-15T13:19:12.091990+0000","last_undegraded":"2024-05-15T13:19:12.091990+0000","last_fullsized":"2024-05-15T13:19:12.091990+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:58.796977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2883584,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":2816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"94'139","reported_seq":809,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:11.837146+0000","last_change":"2024-05-15T13:13:17.391026+0000","last_active":"2024-05-15T13:19:11.837146+0000","last_peered":"2024-05-15T13:19:11.837146+0000","last_clean":"2024-05-15T13:19:11.837146+0000","last_became_active":"2024-05-15T13:13:16.478031+0000","last_became_peered":"2024-05-15T13:13:16.478031+0000","last_unstale":"2024-05-15T13:19:11.837146+0000","last_undegraded":"2024-05-15T13:19:11.837146+0000","last_fullsized":"2024-05-15T13:19:11.837146+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:17.390903+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:17.390903+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,11],"acting":[1,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"746'44","reported_seq":70,"reported_epoch":747,"state":"active+clean","last_fresh":"2024-05-15T13:19:11.375966+0000","last_change":"2024-05-15T13:18:51.770698+0000","last_active":"2024-05-15T13:19:11.375966+0000","last_peered":"2024-05-15T13:19:11.375966+0000","last_clean":"2024-05-15T13:19:11.375966+0000","last_became_active":"2024-05-15T13:18:51.770335+0000","last_became_peered":"2024-05-15T13:18:51.770335+0000","last_unstale":"2024-05-15T13:19:11.375966+0000","last_undegraded":"2024-05-15T13:19:11.375966+0000","last_fullsized":"2024-05-15T13:19:11.375966+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:09.106204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2883584,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":2816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,9],"acting":[10,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"94'139","reported_seq":1430,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:11.834622+0000","last_change":"2024-05-15T13:19:10.071697+0000","last_active":"2024-05-15T13:19:11.834622+0000","last_peered":"2024-05-15T13:19:11.834622+0000","last_clean":"2024-05-15T13:19:11.834622+0000","last_became_active":"2024-05-15T13:13:16.484252+0000","last_became_peered":"2024-05-15T13:13:16.484252+0000","last_unstale":"2024-05-15T13:19:11.834622+0000","last_undegraded":"2024-05-15T13:19:11.834622+0000","last_fullsized":"2024-05-15T13:19:11.834622+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:10.071555+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:10.071555+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:27.164091+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"94'139","reported_seq":1412,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:12.091768+0000","last_change":"2024-05-15T13:19:07.405009+0000","last_active":"2024-05-15T13:19:12.091768+0000","last_peered":"2024-05-15T13:19:12.091768+0000","last_clean":"2024-05-15T13:19:12.091768+0000","last_became_active":"2024-05-15T13:13:16.472800+0000","last_became_peered":"2024-05-15T13:13:16.472800+0000","last_unstale":"2024-05-15T13:19:12.091768+0000","last_undegraded":"2024-05-15T13:19:12.091768+0000","last_fullsized":"2024-05-15T13:19:12.091768+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:07.404921+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:07.404921+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:13.770633+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,8],"acting":[5,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"94'139","reported_seq":1178,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:12.609497+0000","last_change":"2024-05-15T13:18:30.438040+0000","last_active":"2024-05-15T13:19:12.609497+0000","last_peered":"2024-05-15T13:19:12.609497+0000","last_clean":"2024-05-15T13:19:12.609497+0000","last_became_active":"2024-05-15T13:13:16.476744+0000","last_became_peered":"2024-05-15T13:13:16.476744+0000","last_unstale":"2024-05-15T13:19:12.609497+0000","last_undegraded":"2024-05-15T13:19:12.609497+0000","last_fullsized":"2024-05-15T13:19:12.609497+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:18:30.437917+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:18:30.437917+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:31.780510+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"94'139","reported_seq":1428,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:12.091836+0000","last_change":"2024-05-15T13:18:06.341818+0000","last_active":"2024-05-15T13:19:12.091836+0000","last_peered":"2024-05-15T13:19:12.091836+0000","last_clean":"2024-05-15T13:19:12.091836+0000","last_became_active":"2024-05-15T13:13:49.926687+0000","last_became_peered":"2024-05-15T13:13:49.926687+0000","last_unstale":"2024-05-15T13:19:12.091836+0000","last_undegraded":"2024-05-15T13:19:12.091836+0000","last_fullsized":"2024-05-15T13:19:12.091836+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:18:06.340555+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:18:06.340555+0000","last_clean_scrub_stamp":"2024-05-15T13:18:06.340555+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:19:14.400006+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"94'139","reported_seq":1420,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-05-15T13:19:11.837248+0000","last_change":"2024-05-15T13:13:20.455813+0000","last_active":"2024-05-15T13:19:11.837248+0000","last_peered":"2024-05-15T13:19:11.837248+0000","last_clean":"2024-05-15T13:19:11.837248+0000","last_became_active":"2024-05-15T13:13:16.478488+0000","last_became_peered":"2024-05-15T13:13:16.478488+0000","last_unstale":"2024-05-15T13:19:11.837248+0000","last_undegraded":"2024-05-15T13:19:11.837248+0000","last_fullsized":"2024-05-15T13:19:11.837248+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:13:20.455652+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:13:20.455652+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":47316992,"num_objects":722,"num_object_clones":0,"num_object_copies":2166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":722,"num_write_kb":46208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":140902400,"data_stored":140902400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":722,"ondisk_log_size":722,"up":48,"acting":48,"num_store_stats":9},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057046,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":946128,"kb_used_data":359616,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":92799024,"statfs":{"total":95995035648,"available":95026200576,"internally_reserved":0,"allocated":368246784,"data_stored":367556070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":600570310},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351386,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":972720,"kb_used_data":291816,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":92772432,"statfs":{"total":95995035648,"available":94998970368,"internally_reserved":0,"allocated":298819584,"data_stored":298137850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":697235904},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514911,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":837296,"kb_used_data":21736,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":92907856,"statfs":{"total":95995035648,"available":95137644544,"internally_reserved":0,"allocated":22257664,"data_stored":21575930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":835123648},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874660,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":464944,"kb_used_data":10344,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":93280208,"statfs":{"total":95995035648,"available":95518932992,"internally_reserved":0,"allocated":10592256,"data_stored":9910522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":240518169000,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430640,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":741392,"kb_used_data":2176,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":93003760,"statfs":{"total":95995035648,"available":95235850240,"internally_reserved":0,"allocated":2228224,"data_stored":1537510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":756939206},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575304,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":884876,"kb_used_data":19328,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":92860276,"statfs":{"total":95995035648,"available":95088922624,"internally_reserved":0,"allocated":19791872,"data_stored":19101158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":886307269},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738834,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":766864,"kb_used_data":9600,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92978288,"statfs":{"total":95995035648,"available":95209766912,"internally_reserved":0,"allocated":9830400,"data_stored":9139686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":775420357},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411781,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":799636,"kb_used_data":249224,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":92945516,"statfs":{"total":95995035648,"available":95176208384,"internally_reserved":0,"allocated":255205376,"data_stored":254511110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150142,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":948948,"kb_used_data":11208,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92796204,"statfs":{"total":95995035648,"available":95023312896,"internally_reserved":0,"allocated":11476992,"data_stored":10782726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":960231879},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986618,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692277,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1260120,"kb_used_data":637512,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":92485032,"statfs":{"total":95995035648,"available":94704672768,"internally_reserved":0,"allocated":652812288,"data_stored":652118022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7602176,"data_stored":7602176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":17891328,"data_stored":17891328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18087936,"data_stored":18087936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9043968,"data_stored":9043968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":18219008,"data_stored":18219008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8716288,"data_stored":8716288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20381696,"data_stored":20381696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":16580608,"data_stored":16580608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24379392,"data_stored":24379392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:19:16.420 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool set .mgr pgp_num 21 2024-05-15T13:19:16.422 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:19:16.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:16 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:19:16.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:16 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:19:16.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:16 smithi177 ceph-mon[44303]: 1.d scrub ok 2024-05-15T13:19:16.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:16 smithi177 ceph-mon[44303]: osdmap e749: 12 total, 12 up, 10 in 2024-05-15T13:19:16.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:16 smithi177 ceph-mon[44303]: from='client.16362 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:19:16.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:16 smithi177 ceph-mon[44303]: 1.13 scrub starts 2024-05-15T13:19:16.457 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:19:16.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:16.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:19:16.470 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:19:16.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:16 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:19:16.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:16 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:19:16.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:16 smithi120 ceph-mon[43961]: 1.d scrub ok 2024-05-15T13:19:16.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:16 smithi120 ceph-mon[43961]: osdmap e749: 12 total, 12 up, 10 in 2024-05-15T13:19:16.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:16 smithi120 ceph-mon[43961]: from='client.16362 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:19:16.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:16 smithi120 ceph-mon[43961]: 1.13 scrub starts 2024-05-15T13:19:16.566 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:19:16.595 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:16.695 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:19:16.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:16.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:19:16.799 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:16.827 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:16.976 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:16.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:16.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:16.983 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 8 16 722 706 5.51493 0 - 0.00484404 2024-05-15T13:19:17.080 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:19:17.112 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:17.134 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:19:17.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:17.137 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:19:17.238 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:17.265 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:17 smithi027 ceph-mon[39795]: 1.13 scrub ok 2024-05-15T13:19:17.266 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:17 smithi027 ceph-mon[39795]: pgmap v1937: 37 pgs: 37 active+clean; 46 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 6.1 MiB/s wr, 96 op/s 2024-05-15T13:19:17.268 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:17.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:17 smithi177 ceph-mon[44303]: 1.13 scrub ok 2024-05-15T13:19:17.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:17 smithi177 ceph-mon[44303]: pgmap v1937: 37 pgs: 37 active+clean; 46 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 6.1 MiB/s wr, 96 op/s 2024-05-15T13:19:17.499 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:19:17.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:17 smithi120 ceph-mon[43961]: 1.13 scrub ok 2024-05-15T13:19:17.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:17 smithi120 ceph-mon[43961]: pgmap v1937: 37 pgs: 37 active+clean; 46 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 6.1 MiB/s wr, 96 op/s 2024-05-15T13:19:17.825 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:17.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:17.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:17.868 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:19:17.929 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:19:17.961 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:17.982 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:19:17.983 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 9 16 722 706 4.90217 0 - 0.00484404 2024-05-15T13:19:17.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:17.985 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:19:18.086 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:19:18.119 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:18.320 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:19:18.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:18.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:19:18.340 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:18 smithi027 ceph-mon[39795]: 1.12 scrub starts 2024-05-15T13:19:18.340 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:18 smithi027 ceph-mon[39795]: 1.12 scrub ok 2024-05-15T13:19:18.340 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:18 smithi027 ceph-mon[39795]: 1.3 scrub starts 2024-05-15T13:19:18.424 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:19:18.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:18 smithi177 ceph-mon[44303]: 1.12 scrub starts 2024-05-15T13:19:18.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:18 smithi177 ceph-mon[44303]: 1.12 scrub ok 2024-05-15T13:19:18.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:18 smithi177 ceph-mon[44303]: 1.3 scrub starts 2024-05-15T13:19:18.456 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:18.478 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:19:18.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:18.481 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:19:18.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:18 smithi120 ceph-mon[43961]: 1.12 scrub starts 2024-05-15T13:19:18.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:18 smithi120 ceph-mon[43961]: 1.12 scrub ok 2024-05-15T13:19:18.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:18 smithi120 ceph-mon[43961]: 1.3 scrub starts 2024-05-15T13:19:18.582 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:19:18.614 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:18.636 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:19:18.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:18.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:19:18.740 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:19:18.772 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:18.793 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:19:18.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:18.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:19:18.898 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:18.927 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:18.983 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 10 16 722 706 4.41197 0 - 0.00484404 2024-05-15T13:19:19.177 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:19.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:19.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:19.282 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:19.310 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:19.411 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:19.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:19.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:19.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:19 smithi177 ceph-mon[44303]: 1.3 scrub ok 2024-05-15T13:19:19.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:19 smithi177 ceph-mon[44303]: pgmap v1938: 37 pgs: 37 active+clean; 46 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 4.4 MiB/s wr, 69 op/s 2024-05-15T13:19:19.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:19 smithi177 ceph-mon[44303]: osdmap e750: 12 total, 12 up, 10 in 2024-05-15T13:19:19.453 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:19 smithi027 ceph-mon[39795]: 1.3 scrub ok 2024-05-15T13:19:19.453 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:19 smithi027 ceph-mon[39795]: pgmap v1938: 37 pgs: 37 active+clean; 46 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 4.4 MiB/s wr, 69 op/s 2024-05-15T13:19:19.453 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:19 smithi027 ceph-mon[39795]: osdmap e750: 12 total, 12 up, 10 in 2024-05-15T13:19:19.515 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:19:19.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:19 smithi120 ceph-mon[43961]: 1.3 scrub ok 2024-05-15T13:19:19.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:19 smithi120 ceph-mon[43961]: pgmap v1938: 37 pgs: 37 active+clean; 46 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 4.4 MiB/s wr, 69 op/s 2024-05-15T13:19:19.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:19 smithi120 ceph-mon[43961]: osdmap e750: 12 total, 12 up, 10 in 2024-05-15T13:19:19.549 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:19.570 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:19:19.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:19.574 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:19:19.675 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:19:19.704 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:19.983 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 11 16 722 706 4.01087 0 - 0.00484404 2024-05-15T13:19:20.654 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:19:20.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:20.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:19:20.758 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:20.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:20 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1121026439' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:19:20.913 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:20.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:20.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:20.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:20 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1121026439' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:19:20.983 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 12 16 722 706 3.67662 0 - 0.00484404 2024-05-15T13:19:21.017 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:19:21.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:20 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1121026439' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:19:21.049 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:21.071 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:19:21.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:21.074 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:19:21.175 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:19:21.207 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:21.228 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:19:21.228 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T13:19:21.228 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-05-15T13:19:21.228 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-05-15T13:19:21.228 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:19:21.229 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:19:21.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:21.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:19:21.332 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:19:21.362 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:21.383 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:19:21.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:21.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:19:21.487 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:19:21.516 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:21.536 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:19:21.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:21.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:19:21.641 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:19:21.672 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:21.694 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:19:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:21.697 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:19:21.788 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:21 smithi120 ceph-mon[43961]: pgmap v1940: 37 pgs: 37 active+clean; 46 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 3.9 MiB/s wr, 61 op/s 2024-05-15T13:19:21.798 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:21.829 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:21.847 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:19:21.847 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:19:21.847 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:19:21.847 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:19:21.848 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:19:21.848 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:19:21.848 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:19:21.848 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:19:21.848 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:19:21.848 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:19:21.848 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:19:21.848 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:19:21.848 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:19:21.848 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:19:21.848 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:19:21.848 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:19:21.848 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:19:21.849 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:19:21.849 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:19:21.849 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:19:21.849 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:19:21.849 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:19:21.849 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:19:21.849 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:19:21.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:21 smithi177 ceph-mon[44303]: pgmap v1940: 37 pgs: 37 active+clean; 46 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 3.9 MiB/s wr, 61 op/s 2024-05-15T13:19:21.984 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 13 16 722 706 3.39379 0 - 0.00484404 2024-05-15T13:19:22.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:21 smithi027 ceph-mon[39795]: pgmap v1940: 37 pgs: 37 active+clean; 46 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 3.9 MiB/s wr, 61 op/s 2024-05-15T13:19:22.080 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:22.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:22.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:22.185 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:19:22.217 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:22.239 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:19:22.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:22.243 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:19:22.344 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:19:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:22.572 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:19:22.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:22.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:19:22.645 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:22 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2823775712' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:19:22.676 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:19:22.702 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:19:22.705 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:22.906 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:19:22.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:22.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:19:22.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:22 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2823775712' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:19:22.984 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 14 16 722 706 3.15138 0 - 0.00484404 2024-05-15T13:19:23.010 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:19:23.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:22 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2823775712' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:19:23.043 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:23.064 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:19:23.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:23.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:19:23.169 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:19:23.200 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:23.221 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:19:23.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:23.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:19:23.325 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:19:23.358 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-05-15T13:19:23.359 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:23.378 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:19:23.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:23.381 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:19:23.482 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:19:23.515 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:23.536 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:19:23.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:23.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:19:23.641 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:19:23.670 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:23.691 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:19:23.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:23.694 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:19:23.788 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:23 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:19:23.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:23 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:19:23.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:23 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:19:23.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:23 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:19:23.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:23 smithi120 ceph-mon[43961]: pgmap v1941: 37 pgs: 37 active+clean; 46 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:19:23.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:23 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:19:23.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:23 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2823775712' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:19:23.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:23 smithi120 ceph-mon[43961]: osdmap e751: 12 total, 12 up, 10 in 2024-05-15T13:19:23.794 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:19:23.824 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:23.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:23 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:19:23.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:23 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:19:23.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:23 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:19:23.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:23 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:19:23.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:23 smithi177 ceph-mon[44303]: pgmap v1941: 37 pgs: 37 active+clean; 46 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:19:23.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:23 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:19:23.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:23 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2823775712' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:19:23.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:23 smithi177 ceph-mon[44303]: osdmap e751: 12 total, 12 up, 10 in 2024-05-15T13:19:23.984 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 15 16 722 706 2.94128 0 - 0.00484404 2024-05-15T13:19:24.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:23 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:19:24.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:23 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:19:24.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:23 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:19:24.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:23 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:19:24.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:23 smithi027 ceph-mon[39795]: pgmap v1941: 37 pgs: 37 active+clean; 46 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail 2024-05-15T13:19:24.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:23 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:19:24.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:23 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2823775712' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:19:24.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:23 smithi027 ceph-mon[39795]: osdmap e751: 12 total, 12 up, 10 in 2024-05-15T13:19:24.072 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:19:24.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:24.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:19:24.177 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:19:24.210 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:24.234 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:19:24.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:24.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:19:24.339 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:19:24.367 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:24.387 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:19:24.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:24.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:19:24.491 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:19:24.541 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:24.557 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:19:24.757 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:19:24.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:24.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:19:24.826 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:19:24.870 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:19:24.899 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:24.921 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:19:24.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:24.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:19:24.984 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 16 16 3581 3565 13.9236 11.9125 0.00339062 0.0718023 2024-05-15T13:19:25.024 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:25.070 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:25.105 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:25.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:25.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:25.191 INFO:teuthology.orchestra.run.smithi027.stderr:set pool 1 pgp_num to 21 2024-05-15T13:19:25.211 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:19:25.251 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:25.284 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:19:25.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:25.287 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:19:25.388 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:19:25.426 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:25 smithi027 ceph-mon[39795]: osdmap e752: 12 total, 12 up, 10 in 2024-05-15T13:19:25.426 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:25 smithi027 ceph-mon[39795]: pgmap v1944: 37 pgs: 37 active+clean; 49 MiB data, 8.1 GiB used, 886 GiB / 894 GiB avail; 501 KiB/s wr, 7 op/s 2024-05-15T13:19:25.426 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:25 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1484493233' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-05-15T13:19:25.430 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:25.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:25 smithi177 ceph-mon[44303]: osdmap e752: 12 total, 12 up, 10 in 2024-05-15T13:19:25.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:25 smithi177 ceph-mon[44303]: pgmap v1944: 37 pgs: 37 active+clean; 49 MiB data, 8.1 GiB used, 886 GiB / 894 GiB avail; 501 KiB/s wr, 7 op/s 2024-05-15T13:19:25.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:25 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1484493233' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-05-15T13:19:25.465 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:19:25.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:25.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:19:25.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:25 smithi120 ceph-mon[43961]: osdmap e752: 12 total, 12 up, 10 in 2024-05-15T13:19:25.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:25 smithi120 ceph-mon[43961]: pgmap v1944: 37 pgs: 37 active+clean; 49 MiB data, 8.1 GiB used, 886 GiB / 894 GiB avail; 501 KiB/s wr, 7 op/s 2024-05-15T13:19:25.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:25 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1484493233' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-05-15T13:19:25.570 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:19:25.598 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:25.619 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:19:25.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:25.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:19:25.723 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:19:25.751 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:25.949 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:19:25.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:25.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:19:25.977 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:19:25.984 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 17 16 6343 6327 23.2574 172.625 0.00148078 0.042863 2024-05-15T13:19:26.055 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:19:26.081 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:26.099 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:19:26.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:26.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:19:26.203 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:19:26.255 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:26.333 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:19:26.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:26.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:19:26.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:26 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1484493233' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-05-15T13:19:26.371 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:26 smithi027 ceph-mon[39795]: osdmap e753: 12 total, 12 up, 10 in 2024-05-15T13:19:26.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:26 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1484493233' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-05-15T13:19:26.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:26 smithi177 ceph-mon[44303]: osdmap e753: 12 total, 12 up, 10 in 2024-05-15T13:19:26.439 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:19:26.466 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:26 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1484493233' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-05-15T13:19:26.467 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:26 smithi120 ceph-mon[43961]: osdmap e753: 12 total, 12 up, 10 in 2024-05-15T13:19:26.469 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:26.490 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:19:26.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:26.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:19:26.593 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:19:26.623 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:26.644 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:19:26.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:26.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:19:26.749 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:19:26.774 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:26.794 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:19:26.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:26.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:19:26.813 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:19:26.897 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:26.925 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:26.985 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 18 16 8113 8097 28.1102 110.625 0.00124335 0.0348958 2024-05-15T13:19:27.487 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:27 smithi027 ceph-mon[39795]: pgmap v1946: 37 pgs: 37 active+clean; 96 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 8.4 MiB/s wr, 133 op/s 2024-05-15T13:19:27.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:27 smithi120 ceph-mon[43961]: pgmap v1946: 37 pgs: 37 active+clean; 96 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 8.4 MiB/s wr, 133 op/s 2024-05-15T13:19:27.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:27 smithi177 ceph-mon[44303]: pgmap v1946: 37 pgs: 37 active+clean; 96 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 8.4 MiB/s wr, 133 op/s 2024-05-15T13:19:27.783 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:19:27.783 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T13:19:27.783 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T13:19:27.783 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:19:27.783 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.783 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:258455 6.e 6:7555f542:::benchmark_data_smithi177_122278_object128084:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-05-15T13:19:27.783 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:12:01.366853+0000", 2024-05-15T13:19:27.783 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 446.41629031399998, 2024-05-15T13:19:27.783 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3991274410000001, 2024-05-15T13:19:27.784 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:27.784 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:27.784 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:27.784 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:19:27.784 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:19:27.784 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 258455 2024-05-15T13:19:27.784 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.784 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:27.784 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.784 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:27.784 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.366853+0000", 2024-05-15T13:19:27.784 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.784 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.785 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.785 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:27.785 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.366853+0000", 2024-05-15T13:19:27.785 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.785 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.785 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.785 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:27.785 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.366854+0000", 2024-05-15T13:19:27.785 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.403e-06 2024-05-15T13:19:27.785 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.785 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.785 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:27.785 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.366860+0000", 2024-05-15T13:19:27.786 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9220000000000002e-06 2024-05-15T13:19:27.786 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.786 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.786 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:27.786 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.366874+0000", 2024-05-15T13:19:27.786 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3200000000000001e-05 2024-05-15T13:19:27.786 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.786 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.786 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:27.786 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.366881+0000", 2024-05-15T13:19:27.786 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3860000000000001e-06 2024-05-15T13:19:27.786 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.786 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.787 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.787 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.366953+0000", 2024-05-15T13:19:27.787 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1713999999999998e-05 2024-05-15T13:19:27.787 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.787 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.787 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:27.787 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.366966+0000", 2024-05-15T13:19:27.787 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3345e-05 2024-05-15T13:19:27.787 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.787 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.787 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.787 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:02.325917+0000", 2024-05-15T13:19:27.787 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.95895053799999996 2024-05-15T13:19:27.787 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.788 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.788 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:27.788 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:02.325934+0000", 2024-05-15T13:19:27.788 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7076999999999999e-05 2024-05-15T13:19:27.788 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.788 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.788 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.788 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.737349+0000", 2024-05-15T13:19:27.788 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4114149550000001 2024-05-15T13:19:27.788 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.788 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.788 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:27.788 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.737467+0000", 2024-05-15T13:19:27.789 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011877 2024-05-15T13:19:27.789 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.789 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.789 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,5,11", 2024-05-15T13:19:27.789 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.737592+0000", 2024-05-15T13:19:27.789 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000124945 2024-05-15T13:19:27.789 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.789 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.789 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.789 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.758516+0000", 2024-05-15T13:19:27.789 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.020923481000000001 2024-05-15T13:19:27.789 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.789 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.790 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.790 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.758559+0000", 2024-05-15T13:19:27.790 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3485999999999997e-05 2024-05-15T13:19:27.790 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.790 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.790 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.790 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.758638+0000", 2024-05-15T13:19:27.790 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9011999999999995e-05 2024-05-15T13:19:27.790 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.790 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.790 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:27.790 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.765793+0000", 2024-05-15T13:19:27.790 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.007155069 2024-05-15T13:19:27.791 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.791 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.791 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:27.791 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.765860+0000", 2024-05-15T13:19:27.791 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6693999999999995e-05 2024-05-15T13:19:27.791 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.791 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.791 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:27.791 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.765980+0000", 2024-05-15T13:19:27.791 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120444 2024-05-15T13:19:27.791 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.791 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:27.791 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.791 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.791 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.792 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:258464 6.e 6:7324b118:::benchmark_data_smithi177_122278_object128093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-05-15T13:19:27.792 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:12:01.384839+0000", 2024-05-15T13:19:27.792 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 446.39830389899998, 2024-05-15T13:19:27.792 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3813195820000002, 2024-05-15T13:19:27.792 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:27.792 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:27.792 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:27.792 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:19:27.792 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:19:27.792 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 258464 2024-05-15T13:19:27.792 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.792 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:27.792 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.793 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:27.793 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.384839+0000", 2024-05-15T13:19:27.793 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.793 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.793 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.793 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:27.793 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.384839+0000", 2024-05-15T13:19:27.793 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.793 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.793 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.793 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:27.793 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.384841+0000", 2024-05-15T13:19:27.793 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6959999999999999e-06 2024-05-15T13:19:27.793 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.794 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.794 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:27.794 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.384851+0000", 2024-05-15T13:19:27.794 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5559999999999996e-06 2024-05-15T13:19:27.794 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.794 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.794 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:27.794 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.384869+0000", 2024-05-15T13:19:27.794 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8712999999999999e-05 2024-05-15T13:19:27.794 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.794 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.794 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:27.794 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.384879+0000", 2024-05-15T13:19:27.794 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1290000000000002e-06 2024-05-15T13:19:27.795 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.795 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.795 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.795 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.384935+0000", 2024-05-15T13:19:27.795 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6561999999999998e-05 2024-05-15T13:19:27.795 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.795 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.795 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:27.795 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.384943+0000", 2024-05-15T13:19:27.795 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9270000000000005e-06 2024-05-15T13:19:27.795 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.795 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.795 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.796 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:02.325970+0000", 2024-05-15T13:19:27.796 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.94102689699999997 2024-05-15T13:19:27.796 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.796 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.796 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:27.796 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:02.325978+0000", 2024-05-15T13:19:27.796 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8469999999999996e-06 2024-05-15T13:19:27.796 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.796 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.796 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.796 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.743320+0000", 2024-05-15T13:19:27.796 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.417342042 2024-05-15T13:19:27.796 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.796 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.797 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:27.797 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.743461+0000", 2024-05-15T13:19:27.797 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000141485 2024-05-15T13:19:27.797 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.797 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.797 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,5,11", 2024-05-15T13:19:27.797 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.743606+0000", 2024-05-15T13:19:27.797 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014419500000000001 2024-05-15T13:19:27.797 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.797 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.797 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.797 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.758599+0000", 2024-05-15T13:19:27.797 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014993737 2024-05-15T13:19:27.798 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.798 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.798 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.798 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.758678+0000", 2024-05-15T13:19:27.798 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9093999999999999e-05 2024-05-15T13:19:27.798 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.798 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.798 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.798 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.758719+0000", 2024-05-15T13:19:27.798 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0327999999999997e-05 2024-05-15T13:19:27.798 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.798 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.798 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:27.799 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.765989+0000", 2024-05-15T13:19:27.799 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.007270014 2024-05-15T13:19:27.799 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.799 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.799 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:27.799 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.766039+0000", 2024-05-15T13:19:27.799 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0723000000000002e-05 2024-05-15T13:19:27.799 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.799 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.799 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:27.799 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.766159+0000", 2024-05-15T13:19:27.799 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011963700000000001 2024-05-15T13:19:27.799 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.799 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:27.800 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.800 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.800 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.800 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:258470 6.e 6:75cb7168:::benchmark_data_smithi177_122278_object128099:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-05-15T13:19:27.800 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:12:01.402708+0000", 2024-05-15T13:19:27.800 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 446.38043568199998, 2024-05-15T13:19:27.800 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3637355700000002, 2024-05-15T13:19:27.800 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:27.800 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:27.800 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:27.800 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:19:27.800 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:19:27.800 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 258470 2024-05-15T13:19:27.800 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.800 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:27.801 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.801 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:27.801 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.402708+0000", 2024-05-15T13:19:27.801 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.801 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.801 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.801 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:27.801 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.402708+0000", 2024-05-15T13:19:27.801 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.801 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.801 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.801 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:27.801 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.402710+0000", 2024-05-15T13:19:27.801 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.996e-06 2024-05-15T13:19:27.802 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.802 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.802 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:27.802 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.402717+0000", 2024-05-15T13:19:27.802 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7530000000000001e-06 2024-05-15T13:19:27.802 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.802 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.802 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:27.802 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.402734+0000", 2024-05-15T13:19:27.802 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6935999999999998e-05 2024-05-15T13:19:27.802 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.802 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.802 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:27.802 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.402743+0000", 2024-05-15T13:19:27.803 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2470000000000008e-06 2024-05-15T13:19:27.803 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.803 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.803 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.803 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.402800+0000", 2024-05-15T13:19:27.803 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7482000000000003e-05 2024-05-15T13:19:27.803 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.803 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.803 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:27.803 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.402810+0000", 2024-05-15T13:19:27.803 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.73e-06 2024-05-15T13:19:27.803 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.803 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.803 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.804 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:02.326006+0000", 2024-05-15T13:19:27.804 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.92319602899999997 2024-05-15T13:19:27.804 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.804 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.804 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:27.804 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:02.326012+0000", 2024-05-15T13:19:27.804 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6259999999999997e-06 2024-05-15T13:19:27.804 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.804 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.804 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.804 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.748628+0000", 2024-05-15T13:19:27.804 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.422615526 2024-05-15T13:19:27.804 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.805 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.805 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:27.805 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.748739+0000", 2024-05-15T13:19:27.805 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000111481 2024-05-15T13:19:27.805 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.805 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.805 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,5,11", 2024-05-15T13:19:27.805 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.748863+0000", 2024-05-15T13:19:27.805 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000123044 2024-05-15T13:19:27.805 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.805 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.805 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.805 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.765775+0000", 2024-05-15T13:19:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016912316 2024-05-15T13:19:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.766166+0000", 2024-05-15T13:19:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00039137200000000002 2024-05-15T13:19:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.766218+0000", 2024-05-15T13:19:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1705e-05 2024-05-15T13:19:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.766295+0000", 2024-05-15T13:19:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7287999999999996e-05 2024-05-15T13:19:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.766344+0000", 2024-05-15T13:19:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.922e-05 2024-05-15T13:19:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.766443+0000", 2024-05-15T13:19:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8819000000000004e-05 2024-05-15T13:19:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:3273 7.e 7:7aa309c8:::benchmark_data_smithi177_134484_object3272:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-05-15T13:19:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:12:43.971551+0000", 2024-05-15T13:19:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 403.81159273399999, 2024-05-15T13:19:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.204857286, 2024-05-15T13:19:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:19:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:19:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 3273 2024-05-15T13:19:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.971551+0000", 2024-05-15T13:19:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.971551+0000", 2024-05-15T13:19:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.971552+0000", 2024-05-15T13:19:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1319999999999999e-06 2024-05-15T13:19:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.971580+0000", 2024-05-15T13:19:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8459999999999999e-05 2024-05-15T13:19:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.971591+0000", 2024-05-15T13:19:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.093e-05 2024-05-15T13:19:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.971598+0000", 2024-05-15T13:19:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2729999999999997e-06 2024-05-15T13:19:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.971767+0000", 2024-05-15T13:19:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016906199999999999 2024-05-15T13:19:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.971942+0000", 2024-05-15T13:19:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000174203 2024-05-15T13:19:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:19:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.972076+0000", 2024-05-15T13:19:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013431900000000001 2024-05-15T13:19:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.972642+0000", 2024-05-15T13:19:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00056553099999999998 2024-05-15T13:19:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.973874+0000", 2024-05-15T13:19:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001232784 2024-05-15T13:19:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.813 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.813 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.176292+0000", 2024-05-15T13:19:27.813 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20241753400000001 2024-05-15T13:19:27.813 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.813 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.813 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:27.813 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.176355+0000", 2024-05-15T13:19:27.813 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3385000000000004e-05 2024-05-15T13:19:27.813 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.813 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.813 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:27.813 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.176408+0000", 2024-05-15T13:19:27.813 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2673000000000002e-05 2024-05-15T13:19:27.813 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.814 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:27.814 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.814 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.814 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.814 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:3275 7.e 7:7aa86ea8:::benchmark_data_smithi177_134484_object3274:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-05-15T13:19:27.814 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:12:43.972997+0000", 2024-05-15T13:19:27.814 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 403.81014639300003, 2024-05-15T13:19:27.814 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20358997200000001, 2024-05-15T13:19:27.814 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:27.814 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:27.814 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:27.814 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:19:27.814 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:19:27.815 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 3275 2024-05-15T13:19:27.815 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.815 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:27.815 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.815 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:27.815 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.972997+0000", 2024-05-15T13:19:27.815 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.815 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.815 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.815 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:27.815 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.972997+0000", 2024-05-15T13:19:27.815 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.815 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.815 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.816 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:27.816 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.972998+0000", 2024-05-15T13:19:27.816 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2600000000000001e-07 2024-05-15T13:19:27.816 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.816 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.816 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:27.816 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.973164+0000", 2024-05-15T13:19:27.816 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016599799999999999 2024-05-15T13:19:27.816 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.816 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.816 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:27.816 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.973181+0000", 2024-05-15T13:19:27.816 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7002000000000001e-05 2024-05-15T13:19:27.816 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.816 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.816 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:27.817 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.973189+0000", 2024-05-15T13:19:27.817 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5140000000000001e-06 2024-05-15T13:19:27.817 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.817 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.817 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.817 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.973249+0000", 2024-05-15T13:19:27.817 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9423999999999999e-05 2024-05-15T13:19:27.817 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.817 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.817 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:27.817 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.973390+0000", 2024-05-15T13:19:27.817 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000140815 2024-05-15T13:19:27.817 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.818 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.818 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:19:27.818 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.973532+0000", 2024-05-15T13:19:27.818 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014255299999999999 2024-05-15T13:19:27.818 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.818 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.818 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:27.818 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.974164+0000", 2024-05-15T13:19:27.818 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00063157199999999995 2024-05-15T13:19:27.818 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.818 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.818 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.818 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.975000+0000", 2024-05-15T13:19:27.818 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00083587799999999997 2024-05-15T13:19:27.819 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.819 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.819 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.819 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.176467+0000", 2024-05-15T13:19:27.819 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20146778200000001 2024-05-15T13:19:27.819 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.819 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.819 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:27.819 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.176538+0000", 2024-05-15T13:19:27.819 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0439999999999996e-05 2024-05-15T13:19:27.819 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.819 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.819 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:27.819 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.176587+0000", 2024-05-15T13:19:27.819 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9168000000000001e-05 2024-05-15T13:19:27.820 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.820 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:27.820 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.820 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.820 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.820 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:3295 7.e 7:7db7815a:::benchmark_data_smithi177_134484_object3294:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-05-15T13:19:27.820 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:12:43.978971+0000", 2024-05-15T13:19:27.820 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 403.80417192900001, 2024-05-15T13:19:27.820 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19820074200000001, 2024-05-15T13:19:27.820 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:27.820 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:27.820 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:27.820 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:19:27.820 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:19:27.820 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 3295 2024-05-15T13:19:27.821 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.821 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:27.821 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.821 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:27.821 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.978971+0000", 2024-05-15T13:19:27.821 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.821 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.821 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.821 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:27.821 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.978971+0000", 2024-05-15T13:19:27.821 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.821 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.821 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.821 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:27.821 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.978973+0000", 2024-05-15T13:19:27.821 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2249999999999999e-06 2024-05-15T13:19:27.822 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.822 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.822 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:27.822 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.979010+0000", 2024-05-15T13:19:27.822 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6899e-05 2024-05-15T13:19:27.822 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.822 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.822 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:27.822 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.979023+0000", 2024-05-15T13:19:27.822 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3356999999999999e-05 2024-05-15T13:19:27.822 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.822 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.822 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:27.822 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.979029+0000", 2024-05-15T13:19:27.822 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8019999999999997e-06 2024-05-15T13:19:27.823 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.823 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.823 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.823 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.979076+0000", 2024-05-15T13:19:27.823 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7490000000000001e-05 2024-05-15T13:19:27.823 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.823 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.823 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:27.823 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.979211+0000", 2024-05-15T13:19:27.823 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013521599999999999 2024-05-15T13:19:27.823 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.823 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.823 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:19:27.824 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.979364+0000", 2024-05-15T13:19:27.824 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015244600000000001 2024-05-15T13:19:27.824 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.824 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.824 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:27.824 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.980080+0000", 2024-05-15T13:19:27.824 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00071620400000000002 2024-05-15T13:19:27.824 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.824 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.824 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.824 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.980309+0000", 2024-05-15T13:19:27.824 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00022928400000000001 2024-05-15T13:19:27.824 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.825 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.825 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.825 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.177053+0000", 2024-05-15T13:19:27.825 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.196743905 2024-05-15T13:19:27.825 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.825 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.825 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:27.825 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.177132+0000", 2024-05-15T13:19:27.825 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8337000000000006e-05 2024-05-15T13:19:27.825 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.825 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.825 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:27.825 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.177172+0000", 2024-05-15T13:19:27.825 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0577000000000001e-05 2024-05-15T13:19:27.826 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.826 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:27.826 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.826 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.826 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.826 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:3298 7.e 7:764ed9c4:::benchmark_data_smithi177_134484_object3297:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-05-15T13:19:27.826 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:12:43.980204+0000", 2024-05-15T13:19:27.826 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 403.80293889799998, 2024-05-15T13:19:27.826 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.19709568299999999, 2024-05-15T13:19:27.826 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:27.826 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:27.826 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:27.826 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:19:27.827 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:19:27.827 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 3298 2024-05-15T13:19:27.827 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.827 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:27.827 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.827 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:27.827 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.980204+0000", 2024-05-15T13:19:27.827 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.827 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.827 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.827 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:27.827 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.980204+0000", 2024-05-15T13:19:27.827 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.828 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.828 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.828 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:27.828 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.980205+0000", 2024-05-15T13:19:27.828 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.61e-07 2024-05-15T13:19:27.828 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.828 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.828 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:27.828 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.981349+0000", 2024-05-15T13:19:27.828 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001143638 2024-05-15T13:19:27.828 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.828 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.828 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:27.828 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.981362+0000", 2024-05-15T13:19:27.829 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3349999999999999e-05 2024-05-15T13:19:27.829 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.829 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.829 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:27.829 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.981368+0000", 2024-05-15T13:19:27.829 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.728e-06 2024-05-15T13:19:27.829 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.829 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.829 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.829 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.981408+0000", 2024-05-15T13:19:27.829 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0071000000000003e-05 2024-05-15T13:19:27.829 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.829 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.829 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:27.829 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.981495+0000", 2024-05-15T13:19:27.830 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7231000000000006e-05 2024-05-15T13:19:27.830 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.830 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.830 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:19:27.830 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.981584+0000", 2024-05-15T13:19:27.830 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8769000000000003e-05 2024-05-15T13:19:27.830 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.830 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.830 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:27.830 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.982669+0000", 2024-05-15T13:19:27.830 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010852100000000001 2024-05-15T13:19:27.830 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.830 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.830 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.831 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.982968+0000", 2024-05-15T13:19:27.831 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000298938 2024-05-15T13:19:27.831 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.831 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.831 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.831 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.177225+0000", 2024-05-15T13:19:27.831 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.194256869 2024-05-15T13:19:27.831 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.831 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.831 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:27.831 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.177260+0000", 2024-05-15T13:19:27.831 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5284999999999998e-05 2024-05-15T13:19:27.831 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.832 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.832 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:27.832 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.177300+0000", 2024-05-15T13:19:27.832 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9833000000000003e-05 2024-05-15T13:19:27.832 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.832 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:27.832 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.832 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.832 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.832 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:3304 7.e 7:76095dc0:::benchmark_data_smithi177_134484_object3303:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-05-15T13:19:27.832 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:12:43.981909+0000", 2024-05-15T13:19:27.832 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 403.801233895, 2024-05-15T13:19:27.832 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.195526596, 2024-05-15T13:19:27.832 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:27.833 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:27.833 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:27.833 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:19:27.833 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:19:27.833 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 3304 2024-05-15T13:19:27.833 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.833 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:27.833 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.833 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:27.833 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.981909+0000", 2024-05-15T13:19:27.833 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.833 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.833 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.833 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:27.834 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.981909+0000", 2024-05-15T13:19:27.834 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.834 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.834 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.834 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:27.834 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.981910+0000", 2024-05-15T13:19:27.834 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6700000000000002e-07 2024-05-15T13:19:27.834 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.834 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.834 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:27.834 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.981937+0000", 2024-05-15T13:19:27.834 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6414000000000002e-05 2024-05-15T13:19:27.834 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.834 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.835 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:27.835 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.981948+0000", 2024-05-15T13:19:27.835 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0767e-05 2024-05-15T13:19:27.835 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.835 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.835 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:27.835 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.981957+0000", 2024-05-15T13:19:27.835 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1220000000000001e-06 2024-05-15T13:19:27.835 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.835 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.835 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.835 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.982027+0000", 2024-05-15T13:19:27.835 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0456000000000004e-05 2024-05-15T13:19:27.835 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.836 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.836 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:27.836 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.982157+0000", 2024-05-15T13:19:27.836 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013003900000000001 2024-05-15T13:19:27.836 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.836 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.836 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:19:27.836 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.982293+0000", 2024-05-15T13:19:27.836 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013626600000000001 2024-05-15T13:19:27.836 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.836 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.836 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:27.836 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.982981+0000", 2024-05-15T13:19:27.836 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00068772199999999999 2024-05-15T13:19:27.836 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.837 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.837 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.837 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.983142+0000", 2024-05-15T13:19:27.837 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016077999999999999 2024-05-15T13:19:27.837 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.837 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.837 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.837 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.177346+0000", 2024-05-15T13:19:27.837 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.194204241 2024-05-15T13:19:27.837 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.837 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.837 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:27.837 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.177397+0000", 2024-05-15T13:19:27.838 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1099e-05 2024-05-15T13:19:27.838 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.838 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.838 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:27.838 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.177436+0000", 2024-05-15T13:19:27.838 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8822999999999997e-05 2024-05-15T13:19:27.838 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.838 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:27.838 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.838 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.838 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.838 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:3326 7.e 7:720ef385:::benchmark_data_smithi177_134484_object3325:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-05-15T13:19:27.838 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:12:43.989811+0000", 2024-05-15T13:19:27.839 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 403.79333274099997, 2024-05-15T13:19:27.839 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.18775583000000001, 2024-05-15T13:19:27.839 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:27.839 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:27.839 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:27.839 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:19:27.839 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:19:27.839 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 3326 2024-05-15T13:19:27.839 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.839 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:27.839 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.839 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:27.839 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.989811+0000", 2024-05-15T13:19:27.839 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.839 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.840 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.840 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:27.840 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.989811+0000", 2024-05-15T13:19:27.840 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.840 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.840 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.840 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:27.840 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.989812+0000", 2024-05-15T13:19:27.840 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5e-06 2024-05-15T13:19:27.840 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.840 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.840 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:27.840 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.989853+0000", 2024-05-15T13:19:27.840 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1321e-05 2024-05-15T13:19:27.840 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.841 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.841 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:27.841 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.989869+0000", 2024-05-15T13:19:27.841 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5226e-05 2024-05-15T13:19:27.841 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.841 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.841 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:27.841 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.989877+0000", 2024-05-15T13:19:27.841 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5779999999999999e-06 2024-05-15T13:19:27.841 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.841 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.841 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.841 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.989946+0000", 2024-05-15T13:19:27.841 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8591999999999997e-05 2024-05-15T13:19:27.841 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.842 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.842 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:27.842 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.990089+0000", 2024-05-15T13:19:27.842 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014301900000000001 2024-05-15T13:19:27.842 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.842 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.842 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:19:27.842 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.990256+0000", 2024-05-15T13:19:27.842 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016755200000000001 2024-05-15T13:19:27.842 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.842 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.842 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:27.842 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.990734+0000", 2024-05-15T13:19:27.842 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00047789399999999998 2024-05-15T13:19:27.843 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.843 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.843 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.843 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:43.991350+0000", 2024-05-15T13:19:27.843 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00061603500000000002 2024-05-15T13:19:27.843 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.843 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.843 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.843 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.177479+0000", 2024-05-15T13:19:27.843 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.18612863299999999 2024-05-15T13:19:27.843 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.843 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.843 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:27.844 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.177530+0000", 2024-05-15T13:19:27.844 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0748000000000003e-05 2024-05-15T13:19:27.844 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.844 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.844 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:27.844 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.177566+0000", 2024-05-15T13:19:27.844 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6732e-05 2024-05-15T13:19:27.844 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.844 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:27.844 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.844 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.844 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.844 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:3356 7.e 7:72381255:::benchmark_data_smithi177_134484_object3355:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-05-15T13:19:27.844 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:12:44.003226+0000", 2024-05-15T13:19:27.845 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 403.77991725300001, 2024-05-15T13:19:27.845 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.17443794400000001, 2024-05-15T13:19:27.845 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:27.845 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:27.845 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:27.845 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:19:27.845 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:19:27.845 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 3356 2024-05-15T13:19:27.845 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.845 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:27.845 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.845 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:27.845 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.003226+0000", 2024-05-15T13:19:27.845 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.845 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.846 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.846 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:27.846 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.003226+0000", 2024-05-15T13:19:27.846 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.846 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.846 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.846 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:27.846 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.003228+0000", 2024-05-15T13:19:27.846 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4300000000000001e-06 2024-05-15T13:19:27.846 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.846 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.846 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:27.846 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.003290+0000", 2024-05-15T13:19:27.847 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2707999999999997e-05 2024-05-15T13:19:27.847 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.847 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.847 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:27.847 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.003303+0000", 2024-05-15T13:19:27.847 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2787e-05 2024-05-15T13:19:27.847 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.847 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.847 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:27.847 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.003309+0000", 2024-05-15T13:19:27.847 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3829999999999997e-06 2024-05-15T13:19:27.847 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.847 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.847 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.847 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.003363+0000", 2024-05-15T13:19:27.848 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3128999999999999e-05 2024-05-15T13:19:27.848 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.848 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.848 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:27.848 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.003448+0000", 2024-05-15T13:19:27.848 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5072000000000001e-05 2024-05-15T13:19:27.848 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.848 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.848 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:19:27.848 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.003538+0000", 2024-05-15T13:19:27.848 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0093999999999995e-05 2024-05-15T13:19:27.848 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.848 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.848 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:27.849 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.003983+0000", 2024-05-15T13:19:27.849 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00044510600000000002 2024-05-15T13:19:27.849 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.849 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.849 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.849 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.004425+0000", 2024-05-15T13:19:27.849 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00044196099999999999 2024-05-15T13:19:27.849 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.849 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.849 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.849 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.177608+0000", 2024-05-15T13:19:27.849 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.17318303600000001 2024-05-15T13:19:27.849 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.849 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.849 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:27.850 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.177642+0000", 2024-05-15T13:19:27.850 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4301000000000002e-05 2024-05-15T13:19:27.850 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.850 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.850 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:27.850 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.177664+0000", 2024-05-15T13:19:27.850 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1937e-05 2024-05-15T13:19:27.850 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.850 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:27.850 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.850 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.850 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.850 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:3366 7.e 7:7ef13b65:::benchmark_data_smithi177_134484_object3365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-05-15T13:19:27.850 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:12:44.016932+0000", 2024-05-15T13:19:27.851 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 403.76621096000002, 2024-05-15T13:19:27.851 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16082327299999999, 2024-05-15T13:19:27.851 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:27.851 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:27.851 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:27.851 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:19:27.851 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:19:27.851 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 3366 2024-05-15T13:19:27.851 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.851 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:27.851 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.851 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:27.851 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.016932+0000", 2024-05-15T13:19:27.851 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.852 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.852 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.852 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:27.852 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.016932+0000", 2024-05-15T13:19:27.852 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.852 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.852 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.852 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:27.852 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.016934+0000", 2024-05-15T13:19:27.852 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3519999999999999e-06 2024-05-15T13:19:27.852 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.852 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.852 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:27.852 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.017015+0000", 2024-05-15T13:19:27.852 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1211000000000006e-05 2024-05-15T13:19:27.853 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.853 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.853 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:27.853 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.017030+0000", 2024-05-15T13:19:27.853 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5274e-05 2024-05-15T13:19:27.853 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.853 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.853 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:27.853 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.017039+0000", 2024-05-15T13:19:27.853 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2050000000000008e-06 2024-05-15T13:19:27.853 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.853 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.853 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.853 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.017103+0000", 2024-05-15T13:19:27.854 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3415000000000002e-05 2024-05-15T13:19:27.854 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.854 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.854 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:27.854 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.017253+0000", 2024-05-15T13:19:27.854 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014978800000000001 2024-05-15T13:19:27.854 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.854 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.854 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:19:27.854 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.017417+0000", 2024-05-15T13:19:27.854 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016484299999999999 2024-05-15T13:19:27.854 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.854 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.854 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:27.855 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.017894+0000", 2024-05-15T13:19:27.855 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00047674099999999998 2024-05-15T13:19:27.855 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.855 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.855 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.855 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.018182+0000", 2024-05-15T13:19:27.855 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00028727599999999999 2024-05-15T13:19:27.855 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.855 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.855 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.855 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.177687+0000", 2024-05-15T13:19:27.855 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.159505229 2024-05-15T13:19:27.855 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.856 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.856 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:27.856 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.177728+0000", 2024-05-15T13:19:27.856 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1359999999999997e-05 2024-05-15T13:19:27.856 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.856 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.856 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:27.856 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:44.177756+0000", 2024-05-15T13:19:27.856 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7579000000000001e-05 2024-05-15T13:19:27.856 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.856 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:27.856 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.856 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.856 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.857 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:104914 7.8 7:1a081bbc:::benchmark_data_smithi177_134484_object104913:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e643)", 2024-05-15T13:19:27.857 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:14:01.857648+0000", 2024-05-15T13:19:27.857 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 325.92549492400002, 2024-05-15T13:19:27.857 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.214819595, 2024-05-15T13:19:27.857 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:27.857 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:27.857 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:27.857 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:19:27.857 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:19:27.857 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 104914 2024-05-15T13:19:27.857 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.857 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:27.857 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.857 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:27.858 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:01.857648+0000", 2024-05-15T13:19:27.858 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.858 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.858 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.858 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:27.858 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:01.857648+0000", 2024-05-15T13:19:27.858 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.858 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.858 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.858 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:27.858 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:01.857650+0000", 2024-05-15T13:19:27.858 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.083e-06 2024-05-15T13:19:27.858 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.859 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.859 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:27.859 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:01.857705+0000", 2024-05-15T13:19:27.859 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5495000000000003e-05 2024-05-15T13:19:27.859 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.859 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.859 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:27.859 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:01.857720+0000", 2024-05-15T13:19:27.859 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4643e-05 2024-05-15T13:19:27.859 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.859 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.859 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:27.859 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:01.857727+0000", 2024-05-15T13:19:27.859 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2690000000000002e-06 2024-05-15T13:19:27.859 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.860 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.860 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.860 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:01.857797+0000", 2024-05-15T13:19:27.860 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0357000000000004e-05 2024-05-15T13:19:27.860 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.860 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.860 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:27.860 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:01.857894+0000", 2024-05-15T13:19:27.860 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6570999999999994e-05 2024-05-15T13:19:27.860 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.860 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.860 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,10", 2024-05-15T13:19:27.860 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:01.857986+0000", 2024-05-15T13:19:27.860 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1961999999999998e-05 2024-05-15T13:19:27.860 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.861 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.861 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:27.861 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:01.860632+0000", 2024-05-15T13:19:27.861 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0026463210000000001 2024-05-15T13:19:27.861 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.861 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.861 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.861 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:01.860869+0000", 2024-05-15T13:19:27.861 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00023695300000000001 2024-05-15T13:19:27.861 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.861 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.861 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.861 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:02.072385+0000", 2024-05-15T13:19:27.861 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.21151613799999999 2024-05-15T13:19:27.861 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.861 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.862 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:27.862 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:02.072427+0000", 2024-05-15T13:19:27.862 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1677000000000001e-05 2024-05-15T13:19:27.862 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.862 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.862 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:27.862 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:02.072468+0000", 2024-05-15T13:19:27.862 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1125999999999999e-05 2024-05-15T13:19:27.862 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.862 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:27.862 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.862 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.862 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.862 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:123162 7.8 7:1982b225:::benchmark_data_smithi177_134484_object14097:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e654)", 2024-05-15T13:19:27.863 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:14:44.823383+0000", 2024-05-15T13:19:27.863 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 282.95976067999999, 2024-05-15T13:19:27.863 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.153003316, 2024-05-15T13:19:27.863 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:27.863 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:27.863 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:27.863 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:19:27.863 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:19:27.863 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 123162 2024-05-15T13:19:27.863 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.863 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:27.863 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.863 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:27.863 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:44.823383+0000", 2024-05-15T13:19:27.863 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.864 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.864 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:27.864 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:44.823383+0000", 2024-05-15T13:19:27.864 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.864 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.864 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:27.864 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:44.823384+0000", 2024-05-15T13:19:27.864 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1349999999999999e-06 2024-05-15T13:19:27.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.864 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.864 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:27.864 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:44.823389+0000", 2024-05-15T13:19:27.865 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7079999999999996e-06 2024-05-15T13:19:27.865 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.865 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.865 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:27.865 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:44.823403+0000", 2024-05-15T13:19:27.865 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4861999999999999e-05 2024-05-15T13:19:27.865 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.865 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.865 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:27.865 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:44.823410+0000", 2024-05-15T13:19:27.865 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0040000000000002e-06 2024-05-15T13:19:27.865 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.865 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.865 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.866 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:44.838134+0000", 2024-05-15T13:19:27.866 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014723415 2024-05-15T13:19:27.866 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.866 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.866 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-05-15T13:19:27.866 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:44.838248+0000", 2024-05-15T13:19:27.866 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000113891 2024-05-15T13:19:27.866 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.866 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.866 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.866 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:44.912613+0000", 2024-05-15T13:19:27.866 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.074365224999999993 2024-05-15T13:19:27.866 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.867 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.867 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:27.867 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:44.912726+0000", 2024-05-15T13:19:27.867 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000112681 2024-05-15T13:19:27.867 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.867 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.867 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3,10", 2024-05-15T13:19:27.867 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:44.912877+0000", 2024-05-15T13:19:27.867 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015184000000000001 2024-05-15T13:19:27.867 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.867 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.867 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.867 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:44.936306+0000", 2024-05-15T13:19:27.867 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.02342878 2024-05-15T13:19:27.867 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.867 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.868 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:27.868 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:44.936396+0000", 2024-05-15T13:19:27.868 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9870999999999994e-05 2024-05-15T13:19:27.868 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.868 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.868 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.868 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:44.972497+0000", 2024-05-15T13:19:27.868 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.036100689999999998 2024-05-15T13:19:27.868 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.868 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.868 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:27.868 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:44.972556+0000", 2024-05-15T13:19:27.868 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9030999999999998e-05 2024-05-15T13:19:27.868 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.868 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.868 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:27.869 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:44.976386+0000", 2024-05-15T13:19:27.869 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003830183 2024-05-15T13:19:27.869 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.869 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:27.869 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.869 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.869 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.869 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:128959 7.e 7:761d5d4d:::benchmark_data_smithi177_134484_object19894:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e661)", 2024-05-15T13:19:27.869 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:14:59.753971+0000", 2024-05-15T13:19:27.869 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 268.029172776, 2024-05-15T13:19:27.869 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.22551051599999999, 2024-05-15T13:19:27.869 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:27.869 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:27.869 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:27.869 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:19:27.869 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:19:27.870 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 128959 2024-05-15T13:19:27.870 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.870 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:27.870 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.870 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:27.870 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:59.753971+0000", 2024-05-15T13:19:27.870 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.870 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.870 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.870 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:27.870 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:59.753971+0000", 2024-05-15T13:19:27.870 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.870 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.871 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.871 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:27.871 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:59.753972+0000", 2024-05-15T13:19:27.871 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.562e-06 2024-05-15T13:19:27.871 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.871 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.871 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:27.871 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:59.753979+0000", 2024-05-15T13:19:27.871 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.25e-06 2024-05-15T13:19:27.871 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.871 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.871 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:27.871 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:59.753995+0000", 2024-05-15T13:19:27.871 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5290000000000001e-05 2024-05-15T13:19:27.871 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.871 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.872 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:27.872 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:59.754004+0000", 2024-05-15T13:19:27.872 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7960000000000007e-06 2024-05-15T13:19:27.872 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.872 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.872 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.872 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:59.754077+0000", 2024-05-15T13:19:27.872 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2352000000000001e-05 2024-05-15T13:19:27.872 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.872 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.872 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:27.872 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:59.754244+0000", 2024-05-15T13:19:27.872 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016679499999999999 2024-05-15T13:19:27.872 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.872 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.872 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 9,10", 2024-05-15T13:19:27.873 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:59.754378+0000", 2024-05-15T13:19:27.873 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000134447 2024-05-15T13:19:27.873 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.873 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.873 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.873 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:59.758335+0000", 2024-05-15T13:19:27.873 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039573669999999998 2024-05-15T13:19:27.873 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.873 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.873 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:27.873 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:59.772934+0000", 2024-05-15T13:19:27.873 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014598336 2024-05-15T13:19:27.873 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.873 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.873 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:27.873 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:59.979372+0000", 2024-05-15T13:19:27.874 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.206438181 2024-05-15T13:19:27.874 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.874 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.874 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:27.874 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:59.979433+0000", 2024-05-15T13:19:27.874 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0915000000000002e-05 2024-05-15T13:19:27.874 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.874 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.874 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:27.874 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:14:59.979481+0000", 2024-05-15T13:19:27.874 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8225e-05 2024-05-15T13:19:27.874 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.874 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:27.874 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.874 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.875 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.875 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.26044.0:146924 7.2 e679/599 7:488bd821:::benchmark_data_smithi177_134484_object37859:head v 679'12714, mlcod=678'12712)", 2024-05-15T13:19:27.875 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:15:43.640719+0000", 2024-05-15T13:19:27.875 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 224.142424154, 2024-05-15T13:19:27.875 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.17082966099999999, 2024-05-15T13:19:27.875 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:27.875 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:27.875 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:27.875 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.875 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:27.875 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:43.640719+0000", 2024-05-15T13:19:27.875 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.875 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.875 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.875 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:27.875 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:43.640719+0000", 2024-05-15T13:19:27.876 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.876 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.876 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.876 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:27.876 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:43.640720+0000", 2024-05-15T13:19:27.876 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0139999999999999e-06 2024-05-15T13:19:27.876 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.876 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.876 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:27.876 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:43.640730+0000", 2024-05-15T13:19:27.876 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9410000000000002e-06 2024-05-15T13:19:27.876 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.876 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.876 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:27.876 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:43.640747+0000", 2024-05-15T13:19:27.877 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6433999999999999e-05 2024-05-15T13:19:27.877 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.877 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.877 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:27.877 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:43.640755+0000", 2024-05-15T13:19:27.877 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7059999999999993e-06 2024-05-15T13:19:27.877 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.877 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.877 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.877 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:43.640854+0000", 2024-05-15T13:19:27.877 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9011000000000005e-05 2024-05-15T13:19:27.877 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.877 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.877 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:27.877 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:43.640878+0000", 2024-05-15T13:19:27.877 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.332e-05 2024-05-15T13:19:27.878 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.878 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.878 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:27.878 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:43.811504+0000", 2024-05-15T13:19:27.878 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.170626202 2024-05-15T13:19:27.878 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.878 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.878 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:27.878 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:43.811549+0000", 2024-05-15T13:19:27.878 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5033000000000001e-05 2024-05-15T13:19:27.878 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.878 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:27.878 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.879 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.879 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.879 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.26044.0:146929 7.2 e679/599 7:4d13d340:::benchmark_data_smithi177_134484_object37864:head v 679'12715, mlcod=678'12712)", 2024-05-15T13:19:27.879 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:15:43.661455+0000", 2024-05-15T13:19:27.879 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 224.12168829699999, 2024-05-15T13:19:27.879 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.15012383600000001, 2024-05-15T13:19:27.879 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:27.879 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:27.879 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:27.879 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.879 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:27.879 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:43.661455+0000", 2024-05-15T13:19:27.879 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.879 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.879 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.879 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:27.880 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:43.661455+0000", 2024-05-15T13:19:27.880 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.880 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.880 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.880 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:27.880 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:43.661457+0000", 2024-05-15T13:19:27.880 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7850000000000001e-06 2024-05-15T13:19:27.880 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.880 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.880 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:27.880 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:43.661468+0000", 2024-05-15T13:19:27.880 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1547e-05 2024-05-15T13:19:27.880 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.880 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.880 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:27.880 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:43.661490+0000", 2024-05-15T13:19:27.881 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1381000000000001e-05 2024-05-15T13:19:27.881 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.881 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.881 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:27.881 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:43.661499+0000", 2024-05-15T13:19:27.881 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6080000000000002e-06 2024-05-15T13:19:27.881 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.881 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.881 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.881 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:43.661599+0000", 2024-05-15T13:19:27.881 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9159000000000004e-05 2024-05-15T13:19:27.881 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.881 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.882 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:27.882 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:43.661622+0000", 2024-05-15T13:19:27.882 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3269999999999999e-05 2024-05-15T13:19:27.882 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.882 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.882 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:27.882 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:43.811552+0000", 2024-05-15T13:19:27.882 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.14992976799999999 2024-05-15T13:19:27.882 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.882 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.882 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:27.882 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:15:43.811579+0000", 2024-05-15T13:19:27.882 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7317999999999998e-05 2024-05-15T13:19:27.882 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.883 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:27.883 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.883 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.883 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.883 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.26299.0:4273 8.d e753/739 8:ba63f2d3:::benchmark_data_smithi177_149673_object4272:head v 753'262, mlcod=752'260)", 2024-05-15T13:19:27.883 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:19:25.193836+0000", 2024-05-15T13:19:27.883 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 2.5893078430000003, 2024-05-15T13:19:27.883 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014695613999999999, 2024-05-15T13:19:27.883 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:27.883 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:27.883 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:27.883 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.883 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:27.883 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.193836+0000", 2024-05-15T13:19:27.884 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.884 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.884 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.884 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:27.884 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.193836+0000", 2024-05-15T13:19:27.884 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.884 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.884 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.884 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:27.884 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.193836+0000", 2024-05-15T13:19:27.884 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0900000000000001e-07 2024-05-15T13:19:27.884 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.884 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.884 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:27.884 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.193862+0000", 2024-05-15T13:19:27.884 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5401999999999998e-05 2024-05-15T13:19:27.885 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.885 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.885 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:27.885 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.193875+0000", 2024-05-15T13:19:27.885 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3431999999999999e-05 2024-05-15T13:19:27.885 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.885 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.885 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:27.885 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.193882+0000", 2024-05-15T13:19:27.885 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1339999999999998e-06 2024-05-15T13:19:27.885 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.885 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.885 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.885 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.193939+0000", 2024-05-15T13:19:27.885 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6388000000000003e-05 2024-05-15T13:19:27.886 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.886 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.886 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:27.886 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.193968+0000", 2024-05-15T13:19:27.886 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9471999999999998e-05 2024-05-15T13:19:27.886 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.886 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.886 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:27.886 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.208494+0000", 2024-05-15T13:19:27.886 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014526061999999999 2024-05-15T13:19:27.886 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.886 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.886 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:27.887 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.208531+0000", 2024-05-15T13:19:27.887 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7014999999999997e-05 2024-05-15T13:19:27.887 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.887 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:27.887 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.887 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.887 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.887 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.26299.0:4283 8.1 e753/739 8:828fc4e9:::benchmark_data_smithi177_149673_object4282:head v 753'288, mlcod=752'286)", 2024-05-15T13:19:27.887 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:19:25.195725+0000", 2024-05-15T13:19:27.887 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 2.587418735, 2024-05-15T13:19:27.887 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012542731999999999, 2024-05-15T13:19:27.887 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:27.887 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:27.887 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:27.887 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.888 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:27.888 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.195725+0000", 2024-05-15T13:19:27.888 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.888 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.888 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.888 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:27.888 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.195725+0000", 2024-05-15T13:19:27.888 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.888 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.888 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.888 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:27.888 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.195726+0000", 2024-05-15T13:19:27.888 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9700000000000005e-07 2024-05-15T13:19:27.888 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.889 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.889 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:27.889 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.195749+0000", 2024-05-15T13:19:27.889 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3810000000000001e-05 2024-05-15T13:19:27.889 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.889 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.889 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:27.889 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.195762+0000", 2024-05-15T13:19:27.889 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2522e-05 2024-05-15T13:19:27.889 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.889 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.889 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:27.889 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.195768+0000", 2024-05-15T13:19:27.889 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9850000000000003e-06 2024-05-15T13:19:27.890 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.890 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.890 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.890 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.198384+0000", 2024-05-15T13:19:27.890 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0026165580000000002 2024-05-15T13:19:27.890 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.890 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.890 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:27.890 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.198399+0000", 2024-05-15T13:19:27.890 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4859e-05 2024-05-15T13:19:27.890 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.890 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.890 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:27.890 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.208219+0000", 2024-05-15T13:19:27.891 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0098199529999999993 2024-05-15T13:19:27.891 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.891 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.891 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:27.891 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.208267+0000", 2024-05-15T13:19:27.891 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8148e-05 2024-05-15T13:19:27.891 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.891 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:27.891 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.891 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.891 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.891 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.26299.0:4284 8.d e753/739 8:b99acced:::benchmark_data_smithi177_149673_object4283:head v 753'263, mlcod=752'260)", 2024-05-15T13:19:27.891 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:19:25.195818+0000", 2024-05-15T13:19:27.891 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 2.587325168, 2024-05-15T13:19:27.892 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012731357, 2024-05-15T13:19:27.892 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:27.892 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:27.892 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:27.892 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.892 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:27.892 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.195818+0000", 2024-05-15T13:19:27.892 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.892 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.892 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.892 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:27.892 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.195818+0000", 2024-05-15T13:19:27.892 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.892 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.893 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.893 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:27.893 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.195819+0000", 2024-05-15T13:19:27.893 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6000000000000003e-07 2024-05-15T13:19:27.893 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.893 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.893 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:27.893 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.195847+0000", 2024-05-15T13:19:27.893 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7668e-05 2024-05-15T13:19:27.893 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.893 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.893 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:27.893 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.195857+0000", 2024-05-15T13:19:27.893 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0538e-05 2024-05-15T13:19:27.894 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.894 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.894 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:27.894 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.195864+0000", 2024-05-15T13:19:27.894 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6429999999999998e-06 2024-05-15T13:19:27.894 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.894 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.894 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.894 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.195934+0000", 2024-05-15T13:19:27.894 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9746000000000006e-05 2024-05-15T13:19:27.894 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.894 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.894 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:27.894 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.195952+0000", 2024-05-15T13:19:27.895 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8729999999999999e-05 2024-05-15T13:19:27.895 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.895 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.895 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:27.895 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.208533+0000", 2024-05-15T13:19:27.895 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012581083 2024-05-15T13:19:27.895 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.895 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.895 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:27.895 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.208550+0000", 2024-05-15T13:19:27.895 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6189e-05 2024-05-15T13:19:27.895 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.895 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:27.895 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.896 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.896 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.896 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.26299.0:4297 8.6 e753/739 8:690901c1:::benchmark_data_smithi177_149673_object4296:head v 753'288, mlcod=752'286)", 2024-05-15T13:19:27.896 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:19:25.200375+0000", 2024-05-15T13:19:27.896 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 2.5827685499999999, 2024-05-15T13:19:27.896 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0095285649999999993, 2024-05-15T13:19:27.896 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:27.896 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:27.896 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:27.896 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.896 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:27.896 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.200375+0000", 2024-05-15T13:19:27.896 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.896 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.897 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.897 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:27.897 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.200375+0000", 2024-05-15T13:19:27.897 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:27.897 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.897 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.897 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:27.897 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.200376+0000", 2024-05-15T13:19:27.897 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4300000000000002e-07 2024-05-15T13:19:27.897 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.897 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.897 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:27.897 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.200406+0000", 2024-05-15T13:19:27.898 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0266000000000001e-05 2024-05-15T13:19:27.898 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.898 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.898 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:27.898 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.200415+0000", 2024-05-15T13:19:27.898 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0860000000000008e-06 2024-05-15T13:19:27.898 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.898 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.898 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:27.898 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.200421+0000", 2024-05-15T13:19:27.898 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5139999999999998e-06 2024-05-15T13:19:27.898 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.898 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.898 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:27.899 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.208179+0000", 2024-05-15T13:19:27.899 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0077580039999999998 2024-05-15T13:19:27.899 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.899 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.899 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:27.899 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.208194+0000", 2024-05-15T13:19:27.899 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4615e-05 2024-05-15T13:19:27.899 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.899 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.899 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:27.899 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.209872+0000", 2024-05-15T13:19:27.899 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001678407 2024-05-15T13:19:27.899 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:27.899 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:27.900 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:27.900 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:19:25.209903+0000", 2024-05-15T13:19:27.900 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0929999999999997e-05 2024-05-15T13:19:27.900 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.900 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:27.900 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.900 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:27.900 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:27.900 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:19:27.985 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 19 16 8113 8097 26.6308 0 - 0.0348958 2024-05-15T13:19:28.604 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:28.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:28.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:28.626 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-05-15T13:19:28.633 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:19:28.710 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:19:28.742 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:28.764 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:19:28.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:28.768 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:19:28.869 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:19:28.898 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:28.920 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:19:28.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:28.923 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:19:28.985 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T13:19:28.988624+0000 min lat: 0.00105016 max lat: 15.0039 avg lat: 0.0348958 2024-05-15T13:19:28.985 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:19:28.985 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 20 16 8113 8097 25.2993 0 - 0.0348958 2024-05-15T13:19:29.024 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:19:29.052 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:29.071 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:19:29.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:29.075 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:19:29.176 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:19:29.205 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:29.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:29 smithi027 ceph-mon[39795]: pgmap v1947: 37 pgs: 37 active+clean; 360 MiB data, 9.0 GiB used, 885 GiB / 894 GiB avail; 52 MiB/s wr, 837 op/s 2024-05-15T13:19:29.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:29 smithi027 ceph-mon[39795]: osdmap e754: 12 total, 12 up, 10 in 2024-05-15T13:19:29.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:29 smithi177 ceph-mon[44303]: pgmap v1947: 37 pgs: 37 active+clean; 360 MiB data, 9.0 GiB used, 885 GiB / 894 GiB avail; 52 MiB/s wr, 837 op/s 2024-05-15T13:19:29.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:29 smithi177 ceph-mon[44303]: osdmap e754: 12 total, 12 up, 10 in 2024-05-15T13:19:29.985 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 21 16 8113 8097 24.0946 0 - 0.0348958 2024-05-15T13:19:30.031 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:19:30.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:29 smithi120 ceph-mon[43961]: pgmap v1947: 37 pgs: 37 active+clean; 360 MiB data, 9.0 GiB used, 885 GiB / 894 GiB avail; 52 MiB/s wr, 837 op/s 2024-05-15T13:19:30.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:29 smithi120 ceph-mon[43961]: osdmap e754: 12 total, 12 up, 10 in 2024-05-15T13:19:30.051 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:19:30.054 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:19:30.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:30.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:19:30.071 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:19:30.091 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:19:30.112 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:19:30.132 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:19:30.153 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:19:30.159 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:19:30.174 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:19:30.191 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:30.195 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:19:30.214 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:19:30.216 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:19:30.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:30.217 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:19:30.237 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:19:30.257 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:19:30.259 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:19:30.259 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:19:30.259 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:19:30.259 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:19:30.259 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:19:30.259 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:19:30.259 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:19:30.259 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:19:30.259 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:19:30.260 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:19:30.260 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:19:30.260 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:19:30.318 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:19:30.350 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:30.371 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:19:30.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:30.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:19:30.475 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:19:30.507 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:30.529 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:19:30.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:30.533 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:19:30.633 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:19:30.665 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:30.717 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:19:30.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:30.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:19:30.735 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:19:30.822 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:19:30.854 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:30.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:30.902 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:19:30.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:19:30.922 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:30 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1506869652' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:19:30.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:30 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1506869652' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:19:30.985 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 22 16 8113 8097 22.9994 0 - 0.0348958 2024-05-15T13:19:31.003 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:19:31.034 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:31.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:30 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1506869652' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:19:31.057 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:19:31.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:31.060 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:19:31.161 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:19:31.191 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:31.212 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:19:31.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:31.216 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:19:31.317 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:31.347 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:31.472 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:31.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:31.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:31.576 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:19:31.609 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:31.630 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:19:31.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:31.633 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:19:31.734 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:19:31.767 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:31.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:31 smithi027 ceph-mon[39795]: pgmap v1949: 37 pgs: 37 active+clean; 497 MiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-05-15T13:19:31.788 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:19:31.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:31.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:19:31.892 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:31.921 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:31.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:31 smithi177 ceph-mon[44303]: pgmap v1949: 37 pgs: 37 active+clean; 497 MiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-05-15T13:19:31.985 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 23 16 8113 8097 21.9994 0 - 0.0348958 2024-05-15T13:19:31.994 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:31.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:31.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:32.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:31 smithi120 ceph-mon[43961]: pgmap v1949: 37 pgs: 37 active+clean; 497 MiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-05-15T13:19:32.098 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:19:32.131 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:32.152 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:19:32.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:32.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:19:32.257 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:19:32.289 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:32.311 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:19:32.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:32.315 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:19:32.416 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:19:32.448 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:32.470 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:19:32.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:32.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:19:32.574 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:19:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:32.728 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:19:32.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:32.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:19:32.788 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:32 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:19:32.788 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:32 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:19:32.832 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:19:32.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:32 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:19:32.859 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:32 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:19:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:32.923 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 8, 9, 10, 11] out_osds: [7, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:19:32.923 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:19:32.924 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-05-15T13:19:32.924 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd in 7 2024-05-15T13:19:32.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:32 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:19:32.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:32 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:19:32.986 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 24 16 8113 8097 21.0828 0 - 0.0348958 2024-05-15T13:19:33.062 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:19:33.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:33.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:19:33.153 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:19:33.153 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:19:33.166 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:33.193 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:33.342 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:33.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:33.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:33.447 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:19:33.475 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:33.576 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:19:33.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:33.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:19:33.682 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:19:33.714 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:33.736 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:19:33.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:33.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:19:33.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:33 smithi027 ceph-mon[39795]: pgmap v1950: 37 pgs: 37 active+clean; 507 MiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 57 MiB/s wr, 915 op/s 2024-05-15T13:19:33.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:33 smithi027 ceph-mon[39795]: osdmap e755: 12 total, 12 up, 10 in 2024-05-15T13:19:33.840 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:19:33.872 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:33.894 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:19:33.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:33.897 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:19:33.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:33 smithi177 ceph-mon[44303]: pgmap v1950: 37 pgs: 37 active+clean; 507 MiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 57 MiB/s wr, 915 op/s 2024-05-15T13:19:33.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:33 smithi177 ceph-mon[44303]: osdmap e755: 12 total, 12 up, 10 in 2024-05-15T13:19:33.986 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 25 16 8113 8097 20.2395 0 - 0.0348958 2024-05-15T13:19:33.998 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:19:34.028 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:34.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:33 smithi120 ceph-mon[43961]: pgmap v1950: 37 pgs: 37 active+clean; 507 MiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 57 MiB/s wr, 915 op/s 2024-05-15T13:19:34.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:33 smithi120 ceph-mon[43961]: osdmap e755: 12 total, 12 up, 10 in 2024-05-15T13:19:34.049 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:19:34.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:34.054 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:19:34.073 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-05-15T13:19:34.155 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:34.185 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:34.193 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:19:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:34.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:34.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:34.277 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:19:34.314 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:19:34.346 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:34.367 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:19:34.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:34.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:19:34.417 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:19:34.470 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:19:34.499 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:34.552 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:19:34.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:34.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:19:34.658 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:19:34.690 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:34.711 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:19:34.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:34.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:19:34.811 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:34 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/197822484' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:19:34.811 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:34 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:19:34.811 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:34 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/197822484' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:19:34.811 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:34 smithi027 ceph-mon[39795]: osdmap e756: 12 total, 12 up, 10 in 2024-05-15T13:19:34.815 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:19:34.845 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:34.865 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:19:34.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:34.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:19:34.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:34 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/197822484' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:19:34.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:34 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:19:34.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:34 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/197822484' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:19:34.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:34 smithi177 ceph-mon[44303]: osdmap e756: 12 total, 12 up, 10 in 2024-05-15T13:19:34.969 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:19:34.986 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 26 16 8113 8097 19.461 0 - 0.0348958 2024-05-15T13:19:34.997 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:35.019 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:19:35.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:35.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:19:35.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:34 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/197822484' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:19:35.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:34 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:19:35.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:34 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/197822484' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:19:35.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:34 smithi120 ceph-mon[43961]: osdmap e756: 12 total, 12 up, 10 in 2024-05-15T13:19:35.123 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:19:35.156 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:35.177 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:19:35.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:35.181 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:19:35.282 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:19:35.315 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:35.336 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:19:35.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:35.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:19:35.440 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:19:35.469 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:35.489 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:19:35.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:35.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:19:35.493 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:19:35.593 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:19:35.623 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:35.885 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:35 smithi027 ceph-mon[39795]: pgmap v1953: 37 pgs: 37 active+clean; 508 MiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 25 MiB/s wr, 394 op/s 2024-05-15T13:19:35.885 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:19:35.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:35 smithi177 ceph-mon[44303]: pgmap v1953: 37 pgs: 37 active+clean; 508 MiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 25 MiB/s wr, 394 op/s 2024-05-15T13:19:35.986 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 27 16 8113 8097 18.7402 0 - 0.0348958 2024-05-15T13:19:36.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:35 smithi120 ceph-mon[43961]: pgmap v1953: 37 pgs: 37 active+clean; 508 MiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 25 MiB/s wr, 394 op/s 2024-05-15T13:19:36.473 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:19:36.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:36.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:19:36.577 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:19:36.610 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:36.631 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:19:36.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:36.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:19:36.735 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:19:36.764 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:36.784 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:19:36.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:36.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:19:36.813 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:19:36.888 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:19:36.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:36 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:19:36.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:36 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:19:36.920 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:36.941 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:19:36.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:36.945 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:19:36.986 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 28 16 8113 8097 18.071 0 - 0.0348958 2024-05-15T13:19:37.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:36 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:19:37.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:36 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:19:37.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:36 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:19:37.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:36 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:19:37.046 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:19:37.076 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:37.096 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:19:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:37.100 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:19:37.201 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:19:37.229 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:37.248 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:19:37.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:37.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:19:37.353 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:19:37.379 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:37.399 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:19:37.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:37.402 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:19:37.503 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:19:37.534 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:37.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:37 smithi177 ceph-mon[44303]: pgmap v1954: 37 pgs: 37 active+clean; 508 MiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 8.6 MiB/s wr, 136 op/s 2024-05-15T13:19:37.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:37 smithi177 ceph-mon[44303]: osdmap e757: 12 total, 12 up, 10 in 2024-05-15T13:19:37.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:37 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2706999032' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-05-15T13:19:37.986 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 29 16 8113 8097 17.4479 0 - 0.0348958 2024-05-15T13:19:38.011 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:37 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-a[39772]: 2024-05-15T13:19:37.661+0000 7f8daffd2700 -1 mon.a@0(leader).osd e757 definitely_dead 0 2024-05-15T13:19:38.012 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:37 smithi027 ceph-mon[39795]: pgmap v1954: 37 pgs: 37 active+clean; 508 MiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 8.6 MiB/s wr, 136 op/s 2024-05-15T13:19:38.012 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:37 smithi027 ceph-mon[39795]: osdmap e757: 12 total, 12 up, 10 in 2024-05-15T13:19:38.012 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:37 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2706999032' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-05-15T13:19:38.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:37 smithi120 ceph-mon[43961]: pgmap v1954: 37 pgs: 37 active+clean; 508 MiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 8.6 MiB/s wr, 136 op/s 2024-05-15T13:19:38.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:37 smithi120 ceph-mon[43961]: osdmap e757: 12 total, 12 up, 10 in 2024-05-15T13:19:38.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:37 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2706999032' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-05-15T13:19:38.864 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:38 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:19:38.865 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:38 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:19:38.865 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:38 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2706999032' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-05-15T13:19:38.865 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:38 smithi027 ceph-mon[39795]: osdmap e758: 12 total, 12 up, 11 in 2024-05-15T13:19:38.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:38 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:19:38.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:38 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:19:38.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:38 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2706999032' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-05-15T13:19:38.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:38 smithi177 ceph-mon[44303]: osdmap e758: 12 total, 12 up, 11 in 2024-05-15T13:19:38.975 INFO:teuthology.orchestra.run.smithi027.stderr:marked in osd.7. 2024-05-15T13:19:38.986 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 30 16 8113 8097 16.8663 0 - 0.0348958 2024-05-15T13:19:39.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:38 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:19:39.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:38 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:19:39.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:38 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2706999032' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-05-15T13:19:39.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:38 smithi120 ceph-mon[43961]: osdmap e758: 12 total, 12 up, 11 in 2024-05-15T13:19:39.284 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:19:39.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:39.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:19:39.389 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:19:39.420 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:39.441 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:19:39.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:39.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:19:39.502 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:19:39.502 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:19:39.503 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:19:39.503 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:19:39.517 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:19:39.541 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:19:39.545 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:39.566 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:19:39.578 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:39.593 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:19:39.614 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:19:39.636 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:19:39.657 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:19:39.677 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:19:39.678 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:19:39.678 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:19:39.679 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:19:39.679 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:19:39.679 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:19:39.679 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:19:39.679 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:19:39.679 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:19:39.679 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:19:39.679 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:19:39.679 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:19:39.679 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:19:39.832 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:39 smithi027 ceph-mon[39795]: pgmap v1957: 37 pgs: 3 peering, 34 active+clean; 508 MiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 173 KiB/s wr, 2 op/s 2024-05-15T13:19:39.832 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:39 smithi027 ceph-mon[39795]: osdmap e759: 12 total, 12 up, 11 in 2024-05-15T13:19:39.832 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:39 smithi027 ceph-mon[39795]: osdmap e760: 12 total, 12 up, 11 in 2024-05-15T13:19:39.832 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:39 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3615378841' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:19:39.833 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-05-15T13:19:39.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:39 smithi177 ceph-mon[44303]: pgmap v1957: 37 pgs: 3 peering, 34 active+clean; 508 MiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 173 KiB/s wr, 2 op/s 2024-05-15T13:19:39.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:39 smithi177 ceph-mon[44303]: osdmap e759: 12 total, 12 up, 11 in 2024-05-15T13:19:39.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:39 smithi177 ceph-mon[44303]: osdmap e760: 12 total, 12 up, 11 in 2024-05-15T13:19:39.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:39 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3615378841' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:19:39.987 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 31 16 8113 8097 16.3222 0 - 0.0348958 2024-05-15T13:19:40.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:39 smithi120 ceph-mon[43961]: pgmap v1957: 37 pgs: 3 peering, 34 active+clean; 508 MiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 173 KiB/s wr, 2 op/s 2024-05-15T13:19:40.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:39 smithi120 ceph-mon[43961]: osdmap e759: 12 total, 12 up, 11 in 2024-05-15T13:19:40.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:39 smithi120 ceph-mon[43961]: osdmap e760: 12 total, 12 up, 11 in 2024-05-15T13:19:40.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:39 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3615378841' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:19:40.835 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:19:40.883 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:19:40.884 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T13:19:40.884 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-05-15T13:19:40.884 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-05-15T13:19:40.884 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:19:40.987 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 32 16 8113 8097 15.8122 0 - 0.0348958 2024-05-15T13:19:41.328 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:41.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:41.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:41.432 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:19:41.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:41 smithi177 ceph-mon[44303]: osdmap e761: 12 total, 12 up, 11 in 2024-05-15T13:19:41.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:41 smithi177 ceph-mon[44303]: pgmap v1961: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 4 peering, 31 active+clean; 508 MiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 483/24345 objects misplaced (1.984%); 245 KiB/s, 3 objects/s recovering 2024-05-15T13:19:41.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:41 smithi027 ceph-mon[39795]: osdmap e761: 12 total, 12 up, 11 in 2024-05-15T13:19:41.450 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:41 smithi027 ceph-mon[39795]: pgmap v1961: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 4 peering, 31 active+clean; 508 MiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 483/24345 objects misplaced (1.984%); 245 KiB/s, 3 objects/s recovering 2024-05-15T13:19:41.461 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:41.480 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:19:41.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:41.484 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:19:41.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:41 smithi120 ceph-mon[43961]: osdmap e761: 12 total, 12 up, 11 in 2024-05-15T13:19:41.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:41 smithi120 ceph-mon[43961]: pgmap v1961: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 4 peering, 31 active+clean; 508 MiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 483/24345 objects misplaced (1.984%); 245 KiB/s, 3 objects/s recovering 2024-05-15T13:19:41.584 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:19:41.614 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:41.636 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:19:41.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:41.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:19:41.740 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:19:41.771 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:41.791 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:19:41.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:41.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:19:41.896 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:19:41.925 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:41.945 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:19:41.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:41.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:19:41.987 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 33 16 8191 8175 15.4807 0.325 0.00154478 0.063915 2024-05-15T13:19:42.049 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:19:42.077 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:42.098 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:19:42.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:42.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:19:42.202 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:19:42.244 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:42.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:42 smithi177 ceph-mon[44303]: osdmap e762: 12 total, 12 up, 11 in 2024-05-15T13:19:42.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:42 smithi027 ceph-mon[39795]: osdmap e762: 12 total, 12 up, 11 in 2024-05-15T13:19:42.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:42 smithi120 ceph-mon[43961]: osdmap e762: 12 total, 12 up, 11 in 2024-05-15T13:19:42.566 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:19:42.909 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-05-15T13:19:42.987 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 34 16 10353 10337 18.9991 135.125 0.00684058 0.0526274 2024-05-15T13:19:43.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:43 smithi027 ceph-mon[39795]: osdmap e763: 12 total, 12 up, 11 in 2024-05-15T13:19:43.195 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:43 smithi027 ceph-mon[39795]: pgmap v1964: 37 pgs: 1 active+recovering+undersized+remapped, 8 peering, 28 active+clean; 508 MiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 3.8 KiB/s rd, 135 KiB/s wr, 10 op/s; 483/24351 objects misplaced (1.983%); 372 KiB/s, 4 objects/s recovering 2024-05-15T13:19:43.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:43 smithi120 ceph-mon[43961]: osdmap e763: 12 total, 12 up, 11 in 2024-05-15T13:19:43.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:43 smithi120 ceph-mon[43961]: pgmap v1964: 37 pgs: 1 active+recovering+undersized+remapped, 8 peering, 28 active+clean; 508 MiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 3.8 KiB/s rd, 135 KiB/s wr, 10 op/s; 483/24351 objects misplaced (1.983%); 372 KiB/s, 4 objects/s recovering 2024-05-15T13:19:43.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:43 smithi177 ceph-mon[44303]: osdmap e763: 12 total, 12 up, 11 in 2024-05-15T13:19:43.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:43 smithi177 ceph-mon[44303]: pgmap v1964: 37 pgs: 1 active+recovering+undersized+remapped, 8 peering, 28 active+clean; 508 MiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 3.8 KiB/s rd, 135 KiB/s wr, 10 op/s; 483/24351 objects misplaced (1.983%); 372 KiB/s, 4 objects/s recovering 2024-05-15T13:19:43.987 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 35 16 13274 13258 23.6717 182.562 0.00126614 0.0421274 2024-05-15T13:19:44.834 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 8, 9, 10, 11, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:19:44.834 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:19:44.835 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-05-15T13:19:44.835 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd dump -f json-pretty 2024-05-15T13:19:44.988 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 36 16 15967 15951 27.6883 168.312 0.00539223 0.0361113 2024-05-15T13:19:45.169 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:45 smithi027 ceph-mon[39795]: osdmap e764: 12 total, 12 up, 11 in 2024-05-15T13:19:45.170 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:45 smithi027 ceph-mon[39795]: pgmap v1966: 37 pgs: 4 peering, 33 active+clean; 645 MiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 3.4 KiB/s rd, 31 MiB/s wr, 505 op/s; 22 MiB/s, 348 objects/s recovering 2024-05-15T13:19:45.214 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:19:45.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:45.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:19:45.320 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:19:45.331 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:19:45.355 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:45.387 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:19:45.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:45.391 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:19:45.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:45 smithi177 ceph-mon[44303]: osdmap e764: 12 total, 12 up, 11 in 2024-05-15T13:19:45.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:45 smithi177 ceph-mon[44303]: pgmap v1966: 37 pgs: 4 peering, 33 active+clean; 645 MiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 3.4 KiB/s rd, 31 MiB/s wr, 505 op/s; 22 MiB/s, 348 objects/s recovering 2024-05-15T13:19:45.492 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:19:45.529 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:45.538 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:45 smithi120 ceph-mon[43961]: osdmap e764: 12 total, 12 up, 11 in 2024-05-15T13:19:45.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:45 smithi120 ceph-mon[43961]: pgmap v1966: 37 pgs: 4 peering, 33 active+clean; 645 MiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 3.4 KiB/s rd, 31 MiB/s wr, 505 op/s; 22 MiB/s, 348 objects/s recovering 2024-05-15T13:19:45.562 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:19:45.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:45.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:19:45.667 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:19:45.712 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:45.755 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:19:45.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:45.759 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:19:45.860 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:19:45.910 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:45.988 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 37 16 18485 18469 31.1927 157.375 0.00592249 0.0320524 2024-05-15T13:19:46.044 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:19:46.049 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:19:46.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:46.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:19:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:19:46.155 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:46.200 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:46.417 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:19:46.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:46 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/23123933' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:19:46.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:46 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:19:46.445 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:46 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:19:46.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:46 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/23123933' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:19:46.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:46 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:19:46.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:46 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:19:46.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:46 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/23123933' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:19:46.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:46 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:19:46.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:46 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:19:46.988 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 38 16 20943 20927 34.4139 153.625 0.00190246 0.0290036 2024-05-15T13:19:47.264 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:47.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:47.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:47.376 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:19:47.409 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:47.433 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:19:47.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:47.438 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:19:47.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:47 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:19:47.529 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:47 smithi027 ceph-mon[39795]: osdmap e765: 12 total, 12 up, 11 in 2024-05-15T13:19:47.529 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:47 smithi027 ceph-mon[39795]: pgmap v1968: 37 pgs: 37 active+clean; 746 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 44 MiB/s wr, 705 op/s; 18 MiB/s, 283 objects/s recovering 2024-05-15T13:19:47.538 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:19:47.567 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:47.590 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:19:47.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:47.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:19:47.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:47 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:19:47.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:47 smithi177 ceph-mon[44303]: osdmap e765: 12 total, 12 up, 11 in 2024-05-15T13:19:47.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:47 smithi177 ceph-mon[44303]: pgmap v1968: 37 pgs: 37 active+clean; 746 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 44 MiB/s wr, 705 op/s; 18 MiB/s, 283 objects/s recovering 2024-05-15T13:19:47.695 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:19:47.755 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:47.788 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:47 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:19:47.788 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:47 smithi120 ceph-mon[43961]: osdmap e765: 12 total, 12 up, 11 in 2024-05-15T13:19:47.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:47 smithi120 ceph-mon[43961]: pgmap v1968: 37 pgs: 37 active+clean; 746 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 44 MiB/s wr, 705 op/s; 18 MiB/s, 283 objects/s recovering 2024-05-15T13:19:47.991 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 39 15 23033 23018 36.882 130.688 0.00512643 0.0271088 2024-05-15T13:19:48.014 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:19:48.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:48.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:19:48.121 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:19:48.152 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:48.252 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:19:48.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:48.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:19:48.356 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:19:48.398 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:48.477 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:19:48.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:48.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:19:48.584 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:19:48.630 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:48.677 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:19:48.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:48 smithi027 ceph-mon[39795]: 1.e scrub starts 2024-05-15T13:19:48.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:48 smithi027 ceph-mon[39795]: 1.11 scrub starts 2024-05-15T13:19:48.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:48 smithi027 ceph-mon[39795]: 1.e scrub ok 2024-05-15T13:19:48.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:48 smithi027 ceph-mon[39795]: 1.11 scrub ok 2024-05-15T13:19:48.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:48.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:19:48.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:48 smithi177 ceph-mon[44303]: 1.e scrub starts 2024-05-15T13:19:48.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:48 smithi177 ceph-mon[44303]: 1.11 scrub starts 2024-05-15T13:19:48.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:48 smithi177 ceph-mon[44303]: 1.e scrub ok 2024-05-15T13:19:48.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:48 smithi177 ceph-mon[44303]: 1.11 scrub ok 2024-05-15T13:19:48.784 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:19:48.788 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:48 smithi120 ceph-mon[43961]: 1.e scrub starts 2024-05-15T13:19:48.788 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:48 smithi120 ceph-mon[43961]: 1.11 scrub starts 2024-05-15T13:19:48.788 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:48 smithi120 ceph-mon[43961]: 1.e scrub ok 2024-05-15T13:19:48.788 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:48 smithi120 ceph-mon[43961]: 1.11 scrub ok 2024-05-15T13:19:48.813 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:48.835 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:19:48.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:48.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:19:48.939 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:19:48.983 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:48.990 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T13:19:48.993476+0000 min lat: 0.00105016 max lat: 15.0048 avg lat: 0.0256559 2024-05-15T13:19:48.990 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:19:48.990 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 40 16 24959 24943 38.9658 120.312 0.00729783 0.0256559 2024-05-15T13:19:49.015 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:19:49.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:49.020 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:19:49.121 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:19:49.166 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:49.427 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:49 smithi177 ceph-mon[44303]: 1.9 scrub starts 2024-05-15T13:19:49.632 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:19:49.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:49.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:19:49.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:49 smithi177 ceph-mon[44303]: 1.9 scrub ok 2024-05-15T13:19:49.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:49 smithi177 ceph-mon[44303]: pgmap v1969: 37 pgs: 37 active+clean; 1.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 107 MiB/s wr, 1.71k op/s; 15 MiB/s, 239 objects/s recovering 2024-05-15T13:19:49.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:49 smithi177 ceph-mon[44303]: osdmap e766: 12 total, 12 up, 11 in 2024-05-15T13:19:49.737 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:19:49.763 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:49 smithi120 ceph-mon[43961]: 1.9 scrub starts 2024-05-15T13:19:49.763 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:49 smithi120 ceph-mon[43961]: 1.9 scrub ok 2024-05-15T13:19:49.763 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:49 smithi120 ceph-mon[43961]: pgmap v1969: 37 pgs: 37 active+clean; 1.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 107 MiB/s wr, 1.71k op/s; 15 MiB/s, 239 objects/s recovering 2024-05-15T13:19:49.763 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:49 smithi120 ceph-mon[43961]: osdmap e766: 12 total, 12 up, 11 in 2024-05-15T13:19:49.765 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:49.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:49 smithi027 ceph-mon[39795]: 1.9 scrub starts 2024-05-15T13:19:49.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:49 smithi027 ceph-mon[39795]: 1.9 scrub ok 2024-05-15T13:19:49.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:49 smithi027 ceph-mon[39795]: pgmap v1969: 37 pgs: 37 active+clean; 1.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 107 MiB/s wr, 1.71k op/s; 15 MiB/s, 239 objects/s recovering 2024-05-15T13:19:49.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:49 smithi027 ceph-mon[39795]: osdmap e766: 12 total, 12 up, 11 in 2024-05-15T13:19:49.786 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:19:49.790 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:19:49.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:49.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:19:49.892 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:19:49.940 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:49.990 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 41 16 26817 26801 40.8472 116.125 0.0599077 0.0244656 2024-05-15T13:19:50.690 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:19:50.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:50.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:19:50.794 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:50.847 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:50.990 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 42 16 28724 28708 42.7119 119.188 0.00876861 0.0234064 2024-05-15T13:19:51.686 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:51.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:51.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:51.782 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:19:51.782 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:19:51.782 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 766, 2024-05-15T13:19:51.782 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "deaea04e-12b7-11ef-bc98-c7b262605968", 2024-05-15T13:19:51.782 INFO:teuthology.orchestra.run.smithi027.stdout: "created": "2024-05-15T12:38:21.244556+0000", 2024-05-15T13:19:51.782 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-05-15T13:19:49.169222+0000", 2024-05-15T13:19:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "last_up_change": "2024-05-15T12:45:50.876090+0000", 2024-05-15T13:19:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "last_in_change": "2024-05-15T13:19:37.666910+0000", 2024-05-15T13:19:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-15T13:19:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_num": 5804032, 2024-05-15T13:19:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_set": [ 2024-05-15T13:19:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "nodeep-scrub", 2024-05-15T13:19:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "pglog_hardlimit", 2024-05-15T13:19:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "purged_snapdirs", 2024-05-15T13:19:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_deletes", 2024-05-15T13:19:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "sortbitwise" 2024-05-15T13:19:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:19:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_version": 26, 2024-05-15T13:19:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "full_ratio": 0.94999998807907104, 2024-05-15T13:19:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-05-15T13:19:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-15T13:19:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_snapshot": "", 2024-05-15T13:19:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_max": 8, 2024-05-15T13:19:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "max_osd": 12, 2024-05-15T13:19:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "require_min_compat_client": "luminous", 2024-05-15T13:19:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "min_compat_client": "luminous", 2024-05-15T13:19:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "require_osd_release": "squid", 2024-05-15T13:19:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_crimson": false, 2024-05-15T13:19:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "pools": [ 2024-05-15T13:19:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 1, 2024-05-15T13:19:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": ".mgr", 2024-05-15T13:19:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "create_time": "2024-05-15T12:41:57.948613+0000", 2024-05-15T13:19:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-15T13:19:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_names": "hashpspool", 2024-05-15T13:19:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "type": 1, 2024-05-15T13:19:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 3, 2024-05-15T13:19:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": 2, 2024-05-15T13:19:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_rule": 0, 2024-05-15T13:19:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_count": 0, 2024-05-15T13:19:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_target": 0, 2024-05-15T13:19:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T13:19:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T13:19:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "object_hash": 2, 2024-05-15T13:19:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscale_mode": "off", 2024-05-15T13:19:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": 21, 2024-05-15T13:19:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num": 21, 2024-05-15T13:19:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num_target": 21, 2024-05-15T13:19:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_target": 21, 2024-05-15T13:19:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_pending": 21, 2024-05-15T13:19:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "last_pg_merge_meta": { 2024-05-15T13:19:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "source_pgid": "0.0", 2024-05-15T13:19:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "ready_epoch": 0, 2024-05-15T13:19:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_started": 0, 2024-05-15T13:19:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_clean": 0, 2024-05-15T13:19:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "source_version": "0'0", 2024-05-15T13:19:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "target_version": "0'0" 2024-05-15T13:19:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "last_change": "753", 2024-05-15T13:19:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend": "0", 2024-05-15T13:19:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T13:19:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_preluminous": "614", 2024-05-15T13:19:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "auid": 0, 2024-05-15T13:19:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_mode": "selfmanaged", 2024-05-15T13:19:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_seq": 0, 2024-05-15T13:19:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_epoch": 0, 2024-05-15T13:19:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_snaps": [], 2024-05-15T13:19:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps": "[]", 2024-05-15T13:19:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_bytes": 0, 2024-05-15T13:19:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_objects": 0, 2024-05-15T13:19:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "tiers": [], 2024-05-15T13:19:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "tier_of": -1, 2024-05-15T13:19:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "read_tier": -1, 2024-05-15T13:19:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "write_tier": -1, 2024-05-15T13:19:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_mode": "none", 2024-05-15T13:19:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_bytes": 0, 2024-05-15T13:19:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_objects": 0, 2024-05-15T13:19:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T13:19:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T13:19:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T13:19:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_flush_age": 0, 2024-05-15T13:19:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_evict_age": 0, 2024-05-15T13:19:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profile": "", 2024-05-15T13:19:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_params": { 2024-05-15T13:19:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "none" 2024-05-15T13:19:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_period": 0, 2024-05-15T13:19:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_count": 0, 2024-05-15T13:19:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "use_gmt_hitset": true, 2024-05-15T13:19:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "min_read_recency_for_promote": 0, 2024-05-15T13:19:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "min_write_recency_for_promote": 0, 2024-05-15T13:19:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T13:19:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_search_last_n": 0, 2024-05-15T13:19:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "grade_table": [], 2024-05-15T13:19:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_width": 0, 2024-05-15T13:19:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "expected_num_objects": 0, 2024-05-15T13:19:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "fast_read": false, 2024-05-15T13:19:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "options": { 2024-05-15T13:19:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_max": 32, 2024-05-15T13:19:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_min": 1 2024-05-15T13:19:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "application_metadata": { 2024-05-15T13:19:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": {} 2024-05-15T13:19:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: "read_balance": { 2024-05-15T13:19:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: "score_type": "Fair distribution", 2024-05-15T13:19:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: "score_acting": 2.2799999713897705, 2024-05-15T13:19:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: "score_stable": 2.2799999713897705, 2024-05-15T13:19:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: "optimal_score": 0.92000001668930054, 2024-05-15T13:19:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_acting": 2.0999999046325684, 2024-05-15T13:19:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_stable": 2.0999999046325684, 2024-05-15T13:19:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity_weighted": 0.94652295112609863, 2024-05-15T13:19:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity": 0.94999998807907104, 2024-05-15T13:19:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-05-15T13:19:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 8, 2024-05-15T13:19:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": "unique_pool_6", 2024-05-15T13:19:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "create_time": "2024-05-15T13:18:50.685449+0000", 2024-05-15T13:19:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-15T13:19:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_names": "hashpspool", 2024-05-15T13:19:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "type": 1, 2024-05-15T13:19:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 3, 2024-05-15T13:19:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": 2, 2024-05-15T13:19:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_rule": 0, 2024-05-15T13:19:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_count": 0, 2024-05-15T13:19:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_target": 0, 2024-05-15T13:19:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T13:19:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T13:19:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "object_hash": 2, 2024-05-15T13:19:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscale_mode": "off", 2024-05-15T13:19:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": 16, 2024-05-15T13:19:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num": 16, 2024-05-15T13:19:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num_target": 16, 2024-05-15T13:19:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_target": 16, 2024-05-15T13:19:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_pending": 16, 2024-05-15T13:19:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "last_pg_merge_meta": { 2024-05-15T13:19:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "source_pgid": "0.0", 2024-05-15T13:19:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "ready_epoch": 0, 2024-05-15T13:19:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_started": 0, 2024-05-15T13:19:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_clean": 0, 2024-05-15T13:19:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "source_version": "0'0", 2024-05-15T13:19:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "target_version": "0'0" 2024-05-15T13:19:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "last_change": "745", 2024-05-15T13:19:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend": "0", 2024-05-15T13:19:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T13:19:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T13:19:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "auid": 0, 2024-05-15T13:19:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_mode": "selfmanaged", 2024-05-15T13:19:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_seq": 0, 2024-05-15T13:19:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_epoch": 0, 2024-05-15T13:19:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_snaps": [], 2024-05-15T13:19:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps": "[]", 2024-05-15T13:19:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_bytes": 0, 2024-05-15T13:19:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_objects": 0, 2024-05-15T13:19:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "tiers": [], 2024-05-15T13:19:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "tier_of": -1, 2024-05-15T13:19:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "read_tier": -1, 2024-05-15T13:19:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "write_tier": -1, 2024-05-15T13:19:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_mode": "none", 2024-05-15T13:19:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_bytes": 0, 2024-05-15T13:19:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_objects": 0, 2024-05-15T13:19:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T13:19:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T13:19:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T13:19:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_flush_age": 0, 2024-05-15T13:19:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_evict_age": 0, 2024-05-15T13:19:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profile": "", 2024-05-15T13:19:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_params": { 2024-05-15T13:19:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "none" 2024-05-15T13:19:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_period": 0, 2024-05-15T13:19:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_count": 0, 2024-05-15T13:19:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "use_gmt_hitset": true, 2024-05-15T13:19:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "min_read_recency_for_promote": 0, 2024-05-15T13:19:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "min_write_recency_for_promote": 0, 2024-05-15T13:19:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T13:19:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_search_last_n": 0, 2024-05-15T13:19:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "grade_table": [], 2024-05-15T13:19:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_width": 0, 2024-05-15T13:19:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "expected_num_objects": 0, 2024-05-15T13:19:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "fast_read": false, 2024-05-15T13:19:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "options": {}, 2024-05-15T13:19:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "application_metadata": { 2024-05-15T13:19:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "rados": {} 2024-05-15T13:19:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: "read_balance": { 2024-05-15T13:19:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: "score_type": "Fair distribution", 2024-05-15T13:19:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: "score_acting": 3.0099999904632568, 2024-05-15T13:19:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: "score_stable": 3.0099999904632568, 2024-05-15T13:19:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: "optimal_score": 0.82999998331069946, 2024-05-15T13:19:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_acting": 2.5, 2024-05-15T13:19:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_stable": 2.5, 2024-05-15T13:19:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity_weighted": 0.94652295112609863, 2024-05-15T13:19:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity": 0.94999998807907104, 2024-05-15T13:19:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-05-15T13:19:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:19:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T13:19:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 0, 2024-05-15T13:19:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "64fb3ad2-08a7-4106-a07a-6c776c877faf", 2024-05-15T13:19:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:19:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:19:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:19:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:19:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:19:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:19:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 8, 2024-05-15T13:19:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 762, 2024-05-15T13:19:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:19:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:19:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:19:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6802", 2024-05-15T13:19:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:19:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6803", 2024-05-15T13:19:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:19:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:19:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6804", 2024-05-15T13:19:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:19:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6805", 2024-05-15T13:19:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:19:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:19:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6808", 2024-05-15T13:19:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:19:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6809", 2024-05-15T13:19:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:19:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:19:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6806", 2024-05-15T13:19:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:19:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6807", 2024-05-15T13:19:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:19:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6803/4239116935", 2024-05-15T13:19:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6805/4239116935", 2024-05-15T13:19:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6809/4239116935", 2024-05-15T13:19:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6807/4239116935", 2024-05-15T13:19:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:19:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:19:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:19:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 1, 2024-05-15T13:19:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "9c16dd90-9d51-48b8-a296-1c8879ba1bc8", 2024-05-15T13:19:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:19:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:19:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:19:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:19:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:19:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:19:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 13, 2024-05-15T13:19:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 760, 2024-05-15T13:19:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:19:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:19:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:19:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6810", 2024-05-15T13:19:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:19:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6811", 2024-05-15T13:19:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:19:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:19:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6812", 2024-05-15T13:19:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:19:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6813", 2024-05-15T13:19:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:19:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:19:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6816", 2024-05-15T13:19:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:19:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6817", 2024-05-15T13:19:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:19:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:19:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6814", 2024-05-15T13:19:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:19:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6815", 2024-05-15T13:19:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:19:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6811/1466270637", 2024-05-15T13:19:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6813/1466270637", 2024-05-15T13:19:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6817/1466270637", 2024-05-15T13:19:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6815/1466270637", 2024-05-15T13:19:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:19:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:19:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:19:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 2, 2024-05-15T13:19:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "cbe3ba13-3ea3-4891-8d2d-6b85581a732c", 2024-05-15T13:19:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:19:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:19:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:19:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:19:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:19:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:19:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 18, 2024-05-15T13:19:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 761, 2024-05-15T13:19:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:19:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:19:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:19:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6818", 2024-05-15T13:19:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:19:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6819", 2024-05-15T13:19:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:19:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:19:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6820", 2024-05-15T13:19:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:19:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6821", 2024-05-15T13:19:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:19:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:19:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6824", 2024-05-15T13:19:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:19:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6825", 2024-05-15T13:19:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:19:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:19:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6822", 2024-05-15T13:19:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:19:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6823", 2024-05-15T13:19:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:19:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6819/3285413897", 2024-05-15T13:19:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6821/3285413897", 2024-05-15T13:19:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6825/3285413897", 2024-05-15T13:19:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6823/3285413897", 2024-05-15T13:19:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:19:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:19:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:19:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 3, 2024-05-15T13:19:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "2e83735b-b944-4f87-8080-b994d4b96897", 2024-05-15T13:19:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:19:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:19:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:19:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:19:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:19:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:19:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 26, 2024-05-15T13:19:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 739, 2024-05-15T13:19:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:19:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:19:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:19:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6826", 2024-05-15T13:19:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:19:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6827", 2024-05-15T13:19:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:19:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:19:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6828", 2024-05-15T13:19:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:19:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6829", 2024-05-15T13:19:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:19:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:19:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6832", 2024-05-15T13:19:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:19:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6833", 2024-05-15T13:19:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:19:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:19:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6830", 2024-05-15T13:19:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:19:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6831", 2024-05-15T13:19:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:19:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6827/3008975244", 2024-05-15T13:19:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6829/3008975244", 2024-05-15T13:19:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6833/3008975244", 2024-05-15T13:19:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6831/3008975244", 2024-05-15T13:19:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:19:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:19:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:19:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 4, 2024-05-15T13:19:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "7c39c1dd-3867-46f7-8372-7688fa13b16a", 2024-05-15T13:19:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:19:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 0, 2024-05-15T13:19:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 0, 2024-05-15T13:19:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:19:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:19:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:19:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 31, 2024-05-15T13:19:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 509, 2024-05-15T13:19:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:19:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:19:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:19:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6800", 2024-05-15T13:19:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:19:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6801", 2024-05-15T13:19:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:19:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:19:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6802", 2024-05-15T13:19:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:19:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6803", 2024-05-15T13:19:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:19:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:19:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6806", 2024-05-15T13:19:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:19:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6807", 2024-05-15T13:19:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:19:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:19:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6804", 2024-05-15T13:19:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:19:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6805", 2024-05-15T13:19:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:19:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6801/3877842957", 2024-05-15T13:19:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6803/3877842957", 2024-05-15T13:19:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6807/3877842957", 2024-05-15T13:19:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6805/3877842957", 2024-05-15T13:19:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:19:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:19:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:19:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 5, 2024-05-15T13:19:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "a6dc220f-aec0-4454-9695-b44c0b9770e0", 2024-05-15T13:19:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:19:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:19:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:19:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:19:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:19:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:19:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 40, 2024-05-15T13:19:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 761, 2024-05-15T13:19:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:19:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:19:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:19:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6808", 2024-05-15T13:19:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:19:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6809", 2024-05-15T13:19:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:19:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:19:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6810", 2024-05-15T13:19:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:19:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6811", 2024-05-15T13:19:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:19:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:19:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6814", 2024-05-15T13:19:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:19:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6815", 2024-05-15T13:19:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:19:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:19:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6812", 2024-05-15T13:19:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:19:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6813", 2024-05-15T13:19:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:19:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6809/1600179183", 2024-05-15T13:19:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6811/1600179183", 2024-05-15T13:19:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6815/1600179183", 2024-05-15T13:19:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6813/1600179183", 2024-05-15T13:19:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:19:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:19:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:19:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 6, 2024-05-15T13:19:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b", 2024-05-15T13:19:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:19:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:19:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:19:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:19:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:19:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:19:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 48, 2024-05-15T13:19:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 717, 2024-05-15T13:19:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:19:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:19:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:19:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6816", 2024-05-15T13:19:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:19:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6817", 2024-05-15T13:19:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:19:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:19:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6818", 2024-05-15T13:19:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:19:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6819", 2024-05-15T13:19:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:19:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:19:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6822", 2024-05-15T13:19:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:19:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6823", 2024-05-15T13:19:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:19:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:19:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6820", 2024-05-15T13:19:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:19:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6821", 2024-05-15T13:19:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:19:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6817/3323706735", 2024-05-15T13:19:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6819/3323706735", 2024-05-15T13:19:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6823/3323706735", 2024-05-15T13:19:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6821/3323706735", 2024-05-15T13:19:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:19:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:19:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:19:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 7, 2024-05-15T13:19:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "6ef085d8-f59b-4033-8280-2c8f78693e30", 2024-05-15T13:19:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:19:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:19:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:19:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:19:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:19:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:19:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 56, 2024-05-15T13:19:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 761, 2024-05-15T13:19:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:19:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:19:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:19:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6824", 2024-05-15T13:19:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:19:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6825", 2024-05-15T13:19:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:19:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:19:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6826", 2024-05-15T13:19:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:19:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6827", 2024-05-15T13:19:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:19:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:19:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6830", 2024-05-15T13:19:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:19:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6831", 2024-05-15T13:19:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:19:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:19:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6828", 2024-05-15T13:19:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:19:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6829", 2024-05-15T13:19:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:19:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6825/2960616153", 2024-05-15T13:19:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6827/2960616153", 2024-05-15T13:19:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6831/2960616153", 2024-05-15T13:19:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6829/2960616153", 2024-05-15T13:19:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:19:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:19:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:19:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 8, 2024-05-15T13:19:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "26514b11-d02b-46e9-a4e8-3da193fcdbcc", 2024-05-15T13:19:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:19:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:19:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:19:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:19:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:19:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:19:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 65, 2024-05-15T13:19:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 739, 2024-05-15T13:19:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:19:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:19:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:19:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6800", 2024-05-15T13:19:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:19:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6801", 2024-05-15T13:19:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:19:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:19:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6802", 2024-05-15T13:19:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:19:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6803", 2024-05-15T13:19:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:19:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:19:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6806", 2024-05-15T13:19:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:19:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6807", 2024-05-15T13:19:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:19:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:19:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6804", 2024-05-15T13:19:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:19:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6805", 2024-05-15T13:19:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:19:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6801/1132459315", 2024-05-15T13:19:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6803/1132459315", 2024-05-15T13:19:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6807/1132459315", 2024-05-15T13:19:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6805/1132459315", 2024-05-15T13:19:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:19:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:19:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:19:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 9, 2024-05-15T13:19:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "6f9497bc-e962-409d-8a2c-0c634cc1124f", 2024-05-15T13:19:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:19:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:19:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:19:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 0.9062042236328125, 2024-05-15T13:19:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:19:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:19:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 76, 2024-05-15T13:19:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 739, 2024-05-15T13:19:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:19:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:19:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:19:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6808", 2024-05-15T13:19:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:19:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6809", 2024-05-15T13:19:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:19:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:19:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6810", 2024-05-15T13:19:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:19:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6811", 2024-05-15T13:19:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:19:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:19:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6814", 2024-05-15T13:19:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:19:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6815", 2024-05-15T13:19:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:19:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:19:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6812", 2024-05-15T13:19:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:19:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6813", 2024-05-15T13:19:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:19:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6809/2710457646", 2024-05-15T13:19:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6811/2710457646", 2024-05-15T13:19:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6815/2710457646", 2024-05-15T13:19:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6813/2710457646", 2024-05-15T13:19:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:19:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:19:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:19:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 10, 2024-05-15T13:19:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "f852876f-c7d1-4039-90c4-5ab09dbd3e32", 2024-05-15T13:19:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:19:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:19:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:19:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 0.4520721435546875, 2024-05-15T13:19:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:19:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:19:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 81, 2024-05-15T13:19:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 739, 2024-05-15T13:19:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:19:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:19:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:19:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6816", 2024-05-15T13:19:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:19:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6817", 2024-05-15T13:19:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:19:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:19:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6818", 2024-05-15T13:19:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:19:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6819", 2024-05-15T13:19:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:19:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:19:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6822", 2024-05-15T13:19:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:19:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6823", 2024-05-15T13:19:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:19:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:19:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6820", 2024-05-15T13:19:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:19:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6821", 2024-05-15T13:19:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:19:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6817/3236900263", 2024-05-15T13:19:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6819/3236900263", 2024-05-15T13:19:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6823/3236900263", 2024-05-15T13:19:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6821/3236900263", 2024-05-15T13:19:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:19:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:19:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:19:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 11, 2024-05-15T13:19:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "cccc53c5-8476-4d55-b160-4bdc5cf92902", 2024-05-15T13:19:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:19:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:19:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:19:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:19:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:19:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:19:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 90, 2024-05-15T13:19:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 762, 2024-05-15T13:19:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:19:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:19:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:19:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6824", 2024-05-15T13:19:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:19:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6825", 2024-05-15T13:19:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:19:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:19:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6826", 2024-05-15T13:19:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:19:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6827", 2024-05-15T13:19:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:19:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:19:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6830", 2024-05-15T13:19:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:19:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6831", 2024-05-15T13:19:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:19:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:19:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:19:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:19:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6828", 2024-05-15T13:19:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:19:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:19:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6829", 2024-05-15T13:19:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:19:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6825/681011628", 2024-05-15T13:19:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6827/681011628", 2024-05-15T13:19:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6831/681011628", 2024-05-15T13:19:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6829/681011628", 2024-05-15T13:19:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:19:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:19:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:19:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:19:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_xinfo": [ 2024-05-15T13:19:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 0, 2024-05-15T13:19:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:19:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:19:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:19:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:19:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:19:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:19:37.561837+0000", 2024-05-15T13:19:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:19:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 1, 2024-05-15T13:19:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:19:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:19:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:19:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:19:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:19:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:19:36.547067+0000", 2024-05-15T13:19:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:19:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 2, 2024-05-15T13:19:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:19:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:19:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:19:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:19:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:19:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:19:01.527604+0000", 2024-05-15T13:19:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:19:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 3, 2024-05-15T13:19:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:19:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:19:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:19:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:19:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:19:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:19:10.780595+0000", 2024-05-15T13:19:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:19:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 4, 2024-05-15T13:19:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:19:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:19:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:19:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:19:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 65536, 2024-05-15T13:19:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:19:14.581288+0000", 2024-05-15T13:19:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:19:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 5, 2024-05-15T13:19:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:19:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:19:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:19:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:19:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:19:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:18:47.804489+0000", 2024-05-15T13:19:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:19:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 6, 2024-05-15T13:19:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:19:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:19:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:19:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:19:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:19:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:19:31.874314+0000", 2024-05-15T13:19:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:19:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 7, 2024-05-15T13:19:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:19:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:19:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:19:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:19:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:19:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:18:53.276860+0000", 2024-05-15T13:19:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:19:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 8, 2024-05-15T13:19:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:19:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:19:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:19:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:19:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:19:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:19:06.087986+0000", 2024-05-15T13:19:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:19:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 9, 2024-05-15T13:19:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:19:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:19:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:19:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:19:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:19:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:19:22.532588+0000", 2024-05-15T13:19:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:19:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 10, 2024-05-15T13:19:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:19:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:19:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:19:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:19:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:19:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:19:21.918876+0000", 2024-05-15T13:19:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:19:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 11, 2024-05-15T13:19:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:19:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:19:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:19:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:19:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:19:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:18:41.372754+0000", 2024-05-15T13:19:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:19:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:19:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap": [ 2024-05-15T13:19:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "pgid": "1.0", 2024-05-15T13:19:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T13:19:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-05-15T13:19:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: 6, 2024-05-15T13:19:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: 10 2024-05-15T13:19:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:19:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap_items": [], 2024-05-15T13:19:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap_primaries": [], 2024-05-15T13:19:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_temp": [], 2024-05-15T13:19:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_temp": [], 2024-05-15T13:19:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklist": { 2024-05-15T13:19:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6801/2530617473": "2024-05-16T12:39:19.409195+0000", 2024-05-15T13:19:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6800/2530617473": "2024-05-16T12:39:19.409195+0000", 2024-05-15T13:19:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/1880261204": "2024-05-16T12:39:19.409195+0000", 2024-05-15T13:19:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6801/3441136471": "2024-05-16T12:38:50.931011+0000", 2024-05-15T13:19:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/4291446247": "2024-05-16T12:39:19.409195+0000", 2024-05-15T13:19:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/580128209": "2024-05-16T12:38:50.931011+0000", 2024-05-15T13:19:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6800/3441136471": "2024-05-16T12:38:50.931011+0000", 2024-05-15T13:19:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/3011229857": "2024-05-16T12:38:50.931011+0000", 2024-05-15T13:19:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/2109693913": "2024-05-16T12:38:50.931011+0000", 2024-05-15T13:19:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/2445663024": "2024-05-16T12:39:19.409195+0000" 2024-05-15T13:19:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "range_blocklist": {}, 2024-05-15T13:19:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profiles": { 2024-05-15T13:19:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "default": { 2024-05-15T13:19:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "crush-failure-domain": "osd", 2024-05-15T13:19:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "k": "2", 2024-05-15T13:19:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "m": "1", 2024-05-15T13:19:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "plugin": "jerasure", 2024-05-15T13:19:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "technique": "reed_sol_van" 2024-05-15T13:19:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps_queue": [], 2024-05-15T13:19:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "new_removed_snaps": [], 2024-05-15T13:19:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "new_purged_snaps": [], 2024-05-15T13:19:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_node_flags": {}, 2024-05-15T13:19:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "device_class_flags": {}, 2024-05-15T13:19:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode": { 2024-05-15T13:19:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode_enabled": false, 2024-05-15T13:19:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_bucket_count": 0, 2024-05-15T13:19:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "degraded_stretch_mode": 0, 2024-05-15T13:19:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "recovering_stretch_mode": 0, 2024-05-15T13:19:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode_bucket": 0 2024-05-15T13:19:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:51.879 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:19:51.879 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:51.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:51 smithi027 ceph-mon[39795]: pgmap v1971: 37 pgs: 37 active+clean; 1.4 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 131 MiB/s wr, 2.10k op/s; 4.7 MiB/s, 75 objects/s recovering 2024-05-15T13:19:51.919 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:51.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:51 smithi177 ceph-mon[44303]: pgmap v1971: 37 pgs: 37 active+clean; 1.4 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 131 MiB/s wr, 2.10k op/s; 4.7 MiB/s, 75 objects/s recovering 2024-05-15T13:19:51.990 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 43 16 30875 30859 44.8446 134.438 0.0034253 0.0222944 2024-05-15T13:19:52.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:51 smithi120 ceph-mon[43961]: pgmap v1971: 37 pgs: 37 active+clean; 1.4 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 131 MiB/s wr, 2.10k op/s; 4.7 MiB/s, 75 objects/s recovering 2024-05-15T13:19:52.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:52 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2252679318' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T13:19:52.994 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 44 16 32754 32738 46.4903 117.438 0.00873762 0.0215044 2024-05-15T13:19:53.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:52 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2252679318' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T13:19:53.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:52 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2252679318' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T13:19:53.657 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:53 smithi027 ceph-mon[39795]: pgmap v1972: 37 pgs: 37 active+clean; 1.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-05-15T13:19:53.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:53 smithi177 ceph-mon[44303]: pgmap v1972: 37 pgs: 37 active+clean; 1.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-05-15T13:19:53.994 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 45 16 34830 34814 48.34 129.75 0.00739498 0.0206802 2024-05-15T13:19:54.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:53 smithi120 ceph-mon[43961]: pgmap v1972: 37 pgs: 37 active+clean; 1.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-05-15T13:19:54.996 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 46 16 36614 36598 49.7115 111.5 0.0110927 0.0201105 2024-05-15T13:19:55.379 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:55.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:55.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:55.418 INFO:tasks.thrashosds.thrasher:j is {'epoch': 766, 'fsid': 'deaea04e-12b7-11ef-bc98-c7b262605968', 'created': '2024-05-15T12:38:21.244556+0000', 'modified': '2024-05-15T13:19:49.169222+0000', 'last_up_change': '2024-05-15T12:45:50.876090+0000', 'last_in_change': '2024-05-15T13:19:37.666910+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-15T12:41:57.948613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '753', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '614', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2799999713897705, 'score_stable': 2.2799999713897705, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.0999999046325684, 'raw_score_stable': 2.0999999046325684, 'primary_affinity_weighted': 0.9465229511260986, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 8, 'pool_name': 'unique_pool_6', 'create_time': '2024-05-15T13:18:50.685449+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '745', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.009999990463257, 'score_stable': 3.009999990463257, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9465229511260986, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': '64fb3ad2-08a7-4106-a07a-6c776c877faf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 762, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6802', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6803', 'nonce': 4239116935}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6804', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6805', 'nonce': 4239116935}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6808', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6809', 'nonce': 4239116935}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6806', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6807', 'nonce': 4239116935}]}, 'public_addr': '172.21.15.27:6803/4239116935', 'cluster_addr': '172.21.15.27:6805/4239116935', 'heartbeat_back_addr': '172.21.15.27:6809/4239116935', 'heartbeat_front_addr': '172.21.15.27:6807/4239116935', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '9c16dd90-9d51-48b8-a296-1c8879ba1bc8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 760, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6810', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6811', 'nonce': 1466270637}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6812', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6813', 'nonce': 1466270637}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6816', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6817', 'nonce': 1466270637}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6814', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6815', 'nonce': 1466270637}]}, 'public_addr': '172.21.15.27:6811/1466270637', 'cluster_addr': '172.21.15.27:6813/1466270637', 'heartbeat_back_addr': '172.21.15.27:6817/1466270637', 'heartbeat_front_addr': '172.21.15.27:6815/1466270637', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'cbe3ba13-3ea3-4891-8d2d-6b85581a732c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 761, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6818', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6819', 'nonce': 3285413897}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6820', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6821', 'nonce': 3285413897}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6824', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6825', 'nonce': 3285413897}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6822', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6823', 'nonce': 3285413897}]}, 'public_addr': '172.21.15.27:6819/3285413897', 'cluster_addr': '172.21.15.27:6821/3285413897', 'heartbeat_back_addr': '172.21.15.27:6825/3285413897', 'heartbeat_front_addr': '172.21.15.27:6823/3285413897', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2e83735b-b944-4f87-8080-b994d4b96897', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 739, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6826', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6827', 'nonce': 3008975244}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6828', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6829', 'nonce': 3008975244}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6833', 'nonce': 3008975244}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6830', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6831', 'nonce': 3008975244}]}, 'public_addr': '172.21.15.27:6827/3008975244', 'cluster_addr': '172.21.15.27:6829/3008975244', 'heartbeat_back_addr': '172.21.15.27:6833/3008975244', 'heartbeat_front_addr': '172.21.15.27:6831/3008975244', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7c39c1dd-3867-46f7-8372-7688fa13b16a', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 509, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6800', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6801', 'nonce': 3877842957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6802', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6803', 'nonce': 3877842957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6806', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6807', 'nonce': 3877842957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6804', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6805', 'nonce': 3877842957}]}, 'public_addr': '172.21.15.120:6801/3877842957', 'cluster_addr': '172.21.15.120:6803/3877842957', 'heartbeat_back_addr': '172.21.15.120:6807/3877842957', 'heartbeat_front_addr': '172.21.15.120:6805/3877842957', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a6dc220f-aec0-4454-9695-b44c0b9770e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 761, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6808', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6809', 'nonce': 1600179183}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6810', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6811', 'nonce': 1600179183}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6814', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6815', 'nonce': 1600179183}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6812', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6813', 'nonce': 1600179183}]}, 'public_addr': '172.21.15.120:6809/1600179183', 'cluster_addr': '172.21.15.120:6811/1600179183', 'heartbeat_back_addr': '172.21.15.120:6815/1600179183', 'heartbeat_front_addr': '172.21.15.120:6813/1600179183', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 717, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6816', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6817', 'nonce': 3323706735}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6818', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6819', 'nonce': 3323706735}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6822', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6823', 'nonce': 3323706735}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6820', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6821', 'nonce': 3323706735}]}, 'public_addr': '172.21.15.120:6817/3323706735', 'cluster_addr': '172.21.15.120:6819/3323706735', 'heartbeat_back_addr': '172.21.15.120:6823/3323706735', 'heartbeat_front_addr': '172.21.15.120:6821/3323706735', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6ef085d8-f59b-4033-8280-2c8f78693e30', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 761, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6824', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6825', 'nonce': 2960616153}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6826', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6827', 'nonce': 2960616153}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6830', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6831', 'nonce': 2960616153}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6828', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6829', 'nonce': 2960616153}]}, 'public_addr': '172.21.15.120:6825/2960616153', 'cluster_addr': '172.21.15.120:6827/2960616153', 'heartbeat_back_addr': '172.21.15.120:6831/2960616153', 'heartbeat_front_addr': '172.21.15.120:6829/2960616153', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '26514b11-d02b-46e9-a4e8-3da193fcdbcc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 739, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6800', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6801', 'nonce': 1132459315}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6802', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6803', 'nonce': 1132459315}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6806', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6807', 'nonce': 1132459315}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6804', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6805', 'nonce': 1132459315}]}, 'public_addr': '172.21.15.177:6801/1132459315', 'cluster_addr': '172.21.15.177:6803/1132459315', 'heartbeat_back_addr': '172.21.15.177:6807/1132459315', 'heartbeat_front_addr': '172.21.15.177:6805/1132459315', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '6f9497bc-e962-409d-8a2c-0c634cc1124f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9062042236328125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 739, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6808', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6809', 'nonce': 2710457646}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6810', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6811', 'nonce': 2710457646}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6814', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6815', 'nonce': 2710457646}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6812', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6813', 'nonce': 2710457646}]}, 'public_addr': '172.21.15.177:6809/2710457646', 'cluster_addr': '172.21.15.177:6811/2710457646', 'heartbeat_back_addr': '172.21.15.177:6815/2710457646', 'heartbeat_front_addr': '172.21.15.177:6813/2710457646', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'f852876f-c7d1-4039-90c4-5ab09dbd3e32', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.4520721435546875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 81, 'up_thru': 739, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6816', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6817', 'nonce': 3236900263}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6818', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6819', 'nonce': 3236900263}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6822', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6823', 'nonce': 3236900263}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6820', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6821', 'nonce': 3236900263}]}, 'public_addr': '172.21.15.177:6817/3236900263', 'cluster_addr': '172.21.15.177:6819/3236900263', 'heartbeat_back_addr': '172.21.15.177:6823/3236900263', 'heartbeat_front_addr': '172.21.15.177:6821/3236900263', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'cccc53c5-8476-4d55-b160-4bdc5cf92902', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 90, 'up_thru': 762, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6824', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6825', 'nonce': 681011628}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6826', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6827', 'nonce': 681011628}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6830', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6831', 'nonce': 681011628}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6828', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6829', 'nonce': 681011628}]}, 'public_addr': '172.21.15.177:6825/681011628', 'cluster_addr': '172.21.15.177:6827/681011628', 'heartbeat_back_addr': '172.21.15.177:6831/681011628', 'heartbeat_front_addr': '172.21.15.177:6829/681011628', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:19:37.561837+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:19:36.547067+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:19:01.527604+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:19:10.780595+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-15T13:19:14.581288+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:18:47.804489+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:19:31.874314+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:18:53.276860+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:19:06.087986+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:19:22.532588+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:19:21.918876+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:18:41.372754+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [1, 6, 10]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.27:6801/2530617473': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:6800/2530617473': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:0/1880261204': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:6801/3441136471': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/4291446247': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:0/580128209': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:6800/3441136471': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/3011229857': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/2109693913': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/2445663024': '2024-05-16T12:39:19.409195+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-05-15T13:19:55.419 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:19:55.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:55 smithi177 ceph-mon[44303]: pgmap v1973: 37 pgs: 37 active+clean; 2.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 153 MiB/s wr, 2.44k op/s 2024-05-15T13:19:55.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:55 smithi177 ceph-mon[44303]: 1.14 scrub starts 2024-05-15T13:19:55.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:55 smithi177 ceph-mon[44303]: 1.14 scrub ok 2024-05-15T13:19:55.488 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:55.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:55 smithi027 ceph-mon[39795]: pgmap v1973: 37 pgs: 37 active+clean; 2.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 153 MiB/s wr, 2.44k op/s 2024-05-15T13:19:55.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:55 smithi027 ceph-mon[39795]: 1.14 scrub starts 2024-05-15T13:19:55.492 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:55 smithi027 ceph-mon[39795]: 1.14 scrub ok 2024-05-15T13:19:55.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:55 smithi120 ceph-mon[43961]: pgmap v1973: 37 pgs: 37 active+clean; 2.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 153 MiB/s wr, 2.44k op/s 2024-05-15T13:19:55.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:55 smithi120 ceph-mon[43961]: 1.14 scrub starts 2024-05-15T13:19:55.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:55 smithi120 ceph-mon[43961]: 1.14 scrub ok 2024-05-15T13:19:55.574 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:55.610 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:19:55.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:55.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:19:55.719 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:19:55.769 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:55.805 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:19:55.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:55.809 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:19:55.910 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:19:55.967 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:55.995 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:19:55.996 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 47 16 38343 38327 50.9515 108.062 0.00635315 0.0196201 2024-05-15T13:19:56.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:56.001 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:19:56.102 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:19:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:56.596 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:19:56.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:56.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:19:56.608 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:19:56.669 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:19:56.703 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:19:56.731 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:56.765 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:19:56.767 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:19:56.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:56.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:19:56.798 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:19:56.860 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:19:56.861 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T13:19:56.861 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T13:19:56.861 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:19:56.861 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.861 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:231812 6.6 6:6442f1b2:::benchmark_data_smithi177_122278_object101441:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-05-15T13:19:56.861 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:24.777935+0000", 2024-05-15T13:19:56.861 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 512.07938312099998, 2024-05-15T13:19:56.861 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.63502409599999998, 2024-05-15T13:19:56.861 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:56.861 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:56.861 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:56.861 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:19:56.862 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:19:56.862 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 231812 2024-05-15T13:19:56.862 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.862 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:56.862 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.862 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:56.862 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.777935+0000", 2024-05-15T13:19:56.862 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.862 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.862 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.862 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:56.862 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.777935+0000", 2024-05-15T13:19:56.862 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.863 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.863 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.863 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:56.863 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.777936+0000", 2024-05-15T13:19:56.863 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3200000000000001e-06 2024-05-15T13:19:56.863 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.863 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.863 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:56.863 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.777944+0000", 2024-05-15T13:19:56.863 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7270000000000007e-06 2024-05-15T13:19:56.863 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.863 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.863 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:56.863 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.777986+0000", 2024-05-15T13:19:56.863 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2321999999999999e-05 2024-05-15T13:19:56.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.864 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.864 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:56.864 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.777996+0000", 2024-05-15T13:19:56.864 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0138e-05 2024-05-15T13:19:56.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.864 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.864 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.864 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.778036+0000", 2024-05-15T13:19:56.864 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9994999999999999e-05 2024-05-15T13:19:56.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.864 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.864 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:56.864 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.778140+0000", 2024-05-15T13:19:56.865 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000103613 2024-05-15T13:19:56.865 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.865 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.865 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 4,6", 2024-05-15T13:19:56.865 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.778280+0000", 2024-05-15T13:19:56.865 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013989499999999999 2024-05-15T13:19:56.865 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.865 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.865 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.865 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.795584+0000", 2024-05-15T13:19:56.865 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.017304430999999999 2024-05-15T13:19:56.865 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.865 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.865 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:56.865 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.795596+0000", 2024-05-15T13:19:56.865 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1586999999999999e-05 2024-05-15T13:19:56.866 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.866 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.866 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.866 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.412847+0000", 2024-05-15T13:19:56.866 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.61725165400000004 2024-05-15T13:19:56.866 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.866 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.866 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:56.866 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.412906+0000", 2024-05-15T13:19:56.866 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.825e-05 2024-05-15T13:19:56.866 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.866 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.866 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:56.866 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.412959+0000", 2024-05-15T13:19:56.867 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3164000000000001e-05 2024-05-15T13:19:56.867 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.867 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:56.867 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.867 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.867 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.867 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:231813 6.b 6:d0f442b2:::benchmark_data_smithi177_122278_object101442:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-05-15T13:19:56.867 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:24.778010+0000", 2024-05-15T13:19:56.867 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 512.07930837900005, 2024-05-15T13:19:56.867 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.62571494599999999, 2024-05-15T13:19:56.867 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:56.867 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:56.867 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:56.867 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:19:56.867 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:19:56.867 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 231813 2024-05-15T13:19:56.868 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.868 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:56.868 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.868 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:56.868 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.778010+0000", 2024-05-15T13:19:56.868 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.868 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.868 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.868 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:56.868 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.778010+0000", 2024-05-15T13:19:56.868 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.868 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.868 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.868 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:56.869 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.778010+0000", 2024-05-15T13:19:56.869 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9299999999999998e-07 2024-05-15T13:19:56.869 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.869 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.869 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:56.869 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.778014+0000", 2024-05-15T13:19:56.869 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9609999999999999e-06 2024-05-15T13:19:56.869 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.869 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.869 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:56.869 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.778020+0000", 2024-05-15T13:19:56.869 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1310000000000003e-06 2024-05-15T13:19:56.869 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.869 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.869 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:56.870 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.778026+0000", 2024-05-15T13:19:56.870 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9549999999999997e-06 2024-05-15T13:19:56.870 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.870 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.870 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.870 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.778105+0000", 2024-05-15T13:19:56.870 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9268000000000001e-05 2024-05-15T13:19:56.870 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.870 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.870 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:56.870 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.778587+0000", 2024-05-15T13:19:56.870 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00048155499999999999 2024-05-15T13:19:56.870 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.870 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.870 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:19:56.870 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.778702+0000", 2024-05-15T13:19:56.871 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000114611 2024-05-15T13:19:56.871 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.871 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.871 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.871 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.790480+0000", 2024-05-15T13:19:56.871 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011778857 2024-05-15T13:19:56.871 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.871 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.871 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:56.871 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.790665+0000", 2024-05-15T13:19:56.871 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00018451700000000001 2024-05-15T13:19:56.871 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.871 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.871 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.871 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.403621+0000", 2024-05-15T13:19:56.872 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.61295581099999996 2024-05-15T13:19:56.872 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.872 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.872 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:56.872 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.403678+0000", 2024-05-15T13:19:56.872 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7123999999999997e-05 2024-05-15T13:19:56.872 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.872 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.872 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:56.872 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.403724+0000", 2024-05-15T13:19:56.872 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6563000000000001e-05 2024-05-15T13:19:56.872 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.872 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:56.872 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.872 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.872 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.873 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:231821 6.b 6:d2bfd109:::benchmark_data_smithi177_122278_object101450:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-05-15T13:19:56.873 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:24.781818+0000", 2024-05-15T13:19:56.873 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 512.075499455, 2024-05-15T13:19:56.873 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.62201545000000003, 2024-05-15T13:19:56.873 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:56.873 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:56.873 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:56.873 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:19:56.873 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:19:56.873 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 231821 2024-05-15T13:19:56.873 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.873 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:56.873 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.873 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:56.874 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.781818+0000", 2024-05-15T13:19:56.874 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.874 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.874 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.874 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:56.874 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.781818+0000", 2024-05-15T13:19:56.874 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.874 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.874 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.874 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:56.874 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.781820+0000", 2024-05-15T13:19:56.875 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4729999999999999e-06 2024-05-15T13:19:56.875 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.875 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.875 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:56.875 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.781823+0000", 2024-05-15T13:19:56.875 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5030000000000002e-06 2024-05-15T13:19:56.875 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.875 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.875 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:56.875 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.781832+0000", 2024-05-15T13:19:56.875 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5879999999999998e-06 2024-05-15T13:19:56.875 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.875 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.876 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:56.876 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.781838+0000", 2024-05-15T13:19:56.876 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9859999999999999e-06 2024-05-15T13:19:56.876 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.876 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.876 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.876 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.784073+0000", 2024-05-15T13:19:56.876 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0022349470000000001 2024-05-15T13:19:56.876 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.876 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.876 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:56.876 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.784223+0000", 2024-05-15T13:19:56.876 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015028 2024-05-15T13:19:56.876 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.876 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.877 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:19:56.877 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.784354+0000", 2024-05-15T13:19:56.877 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013109800000000001 2024-05-15T13:19:56.877 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.877 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.877 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:56.877 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.790671+0000", 2024-05-15T13:19:56.877 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0063164620000000001 2024-05-15T13:19:56.877 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.877 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.877 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.877 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.804013+0000", 2024-05-15T13:19:56.877 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013341896000000001 2024-05-15T13:19:56.877 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.878 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.878 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.878 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.403761+0000", 2024-05-15T13:19:56.878 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.59974872700000004 2024-05-15T13:19:56.878 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.878 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.878 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:56.878 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.403806+0000", 2024-05-15T13:19:56.878 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4109000000000001e-05 2024-05-15T13:19:56.878 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.878 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.878 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:56.878 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.403834+0000", 2024-05-15T13:19:56.878 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8381000000000002e-05 2024-05-15T13:19:56.879 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.879 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:56.879 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.879 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.879 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.879 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:231824 6.c 6:3b0f3181:::benchmark_data_smithi177_122278_object101453:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-05-15T13:19:56.879 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:24.785601+0000", 2024-05-15T13:19:56.879 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 512.07171727699995, 2024-05-15T13:19:56.879 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.60745241100000003, 2024-05-15T13:19:56.879 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:56.879 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:56.879 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:56.879 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:19:56.879 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:19:56.879 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 231824 2024-05-15T13:19:56.879 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.880 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:56.880 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.880 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:56.880 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.785601+0000", 2024-05-15T13:19:56.880 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.880 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.880 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.880 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:56.880 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.785601+0000", 2024-05-15T13:19:56.880 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.880 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.880 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.880 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:56.880 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.785602+0000", 2024-05-15T13:19:56.881 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5090000000000001e-06 2024-05-15T13:19:56.881 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.881 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.881 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:56.881 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.785607+0000", 2024-05-15T13:19:56.881 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1490000000000002e-06 2024-05-15T13:19:56.881 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.881 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.881 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:56.881 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.785623+0000", 2024-05-15T13:19:56.881 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5920999999999999e-05 2024-05-15T13:19:56.881 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.881 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.881 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:56.881 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.785631+0000", 2024-05-15T13:19:56.882 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8390000000000007e-06 2024-05-15T13:19:56.882 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.882 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.882 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.882 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.785711+0000", 2024-05-15T13:19:56.882 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9919000000000005e-05 2024-05-15T13:19:56.882 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.882 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.882 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:56.882 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.785833+0000", 2024-05-15T13:19:56.882 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000121741 2024-05-15T13:19:56.882 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.882 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.882 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,6", 2024-05-15T13:19:56.883 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.785939+0000", 2024-05-15T13:19:56.883 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000106259 2024-05-15T13:19:56.883 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.883 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.883 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:56.883 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.791668+0000", 2024-05-15T13:19:56.883 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0057289660000000003 2024-05-15T13:19:56.883 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.883 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.883 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.883 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.804402+0000", 2024-05-15T13:19:56.883 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012734519999999999 2024-05-15T13:19:56.883 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.883 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.883 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.884 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.392923+0000", 2024-05-15T13:19:56.884 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.58852016500000004 2024-05-15T13:19:56.884 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.884 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.884 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:56.884 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.392992+0000", 2024-05-15T13:19:56.884 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9533e-05 2024-05-15T13:19:56.884 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.884 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.884 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:56.884 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.393053+0000", 2024-05-15T13:19:56.884 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0890000000000001e-05 2024-05-15T13:19:56.884 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.884 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:56.884 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.885 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.885 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.885 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:231825 6.6 6:649c106d:::benchmark_data_smithi177_122278_object101454:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-05-15T13:19:56.885 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:24.789145+0000", 2024-05-15T13:19:56.885 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 512.06817317000002, 2024-05-15T13:19:56.885 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.62394155200000001, 2024-05-15T13:19:56.885 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:56.885 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:56.885 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:56.885 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:19:56.885 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:19:56.885 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 231825 2024-05-15T13:19:56.885 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.885 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:56.885 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.885 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:56.886 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.789145+0000", 2024-05-15T13:19:56.886 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.886 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.886 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.886 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:56.886 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.789145+0000", 2024-05-15T13:19:56.886 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.886 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.886 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.886 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:56.886 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.789146+0000", 2024-05-15T13:19:56.886 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.587e-06 2024-05-15T13:19:56.886 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.887 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.887 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:56.887 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.789154+0000", 2024-05-15T13:19:56.887 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6410000000000002e-06 2024-05-15T13:19:56.887 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.887 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.887 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:56.887 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.789166+0000", 2024-05-15T13:19:56.887 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2371000000000001e-05 2024-05-15T13:19:56.887 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.887 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.887 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:56.887 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.789175+0000", 2024-05-15T13:19:56.887 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8170000000000005e-06 2024-05-15T13:19:56.887 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.888 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.888 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.888 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.790431+0000", 2024-05-15T13:19:56.888 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001255993 2024-05-15T13:19:56.888 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.888 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.888 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:56.888 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.790566+0000", 2024-05-15T13:19:56.888 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000135086 2024-05-15T13:19:56.888 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.888 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.888 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 4,6", 2024-05-15T13:19:56.888 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.790706+0000", 2024-05-15T13:19:56.888 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014011300000000001 2024-05-15T13:19:56.888 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.888 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.889 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:56.889 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.805985+0000", 2024-05-15T13:19:56.889 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.015278424000000001 2024-05-15T13:19:56.889 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.889 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.889 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.889 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.806034+0000", 2024-05-15T13:19:56.889 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9536000000000001e-05 2024-05-15T13:19:56.889 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.889 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.889 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.889 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.412999+0000", 2024-05-15T13:19:56.889 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.60696444100000002 2024-05-15T13:19:56.889 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.889 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.890 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:56.890 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.413045+0000", 2024-05-15T13:19:56.890 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6068e-05 2024-05-15T13:19:56.890 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.890 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.890 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:56.890 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.413086+0000", 2024-05-15T13:19:56.890 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1474999999999998e-05 2024-05-15T13:19:56.890 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.890 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:56.890 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.890 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.890 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.890 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:231830 6.6 6:6584125c:::benchmark_data_smithi177_122278_object101459:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-05-15T13:19:56.891 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:24.791648+0000", 2024-05-15T13:19:56.891 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 512.06567017999998, 2024-05-15T13:19:56.891 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.62152610600000002, 2024-05-15T13:19:56.891 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:56.891 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:56.891 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:56.891 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:19:56.891 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:19:56.891 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 231830 2024-05-15T13:19:56.891 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.891 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:56.891 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.891 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:56.891 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.791648+0000", 2024-05-15T13:19:56.891 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.892 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.892 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.892 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:56.892 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.791648+0000", 2024-05-15T13:19:56.892 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.892 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.892 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.892 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:56.892 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.791649+0000", 2024-05-15T13:19:56.892 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2600000000000001e-07 2024-05-15T13:19:56.892 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.892 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.892 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:56.892 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.791653+0000", 2024-05-15T13:19:56.892 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5730000000000001e-06 2024-05-15T13:19:56.893 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.893 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.893 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:56.893 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.791664+0000", 2024-05-15T13:19:56.893 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0932e-05 2024-05-15T13:19:56.893 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.893 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.893 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:56.893 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.791672+0000", 2024-05-15T13:19:56.893 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2640000000000008e-06 2024-05-15T13:19:56.893 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.893 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.893 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.893 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.797311+0000", 2024-05-15T13:19:56.894 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0056389580000000003 2024-05-15T13:19:56.894 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.894 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.894 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:56.894 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.797434+0000", 2024-05-15T13:19:56.894 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000123158 2024-05-15T13:19:56.894 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.894 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.894 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 4,6", 2024-05-15T13:19:56.894 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.797549+0000", 2024-05-15T13:19:56.894 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000114349 2024-05-15T13:19:56.894 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.894 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.894 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:56.895 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.805992+0000", 2024-05-15T13:19:56.895 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0084436589999999992 2024-05-15T13:19:56.895 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.895 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.895 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.895 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.810665+0000", 2024-05-15T13:19:56.895 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.004672781 2024-05-15T13:19:56.895 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.895 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.895 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.895 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.413126+0000", 2024-05-15T13:19:56.895 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.60246056800000003 2024-05-15T13:19:56.895 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.895 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.895 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:56.896 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.413152+0000", 2024-05-15T13:19:56.896 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5854e-05 2024-05-15T13:19:56.896 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.896 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.896 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:56.896 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.413174+0000", 2024-05-15T13:19:56.896 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2184e-05 2024-05-15T13:19:56.896 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.896 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:56.896 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.896 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.896 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.896 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:231835 6.b 6:dc436550:::benchmark_data_smithi177_122278_object101464:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-05-15T13:19:56.896 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:24.797631+0000", 2024-05-15T13:19:56.896 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 512.05968704400004, 2024-05-15T13:19:56.896 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.61617096299999996, 2024-05-15T13:19:56.897 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:56.897 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:56.897 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:56.897 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:19:56.897 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:19:56.897 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 231835 2024-05-15T13:19:56.897 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.897 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:56.897 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.897 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:56.897 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.797631+0000", 2024-05-15T13:19:56.897 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.897 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.897 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.898 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:56.898 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.797631+0000", 2024-05-15T13:19:56.898 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.898 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.898 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.898 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:56.898 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.797632+0000", 2024-05-15T13:19:56.898 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.319e-06 2024-05-15T13:19:56.898 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.898 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.898 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:56.898 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.797637+0000", 2024-05-15T13:19:56.898 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0529999999999997e-06 2024-05-15T13:19:56.898 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.899 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.899 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:56.899 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.797648+0000", 2024-05-15T13:19:56.899 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1229e-05 2024-05-15T13:19:56.899 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.899 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.899 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:56.899 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.797656+0000", 2024-05-15T13:19:56.899 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4429999999999997e-06 2024-05-15T13:19:56.899 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.899 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.899 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.899 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.799212+0000", 2024-05-15T13:19:56.899 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001556338 2024-05-15T13:19:56.900 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.900 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.900 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:56.900 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.799306+0000", 2024-05-15T13:19:56.900 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3968999999999994e-05 2024-05-15T13:19:56.900 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.900 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.900 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:19:56.900 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.799396+0000", 2024-05-15T13:19:56.900 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9347999999999999e-05 2024-05-15T13:19:56.900 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.900 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.900 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:56.900 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.804125+0000", 2024-05-15T13:19:56.900 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0047292610000000002 2024-05-15T13:19:56.901 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.901 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.901 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.901 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.809760+0000", 2024-05-15T13:19:56.901 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0056351780000000002 2024-05-15T13:19:56.901 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.901 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.901 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.901 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.413702+0000", 2024-05-15T13:19:56.901 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.60394203899999999 2024-05-15T13:19:56.901 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.901 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.901 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:56.901 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.413760+0000", 2024-05-15T13:19:56.902 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8316000000000002e-05 2024-05-15T13:19:56.902 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.902 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.902 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:56.902 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.413802+0000", 2024-05-15T13:19:56.902 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1470000000000001e-05 2024-05-15T13:19:56.902 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.902 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:56.902 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.902 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.902 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.902 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:231839 6.6 6:69cffe07:::benchmark_data_smithi177_122278_object101468:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-05-15T13:19:56.902 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:24.799804+0000", 2024-05-15T13:19:56.902 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 512.05751358299995, 2024-05-15T13:19:56.903 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.61343175699999997, 2024-05-15T13:19:56.903 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:56.903 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:56.903 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:56.903 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:19:56.903 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:19:56.903 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 231839 2024-05-15T13:19:56.903 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.903 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:56.903 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.903 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:56.903 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.799804+0000", 2024-05-15T13:19:56.903 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.903 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.903 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.904 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:56.904 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.799804+0000", 2024-05-15T13:19:56.904 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.904 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.904 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.904 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:56.904 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.799805+0000", 2024-05-15T13:19:56.904 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2200000000000004e-07 2024-05-15T13:19:56.904 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.904 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.904 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:56.904 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.799808+0000", 2024-05-15T13:19:56.904 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.129e-06 2024-05-15T13:19:56.905 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.905 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.905 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:56.905 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.799815+0000", 2024-05-15T13:19:56.905 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.272e-06 2024-05-15T13:19:56.905 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.905 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.905 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:56.905 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.799819+0000", 2024-05-15T13:19:56.905 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7479999999999999e-06 2024-05-15T13:19:56.905 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.905 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.905 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.905 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.802156+0000", 2024-05-15T13:19:56.905 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0023374070000000001 2024-05-15T13:19:56.906 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.906 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.906 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:56.906 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.802241+0000", 2024-05-15T13:19:56.906 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4587999999999997e-05 2024-05-15T13:19:56.906 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.906 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.906 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 4,6", 2024-05-15T13:19:56.906 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.802325+0000", 2024-05-15T13:19:56.906 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3935000000000002e-05 2024-05-15T13:19:56.906 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.906 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.906 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:56.906 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.806115+0000", 2024-05-15T13:19:56.906 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0037901440000000001 2024-05-15T13:19:56.906 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.907 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.907 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.907 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.810760+0000", 2024-05-15T13:19:56.907 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046450409999999999 2024-05-15T13:19:56.907 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.907 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.907 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.907 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.413194+0000", 2024-05-15T13:19:56.907 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.60243361600000001 2024-05-15T13:19:56.907 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.907 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.907 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:56.907 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.413217+0000", 2024-05-15T13:19:56.907 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3366999999999998e-05 2024-05-15T13:19:56.908 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.908 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.908 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:56.908 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.413236+0000", 2024-05-15T13:19:56.908 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8987999999999998e-05 2024-05-15T13:19:56.908 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.908 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:56.908 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.908 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.908 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.908 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:231841 6.b 6:de5b11fe:::benchmark_data_smithi177_122278_object101470:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-05-15T13:19:56.908 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:24.805690+0000", 2024-05-15T13:19:56.908 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 512.05162761300005, 2024-05-15T13:19:56.908 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.60825682000000003, 2024-05-15T13:19:56.908 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:56.909 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:56.909 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:56.909 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:19:56.909 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:19:56.909 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 231841 2024-05-15T13:19:56.909 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.909 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:56.909 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.909 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:56.909 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.805690+0000", 2024-05-15T13:19:56.909 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.909 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.909 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.909 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:56.909 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.805690+0000", 2024-05-15T13:19:56.910 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.910 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.910 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.910 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:56.910 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.805692+0000", 2024-05-15T13:19:56.910 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3030000000000001e-06 2024-05-15T13:19:56.910 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.910 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.910 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:56.910 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.805697+0000", 2024-05-15T13:19:56.910 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4859999999999997e-06 2024-05-15T13:19:56.910 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.910 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.910 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:56.910 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.805708+0000", 2024-05-15T13:19:56.911 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0859999999999999e-05 2024-05-15T13:19:56.911 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.911 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.911 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:56.911 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.805718+0000", 2024-05-15T13:19:56.911 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7960000000000007e-06 2024-05-15T13:19:56.911 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.911 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.911 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.911 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.805779+0000", 2024-05-15T13:19:56.911 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0936000000000001e-05 2024-05-15T13:19:56.911 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.911 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.911 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:56.911 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.805906+0000", 2024-05-15T13:19:56.912 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012703700000000001 2024-05-15T13:19:56.912 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.912 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.912 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,6", 2024-05-15T13:19:56.912 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.806043+0000", 2024-05-15T13:19:56.912 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000137188 2024-05-15T13:19:56.912 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.912 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.912 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:56.912 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.809914+0000", 2024-05-15T13:19:56.912 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038714370000000001 2024-05-15T13:19:56.912 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.912 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.912 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.913 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:24.811291+0000", 2024-05-15T13:19:56.913 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0013762690000000001 2024-05-15T13:19:56.913 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.913 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.913 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.913 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.413850+0000", 2024-05-15T13:19:56.913 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.60255945600000005 2024-05-15T13:19:56.913 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.913 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.913 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:56.913 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.413892+0000", 2024-05-15T13:19:56.913 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1860999999999998e-05 2024-05-15T13:19:56.913 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.913 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.914 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:56.914 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:25.413947+0000", 2024-05-15T13:19:56.914 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5191000000000001e-05 2024-05-15T13:19:56.914 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.914 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:56.914 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.914 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.914 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.914 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:246903 6.6 6:6b0bff77:::benchmark_data_smithi177_122278_object116532:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e569)", 2024-05-15T13:19:56.914 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:43.363820+0000", 2024-05-15T13:19:56.914 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 493.49349758599999, 2024-05-15T13:19:56.914 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.470989989, 2024-05-15T13:19:56.914 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:56.914 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:19:56.915 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:56.915 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:19:56.915 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:19:56.915 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 246903 2024-05-15T13:19:56.915 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.915 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:56.915 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.915 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:56.915 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.363820+0000", 2024-05-15T13:19:56.915 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.915 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.915 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.915 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:56.915 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.363820+0000", 2024-05-15T13:19:56.915 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.915 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.916 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.916 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:56.916 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.363822+0000", 2024-05-15T13:19:56.916 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.626e-06 2024-05-15T13:19:56.916 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.916 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.916 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:56.916 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.363826+0000", 2024-05-15T13:19:56.916 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8099999999999999e-06 2024-05-15T13:19:56.916 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.916 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.916 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:56.916 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.363836+0000", 2024-05-15T13:19:56.916 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0429e-05 2024-05-15T13:19:56.916 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.917 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.917 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:56.917 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.363846+0000", 2024-05-15T13:19:56.917 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3479999999999993e-06 2024-05-15T13:19:56.917 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.917 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.917 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.917 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.364320+0000", 2024-05-15T13:19:56.917 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00047445300000000001 2024-05-15T13:19:56.917 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.917 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.917 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:56.917 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.364453+0000", 2024-05-15T13:19:56.917 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013268399999999999 2024-05-15T13:19:56.917 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.918 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.918 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 4,6", 2024-05-15T13:19:56.918 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.364576+0000", 2024-05-15T13:19:56.918 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000123795 2024-05-15T13:19:56.918 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.918 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.918 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.918 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.379974+0000", 2024-05-15T13:19:56.918 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.015397350000000001 2024-05-15T13:19:56.918 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.918 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.918 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:56.918 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.379986+0000", 2024-05-15T13:19:56.918 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2012e-05 2024-05-15T13:19:56.919 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.919 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.919 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.919 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.550082+0000", 2024-05-15T13:19:56.919 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.17009616899999999 2024-05-15T13:19:56.919 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.919 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.919 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:56.919 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.550121+0000", 2024-05-15T13:19:56.919 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8840000000000001e-05 2024-05-15T13:19:56.919 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.919 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.919 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.919 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.820493+0000", 2024-05-15T13:19:56.919 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.270371975 2024-05-15T13:19:56.920 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.920 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.920 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:19:56.920 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.820579+0000", 2024-05-15T13:19:56.920 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.577e-05 2024-05-15T13:19:56.920 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.920 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.920 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.920 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.834751+0000", 2024-05-15T13:19:56.920 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014172278999999999 2024-05-15T13:19:56.920 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.920 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.920 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:56.920 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.834810+0000", 2024-05-15T13:19:56.921 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9449e-05 2024-05-15T13:19:56.921 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.921 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:56.921 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.921 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.921 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.921 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:246897 6.2 6:4c4c5cf5:::benchmark_data_smithi177_122278_object116526:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e570)", 2024-05-15T13:19:56.921 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:43.369720+0000", 2024-05-15T13:19:56.921 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 493.487597736, 2024-05-15T13:19:56.921 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5607889859999999, 2024-05-15T13:19:56.921 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:56.921 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:19:56.921 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:56.921 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:19:56.922 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:19:56.922 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 246897 2024-05-15T13:19:56.922 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.922 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:56.922 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.922 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:56.922 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.369720+0000", 2024-05-15T13:19:56.922 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.922 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.922 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.922 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:56.922 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.369720+0000", 2024-05-15T13:19:56.922 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.922 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.923 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.923 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:56.923 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.369721+0000", 2024-05-15T13:19:56.923 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4999999999999998e-07 2024-05-15T13:19:56.923 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.923 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.923 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:56.923 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.369723+0000", 2024-05-15T13:19:56.923 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5799999999999999e-06 2024-05-15T13:19:56.923 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.923 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.923 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:56.923 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.369730+0000", 2024-05-15T13:19:56.923 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0559999999999999e-06 2024-05-15T13:19:56.924 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.924 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.924 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:56.924 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.369736+0000", 2024-05-15T13:19:56.924 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3519999999999997e-06 2024-05-15T13:19:56.924 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.924 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.924 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.924 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.372247+0000", 2024-05-15T13:19:56.924 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025113879999999998 2024-05-15T13:19:56.924 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.924 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.924 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:56.925 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.372256+0000", 2024-05-15T13:19:56.925 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3649999999999993e-06 2024-05-15T13:19:56.925 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.925 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.925 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.925 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.550153+0000", 2024-05-15T13:19:56.925 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.17789643799999999 2024-05-15T13:19:56.925 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.925 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.925 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:56.925 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.550164+0000", 2024-05-15T13:19:56.925 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1225e-05 2024-05-15T13:19:56.925 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.925 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.926 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.926 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.891374+0000", 2024-05-15T13:19:56.926 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3412096060000001 2024-05-15T13:19:56.926 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.926 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.926 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:19:56.926 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.891501+0000", 2024-05-15T13:19:56.926 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012761000000000001 2024-05-15T13:19:56.926 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.926 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.926 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.926 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.930426+0000", 2024-05-15T13:19:56.926 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.038924875999999997 2024-05-15T13:19:56.927 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.927 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.927 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:56.927 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.930509+0000", 2024-05-15T13:19:56.927 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3040000000000005e-05 2024-05-15T13:19:56.927 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.927 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:56.927 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.927 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.927 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.927 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:246899 6.2 6:42411f6d:::benchmark_data_smithi177_122278_object116528:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e570)", 2024-05-15T13:19:56.927 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:43.369764+0000", 2024-05-15T13:19:56.927 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 493.487553682, 2024-05-15T13:19:56.928 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.561071753, 2024-05-15T13:19:56.928 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:56.928 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:19:56.928 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:56.928 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:19:56.928 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:19:56.928 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 246899 2024-05-15T13:19:56.928 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.928 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:56.928 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.928 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:56.928 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.369764+0000", 2024-05-15T13:19:56.928 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.929 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.929 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.929 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:56.929 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.369764+0000", 2024-05-15T13:19:56.929 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.929 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.929 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.929 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:56.929 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.369765+0000", 2024-05-15T13:19:56.929 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7899999999999999e-07 2024-05-15T13:19:56.929 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.929 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.929 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:56.929 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.369767+0000", 2024-05-15T13:19:56.930 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5110000000000002e-06 2024-05-15T13:19:56.930 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.930 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.930 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:56.930 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.369774+0000", 2024-05-15T13:19:56.930 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2520000000000002e-06 2024-05-15T13:19:56.930 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.930 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.930 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:56.930 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.369780+0000", 2024-05-15T13:19:56.930 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2569999999999998e-06 2024-05-15T13:19:56.930 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.930 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.931 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.931 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.372279+0000", 2024-05-15T13:19:56.931 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0024989119999999998 2024-05-15T13:19:56.931 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.931 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.931 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:56.931 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.372285+0000", 2024-05-15T13:19:56.931 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8249999999999999e-06 2024-05-15T13:19:56.931 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.931 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.931 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.931 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.550305+0000", 2024-05-15T13:19:56.931 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.178019702 2024-05-15T13:19:56.932 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.932 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.932 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:56.932 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.550314+0000", 2024-05-15T13:19:56.932 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2469999999999997e-06 2024-05-15T13:19:56.932 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.932 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.932 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.932 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.891791+0000", 2024-05-15T13:19:56.932 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.341476654 2024-05-15T13:19:56.932 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.932 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.932 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:19:56.932 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.891893+0000", 2024-05-15T13:19:56.933 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000102322 2024-05-15T13:19:56.933 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.933 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.933 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.933 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.930787+0000", 2024-05-15T13:19:56.933 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.038893628 2024-05-15T13:19:56.933 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.933 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.933 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:56.933 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.930836+0000", 2024-05-15T13:19:56.933 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8964000000000001e-05 2024-05-15T13:19:56.933 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.933 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:56.933 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.934 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.934 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.934 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:246917 6.6 6:670f3d4a:::benchmark_data_smithi177_122278_object116546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e570)", 2024-05-15T13:19:56.934 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:43.383438+0000", 2024-05-15T13:19:56.934 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 493.47387961200002, 2024-05-15T13:19:56.934 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4579405990000001, 2024-05-15T13:19:56.934 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:56.934 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:56.934 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:56.934 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:19:56.934 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:19:56.934 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 246917 2024-05-15T13:19:56.934 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.935 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:56.935 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.935 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:56.935 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.383438+0000", 2024-05-15T13:19:56.935 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.935 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.935 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.935 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:56.935 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.383438+0000", 2024-05-15T13:19:56.935 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.935 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.935 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.935 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:56.936 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.383440+0000", 2024-05-15T13:19:56.936 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3489999999999999e-06 2024-05-15T13:19:56.936 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.936 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.936 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:56.936 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.383444+0000", 2024-05-15T13:19:56.936 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0419999999999996e-06 2024-05-15T13:19:56.936 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.936 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.936 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:56.936 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.383453+0000", 2024-05-15T13:19:56.936 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3919999999999992e-06 2024-05-15T13:19:56.936 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.936 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.937 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:56.937 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.383460+0000", 2024-05-15T13:19:56.937 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0570000000000004e-06 2024-05-15T13:19:56.937 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.937 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.937 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.937 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.383520+0000", 2024-05-15T13:19:56.937 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9784000000000002e-05 2024-05-15T13:19:56.937 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.937 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.937 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:56.937 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.383530+0000", 2024-05-15T13:19:56.938 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0115000000000001e-05 2024-05-15T13:19:56.938 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.938 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.938 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.938 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.550191+0000", 2024-05-15T13:19:56.938 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16666066900000001 2024-05-15T13:19:56.938 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.938 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.938 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:56.938 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.550200+0000", 2024-05-15T13:19:56.938 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7310000000000005e-06 2024-05-15T13:19:56.938 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.938 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.938 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.939 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.820778+0000", 2024-05-15T13:19:56.939 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2705771860000001 2024-05-15T13:19:56.939 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.939 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.939 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:56.939 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.820857+0000", 2024-05-15T13:19:56.939 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9056999999999999e-05 2024-05-15T13:19:56.939 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.939 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.939 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 4,5,6", 2024-05-15T13:19:56.939 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.820942+0000", 2024-05-15T13:19:56.939 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4973000000000001e-05 2024-05-15T13:19:56.939 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.939 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.940 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:56.940 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.834400+0000", 2024-05-15T13:19:56.940 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013458659 2024-05-15T13:19:56.940 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.940 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.940 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.940 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.834442+0000", 2024-05-15T13:19:56.940 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1399000000000001e-05 2024-05-15T13:19:56.940 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.940 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.940 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.940 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.834486+0000", 2024-05-15T13:19:56.940 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4295000000000002e-05 2024-05-15T13:19:56.940 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.941 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.941 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.941 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.841159+0000", 2024-05-15T13:19:56.941 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0066732750000000002 2024-05-15T13:19:56.941 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.941 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.941 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:56.941 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.841232+0000", 2024-05-15T13:19:56.941 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3144000000000003e-05 2024-05-15T13:19:56.941 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.941 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.941 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:56.941 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.841379+0000", 2024-05-15T13:19:56.942 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000146472 2024-05-15T13:19:56.942 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.942 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:56.942 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.942 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.942 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.942 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:246921 6.2 6:41aebe24:::benchmark_data_smithi177_122278_object116550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e570)", 2024-05-15T13:19:56.942 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:43.386972+0000", 2024-05-15T13:19:56.942 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 493.47034545000002, 2024-05-15T13:19:56.942 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5430468429999999, 2024-05-15T13:19:56.942 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:56.942 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:56.942 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:56.943 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:19:56.943 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:19:56.943 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 246921 2024-05-15T13:19:56.943 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.943 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:56.943 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.943 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:56.943 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.386972+0000", 2024-05-15T13:19:56.943 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.943 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.943 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.943 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:56.943 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.386972+0000", 2024-05-15T13:19:56.944 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.944 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.944 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.944 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:56.944 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.386973+0000", 2024-05-15T13:19:56.944 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3099999999999996e-07 2024-05-15T13:19:56.944 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.944 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.944 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:56.944 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.386979+0000", 2024-05-15T13:19:56.944 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2290000000000002e-06 2024-05-15T13:19:56.944 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.944 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.944 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:56.945 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.386990+0000", 2024-05-15T13:19:56.945 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1746000000000001e-05 2024-05-15T13:19:56.945 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.945 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.945 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:56.945 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.386996+0000", 2024-05-15T13:19:56.945 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0789999999999997e-06 2024-05-15T13:19:56.945 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.945 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.945 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.945 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.387057+0000", 2024-05-15T13:19:56.945 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0868000000000001e-05 2024-05-15T13:19:56.945 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.946 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.946 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:56.946 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.387065+0000", 2024-05-15T13:19:56.946 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3590000000000002e-06 2024-05-15T13:19:56.946 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.946 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.946 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.946 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.550345+0000", 2024-05-15T13:19:56.946 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16328046800000001 2024-05-15T13:19:56.946 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.946 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.946 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:56.946 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.550353+0000", 2024-05-15T13:19:56.947 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1079999999999993e-06 2024-05-15T13:19:56.947 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.947 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.947 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.947 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.891913+0000", 2024-05-15T13:19:56.947 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.341560114 2024-05-15T13:19:56.947 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.947 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.947 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:56.947 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.895428+0000", 2024-05-15T13:19:56.947 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003515097 2024-05-15T13:19:56.947 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.947 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.948 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2,4,11", 2024-05-15T13:19:56.948 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.895574+0000", 2024-05-15T13:19:56.948 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014553100000000001 2024-05-15T13:19:56.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.948 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.948 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:56.948 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.909245+0000", 2024-05-15T13:19:56.948 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013670836 2024-05-15T13:19:56.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.948 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.948 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.948 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.909405+0000", 2024-05-15T13:19:56.948 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016029 2024-05-15T13:19:56.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.949 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.949 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.949 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.919680+0000", 2024-05-15T13:19:56.949 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010274617 2024-05-15T13:19:56.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.949 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.949 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.949 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.924858+0000", 2024-05-15T13:19:56.949 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0051782140000000004 2024-05-15T13:19:56.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.949 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.949 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:56.949 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.924900+0000", 2024-05-15T13:19:56.949 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2197999999999998e-05 2024-05-15T13:19:56.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.950 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.950 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:56.950 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.930019+0000", 2024-05-15T13:19:56.950 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0051191580000000004 2024-05-15T13:19:56.950 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.950 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:56.950 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.950 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.950 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:246929 6.6 6:61094500:::benchmark_data_smithi177_122278_object116558:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e570)", 2024-05-15T13:19:56.950 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:43.406777+0000", 2024-05-15T13:19:56.954 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 493.45054079599998, 2024-05-15T13:19:56.954 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4348986909999999, 2024-05-15T13:19:56.954 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:56.954 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:56.954 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:56.954 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:19:56.954 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:19:56.954 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 246929 2024-05-15T13:19:56.954 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.954 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:56.955 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.955 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:56.955 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.406777+0000", 2024-05-15T13:19:56.955 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.955 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.955 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.955 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:56.955 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.406777+0000", 2024-05-15T13:19:56.955 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.955 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.955 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.955 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:56.955 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.406778+0000", 2024-05-15T13:19:56.955 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2589999999999999e-06 2024-05-15T13:19:56.956 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.956 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.956 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:56.956 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.406783+0000", 2024-05-15T13:19:56.956 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8439999999999997e-06 2024-05-15T13:19:56.956 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.956 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.956 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:56.956 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.406794+0000", 2024-05-15T13:19:56.956 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0373999999999999e-05 2024-05-15T13:19:56.956 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.956 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.956 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:56.957 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.406800+0000", 2024-05-15T13:19:56.957 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9029999999999998e-06 2024-05-15T13:19:56.957 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.957 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.957 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.957 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.406855+0000", 2024-05-15T13:19:56.957 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4126000000000003e-05 2024-05-15T13:19:56.957 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.957 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.957 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:56.957 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.406863+0000", 2024-05-15T13:19:56.957 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5450000000000003e-06 2024-05-15T13:19:56.957 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.958 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.958 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.958 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.550237+0000", 2024-05-15T13:19:56.958 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.143374002 2024-05-15T13:19:56.958 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.958 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.958 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:56.958 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.550247+0000", 2024-05-15T13:19:56.958 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6099999999999995e-06 2024-05-15T13:19:56.958 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.958 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.958 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.958 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.825230+0000", 2024-05-15T13:19:56.959 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2749827460000001 2024-05-15T13:19:56.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.959 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.959 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:56.959 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.825376+0000", 2024-05-15T13:19:56.959 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000146549 2024-05-15T13:19:56.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.959 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.959 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 4,5,6", 2024-05-15T13:19:56.959 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.825525+0000", 2024-05-15T13:19:56.959 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014925 2024-05-15T13:19:56.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.959 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.960 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:56.960 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.834454+0000", 2024-05-15T13:19:56.960 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0089283750000000005 2024-05-15T13:19:56.960 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.960 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.960 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.960 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.834524+0000", 2024-05-15T13:19:56.960 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9919000000000006e-05 2024-05-15T13:19:56.960 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.960 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.960 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.960 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.834848+0000", 2024-05-15T13:19:56.960 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00032469699999999998 2024-05-15T13:19:56.960 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.961 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.961 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.961 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.841451+0000", 2024-05-15T13:19:56.961 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0066024040000000001 2024-05-15T13:19:56.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.961 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.961 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:56.961 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.841526+0000", 2024-05-15T13:19:56.961 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4938999999999995e-05 2024-05-15T13:19:56.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.961 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.961 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:56.961 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.841676+0000", 2024-05-15T13:19:56.961 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000150149 2024-05-15T13:19:56.962 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.962 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:56.962 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.962 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.962 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.962 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:246935 6.6 6:6a277912:::benchmark_data_smithi177_122278_object116564:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e570)", 2024-05-15T13:19:56.962 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:11:43.432656+0000", 2024-05-15T13:19:56.962 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 493.42466149500001, 2024-05-15T13:19:56.962 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.409276258, 2024-05-15T13:19:56.962 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:56.962 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:56.962 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:56.962 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:19:56.963 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:19:56.963 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 246935 2024-05-15T13:19:56.963 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.963 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:56.963 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.963 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:56.963 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.432656+0000", 2024-05-15T13:19:56.963 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.963 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.963 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.963 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:56.963 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.432656+0000", 2024-05-15T13:19:56.963 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.963 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.963 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.964 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:56.964 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.432657+0000", 2024-05-15T13:19:56.964 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0109999999999999e-06 2024-05-15T13:19:56.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.964 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.964 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:56.964 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.432662+0000", 2024-05-15T13:19:56.964 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5630000000000002e-06 2024-05-15T13:19:56.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.964 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.964 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:56.964 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.432674+0000", 2024-05-15T13:19:56.964 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1535999999999999e-05 2024-05-15T13:19:56.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.965 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.965 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:56.965 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.432681+0000", 2024-05-15T13:19:56.965 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1339999999999998e-06 2024-05-15T13:19:56.965 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.965 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.965 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.965 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.432746+0000", 2024-05-15T13:19:56.965 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5418000000000003e-05 2024-05-15T13:19:56.965 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.965 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.965 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:56.965 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.432758+0000", 2024-05-15T13:19:56.965 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2177999999999999e-05 2024-05-15T13:19:56.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.966 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.966 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.966 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.550285+0000", 2024-05-15T13:19:56.966 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.117526906 2024-05-15T13:19:56.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.966 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.966 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:56.966 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:43.550294+0000", 2024-05-15T13:19:56.966 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9910000000000008e-06 2024-05-15T13:19:56.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.966 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.966 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.966 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.830143+0000", 2024-05-15T13:19:56.967 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.279848791 2024-05-15T13:19:56.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.967 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.967 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:56.967 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.830250+0000", 2024-05-15T13:19:56.967 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000107227 2024-05-15T13:19:56.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.967 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.967 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 4,5,6", 2024-05-15T13:19:56.967 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.830337+0000", 2024-05-15T13:19:56.967 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7053999999999997e-05 2024-05-15T13:19:56.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.967 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.967 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:56.967 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.834494+0000", 2024-05-15T13:19:56.967 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041568070000000002 2024-05-15T13:19:56.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.968 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.968 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.834875+0000", 2024-05-15T13:19:56.968 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00038097000000000001 2024-05-15T13:19:56.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.968 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.968 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.839754+0000", 2024-05-15T13:19:56.968 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0048794839999999999 2024-05-15T13:19:56.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.968 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.968 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.841734+0000", 2024-05-15T13:19:56.969 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001979976 2024-05-15T13:19:56.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.969 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.969 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:56.969 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.841805+0000", 2024-05-15T13:19:56.969 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0947999999999999e-05 2024-05-15T13:19:56.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.969 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.969 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:56.969 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:11:44.841933+0000", 2024-05-15T13:19:56.969 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012726400000000001 2024-05-15T13:19:56.969 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.969 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:56.969 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.970 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.970 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:258418 6.6 6:6eee94c7:::benchmark_data_smithi177_122278_object128047:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-05-15T13:19:56.970 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:12:01.318168+0000", 2024-05-15T13:19:56.970 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 475.53914948300002, 2024-05-15T13:19:56.970 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4465179770000001, 2024-05-15T13:19:56.970 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:56.970 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:56.970 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:56.970 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:19:56.970 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:19:56.970 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 258418 2024-05-15T13:19:56.970 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.970 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:56.970 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.970 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:56.971 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.318168+0000", 2024-05-15T13:19:56.971 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.971 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.971 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:56.971 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.318168+0000", 2024-05-15T13:19:56.971 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.971 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.971 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:56.971 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.318170+0000", 2024-05-15T13:19:56.971 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6759999999999999e-06 2024-05-15T13:19:56.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.971 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.971 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:56.972 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.318178+0000", 2024-05-15T13:19:56.972 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4410000000000004e-06 2024-05-15T13:19:56.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.972 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.972 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:56.972 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.318192+0000", 2024-05-15T13:19:56.972 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4192999999999999e-05 2024-05-15T13:19:56.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.972 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.972 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:56.972 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.318201+0000", 2024-05-15T13:19:56.972 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7510000000000003e-06 2024-05-15T13:19:56.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.972 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.972 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.973 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.318271+0000", 2024-05-15T13:19:56.973 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9999999999999994e-05 2024-05-15T13:19:56.973 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.973 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.973 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-05-15T13:19:56.973 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.318278+0000", 2024-05-15T13:19:56.973 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.922e-06 2024-05-15T13:19:56.973 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.973 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.973 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.973 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.322418+0000", 2024-05-15T13:19:56.973 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041395420000000004 2024-05-15T13:19:56.973 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.973 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.973 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:56.973 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.322426+0000", 2024-05-15T13:19:56.974 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2460000000000003e-06 2024-05-15T13:19:56.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.974 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.974 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.974 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:02.332338+0000", 2024-05-15T13:19:56.974 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.009911453 2024-05-15T13:19:56.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.974 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.974 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:56.974 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:02.332353+0000", 2024-05-15T13:19:56.974 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5624000000000002e-05 2024-05-15T13:19:56.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.974 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.975 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.975 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.735938+0000", 2024-05-15T13:19:56.975 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4035849790000001 2024-05-15T13:19:56.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.975 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.975 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:56.975 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.736023+0000", 2024-05-15T13:19:56.975 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4925000000000004e-05 2024-05-15T13:19:56.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.975 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.975 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,6,11", 2024-05-15T13:19:56.975 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.736104+0000", 2024-05-15T13:19:56.975 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0785000000000007e-05 2024-05-15T13:19:56.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.976 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.976 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.976 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.764098+0000", 2024-05-15T13:19:56.976 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.027994027000000001 2024-05-15T13:19:56.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.976 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.976 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:56.976 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.764238+0000", 2024-05-15T13:19:56.976 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000139586 2024-05-15T13:19:56.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.976 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.976 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.976 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.764303+0000", 2024-05-15T13:19:56.976 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5733000000000002e-05 2024-05-15T13:19:56.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.977 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.977 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.977 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.764494+0000", 2024-05-15T13:19:56.977 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000191079 2024-05-15T13:19:56.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.977 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.977 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:56.977 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.764560+0000", 2024-05-15T13:19:56.977 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5732e-05 2024-05-15T13:19:56.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.977 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.977 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:56.977 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.764686+0000", 2024-05-15T13:19:56.978 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012628300000000001 2024-05-15T13:19:56.978 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.978 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:56.978 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.978 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.978 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.978 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:258437 6.6 6:680b743c:::benchmark_data_smithi177_122278_object128066:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-05-15T13:19:56.978 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:12:01.345323+0000", 2024-05-15T13:19:56.978 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 475.51199463900002, 2024-05-15T13:19:56.978 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.419589931, 2024-05-15T13:19:56.978 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:56.978 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:56.978 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:56.978 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:19:56.978 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:19:56.979 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 258437 2024-05-15T13:19:56.979 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.979 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:56.979 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.979 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:56.979 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.345323+0000", 2024-05-15T13:19:56.979 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.979 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.979 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.979 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:56.979 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.345323+0000", 2024-05-15T13:19:56.979 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.979 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.979 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.979 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:56.980 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.345325+0000", 2024-05-15T13:19:56.980 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0480000000000001e-06 2024-05-15T13:19:56.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.980 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.980 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:56.980 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.345335+0000", 2024-05-15T13:19:56.980 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5000000000000005e-06 2024-05-15T13:19:56.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.980 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.980 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:56.980 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.345351+0000", 2024-05-15T13:19:56.980 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5985999999999999e-05 2024-05-15T13:19:56.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.981 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.981 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:56.981 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.345359+0000", 2024-05-15T13:19:56.981 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1859999999999992e-06 2024-05-15T13:19:56.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.981 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.981 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.981 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.345435+0000", 2024-05-15T13:19:56.981 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5519000000000007e-05 2024-05-15T13:19:56.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.981 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.981 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:56.981 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.345446+0000", 2024-05-15T13:19:56.981 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1443e-05 2024-05-15T13:19:56.982 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.982 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.982 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.982 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:02.332484+0000", 2024-05-15T13:19:56.982 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.98703755999999998 2024-05-15T13:19:56.982 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.982 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.982 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:56.982 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:02.332494+0000", 2024-05-15T13:19:56.982 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0529e-05 2024-05-15T13:19:56.982 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.982 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.982 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.983 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.740322+0000", 2024-05-15T13:19:56.983 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.407828335 2024-05-15T13:19:56.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.983 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.983 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:56.983 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.740427+0000", 2024-05-15T13:19:56.983 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000104786 2024-05-15T13:19:56.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.983 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.983 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,6,11", 2024-05-15T13:19:56.983 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.741107+0000", 2024-05-15T13:19:56.983 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00067936199999999998 2024-05-15T13:19:56.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.983 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.984 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.984 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.764176+0000", 2024-05-15T13:19:56.984 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.023069301 2024-05-15T13:19:56.984 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.984 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.984 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:56.984 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.764246+0000", 2024-05-15T13:19:56.984 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9820999999999994e-05 2024-05-15T13:19:56.984 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.984 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.984 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.984 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.764368+0000", 2024-05-15T13:19:56.984 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000122538 2024-05-15T13:19:56.985 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.985 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.985 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.985 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.764748+0000", 2024-05-15T13:19:56.985 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00037968600000000001 2024-05-15T13:19:56.985 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.985 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.985 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:56.985 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.764803+0000", 2024-05-15T13:19:56.985 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4950000000000001e-05 2024-05-15T13:19:56.985 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.985 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.985 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:56.986 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.764913+0000", 2024-05-15T13:19:56.986 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000110381 2024-05-15T13:19:56.986 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.986 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:56.986 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.986 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.986 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.986 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:258456 6.6 6:6189fcac:::benchmark_data_smithi177_122278_object128085:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-05-15T13:19:56.986 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:12:01.370330+0000", 2024-05-15T13:19:56.986 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 475.48698839299999, 2024-05-15T13:19:56.986 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3948201789999999, 2024-05-15T13:19:56.986 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:56.986 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:56.987 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:56.987 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:19:56.987 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:19:56.987 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 258456 2024-05-15T13:19:56.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.987 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:56.987 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.987 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:56.987 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.370330+0000", 2024-05-15T13:19:56.987 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.987 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.987 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:56.988 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.370330+0000", 2024-05-15T13:19:56.988 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.988 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.988 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.988 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:56.988 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.370331+0000", 2024-05-15T13:19:56.988 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9080000000000002e-06 2024-05-15T13:19:56.988 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.988 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.988 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:56.988 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.370338+0000", 2024-05-15T13:19:56.988 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9720000000000003e-06 2024-05-15T13:19:56.988 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.988 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.989 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:56.989 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.370353+0000", 2024-05-15T13:19:56.989 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4564e-05 2024-05-15T13:19:56.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.989 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.989 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:56.989 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.370361+0000", 2024-05-15T13:19:56.989 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6129999999999998e-06 2024-05-15T13:19:56.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.989 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.989 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.989 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.370423+0000", 2024-05-15T13:19:56.989 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2825999999999998e-05 2024-05-15T13:19:56.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.990 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.990 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:56.990 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.370432+0000", 2024-05-15T13:19:56.990 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9339999999999995e-06 2024-05-15T13:19:56.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.990 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.990 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.990 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:02.332546+0000", 2024-05-15T13:19:56.990 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.96211395799999999 2024-05-15T13:19:56.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.990 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.990 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:56.990 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:02.332557+0000", 2024-05-15T13:19:56.990 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0507e-05 2024-05-15T13:19:56.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.991 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.991 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.991 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.746457+0000", 2024-05-15T13:19:56.991 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.413900253 2024-05-15T13:19:56.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.991 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.991 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:56.991 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.746597+0000", 2024-05-15T13:19:56.991 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000140197 2024-05-15T13:19:56.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.991 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.991 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,6,11", 2024-05-15T13:19:56.991 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.746739+0000", 2024-05-15T13:19:56.991 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000142208 2024-05-15T13:19:56.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.992 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.992 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.992 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.764228+0000", 2024-05-15T13:19:56.992 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.017488284999999999 2024-05-15T13:19:56.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.992 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.992 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:56.992 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.764252+0000", 2024-05-15T13:19:56.992 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4564999999999999e-05 2024-05-15T13:19:56.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.992 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.992 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.992 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.764436+0000", 2024-05-15T13:19:56.993 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00018326700000000001 2024-05-15T13:19:56.993 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.993 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.993 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:56.993 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.764972+0000", 2024-05-15T13:19:56.993 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00053625999999999999 2024-05-15T13:19:56.993 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.993 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.993 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:56.993 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.765030+0000", 2024-05-15T13:19:56.993 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8069999999999998e-05 2024-05-15T13:19:56.993 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.993 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.993 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:56.994 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.765150+0000", 2024-05-15T13:19:56.994 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011979199999999999 2024-05-15T13:19:56.994 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.994 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:56.994 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:56.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.994 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.994 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.25735.0:258463 6.6 6:689fdf6c:::benchmark_data_smithi177_122278_object128092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-05-15T13:19:56.994 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:12:01.384661+0000", 2024-05-15T13:19:56.994 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 475.47265732900001, 2024-05-15T13:19:56.994 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3807457240000001, 2024-05-15T13:19:56.994 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:19:56.994 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:19:56.994 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:19:56.995 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.25735", 2024-05-15T13:19:56.995 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:19:56.995 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 258463 2024-05-15T13:19:56.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.995 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:19:56.995 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.995 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:19:56.995 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.384661+0000", 2024-05-15T13:19:56.995 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.995 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.995 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:19:56.995 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.384661+0000", 2024-05-15T13:19:56.995 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:19:56.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.996 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.996 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:19:56.996 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.384662+0000", 2024-05-15T13:19:56.996 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.096e-06 2024-05-15T13:19:56.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.996 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.996 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:19:56.996 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.384665+0000", 2024-05-15T13:19:56.996 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6689999999999999e-06 2024-05-15T13:19:56.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.996 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.996 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:19:56.996 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.384673+0000", 2024-05-15T13:19:56.997 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2450000000000001e-06 2024-05-15T13:19:56.997 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.997 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.997 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:19:56.997 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.384677+0000", 2024-05-15T13:19:56.997 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6290000000000001e-06 2024-05-15T13:19:56.997 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.997 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.997 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.997 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.384734+0000", 2024-05-15T13:19:56.997 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6304999999999997e-05 2024-05-15T13:19:56.997 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.997 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.997 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:56.997 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:01.384741+0000", 2024-05-15T13:19:56.998 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2749999999999998e-06 2024-05-15T13:19:56.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.998 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.998 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.998 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:02.332663+0000", 2024-05-15T13:19:56.998 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.94792191100000001 2024-05-15T13:19:56.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.998 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.998 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:19:56.998 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:02.332673+0000", 2024-05-15T13:19:56.998 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0186e-05 2024-05-15T13:19:56.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.998 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.999 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:19:56.999 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.751639+0000", 2024-05-15T13:19:56.999 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4189664470000001 2024-05-15T13:19:56.999 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.999 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.999 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:19:56.999 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.751765+0000", 2024-05-15T13:19:56.999 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012604599999999999 2024-05-15T13:19:56.999 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.999 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.999 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,6,11", 2024-05-15T13:19:56.999 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.751901+0000", 2024-05-15T13:19:56.999 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000136002 2024-05-15T13:19:56.999 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:56.999 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:56.999 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:19:57.000 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.764260+0000", 2024-05-15T13:19:57.000 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012358535 2024-05-15T13:19:57.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:57.000 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:57.000 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:57.000 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.765194+0000", 2024-05-15T13:19:57.000 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00093413399999999998 2024-05-15T13:19:57.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:57.000 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:57.000 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:57.000 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.765228+0000", 2024-05-15T13:19:57.000 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4152000000000001e-05 2024-05-15T13:19:57.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:57.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:57.001 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:19:57.001 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.765304+0000", 2024-05-15T13:19:57.001 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5668999999999997e-05 2024-05-15T13:19:57.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:57.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:57.001 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:19:57.001 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.765346+0000", 2024-05-15T13:19:57.001 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2394000000000001e-05 2024-05-15T13:19:57.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:19:57.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:19:57.001 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:19:57.001 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:12:03.765406+0000", 2024-05-15T13:19:57.001 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0028999999999997e-05 2024-05-15T13:19:57.001 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:57.002 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:57.002 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:57.002 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:19:57.002 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:19:57.002 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:19:57.002 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:19:57.003 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 48 16 40184 40168 52.2867 115.062 0.00569606 0.0191216 2024-05-15T13:19:57.027 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:19:57.029 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:57.052 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:19:57.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:57.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:19:57.061 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:19:57.090 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:19:57.123 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:19:57.152 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:19:57.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:19:57.200 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:57.204 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:19:57.222 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:19:57.260 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:19:57.293 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:19:57.328 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:19:57.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:19:57.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:19:57.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:19:57.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:19:57.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:19:57.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:19:57.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:19:57.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:19:57.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:19:57.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:19:57.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:19:57.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:19:57.582 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:19:57.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:19:57.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:19:57.689 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:19:57.739 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:19:57.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:57 smithi177 ceph-mon[44303]: 1.5 scrub starts 2024-05-15T13:19:57.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:57 smithi177 ceph-mon[44303]: 1.5 scrub ok 2024-05-15T13:19:57.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:57 smithi177 ceph-mon[44303]: pgmap v1974: 37 pgs: 37 active+clean; 2.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-05-15T13:19:57.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:57 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3171425638' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:19:57.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:57 smithi177 ceph-mon[44303]: 8.7 scrub starts 2024-05-15T13:19:57.997 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 49 16 41959 41943 53.4825 110.938 0.00555815 0.0186922 2024-05-15T13:19:58.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:57 smithi027 ceph-mon[39795]: 1.5 scrub starts 2024-05-15T13:19:58.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:57 smithi027 ceph-mon[39795]: 1.5 scrub ok 2024-05-15T13:19:58.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:57 smithi027 ceph-mon[39795]: pgmap v1974: 37 pgs: 37 active+clean; 2.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-05-15T13:19:58.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:57 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3171425638' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:19:58.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:57 smithi027 ceph-mon[39795]: 8.7 scrub starts 2024-05-15T13:19:58.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:57 smithi120 ceph-mon[43961]: 1.5 scrub starts 2024-05-15T13:19:58.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:57 smithi120 ceph-mon[43961]: 1.5 scrub ok 2024-05-15T13:19:58.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:57 smithi120 ceph-mon[43961]: pgmap v1974: 37 pgs: 37 active+clean; 2.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-05-15T13:19:58.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:57 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3171425638' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:19:58.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:57 smithi120 ceph-mon[43961]: 8.7 scrub starts 2024-05-15T13:19:58.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:58 smithi177 ceph-mon[44303]: 8.7 scrub ok 2024-05-15T13:19:58.997 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 50 16 43532 43516 54.3788 98.3125 0.00530704 0.0183851 2024-05-15T13:19:59.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:58 smithi027 ceph-mon[39795]: 8.7 scrub ok 2024-05-15T13:19:59.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:58 smithi120 ceph-mon[43961]: 8.7 scrub ok 2024-05-15T13:19:59.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:19:59 smithi177 ceph-mon[44303]: pgmap v1975: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 2.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-05-15T13:19:59.998 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 51 16 45137 45121 55.2785 100.312 0.0122729 0.0180849 2024-05-15T13:20:00.018 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:19:59 smithi027 ceph-mon[39795]: pgmap v1975: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 2.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-05-15T13:20:00.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:19:59 smithi120 ceph-mon[43961]: pgmap v1975: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 2.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-05-15T13:20:00.063 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-05-15T13:20:00.379 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:20:00.510 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:20:00.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:00.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:20:00.619 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:20:00.670 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:00.705 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:20:00.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:00.713 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:20:00.779 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:00 smithi027 ceph-mon[39795]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-05-15T13:20:00.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:00 smithi027 ceph-mon[39795]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-05-15T13:20:00.813 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:20:00.855 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:00.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:00 smithi177 ceph-mon[44303]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-05-15T13:20:00.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:00 smithi177 ceph-mon[44303]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-05-15T13:20:00.998 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 52 16 46661 46645 56.0468 95.25 0.0024746 0.0177822 2024-05-15T13:20:01.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:00 smithi120 ceph-mon[43961]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-05-15T13:20:01.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:00 smithi120 ceph-mon[43961]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-05-15T13:20:01.412 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:20:01.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:01.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:20:01.519 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:20:01.573 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:01.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:01.631 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:20:01.631 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:20:01.732 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:20:01.773 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:01.788 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:01 smithi120 ceph-mon[43961]: pgmap v1976: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 2.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 115 MiB/s wr, 1.83k op/s 2024-05-15T13:20:01.788 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:01 smithi120 ceph-mon[43961]: 8.4 scrub starts 2024-05-15T13:20:01.931 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:20:01.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:01 smithi177 ceph-mon[44303]: pgmap v1976: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 2.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 115 MiB/s wr, 1.83k op/s 2024-05-15T13:20:01.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:01 smithi177 ceph-mon[44303]: 8.4 scrub starts 2024-05-15T13:20:01.939 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:01 smithi027 ceph-mon[39795]: pgmap v1976: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 2.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 115 MiB/s wr, 1.83k op/s 2024-05-15T13:20:01.939 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:01 smithi027 ceph-mon[39795]: 8.4 scrub starts 2024-05-15T13:20:01.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:01.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:20:01.998 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 53 16 48352 48336 56.983 105.688 0.0139424 0.0175442 2024-05-15T13:20:02.040 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:20:02.078 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:02.209 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:20:02.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:02.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:20:02.315 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:20:02.373 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:02.407 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:20:02.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:02.412 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:20:02.513 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:20:02.550 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:02.560 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:20:02.910 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:20:02.912 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:02 smithi027 ceph-mon[39795]: 8.4 scrub ok 2024-05-15T13:20:02.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:02.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:20:02.998 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 54 16 49736 49720 57.5293 86.5 0.00764905 0.0173778 2024-05-15T13:20:03.015 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:20:03.038 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:02 smithi120 ceph-mon[43961]: 8.4 scrub ok 2024-05-15T13:20:03.073 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:03.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:02 smithi177 ceph-mon[44303]: 8.4 scrub ok 2024-05-15T13:20:03.700 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:20:03.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:03 smithi120 ceph-mon[43961]: 8.6 scrub starts 2024-05-15T13:20:03.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:03 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:20:03.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:03 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:20:03.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:03 smithi120 ceph-mon[43961]: pgmap v1977: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 2.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-05-15T13:20:03.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:03 smithi120 ceph-mon[43961]: 8.6 scrub ok 2024-05-15T13:20:03.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:03 smithi120 ceph-mon[43961]: osdmap e767: 12 total, 12 up, 11 in 2024-05-15T13:20:03.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:03 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4086569997' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:20:03.938 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:20:03.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:03.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:20:03.998 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 55 16 51091 51075 58.0229 84.6875 0.00561478 0.01723 2024-05-15T13:20:04.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:03 smithi027 ceph-mon[39795]: 8.6 scrub starts 2024-05-15T13:20:04.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:03 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:20:04.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:03 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:20:04.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:03 smithi027 ceph-mon[39795]: pgmap v1977: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 2.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-05-15T13:20:04.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:03 smithi027 ceph-mon[39795]: 8.6 scrub ok 2024-05-15T13:20:04.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:03 smithi027 ceph-mon[39795]: osdmap e767: 12 total, 12 up, 11 in 2024-05-15T13:20:04.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:03 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4086569997' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:20:04.041 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:20:04.041 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:20:04.042 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:20:04.102 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:03 smithi177 ceph-mon[44303]: 8.6 scrub starts 2024-05-15T13:20:04.102 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:03 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:20:04.102 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:03 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:20:04.102 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:03 smithi177 ceph-mon[44303]: pgmap v1977: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 2.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-05-15T13:20:04.102 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:03 smithi177 ceph-mon[44303]: 8.6 scrub ok 2024-05-15T13:20:04.102 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:03 smithi177 ceph-mon[44303]: osdmap e767: 12 total, 12 up, 11 in 2024-05-15T13:20:04.103 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:03 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4086569997' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:20:04.108 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:04.150 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:20:04.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:04.157 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:20:04.258 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:20:04.292 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:04.318 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:20:04.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:04.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:20:04.424 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:20:04.455 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:04.478 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:20:04.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:04.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:20:04.593 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:20:04.621 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:04.822 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:20:04.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:04.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:20:04.856 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-05-15T13:20:04.923 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:04 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:20:04.923 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:04 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:20:04.923 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:04 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:20:04.923 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:04 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T13:20:04.923 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:04 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4086569997' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:20:04.923 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:04 smithi120 ceph-mon[43961]: osdmap e768: 12 total, 12 up, 11 in 2024-05-15T13:20:04.929 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:20:04.958 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:04.982 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:20:04.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:04.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:20:05.000 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 56 16 52716 52700 58.7981 101.562 0.0081982 0.0170018 2024-05-15T13:20:05.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:04 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:20:05.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:04 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:20:05.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:04 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:20:05.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:04 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T13:20:05.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:04 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4086569997' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:20:05.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:04 smithi027 ceph-mon[39795]: osdmap e768: 12 total, 12 up, 11 in 2024-05-15T13:20:05.087 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:20:05.132 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:04 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:20:05.132 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:04 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:20:05.132 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:04 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:20:05.132 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:04 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T13:20:05.132 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:04 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4086569997' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:20:05.132 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:04 smithi177 ceph-mon[44303]: osdmap e768: 12 total, 12 up, 11 in 2024-05-15T13:20:05.138 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:05.169 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:20:05.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:05.173 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:20:05.195 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:20:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:20:05.204 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:20:05.273 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:20:05.322 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:05.366 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:20:05.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:05.373 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:20:05.429 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:20:05.474 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:20:05.514 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:05.593 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:20:05.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:05.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:20:05.702 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:20:05.730 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:05 smithi120 ceph-mon[43961]: pgmap v1980: 37 pgs: 37 active+clean; 3.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-05-15T13:20:05.733 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:05.779 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:05 smithi027 ceph-mon[39795]: pgmap v1980: 37 pgs: 37 active+clean; 3.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-05-15T13:20:05.970 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":1980,"stamp":"2024-05-15T13:20:04.599669+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3224240672,"num_objects":49191,"num_object_clones":0,"num_object_copies":147573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49189,"num_write_kb":3148096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1550,"num_bytes_recovered":101580800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52108,"ondisk_log_size":52108,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":116,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":17615612,"kb_used_data":10120144,"kb_used_omap":17,"kb_used_meta":7495342,"kb_avail":1013581060,"statfs":{"total":1055945392128,"available":1037907005440,"internally_reserved":0,"allocated":10363027456,"data_stored":10354532339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17546,"internal_metadata":7675231094},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1006436352,"num_objects":15357,"num_object_clones":0,"num_object_copies":46071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15357,"num_write_kb":982848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003734"},"pg_stats":[{"pgid":"1.14","version":"94'139","reported_seq":882,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T13:19:54.924679+0000","last_change":"2024-05-15T13:19:54.924720+0000","last_active":"2024-05-15T13:19:54.924679+0000","last_peered":"2024-05-15T13:19:54.924679+0000","last_clean":"2024-05-15T13:19:54.924679+0000","last_became_active":"2024-05-15T13:19:41.190796+0000","last_became_peered":"2024-05-15T13:19:41.190796+0000","last_unstale":"2024-05-15T13:19:54.924679+0000","last_undegraded":"2024-05-15T13:19:54.924679+0000","last_fullsized":"2024-05-15T13:19:54.924679+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:54.924609+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:19:54.924609+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:21:24.432915+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"767'3205","reported_seq":3271,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-05-15T13:20:03.545848+0000","last_change":"2024-05-15T13:18:51.769954+0000","last_active":"2024-05-15T13:20:03.545848+0000","last_peered":"2024-05-15T13:20:03.545848+0000","last_clean":"2024-05-15T13:20:03.545848+0000","last_became_active":"2024-05-15T13:18:51.769627+0000","last_became_peered":"2024-05-15T13:18:51.769627+0000","last_unstale":"2024-05-15T13:20:03.545848+0000","last_undegraded":"2024-05-15T13:20:03.545848+0000","last_fullsized":"2024-05-15T13:20:03.545848+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":3205,"log_dups_size":0,"ondisk_log_size":3205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:11.531828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210042880,"num_objects":3205,"num_object_clones":0,"num_object_copies":9615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3205,"num_write_kb":205120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,0],"acting":[5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":1442,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-05-15T13:20:02.695634+0000","last_change":"2024-05-15T13:19:11.513679+0000","last_active":"2024-05-15T13:20:02.695634+0000","last_peered":"2024-05-15T13:20:02.695634+0000","last_clean":"2024-05-15T13:20:02.695634+0000","last_became_active":"2024-05-15T13:13:11.361366+0000","last_became_peered":"2024-05-15T13:13:11.361366+0000","last_unstale":"2024-05-15T13:20:02.695634+0000","last_undegraded":"2024-05-15T13:20:02.695634+0000","last_fullsized":"2024-05-15T13:20:02.695634+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:11.513579+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:11.513579+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:30.445199+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.d","version":"767'3115","reported_seq":4226,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-05-15T13:20:03.037931+0000","last_change":"2024-05-15T13:19:42.196170+0000","last_active":"2024-05-15T13:20:03.037931+0000","last_peered":"2024-05-15T13:20:03.037931+0000","last_clean":"2024-05-15T13:20:03.037931+0000","last_became_active":"2024-05-15T13:19:42.194276+0000","last_became_peered":"2024-05-15T13:19:42.194276+0000","last_unstale":"2024-05-15T13:20:03.037931+0000","last_undegraded":"2024-05-15T13:20:03.037931+0000","last_fullsized":"2024-05-15T13:20:03.037931+0000","mapping_epoch":762,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":763,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":3115,"log_dups_size":0,"ondisk_log_size":3115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:05.558049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204144640,"num_objects":3115,"num_object_clones":0,"num_object_copies":9345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3115,"num_write_kb":199360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":517,"num_bytes_recovered":33882112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"94'139","reported_seq":1460,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-05-15T13:20:02.697852+0000","last_change":"2024-05-15T13:19:41.190272+0000","last_active":"2024-05-15T13:20:02.697852+0000","last_peered":"2024-05-15T13:20:02.697852+0000","last_clean":"2024-05-15T13:20:02.697852+0000","last_became_active":"2024-05-15T13:19:41.189861+0000","last_became_peered":"2024-05-15T13:19:41.189861+0000","last_unstale":"2024-05-15T13:20:02.697852+0000","last_undegraded":"2024-05-15T13:20:02.697852+0000","last_fullsized":"2024-05-15T13:20:02.697852+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:11.007920+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:19:11.007920+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:26.370486+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"766'2886","reported_seq":2950,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T13:20:00.578001+0000","last_change":"2024-05-15T13:18:51.771623+0000","last_active":"2024-05-15T13:20:00.578001+0000","last_peered":"2024-05-15T13:20:00.578001+0000","last_clean":"2024-05-15T13:20:00.578001+0000","last_became_active":"2024-05-15T13:18:51.771276+0000","last_became_peered":"2024-05-15T13:18:51.771276+0000","last_unstale":"2024-05-15T13:20:00.578001+0000","last_undegraded":"2024-05-15T13:20:00.578001+0000","last_fullsized":"2024-05-15T13:20:00.578001+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":2886,"log_dups_size":0,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:16.197016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189136896,"num_objects":2886,"num_object_clones":0,"num_object_copies":8658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":184704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"94'139","reported_seq":1203,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T13:19:56.237599+0000","last_change":"2024-05-15T13:19:56.237641+0000","last_active":"2024-05-15T13:19:56.237599+0000","last_peered":"2024-05-15T13:19:56.237599+0000","last_clean":"2024-05-15T13:19:56.237599+0000","last_became_active":"2024-05-15T13:14:54.577458+0000","last_became_peered":"2024-05-15T13:14:54.577458+0000","last_unstale":"2024-05-15T13:19:56.237599+0000","last_undegraded":"2024-05-15T13:19:56.237599+0000","last_fullsized":"2024-05-15T13:19:56.237599+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:56.237508+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:18:24.970285+0000","last_clean_scrub_stamp":"2024-05-15T13:19:56.237508+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:21:11.319936+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"767'3133","reported_seq":3199,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-05-15T13:20:03.054810+0000","last_change":"2024-05-15T13:18:51.825618+0000","last_active":"2024-05-15T13:20:03.054810+0000","last_peered":"2024-05-15T13:20:03.054810+0000","last_clean":"2024-05-15T13:20:03.054810+0000","last_became_active":"2024-05-15T13:18:51.825302+0000","last_became_peered":"2024-05-15T13:18:51.825302+0000","last_unstale":"2024-05-15T13:20:03.054810+0000","last_undegraded":"2024-05-15T13:20:03.054810+0000","last_fullsized":"2024-05-15T13:20:03.054810+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":3133,"log_dups_size":0,"ondisk_log_size":3133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:07.838359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205324288,"num_objects":3133,"num_object_clones":0,"num_object_copies":9399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3133,"num_write_kb":200512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"94'139","reported_seq":1442,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T13:19:49.186303+0000","last_change":"2024-05-15T13:19:04.701370+0000","last_active":"2024-05-15T13:19:49.186303+0000","last_peered":"2024-05-15T13:19:49.186303+0000","last_clean":"2024-05-15T13:19:49.186303+0000","last_became_active":"2024-05-15T13:13:11.373251+0000","last_became_peered":"2024-05-15T13:13:11.373251+0000","last_unstale":"2024-05-15T13:19:49.186303+0000","last_undegraded":"2024-05-15T13:19:49.186303+0000","last_fullsized":"2024-05-15T13:19:49.186303+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:04.701253+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:04.701253+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:20.202986+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.e","version":"768'3218","reported_seq":4293,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-05-15T13:20:04.416955+0000","last_change":"2024-05-15T13:19:42.195471+0000","last_active":"2024-05-15T13:20:04.416955+0000","last_peered":"2024-05-15T13:20:04.416955+0000","last_clean":"2024-05-15T13:20:04.416955+0000","last_became_active":"2024-05-15T13:19:42.193904+0000","last_became_peered":"2024-05-15T13:19:42.193904+0000","last_unstale":"2024-05-15T13:20:04.416955+0000","last_undegraded":"2024-05-15T13:20:04.416955+0000","last_fullsized":"2024-05-15T13:20:04.416955+0000","mapping_epoch":762,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":763,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":3218,"log_dups_size":0,"ondisk_log_size":3218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:14.638075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210894848,"num_objects":3218,"num_object_clones":0,"num_object_copies":9654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3218,"num_write_kb":205952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":498,"num_bytes_recovered":32636928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"94'139","reported_seq":1212,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-05-15T13:20:04.054909+0000","last_change":"2024-05-15T13:19:06.027430+0000","last_active":"2024-05-15T13:20:04.054909+0000","last_peered":"2024-05-15T13:20:04.054909+0000","last_clean":"2024-05-15T13:20:04.054909+0000","last_became_active":"2024-05-15T13:14:11.522490+0000","last_became_peered":"2024-05-15T13:14:11.522490+0000","last_unstale":"2024-05-15T13:20:04.054909+0000","last_undegraded":"2024-05-15T13:20:04.054909+0000","last_fullsized":"2024-05-15T13:20:04.054909+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:06.027331+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:06.027331+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:29.041654+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"767'3174","reported_seq":3240,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-05-15T13:20:03.533265+0000","last_change":"2024-05-15T13:18:51.770676+0000","last_active":"2024-05-15T13:20:03.533265+0000","last_peered":"2024-05-15T13:20:03.533265+0000","last_clean":"2024-05-15T13:20:03.533265+0000","last_became_active":"2024-05-15T13:18:51.770286+0000","last_became_peered":"2024-05-15T13:18:51.770286+0000","last_unstale":"2024-05-15T13:20:03.533265+0000","last_undegraded":"2024-05-15T13:20:03.533265+0000","last_fullsized":"2024-05-15T13:20:03.533265+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":3174,"log_dups_size":0,"ondisk_log_size":3174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:05.255233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208011264,"num_objects":3174,"num_object_clones":0,"num_object_copies":9522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3174,"num_write_kb":203136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"94'139","reported_seq":1220,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T13:19:49.186750+0000","last_change":"2024-05-15T13:19:12.802916+0000","last_active":"2024-05-15T13:19:49.186750+0000","last_peered":"2024-05-15T13:19:49.186750+0000","last_clean":"2024-05-15T13:19:49.186750+0000","last_became_active":"2024-05-15T13:15:43.614970+0000","last_became_peered":"2024-05-15T13:15:43.614970+0000","last_unstale":"2024-05-15T13:19:49.186750+0000","last_undegraded":"2024-05-15T13:19:49.186750+0000","last_fullsized":"2024-05-15T13:19:49.186750+0000","mapping_epoch":678,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:12.802814+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:12.802814+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:14.554397+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,0],"acting":[9,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.b","version":"766'2851","reported_seq":2914,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T13:20:00.293259+0000","last_change":"2024-05-15T13:18:51.768048+0000","last_active":"2024-05-15T13:20:00.293259+0000","last_peered":"2024-05-15T13:20:00.293259+0000","last_clean":"2024-05-15T13:20:00.293259+0000","last_became_active":"2024-05-15T13:18:51.767604+0000","last_became_peered":"2024-05-15T13:18:51.767604+0000","last_unstale":"2024-05-15T13:20:00.293259+0000","last_undegraded":"2024-05-15T13:20:00.293259+0000","last_fullsized":"2024-05-15T13:20:00.293259+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":2851,"log_dups_size":0,"ondisk_log_size":2851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:01.870097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186843136,"num_objects":2851,"num_object_clones":0,"num_object_copies":8553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2851,"num_write_kb":182464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"94'139","reported_seq":1456,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T13:19:49.185306+0000","last_change":"2024-05-15T13:19:41.189948+0000","last_active":"2024-05-15T13:19:49.185306+0000","last_peered":"2024-05-15T13:19:49.185306+0000","last_clean":"2024-05-15T13:19:49.185306+0000","last_became_active":"2024-05-15T13:19:41.189488+0000","last_became_peered":"2024-05-15T13:19:41.189488+0000","last_unstale":"2024-05-15T13:19:49.185306+0000","last_undegraded":"2024-05-15T13:19:49.185306+0000","last_fullsized":"2024-05-15T13:19:49.185306+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:03.412703+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:03.412703+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:29.583917+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"766'2864","reported_seq":2928,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T13:20:00.305869+0000","last_change":"2024-05-15T13:18:51.823543+0000","last_active":"2024-05-15T13:20:00.305869+0000","last_peered":"2024-05-15T13:20:00.305869+0000","last_clean":"2024-05-15T13:20:00.305869+0000","last_became_active":"2024-05-15T13:18:51.823137+0000","last_became_peered":"2024-05-15T13:18:51.823137+0000","last_unstale":"2024-05-15T13:20:00.305869+0000","last_undegraded":"2024-05-15T13:20:00.305869+0000","last_fullsized":"2024-05-15T13:20:00.305869+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":2864,"log_dups_size":0,"ondisk_log_size":2864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:12.114852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187695104,"num_objects":2864,"num_object_clones":0,"num_object_copies":8592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2864,"num_write_kb":183296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"94'139","reported_seq":1469,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-05-15T13:20:02.696026+0000","last_change":"2024-05-15T13:19:17.678528+0000","last_active":"2024-05-15T13:20:02.696026+0000","last_peered":"2024-05-15T13:20:02.696026+0000","last_clean":"2024-05-15T13:20:02.696026+0000","last_became_active":"2024-05-15T13:14:54.570004+0000","last_became_peered":"2024-05-15T13:14:54.570004+0000","last_unstale":"2024-05-15T13:20:02.696026+0000","last_undegraded":"2024-05-15T13:20:02.696026+0000","last_fullsized":"2024-05-15T13:20:02.696026+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:17.678421+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:17.678421+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:44.430362+0000","scrub_duration":174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"767'3078","reported_seq":3144,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-05-15T13:20:02.725686+0000","last_change":"2024-05-15T13:18:51.824173+0000","last_active":"2024-05-15T13:20:02.725686+0000","last_peered":"2024-05-15T13:20:02.725686+0000","last_clean":"2024-05-15T13:20:02.725686+0000","last_became_active":"2024-05-15T13:18:51.823777+0000","last_became_peered":"2024-05-15T13:18:51.823777+0000","last_unstale":"2024-05-15T13:20:02.725686+0000","last_undegraded":"2024-05-15T13:20:02.725686+0000","last_fullsized":"2024-05-15T13:20:02.725686+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":3078,"log_dups_size":0,"ondisk_log_size":3078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:20.118351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201719808,"num_objects":3078,"num_object_clones":0,"num_object_copies":9234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3078,"num_write_kb":196992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"94'139","reported_seq":1446,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-05-15T13:20:02.699615+0000","last_change":"2024-05-15T13:19:08.450681+0000","last_active":"2024-05-15T13:20:02.699615+0000","last_peered":"2024-05-15T13:20:02.699615+0000","last_clean":"2024-05-15T13:20:02.699615+0000","last_became_active":"2024-05-15T13:06:46.426908+0000","last_became_peered":"2024-05-15T13:06:46.426908+0000","last_unstale":"2024-05-15T13:20:02.699615+0000","last_undegraded":"2024-05-15T13:20:02.699615+0000","last_fullsized":"2024-05-15T13:20:02.699615+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:08.450567+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:08.450567+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:35.499808+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.0","version":"767'3135","reported_seq":3199,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-05-15T13:20:02.922383+0000","last_change":"2024-05-15T13:18:51.770583+0000","last_active":"2024-05-15T13:20:02.922383+0000","last_peered":"2024-05-15T13:20:02.922383+0000","last_clean":"2024-05-15T13:20:02.922383+0000","last_became_active":"2024-05-15T13:18:51.770274+0000","last_became_peered":"2024-05-15T13:18:51.770274+0000","last_unstale":"2024-05-15T13:20:02.922383+0000","last_undegraded":"2024-05-15T13:20:02.922383+0000","last_fullsized":"2024-05-15T13:20:02.922383+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":3135,"log_dups_size":0,"ondisk_log_size":3135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:11.283287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205455360,"num_objects":3135,"num_object_clones":0,"num_object_copies":9405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3135,"num_write_kb":200640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"94'139","reported_seq":818,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T13:19:49.185281+0000","last_change":"2024-05-15T13:19:48.425693+0000","last_active":"2024-05-15T13:19:49.185281+0000","last_peered":"2024-05-15T13:19:49.185281+0000","last_clean":"2024-05-15T13:19:49.185281+0000","last_became_active":"2024-05-15T13:19:41.191244+0000","last_became_peered":"2024-05-15T13:19:41.191244+0000","last_unstale":"2024-05-15T13:19:49.185281+0000","last_undegraded":"2024-05-15T13:19:49.185281+0000","last_fullsized":"2024-05-15T13:19:49.185281+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:48.425609+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:48.425609+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:21:16.556596+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3","version":"767'3041","reported_seq":3107,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-05-15T13:20:03.039541+0000","last_change":"2024-05-15T13:18:51.769282+0000","last_active":"2024-05-15T13:20:03.039541+0000","last_peered":"2024-05-15T13:20:03.039541+0000","last_clean":"2024-05-15T13:20:03.039541+0000","last_became_active":"2024-05-15T13:18:51.768837+0000","last_became_peered":"2024-05-15T13:18:51.768837+0000","last_unstale":"2024-05-15T13:20:03.039541+0000","last_undegraded":"2024-05-15T13:20:03.039541+0000","last_fullsized":"2024-05-15T13:20:03.039541+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":3041,"log_dups_size":0,"ondisk_log_size":3041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:14.139438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199294976,"num_objects":3041,"num_object_clones":0,"num_object_copies":9123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3041,"num_write_kb":194624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"94'139","reported_seq":1214,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T13:19:49.186367+0000","last_change":"2024-05-15T13:19:11.717256+0000","last_active":"2024-05-15T13:19:49.186367+0000","last_peered":"2024-05-15T13:19:49.186367+0000","last_clean":"2024-05-15T13:19:49.186367+0000","last_became_active":"2024-05-15T13:15:43.611913+0000","last_became_peered":"2024-05-15T13:15:43.611913+0000","last_unstale":"2024-05-15T13:19:49.186367+0000","last_undegraded":"2024-05-15T13:19:49.186367+0000","last_fullsized":"2024-05-15T13:19:49.186367+0000","mapping_epoch":678,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:11.717153+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:11.717153+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:36.104184+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.7","version":"767'3153","reported_seq":3254,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-05-15T13:20:02.736439+0000","last_change":"2024-05-15T13:19:58.014629+0000","last_active":"2024-05-15T13:20:02.736439+0000","last_peered":"2024-05-15T13:20:02.736439+0000","last_clean":"2024-05-15T13:20:02.736439+0000","last_became_active":"2024-05-15T13:18:51.767342+0000","last_became_peered":"2024-05-15T13:18:51.767342+0000","last_unstale":"2024-05-15T13:20:02.736439+0000","last_undegraded":"2024-05-15T13:20:02.736439+0000","last_fullsized":"2024-05-15T13:20:02.736439+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'2700","last_scrub_stamp":"2024-05-15T13:19:58.014414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:19:58.014414+0000","objects_scrubbed":2675,"log_size":3153,"log_dups_size":0,"ondisk_log_size":3153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:21:12.282602+0000","scrub_duration":605,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206635008,"num_objects":3153,"num_object_clones":0,"num_object_copies":9459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3153,"num_write_kb":201792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"94'139","reported_seq":821,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T13:19:49.185566+0000","last_change":"2024-05-15T13:19:47.359917+0000","last_active":"2024-05-15T13:19:49.185566+0000","last_peered":"2024-05-15T13:19:49.185566+0000","last_clean":"2024-05-15T13:19:49.185566+0000","last_became_active":"2024-05-15T13:19:41.189588+0000","last_became_peered":"2024-05-15T13:19:41.189588+0000","last_unstale":"2024-05-15T13:19:49.185566+0000","last_undegraded":"2024-05-15T13:19:49.185566+0000","last_fullsized":"2024-05-15T13:19:49.185566+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:47.359837+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:47.359837+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:21:11.223532+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.6","version":"767'3123","reported_seq":4309,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-05-15T13:20:03.049664+0000","last_change":"2024-05-15T13:20:02.624362+0000","last_active":"2024-05-15T13:20:03.049664+0000","last_peered":"2024-05-15T13:20:03.049664+0000","last_clean":"2024-05-15T13:20:03.049664+0000","last_became_active":"2024-05-15T13:19:42.194427+0000","last_became_peered":"2024-05-15T13:19:42.194427+0000","last_unstale":"2024-05-15T13:20:03.049664+0000","last_undegraded":"2024-05-15T13:20:03.049664+0000","last_fullsized":"2024-05-15T13:20:03.049664+0000","mapping_epoch":762,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":763,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'3075","last_scrub_stamp":"2024-05-15T13:20:02.624105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:20:02.624105+0000","objects_scrubbed":3056,"log_size":3123,"log_dups_size":0,"ondisk_log_size":3123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:21:17.804108+0000","scrub_duration":653,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204668928,"num_objects":3123,"num_object_clones":0,"num_object_copies":9369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3123,"num_write_kb":199872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":535,"num_bytes_recovered":35061760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7],"acting":[11,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"94'139","reported_seq":1211,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-05-15T13:20:04.055110+0000","last_change":"2024-05-15T13:19:03.984125+0000","last_active":"2024-05-15T13:20:04.055110+0000","last_peered":"2024-05-15T13:20:04.055110+0000","last_clean":"2024-05-15T13:20:04.055110+0000","last_became_active":"2024-05-15T13:13:16.472318+0000","last_became_peered":"2024-05-15T13:13:16.472318+0000","last_unstale":"2024-05-15T13:20:04.055110+0000","last_undegraded":"2024-05-15T13:20:04.055110+0000","last_fullsized":"2024-05-15T13:20:04.055110+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:03.983993+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:03.983993+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:24.592184+0000","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"767'3075","reported_seq":3141,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-05-15T13:20:02.731518+0000","last_change":"2024-05-15T13:18:51.767865+0000","last_active":"2024-05-15T13:20:02.731518+0000","last_peered":"2024-05-15T13:20:02.731518+0000","last_clean":"2024-05-15T13:20:02.731518+0000","last_became_active":"2024-05-15T13:18:51.767454+0000","last_became_peered":"2024-05-15T13:18:51.767454+0000","last_unstale":"2024-05-15T13:20:02.731518+0000","last_undegraded":"2024-05-15T13:20:02.731518+0000","last_fullsized":"2024-05-15T13:20:02.731518+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":3075,"log_dups_size":0,"ondisk_log_size":3075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:11.694078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201523200,"num_objects":3075,"num_object_clones":0,"num_object_copies":9225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3075,"num_write_kb":196800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,11],"acting":[1,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"94'139","reported_seq":869,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-05-15T13:20:02.702408+0000","last_change":"2024-05-15T13:19:11.928809+0000","last_active":"2024-05-15T13:20:02.702408+0000","last_peered":"2024-05-15T13:20:02.702408+0000","last_clean":"2024-05-15T13:20:02.702408+0000","last_became_active":"2024-05-15T13:13:16.476972+0000","last_became_peered":"2024-05-15T13:13:16.476972+0000","last_unstale":"2024-05-15T13:20:02.702408+0000","last_undegraded":"2024-05-15T13:20:02.702408+0000","last_fullsized":"2024-05-15T13:20:02.702408+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:11.928703+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:19:11.928703+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:14.003218+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"767'3112","reported_seq":3215,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-05-15T13:20:03.543353+0000","last_change":"2024-05-15T13:20:01.555388+0000","last_active":"2024-05-15T13:20:03.543353+0000","last_peered":"2024-05-15T13:20:03.543353+0000","last_clean":"2024-05-15T13:20:03.543353+0000","last_became_active":"2024-05-15T13:18:51.769098+0000","last_became_peered":"2024-05-15T13:18:51.769098+0000","last_unstale":"2024-05-15T13:20:03.543353+0000","last_undegraded":"2024-05-15T13:20:03.543353+0000","last_fullsized":"2024-05-15T13:20:03.543353+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'2938","last_scrub_stamp":"2024-05-15T13:20:01.555250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:20:01.555250+0000","objects_scrubbed":2917,"log_size":3112,"log_dups_size":0,"ondisk_log_size":3112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:21:29.698823+0000","scrub_duration":479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203948032,"num_objects":3112,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3112,"num_write_kb":199168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"94'139","reported_seq":858,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T13:19:49.185553+0000","last_change":"2024-05-15T13:19:41.189851+0000","last_active":"2024-05-15T13:19:49.185553+0000","last_peered":"2024-05-15T13:19:49.185553+0000","last_clean":"2024-05-15T13:19:49.185553+0000","last_became_active":"2024-05-15T13:19:41.189366+0000","last_became_peered":"2024-05-15T13:19:41.189366+0000","last_unstale":"2024-05-15T13:19:49.185553+0000","last_undegraded":"2024-05-15T13:19:49.185553+0000","last_fullsized":"2024-05-15T13:19:49.185553+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:14.630558+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:14.630558+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:15.006880+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"766'3026","reported_seq":3090,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T13:20:01.905182+0000","last_change":"2024-05-15T13:18:51.770698+0000","last_active":"2024-05-15T13:20:01.905182+0000","last_peered":"2024-05-15T13:20:01.905182+0000","last_clean":"2024-05-15T13:20:01.905182+0000","last_became_active":"2024-05-15T13:18:51.770335+0000","last_became_peered":"2024-05-15T13:18:51.770335+0000","last_unstale":"2024-05-15T13:20:01.905182+0000","last_undegraded":"2024-05-15T13:20:01.905182+0000","last_fullsized":"2024-05-15T13:20:01.905182+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":739,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:18:50.745722+0000","last_clean_scrub_stamp":"2024-05-15T13:18:50.745722+0000","objects_scrubbed":0,"log_size":3026,"log_dups_size":0,"ondisk_log_size":3026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:09.106204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198311936,"num_objects":3026,"num_object_clones":0,"num_object_copies":9078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3026,"num_write_kb":193664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,9],"acting":[10,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"94'139","reported_seq":1470,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-05-15T13:20:02.697646+0000","last_change":"2024-05-15T13:19:10.071697+0000","last_active":"2024-05-15T13:20:02.697646+0000","last_peered":"2024-05-15T13:20:02.697646+0000","last_clean":"2024-05-15T13:20:02.697646+0000","last_became_active":"2024-05-15T13:13:16.484252+0000","last_became_peered":"2024-05-15T13:13:16.484252+0000","last_unstale":"2024-05-15T13:20:02.697646+0000","last_undegraded":"2024-05-15T13:20:02.697646+0000","last_fullsized":"2024-05-15T13:20:02.697646+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:10.071555+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:10.071555+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:25.239137+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"94'139","reported_seq":1460,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-05-15T13:20:02.699562+0000","last_change":"2024-05-15T13:19:41.191437+0000","last_active":"2024-05-15T13:20:02.699562+0000","last_peered":"2024-05-15T13:20:02.699562+0000","last_clean":"2024-05-15T13:20:02.699562+0000","last_became_active":"2024-05-15T13:19:41.190923+0000","last_became_peered":"2024-05-15T13:19:41.190923+0000","last_unstale":"2024-05-15T13:20:02.699562+0000","last_undegraded":"2024-05-15T13:20:02.699562+0000","last_fullsized":"2024-05-15T13:20:02.699562+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:07.404921+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:07.404921+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:07.636921+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"94'139","reported_seq":1227,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-05-15T13:20:03.733539+0000","last_change":"2024-05-15T13:19:47.571937+0000","last_active":"2024-05-15T13:20:03.733539+0000","last_peered":"2024-05-15T13:20:03.733539+0000","last_clean":"2024-05-15T13:20:03.733539+0000","last_became_active":"2024-05-15T13:13:16.476744+0000","last_became_peered":"2024-05-15T13:13:16.476744+0000","last_unstale":"2024-05-15T13:20:03.733539+0000","last_undegraded":"2024-05-15T13:20:03.733539+0000","last_fullsized":"2024-05-15T13:20:03.733539+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:47.571790+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:47.571790+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:53.109347+0000","scrub_duration":251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"94'139","reported_seq":1475,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-05-15T13:20:02.699554+0000","last_change":"2024-05-15T13:19:17.283425+0000","last_active":"2024-05-15T13:20:02.699554+0000","last_peered":"2024-05-15T13:20:02.699554+0000","last_clean":"2024-05-15T13:20:02.699554+0000","last_became_active":"2024-05-15T13:13:49.926687+0000","last_became_peered":"2024-05-15T13:13:49.926687+0000","last_unstale":"2024-05-15T13:20:02.699554+0000","last_undegraded":"2024-05-15T13:20:02.699554+0000","last_fullsized":"2024-05-15T13:20:02.699554+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:17.283279+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:18:06.340555+0000","last_clean_scrub_stamp":"2024-05-15T13:19:17.283279+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:30.684201+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"94'139","reported_seq":1467,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-05-15T13:20:02.695903+0000","last_change":"2024-05-15T13:19:15.681301+0000","last_active":"2024-05-15T13:20:02.695903+0000","last_peered":"2024-05-15T13:20:02.695903+0000","last_clean":"2024-05-15T13:20:02.695903+0000","last_became_active":"2024-05-15T13:13:16.478488+0000","last_became_peered":"2024-05-15T13:13:16.478488+0000","last_unstale":"2024-05-15T13:20:02.695903+0000","last_undegraded":"2024-05-15T13:20:02.695903+0000","last_fullsized":"2024-05-15T13:20:02.695903+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:19:15.681199+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:19:15.681199+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:20:17.078840+0000","scrub_duration":178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3223650304,"num_objects":49189,"num_object_clones":0,"num_object_copies":147567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49189,"num_write_kb":3148096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1550,"num_bytes_recovered":101580800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9580838912,"data_stored":9580838912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49189,"ondisk_log_size":49189,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057056,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2313264,"kb_used_data":1632356,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":91431888,"statfs":{"total":95995035648,"available":93626253312,"internally_reserved":0,"allocated":1671532544,"data_stored":1670758388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":697235910},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351396,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1954208,"kb_used_data":1246936,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":91790944,"statfs":{"total":95995035648,"available":93993926656,"internally_reserved":0,"allocated":1276862464,"data_stored":1276092773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":724236736},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514921,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2105952,"kb_used_data":1258456,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":91639200,"statfs":{"total":95995035648,"available":93838540800,"internally_reserved":0,"allocated":1288658944,"data_stored":1287889253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":867826112},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874670,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":979808,"kb_used_data":529368,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":92765344,"statfs":{"total":95995035648,"available":94991712256,"internally_reserved":0,"allocated":542072832,"data_stored":541303141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169010,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":747104,"kb_used_data":549336,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":92998048,"statfs":{"total":95995035648,"available":95230001152,"internally_reserved":0,"allocated":562520064,"data_stored":561750373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":202504650},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430650,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":741604,"kb_used_data":2392,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":93003548,"statfs":{"total":95995035648,"available":95235633152,"internally_reserved":0,"allocated":2449408,"data_stored":1679717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":756939206},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575314,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2075104,"kb_used_data":1173464,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":91670048,"statfs":{"total":95995035648,"available":93870129152,"internally_reserved":0,"allocated":1201627136,"data_stored":1200857445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":923269573},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1418688,"kb_used_data":621168,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":92326464,"statfs":{"total":95995035648,"available":94542299136,"internally_reserved":0,"allocated":636076032,"data_stored":635297411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":816642501},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411791,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1671464,"kb_used_data":1098848,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92073688,"statfs":{"total":95995035648,"available":94283456512,"internally_reserved":0,"allocated":1125220352,"data_stored":1124447109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150152,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1529704,"kb_used_data":587808,"kb_used_omap":1,"kb_used_meta":941886,"kb_avail":92215448,"statfs":{"total":95995035648,"available":94428618752,"internally_reserved":0,"allocated":601915392,"data_stored":601142149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":964491719},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986628,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692287,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2078712,"kb_used_data":1420012,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":91666440,"statfs":{"total":95995035648,"available":93866434560,"internally_reserved":0,"allocated":1454092288,"data_stored":1453314580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":636616704,"data_stored":636616704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1213595648,"data_stored":1213595648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1010696192,"data_stored":1010696192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":613023744,"data_stored":613023744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1034485760,"data_stored":1034485760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":571473920,"data_stored":571473920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":561774592,"data_stored":561774592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1322975232,"data_stored":1322975232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1187971072,"data_stored":1187971072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1428226048,"data_stored":1428226048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:20:05.973 INFO:tasks.thrashosds.thrasher:Setting 8.d to [0, 5, 6] 2024-05-15T13:20:05.973 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '8.d', '0', '5', '6'] 2024-05-15T13:20:05.973 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pg-upmap 8.d 0 5 6 2024-05-15T13:20:06.001 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 57 15 54305 54290 59.5087 99.375 0.0128969 0.0167992 2024-05-15T13:20:06.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:05 smithi177 ceph-mon[44303]: pgmap v1980: 37 pgs: 37 active+clean; 3.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-05-15T13:20:06.587 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:20:06.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:06.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:20:06.694 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:20:06.724 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:06.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:06 smithi027 ceph-mon[39795]: from='client.26353 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:20:06.779 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:06 smithi027 ceph-mon[39795]: 8.8 scrub starts 2024-05-15T13:20:06.873 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:20:06.876 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:06 smithi120 ceph-mon[43961]: from='client.26353 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:20:06.876 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:06 smithi120 ceph-mon[43961]: 8.8 scrub starts 2024-05-15T13:20:06.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:06.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:20:06.982 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:20:07.001 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 58 15 55851 55836 60.1483 96.625 0.0093187 0.0166209 2024-05-15T13:20:07.013 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:07.034 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:20:07.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:07.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:20:07.111 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:20:07.139 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:20:07.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:06 smithi177 ceph-mon[44303]: from='client.26353 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:20:07.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:06 smithi177 ceph-mon[44303]: 8.8 scrub starts 2024-05-15T13:20:07.213 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:07.252 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:20:07.252 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:20:08.001 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 59 16 57378 57362 60.7451 95.375 0.0114316 0.0164574 2024-05-15T13:20:08.026 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:07 smithi120 ceph-mon[43961]: pgmap v1981: 37 pgs: 37 active+clean; 3.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-05-15T13:20:08.026 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:07 smithi120 ceph-mon[43961]: 8.8 scrub ok 2024-05-15T13:20:08.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:07 smithi027 ceph-mon[39795]: pgmap v1981: 37 pgs: 37 active+clean; 3.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-05-15T13:20:08.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:07 smithi027 ceph-mon[39795]: 8.8 scrub ok 2024-05-15T13:20:08.059 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-05-15T13:20:08.170 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:20:08.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:08.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:20:08.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:07 smithi177 ceph-mon[44303]: pgmap v1981: 37 pgs: 37 active+clean; 3.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-05-15T13:20:08.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:07 smithi177 ceph-mon[44303]: 8.8 scrub ok 2024-05-15T13:20:08.275 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:20:08.332 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:20:08.476 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:08.636 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:20:08.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:08.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:20:08.696 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:20:08.728 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:08 smithi027 ceph-mon[39795]: 1.10 scrub starts 2024-05-15T13:20:08.728 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:08 smithi027 ceph-mon[39795]: 1.10 scrub ok 2024-05-15T13:20:08.742 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:20:08.794 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:08.824 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:20:08.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:08.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:20:08.897 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:08 smithi120 ceph-mon[43961]: 1.10 scrub starts 2024-05-15T13:20:08.897 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:08 smithi120 ceph-mon[43961]: 1.10 scrub ok 2024-05-15T13:20:08.932 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:20:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:09.002 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T13:20:09.004824+0000 min lat: 0.00105016 max lat: 15.0048 avg lat: 0.0163802 2024-05-15T13:20:09.002 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:20:09.002 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 60 16 58626 58610 61.0325 78 0.00683895 0.0163802 2024-05-15T13:20:09.161 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:20:09.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:08 smithi177 ceph-mon[44303]: 1.10 scrub starts 2024-05-15T13:20:09.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:08 smithi177 ceph-mon[44303]: 1.10 scrub ok 2024-05-15T13:20:09.211 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:20:09.242 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:20:09.269 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:20:09.300 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:20:09.327 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:20:09.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:09.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:20:09.345 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:20:09.432 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:20:09.481 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:09.517 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:20:09.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:09.522 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:20:09.574 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:20:09.603 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:20:09.623 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:20:09.647 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:20:09.679 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:09.686 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:20:09.736 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:20:09.744 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:09 smithi177 ceph-mon[44303]: pgmap v1982: 37 pgs: 37 active+clean; 3.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-05-15T13:20:09.744 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:09 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:20:09.745 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:09 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:20:09.745 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:09 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2263557128' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:20:09.745 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:09 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:20:09.775 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:20:09.804 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:20:09.809 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:20:09.810 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:20:09.810 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:20:09.810 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:20:09.810 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:20:09.810 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:20:09.810 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:20:09.810 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:20:09.810 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:20:09.810 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:20:09.810 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:20:09.810 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:20:10.002 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 61 16 59651 59635 61.0821 64.0625 0.00861611 0.0163674 2024-05-15T13:20:10.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:09 smithi027 ceph-mon[39795]: pgmap v1982: 37 pgs: 37 active+clean; 3.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-05-15T13:20:10.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:09 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:20:10.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:09 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:20:10.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:09 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2263557128' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:20:10.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:09 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:20:10.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:09 smithi120 ceph-mon[43961]: pgmap v1982: 37 pgs: 37 active+clean; 3.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-05-15T13:20:10.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:09 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:20:10.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:09 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:20:10.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:09 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2263557128' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:20:10.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:09 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:20:10.509 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:20:10.509 INFO:teuthology.orchestra.run.smithi120.stdout: "size": 20, 2024-05-15T13:20:10.510 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 600, 2024-05-15T13:20:10.510 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [ 2024-05-15T13:20:10.510 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.510 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.25735.0:196508 6.2 6:4d78cb2f:::benchmark_data_smithi177_122278_object66137:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e554)", 2024-05-15T13:20:10.510 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:10:41.035802+0000", 2024-05-15T13:20:10.510 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 569.47391727199999, 2024-05-15T13:20:10.510 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.12579068900000001, 2024-05-15T13:20:10.510 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:10.510 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:10.510 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:20:10.510 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.25735", 2024-05-15T13:20:10.510 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3529446103", 2024-05-15T13:20:10.510 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 196508 2024-05-15T13:20:10.511 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.511 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:10.511 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.511 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:10.511 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.035802+0000", 2024-05-15T13:20:10.511 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.511 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.511 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.511 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:10.511 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.035802+0000", 2024-05-15T13:20:10.511 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.511 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.512 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.512 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:10.512 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.035804+0000", 2024-05-15T13:20:10.512 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3179999999999999e-06 2024-05-15T13:20:10.512 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.512 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.512 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:10.512 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.035809+0000", 2024-05-15T13:20:10.512 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1209999999999998e-06 2024-05-15T13:20:10.512 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.512 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.512 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:10.512 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.035821+0000", 2024-05-15T13:20:10.513 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2696e-05 2024-05-15T13:20:10.513 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.513 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.513 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:10.513 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.035830+0000", 2024-05-15T13:20:10.513 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.4139999999999994e-06 2024-05-15T13:20:10.513 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.513 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.513 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:10.513 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.035885+0000", 2024-05-15T13:20:10.513 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5578999999999998e-05 2024-05-15T13:20:10.513 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.513 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.514 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:10.514 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.036001+0000", 2024-05-15T13:20:10.514 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000115337 2024-05-15T13:20:10.514 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.514 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.514 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 2,3", 2024-05-15T13:20:10.514 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.036122+0000", 2024-05-15T13:20:10.514 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012173 2024-05-15T13:20:10.514 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.514 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.514 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:20:10.514 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.040431+0000", 2024-05-15T13:20:10.514 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.004308709 2024-05-15T13:20:10.515 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.515 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.515 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:20:10.515 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.046149+0000", 2024-05-15T13:20:10.515 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0057182650000000002 2024-05-15T13:20:10.515 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.515 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.515 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:20:10.515 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.161517+0000", 2024-05-15T13:20:10.515 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.115367764 2024-05-15T13:20:10.515 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.515 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.516 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:10.516 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.161559+0000", 2024-05-15T13:20:10.516 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2145999999999999e-05 2024-05-15T13:20:10.516 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.516 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.516 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:10.516 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:10:41.161593+0000", 2024-05-15T13:20:10.516 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.3609999999999998e-05 2024-05-15T13:20:10.516 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.516 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:10.516 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.516 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.516 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.517 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.25735.0:248245 6.2 e574/571 6:4ba12033:::benchmark_data_smithi177_122278_object117874:head v 574'15418, mlcod=573'15416)", 2024-05-15T13:20:10.517 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:46.607600+0000", 2024-05-15T13:20:10.517 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 503.90211898500002, 2024-05-15T13:20:10.517 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.163291408, 2024-05-15T13:20:10.517 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:10.517 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:10.517 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:10.517 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.517 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:10.517 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.607600+0000", 2024-05-15T13:20:10.517 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.517 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.517 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.518 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:10.518 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.607600+0000", 2024-05-15T13:20:10.518 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.518 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.518 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.518 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:10.518 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.607601+0000", 2024-05-15T13:20:10.518 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7499999999999999e-07 2024-05-15T13:20:10.518 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.518 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.518 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:10.518 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.607607+0000", 2024-05-15T13:20:10.518 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6459999999999998e-06 2024-05-15T13:20:10.519 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.519 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.519 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:10.519 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.607616+0000", 2024-05-15T13:20:10.519 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8230000000000001e-06 2024-05-15T13:20:10.519 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.519 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.519 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:10.519 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.607621+0000", 2024-05-15T13:20:10.519 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3040000000000004e-06 2024-05-15T13:20:10.519 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.519 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.519 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:10.519 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.607660+0000", 2024-05-15T13:20:10.520 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.9059e-05 2024-05-15T13:20:10.520 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.520 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.520 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:10.520 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.607675+0000", 2024-05-15T13:20:10.520 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5393999999999999e-05 2024-05-15T13:20:10.520 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.520 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.520 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:10.520 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.770845+0000", 2024-05-15T13:20:10.520 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.16316941600000001 2024-05-15T13:20:10.520 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.520 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.521 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:10.521 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.770892+0000", 2024-05-15T13:20:10.521 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6990999999999998e-05 2024-05-15T13:20:10.521 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.521 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:10.521 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.521 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.521 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.521 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.25735.0:248250 6.2 e574/571 6:4170a72f:::benchmark_data_smithi177_122278_object117879:head v 574'15419, mlcod=573'15416)", 2024-05-15T13:20:10.521 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:46.609313+0000", 2024-05-15T13:20:10.521 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 503.90040660900002, 2024-05-15T13:20:10.521 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.16161126000000001, 2024-05-15T13:20:10.521 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:10.521 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:10.522 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:10.522 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.522 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:10.522 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.609313+0000", 2024-05-15T13:20:10.522 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.522 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.522 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.522 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:10.522 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.609313+0000", 2024-05-15T13:20:10.522 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.522 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.522 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.522 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:10.523 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.609313+0000", 2024-05-15T13:20:10.523 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9699999999999996e-07 2024-05-15T13:20:10.523 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.523 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.523 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:10.523 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.609319+0000", 2024-05-15T13:20:10.523 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.976e-06 2024-05-15T13:20:10.523 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.523 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.523 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:10.523 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.609333+0000", 2024-05-15T13:20:10.523 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3105000000000001e-05 2024-05-15T13:20:10.524 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.524 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.524 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:10.524 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.609339+0000", 2024-05-15T13:20:10.524 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4529999999999999e-06 2024-05-15T13:20:10.524 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.524 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.524 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:10.524 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.611422+0000", 2024-05-15T13:20:10.524 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0020825980000000002 2024-05-15T13:20:10.524 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.524 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.524 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:10.525 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.611448+0000", 2024-05-15T13:20:10.525 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.6157e-05 2024-05-15T13:20:10.525 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.525 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.525 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:10.525 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.770895+0000", 2024-05-15T13:20:10.525 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.15944773300000001 2024-05-15T13:20:10.525 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.525 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.525 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:10.525 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.770924+0000", 2024-05-15T13:20:10.525 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.8640999999999999e-05 2024-05-15T13:20:10.525 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.526 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:10.526 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.526 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.526 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.526 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.25735.0:248257 6.2 e574/571 6:49f2ada0:::benchmark_data_smithi177_122278_object117886:head v 574'15420, mlcod=573'15416)", 2024-05-15T13:20:10.526 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:46.623199+0000", 2024-05-15T13:20:10.526 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 503.88652004099998, 2024-05-15T13:20:10.526 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.14777490400000001, 2024-05-15T13:20:10.526 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:10.526 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:10.526 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:10.527 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.527 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:10.527 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.623199+0000", 2024-05-15T13:20:10.527 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.527 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.527 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.527 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:10.527 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.623199+0000", 2024-05-15T13:20:10.527 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.527 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.527 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.527 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:10.527 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.623205+0000", 2024-05-15T13:20:10.527 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6319999999999996e-06 2024-05-15T13:20:10.528 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.528 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.528 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:10.528 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.623216+0000", 2024-05-15T13:20:10.528 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0730000000000001e-05 2024-05-15T13:20:10.528 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.528 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.528 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:10.528 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.623230+0000", 2024-05-15T13:20:10.528 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4622e-05 2024-05-15T13:20:10.528 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.528 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.528 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:10.528 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.623239+0000", 2024-05-15T13:20:10.528 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0289999999999994e-06 2024-05-15T13:20:10.529 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.529 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.529 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:10.529 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.623326+0000", 2024-05-15T13:20:10.529 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.6725999999999996e-05 2024-05-15T13:20:10.529 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.529 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.529 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:10.529 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.623343+0000", 2024-05-15T13:20:10.529 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6901999999999998e-05 2024-05-15T13:20:10.529 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.530 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.530 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:10.530 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.770947+0000", 2024-05-15T13:20:10.530 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.14760426200000001 2024-05-15T13:20:10.530 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.530 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.530 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:10.530 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.770974+0000", 2024-05-15T13:20:10.530 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.7001000000000001e-05 2024-05-15T13:20:10.530 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.530 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:10.530 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.530 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.531 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.531 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.25735.0:248324 6.7 e574/571 6:ea825df0:::benchmark_data_smithi177_122278_object117953:head v 574'15608, mlcod=573'15606)", 2024-05-15T13:20:10.531 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:46.710949+0000", 2024-05-15T13:20:10.531 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 503.79877044900002, 2024-05-15T13:20:10.531 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.12661940199999999, 2024-05-15T13:20:10.531 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:10.531 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:10.531 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:10.531 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.531 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:10.531 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.710949+0000", 2024-05-15T13:20:10.531 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.531 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.532 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.532 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:10.532 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.710949+0000", 2024-05-15T13:20:10.532 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.532 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.532 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.532 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:10.532 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.710950+0000", 2024-05-15T13:20:10.532 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.3200000000000003e-07 2024-05-15T13:20:10.532 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.532 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.532 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:10.533 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.710957+0000", 2024-05-15T13:20:10.533 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.584e-06 2024-05-15T13:20:10.533 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.533 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.533 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:10.533 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.710969+0000", 2024-05-15T13:20:10.533 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2758000000000001e-05 2024-05-15T13:20:10.533 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.533 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.533 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:10.533 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.710977+0000", 2024-05-15T13:20:10.534 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4510000000000003e-06 2024-05-15T13:20:10.534 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.534 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.534 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:10.534 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.711055+0000", 2024-05-15T13:20:10.534 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8183999999999995e-05 2024-05-15T13:20:10.534 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.534 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.534 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:10.534 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.711079+0000", 2024-05-15T13:20:10.534 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.4189000000000002e-05 2024-05-15T13:20:10.534 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.534 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.535 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:10.535 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.837519+0000", 2024-05-15T13:20:10.535 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.126439791 2024-05-15T13:20:10.535 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.535 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.535 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:10.535 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.837568+0000", 2024-05-15T13:20:10.535 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9512999999999998e-05 2024-05-15T13:20:10.535 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.535 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:10.535 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.535 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.535 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.536 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.25735.0:248328 6.e e574/571 6:7f01531a:::benchmark_data_smithi177_122278_object117957:head v 574'15449, mlcod=574'15447)", 2024-05-15T13:20:10.536 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:46.713975+0000", 2024-05-15T13:20:10.536 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 503.79574451799999, 2024-05-15T13:20:10.536 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.19530534099999999, 2024-05-15T13:20:10.536 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:10.536 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:10.536 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:10.536 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.536 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:10.536 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.713975+0000", 2024-05-15T13:20:10.536 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.536 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.536 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.537 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:10.537 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.713975+0000", 2024-05-15T13:20:10.537 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.537 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.537 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.537 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:10.537 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.713976+0000", 2024-05-15T13:20:10.537 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.062e-06 2024-05-15T13:20:10.537 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.537 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.537 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:10.537 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.713984+0000", 2024-05-15T13:20:10.537 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2360000000000004e-06 2024-05-15T13:20:10.538 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.538 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.538 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:10.538 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.713997+0000", 2024-05-15T13:20:10.538 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2744e-05 2024-05-15T13:20:10.538 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.538 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.538 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:10.538 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.714005+0000", 2024-05-15T13:20:10.538 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1189999999999997e-06 2024-05-15T13:20:10.538 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.538 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.538 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:10.539 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.714113+0000", 2024-05-15T13:20:10.539 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00010806699999999999 2024-05-15T13:20:10.539 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.539 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.539 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:10.539 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.714137+0000", 2024-05-15T13:20:10.539 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.3852000000000001e-05 2024-05-15T13:20:10.539 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.539 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.539 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:10.539 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.909233+0000", 2024-05-15T13:20:10.539 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.19509637099999999 2024-05-15T13:20:10.540 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.540 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.540 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:10.540 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.909280+0000", 2024-05-15T13:20:10.540 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.689e-05 2024-05-15T13:20:10.540 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.540 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:10.540 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.540 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.540 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.540 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.25735.0:248322 6.6 e574/571 6:6577cc70:::benchmark_data_smithi177_122278_object117951:head v 574'15438, mlcod=574'15436)", 2024-05-15T13:20:10.540 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:46.715909+0000", 2024-05-15T13:20:10.540 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 503.79381022400003, 2024-05-15T13:20:10.541 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.184760445, 2024-05-15T13:20:10.541 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:10.541 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:10.541 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:10.541 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.541 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:10.541 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.715909+0000", 2024-05-15T13:20:10.541 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.541 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.541 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.541 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:10.541 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.715909+0000", 2024-05-15T13:20:10.541 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.541 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.542 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.542 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:10.542 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.715910+0000", 2024-05-15T13:20:10.542 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7499999999999999e-07 2024-05-15T13:20:10.542 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.542 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.542 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:10.542 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.715918+0000", 2024-05-15T13:20:10.542 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2539999999999992e-06 2024-05-15T13:20:10.542 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.542 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.542 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:10.543 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.715930+0000", 2024-05-15T13:20:10.543 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1637999999999999e-05 2024-05-15T13:20:10.543 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.543 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.543 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:10.543 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.715938+0000", 2024-05-15T13:20:10.543 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9209999999999992e-06 2024-05-15T13:20:10.543 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.543 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.543 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:10.543 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.716170+0000", 2024-05-15T13:20:10.543 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00023183399999999999 2024-05-15T13:20:10.544 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.544 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.544 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:10.544 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.716199+0000", 2024-05-15T13:20:10.544 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.8819e-05 2024-05-15T13:20:10.544 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.544 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.544 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:10.544 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.900560+0000", 2024-05-15T13:20:10.544 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.184361517 2024-05-15T13:20:10.544 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.544 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.544 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:10.544 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.900670+0000", 2024-05-15T13:20:10.544 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00010958700000000001 2024-05-15T13:20:10.545 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.545 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:10.545 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.545 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.545 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.545 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.25735.0:248330 6.6 e574/571 6:61758278:::benchmark_data_smithi177_122278_object117959:head v 574'15439, mlcod=574'15436)", 2024-05-15T13:20:10.545 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:46.720997+0000", 2024-05-15T13:20:10.545 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 503.78872285300002, 2024-05-15T13:20:10.545 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.19551647699999999, 2024-05-15T13:20:10.545 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:10.545 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:10.545 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:10.545 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.545 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:10.546 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.720997+0000", 2024-05-15T13:20:10.546 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.546 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.546 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.546 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:10.546 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.720997+0000", 2024-05-15T13:20:10.546 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.546 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.546 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.546 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:10.546 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.720997+0000", 2024-05-15T13:20:10.546 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7400000000000002e-07 2024-05-15T13:20:10.546 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.547 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.547 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:10.547 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.721005+0000", 2024-05-15T13:20:10.547 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.926e-06 2024-05-15T13:20:10.547 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.547 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.547 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:10.547 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.721024+0000", 2024-05-15T13:20:10.547 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9082000000000001e-05 2024-05-15T13:20:10.547 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.547 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.547 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:10.547 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.721033+0000", 2024-05-15T13:20:10.547 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.6209999999999993e-06 2024-05-15T13:20:10.547 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.548 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.548 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:10.548 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.771142+0000", 2024-05-15T13:20:10.548 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.050108869 2024-05-15T13:20:10.548 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.548 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.548 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:10.548 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.771163+0000", 2024-05-15T13:20:10.548 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1069000000000001e-05 2024-05-15T13:20:10.548 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.548 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.548 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:10.548 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.916456+0000", 2024-05-15T13:20:10.548 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.14529314800000001 2024-05-15T13:20:10.548 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.549 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.549 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:10.549 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.916513+0000", 2024-05-15T13:20:10.549 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6888000000000001e-05 2024-05-15T13:20:10.549 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.549 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:10.549 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.549 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.549 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.549 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.25735.0:248331 6.e e574/571 6:7ecac96b:::benchmark_data_smithi177_122278_object117960:head v 574'15450, mlcod=574'15447)", 2024-05-15T13:20:10.549 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:46.724755+0000", 2024-05-15T13:20:10.549 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 503.78496407199998, 2024-05-15T13:20:10.549 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.20206249500000001, 2024-05-15T13:20:10.549 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:10.550 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:10.550 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:10.550 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.550 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:10.550 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.724755+0000", 2024-05-15T13:20:10.550 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.550 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.550 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.550 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:10.550 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.724755+0000", 2024-05-15T13:20:10.550 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.550 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.550 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.550 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:10.551 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.724757+0000", 2024-05-15T13:20:10.551 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1400000000000001e-06 2024-05-15T13:20:10.551 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.551 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.551 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:10.551 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.724766+0000", 2024-05-15T13:20:10.551 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.7890000000000006e-06 2024-05-15T13:20:10.551 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.551 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.551 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:10.551 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.724784+0000", 2024-05-15T13:20:10.551 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7980999999999999e-05 2024-05-15T13:20:10.551 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.551 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.551 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:10.552 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.724794+0000", 2024-05-15T13:20:10.552 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.2250000000000006e-06 2024-05-15T13:20:10.552 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.552 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.552 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:10.552 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.837454+0000", 2024-05-15T13:20:10.552 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.112660314 2024-05-15T13:20:10.552 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.552 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.552 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:10.552 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.837478+0000", 2024-05-15T13:20:10.552 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.4124000000000001e-05 2024-05-15T13:20:10.552 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.552 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.553 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:10.553 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.926774+0000", 2024-05-15T13:20:10.553 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.089295853999999994 2024-05-15T13:20:10.553 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.553 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.553 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:10.554 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.926818+0000", 2024-05-15T13:20:10.554 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4067999999999999e-05 2024-05-15T13:20:10.554 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.554 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:10.554 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.554 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.555 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.555 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.25735.0:248338 6.7 e574/571 6:ec037bfb:::benchmark_data_smithi177_122278_object117967:head v 574'15609, mlcod=573'15606)", 2024-05-15T13:20:10.555 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:46.733951+0000", 2024-05-15T13:20:10.555 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 503.77576862199999, 2024-05-15T13:20:10.555 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.16666650799999999, 2024-05-15T13:20:10.555 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:10.555 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:10.555 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:10.555 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.555 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:10.555 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.733951+0000", 2024-05-15T13:20:10.555 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.555 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.555 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.555 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:10.556 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.733951+0000", 2024-05-15T13:20:10.556 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.556 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.556 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.556 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:10.556 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.733952+0000", 2024-05-15T13:20:10.556 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1999999999999999e-06 2024-05-15T13:20:10.556 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.556 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.556 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:10.556 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.733963+0000", 2024-05-15T13:20:10.556 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1378e-05 2024-05-15T13:20:10.556 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.557 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.557 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:10.557 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.733980+0000", 2024-05-15T13:20:10.557 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6252999999999998e-05 2024-05-15T13:20:10.557 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.557 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.557 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:10.557 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.733987+0000", 2024-05-15T13:20:10.557 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2119999999999997e-06 2024-05-15T13:20:10.557 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.557 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.557 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:10.558 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.770685+0000", 2024-05-15T13:20:10.558 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.036698276000000002 2024-05-15T13:20:10.558 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.558 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.558 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:10.558 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.770711+0000", 2024-05-15T13:20:10.558 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.6197e-05 2024-05-15T13:20:10.558 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.558 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.558 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:10.558 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.900571+0000", 2024-05-15T13:20:10.558 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.12985972900000001 2024-05-15T13:20:10.559 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.559 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.559 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:10.559 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.900617+0000", 2024-05-15T13:20:10.559 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6263000000000001e-05 2024-05-15T13:20:10.559 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.559 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:10.559 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.559 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.559 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.559 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.25735.0:248339 6.6 e574/571 6:6c57f15f:::benchmark_data_smithi177_122278_object117968:head v 574'15440, mlcod=574'15436)", 2024-05-15T13:20:10.559 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:46.734489+0000", 2024-05-15T13:20:10.559 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 503.77523032200003, 2024-05-15T13:20:10.559 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.19236523, 2024-05-15T13:20:10.560 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:10.560 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:10.560 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:10.560 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.560 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:10.560 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.734489+0000", 2024-05-15T13:20:10.560 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.560 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.560 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.560 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:10.560 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.734489+0000", 2024-05-15T13:20:10.560 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.560 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.560 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.561 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:10.561 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.734490+0000", 2024-05-15T13:20:10.561 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0100000000000003e-07 2024-05-15T13:20:10.561 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.561 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.561 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:10.561 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.734498+0000", 2024-05-15T13:20:10.561 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4780000000000002e-06 2024-05-15T13:20:10.561 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.561 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.561 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:10.561 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.734511+0000", 2024-05-15T13:20:10.561 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.362e-05 2024-05-15T13:20:10.561 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.562 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.562 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:10.562 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.734518+0000", 2024-05-15T13:20:10.562 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2529999999999998e-06 2024-05-15T13:20:10.562 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.562 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.562 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:10.562 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.837468+0000", 2024-05-15T13:20:10.562 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.102949173 2024-05-15T13:20:10.562 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.562 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.562 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:10.562 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.837486+0000", 2024-05-15T13:20:10.562 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.859e-05 2024-05-15T13:20:10.563 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.563 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.563 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:10.563 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.926820+0000", 2024-05-15T13:20:10.563 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.089334332000000002 2024-05-15T13:20:10.563 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.563 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.563 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:10.563 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.926854+0000", 2024-05-15T13:20:10.563 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.3883000000000001e-05 2024-05-15T13:20:10.563 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.563 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:10.563 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.563 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.564 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.564 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.25735.0:248342 6.7 e574/571 6:e1036766:::benchmark_data_smithi177_122278_object117971:head v 574'15610, mlcod=573'15606)", 2024-05-15T13:20:10.564 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:46.739919+0000", 2024-05-15T13:20:10.564 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 503.76980050899999, 2024-05-15T13:20:10.564 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.16516624099999999, 2024-05-15T13:20:10.564 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:10.564 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:10.564 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:10.564 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.564 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:10.564 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.739919+0000", 2024-05-15T13:20:10.564 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.564 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.564 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.565 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:10.565 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.739919+0000", 2024-05-15T13:20:10.565 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.565 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.565 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.565 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:10.565 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.739920+0000", 2024-05-15T13:20:10.565 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.018e-06 2024-05-15T13:20:10.565 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.565 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.565 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:10.565 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.739929+0000", 2024-05-15T13:20:10.565 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1900000000000001e-06 2024-05-15T13:20:10.565 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.565 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.566 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:10.566 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.739945+0000", 2024-05-15T13:20:10.566 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6237000000000001e-05 2024-05-15T13:20:10.566 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.566 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.566 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:10.566 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.739954+0000", 2024-05-15T13:20:10.566 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.5320000000000006e-06 2024-05-15T13:20:10.566 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.566 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.566 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:10.566 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.837600+0000", 2024-05-15T13:20:10.566 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.097646175000000002 2024-05-15T13:20:10.567 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.567 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.567 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:10.567 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.837622+0000", 2024-05-15T13:20:10.567 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1571000000000001e-05 2024-05-15T13:20:10.567 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.567 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.567 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:10.567 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.905042+0000", 2024-05-15T13:20:10.567 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.067420590000000002 2024-05-15T13:20:10.567 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.567 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.567 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:10.567 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.905085+0000", 2024-05-15T13:20:10.568 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2928e-05 2024-05-15T13:20:10.568 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.568 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:10.568 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.568 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.568 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.568 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.25735.0:248360 6.e e574/571 6:7cd3bbba:::benchmark_data_smithi177_122278_object117989:head v 574'15451, mlcod=574'15447)", 2024-05-15T13:20:10.568 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:46.773487+0000", 2024-05-15T13:20:10.568 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 503.73623293999998, 2024-05-15T13:20:10.568 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.153413733, 2024-05-15T13:20:10.568 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:10.568 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:10.568 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:10.568 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.568 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:10.569 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.773487+0000", 2024-05-15T13:20:10.569 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.569 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.569 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.569 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:10.569 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.773487+0000", 2024-05-15T13:20:10.569 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.569 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.569 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.569 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:10.569 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.773487+0000", 2024-05-15T13:20:10.569 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.47e-07 2024-05-15T13:20:10.569 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.569 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.570 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:10.570 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.773496+0000", 2024-05-15T13:20:10.570 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.3040000000000005e-06 2024-05-15T13:20:10.570 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.570 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.570 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:10.570 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.773507+0000", 2024-05-15T13:20:10.570 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.171e-05 2024-05-15T13:20:10.570 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.570 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.570 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:10.570 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.773516+0000", 2024-05-15T13:20:10.570 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.3829999999999992e-06 2024-05-15T13:20:10.571 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.571 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.571 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:10.571 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.900595+0000", 2024-05-15T13:20:10.571 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.12707911799999999 2024-05-15T13:20:10.571 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.571 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.571 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:10.571 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.900622+0000", 2024-05-15T13:20:10.571 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.7055000000000001e-05 2024-05-15T13:20:10.571 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.571 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.571 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:10.571 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.926857+0000", 2024-05-15T13:20:10.572 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.026234634 2024-05-15T13:20:10.572 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.572 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.572 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:10.572 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.926900+0000", 2024-05-15T13:20:10.572 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3682e-05 2024-05-15T13:20:10.572 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.572 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:10.572 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.572 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.572 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.572 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.25735.0:248361 6.6 e574/571 6:6c92b8af:::benchmark_data_smithi177_122278_object117990:head v 574'15441, mlcod=574'15436)", 2024-05-15T13:20:10.572 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:46.781860+0000", 2024-05-15T13:20:10.572 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 503.72785937800001, 2024-05-15T13:20:10.573 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.14507964100000001, 2024-05-15T13:20:10.573 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:10.573 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:10.573 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:10.573 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.573 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:10.573 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.781860+0000", 2024-05-15T13:20:10.573 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.573 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.573 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.573 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:10.573 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.781860+0000", 2024-05-15T13:20:10.573 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.573 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.573 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.574 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:10.574 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.781861+0000", 2024-05-15T13:20:10.574 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.39e-06 2024-05-15T13:20:10.574 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.574 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.574 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:10.574 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.781871+0000", 2024-05-15T13:20:10.574 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.3079999999999997e-06 2024-05-15T13:20:10.574 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.574 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.574 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:10.574 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.781888+0000", 2024-05-15T13:20:10.574 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7609e-05 2024-05-15T13:20:10.575 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.575 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.575 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:10.575 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.781898+0000", 2024-05-15T13:20:10.575 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.6600000000000007e-06 2024-05-15T13:20:10.575 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.575 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.575 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:10.575 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.900702+0000", 2024-05-15T13:20:10.575 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.118804351 2024-05-15T13:20:10.575 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.575 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.575 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:10.576 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.900730+0000", 2024-05-15T13:20:10.576 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.7375000000000001e-05 2024-05-15T13:20:10.576 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.576 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.576 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:10.576 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.926903+0000", 2024-05-15T13:20:10.576 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.026172850000000001 2024-05-15T13:20:10.576 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.576 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.576 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:10.576 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.926940+0000", 2024-05-15T13:20:10.576 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7098000000000003e-05 2024-05-15T13:20:10.577 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.577 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:10.577 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.577 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.577 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.577 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.25735.0:248364 6.e e574/571 6:7fb34270:::benchmark_data_smithi177_122278_object117993:head v 574'15452, mlcod=574'15447)", 2024-05-15T13:20:10.577 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:46.784744+0000", 2024-05-15T13:20:10.577 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 503.72497504799998, 2024-05-15T13:20:10.577 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.14222351599999999, 2024-05-15T13:20:10.577 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:10.577 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:10.577 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:10.577 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.578 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:10.578 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.784744+0000", 2024-05-15T13:20:10.578 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.578 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.578 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.578 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:10.578 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.784744+0000", 2024-05-15T13:20:10.578 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.578 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.578 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.578 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:10.578 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.784745+0000", 2024-05-15T13:20:10.578 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.8599999999999996e-07 2024-05-15T13:20:10.578 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.579 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.579 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:10.579 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.784755+0000", 2024-05-15T13:20:10.579 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.2420000000000006e-06 2024-05-15T13:20:10.579 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.579 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.579 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:10.579 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.784771+0000", 2024-05-15T13:20:10.579 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6239000000000002e-05 2024-05-15T13:20:10.579 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.579 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.579 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:10.579 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.784785+0000", 2024-05-15T13:20:10.579 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3781000000000001e-05 2024-05-15T13:20:10.579 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.580 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.580 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:10.580 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.904957+0000", 2024-05-15T13:20:10.580 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.120172524 2024-05-15T13:20:10.580 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.580 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.580 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:10.580 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.904973+0000", 2024-05-15T13:20:10.580 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5860000000000001e-05 2024-05-15T13:20:10.580 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.580 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.580 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:10.580 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.926942+0000", 2024-05-15T13:20:10.581 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.021969085999999999 2024-05-15T13:20:10.581 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.581 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.581 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:10.581 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.926968+0000", 2024-05-15T13:20:10.581 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.5797999999999999e-05 2024-05-15T13:20:10.581 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.581 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:10.581 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.581 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.581 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.581 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.25735.0:248366 6.6 e574/571 6:6e708c4a:::benchmark_data_smithi177_122278_object117995:head v 574'15442, mlcod=574'15436)", 2024-05-15T13:20:10.581 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:46.787184+0000", 2024-05-15T13:20:10.581 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 503.72253552000001, 2024-05-15T13:20:10.582 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.13981651, 2024-05-15T13:20:10.582 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:10.582 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:10.582 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:10.582 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.582 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:10.582 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.787184+0000", 2024-05-15T13:20:10.582 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.582 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.582 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.582 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:10.582 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.787184+0000", 2024-05-15T13:20:10.582 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.582 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.583 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.583 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:10.583 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.787185+0000", 2024-05-15T13:20:10.583 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.9199999999999999e-07 2024-05-15T13:20:10.583 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.583 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.583 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:10.583 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.787194+0000", 2024-05-15T13:20:10.583 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7909999999999993e-06 2024-05-15T13:20:10.583 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.583 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.583 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:10.583 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.787212+0000", 2024-05-15T13:20:10.583 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7867e-05 2024-05-15T13:20:10.584 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.584 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.584 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:10.584 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.787220+0000", 2024-05-15T13:20:10.584 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.4009999999999997e-06 2024-05-15T13:20:10.584 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.584 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.584 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:10.584 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.909248+0000", 2024-05-15T13:20:10.584 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.122028319 2024-05-15T13:20:10.584 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.584 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.584 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:10.585 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.909264+0000", 2024-05-15T13:20:10.585 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5489e-05 2024-05-15T13:20:10.585 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.585 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.585 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:10.585 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.926970+0000", 2024-05-15T13:20:10.585 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.017706612 2024-05-15T13:20:10.585 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.585 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.585 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:10.585 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.927000+0000", 2024-05-15T13:20:10.585 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.0039000000000002e-05 2024-05-15T13:20:10.585 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.585 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:10.585 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.585 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.586 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.586 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.25735.0:248372 6.6 e574/571 6:6b31ad78:::benchmark_data_smithi177_122278_object118001:head v 574'15443, mlcod=574'15436)", 2024-05-15T13:20:10.586 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:11:46.805438+0000", 2024-05-15T13:20:10.586 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 503.70428186800001, 2024-05-15T13:20:10.586 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.12633702299999999, 2024-05-15T13:20:10.586 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:10.586 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:10.586 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:10.586 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.586 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:10.586 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.805438+0000", 2024-05-15T13:20:10.586 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.586 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.587 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.587 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:10.587 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.805438+0000", 2024-05-15T13:20:10.587 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:10.587 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.587 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.587 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:10.587 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.805439+0000", 2024-05-15T13:20:10.587 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0559999999999999e-06 2024-05-15T13:20:10.587 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.587 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.587 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:10.587 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.805449+0000", 2024-05-15T13:20:10.587 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0574000000000001e-05 2024-05-15T13:20:10.587 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.588 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.588 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:10.588 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.805470+0000", 2024-05-15T13:20:10.588 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0922000000000001e-05 2024-05-15T13:20:10.588 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.588 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.588 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:10.588 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.805479+0000", 2024-05-15T13:20:10.588 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.3489999999999998e-06 2024-05-15T13:20:10.588 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.588 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.588 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:10.588 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.913221+0000", 2024-05-15T13:20:10.588 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.107741961 2024-05-15T13:20:10.589 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.589 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.589 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:10.589 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.913237+0000", 2024-05-15T13:20:10.589 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5727e-05 2024-05-15T13:20:10.589 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.589 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.589 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:10.589 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.931721+0000", 2024-05-15T13:20:10.589 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.018484236000000001 2024-05-15T13:20:10.589 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:10.589 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:10.589 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:10.589 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:11:46.931775+0000", 2024-05-15T13:20:10.590 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3198000000000001e-05 2024-05-15T13:20:10.590 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.590 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:10.590 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.590 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:10.590 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:10.590 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:20:10.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:10 smithi120 ceph-mon[43961]: 8.2 scrub starts 2024-05-15T13:20:10.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:10 smithi120 ceph-mon[43961]: osdmap e769: 12 total, 12 up, 11 in 2024-05-15T13:20:10.846 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:20:10.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:10.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:20:10.950 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:20:10.983 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:11.002 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 62 16 61120 61104 61.5775 91.8125 0.0107773 0.0162349 2024-05-15T13:20:11.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:10 smithi027 ceph-mon[39795]: 8.2 scrub starts 2024-05-15T13:20:11.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:10 smithi027 ceph-mon[39795]: osdmap e769: 12 total, 12 up, 11 in 2024-05-15T13:20:11.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:10 smithi177 ceph-mon[44303]: 8.2 scrub starts 2024-05-15T13:20:11.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:10 smithi177 ceph-mon[44303]: osdmap e769: 12 total, 12 up, 11 in 2024-05-15T13:20:11.309 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-05-15T13:20:11.333 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:20:11.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:11.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:20:11.438 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:20:11.467 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:11.497 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:20:11.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:11.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:20:11.604 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:20:11.633 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:11.656 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:20:11.661 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:20:11.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:11.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:20:11.767 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:20:11.822 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:11.948 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:11 smithi120 ceph-mon[43961]: 8.2 scrub ok 2024-05-15T13:20:11.948 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:11 smithi120 ceph-mon[43961]: pgmap v1984: 37 pgs: 37 active+clean; 3.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-05-15T13:20:11.948 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:11 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:20:11.948 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:11 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:20:11.948 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:11 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:20:11.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:11 smithi027 ceph-mon[39795]: 8.2 scrub ok 2024-05-15T13:20:11.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:11 smithi027 ceph-mon[39795]: pgmap v1984: 37 pgs: 37 active+clean; 3.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-05-15T13:20:11.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:11 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:20:11.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:11 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:20:11.949 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:11 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:20:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:20:11.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:11.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:20:12.002 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 63 16 62615 62599 62.083 93.4375 0.0100524 0.0161037 2024-05-15T13:20:12.070 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:20:12.102 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:12.159 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:20:12.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:12.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:20:12.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:11 smithi177 ceph-mon[44303]: 8.2 scrub ok 2024-05-15T13:20:12.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:11 smithi177 ceph-mon[44303]: pgmap v1984: 37 pgs: 37 active+clean; 3.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-05-15T13:20:12.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:11 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:20:12.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:11 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:20:12.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:11 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:20:12.268 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:20:12.296 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:12.550 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:20:12.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:12.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:20:12.654 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:20:12.680 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:12.705 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:20:12.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:12.714 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:20:12.815 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:20:12.863 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:12 smithi027 ceph-mon[39795]: 8.5 scrub starts 2024-05-15T13:20:12.867 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:12.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:12 smithi177 ceph-mon[44303]: 8.5 scrub starts 2024-05-15T13:20:13.003 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 64 16 63838 63822 62.3057 76.4375 0.00628259 0.016045 2024-05-15T13:20:13.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:12 smithi120 ceph-mon[43961]: 8.5 scrub starts 2024-05-15T13:20:13.799 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:20:13.799 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [ 2024-05-15T13:20:13.799 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:13.799 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26299.0:64857 8.8 8:1c6de484:::benchmark_data_smithi177_149673_object64856:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e769)", 2024-05-15T13:20:13.799 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:20:13.796209+0000", 2024-05-15T13:20:13.799 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.0043882390000000004, 2024-05-15T13:20:13.800 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0044113829999999996, 2024-05-15T13:20:13.800 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:13.800 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:20:13.800 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:20:13.800 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26299", 2024-05-15T13:20:13.800 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:20:13.800 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 64857 2024-05-15T13:20:13.800 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:13.800 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:13.800 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:13.800 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:13.800 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:13.796209+0000", 2024-05-15T13:20:13.800 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:13.800 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:13.800 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:13.801 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:13.801 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:13.796209+0000", 2024-05-15T13:20:13.801 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:13.801 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:13.801 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:13.801 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:13.801 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:13.796210+0000", 2024-05-15T13:20:13.801 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2219999999999999e-06 2024-05-15T13:20:13.801 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:13.801 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:13.801 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:13.801 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:13.796231+0000", 2024-05-15T13:20:13.801 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0378999999999999e-05 2024-05-15T13:20:13.801 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:13.801 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:13.801 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:13.802 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:13.796242+0000", 2024-05-15T13:20:13.802 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.133e-05 2024-05-15T13:20:13.802 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:13.802 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:13.802 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:13.802 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:13.796248+0000", 2024-05-15T13:20:13.802 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.0070000000000003e-06 2024-05-15T13:20:13.802 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:13.802 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:13.802 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:13.802 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:13.796294+0000", 2024-05-15T13:20:13.802 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6094999999999998e-05 2024-05-15T13:20:13.802 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:13.802 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:13.803 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:13.803 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:13.796449+0000", 2024-05-15T13:20:13.803 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015460100000000001 2024-05-15T13:20:13.803 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:13.803 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:13.803 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 3,9", 2024-05-15T13:20:13.803 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:13.796621+0000", 2024-05-15T13:20:13.803 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00017230899999999999 2024-05-15T13:20:13.803 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:13.803 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:13.803 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:20:13.804 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:13.798468+0000", 2024-05-15T13:20:13.804 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0018473210000000001 2024-05-15T13:20:13.804 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:13.804 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:13.804 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:13.804 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:13.804 INFO:teuthology.orchestra.run.smithi120.stdout: ], 2024-05-15T13:20:13.804 INFO:teuthology.orchestra.run.smithi120.stdout: "num_ops": 1 2024-05-15T13:20:13.804 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:20:14.008 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 65 16 65112 65096 62.5673 79.625 0.0145317 0.0159783 2024-05-15T13:20:14.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:13 smithi120 ceph-mon[43961]: pgmap v1985: 37 pgs: 37 active+clean; 3.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-05-15T13:20:14.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:13 smithi120 ceph-mon[43961]: 8.5 scrub ok 2024-05-15T13:20:14.163 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:13 smithi027 ceph-mon[39795]: pgmap v1985: 37 pgs: 37 active+clean; 3.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-05-15T13:20:14.163 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:13 smithi027 ceph-mon[39795]: 8.5 scrub ok 2024-05-15T13:20:14.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:13 smithi177 ceph-mon[44303]: pgmap v1985: 37 pgs: 37 active+clean; 3.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-05-15T13:20:14.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:13 smithi177 ceph-mon[44303]: 8.5 scrub ok 2024-05-15T13:20:14.437 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:20:14.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:14.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:20:14.543 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:20:14.577 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:14.604 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:20:14.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:14.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:20:14.624 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-05-15T13:20:14.709 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:20:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:14.822 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:20:14.824 INFO:teuthology.orchestra.run.smithi027.stderr:set 8.d pg_upmap mapping to [0,5,6] 2024-05-15T13:20:14.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:14.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:20:14.930 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:20:14.966 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:20:14.976 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:15.009 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 66 16 66120 66104 62.5738 63 0.00334175 0.0159573 2024-05-15T13:20:15.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:14 smithi027 ceph-mon[39795]: 8.b scrub starts 2024-05-15T13:20:15.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:14 smithi027 ceph-mon[39795]: 8.9 deep-scrub starts 2024-05-15T13:20:15.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:14 smithi027 ceph-mon[39795]: 8.d scrub starts 2024-05-15T13:20:15.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:14 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2661700004' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "8.d", "id": [0, 5, 6]}]: dispatch 2024-05-15T13:20:15.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:14 smithi120 ceph-mon[43961]: 8.b scrub starts 2024-05-15T13:20:15.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:14 smithi120 ceph-mon[43961]: 8.9 deep-scrub starts 2024-05-15T13:20:15.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:14 smithi120 ceph-mon[43961]: 8.d scrub starts 2024-05-15T13:20:15.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:14 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2661700004' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "8.d", "id": [0, 5, 6]}]: dispatch 2024-05-15T13:20:15.063 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:20:15.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:15.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:20:15.168 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:20:15.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:14 smithi177 ceph-mon[44303]: 8.b scrub starts 2024-05-15T13:20:15.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:14 smithi177 ceph-mon[44303]: 8.9 deep-scrub starts 2024-05-15T13:20:15.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:14 smithi177 ceph-mon[44303]: 8.d scrub starts 2024-05-15T13:20:15.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:14 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2661700004' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "8.d", "id": [0, 5, 6]}]: dispatch 2024-05-15T13:20:15.201 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:15.226 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:20:15.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:15.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:20:15.344 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:20:15.371 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:15.390 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:20:15.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:15.394 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:20:15.495 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:20:15.525 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:15.580 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:20:15.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:15.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:20:15.684 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:20:15.712 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:15.787 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:20:15.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:15.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:20:15.823 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:20:15.891 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:20:15.919 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:15.938 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:20:15.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:15.942 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:20:16.008 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 67 16 66155 66139 61.6728 2.1875 0.00324794 0.0159519 2024-05-15T13:20:16.043 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:20:16.069 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:15 smithi027 ceph-mon[39795]: 8.b scrub ok 2024-05-15T13:20:16.069 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:15 smithi027 ceph-mon[39795]: 8.a scrub starts 2024-05-15T13:20:16.069 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:15 smithi027 ceph-mon[39795]: pgmap v1986: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-05-15T13:20:16.069 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:15 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2661700004' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "8.d", "id": [0, 5, 6]}]': finished 2024-05-15T13:20:16.070 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:15 smithi027 ceph-mon[39795]: osdmap e770: 12 total, 12 up, 11 in 2024-05-15T13:20:16.070 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:15 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/274450640' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:20:16.070 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:15 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:20:16.072 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:16.091 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:20:16.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:16.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:20:16.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:15 smithi177 ceph-mon[44303]: 8.b scrub ok 2024-05-15T13:20:16.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:15 smithi177 ceph-mon[44303]: 8.a scrub starts 2024-05-15T13:20:16.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:15 smithi177 ceph-mon[44303]: pgmap v1986: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-05-15T13:20:16.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:15 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2661700004' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "8.d", "id": [0, 5, 6]}]': finished 2024-05-15T13:20:16.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:15 smithi177 ceph-mon[44303]: osdmap e770: 12 total, 12 up, 11 in 2024-05-15T13:20:16.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:15 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/274450640' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:20:16.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:15 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:20:16.195 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:20:16.200 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:20:16.219 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:15 smithi120 ceph-mon[43961]: 8.b scrub ok 2024-05-15T13:20:16.219 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:15 smithi120 ceph-mon[43961]: 8.a scrub starts 2024-05-15T13:20:16.219 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:15 smithi120 ceph-mon[43961]: pgmap v1986: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-05-15T13:20:16.219 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:15 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2661700004' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "8.d", "id": [0, 5, 6]}]': finished 2024-05-15T13:20:16.219 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:15 smithi120 ceph-mon[43961]: osdmap e770: 12 total, 12 up, 11 in 2024-05-15T13:20:16.219 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:15 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/274450640' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:20:16.219 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:15 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:20:16.221 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:16.346 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:20:16.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:16.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:20:16.450 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:20:16.477 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:16.498 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:20:16.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:16.501 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:20:16.602 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:20:16.630 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:16.686 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:20:16.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:16.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:20:16.733 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:20:16.792 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:20:16.822 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:16.843 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:20:16.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:16.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:20:16.947 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:20:16.976 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:16 smithi027 ceph-mon[39795]: 8.e scrub starts 2024-05-15T13:20:16.976 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:16 smithi027 ceph-mon[39795]: 8.a scrub ok 2024-05-15T13:20:16.976 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:16 smithi027 ceph-mon[39795]: 8.e scrub ok 2024-05-15T13:20:16.976 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:16 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:20:16.976 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:16 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:20:16.976 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:16 smithi027 ceph-mon[39795]: osdmap e771: 12 total, 12 up, 11 in 2024-05-15T13:20:16.978 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:17.009 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 68 16 66155 66139 60.7661 0 - 0.0159519 2024-05-15T13:20:17.021 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:20:17.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:17.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:20:17.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:16 smithi120 ceph-mon[43961]: 8.e scrub starts 2024-05-15T13:20:17.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:16 smithi120 ceph-mon[43961]: 8.a scrub ok 2024-05-15T13:20:17.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:16 smithi120 ceph-mon[43961]: 8.e scrub ok 2024-05-15T13:20:17.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:16 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:20:17.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:16 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:20:17.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:16 smithi120 ceph-mon[43961]: osdmap e771: 12 total, 12 up, 11 in 2024-05-15T13:20:17.127 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:20:17.135 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:20:17.135 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T13:20:17.135 INFO:teuthology.orchestra.run.smithi120.stdout: "complaint_time": 30, 2024-05-15T13:20:17.135 INFO:teuthology.orchestra.run.smithi120.stdout: "num_blocked_ops": 0 2024-05-15T13:20:17.135 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:20:17.151 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:16 smithi177 ceph-mon[44303]: 8.e scrub starts 2024-05-15T13:20:17.151 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:16 smithi177 ceph-mon[44303]: 8.a scrub ok 2024-05-15T13:20:17.151 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:16 smithi177 ceph-mon[44303]: 8.e scrub ok 2024-05-15T13:20:17.151 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:16 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:20:17.151 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:16 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:20:17.152 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:16 smithi177 ceph-mon[44303]: osdmap e771: 12 total, 12 up, 11 in 2024-05-15T13:20:17.153 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:17.172 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:20:17.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:17.175 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:20:17.276 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:20:17.322 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:17.365 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:20:17.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:17.371 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:20:17.472 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:20:17.507 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:18.009 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 69 16 67093 67077 60.735 29.3125 0.00981504 0.0164596 2024-05-15T13:20:18.107 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:17 smithi027 ceph-mon[39795]: pgmap v1989: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-05-15T13:20:18.108 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:17 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:20:18.108 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:17 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:20:18.108 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:17 smithi027 ceph-mon[39795]: osdmap e772: 12 total, 12 up, 11 in 2024-05-15T13:20:18.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:17 smithi177 ceph-mon[44303]: pgmap v1989: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-05-15T13:20:18.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:17 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:20:18.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:17 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:20:18.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:17 smithi177 ceph-mon[44303]: osdmap e772: 12 total, 12 up, 11 in 2024-05-15T13:20:18.289 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:17 smithi120 ceph-mon[43961]: pgmap v1989: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-05-15T13:20:18.289 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:17 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:20:18.289 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:17 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:20:18.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:17 smithi120 ceph-mon[43961]: osdmap e772: 12 total, 12 up, 11 in 2024-05-15T13:20:18.695 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:20:19.009 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 70 16 68257 68241 60.9065 72.75 0.00960657 0.0164138 2024-05-15T13:20:19.162 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:20:19.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:19.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:20:19.269 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:20:19.301 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:19.325 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:20:19.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:19.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:20:19.430 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:20:19.496 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:19.715 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:20:19.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:19.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:20:19.820 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:20:19.868 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:19.903 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:20:19.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:19.910 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:20:20.009 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 71 16 69407 69391 61.0609 71.875 0.00983129 0.016372 2024-05-15T13:20:20.011 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:20:20.041 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:20.164 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:19 smithi120 ceph-mon[43961]: pgmap v1991: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s; 12252/198708 objects misplaced (6.166%); 85 KiB/s, 1 objects/s recovering 2024-05-15T13:20:20.169 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:20:20.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:20.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:20:20.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:19 smithi177 ceph-mon[44303]: pgmap v1991: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s; 12252/198708 objects misplaced (6.166%); 85 KiB/s, 1 objects/s recovering 2024-05-15T13:20:20.277 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:20:20.279 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:19 smithi027 ceph-mon[39795]: pgmap v1991: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s; 12252/198708 objects misplaced (6.166%); 85 KiB/s, 1 objects/s recovering 2024-05-15T13:20:20.308 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:20.511 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-05-15T13:20:20.513 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:20:20.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:20.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:20:20.605 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 8, 9, 10, 11, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:20:20.605 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:20:20.605 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-05-15T13:20:20.605 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-05-15T13:20:20.616 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:20:20.652 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:20.685 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:20:20.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:20.689 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:20:20.725 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:20:20.725 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:20:20.725 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:20:20.756 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:20:20.790 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:20:20.809 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:20:20.825 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:20:20.831 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:20.855 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:20:20.857 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:20:20.859 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:20:20.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:20.861 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:20:20.901 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:20:20.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:20 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:20:20.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:20 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:20:20.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:20 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4101106300' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:20:20.947 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:20:20.962 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:20:20.980 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:20:20.995 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:21.009 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 72 16 70371 70355 61.0495 60.25 0.0212326 0.0163745 2024-05-15T13:20:21.012 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:20:21.026 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:20:21.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:20 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:20:21.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:20 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:20:21.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:20 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4101106300' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:20:21.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:21.030 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:20:21.042 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:20:21.102 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:20 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:20:21.102 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:20 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:20:21.102 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:20 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4101106300' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:20:21.131 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:20:21.157 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:21.176 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:20:21.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:21.179 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:20:21.280 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:20:21.284 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:20:21.285 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:20:21.285 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:20:21.285 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:20:21.285 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:20:21.285 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:20:21.285 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:20:21.285 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:20:21.285 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:20:21.285 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:20:21.285 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:20:21.286 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:20:21.286 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:20:21.309 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:21.336 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:20:21.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:21.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:20:21.441 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:20:21.479 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:21.685 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:20:21.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:21.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:20:21.789 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:20:21.849 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:22.010 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 73 16 71428 71412 61.1177 66.0625 0.016028 0.0163572 2024-05-15T13:20:22.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:21 smithi120 ceph-mon[43961]: 8.9 deep-scrub ok 2024-05-15T13:20:22.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:21 smithi120 ceph-mon[43961]: pgmap v1992: 37 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 35 active+clean; 4.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 44 MiB/s wr, 698 op/s; 12252/202461 objects misplaced (6.052%); 85 KiB/s, 1 objects/s recovering 2024-05-15T13:20:22.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:21 smithi120 ceph-mon[43961]: osdmap e773: 12 total, 12 up, 11 in 2024-05-15T13:20:22.122 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:20:22.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:22.127 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:21 smithi027 ceph-mon[39795]: 8.9 deep-scrub ok 2024-05-15T13:20:22.127 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:21 smithi027 ceph-mon[39795]: pgmap v1992: 37 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 35 active+clean; 4.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 44 MiB/s wr, 698 op/s; 12252/202461 objects misplaced (6.052%); 85 KiB/s, 1 objects/s recovering 2024-05-15T13:20:22.127 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:21 smithi027 ceph-mon[39795]: osdmap e773: 12 total, 12 up, 11 in 2024-05-15T13:20:22.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:20:22.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:21 smithi177 ceph-mon[44303]: 8.9 deep-scrub ok 2024-05-15T13:20:22.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:21 smithi177 ceph-mon[44303]: pgmap v1992: 37 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 35 active+clean; 4.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 44 MiB/s wr, 698 op/s; 12252/202461 objects misplaced (6.052%); 85 KiB/s, 1 objects/s recovering 2024-05-15T13:20:22.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:21 smithi177 ceph-mon[44303]: osdmap e773: 12 total, 12 up, 11 in 2024-05-15T13:20:22.228 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:20:22.299 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:22.882 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:20:22.882 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:20:23.010 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 74 16 72540 72524 61.2309 69.5 0.00441904 0.0163248 2024-05-15T13:20:23.715 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-05-15T13:20:24.010 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 75 16 73699 73683 61.3802 72.4375 0.0116136 0.0162867 2024-05-15T13:20:24.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:23 smithi120 ceph-mon[43961]: pgmap v1994: 37 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 35 active+clean; 4.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 34 MiB/s wr, 539 op/s; 12252/204126 objects misplaced (6.002%); 75 KiB/s, 1 objects/s recovering 2024-05-15T13:20:24.055 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:20:24.130 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:20:24.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:23 smithi027 ceph-mon[39795]: pgmap v1994: 37 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 35 active+clean; 4.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 34 MiB/s wr, 539 op/s; 12252/204126 objects misplaced (6.002%); 75 KiB/s, 1 objects/s recovering 2024-05-15T13:20:24.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:24.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:20:24.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:23 smithi177 ceph-mon[44303]: pgmap v1994: 37 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 35 active+clean; 4.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 34 MiB/s wr, 539 op/s; 12252/204126 objects misplaced (6.002%); 75 KiB/s, 1 objects/s recovering 2024-05-15T13:20:24.236 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:20:24.283 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:24.316 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:20:24.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:24.326 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:20:24.427 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:20:24.459 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:24.535 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:20:24.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:24.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:20:24.642 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:20:24.673 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:24.924 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:20:24.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:24.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:20:25.010 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 76 16 74759 74743 61.4441 66.25 0.0115015 0.0162705 2024-05-15T13:20:25.027 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:20:25.070 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:25.118 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:20:25.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:25.124 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:20:25.225 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:20:25.254 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:25.435 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:20:25.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:25.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:20:25.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:25 smithi177 ceph-mon[44303]: pgmap v1995: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 61 MiB/s wr, 970 op/s; 11964/216438 objects misplaced (5.528%); 832 KiB/s, 12 objects/s recovering 2024-05-15T13:20:25.528 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:25 smithi027 ceph-mon[39795]: pgmap v1995: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 61 MiB/s wr, 970 op/s; 11964/216438 objects misplaced (5.528%); 832 KiB/s, 12 objects/s recovering 2024-05-15T13:20:25.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:25 smithi120 ceph-mon[43961]: pgmap v1995: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 61 MiB/s wr, 970 op/s; 11964/216438 objects misplaced (5.528%); 832 KiB/s, 12 objects/s recovering 2024-05-15T13:20:25.539 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:20:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:25.652 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:20:25.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:25.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:20:25.806 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:20:25.858 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:26.011 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 77 16 75964 75948 61.623 75.3125 0.00629846 0.0162226 2024-05-15T13:20:26.103 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:20:26.104 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:20:26.170 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:20:26.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=54,logm=6} crc {auth=1523789570,config=1059521453,health=3340005642,kv=3071533163,logm=1156766490}) 2024-05-15T13:20:26.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1293243934}) 2024-05-15T13:20:26.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4128545023}) 2024-05-15T13:20:26.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2914657478}) 2024-05-15T13:20:26.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3175767099}) 2024-05-15T13:20:26.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3712414518}) 2024-05-15T13:20:26.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2632028236}) 2024-05-15T13:20:26.214 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3323646670}) 2024-05-15T13:20:26.214 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3971530150}) 2024-05-15T13:20:26.214 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3384572104}) 2024-05-15T13:20:26.214 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4012413271}) 2024-05-15T13:20:26.214 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2711521225}) 2024-05-15T13:20:26.214 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3595223928}) 2024-05-15T13:20:26.214 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=941277779}) 2024-05-15T13:20:26.214 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3262401199}) 2024-05-15T13:20:26.214 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1954048094}) 2024-05-15T13:20:26.214 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2815507416}) 2024-05-15T13:20:26.214 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1155445190}) 2024-05-15T13:20:26.214 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1117640751}) 2024-05-15T13:20:26.214 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=529599941}) 2024-05-15T13:20:26.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1402418334}) 2024-05-15T13:20:26.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1696210166}) 2024-05-15T13:20:26.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1662546189}) 2024-05-15T13:20:26.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=431678968}) 2024-05-15T13:20:26.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2142779818}) 2024-05-15T13:20:26.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1268787597}) 2024-05-15T13:20:26.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1099537185}) 2024-05-15T13:20:26.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3370601811}) 2024-05-15T13:20:26.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=614262487}) 2024-05-15T13:20:26.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2788622293}) 2024-05-15T13:20:26.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1845395427}) 2024-05-15T13:20:26.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4113196266}) 2024-05-15T13:20:26.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803618606}) 2024-05-15T13:20:26.215 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1979872812}) 2024-05-15T13:20:26.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2170057874}) 2024-05-15T13:20:26.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=890839764}) 2024-05-15T13:20:26.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3129859602}) 2024-05-15T13:20:26.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=622038997}) 2024-05-15T13:20:26.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=858552447}) 2024-05-15T13:20:26.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1665639042}) 2024-05-15T13:20:26.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1062985306}) 2024-05-15T13:20:26.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=737210025}) 2024-05-15T13:20:26.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1634918800}) 2024-05-15T13:20:26.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3594602113}) 2024-05-15T13:20:26.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2314792337}) 2024-05-15T13:20:26.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2619639892}) 2024-05-15T13:20:26.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2491717187}) 2024-05-15T13:20:26.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2436036093}) 2024-05-15T13:20:26.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2680209675}) 2024-05-15T13:20:26.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3316015312}) 2024-05-15T13:20:26.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1730349625}) 2024-05-15T13:20:26.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=73063541}) 2024-05-15T13:20:26.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=272014517}) 2024-05-15T13:20:26.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3779991686}) 2024-05-15T13:20:26.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=991191325}) 2024-05-15T13:20:26.364 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:20:26.487 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=54,logm=6} crc {auth=1523789570,config=1059521453,health=3340005642,kv=3071533163,logm=1156766490}) 2024-05-15T13:20:26.487 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1293243934}) 2024-05-15T13:20:26.487 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4128545023}) 2024-05-15T13:20:26.487 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2914657478}) 2024-05-15T13:20:26.487 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3175767099}) 2024-05-15T13:20:26.487 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3712414518}) 2024-05-15T13:20:26.487 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2632028236}) 2024-05-15T13:20:26.487 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3323646670}) 2024-05-15T13:20:26.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3971530150}) 2024-05-15T13:20:26.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3384572104}) 2024-05-15T13:20:26.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4012413271}) 2024-05-15T13:20:26.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2711521225}) 2024-05-15T13:20:26.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3595223928}) 2024-05-15T13:20:26.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=941277779}) 2024-05-15T13:20:26.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3262401199}) 2024-05-15T13:20:26.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1954048094}) 2024-05-15T13:20:26.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2815507416}) 2024-05-15T13:20:26.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1155445190}) 2024-05-15T13:20:26.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1117640751}) 2024-05-15T13:20:26.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=529599941}) 2024-05-15T13:20:26.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1402418334}) 2024-05-15T13:20:26.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1696210166}) 2024-05-15T13:20:26.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1662546189}) 2024-05-15T13:20:26.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=431678968}) 2024-05-15T13:20:26.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2142779818}) 2024-05-15T13:20:26.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1268787597}) 2024-05-15T13:20:26.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1099537185}) 2024-05-15T13:20:26.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3370601811}) 2024-05-15T13:20:26.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=614262487}) 2024-05-15T13:20:26.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2788622293}) 2024-05-15T13:20:26.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1845395427}) 2024-05-15T13:20:26.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4113196266}) 2024-05-15T13:20:26.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803618606}) 2024-05-15T13:20:26.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1979872812}) 2024-05-15T13:20:26.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2170057874}) 2024-05-15T13:20:26.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=890839764}) 2024-05-15T13:20:26.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3129859602}) 2024-05-15T13:20:26.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=622038997}) 2024-05-15T13:20:26.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=858552447}) 2024-05-15T13:20:26.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1665639042}) 2024-05-15T13:20:26.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1062985306}) 2024-05-15T13:20:26.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=737210025}) 2024-05-15T13:20:26.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1634918800}) 2024-05-15T13:20:26.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3594602113}) 2024-05-15T13:20:26.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2314792337}) 2024-05-15T13:20:26.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2619639892}) 2024-05-15T13:20:26.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2491717187}) 2024-05-15T13:20:26.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2436036093}) 2024-05-15T13:20:26.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2680209675}) 2024-05-15T13:20:26.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3316015312}) 2024-05-15T13:20:26.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1730349625}) 2024-05-15T13:20:26.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=73063541}) 2024-05-15T13:20:26.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=272014517}) 2024-05-15T13:20:26.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3779991686}) 2024-05-15T13:20:26.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=991191325}) 2024-05-15T13:20:26.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4100262294}) 2024-05-15T13:20:26.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1550487118}) 2024-05-15T13:20:26.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1105149251}) 2024-05-15T13:20:26.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1111523456}) 2024-05-15T13:20:26.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=200088216}) 2024-05-15T13:20:26.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=869560824}) 2024-05-15T13:20:26.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1989585682}) 2024-05-15T13:20:26.492 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2044245359}) 2024-05-15T13:20:26.492 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4279206781}) 2024-05-15T13:20:26.492 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3896353176}) 2024-05-15T13:20:26.492 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=38,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=29} crc {logm=961570160,mdsmap=3984425268,mgr=76093979,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=3001467793,mon_config_key=1048937457}) 2024-05-15T13:20:26.492 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=848282982}) 2024-05-15T13:20:26.492 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1869559506}) 2024-05-15T13:20:26.492 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=46,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=36} crc {mon_config_key=2074278679,monmap=2435304556,osd_metadata=1212035326,osd_pg_creating=3911967230,osdmap=1566932862}) 2024-05-15T13:20:26.492 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=79} crc {osdmap=804551354}) 2024-05-15T13:20:26.529 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4100262294}) 2024-05-15T13:20:26.529 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1550487118}) 2024-05-15T13:20:26.529 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1105149251}) 2024-05-15T13:20:26.529 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1111523456}) 2024-05-15T13:20:26.529 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=200088216}) 2024-05-15T13:20:26.529 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=869560824}) 2024-05-15T13:20:26.529 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1989585682}) 2024-05-15T13:20:26.529 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2044245359}) 2024-05-15T13:20:26.529 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4279206781}) 2024-05-15T13:20:26.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3896353176}) 2024-05-15T13:20:26.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=38,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=29} crc {logm=961570160,mdsmap=3984425268,mgr=76093979,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=3001467793,mon_config_key=1048937457}) 2024-05-15T13:20:26.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=848282982}) 2024-05-15T13:20:26.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1869559506}) 2024-05-15T13:20:26.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=46,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=36} crc {mon_config_key=2074278679,monmap=2435304556,osd_metadata=1212035326,osd_pg_creating=3911967230,osdmap=1566932862}) 2024-05-15T13:20:26.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=79} crc {osdmap=804551354}) 2024-05-15T13:20:26.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=54,logm=6} crc {auth=1523789570,config=1059521453,health=3340005642,kv=3071533163,logm=1156766490}) 2024-05-15T13:20:26.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1293243934}) 2024-05-15T13:20:26.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4128545023}) 2024-05-15T13:20:26.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2914657478}) 2024-05-15T13:20:26.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3175767099}) 2024-05-15T13:20:26.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3712414518}) 2024-05-15T13:20:26.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2632028236}) 2024-05-15T13:20:26.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3323646670}) 2024-05-15T13:20:26.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3971530150}) 2024-05-15T13:20:26.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3384572104}) 2024-05-15T13:20:26.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4012413271}) 2024-05-15T13:20:26.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2711521225}) 2024-05-15T13:20:26.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3595223928}) 2024-05-15T13:20:26.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=941277779}) 2024-05-15T13:20:26.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3262401199}) 2024-05-15T13:20:26.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1954048094}) 2024-05-15T13:20:26.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2815507416}) 2024-05-15T13:20:26.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1155445190}) 2024-05-15T13:20:26.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1117640751}) 2024-05-15T13:20:26.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=529599941}) 2024-05-15T13:20:26.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1402418334}) 2024-05-15T13:20:26.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1696210166}) 2024-05-15T13:20:26.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1662546189}) 2024-05-15T13:20:26.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=431678968}) 2024-05-15T13:20:26.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2142779818}) 2024-05-15T13:20:26.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1268787597}) 2024-05-15T13:20:26.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1099537185}) 2024-05-15T13:20:26.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3370601811}) 2024-05-15T13:20:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=614262487}) 2024-05-15T13:20:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2788622293}) 2024-05-15T13:20:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1845395427}) 2024-05-15T13:20:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4113196266}) 2024-05-15T13:20:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803618606}) 2024-05-15T13:20:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1979872812}) 2024-05-15T13:20:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2170057874}) 2024-05-15T13:20:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=890839764}) 2024-05-15T13:20:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3129859602}) 2024-05-15T13:20:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=622038997}) 2024-05-15T13:20:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=858552447}) 2024-05-15T13:20:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1665639042}) 2024-05-15T13:20:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1062985306}) 2024-05-15T13:20:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=737210025}) 2024-05-15T13:20:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1634918800}) 2024-05-15T13:20:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3594602113}) 2024-05-15T13:20:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2314792337}) 2024-05-15T13:20:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2619639892}) 2024-05-15T13:20:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2491717187}) 2024-05-15T13:20:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2436036093}) 2024-05-15T13:20:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2680209675}) 2024-05-15T13:20:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3316015312}) 2024-05-15T13:20:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1730349625}) 2024-05-15T13:20:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=73063541}) 2024-05-15T13:20:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=272014517}) 2024-05-15T13:20:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3779991686}) 2024-05-15T13:20:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=991191325}) 2024-05-15T13:20:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4100262294}) 2024-05-15T13:20:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1550487118}) 2024-05-15T13:20:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1105149251}) 2024-05-15T13:20:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1111523456}) 2024-05-15T13:20:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=200088216}) 2024-05-15T13:20:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=869560824}) 2024-05-15T13:20:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1989585682}) 2024-05-15T13:20:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2044245359}) 2024-05-15T13:20:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4279206781}) 2024-05-15T13:20:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3896353176}) 2024-05-15T13:20:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=38,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=29} crc {logm=961570160,mdsmap=3984425268,mgr=76093979,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=3001467793,mon_config_key=1048937457}) 2024-05-15T13:20:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=848282982}) 2024-05-15T13:20:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1869559506}) 2024-05-15T13:20:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=46,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=36} crc {mon_config_key=2074278679,monmap=2435304556,osd_metadata=1212035326,osd_pg_creating=3911967230,osdmap=1566932862}) 2024-05-15T13:20:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=79} crc {osdmap=804551354}) 2024-05-15T13:20:26.853 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-05-15T13:20:27.012 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 78 16 77121 77105 61.7592 72.3125 0.00647407 0.016186 2024-05-15T13:20:27.183 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:20:27.447 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:20:27.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:27.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:20:27.470 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:27 smithi027 ceph-mon[39795]: pgmap v1996: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 56 MiB/s wr, 893 op/s; 11964/219360 objects misplaced (5.454%); 680 KiB/s, 10 objects/s recovering 2024-05-15T13:20:27.470 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:27 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:20:27.470 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:27 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:20:27.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:27 smithi120 ceph-mon[43961]: pgmap v1996: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 56 MiB/s wr, 893 op/s; 11964/219360 objects misplaced (5.454%); 680 KiB/s, 10 objects/s recovering 2024-05-15T13:20:27.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:27 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:20:27.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:27 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:20:27.558 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:20:27.611 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:27 smithi177 ceph-mon[44303]: pgmap v1996: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 56 MiB/s wr, 893 op/s; 11964/219360 objects misplaced (5.454%); 680 KiB/s, 10 objects/s recovering 2024-05-15T13:20:27.612 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:27 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:20:27.612 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:27 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:20:27.616 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:27.649 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:20:27.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:27.654 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:20:27.755 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:20:27.800 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:27.824 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:20:27.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:27.830 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:20:27.931 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:20:27.970 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:28.012 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 79 16 78162 78146 61.801 65.0625 0.00922674 0.0161762 2024-05-15T13:20:28.289 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:28 smithi120 ceph-mon[43961]: osdmap e774: 12 total, 12 up, 11 in 2024-05-15T13:20:28.468 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:28 smithi027 ceph-mon[39795]: osdmap e774: 12 total, 12 up, 11 in 2024-05-15T13:20:28.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:28 smithi177 ceph-mon[44303]: osdmap e774: 12 total, 12 up, 11 in 2024-05-15T13:20:29.014 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T13:20:29.016598+0000 min lat: 0.00105016 max lat: 15.0048 avg lat: 0.0161331 2024-05-15T13:20:29.014 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:20:29.014 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 80 16 79365 79349 61.9674 75.1875 0.0211464 0.0161331 2024-05-15T13:20:29.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:29 smithi027 ceph-mon[39795]: pgmap v1998: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s; 11631/228846 objects misplaced (5.082%); 1.6 MiB/s, 25 objects/s recovering 2024-05-15T13:20:29.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:29 smithi120 ceph-mon[43961]: pgmap v1998: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s; 11631/228846 objects misplaced (5.082%); 1.6 MiB/s, 25 objects/s recovering 2024-05-15T13:20:29.633 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:20:29.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:29.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:20:29.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:29 smithi177 ceph-mon[44303]: pgmap v1998: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s; 11631/228846 objects misplaced (5.082%); 1.6 MiB/s, 25 objects/s recovering 2024-05-15T13:20:29.736 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:20:29.777 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:29.810 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:20:29.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:29.814 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:20:29.915 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:20:29.937 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:20:29.986 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:30.014 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 81 16 80475 80459 62.0587 69.375 0.0134256 0.0161079 2024-05-15T13:20:30.261 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:20:30.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:30.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:20:30.284 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:20:30.370 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:20:30.411 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:30.442 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:20:30.443 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:20:30.443 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:20:30.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:30.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:20:30.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:30 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1626164270' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:20:30.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:30 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:20:30.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:30 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1626164270' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:20:30.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:30 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:20:30.549 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:20:30.597 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:30 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1626164270' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:20:30.597 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:30 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:20:30.600 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:30.636 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:20:30.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:30.642 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:20:30.743 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:20:30.783 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:30.813 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:20:30.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:30.824 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:20:30.925 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:20:30.956 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:31.014 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 82 16 81610 81594 62.1669 70.9375 0.0082042 0.0160816 2024-05-15T13:20:31.208 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:20:31.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:31.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:20:31.218 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-05-15T13:20:31.314 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:20:31.352 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:31.387 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:20:31.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:31.391 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:20:31.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:31 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:20:31.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:31 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:20:31.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:31 smithi027 ceph-mon[39795]: osdmap e775: 12 total, 12 up, 11 in 2024-05-15T13:20:31.447 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:31 smithi027 ceph-mon[39795]: pgmap v2000: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s; 11631/235272 objects misplaced (4.944%); 1.6 MiB/s, 25 objects/s recovering 2024-05-15T13:20:31.491 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:20:31.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:31 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:20:31.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:31 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:20:31.535 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:31 smithi120 ceph-mon[43961]: osdmap e775: 12 total, 12 up, 11 in 2024-05-15T13:20:31.536 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:31 smithi120 ceph-mon[43961]: pgmap v2000: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s; 11631/235272 objects misplaced (4.944%); 1.6 MiB/s, 25 objects/s recovering 2024-05-15T13:20:31.538 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:31.569 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:20:31.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:31.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:20:31.588 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:20:31.676 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:20:31.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:31 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:20:31.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:31 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:20:31.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:31 smithi177 ceph-mon[44303]: osdmap e775: 12 total, 12 up, 11 in 2024-05-15T13:20:31.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:31 smithi177 ceph-mon[44303]: pgmap v2000: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s; 11631/235272 objects misplaced (4.944%); 1.6 MiB/s, 25 objects/s recovering 2024-05-15T13:20:31.737 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:31.873 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:20:31.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:31.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:20:31.980 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:20:32.013 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:32.014 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 83 16 82742 82726 62.2702 70.75 0.0140268 0.0160545 2024-05-15T13:20:32.116 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:20:32.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:32.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:20:32.220 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:20:32.253 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:32.840 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-05-15T13:20:33.014 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 84 16 83934 83918 62.4157 74.5 0.0169643 0.016017 2024-05-15T13:20:33.307 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:20:33.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:33.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:20:33.413 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:20:33.455 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:33.497 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:20:33.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:33.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:20:33.603 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:20:33.635 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:33.806 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:20:33.914 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:33 smithi120 ceph-mon[43961]: pgmap v2001: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.8 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 54 MiB/s wr, 871 op/s; 11631/237360 objects misplaced (4.900%); 888 KiB/s, 13 objects/s recovering 2024-05-15T13:20:33.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:33 smithi177 ceph-mon[44303]: pgmap v2001: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.8 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 54 MiB/s wr, 871 op/s; 11631/237360 objects misplaced (4.900%); 888 KiB/s, 13 objects/s recovering 2024-05-15T13:20:33.985 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:20:33.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:33.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:20:34.014 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 85 16 85016 85000 62.4769 67.625 0.0150383 0.0160013 2024-05-15T13:20:34.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:33 smithi027 ceph-mon[39795]: pgmap v2001: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.8 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 54 MiB/s wr, 871 op/s; 11631/237360 objects misplaced (4.900%); 888 KiB/s, 13 objects/s recovering 2024-05-15T13:20:34.089 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:20:34.123 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:34.251 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:20:34.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:34.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:20:34.356 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:20:34.373 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:20:34.373 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:20:34.393 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:34.419 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:20:34.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:34.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:20:34.524 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:20:34.583 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:35.014 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 86 16 86078 86062 62.5221 66.375 0.00722443 0.0159884 2024-05-15T13:20:35.161 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-05-15T13:20:35.321 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:20:35.486 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:20:35.672 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:35 smithi027 ceph-mon[39795]: pgmap v2002: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.1 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 81 MiB/s wr, 1.29k op/s; 11288/250332 objects misplaced (4.509%); 1.8 MiB/s, 28 objects/s recovering 2024-05-15T13:20:35.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:35 smithi177 ceph-mon[44303]: pgmap v2002: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.1 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 81 MiB/s wr, 1.29k op/s; 11288/250332 objects misplaced (4.509%); 1.8 MiB/s, 28 objects/s recovering 2024-05-15T13:20:36.014 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 87 16 87105 87089 62.5412 64.1875 0.0146748 0.0159846 2024-05-15T13:20:36.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:35 smithi120 ceph-mon[43961]: pgmap v2002: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.1 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 81 MiB/s wr, 1.29k op/s; 11288/250332 objects misplaced (4.509%); 1.8 MiB/s, 28 objects/s recovering 2024-05-15T13:20:36.456 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:20:36.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:36.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:20:36.568 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:20:36.603 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:36.640 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:20:36.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:36.644 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:20:36.675 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:20:36.675 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:20:36.675 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:20:36.677 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:20:36.713 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:20:36.745 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:20:36.746 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:20:36.769 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:20:36.773 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:36.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:36 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2926977362' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:20:36.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:36 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2926977362' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:20:36.789 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:20:36.792 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:20:36.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:36.796 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:20:36.810 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:20:36.830 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:20:36.851 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:20:36.871 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:20:36.872 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:20:36.873 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:20:36.873 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:20:36.873 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:20:36.873 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:20:36.873 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:20:36.873 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:20:36.873 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:20:36.873 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:20:36.873 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:20:36.873 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:20:36.873 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:20:36.896 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:20:36.929 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:36.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:36 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2926977362' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:20:37.014 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 88 16 87994 87978 62.4619 55.5625 0.0127125 0.0160052 2024-05-15T13:20:37.565 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:20:37.565 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T13:20:37.566 INFO:teuthology.orchestra.run.smithi120.stdout: "num_ops": 0 2024-05-15T13:20:37.566 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:20:37.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:37 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:20:37.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:37 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:20:37.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:37 smithi120 ceph-mon[43961]: pgmap v2003: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s; 11288/253329 objects misplaced (4.456%); 1.5 MiB/s, 23 objects/s recovering 2024-05-15T13:20:37.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:37 smithi120 ceph-mon[43961]: osdmap e776: 12 total, 12 up, 11 in 2024-05-15T13:20:38.014 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 89 16 89089 89073 62.5289 68.4375 0.00444721 0.0159736 2024-05-15T13:20:38.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:37 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:20:38.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:37 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:20:38.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:37 smithi027 ceph-mon[39795]: pgmap v2003: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s; 11288/253329 objects misplaced (4.456%); 1.5 MiB/s, 23 objects/s recovering 2024-05-15T13:20:38.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:37 smithi027 ceph-mon[39795]: osdmap e776: 12 total, 12 up, 11 in 2024-05-15T13:20:38.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:37 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:20:38.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:37 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:20:38.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:37 smithi177 ceph-mon[44303]: pgmap v2003: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s; 11288/253329 objects misplaced (4.456%); 1.5 MiB/s, 23 objects/s recovering 2024-05-15T13:20:38.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:37 smithi177 ceph-mon[44303]: osdmap e776: 12 total, 12 up, 11 in 2024-05-15T13:20:38.461 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-05-15T13:20:38.793 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:20:39.015 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Total time run: 90.0202 2024-05-15T13:20:39.015 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Total writes made: 90059 2024-05-15T13:20:39.015 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Write size: 65536 2024-05-15T13:20:39.015 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Object size: 65536 2024-05-15T13:20:39.015 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Bandwidth (MB/sec): 62.527 2024-05-15T13:20:39.015 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Stddev Bandwidth: 52.1168 2024-05-15T13:20:39.015 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Max bandwidth (MB/sec): 182.562 2024-05-15T13:20:39.015 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Min bandwidth (MB/sec): 0 2024-05-15T13:20:39.015 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Average IOPS: 1000 2024-05-15T13:20:39.016 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Stddev IOPS: 833.878 2024-05-15T13:20:39.016 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Max IOPS: 2921 2024-05-15T13:20:39.016 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Min IOPS: 0 2024-05-15T13:20:39.016 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Average Latency(s): 0.0159888 2024-05-15T13:20:39.016 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Stddev Latency(s): 0.284068 2024-05-15T13:20:39.016 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Max latency(s): 15.0048 2024-05-15T13:20:39.016 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Min latency(s): 0.00105016 2024-05-15T13:20:39.029 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Cleaning up (deleting benchmark objects) 2024-05-15T13:20:40.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:39 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:20:40.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:39 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:20:40.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:39 smithi027 ceph-mon[39795]: pgmap v2005: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.3 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s; 10946/261990 objects misplaced (4.178%); 1.7 MiB/s, 27 objects/s recovering 2024-05-15T13:20:40.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:39 smithi027 ceph-mon[39795]: osdmap e777: 12 total, 12 up, 11 in 2024-05-15T13:20:40.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:39 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:20:40.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:39 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:20:40.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:39 smithi120 ceph-mon[43961]: pgmap v2005: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.3 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s; 10946/261990 objects misplaced (4.178%); 1.7 MiB/s, 27 objects/s recovering 2024-05-15T13:20:40.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:39 smithi120 ceph-mon[43961]: osdmap e777: 12 total, 12 up, 11 in 2024-05-15T13:20:40.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:39 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:20:40.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:39 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:20:40.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:39 smithi177 ceph-mon[44303]: pgmap v2005: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.3 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s; 10946/261990 objects misplaced (4.178%); 1.7 MiB/s, 27 objects/s recovering 2024-05-15T13:20:40.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:39 smithi177 ceph-mon[44303]: osdmap e777: 12 total, 12 up, 11 in 2024-05-15T13:20:40.722 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:20:40.722 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T13:20:40.722 INFO:teuthology.orchestra.run.smithi120.stdout: "complaint_time": 30, 2024-05-15T13:20:40.722 INFO:teuthology.orchestra.run.smithi120.stdout: "num_blocked_ops": 0 2024-05-15T13:20:40.722 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:20:41.608 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-05-15T13:20:41.931 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:20:42.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:41 smithi027 ceph-mon[39795]: pgmap v2007: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.4 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 78 MiB/s wr, 1.27k op/s; 10946/266520 objects misplaced (4.107%); 1.8 MiB/s, 28 objects/s recovering 2024-05-15T13:20:42.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:41 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:20:42.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:41 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:20:42.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:41 smithi027 ceph-mon[39795]: osdmap e778: 12 total, 12 up, 11 in 2024-05-15T13:20:42.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:41 smithi120 ceph-mon[43961]: pgmap v2007: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.4 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 78 MiB/s wr, 1.27k op/s; 10946/266520 objects misplaced (4.107%); 1.8 MiB/s, 28 objects/s recovering 2024-05-15T13:20:42.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:41 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:20:42.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:41 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:20:42.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:41 smithi120 ceph-mon[43961]: osdmap e778: 12 total, 12 up, 11 in 2024-05-15T13:20:42.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:41 smithi177 ceph-mon[44303]: pgmap v2007: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.4 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 78 MiB/s wr, 1.27k op/s; 10946/266520 objects misplaced (4.107%); 1.8 MiB/s, 28 objects/s recovering 2024-05-15T13:20:42.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:41 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:20:42.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:41 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:20:42.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:41 smithi177 ceph-mon[44303]: osdmap e778: 12 total, 12 up, 11 in 2024-05-15T13:20:42.281 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:20:42.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:42.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:20:42.404 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:20:42.449 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:42.486 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:20:42.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:42.497 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:20:42.597 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:20:42.643 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:42.671 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:20:42.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:42.674 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:20:42.754 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:42 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:20:42.754 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:42 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:20:42.754 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:42 smithi027 ceph-mon[39795]: osdmap e779: 12 total, 12 up, 11 in 2024-05-15T13:20:42.755 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:42 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:20:42.755 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:42 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:20:42.755 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:42 smithi177 ceph-mon[44303]: osdmap e779: 12 total, 12 up, 11 in 2024-05-15T13:20:42.775 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:20:42.809 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:43.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:42 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:20:43.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:42 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:20:43.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:42 smithi120 ceph-mon[43961]: osdmap e779: 12 total, 12 up, 11 in 2024-05-15T13:20:43.094 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:20:43.961 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:20:43.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:43.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:20:44.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:43 smithi027 ceph-mon[39795]: pgmap v2010: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.4 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 53 MiB/s wr, 937 op/s; 10946/266772 objects misplaced (4.103%); 1.2 MiB/s, 19 objects/s recovering 2024-05-15T13:20:44.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:43 smithi120 ceph-mon[43961]: pgmap v2010: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.4 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 53 MiB/s wr, 937 op/s; 10946/266772 objects misplaced (4.103%); 1.2 MiB/s, 19 objects/s recovering 2024-05-15T13:20:44.065 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:20:44.108 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:43 smithi177 ceph-mon[44303]: pgmap v2010: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.4 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 53 MiB/s wr, 937 op/s; 10946/266772 objects misplaced (4.103%); 1.2 MiB/s, 19 objects/s recovering 2024-05-15T13:20:44.120 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:44.144 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:20:44.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:44.150 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:20:44.251 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:20:44.286 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:44.316 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:20:44.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:44.324 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:20:44.425 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:20:44.460 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:44.486 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:20:44.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:44.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:20:44.590 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:20:44.617 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:44.816 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:20:44.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:44.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:20:44.921 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:20:44.958 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:44.987 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:20:44.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:44.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:20:45.091 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:20:45.124 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:45.136 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:20:45.291 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:20:45.291 INFO:teuthology.orchestra.run.smithi120.stdout: "size": 20, 2024-05-15T13:20:45.291 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 600, 2024-05-15T13:20:45.291 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [ 2024-05-15T13:20:45.291 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.291 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26299.0:21385 8.e e765/762 8:7ad0f1fb:::benchmark_data_smithi177_149673_object21384:head v 765'1335, mlcod=765'1333)", 2024-05-15T13:20:45.291 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:19:47.265360+0000", 2024-05-15T13:20:45.292 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 58.025614582000003, 2024-05-15T13:20:45.292 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.011842765999999999, 2024-05-15T13:20:45.292 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:45.292 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:45.292 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:45.292 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.292 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:45.292 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.265360+0000", 2024-05-15T13:20:45.292 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.292 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.292 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.293 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:45.293 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.265360+0000", 2024-05-15T13:20:45.293 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.293 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.293 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.293 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:45.293 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.265361+0000", 2024-05-15T13:20:45.293 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3099999999999997e-07 2024-05-15T13:20:45.293 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.293 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.294 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:45.294 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.265418+0000", 2024-05-15T13:20:45.294 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7210999999999998e-05 2024-05-15T13:20:45.294 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.294 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.294 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:45.294 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.265433+0000", 2024-05-15T13:20:45.294 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5024999999999999e-05 2024-05-15T13:20:45.294 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.294 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.294 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:45.294 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.265442+0000", 2024-05-15T13:20:45.294 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.9560000000000003e-06 2024-05-15T13:20:45.294 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.295 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.295 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:45.295 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.265493+0000", 2024-05-15T13:20:45.295 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0856000000000002e-05 2024-05-15T13:20:45.295 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.295 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.295 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:45.295 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.265512+0000", 2024-05-15T13:20:45.295 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9181000000000002e-05 2024-05-15T13:20:45.295 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.295 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.295 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:45.295 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.277166+0000", 2024-05-15T13:20:45.295 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.011653894 2024-05-15T13:20:45.296 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.296 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.296 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:45.296 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.277203+0000", 2024-05-15T13:20:45.296 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.6912000000000002e-05 2024-05-15T13:20:45.296 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.296 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:45.296 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.296 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.296 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.296 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26299.0:21389 8.e e765/762 8:7d3ed34a:::benchmark_data_smithi177_149673_object21388:head v 765'1336, mlcod=765'1333)", 2024-05-15T13:20:45.296 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:19:47.267744+0000", 2024-05-15T13:20:45.296 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 58.023230677999997, 2024-05-15T13:20:45.296 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0094781220000000003, 2024-05-15T13:20:45.297 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:45.297 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:45.297 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:45.297 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.297 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:45.297 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.267744+0000", 2024-05-15T13:20:45.297 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.298 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.298 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.298 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:45.298 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.267744+0000", 2024-05-15T13:20:45.298 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.298 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.298 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.298 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:45.298 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.267744+0000", 2024-05-15T13:20:45.298 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6599999999999996e-07 2024-05-15T13:20:45.298 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.298 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.299 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:45.299 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.267812+0000", 2024-05-15T13:20:45.299 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7589000000000006e-05 2024-05-15T13:20:45.299 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.299 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.299 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:45.299 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.267821+0000", 2024-05-15T13:20:45.299 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1349999999999998e-06 2024-05-15T13:20:45.299 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.299 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.299 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:45.299 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.267826+0000", 2024-05-15T13:20:45.299 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9849999999999997e-06 2024-05-15T13:20:45.299 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.300 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.300 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:45.300 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.267858+0000", 2024-05-15T13:20:45.300 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.2162999999999997e-05 2024-05-15T13:20:45.300 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.300 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.300 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:45.300 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.267872+0000", 2024-05-15T13:20:45.300 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3497e-05 2024-05-15T13:20:45.300 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.300 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.300 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:45.300 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.277206+0000", 2024-05-15T13:20:45.300 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0093337699999999999 2024-05-15T13:20:45.301 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.301 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.301 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:45.301 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.277222+0000", 2024-05-15T13:20:45.301 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6416999999999999e-05 2024-05-15T13:20:45.301 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.301 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:45.301 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.301 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.301 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.301 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26299.0:21391 8.e e765/762 8:78226f0b:::benchmark_data_smithi177_149673_object21390:head v 765'1337, mlcod=765'1333)", 2024-05-15T13:20:45.301 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:19:47.267836+0000", 2024-05-15T13:20:45.301 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 58.023138287999998, 2024-05-15T13:20:45.301 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0094068910000000006, 2024-05-15T13:20:45.302 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:45.302 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:45.302 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:45.302 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.302 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:45.302 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.267836+0000", 2024-05-15T13:20:45.302 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.302 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.302 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.302 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:45.302 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.267836+0000", 2024-05-15T13:20:45.302 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.302 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.302 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.303 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:45.303 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.267836+0000", 2024-05-15T13:20:45.303 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9399999999999999e-07 2024-05-15T13:20:45.303 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.303 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.303 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:45.303 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.267912+0000", 2024-05-15T13:20:45.303 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.6031000000000004e-05 2024-05-15T13:20:45.303 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.303 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.304 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:45.304 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.267917+0000", 2024-05-15T13:20:45.304 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7639999999999996e-06 2024-05-15T13:20:45.304 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.304 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.304 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:45.304 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.267921+0000", 2024-05-15T13:20:45.304 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5269999999999999e-06 2024-05-15T13:20:45.304 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.304 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.304 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:45.304 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.268231+0000", 2024-05-15T13:20:45.304 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00030999499999999998 2024-05-15T13:20:45.305 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.305 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.305 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:45.305 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.268245+0000", 2024-05-15T13:20:45.305 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4386000000000001e-05 2024-05-15T13:20:45.305 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.305 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.305 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:45.305 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.277223+0000", 2024-05-15T13:20:45.305 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0089782990000000003 2024-05-15T13:20:45.305 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.305 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.305 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:45.305 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:47.277243+0000", 2024-05-15T13:20:45.305 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9695e-05 2024-05-15T13:20:45.306 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.306 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:45.306 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.306 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.306 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.306 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26299.0:37144 8.e e766/762 8:736c1bf8:::benchmark_data_smithi177_149673_object37143:head v 766'2389, mlcod=766'2378)", 2024-05-15T13:20:45.306 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:19:55.398751+0000", 2024-05-15T13:20:45.306 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 49.892223856000001, 2024-05-15T13:20:45.306 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0087160269999999995, 2024-05-15T13:20:45.306 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:45.306 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:45.306 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:45.306 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.307 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:45.307 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.398751+0000", 2024-05-15T13:20:45.307 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.307 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.307 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.307 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:45.307 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.398751+0000", 2024-05-15T13:20:45.307 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.307 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.307 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.308 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:45.308 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.398751+0000", 2024-05-15T13:20:45.308 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.8400000000000001e-07 2024-05-15T13:20:45.308 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.308 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.308 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:45.308 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.398799+0000", 2024-05-15T13:20:45.308 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8544000000000001e-05 2024-05-15T13:20:45.308 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.308 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.308 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:45.308 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.398807+0000", 2024-05-15T13:20:45.309 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2389999999999996e-06 2024-05-15T13:20:45.309 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.309 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.309 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:45.309 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.398810+0000", 2024-05-15T13:20:45.309 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.326e-06 2024-05-15T13:20:45.309 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.309 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.309 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:45.309 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.403570+0000", 2024-05-15T13:20:45.309 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0047596940000000001 2024-05-15T13:20:45.309 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.309 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.309 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:45.310 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.403583+0000", 2024-05-15T13:20:45.310 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3712e-05 2024-05-15T13:20:45.310 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.310 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.310 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:45.310 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.407426+0000", 2024-05-15T13:20:45.310 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0038426189999999998 2024-05-15T13:20:45.310 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.310 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.310 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:45.310 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.407467+0000", 2024-05-15T13:20:45.310 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0708999999999999e-05 2024-05-15T13:20:45.311 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.311 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:45.311 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.311 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.311 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.311 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26299.0:37147 8.e e766/762 8:77e8b3f6:::benchmark_data_smithi177_149673_object37146:head v 766'2390, mlcod=766'2378)", 2024-05-15T13:20:45.311 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:19:55.398815+0000", 2024-05-15T13:20:45.311 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 49.892159692, 2024-05-15T13:20:45.311 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0086806399999999999, 2024-05-15T13:20:45.311 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:45.311 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:45.311 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:45.312 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.312 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:45.312 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.398815+0000", 2024-05-15T13:20:45.312 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.312 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.312 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.312 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:45.312 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.398815+0000", 2024-05-15T13:20:45.312 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.312 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.312 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.312 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:45.312 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.398815+0000", 2024-05-15T13:20:45.313 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.29e-07 2024-05-15T13:20:45.313 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.313 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.313 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:45.313 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.398888+0000", 2024-05-15T13:20:45.313 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3534000000000004e-05 2024-05-15T13:20:45.313 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.313 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.313 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:45.313 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.398894+0000", 2024-05-15T13:20:45.313 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6060000000000002e-06 2024-05-15T13:20:45.313 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.314 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.314 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:45.314 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.398897+0000", 2024-05-15T13:20:45.314 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.1729999999999999e-06 2024-05-15T13:20:45.314 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.314 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.314 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:45.314 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.404434+0000", 2024-05-15T13:20:45.314 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0055367419999999999 2024-05-15T13:20:45.314 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.314 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.314 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:45.314 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.404447+0000", 2024-05-15T13:20:45.315 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2599000000000001e-05 2024-05-15T13:20:45.315 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.315 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.315 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:45.315 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.407469+0000", 2024-05-15T13:20:45.315 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00302192 2024-05-15T13:20:45.315 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.315 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.315 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:45.315 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.407495+0000", 2024-05-15T13:20:45.315 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.6937e-05 2024-05-15T13:20:45.315 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.315 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:45.315 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.316 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.316 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.316 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26299.0:37148 8.e e766/762 8:710de698:::benchmark_data_smithi177_149673_object37147:head v 766'2391, mlcod=766'2378)", 2024-05-15T13:20:45.316 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:19:55.398902+0000", 2024-05-15T13:20:45.316 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 49.892072259999999, 2024-05-15T13:20:45.316 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0094458909999999997, 2024-05-15T13:20:45.316 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:45.316 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:45.316 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:45.316 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.316 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:45.316 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.398902+0000", 2024-05-15T13:20:45.316 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.316 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.317 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.317 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:45.317 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.398902+0000", 2024-05-15T13:20:45.317 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.317 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.317 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.317 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:45.317 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.398902+0000", 2024-05-15T13:20:45.317 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.48e-07 2024-05-15T13:20:45.317 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.317 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.317 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:45.317 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.399003+0000", 2024-05-15T13:20:45.317 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00010072 2024-05-15T13:20:45.318 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.318 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.318 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:45.318 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.399009+0000", 2024-05-15T13:20:45.318 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7250000000000002e-06 2024-05-15T13:20:45.318 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.318 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.318 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:45.318 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.399012+0000", 2024-05-15T13:20:45.318 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.6100000000000002e-06 2024-05-15T13:20:45.318 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.318 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.318 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:45.319 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.404998+0000", 2024-05-15T13:20:45.319 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0059852550000000001 2024-05-15T13:20:45.319 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.319 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.319 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:45.319 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.405010+0000", 2024-05-15T13:20:45.319 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1953000000000001e-05 2024-05-15T13:20:45.319 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.319 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.319 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:45.319 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.408305+0000", 2024-05-15T13:20:45.319 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0032958029999999999 2024-05-15T13:20:45.319 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.319 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.320 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:45.320 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.408348+0000", 2024-05-15T13:20:45.320 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2676999999999998e-05 2024-05-15T13:20:45.320 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.320 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:45.320 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.320 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.320 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.320 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26299.0:37154 8.e e766/762 8:7c105233:::benchmark_data_smithi177_149673_object37153:head v 766'2392, mlcod=766'2378)", 2024-05-15T13:20:45.320 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:19:55.399019+0000", 2024-05-15T13:20:45.320 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 49.891955453999998, 2024-05-15T13:20:45.320 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0093578659999999994, 2024-05-15T13:20:45.320 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:45.320 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:45.321 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:45.321 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.321 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:45.321 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.399019+0000", 2024-05-15T13:20:45.321 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.321 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.321 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.321 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:45.321 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.399019+0000", 2024-05-15T13:20:45.321 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.321 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.321 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.321 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:45.322 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.399019+0000", 2024-05-15T13:20:45.322 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5300000000000001e-07 2024-05-15T13:20:45.322 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.322 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.322 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:45.322 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.399063+0000", 2024-05-15T13:20:45.322 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3853000000000002e-05 2024-05-15T13:20:45.322 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.322 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.322 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:45.322 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.399068+0000", 2024-05-15T13:20:45.322 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7659999999999998e-06 2024-05-15T13:20:45.322 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.323 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.323 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:45.323 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.399071+0000", 2024-05-15T13:20:45.323 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.1370000000000002e-06 2024-05-15T13:20:45.323 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.323 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.323 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:45.323 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.405572+0000", 2024-05-15T13:20:45.323 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0065008940000000001 2024-05-15T13:20:45.324 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.324 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.324 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:45.324 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.405586+0000", 2024-05-15T13:20:45.324 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.455e-05 2024-05-15T13:20:45.324 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.324 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.324 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:45.324 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.408350+0000", 2024-05-15T13:20:45.324 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0027635619999999998 2024-05-15T13:20:45.324 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.325 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.325 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:45.325 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.408377+0000", 2024-05-15T13:20:45.325 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.6951e-05 2024-05-15T13:20:45.325 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.325 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:45.325 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.325 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.325 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.325 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26299.0:37193 8.e e766/762 8:74dd9d29:::benchmark_data_smithi177_149673_object37192:head v 766'2393, mlcod=766'2378)", 2024-05-15T13:20:45.325 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:19:55.399075+0000", 2024-05-15T13:20:45.325 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 49.891899051999999, 2024-05-15T13:20:45.325 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0093298019999999999, 2024-05-15T13:20:45.326 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:45.326 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:45.326 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:45.326 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.326 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:45.326 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.399075+0000", 2024-05-15T13:20:45.326 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.326 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.326 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.326 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:45.327 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.399075+0000", 2024-05-15T13:20:45.327 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.327 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.327 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.327 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:45.327 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.399076+0000", 2024-05-15T13:20:45.327 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.43e-07 2024-05-15T13:20:45.327 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.327 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.327 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:45.327 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.399126+0000", 2024-05-15T13:20:45.327 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0482000000000002e-05 2024-05-15T13:20:45.327 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.328 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.328 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:45.328 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.399134+0000", 2024-05-15T13:20:45.328 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.3080000000000008e-06 2024-05-15T13:20:45.328 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.328 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.328 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:45.328 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.399138+0000", 2024-05-15T13:20:45.328 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.6289999999999999e-06 2024-05-15T13:20:45.328 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.328 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.328 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:45.328 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.406226+0000", 2024-05-15T13:20:45.328 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0070880600000000002 2024-05-15T13:20:45.328 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.329 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.329 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:45.329 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.406242+0000", 2024-05-15T13:20:45.329 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5804999999999999e-05 2024-05-15T13:20:45.329 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.329 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.329 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:45.329 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.408379+0000", 2024-05-15T13:20:45.329 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0021371290000000002 2024-05-15T13:20:45.329 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.329 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.329 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:45.329 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.408405+0000", 2024-05-15T13:20:45.329 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.6245999999999999e-05 2024-05-15T13:20:45.330 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.330 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:45.330 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.330 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.330 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.330 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26299.0:37223 8.e e766/762 8:75f4d0ac:::benchmark_data_smithi177_149673_object37222:head v 766'2394, mlcod=766'2378)", 2024-05-15T13:20:45.330 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:19:55.399143+0000", 2024-05-15T13:20:45.330 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 49.891831224000001, 2024-05-15T13:20:45.330 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0092918380000000002, 2024-05-15T13:20:45.330 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:45.330 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:45.330 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:45.330 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.330 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:45.331 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.399143+0000", 2024-05-15T13:20:45.331 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.331 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.331 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.331 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:45.331 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.399143+0000", 2024-05-15T13:20:45.331 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.331 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.331 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.331 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:45.331 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.399144+0000", 2024-05-15T13:20:45.331 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.4299999999999999e-07 2024-05-15T13:20:45.331 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.331 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.332 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:45.332 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.399194+0000", 2024-05-15T13:20:45.332 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0241999999999998e-05 2024-05-15T13:20:45.332 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.332 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.332 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:45.332 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.399201+0000", 2024-05-15T13:20:45.332 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8580000000000002e-06 2024-05-15T13:20:45.332 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.332 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.332 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:45.332 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.399206+0000", 2024-05-15T13:20:45.332 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2440000000000001e-06 2024-05-15T13:20:45.332 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.333 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.333 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:45.333 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.406812+0000", 2024-05-15T13:20:45.333 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0076065680000000002 2024-05-15T13:20:45.333 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.333 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.333 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:45.333 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.406830+0000", 2024-05-15T13:20:45.333 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7646e-05 2024-05-15T13:20:45.333 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.333 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.333 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:45.333 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.408408+0000", 2024-05-15T13:20:45.333 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001578082 2024-05-15T13:20:45.334 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.334 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.334 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:45.334 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:19:55.408435+0000", 2024-05-15T13:20:45.334 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.6954999999999999e-05 2024-05-15T13:20:45.334 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.334 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:45.334 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.334 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.334 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.334 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26299.0:69768 8.e e772/762 8:78246d29:::benchmark_data_smithi177_149673_object69767:head v 772'4325, mlcod=772'4323)", 2024-05-15T13:20:45.334 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:20:20.315867+0000", 2024-05-15T13:20:45.334 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 24.975107879999999, 2024-05-15T13:20:45.334 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.010226305999999999, 2024-05-15T13:20:45.335 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:45.335 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:45.335 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:45.335 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.335 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:45.335 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:20.315867+0000", 2024-05-15T13:20:45.335 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.335 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.335 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.335 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:45.335 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:20.315867+0000", 2024-05-15T13:20:45.335 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.335 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.335 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.336 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:45.336 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:20.315868+0000", 2024-05-15T13:20:45.336 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.062e-06 2024-05-15T13:20:45.336 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.336 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.336 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:45.336 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:20.315935+0000", 2024-05-15T13:20:45.336 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7535999999999998e-05 2024-05-15T13:20:45.336 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.336 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.336 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:45.336 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:20.315954+0000", 2024-05-15T13:20:45.336 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9211e-05 2024-05-15T13:20:45.337 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.337 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.337 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:45.337 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:20.315963+0000", 2024-05-15T13:20:45.337 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0080000000000008e-06 2024-05-15T13:20:45.337 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.337 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.337 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:45.337 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:20.316036+0000", 2024-05-15T13:20:45.337 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2798000000000003e-05 2024-05-15T13:20:45.337 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.337 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.337 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:45.337 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:20.316057+0000", 2024-05-15T13:20:45.338 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0381999999999999e-05 2024-05-15T13:20:45.338 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.338 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.338 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:45.338 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:20.326037+0000", 2024-05-15T13:20:45.338 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0099807130000000004 2024-05-15T13:20:45.338 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.338 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.338 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:45.338 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:20.326093+0000", 2024-05-15T13:20:45.338 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5596000000000001e-05 2024-05-15T13:20:45.338 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.338 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:45.339 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.339 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.339 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.339 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26299.0:90404 8.e e777/762 8:776f4e56:::benchmark_data_smithi177_149673_object342:head v 777'5622, mlcod=777'5619)", 2024-05-15T13:20:45.339 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:20:39.317450+0000", 2024-05-15T13:20:45.339 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 5.9735243120000003, 2024-05-15T13:20:45.339 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0080478029999999992, 2024-05-15T13:20:45.339 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:45.339 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:45.339 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:45.339 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.339 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:45.339 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:39.317450+0000", 2024-05-15T13:20:45.339 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.340 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.340 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.340 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:45.340 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:39.317450+0000", 2024-05-15T13:20:45.340 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.340 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.340 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.340 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:45.340 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:39.317451+0000", 2024-05-15T13:20:45.340 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1170000000000001e-06 2024-05-15T13:20:45.340 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.340 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.340 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:45.340 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:39.317459+0000", 2024-05-15T13:20:45.340 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2110000000000001e-06 2024-05-15T13:20:45.341 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.341 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.341 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:45.341 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:39.317471+0000", 2024-05-15T13:20:45.341 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2892e-05 2024-05-15T13:20:45.341 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.341 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.341 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:45.341 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:39.317480+0000", 2024-05-15T13:20:45.341 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1559999999999995e-06 2024-05-15T13:20:45.341 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.341 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.341 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:45.341 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:39.317774+0000", 2024-05-15T13:20:45.342 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00029484699999999999 2024-05-15T13:20:45.342 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.342 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.342 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:45.342 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:39.317798+0000", 2024-05-15T13:20:45.342 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.3969e-05 2024-05-15T13:20:45.342 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.342 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.343 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:45.343 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:39.325445+0000", 2024-05-15T13:20:45.343 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0076462910000000004 2024-05-15T13:20:45.343 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.343 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.343 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:45.343 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:39.325498+0000", 2024-05-15T13:20:45.343 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3319999999999998e-05 2024-05-15T13:20:45.343 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.343 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:45.343 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.343 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.343 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.343 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26299.0:91428 8.6 e777/762 8:64693049:::benchmark_data_smithi177_149673_object1366:head v 777'5674, mlcod=777'5672)", 2024-05-15T13:20:45.344 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:20:40.171273+0000", 2024-05-15T13:20:45.344 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 5.1197019150000003, 2024-05-15T13:20:45.344 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0081339080000000005, 2024-05-15T13:20:45.344 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:45.344 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:45.344 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:45.344 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.344 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:45.344 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:40.171273+0000", 2024-05-15T13:20:45.344 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.344 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.344 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.344 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:45.345 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:40.171273+0000", 2024-05-15T13:20:45.345 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.345 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.345 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.345 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:45.345 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:40.171274+0000", 2024-05-15T13:20:45.345 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1039999999999999e-06 2024-05-15T13:20:45.345 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.345 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.345 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:45.345 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:40.171280+0000", 2024-05-15T13:20:45.345 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9859999999999999e-06 2024-05-15T13:20:45.345 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.345 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.346 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:45.346 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:40.171293+0000", 2024-05-15T13:20:45.346 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3563e-05 2024-05-15T13:20:45.346 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.346 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.346 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:45.346 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:40.171301+0000", 2024-05-15T13:20:45.346 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9589999999999995e-06 2024-05-15T13:20:45.346 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.346 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.346 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:45.346 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:40.171382+0000", 2024-05-15T13:20:45.346 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0331e-05 2024-05-15T13:20:45.346 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.347 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.347 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:45.347 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:40.171406+0000", 2024-05-15T13:20:45.347 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.4006999999999999e-05 2024-05-15T13:20:45.347 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.347 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.347 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:45.347 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:40.179364+0000", 2024-05-15T13:20:45.347 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0079581300000000008 2024-05-15T13:20:45.347 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.347 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.347 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:45.347 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:40.179406+0000", 2024-05-15T13:20:45.347 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2827999999999997e-05 2024-05-15T13:20:45.348 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.348 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:45.348 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.348 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.348 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.348 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26299.0:92809 8.6 e778/762 8:6d477aa1:::benchmark_data_smithi177_149673_object2747:head v 778'5775, mlcod=778'5773)", 2024-05-15T13:20:45.348 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:20:41.564621+0000", 2024-05-15T13:20:45.348 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 3.7263536999999998, 2024-05-15T13:20:45.348 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0085760969999999995, 2024-05-15T13:20:45.348 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:45.348 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:45.348 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:45.348 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.348 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:45.348 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.564621+0000", 2024-05-15T13:20:45.349 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.349 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.349 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.349 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:45.349 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.564621+0000", 2024-05-15T13:20:45.349 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.349 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.349 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.349 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:45.349 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.564622+0000", 2024-05-15T13:20:45.349 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.4900000000000004e-07 2024-05-15T13:20:45.349 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.349 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.349 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:45.350 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.564630+0000", 2024-05-15T13:20:45.350 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.4020000000000002e-06 2024-05-15T13:20:45.350 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.350 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.350 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:45.350 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.564645+0000", 2024-05-15T13:20:45.350 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4878000000000001e-05 2024-05-15T13:20:45.350 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.350 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.350 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:45.350 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.564652+0000", 2024-05-15T13:20:45.350 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3150000000000003e-06 2024-05-15T13:20:45.350 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.351 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.351 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:45.351 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.564708+0000", 2024-05-15T13:20:45.351 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5868000000000001e-05 2024-05-15T13:20:45.351 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.351 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.351 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:45.351 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.564735+0000", 2024-05-15T13:20:45.351 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.7067999999999999e-05 2024-05-15T13:20:45.351 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.351 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.351 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:45.351 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.573152+0000", 2024-05-15T13:20:45.351 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0084169480000000005 2024-05-15T13:20:45.352 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.352 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.352 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:45.352 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.573197+0000", 2024-05-15T13:20:45.352 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4669000000000002e-05 2024-05-15T13:20:45.352 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.352 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:45.352 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.352 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.352 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.352 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26299.0:92813 8.6 e778/762 8:6348d723:::benchmark_data_smithi177_149673_object2751:head v 778'5776, mlcod=778'5773)", 2024-05-15T13:20:45.352 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:20:41.564670+0000", 2024-05-15T13:20:45.352 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 3.7263046179999999, 2024-05-15T13:20:45.352 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.008559232, 2024-05-15T13:20:45.353 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:45.353 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:45.353 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:45.353 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.353 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:45.353 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.564670+0000", 2024-05-15T13:20:45.353 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.353 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.353 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.353 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:45.353 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.564670+0000", 2024-05-15T13:20:45.353 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.353 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.353 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.353 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:45.354 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.564670+0000", 2024-05-15T13:20:45.354 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2000000000000001e-07 2024-05-15T13:20:45.354 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.354 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.354 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:45.354 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.564674+0000", 2024-05-15T13:20:45.354 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7450000000000001e-06 2024-05-15T13:20:45.354 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.354 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.354 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:45.354 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.564677+0000", 2024-05-15T13:20:45.354 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.3529999999999999e-06 2024-05-15T13:20:45.354 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.354 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.355 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:45.355 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.564680+0000", 2024-05-15T13:20:45.355 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.96e-06 2024-05-15T13:20:45.355 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.355 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.355 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:45.355 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.566987+0000", 2024-05-15T13:20:45.355 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0023067259999999998 2024-05-15T13:20:45.355 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.355 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.355 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:45.355 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.567002+0000", 2024-05-15T13:20:45.355 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4639e-05 2024-05-15T13:20:45.355 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.356 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.356 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:45.356 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.573202+0000", 2024-05-15T13:20:45.356 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0062007169999999997 2024-05-15T13:20:45.356 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.356 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.356 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:45.356 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.573229+0000", 2024-05-15T13:20:45.356 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.6872e-05 2024-05-15T13:20:45.356 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.356 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:45.356 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.356 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.356 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.356 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26299.0:92818 8.6 e778/762 8:690b5bcb:::benchmark_data_smithi177_149673_object2756:head v 778'5777, mlcod=778'5773)", 2024-05-15T13:20:45.357 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:20:41.564697+0000", 2024-05-15T13:20:45.357 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 3.7262770779999999, 2024-05-15T13:20:45.357 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0085598489999999996, 2024-05-15T13:20:45.357 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:45.357 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:45.357 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:45.357 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.357 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:45.357 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.564697+0000", 2024-05-15T13:20:45.357 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.357 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.357 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.357 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:45.357 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.564697+0000", 2024-05-15T13:20:45.358 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.358 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.358 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.358 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:45.358 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.564698+0000", 2024-05-15T13:20:45.358 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5099999999999999e-07 2024-05-15T13:20:45.358 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.358 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.358 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:45.358 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.564704+0000", 2024-05-15T13:20:45.358 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.0750000000000003e-06 2024-05-15T13:20:45.358 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.358 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.358 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:45.359 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.564707+0000", 2024-05-15T13:20:45.359 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5429999999999998e-06 2024-05-15T13:20:45.359 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.359 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.359 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:45.359 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.564710+0000", 2024-05-15T13:20:45.359 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.526e-06 2024-05-15T13:20:45.359 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.359 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.359 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:45.359 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.568931+0000", 2024-05-15T13:20:45.359 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0042213119999999996 2024-05-15T13:20:45.359 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.359 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.359 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:45.360 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.568945+0000", 2024-05-15T13:20:45.360 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4216000000000001e-05 2024-05-15T13:20:45.360 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.360 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.360 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:45.360 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.573232+0000", 2024-05-15T13:20:45.360 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0042863320000000003 2024-05-15T13:20:45.360 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.360 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.360 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:45.360 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.573257+0000", 2024-05-15T13:20:45.360 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.5694000000000002e-05 2024-05-15T13:20:45.360 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.360 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:45.361 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.361 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.361 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.361 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26299.0:93128 8.6 e779/762 8:6befa848:::benchmark_data_smithi177_149673_object3066:head v 779'5796, mlcod=779'5794)", 2024-05-15T13:20:45.361 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:20:41.890671+0000", 2024-05-15T13:20:45.361 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 3.4003036280000001, 2024-05-15T13:20:45.361 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0088763069999999999, 2024-05-15T13:20:45.361 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:45.361 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:45.361 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:45.361 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.361 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:45.361 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.890671+0000", 2024-05-15T13:20:45.361 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.361 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.362 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.362 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:45.362 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.890671+0000", 2024-05-15T13:20:45.362 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.362 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.362 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.362 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:45.362 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.890672+0000", 2024-05-15T13:20:45.362 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.387e-06 2024-05-15T13:20:45.362 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.362 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.362 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:45.362 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.890682+0000", 2024-05-15T13:20:45.363 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.3510000000000008e-06 2024-05-15T13:20:45.363 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.363 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.363 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:45.363 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.890701+0000", 2024-05-15T13:20:45.363 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9329000000000001e-05 2024-05-15T13:20:45.363 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.363 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.363 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:45.363 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.890711+0000", 2024-05-15T13:20:45.363 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.7229999999999999e-06 2024-05-15T13:20:45.363 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.363 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.363 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:45.364 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.890832+0000", 2024-05-15T13:20:45.364 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012087300000000001 2024-05-15T13:20:45.364 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.364 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.364 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:45.364 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.890854+0000", 2024-05-15T13:20:45.364 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2359000000000001e-05 2024-05-15T13:20:45.364 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.364 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.364 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:45.364 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.899496+0000", 2024-05-15T13:20:45.364 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0086422559999999992 2024-05-15T13:20:45.364 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.364 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.365 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:45.365 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.899547+0000", 2024-05-15T13:20:45.365 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1029000000000002e-05 2024-05-15T13:20:45.365 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.365 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:45.365 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.365 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.365 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.365 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26299.0:93132 8.e e779/762 8:7cf5f679:::benchmark_data_smithi177_149673_object3070:head v 779'5779, mlcod=779'5777)", 2024-05-15T13:20:45.365 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:20:41.890754+0000", 2024-05-15T13:20:45.365 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 3.4002205750000001, 2024-05-15T13:20:45.365 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0088281430000000001, 2024-05-15T13:20:45.365 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:45.365 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:45.365 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:45.366 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.366 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:45.366 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.890754+0000", 2024-05-15T13:20:45.366 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.366 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.366 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.366 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:45.366 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.890754+0000", 2024-05-15T13:20:45.366 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.366 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.366 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.366 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:45.366 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.890754+0000", 2024-05-15T13:20:45.367 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6400000000000003e-07 2024-05-15T13:20:45.367 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.367 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.367 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:45.367 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.890759+0000", 2024-05-15T13:20:45.367 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.826e-06 2024-05-15T13:20:45.367 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.367 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.367 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:45.367 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.890766+0000", 2024-05-15T13:20:45.367 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.002e-06 2024-05-15T13:20:45.367 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.367 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.367 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:45.368 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.890771+0000", 2024-05-15T13:20:45.368 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0690000000000001e-06 2024-05-15T13:20:45.368 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.368 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.368 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:45.368 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.893804+0000", 2024-05-15T13:20:45.368 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.003032767 2024-05-15T13:20:45.368 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.368 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.368 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:45.368 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.893826+0000", 2024-05-15T13:20:45.368 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2361000000000002e-05 2024-05-15T13:20:45.368 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.368 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.368 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:45.369 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.899551+0000", 2024-05-15T13:20:45.369 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0057245409999999997 2024-05-15T13:20:45.369 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.369 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.369 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:45.369 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.899582+0000", 2024-05-15T13:20:45.369 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.1112999999999998e-05 2024-05-15T13:20:45.369 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.369 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:45.369 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.369 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.369 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.369 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26299.0:93189 8.6 e779/762 8:6a2decdc:::benchmark_data_smithi177_149673_object3127:head v 779'5801, mlcod=779'5799)", 2024-05-15T13:20:45.369 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:20:41.946125+0000", 2024-05-15T13:20:45.370 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 3.3448497079999999, 2024-05-15T13:20:45.370 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.010864217000000001, 2024-05-15T13:20:45.370 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:45.370 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:45.370 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:45.370 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.370 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:45.370 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.946125+0000", 2024-05-15T13:20:45.370 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.370 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.370 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.370 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:45.370 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.946125+0000", 2024-05-15T13:20:45.370 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.371 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.371 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.371 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:45.371 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.946126+0000", 2024-05-15T13:20:45.371 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0999999999999997e-07 2024-05-15T13:20:45.371 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.371 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.371 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:45.371 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.946133+0000", 2024-05-15T13:20:45.371 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.1779999999999997e-06 2024-05-15T13:20:45.371 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.371 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.371 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:45.371 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.946146+0000", 2024-05-15T13:20:45.372 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3192e-05 2024-05-15T13:20:45.372 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.372 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.372 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:45.372 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.946154+0000", 2024-05-15T13:20:45.372 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.5680000000000002e-06 2024-05-15T13:20:45.372 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.372 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.372 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:45.372 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.946210+0000", 2024-05-15T13:20:45.372 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6202999999999997e-05 2024-05-15T13:20:45.372 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.372 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.372 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:45.372 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.946232+0000", 2024-05-15T13:20:45.373 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1849e-05 2024-05-15T13:20:45.373 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.373 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.373 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:45.373 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.956932+0000", 2024-05-15T13:20:45.373 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.010699916 2024-05-15T13:20:45.373 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.373 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.373 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:45.373 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:41.956989+0000", 2024-05-15T13:20:45.373 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7500999999999997e-05 2024-05-15T13:20:45.373 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.373 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:45.373 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.373 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.374 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.374 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26299.0:95544 8.e e779/762 8:79959d87:::benchmark_data_smithi177_149673_object5482:head v 779'5931, mlcod=779'5929)", 2024-05-15T13:20:45.374 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:20:44.175073+0000", 2024-05-15T13:20:45.374 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 1.115901663, 2024-05-15T13:20:45.374 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.010457744, 2024-05-15T13:20:45.374 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:45.374 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:45.374 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:45.374 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.374 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:45.374 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:44.175073+0000", 2024-05-15T13:20:45.374 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.374 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.374 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.375 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:45.375 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:44.175073+0000", 2024-05-15T13:20:45.375 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.375 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.375 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.375 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:45.375 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:44.175073+0000", 2024-05-15T13:20:45.375 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1799999999999995e-07 2024-05-15T13:20:45.375 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.375 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.375 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:45.375 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:44.175078+0000", 2024-05-15T13:20:45.375 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9389999999999996e-06 2024-05-15T13:20:45.375 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.375 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.376 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:45.376 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:44.175090+0000", 2024-05-15T13:20:45.376 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1107999999999999e-05 2024-05-15T13:20:45.376 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.376 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.376 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:45.376 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:44.175095+0000", 2024-05-15T13:20:45.376 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3499999999999996e-06 2024-05-15T13:20:45.376 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.376 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.376 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:45.376 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:44.175146+0000", 2024-05-15T13:20:45.376 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1252999999999999e-05 2024-05-15T13:20:45.376 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.377 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.377 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:45.377 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:44.175167+0000", 2024-05-15T13:20:45.377 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0601000000000001e-05 2024-05-15T13:20:45.377 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.377 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.377 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:45.377 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:44.185488+0000", 2024-05-15T13:20:45.377 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.010321415 2024-05-15T13:20:45.377 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.377 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.377 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:45.377 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:44.185531+0000", 2024-05-15T13:20:45.377 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2459999999999997e-05 2024-05-15T13:20:45.377 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.378 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:45.378 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.378 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.378 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.378 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26299.0:95746 8.6 e779/762 8:6e38ff31:::benchmark_data_smithi177_149673_object5684:head v 779'5983, mlcod=779'5980)", 2024-05-15T13:20:45.378 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:20:44.385185+0000", 2024-05-15T13:20:45.378 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.90578915699999996, 2024-05-15T13:20:45.378 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0079429770000000004, 2024-05-15T13:20:45.378 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:20:45.378 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:20:45.378 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:20:45.378 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.378 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:20:45.379 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:44.385185+0000", 2024-05-15T13:20:45.379 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.379 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.379 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.379 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:20:45.379 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:44.385185+0000", 2024-05-15T13:20:45.379 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:20:45.379 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.379 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.379 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:20:45.379 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:44.385186+0000", 2024-05-15T13:20:45.379 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.0800000000000001e-07 2024-05-15T13:20:45.379 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.379 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.380 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:20:45.380 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:44.385190+0000", 2024-05-15T13:20:45.380 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3440000000000003e-06 2024-05-15T13:20:45.380 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.380 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.380 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:20:45.380 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:44.385195+0000", 2024-05-15T13:20:45.380 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6990000000000002e-06 2024-05-15T13:20:45.380 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.380 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.380 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:20:45.380 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:44.385198+0000", 2024-05-15T13:20:45.380 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.523e-06 2024-05-15T13:20:45.380 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.380 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.381 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:20:45.381 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:44.387610+0000", 2024-05-15T13:20:45.381 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.002412076 2024-05-15T13:20:45.381 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.381 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.381 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:20:45.381 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:44.387631+0000", 2024-05-15T13:20:45.381 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0333999999999999e-05 2024-05-15T13:20:45.381 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.381 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.381 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:20:45.381 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:44.393076+0000", 2024-05-15T13:20:45.381 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0054457180000000004 2024-05-15T13:20:45.381 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:20:45.382 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:20:45.382 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:20:45.382 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:20:44.393128+0000", 2024-05-15T13:20:45.382 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1975000000000003e-05 2024-05-15T13:20:45.382 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.382 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:45.382 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.382 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:20:45.382 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:20:45.382 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:20:45.531 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:45 smithi027 ceph-mon[39795]: pgmap v2011: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.3 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 170 B/s rd, 28 MiB/s wr, 1.10k op/s; 10562/258510 objects misplaced (4.086%); 1.3 MiB/s, 21 objects/s recovering 2024-05-15T13:20:45.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:45 smithi120 ceph-mon[43961]: pgmap v2011: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.3 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 170 B/s rd, 28 MiB/s wr, 1.10k op/s; 10562/258510 objects misplaced (4.086%); 1.3 MiB/s, 21 objects/s recovering 2024-05-15T13:20:45.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:45 smithi177 ceph-mon[44303]: pgmap v2011: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.3 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 170 B/s rd, 28 MiB/s wr, 1.10k op/s; 10562/258510 objects misplaced (4.086%); 1.3 MiB/s, 21 objects/s recovering 2024-05-15T13:20:46.086 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-05-15T13:20:46.519 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:20:47.611 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:20:47.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:47.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:20:47.663 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-05-15T13:20:47.672 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:20:47.727 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:20:47.788 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:47.877 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:47 smithi177 ceph-mon[44303]: pgmap v2012: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.2 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 129 B/s rd, 14 MiB/s wr, 825 op/s; 10562/255792 objects misplaced (4.129%); 1.0 MiB/s, 16 objects/s recovering 2024-05-15T13:20:47.914 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:20:47.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:47.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:20:47.926 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:47 smithi027 ceph-mon[39795]: pgmap v2012: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.2 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 129 B/s rd, 14 MiB/s wr, 825 op/s; 10562/255792 objects misplaced (4.129%); 1.0 MiB/s, 16 objects/s recovering 2024-05-15T13:20:48.021 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:20:48.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:47 smithi120 ceph-mon[43961]: pgmap v2012: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.2 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 129 B/s rd, 14 MiB/s wr, 825 op/s; 10562/255792 objects misplaced (4.129%); 1.0 MiB/s, 16 objects/s recovering 2024-05-15T13:20:48.069 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:48.164 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:20:48.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:48.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:20:48.269 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:20:48.299 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:48.322 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:20:48.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:48.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:20:48.428 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:20:48.457 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:48.479 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:20:48.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:48.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:20:48.584 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:20:48.649 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:48.771 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:20:48.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:48.775 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:20:48.876 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:20:48.942 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:48.957 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:20:48.957 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T13:20:48.957 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.957 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26299.0:100250 8.b 8:d03d57fe:::benchmark_data_smithi177_149673_object10188:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e779)", 2024-05-15T13:20:48.957 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:20:48.938917+0000", 2024-05-15T13:20:48.957 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.016901935999999999, 2024-05-15T13:20:48.957 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.016934792000000001, 2024-05-15T13:20:48.957 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:20:48.957 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:20:48.957 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:20:48.957 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26299", 2024-05-15T13:20:48.957 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:20:48.958 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 100250 2024-05-15T13:20:48.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.958 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:20:48.958 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.958 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:20:48.958 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.938917+0000", 2024-05-15T13:20:48.958 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:20:48.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.958 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.958 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:20:48.958 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.938917+0000", 2024-05-15T13:20:48.958 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:20:48.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:20:48.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.938919+0000", 2024-05-15T13:20:48.959 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1569999999999998e-06 2024-05-15T13:20:48.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:20:48.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.938928+0000", 2024-05-15T13:20:48.959 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7790000000000001e-06 2024-05-15T13:20:48.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:20:48.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.938946+0000", 2024-05-15T13:20:48.960 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7972e-05 2024-05-15T13:20:48.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.960 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.960 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:20:48.960 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.938958+0000", 2024-05-15T13:20:48.960 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2544999999999999e-05 2024-05-15T13:20:48.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.960 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.960 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:20:48.960 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.940347+0000", 2024-05-15T13:20:48.960 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001389036 2024-05-15T13:20:48.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.960 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.960 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:20:48.960 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.940465+0000", 2024-05-15T13:20:48.961 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117742 2024-05-15T13:20:48.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.961 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,11", 2024-05-15T13:20:48.961 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.940578+0000", 2024-05-15T13:20:48.961 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011251699999999999 2024-05-15T13:20:48.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.961 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:20:48.961 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.947435+0000", 2024-05-15T13:20:48.961 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0068575429999999998 2024-05-15T13:20:48.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.961 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:20:48.962 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.950683+0000", 2024-05-15T13:20:48.962 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003248201 2024-05-15T13:20:48.962 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:20:48.962 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:20:48.962 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:20:48.962 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.962 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.962 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26299.0:100257 8.b 8:d7ad5b0c:::benchmark_data_smithi177_149673_object10195:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e779)", 2024-05-15T13:20:48.962 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:20:48.942718+0000", 2024-05-15T13:20:48.962 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.013100752, 2024-05-15T13:20:48.962 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013225693, 2024-05-15T13:20:48.962 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:20:48.962 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:20:48.963 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:20:48.963 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26299", 2024-05-15T13:20:48.963 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:20:48.963 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 100257 2024-05-15T13:20:48.963 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.963 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:20:48.963 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.963 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:20:48.963 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.942718+0000", 2024-05-15T13:20:48.963 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:20:48.963 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.963 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.963 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:20:48.964 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.942718+0000", 2024-05-15T13:20:48.964 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:20:48.964 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.964 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.964 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:20:48.964 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.942720+0000", 2024-05-15T13:20:48.964 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.73e-06 2024-05-15T13:20:48.964 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.964 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.964 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:20:48.964 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.942726+0000", 2024-05-15T13:20:48.964 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.601e-06 2024-05-15T13:20:48.964 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.964 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.965 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:20:48.965 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.942741+0000", 2024-05-15T13:20:48.965 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4283e-05 2024-05-15T13:20:48.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.965 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.965 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:20:48.965 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.942750+0000", 2024-05-15T13:20:48.965 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6809999999999993e-06 2024-05-15T13:20:48.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.965 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.965 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:20:48.965 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.943466+0000", 2024-05-15T13:20:48.965 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00071580099999999996 2024-05-15T13:20:48.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.965 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.966 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:20:48.966 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.943547+0000", 2024-05-15T13:20:48.966 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0589999999999999e-05 2024-05-15T13:20:48.966 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.966 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.966 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,11", 2024-05-15T13:20:48.966 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.943644+0000", 2024-05-15T13:20:48.966 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7588000000000001e-05 2024-05-15T13:20:48.966 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.966 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.966 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:20:48.966 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.947441+0000", 2024-05-15T13:20:48.967 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037966670000000001 2024-05-15T13:20:48.967 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.967 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.967 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:20:48.967 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.954866+0000", 2024-05-15T13:20:48.967 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0074252500000000004 2024-05-15T13:20:48.967 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:20:48.967 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:20:48.967 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:20:48.967 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.967 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.967 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26299.0:100262 8.b 8:dcdc3030:::benchmark_data_smithi177_149673_object10200:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e779)", 2024-05-15T13:20:48.967 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:20:48.951417+0000", 2024-05-15T13:20:48.967 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0044020650000000001, 2024-05-15T13:20:48.968 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0046090209999999996, 2024-05-15T13:20:48.968 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:20:48.968 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:20:48.968 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:20:48.968 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26299", 2024-05-15T13:20:48.968 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:20:48.968 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 100262 2024-05-15T13:20:48.968 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.968 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:20:48.968 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.968 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:20:48.968 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.951417+0000", 2024-05-15T13:20:48.968 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:20:48.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.969 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.969 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:20:48.969 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.951417+0000", 2024-05-15T13:20:48.969 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:20:48.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.969 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.969 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:20:48.969 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.951418+0000", 2024-05-15T13:20:48.969 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7e-06 2024-05-15T13:20:48.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.969 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.969 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:20:48.969 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.951425+0000", 2024-05-15T13:20:48.970 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5050000000000004e-06 2024-05-15T13:20:48.970 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.970 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.970 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:20:48.970 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.951441+0000", 2024-05-15T13:20:48.970 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5736999999999999e-05 2024-05-15T13:20:48.970 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.970 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.970 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:20:48.970 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.951450+0000", 2024-05-15T13:20:48.970 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0890000000000006e-06 2024-05-15T13:20:48.971 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.971 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.971 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:20:48.971 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.951504+0000", 2024-05-15T13:20:48.971 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4502000000000001e-05 2024-05-15T13:20:48.971 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.971 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.971 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:20:48.971 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.951620+0000", 2024-05-15T13:20:48.971 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011537800000000001 2024-05-15T13:20:48.971 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.971 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.971 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,11", 2024-05-15T13:20:48.971 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.951752+0000", 2024-05-15T13:20:48.972 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013232000000000001 2024-05-15T13:20:48.972 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.972 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.972 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:20:48.972 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.954993+0000", 2024-05-15T13:20:48.972 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0032411620000000001 2024-05-15T13:20:48.972 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:20:48.972 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:20:48.972 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:20:48.972 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:20:48.955097+0000", 2024-05-15T13:20:48.972 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010364 2024-05-15T13:20:48.972 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:20:48.972 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:20:48.972 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:20:48.972 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:20:48.973 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-05-15T13:20:48.973 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 3 2024-05-15T13:20:48.973 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:20:49.688 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-05-15T13:20:49.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:49 smithi177 ceph-mon[44303]: pgmap v2013: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.0 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 127 B/s rd, 7.7 MiB/s wr, 1.04k op/s; 10179/247356 objects misplaced (4.115%); 2.0 MiB/s, 31 objects/s recovering 2024-05-15T13:20:50.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:49 smithi027 ceph-mon[39795]: pgmap v2013: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.0 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 127 B/s rd, 7.7 MiB/s wr, 1.04k op/s; 10179/247356 objects misplaced (4.115%); 2.0 MiB/s, 31 objects/s recovering 2024-05-15T13:20:50.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:49 smithi120 ceph-mon[43961]: pgmap v2013: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.0 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 127 B/s rd, 7.7 MiB/s wr, 1.04k op/s; 10179/247356 objects misplaced (4.115%); 2.0 MiB/s, 31 objects/s recovering 2024-05-15T13:20:50.108 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:20:50.722 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:20:50.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:50.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:20:50.743 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:20:50.829 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:20:50.892 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:51.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:50 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:20:51.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:50 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:20:51.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:50 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:20:51.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:50 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:20:51.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:50 smithi027 ceph-mon[39795]: osdmap e780: 12 total, 12 up, 11 in 2024-05-15T13:20:51.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:50 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3229865305' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:20:51.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:50 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:20:51.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:50 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:20:51.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:50 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:20:51.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:50 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:20:51.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:50 smithi120 ceph-mon[43961]: osdmap e780: 12 total, 12 up, 11 in 2024-05-15T13:20:51.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:50 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3229865305' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:20:51.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:50 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:20:51.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:50 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:20:51.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:50 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:20:51.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:50 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:20:51.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:50 smithi177 ceph-mon[44303]: osdmap e780: 12 total, 12 up, 11 in 2024-05-15T13:20:51.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:50 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3229865305' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:20:51.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:51 smithi177 ceph-mon[44303]: pgmap v2015: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.9 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 115 B/s rd, 5.3 MiB/s wr, 1.11k op/s; 10179/241746 objects misplaced (4.211%); 1.8 MiB/s, 28 objects/s recovering 2024-05-15T13:20:51.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:51 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:20:51.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:51 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3229865305' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:20:51.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:51 smithi177 ceph-mon[44303]: osdmap e781: 12 total, 12 up, 11 in 2024-05-15T13:20:52.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:51 smithi027 ceph-mon[39795]: pgmap v2015: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.9 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 115 B/s rd, 5.3 MiB/s wr, 1.11k op/s; 10179/241746 objects misplaced (4.211%); 1.8 MiB/s, 28 objects/s recovering 2024-05-15T13:20:52.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:51 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:20:52.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:51 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3229865305' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:20:52.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:51 smithi027 ceph-mon[39795]: osdmap e781: 12 total, 12 up, 11 in 2024-05-15T13:20:52.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:51 smithi120 ceph-mon[43961]: pgmap v2015: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.9 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 115 B/s rd, 5.3 MiB/s wr, 1.11k op/s; 10179/241746 objects misplaced (4.211%); 1.8 MiB/s, 28 objects/s recovering 2024-05-15T13:20:52.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:51 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:20:52.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:51 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3229865305' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:20:52.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:51 smithi120 ceph-mon[43961]: osdmap e781: 12 total, 12 up, 11 in 2024-05-15T13:20:52.067 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:20:52.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:52.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:20:52.174 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:20:52.207 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:52.771 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:20:52.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:52.776 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:20:52.877 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:20:52.948 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:52.985 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:20:52.985 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:20:52.985 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:20:52.985 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:20:52.985 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:20:52.985 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:20:52.986 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:20:52.996 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:20:52.997 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:20:53.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:53.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:20:53.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:52 smithi027 ceph-mon[39795]: 1.1 scrub starts 2024-05-15T13:20:53.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:52 smithi027 ceph-mon[39795]: 1.1 scrub ok 2024-05-15T13:20:53.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:52 smithi027 ceph-mon[39795]: 8.0 scrub starts 2024-05-15T13:20:53.019 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:20:53.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:52 smithi120 ceph-mon[43961]: 1.1 scrub starts 2024-05-15T13:20:53.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:52 smithi120 ceph-mon[43961]: 1.1 scrub ok 2024-05-15T13:20:53.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:52 smithi120 ceph-mon[43961]: 8.0 scrub starts 2024-05-15T13:20:53.054 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:20:53.081 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:20:53.102 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:52 smithi177 ceph-mon[44303]: 1.1 scrub starts 2024-05-15T13:20:53.102 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:52 smithi177 ceph-mon[44303]: 1.1 scrub ok 2024-05-15T13:20:53.102 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:52 smithi177 ceph-mon[44303]: 8.0 scrub starts 2024-05-15T13:20:53.105 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:20:53.111 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:20:53.140 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:20:53.142 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:53.154 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:20:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:20:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:20:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:20:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:20:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:20:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:20:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:20:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:20:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:20:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:20:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:20:53.257 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:20:53.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:53.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:20:53.365 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:20:53.395 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:53.417 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:20:53.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:53.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:20:53.444 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:20:53.444 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:20:53.521 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:20:53.569 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:53.597 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:20:53.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:53.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:20:53.702 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:20:53.735 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:54.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:53 smithi027 ceph-mon[39795]: pgmap v2017: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 779 op/s; 10179/239775 objects misplaced (4.245%); 1023 KiB/s, 15 objects/s recovering 2024-05-15T13:20:54.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:53 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/935735407' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:20:54.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:53 smithi027 ceph-mon[39795]: 8.0 scrub ok 2024-05-15T13:20:54.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:53 smithi120 ceph-mon[43961]: pgmap v2017: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 779 op/s; 10179/239775 objects misplaced (4.245%); 1023 KiB/s, 15 objects/s recovering 2024-05-15T13:20:54.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:53 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/935735407' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:20:54.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:53 smithi120 ceph-mon[43961]: 8.0 scrub ok 2024-05-15T13:20:54.092 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:20:54.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:54.099 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:53 smithi177 ceph-mon[44303]: pgmap v2017: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 779 op/s; 10179/239775 objects misplaced (4.245%); 1023 KiB/s, 15 objects/s recovering 2024-05-15T13:20:54.099 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:53 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/935735407' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:20:54.099 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:53 smithi177 ceph-mon[44303]: 8.0 scrub ok 2024-05-15T13:20:54.100 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:20:54.201 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:20:54.229 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:54.250 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:20:54.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:54.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:20:54.356 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:20:54.411 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:54.585 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-05-15T13:20:54.931 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:20:55.052 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:20:55.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:55 smithi027 ceph-mon[39795]: 8.c scrub starts 2024-05-15T13:20:55.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:55 smithi027 ceph-mon[39795]: pgmap v2018: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s; 9704/229443 objects misplaced (4.229%); 2.2 MiB/s, 35 objects/s recovering 2024-05-15T13:20:55.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:55 smithi120 ceph-mon[43961]: 8.c scrub starts 2024-05-15T13:20:55.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:55 smithi120 ceph-mon[43961]: pgmap v2018: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s; 9704/229443 objects misplaced (4.229%); 2.2 MiB/s, 35 objects/s recovering 2024-05-15T13:20:55.587 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:55 smithi177 ceph-mon[44303]: 8.c scrub starts 2024-05-15T13:20:55.588 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:55 smithi177 ceph-mon[44303]: pgmap v2018: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s; 9704/229443 objects misplaced (4.229%); 2.2 MiB/s, 35 objects/s recovering 2024-05-15T13:20:56.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:56 smithi177 ceph-mon[44303]: 8.c scrub ok 2024-05-15T13:20:56.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:56 smithi177 ceph-mon[44303]: 1.13 scrub starts 2024-05-15T13:20:56.531 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:56 smithi027 ceph-mon[39795]: 8.c scrub ok 2024-05-15T13:20:56.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:56 smithi027 ceph-mon[39795]: 1.13 scrub starts 2024-05-15T13:20:56.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:56 smithi120 ceph-mon[43961]: 8.c scrub ok 2024-05-15T13:20:56.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:56 smithi120 ceph-mon[43961]: 1.13 scrub starts 2024-05-15T13:20:57.112 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:20:57.112 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:20:57.227 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:57 smithi027 ceph-mon[39795]: 1.13 scrub ok 2024-05-15T13:20:57.227 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:57 smithi027 ceph-mon[39795]: 8.1 scrub starts 2024-05-15T13:20:57.227 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:57 smithi027 ceph-mon[39795]: 1.d scrub starts 2024-05-15T13:20:57.227 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:57 smithi027 ceph-mon[39795]: pgmap v2019: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.6 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 839 op/s; 9704/227211 objects misplaced (4.271%); 1.2 MiB/s, 19 objects/s recovering 2024-05-15T13:20:57.227 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:57 smithi027 ceph-mon[39795]: 1.d scrub ok 2024-05-15T13:20:57.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:57 smithi177 ceph-mon[44303]: 1.13 scrub ok 2024-05-15T13:20:57.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:57 smithi177 ceph-mon[44303]: 8.1 scrub starts 2024-05-15T13:20:57.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:57 smithi177 ceph-mon[44303]: 1.d scrub starts 2024-05-15T13:20:57.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:57 smithi177 ceph-mon[44303]: pgmap v2019: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.6 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 839 op/s; 9704/227211 objects misplaced (4.271%); 1.2 MiB/s, 19 objects/s recovering 2024-05-15T13:20:57.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:57 smithi177 ceph-mon[44303]: 1.d scrub ok 2024-05-15T13:20:57.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:57 smithi120 ceph-mon[43961]: 1.13 scrub ok 2024-05-15T13:20:57.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:57 smithi120 ceph-mon[43961]: 8.1 scrub starts 2024-05-15T13:20:57.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:57 smithi120 ceph-mon[43961]: 1.d scrub starts 2024-05-15T13:20:57.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:57 smithi120 ceph-mon[43961]: pgmap v2019: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.6 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 839 op/s; 9704/227211 objects misplaced (4.271%); 1.2 MiB/s, 19 objects/s recovering 2024-05-15T13:20:57.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:57 smithi120 ceph-mon[43961]: 1.d scrub ok 2024-05-15T13:20:57.689 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:20:57.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:57.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:20:57.796 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:20:57.818 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-05-15T13:20:57.825 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:57.847 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:20:57.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:57.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:20:57.951 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:20:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:58.214 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:20:58.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:58 smithi027 ceph-mon[39795]: 1.6 scrub starts 2024-05-15T13:20:58.511 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:58 smithi177 ceph-mon[44303]: 1.6 scrub starts 2024-05-15T13:20:58.511 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:58 smithi177 ceph-mon[44303]: 1.6 scrub ok 2024-05-15T13:20:58.511 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:58 smithi177 ceph-mon[44303]: 8.1 scrub ok 2024-05-15T13:20:58.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:58 smithi120 ceph-mon[43961]: 1.6 scrub starts 2024-05-15T13:20:58.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:58 smithi120 ceph-mon[43961]: 1.6 scrub ok 2024-05-15T13:20:58.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:58 smithi120 ceph-mon[43961]: 8.1 scrub ok 2024-05-15T13:20:58.757 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:58 smithi027 ceph-mon[39795]: 1.6 scrub ok 2024-05-15T13:20:58.757 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:58 smithi027 ceph-mon[39795]: 8.1 scrub ok 2024-05-15T13:20:58.777 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:20:58.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:58.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:20:58.821 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:20:58.883 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:20:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:58.989 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:20:58.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:58.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:20:59.098 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:20:59.126 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:59.148 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:20:59.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:20:59.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:20:59.255 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:20:59.306 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:20:59.432 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:59 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T13:20:59.432 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:59 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T13:20:59.432 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:59 smithi027 ceph-mon[39795]: 1.2 scrub starts 2024-05-15T13:20:59.432 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:59 smithi027 ceph-mon[39795]: 1.2 scrub ok 2024-05-15T13:20:59.432 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:20:59 smithi027 ceph-mon[39795]: pgmap v2020: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 864 op/s; 8982/220476 objects misplaced (4.074%); 2.8 MiB/s, 44 objects/s recovering 2024-05-15T13:20:59.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:59 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T13:20:59.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:59 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T13:20:59.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:59 smithi177 ceph-mon[44303]: 1.2 scrub starts 2024-05-15T13:20:59.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:59 smithi177 ceph-mon[44303]: 1.2 scrub ok 2024-05-15T13:20:59.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:20:59 smithi177 ceph-mon[44303]: pgmap v2020: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 864 op/s; 8982/220476 objects misplaced (4.074%); 2.8 MiB/s, 44 objects/s recovering 2024-05-15T13:20:59.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:59 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T13:20:59.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:59 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T13:20:59.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:59 smithi120 ceph-mon[43961]: 1.2 scrub starts 2024-05-15T13:20:59.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:59 smithi120 ceph-mon[43961]: 1.2 scrub ok 2024-05-15T13:20:59.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:20:59 smithi120 ceph-mon[43961]: pgmap v2020: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 864 op/s; 8982/220476 objects misplaced (4.074%); 2.8 MiB/s, 44 objects/s recovering 2024-05-15T13:20:59.728 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:21:00.177 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:21:00.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:00.188 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:21:00.289 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:21:00.327 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:01.517 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:21:01.518 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:21:01.897 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:01 smithi177 ceph-mon[44303]: pgmap v2021: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 869 op/s; 8982/215661 objects misplaced (4.165%); 2.5 MiB/s, 39 objects/s recovering 2024-05-15T13:21:02.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:01 smithi027 ceph-mon[39795]: pgmap v2021: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 869 op/s; 8982/215661 objects misplaced (4.165%); 2.5 MiB/s, 39 objects/s recovering 2024-05-15T13:21:02.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:01 smithi120 ceph-mon[43961]: pgmap v2021: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 869 op/s; 8982/215661 objects misplaced (4.165%); 2.5 MiB/s, 39 objects/s recovering 2024-05-15T13:21:02.391 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:21:02.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:02.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:21:02.502 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:21:02.531 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:02.555 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:21:02.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:02.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:21:02.659 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:21:02.678 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:02 smithi027 ceph-mon[39795]: 1.a scrub starts 2024-05-15T13:21:02.678 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:02 smithi027 ceph-mon[39795]: 1.a scrub ok 2024-05-15T13:21:02.678 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:02 smithi027 ceph-mon[39795]: 1.3 scrub starts 2024-05-15T13:21:02.678 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:02 smithi027 ceph-mon[39795]: 1.c scrub starts 2024-05-15T13:21:02.678 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:02 smithi027 ceph-mon[39795]: 1.3 scrub ok 2024-05-15T13:21:02.678 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:02 smithi027 ceph-mon[39795]: 1.c scrub ok 2024-05-15T13:21:02.689 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:02.712 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:21:02.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:02.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:21:02.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:02 smithi120 ceph-mon[43961]: 1.a scrub starts 2024-05-15T13:21:02.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:02 smithi120 ceph-mon[43961]: 1.a scrub ok 2024-05-15T13:21:02.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:02 smithi120 ceph-mon[43961]: 1.3 scrub starts 2024-05-15T13:21:02.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:02 smithi120 ceph-mon[43961]: 1.c scrub starts 2024-05-15T13:21:02.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:02 smithi120 ceph-mon[43961]: 1.3 scrub ok 2024-05-15T13:21:02.789 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:02 smithi120 ceph-mon[43961]: 1.c scrub ok 2024-05-15T13:21:02.816 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:21:02.870 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-05-15T13:21:02.870 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:02.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:02 smithi177 ceph-mon[44303]: 1.a scrub starts 2024-05-15T13:21:02.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:02 smithi177 ceph-mon[44303]: 1.a scrub ok 2024-05-15T13:21:02.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:02 smithi177 ceph-mon[44303]: 1.3 scrub starts 2024-05-15T13:21:02.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:02 smithi177 ceph-mon[44303]: 1.c scrub starts 2024-05-15T13:21:02.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:02 smithi177 ceph-mon[44303]: 1.3 scrub ok 2024-05-15T13:21:02.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:02 smithi177 ceph-mon[44303]: 1.c scrub ok 2024-05-15T13:21:03.231 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:21:03.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:03.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:21:03.242 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:21:03.339 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:21:03.387 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:03.524 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:21:03.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:03.526 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:21:03.627 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:21:03.658 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:03.681 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:21:03.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:03.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:21:03.726 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:03 smithi027 ceph-mon[39795]: pgmap v2022: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 742 op/s; 8982/214269 objects misplaced (4.192%); 2.1 MiB/s, 33 objects/s recovering 2024-05-15T13:21:03.771 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-05-15T13:21:03.787 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:21:03.884 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:04.029 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:21:04.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:04.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:21:04.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:03 smithi120 ceph-mon[43961]: pgmap v2022: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 742 op/s; 8982/214269 objects misplaced (4.192%); 2.1 MiB/s, 33 objects/s recovering 2024-05-15T13:21:04.134 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:21:04.173 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:03 smithi177 ceph-mon[44303]: pgmap v2022: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 742 op/s; 8982/214269 objects misplaced (4.192%); 2.1 MiB/s, 33 objects/s recovering 2024-05-15T13:21:04.177 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:04.219 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:21:04.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:04.237 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:21:04.338 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:21:04.339 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:21:04.371 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:04.626 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:21:04.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:04.639 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:21:04.740 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:21:04.808 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:05.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:05 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:21:05.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:05 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:21:05.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:05 smithi027 ceph-mon[39795]: pgmap v2023: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 958 op/s; 8174/205263 objects misplaced (3.982%); 3.5 MiB/s, 55 objects/s recovering 2024-05-15T13:21:05.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:05 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2872050008' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:21:05.210 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:05 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:21:05.213 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:21:05.232 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:21:05.233 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:21:05.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:05 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:21:05.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:05 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:21:05.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:05 smithi120 ceph-mon[43961]: pgmap v2023: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 958 op/s; 8174/205263 objects misplaced (3.982%); 3.5 MiB/s, 55 objects/s recovering 2024-05-15T13:21:05.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:05 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2872050008' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:21:05.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:05 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:21:05.635 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:05 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:21:05.635 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:05 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:21:05.636 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:05 smithi177 ceph-mon[44303]: pgmap v2023: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 958 op/s; 8174/205263 objects misplaced (3.982%); 3.5 MiB/s, 55 objects/s recovering 2024-05-15T13:21:05.636 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:05 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2872050008' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:21:05.636 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:05 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:21:05.981 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:21:05.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:05.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:21:06.013 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-05-15T13:21:06.090 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:21:06.118 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:06.139 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:21:06.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:06.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:21:06.245 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:21:06.273 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:06.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:06 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:21:06.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:06 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T13:21:06.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:06 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:21:06.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:06 smithi027 ceph-mon[39795]: osdmap e782: 12 total, 12 up, 11 in 2024-05-15T13:21:06.294 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:21:06.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:06.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:21:06.398 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:21:06.421 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:21:06.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:06 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:21:06.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:06 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T13:21:06.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:06 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:21:06.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:06 smithi177 ceph-mon[44303]: osdmap e782: 12 total, 12 up, 11 in 2024-05-15T13:21:06.447 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:06.482 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:21:06.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:06.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:21:06.493 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:21:06.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:06 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:21:06.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:06 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T13:21:06.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:06 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:21:06.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:06 smithi120 ceph-mon[43961]: osdmap e782: 12 total, 12 up, 11 in 2024-05-15T13:21:06.588 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:21:06.631 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:06.718 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:21:06.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:06.722 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:21:06.823 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:21:06.876 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:07.229 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:07 smithi027 ceph-mon[39795]: 8.f scrub starts 2024-05-15T13:21:07.229 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:07 smithi027 ceph-mon[39795]: pgmap v2025: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 808 op/s; 8174/202944 objects misplaced (4.028%); 3.2 MiB/s, 50 objects/s recovering 2024-05-15T13:21:07.498 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:07 smithi177 ceph-mon[44303]: 8.f scrub starts 2024-05-15T13:21:07.498 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:07 smithi177 ceph-mon[44303]: pgmap v2025: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 808 op/s; 8174/202944 objects misplaced (4.028%); 3.2 MiB/s, 50 objects/s recovering 2024-05-15T13:21:07.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:07 smithi120 ceph-mon[43961]: 8.f scrub starts 2024-05-15T13:21:07.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:07 smithi120 ceph-mon[43961]: pgmap v2025: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 4.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 808 op/s; 8174/202944 objects misplaced (4.028%); 3.2 MiB/s, 50 objects/s recovering 2024-05-15T13:21:08.519 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:21:08.519 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:21:08.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:08 smithi027 ceph-mon[39795]: 8.3 deep-scrub starts 2024-05-15T13:21:08.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:08 smithi027 ceph-mon[39795]: 8.f scrub ok 2024-05-15T13:21:08.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:08 smithi120 ceph-mon[43961]: 8.3 deep-scrub starts 2024-05-15T13:21:08.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:08 smithi120 ceph-mon[43961]: 8.f scrub ok 2024-05-15T13:21:08.550 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:08 smithi177 ceph-mon[44303]: 8.3 deep-scrub starts 2024-05-15T13:21:08.550 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:08 smithi177 ceph-mon[44303]: 8.f scrub ok 2024-05-15T13:21:09.150 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:21:09.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:09.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:21:09.258 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:21:09.279 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-05-15T13:21:09.288 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:09.308 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:21:09.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:09.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:21:09.366 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:09 smithi027 ceph-mon[39795]: pgmap v2026: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 4.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 799 op/s; 7451/196458 objects misplaced (3.793%); 3.2 MiB/s, 50 objects/s recovering 2024-05-15T13:21:09.412 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:21:09.470 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:09.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:09 smithi120 ceph-mon[43961]: pgmap v2026: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 4.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 799 op/s; 7451/196458 objects misplaced (3.793%); 3.2 MiB/s, 50 objects/s recovering 2024-05-15T13:21:09.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:09 smithi177 ceph-mon[44303]: pgmap v2026: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 4.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 799 op/s; 7451/196458 objects misplaced (3.793%); 3.2 MiB/s, 50 objects/s recovering 2024-05-15T13:21:09.719 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:21:10.361 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:21:10.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:10.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:21:10.471 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:21:10.540 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:10.701 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:21:10.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:10.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:21:10.808 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:21:10.850 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:10.992 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:21:10.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:10.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:21:11.097 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:21:11.146 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:11.634 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:21:11.680 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:21:11.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:11 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:21:11.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:11 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:21:11.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:11 smithi177 ceph-mon[44303]: osdmap e783: 12 total, 12 up, 11 in 2024-05-15T13:21:11.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:11 smithi177 ceph-mon[44303]: pgmap v2028: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 961 op/s; 7451/191172 objects misplaced (3.898%); 4.0 MiB/s, 63 objects/s recovering 2024-05-15T13:21:11.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:11 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:21:11.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:11 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2982163157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:21:11.725 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:21:11.760 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:21:11.769 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:21:11.769 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:21:11.818 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:21:11.861 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:21:11.912 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:21:11.954 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:21:11.994 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:21:11.995 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:21:12.036 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:11 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:21:12.036 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:11 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:21:12.036 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:11 smithi027 ceph-mon[39795]: osdmap e783: 12 total, 12 up, 11 in 2024-05-15T13:21:12.036 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:11 smithi027 ceph-mon[39795]: pgmap v2028: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 961 op/s; 7451/191172 objects misplaced (3.898%); 4.0 MiB/s, 63 objects/s recovering 2024-05-15T13:21:12.036 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:11 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:21:12.036 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:11 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2982163157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:21:12.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:11 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:21:12.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:11 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:21:12.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:11 smithi120 ceph-mon[43961]: osdmap e783: 12 total, 12 up, 11 in 2024-05-15T13:21:12.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:11 smithi120 ceph-mon[43961]: pgmap v2028: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 961 op/s; 7451/191172 objects misplaced (3.898%); 4.0 MiB/s, 63 objects/s recovering 2024-05-15T13:21:12.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:11 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:21:12.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:11 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2982163157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:21:12.056 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:21:12.093 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:21:12.127 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:21:12.170 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:21:12.170 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:21:12.170 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:21:12.170 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:21:12.170 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:21:12.170 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:21:12.170 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:21:12.170 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:21:12.170 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:21:12.170 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:21:12.171 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:21:12.171 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:21:12.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:12 smithi177 ceph-mon[44303]: 1.b scrub starts 2024-05-15T13:21:12.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:12 smithi177 ceph-mon[44303]: 1.b scrub ok 2024-05-15T13:21:13.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:12 smithi027 ceph-mon[39795]: 1.b scrub starts 2024-05-15T13:21:13.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:12 smithi027 ceph-mon[39795]: 1.b scrub ok 2024-05-15T13:21:13.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:12 smithi120 ceph-mon[43961]: 1.b scrub starts 2024-05-15T13:21:13.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:12 smithi120 ceph-mon[43961]: 1.b scrub ok 2024-05-15T13:21:13.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:13 smithi177 ceph-mon[44303]: 1.5 scrub starts 2024-05-15T13:21:13.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:13 smithi177 ceph-mon[44303]: 1.5 scrub ok 2024-05-15T13:21:13.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:13 smithi177 ceph-mon[44303]: pgmap v2029: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 611 op/s; 7451/190566 objects misplaced (3.910%); 1.9 MiB/s, 30 objects/s recovering 2024-05-15T13:21:14.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:13 smithi027 ceph-mon[39795]: 1.5 scrub starts 2024-05-15T13:21:14.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:13 smithi027 ceph-mon[39795]: 1.5 scrub ok 2024-05-15T13:21:14.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:13 smithi027 ceph-mon[39795]: pgmap v2029: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 611 op/s; 7451/190566 objects misplaced (3.910%); 1.9 MiB/s, 30 objects/s recovering 2024-05-15T13:21:14.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:13 smithi120 ceph-mon[43961]: 1.5 scrub starts 2024-05-15T13:21:14.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:13 smithi120 ceph-mon[43961]: 1.5 scrub ok 2024-05-15T13:21:14.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:13 smithi120 ceph-mon[43961]: pgmap v2029: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 611 op/s; 7451/190566 objects misplaced (3.910%); 1.9 MiB/s, 30 objects/s recovering 2024-05-15T13:21:14.424 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:21:14.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:14.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:21:14.466 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-05-15T13:21:14.537 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:21:14.588 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:14.621 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:21:14.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:14.626 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:21:14.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:21:14.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:21:14.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:14 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:21:14.726 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:21:14.738 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:21:14.739 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:21:14.739 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:14 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:21:14.782 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:14.836 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:21:14.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:14.844 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:21:14.889 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:21:14.945 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:21:14.972 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:21:14.972 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:21:14.972 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:14 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:21:14.974 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:14.997 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:21:15.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:15.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:21:15.101 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:21:15.151 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:15.177 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:21:15.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:15.186 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:21:15.287 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:21:15.330 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:15.440 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:21:15.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:15.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:21:15.479 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-05-15T13:21:15.548 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:21:15.625 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:21:15.637 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:15.678 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:21:15.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:15 smithi177 ceph-mon[44303]: 1.e scrub starts 2024-05-15T13:21:15.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:15 smithi177 ceph-mon[44303]: 1.e scrub ok 2024-05-15T13:21:15.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:15 smithi177 ceph-mon[44303]: pgmap v2030: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 806 op/s; 6801/182487 objects misplaced (3.727%); 3.0 MiB/s, 48 objects/s recovering 2024-05-15T13:21:15.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:15.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:21:15.790 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:21:15.818 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:15.841 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:21:15.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:15.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:21:15.948 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:21:15.973 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:15 smithi027 ceph-mon[39795]: 1.e scrub starts 2024-05-15T13:21:15.973 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:15 smithi027 ceph-mon[39795]: 1.e scrub ok 2024-05-15T13:21:15.973 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:15 smithi027 ceph-mon[39795]: pgmap v2030: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 806 op/s; 6801/182487 objects misplaced (3.727%); 3.0 MiB/s, 48 objects/s recovering 2024-05-15T13:21:15.984 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:16.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:15 smithi120 ceph-mon[43961]: 1.e scrub starts 2024-05-15T13:21:16.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:15 smithi120 ceph-mon[43961]: 1.e scrub ok 2024-05-15T13:21:16.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:15 smithi120 ceph-mon[43961]: pgmap v2030: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 806 op/s; 6801/182487 objects misplaced (3.727%); 3.0 MiB/s, 48 objects/s recovering 2024-05-15T13:21:16.886 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:21:16.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:16.890 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:21:16.991 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:21:17.038 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:17.088 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:21:17.593 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:21:17.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:17.598 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:21:17.671 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:21:17.699 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:21:17.747 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:17.789 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:21:17.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:17.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:21:17.896 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:21:17.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:17 smithi177 ceph-mon[44303]: 8.7 scrub starts 2024-05-15T13:21:17.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:17 smithi177 ceph-mon[44303]: pgmap v2031: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 746 op/s; 6801/180525 objects misplaced (3.767%); 2.9 MiB/s, 45 objects/s recovering 2024-05-15T13:21:17.935 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:17.968 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:21:17.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:17.973 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:21:17.979 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:21:17.979 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T13:21:17.979 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-05-15T13:21:17.979 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-05-15T13:21:17.979 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:21:18.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:17 smithi027 ceph-mon[39795]: 8.7 scrub starts 2024-05-15T13:21:18.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:17 smithi027 ceph-mon[39795]: pgmap v2031: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 746 op/s; 6801/180525 objects misplaced (3.767%); 2.9 MiB/s, 45 objects/s recovering 2024-05-15T13:21:18.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:17 smithi120 ceph-mon[43961]: 8.7 scrub starts 2024-05-15T13:21:18.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:17 smithi120 ceph-mon[43961]: pgmap v2031: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 746 op/s; 6801/180525 objects misplaced (3.767%); 2.9 MiB/s, 45 objects/s recovering 2024-05-15T13:21:18.075 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:21:18.137 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:18.790 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-05-15T13:21:19.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:18 smithi027 ceph-mon[39795]: 8.7 scrub ok 2024-05-15T13:21:19.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:18 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:21:19.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:18 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:21:19.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:18 smithi120 ceph-mon[43961]: 8.7 scrub ok 2024-05-15T13:21:19.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:18 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:21:19.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:18 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:21:19.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:18 smithi177 ceph-mon[44303]: 8.7 scrub ok 2024-05-15T13:21:19.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:18 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:21:19.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:18 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:21:19.249 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:21:19.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:19 smithi177 ceph-mon[44303]: pgmap v2032: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 707 op/s; 6111/175233 objects misplaced (3.487%); 2.8 MiB/s, 44 objects/s recovering 2024-05-15T13:21:19.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:19 smithi177 ceph-mon[44303]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-05-15T13:21:19.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:19 smithi177 ceph-mon[44303]: osdmap e784: 12 total, 12 up, 11 in 2024-05-15T13:21:20.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:19 smithi027 ceph-mon[39795]: pgmap v2032: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 707 op/s; 6111/175233 objects misplaced (3.487%); 2.8 MiB/s, 44 objects/s recovering 2024-05-15T13:21:20.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:19 smithi027 ceph-mon[39795]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-05-15T13:21:20.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:19 smithi027 ceph-mon[39795]: osdmap e784: 12 total, 12 up, 11 in 2024-05-15T13:21:20.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:19 smithi120 ceph-mon[43961]: pgmap v2032: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 707 op/s; 6111/175233 objects misplaced (3.487%); 2.8 MiB/s, 44 objects/s recovering 2024-05-15T13:21:20.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:19 smithi120 ceph-mon[43961]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-05-15T13:21:20.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:19 smithi120 ceph-mon[43961]: osdmap e784: 12 total, 12 up, 11 in 2024-05-15T13:21:20.520 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:21:20.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:20.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:21:20.627 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:21:20.660 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:20.683 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:21:20.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:20.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:21:20.791 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:21:20.824 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:20.846 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:21:20.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:20.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:21:20.952 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:21:20.980 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:21.001 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:21:21.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:21.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:21:21.105 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:21:21.135 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:21.156 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:21:21.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:21.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:21:21.260 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:21:21.289 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:21.311 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:21:21.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:21.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:21:21.416 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:21:21.472 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:21.720 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:21:21.750 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:21 smithi027 ceph-mon[39795]: 1.9 scrub starts 2024-05-15T13:21:21.750 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:21 smithi027 ceph-mon[39795]: pgmap v2034: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.5 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 651 op/s; 6111/171621 objects misplaced (3.561%); 2.8 MiB/s, 44 objects/s recovering 2024-05-15T13:21:21.750 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:21 smithi027 ceph-mon[39795]: 1.9 scrub ok 2024-05-15T13:21:21.750 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:21 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4114243336' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:21:21.750 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:21 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:21:21.750 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:21 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:21:21.751 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:21 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:21:21.751 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:21 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:21:21.751 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:21 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:21:21.943 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:21 smithi177 ceph-mon[44303]: 1.9 scrub starts 2024-05-15T13:21:21.943 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:21 smithi177 ceph-mon[44303]: pgmap v2034: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.5 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 651 op/s; 6111/171621 objects misplaced (3.561%); 2.8 MiB/s, 44 objects/s recovering 2024-05-15T13:21:21.943 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:21 smithi177 ceph-mon[44303]: 1.9 scrub ok 2024-05-15T13:21:21.944 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:21 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4114243336' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:21:21.944 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:21 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:21:21.944 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:21 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:21:21.944 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:21 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:21:21.944 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:21 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:21:21.944 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:21 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:21:22.018 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:21:22.018 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-05-15T13:21:22.018 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-05-15T13:21:22.018 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T13:21:22.018 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.018 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:104910 7.1 7:8d3c840f:::benchmark_data_smithi177_134484_object104909:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e643)", 2024-05-15T13:21:22.018 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:14:01.846738+0000", 2024-05-15T13:21:22.019 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 440.16893580599998, 2024-05-15T13:21:22.019 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.237600903, 2024-05-15T13:21:22.019 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:22.019 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:22.019 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:22.019 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:22.019 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:22.019 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 104910 2024-05-15T13:21:22.019 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.019 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:22.019 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.019 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:22.019 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.846738+0000", 2024-05-15T13:21:22.019 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.020 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.020 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.020 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:22.020 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.846738+0000", 2024-05-15T13:21:22.020 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.020 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.020 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.020 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:22.020 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.846740+0000", 2024-05-15T13:21:22.020 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.607e-06 2024-05-15T13:21:22.020 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.020 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.020 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:22.021 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.846763+0000", 2024-05-15T13:21:22.021 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3178999999999999e-05 2024-05-15T13:21:22.021 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.021 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.021 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:22.021 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.846777+0000", 2024-05-15T13:21:22.021 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4229999999999999e-05 2024-05-15T13:21:22.021 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.021 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.021 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:22.021 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.846785+0000", 2024-05-15T13:21:22.021 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0439999999999996e-06 2024-05-15T13:21:22.021 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.022 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.022 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.022 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.846843+0000", 2024-05-15T13:21:22.022 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7778000000000001e-05 2024-05-15T13:21:22.022 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.022 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.022 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:22.022 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.846952+0000", 2024-05-15T13:21:22.022 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109555 2024-05-15T13:21:22.022 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.022 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.022 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,11", 2024-05-15T13:21:22.023 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.847062+0000", 2024-05-15T13:21:22.023 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010967999999999999 2024-05-15T13:21:22.023 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.023 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.023 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:22.023 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.852490+0000", 2024-05-15T13:21:22.023 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0054281329999999999 2024-05-15T13:21:22.023 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.023 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.023 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.023 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.854857+0000", 2024-05-15T13:21:22.023 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023670449999999999 2024-05-15T13:21:22.023 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.024 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.024 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.024 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:02.080819+0000", 2024-05-15T13:21:22.024 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.22596133600000001 2024-05-15T13:21:22.024 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.024 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.024 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:22.024 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:02.080879+0000", 2024-05-15T13:21:22.024 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0562000000000001e-05 2024-05-15T13:21:22.024 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.024 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.024 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:22.024 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:02.084339+0000", 2024-05-15T13:21:22.024 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0034597540000000002 2024-05-15T13:21:22.025 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.025 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:22.025 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.025 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.025 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.025 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:104911 7.1 7:8a9b5ef6:::benchmark_data_smithi177_134484_object104910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e643)", 2024-05-15T13:21:22.025 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:14:01.849191+0000", 2024-05-15T13:21:22.025 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 440.16648326299998, 2024-05-15T13:21:22.025 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.237878706, 2024-05-15T13:21:22.025 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:22.025 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:22.025 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:22.025 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:22.025 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:22.026 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 104911 2024-05-15T13:21:22.026 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.026 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:22.026 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.026 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:22.026 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.849191+0000", 2024-05-15T13:21:22.026 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.026 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.026 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.026 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:22.026 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.849191+0000", 2024-05-15T13:21:22.026 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.026 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.027 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.027 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:22.027 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.849192+0000", 2024-05-15T13:21:22.027 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8899999999999998e-07 2024-05-15T13:21:22.027 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.027 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.027 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:22.027 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.849215+0000", 2024-05-15T13:21:22.027 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3530999999999999e-05 2024-05-15T13:21:22.027 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.027 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.027 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:22.027 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.849227+0000", 2024-05-15T13:21:22.027 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1493e-05 2024-05-15T13:21:22.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.028 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.028 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:22.028 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.849234+0000", 2024-05-15T13:21:22.028 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9360000000000002e-06 2024-05-15T13:21:22.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.028 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.028 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.028 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.849778+0000", 2024-05-15T13:21:22.028 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00054412600000000003 2024-05-15T13:21:22.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.028 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.028 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:22.028 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.849924+0000", 2024-05-15T13:21:22.029 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000146775 2024-05-15T13:21:22.029 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.029 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.029 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,11", 2024-05-15T13:21:22.029 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.850032+0000", 2024-05-15T13:21:22.029 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000107667 2024-05-15T13:21:22.029 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.029 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.029 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:22.029 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.854819+0000", 2024-05-15T13:21:22.029 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004786434 2024-05-15T13:21:22.029 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.029 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.029 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.030 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.854886+0000", 2024-05-15T13:21:22.030 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7799999999999995e-05 2024-05-15T13:21:22.030 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.030 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.030 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.030 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:02.084392+0000", 2024-05-15T13:21:22.030 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.22950580700000001 2024-05-15T13:21:22.030 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.030 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.030 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:22.030 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:02.084433+0000", 2024-05-15T13:21:22.030 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1211999999999998e-05 2024-05-15T13:21:22.030 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.030 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.031 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:22.031 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:02.087069+0000", 2024-05-15T13:21:22.031 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0026359360000000002 2024-05-15T13:21:22.031 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.031 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:22.031 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.031 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.031 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.031 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:104912 7.1 7:8a349762:::benchmark_data_smithi177_134484_object104911:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e643)", 2024-05-15T13:21:22.031 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:14:01.852031+0000", 2024-05-15T13:21:22.031 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 440.16364266699998, 2024-05-15T13:21:22.031 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.23767824100000001, 2024-05-15T13:21:22.031 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:22.032 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:22.032 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:22.032 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:22.032 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:22.032 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 104912 2024-05-15T13:21:22.032 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.032 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:22.032 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.032 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:22.032 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.852031+0000", 2024-05-15T13:21:22.032 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.032 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.032 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.032 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:22.033 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.852031+0000", 2024-05-15T13:21:22.033 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.033 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.033 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.033 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:22.033 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.852032+0000", 2024-05-15T13:21:22.033 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1340000000000001e-06 2024-05-15T13:21:22.033 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.033 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.033 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:22.033 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.852053+0000", 2024-05-15T13:21:22.033 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1058999999999999e-05 2024-05-15T13:21:22.033 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.033 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.034 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:22.034 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.852066+0000", 2024-05-15T13:21:22.034 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2649e-05 2024-05-15T13:21:22.034 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.034 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.034 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:22.034 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.852075+0000", 2024-05-15T13:21:22.034 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9490000000000002e-06 2024-05-15T13:21:22.034 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.034 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.034 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.034 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.852515+0000", 2024-05-15T13:21:22.034 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00044034500000000003 2024-05-15T13:21:22.034 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.035 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.035 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:22.035 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.852649+0000", 2024-05-15T13:21:22.035 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013374200000000001 2024-05-15T13:21:22.035 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.035 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.035 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,11", 2024-05-15T13:21:22.035 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.852760+0000", 2024-05-15T13:21:22.035 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111257 2024-05-15T13:21:22.035 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.035 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.035 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:22.035 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.854946+0000", 2024-05-15T13:21:22.036 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021859560000000002 2024-05-15T13:21:22.036 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.036 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.036 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.036 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:01.855185+0000", 2024-05-15T13:21:22.036 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000238354 2024-05-15T13:21:22.036 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.036 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.036 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.036 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:02.087144+0000", 2024-05-15T13:21:22.036 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.231959055 2024-05-15T13:21:22.036 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.036 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.036 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:22.037 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:02.087209+0000", 2024-05-15T13:21:22.037 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5131999999999999e-05 2024-05-15T13:21:22.037 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.037 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.037 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:22.037 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:02.089709+0000", 2024-05-15T13:21:22.037 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002500609 2024-05-15T13:21:22.037 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.037 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:22.037 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.037 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.037 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.037 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:124362 7.1 7:8eb25af3:::benchmark_data_smithi177_134484_object15297:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e656)", 2024-05-15T13:21:22.037 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:14:47.558066+0000", 2024-05-15T13:21:22.038 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 394.45760813800001, 2024-05-15T13:21:22.038 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17389491400000001, 2024-05-15T13:21:22.038 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:22.038 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:22.038 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:22.038 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:22.038 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:22.038 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 124362 2024-05-15T13:21:22.038 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.038 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:22.038 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.038 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:22.038 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.558066+0000", 2024-05-15T13:21:22.039 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.039 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.039 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.039 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:22.039 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.558066+0000", 2024-05-15T13:21:22.039 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.039 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.039 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.039 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:22.039 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.558067+0000", 2024-05-15T13:21:22.039 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.282e-06 2024-05-15T13:21:22.039 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.039 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.039 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:22.040 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.558073+0000", 2024-05-15T13:21:22.040 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5389999999999998e-06 2024-05-15T13:21:22.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.040 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.040 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:22.040 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.558088+0000", 2024-05-15T13:21:22.040 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4980000000000001e-05 2024-05-15T13:21:22.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.040 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.040 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:22.040 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.558095+0000", 2024-05-15T13:21:22.040 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9659999999999996e-06 2024-05-15T13:21:22.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.040 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.041 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.041 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.595027+0000", 2024-05-15T13:21:22.041 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.036931339000000001 2024-05-15T13:21:22.041 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.041 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.041 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for degraded object", 2024-05-15T13:21:22.041 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.595591+0000", 2024-05-15T13:21:22.041 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00056457499999999999 2024-05-15T13:21:22.041 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.041 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.041 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.041 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.647803+0000", 2024-05-15T13:21:22.041 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.052211712 2024-05-15T13:21:22.041 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.041 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.042 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:22.042 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.647877+0000", 2024-05-15T13:21:22.042 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4320999999999995e-05 2024-05-15T13:21:22.042 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.042 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.042 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,3", 2024-05-15T13:21:22.042 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.647967+0000", 2024-05-15T13:21:22.042 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9899000000000001e-05 2024-05-15T13:21:22.042 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.042 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.042 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:22.042 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.669474+0000", 2024-05-15T13:21:22.042 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021507150999999999 2024-05-15T13:21:22.042 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.043 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.043 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.043 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.669529+0000", 2024-05-15T13:21:22.043 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4085000000000001e-05 2024-05-15T13:21:22.043 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.043 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.043 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.043 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.727693+0000", 2024-05-15T13:21:22.043 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.058164902999999997 2024-05-15T13:21:22.043 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.043 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.043 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:22.043 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.727758+0000", 2024-05-15T13:21:22.043 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4752000000000006e-05 2024-05-15T13:21:22.043 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.044 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.044 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:22.044 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.731961+0000", 2024-05-15T13:21:22.044 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0042024100000000002 2024-05-15T13:21:22.044 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.044 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:22.044 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.044 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.044 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.044 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:124387 7.1 7:8a34d5db:::benchmark_data_smithi177_134484_object15322:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e656)", 2024-05-15T13:21:22.044 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:14:47.629277+0000", 2024-05-15T13:21:22.044 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 394.38639733799999, 2024-05-15T13:21:22.044 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16690745400000001, 2024-05-15T13:21:22.044 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:22.045 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:22.045 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:22.045 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:22.045 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:22.045 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 124387 2024-05-15T13:21:22.045 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.045 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:22.045 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.045 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:22.045 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.629277+0000", 2024-05-15T13:21:22.045 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.045 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.045 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.045 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:22.046 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.629277+0000", 2024-05-15T13:21:22.046 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.046 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.046 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.046 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:22.046 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.629277+0000", 2024-05-15T13:21:22.046 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6199999999999999e-07 2024-05-15T13:21:22.046 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.046 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.046 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:22.046 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.629279+0000", 2024-05-15T13:21:22.046 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4710000000000001e-06 2024-05-15T13:21:22.046 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.046 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.047 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:22.047 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.629286+0000", 2024-05-15T13:21:22.047 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5710000000000003e-06 2024-05-15T13:21:22.047 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.047 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.047 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:22.047 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.629291+0000", 2024-05-15T13:21:22.047 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2820000000000004e-06 2024-05-15T13:21:22.047 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.047 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.047 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.047 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.679108+0000", 2024-05-15T13:21:22.047 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.049816711999999999 2024-05-15T13:21:22.047 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.048 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.048 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for degraded object", 2024-05-15T13:21:22.048 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.679718+0000", 2024-05-15T13:21:22.048 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000610484 2024-05-15T13:21:22.048 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.048 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.048 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.048 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.759057+0000", 2024-05-15T13:21:22.048 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.079338800000000001 2024-05-15T13:21:22.048 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.048 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.048 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:22.048 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.759133+0000", 2024-05-15T13:21:22.048 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6120999999999998e-05 2024-05-15T13:21:22.049 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.049 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.049 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,3", 2024-05-15T13:21:22.049 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.759239+0000", 2024-05-15T13:21:22.049 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010587299999999999 2024-05-15T13:21:22.049 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.049 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.049 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:22.049 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.772310+0000", 2024-05-15T13:21:22.049 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013070576 2024-05-15T13:21:22.049 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.049 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.049 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.049 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.779869+0000", 2024-05-15T13:21:22.050 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0075594929999999996 2024-05-15T13:21:22.050 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.050 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.050 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.050 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.793336+0000", 2024-05-15T13:21:22.050 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013466506 2024-05-15T13:21:22.050 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.050 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.050 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:22.050 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.793377+0000", 2024-05-15T13:21:22.050 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0790000000000001e-05 2024-05-15T13:21:22.050 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.050 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.050 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:22.051 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:47.796184+0000", 2024-05-15T13:21:22.051 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0028074129999999999 2024-05-15T13:21:22.051 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.051 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:22.051 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.051 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.051 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.051 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:125691 7.1 7:88f1c88c:::benchmark_data_smithi177_134484_object16626:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e656)", 2024-05-15T13:21:22.051 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:14:50.401612+0000", 2024-05-15T13:21:22.051 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 391.61406207700003, 2024-05-15T13:21:22.051 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.460035682, 2024-05-15T13:21:22.051 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:22.051 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-05-15T13:21:22.051 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:22.052 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:22.052 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:22.052 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125691 2024-05-15T13:21:22.052 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.052 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:22.052 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.052 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:22.052 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.401612+0000", 2024-05-15T13:21:22.052 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.052 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.052 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.052 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:22.052 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.401612+0000", 2024-05-15T13:21:22.052 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.053 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.053 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.053 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:22.053 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.401614+0000", 2024-05-15T13:21:22.053 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7379999999999999e-06 2024-05-15T13:21:22.053 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.053 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.053 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:22.053 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.401621+0000", 2024-05-15T13:21:22.053 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.046e-06 2024-05-15T13:21:22.053 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.053 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.053 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:22.053 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.401635+0000", 2024-05-15T13:21:22.054 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4905999999999999e-05 2024-05-15T13:21:22.054 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.054 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.054 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:22.054 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.401646+0000", 2024-05-15T13:21:22.054 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0237e-05 2024-05-15T13:21:22.054 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.054 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.054 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.054 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.408622+0000", 2024-05-15T13:21:22.054 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0069759100000000001 2024-05-15T13:21:22.054 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.054 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.054 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for degraded object", 2024-05-15T13:21:22.055 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.409442+0000", 2024-05-15T13:21:22.055 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00082062300000000003 2024-05-15T13:21:22.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.055 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.055 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.451944+0000", 2024-05-15T13:21:22.055 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042501701000000003 2024-05-15T13:21:22.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.055 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:22.055 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.452045+0000", 2024-05-15T13:21:22.055 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000100741 2024-05-15T13:21:22.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.055 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,3", 2024-05-15T13:21:22.056 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.452200+0000", 2024-05-15T13:21:22.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015501499999999999 2024-05-15T13:21:22.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.056 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.056 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.056 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.462291+0000", 2024-05-15T13:21:22.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010090988 2024-05-15T13:21:22.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.056 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.056 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:22.056 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.462318+0000", 2024-05-15T13:21:22.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.72e-05 2024-05-15T13:21:22.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.056 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.056 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.057 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:51.454260+0000", 2024-05-15T13:21:22.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.99194180099999996 2024-05-15T13:21:22.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.057 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.057 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:22.057 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:51.454275+0000", 2024-05-15T13:21:22.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5344000000000001e-05 2024-05-15T13:21:22.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.057 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.057 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.057 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.829597+0000", 2024-05-15T13:21:22.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.375321722 2024-05-15T13:21:22.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.057 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.058 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for missing object", 2024-05-15T13:21:22.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.829780+0000", 2024-05-15T13:21:22.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018291100000000001 2024-05-15T13:21:22.058 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.058 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.861585+0000", 2024-05-15T13:21:22.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.031805012000000001 2024-05-15T13:21:22.058 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.058 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:22.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.861647+0000", 2024-05-15T13:21:22.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2786999999999994e-05 2024-05-15T13:21:22.058 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.058 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:22.059 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.059 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.059 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.059 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:125727 7.1 7:83e9791e:::benchmark_data_smithi177_134484_object16662:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-05-15T13:21:22.059 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:14:50.493635+0000", 2024-05-15T13:21:22.059 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 391.52203846700002, 2024-05-15T13:21:22.059 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4217584109999999, 2024-05-15T13:21:22.059 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:22.059 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:22.059 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:22.059 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:22.059 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:22.059 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125727 2024-05-15T13:21:22.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.060 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:22.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:22.060 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.493635+0000", 2024-05-15T13:21:22.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:22.060 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.493635+0000", 2024-05-15T13:21:22.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:22.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.493637+0000", 2024-05-15T13:21:22.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7060000000000001e-06 2024-05-15T13:21:22.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:22.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.493642+0000", 2024-05-15T13:21:22.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7709999999999997e-06 2024-05-15T13:21:22.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:22.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.493653+0000", 2024-05-15T13:21:22.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1402e-05 2024-05-15T13:21:22.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.062 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:22.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.493659+0000", 2024-05-15T13:21:22.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.677e-06 2024-05-15T13:21:22.062 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.062 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.493714+0000", 2024-05-15T13:21:22.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4780000000000001e-05 2024-05-15T13:21:22.062 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.062 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:22.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.493721+0000", 2024-05-15T13:21:22.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.221e-06 2024-05-15T13:21:22.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.063 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.063 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:51.454346+0000", 2024-05-15T13:21:22.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96062528700000005 2024-05-15T13:21:22.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.063 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:22.063 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:51.454356+0000", 2024-05-15T13:21:22.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7380000000000006e-06 2024-05-15T13:21:22.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.063 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.063 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.833331+0000", 2024-05-15T13:21:22.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.378974605 2024-05-15T13:21:22.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for degraded object", 2024-05-15T13:21:22.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.834150+0000", 2024-05-15T13:21:22.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00081903800000000001 2024-05-15T13:21:22.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.865160+0000", 2024-05-15T13:21:22.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.031010469999999998 2024-05-15T13:21:22.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:22.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.865272+0000", 2024-05-15T13:21:22.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111772 2024-05-15T13:21:22.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,6", 2024-05-15T13:21:22.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.865426+0000", 2024-05-15T13:21:22.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000154528 2024-05-15T13:21:22.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:22.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.886495+0000", 2024-05-15T13:21:22.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021068606 2024-05-15T13:21:22.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.066 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.911827+0000", 2024-05-15T13:21:22.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.025332210000000001 2024-05-15T13:21:22.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.066 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.911889+0000", 2024-05-15T13:21:22.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1371000000000006e-05 2024-05-15T13:21:22.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:22.066 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.911956+0000", 2024-05-15T13:21:22.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6963000000000002e-05 2024-05-15T13:21:22.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.067 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:22.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.915394+0000", 2024-05-15T13:21:22.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0034382660000000002 2024-05-15T13:21:22.067 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.067 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:22.067 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.067 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:125732 7.1 7:8dddf723:::benchmark_data_smithi177_134484_object16667:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-05-15T13:21:22.067 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:14:50.500256+0000", 2024-05-15T13:21:22.067 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 391.51541798599999, 2024-05-15T13:21:22.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.418168004, 2024-05-15T13:21:22.067 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:22.068 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:22.068 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:22.068 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:22.068 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:22.068 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125732 2024-05-15T13:21:22.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.068 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:22.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:22.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.500256+0000", 2024-05-15T13:21:22.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:22.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.500256+0000", 2024-05-15T13:21:22.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:22.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.500257+0000", 2024-05-15T13:21:22.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0750000000000001e-06 2024-05-15T13:21:22.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:22.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.500262+0000", 2024-05-15T13:21:22.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7199999999999997e-06 2024-05-15T13:21:22.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:22.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.500273+0000", 2024-05-15T13:21:22.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1634000000000001e-05 2024-05-15T13:21:22.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:22.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.500279+0000", 2024-05-15T13:21:22.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9859999999999999e-06 2024-05-15T13:21:22.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.500327+0000", 2024-05-15T13:21:22.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7465e-05 2024-05-15T13:21:22.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:22.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.500334+0000", 2024-05-15T13:21:22.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9009999999999997e-06 2024-05-15T13:21:22.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:51.454397+0000", 2024-05-15T13:21:22.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.95406309099999997 2024-05-15T13:21:22.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:22.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:51.454407+0000", 2024-05-15T13:21:22.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0142999999999999e-05 2024-05-15T13:21:22.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.072 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.072 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.834192+0000", 2024-05-15T13:21:22.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3797852580000001 2024-05-15T13:21:22.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.072 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for degraded object", 2024-05-15T13:21:22.072 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.834901+0000", 2024-05-15T13:21:22.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00070850299999999998 2024-05-15T13:21:22.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.072 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.875781+0000", 2024-05-15T13:21:22.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040880809999999997 2024-05-15T13:21:22.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:22.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.875930+0000", 2024-05-15T13:21:22.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000148782 2024-05-15T13:21:22.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,6", 2024-05-15T13:21:22.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.876091+0000", 2024-05-15T13:21:22.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016048599999999999 2024-05-15T13:21:22.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:22.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.893630+0000", 2024-05-15T13:21:22.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017538818000000001 2024-05-15T13:21:22.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.915455+0000", 2024-05-15T13:21:22.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021825496999999999 2024-05-15T13:21:22.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.915524+0000", 2024-05-15T13:21:22.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8478000000000004e-05 2024-05-15T13:21:22.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:22.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.915576+0000", 2024-05-15T13:21:22.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2463000000000002e-05 2024-05-15T13:21:22.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:22.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.918424+0000", 2024-05-15T13:21:22.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0028478940000000001 2024-05-15T13:21:22.075 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.075 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:22.075 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.076 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:125737 7.1 7:8d86fd0e:::benchmark_data_smithi177_134484_object16672:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-05-15T13:21:22.076 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:14:50.509603+0000", 2024-05-15T13:21:22.076 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 391.50607100799999, 2024-05-15T13:21:22.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4107393940000001, 2024-05-15T13:21:22.076 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:22.076 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:22.076 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:22.076 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:22.076 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:22.076 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125737 2024-05-15T13:21:22.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.076 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:22.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:22.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.509603+0000", 2024-05-15T13:21:22.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:22.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.509603+0000", 2024-05-15T13:21:22.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:22.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.509604+0000", 2024-05-15T13:21:22.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.333e-06 2024-05-15T13:21:22.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:22.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.509610+0000", 2024-05-15T13:21:22.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5400000000000003e-06 2024-05-15T13:21:22.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:22.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.509625+0000", 2024-05-15T13:21:22.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5121e-05 2024-05-15T13:21:22.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:22.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.509633+0000", 2024-05-15T13:21:22.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.524e-06 2024-05-15T13:21:22.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.509698+0000", 2024-05-15T13:21:22.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5093999999999997e-05 2024-05-15T13:21:22.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:22.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.509710+0000", 2024-05-15T13:21:22.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1411e-05 2024-05-15T13:21:22.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:51.454430+0000", 2024-05-15T13:21:22.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.94472033899999996 2024-05-15T13:21:22.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:22.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:51.454435+0000", 2024-05-15T13:21:22.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4909999999999999e-06 2024-05-15T13:21:22.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.834957+0000", 2024-05-15T13:21:22.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.380522389 2024-05-15T13:21:22.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for degraded object", 2024-05-15T13:21:22.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.835631+0000", 2024-05-15T13:21:22.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00067434000000000005 2024-05-15T13:21:22.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.886549+0000", 2024-05-15T13:21:22.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.050917385000000003 2024-05-15T13:21:22.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:22.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.886681+0000", 2024-05-15T13:21:22.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013227900000000001 2024-05-15T13:21:22.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,6", 2024-05-15T13:21:22.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.886850+0000", 2024-05-15T13:21:22.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016862600000000001 2024-05-15T13:21:22.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:22.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.900016+0000", 2024-05-15T13:21:22.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013166493 2024-05-15T13:21:22.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.918492+0000", 2024-05-15T13:21:22.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.018475933999999999 2024-05-15T13:21:22.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.918540+0000", 2024-05-15T13:21:22.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7367999999999998e-05 2024-05-15T13:21:22.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:22.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.918588+0000", 2024-05-15T13:21:22.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8983999999999997e-05 2024-05-15T13:21:22.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:22.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.920342+0000", 2024-05-15T13:21:22.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017537430000000001 2024-05-15T13:21:22.083 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.084 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:22.084 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.084 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:125755 7.1 7:8162cd9a:::benchmark_data_smithi177_134484_object16690:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-05-15T13:21:22.084 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:14:50.561314+0000", 2024-05-15T13:21:22.084 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 391.45435997099997, 2024-05-15T13:21:22.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.361023903, 2024-05-15T13:21:22.084 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:22.084 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:22.084 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:22.084 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:22.084 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:22.084 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125755 2024-05-15T13:21:22.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.085 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:22.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:22.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.561314+0000", 2024-05-15T13:21:22.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:22.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.561314+0000", 2024-05-15T13:21:22.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:22.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.561316+0000", 2024-05-15T13:21:22.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8619999999999999e-06 2024-05-15T13:21:22.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:22.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.561322+0000", 2024-05-15T13:21:22.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3400000000000003e-06 2024-05-15T13:21:22.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:22.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.561338+0000", 2024-05-15T13:21:22.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6328e-05 2024-05-15T13:21:22.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:22.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.561346+0000", 2024-05-15T13:21:22.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7090000000000002e-06 2024-05-15T13:21:22.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.561399+0000", 2024-05-15T13:21:22.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3072000000000003e-05 2024-05-15T13:21:22.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:22.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.561408+0000", 2024-05-15T13:21:22.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7080000000000003e-06 2024-05-15T13:21:22.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:51.454467+0000", 2024-05-15T13:21:22.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.89305920999999999 2024-05-15T13:21:22.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:22.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:51.454476+0000", 2024-05-15T13:21:22.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4950000000000008e-06 2024-05-15T13:21:22.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.835676+0000", 2024-05-15T13:21:22.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.381199982 2024-05-15T13:21:22.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for degraded object", 2024-05-15T13:21:22.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.836339+0000", 2024-05-15T13:21:22.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00066347900000000002 2024-05-15T13:21:22.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.896873+0000", 2024-05-15T13:21:22.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.060534250999999997 2024-05-15T13:21:22.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:22.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.896949+0000", 2024-05-15T13:21:22.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5315999999999995e-05 2024-05-15T13:21:22.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,6", 2024-05-15T13:21:22.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.897067+0000", 2024-05-15T13:21:22.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011851200000000001 2024-05-15T13:21:22.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:22.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.911775+0000", 2024-05-15T13:21:22.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014707896 2024-05-15T13:21:22.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.920393+0000", 2024-05-15T13:21:22.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0086179640000000005 2024-05-15T13:21:22.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.920433+0000", 2024-05-15T13:21:22.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0185000000000002e-05 2024-05-15T13:21:22.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:22.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.920471+0000", 2024-05-15T13:21:22.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7583000000000003e-05 2024-05-15T13:21:22.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:22.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.922338+0000", 2024-05-15T13:21:22.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001867011 2024-05-15T13:21:22.092 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.092 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:22.092 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.092 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:125790 7.1 7:8b578205:::benchmark_data_smithi177_134484_object16725:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-05-15T13:21:22.092 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:14:50.684303+0000", 2024-05-15T13:21:22.092 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 391.331371322, 2024-05-15T13:21:22.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2537769320000001, 2024-05-15T13:21:22.093 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:22.093 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:22.093 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:22.093 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:22.093 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:22.093 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 125790 2024-05-15T13:21:22.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.093 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:22.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:22.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.684303+0000", 2024-05-15T13:21:22.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:22.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.684303+0000", 2024-05-15T13:21:22.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:22.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.684304+0000", 2024-05-15T13:21:22.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2750000000000001e-06 2024-05-15T13:21:22.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:22.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.684308+0000", 2024-05-15T13:21:22.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5700000000000003e-06 2024-05-15T13:21:22.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:22.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.684319+0000", 2024-05-15T13:21:22.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0956e-05 2024-05-15T13:21:22.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:22.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.684325+0000", 2024-05-15T13:21:22.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4419999999999997e-06 2024-05-15T13:21:22.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.684375+0000", 2024-05-15T13:21:22.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0000999999999998e-05 2024-05-15T13:21:22.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:22.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:50.684383+0000", 2024-05-15T13:21:22.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7139999999999999e-06 2024-05-15T13:21:22.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:51.454492+0000", 2024-05-15T13:21:22.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.77010831700000004 2024-05-15T13:21:22.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:22.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:51.454496+0000", 2024-05-15T13:21:22.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2819999999999998e-06 2024-05-15T13:21:22.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.836376+0000", 2024-05-15T13:21:22.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3818802809999999 2024-05-15T13:21:22.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for degraded object", 2024-05-15T13:21:22.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.836870+0000", 2024-05-15T13:21:22.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00049342699999999995 2024-05-15T13:21:22.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.907826+0000", 2024-05-15T13:21:22.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.070956672999999998 2024-05-15T13:21:22.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:22.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.907918+0000", 2024-05-15T13:21:22.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1088000000000006e-05 2024-05-15T13:21:22.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,6", 2024-05-15T13:21:22.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.908020+0000", 2024-05-15T13:21:22.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102112 2024-05-15T13:21:22.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:22.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.915474+0000", 2024-05-15T13:21:22.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0074543760000000004 2024-05-15T13:21:22.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.922397+0000", 2024-05-15T13:21:22.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.006922648 2024-05-15T13:21:22.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.934438+0000", 2024-05-15T13:21:22.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012040945000000001 2024-05-15T13:21:22.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:22.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.934511+0000", 2024-05-15T13:21:22.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3694000000000003e-05 2024-05-15T13:21:22.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:22.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:14:52.938079+0000", 2024-05-15T13:21:22.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003568131 2024-05-15T13:21:22.100 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.100 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:22.100 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.101 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26299.0:25986 8.2 8:4b5b0f50:::benchmark_data_smithi177_149673_object25985:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e766)", 2024-05-15T13:21:22.101 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:19:49.474991+0000", 2024-05-15T13:21:22.101 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.540682383999993, 2024-05-15T13:21:22.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.209466822, 2024-05-15T13:21:22.101 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:22.101 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:22.101 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:22.101 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26299", 2024-05-15T13:21:22.101 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:21:22.101 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 25986 2024-05-15T13:21:22.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.101 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:22.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:22.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.474991+0000", 2024-05-15T13:21:22.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:22.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.474991+0000", 2024-05-15T13:21:22.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:22.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.474993+0000", 2024-05-15T13:21:22.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0899999999999999e-06 2024-05-15T13:21:22.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:22.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.475011+0000", 2024-05-15T13:21:22.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8927999999999999e-05 2024-05-15T13:21:22.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:22.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.475024+0000", 2024-05-15T13:21:22.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2761e-05 2024-05-15T13:21:22.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:22.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.475032+0000", 2024-05-15T13:21:22.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6580000000000002e-06 2024-05-15T13:21:22.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.475076+0000", 2024-05-15T13:21:22.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3778e-05 2024-05-15T13:21:22.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:22.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.475190+0000", 2024-05-15T13:21:22.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114184 2024-05-15T13:21:22.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:22.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.475330+0000", 2024-05-15T13:21:22.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014042199999999999 2024-05-15T13:21:22.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:22.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.478029+0000", 2024-05-15T13:21:22.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0026990069999999998 2024-05-15T13:21:22.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.478666+0000", 2024-05-15T13:21:22.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00063672900000000005 2024-05-15T13:21:22.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.684347+0000", 2024-05-15T13:21:22.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20568076599999999 2024-05-15T13:21:22.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:22.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.684410+0000", 2024-05-15T13:21:22.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3639000000000005e-05 2024-05-15T13:21:22.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:22.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.684458+0000", 2024-05-15T13:21:22.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7859999999999999e-05 2024-05-15T13:21:22.106 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.107 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:22.107 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.107 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26299.0:25993 8.2 8:4524888d:::benchmark_data_smithi177_149673_object25992:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e766)", 2024-05-15T13:21:22.107 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:19:49.476689+0000", 2024-05-15T13:21:22.107 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.538984381999995, 2024-05-15T13:21:22.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20791063200000001, 2024-05-15T13:21:22.107 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:22.107 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:22.107 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:22.107 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26299", 2024-05-15T13:21:22.107 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:21:22.107 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 25993 2024-05-15T13:21:22.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.108 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:22.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:22.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.476689+0000", 2024-05-15T13:21:22.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:22.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.476689+0000", 2024-05-15T13:21:22.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:22.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.476691+0000", 2024-05-15T13:21:22.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4699999999999999e-06 2024-05-15T13:21:22.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:22.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.476720+0000", 2024-05-15T13:21:22.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9359000000000001e-05 2024-05-15T13:21:22.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:22.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.476733+0000", 2024-05-15T13:21:22.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.291e-05 2024-05-15T13:21:22.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:22.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.476742+0000", 2024-05-15T13:21:22.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3790000000000006e-06 2024-05-15T13:21:22.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.476800+0000", 2024-05-15T13:21:22.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8459999999999999e-05 2024-05-15T13:21:22.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:22.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.476931+0000", 2024-05-15T13:21:22.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000130841 2024-05-15T13:21:22.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:22.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.477064+0000", 2024-05-15T13:21:22.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013344499999999999 2024-05-15T13:21:22.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:22.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.478265+0000", 2024-05-15T13:21:22.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001200172 2024-05-15T13:21:22.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.480066+0000", 2024-05-15T13:21:22.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0018016040000000001 2024-05-15T13:21:22.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.684504+0000", 2024-05-15T13:21:22.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20443746900000001 2024-05-15T13:21:22.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:22.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.684570+0000", 2024-05-15T13:21:22.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6866999999999995e-05 2024-05-15T13:21:22.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:22.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.684600+0000", 2024-05-15T13:21:22.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9655999999999999e-05 2024-05-15T13:21:22.113 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.113 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:22.113 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.113 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26299.0:26007 8.2 8:48fdd64c:::benchmark_data_smithi177_149673_object26006:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e766)", 2024-05-15T13:21:22.113 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:19:49.483394+0000", 2024-05-15T13:21:22.113 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.532280115000006, 2024-05-15T13:21:22.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.204105601, 2024-05-15T13:21:22.113 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:22.113 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:22.113 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:22.113 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26299", 2024-05-15T13:21:22.113 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:21:22.114 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 26007 2024-05-15T13:21:22.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.114 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:22.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:22.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.483394+0000", 2024-05-15T13:21:22.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:22.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.483394+0000", 2024-05-15T13:21:22.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:22.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.483395+0000", 2024-05-15T13:21:22.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.559e-06 2024-05-15T13:21:22.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:22.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.483420+0000", 2024-05-15T13:21:22.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4294000000000002e-05 2024-05-15T13:21:22.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:22.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.483432+0000", 2024-05-15T13:21:22.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2763e-05 2024-05-15T13:21:22.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:22.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.483441+0000", 2024-05-15T13:21:22.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5010000000000004e-06 2024-05-15T13:21:22.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.483501+0000", 2024-05-15T13:21:22.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0126e-05 2024-05-15T13:21:22.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:22.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.483691+0000", 2024-05-15T13:21:22.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00019036300000000001 2024-05-15T13:21:22.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:22.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.483850+0000", 2024-05-15T13:21:22.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015913799999999999 2024-05-15T13:21:22.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:22.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.486875+0000", 2024-05-15T13:21:22.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0030243179999999998 2024-05-15T13:21:22.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.486927+0000", 2024-05-15T13:21:22.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2259000000000002e-05 2024-05-15T13:21:22.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.687409+0000", 2024-05-15T13:21:22.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.200481938 2024-05-15T13:21:22.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:22.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.687455+0000", 2024-05-15T13:21:22.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5735000000000002e-05 2024-05-15T13:21:22.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:22.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.687499+0000", 2024-05-15T13:21:22.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4607000000000002e-05 2024-05-15T13:21:22.119 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.119 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:22.119 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.119 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26299.0:26009 8.2 8:4614c74a:::benchmark_data_smithi177_149673_object26008:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e766)", 2024-05-15T13:21:22.119 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:19:49.483559+0000", 2024-05-15T13:21:22.119 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.532114578999995, 2024-05-15T13:21:22.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20412291900000001, 2024-05-15T13:21:22.119 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:22.120 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:22.120 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:22.120 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26299", 2024-05-15T13:21:22.120 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:21:22.120 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 26009 2024-05-15T13:21:22.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.120 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:22.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:22.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.483559+0000", 2024-05-15T13:21:22.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:22.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.483559+0000", 2024-05-15T13:21:22.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:22.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.483560+0000", 2024-05-15T13:21:22.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8599999999999997e-07 2024-05-15T13:21:22.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:22.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.483590+0000", 2024-05-15T13:21:22.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0318e-05 2024-05-15T13:21:22.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:22.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.483603+0000", 2024-05-15T13:21:22.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2655e-05 2024-05-15T13:21:22.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:22.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.483610+0000", 2024-05-15T13:21:22.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3339999999999996e-06 2024-05-15T13:21:22.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.485563+0000", 2024-05-15T13:21:22.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0019524410000000001 2024-05-15T13:21:22.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:22.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.485710+0000", 2024-05-15T13:21:22.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000147207 2024-05-15T13:21:22.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:22.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.485841+0000", 2024-05-15T13:21:22.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013113600000000001 2024-05-15T13:21:22.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.487280+0000", 2024-05-15T13:21:22.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001438644 2024-05-15T13:21:22.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:22.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.488899+0000", 2024-05-15T13:21:22.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0016188800000000001 2024-05-15T13:21:22.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.687576+0000", 2024-05-15T13:21:22.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19867738099999999 2024-05-15T13:21:22.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:22.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.687634+0000", 2024-05-15T13:21:22.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8251999999999997e-05 2024-05-15T13:21:22.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:22.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.687682+0000", 2024-05-15T13:21:22.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7885e-05 2024-05-15T13:21:22.125 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.125 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:22.125 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.125 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26299.0:26073 8.2 8:481fd052:::benchmark_data_smithi177_149673_object26072:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e766)", 2024-05-15T13:21:22.125 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:19:49.509094+0000", 2024-05-15T13:21:22.125 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.506579383000002, 2024-05-15T13:21:22.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17961796199999999, 2024-05-15T13:21:22.125 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:22.125 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:22.125 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:22.126 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26299", 2024-05-15T13:21:22.126 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:21:22.126 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 26073 2024-05-15T13:21:22.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.126 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:22.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:22.126 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.509094+0000", 2024-05-15T13:21:22.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:22.126 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.509094+0000", 2024-05-15T13:21:22.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:22.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.509096+0000", 2024-05-15T13:21:22.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6309999999999999e-06 2024-05-15T13:21:22.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:22.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.509114+0000", 2024-05-15T13:21:22.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8241999999999999e-05 2024-05-15T13:21:22.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:22.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.509131+0000", 2024-05-15T13:21:22.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6915e-05 2024-05-15T13:21:22.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:22.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.509139+0000", 2024-05-15T13:21:22.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9300000000000003e-06 2024-05-15T13:21:22.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.509184+0000", 2024-05-15T13:21:22.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5222000000000002e-05 2024-05-15T13:21:22.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:22.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.509279+0000", 2024-05-15T13:21:22.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4949000000000002e-05 2024-05-15T13:21:22.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:22.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.509363+0000", 2024-05-15T13:21:22.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3633000000000004e-05 2024-05-15T13:21:22.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:22.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.510378+0000", 2024-05-15T13:21:22.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010149239999999999 2024-05-15T13:21:22.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.513328+0000", 2024-05-15T13:21:22.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002950121 2024-05-15T13:21:22.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.688593+0000", 2024-05-15T13:21:22.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17526512999999999 2024-05-15T13:21:22.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:22.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.688671+0000", 2024-05-15T13:21:22.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7507000000000002e-05 2024-05-15T13:21:22.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:22.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.688712+0000", 2024-05-15T13:21:22.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1758000000000002e-05 2024-05-15T13:21:22.131 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.131 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:22.131 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.131 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26299.0:26083 8.2 8:4a2f7d57:::benchmark_data_smithi177_149673_object26082:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e766)", 2024-05-15T13:21:22.131 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:19:49.513874+0000", 2024-05-15T13:21:22.131 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.501799641999995, 2024-05-15T13:21:22.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17583469299999999, 2024-05-15T13:21:22.131 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:22.131 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:22.132 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:22.132 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26299", 2024-05-15T13:21:22.132 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:21:22.132 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 26083 2024-05-15T13:21:22.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.132 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:22.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:22.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.513874+0000", 2024-05-15T13:21:22.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:22.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.513874+0000", 2024-05-15T13:21:22.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:22.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.513875+0000", 2024-05-15T13:21:22.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.28e-06 2024-05-15T13:21:22.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:22.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.513894+0000", 2024-05-15T13:21:22.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8266000000000001e-05 2024-05-15T13:21:22.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:22.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.513905+0000", 2024-05-15T13:21:22.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1314999999999999e-05 2024-05-15T13:21:22.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:22.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.513911+0000", 2024-05-15T13:21:22.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5280000000000003e-06 2024-05-15T13:21:22.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.513950+0000", 2024-05-15T13:21:22.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8980999999999998e-05 2024-05-15T13:21:22.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:22.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.514054+0000", 2024-05-15T13:21:22.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104033 2024-05-15T13:21:22.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:22.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.514159+0000", 2024-05-15T13:21:22.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105151 2024-05-15T13:21:22.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.516960+0000", 2024-05-15T13:21:22.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0028012610000000002 2024-05-15T13:21:22.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:22.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.516972+0000", 2024-05-15T13:21:22.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2078e-05 2024-05-15T13:21:22.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.689609+0000", 2024-05-15T13:21:22.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17263690100000001 2024-05-15T13:21:22.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:22.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.689670+0000", 2024-05-15T13:21:22.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1070999999999999e-05 2024-05-15T13:21:22.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:22.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.689709+0000", 2024-05-15T13:21:22.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8828000000000002e-05 2024-05-15T13:21:22.137 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.137 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:22.137 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.137 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26299.0:26094 8.2 8:42622b4b:::benchmark_data_smithi177_149673_object26093:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e766)", 2024-05-15T13:21:22.137 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:19:49.520995+0000", 2024-05-15T13:21:22.137 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.494679323, 2024-05-15T13:21:22.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16885046400000001, 2024-05-15T13:21:22.137 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:22.137 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:22.137 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:22.138 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26299", 2024-05-15T13:21:22.138 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:21:22.138 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 26094 2024-05-15T13:21:22.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.138 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:22.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:22.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.520995+0000", 2024-05-15T13:21:22.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:22.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.520995+0000", 2024-05-15T13:21:22.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:22.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.520996+0000", 2024-05-15T13:21:22.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4899999999999999e-06 2024-05-15T13:21:22.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:22.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.521021+0000", 2024-05-15T13:21:22.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5462000000000001e-05 2024-05-15T13:21:22.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:22.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.521039+0000", 2024-05-15T13:21:22.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7781000000000001e-05 2024-05-15T13:21:22.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:22.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.521049+0000", 2024-05-15T13:21:22.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3479999999999993e-06 2024-05-15T13:21:22.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.521324+0000", 2024-05-15T13:21:22.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00027493199999999998 2024-05-15T13:21:22.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:22.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.521475+0000", 2024-05-15T13:21:22.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000151545 2024-05-15T13:21:22.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:22.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.521610+0000", 2024-05-15T13:21:22.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000135368 2024-05-15T13:21:22.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:22.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.523024+0000", 2024-05-15T13:21:22.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014135930000000001 2024-05-15T13:21:22.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.524201+0000", 2024-05-15T13:21:22.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011765499999999999 2024-05-15T13:21:22.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.689750+0000", 2024-05-15T13:21:22.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.165549268 2024-05-15T13:21:22.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:22.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.689806+0000", 2024-05-15T13:21:22.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6359e-05 2024-05-15T13:21:22.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:22.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.689845+0000", 2024-05-15T13:21:22.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8767999999999999e-05 2024-05-15T13:21:22.143 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.143 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:22.143 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.143 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26299.0:26108 8.2 8:409a8ef9:::benchmark_data_smithi177_149673_object26107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e766)", 2024-05-15T13:21:22.143 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:19:49.525613+0000", 2024-05-15T13:21:22.143 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.490060514000007, 2024-05-15T13:21:22.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.164355427, 2024-05-15T13:21:22.143 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:22.143 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:22.144 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:22.144 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26299", 2024-05-15T13:21:22.144 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:21:22.144 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 26108 2024-05-15T13:21:22.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.144 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:22.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:22.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.525613+0000", 2024-05-15T13:21:22.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:22.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.525613+0000", 2024-05-15T13:21:22.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:22.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.525616+0000", 2024-05-15T13:21:22.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2010000000000002e-06 2024-05-15T13:21:22.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:22.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.525640+0000", 2024-05-15T13:21:22.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4358e-05 2024-05-15T13:21:22.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:22.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.525654+0000", 2024-05-15T13:21:22.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3935e-05 2024-05-15T13:21:22.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:22.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.525661+0000", 2024-05-15T13:21:22.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7499999999999997e-06 2024-05-15T13:21:22.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.525715+0000", 2024-05-15T13:21:22.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4799999999999997e-05 2024-05-15T13:21:22.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:22.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.525878+0000", 2024-05-15T13:21:22.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016271600000000001 2024-05-15T13:21:22.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:22.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.526022+0000", 2024-05-15T13:21:22.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014405399999999999 2024-05-15T13:21:22.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:22.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.528405+0000", 2024-05-15T13:21:22.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002383263 2024-05-15T13:21:22.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.528693+0000", 2024-05-15T13:21:22.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028776600000000001 2024-05-15T13:21:22.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.689883+0000", 2024-05-15T13:21:22.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16118950600000001 2024-05-15T13:21:22.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:22.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.689933+0000", 2024-05-15T13:21:22.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9985999999999999e-05 2024-05-15T13:21:22.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:22.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.689969+0000", 2024-05-15T13:21:22.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6091999999999999e-05 2024-05-15T13:21:22.149 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.149 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:22.149 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.149 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26299.0:26110 8.2 8:401ccc50:::benchmark_data_smithi177_149673_object26109:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e766)", 2024-05-15T13:21:22.149 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:19:49.525884+0000", 2024-05-15T13:21:22.149 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 92.489789396000006, 2024-05-15T13:21:22.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16472099000000001, 2024-05-15T13:21:22.150 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:22.150 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:22.150 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:22.150 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26299", 2024-05-15T13:21:22.150 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:21:22.150 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 26110 2024-05-15T13:21:22.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.150 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:22.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:22.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.525884+0000", 2024-05-15T13:21:22.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:22.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.525884+0000", 2024-05-15T13:21:22.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:22.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:22.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.525885+0000", 2024-05-15T13:21:22.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4199999999999995e-07 2024-05-15T13:21:22.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:22.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.525928+0000", 2024-05-15T13:21:22.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3290000000000001e-05 2024-05-15T13:21:22.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:22.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.525938+0000", 2024-05-15T13:21:22.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3209999999999994e-06 2024-05-15T13:21:22.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:22.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.525944+0000", 2024-05-15T13:21:22.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7610000000000004e-06 2024-05-15T13:21:22.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:22.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.527252+0000", 2024-05-15T13:21:22.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013083350000000001 2024-05-15T13:21:22.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:22.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.527394+0000", 2024-05-15T13:21:22.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014174700000000001 2024-05-15T13:21:22.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:22.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.527551+0000", 2024-05-15T13:21:22.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015698400000000001 2024-05-15T13:21:22.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:22.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.528615+0000", 2024-05-15T13:21:22.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001063905 2024-05-15T13:21:22.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.528873+0000", 2024-05-15T13:21:22.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025878199999999999 2024-05-15T13:21:22.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:22.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.690503+0000", 2024-05-15T13:21:22.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.16162981900000001 2024-05-15T13:21:22.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:22.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.690552+0000", 2024-05-15T13:21:22.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8384000000000003e-05 2024-05-15T13:21:22.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:22.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:22.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:22.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:49.690605+0000", 2024-05-15T13:21:22.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3919999999999999e-05 2024-05-15T13:21:22.155 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.155 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:22.155 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.155 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:22.155 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:22.155 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:21:22.156 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:21 smithi120 ceph-mon[43961]: 1.9 scrub starts 2024-05-15T13:21:22.156 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:21 smithi120 ceph-mon[43961]: pgmap v2034: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.5 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 651 op/s; 6111/171621 objects misplaced (3.561%); 2.8 MiB/s, 44 objects/s recovering 2024-05-15T13:21:22.156 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:21 smithi120 ceph-mon[43961]: 1.9 scrub ok 2024-05-15T13:21:22.156 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:21 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4114243336' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:21:22.156 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:21 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:21:22.156 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:21 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:21:22.156 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:21 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:21:22.157 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:21 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:21:22.157 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:21 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:21:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:21:22.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:22.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:21:22.351 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:21:22.382 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:22.408 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:21:22.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:22.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:21:22.515 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:21:22.570 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:22.782 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-05-15T13:21:23.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:22 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:21:23.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:22 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:21:23.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:22 smithi027 ceph-mon[39795]: osdmap e785: 12 total, 12 up, 11 in 2024-05-15T13:21:23.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:22 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:21:23.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:22 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:21:23.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:22 smithi120 ceph-mon[43961]: osdmap e785: 12 total, 12 up, 11 in 2024-05-15T13:21:23.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:22 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:21:23.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:22 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:21:23.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:22 smithi177 ceph-mon[44303]: osdmap e785: 12 total, 12 up, 11 in 2024-05-15T13:21:23.216 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:21:23.638 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:21:23.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:23.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:21:23.743 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:21:23.788 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:23.798 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:23 smithi177 ceph-mon[44303]: pgmap v2036: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 505 op/s; 6111/170349 objects misplaced (3.587%); 1.8 MiB/s, 28 objects/s recovering 2024-05-15T13:21:23.932 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:21:23.938 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:23 smithi027 ceph-mon[39795]: pgmap v2036: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 505 op/s; 6111/170349 objects misplaced (3.587%); 1.8 MiB/s, 28 objects/s recovering 2024-05-15T13:21:23.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:23.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:21:24.039 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:21:24.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:23 smithi120 ceph-mon[43961]: pgmap v2036: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 505 op/s; 6111/170349 objects misplaced (3.587%); 1.8 MiB/s, 28 objects/s recovering 2024-05-15T13:21:24.108 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:24.509 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:21:24.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:24.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:21:24.620 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:21:24.664 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:25.044 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:21:25.251 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:25 smithi177 ceph-mon[44303]: pgmap v2037: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 754 op/s; 5403/162423 objects misplaced (3.326%); 3.6 MiB/s, 58 objects/s recovering 2024-05-15T13:21:25.439 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:21:25.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:25.445 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:21:25.529 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:25 smithi027 ceph-mon[39795]: pgmap v2037: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 754 op/s; 5403/162423 objects misplaced (3.326%); 3.6 MiB/s, 58 objects/s recovering 2024-05-15T13:21:25.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:25 smithi120 ceph-mon[43961]: pgmap v2037: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 754 op/s; 5403/162423 objects misplaced (3.326%); 3.6 MiB/s, 58 objects/s recovering 2024-05-15T13:21:25.545 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:21:25.573 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:25.596 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:21:25.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:25.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:21:25.703 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:21:25.733 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:25.753 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:21:25.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:25.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:21:25.857 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:21:25.888 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:25.911 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:21:25.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:25.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:21:25.933 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-05-15T13:21:26.015 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:21:26.074 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:21:26.074 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T13:21:26.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.074 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26299.0:128266 8.3 8:c2ba81c2:::benchmark_data_smithi177_149673_object38204:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e785)", 2024-05-15T13:21:26.074 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:21:26.004757+0000", 2024-05-15T13:21:26.074 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.069963116000000006, 2024-05-15T13:21:26.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.069992993000000003, 2024-05-15T13:21:26.074 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:26.074 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:21:26.074 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:26.074 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26299", 2024-05-15T13:21:26.074 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:21:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 128266 2024-05-15T13:21:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.004757+0000", 2024-05-15T13:21:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.004757+0000", 2024-05-15T13:21:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.004759+0000", 2024-05-15T13:21:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2979999999999999e-06 2024-05-15T13:21:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.004764+0000", 2024-05-15T13:21:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1200000000000001e-06 2024-05-15T13:21:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.004778+0000", 2024-05-15T13:21:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.38e-05 2024-05-15T13:21:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.004785+0000", 2024-05-15T13:21:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6850000000000001e-06 2024-05-15T13:21:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.016055+0000", 2024-05-15T13:21:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.01126952 2024-05-15T13:21:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.016187+0000", 2024-05-15T13:21:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013202400000000001 2024-05-15T13:21:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,5", 2024-05-15T13:21:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.016337+0000", 2024-05-15T13:21:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000150323 2024-05-15T13:21:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.021104+0000", 2024-05-15T13:21:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0047664350000000003 2024-05-15T13:21:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.022203+0000", 2024-05-15T13:21:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010989579999999999 2024-05-15T13:21:26.078 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26299.0:128309 8.f 8:f9a41b7c:::benchmark_data_smithi177_149673_object38247:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e785)", 2024-05-15T13:21:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:21:26.065180+0000", 2024-05-15T13:21:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.009540715, 2024-05-15T13:21:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0096291769999999992, 2024-05-15T13:21:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:21:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26299", 2024-05-15T13:21:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:21:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 128309 2024-05-15T13:21:26.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.065180+0000", 2024-05-15T13:21:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.065180+0000", 2024-05-15T13:21:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.065182+0000", 2024-05-15T13:21:26.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.75e-06 2024-05-15T13:21:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.065189+0000", 2024-05-15T13:21:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6739999999999997e-06 2024-05-15T13:21:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.065208+0000", 2024-05-15T13:21:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9018999999999999e-05 2024-05-15T13:21:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.065219+0000", 2024-05-15T13:21:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0377000000000001e-05 2024-05-15T13:21:26.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:26.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.065295+0000", 2024-05-15T13:21:26.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6111000000000004e-05 2024-05-15T13:21:26.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:26.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.065431+0000", 2024-05-15T13:21:26.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013627299999999999 2024-05-15T13:21:26.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,2", 2024-05-15T13:21:26.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.065553+0000", 2024-05-15T13:21:26.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000122037 2024-05-15T13:21:26.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:26.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.069203+0000", 2024-05-15T13:21:26.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0036500320000000001 2024-05-15T13:21:26.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:26.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.074418+0000", 2024-05-15T13:21:26.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0052148439999999997 2024-05-15T13:21:26.083 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:26.083 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:26.083 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.26299.0:128239 8.3 e785/739)", 2024-05-15T13:21:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:21:26.069745+0000", 2024-05-15T13:21:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0049759419999999997, 2024-05-15T13:21:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.005111544, 2024-05-15T13:21:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "queued for pg", 2024-05-15T13:21:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.069745+0000", 2024-05-15T13:21:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.069745+0000", 2024-05-15T13:21:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.069745+0000", 2024-05-15T13:21:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9500000000000002e-07 2024-05-15T13:21:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.069749+0000", 2024-05-15T13:21:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.76e-06 2024-05-15T13:21:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.069765+0000", 2024-05-15T13:21:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.554e-05 2024-05-15T13:21:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.069772+0000", 2024-05-15T13:21:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7889999999999995e-06 2024-05-15T13:21:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.26299.0:128249 8.3 e785/739)", 2024-05-15T13:21:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:21:26.069786+0000", 2024-05-15T13:21:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0049351079999999997, 2024-05-15T13:21:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.005097028, 2024-05-15T13:21:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "queued for pg", 2024-05-15T13:21:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.069786+0000", 2024-05-15T13:21:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:26.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.069786+0000", 2024-05-15T13:21:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.069786+0000", 2024-05-15T13:21:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1899999999999999e-07 2024-05-15T13:21:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.069788+0000", 2024-05-15T13:21:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.407e-06 2024-05-15T13:21:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:26.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.069797+0000", 2024-05-15T13:21:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6260000000000001e-06 2024-05-15T13:21:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.069801+0000", 2024-05-15T13:21:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3359999999999997e-06 2024-05-15T13:21:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(osd.11.0:107576 8.3 e785/739)", 2024-05-15T13:21:26.089 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:21:26.069816+0000", 2024-05-15T13:21:26.090 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0049044320000000002, 2024-05-15T13:21:26.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0050934730000000003, 2024-05-15T13:21:26.090 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "queued for pg", 2024-05-15T13:21:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.069816+0000", 2024-05-15T13:21:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.069816+0000", 2024-05-15T13:21:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:26.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.069816+0000", 2024-05-15T13:21:26.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5199999999999998e-07 2024-05-15T13:21:26.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:26.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.069819+0000", 2024-05-15T13:21:26.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3029999999999998e-06 2024-05-15T13:21:26.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:26.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.069826+0000", 2024-05-15T13:21:26.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2239999999999998e-06 2024-05-15T13:21:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.069830+0000", 2024-05-15T13:21:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2239999999999997e-06 2024-05-15T13:21:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.26299.0:128266 8.3 e785/739)", 2024-05-15T13:21:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:21:26.072584+0000", 2024-05-15T13:21:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0021368820000000001, 2024-05-15T13:21:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002351255, 2024-05-15T13:21:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:26.093 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "queued for pg", 2024-05-15T13:21:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.072584+0000", 2024-05-15T13:21:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.072584+0000", 2024-05-15T13:21:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.072584+0000", 2024-05-15T13:21:26.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6400000000000002e-07 2024-05-15T13:21:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.072589+0000", 2024-05-15T13:21:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3699999999999997e-06 2024-05-15T13:21:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.072603+0000", 2024-05-15T13:21:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4117000000000001e-05 2024-05-15T13:21:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.072610+0000", 2024-05-15T13:21:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.233e-06 2024-05-15T13:21:26.095 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.26299.0:128311 8.5 e785/739 8:a662f960:::benchmark_data_smithi177_149673_object38249:head v 785'8032, mlcod=785'8030)", 2024-05-15T13:21:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:21:26.072632+0000", 2024-05-15T13:21:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0020885209999999999, 2024-05-15T13:21:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023325379999999999, 2024-05-15T13:21:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T13:21:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.072632+0000", 2024-05-15T13:21:26.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.072632+0000", 2024-05-15T13:21:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.072632+0000", 2024-05-15T13:21:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.03e-07 2024-05-15T13:21:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:26.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.072636+0000", 2024-05-15T13:21:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4869999999999998e-06 2024-05-15T13:21:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.072642+0000", 2024-05-15T13:21:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0340000000000002e-06 2024-05-15T13:21:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.072647+0000", 2024-05-15T13:21:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6959999999999996e-06 2024-05-15T13:21:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.072704+0000", 2024-05-15T13:21:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7451999999999998e-05 2024-05-15T13:21:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.072723+0000", 2024-05-15T13:21:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8644e-05 2024-05-15T13:21:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26299.0:128239 8.3 8:ce5b8b2c:::benchmark_data_smithi177_149673_object38177:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e785)", 2024-05-15T13:21:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:21:25.939422+0000", 2024-05-15T13:21:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.13529812099999999, 2024-05-15T13:21:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.135579114, 2024-05-15T13:21:26.099 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:21:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26299", 2024-05-15T13:21:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:21:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 128239 2024-05-15T13:21:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:25.939422+0000", 2024-05-15T13:21:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:25.939422+0000", 2024-05-15T13:21:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:25.939423+0000", 2024-05-15T13:21:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.89e-07 2024-05-15T13:21:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:25.939425+0000", 2024-05-15T13:21:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4839999999999998e-06 2024-05-15T13:21:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:25.939431+0000", 2024-05-15T13:21:26.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3990000000000003e-06 2024-05-15T13:21:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:25.939435+0000", 2024-05-15T13:21:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6759999999999998e-06 2024-05-15T13:21:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.007921+0000", 2024-05-15T13:21:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.068486061000000001 2024-05-15T13:21:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:26.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.008051+0000", 2024-05-15T13:21:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012908400000000001 2024-05-15T13:21:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,5", 2024-05-15T13:21:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.008172+0000", 2024-05-15T13:21:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000121854 2024-05-15T13:21:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.015980+0000", 2024-05-15T13:21:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0078070800000000001 2024-05-15T13:21:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:26.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.021858+0000", 2024-05-15T13:21:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0058787249999999996 2024-05-15T13:21:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26299.0:128283 8.3 8:c9f4282f:::benchmark_data_smithi177_149673_object38221:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e785)", 2024-05-15T13:21:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:21:26.032227+0000", 2024-05-15T13:21:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.042493476000000002, 2024-05-15T13:21:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042825844000000002, 2024-05-15T13:21:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:21:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:26.104 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26299", 2024-05-15T13:21:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:21:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 128283 2024-05-15T13:21:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.032227+0000", 2024-05-15T13:21:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.032227+0000", 2024-05-15T13:21:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.032229+0000", 2024-05-15T13:21:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5889999999999999e-06 2024-05-15T13:21:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.032235+0000", 2024-05-15T13:21:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4219999999999997e-06 2024-05-15T13:21:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.032249+0000", 2024-05-15T13:21:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3685e-05 2024-05-15T13:21:26.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.032257+0000", 2024-05-15T13:21:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2090000000000005e-06 2024-05-15T13:21:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.037739+0000", 2024-05-15T13:21:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0054817429999999999 2024-05-15T13:21:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.037874+0000", 2024-05-15T13:21:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013501999999999999 2024-05-15T13:21:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,5", 2024-05-15T13:21:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.038008+0000", 2024-05-15T13:21:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013397300000000001 2024-05-15T13:21:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.043248+0000", 2024-05-15T13:21:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0052398540000000004 2024-05-15T13:21:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.048579+0000", 2024-05-15T13:21:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0053310830000000003 2024-05-15T13:21:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:26.108 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26299.0:128290 8.3 8:cbe9b90c:::benchmark_data_smithi177_149673_object38228:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e785)", 2024-05-15T13:21:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:21:26.039740+0000", 2024-05-15T13:21:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.034980116999999998, 2024-05-15T13:21:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.035360763000000003, 2024-05-15T13:21:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:21:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26299", 2024-05-15T13:21:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:21:26.109 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 128290 2024-05-15T13:21:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.039740+0000", 2024-05-15T13:21:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.039740+0000", 2024-05-15T13:21:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.039742+0000", 2024-05-15T13:21:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.398e-06 2024-05-15T13:21:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.039748+0000", 2024-05-15T13:21:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6910000000000002e-06 2024-05-15T13:21:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.039758+0000", 2024-05-15T13:21:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0370999999999999e-05 2024-05-15T13:21:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.039766+0000", 2024-05-15T13:21:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2509999999999994e-06 2024-05-15T13:21:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.043276+0000", 2024-05-15T13:21:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035095500000000002 2024-05-15T13:21:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.043404+0000", 2024-05-15T13:21:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000128281 2024-05-15T13:21:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,5", 2024-05-15T13:21:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.043524+0000", 2024-05-15T13:21:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000119833 2024-05-15T13:21:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.048560+0000", 2024-05-15T13:21:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0050366029999999997 2024-05-15T13:21:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.048619+0000", 2024-05-15T13:21:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8692e-05 2024-05-15T13:21:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:26.113 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26299.0:128249 8.3 8:c443bfe6:::benchmark_data_smithi177_149673_object38187:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e785)", 2024-05-15T13:21:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:21:25.956883+0000", 2024-05-15T13:21:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.117837309, 2024-05-15T13:21:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.118265735, 2024-05-15T13:21:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:21:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26299", 2024-05-15T13:21:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:21:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 128249 2024-05-15T13:21:26.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:25.956883+0000", 2024-05-15T13:21:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:25.956883+0000", 2024-05-15T13:21:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:25.956884+0000", 2024-05-15T13:21:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1799999999999999e-06 2024-05-15T13:21:26.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:25.956889+0000", 2024-05-15T13:21:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8679999999999998e-06 2024-05-15T13:21:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:25.956902+0000", 2024-05-15T13:21:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2340999999999999e-05 2024-05-15T13:21:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:25.956909+0000", 2024-05-15T13:21:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.092e-06 2024-05-15T13:21:26.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.012003+0000", 2024-05-15T13:21:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.055094028000000003 2024-05-15T13:21:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.012124+0000", 2024-05-15T13:21:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012167699999999999 2024-05-15T13:21:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 1,5", 2024-05-15T13:21:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.012229+0000", 2024-05-15T13:21:26.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104399 2024-05-15T13:21:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.016023+0000", 2024-05-15T13:21:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037946059999999998 2024-05-15T13:21:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:21:26.021868+0000", 2024-05-15T13:21:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0058445679999999996 2024-05-15T13:21:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:26.118 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-05-15T13:21:26.119 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 11 2024-05-15T13:21:26.119 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:21:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:26.149 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:21:26.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:26.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:21:26.262 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:21:26.316 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:26.351 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:21:26.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:26.355 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:21:26.456 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:21:26.492 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:26.519 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:21:26.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:26.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:21:26.551 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:21:26.625 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:21:26.659 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:26.681 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:21:26.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:26.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:21:26.786 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:21:26.813 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:26.831 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-05-15T13:21:26.836 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:21:26.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:26.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:21:26.941 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:21:26.970 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:26.994 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:21:26.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:26.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:21:27.098 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:21:27.126 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:27.146 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:21:27.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:27.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:21:27.249 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:21:27.252 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:21:27.289 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:27.572 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:21:27.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:27.579 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:21:27.680 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:21:27.744 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:27.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:27 smithi177 ceph-mon[44303]: pgmap v2038: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 614 op/s; 5403/160464 objects misplaced (3.367%); 1.8 MiB/s, 29 objects/s recovering 2024-05-15T13:21:28.035 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:27 smithi027 ceph-mon[39795]: pgmap v2038: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 614 op/s; 5403/160464 objects misplaced (3.367%); 1.8 MiB/s, 29 objects/s recovering 2024-05-15T13:21:28.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:27 smithi120 ceph-mon[43961]: pgmap v2038: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 614 op/s; 5403/160464 objects misplaced (3.367%); 1.8 MiB/s, 29 objects/s recovering 2024-05-15T13:21:28.159 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:21:28.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:28.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:21:28.271 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:21:28.300 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:28.328 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:21:28.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:28.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:21:28.432 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:21:28.467 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:28.486 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:21:28.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:28.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:21:28.591 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:21:28.649 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:29.394 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:21:29.394 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T13:21:29.394 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-05-15T13:21:29.394 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-05-15T13:21:29.394 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:21:29.514 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:21:29.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:29.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:21:29.576 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:21:29.622 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:21:29.629 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:21:29.670 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:29.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:29 smithi027 ceph-mon[39795]: pgmap v2039: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.1 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 632 op/s; 4678/154710 objects misplaced (3.024%); 3.0 MiB/s, 47 objects/s recovering 2024-05-15T13:21:30.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:29 smithi120 ceph-mon[43961]: pgmap v2039: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.1 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 632 op/s; 4678/154710 objects misplaced (3.024%); 3.0 MiB/s, 47 objects/s recovering 2024-05-15T13:21:30.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:29 smithi177 ceph-mon[44303]: pgmap v2039: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.1 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 632 op/s; 4678/154710 objects misplaced (3.024%); 3.0 MiB/s, 47 objects/s recovering 2024-05-15T13:21:31.249 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:21:31.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:31.252 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:21:31.277 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-05-15T13:21:31.353 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:21:31.428 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:31.469 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:21:31.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:31.478 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:21:31.579 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:21:31.642 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:31.671 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:21:31.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:31.676 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:21:31.739 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:21:31.777 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:21:31.825 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:31.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:31 smithi177 ceph-mon[44303]: pgmap v2040: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 687 op/s; 4678/150993 objects misplaced (3.098%); 3.0 MiB/s, 47 objects/s recovering 2024-05-15T13:21:32.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:31 smithi027 ceph-mon[39795]: pgmap v2040: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 687 op/s; 4678/150993 objects misplaced (3.098%); 3.0 MiB/s, 47 objects/s recovering 2024-05-15T13:21:32.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:31 smithi120 ceph-mon[43961]: pgmap v2040: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 687 op/s; 4678/150993 objects misplaced (3.098%); 3.0 MiB/s, 47 objects/s recovering 2024-05-15T13:21:33.002 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:21:33.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:33.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:21:33.112 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:21:33.187 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:33.221 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:21:33.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:33.226 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:21:33.327 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:21:33.386 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:33.725 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:33 smithi027 ceph-mon[39795]: pgmap v2041: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 650 op/s; 4678/149640 objects misplaced (3.126%); 2.7 MiB/s, 43 objects/s recovering 2024-05-15T13:21:33.816 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:21:33.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:33.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:21:33.824 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:21:33.856 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:21:33.856 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-05-15T13:21:33.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-05-15T13:21:33.856 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T13:21:33.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.856 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:145960 7.5 7:a1515312:::benchmark_data_smithi177_134484_object36895:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:21:33.856 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.569536+0000", 2024-05-15T13:21:33.856 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 354.28252889999999, 2024-05-15T13:21:33.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2435298179999998, 2024-05-15T13:21:33.856 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:33.857 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:33.857 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:33.857 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:33.857 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:33.857 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 145960 2024-05-15T13:21:33.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.857 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:33.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:33.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569536+0000", 2024-05-15T13:21:33.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:33.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569536+0000", 2024-05-15T13:21:33.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:33.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569538+0000", 2024-05-15T13:21:33.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4789999999999999e-06 2024-05-15T13:21:33.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:33.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569545+0000", 2024-05-15T13:21:33.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5299999999999999e-06 2024-05-15T13:21:33.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:33.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569561+0000", 2024-05-15T13:21:33.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5747e-05 2024-05-15T13:21:33.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:33.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569575+0000", 2024-05-15T13:21:33.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3915e-05 2024-05-15T13:21:33.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569651+0000", 2024-05-15T13:21:33.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6126999999999998e-05 2024-05-15T13:21:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569660+0000", 2024-05-15T13:21:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5480000000000007e-06 2024-05-15T13:21:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530145+0000", 2024-05-15T13:21:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96048494399999995 2024-05-15T13:21:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530173+0000", 2024-05-15T13:21:33.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7702999999999999e-05 2024-05-15T13:21:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.800476+0000", 2024-05-15T13:21:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2703030390000001 2024-05-15T13:21:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.800594+0000", 2024-05-15T13:21:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117899 2024-05-15T13:21:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.800698+0000", 2024-05-15T13:21:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104308 2024-05-15T13:21:33.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.810577+0000", 2024-05-15T13:21:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0098783619999999999 2024-05-15T13:21:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.810611+0000", 2024-05-15T13:21:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4394000000000003e-05 2024-05-15T13:21:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.810670+0000", 2024-05-15T13:21:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9113999999999997e-05 2024-05-15T13:21:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.810718+0000", 2024-05-15T13:21:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7692999999999999e-05 2024-05-15T13:21:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.813066+0000", 2024-05-15T13:21:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023480160000000001 2024-05-15T13:21:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.863 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:145962 7.5 7:a16081f3:::benchmark_data_smithi177_134484_object36897:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:21:33.864 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.569605+0000", 2024-05-15T13:21:33.864 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 354.28245960200002, 2024-05-15T13:21:33.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2554589759999999, 2024-05-15T13:21:33.864 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:33.864 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:33.864 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:33.864 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:33.864 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:33.864 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 145962 2024-05-15T13:21:33.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.864 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:33.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:33.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569605+0000", 2024-05-15T13:21:33.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:33.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569605+0000", 2024-05-15T13:21:33.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:33.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569606+0000", 2024-05-15T13:21:33.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8899999999999999e-07 2024-05-15T13:21:33.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:33.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569609+0000", 2024-05-15T13:21:33.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3390000000000001e-06 2024-05-15T13:21:33.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:33.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569617+0000", 2024-05-15T13:21:33.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3019999999999997e-06 2024-05-15T13:21:33.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:33.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569623+0000", 2024-05-15T13:21:33.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9370000000000001e-06 2024-05-15T13:21:33.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569685+0000", 2024-05-15T13:21:33.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2867000000000006e-05 2024-05-15T13:21:33.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.569691+0000", 2024-05-15T13:21:33.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8000000000000004e-06 2024-05-15T13:21:33.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530285+0000", 2024-05-15T13:21:33.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96059413500000002 2024-05-15T13:21:33.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530298+0000", 2024-05-15T13:21:33.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2278e-05 2024-05-15T13:21:33.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.806841+0000", 2024-05-15T13:21:33.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2765429770000001 2024-05-15T13:21:33.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:33.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.806953+0000", 2024-05-15T13:21:33.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011249200000000001 2024-05-15T13:21:33.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:33.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.807060+0000", 2024-05-15T13:21:33.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000107354 2024-05-15T13:21:33.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:33.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.822037+0000", 2024-05-15T13:21:33.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014976635 2024-05-15T13:21:33.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.822097+0000", 2024-05-15T13:21:33.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9883999999999998e-05 2024-05-15T13:21:33.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.822147+0000", 2024-05-15T13:21:33.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0435000000000001e-05 2024-05-15T13:21:33.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:33.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.822189+0000", 2024-05-15T13:21:33.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1446999999999998e-05 2024-05-15T13:21:33.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.871 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:33.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.825064+0000", 2024-05-15T13:21:33.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002875505 2024-05-15T13:21:33.871 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.871 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:33.871 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.871 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:145966 7.5 7:aee1f7e1:::benchmark_data_smithi177_134484_object36901:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:21:33.871 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.588210+0000", 2024-05-15T13:21:33.872 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 354.26385494499999, 2024-05-15T13:21:33.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2397243819999999, 2024-05-15T13:21:33.872 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:33.872 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:33.872 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:33.872 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:33.872 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:33.872 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 145966 2024-05-15T13:21:33.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.872 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:33.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:33.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.588210+0000", 2024-05-15T13:21:33.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:33.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.588210+0000", 2024-05-15T13:21:33.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:33.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.588212+0000", 2024-05-15T13:21:33.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8470000000000001e-06 2024-05-15T13:21:33.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:33.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.588219+0000", 2024-05-15T13:21:33.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1239999999999999e-06 2024-05-15T13:21:33.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:33.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.588232+0000", 2024-05-15T13:21:33.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2988e-05 2024-05-15T13:21:33.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:33.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.588240+0000", 2024-05-15T13:21:33.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1110000000000008e-06 2024-05-15T13:21:33.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.588295+0000", 2024-05-15T13:21:33.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5294000000000003e-05 2024-05-15T13:21:33.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.588303+0000", 2024-05-15T13:21:33.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7810000000000005e-06 2024-05-15T13:21:33.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530361+0000", 2024-05-15T13:21:33.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.94205747900000003 2024-05-15T13:21:33.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530373+0000", 2024-05-15T13:21:33.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2500000000000001e-05 2024-05-15T13:21:33.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.813130+0000", 2024-05-15T13:21:33.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2827566130000001 2024-05-15T13:21:33.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:33.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.813244+0000", 2024-05-15T13:21:33.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114652 2024-05-15T13:21:33.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:33.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.813419+0000", 2024-05-15T13:21:33.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017432600000000001 2024-05-15T13:21:33.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:33.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.825083+0000", 2024-05-15T13:21:33.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011664091999999999 2024-05-15T13:21:33.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.825153+0000", 2024-05-15T13:21:33.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0055999999999995e-05 2024-05-15T13:21:33.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.825293+0000", 2024-05-15T13:21:33.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014042699999999999 2024-05-15T13:21:33.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:33.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.825361+0000", 2024-05-15T13:21:33.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7619000000000004e-05 2024-05-15T13:21:33.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:33.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.827934+0000", 2024-05-15T13:21:33.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025734730000000002 2024-05-15T13:21:33.879 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.879 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:33.879 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.879 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:145976 7.5 7:ab177066:::benchmark_data_smithi177_134484_object36911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:21:33.879 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.611889+0000", 2024-05-15T13:21:33.879 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 354.24017596200002, 2024-05-15T13:21:33.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2317611080000002, 2024-05-15T13:21:33.879 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:33.879 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:33.879 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:33.880 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:33.880 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:33.881 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 145976 2024-05-15T13:21:33.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.881 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:33.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:33.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.611889+0000", 2024-05-15T13:21:33.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:33.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.611889+0000", 2024-05-15T13:21:33.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.882 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.882 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:33.882 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.611890+0000", 2024-05-15T13:21:33.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2810000000000001e-06 2024-05-15T13:21:33.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.882 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.882 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:33.882 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.611897+0000", 2024-05-15T13:21:33.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.81e-06 2024-05-15T13:21:33.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.882 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:33.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.611910+0000", 2024-05-15T13:21:33.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2934e-05 2024-05-15T13:21:33.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:33.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.611917+0000", 2024-05-15T13:21:33.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7179999999999999e-06 2024-05-15T13:21:33.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.611990+0000", 2024-05-15T13:21:33.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3441000000000004e-05 2024-05-15T13:21:33.884 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.884 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.884 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.884 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.612002+0000", 2024-05-15T13:21:33.884 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1928e-05 2024-05-15T13:21:33.884 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.884 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.884 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.884 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530420+0000", 2024-05-15T13:21:33.884 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.91841766199999997 2024-05-15T13:21:33.884 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.884 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530430+0000", 2024-05-15T13:21:33.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9159999999999996e-06 2024-05-15T13:21:33.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.818055+0000", 2024-05-15T13:21:33.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.287625435 2024-05-15T13:21:33.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:33.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.818242+0000", 2024-05-15T13:21:33.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018715100000000001 2024-05-15T13:21:33.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:33.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.818409+0000", 2024-05-15T13:21:33.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016631600000000001 2024-05-15T13:21:33.886 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.886 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:33.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.825240+0000", 2024-05-15T13:21:33.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0068310080000000004 2024-05-15T13:21:33.886 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.886 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.828014+0000", 2024-05-15T13:21:33.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00277392 2024-05-15T13:21:33.886 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.887 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.887 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.840375+0000", 2024-05-15T13:21:33.887 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012361158000000001 2024-05-15T13:21:33.887 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.887 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.887 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:33.887 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.840445+0000", 2024-05-15T13:21:33.887 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9921999999999999e-05 2024-05-15T13:21:33.887 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.887 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.887 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:33.887 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.843650+0000", 2024-05-15T13:21:33.887 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0032055080000000001 2024-05-15T13:21:33.887 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.887 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:33.888 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.888 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.888 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.888 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:145984 7.5 7:a3dbbd39:::benchmark_data_smithi177_134484_object36919:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:21:33.888 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.636999+0000", 2024-05-15T13:21:33.888 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 354.215066068, 2024-05-15T13:21:33.888 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2198527179999998, 2024-05-15T13:21:33.888 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:33.888 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:33.888 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:33.888 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:33.888 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:33.888 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 145984 2024-05-15T13:21:33.889 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.889 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:33.889 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.889 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:33.889 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.636999+0000", 2024-05-15T13:21:33.889 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.889 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.889 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.889 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:33.889 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.636999+0000", 2024-05-15T13:21:33.889 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.889 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.889 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.889 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:33.890 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.637000+0000", 2024-05-15T13:21:33.890 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4219999999999999e-06 2024-05-15T13:21:33.890 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.890 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.890 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:33.890 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.637008+0000", 2024-05-15T13:21:33.890 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7479999999999993e-06 2024-05-15T13:21:33.890 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.890 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.890 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:33.890 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.637025+0000", 2024-05-15T13:21:33.890 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7059999999999999e-05 2024-05-15T13:21:33.890 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.890 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.890 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:33.891 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.637033+0000", 2024-05-15T13:21:33.891 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9640000000000003e-06 2024-05-15T13:21:33.891 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.891 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.891 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.637108+0000", 2024-05-15T13:21:33.891 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4603000000000004e-05 2024-05-15T13:21:33.891 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.891 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.891 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.637121+0000", 2024-05-15T13:21:33.891 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3145e-05 2024-05-15T13:21:33.891 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.891 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.892 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530492+0000", 2024-05-15T13:21:33.892 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.89337067400000003 2024-05-15T13:21:33.892 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.892 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.892 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.892 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530500+0000", 2024-05-15T13:21:33.892 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4330000000000004e-06 2024-05-15T13:21:33.892 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.892 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.892 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.892 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.828066+0000", 2024-05-15T13:21:33.892 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2975655580000001 2024-05-15T13:21:33.893 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.893 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.893 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:33.893 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.828240+0000", 2024-05-15T13:21:33.893 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017429500000000001 2024-05-15T13:21:33.893 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.893 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.893 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:33.893 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.828413+0000", 2024-05-15T13:21:33.893 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000173327 2024-05-15T13:21:33.893 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.893 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.893 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:33.894 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.843665+0000", 2024-05-15T13:21:33.894 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015251951999999999 2024-05-15T13:21:33.894 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.894 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.894 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.894 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.854202+0000", 2024-05-15T13:21:33.894 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010536634 2024-05-15T13:21:33.894 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.894 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.894 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.894 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.854318+0000", 2024-05-15T13:21:33.894 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011631199999999999 2024-05-15T13:21:33.895 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.895 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.895 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:33.895 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.854366+0000", 2024-05-15T13:21:33.895 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7944e-05 2024-05-15T13:21:33.895 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.895 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.895 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:33.895 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.856852+0000", 2024-05-15T13:21:33.895 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024856470000000001 2024-05-15T13:21:33.895 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.895 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:33.895 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.895 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.895 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.896 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:146006 7.5 7:a497d408:::benchmark_data_smithi177_134484_object36941:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:21:33.896 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.688862+0000", 2024-05-15T13:21:33.896 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 354.16320335099999, 2024-05-15T13:21:33.896 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.18289022, 2024-05-15T13:21:33.896 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:33.896 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:33.896 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:33.896 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:33.896 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:33.896 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 146006 2024-05-15T13:21:33.896 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.896 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:33.896 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.896 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:33.896 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.688862+0000", 2024-05-15T13:21:33.896 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.897 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.897 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.897 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:33.897 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.688862+0000", 2024-05-15T13:21:33.897 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.897 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.897 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.897 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:33.897 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.688863+0000", 2024-05-15T13:21:33.897 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2300000000000001e-06 2024-05-15T13:21:33.897 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.897 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.897 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:33.898 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.688868+0000", 2024-05-15T13:21:33.898 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5249999999999996e-06 2024-05-15T13:21:33.898 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.898 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.898 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:33.898 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.688880+0000", 2024-05-15T13:21:33.898 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1637999999999999e-05 2024-05-15T13:21:33.898 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.898 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.898 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:33.898 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.688888+0000", 2024-05-15T13:21:33.898 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0949999999999996e-06 2024-05-15T13:21:33.898 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.898 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.899 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.899 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.688951+0000", 2024-05-15T13:21:33.899 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3332999999999998e-05 2024-05-15T13:21:33.899 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.899 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.899 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.899 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.688963+0000", 2024-05-15T13:21:33.899 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.172e-05 2024-05-15T13:21:33.899 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.899 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.899 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.899 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530549+0000", 2024-05-15T13:21:33.899 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.84158606400000002 2024-05-15T13:21:33.899 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.899 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.900 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.900 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530562+0000", 2024-05-15T13:21:33.900 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3243e-05 2024-05-15T13:21:33.900 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.900 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.900 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.900 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.832724+0000", 2024-05-15T13:21:33.900 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3021619609999999 2024-05-15T13:21:33.900 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.900 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.900 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:33.900 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.832845+0000", 2024-05-15T13:21:33.900 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120609 2024-05-15T13:21:33.900 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.901 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.901 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:33.901 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.832953+0000", 2024-05-15T13:21:33.901 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108044 2024-05-15T13:21:33.901 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.901 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.901 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:33.901 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.850232+0000", 2024-05-15T13:21:33.901 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017278853 2024-05-15T13:21:33.901 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.901 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.901 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.901 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.854263+0000", 2024-05-15T13:21:33.901 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0040310479999999997 2024-05-15T13:21:33.902 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.902 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.902 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.902 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.868854+0000", 2024-05-15T13:21:33.902 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014591425 2024-05-15T13:21:33.902 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.902 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.902 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:33.902 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.868931+0000", 2024-05-15T13:21:33.902 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6642000000000002e-05 2024-05-15T13:21:33.902 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.902 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.902 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:33.903 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.871752+0000", 2024-05-15T13:21:33.903 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0028207900000000001 2024-05-15T13:21:33.903 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.903 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:33.903 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.903 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.903 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.903 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:146038 7.5 7:a8212863:::benchmark_data_smithi177_134484_object36973:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:21:33.903 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.762883+0000", 2024-05-15T13:21:33.903 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 354.089182365, 2024-05-15T13:21:33.903 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1114912019999998, 2024-05-15T13:21:33.903 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:33.904 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:33.904 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:33.904 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:33.904 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:33.904 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 146038 2024-05-15T13:21:33.904 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.904 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:33.904 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.904 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:33.904 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.762883+0000", 2024-05-15T13:21:33.904 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.904 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.904 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.905 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:33.905 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.762883+0000", 2024-05-15T13:21:33.905 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.905 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.905 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.905 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:33.905 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.762884+0000", 2024-05-15T13:21:33.905 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.469e-06 2024-05-15T13:21:33.905 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.905 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.905 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:33.905 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.762890+0000", 2024-05-15T13:21:33.905 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.117e-06 2024-05-15T13:21:33.906 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.906 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.906 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:33.906 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.762904+0000", 2024-05-15T13:21:33.906 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3906e-05 2024-05-15T13:21:33.906 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.906 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.906 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:33.906 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.762913+0000", 2024-05-15T13:21:33.906 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5269999999999998e-06 2024-05-15T13:21:33.906 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.906 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.907 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.907 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.762973+0000", 2024-05-15T13:21:33.907 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0609000000000002e-05 2024-05-15T13:21:33.907 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.907 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.907 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.907 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.762986+0000", 2024-05-15T13:21:33.907 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2736e-05 2024-05-15T13:21:33.907 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.907 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.907 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.907 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530624+0000", 2024-05-15T13:21:33.907 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.76763827900000003 2024-05-15T13:21:33.907 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.908 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.908 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.908 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530633+0000", 2024-05-15T13:21:33.908 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6969999999999999e-06 2024-05-15T13:21:33.908 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.908 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.908 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.908 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.836480+0000", 2024-05-15T13:21:33.908 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3058468300000001 2024-05-15T13:21:33.908 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.908 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.908 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:33.908 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.836597+0000", 2024-05-15T13:21:33.908 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117296 2024-05-15T13:21:33.908 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.909 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.909 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:33.909 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.836717+0000", 2024-05-15T13:21:33.909 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000119752 2024-05-15T13:21:33.909 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.909 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.909 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:33.909 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.850239+0000", 2024-05-15T13:21:33.909 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013522008 2024-05-15T13:21:33.909 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.909 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.909 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.909 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.856916+0000", 2024-05-15T13:21:33.909 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0066768829999999998 2024-05-15T13:21:33.909 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.910 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.910 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.910 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.871813+0000", 2024-05-15T13:21:33.910 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.01489701 2024-05-15T13:21:33.910 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.910 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.910 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:33.910 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.871858+0000", 2024-05-15T13:21:33.910 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5318999999999998e-05 2024-05-15T13:21:33.910 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.910 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.910 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:33.910 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.874374+0000", 2024-05-15T13:21:33.910 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025157640000000002 2024-05-15T13:21:33.911 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.911 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:33.911 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.911 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.911 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.911 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:146045 7.5 7:a967a4c6:::benchmark_data_smithi177_134484_object36980:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:21:33.911 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.777741+0000", 2024-05-15T13:21:33.911 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 354.07432439299998, 2024-05-15T13:21:33.911 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1091810830000002, 2024-05-15T13:21:33.911 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:33.911 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:33.911 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:33.911 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:33.911 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:33.911 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 146045 2024-05-15T13:21:33.911 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.912 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:33.912 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.912 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:33.912 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.777741+0000", 2024-05-15T13:21:33.912 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.912 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.912 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.912 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:33.912 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.777741+0000", 2024-05-15T13:21:33.912 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.912 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.912 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.912 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:33.912 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.777742+0000", 2024-05-15T13:21:33.912 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.288e-06 2024-05-15T13:21:33.912 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.913 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.913 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:33.913 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.777748+0000", 2024-05-15T13:21:33.913 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0009999999999998e-06 2024-05-15T13:21:33.913 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.913 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.913 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:33.913 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.777760+0000", 2024-05-15T13:21:33.913 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1987999999999999e-05 2024-05-15T13:21:33.913 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.913 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.913 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:33.913 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.777768+0000", 2024-05-15T13:21:33.913 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2360000000000004e-06 2024-05-15T13:21:33.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.914 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.777828+0000", 2024-05-15T13:21:33.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9814e-05 2024-05-15T13:21:33.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.914 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.777838+0000", 2024-05-15T13:21:33.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7350000000000008e-06 2024-05-15T13:21:33.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530677+0000", 2024-05-15T13:21:33.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.75283949299999997 2024-05-15T13:21:33.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.915 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530689+0000", 2024-05-15T13:21:33.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2292e-05 2024-05-15T13:21:33.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.915 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.843709+0000", 2024-05-15T13:21:33.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.313019911 2024-05-15T13:21:33.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.916 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:33.916 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.843836+0000", 2024-05-15T13:21:33.916 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012651599999999999 2024-05-15T13:21:33.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.916 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:33.916 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.843950+0000", 2024-05-15T13:21:33.916 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011428499999999999 2024-05-15T13:21:33.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.916 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:33.917 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.854275+0000", 2024-05-15T13:21:33.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010324453000000001 2024-05-15T13:21:33.917 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.917 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.917 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.874462+0000", 2024-05-15T13:21:33.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.020187159 2024-05-15T13:21:33.917 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.917 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.917 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.883398+0000", 2024-05-15T13:21:33.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0089358379999999998 2024-05-15T13:21:33.917 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:33.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.883455+0000", 2024-05-15T13:21:33.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7015000000000002e-05 2024-05-15T13:21:33.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:33.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.886922+0000", 2024-05-15T13:21:33.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0034670590000000002 2024-05-15T13:21:33.918 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.918 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:33.918 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.919 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:146047 7.5 7:af883097:::benchmark_data_smithi177_134484_object36982:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:21:33.919 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.783367+0000", 2024-05-15T13:21:33.919 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 354.06869802400001, 2024-05-15T13:21:33.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1068296379999998, 2024-05-15T13:21:33.919 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:33.919 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:33.919 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:33.919 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:33.919 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:33.919 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 146047 2024-05-15T13:21:33.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.919 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:33.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.919 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:33.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.783367+0000", 2024-05-15T13:21:33.920 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.920 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.920 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.920 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:33.920 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.783367+0000", 2024-05-15T13:21:33.920 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.920 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.920 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.920 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:33.920 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.783369+0000", 2024-05-15T13:21:33.920 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6029999999999999e-06 2024-05-15T13:21:33.920 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.920 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.920 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:33.920 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.783374+0000", 2024-05-15T13:21:33.921 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8100000000000003e-06 2024-05-15T13:21:33.921 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.921 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.921 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:33.921 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.783387+0000", 2024-05-15T13:21:33.921 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2296999999999999e-05 2024-05-15T13:21:33.921 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.921 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.921 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:33.921 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.783395+0000", 2024-05-15T13:21:33.921 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1689999999999992e-06 2024-05-15T13:21:33.921 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.921 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.922 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.922 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.783459+0000", 2024-05-15T13:21:33.922 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3909000000000001e-05 2024-05-15T13:21:33.922 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.922 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.922 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.922 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.783465+0000", 2024-05-15T13:21:33.922 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7279999999999998e-06 2024-05-15T13:21:33.922 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.922 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.922 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.922 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530725+0000", 2024-05-15T13:21:33.922 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.74725946799999998 2024-05-15T13:21:33.922 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530730+0000", 2024-05-15T13:21:33.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0000000000000004e-06 2024-05-15T13:21:33.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.850301+0000", 2024-05-15T13:21:33.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.319570774 2024-05-15T13:21:33.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:33.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.850451+0000", 2024-05-15T13:21:33.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000150266 2024-05-15T13:21:33.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.924 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:33.924 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.850597+0000", 2024-05-15T13:21:33.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014563600000000001 2024-05-15T13:21:33.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.924 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:33.924 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.864951+0000", 2024-05-15T13:21:33.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014353899999999999 2024-05-15T13:21:33.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.924 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.924 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.883335+0000", 2024-05-15T13:21:33.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.018384939999999999 2024-05-15T13:21:33.925 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.925 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.925 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.925 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.887036+0000", 2024-05-15T13:21:33.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037008459999999998 2024-05-15T13:21:33.925 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.925 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.925 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:33.925 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.887084+0000", 2024-05-15T13:21:33.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7394000000000001e-05 2024-05-15T13:21:33.925 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.925 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.925 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:33.925 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.890197+0000", 2024-05-15T13:21:33.926 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031128979999999998 2024-05-15T13:21:33.926 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.926 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:33.926 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.926 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.926 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.926 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:146049 7.5 7:ac75f5ad:::benchmark_data_smithi177_134484_object36984:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:21:33.926 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.791535+0000", 2024-05-15T13:21:33.926 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 354.06053006299999, 2024-05-15T13:21:33.926 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1017435240000002, 2024-05-15T13:21:33.926 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:33.926 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:33.926 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:33.927 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:33.927 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:33.927 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 146049 2024-05-15T13:21:33.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.927 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:33.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.927 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:33.927 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.791535+0000", 2024-05-15T13:21:33.927 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.927 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:33.927 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.791535+0000", 2024-05-15T13:21:33.927 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.928 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.928 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.928 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:33.928 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.791536+0000", 2024-05-15T13:21:33.928 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0529999999999999e-06 2024-05-15T13:21:33.928 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.928 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.928 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:33.928 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.791541+0000", 2024-05-15T13:21:33.928 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9300000000000002e-06 2024-05-15T13:21:33.928 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.928 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.928 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:33.929 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.791552+0000", 2024-05-15T13:21:33.929 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1117e-05 2024-05-15T13:21:33.929 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.929 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.929 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:33.929 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.791558+0000", 2024-05-15T13:21:33.929 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8209999999999998e-06 2024-05-15T13:21:33.929 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.929 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.929 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.929 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.791613+0000", 2024-05-15T13:21:33.929 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4722000000000002e-05 2024-05-15T13:21:33.929 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.929 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.930 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.930 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.791621+0000", 2024-05-15T13:21:33.930 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2989999999999997e-06 2024-05-15T13:21:33.930 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.930 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.930 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.930 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530771+0000", 2024-05-15T13:21:33.930 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.73915060200000005 2024-05-15T13:21:33.930 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.930 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.930 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.930 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530783+0000", 2024-05-15T13:21:33.931 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1513e-05 2024-05-15T13:21:33.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.931 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.931 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.931 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.856950+0000", 2024-05-15T13:21:33.931 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.326167332 2024-05-15T13:21:33.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.931 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.931 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:33.931 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.857060+0000", 2024-05-15T13:21:33.931 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010977899999999999 2024-05-15T13:21:33.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.931 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.931 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:33.931 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.857169+0000", 2024-05-15T13:21:33.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010908699999999999 2024-05-15T13:21:33.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.932 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.932 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:33.932 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.871772+0000", 2024-05-15T13:21:33.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014602364 2024-05-15T13:21:33.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.932 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.932 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.932 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.886979+0000", 2024-05-15T13:21:33.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015207001 2024-05-15T13:21:33.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.932 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.932 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.933 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.890279+0000", 2024-05-15T13:21:33.933 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0033007190000000001 2024-05-15T13:21:33.933 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.933 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.933 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:33.933 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.890345+0000", 2024-05-15T13:21:33.933 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5369000000000004e-05 2024-05-15T13:21:33.933 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.933 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.933 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:33.933 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.893278+0000", 2024-05-15T13:21:33.933 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0029338160000000001 2024-05-15T13:21:33.933 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.934 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:33.934 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.934 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.934 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.934 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:146056 7.5 7:a6b3a042:::benchmark_data_smithi177_134484_object36991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:21:33.934 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.815388+0000", 2024-05-15T13:21:33.934 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 354.03667676999999, 2024-05-15T13:21:33.934 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0813361119999998, 2024-05-15T13:21:33.934 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:33.934 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:33.934 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:33.934 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:33.934 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:33.935 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 146056 2024-05-15T13:21:33.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.935 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:33.935 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.935 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:33.935 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.815388+0000", 2024-05-15T13:21:33.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.935 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.935 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:33.935 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.815388+0000", 2024-05-15T13:21:33.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.936 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.936 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:33.936 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.815390+0000", 2024-05-15T13:21:33.936 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9230000000000001e-06 2024-05-15T13:21:33.936 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.936 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.936 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:33.936 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.815397+0000", 2024-05-15T13:21:33.936 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6880000000000002e-06 2024-05-15T13:21:33.936 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.936 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.936 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:33.936 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.815412+0000", 2024-05-15T13:21:33.937 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5184999999999999e-05 2024-05-15T13:21:33.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.937 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.937 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:33.937 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.815421+0000", 2024-05-15T13:21:33.937 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7360000000000007e-06 2024-05-15T13:21:33.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.937 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.937 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.937 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.815486+0000", 2024-05-15T13:21:33.937 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5191000000000007e-05 2024-05-15T13:21:33.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.937 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.937 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.938 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.815495+0000", 2024-05-15T13:21:33.938 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0089999999999996e-06 2024-05-15T13:21:33.938 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.938 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.938 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.938 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530838+0000", 2024-05-15T13:21:33.938 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.71534338099999994 2024-05-15T13:21:33.938 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.938 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.938 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.938 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530850+0000", 2024-05-15T13:21:33.938 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1725999999999999e-05 2024-05-15T13:21:33.938 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.939 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.939 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.861247+0000", 2024-05-15T13:21:33.939 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3303964509999999 2024-05-15T13:21:33.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.939 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:33.939 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.861408+0000", 2024-05-15T13:21:33.939 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016124800000000001 2024-05-15T13:21:33.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.939 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:33.939 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.861518+0000", 2024-05-15T13:21:33.939 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000110092 2024-05-15T13:21:33.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.940 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.940 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:33.940 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.874481+0000", 2024-05-15T13:21:33.940 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012963595 2024-05-15T13:21:33.940 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.940 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.940 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.940 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.893333+0000", 2024-05-15T13:21:33.940 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.018851402999999999 2024-05-15T13:21:33.940 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.940 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.940 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.940 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.893362+0000", 2024-05-15T13:21:33.940 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9371999999999999e-05 2024-05-15T13:21:33.940 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.941 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:33.941 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.893408+0000", 2024-05-15T13:21:33.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5445000000000003e-05 2024-05-15T13:21:33.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.941 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:33.941 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.896724+0000", 2024-05-15T13:21:33.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0033166670000000001 2024-05-15T13:21:33.941 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.941 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:33.941 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.942 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.942 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:146091 7.5 7:aeed9694:::benchmark_data_smithi177_134484_object37026:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:21:33.942 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.950203+0000", 2024-05-15T13:21:33.942 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 353.90186175399998, 2024-05-15T13:21:33.942 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9582113840000002, 2024-05-15T13:21:33.942 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:33.942 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:33.942 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:33.942 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:33.942 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:33.942 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 146091 2024-05-15T13:21:33.942 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.942 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:33.942 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.942 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:33.943 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.950203+0000", 2024-05-15T13:21:33.943 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.943 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.943 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.943 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:33.943 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.950203+0000", 2024-05-15T13:21:33.943 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.943 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.943 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.943 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:33.943 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.950204+0000", 2024-05-15T13:21:33.943 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.175e-06 2024-05-15T13:21:33.943 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.943 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.943 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:33.944 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.950211+0000", 2024-05-15T13:21:33.944 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7989999999999996e-06 2024-05-15T13:21:33.944 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.944 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.944 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:33.944 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.950225+0000", 2024-05-15T13:21:33.944 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3940999999999999e-05 2024-05-15T13:21:33.944 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.944 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.944 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:33.944 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.950232+0000", 2024-05-15T13:21:33.944 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.489e-06 2024-05-15T13:21:33.944 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.944 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.944 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.945 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.950293+0000", 2024-05-15T13:21:33.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1647000000000001e-05 2024-05-15T13:21:33.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.945 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.945 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.945 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.950301+0000", 2024-05-15T13:21:33.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6109999999999996e-06 2024-05-15T13:21:33.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.945 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.945 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.945 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530901+0000", 2024-05-15T13:21:33.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.580600373 2024-05-15T13:21:33.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.945 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.946 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.946 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530913+0000", 2024-05-15T13:21:33.946 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1718e-05 2024-05-15T13:21:33.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.946 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.946 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.946 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.865005+0000", 2024-05-15T13:21:33.946 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3340925129999999 2024-05-15T13:21:33.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.946 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.946 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:33.946 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.865116+0000", 2024-05-15T13:21:33.946 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011082300000000001 2024-05-15T13:21:33.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:33.947 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.865238+0000", 2024-05-15T13:21:33.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000121303 2024-05-15T13:21:33.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:33.947 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.874491+0000", 2024-05-15T13:21:33.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0092530589999999992 2024-05-15T13:21:33.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.947 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.896804+0000", 2024-05-15T13:21:33.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.022313625 2024-05-15T13:21:33.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.948 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.948 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.948 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.905865+0000", 2024-05-15T13:21:33.948 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0090606160000000005 2024-05-15T13:21:33.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.948 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.948 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:33.948 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.905905+0000", 2024-05-15T13:21:33.948 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9866000000000001e-05 2024-05-15T13:21:33.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.948 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.948 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:33.948 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.908415+0000", 2024-05-15T13:21:33.948 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025098260000000002 2024-05-15T13:21:33.949 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.949 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:33.949 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.949 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.949 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:146097 7.5 7:a626cefc:::benchmark_data_smithi177_134484_object37032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:21:33.949 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.982750+0000", 2024-05-15T13:21:33.949 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 353.869315331, 2024-05-15T13:21:33.949 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9281458680000001, 2024-05-15T13:21:33.949 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:33.949 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:33.949 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:33.949 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:33.949 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:33.949 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 146097 2024-05-15T13:21:33.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.950 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:33.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:33.950 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.982750+0000", 2024-05-15T13:21:33.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:33.950 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.982750+0000", 2024-05-15T13:21:33.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:33.950 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.982751+0000", 2024-05-15T13:21:33.951 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.339e-06 2024-05-15T13:21:33.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.951 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.951 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:33.951 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.982757+0000", 2024-05-15T13:21:33.951 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6350000000000003e-06 2024-05-15T13:21:33.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.951 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.951 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:33.951 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.982769+0000", 2024-05-15T13:21:33.951 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2581e-05 2024-05-15T13:21:33.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.951 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.951 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:33.951 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.982776+0000", 2024-05-15T13:21:33.951 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3729999999999998e-06 2024-05-15T13:21:33.952 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.952 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.952 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.952 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.982828+0000", 2024-05-15T13:21:33.952 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2868000000000003e-05 2024-05-15T13:21:33.952 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.952 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.952 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.952 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.982838+0000", 2024-05-15T13:21:33.952 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3270000000000007e-06 2024-05-15T13:21:33.952 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.952 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.952 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.953 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530958+0000", 2024-05-15T13:21:33.953 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.548119837 2024-05-15T13:21:33.953 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.953 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.953 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.953 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.530969+0000", 2024-05-15T13:21:33.953 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1474000000000001e-05 2024-05-15T13:21:33.953 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.953 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.953 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.953 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.874519+0000", 2024-05-15T13:21:33.953 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.343550003 2024-05-15T13:21:33.953 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.954 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.954 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:33.954 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.874697+0000", 2024-05-15T13:21:33.954 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017830999999999999 2024-05-15T13:21:33.954 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.954 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.954 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:33.954 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.874814+0000", 2024-05-15T13:21:33.954 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000116564 2024-05-15T13:21:33.954 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.954 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.954 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:33.954 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.890221+0000", 2024-05-15T13:21:33.954 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015406716000000001 2024-05-15T13:21:33.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.955 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.955 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.955 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.896866+0000", 2024-05-15T13:21:33.955 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0066452270000000001 2024-05-15T13:21:33.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.955 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.955 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.955 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.908480+0000", 2024-05-15T13:21:33.955 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0116144 2024-05-15T13:21:33.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.955 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.955 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:33.955 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.908524+0000", 2024-05-15T13:21:33.956 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4098999999999999e-05 2024-05-15T13:21:33.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.956 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.956 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:33.956 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.910896+0000", 2024-05-15T13:21:33.956 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002371115 2024-05-15T13:21:33.956 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.956 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:33.956 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.956 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.956 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:146098 7.5 7:a23e1871:::benchmark_data_smithi177_134484_object37033:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:21:33.956 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:39.990742+0000", 2024-05-15T13:21:33.956 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 353.86132267200003, 2024-05-15T13:21:33.957 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9231258900000001, 2024-05-15T13:21:33.957 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:33.957 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:33.957 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:33.957 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:33.957 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:33.957 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 146098 2024-05-15T13:21:33.957 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.957 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:33.957 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.957 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:33.957 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.990742+0000", 2024-05-15T13:21:33.957 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.957 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.957 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.958 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:33.958 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.990742+0000", 2024-05-15T13:21:33.958 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.958 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.958 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:33.958 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.990744+0000", 2024-05-15T13:21:33.958 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4079999999999999e-06 2024-05-15T13:21:33.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.958 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.958 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:33.958 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.990749+0000", 2024-05-15T13:21:33.958 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5140000000000001e-06 2024-05-15T13:21:33.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.958 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.958 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:33.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.990761+0000", 2024-05-15T13:21:33.959 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2225000000000001e-05 2024-05-15T13:21:33.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:33.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.990768+0000", 2024-05-15T13:21:33.959 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0339999999999999e-06 2024-05-15T13:21:33.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.990823+0000", 2024-05-15T13:21:33.959 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4178000000000002e-05 2024-05-15T13:21:33.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:39.990830+0000", 2024-05-15T13:21:33.960 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5290000000000003e-06 2024-05-15T13:21:33.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.960 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.960 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.960 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.531010+0000", 2024-05-15T13:21:33.960 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.54018028200000001 2024-05-15T13:21:33.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.960 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.960 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.960 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.531017+0000", 2024-05-15T13:21:33.960 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9789999999999996e-06 2024-05-15T13:21:33.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.960 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.960 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.960 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.878483+0000", 2024-05-15T13:21:33.960 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3474650509999999 2024-05-15T13:21:33.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.961 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:33.961 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.878650+0000", 2024-05-15T13:21:33.961 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000167897 2024-05-15T13:21:33.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.961 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:33.961 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.878805+0000", 2024-05-15T13:21:33.961 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015463400000000001 2024-05-15T13:21:33.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.961 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:33.961 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.893292+0000", 2024-05-15T13:21:33.962 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014487458 2024-05-15T13:21:33.962 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.962 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.962 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.962 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.896924+0000", 2024-05-15T13:21:33.962 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0036316970000000001 2024-05-15T13:21:33.962 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.962 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.962 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.962 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.910956+0000", 2024-05-15T13:21:33.962 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014032205000000001 2024-05-15T13:21:33.962 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.962 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.962 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:33.963 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.911017+0000", 2024-05-15T13:21:33.963 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0924000000000001e-05 2024-05-15T13:21:33.963 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.963 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.963 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:33.963 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.913868+0000", 2024-05-15T13:21:33.963 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0028508750000000001 2024-05-15T13:21:33.963 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.963 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:33.963 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.963 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.963 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.963 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:146107 7.5 7:a81c3c36:::benchmark_data_smithi177_134484_object37042:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:21:33.963 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:40.061892+0000", 2024-05-15T13:21:33.964 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 353.79017308900001, 2024-05-15T13:21:33.964 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8648366159999998, 2024-05-15T13:21:33.964 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:33.964 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:33.964 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:33.964 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:33.964 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:33.964 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 146107 2024-05-15T13:21:33.964 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.964 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:33.964 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.964 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:33.964 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.061892+0000", 2024-05-15T13:21:33.964 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.965 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.965 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:33.965 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.061892+0000", 2024-05-15T13:21:33.965 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.965 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.965 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:33.965 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.061893+0000", 2024-05-15T13:21:33.965 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.57e-06 2024-05-15T13:21:33.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.965 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.965 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:33.965 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.061899+0000", 2024-05-15T13:21:33.966 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2780000000000001e-06 2024-05-15T13:21:33.966 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:21:33.967 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.968 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.968 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:33.968 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.061912+0000", 2024-05-15T13:21:33.968 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2865e-05 2024-05-15T13:21:33.968 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.968 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.968 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:33.968 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.061918+0000", 2024-05-15T13:21:33.968 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5400000000000001e-06 2024-05-15T13:21:33.968 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.968 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.968 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.968 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.061970+0000", 2024-05-15T13:21:33.969 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1683e-05 2024-05-15T13:21:33.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.969 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.969 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.969 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.061978+0000", 2024-05-15T13:21:33.969 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8140000000000001e-06 2024-05-15T13:21:33.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.969 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.969 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.969 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.531056+0000", 2024-05-15T13:21:33.969 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46907885199999999 2024-05-15T13:21:33.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.969 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.969 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.970 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.531066+0000", 2024-05-15T13:21:33.970 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6730000000000004e-06 2024-05-15T13:21:33.970 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.970 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.970 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.970 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.896961+0000", 2024-05-15T13:21:33.970 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3658945419999999 2024-05-15T13:21:33.970 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.970 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.970 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:33.970 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.897084+0000", 2024-05-15T13:21:33.970 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000123349 2024-05-15T13:21:33.971 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.971 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.971 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:33.971 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.897193+0000", 2024-05-15T13:21:33.971 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108665 2024-05-15T13:21:33.971 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.971 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.971 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:33.971 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.910909+0000", 2024-05-15T13:21:33.971 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013716425000000001 2024-05-15T13:21:33.971 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.971 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.971 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.972 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.913942+0000", 2024-05-15T13:21:33.972 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003032603 2024-05-15T13:21:33.972 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.972 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.972 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.972 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.923664+0000", 2024-05-15T13:21:33.972 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0097224900000000003 2024-05-15T13:21:33.972 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.972 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.972 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:33.972 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.923730+0000", 2024-05-15T13:21:33.972 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5875000000000002e-05 2024-05-15T13:21:33.972 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.973 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.973 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:33.973 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.926729+0000", 2024-05-15T13:21:33.973 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0029983919999999999 2024-05-15T13:21:33.973 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.973 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:33.973 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.973 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.973 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.973 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:146109 7.5 7:a65de73c:::benchmark_data_smithi177_134484_object37044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-05-15T13:21:33.973 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:40.073893+0000", 2024-05-15T13:21:33.973 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 353.77817210500001, 2024-05-15T13:21:33.973 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.856148006, 2024-05-15T13:21:33.973 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:33.974 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:33.974 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:33.974 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:21:33.974 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:21:33.974 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 146109 2024-05-15T13:21:33.974 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.974 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:33.974 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.974 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:33.974 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.073893+0000", 2024-05-15T13:21:33.974 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.974 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.974 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.975 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:33.975 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.073893+0000", 2024-05-15T13:21:33.975 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.975 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.975 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.975 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:33.975 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.073894+0000", 2024-05-15T13:21:33.975 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3680000000000001e-06 2024-05-15T13:21:33.975 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.975 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.975 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:33.975 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.073899+0000", 2024-05-15T13:21:33.975 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9910000000000002e-06 2024-05-15T13:21:33.975 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.976 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.976 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:33.976 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.073911+0000", 2024-05-15T13:21:33.976 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1781e-05 2024-05-15T13:21:33.976 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.976 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.976 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:33.976 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.073919+0000", 2024-05-15T13:21:33.976 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1559999999999995e-06 2024-05-15T13:21:33.976 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.976 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.976 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.976 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.074002+0000", 2024-05-15T13:21:33.976 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2902e-05 2024-05-15T13:21:33.977 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.977 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.977 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.977 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.074010+0000", 2024-05-15T13:21:33.977 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4200000000000007e-06 2024-05-15T13:21:33.977 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.977 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.977 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.977 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.531086+0000", 2024-05-15T13:21:33.977 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.45707555799999999 2024-05-15T13:21:33.977 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.977 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.977 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.977 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:40.531096+0000", 2024-05-15T13:21:33.978 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0284e-05 2024-05-15T13:21:33.978 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.978 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.978 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.978 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.900952+0000", 2024-05-15T13:21:33.978 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.369855265 2024-05-15T13:21:33.978 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.978 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.978 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:33.978 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.901099+0000", 2024-05-15T13:21:33.978 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014783600000000001 2024-05-15T13:21:33.978 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.978 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.979 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 0,9", 2024-05-15T13:21:33.979 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.901255+0000", 2024-05-15T13:21:33.979 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015536400000000001 2024-05-15T13:21:33.979 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.979 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.979 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:33.979 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.913959+0000", 2024-05-15T13:21:33.979 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012703743 2024-05-15T13:21:33.979 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.979 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.979 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.979 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.914003+0000", 2024-05-15T13:21:33.979 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4048999999999998e-05 2024-05-15T13:21:33.979 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.980 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.980 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.980 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.926805+0000", 2024-05-15T13:21:33.980 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012802167999999999 2024-05-15T13:21:33.980 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.980 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.980 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:33.980 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.926847+0000", 2024-05-15T13:21:33.980 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2052999999999999e-05 2024-05-15T13:21:33.980 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.980 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.980 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:33.980 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:41.930041+0000", 2024-05-15T13:21:33.980 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003194068 2024-05-15T13:21:33.980 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.981 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:33.981 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.981 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.981 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.981 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26299.0:8122 8.6 8:66a7198d:::benchmark_data_smithi177_149673_object8121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e762)", 2024-05-15T13:21:33.981 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:19:41.632057+0000", 2024-05-15T13:21:33.981 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 112.220007563, 2024-05-15T13:21:33.981 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.56742827399999995, 2024-05-15T13:21:33.981 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:33.981 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:33.981 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:33.981 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26299", 2024-05-15T13:21:33.981 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:21:33.981 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 8122 2024-05-15T13:21:33.982 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.982 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:33.982 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.982 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:33.982 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.632057+0000", 2024-05-15T13:21:33.982 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.982 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.982 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.982 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:33.982 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.632057+0000", 2024-05-15T13:21:33.982 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.982 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.982 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.983 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:33.983 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.632059+0000", 2024-05-15T13:21:33.983 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2419999999999999e-06 2024-05-15T13:21:33.983 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.983 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.983 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:33.983 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.632119+0000", 2024-05-15T13:21:33.983 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0786999999999999e-05 2024-05-15T13:21:33.983 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.983 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.983 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:33.983 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.632134+0000", 2024-05-15T13:21:33.983 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4219999999999999e-05 2024-05-15T13:21:33.983 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.983 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.984 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:33.984 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.632143+0000", 2024-05-15T13:21:33.984 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5599999999999999e-06 2024-05-15T13:21:33.984 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.984 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.984 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.984 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.632220+0000", 2024-05-15T13:21:33.984 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6531000000000003e-05 2024-05-15T13:21:33.984 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.984 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.984 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.984 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.632227+0000", 2024-05-15T13:21:33.984 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9879999999999998e-06 2024-05-15T13:21:33.984 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.985 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.985 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.985 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.195124+0000", 2024-05-15T13:21:33.985 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.56289726600000001 2024-05-15T13:21:33.985 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.985 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.985 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:33.985 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.195269+0000", 2024-05-15T13:21:33.985 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000144728 2024-05-15T13:21:33.985 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.985 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.985 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7,10", 2024-05-15T13:21:33.985 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.195431+0000", 2024-05-15T13:21:33.985 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016216099999999999 2024-05-15T13:21:33.986 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.986 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.986 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.986 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.198170+0000", 2024-05-15T13:21:33.986 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0027386210000000001 2024-05-15T13:21:33.986 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.986 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.986 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.986 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.198215+0000", 2024-05-15T13:21:33.986 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5420999999999998e-05 2024-05-15T13:21:33.986 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.986 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.986 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:33.987 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.199354+0000", 2024-05-15T13:21:33.987 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011392399999999999 2024-05-15T13:21:33.987 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.987 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.987 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:33.987 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.199408+0000", 2024-05-15T13:21:33.987 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3940999999999997e-05 2024-05-15T13:21:33.987 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.987 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.987 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:33.987 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.199486+0000", 2024-05-15T13:21:33.987 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7568e-05 2024-05-15T13:21:33.987 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.988 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:33.988 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.988 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.988 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.988 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26299.0:8164 8.6 8:68ab8acd:::benchmark_data_smithi177_149673_object8163:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e762)", 2024-05-15T13:21:33.988 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:19:41.644436+0000", 2024-05-15T13:21:33.988 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 112.207628832, 2024-05-15T13:21:33.988 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.55521590899999995, 2024-05-15T13:21:33.988 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:33.988 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:33.988 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:33.988 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26299", 2024-05-15T13:21:33.988 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:21:33.989 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 8164 2024-05-15T13:21:33.989 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.989 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:33.989 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.989 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:33.989 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.644436+0000", 2024-05-15T13:21:33.989 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.989 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.989 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.989 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:33.989 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.644436+0000", 2024-05-15T13:21:33.989 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.989 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.990 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.990 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:33.990 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.644437+0000", 2024-05-15T13:21:33.990 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5199999999999996e-07 2024-05-15T13:21:33.990 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.990 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.990 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:33.990 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.644458+0000", 2024-05-15T13:21:33.990 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0709000000000001e-05 2024-05-15T13:21:33.990 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.990 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.990 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:33.990 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.644466+0000", 2024-05-15T13:21:33.990 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4100000000000008e-06 2024-05-15T13:21:33.991 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.991 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.991 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:33.991 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.644472+0000", 2024-05-15T13:21:33.991 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1190000000000002e-06 2024-05-15T13:21:33.991 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.991 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.991 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.991 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.644530+0000", 2024-05-15T13:21:33.991 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8281e-05 2024-05-15T13:21:33.991 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.991 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.991 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.991 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.644539+0000", 2024-05-15T13:21:33.991 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5299999999999996e-06 2024-05-15T13:21:33.992 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.992 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.992 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.992 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.196581+0000", 2024-05-15T13:21:33.992 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.55204165400000005 2024-05-15T13:21:33.992 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.992 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.992 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:33.992 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.196698+0000", 2024-05-15T13:21:33.992 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117347 2024-05-15T13:21:33.992 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.992 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.992 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7,10", 2024-05-15T13:21:33.992 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.196850+0000", 2024-05-15T13:21:33.992 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000152607 2024-05-15T13:21:33.993 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.993 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.993 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.993 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.198263+0000", 2024-05-15T13:21:33.993 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001412663 2024-05-15T13:21:33.993 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.993 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.993 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:33.993 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.198303+0000", 2024-05-15T13:21:33.993 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9410999999999999e-05 2024-05-15T13:21:33.993 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.993 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.993 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:33.993 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.199586+0000", 2024-05-15T13:21:33.993 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012839049999999999 2024-05-15T13:21:33.994 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.994 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.994 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:33.994 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.199620+0000", 2024-05-15T13:21:33.994 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3698000000000002e-05 2024-05-15T13:21:33.994 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.994 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.994 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:33.994 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.199652+0000", 2024-05-15T13:21:33.994 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1922999999999999e-05 2024-05-15T13:21:33.994 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.994 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:33.994 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:33.995 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.995 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.995 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26299.0:8166 8.6 8:6d8ccde4:::benchmark_data_smithi177_149673_object8165:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e762)", 2024-05-15T13:21:33.995 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:19:41.644490+0000", 2024-05-15T13:21:33.995 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 112.207574745, 2024-05-15T13:21:33.995 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.555253157, 2024-05-15T13:21:33.995 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:33.995 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:33.995 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:33.995 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26299", 2024-05-15T13:21:33.995 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:21:33.995 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 8166 2024-05-15T13:21:33.995 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.996 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:33.996 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.996 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:33.996 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.644490+0000", 2024-05-15T13:21:33.996 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.996 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.996 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.996 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:33.996 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.644490+0000", 2024-05-15T13:21:33.996 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:33.996 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.996 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.996 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:33.996 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.644491+0000", 2024-05-15T13:21:33.997 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4199999999999995e-07 2024-05-15T13:21:33.997 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.997 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.997 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:33.997 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.644518+0000", 2024-05-15T13:21:33.997 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7311e-05 2024-05-15T13:21:33.997 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.997 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.997 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:33.997 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.644526+0000", 2024-05-15T13:21:33.997 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6990000000000003e-06 2024-05-15T13:21:33.997 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.997 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.998 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:33.998 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.644532+0000", 2024-05-15T13:21:33.998 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9789999999999998e-06 2024-05-15T13:21:33.998 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.998 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.998 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.998 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.644571+0000", 2024-05-15T13:21:33.998 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9399e-05 2024-05-15T13:21:33.998 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.998 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.998 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:33.998 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.644578+0000", 2024-05-15T13:21:33.998 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7089999999999996e-06 2024-05-15T13:21:33.999 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.999 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.999 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:33.999 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.197391+0000", 2024-05-15T13:21:33.999 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.55281344099999996 2024-05-15T13:21:33.999 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.999 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.999 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:33.999 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.197496+0000", 2024-05-15T13:21:33.999 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010490899999999999 2024-05-15T13:21:33.999 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:33.999 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:33.999 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7,10", 2024-05-15T13:21:33.999 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.197653+0000", 2024-05-15T13:21:34.000 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015708000000000001 2024-05-15T13:21:34.000 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:34.000 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:34.000 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:34.000 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.199263+0000", 2024-05-15T13:21:34.000 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001609536 2024-05-15T13:21:34.000 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:34.000 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:34.000 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:34.000 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.199303+0000", 2024-05-15T13:21:34.000 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9706000000000002e-05 2024-05-15T13:21:34.000 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:34.000 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:34.001 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:34.001 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.199662+0000", 2024-05-15T13:21:34.001 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00035904900000000001 2024-05-15T13:21:34.001 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:34.001 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:34.001 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:34.001 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.199704+0000", 2024-05-15T13:21:34.001 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2292000000000001e-05 2024-05-15T13:21:34.001 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:34.001 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:34.001 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:34.001 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.199743+0000", 2024-05-15T13:21:34.001 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9405e-05 2024-05-15T13:21:34.002 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:34.002 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:34.002 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:34.002 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:34.002 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:34.002 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26299.0:8184 8.6 8:670b696b:::benchmark_data_smithi177_149673_object8183:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e762)", 2024-05-15T13:21:34.002 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:19:41.651262+0000", 2024-05-15T13:21:34.002 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 112.200803142, 2024-05-15T13:21:34.002 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.55514700900000002, 2024-05-15T13:21:34.002 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:21:34.002 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:21:34.002 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:21:34.003 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26299", 2024-05-15T13:21:34.003 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:21:34.003 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 8184 2024-05-15T13:21:34.003 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:34.003 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:21:34.003 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:34.003 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:21:34.003 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.651262+0000", 2024-05-15T13:21:34.003 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:34.003 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:34.003 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:34.003 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:21:34.003 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.651262+0000", 2024-05-15T13:21:34.003 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:21:34.004 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:34.004 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:34.004 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:21:34.004 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.651263+0000", 2024-05-15T13:21:34.004 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4200000000000004e-07 2024-05-15T13:21:34.004 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:34.004 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:34.004 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:21:34.004 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.651291+0000", 2024-05-15T13:21:34.004 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7903e-05 2024-05-15T13:21:34.004 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:34.004 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:34.004 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:21:34.004 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.651304+0000", 2024-05-15T13:21:34.005 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2853999999999999e-05 2024-05-15T13:21:34.005 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:34.005 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:34.005 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:21:34.005 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.651314+0000", 2024-05-15T13:21:34.005 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0032e-05 2024-05-15T13:21:34.005 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:34.005 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:34.005 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:34.005 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.651367+0000", 2024-05-15T13:21:34.005 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3264999999999999e-05 2024-05-15T13:21:34.005 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:34.005 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:34.006 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:21:34.006 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:41.651373+0000", 2024-05-15T13:21:34.006 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1580000000000002e-06 2024-05-15T13:21:34.006 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:34.006 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:34.006 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:21:34.006 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.198339+0000", 2024-05-15T13:21:34.006 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.54696618100000005 2024-05-15T13:21:34.006 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:34.006 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:34.006 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:21:34.006 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.198502+0000", 2024-05-15T13:21:34.006 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016266300000000001 2024-05-15T13:21:34.007 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:34.007 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:34.007 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7,10", 2024-05-15T13:21:34.007 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.198660+0000", 2024-05-15T13:21:34.007 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000157985 2024-05-15T13:21:34.007 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:34.007 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:34.007 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:21:34.007 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.199757+0000", 2024-05-15T13:21:34.007 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010974050000000001 2024-05-15T13:21:34.007 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:34.007 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:34.007 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:34.008 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.199805+0000", 2024-05-15T13:21:34.008 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7868999999999999e-05 2024-05-15T13:21:34.008 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:34.008 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:34.008 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:21:34.008 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.206306+0000", 2024-05-15T13:21:34.008 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0065009270000000001 2024-05-15T13:21:34.008 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:34.008 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:34.008 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:21:34.008 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.206363+0000", 2024-05-15T13:21:34.008 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6944999999999998e-05 2024-05-15T13:21:34.008 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:21:34.008 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:21:34.008 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:21:34.008 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:19:42.206409+0000", 2024-05-15T13:21:34.009 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5880000000000001e-05 2024-05-15T13:21:34.009 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:34.009 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:34.009 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:34.009 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:21:34.009 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:21:34.009 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:21:34.010 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:21:34.010 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:21:34.010 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:21:34.010 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:33 smithi177 ceph-mon[44303]: pgmap v2041: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 650 op/s; 4678/149640 objects misplaced (3.126%); 2.7 MiB/s, 43 objects/s recovering 2024-05-15T13:21:34.010 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:21:34.033 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:21:34.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:33 smithi120 ceph-mon[43961]: pgmap v2041: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 3.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 650 op/s; 4678/149640 objects misplaced (3.126%); 2.7 MiB/s, 43 objects/s recovering 2024-05-15T13:21:34.078 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:21:34.082 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:34.133 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:21:34.168 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:21:34.196 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:21:34.247 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:21:34.292 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:21:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:21:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:21:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:21:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:21:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:21:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:21:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:21:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:21:34.314 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:21:34.314 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:21:34.314 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:21:34.314 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:21:34.968 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:21:34.970 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:34 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2054799035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:21:34.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:34.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:21:35.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:34 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2054799035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:21:35.073 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:21:35.118 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:34 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2054799035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:21:35.131 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:35.454 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:21:35.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:35.459 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:21:35.495 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-05-15T13:21:35.561 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:21:35.629 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:35.754 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:35 smithi177 ceph-mon[44303]: 8.3 deep-scrub ok 2024-05-15T13:21:35.755 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:35 smithi177 ceph-mon[44303]: pgmap v2042: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 2.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 789 op/s; 3863/141900 objects misplaced (2.722%); 3.9 MiB/s, 62 objects/s recovering 2024-05-15T13:21:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:21:35.999 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:35 smithi027 ceph-mon[39795]: 8.3 deep-scrub ok 2024-05-15T13:21:35.999 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:35 smithi027 ceph-mon[39795]: pgmap v2042: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 2.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 789 op/s; 3863/141900 objects misplaced (2.722%); 3.9 MiB/s, 62 objects/s recovering 2024-05-15T13:21:36.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:36.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:21:36.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:35 smithi120 ceph-mon[43961]: 8.3 deep-scrub ok 2024-05-15T13:21:36.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:35 smithi120 ceph-mon[43961]: pgmap v2042: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 2.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 789 op/s; 3863/141900 objects misplaced (2.722%); 3.9 MiB/s, 62 objects/s recovering 2024-05-15T13:21:36.053 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.8 reset_purged_snaps_last 2024-05-15T13:21:36.102 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:21:36.164 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:21:36.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:36.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:21:36.501 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:21:36.553 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:36.586 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:21:36.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:36.595 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:21:36.696 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:21:36.771 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:36.951 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:21:36.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:36.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:21:37.058 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:21:37.108 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:37.146 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:21:37.152 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:21:37.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:37.153 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:21:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:21:37.254 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:21:37.281 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:37.302 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:21:37.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:37.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:21:37.405 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:21:37.447 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:37.486 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:21:37.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:37.491 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:21:37.592 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:21:37.620 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:37.642 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:21:37.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:37.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:21:37.746 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:21:37.790 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:37 smithi177 ceph-mon[44303]: pgmap v2043: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 2.8 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 626 op/s; 3863/139863 objects misplaced (2.762%); 2.7 MiB/s, 42 objects/s recovering 2024-05-15T13:21:37.790 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:37 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/577534253' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:21:37.802 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:21:37.815 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:37.852 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:21:37.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:37.858 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:21:37.959 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:21:37.998 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:38.026 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:21:38.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:37 smithi027 ceph-mon[39795]: pgmap v2043: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 2.8 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 626 op/s; 3863/139863 objects misplaced (2.762%); 2.7 MiB/s, 42 objects/s recovering 2024-05-15T13:21:38.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:37 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/577534253' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:21:38.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:38.032 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:21:38.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:37 smithi120 ceph-mon[43961]: pgmap v2043: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 35 active+clean; 2.8 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 626 op/s; 3863/139863 objects misplaced (2.762%); 2.7 MiB/s, 42 objects/s recovering 2024-05-15T13:21:38.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:37 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/577534253' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:21:38.132 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:21:38.183 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:38.215 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:21:38.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:38.222 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:21:38.323 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:21:38.371 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:38.408 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:21:38.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:38.421 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:21:38.522 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:21:38.578 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:38.808 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:21:38.876 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:38 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:21:38.876 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:38 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/577534253' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:21:38.876 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:38 smithi027 ceph-mon[39795]: osdmap e786: 12 total, 12 up, 11 in 2024-05-15T13:21:39.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:38 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:21:39.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:38 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/577534253' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:21:39.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:38 smithi120 ceph-mon[43961]: osdmap e786: 12 total, 12 up, 11 in 2024-05-15T13:21:39.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:38 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:21:39.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:38 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/577534253' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:21:39.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:38 smithi177 ceph-mon[44303]: osdmap e786: 12 total, 12 up, 11 in 2024-05-15T13:21:40.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:39 smithi177 ceph-mon[44303]: pgmap v2045: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 2.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 715 op/s; 3162/133221 objects misplaced (2.373%); 3.2 MiB/s, 50 objects/s recovering 2024-05-15T13:21:40.279 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:39 smithi027 ceph-mon[39795]: pgmap v2045: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 2.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 715 op/s; 3162/133221 objects misplaced (2.373%); 3.2 MiB/s, 50 objects/s recovering 2024-05-15T13:21:40.289 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:39 smithi120 ceph-mon[43961]: pgmap v2045: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 2.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 715 op/s; 3162/133221 objects misplaced (2.373%); 3.2 MiB/s, 50 objects/s recovering 2024-05-15T13:21:40.753 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:21:40.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:40.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:21:40.864 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:21:40.894 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:40.924 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:21:40.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:40.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:21:41.028 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:21:41.056 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:41.081 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:21:41.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:41.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:21:41.185 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:21:41.239 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:41.266 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:21:41.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:41.278 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:21:41.379 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:21:41.414 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:41.439 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:21:41.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:41.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:21:41.545 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:21:41.587 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:41.619 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:21:41.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:41.625 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:21:41.727 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:21:41.795 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:41.829 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:21:41.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:41.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:21:41.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:41 smithi177 ceph-mon[44303]: pgmap v2046: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 2.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 3162/128559 objects misplaced (2.460%); 3.2 MiB/s, 50 objects/s recovering 2024-05-15T13:21:41.938 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:21:41.987 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:42.028 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:21:42.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:42.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:21:42.137 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:21:42.184 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:42.224 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:21:42.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:42.235 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:21:42.279 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:41 smithi027 ceph-mon[39795]: pgmap v2046: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 2.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 3162/128559 objects misplaced (2.460%); 3.2 MiB/s, 50 objects/s recovering 2024-05-15T13:21:42.289 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:41 smithi120 ceph-mon[43961]: pgmap v2046: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 2.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 3162/128559 objects misplaced (2.460%); 3.2 MiB/s, 50 objects/s recovering 2024-05-15T13:21:42.336 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:21:42.383 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:42.413 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:21:42.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:42.416 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:21:42.517 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:21:42.560 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:42.593 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:21:42.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:42.599 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:21:42.700 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:21:42.754 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:42.789 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:21:42.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:42.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:21:42.854 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:42 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:21:42.854 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:42 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:21:42.854 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:42 smithi177 ceph-mon[44303]: osdmap e787: 12 total, 12 up, 11 in 2024-05-15T13:21:42.885 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:21:42.897 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:21:42.925 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:42.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:42.950 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:21:42.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:21:43.052 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:21:43.082 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:43.103 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:21:43.106 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:42 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:21:43.106 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:42 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:21:43.106 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:42 smithi120 ceph-mon[43961]: osdmap e787: 12 total, 12 up, 11 in 2024-05-15T13:21:43.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:43.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:21:43.208 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:21:43.236 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:43.258 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:21:43.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:43.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:21:43.279 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:42 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:21:43.279 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:42 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:21:43.279 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:42 smithi027 ceph-mon[39795]: osdmap e787: 12 total, 12 up, 11 in 2024-05-15T13:21:43.361 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:21:43.389 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:43.410 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:21:43.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:43.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:21:43.516 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:21:43.562 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:44.058 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:21:44.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:44.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:21:44.069 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:43 smithi027 ceph-mon[39795]: pgmap v2048: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 2.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 620 op/s; 3162/126999 objects misplaced (2.490%); 1.8 MiB/s, 29 objects/s recovering 2024-05-15T13:21:44.167 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:21:44.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:43 smithi177 ceph-mon[44303]: pgmap v2048: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 2.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 620 op/s; 3162/126999 objects misplaced (2.490%); 1.8 MiB/s, 29 objects/s recovering 2024-05-15T13:21:44.209 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:44.261 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:21:44.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:44.271 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:21:44.289 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:43 smithi120 ceph-mon[43961]: pgmap v2048: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 2.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 620 op/s; 3162/126999 objects misplaced (2.490%); 1.8 MiB/s, 29 objects/s recovering 2024-05-15T13:21:44.371 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:21:44.420 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:44.437 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:21:44.451 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:21:44.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:44.458 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:21:44.559 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:21:44.621 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:45.243 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:45 smithi027 ceph-mon[39795]: pgmap v2049: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 2.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 926 op/s; 2343/117609 objects misplaced (1.992%); 4.0 MiB/s, 63 objects/s recovering 2024-05-15T13:21:45.243 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:45 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:21:45.243 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:45 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:21:45.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:45 smithi120 ceph-mon[43961]: pgmap v2049: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 2.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 926 op/s; 2343/117609 objects misplaced (1.992%); 4.0 MiB/s, 63 objects/s recovering 2024-05-15T13:21:45.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:45 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:21:45.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:45 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:21:45.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:45 smithi177 ceph-mon[44303]: pgmap v2049: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 2.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 926 op/s; 2343/117609 objects misplaced (1.992%); 4.0 MiB/s, 63 objects/s recovering 2024-05-15T13:21:45.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:45 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:21:45.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:45 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:21:45.834 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:21:45.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:45.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:21:45.851 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.9 reset_purged_snaps_last 2024-05-15T13:21:45.939 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:21:45.969 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:45.994 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:21:46.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:46.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:21:46.102 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:21:46.157 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:46.283 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:21:46.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:46.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:21:46.388 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:21:46.416 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:46.440 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:21:46.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:46.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:21:46.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:46 smithi027 ceph-mon[39795]: osdmap e788: 12 total, 12 up, 11 in 2024-05-15T13:21:46.539 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:46 smithi120 ceph-mon[43961]: osdmap e788: 12 total, 12 up, 11 in 2024-05-15T13:21:46.544 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:21:46.577 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:46 smithi177 ceph-mon[44303]: osdmap e788: 12 total, 12 up, 11 in 2024-05-15T13:21:46.578 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:46.598 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:21:46.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:46.600 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:21:46.695 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:21:46.701 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:21:46.727 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:21:46.730 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:46.752 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:21:46.752 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:21:46.756 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:21:46.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:46.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:21:46.766 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:21:46.805 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:21:46.844 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:21:46.860 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:21:46.878 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:21:46.889 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:46.910 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:21:46.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:46.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:21:46.919 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:21:46.955 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:21:47.004 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:21:47.013 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:21:47.030 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:21:47.041 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:47.056 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:21:47.061 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:21:47.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:47.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:21:47.101 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:21:47.138 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:21:47.154 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:21:47.154 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:21:47.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:21:47.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:21:47.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:21:47.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:21:47.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:21:47.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:21:47.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:21:47.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:21:47.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:21:47.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:21:47.166 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:21:47.195 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:47.216 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:21:47.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:47.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:21:47.289 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:47 smithi120 ceph-mon[43961]: pgmap v2051: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 2.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 745 op/s; 2343/115314 objects misplaced (2.032%); 2.1 MiB/s, 34 objects/s recovering 2024-05-15T13:21:47.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:47 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4198485807' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:21:47.321 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:21:47.369 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:47.398 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:21:47.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:47.404 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:21:47.505 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:21:47.529 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:47 smithi027 ceph-mon[39795]: pgmap v2051: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 2.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 745 op/s; 2343/115314 objects misplaced (2.032%); 2.1 MiB/s, 34 objects/s recovering 2024-05-15T13:21:47.529 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:47 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4198485807' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:21:47.531 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:47 smithi177 ceph-mon[44303]: pgmap v2051: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 2.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 745 op/s; 2343/115314 objects misplaced (2.032%); 2.1 MiB/s, 34 objects/s recovering 2024-05-15T13:21:47.531 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:47 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4198485807' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:21:47.533 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:47.554 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:21:47.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:47.557 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:21:47.641 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-05-15T13:21:47.658 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:21:47.703 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:47.743 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:21:47.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:47.754 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:21:47.855 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:21:47.957 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:48.139 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:21:49.019 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:21:49.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:49.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:21:49.124 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:21:49.195 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:49.226 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:21:49.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:49.237 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:21:49.338 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:21:49.367 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:49.389 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:21:49.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:49.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:21:49.496 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:21:49.557 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:49.597 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:21:49.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:49.600 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:21:49.702 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:21:49.760 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:49.819 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:21:49.821 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:49 smithi027 ceph-mon[39795]: pgmap v2052: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 2.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 815 op/s; 1401/108978 objects misplaced (1.286%); 4.6 MiB/s, 73 objects/s recovering 2024-05-15T13:21:49.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:49.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:21:49.924 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:21:49.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:49 smithi177 ceph-mon[44303]: pgmap v2052: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 2.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 815 op/s; 1401/108978 objects misplaced (1.286%); 4.6 MiB/s, 73 objects/s recovering 2024-05-15T13:21:49.972 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:50.030 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:21:50.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:49 smithi120 ceph-mon[43961]: pgmap v2052: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 2.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 815 op/s; 1401/108978 objects misplaced (1.286%); 4.6 MiB/s, 73 objects/s recovering 2024-05-15T13:21:50.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:50.041 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:21:50.142 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:21:50.173 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:50.199 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:21:50.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:50.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:21:50.305 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:21:50.334 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:50.353 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:21:50.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:50.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:21:50.457 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:21:50.538 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:50.570 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:21:50.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:50.575 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:21:50.676 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:21:50.736 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:50.779 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:21:50.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:50.786 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:21:50.887 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:21:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:51.058 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:21:51.066 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:21:51.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:51.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:21:51.069 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:21:51.169 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:21:51.268 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:51.782 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:21:51.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:51 smithi027 ceph-mon[39795]: pgmap v2053: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 2.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 868 op/s; 1401/104766 objects misplaced (1.337%); 4.2 MiB/s, 66 objects/s recovering 2024-05-15T13:21:51.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:51 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2580027183' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:21:51.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:51 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:21:52.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:51 smithi120 ceph-mon[43961]: pgmap v2053: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 2.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 868 op/s; 1401/104766 objects misplaced (1.337%); 4.2 MiB/s, 66 objects/s recovering 2024-05-15T13:21:52.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:51 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2580027183' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:21:52.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:51 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:21:52.154 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:21:52.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:52.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:21:52.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:51 smithi177 ceph-mon[44303]: pgmap v2053: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 2.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 868 op/s; 1401/104766 objects misplaced (1.337%); 4.2 MiB/s, 66 objects/s recovering 2024-05-15T13:21:52.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:51 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2580027183' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:21:52.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:51 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:21:52.258 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:21:52.327 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:52.356 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:21:52.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:52.367 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:21:52.468 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:21:52.510 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:52.559 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:21:52.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:52.563 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:21:52.665 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:21:52.695 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:52.720 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:21:52.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:52.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:21:52.824 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:21:52.854 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:52.884 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:21:52.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:52.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:21:52.988 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:21:53.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:52 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:21:53.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:52 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:21:53.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:52 smithi027 ceph-mon[39795]: osdmap e789: 12 total, 12 up, 11 in 2024-05-15T13:21:53.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:52 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:21:53.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:52 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:21:53.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:52 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:21:53.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:52 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:21:53.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:52 smithi120 ceph-mon[43961]: osdmap e789: 12 total, 12 up, 11 in 2024-05-15T13:21:53.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:52 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:21:53.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:52 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:21:53.044 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:52 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:21:53.044 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:52 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:21:53.044 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:52 smithi177 ceph-mon[44303]: osdmap e789: 12 total, 12 up, 11 in 2024-05-15T13:21:53.044 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:52 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:21:53.044 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:52 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:21:53.047 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:53.089 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:21:53.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:53.092 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:21:53.193 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:21:53.261 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:53.545 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:21:53.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:53.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:21:53.654 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:21:53.710 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:53.765 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:21:53.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:53.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:21:53.874 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:21:53.929 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:53 smithi027 ceph-mon[39795]: pgmap v2055: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 2.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 596 op/s; 1401/103293 objects misplaced (1.356%); 2.5 MiB/s, 39 objects/s recovering 2024-05-15T13:21:53.929 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:53 smithi027 ceph-mon[39795]: osdmap e790: 12 total, 12 up, 11 in 2024-05-15T13:21:53.929 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:53 smithi027 ceph-mon[39795]: 8.5 scrub starts 2024-05-15T13:21:53.933 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:54.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:53 smithi120 ceph-mon[43961]: pgmap v2055: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 2.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 596 op/s; 1401/103293 objects misplaced (1.356%); 2.5 MiB/s, 39 objects/s recovering 2024-05-15T13:21:54.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:53 smithi120 ceph-mon[43961]: osdmap e790: 12 total, 12 up, 11 in 2024-05-15T13:21:54.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:53 smithi120 ceph-mon[43961]: 8.5 scrub starts 2024-05-15T13:21:54.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:53 smithi177 ceph-mon[44303]: pgmap v2055: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 2.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 596 op/s; 1401/103293 objects misplaced (1.356%); 2.5 MiB/s, 39 objects/s recovering 2024-05-15T13:21:54.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:53 smithi177 ceph-mon[44303]: osdmap e790: 12 total, 12 up, 11 in 2024-05-15T13:21:54.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:53 smithi177 ceph-mon[44303]: 8.5 scrub starts 2024-05-15T13:21:54.510 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.10 reset_purged_snaps_last 2024-05-15T13:21:54.596 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:21:54.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:54.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:21:54.707 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:21:54.758 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:54.764 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:54 smithi027 ceph-mon[39795]: 1.4 scrub starts 2024-05-15T13:21:54.764 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:54 smithi027 ceph-mon[39795]: 1.4 scrub ok 2024-05-15T13:21:54.765 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:54 smithi027 ceph-mon[39795]: 8.5 scrub ok 2024-05-15T13:21:54.765 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:21:54.790 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:21:54.790 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T13:21:54.790 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-05-15T13:21:54.790 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-05-15T13:21:54.790 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:21:54.791 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:21:54.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:54.793 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:21:54.894 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:21:54.940 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:55.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:54 smithi120 ceph-mon[43961]: 1.4 scrub starts 2024-05-15T13:21:55.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:54 smithi120 ceph-mon[43961]: 1.4 scrub ok 2024-05-15T13:21:55.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:54 smithi120 ceph-mon[43961]: 8.5 scrub ok 2024-05-15T13:21:55.082 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:21:55.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:55.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:21:55.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:54 smithi177 ceph-mon[44303]: 1.4 scrub starts 2024-05-15T13:21:55.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:54 smithi177 ceph-mon[44303]: 1.4 scrub ok 2024-05-15T13:21:55.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:54 smithi177 ceph-mon[44303]: 8.5 scrub ok 2024-05-15T13:21:55.189 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:21:55.266 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:55.295 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:21:55.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:55.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:21:55.403 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:21:55.431 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:55.452 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:21:55.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:55.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:21:55.556 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:21:55.585 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:55.591 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:21:55.790 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:21:55.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:55.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:21:55.894 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:21:55.920 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:55.938 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:21:55.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:55.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:21:56.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:55 smithi027 ceph-mon[39795]: pgmap v2057: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 877 op/s; 300/94260 objects misplaced (0.318%); 5.3 MiB/s, 85 objects/s recovering 2024-05-15T13:21:56.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:55 smithi027 ceph-mon[39795]: osdmap e791: 12 total, 12 up, 11 in 2024-05-15T13:21:56.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:55 smithi120 ceph-mon[43961]: pgmap v2057: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 877 op/s; 300/94260 objects misplaced (0.318%); 5.3 MiB/s, 85 objects/s recovering 2024-05-15T13:21:56.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:55 smithi120 ceph-mon[43961]: osdmap e791: 12 total, 12 up, 11 in 2024-05-15T13:21:56.043 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:21:56.068 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:55 smithi177 ceph-mon[44303]: pgmap v2057: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 877 op/s; 300/94260 objects misplaced (0.318%); 5.3 MiB/s, 85 objects/s recovering 2024-05-15T13:21:56.068 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:55 smithi177 ceph-mon[44303]: osdmap e791: 12 total, 12 up, 11 in 2024-05-15T13:21:56.070 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:56.089 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:21:56.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:56.093 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:21:56.194 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:21:56.220 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:56.239 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:21:56.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:56.242 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:21:56.346 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:21:56.399 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:56.668 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:21:56.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:56.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:21:56.717 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-05-15T13:21:56.779 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:21:56.842 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:56.887 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:21:56.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:56.894 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:21:56.961 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:56 smithi027 ceph-mon[39795]: 1.14 scrub starts 2024-05-15T13:21:56.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:56 smithi027 ceph-mon[39795]: 1.14 scrub ok 2024-05-15T13:21:56.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:56 smithi027 ceph-mon[39795]: 1.12 scrub starts 2024-05-15T13:21:56.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:56 smithi027 ceph-mon[39795]: 8.6 scrub starts 2024-05-15T13:21:56.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:56 smithi027 ceph-mon[39795]: 1.12 scrub ok 2024-05-15T13:21:56.962 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:56 smithi027 ceph-mon[39795]: osdmap e792: 12 total, 12 up, 11 in 2024-05-15T13:21:56.994 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:21:57.019 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:56 smithi120 ceph-mon[43961]: 1.14 scrub starts 2024-05-15T13:21:57.019 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:56 smithi120 ceph-mon[43961]: 1.14 scrub ok 2024-05-15T13:21:57.020 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:56 smithi120 ceph-mon[43961]: 1.12 scrub starts 2024-05-15T13:21:57.020 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:56 smithi120 ceph-mon[43961]: 8.6 scrub starts 2024-05-15T13:21:57.020 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:56 smithi120 ceph-mon[43961]: 1.12 scrub ok 2024-05-15T13:21:57.020 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:56 smithi120 ceph-mon[43961]: osdmap e792: 12 total, 12 up, 11 in 2024-05-15T13:21:57.022 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:57.044 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:21:57.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:57.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:21:57.148 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:21:57.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:56 smithi177 ceph-mon[44303]: 1.14 scrub starts 2024-05-15T13:21:57.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:56 smithi177 ceph-mon[44303]: 1.14 scrub ok 2024-05-15T13:21:57.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:56 smithi177 ceph-mon[44303]: 1.12 scrub starts 2024-05-15T13:21:57.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:56 smithi177 ceph-mon[44303]: 8.6 scrub starts 2024-05-15T13:21:57.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:56 smithi177 ceph-mon[44303]: 1.12 scrub ok 2024-05-15T13:21:57.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:56 smithi177 ceph-mon[44303]: osdmap e792: 12 total, 12 up, 11 in 2024-05-15T13:21:57.200 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:57.230 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:21:57.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:57.244 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:21:57.326 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:21:57.333 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:21:57.344 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:21:57.372 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:57.384 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:21:57.392 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:21:57.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:57.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:21:57.429 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:21:57.468 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:21:57.497 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:21:57.497 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:21:57.534 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:21:57.545 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:57.564 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:21:57.583 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:21:57.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:57.586 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:21:57.599 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:21:57.644 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:21:57.680 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:21:57.688 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:21:57.723 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:21:57.748 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:57.845 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:21:57.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:57.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:21:57.926 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:57 smithi027 ceph-mon[39795]: 8.6 scrub ok 2024-05-15T13:21:57.927 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:57 smithi027 ceph-mon[39795]: pgmap v2060: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 1.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 818 op/s; 300/92016 objects misplaced (0.326%); 4.7 MiB/s, 74 objects/s recovering 2024-05-15T13:21:57.927 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:57 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2892225770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:21:57.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:57 smithi177 ceph-mon[44303]: 8.6 scrub ok 2024-05-15T13:21:57.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:57 smithi177 ceph-mon[44303]: pgmap v2060: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 1.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 818 op/s; 300/92016 objects misplaced (0.326%); 4.7 MiB/s, 74 objects/s recovering 2024-05-15T13:21:57.943 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:57 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2892225770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:21:57.952 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:21:57.979 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:21:58.024 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:21:58.024 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:58.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:57 smithi120 ceph-mon[43961]: 8.6 scrub ok 2024-05-15T13:21:58.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:57 smithi120 ceph-mon[43961]: pgmap v2060: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 1.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 818 op/s; 300/92016 objects misplaced (0.326%); 4.7 MiB/s, 74 objects/s recovering 2024-05-15T13:21:58.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:57 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2892225770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:21:58.072 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:21:58.087 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:21:58.087 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:21:58.087 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:21:58.087 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:21:58.088 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:21:58.088 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:21:58.088 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:21:58.088 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:21:58.088 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:21:58.088 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:21:58.088 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:21:58.088 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:21:59.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:58 smithi027 ceph-mon[39795]: 8.d scrub starts 2024-05-15T13:21:59.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:58 smithi027 ceph-mon[39795]: 1.1 scrub starts 2024-05-15T13:21:59.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:58 smithi027 ceph-mon[39795]: 1.1 scrub ok 2024-05-15T13:21:59.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:58 smithi027 ceph-mon[39795]: 8.d scrub ok 2024-05-15T13:21:59.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:58 smithi027 ceph-mon[39795]: 1.8 scrub starts 2024-05-15T13:21:59.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:58 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:21:59.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:58 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:21:59.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:58 smithi027 ceph-mon[39795]: 1.8 scrub ok 2024-05-15T13:21:59.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:58 smithi027 ceph-mon[39795]: osdmap e793: 12 total, 12 up, 11 in 2024-05-15T13:21:59.039 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:58 smithi120 ceph-mon[43961]: 8.d scrub starts 2024-05-15T13:21:59.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:58 smithi120 ceph-mon[43961]: 1.1 scrub starts 2024-05-15T13:21:59.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:58 smithi120 ceph-mon[43961]: 1.1 scrub ok 2024-05-15T13:21:59.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:58 smithi120 ceph-mon[43961]: 8.d scrub ok 2024-05-15T13:21:59.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:58 smithi120 ceph-mon[43961]: 1.8 scrub starts 2024-05-15T13:21:59.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:58 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:21:59.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:58 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:21:59.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:58 smithi120 ceph-mon[43961]: 1.8 scrub ok 2024-05-15T13:21:59.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:58 smithi120 ceph-mon[43961]: osdmap e793: 12 total, 12 up, 11 in 2024-05-15T13:21:59.111 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:21:59.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:59.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:21:59.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:58 smithi177 ceph-mon[44303]: 8.d scrub starts 2024-05-15T13:21:59.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:58 smithi177 ceph-mon[44303]: 1.1 scrub starts 2024-05-15T13:21:59.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:58 smithi177 ceph-mon[44303]: 1.1 scrub ok 2024-05-15T13:21:59.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:58 smithi177 ceph-mon[44303]: 8.d scrub ok 2024-05-15T13:21:59.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:58 smithi177 ceph-mon[44303]: 1.8 scrub starts 2024-05-15T13:21:59.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:58 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:21:59.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:58 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:21:59.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:58 smithi177 ceph-mon[44303]: 1.8 scrub ok 2024-05-15T13:21:59.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:58 smithi177 ceph-mon[44303]: osdmap e793: 12 total, 12 up, 11 in 2024-05-15T13:21:59.225 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:21:59.269 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:59.757 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:21:59.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:59.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:21:59.872 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:21:59.900 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:21:59.921 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:21:59.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:21:59.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:22:00.025 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:22:00.073 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:59 smithi027 ceph-mon[39795]: 1.11 scrub starts 2024-05-15T13:22:00.073 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:59 smithi027 ceph-mon[39795]: 8.2 scrub starts 2024-05-15T13:22:00.074 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:59 smithi027 ceph-mon[39795]: 1.11 scrub ok 2024-05-15T13:22:00.074 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:59 smithi027 ceph-mon[39795]: 8.8 scrub starts 2024-05-15T13:22:00.074 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:59 smithi027 ceph-mon[39795]: pgmap v2062: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 1.8 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 799 op/s; 300/86958 objects misplaced (0.345%); 3.9 MiB/s, 62 objects/s recovering 2024-05-15T13:22:00.074 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:59 smithi027 ceph-mon[39795]: 8.2 scrub ok 2024-05-15T13:22:00.074 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:59 smithi027 ceph-mon[39795]: 8.8 scrub ok 2024-05-15T13:22:00.074 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:59 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:22:00.074 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:59 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:22:00.074 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:59 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:22:00.074 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:21:59 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:22:00.078 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:00.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:59 smithi177 ceph-mon[44303]: 1.11 scrub starts 2024-05-15T13:22:00.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:59 smithi177 ceph-mon[44303]: 8.2 scrub starts 2024-05-15T13:22:00.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:59 smithi177 ceph-mon[44303]: 1.11 scrub ok 2024-05-15T13:22:00.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:59 smithi177 ceph-mon[44303]: 8.8 scrub starts 2024-05-15T13:22:00.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:59 smithi177 ceph-mon[44303]: pgmap v2062: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 1.8 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 799 op/s; 300/86958 objects misplaced (0.345%); 3.9 MiB/s, 62 objects/s recovering 2024-05-15T13:22:00.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:59 smithi177 ceph-mon[44303]: 8.2 scrub ok 2024-05-15T13:22:00.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:59 smithi177 ceph-mon[44303]: 8.8 scrub ok 2024-05-15T13:22:00.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:59 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:22:00.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:59 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:22:00.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:59 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:22:00.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:21:59 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:22:00.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:59 smithi120 ceph-mon[43961]: 1.11 scrub starts 2024-05-15T13:22:00.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:59 smithi120 ceph-mon[43961]: 8.2 scrub starts 2024-05-15T13:22:00.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:59 smithi120 ceph-mon[43961]: 1.11 scrub ok 2024-05-15T13:22:00.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:59 smithi120 ceph-mon[43961]: 8.8 scrub starts 2024-05-15T13:22:00.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:59 smithi120 ceph-mon[43961]: pgmap v2062: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 1.8 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 799 op/s; 300/86958 objects misplaced (0.345%); 3.9 MiB/s, 62 objects/s recovering 2024-05-15T13:22:00.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:59 smithi120 ceph-mon[43961]: 8.2 scrub ok 2024-05-15T13:22:00.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:59 smithi120 ceph-mon[43961]: 8.8 scrub ok 2024-05-15T13:22:00.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:59 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:22:00.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:59 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:22:00.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:59 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:22:00.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:21:59 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:22:00.962 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:22:00.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:00.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:22:01.067 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:22:01.104 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:00 smithi027 ceph-mon[39795]: 8.e scrub starts 2024-05-15T13:22:01.104 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:00 smithi027 ceph-mon[39795]: 8.e scrub ok 2024-05-15T13:22:01.104 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:00 smithi027 ceph-mon[39795]: osdmap e794: 12 total, 12 up, 11 in 2024-05-15T13:22:01.106 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:01.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:00 smithi177 ceph-mon[44303]: 8.e scrub starts 2024-05-15T13:22:01.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:00 smithi177 ceph-mon[44303]: 8.e scrub ok 2024-05-15T13:22:01.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:00 smithi177 ceph-mon[44303]: osdmap e794: 12 total, 12 up, 11 in 2024-05-15T13:22:01.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:00 smithi120 ceph-mon[43961]: 8.e scrub starts 2024-05-15T13:22:01.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:00 smithi120 ceph-mon[43961]: 8.e scrub ok 2024-05-15T13:22:01.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:00 smithi120 ceph-mon[43961]: osdmap e794: 12 total, 12 up, 11 in 2024-05-15T13:22:01.961 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:22:01.983 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:22:01.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:01.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:22:02.088 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:22:02.112 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:01 smithi120 ceph-mon[43961]: 1.f scrub starts 2024-05-15T13:22:02.112 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:01 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:22:02.113 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:01 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:22:02.113 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:01 smithi120 ceph-mon[43961]: 1.f scrub ok 2024-05-15T13:22:02.113 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:01 smithi120 ceph-mon[43961]: pgmap v2064: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 1.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 738 op/s; 1.2 MiB/s, 18 objects/s recovering 2024-05-15T13:22:02.113 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:01 smithi120 ceph-mon[43961]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-05-15T13:22:02.113 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:01 smithi120 ceph-mon[43961]: osdmap e795: 12 total, 12 up, 11 in 2024-05-15T13:22:02.113 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:01 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3456302606' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:22:02.113 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:01 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:22:02.114 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:02.140 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:22:02.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:02.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:22:02.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:01 smithi177 ceph-mon[44303]: 1.f scrub starts 2024-05-15T13:22:02.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:01 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:22:02.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:01 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:22:02.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:01 smithi177 ceph-mon[44303]: 1.f scrub ok 2024-05-15T13:22:02.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:01 smithi177 ceph-mon[44303]: pgmap v2064: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 1.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 738 op/s; 1.2 MiB/s, 18 objects/s recovering 2024-05-15T13:22:02.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:01 smithi177 ceph-mon[44303]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-05-15T13:22:02.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:01 smithi177 ceph-mon[44303]: osdmap e795: 12 total, 12 up, 11 in 2024-05-15T13:22:02.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:01 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3456302606' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:22:02.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:01 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:22:02.250 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:22:02.279 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:01 smithi027 ceph-mon[39795]: 1.f scrub starts 2024-05-15T13:22:02.279 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:01 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:22:02.279 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:01 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:22:02.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:01 smithi027 ceph-mon[39795]: 1.f scrub ok 2024-05-15T13:22:02.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:01 smithi027 ceph-mon[39795]: pgmap v2064: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 1.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 738 op/s; 1.2 MiB/s, 18 objects/s recovering 2024-05-15T13:22:02.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:01 smithi027 ceph-mon[39795]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-05-15T13:22:02.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:01 smithi027 ceph-mon[39795]: osdmap e795: 12 total, 12 up, 11 in 2024-05-15T13:22:02.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:01 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3456302606' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:22:02.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:01 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:22:02.291 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:03.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:02 smithi177 ceph-mon[44303]: 1.7 scrub starts 2024-05-15T13:22:03.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:02 smithi177 ceph-mon[44303]: 1.7 scrub ok 2024-05-15T13:22:03.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:02 smithi177 ceph-mon[44303]: 8.b scrub starts 2024-05-15T13:22:03.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:02 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:22:03.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:02 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T13:22:03.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:02 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:22:03.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:02 smithi177 ceph-mon[44303]: osdmap e796: 12 total, 12 up, 11 in 2024-05-15T13:22:03.287 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:02 smithi027 ceph-mon[39795]: 1.7 scrub starts 2024-05-15T13:22:03.288 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:02 smithi027 ceph-mon[39795]: 1.7 scrub ok 2024-05-15T13:22:03.288 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:02 smithi027 ceph-mon[39795]: 8.b scrub starts 2024-05-15T13:22:03.288 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:02 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:22:03.288 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:02 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T13:22:03.288 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:02 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:22:03.288 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:02 smithi027 ceph-mon[39795]: osdmap e796: 12 total, 12 up, 11 in 2024-05-15T13:22:03.289 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:02 smithi120 ceph-mon[43961]: 1.7 scrub starts 2024-05-15T13:22:03.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:02 smithi120 ceph-mon[43961]: 1.7 scrub ok 2024-05-15T13:22:03.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:02 smithi120 ceph-mon[43961]: 8.b scrub starts 2024-05-15T13:22:03.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:02 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:22:03.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:02 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T13:22:03.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:02 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:22:03.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:02 smithi120 ceph-mon[43961]: osdmap e796: 12 total, 12 up, 11 in 2024-05-15T13:22:04.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:03 smithi177 ceph-mon[44303]: 8.b scrub ok 2024-05-15T13:22:04.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:03 smithi177 ceph-mon[44303]: 8.4 scrub starts 2024-05-15T13:22:04.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:03 smithi177 ceph-mon[44303]: pgmap v2067: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 1.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 463 op/s; 1.4 MiB/s, 22 objects/s recovering 2024-05-15T13:22:04.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:03 smithi177 ceph-mon[44303]: 8.4 scrub ok 2024-05-15T13:22:04.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:03 smithi027 ceph-mon[39795]: 8.b scrub ok 2024-05-15T13:22:04.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:03 smithi027 ceph-mon[39795]: 8.4 scrub starts 2024-05-15T13:22:04.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:03 smithi027 ceph-mon[39795]: pgmap v2067: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 1.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 463 op/s; 1.4 MiB/s, 22 objects/s recovering 2024-05-15T13:22:04.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:03 smithi027 ceph-mon[39795]: 8.4 scrub ok 2024-05-15T13:22:04.289 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:03 smithi120 ceph-mon[43961]: 8.b scrub ok 2024-05-15T13:22:04.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:03 smithi120 ceph-mon[43961]: 8.4 scrub starts 2024-05-15T13:22:04.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:03 smithi120 ceph-mon[43961]: pgmap v2067: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 1.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 463 op/s; 1.4 MiB/s, 22 objects/s recovering 2024-05-15T13:22:04.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:03 smithi120 ceph-mon[43961]: 8.4 scrub ok 2024-05-15T13:22:05.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:04 smithi177 ceph-mon[44303]: 1.a deep-scrub starts 2024-05-15T13:22:05.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:04 smithi177 ceph-mon[44303]: 1.a deep-scrub ok 2024-05-15T13:22:05.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:04 smithi177 ceph-mon[44303]: 1.10 scrub starts 2024-05-15T13:22:05.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:04 smithi177 ceph-mon[44303]: 1.10 scrub ok 2024-05-15T13:22:05.279 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:04 smithi027 ceph-mon[39795]: 1.a deep-scrub starts 2024-05-15T13:22:05.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:04 smithi027 ceph-mon[39795]: 1.a deep-scrub ok 2024-05-15T13:22:05.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:04 smithi027 ceph-mon[39795]: 1.10 scrub starts 2024-05-15T13:22:05.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:04 smithi027 ceph-mon[39795]: 1.10 scrub ok 2024-05-15T13:22:05.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:04 smithi120 ceph-mon[43961]: 1.a deep-scrub starts 2024-05-15T13:22:05.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:04 smithi120 ceph-mon[43961]: 1.a deep-scrub ok 2024-05-15T13:22:05.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:04 smithi120 ceph-mon[43961]: 1.10 scrub starts 2024-05-15T13:22:05.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:04 smithi120 ceph-mon[43961]: 1.10 scrub ok 2024-05-15T13:22:06.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:05 smithi027 ceph-mon[39795]: 8.a scrub starts 2024-05-15T13:22:06.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:05 smithi027 ceph-mon[39795]: 8.9 scrub starts 2024-05-15T13:22:06.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:05 smithi027 ceph-mon[39795]: pgmap v2068: 37 pgs: 37 active+clean; 1.5 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 850 op/s; 1.0 MiB/s, 16 objects/s recovering 2024-05-15T13:22:06.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:05 smithi027 ceph-mon[39795]: 8.9 scrub ok 2024-05-15T13:22:06.078 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:22:06.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:06.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:22:06.184 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:22:06.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:05 smithi177 ceph-mon[44303]: 8.a scrub starts 2024-05-15T13:22:06.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:05 smithi177 ceph-mon[44303]: 8.9 scrub starts 2024-05-15T13:22:06.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:05 smithi177 ceph-mon[44303]: pgmap v2068: 37 pgs: 37 active+clean; 1.5 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 850 op/s; 1.0 MiB/s, 16 objects/s recovering 2024-05-15T13:22:06.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:05 smithi177 ceph-mon[44303]: 8.9 scrub ok 2024-05-15T13:22:06.240 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:06.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:05 smithi120 ceph-mon[43961]: 8.a scrub starts 2024-05-15T13:22:06.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:05 smithi120 ceph-mon[43961]: 8.9 scrub starts 2024-05-15T13:22:06.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:05 smithi120 ceph-mon[43961]: pgmap v2068: 37 pgs: 37 active+clean; 1.5 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 850 op/s; 1.0 MiB/s, 16 objects/s recovering 2024-05-15T13:22:06.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:05 smithi120 ceph-mon[43961]: 8.9 scrub ok 2024-05-15T13:22:06.300 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:22:06.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:06.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:22:06.408 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:22:06.467 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:06.631 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:22:06.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:06.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:22:06.739 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:22:06.767 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:06.793 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:22:06.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:06.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:22:06.899 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:22:06.951 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:06 smithi027 ceph-mon[39795]: 8.a scrub ok 2024-05-15T13:22:06.951 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:06 smithi027 ceph-mon[39795]: 8.c scrub starts 2024-05-15T13:22:06.951 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:06 smithi027 ceph-mon[39795]: 8.c scrub ok 2024-05-15T13:22:06.954 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:07.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:06 smithi120 ceph-mon[43961]: 8.a scrub ok 2024-05-15T13:22:07.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:06 smithi120 ceph-mon[43961]: 8.c scrub starts 2024-05-15T13:22:07.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:06 smithi120 ceph-mon[43961]: 8.c scrub ok 2024-05-15T13:22:07.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:06 smithi177 ceph-mon[44303]: 8.a scrub ok 2024-05-15T13:22:07.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:06 smithi177 ceph-mon[44303]: 8.c scrub starts 2024-05-15T13:22:07.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:06 smithi177 ceph-mon[44303]: 8.c scrub ok 2024-05-15T13:22:08.034 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:22:08.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:08.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:22:08.145 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:22:08.173 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:07 smithi120 ceph-mon[43961]: pgmap v2069: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 1.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 684 op/s 2024-05-15T13:22:08.173 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:07 smithi120 ceph-mon[43961]: 8.0 scrub starts 2024-05-15T13:22:08.175 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:08.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:07 smithi177 ceph-mon[44303]: pgmap v2069: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 1.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 684 op/s 2024-05-15T13:22:08.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:07 smithi177 ceph-mon[44303]: 8.0 scrub starts 2024-05-15T13:22:08.195 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:22:08.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:08.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:22:08.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:07 smithi027 ceph-mon[39795]: pgmap v2069: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 1.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 684 op/s 2024-05-15T13:22:08.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:07 smithi027 ceph-mon[39795]: 8.0 scrub starts 2024-05-15T13:22:08.303 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:22:08.331 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:08.350 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:22:08.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:08.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:22:08.454 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:22:08.512 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:08.565 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:22:08.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:08.572 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:22:08.673 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:22:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:08.866 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:08 smithi177 ceph-mon[44303]: 8.0 scrub ok 2024-05-15T13:22:09.019 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:08 smithi027 ceph-mon[39795]: 8.0 scrub ok 2024-05-15T13:22:09.096 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:22:09.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:09.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:22:09.193 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:22:09.193 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:22:09.202 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:22:09.246 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:09.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:08 smithi120 ceph-mon[43961]: 8.0 scrub ok 2024-05-15T13:22:09.299 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:22:09.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:09.303 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:22:09.404 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:22:09.444 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:09.472 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:22:09.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:09.478 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:22:09.579 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:22:09.607 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:09.637 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:22:09.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:09.646 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:22:09.747 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:22:09.817 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:09.867 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:09 smithi027 ceph-mon[39795]: pgmap v2070: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 1.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 786 op/s 2024-05-15T13:22:09.871 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:09 smithi177 ceph-mon[44303]: pgmap v2070: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 1.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 786 op/s 2024-05-15T13:22:09.879 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:22:09.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:09.883 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:22:09.972 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-05-15T13:22:09.983 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:22:10.020 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:10.061 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:22:10.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:10.070 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:22:10.079 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:22:10.171 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:22:10.200 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:09 smithi120 ceph-mon[43961]: pgmap v2070: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 1.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 786 op/s 2024-05-15T13:22:10.203 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:10.225 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:22:10.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:10.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:22:10.333 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:22:10.374 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:10.417 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:22:10.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:10.423 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:22:10.430 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:22:10.523 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:22:10.586 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:10.619 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:22:10.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:10.628 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:22:10.705 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:22:10.728 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:22:10.762 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:22:10.766 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:10.792 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:22:10.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:10.798 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:22:10.899 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:22:10.982 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:11.016 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:22:11.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:11.020 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:22:11.122 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:22:11.148 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:11.168 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:22:11.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:11.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:22:11.271 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:22:11.322 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:12.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:11 smithi027 ceph-mon[39795]: 1.6 scrub starts 2024-05-15T13:22:12.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:11 smithi027 ceph-mon[39795]: 1.6 scrub ok 2024-05-15T13:22:12.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:11 smithi027 ceph-mon[39795]: pgmap v2071: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 1.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 791 op/s 2024-05-15T13:22:12.095 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.11 reset_purged_snaps_last 2024-05-15T13:22:12.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:11 smithi177 ceph-mon[44303]: 1.6 scrub starts 2024-05-15T13:22:12.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:11 smithi177 ceph-mon[44303]: 1.6 scrub ok 2024-05-15T13:22:12.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:11 smithi177 ceph-mon[44303]: pgmap v2071: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 1.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 791 op/s 2024-05-15T13:22:12.254 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:22:12.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:12.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:22:12.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:11 smithi120 ceph-mon[43961]: 1.6 scrub starts 2024-05-15T13:22:12.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:11 smithi120 ceph-mon[43961]: 1.6 scrub ok 2024-05-15T13:22:12.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:11 smithi120 ceph-mon[43961]: pgmap v2071: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 1.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 791 op/s 2024-05-15T13:22:12.360 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:22:12.425 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:12.461 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:22:12.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:12.474 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:22:12.575 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:22:12.609 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:12.672 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:22:12.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:12.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:22:12.779 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:22:12.832 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:12.873 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:22:12.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:12.880 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:22:12.981 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:22:13.029 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:13.058 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:22:13.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:13.067 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:22:13.168 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:22:13.203 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:13.226 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:22:13.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:13.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:22:13.331 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:22:13.388 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:13.428 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:22:13.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:13.433 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:22:13.534 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:22:13.588 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:13.623 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:22:13.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:13.635 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:22:13.736 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:22:13.766 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:13.788 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:22:13.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:13.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:22:13.893 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:22:13.921 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:13.941 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:22:13.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:13.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:22:14.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:13 smithi120 ceph-mon[43961]: pgmap v2072: 37 pgs: 37 active+clean; 1.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 761 op/s 2024-05-15T13:22:14.045 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:22:14.117 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:14.140 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:22:14.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:14.143 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:22:14.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:13 smithi177 ceph-mon[44303]: pgmap v2072: 37 pgs: 37 active+clean; 1.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 761 op/s 2024-05-15T13:22:14.244 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:22:14.274 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:13 smithi027 ceph-mon[39795]: pgmap v2072: 37 pgs: 37 active+clean; 1.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 761 op/s 2024-05-15T13:22:14.275 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:14.295 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:22:14.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:14.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:22:14.397 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:22:14.397 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:22:14.402 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:22:14.429 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:14.452 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:22:14.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:14.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:22:14.556 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:22:14.618 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:14.910 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:14 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:22:15.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:14 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:22:15.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:14 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:22:16.093 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:22:16.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:16.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:22:16.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:15 smithi177 ceph-mon[44303]: pgmap v2073: 37 pgs: 37 active+clean; 1004 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 907 op/s 2024-05-15T13:22:16.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:15 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:22:16.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:15 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:22:16.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:15 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:22:16.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:15 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:22:16.202 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:22:16.228 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:15 smithi120 ceph-mon[43961]: pgmap v2073: 37 pgs: 37 active+clean; 1004 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 907 op/s 2024-05-15T13:22:16.228 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:15 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:22:16.228 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:15 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:22:16.228 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:15 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:22:16.228 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:15 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:22:16.230 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:16.253 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:22:16.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:16.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:22:16.279 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:15 smithi027 ceph-mon[39795]: pgmap v2073: 37 pgs: 37 active+clean; 1004 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 907 op/s 2024-05-15T13:22:16.279 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:15 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:22:16.279 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:15 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:22:16.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:15 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:22:16.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:15 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:22:16.358 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:22:16.388 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:16.410 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:22:16.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:16.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:22:16.434 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:22:16.476 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:22:16.509 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:22:16.514 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:22:16.544 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:22:16.572 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:16.581 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:22:16.610 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:22:16.615 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:22:16.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:16.617 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:22:16.654 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:22:16.681 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:22:16.717 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:22:16.740 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:22:16.745 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:16.767 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:22:16.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:16.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:22:16.780 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:22:16.836 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:22:16.848 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:22:16.871 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:22:16.872 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:22:16.909 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:22:16.910 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:22:16.910 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:22:16.910 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:22:16.910 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:22:16.910 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:22:16.910 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:22:16.910 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:22:16.910 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:22:16.910 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:22:16.910 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:22:16.910 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:22:16.911 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-05-15T13:22:16.912 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:16.942 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:22:16.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:16.945 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:22:16.963 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:22:17.046 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:22:17.074 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:17.094 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:22:17.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:17.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:22:17.198 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:22:17.225 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:17.247 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:22:17.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:17.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:22:17.353 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:22:17.379 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:17 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:22:17.379 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:17 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:22:17.379 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:17 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1900201674' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:22:17.379 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:17 smithi120 ceph-mon[43961]: pgmap v2074: 37 pgs: 37 active+clean; 960 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 710 op/s 2024-05-15T13:22:17.379 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:17 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:22:17.379 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:17 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:22:17.379 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:17 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:22:17.381 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:17.402 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:22:17.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:17.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:22:17.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:17 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:22:17.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:17 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:22:17.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:17 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1900201674' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:22:17.478 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:17 smithi027 ceph-mon[39795]: pgmap v2074: 37 pgs: 37 active+clean; 960 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 710 op/s 2024-05-15T13:22:17.479 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:17 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:22:17.479 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:17 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:22:17.479 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:17 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:22:17.507 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:22:17.565 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:17.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:17 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:22:17.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:17 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:22:17.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:17 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1900201674' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:22:17.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:17 smithi177 ceph-mon[44303]: pgmap v2074: 37 pgs: 37 active+clean; 960 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 710 op/s 2024-05-15T13:22:17.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:17 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:22:17.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:17 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:22:17.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:17 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:22:18.431 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:22:18.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:18.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:22:18.540 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:22:18.590 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:18 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T13:22:18.590 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:18 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T13:22:18.592 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:18.635 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:22:18.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:18.639 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:22:18.740 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:22:18.765 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:18 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T13:22:18.765 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:18 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T13:22:18.767 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:18.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:18 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T13:22:18.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:18 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T13:22:18.790 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:22:18.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:18.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:22:18.894 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:22:18.922 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:18.944 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:22:18.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:18.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:22:19.048 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:22:19.106 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:19.154 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:22:19.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:19.164 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:22:19.265 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:22:19.319 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:19.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:19 smithi177 ceph-mon[44303]: 8.1 scrub starts 2024-05-15T13:22:19.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:19 smithi177 ceph-mon[44303]: 8.1 scrub ok 2024-05-15T13:22:19.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:19 smithi177 ceph-mon[44303]: pgmap v2075: 37 pgs: 37 active+clean; 845 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 804 op/s 2024-05-15T13:22:19.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:19 smithi027 ceph-mon[39795]: 8.1 scrub starts 2024-05-15T13:22:19.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:19 smithi027 ceph-mon[39795]: 8.1 scrub ok 2024-05-15T13:22:19.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:19 smithi027 ceph-mon[39795]: pgmap v2075: 37 pgs: 37 active+clean; 845 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 804 op/s 2024-05-15T13:22:19.790 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:19 smithi120 ceph-mon[43961]: 8.1 scrub starts 2024-05-15T13:22:19.790 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:19 smithi120 ceph-mon[43961]: 8.1 scrub ok 2024-05-15T13:22:19.790 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:19 smithi120 ceph-mon[43961]: pgmap v2075: 37 pgs: 37 active+clean; 845 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 804 op/s 2024-05-15T13:22:20.786 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:22:20.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:20.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:22:20.897 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:22:20.925 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:20.951 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:22:20.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:20.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:22:21.056 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:22:21.121 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:21.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:21 smithi027 ceph-mon[39795]: pgmap v2076: 37 pgs: 37 active+clean; 743 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 770 op/s 2024-05-15T13:22:22.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:21 smithi120 ceph-mon[43961]: pgmap v2076: 37 pgs: 37 active+clean; 743 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 770 op/s 2024-05-15T13:22:22.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:21 smithi177 ceph-mon[44303]: pgmap v2076: 37 pgs: 37 active+clean; 743 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 770 op/s 2024-05-15T13:22:22.306 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:22:22.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:22.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:22:22.428 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:22:22.466 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:22.737 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:22 smithi027 ceph-mon[39795]: 1.13 scrub starts 2024-05-15T13:22:22.737 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:22 smithi027 ceph-mon[39795]: 1.13 scrub ok 2024-05-15T13:22:22.737 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:22 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2698764978' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:22:22.737 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:22 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:22:22.901 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-05-15T13:22:22.998 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:22:23.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:23.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:22:23.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:22 smithi120 ceph-mon[43961]: 1.13 scrub starts 2024-05-15T13:22:23.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:22 smithi120 ceph-mon[43961]: 1.13 scrub ok 2024-05-15T13:22:23.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:22 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2698764978' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:22:23.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:22 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:22:23.108 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:22:23.129 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:22:23.177 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:22 smithi177 ceph-mon[44303]: 1.13 scrub starts 2024-05-15T13:22:23.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:22 smithi177 ceph-mon[44303]: 1.13 scrub ok 2024-05-15T13:22:23.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:22 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2698764978' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:22:23.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:22 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:22:23.183 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:23.228 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:22:23.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:23.234 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:22:23.335 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:22:23.363 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:23.384 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:22:23.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:23.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:22:23.488 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:22:23.542 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:22:23.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:23.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:22:23.805 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:22:23.838 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:22:23.871 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:22:23.902 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:24.029 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:23 smithi027 ceph-mon[39795]: pgmap v2077: 37 pgs: 37 active+clean; 714 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 668 op/s 2024-05-15T13:22:24.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:23 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:22:24.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:23 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:22:24.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:23 smithi027 ceph-mon[39795]: osdmap e797: 12 total, 12 up, 11 in 2024-05-15T13:22:24.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:23 smithi120 ceph-mon[43961]: pgmap v2077: 37 pgs: 37 active+clean; 714 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 668 op/s 2024-05-15T13:22:24.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:23 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:22:24.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:23 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:22:24.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:23 smithi120 ceph-mon[43961]: osdmap e797: 12 total, 12 up, 11 in 2024-05-15T13:22:24.109 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:22:24.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:24.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:22:24.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:23 smithi177 ceph-mon[44303]: pgmap v2077: 37 pgs: 37 active+clean; 714 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 668 op/s 2024-05-15T13:22:24.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:23 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:22:24.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:23 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:22:24.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:23 smithi177 ceph-mon[44303]: osdmap e797: 12 total, 12 up, 11 in 2024-05-15T13:22:24.220 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:22:24.247 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:24.267 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:22:24.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:24.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:22:24.371 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:22:24.399 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:24.422 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:22:24.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:24.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:22:24.528 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:22:24.556 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:24.576 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:22:24.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:24.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:22:24.681 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:22:24.736 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:24.802 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:22:24.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:24.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:22:24.910 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:22:24.959 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:24.992 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:22:24.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:24.999 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:22:25.048 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:22:25.100 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:22:25.147 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:25.176 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:22:25.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:25.181 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:22:25.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:25 smithi177 ceph-mon[44303]: pgmap v2079: 37 pgs: 37 active+clean; 531 MiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 756 op/s 2024-05-15T13:22:25.282 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:22:25.328 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:25.359 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:22:25.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:25.367 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:22:25.392 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:25 smithi027 ceph-mon[39795]: pgmap v2079: 37 pgs: 37 active+clean; 531 MiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 756 op/s 2024-05-15T13:22:25.467 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:22:25.493 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:25 smithi120 ceph-mon[43961]: pgmap v2079: 37 pgs: 37 active+clean; 531 MiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 756 op/s 2024-05-15T13:22:25.495 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:25.517 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:22:25.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:25.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:22:25.621 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:22:25.667 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:25.706 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:22:25.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:25.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:22:25.792 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:22:25.811 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:22:25.862 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:25.909 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:22:25.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:25.917 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:22:26.018 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:22:26.045 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:26.066 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:22:26.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:26.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:22:26.171 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:22:26.199 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:26.218 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:22:26.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:26.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:22:26.322 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:22:26.381 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:26.431 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:22:26.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:26.439 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:22:26.540 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:22:26.581 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:26.601 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:22:26.622 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:22:26.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:26.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:22:26.737 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:22:26.791 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:26.832 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:22:26.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:26.839 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:22:26.940 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:22:26.969 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:26.990 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:22:26.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:26.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:22:27.094 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:22:27.131 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:27.156 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:22:27.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:27.164 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:22:27.263 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:27 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:22:27.264 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:27 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:22:27.264 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:27 smithi120 ceph-mon[43961]: osdmap e798: 12 total, 12 up, 11 in 2024-05-15T13:22:27.264 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:27 smithi120 ceph-mon[43961]: pgmap v2081: 37 pgs: 37 active+clean; 489 MiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 712 op/s 2024-05-15T13:22:27.264 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:22:27.292 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:27.316 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:22:27.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:27.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:22:27.423 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:22:27.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:27 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:22:27.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:27 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:22:27.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:27 smithi177 ceph-mon[44303]: osdmap e798: 12 total, 12 up, 11 in 2024-05-15T13:22:27.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:27 smithi177 ceph-mon[44303]: pgmap v2081: 37 pgs: 37 active+clean; 489 MiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 712 op/s 2024-05-15T13:22:27.476 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:27.515 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:22:27.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:27.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:22:27.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:27 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:22:27.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:27 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:22:27.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:27 smithi027 ceph-mon[39795]: osdmap e798: 12 total, 12 up, 11 in 2024-05-15T13:22:27.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:27 smithi027 ceph-mon[39795]: pgmap v2081: 37 pgs: 37 active+clean; 489 MiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 712 op/s 2024-05-15T13:22:27.623 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:22:27.651 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:27.672 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:22:27.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:27.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:22:27.777 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:22:27.834 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:28.675 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:22:28.676 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T13:22:28.676 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-05-15T13:22:28.676 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-05-15T13:22:28.676 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:22:30.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:29 smithi027 ceph-mon[39795]: pgmap v2082: 37 pgs: 37 active+clean; 368 MiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 750 op/s 2024-05-15T13:22:30.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:29 smithi120 ceph-mon[43961]: pgmap v2082: 37 pgs: 37 active+clean; 368 MiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 750 op/s 2024-05-15T13:22:30.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:29 smithi177 ceph-mon[44303]: pgmap v2082: 37 pgs: 37 active+clean; 368 MiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 750 op/s 2024-05-15T13:22:30.207 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:22:30.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:30.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:22:30.313 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:22:30.342 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:30.369 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:22:30.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:30.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:22:30.473 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:22:30.539 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:30.578 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:22:30.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:30.584 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:22:30.685 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:22:30.714 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:30.735 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:22:30.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:30.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:22:30.840 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:22:30.881 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:30 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:22:30.881 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:30 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:22:30.881 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:30 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:22:30.881 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:30 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:22:30.881 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:30 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:22:30.881 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:30 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:22:30.882 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:30 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:22:30.884 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:30.936 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:22:30.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:30.942 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:22:31.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:30 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:22:31.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:30 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:22:31.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:30 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:22:31.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:30 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:22:31.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:30 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:22:31.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:30 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:22:31.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:30 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:22:31.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:30 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:22:31.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:30 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:22:31.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:30 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:22:31.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:30 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:22:31.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:30 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:22:31.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:30 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:22:31.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:30 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:22:31.043 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:22:31.085 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:31.272 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:22:31.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:31.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:22:31.381 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:22:31.432 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:31.478 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:22:31.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:31.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:22:31.608 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:22:31.635 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:31.657 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:22:31.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:31.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:22:31.761 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:22:31.841 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:31.874 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:22:31.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:31.883 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:22:31.944 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-05-15T13:22:31.983 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:22:31.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:31 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:22:31.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:31 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:22:31.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:31 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:22:31.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:31 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:22:31.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:31 smithi027 ceph-mon[39795]: osdmap e799: 12 total, 12 up, 11 in 2024-05-15T13:22:32.039 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:32.039 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:31 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:22:32.039 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:31 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:22:32.039 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:31 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:22:32.039 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:31 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:22:32.039 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:31 smithi177 ceph-mon[44303]: osdmap e799: 12 total, 12 up, 11 in 2024-05-15T13:22:32.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:31 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:22:32.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:31 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:22:32.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:31 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:22:32.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:31 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:22:32.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:31 smithi120 ceph-mon[43961]: osdmap e799: 12 total, 12 up, 11 in 2024-05-15T13:22:32.065 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:22:32.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:32.068 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:22:32.169 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:22:32.237 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:32.260 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:22:32.318 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:22:32.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:32.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:22:32.429 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:22:32.457 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:32.479 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:22:32.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:32.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:22:32.584 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:22:32.626 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:22:32.630 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:32.662 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:22:32.668 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:22:32.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:32.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:22:32.695 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:22:32.726 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:22:32.759 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:22:32.772 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:22:32.793 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:22:32.818 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:32.823 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:22:32.843 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:22:32.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:32.852 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:22:32.854 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:22:32.903 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:22:32.944 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:22:32.953 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:22:32.978 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:22:32.980 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:32.997 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:32 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:22:32.997 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:32 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:22:32.997 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:32 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3781049679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:22:33.001 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:22:33.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:33.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:22:33.005 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:32 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:22:33.005 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:32 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:22:33.005 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:32 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3781049679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:22:33.024 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:22:33.028 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:22:33.028 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:22:33.028 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:22:33.028 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:22:33.028 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:22:33.028 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:22:33.029 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:22:33.029 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:22:33.029 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:22:33.029 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:22:33.029 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:22:33.029 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:22:33.106 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:22:33.133 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:33.154 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:22:33.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:33.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:22:33.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:32 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:22:33.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:32 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:22:33.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:32 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3781049679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:22:33.257 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:22:33.305 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:33.507 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:22:33.514 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:22:33.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:33.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:22:33.625 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:22:33.681 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:33.711 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:22:33.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:33.724 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:22:33.746 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Removed 90059 objects 2024-05-15T13:22:33.746 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Clean up completed and total clean up time :114.716 2024-05-15T13:22:33.825 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:22:33.851 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:33.869 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:22:33.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:33.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:22:33.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:33 smithi177 ceph-mon[44303]: pgmap v2085: 37 pgs: 37 active+clean; 235 MiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 592 op/s 2024-05-15T13:22:33.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:33 smithi177 ceph-mon[44303]: osdmap e800: 12 total, 12 up, 11 in 2024-05-15T13:22:33.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:33 smithi027 ceph-mon[39795]: pgmap v2085: 37 pgs: 37 active+clean; 235 MiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 592 op/s 2024-05-15T13:22:33.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:33 smithi027 ceph-mon[39795]: osdmap e800: 12 total, 12 up, 11 in 2024-05-15T13:22:33.973 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:22:33.999 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:34.019 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:22:34.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:34.022 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:22:34.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:33 smithi120 ceph-mon[43961]: pgmap v2085: 37 pgs: 37 active+clean; 235 MiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 592 op/s 2024-05-15T13:22:34.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:33 smithi120 ceph-mon[43961]: osdmap e800: 12 total, 12 up, 11 in 2024-05-15T13:22:34.122 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:22:34.149 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:34.168 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:22:34.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:34.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:22:34.272 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:22:34.298 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:34.317 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:22:34.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:34.320 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:22:34.421 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:22:34.448 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:34.467 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:22:34.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:34.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:22:34.571 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:22:34.598 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:34.617 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:22:34.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:34.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:22:34.721 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:22:34.749 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:34.769 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:22:34.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:34.772 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:22:34.873 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:22:34.901 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:34.921 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:22:34.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:34.924 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:22:35.024 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:22:35.054 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:35.127 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:22:35.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:35.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:22:35.231 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:22:35.260 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:35.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:35 smithi027 ceph-mon[39795]: pgmap v2087: 37 pgs: 37 active+clean; 57 MiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 862 op/s 2024-05-15T13:22:35.281 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:22:35.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:35.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:22:35.385 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:22:35.414 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:35.453 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:22:35.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:35.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:22:35.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:35 smithi120 ceph-mon[43961]: pgmap v2087: 37 pgs: 37 active+clean; 57 MiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 862 op/s 2024-05-15T13:22:35.557 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:22:35.587 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:35 smithi177 ceph-mon[44303]: pgmap v2087: 37 pgs: 37 active+clean; 57 MiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 862 op/s 2024-05-15T13:22:35.590 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:35.611 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:22:35.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:35.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:22:35.716 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:22:35.747 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:35.767 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:22:35.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:35.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:22:35.871 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:22:35.900 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:35.919 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:22:35.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:35.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:22:36.023 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:22:36.051 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:36.150 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:22:36.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:36.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:22:36.238 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:22:36.238 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:22:36.254 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:22:36.284 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:36.296 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 8, 9, 10, 11, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:22:36.296 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:22:36.297 INFO:tasks.thrashosds.thrasher:Removing osd 8, in_osds are: [0, 1, 2, 3, 5, 6, 8, 9, 10, 11, 7] 2024-05-15T13:22:36.297 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd out 8 2024-05-15T13:22:36.326 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_6 2024-05-15T13:22:36.326 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool rm unique_pool_6 unique_pool_6 --yes-i-really-really-mean-it 2024-05-15T13:22:36.521 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:22:36.738 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:22:36.739 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:36 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3983382855' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:22:36.739 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:36 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:22:36.740 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:36 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3983382855' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:22:36.740 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:36 smithi027 ceph-mon[39795]: osdmap e801: 12 total, 12 up, 11 in 2024-05-15T13:22:36.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:36.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:22:36.842 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:22:36.870 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:36.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:36 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3983382855' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:22:36.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:36 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:22:36.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:36 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3983382855' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:22:36.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:36 smithi177 ceph-mon[44303]: osdmap e801: 12 total, 12 up, 11 in 2024-05-15T13:22:37.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:36 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3983382855' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:22:37.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:36 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:22:37.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:36 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3983382855' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:22:37.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:36 smithi120 ceph-mon[43961]: osdmap e801: 12 total, 12 up, 11 in 2024-05-15T13:22:37.119 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:22:37.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:37.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:22:37.139 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-05-15T13:22:37.151 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:22:37.224 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:22:37.256 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:37.277 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:22:37.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:37.281 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:22:37.382 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:22:37.413 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:37.434 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:22:37.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:37.437 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:22:37.538 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:22:37.567 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:37.587 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:22:37.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:37.590 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:22:37.660 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:22:37.691 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:22:37.718 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:37.738 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:22:37.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:37.741 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:22:37.842 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:22:37.872 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:37 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:22:37.872 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:37 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:22:37.872 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:37 smithi120 ceph-mon[43961]: pgmap v2089: 37 pgs: 37 active+clean; 26 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 644 op/s 2024-05-15T13:22:37.874 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:37.895 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:22:37.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:37.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:22:37.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:37 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:22:37.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:37 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:22:37.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:37 smithi027 ceph-mon[39795]: pgmap v2089: 37 pgs: 37 active+clean; 26 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 644 op/s 2024-05-15T13:22:37.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:37 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:22:37.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:37 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:22:37.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:37 smithi177 ceph-mon[44303]: pgmap v2089: 37 pgs: 37 active+clean; 26 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 644 op/s 2024-05-15T13:22:37.999 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:22:38.030 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:38.051 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:22:38.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:38.054 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:22:38.155 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:22:38.182 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:38.202 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:22:38.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:38.205 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:22:38.306 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:22:38.333 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:38.351 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:22:38.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:38.354 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:22:38.407 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:22:38.408 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:22:38.408 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:22:38.455 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:22:38.488 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:38.509 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:22:38.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:38.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:22:38.614 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:22:38.643 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:38.664 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:22:38.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:38.667 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:22:38.768 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:22:38.801 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:38.822 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:22:38.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:38.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:22:38.926 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:22:38.956 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:39.662 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:22:39.966 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:39 smithi027 ceph-mon[39795]: pgmap v2090: 37 pgs: 37 active+clean; 833 KiB data, 7.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 505 op/s 2024-05-15T13:22:40.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:39 smithi120 ceph-mon[43961]: pgmap v2090: 37 pgs: 37 active+clean; 833 KiB data, 7.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 505 op/s 2024-05-15T13:22:40.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:39 smithi177 ceph-mon[44303]: pgmap v2090: 37 pgs: 37 active+clean; 833 KiB data, 7.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 505 op/s 2024-05-15T13:22:40.839 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:22:40.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:40.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:22:40.944 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:22:40.977 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:40.998 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:22:41.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:41.002 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:22:41.103 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:22:41.134 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:41.156 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:22:41.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:41.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:22:41.260 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:22:41.292 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:41.314 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:22:41.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:41.318 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:22:41.419 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:22:41.449 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:41.470 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:22:41.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:41.473 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:22:41.574 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:22:41.603 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:41.623 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:22:41.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:41.626 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:22:41.727 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:22:41.757 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:41.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:41 smithi177 ceph-mon[44303]: pgmap v2091: 37 pgs: 37 active+clean; 577 KiB data, 7.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 468 op/s 2024-05-15T13:22:42.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:41 smithi027 ceph-mon[39795]: pgmap v2091: 37 pgs: 37 active+clean; 577 KiB data, 7.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 468 op/s 2024-05-15T13:22:42.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:41 smithi120 ceph-mon[43961]: pgmap v2091: 37 pgs: 37 active+clean; 577 KiB data, 7.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 468 op/s 2024-05-15T13:22:42.407 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:22:42.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:42.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:22:42.512 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:22:42.544 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:42.565 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:22:42.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:42.569 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:22:42.670 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:22:42.700 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:42.773 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:22:42.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:42.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:22:42.878 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:22:42.906 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:43.004 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:22:43.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:43.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:22:43.108 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:22:43.137 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:43.239 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:22:43.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:43.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:22:43.343 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:22:43.375 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:43.392 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:22:43.397 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:22:43.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:43.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:22:43.501 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:22:43.532 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:43.554 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:22:43.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:43.557 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:22:43.658 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:22:43.688 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:43.708 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:22:43.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:43.712 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:22:43.813 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:22:43.842 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:43.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:43 smithi177 ceph-mon[44303]: pgmap v2092: 37 pgs: 37 active+clean; 577 KiB data, 7.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 377 op/s 2024-05-15T13:22:43.940 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:43 smithi027 ceph-mon[39795]: pgmap v2092: 37 pgs: 37 active+clean; 577 KiB data, 7.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 377 op/s 2024-05-15T13:22:43.940 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:43 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-a[39772]: 2024-05-15T13:22:43.804+0000 7f8daffd2700 -1 mon.a@0(leader).osd e801 definitely_dead 0 2024-05-15T13:22:43.991 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:22:43.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:43.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:22:44.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:43 smithi120 ceph-mon[43961]: pgmap v2092: 37 pgs: 37 active+clean; 577 KiB data, 7.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 377 op/s 2024-05-15T13:22:44.096 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:22:44.125 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:44.145 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:22:44.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:44.149 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:22:44.250 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:22:44.259 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:22:44.259 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:22:44.280 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:44.730 INFO:teuthology.orchestra.run.smithi027.stderr:marked out osd.8. 2024-05-15T13:22:45.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:44 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1888864256' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-05-15T13:22:45.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:44 smithi027 ceph-mon[39795]: Client client.admin marked osd.8 out, while it was still marked up 2024-05-15T13:22:45.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:44 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1888864256' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-05-15T13:22:45.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:44 smithi120 ceph-mon[43961]: Client client.admin marked osd.8 out, while it was still marked up 2024-05-15T13:22:45.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:44 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1888864256' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-05-15T13:22:45.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:44 smithi177 ceph-mon[44303]: Client client.admin marked osd.8 out, while it was still marked up 2024-05-15T13:22:45.735 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:45 smithi027 ceph-mon[39795]: pgmap v2093: 37 pgs: 37 active+clean; 577 KiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 90 op/s 2024-05-15T13:22:45.735 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:45 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1888864256' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["8"]}]': finished 2024-05-15T13:22:45.735 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:45 smithi027 ceph-mon[39795]: osdmap e802: 12 total, 12 up, 10 in 2024-05-15T13:22:46.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:45 smithi120 ceph-mon[43961]: pgmap v2093: 37 pgs: 37 active+clean; 577 KiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 90 op/s 2024-05-15T13:22:46.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:45 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1888864256' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["8"]}]': finished 2024-05-15T13:22:46.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:45 smithi120 ceph-mon[43961]: osdmap e802: 12 total, 12 up, 10 in 2024-05-15T13:22:46.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:45 smithi177 ceph-mon[44303]: pgmap v2093: 37 pgs: 37 active+clean; 577 KiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 90 op/s 2024-05-15T13:22:46.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:45 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1888864256' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["8"]}]': finished 2024-05-15T13:22:46.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:45 smithi177 ceph-mon[44303]: osdmap e802: 12 total, 12 up, 10 in 2024-05-15T13:22:47.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:46 smithi027 ceph-mon[39795]: osdmap e803: 12 total, 12 up, 10 in 2024-05-15T13:22:47.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:46 smithi027 ceph-mon[39795]: osdmap e804: 12 total, 12 up, 10 in 2024-05-15T13:22:47.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:46 smithi120 ceph-mon[43961]: osdmap e803: 12 total, 12 up, 10 in 2024-05-15T13:22:47.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:46 smithi120 ceph-mon[43961]: osdmap e804: 12 total, 12 up, 10 in 2024-05-15T13:22:47.175 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-05-15T13:22:47.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:46 smithi177 ceph-mon[44303]: osdmap e803: 12 total, 12 up, 10 in 2024-05-15T13:22:47.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:46 smithi177 ceph-mon[44303]: osdmap e804: 12 total, 12 up, 10 in 2024-05-15T13:22:47.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-05-15T13:22:46.955+0000 7f765e7fc700 0 --2- 172.21.15.27:0/449283870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76601084a0 0x7f7660108920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-05-15T13:22:47.735 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_6' removed 2024-05-15T13:22:48.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:47 smithi177 ceph-mon[44303]: pgmap v2096: 37 pgs: 3 remapped+peering, 2 peering, 32 active+clean; 577 KiB data, 7.5 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 0 op/s 2024-05-15T13:22:48.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:47 smithi177 ceph-mon[44303]: 8.b starting backfill to osd.7 from (0'0,0'0] MAX to 800'11184 2024-05-15T13:22:48.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:47 smithi177 ceph-mon[44303]: 8.c starting backfill to osd.7 from (0'0,0'0] MAX to 800'11274 2024-05-15T13:22:48.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:47 smithi177 ceph-mon[44303]: 8.a starting backfill to osd.3 from (0'0,0'0] MAX to 800'11172 2024-05-15T13:22:48.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:47 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3276012592' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T13:22:48.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:47 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3276012592' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T13:22:48.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:47 smithi177 ceph-mon[44303]: osdmap e805: 12 total, 12 up, 10 in 2024-05-15T13:22:48.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:47 smithi027 ceph-mon[39795]: pgmap v2096: 37 pgs: 3 remapped+peering, 2 peering, 32 active+clean; 577 KiB data, 7.5 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 0 op/s 2024-05-15T13:22:48.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:47 smithi027 ceph-mon[39795]: 8.b starting backfill to osd.7 from (0'0,0'0] MAX to 800'11184 2024-05-15T13:22:48.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:47 smithi027 ceph-mon[39795]: 8.c starting backfill to osd.7 from (0'0,0'0] MAX to 800'11274 2024-05-15T13:22:48.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:47 smithi027 ceph-mon[39795]: 8.a starting backfill to osd.3 from (0'0,0'0] MAX to 800'11172 2024-05-15T13:22:48.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:47 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3276012592' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T13:22:48.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:47 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3276012592' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T13:22:48.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:47 smithi027 ceph-mon[39795]: osdmap e805: 12 total, 12 up, 10 in 2024-05-15T13:22:48.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:47 smithi120 ceph-mon[43961]: pgmap v2096: 37 pgs: 3 remapped+peering, 2 peering, 32 active+clean; 577 KiB data, 7.5 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 0 op/s 2024-05-15T13:22:48.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:47 smithi120 ceph-mon[43961]: 8.b starting backfill to osd.7 from (0'0,0'0] MAX to 800'11184 2024-05-15T13:22:48.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:47 smithi120 ceph-mon[43961]: 8.c starting backfill to osd.7 from (0'0,0'0] MAX to 800'11274 2024-05-15T13:22:48.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:47 smithi120 ceph-mon[43961]: 8.a starting backfill to osd.3 from (0'0,0'0] MAX to 800'11172 2024-05-15T13:22:48.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:47 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3276012592' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T13:22:48.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:47 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3276012592' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T13:22:48.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:47 smithi120 ceph-mon[43961]: osdmap e805: 12 total, 12 up, 10 in 2024-05-15T13:22:48.292 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:22:48.311 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:22:48.331 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:22:48.351 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:22:48.371 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:22:48.392 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:22:48.413 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:22:48.433 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:22:48.454 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:22:48.474 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:22:48.496 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:22:48.496 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:22:48.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:48.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:22:48.521 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:22:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:22:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:22:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:22:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:22:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:22:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:22:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:22:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:22:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:22:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:22:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:22:48.524 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:22:48.601 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:22:48.633 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:48.655 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:22:48.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:48.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:22:48.759 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:22:48.787 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:48.808 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:22:48.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:48.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:22:48.912 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:22:48.942 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:49.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:48 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2781229624' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:22:49.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:48 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2917373555' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:22:49.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:48 smithi120 ceph-mon[43961]: osdmap e806: 12 total, 12 up, 10 in 2024-05-15T13:22:49.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:48 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2781229624' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:22:49.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:48 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2917373555' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:22:49.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:48 smithi177 ceph-mon[44303]: osdmap e806: 12 total, 12 up, 10 in 2024-05-15T13:22:49.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:48 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2781229624' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:22:49.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:48 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2917373555' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:22:49.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:48 smithi027 ceph-mon[39795]: osdmap e806: 12 total, 12 up, 10 in 2024-05-15T13:22:49.704 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:22:49.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:49.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:22:49.725 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-05-15T13:22:49.726 INFO:tasks.radosbench:Beginning radosbench... 2024-05-15T13:22:49.727 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_7 2024-05-15T13:22:49.727 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool create unique_pool_7 16 2024-05-15T13:22:49.728 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:22:49.739 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:22:49.810 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:22:49.836 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:49.855 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:22:49.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:49.859 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:22:49.960 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:22:49.971 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:49 smithi027 ceph-mon[39795]: pgmap v2099: 21 pgs: 3 remapped+peering, 3 peering, 15 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:22:49.971 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:49 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:22:49.971 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:49 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2917373555' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:22:49.971 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:49 smithi027 ceph-mon[39795]: osdmap e807: 12 total, 12 up, 10 in 2024-05-15T13:22:49.993 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:50.014 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:22:50.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:50.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:22:50.119 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:22:50.144 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:49 smithi177 ceph-mon[44303]: pgmap v2099: 21 pgs: 3 remapped+peering, 3 peering, 15 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:22:50.144 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:49 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:22:50.144 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:49 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2917373555' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:22:50.144 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:49 smithi177 ceph-mon[44303]: osdmap e807: 12 total, 12 up, 10 in 2024-05-15T13:22:50.146 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:50.164 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:22:50.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:50.167 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:22:50.268 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:22:50.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:49 smithi120 ceph-mon[43961]: pgmap v2099: 21 pgs: 3 remapped+peering, 3 peering, 15 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:22:50.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:49 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:22:50.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:49 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2917373555' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:22:50.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:49 smithi120 ceph-mon[43961]: osdmap e807: 12 total, 12 up, 10 in 2024-05-15T13:22:50.294 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:50.313 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:22:50.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:50.316 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:22:50.417 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:22:50.446 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:50.695 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:22:50.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:50.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:22:50.801 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:22:50.834 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:50.856 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:22:50.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:50.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:22:50.960 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:22:50.990 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:51.010 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:22:51.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:51.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:22:51.115 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:22:51.145 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:51.243 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:22:51.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:51.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:22:51.348 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:22:51.380 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:51.381 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:22:51.402 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:22:51.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:51.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:22:51.506 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:22:51.538 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:51.559 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:22:51.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:51.563 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:22:51.568 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:22:51.664 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:22:51.694 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:51.714 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:22:51.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:51.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:22:51.819 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:22:51.847 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:51.867 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:22:51.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:51.871 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:22:51.972 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:22:52.000 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:52.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:51 smithi120 ceph-mon[43961]: pgmap v2102: 21 pgs: 3 remapped+peering, 3 peering, 15 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:22:52.066 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:51 smithi027 ceph-mon[39795]: pgmap v2102: 21 pgs: 3 remapped+peering, 3 peering, 15 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:22:52.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:51 smithi177 ceph-mon[44303]: pgmap v2102: 21 pgs: 3 remapped+peering, 3 peering, 15 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:22:52.448 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:22:52.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:52.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:22:52.520 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 9, 10, 11, 7] out_osds: [4, 8] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:22:52.520 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:22:52.520 INFO:tasks.thrashosds.thrasher:inject_pause on osd.10 2024-05-15T13:22:52.520 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-15T13:22:52.521 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-15T13:22:52.521 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 config set filestore_inject_stall 3 2024-05-15T13:22:52.553 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:22:52.582 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:52.655 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:22:52.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:52.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:22:52.759 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:22:52.784 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:22:52.789 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:52.794 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:22:52.804 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:22:52.805 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T13:22:52.805 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-05-15T13:22:52.805 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-05-15T13:22:52.805 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:22:52.810 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:22:52.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:52.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:22:52.916 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:22:52.948 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:52.969 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:22:52.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:52.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:22:53.054 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:52 smithi027 ceph-mon[39795]: 1.5 scrub starts 2024-05-15T13:22:53.073 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:22:53.101 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:52 smithi120 ceph-mon[43961]: 1.5 scrub starts 2024-05-15T13:22:53.103 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:53.124 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:22:53.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:53.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:22:53.131 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:52 smithi177 ceph-mon[44303]: 1.5 scrub starts 2024-05-15T13:22:53.228 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:22:53.256 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:53.329 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:22:53.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:53.333 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:22:53.434 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:22:53.465 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:53.487 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:22:53.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:53.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:22:53.591 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:22:53.605 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:22:53.621 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:53.641 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:22:53.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:53.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:22:53.745 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:22:53.775 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:53.795 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:22:53.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:53.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:22:53.900 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:22:53.928 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:54.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:53 smithi120 ceph-mon[43961]: 1.5 scrub ok 2024-05-15T13:22:54.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:53 smithi120 ceph-mon[43961]: pgmap v2103: 21 pgs: 2 peering, 19 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:22:54.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:53 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:22:54.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:53 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:22:54.099 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:53 smithi027 ceph-mon[39795]: 1.5 scrub ok 2024-05-15T13:22:54.099 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:53 smithi027 ceph-mon[39795]: pgmap v2103: 21 pgs: 2 peering, 19 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:22:54.100 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:53 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:22:54.100 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:53 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:22:54.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:53 smithi177 ceph-mon[44303]: 1.5 scrub ok 2024-05-15T13:22:54.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:53 smithi177 ceph-mon[44303]: pgmap v2103: 21 pgs: 2 peering, 19 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:22:54.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:53 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:22:54.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:53 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:22:54.971 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:22:54.971 INFO:teuthology.orchestra.run.smithi177.stdout: "success": "" 2024-05-15T13:22:54.971 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:22:55.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:54 smithi027 ceph-mon[39795]: osdmap e808: 12 total, 12 up, 10 in 2024-05-15T13:22:55.087 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-05-15T13:22:55.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:54 smithi177 ceph-mon[44303]: osdmap e808: 12 total, 12 up, 10 in 2024-05-15T13:22:55.279 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:22:55.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:55.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:22:55.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:54 smithi120 ceph-mon[43961]: osdmap e808: 12 total, 12 up, 10 in 2024-05-15T13:22:55.385 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:22:55.414 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:55.501 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:22:55.549 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:22:55.569 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:22:55.588 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:22:55.607 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:22:55.628 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:22:55.648 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:22:55.664 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:22:55.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:55.667 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:22:55.671 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:22:55.691 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:22:55.711 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:22:55.731 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:22:55.750 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:22:55.768 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:22:55.772 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:22:55.773 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:22:55.773 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:22:55.773 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:22:55.773 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:22:55.773 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:22:55.773 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:22:55.774 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:22:55.774 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:22:55.774 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:22:55.774 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:22:55.774 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:22:55.774 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:22:55.781 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:22:55.794 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:55.813 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:22:55.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:55.817 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:22:55.918 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:22:55.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:55 smithi177 ceph-mon[44303]: pgmap v2105: 21 pgs: 21 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:22:55.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:55 smithi177 ceph-mon[44303]: 1.3 scrub starts 2024-05-15T13:22:55.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:55 smithi177 ceph-mon[44303]: 1.3 scrub ok 2024-05-15T13:22:55.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:55 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/872271495' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:22:55.944 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:55.963 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:22:55.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:55.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:22:56.021 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:55 smithi027 ceph-mon[39795]: pgmap v2105: 21 pgs: 21 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:22:56.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:55 smithi027 ceph-mon[39795]: 1.3 scrub starts 2024-05-15T13:22:56.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:55 smithi027 ceph-mon[39795]: 1.3 scrub ok 2024-05-15T13:22:56.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:55 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/872271495' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:22:56.067 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:22:56.094 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:56.112 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:22:56.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:56.116 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:22:56.217 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:22:56.247 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:56.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:55 smithi120 ceph-mon[43961]: pgmap v2105: 21 pgs: 21 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:22:56.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:55 smithi120 ceph-mon[43961]: 1.3 scrub starts 2024-05-15T13:22:56.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:55 smithi120 ceph-mon[43961]: 1.3 scrub ok 2024-05-15T13:22:56.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:55 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/872271495' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:22:56.346 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:22:56.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:56.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:22:56.450 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:22:56.477 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:57.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:56 smithi177 ceph-mon[44303]: 1.b scrub starts 2024-05-15T13:22:57.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:56 smithi177 ceph-mon[44303]: 1.b scrub ok 2024-05-15T13:22:57.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:56 smithi027 ceph-mon[39795]: 1.b scrub starts 2024-05-15T13:22:57.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:56 smithi027 ceph-mon[39795]: 1.b scrub ok 2024-05-15T13:22:57.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:56 smithi120 ceph-mon[43961]: 1.b scrub starts 2024-05-15T13:22:57.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:56 smithi120 ceph-mon[43961]: 1.b scrub ok 2024-05-15T13:22:57.328 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:22:57.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:57.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:22:57.434 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:22:57.463 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:57.662 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:22:57.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:57.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:22:57.766 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:22:57.795 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_7' created 2024-05-15T13:22:57.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:57 smithi027 ceph-mon[39795]: 1.c scrub starts 2024-05-15T13:22:57.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:57 smithi027 ceph-mon[39795]: 1.c scrub ok 2024-05-15T13:22:57.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:57 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2752306926' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-05-15T13:22:57.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:57 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-05-15T13:22:57.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:57 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:22:57.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:57 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:22:57.904 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:57 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:22:57.904 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:57 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:22:57.904 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:57 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:22:57.904 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:57 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:22:57.904 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:57 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:22:58.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:57 smithi177 ceph-mon[44303]: 1.c scrub starts 2024-05-15T13:22:58.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:57 smithi177 ceph-mon[44303]: 1.c scrub ok 2024-05-15T13:22:58.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:57 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2752306926' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-05-15T13:22:58.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:57 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-05-15T13:22:58.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:57 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:22:58.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:57 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:22:58.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:57 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:22:58.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:57 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:22:58.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:57 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:22:58.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:57 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:22:58.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:57 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:22:58.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:57 smithi120 ceph-mon[43961]: 1.c scrub starts 2024-05-15T13:22:58.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:57 smithi120 ceph-mon[43961]: 1.c scrub ok 2024-05-15T13:22:58.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:57 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2752306926' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-05-15T13:22:58.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:57 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-05-15T13:22:58.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:57 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:22:58.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:57 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:22:58.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:57 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:22:58.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:57 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:22:58.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:57 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:22:58.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:57 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:22:58.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:57 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:22:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:22:58.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:58.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:22:58.980 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:22:59.013 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:59.035 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:22:59.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:59.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:22:59.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:58 smithi027 ceph-mon[39795]: 1.2 scrub starts 2024-05-15T13:22:59.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:58 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-05-15T13:22:59.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:58 smithi027 ceph-mon[39795]: osdmap e809: 12 total, 12 up, 10 in 2024-05-15T13:22:59.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:58 smithi027 ceph-mon[39795]: 1.2 scrub ok 2024-05-15T13:22:59.140 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:22:59.170 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:58 smithi177 ceph-mon[44303]: 1.2 scrub starts 2024-05-15T13:22:59.170 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:58 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-05-15T13:22:59.170 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:58 smithi177 ceph-mon[44303]: osdmap e809: 12 total, 12 up, 10 in 2024-05-15T13:22:59.171 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:58 smithi177 ceph-mon[44303]: 1.2 scrub ok 2024-05-15T13:22:59.172 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:59.194 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:22:59.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:59.197 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:22:59.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:58 smithi120 ceph-mon[43961]: 1.2 scrub starts 2024-05-15T13:22:59.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:58 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-05-15T13:22:59.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:58 smithi120 ceph-mon[43961]: osdmap e809: 12 total, 12 up, 10 in 2024-05-15T13:22:59.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:58 smithi120 ceph-mon[43961]: 1.2 scrub ok 2024-05-15T13:22:59.298 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:22:59.329 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:22:59.672 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:22:59.777 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool application enable unique_pool_7 rados --yes-i-really-mean-it || true 2024-05-15T13:22:59.980 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:22:59.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:59 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:22:59.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:59 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:22:59.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:59 smithi027 ceph-mon[39795]: pgmap v2108: 37 pgs: 6 creating+peering, 10 unknown, 21 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:22:59.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:59 smithi027 ceph-mon[39795]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T13:22:59.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:59 smithi027 ceph-mon[39795]: osdmap e810: 12 total, 12 up, 10 in 2024-05-15T13:22:59.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:22:59 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3204148111' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:22:59.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:22:59.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:23:00.085 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:23:00.115 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:00.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:59 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:23:00.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:59 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:23:00.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:59 smithi177 ceph-mon[44303]: pgmap v2108: 37 pgs: 6 creating+peering, 10 unknown, 21 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:23:00.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:59 smithi177 ceph-mon[44303]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T13:23:00.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:59 smithi177 ceph-mon[44303]: osdmap e810: 12 total, 12 up, 10 in 2024-05-15T13:23:00.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:22:59 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3204148111' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:23:00.238 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:23:00.239 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:23:00.239 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:23:00.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:00.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:23:00.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:59 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:23:00.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:59 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:23:00.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:59 smithi120 ceph-mon[43961]: pgmap v2108: 37 pgs: 6 creating+peering, 10 unknown, 21 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:23:00.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:59 smithi120 ceph-mon[43961]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T13:23:00.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:59 smithi120 ceph-mon[43961]: osdmap e810: 12 total, 12 up, 10 in 2024-05-15T13:23:00.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:22:59 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3204148111' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:23:00.342 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:23:00.375 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:00.375 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:23:00.396 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:23:00.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:00.400 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:23:00.501 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:23:00.532 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:00.554 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:23:00.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:00.557 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:23:00.658 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:23:00.671 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 9, 10, 11, 7] out_osds: [4, 8] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:23:00.671 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:23:00.671 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-05-15T13:23:00.671 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd dump -f json-pretty 2024-05-15T13:23:00.687 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:00.709 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:23:00.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:00.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:23:00.816 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:23:00.849 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:00.871 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:23:00.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:00.875 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:23:00.976 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:23:01.002 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:00 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:23:01.002 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:00 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3204148111' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:23:01.002 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:00 smithi027 ceph-mon[39795]: osdmap e811: 12 total, 12 up, 10 in 2024-05-15T13:23:01.003 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:01.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:00 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:23:01.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:00 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3204148111' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:23:01.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:00 smithi177 ceph-mon[44303]: osdmap e811: 12 total, 12 up, 10 in 2024-05-15T13:23:01.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:00 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:23:01.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:00 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3204148111' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:23:01.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:00 smithi120 ceph-mon[43961]: osdmap e811: 12 total, 12 up, 10 in 2024-05-15T13:23:01.640 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:23:01.642 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:23:01.656 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:23:01.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:01.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:23:01.684 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:23:01.690 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-05-15T13:23:01.762 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:23:01.795 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:01.816 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:23:01.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:01.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:23:01.923 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:23:01.945 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:01 smithi027 ceph-mon[39795]: pgmap v2111: 37 pgs: 10 creating+peering, 6 unknown, 21 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:23:01.956 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:02.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:01 smithi177 ceph-mon[44303]: pgmap v2111: 37 pgs: 10 creating+peering, 6 unknown, 21 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:23:02.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:01 smithi120 ceph-mon[43961]: pgmap v2111: 37 pgs: 10 creating+peering, 6 unknown, 21 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:23:02.864 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:02 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:23:03.041 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:23:03.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:02 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:23:03.208 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:23:03.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:03.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:23:03.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:02 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:23:03.312 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:23:03.344 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:03.366 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:23:03.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:03.370 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:23:03.471 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:23:03.501 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:03.555 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:23:03.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:03.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:23:03.660 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:23:03.691 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:03.711 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:23:03.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:03.715 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:23:03.816 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:23:03.844 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:03.865 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:23:03.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:03.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:23:03.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:03 smithi177 ceph-mon[44303]: pgmap v2112: 37 pgs: 10 creating+peering, 27 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:23:03.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:03 smithi177 ceph-mon[44303]: osdmap e812: 12 total, 12 up, 10 in 2024-05-15T13:23:03.969 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:23:03.996 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:04.015 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:23:04.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:04.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:23:04.119 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:23:04.148 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:04.168 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:23:04.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:04.171 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:23:04.253 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:03 smithi027 ceph-mon[39795]: pgmap v2112: 37 pgs: 10 creating+peering, 27 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:23:04.253 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:03 smithi027 ceph-mon[39795]: osdmap e812: 12 total, 12 up, 10 in 2024-05-15T13:23:04.272 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:23:04.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:03 smithi120 ceph-mon[43961]: pgmap v2112: 37 pgs: 10 creating+peering, 27 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:23:04.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:03 smithi120 ceph-mon[43961]: osdmap e812: 12 total, 12 up, 10 in 2024-05-15T13:23:04.299 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:04.318 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:23:04.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:04.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:23:04.422 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:23:04.451 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:04.470 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:23:04.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:04.473 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:23:04.574 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:23:04.605 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:04.679 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:23:04.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:04.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:23:04.783 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:23:04.812 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:04.832 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:23:04.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:04.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:23:04.924 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:23:04.936 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:23:04.944 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:23:04.964 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:23:04.965 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:04.987 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:23:05.007 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:23:05.027 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:23:05.049 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:23:05.069 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:23:05.089 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:23:05.110 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:23:05.131 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:23:05.144 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:04 smithi027 ceph-mon[39795]: 1.d deep-scrub starts 2024-05-15T13:23:05.145 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:04 smithi027 ceph-mon[39795]: 1.d deep-scrub ok 2024-05-15T13:23:05.153 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:23:05.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:23:05.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:23:05.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:23:05.156 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:23:05.156 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:23:05.156 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:23:05.156 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:23:05.156 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:23:05.156 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:23:05.156 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:23:05.156 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:23:05.156 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:23:05.165 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:23:05.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:05.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:23:05.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:04 smithi177 ceph-mon[44303]: 1.d deep-scrub starts 2024-05-15T13:23:05.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:04 smithi177 ceph-mon[44303]: 1.d deep-scrub ok 2024-05-15T13:23:05.270 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:23:05.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:04 smithi120 ceph-mon[43961]: 1.d deep-scrub starts 2024-05-15T13:23:05.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:04 smithi120 ceph-mon[43961]: 1.d deep-scrub ok 2024-05-15T13:23:05.298 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:05.317 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:23:05.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:05.320 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:23:05.421 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:23:05.450 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:06.030 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:23:06.030 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:23:06.030 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 812, 2024-05-15T13:23:06.030 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "deaea04e-12b7-11ef-bc98-c7b262605968", 2024-05-15T13:23:06.031 INFO:teuthology.orchestra.run.smithi027.stdout: "created": "2024-05-15T12:38:21.244556+0000", 2024-05-15T13:23:06.031 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-05-15T13:23:02.855133+0000", 2024-05-15T13:23:06.031 INFO:teuthology.orchestra.run.smithi027.stdout: "last_up_change": "2024-05-15T12:45:50.876090+0000", 2024-05-15T13:23:06.031 INFO:teuthology.orchestra.run.smithi027.stdout: "last_in_change": "2024-05-15T13:22:44.721594+0000", 2024-05-15T13:23:06.031 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-15T13:23:06.031 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_num": 5799936, 2024-05-15T13:23:06.031 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_set": [ 2024-05-15T13:23:06.031 INFO:teuthology.orchestra.run.smithi027.stdout: "pglog_hardlimit", 2024-05-15T13:23:06.031 INFO:teuthology.orchestra.run.smithi027.stdout: "purged_snapdirs", 2024-05-15T13:23:06.031 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_deletes", 2024-05-15T13:23:06.031 INFO:teuthology.orchestra.run.smithi027.stdout: "sortbitwise" 2024-05-15T13:23:06.031 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:23:06.031 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_version": 26, 2024-05-15T13:23:06.031 INFO:teuthology.orchestra.run.smithi027.stdout: "full_ratio": 0.94999998807907104, 2024-05-15T13:23:06.032 INFO:teuthology.orchestra.run.smithi027.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-05-15T13:23:06.032 INFO:teuthology.orchestra.run.smithi027.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-15T13:23:06.032 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_snapshot": "", 2024-05-15T13:23:06.032 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_max": 9, 2024-05-15T13:23:06.032 INFO:teuthology.orchestra.run.smithi027.stdout: "max_osd": 12, 2024-05-15T13:23:06.032 INFO:teuthology.orchestra.run.smithi027.stdout: "require_min_compat_client": "luminous", 2024-05-15T13:23:06.032 INFO:teuthology.orchestra.run.smithi027.stdout: "min_compat_client": "luminous", 2024-05-15T13:23:06.032 INFO:teuthology.orchestra.run.smithi027.stdout: "require_osd_release": "squid", 2024-05-15T13:23:06.032 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_crimson": false, 2024-05-15T13:23:06.032 INFO:teuthology.orchestra.run.smithi027.stdout: "pools": [ 2024-05-15T13:23:06.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.032 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 1, 2024-05-15T13:23:06.032 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": ".mgr", 2024-05-15T13:23:06.033 INFO:teuthology.orchestra.run.smithi027.stdout: "create_time": "2024-05-15T12:41:57.948613+0000", 2024-05-15T13:23:06.033 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-15T13:23:06.033 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_names": "hashpspool", 2024-05-15T13:23:06.033 INFO:teuthology.orchestra.run.smithi027.stdout: "type": 1, 2024-05-15T13:23:06.033 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 3, 2024-05-15T13:23:06.033 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": 2, 2024-05-15T13:23:06.033 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_rule": 0, 2024-05-15T13:23:06.033 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_count": 0, 2024-05-15T13:23:06.033 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_target": 0, 2024-05-15T13:23:06.033 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T13:23:06.033 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T13:23:06.033 INFO:teuthology.orchestra.run.smithi027.stdout: "object_hash": 2, 2024-05-15T13:23:06.033 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscale_mode": "off", 2024-05-15T13:23:06.033 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": 21, 2024-05-15T13:23:06.034 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num": 21, 2024-05-15T13:23:06.034 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num_target": 21, 2024-05-15T13:23:06.034 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_target": 21, 2024-05-15T13:23:06.034 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_pending": 21, 2024-05-15T13:23:06.034 INFO:teuthology.orchestra.run.smithi027.stdout: "last_pg_merge_meta": { 2024-05-15T13:23:06.034 INFO:teuthology.orchestra.run.smithi027.stdout: "source_pgid": "0.0", 2024-05-15T13:23:06.034 INFO:teuthology.orchestra.run.smithi027.stdout: "ready_epoch": 0, 2024-05-15T13:23:06.034 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_started": 0, 2024-05-15T13:23:06.034 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_clean": 0, 2024-05-15T13:23:06.034 INFO:teuthology.orchestra.run.smithi027.stdout: "source_version": "0'0", 2024-05-15T13:23:06.034 INFO:teuthology.orchestra.run.smithi027.stdout: "target_version": "0'0" 2024-05-15T13:23:06.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.035 INFO:teuthology.orchestra.run.smithi027.stdout: "last_change": "753", 2024-05-15T13:23:06.035 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend": "0", 2024-05-15T13:23:06.035 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T13:23:06.035 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_preluminous": "614", 2024-05-15T13:23:06.035 INFO:teuthology.orchestra.run.smithi027.stdout: "auid": 0, 2024-05-15T13:23:06.035 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_mode": "selfmanaged", 2024-05-15T13:23:06.035 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_seq": 0, 2024-05-15T13:23:06.035 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_epoch": 0, 2024-05-15T13:23:06.035 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_snaps": [], 2024-05-15T13:23:06.035 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps": "[]", 2024-05-15T13:23:06.035 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_bytes": 0, 2024-05-15T13:23:06.035 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_objects": 0, 2024-05-15T13:23:06.035 INFO:teuthology.orchestra.run.smithi027.stdout: "tiers": [], 2024-05-15T13:23:06.035 INFO:teuthology.orchestra.run.smithi027.stdout: "tier_of": -1, 2024-05-15T13:23:06.036 INFO:teuthology.orchestra.run.smithi027.stdout: "read_tier": -1, 2024-05-15T13:23:06.036 INFO:teuthology.orchestra.run.smithi027.stdout: "write_tier": -1, 2024-05-15T13:23:06.036 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_mode": "none", 2024-05-15T13:23:06.036 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_bytes": 0, 2024-05-15T13:23:06.036 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_objects": 0, 2024-05-15T13:23:06.036 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T13:23:06.036 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T13:23:06.036 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T13:23:06.036 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_flush_age": 0, 2024-05-15T13:23:06.036 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_evict_age": 0, 2024-05-15T13:23:06.036 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profile": "", 2024-05-15T13:23:06.036 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_params": { 2024-05-15T13:23:06.036 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "none" 2024-05-15T13:23:06.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.037 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_period": 0, 2024-05-15T13:23:06.037 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_count": 0, 2024-05-15T13:23:06.037 INFO:teuthology.orchestra.run.smithi027.stdout: "use_gmt_hitset": true, 2024-05-15T13:23:06.037 INFO:teuthology.orchestra.run.smithi027.stdout: "min_read_recency_for_promote": 0, 2024-05-15T13:23:06.037 INFO:teuthology.orchestra.run.smithi027.stdout: "min_write_recency_for_promote": 0, 2024-05-15T13:23:06.037 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T13:23:06.037 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_search_last_n": 0, 2024-05-15T13:23:06.037 INFO:teuthology.orchestra.run.smithi027.stdout: "grade_table": [], 2024-05-15T13:23:06.037 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_width": 0, 2024-05-15T13:23:06.037 INFO:teuthology.orchestra.run.smithi027.stdout: "expected_num_objects": 0, 2024-05-15T13:23:06.037 INFO:teuthology.orchestra.run.smithi027.stdout: "fast_read": false, 2024-05-15T13:23:06.037 INFO:teuthology.orchestra.run.smithi027.stdout: "options": { 2024-05-15T13:23:06.037 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_max": 32, 2024-05-15T13:23:06.038 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_min": 1 2024-05-15T13:23:06.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.038 INFO:teuthology.orchestra.run.smithi027.stdout: "application_metadata": { 2024-05-15T13:23:06.038 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": {} 2024-05-15T13:23:06.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.038 INFO:teuthology.orchestra.run.smithi027.stdout: "read_balance": { 2024-05-15T13:23:06.038 INFO:teuthology.orchestra.run.smithi027.stdout: "score_type": "Fair distribution", 2024-05-15T13:23:06.038 INFO:teuthology.orchestra.run.smithi027.stdout: "score_acting": 2.2899999618530273, 2024-05-15T13:23:06.038 INFO:teuthology.orchestra.run.smithi027.stdout: "score_stable": 2.2899999618530273, 2024-05-15T13:23:06.038 INFO:teuthology.orchestra.run.smithi027.stdout: "optimal_score": 0.82999998331069946, 2024-05-15T13:23:06.038 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_acting": 1.8999999761581421, 2024-05-15T13:23:06.038 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_stable": 1.8999999761581421, 2024-05-15T13:23:06.038 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity_weighted": 0.94652295112609863, 2024-05-15T13:23:06.039 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity": 0.94999998807907104, 2024-05-15T13:23:06.039 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-05-15T13:23:06.039 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.039 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.039 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.039 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 9, 2024-05-15T13:23:06.039 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": "unique_pool_7", 2024-05-15T13:23:06.039 INFO:teuthology.orchestra.run.smithi027.stdout: "create_time": "2024-05-15T13:22:57.418245+0000", 2024-05-15T13:23:06.039 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-05-15T13:23:06.039 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_names": "hashpspool", 2024-05-15T13:23:06.039 INFO:teuthology.orchestra.run.smithi027.stdout: "type": 1, 2024-05-15T13:23:06.039 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 3, 2024-05-15T13:23:06.039 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": 2, 2024-05-15T13:23:06.039 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_rule": 0, 2024-05-15T13:23:06.040 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_count": 0, 2024-05-15T13:23:06.040 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_target": 0, 2024-05-15T13:23:06.040 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T13:23:06.040 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T13:23:06.040 INFO:teuthology.orchestra.run.smithi027.stdout: "object_hash": 2, 2024-05-15T13:23:06.040 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscale_mode": "off", 2024-05-15T13:23:06.040 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": 16, 2024-05-15T13:23:06.040 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num": 16, 2024-05-15T13:23:06.040 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num_target": 16, 2024-05-15T13:23:06.040 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_target": 16, 2024-05-15T13:23:06.040 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_pending": 16, 2024-05-15T13:23:06.040 INFO:teuthology.orchestra.run.smithi027.stdout: "last_pg_merge_meta": { 2024-05-15T13:23:06.040 INFO:teuthology.orchestra.run.smithi027.stdout: "source_pgid": "0.0", 2024-05-15T13:23:06.040 INFO:teuthology.orchestra.run.smithi027.stdout: "ready_epoch": 0, 2024-05-15T13:23:06.041 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_started": 0, 2024-05-15T13:23:06.041 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_clean": 0, 2024-05-15T13:23:06.041 INFO:teuthology.orchestra.run.smithi027.stdout: "source_version": "0'0", 2024-05-15T13:23:06.041 INFO:teuthology.orchestra.run.smithi027.stdout: "target_version": "0'0" 2024-05-15T13:23:06.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.041 INFO:teuthology.orchestra.run.smithi027.stdout: "last_change": "811", 2024-05-15T13:23:06.041 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend": "0", 2024-05-15T13:23:06.041 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T13:23:06.041 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T13:23:06.041 INFO:teuthology.orchestra.run.smithi027.stdout: "auid": 0, 2024-05-15T13:23:06.041 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_mode": "selfmanaged", 2024-05-15T13:23:06.041 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_seq": 0, 2024-05-15T13:23:06.041 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_epoch": 0, 2024-05-15T13:23:06.041 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_snaps": [], 2024-05-15T13:23:06.042 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps": "[]", 2024-05-15T13:23:06.042 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_bytes": 0, 2024-05-15T13:23:06.042 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_objects": 0, 2024-05-15T13:23:06.042 INFO:teuthology.orchestra.run.smithi027.stdout: "tiers": [], 2024-05-15T13:23:06.042 INFO:teuthology.orchestra.run.smithi027.stdout: "tier_of": -1, 2024-05-15T13:23:06.042 INFO:teuthology.orchestra.run.smithi027.stdout: "read_tier": -1, 2024-05-15T13:23:06.042 INFO:teuthology.orchestra.run.smithi027.stdout: "write_tier": -1, 2024-05-15T13:23:06.042 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_mode": "none", 2024-05-15T13:23:06.042 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_bytes": 0, 2024-05-15T13:23:06.042 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_objects": 0, 2024-05-15T13:23:06.042 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T13:23:06.042 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T13:23:06.042 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T13:23:06.042 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_flush_age": 0, 2024-05-15T13:23:06.043 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_evict_age": 0, 2024-05-15T13:23:06.043 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profile": "", 2024-05-15T13:23:06.043 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_params": { 2024-05-15T13:23:06.043 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "none" 2024-05-15T13:23:06.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.043 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_period": 0, 2024-05-15T13:23:06.043 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_count": 0, 2024-05-15T13:23:06.043 INFO:teuthology.orchestra.run.smithi027.stdout: "use_gmt_hitset": true, 2024-05-15T13:23:06.043 INFO:teuthology.orchestra.run.smithi027.stdout: "min_read_recency_for_promote": 0, 2024-05-15T13:23:06.043 INFO:teuthology.orchestra.run.smithi027.stdout: "min_write_recency_for_promote": 0, 2024-05-15T13:23:06.043 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T13:23:06.043 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_search_last_n": 0, 2024-05-15T13:23:06.043 INFO:teuthology.orchestra.run.smithi027.stdout: "grade_table": [], 2024-05-15T13:23:06.043 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_width": 0, 2024-05-15T13:23:06.044 INFO:teuthology.orchestra.run.smithi027.stdout: "expected_num_objects": 0, 2024-05-15T13:23:06.044 INFO:teuthology.orchestra.run.smithi027.stdout: "fast_read": false, 2024-05-15T13:23:06.044 INFO:teuthology.orchestra.run.smithi027.stdout: "options": {}, 2024-05-15T13:23:06.044 INFO:teuthology.orchestra.run.smithi027.stdout: "application_metadata": {}, 2024-05-15T13:23:06.044 INFO:teuthology.orchestra.run.smithi027.stdout: "read_balance": { 2024-05-15T13:23:06.044 INFO:teuthology.orchestra.run.smithi027.stdout: "score_type": "Fair distribution", 2024-05-15T13:23:06.044 INFO:teuthology.orchestra.run.smithi027.stdout: "score_acting": 3, 2024-05-15T13:23:06.044 INFO:teuthology.orchestra.run.smithi027.stdout: "score_stable": 3, 2024-05-15T13:23:06.044 INFO:teuthology.orchestra.run.smithi027.stdout: "optimal_score": 0.75, 2024-05-15T13:23:06.044 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_acting": 2.25, 2024-05-15T13:23:06.044 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_stable": 2.25, 2024-05-15T13:23:06.044 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity_weighted": 0.94652295112609863, 2024-05-15T13:23:06.044 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity": 0.94999998807907104, 2024-05-15T13:23:06.044 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-05-15T13:23:06.044 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.045 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.045 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:23:06.045 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T13:23:06.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.045 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 0, 2024-05-15T13:23:06.045 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "64fb3ad2-08a7-4106-a07a-6c776c877faf", 2024-05-15T13:23:06.045 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:23:06.045 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:23:06.045 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:23:06.045 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:23:06.045 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:23:06.045 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:23:06.045 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 8, 2024-05-15T13:23:06.045 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 805, 2024-05-15T13:23:06.046 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:23:06.046 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:23:06.046 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:23:06.046 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.046 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.046 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.046 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6802", 2024-05-15T13:23:06.046 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:23:06.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.046 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.046 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.046 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6803", 2024-05-15T13:23:06.046 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:23:06.047 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.047 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.047 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:23:06.047 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.047 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.047 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.047 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6804", 2024-05-15T13:23:06.047 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:23:06.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.047 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.047 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.047 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6805", 2024-05-15T13:23:06.047 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:23:06.048 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.048 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.048 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:23:06.048 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.048 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.048 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.048 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6808", 2024-05-15T13:23:06.048 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:23:06.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.048 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.048 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.048 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6809", 2024-05-15T13:23:06.048 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:23:06.049 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.049 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.049 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:23:06.049 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.049 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.049 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.049 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6806", 2024-05-15T13:23:06.049 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:23:06.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.049 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.049 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.049 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6807", 2024-05-15T13:23:06.049 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4239116935 2024-05-15T13:23:06.049 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.050 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.050 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6803/4239116935", 2024-05-15T13:23:06.050 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6805/4239116935", 2024-05-15T13:23:06.050 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6809/4239116935", 2024-05-15T13:23:06.050 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6807/4239116935", 2024-05-15T13:23:06.050 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:23:06.050 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:23:06.050 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:23:06.050 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.050 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 1, 2024-05-15T13:23:06.050 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "9c16dd90-9d51-48b8-a296-1c8879ba1bc8", 2024-05-15T13:23:06.051 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:23:06.051 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:23:06.051 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:23:06.051 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:23:06.051 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:23:06.051 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:23:06.051 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 13, 2024-05-15T13:23:06.051 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 809, 2024-05-15T13:23:06.051 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:23:06.051 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:23:06.051 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:23:06.051 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.051 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.052 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6810", 2024-05-15T13:23:06.052 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:23:06.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.052 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.052 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6811", 2024-05-15T13:23:06.052 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:23:06.052 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.052 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.052 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:23:06.052 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.052 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.053 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6812", 2024-05-15T13:23:06.053 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:23:06.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.053 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.053 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6813", 2024-05-15T13:23:06.053 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:23:06.053 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.053 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.053 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:23:06.053 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.053 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.054 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6816", 2024-05-15T13:23:06.054 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:23:06.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.054 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.054 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.054 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6817", 2024-05-15T13:23:06.054 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:23:06.054 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.054 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.054 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:23:06.054 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.054 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.055 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.055 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6814", 2024-05-15T13:23:06.055 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:23:06.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.055 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.055 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6815", 2024-05-15T13:23:06.055 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1466270637 2024-05-15T13:23:06.055 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.055 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.055 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6811/1466270637", 2024-05-15T13:23:06.055 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6813/1466270637", 2024-05-15T13:23:06.055 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6817/1466270637", 2024-05-15T13:23:06.056 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6815/1466270637", 2024-05-15T13:23:06.056 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:23:06.056 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:23:06.056 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:23:06.056 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.056 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 2, 2024-05-15T13:23:06.056 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "cbe3ba13-3ea3-4891-8d2d-6b85581a732c", 2024-05-15T13:23:06.056 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:23:06.056 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:23:06.056 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:23:06.056 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:23:06.056 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:23:06.057 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:23:06.057 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 18, 2024-05-15T13:23:06.057 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 803, 2024-05-15T13:23:06.057 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:23:06.057 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:23:06.057 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:23:06.057 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.057 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.057 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6818", 2024-05-15T13:23:06.057 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:23:06.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.057 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.058 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6819", 2024-05-15T13:23:06.058 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:23:06.058 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.058 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.058 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:23:06.058 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.058 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.058 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.058 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6820", 2024-05-15T13:23:06.058 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:23:06.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.058 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.058 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.059 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6821", 2024-05-15T13:23:06.059 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:23:06.059 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.059 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.059 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:23:06.059 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.059 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.059 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6824", 2024-05-15T13:23:06.059 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:23:06.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.059 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.060 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6825", 2024-05-15T13:23:06.060 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:23:06.060 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.060 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.060 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:23:06.060 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.060 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.060 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6822", 2024-05-15T13:23:06.060 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:23:06.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.060 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.061 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6823", 2024-05-15T13:23:06.061 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3285413897 2024-05-15T13:23:06.061 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.061 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.061 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6819/3285413897", 2024-05-15T13:23:06.061 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6821/3285413897", 2024-05-15T13:23:06.061 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6825/3285413897", 2024-05-15T13:23:06.061 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6823/3285413897", 2024-05-15T13:23:06.061 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:23:06.061 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:23:06.061 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:23:06.061 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.061 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.062 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 3, 2024-05-15T13:23:06.062 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "2e83735b-b944-4f87-8080-b994d4b96897", 2024-05-15T13:23:06.062 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:23:06.062 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:23:06.062 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:23:06.062 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:23:06.062 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:23:06.062 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:23:06.062 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 26, 2024-05-15T13:23:06.062 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 809, 2024-05-15T13:23:06.062 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:23:06.062 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:23:06.062 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:23:06.062 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.063 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.063 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6826", 2024-05-15T13:23:06.063 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:23:06.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.063 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.063 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6827", 2024-05-15T13:23:06.063 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:23:06.063 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.063 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.063 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:23:06.063 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.064 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.064 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.064 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6828", 2024-05-15T13:23:06.064 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:23:06.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.064 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.064 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.064 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6829", 2024-05-15T13:23:06.064 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:23:06.064 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.064 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.064 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:23:06.064 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.064 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.065 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.065 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6832", 2024-05-15T13:23:06.065 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:23:06.065 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.065 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.065 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.065 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6833", 2024-05-15T13:23:06.065 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:23:06.065 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.065 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.065 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.065 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:23:06.065 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.066 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.066 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.066 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6830", 2024-05-15T13:23:06.066 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:23:06.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.066 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.066 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.066 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6831", 2024-05-15T13:23:06.066 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3008975244 2024-05-15T13:23:06.066 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.066 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.066 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6827/3008975244", 2024-05-15T13:23:06.066 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6829/3008975244", 2024-05-15T13:23:06.067 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6833/3008975244", 2024-05-15T13:23:06.067 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6831/3008975244", 2024-05-15T13:23:06.067 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:23:06.067 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:23:06.067 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:23:06.067 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.067 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.067 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 4, 2024-05-15T13:23:06.067 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "7c39c1dd-3867-46f7-8372-7688fa13b16a", 2024-05-15T13:23:06.067 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:23:06.067 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 0, 2024-05-15T13:23:06.067 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 0, 2024-05-15T13:23:06.068 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:23:06.068 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:23:06.068 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:23:06.068 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 31, 2024-05-15T13:23:06.068 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 509, 2024-05-15T13:23:06.068 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:23:06.068 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:23:06.068 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:23:06.068 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.068 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.068 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.068 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6800", 2024-05-15T13:23:06.068 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:23:06.068 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.068 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6801", 2024-05-15T13:23:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:23:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:23:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6802", 2024-05-15T13:23:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:23:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6803", 2024-05-15T13:23:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:23:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:23:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6806", 2024-05-15T13:23:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:23:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.071 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6807", 2024-05-15T13:23:06.071 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:23:06.071 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.071 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.071 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.071 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:23:06.071 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.071 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.071 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.071 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6804", 2024-05-15T13:23:06.071 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:23:06.071 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.071 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.071 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6805", 2024-05-15T13:23:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3877842957 2024-05-15T13:23:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6801/3877842957", 2024-05-15T13:23:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6803/3877842957", 2024-05-15T13:23:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6807/3877842957", 2024-05-15T13:23:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6805/3877842957", 2024-05-15T13:23:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:23:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:23:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:23:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.073 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.073 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 5, 2024-05-15T13:23:06.073 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "a6dc220f-aec0-4454-9695-b44c0b9770e0", 2024-05-15T13:23:06.073 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:23:06.073 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:23:06.073 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:23:06.073 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:23:06.073 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:23:06.073 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:23:06.073 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 40, 2024-05-15T13:23:06.073 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 809, 2024-05-15T13:23:06.073 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:23:06.073 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:23:06.073 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:23:06.074 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.074 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.074 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.074 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6808", 2024-05-15T13:23:06.074 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:23:06.074 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.074 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.074 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.074 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6809", 2024-05-15T13:23:06.074 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:23:06.074 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.074 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.074 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.074 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:23:06.075 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.075 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.075 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.075 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6810", 2024-05-15T13:23:06.075 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:23:06.075 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.075 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.075 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.075 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6811", 2024-05-15T13:23:06.075 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:23:06.075 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.075 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.075 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.075 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:23:06.075 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.076 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.076 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.076 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6814", 2024-05-15T13:23:06.076 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:23:06.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.076 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.076 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.076 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6815", 2024-05-15T13:23:06.076 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:23:06.076 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.076 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.076 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:23:06.076 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.076 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.077 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.077 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6812", 2024-05-15T13:23:06.077 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:23:06.077 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.077 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.077 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.077 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6813", 2024-05-15T13:23:06.077 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1600179183 2024-05-15T13:23:06.077 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.077 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.077 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.077 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6809/1600179183", 2024-05-15T13:23:06.077 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6811/1600179183", 2024-05-15T13:23:06.077 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6815/1600179183", 2024-05-15T13:23:06.077 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6813/1600179183", 2024-05-15T13:23:06.078 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:23:06.078 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:23:06.078 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:23:06.078 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.078 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.078 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.078 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 6, 2024-05-15T13:23:06.078 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b", 2024-05-15T13:23:06.078 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:23:06.078 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:23:06.078 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:23:06.078 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:23:06.078 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:23:06.078 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:23:06.079 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 48, 2024-05-15T13:23:06.079 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 717, 2024-05-15T13:23:06.079 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:23:06.079 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:23:06.079 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:23:06.079 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.079 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.079 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.079 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6816", 2024-05-15T13:23:06.079 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:23:06.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.079 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.079 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.079 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6817", 2024-05-15T13:23:06.080 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:23:06.080 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.080 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.080 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.080 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:23:06.080 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.080 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.080 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.080 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6818", 2024-05-15T13:23:06.080 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:23:06.080 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.080 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.080 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.080 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6819", 2024-05-15T13:23:06.080 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:23:06.081 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.081 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.081 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.081 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:23:06.081 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.081 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.081 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.081 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6822", 2024-05-15T13:23:06.081 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:23:06.081 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.081 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.081 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.081 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6823", 2024-05-15T13:23:06.081 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:23:06.081 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.082 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.082 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:23:06.082 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.082 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.082 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.082 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6820", 2024-05-15T13:23:06.082 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:23:06.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.082 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.082 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.082 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6821", 2024-05-15T13:23:06.082 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3323706735 2024-05-15T13:23:06.082 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.083 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.083 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.083 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6817/3323706735", 2024-05-15T13:23:06.083 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6819/3323706735", 2024-05-15T13:23:06.083 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6823/3323706735", 2024-05-15T13:23:06.083 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6821/3323706735", 2024-05-15T13:23:06.083 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:23:06.083 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:23:06.083 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:23:06.083 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.083 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.083 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.084 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 7, 2024-05-15T13:23:06.084 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "6ef085d8-f59b-4033-8280-2c8f78693e30", 2024-05-15T13:23:06.084 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:23:06.084 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:23:06.084 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:23:06.084 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:23:06.084 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:23:06.084 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:23:06.084 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 56, 2024-05-15T13:23:06.084 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 809, 2024-05-15T13:23:06.084 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:23:06.084 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:23:06.085 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:23:06.085 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.085 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.085 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.085 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6824", 2024-05-15T13:23:06.085 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:23:06.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.085 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.085 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.085 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6825", 2024-05-15T13:23:06.085 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:23:06.085 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.085 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.086 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:23:06.086 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.086 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.086 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.086 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6826", 2024-05-15T13:23:06.086 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:23:06.086 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.086 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.086 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.086 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6827", 2024-05-15T13:23:06.086 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:23:06.086 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.086 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.086 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.087 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:23:06.087 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.087 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.087 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6830", 2024-05-15T13:23:06.087 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:23:06.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.087 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.087 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6831", 2024-05-15T13:23:06.087 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:23:06.087 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.087 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.088 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:23:06.088 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.088 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.088 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.088 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6828", 2024-05-15T13:23:06.088 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:23:06.088 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.088 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.088 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.088 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.120:6829", 2024-05-15T13:23:06.088 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2960616153 2024-05-15T13:23:06.088 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.088 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.088 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.088 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.120:6825/2960616153", 2024-05-15T13:23:06.089 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.120:6827/2960616153", 2024-05-15T13:23:06.089 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.120:6831/2960616153", 2024-05-15T13:23:06.089 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.120:6829/2960616153", 2024-05-15T13:23:06.089 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:23:06.089 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:23:06.089 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:23:06.089 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.089 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.089 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.089 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 8, 2024-05-15T13:23:06.089 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "26514b11-d02b-46e9-a4e8-3da193fcdbcc", 2024-05-15T13:23:06.089 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:23:06.089 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 0, 2024-05-15T13:23:06.089 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 0, 2024-05-15T13:23:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:23:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:23:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:23:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 65, 2024-05-15T13:23:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 739, 2024-05-15T13:23:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:23:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:23:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:23:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6800", 2024-05-15T13:23:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:23:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6801", 2024-05-15T13:23:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:23:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:23:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6802", 2024-05-15T13:23:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:23:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6803", 2024-05-15T13:23:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:23:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:23:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6806", 2024-05-15T13:23:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:23:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6807", 2024-05-15T13:23:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:23:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:23:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6804", 2024-05-15T13:23:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:23:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.094 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.094 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6805", 2024-05-15T13:23:06.094 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132459315 2024-05-15T13:23:06.094 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.094 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.094 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.094 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6801/1132459315", 2024-05-15T13:23:06.094 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6803/1132459315", 2024-05-15T13:23:06.094 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6807/1132459315", 2024-05-15T13:23:06.094 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6805/1132459315", 2024-05-15T13:23:06.094 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:23:06.094 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:23:06.094 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:23:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 9, 2024-05-15T13:23:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "6f9497bc-e962-409d-8a2c-0c634cc1124f", 2024-05-15T13:23:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:23:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:23:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:23:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 0.9062042236328125, 2024-05-15T13:23:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:23:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:23:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 76, 2024-05-15T13:23:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 809, 2024-05-15T13:23:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:23:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:23:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:23:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6808", 2024-05-15T13:23:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:23:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6809", 2024-05-15T13:23:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:23:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:23:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6810", 2024-05-15T13:23:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:23:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6811", 2024-05-15T13:23:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:23:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:23:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6814", 2024-05-15T13:23:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:23:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6815", 2024-05-15T13:23:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:23:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:23:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6812", 2024-05-15T13:23:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:23:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6813", 2024-05-15T13:23:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2710457646 2024-05-15T13:23:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6809/2710457646", 2024-05-15T13:23:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6811/2710457646", 2024-05-15T13:23:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6815/2710457646", 2024-05-15T13:23:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6813/2710457646", 2024-05-15T13:23:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:23:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:23:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:23:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 10, 2024-05-15T13:23:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "f852876f-c7d1-4039-90c4-5ab09dbd3e32", 2024-05-15T13:23:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:23:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:23:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:23:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 0.4520721435546875, 2024-05-15T13:23:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:23:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:23:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 81, 2024-05-15T13:23:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 809, 2024-05-15T13:23:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:23:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:23:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:23:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6816", 2024-05-15T13:23:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:23:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.102 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.102 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6817", 2024-05-15T13:23:06.102 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:23:06.102 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.102 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.102 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.102 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:23:06.102 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.102 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.102 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.102 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6818", 2024-05-15T13:23:06.102 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:23:06.102 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.103 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.103 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.103 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6819", 2024-05-15T13:23:06.103 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:23:06.103 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.103 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.103 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.103 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:23:06.103 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.103 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.103 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.103 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6822", 2024-05-15T13:23:06.103 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:23:06.103 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.104 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.104 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.104 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6823", 2024-05-15T13:23:06.104 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:23:06.104 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.104 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.104 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:23:06.104 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.104 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.104 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.104 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6820", 2024-05-15T13:23:06.104 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:23:06.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.105 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.105 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.105 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6821", 2024-05-15T13:23:06.105 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3236900263 2024-05-15T13:23:06.105 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.105 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.105 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.105 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6817/3236900263", 2024-05-15T13:23:06.105 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6819/3236900263", 2024-05-15T13:23:06.105 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6823/3236900263", 2024-05-15T13:23:06.105 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6821/3236900263", 2024-05-15T13:23:06.105 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:23:06.105 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:23:06.105 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:23:06.106 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.106 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.106 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 11, 2024-05-15T13:23:06.106 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "cccc53c5-8476-4d55-b160-4bdc5cf92902", 2024-05-15T13:23:06.106 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-05-15T13:23:06.106 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-05-15T13:23:06.106 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-05-15T13:23:06.106 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-05-15T13:23:06.106 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-05-15T13:23:06.106 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-05-15T13:23:06.106 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 90, 2024-05-15T13:23:06.106 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 809, 2024-05-15T13:23:06.106 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-05-15T13:23:06.107 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-05-15T13:23:06.107 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-05-15T13:23:06.107 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.107 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.107 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.107 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6824", 2024-05-15T13:23:06.107 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:23:06.107 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.107 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.107 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.107 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6825", 2024-05-15T13:23:06.107 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:23:06.107 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.107 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.107 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.108 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-05-15T13:23:06.108 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.108 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.108 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.108 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6826", 2024-05-15T13:23:06.108 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:23:06.108 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.108 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.108 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.108 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6827", 2024-05-15T13:23:06.108 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:23:06.108 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.108 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.108 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.109 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-05-15T13:23:06.109 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.109 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.109 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.109 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6830", 2024-05-15T13:23:06.109 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:23:06.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.109 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.109 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.109 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6831", 2024-05-15T13:23:06.109 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:23:06.109 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.109 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.110 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-05-15T13:23:06.110 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-05-15T13:23:06.110 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.110 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-05-15T13:23:06.110 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6828", 2024-05-15T13:23:06.110 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:23:06.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.110 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.110 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-05-15T13:23:06.110 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.177:6829", 2024-05-15T13:23:06.110 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 681011628 2024-05-15T13:23:06.110 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.110 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.111 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.177:6825/681011628", 2024-05-15T13:23:06.111 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.177:6827/681011628", 2024-05-15T13:23:06.111 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.177:6831/681011628", 2024-05-15T13:23:06.111 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.177:6829/681011628", 2024-05-15T13:23:06.111 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-05-15T13:23:06.111 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-05-15T13:23:06.111 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-05-15T13:23:06.111 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.111 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.111 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:23:06.111 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_xinfo": [ 2024-05-15T13:23:06.111 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.111 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 0, 2024-05-15T13:23:06.111 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:23:06.111 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:23:06.112 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:23:06.112 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:23:06.112 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:23:06.112 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:22:00.146190+0000", 2024-05-15T13:23:06.112 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:23:06.112 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.112 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.112 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 1, 2024-05-15T13:23:06.112 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:23:06.112 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:23:06.112 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:23:06.112 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:23:06.112 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:23:06.112 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:21:59.037448+0000", 2024-05-15T13:23:06.113 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:23:06.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.113 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.113 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 2, 2024-05-15T13:23:06.113 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:23:06.113 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:23:06.113 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:23:06.113 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:23:06.113 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:23:06.113 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:22:32.063751+0000", 2024-05-15T13:23:06.113 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:23:06.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.113 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.113 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 3, 2024-05-15T13:23:06.113 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:23:06.114 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:23:06.114 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:23:06.114 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:23:06.114 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:23:06.114 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:21:57.570001+0000", 2024-05-15T13:23:06.114 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:23:06.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.114 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.114 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 4, 2024-05-15T13:23:06.114 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:23:06.114 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:23:06.114 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:23:06.114 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:23:06.114 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 65536, 2024-05-15T13:23:06.115 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:22:26.141534+0000", 2024-05-15T13:23:06.115 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:23:06.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.115 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 5, 2024-05-15T13:23:06.115 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:23:06.115 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:23:06.115 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:23:06.115 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:23:06.115 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:23:06.115 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:22:36.465809+0000", 2024-05-15T13:23:06.115 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:23:06.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.115 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 6, 2024-05-15T13:23:06.116 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:23:06.116 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:23:06.116 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:23:06.116 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:23:06.116 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:23:06.116 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:22:52.866996+0000", 2024-05-15T13:23:06.116 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:23:06.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.116 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.116 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 7, 2024-05-15T13:23:06.116 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:23:06.116 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:23:06.116 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:23:06.116 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:23:06.117 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:23:06.117 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:22:30.559104+0000", 2024-05-15T13:23:06.117 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:23:06.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.117 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 8, 2024-05-15T13:23:06.117 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:23:06.117 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:23:06.117 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:23:06.117 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:23:06.117 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 65536, 2024-05-15T13:23:06.117 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:22:57.999765+0000", 2024-05-15T13:23:06.117 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:23:06.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.118 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 9, 2024-05-15T13:23:06.118 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:23:06.118 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:23:06.118 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:23:06.118 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:23:06.118 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:23:06.118 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:21:58.967676+0000", 2024-05-15T13:23:06.118 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:23:06.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.118 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.118 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 10, 2024-05-15T13:23:06.118 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:23:06.118 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:23:06.118 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:23:06.119 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:23:06.119 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:23:06.119 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:23:01.971114+0000", 2024-05-15T13:23:06.119 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:23:06.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.119 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.119 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 11, 2024-05-15T13:23:06.119 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-05-15T13:23:06.119 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-05-15T13:23:06.119 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-05-15T13:23:06.119 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540701547738038271, 2024-05-15T13:23:06.119 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-05-15T13:23:06.119 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-05-15T13:22:29.964222+0000", 2024-05-15T13:23:06.119 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-05-15T13:23:06.120 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.120 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:23:06.120 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap": [ 2024-05-15T13:23:06.120 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:06.120 INFO:teuthology.orchestra.run.smithi027.stdout: "pgid": "1.0", 2024-05-15T13:23:06.120 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-05-15T13:23:06.120 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-05-15T13:23:06.120 INFO:teuthology.orchestra.run.smithi027.stdout: 6, 2024-05-15T13:23:06.120 INFO:teuthology.orchestra.run.smithi027.stdout: 10 2024-05-15T13:23:06.120 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:06.120 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.120 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:23:06.120 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap_items": [], 2024-05-15T13:23:06.120 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap_primaries": [], 2024-05-15T13:23:06.120 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_temp": [], 2024-05-15T13:23:06.121 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_temp": [], 2024-05-15T13:23:06.121 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklist": { 2024-05-15T13:23:06.121 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6801/2530617473": "2024-05-16T12:39:19.409195+0000", 2024-05-15T13:23:06.121 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6800/2530617473": "2024-05-16T12:39:19.409195+0000", 2024-05-15T13:23:06.121 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/1880261204": "2024-05-16T12:39:19.409195+0000", 2024-05-15T13:23:06.121 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6801/3441136471": "2024-05-16T12:38:50.931011+0000", 2024-05-15T13:23:06.121 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/4291446247": "2024-05-16T12:39:19.409195+0000", 2024-05-15T13:23:06.121 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/580128209": "2024-05-16T12:38:50.931011+0000", 2024-05-15T13:23:06.121 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6800/3441136471": "2024-05-16T12:38:50.931011+0000", 2024-05-15T13:23:06.121 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/3011229857": "2024-05-16T12:38:50.931011+0000", 2024-05-15T13:23:06.121 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/2109693913": "2024-05-16T12:38:50.931011+0000", 2024-05-15T13:23:06.121 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/2445663024": "2024-05-16T12:39:19.409195+0000" 2024-05-15T13:23:06.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.121 INFO:teuthology.orchestra.run.smithi027.stdout: "range_blocklist": {}, 2024-05-15T13:23:06.121 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profiles": { 2024-05-15T13:23:06.122 INFO:teuthology.orchestra.run.smithi027.stdout: "default": { 2024-05-15T13:23:06.122 INFO:teuthology.orchestra.run.smithi027.stdout: "crush-failure-domain": "osd", 2024-05-15T13:23:06.122 INFO:teuthology.orchestra.run.smithi027.stdout: "k": "2", 2024-05-15T13:23:06.122 INFO:teuthology.orchestra.run.smithi027.stdout: "m": "1", 2024-05-15T13:23:06.122 INFO:teuthology.orchestra.run.smithi027.stdout: "plugin": "jerasure", 2024-05-15T13:23:06.122 INFO:teuthology.orchestra.run.smithi027.stdout: "technique": "reed_sol_van" 2024-05-15T13:23:06.122 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:06.122 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps_queue": [], 2024-05-15T13:23:06.122 INFO:teuthology.orchestra.run.smithi027.stdout: "new_removed_snaps": [], 2024-05-15T13:23:06.122 INFO:teuthology.orchestra.run.smithi027.stdout: "new_purged_snaps": [], 2024-05-15T13:23:06.122 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_node_flags": {}, 2024-05-15T13:23:06.122 INFO:teuthology.orchestra.run.smithi027.stdout: "device_class_flags": {}, 2024-05-15T13:23:06.122 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode": { 2024-05-15T13:23:06.123 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode_enabled": false, 2024-05-15T13:23:06.123 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_bucket_count": 0, 2024-05-15T13:23:06.123 INFO:teuthology.orchestra.run.smithi027.stdout: "degraded_stretch_mode": 0, 2024-05-15T13:23:06.123 INFO:teuthology.orchestra.run.smithi027.stdout: "recovering_stretch_mode": 0, 2024-05-15T13:23:06.123 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode_bucket": 0 2024-05-15T13:23:06.123 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:06.123 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:23:06.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:05 smithi177 ceph-mon[44303]: pgmap v2114: 37 pgs: 37 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:23:06.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:05 smithi177 ceph-mon[44303]: 1.4 scrub starts 2024-05-15T13:23:06.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:05 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/894190942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:23:06.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:05 smithi177 ceph-mon[44303]: 1.4 scrub ok 2024-05-15T13:23:06.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:05 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/936460396' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T13:23:06.207 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:23:06.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:06.211 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:05 smithi027 ceph-mon[39795]: pgmap v2114: 37 pgs: 37 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:23:06.211 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:05 smithi027 ceph-mon[39795]: 1.4 scrub starts 2024-05-15T13:23:06.211 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:05 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/894190942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:23:06.211 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:05 smithi027 ceph-mon[39795]: 1.4 scrub ok 2024-05-15T13:23:06.211 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:05 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/936460396' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T13:23:06.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:23:06.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:05 smithi120 ceph-mon[43961]: pgmap v2114: 37 pgs: 37 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:23:06.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:05 smithi120 ceph-mon[43961]: 1.4 scrub starts 2024-05-15T13:23:06.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:05 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/894190942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:23:06.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:05 smithi120 ceph-mon[43961]: 1.4 scrub ok 2024-05-15T13:23:06.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:05 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/936460396' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T13:23:06.313 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:23:06.342 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:06.641 INFO:tasks.thrashosds.thrasher:j is {'epoch': 812, 'fsid': 'deaea04e-12b7-11ef-bc98-c7b262605968', 'created': '2024-05-15T12:38:21.244556+0000', 'modified': '2024-05-15T13:23:02.855133+0000', 'last_up_change': '2024-05-15T12:45:50.876090+0000', 'last_in_change': '2024-05-15T13:22:44.721594+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-15T12:41:57.948613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '753', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '614', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2899999618530273, 'score_stable': 2.2899999618530273, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.899999976158142, 'raw_score_stable': 1.899999976158142, 'primary_affinity_weighted': 0.9465229511260986, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 9, 'pool_name': 'unique_pool_7', 'create_time': '2024-05-15T13:22:57.418245+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '811', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 0.75, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9465229511260986, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': '64fb3ad2-08a7-4106-a07a-6c776c877faf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 805, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6802', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6803', 'nonce': 4239116935}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6804', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6805', 'nonce': 4239116935}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6808', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6809', 'nonce': 4239116935}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6806', 'nonce': 4239116935}, {'type': 'v1', 'addr': '172.21.15.27:6807', 'nonce': 4239116935}]}, 'public_addr': '172.21.15.27:6803/4239116935', 'cluster_addr': '172.21.15.27:6805/4239116935', 'heartbeat_back_addr': '172.21.15.27:6809/4239116935', 'heartbeat_front_addr': '172.21.15.27:6807/4239116935', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '9c16dd90-9d51-48b8-a296-1c8879ba1bc8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 809, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6810', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6811', 'nonce': 1466270637}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6812', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6813', 'nonce': 1466270637}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6816', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6817', 'nonce': 1466270637}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6814', 'nonce': 1466270637}, {'type': 'v1', 'addr': '172.21.15.27:6815', 'nonce': 1466270637}]}, 'public_addr': '172.21.15.27:6811/1466270637', 'cluster_addr': '172.21.15.27:6813/1466270637', 'heartbeat_back_addr': '172.21.15.27:6817/1466270637', 'heartbeat_front_addr': '172.21.15.27:6815/1466270637', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'cbe3ba13-3ea3-4891-8d2d-6b85581a732c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 803, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6818', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6819', 'nonce': 3285413897}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6820', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6821', 'nonce': 3285413897}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6824', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6825', 'nonce': 3285413897}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6822', 'nonce': 3285413897}, {'type': 'v1', 'addr': '172.21.15.27:6823', 'nonce': 3285413897}]}, 'public_addr': '172.21.15.27:6819/3285413897', 'cluster_addr': '172.21.15.27:6821/3285413897', 'heartbeat_back_addr': '172.21.15.27:6825/3285413897', 'heartbeat_front_addr': '172.21.15.27:6823/3285413897', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2e83735b-b944-4f87-8080-b994d4b96897', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 809, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6826', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6827', 'nonce': 3008975244}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6828', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6829', 'nonce': 3008975244}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6833', 'nonce': 3008975244}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6830', 'nonce': 3008975244}, {'type': 'v1', 'addr': '172.21.15.27:6831', 'nonce': 3008975244}]}, 'public_addr': '172.21.15.27:6827/3008975244', 'cluster_addr': '172.21.15.27:6829/3008975244', 'heartbeat_back_addr': '172.21.15.27:6833/3008975244', 'heartbeat_front_addr': '172.21.15.27:6831/3008975244', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7c39c1dd-3867-46f7-8372-7688fa13b16a', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 509, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6800', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6801', 'nonce': 3877842957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6802', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6803', 'nonce': 3877842957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6806', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6807', 'nonce': 3877842957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6804', 'nonce': 3877842957}, {'type': 'v1', 'addr': '172.21.15.120:6805', 'nonce': 3877842957}]}, 'public_addr': '172.21.15.120:6801/3877842957', 'cluster_addr': '172.21.15.120:6803/3877842957', 'heartbeat_back_addr': '172.21.15.120:6807/3877842957', 'heartbeat_front_addr': '172.21.15.120:6805/3877842957', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a6dc220f-aec0-4454-9695-b44c0b9770e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 809, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6808', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6809', 'nonce': 1600179183}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6810', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6811', 'nonce': 1600179183}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6814', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6815', 'nonce': 1600179183}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6812', 'nonce': 1600179183}, {'type': 'v1', 'addr': '172.21.15.120:6813', 'nonce': 1600179183}]}, 'public_addr': '172.21.15.120:6809/1600179183', 'cluster_addr': '172.21.15.120:6811/1600179183', 'heartbeat_back_addr': '172.21.15.120:6815/1600179183', 'heartbeat_front_addr': '172.21.15.120:6813/1600179183', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 717, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6816', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6817', 'nonce': 3323706735}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6818', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6819', 'nonce': 3323706735}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6822', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6823', 'nonce': 3323706735}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6820', 'nonce': 3323706735}, {'type': 'v1', 'addr': '172.21.15.120:6821', 'nonce': 3323706735}]}, 'public_addr': '172.21.15.120:6817/3323706735', 'cluster_addr': '172.21.15.120:6819/3323706735', 'heartbeat_back_addr': '172.21.15.120:6823/3323706735', 'heartbeat_front_addr': '172.21.15.120:6821/3323706735', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6ef085d8-f59b-4033-8280-2c8f78693e30', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 809, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6824', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6825', 'nonce': 2960616153}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6826', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6827', 'nonce': 2960616153}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6830', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6831', 'nonce': 2960616153}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6828', 'nonce': 2960616153}, {'type': 'v1', 'addr': '172.21.15.120:6829', 'nonce': 2960616153}]}, 'public_addr': '172.21.15.120:6825/2960616153', 'cluster_addr': '172.21.15.120:6827/2960616153', 'heartbeat_back_addr': '172.21.15.120:6831/2960616153', 'heartbeat_front_addr': '172.21.15.120:6829/2960616153', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '26514b11-d02b-46e9-a4e8-3da193fcdbcc', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 739, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6800', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6801', 'nonce': 1132459315}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6802', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6803', 'nonce': 1132459315}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6806', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6807', 'nonce': 1132459315}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6804', 'nonce': 1132459315}, {'type': 'v1', 'addr': '172.21.15.177:6805', 'nonce': 1132459315}]}, 'public_addr': '172.21.15.177:6801/1132459315', 'cluster_addr': '172.21.15.177:6803/1132459315', 'heartbeat_back_addr': '172.21.15.177:6807/1132459315', 'heartbeat_front_addr': '172.21.15.177:6805/1132459315', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '6f9497bc-e962-409d-8a2c-0c634cc1124f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9062042236328125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 809, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6808', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6809', 'nonce': 2710457646}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6810', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6811', 'nonce': 2710457646}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6814', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6815', 'nonce': 2710457646}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6812', 'nonce': 2710457646}, {'type': 'v1', 'addr': '172.21.15.177:6813', 'nonce': 2710457646}]}, 'public_addr': '172.21.15.177:6809/2710457646', 'cluster_addr': '172.21.15.177:6811/2710457646', 'heartbeat_back_addr': '172.21.15.177:6815/2710457646', 'heartbeat_front_addr': '172.21.15.177:6813/2710457646', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'f852876f-c7d1-4039-90c4-5ab09dbd3e32', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.4520721435546875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 81, 'up_thru': 809, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6816', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6817', 'nonce': 3236900263}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6818', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6819', 'nonce': 3236900263}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6822', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6823', 'nonce': 3236900263}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6820', 'nonce': 3236900263}, {'type': 'v1', 'addr': '172.21.15.177:6821', 'nonce': 3236900263}]}, 'public_addr': '172.21.15.177:6817/3236900263', 'cluster_addr': '172.21.15.177:6819/3236900263', 'heartbeat_back_addr': '172.21.15.177:6823/3236900263', 'heartbeat_front_addr': '172.21.15.177:6821/3236900263', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'cccc53c5-8476-4d55-b160-4bdc5cf92902', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 90, 'up_thru': 809, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6824', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6825', 'nonce': 681011628}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6826', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6827', 'nonce': 681011628}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6830', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6831', 'nonce': 681011628}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6828', 'nonce': 681011628}, {'type': 'v1', 'addr': '172.21.15.177:6829', 'nonce': 681011628}]}, 'public_addr': '172.21.15.177:6825/681011628', 'cluster_addr': '172.21.15.177:6827/681011628', 'heartbeat_back_addr': '172.21.15.177:6831/681011628', 'heartbeat_front_addr': '172.21.15.177:6829/681011628', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:22:00.146190+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:21:59.037448+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:22:32.063751+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:21:57.570001+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-15T13:22:26.141534+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:22:36.465809+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:22:52.866996+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:22:30.559104+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-15T13:22:57.999765+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:21:58.967676+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:23:01.971114+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T13:22:29.964222+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [1, 6, 10]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.27:6801/2530617473': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:6800/2530617473': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:0/1880261204': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:6801/3441136471': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/4291446247': '2024-05-16T12:39:19.409195+0000', '172.21.15.27:0/580128209': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:6800/3441136471': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/3011229857': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/2109693913': '2024-05-16T12:38:50.931011+0000', '172.21.15.27:0/2445663024': '2024-05-16T12:39:19.409195+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-05-15T13:23:06.641 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:23:07.541 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:23:07.581 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:23:07.792 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:23:07.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:07.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:23:07.896 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:23:07.926 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:07.946 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:23:07.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:07.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:23:08.026 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:23:08.050 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:23:08.053 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:07 smithi027 ceph-mon[39795]: pgmap v2115: 37 pgs: 37 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:23:08.053 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:07 smithi027 ceph-mon[39795]: 1.e deep-scrub starts 2024-05-15T13:23:08.053 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:07 smithi027 ceph-mon[39795]: 1.e deep-scrub ok 2024-05-15T13:23:08.083 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:08.104 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:23:08.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:08.107 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:23:08.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:07 smithi177 ceph-mon[44303]: pgmap v2115: 37 pgs: 37 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:23:08.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:07 smithi177 ceph-mon[44303]: 1.e deep-scrub starts 2024-05-15T13:23:08.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:07 smithi177 ceph-mon[44303]: 1.e deep-scrub ok 2024-05-15T13:23:08.209 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:23:08.238 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:08.259 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:23:08.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:08.262 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:23:08.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:07 smithi120 ceph-mon[43961]: pgmap v2115: 37 pgs: 37 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:23:08.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:07 smithi120 ceph-mon[43961]: 1.e deep-scrub starts 2024-05-15T13:23:08.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:07 smithi120 ceph-mon[43961]: 1.e deep-scrub ok 2024-05-15T13:23:08.363 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:23:08.392 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:08.412 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:23:08.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:08.415 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:23:08.516 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:23:08.550 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:09.136 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:23:09.163 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:08 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1377137174' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T13:23:09.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:08 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1377137174' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T13:23:09.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:08 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1377137174' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T13:23:09.377 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rados' on pool 'unique_pool_7' 2024-05-15T13:23:10.093 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:09 smithi027 ceph-mon[39795]: pgmap v2116: 37 pgs: 37 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:23:10.093 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:09 smithi027 ceph-mon[39795]: 1.12 scrub starts 2024-05-15T13:23:10.093 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:09 smithi027 ceph-mon[39795]: 1.12 scrub ok 2024-05-15T13:23:10.093 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:09 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1377137174' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T13:23:10.093 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:09 smithi027 ceph-mon[39795]: osdmap e813: 12 total, 12 up, 10 in 2024-05-15T13:23:10.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:09 smithi177 ceph-mon[44303]: pgmap v2116: 37 pgs: 37 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:23:10.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:09 smithi177 ceph-mon[44303]: 1.12 scrub starts 2024-05-15T13:23:10.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:09 smithi177 ceph-mon[44303]: 1.12 scrub ok 2024-05-15T13:23:10.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:09 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1377137174' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T13:23:10.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:09 smithi177 ceph-mon[44303]: osdmap e813: 12 total, 12 up, 10 in 2024-05-15T13:23:10.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:09 smithi120 ceph-mon[43961]: pgmap v2116: 37 pgs: 37 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:23:10.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:09 smithi120 ceph-mon[43961]: 1.12 scrub starts 2024-05-15T13:23:10.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:09 smithi120 ceph-mon[43961]: 1.12 scrub ok 2024-05-15T13:23:10.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:09 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1377137174' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T13:23:10.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:09 smithi120 ceph-mon[43961]: osdmap e813: 12 total, 12 up, 10 in 2024-05-15T13:23:10.529 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:23:10.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:10.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:23:10.636 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:23:10.668 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:10.690 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:23:10.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:10.694 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:23:10.795 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:23:10.827 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:10.926 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:23:10.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:10.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:23:11.031 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:23:11.075 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:11.096 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:23:11.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:11.099 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:23:11.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:10 smithi177 ceph-mon[44303]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T13:23:11.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:10 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T13:23:11.201 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:23:11.212 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:10 smithi027 ceph-mon[39795]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T13:23:11.212 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:10 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T13:23:11.231 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:10 smithi120 ceph-mon[43961]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T13:23:11.231 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:10 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T13:23:11.233 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:11.254 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:23:11.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:11.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:23:11.358 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:23:11.388 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:11.637 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:23:11.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:11.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:23:11.744 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:23:11.752 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:23:11.753 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T13:23:11.753 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-05-15T13:23:11.753 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:23:11.776 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:11.797 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:23:11.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:11.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:23:11.902 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:23:11.931 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:11.953 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:23:11.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:11.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:23:12.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:11 smithi120 ceph-mon[43961]: pgmap v2118: 37 pgs: 37 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:23:12.057 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:23:12.085 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:12.104 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:23:12.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:12.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:23:12.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:11 smithi177 ceph-mon[44303]: pgmap v2118: 37 pgs: 37 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:23:12.208 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:23:12.211 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:11 smithi027 ceph-mon[39795]: pgmap v2118: 37 pgs: 37 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:23:12.236 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:12.241 DEBUG:teuthology.orchestra.run.smithi177:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_7 bench 90 write' 2024-05-15T13:23:12.244 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-05-15T13:23:12.244 INFO:teuthology.orchestra.run:waiting for 3000 2024-05-15T13:23:12.257 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:23:12.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:12.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:23:12.294 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:hints = 1 2024-05-15T13:23:12.294 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-05-15T13:23:12.294 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Object prefix: benchmark_data_smithi177_159919 2024-05-15T13:23:12.294 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:23:12.294 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 0 0 0 0 0 0 - 0 2024-05-15T13:23:12.361 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:23:12.398 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:12.422 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:23:12.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:12.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:23:12.526 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:23:12.546 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-05-15T13:23:12.552 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:12.582 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:23:12.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:12.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:23:12.689 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:23:12.745 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:12.798 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:23:12.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:12.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:23:12.906 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:23:12.934 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:12.954 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:23:12.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:12.957 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:23:13.058 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:23:13.101 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:13.122 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:23:13.125 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:23:13.128 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:23:13.129 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:23:13.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:13.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:23:13.161 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:12 smithi027 ceph-mon[39795]: 1.14 deep-scrub starts 2024-05-15T13:23:13.161 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:12 smithi027 ceph-mon[39795]: 1.14 deep-scrub ok 2024-05-15T13:23:13.161 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:12 smithi027 ceph-mon[39795]: 1.9 scrub starts 2024-05-15T13:23:13.161 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:12 smithi027 ceph-mon[39795]: 1.9 scrub ok 2024-05-15T13:23:13.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:12 smithi177 ceph-mon[44303]: 1.14 deep-scrub starts 2024-05-15T13:23:13.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:12 smithi177 ceph-mon[44303]: 1.14 deep-scrub ok 2024-05-15T13:23:13.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:12 smithi177 ceph-mon[44303]: 1.9 scrub starts 2024-05-15T13:23:13.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:12 smithi177 ceph-mon[44303]: 1.9 scrub ok 2024-05-15T13:23:13.234 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:23:13.275 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:13.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:12 smithi120 ceph-mon[43961]: 1.14 deep-scrub starts 2024-05-15T13:23:13.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:12 smithi120 ceph-mon[43961]: 1.14 deep-scrub ok 2024-05-15T13:23:13.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:12 smithi120 ceph-mon[43961]: 1.9 scrub starts 2024-05-15T13:23:13.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:12 smithi120 ceph-mon[43961]: 1.9 scrub ok 2024-05-15T13:23:13.294 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 1 15 3056 3041 189.996 190.062 0.0026777 0.00524966 2024-05-15T13:23:13.311 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:23:13.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:13.315 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:23:13.416 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:23:13.456 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:13.487 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:23:13.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:13.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:23:13.591 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:23:13.644 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:13.937 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:13 smithi027 ceph-mon[39795]: pgmap v2119: 37 pgs: 37 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:23:13.937 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:13 smithi027 ceph-mon[39795]: from='client.16626 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:23:14.097 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:23:14.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:14.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:23:14.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:13 smithi177 ceph-mon[44303]: pgmap v2119: 37 pgs: 37 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:23:14.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:13 smithi177 ceph-mon[44303]: from='client.16626 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:23:14.203 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:23:14.254 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:14.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:13 smithi120 ceph-mon[43961]: pgmap v2119: 37 pgs: 37 active+clean; 577 KiB data, 7.4 GiB used, 887 GiB / 894 GiB avail 2024-05-15T13:23:14.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:13 smithi120 ceph-mon[43961]: from='client.16626 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:23:14.294 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 2 16 7378 7362 230.012 270.062 0.00283119 0.0043407 2024-05-15T13:23:14.894 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:23:14.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:14.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:23:15.002 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:23:15.049 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:23:15.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:15.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:23:15.216 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:23:15.260 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:15.288 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:23:15.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:15.293 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:23:15.294 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 3 15 11348 11333 236.063 248.188 0.00631593 0.00423039 2024-05-15T13:23:15.393 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:23:15.432 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:15.463 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:23:15.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:15.467 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:23:15.568 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:23:15.617 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:15.661 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:23:15.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:15.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:23:15.768 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:23:15.811 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:15.837 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:23:15.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:15.842 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:23:15.943 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:23:15.950 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:15 smithi027 ceph-mon[39795]: pgmap v2120: 37 pgs: 37 active+clean; 123 MiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 12 MiB/s wr, 195 op/s 2024-05-15T13:23:15.950 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:15 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2773606742' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:23:15.950 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:15 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:23:15.963 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:23:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:16.057 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2119,"stamp":"2024-05-15T13:23:12.670172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":112,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":7753656,"kb_used_data":135236,"kb_used_omap":15,"kb_used_meta":7618288,"kb_avail":929697864,"statfs":{"total":959950356480,"available":952010612736,"internally_reserved":0,"allocated":138481664,"data_stored":135333723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15951,"internal_metadata":7801127345},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.807135"},"pg_stats":[{"pgid":"1.14","version":"94'139","reported_seq":984,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-05-15T13:23:02.865276+0000","last_change":"2024-05-15T13:21:55.427877+0000","last_active":"2024-05-15T13:23:02.865276+0000","last_peered":"2024-05-15T13:23:02.865276+0000","last_clean":"2024-05-15T13:23:02.865276+0000","last_became_active":"2024-05-15T13:19:41.190796+0000","last_became_peered":"2024-05-15T13:19:41.190796+0000","last_unstale":"2024-05-15T13:23:02.865276+0000","last_undegraded":"2024-05-15T13:23:02.865276+0000","last_fullsized":"2024-05-15T13:23:02.865276+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:21:55.427756+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:21:55.427756+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:23:08.928496+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":18,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-05-15T13:23:09.119603+0000","last_change":"2024-05-15T13:22:58.857344+0000","last_active":"2024-05-15T13:23:09.119603+0000","last_peered":"2024-05-15T13:23:09.119603+0000","last_clean":"2024-05-15T13:23:09.119603+0000","last_became_active":"2024-05-15T13:22:58.857024+0000","last_became_peered":"2024-05-15T13:22:58.857024+0000","last_unstale":"2024-05-15T13:23:09.119603+0000","last_undegraded":"2024-05-15T13:23:09.119603+0000","last_fullsized":"2024-05-15T13:23:09.119603+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:06.225828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":1550,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-05-15T13:23:02.865914+0000","last_change":"2024-05-15T13:22:17.099939+0000","last_active":"2024-05-15T13:23:02.865914+0000","last_peered":"2024-05-15T13:23:02.865914+0000","last_clean":"2024-05-15T13:23:02.865914+0000","last_became_active":"2024-05-15T13:13:11.361366+0000","last_became_peered":"2024-05-15T13:13:11.361366+0000","last_unstale":"2024-05-15T13:23:02.865914+0000","last_undegraded":"2024-05-15T13:23:02.865914+0000","last_fullsized":"2024-05-15T13:23:02.865914+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:17.099775+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:22:17.099775+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:23:35.922403+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":15,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-05-15T13:23:02.865728+0000","last_change":"2024-05-15T13:22:59.336651+0000","last_active":"2024-05-15T13:23:02.865728+0000","last_peered":"2024-05-15T13:23:02.865728+0000","last_clean":"2024-05-15T13:23:02.865728+0000","last_became_active":"2024-05-15T13:22:59.336347+0000","last_became_peered":"2024-05-15T13:22:59.336347+0000","last_unstale":"2024-05-15T13:23:02.865728+0000","last_undegraded":"2024-05-15T13:23:02.865728+0000","last_fullsized":"2024-05-15T13:23:02.865728+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:09.804568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,0],"acting":[3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"94'139","reported_seq":1576,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-05-15T13:23:04.931417+0000","last_change":"2024-05-15T13:23:04.931444+0000","last_active":"2024-05-15T13:23:04.931417+0000","last_peered":"2024-05-15T13:23:04.931417+0000","last_clean":"2024-05-15T13:23:04.931417+0000","last_became_active":"2024-05-15T13:22:48.740313+0000","last_became_peered":"2024-05-15T13:22:48.740313+0000","last_unstale":"2024-05-15T13:23:04.931417+0000","last_undegraded":"2024-05-15T13:23:04.931417+0000","last_fullsized":"2024-05-15T13:23:04.931417+0000","mapping_epoch":805,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":806,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:04.931319+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:23:04.931319+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:12.390796+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":18,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-05-15T13:23:08.876786+0000","last_change":"2024-05-15T13:22:58.856881+0000","last_active":"2024-05-15T13:23:08.876786+0000","last_peered":"2024-05-15T13:23:08.876786+0000","last_clean":"2024-05-15T13:23:08.876786+0000","last_became_active":"2024-05-15T13:22:58.856595+0000","last_became_peered":"2024-05-15T13:22:58.856595+0000","last_unstale":"2024-05-15T13:23:08.876786+0000","last_undegraded":"2024-05-15T13:23:08.876786+0000","last_fullsized":"2024-05-15T13:23:08.876786+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:09.943761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"94'139","reported_seq":1314,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-05-15T13:23:09.119857+0000","last_change":"2024-05-15T13:22:51.887150+0000","last_active":"2024-05-15T13:23:09.119857+0000","last_peered":"2024-05-15T13:23:09.119857+0000","last_clean":"2024-05-15T13:23:09.119857+0000","last_became_active":"2024-05-15T13:14:54.577458+0000","last_became_peered":"2024-05-15T13:14:54.577458+0000","last_unstale":"2024-05-15T13:23:09.119857+0000","last_undegraded":"2024-05-15T13:23:09.119857+0000","last_fullsized":"2024-05-15T13:23:09.119857+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:51.887016+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:18:24.970285+0000","last_clean_scrub_stamp":"2024-05-15T13:22:51.887016+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:12.953549+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":18,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-05-15T13:23:08.876944+0000","last_change":"2024-05-15T13:22:58.852680+0000","last_active":"2024-05-15T13:23:08.876944+0000","last_peered":"2024-05-15T13:23:08.876944+0000","last_clean":"2024-05-15T13:23:08.876944+0000","last_became_active":"2024-05-15T13:22:58.852354+0000","last_became_peered":"2024-05-15T13:22:58.852354+0000","last_unstale":"2024-05-15T13:23:08.876944+0000","last_undegraded":"2024-05-15T13:23:08.876944+0000","last_fullsized":"2024-05-15T13:23:08.876944+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:07.754518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,1],"acting":[7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"94'139","reported_seq":1562,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-05-15T13:23:09.119610+0000","last_change":"2024-05-15T13:22:48.741009+0000","last_active":"2024-05-15T13:23:09.119610+0000","last_peered":"2024-05-15T13:23:09.119610+0000","last_clean":"2024-05-15T13:23:09.119610+0000","last_became_active":"2024-05-15T13:22:48.740661+0000","last_became_peered":"2024-05-15T13:22:48.740661+0000","last_unstale":"2024-05-15T13:23:09.119610+0000","last_undegraded":"2024-05-15T13:23:09.119610+0000","last_fullsized":"2024-05-15T13:23:09.119610+0000","mapping_epoch":805,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":806,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:10.530048+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:22:10.530048+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:23:39.787309+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,6],"acting":[9,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":15,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-05-15T13:23:04.116081+0000","last_change":"2024-05-15T13:22:58.857932+0000","last_active":"2024-05-15T13:23:04.116081+0000","last_peered":"2024-05-15T13:23:04.116081+0000","last_clean":"2024-05-15T13:23:04.116081+0000","last_became_active":"2024-05-15T13:22:58.857554+0000","last_became_peered":"2024-05-15T13:22:58.857554+0000","last_unstale":"2024-05-15T13:23:04.116081+0000","last_undegraded":"2024-05-15T13:23:04.116081+0000","last_fullsized":"2024-05-15T13:23:04.116081+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:14.662053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"94'139","reported_seq":1311,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-05-15T13:23:08.932927+0000","last_change":"2024-05-15T13:22:01.216764+0000","last_active":"2024-05-15T13:23:08.932927+0000","last_peered":"2024-05-15T13:23:08.932927+0000","last_clean":"2024-05-15T13:23:08.932927+0000","last_became_active":"2024-05-15T13:14:11.522490+0000","last_became_peered":"2024-05-15T13:14:11.522490+0000","last_unstale":"2024-05-15T13:23:08.932927+0000","last_undegraded":"2024-05-15T13:23:08.932927+0000","last_fullsized":"2024-05-15T13:23:08.932927+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:01.216619+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:22:01.216619+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:23:20.549546+0000","scrub_duration":208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":18,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-05-15T13:23:09.119728+0000","last_change":"2024-05-15T13:22:58.856422+0000","last_active":"2024-05-15T13:23:09.119728+0000","last_peered":"2024-05-15T13:23:09.119728+0000","last_clean":"2024-05-15T13:23:09.119728+0000","last_became_active":"2024-05-15T13:22:58.855908+0000","last_became_peered":"2024-05-15T13:22:58.855908+0000","last_unstale":"2024-05-15T13:23:09.119728+0000","last_undegraded":"2024-05-15T13:23:09.119728+0000","last_fullsized":"2024-05-15T13:23:09.119728+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:25.461974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"94'139","reported_seq":1331,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-05-15T13:23:09.119831+0000","last_change":"2024-05-15T13:21:57.467175+0000","last_active":"2024-05-15T13:23:09.119831+0000","last_peered":"2024-05-15T13:23:09.119831+0000","last_clean":"2024-05-15T13:23:09.119831+0000","last_became_active":"2024-05-15T13:15:43.614970+0000","last_became_peered":"2024-05-15T13:15:43.614970+0000","last_unstale":"2024-05-15T13:23:09.119831+0000","last_undegraded":"2024-05-15T13:23:09.119831+0000","last_fullsized":"2024-05-15T13:23:09.119831+0000","mapping_epoch":678,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:21:57.467091+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:21:57.467091+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:23:19.054075+0000","scrub_duration":246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,0],"acting":[9,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":15,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-05-15T13:23:04.115903+0000","last_change":"2024-05-15T13:22:58.858222+0000","last_active":"2024-05-15T13:23:04.115903+0000","last_peered":"2024-05-15T13:23:04.115903+0000","last_clean":"2024-05-15T13:23:04.115903+0000","last_became_active":"2024-05-15T13:22:58.857879+0000","last_became_peered":"2024-05-15T13:22:58.857879+0000","last_unstale":"2024-05-15T13:23:04.115903+0000","last_undegraded":"2024-05-15T13:23:04.115903+0000","last_fullsized":"2024-05-15T13:23:04.115903+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:14.628067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"94'139","reported_seq":1576,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-05-15T13:23:08.876667+0000","last_change":"2024-05-15T13:22:57.893531+0000","last_active":"2024-05-15T13:23:08.876667+0000","last_peered":"2024-05-15T13:23:08.876667+0000","last_clean":"2024-05-15T13:23:08.876667+0000","last_became_active":"2024-05-15T13:22:48.739056+0000","last_became_peered":"2024-05-15T13:22:48.739056+0000","last_unstale":"2024-05-15T13:23:08.876667+0000","last_undegraded":"2024-05-15T13:23:08.876667+0000","last_fullsized":"2024-05-15T13:23:08.876667+0000","mapping_epoch":805,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":806,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:57.893417+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.893417+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:05.565057+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":15,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-05-15T13:23:04.116134+0000","last_change":"2024-05-15T13:22:58.858310+0000","last_active":"2024-05-15T13:23:04.116134+0000","last_peered":"2024-05-15T13:23:04.116134+0000","last_clean":"2024-05-15T13:23:04.116134+0000","last_became_active":"2024-05-15T13:22:58.857910+0000","last_became_peered":"2024-05-15T13:22:58.857910+0000","last_unstale":"2024-05-15T13:23:04.116134+0000","last_undegraded":"2024-05-15T13:23:04.116134+0000","last_fullsized":"2024-05-15T13:23:04.116134+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:24.299022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"94'139","reported_seq":1585,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-05-15T13:23:02.866162+0000","last_change":"2024-05-15T13:22:55.272017+0000","last_active":"2024-05-15T13:23:02.866162+0000","last_peered":"2024-05-15T13:23:02.866162+0000","last_clean":"2024-05-15T13:23:02.866162+0000","last_became_active":"2024-05-15T13:22:48.738778+0000","last_became_peered":"2024-05-15T13:22:48.738778+0000","last_unstale":"2024-05-15T13:23:02.866162+0000","last_undegraded":"2024-05-15T13:23:02.866162+0000","last_fullsized":"2024-05-15T13:23:02.866162+0000","mapping_epoch":805,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":806,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:55.271911+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:22:55.271911+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:10.155748+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":15,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-05-15T13:23:02.865900+0000","last_change":"2024-05-15T13:22:59.208928+0000","last_active":"2024-05-15T13:23:02.865900+0000","last_peered":"2024-05-15T13:23:02.865900+0000","last_clean":"2024-05-15T13:23:02.865900+0000","last_became_active":"2024-05-15T13:22:59.208576+0000","last_became_peered":"2024-05-15T13:22:59.208576+0000","last_unstale":"2024-05-15T13:23:02.865900+0000","last_undegraded":"2024-05-15T13:23:02.865900+0000","last_fullsized":"2024-05-15T13:23:02.865900+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:21.326850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"94'139","reported_seq":1545,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-05-15T13:23:04.116241+0000","last_change":"2024-05-15T13:21:57.620102+0000","last_active":"2024-05-15T13:23:04.116241+0000","last_peered":"2024-05-15T13:23:04.116241+0000","last_clean":"2024-05-15T13:23:04.116241+0000","last_became_active":"2024-05-15T13:06:46.426908+0000","last_became_peered":"2024-05-15T13:06:46.426908+0000","last_unstale":"2024-05-15T13:23:04.116241+0000","last_undegraded":"2024-05-15T13:23:04.116241+0000","last_fullsized":"2024-05-15T13:23:04.116241+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:21:57.619988+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:21:57.619988+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:23:21.713511+0000","scrub_duration":89,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":15,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-05-15T13:23:02.865136+0000","last_change":"2024-05-15T13:22:59.209016+0000","last_active":"2024-05-15T13:23:02.865136+0000","last_peered":"2024-05-15T13:23:02.865136+0000","last_clean":"2024-05-15T13:23:02.865136+0000","last_became_active":"2024-05-15T13:22:59.208619+0000","last_became_peered":"2024-05-15T13:22:59.208619+0000","last_unstale":"2024-05-15T13:23:02.865136+0000","last_undegraded":"2024-05-15T13:23:02.865136+0000","last_fullsized":"2024-05-15T13:23:02.865136+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:03.738999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"94'139","reported_seq":921,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-05-15T13:23:08.876652+0000","last_change":"2024-05-15T13:21:20.642230+0000","last_active":"2024-05-15T13:23:08.876652+0000","last_peered":"2024-05-15T13:23:08.876652+0000","last_clean":"2024-05-15T13:23:08.876652+0000","last_became_active":"2024-05-15T13:19:41.191244+0000","last_became_peered":"2024-05-15T13:19:41.191244+0000","last_unstale":"2024-05-15T13:23:08.876652+0000","last_undegraded":"2024-05-15T13:23:08.876652+0000","last_fullsized":"2024-05-15T13:23:08.876652+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:21:20.642108+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:21:20.642108+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":15,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-05-15T13:23:02.865599+0000","last_change":"2024-05-15T13:22:58.851932+0000","last_active":"2024-05-15T13:23:02.865599+0000","last_peered":"2024-05-15T13:23:02.865599+0000","last_clean":"2024-05-15T13:23:02.865599+0000","last_became_active":"2024-05-15T13:22:58.851656+0000","last_became_peered":"2024-05-15T13:22:58.851656+0000","last_unstale":"2024-05-15T13:23:02.865599+0000","last_undegraded":"2024-05-15T13:23:02.865599+0000","last_fullsized":"2024-05-15T13:23:02.865599+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:27.196653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"94'139","reported_seq":1326,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-05-15T13:23:09.119619+0000","last_change":"2024-05-15T13:22:03.418470+0000","last_active":"2024-05-15T13:23:09.119619+0000","last_peered":"2024-05-15T13:23:09.119619+0000","last_clean":"2024-05-15T13:23:09.119619+0000","last_became_active":"2024-05-15T13:15:43.611913+0000","last_became_peered":"2024-05-15T13:15:43.611913+0000","last_unstale":"2024-05-15T13:23:09.119619+0000","last_undegraded":"2024-05-15T13:23:09.119619+0000","last_fullsized":"2024-05-15T13:23:09.119619+0000","mapping_epoch":678,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:03.418345+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:22:03.418345+0000","last_clean_scrub_stamp":"2024-05-15T13:22:03.418345+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:23:30.712825+0000","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":15,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-05-15T13:23:04.115970+0000","last_change":"2024-05-15T13:22:58.859332+0000","last_active":"2024-05-15T13:23:04.115970+0000","last_peered":"2024-05-15T13:23:04.115970+0000","last_clean":"2024-05-15T13:23:04.115970+0000","last_became_active":"2024-05-15T13:22:58.859007+0000","last_became_peered":"2024-05-15T13:22:58.859007+0000","last_unstale":"2024-05-15T13:23:04.115970+0000","last_undegraded":"2024-05-15T13:23:04.115970+0000","last_fullsized":"2024-05-15T13:23:04.115970+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:07.367580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"94'139","reported_seq":941,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-05-15T13:23:08.876938+0000","last_change":"2024-05-15T13:23:06.819279+0000","last_active":"2024-05-15T13:23:08.876938+0000","last_peered":"2024-05-15T13:23:08.876938+0000","last_clean":"2024-05-15T13:23:08.876938+0000","last_became_active":"2024-05-15T13:22:48.738361+0000","last_became_peered":"2024-05-15T13:22:48.738361+0000","last_unstale":"2024-05-15T13:23:08.876938+0000","last_undegraded":"2024-05-15T13:23:08.876938+0000","last_fullsized":"2024-05-15T13:23:08.876938+0000","mapping_epoch":805,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":806,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:06.819156+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:23:06.819156+0000","last_clean_scrub_stamp":"2024-05-15T13:23:06.819156+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:14.267479+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":18,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-05-15T13:23:09.119985+0000","last_change":"2024-05-15T13:22:58.852908+0000","last_active":"2024-05-15T13:23:09.119985+0000","last_peered":"2024-05-15T13:23:09.119985+0000","last_clean":"2024-05-15T13:23:09.119985+0000","last_became_active":"2024-05-15T13:22:58.852401+0000","last_became_peered":"2024-05-15T13:22:58.852401+0000","last_unstale":"2024-05-15T13:23:09.119985+0000","last_undegraded":"2024-05-15T13:23:09.119985+0000","last_fullsized":"2024-05-15T13:23:09.119985+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:23.299876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"94'139","reported_seq":1310,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-05-15T13:23:08.932950+0000","last_change":"2024-05-15T13:22:00.244496+0000","last_active":"2024-05-15T13:23:08.932950+0000","last_peered":"2024-05-15T13:23:08.932950+0000","last_clean":"2024-05-15T13:23:08.932950+0000","last_became_active":"2024-05-15T13:13:16.472318+0000","last_became_peered":"2024-05-15T13:13:16.472318+0000","last_unstale":"2024-05-15T13:23:08.932950+0000","last_undegraded":"2024-05-15T13:23:08.932950+0000","last_fullsized":"2024-05-15T13:23:08.932950+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:00.244355+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:22:00.244355+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:23:28.659900+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":15,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-05-15T13:23:02.866021+0000","last_change":"2024-05-15T13:22:59.336585+0000","last_active":"2024-05-15T13:23:02.866021+0000","last_peered":"2024-05-15T13:23:02.866021+0000","last_clean":"2024-05-15T13:23:02.866021+0000","last_became_active":"2024-05-15T13:22:59.336258+0000","last_became_peered":"2024-05-15T13:22:59.336258+0000","last_unstale":"2024-05-15T13:23:02.866021+0000","last_undegraded":"2024-05-15T13:23:02.866021+0000","last_fullsized":"2024-05-15T13:23:02.866021+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:07.882852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"94'139","reported_seq":976,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-05-15T13:23:02.865681+0000","last_change":"2024-05-15T13:22:56.794557+0000","last_active":"2024-05-15T13:23:02.865681+0000","last_peered":"2024-05-15T13:23:02.865681+0000","last_clean":"2024-05-15T13:23:02.865681+0000","last_became_active":"2024-05-15T13:13:16.476972+0000","last_became_peered":"2024-05-15T13:13:16.476972+0000","last_unstale":"2024-05-15T13:23:02.865681+0000","last_undegraded":"2024-05-15T13:23:02.865681+0000","last_fullsized":"2024-05-15T13:23:02.865681+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:56.794428+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:22:56.794428+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:09.214531+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":15,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-05-15T13:23:03.791644+0000","last_change":"2024-05-15T13:22:58.856916+0000","last_active":"2024-05-15T13:23:03.791644+0000","last_peered":"2024-05-15T13:23:03.791644+0000","last_clean":"2024-05-15T13:23:03.791644+0000","last_became_active":"2024-05-15T13:22:58.856523+0000","last_became_peered":"2024-05-15T13:22:58.856523+0000","last_unstale":"2024-05-15T13:23:03.791644+0000","last_undegraded":"2024-05-15T13:23:03.791644+0000","last_fullsized":"2024-05-15T13:23:03.791644+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:24.838774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"94'139","reported_seq":978,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-05-15T13:23:08.876857+0000","last_change":"2024-05-15T13:23:03.856369+0000","last_active":"2024-05-15T13:23:08.876857+0000","last_peered":"2024-05-15T13:23:08.876857+0000","last_clean":"2024-05-15T13:23:08.876857+0000","last_became_active":"2024-05-15T13:22:48.739212+0000","last_became_peered":"2024-05-15T13:22:48.739212+0000","last_unstale":"2024-05-15T13:23:08.876857+0000","last_undegraded":"2024-05-15T13:23:08.876857+0000","last_fullsized":"2024-05-15T13:23:08.876857+0000","mapping_epoch":805,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":806,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:03.856199+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:23:03.856199+0000","last_clean_scrub_stamp":"2024-05-15T13:23:03.856199+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:28.846590+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":18,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-05-15T13:23:09.119664+0000","last_change":"2024-05-15T13:22:58.857200+0000","last_active":"2024-05-15T13:23:09.119664+0000","last_peered":"2024-05-15T13:23:09.119664+0000","last_clean":"2024-05-15T13:23:09.119664+0000","last_became_active":"2024-05-15T13:22:58.856850+0000","last_became_peered":"2024-05-15T13:22:58.856850+0000","last_unstale":"2024-05-15T13:23:09.119664+0000","last_undegraded":"2024-05-15T13:23:09.119664+0000","last_fullsized":"2024-05-15T13:23:09.119664+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:09.584566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"94'139","reported_seq":1577,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-05-15T13:23:03.791650+0000","last_change":"2024-05-15T13:22:56.003209+0000","last_active":"2024-05-15T13:23:03.791650+0000","last_peered":"2024-05-15T13:23:03.791650+0000","last_clean":"2024-05-15T13:23:03.791650+0000","last_became_active":"2024-05-15T13:13:16.484252+0000","last_became_peered":"2024-05-15T13:13:16.484252+0000","last_unstale":"2024-05-15T13:23:03.791650+0000","last_undegraded":"2024-05-15T13:23:03.791650+0000","last_fullsized":"2024-05-15T13:23:03.791650+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:56.003105+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:22:56.003105+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:03.248568+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"94'139","reported_seq":1568,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-05-15T13:23:04.115967+0000","last_change":"2024-05-15T13:22:03.596248+0000","last_active":"2024-05-15T13:23:04.115967+0000","last_peered":"2024-05-15T13:23:04.115967+0000","last_clean":"2024-05-15T13:23:04.115967+0000","last_became_active":"2024-05-15T13:19:41.190923+0000","last_became_peered":"2024-05-15T13:19:41.190923+0000","last_unstale":"2024-05-15T13:23:04.115967+0000","last_undegraded":"2024-05-15T13:23:04.115967+0000","last_fullsized":"2024-05-15T13:23:04.115967+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:03.596155+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:22:03.596155+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:23:16.872136+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"94'139","reported_seq":1336,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-05-15T13:23:09.119026+0000","last_change":"2024-05-15T13:22:48.740097+0000","last_active":"2024-05-15T13:23:09.119026+0000","last_peered":"2024-05-15T13:23:09.119026+0000","last_clean":"2024-05-15T13:23:09.119026+0000","last_became_active":"2024-05-15T13:22:48.739476+0000","last_became_peered":"2024-05-15T13:22:48.739476+0000","last_unstale":"2024-05-15T13:23:09.119026+0000","last_undegraded":"2024-05-15T13:23:09.119026+0000","last_fullsized":"2024-05-15T13:23:09.119026+0000","mapping_epoch":805,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":806,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:21:58.287523+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:21:58.287523+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:23:23.394307+0000","scrub_duration":281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,3],"acting":[0,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"94'139","reported_seq":1575,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-05-15T13:23:04.116004+0000","last_change":"2024-05-15T13:21:55.646433+0000","last_active":"2024-05-15T13:23:04.116004+0000","last_peered":"2024-05-15T13:23:04.116004+0000","last_clean":"2024-05-15T13:23:04.116004+0000","last_became_active":"2024-05-15T13:13:49.926687+0000","last_became_peered":"2024-05-15T13:13:49.926687+0000","last_unstale":"2024-05-15T13:23:04.116004+0000","last_undegraded":"2024-05-15T13:23:04.116004+0000","last_fullsized":"2024-05-15T13:23:04.116004+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:21:55.646304+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:18:06.340555+0000","last_clean_scrub_stamp":"2024-05-15T13:21:55.646304+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:23:07.903697+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"94'139","reported_seq":1575,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-05-15T13:23:02.866070+0000","last_change":"2024-05-15T13:22:21.308402+0000","last_active":"2024-05-15T13:23:02.866070+0000","last_peered":"2024-05-15T13:23:02.866070+0000","last_clean":"2024-05-15T13:23:02.866070+0000","last_became_active":"2024-05-15T13:13:16.478488+0000","last_became_peered":"2024-05-15T13:13:16.478488+0000","last_unstale":"2024-05-15T13:23:02.866070+0000","last_undegraded":"2024-05-15T13:23:02.866070+0000","last_fullsized":"2024-05-15T13:23:02.866070+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:21.308290+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:22:21.308290+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:23:32.509388+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":9},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057093,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":883744,"kb_used_data":126488,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92861408,"statfs":{"total":95995035648,"available":95090081792,"internally_reserved":0,"allocated":129523712,"data_stored":129212071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":775420358},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351434,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758064,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92987088,"statfs":{"total":95995035648,"available":95218778112,"internally_reserved":0,"allocated":823296,"data_stored":507480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":775420352},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514959,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":924656,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":92820496,"statfs":{"total":95995035648,"available":95048187904,"internally_reserved":0,"allocated":823296,"data_stored":507480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":946010560},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":0,"seq":279172874708,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169049,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":262448,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":93482704,"statfs":{"total":95995035648,"available":95726288896,"internally_reserved":0,"allocated":823296,"data_stored":507480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":267909578},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430687,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":754224,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":753086,"kb_avail":92990928,"statfs":{"total":95995035648,"available":95222710272,"internally_reserved":0,"allocated":1150976,"data_stored":835160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":771160518},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575351,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":956580,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":955774,"kb_avail":92788572,"statfs":{"total":95995035648,"available":95015497728,"internally_reserved":0,"allocated":811008,"data_stored":499367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":978713029},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738882,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":834736,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":833598,"kb_avail":92910416,"statfs":{"total":95995035648,"available":95140265984,"internally_reserved":0,"allocated":1150976,"data_stored":835160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":853604805},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411828,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":623596,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93121556,"statfs":{"total":95995035648,"available":95356473344,"internally_reserved":0,"allocated":1015808,"data_stored":700615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150190,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":975148,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":973822,"kb_avail":92770004,"statfs":{"total":95995035648,"available":94996484096,"internally_reserved":0,"allocated":1343488,"data_stored":1028295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":997194183},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986666,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692324,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":780460,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":92964692,"statfs":{"total":95995035648,"available":95195844608,"internally_reserved":0,"allocated":1015808,"data_stored":700615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":798161349},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:23:16.060 INFO:tasks.thrashosds.thrasher:Setting 1.6 to [1, 3, 4, 0, 9, 2] 2024-05-15T13:23:16.060 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.6', '1', '3', '4', '0', '9', '2'] 2024-05-15T13:23:16.060 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pg-upmap-items 1.6 1 3 4 0 9 2 2024-05-15T13:23:16.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:15 smithi177 ceph-mon[44303]: pgmap v2120: 37 pgs: 37 active+clean; 123 MiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 12 MiB/s wr, 195 op/s 2024-05-15T13:23:16.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:15 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2773606742' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:23:16.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:15 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:23:16.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:15 smithi120 ceph-mon[43961]: pgmap v2120: 37 pgs: 37 active+clean; 123 MiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 12 MiB/s wr, 195 op/s 2024-05-15T13:23:16.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:15 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2773606742' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:23:16.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:15 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:23:16.294 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 4 16 14713 14697 229.603 210.25 0.00159521 0.00429877 2024-05-15T13:23:16.470 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:23:16.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:16.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:23:16.576 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:23:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:16.829 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:23:16.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:16.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:23:16.843 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:23:16.881 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:23:16.935 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:23:16.975 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:17.004 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:23:17.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:17.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:23:17.110 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:23:17.124 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:23:17.156 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:16 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:23:17.156 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:16 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:23:17.156 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:16 smithi027 ceph-mon[39795]: osdmap e814: 12 total, 12 up, 10 in 2024-05-15T13:23:17.156 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:16 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1089991271' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:23:17.160 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:17.162 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:23:17.180 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:23:17.203 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:23:17.204 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:23:17.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:17.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:23:17.250 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:23:17.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:16 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:23:17.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:16 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:23:17.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:16 smithi120 ceph-mon[43961]: osdmap e814: 12 total, 12 up, 10 in 2024-05-15T13:23:17.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:16 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1089991271' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:23:17.292 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:23:17.294 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 5 16 17028 17012 212.618 144.688 0.00535676 0.00469915 2024-05-15T13:23:17.309 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:23:17.328 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:23:17.350 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:17.375 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:23:17.411 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:23:17.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:16 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:23:17.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:16 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:23:17.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:16 smithi177 ceph-mon[44303]: osdmap e814: 12 total, 12 up, 10 in 2024-05-15T13:23:17.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:16 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1089991271' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:23:17.441 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:23:17.454 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:23:17.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:17.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:23:17.477 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:23:17.483 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:23:17.483 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:23:17.484 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:23:17.484 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:23:17.484 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:23:17.484 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:23:17.484 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:23:17.484 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:23:17.484 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:23:17.484 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:23:17.484 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:23:17.484 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:23:17.560 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:23:17.609 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:18.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:17 smithi027 ceph-mon[39795]: pgmap v2122: 37 pgs: 37 active+clean; 328 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 41 MiB/s wr, 654 op/s 2024-05-15T13:23:18.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:17 smithi120 ceph-mon[43961]: pgmap v2122: 37 pgs: 37 active+clean; 328 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 41 MiB/s wr, 654 op/s 2024-05-15T13:23:18.294 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 6 16 20326 20310 211.533 206.125 0.00534116 0.00472251 2024-05-15T13:23:18.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:17 smithi177 ceph-mon[44303]: pgmap v2122: 37 pgs: 37 active+clean; 328 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 41 MiB/s wr, 654 op/s 2024-05-15T13:23:19.076 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:23:19.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:19.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:23:19.182 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:23:19.208 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:19.227 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:23:19.229 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:18 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:23:19.229 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:18 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:23:19.229 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:18 smithi177 ceph-mon[44303]: osdmap e815: 12 total, 12 up, 10 in 2024-05-15T13:23:19.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:19.230 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:23:19.241 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:23:19.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:18 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:23:19.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:18 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:23:19.270 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:18 smithi027 ceph-mon[39795]: osdmap e815: 12 total, 12 up, 10 in 2024-05-15T13:23:19.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:18 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:23:19.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:18 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:23:19.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:18 smithi120 ceph-mon[43961]: osdmap e815: 12 total, 12 up, 10 in 2024-05-15T13:23:19.295 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 7 16 22431 22415 200.105 131.562 0.00197803 0.00483661 2024-05-15T13:23:19.331 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:23:19.357 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:19.376 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:23:19.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:19.379 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:23:19.480 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:23:19.508 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:20.261 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:19 smithi027 ceph-mon[39795]: pgmap v2124: 37 pgs: 37 active+clean; 859 MiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 107 MiB/s wr, 1.72k op/s 2024-05-15T13:23:20.261 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:19 smithi027 ceph-mon[39795]: osdmap e816: 12 total, 12 up, 10 in 2024-05-15T13:23:20.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:19 smithi120 ceph-mon[43961]: pgmap v2124: 37 pgs: 37 active+clean; 859 MiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 107 MiB/s wr, 1.72k op/s 2024-05-15T13:23:20.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:19 smithi120 ceph-mon[43961]: osdmap e816: 12 total, 12 up, 10 in 2024-05-15T13:23:20.295 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 8 16 22431 22415 175.093 0 - 0.00483661 2024-05-15T13:23:20.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:19 smithi177 ceph-mon[44303]: pgmap v2124: 37 pgs: 37 active+clean; 859 MiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 107 MiB/s wr, 1.72k op/s 2024-05-15T13:23:20.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:19 smithi177 ceph-mon[44303]: osdmap e816: 12 total, 12 up, 10 in 2024-05-15T13:23:21.295 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 9 16 22431 22415 155.638 0 - 0.00483661 2024-05-15T13:23:21.745 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:23:21.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:21.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:23:21.849 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:23:21.882 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:21.904 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:23:21.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:21.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:23:22.009 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:23:22.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:21 smithi027 ceph-mon[39795]: pgmap v2126: 37 pgs: 37 active+clean; 1.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 167 MiB/s wr, 2.66k op/s 2024-05-15T13:23:22.042 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:22.063 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:23:22.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:22.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:23:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:23:22.169 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:23:22.200 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:22.223 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:23:22.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:22.227 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:23:22.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:21 smithi120 ceph-mon[43961]: pgmap v2126: 37 pgs: 37 active+clean; 1.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 167 MiB/s wr, 2.66k op/s 2024-05-15T13:23:22.295 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 10 16 22431 22415 140.075 0 - 0.00483661 2024-05-15T13:23:22.328 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:23:22.361 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:22.383 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:23:22.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:22.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:23:22.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:21 smithi177 ceph-mon[44303]: pgmap v2126: 37 pgs: 37 active+clean; 1.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 167 MiB/s wr, 2.66k op/s 2024-05-15T13:23:22.486 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:23:22.487 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:23:22.519 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:23.295 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 11 16 22431 22415 127.341 0 - 0.00483661 2024-05-15T13:23:23.780 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:23:23.869 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:23:23.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:23.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:23:23.983 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:23:24.015 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:24.037 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:23:24.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:24.040 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:23:24.141 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:23:24.172 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:24.193 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:23:24.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:24.197 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:23:24.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:23 smithi027 ceph-mon[39795]: pgmap v2127: 37 pgs: 37 active+clean; 1.3 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-05-15T13:23:24.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:23 smithi120 ceph-mon[43961]: pgmap v2127: 37 pgs: 37 active+clean; 1.3 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-05-15T13:23:24.295 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 12 16 22431 22415 116.729 0 - 0.00483661 2024-05-15T13:23:24.298 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:23:24.326 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:24.380 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:23:24.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:24.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:23:24.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:23 smithi177 ceph-mon[44303]: pgmap v2127: 37 pgs: 37 active+clean; 1.3 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-05-15T13:23:24.483 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:23:24.516 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:24.538 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:23:24.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:24.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:23:24.642 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:23:24.672 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:24.921 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:23:24.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:24.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:23:25.025 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:23:25.054 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:25.153 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:23:25.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:25.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:23:25.258 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:23:25.290 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:25.296 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 13 16 22431 22415 107.75 0 - 0.00483661 2024-05-15T13:23:25.311 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:23:25.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:25.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:23:25.416 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:23:25.449 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:25.471 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:23:25.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:25.474 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:23:25.495 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:25 smithi027 ceph-mon[39795]: osdmap e817: 12 total, 12 up, 10 in 2024-05-15T13:23:25.495 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:25 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:23:25.496 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:25 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:23:25.496 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:25 smithi027 ceph-mon[39795]: pgmap v2129: 37 pgs: 37 active+clean; 1.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 131 MiB/s wr, 2.10k op/s 2024-05-15T13:23:25.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:25 smithi120 ceph-mon[43961]: osdmap e817: 12 total, 12 up, 10 in 2024-05-15T13:23:25.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:25 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:23:25.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:25 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:23:25.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:25 smithi120 ceph-mon[43961]: pgmap v2129: 37 pgs: 37 active+clean; 1.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 131 MiB/s wr, 2.10k op/s 2024-05-15T13:23:25.575 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:23:25.605 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:25.626 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:23:25.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:25.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:23:25.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:25 smithi177 ceph-mon[44303]: osdmap e817: 12 total, 12 up, 10 in 2024-05-15T13:23:25.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:25 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:23:25.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:25 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:23:25.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:25 smithi177 ceph-mon[44303]: pgmap v2129: 37 pgs: 37 active+clean; 1.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 131 MiB/s wr, 2.10k op/s 2024-05-15T13:23:25.731 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:23:25.762 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:23:26.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:26.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:23:26.217 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:23:26.249 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:26.255 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:23:26.255 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:23:26.262 INFO:teuthology.orchestra.run.smithi027.stderr:set 1.6 pg_upmap_items mapping to [1->3,4->0,9->2] 2024-05-15T13:23:26.273 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:23:26.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:26.276 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:23:26.296 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 14 16 22431 22415 100.053 0 - 0.00483661 2024-05-15T13:23:26.377 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:23:26.403 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:26 smithi027 ceph-mon[39795]: osdmap e818: 12 total, 12 up, 10 in 2024-05-15T13:23:26.403 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:26 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3372419360' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.6", "id": [1, 3, 4, 0, 9, 2]}]: dispatch 2024-05-15T13:23:26.405 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:26.460 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:23:26.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:26.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:23:26.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:26 smithi120 ceph-mon[43961]: osdmap e818: 12 total, 12 up, 10 in 2024-05-15T13:23:26.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:26 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3372419360' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.6", "id": [1, 3, 4, 0, 9, 2]}]: dispatch 2024-05-15T13:23:26.564 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:23:26.591 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:26.665 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:23:26.665 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:23:26.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:26.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:23:26.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:26 smithi177 ceph-mon[44303]: osdmap e818: 12 total, 12 up, 10 in 2024-05-15T13:23:26.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:26 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3372419360' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.6", "id": [1, 3, 4, 0, 9, 2]}]: dispatch 2024-05-15T13:23:26.769 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:23:26.801 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:26.823 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:23:26.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:26.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:23:26.928 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:23:26.959 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:26.982 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:23:26.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:26.985 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:23:27.086 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:23:27.116 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:27.266 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:23:27.296 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 15 16 22431 22415 93.3827 0 - 0.00483661 2024-05-15T13:23:27.315 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:23:27.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:27.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:23:27.361 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:27 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:23:27.361 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:27 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:23:27.361 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:27 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3372419360' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.6", "id": [1, 3, 4, 0, 9, 2]}]': finished 2024-05-15T13:23:27.361 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:27 smithi027 ceph-mon[39795]: osdmap e819: 12 total, 12 up, 10 in 2024-05-15T13:23:27.361 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:27 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2303321796' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:23:27.361 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:27 smithi027 ceph-mon[39795]: pgmap v2132: 37 pgs: 37 active+clean; 1.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 47 MiB/s wr, 746 op/s 2024-05-15T13:23:27.419 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:23:27.448 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:27.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:27 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:23:27.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:27 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:23:27.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:27 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3372419360' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.6", "id": [1, 3, 4, 0, 9, 2]}]': finished 2024-05-15T13:23:27.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:27 smithi120 ceph-mon[43961]: osdmap e819: 12 total, 12 up, 10 in 2024-05-15T13:23:27.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:27 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2303321796' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:23:27.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:27 smithi120 ceph-mon[43961]: pgmap v2132: 37 pgs: 37 active+clean; 1.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 47 MiB/s wr, 746 op/s 2024-05-15T13:23:27.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:27 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:23:27.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:27 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:23:27.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:27 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3372419360' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.6", "id": [1, 3, 4, 0, 9, 2]}]': finished 2024-05-15T13:23:27.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:27 smithi177 ceph-mon[44303]: osdmap e819: 12 total, 12 up, 10 in 2024-05-15T13:23:27.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:27 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2303321796' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:23:27.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:27 smithi177 ceph-mon[44303]: pgmap v2132: 37 pgs: 37 active+clean; 1.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 47 MiB/s wr, 746 op/s 2024-05-15T13:23:28.296 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 16 16 22431 22415 87.5462 0 - 0.00483661 2024-05-15T13:23:28.384 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:28 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:23:28.384 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:28 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2303321796' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:23:28.384 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:28 smithi027 ceph-mon[39795]: osdmap e820: 12 total, 12 up, 10 in 2024-05-15T13:23:28.384 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:28 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:23:28.385 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:28 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:23:28.398 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:23:28.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:28.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:23:28.503 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:23:28.532 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:28 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:23:28.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:28 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2303321796' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:23:28.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:28 smithi120 ceph-mon[43961]: osdmap e820: 12 total, 12 up, 10 in 2024-05-15T13:23:28.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:28 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:23:28.533 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:28 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:23:28.534 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:28.557 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:23:28.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:28.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:23:28.661 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:23:28.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:28 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:23:28.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:28 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2303321796' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:23:28.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:28 smithi177 ceph-mon[44303]: osdmap e820: 12 total, 12 up, 10 in 2024-05-15T13:23:28.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:28 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:23:28.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:28 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:23:28.690 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:28.940 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:23:28.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:28.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:23:29.045 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:23:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:23:29.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:29.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:23:29.296 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-05-15T13:23:29.296 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 17 16 22431 22415 82.3964 0 - 0.00483661 2024-05-15T13:23:29.376 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:23:29.408 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:29.430 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:23:29.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:29.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:23:29.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:29 smithi027 ceph-mon[39795]: osdmap e821: 12 total, 12 up, 10 in 2024-05-15T13:23:29.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:29 smithi027 ceph-mon[39795]: pgmap v2135: 37 pgs: 37 active+clean; 1.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail 2024-05-15T13:23:29.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:29 smithi027 ceph-mon[39795]: osdmap e822: 12 total, 12 up, 10 in 2024-05-15T13:23:29.534 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:23:29.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:29 smithi120 ceph-mon[43961]: osdmap e821: 12 total, 12 up, 10 in 2024-05-15T13:23:29.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:29 smithi120 ceph-mon[43961]: pgmap v2135: 37 pgs: 37 active+clean; 1.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail 2024-05-15T13:23:29.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:29 smithi120 ceph-mon[43961]: osdmap e822: 12 total, 12 up, 10 in 2024-05-15T13:23:29.566 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:29.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:29 smithi177 ceph-mon[44303]: osdmap e821: 12 total, 12 up, 10 in 2024-05-15T13:23:29.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:29 smithi177 ceph-mon[44303]: pgmap v2135: 37 pgs: 37 active+clean; 1.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail 2024-05-15T13:23:29.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:29 smithi177 ceph-mon[44303]: osdmap e822: 12 total, 12 up, 10 in 2024-05-15T13:23:29.690 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:23:29.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:29.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:23:29.794 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:23:29.818 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:23:29.826 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:23:29.826 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:29.848 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:23:29.848 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:23:29.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:29.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:23:29.867 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:23:29.887 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:23:29.907 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:23:29.928 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:23:29.949 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:23:29.952 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:23:29.969 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:23:29.983 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:29.994 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:23:30.016 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:23:30.036 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:23:30.057 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:23:30.058 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:23:30.059 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:23:30.059 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:23:30.059 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:23:30.059 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:23:30.059 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:23:30.059 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:23:30.059 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:23:30.059 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:23:30.059 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:23:30.059 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:23:30.059 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:23:30.081 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:23:30.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:30.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:23:30.187 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:23:30.220 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:30.242 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:23:30.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:30.245 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:23:30.296 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 18 16 22431 22415 77.8189 0 - 0.00483661 2024-05-15T13:23:30.346 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:23:30.376 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:30.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:30 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2458969415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:23:30.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:30 smithi027 ceph-mon[39795]: osdmap e823: 12 total, 12 up, 10 in 2024-05-15T13:23:30.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:30 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2458969415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:23:30.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:30 smithi120 ceph-mon[43961]: osdmap e823: 12 total, 12 up, 10 in 2024-05-15T13:23:30.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:30 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2458969415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:23:30.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:30 smithi177 ceph-mon[44303]: osdmap e823: 12 total, 12 up, 10 in 2024-05-15T13:23:30.966 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:23:31.226 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:23:31.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:31.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:23:31.296 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 19 16 22431 22415 73.7233 0 - 0.00483661 2024-05-15T13:23:31.331 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:23:31.359 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:31.398 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:23:31.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:31.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:23:31.507 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:23:31.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:31 smithi027 ceph-mon[39795]: pgmap v2138: 37 pgs: 37 active+clean; 1.4 GiB data, 12 GiB used, 883 GiB / 894 GiB avail 2024-05-15T13:23:31.534 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:31.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:31 smithi120 ceph-mon[43961]: pgmap v2138: 37 pgs: 37 active+clean; 1.4 GiB data, 12 GiB used, 883 GiB / 894 GiB avail 2024-05-15T13:23:31.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:31 smithi177 ceph-mon[44303]: pgmap v2138: 37 pgs: 37 active+clean; 1.4 GiB data, 12 GiB used, 883 GiB / 894 GiB avail 2024-05-15T13:23:31.697 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:23:31.883 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:23:31.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:31.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:23:31.987 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:23:32.020 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:32.041 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:23:32.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:32.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:23:32.142 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 9, 10, 11, 7] out_osds: [4, 8] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:23:32.142 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:23:32.142 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-05-15T13:23:32.142 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-15T13:23:32.142 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-15T13:23:32.143 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-05-15T13:23:32.145 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:23:32.213 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:32.233 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:23:32.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:32.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:23:32.296 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T13:23:32.298234+0000 min lat: 0.00110277 max lat: 0.207024 avg lat: 0.00483661 2024-05-15T13:23:32.297 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:23:32.297 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 20 16 22431 22415 70.0372 0 - 0.00483661 2024-05-15T13:23:32.337 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:23:32.365 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:32.381 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:23:32.381 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T13:23:32.381 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T13:23:32.381 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:23:32.381 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.381 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:145400 8.0 8:04437cde:::benchmark_data_smithi177_149673_object55338:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e788)", 2024-05-15T13:23:32.381 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:21:48.435889+0000", 2024-05-15T13:23:32.381 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 103.943251987, 2024-05-15T13:23:32.381 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.039006181000000001, 2024-05-15T13:23:32.381 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:23:32.381 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:23:32.381 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:23:32.382 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:23:32.382 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:23:32.382 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 145400 2024-05-15T13:23:32.382 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.382 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:23:32.382 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.382 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:23:32.382 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:48.435889+0000", 2024-05-15T13:23:32.382 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.382 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.382 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.385 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:23:32.385 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:48.435889+0000", 2024-05-15T13:23:32.385 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.385 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.385 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.385 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:23:32.385 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:48.435890+0000", 2024-05-15T13:23:32.385 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.66e-06 2024-05-15T13:23:32.385 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.385 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.386 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:23:32.386 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:48.435898+0000", 2024-05-15T13:23:32.386 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9319999999999996e-06 2024-05-15T13:23:32.386 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.386 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.386 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:23:32.386 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:48.435917+0000", 2024-05-15T13:23:32.386 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9114999999999999e-05 2024-05-15T13:23:32.386 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.386 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.386 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:23:32.386 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:48.435927+0000", 2024-05-15T13:23:32.386 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9909999999999997e-06 2024-05-15T13:23:32.387 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.387 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.387 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:23:32.387 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:48.450421+0000", 2024-05-15T13:23:32.387 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014494152999999999 2024-05-15T13:23:32.387 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.387 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.387 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:23:32.387 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:48.450563+0000", 2024-05-15T13:23:32.387 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014203600000000001 2024-05-15T13:23:32.387 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.387 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.387 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:23:32.387 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:48.450690+0000", 2024-05-15T13:23:32.388 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012637799999999999 2024-05-15T13:23:32.388 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.388 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.388 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:23:32.388 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:48.454653+0000", 2024-05-15T13:23:32.388 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039627539999999998 2024-05-15T13:23:32.388 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.388 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.388 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.388 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:48.454856+0000", 2024-05-15T13:23:32.388 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000203308 2024-05-15T13:23:32.388 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.388 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.388 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.389 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:48.474768+0000", 2024-05-15T13:23:32.389 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.019912005999999999 2024-05-15T13:23:32.389 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.389 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.389 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:23:32.389 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:48.474824+0000", 2024-05-15T13:23:32.389 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5976000000000001e-05 2024-05-15T13:23:32.389 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.389 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.389 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:23:32.389 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:48.474895+0000", 2024-05-15T13:23:32.389 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0871999999999995e-05 2024-05-15T13:23:32.389 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.389 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:32.389 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.390 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.390 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.390 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:149542 8.0 8:055e5c72:::benchmark_data_smithi177_149673_object59480:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e790)", 2024-05-15T13:23:32.390 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:21:53.615844+0000", 2024-05-15T13:23:32.390 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 98.763296639000004, 2024-05-15T13:23:32.390 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.041873212, 2024-05-15T13:23:32.390 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:23:32.390 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:23:32.390 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:23:32.390 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:23:32.390 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:23:32.390 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 149542 2024-05-15T13:23:32.390 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.391 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:23:32.391 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.391 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:23:32.391 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:53.615844+0000", 2024-05-15T13:23:32.391 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.391 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.391 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.391 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:23:32.391 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:53.615844+0000", 2024-05-15T13:23:32.391 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.391 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.391 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.391 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:23:32.391 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:53.615844+0000", 2024-05-15T13:23:32.391 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9200000000000002e-07 2024-05-15T13:23:32.391 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.391 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.392 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:23:32.392 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:53.615847+0000", 2024-05-15T13:23:32.392 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4739999999999999e-06 2024-05-15T13:23:32.392 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.392 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.392 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:23:32.392 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:53.615853+0000", 2024-05-15T13:23:32.392 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9240000000000004e-06 2024-05-15T13:23:32.392 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.392 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.392 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:23:32.392 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:53.615858+0000", 2024-05-15T13:23:32.392 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0119999999999996e-06 2024-05-15T13:23:32.392 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.392 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.393 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:23:32.393 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:53.628519+0000", 2024-05-15T13:23:32.393 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012661194000000001 2024-05-15T13:23:32.393 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.393 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.393 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:23:32.393 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:53.628593+0000", 2024-05-15T13:23:32.393 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3829000000000001e-05 2024-05-15T13:23:32.393 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.393 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.393 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:23:32.393 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:53.628653+0000", 2024-05-15T13:23:32.393 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0089e-05 2024-05-15T13:23:32.393 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.394 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.394 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:23:32.394 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:53.638344+0000", 2024-05-15T13:23:32.394 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0096910360000000001 2024-05-15T13:23:32.394 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.394 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.394 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.394 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:53.649172+0000", 2024-05-15T13:23:32.394 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010828323000000001 2024-05-15T13:23:32.394 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.394 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.394 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.394 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:53.657623+0000", 2024-05-15T13:23:32.394 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0084504039999999999 2024-05-15T13:23:32.395 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.395 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.395 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:23:32.395 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:53.657676+0000", 2024-05-15T13:23:32.395 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3538000000000001e-05 2024-05-15T13:23:32.395 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.395 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.395 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:23:32.395 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:53.657717+0000", 2024-05-15T13:23:32.395 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0997000000000001e-05 2024-05-15T13:23:32.395 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.395 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:32.395 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.395 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.395 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.395 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:150341 8.0 8:00d7777a:::benchmark_data_smithi177_149673_object60279:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e790)", 2024-05-15T13:23:32.395 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:21:54.576102+0000", 2024-05-15T13:23:32.396 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 97.803038544000003, 2024-05-15T13:23:32.396 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.038846389000000002, 2024-05-15T13:23:32.396 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:23:32.396 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:23:32.396 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:23:32.396 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:23:32.396 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:23:32.396 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 150341 2024-05-15T13:23:32.396 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.396 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:23:32.396 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.396 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:23:32.396 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.576102+0000", 2024-05-15T13:23:32.397 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.397 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.397 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.397 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:23:32.397 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.576102+0000", 2024-05-15T13:23:32.397 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.397 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.397 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.397 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:23:32.397 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.576103+0000", 2024-05-15T13:23:32.397 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9800000000000003e-07 2024-05-15T13:23:32.397 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.397 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.397 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:23:32.398 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.576119+0000", 2024-05-15T13:23:32.398 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6041000000000001e-05 2024-05-15T13:23:32.398 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.398 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.398 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:23:32.398 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.576133+0000", 2024-05-15T13:23:32.398 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4572999999999999e-05 2024-05-15T13:23:32.398 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.398 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.398 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:23:32.398 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.576142+0000", 2024-05-15T13:23:32.398 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2859999999999999e-06 2024-05-15T13:23:32.398 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.398 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.399 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:23:32.399 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.593144+0000", 2024-05-15T13:23:32.399 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.017002685999999999 2024-05-15T13:23:32.399 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.399 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.399 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:23:32.399 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.593264+0000", 2024-05-15T13:23:32.399 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000119196 2024-05-15T13:23:32.399 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.399 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.399 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:23:32.399 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.593374+0000", 2024-05-15T13:23:32.399 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000110098 2024-05-15T13:23:32.399 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.399 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.400 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:23:32.400 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.607386+0000", 2024-05-15T13:23:32.400 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014012785999999999 2024-05-15T13:23:32.400 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.400 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.400 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.400 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.607431+0000", 2024-05-15T13:23:32.400 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4348999999999998e-05 2024-05-15T13:23:32.400 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.400 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.400 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.400 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.614845+0000", 2024-05-15T13:23:32.400 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0074141420000000003 2024-05-15T13:23:32.400 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.401 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.401 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:23:32.401 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.614903+0000", 2024-05-15T13:23:32.401 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7595e-05 2024-05-15T13:23:32.401 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.401 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.401 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:23:32.401 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.614948+0000", 2024-05-15T13:23:32.401 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5838999999999999e-05 2024-05-15T13:23:32.401 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.401 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:32.401 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.402 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.402 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.402 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:150356 8.0 8:0ca2616f:::benchmark_data_smithi177_149673_object60294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e790)", 2024-05-15T13:23:32.402 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:21:54.596926+0000", 2024-05-15T13:23:32.402 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 97.782214272000004, 2024-05-15T13:23:32.402 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.044741580000000003, 2024-05-15T13:23:32.402 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:23:32.402 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:23:32.402 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:23:32.402 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:23:32.402 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:23:32.402 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 150356 2024-05-15T13:23:32.402 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.402 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:23:32.403 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.403 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:23:32.403 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.596926+0000", 2024-05-15T13:23:32.403 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.403 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.403 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.403 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:23:32.403 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.596926+0000", 2024-05-15T13:23:32.403 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.403 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.403 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.403 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:23:32.403 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.596928+0000", 2024-05-15T13:23:32.403 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4959999999999999e-06 2024-05-15T13:23:32.404 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.404 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.404 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:23:32.404 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.596933+0000", 2024-05-15T13:23:32.404 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3959999999999997e-06 2024-05-15T13:23:32.404 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.404 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.404 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:23:32.404 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.596948+0000", 2024-05-15T13:23:32.404 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5033e-05 2024-05-15T13:23:32.404 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.404 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.404 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:23:32.404 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.596957+0000", 2024-05-15T13:23:32.405 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4160000000000004e-06 2024-05-15T13:23:32.405 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.405 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.405 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:23:32.405 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.624514+0000", 2024-05-15T13:23:32.405 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.027557422000000002 2024-05-15T13:23:32.405 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.405 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.405 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:23:32.405 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.624655+0000", 2024-05-15T13:23:32.405 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000140999 2024-05-15T13:23:32.405 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.405 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.405 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:23:32.405 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.624782+0000", 2024-05-15T13:23:32.406 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012732999999999999 2024-05-15T13:23:32.406 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.406 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.406 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.406 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.629741+0000", 2024-05-15T13:23:32.406 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0049585569999999997 2024-05-15T13:23:32.406 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.406 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.406 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.406 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.634846+0000", 2024-05-15T13:23:32.406 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0051051910000000002 2024-05-15T13:23:32.406 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.406 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.407 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:23:32.407 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.641568+0000", 2024-05-15T13:23:32.407 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0067215410000000001 2024-05-15T13:23:32.407 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.407 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.407 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:23:32.407 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.641625+0000", 2024-05-15T13:23:32.407 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7867e-05 2024-05-15T13:23:32.407 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.407 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.407 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:23:32.407 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.641668+0000", 2024-05-15T13:23:32.407 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2332000000000001e-05 2024-05-15T13:23:32.407 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.407 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:32.408 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.408 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.408 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.408 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:150362 8.0 8:0b1d8878:::benchmark_data_smithi177_149673_object60300:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e790)", 2024-05-15T13:23:32.408 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:21:54.605546+0000", 2024-05-15T13:23:32.408 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 97.773594962999994, 2024-05-15T13:23:32.408 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.046411743999999998, 2024-05-15T13:23:32.408 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:23:32.408 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:23:32.408 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:23:32.408 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:23:32.408 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:23:32.408 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 150362 2024-05-15T13:23:32.408 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.409 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:23:32.409 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.409 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:23:32.409 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.605546+0000", 2024-05-15T13:23:32.409 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.409 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.409 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.409 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:23:32.409 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.605546+0000", 2024-05-15T13:23:32.409 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.409 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.410 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.410 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:23:32.410 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.605547+0000", 2024-05-15T13:23:32.410 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7990000000000001e-06 2024-05-15T13:23:32.410 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.410 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.410 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:23:32.410 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.605555+0000", 2024-05-15T13:23:32.410 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3860000000000001e-06 2024-05-15T13:23:32.410 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.410 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.410 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:23:32.410 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.605570+0000", 2024-05-15T13:23:32.410 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5333999999999999e-05 2024-05-15T13:23:32.411 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.411 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.411 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:23:32.411 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.605580+0000", 2024-05-15T13:23:32.411 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4550000000000001e-06 2024-05-15T13:23:32.411 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.411 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.411 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:23:32.411 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.629809+0000", 2024-05-15T13:23:32.411 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.024229967000000002 2024-05-15T13:23:32.411 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.412 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.412 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:23:32.412 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.629943+0000", 2024-05-15T13:23:32.412 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013343400000000001 2024-05-15T13:23:32.412 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.412 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.412 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:23:32.412 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.630060+0000", 2024-05-15T13:23:32.412 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011667099999999999 2024-05-15T13:23:32.412 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.412 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.412 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.412 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.637801+0000", 2024-05-15T13:23:32.412 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0077410550000000002 2024-05-15T13:23:32.412 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.413 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.413 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.413 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.638968+0000", 2024-05-15T13:23:32.413 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001167843 2024-05-15T13:23:32.413 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.413 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.413 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:23:32.413 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.651853+0000", 2024-05-15T13:23:32.413 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012884815000000001 2024-05-15T13:23:32.413 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.413 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.413 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:23:32.413 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.651895+0000", 2024-05-15T13:23:32.413 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1983000000000001e-05 2024-05-15T13:23:32.413 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.414 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.414 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:23:32.414 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:54.651957+0000", 2024-05-15T13:23:32.414 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2001999999999994e-05 2024-05-15T13:23:32.414 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.414 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:32.414 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.414 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.414 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.414 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:167325 8.0 8:04c13d76:::benchmark_data_smithi177_149673_object77263:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-05-15T13:23:32.414 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:22:17.212684+0000", 2024-05-15T13:23:32.414 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 75.166456681, 2024-05-15T13:23:32.414 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.036165791000000003, 2024-05-15T13:23:32.414 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:23:32.415 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:23:32.415 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:23:32.415 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:23:32.415 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:23:32.415 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 167325 2024-05-15T13:23:32.415 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.415 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:23:32.415 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.415 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:23:32.415 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:17.212684+0000", 2024-05-15T13:23:32.415 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.415 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.415 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.416 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:23:32.416 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:17.212684+0000", 2024-05-15T13:23:32.416 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.416 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.416 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.416 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:23:32.416 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:17.212684+0000", 2024-05-15T13:23:32.416 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5599999999999995e-07 2024-05-15T13:23:32.416 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.416 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.416 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:23:32.416 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:17.212691+0000", 2024-05-15T13:23:32.416 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0199999999999997e-06 2024-05-15T13:23:32.416 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.416 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.416 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:23:32.417 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:17.212700+0000", 2024-05-15T13:23:32.417 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6330000000000002e-06 2024-05-15T13:23:32.417 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.417 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.417 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:23:32.417 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:17.212706+0000", 2024-05-15T13:23:32.417 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6359999999999999e-06 2024-05-15T13:23:32.417 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.417 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.417 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:23:32.417 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:17.231784+0000", 2024-05-15T13:23:32.417 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.019078054000000001 2024-05-15T13:23:32.417 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.417 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.418 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:23:32.418 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:17.231917+0000", 2024-05-15T13:23:32.418 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000133268 2024-05-15T13:23:32.418 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.418 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.418 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:23:32.418 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:17.232047+0000", 2024-05-15T13:23:32.418 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013031800000000001 2024-05-15T13:23:32.418 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.418 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.418 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:23:32.418 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:17.242348+0000", 2024-05-15T13:23:32.418 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010300501 2024-05-15T13:23:32.419 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.419 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.419 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.419 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:17.242714+0000", 2024-05-15T13:23:32.419 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00036573200000000001 2024-05-15T13:23:32.419 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.419 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.419 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.419 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:17.248739+0000", 2024-05-15T13:23:32.419 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0060257740000000002 2024-05-15T13:23:32.419 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.419 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.419 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:23:32.419 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:17.248798+0000", 2024-05-15T13:23:32.419 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8594999999999997e-05 2024-05-15T13:23:32.420 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.420 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.420 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:23:32.420 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:17.248850+0000", 2024-05-15T13:23:32.420 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1703999999999998e-05 2024-05-15T13:23:32.420 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.420 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:32.420 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.420 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.420 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.420 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:168397 8.0 8:0871a746:::benchmark_data_smithi177_149673_object78335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-05-15T13:23:32.420 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:22:18.552919+0000", 2024-05-15T13:23:32.420 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 73.826221305999994, 2024-05-15T13:23:32.420 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.037806711999999999, 2024-05-15T13:23:32.420 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:23:32.421 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:23:32.421 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:23:32.421 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:23:32.421 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:23:32.421 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 168397 2024-05-15T13:23:32.421 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.421 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:23:32.421 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.421 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:23:32.421 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:18.552919+0000", 2024-05-15T13:23:32.421 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.421 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.421 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.421 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:23:32.422 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:18.552919+0000", 2024-05-15T13:23:32.422 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.422 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.422 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.422 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:23:32.422 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:18.552921+0000", 2024-05-15T13:23:32.422 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.533e-06 2024-05-15T13:23:32.422 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.422 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.422 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:23:32.422 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:18.552927+0000", 2024-05-15T13:23:32.422 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7209999999999997e-06 2024-05-15T13:23:32.422 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.422 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.422 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:23:32.423 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:18.552941+0000", 2024-05-15T13:23:32.423 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3923999999999999e-05 2024-05-15T13:23:32.423 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.423 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.423 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:23:32.423 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:18.552949+0000", 2024-05-15T13:23:32.423 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3860000000000001e-06 2024-05-15T13:23:32.423 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.423 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.423 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:23:32.423 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:18.553002+0000", 2024-05-15T13:23:32.423 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3060999999999999e-05 2024-05-15T13:23:32.423 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.423 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.423 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:23:32.424 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:18.553119+0000", 2024-05-15T13:23:32.424 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000117669 2024-05-15T13:23:32.424 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.424 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.424 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:23:32.424 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:18.553250+0000", 2024-05-15T13:23:32.424 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013046100000000001 2024-05-15T13:23:32.424 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.424 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.424 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.424 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:18.562639+0000", 2024-05-15T13:23:32.424 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0093894059999999994 2024-05-15T13:23:32.424 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.424 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.425 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:23:32.425 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:18.566304+0000", 2024-05-15T13:23:32.425 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003664311 2024-05-15T13:23:32.425 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.425 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.425 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.425 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:18.590613+0000", 2024-05-15T13:23:32.425 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.024309042999999999 2024-05-15T13:23:32.425 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.425 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.425 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:23:32.425 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:18.590673+0000", 2024-05-15T13:23:32.425 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0650999999999999e-05 2024-05-15T13:23:32.425 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.426 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.426 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:23:32.426 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:18.590726+0000", 2024-05-15T13:23:32.426 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2546000000000001e-05 2024-05-15T13:23:32.426 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.426 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:32.426 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.426 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.426 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.426 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:169855 8.0 8:0f2bcfaf:::benchmark_data_smithi177_149673_object79793:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-05-15T13:23:32.426 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:22:20.399169+0000", 2024-05-15T13:23:32.426 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 71.979971327000001, 2024-05-15T13:23:32.426 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.041817794999999998, 2024-05-15T13:23:32.426 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:23:32.427 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:23:32.427 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:23:32.427 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:23:32.427 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:23:32.427 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 169855 2024-05-15T13:23:32.427 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.427 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:23:32.427 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.427 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:23:32.427 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399169+0000", 2024-05-15T13:23:32.427 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.427 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.427 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.427 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:23:32.427 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399169+0000", 2024-05-15T13:23:32.428 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.428 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.428 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.428 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:23:32.428 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399171+0000", 2024-05-15T13:23:32.428 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5680000000000001e-06 2024-05-15T13:23:32.428 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.428 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.428 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:23:32.428 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399178+0000", 2024-05-15T13:23:32.428 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0870000000000001e-06 2024-05-15T13:23:32.428 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.428 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.428 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:23:32.429 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399194+0000", 2024-05-15T13:23:32.429 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6334e-05 2024-05-15T13:23:32.429 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.429 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.429 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:23:32.429 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399203+0000", 2024-05-15T13:23:32.429 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6000000000000007e-06 2024-05-15T13:23:32.429 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.429 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.429 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:23:32.429 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399278+0000", 2024-05-15T13:23:32.429 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5610000000000003e-05 2024-05-15T13:23:32.429 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.430 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.430 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:23:32.430 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399438+0000", 2024-05-15T13:23:32.430 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015956499999999999 2024-05-15T13:23:32.430 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.430 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.430 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:23:32.430 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399563+0000", 2024-05-15T13:23:32.430 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000124747 2024-05-15T13:23:32.430 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.430 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.430 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.430 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.412792+0000", 2024-05-15T13:23:32.431 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013229388999999999 2024-05-15T13:23:32.431 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.431 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.431 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:23:32.431 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.412807+0000", 2024-05-15T13:23:32.431 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4599e-05 2024-05-15T13:23:32.431 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.431 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.431 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.431 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.440875+0000", 2024-05-15T13:23:32.431 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.028068823 2024-05-15T13:23:32.431 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.431 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.431 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:23:32.432 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.440936+0000", 2024-05-15T13:23:32.432 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0887000000000002e-05 2024-05-15T13:23:32.432 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.432 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.432 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:23:32.432 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.440987+0000", 2024-05-15T13:23:32.432 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0586e-05 2024-05-15T13:23:32.432 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.432 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:32.432 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.432 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.432 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.432 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:169857 8.0 8:0c8e09ba:::benchmark_data_smithi177_149673_object79795:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-05-15T13:23:32.432 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:22:20.399238+0000", 2024-05-15T13:23:32.433 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 71.979902663999994, 2024-05-15T13:23:32.433 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.041864540999999998, 2024-05-15T13:23:32.433 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:23:32.433 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:23:32.433 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:23:32.433 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:23:32.433 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:23:32.433 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 169857 2024-05-15T13:23:32.433 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.433 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:23:32.433 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.433 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:23:32.433 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399238+0000", 2024-05-15T13:23:32.433 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.433 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.434 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.434 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:23:32.434 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399238+0000", 2024-05-15T13:23:32.434 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.434 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.434 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.435 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:23:32.435 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399238+0000", 2024-05-15T13:23:32.435 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4000000000000002e-07 2024-05-15T13:23:32.435 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.435 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.435 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:23:32.436 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399245+0000", 2024-05-15T13:23:32.436 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6699999999999997e-06 2024-05-15T13:23:32.436 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.436 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.436 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:23:32.436 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399251+0000", 2024-05-15T13:23:32.436 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2580000000000001e-06 2024-05-15T13:23:32.437 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.437 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.437 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:23:32.437 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399257+0000", 2024-05-15T13:23:32.437 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7189999999999998e-06 2024-05-15T13:23:32.437 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.438 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.438 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:23:32.438 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.403691+0000", 2024-05-15T13:23:32.438 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0044336269999999999 2024-05-15T13:23:32.438 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.438 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.439 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:23:32.439 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.403844+0000", 2024-05-15T13:23:32.439 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015372500000000001 2024-05-15T13:23:32.439 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.439 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.439 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:23:32.439 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.403971+0000", 2024-05-15T13:23:32.440 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000126754 2024-05-15T13:23:32.440 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.440 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.440 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.440 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.412843+0000", 2024-05-15T13:23:32.440 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0088714580000000005 2024-05-15T13:23:32.440 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.441 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.441 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:23:32.441 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.412890+0000", 2024-05-15T13:23:32.441 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7831000000000003e-05 2024-05-15T13:23:32.441 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.441 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.442 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.442 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.441029+0000", 2024-05-15T13:23:32.442 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.028138691 2024-05-15T13:23:32.442 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.442 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.442 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:23:32.443 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.441062+0000", 2024-05-15T13:23:32.443 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2648000000000003e-05 2024-05-15T13:23:32.443 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.443 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.443 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:23:32.443 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.441102+0000", 2024-05-15T13:23:32.443 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0620000000000001e-05 2024-05-15T13:23:32.444 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.444 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:32.444 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.444 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.444 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.444 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:169859 8.0 8:04584fc8:::benchmark_data_smithi177_149673_object79797:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-05-15T13:23:32.445 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:22:20.399290+0000", 2024-05-15T13:23:32.445 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 71.979850228999993, 2024-05-15T13:23:32.445 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.041911802999999997, 2024-05-15T13:23:32.445 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:23:32.445 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:23:32.445 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:23:32.446 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:23:32.446 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:23:32.446 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 169859 2024-05-15T13:23:32.446 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.446 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:23:32.446 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.447 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:23:32.447 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399290+0000", 2024-05-15T13:23:32.447 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.447 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.447 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.447 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:23:32.448 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399290+0000", 2024-05-15T13:23:32.448 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.448 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.448 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.448 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:23:32.448 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399291+0000", 2024-05-15T13:23:32.449 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9699999999999996e-07 2024-05-15T13:23:32.449 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.449 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.449 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:23:32.449 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399295+0000", 2024-05-15T13:23:32.449 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8909999999999997e-06 2024-05-15T13:23:32.450 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.450 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.450 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:23:32.450 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399308+0000", 2024-05-15T13:23:32.450 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3441000000000001e-05 2024-05-15T13:23:32.450 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.451 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.451 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:23:32.451 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399315+0000", 2024-05-15T13:23:32.451 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9210000000000003e-06 2024-05-15T13:23:32.451 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.451 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.451 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:23:32.452 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.408521+0000", 2024-05-15T13:23:32.452 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0092057240000000002 2024-05-15T13:23:32.452 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.452 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.452 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:23:32.452 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.408636+0000", 2024-05-15T13:23:32.453 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000114955 2024-05-15T13:23:32.453 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.453 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.453 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:23:32.453 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.408745+0000", 2024-05-15T13:23:32.454 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000108964 2024-05-15T13:23:32.454 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.454 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.454 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.454 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.412882+0000", 2024-05-15T13:23:32.454 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.004137595 2024-05-15T13:23:32.455 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.455 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.455 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:23:32.455 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.417631+0000", 2024-05-15T13:23:32.455 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0047486840000000004 2024-05-15T13:23:32.455 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.455 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.456 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.456 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.441150+0000", 2024-05-15T13:23:32.456 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.023518693 2024-05-15T13:23:32.456 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.456 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.456 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:23:32.456 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.441179+0000", 2024-05-15T13:23:32.456 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9675e-05 2024-05-15T13:23:32.456 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.456 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.457 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:23:32.457 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.441202+0000", 2024-05-15T13:23:32.457 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2663e-05 2024-05-15T13:23:32.457 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.457 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:32.457 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.457 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.458 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.458 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:169862 8.0 8:02e02983:::benchmark_data_smithi177_149673_object79800:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-05-15T13:23:32.458 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:22:20.399507+0000", 2024-05-15T13:23:32.458 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 71.979633133999997, 2024-05-15T13:23:32.458 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.041797675999999999, 2024-05-15T13:23:32.458 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:23:32.458 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:23:32.458 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:23:32.459 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:23:32.459 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:23:32.459 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 169862 2024-05-15T13:23:32.459 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.459 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:23:32.459 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.459 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:23:32.459 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399507+0000", 2024-05-15T13:23:32.460 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.460 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.460 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.460 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:23:32.460 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399507+0000", 2024-05-15T13:23:32.460 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.460 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.460 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.461 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:23:32.461 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399508+0000", 2024-05-15T13:23:32.461 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5799999999999998e-07 2024-05-15T13:23:32.461 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.461 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.461 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:23:32.461 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399516+0000", 2024-05-15T13:23:32.461 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4089999999999997e-06 2024-05-15T13:23:32.462 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.462 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.462 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:23:32.462 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399526+0000", 2024-05-15T13:23:32.462 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.01e-05 2024-05-15T13:23:32.462 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.462 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.462 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:23:32.462 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.399533+0000", 2024-05-15T13:23:32.462 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4630000000000004e-06 2024-05-15T13:23:32.462 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.462 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.463 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:23:32.463 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.412916+0000", 2024-05-15T13:23:32.463 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013383183 2024-05-15T13:23:32.463 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.463 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.463 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:23:32.463 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.413037+0000", 2024-05-15T13:23:32.463 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120974 2024-05-15T13:23:32.463 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.463 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.463 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:23:32.463 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.413171+0000", 2024-05-15T13:23:32.463 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013397699999999999 2024-05-15T13:23:32.464 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.464 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.464 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.464 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.417669+0000", 2024-05-15T13:23:32.464 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0044976529999999999 2024-05-15T13:23:32.464 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.464 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.464 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:23:32.464 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.426244+0000", 2024-05-15T13:23:32.464 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0085748600000000001 2024-05-15T13:23:32.464 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.464 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.464 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.464 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.441240+0000", 2024-05-15T13:23:32.465 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014996221000000001 2024-05-15T13:23:32.465 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.465 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.465 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:23:32.465 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.441269+0000", 2024-05-15T13:23:32.465 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8909000000000001e-05 2024-05-15T13:23:32.465 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.465 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.465 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:23:32.465 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:20.441305+0000", 2024-05-15T13:23:32.465 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6069000000000003e-05 2024-05-15T13:23:32.465 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.466 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:32.466 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.466 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.466 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.466 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:171887 8.0 8:065a1aa2:::benchmark_data_smithi177_149673_object81825:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e797)", 2024-05-15T13:23:32.466 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:22:22.993053+0000", 2024-05-15T13:23:32.466 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 69.386087232999998, 2024-05-15T13:23:32.466 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.048286281, 2024-05-15T13:23:32.466 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:23:32.466 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:23:32.467 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:23:32.467 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:23:32.467 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:23:32.467 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 171887 2024-05-15T13:23:32.467 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.467 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:23:32.467 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.467 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:23:32.467 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:22.993053+0000", 2024-05-15T13:23:32.467 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.467 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.467 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.467 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:23:32.467 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:22.993053+0000", 2024-05-15T13:23:32.467 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.468 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.468 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.468 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:23:32.468 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:22.993055+0000", 2024-05-15T13:23:32.468 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9120000000000001e-06 2024-05-15T13:23:32.468 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.468 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.468 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:23:32.468 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:22.993065+0000", 2024-05-15T13:23:32.468 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0074999999999999e-05 2024-05-15T13:23:32.468 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.468 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.468 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:23:32.468 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:22.993083+0000", 2024-05-15T13:23:32.468 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7918e-05 2024-05-15T13:23:32.469 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.469 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.469 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:23:32.469 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:22.993092+0000", 2024-05-15T13:23:32.469 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9570000000000008e-06 2024-05-15T13:23:32.469 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.469 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.469 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:23:32.469 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:22.993174+0000", 2024-05-15T13:23:32.469 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1691000000000001e-05 2024-05-15T13:23:32.469 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.469 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.469 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:23:32.469 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:22.993294+0000", 2024-05-15T13:23:32.470 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000119836 2024-05-15T13:23:32.470 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.470 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.470 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:23:32.470 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:22.993404+0000", 2024-05-15T13:23:32.470 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011010699999999999 2024-05-15T13:23:32.470 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.470 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.470 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.470 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:23.004689+0000", 2024-05-15T13:23:32.470 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011285702999999999 2024-05-15T13:23:32.470 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.470 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.470 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:23:32.471 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:23.011339+0000", 2024-05-15T13:23:32.471 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0066493869999999997 2024-05-15T13:23:32.471 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.471 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.471 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.471 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:23.041211+0000", 2024-05-15T13:23:32.471 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.029871724999999998 2024-05-15T13:23:32.471 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.471 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.471 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:23:32.471 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:23.041281+0000", 2024-05-15T13:23:32.471 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0856000000000001e-05 2024-05-15T13:23:32.471 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.471 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.472 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:23:32.472 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:23.041340+0000", 2024-05-15T13:23:32.472 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8114e-05 2024-05-15T13:23:32.472 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.472 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:32.472 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.472 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.472 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.472 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.26299.0:174503 8.8 e798/739 8:1cd0c7cd:::benchmark_data_smithi177_149673_object84441:head v 798'11001, mlcod=798'10998)", 2024-05-15T13:23:32.472 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:22:26.630325+0000", 2024-05-15T13:23:32.472 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 65.748815293999996, 2024-05-15T13:23:32.472 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.036318105000000003, 2024-05-15T13:23:32.472 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:23:32.472 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:23:32.472 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:23:32.473 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.473 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:23:32.473 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:26.630325+0000", 2024-05-15T13:23:32.473 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.473 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.473 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.473 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:23:32.473 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:26.630325+0000", 2024-05-15T13:23:32.473 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.473 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.473 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.473 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:23:32.474 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:26.630327+0000", 2024-05-15T13:23:32.474 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3379999999999999e-06 2024-05-15T13:23:32.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.474 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.474 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:23:32.474 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:26.630341+0000", 2024-05-15T13:23:32.474 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4538e-05 2024-05-15T13:23:32.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.474 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.474 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:23:32.474 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:26.630355+0000", 2024-05-15T13:23:32.474 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3903999999999999e-05 2024-05-15T13:23:32.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.475 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.475 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:23:32.475 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:26.630364+0000", 2024-05-15T13:23:32.475 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5539999999999998e-06 2024-05-15T13:23:32.475 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.475 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.475 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:23:32.475 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:26.635058+0000", 2024-05-15T13:23:32.475 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046940879999999999 2024-05-15T13:23:32.475 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.475 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.475 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:23:32.475 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:26.635080+0000", 2024-05-15T13:23:32.475 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2342000000000001e-05 2024-05-15T13:23:32.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.476 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.476 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:23:32.476 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:26.666596+0000", 2024-05-15T13:23:32.476 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.031515804000000001 2024-05-15T13:23:32.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.476 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.476 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:23:32.476 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:26.666643+0000", 2024-05-15T13:23:32.476 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7537000000000002e-05 2024-05-15T13:23:32.476 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.476 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:32.476 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.476 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.477 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:175015 8.0 8:056c4c81:::benchmark_data_smithi177_149673_object84953:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e798)", 2024-05-15T13:23:32.477 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:22:27.277039+0000", 2024-05-15T13:23:32.477 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 65.102101650999998, 2024-05-15T13:23:32.477 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.036863595999999998, 2024-05-15T13:23:32.477 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:23:32.477 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:23:32.477 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:23:32.477 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:23:32.477 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:23:32.477 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 175015 2024-05-15T13:23:32.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.477 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:23:32.477 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.477 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:23:32.477 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:27.277039+0000", 2024-05-15T13:23:32.478 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.478 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.478 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:23:32.478 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:27.277039+0000", 2024-05-15T13:23:32.478 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.478 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.478 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:23:32.478 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:27.277039+0000", 2024-05-15T13:23:32.478 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5999999999999999e-07 2024-05-15T13:23:32.478 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.478 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:23:32.479 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:27.277042+0000", 2024-05-15T13:23:32.479 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8700000000000001e-06 2024-05-15T13:23:32.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.479 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:23:32.479 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:27.277050+0000", 2024-05-15T13:23:32.479 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6359999999999994e-06 2024-05-15T13:23:32.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.479 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:23:32.479 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:27.277055+0000", 2024-05-15T13:23:32.479 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2449999999999998e-06 2024-05-15T13:23:32.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.480 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:23:32.480 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:27.282461+0000", 2024-05-15T13:23:32.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0054063540000000004 2024-05-15T13:23:32.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.480 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.480 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:23:32.480 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:27.282598+0000", 2024-05-15T13:23:32.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013620400000000001 2024-05-15T13:23:32.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.480 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.480 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:23:32.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:27.282726+0000", 2024-05-15T13:23:32.481 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012840799999999999 2024-05-15T13:23:32.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:23:32.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:27.286624+0000", 2024-05-15T13:23:32.481 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038982560000000001 2024-05-15T13:23:32.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:27.287088+0000", 2024-05-15T13:23:32.481 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00046385000000000003 2024-05-15T13:23:32.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.482 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:27.313770+0000", 2024-05-15T13:23:32.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.026682035 2024-05-15T13:23:32.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.482 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:23:32.482 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:27.313841+0000", 2024-05-15T13:23:32.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0516e-05 2024-05-15T13:23:32.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.482 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:23:32.482 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:27.313902+0000", 2024-05-15T13:23:32.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1761999999999996e-05 2024-05-15T13:23:32.482 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.482 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:32.483 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.483 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:175720 8.0 8:09df40d3:::benchmark_data_smithi177_149673_object85658:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e798)", 2024-05-15T13:23:32.483 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:22:28.182160+0000", 2024-05-15T13:23:32.483 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 64.196980460999995, 2024-05-15T13:23:32.483 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.039232786999999998, 2024-05-15T13:23:32.483 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:23:32.483 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:23:32.483 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:23:32.483 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:23:32.483 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:23:32.483 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 175720 2024-05-15T13:23:32.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.484 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:23:32.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.484 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:23:32.484 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.182160+0000", 2024-05-15T13:23:32.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.484 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:23:32.484 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.182160+0000", 2024-05-15T13:23:32.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.484 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:23:32.484 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.182161+0000", 2024-05-15T13:23:32.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2049999999999999e-06 2024-05-15T13:23:32.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:23:32.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.182167+0000", 2024-05-15T13:23:32.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1800000000000001e-06 2024-05-15T13:23:32.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:23:32.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.182181+0000", 2024-05-15T13:23:32.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3424999999999999e-05 2024-05-15T13:23:32.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:23:32.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.182190+0000", 2024-05-15T13:23:32.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9849999999999995e-06 2024-05-15T13:23:32.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:23:32.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.186557+0000", 2024-05-15T13:23:32.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0043670710000000001 2024-05-15T13:23:32.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:23:32.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.186648+0000", 2024-05-15T13:23:32.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1297000000000004e-05 2024-05-15T13:23:32.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:23:32.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.186743+0000", 2024-05-15T13:23:32.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4363000000000004e-05 2024-05-15T13:23:32.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:23:32.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.196320+0000", 2024-05-15T13:23:32.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0095775300000000008 2024-05-15T13:23:32.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.202084+0000", 2024-05-15T13:23:32.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0057634720000000004 2024-05-15T13:23:32.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.221269+0000", 2024-05-15T13:23:32.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.019184995999999999 2024-05-15T13:23:32.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:23:32.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.221341+0000", 2024-05-15T13:23:32.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2885000000000004e-05 2024-05-15T13:23:32.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:23:32.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.221393+0000", 2024-05-15T13:23:32.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1378000000000002e-05 2024-05-15T13:23:32.489 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.489 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:32.489 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.489 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:175729 8.0 8:0f8897df:::benchmark_data_smithi177_149673_object85667:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e798)", 2024-05-15T13:23:32.489 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:22:28.182209+0000", 2024-05-15T13:23:32.489 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 64.196931264, 2024-05-15T13:23:32.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.03931668, 2024-05-15T13:23:32.489 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:23:32.489 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:23:32.489 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:23:32.489 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:23:32.489 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:23:32.490 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 175729 2024-05-15T13:23:32.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.490 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:23:32.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:23:32.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.182209+0000", 2024-05-15T13:23:32.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:23:32.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.182209+0000", 2024-05-15T13:23:32.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:23:32.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.182210+0000", 2024-05-15T13:23:32.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5499999999999998e-07 2024-05-15T13:23:32.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.491 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:23:32.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.182213+0000", 2024-05-15T13:23:32.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8720000000000002e-06 2024-05-15T13:23:32.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.491 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:23:32.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.182219+0000", 2024-05-15T13:23:32.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8129999999999998e-06 2024-05-15T13:23:32.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:23:32.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.182225+0000", 2024-05-15T13:23:32.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.412e-06 2024-05-15T13:23:32.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:23:32.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.191479+0000", 2024-05-15T13:23:32.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0092545809999999996 2024-05-15T13:23:32.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:23:32.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.191615+0000", 2024-05-15T13:23:32.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013537400000000001 2024-05-15T13:23:32.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.493 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:23:32.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.191743+0000", 2024-05-15T13:23:32.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012815 2024-05-15T13:23:32.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.493 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:23:32.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.196475+0000", 2024-05-15T13:23:32.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0047320139999999997 2024-05-15T13:23:32.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.493 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.202102+0000", 2024-05-15T13:23:32.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0056274369999999999 2024-05-15T13:23:32.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.221454+0000", 2024-05-15T13:23:32.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.019351323 2024-05-15T13:23:32.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:23:32.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.221492+0000", 2024-05-15T13:23:32.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8825000000000002e-05 2024-05-15T13:23:32.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:23:32.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:28.221526+0000", 2024-05-15T13:23:32.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3423999999999997e-05 2024-05-15T13:23:32.495 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.495 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:32.495 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.495 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:177235 8.0 8:0bf062a5:::benchmark_data_smithi177_149673_object87173:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e798)", 2024-05-15T13:23:32.495 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:22:30.035248+0000", 2024-05-15T13:23:32.495 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 62.343892580999999, 2024-05-15T13:23:32.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.042007252000000002, 2024-05-15T13:23:32.495 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:23:32.495 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:23:32.495 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:23:32.496 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:23:32.496 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:23:32.496 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 177235 2024-05-15T13:23:32.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.496 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:23:32.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:23:32.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.035248+0000", 2024-05-15T13:23:32.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:23:32.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.035248+0000", 2024-05-15T13:23:32.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:23:32.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.035249+0000", 2024-05-15T13:23:32.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.229e-06 2024-05-15T13:23:32.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:23:32.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.035261+0000", 2024-05-15T13:23:32.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.152e-05 2024-05-15T13:23:32.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:23:32.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.035277+0000", 2024-05-15T13:23:32.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6333000000000001e-05 2024-05-15T13:23:32.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:23:32.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.035285+0000", 2024-05-15T13:23:32.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3280000000000006e-06 2024-05-15T13:23:32.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:23:32.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.039278+0000", 2024-05-15T13:23:32.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039928660000000003 2024-05-15T13:23:32.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:23:32.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.039395+0000", 2024-05-15T13:23:32.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000116323 2024-05-15T13:23:32.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:23:32.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.039506+0000", 2024-05-15T13:23:32.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011182499999999999 2024-05-15T13:23:32.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:23:32.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.044040+0000", 2024-05-15T13:23:32.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.004533937 2024-05-15T13:23:32.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.045378+0000", 2024-05-15T13:23:32.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0013377879999999999 2024-05-15T13:23:32.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.077155+0000", 2024-05-15T13:23:32.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.031777092999999999 2024-05-15T13:23:32.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:23:32.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.077220+0000", 2024-05-15T13:23:32.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4991000000000002e-05 2024-05-15T13:23:32.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:23:32.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.077255+0000", 2024-05-15T13:23:32.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5018999999999998e-05 2024-05-15T13:23:32.501 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.501 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:32.501 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.501 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:177611 8.0 8:0bc9fb9c:::benchmark_data_smithi177_149673_object87549:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e798)", 2024-05-15T13:23:32.501 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:22:30.493539+0000", 2024-05-15T13:23:32.501 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 61.885601227000002, 2024-05-15T13:23:32.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.054474819000000001, 2024-05-15T13:23:32.501 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:23:32.502 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:23:32.502 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:23:32.502 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:23:32.502 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:23:32.502 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 177611 2024-05-15T13:23:32.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.502 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:23:32.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:23:32.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.493539+0000", 2024-05-15T13:23:32.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:23:32.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.493539+0000", 2024-05-15T13:23:32.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:23:32.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.493540+0000", 2024-05-15T13:23:32.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2800000000000003e-07 2024-05-15T13:23:32.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:23:32.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.493542+0000", 2024-05-15T13:23:32.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9400000000000001e-06 2024-05-15T13:23:32.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:23:32.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.493549+0000", 2024-05-15T13:23:32.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1600000000000001e-06 2024-05-15T13:23:32.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:23:32.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.493553+0000", 2024-05-15T13:23:32.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7979999999999996e-06 2024-05-15T13:23:32.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:23:32.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.502706+0000", 2024-05-15T13:23:32.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00915295 2024-05-15T13:23:32.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:23:32.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.502789+0000", 2024-05-15T13:23:32.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2750999999999994e-05 2024-05-15T13:23:32.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:23:32.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.502874+0000", 2024-05-15T13:23:32.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4727000000000004e-05 2024-05-15T13:23:32.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:23:32.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.509244+0000", 2024-05-15T13:23:32.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0063700370000000003 2024-05-15T13:23:32.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.512785+0000", 2024-05-15T13:23:32.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003541326 2024-05-15T13:23:32.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.547891+0000", 2024-05-15T13:23:32.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.035105405999999999 2024-05-15T13:23:32.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:23:32.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.547959+0000", 2024-05-15T13:23:32.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8707000000000005e-05 2024-05-15T13:23:32.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:23:32.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.548014+0000", 2024-05-15T13:23:32.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4688999999999998e-05 2024-05-15T13:23:32.507 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.507 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:32.508 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.508 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:177616 8.0 8:096b7157:::benchmark_data_smithi177_149673_object87554:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e798)", 2024-05-15T13:23:32.508 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:22:30.499487+0000", 2024-05-15T13:23:32.508 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 61.879653132000001, 2024-05-15T13:23:32.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.048633316000000003, 2024-05-15T13:23:32.508 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:23:32.508 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:23:32.508 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:23:32.508 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:23:32.508 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:23:32.508 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 177616 2024-05-15T13:23:32.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.509 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:23:32.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:23:32.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.499487+0000", 2024-05-15T13:23:32.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:23:32.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.499487+0000", 2024-05-15T13:23:32.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:23:32.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.499489+0000", 2024-05-15T13:23:32.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.189e-06 2024-05-15T13:23:32.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:23:32.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.499494+0000", 2024-05-15T13:23:32.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9649999999999999e-06 2024-05-15T13:23:32.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:23:32.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.499503+0000", 2024-05-15T13:23:32.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.448e-06 2024-05-15T13:23:32.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:23:32.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.499509+0000", 2024-05-15T13:23:32.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8579999999999996e-06 2024-05-15T13:23:32.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:23:32.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.505705+0000", 2024-05-15T13:23:32.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0061958430000000004 2024-05-15T13:23:32.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:23:32.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.505776+0000", 2024-05-15T13:23:32.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1407000000000003e-05 2024-05-15T13:23:32.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:23:32.512 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.505854+0000", 2024-05-15T13:23:32.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8331000000000006e-05 2024-05-15T13:23:32.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:23:32.512 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.509355+0000", 2024-05-15T13:23:32.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035008040000000002 2024-05-15T13:23:32.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.512833+0000", 2024-05-15T13:23:32.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034781679999999998 2024-05-15T13:23:32.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.548046+0000", 2024-05-15T13:23:32.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.035212833999999998 2024-05-15T13:23:32.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:23:32.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.548081+0000", 2024-05-15T13:23:32.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5244999999999999e-05 2024-05-15T13:23:32.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.514 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:23:32.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:30.548121+0000", 2024-05-15T13:23:32.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9224000000000003e-05 2024-05-15T13:23:32.514 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.514 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:32.514 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.514 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:178714 8.0 8:088c85d4:::benchmark_data_smithi177_149673_object88652:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e799)", 2024-05-15T13:23:32.514 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:22:31.886879+0000", 2024-05-15T13:23:32.514 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 60.492261308000003, 2024-05-15T13:23:32.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.037953923000000001, 2024-05-15T13:23:32.515 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:23:32.515 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:23:32.515 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:23:32.515 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:23:32.515 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:23:32.515 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 178714 2024-05-15T13:23:32.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.515 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:23:32.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:23:32.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:31.886879+0000", 2024-05-15T13:23:32.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:23:32.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:31.886879+0000", 2024-05-15T13:23:32.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:23:32.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:23:32.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:31.886880+0000", 2024-05-15T13:23:32.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3500000000000005e-07 2024-05-15T13:23:32.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:23:32.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:31.886884+0000", 2024-05-15T13:23:32.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1110000000000001e-06 2024-05-15T13:23:32.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.517 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:23:32.517 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:31.886896+0000", 2024-05-15T13:23:32.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.184e-05 2024-05-15T13:23:32.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.517 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:23:32.517 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:31.886905+0000", 2024-05-15T13:23:32.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8540000000000003e-06 2024-05-15T13:23:32.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.517 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:23:32.517 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:31.894134+0000", 2024-05-15T13:23:32.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0072294020000000002 2024-05-15T13:23:32.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:23:32.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:31.894268+0000", 2024-05-15T13:23:32.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000134263 2024-05-15T13:23:32.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,9", 2024-05-15T13:23:32.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:31.894387+0000", 2024-05-15T13:23:32.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000118187 2024-05-15T13:23:32.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:31.900686+0000", 2024-05-15T13:23:32.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0062993830000000004 2024-05-15T13:23:32.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:23:32.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:31.906411+0000", 2024-05-15T13:23:32.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0057247679999999999 2024-05-15T13:23:32.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:32.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:31.924725+0000", 2024-05-15T13:23:32.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.018314006000000001 2024-05-15T13:23:32.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:23:32.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:31.924784+0000", 2024-05-15T13:23:32.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8755000000000002e-05 2024-05-15T13:23:32.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:23:32.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:23:32.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:23:32.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:22:31.924833+0000", 2024-05-15T13:23:32.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9518999999999998e-05 2024-05-15T13:23:32.520 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.520 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:32.520 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.520 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:23:32.520 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:23:32.520 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:23:32.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:32.522 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:23:32.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:23:32.522 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:23:32.623 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:23:32.656 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:32.677 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:23:32.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:32.681 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:23:32.782 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:23:32.835 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:32.995 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:23:32.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:32.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:23:33.100 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:23:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:33.297 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 21 16 24210 24194 71.9961 7.94196 0.00354522 0.0138852 2024-05-15T13:23:33.608 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:23:33.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:33.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:23:33.620 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-05-15T13:23:33.707 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:33 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:23:33.707 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:33 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:23:33.707 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:33 smithi027 ceph-mon[39795]: osdmap e824: 12 total, 12 up, 10 in 2024-05-15T13:23:33.707 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:33 smithi027 ceph-mon[39795]: pgmap v2140: 37 pgs: 37 active+clean; 1.4 GiB data, 12 GiB used, 883 GiB / 894 GiB avail 2024-05-15T13:23:33.713 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:23:33.730 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:23:33.760 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:33.790 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:33 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:23:33.790 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:33 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:23:33.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:33 smithi120 ceph-mon[43961]: osdmap e824: 12 total, 12 up, 10 in 2024-05-15T13:23:33.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:33 smithi120 ceph-mon[43961]: pgmap v2140: 37 pgs: 37 active+clean; 1.4 GiB data, 12 GiB used, 883 GiB / 894 GiB avail 2024-05-15T13:23:33.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:33 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:23:33.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:33 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:23:33.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:33 smithi177 ceph-mon[44303]: osdmap e824: 12 total, 12 up, 10 in 2024-05-15T13:23:33.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:33 smithi177 ceph-mon[44303]: pgmap v2140: 37 pgs: 37 active+clean; 1.4 GiB data, 12 GiB used, 883 GiB / 894 GiB avail 2024-05-15T13:23:34.034 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:23:34.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:34.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:23:34.139 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:23:34.194 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:34.297 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 22 16 27010 26994 76.6769 175 0.00485506 0.0130379 2024-05-15T13:23:34.402 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:23:34.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:34.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:23:34.507 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:23:34.561 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:34.591 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:23:34.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:34.596 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:23:34.646 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:23:34.698 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:23:34.741 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:35.262 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:35 smithi120 ceph-mon[43961]: osdmap e825: 12 total, 12 up, 10 in 2024-05-15T13:23:35.262 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:35 smithi120 ceph-mon[43961]: pgmap v2142: 37 pgs: 37 active+clean; 1.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 12 MiB/s wr, 194 op/s 2024-05-15T13:23:35.262 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:35 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/752203755' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:23:35.262 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:35 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:23:35.273 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:23:35.297 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 23 16 29661 29645 80.546 165.688 0.00736829 0.012411 2024-05-15T13:23:35.398 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:23:35.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:35.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:23:35.420 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:23:35.501 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:23:35.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:35 smithi027 ceph-mon[39795]: osdmap e825: 12 total, 12 up, 10 in 2024-05-15T13:23:35.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:35 smithi027 ceph-mon[39795]: pgmap v2142: 37 pgs: 37 active+clean; 1.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 12 MiB/s wr, 194 op/s 2024-05-15T13:23:35.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:35 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/752203755' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:23:35.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:35 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:23:35.538 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:35 smithi177 ceph-mon[44303]: osdmap e825: 12 total, 12 up, 10 in 2024-05-15T13:23:35.538 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:35 smithi177 ceph-mon[44303]: pgmap v2142: 37 pgs: 37 active+clean; 1.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 12 MiB/s wr, 194 op/s 2024-05-15T13:23:35.538 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:35 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/752203755' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:23:35.538 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:35 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:23:35.540 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:35.573 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:23:35.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:35.579 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:23:35.680 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:23:35.721 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:35.750 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:23:35.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:35.755 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:23:35.856 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:23:35.903 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:36.264 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:36 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:23:36.264 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:36 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:23:36.264 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:36 smithi120 ceph-mon[43961]: osdmap e826: 12 total, 12 up, 10 in 2024-05-15T13:23:36.297 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 24 16 32216 32200 83.8428 159.688 0.00261459 0.0119085 2024-05-15T13:23:36.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:36 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:23:36.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:36 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:23:36.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:36 smithi027 ceph-mon[39795]: osdmap e826: 12 total, 12 up, 10 in 2024-05-15T13:23:36.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:36 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:23:36.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:36 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:23:36.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:36 smithi177 ceph-mon[44303]: osdmap e826: 12 total, 12 up, 10 in 2024-05-15T13:23:36.796 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:23:36.797 INFO:teuthology.orchestra.run.smithi120.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-05-15T13:23:36.797 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:23:37.297 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 25 16 34511 34495 86.226 143.438 0.00397441 0.0115931 2024-05-15T13:23:37.531 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:37 smithi027 ceph-mon[39795]: 1.11 scrub starts 2024-05-15T13:23:37.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:37 smithi027 ceph-mon[39795]: 1.11 scrub ok 2024-05-15T13:23:37.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:37 smithi027 ceph-mon[39795]: pgmap v2144: 37 pgs: 37 active+clean; 1.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 33 MiB/s wr, 532 op/s 2024-05-15T13:23:37.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:37 smithi120 ceph-mon[43961]: 1.11 scrub starts 2024-05-15T13:23:37.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:37 smithi120 ceph-mon[43961]: 1.11 scrub ok 2024-05-15T13:23:37.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:37 smithi120 ceph-mon[43961]: pgmap v2144: 37 pgs: 37 active+clean; 1.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 33 MiB/s wr, 532 op/s 2024-05-15T13:23:37.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:37 smithi177 ceph-mon[44303]: 1.11 scrub starts 2024-05-15T13:23:37.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:37 smithi177 ceph-mon[44303]: 1.11 scrub ok 2024-05-15T13:23:37.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:37 smithi177 ceph-mon[44303]: pgmap v2144: 37 pgs: 37 active+clean; 1.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 33 MiB/s wr, 532 op/s 2024-05-15T13:23:38.160 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:23:38.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:38.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:23:38.266 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:23:38.295 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:38.297 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 26 16 36761 36745 88.3177 140.625 0.00180233 0.0112947 2024-05-15T13:23:38.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:38 smithi120 ceph-mon[43961]: 1.6 scrub starts 2024-05-15T13:23:38.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:38 smithi120 ceph-mon[43961]: 1.6 scrub ok 2024-05-15T13:23:38.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:38 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:23:38.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:38 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:23:38.543 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:23:38.589 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:38 smithi027 ceph-mon[39795]: 1.6 scrub starts 2024-05-15T13:23:38.589 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:38 smithi027 ceph-mon[39795]: 1.6 scrub ok 2024-05-15T13:23:38.589 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:38 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:23:38.589 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:38 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:23:38.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:38 smithi177 ceph-mon[44303]: 1.6 scrub starts 2024-05-15T13:23:38.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:38 smithi177 ceph-mon[44303]: 1.6 scrub ok 2024-05-15T13:23:38.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:38 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:23:38.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:38 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:23:38.981 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:23:39.297 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 27 15 38819 38804 89.8125 128.688 0.00715414 0.01113 2024-05-15T13:23:39.435 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:39 smithi120 ceph-mon[43961]: 1.13 scrub starts 2024-05-15T13:23:39.436 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:39 smithi120 ceph-mon[43961]: 1.13 scrub ok 2024-05-15T13:23:39.436 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:39 smithi120 ceph-mon[43961]: osdmap e827: 12 total, 12 up, 10 in 2024-05-15T13:23:39.436 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:39 smithi120 ceph-mon[43961]: 1.1 scrub starts 2024-05-15T13:23:39.436 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:39 smithi120 ceph-mon[43961]: 1.1 scrub ok 2024-05-15T13:23:39.436 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:39 smithi120 ceph-mon[43961]: pgmap v2146: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 2.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-05-15T13:23:39.450 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:23:39.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:39.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:23:39.555 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:23:39.597 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:39.620 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:23:39.651 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:23:39.683 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:23:39.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:39 smithi177 ceph-mon[44303]: 1.13 scrub starts 2024-05-15T13:23:39.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:39 smithi177 ceph-mon[44303]: 1.13 scrub ok 2024-05-15T13:23:39.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:39 smithi177 ceph-mon[44303]: osdmap e827: 12 total, 12 up, 10 in 2024-05-15T13:23:39.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:39 smithi177 ceph-mon[44303]: 1.1 scrub starts 2024-05-15T13:23:39.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:39 smithi177 ceph-mon[44303]: 1.1 scrub ok 2024-05-15T13:23:39.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:39 smithi177 ceph-mon[44303]: pgmap v2146: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 2.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-05-15T13:23:39.719 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:23:39.757 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:23:39.770 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:23:39.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:39.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:23:39.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:39 smithi027 ceph-mon[39795]: 1.13 scrub starts 2024-05-15T13:23:39.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:39 smithi027 ceph-mon[39795]: 1.13 scrub ok 2024-05-15T13:23:39.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:39 smithi027 ceph-mon[39795]: osdmap e827: 12 total, 12 up, 10 in 2024-05-15T13:23:39.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:39 smithi027 ceph-mon[39795]: 1.1 scrub starts 2024-05-15T13:23:39.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:39 smithi027 ceph-mon[39795]: 1.1 scrub ok 2024-05-15T13:23:39.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:39 smithi027 ceph-mon[39795]: pgmap v2146: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 2.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-05-15T13:23:39.791 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:23:39.826 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:23:39.864 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:23:39.875 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:23:39.887 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:23:39.887 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:23:39.902 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:23:39.934 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:39.935 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:23:39.966 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:23:40.002 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:23:40.005 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:23:40.006 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:23:40.006 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:23:40.006 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:23:40.006 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:23:40.006 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:23:40.006 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:23:40.006 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:23:40.006 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:23:40.006 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:23:40.006 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:23:40.006 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:23:40.297 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 28 16 40841 40825 91.1156 126.312 0.00233546 0.0109244 2024-05-15T13:23:40.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:40 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T13:23:40.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:40 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T13:23:40.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:40 smithi120 ceph-mon[43961]: 1.a scrub starts 2024-05-15T13:23:40.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:40 smithi120 ceph-mon[43961]: 1.a scrub ok 2024-05-15T13:23:40.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:40 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3010374615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:23:40.589 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:23:40.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:40.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:23:40.595 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:40 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T13:23:40.595 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:40 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T13:23:40.595 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:40 smithi027 ceph-mon[39795]: 1.a scrub starts 2024-05-15T13:23:40.595 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:40 smithi027 ceph-mon[39795]: 1.a scrub ok 2024-05-15T13:23:40.595 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:40 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3010374615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:23:40.645 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-05-15T13:23:40.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:40 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T13:23:40.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:40 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T13:23:40.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:40 smithi177 ceph-mon[44303]: 1.a scrub starts 2024-05-15T13:23:40.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:40 smithi177 ceph-mon[44303]: 1.a scrub ok 2024-05-15T13:23:40.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:40 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3010374615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:23:40.696 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:23:40.745 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:40.784 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:23:40.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:40.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:23:40.889 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:23:40.935 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:41.036 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:23:41.297 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 29 16 43104 43088 92.8504 141.438 0.00697331 0.0107659 2024-05-15T13:23:41.398 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:23:41.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:41.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:23:41.505 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:23:41.553 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:41.603 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:23:41.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:41.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:23:41.607 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:41 smithi120 ceph-mon[43961]: pgmap v2147: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 2.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-05-15T13:23:41.642 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:41 smithi027 ceph-mon[39795]: pgmap v2147: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 2.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-05-15T13:23:41.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:41 smithi177 ceph-mon[44303]: pgmap v2147: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 2.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-05-15T13:23:41.708 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:23:41.736 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:42.298 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 30 16 45009 44993 93.7237 119.062 0.00636243 0.0106654 2024-05-15T13:23:42.486 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:23:42.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:42.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:23:42.596 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:23:42.639 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:42.848 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:23:42.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:42.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:23:42.953 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:23:43.009 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:43.298 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 31 16 47167 47151 95.0508 134.875 0.007287 0.0105166 2024-05-15T13:23:43.399 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:23:43.399 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:23:43.406 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:23:43.578 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:23:43.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:43.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:23:43.683 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:23:43.722 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:43.738 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:43 smithi027 ceph-mon[39795]: pgmap v2148: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 2.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-05-15T13:23:43.738 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:43 smithi027 ceph-mon[39795]: 1.10 scrub starts 2024-05-15T13:23:43.738 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:43 smithi027 ceph-mon[39795]: 1.10 scrub ok 2024-05-15T13:23:43.854 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:23:43.874 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:23:43.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:43.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:23:43.983 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:23:44.033 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:44.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:43 smithi120 ceph-mon[43961]: pgmap v2148: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 2.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-05-15T13:23:44.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:43 smithi120 ceph-mon[43961]: 1.10 scrub starts 2024-05-15T13:23:44.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:43 smithi120 ceph-mon[43961]: 1.10 scrub ok 2024-05-15T13:23:44.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:43 smithi177 ceph-mon[44303]: pgmap v2148: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 2.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-05-15T13:23:44.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:43 smithi177 ceph-mon[44303]: 1.10 scrub starts 2024-05-15T13:23:44.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:43 smithi177 ceph-mon[44303]: 1.10 scrub ok 2024-05-15T13:23:44.210 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 9, 10, 11, 7] out_osds: [4, 8] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:23:44.211 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:23:44.211 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-05-15T13:23:44.211 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:23:44.215 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-05-15T13:23:44.298 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 32 16 48880 48864 95.4256 107.062 0.00232697 0.0104325 2024-05-15T13:23:44.536 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:23:44.592 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:23:44.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:44.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:23:44.696 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:23:44.743 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:44.759 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:23:44.770 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:23:44.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:44.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:23:44.879 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:23:44.925 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:44 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/284671563' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:23:44.928 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:45.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:44 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/284671563' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:23:45.139 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:23:45.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:45.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:23:45.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:44 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/284671563' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:23:45.246 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:23:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:45.298 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 33 16 50196 50180 95.0261 82.25 0.00897042 0.0105196 2024-05-15T13:23:45.912 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:45 smithi120 ceph-mon[43961]: pgmap v2149: 37 pgs: 37 active+clean; 2.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 146 MiB/s wr, 2.34k op/s 2024-05-15T13:23:45.912 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:45 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:23:45.912 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:45 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T13:23:45.912 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:45 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/284671563' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:23:45.912 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:45 smithi120 ceph-mon[43961]: osdmap e828: 12 total, 12 up, 10 in 2024-05-15T13:23:45.953 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:23:45.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:45.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:23:45.960 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:45 smithi027 ceph-mon[39795]: pgmap v2149: 37 pgs: 37 active+clean; 2.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 146 MiB/s wr, 2.34k op/s 2024-05-15T13:23:45.960 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:45 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:23:45.960 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:45 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T13:23:45.961 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:45 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/284671563' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:23:45.961 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:45 smithi027 ceph-mon[39795]: osdmap e828: 12 total, 12 up, 10 in 2024-05-15T13:23:46.061 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:23:46.106 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:45 smithi177 ceph-mon[44303]: pgmap v2149: 37 pgs: 37 active+clean; 2.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 146 MiB/s wr, 2.34k op/s 2024-05-15T13:23:46.106 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:45 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:23:46.106 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:45 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T13:23:46.106 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:45 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/284671563' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:23:46.106 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:45 smithi177 ceph-mon[44303]: osdmap e828: 12 total, 12 up, 10 in 2024-05-15T13:23:46.109 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:46.141 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:23:46.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:46.148 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:23:46.249 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:23:46.298 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 34 16 51863 51847 95.2953 104.188 0.00260441 0.01049 2024-05-15T13:23:46.300 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:46.333 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:23:46.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:46.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:23:46.438 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:23:46.479 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:46.511 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:23:46.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:46.518 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:23:46.619 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:23:46.662 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:46.688 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:23:46.695 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:23:46.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:46.701 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:23:46.802 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:23:46.851 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:46.915 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:23:46.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:46.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:23:47.023 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:23:47.076 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:47.298 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 35 16 53843 53827 96.1073 123.75 0.00842181 0.0104009 2024-05-15T13:23:47.638 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:23:47.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:47.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:23:47.743 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:23:47.753 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:47 smithi027 ceph-mon[39795]: pgmap v2151: 37 pgs: 37 active+clean; 2.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-05-15T13:23:47.754 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:47 smithi027 ceph-mon[39795]: 1.8 scrub starts 2024-05-15T13:23:47.754 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:47 smithi027 ceph-mon[39795]: 1.8 scrub ok 2024-05-15T13:23:47.795 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:48.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:47 smithi120 ceph-mon[43961]: pgmap v2151: 37 pgs: 37 active+clean; 2.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-05-15T13:23:48.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:47 smithi120 ceph-mon[43961]: 1.8 scrub starts 2024-05-15T13:23:48.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:47 smithi120 ceph-mon[43961]: 1.8 scrub ok 2024-05-15T13:23:48.074 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:23:48.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:48.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:23:48.180 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:23:48.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:47 smithi177 ceph-mon[44303]: pgmap v2151: 37 pgs: 37 active+clean; 2.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-05-15T13:23:48.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:47 smithi177 ceph-mon[44303]: 1.8 scrub starts 2024-05-15T13:23:48.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:47 smithi177 ceph-mon[44303]: 1.8 scrub ok 2024-05-15T13:23:48.193 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:23:48.209 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:48.223 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:23:48.264 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:23:48.298 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 36 16 55682 55666 96.63 114.938 0.00908215 0.0103447 2024-05-15T13:23:48.301 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:23:48.336 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:23:48.371 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:23:48.396 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:23:48.434 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:23:48.469 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:23:48.502 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:23:48.535 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:23:48.605 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:23:48.606 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:23:48.606 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:23:48.606 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:23:48.606 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:23:48.606 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:23:48.606 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:23:48.606 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:23:48.606 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:23:48.607 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:23:48.607 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:23:48.607 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:23:48.607 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:23:48.735 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:23:49.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:48 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2900900648' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:23:49.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:48 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2900900648' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:23:49.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:48 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2900900648' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:23:49.192 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:23:49.192 INFO:teuthology.orchestra.run.smithi120.stdout: "size": 20, 2024-05-15T13:23:49.192 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 600, 2024-05-15T13:23:49.192 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [] 2024-05-15T13:23:49.192 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:23:49.299 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 37 16 57287 57271 96.7293 100.312 0.00307627 0.0103291 2024-05-15T13:23:49.574 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:23:49.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:49.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:23:49.681 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:23:49.723 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:49.752 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:23:49.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:49.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:23:49.857 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:23:49.871 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:49 smithi027 ceph-mon[39795]: pgmap v2152: 37 pgs: 37 active+clean; 3.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-05-15T13:23:49.871 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:49 smithi027 ceph-mon[39795]: osdmap e829: 12 total, 12 up, 10 in 2024-05-15T13:23:49.905 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:50.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:49 smithi120 ceph-mon[43961]: pgmap v2152: 37 pgs: 37 active+clean; 3.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-05-15T13:23:50.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:49 smithi120 ceph-mon[43961]: osdmap e829: 12 total, 12 up, 10 in 2024-05-15T13:23:50.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:49 smithi177 ceph-mon[44303]: pgmap v2152: 37 pgs: 37 active+clean; 3.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-05-15T13:23:50.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:49 smithi177 ceph-mon[44303]: osdmap e829: 12 total, 12 up, 10 in 2024-05-15T13:23:50.299 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 38 16 58913 58897 96.858 101.625 0.00801982 0.0103204 2024-05-15T13:23:50.371 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:23:50.763 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:23:50.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:50.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:23:50.868 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:23:50.910 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:50.941 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:23:50.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:50.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:23:51.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:50 smithi027 ceph-mon[39795]: 1.7 scrub starts 2024-05-15T13:23:51.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:50 smithi027 ceph-mon[39795]: 1.7 scrub ok 2024-05-15T13:23:51.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:50 smithi120 ceph-mon[43961]: 1.7 scrub starts 2024-05-15T13:23:51.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:50 smithi120 ceph-mon[43961]: 1.7 scrub ok 2024-05-15T13:23:51.047 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:23:51.097 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:51.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:50 smithi177 ceph-mon[44303]: 1.7 scrub starts 2024-05-15T13:23:51.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:50 smithi177 ceph-mon[44303]: 1.7 scrub ok 2024-05-15T13:23:51.299 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 39 16 60535 60519 96.9736 101.375 0.00553605 0.0103081 2024-05-15T13:23:51.422 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-05-15T13:23:51.464 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:23:51.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:51.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:23:51.574 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:23:51.614 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:51.647 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:23:51.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:51.653 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:23:51.754 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:23:51.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:51 smithi027 ceph-mon[39795]: pgmap v2154: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-05-15T13:23:51.795 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:51.800 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:23:51.819 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:23:51.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:51.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:23:51.922 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:23:51.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:51 smithi177 ceph-mon[44303]: pgmap v2154: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-05-15T13:23:51.958 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:52.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:51 smithi120 ceph-mon[43961]: pgmap v2154: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-05-15T13:23:52.067 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:23:52.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:52.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:23:52.175 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:23:52.219 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:52.299 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T13:23:52.300367+0000 min lat: 0.00110277 max lat: 13.6712 avg lat: 0.0103218 2024-05-15T13:23:52.299 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:23:52.299 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 40 16 62006 61990 96.8475 91.9375 0.00522464 0.0103218 2024-05-15T13:23:52.373 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:23:52.882 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:23:52.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:52.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:23:52.987 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:23:53.035 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:53.299 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 41 16 63726 63710 97.107 107.5 0.00563002 0.0102937 2024-05-15T13:23:54.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:53 smithi027 ceph-mon[39795]: pgmap v2155: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.5 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-05-15T13:23:54.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:53 smithi120 ceph-mon[43961]: pgmap v2155: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.5 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-05-15T13:23:54.048 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:23:54.062 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:23:54.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:53 smithi177 ceph-mon[44303]: pgmap v2155: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.5 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-05-15T13:23:54.300 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 42 16 65465 65449 97.3802 108.688 0.00640228 0.0102646 2024-05-15T13:23:55.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:54 smithi027 ceph-mon[39795]: from='client.16671 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:23:55.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:54 smithi120 ceph-mon[43961]: from='client.16671 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:23:55.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:54 smithi177 ceph-mon[44303]: from='client.16671 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:23:55.300 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 43 16 67152 67136 97.5673 105.438 0.00681707 0.0102456 2024-05-15T13:23:55.364 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:23:55.364 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [ 2024-05-15T13:23:55.364 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.365 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:67241 9.a 9:525b50f3:::benchmark_data_smithi177_159919_object67240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e829)", 2024-05-15T13:23:55.365 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:23:55.357296+0000", 2024-05-15T13:23:55.365 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.0076380770000000001, 2024-05-15T13:23:55.365 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0076614509999999997, 2024-05-15T13:23:55.365 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:23:55.365 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:23:55.365 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:23:55.365 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:23:55.365 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:23:55.365 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 67241 2024-05-15T13:23:55.365 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.366 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:23:55.366 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.366 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:23:55.366 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.357296+0000", 2024-05-15T13:23:55.366 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:23:55.366 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.366 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.366 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:23:55.367 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.357296+0000", 2024-05-15T13:23:55.367 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:23:55.367 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.367 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.367 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:23:55.367 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.357297+0000", 2024-05-15T13:23:55.367 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.226e-06 2024-05-15T13:23:55.367 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.368 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.368 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:23:55.368 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.357471+0000", 2024-05-15T13:23:55.368 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000173804 2024-05-15T13:23:55.368 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.368 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.368 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:23:55.368 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.357480+0000", 2024-05-15T13:23:55.369 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1989999999999995e-06 2024-05-15T13:23:55.369 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.369 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.369 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:23:55.369 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.357487+0000", 2024-05-15T13:23:55.369 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7460000000000003e-06 2024-05-15T13:23:55.369 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.369 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.369 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:23:55.370 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.357546+0000", 2024-05-15T13:23:55.370 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8449999999999998e-05 2024-05-15T13:23:55.370 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.370 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.370 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:23:55.370 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.357693+0000", 2024-05-15T13:23:55.370 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014713000000000001 2024-05-15T13:23:55.370 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.370 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.370 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,7", 2024-05-15T13:23:55.370 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.357854+0000", 2024-05-15T13:23:55.371 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000161297 2024-05-15T13:23:55.371 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.371 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.371 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:55.371 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.360419+0000", 2024-05-15T13:23:55.371 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.002564733 2024-05-15T13:23:55.371 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.371 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.371 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:23:55.371 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.364402+0000", 2024-05-15T13:23:55.371 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0039826870000000004 2024-05-15T13:23:55.371 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:23:55.372 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:23:55.372 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:23:55.372 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.372 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.372 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26536.0:67236 9.d e829/809 9:b63cec8f:::benchmark_data_smithi177_159919_object67235:head v 829'4229, mlcod=829'4227)", 2024-05-15T13:23:55.372 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:23:55.359388+0000", 2024-05-15T13:23:55.372 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.0055466259999999998, 2024-05-15T13:23:55.372 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0056251499999999998, 2024-05-15T13:23:55.372 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:23:55.372 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "started", 2024-05-15T13:23:55.372 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:23:55.372 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.373 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:23:55.373 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.359388+0000", 2024-05-15T13:23:55.373 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:23:55.373 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.373 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.373 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:23:55.373 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.359388+0000", 2024-05-15T13:23:55.373 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:23:55.373 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.373 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.373 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:23:55.373 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.359388+0000", 2024-05-15T13:23:55.373 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3900000000000004e-07 2024-05-15T13:23:55.374 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.374 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.374 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:23:55.374 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.359410+0000", 2024-05-15T13:23:55.374 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1109999999999999e-05 2024-05-15T13:23:55.374 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.374 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.374 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:23:55.374 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.359419+0000", 2024-05-15T13:23:55.374 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0990000000000005e-06 2024-05-15T13:23:55.374 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.375 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.375 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:23:55.375 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.359425+0000", 2024-05-15T13:23:55.375 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.0719999999999996e-06 2024-05-15T13:23:55.375 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.375 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.375 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:23:55.375 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.363270+0000", 2024-05-15T13:23:55.375 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0038450950000000002 2024-05-15T13:23:55.375 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.375 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.375 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:23:55.376 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.363282+0000", 2024-05-15T13:23:55.376 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2554999999999999e-05 2024-05-15T13:23:55.376 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:23:55.376 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:23:55.376 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:23:55.376 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.376 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.376 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26536.0:67238 9.d e829/809)", 2024-05-15T13:23:55.376 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:23:55.361631+0000", 2024-05-15T13:23:55.376 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.003303655, 2024-05-15T13:23:55.376 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0034150080000000002, 2024-05-15T13:23:55.376 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:23:55.377 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "queued for pg", 2024-05-15T13:23:55.377 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:23:55.377 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.377 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:23:55.377 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.361631+0000", 2024-05-15T13:23:55.377 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:23:55.377 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.377 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.377 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:23:55.377 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.361631+0000", 2024-05-15T13:23:55.377 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:23:55.377 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.378 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.378 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:23:55.378 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.361631+0000", 2024-05-15T13:23:55.378 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0399999999999995e-07 2024-05-15T13:23:55.378 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.378 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.378 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:23:55.378 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.361652+0000", 2024-05-15T13:23:55.378 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0852999999999998e-05 2024-05-15T13:23:55.378 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.378 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.379 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:23:55.379 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.361666+0000", 2024-05-15T13:23:55.379 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4161e-05 2024-05-15T13:23:55.379 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.379 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.379 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:23:55.379 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.361673+0000", 2024-05-15T13:23:55.379 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3400000000000003e-06 2024-05-15T13:23:55.379 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:23:55.379 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:23:55.379 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:23:55.379 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.380 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.380 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:67248 9.f 9:f9234bcb:::benchmark_data_smithi177_159919_object67247:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e829)", 2024-05-15T13:23:55.380 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:23:55.358997+0000", 2024-05-15T13:23:55.380 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.0059374909999999996, 2024-05-15T13:23:55.380 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.006109784, 2024-05-15T13:23:55.380 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:23:55.380 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:23:55.380 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:23:55.380 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:23:55.380 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:23:55.380 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 67248 2024-05-15T13:23:55.380 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.381 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:23:55.381 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.381 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:23:55.381 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.358997+0000", 2024-05-15T13:23:55.381 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:23:55.381 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.381 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.381 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:23:55.381 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.358997+0000", 2024-05-15T13:23:55.381 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:23:55.381 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.381 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.381 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:23:55.382 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.358998+0000", 2024-05-15T13:23:55.382 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0299999999999997e-07 2024-05-15T13:23:55.382 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.382 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.382 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:23:55.382 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.361983+0000", 2024-05-15T13:23:55.382 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0029856259999999999 2024-05-15T13:23:55.382 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.382 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.382 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:23:55.382 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.361998+0000", 2024-05-15T13:23:55.383 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4418e-05 2024-05-15T13:23:55.383 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.383 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.383 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:23:55.383 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.362005+0000", 2024-05-15T13:23:55.383 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.6129999999999998e-06 2024-05-15T13:23:55.383 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.383 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.383 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:23:55.383 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.362060+0000", 2024-05-15T13:23:55.383 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.4486e-05 2024-05-15T13:23:55.383 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.384 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.384 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:23:55.384 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.362208+0000", 2024-05-15T13:23:55.384 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014810999999999999 2024-05-15T13:23:55.384 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.384 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.384 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,9", 2024-05-15T13:23:55.385 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.362348+0000", 2024-05-15T13:23:55.385 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014050299999999999 2024-05-15T13:23:55.385 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.385 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.385 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:23:55.385 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.364501+0000", 2024-05-15T13:23:55.385 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0021520150000000002 2024-05-15T13:23:55.386 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.386 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.386 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:23:55.386 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.364882+0000", 2024-05-15T13:23:55.386 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00038106199999999999 2024-05-15T13:23:55.386 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:23:55.386 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:23:55.386 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:23:55.387 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.387 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.387 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26536.0:67247 9.d e829/809)", 2024-05-15T13:23:55.387 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:23:55.364292+0000", 2024-05-15T13:23:55.387 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.00064260299999999995, 2024-05-15T13:23:55.387 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00089364499999999999, 2024-05-15T13:23:55.387 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:23:55.387 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "queued for pg", 2024-05-15T13:23:55.387 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:23:55.388 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.388 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:23:55.388 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.364292+0000", 2024-05-15T13:23:55.388 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:23:55.388 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.388 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.388 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:23:55.388 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.364292+0000", 2024-05-15T13:23:55.388 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:23:55.388 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.388 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.388 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:23:55.388 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.364292+0000", 2024-05-15T13:23:55.389 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4199999999999995e-07 2024-05-15T13:23:55.389 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.389 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.389 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:23:55.389 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.364315+0000", 2024-05-15T13:23:55.389 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2963999999999999e-05 2024-05-15T13:23:55.389 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.389 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.389 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:23:55.389 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.364324+0000", 2024-05-15T13:23:55.389 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.986e-06 2024-05-15T13:23:55.389 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:23:55.389 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:23:55.390 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:23:55.390 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:55.364332+0000", 2024-05-15T13:23:55.390 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3610000000000003e-06 2024-05-15T13:23:55.390 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:23:55.390 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:23:55.390 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:23:55.390 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:23:55.390 INFO:teuthology.orchestra.run.smithi120.stdout: ], 2024-05-15T13:23:55.390 INFO:teuthology.orchestra.run.smithi120.stdout: "num_ops": 5 2024-05-15T13:23:55.390 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:23:55.712 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2155,"stamp":"2024-05-15T13:23:52.680332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3764716064,"num_objects":57438,"num_object_clones":0,"num_object_copies":172314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57436,"num_write_kb":3675904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60355,"ondisk_log_size":60355,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":112,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":18857668,"kb_used_data":10918572,"kb_used_omap":15,"kb_used_meta":7938992,"kb_avail":918593852,"statfs":{"total":959950356480,"available":940640104448,"internally_reserved":0,"allocated":11180617728,"data_stored":11176765794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15951,"internal_metadata":8129528241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":781254656,"num_objects":11921,"num_object_clones":0,"num_object_copies":35763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11921,"num_write_kb":762944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003094"},"pg_stats":[{"pgid":"1.14","version":"94'139","reported_seq":1024,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-05-15T13:23:44.751087+0000","last_change":"2024-05-15T13:23:11.573603+0000","last_active":"2024-05-15T13:23:44.751087+0000","last_peered":"2024-05-15T13:23:44.751087+0000","last_clean":"2024-05-15T13:23:44.751087+0000","last_became_active":"2024-05-15T13:19:41.190796+0000","last_became_peered":"2024-05-15T13:19:41.190796+0000","last_unstale":"2024-05-15T13:23:44.751087+0000","last_undegraded":"2024-05-15T13:23:44.751087+0000","last_fullsized":"2024-05-15T13:23:44.751087+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:11.573492+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:23:11.573492+0000","last_clean_scrub_stamp":"2024-05-15T13:23:11.573492+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:27.815536+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"829'3680","reported_seq":3730,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-05-15T13:23:50.706144+0000","last_change":"2024-05-15T13:22:58.857344+0000","last_active":"2024-05-15T13:23:50.706144+0000","last_peered":"2024-05-15T13:23:50.706144+0000","last_clean":"2024-05-15T13:23:50.706144+0000","last_became_active":"2024-05-15T13:22:58.857024+0000","last_became_peered":"2024-05-15T13:22:58.857024+0000","last_unstale":"2024-05-15T13:23:50.706144+0000","last_undegraded":"2024-05-15T13:23:50.706144+0000","last_fullsized":"2024-05-15T13:23:50.706144+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":3680,"log_dups_size":0,"ondisk_log_size":3680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:06.225828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241172480,"num_objects":3680,"num_object_clones":0,"num_object_copies":11040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3680,"num_write_kb":235520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":1590,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-05-15T13:23:44.752089+0000","last_change":"2024-05-15T13:23:38.906554+0000","last_active":"2024-05-15T13:23:44.752089+0000","last_peered":"2024-05-15T13:23:44.752089+0000","last_clean":"2024-05-15T13:23:44.752089+0000","last_became_active":"2024-05-15T13:13:11.361366+0000","last_became_peered":"2024-05-15T13:13:11.361366+0000","last_unstale":"2024-05-15T13:23:44.752089+0000","last_undegraded":"2024-05-15T13:23:44.752089+0000","last_fullsized":"2024-05-15T13:23:44.752089+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:38.906446+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:23:38.906446+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:53.403210+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.c","version":"828'3530","reported_seq":3577,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-05-15T13:23:48.021961+0000","last_change":"2024-05-15T13:22:59.336651+0000","last_active":"2024-05-15T13:23:48.021961+0000","last_peered":"2024-05-15T13:23:48.021961+0000","last_clean":"2024-05-15T13:23:48.021961+0000","last_became_active":"2024-05-15T13:22:59.336347+0000","last_became_peered":"2024-05-15T13:22:59.336347+0000","last_unstale":"2024-05-15T13:23:48.021961+0000","last_undegraded":"2024-05-15T13:23:48.021961+0000","last_fullsized":"2024-05-15T13:23:48.021961+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":3530,"log_dups_size":0,"ondisk_log_size":3530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:05.826288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231342080,"num_objects":3530,"num_object_clones":0,"num_object_copies":10590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3530,"num_write_kb":225920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,0],"acting":[3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"94'139","reported_seq":1609,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-05-15T13:23:44.750979+0000","last_change":"2024-05-15T13:23:04.931444+0000","last_active":"2024-05-15T13:23:44.750979+0000","last_peered":"2024-05-15T13:23:44.750979+0000","last_clean":"2024-05-15T13:23:44.750979+0000","last_became_active":"2024-05-15T13:22:48.740313+0000","last_became_peered":"2024-05-15T13:22:48.740313+0000","last_unstale":"2024-05-15T13:23:44.750979+0000","last_undegraded":"2024-05-15T13:23:44.750979+0000","last_fullsized":"2024-05-15T13:23:44.750979+0000","mapping_epoch":805,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":806,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:04.931319+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:23:04.931319+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:12.390796+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"829'3787","reported_seq":3836,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-05-15T13:23:50.855572+0000","last_change":"2024-05-15T13:22:58.856881+0000","last_active":"2024-05-15T13:23:50.855572+0000","last_peered":"2024-05-15T13:23:50.855572+0000","last_clean":"2024-05-15T13:23:50.855572+0000","last_became_active":"2024-05-15T13:22:58.856595+0000","last_became_peered":"2024-05-15T13:22:58.856595+0000","last_unstale":"2024-05-15T13:23:50.855572+0000","last_undegraded":"2024-05-15T13:23:50.855572+0000","last_fullsized":"2024-05-15T13:23:50.855572+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":3787,"log_dups_size":0,"ondisk_log_size":3787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:09.943761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248184832,"num_objects":3787,"num_object_clones":0,"num_object_copies":11361,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3787,"num_write_kb":242368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"94'139","reported_seq":1347,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-05-15T13:23:49.259978+0000","last_change":"2024-05-15T13:22:51.887150+0000","last_active":"2024-05-15T13:23:49.259978+0000","last_peered":"2024-05-15T13:23:49.259978+0000","last_clean":"2024-05-15T13:23:49.259978+0000","last_became_active":"2024-05-15T13:14:54.577458+0000","last_became_peered":"2024-05-15T13:14:54.577458+0000","last_unstale":"2024-05-15T13:23:49.259978+0000","last_undegraded":"2024-05-15T13:23:49.259978+0000","last_fullsized":"2024-05-15T13:23:49.259978+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:51.887016+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:18:24.970285+0000","last_clean_scrub_stamp":"2024-05-15T13:22:51.887016+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:12.953549+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"829'3653","reported_seq":3702,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-05-15T13:23:50.870864+0000","last_change":"2024-05-15T13:22:58.852680+0000","last_active":"2024-05-15T13:23:50.870864+0000","last_peered":"2024-05-15T13:23:50.870864+0000","last_clean":"2024-05-15T13:23:50.870864+0000","last_became_active":"2024-05-15T13:22:58.852354+0000","last_became_peered":"2024-05-15T13:22:58.852354+0000","last_unstale":"2024-05-15T13:23:50.870864+0000","last_undegraded":"2024-05-15T13:23:50.870864+0000","last_fullsized":"2024-05-15T13:23:50.870864+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":3653,"log_dups_size":0,"ondisk_log_size":3653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:07.754518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239403008,"num_objects":3653,"num_object_clones":0,"num_object_copies":10959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3653,"num_write_kb":233792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,1],"acting":[7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"94'139","reported_seq":1564,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-05-15T13:23:44.751169+0000","last_change":"2024-05-15T13:23:37.717835+0000","last_active":"2024-05-15T13:23:44.751169+0000","last_peered":"2024-05-15T13:23:44.751169+0000","last_clean":"2024-05-15T13:23:44.751169+0000","last_became_active":"2024-05-15T13:23:30.257493+0000","last_became_peered":"2024-05-15T13:23:30.257493+0000","last_unstale":"2024-05-15T13:23:44.751169+0000","last_undegraded":"2024-05-15T13:23:44.751169+0000","last_fullsized":"2024-05-15T13:23:44.751169+0000","mapping_epoch":822,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":823,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:37.717703+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:23:37.717703+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:53.621219+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,6],"acting":[2,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.f","version":"828'3612","reported_seq":3660,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-05-15T13:23:48.707660+0000","last_change":"2024-05-15T13:23:32.734479+0000","last_active":"2024-05-15T13:23:48.707660+0000","last_peered":"2024-05-15T13:23:48.707660+0000","last_clean":"2024-05-15T13:23:48.707660+0000","last_became_active":"2024-05-15T13:22:58.857554+0000","last_became_peered":"2024-05-15T13:22:58.857554+0000","last_unstale":"2024-05-15T13:23:48.707660+0000","last_undegraded":"2024-05-15T13:23:48.707660+0000","last_fullsized":"2024-05-15T13:23:48.707660+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":3612,"log_dups_size":0,"ondisk_log_size":3612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:00.904125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236716032,"num_objects":3612,"num_object_clones":0,"num_object_copies":10836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3612,"num_write_kb":231168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"94'139","reported_seq":1347,"reported_epoch":829,"state":"active+clean+scrubbing","last_fresh":"2024-05-15T13:23:49.332114+0000","last_change":"2024-05-15T13:23:49.240227+0000","last_active":"2024-05-15T13:23:49.332114+0000","last_peered":"2024-05-15T13:23:49.332114+0000","last_clean":"2024-05-15T13:23:49.332114+0000","last_became_active":"2024-05-15T13:14:11.522490+0000","last_became_peered":"2024-05-15T13:14:11.522490+0000","last_unstale":"2024-05-15T13:23:49.332114+0000","last_undegraded":"2024-05-15T13:23:49.332114+0000","last_fullsized":"2024-05-15T13:23:49.332114+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:01.216619+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:22:01.216619+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"829'3925","reported_seq":3975,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-05-15T13:23:51.995200+0000","last_change":"2024-05-15T13:22:58.856422+0000","last_active":"2024-05-15T13:23:51.995200+0000","last_peered":"2024-05-15T13:23:51.995200+0000","last_clean":"2024-05-15T13:23:51.995200+0000","last_became_active":"2024-05-15T13:22:58.855908+0000","last_became_peered":"2024-05-15T13:22:58.855908+0000","last_unstale":"2024-05-15T13:23:51.995200+0000","last_undegraded":"2024-05-15T13:23:51.995200+0000","last_fullsized":"2024-05-15T13:23:51.995200+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":3925,"log_dups_size":0,"ondisk_log_size":3925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:25.461974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257228800,"num_objects":3925,"num_object_clones":0,"num_object_copies":11775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":251200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"94'139","reported_seq":1371,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-05-15T13:23:49.262126+0000","last_change":"2024-05-15T13:23:38.468949+0000","last_active":"2024-05-15T13:23:49.262126+0000","last_peered":"2024-05-15T13:23:49.262126+0000","last_clean":"2024-05-15T13:23:49.262126+0000","last_became_active":"2024-05-15T13:15:43.614970+0000","last_became_peered":"2024-05-15T13:15:43.614970+0000","last_unstale":"2024-05-15T13:23:49.262126+0000","last_undegraded":"2024-05-15T13:23:49.262126+0000","last_fullsized":"2024-05-15T13:23:49.262126+0000","mapping_epoch":678,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:38.468826+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:23:38.468826+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:38.790352+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,0],"acting":[9,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.a","version":"828'3418","reported_seq":3466,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-05-15T13:23:48.715314+0000","last_change":"2024-05-15T13:22:58.858222+0000","last_active":"2024-05-15T13:23:48.715314+0000","last_peered":"2024-05-15T13:23:48.715314+0000","last_clean":"2024-05-15T13:23:48.715314+0000","last_became_active":"2024-05-15T13:22:58.857879+0000","last_became_peered":"2024-05-15T13:22:58.857879+0000","last_unstale":"2024-05-15T13:23:48.715314+0000","last_undegraded":"2024-05-15T13:23:48.715314+0000","last_fullsized":"2024-05-15T13:23:48.715314+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":3418,"log_dups_size":0,"ondisk_log_size":3418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:12.095975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224002048,"num_objects":3418,"num_object_clones":0,"num_object_copies":10254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3418,"num_write_kb":218752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"94'139","reported_seq":1608,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-05-15T13:23:49.265617+0000","last_change":"2024-05-15T13:22:57.893531+0000","last_active":"2024-05-15T13:23:49.265617+0000","last_peered":"2024-05-15T13:23:49.265617+0000","last_clean":"2024-05-15T13:23:49.265617+0000","last_became_active":"2024-05-15T13:22:48.739056+0000","last_became_peered":"2024-05-15T13:22:48.739056+0000","last_unstale":"2024-05-15T13:23:49.265617+0000","last_undegraded":"2024-05-15T13:23:49.265617+0000","last_fullsized":"2024-05-15T13:23:49.265617+0000","mapping_epoch":805,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":806,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:57.893417+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.893417+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:05.565057+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"828'3552","reported_seq":3600,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-05-15T13:23:48.715359+0000","last_change":"2024-05-15T13:22:58.858310+0000","last_active":"2024-05-15T13:23:48.715359+0000","last_peered":"2024-05-15T13:23:48.715359+0000","last_clean":"2024-05-15T13:23:48.715359+0000","last_became_active":"2024-05-15T13:22:58.857910+0000","last_became_peered":"2024-05-15T13:22:58.857910+0000","last_unstale":"2024-05-15T13:23:48.715359+0000","last_undegraded":"2024-05-15T13:23:48.715359+0000","last_fullsized":"2024-05-15T13:23:48.715359+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":3552,"log_dups_size":0,"ondisk_log_size":3552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:27.081337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232783872,"num_objects":3552,"num_object_clones":0,"num_object_copies":10656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3552,"num_write_kb":227328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"94'139","reported_seq":1618,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-05-15T13:23:44.752392+0000","last_change":"2024-05-15T13:22:55.272017+0000","last_active":"2024-05-15T13:23:44.752392+0000","last_peered":"2024-05-15T13:23:44.752392+0000","last_clean":"2024-05-15T13:23:44.752392+0000","last_became_active":"2024-05-15T13:22:48.738778+0000","last_became_peered":"2024-05-15T13:22:48.738778+0000","last_unstale":"2024-05-15T13:23:44.752392+0000","last_undegraded":"2024-05-15T13:23:44.752392+0000","last_fullsized":"2024-05-15T13:23:44.752392+0000","mapping_epoch":805,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":806,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:55.271911+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:22:55.271911+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:10.155748+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.0","version":"828'3369","reported_seq":3417,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-05-15T13:23:48.150083+0000","last_change":"2024-05-15T13:22:59.208928+0000","last_active":"2024-05-15T13:23:48.150083+0000","last_peered":"2024-05-15T13:23:48.150083+0000","last_clean":"2024-05-15T13:23:48.150083+0000","last_became_active":"2024-05-15T13:22:59.208576+0000","last_became_peered":"2024-05-15T13:22:59.208576+0000","last_unstale":"2024-05-15T13:23:48.150083+0000","last_undegraded":"2024-05-15T13:23:48.150083+0000","last_fullsized":"2024-05-15T13:23:48.150083+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":3369,"log_dups_size":0,"ondisk_log_size":3369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:17.422924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220790784,"num_objects":3369,"num_object_clones":0,"num_object_copies":10107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3369,"num_write_kb":215616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"94'139","reported_seq":1585,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-05-15T13:23:47.391707+0000","last_change":"2024-05-15T13:23:47.391758+0000","last_active":"2024-05-15T13:23:47.391707+0000","last_peered":"2024-05-15T13:23:47.391707+0000","last_clean":"2024-05-15T13:23:47.391707+0000","last_became_active":"2024-05-15T13:06:46.426908+0000","last_became_peered":"2024-05-15T13:06:46.426908+0000","last_unstale":"2024-05-15T13:23:47.391707+0000","last_undegraded":"2024-05-15T13:23:47.391707+0000","last_fullsized":"2024-05-15T13:23:47.391707+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:47.391631+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:23:47.391631+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:50.197691+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"828'3386","reported_seq":3434,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-05-15T13:23:48.037305+0000","last_change":"2024-05-15T13:22:59.209016+0000","last_active":"2024-05-15T13:23:48.037305+0000","last_peered":"2024-05-15T13:23:48.037305+0000","last_clean":"2024-05-15T13:23:48.037305+0000","last_became_active":"2024-05-15T13:22:59.208619+0000","last_became_peered":"2024-05-15T13:22:59.208619+0000","last_unstale":"2024-05-15T13:23:48.037305+0000","last_undegraded":"2024-05-15T13:23:48.037305+0000","last_fullsized":"2024-05-15T13:23:48.037305+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":3386,"log_dups_size":0,"ondisk_log_size":3386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:18.411120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221904896,"num_objects":3386,"num_object_clones":0,"num_object_copies":10158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3386,"num_write_kb":216704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"94'139","reported_seq":960,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-05-15T13:23:49.265307+0000","last_change":"2024-05-15T13:23:12.032052+0000","last_active":"2024-05-15T13:23:49.265307+0000","last_peered":"2024-05-15T13:23:49.265307+0000","last_clean":"2024-05-15T13:23:49.265307+0000","last_became_active":"2024-05-15T13:19:41.191244+0000","last_became_peered":"2024-05-15T13:19:41.191244+0000","last_unstale":"2024-05-15T13:23:49.265307+0000","last_undegraded":"2024-05-15T13:23:49.265307+0000","last_fullsized":"2024-05-15T13:23:49.265307+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:12.031939+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:23:12.031939+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:20.315779+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2","version":"828'3510","reported_seq":3558,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-05-15T13:23:48.041461+0000","last_change":"2024-05-15T13:22:58.851932+0000","last_active":"2024-05-15T13:23:48.041461+0000","last_peered":"2024-05-15T13:23:48.041461+0000","last_clean":"2024-05-15T13:23:48.041461+0000","last_became_active":"2024-05-15T13:22:58.851656+0000","last_became_peered":"2024-05-15T13:22:58.851656+0000","last_unstale":"2024-05-15T13:23:48.041461+0000","last_undegraded":"2024-05-15T13:23:48.041461+0000","last_fullsized":"2024-05-15T13:23:48.041461+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":3510,"log_dups_size":0,"ondisk_log_size":3510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:27.197438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230031360,"num_objects":3510,"num_object_clones":0,"num_object_copies":10530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3510,"num_write_kb":224640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"94'139","reported_seq":1366,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-05-15T13:23:49.262268+0000","last_change":"2024-05-15T13:23:39.504665+0000","last_active":"2024-05-15T13:23:49.262268+0000","last_peered":"2024-05-15T13:23:49.262268+0000","last_clean":"2024-05-15T13:23:49.262268+0000","last_became_active":"2024-05-15T13:15:43.611913+0000","last_became_peered":"2024-05-15T13:15:43.611913+0000","last_unstale":"2024-05-15T13:23:49.262268+0000","last_undegraded":"2024-05-15T13:23:49.262268+0000","last_fullsized":"2024-05-15T13:23:49.262268+0000","mapping_epoch":678,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:39.504559+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:22:03.418345+0000","last_clean_scrub_stamp":"2024-05-15T13:23:39.504559+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:54.691049+0000","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.6","version":"828'3499","reported_seq":3547,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-05-15T13:23:48.702341+0000","last_change":"2024-05-15T13:22:58.859332+0000","last_active":"2024-05-15T13:23:48.702341+0000","last_peered":"2024-05-15T13:23:48.702341+0000","last_clean":"2024-05-15T13:23:48.702341+0000","last_became_active":"2024-05-15T13:22:58.859007+0000","last_became_peered":"2024-05-15T13:22:58.859007+0000","last_unstale":"2024-05-15T13:23:48.702341+0000","last_undegraded":"2024-05-15T13:23:48.702341+0000","last_fullsized":"2024-05-15T13:23:48.702341+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":3499,"log_dups_size":0,"ondisk_log_size":3499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:12.582799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229310464,"num_objects":3499,"num_object_clones":0,"num_object_copies":10497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3499,"num_write_kb":223936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"94'139","reported_seq":972,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-05-15T13:23:49.266066+0000","last_change":"2024-05-15T13:23:06.819279+0000","last_active":"2024-05-15T13:23:49.266066+0000","last_peered":"2024-05-15T13:23:49.266066+0000","last_clean":"2024-05-15T13:23:49.266066+0000","last_became_active":"2024-05-15T13:22:48.738361+0000","last_became_peered":"2024-05-15T13:22:48.738361+0000","last_unstale":"2024-05-15T13:23:49.266066+0000","last_undegraded":"2024-05-15T13:23:49.266066+0000","last_fullsized":"2024-05-15T13:23:49.266066+0000","mapping_epoch":805,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":806,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:06.819156+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:23:06.819156+0000","last_clean_scrub_stamp":"2024-05-15T13:23:06.819156+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:14.267479+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"829'3893","reported_seq":3943,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-05-15T13:23:52.004691+0000","last_change":"2024-05-15T13:22:58.852908+0000","last_active":"2024-05-15T13:23:52.004691+0000","last_peered":"2024-05-15T13:23:52.004691+0000","last_clean":"2024-05-15T13:23:52.004691+0000","last_became_active":"2024-05-15T13:22:58.852401+0000","last_became_peered":"2024-05-15T13:22:58.852401+0000","last_unstale":"2024-05-15T13:23:52.004691+0000","last_undegraded":"2024-05-15T13:23:52.004691+0000","last_fullsized":"2024-05-15T13:23:52.004691+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":3893,"log_dups_size":0,"ondisk_log_size":3893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:23.299876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255131648,"num_objects":3893,"num_object_clones":0,"num_object_copies":11679,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3893,"num_write_kb":249152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"94'139","reported_seq":1342,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-05-15T13:23:49.332102+0000","last_change":"2024-05-15T13:22:00.244496+0000","last_active":"2024-05-15T13:23:49.332102+0000","last_peered":"2024-05-15T13:23:49.332102+0000","last_clean":"2024-05-15T13:23:49.332102+0000","last_became_active":"2024-05-15T13:13:16.472318+0000","last_became_peered":"2024-05-15T13:13:16.472318+0000","last_unstale":"2024-05-15T13:23:49.332102+0000","last_undegraded":"2024-05-15T13:23:49.332102+0000","last_fullsized":"2024-05-15T13:23:49.332102+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:00.244355+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:22:00.244355+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"828'3425","reported_seq":3473,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-05-15T13:23:48.154591+0000","last_change":"2024-05-15T13:22:59.336585+0000","last_active":"2024-05-15T13:23:48.154591+0000","last_peered":"2024-05-15T13:23:48.154591+0000","last_clean":"2024-05-15T13:23:48.154591+0000","last_became_active":"2024-05-15T13:22:59.336258+0000","last_became_peered":"2024-05-15T13:22:59.336258+0000","last_unstale":"2024-05-15T13:23:48.154591+0000","last_undegraded":"2024-05-15T13:23:48.154591+0000","last_fullsized":"2024-05-15T13:23:48.154591+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":3425,"log_dups_size":0,"ondisk_log_size":3425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:04.475753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224460800,"num_objects":3425,"num_object_clones":0,"num_object_copies":10275,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3425,"num_write_kb":219200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"94'139","reported_seq":1008,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-05-15T13:23:44.755297+0000","last_change":"2024-05-15T13:22:56.794557+0000","last_active":"2024-05-15T13:23:44.755297+0000","last_peered":"2024-05-15T13:23:44.755297+0000","last_clean":"2024-05-15T13:23:44.755297+0000","last_became_active":"2024-05-15T13:13:16.476972+0000","last_became_peered":"2024-05-15T13:13:16.476972+0000","last_unstale":"2024-05-15T13:23:44.755297+0000","last_undegraded":"2024-05-15T13:23:44.755297+0000","last_fullsized":"2024-05-15T13:23:44.755297+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:56.794428+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:22:56.794428+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:09.214531+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"828'3470","reported_seq":3518,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-05-15T13:23:47.866596+0000","last_change":"2024-05-15T13:22:58.856916+0000","last_active":"2024-05-15T13:23:47.866596+0000","last_peered":"2024-05-15T13:23:47.866596+0000","last_clean":"2024-05-15T13:23:47.866596+0000","last_became_active":"2024-05-15T13:22:58.856523+0000","last_became_peered":"2024-05-15T13:22:58.856523+0000","last_unstale":"2024-05-15T13:23:47.866596+0000","last_undegraded":"2024-05-15T13:23:47.866596+0000","last_fullsized":"2024-05-15T13:23:47.866596+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":3470,"log_dups_size":0,"ondisk_log_size":3470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:24.728154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227409920,"num_objects":3470,"num_object_clones":0,"num_object_copies":10410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3470,"num_write_kb":222080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"94'139","reported_seq":1009,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-05-15T13:23:49.265874+0000","last_change":"2024-05-15T13:23:03.856369+0000","last_active":"2024-05-15T13:23:49.265874+0000","last_peered":"2024-05-15T13:23:49.265874+0000","last_clean":"2024-05-15T13:23:49.265874+0000","last_became_active":"2024-05-15T13:22:48.739212+0000","last_became_peered":"2024-05-15T13:22:48.739212+0000","last_unstale":"2024-05-15T13:23:49.265874+0000","last_undegraded":"2024-05-15T13:23:49.265874+0000","last_fullsized":"2024-05-15T13:23:49.265874+0000","mapping_epoch":805,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":806,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:03.856199+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:23:03.856199+0000","last_clean_scrub_stamp":"2024-05-15T13:23:03.856199+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:28.846590+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3","version":"829'3727","reported_seq":3777,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-05-15T13:23:50.699837+0000","last_change":"2024-05-15T13:22:58.857200+0000","last_active":"2024-05-15T13:23:50.699837+0000","last_peered":"2024-05-15T13:23:50.699837+0000","last_clean":"2024-05-15T13:23:50.699837+0000","last_became_active":"2024-05-15T13:22:58.856850+0000","last_became_peered":"2024-05-15T13:22:58.856850+0000","last_unstale":"2024-05-15T13:23:50.699837+0000","last_undegraded":"2024-05-15T13:23:50.699837+0000","last_fullsized":"2024-05-15T13:23:50.699837+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":3727,"log_dups_size":0,"ondisk_log_size":3727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:09.584566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244252672,"num_objects":3727,"num_object_clones":0,"num_object_copies":11181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3727,"num_write_kb":238528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"94'139","reported_seq":1609,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-05-15T13:23:44.750989+0000","last_change":"2024-05-15T13:22:56.003209+0000","last_active":"2024-05-15T13:23:44.750989+0000","last_peered":"2024-05-15T13:23:44.750989+0000","last_clean":"2024-05-15T13:23:44.750989+0000","last_became_active":"2024-05-15T13:13:16.484252+0000","last_became_peered":"2024-05-15T13:13:16.484252+0000","last_unstale":"2024-05-15T13:23:44.750989+0000","last_undegraded":"2024-05-15T13:23:44.750989+0000","last_fullsized":"2024-05-15T13:23:44.750989+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:56.003105+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:22:56.003105+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:03.248568+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"94'139","reported_seq":1608,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-05-15T13:23:44.755844+0000","last_change":"2024-05-15T13:23:43.413436+0000","last_active":"2024-05-15T13:23:44.755844+0000","last_peered":"2024-05-15T13:23:44.755844+0000","last_clean":"2024-05-15T13:23:44.755844+0000","last_became_active":"2024-05-15T13:19:41.190923+0000","last_became_peered":"2024-05-15T13:19:41.190923+0000","last_unstale":"2024-05-15T13:23:44.755844+0000","last_undegraded":"2024-05-15T13:23:44.755844+0000","last_fullsized":"2024-05-15T13:23:44.755844+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:43.413325+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:23:43.413325+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:25:06.603502+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"94'139","reported_seq":1376,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-05-15T13:23:49.264512+0000","last_change":"2024-05-15T13:23:36.340763+0000","last_active":"2024-05-15T13:23:49.264512+0000","last_peered":"2024-05-15T13:23:49.264512+0000","last_clean":"2024-05-15T13:23:49.264512+0000","last_became_active":"2024-05-15T13:22:48.739476+0000","last_became_peered":"2024-05-15T13:22:48.739476+0000","last_unstale":"2024-05-15T13:23:49.264512+0000","last_undegraded":"2024-05-15T13:23:49.264512+0000","last_fullsized":"2024-05-15T13:23:49.264512+0000","mapping_epoch":805,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":806,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:36.340646+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:23:36.340646+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:39.354164+0000","scrub_duration":267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,3],"acting":[0,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"94'139","reported_seq":1615,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-05-15T13:23:44.756086+0000","last_change":"2024-05-15T13:23:08.852185+0000","last_active":"2024-05-15T13:23:44.756086+0000","last_peered":"2024-05-15T13:23:44.756086+0000","last_clean":"2024-05-15T13:23:44.756086+0000","last_became_active":"2024-05-15T13:13:49.926687+0000","last_became_peered":"2024-05-15T13:13:49.926687+0000","last_unstale":"2024-05-15T13:23:44.756086+0000","last_undegraded":"2024-05-15T13:23:44.756086+0000","last_fullsized":"2024-05-15T13:23:44.756086+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:08.852053+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:18:06.340555+0000","last_clean_scrub_stamp":"2024-05-15T13:23:08.852053+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:33.753420+0000","scrub_duration":121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"94'139","reported_seq":1616,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-05-15T13:23:44.752218+0000","last_change":"2024-05-15T13:23:38.179779+0000","last_active":"2024-05-15T13:23:44.752218+0000","last_peered":"2024-05-15T13:23:44.752218+0000","last_clean":"2024-05-15T13:23:44.752218+0000","last_became_active":"2024-05-15T13:13:16.478488+0000","last_became_peered":"2024-05-15T13:13:16.478488+0000","last_unstale":"2024-05-15T13:23:44.752218+0000","last_undegraded":"2024-05-15T13:23:44.752218+0000","last_fullsized":"2024-05-15T13:23:44.752218+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:38.179664+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:23:38.179664+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:45.486955+0000","scrub_duration":286,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":3764125696,"num_objects":57436,"num_object_clones":0,"num_object_copies":172308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57436,"num_write_kb":3675904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11271864320,"data_stored":11271864320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57436,"ondisk_log_size":57436,"up":48,"acting":48,"num_store_stats":9},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057101,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1698208,"kb_used_data":904856,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":92046944,"statfs":{"total":95995035648,"available":94256070656,"internally_reserved":0,"allocated":926572544,"data_stored":926189762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":812382662},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351442,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2082412,"kb_used_data":1230756,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":91662740,"statfs":{"total":95995035648,"available":93862645760,"internally_reserved":0,"allocated":1260294144,"data_stored":1259906876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":872085952},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514967,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2376364,"kb_used_data":1420580,"kb_used_omap":1,"kb_used_meta":955774,"kb_avail":91368788,"statfs":{"total":95995035648,"available":93561638912,"internally_reserved":0,"allocated":1454673920,"data_stored":1454286652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":978713024},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":0,"seq":279172874716,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169057,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1479856,"kb_used_data":1165476,"kb_used_omap":1,"kb_used_meta":314366,"kb_avail":92265296,"statfs":{"total":95995035648,"available":94479663104,"internally_reserved":0,"allocated":1193447424,"data_stored":1193060156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":321911242},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430695,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1215584,"kb_used_data":448600,"kb_used_omap":1,"kb_used_meta":766974,"kb_avail":92529568,"statfs":{"total":95995035648,"available":94750277632,"internally_reserved":0,"allocated":459366400,"data_stored":458983618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":785381830},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575359,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2947424,"kb_used_data":1969432,"kb_used_omap":1,"kb_used_meta":977982,"kb_avail":90797728,"statfs":{"total":95995035648,"available":92976873472,"internally_reserved":0,"allocated":2016698368,"data_stored":2016315586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1001454021},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738890,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1305248,"kb_used_data":457752,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":92439904,"statfs":{"total":95995035648,"available":94658461696,"internally_reserved":0,"allocated":468738048,"data_stored":468355266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":867826117},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411836,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":623788,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93121364,"statfs":{"total":95995035648,"available":95356276736,"internally_reserved":0,"allocated":1212416,"data_stored":826082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150198,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2297192,"kb_used_data":1309472,"kb_used_omap":1,"kb_used_meta":987710,"kb_avail":91447960,"statfs":{"total":95995035648,"available":93642711040,"internally_reserved":0,"allocated":1340899328,"data_stored":1340512994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1011415495},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986674,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692332,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2831592,"kb_used_data":2010464,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":90913560,"statfs":{"total":95995035648,"available":93095485440,"internally_reserved":0,"allocated":2058715136,"data_stored":2058328802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":840825285},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471793664,"data_stored":471793664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2036334592,"data_stored":2036334592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1362886656,"data_stored":1362886656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2090401792,"data_stored":2090401792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":459079680,"data_stored":459079680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1220083712,"data_stored":1220083712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1468727296,"data_stored":1468727296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1262223360,"data_stored":1262223360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":900333568,"data_stored":900333568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:23:55.715 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-05-15T13:23:55.715 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool set .mgr pg_num 31 2024-05-15T13:23:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:23:55.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:55.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:23:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:23:55.815 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:23:55.843 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:23:55.904 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:56.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:55 smithi027 ceph-mon[39795]: 1.f scrub starts 2024-05-15T13:23:56.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:55 smithi027 ceph-mon[39795]: 1.f scrub ok 2024-05-15T13:23:56.034 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:55 smithi027 ceph-mon[39795]: pgmap v2156: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-05-15T13:23:56.034 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:55 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:23:56.034 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:55 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:23:56.034 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:55 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1379704113' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:23:56.185 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:55 smithi120 ceph-mon[43961]: 1.f scrub starts 2024-05-15T13:23:56.185 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:55 smithi120 ceph-mon[43961]: 1.f scrub ok 2024-05-15T13:23:56.186 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:55 smithi120 ceph-mon[43961]: pgmap v2156: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-05-15T13:23:56.186 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:55 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:23:56.186 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:55 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:23:56.186 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:55 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1379704113' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:23:56.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:55 smithi177 ceph-mon[44303]: 1.f scrub starts 2024-05-15T13:23:56.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:55 smithi177 ceph-mon[44303]: 1.f scrub ok 2024-05-15T13:23:56.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:55 smithi177 ceph-mon[44303]: pgmap v2156: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-05-15T13:23:56.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:55 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:23:56.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:55 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:23:56.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:55 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1379704113' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:23:56.228 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-05-15T13:23:56.300 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 44 16 68729 68713 97.5898 98.5625 0.00618641 0.0102431 2024-05-15T13:23:56.362 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:23:56.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:56.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:23:56.471 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:23:56.546 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:56.628 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:23:56.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:56.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:23:56.637 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:23:56.733 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:23:56.760 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:56.961 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:23:56.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:56.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:23:57.010 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:56 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:23:57.011 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:56 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:23:57.011 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:56 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:23:57.011 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:56 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1379704113' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:23:57.011 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:56 smithi120 ceph-mon[43961]: osdmap e830: 12 total, 12 up, 10 in 2024-05-15T13:23:57.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:56 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:23:57.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:56 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:23:57.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:56 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:23:57.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:56 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1379704113' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:23:57.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:56 smithi027 ceph-mon[39795]: osdmap e830: 12 total, 12 up, 10 in 2024-05-15T13:23:57.067 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:23:57.118 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:57.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:56 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:23:57.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:56 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:23:57.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:56 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:23:57.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:56 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1379704113' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:23:57.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:56 smithi177 ceph-mon[44303]: osdmap e830: 12 total, 12 up, 10 in 2024-05-15T13:23:57.300 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 45 16 70086 70070 97.3057 84.8125 0.00648989 0.010273 2024-05-15T13:23:57.373 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:23:57.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:57.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:23:57.478 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:23:57.527 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:57.566 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:23:57.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:57.572 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:23:57.673 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:23:57.726 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:57.757 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:23:57.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:23:57.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:23:57.804 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:57 smithi177 ceph-mon[44303]: pgmap v2158: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-05-15T13:23:57.804 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:57 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:23:57.863 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:23:57.910 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:23:57.962 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:23:58.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:57 smithi120 ceph-mon[43961]: pgmap v2158: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-05-15T13:23:58.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:57 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:23:58.226 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:57 smithi027 ceph-mon[39795]: pgmap v2158: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-05-15T13:23:58.226 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:57 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:23:58.300 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 46 16 71357 71341 96.9171 79.4375 0.00606918 0.0103099 2024-05-15T13:23:59.070 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:23:59.304 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 47 16 72932 72916 96.9429 98.4375 0.00594912 0.0103116 2024-05-15T13:23:59.795 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:23:59.839 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:23:59.860 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:23:59.886 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:23:59.901 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:23:59.964 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:23:59.995 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:24:00.021 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:24:00.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:00.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:24:00.034 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:24:00.065 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:24:00.093 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:24:00.127 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:24:00.133 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:24:00.174 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:59 smithi027 ceph-mon[39795]: pgmap v2159: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-05-15T13:24:00.174 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:59 smithi027 ceph-mon[39795]: osdmap e831: 12 total, 12 up, 10 in 2024-05-15T13:24:00.174 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:23:59 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1883058086' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:24:00.177 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:00.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:59 smithi177 ceph-mon[44303]: pgmap v2159: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-05-15T13:24:00.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:59 smithi177 ceph-mon[44303]: osdmap e831: 12 total, 12 up, 10 in 2024-05-15T13:24:00.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:23:59 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1883058086' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:24:00.201 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:24:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:24:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:24:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:24:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:24:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:24:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:24:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:24:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:24:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:24:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:24:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:24:00.203 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:24:00.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:59 smithi120 ceph-mon[43961]: pgmap v2159: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-05-15T13:24:00.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:59 smithi120 ceph-mon[43961]: osdmap e831: 12 total, 12 up, 10 in 2024-05-15T13:24:00.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:23:59 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1883058086' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:24:00.303 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 48 16 74236 74220 96.6211 81.5 0.00653658 0.0103453 2024-05-15T13:24:00.839 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:24:00.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:00.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:24:00.950 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:24:01.006 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:01.030 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:24:01.030 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:24:01.304 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 49 16 75555 75539 96.3315 82.4375 0.00622648 0.0103768 2024-05-15T13:24:01.364 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:24:01.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:01.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:24:01.477 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:24:01.530 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:02.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:01 smithi027 ceph-mon[39795]: pgmap v2161: 37 pgs: 37 active+clean; 4.3 GiB data, 21 GiB used, 874 GiB / 894 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-05-15T13:24:02.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:01 smithi177 ceph-mon[44303]: pgmap v2161: 37 pgs: 37 active+clean; 4.3 GiB data, 21 GiB used, 874 GiB / 894 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-05-15T13:24:02.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:01 smithi120 ceph-mon[43961]: pgmap v2161: 37 pgs: 37 active+clean; 4.3 GiB data, 21 GiB used, 874 GiB / 894 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-05-15T13:24:02.304 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 50 16 77038 77022 96.2585 92.6875 0.0165133 0.0103851 2024-05-15T13:24:02.602 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:24:02.706 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:24:02.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:02.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:24:02.735 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-05-15T13:24:02.812 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:24:02.865 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:02.907 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:24:02.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:02.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:24:03.012 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:24:03.045 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:03.073 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:24:03.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:03.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:24:03.184 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:24:03.189 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:24:03.237 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:03.304 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 51 16 78463 78447 96.1173 89.0625 0.00815686 0.0103999 2024-05-15T13:24:03.602 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:24:03.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:03.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:24:03.707 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:24:03.767 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:set pool 1 pg_num to 31 2024-05-15T13:24:04.087 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:24:04.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:03 smithi027 ceph-mon[39795]: pgmap v2162: 37 pgs: 37 active+clean; 4.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-05-15T13:24:04.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:03 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2817719773' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-05-15T13:24:04.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:03 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-05-15T13:24:04.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:03 smithi177 ceph-mon[44303]: pgmap v2162: 37 pgs: 37 active+clean; 4.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-05-15T13:24:04.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:03 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2817719773' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-05-15T13:24:04.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:03 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-05-15T13:24:04.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:03 smithi120 ceph-mon[43961]: pgmap v2162: 37 pgs: 37 active+clean; 4.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-05-15T13:24:04.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:03 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2817719773' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-05-15T13:24:04.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:03 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-05-15T13:24:04.304 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 52 16 79955 79939 96.062 93.25 0.0136052 0.0104053 2024-05-15T13:24:05.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:04 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-05-15T13:24:05.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:04 smithi120 ceph-mon[43961]: osdmap e832: 12 total, 12 up, 10 in 2024-05-15T13:24:05.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:04 smithi120 ceph-mon[43961]: osdmap e833: 12 total, 12 up, 10 in 2024-05-15T13:24:05.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:04 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-05-15T13:24:05.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:04 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-05-15T13:24:05.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:04 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-05-15T13:24:05.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:04 smithi177 ceph-mon[44303]: osdmap e832: 12 total, 12 up, 10 in 2024-05-15T13:24:05.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:04 smithi177 ceph-mon[44303]: osdmap e833: 12 total, 12 up, 10 in 2024-05-15T13:24:05.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:04 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-05-15T13:24:05.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:04 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-05-15T13:24:05.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:04 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-05-15T13:24:05.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:04 smithi027 ceph-mon[39795]: osdmap e832: 12 total, 12 up, 10 in 2024-05-15T13:24:05.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:04 smithi027 ceph-mon[39795]: osdmap e833: 12 total, 12 up, 10 in 2024-05-15T13:24:05.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:04 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-05-15T13:24:05.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:04 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-05-15T13:24:05.304 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 53 16 81094 81078 95.5926 71.1875 0.0127077 0.0104575 2024-05-15T13:24:05.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:05 smithi027 ceph-mon[39795]: pgmap v2165: 37 pgs: 37 active+clean; 4.7 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-05-15T13:24:05.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:05 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-05-15T13:24:05.869 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:05 smithi027 ceph-mon[39795]: osdmap e834: 12 total, 12 up, 10 in 2024-05-15T13:24:06.011 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:24:06.012 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:24:06.041 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:24:06.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:06.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:24:06.148 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:24:06.187 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:05 smithi120 ceph-mon[43961]: pgmap v2165: 37 pgs: 37 active+clean; 4.7 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-05-15T13:24:06.187 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:05 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-05-15T13:24:06.187 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:05 smithi120 ceph-mon[43961]: osdmap e834: 12 total, 12 up, 10 in 2024-05-15T13:24:06.190 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:06.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:05 smithi177 ceph-mon[44303]: pgmap v2165: 37 pgs: 37 active+clean; 4.7 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-05-15T13:24:06.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:05 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-05-15T13:24:06.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:05 smithi177 ceph-mon[44303]: osdmap e834: 12 total, 12 up, 10 in 2024-05-15T13:24:06.220 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:24:06.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:06.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:24:06.307 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 54 16 82567 82551 95.5226 92.0625 0.00899133 0.0104634 2024-05-15T13:24:06.326 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:24:06.371 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:06.400 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:24:06.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:06.405 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:24:06.506 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:24:06.546 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:06.658 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:24:06.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:06.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:24:06.766 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:24:06.805 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:06.835 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:24:06.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:06.839 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:24:06.940 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:24:06.999 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:07.129 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:24:07.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:07.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:24:07.234 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:24:07.278 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:07.307 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 55 16 83993 83977 95.4061 89.125 0.0107059 0.0104776 2024-05-15T13:24:07.339 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:24:07.437 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:24:07.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:07.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:24:07.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:07 smithi120 ceph-mon[43961]: osdmap e835: 12 total, 12 up, 10 in 2024-05-15T13:24:07.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:07 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:24:07.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:07 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:24:07.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:07 smithi120 ceph-mon[43961]: pgmap v2168: 47 pgs: 4 peering, 10 unknown, 33 active+clean; 4.8 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-05-15T13:24:07.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:07 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-05-15T13:24:07.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:07 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-05-15T13:24:07.543 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:24:07.589 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:07 smithi027 ceph-mon[39795]: osdmap e835: 12 total, 12 up, 10 in 2024-05-15T13:24:07.590 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:07 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:24:07.590 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:07 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:24:07.590 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:07 smithi027 ceph-mon[39795]: pgmap v2168: 47 pgs: 4 peering, 10 unknown, 33 active+clean; 4.8 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-05-15T13:24:07.590 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:07 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-05-15T13:24:07.590 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:07 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-05-15T13:24:07.594 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:07.686 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:24:07.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:07 smithi177 ceph-mon[44303]: osdmap e835: 12 total, 12 up, 10 in 2024-05-15T13:24:07.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:07 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:24:07.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:07 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:24:07.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:07 smithi177 ceph-mon[44303]: pgmap v2168: 47 pgs: 4 peering, 10 unknown, 33 active+clean; 4.8 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-05-15T13:24:07.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:07 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-05-15T13:24:07.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:07 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-05-15T13:24:07.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:07.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:24:07.798 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:24:07.838 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:07.867 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:24:07.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:07.873 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:24:07.974 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:24:08.011 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:08.097 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:24:08.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:08.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:24:08.134 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-05-15T13:24:08.206 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:24:08.253 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:08.307 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 56 16 85262 85246 95.1187 79.3125 0.0111202 0.0105088 2024-05-15T13:24:08.315 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:08 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:24:08.315 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:08 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:24:08.315 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:08 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-05-15T13:24:08.315 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:08 smithi027 ceph-mon[39795]: osdmap e836: 12 total, 12 up, 10 in 2024-05-15T13:24:08.315 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:08 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:24:08.315 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:08 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:24:08.315 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:08 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1592107353' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:24:08.316 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:24:08.577 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:24:08.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:08 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:24:08.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:08 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:24:08.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:08 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-05-15T13:24:08.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:08 smithi177 ceph-mon[44303]: osdmap e836: 12 total, 12 up, 10 in 2024-05-15T13:24:08.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:08 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:24:08.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:08 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:24:08.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:08 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1592107353' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:24:08.719 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:24:08.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:08.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:24:08.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:08 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:24:08.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:08 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:24:08.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:08 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-05-15T13:24:08.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:08 smithi120 ceph-mon[43961]: osdmap e836: 12 total, 12 up, 10 in 2024-05-15T13:24:08.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:08 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:24:08.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:08 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:24:08.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:08 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1592107353' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:24:08.825 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:24:08.866 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:09.307 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 57 16 86486 86470 94.792 76.5 0.00801227 0.0105455 2024-05-15T13:24:09.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:09 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:24:09.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:09 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1592107353' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:24:09.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:09 smithi120 ceph-mon[43961]: osdmap e837: 12 total, 12 up, 10 in 2024-05-15T13:24:09.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:09 smithi120 ceph-mon[43961]: pgmap v2171: 47 pgs: 1 remapped, 4 peering, 8 unknown, 34 active+clean; 5.0 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-05-15T13:24:09.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:09 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:24:09.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:09 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:24:09.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:09 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:24:09.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:09 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:24:09.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:09 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1592107353' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:24:09.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:09 smithi177 ceph-mon[44303]: osdmap e837: 12 total, 12 up, 10 in 2024-05-15T13:24:09.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:09 smithi177 ceph-mon[44303]: pgmap v2171: 47 pgs: 1 remapped, 4 peering, 8 unknown, 34 active+clean; 5.0 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-05-15T13:24:09.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:09 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:24:09.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:09 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:24:09.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:09 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:24:09.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:09 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:24:09.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:09 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1592107353' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:24:09.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:09 smithi027 ceph-mon[39795]: osdmap e837: 12 total, 12 up, 10 in 2024-05-15T13:24:09.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:09 smithi027 ceph-mon[39795]: pgmap v2171: 47 pgs: 1 remapped, 4 peering, 8 unknown, 34 active+clean; 5.0 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-05-15T13:24:09.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:09 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:24:09.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:09 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:24:09.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:09 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:24:09.832 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:24:09.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:09.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:24:09.938 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:24:09.983 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:10.013 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:24:10.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:10.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:24:10.120 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:24:10.160 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:10.307 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 58 16 87918 87902 94.7006 89.5 0.00981886 0.0105555 2024-05-15T13:24:10.373 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:24:10.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:10.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:24:10.478 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:24:10.529 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:10.530 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:10 smithi027 ceph-mon[39795]: osdmap e838: 12 total, 12 up, 10 in 2024-05-15T13:24:10.565 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:24:10.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:10.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:24:10.670 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:24:10.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:10 smithi177 ceph-mon[44303]: osdmap e838: 12 total, 12 up, 10 in 2024-05-15T13:24:10.711 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:10 smithi120 ceph-mon[43961]: osdmap e838: 12 total, 12 up, 10 in 2024-05-15T13:24:10.716 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:10.795 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:24:10.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:10.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:24:10.902 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:24:10.964 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:11.001 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:24:11.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:11.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:24:11.108 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:24:11.112 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:24:11.112 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:24:11.114 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:24:11.146 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:24:11.149 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:11.173 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:24:11.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:11.178 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:24:11.179 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:24:11.212 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:24:11.251 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:24:11.278 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:24:11.285 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:24:11.307 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 59 16 89299 89283 94.5583 86.3125 0.0135523 0.0105715 2024-05-15T13:24:11.321 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:24:11.327 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:11.367 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:24:11.400 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:24:11.421 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:24:11.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:11 smithi177 ceph-mon[44303]: osdmap e839: 12 total, 12 up, 10 in 2024-05-15T13:24:11.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:11 smithi177 ceph-mon[44303]: pgmap v2174: 47 pgs: 3 active+undersized+remapped, 1 remapped+peering, 1 remapped, 4 peering, 4 unknown, 34 active+clean; 5.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 511 B/s rd, 78 MiB/s wr, 1.25k op/s; 56 KiB/s, 0 objects/s recovering 2024-05-15T13:24:11.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:11 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3946970073' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:24:11.441 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:24:11.469 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:24:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:24:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:24:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:24:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:24:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:24:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:24:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:24:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:24:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:24:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:24:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:24:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:24:11.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:11 smithi120 ceph-mon[43961]: osdmap e839: 12 total, 12 up, 10 in 2024-05-15T13:24:11.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:11 smithi120 ceph-mon[43961]: pgmap v2174: 47 pgs: 3 active+undersized+remapped, 1 remapped+peering, 1 remapped, 4 peering, 4 unknown, 34 active+clean; 5.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 511 B/s rd, 78 MiB/s wr, 1.25k op/s; 56 KiB/s, 0 objects/s recovering 2024-05-15T13:24:11.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:11 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3946970073' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:24:11.718 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 9, 10, 11, 7] out_osds: [4, 8] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:24:11.718 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:24:11.718 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_7 2024-05-15T13:24:11.719 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:24:11.746 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:11 smithi027 ceph-mon[39795]: osdmap e839: 12 total, 12 up, 10 in 2024-05-15T13:24:11.746 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:11 smithi027 ceph-mon[39795]: pgmap v2174: 47 pgs: 3 active+undersized+remapped, 1 remapped+peering, 1 remapped, 4 peering, 4 unknown, 34 active+clean; 5.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 511 B/s rd, 78 MiB/s wr, 1.25k op/s; 56 KiB/s, 0 objects/s recovering 2024-05-15T13:24:11.746 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:11 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3946970073' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:24:12.086 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:24:12.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:12.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:24:12.192 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:24:12.237 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:12.298 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:24:12.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:12.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:24:12.307 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T13:24:12.308717+0000 min lat: 0.00110277 max lat: 13.6712 avg lat: 0.0106436 2024-05-15T13:24:12.307 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:24:12.307 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 60 16 90192 90176 93.9126 55.8125 0.0068632 0.0106436 2024-05-15T13:24:12.401 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:24:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:24:12.448 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:12.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:12 smithi027 ceph-mon[39795]: osdmap e840: 12 total, 12 up, 10 in 2024-05-15T13:24:12.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:12 smithi177 ceph-mon[44303]: osdmap e840: 12 total, 12 up, 10 in 2024-05-15T13:24:12.786 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:24:12.790 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:12 smithi120 ceph-mon[43961]: osdmap e840: 12 total, 12 up, 10 in 2024-05-15T13:24:12.907 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:24:12.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:12.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:24:13.025 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:24:13.074 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:13.262 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:24:13.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:13.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:24:13.299 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-05-15T13:24:13.307 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 61 16 91467 91451 93.6793 79.6875 0.00718765 0.0106704 2024-05-15T13:24:13.370 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:24:13.376 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:13 smithi027 ceph-mon[39795]: pgmap v2176: 47 pgs: 3 peering, 1 active+undersized+remapped, 1 remapped+peering, 1 remapped, 2 unknown, 39 active+clean; 5.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 3.0 KiB/s rd, 59 MiB/s wr, 953 op/s; 126 KiB/s, 0 objects/s recovering 2024-05-15T13:24:13.413 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:13.444 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:24:13.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:13.452 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:24:13.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:13 smithi120 ceph-mon[43961]: pgmap v2176: 47 pgs: 3 peering, 1 active+undersized+remapped, 1 remapped+peering, 1 remapped, 2 unknown, 39 active+clean; 5.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 3.0 KiB/s rd, 59 MiB/s wr, 953 op/s; 126 KiB/s, 0 objects/s recovering 2024-05-15T13:24:13.552 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:24:13.598 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:13.628 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:24:13.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:13.636 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:24:13.636 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:13 smithi177 ceph-mon[44303]: pgmap v2176: 47 pgs: 3 peering, 1 active+undersized+remapped, 1 remapped+peering, 1 remapped, 2 unknown, 39 active+clean; 5.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 3.0 KiB/s rd, 59 MiB/s wr, 953 op/s; 126 KiB/s, 0 objects/s recovering 2024-05-15T13:24:13.737 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:24:13.746 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:24:13.777 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:13.823 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:24:13.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:13.832 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:24:13.912 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:24:13.932 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:24:13.964 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:14.002 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:24:14.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:14.011 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:24:14.112 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:24:14.151 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:14.188 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:24:14.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:14.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:24:14.292 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:24:14.310 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 62 16 92771 92755 93.4784 81.5 0.011666 0.010694 2024-05-15T13:24:14.347 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:24:14.350 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:14.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:14 smithi027 ceph-mon[39795]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-05-15T13:24:14.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:14 smithi027 ceph-mon[39795]: osdmap e841: 12 total, 12 up, 10 in 2024-05-15T13:24:14.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:14 smithi177 ceph-mon[44303]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-05-15T13:24:14.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:14 smithi177 ceph-mon[44303]: osdmap e841: 12 total, 12 up, 10 in 2024-05-15T13:24:14.707 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:24:14.708 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:14 smithi120 ceph-mon[43961]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-05-15T13:24:14.708 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:14 smithi120 ceph-mon[43961]: osdmap e841: 12 total, 12 up, 10 in 2024-05-15T13:24:14.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:14.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:24:14.811 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:24:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:15.313 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 63 16 94109 94093 93.3181 83.625 0.0166315 0.0107122 2024-05-15T13:24:15.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:15 smithi120 ceph-mon[43961]: pgmap v2178: 47 pgs: 3 peering, 44 active+clean; 5.5 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 2.4 KiB/s rd, 75 MiB/s wr, 1.21k op/s; 140 KiB/s, 0 objects/s recovering 2024-05-15T13:24:15.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:15 smithi177 ceph-mon[44303]: pgmap v2178: 47 pgs: 3 peering, 44 active+clean; 5.5 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 2.4 KiB/s rd, 75 MiB/s wr, 1.21k op/s; 140 KiB/s, 0 objects/s recovering 2024-05-15T13:24:15.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:15 smithi027 ceph-mon[39795]: pgmap v2178: 47 pgs: 3 peering, 44 active+clean; 5.5 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 2.4 KiB/s rd, 75 MiB/s wr, 1.21k op/s; 140 KiB/s, 0 objects/s recovering 2024-05-15T13:24:16.313 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 64 16 95457 95441 93.1763 84.25 0.014358 0.0107286 2024-05-15T13:24:16.333 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:24:16.333 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T13:24:16.333 INFO:teuthology.orchestra.run.smithi120.stdout: "complaint_time": 30, 2024-05-15T13:24:16.333 INFO:teuthology.orchestra.run.smithi120.stdout: "num_blocked_ops": 0 2024-05-15T13:24:16.333 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:24:16.664 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:24:16.665 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:24:16.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:16 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:24:16.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:16 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:24:16.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:16 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:24:16.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:16 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:24:16.790 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:16 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:24:16.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:16 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:24:17.149 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:24:17.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:17.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:24:17.262 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:24:17.308 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:17.315 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 65 16 96790 96774 93.0217 83.3125 0.00865687 0.010746 2024-05-15T13:24:17.349 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:24:17.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:17.353 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:24:17.454 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:24:17.506 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:17.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:17 smithi177 ceph-mon[44303]: osdmap e842: 12 total, 12 up, 10 in 2024-05-15T13:24:17.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:17 smithi177 ceph-mon[44303]: from='client.16695 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:24:17.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:17 smithi177 ceph-mon[44303]: pgmap v2180: 47 pgs: 47 active+clean; 5.6 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 1.8 KiB/s rd, 83 MiB/s wr, 1.33k op/s; 88 KiB/s, 0 objects/s recovering 2024-05-15T13:24:17.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:17 smithi027 ceph-mon[39795]: osdmap e842: 12 total, 12 up, 10 in 2024-05-15T13:24:17.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:17 smithi027 ceph-mon[39795]: from='client.16695 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:24:17.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:17 smithi027 ceph-mon[39795]: pgmap v2180: 47 pgs: 47 active+clean; 5.6 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 1.8 KiB/s rd, 83 MiB/s wr, 1.33k op/s; 88 KiB/s, 0 objects/s recovering 2024-05-15T13:24:17.749 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2179,"stamp":"2024-05-15T13:24:16.404461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6037534392,"num_objects":92119,"num_object_clones":0,"num_object_copies":276357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92119,"num_whiteouts":0,"num_read":14,"num_read_kb":13,"num_write":92137,"num_write_kb":5895865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":765792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96427,"ondisk_log_size":96427,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":25683520,"kb_used_data":17616588,"kb_used_omap":15,"kb_used_meta":8066736,"kb_avail":911768000,"statfs":{"total":959950356480,"available":933650432000,"internally_reserved":0,"allocated":18039386112,"data_stored":18035023378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15951,"internal_metadata":8260338097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,7,8,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":520606119,"num_objects":7943,"num_object_clones":0,"num_object_copies":23829,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7943,"num_whiteouts":0,"num_read":12,"num_read_kb":11,"num_write":7959,"num_write_kb":508620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":537839,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.064254"},"pg_stats":[{"pgid":"1.1e","version":"94'139","reported_seq":916,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:11.515341+0000","last_change":"2024-05-15T13:24:11.515341+0000","last_active":"2024-05-15T13:24:11.515341+0000","last_peered":"2024-05-15T13:24:11.515341+0000","last_clean":"2024-05-15T13:24:11.515341+0000","last_became_active":"2024-05-15T13:24:11.445146+0000","last_became_peered":"2024-05-15T13:24:11.445146+0000","last_unstale":"2024-05-15T13:24:11.515341+0000","last_undegraded":"2024-05-15T13:24:11.515341+0000","last_fullsized":"2024-05-15T13:24:11.515341+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:06.819156+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:23:06.819156+0000","last_clean_scrub_stamp":"2024-05-15T13:23:06.819156+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,10],"acting":[11,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"94'139","reported_seq":864,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:11.375115+0000","last_change":"2024-05-15T13:24:11.375115+0000","last_active":"2024-05-15T13:24:11.375115+0000","last_peered":"2024-05-15T13:24:11.375115+0000","last_clean":"2024-05-15T13:24:11.375115+0000","last_became_active":"2024-05-15T13:24:11.374741+0000","last_became_peered":"2024-05-15T13:24:11.374741+0000","last_unstale":"2024-05-15T13:24:11.375115+0000","last_undegraded":"2024-05-15T13:24:11.375115+0000","last_fullsized":"2024-05-15T13:24:11.375115+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:03.856199+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:23:03.856199+0000","last_clean_scrub_stamp":"2024-05-15T13:23:03.856199+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"94'139","reported_seq":576,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-05-15T13:24:14.284002+0000","last_change":"2024-05-15T13:24:11.372542+0000","last_active":"2024-05-15T13:24:14.284002+0000","last_peered":"2024-05-15T13:24:14.284002+0000","last_clean":"2024-05-15T13:24:14.284002+0000","last_became_active":"2024-05-15T13:24:11.372136+0000","last_became_peered":"2024-05-15T13:24:11.372136+0000","last_unstale":"2024-05-15T13:24:14.284002+0000","last_undegraded":"2024-05-15T13:24:14.284002+0000","last_fullsized":"2024-05-15T13:24:14.284002+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:56.794428+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:22:56.794428+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:17.268838+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8945,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":227953,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,11],"acting":[6,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1b","version":"94'139","reported_seq":1153,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-05-15T13:24:14.288378+0000","last_change":"2024-05-15T13:24:11.374088+0000","last_active":"2024-05-15T13:24:14.288378+0000","last_peered":"2024-05-15T13:24:14.288378+0000","last_clean":"2024-05-15T13:24:14.288378+0000","last_became_active":"2024-05-15T13:24:11.373634+0000","last_became_peered":"2024-05-15T13:24:11.373634+0000","last_unstale":"2024-05-15T13:24:14.288378+0000","last_undegraded":"2024-05-15T13:24:14.288378+0000","last_fullsized":"2024-05-15T13:24:14.288378+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:56.003105+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:22:56.003105+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:16.398801+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.1a","version":"94'139","reported_seq":1032,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-05-15T13:24:14.283202+0000","last_change":"2024-05-15T13:24:11.446766+0000","last_active":"2024-05-15T13:24:14.283202+0000","last_peered":"2024-05-15T13:24:14.283202+0000","last_clean":"2024-05-15T13:24:14.283202+0000","last_became_active":"2024-05-15T13:24:11.444818+0000","last_became_peered":"2024-05-15T13:24:11.444818+0000","last_unstale":"2024-05-15T13:24:14.283202+0000","last_undegraded":"2024-05-15T13:24:14.283202+0000","last_fullsized":"2024-05-15T13:24:14.283202+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:39.504559+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:22:03.418345+0000","last_clean_scrub_stamp":"2024-05-15T13:23:39.504559+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:17.343960+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"94'139","reported_seq":810,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:11.375581+0000","last_change":"2024-05-15T13:24:11.375145+0000","last_active":"2024-05-15T13:24:11.375581+0000","last_peered":"2024-05-15T13:24:11.375581+0000","last_clean":"2024-05-15T13:24:11.375581+0000","last_became_active":"2024-05-15T13:24:11.374739+0000","last_became_peered":"2024-05-15T13:24:11.374739+0000","last_unstale":"2024-05-15T13:24:11.375581+0000","last_undegraded":"2024-05-15T13:24:11.375581+0000","last_fullsized":"2024-05-15T13:24:11.375581+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:12.031939+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:23:12.031939+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:16.884412+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":8,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":168295,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"94'139","reported_seq":1630,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:11.368655+0000","last_change":"2024-05-15T13:24:11.368655+0000","last_active":"2024-05-15T13:24:11.368655+0000","last_peered":"2024-05-15T13:24:11.368655+0000","last_clean":"2024-05-15T13:24:11.368655+0000","last_became_active":"2024-05-15T13:24:11.368347+0000","last_became_peered":"2024-05-15T13:24:11.368347+0000","last_unstale":"2024-05-15T13:24:11.368655+0000","last_undegraded":"2024-05-15T13:24:11.368655+0000","last_fullsized":"2024-05-15T13:24:11.368655+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:47.391631+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:23:47.391631+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131344,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"94'139","reported_seq":546,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:11.385586+0000","last_change":"2024-05-15T13:24:11.379074+0000","last_active":"2024-05-15T13:24:11.385586+0000","last_peered":"2024-05-15T13:24:11.385586+0000","last_clean":"2024-05-15T13:24:11.385586+0000","last_became_active":"2024-05-15T13:24:11.378731+0000","last_became_peered":"2024-05-15T13:24:11.378731+0000","last_unstale":"2024-05-15T13:24:11.385586+0000","last_undegraded":"2024-05-15T13:24:11.385586+0000","last_fullsized":"2024-05-15T13:24:11.385586+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:49.429485+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:23:49.429485+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:16.884433+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":84148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.16","version":"94'139","reported_seq":1591,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:11.383713+0000","last_change":"2024-05-15T13:24:11.380802+0000","last_active":"2024-05-15T13:24:11.383713+0000","last_peered":"2024-05-15T13:24:11.383713+0000","last_clean":"2024-05-15T13:24:11.383713+0000","last_became_active":"2024-05-15T13:24:11.380273+0000","last_became_peered":"2024-05-15T13:24:11.380273+0000","last_unstale":"2024-05-15T13:24:11.383713+0000","last_undegraded":"2024-05-15T13:24:11.383713+0000","last_fullsized":"2024-05-15T13:24:11.383713+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:37.717703+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:23:37.717703+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:17.164220+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"94'139","reported_seq":545,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-05-15T13:24:14.283053+0000","last_change":"2024-05-15T13:24:11.446049+0000","last_active":"2024-05-15T13:24:14.283053+0000","last_peered":"2024-05-15T13:24:14.283053+0000","last_clean":"2024-05-15T13:24:14.283053+0000","last_became_active":"2024-05-15T13:24:11.444781+0000","last_became_peered":"2024-05-15T13:24:11.444781+0000","last_unstale":"2024-05-15T13:24:14.283053+0000","last_undegraded":"2024-05-15T13:24:14.283053+0000","last_fullsized":"2024-05-15T13:24:14.283053+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:51.887016+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:18:24.970285+0000","last_clean_scrub_stamp":"2024-05-15T13:22:51.887016+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:17.343929+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":84148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"94'139","reported_seq":1051,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:11.364447+0000","last_change":"2024-05-15T13:23:11.573603+0000","last_active":"2024-05-15T13:24:11.364447+0000","last_peered":"2024-05-15T13:24:11.364447+0000","last_clean":"2024-05-15T13:24:11.364447+0000","last_became_active":"2024-05-15T13:19:41.190796+0000","last_became_peered":"2024-05-15T13:19:41.190796+0000","last_unstale":"2024-05-15T13:24:11.364447+0000","last_undegraded":"2024-05-15T13:24:11.364447+0000","last_fullsized":"2024-05-15T13:24:11.364447+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:11.573492+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:23:11.573492+0000","last_clean_scrub_stamp":"2024-05-15T13:23:11.573492+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:25.130907+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"841'5852","reported_seq":5926,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-05-15T13:24:15.672080+0000","last_change":"2024-05-15T13:22:58.857344+0000","last_active":"2024-05-15T13:24:15.672080+0000","last_peered":"2024-05-15T13:24:15.672080+0000","last_clean":"2024-05-15T13:24:15.672080+0000","last_became_active":"2024-05-15T13:22:58.857024+0000","last_became_peered":"2024-05-15T13:22:58.857024+0000","last_unstale":"2024-05-15T13:24:15.672080+0000","last_undegraded":"2024-05-15T13:24:15.672080+0000","last_fullsized":"2024-05-15T13:24:15.672080+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":5852,"log_dups_size":0,"ondisk_log_size":5852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:16.398839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383516672,"num_objects":5852,"num_object_clones":0,"num_object_copies":17556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5852,"num_write_kb":374528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":1617,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:11.367289+0000","last_change":"2024-05-15T13:23:38.906554+0000","last_active":"2024-05-15T13:24:11.367289+0000","last_peered":"2024-05-15T13:24:11.367289+0000","last_clean":"2024-05-15T13:24:11.367289+0000","last_became_active":"2024-05-15T13:13:11.361366+0000","last_became_peered":"2024-05-15T13:13:11.361366+0000","last_unstale":"2024-05-15T13:24:11.367289+0000","last_undegraded":"2024-05-15T13:24:11.367289+0000","last_fullsized":"2024-05-15T13:24:11.367289+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:38.906446+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:23:38.906446+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:53.778316+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.c","version":"840'5655","reported_seq":5727,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:13.043937+0000","last_change":"2024-05-15T13:22:59.336651+0000","last_active":"2024-05-15T13:24:13.043937+0000","last_peered":"2024-05-15T13:24:13.043937+0000","last_clean":"2024-05-15T13:24:13.043937+0000","last_became_active":"2024-05-15T13:22:59.336347+0000","last_became_peered":"2024-05-15T13:22:59.336347+0000","last_unstale":"2024-05-15T13:24:13.043937+0000","last_undegraded":"2024-05-15T13:24:13.043937+0000","last_fullsized":"2024-05-15T13:24:13.043937+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":5655,"log_dups_size":0,"ondisk_log_size":5655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:15.662668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370606080,"num_objects":5655,"num_object_clones":0,"num_object_copies":16965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5655,"num_write_kb":361920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,0],"acting":[3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"94'139","reported_seq":1636,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:11.362365+0000","last_change":"2024-05-15T13:23:04.931444+0000","last_active":"2024-05-15T13:24:11.362365+0000","last_peered":"2024-05-15T13:24:11.362365+0000","last_clean":"2024-05-15T13:24:11.362365+0000","last_became_active":"2024-05-15T13:22:48.740313+0000","last_became_peered":"2024-05-15T13:22:48.740313+0000","last_unstale":"2024-05-15T13:24:11.362365+0000","last_undegraded":"2024-05-15T13:24:11.362365+0000","last_fullsized":"2024-05-15T13:24:11.362365+0000","mapping_epoch":805,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":806,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:04.931319+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:23:04.931319+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:33.570872+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"841'6024","reported_seq":6097,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-05-15T13:24:15.888884+0000","last_change":"2024-05-15T13:22:58.856881+0000","last_active":"2024-05-15T13:24:15.888884+0000","last_peered":"2024-05-15T13:24:15.888884+0000","last_clean":"2024-05-15T13:24:15.888884+0000","last_became_active":"2024-05-15T13:22:58.856595+0000","last_became_peered":"2024-05-15T13:22:58.856595+0000","last_unstale":"2024-05-15T13:24:15.888884+0000","last_undegraded":"2024-05-15T13:24:15.888884+0000","last_fullsized":"2024-05-15T13:24:15.888884+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":6024,"log_dups_size":0,"ondisk_log_size":6024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394788864,"num_objects":6024,"num_object_clones":0,"num_object_copies":18072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6024,"num_write_kb":385536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"94'139","reported_seq":1377,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:11.369432+0000","last_change":"2024-05-15T13:24:06.307879+0000","last_active":"2024-05-15T13:24:11.369432+0000","last_peered":"2024-05-15T13:24:11.369432+0000","last_clean":"2024-05-15T13:24:11.369432+0000","last_became_active":"2024-05-15T13:24:06.307598+0000","last_became_peered":"2024-05-15T13:24:06.307598+0000","last_unstale":"2024-05-15T13:24:11.369432+0000","last_undegraded":"2024-05-15T13:24:11.369432+0000","last_fullsized":"2024-05-15T13:24:11.369432+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:51.887016+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:18:24.970285+0000","last_clean_scrub_stamp":"2024-05-15T13:22:51.887016+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:12.835235+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"841'5832","reported_seq":5905,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-05-15T13:24:15.888590+0000","last_change":"2024-05-15T13:22:58.852680+0000","last_active":"2024-05-15T13:24:15.888590+0000","last_peered":"2024-05-15T13:24:15.888590+0000","last_clean":"2024-05-15T13:24:15.888590+0000","last_became_active":"2024-05-15T13:22:58.852354+0000","last_became_peered":"2024-05-15T13:22:58.852354+0000","last_unstale":"2024-05-15T13:24:15.888590+0000","last_undegraded":"2024-05-15T13:24:15.888590+0000","last_fullsized":"2024-05-15T13:24:15.888590+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":5832,"log_dups_size":0,"ondisk_log_size":5832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:19.342516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382205952,"num_objects":5832,"num_object_clones":0,"num_object_copies":17496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5832,"num_write_kb":373248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,1],"acting":[7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"94'139","reported_seq":1596,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:11.364613+0000","last_change":"2024-05-15T13:24:06.300535+0000","last_active":"2024-05-15T13:24:11.364613+0000","last_peered":"2024-05-15T13:24:11.364613+0000","last_clean":"2024-05-15T13:24:11.364613+0000","last_became_active":"2024-05-15T13:24:06.300206+0000","last_became_peered":"2024-05-15T13:24:06.300206+0000","last_unstale":"2024-05-15T13:24:11.364613+0000","last_undegraded":"2024-05-15T13:24:11.364613+0000","last_fullsized":"2024-05-15T13:24:11.364613+0000","mapping_epoch":822,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:37.717703+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:23:37.717703+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:12.556123+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,6],"acting":[2,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.f","version":"840'5889","reported_seq":5963,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:13.743203+0000","last_change":"2024-05-15T13:23:32.734479+0000","last_active":"2024-05-15T13:24:13.743203+0000","last_peered":"2024-05-15T13:24:13.743203+0000","last_clean":"2024-05-15T13:24:13.743203+0000","last_became_active":"2024-05-15T13:22:58.857554+0000","last_became_peered":"2024-05-15T13:22:58.857554+0000","last_unstale":"2024-05-15T13:24:13.743203+0000","last_undegraded":"2024-05-15T13:24:13.743203+0000","last_fullsized":"2024-05-15T13:24:13.743203+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":5889,"log_dups_size":0,"ondisk_log_size":5889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385941504,"num_objects":5889,"num_object_clones":0,"num_object_copies":17667,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5889,"num_write_kb":376896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"94'139","reported_seq":1384,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-05-15T13:24:14.284002+0000","last_change":"2024-05-15T13:24:06.313105+0000","last_active":"2024-05-15T13:24:14.284002+0000","last_peered":"2024-05-15T13:24:14.284002+0000","last_clean":"2024-05-15T13:24:14.284002+0000","last_became_active":"2024-05-15T13:24:06.312706+0000","last_became_peered":"2024-05-15T13:24:06.312706+0000","last_unstale":"2024-05-15T13:24:14.284002+0000","last_undegraded":"2024-05-15T13:24:14.284002+0000","last_fullsized":"2024-05-15T13:24:14.284002+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:49.429485+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:23:49.429485+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:18.265562+0000","scrub_duration":188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"840'5676","reported_seq":5748,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:11.971209+0000","last_change":"2024-05-15T13:22:58.856422+0000","last_active":"2024-05-15T13:24:11.971209+0000","last_peered":"2024-05-15T13:24:11.971209+0000","last_clean":"2024-05-15T13:24:11.971209+0000","last_became_active":"2024-05-15T13:22:58.855908+0000","last_became_peered":"2024-05-15T13:22:58.855908+0000","last_unstale":"2024-05-15T13:24:11.971209+0000","last_undegraded":"2024-05-15T13:24:11.971209+0000","last_fullsized":"2024-05-15T13:24:11.971209+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":5676,"log_dups_size":0,"ondisk_log_size":5676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:25.461974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371982336,"num_objects":5676,"num_object_clones":0,"num_object_copies":17028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5676,"num_write_kb":363264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"94'139","reported_seq":1399,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-05-15T13:24:14.290317+0000","last_change":"2024-05-15T13:23:38.468949+0000","last_active":"2024-05-15T13:24:14.290317+0000","last_peered":"2024-05-15T13:24:14.290317+0000","last_clean":"2024-05-15T13:24:14.290317+0000","last_became_active":"2024-05-15T13:15:43.614970+0000","last_became_peered":"2024-05-15T13:15:43.614970+0000","last_unstale":"2024-05-15T13:24:14.290317+0000","last_undegraded":"2024-05-15T13:24:14.290317+0000","last_fullsized":"2024-05-15T13:24:14.290317+0000","mapping_epoch":678,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:38.468826+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:23:38.468826+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:25:01.495545+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,0],"acting":[9,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.a","version":"840'5634","reported_seq":5706,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:13.724482+0000","last_change":"2024-05-15T13:22:58.858222+0000","last_active":"2024-05-15T13:24:13.724482+0000","last_peered":"2024-05-15T13:24:13.724482+0000","last_clean":"2024-05-15T13:24:13.724482+0000","last_became_active":"2024-05-15T13:22:58.857879+0000","last_became_peered":"2024-05-15T13:22:58.857879+0000","last_unstale":"2024-05-15T13:24:13.724482+0000","last_undegraded":"2024-05-15T13:24:13.724482+0000","last_fullsized":"2024-05-15T13:24:13.724482+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":5634,"log_dups_size":0,"ondisk_log_size":5634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369229824,"num_objects":5634,"num_object_clones":0,"num_object_copies":16902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5634,"num_write_kb":360576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"94'139","reported_seq":1635,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-05-15T13:24:14.283308+0000","last_change":"2024-05-15T13:22:57.893531+0000","last_active":"2024-05-15T13:24:14.283308+0000","last_peered":"2024-05-15T13:24:14.283308+0000","last_clean":"2024-05-15T13:24:14.283308+0000","last_became_active":"2024-05-15T13:22:48.739056+0000","last_became_peered":"2024-05-15T13:22:48.739056+0000","last_unstale":"2024-05-15T13:24:14.283308+0000","last_undegraded":"2024-05-15T13:24:14.283308+0000","last_fullsized":"2024-05-15T13:24:14.283308+0000","mapping_epoch":805,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":806,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:57.893417+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.893417+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:26.390879+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"840'5824","reported_seq":5896,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:13.733030+0000","last_change":"2024-05-15T13:22:58.858310+0000","last_active":"2024-05-15T13:24:13.733030+0000","last_peered":"2024-05-15T13:24:13.733030+0000","last_clean":"2024-05-15T13:24:13.733030+0000","last_became_active":"2024-05-15T13:22:58.857910+0000","last_became_peered":"2024-05-15T13:22:58.857910+0000","last_unstale":"2024-05-15T13:24:13.733030+0000","last_undegraded":"2024-05-15T13:24:13.733030+0000","last_fullsized":"2024-05-15T13:24:13.733030+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":5824,"log_dups_size":0,"ondisk_log_size":5824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:27.081337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381681664,"num_objects":5824,"num_object_clones":0,"num_object_copies":17472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5824,"num_write_kb":372736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"94'139","reported_seq":1645,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:11.367869+0000","last_change":"2024-05-15T13:22:55.272017+0000","last_active":"2024-05-15T13:24:11.367869+0000","last_peered":"2024-05-15T13:24:11.367869+0000","last_clean":"2024-05-15T13:24:11.367869+0000","last_became_active":"2024-05-15T13:22:48.738778+0000","last_became_peered":"2024-05-15T13:22:48.738778+0000","last_unstale":"2024-05-15T13:24:11.367869+0000","last_undegraded":"2024-05-15T13:24:11.367869+0000","last_fullsized":"2024-05-15T13:24:11.367869+0000","mapping_epoch":805,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":806,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:55.271911+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:22:55.271911+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:14.076929+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.0","version":"840'5570","reported_seq":5642,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:13.183869+0000","last_change":"2024-05-15T13:22:59.208928+0000","last_active":"2024-05-15T13:24:13.183869+0000","last_peered":"2024-05-15T13:24:13.183869+0000","last_clean":"2024-05-15T13:24:13.183869+0000","last_became_active":"2024-05-15T13:22:59.208576+0000","last_became_peered":"2024-05-15T13:22:59.208576+0000","last_unstale":"2024-05-15T13:24:13.183869+0000","last_undegraded":"2024-05-15T13:24:13.183869+0000","last_fullsized":"2024-05-15T13:24:13.183869+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":5570,"log_dups_size":0,"ondisk_log_size":5570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:17.422924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365035520,"num_objects":5570,"num_object_clones":0,"num_object_copies":16710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5570,"num_write_kb":356480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"94'139","reported_seq":1618,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:11.361663+0000","last_change":"2024-05-15T13:24:06.308064+0000","last_active":"2024-05-15T13:24:11.361663+0000","last_peered":"2024-05-15T13:24:11.361663+0000","last_clean":"2024-05-15T13:24:11.361663+0000","last_became_active":"2024-05-15T13:24:06.307697+0000","last_became_peered":"2024-05-15T13:24:06.307697+0000","last_unstale":"2024-05-15T13:24:11.361663+0000","last_undegraded":"2024-05-15T13:24:11.361663+0000","last_fullsized":"2024-05-15T13:24:11.361663+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:47.391631+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:23:47.391631+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:13.149650+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131344,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"840'5696","reported_seq":5768,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:12.997318+0000","last_change":"2024-05-15T13:22:59.209016+0000","last_active":"2024-05-15T13:24:12.997318+0000","last_peered":"2024-05-15T13:24:12.997318+0000","last_clean":"2024-05-15T13:24:12.997318+0000","last_became_active":"2024-05-15T13:22:59.208619+0000","last_became_peered":"2024-05-15T13:22:59.208619+0000","last_unstale":"2024-05-15T13:24:12.997318+0000","last_undegraded":"2024-05-15T13:24:12.997318+0000","last_fullsized":"2024-05-15T13:24:12.997318+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":5696,"log_dups_size":0,"ondisk_log_size":5696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:18.411120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373293056,"num_objects":5696,"num_object_clones":0,"num_object_copies":17088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5696,"num_write_kb":364544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"94'139","reported_seq":993,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-05-15T13:24:14.283194+0000","last_change":"2024-05-15T13:24:06.300741+0000","last_active":"2024-05-15T13:24:14.283194+0000","last_peered":"2024-05-15T13:24:14.283194+0000","last_clean":"2024-05-15T13:24:14.283194+0000","last_became_active":"2024-05-15T13:24:06.300367+0000","last_became_peered":"2024-05-15T13:24:06.300367+0000","last_unstale":"2024-05-15T13:24:14.283194+0000","last_undegraded":"2024-05-15T13:24:14.283194+0000","last_fullsized":"2024-05-15T13:24:14.283194+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:12.031939+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:23:12.031939+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:18.351447+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2","version":"840'5737","reported_seq":5808,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:13.061008+0000","last_change":"2024-05-15T13:22:58.851932+0000","last_active":"2024-05-15T13:24:13.061008+0000","last_peered":"2024-05-15T13:24:13.061008+0000","last_clean":"2024-05-15T13:24:13.061008+0000","last_became_active":"2024-05-15T13:22:58.851656+0000","last_became_peered":"2024-05-15T13:22:58.851656+0000","last_unstale":"2024-05-15T13:24:13.061008+0000","last_undegraded":"2024-05-15T13:24:13.061008+0000","last_fullsized":"2024-05-15T13:24:13.061008+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":5737,"log_dups_size":0,"ondisk_log_size":5737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:27.197438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375980032,"num_objects":5737,"num_object_clones":0,"num_object_copies":17211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5737,"num_write_kb":367168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"94'139","reported_seq":1398,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-05-15T13:24:14.288435+0000","last_change":"2024-05-15T13:24:06.304310+0000","last_active":"2024-05-15T13:24:14.288435+0000","last_peered":"2024-05-15T13:24:14.288435+0000","last_clean":"2024-05-15T13:24:14.288435+0000","last_became_active":"2024-05-15T13:24:06.304008+0000","last_became_peered":"2024-05-15T13:24:06.304008+0000","last_unstale":"2024-05-15T13:24:14.288435+0000","last_undegraded":"2024-05-15T13:24:14.288435+0000","last_fullsized":"2024-05-15T13:24:14.288435+0000","mapping_epoch":678,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:39.504559+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:22:03.418345+0000","last_clean_scrub_stamp":"2024-05-15T13:23:39.504559+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:17.439118+0000","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.6","version":"840'5671","reported_seq":5743,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:13.731660+0000","last_change":"2024-05-15T13:22:58.859332+0000","last_active":"2024-05-15T13:24:13.731660+0000","last_peered":"2024-05-15T13:24:13.731660+0000","last_clean":"2024-05-15T13:24:13.731660+0000","last_became_active":"2024-05-15T13:22:58.859007+0000","last_became_peered":"2024-05-15T13:22:58.859007+0000","last_unstale":"2024-05-15T13:24:13.731660+0000","last_undegraded":"2024-05-15T13:24:13.731660+0000","last_fullsized":"2024-05-15T13:24:13.731660+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":5671,"log_dups_size":0,"ondisk_log_size":5671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371654656,"num_objects":5671,"num_object_clones":0,"num_object_copies":17013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5671,"num_write_kb":362944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"94'139","reported_seq":1005,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-05-15T13:24:14.283413+0000","last_change":"2024-05-15T13:24:06.304234+0000","last_active":"2024-05-15T13:24:14.283413+0000","last_peered":"2024-05-15T13:24:14.283413+0000","last_clean":"2024-05-15T13:24:14.283413+0000","last_became_active":"2024-05-15T13:24:06.303921+0000","last_became_peered":"2024-05-15T13:24:06.303921+0000","last_unstale":"2024-05-15T13:24:14.283413+0000","last_undegraded":"2024-05-15T13:24:14.283413+0000","last_fullsized":"2024-05-15T13:24:14.283413+0000","mapping_epoch":805,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:06.819156+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:23:06.819156+0000","last_clean_scrub_stamp":"2024-05-15T13:23:06.819156+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:18.351481+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"840'5672","reported_seq":5744,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:12.005023+0000","last_change":"2024-05-15T13:22:58.852908+0000","last_active":"2024-05-15T13:24:12.005023+0000","last_peered":"2024-05-15T13:24:12.005023+0000","last_clean":"2024-05-15T13:24:12.005023+0000","last_became_active":"2024-05-15T13:22:58.852401+0000","last_became_peered":"2024-05-15T13:22:58.852401+0000","last_unstale":"2024-05-15T13:24:12.005023+0000","last_undegraded":"2024-05-15T13:24:12.005023+0000","last_fullsized":"2024-05-15T13:24:12.005023+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":5672,"log_dups_size":0,"ondisk_log_size":5672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:23.299876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371720192,"num_objects":5672,"num_object_clones":0,"num_object_copies":17016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5672,"num_write_kb":363008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"94'139","reported_seq":1378,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-05-15T13:24:14.284201+0000","last_change":"2024-05-15T13:23:54.488876+0000","last_active":"2024-05-15T13:24:14.284201+0000","last_peered":"2024-05-15T13:24:14.284201+0000","last_clean":"2024-05-15T13:24:14.284201+0000","last_became_active":"2024-05-15T13:13:16.472318+0000","last_became_peered":"2024-05-15T13:13:16.472318+0000","last_unstale":"2024-05-15T13:24:14.284201+0000","last_undegraded":"2024-05-15T13:24:14.284201+0000","last_fullsized":"2024-05-15T13:24:14.284201+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:54.488739+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:23:54.488739+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:25:07.213480+0000","scrub_duration":245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"840'5708","reported_seq":5780,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:13.171912+0000","last_change":"2024-05-15T13:22:59.336585+0000","last_active":"2024-05-15T13:24:13.171912+0000","last_peered":"2024-05-15T13:24:13.171912+0000","last_clean":"2024-05-15T13:24:13.171912+0000","last_became_active":"2024-05-15T13:22:59.336258+0000","last_became_peered":"2024-05-15T13:22:59.336258+0000","last_unstale":"2024-05-15T13:24:13.171912+0000","last_undegraded":"2024-05-15T13:24:13.171912+0000","last_fullsized":"2024-05-15T13:24:13.171912+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":5708,"log_dups_size":0,"ondisk_log_size":5708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:14.076891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374079488,"num_objects":5708,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5708,"num_write_kb":365312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"94'139","reported_seq":1041,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:11.371233+0000","last_change":"2024-05-15T13:24:06.308683+0000","last_active":"2024-05-15T13:24:11.371233+0000","last_peered":"2024-05-15T13:24:11.371233+0000","last_clean":"2024-05-15T13:24:11.371233+0000","last_became_active":"2024-05-15T13:24:06.308068+0000","last_became_peered":"2024-05-15T13:24:06.308068+0000","last_unstale":"2024-05-15T13:24:11.371233+0000","last_undegraded":"2024-05-15T13:24:11.371233+0000","last_fullsized":"2024-05-15T13:24:11.371233+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:56.794428+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:22:56.794428+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:12.623652+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163840,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"840'5726","reported_seq":5798,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:12.849875+0000","last_change":"2024-05-15T13:22:58.856916+0000","last_active":"2024-05-15T13:24:12.849875+0000","last_peered":"2024-05-15T13:24:12.849875+0000","last_clean":"2024-05-15T13:24:12.849875+0000","last_became_active":"2024-05-15T13:22:58.856523+0000","last_became_peered":"2024-05-15T13:22:58.856523+0000","last_unstale":"2024-05-15T13:24:12.849875+0000","last_undegraded":"2024-05-15T13:24:12.849875+0000","last_fullsized":"2024-05-15T13:24:12.849875+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":5726,"log_dups_size":0,"ondisk_log_size":5726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:24.728154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375259136,"num_objects":5726,"num_object_clones":0,"num_object_copies":17178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5726,"num_write_kb":366464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"94'139","reported_seq":1042,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-05-15T13:24:14.283383+0000","last_change":"2024-05-15T13:24:06.301362+0000","last_active":"2024-05-15T13:24:14.283383+0000","last_peered":"2024-05-15T13:24:14.283383+0000","last_clean":"2024-05-15T13:24:14.283383+0000","last_became_active":"2024-05-15T13:24:06.300953+0000","last_became_peered":"2024-05-15T13:24:06.300953+0000","last_unstale":"2024-05-15T13:24:14.283383+0000","last_undegraded":"2024-05-15T13:24:14.283383+0000","last_fullsized":"2024-05-15T13:24:14.283383+0000","mapping_epoch":805,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:03.856199+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:23:03.856199+0000","last_clean_scrub_stamp":"2024-05-15T13:23:03.856199+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:18.351512+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3","version":"841'5952","reported_seq":6026,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-05-15T13:24:15.717131+0000","last_change":"2024-05-15T13:22:58.857200+0000","last_active":"2024-05-15T13:24:15.717131+0000","last_peered":"2024-05-15T13:24:15.717131+0000","last_clean":"2024-05-15T13:24:15.717131+0000","last_became_active":"2024-05-15T13:22:58.856850+0000","last_became_peered":"2024-05-15T13:22:58.856850+0000","last_unstale":"2024-05-15T13:24:15.717131+0000","last_undegraded":"2024-05-15T13:24:15.717131+0000","last_fullsized":"2024-05-15T13:24:15.717131+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:22:57.831174+0000","last_clean_scrub_stamp":"2024-05-15T13:22:57.831174+0000","objects_scrubbed":0,"log_size":5952,"log_dups_size":0,"ondisk_log_size":5952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:16.399162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390070272,"num_objects":5952,"num_object_clones":0,"num_object_copies":17856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5952,"num_write_kb":380928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"94'139","reported_seq":1641,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:11.362160+0000","last_change":"2024-05-15T13:24:06.309304+0000","last_active":"2024-05-15T13:24:11.362160+0000","last_peered":"2024-05-15T13:24:11.362160+0000","last_clean":"2024-05-15T13:24:11.362160+0000","last_became_active":"2024-05-15T13:24:06.308587+0000","last_became_peered":"2024-05-15T13:24:06.308587+0000","last_unstale":"2024-05-15T13:24:11.362160+0000","last_undegraded":"2024-05-15T13:24:11.362160+0000","last_fullsized":"2024-05-15T13:24:11.362160+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:22:56.003105+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:22:56.003105+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:13.229828+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"94'139","reported_seq":1635,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:11.361493+0000","last_change":"2024-05-15T13:23:43.413436+0000","last_active":"2024-05-15T13:24:11.361493+0000","last_peered":"2024-05-15T13:24:11.361493+0000","last_clean":"2024-05-15T13:24:11.361493+0000","last_became_active":"2024-05-15T13:19:41.190923+0000","last_became_peered":"2024-05-15T13:19:41.190923+0000","last_unstale":"2024-05-15T13:24:11.361493+0000","last_undegraded":"2024-05-15T13:24:11.361493+0000","last_fullsized":"2024-05-15T13:24:11.361493+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:43.413325+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:23:43.413325+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:25:04.504248+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"94'139","reported_seq":1403,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-05-15T13:24:14.289498+0000","last_change":"2024-05-15T13:23:36.340763+0000","last_active":"2024-05-15T13:24:14.289498+0000","last_peered":"2024-05-15T13:24:14.289498+0000","last_clean":"2024-05-15T13:24:14.289498+0000","last_became_active":"2024-05-15T13:22:48.739476+0000","last_became_peered":"2024-05-15T13:22:48.739476+0000","last_unstale":"2024-05-15T13:24:14.289498+0000","last_undegraded":"2024-05-15T13:24:14.289498+0000","last_fullsized":"2024-05-15T13:24:14.289498+0000","mapping_epoch":805,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":806,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:36.340646+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:23:36.340646+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:45.341580+0000","scrub_duration":267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,3],"acting":[0,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"94'139","reported_seq":1642,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:11.361494+0000","last_change":"2024-05-15T13:23:08.852185+0000","last_active":"2024-05-15T13:24:11.361494+0000","last_peered":"2024-05-15T13:24:11.361494+0000","last_clean":"2024-05-15T13:24:11.361494+0000","last_became_active":"2024-05-15T13:13:49.926687+0000","last_became_peered":"2024-05-15T13:13:49.926687+0000","last_unstale":"2024-05-15T13:24:11.361494+0000","last_undegraded":"2024-05-15T13:24:11.361494+0000","last_fullsized":"2024-05-15T13:24:11.361494+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:08.852053+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:18:06.340555+0000","last_clean_scrub_stamp":"2024-05-15T13:23:08.852053+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:16.137996+0000","scrub_duration":121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"94'139","reported_seq":1643,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-05-15T13:24:11.367665+0000","last_change":"2024-05-15T13:23:38.179779+0000","last_active":"2024-05-15T13:24:11.367665+0000","last_peered":"2024-05-15T13:24:11.367665+0000","last_clean":"2024-05-15T13:24:11.367665+0000","last_became_active":"2024-05-15T13:13:16.478488+0000","last_became_peered":"2024-05-15T13:13:16.478488+0000","last_unstale":"2024-05-15T13:24:11.367665+0000","last_undegraded":"2024-05-15T13:24:11.367665+0000","last_fullsized":"2024-05-15T13:24:11.367665+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:23:38.179664+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:23:38.179664+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:24:47.936598+0000","scrub_duration":286,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6037045248,"num_objects":92118,"num_object_clones":0,"num_object_copies":276354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92118,"num_write_kb":5895552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18122735616,"data_stored":18122735616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92118,"ondisk_log_size":92118,"up":48,"acting":48,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":489144,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":13,"num_write":19,"num_write_kb":313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":765792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057106,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2230000,"kb_used_data":1436584,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":91515152,"statfs":{"total":95995035648,"available":93711515648,"internally_reserved":0,"allocated":1471062016,"data_stored":1470628154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":812382662},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351446,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2651000,"kb_used_data":1795176,"kb_used_omap":1,"kb_used_meta":855806,"kb_avail":91094152,"statfs":{"total":95995035648,"available":93280411648,"internally_reserved":0,"allocated":1838260224,"data_stored":1837826362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":876345792},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514972,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3224000,"kb_used_data":2268212,"kb_used_omap":1,"kb_used_meta":955774,"kb_avail":90521152,"statfs":{"total":95995035648,"available":92693659648,"internally_reserved":0,"allocated":2322649088,"data_stored":2322210740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":978713024},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":0,"seq":279172874721,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169062,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2185028,"kb_used_data":1883132,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":91560124,"statfs":{"total":95995035648,"available":93757566976,"internally_reserved":0,"allocated":1928327168,"data_stored":1927885268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":309131722},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430700,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1501104,"kb_used_data":734120,"kb_used_omap":1,"kb_used_meta":766974,"kb_avail":92244048,"statfs":{"total":95995035648,"available":94457905152,"internally_reserved":0,"allocated":751738880,"data_stored":751305018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":785381830},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575364,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4344368,"kb_used_data":3280296,"kb_used_omap":1,"kb_used_meta":1064062,"kb_avail":89400784,"statfs":{"total":95995035648,"available":91546402816,"internally_reserved":0,"allocated":3359023104,"data_stored":3358589242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1089599941},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738895,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1588352,"kb_used_data":740852,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":92156800,"statfs":{"total":95995035648,"available":94368563200,"internally_reserved":0,"allocated":758632448,"data_stored":758194100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":867826117},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411841,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":623940,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93121212,"statfs":{"total":95995035648,"available":95356121088,"internally_reserved":0,"allocated":1359872,"data_stored":922458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150203,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3202292,"kb_used_data":2182632,"kb_used_omap":1,"kb_used_meta":1019646,"kb_avail":90542860,"statfs":{"total":95995035648,"available":92715888640,"internally_reserved":0,"allocated":2235015168,"data_stored":2234581306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1044117959},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986679,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692337,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4133436,"kb_used_data":3294256,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":89611716,"statfs":{"total":95995035648,"available":91762397184,"internally_reserved":0,"allocated":3373318144,"data_stored":3372880730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":859306437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":759234560,"data_stored":759234560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3361603584,"data_stored":3361603584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2245591040,"data_stored":2245591040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3403153408,"data_stored":3403153408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":752418816,"data_stored":752418816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1943273472,"data_stored":1943273472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2333736960,"data_stored":2333736960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1841430528,"data_stored":1841430528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1482293248,"data_stored":1482293248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:24:17.751 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool set unique_pool_7 pgp_num 16 2024-05-15T13:24:17.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:17 smithi120 ceph-mon[43961]: osdmap e842: 12 total, 12 up, 10 in 2024-05-15T13:24:17.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:17 smithi120 ceph-mon[43961]: from='client.16695 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:24:17.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:17 smithi120 ceph-mon[43961]: pgmap v2180: 47 pgs: 47 active+clean; 5.6 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 1.8 KiB/s rd, 83 MiB/s wr, 1.33k op/s; 88 KiB/s, 0 objects/s recovering 2024-05-15T13:24:17.860 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:24:17.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:17.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:24:17.968 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:24:18.010 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:18.033 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:24:18.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:18.036 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:24:18.137 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:24:18.169 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:18.198 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:24:18.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:18.205 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:24:18.227 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:24:18.306 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:24:18.313 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:24:18.315 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 66 16 97832 97816 92.599 65.125 0.019259 0.0107954 2024-05-15T13:24:18.349 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:18.426 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:18 smithi027 ceph-mon[39795]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-05-15T13:24:18.426 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:18 smithi177 ceph-mon[44303]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-05-15T13:24:18.476 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:24:18.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:18.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:24:18.519 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-05-15T13:24:18.585 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:24:18.594 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:18 smithi120 ceph-mon[43961]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-05-15T13:24:18.622 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:18.654 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:24:18.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:18.659 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:24:18.760 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:24:18.819 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:18.847 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:24:18.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:18.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:24:18.947 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:24:18.954 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:24:19.000 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:19.264 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:24:19.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:19.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:24:19.316 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 67 16 99090 99074 92.3904 78.625 0.0116017 0.0108197 2024-05-15T13:24:19.368 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:24:19.414 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:19.445 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:24:19.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:19.453 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:24:19.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:19 smithi120 ceph-mon[43961]: pgmap v2181: 47 pgs: 47 active+clean; 5.8 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 1.1 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 60 KiB/s, 0 objects/s recovering 2024-05-15T13:24:19.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:19 smithi120 ceph-mon[43961]: osdmap e843: 12 total, 12 up, 10 in 2024-05-15T13:24:19.555 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:24:19.597 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:19.627 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:24:19.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:19.641 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:24:19.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:19 smithi177 ceph-mon[44303]: pgmap v2181: 47 pgs: 47 active+clean; 5.8 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 1.1 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 60 KiB/s, 0 objects/s recovering 2024-05-15T13:24:19.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:19 smithi177 ceph-mon[44303]: osdmap e843: 12 total, 12 up, 10 in 2024-05-15T13:24:19.742 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:24:19.779 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:19.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:19 smithi027 ceph-mon[39795]: pgmap v2181: 47 pgs: 47 active+clean; 5.8 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 1.1 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 60 KiB/s, 0 objects/s recovering 2024-05-15T13:24:19.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:19 smithi027 ceph-mon[39795]: osdmap e843: 12 total, 12 up, 10 in 2024-05-15T13:24:20.315 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 68 16 100308 100292 92.1512 76.125 0.00471175 0.010848 2024-05-15T13:24:20.674 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:24:20.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:20.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:24:20.781 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:24:20.822 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:20.907 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:24:20.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:20.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:24:21.013 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:24:21.058 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:21.084 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:24:21.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:21.087 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:24:21.188 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:24:21.240 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:21.282 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:24:21.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:21.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:24:21.315 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 69 16 101555 101539 91.9451 77.9375 0.00827807 0.0108721 2024-05-15T13:24:21.391 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:24:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:21.754 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:21 smithi027 ceph-mon[39795]: pgmap v2183: 47 pgs: 47 active+clean; 5.9 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 62 MiB/s wr, 991 op/s 2024-05-15T13:24:21.917 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:24:21.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:21.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:24:22.024 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:24:22.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:21 smithi120 ceph-mon[43961]: pgmap v2183: 47 pgs: 47 active+clean; 5.9 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 62 MiB/s wr, 991 op/s 2024-05-15T13:24:22.048 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:21 smithi177 ceph-mon[44303]: pgmap v2183: 47 pgs: 47 active+clean; 5.9 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 62 MiB/s wr, 991 op/s 2024-05-15T13:24:22.050 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:22.070 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:24:22.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:22.073 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:24:22.175 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:24:22.259 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:22.316 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 70 16 102668 102652 91.6253 69.5625 0.0141888 0.0109099 2024-05-15T13:24:22.595 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:24:22.595 INFO:teuthology.orchestra.run.smithi120.stdout: "size": 20, 2024-05-15T13:24:22.595 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 600, 2024-05-15T13:24:22.595 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [ 2024-05-15T13:24:22.596 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.596 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:125708 7.7 7.7b696437 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e657)", 2024-05-15T13:24:22.596 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:14:50.450084+0000", 2024-05-15T13:24:22.596 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 572.14396628099996, 2024-05-15T13:24:22.596 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.004634933, 2024-05-15T13:24:22.596 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:22.596 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:24:22.596 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:22.596 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:24:22.596 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:24:22.596 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 125708 2024-05-15T13:24:22.596 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.596 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:22.596 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.596 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:22.596 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.450084+0000", 2024-05-15T13:24:22.597 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.597 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.597 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.597 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:22.597 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.450084+0000", 2024-05-15T13:24:22.597 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.597 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.597 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.597 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:22.597 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.450085+0000", 2024-05-15T13:24:22.597 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.393e-06 2024-05-15T13:24:22.597 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.597 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.597 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:22.597 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.450092+0000", 2024-05-15T13:24:22.598 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6059999999999999e-06 2024-05-15T13:24:22.598 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.598 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.598 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:22.598 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.450108+0000", 2024-05-15T13:24:22.598 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5543999999999999e-05 2024-05-15T13:24:22.598 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.598 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.598 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:22.598 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.450116+0000", 2024-05-15T13:24:22.601 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8650000000000007e-06 2024-05-15T13:24:22.601 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.601 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.601 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.602 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.456666+0000", 2024-05-15T13:24:22.602 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0065495869999999999 2024-05-15T13:24:22.602 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.602 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.602 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:24:22.602 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.456675+0000", 2024-05-15T13:24:22.602 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.054e-06 2024-05-15T13:24:22.602 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.602 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.602 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.602 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454707+0000", 2024-05-15T13:24:22.602 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.99803161399999996 2024-05-15T13:24:22.602 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.603 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.603 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:22.603 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454719+0000", 2024-05-15T13:24:22.603 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2269999999999999e-05 2024-05-15T13:24:22.603 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.603 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:22.603 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.603 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.603 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.603 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:125717 7.d 7.75ad9b4d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-05-15T13:24:22.603 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:14:50.461218+0000", 2024-05-15T13:24:22.603 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 572.13283235100005, 2024-05-15T13:24:22.603 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.99365158099999995, 2024-05-15T13:24:22.603 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:22.603 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:24:22.604 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:22.604 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:24:22.604 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:24:22.604 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 125717 2024-05-15T13:24:22.604 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.604 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:22.604 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.604 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:22.604 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.461218+0000", 2024-05-15T13:24:22.604 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.604 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.604 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.604 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:22.604 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.461218+0000", 2024-05-15T13:24:22.604 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.604 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.605 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.605 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:22.605 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.461219+0000", 2024-05-15T13:24:22.605 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.449e-06 2024-05-15T13:24:22.605 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.605 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.605 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:22.605 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.461225+0000", 2024-05-15T13:24:22.605 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7919999999999998e-06 2024-05-15T13:24:22.605 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.605 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.605 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:22.605 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.461237+0000", 2024-05-15T13:24:22.605 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2187e-05 2024-05-15T13:24:22.605 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.605 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.606 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:22.606 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.461260+0000", 2024-05-15T13:24:22.606 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2277999999999999e-05 2024-05-15T13:24:22.606 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.606 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.606 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.606 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.463106+0000", 2024-05-15T13:24:22.606 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001846721 2024-05-15T13:24:22.606 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.606 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.606 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:24:22.606 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.463117+0000", 2024-05-15T13:24:22.606 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0231000000000001e-05 2024-05-15T13:24:22.606 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.607 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.607 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.607 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454858+0000", 2024-05-15T13:24:22.607 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.99174162499999996 2024-05-15T13:24:22.607 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.607 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.607 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:22.607 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454870+0000", 2024-05-15T13:24:22.607 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1297999999999999e-05 2024-05-15T13:24:22.607 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.607 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:22.607 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.607 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.607 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.607 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:125719 7.7 7.fd392cc7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-05-15T13:24:22.607 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:14:50.470608+0000", 2024-05-15T13:24:22.608 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 572.12344276900001, 2024-05-15T13:24:22.608 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.98416746300000002, 2024-05-15T13:24:22.608 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:22.608 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:24:22.608 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:22.608 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:24:22.608 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:24:22.608 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 125719 2024-05-15T13:24:22.608 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.608 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:22.608 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.608 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:22.608 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.470608+0000", 2024-05-15T13:24:22.608 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.608 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.608 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.609 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:22.609 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.470608+0000", 2024-05-15T13:24:22.609 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.609 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.609 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.609 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:22.609 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.470609+0000", 2024-05-15T13:24:22.609 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.361e-06 2024-05-15T13:24:22.609 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.609 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.609 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:22.609 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.470616+0000", 2024-05-15T13:24:22.609 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.52e-06 2024-05-15T13:24:22.609 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.609 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.609 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:22.610 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.470630+0000", 2024-05-15T13:24:22.610 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3389e-05 2024-05-15T13:24:22.610 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.610 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.610 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:22.610 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.470641+0000", 2024-05-15T13:24:22.610 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1181e-05 2024-05-15T13:24:22.610 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.610 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.610 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.610 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.470699+0000", 2024-05-15T13:24:22.610 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7704000000000002e-05 2024-05-15T13:24:22.610 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.610 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.610 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:24:22.611 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.470709+0000", 2024-05-15T13:24:22.611 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.825e-06 2024-05-15T13:24:22.611 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.611 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.611 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.611 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454764+0000", 2024-05-15T13:24:22.611 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.98405541100000005 2024-05-15T13:24:22.611 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.611 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.612 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:22.612 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454775+0000", 2024-05-15T13:24:22.612 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1072e-05 2024-05-15T13:24:22.612 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.613 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:22.613 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.613 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.613 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.613 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:125735 7.d 7.cdea4a0d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-05-15T13:24:22.613 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:14:50.508723+0000", 2024-05-15T13:24:22.613 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 572.08532737999997, 2024-05-15T13:24:22.613 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.94617932000000005, 2024-05-15T13:24:22.613 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:22.613 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:24:22.613 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:22.613 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:24:22.614 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:24:22.614 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 125735 2024-05-15T13:24:22.614 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.614 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:22.614 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.614 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:22.614 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.508723+0000", 2024-05-15T13:24:22.614 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.614 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.614 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.614 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:22.614 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.508723+0000", 2024-05-15T13:24:22.614 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.614 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.615 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.615 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:22.615 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.508731+0000", 2024-05-15T13:24:22.615 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7500000000000003e-06 2024-05-15T13:24:22.615 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.615 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.615 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:22.615 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.508737+0000", 2024-05-15T13:24:22.615 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8259999999999998e-06 2024-05-15T13:24:22.615 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.615 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.615 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:22.615 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.508749+0000", 2024-05-15T13:24:22.615 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2456999999999999e-05 2024-05-15T13:24:22.615 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.615 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.616 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:22.616 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.508757+0000", 2024-05-15T13:24:22.616 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9570000000000002e-06 2024-05-15T13:24:22.616 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.616 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.616 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.616 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.508805+0000", 2024-05-15T13:24:22.616 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8238999999999997e-05 2024-05-15T13:24:22.616 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.616 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.616 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:24:22.616 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.508814+0000", 2024-05-15T13:24:22.616 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1989999999999995e-06 2024-05-15T13:24:22.616 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.617 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.617 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.617 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454896+0000", 2024-05-15T13:24:22.617 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.94608192899999999 2024-05-15T13:24:22.617 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.617 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.617 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:22.617 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454902+0000", 2024-05-15T13:24:22.617 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9630000000000003e-06 2024-05-15T13:24:22.617 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.617 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:22.617 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.617 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.617 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.617 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:125762 7.7 7.81691657 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-05-15T13:24:22.617 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:14:50.586614+0000", 2024-05-15T13:24:22.618 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 572.00743662299999, 2024-05-15T13:24:22.618 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.86822153999999996, 2024-05-15T13:24:22.618 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:22.618 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:24:22.618 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:22.618 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:24:22.618 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:24:22.618 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 125762 2024-05-15T13:24:22.618 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.618 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:22.618 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.618 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:22.618 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.586614+0000", 2024-05-15T13:24:22.618 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.619 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.619 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.619 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:22.619 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.586614+0000", 2024-05-15T13:24:22.619 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.619 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.619 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.619 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:22.619 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.586615+0000", 2024-05-15T13:24:22.619 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.466e-06 2024-05-15T13:24:22.619 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.619 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.619 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:22.619 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.586621+0000", 2024-05-15T13:24:22.619 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9150000000000001e-06 2024-05-15T13:24:22.620 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.620 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.620 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:22.620 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.586637+0000", 2024-05-15T13:24:22.620 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6158999999999999e-05 2024-05-15T13:24:22.620 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.620 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.620 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:22.620 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.586660+0000", 2024-05-15T13:24:22.620 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2296000000000001e-05 2024-05-15T13:24:22.620 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.620 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.620 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.620 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.586741+0000", 2024-05-15T13:24:22.621 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1347000000000006e-05 2024-05-15T13:24:22.621 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.621 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.621 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:24:22.621 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.586752+0000", 2024-05-15T13:24:22.621 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0893000000000001e-05 2024-05-15T13:24:22.621 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.621 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.621 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.621 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454813+0000", 2024-05-15T13:24:22.621 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.86806092599999995 2024-05-15T13:24:22.621 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.621 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.621 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:22.621 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454835+0000", 2024-05-15T13:24:22.622 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2538e-05 2024-05-15T13:24:22.622 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.622 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:22.622 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.622 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.622 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.622 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:125785 7.d 7.270a8e6d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-05-15T13:24:22.622 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:14:50.672780+0000", 2024-05-15T13:24:22.622 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 571.92127035999999, 2024-05-15T13:24:22.622 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.78215774000000005, 2024-05-15T13:24:22.622 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:22.622 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:24:22.622 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:22.623 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:24:22.623 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:24:22.623 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 125785 2024-05-15T13:24:22.623 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.623 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:22.623 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.623 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:22.623 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.672780+0000", 2024-05-15T13:24:22.623 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.623 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.623 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.623 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:22.623 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.672780+0000", 2024-05-15T13:24:22.623 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.623 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.624 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.624 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:22.624 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.672782+0000", 2024-05-15T13:24:22.624 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.635e-06 2024-05-15T13:24:22.624 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.624 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.624 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:22.624 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.672789+0000", 2024-05-15T13:24:22.624 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2649999999999999e-06 2024-05-15T13:24:22.624 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.624 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.624 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:22.624 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.672804+0000", 2024-05-15T13:24:22.624 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5328999999999998e-05 2024-05-15T13:24:22.624 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.624 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.625 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:22.625 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.672812+0000", 2024-05-15T13:24:22.625 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1699999999999997e-06 2024-05-15T13:24:22.625 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.625 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.625 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.625 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.672869+0000", 2024-05-15T13:24:22.625 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6779e-05 2024-05-15T13:24:22.625 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.625 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.625 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:24:22.625 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.672878+0000", 2024-05-15T13:24:22.625 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1670000000000005e-06 2024-05-15T13:24:22.625 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.625 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.625 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.626 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454929+0000", 2024-05-15T13:24:22.626 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.78205035000000001 2024-05-15T13:24:22.626 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.626 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.626 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:22.626 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454938+0000", 2024-05-15T13:24:22.626 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0450000000000006e-06 2024-05-15T13:24:22.626 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.626 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:22.626 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.626 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.626 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.626 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:125787 7.7 7.a38c5a27 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-05-15T13:24:22.626 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:14:50.678532+0000", 2024-05-15T13:24:22.627 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 571.91551818300002, 2024-05-15T13:24:22.627 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.77632293600000002, 2024-05-15T13:24:22.627 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:22.627 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:24:22.627 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:22.627 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:24:22.627 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:24:22.627 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 125787 2024-05-15T13:24:22.627 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.627 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:22.627 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.627 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:22.627 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.678532+0000", 2024-05-15T13:24:22.627 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.628 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.628 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.628 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:22.628 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.678532+0000", 2024-05-15T13:24:22.628 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.628 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.628 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.628 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:22.628 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.678533+0000", 2024-05-15T13:24:22.628 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.215e-06 2024-05-15T13:24:22.628 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.628 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.628 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:22.628 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.678541+0000", 2024-05-15T13:24:22.629 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2250000000000003e-06 2024-05-15T13:24:22.629 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.629 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.629 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:22.629 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.678558+0000", 2024-05-15T13:24:22.629 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7462e-05 2024-05-15T13:24:22.629 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.629 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.629 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:22.629 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.678565+0000", 2024-05-15T13:24:22.629 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.1400000000000002e-06 2024-05-15T13:24:22.629 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.629 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.629 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.629 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.678614+0000", 2024-05-15T13:24:22.630 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8331000000000002e-05 2024-05-15T13:24:22.630 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.630 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.630 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:24:22.630 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.678619+0000", 2024-05-15T13:24:22.630 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9370000000000001e-06 2024-05-15T13:24:22.630 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.630 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.630 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.630 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454844+0000", 2024-05-15T13:24:22.630 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.77622497099999999 2024-05-15T13:24:22.630 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.630 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.630 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:22.630 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454855+0000", 2024-05-15T13:24:22.631 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0655e-05 2024-05-15T13:24:22.631 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.631 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:22.631 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.631 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.631 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.631 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:125796 7.d 7.e7ec442d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-05-15T13:24:22.631 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:14:50.716834+0000", 2024-05-15T13:24:22.631 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 571.87721606399998, 2024-05-15T13:24:22.631 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.73814268500000002, 2024-05-15T13:24:22.631 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:22.631 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:24:22.631 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:22.631 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:24:22.632 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:24:22.632 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 125796 2024-05-15T13:24:22.632 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.632 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:22.632 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.632 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:22.632 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.716834+0000", 2024-05-15T13:24:22.632 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.632 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.632 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.632 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:22.632 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.716834+0000", 2024-05-15T13:24:22.632 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.632 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.633 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.633 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:22.633 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.716836+0000", 2024-05-15T13:24:22.633 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.311e-06 2024-05-15T13:24:22.633 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.633 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.633 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:22.633 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.716842+0000", 2024-05-15T13:24:22.633 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.0279999999999997e-06 2024-05-15T13:24:22.633 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.633 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.633 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:22.633 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.716854+0000", 2024-05-15T13:24:22.633 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2130000000000001e-05 2024-05-15T13:24:22.633 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.634 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.634 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:22.634 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.716860+0000", 2024-05-15T13:24:22.634 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4500000000000001e-06 2024-05-15T13:24:22.634 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.634 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.634 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.634 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.716905+0000", 2024-05-15T13:24:22.634 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4802000000000002e-05 2024-05-15T13:24:22.634 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.634 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.634 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:24:22.634 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.716912+0000", 2024-05-15T13:24:22.634 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4880000000000001e-06 2024-05-15T13:24:22.634 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.635 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.635 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.635 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454969+0000", 2024-05-15T13:24:22.635 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.73805667200000002 2024-05-15T13:24:22.635 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.635 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.635 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:22.635 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454977+0000", 2024-05-15T13:24:22.635 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8040000000000002e-06 2024-05-15T13:24:22.635 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.635 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:22.635 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.635 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.635 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.636 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:125797 7.7 7.a8e9b387 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-05-15T13:24:22.636 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:14:50.718728+0000", 2024-05-15T13:24:22.636 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 571.87532230199997, 2024-05-15T13:24:22.636 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.736156488, 2024-05-15T13:24:22.636 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:22.636 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:24:22.636 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:22.636 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:24:22.636 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:24:22.636 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 125797 2024-05-15T13:24:22.636 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.636 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:22.636 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.636 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:22.636 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.718728+0000", 2024-05-15T13:24:22.637 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.637 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.637 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.637 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:22.637 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.718728+0000", 2024-05-15T13:24:22.637 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.637 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.637 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.637 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:22.637 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.718729+0000", 2024-05-15T13:24:22.637 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.95e-07 2024-05-15T13:24:22.637 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.637 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.637 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:22.637 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.718733+0000", 2024-05-15T13:24:22.638 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0749999999999999e-06 2024-05-15T13:24:22.638 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.638 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.638 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:22.638 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.718742+0000", 2024-05-15T13:24:22.638 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.9490000000000002e-06 2024-05-15T13:24:22.638 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.638 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.638 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:22.638 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.718748+0000", 2024-05-15T13:24:22.638 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5670000000000002e-06 2024-05-15T13:24:22.638 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.638 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.638 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.639 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.718789+0000", 2024-05-15T13:24:22.639 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0921999999999999e-05 2024-05-15T13:24:22.639 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.639 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.639 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:24:22.639 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:50.718795+0000", 2024-05-15T13:24:22.639 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4679999999999997e-06 2024-05-15T13:24:22.639 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.639 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.639 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.639 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454876+0000", 2024-05-15T13:24:22.639 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.736080717 2024-05-15T13:24:22.639 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.639 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.639 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:22.640 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:14:51.454885+0000", 2024-05-15T13:24:22.640 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7949999999999996e-06 2024-05-15T13:24:22.640 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.640 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:22.640 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.640 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.640 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.640 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:203794 7.d 7:b8521088:::benchmark_data_smithi177_134484_object94729:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e717)", 2024-05-15T13:24:22.640 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:17:47.880721+0000", 2024-05-15T13:24:22.640 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 394.71332891399999, 2024-05-15T13:24:22.640 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.037491414, 2024-05-15T13:24:22.640 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:22.640 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:24:22.640 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:22.640 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:24:22.641 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:24:22.641 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 203794 2024-05-15T13:24:22.641 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.641 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:22.641 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.641 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:22.641 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.880721+0000", 2024-05-15T13:24:22.641 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.641 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.641 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.641 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:22.641 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.880721+0000", 2024-05-15T13:24:22.641 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.641 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.642 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.642 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:22.642 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.880723+0000", 2024-05-15T13:24:22.642 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1489999999999999e-06 2024-05-15T13:24:22.642 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.642 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.642 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:22.642 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.880735+0000", 2024-05-15T13:24:22.642 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2146e-05 2024-05-15T13:24:22.642 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.642 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.642 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:22.642 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.880745+0000", 2024-05-15T13:24:22.642 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0139999999999999e-05 2024-05-15T13:24:22.643 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.643 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.643 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:22.643 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.880751+0000", 2024-05-15T13:24:22.643 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1840000000000004e-06 2024-05-15T13:24:22.643 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.643 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.643 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.643 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.884974+0000", 2024-05-15T13:24:22.643 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0042230940000000002 2024-05-15T13:24:22.643 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.643 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.643 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op must wait for map", 2024-05-15T13:24:22.643 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.884979+0000", 2024-05-15T13:24:22.643 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8069999999999999e-06 2024-05-15T13:24:22.644 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.644 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.644 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.644 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.886333+0000", 2024-05-15T13:24:22.644 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00135432 2024-05-15T13:24:22.644 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.644 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.644 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:24:22.644 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.886342+0000", 2024-05-15T13:24:22.644 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7900000000000005e-06 2024-05-15T13:24:22.644 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.644 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.644 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.644 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.918161+0000", 2024-05-15T13:24:22.645 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.031818468 2024-05-15T13:24:22.645 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.645 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.645 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:22.645 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.918213+0000", 2024-05-15T13:24:22.645 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2315999999999999e-05 2024-05-15T13:24:22.645 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.645 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:22.645 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.645 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.645 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.645 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:203795 7.d 7:ba59890f:::benchmark_data_smithi177_134484_object94730:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e717)", 2024-05-15T13:24:22.645 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:17:47.880762+0000", 2024-05-15T13:24:22.645 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 394.71328867800003, 2024-05-15T13:24:22.645 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0378411489999999, 2024-05-15T13:24:22.646 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:22.646 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:24:22.646 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:22.646 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:24:22.646 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:24:22.646 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 203795 2024-05-15T13:24:22.646 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.646 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:22.646 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.646 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:22.646 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.880762+0000", 2024-05-15T13:24:22.646 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.646 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.646 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.646 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:22.647 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.880762+0000", 2024-05-15T13:24:22.647 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.647 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.647 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.647 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:22.647 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.880762+0000", 2024-05-15T13:24:22.647 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.0899999999999997e-07 2024-05-15T13:24:22.647 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.647 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.647 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:22.647 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.880764+0000", 2024-05-15T13:24:22.647 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.8920000000000001e-06 2024-05-15T13:24:22.647 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.647 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.647 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:22.648 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.880769+0000", 2024-05-15T13:24:22.648 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9690000000000002e-06 2024-05-15T13:24:22.648 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.648 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.648 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:22.648 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.880772+0000", 2024-05-15T13:24:22.648 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.3589999999999999e-06 2024-05-15T13:24:22.648 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.648 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.648 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.648 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.885051+0000", 2024-05-15T13:24:22.648 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0042785569999999997 2024-05-15T13:24:22.648 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.648 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.649 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting_for_map not empty", 2024-05-15T13:24:22.649 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.885055+0000", 2024-05-15T13:24:22.649 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7670000000000001e-06 2024-05-15T13:24:22.649 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.649 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.649 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.649 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.886367+0000", 2024-05-15T13:24:22.649 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0013126030000000001 2024-05-15T13:24:22.649 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.649 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.649 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:24:22.649 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.886375+0000", 2024-05-15T13:24:22.649 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7370000000000006e-06 2024-05-15T13:24:22.649 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.649 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.650 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.650 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.918568+0000", 2024-05-15T13:24:22.650 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0321926770000001 2024-05-15T13:24:22.650 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.650 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.650 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:22.650 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.918603+0000", 2024-05-15T13:24:22.650 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5278999999999999e-05 2024-05-15T13:24:22.650 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.650 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:22.650 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.650 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.650 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.650 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:203834 7.d 7:b4387020:::benchmark_data_smithi177_134484_object94769:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e717)", 2024-05-15T13:24:22.651 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:17:47.961413+0000", 2024-05-15T13:24:22.651 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 394.632637784, 2024-05-15T13:24:22.651 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.976490144, 2024-05-15T13:24:22.651 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:22.651 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:22.651 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:22.651 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:24:22.651 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:24:22.651 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 203834 2024-05-15T13:24:22.651 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.651 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:22.651 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.651 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:22.651 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.961413+0000", 2024-05-15T13:24:22.651 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.652 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.652 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.652 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:22.652 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.961413+0000", 2024-05-15T13:24:22.652 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.652 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.652 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.652 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:22.652 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.961414+0000", 2024-05-15T13:24:22.652 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.6200000000000006e-07 2024-05-15T13:24:22.652 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.652 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.652 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:22.652 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.961418+0000", 2024-05-15T13:24:22.652 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2050000000000004e-06 2024-05-15T13:24:22.652 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.653 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.653 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:22.653 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.961427+0000", 2024-05-15T13:24:22.653 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1609999999999992e-06 2024-05-15T13:24:22.653 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.653 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.653 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:22.653 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.961434+0000", 2024-05-15T13:24:22.653 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3379999999999999e-06 2024-05-15T13:24:22.653 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.653 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.653 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.653 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.961493+0000", 2024-05-15T13:24:22.653 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8452000000000002e-05 2024-05-15T13:24:22.653 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.654 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.654 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:24:22.654 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.961504+0000", 2024-05-15T13:24:22.654 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0967e-05 2024-05-15T13:24:22.654 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.654 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.654 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.654 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.918637+0000", 2024-05-15T13:24:22.654 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.95713334900000002 2024-05-15T13:24:22.654 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.654 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.654 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:22.654 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.918737+0000", 2024-05-15T13:24:22.654 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00010035300000000001 2024-05-15T13:24:22.655 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.655 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.655 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:24:22.655 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.918814+0000", 2024-05-15T13:24:22.655 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.6732999999999999e-05 2024-05-15T13:24:22.655 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.655 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.655 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:22.655 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.928052+0000", 2024-05-15T13:24:22.655 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0092382420000000007 2024-05-15T13:24:22.655 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.655 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.655 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:22.655 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.937744+0000", 2024-05-15T13:24:22.655 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0096913030000000001 2024-05-15T13:24:22.655 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.656 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.656 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:22.656 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.937827+0000", 2024-05-15T13:24:22.656 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.3150000000000002e-05 2024-05-15T13:24:22.656 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.656 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.656 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:22.656 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.937869+0000", 2024-05-15T13:24:22.656 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2321999999999999e-05 2024-05-15T13:24:22.656 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.656 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.656 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:22.656 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.937903+0000", 2024-05-15T13:24:22.656 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.3606999999999999e-05 2024-05-15T13:24:22.656 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.656 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:22.657 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.657 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.657 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.657 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:203841 7.d 7:b681c48f:::benchmark_data_smithi177_134484_object94776:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e717)", 2024-05-15T13:24:22.657 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:17:47.974383+0000", 2024-05-15T13:24:22.657 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 394.61966701099999, 2024-05-15T13:24:22.657 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.96827790499999999, 2024-05-15T13:24:22.657 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:22.657 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:22.657 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:22.657 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:24:22.657 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:24:22.657 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 203841 2024-05-15T13:24:22.657 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.657 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:22.657 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.658 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:22.658 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.974383+0000", 2024-05-15T13:24:22.658 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.658 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.658 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.658 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:22.658 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.974383+0000", 2024-05-15T13:24:22.658 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.658 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.658 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.658 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:22.658 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.974384+0000", 2024-05-15T13:24:22.658 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0299999999999998e-07 2024-05-15T13:24:22.659 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.659 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.659 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:22.659 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.974388+0000", 2024-05-15T13:24:22.659 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5999999999999998e-06 2024-05-15T13:24:22.659 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.659 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.659 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:22.659 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.974398+0000", 2024-05-15T13:24:22.659 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0456e-05 2024-05-15T13:24:22.659 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.659 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.659 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:22.659 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.974405+0000", 2024-05-15T13:24:22.659 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3490000000000003e-06 2024-05-15T13:24:22.659 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.660 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.660 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.660 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.974475+0000", 2024-05-15T13:24:22.660 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9918000000000004e-05 2024-05-15T13:24:22.660 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.660 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.660 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:24:22.660 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:47.974483+0000", 2024-05-15T13:24:22.660 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0900000000000005e-06 2024-05-15T13:24:22.660 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.660 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.660 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.660 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.921920+0000", 2024-05-15T13:24:22.660 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.94743623499999996 2024-05-15T13:24:22.660 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.660 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.661 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:22.661 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.922002+0000", 2024-05-15T13:24:22.661 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2782999999999997e-05 2024-05-15T13:24:22.661 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.661 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.661 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:24:22.661 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.922078+0000", 2024-05-15T13:24:22.661 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.5318e-05 2024-05-15T13:24:22.661 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.661 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.661 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:22.661 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.932666+0000", 2024-05-15T13:24:22.661 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.010588389 2024-05-15T13:24:22.661 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.661 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.661 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:22.662 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.942546+0000", 2024-05-15T13:24:22.662 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0098796449999999994 2024-05-15T13:24:22.662 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.662 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.662 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:22.662 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.942592+0000", 2024-05-15T13:24:22.662 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6078999999999997e-05 2024-05-15T13:24:22.662 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.662 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.662 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:22.662 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.942630+0000", 2024-05-15T13:24:22.662 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.8538999999999998e-05 2024-05-15T13:24:22.662 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.662 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.663 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:22.663 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.942661+0000", 2024-05-15T13:24:22.663 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.0800999999999999e-05 2024-05-15T13:24:22.663 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.663 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:22.663 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.663 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.663 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.663 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:203878 7.d 7:bee932d0:::benchmark_data_smithi177_134484_object94813:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e717)", 2024-05-15T13:24:22.663 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:17:48.046845+0000", 2024-05-15T13:24:22.663 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 394.54720497, 2024-05-15T13:24:22.663 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.90893268100000002, 2024-05-15T13:24:22.663 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:22.663 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:22.663 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:22.663 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:24:22.663 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:24:22.664 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 203878 2024-05-15T13:24:22.664 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.664 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:22.664 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.664 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:22.664 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.046845+0000", 2024-05-15T13:24:22.664 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.664 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.664 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.664 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:22.664 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.046845+0000", 2024-05-15T13:24:22.664 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.664 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.664 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.664 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:22.664 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.046846+0000", 2024-05-15T13:24:22.665 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2999999999999999e-07 2024-05-15T13:24:22.665 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.665 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.665 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:22.665 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.046851+0000", 2024-05-15T13:24:22.665 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6789999999999996e-06 2024-05-15T13:24:22.665 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.665 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.665 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:22.665 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.046862+0000", 2024-05-15T13:24:22.665 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1022000000000001e-05 2024-05-15T13:24:22.665 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.665 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.665 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:22.665 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.046868+0000", 2024-05-15T13:24:22.666 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3400000000000003e-06 2024-05-15T13:24:22.666 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.666 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.666 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.666 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.046916+0000", 2024-05-15T13:24:22.666 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8242999999999999e-05 2024-05-15T13:24:22.666 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.666 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.666 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:24:22.666 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.046924+0000", 2024-05-15T13:24:22.666 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.5909999999999998e-06 2024-05-15T13:24:22.666 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.666 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.666 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.666 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.924015+0000", 2024-05-15T13:24:22.667 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.87709068800000001 2024-05-15T13:24:22.667 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.667 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.667 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:22.667 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.924086+0000", 2024-05-15T13:24:22.667 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0927999999999996e-05 2024-05-15T13:24:22.667 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.667 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.667 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:24:22.667 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.924164+0000", 2024-05-15T13:24:22.667 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8641999999999997e-05 2024-05-15T13:24:22.667 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.667 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.667 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:22.667 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.932691+0000", 2024-05-15T13:24:22.667 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0085268449999999999 2024-05-15T13:24:22.668 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.668 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.668 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:22.668 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.946758+0000", 2024-05-15T13:24:22.668 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.014066665000000001 2024-05-15T13:24:22.668 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.668 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.668 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:22.668 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.955703+0000", 2024-05-15T13:24:22.668 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0089455569999999998 2024-05-15T13:24:22.668 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.668 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.668 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:22.668 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.955748+0000", 2024-05-15T13:24:22.668 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4212999999999998e-05 2024-05-15T13:24:22.668 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.669 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.669 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:22.669 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.955778+0000", 2024-05-15T13:24:22.669 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.0437999999999999e-05 2024-05-15T13:24:22.669 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.669 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:22.669 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.669 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.669 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.669 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:203920 7.d 7:b9fb0590:::benchmark_data_smithi177_134484_object94855:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e717)", 2024-05-15T13:24:22.669 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:17:48.120993+0000", 2024-05-15T13:24:22.669 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 394.47305750700002, 2024-05-15T13:24:22.669 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.84822232900000005, 2024-05-15T13:24:22.669 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:22.669 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:22.669 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:22.670 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:24:22.670 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:24:22.670 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 203920 2024-05-15T13:24:22.670 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.670 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:22.670 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.670 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:22.670 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.120993+0000", 2024-05-15T13:24:22.670 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.670 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.670 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.670 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:22.670 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.120993+0000", 2024-05-15T13:24:22.670 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.670 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.671 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.671 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:22.671 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.120995+0000", 2024-05-15T13:24:22.671 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1859999999999999e-06 2024-05-15T13:24:22.671 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.671 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.671 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:22.671 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.121007+0000", 2024-05-15T13:24:22.671 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1503e-05 2024-05-15T13:24:22.671 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.671 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.671 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:22.671 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.121036+0000", 2024-05-15T13:24:22.671 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.9886000000000002e-05 2024-05-15T13:24:22.671 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.671 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.672 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:22.672 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.121053+0000", 2024-05-15T13:24:22.672 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6673000000000001e-05 2024-05-15T13:24:22.672 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.672 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.672 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.672 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.121166+0000", 2024-05-15T13:24:22.672 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00011276400000000001 2024-05-15T13:24:22.672 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.672 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.672 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:24:22.672 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.121186+0000", 2024-05-15T13:24:22.672 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0146999999999999e-05 2024-05-15T13:24:22.672 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.672 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.673 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.673 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.925974+0000", 2024-05-15T13:24:22.673 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.80478809500000004 2024-05-15T13:24:22.673 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.673 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.673 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:22.673 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.926075+0000", 2024-05-15T13:24:22.673 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000101102 2024-05-15T13:24:22.673 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.673 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.673 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:24:22.673 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.926159+0000", 2024-05-15T13:24:22.673 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.3763000000000004e-05 2024-05-15T13:24:22.673 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.673 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.673 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:22.673 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.937676+0000", 2024-05-15T13:24:22.674 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.011517490999999999 2024-05-15T13:24:22.674 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.674 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.674 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:22.674 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.951007+0000", 2024-05-15T13:24:22.674 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.013330933999999999 2024-05-15T13:24:22.674 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.674 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.674 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:22.674 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.969101+0000", 2024-05-15T13:24:22.674 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.018093227999999999 2024-05-15T13:24:22.674 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.674 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.674 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:22.674 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.969163+0000", 2024-05-15T13:24:22.675 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.2281999999999998e-05 2024-05-15T13:24:22.675 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.675 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.675 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:22.675 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.969215+0000", 2024-05-15T13:24:22.675 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2274999999999997e-05 2024-05-15T13:24:22.675 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.675 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:22.675 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.675 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.675 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.675 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:203940 7.d 7:bd5a2a1b:::benchmark_data_smithi177_134484_object94875:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e717)", 2024-05-15T13:24:22.675 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:17:48.156580+0000", 2024-05-15T13:24:22.675 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 394.437470287, 2024-05-15T13:24:22.675 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.81272291399999996, 2024-05-15T13:24:22.675 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:22.676 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:22.676 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:22.676 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:24:22.676 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:24:22.676 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 203940 2024-05-15T13:24:22.676 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.676 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:22.676 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.676 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:22.676 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.156580+0000", 2024-05-15T13:24:22.676 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.676 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.676 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.676 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:22.676 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.156580+0000", 2024-05-15T13:24:22.676 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.676 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.677 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.677 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:22.677 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.156581+0000", 2024-05-15T13:24:22.677 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2359999999999999e-06 2024-05-15T13:24:22.677 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.677 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.677 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:22.677 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.156586+0000", 2024-05-15T13:24:22.677 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9690000000000002e-06 2024-05-15T13:24:22.677 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.677 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.677 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:22.677 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.156598+0000", 2024-05-15T13:24:22.677 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1518000000000001e-05 2024-05-15T13:24:22.677 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.677 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.678 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:22.678 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.156604+0000", 2024-05-15T13:24:22.678 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9619999999999998e-06 2024-05-15T13:24:22.678 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.678 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.678 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.678 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.156653+0000", 2024-05-15T13:24:22.678 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8798000000000003e-05 2024-05-15T13:24:22.678 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.678 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.678 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:24:22.678 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.156659+0000", 2024-05-15T13:24:22.678 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.5470000000000002e-06 2024-05-15T13:24:22.679 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.679 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.679 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.679 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.928092+0000", 2024-05-15T13:24:22.679 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.77143289900000001 2024-05-15T13:24:22.679 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.679 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.679 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:22.679 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.928159+0000", 2024-05-15T13:24:22.679 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6659e-05 2024-05-15T13:24:22.679 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.679 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.679 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:24:22.679 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.928223+0000", 2024-05-15T13:24:22.679 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4227999999999996e-05 2024-05-15T13:24:22.680 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.680 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.680 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:22.680 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.937680+0000", 2024-05-15T13:24:22.680 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0094574329999999995 2024-05-15T13:24:22.680 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.680 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.680 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:22.680 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.954311+0000", 2024-05-15T13:24:22.680 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.016631168000000002 2024-05-15T13:24:22.680 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.680 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.680 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:22.680 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.969244+0000", 2024-05-15T13:24:22.681 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.01493296 2024-05-15T13:24:22.681 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.681 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.681 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:22.681 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.969277+0000", 2024-05-15T13:24:22.681 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.2459999999999998e-05 2024-05-15T13:24:22.681 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.681 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.681 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:22.681 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.969303+0000", 2024-05-15T13:24:22.681 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.6077000000000001e-05 2024-05-15T13:24:22.681 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.681 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:22.681 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.681 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.682 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.682 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:203941 7.d 7:b6c0c7f4:::benchmark_data_smithi177_134484_object94876:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e717)", 2024-05-15T13:24:22.682 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:17:48.157973+0000", 2024-05-15T13:24:22.682 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 394.43607782499998, 2024-05-15T13:24:22.682 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.811467563, 2024-05-15T13:24:22.682 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:22.682 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:22.682 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:22.682 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:24:22.682 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:24:22.682 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 203941 2024-05-15T13:24:22.682 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.682 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:22.682 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.682 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:22.682 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.157973+0000", 2024-05-15T13:24:22.683 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.683 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.683 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.683 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:22.683 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.157973+0000", 2024-05-15T13:24:22.683 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.683 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.683 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.683 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:22.683 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.157973+0000", 2024-05-15T13:24:22.683 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3300000000000002e-07 2024-05-15T13:24:22.683 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.683 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.683 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:22.683 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.157976+0000", 2024-05-15T13:24:22.684 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.2459999999999999e-06 2024-05-15T13:24:22.684 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.684 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.684 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:22.684 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.157984+0000", 2024-05-15T13:24:22.684 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8019999999999992e-06 2024-05-15T13:24:22.684 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.684 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.684 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:22.684 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.157995+0000", 2024-05-15T13:24:22.684 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1228e-05 2024-05-15T13:24:22.684 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.684 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.684 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.684 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.158049+0000", 2024-05-15T13:24:22.685 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3291999999999997e-05 2024-05-15T13:24:22.685 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.685 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.685 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:24:22.685 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.158059+0000", 2024-05-15T13:24:22.685 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0066e-05 2024-05-15T13:24:22.685 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.685 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.685 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.685 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.930497+0000", 2024-05-15T13:24:22.685 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.77243796300000001 2024-05-15T13:24:22.685 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.685 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.685 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:22.685 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.930588+0000", 2024-05-15T13:24:22.685 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1509999999999996e-05 2024-05-15T13:24:22.686 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.686 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.686 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:24:22.686 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.930671+0000", 2024-05-15T13:24:22.686 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.3034999999999994e-05 2024-05-15T13:24:22.686 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.686 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.686 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:22.686 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.937778+0000", 2024-05-15T13:24:22.686 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0071063369999999999 2024-05-15T13:24:22.686 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.686 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.686 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:22.686 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.960301+0000", 2024-05-15T13:24:22.687 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.022523461000000002 2024-05-15T13:24:22.687 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.687 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.687 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:22.687 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.969350+0000", 2024-05-15T13:24:22.687 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0090489999999999998 2024-05-15T13:24:22.687 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.687 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.687 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:22.687 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.969400+0000", 2024-05-15T13:24:22.687 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0074000000000002e-05 2024-05-15T13:24:22.687 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.687 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.687 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:22.687 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.969440+0000", 2024-05-15T13:24:22.687 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0015999999999998e-05 2024-05-15T13:24:22.688 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.688 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:22.688 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.688 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.688 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.688 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:203947 7.d 7:bc254b26:::benchmark_data_smithi177_134484_object94882:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e717)", 2024-05-15T13:24:22.688 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:17:48.167122+0000", 2024-05-15T13:24:22.688 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 394.42692799600002, 2024-05-15T13:24:22.688 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.80240224699999996, 2024-05-15T13:24:22.688 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:22.688 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:22.688 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:22.688 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:24:22.688 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:24:22.688 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 203947 2024-05-15T13:24:22.688 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.689 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:22.689 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.689 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:22.689 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.167122+0000", 2024-05-15T13:24:22.689 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.689 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.689 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.689 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:22.689 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.167122+0000", 2024-05-15T13:24:22.689 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.689 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.689 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.689 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:22.689 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.167123+0000", 2024-05-15T13:24:22.689 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9400000000000004e-07 2024-05-15T13:24:22.689 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.690 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.690 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:22.690 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.167127+0000", 2024-05-15T13:24:22.690 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2390000000000004e-06 2024-05-15T13:24:22.690 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.690 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.690 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:22.690 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.167137+0000", 2024-05-15T13:24:22.690 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.5000000000000005e-06 2024-05-15T13:24:22.690 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.690 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.690 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:22.690 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.167142+0000", 2024-05-15T13:24:22.690 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2120000000000002e-06 2024-05-15T13:24:22.690 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.691 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.691 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.691 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.167186+0000", 2024-05-15T13:24:22.691 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3544999999999997e-05 2024-05-15T13:24:22.691 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.691 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.691 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:24:22.691 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.167193+0000", 2024-05-15T13:24:22.691 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2409999999999998e-06 2024-05-15T13:24:22.691 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.691 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.691 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.691 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.932738+0000", 2024-05-15T13:24:22.691 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.76554499399999998 2024-05-15T13:24:22.691 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.691 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.692 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:22.692 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.932812+0000", 2024-05-15T13:24:22.692 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4169000000000001e-05 2024-05-15T13:24:22.692 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.692 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.692 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:24:22.692 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.932881+0000", 2024-05-15T13:24:22.692 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8465000000000003e-05 2024-05-15T13:24:22.692 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.692 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.692 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:22.692 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.937782+0000", 2024-05-15T13:24:22.692 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0049019420000000003 2024-05-15T13:24:22.692 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.692 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.693 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:22.693 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.966931+0000", 2024-05-15T13:24:22.693 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.029149040000000001 2024-05-15T13:24:22.693 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.693 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.693 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:22.693 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.969477+0000", 2024-05-15T13:24:22.693 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0025453759999999998 2024-05-15T13:24:22.693 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.693 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.693 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:22.693 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.969504+0000", 2024-05-15T13:24:22.693 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.6961000000000002e-05 2024-05-15T13:24:22.693 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.693 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.693 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:22.694 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.969525+0000", 2024-05-15T13:24:22.694 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0769000000000001e-05 2024-05-15T13:24:22.694 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.694 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:22.694 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.694 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.694 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.694 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:203952 7.d 7:b9adb96a:::benchmark_data_smithi177_134484_object94887:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e717)", 2024-05-15T13:24:22.694 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:17:48.174147+0000", 2024-05-15T13:24:22.694 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 394.41990376699999, 2024-05-15T13:24:22.694 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.80206750999999998, 2024-05-15T13:24:22.694 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:22.694 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:22.694 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:22.694 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:24:22.695 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:24:22.695 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 203952 2024-05-15T13:24:22.695 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.695 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:22.695 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.695 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:22.695 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.174147+0000", 2024-05-15T13:24:22.695 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.695 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.695 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.695 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:22.695 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.174147+0000", 2024-05-15T13:24:22.695 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.695 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.695 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.695 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:22.696 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.174148+0000", 2024-05-15T13:24:22.696 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.048e-06 2024-05-15T13:24:22.696 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.696 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.696 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:22.696 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.174152+0000", 2024-05-15T13:24:22.696 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.1749999999999998e-06 2024-05-15T13:24:22.696 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.696 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.696 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:22.696 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.174163+0000", 2024-05-15T13:24:22.696 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1192000000000001e-05 2024-05-15T13:24:22.696 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.696 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.696 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:22.696 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.174169+0000", 2024-05-15T13:24:22.697 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.2400000000000004e-06 2024-05-15T13:24:22.697 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.697 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.697 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.697 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.174221+0000", 2024-05-15T13:24:22.697 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2070999999999997e-05 2024-05-15T13:24:22.697 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.697 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.697 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:24:22.697 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.174230+0000", 2024-05-15T13:24:22.697 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.9220000000000003e-06 2024-05-15T13:24:22.697 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.697 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.697 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.697 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.935168+0000", 2024-05-15T13:24:22.697 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.76093758300000003 2024-05-15T13:24:22.698 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.698 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.698 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:22.698 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.935259+0000", 2024-05-15T13:24:22.698 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0834999999999993e-05 2024-05-15T13:24:22.698 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.698 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.698 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:24:22.698 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.935344+0000", 2024-05-15T13:24:22.698 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.5432000000000005e-05 2024-05-15T13:24:22.698 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.698 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.698 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:22.698 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.942669+0000", 2024-05-15T13:24:22.698 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0073251380000000001 2024-05-15T13:24:22.698 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.699 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.699 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:22.699 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.972980+0000", 2024-05-15T13:24:22.699 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.030310836000000001 2024-05-15T13:24:22.699 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.699 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.699 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:22.699 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.976092+0000", 2024-05-15T13:24:22.699 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0031118700000000001 2024-05-15T13:24:22.699 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.699 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.699 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:22.699 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.976169+0000", 2024-05-15T13:24:22.699 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.6855999999999997e-05 2024-05-15T13:24:22.699 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.699 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.699 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:22.700 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.976214+0000", 2024-05-15T13:24:22.700 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.5312000000000003e-05 2024-05-15T13:24:22.700 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.700 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:22.700 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.700 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.700 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.700 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26044.0:203963 7.d 7:b39e1d92:::benchmark_data_smithi177_134484_object94898:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e717)", 2024-05-15T13:24:22.700 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:17:48.198585+0000", 2024-05-15T13:24:22.700 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 394.39546571099999, 2024-05-15T13:24:22.700 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.79741139400000005, 2024-05-15T13:24:22.700 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:22.700 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:22.700 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:22.700 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26044", 2024-05-15T13:24:22.700 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:24:22.701 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 203963 2024-05-15T13:24:22.701 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.701 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:22.701 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.701 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:22.701 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.198585+0000", 2024-05-15T13:24:22.701 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.701 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.701 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.701 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:22.701 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.198585+0000", 2024-05-15T13:24:22.701 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:22.701 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.701 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.701 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:22.701 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.198586+0000", 2024-05-15T13:24:22.702 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1149999999999999e-06 2024-05-15T13:24:22.702 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.702 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.702 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:22.702 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.198590+0000", 2024-05-15T13:24:22.702 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.5349999999999998e-06 2024-05-15T13:24:22.702 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.702 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.702 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:22.702 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.198602+0000", 2024-05-15T13:24:22.702 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2033e-05 2024-05-15T13:24:22.702 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.702 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.702 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:22.702 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.198609+0000", 2024-05-15T13:24:22.703 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4160000000000001e-06 2024-05-15T13:24:22.703 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.703 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.703 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.703 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.198669+0000", 2024-05-15T13:24:22.703 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.0322000000000003e-05 2024-05-15T13:24:22.703 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.703 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.703 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:24:22.703 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.198680+0000", 2024-05-15T13:24:22.703 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0847e-05 2024-05-15T13:24:22.703 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.703 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.703 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:22.703 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.937932+0000", 2024-05-15T13:24:22.703 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.73925251599999997 2024-05-15T13:24:22.703 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.704 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.704 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:22.704 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.938006+0000", 2024-05-15T13:24:22.704 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3862000000000005e-05 2024-05-15T13:24:22.704 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.704 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.704 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 0,1", 2024-05-15T13:24:22.704 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.938073+0000", 2024-05-15T13:24:22.704 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6754000000000005e-05 2024-05-15T13:24:22.704 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.704 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.704 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:22.704 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.946687+0000", 2024-05-15T13:24:22.704 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0086143499999999998 2024-05-15T13:24:22.704 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.704 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.705 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:22.705 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.982034+0000", 2024-05-15T13:24:22.705 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.035346133000000002 2024-05-15T13:24:22.705 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.705 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.705 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:22.705 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.995879+0000", 2024-05-15T13:24:22.705 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.013845975 2024-05-15T13:24:22.705 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.705 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.705 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:22.705 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.995944+0000", 2024-05-15T13:24:22.705 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4039000000000001e-05 2024-05-15T13:24:22.705 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:22.705 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:22.705 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:22.706 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:17:48.995996+0000", 2024-05-15T13:24:22.706 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2497000000000002e-05 2024-05-15T13:24:22.706 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.706 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:22.706 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.706 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:22.706 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:22.706 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:24:22.817 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:24:22.972 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:24:23.009 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:24:23.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:22 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1614550663' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:24:23.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:22 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:24:23.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:22 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1614550663' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:24:23.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:22 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:24:23.048 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:24:23.077 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:24:23.120 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:24:23.161 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:24:23.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:22 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1614550663' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:24:23.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:22 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:24:23.198 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:24:23.240 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:24:23.273 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:24:23.310 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:24:23.316 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 71 16 103872 103856 91.3947 75.25 0.0143896 0.0109371 2024-05-15T13:24:23.348 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:24:23.392 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:24:23.400 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:24:23.400 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:24:23.400 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:24:23.400 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:24:23.401 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:24:23.401 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:24:23.401 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:24:23.401 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:24:23.401 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:24:23.401 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:24:23.401 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:24:23.401 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:24:23.437 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:24:23.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:23.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:24:23.545 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:24:23.579 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:23.611 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:24:23.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:23.614 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:24:23.621 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-05-15T13:24:23.715 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:24:23.759 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:23 smithi027 ceph-mon[39795]: pgmap v2184: 47 pgs: 47 active+clean; 6.1 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-05-15T13:24:23.760 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:23 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:24:23.760 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:23 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:24:23.760 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:23 smithi027 ceph-mon[39795]: osdmap e844: 12 total, 12 up, 10 in 2024-05-15T13:24:23.760 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:23 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/879367891' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:24:23.770 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:23.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:23 smithi177 ceph-mon[44303]: pgmap v2184: 47 pgs: 47 active+clean; 6.1 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-05-15T13:24:23.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:23 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:24:23.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:23 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:24:23.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:23 smithi177 ceph-mon[44303]: osdmap e844: 12 total, 12 up, 10 in 2024-05-15T13:24:23.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:23 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/879367891' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:24:24.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:23 smithi120 ceph-mon[43961]: pgmap v2184: 47 pgs: 47 active+clean; 6.1 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-05-15T13:24:24.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:23 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:24:24.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:23 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:24:24.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:23 smithi120 ceph-mon[43961]: osdmap e844: 12 total, 12 up, 10 in 2024-05-15T13:24:24.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:23 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/879367891' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:24:24.052 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:24:24.231 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:24:24.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:24.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:24:24.316 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 72 16 105013 104997 91.1157 71.3125 0.0188032 0.0109711 2024-05-15T13:24:24.337 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:24:24.387 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:24.425 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:24:24.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:24.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:24:24.532 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:24:24.584 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:24.657 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:24:24.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:24.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:24:24.762 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:24:24.849 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:25.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:24 smithi027 ceph-mon[39795]: 1.1e scrub starts 2024-05-15T13:24:25.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:24 smithi027 ceph-mon[39795]: 1.1e scrub ok 2024-05-15T13:24:25.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:24 smithi027 ceph-mon[39795]: 1.1d scrub starts 2024-05-15T13:24:25.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:24 smithi027 ceph-mon[39795]: 1.7 scrub starts 2024-05-15T13:24:25.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:24 smithi027 ceph-mon[39795]: 1.1d scrub ok 2024-05-15T13:24:25.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:24 smithi027 ceph-mon[39795]: 1.a scrub starts 2024-05-15T13:24:25.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:24 smithi027 ceph-mon[39795]: 1.a scrub ok 2024-05-15T13:24:25.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:24 smithi027 ceph-mon[39795]: 1.7 scrub ok 2024-05-15T13:24:25.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:24 smithi027 ceph-mon[39795]: 1.19 scrub starts 2024-05-15T13:24:25.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:24 smithi027 ceph-mon[39795]: 1.19 scrub ok 2024-05-15T13:24:25.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:24 smithi027 ceph-mon[39795]: 1.18 scrub starts 2024-05-15T13:24:25.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:24 smithi027 ceph-mon[39795]: 1.18 scrub ok 2024-05-15T13:24:25.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:24 smithi027 ceph-mon[39795]: osdmap e845: 12 total, 12 up, 10 in 2024-05-15T13:24:25.044 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:24 smithi120 ceph-mon[43961]: 1.1e scrub starts 2024-05-15T13:24:25.045 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:24 smithi120 ceph-mon[43961]: 1.1e scrub ok 2024-05-15T13:24:25.045 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:24 smithi120 ceph-mon[43961]: 1.1d scrub starts 2024-05-15T13:24:25.045 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:24 smithi120 ceph-mon[43961]: 1.7 scrub starts 2024-05-15T13:24:25.045 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:24 smithi120 ceph-mon[43961]: 1.1d scrub ok 2024-05-15T13:24:25.045 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:24 smithi120 ceph-mon[43961]: 1.a scrub starts 2024-05-15T13:24:25.045 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:24 smithi120 ceph-mon[43961]: 1.a scrub ok 2024-05-15T13:24:25.045 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:24 smithi120 ceph-mon[43961]: 1.7 scrub ok 2024-05-15T13:24:25.045 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:24 smithi120 ceph-mon[43961]: 1.19 scrub starts 2024-05-15T13:24:25.045 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:24 smithi120 ceph-mon[43961]: 1.19 scrub ok 2024-05-15T13:24:25.045 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:24 smithi120 ceph-mon[43961]: 1.18 scrub starts 2024-05-15T13:24:25.045 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:24 smithi120 ceph-mon[43961]: 1.18 scrub ok 2024-05-15T13:24:25.045 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:24 smithi120 ceph-mon[43961]: osdmap e845: 12 total, 12 up, 10 in 2024-05-15T13:24:25.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:24 smithi177 ceph-mon[44303]: 1.1e scrub starts 2024-05-15T13:24:25.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:24 smithi177 ceph-mon[44303]: 1.1e scrub ok 2024-05-15T13:24:25.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:24 smithi177 ceph-mon[44303]: 1.1d scrub starts 2024-05-15T13:24:25.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:24 smithi177 ceph-mon[44303]: 1.7 scrub starts 2024-05-15T13:24:25.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:24 smithi177 ceph-mon[44303]: 1.1d scrub ok 2024-05-15T13:24:25.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:24 smithi177 ceph-mon[44303]: 1.a scrub starts 2024-05-15T13:24:25.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:24 smithi177 ceph-mon[44303]: 1.a scrub ok 2024-05-15T13:24:25.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:24 smithi177 ceph-mon[44303]: 1.7 scrub ok 2024-05-15T13:24:25.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:24 smithi177 ceph-mon[44303]: 1.19 scrub starts 2024-05-15T13:24:25.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:24 smithi177 ceph-mon[44303]: 1.19 scrub ok 2024-05-15T13:24:25.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:24 smithi177 ceph-mon[44303]: 1.18 scrub starts 2024-05-15T13:24:25.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:24 smithi177 ceph-mon[44303]: 1.18 scrub ok 2024-05-15T13:24:25.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:24 smithi177 ceph-mon[44303]: osdmap e845: 12 total, 12 up, 10 in 2024-05-15T13:24:25.316 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 73 16 106163 106147 90.8521 71.875 0.0068578 0.0110026 2024-05-15T13:24:25.421 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:24:25.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:25.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:24:25.526 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:24:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:25.670 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:24:25.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:25.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:24:25.778 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:24:25.814 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:25.838 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:24:25.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:25.842 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:24:25.943 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:24:25.971 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:25 smithi120 ceph-mon[43961]: 1.16 scrub starts 2024-05-15T13:24:25.971 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:25 smithi120 ceph-mon[43961]: 1.16 scrub ok 2024-05-15T13:24:25.971 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:25 smithi120 ceph-mon[43961]: 1.9 scrub starts 2024-05-15T13:24:25.971 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:25 smithi120 ceph-mon[43961]: 1.1b scrub starts 2024-05-15T13:24:25.971 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:25 smithi120 ceph-mon[43961]: 1.9 scrub ok 2024-05-15T13:24:25.971 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:25 smithi120 ceph-mon[43961]: pgmap v2187: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.3 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-05-15T13:24:25.971 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:25 smithi120 ceph-mon[43961]: 1.5 scrub starts 2024-05-15T13:24:25.971 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:25 smithi120 ceph-mon[43961]: 1.1b scrub ok 2024-05-15T13:24:25.971 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:25 smithi120 ceph-mon[43961]: 1.8 scrub starts 2024-05-15T13:24:25.972 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:25 smithi120 ceph-mon[43961]: 1.8 scrub ok 2024-05-15T13:24:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:26.034 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:25 smithi027 ceph-mon[39795]: 1.16 scrub starts 2024-05-15T13:24:26.034 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:25 smithi027 ceph-mon[39795]: 1.16 scrub ok 2024-05-15T13:24:26.034 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:25 smithi027 ceph-mon[39795]: 1.9 scrub starts 2024-05-15T13:24:26.034 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:25 smithi027 ceph-mon[39795]: 1.1b scrub starts 2024-05-15T13:24:26.034 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:25 smithi027 ceph-mon[39795]: 1.9 scrub ok 2024-05-15T13:24:26.034 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:25 smithi027 ceph-mon[39795]: pgmap v2187: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.3 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-05-15T13:24:26.034 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:25 smithi027 ceph-mon[39795]: 1.5 scrub starts 2024-05-15T13:24:26.034 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:25 smithi027 ceph-mon[39795]: 1.1b scrub ok 2024-05-15T13:24:26.034 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:25 smithi027 ceph-mon[39795]: 1.8 scrub starts 2024-05-15T13:24:26.034 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:25 smithi027 ceph-mon[39795]: 1.8 scrub ok 2024-05-15T13:24:26.059 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:24:26.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:26.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:24:26.163 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:24:26.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:25 smithi177 ceph-mon[44303]: 1.16 scrub starts 2024-05-15T13:24:26.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:25 smithi177 ceph-mon[44303]: 1.16 scrub ok 2024-05-15T13:24:26.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:25 smithi177 ceph-mon[44303]: 1.9 scrub starts 2024-05-15T13:24:26.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:25 smithi177 ceph-mon[44303]: 1.1b scrub starts 2024-05-15T13:24:26.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:25 smithi177 ceph-mon[44303]: 1.9 scrub ok 2024-05-15T13:24:26.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:25 smithi177 ceph-mon[44303]: pgmap v2187: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.3 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-05-15T13:24:26.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:25 smithi177 ceph-mon[44303]: 1.5 scrub starts 2024-05-15T13:24:26.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:25 smithi177 ceph-mon[44303]: 1.1b scrub ok 2024-05-15T13:24:26.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:25 smithi177 ceph-mon[44303]: 1.8 scrub starts 2024-05-15T13:24:26.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:25 smithi177 ceph-mon[44303]: 1.8 scrub ok 2024-05-15T13:24:26.216 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:26.316 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 74 16 107119 107103 90.4319 59.75 0.0151966 0.0110536 2024-05-15T13:24:26.478 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:24:26.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:26.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:24:26.585 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:24:26.641 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:26.691 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:24:26.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:26.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:24:26.797 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:24:26.829 INFO:teuthology.orchestra.run.smithi027.stderr:set pool 9 pgp_num to 16 2024-05-15T13:24:26.879 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:26.885 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:24:26.914 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:24:26.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:26 smithi027 ceph-mon[39795]: 1.5 scrub ok 2024-05-15T13:24:26.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:26 smithi027 ceph-mon[39795]: 9.4 scrub starts 2024-05-15T13:24:26.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:26 smithi027 ceph-mon[39795]: 1.e scrub starts 2024-05-15T13:24:26.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:26 smithi027 ceph-mon[39795]: 1.e scrub ok 2024-05-15T13:24:26.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:26 smithi027 ceph-mon[39795]: 1.17 scrub starts 2024-05-15T13:24:26.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:26 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:24:26.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:26 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:24:26.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:26 smithi027 ceph-mon[39795]: 9.1 scrub starts 2024-05-15T13:24:26.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:26 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/675234294' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num", "val": "16"}]: dispatch 2024-05-15T13:24:26.917 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:26 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num", "val": "16"}]: dispatch 2024-05-15T13:24:26.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:26.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:24:27.019 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:24:27.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:26 smithi120 ceph-mon[43961]: 1.5 scrub ok 2024-05-15T13:24:27.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:26 smithi120 ceph-mon[43961]: 9.4 scrub starts 2024-05-15T13:24:27.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:26 smithi120 ceph-mon[43961]: 1.e scrub starts 2024-05-15T13:24:27.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:26 smithi120 ceph-mon[43961]: 1.e scrub ok 2024-05-15T13:24:27.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:26 smithi120 ceph-mon[43961]: 1.17 scrub starts 2024-05-15T13:24:27.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:26 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:24:27.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:26 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:24:27.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:26 smithi120 ceph-mon[43961]: 9.1 scrub starts 2024-05-15T13:24:27.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:26 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/675234294' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num", "val": "16"}]: dispatch 2024-05-15T13:24:27.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:26 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num", "val": "16"}]: dispatch 2024-05-15T13:24:27.054 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:26 smithi177 ceph-mon[44303]: 1.5 scrub ok 2024-05-15T13:24:27.054 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:26 smithi177 ceph-mon[44303]: 9.4 scrub starts 2024-05-15T13:24:27.054 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:26 smithi177 ceph-mon[44303]: 1.e scrub starts 2024-05-15T13:24:27.054 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:26 smithi177 ceph-mon[44303]: 1.e scrub ok 2024-05-15T13:24:27.054 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:26 smithi177 ceph-mon[44303]: 1.17 scrub starts 2024-05-15T13:24:27.054 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:26 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:24:27.054 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:26 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:24:27.054 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:26 smithi177 ceph-mon[44303]: 9.1 scrub starts 2024-05-15T13:24:27.055 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:26 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/675234294' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num", "val": "16"}]: dispatch 2024-05-15T13:24:27.055 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:26 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num", "val": "16"}]: dispatch 2024-05-15T13:24:27.056 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:27.080 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:24:27.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:27.086 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:24:27.133 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:24:27.133 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:24:27.187 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:24:27.261 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:27.304 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:24:27.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:27.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:24:27.316 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 75 16 108064 108048 90.0136 59.0625 0.0149102 0.0111047 2024-05-15T13:24:27.412 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:24:27.454 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:27.482 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:24:27.489 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:24:27.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:27.492 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:24:27.593 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:24:27.636 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:28.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:27 smithi120 ceph-mon[43961]: 1.17 scrub ok 2024-05-15T13:24:28.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:27 smithi120 ceph-mon[43961]: 1.b scrub starts 2024-05-15T13:24:28.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:27 smithi120 ceph-mon[43961]: 1.b scrub ok 2024-05-15T13:24:28.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:27 smithi120 ceph-mon[43961]: 1.d scrub starts 2024-05-15T13:24:28.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:27 smithi120 ceph-mon[43961]: 1.d scrub ok 2024-05-15T13:24:28.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:27 smithi120 ceph-mon[43961]: pgmap v2188: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.4 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-05-15T13:24:28.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:27 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num", "val": "16"}]': finished 2024-05-15T13:24:28.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:27 smithi120 ceph-mon[43961]: osdmap e846: 12 total, 12 up, 10 in 2024-05-15T13:24:28.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:27 smithi120 ceph-mon[43961]: 1.14 scrub starts 2024-05-15T13:24:28.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:27 smithi120 ceph-mon[43961]: 1.14 scrub ok 2024-05-15T13:24:28.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:27 smithi177 ceph-mon[44303]: 1.17 scrub ok 2024-05-15T13:24:28.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:27 smithi177 ceph-mon[44303]: 1.b scrub starts 2024-05-15T13:24:28.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:27 smithi177 ceph-mon[44303]: 1.b scrub ok 2024-05-15T13:24:28.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:27 smithi177 ceph-mon[44303]: 1.d scrub starts 2024-05-15T13:24:28.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:27 smithi177 ceph-mon[44303]: 1.d scrub ok 2024-05-15T13:24:28.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:27 smithi177 ceph-mon[44303]: pgmap v2188: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.4 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-05-15T13:24:28.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:27 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num", "val": "16"}]': finished 2024-05-15T13:24:28.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:27 smithi177 ceph-mon[44303]: osdmap e846: 12 total, 12 up, 10 in 2024-05-15T13:24:28.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:27 smithi177 ceph-mon[44303]: 1.14 scrub starts 2024-05-15T13:24:28.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:27 smithi177 ceph-mon[44303]: 1.14 scrub ok 2024-05-15T13:24:28.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:27 smithi027 ceph-mon[39795]: 1.17 scrub ok 2024-05-15T13:24:28.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:27 smithi027 ceph-mon[39795]: 1.b scrub starts 2024-05-15T13:24:28.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:27 smithi027 ceph-mon[39795]: 1.b scrub ok 2024-05-15T13:24:28.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:27 smithi027 ceph-mon[39795]: 1.d scrub starts 2024-05-15T13:24:28.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:27 smithi027 ceph-mon[39795]: 1.d scrub ok 2024-05-15T13:24:28.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:27 smithi027 ceph-mon[39795]: pgmap v2188: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.4 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-05-15T13:24:28.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:27 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num", "val": "16"}]': finished 2024-05-15T13:24:28.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:27 smithi027 ceph-mon[39795]: osdmap e846: 12 total, 12 up, 10 in 2024-05-15T13:24:28.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:27 smithi027 ceph-mon[39795]: 1.14 scrub starts 2024-05-15T13:24:28.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:27 smithi027 ceph-mon[39795]: 1.14 scrub ok 2024-05-15T13:24:28.293 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:24:28.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:28.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:24:28.316 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 76 16 108916 108900 89.5299 53.25 0.0153385 0.0111643 2024-05-15T13:24:28.397 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:24:28.430 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:29.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:28 smithi177 ceph-mon[44303]: 9.0 scrub starts 2024-05-15T13:24:29.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:28 smithi177 ceph-mon[44303]: 9.5 scrub starts 2024-05-15T13:24:29.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:28 smithi177 ceph-mon[44303]: 1.1c scrub starts 2024-05-15T13:24:29.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:28 smithi177 ceph-mon[44303]: 1.1c scrub ok 2024-05-15T13:24:29.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:28 smithi177 ceph-mon[44303]: 9.4 scrub ok 2024-05-15T13:24:29.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:28 smithi177 ceph-mon[44303]: 9.3 scrub starts 2024-05-15T13:24:29.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:28 smithi177 ceph-mon[44303]: 9.1 scrub ok 2024-05-15T13:24:29.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:28 smithi177 ceph-mon[44303]: 9.9 scrub starts 2024-05-15T13:24:29.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:28 smithi177 ceph-mon[44303]: 1.c scrub starts 2024-05-15T13:24:29.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:28 smithi177 ceph-mon[44303]: 1.6 scrub starts 2024-05-15T13:24:29.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:28 smithi177 ceph-mon[44303]: 1.c scrub ok 2024-05-15T13:24:29.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:28 smithi177 ceph-mon[44303]: 1.6 scrub ok 2024-05-15T13:24:29.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:28 smithi177 ceph-mon[44303]: 9.f scrub starts 2024-05-15T13:24:29.224 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:24:29.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:28 smithi027 ceph-mon[39795]: 9.0 scrub starts 2024-05-15T13:24:29.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:28 smithi027 ceph-mon[39795]: 9.5 scrub starts 2024-05-15T13:24:29.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:28 smithi027 ceph-mon[39795]: 1.1c scrub starts 2024-05-15T13:24:29.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:28 smithi027 ceph-mon[39795]: 1.1c scrub ok 2024-05-15T13:24:29.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:28 smithi027 ceph-mon[39795]: 9.4 scrub ok 2024-05-15T13:24:29.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:28 smithi027 ceph-mon[39795]: 9.3 scrub starts 2024-05-15T13:24:29.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:28 smithi027 ceph-mon[39795]: 9.1 scrub ok 2024-05-15T13:24:29.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:28 smithi027 ceph-mon[39795]: 9.9 scrub starts 2024-05-15T13:24:29.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:28 smithi027 ceph-mon[39795]: 1.c scrub starts 2024-05-15T13:24:29.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:28 smithi027 ceph-mon[39795]: 1.6 scrub starts 2024-05-15T13:24:29.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:28 smithi027 ceph-mon[39795]: 1.c scrub ok 2024-05-15T13:24:29.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:28 smithi027 ceph-mon[39795]: 1.6 scrub ok 2024-05-15T13:24:29.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:28 smithi027 ceph-mon[39795]: 9.f scrub starts 2024-05-15T13:24:29.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:28 smithi120 ceph-mon[43961]: 9.0 scrub starts 2024-05-15T13:24:29.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:28 smithi120 ceph-mon[43961]: 9.5 scrub starts 2024-05-15T13:24:29.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:28 smithi120 ceph-mon[43961]: 1.1c scrub starts 2024-05-15T13:24:29.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:28 smithi120 ceph-mon[43961]: 1.1c scrub ok 2024-05-15T13:24:29.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:28 smithi120 ceph-mon[43961]: 9.4 scrub ok 2024-05-15T13:24:29.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:28 smithi120 ceph-mon[43961]: 9.3 scrub starts 2024-05-15T13:24:29.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:28 smithi120 ceph-mon[43961]: 9.1 scrub ok 2024-05-15T13:24:29.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:28 smithi120 ceph-mon[43961]: 9.9 scrub starts 2024-05-15T13:24:29.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:28 smithi120 ceph-mon[43961]: 1.c scrub starts 2024-05-15T13:24:29.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:28 smithi120 ceph-mon[43961]: 1.6 scrub starts 2024-05-15T13:24:29.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:28 smithi120 ceph-mon[43961]: 1.c scrub ok 2024-05-15T13:24:29.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:28 smithi120 ceph-mon[43961]: 1.6 scrub ok 2024-05-15T13:24:29.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:28 smithi120 ceph-mon[43961]: 9.f scrub starts 2024-05-15T13:24:29.316 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 77 16 109551 109535 88.8828 39.6875 0.0292772 0.011247 2024-05-15T13:24:29.501 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:24:29.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:29.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:24:29.527 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-05-15T13:24:29.608 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:24:29.682 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:29.731 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:24:29.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:29.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:24:29.834 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:24:29.899 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:29.942 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:24:29.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:29.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:24:30.002 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:24:30.049 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:24:30.068 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:29 smithi027 ceph-mon[39795]: 1.3 scrub starts 2024-05-15T13:24:30.068 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:29 smithi027 ceph-mon[39795]: 1.3 scrub ok 2024-05-15T13:24:30.068 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:29 smithi027 ceph-mon[39795]: 9.5 scrub ok 2024-05-15T13:24:30.068 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:29 smithi027 ceph-mon[39795]: 9.8 scrub starts 2024-05-15T13:24:30.068 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:29 smithi027 ceph-mon[39795]: pgmap v2190: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 6.5 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-05-15T13:24:30.068 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:29 smithi027 ceph-mon[39795]: 9.0 scrub ok 2024-05-15T13:24:30.068 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:29 smithi027 ceph-mon[39795]: 9.a scrub starts 2024-05-15T13:24:30.068 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:29 smithi027 ceph-mon[39795]: osdmap e847: 12 total, 12 up, 10 in 2024-05-15T13:24:30.097 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:29 smithi177 ceph-mon[44303]: 1.3 scrub starts 2024-05-15T13:24:30.097 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:29 smithi177 ceph-mon[44303]: 1.3 scrub ok 2024-05-15T13:24:30.097 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:29 smithi177 ceph-mon[44303]: 9.5 scrub ok 2024-05-15T13:24:30.097 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:29 smithi177 ceph-mon[44303]: 9.8 scrub starts 2024-05-15T13:24:30.097 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:29 smithi177 ceph-mon[44303]: pgmap v2190: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 6.5 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-05-15T13:24:30.097 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:29 smithi177 ceph-mon[44303]: 9.0 scrub ok 2024-05-15T13:24:30.097 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:29 smithi177 ceph-mon[44303]: 9.a scrub starts 2024-05-15T13:24:30.097 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:29 smithi177 ceph-mon[44303]: osdmap e847: 12 total, 12 up, 10 in 2024-05-15T13:24:30.103 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:30.144 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:24:30.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:30.148 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:24:30.249 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:24:30.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:29 smithi120 ceph-mon[43961]: 1.3 scrub starts 2024-05-15T13:24:30.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:29 smithi120 ceph-mon[43961]: 1.3 scrub ok 2024-05-15T13:24:30.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:29 smithi120 ceph-mon[43961]: 9.5 scrub ok 2024-05-15T13:24:30.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:29 smithi120 ceph-mon[43961]: 9.8 scrub starts 2024-05-15T13:24:30.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:29 smithi120 ceph-mon[43961]: pgmap v2190: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 6.5 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-05-15T13:24:30.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:29 smithi120 ceph-mon[43961]: 9.0 scrub ok 2024-05-15T13:24:30.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:29 smithi120 ceph-mon[43961]: 9.a scrub starts 2024-05-15T13:24:30.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:29 smithi120 ceph-mon[43961]: osdmap e847: 12 total, 12 up, 10 in 2024-05-15T13:24:30.295 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:30.316 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 78 16 110512 110496 88.5133 60.0625 0.0144255 0.0112933 2024-05-15T13:24:30.320 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:24:30.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:30.328 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:24:30.429 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:24:30.464 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:30.492 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:24:30.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:30.498 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:24:30.599 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:24:30.654 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:30.681 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:24:30.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:30.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:24:30.789 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:24:30.828 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:30.854 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:24:30.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:30.858 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:24:30.959 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:24:31.004 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:30 smithi120 ceph-mon[43961]: 9.9 scrub ok 2024-05-15T13:24:31.004 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:30 smithi120 ceph-mon[43961]: 1.15 scrub starts 2024-05-15T13:24:31.004 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:30 smithi120 ceph-mon[43961]: 9.3 scrub ok 2024-05-15T13:24:31.004 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:30 smithi120 ceph-mon[43961]: 1.15 scrub ok 2024-05-15T13:24:31.005 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:30 smithi120 ceph-mon[43961]: 1.1a scrub starts 2024-05-15T13:24:31.005 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:30 smithi120 ceph-mon[43961]: 1.1a scrub ok 2024-05-15T13:24:31.005 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:30 smithi120 ceph-mon[43961]: 9.f scrub ok 2024-05-15T13:24:31.008 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:31.123 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:24:31.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:31.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:24:31.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:30 smithi177 ceph-mon[44303]: 9.9 scrub ok 2024-05-15T13:24:31.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:30 smithi177 ceph-mon[44303]: 1.15 scrub starts 2024-05-15T13:24:31.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:30 smithi177 ceph-mon[44303]: 9.3 scrub ok 2024-05-15T13:24:31.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:30 smithi177 ceph-mon[44303]: 1.15 scrub ok 2024-05-15T13:24:31.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:30 smithi177 ceph-mon[44303]: 1.1a scrub starts 2024-05-15T13:24:31.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:30 smithi177 ceph-mon[44303]: 1.1a scrub ok 2024-05-15T13:24:31.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:30 smithi177 ceph-mon[44303]: 9.f scrub ok 2024-05-15T13:24:31.231 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:24:31.250 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:30 smithi027 ceph-mon[39795]: 9.9 scrub ok 2024-05-15T13:24:31.250 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:30 smithi027 ceph-mon[39795]: 1.15 scrub starts 2024-05-15T13:24:31.250 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:30 smithi027 ceph-mon[39795]: 9.3 scrub ok 2024-05-15T13:24:31.250 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:30 smithi027 ceph-mon[39795]: 1.15 scrub ok 2024-05-15T13:24:31.250 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:30 smithi027 ceph-mon[39795]: 1.1a scrub starts 2024-05-15T13:24:31.250 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:30 smithi027 ceph-mon[39795]: 1.1a scrub ok 2024-05-15T13:24:31.250 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:30 smithi027 ceph-mon[39795]: 9.f scrub ok 2024-05-15T13:24:31.277 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:24:31.280 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:31.317 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 79 16 111548 111532 88.2125 64.75 0.0107146 0.0113317 2024-05-15T13:24:32.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:31 smithi120 ceph-mon[43961]: 9.8 scrub ok 2024-05-15T13:24:32.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:31 smithi120 ceph-mon[43961]: 9.e scrub starts 2024-05-15T13:24:32.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:31 smithi120 ceph-mon[43961]: pgmap v2192: 47 pgs: 4 active+clean+scrubbing, 43 active+clean; 6.6 GiB data, 28 GiB used, 866 GiB / 894 GiB avail; 52 MiB/s wr, 824 op/s 2024-05-15T13:24:32.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:31 smithi120 ceph-mon[43961]: 9.7 scrub starts 2024-05-15T13:24:32.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:31 smithi120 ceph-mon[43961]: 9.a scrub ok 2024-05-15T13:24:32.143 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:24:32.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:32.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:24:32.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:31 smithi177 ceph-mon[44303]: 9.8 scrub ok 2024-05-15T13:24:32.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:31 smithi177 ceph-mon[44303]: 9.e scrub starts 2024-05-15T13:24:32.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:31 smithi177 ceph-mon[44303]: pgmap v2192: 47 pgs: 4 active+clean+scrubbing, 43 active+clean; 6.6 GiB data, 28 GiB used, 866 GiB / 894 GiB avail; 52 MiB/s wr, 824 op/s 2024-05-15T13:24:32.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:31 smithi177 ceph-mon[44303]: 9.7 scrub starts 2024-05-15T13:24:32.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:31 smithi177 ceph-mon[44303]: 9.a scrub ok 2024-05-15T13:24:32.255 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:24:32.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:31 smithi027 ceph-mon[39795]: 9.8 scrub ok 2024-05-15T13:24:32.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:31 smithi027 ceph-mon[39795]: 9.e scrub starts 2024-05-15T13:24:32.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:31 smithi027 ceph-mon[39795]: pgmap v2192: 47 pgs: 4 active+clean+scrubbing, 43 active+clean; 6.6 GiB data, 28 GiB used, 866 GiB / 894 GiB avail; 52 MiB/s wr, 824 op/s 2024-05-15T13:24:32.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:31 smithi027 ceph-mon[39795]: 9.7 scrub starts 2024-05-15T13:24:32.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:31 smithi027 ceph-mon[39795]: 9.a scrub ok 2024-05-15T13:24:32.305 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:32.316 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:2024-05-15T13:24:32.317824+0000 min lat: 0.00110277 max lat: 13.6712 avg lat: 0.0113724 2024-05-15T13:24:32.317 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T13:24:32.317 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 80 16 112558 112542 87.8989 63.125 0.0259988 0.0113724 2024-05-15T13:24:32.374 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:24:32.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:32.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:24:32.480 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:24:32.599 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:32.846 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:32 smithi027 ceph-mon[39795]: 9.c scrub starts 2024-05-15T13:24:32.846 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:32 smithi027 ceph-mon[39795]: 9.e scrub ok 2024-05-15T13:24:33.077 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:24:33.077 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:24:33.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:32 smithi177 ceph-mon[44303]: 9.c scrub starts 2024-05-15T13:24:33.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:32 smithi177 ceph-mon[44303]: 9.e scrub ok 2024-05-15T13:24:33.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:32 smithi120 ceph-mon[43961]: 9.c scrub starts 2024-05-15T13:24:33.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:32 smithi120 ceph-mon[43961]: 9.e scrub ok 2024-05-15T13:24:33.317 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 81 16 113580 113564 87.6023 63.875 0.0146267 0.0114108 2024-05-15T13:24:33.881 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-05-15T13:24:34.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:33 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:24:34.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:33 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:24:34.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:33 smithi177 ceph-mon[44303]: 9.7 scrub ok 2024-05-15T13:24:34.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:33 smithi177 ceph-mon[44303]: 9.d scrub starts 2024-05-15T13:24:34.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:33 smithi177 ceph-mon[44303]: pgmap v2193: 47 pgs: 6 active+clean+scrubbing, 41 active+clean; 6.7 GiB data, 28 GiB used, 866 GiB / 894 GiB avail; 54 MiB/s wr, 858 op/s 2024-05-15T13:24:34.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:33 smithi177 ceph-mon[44303]: osdmap e848: 12 total, 12 up, 10 in 2024-05-15T13:24:34.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:33 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2849988476' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:24:34.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:33 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:24:34.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:33 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:24:34.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:33 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:24:34.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:33 smithi027 ceph-mon[39795]: 9.7 scrub ok 2024-05-15T13:24:34.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:33 smithi027 ceph-mon[39795]: 9.d scrub starts 2024-05-15T13:24:34.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:33 smithi027 ceph-mon[39795]: pgmap v2193: 47 pgs: 6 active+clean+scrubbing, 41 active+clean; 6.7 GiB data, 28 GiB used, 866 GiB / 894 GiB avail; 54 MiB/s wr, 858 op/s 2024-05-15T13:24:34.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:33 smithi027 ceph-mon[39795]: osdmap e848: 12 total, 12 up, 10 in 2024-05-15T13:24:34.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:33 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2849988476' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:24:34.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:33 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:24:34.301 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:33 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:24:34.301 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:33 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:24:34.301 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:33 smithi120 ceph-mon[43961]: 9.7 scrub ok 2024-05-15T13:24:34.301 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:33 smithi120 ceph-mon[43961]: 9.d scrub starts 2024-05-15T13:24:34.301 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:33 smithi120 ceph-mon[43961]: pgmap v2193: 47 pgs: 6 active+clean+scrubbing, 41 active+clean; 6.7 GiB data, 28 GiB used, 866 GiB / 894 GiB avail; 54 MiB/s wr, 858 op/s 2024-05-15T13:24:34.301 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:33 smithi120 ceph-mon[43961]: osdmap e848: 12 total, 12 up, 10 in 2024-05-15T13:24:34.301 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:33 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2849988476' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:24:34.301 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:33 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:24:34.316 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:24:34.317 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 82 16 114417 114401 87.1719 52.3125 0.0235097 0.0114671 2024-05-15T13:24:34.580 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:24:34.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:34.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:24:34.687 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:24:34.700 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:24:34.723 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:34.775 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:24:34.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:34.780 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:24:34.873 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 9, 10, 11, 7] out_osds: [4, 8] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:24:34.873 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:24:34.874 INFO:tasks.thrashosds.thrasher:inject_pause on osd.8 2024-05-15T13:24:34.874 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-15T13:24:34.874 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-15T13:24:34.874 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 config set filestore_inject_stall 3 2024-05-15T13:24:34.882 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:24:34.962 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:35.053 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:24:35.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:35.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:24:35.159 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:24:35.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:34 smithi177 ceph-mon[44303]: 9.c scrub ok 2024-05-15T13:24:35.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:34 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:24:35.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:34 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T13:24:35.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:34 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:24:35.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:34 smithi177 ceph-mon[44303]: osdmap e849: 12 total, 12 up, 10 in 2024-05-15T13:24:35.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:34 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:24:35.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:34 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:24:35.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:34 smithi177 ceph-mon[44303]: 9.b deep-scrub starts 2024-05-15T13:24:35.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:34 smithi027 ceph-mon[39795]: 9.c scrub ok 2024-05-15T13:24:35.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:34 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:24:35.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:34 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T13:24:35.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:34 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:24:35.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:34 smithi027 ceph-mon[39795]: osdmap e849: 12 total, 12 up, 10 in 2024-05-15T13:24:35.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:34 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:24:35.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:34 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:24:35.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:34 smithi027 ceph-mon[39795]: 9.b deep-scrub starts 2024-05-15T13:24:35.202 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:35.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:34 smithi120 ceph-mon[43961]: 9.c scrub ok 2024-05-15T13:24:35.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:34 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:24:35.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:34 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T13:24:35.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:34 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:24:35.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:34 smithi120 ceph-mon[43961]: osdmap e849: 12 total, 12 up, 10 in 2024-05-15T13:24:35.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:34 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:24:35.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:34 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:24:35.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:34 smithi120 ceph-mon[43961]: 9.b deep-scrub starts 2024-05-15T13:24:35.317 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 83 16 115328 115312 86.8077 56.9375 0.0129016 0.0115135 2024-05-15T13:24:35.365 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:24:35.501 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:24:35.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:35.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:24:35.607 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:24:35.633 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:35.735 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:24:35.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:35.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:24:35.844 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:24:35.901 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:24:35.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:35.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:24:36.040 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:24:36.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:35 smithi120 ceph-mon[43961]: 9.d scrub ok 2024-05-15T13:24:36.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:35 smithi120 ceph-mon[43961]: pgmap v2196: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 6.9 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 59 MiB/s wr, 947 op/s 2024-05-15T13:24:36.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:35 smithi120 ceph-mon[43961]: osdmap e850: 12 total, 12 up, 10 in 2024-05-15T13:24:36.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:35 smithi120 ceph-mon[43961]: 1.4 deep-scrub starts 2024-05-15T13:24:36.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:35 smithi120 ceph-mon[43961]: 1.4 deep-scrub ok 2024-05-15T13:24:36.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:35 smithi120 ceph-mon[43961]: 1.12 scrub starts 2024-05-15T13:24:36.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:35 smithi120 ceph-mon[43961]: 1.12 scrub ok 2024-05-15T13:24:36.073 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:36.133 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:24:36.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:36.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:24:36.164 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:24:36.196 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:24:36.228 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:35 smithi177 ceph-mon[44303]: 9.d scrub ok 2024-05-15T13:24:36.228 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:35 smithi177 ceph-mon[44303]: pgmap v2196: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 6.9 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 59 MiB/s wr, 947 op/s 2024-05-15T13:24:36.228 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:35 smithi177 ceph-mon[44303]: osdmap e850: 12 total, 12 up, 10 in 2024-05-15T13:24:36.228 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:35 smithi177 ceph-mon[44303]: 1.4 deep-scrub starts 2024-05-15T13:24:36.228 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:35 smithi177 ceph-mon[44303]: 1.4 deep-scrub ok 2024-05-15T13:24:36.228 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:35 smithi177 ceph-mon[44303]: 1.12 scrub starts 2024-05-15T13:24:36.228 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:35 smithi177 ceph-mon[44303]: 1.12 scrub ok 2024-05-15T13:24:36.237 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:24:36.241 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:24:36.276 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:24:36.279 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:36.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:35 smithi027 ceph-mon[39795]: 9.d scrub ok 2024-05-15T13:24:36.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:35 smithi027 ceph-mon[39795]: pgmap v2196: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 6.9 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 59 MiB/s wr, 947 op/s 2024-05-15T13:24:36.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:35 smithi027 ceph-mon[39795]: osdmap e850: 12 total, 12 up, 10 in 2024-05-15T13:24:36.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:35 smithi027 ceph-mon[39795]: 1.4 deep-scrub starts 2024-05-15T13:24:36.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:35 smithi027 ceph-mon[39795]: 1.4 deep-scrub ok 2024-05-15T13:24:36.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:35 smithi027 ceph-mon[39795]: 1.12 scrub starts 2024-05-15T13:24:36.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:35 smithi027 ceph-mon[39795]: 1.12 scrub ok 2024-05-15T13:24:36.310 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:24:36.322 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 84 16 116027 116011 86.2888 43.6875 0.00746818 0.0115843 2024-05-15T13:24:36.350 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:24:36.392 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:24:36.398 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:24:36.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:36.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:24:36.429 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:24:36.477 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:24:36.502 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:24:36.518 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:24:36.559 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:24:36.574 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:36.601 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:24:36.620 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:24:36.620 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:24:36.620 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:24:36.620 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:24:36.620 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:24:36.621 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:24:36.621 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:24:36.621 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:24:36.621 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:24:36.621 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:24:36.621 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:24:36.621 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:24:36.621 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:24:36.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:36.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:24:36.727 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:24:36.759 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:36.818 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:24:36.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:36.821 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:24:36.922 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:24:36.974 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:36 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:24:36.974 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:36 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:24:36.975 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:36 smithi027 ceph-mon[39795]: 1.2 scrub starts 2024-05-15T13:24:36.975 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:36 smithi027 ceph-mon[39795]: 1.2 scrub ok 2024-05-15T13:24:36.975 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:36 smithi027 ceph-mon[39795]: 9.2 deep-scrub starts 2024-05-15T13:24:36.977 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:37.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:36 smithi027 ceph-mon[39795]: osdmap e851: 12 total, 12 up, 10 in 2024-05-15T13:24:37.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:36 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1944605961' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:24:37.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:36 smithi027 ceph-mon[39795]: 9.6 scrub starts 2024-05-15T13:24:37.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:36 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:24:37.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:36 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:24:37.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:36 smithi120 ceph-mon[43961]: 1.2 scrub starts 2024-05-15T13:24:37.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:36 smithi120 ceph-mon[43961]: 1.2 scrub ok 2024-05-15T13:24:37.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:36 smithi120 ceph-mon[43961]: 9.2 deep-scrub starts 2024-05-15T13:24:37.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:36 smithi120 ceph-mon[43961]: osdmap e851: 12 total, 12 up, 10 in 2024-05-15T13:24:37.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:36 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1944605961' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:24:37.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:36 smithi120 ceph-mon[43961]: 9.6 scrub starts 2024-05-15T13:24:37.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:36 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:24:37.318 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:36 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:24:37.318 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:36 smithi177 ceph-mon[44303]: 1.2 scrub starts 2024-05-15T13:24:37.318 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:36 smithi177 ceph-mon[44303]: 1.2 scrub ok 2024-05-15T13:24:37.318 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:36 smithi177 ceph-mon[44303]: 9.2 deep-scrub starts 2024-05-15T13:24:37.318 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:36 smithi177 ceph-mon[44303]: osdmap e851: 12 total, 12 up, 10 in 2024-05-15T13:24:37.318 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:36 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1944605961' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:24:37.318 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:36 smithi177 ceph-mon[44303]: 9.6 scrub starts 2024-05-15T13:24:37.322 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 85 16 116746 116730 85.8023 44.9375 0.0106357 0.0116484 2024-05-15T13:24:37.338 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:24:37.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:37.343 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:24:37.443 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:24:37.496 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:37.535 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:24:37.542 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:24:37.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:37.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:24:37.648 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:24:37.715 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:24:37.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:37.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:24:37.859 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:24:37.896 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:37.956 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:24:37.993 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:37 smithi027 ceph-mon[39795]: pgmap v2199: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 7.0 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-05-15T13:24:38.007 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:24:38.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:38.011 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:24:38.112 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:24:38.150 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:38.171 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:24:38.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:38.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:24:38.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:37 smithi177 ceph-mon[44303]: pgmap v2199: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 7.0 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-05-15T13:24:38.216 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:24:38.216 INFO:teuthology.orchestra.run.smithi177.stdout: "success": "" 2024-05-15T13:24:38.217 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:24:38.245 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:24:38.245 INFO:teuthology.orchestra.run.smithi120.stdout: "size": 20, 2024-05-15T13:24:38.245 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 600, 2024-05-15T13:24:38.245 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [ 2024-05-15T13:24:38.246 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.246 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:1548 9.e 9:791dab7e:::benchmark_data_smithi177_159919_object1547:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-05-15T13:24:38.246 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:23:12.794847+0000", 2024-05-15T13:24:38.246 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 85.445509028000004, 2024-05-15T13:24:38.246 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.197658893, 2024-05-15T13:24:38.246 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:38.246 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:38.246 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:38.246 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:24:38.246 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:24:38.246 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 1548 2024-05-15T13:24:38.246 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.246 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:38.247 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.247 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:38.247 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.794847+0000", 2024-05-15T13:24:38.247 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.247 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.247 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.247 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:38.247 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.794847+0000", 2024-05-15T13:24:38.247 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.247 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.247 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.247 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:38.247 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.794848+0000", 2024-05-15T13:24:38.248 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3009999999999999e-06 2024-05-15T13:24:38.248 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.248 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.248 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:38.248 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.794922+0000", 2024-05-15T13:24:38.248 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3573999999999997e-05 2024-05-15T13:24:38.248 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.248 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.248 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:38.248 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.794934+0000", 2024-05-15T13:24:38.248 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2245e-05 2024-05-15T13:24:38.248 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.248 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.248 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:38.248 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.794942+0000", 2024-05-15T13:24:38.249 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8420000000000005e-06 2024-05-15T13:24:38.249 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.249 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.249 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:38.249 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.794995+0000", 2024-05-15T13:24:38.249 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2880000000000002e-05 2024-05-15T13:24:38.249 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.249 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.249 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:38.249 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.795130+0000", 2024-05-15T13:24:38.249 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013524899999999999 2024-05-15T13:24:38.250 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.250 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.250 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,10", 2024-05-15T13:24:38.250 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.795243+0000", 2024-05-15T13:24:38.250 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000112935 2024-05-15T13:24:38.250 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.250 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.250 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:38.250 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.795611+0000", 2024-05-15T13:24:38.250 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00036798100000000003 2024-05-15T13:24:38.250 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.250 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.251 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.251 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.797242+0000", 2024-05-15T13:24:38.251 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001631148 2024-05-15T13:24:38.251 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.251 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.251 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.251 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.992355+0000", 2024-05-15T13:24:38.251 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.195112805 2024-05-15T13:24:38.251 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.251 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.251 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:38.251 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.992409+0000", 2024-05-15T13:24:38.252 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3968000000000002e-05 2024-05-15T13:24:38.252 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.252 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.252 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:38.252 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.992506+0000", 2024-05-15T13:24:38.252 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.6965000000000004e-05 2024-05-15T13:24:38.252 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.252 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:38.252 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.252 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.252 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.252 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:1566 9.e 9:7fc71cf7:::benchmark_data_smithi177_159919_object1565:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-05-15T13:24:38.252 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:23:12.806399+0000", 2024-05-15T13:24:38.252 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 85.433956991000002, 2024-05-15T13:24:38.253 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.18664703199999999, 2024-05-15T13:24:38.253 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:38.253 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:38.253 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:38.253 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:24:38.253 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:24:38.253 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 1566 2024-05-15T13:24:38.253 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.253 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:38.253 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.253 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:38.253 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.806399+0000", 2024-05-15T13:24:38.253 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.253 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.253 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.254 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:38.254 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.806399+0000", 2024-05-15T13:24:38.254 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.254 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.254 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.254 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:38.254 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.806400+0000", 2024-05-15T13:24:38.254 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6899999999999997e-07 2024-05-15T13:24:38.254 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.254 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.254 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:38.254 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.806441+0000", 2024-05-15T13:24:38.255 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.1856e-05 2024-05-15T13:24:38.255 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.255 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.255 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:38.255 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.806453+0000", 2024-05-15T13:24:38.255 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1782e-05 2024-05-15T13:24:38.255 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.255 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.255 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:38.255 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.806461+0000", 2024-05-15T13:24:38.255 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3470000000000001e-06 2024-05-15T13:24:38.255 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.255 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.255 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:38.255 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.806518+0000", 2024-05-15T13:24:38.256 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7108000000000003e-05 2024-05-15T13:24:38.256 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.256 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.256 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:38.256 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.806638+0000", 2024-05-15T13:24:38.256 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000120617 2024-05-15T13:24:38.256 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.256 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.256 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,10", 2024-05-15T13:24:38.256 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.806792+0000", 2024-05-15T13:24:38.256 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015374799999999999 2024-05-15T13:24:38.256 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.256 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.257 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:38.257 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.808109+0000", 2024-05-15T13:24:38.257 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001316808 2024-05-15T13:24:38.257 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.257 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.257 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.257 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.809496+0000", 2024-05-15T13:24:38.257 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001386895 2024-05-15T13:24:38.257 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.257 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.257 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.257 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.992798+0000", 2024-05-15T13:24:38.257 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.18330248399999999 2024-05-15T13:24:38.257 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.257 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.257 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:38.258 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.992837+0000", 2024-05-15T13:24:38.258 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.8291000000000003e-05 2024-05-15T13:24:38.258 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.258 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.258 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:38.258 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.993046+0000", 2024-05-15T13:24:38.258 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00020952699999999999 2024-05-15T13:24:38.258 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.258 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:38.258 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.258 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.258 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.258 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:1568 9.e 9:739515c1:::benchmark_data_smithi177_159919_object1567:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-05-15T13:24:38.258 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:23:12.807094+0000", 2024-05-15T13:24:38.258 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 85.433262450000001, 2024-05-15T13:24:38.259 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.18614104000000001, 2024-05-15T13:24:38.259 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:38.259 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:38.259 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:38.259 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:24:38.259 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:24:38.259 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 1568 2024-05-15T13:24:38.259 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.259 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:38.259 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.259 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:38.259 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.807094+0000", 2024-05-15T13:24:38.259 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.259 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.260 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.260 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:38.260 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.807094+0000", 2024-05-15T13:24:38.260 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.260 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.260 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.260 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:38.260 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.807094+0000", 2024-05-15T13:24:38.260 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1900000000000002e-07 2024-05-15T13:24:38.260 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.260 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.260 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:38.260 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.807121+0000", 2024-05-15T13:24:38.260 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.7209999999999999e-05 2024-05-15T13:24:38.260 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.261 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.261 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:38.261 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.807130+0000", 2024-05-15T13:24:38.261 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.4919999999999993e-06 2024-05-15T13:24:38.261 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.261 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.261 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:38.261 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.807136+0000", 2024-05-15T13:24:38.261 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3099999999999997e-06 2024-05-15T13:24:38.261 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.261 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.261 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:38.261 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.807185+0000", 2024-05-15T13:24:38.261 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8591000000000003e-05 2024-05-15T13:24:38.261 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.262 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.262 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:38.262 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.807326+0000", 2024-05-15T13:24:38.262 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014138599999999999 2024-05-15T13:24:38.262 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.262 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.262 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,10", 2024-05-15T13:24:38.262 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.807472+0000", 2024-05-15T13:24:38.262 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000145974 2024-05-15T13:24:38.262 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.262 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.262 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:38.262 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.808114+0000", 2024-05-15T13:24:38.262 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00064211100000000001 2024-05-15T13:24:38.262 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.263 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.263 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.263 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.810824+0000", 2024-05-15T13:24:38.263 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0027097800000000002 2024-05-15T13:24:38.263 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.263 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.263 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.263 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.993092+0000", 2024-05-15T13:24:38.263 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.18226836599999999 2024-05-15T13:24:38.263 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.263 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.263 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:38.263 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.993151+0000", 2024-05-15T13:24:38.263 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8606000000000001e-05 2024-05-15T13:24:38.263 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.263 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.264 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:38.264 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.993235+0000", 2024-05-15T13:24:38.264 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.3595000000000002e-05 2024-05-15T13:24:38.264 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.264 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:38.264 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.264 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.264 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.264 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:1575 9.e 9:75edc339:::benchmark_data_smithi177_159919_object1574:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-05-15T13:24:38.264 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:23:12.807420+0000", 2024-05-15T13:24:38.264 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 85.432935822000005, 2024-05-15T13:24:38.264 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.18604438400000001, 2024-05-15T13:24:38.264 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:38.264 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:38.264 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:38.265 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:24:38.265 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:24:38.265 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 1575 2024-05-15T13:24:38.265 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.265 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:38.265 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.265 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:38.265 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.807420+0000", 2024-05-15T13:24:38.265 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.265 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.265 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.265 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:38.266 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.807420+0000", 2024-05-15T13:24:38.266 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.266 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.266 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.266 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:38.266 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.807421+0000", 2024-05-15T13:24:38.266 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.1999999999999999e-07 2024-05-15T13:24:38.266 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.266 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.266 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:38.266 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.807499+0000", 2024-05-15T13:24:38.266 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8484000000000003e-05 2024-05-15T13:24:38.266 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.266 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.267 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:38.267 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.807510+0000", 2024-05-15T13:24:38.267 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0298e-05 2024-05-15T13:24:38.267 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.267 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.267 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:38.267 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.807517+0000", 2024-05-15T13:24:38.267 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.934e-06 2024-05-15T13:24:38.267 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.267 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.267 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:38.267 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.807736+0000", 2024-05-15T13:24:38.267 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000218903 2024-05-15T13:24:38.267 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.267 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.267 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:38.268 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.807814+0000", 2024-05-15T13:24:38.268 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8038000000000001e-05 2024-05-15T13:24:38.268 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.268 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.268 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,10", 2024-05-15T13:24:38.268 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.807910+0000", 2024-05-15T13:24:38.268 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.6102999999999998e-05 2024-05-15T13:24:38.268 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.268 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.268 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:38.268 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.808256+0000", 2024-05-15T13:24:38.268 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00034599700000000001 2024-05-15T13:24:38.268 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.269 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.269 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.269 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.810857+0000", 2024-05-15T13:24:38.269 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0026011129999999999 2024-05-15T13:24:38.269 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.269 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.269 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.269 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.993346+0000", 2024-05-15T13:24:38.269 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.18248935299999999 2024-05-15T13:24:38.269 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.269 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.269 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:38.269 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.993394+0000", 2024-05-15T13:24:38.269 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8062999999999997e-05 2024-05-15T13:24:38.270 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.270 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.270 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:38.270 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.993465+0000", 2024-05-15T13:24:38.270 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0377999999999996e-05 2024-05-15T13:24:38.270 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.270 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:38.270 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.270 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.270 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.270 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:1548 9.e 9:791dab7e:::benchmark_data_smithi177_159919_object1547:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e813)", 2024-05-15T13:24:38.271 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:23:12.822271+0000", 2024-05-15T13:24:38.271 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 85.418085340000005, 2024-05-15T13:24:38.271 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.17015498200000001, 2024-05-15T13:24:38.271 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:38.271 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "waiting for ondisk", 2024-05-15T13:24:38.271 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:38.271 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:24:38.271 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:24:38.271 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 1548 2024-05-15T13:24:38.271 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.271 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:38.271 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.271 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:38.271 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822271+0000", 2024-05-15T13:24:38.272 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.272 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.272 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.272 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:38.272 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822271+0000", 2024-05-15T13:24:38.272 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.272 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.272 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.272 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:38.272 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822271+0000", 2024-05-15T13:24:38.272 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.9100000000000003e-07 2024-05-15T13:24:38.272 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.272 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.272 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:38.272 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822447+0000", 2024-05-15T13:24:38.273 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00017546099999999999 2024-05-15T13:24:38.273 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.273 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.273 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:38.273 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822459+0000", 2024-05-15T13:24:38.273 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2493e-05 2024-05-15T13:24:38.273 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.273 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.273 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:38.273 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822467+0000", 2024-05-15T13:24:38.273 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.5789999999999998e-06 2024-05-15T13:24:38.273 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.273 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.273 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:38.274 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822534+0000", 2024-05-15T13:24:38.274 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7570999999999994e-05 2024-05-15T13:24:38.274 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.274 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.274 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for ondisk", 2024-05-15T13:24:38.274 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822617+0000", 2024-05-15T13:24:38.274 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2348000000000005e-05 2024-05-15T13:24:38.274 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.274 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.274 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:38.274 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.992426+0000", 2024-05-15T13:24:38.275 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.16980883899999999 2024-05-15T13:24:38.275 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.275 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:38.275 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.275 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.275 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.275 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:1566 9.e 9:7fc71cf7:::benchmark_data_smithi177_159919_object1565:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e813)", 2024-05-15T13:24:38.275 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:23:12.822491+0000", 2024-05-15T13:24:38.275 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 85.417865066999994, 2024-05-15T13:24:38.275 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.17035731800000001, 2024-05-15T13:24:38.275 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:38.275 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "waiting for ondisk", 2024-05-15T13:24:38.275 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:38.275 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:24:38.276 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:24:38.276 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 1566 2024-05-15T13:24:38.276 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.276 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:38.276 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.276 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:38.276 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822491+0000", 2024-05-15T13:24:38.276 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.276 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.276 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.276 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:38.276 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822491+0000", 2024-05-15T13:24:38.276 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.276 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.277 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.277 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:38.277 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822492+0000", 2024-05-15T13:24:38.277 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.68e-07 2024-05-15T13:24:38.277 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.277 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.277 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:38.277 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822558+0000", 2024-05-15T13:24:38.277 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6297000000000006e-05 2024-05-15T13:24:38.277 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.277 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.277 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:38.277 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822566+0000", 2024-05-15T13:24:38.277 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.011e-06 2024-05-15T13:24:38.278 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.278 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.278 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:38.278 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822572+0000", 2024-05-15T13:24:38.278 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6719999999999998e-06 2024-05-15T13:24:38.278 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.278 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.278 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:38.278 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822660+0000", 2024-05-15T13:24:38.278 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7744999999999994e-05 2024-05-15T13:24:38.278 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.278 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.279 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for ondisk", 2024-05-15T13:24:38.279 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822784+0000", 2024-05-15T13:24:38.279 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000123452 2024-05-15T13:24:38.279 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.279 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.279 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:38.279 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.992848+0000", 2024-05-15T13:24:38.279 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.170064573 2024-05-15T13:24:38.279 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.279 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:38.279 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.279 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.279 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.279 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:1568 9.e 9:739515c1:::benchmark_data_smithi177_159919_object1567:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e813)", 2024-05-15T13:24:38.280 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:23:12.822588+0000", 2024-05-15T13:24:38.280 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 85.417767620000006, 2024-05-15T13:24:38.280 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.170583348, 2024-05-15T13:24:38.280 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:38.280 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "waiting for ondisk", 2024-05-15T13:24:38.280 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:38.280 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:24:38.280 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:24:38.280 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 1568 2024-05-15T13:24:38.280 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.280 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:38.280 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.281 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:38.281 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822588+0000", 2024-05-15T13:24:38.281 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.281 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.281 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.281 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:38.281 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822588+0000", 2024-05-15T13:24:38.281 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.281 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.281 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.281 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:38.281 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822589+0000", 2024-05-15T13:24:38.281 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.0399999999999996e-07 2024-05-15T13:24:38.281 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.282 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.282 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:38.282 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822637+0000", 2024-05-15T13:24:38.282 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8316000000000003e-05 2024-05-15T13:24:38.282 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.282 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.282 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:38.282 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822645+0000", 2024-05-15T13:24:38.282 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7330000000000003e-06 2024-05-15T13:24:38.282 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.282 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.282 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:38.282 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822651+0000", 2024-05-15T13:24:38.282 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3080000000000004e-06 2024-05-15T13:24:38.283 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.283 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.283 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:38.283 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822844+0000", 2024-05-15T13:24:38.283 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000192368 2024-05-15T13:24:38.283 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.283 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.283 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for ondisk", 2024-05-15T13:24:38.283 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822888+0000", 2024-05-15T13:24:38.283 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4443000000000001e-05 2024-05-15T13:24:38.283 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.283 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.283 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:38.284 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.993172+0000", 2024-05-15T13:24:38.284 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.17028357599999999 2024-05-15T13:24:38.284 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.284 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:38.284 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.284 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.284 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.284 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:1575 9.e 9:75edc339:::benchmark_data_smithi177_159919_object1574:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e813)", 2024-05-15T13:24:38.284 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:23:12.822678+0000", 2024-05-15T13:24:38.284 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 85.417678201000001, 2024-05-15T13:24:38.284 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.170730935, 2024-05-15T13:24:38.284 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:38.284 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "waiting for ondisk", 2024-05-15T13:24:38.285 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:38.285 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:24:38.285 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:24:38.285 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 1575 2024-05-15T13:24:38.285 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.285 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:38.285 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.285 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:38.285 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822678+0000", 2024-05-15T13:24:38.285 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.285 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.285 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.285 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:38.285 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822678+0000", 2024-05-15T13:24:38.285 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.285 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.286 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.286 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:38.286 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822678+0000", 2024-05-15T13:24:38.286 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.4199999999999996e-07 2024-05-15T13:24:38.286 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.286 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.286 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:38.286 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822706+0000", 2024-05-15T13:24:38.286 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.7960999999999999e-05 2024-05-15T13:24:38.286 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.286 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.286 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:38.286 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822738+0000", 2024-05-15T13:24:38.287 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.1198999999999997e-05 2024-05-15T13:24:38.287 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.287 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.287 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:38.287 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822745+0000", 2024-05-15T13:24:38.287 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2969999999999998e-06 2024-05-15T13:24:38.287 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.287 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.287 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:38.287 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822920+0000", 2024-05-15T13:24:38.287 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00017525400000000001 2024-05-15T13:24:38.287 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.287 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.287 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for ondisk", 2024-05-15T13:24:38.287 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.822979+0000", 2024-05-15T13:24:38.287 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8751e-05 2024-05-15T13:24:38.288 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.288 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.288 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:38.288 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.993409+0000", 2024-05-15T13:24:38.288 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.17042993100000001 2024-05-15T13:24:38.288 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.288 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:38.288 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.288 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.288 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.288 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:1659 9.e 9:7c99cf1e:::benchmark_data_smithi177_159919_object1658:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-05-15T13:24:38.288 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:23:12.833551+0000", 2024-05-15T13:24:38.288 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 85.406804765000004, 2024-05-15T13:24:38.288 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.160055116, 2024-05-15T13:24:38.288 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:38.288 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:38.289 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:38.289 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:24:38.289 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:24:38.289 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 1659 2024-05-15T13:24:38.289 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.289 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:38.289 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.289 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:38.289 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.833551+0000", 2024-05-15T13:24:38.289 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.289 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.289 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.289 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:38.289 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.833551+0000", 2024-05-15T13:24:38.289 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.289 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.290 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.290 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:38.290 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.833552+0000", 2024-05-15T13:24:38.290 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.9899999999999999e-07 2024-05-15T13:24:38.290 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.290 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.290 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:38.290 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.833648+0000", 2024-05-15T13:24:38.290 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.6299999999999996e-05 2024-05-15T13:24:38.290 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.290 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.290 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:38.290 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.833662+0000", 2024-05-15T13:24:38.290 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3227999999999999e-05 2024-05-15T13:24:38.291 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.291 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.291 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:38.291 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.833669+0000", 2024-05-15T13:24:38.291 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.413e-06 2024-05-15T13:24:38.291 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.291 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.291 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:38.291 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.833721+0000", 2024-05-15T13:24:38.291 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2094e-05 2024-05-15T13:24:38.291 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.291 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.291 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:38.291 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.833827+0000", 2024-05-15T13:24:38.292 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00010587200000000001 2024-05-15T13:24:38.292 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.292 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.292 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,10", 2024-05-15T13:24:38.292 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.833964+0000", 2024-05-15T13:24:38.292 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000136623 2024-05-15T13:24:38.292 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.292 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.292 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:38.292 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.834355+0000", 2024-05-15T13:24:38.292 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00039170999999999997 2024-05-15T13:24:38.292 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.292 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.292 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.293 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.834703+0000", 2024-05-15T13:24:38.293 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00034733600000000001 2024-05-15T13:24:38.293 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.293 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.293 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.293 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.993514+0000", 2024-05-15T13:24:38.293 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.158811338 2024-05-15T13:24:38.293 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.293 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.293 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:38.293 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.993565+0000", 2024-05-15T13:24:38.293 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1243999999999999e-05 2024-05-15T13:24:38.293 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.293 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.293 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:38.294 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:12.993606+0000", 2024-05-15T13:24:38.294 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.1059000000000001e-05 2024-05-15T13:24:38.294 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.294 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:38.294 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.294 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.294 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.294 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:59060 9.d 9:b146a532:::benchmark_data_smithi177_159919_object59059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e829)", 2024-05-15T13:24:38.294 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:23:50.380258+0000", 2024-05-15T13:24:38.294 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 47.860097572999997, 2024-05-15T13:24:38.294 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.13987945299999999, 2024-05-15T13:24:38.294 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:38.294 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:38.294 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:38.295 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:24:38.295 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:24:38.295 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 59060 2024-05-15T13:24:38.295 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.295 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:38.295 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.295 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:38.295 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:50.380258+0000", 2024-05-15T13:24:38.295 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.295 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.295 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.295 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:38.295 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:50.380258+0000", 2024-05-15T13:24:38.295 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.296 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.296 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.296 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:38.296 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:50.380259+0000", 2024-05-15T13:24:38.296 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4799999999999998e-07 2024-05-15T13:24:38.296 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.296 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.296 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:38.296 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:50.380380+0000", 2024-05-15T13:24:38.296 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012055799999999999 2024-05-15T13:24:38.296 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.296 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.296 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:38.296 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:50.380386+0000", 2024-05-15T13:24:38.296 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8210000000000004e-06 2024-05-15T13:24:38.297 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.297 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.297 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:38.297 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:50.380392+0000", 2024-05-15T13:24:38.297 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1059999999999999e-06 2024-05-15T13:24:38.297 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.297 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.297 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:38.297 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:50.380428+0000", 2024-05-15T13:24:38.297 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.6192999999999997e-05 2024-05-15T13:24:38.297 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.297 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.297 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:38.297 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:50.380506+0000", 2024-05-15T13:24:38.297 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8573999999999997e-05 2024-05-15T13:24:38.298 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.298 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.298 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,5", 2024-05-15T13:24:38.298 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:50.380583+0000", 2024-05-15T13:24:38.298 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.6763999999999999e-05 2024-05-15T13:24:38.298 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.298 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.298 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:38.298 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:50.383306+0000", 2024-05-15T13:24:38.298 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.002722683 2024-05-15T13:24:38.298 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.298 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.298 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.298 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:50.384278+0000", 2024-05-15T13:24:38.299 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00097208400000000001 2024-05-15T13:24:38.299 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.299 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.299 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.299 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:50.520035+0000", 2024-05-15T13:24:38.299 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.13575753700000001 2024-05-15T13:24:38.299 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.299 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.299 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:38.299 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:50.520090+0000", 2024-05-15T13:24:38.299 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5053999999999998e-05 2024-05-15T13:24:38.299 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.299 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.300 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:38.300 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:50.520138+0000", 2024-05-15T13:24:38.300 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7431e-05 2024-05-15T13:24:38.300 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.300 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:38.300 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.300 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.300 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.300 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:71228 9.d 9:b5497010:::benchmark_data_smithi177_159919_object71227:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e830)", 2024-05-15T13:24:38.300 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:23:58.077735+0000", 2024-05-15T13:24:38.300 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 40.162620685, 2024-05-15T13:24:38.300 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.202911431, 2024-05-15T13:24:38.300 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:38.300 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:38.300 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:38.301 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:24:38.301 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:24:38.301 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 71228 2024-05-15T13:24:38.301 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.301 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:38.301 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.301 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:38.301 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.077735+0000", 2024-05-15T13:24:38.301 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.301 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.301 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.301 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:38.301 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.077735+0000", 2024-05-15T13:24:38.301 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.302 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.302 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.302 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:38.302 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.077737+0000", 2024-05-15T13:24:38.302 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2669999999999999e-06 2024-05-15T13:24:38.302 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.302 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.302 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:38.302 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.077863+0000", 2024-05-15T13:24:38.302 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000126425 2024-05-15T13:24:38.302 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.302 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.302 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:38.303 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.077945+0000", 2024-05-15T13:24:38.303 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1644000000000006e-05 2024-05-15T13:24:38.303 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.303 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.303 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:38.303 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.077956+0000", 2024-05-15T13:24:38.303 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1124000000000001e-05 2024-05-15T13:24:38.303 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.303 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.303 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:38.303 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.078013+0000", 2024-05-15T13:24:38.303 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6932999999999999e-05 2024-05-15T13:24:38.304 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.304 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.304 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:38.304 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.078153+0000", 2024-05-15T13:24:38.304 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014070400000000001 2024-05-15T13:24:38.304 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.304 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.304 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,5", 2024-05-15T13:24:38.304 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.078285+0000", 2024-05-15T13:24:38.304 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013154200000000001 2024-05-15T13:24:38.304 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.304 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.304 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:38.304 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.080645+0000", 2024-05-15T13:24:38.305 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0023598849999999999 2024-05-15T13:24:38.305 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.305 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.305 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.305 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.084591+0000", 2024-05-15T13:24:38.305 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0039466120000000004 2024-05-15T13:24:38.305 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.305 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.305 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.305 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.280566+0000", 2024-05-15T13:24:38.305 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.19597426500000001 2024-05-15T13:24:38.305 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.305 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.305 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:38.305 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.280604+0000", 2024-05-15T13:24:38.305 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.8625999999999999e-05 2024-05-15T13:24:38.306 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.306 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.306 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:38.306 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.280647+0000", 2024-05-15T13:24:38.306 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2404000000000003e-05 2024-05-15T13:24:38.306 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.306 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:38.306 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.306 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.306 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.306 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:71243 9.d 9:b391a2ac:::benchmark_data_smithi177_159919_object71242:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e830)", 2024-05-15T13:24:38.306 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:23:58.090175+0000", 2024-05-15T13:24:38.307 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 40.150181242000002, 2024-05-15T13:24:38.307 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.19254369599999999, 2024-05-15T13:24:38.307 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:38.307 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:38.307 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:38.307 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:24:38.307 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:24:38.307 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 71243 2024-05-15T13:24:38.307 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.307 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:38.307 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.307 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:38.307 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.090175+0000", 2024-05-15T13:24:38.307 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.307 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.307 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.308 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:38.308 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.090175+0000", 2024-05-15T13:24:38.308 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.308 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.308 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.308 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:38.308 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.090176+0000", 2024-05-15T13:24:38.308 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1399999999999995e-07 2024-05-15T13:24:38.308 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.308 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.308 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:38.308 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.090228+0000", 2024-05-15T13:24:38.308 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2472000000000002e-05 2024-05-15T13:24:38.308 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.308 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.309 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:38.309 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.090237+0000", 2024-05-15T13:24:38.309 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.5210000000000002e-06 2024-05-15T13:24:38.309 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.309 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.309 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:38.309 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.090242+0000", 2024-05-15T13:24:38.309 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3279999999999996e-06 2024-05-15T13:24:38.309 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.309 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.309 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:38.309 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.090300+0000", 2024-05-15T13:24:38.309 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8109999999999998e-05 2024-05-15T13:24:38.309 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.309 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.309 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:38.310 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.090397+0000", 2024-05-15T13:24:38.310 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.7330000000000005e-05 2024-05-15T13:24:38.310 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.310 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.310 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,5", 2024-05-15T13:24:38.310 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.090488+0000", 2024-05-15T13:24:38.310 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.0447999999999999e-05 2024-05-15T13:24:38.310 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.310 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.310 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:38.310 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.094636+0000", 2024-05-15T13:24:38.310 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.004147838 2024-05-15T13:24:38.310 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.310 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.310 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.311 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.098626+0000", 2024-05-15T13:24:38.311 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0039906189999999999 2024-05-15T13:24:38.311 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.311 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.311 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.311 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.282623+0000", 2024-05-15T13:24:38.311 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.183996719 2024-05-15T13:24:38.311 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.311 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.311 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:38.311 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.282676+0000", 2024-05-15T13:24:38.311 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2451999999999999e-05 2024-05-15T13:24:38.311 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.311 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.311 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:38.311 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.282718+0000", 2024-05-15T13:24:38.312 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2945000000000003e-05 2024-05-15T13:24:38.312 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.312 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:38.312 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.312 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.312 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.312 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:71248 9.d 9:b4ea0935:::benchmark_data_smithi177_159919_object71247:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e830)", 2024-05-15T13:24:38.312 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:23:58.090284+0000", 2024-05-15T13:24:38.312 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 40.150071543999999, 2024-05-15T13:24:38.312 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.193879779, 2024-05-15T13:24:38.312 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:38.312 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:38.312 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:38.312 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:24:38.312 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:24:38.313 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 71248 2024-05-15T13:24:38.313 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.313 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:38.313 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.313 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:38.313 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.090284+0000", 2024-05-15T13:24:38.313 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.313 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.313 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.313 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:38.313 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.090284+0000", 2024-05-15T13:24:38.313 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.313 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.314 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.314 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:38.314 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.090285+0000", 2024-05-15T13:24:38.314 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.5600000000000004e-07 2024-05-15T13:24:38.314 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.314 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.314 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:38.314 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.090449+0000", 2024-05-15T13:24:38.314 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000163819 2024-05-15T13:24:38.314 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.314 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.314 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:38.314 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.090457+0000", 2024-05-15T13:24:38.314 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4540000000000001e-06 2024-05-15T13:24:38.315 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.315 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.315 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:38.315 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.090461+0000", 2024-05-15T13:24:38.315 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7160000000000002e-06 2024-05-15T13:24:38.315 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.315 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.315 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:38.315 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.092428+0000", 2024-05-15T13:24:38.315 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0019662049999999999 2024-05-15T13:24:38.315 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.315 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.315 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:38.315 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.092544+0000", 2024-05-15T13:24:38.316 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000116123 2024-05-15T13:24:38.316 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.316 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.316 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,5", 2024-05-15T13:24:38.316 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.092640+0000", 2024-05-15T13:24:38.316 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.6335999999999994e-05 2024-05-15T13:24:38.316 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.316 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.316 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:38.316 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.094746+0000", 2024-05-15T13:24:38.316 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0021060620000000001 2024-05-15T13:24:38.316 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.316 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.316 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.317 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.099232+0000", 2024-05-15T13:24:38.317 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0044854730000000002 2024-05-15T13:24:38.317 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.317 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.317 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.317 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.284067+0000", 2024-05-15T13:24:38.317 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.18483507699999999 2024-05-15T13:24:38.317 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.317 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.317 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:38.317 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.284124+0000", 2024-05-15T13:24:38.317 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7605999999999997e-05 2024-05-15T13:24:38.318 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.318 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.318 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:38.318 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.284164+0000", 2024-05-15T13:24:38.318 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0052000000000002e-05 2024-05-15T13:24:38.318 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.318 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:38.318 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.318 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.318 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.318 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:71250 9.d 9:ba70e9d4:::benchmark_data_smithi177_159919_object71249:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e830)", 2024-05-15T13:24:38.318 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:23:58.095681+0000", 2024-05-15T13:24:38.318 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 40.144675399, 2024-05-15T13:24:38.319 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.188623124, 2024-05-15T13:24:38.319 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:38.319 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:38.319 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:38.319 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:24:38.319 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:24:38.319 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 71250 2024-05-15T13:24:38.319 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.319 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:38.319 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.319 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:38.319 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.095681+0000", 2024-05-15T13:24:38.319 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.319 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.319 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.319 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:38.320 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.095681+0000", 2024-05-15T13:24:38.320 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.320 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.320 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.320 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:38.320 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.095682+0000", 2024-05-15T13:24:38.320 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.9900000000000009e-07 2024-05-15T13:24:38.320 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.320 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.320 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:38.320 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.095747+0000", 2024-05-15T13:24:38.320 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4923000000000002e-05 2024-05-15T13:24:38.320 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.320 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.320 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:38.320 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.095757+0000", 2024-05-15T13:24:38.321 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0492000000000001e-05 2024-05-15T13:24:38.321 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.321 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.321 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:38.321 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.095763+0000", 2024-05-15T13:24:38.321 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4359999999999999e-06 2024-05-15T13:24:38.321 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.321 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.321 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:38.321 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.095808+0000", 2024-05-15T13:24:38.321 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4314000000000003e-05 2024-05-15T13:24:38.321 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.321 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.321 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:38.321 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.095903+0000", 2024-05-15T13:24:38.322 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.5719999999999998e-05 2024-05-15T13:24:38.322 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.322 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.322 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,5", 2024-05-15T13:24:38.322 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.095996+0000", 2024-05-15T13:24:38.322 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.2455999999999997e-05 2024-05-15T13:24:38.322 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.322 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.322 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:38.322 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.099248+0000", 2024-05-15T13:24:38.322 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0032522649999999998 2024-05-15T13:24:38.322 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.322 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.322 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.322 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.102377+0000", 2024-05-15T13:24:38.323 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0031283180000000002 2024-05-15T13:24:38.323 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.323 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.323 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.323 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.284204+0000", 2024-05-15T13:24:38.323 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.18182726799999999 2024-05-15T13:24:38.323 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.323 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.323 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:38.323 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.284262+0000", 2024-05-15T13:24:38.323 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8010000000000002e-05 2024-05-15T13:24:38.323 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.323 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.323 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:38.323 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.284304+0000", 2024-05-15T13:24:38.324 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.1922999999999998e-05 2024-05-15T13:24:38.324 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.324 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:38.324 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.324 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.324 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.324 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:71268 9.d 9:b4f37617:::benchmark_data_smithi177_159919_object71267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e830)", 2024-05-15T13:24:38.324 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:23:58.108201+0000", 2024-05-15T13:24:38.324 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 40.132154890999999, 2024-05-15T13:24:38.324 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.17924968399999999, 2024-05-15T13:24:38.324 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:38.324 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:38.324 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:38.324 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:24:38.324 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:24:38.324 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 71268 2024-05-15T13:24:38.324 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.325 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:38.325 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.325 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:38.325 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.108201+0000", 2024-05-15T13:24:38.325 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.325 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.325 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.325 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:38.325 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.108201+0000", 2024-05-15T13:24:38.325 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.325 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.325 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.325 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:38.325 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.108209+0000", 2024-05-15T13:24:38.325 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1650000000000006e-06 2024-05-15T13:24:38.326 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.326 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.326 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:38.326 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.108258+0000", 2024-05-15T13:24:38.326 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8899000000000001e-05 2024-05-15T13:24:38.326 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.326 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.326 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:38.326 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.108276+0000", 2024-05-15T13:24:38.326 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7867e-05 2024-05-15T13:24:38.326 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.326 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.326 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:38.326 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.108283+0000", 2024-05-15T13:24:38.327 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6789999999999999e-06 2024-05-15T13:24:38.327 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.327 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.327 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:38.327 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.108345+0000", 2024-05-15T13:24:38.327 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1861999999999998e-05 2024-05-15T13:24:38.327 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.327 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.327 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:38.327 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.108477+0000", 2024-05-15T13:24:38.327 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000132364 2024-05-15T13:24:38.327 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.327 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.327 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,5", 2024-05-15T13:24:38.327 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.108618+0000", 2024-05-15T13:24:38.328 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014066799999999999 2024-05-15T13:24:38.328 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.328 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.328 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:38.328 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.111041+0000", 2024-05-15T13:24:38.328 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0024236829999999998 2024-05-15T13:24:38.328 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.328 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.328 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.328 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.111424+0000", 2024-05-15T13:24:38.328 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00038226699999999999 2024-05-15T13:24:38.328 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.328 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.329 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.329 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.287347+0000", 2024-05-15T13:24:38.329 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.175923896 2024-05-15T13:24:38.329 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.329 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.329 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:38.329 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.287403+0000", 2024-05-15T13:24:38.329 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5241999999999997e-05 2024-05-15T13:24:38.329 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.329 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.329 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:38.329 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.287451+0000", 2024-05-15T13:24:38.330 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8092e-05 2024-05-15T13:24:38.330 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.330 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:38.330 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.330 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.330 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.330 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:71280 9.d 9:bf08c608:::benchmark_data_smithi177_159919_object71279:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e830)", 2024-05-15T13:24:38.330 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:23:58.115733+0000", 2024-05-15T13:24:38.330 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 40.124623317999998, 2024-05-15T13:24:38.330 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.17220660800000001, 2024-05-15T13:24:38.330 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:38.330 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:38.330 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:38.331 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:24:38.331 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:24:38.331 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 71280 2024-05-15T13:24:38.331 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.331 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:38.331 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.331 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:38.331 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.115733+0000", 2024-05-15T13:24:38.331 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.331 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.331 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.331 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:38.332 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.115733+0000", 2024-05-15T13:24:38.332 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.332 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.332 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.332 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:38.332 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.115734+0000", 2024-05-15T13:24:38.333 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.353e-06 2024-05-15T13:24:38.333 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.333 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.333 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:38.333 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.115800+0000", 2024-05-15T13:24:38.333 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6149000000000007e-05 2024-05-15T13:24:38.334 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.334 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.334 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:38.334 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.115813+0000", 2024-05-15T13:24:38.334 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3251000000000001e-05 2024-05-15T13:24:38.334 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.335 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.335 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:38.335 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.115821+0000", 2024-05-15T13:24:38.335 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4549999999999998e-06 2024-05-15T13:24:38.335 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.335 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.336 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:38.336 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.116095+0000", 2024-05-15T13:24:38.336 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000273849 2024-05-15T13:24:38.336 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.336 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.337 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:38.337 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.116224+0000", 2024-05-15T13:24:38.337 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000129293 2024-05-15T13:24:38.337 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.337 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.337 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,5", 2024-05-15T13:24:38.337 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.116366+0000", 2024-05-15T13:24:38.338 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000142402 2024-05-15T13:24:38.338 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.338 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.338 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:38.338 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.118638+0000", 2024-05-15T13:24:38.338 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0022717219999999999 2024-05-15T13:24:38.339 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.339 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.339 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.339 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.120643+0000", 2024-05-15T13:24:38.339 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0020052189999999999 2024-05-15T13:24:38.339 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.340 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.340 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.340 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.287838+0000", 2024-05-15T13:24:38.340 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.16719453200000001 2024-05-15T13:24:38.340 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.340 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.341 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:38.341 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.287896+0000", 2024-05-15T13:24:38.341 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7637999999999999e-05 2024-05-15T13:24:38.341 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.341 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.341 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:38.341 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.287939+0000", 2024-05-15T13:24:38.342 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3745000000000002e-05 2024-05-15T13:24:38.342 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.342 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:38.342 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.342 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.343 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.343 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:71292 9.d 9:b1656549:::benchmark_data_smithi177_159919_object71291:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e830)", 2024-05-15T13:24:38.343 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:23:58.128266+0000", 2024-05-15T13:24:38.343 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 40.112089886, 2024-05-15T13:24:38.343 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.163382732, 2024-05-15T13:24:38.343 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:38.343 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:38.344 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:38.344 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:24:38.344 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:24:38.344 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 71292 2024-05-15T13:24:38.344 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.344 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:38.345 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.345 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:38.345 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.128266+0000", 2024-05-15T13:24:38.345 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.345 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.345 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.345 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:38.345 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.128266+0000", 2024-05-15T13:24:38.345 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.345 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.346 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.346 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:38.346 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.128268+0000", 2024-05-15T13:24:38.346 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5400000000000001e-06 2024-05-15T13:24:38.346 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.346 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.347 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:38.347 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:24:38.349 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 86 16 117454 117438 85.3157 44.25 0.00795977 0.0117176 2024-05-15T13:24:38.349 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.128328+0000", 2024-05-15T13:24:38.349 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.0276999999999999e-05 2024-05-15T13:24:38.349 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.349 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.350 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:38.350 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.128342+0000", 2024-05-15T13:24:38.350 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3822000000000001e-05 2024-05-15T13:24:38.350 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.353 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.354 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:38.354 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.128350+0000", 2024-05-15T13:24:38.354 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.5089999999999993e-06 2024-05-15T13:24:38.354 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.354 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.355 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:38.355 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.128432+0000", 2024-05-15T13:24:38.355 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1382000000000001e-05 2024-05-15T13:24:38.355 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.355 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.355 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:38.356 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.128592+0000", 2024-05-15T13:24:38.356 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015985399999999999 2024-05-15T13:24:38.356 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.356 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.356 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,5", 2024-05-15T13:24:38.356 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.128753+0000", 2024-05-15T13:24:38.357 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00016124999999999999 2024-05-15T13:24:38.357 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.357 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.357 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.357 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.131131+0000", 2024-05-15T13:24:38.357 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.002378401 2024-05-15T13:24:38.358 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.358 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.358 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:38.358 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.131235+0000", 2024-05-15T13:24:38.358 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000103464 2024-05-15T13:24:38.358 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.358 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.359 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.359 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.291538+0000", 2024-05-15T13:24:38.359 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.160303108 2024-05-15T13:24:38.359 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.359 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.359 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:38.359 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.291601+0000", 2024-05-15T13:24:38.360 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3279000000000002e-05 2024-05-15T13:24:38.360 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.360 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.360 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:38.360 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.291649+0000", 2024-05-15T13:24:38.360 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7846000000000002e-05 2024-05-15T13:24:38.361 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.361 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:38.361 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.361 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.361 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.361 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:71299 9.d 9:bf98ad41:::benchmark_data_smithi177_159919_object71298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e830)", 2024-05-15T13:24:38.361 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:23:58.136697+0000", 2024-05-15T13:24:38.362 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 40.103659352999998, 2024-05-15T13:24:38.362 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.16297299300000001, 2024-05-15T13:24:38.362 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:38.362 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:38.362 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:38.362 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:24:38.363 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:24:38.363 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 71299 2024-05-15T13:24:38.363 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.363 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:38.363 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.363 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:38.364 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.136697+0000", 2024-05-15T13:24:38.364 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.364 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.364 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.364 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:38.364 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.136697+0000", 2024-05-15T13:24:38.364 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.365 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.365 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.365 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:38.365 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.136698+0000", 2024-05-15T13:24:38.365 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2219999999999999e-06 2024-05-15T13:24:38.365 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.366 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.366 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:38.366 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.136751+0000", 2024-05-15T13:24:38.366 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3087000000000002e-05 2024-05-15T13:24:38.366 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.367 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.367 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:38.367 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.136763+0000", 2024-05-15T13:24:38.367 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1721e-05 2024-05-15T13:24:38.367 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.367 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.368 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:38.368 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.136769+0000", 2024-05-15T13:24:38.368 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9190000000000004e-06 2024-05-15T13:24:38.368 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.368 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.368 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:38.369 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.136830+0000", 2024-05-15T13:24:38.369 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1532999999999995e-05 2024-05-15T13:24:38.369 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.369 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.369 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:38.369 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.136940+0000", 2024-05-15T13:24:38.370 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000110223 2024-05-15T13:24:38.370 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.370 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.370 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,5", 2024-05-15T13:24:38.370 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.137029+0000", 2024-05-15T13:24:38.370 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8938999999999997e-05 2024-05-15T13:24:38.371 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.371 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.371 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.371 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.139356+0000", 2024-05-15T13:24:38.371 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0023266179999999999 2024-05-15T13:24:38.371 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.372 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.372 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:38.372 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.139521+0000", 2024-05-15T13:24:38.372 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000164661 2024-05-15T13:24:38.372 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.372 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.373 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.373 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.299569+0000", 2024-05-15T13:24:38.373 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.16004874599999999 2024-05-15T13:24:38.373 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.373 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.373 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:38.374 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.299626+0000", 2024-05-15T13:24:38.374 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7006999999999998e-05 2024-05-15T13:24:38.374 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.374 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.374 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:38.374 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.299670+0000", 2024-05-15T13:24:38.375 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3316999999999999e-05 2024-05-15T13:24:38.375 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.375 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:38.375 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.375 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.375 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.376 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:71309 9.d 9:bc693764:::benchmark_data_smithi177_159919_object71308:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e830)", 2024-05-15T13:24:38.376 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:23:58.146612+0000", 2024-05-15T13:24:38.376 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 40.093744297000001, 2024-05-15T13:24:38.376 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.15321568499999999, 2024-05-15T13:24:38.376 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:38.376 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:38.376 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:38.377 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:24:38.377 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:24:38.377 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 71309 2024-05-15T13:24:38.377 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.377 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:38.377 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.378 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:38.378 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.146612+0000", 2024-05-15T13:24:38.378 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.378 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.378 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.378 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:38.379 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.146612+0000", 2024-05-15T13:24:38.379 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.379 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.379 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.379 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:38.379 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.146613+0000", 2024-05-15T13:24:38.380 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.4600000000000003e-07 2024-05-15T13:24:38.380 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.380 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.380 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:38.380 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.146683+0000", 2024-05-15T13:24:38.380 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0078000000000002e-05 2024-05-15T13:24:38.381 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.381 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.381 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:38.381 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.146694+0000", 2024-05-15T13:24:38.381 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1088e-05 2024-05-15T13:24:38.381 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.381 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.382 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:38.382 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.146700+0000", 2024-05-15T13:24:38.382 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8810000000000001e-06 2024-05-15T13:24:38.382 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.382 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.382 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:38.383 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.146756+0000", 2024-05-15T13:24:38.383 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5927000000000002e-05 2024-05-15T13:24:38.383 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.383 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.383 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:38.383 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.146880+0000", 2024-05-15T13:24:38.383 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000124354 2024-05-15T13:24:38.384 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.384 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.384 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,5", 2024-05-15T13:24:38.384 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.147024+0000", 2024-05-15T13:24:38.384 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000144289 2024-05-15T13:24:38.384 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.385 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.385 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:38.385 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.148947+0000", 2024-05-15T13:24:38.385 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0019223039999999999 2024-05-15T13:24:38.385 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.385 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.385 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.385 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.150075+0000", 2024-05-15T13:24:38.385 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0011287770000000001 2024-05-15T13:24:38.385 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.385 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.386 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.386 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.299738+0000", 2024-05-15T13:24:38.386 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.14966309799999999 2024-05-15T13:24:38.386 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.386 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.386 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:38.386 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.299790+0000", 2024-05-15T13:24:38.386 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1118000000000001e-05 2024-05-15T13:24:38.386 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.386 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.386 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:38.386 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:23:58.299827+0000", 2024-05-15T13:24:38.386 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.7824999999999998e-05 2024-05-15T13:24:38.386 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.386 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:38.387 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.387 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.387 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.387 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:97041 9.d 9:bd789d55:::benchmark_data_smithi177_159919_object97040:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e842)", 2024-05-15T13:24:38.387 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:24:17.498349+0000", 2024-05-15T13:24:38.387 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 20.742007069, 2024-05-15T13:24:38.387 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.184198897, 2024-05-15T13:24:38.387 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:24:38.387 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:24:38.387 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:24:38.387 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:24:38.387 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:24:38.387 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 97041 2024-05-15T13:24:38.387 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.388 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:24:38.388 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.388 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:24:38.388 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:24:17.498349+0000", 2024-05-15T13:24:38.388 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.388 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.388 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.388 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:24:38.388 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:24:17.498349+0000", 2024-05-15T13:24:38.388 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:24:38.388 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.388 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.388 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:24:38.388 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:24:17.498350+0000", 2024-05-15T13:24:38.388 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.2799999999999996e-07 2024-05-15T13:24:38.388 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.389 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.389 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:24:38.389 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:24:17.498523+0000", 2024-05-15T13:24:38.389 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00017390600000000001 2024-05-15T13:24:38.389 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.389 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.389 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:24:38.389 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:24:17.498533+0000", 2024-05-15T13:24:38.389 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.3789999999999995e-06 2024-05-15T13:24:38.389 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.389 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.389 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:24:38.389 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:24:17.498538+0000", 2024-05-15T13:24:38.389 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.164e-06 2024-05-15T13:24:38.389 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.390 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.390 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:24:38.390 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:24:17.498590+0000", 2024-05-15T13:24:38.390 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2376000000000001e-05 2024-05-15T13:24:38.390 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.390 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.390 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:24:38.390 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:24:17.498794+0000", 2024-05-15T13:24:38.390 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000203389 2024-05-15T13:24:38.390 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.390 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.390 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,5", 2024-05-15T13:24:38.390 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:24:17.498940+0000", 2024-05-15T13:24:38.390 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014624000000000001 2024-05-15T13:24:38.391 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.391 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.391 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.391 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:24:17.505845+0000", 2024-05-15T13:24:38.391 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0069049599999999999 2024-05-15T13:24:38.391 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.391 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.391 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:24:38.391 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:24:17.505930+0000", 2024-05-15T13:24:38.391 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.5321000000000005e-05 2024-05-15T13:24:38.391 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.391 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.391 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:24:38.391 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:24:17.682438+0000", 2024-05-15T13:24:38.391 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.17650753899999999 2024-05-15T13:24:38.392 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.392 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.392 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:24:38.392 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:24:17.682501+0000", 2024-05-15T13:24:38.392 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3350999999999997e-05 2024-05-15T13:24:38.392 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:24:38.392 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:24:38.392 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:24:38.392 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:24:17.682548+0000", 2024-05-15T13:24:38.392 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6644000000000003e-05 2024-05-15T13:24:38.392 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.392 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:38.392 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.392 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:24:38.392 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:24:38.392 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:24:38.393 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:37 smithi120 ceph-mon[43961]: pgmap v2199: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 7.0 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-05-15T13:24:38.426 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:38.450 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:24:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:38.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:24:38.556 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:24:38.595 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:38.961 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:24:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:38.965 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:24:39.067 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:24:39.116 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:39.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:39 smithi177 ceph-mon[44303]: 9.6 scrub ok 2024-05-15T13:24:39.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:39 smithi027 ceph-mon[39795]: 9.6 scrub ok 2024-05-15T13:24:39.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:39 smithi120 ceph-mon[43961]: 9.6 scrub ok 2024-05-15T13:24:39.326 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 87 16 118220 118204 84.8855 47.875 0.00862546 0.0117739 2024-05-15T13:24:39.488 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:24:40.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:40 smithi027 ceph-mon[39795]: pgmap v2200: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.1 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-05-15T13:24:40.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:40 smithi120 ceph-mon[43961]: pgmap v2200: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.1 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-05-15T13:24:40.326 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 88 16 118898 118882 84.4025 42.375 0.0104812 0.0118408 2024-05-15T13:24:40.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:40 smithi177 ceph-mon[44303]: pgmap v2200: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.1 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-05-15T13:24:40.594 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:24:40.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:40.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:24:40.649 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:24:40.702 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:24:40.743 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:40.765 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:24:40.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:40.774 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:24:40.875 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:24:40.905 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:40.925 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:24:40.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:40.931 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:24:41.032 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:24:41.088 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:41.194 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:24:41.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:41.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:24:41.299 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:24:41.326 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 89 16 119549 119533 83.9114 40.6875 0.00714944 0.011911 2024-05-15T13:24:41.337 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:41.360 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:24:41.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:41.366 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:24:41.467 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:24:41.497 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:41.537 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:24:41.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:41.544 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:24:41.645 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:24:41.688 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-05-15T13:24:41.707 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:42.022 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:42 smithi027 ceph-mon[39795]: pgmap v2201: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 7.1 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 38 MiB/s wr, 611 op/s 2024-05-15T13:24:42.028 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:42 smithi177 ceph-mon[44303]: pgmap v2201: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 7.1 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 38 MiB/s wr, 611 op/s 2024-05-15T13:24:42.061 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:24:42.088 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:24:42.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:42.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:24:42.194 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:24:42.232 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:42.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:42 smithi120 ceph-mon[43961]: pgmap v2201: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 7.1 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 38 MiB/s wr, 611 op/s 2024-05-15T13:24:42.314 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:24:42.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:42.319 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:24:42.326 INFO:tasks.radosbench.radosbench.2.smithi177.stdout: 90 12 120240 120228 83.4618 43.4375 0.00723635 0.0119674 2024-05-15T13:24:42.420 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:24:42.452 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:42.808 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:24:42.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:42.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:24:42.914 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:24:42.942 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:43.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:43 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3625177958' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:24:43.027 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:43 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:24:43.031 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:24:43.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:43 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3625177958' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:24:43.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:43 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:24:43.326 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Total time run: 90.1266 2024-05-15T13:24:43.327 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Total writes made: 120240 2024-05-15T13:24:43.327 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Write size: 65536 2024-05-15T13:24:43.327 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Object size: 65536 2024-05-15T13:24:43.327 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Bandwidth (MB/sec): 83.3827 2024-05-15T13:24:43.327 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Stddev Bandwidth: 55.3937 2024-05-15T13:24:43.327 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Max bandwidth (MB/sec): 270.062 2024-05-15T13:24:43.327 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Min bandwidth (MB/sec): 0 2024-05-15T13:24:43.327 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Average IOPS: 1334 2024-05-15T13:24:43.327 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Stddev IOPS: 886.3 2024-05-15T13:24:43.327 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Max IOPS: 4321 2024-05-15T13:24:43.327 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Min IOPS: 0 2024-05-15T13:24:43.327 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Average Latency(s): 0.0119849 2024-05-15T13:24:43.327 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Stddev Latency(s): 0.158019 2024-05-15T13:24:43.327 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Max latency(s): 13.6712 2024-05-15T13:24:43.327 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Min latency(s): 0.00110277 2024-05-15T13:24:43.332 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:43 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3625177958' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:24:43.332 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:43 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:24:43.333 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Cleaning up (deleting benchmark objects) 2024-05-15T13:24:43.393 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:24:43.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:43.400 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:24:43.501 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:24:43.537 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:43.656 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:24:43.684 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:24:43.697 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:24:43.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:43.702 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:24:43.714 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:24:43.752 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:24:43.785 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:24:43.802 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:24:43.839 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:24:43.845 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:43.878 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:24:43.915 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:24:43.948 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:24:43.950 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:24:43.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:43.955 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:24:43.971 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 9, 10, 11, 7] out_osds: [4, 8] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:24:43.971 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:24:43.972 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-05-15T13:24:43.972 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd in 4 2024-05-15T13:24:43.988 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:24:44.027 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:24:44.027 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:24:44.036 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:24:44.055 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:24:44.074 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:24:44.087 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:44.100 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:24:44.100 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:24:44.100 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:24:44.100 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:24:44.100 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:24:44.100 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:24:44.100 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:24:44.100 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:24:44.100 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:24:44.100 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:24:44.100 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:24:44.100 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:24:44.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:44 smithi027 ceph-mon[39795]: pgmap v2202: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 7.2 GiB data, 30 GiB used, 865 GiB / 894 GiB avail; 43 MiB/s wr, 694 op/s 2024-05-15T13:24:44.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:44 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:24:44.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:44 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:24:44.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:44 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:24:44.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:44 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:24:44.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:44 smithi027 ceph-mon[39795]: osdmap e852: 12 total, 12 up, 10 in 2024-05-15T13:24:44.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:44 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/643258221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:24:44.290 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:44 smithi120 ceph-mon[43961]: pgmap v2202: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 7.2 GiB data, 30 GiB used, 865 GiB / 894 GiB avail; 43 MiB/s wr, 694 op/s 2024-05-15T13:24:44.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:44 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:24:44.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:44 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:24:44.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:44 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:24:44.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:44 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:24:44.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:44 smithi120 ceph-mon[43961]: osdmap e852: 12 total, 12 up, 10 in 2024-05-15T13:24:44.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:44 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/643258221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:24:44.295 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:24:44.299 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:44 smithi177 ceph-mon[44303]: pgmap v2202: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 7.2 GiB data, 30 GiB used, 865 GiB / 894 GiB avail; 43 MiB/s wr, 694 op/s 2024-05-15T13:24:44.299 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:44 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:24:44.299 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:44 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:24:44.299 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:44 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:24:44.299 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:44 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:24:44.299 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:44 smithi177 ceph-mon[44303]: osdmap e852: 12 total, 12 up, 10 in 2024-05-15T13:24:44.299 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:44 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/643258221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:24:44.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:44.300 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:24:44.401 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:24:44.455 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:44.722 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:24:44.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:44.729 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:24:44.731 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-05-15T13:24:44.830 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:24:44.863 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:44.884 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:24:44.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:44.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:24:44.903 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:24:44.988 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:24:45.028 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:45.070 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:24:45.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:45.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:24:45.136 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:24:45.175 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:24:45.208 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:45.237 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:24:45.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:45.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:24:45.348 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:24:45.397 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:45.436 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:24:45.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:45.444 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:24:45.512 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:45 smithi027 ceph-mon[39795]: osdmap e853: 12 total, 12 up, 10 in 2024-05-15T13:24:45.512 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:45 smithi027 ceph-mon[39795]: pgmap v2205: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 7.3 GiB data, 30 GiB used, 864 GiB / 894 GiB avail; 47 MiB/s wr, 764 op/s 2024-05-15T13:24:45.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:45 smithi120 ceph-mon[43961]: osdmap e853: 12 total, 12 up, 10 in 2024-05-15T13:24:45.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:45 smithi120 ceph-mon[43961]: pgmap v2205: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 7.3 GiB data, 30 GiB used, 864 GiB / 894 GiB avail; 47 MiB/s wr, 764 op/s 2024-05-15T13:24:45.545 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:24:45.609 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:45.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:45 smithi177 ceph-mon[44303]: osdmap e853: 12 total, 12 up, 10 in 2024-05-15T13:24:45.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:45 smithi177 ceph-mon[44303]: pgmap v2205: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 7.3 GiB data, 30 GiB used, 864 GiB / 894 GiB avail; 47 MiB/s wr, 764 op/s 2024-05-15T13:24:45.810 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:24:45.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:45.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:24:45.920 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:24:45.948 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:45.967 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:24:45.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:45.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:24:46.071 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:24:46.091 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:24:46.113 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:46.552 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:24:46.886 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:24:47.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:47 smithi027 ceph-mon[39795]: pgmap v2206: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 7.3 GiB data, 30 GiB used, 864 GiB / 894 GiB avail; 127 B/s rd, 34 MiB/s wr, 598 op/s 2024-05-15T13:24:48.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:47 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-a[39772]: 2024-05-15T13:24:47.784+0000 7f8daffd2700 -1 mon.a@0(leader).osd e853 definitely_dead 0 2024-05-15T13:24:48.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:47 smithi120 ceph-mon[43961]: pgmap v2206: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 7.3 GiB data, 30 GiB used, 864 GiB / 894 GiB avail; 127 B/s rd, 34 MiB/s wr, 598 op/s 2024-05-15T13:24:48.140 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:24:48.145 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:47 smithi177 ceph-mon[44303]: pgmap v2206: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 7.3 GiB data, 30 GiB used, 864 GiB / 894 GiB avail; 127 B/s rd, 34 MiB/s wr, 598 op/s 2024-05-15T13:24:48.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:48.146 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:24:48.247 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:24:48.295 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:24:48.502 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:24:48.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:48.506 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:24:48.607 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:24:48.696 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:48.737 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:24:48.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:48.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:24:48.785 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:48 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3608451138' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-05-15T13:24:48.785 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:48 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-05-15T13:24:48.800 INFO:teuthology.orchestra.run.smithi027.stderr:marked in osd.4. 2024-05-15T13:24:48.847 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:24:48.874 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:48.907 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:48 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3608451138' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-05-15T13:24:48.908 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:48 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-05-15T13:24:48.910 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:24:48.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:48.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:24:49.015 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:24:49.026 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:24:49.026 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:24:49.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:48 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3608451138' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-05-15T13:24:49.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:48 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-05-15T13:24:49.052 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:49.077 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:24:49.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:49.081 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:24:49.182 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:24:49.209 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:49.228 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:24:49.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:49.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:24:49.332 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:24:49.360 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:49.752 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-05-15T13:24:49.814 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:24:49.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:49.817 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:24:49.918 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:24:49.945 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:49.963 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:24:49.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:49.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:24:50.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:49 smithi120 ceph-mon[43961]: pgmap v2207: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 7.2 GiB data, 30 GiB used, 864 GiB / 894 GiB avail; 127 B/s rd, 26 MiB/s wr, 614 op/s 2024-05-15T13:24:50.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:49 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-05-15T13:24:50.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:49 smithi120 ceph-mon[43961]: osdmap e854: 12 total, 12 up, 11 in 2024-05-15T13:24:50.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:49 smithi120 ceph-mon[43961]: osdmap e855: 12 total, 12 up, 11 in 2024-05-15T13:24:50.067 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:24:50.095 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:49 smithi177 ceph-mon[44303]: pgmap v2207: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 7.2 GiB data, 30 GiB used, 864 GiB / 894 GiB avail; 127 B/s rd, 26 MiB/s wr, 614 op/s 2024-05-15T13:24:50.095 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:49 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-05-15T13:24:50.095 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:49 smithi177 ceph-mon[44303]: osdmap e854: 12 total, 12 up, 11 in 2024-05-15T13:24:50.095 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:49 smithi177 ceph-mon[44303]: osdmap e855: 12 total, 12 up, 11 in 2024-05-15T13:24:50.097 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:50.102 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:24:50.120 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:24:50.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:50.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:24:50.225 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:24:50.253 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:50.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:49 smithi027 ceph-mon[39795]: pgmap v2207: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 7.2 GiB data, 30 GiB used, 864 GiB / 894 GiB avail; 127 B/s rd, 26 MiB/s wr, 614 op/s 2024-05-15T13:24:50.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:49 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-05-15T13:24:50.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:49 smithi027 ceph-mon[39795]: osdmap e854: 12 total, 12 up, 11 in 2024-05-15T13:24:50.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:49 smithi027 ceph-mon[39795]: osdmap e855: 12 total, 12 up, 11 in 2024-05-15T13:24:50.353 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:24:50.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:50.357 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:24:50.458 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:24:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:51.329 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:24:51.651 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:51 smithi177 ceph-mon[44303]: osdmap e856: 12 total, 12 up, 11 in 2024-05-15T13:24:51.651 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:51 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2431820610' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:24:51.651 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:51 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:24:51.651 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:51 smithi177 ceph-mon[44303]: pgmap v2211: 47 pgs: 1 remapped+peering, 2 active+clean+scrubbing+deep, 44 active+clean; 7.2 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 170 B/s rd, 2.9 MiB/s wr, 519 op/s 2024-05-15T13:24:51.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:51 smithi027 ceph-mon[39795]: osdmap e856: 12 total, 12 up, 11 in 2024-05-15T13:24:51.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:51 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2431820610' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:24:51.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:51 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:24:51.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:51 smithi027 ceph-mon[39795]: pgmap v2211: 47 pgs: 1 remapped+peering, 2 active+clean+scrubbing+deep, 44 active+clean; 7.2 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 170 B/s rd, 2.9 MiB/s wr, 519 op/s 2024-05-15T13:24:51.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:51 smithi120 ceph-mon[43961]: osdmap e856: 12 total, 12 up, 11 in 2024-05-15T13:24:51.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:51 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2431820610' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:24:51.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:51 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:24:51.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:51 smithi120 ceph-mon[43961]: pgmap v2211: 47 pgs: 1 remapped+peering, 2 active+clean+scrubbing+deep, 44 active+clean; 7.2 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 170 B/s rd, 2.9 MiB/s wr, 519 op/s 2024-05-15T13:24:52.138 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:24:52.138 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:24:52.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:52 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:24:52.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:52 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:24:52.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:52 smithi177 ceph-mon[44303]: osdmap e857: 12 total, 12 up, 11 in 2024-05-15T13:24:52.709 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:52 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:24:52.709 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:52 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:24:52.710 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:52 smithi027 ceph-mon[39795]: osdmap e857: 12 total, 12 up, 11 in 2024-05-15T13:24:52.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:52 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:24:52.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:52 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:24:52.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:52 smithi120 ceph-mon[43961]: osdmap e857: 12 total, 12 up, 11 in 2024-05-15T13:24:53.464 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:24:53.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:53.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:24:53.515 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-05-15T13:24:53.568 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:24:53.598 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:53 smithi027 ceph-mon[39795]: osdmap e858: 12 total, 12 up, 11 in 2024-05-15T13:24:53.598 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:53 smithi027 ceph-mon[39795]: pgmap v2214: 47 pgs: 1 remapped+peering, 3 active+undersized+remapped, 2 active+clean+scrubbing+deep, 41 active+clean; 7.1 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 437 op/s 2024-05-15T13:24:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:53.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:53 smithi177 ceph-mon[44303]: osdmap e858: 12 total, 12 up, 11 in 2024-05-15T13:24:53.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:53 smithi177 ceph-mon[44303]: pgmap v2214: 47 pgs: 1 remapped+peering, 3 active+undersized+remapped, 2 active+clean+scrubbing+deep, 41 active+clean; 7.1 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 437 op/s 2024-05-15T13:24:53.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:53 smithi120 ceph-mon[43961]: osdmap e858: 12 total, 12 up, 11 in 2024-05-15T13:24:53.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:53 smithi120 ceph-mon[43961]: pgmap v2214: 47 pgs: 1 remapped+peering, 3 active+undersized+remapped, 2 active+clean+scrubbing+deep, 41 active+clean; 7.1 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 437 op/s 2024-05-15T13:24:53.834 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:24:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:24:53.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:53.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:24:54.000 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-05-15T13:24:54.078 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:24:54.112 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:54.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:54.147 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:24:54.147 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:24:54.248 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:24:54.284 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:54.324 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:24:54.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:54.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:24:54.430 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:24:54.462 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:54.505 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:24:54.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:54.509 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:24:54.610 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:24:54.642 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:54.726 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:24:54.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:54.733 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:24:54.834 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:24:54.887 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:54.917 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:24:54.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:54.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:24:55.020 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:24:55.076 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:55.109 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:24:55.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:55.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:24:55.215 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:24:55.252 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:55.281 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:24:55.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:55.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:24:55.388 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:24:55.405 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:24:55.428 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:55.434 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:24:55.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:55 smithi177 ceph-mon[44303]: osdmap e859: 12 total, 12 up, 11 in 2024-05-15T13:24:55.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:55 smithi177 ceph-mon[44303]: pgmap v2216: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+remapped, 42 active+clean; 7.0 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 495 op/s; 125/345372 objects degraded (0.036%); 14305/345372 objects misplaced (4.142%); 1.4 MiB/s, 24 objects/s recovering 2024-05-15T13:24:55.455 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:24:55.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:55.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:24:55.464 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:24:55.499 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:24:55.526 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:24:55.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:55 smithi120 ceph-mon[43961]: osdmap e859: 12 total, 12 up, 11 in 2024-05-15T13:24:55.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:55 smithi120 ceph-mon[43961]: pgmap v2216: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+remapped, 42 active+clean; 7.0 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 495 op/s; 125/345372 objects degraded (0.036%); 14305/345372 objects misplaced (4.142%); 1.4 MiB/s, 24 objects/s recovering 2024-05-15T13:24:55.560 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:24:55.569 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:24:55.601 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:24:55.612 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:55.630 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:24:55.649 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:24:55.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:55.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:24:55.660 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:24:55.702 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:24:55.732 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:24:55.754 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:24:55.764 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:24:55.766 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:24:55.766 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:24:55.766 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:24:55.766 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:24:55.766 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:24:55.766 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:24:55.766 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:24:55.766 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:24:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:24:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:24:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:24:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:24:55.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:55 smithi027 ceph-mon[39795]: osdmap e859: 12 total, 12 up, 11 in 2024-05-15T13:24:55.780 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:55 smithi027 ceph-mon[39795]: pgmap v2216: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+remapped, 42 active+clean; 7.0 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 495 op/s; 125/345372 objects degraded (0.036%); 14305/345372 objects misplaced (4.142%); 1.4 MiB/s, 24 objects/s recovering 2024-05-15T13:24:55.804 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:55.829 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:24:55.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:55.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:24:55.841 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:24:55.841 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T13:24:55.841 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-05-15T13:24:55.841 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:24:55.936 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:24:55.978 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:56.013 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:24:56.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:56.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:24:56.118 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:24:56.170 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:56.211 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:24:56.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:56.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:24:56.315 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:24:56.356 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:56.558 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:56 smithi177 ceph-mon[44303]: Health check failed: Degraded data redundancy: 125/345372 objects degraded (0.036%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T13:24:56.558 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:56 smithi177 ceph-mon[44303]: from='client.16728 172.21.15.27:0/1668762887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:24:56.560 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:24:56.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:56.564 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:24:56.588 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-05-15T13:24:56.656 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:24:56.665 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:24:56.691 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:56 smithi027 ceph-mon[39795]: Health check failed: Degraded data redundancy: 125/345372 objects degraded (0.036%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T13:24:56.691 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:56 smithi027 ceph-mon[39795]: from='client.16728 172.21.15.27:0/1668762887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:24:56.703 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:56.731 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:24:56.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:56.738 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:24:56.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:56 smithi120 ceph-mon[43961]: Health check failed: Degraded data redundancy: 125/345372 objects degraded (0.036%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T13:24:56.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:56 smithi120 ceph-mon[43961]: from='client.16728 172.21.15.27:0/1668762887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:24:56.839 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:24:56.879 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:56.906 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:24:56.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:56.910 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:24:57.009 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:24:57.011 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:24:57.052 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:57.311 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:24:57.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:57.317 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:24:57.418 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:24:57.450 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:57.512 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:24:57.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:57.515 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:24:57.540 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:57 smithi177 ceph-mon[44303]: pgmap v2217: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 7.0 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 506 op/s; 125/342825 objects degraded (0.036%); 14305/342825 objects misplaced (4.173%); 1.0 MiB/s, 17 objects/s recovering 2024-05-15T13:24:57.616 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:24:57.647 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:57 smithi027 ceph-mon[39795]: pgmap v2217: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 7.0 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 506 op/s; 125/342825 objects degraded (0.036%); 14305/342825 objects misplaced (4.173%); 1.0 MiB/s, 17 objects/s recovering 2024-05-15T13:24:57.649 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:57.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:57 smithi120 ceph-mon[43961]: pgmap v2217: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 7.0 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 506 op/s; 125/342825 objects degraded (0.036%); 14305/342825 objects misplaced (4.173%); 1.0 MiB/s, 17 objects/s recovering 2024-05-15T13:24:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:24:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:58.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:24:58.540 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:24:58.600 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:24:58.628 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:58.727 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:24:58.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:58.735 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:24:58.836 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:24:58.888 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:58.915 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:24:58.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:58.918 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:24:59.000 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 9, 10, 11, 7, 4] out_osds: [8] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:24:59.001 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:24:59.001 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 5, 6, 9, 10, 11, 7, 4] 2024-05-15T13:24:59.001 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd out 1 2024-05-15T13:24:59.018 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:24:59.101 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:59.263 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:24:59.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:24:59.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:24:59.369 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:24:59.439 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:24:59.770 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:24:59 smithi027 ceph-mon[39795]: pgmap v2218: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 6.9 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 536 op/s; 328/338271 objects degraded (0.097%); 14029/338271 objects misplaced (4.147%); 1.4 MiB/s, 24 objects/s recovering 2024-05-15T13:24:59.818 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:24:59.818 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T13:24:59.818 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-05-15T13:24:59.819 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-05-15T13:24:59.819 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:25:00.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:24:59 smithi120 ceph-mon[43961]: pgmap v2218: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 6.9 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 536 op/s; 328/338271 objects degraded (0.097%); 14029/338271 objects misplaced (4.147%); 1.4 MiB/s, 24 objects/s recovering 2024-05-15T13:25:00.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:24:59 smithi177 ceph-mon[44303]: pgmap v2218: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 6.9 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 536 op/s; 328/338271 objects degraded (0.097%); 14029/338271 objects misplaced (4.147%); 1.4 MiB/s, 24 objects/s recovering 2024-05-15T13:25:00.427 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:25:00.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:00.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:25:00.528 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:25:00.536 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:25:00.580 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-05-15T13:25:00.605 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:00.823 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:00 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 519/333936 objects degraded (0.155%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T13:25:00.823 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:00 smithi177 ceph-mon[44303]: osdmap e860: 12 total, 12 up, 11 in 2024-05-15T13:25:00.925 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:25:01.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:00 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 519/333936 objects degraded (0.155%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T13:25:01.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:00 smithi027 ceph-mon[39795]: osdmap e860: 12 total, 12 up, 11 in 2024-05-15T13:25:01.085 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:25:01.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:01.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:25:01.143 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:25:01.196 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:25:01.228 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:01.263 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:25:01.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:01.267 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:25:01.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:00 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 519/333936 objects degraded (0.155%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T13:25:01.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:00 smithi120 ceph-mon[43961]: osdmap e860: 12 total, 12 up, 11 in 2024-05-15T13:25:01.367 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:25:01.413 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:01.455 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:25:01.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:01.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:25:01.561 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:25:01.647 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:02.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:01 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:25:02.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:01 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:25:02.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:01 smithi027 ceph-mon[39795]: pgmap v2219: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 6.8 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 616 op/s; 519/333936 objects degraded (0.155%); 13768/333936 objects misplaced (4.123%); 1.8 MiB/s, 29 objects/s recovering 2024-05-15T13:25:02.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:01 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:25:02.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:01 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:25:02.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:01 smithi177 ceph-mon[44303]: pgmap v2219: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 6.8 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 616 op/s; 519/333936 objects degraded (0.155%); 13768/333936 objects misplaced (4.123%); 1.8 MiB/s, 29 objects/s recovering 2024-05-15T13:25:02.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:01 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:25:02.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:01 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:25:02.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:01 smithi120 ceph-mon[43961]: pgmap v2219: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 6.8 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 616 op/s; 519/333936 objects degraded (0.155%); 13768/333936 objects misplaced (4.123%); 1.8 MiB/s, 29 objects/s recovering 2024-05-15T13:25:02.405 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:25:02.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:02.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:25:02.427 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:25:02.515 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:25:02.556 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:02.814 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:25:02.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:02.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:25:02.923 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:25:02.960 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:02.993 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:25:03.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:03.007 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:25:03.108 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:25:03.180 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:03.293 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:25:03.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:03.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:25:03.414 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:25:03.453 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:03.775 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:25:03.775 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-05-15T13:25:03.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-05-15T13:25:03.775 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T13:25:03.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.775 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:143291 7.a 7:5365d091:::benchmark_data_smithi177_134484_object34226:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e672)", 2024-05-15T13:25:03.775 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:33.117878+0000", 2024-05-15T13:25:03.775 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 570.65422477200002, 2024-05-15T13:25:03.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21058098, 2024-05-15T13:25:03.776 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:25:03.776 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:25:03.776 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:25:03.776 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:25:03.776 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:25:03.776 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 143291 2024-05-15T13:25:03.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.776 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:25:03.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:25:03.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.117878+0000", 2024-05-15T13:25:03.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:25:03.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.117878+0000", 2024-05-15T13:25:03.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:25:03.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.117880+0000", 2024-05-15T13:25:03.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.702e-06 2024-05-15T13:25:03.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:25:03.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.117887+0000", 2024-05-15T13:25:03.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6070000000000002e-06 2024-05-15T13:25:03.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:25:03.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.117902+0000", 2024-05-15T13:25:03.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5211999999999999e-05 2024-05-15T13:25:03.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:25:03.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.117911+0000", 2024-05-15T13:25:03.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9900000000000003e-06 2024-05-15T13:25:03.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:25:03.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.117970+0000", 2024-05-15T13:25:03.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8576999999999998e-05 2024-05-15T13:25:03.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:25:03.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.118102+0000", 2024-05-15T13:25:03.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000132046 2024-05-15T13:25:03.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,11", 2024-05-15T13:25:03.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.118223+0000", 2024-05-15T13:25:03.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120726 2024-05-15T13:25:03.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:25:03.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.121922+0000", 2024-05-15T13:25:03.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003699529 2024-05-15T13:25:03.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.130040+0000", 2024-05-15T13:25:03.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0081178489999999999 2024-05-15T13:25:03.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.328385+0000", 2024-05-15T13:25:03.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19834512300000001 2024-05-15T13:25:03.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:25:03.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.328423+0000", 2024-05-15T13:25:03.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8148999999999997e-05 2024-05-15T13:25:03.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:25:03.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.328459+0000", 2024-05-15T13:25:03.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5469999999999998e-05 2024-05-15T13:25:03.781 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.781 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:25:03.781 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.781 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26044.0:143293 7.a 7:59b1bca6:::benchmark_data_smithi177_134484_object34228:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e672)", 2024-05-15T13:25:03.781 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:15:33.123026+0000", 2024-05-15T13:25:03.781 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 570.64907634099995, 2024-05-15T13:25:03.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21120455499999999, 2024-05-15T13:25:03.781 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:25:03.781 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:25:03.782 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:25:03.782 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26044", 2024-05-15T13:25:03.782 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:25:03.782 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 143293 2024-05-15T13:25:03.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.782 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:25:03.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:25:03.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.123026+0000", 2024-05-15T13:25:03.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:25:03.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.123026+0000", 2024-05-15T13:25:03.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:25:03.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.123040+0000", 2024-05-15T13:25:03.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4173e-05 2024-05-15T13:25:03.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:25:03.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.123047+0000", 2024-05-15T13:25:03.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.229e-06 2024-05-15T13:25:03.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:25:03.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.123062+0000", 2024-05-15T13:25:03.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5783000000000001e-05 2024-05-15T13:25:03.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:25:03.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.123071+0000", 2024-05-15T13:25:03.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7479999999999999e-06 2024-05-15T13:25:03.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:25:03.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.123129+0000", 2024-05-15T13:25:03.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7324999999999997e-05 2024-05-15T13:25:03.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:25:03.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.123258+0000", 2024-05-15T13:25:03.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012974199999999999 2024-05-15T13:25:03.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 2,11", 2024-05-15T13:25:03.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.123379+0000", 2024-05-15T13:25:03.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120835 2024-05-15T13:25:03.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:25:03.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.127696+0000", 2024-05-15T13:25:03.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0043172130000000003 2024-05-15T13:25:03.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.130347+0000", 2024-05-15T13:25:03.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0026510650000000002 2024-05-15T13:25:03.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.334144+0000", 2024-05-15T13:25:03.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20379657400000001 2024-05-15T13:25:03.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.786 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:25:03.786 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.334197+0000", 2024-05-15T13:25:03.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3368000000000001e-05 2024-05-15T13:25:03.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:25:03.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:15:33.334231+0000", 2024-05-15T13:25:03.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3500000000000001e-05 2024-05-15T13:25:03.787 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.787 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:25:03.787 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.787 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:36713 9.3 9:cbaa933d:::benchmark_data_smithi177_159919_object36712:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-05-15T13:25:03.787 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:23:38.230573+0000", 2024-05-15T13:25:03.787 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 85.541529252000004, 2024-05-15T13:25:03.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20847062899999999, 2024-05-15T13:25:03.787 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:25:03.788 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:25:03.788 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:25:03.788 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:25:03.788 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:25:03.788 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 36713 2024-05-15T13:25:03.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.788 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:25:03.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:25:03.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.230573+0000", 2024-05-15T13:25:03.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:25:03.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.230573+0000", 2024-05-15T13:25:03.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:25:03.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.230574+0000", 2024-05-15T13:25:03.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.54e-07 2024-05-15T13:25:03.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:25:03.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.230594+0000", 2024-05-15T13:25:03.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9775e-05 2024-05-15T13:25:03.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:25:03.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.230603+0000", 2024-05-15T13:25:03.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8279999999999992e-06 2024-05-15T13:25:03.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:25:03.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.230609+0000", 2024-05-15T13:25:03.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7460000000000003e-06 2024-05-15T13:25:03.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:25:03.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.230741+0000", 2024-05-15T13:25:03.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000131478 2024-05-15T13:25:03.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:25:03.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.230863+0000", 2024-05-15T13:25:03.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012192 2024-05-15T13:25:03.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,5", 2024-05-15T13:25:03.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.230987+0000", 2024-05-15T13:25:03.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000124292 2024-05-15T13:25:03.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:25:03.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.233788+0000", 2024-05-15T13:25:03.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002801132 2024-05-15T13:25:03.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.234238+0000", 2024-05-15T13:25:03.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00044965899999999999 2024-05-15T13:25:03.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.438929+0000", 2024-05-15T13:25:03.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20469066 2024-05-15T13:25:03.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:25:03.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.438995+0000", 2024-05-15T13:25:03.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6357000000000002e-05 2024-05-15T13:25:03.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:25:03.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.439044+0000", 2024-05-15T13:25:03.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9027999999999999e-05 2024-05-15T13:25:03.793 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.793 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:25:03.793 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.793 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:36715 9.3 9:c5fd2772:::benchmark_data_smithi177_159919_object36714:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-05-15T13:25:03.793 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:23:38.230626+0000", 2024-05-15T13:25:03.793 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 85.541476203000002, 2024-05-15T13:25:03.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.208550286, 2024-05-15T13:25:03.793 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:25:03.793 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:25:03.794 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:25:03.794 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:25:03.794 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:25:03.794 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 36715 2024-05-15T13:25:03.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.794 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:25:03.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:25:03.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.230626+0000", 2024-05-15T13:25:03.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:25:03.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.230626+0000", 2024-05-15T13:25:03.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:25:03.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.230627+0000", 2024-05-15T13:25:03.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5499999999999999e-07 2024-05-15T13:25:03.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:25:03.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.230662+0000", 2024-05-15T13:25:03.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4799999999999999e-05 2024-05-15T13:25:03.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:25:03.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.230667+0000", 2024-05-15T13:25:03.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1490000000000002e-06 2024-05-15T13:25:03.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:25:03.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.230671+0000", 2024-05-15T13:25:03.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9400000000000004e-06 2024-05-15T13:25:03.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:25:03.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.232228+0000", 2024-05-15T13:25:03.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001557016 2024-05-15T13:25:03.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:25:03.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.232378+0000", 2024-05-15T13:25:03.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014990199999999999 2024-05-15T13:25:03.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,5", 2024-05-15T13:25:03.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.232521+0000", 2024-05-15T13:25:03.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000143743 2024-05-15T13:25:03.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:25:03.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.233793+0000", 2024-05-15T13:25:03.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012721799999999999 2024-05-15T13:25:03.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.234334+0000", 2024-05-15T13:25:03.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00054032699999999995 2024-05-15T13:25:03.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.439085+0000", 2024-05-15T13:25:03.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20475091400000001 2024-05-15T13:25:03.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:25:03.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.439142+0000", 2024-05-15T13:25:03.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7408999999999999e-05 2024-05-15T13:25:03.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:25:03.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.439177+0000", 2024-05-15T13:25:03.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4551000000000002e-05 2024-05-15T13:25:03.798 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.799 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:25:03.799 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.799 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:36727 9.3 9:ca43e0e6:::benchmark_data_smithi177_159919_object36726:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-05-15T13:25:03.799 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:23:38.235964+0000", 2024-05-15T13:25:03.799 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 85.536138874000002, 2024-05-15T13:25:03.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.203335341, 2024-05-15T13:25:03.799 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:25:03.799 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:25:03.799 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:25:03.799 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:25:03.799 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:25:03.799 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 36727 2024-05-15T13:25:03.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.799 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:25:03.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:25:03.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.235964+0000", 2024-05-15T13:25:03.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:25:03.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.235964+0000", 2024-05-15T13:25:03.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:25:03.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.235965+0000", 2024-05-15T13:25:03.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.54e-07 2024-05-15T13:25:03.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:25:03.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.236014+0000", 2024-05-15T13:25:03.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8965999999999998e-05 2024-05-15T13:25:03.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:25:03.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.236024+0000", 2024-05-15T13:25:03.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.023e-05 2024-05-15T13:25:03.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.801 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:25:03.801 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.236031+0000", 2024-05-15T13:25:03.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9639999999999997e-06 2024-05-15T13:25:03.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:25:03.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.236078+0000", 2024-05-15T13:25:03.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7576e-05 2024-05-15T13:25:03.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:25:03.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.236208+0000", 2024-05-15T13:25:03.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000130075 2024-05-15T13:25:03.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,5", 2024-05-15T13:25:03.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.236329+0000", 2024-05-15T13:25:03.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012106 2024-05-15T13:25:03.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:25:03.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.239977+0000", 2024-05-15T13:25:03.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0036474260000000001 2024-05-15T13:25:03.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.240012+0000", 2024-05-15T13:25:03.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5181000000000001e-05 2024-05-15T13:25:03.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.439213+0000", 2024-05-15T13:25:03.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19920136399999999 2024-05-15T13:25:03.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:25:03.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.439263+0000", 2024-05-15T13:25:03.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9815999999999999e-05 2024-05-15T13:25:03.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:25:03.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.439299+0000", 2024-05-15T13:25:03.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5828999999999999e-05 2024-05-15T13:25:03.804 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.804 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:25:03.804 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.805 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:36728 9.3 9:c8284c38:::benchmark_data_smithi177_159919_object36727:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-05-15T13:25:03.805 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:23:38.236064+0000", 2024-05-15T13:25:03.805 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 85.536038959999999, 2024-05-15T13:25:03.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20449136000000001, 2024-05-15T13:25:03.805 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:25:03.805 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:25:03.805 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:25:03.805 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:25:03.805 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:25:03.805 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 36728 2024-05-15T13:25:03.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.805 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:25:03.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:25:03.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.236064+0000", 2024-05-15T13:25:03.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:25:03.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.236064+0000", 2024-05-15T13:25:03.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:25:03.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.236064+0000", 2024-05-15T13:25:03.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7599999999999997e-07 2024-05-15T13:25:03.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:25:03.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.236106+0000", 2024-05-15T13:25:03.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1817000000000003e-05 2024-05-15T13:25:03.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:25:03.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.236114+0000", 2024-05-15T13:25:03.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3359999999999995e-06 2024-05-15T13:25:03.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:25:03.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.236121+0000", 2024-05-15T13:25:03.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8079999999999999e-06 2024-05-15T13:25:03.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:25:03.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.237455+0000", 2024-05-15T13:25:03.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001334268 2024-05-15T13:25:03.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:25:03.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.237587+0000", 2024-05-15T13:25:03.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013109600000000001 2024-05-15T13:25:03.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,5", 2024-05-15T13:25:03.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.237723+0000", 2024-05-15T13:25:03.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000136935 2024-05-15T13:25:03.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:25:03.808 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.240024+0000", 2024-05-15T13:25:03.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023009710000000002 2024-05-15T13:25:03.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.240234+0000", 2024-05-15T13:25:03.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000209884 2024-05-15T13:25:03.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.440465+0000", 2024-05-15T13:25:03.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20023062899999999 2024-05-15T13:25:03.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:25:03.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.440510+0000", 2024-05-15T13:25:03.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5491000000000002e-05 2024-05-15T13:25:03.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:25:03.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.440555+0000", 2024-05-15T13:25:03.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4549000000000003e-05 2024-05-15T13:25:03.810 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.810 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:25:03.810 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.810 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:36731 9.3 9:c7966a29:::benchmark_data_smithi177_159919_object36730:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-05-15T13:25:03.810 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:23:38.238104+0000", 2024-05-15T13:25:03.810 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 85.533998232000002, 2024-05-15T13:25:03.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20460542400000001, 2024-05-15T13:25:03.810 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:25:03.811 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:25:03.811 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:25:03.811 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:25:03.811 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:25:03.811 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 36731 2024-05-15T13:25:03.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.811 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:25:03.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:25:03.811 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.238104+0000", 2024-05-15T13:25:03.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:25:03.811 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.238104+0000", 2024-05-15T13:25:03.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:25:03.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.238105+0000", 2024-05-15T13:25:03.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7999999999999993e-07 2024-05-15T13:25:03.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:25:03.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.238158+0000", 2024-05-15T13:25:03.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2475999999999997e-05 2024-05-15T13:25:03.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:25:03.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.238170+0000", 2024-05-15T13:25:03.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.171e-05 2024-05-15T13:25:03.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:25:03.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.238177+0000", 2024-05-15T13:25:03.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9489999999999996e-06 2024-05-15T13:25:03.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:25:03.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.238794+0000", 2024-05-15T13:25:03.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00061624200000000003 2024-05-15T13:25:03.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:25:03.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.238921+0000", 2024-05-15T13:25:03.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012677700000000001 2024-05-15T13:25:03.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,5", 2024-05-15T13:25:03.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.239042+0000", 2024-05-15T13:25:03.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012154099999999999 2024-05-15T13:25:03.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:25:03.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.240079+0000", 2024-05-15T13:25:03.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010371919999999999 2024-05-15T13:25:03.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.240307+0000", 2024-05-15T13:25:03.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022725499999999999 2024-05-15T13:25:03.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.442627+0000", 2024-05-15T13:25:03.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20232092600000001 2024-05-15T13:25:03.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:25:03.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.442670+0000", 2024-05-15T13:25:03.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3037999999999997e-05 2024-05-15T13:25:03.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:25:03.815 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.442710+0000", 2024-05-15T13:25:03.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9338000000000002e-05 2024-05-15T13:25:03.816 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.816 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:25:03.816 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.816 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:36740 9.3 9:cd844683:::benchmark_data_smithi177_159919_object36739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-05-15T13:25:03.816 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:23:38.242986+0000", 2024-05-15T13:25:03.816 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 85.529116806999994, 2024-05-15T13:25:03.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19987716799999999, 2024-05-15T13:25:03.816 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:25:03.816 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:25:03.816 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:25:03.816 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:25:03.816 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:25:03.816 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 36740 2024-05-15T13:25:03.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.817 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:25:03.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:25:03.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.242986+0000", 2024-05-15T13:25:03.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:25:03.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.242986+0000", 2024-05-15T13:25:03.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:25:03.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.242987+0000", 2024-05-15T13:25:03.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.35e-06 2024-05-15T13:25:03.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:25:03.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.243012+0000", 2024-05-15T13:25:03.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4641e-05 2024-05-15T13:25:03.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:25:03.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.243025+0000", 2024-05-15T13:25:03.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3056e-05 2024-05-15T13:25:03.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:25:03.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.243031+0000", 2024-05-15T13:25:03.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3799999999999999e-06 2024-05-15T13:25:03.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:25:03.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.243089+0000", 2024-05-15T13:25:03.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7521e-05 2024-05-15T13:25:03.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:25:03.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.243227+0000", 2024-05-15T13:25:03.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000138486 2024-05-15T13:25:03.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,5", 2024-05-15T13:25:03.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.243348+0000", 2024-05-15T13:25:03.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120796 2024-05-15T13:25:03.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:25:03.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.244473+0000", 2024-05-15T13:25:03.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011249599999999999 2024-05-15T13:25:03.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.247121+0000", 2024-05-15T13:25:03.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0026478539999999998 2024-05-15T13:25:03.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.442763+0000", 2024-05-15T13:25:03.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19564214699999999 2024-05-15T13:25:03.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:25:03.821 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.442819+0000", 2024-05-15T13:25:03.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5670000000000001e-05 2024-05-15T13:25:03.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.821 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:25:03.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.442863+0000", 2024-05-15T13:25:03.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4307000000000001e-05 2024-05-15T13:25:03.822 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.822 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:25:03.822 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.822 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:36746 9.3 9:c8000dda:::benchmark_data_smithi177_159919_object36745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-05-15T13:25:03.822 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:23:38.245734+0000", 2024-05-15T13:25:03.822 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 85.526368304000002, 2024-05-15T13:25:03.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19730014300000001, 2024-05-15T13:25:03.822 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:25:03.822 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:25:03.823 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:25:03.823 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:25:03.823 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:25:03.823 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 36746 2024-05-15T13:25:03.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.823 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:25:03.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:25:03.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.245734+0000", 2024-05-15T13:25:03.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:25:03.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.245734+0000", 2024-05-15T13:25:03.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:25:03.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.245735+0000", 2024-05-15T13:25:03.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.879999999999999e-07 2024-05-15T13:25:03.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:25:03.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.245755+0000", 2024-05-15T13:25:03.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9786999999999999e-05 2024-05-15T13:25:03.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:25:03.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.245766+0000", 2024-05-15T13:25:03.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0614e-05 2024-05-15T13:25:03.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:25:03.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.245774+0000", 2024-05-15T13:25:03.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3799999999999994e-06 2024-05-15T13:25:03.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:25:03.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.245825+0000", 2024-05-15T13:25:03.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0659999999999999e-05 2024-05-15T13:25:03.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:25:03.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.245928+0000", 2024-05-15T13:25:03.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103198 2024-05-15T13:25:03.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,5", 2024-05-15T13:25:03.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.246019+0000", 2024-05-15T13:25:03.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0698000000000005e-05 2024-05-15T13:25:03.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.247243+0000", 2024-05-15T13:25:03.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001223954 2024-05-15T13:25:03.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:25:03.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.247258+0000", 2024-05-15T13:25:03.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4948e-05 2024-05-15T13:25:03.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.442939+0000", 2024-05-15T13:25:03.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.195681356 2024-05-15T13:25:03.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:25:03.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.442994+0000", 2024-05-15T13:25:03.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4697999999999997e-05 2024-05-15T13:25:03.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.827 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:25:03.827 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.443034+0000", 2024-05-15T13:25:03.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0862000000000003e-05 2024-05-15T13:25:03.828 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.828 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:25:03.828 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.828 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:36757 9.3 9:cbf650f4:::benchmark_data_smithi177_159919_object36756:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-05-15T13:25:03.828 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:23:38.253555+0000", 2024-05-15T13:25:03.828 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 85.518548061999994, 2024-05-15T13:25:03.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18962374300000001, 2024-05-15T13:25:03.828 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:25:03.828 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:25:03.828 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:25:03.828 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:25:03.828 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:25:03.828 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 36757 2024-05-15T13:25:03.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.829 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:25:03.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:25:03.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.253555+0000", 2024-05-15T13:25:03.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:25:03.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.253555+0000", 2024-05-15T13:25:03.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:25:03.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.253556+0000", 2024-05-15T13:25:03.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3909999999999999e-06 2024-05-15T13:25:03.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:25:03.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.253579+0000", 2024-05-15T13:25:03.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.349e-05 2024-05-15T13:25:03.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:25:03.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.253592+0000", 2024-05-15T13:25:03.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2376e-05 2024-05-15T13:25:03.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:25:03.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.253598+0000", 2024-05-15T13:25:03.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6599999999999998e-06 2024-05-15T13:25:03.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:25:03.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.253653+0000", 2024-05-15T13:25:03.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4231000000000003e-05 2024-05-15T13:25:03.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:25:03.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.253817+0000", 2024-05-15T13:25:03.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016412299999999999 2024-05-15T13:25:03.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,5", 2024-05-15T13:25:03.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.253971+0000", 2024-05-15T13:25:03.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015406099999999999 2024-05-15T13:25:03.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:25:03.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.255164+0000", 2024-05-15T13:25:03.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011927979999999999 2024-05-15T13:25:03.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.255492+0000", 2024-05-15T13:25:03.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00032830000000000001 2024-05-15T13:25:03.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.443087+0000", 2024-05-15T13:25:03.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18759493499999999 2024-05-15T13:25:03.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:25:03.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.443138+0000", 2024-05-15T13:25:03.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1270999999999998e-05 2024-05-15T13:25:03.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.833 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:25:03.833 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.443178+0000", 2024-05-15T13:25:03.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0107000000000001e-05 2024-05-15T13:25:03.833 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.834 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:25:03.834 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.834 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:36760 9.3 9:c31d8860:::benchmark_data_smithi177_159919_object36759:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-05-15T13:25:03.834 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:23:38.256044+0000", 2024-05-15T13:25:03.834 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 85.516058821000001, 2024-05-15T13:25:03.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.187325252, 2024-05-15T13:25:03.834 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:25:03.834 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:25:03.834 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:25:03.834 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:25:03.834 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:25:03.834 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 36760 2024-05-15T13:25:03.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.835 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:25:03.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:25:03.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.256044+0000", 2024-05-15T13:25:03.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:25:03.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.256044+0000", 2024-05-15T13:25:03.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:25:03.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.256045+0000", 2024-05-15T13:25:03.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0440000000000001e-06 2024-05-15T13:25:03.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:25:03.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.256067+0000", 2024-05-15T13:25:03.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.173e-05 2024-05-15T13:25:03.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:25:03.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.256077+0000", 2024-05-15T13:25:03.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0730999999999999e-05 2024-05-15T13:25:03.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:25:03.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.256085+0000", 2024-05-15T13:25:03.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7689999999999996e-06 2024-05-15T13:25:03.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:25:03.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.256126+0000", 2024-05-15T13:25:03.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0861000000000001e-05 2024-05-15T13:25:03.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:25:03.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.256230+0000", 2024-05-15T13:25:03.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104319 2024-05-15T13:25:03.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,5", 2024-05-15T13:25:03.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.256319+0000", 2024-05-15T13:25:03.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8943000000000005e-05 2024-05-15T13:25:03.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:25:03.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.257325+0000", 2024-05-15T13:25:03.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010060399999999999 2024-05-15T13:25:03.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.257393+0000", 2024-05-15T13:25:03.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7397999999999994e-05 2024-05-15T13:25:03.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.443280+0000", 2024-05-15T13:25:03.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.185887213 2024-05-15T13:25:03.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.839 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:25:03.839 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.443333+0000", 2024-05-15T13:25:03.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2963000000000001e-05 2024-05-15T13:25:03.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.839 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:25:03.839 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:38.443369+0000", 2024-05-15T13:25:03.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6241000000000001e-05 2024-05-15T13:25:03.839 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.839 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:25:03.839 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.839 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:69282 9.8 9:18611e28:::benchmark_data_smithi177_159919_object69281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e830)", 2024-05-15T13:25:03.840 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:23:56.662084+0000", 2024-05-15T13:25:03.840 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 67.110018171999997, 2024-05-15T13:25:03.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.206657958, 2024-05-15T13:25:03.840 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:25:03.840 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:25:03.840 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:25:03.840 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:25:03.840 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:25:03.840 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 69282 2024-05-15T13:25:03.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.840 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:25:03.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:25:03.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.662084+0000", 2024-05-15T13:25:03.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:25:03.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.662084+0000", 2024-05-15T13:25:03.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:25:03.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.662086+0000", 2024-05-15T13:25:03.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.46e-06 2024-05-15T13:25:03.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:25:03.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.662112+0000", 2024-05-15T13:25:03.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5737000000000001e-05 2024-05-15T13:25:03.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:25:03.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.662124+0000", 2024-05-15T13:25:03.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1939999999999999e-05 2024-05-15T13:25:03.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:25:03.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.662132+0000", 2024-05-15T13:25:03.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6829999999999997e-06 2024-05-15T13:25:03.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:25:03.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.662177+0000", 2024-05-15T13:25:03.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4428999999999998e-05 2024-05-15T13:25:03.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:25:03.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.662317+0000", 2024-05-15T13:25:03.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014003600000000001 2024-05-15T13:25:03.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,11", 2024-05-15T13:25:03.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.662480+0000", 2024-05-15T13:25:03.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000163578 2024-05-15T13:25:03.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:25:03.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.666884+0000", 2024-05-15T13:25:03.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0044037859999999998 2024-05-15T13:25:03.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.669336+0000", 2024-05-15T13:25:03.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024515050000000001 2024-05-15T13:25:03.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.868649+0000", 2024-05-15T13:25:03.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19931332299999999 2024-05-15T13:25:03.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:25:03.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.868695+0000", 2024-05-15T13:25:03.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5643999999999999e-05 2024-05-15T13:25:03.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:25:03.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.868742+0000", 2024-05-15T13:25:03.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7837000000000003e-05 2024-05-15T13:25:03.845 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.845 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:25:03.845 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.845 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:69283 9.8 9:16f9d400:::benchmark_data_smithi177_159919_object69282:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e830)", 2024-05-15T13:25:03.845 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:23:56.662148+0000", 2024-05-15T13:25:03.845 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 67.109954732000006, 2024-05-15T13:25:03.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21156636500000001, 2024-05-15T13:25:03.846 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:25:03.846 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:25:03.846 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:25:03.846 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:25:03.846 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:25:03.846 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 69283 2024-05-15T13:25:03.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.846 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:25:03.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:25:03.846 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.662148+0000", 2024-05-15T13:25:03.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:25:03.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.662148+0000", 2024-05-15T13:25:03.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:25:03.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.662148+0000", 2024-05-15T13:25:03.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0200000000000002e-07 2024-05-15T13:25:03.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:25:03.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.662172+0000", 2024-05-15T13:25:03.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3467000000000001e-05 2024-05-15T13:25:03.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:25:03.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.662180+0000", 2024-05-15T13:25:03.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8879999999999997e-06 2024-05-15T13:25:03.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:25:03.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.662186+0000", 2024-05-15T13:25:03.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0909999999999998e-06 2024-05-15T13:25:03.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:25:03.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.664412+0000", 2024-05-15T13:25:03.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002226207 2024-05-15T13:25:03.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:25:03.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.664541+0000", 2024-05-15T13:25:03.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000129286 2024-05-15T13:25:03.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,11", 2024-05-15T13:25:03.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.664676+0000", 2024-05-15T13:25:03.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013491600000000001 2024-05-15T13:25:03.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.669385+0000", 2024-05-15T13:25:03.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0047089389999999997 2024-05-15T13:25:03.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:25:03.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.669618+0000", 2024-05-15T13:25:03.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023246200000000001 2024-05-15T13:25:03.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.873609+0000", 2024-05-15T13:25:03.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20399165499999999 2024-05-15T13:25:03.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:25:03.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.873668+0000", 2024-05-15T13:25:03.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9181999999999997e-05 2024-05-15T13:25:03.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:25:03.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.873714+0000", 2024-05-15T13:25:03.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5769999999999997e-05 2024-05-15T13:25:03.851 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.851 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:25:03.851 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.851 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:69285 9.8 9:130d1b58:::benchmark_data_smithi177_159919_object69284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e830)", 2024-05-15T13:25:03.851 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:23:56.663630+0000", 2024-05-15T13:25:03.851 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 67.108472992000003, 2024-05-15T13:25:03.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21388459300000001, 2024-05-15T13:25:03.851 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:25:03.852 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:25:03.852 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:25:03.852 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:25:03.852 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:25:03.852 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 69285 2024-05-15T13:25:03.852 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.852 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:25:03.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.852 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:25:03.852 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.663630+0000", 2024-05-15T13:25:03.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.852 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.852 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:25:03.853 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.663630+0000", 2024-05-15T13:25:03.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.853 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:25:03.853 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.663630+0000", 2024-05-15T13:25:03.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5300000000000003e-07 2024-05-15T13:25:03.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.853 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:25:03.853 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.663651+0000", 2024-05-15T13:25:03.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0449e-05 2024-05-15T13:25:03.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.853 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:25:03.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.663665+0000", 2024-05-15T13:25:03.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.448e-05 2024-05-15T13:25:03.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:25:03.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.663673+0000", 2024-05-15T13:25:03.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9710000000000004e-06 2024-05-15T13:25:03.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:25:03.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.666921+0000", 2024-05-15T13:25:03.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0032472220000000001 2024-05-15T13:25:03.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:25:03.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.667081+0000", 2024-05-15T13:25:03.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016043900000000001 2024-05-15T13:25:03.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,11", 2024-05-15T13:25:03.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.667254+0000", 2024-05-15T13:25:03.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017274900000000001 2024-05-15T13:25:03.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:25:03.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.669842+0000", 2024-05-15T13:25:03.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00258866 2024-05-15T13:25:03.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.670002+0000", 2024-05-15T13:25:03.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015945 2024-05-15T13:25:03.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.877435+0000", 2024-05-15T13:25:03.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20743344399999999 2024-05-15T13:25:03.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:25:03.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.877471+0000", 2024-05-15T13:25:03.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6142000000000001e-05 2024-05-15T13:25:03.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:25:03.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.877514+0000", 2024-05-15T13:25:03.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2734000000000001e-05 2024-05-15T13:25:03.857 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.857 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:25:03.857 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.857 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:69301 9.8 9:10607aa7:::benchmark_data_smithi177_159919_object69300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e830)", 2024-05-15T13:25:03.857 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:23:56.676170+0000", 2024-05-15T13:25:03.857 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 67.095932558000001, 2024-05-15T13:25:03.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20151471200000001, 2024-05-15T13:25:03.857 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:25:03.858 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:25:03.858 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:25:03.858 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:25:03.858 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:25:03.858 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 69301 2024-05-15T13:25:03.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.858 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:25:03.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:25:03.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.676170+0000", 2024-05-15T13:25:03.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:25:03.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.676170+0000", 2024-05-15T13:25:03.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:25:03.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.676172+0000", 2024-05-15T13:25:03.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9319999999999999e-06 2024-05-15T13:25:03.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:25:03.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.676192+0000", 2024-05-15T13:25:03.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9522e-05 2024-05-15T13:25:03.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:25:03.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.676202+0000", 2024-05-15T13:25:03.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.081e-05 2024-05-15T13:25:03.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:25:03.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.676209+0000", 2024-05-15T13:25:03.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2269999999999998e-06 2024-05-15T13:25:03.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:25:03.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.676252+0000", 2024-05-15T13:25:03.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3182000000000001e-05 2024-05-15T13:25:03.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:25:03.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.676367+0000", 2024-05-15T13:25:03.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011527399999999999 2024-05-15T13:25:03.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,11", 2024-05-15T13:25:03.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.676467+0000", 2024-05-15T13:25:03.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9563999999999997e-05 2024-05-15T13:25:03.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:25:03.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.678559+0000", 2024-05-15T13:25:03.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002092923 2024-05-15T13:25:03.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.679842+0000", 2024-05-15T13:25:03.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001282313 2024-05-15T13:25:03.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.877575+0000", 2024-05-15T13:25:03.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19773342399999999 2024-05-15T13:25:03.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:25:03.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.877639+0000", 2024-05-15T13:25:03.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4018999999999998e-05 2024-05-15T13:25:03.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:25:03.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.877685+0000", 2024-05-15T13:25:03.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5522000000000002e-05 2024-05-15T13:25:03.863 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.863 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:25:03.863 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.863 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:69328 9.8 9:1c8df94f:::benchmark_data_smithi177_159919_object69327:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e830)", 2024-05-15T13:25:03.863 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:23:56.694032+0000", 2024-05-15T13:25:03.863 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 67.078070156999999, 2024-05-15T13:25:03.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.184262551, 2024-05-15T13:25:03.863 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:25:03.864 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:25:03.864 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:25:03.864 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:25:03.864 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:25:03.864 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 69328 2024-05-15T13:25:03.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.864 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:25:03.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:25:03.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.694032+0000", 2024-05-15T13:25:03.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:25:03.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.694032+0000", 2024-05-15T13:25:03.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:25:03.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.694034+0000", 2024-05-15T13:25:03.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4610000000000001e-06 2024-05-15T13:25:03.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:25:03.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.694056+0000", 2024-05-15T13:25:03.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1608e-05 2024-05-15T13:25:03.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:25:03.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.694068+0000", 2024-05-15T13:25:03.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2425e-05 2024-05-15T13:25:03.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:25:03.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.694075+0000", 2024-05-15T13:25:03.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2930000000000003e-06 2024-05-15T13:25:03.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:25:03.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.694129+0000", 2024-05-15T13:25:03.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3835000000000002e-05 2024-05-15T13:25:03.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:25:03.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.694295+0000", 2024-05-15T13:25:03.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000166002 2024-05-15T13:25:03.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,11", 2024-05-15T13:25:03.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.694483+0000", 2024-05-15T13:25:03.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000187496 2024-05-15T13:25:03.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:25:03.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.701671+0000", 2024-05-15T13:25:03.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0071880090000000004 2024-05-15T13:25:03.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.701705+0000", 2024-05-15T13:25:03.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4019000000000001e-05 2024-05-15T13:25:03.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.878197+0000", 2024-05-15T13:25:03.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17649258900000001 2024-05-15T13:25:03.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:25:03.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.878253+0000", 2024-05-15T13:25:03.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5577e-05 2024-05-15T13:25:03.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:25:03.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:56.878295+0000", 2024-05-15T13:25:03.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2237000000000002e-05 2024-05-15T13:25:03.869 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.869 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:25:03.869 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.869 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:97031 9.8 9:11bc201b:::benchmark_data_smithi177_159919_object97030:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e842)", 2024-05-15T13:25:03.869 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:24:17.493235+0000", 2024-05-15T13:25:03.869 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 46.278867794999996, 2024-05-15T13:25:03.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.185545829, 2024-05-15T13:25:03.869 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:25:03.870 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:25:03.870 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:25:03.870 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:25:03.870 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:25:03.870 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 97031 2024-05-15T13:25:03.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.870 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:25:03.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:25:03.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.493235+0000", 2024-05-15T13:25:03.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.871 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:25:03.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.493235+0000", 2024-05-15T13:25:03.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.871 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:25:03.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.493236+0000", 2024-05-15T13:25:03.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2699999999999999e-06 2024-05-15T13:25:03.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.871 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:25:03.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.493261+0000", 2024-05-15T13:25:03.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4824000000000002e-05 2024-05-15T13:25:03.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:25:03.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.493274+0000", 2024-05-15T13:25:03.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3307e-05 2024-05-15T13:25:03.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:25:03.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.493283+0000", 2024-05-15T13:25:03.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6629999999999999e-06 2024-05-15T13:25:03.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:25:03.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.493348+0000", 2024-05-15T13:25:03.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4764999999999994e-05 2024-05-15T13:25:03.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:25:03.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.493512+0000", 2024-05-15T13:25:03.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000163898 2024-05-15T13:25:03.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,11", 2024-05-15T13:25:03.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.493671+0000", 2024-05-15T13:25:03.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015935299999999999 2024-05-15T13:25:03.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:25:03.873 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.499346+0000", 2024-05-15T13:25:03.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0056749449999999998 2024-05-15T13:25:03.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.499407+0000", 2024-05-15T13:25:03.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0993999999999999e-05 2024-05-15T13:25:03.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.678708+0000", 2024-05-15T13:25:03.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17930116199999999 2024-05-15T13:25:03.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:25:03.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.678748+0000", 2024-05-15T13:25:03.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9691000000000003e-05 2024-05-15T13:25:03.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:25:03.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.678781+0000", 2024-05-15T13:25:03.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2957000000000003e-05 2024-05-15T13:25:03.875 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.875 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:25:03.875 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.875 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:97033 9.8 9:12e700c8:::benchmark_data_smithi177_159919_object97032:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e842)", 2024-05-15T13:25:03.875 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:24:17.493555+0000", 2024-05-15T13:25:03.875 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 46.278547541000002, 2024-05-15T13:25:03.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18534941199999999, 2024-05-15T13:25:03.875 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:25:03.876 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:25:03.876 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:25:03.876 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:25:03.876 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:25:03.876 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 97033 2024-05-15T13:25:03.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.876 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:25:03.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:25:03.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.493555+0000", 2024-05-15T13:25:03.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:25:03.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.493555+0000", 2024-05-15T13:25:03.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:25:03.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.493556+0000", 2024-05-15T13:25:03.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.47e-07 2024-05-15T13:25:03.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:25:03.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.493584+0000", 2024-05-15T13:25:03.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8050000000000001e-05 2024-05-15T13:25:03.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:25:03.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.493594+0000", 2024-05-15T13:25:03.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.91e-06 2024-05-15T13:25:03.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:25:03.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.493600+0000", 2024-05-15T13:25:03.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8719999999999999e-06 2024-05-15T13:25:03.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:25:03.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.496671+0000", 2024-05-15T13:25:03.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0030708879999999999 2024-05-15T13:25:03.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:25:03.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.496825+0000", 2024-05-15T13:25:03.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015424699999999999 2024-05-15T13:25:03.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,11", 2024-05-15T13:25:03.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.496919+0000", 2024-05-15T13:25:03.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3727000000000006e-05 2024-05-15T13:25:03.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:25:03.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.499485+0000", 2024-05-15T13:25:03.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025665869999999999 2024-05-15T13:25:03.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.501725+0000", 2024-05-15T13:25:03.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0022396130000000001 2024-05-15T13:25:03.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.678813+0000", 2024-05-15T13:25:03.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.17708763699999999 2024-05-15T13:25:03.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:25:03.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.678861+0000", 2024-05-15T13:25:03.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8162999999999999e-05 2024-05-15T13:25:03.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:25:03.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.678904+0000", 2024-05-15T13:25:03.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3770999999999998e-05 2024-05-15T13:25:03.881 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.881 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:25:03.881 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.881 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:97047 9.3 9:c9bee0bf:::benchmark_data_smithi177_159919_object97046:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e842)", 2024-05-15T13:25:03.881 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:24:17.505706+0000", 2024-05-15T13:25:03.881 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 46.266396567999998, 2024-05-15T13:25:03.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19135913500000001, 2024-05-15T13:25:03.881 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:25:03.881 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:25:03.881 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:25:03.881 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:25:03.882 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:25:03.882 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 97047 2024-05-15T13:25:03.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.882 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:25:03.882 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.882 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:25:03.882 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.505706+0000", 2024-05-15T13:25:03.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.882 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.882 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:25:03.882 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.505706+0000", 2024-05-15T13:25:03.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:25:03.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.505707+0000", 2024-05-15T13:25:03.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.248e-06 2024-05-15T13:25:03.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:25:03.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.505726+0000", 2024-05-15T13:25:03.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8474e-05 2024-05-15T13:25:03.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:25:03.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.505737+0000", 2024-05-15T13:25:03.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1455e-05 2024-05-15T13:25:03.884 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.884 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.884 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:25:03.884 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.505746+0000", 2024-05-15T13:25:03.884 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5140000000000001e-06 2024-05-15T13:25:03.884 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.884 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.884 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:25:03.884 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.505803+0000", 2024-05-15T13:25:03.884 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7025999999999999e-05 2024-05-15T13:25:03.884 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.884 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.884 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:25:03.884 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.505944+0000", 2024-05-15T13:25:03.884 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014168899999999999 2024-05-15T13:25:03.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,5", 2024-05-15T13:25:03.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.506035+0000", 2024-05-15T13:25:03.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0883000000000004e-05 2024-05-15T13:25:03.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:25:03.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.508721+0000", 2024-05-15T13:25:03.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0026860949999999999 2024-05-15T13:25:03.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.509044+0000", 2024-05-15T13:25:03.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00032288999999999999 2024-05-15T13:25:03.886 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.886 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.696961+0000", 2024-05-15T13:25:03.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18791664499999999 2024-05-15T13:25:03.886 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.886 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:25:03.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.697019+0000", 2024-05-15T13:25:03.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8136e-05 2024-05-15T13:25:03.886 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.886 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:25:03.887 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.697065+0000", 2024-05-15T13:25:03.887 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6079999999999999e-05 2024-05-15T13:25:03.887 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.887 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:25:03.887 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.887 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.887 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.887 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:97056 9.3 9:c9893693:::benchmark_data_smithi177_159919_object97055:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e842)", 2024-05-15T13:25:03.887 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:24:17.514249+0000", 2024-05-15T13:25:03.887 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 46.257853642000001, 2024-05-15T13:25:03.887 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.183048771, 2024-05-15T13:25:03.887 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:25:03.887 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:25:03.887 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:25:03.887 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:25:03.887 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:25:03.888 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 97056 2024-05-15T13:25:03.888 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.888 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:25:03.888 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.888 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:25:03.888 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.514249+0000", 2024-05-15T13:25:03.888 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.888 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.888 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.888 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:25:03.888 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.514249+0000", 2024-05-15T13:25:03.888 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:03.888 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.888 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.889 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:25:03.889 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.514250+0000", 2024-05-15T13:25:03.889 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0860000000000001e-06 2024-05-15T13:25:03.889 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.889 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.889 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:25:03.889 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.514274+0000", 2024-05-15T13:25:03.889 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3594000000000001e-05 2024-05-15T13:25:03.889 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.889 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.889 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:25:03.889 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.514287+0000", 2024-05-15T13:25:03.889 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2978e-05 2024-05-15T13:25:03.889 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.889 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.890 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:25:03.890 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.514294+0000", 2024-05-15T13:25:03.890 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8140000000000001e-06 2024-05-15T13:25:03.890 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.890 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.890 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:25:03.890 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.514343+0000", 2024-05-15T13:25:03.890 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8566999999999998e-05 2024-05-15T13:25:03.890 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.890 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.890 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:25:03.890 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.514444+0000", 2024-05-15T13:25:03.890 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000100985 2024-05-15T13:25:03.890 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.891 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,5", 2024-05-15T13:25:03.891 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.514532+0000", 2024-05-15T13:25:03.891 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7515999999999994e-05 2024-05-15T13:25:03.891 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.891 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:25:03.891 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.517007+0000", 2024-05-15T13:25:03.891 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024750269999999999 2024-05-15T13:25:03.891 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.891 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.891 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.517295+0000", 2024-05-15T13:25:03.891 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028820300000000001 2024-05-15T13:25:03.891 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.892 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:03.892 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.697202+0000", 2024-05-15T13:25:03.892 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.179906972 2024-05-15T13:25:03.892 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.892 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.892 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:25:03.892 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.697256+0000", 2024-05-15T13:25:03.892 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4237999999999998e-05 2024-05-15T13:25:03.892 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:03.892 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:03.892 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:25:03.892 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:17.697298+0000", 2024-05-15T13:25:03.892 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1791e-05 2024-05-15T13:25:03.892 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.892 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:25:03.893 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.893 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:03.893 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:25:03.893 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:25:03.893 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:25:04.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:03 smithi177 ceph-mon[44303]: pgmap v2221: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 6.7 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 519/329256 objects degraded (0.158%); 13768/329256 objects misplaced (4.182%); 1.0 MiB/s, 16 objects/s recovering 2024-05-15T13:25:04.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:03 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/154513835' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:25:04.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:03 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:25:04.286 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:03 smithi027 ceph-mon[39795]: pgmap v2221: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 6.7 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 519/329256 objects degraded (0.158%); 13768/329256 objects misplaced (4.182%); 1.0 MiB/s, 16 objects/s recovering 2024-05-15T13:25:04.286 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:03 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/154513835' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:25:04.286 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:03 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:25:04.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:03 smithi120 ceph-mon[43961]: pgmap v2221: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 6.7 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 519/329256 objects degraded (0.158%); 13768/329256 objects misplaced (4.182%); 1.0 MiB/s, 16 objects/s recovering 2024-05-15T13:25:04.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:03 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/154513835' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:25:04.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:03 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:25:04.567 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-05-15T13:25:04.851 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:04 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:25:04.851 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:04 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:25:04.851 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:04 smithi177 ceph-mon[44303]: osdmap e861: 12 total, 12 up, 11 in 2024-05-15T13:25:04.920 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:25:05.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:04 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:25:05.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:04 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:25:05.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:04 smithi027 ceph-mon[39795]: osdmap e861: 12 total, 12 up, 11 in 2024-05-15T13:25:05.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:04 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:25:05.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:04 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:25:05.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:04 smithi120 ceph-mon[43961]: osdmap e861: 12 total, 12 up, 11 in 2024-05-15T13:25:05.328 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:25:05.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:05.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:25:05.437 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:25:05.488 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:05.532 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:25:05.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:05.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:25:05.641 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:25:05.694 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:05.718 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:25:05.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:05.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:25:05.826 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:25:05.862 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:06.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:05 smithi177 ceph-mon[44303]: pgmap v2223: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 6.5 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 885 op/s; 874/321537 objects degraded (0.272%); 13259/321537 objects misplaced (4.124%); 2.3 MiB/s, 37 objects/s recovering 2024-05-15T13:25:06.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:05 smithi177 ceph-mon[44303]: 9.b scrub starts 2024-05-15T13:25:06.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:05 smithi027 ceph-mon[39795]: pgmap v2223: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 6.5 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 885 op/s; 874/321537 objects degraded (0.272%); 13259/321537 objects misplaced (4.124%); 2.3 MiB/s, 37 objects/s recovering 2024-05-15T13:25:06.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:05 smithi027 ceph-mon[39795]: 9.b scrub starts 2024-05-15T13:25:06.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:05 smithi120 ceph-mon[43961]: pgmap v2223: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 6.5 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 885 op/s; 874/321537 objects degraded (0.272%); 13259/321537 objects misplaced (4.124%); 2.3 MiB/s, 37 objects/s recovering 2024-05-15T13:25:06.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:05 smithi120 ceph-mon[43961]: 9.b scrub starts 2024-05-15T13:25:06.321 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:25:06.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:06.325 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:25:06.425 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:25:06.455 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:06.599 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:25:06.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:06.605 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:25:06.706 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:25:06.742 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:06.862 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:06 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:25:06.862 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:06 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:25:06.884 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:25:06.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:06.887 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:25:06.988 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:25:07.026 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:07.057 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:25:07.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:07.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:25:07.112 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:25:07.112 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T13:25:07.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:07.113 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:136792 9.7 9:eace7bd4:::benchmark_data_smithi177_159919_object16549:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e862)", 2024-05-15T13:25:07.113 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:25:07.106788+0000", 2024-05-15T13:25:07.113 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.0056628620000000003, 2024-05-15T13:25:07.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0056954589999999999, 2024-05-15T13:25:07.113 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:25:07.113 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:25:07.113 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:25:07.113 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:25:07.113 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:25:07.113 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 136792 2024-05-15T13:25:07.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:07.113 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:25:07.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:07.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:25:07.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:25:07.106788+0000", 2024-05-15T13:25:07.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:07.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:07.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:07.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:25:07.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:25:07.106788+0000", 2024-05-15T13:25:07.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:25:07.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:07.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:07.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:25:07.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:25:07.106789+0000", 2024-05-15T13:25:07.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4720000000000001e-06 2024-05-15T13:25:07.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:07.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:07.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:25:07.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:25:07.106794+0000", 2024-05-15T13:25:07.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4360000000000001e-06 2024-05-15T13:25:07.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:07.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:07.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:25:07.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:25:07.106808+0000", 2024-05-15T13:25:07.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3672999999999999e-05 2024-05-15T13:25:07.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:07.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:07.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:25:07.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:25:07.106818+0000", 2024-05-15T13:25:07.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4089999999999992e-06 2024-05-15T13:25:07.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:07.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:07.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:25:07.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:25:07.106879+0000", 2024-05-15T13:25:07.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1815000000000004e-05 2024-05-15T13:25:07.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:07.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:07.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:25:07.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:25:07.107014+0000", 2024-05-15T13:25:07.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000135112 2024-05-15T13:25:07.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:07.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:07.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,7", 2024-05-15T13:25:07.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:25:07.107130+0000", 2024-05-15T13:25:07.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115129 2024-05-15T13:25:07.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:25:07.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:25:07.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:07.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:25:07.111137+0000", 2024-05-15T13:25:07.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0040074280000000004 2024-05-15T13:25:07.117 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:07.117 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:25:07.117 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:07.117 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:25:07.117 INFO:teuthology.orchestra.run.smithi177.stdout: ], 2024-05-15T13:25:07.117 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 1 2024-05-15T13:25:07.117 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:25:07.162 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:25:07.191 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:07.216 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:25:07.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:07.221 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:25:07.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:06 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:25:07.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:06 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:25:07.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:06 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:25:07.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:06 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:25:07.322 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:25:07.373 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:07.964 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:25:08.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:07 smithi177 ceph-mon[44303]: pgmap v2224: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 6.5 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 822 op/s; 874/318510 objects degraded (0.274%); 13259/318510 objects misplaced (4.163%); 1.7 MiB/s, 27 objects/s recovering 2024-05-15T13:25:08.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:07 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 874/318510 objects degraded (0.274%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T13:25:08.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:07 smithi177 ceph-mon[44303]: osdmap e862: 12 total, 12 up, 11 in 2024-05-15T13:25:08.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:07 smithi177 ceph-mon[44303]: 9.b scrub ok 2024-05-15T13:25:08.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:07 smithi177 ceph-mon[44303]: 1.10 scrub starts 2024-05-15T13:25:08.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:07 smithi177 ceph-mon[44303]: 1.10 scrub ok 2024-05-15T13:25:08.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:07 smithi027 ceph-mon[39795]: pgmap v2224: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 6.5 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 822 op/s; 874/318510 objects degraded (0.274%); 13259/318510 objects misplaced (4.163%); 1.7 MiB/s, 27 objects/s recovering 2024-05-15T13:25:08.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:07 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 874/318510 objects degraded (0.274%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T13:25:08.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:07 smithi027 ceph-mon[39795]: osdmap e862: 12 total, 12 up, 11 in 2024-05-15T13:25:08.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:07 smithi027 ceph-mon[39795]: 9.b scrub ok 2024-05-15T13:25:08.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:07 smithi027 ceph-mon[39795]: 1.10 scrub starts 2024-05-15T13:25:08.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:07 smithi027 ceph-mon[39795]: 1.10 scrub ok 2024-05-15T13:25:08.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:07 smithi120 ceph-mon[43961]: pgmap v2224: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 6.5 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 822 op/s; 874/318510 objects degraded (0.274%); 13259/318510 objects misplaced (4.163%); 1.7 MiB/s, 27 objects/s recovering 2024-05-15T13:25:08.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:07 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 874/318510 objects degraded (0.274%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T13:25:08.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:07 smithi120 ceph-mon[43961]: osdmap e862: 12 total, 12 up, 11 in 2024-05-15T13:25:08.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:07 smithi120 ceph-mon[43961]: 9.b scrub ok 2024-05-15T13:25:08.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:07 smithi120 ceph-mon[43961]: 1.10 scrub starts 2024-05-15T13:25:08.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:07 smithi120 ceph-mon[43961]: 1.10 scrub ok 2024-05-15T13:25:08.638 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:25:08.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:08.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:25:08.743 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-05-15T13:25:08.750 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:25:08.819 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:08.849 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:25:08.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:08.854 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:25:08.956 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:25:09.011 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:09.092 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:25:10.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:09 smithi177 ceph-mon[44303]: pgmap v2226: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 6.4 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 770 op/s; 1052/314034 objects degraded (0.335%); 13008/314034 objects misplaced (4.142%); 1.7 MiB/s, 28 objects/s recovering 2024-05-15T13:25:10.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:09 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:25:10.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:09 smithi177 ceph-mon[44303]: osdmap e863: 12 total, 12 up, 11 in 2024-05-15T13:25:10.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:09 smithi027 ceph-mon[39795]: pgmap v2226: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 6.4 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 770 op/s; 1052/314034 objects degraded (0.335%); 13008/314034 objects misplaced (4.142%); 1.7 MiB/s, 28 objects/s recovering 2024-05-15T13:25:10.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:09 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:25:10.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:09 smithi027 ceph-mon[39795]: osdmap e863: 12 total, 12 up, 11 in 2024-05-15T13:25:10.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:09 smithi120 ceph-mon[43961]: pgmap v2226: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 6.4 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 770 op/s; 1052/314034 objects degraded (0.335%); 13008/314034 objects misplaced (4.142%); 1.7 MiB/s, 28 objects/s recovering 2024-05-15T13:25:10.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:09 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:25:10.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:09 smithi120 ceph-mon[43961]: osdmap e863: 12 total, 12 up, 11 in 2024-05-15T13:25:10.985 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:25:10.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:10.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:25:11.092 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:25:11.130 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:25:11.161 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:10 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:25:11.161 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:10 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:25:11.161 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:10 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:25:11.161 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:10 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:25:11.161 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:10 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:25:11.170 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:11.176 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:10 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:25:11.176 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:10 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:25:11.176 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:10 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:25:11.176 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:10 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:25:11.176 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:10 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:25:11.182 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:25:11.227 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:25:11.263 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:25:11.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:10 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:25:11.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:10 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:25:11.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:10 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:25:11.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:10 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:25:11.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:10 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:25:11.315 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:25:11.346 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:25:11.380 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:25:11.410 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:25:11.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:11.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:25:11.435 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:25:11.465 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:25:11.503 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:25:11.523 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:25:11.535 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:25:11.556 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:11.563 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:25:11.589 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:25:11.593 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:25:11.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:11.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:25:11.596 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:25:11.596 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:25:11.596 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:25:11.596 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:25:11.596 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:25:11.596 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:25:11.596 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:25:11.596 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:25:11.596 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:25:11.596 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:25:11.596 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:25:11.597 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:25:11.696 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:25:11.748 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:11.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:11 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-a[39772]: 2024-05-15T13:25:11.465+0000 7f8daffd2700 -1 mon.a@0(leader).osd e864 definitely_dead 0 2024-05-15T13:25:11.788 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:25:11.789 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:25:11.798 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:25:11.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:11.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:25:11.905 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:25:11.936 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:11.959 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:25:11.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:11.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:25:11.995 INFO:teuthology.orchestra.run.smithi027.stderr:marked out osd.1. 2024-05-15T13:25:12.063 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:25:12.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:11 smithi027 ceph-mon[39795]: 1.11 scrub starts 2024-05-15T13:25:12.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:11 smithi027 ceph-mon[39795]: 1.11 scrub ok 2024-05-15T13:25:12.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:11 smithi027 ceph-mon[39795]: pgmap v2228: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 6.3 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 735 op/s; 1223/309579 objects degraded (0.395%); 12750/309579 objects misplaced (4.118%); 1.4 MiB/s, 23 objects/s recovering 2024-05-15T13:25:12.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:11 smithi027 ceph-mon[39795]: osdmap e864: 12 total, 12 up, 11 in 2024-05-15T13:25:12.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:11 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3964410793' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:25:12.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:11 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4055435818' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-05-15T13:25:12.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:11 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-05-15T13:25:12.095 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:11 smithi027 ceph-mon[39795]: Client client.admin marked osd.1 out, while it was still marked up 2024-05-15T13:25:12.097 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:12.145 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:11 smithi177 ceph-mon[44303]: 1.11 scrub starts 2024-05-15T13:25:12.145 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:11 smithi177 ceph-mon[44303]: 1.11 scrub ok 2024-05-15T13:25:12.146 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:11 smithi177 ceph-mon[44303]: pgmap v2228: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 6.3 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 735 op/s; 1223/309579 objects degraded (0.395%); 12750/309579 objects misplaced (4.118%); 1.4 MiB/s, 23 objects/s recovering 2024-05-15T13:25:12.146 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:11 smithi177 ceph-mon[44303]: osdmap e864: 12 total, 12 up, 11 in 2024-05-15T13:25:12.146 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:11 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3964410793' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:25:12.146 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:11 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4055435818' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-05-15T13:25:12.146 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:11 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-05-15T13:25:12.146 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:11 smithi177 ceph-mon[44303]: Client client.admin marked osd.1 out, while it was still marked up 2024-05-15T13:25:12.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:11 smithi120 ceph-mon[43961]: 1.11 scrub starts 2024-05-15T13:25:12.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:11 smithi120 ceph-mon[43961]: 1.11 scrub ok 2024-05-15T13:25:12.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:11 smithi120 ceph-mon[43961]: pgmap v2228: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 6.3 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 735 op/s; 1223/309579 objects degraded (0.395%); 12750/309579 objects misplaced (4.118%); 1.4 MiB/s, 23 objects/s recovering 2024-05-15T13:25:12.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:11 smithi120 ceph-mon[43961]: osdmap e864: 12 total, 12 up, 11 in 2024-05-15T13:25:12.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:11 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3964410793' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:25:12.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:11 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4055435818' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-05-15T13:25:12.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:11 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-05-15T13:25:12.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:11 smithi120 ceph-mon[43961]: Client client.admin marked osd.1 out, while it was still marked up 2024-05-15T13:25:12.428 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-05-15T13:25:12.690 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:25:13.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:12 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-05-15T13:25:13.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:12 smithi177 ceph-mon[44303]: osdmap e865: 12 total, 12 up, 10 in 2024-05-15T13:25:13.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:12 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-05-15T13:25:13.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:12 smithi027 ceph-mon[39795]: osdmap e865: 12 total, 12 up, 10 in 2024-05-15T13:25:13.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:12 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-05-15T13:25:13.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:12 smithi120 ceph-mon[43961]: osdmap e865: 12 total, 12 up, 10 in 2024-05-15T13:25:14.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:13 smithi177 ceph-mon[44303]: pgmap v2231: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 6.2 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 744 op/s; 1223/305415 objects degraded (0.400%); 12750/305415 objects misplaced (4.175%); 1.6 MiB/s, 27 objects/s recovering 2024-05-15T13:25:14.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:13 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 1223/305415 objects degraded (0.400%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T13:25:14.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:13 smithi177 ceph-mon[44303]: osdmap e866: 12 total, 12 up, 10 in 2024-05-15T13:25:14.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:13 smithi177 ceph-mon[44303]: osdmap e867: 12 total, 12 up, 10 in 2024-05-15T13:25:14.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:13 smithi027 ceph-mon[39795]: pgmap v2231: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 6.2 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 744 op/s; 1223/305415 objects degraded (0.400%); 12750/305415 objects misplaced (4.175%); 1.6 MiB/s, 27 objects/s recovering 2024-05-15T13:25:14.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:13 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 1223/305415 objects degraded (0.400%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T13:25:14.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:13 smithi027 ceph-mon[39795]: osdmap e866: 12 total, 12 up, 10 in 2024-05-15T13:25:14.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:13 smithi027 ceph-mon[39795]: osdmap e867: 12 total, 12 up, 10 in 2024-05-15T13:25:14.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:13 smithi120 ceph-mon[43961]: pgmap v2231: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 6.2 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 744 op/s; 1223/305415 objects degraded (0.400%); 12750/305415 objects misplaced (4.175%); 1.6 MiB/s, 27 objects/s recovering 2024-05-15T13:25:14.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:13 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 1223/305415 objects degraded (0.400%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T13:25:14.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:13 smithi120 ceph-mon[43961]: osdmap e866: 12 total, 12 up, 10 in 2024-05-15T13:25:14.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:13 smithi120 ceph-mon[43961]: osdmap e867: 12 total, 12 up, 10 in 2024-05-15T13:25:14.402 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:25:14.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:14.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:25:14.507 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:25:14.611 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:14.674 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:25:14.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:14.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:25:14.780 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:25:14.812 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:14.888 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:25:14.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:14.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:25:14.997 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:25:15.005 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:15 smithi120 ceph-mon[43961]: 9.b starting backfill to osd.9 from (0'0,0'0] MAX to 864'11600 2024-05-15T13:25:15.006 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:15 smithi120 ceph-mon[43961]: 9.2 starting backfill to osd.11 from (0'0,0'0] MAX to 864'11159 2024-05-15T13:25:15.006 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:15 smithi120 ceph-mon[43961]: 1.f scrub starts 2024-05-15T13:25:15.006 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:15 smithi120 ceph-mon[43961]: osdmap e868: 12 total, 12 up, 10 in 2024-05-15T13:25:15.006 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:15 smithi120 ceph-mon[43961]: 1.f scrub ok 2024-05-15T13:25:15.054 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:25:15.054 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:25:15.078 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:15 smithi027 ceph-mon[39795]: 9.b starting backfill to osd.9 from (0'0,0'0] MAX to 864'11600 2024-05-15T13:25:15.078 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:15 smithi027 ceph-mon[39795]: 9.2 starting backfill to osd.11 from (0'0,0'0] MAX to 864'11159 2024-05-15T13:25:15.078 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:15 smithi027 ceph-mon[39795]: 1.f scrub starts 2024-05-15T13:25:15.078 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:15 smithi027 ceph-mon[39795]: osdmap e868: 12 total, 12 up, 10 in 2024-05-15T13:25:15.078 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:15 smithi027 ceph-mon[39795]: 1.f scrub ok 2024-05-15T13:25:15.094 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:15.144 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:25:15.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:15.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:25:15.249 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:25:15.277 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:15 smithi177 ceph-mon[44303]: 9.b starting backfill to osd.9 from (0'0,0'0] MAX to 864'11600 2024-05-15T13:25:15.277 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:15 smithi177 ceph-mon[44303]: 9.2 starting backfill to osd.11 from (0'0,0'0] MAX to 864'11159 2024-05-15T13:25:15.278 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:15 smithi177 ceph-mon[44303]: 1.f scrub starts 2024-05-15T13:25:15.278 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:15 smithi177 ceph-mon[44303]: osdmap e868: 12 total, 12 up, 10 in 2024-05-15T13:25:15.278 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:15 smithi177 ceph-mon[44303]: 1.f scrub ok 2024-05-15T13:25:15.280 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:15.303 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:25:15.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:15.306 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:25:15.407 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:25:15.437 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:15.476 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:25:15.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:15.484 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:25:15.585 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:25:15.618 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:15.771 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:25:15.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:15.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:25:15.871 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-05-15T13:25:15.876 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:25:15.924 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:15.955 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:25:15.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:15.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:25:16.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:16 smithi027 ceph-mon[39795]: pgmap v2235: 47 pgs: 1 active+clean+scrubbing, 3 peering, 7 remapped+peering, 2 unknown, 3 active+recovering+undersized+degraded+remapped, 31 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 498 op/s; 1364/301215 objects degraded (0.453%); 18587/301215 objects misplaced (6.171%); 2.1 MiB/s, 33 objects/s recovering 2024-05-15T13:25:16.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:16 smithi027 ceph-mon[39795]: osdmap e869: 12 total, 12 up, 10 in 2024-05-15T13:25:16.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:16 smithi027 ceph-mon[39795]: Health check failed: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-05-15T13:25:16.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:16 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:25:16.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:16 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:25:16.063 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:25:16.097 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:16.161 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:25:16.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:16 smithi177 ceph-mon[44303]: pgmap v2235: 47 pgs: 1 active+clean+scrubbing, 3 peering, 7 remapped+peering, 2 unknown, 3 active+recovering+undersized+degraded+remapped, 31 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 498 op/s; 1364/301215 objects degraded (0.453%); 18587/301215 objects misplaced (6.171%); 2.1 MiB/s, 33 objects/s recovering 2024-05-15T13:25:16.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:16 smithi177 ceph-mon[44303]: osdmap e869: 12 total, 12 up, 10 in 2024-05-15T13:25:16.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:16 smithi177 ceph-mon[44303]: Health check failed: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-05-15T13:25:16.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:16 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:25:16.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:16 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:25:16.248 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:25:16.252 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:16 smithi120 ceph-mon[43961]: pgmap v2235: 47 pgs: 1 active+clean+scrubbing, 3 peering, 7 remapped+peering, 2 unknown, 3 active+recovering+undersized+degraded+remapped, 31 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 498 op/s; 1364/301215 objects degraded (0.453%); 18587/301215 objects misplaced (6.171%); 2.1 MiB/s, 33 objects/s recovering 2024-05-15T13:25:16.252 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:16 smithi120 ceph-mon[43961]: osdmap e869: 12 total, 12 up, 10 in 2024-05-15T13:25:16.252 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:16 smithi120 ceph-mon[43961]: Health check failed: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-05-15T13:25:16.252 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:16 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:25:16.252 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:16 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:25:16.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:16.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:25:16.354 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:25:16.382 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:16.416 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:25:16.421 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:25:16.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:16.424 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:25:16.525 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:25:16.558 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:25:16.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:16.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:25:16.684 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:25:16.714 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:16.742 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:25:17.006 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:17 smithi177 ceph-mon[44303]: 1.16 scrub starts 2024-05-15T13:25:17.006 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:17 smithi177 ceph-mon[44303]: 1.16 scrub ok 2024-05-15T13:25:17.006 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:17 smithi177 ceph-mon[44303]: osdmap e870: 12 total, 12 up, 10 in 2024-05-15T13:25:17.006 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:17 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3642192283' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:25:17.006 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:17 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:25:17.006 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:17 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:25:17.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:17 smithi027 ceph-mon[39795]: 1.16 scrub starts 2024-05-15T13:25:17.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:17 smithi027 ceph-mon[39795]: 1.16 scrub ok 2024-05-15T13:25:17.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:17 smithi027 ceph-mon[39795]: osdmap e870: 12 total, 12 up, 10 in 2024-05-15T13:25:17.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:17 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3642192283' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:25:17.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:17 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:25:17.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:17 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:25:17.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:17 smithi120 ceph-mon[43961]: 1.16 scrub starts 2024-05-15T13:25:17.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:17 smithi120 ceph-mon[43961]: 1.16 scrub ok 2024-05-15T13:25:17.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:17 smithi120 ceph-mon[43961]: osdmap e870: 12 total, 12 up, 10 in 2024-05-15T13:25:17.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:17 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3642192283' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:25:17.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:17 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:25:17.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:17 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:25:17.349 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:25:17.465 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:25:17.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:17.468 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:25:17.569 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:25:17.597 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:17.746 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:25:17.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:17.749 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:25:17.850 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:25:17.896 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:17.933 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:25:17.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:17.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:25:18.007 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:18 smithi177 ceph-mon[44303]: pgmap v2238: 47 pgs: 1 active+clean+scrubbing, 4 peering, 6 remapped+peering, 2 unknown, 5 active+recovering+undersized+degraded+remapped, 29 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 392 op/s; 1367/300483 objects degraded (0.455%); 31045/300483 objects misplaced (10.332%); 3.8 MiB/s, 60 objects/s recovering 2024-05-15T13:25:18.007 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:18 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:25:18.028 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:25:18.028 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:25:18.045 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:25:18.077 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:18.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:18 smithi027 ceph-mon[39795]: pgmap v2238: 47 pgs: 1 active+clean+scrubbing, 4 peering, 6 remapped+peering, 2 unknown, 5 active+recovering+undersized+degraded+remapped, 29 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 392 op/s; 1367/300483 objects degraded (0.455%); 31045/300483 objects misplaced (10.332%); 3.8 MiB/s, 60 objects/s recovering 2024-05-15T13:25:18.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:18 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:25:18.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:18 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3642192283' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:25:18.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:18 smithi027 ceph-mon[39795]: osdmap e871: 12 total, 12 up, 10 in 2024-05-15T13:25:18.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:18 smithi120 ceph-mon[43961]: pgmap v2238: 47 pgs: 1 active+clean+scrubbing, 4 peering, 6 remapped+peering, 2 unknown, 5 active+recovering+undersized+degraded+remapped, 29 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 392 op/s; 1367/300483 objects degraded (0.455%); 31045/300483 objects misplaced (10.332%); 3.8 MiB/s, 60 objects/s recovering 2024-05-15T13:25:18.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:18 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:25:18.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:18 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3642192283' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:25:18.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:18 smithi120 ceph-mon[43961]: osdmap e871: 12 total, 12 up, 10 in 2024-05-15T13:25:18.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:18 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3642192283' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:25:18.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:18 smithi177 ceph-mon[44303]: osdmap e871: 12 total, 12 up, 10 in 2024-05-15T13:25:18.428 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:25:18.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:18.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:25:18.532 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:25:18.562 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:18.591 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:25:18.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:18.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:25:18.696 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:25:18.738 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-05-15T13:25:18.741 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:18.794 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:25:18.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:18.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:25:18.898 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:25:18.927 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:18.948 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:25:18.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:18.951 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:25:19.040 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:25:19.051 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:25:19.109 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:19.141 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:25:19.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:19.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:25:19.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:19 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 1367/300294 objects degraded (0.455%), 5 pgs degraded (PG_DEGRADED) 2024-05-15T13:25:19.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:19 smithi027 ceph-mon[39795]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-05-15T13:25:19.248 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:25:19.278 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:19.302 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:25:19.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:19.306 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:25:19.311 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:19 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 1367/300294 objects degraded (0.455%), 5 pgs degraded (PG_DEGRADED) 2024-05-15T13:25:19.311 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:19 smithi177 ceph-mon[44303]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-05-15T13:25:19.407 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:25:19.458 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:19 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 1367/300294 objects degraded (0.455%), 5 pgs degraded (PG_DEGRADED) 2024-05-15T13:25:19.458 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:19 smithi120 ceph-mon[43961]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-05-15T13:25:19.461 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:19.490 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:25:19.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:19.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:25:19.533 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:25:19.553 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:25:19.575 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:25:19.596 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:25:19.597 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:25:19.666 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:19.717 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:25:19.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:19.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:25:19.824 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:25:19.886 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:19.913 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:25:19.920 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 9, 10, 11, 7, 4] out_osds: [8, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:25:19.921 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:25:19.921 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-05-15T13:25:19.921 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-05-15T13:25:19.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:19.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:25:20.023 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:25:20.053 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:20.230 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:25:20.280 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:20 smithi027 ceph-mon[39795]: pgmap v2240: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 4 peering, 5 remapped+peering, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 106 op/s; 1367/300294 objects degraded (0.455%); 34968/300294 objects misplaced (11.645%); 35 MiB/s, 554 objects/s recovering 2024-05-15T13:25:20.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:20 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2425778553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:25:20.351 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:20 smithi177 ceph-mon[44303]: pgmap v2240: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 4 peering, 5 remapped+peering, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 106 op/s; 1367/300294 objects degraded (0.455%); 34968/300294 objects misplaced (11.645%); 35 MiB/s, 554 objects/s recovering 2024-05-15T13:25:20.351 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:20 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2425778553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:25:20.403 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:25:20.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:20.407 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:25:20.426 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:25:20.508 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:25:20.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:20 smithi120 ceph-mon[43961]: pgmap v2240: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 4 peering, 5 remapped+peering, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 106 op/s; 1367/300294 objects degraded (0.455%); 34968/300294 objects misplaced (11.645%); 35 MiB/s, 554 objects/s recovering 2024-05-15T13:25:20.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:20 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2425778553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:25:20.562 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:20.596 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:25:20.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:20.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:25:20.707 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:25:20.738 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:20.812 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:25:20.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:20.815 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:25:20.916 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:25:20.948 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:20.958 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:25:20.958 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:25:21.098 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:25:21.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:21.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:25:21.202 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:25:21.231 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:21.251 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:25:21.252 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:21 smithi177 ceph-mon[44303]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T13:25:21.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:21.254 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:25:21.355 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:25:21.383 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:21.531 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:21 smithi027 ceph-mon[39795]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T13:25:21.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:21 smithi120 ceph-mon[43961]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T13:25:21.694 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:25:21.722 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-05-15T13:25:21.734 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:25:21.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:21.737 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:25:21.838 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:25:21.890 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:21.930 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:25:21.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:21.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:25:22.025 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:25:22.037 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:25:22.094 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:22.131 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:25:22.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:22.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:25:22.236 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:25:22.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:22 smithi027 ceph-mon[39795]: pgmap v2241: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 36 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 59 op/s; 8831/300144 objects degraded (2.942%); 53236/300144 objects misplaced (17.737%); 31 MiB/s, 496 objects/s recovering 2024-05-15T13:25:22.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:22 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:25:22.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:22 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:25:22.269 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:22 smithi027 ceph-mon[39795]: osdmap e872: 12 total, 12 up, 10 in 2024-05-15T13:25:22.272 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:22.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:22 smithi120 ceph-mon[43961]: pgmap v2241: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 36 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 59 op/s; 8831/300144 objects degraded (2.942%); 53236/300144 objects misplaced (17.737%); 31 MiB/s, 496 objects/s recovering 2024-05-15T13:25:22.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:22 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:25:22.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:22 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:25:22.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:22 smithi120 ceph-mon[43961]: osdmap e872: 12 total, 12 up, 10 in 2024-05-15T13:25:22.299 INFO:teuthology.orchestra.run.smithi027.stderr:no valid command found; 10 closest matches: 2024-05-15T13:25:22.299 INFO:teuthology.orchestra.run.smithi027.stderr:0 2024-05-15T13:25:22.299 INFO:teuthology.orchestra.run.smithi027.stderr:1 2024-05-15T13:25:22.299 INFO:teuthology.orchestra.run.smithi027.stderr:2 2024-05-15T13:25:22.299 INFO:teuthology.orchestra.run.smithi027.stderr:abort 2024-05-15T13:25:22.299 INFO:teuthology.orchestra.run.smithi027.stderr:assert 2024-05-15T13:25:22.299 INFO:teuthology.orchestra.run.smithi027.stderr:bench [] [] [] [] 2024-05-15T13:25:22.299 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs debug_inject_read_zeros 2024-05-15T13:25:22.299 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs files list 2024-05-15T13:25:22.299 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs stats 2024-05-15T13:25:22.299 INFO:teuthology.orchestra.run.smithi027.stderr:bluestore allocator dump block 2024-05-15T13:25:22.300 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: invalid command 2024-05-15T13:25:22.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:22 smithi177 ceph-mon[44303]: pgmap v2241: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 36 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 59 op/s; 8831/300144 objects degraded (2.942%); 53236/300144 objects misplaced (17.737%); 31 MiB/s, 496 objects/s recovering 2024-05-15T13:25:22.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:22 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:25:22.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:22 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:25:22.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:22 smithi177 ceph-mon[44303]: osdmap e872: 12 total, 12 up, 10 in 2024-05-15T13:25:22.522 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:25:22.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:22.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:25:22.626 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:25:22.670 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:22.711 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:25:22.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:22.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:25:22.817 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:25:22.849 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:23.397 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:25:23.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:25:23.400 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:25:23.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:25:23.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:25:23.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:25:23.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:25:23.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:25:23.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:23.402 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:25:23.503 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:25:23.531 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:23.531 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:25:23.531 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:25:23.531 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:25:23.531 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:25:23.531 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:25:23.531 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:25:23.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:25:23.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:25:23.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:25:23.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:25:23.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:25:23.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:25:23.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:25:23.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:25:23.656 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:25:23.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:23.659 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:25:23.671 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T13:25:23.671 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T13:25:23.671 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-05-15T13:25:23.760 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:25:23.790 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:23.811 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:25:23.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:23.814 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:25:23.916 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:25:23.945 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:23.957 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:25:23.958 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:25:23.970 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:25:23.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:23.973 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:25:24.074 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:25:24.102 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:24.162 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:25:24.225 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:25:24.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:24.228 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:25:24.329 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:25:24.379 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:24.421 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:25:24.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:24.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:25:24.532 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:25:24.561 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:24 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 8831/300144 objects degraded (2.942%), 7 pgs degraded (PG_DEGRADED) 2024-05-15T13:25:24.561 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:24 smithi177 ceph-mon[44303]: osdmap e873: 12 total, 12 up, 10 in 2024-05-15T13:25:24.563 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:24.659 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:24 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 8831/300144 objects degraded (2.942%), 7 pgs degraded (PG_DEGRADED) 2024-05-15T13:25:24.659 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:24 smithi027 ceph-mon[39795]: osdmap e873: 12 total, 12 up, 10 in 2024-05-15T13:25:24.687 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:25:24.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:24.690 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:25:24.709 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-05-15T13:25:24.791 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:25:24.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:24 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 8831/300144 objects degraded (2.942%), 7 pgs degraded (PG_DEGRADED) 2024-05-15T13:25:24.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:24 smithi120 ceph-mon[43961]: osdmap e873: 12 total, 12 up, 10 in 2024-05-15T13:25:24.819 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:24.838 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:25:24.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:24.841 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:25:24.943 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:25:24.971 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:25.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:25 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:25:25.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:25 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:25:25.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:25 smithi177 ceph-mon[44303]: pgmap v2245: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 37 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 12 op/s; 8831/300144 objects degraded (2.942%); 49212/300144 objects misplaced (16.396%); 58 MiB/s, 935 objects/s recovering 2024-05-15T13:25:25.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:25 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:25:25.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:25 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:25:25.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:25 smithi027 ceph-mon[39795]: pgmap v2245: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 37 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 12 op/s; 8831/300144 objects degraded (2.942%); 49212/300144 objects misplaced (16.396%); 58 MiB/s, 935 objects/s recovering 2024-05-15T13:25:25.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:25 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:25:25.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:25 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:25:25.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:25 smithi120 ceph-mon[43961]: pgmap v2245: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 37 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 12 op/s; 8831/300144 objects degraded (2.942%); 49212/300144 objects misplaced (16.396%); 58 MiB/s, 935 objects/s recovering 2024-05-15T13:25:26.364 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: osdmap e874: 12 total, 12 up, 10 in 2024-05-15T13:25:26.365 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=59} crc {auth=3727032478,config=1059521453,health=656271526,kv=2771505923}) 2024-05-15T13:25:26.365 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=562633052}) 2024-05-15T13:25:26.365 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3974286832}) 2024-05-15T13:25:26.365 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2760381946}) 2024-05-15T13:25:26.365 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1195902242}) 2024-05-15T13:25:26.365 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3547126758}) 2024-05-15T13:25:26.365 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=137419636}) 2024-05-15T13:25:26.365 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2742590085}) 2024-05-15T13:25:26.365 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=354886503}) 2024-05-15T13:25:26.365 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=219649518}) 2024-05-15T13:25:26.365 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3732176288}) 2024-05-15T13:25:26.376 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:25:26.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=846292397}) 2024-05-15T13:25:26.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1674790404}) 2024-05-15T13:25:26.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3571758622}) 2024-05-15T13:25:26.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2048867150}) 2024-05-15T13:25:26.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=947768696}) 2024-05-15T13:25:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4209590272}) 2024-05-15T13:25:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3206362821}) 2024-05-15T13:25:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=797841427}) 2024-05-15T13:25:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2569765564}) 2024-05-15T13:25:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2552906282}) 2024-05-15T13:25:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1113067590}) 2024-05-15T13:25:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3586943393}) 2024-05-15T13:25:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4010813643}) 2024-05-15T13:25:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=68202620}) 2024-05-15T13:25:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2195205994}) 2024-05-15T13:25:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=269475152}) 2024-05-15T13:25:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2196281583}) 2024-05-15T13:25:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3795383125}) 2024-05-15T13:25:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=336187104}) 2024-05-15T13:25:26.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1062473678}) 2024-05-15T13:25:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3127181941}) 2024-05-15T13:25:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1403669791}) 2024-05-15T13:25:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2009925847}) 2024-05-15T13:25:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4133727929}) 2024-05-15T13:25:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3039099997}) 2024-05-15T13:25:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3412257710}) 2024-05-15T13:25:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=183000082}) 2024-05-15T13:25:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3813338609}) 2024-05-15T13:25:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=282467982}) 2024-05-15T13:25:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2783874054}) 2024-05-15T13:25:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3731307992}) 2024-05-15T13:25:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1218932475}) 2024-05-15T13:25:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=52465580}) 2024-05-15T13:25:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=251017902}) 2024-05-15T13:25:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2288950160}) 2024-05-15T13:25:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1690870073}) 2024-05-15T13:25:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1280297524}) 2024-05-15T13:25:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=459506690}) 2024-05-15T13:25:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2425002234}) 2024-05-15T13:25:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1423607182}) 2024-05-15T13:25:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1693795940}) 2024-05-15T13:25:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=209194341}) 2024-05-15T13:25:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1606181704}) 2024-05-15T13:25:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1581111896}) 2024-05-15T13:25:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2109747251}) 2024-05-15T13:25:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1159425918}) 2024-05-15T13:25:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=664390873}) 2024-05-15T13:25:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3282106898}) 2024-05-15T13:25:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4119617330}) 2024-05-15T13:25:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4041446488}) 2024-05-15T13:25:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=198309354}) 2024-05-15T13:25:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1698740385}) 2024-05-15T13:25:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=372269868}) 2024-05-15T13:25:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2922959898}) 2024-05-15T13:25:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=804423591}) 2024-05-15T13:25:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2235537078}) 2024-05-15T13:25:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2668198477}) 2024-05-15T13:25:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=939719702}) 2024-05-15T13:25:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=617245647}) 2024-05-15T13:25:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3783645496}) 2024-05-15T13:25:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3317023565}) 2024-05-15T13:25:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1481337511}) 2024-05-15T13:25:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=97,mdsmap=3} crc {logm=1122068573,mdsmap=3984425268}) 2024-05-15T13:25:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=13,mon_config_key=66} crc {mgr=76093979,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=2519129591,mon_config_key=1488343634}) 2024-05-15T13:25:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=287011682}) 2024-05-15T13:25:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2706520274}) 2024-05-15T13:25:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=9,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=73} crc {mon_config_key=3918367541,monmap=2435304556,osd_metadata=1212035326,osd_pg_creating=162581160,osdmap=3722839559}) 2024-05-15T13:25:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3547802431}) 2024-05-15T13:25:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=45} crc {osdmap=610611873}) 2024-05-15T13:25:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:26 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3678544733' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:25:26.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: osdmap e874: 12 total, 12 up, 10 in 2024-05-15T13:25:26.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=59} crc {auth=3727032478,config=1059521453,health=656271526,kv=2771505923}) 2024-05-15T13:25:26.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=562633052}) 2024-05-15T13:25:26.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3974286832}) 2024-05-15T13:25:26.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2760381946}) 2024-05-15T13:25:26.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1195902242}) 2024-05-15T13:25:26.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3547126758}) 2024-05-15T13:25:26.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=137419636}) 2024-05-15T13:25:26.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2742590085}) 2024-05-15T13:25:26.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=354886503}) 2024-05-15T13:25:26.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=219649518}) 2024-05-15T13:25:26.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3732176288}) 2024-05-15T13:25:26.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=846292397}) 2024-05-15T13:25:26.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1674790404}) 2024-05-15T13:25:26.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3571758622}) 2024-05-15T13:25:26.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2048867150}) 2024-05-15T13:25:26.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=947768696}) 2024-05-15T13:25:26.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4209590272}) 2024-05-15T13:25:26.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3206362821}) 2024-05-15T13:25:26.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=797841427}) 2024-05-15T13:25:26.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2569765564}) 2024-05-15T13:25:26.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2552906282}) 2024-05-15T13:25:26.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1113067590}) 2024-05-15T13:25:26.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3586943393}) 2024-05-15T13:25:26.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4010813643}) 2024-05-15T13:25:26.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=68202620}) 2024-05-15T13:25:26.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2195205994}) 2024-05-15T13:25:26.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=269475152}) 2024-05-15T13:25:26.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2196281583}) 2024-05-15T13:25:26.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3795383125}) 2024-05-15T13:25:26.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=336187104}) 2024-05-15T13:25:26.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1062473678}) 2024-05-15T13:25:26.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3127181941}) 2024-05-15T13:25:26.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1403669791}) 2024-05-15T13:25:26.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2009925847}) 2024-05-15T13:25:26.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4133727929}) 2024-05-15T13:25:26.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3039099997}) 2024-05-15T13:25:26.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3412257710}) 2024-05-15T13:25:26.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=183000082}) 2024-05-15T13:25:26.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3813338609}) 2024-05-15T13:25:26.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=282467982}) 2024-05-15T13:25:26.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2783874054}) 2024-05-15T13:25:26.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3731307992}) 2024-05-15T13:25:26.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1218932475}) 2024-05-15T13:25:26.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=52465580}) 2024-05-15T13:25:26.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=251017902}) 2024-05-15T13:25:26.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2288950160}) 2024-05-15T13:25:26.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1690870073}) 2024-05-15T13:25:26.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1280297524}) 2024-05-15T13:25:26.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=459506690}) 2024-05-15T13:25:26.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2425002234}) 2024-05-15T13:25:26.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1423607182}) 2024-05-15T13:25:26.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1693795940}) 2024-05-15T13:25:26.785 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=209194341}) 2024-05-15T13:25:26.785 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1606181704}) 2024-05-15T13:25:26.785 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1581111896}) 2024-05-15T13:25:26.785 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2109747251}) 2024-05-15T13:25:26.785 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1159425918}) 2024-05-15T13:25:26.785 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=664390873}) 2024-05-15T13:25:26.785 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3282106898}) 2024-05-15T13:25:26.785 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4119617330}) 2024-05-15T13:25:26.785 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4041446488}) 2024-05-15T13:25:26.785 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=198309354}) 2024-05-15T13:25:26.785 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1698740385}) 2024-05-15T13:25:26.785 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=372269868}) 2024-05-15T13:25:26.785 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2922959898}) 2024-05-15T13:25:26.785 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=804423591}) 2024-05-15T13:25:26.785 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2235537078}) 2024-05-15T13:25:26.785 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2668198477}) 2024-05-15T13:25:26.786 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=939719702}) 2024-05-15T13:25:26.786 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=617245647}) 2024-05-15T13:25:26.786 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3783645496}) 2024-05-15T13:25:26.786 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3317023565}) 2024-05-15T13:25:26.786 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1481337511}) 2024-05-15T13:25:26.786 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=97,mdsmap=3} crc {logm=1122068573,mdsmap=3984425268}) 2024-05-15T13:25:26.786 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=13,mon_config_key=66} crc {mgr=76093979,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=2519129591,mon_config_key=1488343634}) 2024-05-15T13:25:26.786 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=287011682}) 2024-05-15T13:25:26.786 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2706520274}) 2024-05-15T13:25:26.786 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=9,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=73} crc {mon_config_key=3918367541,monmap=2435304556,osd_metadata=1212035326,osd_pg_creating=162581160,osdmap=3722839559}) 2024-05-15T13:25:26.786 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3547802431}) 2024-05-15T13:25:26.786 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=45} crc {osdmap=610611873}) 2024-05-15T13:25:26.786 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:26 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3678544733' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:25:26.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: osdmap e874: 12 total, 12 up, 10 in 2024-05-15T13:25:26.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=59} crc {auth=3727032478,config=1059521453,health=656271526,kv=2771505923}) 2024-05-15T13:25:26.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=562633052}) 2024-05-15T13:25:26.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3974286832}) 2024-05-15T13:25:26.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2760381946}) 2024-05-15T13:25:26.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1195902242}) 2024-05-15T13:25:26.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3547126758}) 2024-05-15T13:25:26.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=137419636}) 2024-05-15T13:25:26.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2742590085}) 2024-05-15T13:25:26.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=354886503}) 2024-05-15T13:25:26.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=219649518}) 2024-05-15T13:25:26.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3732176288}) 2024-05-15T13:25:26.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=846292397}) 2024-05-15T13:25:26.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1674790404}) 2024-05-15T13:25:26.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3571758622}) 2024-05-15T13:25:26.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2048867150}) 2024-05-15T13:25:26.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=947768696}) 2024-05-15T13:25:26.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4209590272}) 2024-05-15T13:25:26.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3206362821}) 2024-05-15T13:25:26.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=797841427}) 2024-05-15T13:25:26.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2569765564}) 2024-05-15T13:25:26.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2552906282}) 2024-05-15T13:25:26.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1113067590}) 2024-05-15T13:25:26.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3586943393}) 2024-05-15T13:25:26.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4010813643}) 2024-05-15T13:25:26.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=68202620}) 2024-05-15T13:25:26.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2195205994}) 2024-05-15T13:25:26.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=269475152}) 2024-05-15T13:25:26.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2196281583}) 2024-05-15T13:25:26.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3795383125}) 2024-05-15T13:25:26.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=336187104}) 2024-05-15T13:25:26.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1062473678}) 2024-05-15T13:25:26.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3127181941}) 2024-05-15T13:25:26.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1403669791}) 2024-05-15T13:25:26.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2009925847}) 2024-05-15T13:25:26.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4133727929}) 2024-05-15T13:25:26.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3039099997}) 2024-05-15T13:25:26.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3412257710}) 2024-05-15T13:25:26.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=183000082}) 2024-05-15T13:25:26.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3813338609}) 2024-05-15T13:25:26.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=282467982}) 2024-05-15T13:25:26.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2783874054}) 2024-05-15T13:25:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3731307992}) 2024-05-15T13:25:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1218932475}) 2024-05-15T13:25:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=52465580}) 2024-05-15T13:25:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=251017902}) 2024-05-15T13:25:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2288950160}) 2024-05-15T13:25:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1690870073}) 2024-05-15T13:25:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1280297524}) 2024-05-15T13:25:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=459506690}) 2024-05-15T13:25:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2425002234}) 2024-05-15T13:25:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1423607182}) 2024-05-15T13:25:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1693795940}) 2024-05-15T13:25:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=209194341}) 2024-05-15T13:25:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1606181704}) 2024-05-15T13:25:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1581111896}) 2024-05-15T13:25:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2109747251}) 2024-05-15T13:25:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1159425918}) 2024-05-15T13:25:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=664390873}) 2024-05-15T13:25:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3282106898}) 2024-05-15T13:25:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4119617330}) 2024-05-15T13:25:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4041446488}) 2024-05-15T13:25:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=198309354}) 2024-05-15T13:25:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1698740385}) 2024-05-15T13:25:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=372269868}) 2024-05-15T13:25:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2922959898}) 2024-05-15T13:25:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=804423591}) 2024-05-15T13:25:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2235537078}) 2024-05-15T13:25:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2668198477}) 2024-05-15T13:25:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=939719702}) 2024-05-15T13:25:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=617245647}) 2024-05-15T13:25:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3783645496}) 2024-05-15T13:25:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3317023565}) 2024-05-15T13:25:26.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1481337511}) 2024-05-15T13:25:26.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=97,mdsmap=3} crc {logm=1122068573,mdsmap=3984425268}) 2024-05-15T13:25:26.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=13,mon_config_key=66} crc {mgr=76093979,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=2519129591,mon_config_key=1488343634}) 2024-05-15T13:25:26.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=287011682}) 2024-05-15T13:25:26.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2706520274}) 2024-05-15T13:25:26.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=9,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=73} crc {mon_config_key=3918367541,monmap=2435304556,osd_metadata=1212035326,osd_pg_creating=162581160,osdmap=3722839559}) 2024-05-15T13:25:26.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3547802431}) 2024-05-15T13:25:26.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=45} crc {osdmap=610611873}) 2024-05-15T13:25:26.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:26 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3678544733' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:25:27.531 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:27 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:25:27.531 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:27 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3678544733' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:25:27.531 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:27 smithi027 ceph-mon[39795]: osdmap e875: 12 total, 12 up, 10 in 2024-05-15T13:25:27.531 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:27 smithi027 ceph-mon[39795]: pgmap v2248: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 37 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 48807/300144 objects misplaced (16.261%); 54 MiB/s, 866 objects/s recovering 2024-05-15T13:25:27.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:27 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:25:27.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:27 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3678544733' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:25:27.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:27 smithi177 ceph-mon[44303]: osdmap e875: 12 total, 12 up, 10 in 2024-05-15T13:25:27.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:27 smithi177 ceph-mon[44303]: pgmap v2248: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 37 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 48807/300144 objects misplaced (16.261%); 54 MiB/s, 866 objects/s recovering 2024-05-15T13:25:27.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:27 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:25:27.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:27 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3678544733' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:25:27.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:27 smithi120 ceph-mon[43961]: osdmap e875: 12 total, 12 up, 10 in 2024-05-15T13:25:27.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:27 smithi120 ceph-mon[43961]: pgmap v2248: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 37 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 48807/300144 objects misplaced (16.261%); 54 MiB/s, 866 objects/s recovering 2024-05-15T13:25:29.055 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:25:29.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:29.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:25:29.160 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:25:29.224 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:29.265 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:25:29.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:29.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:25:29.374 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:25:29.409 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:29.611 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:25:29.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:29.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:25:29.716 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:25:29.743 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:29.763 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:25:29.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:29.766 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:25:29.868 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:25:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:30.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:29 smithi027 ceph-mon[39795]: pgmap v2249: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 47890/300144 objects misplaced (15.956%); 52 MiB/s, 825 objects/s recovering 2024-05-15T13:25:30.046 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:29 smithi120 ceph-mon[43961]: pgmap v2249: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 47890/300144 objects misplaced (15.956%); 52 MiB/s, 825 objects/s recovering 2024-05-15T13:25:30.097 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:25:30.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:30.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:25:30.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:29 smithi177 ceph-mon[44303]: pgmap v2249: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 47890/300144 objects misplaced (15.956%); 52 MiB/s, 825 objects/s recovering 2024-05-15T13:25:30.203 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:25:30.245 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:30.272 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:25:30.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:30.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:25:30.376 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:25:30.405 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:30.583 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:25:30.583 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:25:30.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:30.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:25:30.624 INFO:teuthology.orchestra.run.smithi027.stderr:no valid command found; 10 closest matches: 2024-05-15T13:25:30.624 INFO:teuthology.orchestra.run.smithi027.stderr:0 2024-05-15T13:25:30.624 INFO:teuthology.orchestra.run.smithi027.stderr:1 2024-05-15T13:25:30.624 INFO:teuthology.orchestra.run.smithi027.stderr:2 2024-05-15T13:25:30.625 INFO:teuthology.orchestra.run.smithi027.stderr:abort 2024-05-15T13:25:30.625 INFO:teuthology.orchestra.run.smithi027.stderr:assert 2024-05-15T13:25:30.625 INFO:teuthology.orchestra.run.smithi027.stderr:bench [] [] [] [] 2024-05-15T13:25:30.625 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs debug_inject_read_zeros 2024-05-15T13:25:30.625 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs files list 2024-05-15T13:25:30.625 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs stats 2024-05-15T13:25:30.625 INFO:teuthology.orchestra.run.smithi027.stderr:bluestore allocator dump block 2024-05-15T13:25:30.625 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: invalid command 2024-05-15T13:25:30.688 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:25:30.717 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:30.842 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:25:30.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:30.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:25:30.946 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:25:30.973 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:30.992 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:25:30.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:30.997 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:25:31.098 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:25:31.124 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:31.142 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:25:31.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:31.145 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:25:31.246 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:25:31.285 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:31.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:31.336 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:25:31.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:25:31.437 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:25:31.464 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:31.483 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:25:31.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:31.487 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:25:31.588 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:25:31.637 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:31.671 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:25:31.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:31.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:25:31.781 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:25:31.812 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:31.936 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:25:31.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:31.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:25:31.958 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T13:25:31.959 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T13:25:31.959 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-05-15T13:25:32.039 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:25:32.045 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:31 smithi120 ceph-mon[43961]: pgmap v2250: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 47284/300144 objects misplaced (15.754%); 21 MiB/s, 331 objects/s recovering 2024-05-15T13:25:32.064 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:31 smithi177 ceph-mon[44303]: pgmap v2250: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 47284/300144 objects misplaced (15.754%); 21 MiB/s, 331 objects/s recovering 2024-05-15T13:25:32.066 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:32.079 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:31 smithi027 ceph-mon[39795]: pgmap v2250: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 47284/300144 objects misplaced (15.754%); 21 MiB/s, 331 objects/s recovering 2024-05-15T13:25:32.086 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:25:32.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:32.089 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:25:32.106 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:25:32.189 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:25:32.220 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:32.571 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:25:32.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:32.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:25:32.581 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:25:32.676 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:25:32.702 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:32.721 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:25:32.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:32.724 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:25:32.825 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:25:32.851 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:32.869 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:25:32.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:32.872 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:25:32.973 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:25:33.026 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:33.079 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:25:33.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:33.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:25:33.184 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:25:33.225 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:33.256 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:25:33.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:33.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:25:33.360 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:25:33.390 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:34.004 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:33 smithi027 ceph-mon[39795]: pgmap v2251: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 46866/300144 objects misplaced (15.615%); 18 MiB/s, 293 objects/s recovering 2024-05-15T13:25:34.125 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:25:34.125 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T13:25:34.125 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-05-15T13:25:34.125 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:25:34.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:33 smithi177 ceph-mon[44303]: pgmap v2251: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 46866/300144 objects misplaced (15.615%); 18 MiB/s, 293 objects/s recovering 2024-05-15T13:25:34.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:33 smithi120 ceph-mon[43961]: pgmap v2251: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 46866/300144 objects misplaced (15.615%); 18 MiB/s, 293 objects/s recovering 2024-05-15T13:25:34.357 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:25:34.802 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-05-15T13:25:35.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:35 smithi177 ceph-mon[44303]: pgmap v2252: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 46250/300144 objects misplaced (15.409%); 20 MiB/s, 315 objects/s recovering 2024-05-15T13:25:35.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:35 smithi027 ceph-mon[39795]: pgmap v2252: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 46250/300144 objects misplaced (15.409%); 20 MiB/s, 315 objects/s recovering 2024-05-15T13:25:35.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:35 smithi120 ceph-mon[43961]: pgmap v2252: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 46250/300144 objects misplaced (15.409%); 20 MiB/s, 315 objects/s recovering 2024-05-15T13:25:36.142 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:25:36.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:36.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:25:36.247 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:25:36.277 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:36.350 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:25:36.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:36.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:25:36.456 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:25:36.484 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:36.626 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:25:36.632 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:25:36.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:36.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:25:36.655 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:25:36.737 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:25:36.767 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:36.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:36 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/302905921' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:25:36.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:36 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:25:36.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:36 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/302905921' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:25:36.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:36 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:25:37.044 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:36 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/302905921' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:25:37.045 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:36 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:25:37.316 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:25:37.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:37.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:25:37.350 INFO:teuthology.orchestra.run.smithi027.stderr:no valid command found; 10 closest matches: 2024-05-15T13:25:37.350 INFO:teuthology.orchestra.run.smithi027.stderr:0 2024-05-15T13:25:37.350 INFO:teuthology.orchestra.run.smithi027.stderr:1 2024-05-15T13:25:37.351 INFO:teuthology.orchestra.run.smithi027.stderr:2 2024-05-15T13:25:37.351 INFO:teuthology.orchestra.run.smithi027.stderr:abort 2024-05-15T13:25:37.351 INFO:teuthology.orchestra.run.smithi027.stderr:assert 2024-05-15T13:25:37.351 INFO:teuthology.orchestra.run.smithi027.stderr:bench [] [] [] [] 2024-05-15T13:25:37.351 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs debug_inject_read_zeros 2024-05-15T13:25:37.351 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs files list 2024-05-15T13:25:37.351 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs stats 2024-05-15T13:25:37.351 INFO:teuthology.orchestra.run.smithi027.stderr:bluestore allocator dump block 2024-05-15T13:25:37.351 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: invalid command 2024-05-15T13:25:37.421 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:25:37.738 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:37.769 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:25:37.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:37.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:25:37.876 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:25:37.932 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:37 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:25:37.933 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:37 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:25:37.933 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:37 smithi120 ceph-mon[43961]: osdmap e876: 12 total, 12 up, 10 in 2024-05-15T13:25:37.933 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:37 smithi120 ceph-mon[43961]: pgmap v2254: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 45833/300144 objects misplaced (15.270%); 19 MiB/s, 297 objects/s recovering 2024-05-15T13:25:37.939 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:37.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:37 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:25:37.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:37 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:25:37.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:37 smithi177 ceph-mon[44303]: osdmap e876: 12 total, 12 up, 10 in 2024-05-15T13:25:37.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:37 smithi177 ceph-mon[44303]: pgmap v2254: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 45833/300144 objects misplaced (15.270%); 19 MiB/s, 297 objects/s recovering 2024-05-15T13:25:37.965 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:25:37.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:37.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:25:38.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:37 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:25:38.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:37 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:25:38.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:37 smithi027 ceph-mon[39795]: osdmap e876: 12 total, 12 up, 10 in 2024-05-15T13:25:38.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:37 smithi027 ceph-mon[39795]: pgmap v2254: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 45833/300144 objects misplaced (15.270%); 19 MiB/s, 297 objects/s recovering 2024-05-15T13:25:38.071 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:25:38.097 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:38.117 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:25:38.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:38.120 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:25:38.221 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:25:38.258 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:38.300 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:25:38.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:38.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:25:38.406 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:25:38.437 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:38.537 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:25:38.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:38.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:25:38.560 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T13:25:38.561 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T13:25:38.561 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-05-15T13:25:38.641 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:25:38.673 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:39.055 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:25:39.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:39.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:25:39.159 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:25:39.207 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:39.248 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:25:39.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:39.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:25:39.353 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:25:39.381 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:39.584 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:25:39.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:39.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:25:39.688 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:25:39.717 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:39.841 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:25:39.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:39.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:25:39.945 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:25:39.971 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:39.991 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:25:39.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:39.994 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:25:40.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:39 smithi027 ceph-mon[39795]: pgmap v2255: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 45833/300144 objects misplaced (15.270%); 13 MiB/s, 205 objects/s recovering 2024-05-15T13:25:40.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:39 smithi027 ceph-mon[39795]: osdmap e877: 12 total, 12 up, 10 in 2024-05-15T13:25:40.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:39 smithi120 ceph-mon[43961]: pgmap v2255: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 45833/300144 objects misplaced (15.270%); 13 MiB/s, 205 objects/s recovering 2024-05-15T13:25:40.041 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:39 smithi120 ceph-mon[43961]: osdmap e877: 12 total, 12 up, 10 in 2024-05-15T13:25:40.094 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:25:40.104 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:25:40.130 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:25:40.130 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T13:25:40.130 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-05-15T13:25:40.131 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-05-15T13:25:40.131 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:25:40.140 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:40.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:39 smithi177 ceph-mon[44303]: pgmap v2255: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 45833/300144 objects misplaced (15.270%); 13 MiB/s, 205 objects/s recovering 2024-05-15T13:25:40.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:39 smithi177 ceph-mon[44303]: osdmap e877: 12 total, 12 up, 10 in 2024-05-15T13:25:40.282 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:25:40.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:40.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:25:40.390 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:25:40.416 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:40.436 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:25:40.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:40.439 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:25:40.540 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:25:40.567 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:40.587 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:25:40.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:40.590 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:25:40.691 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:25:40.742 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:40.779 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:25:40.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:40.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:25:40.886 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:25:40.912 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:40.931 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:25:40.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:40.934 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:25:41.035 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:25:41.090 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:41.121 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:25:41.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:41.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:25:41.230 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:25:41.256 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:41.275 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:25:41.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:41.278 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:25:41.379 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:25:41.435 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:41.463 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:25:41.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:41.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:25:41.568 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:25:41.592 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:25:41.608 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:41.640 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:25:41.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:41.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:25:41.749 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:25:41.799 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:41.863 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:25:41.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:41.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:25:41.925 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-05-15T13:25:41.967 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:25:42.021 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:42.057 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:25:42.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:42.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:25:42.062 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:41 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:25:42.062 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:41 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:25:42.062 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:41 smithi120 ceph-mon[43961]: pgmap v2257: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 45199/300144 objects misplaced (15.059%); 13 MiB/s, 208 objects/s recovering 2024-05-15T13:25:42.062 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:41 smithi120 ceph-mon[43961]: osdmap e878: 12 total, 12 up, 10 in 2024-05-15T13:25:42.123 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:41 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:25:42.124 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:41 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:25:42.124 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:41 smithi027 ceph-mon[39795]: pgmap v2257: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 45199/300144 objects misplaced (15.059%); 13 MiB/s, 208 objects/s recovering 2024-05-15T13:25:42.124 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:41 smithi027 ceph-mon[39795]: osdmap e878: 12 total, 12 up, 10 in 2024-05-15T13:25:42.162 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:25:42.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:41 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:25:42.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:41 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:25:42.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:41 smithi177 ceph-mon[44303]: pgmap v2257: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 45199/300144 objects misplaced (15.059%); 13 MiB/s, 208 objects/s recovering 2024-05-15T13:25:42.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:41 smithi177 ceph-mon[44303]: osdmap e878: 12 total, 12 up, 10 in 2024-05-15T13:25:42.188 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:42.207 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:25:42.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:42.210 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:25:42.311 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:25:42.337 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:42.356 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:25:42.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:42.359 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:25:42.399 INFO:teuthology.orchestra.run.smithi027.stderr:no valid command found; 10 closest matches: 2024-05-15T13:25:42.399 INFO:teuthology.orchestra.run.smithi027.stderr:0 2024-05-15T13:25:42.399 INFO:teuthology.orchestra.run.smithi027.stderr:1 2024-05-15T13:25:42.399 INFO:teuthology.orchestra.run.smithi027.stderr:2 2024-05-15T13:25:42.399 INFO:teuthology.orchestra.run.smithi027.stderr:abort 2024-05-15T13:25:42.399 INFO:teuthology.orchestra.run.smithi027.stderr:assert 2024-05-15T13:25:42.399 INFO:teuthology.orchestra.run.smithi027.stderr:bench [] [] [] [] 2024-05-15T13:25:42.399 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs debug_inject_read_zeros 2024-05-15T13:25:42.399 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs files list 2024-05-15T13:25:42.399 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs stats 2024-05-15T13:25:42.399 INFO:teuthology.orchestra.run.smithi027.stderr:bluestore allocator dump block 2024-05-15T13:25:42.399 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: invalid command 2024-05-15T13:25:42.400 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:25:42.402 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:25:42.460 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:25:42.491 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:44.166 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:43 smithi027 ceph-mon[39795]: pgmap v2259: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 44765/300144 objects misplaced (14.915%); 11 MiB/s, 175 objects/s recovering 2024-05-15T13:25:44.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:43 smithi177 ceph-mon[44303]: pgmap v2259: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 44765/300144 objects misplaced (14.915%); 11 MiB/s, 175 objects/s recovering 2024-05-15T13:25:44.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:43 smithi120 ceph-mon[43961]: pgmap v2259: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 44765/300144 objects misplaced (14.915%); 11 MiB/s, 175 objects/s recovering 2024-05-15T13:25:45.232 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T13:25:45.233 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T13:25:45.233 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-05-15T13:25:45.242 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:25:45.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:45.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:25:45.346 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:25:45.374 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:45.497 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:25:45.499 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:45 smithi027 ceph-mon[39795]: osdmap e879: 12 total, 12 up, 10 in 2024-05-15T13:25:45.499 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:45 smithi027 ceph-mon[39795]: pgmap v2261: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 44115/300144 objects misplaced (14.698%); 18 MiB/s, 286 objects/s recovering 2024-05-15T13:25:45.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:45.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:25:45.601 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:25:45.631 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:45.638 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:25:45.638 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-05-15T13:25:45.638 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-05-15T13:25:45.638 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:25:45.638 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.638 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:197998 7.4 7:29af6335:::benchmark_data_smithi177_134484_object88933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e713)", 2024-05-15T13:25:45.638 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:17:36.732280+0000", 2024-05-15T13:25:45.639 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 488.90285711400003, 2024-05-15T13:25:45.639 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0520236970000001, 2024-05-15T13:25:45.639 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:25:45.639 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:25:45.639 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:25:45.639 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:25:45.639 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:25:45.639 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 197998 2024-05-15T13:25:45.639 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.639 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:25:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:25:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.732280+0000", 2024-05-15T13:25:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:25:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.732280+0000", 2024-05-15T13:25:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:25:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.732281+0000", 2024-05-15T13:25:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9499999999999998e-07 2024-05-15T13:25:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:25:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.732283+0000", 2024-05-15T13:25:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5450000000000002e-06 2024-05-15T13:25:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:25:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.732290+0000", 2024-05-15T13:25:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4729999999999997e-06 2024-05-15T13:25:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:25:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.732295+0000", 2024-05-15T13:25:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6779999999999997e-06 2024-05-15T13:25:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.732356+0000", 2024-05-15T13:25:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0906999999999998e-05 2024-05-15T13:25:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:25:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.732505+0000", 2024-05-15T13:25:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014865700000000001 2024-05-15T13:25:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 1,9", 2024-05-15T13:25:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.732634+0000", 2024-05-15T13:25:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012876000000000001 2024-05-15T13:25:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.743037+0000", 2024-05-15T13:25:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010403333000000001 2024-05-15T13:25:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:25:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.743050+0000", 2024-05-15T13:25:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3050999999999999e-05 2024-05-15T13:25:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.784291+0000", 2024-05-15T13:25:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.041241208 2024-05-15T13:25:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:25:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.784304+0000", 2024-05-15T13:25:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.269e-05 2024-05-15T13:25:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:25:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:198002 7.4 7.e8c35024 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e713)", 2024-05-15T13:25:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:17:36.738153+0000", 2024-05-15T13:25:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 488.89698439, 2024-05-15T13:25:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0563994759999999, 2024-05-15T13:25:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:25:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:25:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:25:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:25:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:25:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 198002 2024-05-15T13:25:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:25:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:25:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.738153+0000", 2024-05-15T13:25:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:25:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.738153+0000", 2024-05-15T13:25:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:25:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.738154+0000", 2024-05-15T13:25:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3769999999999999e-06 2024-05-15T13:25:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:25:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.738159+0000", 2024-05-15T13:25:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.651e-06 2024-05-15T13:25:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:25:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.738171+0000", 2024-05-15T13:25:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1792e-05 2024-05-15T13:25:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:25:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.738180+0000", 2024-05-15T13:25:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7299999999999994e-06 2024-05-15T13:25:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.743241+0000", 2024-05-15T13:25:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00506122 2024-05-15T13:25:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:25:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.743249+0000", 2024-05-15T13:25:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3359999999999995e-06 2024-05-15T13:25:45.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.649 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.649 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.649 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.794543+0000", 2024-05-15T13:25:45.649 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0512938540000001 2024-05-15T13:25:45.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.649 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.649 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:25:45.649 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.794553+0000", 2024-05-15T13:25:45.649 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.516e-06 2024-05-15T13:25:45.649 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:25:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:198003 7.4 7.a1de35c4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e713)", 2024-05-15T13:25:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:17:36.738195+0000", 2024-05-15T13:25:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 488.89694275900001, 2024-05-15T13:25:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.056436962, 2024-05-15T13:25:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:25:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:25:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:25:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:25:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:25:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 198003 2024-05-15T13:25:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:25:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:25:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.738195+0000", 2024-05-15T13:25:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:25:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.738195+0000", 2024-05-15T13:25:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:25:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.738195+0000", 2024-05-15T13:25:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2599999999999998e-07 2024-05-15T13:25:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:25:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.738198+0000", 2024-05-15T13:25:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4229999999999999e-06 2024-05-15T13:25:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:25:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.738204+0000", 2024-05-15T13:25:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3550000000000001e-06 2024-05-15T13:25:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:25:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.738209+0000", 2024-05-15T13:25:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5319999999999997e-06 2024-05-15T13:25:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.743845+0000", 2024-05-15T13:25:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0056355650000000004 2024-05-15T13:25:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:25:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.743854+0000", 2024-05-15T13:25:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3789999999999995e-06 2024-05-15T13:25:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.794625+0000", 2024-05-15T13:25:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.050770083 2024-05-15T13:25:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:25:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.794632+0000", 2024-05-15T13:25:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.199e-06 2024-05-15T13:25:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:25:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:198013 7.4 7.776a96f4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e714)", 2024-05-15T13:25:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:17:36.758925+0000", 2024-05-15T13:25:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 488.87621280000002, 2024-05-15T13:25:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.035730035, 2024-05-15T13:25:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:25:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:25:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:25:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:25:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:25:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 198013 2024-05-15T13:25:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:25:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:25:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.758925+0000", 2024-05-15T13:25:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:25:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.758925+0000", 2024-05-15T13:25:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:25:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.758926+0000", 2024-05-15T13:25:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2640000000000001e-06 2024-05-15T13:25:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:25:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.758932+0000", 2024-05-15T13:25:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6080000000000003e-06 2024-05-15T13:25:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:25:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.758941+0000", 2024-05-15T13:25:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1800000000000002e-06 2024-05-15T13:25:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:25:45.658 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.758958+0000", 2024-05-15T13:25:45.658 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7374e-05 2024-05-15T13:25:45.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.658 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.658 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.658 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.759464+0000", 2024-05-15T13:25:45.658 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00050630800000000002 2024-05-15T13:25:45.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.658 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.658 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:25:45.659 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.759474+0000", 2024-05-15T13:25:45.659 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8069999999999994e-06 2024-05-15T13:25:45.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.659 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.659 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.659 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.794648+0000", 2024-05-15T13:25:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0351739069999999 2024-05-15T13:25:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:25:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.794655+0000", 2024-05-15T13:25:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5869999999999998e-06 2024-05-15T13:25:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:25:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26044.0:198021 7.4 7.e4a21094 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e714)", 2024-05-15T13:25:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:17:36.769084+0000", 2024-05-15T13:25:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 488.86605348400002, 2024-05-15T13:25:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.025592815, 2024-05-15T13:25:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:25:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:25:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:25:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26044", 2024-05-15T13:25:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/90514387", 2024-05-15T13:25:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 198021 2024-05-15T13:25:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:25:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:25:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.769084+0000", 2024-05-15T13:25:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:25:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.769084+0000", 2024-05-15T13:25:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:25:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.769085+0000", 2024-05-15T13:25:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0299999999999997e-07 2024-05-15T13:25:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:25:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.769089+0000", 2024-05-15T13:25:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3710000000000002e-06 2024-05-15T13:25:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:25:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.769100+0000", 2024-05-15T13:25:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0913e-05 2024-05-15T13:25:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:25:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.769107+0000", 2024-05-15T13:25:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6919999999999996e-06 2024-05-15T13:25:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.773105+0000", 2024-05-15T13:25:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039984110000000003 2024-05-15T13:25:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:25:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:36.773120+0000", 2024-05-15T13:25:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5073e-05 2024-05-15T13:25:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.794671+0000", 2024-05-15T13:25:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0215501279999999 2024-05-15T13:25:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:25:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:17:37.794677+0000", 2024-05-15T13:25:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3239999999999999e-06 2024-05-15T13:25:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:25:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:65954 8.d 8.18a510cd (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e770)", 2024-05-15T13:25:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:20:14.818898+0000", 2024-05-15T13:25:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 330.81623992999999, 2024-05-15T13:25:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.003652948, 2024-05-15T13:25:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:25:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:25:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:25:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:25:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:25:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 65954 2024-05-15T13:25:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:25:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:25:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.818898+0000", 2024-05-15T13:25:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:25:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.818898+0000", 2024-05-15T13:25:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:25:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.818899+0000", 2024-05-15T13:25:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.311e-06 2024-05-15T13:25:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:25:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.818939+0000", 2024-05-15T13:25:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0302999999999997e-05 2024-05-15T13:25:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:25:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.818957+0000", 2024-05-15T13:25:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7821000000000001e-05 2024-05-15T13:25:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:25:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.818966+0000", 2024-05-15T13:25:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9930000000000001e-06 2024-05-15T13:25:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.820769+0000", 2024-05-15T13:25:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001802723 2024-05-15T13:25:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:25:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.820774+0000", 2024-05-15T13:25:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.507e-06 2024-05-15T13:25:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:15.822541+0000", 2024-05-15T13:25:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0017663000000001 2024-05-15T13:25:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:25:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:15.822551+0000", 2024-05-15T13:25:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9899999999999992e-06 2024-05-15T13:25:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:25:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:65959 8.d 8.5c53412d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e770)", 2024-05-15T13:25:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:20:14.829286+0000", 2024-05-15T13:25:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 330.80585142400002, 2024-05-15T13:25:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99330206899999995, 2024-05-15T13:25:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:25:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:25:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:25:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:25:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:25:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 65959 2024-05-15T13:25:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:25:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:25:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.829286+0000", 2024-05-15T13:25:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:25:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.829286+0000", 2024-05-15T13:25:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:25:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.829287+0000", 2024-05-15T13:25:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1399999999999997e-07 2024-05-15T13:25:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:25:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.829328+0000", 2024-05-15T13:25:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1276000000000003e-05 2024-05-15T13:25:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:25:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.829336+0000", 2024-05-15T13:25:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2819999999999996e-06 2024-05-15T13:25:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:25:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.829343+0000", 2024-05-15T13:25:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6980000000000001e-06 2024-05-15T13:25:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.829408+0000", 2024-05-15T13:25:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4734999999999996e-05 2024-05-15T13:25:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:25:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.829417+0000", 2024-05-15T13:25:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5699999999999999e-06 2024-05-15T13:25:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:15.822579+0000", 2024-05-15T13:25:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99316155800000006 2024-05-15T13:25:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:25:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:15.822588+0000", 2024-05-15T13:25:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3360000000000001e-06 2024-05-15T13:25:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:25:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:65954 8.d 8.18a510cd (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e770)", 2024-05-15T13:25:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:20:14.834471+0000", 2024-05-15T13:25:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 330.80066610599999, 2024-05-15T13:25:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.98820525699999995, 2024-05-15T13:25:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:25:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:25:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:25:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:25:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:25:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 65954 2024-05-15T13:25:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:25:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:25:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.834471+0000", 2024-05-15T13:25:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:25:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.834471+0000", 2024-05-15T13:25:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:25:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.834473+0000", 2024-05-15T13:25:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.217e-06 2024-05-15T13:25:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:25:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.834699+0000", 2024-05-15T13:25:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00022629 2024-05-15T13:25:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:25:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.834722+0000", 2024-05-15T13:25:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3295e-05 2024-05-15T13:25:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:25:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.834742+0000", 2024-05-15T13:25:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9942999999999999e-05 2024-05-15T13:25:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.834972+0000", 2024-05-15T13:25:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00022969200000000001 2024-05-15T13:25:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:25:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.834981+0000", 2024-05-15T13:25:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0210000000000005e-06 2024-05-15T13:25:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:15.822663+0000", 2024-05-15T13:25:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.98768213500000002 2024-05-15T13:25:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:25:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:15.822677+0000", 2024-05-15T13:25:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3664e-05 2024-05-15T13:25:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:25:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:65959 8.d 8.5c53412d (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e770)", 2024-05-15T13:25:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:20:14.834860+0000", 2024-05-15T13:25:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 330.800277757, 2024-05-15T13:25:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.98785461100000005, 2024-05-15T13:25:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:25:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:25:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:25:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:25:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:25:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 65959 2024-05-15T13:25:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:25:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:25:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.834860+0000", 2024-05-15T13:25:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:25:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.834860+0000", 2024-05-15T13:25:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:25:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.834860+0000", 2024-05-15T13:25:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9800000000000003e-07 2024-05-15T13:25:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:25:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.834904+0000", 2024-05-15T13:25:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3433000000000002e-05 2024-05-15T13:25:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:25:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.834913+0000", 2024-05-15T13:25:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1670000000000005e-06 2024-05-15T13:25:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:25:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.834919+0000", 2024-05-15T13:25:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3029999999999996e-06 2024-05-15T13:25:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.835004+0000", 2024-05-15T13:25:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5179000000000006e-05 2024-05-15T13:25:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:25:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.835010+0000", 2024-05-15T13:25:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6289999999999998e-06 2024-05-15T13:25:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:15.822705+0000", 2024-05-15T13:25:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.98769449300000001 2024-05-15T13:25:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:25:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:15.822714+0000", 2024-05-15T13:25:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8090000000000004e-06 2024-05-15T13:25:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:25:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:65961 8.d 8.ad0d2acd (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e770)", 2024-05-15T13:25:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:20:14.836164+0000", 2024-05-15T13:25:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 330.79897366699998, 2024-05-15T13:25:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.98665524500000001, 2024-05-15T13:25:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:25:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-05-15T13:25:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:25:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:25:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:25:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 65961 2024-05-15T13:25:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:25:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:25:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.836164+0000", 2024-05-15T13:25:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:25:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.836164+0000", 2024-05-15T13:25:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:25:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.836165+0000", 2024-05-15T13:25:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.128e-06 2024-05-15T13:25:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:25:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.836326+0000", 2024-05-15T13:25:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000160588 2024-05-15T13:25:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:25:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.836340+0000", 2024-05-15T13:25:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4667e-05 2024-05-15T13:25:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:25:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.836349+0000", 2024-05-15T13:25:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5620000000000004e-06 2024-05-15T13:25:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.836403+0000", 2024-05-15T13:25:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3875000000000002e-05 2024-05-15T13:25:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:25:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:14.836409+0000", 2024-05-15T13:25:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9200000000000001e-06 2024-05-15T13:25:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:15.822809+0000", 2024-05-15T13:25:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.98640069200000002 2024-05-15T13:25:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:25:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:20:15.822819+0000", 2024-05-15T13:25:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8130000000000007e-06 2024-05-15T13:25:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:25:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:150890 8.d 8:bd1a1876:::benchmark_data_smithi177_149673_object60828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e791)", 2024-05-15T13:25:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:21:55.270507+0000", 2024-05-15T13:25:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 230.364630871, 2024-05-15T13:25:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.017799788, 2024-05-15T13:25:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:25:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:25:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:25:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:25:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:25:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 150890 2024-05-15T13:25:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:25:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:25:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:55.270507+0000", 2024-05-15T13:25:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:25:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:55.270507+0000", 2024-05-15T13:25:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:25:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:55.270508+0000", 2024-05-15T13:25:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7e-06 2024-05-15T13:25:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:25:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:55.270515+0000", 2024-05-15T13:25:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0979999999999996e-06 2024-05-15T13:25:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:25:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:55.270529+0000", 2024-05-15T13:25:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3508e-05 2024-05-15T13:25:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:25:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:55.270537+0000", 2024-05-15T13:25:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5210000000000002e-06 2024-05-15T13:25:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:55.271061+0000", 2024-05-15T13:25:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00052312799999999998 2024-05-15T13:25:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:25:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:55.271068+0000", 2024-05-15T13:25:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3599999999999998e-06 2024-05-15T13:25:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:56.277999+0000", 2024-05-15T13:25:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0069305669999999 2024-05-15T13:25:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:25:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:56.278159+0000", 2024-05-15T13:25:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000160218 2024-05-15T13:25:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,6", 2024-05-15T13:25:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:56.278259+0000", 2024-05-15T13:25:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000100369 2024-05-15T13:25:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:56.288163+0000", 2024-05-15T13:25:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0099034120000000003 2024-05-15T13:25:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:56.288208+0000", 2024-05-15T13:25:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5232999999999999e-05 2024-05-15T13:25:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:25:45.695 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:56.288219+0000", 2024-05-15T13:25:45.695 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0967e-05 2024-05-15T13:25:45.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.695 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.695 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:25:45.695 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:56.288264+0000", 2024-05-15T13:25:45.695 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5661000000000002e-05 2024-05-15T13:25:45.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.695 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.695 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:25:45.695 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:56.288306+0000", 2024-05-15T13:25:45.695 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2045999999999997e-05 2024-05-15T13:25:45.695 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.695 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:25:45.695 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.696 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.696 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.696 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:150894 8.d 8:be816ae0:::benchmark_data_smithi177_149673_object60832:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e791)", 2024-05-15T13:25:45.696 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:21:55.273305+0000", 2024-05-15T13:25:45.696 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 230.36183268600001, 2024-05-15T13:25:45.696 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.018677769, 2024-05-15T13:25:45.696 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:25:45.696 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:25:45.696 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:25:45.696 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:25:45.696 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:25:45.696 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 150894 2024-05-15T13:25:45.696 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.697 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:25:45.697 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.697 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:25:45.697 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:55.273305+0000", 2024-05-15T13:25:45.697 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.697 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.697 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.697 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:25:45.697 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:55.273305+0000", 2024-05-15T13:25:45.697 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.697 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.697 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.697 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:25:45.697 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:55.273307+0000", 2024-05-15T13:25:45.698 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2259999999999999e-06 2024-05-15T13:25:45.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.698 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.698 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:25:45.698 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:55.273316+0000", 2024-05-15T13:25:45.698 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.789e-06 2024-05-15T13:25:45.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.698 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.698 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:25:45.698 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:55.273340+0000", 2024-05-15T13:25:45.698 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4054999999999999e-05 2024-05-15T13:25:45.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.698 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.699 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:25:45.699 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:55.273351+0000", 2024-05-15T13:25:45.699 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0617e-05 2024-05-15T13:25:45.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.699 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.699 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.699 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:55.273411+0000", 2024-05-15T13:25:45.699 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0791000000000001e-05 2024-05-15T13:25:45.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.699 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.699 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:25:45.699 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:55.273419+0000", 2024-05-15T13:25:45.699 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1850000000000004e-06 2024-05-15T13:25:45.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.700 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.700 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.700 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:56.281722+0000", 2024-05-15T13:25:45.700 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.008302458 2024-05-15T13:25:45.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.700 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.700 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:25:45.700 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:56.281807+0000", 2024-05-15T13:25:45.700 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5113999999999999e-05 2024-05-15T13:25:45.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.700 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.700 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,6", 2024-05-15T13:25:45.700 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:56.281894+0000", 2024-05-15T13:25:45.700 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6947999999999995e-05 2024-05-15T13:25:45.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.701 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.701 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:45.701 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:56.288476+0000", 2024-05-15T13:25:45.701 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0065817430000000001 2024-05-15T13:25:45.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.701 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.701 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:25:45.701 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:56.288555+0000", 2024-05-15T13:25:45.701 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9711999999999998e-05 2024-05-15T13:25:45.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.701 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.701 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:45.701 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:56.291874+0000", 2024-05-15T13:25:45.702 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003318779 2024-05-15T13:25:45.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.702 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.702 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:25:45.702 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:56.291935+0000", 2024-05-15T13:25:45.702 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0608e-05 2024-05-15T13:25:45.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.702 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.702 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:25:45.702 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:56.291983+0000", 2024-05-15T13:25:45.702 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7744000000000002e-05 2024-05-15T13:25:45.702 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.702 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:25:45.702 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.703 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.703 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26299.0:150903 8.d 8:bedef465:::benchmark_data_smithi177_149673_object60841:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e791)", 2024-05-15T13:25:45.703 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:21:55.281099+0000", 2024-05-15T13:25:45.703 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 230.35403882599999, 2024-05-15T13:25:45.703 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.011080899, 2024-05-15T13:25:45.703 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:25:45.703 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:25:45.703 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:25:45.703 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26299", 2024-05-15T13:25:45.703 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/3356133509", 2024-05-15T13:25:45.703 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 150903 2024-05-15T13:25:45.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.703 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:25:45.704 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.704 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:25:45.704 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:55.281099+0000", 2024-05-15T13:25:45.704 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.704 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.704 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:25:45.704 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:55.281099+0000", 2024-05-15T13:25:45.704 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.704 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.704 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:25:45.704 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:55.281101+0000", 2024-05-15T13:25:45.705 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9130000000000001e-06 2024-05-15T13:25:45.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.705 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.705 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:25:45.705 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:55.281116+0000", 2024-05-15T13:25:45.705 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5831000000000001e-05 2024-05-15T13:25:45.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.705 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.705 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:25:45.705 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:55.281140+0000", 2024-05-15T13:25:45.705 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3110000000000001e-05 2024-05-15T13:25:45.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.705 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.705 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:25:45.706 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:55.281150+0000", 2024-05-15T13:25:45.706 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0631e-05 2024-05-15T13:25:45.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.706 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.706 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.706 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:55.281216+0000", 2024-05-15T13:25:45.706 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5927999999999996e-05 2024-05-15T13:25:45.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.706 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.706 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-05-15T13:25:45.706 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:55.281228+0000", 2024-05-15T13:25:45.706 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1590000000000001e-05 2024-05-15T13:25:45.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.706 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.707 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.707 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:56.284585+0000", 2024-05-15T13:25:45.707 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.003357598 2024-05-15T13:25:45.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.707 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.707 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:25:45.707 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:56.284671+0000", 2024-05-15T13:25:45.707 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5748000000000006e-05 2024-05-15T13:25:45.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.707 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.707 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 5,6", 2024-05-15T13:25:45.707 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:56.284739+0000", 2024-05-15T13:25:45.707 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8291000000000001e-05 2024-05-15T13:25:45.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.708 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.708 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:45.708 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:56.288541+0000", 2024-05-15T13:25:45.708 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038015800000000002 2024-05-15T13:25:45.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.708 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.708 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:25:45.708 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:56.288564+0000", 2024-05-15T13:25:45.708 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3604e-05 2024-05-15T13:25:45.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.708 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.708 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:25:45.708 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:56.292064+0000", 2024-05-15T13:25:45.708 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034995030000000002 2024-05-15T13:25:45.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.709 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.709 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-05-15T13:25:45.709 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:56.292132+0000", 2024-05-15T13:25:45.709 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7651999999999995e-05 2024-05-15T13:25:45.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.709 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.709 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:25:45.709 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:21:56.292180+0000", 2024-05-15T13:25:45.709 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7920000000000002e-05 2024-05-15T13:25:45.709 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.709 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:25:45.709 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.709 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.710 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDScrubReserve(1.11,GRANT,e:863,from:3,reservation_nonce:0)", 2024-05-15T13:25:45.710 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:25:10.456577+0000", 2024-05-15T13:25:45.710 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 35.178560867000002, 2024-05-15T13:25:45.710 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00068896599999999995, 2024-05-15T13:25:45.710 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:25:45.710 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:25:45.710 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:25:45.710 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.710 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:25:45.710 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.456577+0000", 2024-05-15T13:25:45.711 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.711 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.711 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:25:45.711 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.456577+0000", 2024-05-15T13:25:45.711 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.711 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.711 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:25:45.711 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.456578+0000", 2024-05-15T13:25:45.711 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1200000000000001e-07 2024-05-15T13:25:45.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.711 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.711 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:25:45.712 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.456582+0000", 2024-05-15T13:25:45.712 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0369999999999996e-06 2024-05-15T13:25:45.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.712 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.712 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:25:45.712 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.456591+0000", 2024-05-15T13:25:45.712 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3109999999999995e-06 2024-05-15T13:25:45.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.712 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.712 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:25:45.713 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.456599+0000", 2024-05-15T13:25:45.713 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8129999999999996e-06 2024-05-15T13:25:45.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.713 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.713 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.713 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.457206+0000", 2024-05-15T13:25:45.713 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00060719500000000004 2024-05-15T13:25:45.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.713 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.713 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:25:45.713 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.457223+0000", 2024-05-15T13:25:45.713 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6730000000000001e-05 2024-05-15T13:25:45.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.714 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.714 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:25:45.714 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.457266+0000", 2024-05-15T13:25:45.714 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2967999999999999e-05 2024-05-15T13:25:45.714 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.714 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:25:45.714 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.714 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.714 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.714 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDScrubReserve(1.11,GRANT,e:863,from:9,reservation_nonce:0)", 2024-05-15T13:25:45.714 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:25:10.458386+0000", 2024-05-15T13:25:45.714 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 35.176751660999997, 2024-05-15T13:25:45.714 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000253787, 2024-05-15T13:25:45.715 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:25:45.715 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:25:45.715 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:25:45.715 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.715 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:25:45.715 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.458386+0000", 2024-05-15T13:25:45.715 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.715 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.715 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.715 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:25:45.715 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.458386+0000", 2024-05-15T13:25:45.715 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.715 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.715 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.715 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:25:45.716 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.458387+0000", 2024-05-15T13:25:45.716 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0299999999999998e-07 2024-05-15T13:25:45.716 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.716 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.716 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:25:45.716 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.458390+0000", 2024-05-15T13:25:45.716 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8299999999999998e-06 2024-05-15T13:25:45.716 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.716 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.716 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:25:45.716 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.458402+0000", 2024-05-15T13:25:45.716 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1277999999999999e-05 2024-05-15T13:25:45.716 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.717 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.717 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:25:45.717 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.458409+0000", 2024-05-15T13:25:45.717 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.289e-06 2024-05-15T13:25:45.717 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.717 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.717 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.717 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.458477+0000", 2024-05-15T13:25:45.717 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7847000000000003e-05 2024-05-15T13:25:45.717 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.717 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.717 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:25:45.717 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.458492+0000", 2024-05-15T13:25:45.717 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5549999999999999e-05 2024-05-15T13:25:45.717 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.717 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.718 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:25:45.718 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.458640+0000", 2024-05-15T13:25:45.718 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014729000000000001 2024-05-15T13:25:45.718 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.718 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:25:45.718 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.718 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.718 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.718 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "rep_scrubmap(1.11 e805 from shard 9)", 2024-05-15T13:25:45.718 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:25:10.590566+0000", 2024-05-15T13:25:45.718 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 35.044571511999997, 2024-05-15T13:25:45.718 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000151667, 2024-05-15T13:25:45.719 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:25:45.719 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:25:45.719 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:25:45.719 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.719 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:25:45.719 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.590566+0000", 2024-05-15T13:25:45.719 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.719 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.719 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.719 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:25:45.719 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.590566+0000", 2024-05-15T13:25:45.719 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.719 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.719 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.719 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:25:45.719 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.590567+0000", 2024-05-15T13:25:45.720 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.384e-06 2024-05-15T13:25:45.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.720 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.720 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:25:45.720 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.590576+0000", 2024-05-15T13:25:45.720 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.064e-06 2024-05-15T13:25:45.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.720 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.720 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:25:45.720 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.590596+0000", 2024-05-15T13:25:45.720 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9624e-05 2024-05-15T13:25:45.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.720 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.720 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:25:45.721 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.590606+0000", 2024-05-15T13:25:45.721 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7489999999999993e-06 2024-05-15T13:25:45.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.721 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.721 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.721 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.590691+0000", 2024-05-15T13:25:45.721 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5085000000000003e-05 2024-05-15T13:25:45.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.721 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.721 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:25:45.721 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.590703+0000", 2024-05-15T13:25:45.721 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2269999999999999e-05 2024-05-15T13:25:45.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.721 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.721 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:25:45.722 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.590718+0000", 2024-05-15T13:25:45.722 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4491000000000001e-05 2024-05-15T13:25:45.722 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.722 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:25:45.722 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.722 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.722 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.722 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "rep_scrubmap(1.11 e805 from shard 3)", 2024-05-15T13:25:45.722 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:25:10.656774+0000", 2024-05-15T13:25:45.722 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 34.978363665000003, 2024-05-15T13:25:45.722 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014612699999999999, 2024-05-15T13:25:45.722 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:25:45.722 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:25:45.722 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:25:45.723 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.723 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:25:45.723 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.656774+0000", 2024-05-15T13:25:45.723 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.723 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.723 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.723 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:25:45.723 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.656774+0000", 2024-05-15T13:25:45.723 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:25:45.723 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.723 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.723 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:25:45.723 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.656775+0000", 2024-05-15T13:25:45.723 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.187e-06 2024-05-15T13:25:45.723 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.724 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.724 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:25:45.724 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.656784+0000", 2024-05-15T13:25:45.724 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0100000000000001e-06 2024-05-15T13:25:45.724 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.724 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.724 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:25:45.724 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.656805+0000", 2024-05-15T13:25:45.724 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0825000000000001e-05 2024-05-15T13:25:45.724 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.724 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.724 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:25:45.724 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.656815+0000", 2024-05-15T13:25:45.724 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0576e-05 2024-05-15T13:25:45.724 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.725 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.725 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:25:45.725 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.656870+0000", 2024-05-15T13:25:45.725 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4438999999999998e-05 2024-05-15T13:25:45.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.725 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.725 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:25:45.725 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.656882+0000", 2024-05-15T13:25:45.725 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2282e-05 2024-05-15T13:25:45.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:25:45.725 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:25:45.725 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-05-15T13:25:45.725 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:25:10.656920+0000", 2024-05-15T13:25:45.725 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7808000000000001e-05 2024-05-15T13:25:45.726 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.726 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:25:45.726 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.726 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:25:45.726 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:25:45.726 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:25:45.727 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:45 smithi177 ceph-mon[44303]: osdmap e879: 12 total, 12 up, 10 in 2024-05-15T13:25:45.727 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:45 smithi177 ceph-mon[44303]: pgmap v2261: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 44115/300144 objects misplaced (14.698%); 18 MiB/s, 286 objects/s recovering 2024-05-15T13:25:45.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:45.728 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:25:45.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:25:45.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:45 smithi120 ceph-mon[43961]: osdmap e879: 12 total, 12 up, 10 in 2024-05-15T13:25:45.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:45 smithi120 ceph-mon[43961]: pgmap v2261: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 44115/300144 objects misplaced (14.698%); 18 MiB/s, 286 objects/s recovering 2024-05-15T13:25:45.829 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:25:45.855 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:45.874 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:25:45.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:45.878 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:25:45.979 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:25:46.023 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:46.062 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:25:46.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:46.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:25:46.171 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:25:46.200 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:46.324 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:25:46.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:46.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:25:46.428 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:25:46.454 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:46.474 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:25:46.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:46.477 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:25:46.578 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:25:46.625 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:46.678 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:25:46.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:46.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:25:46.782 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:25:46.812 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:46.865 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:25:46.866 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:25:46.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:46.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:25:46.900 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-05-15T13:25:46.971 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:25:47.012 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:47.137 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:25:47.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:47.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:25:47.241 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:25:47.269 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:47.468 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:25:47.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:47.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:25:47.573 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:25:47.602 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:47.726 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:25:47.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:47.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:25:47.829 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:25:47.874 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:47.933 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:25:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:47.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:25:48.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:47 smithi027 ceph-mon[39795]: pgmap v2262: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 43671/300144 objects misplaced (14.550%); 15 MiB/s, 235 objects/s recovering 2024-05-15T13:25:48.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:47 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:25:48.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:47 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:25:48.038 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:25:48.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:47 smithi120 ceph-mon[43961]: pgmap v2262: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 43671/300144 objects misplaced (14.550%); 15 MiB/s, 235 objects/s recovering 2024-05-15T13:25:48.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:47 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:25:48.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:47 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:25:48.070 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:48.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:47 smithi177 ceph-mon[44303]: pgmap v2262: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 43671/300144 objects misplaced (14.550%); 15 MiB/s, 235 objects/s recovering 2024-05-15T13:25:48.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:47 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:25:48.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:47 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:25:48.193 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:25:48.193 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:25:48.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:48.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:25:48.298 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:25:48.325 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:48.343 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:25:48.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:48.346 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:25:48.447 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:25:48.497 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:48.536 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:25:48.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:48.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:25:48.645 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:25:48.672 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:48.691 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:25:48.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:48.694 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:25:48.795 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:25:48.802 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:25:48.849 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:48.884 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:25:48.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:48.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:25:48.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:48 smithi177 ceph-mon[44303]: osdmap e880: 12 total, 12 up, 10 in 2024-05-15T13:25:48.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:48 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1776610945' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:25:48.991 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:25:49.024 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:49.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:48 smithi027 ceph-mon[39795]: osdmap e880: 12 total, 12 up, 10 in 2024-05-15T13:25:49.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:48 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1776610945' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:25:49.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:48 smithi120 ceph-mon[43961]: osdmap e880: 12 total, 12 up, 10 in 2024-05-15T13:25:49.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:48 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1776610945' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:25:49.517 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 15 13:25:49 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-3[81304]: 2024-05-15T13:25:49.118+0000 7fd2ac61e700 -1 osd.3 881 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:25:25.610284+0000 front 2024-05-15T13:25:25.610343+0000 (oldest deadline 2024-05-15T13:25:49.110390+0000) 2024-05-15T13:25:49.768 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 15 13:25:49 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-0[58372]: 2024-05-15T13:25:49.516+0000 7f897b1e5700 -1 osd.0 881 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:25:23.336753+0000 front 2024-05-15T13:25:23.336781+0000 (oldest deadline 2024-05-15T13:25:49.236801+0000) 2024-05-15T13:25:49.979 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:25:49.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:49.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:25:50.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:49 smithi027 ceph-mon[39795]: pgmap v2264: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 43671/300144 objects misplaced (14.550%); 12 MiB/s, 192 objects/s recovering 2024-05-15T13:25:50.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:49 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:25:50.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:49 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1776610945' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:25:50.026 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:49 smithi027 ceph-mon[39795]: osdmap e881: 12 total, 12 up, 10 in 2024-05-15T13:25:50.041 INFO:journalctl@ceph.osd.5.smithi120.stdout:May 15 13:25:49 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-5[56546]: 2024-05-15T13:25:49.572+0000 7f0f46d17700 -1 osd.5 881 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:25:23.550056+0000 front 2024-05-15T13:25:23.550965+0000 (oldest deadline 2024-05-15T13:25:49.449195+0000) 2024-05-15T13:25:50.043 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:49 smithi120 ceph-mon[43961]: pgmap v2264: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 43671/300144 objects misplaced (14.550%); 12 MiB/s, 192 objects/s recovering 2024-05-15T13:25:50.043 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:49 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:25:50.043 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:49 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1776610945' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:25:50.043 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:49 smithi120 ceph-mon[43961]: osdmap e881: 12 total, 12 up, 10 in 2024-05-15T13:25:50.084 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:25:50.134 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:50.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:49 smithi177 ceph-mon[44303]: pgmap v2264: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 43671/300144 objects misplaced (14.550%); 12 MiB/s, 192 objects/s recovering 2024-05-15T13:25:50.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:49 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:25:50.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:49 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1776610945' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:25:50.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:49 smithi177 ceph-mon[44303]: osdmap e881: 12 total, 12 up, 10 in 2024-05-15T13:25:50.267 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:25:50.268 INFO:teuthology.orchestra.run.smithi027.stderr:no valid command found; 10 closest matches: 2024-05-15T13:25:50.269 INFO:teuthology.orchestra.run.smithi027.stderr:0 2024-05-15T13:25:50.269 INFO:teuthology.orchestra.run.smithi027.stderr:1 2024-05-15T13:25:50.269 INFO:teuthology.orchestra.run.smithi027.stderr:2 2024-05-15T13:25:50.269 INFO:teuthology.orchestra.run.smithi027.stderr:abort 2024-05-15T13:25:50.269 INFO:teuthology.orchestra.run.smithi027.stderr:assert 2024-05-15T13:25:50.269 INFO:teuthology.orchestra.run.smithi027.stderr:bench [] [] [] [] 2024-05-15T13:25:50.269 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs debug_inject_read_zeros 2024-05-15T13:25:50.269 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs files list 2024-05-15T13:25:50.269 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs stats 2024-05-15T13:25:50.269 INFO:teuthology.orchestra.run.smithi027.stderr:bluestore allocator dump block 2024-05-15T13:25:50.269 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: invalid command 2024-05-15T13:25:50.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:50.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:25:50.375 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:25:50.427 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:25:50 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:25:50.247+0000 7fa328a3e700 -1 osd.6 882 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:25:28.365741+0000 front 2024-05-15T13:25:28.365664+0000 (oldest deadline 2024-05-15T13:25:50.065824+0000) 2024-05-15T13:25:50.429 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:50.455 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:25:50.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:50.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:25:50.511 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 15 13:25:50 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-3[81304]: 2024-05-15T13:25:50.154+0000 7fd2ac61e700 -1 osd.3 882 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:25:25.610284+0000 front 2024-05-15T13:25:25.610343+0000 (oldest deadline 2024-05-15T13:25:49.110390+0000) 2024-05-15T13:25:50.563 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:25:50.610 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:50.645 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:25:50.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:50.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:25:50.761 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:25:50.781 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 15 13:25:50 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-0[58372]: 2024-05-15T13:25:50.510+0000 7f897b1e5700 -1 osd.0 882 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:25:23.336753+0000 front 2024-05-15T13:25:23.336781+0000 (oldest deadline 2024-05-15T13:25:49.236801+0000) 2024-05-15T13:25:50.793 INFO:journalctl@ceph.osd.5.smithi120.stdout:May 15 13:25:50 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-5[56546]: 2024-05-15T13:25:50.529+0000 7f0f46d17700 -1 osd.5 882 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:25:23.550056+0000 front 2024-05-15T13:25:23.550965+0000 (oldest deadline 2024-05-15T13:25:49.449195+0000) 2024-05-15T13:25:50.800 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:50.837 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:25:50.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:50.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:25:50.946 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:25:50.973 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:50.993 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:25:50.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:50.996 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:25:51.097 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:25:51.126 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:50 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:25:51.126 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:50 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:25:51.126 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:50 smithi027 ceph-mon[39795]: osdmap e882: 12 total, 12 up, 10 in 2024-05-15T13:25:51.126 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:50 smithi027 ceph-mon[39795]: osd.4 reported failed by osd.6 2024-05-15T13:25:51.129 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:51.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:50 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:25:51.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:50 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:25:51.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:50 smithi177 ceph-mon[44303]: osdmap e882: 12 total, 12 up, 10 in 2024-05-15T13:25:51.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:50 smithi177 ceph-mon[44303]: osd.4 reported failed by osd.6 2024-05-15T13:25:51.249 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:50 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:25:51.249 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:50 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:25:51.249 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:50 smithi120 ceph-mon[43961]: osdmap e882: 12 total, 12 up, 10 in 2024-05-15T13:25:51.249 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:50 smithi120 ceph-mon[43961]: osd.4 reported failed by osd.6 2024-05-15T13:25:51.252 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:25:51 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:25:51.248+0000 7fa328a3e700 -1 osd.6 882 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:25:28.365741+0000 front 2024-05-15T13:25:28.365664+0000 (oldest deadline 2024-05-15T13:25:50.065824+0000) 2024-05-15T13:25:51.380 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:25:51.383 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 15 13:25:51 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-3[81304]: 2024-05-15T13:25:51.182+0000 7fd2ac61e700 -1 osd.3 882 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:25:25.610284+0000 front 2024-05-15T13:25:25.610343+0000 (oldest deadline 2024-05-15T13:25:49.110390+0000) 2024-05-15T13:25:51.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:51.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:25:51.485 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:25:51.512 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:51.531 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:25:51.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:51.534 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:25:51.635 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:25:51.681 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:51.719 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:25:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T13:25:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-05-15T13:25:51.719 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:25:51.730 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:25:51.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:51.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:25:51.739 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 15 13:25:51 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-0[58372]: 2024-05-15T13:25:51.534+0000 7f897b1e5700 -1 osd.0 882 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:25:23.336753+0000 front 2024-05-15T13:25:23.336781+0000 (oldest deadline 2024-05-15T13:25:49.236801+0000) 2024-05-15T13:25:51.770 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:25:51.792 INFO:journalctl@ceph.osd.5.smithi120.stdout:May 15 13:25:51 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-5[56546]: 2024-05-15T13:25:51.500+0000 7f0f46d17700 -1 osd.5 882 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:25:23.550056+0000 front 2024-05-15T13:25:23.550965+0000 (oldest deadline 2024-05-15T13:25:49.449195+0000) 2024-05-15T13:25:51.838 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:25:51.864 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:51.883 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:25:51.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:51.886 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:25:51.987 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:25:52.012 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:25:51 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:25:51.759+0000 7f7987687700 -1 osd.9 882 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:25:25.937019+0000 front 2024-05-15T13:25:25.936920+0000 (oldest deadline 2024-05-15T13:25:51.236881+0000) 2024-05-15T13:25:52.012 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:51 smithi177 ceph-mon[44303]: 1.6 scrub starts 2024-05-15T13:25:52.013 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:51 smithi177 ceph-mon[44303]: 1.6 scrub ok 2024-05-15T13:25:52.013 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:51 smithi177 ceph-mon[44303]: pgmap v2267: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 43012/300144 objects misplaced (14.330%); 11 MiB/s, 183 objects/s recovering 2024-05-15T13:25:52.013 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:51 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 8831/300144 objects degraded (2.942%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-05-15T13:25:52.013 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:51 smithi177 ceph-mon[44303]: osd.4 reported failed by osd.5 2024-05-15T13:25:52.013 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:51 smithi177 ceph-mon[44303]: osd.4 failed (root=default,host=smithi120) (2 reporters from different osd after 22.206012 >= grace 20.000000) 2024-05-15T13:25:52.013 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:51 smithi177 ceph-mon[44303]: osd.4 reported failed by osd.0 2024-05-15T13:25:52.014 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:52.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:51 smithi027 ceph-mon[39795]: 1.6 scrub starts 2024-05-15T13:25:52.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:51 smithi027 ceph-mon[39795]: 1.6 scrub ok 2024-05-15T13:25:52.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:51 smithi027 ceph-mon[39795]: pgmap v2267: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 43012/300144 objects misplaced (14.330%); 11 MiB/s, 183 objects/s recovering 2024-05-15T13:25:52.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:51 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 8831/300144 objects degraded (2.942%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-05-15T13:25:52.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:51 smithi027 ceph-mon[39795]: osd.4 reported failed by osd.5 2024-05-15T13:25:52.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:51 smithi027 ceph-mon[39795]: osd.4 failed (root=default,host=smithi120) (2 reporters from different osd after 22.206012 >= grace 20.000000) 2024-05-15T13:25:52.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:51 smithi027 ceph-mon[39795]: osd.4 reported failed by osd.0 2024-05-15T13:25:52.032 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:25:52.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:52.036 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:25:52.137 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:25:52.163 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:52.166 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T13:25:52.167 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T13:25:52.167 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.9 'trim stale osdmaps' 2024-05-15T13:25:52.182 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:25:52.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:52.185 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:25:52.286 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:25:52.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:51 smithi120 ceph-mon[43961]: 1.6 scrub starts 2024-05-15T13:25:52.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:51 smithi120 ceph-mon[43961]: 1.6 scrub ok 2024-05-15T13:25:52.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:51 smithi120 ceph-mon[43961]: pgmap v2267: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 43012/300144 objects misplaced (14.330%); 11 MiB/s, 183 objects/s recovering 2024-05-15T13:25:52.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:51 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 8831/300144 objects degraded (2.942%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-05-15T13:25:52.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:51 smithi120 ceph-mon[43961]: osd.4 reported failed by osd.5 2024-05-15T13:25:52.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:51 smithi120 ceph-mon[43961]: osd.4 failed (root=default,host=smithi120) (2 reporters from different osd after 22.206012 >= grace 20.000000) 2024-05-15T13:25:52.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:51 smithi120 ceph-mon[43961]: osd.4 reported failed by osd.0 2024-05-15T13:25:52.313 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:52.437 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:25:52.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:52.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:25:52.544 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:25:52.573 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:52.840 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:25:52.883 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-05-15T13:25:53.121 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:25:53.127 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:52 smithi027 ceph-mon[39795]: 1.14 scrub starts 2024-05-15T13:25:53.127 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:52 smithi027 ceph-mon[39795]: 1.14 scrub ok 2024-05-15T13:25:53.127 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:52 smithi027 ceph-mon[39795]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-15T13:25:53.127 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:52 smithi027 ceph-mon[39795]: osdmap e883: 12 total, 11 up, 10 in 2024-05-15T13:25:53.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:53.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:25:53.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:52 smithi177 ceph-mon[44303]: 1.14 scrub starts 2024-05-15T13:25:53.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:52 smithi177 ceph-mon[44303]: 1.14 scrub ok 2024-05-15T13:25:53.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:52 smithi177 ceph-mon[44303]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-15T13:25:53.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:52 smithi177 ceph-mon[44303]: osdmap e883: 12 total, 11 up, 10 in 2024-05-15T13:25:53.229 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:25:53.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:52 smithi120 ceph-mon[43961]: 1.14 scrub starts 2024-05-15T13:25:53.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:52 smithi120 ceph-mon[43961]: 1.14 scrub ok 2024-05-15T13:25:53.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:52 smithi120 ceph-mon[43961]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-15T13:25:53.281 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:52 smithi120 ceph-mon[43961]: osdmap e883: 12 total, 11 up, 10 in 2024-05-15T13:25:53.285 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:53.323 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:25:53.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:53.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:25:53.428 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:25:53.458 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:53.685 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:25:53.707 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:25:53.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:53.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:25:53.721 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:25:53.813 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:25:53.865 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:53.913 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:25:53.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:53.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:25:53.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:53 smithi027 ceph-mon[39795]: pgmap v2269: 47 pgs: 2 stale+active+clean, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 36 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 42549/300144 objects misplaced (14.176%); 14 MiB/s, 226 objects/s recovering 2024-05-15T13:25:53.983 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:53 smithi027 ceph-mon[39795]: osdmap e884: 12 total, 11 up, 10 in 2024-05-15T13:25:54.024 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:25:54.055 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:54.077 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:25:54.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:54.080 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:25:54.181 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:25:54.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:53 smithi177 ceph-mon[44303]: pgmap v2269: 47 pgs: 2 stale+active+clean, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 36 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 42549/300144 objects misplaced (14.176%); 14 MiB/s, 226 objects/s recovering 2024-05-15T13:25:54.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:53 smithi177 ceph-mon[44303]: osdmap e884: 12 total, 11 up, 10 in 2024-05-15T13:25:54.213 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:54.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:53 smithi120 ceph-mon[43961]: pgmap v2269: 47 pgs: 2 stale+active+clean, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 36 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 8831/300144 objects degraded (2.942%); 42549/300144 objects misplaced (14.176%); 14 MiB/s, 226 objects/s recovering 2024-05-15T13:25:54.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:53 smithi120 ceph-mon[43961]: osdmap e884: 12 total, 11 up, 10 in 2024-05-15T13:25:54.537 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:25:54.559 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:25:54.587 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:25:54.611 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:25:54.631 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:25:54.653 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:25:54.675 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:25:54.675 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:25:54.675 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:25:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:25:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:25:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-05-15T13:25:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:25:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:25:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:25:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:25:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:25:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:25:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:25:54.872 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:54 smithi027 ceph-mon[39795]: osdmap e885: 12 total, 11 up, 10 in 2024-05-15T13:25:55.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:54 smithi177 ceph-mon[44303]: osdmap e885: 12 total, 11 up, 10 in 2024-05-15T13:25:55.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:54 smithi120 ceph-mon[43961]: osdmap e885: 12 total, 11 up, 10 in 2024-05-15T13:25:55.868 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:25:55.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:55.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:25:55.973 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:25:56.010 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:56.051 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:25:56.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:56.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:25:56.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:55 smithi027 ceph-mon[39795]: pgmap v2272: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 2 remapped+peering, 2 stale+active+clean, 3 active+undersized+wait, 5 active+recovering+undersized+degraded+remapped, 33 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 32 op/s; 18/299658 objects degraded (0.006%); 35526/299658 objects misplaced (11.856%); 16 MiB/s, 257 objects/s recovering 2024-05-15T13:25:56.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:25:56.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:55 smithi177 ceph-mon[44303]: pgmap v2272: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 2 remapped+peering, 2 stale+active+clean, 3 active+undersized+wait, 5 active+recovering+undersized+degraded+remapped, 33 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 32 op/s; 18/299658 objects degraded (0.006%); 35526/299658 objects misplaced (11.856%); 16 MiB/s, 257 objects/s recovering 2024-05-15T13:25:56.190 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:56.264 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:25:56.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:56.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:25:56.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:55 smithi120 ceph-mon[43961]: pgmap v2272: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 2 remapped+peering, 2 stale+active+clean, 3 active+undersized+wait, 5 active+recovering+undersized+degraded+remapped, 33 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 32 op/s; 18/299658 objects degraded (0.006%); 35526/299658 objects misplaced (11.856%); 16 MiB/s, 257 objects/s recovering 2024-05-15T13:25:56.372 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:25:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:57.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:56 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 17/299628 objects degraded (0.006%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:25:57.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:56 smithi177 ceph-mon[44303]: osdmap e886: 12 total, 11 up, 10 in 2024-05-15T13:25:57.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:56 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 17/299628 objects degraded (0.006%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:25:57.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:56 smithi027 ceph-mon[39795]: osdmap e886: 12 total, 11 up, 10 in 2024-05-15T13:25:57.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:56 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 17/299628 objects degraded (0.006%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:25:57.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:56 smithi120 ceph-mon[43961]: osdmap e886: 12 total, 11 up, 10 in 2024-05-15T13:25:57.793 INFO:journalctl@ceph.osd.4.smithi120.stdout:May 15 13:25:57 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-4[49322]: 2024-05-15T13:25:57.315+0000 7f9fa019d700 -1 osd.4 886 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T13:25:58.181 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:57 smithi027 ceph-mon[39795]: pgmap v2273: 47 pgs: 1 active+clean+remapped+wait, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 2 remapped+peering, 8 active+undersized+wait, 4 active+recovering+undersized+degraded+remapped, 30 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 28 op/s; 17/299628 objects degraded (0.006%); 36731/299628 objects misplaced (12.259%); 16 MiB/s, 259 objects/s recovering 2024-05-15T13:25:58.181 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:57 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:25:58.182 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:57 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:25:58.182 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:57 smithi027 ceph-mon[39795]: Monitor daemon marked osd.4 down, but it is still running 2024-05-15T13:25:58.182 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:57 smithi027 ceph-mon[39795]: map e883 wrongly marked me down at e883 2024-05-15T13:25:58.182 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:57 smithi027 ceph-mon[39795]: osd.4 marked itself dead as of e883 2024-05-15T13:25:58.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:57 smithi177 ceph-mon[44303]: pgmap v2273: 47 pgs: 1 active+clean+remapped+wait, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 2 remapped+peering, 8 active+undersized+wait, 4 active+recovering+undersized+degraded+remapped, 30 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 28 op/s; 17/299628 objects degraded (0.006%); 36731/299628 objects misplaced (12.259%); 16 MiB/s, 259 objects/s recovering 2024-05-15T13:25:58.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:57 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:25:58.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:57 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:25:58.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:57 smithi177 ceph-mon[44303]: Monitor daemon marked osd.4 down, but it is still running 2024-05-15T13:25:58.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:57 smithi177 ceph-mon[44303]: map e883 wrongly marked me down at e883 2024-05-15T13:25:58.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:57 smithi177 ceph-mon[44303]: osd.4 marked itself dead as of e883 2024-05-15T13:25:58.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:57 smithi120 ceph-mon[43961]: pgmap v2273: 47 pgs: 1 active+clean+remapped+wait, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 2 remapped+peering, 8 active+undersized+wait, 4 active+recovering+undersized+degraded+remapped, 30 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 28 op/s; 17/299628 objects degraded (0.006%); 36731/299628 objects misplaced (12.259%); 16 MiB/s, 259 objects/s recovering 2024-05-15T13:25:58.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:57 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:25:58.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:57 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:25:58.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:57 smithi120 ceph-mon[43961]: Monitor daemon marked osd.4 down, but it is still running 2024-05-15T13:25:58.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:57 smithi120 ceph-mon[43961]: map e883 wrongly marked me down at e883 2024-05-15T13:25:58.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:57 smithi120 ceph-mon[43961]: osd.4 marked itself dead as of e883 2024-05-15T13:25:58.338 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T13:25:58.339 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T13:25:59.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:58 smithi177 ceph-mon[44303]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-15T13:25:59.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:58 smithi177 ceph-mon[44303]: osd.4 [v2:172.21.15.120:6800/3877842957,v1:172.21.15.120:6801/3877842957] boot 2024-05-15T13:25:59.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:58 smithi177 ceph-mon[44303]: osdmap e887: 12 total, 12 up, 10 in 2024-05-15T13:25:59.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:58 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T13:25:59.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:58 smithi027 ceph-mon[39795]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-15T13:25:59.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:58 smithi027 ceph-mon[39795]: osd.4 [v2:172.21.15.120:6800/3877842957,v1:172.21.15.120:6801/3877842957] boot 2024-05-15T13:25:59.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:58 smithi027 ceph-mon[39795]: osdmap e887: 12 total, 12 up, 10 in 2024-05-15T13:25:59.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:58 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T13:25:59.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:58 smithi120 ceph-mon[43961]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-15T13:25:59.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:58 smithi120 ceph-mon[43961]: osd.4 [v2:172.21.15.120:6800/3877842957,v1:172.21.15.120:6801/3877842957] boot 2024-05-15T13:25:59.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:58 smithi120 ceph-mon[43961]: osdmap e887: 12 total, 12 up, 10 in 2024-05-15T13:25:59.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:58 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T13:25:59.506 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:25:59.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:59.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:25:59.612 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:25:59.699 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:59.744 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:25:59.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:25:59.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:25:59.858 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:25:59.896 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:25:59.942 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:59 smithi120 ceph-mon[43961]: pgmap v2276: 47 pgs: 1 peering, 1 active+clean+remapped+wait, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 2 remapped+peering, 7 active+undersized+wait, 4 active+recovering+undersized+degraded+remapped, 30 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 34 op/s; 17/299541 objects degraded (0.006%); 36731/299541 objects misplaced (12.262%); 12 MiB/s, 364 objects/s recovering 2024-05-15T13:25:59.942 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:59 smithi120 ceph-mon[43961]: osdmap e888: 12 total, 12 up, 10 in 2024-05-15T13:25:59.942 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:59 smithi120 ceph-mon[43961]: 1.e scrub starts 2024-05-15T13:25:59.942 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:59 smithi120 ceph-mon[43961]: 1.e scrub ok 2024-05-15T13:25:59.942 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:59 smithi120 ceph-mon[43961]: osdmap e889: 12 total, 12 up, 10 in 2024-05-15T13:25:59.943 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:25:59 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/250779' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:26:00.048 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:26:00.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:00.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:26:00.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:26:00.186 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:59 smithi027 ceph-mon[39795]: pgmap v2276: 47 pgs: 1 peering, 1 active+clean+remapped+wait, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 2 remapped+peering, 7 active+undersized+wait, 4 active+recovering+undersized+degraded+remapped, 30 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 34 op/s; 17/299541 objects degraded (0.006%); 36731/299541 objects misplaced (12.262%); 12 MiB/s, 364 objects/s recovering 2024-05-15T13:26:00.187 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:59 smithi027 ceph-mon[39795]: osdmap e888: 12 total, 12 up, 10 in 2024-05-15T13:26:00.187 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:59 smithi027 ceph-mon[39795]: 1.e scrub starts 2024-05-15T13:26:00.187 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:59 smithi027 ceph-mon[39795]: 1.e scrub ok 2024-05-15T13:26:00.187 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:59 smithi027 ceph-mon[39795]: osdmap e889: 12 total, 12 up, 10 in 2024-05-15T13:26:00.187 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:25:59 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/250779' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:26:00.189 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:00.229 INFO:teuthology.orchestra.run.smithi027.stderr:no valid command found; 10 closest matches: 2024-05-15T13:26:00.229 INFO:teuthology.orchestra.run.smithi027.stderr:0 2024-05-15T13:26:00.229 INFO:teuthology.orchestra.run.smithi027.stderr:1 2024-05-15T13:26:00.229 INFO:teuthology.orchestra.run.smithi027.stderr:2 2024-05-15T13:26:00.229 INFO:teuthology.orchestra.run.smithi027.stderr:abort 2024-05-15T13:26:00.230 INFO:teuthology.orchestra.run.smithi027.stderr:assert 2024-05-15T13:26:00.230 INFO:teuthology.orchestra.run.smithi027.stderr:bench [] [] [] [] 2024-05-15T13:26:00.230 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs debug_inject_read_zeros 2024-05-15T13:26:00.230 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs files list 2024-05-15T13:26:00.230 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs stats 2024-05-15T13:26:00.230 INFO:teuthology.orchestra.run.smithi027.stderr:bluestore allocator dump block 2024-05-15T13:26:00.230 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: invalid command 2024-05-15T13:26:00.340 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:26:00.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:59 smithi177 ceph-mon[44303]: pgmap v2276: 47 pgs: 1 peering, 1 active+clean+remapped+wait, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 2 remapped+peering, 7 active+undersized+wait, 4 active+recovering+undersized+degraded+remapped, 30 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 34 op/s; 17/299541 objects degraded (0.006%); 36731/299541 objects misplaced (12.262%); 12 MiB/s, 364 objects/s recovering 2024-05-15T13:26:00.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:59 smithi177 ceph-mon[44303]: osdmap e888: 12 total, 12 up, 10 in 2024-05-15T13:26:00.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:59 smithi177 ceph-mon[44303]: 1.e scrub starts 2024-05-15T13:26:00.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:59 smithi177 ceph-mon[44303]: 1.e scrub ok 2024-05-15T13:26:00.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:59 smithi177 ceph-mon[44303]: osdmap e889: 12 total, 12 up, 10 in 2024-05-15T13:26:00.684 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:25:59 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/250779' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:26:00.685 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:26:01.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:00 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:26:01.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:00 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/250779' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:26:01.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:00 smithi027 ceph-mon[39795]: osdmap e890: 12 total, 12 up, 10 in 2024-05-15T13:26:01.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:00 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:26:01.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:00 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/250779' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:26:01.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:00 smithi120 ceph-mon[43961]: osdmap e890: 12 total, 12 up, 10 in 2024-05-15T13:26:01.443 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:00 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:26:01.443 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:00 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/250779' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:26:01.443 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:00 smithi177 ceph-mon[44303]: osdmap e890: 12 total, 12 up, 10 in 2024-05-15T13:26:02.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:01 smithi027 ceph-mon[39795]: pgmap v2280: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 1 peering, 1 active+clean+remapped+wait, 2 remapped+peering, 5 active+undersized+wait, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 9 op/s; 17/299520 objects degraded (0.006%); 35789/299520 objects misplaced (11.949%); 15 MiB/s, 516 objects/s recovering 2024-05-15T13:26:02.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:01 smithi120 ceph-mon[43961]: pgmap v2280: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 1 peering, 1 active+clean+remapped+wait, 2 remapped+peering, 5 active+undersized+wait, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 9 op/s; 17/299520 objects degraded (0.006%); 35789/299520 objects misplaced (11.949%); 15 MiB/s, 516 objects/s recovering 2024-05-15T13:26:02.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:01 smithi177 ceph-mon[44303]: pgmap v2280: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 1 peering, 1 active+clean+remapped+wait, 2 remapped+peering, 5 active+undersized+wait, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 6.1 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 9 op/s; 17/299520 objects degraded (0.006%); 35789/299520 objects misplaced (11.949%); 15 MiB/s, 516 objects/s recovering 2024-05-15T13:26:02.649 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:26:02.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:02.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:26:02.660 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T13:26:02.660 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T13:26:02.660 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.10 'trim stale osdmaps' 2024-05-15T13:26:02.756 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:26:02.787 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:02.826 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:26:02.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:02.828 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:26:02.929 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:26:02.974 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:03.226 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:26:03.228 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:03 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 17/298596 objects degraded (0.006%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:26:03.230 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:26:03.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:03.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:26:03.294 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:26:03.294 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T13:26:03.294 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-05-15T13:26:03.294 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-05-15T13:26:03.294 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:26:03.331 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:26:03.409 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:03 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 17/298596 objects degraded (0.006%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:26:03.419 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:03.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:03 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 17/298596 objects degraded (0.006%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T13:26:03.490 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:26:03.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:03.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:26:03.598 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:26:03.713 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:03.746 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:26:03.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:03.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:26:03.855 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:26:03.896 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:03.934 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:26:03.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:03.937 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:26:04.038 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:26:04.097 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:26:04.130 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:04.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:04 smithi177 ceph-mon[44303]: pgmap v2281: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 peering, 2 remapped+peering, 4 active+recovering+undersized+degraded+remapped, 37 active+clean; 6.1 GiB data, 24 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 69 op/s; 17/298596 objects degraded (0.006%); 34043/298596 objects misplaced (11.401%); 16 MiB/s, 468 objects/s recovering 2024-05-15T13:26:04.197 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:26:04.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:04.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:26:04.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:04 smithi027 ceph-mon[39795]: pgmap v2281: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 peering, 2 remapped+peering, 4 active+recovering+undersized+degraded+remapped, 37 active+clean; 6.1 GiB data, 24 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 69 op/s; 17/298596 objects degraded (0.006%); 34043/298596 objects misplaced (11.401%); 16 MiB/s, 468 objects/s recovering 2024-05-15T13:26:04.304 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:26:04.337 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:04.360 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:26:04.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:04.363 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:26:04.464 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:26:04.544 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:04 smithi120 ceph-mon[43961]: pgmap v2281: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 peering, 2 remapped+peering, 4 active+recovering+undersized+degraded+remapped, 37 active+clean; 6.1 GiB data, 24 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 69 op/s; 17/298596 objects degraded (0.006%); 34043/298596 objects misplaced (11.401%); 16 MiB/s, 468 objects/s recovering 2024-05-15T13:26:04.556 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:04.627 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:26:04.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:04.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:26:04.732 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:26:04.764 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:05.531 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:05 smithi027 ceph-mon[39795]: osdmap e891: 12 total, 12 up, 10 in 2024-05-15T13:26:05.531 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:05 smithi027 ceph-mon[39795]: pgmap v2283: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.0 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 270 op/s; 7060/294843 objects degraded (2.394%); 39509/294843 objects misplaced (13.400%); 19 MiB/s, 313 objects/s recovering 2024-05-15T13:26:05.645 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:26:05.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:05.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:26:05.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:05 smithi177 ceph-mon[44303]: osdmap e891: 12 total, 12 up, 10 in 2024-05-15T13:26:05.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:05 smithi177 ceph-mon[44303]: pgmap v2283: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.0 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 270 op/s; 7060/294843 objects degraded (2.394%); 39509/294843 objects misplaced (13.400%); 19 MiB/s, 313 objects/s recovering 2024-05-15T13:26:05.750 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:26:05.780 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:05.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:05 smithi120 ceph-mon[43961]: osdmap e891: 12 total, 12 up, 10 in 2024-05-15T13:26:05.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:05 smithi120 ceph-mon[43961]: pgmap v2283: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.0 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 270 op/s; 7060/294843 objects degraded (2.394%); 39509/294843 objects misplaced (13.400%); 19 MiB/s, 313 objects/s recovering 2024-05-15T13:26:05.981 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:26:05.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:05.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:26:06.011 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-05-15T13:26:06.018 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:26:06.089 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:26:06.241 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:06.275 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:26:06.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:06.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:26:06.381 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:26:06.410 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:06.437 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:26:06.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:06.443 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:26:06.544 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:26:06.545 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:06 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:26:06.545 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:06 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:26:06.583 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:06.605 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:26:06.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:06.613 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:26:06.617 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:06 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:26:06.617 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:06 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:26:06.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:06 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:26:06.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:06 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:26:06.713 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:26:06.761 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:06.792 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:26:06.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:06.795 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:26:06.896 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:26:06.927 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:06.955 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:26:06.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:06.961 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:26:07.062 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:26:07.165 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:07.259 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:26:07.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:07.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:26:07.364 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:26:07.398 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:07.422 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:26:07.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:07.433 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:26:07.493 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:26:07.533 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:26:07.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:07 smithi120 ceph-mon[43961]: osdmap e892: 12 total, 12 up, 10 in 2024-05-15T13:26:07.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:07 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:26:07.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:07 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:26:07.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:07 smithi120 ceph-mon[43961]: pgmap v2285: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.0 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 341 op/s; 7060/292959 objects degraded (2.410%); 39280/292959 objects misplaced (13.408%); 11 MiB/s, 172 objects/s recovering 2024-05-15T13:26:07.656 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:07.669 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:26:07.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:07 smithi177 ceph-mon[44303]: osdmap e892: 12 total, 12 up, 10 in 2024-05-15T13:26:07.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:07 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:26:07.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:07 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:26:07.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:07 smithi177 ceph-mon[44303]: pgmap v2285: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.0 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 341 op/s; 7060/292959 objects degraded (2.410%); 39280/292959 objects misplaced (13.408%); 11 MiB/s, 172 objects/s recovering 2024-05-15T13:26:07.705 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:26:07.720 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:26:07.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:07.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:26:07.739 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:26:07.761 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:26:07.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:07 smithi027 ceph-mon[39795]: osdmap e892: 12 total, 12 up, 10 in 2024-05-15T13:26:07.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:07 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:26:07.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:07 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:26:07.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:07 smithi027 ceph-mon[39795]: pgmap v2285: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.0 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 341 op/s; 7060/292959 objects degraded (2.410%); 39280/292959 objects misplaced (13.408%); 11 MiB/s, 172 objects/s recovering 2024-05-15T13:26:07.787 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:26:07.824 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:26:07.848 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:26:07.885 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:26:07.932 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:26:07.944 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:07.953 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:26:07.976 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:26:07.999 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:26:08.001 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:26:08.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:08.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:26:08.028 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:26:08.029 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:26:08.030 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:26:08.030 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:26:08.030 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:26:08.030 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:26:08.030 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:26:08.030 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:26:08.030 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:26:08.030 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:26:08.030 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:26:08.030 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:26:08.113 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:26:08.157 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:08.185 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:26:08.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:08.188 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:26:08.289 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:26:08.326 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:08.426 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:26:08.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:08.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:26:08.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:08 smithi177 ceph-mon[44303]: osdmap e893: 12 total, 12 up, 10 in 2024-05-15T13:26:08.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:08 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2067844679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:26:08.518 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:08 smithi027 ceph-mon[39795]: osdmap e893: 12 total, 12 up, 10 in 2024-05-15T13:26:08.518 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:08 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2067844679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:26:08.533 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:26:08.640 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:08.686 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:26:08.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:08.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:26:08.692 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:08 smithi120 ceph-mon[43961]: osdmap e893: 12 total, 12 up, 10 in 2024-05-15T13:26:08.692 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:08 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2067844679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:26:08.793 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:26:08.825 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:08.860 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:26:08.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:08.869 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:26:08.970 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:26:09.085 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:09.192 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:26:09.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:09.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:26:09.307 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:26:09.347 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:09.380 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:26:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:26:09.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:09.386 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:26:09.418 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:09 smithi120 ceph-mon[43961]: pgmap v2287: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.9 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 484 op/s; 7080/289881 objects degraded (2.442%); 39148/289881 objects misplaced (13.505%); 9.9 MiB/s, 159 objects/s recovering 2024-05-15T13:26:09.418 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:09 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 7060/292959 objects degraded (2.410%), 6 pgs degraded (PG_DEGRADED) 2024-05-15T13:26:09.463 INFO:teuthology.orchestra.run.smithi027.stderr:no valid command found; 10 closest matches: 2024-05-15T13:26:09.463 INFO:teuthology.orchestra.run.smithi027.stderr:0 2024-05-15T13:26:09.463 INFO:teuthology.orchestra.run.smithi027.stderr:1 2024-05-15T13:26:09.463 INFO:teuthology.orchestra.run.smithi027.stderr:2 2024-05-15T13:26:09.463 INFO:teuthology.orchestra.run.smithi027.stderr:abort 2024-05-15T13:26:09.463 INFO:teuthology.orchestra.run.smithi027.stderr:assert 2024-05-15T13:26:09.464 INFO:teuthology.orchestra.run.smithi027.stderr:bench [] [] [] [] 2024-05-15T13:26:09.464 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs debug_inject_read_zeros 2024-05-15T13:26:09.464 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs files list 2024-05-15T13:26:09.464 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs stats 2024-05-15T13:26:09.464 INFO:teuthology.orchestra.run.smithi027.stderr:bluestore allocator dump block 2024-05-15T13:26:09.464 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: invalid command 2024-05-15T13:26:09.486 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:26:09.533 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:09.591 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:26:09.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:09.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:26:09.663 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:09 smithi027 ceph-mon[39795]: pgmap v2287: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.9 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 484 op/s; 7080/289881 objects degraded (2.442%); 39148/289881 objects misplaced (13.505%); 9.9 MiB/s, 159 objects/s recovering 2024-05-15T13:26:09.663 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:09 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 7060/292959 objects degraded (2.410%), 6 pgs degraded (PG_DEGRADED) 2024-05-15T13:26:09.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:09 smithi177 ceph-mon[44303]: pgmap v2287: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.9 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 484 op/s; 7080/289881 objects degraded (2.442%); 39148/289881 objects misplaced (13.505%); 9.9 MiB/s, 159 objects/s recovering 2024-05-15T13:26:09.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:09 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 7060/292959 objects degraded (2.410%), 6 pgs degraded (PG_DEGRADED) 2024-05-15T13:26:09.703 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:26:09.738 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:10.914 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:26:11.308 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:26:11.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:11.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:26:11.414 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:26:11.452 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:11.476 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:26:11.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:11.479 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:26:11.580 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:26:11.660 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:11.752 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:26:11.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:11.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:26:11.856 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:26:11.884 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:11 smithi027 ceph-mon[39795]: pgmap v2288: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.9 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 479 op/s; 7080/287706 objects degraded (2.461%); 38533/287706 objects misplaced (13.393%); 13 MiB/s, 209 objects/s recovering 2024-05-15T13:26:11.887 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:12.043 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:11 smithi120 ceph-mon[43961]: pgmap v2288: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.9 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 479 op/s; 7080/287706 objects degraded (2.461%); 38533/287706 objects misplaced (13.393%); 13 MiB/s, 209 objects/s recovering 2024-05-15T13:26:12.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:11 smithi177 ceph-mon[44303]: pgmap v2288: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.9 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 479 op/s; 7080/287706 objects degraded (2.461%); 38533/287706 objects misplaced (13.393%); 13 MiB/s, 209 objects/s recovering 2024-05-15T13:26:13.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:12 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:26:13.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:12 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:26:13.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:12 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:26:13.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:12 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:26:13.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:12 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:26:13.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:12 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:26:13.881 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:13 smithi027 ceph-mon[39795]: pgmap v2289: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 401 op/s; 7080/285195 objects degraded (2.483%); 38294/285195 objects misplaced (13.427%); 8.3 MiB/s, 132 objects/s recovering 2024-05-15T13:26:13.882 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:13 smithi027 ceph-mon[39795]: osdmap e894: 12 total, 12 up, 10 in 2024-05-15T13:26:13.883 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T13:26:13.883 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T13:26:13.884 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.11 'trim stale osdmaps' 2024-05-15T13:26:14.003 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:26:14.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:14.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:26:14.111 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:26:14.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:13 smithi177 ceph-mon[44303]: pgmap v2289: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 401 op/s; 7080/285195 objects degraded (2.483%); 38294/285195 objects misplaced (13.427%); 8.3 MiB/s, 132 objects/s recovering 2024-05-15T13:26:14.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:13 smithi177 ceph-mon[44303]: osdmap e894: 12 total, 12 up, 10 in 2024-05-15T13:26:14.228 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:13 smithi120 ceph-mon[43961]: pgmap v2289: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 401 op/s; 7080/285195 objects degraded (2.483%); 38294/285195 objects misplaced (13.427%); 8.3 MiB/s, 132 objects/s recovering 2024-05-15T13:26:14.228 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:13 smithi120 ceph-mon[43961]: osdmap e894: 12 total, 12 up, 10 in 2024-05-15T13:26:14.233 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:14.262 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:26:14.262 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:26:14.289 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:26:14.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:14.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:26:14.397 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:26:14.471 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:14.540 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:26:14.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:14.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:26:14.659 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:26:14.693 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:14.717 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:26:14.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:14.722 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:26:14.823 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:26:14.962 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:15.007 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:26:15.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:15.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:26:15.111 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:26:15.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:14 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 7080/285195 objects degraded (2.483%), 6 pgs degraded (PG_DEGRADED) 2024-05-15T13:26:15.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:14 smithi027 ceph-mon[39795]: osdmap e895: 12 total, 12 up, 10 in 2024-05-15T13:26:15.148 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:15.194 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:14 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 7080/285195 objects degraded (2.483%), 6 pgs degraded (PG_DEGRADED) 2024-05-15T13:26:15.194 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:14 smithi177 ceph-mon[44303]: osdmap e895: 12 total, 12 up, 10 in 2024-05-15T13:26:15.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:14 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 7080/285195 objects degraded (2.483%), 6 pgs degraded (PG_DEGRADED) 2024-05-15T13:26:15.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:14 smithi120 ceph-mon[43961]: osdmap e895: 12 total, 12 up, 10 in 2024-05-15T13:26:15.753 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:26:16.108 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:15 smithi027 ceph-mon[39795]: pgmap v2292: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 490 op/s; 7093/281139 objects degraded (2.523%); 37504/281139 objects misplaced (13.340%); 13 MiB/s, 202 objects/s recovering 2024-05-15T13:26:16.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:15 smithi177 ceph-mon[44303]: pgmap v2292: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 490 op/s; 7093/281139 objects degraded (2.523%); 37504/281139 objects misplaced (13.340%); 13 MiB/s, 202 objects/s recovering 2024-05-15T13:26:16.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:15 smithi120 ceph-mon[43961]: pgmap v2292: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 490 op/s; 7093/281139 objects degraded (2.523%); 37504/281139 objects misplaced (13.340%); 13 MiB/s, 202 objects/s recovering 2024-05-15T13:26:17.001 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-05-15T13:26:17.001 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:26:17.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:17.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:26:17.104 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:26:17.176 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:17.228 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:26:17.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:17.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:26:17.341 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:26:17.377 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:17.404 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:26:17.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:17.408 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:26:17.509 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:26:17.625 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:17.661 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:26:17.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:17.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:26:17.704 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:26:17.777 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:26:17.812 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:17.867 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:17 smithi120 ceph-mon[43961]: pgmap v2293: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 436 op/s; 7093/279408 objects degraded (2.539%); 37263/279408 objects misplaced (13.336%); 13 MiB/s, 201 objects/s recovering 2024-05-15T13:26:18.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:17 smithi027 ceph-mon[39795]: pgmap v2293: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 436 op/s; 7093/279408 objects degraded (2.539%); 37263/279408 objects misplaced (13.336%); 13 MiB/s, 201 objects/s recovering 2024-05-15T13:26:18.195 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:17 smithi177 ceph-mon[44303]: pgmap v2293: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 436 op/s; 7093/279408 objects degraded (2.539%); 37263/279408 objects misplaced (13.336%); 13 MiB/s, 201 objects/s recovering 2024-05-15T13:26:18.876 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:26:19.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:18 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/791452565' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:26:19.196 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:18 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/791452565' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:26:19.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:18 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/791452565' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:26:20.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:19 smithi177 ceph-mon[44303]: pgmap v2294: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.6 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 473 op/s; 7110/276333 objects degraded (2.573%); 37141/276333 objects misplaced (13.441%); 9.6 MiB/s, 155 objects/s recovering 2024-05-15T13:26:20.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:19 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:26:20.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:19 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/791452565' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:26:20.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:19 smithi177 ceph-mon[44303]: osdmap e896: 12 total, 12 up, 10 in 2024-05-15T13:26:20.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:19 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 7110/276333 objects degraded (2.573%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:26:20.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:19 smithi177 ceph-mon[44303]: osdmap e897: 12 total, 12 up, 10 in 2024-05-15T13:26:20.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:19 smithi027 ceph-mon[39795]: pgmap v2294: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.6 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 473 op/s; 7110/276333 objects degraded (2.573%); 37141/276333 objects misplaced (13.441%); 9.6 MiB/s, 155 objects/s recovering 2024-05-15T13:26:20.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:19 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:26:20.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:19 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/791452565' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:26:20.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:19 smithi027 ceph-mon[39795]: osdmap e896: 12 total, 12 up, 10 in 2024-05-15T13:26:20.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:19 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 7110/276333 objects degraded (2.573%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:26:20.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:19 smithi027 ceph-mon[39795]: osdmap e897: 12 total, 12 up, 10 in 2024-05-15T13:26:20.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:19 smithi120 ceph-mon[43961]: pgmap v2294: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.6 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 473 op/s; 7110/276333 objects degraded (2.573%); 37141/276333 objects misplaced (13.441%); 9.6 MiB/s, 155 objects/s recovering 2024-05-15T13:26:20.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:19 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:26:20.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:19 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/791452565' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:26:20.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:19 smithi120 ceph-mon[43961]: osdmap e896: 12 total, 12 up, 10 in 2024-05-15T13:26:20.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:19 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 7110/276333 objects degraded (2.573%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:26:20.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:19 smithi120 ceph-mon[43961]: osdmap e897: 12 total, 12 up, 10 in 2024-05-15T13:26:21.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:20 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:26:21.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:20 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:26:21.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:20 smithi027 ceph-mon[39795]: osdmap e898: 12 total, 12 up, 10 in 2024-05-15T13:26:21.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:20 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:26:21.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:20 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:26:21.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:20 smithi177 ceph-mon[44303]: osdmap e898: 12 total, 12 up, 10 in 2024-05-15T13:26:21.278 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:26:21.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:20 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:26:21.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:20 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:26:21.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:20 smithi120 ceph-mon[43961]: osdmap e898: 12 total, 12 up, 10 in 2024-05-15T13:26:21.566 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:26:21.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:21.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:26:21.673 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:26:21.711 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:21.746 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:26:21.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:21.750 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:26:21.851 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:26:21.988 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:22.039 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:26:22.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:22.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:26:22.160 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:26:22.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:21 smithi177 ceph-mon[44303]: pgmap v2298: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.6 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 386 op/s; 7110/274182 objects degraded (2.593%); 36471/274182 objects misplaced (13.302%); 9.1 MiB/s, 146 objects/s recovering 2024-05-15T13:26:22.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:21 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:26:22.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:21 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:26:22.198 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:22.228 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:26:22.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:22.233 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:26:22.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:21 smithi027 ceph-mon[39795]: pgmap v2298: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.6 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 386 op/s; 7110/274182 objects degraded (2.593%); 36471/274182 objects misplaced (13.302%); 9.1 MiB/s, 146 objects/s recovering 2024-05-15T13:26:22.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:21 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:26:22.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:21 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:26:22.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:21 smithi120 ceph-mon[43961]: pgmap v2298: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.6 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 386 op/s; 7110/274182 objects degraded (2.593%); 36471/274182 objects misplaced (13.302%); 9.1 MiB/s, 146 objects/s recovering 2024-05-15T13:26:22.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:21 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:26:22.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:21 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:26:22.334 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:26:22.362 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:22.393 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:26:22.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:22.400 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:26:22.501 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:26:22.540 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:22.812 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:26:23.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:22 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/903803826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:26:23.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:22 smithi177 ceph-mon[44303]: osdmap e899: 12 total, 12 up, 10 in 2024-05-15T13:26:23.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:22 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/903803826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:26:23.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:22 smithi027 ceph-mon[39795]: osdmap e899: 12 total, 12 up, 10 in 2024-05-15T13:26:23.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:22 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/903803826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:26:23.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:22 smithi120 ceph-mon[43961]: osdmap e899: 12 total, 12 up, 10 in 2024-05-15T13:26:23.435 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:26:23.435 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:26:23.435 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:26:23.435 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:26:23.435 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:26:23.435 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:26:23.435 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:26:23.435 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:26:23.435 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:26:23.435 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:26:23.435 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:26:23.436 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:26:23.436 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:26:23.436 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:26:23.436 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:26:23.436 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:26:23.436 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:26:23.436 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:26:23.436 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:26:23.436 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:26:23.436 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:26:23.437 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:26:23.437 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:26:23.437 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:26:24.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:23 smithi027 ceph-mon[39795]: pgmap v2300: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.5 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 393 op/s; 7110/271614 objects degraded (2.618%); 36199/271614 objects misplaced (13.327%); 13 MiB/s, 201 objects/s recovering 2024-05-15T13:26:24.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:23 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:26:24.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:23 smithi177 ceph-mon[44303]: pgmap v2300: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.5 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 393 op/s; 7110/271614 objects degraded (2.618%); 36199/271614 objects misplaced (13.327%); 13 MiB/s, 201 objects/s recovering 2024-05-15T13:26:24.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:23 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:26:24.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:23 smithi120 ceph-mon[43961]: pgmap v2300: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.5 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 393 op/s; 7110/271614 objects degraded (2.618%); 36199/271614 objects misplaced (13.327%); 13 MiB/s, 201 objects/s recovering 2024-05-15T13:26:24.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:23 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:26:24.296 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:26:24.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:24.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:26:24.402 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:26:24.439 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:24.475 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:26:24.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:24.482 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:26:24.583 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:26:24.871 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:25.194 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:24 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 7110/271614 objects degraded (2.618%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:26:25.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:24 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 7110/271614 objects degraded (2.618%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:26:25.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:24 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 7110/271614 objects degraded (2.618%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:26:25.695 INFO:teuthology.orchestra.run.smithi027.stderr:no valid command found; 10 closest matches: 2024-05-15T13:26:25.695 INFO:teuthology.orchestra.run.smithi027.stderr:0 2024-05-15T13:26:25.695 INFO:teuthology.orchestra.run.smithi027.stderr:1 2024-05-15T13:26:25.695 INFO:teuthology.orchestra.run.smithi027.stderr:2 2024-05-15T13:26:25.696 INFO:teuthology.orchestra.run.smithi027.stderr:abort 2024-05-15T13:26:25.696 INFO:teuthology.orchestra.run.smithi027.stderr:assert 2024-05-15T13:26:25.696 INFO:teuthology.orchestra.run.smithi027.stderr:bench [] [] [] [] 2024-05-15T13:26:25.696 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs debug_inject_read_zeros 2024-05-15T13:26:25.696 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs files list 2024-05-15T13:26:25.696 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs stats 2024-05-15T13:26:25.697 INFO:teuthology.orchestra.run.smithi027.stderr:bluestore allocator dump block 2024-05-15T13:26:25.697 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: invalid command 2024-05-15T13:26:26.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:25 smithi177 ceph-mon[44303]: pgmap v2301: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.5 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 416 op/s; 7142/268995 objects degraded (2.655%); 35883/268995 objects misplaced (13.340%); 12 MiB/s, 185 objects/s recovering 2024-05-15T13:26:26.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:25 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:26:26.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:25 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:26:26.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:25 smithi027 ceph-mon[39795]: pgmap v2301: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.5 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 416 op/s; 7142/268995 objects degraded (2.655%); 35883/268995 objects misplaced (13.340%); 12 MiB/s, 185 objects/s recovering 2024-05-15T13:26:26.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:25 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:26:26.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:25 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:26:26.295 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:25 smithi120 ceph-mon[43961]: pgmap v2301: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.5 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 416 op/s; 7142/268995 objects degraded (2.655%); 35883/268995 objects misplaced (13.340%); 12 MiB/s, 185 objects/s recovering 2024-05-15T13:26:26.295 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:25 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:26:26.295 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:25 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:26:26.946 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:26:27.182 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:26 smithi027 ceph-mon[39795]: osdmap e900: 12 total, 12 up, 10 in 2024-05-15T13:26:27.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:26 smithi120 ceph-mon[43961]: osdmap e900: 12 total, 12 up, 10 in 2024-05-15T13:26:27.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:26 smithi177 ceph-mon[44303]: osdmap e900: 12 total, 12 up, 10 in 2024-05-15T13:26:27.590 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:26:27.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:27.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:26:27.700 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:26:27.836 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:27.907 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:26:27.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:27.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:26:27.985 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T13:26:27.986 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T13:26:27.986 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-05-15T13:26:28.015 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:26:28.063 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:28.148 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:26:28.149 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T13:26:28.149 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-05-15T13:26:28.149 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:26:28.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:27 smithi027 ceph-mon[39795]: pgmap v2303: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 454 op/s; 7142/265431 objects degraded (2.691%); 35159/265431 objects misplaced (13.246%); 11 MiB/s, 176 objects/s recovering 2024-05-15T13:26:28.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:27 smithi120 ceph-mon[43961]: pgmap v2303: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 454 op/s; 7142/265431 objects degraded (2.691%); 35159/265431 objects misplaced (13.246%); 11 MiB/s, 176 objects/s recovering 2024-05-15T13:26:28.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:27 smithi177 ceph-mon[44303]: pgmap v2303: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 454 op/s; 7142/265431 objects degraded (2.691%); 35159/265431 objects misplaced (13.246%); 11 MiB/s, 176 objects/s recovering 2024-05-15T13:26:28.944 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:26:29.179 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:26:29.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:29.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:26:29.297 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:26:29.329 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:29.727 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-05-15T13:26:30.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:29 smithi027 ceph-mon[39795]: pgmap v2304: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 488 op/s; 7171/262446 objects degraded (2.732%); 35019/262446 objects misplaced (13.343%); 9.7 MiB/s, 155 objects/s recovering 2024-05-15T13:26:30.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:29 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 7171/262446 objects degraded (2.732%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:26:30.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:29 smithi027 ceph-mon[39795]: osdmap e901: 12 total, 12 up, 10 in 2024-05-15T13:26:30.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:29 smithi120 ceph-mon[43961]: pgmap v2304: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 488 op/s; 7171/262446 objects degraded (2.732%); 35019/262446 objects misplaced (13.343%); 9.7 MiB/s, 155 objects/s recovering 2024-05-15T13:26:30.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:29 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 7171/262446 objects degraded (2.732%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:26:30.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:29 smithi120 ceph-mon[43961]: osdmap e901: 12 total, 12 up, 10 in 2024-05-15T13:26:30.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:29 smithi177 ceph-mon[44303]: pgmap v2304: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 488 op/s; 7171/262446 objects degraded (2.732%); 35019/262446 objects misplaced (13.343%); 9.7 MiB/s, 155 objects/s recovering 2024-05-15T13:26:30.443 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:29 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 7171/262446 objects degraded (2.732%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:26:30.443 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:29 smithi177 ceph-mon[44303]: osdmap e901: 12 total, 12 up, 10 in 2024-05-15T13:26:30.789 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:26:30.790 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:26:30.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:30.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:26:30.829 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:26:30.897 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:26:30.930 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:30.969 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:26:30.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:30.974 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:26:31.075 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:26:31.168 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:31.235 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:26:31.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:31.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:26:31.351 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:26:31.382 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:32.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:31 smithi027 ceph-mon[39795]: pgmap v2306: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 472 op/s; 7171/260274 objects degraded (2.755%); 34356/260274 objects misplaced (13.200%); 12 MiB/s, 186 objects/s recovering 2024-05-15T13:26:32.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:31 smithi120 ceph-mon[43961]: pgmap v2306: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 472 op/s; 7171/260274 objects degraded (2.755%); 34356/260274 objects misplaced (13.200%); 12 MiB/s, 186 objects/s recovering 2024-05-15T13:26:32.386 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:26:32.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:31 smithi177 ceph-mon[44303]: pgmap v2306: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 472 op/s; 7171/260274 objects degraded (2.755%); 34356/260274 objects misplaced (13.200%); 12 MiB/s, 186 objects/s recovering 2024-05-15T13:26:33.839 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:26:33.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:33.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:26:33.944 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:26:33.980 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:34.008 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:26:34.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:34.016 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:26:34.106 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:33 smithi027 ceph-mon[39795]: pgmap v2307: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.2 GiB data, 22 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 464 op/s; 7171/257832 objects degraded (2.781%); 34089/257832 objects misplaced (13.221%); 12 MiB/s, 184 objects/s recovering 2024-05-15T13:26:34.116 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:26:34.164 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:34.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:33 smithi120 ceph-mon[43961]: pgmap v2307: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.2 GiB data, 22 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 464 op/s; 7171/257832 objects degraded (2.781%); 34089/257832 objects misplaced (13.221%); 12 MiB/s, 184 objects/s recovering 2024-05-15T13:26:34.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:34 smithi177 ceph-mon[44303]: pgmap v2307: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.2 GiB data, 22 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 464 op/s; 7171/257832 objects degraded (2.781%); 34089/257832 objects misplaced (13.221%); 12 MiB/s, 184 objects/s recovering 2024-05-15T13:26:35.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:34 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/583320853' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:26:35.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:34 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 7171/257832 objects degraded (2.781%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:26:35.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:35 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/583320853' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:26:35.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:35 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 7171/257832 objects degraded (2.781%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:26:35.444 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:35 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/583320853' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:26:35.444 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:35 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 7171/257832 objects degraded (2.781%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:26:35.597 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:26:35.821 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:26:35.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:35.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:26:35.926 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:26:36.048 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:36.094 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:26:36.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:36.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:26:36.204 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:26:36.231 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:36 smithi027 ceph-mon[39795]: pgmap v2308: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.2 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 424 op/s; 7221/255072 objects degraded (2.831%); 33744/255072 objects misplaced (13.229%); 10 MiB/s, 162 objects/s recovering 2024-05-15T13:26:36.231 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:36 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:26:36.231 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:36 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/583320853' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:26:36.231 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:36 smithi027 ceph-mon[39795]: osdmap e902: 12 total, 12 up, 10 in 2024-05-15T13:26:36.234 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:36.307 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:36 smithi120 ceph-mon[43961]: pgmap v2308: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.2 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 424 op/s; 7221/255072 objects degraded (2.831%); 33744/255072 objects misplaced (13.229%); 10 MiB/s, 162 objects/s recovering 2024-05-15T13:26:36.307 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:36 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:26:36.307 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:36 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/583320853' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:26:36.307 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:36 smithi120 ceph-mon[43961]: osdmap e902: 12 total, 12 up, 10 in 2024-05-15T13:26:36.384 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:26:36.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:36.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:26:36.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:36 smithi177 ceph-mon[44303]: pgmap v2308: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.2 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 424 op/s; 7221/255072 objects degraded (2.831%); 33744/255072 objects misplaced (13.229%); 10 MiB/s, 162 objects/s recovering 2024-05-15T13:26:36.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:36 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:26:36.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:36 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/583320853' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:26:36.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:36 smithi177 ceph-mon[44303]: osdmap e902: 12 total, 12 up, 10 in 2024-05-15T13:26:36.493 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:26:36.524 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:36.557 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:26:36.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:36.561 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:26:36.662 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:26:36.698 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:37.055 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:26:37.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:37.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:26:37.160 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:26:37.212 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:26:37.238 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:26:37.260 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:26:37.284 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:26:37.291 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:37.307 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:26:37.347 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:26:37.372 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:26:37.374 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:26:37.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:37.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:26:37.418 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:26:37.441 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:26:37.464 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:26:37.476 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:26:37.491 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:26:37.514 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:26:37.518 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:26:37.518 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:26:37.518 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:26:37.518 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:26:37.519 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:26:37.519 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:26:37.519 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:26:37.519 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:26:37.519 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:26:37.519 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:26:37.519 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:26:37.519 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:26:37.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:26:37.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:37 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:26:37.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:37 smithi120 ceph-mon[43961]: pgmap v2310: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 444 op/s; 7231/251781 objects degraded (2.872%); 32972/251781 objects misplaced (13.096%); 13 MiB/s, 212 objects/s recovering 2024-05-15T13:26:37.544 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:37 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3157439258' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:26:37.564 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:37.591 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:26:37.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:37.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:26:37.647 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:26:37.648 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:37 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:26:37.648 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:37 smithi027 ceph-mon[39795]: pgmap v2310: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 444 op/s; 7231/251781 objects degraded (2.872%); 32972/251781 objects misplaced (13.096%); 13 MiB/s, 212 objects/s recovering 2024-05-15T13:26:37.648 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:37 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3157439258' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:26:37.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:26:37.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:37 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:26:37.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:37 smithi177 ceph-mon[44303]: pgmap v2310: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 444 op/s; 7231/251781 objects degraded (2.872%); 32972/251781 objects misplaced (13.096%); 13 MiB/s, 212 objects/s recovering 2024-05-15T13:26:37.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:37 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3157439258' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:26:37.712 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:26:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:38.409 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:26:38.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:38 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:26:38.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:38 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:26:38.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:38 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:26:38.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:38 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:26:38.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:38 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:26:38.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:38 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:26:39.050 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:38 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:26:39.050 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:38 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:26:39.050 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:38 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:26:39.767 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:26:39.767 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T13:26:39.767 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-05-15T13:26:39.767 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-05-15T13:26:39.767 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:26:39.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:39 smithi177 ceph-mon[44303]: pgmap v2311: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 478 op/s; 7242/248910 objects degraded (2.909%); 32844/248910 objects misplaced (13.195%); 12 MiB/s, 193 objects/s recovering 2024-05-15T13:26:39.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:39 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 7231/251781 objects degraded (2.872%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:26:39.943 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:39 smithi177 ceph-mon[44303]: osdmap e903: 12 total, 12 up, 10 in 2024-05-15T13:26:40.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:39 smithi027 ceph-mon[39795]: pgmap v2311: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 478 op/s; 7242/248910 objects degraded (2.909%); 32844/248910 objects misplaced (13.195%); 12 MiB/s, 193 objects/s recovering 2024-05-15T13:26:40.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:39 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 7231/251781 objects degraded (2.872%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:26:40.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:39 smithi027 ceph-mon[39795]: osdmap e903: 12 total, 12 up, 10 in 2024-05-15T13:26:40.055 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:39 smithi120 ceph-mon[43961]: pgmap v2311: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 478 op/s; 7242/248910 objects degraded (2.909%); 32844/248910 objects misplaced (13.195%); 12 MiB/s, 193 objects/s recovering 2024-05-15T13:26:40.055 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:39 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 7231/251781 objects degraded (2.872%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:26:40.055 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:39 smithi120 ceph-mon[43961]: osdmap e903: 12 total, 12 up, 10 in 2024-05-15T13:26:40.115 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:26:40.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:40.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:26:40.229 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:26:40.257 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:40.753 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:26:40.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:40.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:26:40.786 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-05-15T13:26:40.860 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:26:40.893 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:40.916 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:26:40.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:40.920 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:26:41.020 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:26:41.119 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:41.268 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:26:41.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:41.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:26:41.474 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:26:41.504 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:41.527 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:26:41.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:41.534 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:26:41.635 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:26:41.668 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:42.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:41 smithi177 ceph-mon[44303]: pgmap v2313: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 463 op/s; 7303/246714 objects degraded (2.960%); 32072/246714 objects misplaced (13.000%); 12 MiB/s, 198 objects/s recovering 2024-05-15T13:26:42.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:41 smithi027 ceph-mon[39795]: pgmap v2313: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 463 op/s; 7303/246714 objects degraded (2.960%); 32072/246714 objects misplaced (13.000%); 12 MiB/s, 198 objects/s recovering 2024-05-15T13:26:42.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:41 smithi120 ceph-mon[43961]: pgmap v2313: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 463 op/s; 7303/246714 objects degraded (2.960%); 32072/246714 objects misplaced (13.000%); 12 MiB/s, 198 objects/s recovering 2024-05-15T13:26:42.619 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:26:42.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:42.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:26:42.630 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:26:42.725 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:26:42.760 INFO:teuthology.orchestra.run.smithi027.stderr:no valid command found; 10 closest matches: 2024-05-15T13:26:42.760 INFO:teuthology.orchestra.run.smithi027.stderr:0 2024-05-15T13:26:42.760 INFO:teuthology.orchestra.run.smithi027.stderr:1 2024-05-15T13:26:42.760 INFO:teuthology.orchestra.run.smithi027.stderr:2 2024-05-15T13:26:42.760 INFO:teuthology.orchestra.run.smithi027.stderr:abort 2024-05-15T13:26:42.760 INFO:teuthology.orchestra.run.smithi027.stderr:assert 2024-05-15T13:26:42.760 INFO:teuthology.orchestra.run.smithi027.stderr:bench [] [] [] [] 2024-05-15T13:26:42.760 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs debug_inject_read_zeros 2024-05-15T13:26:42.760 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs files list 2024-05-15T13:26:42.761 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs stats 2024-05-15T13:26:42.761 INFO:teuthology.orchestra.run.smithi027.stderr:bluestore allocator dump block 2024-05-15T13:26:42.761 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: invalid command 2024-05-15T13:26:42.846 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:42.891 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:26:42.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:42.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:26:43.004 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:26:43.073 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:43.164 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:26:43.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:43.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:26:43.284 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:26:43.319 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:43.349 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:26:43.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:43.355 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:26:43.456 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:26:43.478 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:26:43.490 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:44.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:43 smithi177 ceph-mon[44303]: pgmap v2314: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 444 op/s; 7303/244398 objects degraded (2.988%); 31796/244398 objects misplaced (13.010%); 12 MiB/s, 196 objects/s recovering 2024-05-15T13:26:44.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:43 smithi027 ceph-mon[39795]: pgmap v2314: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 444 op/s; 7303/244398 objects degraded (2.988%); 31796/244398 objects misplaced (13.010%); 12 MiB/s, 196 objects/s recovering 2024-05-15T13:26:44.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:43 smithi120 ceph-mon[43961]: pgmap v2314: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 5.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 444 op/s; 7303/244398 objects degraded (2.988%); 31796/244398 objects misplaced (13.010%); 12 MiB/s, 196 objects/s recovering 2024-05-15T13:26:44.317 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T13:26:44.317 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T13:26:44.318 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-05-15T13:26:44.614 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:26:45.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:44 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 7303/244398 objects degraded (2.988%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:26:45.188 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:26:45.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:44 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 7303/244398 objects degraded (2.988%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:26:45.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:45.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:26:45.296 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:26:45.297 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:44 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 7303/244398 objects degraded (2.988%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:26:45.327 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:45.460 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:26:45.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:45.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:26:45.564 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:26:45.598 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:45.804 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:26:45.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:45.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:26:45.908 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:26:45.942 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:45.984 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:26:45.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:45.997 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:26:46.098 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:26:46.123 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:45 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:26:46.124 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:45 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:26:46.124 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:45 smithi177 ceph-mon[44303]: pgmap v2315: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 431 op/s; 7307/242448 objects degraded (3.014%); 31654/242448 objects misplaced (13.056%); 11 MiB/s, 175 objects/s recovering 2024-05-15T13:26:46.124 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:45 smithi177 ceph-mon[44303]: osdmap e904: 12 total, 12 up, 10 in 2024-05-15T13:26:46.125 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:46.152 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:26:46.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:46.157 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:26:46.228 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:45 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:26:46.228 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:45 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:26:46.228 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:45 smithi027 ceph-mon[39795]: pgmap v2315: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 431 op/s; 7307/242448 objects degraded (3.014%); 31654/242448 objects misplaced (13.056%); 11 MiB/s, 175 objects/s recovering 2024-05-15T13:26:46.228 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:45 smithi027 ceph-mon[39795]: osdmap e904: 12 total, 12 up, 10 in 2024-05-15T13:26:46.257 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:26:46.289 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:46.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:45 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:26:46.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:45 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:26:46.291 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:45 smithi120 ceph-mon[43961]: pgmap v2315: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 431 op/s; 7307/242448 objects degraded (3.014%); 31654/242448 objects misplaced (13.056%); 11 MiB/s, 175 objects/s recovering 2024-05-15T13:26:46.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:45 smithi120 ceph-mon[43961]: osdmap e904: 12 total, 12 up, 10 in 2024-05-15T13:26:46.330 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:26:46.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:46.336 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:26:46.437 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:26:46.526 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:46.561 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:26:46.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:46.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:26:46.675 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:26:46.705 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:46.741 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:26:46.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:46.752 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:26:46.853 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:26:46.890 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:46.915 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:26:46.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:46.922 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:26:47.023 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:26:47.078 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:47.117 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:26:47.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:47.120 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:26:47.221 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:26:47.261 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:47.291 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:26:47.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:47.294 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:26:47.395 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:26:47.452 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:47.483 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:26:47.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:47.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:26:47.592 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:26:47.623 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:47.702 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:26:47.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:47.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:26:47.808 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:26:47.839 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:47.862 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:26:47.865 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:26:47.865 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:26:47.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:47.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:26:47.967 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:26:47.995 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:48.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:47 smithi177 ceph-mon[44303]: pgmap v2317: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 425 op/s; 7327/238689 objects degraded (3.070%); 30614/238689 objects misplaced (12.826%); 15 MiB/s, 233 objects/s recovering 2024-05-15T13:26:48.216 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:47 smithi027 ceph-mon[39795]: pgmap v2317: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 425 op/s; 7327/238689 objects degraded (3.070%); 30614/238689 objects misplaced (12.826%); 15 MiB/s, 233 objects/s recovering 2024-05-15T13:26:48.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:47 smithi120 ceph-mon[43961]: pgmap v2317: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 425 op/s; 7327/238689 objects degraded (3.070%); 30614/238689 objects misplaced (12.826%); 15 MiB/s, 233 objects/s recovering 2024-05-15T13:26:48.878 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:26:49.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:48 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:26:49.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:48 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:26:49.194 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:48 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/376071150' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:26:49.252 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:26:49.255 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:48 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:26:49.255 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:48 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:26:49.255 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:48 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/376071150' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:26:49.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:49.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:26:49.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:48 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:26:49.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:48 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:26:49.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:48 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/376071150' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:26:49.357 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:26:49.419 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:49.462 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:26:49.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:49.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:26:49.571 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:26:49.606 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:49.682 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:26:49.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:49.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:26:49.794 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:26:49.874 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:49 smithi027 ceph-mon[39795]: pgmap v2318: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.8 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 475 op/s; 7361/235476 objects degraded (3.126%); 30472/235476 objects misplaced (12.941%); 13 MiB/s, 210 objects/s recovering 2024-05-15T13:26:49.874 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:49 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:26:49.874 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:49 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/376071150' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:26:49.874 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:49 smithi027 ceph-mon[39795]: osdmap e905: 12 total, 12 up, 10 in 2024-05-15T13:26:49.874 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:49 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 7361/235476 objects degraded (3.126%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:26:49.881 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:49.947 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:26:49.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:49.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:26:50.054 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:26:50.133 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:26:50.133 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:26:50.155 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:49 smithi120 ceph-mon[43961]: pgmap v2318: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.8 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 475 op/s; 7361/235476 objects degraded (3.126%); 30472/235476 objects misplaced (12.941%); 13 MiB/s, 210 objects/s recovering 2024-05-15T13:26:50.155 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:49 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:26:50.155 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:49 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/376071150' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:26:50.155 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:49 smithi120 ceph-mon[43961]: osdmap e905: 12 total, 12 up, 10 in 2024-05-15T13:26:50.155 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:49 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 7361/235476 objects degraded (3.126%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:26:50.161 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:50.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:49 smithi177 ceph-mon[44303]: pgmap v2318: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.8 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 475 op/s; 7361/235476 objects degraded (3.126%); 30472/235476 objects misplaced (12.941%); 13 MiB/s, 210 objects/s recovering 2024-05-15T13:26:50.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:49 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:26:50.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:49 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/376071150' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:26:50.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:49 smithi177 ceph-mon[44303]: osdmap e905: 12 total, 12 up, 10 in 2024-05-15T13:26:50.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:49 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 7361/235476 objects degraded (3.126%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:26:50.255 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:26:50.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:50.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:26:50.369 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:26:50.465 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:50.534 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:26:50.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:50.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:26:50.645 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:26:50.686 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:50.712 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:26:50.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:50.720 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:26:50.821 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:26:50.856 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:50.884 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:26:50.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:50.889 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:26:50.990 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:26:51.025 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:51.054 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:26:51.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:51.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:26:51.161 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:26:51.191 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:51.223 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:26:51.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:51.228 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:26:51.329 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:26:51.362 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:51.395 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:26:51.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:51.398 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:26:51.499 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:26:51.629 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:51.679 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:26:51.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:51.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:26:51.791 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:26:51.833 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:51.857 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:26:51.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:51.863 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:26:51.929 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-05-15T13:26:51.963 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:26:52.005 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:52.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:51 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:26:52.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:51 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:26:52.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:51 smithi177 ceph-mon[44303]: pgmap v2320: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 461 op/s; 7389/233313 objects degraded (3.167%); 29691/233313 objects misplaced (12.726%); 13 MiB/s, 217 objects/s recovering 2024-05-15T13:26:52.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:51 smithi177 ceph-mon[44303]: osdmap e906: 12 total, 12 up, 10 in 2024-05-15T13:26:52.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:51 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:26:52.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:51 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:26:52.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:51 smithi027 ceph-mon[39795]: pgmap v2320: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 461 op/s; 7389/233313 objects degraded (3.167%); 29691/233313 objects misplaced (12.726%); 13 MiB/s, 217 objects/s recovering 2024-05-15T13:26:52.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:51 smithi027 ceph-mon[39795]: osdmap e906: 12 total, 12 up, 10 in 2024-05-15T13:26:52.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:51 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:26:52.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:51 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:26:52.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:51 smithi120 ceph-mon[43961]: pgmap v2320: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 461 op/s; 7389/233313 objects degraded (3.167%); 29691/233313 objects misplaced (12.726%); 13 MiB/s, 217 objects/s recovering 2024-05-15T13:26:52.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:51 smithi120 ceph-mon[43961]: osdmap e906: 12 total, 12 up, 10 in 2024-05-15T13:26:52.958 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:26:52.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:52.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:26:53.064 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:26:53.097 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:53.142 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:26:53.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:53.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:26:53.255 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:26:53.284 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:53.573 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:26:53.841 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:26:53.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:53.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:26:53.957 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:26:54.060 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:54.087 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:53 smithi027 ceph-mon[39795]: pgmap v2322: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 445 op/s; 7389/230574 objects degraded (3.205%); 29411/230574 objects misplaced (12.756%); 11 MiB/s, 169 objects/s recovering 2024-05-15T13:26:54.087 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:53 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:26:54.087 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:53 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:26:54.106 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:26:54.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:54.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:26:54.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:53 smithi177 ceph-mon[44303]: pgmap v2322: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 445 op/s; 7389/230574 objects degraded (3.205%); 29411/230574 objects misplaced (12.756%); 11 MiB/s, 169 objects/s recovering 2024-05-15T13:26:54.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:53 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:26:54.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:53 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:26:54.211 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:26:54.285 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:53 smithi120 ceph-mon[43961]: pgmap v2322: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 445 op/s; 7389/230574 objects degraded (3.205%); 29411/230574 objects misplaced (12.756%); 11 MiB/s, 169 objects/s recovering 2024-05-15T13:26:54.285 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:53 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:26:54.285 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:53 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:26:54.294 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:54.347 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:26:54.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:54.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:26:54.446 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:26:54.458 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:26:54.521 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:55.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:54 smithi027 ceph-mon[39795]: osdmap e907: 12 total, 12 up, 10 in 2024-05-15T13:26:55.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:54 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 7389/230574 objects degraded (3.205%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:26:55.297 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:54 smithi120 ceph-mon[43961]: osdmap e907: 12 total, 12 up, 10 in 2024-05-15T13:26:55.298 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:54 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 7389/230574 objects degraded (3.205%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:26:55.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:54 smithi177 ceph-mon[44303]: osdmap e907: 12 total, 12 up, 10 in 2024-05-15T13:26:55.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:54 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 7389/230574 objects degraded (3.205%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:26:56.249 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:55 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:26:56.249 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:55 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:26:56.249 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:55 smithi027 ceph-mon[39795]: pgmap v2324: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.6 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 394 op/s; 7410/228378 objects degraded (3.245%); 29271/228378 objects misplaced (12.817%); 10 MiB/s, 166 objects/s recovering 2024-05-15T13:26:56.250 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:55 smithi027 ceph-mon[39795]: osdmap e908: 12 total, 12 up, 10 in 2024-05-15T13:26:56.250 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:55 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2359164963' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:26:56.302 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:55 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:26:56.302 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:55 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:26:56.302 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:55 smithi120 ceph-mon[43961]: pgmap v2324: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.6 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 394 op/s; 7410/228378 objects degraded (3.245%); 29271/228378 objects misplaced (12.817%); 10 MiB/s, 166 objects/s recovering 2024-05-15T13:26:56.302 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:55 smithi120 ceph-mon[43961]: osdmap e908: 12 total, 12 up, 10 in 2024-05-15T13:26:56.302 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:55 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2359164963' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:26:56.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:55 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:26:56.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:55 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:26:56.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:55 smithi177 ceph-mon[44303]: pgmap v2324: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.6 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 394 op/s; 7410/228378 objects degraded (3.245%); 29271/228378 objects misplaced (12.817%); 10 MiB/s, 166 objects/s recovering 2024-05-15T13:26:56.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:55 smithi177 ceph-mon[44303]: osdmap e908: 12 total, 12 up, 10 in 2024-05-15T13:26:56.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:55 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2359164963' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:26:56.485 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:26:56.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:56.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:26:56.592 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:26:56.629 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:56.682 INFO:teuthology.orchestra.run.smithi027.stderr:no valid command found; 10 closest matches: 2024-05-15T13:26:56.682 INFO:teuthology.orchestra.run.smithi027.stderr:0 2024-05-15T13:26:56.682 INFO:teuthology.orchestra.run.smithi027.stderr:1 2024-05-15T13:26:56.683 INFO:teuthology.orchestra.run.smithi027.stderr:2 2024-05-15T13:26:56.683 INFO:teuthology.orchestra.run.smithi027.stderr:abort 2024-05-15T13:26:56.683 INFO:teuthology.orchestra.run.smithi027.stderr:assert 2024-05-15T13:26:56.683 INFO:teuthology.orchestra.run.smithi027.stderr:bench [] [] [] [] 2024-05-15T13:26:56.683 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs debug_inject_read_zeros 2024-05-15T13:26:56.683 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs files list 2024-05-15T13:26:56.683 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs stats 2024-05-15T13:26:56.683 INFO:teuthology.orchestra.run.smithi027.stderr:bluestore allocator dump block 2024-05-15T13:26:56.683 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: invalid command 2024-05-15T13:26:57.486 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:26:57.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:57.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:26:57.592 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:26:57.644 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:57.684 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:26:57.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:57.693 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:26:57.794 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:26:57.827 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:57.999 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T13:26:57.999 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T13:26:58.273 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:26:58.273 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-05-15T13:26:58.273 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.273 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26536.0:167525 9.1 9:8d7e5565:::benchmark_data_smithi177_159919_object47282:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e908)", 2024-05-15T13:26:58.273 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:26:58.257775+0000", 2024-05-15T13:26:58.273 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.014562204, 2024-05-15T13:26:58.273 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014586606, 2024-05-15T13:26:58.273 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:26:58.273 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:26:58.274 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:26:58.274 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26536", 2024-05-15T13:26:58.274 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:26:58.274 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 167525 2024-05-15T13:26:58.274 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.274 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:26:58.274 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.274 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:26:58.274 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.257775+0000", 2024-05-15T13:26:58.274 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:26:58.274 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.274 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.274 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:26:58.274 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.257775+0000", 2024-05-15T13:26:58.275 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:26:58.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.275 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.275 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:26:58.275 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.257775+0000", 2024-05-15T13:26:58.275 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2200000000000004e-07 2024-05-15T13:26:58.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.275 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.275 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:26:58.275 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.257779+0000", 2024-05-15T13:26:58.275 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5080000000000001e-06 2024-05-15T13:26:58.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.275 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.276 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:26:58.276 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.257787+0000", 2024-05-15T13:26:58.276 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0339999999999997e-06 2024-05-15T13:26:58.276 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.276 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.276 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:26:58.276 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.257791+0000", 2024-05-15T13:26:58.276 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.707e-06 2024-05-15T13:26:58.276 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.276 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.276 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:26:58.276 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.257836+0000", 2024-05-15T13:26:58.276 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4685999999999999e-05 2024-05-15T13:26:58.276 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.276 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.277 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:26:58.277 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.257937+0000", 2024-05-15T13:26:58.277 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000101357 2024-05-15T13:26:58.277 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.277 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.277 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6,9", 2024-05-15T13:26:58.277 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.258017+0000", 2024-05-15T13:26:58.277 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9734999999999995e-05 2024-05-15T13:26:58.277 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.277 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.277 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-05-15T13:26:58.277 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.262458+0000", 2024-05-15T13:26:58.277 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0044409779999999999 2024-05-15T13:26:58.278 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.278 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.278 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:26:58.278 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.264183+0000", 2024-05-15T13:26:58.278 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001724969 2024-05-15T13:26:58.278 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:26:58.278 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:26:58.278 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:26:58.278 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.278 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.279 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26536.0:167528 9.c 9:3671f0ab:::benchmark_data_smithi177_159919_object47285:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e908)", 2024-05-15T13:26:58.279 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:26:58.260326+0000", 2024-05-15T13:26:58.279 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.012010735, 2024-05-15T13:26:58.279 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012093763, 2024-05-15T13:26:58.279 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:26:58.279 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:26:58.279 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:26:58.279 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26536", 2024-05-15T13:26:58.279 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:26:58.279 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 167528 2024-05-15T13:26:58.279 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.279 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:26:58.279 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.280 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:26:58.280 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.260326+0000", 2024-05-15T13:26:58.280 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:26:58.280 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.280 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.280 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:26:58.280 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.260326+0000", 2024-05-15T13:26:58.280 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:26:58.280 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.280 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.280 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:26:58.280 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.260327+0000", 2024-05-15T13:26:58.280 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2410000000000001e-06 2024-05-15T13:26:58.280 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.280 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.281 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:26:58.281 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.260331+0000", 2024-05-15T13:26:58.281 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1030000000000004e-06 2024-05-15T13:26:58.281 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.281 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.281 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:26:58.281 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.260342+0000", 2024-05-15T13:26:58.281 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1083999999999999e-05 2024-05-15T13:26:58.281 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.281 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.281 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:26:58.281 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.260349+0000", 2024-05-15T13:26:58.281 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6830000000000002e-06 2024-05-15T13:26:58.282 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.282 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.282 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:26:58.282 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.269661+0000", 2024-05-15T13:26:58.282 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.009311672 2024-05-15T13:26:58.282 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.282 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.282 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:26:58.282 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.269829+0000", 2024-05-15T13:26:58.282 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016806199999999999 2024-05-15T13:26:58.282 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.282 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.282 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 4,9", 2024-05-15T13:26:58.282 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.269991+0000", 2024-05-15T13:26:58.282 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016232700000000001 2024-05-15T13:26:58.283 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:26:58.283 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:26:58.283 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:26:58.283 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.283 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.283 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26536.0:167531 9.c 9.6f5f8bbc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e908)", 2024-05-15T13:26:58.283 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:26:58.263582+0000", 2024-05-15T13:26:58.283 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.0087543480000000003, 2024-05-15T13:26:58.283 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0088799440000000007, 2024-05-15T13:26:58.283 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:26:58.283 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "queued for pg", 2024-05-15T13:26:58.283 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:26:58.283 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26536", 2024-05-15T13:26:58.284 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:26:58.284 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 167531 2024-05-15T13:26:58.284 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.284 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:26:58.284 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.284 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:26:58.284 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.263582+0000", 2024-05-15T13:26:58.284 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:26:58.284 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.284 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.284 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:26:58.284 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.263582+0000", 2024-05-15T13:26:58.284 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:26:58.284 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.284 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.285 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:26:58.285 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.263583+0000", 2024-05-15T13:26:58.285 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0079999999999999e-06 2024-05-15T13:26:58.285 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.285 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.285 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:26:58.285 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.263587+0000", 2024-05-15T13:26:58.285 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4979999999999998e-06 2024-05-15T13:26:58.285 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.285 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.285 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:26:58.285 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.263597+0000", 2024-05-15T13:26:58.285 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0108e-05 2024-05-15T13:26:58.285 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.285 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.286 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:26:58.286 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.263605+0000", 2024-05-15T13:26:58.286 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2830000000000001e-06 2024-05-15T13:26:58.286 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:26:58.286 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:26:58.286 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:26:58.286 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.286 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.286 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.26536.0:166811 9.4 9:2024eb97:::benchmark_data_smithi177_159919_object46568:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e908)", 2024-05-15T13:26:58.286 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:26:56.799511+0000", 2024-05-15T13:26:58.286 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 1.4728255770000001, 2024-05-15T13:26:58.286 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.472985177, 2024-05-15T13:26:58.286 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:26:58.286 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "waiting for missing object", 2024-05-15T13:26:58.287 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-05-15T13:26:58.287 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.26536", 2024-05-15T13:26:58.287 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:26:58.287 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 166811 2024-05-15T13:26:58.287 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.287 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:26:58.287 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.287 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:26:58.287 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:56.799511+0000", 2024-05-15T13:26:58.287 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:26:58.287 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.287 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.287 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:26:58.288 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:56.799511+0000", 2024-05-15T13:26:58.288 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:26:58.288 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.288 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.288 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:26:58.288 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:56.799511+0000", 2024-05-15T13:26:58.288 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9999999999999999e-07 2024-05-15T13:26:58.288 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.288 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.288 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:26:58.288 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:56.799513+0000", 2024-05-15T13:26:58.288 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6670000000000001e-06 2024-05-15T13:26:58.288 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.289 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.289 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:26:58.289 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:56.799518+0000", 2024-05-15T13:26:58.289 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9049999999999996e-06 2024-05-15T13:26:58.289 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.289 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.289 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:26:58.289 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:56.799521+0000", 2024-05-15T13:26:58.290 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4000000000000001e-06 2024-05-15T13:26:58.290 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.290 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.290 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:26:58.290 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:56.799584+0000", 2024-05-15T13:26:58.290 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2810000000000003e-05 2024-05-15T13:26:58.290 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.290 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.290 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for missing object", 2024-05-15T13:26:58.290 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:56.799680+0000", 2024-05-15T13:26:58.290 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5736000000000006e-05 2024-05-15T13:26:58.291 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:26:58.291 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:26:58.291 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:26:58.291 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.291 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.291 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.26536.0:167529 9.a e908/866 9:5ead0508:::benchmark_data_smithi177_159919_object47286:head v 908'10395, mlcod=831'4512)", 2024-05-15T13:26:58.291 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-05-15T13:26:58.268972+0000", 2024-05-15T13:26:58.291 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.003365023, 2024-05-15T13:26:58.291 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035653809999999998, 2024-05-15T13:26:58.291 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-05-15T13:26:58.291 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-05-15T13:26:58.291 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-05-15T13:26:58.291 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.291 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-05-15T13:26:58.291 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.268972+0000", 2024-05-15T13:26:58.292 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:26:58.292 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.292 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.292 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-05-15T13:26:58.292 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.268972+0000", 2024-05-15T13:26:58.292 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-05-15T13:26:58.292 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.292 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.292 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-05-15T13:26:58.292 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.268973+0000", 2024-05-15T13:26:58.292 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0070000000000001e-06 2024-05-15T13:26:58.292 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.292 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.293 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-05-15T13:26:58.293 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.268981+0000", 2024-05-15T13:26:58.293 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4870000000000002e-06 2024-05-15T13:26:58.293 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.293 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.293 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-05-15T13:26:58.293 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.268990+0000", 2024-05-15T13:26:58.293 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2480000000000002e-06 2024-05-15T13:26:58.293 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.293 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.293 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-05-15T13:26:58.293 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.268996+0000", 2024-05-15T13:26:58.293 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5729999999999998e-06 2024-05-15T13:26:58.293 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.294 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.294 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-05-15T13:26:58.294 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.269047+0000", 2024-05-15T13:26:58.294 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1149000000000001e-05 2024-05-15T13:26:58.294 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-05-15T13:26:58.294 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-05-15T13:26:58.294 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-05-15T13:26:58.294 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-05-15T13:26:58.269072+0000", 2024-05-15T13:26:58.294 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4796000000000001e-05 2024-05-15T13:26:58.294 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:26:58.294 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-05-15T13:26:58.294 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:26:58.294 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-05-15T13:26:58.294 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-05-15T13:26:58.295 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 5 2024-05-15T13:26:58.295 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:26:58.295 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:26:58.295 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:26:57 smithi027 ceph-mon[39795]: pgmap v2326: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.6 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 499 op/s; 7484/224322 objects degraded (3.336%); 28178/224322 objects misplaced (12.561%); 13 MiB/s, 210 objects/s recovering 2024-05-15T13:26:58.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:58.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:26:58.297 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:26:57 smithi120 ceph-mon[43961]: pgmap v2326: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.6 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 499 op/s; 7484/224322 objects degraded (3.336%); 28178/224322 objects misplaced (12.561%); 13 MiB/s, 210 objects/s recovering 2024-05-15T13:26:58.397 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:26:58.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:26:57 smithi177 ceph-mon[44303]: pgmap v2326: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.6 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 499 op/s; 7484/224322 objects degraded (3.336%); 28178/224322 objects misplaced (12.561%); 13 MiB/s, 210 objects/s recovering 2024-05-15T13:26:58.512 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:58.556 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:26:58.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:58.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:26:58.609 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:26:58.610 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:26:58.610 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:26:58.610 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:26:58.610 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:26:58.610 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:26:58.610 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:26:58.610 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:26:58.610 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:26:58.610 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:26:58.610 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:26:58.610 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:26:58.611 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:26:58.611 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:26:58.611 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:26:58.611 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:26:58.611 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:26:58.611 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:26:58.611 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:26:58.611 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:26:58.611 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:26:58.611 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:26:58.611 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:26:58.611 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:26:58.673 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:26:58.711 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:58.742 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:26:58.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:58.746 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:26:58.847 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:26:58.965 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:59.048 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:26:59.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:59.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:26:59.160 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:26:59.192 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:26:59.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:59.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:26:59.457 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:26:59.493 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:59.520 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:26:59.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:59.524 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:26:59.625 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:26:59.708 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:59.730 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:26:59.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:59.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:26:59.819 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:26:59.835 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:26:59.851 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-05-15T13:26:59.872 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:26:59.971 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:26:59.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:26:59.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:27:00.012 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:00 smithi120 ceph-mon[43961]: pgmap v2327: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 479 op/s; 7496/221094 objects degraded (3.390%); 28044/221094 objects misplaced (12.684%); 11 MiB/s, 176 objects/s recovering 2024-05-15T13:27:00.012 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:00 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 7496/221094 objects degraded (3.390%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:00.012 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:00 smithi120 ceph-mon[43961]: osdmap e909: 12 total, 12 up, 10 in 2024-05-15T13:27:00.077 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:27:00.086 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:00 smithi027 ceph-mon[39795]: pgmap v2327: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 479 op/s; 7496/221094 objects degraded (3.390%); 28044/221094 objects misplaced (12.684%); 11 MiB/s, 176 objects/s recovering 2024-05-15T13:27:00.086 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:00 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 7496/221094 objects degraded (3.390%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:00.086 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:00 smithi027 ceph-mon[39795]: osdmap e909: 12 total, 12 up, 10 in 2024-05-15T13:27:00.179 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:00.229 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:27:00.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:00.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:27:00.337 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:27:00.427 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:00.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:00 smithi177 ceph-mon[44303]: pgmap v2327: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 479 op/s; 7496/221094 objects degraded (3.390%); 28044/221094 objects misplaced (12.684%); 11 MiB/s, 176 objects/s recovering 2024-05-15T13:27:00.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:00 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 7496/221094 objects degraded (3.390%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:00.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:00 smithi177 ceph-mon[44303]: osdmap e909: 12 total, 12 up, 10 in 2024-05-15T13:27:00.502 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:27:00.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:00.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:27:00.609 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:27:00.710 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:00.774 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:27:00.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:00.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:27:00.886 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:27:00.920 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:01.156 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:27:01.172 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:27:01.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:01.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:27:01.279 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:27:01.309 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:01.376 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:27:01.409 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:27:01.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:01.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:27:01.517 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:27:01.601 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:01.638 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:27:01.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:01.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:27:01.749 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:27:01.845 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:01.884 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:27:01.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:01.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:27:01.993 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:27:02.026 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:02.068 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:27:02.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:02.071 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:27:02.172 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:27:02.211 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:02.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:02 smithi027 ceph-mon[39795]: pgmap v2329: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 466 op/s; 7540/218835 objects degraded (3.446%); 27210/218835 objects misplaced (12.434%); 15 MiB/s, 238 objects/s recovering 2024-05-15T13:27:02.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:02 smithi120 ceph-mon[43961]: pgmap v2329: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 466 op/s; 7540/218835 objects degraded (3.446%); 27210/218835 objects misplaced (12.434%); 15 MiB/s, 238 objects/s recovering 2024-05-15T13:27:02.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:02 smithi177 ceph-mon[44303]: pgmap v2329: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 466 op/s; 7540/218835 objects degraded (3.446%); 27210/218835 objects misplaced (12.434%); 15 MiB/s, 238 objects/s recovering 2024-05-15T13:27:02.663 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:27:02.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:02.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:27:02.767 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:27:02.805 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:03.000 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 9, 10, 11, 7, 4] out_osds: [8, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:27:03.000 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:27:03.000 INFO:tasks.thrashosds.thrasher:Removing osd 11, in_osds are: [0, 2, 3, 5, 6, 9, 10, 11, 7, 4] 2024-05-15T13:27:03.000 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd out 11 2024-05-15T13:27:04.040 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:27:04.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:04 smithi177 ceph-mon[44303]: pgmap v2330: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 508 op/s; 7540/216180 objects degraded (3.488%); 26915/216180 objects misplaced (12.450%); 15 MiB/s, 239 objects/s recovering 2024-05-15T13:27:04.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:04 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/914143620' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:27:04.531 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:04 smithi027 ceph-mon[39795]: pgmap v2330: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 508 op/s; 7540/216180 objects degraded (3.488%); 26915/216180 objects misplaced (12.450%); 15 MiB/s, 239 objects/s recovering 2024-05-15T13:27:04.531 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:04 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/914143620' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:27:04.544 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:04 smithi120 ceph-mon[43961]: pgmap v2330: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 508 op/s; 7540/216180 objects degraded (3.488%); 26915/216180 objects misplaced (12.450%); 15 MiB/s, 239 objects/s recovering 2024-05-15T13:27:04.544 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:04 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/914143620' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:27:05.054 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:05 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:27:05.054 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:05 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/914143620' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:27:05.055 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:05 smithi027 ceph-mon[39795]: osdmap e910: 12 total, 12 up, 10 in 2024-05-15T13:27:05.055 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:05 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 7540/216180 objects degraded (3.488%), 6 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:05.055 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:05 smithi027 ceph-mon[39795]: osdmap e911: 12 total, 12 up, 10 in 2024-05-15T13:27:05.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:05 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:27:05.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:05 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/914143620' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:27:05.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:05 smithi177 ceph-mon[44303]: osdmap e910: 12 total, 12 up, 10 in 2024-05-15T13:27:05.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:05 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 7540/216180 objects degraded (3.488%), 6 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:05.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:05 smithi177 ceph-mon[44303]: osdmap e911: 12 total, 12 up, 10 in 2024-05-15T13:27:05.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:05 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:27:05.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:05 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/914143620' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:27:05.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:05 smithi120 ceph-mon[43961]: osdmap e910: 12 total, 12 up, 10 in 2024-05-15T13:27:05.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:05 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 7540/216180 objects degraded (3.488%), 6 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:05.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:05 smithi120 ceph-mon[43961]: osdmap e911: 12 total, 12 up, 10 in 2024-05-15T13:27:06.062 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:27:06.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:06.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:27:06.167 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:27:06.213 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:27:06.213 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:27:06.243 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:06.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:06 smithi027 ceph-mon[39795]: pgmap v2333: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 393 op/s; 7547/214017 objects degraded (3.526%); 26782/214017 objects misplaced (12.514%); 11 MiB/s, 176 objects/s recovering 2024-05-15T13:27:06.310 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:27:06.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:06.334 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:06 smithi120 ceph-mon[43961]: pgmap v2333: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 393 op/s; 7547/214017 objects degraded (3.526%); 26782/214017 objects misplaced (12.514%); 11 MiB/s, 176 objects/s recovering 2024-05-15T13:27:06.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:27:06.379 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-05-15T13:27:06.379 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-05-15T13:27:06.379 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-05-15T13:27:06.379 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-05-15T13:27:06.379 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-05-15T13:27:06.435 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:27:06.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:06 smithi177 ceph-mon[44303]: pgmap v2333: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 393 op/s; 7547/214017 objects degraded (3.526%); 26782/214017 objects misplaced (12.514%); 11 MiB/s, 176 objects/s recovering 2024-05-15T13:27:06.478 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:06.517 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:27:06.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:06.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:27:06.622 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:27:06.686 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:06.735 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:27:06.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:06.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:27:06.847 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:27:06.886 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:06.910 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:27:06.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:06.914 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:27:07.015 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:27:07.118 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:07.141 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:27:07.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:07.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:27:07.255 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:27:07.283 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:07.309 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:27:07.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:07.312 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:27:07.412 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:27:07.496 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:07.552 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:27:07.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:07.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:27:07.663 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:27:07.681 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:27:07.694 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:07.729 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:27:07.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:07.732 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:27:07.833 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:27:07.876 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:08.074 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:08 smithi177 ceph-mon[44303]: pgmap v2334: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 506 op/s; 7595/209817 objects degraded (3.620%); 25626/209817 objects misplaced (12.214%); 17 MiB/s, 268 objects/s recovering 2024-05-15T13:27:08.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:08 smithi027 ceph-mon[39795]: pgmap v2334: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 506 op/s; 7595/209817 objects degraded (3.620%); 25626/209817 objects misplaced (12.214%); 17 MiB/s, 268 objects/s recovering 2024-05-15T13:27:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:27:08.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:08.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:27:08.433 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:27:08.465 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:08.545 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:08 smithi120 ceph-mon[43961]: pgmap v2334: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 506 op/s; 7595/209817 objects degraded (3.620%); 25626/209817 objects misplaced (12.214%); 17 MiB/s, 268 objects/s recovering 2024-05-15T13:27:08.859 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:27:08.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:08.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:27:08.899 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-05-15T13:27:08.966 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:27:09.016 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:09.040 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:27:09.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:09.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:27:09.151 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:27:09.244 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:09.296 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:27:09.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:09.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:27:09.408 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:27:09.444 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:09.474 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:27:09.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:09.476 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:27:09.577 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:27:09.662 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:09.716 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:27:09.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:09.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:27:09.826 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:27:09.924 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:09.983 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:27:09.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:09.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:27:10.093 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:27:10.126 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:10.149 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:27:10.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:10.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:27:10.175 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:10 smithi027 ceph-mon[39795]: pgmap v2335: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 511 op/s; 7637/206562 objects degraded (3.697%); 25468/206562 objects misplaced (12.329%); 11 MiB/s, 183 objects/s recovering 2024-05-15T13:27:10.175 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:10 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 7637/206562 objects degraded (3.697%), 6 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:10.253 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:27:10.287 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:10.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:10 smithi120 ceph-mon[43961]: pgmap v2335: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 511 op/s; 7637/206562 objects degraded (3.697%); 25468/206562 objects misplaced (12.329%); 11 MiB/s, 183 objects/s recovering 2024-05-15T13:27:10.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:10 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 7637/206562 objects degraded (3.697%), 6 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:10.321 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:27:10.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:10.327 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:27:10.429 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:27:10.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:10 smithi177 ceph-mon[44303]: pgmap v2335: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 511 op/s; 7637/206562 objects degraded (3.697%); 25468/206562 objects misplaced (12.329%); 11 MiB/s, 183 objects/s recovering 2024-05-15T13:27:10.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:10 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 7637/206562 objects degraded (3.697%), 6 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:10.464 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:10.494 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:27:10.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:10.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:27:10.599 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:27:10.678 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:10.717 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:27:10.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:10.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:27:10.825 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:27:10.860 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:10.883 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:27:10.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:10.886 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:27:10.987 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:27:11.019 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:11.025 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:27:11.030 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:27:11.053 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:27:11.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:11.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:27:11.162 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:27:11.193 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:12.114 INFO:teuthology.orchestra.run.smithi027.stderr:marked out osd.11. 2024-05-15T13:27:12.139 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:11 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-a[39772]: 2024-05-15T13:27:11.991+0000 7f8daffd2700 -1 mon.a@0(leader).osd e911 definitely_dead 0 2024-05-15T13:27:12.139 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:12 smithi027 ceph-mon[39795]: pgmap v2336: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 492 op/s; 7706/204354 objects degraded (3.771%); 24576/204354 objects misplaced (12.026%); 14 MiB/s, 231 objects/s recovering 2024-05-15T13:27:12.139 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:12 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1925142213' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-05-15T13:27:12.139 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:12 smithi027 ceph-mon[39795]: Client client.admin marked osd.11 out, while it was still marked up 2024-05-15T13:27:12.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:12 smithi177 ceph-mon[44303]: pgmap v2336: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 492 op/s; 7706/204354 objects degraded (3.771%); 24576/204354 objects misplaced (12.026%); 14 MiB/s, 231 objects/s recovering 2024-05-15T13:27:12.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:12 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1925142213' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-05-15T13:27:12.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:12 smithi177 ceph-mon[44303]: Client client.admin marked osd.11 out, while it was still marked up 2024-05-15T13:27:12.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:12 smithi120 ceph-mon[43961]: pgmap v2336: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 492 op/s; 7706/204354 objects degraded (3.771%); 24576/204354 objects misplaced (12.026%); 14 MiB/s, 231 objects/s recovering 2024-05-15T13:27:12.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:12 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1925142213' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-05-15T13:27:12.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:12 smithi120 ceph-mon[43961]: Client client.admin marked osd.11 out, while it was still marked up 2024-05-15T13:27:13.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:13 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1925142213' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-05-15T13:27:13.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:13 smithi027 ceph-mon[39795]: osdmap e912: 12 total, 12 up, 9 in 2024-05-15T13:27:13.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:13 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2171778639' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:27:13.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:13 smithi027 ceph-mon[39795]: osdmap e913: 12 total, 12 up, 9 in 2024-05-15T13:27:13.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:13 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1925142213' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-05-15T13:27:13.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:13 smithi177 ceph-mon[44303]: osdmap e912: 12 total, 12 up, 9 in 2024-05-15T13:27:13.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:13 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2171778639' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:27:13.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:13 smithi177 ceph-mon[44303]: osdmap e913: 12 total, 12 up, 9 in 2024-05-15T13:27:13.456 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:27:13.456 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:27:13.456 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:27:13.456 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:27:13.456 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:27:13.456 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:27:13.457 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:27:13.457 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:27:13.457 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:27:13.457 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:27:13.457 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:27:13.457 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:27:13.457 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:27:13.457 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:27:13.457 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:27:13.457 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:27:13.457 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:27:13.457 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:27:13.457 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:27:13.458 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:27:13.458 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:27:13.458 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:27:13.458 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:27:13.458 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:27:13.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:13 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1925142213' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-05-15T13:27:13.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:13 smithi120 ceph-mon[43961]: osdmap e912: 12 total, 12 up, 9 in 2024-05-15T13:27:13.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:13 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2171778639' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:27:13.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:13 smithi120 ceph-mon[43961]: osdmap e913: 12 total, 12 up, 9 in 2024-05-15T13:27:14.124 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:14 smithi027 ceph-mon[39795]: pgmap v2338: 47 pgs: 2 peering, 1 unknown, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 35 active+clean; 4.1 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 489 op/s; 7706/201654 objects degraded (3.821%); 24274/201654 objects misplaced (12.037%); 15 MiB/s, 240 objects/s recovering 2024-05-15T13:27:14.124 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:14 smithi027 ceph-mon[39795]: 1.7 scrub starts 2024-05-15T13:27:14.124 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:14 smithi027 ceph-mon[39795]: 1.7 scrub ok 2024-05-15T13:27:14.124 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:14 smithi027 ceph-mon[39795]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-05-15T13:27:14.124 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:14 smithi027 ceph-mon[39795]: osdmap e914: 12 total, 12 up, 9 in 2024-05-15T13:27:14.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:14 smithi177 ceph-mon[44303]: pgmap v2338: 47 pgs: 2 peering, 1 unknown, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 35 active+clean; 4.1 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 489 op/s; 7706/201654 objects degraded (3.821%); 24274/201654 objects misplaced (12.037%); 15 MiB/s, 240 objects/s recovering 2024-05-15T13:27:14.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:14 smithi177 ceph-mon[44303]: 1.7 scrub starts 2024-05-15T13:27:14.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:14 smithi177 ceph-mon[44303]: 1.7 scrub ok 2024-05-15T13:27:14.443 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:14 smithi177 ceph-mon[44303]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-05-15T13:27:14.443 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:14 smithi177 ceph-mon[44303]: osdmap e914: 12 total, 12 up, 9 in 2024-05-15T13:27:14.545 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:14 smithi120 ceph-mon[43961]: pgmap v2338: 47 pgs: 2 peering, 1 unknown, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 35 active+clean; 4.1 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 489 op/s; 7706/201654 objects degraded (3.821%); 24274/201654 objects misplaced (12.037%); 15 MiB/s, 240 objects/s recovering 2024-05-15T13:27:14.546 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:14 smithi120 ceph-mon[43961]: 1.7 scrub starts 2024-05-15T13:27:14.546 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:14 smithi120 ceph-mon[43961]: 1.7 scrub ok 2024-05-15T13:27:14.546 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:14 smithi120 ceph-mon[43961]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-05-15T13:27:14.546 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:14 smithi120 ceph-mon[43961]: osdmap e914: 12 total, 12 up, 9 in 2024-05-15T13:27:15.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:15 smithi027 ceph-mon[39795]: 9.8 starting backfill to osd.7 from (0'0,0'0] MAX to 911'10789 2024-05-15T13:27:15.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:15 smithi027 ceph-mon[39795]: 9.f starting backfill to osd.10 from (0'0,0'0] MAX to 911'11074 2024-05-15T13:27:15.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:15 smithi027 ceph-mon[39795]: 9.2 starting backfill to osd.7 from (0'0,0'0] MAX to 911'13355 2024-05-15T13:27:15.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:15 smithi027 ceph-mon[39795]: 9.9 starting backfill to osd.3 from (0'0,0'0] MAX to 911'11058 2024-05-15T13:27:15.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:15 smithi027 ceph-mon[39795]: 9.5 starting backfill to osd.3 from (0'0,0'0] MAX to 911'10929 2024-05-15T13:27:15.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:15 smithi027 ceph-mon[39795]: 9.0 starting backfill to osd.0 from (0'0,0'0] MAX to 911'10588 2024-05-15T13:27:15.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:15 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 7706/201654 objects degraded (3.821%), 6 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:15.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:15 smithi027 ceph-mon[39795]: osdmap e915: 12 total, 12 up, 9 in 2024-05-15T13:27:15.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:15 smithi177 ceph-mon[44303]: 9.8 starting backfill to osd.7 from (0'0,0'0] MAX to 911'10789 2024-05-15T13:27:15.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:15 smithi177 ceph-mon[44303]: 9.f starting backfill to osd.10 from (0'0,0'0] MAX to 911'11074 2024-05-15T13:27:15.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:15 smithi177 ceph-mon[44303]: 9.2 starting backfill to osd.7 from (0'0,0'0] MAX to 911'13355 2024-05-15T13:27:15.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:15 smithi177 ceph-mon[44303]: 9.9 starting backfill to osd.3 from (0'0,0'0] MAX to 911'11058 2024-05-15T13:27:15.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:15 smithi177 ceph-mon[44303]: 9.5 starting backfill to osd.3 from (0'0,0'0] MAX to 911'10929 2024-05-15T13:27:15.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:15 smithi177 ceph-mon[44303]: 9.0 starting backfill to osd.0 from (0'0,0'0] MAX to 911'10588 2024-05-15T13:27:15.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:15 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 7706/201654 objects degraded (3.821%), 6 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:15.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:15 smithi177 ceph-mon[44303]: osdmap e915: 12 total, 12 up, 9 in 2024-05-15T13:27:15.544 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:15 smithi120 ceph-mon[43961]: 9.8 starting backfill to osd.7 from (0'0,0'0] MAX to 911'10789 2024-05-15T13:27:15.545 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:15 smithi120 ceph-mon[43961]: 9.f starting backfill to osd.10 from (0'0,0'0] MAX to 911'11074 2024-05-15T13:27:15.545 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:15 smithi120 ceph-mon[43961]: 9.2 starting backfill to osd.7 from (0'0,0'0] MAX to 911'13355 2024-05-15T13:27:15.545 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:15 smithi120 ceph-mon[43961]: 9.9 starting backfill to osd.3 from (0'0,0'0] MAX to 911'11058 2024-05-15T13:27:15.545 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:15 smithi120 ceph-mon[43961]: 9.5 starting backfill to osd.3 from (0'0,0'0] MAX to 911'10929 2024-05-15T13:27:15.545 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:15 smithi120 ceph-mon[43961]: 9.0 starting backfill to osd.0 from (0'0,0'0] MAX to 911'10588 2024-05-15T13:27:15.545 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:15 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 7706/201654 objects degraded (3.821%), 6 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:15.545 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:15 smithi120 ceph-mon[43961]: osdmap e915: 12 total, 12 up, 9 in 2024-05-15T13:27:16.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:16 smithi027 ceph-mon[39795]: 1.14 scrub starts 2024-05-15T13:27:16.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:16 smithi027 ceph-mon[39795]: 1.14 scrub ok 2024-05-15T13:27:16.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:16 smithi027 ceph-mon[39795]: pgmap v2342: 47 pgs: 1 remapped+peering, 4 peering, 2 unknown, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 31 active+clean; 4.1 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 338 op/s; 2742/200286 objects degraded (1.369%); 28045/200286 objects misplaced (14.002%); 8.1 MiB/s, 130 objects/s recovering 2024-05-15T13:27:16.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:16 smithi027 ceph-mon[39795]: osdmap e916: 12 total, 12 up, 9 in 2024-05-15T13:27:16.248 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:27:16.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:16.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:27:16.360 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:27:16.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:16 smithi177 ceph-mon[44303]: 1.14 scrub starts 2024-05-15T13:27:16.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:16 smithi177 ceph-mon[44303]: 1.14 scrub ok 2024-05-15T13:27:16.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:16 smithi177 ceph-mon[44303]: pgmap v2342: 47 pgs: 1 remapped+peering, 4 peering, 2 unknown, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 31 active+clean; 4.1 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 338 op/s; 2742/200286 objects degraded (1.369%); 28045/200286 objects misplaced (14.002%); 8.1 MiB/s, 130 objects/s recovering 2024-05-15T13:27:16.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:16 smithi177 ceph-mon[44303]: osdmap e916: 12 total, 12 up, 9 in 2024-05-15T13:27:16.451 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:16 smithi120 ceph-mon[43961]: 1.14 scrub starts 2024-05-15T13:27:16.452 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:16 smithi120 ceph-mon[43961]: 1.14 scrub ok 2024-05-15T13:27:16.452 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:16 smithi120 ceph-mon[43961]: pgmap v2342: 47 pgs: 1 remapped+peering, 4 peering, 2 unknown, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 31 active+clean; 4.1 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 338 op/s; 2742/200286 objects degraded (1.369%); 28045/200286 objects misplaced (14.002%); 8.1 MiB/s, 130 objects/s recovering 2024-05-15T13:27:16.452 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:16 smithi120 ceph-mon[43961]: osdmap e916: 12 total, 12 up, 9 in 2024-05-15T13:27:16.461 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:16.519 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:27:16.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:16.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:27:16.627 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:27:16.667 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:16.703 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:27:16.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:16.708 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:27:16.809 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:27:16.895 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:16.926 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-05-15T13:27:16.926 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:27:16.927 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:27:16.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:16.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:27:17.042 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:27:17.077 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:17.167 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:17 smithi120 ceph-mon[43961]: 1.1d scrub starts 2024-05-15T13:27:17.168 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:17 smithi120 ceph-mon[43961]: 1.1d scrub ok 2024-05-15T13:27:17.168 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:17 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:27:17.168 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:17 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:27:17.168 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:17 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:27:17.168 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:17 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:27:17.168 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:17 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:27:17.168 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:17 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:27:17.168 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:17 smithi120 ceph-mon[43961]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:27:17.168 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:17 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1801137488' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:27:17.168 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:17 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:27:17.208 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:27:17.230 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:17 smithi027 ceph-mon[39795]: 1.1d scrub starts 2024-05-15T13:27:17.230 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:17 smithi027 ceph-mon[39795]: 1.1d scrub ok 2024-05-15T13:27:17.230 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:17 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:27:17.230 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:17 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:27:17.230 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:17 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:27:17.230 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:17 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:27:17.230 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:17 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:27:17.230 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:17 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:27:17.231 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:17 smithi027 ceph-mon[39795]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:27:17.231 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:17 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1801137488' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:27:17.231 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:17 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:27:17.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:17 smithi177 ceph-mon[44303]: 1.1d scrub starts 2024-05-15T13:27:17.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:17 smithi177 ceph-mon[44303]: 1.1d scrub ok 2024-05-15T13:27:17.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:17 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:27:17.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:17 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:27:17.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:17 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:27:17.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:17 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:27:17.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:17 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:27:17.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:17 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:27:17.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:17 smithi177 ceph-mon[44303]: from='mgr.14148 ' entity='mgr.y' 2024-05-15T13:27:17.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:17 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1801137488' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:27:17.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:17 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:27:18.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:18 smithi177 ceph-mon[44303]: pgmap v2344: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 remapped+peering, 5 peering, 2 unknown, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 28 active+clean; 4.0 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 336 op/s; 6338/197619 objects degraded (3.207%); 34427/197619 objects misplaced (17.421%); 23 MiB/s, 370 objects/s recovering 2024-05-15T13:27:18.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:18 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:27:18.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:18 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:27:18.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:18 smithi177 ceph-mon[44303]: osdmap e917: 12 total, 12 up, 9 in 2024-05-15T13:27:18.462 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:27:18.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:18 smithi027 ceph-mon[39795]: pgmap v2344: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 remapped+peering, 5 peering, 2 unknown, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 28 active+clean; 4.0 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 336 op/s; 6338/197619 objects degraded (3.207%); 34427/197619 objects misplaced (17.421%); 23 MiB/s, 370 objects/s recovering 2024-05-15T13:27:18.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:18 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:27:18.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:18 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:27:18.466 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:18 smithi027 ceph-mon[39795]: osdmap e917: 12 total, 12 up, 9 in 2024-05-15T13:27:18.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:18.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:27:18.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:18 smithi120 ceph-mon[43961]: pgmap v2344: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 remapped+peering, 5 peering, 2 unknown, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 28 active+clean; 4.0 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 336 op/s; 6338/197619 objects degraded (3.207%); 34427/197619 objects misplaced (17.421%); 23 MiB/s, 370 objects/s recovering 2024-05-15T13:27:18.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:18 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:27:18.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:18 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:27:18.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:18 smithi120 ceph-mon[43961]: osdmap e917: 12 total, 12 up, 9 in 2024-05-15T13:27:18.567 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:27:18.663 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:18.700 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:27:18.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:18.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:27:18.810 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:27:18.843 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:18.888 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:27:18.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:18.896 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:27:18.997 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:27:19.032 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-05-15T13:27:19.035 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:19.065 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:27:19.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:19.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:27:19.169 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:27:19.208 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:19.232 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:27:19.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:19.235 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:27:19.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:19 smithi027 ceph-mon[39795]: pgmap v2346: 47 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 1 active+recovering+remapped, 3 peering, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 31 active+clean; 4.0 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 333 op/s; 6338/195633 objects degraded (3.240%); 41864/195633 objects misplaced (21.399%); 25 MiB/s, 401 objects/s recovering 2024-05-15T13:27:19.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:19 smithi027 ceph-mon[39795]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T13:27:19.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:19 smithi027 ceph-mon[39795]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-05-15T13:27:19.336 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:27:19.373 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:19.394 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:27:19.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:19.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:27:19.455 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:27:19.501 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:27:19.540 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:19.548 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:19 smithi120 ceph-mon[43961]: pgmap v2346: 47 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 1 active+recovering+remapped, 3 peering, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 31 active+clean; 4.0 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 333 op/s; 6338/195633 objects degraded (3.240%); 41864/195633 objects misplaced (21.399%); 25 MiB/s, 401 objects/s recovering 2024-05-15T13:27:19.548 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:19 smithi120 ceph-mon[43961]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T13:27:19.548 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:19 smithi120 ceph-mon[43961]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-05-15T13:27:19.569 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:27:19.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:19.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:27:19.651 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:27:19.674 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:27:19.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:19 smithi177 ceph-mon[44303]: pgmap v2346: 47 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 1 active+recovering+remapped, 3 peering, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 31 active+clean; 4.0 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 333 op/s; 6338/195633 objects degraded (3.240%); 41864/195633 objects misplaced (21.399%); 25 MiB/s, 401 objects/s recovering 2024-05-15T13:27:19.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:19 smithi177 ceph-mon[44303]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T13:27:19.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:19 smithi177 ceph-mon[44303]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-05-15T13:27:19.711 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:19.747 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:27:19.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:19.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:27:19.812 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:27:19.851 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:27:19.893 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:19.915 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:27:19.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:19.918 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:27:20.019 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:27:20.058 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:20.085 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:27:20.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:20.089 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:27:20.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:27:20.190 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:27:20.225 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:20.262 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:27:20.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:20.268 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:27:20.369 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:27:20.383 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:20 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 6338/195633 objects degraded (3.240%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:20.426 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 9, 10, 7, 4] out_osds: [8, 1, 11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:27:20.427 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:27:20.427 INFO:tasks.thrashosds.thrasher:Adding osd 8 2024-05-15T13:27:20.427 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd in 8 2024-05-15T13:27:20.459 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:20.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:20 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 6338/195633 objects degraded (3.240%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:20.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:20 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 6338/195633 objects degraded (3.240%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:21.489 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:21 smithi027 ceph-mon[39795]: pgmap v2347: 47 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.9 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 377 op/s; 6338/193011 objects degraded (3.284%); 40611/193011 objects misplaced (21.041%); 23 MiB/s, 375 objects/s recovering 2024-05-15T13:27:21.490 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:21 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:27:21.490 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:21 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:27:21.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:21 smithi120 ceph-mon[43961]: pgmap v2347: 47 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.9 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 377 op/s; 6338/193011 objects degraded (3.284%); 40611/193011 objects misplaced (21.041%); 23 MiB/s, 375 objects/s recovering 2024-05-15T13:27:21.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:21 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:27:21.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:21 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:27:21.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:21 smithi177 ceph-mon[44303]: pgmap v2347: 47 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.9 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 377 op/s; 6338/193011 objects degraded (3.284%); 40611/193011 objects misplaced (21.041%); 23 MiB/s, 375 objects/s recovering 2024-05-15T13:27:21.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:21 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:27:21.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:21 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:27:21.780 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:27:21.780 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:27:21.837 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:27:21.866 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:27:21.894 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:27:21.918 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:27:21.966 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:27:22.004 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:27:22.032 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:27:22.107 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:27:22.130 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:27:22.168 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:27:22.200 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:27:22.229 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:27:22.259 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:27:22.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:22.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:27:22.370 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:27:22.438 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:22.470 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:22 smithi027 ceph-mon[39795]: osdmap e918: 12 total, 12 up, 9 in 2024-05-15T13:27:22.470 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:22 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1927993935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:27:22.492 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:27:22.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:22.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:27:22.508 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:22 smithi120 ceph-mon[43961]: osdmap e918: 12 total, 12 up, 9 in 2024-05-15T13:27:22.508 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:22 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1927993935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:27:22.606 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:27:22.641 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:22 smithi177 ceph-mon[44303]: osdmap e918: 12 total, 12 up, 9 in 2024-05-15T13:27:22.641 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:22 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1927993935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:27:22.643 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:22.681 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:27:22.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:22.687 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:27:22.788 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:27:22.821 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:22.840 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:27:22.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:22.847 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:27:22.948 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:27:22.985 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:23.010 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:27:23.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:23.018 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:27:23.119 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:27:23.162 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:23.295 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:23 smithi177 ceph-mon[44303]: pgmap v2349: 47 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.9 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 441 op/s; 6338/190224 objects degraded (3.332%); 39680/190224 objects misplaced (20.860%); 25 MiB/s, 395 objects/s recovering 2024-05-15T13:27:23.720 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:27:23.725 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:23 smithi027 ceph-mon[39795]: pgmap v2349: 47 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.9 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 441 op/s; 6338/190224 objects degraded (3.332%); 39680/190224 objects misplaced (20.860%); 25 MiB/s, 395 objects/s recovering 2024-05-15T13:27:23.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:23.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:27:23.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:23 smithi120 ceph-mon[43961]: pgmap v2349: 47 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.9 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 441 op/s; 6338/190224 objects degraded (3.332%); 39680/190224 objects misplaced (20.860%); 25 MiB/s, 395 objects/s recovering 2024-05-15T13:27:23.827 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:27:23.973 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:24.022 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:27:24.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:24.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:27:24.135 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:27:24.182 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:24.209 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:27:24.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:24.212 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:27:24.313 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:27:24.404 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:24.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:24 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 6338/190224 objects degraded (3.332%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:24.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:24 smithi177 ceph-mon[44303]: osdmap e919: 12 total, 12 up, 9 in 2024-05-15T13:27:24.718 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:24 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 6338/190224 objects degraded (3.332%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:24.718 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:24 smithi027 ceph-mon[39795]: osdmap e919: 12 total, 12 up, 9 in 2024-05-15T13:27:24.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:24 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 6338/190224 objects degraded (3.332%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:24.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:24 smithi120 ceph-mon[43961]: osdmap e919: 12 total, 12 up, 9 in 2024-05-15T13:27:25.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:25 smithi177 ceph-mon[44303]: pgmap v2351: 47 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.8 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 382 op/s; 6338/188385 objects degraded (3.364%); 38981/188385 objects misplaced (20.692%); 20 MiB/s, 314 objects/s recovering 2024-05-15T13:27:25.759 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:25 smithi027 ceph-mon[39795]: pgmap v2351: 47 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.8 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 382 op/s; 6338/188385 objects degraded (3.364%); 38981/188385 objects misplaced (20.692%); 20 MiB/s, 314 objects/s recovering 2024-05-15T13:27:25.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:25 smithi120 ceph-mon[43961]: pgmap v2351: 47 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.8 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 382 op/s; 6338/188385 objects degraded (3.364%); 38981/188385 objects misplaced (20.692%); 20 MiB/s, 314 objects/s recovering 2024-05-15T13:27:27.586 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:27:27.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:27.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:27:27.694 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:27:27.733 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:27.795 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:27:28.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:27 smithi027 ceph-mon[39795]: pgmap v2352: 47 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.7 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 487 op/s; 6338/183924 objects degraded (3.446%); 36857/183924 objects misplaced (20.039%); 24 MiB/s, 381 objects/s recovering 2024-05-15T13:27:28.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:27 smithi120 ceph-mon[43961]: pgmap v2352: 47 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.7 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 487 op/s; 6338/183924 objects degraded (3.446%); 36857/183924 objects misplaced (20.039%); 24 MiB/s, 381 objects/s recovering 2024-05-15T13:27:28.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:27 smithi177 ceph-mon[44303]: pgmap v2352: 47 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.7 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 487 op/s; 6338/183924 objects degraded (3.446%); 36857/183924 objects misplaced (20.039%); 24 MiB/s, 381 objects/s recovering 2024-05-15T13:27:28.490 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:27:28.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:28.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:27:28.580 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:27:28.594 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:27:28.624 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:28.654 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:27:28.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:28.662 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:27:28.763 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:27:28.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:28 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-a[39772]: 2024-05-15T13:27:28.446+0000 7f8daffd2700 -1 mon.a@0(leader).osd e920 definitely_dead 0 2024-05-15T13:27:28.835 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:28.858 INFO:teuthology.orchestra.run.smithi027.stderr:marked in osd.8. 2024-05-15T13:27:28.883 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:27:28.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:28 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:27:28.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:28 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:27:28.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:28 smithi177 ceph-mon[44303]: osdmap e920: 12 total, 12 up, 9 in 2024-05-15T13:27:28.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:28 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/427015117' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["8"]}]: dispatch 2024-05-15T13:27:28.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:28 smithi177 ceph-mon[44303]: from='client.26818 172.21.15.27:0/3789657831' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:27:28.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:28 smithi177 ceph-mon[44303]: from='client.26818 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:27:28.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:27:28.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:27:28.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:27:28.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:28 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:27:29.044 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:28 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:27:29.044 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:28 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:27:29.044 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:28 smithi120 ceph-mon[43961]: osdmap e920: 12 total, 12 up, 9 in 2024-05-15T13:27:29.044 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:28 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/427015117' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["8"]}]: dispatch 2024-05-15T13:27:29.044 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:28 smithi120 ceph-mon[43961]: from='client.26818 172.21.15.27:0/3789657831' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:27:29.044 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:28 smithi120 ceph-mon[43961]: from='client.26818 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:27:29.044 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:27:29.044 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:27:29.044 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:27:29.045 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:28 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:27:29.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:28 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:27:29.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:28 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:27:29.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:28 smithi027 ceph-mon[39795]: osdmap e920: 12 total, 12 up, 9 in 2024-05-15T13:27:29.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:28 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/427015117' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["8"]}]: dispatch 2024-05-15T13:27:29.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:28 smithi027 ceph-mon[39795]: from='client.26818 172.21.15.27:0/3789657831' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:27:29.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:28 smithi027 ceph-mon[39795]: from='client.26818 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:27:29.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:27:29.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:27:29.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:27:29.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:28 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:27:30.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:29 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:27:30.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:29 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/427015117' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["8"]}]': finished 2024-05-15T13:27:30.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:29 smithi177 ceph-mon[44303]: from='client.26818 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:27:30.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:29 smithi177 ceph-mon[44303]: osdmap e921: 12 total, 12 up, 10 in 2024-05-15T13:27:30.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:29 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 6338/183924 objects degraded (3.446%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:30.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:29 smithi177 ceph-mon[44303]: osdmap e922: 12 total, 12 up, 10 in 2024-05-15T13:27:30.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:29 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:27:30.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:29 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/427015117' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["8"]}]': finished 2024-05-15T13:27:30.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:29 smithi027 ceph-mon[39795]: from='client.26818 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:27:30.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:29 smithi027 ceph-mon[39795]: osdmap e921: 12 total, 12 up, 10 in 2024-05-15T13:27:30.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:29 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 6338/183924 objects degraded (3.446%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:30.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:29 smithi027 ceph-mon[39795]: osdmap e922: 12 total, 12 up, 10 in 2024-05-15T13:27:30.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:29 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:27:30.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:29 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/427015117' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["8"]}]': finished 2024-05-15T13:27:30.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:29 smithi120 ceph-mon[43961]: from='client.26818 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:27:30.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:29 smithi120 ceph-mon[43961]: osdmap e921: 12 total, 12 up, 10 in 2024-05-15T13:27:30.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:29 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 6338/183924 objects degraded (3.446%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:30.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:29 smithi120 ceph-mon[43961]: osdmap e922: 12 total, 12 up, 10 in 2024-05-15T13:27:31.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:30 smithi177 ceph-mon[44303]: 9.4 starting backfill to osd.8 from (0'0,0'0] MAX to 920'11328 2024-05-15T13:27:31.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:30 smithi177 ceph-mon[44303]: 9.b starting backfill to osd.8 from (0'0,0'0] MAX to 920'14089 2024-05-15T13:27:31.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:30 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:27:31.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:30 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:27:31.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:30 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:27:31.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:30 smithi177 ceph-mon[44303]: osdmap e923: 12 total, 12 up, 10 in 2024-05-15T13:27:31.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:30 smithi027 ceph-mon[39795]: 9.4 starting backfill to osd.8 from (0'0,0'0] MAX to 920'11328 2024-05-15T13:27:31.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:30 smithi027 ceph-mon[39795]: 9.b starting backfill to osd.8 from (0'0,0'0] MAX to 920'14089 2024-05-15T13:27:31.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:30 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:27:31.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:30 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:27:31.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:30 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:27:31.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:30 smithi027 ceph-mon[39795]: osdmap e923: 12 total, 12 up, 10 in 2024-05-15T13:27:31.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:30 smithi120 ceph-mon[43961]: 9.4 starting backfill to osd.8 from (0'0,0'0] MAX to 920'11328 2024-05-15T13:27:31.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:30 smithi120 ceph-mon[43961]: 9.b starting backfill to osd.8 from (0'0,0'0] MAX to 920'14089 2024-05-15T13:27:31.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:30 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:27:31.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:30 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:27:31.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:30 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:27:31.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:30 smithi120 ceph-mon[43961]: osdmap e923: 12 total, 12 up, 10 in 2024-05-15T13:27:32.194 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:31 smithi177 ceph-mon[44303]: 9.0 starting backfill to osd.8 from (0'0,0'0] MAX to 920'11012 2024-05-15T13:27:32.194 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:31 smithi177 ceph-mon[44303]: pgmap v2358: 47 pgs: 5 remapped+peering, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 30 active+clean; 3.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 767 B/s rd, 17 KiB/s wr, 250 op/s; 6365/180936 objects degraded (3.518%); 34308/180936 objects misplaced (18.961%); 14 MiB/s, 220 objects/s recovering 2024-05-15T13:27:32.194 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:31 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:27:32.194 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:31 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:27:32.194 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:31 smithi177 ceph-mon[44303]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-05-15T13:27:32.194 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:31 smithi177 ceph-mon[44303]: osdmap e924: 12 total, 12 up, 10 in 2024-05-15T13:27:32.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:31 smithi027 ceph-mon[39795]: 9.0 starting backfill to osd.8 from (0'0,0'0] MAX to 920'11012 2024-05-15T13:27:32.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:31 smithi027 ceph-mon[39795]: pgmap v2358: 47 pgs: 5 remapped+peering, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 30 active+clean; 3.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 767 B/s rd, 17 KiB/s wr, 250 op/s; 6365/180936 objects degraded (3.518%); 34308/180936 objects misplaced (18.961%); 14 MiB/s, 220 objects/s recovering 2024-05-15T13:27:32.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:31 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:27:32.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:31 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:27:32.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:31 smithi027 ceph-mon[39795]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-05-15T13:27:32.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:31 smithi027 ceph-mon[39795]: osdmap e924: 12 total, 12 up, 10 in 2024-05-15T13:27:32.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:31 smithi120 ceph-mon[43961]: 9.0 starting backfill to osd.8 from (0'0,0'0] MAX to 920'11012 2024-05-15T13:27:32.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:31 smithi120 ceph-mon[43961]: pgmap v2358: 47 pgs: 5 remapped+peering, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 30 active+clean; 3.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 767 B/s rd, 17 KiB/s wr, 250 op/s; 6365/180936 objects degraded (3.518%); 34308/180936 objects misplaced (18.961%); 14 MiB/s, 220 objects/s recovering 2024-05-15T13:27:32.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:31 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:27:32.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:31 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:27:32.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:31 smithi120 ceph-mon[43961]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-05-15T13:27:32.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:31 smithi120 ceph-mon[43961]: osdmap e924: 12 total, 12 up, 10 in 2024-05-15T13:27:33.052 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:27:33.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:33.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:27:33.161 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:27:33.202 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:33.335 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:27:33.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:33.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:27:33.442 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:27:33.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:33 smithi027 ceph-mon[39795]: osdmap e925: 12 total, 12 up, 10 in 2024-05-15T13:27:33.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:33 smithi027 ceph-mon[39795]: pgmap v2361: 47 pgs: 1 active+remapped+backfill_toofull, 3 peering, 1 active+recovery_wait+remapped, 5 remapped+peering, 2 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 27 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 786 B/s rd, 17 KiB/s wr, 305 op/s; 6378/177759 objects degraded (3.588%); 31958/177759 objects misplaced (17.978%); 34 MiB/s, 547 objects/s recovering 2024-05-15T13:27:33.684 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:33.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:33 smithi177 ceph-mon[44303]: osdmap e925: 12 total, 12 up, 10 in 2024-05-15T13:27:33.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:33 smithi177 ceph-mon[44303]: pgmap v2361: 47 pgs: 1 active+remapped+backfill_toofull, 3 peering, 1 active+recovery_wait+remapped, 5 remapped+peering, 2 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 27 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 786 B/s rd, 17 KiB/s wr, 305 op/s; 6378/177759 objects degraded (3.588%); 31958/177759 objects misplaced (17.978%); 34 MiB/s, 547 objects/s recovering 2024-05-15T13:27:33.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:33 smithi120 ceph-mon[43961]: osdmap e925: 12 total, 12 up, 10 in 2024-05-15T13:27:33.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:33 smithi120 ceph-mon[43961]: pgmap v2361: 47 pgs: 1 active+remapped+backfill_toofull, 3 peering, 1 active+recovery_wait+remapped, 5 remapped+peering, 2 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 27 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 786 B/s rd, 17 KiB/s wr, 305 op/s; 6378/177759 objects degraded (3.588%); 31958/177759 objects misplaced (17.978%); 34 MiB/s, 547 objects/s recovering 2024-05-15T13:27:33.937 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:27:33.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:33.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:27:33.944 INFO:tasks.thrashosds.thrasher:Added osd 8 2024-05-15T13:27:34.045 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:27:34.093 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:34.134 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:27:34.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:34.143 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:27:34.244 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:27:34.370 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:34.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:34 smithi177 ceph-mon[44303]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T13:27:34.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:34 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 6378/177759 objects degraded (3.588%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:34.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:34 smithi027 ceph-mon[39795]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T13:27:34.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:34 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 6378/177759 objects degraded (3.588%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:34.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:34 smithi120 ceph-mon[43961]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-05-15T13:27:34.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:34 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 6378/177759 objects degraded (3.588%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:35.375 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:27:35.644 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:27:35.671 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:27:35.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:35 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:27:35.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:35 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:27:35.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:35 smithi177 ceph-mon[44303]: osdmap e926: 12 total, 12 up, 10 in 2024-05-15T13:27:35.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:35 smithi177 ceph-mon[44303]: pgmap v2363: 47 pgs: 1 active+remapped+backfill_toofull, 3 peering, 1 active+recovery_wait+remapped, 4 remapped+peering, 2 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 28 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 325 op/s; 6015/176616 objects degraded (3.406%); 34447/176616 objects misplaced (19.504%); 42 MiB/s, 675 objects/s recovering 2024-05-15T13:27:35.703 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:27:35.741 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:35 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:27:35.741 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:35 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:27:35.741 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:35 smithi120 ceph-mon[43961]: osdmap e926: 12 total, 12 up, 10 in 2024-05-15T13:27:35.741 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:35 smithi120 ceph-mon[43961]: pgmap v2363: 47 pgs: 1 active+remapped+backfill_toofull, 3 peering, 1 active+recovery_wait+remapped, 4 remapped+peering, 2 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 28 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 325 op/s; 6015/176616 objects degraded (3.406%); 34447/176616 objects misplaced (19.504%); 42 MiB/s, 675 objects/s recovering 2024-05-15T13:27:35.742 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:27:35.756 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:27:35.758 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:27:35.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:35.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:27:35.770 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:27:35.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:35 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:27:35.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:35 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:27:35.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:35 smithi027 ceph-mon[39795]: osdmap e926: 12 total, 12 up, 10 in 2024-05-15T13:27:35.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:35 smithi027 ceph-mon[39795]: pgmap v2363: 47 pgs: 1 active+remapped+backfill_toofull, 3 peering, 1 active+recovery_wait+remapped, 4 remapped+peering, 2 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 28 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 325 op/s; 6015/176616 objects degraded (3.406%); 34447/176616 objects misplaced (19.504%); 42 MiB/s, 675 objects/s recovering 2024-05-15T13:27:35.847 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:27:35.866 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:27:35.903 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:35.931 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:27:35.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:35.935 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:27:35.962 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:27:35.962 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:27:35.962 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:27:35.963 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:27:35.973 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:27:36.036 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:27:36.075 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:36.099 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:27:36.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:36.105 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:27:36.197 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:27:36.201 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:27:36.201 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:27:36.201 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:27:36.201 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:27:36.201 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:27:36.201 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:27:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:27:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:27:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:27:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:27:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:27:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:27:36.205 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:27:36.240 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:36.412 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:36 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2541877251' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:27:36.531 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:36 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2541877251' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:27:36.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:36 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2541877251' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:27:37.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:37 smithi177 ceph-mon[44303]: pgmap v2364: 47 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfill_toofull, 3 peering, 2 active+remapped+backfilling, 4 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 31 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 427 op/s; 6002/173232 objects degraded (3.465%); 33826/173232 objects misplaced (19.526%); 42 MiB/s, 670 objects/s recovering 2024-05-15T13:27:37.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:37 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:27:37.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:37 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:27:37.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:37 smithi027 ceph-mon[39795]: pgmap v2364: 47 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfill_toofull, 3 peering, 2 active+remapped+backfilling, 4 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 31 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 427 op/s; 6002/173232 objects degraded (3.465%); 33826/173232 objects misplaced (19.526%); 42 MiB/s, 670 objects/s recovering 2024-05-15T13:27:37.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:37 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:27:37.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:37 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:27:37.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:37 smithi120 ceph-mon[43961]: pgmap v2364: 47 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfill_toofull, 3 peering, 2 active+remapped+backfilling, 4 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 31 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 427 op/s; 6002/173232 objects degraded (3.465%); 33826/173232 objects misplaced (19.526%); 42 MiB/s, 670 objects/s recovering 2024-05-15T13:27:37.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:37 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:27:37.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:37 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:27:38.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:38 smithi177 ceph-mon[44303]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-05-15T13:27:38.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:38 smithi177 ceph-mon[44303]: osdmap e927: 12 total, 12 up, 10 in 2024-05-15T13:27:38.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:38 smithi027 ceph-mon[39795]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-05-15T13:27:38.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:38 smithi027 ceph-mon[39795]: osdmap e927: 12 total, 12 up, 10 in 2024-05-15T13:27:38.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:38 smithi120 ceph-mon[43961]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-05-15T13:27:38.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:38 smithi120 ceph-mon[43961]: osdmap e927: 12 total, 12 up, 10 in 2024-05-15T13:27:38.944 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 9, 10, 7, 4, 8] out_osds: [1, 11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:27:38.945 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:27:38.945 INFO:tasks.thrashosds.thrasher:Adding osd 11 2024-05-15T13:27:38.945 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd in 11 2024-05-15T13:27:39.353 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:27:39.353 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:27:39.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:39 smithi177 ceph-mon[44303]: pgmap v2366: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovery_wait+remapped, 2 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 420 op/s; 6014/169671 objects degraded (3.545%); 32485/169671 objects misplaced (19.146%); 31 MiB/s, 493 objects/s recovering 2024-05-15T13:27:39.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:39 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 6002/173232 objects degraded (3.465%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:39.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:39 smithi027 ceph-mon[39795]: pgmap v2366: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovery_wait+remapped, 2 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 420 op/s; 6014/169671 objects degraded (3.545%); 32485/169671 objects misplaced (19.146%); 31 MiB/s, 493 objects/s recovering 2024-05-15T13:27:39.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:39 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 6002/173232 objects degraded (3.465%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:39.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:39 smithi120 ceph-mon[43961]: pgmap v2366: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovery_wait+remapped, 2 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 420 op/s; 6014/169671 objects degraded (3.545%); 32485/169671 objects misplaced (19.146%); 31 MiB/s, 493 objects/s recovering 2024-05-15T13:27:39.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:39 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 6002/173232 objects degraded (3.465%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:40.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:40 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:27:40.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:40 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:27:40.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:40 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:27:40.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:40 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:27:40.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:40 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:27:40.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:40 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:27:41.531 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:41 smithi027 ceph-mon[39795]: osdmap e928: 12 total, 12 up, 10 in 2024-05-15T13:27:41.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:41 smithi027 ceph-mon[39795]: pgmap v2368: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovery_wait+remapped, 2 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 386 op/s; 6014/169233 objects degraded (3.554%); 31859/169233 objects misplaced (18.826%); 23 MiB/s, 380 objects/s recovering 2024-05-15T13:27:41.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:41 smithi120 ceph-mon[43961]: osdmap e928: 12 total, 12 up, 10 in 2024-05-15T13:27:41.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:41 smithi120 ceph-mon[43961]: pgmap v2368: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovery_wait+remapped, 2 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 386 op/s; 6014/169233 objects degraded (3.554%); 31859/169233 objects misplaced (18.826%); 23 MiB/s, 380 objects/s recovering 2024-05-15T13:27:41.945 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:41 smithi177 ceph-mon[44303]: osdmap e928: 12 total, 12 up, 10 in 2024-05-15T13:27:41.945 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:41 smithi177 ceph-mon[44303]: pgmap v2368: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovery_wait+remapped, 2 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 386 op/s; 6014/169233 objects degraded (3.554%); 31859/169233 objects misplaced (18.826%); 23 MiB/s, 380 objects/s recovering 2024-05-15T13:27:43.206 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-05-15T13:27:43.273 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:27:43.723 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:27:44.117 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:27:44.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:43 smithi177 ceph-mon[44303]: pgmap v2369: 47 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 515 op/s; 5870/164241 objects degraded (3.574%); 30167/164241 objects misplaced (18.368%); 28 MiB/s, 451 objects/s recovering 2024-05-15T13:27:44.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:43 smithi027 ceph-mon[39795]: pgmap v2369: 47 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 515 op/s; 5870/164241 objects degraded (3.574%); 30167/164241 objects misplaced (18.368%); 28 MiB/s, 451 objects/s recovering 2024-05-15T13:27:44.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:44.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:27:44.297 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:43 smithi120 ceph-mon[43961]: pgmap v2369: 47 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 515 op/s; 5870/164241 objects degraded (3.574%); 30167/164241 objects misplaced (18.368%); 28 MiB/s, 451 objects/s recovering 2024-05-15T13:27:44.392 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:27:44.487 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:44.959 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:27:44.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:44.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:27:45.072 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:27:45.118 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:44 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 5870/164241 objects degraded (3.574%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:45.119 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:44 smithi120 ceph-mon[43961]: osdmap e929: 12 total, 12 up, 10 in 2024-05-15T13:27:45.121 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:45.194 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:44 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 5870/164241 objects degraded (3.574%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:45.194 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:44 smithi177 ceph-mon[44303]: osdmap e929: 12 total, 12 up, 10 in 2024-05-15T13:27:45.264 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:44 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 5870/164241 objects degraded (3.574%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:45.264 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:44 smithi027 ceph-mon[39795]: osdmap e929: 12 total, 12 up, 10 in 2024-05-15T13:27:45.307 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:27:45.927 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:27:46.183 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:45 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:27:46.183 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:45 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:27:46.183 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:45 smithi120 ceph-mon[43961]: pgmap v2371: 47 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 409 op/s; 5887/162483 objects degraded (3.623%); 29217/162483 objects misplaced (17.982%); 25 MiB/s, 400 objects/s recovering 2024-05-15T13:27:46.183 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:45 smithi120 ceph-mon[43961]: osdmap e930: 12 total, 12 up, 10 in 2024-05-15T13:27:46.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:45 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:27:46.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:45 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:27:46.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:45 smithi177 ceph-mon[44303]: pgmap v2371: 47 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 409 op/s; 5887/162483 objects degraded (3.623%); 29217/162483 objects misplaced (17.982%); 25 MiB/s, 400 objects/s recovering 2024-05-15T13:27:46.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:45 smithi177 ceph-mon[44303]: osdmap e930: 12 total, 12 up, 10 in 2024-05-15T13:27:46.203 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:27:46.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:46.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:27:46.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:45 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:27:46.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:45 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:27:46.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:45 smithi027 ceph-mon[39795]: pgmap v2371: 47 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 409 op/s; 5887/162483 objects degraded (3.623%); 29217/162483 objects misplaced (17.982%); 25 MiB/s, 400 objects/s recovering 2024-05-15T13:27:46.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:45 smithi027 ceph-mon[39795]: osdmap e930: 12 total, 12 up, 10 in 2024-05-15T13:27:46.316 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:27:46.348 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:46.702 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:27:46.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:46.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:27:46.809 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:27:46.847 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:47.515 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:27:47.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:47.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:27:47.620 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:27:47.665 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:47.796 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:27:47.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:47.811 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:27:47.912 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:27:47.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:47 smithi177 ceph-mon[44303]: pgmap v2373: 47 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 557 op/s; 5717/158643 objects degraded (3.604%); 28056/158643 objects misplaced (17.685%); 27 MiB/s, 496 objects/s recovering 2024-05-15T13:27:47.961 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:47.992 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:27:47.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:47.996 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:27:48.097 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:27:48.127 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:47 smithi027 ceph-mon[39795]: pgmap v2373: 47 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 557 op/s; 5717/158643 objects degraded (3.604%); 28056/158643 objects misplaced (17.685%); 27 MiB/s, 496 objects/s recovering 2024-05-15T13:27:48.137 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:48.138 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:47 smithi120 ceph-mon[43961]: pgmap v2373: 47 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 557 op/s; 5717/158643 objects degraded (3.604%); 28056/158643 objects misplaced (17.685%); 27 MiB/s, 496 objects/s recovering 2024-05-15T13:27:48.867 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:27:49.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:48 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:27:49.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:48 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:27:49.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:48 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2720564940' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:27:49.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:48 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:27:49.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:48 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:27:49.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:48 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2720564940' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:27:49.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:48 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:27:49.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:48 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:27:49.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:48 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2720564940' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:27:50.175 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:49 smithi027 ceph-mon[39795]: pgmap v2374: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovery_wait+remapped, 2 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 597 op/s; 5779/154884 objects degraded (3.731%); 26640/154884 objects misplaced (17.200%); 30 MiB/s, 523 objects/s recovering 2024-05-15T13:27:50.175 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:49 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:27:50.176 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:49 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2720564940' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:27:50.176 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:49 smithi027 ceph-mon[39795]: osdmap e931: 12 total, 12 up, 10 in 2024-05-15T13:27:50.176 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:49 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 5779/154884 objects degraded (3.731%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:50.176 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:49 smithi027 ceph-mon[39795]: osdmap e932: 12 total, 12 up, 10 in 2024-05-15T13:27:50.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:49 smithi177 ceph-mon[44303]: pgmap v2374: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovery_wait+remapped, 2 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 597 op/s; 5779/154884 objects degraded (3.731%); 26640/154884 objects misplaced (17.200%); 30 MiB/s, 523 objects/s recovering 2024-05-15T13:27:50.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:49 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:27:50.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:49 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2720564940' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:27:50.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:49 smithi177 ceph-mon[44303]: osdmap e931: 12 total, 12 up, 10 in 2024-05-15T13:27:50.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:49 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 5779/154884 objects degraded (3.731%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:50.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:49 smithi177 ceph-mon[44303]: osdmap e932: 12 total, 12 up, 10 in 2024-05-15T13:27:50.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:49 smithi120 ceph-mon[43961]: pgmap v2374: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovery_wait+remapped, 2 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 597 op/s; 5779/154884 objects degraded (3.731%); 26640/154884 objects misplaced (17.200%); 30 MiB/s, 523 objects/s recovering 2024-05-15T13:27:50.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:49 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:27:50.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:49 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2720564940' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:27:50.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:49 smithi120 ceph-mon[43961]: osdmap e931: 12 total, 12 up, 10 in 2024-05-15T13:27:50.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:49 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 5779/154884 objects degraded (3.731%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:50.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:49 smithi120 ceph-mon[43961]: osdmap e932: 12 total, 12 up, 10 in 2024-05-15T13:27:50.474 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:27:50.474 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:27:50.864 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:50 smithi027 ceph-mon[39795]: 1.6 scrub starts 2024-05-15T13:27:50.865 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:50 smithi027 ceph-mon[39795]: 1.6 scrub ok 2024-05-15T13:27:50.865 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:50 smithi027 ceph-mon[39795]: osdmap e933: 12 total, 12 up, 10 in 2024-05-15T13:27:50.922 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:27:50.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:50.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:27:51.028 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:27:51.062 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:51.085 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:27:51.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:51.092 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:27:51.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:50 smithi177 ceph-mon[44303]: 1.6 scrub starts 2024-05-15T13:27:51.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:50 smithi177 ceph-mon[44303]: 1.6 scrub ok 2024-05-15T13:27:51.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:50 smithi177 ceph-mon[44303]: osdmap e933: 12 total, 12 up, 10 in 2024-05-15T13:27:51.193 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:27:51.240 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:51.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:50 smithi120 ceph-mon[43961]: 1.6 scrub starts 2024-05-15T13:27:51.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:50 smithi120 ceph-mon[43961]: 1.6 scrub ok 2024-05-15T13:27:51.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:50 smithi120 ceph-mon[43961]: osdmap e933: 12 total, 12 up, 10 in 2024-05-15T13:27:51.592 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:27:51.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:51.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:27:51.698 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:27:51.737 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:52.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:51 smithi177 ceph-mon[44303]: pgmap v2378: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovery_wait+remapped, 2 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 467 op/s; 5779/154422 objects degraded (3.742%); 26640/154422 objects misplaced (17.251%); 21 MiB/s, 424 objects/s recovering 2024-05-15T13:27:52.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:51 smithi177 ceph-mon[44303]: osdmap e934: 12 total, 12 up, 10 in 2024-05-15T13:27:52.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:51 smithi027 ceph-mon[39795]: pgmap v2378: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovery_wait+remapped, 2 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 467 op/s; 5779/154422 objects degraded (3.742%); 26640/154422 objects misplaced (17.251%); 21 MiB/s, 424 objects/s recovering 2024-05-15T13:27:52.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:51 smithi027 ceph-mon[39795]: osdmap e934: 12 total, 12 up, 10 in 2024-05-15T13:27:52.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:51 smithi120 ceph-mon[43961]: pgmap v2378: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovery_wait+remapped, 2 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 34 active+clean; 3.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 467 op/s; 5779/154422 objects degraded (3.742%); 26640/154422 objects misplaced (17.251%); 21 MiB/s, 424 objects/s recovering 2024-05-15T13:27:52.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:51 smithi120 ceph-mon[43961]: osdmap e934: 12 total, 12 up, 10 in 2024-05-15T13:27:52.646 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:27:53.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:53 smithi177 ceph-mon[44303]: osdmap e935: 12 total, 12 up, 10 in 2024-05-15T13:27:53.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:53 smithi177 ceph-mon[44303]: 9.0 continuing backfill to osd.8 from (824'1600,933'11628] 9:006d0943:::benchmark_data_smithi177_159919_object108274:head to 933'11628 2024-05-15T13:27:53.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:53 smithi177 ceph-mon[44303]: pgmap v2381: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 35 active+clean; 3.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 382 op/s; 5655/150399 objects degraded (3.760%); 25151/150399 objects misplaced (16.723%); 18 MiB/s, 492 objects/s recovering 2024-05-15T13:27:53.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:53 smithi027 ceph-mon[39795]: osdmap e935: 12 total, 12 up, 10 in 2024-05-15T13:27:53.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:53 smithi027 ceph-mon[39795]: 9.0 continuing backfill to osd.8 from (824'1600,933'11628] 9:006d0943:::benchmark_data_smithi177_159919_object108274:head to 933'11628 2024-05-15T13:27:53.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:53 smithi027 ceph-mon[39795]: pgmap v2381: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 35 active+clean; 3.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 382 op/s; 5655/150399 objects degraded (3.760%); 25151/150399 objects misplaced (16.723%); 18 MiB/s, 492 objects/s recovering 2024-05-15T13:27:53.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:53 smithi120 ceph-mon[43961]: osdmap e935: 12 total, 12 up, 10 in 2024-05-15T13:27:53.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:53 smithi120 ceph-mon[43961]: 9.0 continuing backfill to osd.8 from (824'1600,933'11628] 9:006d0943:::benchmark_data_smithi177_159919_object108274:head to 933'11628 2024-05-15T13:27:53.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:53 smithi120 ceph-mon[43961]: pgmap v2381: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 35 active+clean; 3.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 382 op/s; 5655/150399 objects degraded (3.760%); 25151/150399 objects misplaced (16.723%); 18 MiB/s, 492 objects/s recovering 2024-05-15T13:27:53.968 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:27:53.989 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:27:53.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:53.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:27:54.095 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:27:54.177 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:54.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:54 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 5655/150399 objects degraded (3.760%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:54.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:54 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 5655/150399 objects degraded (3.760%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:54.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:54 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 5655/150399 objects degraded (3.760%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:55.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:55 smithi177 ceph-mon[44303]: pgmap v2382: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 36 active+clean; 3.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 348 op/s; 5661/149217 objects degraded (3.794%); 24336/149217 objects misplaced (16.309%); 20 MiB/s, 467 objects/s recovering 2024-05-15T13:27:55.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:55 smithi027 ceph-mon[39795]: pgmap v2382: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 36 active+clean; 3.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 348 op/s; 5661/149217 objects degraded (3.794%); 24336/149217 objects misplaced (16.309%); 20 MiB/s, 467 objects/s recovering 2024-05-15T13:27:55.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:55 smithi120 ceph-mon[43961]: pgmap v2382: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 36 active+clean; 3.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 348 op/s; 5661/149217 objects degraded (3.794%); 24336/149217 objects misplaced (16.309%); 20 MiB/s, 467 objects/s recovering 2024-05-15T13:27:56.369 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:27:56.394 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:27:56.423 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:27:56.462 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:27:56.497 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:27:56.531 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:56 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3927274820' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:27:56.544 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:27:56.574 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:27:56.616 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:27:56.641 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:27:56.677 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:27:56.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:56 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3927274820' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:27:56.702 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:27:56.754 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:27:56.754 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:27:56.754 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:27:56.754 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:27:56.754 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:27:56.754 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:27:56.754 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:27:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:27:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:27:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:27:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:27:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:27:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:27:56.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:56 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3927274820' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:27:57.031 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:56 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-a[39772]: 2024-05-15T13:27:56.663+0000 7f8daffd2700 -1 mon.a@0(leader).osd e935 definitely_dead 0 2024-05-15T13:27:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:marked in osd.11. 2024-05-15T13:27:57.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:57 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/528321715' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-05-15T13:27:57.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:57 smithi177 ceph-mon[44303]: pgmap v2383: 47 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 36 active+clean; 3.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 437 op/s; 5523/146028 objects degraded (3.782%); 26069/146028 objects misplaced (17.852%); 25 MiB/s, 502 objects/s recovering 2024-05-15T13:27:57.710 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:57 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/528321715' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-05-15T13:27:57.711 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:57 smithi120 ceph-mon[43961]: pgmap v2383: 47 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 36 active+clean; 3.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 437 op/s; 5523/146028 objects degraded (3.782%); 26069/146028 objects misplaced (17.852%); 25 MiB/s, 502 objects/s recovering 2024-05-15T13:27:57.752 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-05-15T13:27:57.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:57 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/528321715' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-05-15T13:27:57.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:57 smithi027 ceph-mon[39795]: pgmap v2383: 47 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 36 active+clean; 3.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 437 op/s; 5523/146028 objects degraded (3.782%); 26069/146028 objects misplaced (17.852%); 25 MiB/s, 502 objects/s recovering 2024-05-15T13:27:57.789 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:27:57.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:57.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:27:57.894 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:27:57.922 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:57.945 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:27:57.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:57.948 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:27:58.049 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:27:58.085 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:58.489 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:27:58.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:58 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/528321715' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-05-15T13:27:58.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:58 smithi177 ceph-mon[44303]: osdmap e936: 12 total, 12 up, 11 in 2024-05-15T13:27:58.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:58 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/528321715' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-05-15T13:27:58.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:58 smithi027 ceph-mon[39795]: osdmap e936: 12 total, 12 up, 11 in 2024-05-15T13:27:58.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:58 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/528321715' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-05-15T13:27:58.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:58 smithi120 ceph-mon[43961]: osdmap e936: 12 total, 12 up, 11 in 2024-05-15T13:27:58.844 INFO:tasks.thrashosds.thrasher:Added osd 11 2024-05-15T13:27:58.867 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:27:58.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:27:58.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:27:58.973 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:27:59.005 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:27:59.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:59 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:27:59.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:59 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:27:59.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:59 smithi177 ceph-mon[44303]: osdmap e937: 12 total, 12 up, 11 in 2024-05-15T13:27:59.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:59 smithi177 ceph-mon[44303]: pgmap v2386: 47 pgs: 1 remapped+peering, 5 unknown, 1 active+remapped+backfill_toofull, 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 30 active+clean; 2.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 419 op/s; 5523/142347 objects degraded (3.880%); 25253/142347 objects misplaced (17.740%); 19 MiB/s, 416 objects/s recovering 2024-05-15T13:27:59.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:59 smithi177 ceph-mon[44303]: 1.3 scrub starts 2024-05-15T13:27:59.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:59 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 5523/146028 objects degraded (3.782%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:59.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:27:59 smithi177 ceph-mon[44303]: osdmap e938: 12 total, 12 up, 11 in 2024-05-15T13:27:59.739 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:59 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:27:59.739 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:59 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:27:59.739 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:59 smithi027 ceph-mon[39795]: osdmap e937: 12 total, 12 up, 11 in 2024-05-15T13:27:59.739 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:59 smithi027 ceph-mon[39795]: pgmap v2386: 47 pgs: 1 remapped+peering, 5 unknown, 1 active+remapped+backfill_toofull, 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 30 active+clean; 2.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 419 op/s; 5523/142347 objects degraded (3.880%); 25253/142347 objects misplaced (17.740%); 19 MiB/s, 416 objects/s recovering 2024-05-15T13:27:59.739 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:59 smithi027 ceph-mon[39795]: 1.3 scrub starts 2024-05-15T13:27:59.739 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:59 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 5523/146028 objects degraded (3.782%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:59.739 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:27:59 smithi027 ceph-mon[39795]: osdmap e938: 12 total, 12 up, 11 in 2024-05-15T13:27:59.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:59 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:27:59.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:59 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:27:59.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:59 smithi120 ceph-mon[43961]: osdmap e937: 12 total, 12 up, 11 in 2024-05-15T13:27:59.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:59 smithi120 ceph-mon[43961]: pgmap v2386: 47 pgs: 1 remapped+peering, 5 unknown, 1 active+remapped+backfill_toofull, 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 30 active+clean; 2.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 419 op/s; 5523/142347 objects degraded (3.880%); 25253/142347 objects misplaced (17.740%); 19 MiB/s, 416 objects/s recovering 2024-05-15T13:27:59.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:59 smithi120 ceph-mon[43961]: 1.3 scrub starts 2024-05-15T13:27:59.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:59 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 5523/146028 objects degraded (3.782%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-05-15T13:27:59.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:27:59 smithi120 ceph-mon[43961]: osdmap e938: 12 total, 12 up, 11 in 2024-05-15T13:28:00.361 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:28:00.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:00 smithi177 ceph-mon[44303]: 1.3 scrub ok 2024-05-15T13:28:00.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:00 smithi177 ceph-mon[44303]: 9.8 starting backfill to osd.11 from (825'1700,931'11771] MIN to 935'11974 2024-05-15T13:28:00.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:00 smithi177 ceph-mon[44303]: 9.2 starting backfill to osd.11 from (830'4300,932'14368] MIN to 935'14572 2024-05-15T13:28:00.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:00 smithi177 ceph-mon[44303]: 9.0 continuing backfill to osd.8 from (825'1700,935'11783] 9:03f9c62d:::benchmark_data_smithi177_159919_object97411:head to 935'11783 2024-05-15T13:28:00.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:00 smithi177 ceph-mon[44303]: osdmap e939: 12 total, 12 up, 11 in 2024-05-15T13:28:00.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:00 smithi027 ceph-mon[39795]: 1.3 scrub ok 2024-05-15T13:28:00.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:00 smithi027 ceph-mon[39795]: 9.8 starting backfill to osd.11 from (825'1700,931'11771] MIN to 935'11974 2024-05-15T13:28:00.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:00 smithi027 ceph-mon[39795]: 9.2 starting backfill to osd.11 from (830'4300,932'14368] MIN to 935'14572 2024-05-15T13:28:00.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:00 smithi027 ceph-mon[39795]: 9.0 continuing backfill to osd.8 from (825'1700,935'11783] 9:03f9c62d:::benchmark_data_smithi177_159919_object97411:head to 935'11783 2024-05-15T13:28:00.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:00 smithi027 ceph-mon[39795]: osdmap e939: 12 total, 12 up, 11 in 2024-05-15T13:28:00.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:00 smithi120 ceph-mon[43961]: 1.3 scrub ok 2024-05-15T13:28:00.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:00 smithi120 ceph-mon[43961]: 9.8 starting backfill to osd.11 from (825'1700,931'11771] MIN to 935'11974 2024-05-15T13:28:00.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:00 smithi120 ceph-mon[43961]: 9.2 starting backfill to osd.11 from (830'4300,932'14368] MIN to 935'14572 2024-05-15T13:28:00.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:00 smithi120 ceph-mon[43961]: 9.0 continuing backfill to osd.8 from (825'1700,935'11783] 9:03f9c62d:::benchmark_data_smithi177_159919_object97411:head to 935'11783 2024-05-15T13:28:00.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:00 smithi120 ceph-mon[43961]: osdmap e939: 12 total, 12 up, 11 in 2024-05-15T13:28:00.896 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:28:01.166 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:28:01.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:01.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:28:01.275 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:28:01.307 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:01.331 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:28:01.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:01.335 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:28:01.436 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:28:01.487 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:01.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:01 smithi177 ceph-mon[44303]: pgmap v2389: 47 pgs: 1 peering, 1 active+remapped+backfilling, 1 remapped+peering, 2 unknown, 1 active+remapped+backfill_toofull, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 3 active+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+remapped, 1 active+recovery_wait+degraded+remapped, 28 active+clean; 2.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 326 op/s; 5523/142110 objects degraded (3.886%); 24776/142110 objects misplaced (17.434%); 14 MiB/s, 330 objects/s recovering 2024-05-15T13:28:01.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:01 smithi177 ceph-mon[44303]: osdmap e940: 12 total, 12 up, 11 in 2024-05-15T13:28:01.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:01 smithi027 ceph-mon[39795]: pgmap v2389: 47 pgs: 1 peering, 1 active+remapped+backfilling, 1 remapped+peering, 2 unknown, 1 active+remapped+backfill_toofull, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 3 active+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+remapped, 1 active+recovery_wait+degraded+remapped, 28 active+clean; 2.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 326 op/s; 5523/142110 objects degraded (3.886%); 24776/142110 objects misplaced (17.434%); 14 MiB/s, 330 objects/s recovering 2024-05-15T13:28:01.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:01 smithi027 ceph-mon[39795]: osdmap e940: 12 total, 12 up, 11 in 2024-05-15T13:28:01.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:01 smithi120 ceph-mon[43961]: pgmap v2389: 47 pgs: 1 peering, 1 active+remapped+backfilling, 1 remapped+peering, 2 unknown, 1 active+remapped+backfill_toofull, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 3 active+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 1 active+remapped, 1 active+recovery_wait+degraded+remapped, 28 active+clean; 2.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 326 op/s; 5523/142110 objects degraded (3.886%); 24776/142110 objects misplaced (17.434%); 14 MiB/s, 330 objects/s recovering 2024-05-15T13:28:01.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:01 smithi120 ceph-mon[43961]: osdmap e940: 12 total, 12 up, 11 in 2024-05-15T13:28:02.500 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:28:02.738 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:02 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1663729305' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:28:02.787 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:28:02.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:02.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:28:02.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:02 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1663729305' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:28:02.893 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:28:02.929 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:02.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:02 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1663729305' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:28:03.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:03 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:28:03.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:03 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1663729305' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:28:03.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:03 smithi027 ceph-mon[39795]: osdmap e941: 12 total, 12 up, 11 in 2024-05-15T13:28:03.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:03 smithi027 ceph-mon[39795]: pgmap v2392: 47 pgs: 1 peering, 2 active+remapped+backfilling, 1 remapped+peering, 2 unknown, 2 active+recovering+undersized+remapped, 1 active+recovering, 3 active+undersized+remapped, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped, 29 active+clean; 2.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 222 op/s; 2522/139677 objects degraded (1.806%); 23784/139677 objects misplaced (17.028%); 33 MiB/s, 646 objects/s recovering 2024-05-15T13:28:03.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:03 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:28:03.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:03 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:28:03.783 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:28:03.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:03.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:28:03.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:03 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:28:03.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:03 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1663729305' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:28:03.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:03 smithi120 ceph-mon[43961]: osdmap e941: 12 total, 12 up, 11 in 2024-05-15T13:28:03.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:03 smithi120 ceph-mon[43961]: pgmap v2392: 47 pgs: 1 peering, 2 active+remapped+backfilling, 1 remapped+peering, 2 unknown, 2 active+recovering+undersized+remapped, 1 active+recovering, 3 active+undersized+remapped, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped, 29 active+clean; 2.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 222 op/s; 2522/139677 objects degraded (1.806%); 23784/139677 objects misplaced (17.028%); 33 MiB/s, 646 objects/s recovering 2024-05-15T13:28:03.798 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:03 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:28:03.798 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:03 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:28:03.845 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 9, 10, 7, 4, 8, 11] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T13:28:03.845 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 0 1 2024-05-15T13:28:03.888 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:28:03.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:03 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:28:03.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:03 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1663729305' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:28:03.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:03 smithi177 ceph-mon[44303]: osdmap e941: 12 total, 12 up, 11 in 2024-05-15T13:28:03.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:03 smithi177 ceph-mon[44303]: pgmap v2392: 47 pgs: 1 peering, 2 active+remapped+backfilling, 1 remapped+peering, 2 unknown, 2 active+recovering+undersized+remapped, 1 active+recovering, 3 active+undersized+remapped, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped, 29 active+clean; 2.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 222 op/s; 2522/139677 objects degraded (1.806%); 23784/139677 objects misplaced (17.028%); 33 MiB/s, 646 objects/s recovering 2024-05-15T13:28:03.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:03 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:28:03.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:03 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:28:03.985 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:04.785 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:04 smithi027 ceph-mon[39795]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-05-15T13:28:04.786 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:04 smithi027 ceph-mon[39795]: osdmap e942: 12 total, 12 up, 11 in 2024-05-15T13:28:04.786 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:04 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 2522/139677 objects degraded (1.806%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:04.786 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:04 smithi027 ceph-mon[39795]: osdmap e943: 12 total, 12 up, 11 in 2024-05-15T13:28:04.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:04 smithi120 ceph-mon[43961]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-05-15T13:28:04.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:04 smithi120 ceph-mon[43961]: osdmap e942: 12 total, 12 up, 11 in 2024-05-15T13:28:04.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:04 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 2522/139677 objects degraded (1.806%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:04.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:04 smithi120 ceph-mon[43961]: osdmap e943: 12 total, 12 up, 11 in 2024-05-15T13:28:04.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:04 smithi177 ceph-mon[44303]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-05-15T13:28:04.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:04 smithi177 ceph-mon[44303]: osdmap e942: 12 total, 12 up, 11 in 2024-05-15T13:28:04.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:04 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 2522/139677 objects degraded (1.806%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:04.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:04 smithi177 ceph-mon[44303]: osdmap e943: 12 total, 12 up, 11 in 2024-05-15T13:28:05.391 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:28:05.511 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:28:05.521 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:05 smithi027 ceph-mon[39795]: pgmap v2395: 47 pgs: 1 peering, 2 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 311 op/s; 2522/138366 objects degraded (1.823%); 20162/138366 objects misplaced (14.571%); 36 MiB/s, 689 objects/s recovering 2024-05-15T13:28:05.539 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:28:05.567 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:28:05.599 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:28:05.636 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:28:05.680 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:28:05.713 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:28:05.743 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:28:05.773 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:28:05.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:05 smithi120 ceph-mon[43961]: pgmap v2395: 47 pgs: 1 peering, 2 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 311 op/s; 2522/138366 objects degraded (1.823%); 20162/138366 objects misplaced (14.571%); 36 MiB/s, 689 objects/s recovering 2024-05-15T13:28:05.805 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:28:05.842 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:28:05.861 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:28:05.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:05.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:28:05.879 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:28:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:28:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:28:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:28:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:28:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:28:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:28:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:28:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:28:05.882 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:28:05.882 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:28:05.882 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:28:05.882 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:28:05.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:05 smithi177 ceph-mon[44303]: pgmap v2395: 47 pgs: 1 peering, 2 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 311 op/s; 2522/138366 objects degraded (1.823%); 20162/138366 objects misplaced (14.571%); 36 MiB/s, 689 objects/s recovering 2024-05-15T13:28:05.972 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:28:06.014 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:06.218 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:28:06.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:06.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:28:06.325 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:28:06.387 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:06.431 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:28:06.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:06.435 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:28:06.537 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:28:06.567 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:06.600 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:28:06.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:06.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:28:06.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:06 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2321873957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:28:06.713 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:28:06.779 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:06.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:06 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2321873957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:28:06.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:06 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2321873957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:28:06.815 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:28:06.817 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:28:06.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:06.821 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:28:06.922 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:28:06.963 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:07.004 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:28:07.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:07.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:28:07.109 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:28:07.137 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:07.186 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:28:07.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:07.199 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:28:07.300 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:28:07.358 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:07.395 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:28:07.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:07.402 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:28:07.503 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:28:07.559 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:07.587 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:28:07.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:07.592 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:28:07.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:07 smithi177 ceph-mon[44303]: pgmap v2396: 47 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 39 active+clean; 2.8 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 306 op/s; 2522/135540 objects degraded (1.861%); 17042/135540 objects misplaced (12.573%); 30 MiB/s, 616 objects/s recovering 2024-05-15T13:28:07.693 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:28:07.721 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:07.767 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:28:07.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:07.773 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:28:07.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:07 smithi027 ceph-mon[39795]: pgmap v2396: 47 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 39 active+clean; 2.8 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 306 op/s; 2522/135540 objects degraded (1.861%); 17042/135540 objects misplaced (12.573%); 30 MiB/s, 616 objects/s recovering 2024-05-15T13:28:07.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:07 smithi120 ceph-mon[43961]: pgmap v2396: 47 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 39 active+clean; 2.8 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 306 op/s; 2522/135540 objects degraded (1.861%); 17042/135540 objects misplaced (12.573%); 30 MiB/s, 616 objects/s recovering 2024-05-15T13:28:07.874 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:28:07.954 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:10.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:09 smithi177 ceph-mon[44303]: pgmap v2397: 47 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 39 active+clean; 2.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 451 op/s; 2522/131118 objects degraded (1.923%); 16387/131118 objects misplaced (12.498%); 28 MiB/s, 484 objects/s recovering 2024-05-15T13:28:10.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:09 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 2522/131118 objects degraded (1.923%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:10.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:09 smithi027 ceph-mon[39795]: pgmap v2397: 47 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 39 active+clean; 2.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 451 op/s; 2522/131118 objects degraded (1.923%); 16387/131118 objects misplaced (12.498%); 28 MiB/s, 484 objects/s recovering 2024-05-15T13:28:10.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:09 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 2522/131118 objects degraded (1.923%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:10.295 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:09 smithi120 ceph-mon[43961]: pgmap v2397: 47 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 39 active+clean; 2.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 451 op/s; 2522/131118 objects degraded (1.923%); 16387/131118 objects misplaced (12.498%); 28 MiB/s, 484 objects/s recovering 2024-05-15T13:28:10.295 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:09 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 2522/131118 objects degraded (1.923%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:10.764 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:28:10.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:10.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:28:10.869 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:28:10.908 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:10.952 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:28:10.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:10.963 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:28:11.064 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:28:11.105 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:11.135 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:28:11.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:11.140 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:28:11.240 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:28:11.279 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:11.300 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:28:11.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:11.303 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:28:11.404 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:28:11.481 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:12.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:11 smithi177 ceph-mon[44303]: pgmap v2398: 47 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 39 active+clean; 2.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 378 op/s; 2522/130602 objects degraded (1.931%); 15718/130602 objects misplaced (12.035%); 27 MiB/s, 462 objects/s recovering 2024-05-15T13:28:12.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:11 smithi027 ceph-mon[39795]: pgmap v2398: 47 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 39 active+clean; 2.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 378 op/s; 2522/130602 objects degraded (1.931%); 15718/130602 objects misplaced (12.035%); 27 MiB/s, 462 objects/s recovering 2024-05-15T13:28:12.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:11 smithi120 ceph-mon[43961]: pgmap v2398: 47 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 39 active+clean; 2.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 378 op/s; 2522/130602 objects degraded (1.931%); 15718/130602 objects misplaced (12.035%); 27 MiB/s, 462 objects/s recovering 2024-05-15T13:28:12.404 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:28:13.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:12 smithi027 ceph-mon[39795]: osdmap e944: 12 total, 12 up, 11 in 2024-05-15T13:28:13.051 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:28:13.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:13.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:28:13.154 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:12 smithi120 ceph-mon[43961]: osdmap e944: 12 total, 12 up, 11 in 2024-05-15T13:28:13.158 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:28:13.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:12 smithi177 ceph-mon[44303]: osdmap e944: 12 total, 12 up, 11 in 2024-05-15T13:28:13.216 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:13.258 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:28:13.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:13.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:28:13.367 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:28:13.412 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:13.470 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:28:13.733 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:28:13.733 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:28:14.044 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:13 smithi120 ceph-mon[43961]: pgmap v2400: 47 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 39 active+clean; 2.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 522 op/s; 2522/125154 objects degraded (2.015%); 15040/125154 objects misplaced (12.017%); 25 MiB/s, 480 objects/s recovering 2024-05-15T13:28:14.044 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:13 smithi120 ceph-mon[43961]: osdmap e945: 12 total, 12 up, 11 in 2024-05-15T13:28:14.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:13 smithi177 ceph-mon[44303]: pgmap v2400: 47 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 39 active+clean; 2.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 522 op/s; 2522/125154 objects degraded (2.015%); 15040/125154 objects misplaced (12.017%); 25 MiB/s, 480 objects/s recovering 2024-05-15T13:28:14.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:13 smithi177 ceph-mon[44303]: osdmap e945: 12 total, 12 up, 11 in 2024-05-15T13:28:14.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:13 smithi027 ceph-mon[39795]: pgmap v2400: 47 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 39 active+clean; 2.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 522 op/s; 2522/125154 objects degraded (2.015%); 15040/125154 objects misplaced (12.017%); 25 MiB/s, 480 objects/s recovering 2024-05-15T13:28:14.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:13 smithi027 ceph-mon[39795]: osdmap e945: 12 total, 12 up, 11 in 2024-05-15T13:28:14.549 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-05-15T13:28:14.564 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:28:14.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:14.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:28:14.669 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:28:14.760 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:14.810 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:28:14.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:14.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:28:14.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:14 smithi027 ceph-mon[39795]: 9.0 scrub starts 2024-05-15T13:28:14.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:14 smithi027 ceph-mon[39795]: 9.0 scrub ok 2024-05-15T13:28:14.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:14 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 2522/125154 objects degraded (2.015%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:14.870 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:14 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1436195965' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:28:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.0 to 1 (10000) 2024-05-15T13:28:14.916 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:28:14.949 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:15.092 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:28:15.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:14 smithi177 ceph-mon[44303]: 9.0 scrub starts 2024-05-15T13:28:15.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:14 smithi177 ceph-mon[44303]: 9.0 scrub ok 2024-05-15T13:28:15.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:14 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 2522/125154 objects degraded (2.015%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:15.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:14 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1436195965' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:28:15.295 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:14 smithi120 ceph-mon[43961]: 9.0 scrub starts 2024-05-15T13:28:15.295 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:14 smithi120 ceph-mon[43961]: 9.0 scrub ok 2024-05-15T13:28:15.296 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:14 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 2522/125154 objects degraded (2.015%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:15.296 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:14 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1436195965' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:28:15.610 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:28:15.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:15.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:28:15.718 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:28:15.762 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:15.842 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:28:15.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:15.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:28:15.947 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:28:16.019 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:15 smithi120 ceph-mon[43961]: pgmap v2402: 47 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 39 active+clean; 2.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 530 op/s; 2522/122808 objects degraded (2.054%); 14527/122808 objects misplaced (11.829%); 16 MiB/s, 301 objects/s recovering 2024-05-15T13:28:16.019 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:15 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1436195965' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T13:28:16.019 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:15 smithi120 ceph-mon[43961]: osdmap e946: 12 total, 12 up, 11 in 2024-05-15T13:28:16.019 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:15 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:28:16.019 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:15 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:28:16.022 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:16.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:15 smithi027 ceph-mon[39795]: pgmap v2402: 47 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 39 active+clean; 2.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 530 op/s; 2522/122808 objects degraded (2.054%); 14527/122808 objects misplaced (11.829%); 16 MiB/s, 301 objects/s recovering 2024-05-15T13:28:16.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:15 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1436195965' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T13:28:16.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:15 smithi027 ceph-mon[39795]: osdmap e946: 12 total, 12 up, 11 in 2024-05-15T13:28:16.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:15 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:28:16.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:15 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:28:16.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:15 smithi177 ceph-mon[44303]: pgmap v2402: 47 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 39 active+clean; 2.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 530 op/s; 2522/122808 objects degraded (2.054%); 14527/122808 objects misplaced (11.829%); 16 MiB/s, 301 objects/s recovering 2024-05-15T13:28:16.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:15 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1436195965' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T13:28:16.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:15 smithi177 ceph-mon[44303]: osdmap e946: 12 total, 12 up, 11 in 2024-05-15T13:28:16.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:15 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:28:16.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:15 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:28:16.622 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 2 1 2024-05-15T13:28:16.838 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:28:16.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:16 smithi027 ceph-mon[39795]: osdmap e947: 12 total, 12 up, 11 in 2024-05-15T13:28:16.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:16 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1941658596' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:28:16.912 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:28:17.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:16 smithi177 ceph-mon[44303]: osdmap e947: 12 total, 12 up, 11 in 2024-05-15T13:28:17.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:16 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1941658596' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:28:17.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:16 smithi120 ceph-mon[43961]: osdmap e947: 12 total, 12 up, 11 in 2024-05-15T13:28:17.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:16 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1941658596' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:28:17.429 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:28:17.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:17.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:28:17.533 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:28:17.565 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:17.905 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:17 smithi027 ceph-mon[39795]: 1.14 scrub starts 2024-05-15T13:28:17.905 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:17 smithi027 ceph-mon[39795]: 1.14 scrub ok 2024-05-15T13:28:17.905 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:17 smithi027 ceph-mon[39795]: pgmap v2405: 47 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 536 op/s; 2522/119355 objects degraded (2.113%); 13886/119355 objects misplaced (11.634%); 14 MiB/s, 319 objects/s recovering 2024-05-15T13:28:17.905 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:17 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:28:17.905 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:17 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1941658596' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:28:17.905 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:17 smithi027 ceph-mon[39795]: osdmap e948: 12 total, 12 up, 11 in 2024-05-15T13:28:18.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:17 smithi177 ceph-mon[44303]: 1.14 scrub starts 2024-05-15T13:28:18.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:17 smithi177 ceph-mon[44303]: 1.14 scrub ok 2024-05-15T13:28:18.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:17 smithi177 ceph-mon[44303]: pgmap v2405: 47 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 536 op/s; 2522/119355 objects degraded (2.113%); 13886/119355 objects misplaced (11.634%); 14 MiB/s, 319 objects/s recovering 2024-05-15T13:28:18.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:17 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:28:18.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:17 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1941658596' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:28:18.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:17 smithi177 ceph-mon[44303]: osdmap e948: 12 total, 12 up, 11 in 2024-05-15T13:28:18.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:17 smithi120 ceph-mon[43961]: 1.14 scrub starts 2024-05-15T13:28:18.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:17 smithi120 ceph-mon[43961]: 1.14 scrub ok 2024-05-15T13:28:18.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:17 smithi120 ceph-mon[43961]: pgmap v2405: 47 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 536 op/s; 2522/119355 objects degraded (2.113%); 13886/119355 objects misplaced (11.634%); 14 MiB/s, 319 objects/s recovering 2024-05-15T13:28:18.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:17 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:28:18.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:17 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1941658596' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:28:18.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:17 smithi120 ceph-mon[43961]: osdmap e948: 12 total, 12 up, 11 in 2024-05-15T13:28:18.925 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:18 smithi120 ceph-mon[43961]: osdmap e949: 12 total, 12 up, 11 in 2024-05-15T13:28:19.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:18 smithi177 ceph-mon[44303]: osdmap e949: 12 total, 12 up, 11 in 2024-05-15T13:28:19.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:18 smithi027 ceph-mon[39795]: osdmap e949: 12 total, 12 up, 11 in 2024-05-15T13:28:19.735 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:28:19.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:19.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:28:19.842 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:28:19.884 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:19.911 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:28:19.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:19.915 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:28:20.016 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:28:20.097 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:20.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:19 smithi177 ceph-mon[44303]: pgmap v2408: 47 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 677 op/s; 2522/114681 objects degraded (2.199%); 13226/114681 objects misplaced (11.533%); 16 MiB/s, 379 objects/s recovering 2024-05-15T13:28:20.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:19 smithi177 ceph-mon[44303]: osdmap e950: 12 total, 12 up, 11 in 2024-05-15T13:28:20.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:19 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 2522/114681 objects degraded (2.199%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:20.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:19 smithi177 ceph-mon[44303]: osdmap e951: 12 total, 12 up, 11 in 2024-05-15T13:28:20.201 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:19 smithi027 ceph-mon[39795]: pgmap v2408: 47 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 677 op/s; 2522/114681 objects degraded (2.199%); 13226/114681 objects misplaced (11.533%); 16 MiB/s, 379 objects/s recovering 2024-05-15T13:28:20.201 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:19 smithi027 ceph-mon[39795]: osdmap e950: 12 total, 12 up, 11 in 2024-05-15T13:28:20.202 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:19 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 2522/114681 objects degraded (2.199%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:20.202 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:19 smithi027 ceph-mon[39795]: osdmap e951: 12 total, 12 up, 11 in 2024-05-15T13:28:20.295 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:19 smithi120 ceph-mon[43961]: pgmap v2408: 47 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 677 op/s; 2522/114681 objects degraded (2.199%); 13226/114681 objects misplaced (11.533%); 16 MiB/s, 379 objects/s recovering 2024-05-15T13:28:20.295 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:19 smithi120 ceph-mon[43961]: osdmap e950: 12 total, 12 up, 11 in 2024-05-15T13:28:20.295 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:19 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 2522/114681 objects degraded (2.199%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:20.295 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:19 smithi120 ceph-mon[43961]: osdmap e951: 12 total, 12 up, 11 in 2024-05-15T13:28:20.657 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:28:21.786 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:28:22.001 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:28:22.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:22.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:28:22.110 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:28:22.170 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:21 smithi027 ceph-mon[39795]: pgmap v2411: 47 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 425 op/s; 2522/114243 objects degraded (2.208%); 13226/114243 objects misplaced (11.577%); 8.1 MiB/s, 129 objects/s recovering 2024-05-15T13:28:22.171 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:21 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:28:22.171 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:21 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:28:22.171 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:21 smithi027 ceph-mon[39795]: osdmap e952: 12 total, 12 up, 11 in 2024-05-15T13:28:22.171 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:21 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/890857147' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:28:22.179 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:22.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:21 smithi177 ceph-mon[44303]: pgmap v2411: 47 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 425 op/s; 2522/114243 objects degraded (2.208%); 13226/114243 objects misplaced (11.577%); 8.1 MiB/s, 129 objects/s recovering 2024-05-15T13:28:22.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:21 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:28:22.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:21 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:28:22.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:21 smithi177 ceph-mon[44303]: osdmap e952: 12 total, 12 up, 11 in 2024-05-15T13:28:22.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:21 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/890857147' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:28:22.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:21 smithi120 ceph-mon[43961]: pgmap v2411: 47 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 425 op/s; 2522/114243 objects degraded (2.208%); 13226/114243 objects misplaced (11.577%); 8.1 MiB/s, 129 objects/s recovering 2024-05-15T13:28:22.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:21 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:28:22.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:21 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:28:22.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:21 smithi120 ceph-mon[43961]: osdmap e952: 12 total, 12 up, 11 in 2024-05-15T13:28:22.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:21 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/890857147' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:28:23.990 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:28:23.991 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:28:23.991 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:28:23.991 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:28:23.991 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:28:23.991 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:28:23.991 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:28:23.991 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:28:23.991 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:28:23.991 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:28:23.991 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:28:23.991 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:28:23.992 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:28:23.992 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:28:23.992 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:28:23.992 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:28:23.992 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:28:23.992 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:28:23.992 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:28:23.992 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:28:23.992 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:28:23.992 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:28:23.992 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:28:23.992 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:28:24.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:23 smithi177 ceph-mon[44303]: pgmap v2413: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 545 op/s; 2522/109395 objects degraded (2.305%); 11744/109395 objects misplaced (10.735%); 21 MiB/s, 356 objects/s recovering 2024-05-15T13:28:24.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:23 smithi027 ceph-mon[39795]: pgmap v2413: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 545 op/s; 2522/109395 objects degraded (2.305%); 11744/109395 objects misplaced (10.735%); 21 MiB/s, 356 objects/s recovering 2024-05-15T13:28:24.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:23 smithi120 ceph-mon[43961]: pgmap v2413: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 545 op/s; 2522/109395 objects degraded (2.305%); 11744/109395 objects misplaced (10.735%); 21 MiB/s, 356 objects/s recovering 2024-05-15T13:28:24.482 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:28:24.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:24.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:28:24.588 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:28:24.682 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:24.711 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:28:24.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:24.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:28:24.815 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:28:24.893 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:24.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:24.928 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:28:24.928 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:28:25.029 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:28:25.043 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:24 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 2522/109395 objects degraded (2.305%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:25.043 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:24 smithi120 ceph-mon[43961]: osdmap e953: 12 total, 12 up, 11 in 2024-05-15T13:28:25.116 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:25.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:24 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 2522/109395 objects degraded (2.305%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:25.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:24 smithi177 ceph-mon[44303]: osdmap e953: 12 total, 12 up, 11 in 2024-05-15T13:28:25.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:24 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 2522/109395 objects degraded (2.305%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:25.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:24 smithi027 ceph-mon[39795]: osdmap e953: 12 total, 12 up, 11 in 2024-05-15T13:28:25.428 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:28:26.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:25 smithi027 ceph-mon[39795]: pgmap v2415: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 439 op/s; 2522/106971 objects degraded (2.358%); 11211/106971 objects misplaced (10.480%); 18 MiB/s, 309 objects/s recovering 2024-05-15T13:28:26.297 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:25 smithi120 ceph-mon[43961]: pgmap v2415: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 439 op/s; 2522/106971 objects degraded (2.358%); 11211/106971 objects misplaced (10.480%); 18 MiB/s, 309 objects/s recovering 2024-05-15T13:28:26.301 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:28:26.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:26.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:28:26.411 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:28:26.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:25 smithi177 ceph-mon[44303]: pgmap v2415: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 439 op/s; 2522/106971 objects degraded (2.358%); 11211/106971 objects misplaced (10.480%); 18 MiB/s, 309 objects/s recovering 2024-05-15T13:28:26.458 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:27.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:26 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3584017683' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:28:27.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:26 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:28:27.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:26 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3584017683' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:28:27.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:26 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:28:27.427 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:28:27.445 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:26 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3584017683' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:28:27.445 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:26 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:28:27.474 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:28:27.474 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [ 2024-05-15T13:28:27.474 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.475 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26536.0:207694 9.3 e953/809 9:c22b90b8:::benchmark_data_smithi177_159919_object87451:head v 953'13042, mlcod=953'13039)", 2024-05-15T13:28:27.475 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:28:26.668384+0000", 2024-05-15T13:28:27.475 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.015676321, 2024-05-15T13:28:27.475 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.015708680999999999, 2024-05-15T13:28:27.475 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:27.475 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "started", 2024-05-15T13:28:27.475 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:27.476 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.476 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:27.476 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.668384+0000", 2024-05-15T13:28:27.476 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:27.476 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.476 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.476 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:27.477 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.668384+0000", 2024-05-15T13:28:27.477 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:27.477 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.477 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.477 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:27.477 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.668385+0000", 2024-05-15T13:28:27.477 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.6099999999999999e-07 2024-05-15T13:28:27.478 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.478 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.478 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:27.478 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.668393+0000", 2024-05-15T13:28:27.478 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.4200000000000007e-06 2024-05-15T13:28:27.478 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.479 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.479 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:27.479 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.668410+0000", 2024-05-15T13:28:27.479 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7189000000000001e-05 2024-05-15T13:28:27.479 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.479 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.479 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:27.479 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.668418+0000", 2024-05-15T13:28:27.479 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4909999999999999e-06 2024-05-15T13:28:27.479 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.479 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.479 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:27.480 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.668883+0000", 2024-05-15T13:28:27.480 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00046571399999999999 2024-05-15T13:28:27.480 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.480 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.481 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:28:27.481 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.668903+0000", 2024-05-15T13:28:27.481 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9636999999999999e-05 2024-05-15T13:28:27.482 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:27.482 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:27.482 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:27.482 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.482 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.482 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26536.0:207695 9.d e953/888 9:b152ad24:::benchmark_data_smithi177_159919_object87452:head v 953'13239, mlcod=851'7502)", 2024-05-15T13:28:27.482 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:28:26.674453+0000", 2024-05-15T13:28:27.482 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.0096073540000000002, 2024-05-15T13:28:27.482 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0097052130000000007, 2024-05-15T13:28:27.482 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:27.482 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "started", 2024-05-15T13:28:27.482 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:27.483 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.483 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:27.483 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.674453+0000", 2024-05-15T13:28:27.483 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:27.483 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.483 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.483 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:27.483 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.674453+0000", 2024-05-15T13:28:27.483 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:27.483 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.483 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.483 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:27.483 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.674453+0000", 2024-05-15T13:28:27.484 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.3099999999999999e-07 2024-05-15T13:28:27.484 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.484 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.484 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:27.484 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.674457+0000", 2024-05-15T13:28:27.484 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4700000000000004e-06 2024-05-15T13:28:27.484 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.484 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.484 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:27.484 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.674463+0000", 2024-05-15T13:28:27.484 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8300000000000001e-06 2024-05-15T13:28:27.484 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.484 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.485 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:27.485 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.674467+0000", 2024-05-15T13:28:27.485 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.1370000000000004e-06 2024-05-15T13:28:27.485 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.485 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.485 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:27.485 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.674513+0000", 2024-05-15T13:28:27.485 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.5719000000000001e-05 2024-05-15T13:28:27.485 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.485 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.485 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:28:27.485 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.674540+0000", 2024-05-15T13:28:27.486 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.7262000000000001e-05 2024-05-15T13:28:27.486 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:27.486 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:27.486 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:27.486 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.486 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.486 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_repop(client.26536.0:207699 9.d e953/888 9:bdc32fe5:::benchmark_data_smithi177_159919_object87456:head v 953'13240, mlcod=851'7502)", 2024-05-15T13:28:27.486 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:28:26.674479+0000", 2024-05-15T13:28:27.486 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.0095808950000000007, 2024-05-15T13:28:27.486 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0097353330000000005, 2024-05-15T13:28:27.486 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:27.487 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "started", 2024-05-15T13:28:27.487 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:27.487 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.487 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:27.487 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.674479+0000", 2024-05-15T13:28:27.487 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:27.487 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.487 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.487 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:27.487 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.674479+0000", 2024-05-15T13:28:27.487 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:27.487 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.487 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.487 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:27.488 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.674479+0000", 2024-05-15T13:28:27.488 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.74e-07 2024-05-15T13:28:27.488 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.488 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.488 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:27.488 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.674483+0000", 2024-05-15T13:28:27.488 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.2880000000000001e-06 2024-05-15T13:28:27.488 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.488 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.488 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:27.488 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.674488+0000", 2024-05-15T13:28:27.488 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3900000000000001e-06 2024-05-15T13:28:27.489 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.489 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.489 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:27.489 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.674492+0000", 2024-05-15T13:28:27.489 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.76e-06 2024-05-15T13:28:27.489 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.489 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.489 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:27.489 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.679270+0000", 2024-05-15T13:28:27.489 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.004778671 2024-05-15T13:28:27.489 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.489 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.489 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:28:27.489 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.679292+0000", 2024-05-15T13:28:27.489 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1432000000000001e-05 2024-05-15T13:28:27.490 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:27.490 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:27.490 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:27.490 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.490 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.490 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:207703 9.b 9:d14d5f6c:::benchmark_data_smithi177_159919_object87460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e953)", 2024-05-15T13:28:27.490 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:28:26.678577+0000", 2024-05-15T13:28:27.490 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.0054826559999999998, 2024-05-15T13:28:27.490 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.005693867, 2024-05-15T13:28:27.490 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:27.490 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "waiting for sub ops", 2024-05-15T13:28:27.490 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:28:27.490 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:28:27.491 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:28:27.491 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 207703 2024-05-15T13:28:27.491 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.491 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:27.491 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.491 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:27.491 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.678577+0000", 2024-05-15T13:28:27.491 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:27.491 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.491 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.491 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:27.491 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.678577+0000", 2024-05-15T13:28:27.491 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:27.491 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.491 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.491 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:27.492 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.678579+0000", 2024-05-15T13:28:27.492 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1790000000000001e-06 2024-05-15T13:28:27.492 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.492 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.492 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:27.492 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.678583+0000", 2024-05-15T13:28:27.492 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.938e-06 2024-05-15T13:28:27.492 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.492 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.492 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:27.492 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.678591+0000", 2024-05-15T13:28:27.492 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8380000000000002e-06 2024-05-15T13:28:27.492 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.492 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.493 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:27.493 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.678598+0000", 2024-05-15T13:28:27.493 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4949999999999996e-06 2024-05-15T13:28:27.493 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.493 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.493 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:27.493 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.678628+0000", 2024-05-15T13:28:27.493 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.9686e-05 2024-05-15T13:28:27.493 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.493 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.493 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:28:27.493 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.678768+0000", 2024-05-15T13:28:27.493 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000140576 2024-05-15T13:28:27.493 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.494 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.494 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,8,10", 2024-05-15T13:28:27.494 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.678904+0000", 2024-05-15T13:28:27.494 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013545500000000001 2024-05-15T13:28:27.494 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:27.494 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:27.494 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:27.494 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.494 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.494 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "MOSDPGPushReply(9.b 953/921 [PushReplyOp(9:db72f415:::benchmark_data_smithi177_159919_object89787:head)])", 2024-05-15T13:28:27.494 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:28:26.681194+0000", 2024-05-15T13:28:27.494 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 0.0028660460000000001, 2024-05-15T13:28:27.494 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0031470790000000001, 2024-05-15T13:28:27.494 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:27.495 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:28:27.495 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:27.495 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.495 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:27.495 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.681194+0000", 2024-05-15T13:28:27.495 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:27.495 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.495 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.495 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:27.495 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.681194+0000", 2024-05-15T13:28:27.495 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:27.495 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.495 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.496 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:27.496 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.681194+0000", 2024-05-15T13:28:27.496 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0599999999999999e-07 2024-05-15T13:28:27.496 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.496 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.496 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:27.496 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.681197+0000", 2024-05-15T13:28:27.496 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.7640000000000001e-06 2024-05-15T13:28:27.496 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.496 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.496 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:27.496 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.681202+0000", 2024-05-15T13:28:27.496 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0089999999999998e-06 2024-05-15T13:28:27.496 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.496 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.497 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:27.497 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.681207+0000", 2024-05-15T13:28:27.497 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9960000000000001e-06 2024-05-15T13:28:27.497 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:27.497 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:27.497 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:27.497 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:26.684131+0000", 2024-05-15T13:28:27.497 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0029236689999999998 2024-05-15T13:28:27.497 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:27.497 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:27.497 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:27.497 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:27.497 INFO:teuthology.orchestra.run.smithi120.stdout: ], 2024-05-15T13:28:27.497 INFO:teuthology.orchestra.run.smithi120.stdout: "num_ops": 5 2024-05-15T13:28:27.497 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:28:27.826 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:28:27.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:27.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:28:27.930 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:28:27.988 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.2 to 1 (10000) 2024-05-15T13:28:27.988 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:28.033 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:28:28.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:28.042 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:28:28.143 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:28:28.173 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:27 smithi027 ceph-mon[39795]: pgmap v2416: 47 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 521 op/s; 2522/103083 objects degraded (2.447%); 10210/103083 objects misplaced (9.905%); 22 MiB/s, 368 objects/s recovering 2024-05-15T13:28:28.173 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:27 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T13:28:28.173 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:27 smithi027 ceph-mon[39795]: osdmap e954: 12 total, 12 up, 11 in 2024-05-15T13:28:28.175 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:28.198 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:28:28.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:28.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:28:28.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:27 smithi120 ceph-mon[43961]: pgmap v2416: 47 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 521 op/s; 2522/103083 objects degraded (2.447%); 10210/103083 objects misplaced (9.905%); 22 MiB/s, 368 objects/s recovering 2024-05-15T13:28:28.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:27 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T13:28:28.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:27 smithi120 ceph-mon[43961]: osdmap e954: 12 total, 12 up, 11 in 2024-05-15T13:28:28.303 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:28:28.387 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:28.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:27 smithi177 ceph-mon[44303]: pgmap v2416: 47 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 521 op/s; 2522/103083 objects degraded (2.447%); 10210/103083 objects misplaced (9.905%); 22 MiB/s, 368 objects/s recovering 2024-05-15T13:28:28.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:27 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T13:28:28.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:27 smithi177 ceph-mon[44303]: osdmap e954: 12 total, 12 up, 11 in 2024-05-15T13:28:29.216 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:28:29.254 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 3 1 2024-05-15T13:28:29.681 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:28:29.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:29.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:28:29.790 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:28:29.860 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:30.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:29 smithi027 ceph-mon[39795]: pgmap v2418: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.0 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 698 op/s; 2522/97821 objects degraded (2.578%); 9512/97821 objects misplaced (9.724%); 25 MiB/s, 419 objects/s recovering 2024-05-15T13:28:30.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:29 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 2522/97821 objects degraded (2.578%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:30.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:29 smithi027 ceph-mon[39795]: osdmap e955: 12 total, 12 up, 11 in 2024-05-15T13:28:30.303 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:29 smithi120 ceph-mon[43961]: pgmap v2418: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.0 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 698 op/s; 2522/97821 objects degraded (2.578%); 9512/97821 objects misplaced (9.724%); 25 MiB/s, 419 objects/s recovering 2024-05-15T13:28:30.303 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:29 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 2522/97821 objects degraded (2.578%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:30.303 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:29 smithi120 ceph-mon[43961]: osdmap e955: 12 total, 12 up, 11 in 2024-05-15T13:28:30.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:29 smithi177 ceph-mon[44303]: pgmap v2418: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.0 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 698 op/s; 2522/97821 objects degraded (2.578%); 9512/97821 objects misplaced (9.724%); 25 MiB/s, 419 objects/s recovering 2024-05-15T13:28:30.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:29 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 2522/97821 objects degraded (2.578%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:30.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:29 smithi177 ceph-mon[44303]: osdmap e955: 12 total, 12 up, 11 in 2024-05-15T13:28:31.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:30 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:28:31.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:30 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:28:31.395 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:31 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:28:32.022 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:28:32.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:32 smithi027 ceph-mon[39795]: pgmap v2420: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.0 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 499 op/s; 2522/97341 objects degraded (2.591%); 9512/97341 objects misplaced (9.772%); 14 MiB/s, 230 objects/s recovering 2024-05-15T13:28:32.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:32 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/735850360' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:28:32.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:32 smithi120 ceph-mon[43961]: pgmap v2420: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.0 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 499 op/s; 2522/97341 objects degraded (2.591%); 9512/97341 objects misplaced (9.772%); 14 MiB/s, 230 objects/s recovering 2024-05-15T13:28:32.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:32 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/735850360' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:28:32.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:32 smithi177 ceph-mon[44303]: pgmap v2420: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.0 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 499 op/s; 2522/97341 objects degraded (2.591%); 9512/97341 objects misplaced (9.772%); 14 MiB/s, 230 objects/s recovering 2024-05-15T13:28:32.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:32 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/735850360' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:28:33.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:33 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:28:33.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:33 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/735850360' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:28:33.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:33 smithi027 ceph-mon[39795]: osdmap e956: 12 total, 12 up, 11 in 2024-05-15T13:28:33.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:33 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:28:33.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:33 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:28:33.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:33 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:28:33.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:33 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:28:33.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:33 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:28:33.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:33 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:28:33.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:33 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/735850360' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:28:33.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:33 smithi120 ceph-mon[43961]: osdmap e956: 12 total, 12 up, 11 in 2024-05-15T13:28:33.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:33 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:28:33.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:33 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:28:33.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:33 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:28:33.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:33 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:28:33.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:33 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:28:33.365 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:28:33.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:33 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:28:33.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:33 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/735850360' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:28:33.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:33 smithi177 ceph-mon[44303]: osdmap e956: 12 total, 12 up, 11 in 2024-05-15T13:28:33.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:33 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:28:33.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:33 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:28:33.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:33 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:28:33.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:33 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:28:33.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:33 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:28:34.248 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-05-15T13:28:34.278 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:34 smithi120 ceph-mon[43961]: pgmap v2422: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 637 op/s; 2522/91614 objects degraded (2.753%); 8550/91614 objects misplaced (9.333%); 15 MiB/s, 272 objects/s recovering 2024-05-15T13:28:34.278 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:34 smithi120 ceph-mon[43961]: osdmap e957: 12 total, 12 up, 11 in 2024-05-15T13:28:34.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:34 smithi027 ceph-mon[39795]: pgmap v2422: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 637 op/s; 2522/91614 objects degraded (2.753%); 8550/91614 objects misplaced (9.333%); 15 MiB/s, 272 objects/s recovering 2024-05-15T13:28:34.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:34 smithi027 ceph-mon[39795]: osdmap e957: 12 total, 12 up, 11 in 2024-05-15T13:28:34.299 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:28:34.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:34.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:28:34.405 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:28:34.440 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:34.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:34 smithi177 ceph-mon[44303]: pgmap v2422: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 637 op/s; 2522/91614 objects degraded (2.753%); 8550/91614 objects misplaced (9.333%); 15 MiB/s, 272 objects/s recovering 2024-05-15T13:28:34.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:34 smithi177 ceph-mon[44303]: osdmap e957: 12 total, 12 up, 11 in 2024-05-15T13:28:34.600 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:28:34.993 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:28:35.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:35 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 2522/91614 objects degraded (2.753%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:35.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:35 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1791314444' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:28:35.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:35 smithi027 ceph-mon[39795]: osdmap e958: 12 total, 12 up, 11 in 2024-05-15T13:28:35.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:35 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 2522/91614 objects degraded (2.753%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:35.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:35 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1791314444' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:28:35.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:35 smithi120 ceph-mon[43961]: osdmap e958: 12 total, 12 up, 11 in 2024-05-15T13:28:35.443 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:35 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 2522/91614 objects degraded (2.753%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:35.444 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:35 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1791314444' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:28:35.444 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:35 smithi177 ceph-mon[44303]: osdmap e958: 12 total, 12 up, 11 in 2024-05-15T13:28:35.896 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:28:35.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:35.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:28:36.001 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:28:36.060 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:36.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:36 smithi027 ceph-mon[39795]: pgmap v2424: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 496 op/s; 2522/88881 objects degraded (2.838%); 8032/88881 objects misplaced (9.037%); 13 MiB/s, 234 objects/s recovering 2024-05-15T13:28:36.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:36 smithi120 ceph-mon[43961]: pgmap v2424: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 496 op/s; 2522/88881 objects degraded (2.838%); 8032/88881 objects misplaced (9.037%); 13 MiB/s, 234 objects/s recovering 2024-05-15T13:28:36.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:36 smithi177 ceph-mon[44303]: pgmap v2424: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 496 op/s; 2522/88881 objects degraded (2.838%); 8032/88881 objects misplaced (9.037%); 13 MiB/s, 234 objects/s recovering 2024-05-15T13:28:36.827 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:28:36.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:36.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:28:36.932 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:28:36.972 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:37.017 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:28:37.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:37.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:28:37.124 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:28:37.175 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:37.218 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:28:37.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:37.229 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:28:37.331 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:28:37.361 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:28:37.361 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:28:37.361 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:28:37.361 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:28:37.361 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:28:37.361 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:28:37.362 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:28:37.362 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:28:37.362 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:28:37.362 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:28:37.362 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:28:37.362 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:28:37.362 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:28:37.362 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:28:37.362 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:28:37.363 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:28:37.363 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:28:37.363 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:28:37.363 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:28:37.363 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:28:37.363 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:28:37.363 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:28:37.363 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:28:37.363 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:28:37.430 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:38.031 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:28:38.199 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:28:38.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:38.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:28:38.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:38 smithi027 ceph-mon[39795]: pgmap v2426: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 656 op/s; 2522/85530 objects degraded (2.949%); 7311/85530 objects misplaced (8.548%); 20 MiB/s, 353 objects/s recovering 2024-05-15T13:28:38.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:38 smithi120 ceph-mon[43961]: pgmap v2426: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 656 op/s; 2522/85530 objects degraded (2.949%); 7311/85530 objects misplaced (8.548%); 20 MiB/s, 353 objects/s recovering 2024-05-15T13:28:38.304 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:28:38.343 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:38.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:38 smithi177 ceph-mon[44303]: pgmap v2426: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 656 op/s; 2522/85530 objects degraded (2.949%); 7311/85530 objects misplaced (8.548%); 20 MiB/s, 353 objects/s recovering 2024-05-15T13:28:38.930 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:28:38.930 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:28:38.996 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:28:38.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:39.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:28:39.101 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:28:39.172 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:40.073 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:28:40.164 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:28:40.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:40.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:28:40.272 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:28:40.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:40 smithi027 ceph-mon[39795]: pgmap v2427: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 584 op/s; 2522/80943 objects degraded (3.116%); 6711/80943 objects misplaced (8.291%); 13 MiB/s, 278 objects/s recovering 2024-05-15T13:28:40.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:40 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 2522/80943 objects degraded (3.116%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:40.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:40 smithi027 ceph-mon[39795]: osdmap e959: 12 total, 12 up, 11 in 2024-05-15T13:28:40.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:40 smithi120 ceph-mon[43961]: pgmap v2427: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 584 op/s; 2522/80943 objects degraded (3.116%); 6711/80943 objects misplaced (8.291%); 13 MiB/s, 278 objects/s recovering 2024-05-15T13:28:40.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:40 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 2522/80943 objects degraded (3.116%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:40.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:40 smithi120 ceph-mon[43961]: osdmap e959: 12 total, 12 up, 11 in 2024-05-15T13:28:40.303 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:40.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:40 smithi177 ceph-mon[44303]: pgmap v2427: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 584 op/s; 2522/80943 objects degraded (3.116%); 6711/80943 objects misplaced (8.291%); 13 MiB/s, 278 objects/s recovering 2024-05-15T13:28:40.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:40 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 2522/80943 objects degraded (3.116%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:40.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:40 smithi177 ceph-mon[44303]: osdmap e959: 12 total, 12 up, 11 in 2024-05-15T13:28:41.298 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:41 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:28:41.298 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:41 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:28:41.298 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:41 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:28:41.298 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:41 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:28:41.299 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:41 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:28:41.299 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:41 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:28:41.299 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:41 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:28:41.364 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:28:41.366 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:41 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:28:41.366 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:41 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:28:41.366 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:41 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:28:41.366 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:41 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:28:41.366 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:41 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:28:41.366 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:41 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:28:41.366 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:41 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:28:41.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:41.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:28:41.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:41 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:28:41.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:41 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:28:41.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:41 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:28:41.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:41 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:28:41.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:41 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:28:41.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:41 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:28:41.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:41 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:28:41.468 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:28:41.518 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:41.545 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:28:41.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:41.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:28:41.654 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:28:41.715 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:41.742 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:28:41.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:41.748 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:28:41.848 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:28:41.924 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:42.082 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:28:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.3 to 1 (10000) 2024-05-15T13:28:42.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:42 smithi120 ceph-mon[43961]: pgmap v2429: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 435 op/s; 2522/79749 objects degraded (3.162%); 6711/79749 objects misplaced (8.415%); 9.8 MiB/s, 221 objects/s recovering 2024-05-15T13:28:42.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:42 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:28:42.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:42 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:28:42.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:42 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1098947752' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:28:42.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:42 smithi027 ceph-mon[39795]: pgmap v2429: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 435 op/s; 2522/79749 objects degraded (3.162%); 6711/79749 objects misplaced (8.415%); 9.8 MiB/s, 221 objects/s recovering 2024-05-15T13:28:42.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:42 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:28:42.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:42 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:28:42.394 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:42 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1098947752' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:28:42.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:42 smithi177 ceph-mon[44303]: pgmap v2429: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 435 op/s; 2522/79749 objects degraded (3.162%); 6711/79749 objects misplaced (8.415%); 9.8 MiB/s, 221 objects/s recovering 2024-05-15T13:28:42.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:42 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:28:42.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:42 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:28:42.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:42 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1098947752' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:28:43.444 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:43 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1098947752' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T13:28:43.444 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:43 smithi177 ceph-mon[44303]: osdmap e960: 12 total, 12 up, 11 in 2024-05-15T13:28:43.499 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:43 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1098947752' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T13:28:43.499 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:43 smithi027 ceph-mon[39795]: osdmap e960: 12 total, 12 up, 11 in 2024-05-15T13:28:43.529 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 5 1 2024-05-15T13:28:43.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:43 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1098947752' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T13:28:43.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:43 smithi120 ceph-mon[43961]: osdmap e960: 12 total, 12 up, 11 in 2024-05-15T13:28:44.418 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:28:44.444 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:44 smithi177 ceph-mon[44303]: pgmap v2431: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 621 op/s; 2522/73977 objects degraded (3.409%); 6277/73977 objects misplaced (8.485%); 12 MiB/s, 286 objects/s recovering 2024-05-15T13:28:44.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:44 smithi027 ceph-mon[39795]: pgmap v2431: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 621 op/s; 2522/73977 objects degraded (3.409%); 6277/73977 objects misplaced (8.485%); 12 MiB/s, 286 objects/s recovering 2024-05-15T13:28:44.544 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:44 smithi120 ceph-mon[43961]: pgmap v2431: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 621 op/s; 2522/73977 objects degraded (3.409%); 6277/73977 objects misplaced (8.485%); 12 MiB/s, 286 objects/s recovering 2024-05-15T13:28:45.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:45 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/632541153' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:28:45.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:45 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 2522/73977 objects degraded (3.409%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:45.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:45 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:28:45.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:45 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/632541153' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:28:45.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:45 smithi177 ceph-mon[44303]: osdmap e961: 12 total, 12 up, 11 in 2024-05-15T13:28:45.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:45 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/632541153' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:28:45.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:45 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 2522/73977 objects degraded (3.409%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:45.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:45 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:28:45.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:45 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/632541153' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:28:45.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:45 smithi027 ceph-mon[39795]: osdmap e961: 12 total, 12 up, 11 in 2024-05-15T13:28:45.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:45 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/632541153' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:28:45.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:45 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 2522/73977 objects degraded (3.409%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:45.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:45 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:28:45.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:45 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/632541153' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:28:45.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:45 smithi120 ceph-mon[43961]: osdmap e961: 12 total, 12 up, 11 in 2024-05-15T13:28:46.334 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:28:46.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:46.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:28:46.378 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-05-15T13:28:46.445 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:46 smithi177 ceph-mon[44303]: pgmap v2433: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 557 op/s; 2541/70911 objects degraded (3.583%); 5801/70911 objects misplaced (8.181%); 7.6 MiB/s, 172 objects/s recovering 2024-05-15T13:28:46.447 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:28:46.468 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:46 smithi120 ceph-mon[43961]: pgmap v2433: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 557 op/s; 2541/70911 objects degraded (3.583%); 5801/70911 objects misplaced (8.181%); 7.6 MiB/s, 172 objects/s recovering 2024-05-15T13:28:46.510 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:46.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:46 smithi027 ceph-mon[39795]: pgmap v2433: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 557 op/s; 2541/70911 objects degraded (3.583%); 5801/70911 objects misplaced (8.181%); 7.6 MiB/s, 172 objects/s recovering 2024-05-15T13:28:46.551 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:28:46.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:46.557 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:28:46.658 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:28:46.771 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:46.830 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:28:46.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:46.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:28:46.868 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:28:46.935 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:28:46.994 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:47.031 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:28:47.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:47.034 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:28:47.135 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:28:47.174 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:47.203 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:28:47.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:47.211 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:28:47.312 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:28:47.358 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:47.392 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:28:47.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:47.398 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:28:47.499 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:28:47.533 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:48.390 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:28:48.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:48.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:28:48.444 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:48 smithi177 ceph-mon[44303]: pgmap v2434: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 2541/66732 objects degraded (3.808%); 5366/66732 objects misplaced (8.041%); 9.8 MiB/s, 198 objects/s recovering 2024-05-15T13:28:48.444 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:48 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:28:48.444 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:48 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:28:48.444 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:48 smithi177 ceph-mon[44303]: osdmap e962: 12 total, 12 up, 11 in 2024-05-15T13:28:48.495 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:28:48.498 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:28:48.525 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:28:48.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:48 smithi120 ceph-mon[43961]: pgmap v2434: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 2541/66732 objects degraded (3.808%); 5366/66732 objects misplaced (8.041%); 9.8 MiB/s, 198 objects/s recovering 2024-05-15T13:28:48.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:48 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:28:48.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:48 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:28:48.530 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:48 smithi120 ceph-mon[43961]: osdmap e962: 12 total, 12 up, 11 in 2024-05-15T13:28:48.531 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:48.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:48 smithi027 ceph-mon[39795]: pgmap v2434: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 2541/66732 objects degraded (3.808%); 5366/66732 objects misplaced (8.041%); 9.8 MiB/s, 198 objects/s recovering 2024-05-15T13:28:48.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:48 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:28:48.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:48 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:28:48.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:48 smithi027 ceph-mon[39795]: osdmap e962: 12 total, 12 up, 11 in 2024-05-15T13:28:48.551 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:28:48.572 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:28:48.602 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:28:48.624 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:28:48.653 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:28:48.693 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:28:48.720 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:28:48.751 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:28:48.779 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:28:48.810 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:28:48.811 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:28:48.811 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:28:48.811 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:28:48.812 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:28:48.812 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:28:48.812 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:28:48.812 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:28:48.812 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:28:48.812 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:28:48.812 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:28:48.812 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:28:48.812 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:28:48.993 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:28:48.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:48.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:28:49.098 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:28:49.134 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:49.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:49 smithi120 ceph-mon[43961]: osdmap e963: 12 total, 12 up, 11 in 2024-05-15T13:28:49.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:49 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T13:28:49.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:49 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T13:28:49.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:49 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2019431283' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:28:49.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:49 smithi177 ceph-mon[44303]: osdmap e963: 12 total, 12 up, 11 in 2024-05-15T13:28:49.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:49 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T13:28:49.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:49 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T13:28:49.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:49 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2019431283' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:28:49.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:49 smithi027 ceph-mon[39795]: osdmap e963: 12 total, 12 up, 11 in 2024-05-15T13:28:49.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:49 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T13:28:49.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:49 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T13:28:49.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:49 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2019431283' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:28:49.981 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:28:49.987 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:28:49.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:49.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:28:50.093 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:28:50.128 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:50.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:50 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:28:50.197 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:50 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:28:50.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:50 smithi027 ceph-mon[39795]: pgmap v2437: 47 pgs: 1 active+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 689 op/s; 2569/61569 objects degraded (4.173%); 4640/61569 objects misplaced (7.536%); 12 MiB/s, 260 objects/s recovering 2024-05-15T13:28:50.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:50 smithi027 ceph-mon[39795]: osdmap e964: 12 total, 12 up, 11 in 2024-05-15T13:28:50.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:50 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 2569/61569 objects degraded (4.173%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:50.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:50 smithi027 ceph-mon[39795]: osdmap e965: 12 total, 12 up, 11 in 2024-05-15T13:28:50.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:50 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:28:50.198 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:50 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:28:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:28:50.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:50.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:28:50.366 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:28:50.410 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:50.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:50 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:28:50.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:50 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:28:50.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:50 smithi120 ceph-mon[43961]: pgmap v2437: 47 pgs: 1 active+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 689 op/s; 2569/61569 objects degraded (4.173%); 4640/61569 objects misplaced (7.536%); 12 MiB/s, 260 objects/s recovering 2024-05-15T13:28:50.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:50 smithi120 ceph-mon[43961]: osdmap e964: 12 total, 12 up, 11 in 2024-05-15T13:28:50.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:50 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 2569/61569 objects degraded (4.173%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:50.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:50 smithi120 ceph-mon[43961]: osdmap e965: 12 total, 12 up, 11 in 2024-05-15T13:28:50.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:50 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:28:50.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:50 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:28:50.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:50 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:28:50.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:50 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:28:50.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:50 smithi177 ceph-mon[44303]: pgmap v2437: 47 pgs: 1 active+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 689 op/s; 2569/61569 objects degraded (4.173%); 4640/61569 objects misplaced (7.536%); 12 MiB/s, 260 objects/s recovering 2024-05-15T13:28:50.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:50 smithi177 ceph-mon[44303]: osdmap e964: 12 total, 12 up, 11 in 2024-05-15T13:28:50.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:50 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 2569/61569 objects degraded (4.173%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:50.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:50 smithi177 ceph-mon[44303]: osdmap e965: 12 total, 12 up, 11 in 2024-05-15T13:28:50.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:50 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:28:50.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:50 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:28:51.260 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:28:51.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:51.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:28:51.366 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:28:51.402 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:51.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:51 smithi027 ceph-mon[39795]: osdmap e966: 12 total, 12 up, 11 in 2024-05-15T13:28:51.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:51 smithi027 ceph-mon[39795]: 1.9 scrub starts 2024-05-15T13:28:51.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:51 smithi027 ceph-mon[39795]: 1.f scrub starts 2024-05-15T13:28:51.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:51 smithi027 ceph-mon[39795]: pgmap v2441: 47 pgs: 1 active+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.2 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 456 op/s; 2569/60561 objects degraded (4.242%); 4640/60561 objects misplaced (7.662%); 7.8 MiB/s, 125 objects/s recovering 2024-05-15T13:28:51.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:51 smithi027 ceph-mon[39795]: 1.f scrub ok 2024-05-15T13:28:51.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:51 smithi120 ceph-mon[43961]: osdmap e966: 12 total, 12 up, 11 in 2024-05-15T13:28:51.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:51 smithi120 ceph-mon[43961]: 1.9 scrub starts 2024-05-15T13:28:51.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:51 smithi120 ceph-mon[43961]: 1.f scrub starts 2024-05-15T13:28:51.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:51 smithi120 ceph-mon[43961]: pgmap v2441: 47 pgs: 1 active+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.2 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 456 op/s; 2569/60561 objects degraded (4.242%); 4640/60561 objects misplaced (7.662%); 7.8 MiB/s, 125 objects/s recovering 2024-05-15T13:28:51.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:51 smithi120 ceph-mon[43961]: 1.f scrub ok 2024-05-15T13:28:51.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:51 smithi177 ceph-mon[44303]: osdmap e966: 12 total, 12 up, 11 in 2024-05-15T13:28:51.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:51 smithi177 ceph-mon[44303]: 1.9 scrub starts 2024-05-15T13:28:51.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:51 smithi177 ceph-mon[44303]: 1.f scrub starts 2024-05-15T13:28:51.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:51 smithi177 ceph-mon[44303]: pgmap v2441: 47 pgs: 1 active+undersized+remapped, 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.2 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 456 op/s; 2569/60561 objects degraded (4.242%); 4640/60561 objects misplaced (7.662%); 7.8 MiB/s, 125 objects/s recovering 2024-05-15T13:28:51.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:51 smithi177 ceph-mon[44303]: 1.f scrub ok 2024-05-15T13:28:51.965 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:28:52.518 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:28:52.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:52 smithi027 ceph-mon[39795]: 1.9 scrub ok 2024-05-15T13:28:52.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:52 smithi177 ceph-mon[44303]: 1.9 scrub ok 2024-05-15T13:28:52.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:52 smithi120 ceph-mon[43961]: 1.9 scrub ok 2024-05-15T13:28:52.843 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:28:53.406 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.5 to 1 (10000) 2024-05-15T13:28:53.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:53 smithi177 ceph-mon[44303]: 9.e scrub starts 2024-05-15T13:28:53.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:53 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/905550408' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T13:28:53.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:53 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T13:28:53.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:53 smithi177 ceph-mon[44303]: pgmap v2442: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 573 op/s; 2569/55767 objects degraded (4.607%); 4178/55767 objects misplaced (7.492%); 12 MiB/s, 245 objects/s recovering 2024-05-15T13:28:53.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:53 smithi177 ceph-mon[44303]: 1.7 scrub starts 2024-05-15T13:28:53.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:53 smithi177 ceph-mon[44303]: 1.7 scrub ok 2024-05-15T13:28:53.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:53 smithi027 ceph-mon[39795]: 9.e scrub starts 2024-05-15T13:28:53.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:53 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/905550408' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T13:28:53.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:53 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T13:28:53.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:53 smithi027 ceph-mon[39795]: pgmap v2442: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 573 op/s; 2569/55767 objects degraded (4.607%); 4178/55767 objects misplaced (7.492%); 12 MiB/s, 245 objects/s recovering 2024-05-15T13:28:53.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:53 smithi027 ceph-mon[39795]: 1.7 scrub starts 2024-05-15T13:28:53.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:53 smithi027 ceph-mon[39795]: 1.7 scrub ok 2024-05-15T13:28:53.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:53 smithi120 ceph-mon[43961]: 9.e scrub starts 2024-05-15T13:28:53.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:53 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/905550408' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T13:28:53.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:53 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T13:28:53.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:53 smithi120 ceph-mon[43961]: pgmap v2442: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 573 op/s; 2569/55767 objects degraded (4.607%); 4178/55767 objects misplaced (7.492%); 12 MiB/s, 245 objects/s recovering 2024-05-15T13:28:53.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:53 smithi120 ceph-mon[43961]: 1.7 scrub starts 2024-05-15T13:28:53.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:53 smithi120 ceph-mon[43961]: 1.7 scrub ok 2024-05-15T13:28:54.368 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:28:54.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:54.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:28:54.478 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:28:54.520 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:54.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:54 smithi027 ceph-mon[39795]: 9.e scrub ok 2024-05-15T13:28:54.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:54 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T13:28:54.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:54 smithi027 ceph-mon[39795]: osdmap e967: 12 total, 12 up, 11 in 2024-05-15T13:28:54.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:54 smithi027 ceph-mon[39795]: 1.d scrub starts 2024-05-15T13:28:54.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:54 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 2569/55767 objects degraded (4.607%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:54.579 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:28:54.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:54.584 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:28:54.686 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:28:54.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:54 smithi177 ceph-mon[44303]: 9.e scrub ok 2024-05-15T13:28:54.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:54 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T13:28:54.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:54 smithi177 ceph-mon[44303]: osdmap e967: 12 total, 12 up, 11 in 2024-05-15T13:28:54.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:54 smithi177 ceph-mon[44303]: 1.d scrub starts 2024-05-15T13:28:54.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:54 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 2569/55767 objects degraded (4.607%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:54.742 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:54.769 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:28:54.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:54.775 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:28:54.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:54 smithi120 ceph-mon[43961]: 9.e scrub ok 2024-05-15T13:28:54.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:54 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T13:28:54.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:54 smithi120 ceph-mon[43961]: osdmap e967: 12 total, 12 up, 11 in 2024-05-15T13:28:54.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:54 smithi120 ceph-mon[43961]: 1.d scrub starts 2024-05-15T13:28:54.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:54 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 2569/55767 objects degraded (4.607%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:54.875 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:28:54.901 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:54.933 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:28:54.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:54.938 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:28:55.039 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:28:55.089 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:55.121 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:28:55.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:55.132 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:28:55.234 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:28:55.268 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:55.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:55 smithi177 ceph-mon[44303]: 1.d scrub ok 2024-05-15T13:28:55.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:55 smithi177 ceph-mon[44303]: pgmap v2444: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 525 op/s; 2569/52761 objects degraded (4.869%); 3609/52761 objects misplaced (6.840%); 9.3 MiB/s, 187 objects/s recovering 2024-05-15T13:28:55.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:55 smithi027 ceph-mon[39795]: 1.d scrub ok 2024-05-15T13:28:55.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:55 smithi027 ceph-mon[39795]: pgmap v2444: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 525 op/s; 2569/52761 objects degraded (4.869%); 3609/52761 objects misplaced (6.840%); 9.3 MiB/s, 187 objects/s recovering 2024-05-15T13:28:55.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:55 smithi120 ceph-mon[43961]: 1.d scrub ok 2024-05-15T13:28:55.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:55 smithi120 ceph-mon[43961]: pgmap v2444: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 525 op/s; 2569/52761 objects degraded (4.869%); 3609/52761 objects misplaced (6.840%); 9.3 MiB/s, 187 objects/s recovering 2024-05-15T13:28:56.285 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:28:56.285 INFO:teuthology.orchestra.run.smithi120.stdout: "size": 20, 2024-05-15T13:28:56.285 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 600, 2024-05-15T13:28:56.285 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [ 2024-05-15T13:28:56.285 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.285 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:181048 9.b 9:d0cdfe0a:::benchmark_data_smithi177_159919_object60805:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e920)", 2024-05-15T13:28:56.285 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:28.859428+0000", 2024-05-15T13:28:56.285 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 87.421857187000001, 2024-05-15T13:28:56.285 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.68840314300000005, 2024-05-15T13:28:56.286 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:56.286 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:28:56.286 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:28:56.286 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:28:56.286 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:28:56.286 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 181048 2024-05-15T13:28:56.286 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.286 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:56.286 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.286 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:56.286 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.859428+0000", 2024-05-15T13:28:56.286 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.286 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.287 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.287 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:56.287 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.859428+0000", 2024-05-15T13:28:56.287 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.287 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.287 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.287 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:56.287 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.859430+0000", 2024-05-15T13:28:56.287 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6220000000000001e-06 2024-05-15T13:28:56.287 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.287 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.287 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:56.287 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.859436+0000", 2024-05-15T13:28:56.287 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7660000000000004e-06 2024-05-15T13:28:56.288 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.288 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.288 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:56.288 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.859450+0000", 2024-05-15T13:28:56.288 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3993e-05 2024-05-15T13:28:56.288 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.288 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.288 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:56.288 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.859468+0000", 2024-05-15T13:28:56.288 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.8607999999999999e-05 2024-05-15T13:28:56.288 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.288 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.288 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.289 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.867621+0000", 2024-05-15T13:28:56.289 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0081525150000000008 2024-05-15T13:28:56.289 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.289 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.289 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:28:56.289 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.867779+0000", 2024-05-15T13:28:56.289 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015762799999999999 2024-05-15T13:28:56.289 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.290 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.290 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,9,10", 2024-05-15T13:28:56.290 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.867918+0000", 2024-05-15T13:28:56.290 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013909 2024-05-15T13:28:56.290 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.290 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.290 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.290 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.881414+0000", 2024-05-15T13:28:56.290 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.013496375 2024-05-15T13:28:56.290 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.290 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.290 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:28:56.290 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.881424+0000", 2024-05-15T13:28:56.291 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.4669999999999993e-06 2024-05-15T13:28:56.291 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.291 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.291 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.291 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.536966+0000", 2024-05-15T13:28:56.291 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.65554222100000004 2024-05-15T13:28:56.291 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.291 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.291 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for missing object", 2024-05-15T13:28:56.291 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.537071+0000", 2024-05-15T13:28:56.291 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000105296 2024-05-15T13:28:56.291 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.291 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.291 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.292 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.547761+0000", 2024-05-15T13:28:56.292 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.010689515 2024-05-15T13:28:56.292 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.292 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.292 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:28:56.292 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.547832+0000", 2024-05-15T13:28:56.292 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.1046999999999999e-05 2024-05-15T13:28:56.292 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.292 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:56.292 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.292 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.292 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.292 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:181063 9.b 9:d85e9e9e:::benchmark_data_smithi177_159919_object60820:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e921)", 2024-05-15T13:28:56.292 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:28.894643+0000", 2024-05-15T13:28:56.292 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 87.386642992999995, 2024-05-15T13:28:56.293 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.65333332399999999, 2024-05-15T13:28:56.293 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:56.293 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:28:56.293 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:28:56.293 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:28:56.293 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:28:56.293 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 181063 2024-05-15T13:28:56.293 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.293 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:56.293 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.293 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:56.293 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.894643+0000", 2024-05-15T13:28:56.293 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.294 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.294 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.294 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:56.294 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.894643+0000", 2024-05-15T13:28:56.294 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.294 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.294 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.294 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:56.294 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.894645+0000", 2024-05-15T13:28:56.294 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0640000000000001e-06 2024-05-15T13:28:56.294 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.294 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.294 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:56.294 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.894650+0000", 2024-05-15T13:28:56.295 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.677e-06 2024-05-15T13:28:56.295 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.295 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.295 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:56.295 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.894661+0000", 2024-05-15T13:28:56.295 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0811e-05 2024-05-15T13:28:56.295 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.295 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.295 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:56.295 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.894668+0000", 2024-05-15T13:28:56.295 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6390000000000003e-06 2024-05-15T13:28:56.295 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.295 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.295 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.295 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.897704+0000", 2024-05-15T13:28:56.295 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0030358899999999999 2024-05-15T13:28:56.296 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.296 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.296 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:28:56.296 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.897713+0000", 2024-05-15T13:28:56.296 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.5899999999999997e-06 2024-05-15T13:28:56.296 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.296 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.296 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.296 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.537376+0000", 2024-05-15T13:28:56.296 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.63966236200000004 2024-05-15T13:28:56.296 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.296 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.296 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:28:56.296 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.537738+0000", 2024-05-15T13:28:56.296 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00036273399999999997 2024-05-15T13:28:56.296 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.297 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.297 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,8,10", 2024-05-15T13:28:56.297 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.537883+0000", 2024-05-15T13:28:56.297 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014439499999999999 2024-05-15T13:28:56.297 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.297 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.297 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:28:56.297 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.547365+0000", 2024-05-15T13:28:56.297 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0094823230000000008 2024-05-15T13:28:56.297 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.297 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.297 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:28:56.297 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.547410+0000", 2024-05-15T13:28:56.297 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4571999999999999e-05 2024-05-15T13:28:56.297 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.297 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.298 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:28:56.298 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.547826+0000", 2024-05-15T13:28:56.298 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00041648199999999997 2024-05-15T13:28:56.298 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.298 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.298 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:28:56.298 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.547866+0000", 2024-05-15T13:28:56.298 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.9575e-05 2024-05-15T13:28:56.298 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.298 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.298 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:28:56.298 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.547901+0000", 2024-05-15T13:28:56.299 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.4785e-05 2024-05-15T13:28:56.299 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.299 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.299 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:28:56.299 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.547976+0000", 2024-05-15T13:28:56.299 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.5425000000000004e-05 2024-05-15T13:28:56.299 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.299 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:56.299 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.299 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.300 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.300 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:181080 9.b 9:dc319383:::benchmark_data_smithi177_159919_object60837:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e921)", 2024-05-15T13:28:56.300 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:28.937547+0000", 2024-05-15T13:28:56.300 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 87.343738324, 2024-05-15T13:28:56.300 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.61993279999999995, 2024-05-15T13:28:56.300 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:56.300 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:28:56.300 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:28:56.300 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:28:56.300 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:28:56.300 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 181080 2024-05-15T13:28:56.300 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.300 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:56.300 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.300 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:56.301 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.937547+0000", 2024-05-15T13:28:56.301 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.301 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.301 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.301 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:56.301 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.937547+0000", 2024-05-15T13:28:56.301 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.301 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.301 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.301 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:56.301 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.937548+0000", 2024-05-15T13:28:56.301 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.7699999999999992e-07 2024-05-15T13:28:56.301 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.301 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.301 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:56.302 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.937553+0000", 2024-05-15T13:28:56.302 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.3159999999999998e-06 2024-05-15T13:28:56.302 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.302 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.302 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:56.302 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.937564+0000", 2024-05-15T13:28:56.302 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1155000000000001e-05 2024-05-15T13:28:56.302 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.302 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.302 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:56.302 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.937571+0000", 2024-05-15T13:28:56.302 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.4449999999999999e-06 2024-05-15T13:28:56.302 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.302 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.302 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.303 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.937628+0000", 2024-05-15T13:28:56.303 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6663000000000003e-05 2024-05-15T13:28:56.303 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.303 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.303 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:28:56.303 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.937636+0000", 2024-05-15T13:28:56.303 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.5690000000000005e-06 2024-05-15T13:28:56.303 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.303 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.303 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.303 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.542525+0000", 2024-05-15T13:28:56.303 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.60488827099999998 2024-05-15T13:28:56.303 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.303 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.303 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:28:56.304 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.542682+0000", 2024-05-15T13:28:56.304 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000157131 2024-05-15T13:28:56.304 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.304 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.304 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,8,10", 2024-05-15T13:28:56.304 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.542861+0000", 2024-05-15T13:28:56.304 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00017874699999999999 2024-05-15T13:28:56.304 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.304 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.304 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:28:56.304 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.547464+0000", 2024-05-15T13:28:56.304 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0046034320000000002 2024-05-15T13:28:56.305 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.305 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.305 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:28:56.305 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.547830+0000", 2024-05-15T13:28:56.305 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00036643500000000002 2024-05-15T13:28:56.305 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.305 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.305 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:28:56.305 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.548011+0000", 2024-05-15T13:28:56.305 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00018052500000000001 2024-05-15T13:28:56.305 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.305 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.305 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:28:56.305 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.557347+0000", 2024-05-15T13:28:56.306 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0093356250000000002 2024-05-15T13:28:56.306 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.306 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.306 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:28:56.306 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.557387+0000", 2024-05-15T13:28:56.306 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0429999999999997e-05 2024-05-15T13:28:56.306 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.306 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.306 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:28:56.306 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.557480+0000", 2024-05-15T13:28:56.306 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.3078999999999994e-05 2024-05-15T13:28:56.306 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.307 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:56.307 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.307 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.307 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.307 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:181086 9.b 9:dfa52a23:::benchmark_data_smithi177_159919_object60843:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e921)", 2024-05-15T13:28:56.307 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:28.949663+0000", 2024-05-15T13:28:56.307 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 87.331622659999994, 2024-05-15T13:28:56.307 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.60996531099999995, 2024-05-15T13:28:56.307 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:56.307 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:28:56.307 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:28:56.307 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:28:56.307 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:28:56.308 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 181086 2024-05-15T13:28:56.308 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.308 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:56.308 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.308 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:56.308 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.949663+0000", 2024-05-15T13:28:56.308 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.308 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.308 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.308 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:56.308 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.949663+0000", 2024-05-15T13:28:56.308 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.308 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.308 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.308 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:56.309 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.949663+0000", 2024-05-15T13:28:56.309 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.5200000000000002e-07 2024-05-15T13:28:56.309 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.309 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.309 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:56.309 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.949667+0000", 2024-05-15T13:28:56.309 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.2619999999999999e-06 2024-05-15T13:28:56.309 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.309 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.309 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:56.309 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.949673+0000", 2024-05-15T13:28:56.309 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1519999999999997e-06 2024-05-15T13:28:56.309 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.309 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.309 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:56.309 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.949678+0000", 2024-05-15T13:28:56.310 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.344e-06 2024-05-15T13:28:56.310 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.310 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.310 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.310 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.949746+0000", 2024-05-15T13:28:56.310 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8044000000000001e-05 2024-05-15T13:28:56.310 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.310 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.310 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:28:56.310 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.949763+0000", 2024-05-15T13:28:56.311 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6529e-05 2024-05-15T13:28:56.311 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.311 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.311 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.311 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.548038+0000", 2024-05-15T13:28:56.311 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.59827491399999999 2024-05-15T13:28:56.311 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.311 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.311 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:28:56.311 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.548115+0000", 2024-05-15T13:28:56.311 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.6932000000000001e-05 2024-05-15T13:28:56.311 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.311 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.312 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,8,10", 2024-05-15T13:28:56.312 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.548200+0000", 2024-05-15T13:28:56.312 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.5377999999999995e-05 2024-05-15T13:28:56.312 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.312 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.312 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:28:56.312 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.557284+0000", 2024-05-15T13:28:56.312 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0090843790000000001 2024-05-15T13:28:56.312 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.312 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.312 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:28:56.312 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.557323+0000", 2024-05-15T13:28:56.313 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.8680999999999998e-05 2024-05-15T13:28:56.313 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.313 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.313 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:28:56.313 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.557541+0000", 2024-05-15T13:28:56.313 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00021785500000000001 2024-05-15T13:28:56.313 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.313 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.313 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:28:56.313 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.559439+0000", 2024-05-15T13:28:56.313 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001898521 2024-05-15T13:28:56.313 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.313 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.313 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:28:56.313 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.559503+0000", 2024-05-15T13:28:56.314 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3559999999999995e-05 2024-05-15T13:28:56.314 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.314 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.314 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:28:56.314 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.559628+0000", 2024-05-15T13:28:56.314 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000125308 2024-05-15T13:28:56.314 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.314 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:56.314 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.314 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.314 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.314 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:181112 9.b 9:da199c20:::benchmark_data_smithi177_159919_object60869:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e921)", 2024-05-15T13:28:56.314 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:29.074273+0000", 2024-05-15T13:28:56.314 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 87.207012814999999, 2024-05-15T13:28:56.315 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.48557814399999999, 2024-05-15T13:28:56.315 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:56.315 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:28:56.315 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:28:56.315 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:28:56.315 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:28:56.316 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 181112 2024-05-15T13:28:56.316 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.316 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:56.316 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.316 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:56.316 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.074273+0000", 2024-05-15T13:28:56.316 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.316 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.316 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.316 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:56.316 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.074273+0000", 2024-05-15T13:28:56.316 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.316 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.316 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.316 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:56.316 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.074275+0000", 2024-05-15T13:28:56.317 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9439999999999999e-06 2024-05-15T13:28:56.317 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.317 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.317 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:56.317 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.074282+0000", 2024-05-15T13:28:56.317 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3599999999999998e-06 2024-05-15T13:28:56.317 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.317 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.317 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:56.317 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.074296+0000", 2024-05-15T13:28:56.317 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4286e-05 2024-05-15T13:28:56.317 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.317 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.317 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:56.317 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.074306+0000", 2024-05-15T13:28:56.317 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.329e-06 2024-05-15T13:28:56.318 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.318 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.318 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.318 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.074365+0000", 2024-05-15T13:28:56.318 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8896e-05 2024-05-15T13:28:56.318 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.318 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.318 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:28:56.318 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.074372+0000", 2024-05-15T13:28:56.318 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.554e-06 2024-05-15T13:28:56.318 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.318 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.318 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.319 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.552401+0000", 2024-05-15T13:28:56.319 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.47802923899999999 2024-05-15T13:28:56.319 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.319 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.319 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:28:56.319 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.552516+0000", 2024-05-15T13:28:56.319 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000114955 2024-05-15T13:28:56.319 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.319 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.320 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 1,8,10", 2024-05-15T13:28:56.320 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.552662+0000", 2024-05-15T13:28:56.320 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000145465 2024-05-15T13:28:56.320 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.320 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.320 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:28:56.321 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.557508+0000", 2024-05-15T13:28:56.321 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0048459699999999998 2024-05-15T13:28:56.321 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.321 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.321 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:28:56.321 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.557532+0000", 2024-05-15T13:28:56.321 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.3799999999999999e-05 2024-05-15T13:28:56.322 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.322 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.322 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:28:56.322 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.557545+0000", 2024-05-15T13:28:56.322 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3855e-05 2024-05-15T13:28:56.322 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.323 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.323 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:28:56.323 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.559673+0000", 2024-05-15T13:28:56.323 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.002127112 2024-05-15T13:28:56.323 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.323 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.323 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:28:56.324 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.559720+0000", 2024-05-15T13:28:56.324 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7657999999999997e-05 2024-05-15T13:28:56.324 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.324 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.324 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:28:56.324 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.559851+0000", 2024-05-15T13:28:56.325 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013072100000000001 2024-05-15T13:28:56.325 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.325 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:56.325 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.325 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.325 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.325 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:181935 9.5 9:adaba10a:::benchmark_data_smithi177_159919_object61692:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e925)", 2024-05-15T13:28:56.326 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:32.420153+0000", 2024-05-15T13:28:56.326 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 83.861132463000004, 2024-05-15T13:28:56.326 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.300162441, 2024-05-15T13:28:56.326 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:56.326 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:28:56.326 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:28:56.327 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:28:56.327 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:28:56.327 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 181935 2024-05-15T13:28:56.327 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.327 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:56.327 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.327 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:56.328 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:32.420153+0000", 2024-05-15T13:28:56.328 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.328 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.328 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.328 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:56.328 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:32.420153+0000", 2024-05-15T13:28:56.329 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.329 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.329 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.329 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:56.329 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:32.420155+0000", 2024-05-15T13:28:56.329 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5799999999999999e-06 2024-05-15T13:28:56.330 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.330 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.330 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:56.330 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:32.420159+0000", 2024-05-15T13:28:56.330 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0439999999999997e-06 2024-05-15T13:28:56.330 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.330 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.331 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:56.331 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:32.420168+0000", 2024-05-15T13:28:56.331 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.6849999999999996e-06 2024-05-15T13:28:56.331 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.331 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.332 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:56.332 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:32.420175+0000", 2024-05-15T13:28:56.332 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1850000000000001e-06 2024-05-15T13:28:56.332 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.332 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.332 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.333 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:32.424676+0000", 2024-05-15T13:28:56.333 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0045018319999999999 2024-05-15T13:28:56.333 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.333 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.333 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:28:56.333 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:32.424848+0000", 2024-05-15T13:28:56.334 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00017137599999999999 2024-05-15T13:28:56.334 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.334 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.334 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 3,7,11", 2024-05-15T13:28:56.334 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:32.426463+0000", 2024-05-15T13:28:56.334 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0016154380000000001 2024-05-15T13:28:56.335 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.335 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.335 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:28:56.335 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:32.436130+0000", 2024-05-15T13:28:56.335 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0096663840000000001 2024-05-15T13:28:56.335 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.336 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.336 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:28:56.336 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:32.447996+0000", 2024-05-15T13:28:56.336 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.011865872 2024-05-15T13:28:56.336 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.336 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.337 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:28:56.337 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:32.453326+0000", 2024-05-15T13:28:56.337 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0053300650000000001 2024-05-15T13:28:56.337 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.337 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.337 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:28:56.338 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:32.720113+0000", 2024-05-15T13:28:56.338 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.26678730699999997 2024-05-15T13:28:56.338 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.338 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.338 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:28:56.339 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:32.720180+0000", 2024-05-15T13:28:56.339 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.6707999999999999e-05 2024-05-15T13:28:56.339 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.339 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.339 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:28:56.339 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:32.720316+0000", 2024-05-15T13:28:56.340 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013596499999999999 2024-05-15T13:28:56.340 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.340 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:56.340 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.340 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.340 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.340 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:193547 9.5 9:a23f45c2:::benchmark_data_smithi177_159919_object73304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e935)", 2024-05-15T13:28:56.340 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:57.397207+0000", 2024-05-15T13:28:56.340 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 58.884078916, 2024-05-15T13:28:56.341 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0372574269999999, 2024-05-15T13:28:56.341 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:56.341 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:28:56.341 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:28:56.341 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:28:56.341 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:28:56.341 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 193547 2024-05-15T13:28:56.341 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.341 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:56.342 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.342 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:56.342 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.397207+0000", 2024-05-15T13:28:56.342 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.342 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.342 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.342 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:56.342 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.397207+0000", 2024-05-15T13:28:56.343 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.343 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.343 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.343 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:56.343 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.397209+0000", 2024-05-15T13:28:56.343 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.804e-06 2024-05-15T13:28:56.343 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.343 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.344 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:56.344 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.397216+0000", 2024-05-15T13:28:56.344 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8809999999999996e-06 2024-05-15T13:28:56.344 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.344 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.344 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:56.344 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.397233+0000", 2024-05-15T13:28:56.344 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.6174000000000001e-05 2024-05-15T13:28:56.344 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.345 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.345 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:56.345 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.397243+0000", 2024-05-15T13:28:56.345 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0356e-05 2024-05-15T13:28:56.345 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.345 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.345 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.345 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.403902+0000", 2024-05-15T13:28:56.345 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0066590929999999996 2024-05-15T13:28:56.345 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.345 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.345 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:28:56.345 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.404038+0000", 2024-05-15T13:28:56.345 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000136341 2024-05-15T13:28:56.345 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.346 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.346 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 3,7,11", 2024-05-15T13:28:56.346 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.404168+0000", 2024-05-15T13:28:56.346 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000130068 2024-05-15T13:28:56.346 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.346 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.346 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.346 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.431494+0000", 2024-05-15T13:28:56.346 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.027325461999999998 2024-05-15T13:28:56.346 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.346 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.347 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:28:56.347 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.431504+0000", 2024-05-15T13:28:56.347 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.9389999999999992e-06 2024-05-15T13:28:56.347 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.347 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.347 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.347 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.434449+0000", 2024-05-15T13:28:56.347 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0029454609999999 2024-05-15T13:28:56.347 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.347 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.347 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:28:56.347 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.434464+0000", 2024-05-15T13:28:56.347 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4848000000000001e-05 2024-05-15T13:28:56.347 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.347 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:56.348 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.348 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.348 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.348 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:193550 9.5 9.3a8ec745 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e935)", 2024-05-15T13:28:56.348 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:57.404905+0000", 2024-05-15T13:28:56.348 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 58.876380556000001, 2024-05-15T13:28:56.348 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.029592853, 2024-05-15T13:28:56.348 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:56.348 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:28:56.348 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:28:56.348 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:28:56.348 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:28:56.348 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 193550 2024-05-15T13:28:56.348 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.348 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:56.348 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.349 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:56.349 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.404905+0000", 2024-05-15T13:28:56.349 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.349 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.349 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.349 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:56.349 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.404905+0000", 2024-05-15T13:28:56.349 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.349 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.349 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.349 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:56.349 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.404906+0000", 2024-05-15T13:28:56.349 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.257e-06 2024-05-15T13:28:56.350 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.350 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.350 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:56.350 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.404912+0000", 2024-05-15T13:28:56.350 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.4349999999999996e-06 2024-05-15T13:28:56.350 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.350 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.350 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:56.350 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.404922+0000", 2024-05-15T13:28:56.350 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.009e-05 2024-05-15T13:28:56.350 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.350 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.350 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:56.351 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.404930+0000", 2024-05-15T13:28:56.351 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8229999999999995e-06 2024-05-15T13:28:56.351 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.351 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.351 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.351 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.438959+0000", 2024-05-15T13:28:56.351 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.034028834000000001 2024-05-15T13:28:56.351 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.351 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.351 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:28:56.351 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.438966+0000", 2024-05-15T13:28:56.351 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.841e-06 2024-05-15T13:28:56.351 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.351 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.351 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.352 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.434490+0000", 2024-05-15T13:28:56.352 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.995523987 2024-05-15T13:28:56.352 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.352 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.352 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:28:56.352 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.434498+0000", 2024-05-15T13:28:56.352 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.5859999999999999e-06 2024-05-15T13:28:56.352 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.352 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:56.352 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.352 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.352 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.352 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:193555 9.5 9.bf0ee1c5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e936)", 2024-05-15T13:28:56.352 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:57.416863+0000", 2024-05-15T13:28:56.353 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 58.864422826000002, 2024-05-15T13:28:56.353 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0176650439999999, 2024-05-15T13:28:56.353 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:56.353 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:28:56.353 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:28:56.353 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:28:56.353 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:28:56.353 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 193555 2024-05-15T13:28:56.353 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.353 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:56.353 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.353 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:56.354 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.416863+0000", 2024-05-15T13:28:56.354 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.354 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.354 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.354 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:56.354 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.416863+0000", 2024-05-15T13:28:56.354 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.354 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.354 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.354 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:56.354 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.416864+0000", 2024-05-15T13:28:56.354 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4869999999999999e-06 2024-05-15T13:28:56.355 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.355 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.355 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:56.355 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.416870+0000", 2024-05-15T13:28:56.355 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8680000000000004e-06 2024-05-15T13:28:56.355 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.355 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.355 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:56.355 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.416882+0000", 2024-05-15T13:28:56.355 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1703e-05 2024-05-15T13:28:56.355 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.355 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.355 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.416890+0000", 2024-05-15T13:28:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0260000000000007e-06 2024-05-15T13:28:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.438987+0000", 2024-05-15T13:28:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.022097535000000001 2024-05-15T13:28:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:28:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.438994+0000", 2024-05-15T13:28:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.5289999999999997e-06 2024-05-15T13:28:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.434520+0000", 2024-05-15T13:28:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.99552624999999995 2024-05-15T13:28:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:28:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.434528+0000", 2024-05-15T13:28:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.6459999999999993e-06 2024-05-15T13:28:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:193558 9.f 9.532420f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e936)", 2024-05-15T13:28:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:57.416905+0000", 2024-05-15T13:28:56.358 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 58.864380576999999, 2024-05-15T13:28:56.358 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0857051339999999, 2024-05-15T13:28:56.358 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:56.358 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:28:56.358 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:28:56.358 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:28:56.358 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:28:56.358 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 193558 2024-05-15T13:28:56.358 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.358 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:56.358 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.358 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:56.358 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.416905+0000", 2024-05-15T13:28:56.359 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.359 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.359 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.359 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:56.359 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.416905+0000", 2024-05-15T13:28:56.359 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.359 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.359 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.359 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:56.359 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.416906+0000", 2024-05-15T13:28:56.359 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9399999999999995e-07 2024-05-15T13:28:56.359 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.360 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.360 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:56.360 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.416909+0000", 2024-05-15T13:28:56.360 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.1939999999999998e-06 2024-05-15T13:28:56.360 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.360 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.360 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:56.360 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.416915+0000", 2024-05-15T13:28:56.360 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.251e-06 2024-05-15T13:28:56.360 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.360 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.360 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:56.360 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.416921+0000", 2024-05-15T13:28:56.360 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9370000000000001e-06 2024-05-15T13:28:56.361 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.361 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.361 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.361 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.417010+0000", 2024-05-15T13:28:56.361 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8731000000000001e-05 2024-05-15T13:28:56.361 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.361 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.361 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op must wait for map", 2024-05-15T13:28:56.361 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.417018+0000", 2024-05-15T13:28:56.361 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2740000000000007e-06 2024-05-15T13:28:56.361 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.361 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.361 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.361 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.436397+0000", 2024-05-15T13:28:56.362 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.019378827000000001 2024-05-15T13:28:56.362 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.362 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.362 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:28:56.362 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.436408+0000", 2024-05-15T13:28:56.362 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.129e-05 2024-05-15T13:28:56.362 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.362 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.362 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.362 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.502599+0000", 2024-05-15T13:28:56.362 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.066191356 2024-05-15T13:28:56.362 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.362 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.363 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:28:56.363 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.502610+0000", 2024-05-15T13:28:56.363 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.078e-05 2024-05-15T13:28:56.363 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.363 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:56.363 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.363 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.363 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.363 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:193564 9.5 9.d26c5975 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e936)", 2024-05-15T13:28:56.363 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:57.436474+0000", 2024-05-15T13:28:56.363 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 58.844811313999998, 2024-05-15T13:28:56.363 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.99809523700000002, 2024-05-15T13:28:56.363 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:56.363 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:28:56.364 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:28:56.364 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:28:56.364 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:28:56.364 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 193564 2024-05-15T13:28:56.364 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.364 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:56.364 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.364 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:56.364 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.436474+0000", 2024-05-15T13:28:56.364 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.364 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.364 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.365 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:56.365 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.436474+0000", 2024-05-15T13:28:56.365 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.365 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.365 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.365 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:56.366 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.436476+0000", 2024-05-15T13:28:56.366 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2699999999999999e-06 2024-05-15T13:28:56.366 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.366 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.366 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:56.367 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.436481+0000", 2024-05-15T13:28:56.367 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.6420000000000004e-06 2024-05-15T13:28:56.367 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.367 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.367 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:56.367 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.436491+0000", 2024-05-15T13:28:56.367 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.7799999999999995e-06 2024-05-15T13:28:56.368 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.368 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.368 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:56.368 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.436498+0000", 2024-05-15T13:28:56.368 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8789999999999997e-06 2024-05-15T13:28:56.368 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.369 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.369 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.369 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.439013+0000", 2024-05-15T13:28:56.369 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00251534 2024-05-15T13:28:56.369 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.369 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.370 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:28:56.370 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.439019+0000", 2024-05-15T13:28:56.370 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3070000000000002e-06 2024-05-15T13:28:56.370 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.370 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.370 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.371 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.434562+0000", 2024-05-15T13:28:56.371 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.99554344400000006 2024-05-15T13:28:56.371 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.371 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.371 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:28:56.371 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.434570+0000", 2024-05-15T13:28:56.371 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.5750000000000003e-06 2024-05-15T13:28:56.372 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.372 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:56.372 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.372 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.372 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.372 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:193547 9.5 9.43a2fc45 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e936)", 2024-05-15T13:28:56.373 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:57.454389+0000", 2024-05-15T13:28:56.373 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 58.826896795000003, 2024-05-15T13:28:56.373 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.98021955100000002, 2024-05-15T13:28:56.373 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:56.373 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:28:56.374 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:28:56.374 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:28:56.374 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:28:56.374 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 193547 2024-05-15T13:28:56.374 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.374 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:56.375 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.375 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:56.375 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454389+0000", 2024-05-15T13:28:56.375 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.375 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.375 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.376 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:56.376 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454389+0000", 2024-05-15T13:28:56.376 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.376 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.376 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.376 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:56.377 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454390+0000", 2024-05-15T13:28:56.377 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3319999999999999e-06 2024-05-15T13:28:56.377 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.377 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.377 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:56.377 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454395+0000", 2024-05-15T13:28:56.378 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6369999999999998e-06 2024-05-15T13:28:56.378 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.378 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.378 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:56.378 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454404+0000", 2024-05-15T13:28:56.379 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7450000000000001e-06 2024-05-15T13:28:56.379 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.379 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.379 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:56.379 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454410+0000", 2024-05-15T13:28:56.379 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.195e-06 2024-05-15T13:28:56.380 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.380 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.380 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.380 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454710+0000", 2024-05-15T13:28:56.380 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00030001300000000002 2024-05-15T13:28:56.380 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.381 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.381 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:28:56.381 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454719+0000", 2024-05-15T13:28:56.381 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.7249999999999992e-06 2024-05-15T13:28:56.381 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.381 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.382 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.382 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.434603+0000", 2024-05-15T13:28:56.382 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.97988318600000002 2024-05-15T13:28:56.382 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.382 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.383 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:28:56.383 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.434608+0000", 2024-05-15T13:28:56.383 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7180000000000001e-06 2024-05-15T13:28:56.383 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.383 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:56.383 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.384 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.384 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.384 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:193550 9.5 9.3a8ec745 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e936)", 2024-05-15T13:28:56.384 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:57.454427+0000", 2024-05-15T13:28:56.384 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 58.826858952000002, 2024-05-15T13:28:56.384 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.98020795999999999, 2024-05-15T13:28:56.385 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:56.385 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:28:56.385 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:28:56.385 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:28:56.385 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:28:56.385 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 193550 2024-05-15T13:28:56.385 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.385 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:56.385 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.385 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:56.386 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454427+0000", 2024-05-15T13:28:56.386 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.386 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.386 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.386 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:56.386 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454427+0000", 2024-05-15T13:28:56.386 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.386 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.386 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.386 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:56.386 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454427+0000", 2024-05-15T13:28:56.386 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.68e-07 2024-05-15T13:28:56.386 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.387 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.387 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:56.387 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454430+0000", 2024-05-15T13:28:56.387 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.7930000000000002e-06 2024-05-15T13:28:56.387 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.387 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.387 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:56.387 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454437+0000", 2024-05-15T13:28:56.387 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.345e-06 2024-05-15T13:28:56.387 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.387 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.387 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:56.387 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454443+0000", 2024-05-15T13:28:56.387 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.7440000000000004e-06 2024-05-15T13:28:56.387 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.388 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.388 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.388 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454752+0000", 2024-05-15T13:28:56.388 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00030848099999999999 2024-05-15T13:28:56.388 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.388 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.388 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:28:56.388 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454758+0000", 2024-05-15T13:28:56.388 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3500000000000002e-06 2024-05-15T13:28:56.388 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.388 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.388 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.388 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.434630+0000", 2024-05-15T13:28:56.388 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.979872411 2024-05-15T13:28:56.389 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.389 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.389 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:28:56.389 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.434635+0000", 2024-05-15T13:28:56.389 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.2679999999999996e-06 2024-05-15T13:28:56.389 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.389 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:56.389 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.389 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.389 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.389 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:193555 9.5 9.bf0ee1c5 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e936)", 2024-05-15T13:28:56.389 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:57.454477+0000", 2024-05-15T13:28:56.389 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 58.826808778, 2024-05-15T13:28:56.389 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.98018095599999999, 2024-05-15T13:28:56.390 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:56.390 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:28:56.390 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:28:56.390 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:28:56.390 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:28:56.390 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 193555 2024-05-15T13:28:56.390 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.390 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:56.390 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.390 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:56.390 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454477+0000", 2024-05-15T13:28:56.390 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.390 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.391 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.391 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:56.391 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454477+0000", 2024-05-15T13:28:56.391 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.391 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.391 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.391 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:56.391 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454477+0000", 2024-05-15T13:28:56.391 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0200000000000003e-07 2024-05-15T13:28:56.391 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.391 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.391 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:56.391 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454480+0000", 2024-05-15T13:28:56.391 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.604e-06 2024-05-15T13:28:56.391 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.391 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.392 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:56.392 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454486+0000", 2024-05-15T13:28:56.392 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.5470000000000002e-06 2024-05-15T13:28:56.392 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.392 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.392 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:56.392 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454492+0000", 2024-05-15T13:28:56.392 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9020000000000004e-06 2024-05-15T13:28:56.392 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.392 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.392 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.392 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454778+0000", 2024-05-15T13:28:56.392 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00028617099999999999 2024-05-15T13:28:56.392 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.392 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.392 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:28:56.393 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454785+0000", 2024-05-15T13:28:56.393 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1530000000000002e-06 2024-05-15T13:28:56.393 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.393 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.393 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.393 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.434653+0000", 2024-05-15T13:28:56.393 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.97986870299999995 2024-05-15T13:28:56.393 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.393 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.393 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:28:56.393 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.434658+0000", 2024-05-15T13:28:56.393 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4739999999999999e-06 2024-05-15T13:28:56.393 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.393 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:56.393 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.394 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.394 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.394 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:193558 9.f 9:f0424ca0:::benchmark_data_smithi177_159919_object73315:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e936)", 2024-05-15T13:28:56.394 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:57.454499+0000", 2024-05-15T13:28:56.394 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 58.826786798000001, 2024-05-15T13:28:56.394 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1417647049999999, 2024-05-15T13:28:56.394 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:56.394 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:28:56.394 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:28:56.394 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:28:56.394 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:28:56.395 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 193558 2024-05-15T13:28:56.395 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.395 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:56.395 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.395 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:56.395 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454499+0000", 2024-05-15T13:28:56.395 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.395 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.395 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.395 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:56.395 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454499+0000", 2024-05-15T13:28:56.395 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.395 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.395 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.395 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:56.396 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454499+0000", 2024-05-15T13:28:56.396 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.9900000000000001e-07 2024-05-15T13:28:56.396 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.396 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.396 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:56.396 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454502+0000", 2024-05-15T13:28:56.396 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.3740000000000001e-06 2024-05-15T13:28:56.396 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.396 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.396 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:56.396 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454508+0000", 2024-05-15T13:28:56.396 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.4189999999999999e-06 2024-05-15T13:28:56.396 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.396 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.397 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:56.397 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454513+0000", 2024-05-15T13:28:56.397 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0769999999999998e-06 2024-05-15T13:28:56.397 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.397 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.397 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.397 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454712+0000", 2024-05-15T13:28:56.397 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00019849599999999999 2024-05-15T13:28:56.397 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.397 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.397 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:28:56.397 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454720+0000", 2024-05-15T13:28:56.397 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1210000000000007e-06 2024-05-15T13:28:56.397 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.397 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.397 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.398 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.576531+0000", 2024-05-15T13:28:56.398 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.121810821 2024-05-15T13:28:56.398 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.398 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.398 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:28:56.398 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.576656+0000", 2024-05-15T13:28:56.398 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000125838 2024-05-15T13:28:56.398 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.398 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.398 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 9,11", 2024-05-15T13:28:56.398 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.576796+0000", 2024-05-15T13:28:56.398 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013918900000000001 2024-05-15T13:28:56.398 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.399 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.399 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:28:56.399 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.586278+0000", 2024-05-15T13:28:56.399 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0094820149999999999 2024-05-15T13:28:56.399 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.399 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.399 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:28:56.399 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.591345+0000", 2024-05-15T13:28:56.399 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0050678119999999997 2024-05-15T13:28:56.399 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.399 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.399 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:28:56.399 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.593751+0000", 2024-05-15T13:28:56.399 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0024057940000000002 2024-05-15T13:28:56.400 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.400 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.400 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:28:56.400 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.593823+0000", 2024-05-15T13:28:56.400 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.1614999999999998e-05 2024-05-15T13:28:56.400 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.400 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.400 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:28:56.400 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.596264+0000", 2024-05-15T13:28:56.400 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0024407349999999999 2024-05-15T13:28:56.400 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.400 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:56.400 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.400 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.400 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.401 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:193564 9.5 9.d26c5975 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e936)", 2024-05-15T13:28:56.401 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:57.454569+0000", 2024-05-15T13:28:56.401 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 58.826716537999999, 2024-05-15T13:28:56.401 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.98010632200000003, 2024-05-15T13:28:56.401 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:56.401 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:28:56.401 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:28:56.401 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:28:56.401 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:28:56.401 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 193564 2024-05-15T13:28:56.401 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.401 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:56.401 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.401 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:56.401 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454569+0000", 2024-05-15T13:28:56.402 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.402 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.402 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.402 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:56.402 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454569+0000", 2024-05-15T13:28:56.402 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.402 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.402 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.402 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:56.402 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454569+0000", 2024-05-15T13:28:56.402 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.65e-07 2024-05-15T13:28:56.402 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.402 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.403 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:56.403 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454572+0000", 2024-05-15T13:28:56.403 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1749999999999999e-06 2024-05-15T13:28:56.403 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.403 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.403 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:56.403 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454578+0000", 2024-05-15T13:28:56.403 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.669e-06 2024-05-15T13:28:56.403 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.403 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.403 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:56.403 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454584+0000", 2024-05-15T13:28:56.403 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.4149999999999998e-06 2024-05-15T13:28:56.403 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.404 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.404 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.404 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454812+0000", 2024-05-15T13:28:56.404 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000228626 2024-05-15T13:28:56.404 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.404 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.404 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:28:56.404 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454819+0000", 2024-05-15T13:28:56.404 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7460000000000003e-06 2024-05-15T13:28:56.404 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.404 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.405 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.405 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.434672+0000", 2024-05-15T13:28:56.405 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.97985241899999997 2024-05-15T13:28:56.405 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.405 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.405 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:28:56.405 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.434675+0000", 2024-05-15T13:28:56.405 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.907e-06 2024-05-15T13:28:56.405 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.405 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:56.405 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.405 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.405 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.405 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:193566 9.f 9:ff16f0fd:::benchmark_data_smithi177_159919_object73323:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e936)", 2024-05-15T13:28:56.405 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:57.454590+0000", 2024-05-15T13:28:56.406 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 58.826695845000003, 2024-05-15T13:28:56.406 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.143755565, 2024-05-15T13:28:56.406 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:56.406 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:28:56.406 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:28:56.406 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:28:56.406 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:28:56.406 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 193566 2024-05-15T13:28:56.406 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.406 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:56.406 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.406 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:56.406 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454590+0000", 2024-05-15T13:28:56.407 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.407 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.407 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.407 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:56.407 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454590+0000", 2024-05-15T13:28:56.407 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.407 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.407 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.407 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:56.407 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454590+0000", 2024-05-15T13:28:56.407 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5600000000000001e-07 2024-05-15T13:28:56.407 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.408 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.408 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:56.408 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454592+0000", 2024-05-15T13:28:56.408 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9939999999999999e-06 2024-05-15T13:28:56.408 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.408 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.408 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:56.408 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454598+0000", 2024-05-15T13:28:56.408 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9309999999999996e-06 2024-05-15T13:28:56.408 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.408 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.408 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:56.408 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454603+0000", 2024-05-15T13:28:56.408 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6700000000000002e-06 2024-05-15T13:28:56.409 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.409 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.409 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.409 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454750+0000", 2024-05-15T13:28:56.409 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000147224 2024-05-15T13:28:56.409 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.409 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.409 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:28:56.410 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454756+0000", 2024-05-15T13:28:56.410 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.438e-06 2024-05-15T13:28:56.410 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.410 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.410 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.410 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.581285+0000", 2024-05-15T13:28:56.410 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1265291040000001 2024-05-15T13:28:56.410 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.410 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.410 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:28:56.410 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.581386+0000", 2024-05-15T13:28:56.411 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00010084799999999999 2024-05-15T13:28:56.411 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.411 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.411 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 9,11", 2024-05-15T13:28:56.411 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.581484+0000", 2024-05-15T13:28:56.411 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.7515000000000004e-05 2024-05-15T13:28:56.411 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.411 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.411 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:28:56.411 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.591483+0000", 2024-05-15T13:28:56.411 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0099987700000000006 2024-05-15T13:28:56.411 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.411 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.412 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:28:56.412 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.596321+0000", 2024-05-15T13:28:56.412 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0048386269999999999 2024-05-15T13:28:56.412 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.412 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.412 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:28:56.412 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.596374+0000", 2024-05-15T13:28:56.412 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3189000000000002e-05 2024-05-15T13:28:56.412 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.412 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.412 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:28:56.413 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.596433+0000", 2024-05-15T13:28:56.413 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8919000000000003e-05 2024-05-15T13:28:56.413 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.413 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.413 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:28:56.413 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.598345+0000", 2024-05-15T13:28:56.413 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00191198 2024-05-15T13:28:56.413 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.413 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:56.413 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.413 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.413 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.413 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:193575 9.5 9.afe9a5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e936)", 2024-05-15T13:28:56.413 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:57.470740+0000", 2024-05-15T13:28:56.413 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 58.810545789999999, 2024-05-15T13:28:56.414 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.96395287500000004, 2024-05-15T13:28:56.414 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:56.414 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:28:56.414 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:28:56.414 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:28:56.414 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:28:56.414 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 193575 2024-05-15T13:28:56.414 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.414 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:56.414 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.414 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:56.414 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.470740+0000", 2024-05-15T13:28:56.414 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.415 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.415 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.415 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:56.415 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.470740+0000", 2024-05-15T13:28:56.415 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.415 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.415 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.415 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:56.415 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.470741+0000", 2024-05-15T13:28:56.415 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.316e-06 2024-05-15T13:28:56.415 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.415 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.415 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:56.415 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.470746+0000", 2024-05-15T13:28:56.416 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8910000000000003e-06 2024-05-15T13:28:56.416 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.416 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.416 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:56.416 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.470758+0000", 2024-05-15T13:28:56.416 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1661999999999999e-05 2024-05-15T13:28:56.416 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.416 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.416 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:56.416 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.470765+0000", 2024-05-15T13:28:56.416 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2989999999999999e-06 2024-05-15T13:28:56.416 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.416 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.416 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.416 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.470818+0000", 2024-05-15T13:28:56.416 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3276999999999998e-05 2024-05-15T13:28:56.417 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.417 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.417 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:28:56.417 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.470827+0000", 2024-05-15T13:28:56.417 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.1139999999999995e-06 2024-05-15T13:28:56.417 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.417 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.417 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.417 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.434689+0000", 2024-05-15T13:28:56.417 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.96386125899999997 2024-05-15T13:28:56.417 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.417 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.417 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:28:56.417 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.434693+0000", 2024-05-15T13:28:56.417 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.0570000000000003e-06 2024-05-15T13:28:56.418 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.418 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:56.418 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.418 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.418 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.418 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:193578 9.f 9:f81d9cf5:::benchmark_data_smithi177_159919_object73335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e936)", 2024-05-15T13:28:56.418 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:57.487834+0000", 2024-05-15T13:28:56.418 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 58.793451456, 2024-05-15T13:28:56.418 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1131486740000001, 2024-05-15T13:28:56.418 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:56.418 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:28:56.418 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:28:56.418 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:28:56.418 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:28:56.419 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 193578 2024-05-15T13:28:56.419 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.419 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:56.419 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.419 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:56.419 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.487834+0000", 2024-05-15T13:28:56.419 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.419 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.419 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.419 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:56.419 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.487834+0000", 2024-05-15T13:28:56.419 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.419 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.419 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.419 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:56.419 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.487835+0000", 2024-05-15T13:28:56.420 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3149999999999999e-06 2024-05-15T13:28:56.420 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.420 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.420 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:56.420 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.487841+0000", 2024-05-15T13:28:56.420 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8850000000000004e-06 2024-05-15T13:28:56.420 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.420 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.420 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:56.420 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.487852+0000", 2024-05-15T13:28:56.420 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0773e-05 2024-05-15T13:28:56.420 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.420 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.420 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:56.420 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.487861+0000", 2024-05-15T13:28:56.421 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7409999999999998e-06 2024-05-15T13:28:56.421 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.421 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.421 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.421 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.487914+0000", 2024-05-15T13:28:56.421 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2927000000000003e-05 2024-05-15T13:28:56.421 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.421 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.421 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:28:56.421 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.487920+0000", 2024-05-15T13:28:56.421 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9410000000000004e-06 2024-05-15T13:28:56.421 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.421 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.421 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.422 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.586391+0000", 2024-05-15T13:28:56.422 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0984710929999999 2024-05-15T13:28:56.422 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.422 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.422 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:28:56.422 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.586541+0000", 2024-05-15T13:28:56.422 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014972499999999999 2024-05-15T13:28:56.422 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.422 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.422 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 9,11", 2024-05-15T13:28:56.422 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.586702+0000", 2024-05-15T13:28:56.422 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00016099500000000001 2024-05-15T13:28:56.422 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.422 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.422 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:28:56.423 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.591572+0000", 2024-05-15T13:28:56.423 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0048708939999999997 2024-05-15T13:28:56.423 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.423 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.423 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:28:56.423 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.598394+0000", 2024-05-15T13:28:56.423 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0068220260000000001 2024-05-15T13:28:56.423 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.423 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.423 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:28:56.423 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.598945+0000", 2024-05-15T13:28:56.423 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00055099999999999995 2024-05-15T13:28:56.423 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.423 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.423 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:28:56.424 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.598999+0000", 2024-05-15T13:28:56.424 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.4008000000000002e-05 2024-05-15T13:28:56.424 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.424 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.424 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:28:56.424 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.600983+0000", 2024-05-15T13:28:56.424 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001983351 2024-05-15T13:28:56.424 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.424 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:56.424 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.424 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.424 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.424 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:193589 9.5 9.83b88795 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e936)", 2024-05-15T13:28:56.424 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:57.536484+0000", 2024-05-15T13:28:56.424 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 58.744801668000001, 2024-05-15T13:28:56.425 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.89822722600000005, 2024-05-15T13:28:56.425 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:28:56.425 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:28:56.425 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:28:56.425 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:28:56.425 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:28:56.425 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 193589 2024-05-15T13:28:56.425 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.425 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:28:56.425 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.425 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:28:56.425 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.536484+0000", 2024-05-15T13:28:56.425 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.425 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.425 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.425 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:28:56.426 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.536484+0000", 2024-05-15T13:28:56.426 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:28:56.426 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.426 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.426 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:28:56.426 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.536485+0000", 2024-05-15T13:28:56.426 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.5099999999999998e-07 2024-05-15T13:28:56.426 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.426 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.426 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:28:56.426 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.536490+0000", 2024-05-15T13:28:56.426 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7339999999999999e-06 2024-05-15T13:28:56.426 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.426 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.427 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:28:56.427 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.536536+0000", 2024-05-15T13:28:56.427 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6875000000000001e-05 2024-05-15T13:28:56.427 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.427 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.427 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:28:56.427 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.536545+0000", 2024-05-15T13:28:56.427 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7600000000000008e-06 2024-05-15T13:28:56.427 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.427 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.427 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.427 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.536612+0000", 2024-05-15T13:28:56.427 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7142000000000002e-05 2024-05-15T13:28:56.427 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.427 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.427 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:28:56.428 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.536623+0000", 2024-05-15T13:28:56.428 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1089e-05 2024-05-15T13:28:56.428 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.428 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.428 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:28:56.428 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.434706+0000", 2024-05-15T13:28:56.428 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.89808289100000005 2024-05-15T13:28:56.428 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:28:56.428 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:28:56.428 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:28:56.428 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.434711+0000", 2024-05-15T13:28:56.428 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8840000000000002e-06 2024-05-15T13:28:56.428 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.428 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:56.428 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.429 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:28:56.429 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:28:56.429 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:28:56.486 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:28:56.627 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:28:56.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:56.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:28:56.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:56 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:28:56.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:56 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:28:56.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:56 smithi120 ceph-mon[43961]: osdmap e968: 12 total, 12 up, 11 in 2024-05-15T13:28:56.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:56 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:28:56.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:56 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:28:56.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:56 smithi177 ceph-mon[44303]: osdmap e968: 12 total, 12 up, 11 in 2024-05-15T13:28:56.744 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:28:56.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:56 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:28:56.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:56 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:28:56.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:56 smithi027 ceph-mon[39795]: osdmap e968: 12 total, 12 up, 11 in 2024-05-15T13:28:56.806 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:57.021 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:28:57.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:57.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:28:57.047 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-05-15T13:28:57.127 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:28:57.178 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:57.220 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:28:57.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:57.228 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:28:57.329 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:28:57.340 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 6 1 2024-05-15T13:28:57.359 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:57.556 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:28:57.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:57 smithi027 ceph-mon[39795]: pgmap v2446: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1010 MiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 656 op/s; 2569/48480 objects degraded (5.299%); 3431/48480 objects misplaced (7.077%); 9.8 MiB/s, 236 objects/s recovering 2024-05-15T13:28:57.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:57 smithi120 ceph-mon[43961]: pgmap v2446: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1010 MiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 656 op/s; 2569/48480 objects degraded (5.299%); 3431/48480 objects misplaced (7.077%); 9.8 MiB/s, 236 objects/s recovering 2024-05-15T13:28:57.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:57 smithi177 ceph-mon[44303]: pgmap v2446: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1010 MiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 656 op/s; 2569/48480 objects degraded (5.299%); 3431/48480 objects misplaced (7.077%); 9.8 MiB/s, 236 objects/s recovering 2024-05-15T13:28:59.416 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:28:59.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:28:59.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:28:59.522 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:28:59.577 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:28:59.957 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:59 smithi027 ceph-mon[39795]: pgmap v2447: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 895 MiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 734 op/s; 2576/42930 objects degraded (6.000%); 2819/42930 objects misplaced (6.567%); 11 MiB/s, 237 objects/s recovering 2024-05-15T13:28:59.957 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:59 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 2576/42930 objects degraded (6.000%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:28:59.957 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:28:59 smithi027 ceph-mon[39795]: osdmap e969: 12 total, 12 up, 11 in 2024-05-15T13:29:00.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:59 smithi177 ceph-mon[44303]: pgmap v2447: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 895 MiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 734 op/s; 2576/42930 objects degraded (6.000%); 2819/42930 objects misplaced (6.567%); 11 MiB/s, 237 objects/s recovering 2024-05-15T13:29:00.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:59 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 2576/42930 objects degraded (6.000%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:29:00.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:28:59 smithi177 ceph-mon[44303]: osdmap e969: 12 total, 12 up, 11 in 2024-05-15T13:29:00.237 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:29:00.244 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:29:00.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:00.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:29:00.252 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:59 smithi120 ceph-mon[43961]: pgmap v2447: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 895 MiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 734 op/s; 2576/42930 objects degraded (6.000%); 2819/42930 objects misplaced (6.567%); 11 MiB/s, 237 objects/s recovering 2024-05-15T13:29:00.252 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:59 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 2576/42930 objects degraded (6.000%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:29:00.252 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:28:59 smithi120 ceph-mon[43961]: osdmap e969: 12 total, 12 up, 11 in 2024-05-15T13:29:00.351 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:29:00.401 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:00.456 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:29:00.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:00.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:29:00.611 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:29:00.640 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:29:00.678 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:29:00.684 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:00.700 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:29:00.725 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:29:00.755 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:29:00.795 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:29:00.824 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:29:00.853 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:29:00.880 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:29:00.903 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:29:00.942 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:29:00.980 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:29:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:29:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:29:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:29:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:29:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:29:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:29:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:29:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:29:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:29:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:29:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:29:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:29:01.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:00 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:29:01.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:00 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:29:01.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:00 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:29:01.043 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:00 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:29:01.043 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:00 smithi120 ceph-mon[43961]: 9.d scrub starts 2024-05-15T13:29:01.043 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:00 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1002558483' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:29:01.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:00 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:29:01.120 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:00 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:29:01.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:00 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:29:01.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:00 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:29:01.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:00 smithi027 ceph-mon[39795]: 9.d scrub starts 2024-05-15T13:29:01.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:00 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1002558483' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:29:01.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:00 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:29:01.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:00 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:29:01.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:00 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:29:01.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:00 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:29:01.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:00 smithi177 ceph-mon[44303]: 9.d scrub starts 2024-05-15T13:29:01.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:00 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1002558483' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:29:01.458 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:29:01.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:01.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:29:01.567 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:29:01.620 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:01.649 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:29:01.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:01.658 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:29:01.759 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:29:01.825 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:01.867 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:29:01.877 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:01 smithi177 ceph-mon[44303]: pgmap v2449: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 868 MiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 610 op/s; 2576/41652 objects degraded (6.185%); 2819/41652 objects misplaced (6.768%); 7.9 MiB/s, 165 objects/s recovering 2024-05-15T13:29:01.877 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:01 smithi177 ceph-mon[44303]: 9.d scrub ok 2024-05-15T13:29:01.877 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:01 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:29:01.877 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:01 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:29:01.877 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:01 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1312456296' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:29:01.877 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:01 smithi177 ceph-mon[44303]: 1.6 scrub starts 2024-05-15T13:29:01.877 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:01 smithi177 ceph-mon[44303]: 1.6 scrub ok 2024-05-15T13:29:01.877 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:01 smithi177 ceph-mon[44303]: 1.15 scrub starts 2024-05-15T13:29:01.877 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:01 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:29:02.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:01 smithi027 ceph-mon[39795]: pgmap v2449: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 868 MiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 610 op/s; 2576/41652 objects degraded (6.185%); 2819/41652 objects misplaced (6.768%); 7.9 MiB/s, 165 objects/s recovering 2024-05-15T13:29:02.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:01 smithi027 ceph-mon[39795]: 9.d scrub ok 2024-05-15T13:29:02.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:01 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:29:02.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:01 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:29:02.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:01 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1312456296' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:29:02.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:01 smithi027 ceph-mon[39795]: 1.6 scrub starts 2024-05-15T13:29:02.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:01 smithi027 ceph-mon[39795]: 1.6 scrub ok 2024-05-15T13:29:02.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:01 smithi027 ceph-mon[39795]: 1.15 scrub starts 2024-05-15T13:29:02.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:01 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:29:02.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:01 smithi120 ceph-mon[43961]: pgmap v2449: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 868 MiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 610 op/s; 2576/41652 objects degraded (6.185%); 2819/41652 objects misplaced (6.768%); 7.9 MiB/s, 165 objects/s recovering 2024-05-15T13:29:02.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:01 smithi120 ceph-mon[43961]: 9.d scrub ok 2024-05-15T13:29:02.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:01 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:29:02.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:01 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:29:02.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:01 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1312456296' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:29:02.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:01 smithi120 ceph-mon[43961]: 1.6 scrub starts 2024-05-15T13:29:02.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:01 smithi120 ceph-mon[43961]: 1.6 scrub ok 2024-05-15T13:29:02.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:01 smithi120 ceph-mon[43961]: 1.15 scrub starts 2024-05-15T13:29:02.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:01 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:29:02.635 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:29:02.635 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:29:02.891 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:02 smithi027 ceph-mon[39795]: 1.15 scrub ok 2024-05-15T13:29:02.891 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:02 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1312456296' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:29:02.891 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:02 smithi027 ceph-mon[39795]: osdmap e970: 12 total, 12 up, 11 in 2024-05-15T13:29:03.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:02 smithi177 ceph-mon[44303]: 1.15 scrub ok 2024-05-15T13:29:03.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:02 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1312456296' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:29:03.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:02 smithi177 ceph-mon[44303]: osdmap e970: 12 total, 12 up, 11 in 2024-05-15T13:29:03.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:02 smithi120 ceph-mon[43961]: 1.15 scrub ok 2024-05-15T13:29:03.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:02 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1312456296' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:29:03.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:02 smithi120 ceph-mon[43961]: osdmap e970: 12 total, 12 up, 11 in 2024-05-15T13:29:03.320 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:29:03.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:03.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:29:03.427 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:29:03.479 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:03.580 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:29:03.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:03.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:29:03.692 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:29:03.728 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:03.851 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:29:03.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:03.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:29:03.904 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:29:03.959 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:29:04.008 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:04.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:03 smithi177 ceph-mon[44303]: pgmap v2451: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 748 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 712 op/s; 2576/35880 objects degraded (7.179%); 2599/35880 objects misplaced (7.244%); 6.7 MiB/s, 191 objects/s recovering 2024-05-15T13:29:04.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:03 smithi177 ceph-mon[44303]: 1.2 scrub starts 2024-05-15T13:29:04.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:03 smithi027 ceph-mon[39795]: pgmap v2451: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 748 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 712 op/s; 2576/35880 objects degraded (7.179%); 2599/35880 objects misplaced (7.244%); 6.7 MiB/s, 191 objects/s recovering 2024-05-15T13:29:04.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:03 smithi027 ceph-mon[39795]: 1.2 scrub starts 2024-05-15T13:29:04.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:03 smithi120 ceph-mon[43961]: pgmap v2451: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 748 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 712 op/s; 2576/35880 objects degraded (7.179%); 2599/35880 objects misplaced (7.244%); 6.7 MiB/s, 191 objects/s recovering 2024-05-15T13:29:04.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:03 smithi120 ceph-mon[43961]: 1.2 scrub starts 2024-05-15T13:29:04.300 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:29:04.368 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.6 to 1 (10000) 2024-05-15T13:29:04.381 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:29:04.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:04.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:29:04.423 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-05-15T13:29:04.486 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:29:04.516 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:04.901 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:29:05.194 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:04 smithi177 ceph-mon[44303]: 1.2 scrub ok 2024-05-15T13:29:05.194 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:04 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/657695387' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T13:29:05.194 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:04 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 2576/35880 objects degraded (7.179%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:29:05.194 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:04 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/657695387' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T13:29:05.194 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:04 smithi177 ceph-mon[44303]: osdmap e971: 12 total, 12 up, 11 in 2024-05-15T13:29:05.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:04 smithi027 ceph-mon[39795]: 1.2 scrub ok 2024-05-15T13:29:05.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:04 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/657695387' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T13:29:05.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:04 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 2576/35880 objects degraded (7.179%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:29:05.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:04 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/657695387' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T13:29:05.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:04 smithi027 ceph-mon[39795]: osdmap e971: 12 total, 12 up, 11 in 2024-05-15T13:29:05.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:04 smithi120 ceph-mon[43961]: 1.2 scrub ok 2024-05-15T13:29:05.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:04 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/657695387' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T13:29:05.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:04 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 2576/35880 objects degraded (7.179%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:29:05.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:04 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/657695387' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T13:29:05.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:04 smithi120 ceph-mon[43961]: osdmap e971: 12 total, 12 up, 11 in 2024-05-15T13:29:05.566 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set noscrub 2024-05-15T13:29:05.919 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:05 smithi027 ceph-mon[39795]: pgmap v2453: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 678 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 579 op/s; 2605/32502 objects degraded (8.015%); 1932/32502 objects misplaced (5.944%); 6.6 MiB/s, 160 objects/s recovering 2024-05-15T13:29:05.919 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:05 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:29:05.919 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:05 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:29:05.919 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:05 smithi027 ceph-mon[39795]: osdmap e972: 12 total, 12 up, 11 in 2024-05-15T13:29:05.924 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:05 smithi120 ceph-mon[43961]: pgmap v2453: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 678 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 579 op/s; 2605/32502 objects degraded (8.015%); 1932/32502 objects misplaced (5.944%); 6.6 MiB/s, 160 objects/s recovering 2024-05-15T13:29:05.924 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:05 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:29:05.924 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:05 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:29:05.924 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:05 smithi120 ceph-mon[43961]: osdmap e972: 12 total, 12 up, 11 in 2024-05-15T13:29:06.196 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:05 smithi177 ceph-mon[44303]: pgmap v2453: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 678 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 579 op/s; 2605/32502 objects degraded (8.015%); 1932/32502 objects misplaced (5.944%); 6.6 MiB/s, 160 objects/s recovering 2024-05-15T13:29:06.196 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:05 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:29:06.196 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:05 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:29:06.196 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:05 smithi177 ceph-mon[44303]: osdmap e972: 12 total, 12 up, 11 in 2024-05-15T13:29:06.968 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:29:06.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:06.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:29:07.073 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:29:07.104 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:07.369 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:29:07.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:07.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:29:07.447 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:29:07.447 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:29:07.473 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:29:07.534 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:07.764 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:29:07.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:07.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:29:07.874 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:29:07.907 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:07.965 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 9 1 2024-05-15T13:29:08.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:07 smithi120 ceph-mon[43961]: pgmap v2455: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 583 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 760 op/s; 2605/27954 objects degraded (9.319%); 1693/27954 objects misplaced (6.056%); 9.0 MiB/s, 259 objects/s recovering 2024-05-15T13:29:08.195 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:07 smithi177 ceph-mon[44303]: pgmap v2455: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 583 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 760 op/s; 2605/27954 objects degraded (9.319%); 1693/27954 objects misplaced (6.056%); 9.0 MiB/s, 259 objects/s recovering 2024-05-15T13:29:08.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:07 smithi027 ceph-mon[39795]: pgmap v2455: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 583 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 760 op/s; 2605/27954 objects degraded (9.319%); 1693/27954 objects misplaced (6.056%); 9.0 MiB/s, 259 objects/s recovering 2024-05-15T13:29:08.315 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-05-15T13:29:08.426 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:29:08.468 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:29:08.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:08.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:29:08.576 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:29:08.617 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:08.667 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:29:08.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:08.671 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:29:08.732 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:29:08.772 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:29:08.818 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:09.091 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:29:09.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:09.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:29:09.196 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9 2024-05-15T13:29:09.265 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:09.304 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9" found: no such container 2024-05-15T13:29:09.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:09.312 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.9' 2024-05-15T13:29:09.412 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:29:09.458 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:09.490 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:29:09.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:09.497 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:29:09.598 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:29:09.643 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:09.697 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:29:09.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:09.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:29:09.805 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:29:09.839 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:09.936 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:09 smithi120 ceph-mon[43961]: pgmap v2456: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 460 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 784 op/s; 2645/22071 objects degraded (11.984%); 1043/22071 objects misplaced (4.726%); 9.4 MiB/s, 201 objects/s recovering 2024-05-15T13:29:09.936 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:09 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 2645/22071 objects degraded (11.984%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:29:09.936 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:09 smithi120 ceph-mon[43961]: osdmap e973: 12 total, 12 up, 11 in 2024-05-15T13:29:09.936 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:09 smithi120 ceph-mon[43961]: 1.b scrub starts 2024-05-15T13:29:09.936 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:09 smithi120 ceph-mon[43961]: 1.b scrub ok 2024-05-15T13:29:10.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:09 smithi177 ceph-mon[44303]: pgmap v2456: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 460 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 784 op/s; 2645/22071 objects degraded (11.984%); 1043/22071 objects misplaced (4.726%); 9.4 MiB/s, 201 objects/s recovering 2024-05-15T13:29:10.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:09 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 2645/22071 objects degraded (11.984%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:29:10.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:09 smithi177 ceph-mon[44303]: osdmap e973: 12 total, 12 up, 11 in 2024-05-15T13:29:10.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:09 smithi177 ceph-mon[44303]: 1.b scrub starts 2024-05-15T13:29:10.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:09 smithi177 ceph-mon[44303]: 1.b scrub ok 2024-05-15T13:29:10.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:09 smithi027 ceph-mon[39795]: pgmap v2456: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 460 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 784 op/s; 2645/22071 objects degraded (11.984%); 1043/22071 objects misplaced (4.726%); 9.4 MiB/s, 201 objects/s recovering 2024-05-15T13:29:10.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:09 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 2645/22071 objects degraded (11.984%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:29:10.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:09 smithi027 ceph-mon[39795]: osdmap e973: 12 total, 12 up, 11 in 2024-05-15T13:29:10.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:09 smithi027 ceph-mon[39795]: 1.b scrub starts 2024-05-15T13:29:10.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:09 smithi027 ceph-mon[39795]: 1.b scrub ok 2024-05-15T13:29:11.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:10 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/552560180' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:29:11.043 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:10 smithi120 ceph-mon[43961]: 1.13 deep-scrub starts 2024-05-15T13:29:11.043 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:10 smithi120 ceph-mon[43961]: 1.13 deep-scrub ok 2024-05-15T13:29:11.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:10 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/552560180' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:29:11.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:10 smithi027 ceph-mon[39795]: 1.13 deep-scrub starts 2024-05-15T13:29:11.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:10 smithi027 ceph-mon[39795]: 1.13 deep-scrub ok 2024-05-15T13:29:11.445 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:10 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/552560180' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:29:11.445 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:10 smithi177 ceph-mon[44303]: 1.13 deep-scrub starts 2024-05-15T13:29:11.445 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:10 smithi177 ceph-mon[44303]: 1.13 deep-scrub ok 2024-05-15T13:29:11.898 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:29:11.898 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:29:11.898 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:29:11.898 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:29:11.898 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:29:11.898 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:29:11.898 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:29:11.898 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:29:11.899 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:29:11.899 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:29:11.899 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:29:11.899 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:29:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:29:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:29:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:29:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:29:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:29:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:29:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:29:11.899 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:29:11.900 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:29:11.900 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:29:11.900 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:29:11.900 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:29:12.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:11 smithi027 ceph-mon[39795]: pgmap v2458: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 433 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 608 op/s; 2645/20787 objects degraded (12.724%); 1043/20787 objects misplaced (5.018%); 6.1 MiB/s, 153 objects/s recovering 2024-05-15T13:29:12.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:11 smithi027 ceph-mon[39795]: 1.1e deep-scrub starts 2024-05-15T13:29:12.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:11 smithi027 ceph-mon[39795]: 1.1e deep-scrub ok 2024-05-15T13:29:12.297 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:11 smithi120 ceph-mon[43961]: pgmap v2458: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 433 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 608 op/s; 2645/20787 objects degraded (12.724%); 1043/20787 objects misplaced (5.018%); 6.1 MiB/s, 153 objects/s recovering 2024-05-15T13:29:12.297 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:11 smithi120 ceph-mon[43961]: 1.1e deep-scrub starts 2024-05-15T13:29:12.297 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:11 smithi120 ceph-mon[43961]: 1.1e deep-scrub ok 2024-05-15T13:29:12.444 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:11 smithi177 ceph-mon[44303]: pgmap v2458: 47 pgs: 1 active+recovering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 433 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 608 op/s; 2645/20787 objects degraded (12.724%); 1043/20787 objects misplaced (5.018%); 6.1 MiB/s, 153 objects/s recovering 2024-05-15T13:29:12.444 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:11 smithi177 ceph-mon[44303]: 1.1e deep-scrub starts 2024-05-15T13:29:12.444 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:11 smithi177 ceph-mon[44303]: 1.1e deep-scrub ok 2024-05-15T13:29:12.643 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:29:12.800 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:29:12.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:12.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:29:12.907 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:29:12.953 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:12.997 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:29:13.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:13.001 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:29:13.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:12 smithi120 ceph-mon[43961]: 1.3 scrub starts 2024-05-15T13:29:13.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:12 smithi120 ceph-mon[43961]: 1.3 scrub ok 2024-05-15T13:29:13.102 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:29:13.171 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:13.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:12 smithi027 ceph-mon[39795]: 1.3 scrub starts 2024-05-15T13:29:13.284 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:12 smithi027 ceph-mon[39795]: 1.3 scrub ok 2024-05-15T13:29:13.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:12 smithi177 ceph-mon[44303]: 1.3 scrub starts 2024-05-15T13:29:13.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:12 smithi177 ceph-mon[44303]: 1.3 scrub ok 2024-05-15T13:29:13.610 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:29:13.610 INFO:teuthology.orchestra.run.smithi120.stdout: "size": 20, 2024-05-15T13:29:13.610 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 600, 2024-05-15T13:29:13.610 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [ 2024-05-15T13:29:13.610 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.610 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:181053 9.0 9:0f0f5083:::benchmark_data_smithi177_159919_object60810:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e920)", 2024-05-15T13:29:13.610 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:28.866633+0000", 2024-05-15T13:29:13.610 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 103.875958657, 2024-05-15T13:29:13.610 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.936316927, 2024-05-15T13:29:13.611 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:29:13.611 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:13.611 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:29:13.611 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:29:13.611 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:13.611 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 181053 2024-05-15T13:29:13.611 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.611 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:29:13.611 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.611 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:29:13.611 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.866633+0000", 2024-05-15T13:29:13.611 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.611 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.612 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.612 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:29:13.612 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.866633+0000", 2024-05-15T13:29:13.612 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.612 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.612 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.612 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:29:13.612 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.866634+0000", 2024-05-15T13:29:13.612 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4309999999999999e-06 2024-05-15T13:29:13.612 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.612 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.612 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:29:13.613 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.866641+0000", 2024-05-15T13:29:13.613 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.5100000000000004e-06 2024-05-15T13:29:13.613 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.613 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.613 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:29:13.613 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.866654+0000", 2024-05-15T13:29:13.613 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3591000000000001e-05 2024-05-15T13:29:13.613 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.613 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.613 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:29:13.613 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.866663+0000", 2024-05-15T13:29:13.613 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.5830000000000007e-06 2024-05-15T13:29:13.613 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.613 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.613 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.614 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.869899+0000", 2024-05-15T13:29:13.614 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0032367559999999999 2024-05-15T13:29:13.614 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.614 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.614 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.614 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.869911+0000", 2024-05-15T13:29:13.614 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1902000000000001e-05 2024-05-15T13:29:13.614 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.614 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.614 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.614 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.342202+0000", 2024-05-15T13:29:13.614 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.47229095700000001 2024-05-15T13:29:13.614 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.615 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.615 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.615 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.342213+0000", 2024-05-15T13:29:13.615 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0437e-05 2024-05-15T13:29:13.615 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.615 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.615 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.615 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.795657+0000", 2024-05-15T13:29:13.615 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.453444213 2024-05-15T13:29:13.615 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.615 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.615 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:29:13.615 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.795828+0000", 2024-05-15T13:29:13.615 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000171106 2024-05-15T13:29:13.616 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.616 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.616 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 8,10,11", 2024-05-15T13:29:13.616 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.795998+0000", 2024-05-15T13:29:13.616 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00016979399999999999 2024-05-15T13:29:13.616 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.616 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.616 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.616 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.802388+0000", 2024-05-15T13:29:13.616 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0063902430000000003 2024-05-15T13:29:13.616 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.616 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.616 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:29:13.617 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.802406+0000", 2024-05-15T13:29:13.617 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.8304999999999998e-05 2024-05-15T13:29:13.617 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.617 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.617 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.617 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.802451+0000", 2024-05-15T13:29:13.617 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4271999999999999e-05 2024-05-15T13:29:13.617 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.617 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.617 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.617 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.802500+0000", 2024-05-15T13:29:13.617 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9329000000000002e-05 2024-05-15T13:29:13.617 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.617 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.618 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:29:13.618 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.802559+0000", 2024-05-15T13:29:13.618 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8501000000000001e-05 2024-05-15T13:29:13.618 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.618 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.618 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:29:13.618 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.802950+0000", 2024-05-15T13:29:13.618 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00039099700000000002 2024-05-15T13:29:13.618 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.618 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:29:13.618 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.618 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.618 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.618 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:181078 9.0 9:028bae8c:::benchmark_data_smithi177_159919_object60835:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e921)", 2024-05-15T13:29:13.619 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:28.937113+0000", 2024-05-15T13:29:13.619 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 103.80547783199999, 2024-05-15T13:29:13.619 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.897354194, 2024-05-15T13:29:13.619 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:29:13.619 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:13.619 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:29:13.619 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:29:13.619 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:13.619 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 181078 2024-05-15T13:29:13.619 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.619 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:29:13.619 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.619 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:29:13.620 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.937113+0000", 2024-05-15T13:29:13.620 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.620 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.620 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.620 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:29:13.620 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.937113+0000", 2024-05-15T13:29:13.620 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.620 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.620 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.620 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:29:13.620 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.937116+0000", 2024-05-15T13:29:13.620 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0719999999999998e-06 2024-05-15T13:29:13.620 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.620 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.621 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:29:13.621 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.937124+0000", 2024-05-15T13:29:13.621 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8179999999999993e-06 2024-05-15T13:29:13.621 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.621 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.621 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:29:13.621 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.937145+0000", 2024-05-15T13:29:13.621 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0820000000000001e-05 2024-05-15T13:29:13.621 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.621 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.621 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:29:13.621 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.937156+0000", 2024-05-15T13:29:13.621 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1358e-05 2024-05-15T13:29:13.621 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.622 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.622 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.622 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.937230+0000", 2024-05-15T13:29:13.622 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3571000000000004e-05 2024-05-15T13:29:13.622 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.622 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.622 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.622 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.937242+0000", 2024-05-15T13:29:13.622 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1666e-05 2024-05-15T13:29:13.622 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.622 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.622 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.622 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.342239+0000", 2024-05-15T13:29:13.622 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.40499717699999999 2024-05-15T13:29:13.623 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.623 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.623 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.623 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.342249+0000", 2024-05-15T13:29:13.623 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0454e-05 2024-05-15T13:29:13.623 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.623 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.623 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.623 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.802986+0000", 2024-05-15T13:29:13.623 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.460736445 2024-05-15T13:29:13.623 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.623 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.623 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:29:13.623 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.803123+0000", 2024-05-15T13:29:13.623 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000137088 2024-05-15T13:29:13.624 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.624 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.624 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 8,10,11", 2024-05-15T13:29:13.624 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.803274+0000", 2024-05-15T13:29:13.624 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015153700000000001 2024-05-15T13:29:13.624 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.624 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.624 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.624 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.813180+0000", 2024-05-15T13:29:13.624 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0099055600000000008 2024-05-15T13:29:13.624 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.624 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.624 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:29:13.624 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.813308+0000", 2024-05-15T13:29:13.625 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012800800000000001 2024-05-15T13:29:13.625 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.625 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.625 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.625 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.823670+0000", 2024-05-15T13:29:13.625 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.010362333 2024-05-15T13:29:13.625 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.625 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.625 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.625 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.833637+0000", 2024-05-15T13:29:13.625 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.009966941 2024-05-15T13:29:13.625 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.625 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.625 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:29:13.626 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.833699+0000", 2024-05-15T13:29:13.626 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1310999999999996e-05 2024-05-15T13:29:13.626 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.626 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.626 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:29:13.626 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.834468+0000", 2024-05-15T13:29:13.626 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00076903500000000005 2024-05-15T13:29:13.626 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.626 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:29:13.626 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.626 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.626 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.627 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:181079 9.0 9:0dd37094:::benchmark_data_smithi177_159919_object60836:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e921)", 2024-05-15T13:29:13.627 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:28.937535+0000", 2024-05-15T13:29:13.627 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 103.805056034, 2024-05-15T13:29:13.627 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.897758083, 2024-05-15T13:29:13.627 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:29:13.627 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:13.627 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:29:13.627 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:29:13.627 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:13.627 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 181079 2024-05-15T13:29:13.627 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.627 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:29:13.627 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.627 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:29:13.628 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.937535+0000", 2024-05-15T13:29:13.628 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.628 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.628 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.628 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:29:13.628 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.937535+0000", 2024-05-15T13:29:13.628 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.628 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.628 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.628 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:29:13.629 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.937536+0000", 2024-05-15T13:29:13.629 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7700000000000004e-07 2024-05-15T13:29:13.629 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.629 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.629 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:29:13.629 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.937541+0000", 2024-05-15T13:29:13.629 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.7940000000000002e-06 2024-05-15T13:29:13.629 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.629 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.629 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:29:13.629 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.937555+0000", 2024-05-15T13:29:13.629 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3814e-05 2024-05-15T13:29:13.630 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.630 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.630 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:29:13.630 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.937562+0000", 2024-05-15T13:29:13.630 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3350000000000001e-06 2024-05-15T13:29:13.630 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.630 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.630 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.630 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.937606+0000", 2024-05-15T13:29:13.630 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4450000000000003e-05 2024-05-15T13:29:13.630 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.630 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.631 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.631 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.937614+0000", 2024-05-15T13:29:13.631 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.6009999999999997e-06 2024-05-15T13:29:13.631 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.631 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.631 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.631 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.342277+0000", 2024-05-15T13:29:13.631 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.40466264200000002 2024-05-15T13:29:13.631 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.631 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.631 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.631 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.342283+0000", 2024-05-15T13:29:13.631 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7299999999999999e-06 2024-05-15T13:29:13.631 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.631 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.632 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.632 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.808429+0000", 2024-05-15T13:29:13.632 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4661455509999999 2024-05-15T13:29:13.632 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.632 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.632 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:29:13.632 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.808583+0000", 2024-05-15T13:29:13.632 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015385900000000001 2024-05-15T13:29:13.632 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.632 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.632 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 8,10,11", 2024-05-15T13:29:13.632 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.808762+0000", 2024-05-15T13:29:13.632 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00017934 2024-05-15T13:29:13.632 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.633 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.633 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.633 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.813294+0000", 2024-05-15T13:29:13.633 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0045316540000000004 2024-05-15T13:29:13.633 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.633 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.633 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:29:13.633 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.823618+0000", 2024-05-15T13:29:13.633 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.010323736 2024-05-15T13:29:13.633 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.633 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.633 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.633 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.823765+0000", 2024-05-15T13:29:13.633 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014787599999999999 2024-05-15T13:29:13.633 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.634 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.634 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.634 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.834603+0000", 2024-05-15T13:29:13.634 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.010838096 2024-05-15T13:29:13.634 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.634 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.634 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:29:13.634 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.834665+0000", 2024-05-15T13:29:13.634 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1716000000000003e-05 2024-05-15T13:29:13.634 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.634 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.634 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:29:13.634 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.835293+0000", 2024-05-15T13:29:13.634 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00062811200000000001 2024-05-15T13:29:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:29:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:181093 9.0 9:0b73c612:::benchmark_data_smithi177_159919_object60850:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e921)", 2024-05-15T13:29:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:28.971511+0000", 2024-05-15T13:29:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 103.771080302, 2024-05-15T13:29:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.86449398, 2024-05-15T13:29:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:29:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:29:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:29:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:13.635 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 181093 2024-05-15T13:29:13.636 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.636 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:29:13.636 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.636 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:29:13.636 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.971511+0000", 2024-05-15T13:29:13.636 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.636 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.636 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.636 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:29:13.636 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.971511+0000", 2024-05-15T13:29:13.636 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.636 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.636 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.636 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:29:13.636 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.971513+0000", 2024-05-15T13:29:13.637 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2309999999999999e-06 2024-05-15T13:29:13.637 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.637 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.637 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:29:13.637 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.971523+0000", 2024-05-15T13:29:13.637 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0283e-05 2024-05-15T13:29:13.637 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.637 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.637 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:29:13.637 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.971546+0000", 2024-05-15T13:29:13.637 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2238e-05 2024-05-15T13:29:13.637 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.637 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.637 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:29:13.637 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.971557+0000", 2024-05-15T13:29:13.638 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0959e-05 2024-05-15T13:29:13.638 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.638 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.638 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.638 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.971625+0000", 2024-05-15T13:29:13.638 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.7849999999999996e-05 2024-05-15T13:29:13.638 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.638 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.638 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.638 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.971636+0000", 2024-05-15T13:29:13.638 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1684e-05 2024-05-15T13:29:13.638 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.638 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.639 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.639 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.342308+0000", 2024-05-15T13:29:13.639 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.37067151500000001 2024-05-15T13:29:13.639 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.639 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.639 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.639 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.342315+0000", 2024-05-15T13:29:13.639 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.1400000000000002e-06 2024-05-15T13:29:13.639 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.639 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.639 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.639 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.813335+0000", 2024-05-15T13:29:13.639 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4710199900000001 2024-05-15T13:29:13.639 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.639 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.640 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:29:13.640 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.813436+0000", 2024-05-15T13:29:13.640 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00010111 2024-05-15T13:29:13.640 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.640 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.640 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 8,10,11", 2024-05-15T13:29:13.640 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.813536+0000", 2024-05-15T13:29:13.640 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.9640000000000001e-05 2024-05-15T13:29:13.640 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.640 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.640 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.640 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.823604+0000", 2024-05-15T13:29:13.640 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.010068817000000001 2024-05-15T13:29:13.640 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.640 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.641 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:29:13.641 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.823623+0000", 2024-05-15T13:29:13.641 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.8139e-05 2024-05-15T13:29:13.641 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.641 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.641 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.641 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.823822+0000", 2024-05-15T13:29:13.641 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0001996 2024-05-15T13:29:13.641 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.641 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.641 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.641 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.835476+0000", 2024-05-15T13:29:13.641 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.011653686999999999 2024-05-15T13:29:13.641 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.641 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.642 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:29:13.642 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.835529+0000", 2024-05-15T13:29:13.642 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.2871999999999998e-05 2024-05-15T13:29:13.642 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.642 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.642 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:29:13.642 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.836005+0000", 2024-05-15T13:29:13.642 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00047622499999999999 2024-05-15T13:29:13.642 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.642 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:29:13.642 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.642 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.642 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.643 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:181097 9.0 9:0177c6f1:::benchmark_data_smithi177_159919_object60854:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e921)", 2024-05-15T13:29:13.643 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:28.979234+0000", 2024-05-15T13:29:13.643 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 103.76335747900001, 2024-05-15T13:29:13.643 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.857322245, 2024-05-15T13:29:13.643 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:29:13.643 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:13.643 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:29:13.643 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:29:13.643 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:13.643 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 181097 2024-05-15T13:29:13.643 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.643 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:29:13.643 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.643 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:29:13.644 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.979234+0000", 2024-05-15T13:29:13.644 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.644 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.644 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.644 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:29:13.644 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.979234+0000", 2024-05-15T13:29:13.644 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.644 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.644 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.644 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:29:13.644 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.979235+0000", 2024-05-15T13:29:13.644 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.325e-06 2024-05-15T13:29:13.644 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.644 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.645 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:29:13.645 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.979241+0000", 2024-05-15T13:29:13.645 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1229999999999997e-06 2024-05-15T13:29:13.645 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.645 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.645 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:29:13.645 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.979254+0000", 2024-05-15T13:29:13.645 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2907e-05 2024-05-15T13:29:13.645 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.645 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.645 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:29:13.645 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.979262+0000", 2024-05-15T13:29:13.645 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8730000000000007e-06 2024-05-15T13:29:13.645 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.646 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.646 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.646 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.981273+0000", 2024-05-15T13:29:13.646 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0020110750000000002 2024-05-15T13:29:13.646 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.646 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.646 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.646 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.981281+0000", 2024-05-15T13:29:13.646 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8529999999999992e-06 2024-05-15T13:29:13.646 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.646 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.646 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.646 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.342340+0000", 2024-05-15T13:29:13.646 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.36105868600000002 2024-05-15T13:29:13.647 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.647 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.647 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.647 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.342348+0000", 2024-05-15T13:29:13.647 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.0229999999999992e-06 2024-05-15T13:29:13.647 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.647 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.647 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.647 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.817932+0000", 2024-05-15T13:29:13.647 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.475584784 2024-05-15T13:29:13.647 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.647 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.647 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:29:13.647 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.818089+0000", 2024-05-15T13:29:13.648 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015668 2024-05-15T13:29:13.648 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.648 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.648 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 8,10,11", 2024-05-15T13:29:13.648 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.818256+0000", 2024-05-15T13:29:13.648 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00016670699999999999 2024-05-15T13:29:13.648 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.648 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.648 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.648 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.823709+0000", 2024-05-15T13:29:13.648 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0054533799999999999 2024-05-15T13:29:13.648 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.648 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.648 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:29:13.649 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.823718+0000", 2024-05-15T13:29:13.649 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.3280000000000006e-06 2024-05-15T13:29:13.649 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.649 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.649 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.649 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.823879+0000", 2024-05-15T13:29:13.649 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00016136899999999999 2024-05-15T13:29:13.649 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.649 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.649 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.649 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.836110+0000", 2024-05-15T13:29:13.649 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.012230896 2024-05-15T13:29:13.649 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.649 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.649 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:29:13.649 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.836160+0000", 2024-05-15T13:29:13.650 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.0481e-05 2024-05-15T13:29:13.650 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.650 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.650 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:29:13.650 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.836556+0000", 2024-05-15T13:29:13.650 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00039575500000000002 2024-05-15T13:29:13.650 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.650 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:29:13.650 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.650 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.650 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.650 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:181099 9.0 9:01b34959:::benchmark_data_smithi177_159919_object60856:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e921)", 2024-05-15T13:29:13.650 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:28.980990+0000", 2024-05-15T13:29:13.650 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 103.761601113, 2024-05-15T13:29:13.651 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.856046979, 2024-05-15T13:29:13.651 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:29:13.651 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:13.651 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:29:13.651 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:29:13.651 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:13.651 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 181099 2024-05-15T13:29:13.651 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.651 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:29:13.651 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.651 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:29:13.651 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.980990+0000", 2024-05-15T13:29:13.651 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.651 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.652 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.652 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:29:13.652 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.980990+0000", 2024-05-15T13:29:13.652 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.652 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.652 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.652 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:29:13.652 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.980992+0000", 2024-05-15T13:29:13.652 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5969999999999999e-06 2024-05-15T13:29:13.652 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.652 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.652 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:29:13.652 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.980999+0000", 2024-05-15T13:29:13.652 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0129999999999996e-06 2024-05-15T13:29:13.652 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.653 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.653 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:29:13.653 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.981019+0000", 2024-05-15T13:29:13.653 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0015000000000001e-05 2024-05-15T13:29:13.653 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.653 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.653 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:29:13.653 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.981029+0000", 2024-05-15T13:29:13.653 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0219e-05 2024-05-15T13:29:13.653 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.653 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.653 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.653 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.981296+0000", 2024-05-15T13:29:13.653 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00026673000000000002 2024-05-15T13:29:13.653 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.654 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.654 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.654 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:28.981299+0000", 2024-05-15T13:29:13.654 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.478e-06 2024-05-15T13:29:13.654 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.654 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.654 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.654 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.342375+0000", 2024-05-15T13:29:13.654 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.361075761 2024-05-15T13:29:13.654 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.654 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.654 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.654 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.342381+0000", 2024-05-15T13:29:13.654 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.0510000000000002e-06 2024-05-15T13:29:13.655 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.655 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.655 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.655 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.823921+0000", 2024-05-15T13:29:13.655 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4815399280000001 2024-05-15T13:29:13.655 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.655 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.655 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:29:13.655 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.824034+0000", 2024-05-15T13:29:13.655 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000113132 2024-05-15T13:29:13.655 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.655 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.655 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 8,10,11", 2024-05-15T13:29:13.655 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.824135+0000", 2024-05-15T13:29:13.655 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00010098399999999999 2024-05-15T13:29:13.656 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.656 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.656 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.656 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.834540+0000", 2024-05-15T13:29:13.656 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.010405159000000001 2024-05-15T13:29:13.656 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.656 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.656 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.656 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.835346+0000", 2024-05-15T13:29:13.656 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00080584999999999999 2024-05-15T13:29:13.656 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.656 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.656 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:29:13.657 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.835358+0000", 2024-05-15T13:29:13.657 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1761e-05 2024-05-15T13:29:13.657 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.657 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.657 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.657 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.836612+0000", 2024-05-15T13:29:13.657 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001253702 2024-05-15T13:29:13.657 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.657 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.657 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:29:13.657 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.836667+0000", 2024-05-15T13:29:13.657 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5787e-05 2024-05-15T13:29:13.657 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.657 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.658 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:29:13.658 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.837037+0000", 2024-05-15T13:29:13.658 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00036981200000000002 2024-05-15T13:29:13.658 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.658 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:29:13.658 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.658 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.658 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.658 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:181113 9.0 9:03a921c4:::benchmark_data_smithi177_159919_object60870:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e921)", 2024-05-15T13:29:13.658 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:29.081659+0000", 2024-05-15T13:29:13.658 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 103.66093232599999, 2024-05-15T13:29:13.658 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7673728180000001, 2024-05-15T13:29:13.658 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:29:13.658 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:13.659 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:29:13.659 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:29:13.659 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:13.659 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 181113 2024-05-15T13:29:13.659 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.659 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:29:13.659 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.659 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:29:13.659 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.081659+0000", 2024-05-15T13:29:13.659 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.659 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.659 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.659 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:29:13.660 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.081659+0000", 2024-05-15T13:29:13.660 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.660 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.660 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.660 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:29:13.660 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.081661+0000", 2024-05-15T13:29:13.660 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.565e-06 2024-05-15T13:29:13.660 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.660 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.660 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:29:13.660 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.081670+0000", 2024-05-15T13:29:13.660 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.9199999999999993e-06 2024-05-15T13:29:13.660 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.660 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.661 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:29:13.661 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.081691+0000", 2024-05-15T13:29:13.661 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0777000000000001e-05 2024-05-15T13:29:13.661 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.661 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.661 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:29:13.661 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.081703+0000", 2024-05-15T13:29:13.661 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1357e-05 2024-05-15T13:29:13.661 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.661 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.661 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.661 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.081782+0000", 2024-05-15T13:29:13.661 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8954000000000003e-05 2024-05-15T13:29:13.661 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.662 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.662 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.662 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.081794+0000", 2024-05-15T13:29:13.662 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2449000000000001e-05 2024-05-15T13:29:13.662 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.662 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.662 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.662 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.342433+0000", 2024-05-15T13:29:13.662 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.26063863199999998 2024-05-15T13:29:13.662 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.662 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.662 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.662 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.342437+0000", 2024-05-15T13:29:13.662 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.9330000000000003e-06 2024-05-15T13:29:13.663 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.663 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.663 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.663 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.828119+0000", 2024-05-15T13:29:13.663 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.48568268 2024-05-15T13:29:13.663 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.663 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.663 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:29:13.663 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.828215+0000", 2024-05-15T13:29:13.663 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.5752000000000001e-05 2024-05-15T13:29:13.663 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.663 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.663 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 8,10,11", 2024-05-15T13:29:13.664 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.828371+0000", 2024-05-15T13:29:13.664 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015612699999999999 2024-05-15T13:29:13.664 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.665 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.665 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:29:13.665 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.835364+0000", 2024-05-15T13:29:13.665 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0069925339999999999 2024-05-15T13:29:13.665 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.665 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.665 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.665 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.835419+0000", 2024-05-15T13:29:13.665 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.5379e-05 2024-05-15T13:29:13.665 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.665 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.666 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.666 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.836057+0000", 2024-05-15T13:29:13.666 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000637807 2024-05-15T13:29:13.666 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.666 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.666 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.666 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.848211+0000", 2024-05-15T13:29:13.666 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.01215392 2024-05-15T13:29:13.666 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.667 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.667 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:29:13.667 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.848272+0000", 2024-05-15T13:29:13.667 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.1037000000000005e-05 2024-05-15T13:29:13.667 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.667 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.667 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:29:13.667 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.849032+0000", 2024-05-15T13:29:13.667 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00075999499999999996 2024-05-15T13:29:13.667 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.667 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:29:13.667 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.668 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.668 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.668 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:181119 9.0 9:0740e0e9:::benchmark_data_smithi177_159919_object60876:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e922)", 2024-05-15T13:29:13.668 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:29.559861+0000", 2024-05-15T13:29:13.668 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 103.182730179, 2024-05-15T13:29:13.668 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.332147902, 2024-05-15T13:29:13.668 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:29:13.668 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:13.668 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:29:13.668 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:29:13.668 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:13.669 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 181119 2024-05-15T13:29:13.669 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.669 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:29:13.669 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.669 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:29:13.669 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.559861+0000", 2024-05-15T13:29:13.669 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.669 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.669 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.669 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:29:13.669 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.559861+0000", 2024-05-15T13:29:13.669 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.670 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.670 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.670 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:29:13.670 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.559863+0000", 2024-05-15T13:29:13.670 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7990000000000001e-06 2024-05-15T13:29:13.670 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.670 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.670 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:29:13.670 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.559871+0000", 2024-05-15T13:29:13.670 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9049999999999997e-06 2024-05-15T13:29:13.671 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.671 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.671 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:29:13.671 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.559889+0000", 2024-05-15T13:29:13.671 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.8331000000000001e-05 2024-05-15T13:29:13.671 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.671 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.671 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:29:13.671 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.559900+0000", 2024-05-15T13:29:13.671 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0776e-05 2024-05-15T13:29:13.671 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.559959+0000", 2024-05-15T13:29:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.9456000000000001e-05 2024-05-15T13:29:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.559971+0000", 2024-05-15T13:29:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1379999999999999e-05 2024-05-15T13:29:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.837255+0000", 2024-05-15T13:29:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.277283985 2024-05-15T13:29:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:29:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.837411+0000", 2024-05-15T13:29:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015593200000000001 2024-05-15T13:29:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 8,10,11", 2024-05-15T13:29:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.837596+0000", 2024-05-15T13:29:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000185083 2024-05-15T13:29:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.849081+0000", 2024-05-15T13:29:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.011485386 2024-05-15T13:29:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.849128+0000", 2024-05-15T13:29:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6737999999999999e-05 2024-05-15T13:29:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.849175+0000", 2024-05-15T13:29:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6910999999999999e-05 2024-05-15T13:29:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:29:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.891194+0000", 2024-05-15T13:29:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.042019528 2024-05-15T13:29:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:29:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.891245+0000", 2024-05-15T13:29:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1149000000000001e-05 2024-05-15T13:29:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:29:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.892009+0000", 2024-05-15T13:29:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00076354299999999999 2024-05-15T13:29:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:29:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:181142 9.0 9:0868d3cf:::benchmark_data_smithi177_159919_object60899:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e922)", 2024-05-15T13:29:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:29.629127+0000", 2024-05-15T13:29:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 103.113464436, 2024-05-15T13:29:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.263271772, 2024-05-15T13:29:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:29:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:29:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:29:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 181142 2024-05-15T13:29:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:29:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:29:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.629127+0000", 2024-05-15T13:29:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:29:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.629127+0000", 2024-05-15T13:29:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.679 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.679 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.679 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:29:13.679 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.629129+0000", 2024-05-15T13:29:13.679 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.8700000000000001e-06 2024-05-15T13:29:13.679 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.679 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.679 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:29:13.679 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.629141+0000", 2024-05-15T13:29:13.679 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1990999999999999e-05 2024-05-15T13:29:13.679 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.679 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:29:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.629165+0000", 2024-05-15T13:29:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.4476000000000001e-05 2024-05-15T13:29:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:29:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.629175+0000", 2024-05-15T13:29:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0189e-05 2024-05-15T13:29:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.629246+0000", 2024-05-15T13:29:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0149999999999998e-05 2024-05-15T13:29:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.629256+0000", 2024-05-15T13:29:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0349999999999999e-05 2024-05-15T13:29:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.842793+0000", 2024-05-15T13:29:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.213536707 2024-05-15T13:29:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.682 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:29:13.682 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.842945+0000", 2024-05-15T13:29:13.682 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015246199999999999 2024-05-15T13:29:13.682 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.682 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.682 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 8,10,11", 2024-05-15T13:29:13.682 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.843104+0000", 2024-05-15T13:29:13.682 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000159254 2024-05-15T13:29:13.682 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.682 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.682 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.683 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.859228+0000", 2024-05-15T13:29:13.683 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.016124000999999999 2024-05-15T13:29:13.683 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.683 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.683 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.683 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.859281+0000", 2024-05-15T13:29:13.683 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.3145e-05 2024-05-15T13:29:13.683 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.683 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.683 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.683 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.859331+0000", 2024-05-15T13:29:13.683 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9981999999999997e-05 2024-05-15T13:29:13.684 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.684 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.684 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:29:13.684 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.892019+0000", 2024-05-15T13:29:13.684 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.032687411999999999 2024-05-15T13:29:13.684 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.684 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.684 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:29:13.684 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.892065+0000", 2024-05-15T13:29:13.684 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.5945000000000001e-05 2024-05-15T13:29:13.684 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.684 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.684 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:29:13.684 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.892399+0000", 2024-05-15T13:29:13.685 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00033383799999999998 2024-05-15T13:29:13.685 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.685 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:29:13.685 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.685 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.685 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.685 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:181165 9.0 9:0ee3f67e:::benchmark_data_smithi177_159919_object60922:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e922)", 2024-05-15T13:29:13.685 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:29.707966+0000", 2024-05-15T13:29:13.685 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 103.034625679, 2024-05-15T13:29:13.685 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1848165749999999, 2024-05-15T13:29:13.685 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:29:13.685 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:13.685 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:29:13.686 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:29:13.686 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:13.686 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 181165 2024-05-15T13:29:13.686 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.686 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:29:13.686 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.686 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:29:13.686 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.707966+0000", 2024-05-15T13:29:13.686 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.686 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.686 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.687 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:29:13.687 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.707966+0000", 2024-05-15T13:29:13.687 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.687 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.687 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.687 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:29:13.687 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.707968+0000", 2024-05-15T13:29:13.687 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0090000000000002e-06 2024-05-15T13:29:13.687 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.687 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.687 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:29:13.687 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.707977+0000", 2024-05-15T13:29:13.688 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.4360000000000008e-06 2024-05-15T13:29:13.688 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.688 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.688 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:29:13.688 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.707999+0000", 2024-05-15T13:29:13.688 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2183000000000001e-05 2024-05-15T13:29:13.688 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.688 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.688 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:29:13.689 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.708010+0000", 2024-05-15T13:29:13.689 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1131999999999999e-05 2024-05-15T13:29:13.689 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.689 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.689 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.689 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.708085+0000", 2024-05-15T13:29:13.689 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.5123999999999994e-05 2024-05-15T13:29:13.689 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.689 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.689 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.689 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.708097+0000", 2024-05-15T13:29:13.689 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2033e-05 2024-05-15T13:29:13.690 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.690 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.690 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.690 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.849209+0000", 2024-05-15T13:29:13.690 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.141111674 2024-05-15T13:29:13.690 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.690 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.690 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:29:13.690 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.849345+0000", 2024-05-15T13:29:13.690 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013578 2024-05-15T13:29:13.690 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.691 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.691 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 8,10,11", 2024-05-15T13:29:13.691 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.849494+0000", 2024-05-15T13:29:13.691 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000148918 2024-05-15T13:29:13.691 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.691 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.691 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.691 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.869373+0000", 2024-05-15T13:29:13.691 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.019879221999999998 2024-05-15T13:29:13.691 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.691 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.691 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.691 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.879203+0000", 2024-05-15T13:29:13.692 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0098302449999999996 2024-05-15T13:29:13.692 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.692 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.692 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.692 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.880096+0000", 2024-05-15T13:29:13.692 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00089305799999999996 2024-05-15T13:29:13.692 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.692 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.692 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:29:13.692 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.892407+0000", 2024-05-15T13:29:13.692 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.012310585000000001 2024-05-15T13:29:13.692 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.692 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.693 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:29:13.693 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.892451+0000", 2024-05-15T13:29:13.693 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4409000000000001e-05 2024-05-15T13:29:13.693 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.693 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.693 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:29:13.693 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.892782+0000", 2024-05-15T13:29:13.693 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00033076699999999999 2024-05-15T13:29:13.693 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.693 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:29:13.693 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.693 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.694 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.694 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:181167 9.0 9:0c9fd18d:::benchmark_data_smithi177_159919_object60924:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e922)", 2024-05-15T13:29:13.694 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:29.712623+0000", 2024-05-15T13:29:13.694 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 103.029968037, 2024-05-15T13:29:13.694 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1805542099999999, 2024-05-15T13:29:13.694 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:29:13.694 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:13.694 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:29:13.694 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:29:13.694 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:13.694 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 181167 2024-05-15T13:29:13.694 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.695 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:29:13.695 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.695 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:29:13.695 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.712623+0000", 2024-05-15T13:29:13.695 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.695 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.695 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.695 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:29:13.695 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.712623+0000", 2024-05-15T13:29:13.695 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.695 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.695 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.696 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:29:13.696 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.712625+0000", 2024-05-15T13:29:13.696 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4479999999999999e-06 2024-05-15T13:29:13.696 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.696 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.696 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:29:13.696 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.712631+0000", 2024-05-15T13:29:13.696 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.8109999999999997e-06 2024-05-15T13:29:13.696 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.696 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.696 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:29:13.697 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.712650+0000", 2024-05-15T13:29:13.697 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.8669e-05 2024-05-15T13:29:13.697 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.697 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.697 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:29:13.697 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.712658+0000", 2024-05-15T13:29:13.697 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.1329999999999999e-06 2024-05-15T13:29:13.697 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.697 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.697 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.697 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.712707+0000", 2024-05-15T13:29:13.697 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9094000000000001e-05 2024-05-15T13:29:13.697 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.698 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.698 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.698 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.712715+0000", 2024-05-15T13:29:13.698 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9510000000000006e-06 2024-05-15T13:29:13.698 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.698 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.698 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.698 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.853976+0000", 2024-05-15T13:29:13.698 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1412602729999999 2024-05-15T13:29:13.698 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.698 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.699 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:29:13.699 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.854104+0000", 2024-05-15T13:29:13.699 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012823500000000001 2024-05-15T13:29:13.699 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.699 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.699 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 8,10,11", 2024-05-15T13:29:13.699 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.854202+0000", 2024-05-15T13:29:13.699 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.7707000000000005e-05 2024-05-15T13:29:13.699 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.699 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.699 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.699 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.869413+0000", 2024-05-15T13:29:13.700 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.015211886000000001 2024-05-15T13:29:13.700 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.700 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.700 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.700 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.880056+0000", 2024-05-15T13:29:13.700 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.010642380999999999 2024-05-15T13:29:13.700 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.700 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.700 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.700 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.880208+0000", 2024-05-15T13:29:13.700 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015206200000000001 2024-05-15T13:29:13.700 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.701 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.701 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:29:13.701 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.892791+0000", 2024-05-15T13:29:13.701 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.012582797 2024-05-15T13:29:13.701 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.701 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.701 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:29:13.701 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.892835+0000", 2024-05-15T13:29:13.701 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.477e-05 2024-05-15T13:29:13.701 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.701 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.702 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:29:13.702 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.893177+0000", 2024-05-15T13:29:13.702 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00034199300000000002 2024-05-15T13:29:13.702 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.702 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:29:13.702 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.702 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.702 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.702 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:181191 9.0 9:04219414:::benchmark_data_smithi177_159919_object60948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e922)", 2024-05-15T13:29:13.702 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:29.794890+0000", 2024-05-15T13:29:13.702 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 102.94770096800001, 2024-05-15T13:29:13.702 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0986727730000001, 2024-05-15T13:29:13.703 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:29:13.703 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:13.703 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:29:13.703 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:29:13.703 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:13.703 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 181191 2024-05-15T13:29:13.703 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.703 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:29:13.703 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.703 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:29:13.703 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.794890+0000", 2024-05-15T13:29:13.703 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.704 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.704 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.704 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:29:13.704 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.794890+0000", 2024-05-15T13:29:13.704 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.704 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.704 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.704 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:29:13.704 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.794892+0000", 2024-05-15T13:29:13.704 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.747e-06 2024-05-15T13:29:13.704 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.704 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.704 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:29:13.705 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.794899+0000", 2024-05-15T13:29:13.705 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0960000000000003e-06 2024-05-15T13:29:13.705 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.705 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.705 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:29:13.705 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.794918+0000", 2024-05-15T13:29:13.705 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9082000000000001e-05 2024-05-15T13:29:13.705 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.705 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.705 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:29:13.705 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.794929+0000", 2024-05-15T13:29:13.705 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0334e-05 2024-05-15T13:29:13.705 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.706 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.706 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.706 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.794999+0000", 2024-05-15T13:29:13.706 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0340999999999996e-05 2024-05-15T13:29:13.706 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.706 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.706 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.706 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.795014+0000", 2024-05-15T13:29:13.706 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.5271999999999999e-05 2024-05-15T13:29:13.706 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.706 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.706 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.707 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.859364+0000", 2024-05-15T13:29:13.707 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.064349913 2024-05-15T13:29:13.707 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.707 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.707 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:29:13.707 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.859529+0000", 2024-05-15T13:29:13.707 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00016480100000000001 2024-05-15T13:29:13.707 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.707 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.707 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 8,10,11", 2024-05-15T13:29:13.707 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.859692+0000", 2024-05-15T13:29:13.707 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00016296799999999999 2024-05-15T13:29:13.708 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.708 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.708 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.708 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.880135+0000", 2024-05-15T13:29:13.708 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.020443625999999999 2024-05-15T13:29:13.708 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.708 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.708 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.708 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.880172+0000", 2024-05-15T13:29:13.708 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.6504999999999997e-05 2024-05-15T13:29:13.708 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.708 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.708 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.709 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.880281+0000", 2024-05-15T13:29:13.709 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000109444 2024-05-15T13:29:13.709 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.709 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.709 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:29:13.709 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.893190+0000", 2024-05-15T13:29:13.709 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.012908118 2024-05-15T13:29:13.709 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.709 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.709 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:29:13.709 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.893234+0000", 2024-05-15T13:29:13.709 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4165000000000002e-05 2024-05-15T13:29:13.709 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.709 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.710 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:29:13.710 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.893563+0000", 2024-05-15T13:29:13.710 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00032936100000000002 2024-05-15T13:29:13.710 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.710 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:29:13.710 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.710 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.710 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.710 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:181200 9.0 9:03e2deff:::benchmark_data_smithi177_159919_object60957:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e922)", 2024-05-15T13:29:13.710 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:29.845276+0000", 2024-05-15T13:29:13.710 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 102.897315396, 2024-05-15T13:29:13.710 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.048674551, 2024-05-15T13:29:13.711 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:29:13.711 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:13.711 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:29:13.711 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:29:13.711 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:13.711 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 181200 2024-05-15T13:29:13.711 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.711 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:29:13.711 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.711 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:29:13.711 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.845276+0000", 2024-05-15T13:29:13.711 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.711 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.712 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.712 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:29:13.712 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.845276+0000", 2024-05-15T13:29:13.712 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.712 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.712 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.712 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:29:13.712 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.845278+0000", 2024-05-15T13:29:13.712 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0099999999999998e-06 2024-05-15T13:29:13.712 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.712 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.712 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:29:13.712 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.845286+0000", 2024-05-15T13:29:13.712 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.4600000000000003e-06 2024-05-15T13:29:13.713 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.713 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.713 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:29:13.713 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.845308+0000", 2024-05-15T13:29:13.713 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1378000000000001e-05 2024-05-15T13:29:13.713 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.713 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.713 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:29:13.713 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.845319+0000", 2024-05-15T13:29:13.713 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0976e-05 2024-05-15T13:29:13.713 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.713 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.713 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.714 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.846796+0000", 2024-05-15T13:29:13.714 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00147777 2024-05-15T13:29:13.714 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.714 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.714 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.714 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:29.846808+0000", 2024-05-15T13:29:13.714 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1355999999999999e-05 2024-05-15T13:29:13.714 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.714 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.714 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.714 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.864372+0000", 2024-05-15T13:29:13.714 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.017564342 2024-05-15T13:29:13.715 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.715 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.715 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:29:13.715 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.864512+0000", 2024-05-15T13:29:13.715 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013993400000000001 2024-05-15T13:29:13.715 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.715 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.715 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 8,10,11", 2024-05-15T13:29:13.715 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.864659+0000", 2024-05-15T13:29:13.715 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000146911 2024-05-15T13:29:13.715 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.715 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.715 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.716 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.880245+0000", 2024-05-15T13:29:13.716 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.015585699 2024-05-15T13:29:13.716 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.716 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.716 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.716 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.890943+0000", 2024-05-15T13:29:13.716 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.010698327000000001 2024-05-15T13:29:13.716 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.716 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.716 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.716 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.890992+0000", 2024-05-15T13:29:13.717 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.8687000000000003e-05 2024-05-15T13:29:13.717 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.717 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.717 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:29:13.717 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.893577+0000", 2024-05-15T13:29:13.717 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0025854290000000002 2024-05-15T13:29:13.717 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.717 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.717 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:29:13.717 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.893621+0000", 2024-05-15T13:29:13.717 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4082999999999998e-05 2024-05-15T13:29:13.718 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.718 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.718 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:29:13.718 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:30.893950+0000", 2024-05-15T13:29:13.718 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00032918900000000002 2024-05-15T13:29:13.718 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.718 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:29:13.718 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.718 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.718 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.718 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:193559 9.0 9:05de1ad0:::benchmark_data_smithi177_159919_object73316:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e936)", 2024-05-15T13:29:13.719 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:57.417438+0000", 2024-05-15T13:29:13.719 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 75.325153194999999, 2024-05-15T13:29:13.719 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1072661679999998, 2024-05-15T13:29:13.719 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:29:13.719 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:13.719 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:29:13.719 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:29:13.719 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:13.719 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 193559 2024-05-15T13:29:13.719 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.719 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:29:13.719 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.719 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:29:13.719 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.417438+0000", 2024-05-15T13:29:13.720 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.720 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.720 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.720 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:29:13.720 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.417438+0000", 2024-05-15T13:29:13.720 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.720 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.720 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.720 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:29:13.720 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.417440+0000", 2024-05-15T13:29:13.720 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.699e-06 2024-05-15T13:29:13.720 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.720 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.720 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:29:13.721 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.417446+0000", 2024-05-15T13:29:13.721 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3219999999999998e-06 2024-05-15T13:29:13.721 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.721 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.721 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:29:13.721 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.417459+0000", 2024-05-15T13:29:13.721 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3118e-05 2024-05-15T13:29:13.721 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.721 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.721 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:29:13.721 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.417468+0000", 2024-05-15T13:29:13.721 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.7709999999999995e-06 2024-05-15T13:29:13.721 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.722 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.722 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.722 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.417539+0000", 2024-05-15T13:29:13.722 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0766999999999995e-05 2024-05-15T13:29:13.722 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.722 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.722 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op must wait for map", 2024-05-15T13:29:13.722 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.417546+0000", 2024-05-15T13:29:13.722 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.3239999999999997e-06 2024-05-15T13:29:13.722 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.722 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.723 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.723 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.420414+0000", 2024-05-15T13:29:13.723 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.002867709 2024-05-15T13:29:13.723 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.723 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.723 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.723 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.420422+0000", 2024-05-15T13:29:13.723 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.5459999999999992e-06 2024-05-15T13:29:13.723 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.723 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.723 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.723 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.418626+0000", 2024-05-15T13:29:13.723 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.99820408100000002 2024-05-15T13:29:13.724 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.724 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.724 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.724 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.418639+0000", 2024-05-15T13:29:13.724 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2984e-05 2024-05-15T13:29:13.724 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.724 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.724 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.724 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.453691+0000", 2024-05-15T13:29:13.724 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0350517560000001 2024-05-15T13:29:13.724 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.725 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.725 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:29:13.725 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.453845+0000", 2024-05-15T13:29:13.725 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015417299999999999 2024-05-15T13:29:13.725 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.725 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.725 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 8,10,11", 2024-05-15T13:29:13.725 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.453999+0000", 2024-05-15T13:29:13.725 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00015369400000000001 2024-05-15T13:29:13.725 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.725 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.725 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.725 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.469643+0000", 2024-05-15T13:29:13.726 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.015643688999999999 2024-05-15T13:29:13.726 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.726 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.726 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:29:13.726 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.469732+0000", 2024-05-15T13:29:13.726 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.9219000000000001e-05 2024-05-15T13:29:13.726 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.726 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.726 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.726 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.479194+0000", 2024-05-15T13:29:13.726 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00946169 2024-05-15T13:29:13.726 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.726 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.727 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.727 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.524557+0000", 2024-05-15T13:29:13.727 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.045363497000000003 2024-05-15T13:29:13.727 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.727 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.727 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:29:13.727 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.524604+0000", 2024-05-15T13:29:13.727 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6490999999999999e-05 2024-05-15T13:29:13.727 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.727 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.727 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:29:13.727 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.524704+0000", 2024-05-15T13:29:13.727 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00010063799999999999 2024-05-15T13:29:13.727 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.728 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:29:13.728 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.728 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.728 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.728 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:193565 9.0 9:0605450e:::benchmark_data_smithi177_159919_object73322:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e936)", 2024-05-15T13:29:13.728 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:57.430377+0000", 2024-05-15T13:29:13.728 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 75.312214694000005, 2024-05-15T13:29:13.728 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0945958550000001, 2024-05-15T13:29:13.728 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:29:13.728 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:13.728 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:29:13.728 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:29:13.728 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:13.729 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 193565 2024-05-15T13:29:13.729 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.729 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:29:13.729 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.729 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:29:13.729 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.430377+0000", 2024-05-15T13:29:13.729 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.729 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.729 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.729 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:29:13.729 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.430377+0000", 2024-05-15T13:29:13.729 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.729 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.729 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.730 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:29:13.730 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.430378+0000", 2024-05-15T13:29:13.730 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.291e-06 2024-05-15T13:29:13.730 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.730 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.730 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:29:13.730 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.430382+0000", 2024-05-15T13:29:13.730 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.4530000000000004e-06 2024-05-15T13:29:13.730 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.730 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.731 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:29:13.731 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.430392+0000", 2024-05-15T13:29:13.731 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0086e-05 2024-05-15T13:29:13.731 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.731 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.731 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:29:13.731 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.430403+0000", 2024-05-15T13:29:13.731 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0209e-05 2024-05-15T13:29:13.731 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.731 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.731 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.732 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.431023+0000", 2024-05-15T13:29:13.732 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00062074299999999999 2024-05-15T13:29:13.732 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.732 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.732 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.732 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.431036+0000", 2024-05-15T13:29:13.732 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.2554e-05 2024-05-15T13:29:13.732 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.732 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.732 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.732 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.418664+0000", 2024-05-15T13:29:13.732 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.98762816200000003 2024-05-15T13:29:13.732 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.733 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.733 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.733 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.418672+0000", 2024-05-15T13:29:13.733 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2800000000000003e-06 2024-05-15T13:29:13.733 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.733 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.733 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.733 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.459054+0000", 2024-05-15T13:29:13.733 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.040381751 2024-05-15T13:29:13.733 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.733 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.733 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:29:13.733 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.459180+0000", 2024-05-15T13:29:13.733 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012609000000000001 2024-05-15T13:29:13.734 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.734 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.734 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 8,10,11", 2024-05-15T13:29:13.734 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.459317+0000", 2024-05-15T13:29:13.734 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000137193 2024-05-15T13:29:13.734 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.734 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.734 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.734 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.469710+0000", 2024-05-15T13:29:13.734 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.010393079 2024-05-15T13:29:13.734 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.734 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.734 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:29:13.735 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.469740+0000", 2024-05-15T13:29:13.735 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.9101000000000001e-05 2024-05-15T13:29:13.735 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.735 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.735 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.735 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.479238+0000", 2024-05-15T13:29:13.735 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0094987400000000003 2024-05-15T13:29:13.735 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.735 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.735 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.735 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.524771+0000", 2024-05-15T13:29:13.735 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.045532950000000003 2024-05-15T13:29:13.735 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.736 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.736 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:29:13.736 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.524830+0000", 2024-05-15T13:29:13.736 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.8279000000000002e-05 2024-05-15T13:29:13.736 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.736 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.736 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:29:13.736 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.524972+0000", 2024-05-15T13:29:13.736 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00014289399999999999 2024-05-15T13:29:13.736 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.736 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:29:13.736 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.736 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.736 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.737 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:193570 9.0 9:0959ca73:::benchmark_data_smithi177_159919_object73327:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e936)", 2024-05-15T13:29:13.737 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:57.454945+0000", 2024-05-15T13:29:13.737 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 75.287645846000004, 2024-05-15T13:29:13.737 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0701821210000002, 2024-05-15T13:29:13.737 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:29:13.737 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:13.737 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:29:13.737 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:29:13.737 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:13.737 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 193570 2024-05-15T13:29:13.737 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.737 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:29:13.737 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.738 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:29:13.738 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454945+0000", 2024-05-15T13:29:13.738 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.738 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.738 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.738 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:29:13.738 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454945+0000", 2024-05-15T13:29:13.738 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.738 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.738 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.738 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:29:13.738 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454947+0000", 2024-05-15T13:29:13.739 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.469e-06 2024-05-15T13:29:13.739 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.739 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.739 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:29:13.739 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454955+0000", 2024-05-15T13:29:13.739 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.7780000000000007e-06 2024-05-15T13:29:13.739 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.739 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.739 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:29:13.739 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454977+0000", 2024-05-15T13:29:13.739 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.2330000000000001e-05 2024-05-15T13:29:13.739 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.739 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.740 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:29:13.740 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.454988+0000", 2024-05-15T13:29:13.740 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0548e-05 2024-05-15T13:29:13.740 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.740 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.740 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.740 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.455085+0000", 2024-05-15T13:29:13.740 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.7830000000000004e-05 2024-05-15T13:29:13.740 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.740 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.740 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.740 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.455099+0000", 2024-05-15T13:29:13.740 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3624e-05 2024-05-15T13:29:13.741 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.741 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.741 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.741 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.418694+0000", 2024-05-15T13:29:13.741 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.963595482 2024-05-15T13:29:13.741 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.741 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.741 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.741 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.418702+0000", 2024-05-15T13:29:13.741 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9500000000000001e-06 2024-05-15T13:29:13.741 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.741 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.741 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.742 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.464208+0000", 2024-05-15T13:29:13.742 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0455056009999999 2024-05-15T13:29:13.742 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.742 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.742 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:29:13.742 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.464337+0000", 2024-05-15T13:29:13.742 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000128799 2024-05-15T13:29:13.742 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.742 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.742 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 8,10,11", 2024-05-15T13:29:13.742 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.464476+0000", 2024-05-15T13:29:13.742 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013928599999999999 2024-05-15T13:29:13.742 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.743 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.743 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.743 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.479285+0000", 2024-05-15T13:29:13.743 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.014808684000000001 2024-05-15T13:29:13.743 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.743 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.743 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:29:13.743 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.479302+0000", 2024-05-15T13:29:13.743 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.7144999999999999e-05 2024-05-15T13:29:13.743 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.743 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.743 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.744 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.482941+0000", 2024-05-15T13:29:13.744 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.003639382 2024-05-15T13:29:13.744 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.744 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.744 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.744 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.525014+0000", 2024-05-15T13:29:13.744 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.042072304999999997 2024-05-15T13:29:13.744 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.744 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.744 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:29:13.744 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.525049+0000", 2024-05-15T13:29:13.744 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.5505e-05 2024-05-15T13:29:13.744 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.745 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.745 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:29:13.745 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.525128+0000", 2024-05-15T13:29:13.745 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.8403000000000001e-05 2024-05-15T13:29:13.745 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.745 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:29:13.745 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.745 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.745 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.745 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:193573 9.0 9:0a633edd:::benchmark_data_smithi177_159919_object73330:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e936)", 2024-05-15T13:29:13.745 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:57.455019+0000", 2024-05-15T13:29:13.745 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 75.287572208, 2024-05-15T13:29:13.745 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0703574659999999, 2024-05-15T13:29:13.745 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:29:13.746 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:13.746 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:29:13.746 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:29:13.746 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:13.746 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 193573 2024-05-15T13:29:13.746 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.746 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:29:13.746 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.746 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:29:13.746 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.455019+0000", 2024-05-15T13:29:13.746 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.746 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.747 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.747 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:29:13.747 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.455019+0000", 2024-05-15T13:29:13.747 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.747 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.747 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.747 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:29:13.747 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.455020+0000", 2024-05-15T13:29:13.747 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.4099999999999999e-07 2024-05-15T13:29:13.747 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.747 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.747 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:29:13.747 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.455023+0000", 2024-05-15T13:29:13.747 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.9790000000000002e-06 2024-05-15T13:29:13.748 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.748 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.748 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:29:13.748 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.455031+0000", 2024-05-15T13:29:13.748 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.2600000000000005e-06 2024-05-15T13:29:13.748 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.748 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.748 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:29:13.748 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.455038+0000", 2024-05-15T13:29:13.748 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.0249999999999997e-06 2024-05-15T13:29:13.748 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.748 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.748 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.749 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.455131+0000", 2024-05-15T13:29:13.749 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.2942000000000006e-05 2024-05-15T13:29:13.749 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.749 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.749 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.749 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.455140+0000", 2024-05-15T13:29:13.749 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8200000000000003e-06 2024-05-15T13:29:13.749 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.749 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.749 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.749 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.418724+0000", 2024-05-15T13:29:13.749 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.96358423500000001 2024-05-15T13:29:13.750 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.750 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.750 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.750 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.418738+0000", 2024-05-15T13:29:13.750 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.4431e-05 2024-05-15T13:29:13.750 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.750 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.750 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.750 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.469785+0000", 2024-05-15T13:29:13.750 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.05104654 2024-05-15T13:29:13.750 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.750 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.751 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:29:13.751 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.469925+0000", 2024-05-15T13:29:13.751 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000139705 2024-05-15T13:29:13.751 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.751 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.751 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 8,10,11", 2024-05-15T13:29:13.751 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.470078+0000", 2024-05-15T13:29:13.751 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000153595 2024-05-15T13:29:13.751 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.751 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.751 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:29:13.751 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.479309+0000", 2024-05-15T13:29:13.751 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0092308939999999999 2024-05-15T13:29:13.752 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.752 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.752 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.752 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.480786+0000", 2024-05-15T13:29:13.752 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.001477318 2024-05-15T13:29:13.752 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.752 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.752 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.752 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.488986+0000", 2024-05-15T13:29:13.752 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0082001030000000003 2024-05-15T13:29:13.752 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.752 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.753 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.753 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.525178+0000", 2024-05-15T13:29:13.753 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.036191521999999997 2024-05-15T13:29:13.753 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.753 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.753 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:29:13.753 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.525238+0000", 2024-05-15T13:29:13.753 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.0013999999999998e-05 2024-05-15T13:29:13.753 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.753 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.753 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:29:13.753 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.525377+0000", 2024-05-15T13:29:13.753 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00013854199999999999 2024-05-15T13:29:13.753 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.754 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:29:13.754 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.754 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.754 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.754 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:193574 9.0 9:0180b66f:::benchmark_data_smithi177_159919_object73331:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e936)", 2024-05-15T13:29:13.754 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:27:57.466052+0000", 2024-05-15T13:29:13.754 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 75.276539396999993, 2024-05-15T13:29:13.754 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.0594722839999999, 2024-05-15T13:29:13.754 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:29:13.754 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:13.754 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:29:13.754 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:29:13.755 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:13.755 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 193574 2024-05-15T13:29:13.755 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.755 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:29:13.755 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.755 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:29:13.755 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.466052+0000", 2024-05-15T13:29:13.755 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.755 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.755 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.755 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:29:13.755 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.466052+0000", 2024-05-15T13:29:13.755 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.755 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.756 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.756 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:29:13.756 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.466054+0000", 2024-05-15T13:29:13.756 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9630000000000001e-06 2024-05-15T13:29:13.756 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.756 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.756 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:29:13.756 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.466064+0000", 2024-05-15T13:29:13.756 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 9.7289999999999995e-06 2024-05-15T13:29:13.756 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.756 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.756 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:29:13.756 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.466083+0000", 2024-05-15T13:29:13.756 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.9364999999999998e-05 2024-05-15T13:29:13.757 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.757 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.757 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:29:13.757 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.466095+0000", 2024-05-15T13:29:13.757 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.1593000000000001e-05 2024-05-15T13:29:13.757 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.757 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.757 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.757 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.466219+0000", 2024-05-15T13:29:13.757 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000124769 2024-05-15T13:29:13.757 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.757 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.757 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.758 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:57.466233+0000", 2024-05-15T13:29:13.758 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3674e-05 2024-05-15T13:29:13.758 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.758 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.758 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.758 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.418761+0000", 2024-05-15T13:29:13.758 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.95252785600000001 2024-05-15T13:29:13.758 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.758 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.758 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for peered", 2024-05-15T13:29:13.758 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:58.418769+0000", 2024-05-15T13:29:13.758 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.9510000000000006e-06 2024-05-15T13:29:13.758 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.759 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.759 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.759 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.474299+0000", 2024-05-15T13:29:13.759 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.0555298660000001 2024-05-15T13:29:13.759 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.759 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.759 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:29:13.759 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.474423+0000", 2024-05-15T13:29:13.759 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.000124647 2024-05-15T13:29:13.759 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.759 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.760 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 8,10,11", 2024-05-15T13:29:13.760 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.474510+0000", 2024-05-15T13:29:13.760 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.6427000000000004e-05 2024-05-15T13:29:13.760 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.760 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.760 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:29:13.760 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.479315+0000", 2024-05-15T13:29:13.760 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0048054630000000003 2024-05-15T13:29:13.760 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.760 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.760 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.760 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.480823+0000", 2024-05-15T13:29:13.761 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0015079169999999999 2024-05-15T13:29:13.761 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.761 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.761 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.761 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.489081+0000", 2024-05-15T13:29:13.761 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0082579490000000005 2024-05-15T13:29:13.761 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.761 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.761 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.761 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.525419+0000", 2024-05-15T13:29:13.761 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.036337787000000003 2024-05-15T13:29:13.761 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.762 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.762 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:29:13.762 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.525451+0000", 2024-05-15T13:29:13.762 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.2329999999999997e-05 2024-05-15T13:29:13.762 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.762 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.762 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:29:13.762 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:27:59.525524+0000", 2024-05-15T13:29:13.762 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.2997999999999995e-05 2024-05-15T13:29:13.762 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.762 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:29:13.763 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.763 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.763 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.763 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "MOSDPGPushReply(9.0 941/937 [PushReplyOp(9:0a30986d:::benchmark_data_smithi177_159919_object119058:head)])", 2024-05-15T13:29:13.763 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:28:03.408980+0000", 2024-05-15T13:29:13.763 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 69.333610999000001, 2024-05-15T13:29:13.763 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.051794690999999997, 2024-05-15T13:29:13.763 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:29:13.763 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "reached pg", 2024-05-15T13:29:13.763 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:29:13.763 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.763 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:29:13.764 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:03.408980+0000", 2024-05-15T13:29:13.764 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.764 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.764 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.764 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:29:13.764 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:03.408980+0000", 2024-05-15T13:29:13.764 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.764 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.764 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.764 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:29:13.764 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:03.408980+0000", 2024-05-15T13:29:13.764 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.1400000000000001e-07 2024-05-15T13:29:13.765 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.765 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.765 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:29:13.765 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:03.408984+0000", 2024-05-15T13:29:13.765 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 3.0539999999999999e-06 2024-05-15T13:29:13.765 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.765 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.765 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:29:13.765 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:03.408991+0000", 2024-05-15T13:29:13.765 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.1559999999999998e-06 2024-05-15T13:29:13.765 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.765 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.766 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:29:13.766 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:03.408998+0000", 2024-05-15T13:29:13.766 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 7.6899999999999992e-06 2024-05-15T13:29:13.766 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.766 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.766 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.766 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:03.409050+0000", 2024-05-15T13:29:13.766 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 5.1651999999999999e-05 2024-05-15T13:29:13.766 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.766 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.766 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:29:13.767 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:03.460775+0000", 2024-05-15T13:29:13.767 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.051724924999999998 2024-05-15T13:29:13.767 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.767 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:29:13.767 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.767 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.767 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.767 INFO:teuthology.orchestra.run.smithi120.stdout: "description": "osd_op(client.26536.0:226171 9.0 9:0b464755:::benchmark_data_smithi177_159919_object105928:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e968)", 2024-05-15T13:29:13.767 INFO:teuthology.orchestra.run.smithi120.stdout: "initiated_at": "2024-05-15T13:28:56.980764+0000", 2024-05-15T13:29:13.767 INFO:teuthology.orchestra.run.smithi120.stdout: "age": 15.761827626000001, 2024-05-15T13:29:13.767 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.048042689999999999, 2024-05-15T13:29:13.767 INFO:teuthology.orchestra.run.smithi120.stdout: "type_data": { 2024-05-15T13:29:13.768 INFO:teuthology.orchestra.run.smithi120.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:13.768 INFO:teuthology.orchestra.run.smithi120.stdout: "client_info": { 2024-05-15T13:29:13.768 INFO:teuthology.orchestra.run.smithi120.stdout: "client": "client.26536", 2024-05-15T13:29:13.768 INFO:teuthology.orchestra.run.smithi120.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:13.768 INFO:teuthology.orchestra.run.smithi120.stdout: "tid": 226171 2024-05-15T13:29:13.768 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.768 INFO:teuthology.orchestra.run.smithi120.stdout: "events": [ 2024-05-15T13:29:13.768 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.768 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "initiated", 2024-05-15T13:29:13.768 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:56.980764+0000", 2024-05-15T13:29:13.768 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.768 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.769 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.769 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "header_read", 2024-05-15T13:29:13.769 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:56.980764+0000", 2024-05-15T13:29:13.769 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0 2024-05-15T13:29:13.769 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.769 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.769 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "throttled", 2024-05-15T13:29:13.769 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:56.980766+0000", 2024-05-15T13:29:13.769 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 2.165e-06 2024-05-15T13:29:13.769 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.769 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.770 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "all_read", 2024-05-15T13:29:13.770 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:56.980772+0000", 2024-05-15T13:29:13.770 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 6.3069999999999999e-06 2024-05-15T13:29:13.770 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.770 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.770 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "dispatched", 2024-05-15T13:29:13.770 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:56.980786+0000", 2024-05-15T13:29:13.770 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 1.3886999999999999e-05 2024-05-15T13:29:13.770 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.770 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.770 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "queued_for_pg", 2024-05-15T13:29:13.771 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:56.980795+0000", 2024-05-15T13:29:13.771 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 8.8659999999999995e-06 2024-05-15T13:29:13.771 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.771 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.771 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "reached_pg", 2024-05-15T13:29:13.771 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:56.984151+0000", 2024-05-15T13:29:13.771 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.0033565840000000001 2024-05-15T13:29:13.771 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.771 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.771 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "started", 2024-05-15T13:29:13.771 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:56.984276+0000", 2024-05-15T13:29:13.771 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012464900000000001 2024-05-15T13:29:13.771 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.772 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.772 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "waiting for subops from 8,11", 2024-05-15T13:29:13.772 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:56.984397+0000", 2024-05-15T13:29:13.772 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00012113899999999999 2024-05-15T13:29:13.772 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.772 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.772 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.772 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:57.002191+0000", 2024-05-15T13:29:13.772 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.017794053000000001 2024-05-15T13:29:13.772 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.772 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.773 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:13.773 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:57.002444+0000", 2024-05-15T13:29:13.773 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.00025255399999999999 2024-05-15T13:29:13.773 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.773 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.773 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "op_commit", 2024-05-15T13:29:13.773 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:57.028710+0000", 2024-05-15T13:29:13.773 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 0.026266080000000001 2024-05-15T13:29:13.773 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.773 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.774 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "commit_sent", 2024-05-15T13:29:13.774 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:57.028756+0000", 2024-05-15T13:29:13.774 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.6418000000000002e-05 2024-05-15T13:29:13.774 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-05-15T13:29:13.774 INFO:teuthology.orchestra.run.smithi120.stdout: { 2024-05-15T13:29:13.774 INFO:teuthology.orchestra.run.smithi120.stdout: "event": "done", 2024-05-15T13:29:13.774 INFO:teuthology.orchestra.run.smithi120.stdout: "time": "2024-05-15T13:28:57.028806+0000", 2024-05-15T13:29:13.774 INFO:teuthology.orchestra.run.smithi120.stdout: "duration": 4.9988000000000003e-05 2024-05-15T13:29:13.774 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.774 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:29:13.775 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.775 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-05-15T13:29:13.775 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-05-15T13:29:13.775 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:29:13.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:13 smithi027 ceph-mon[39795]: pgmap v2459: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 308 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 739 op/s; 2645/14751 objects degraded (17.931%); 851/14751 objects misplaced (5.769%); 6.4 MiB/s, 166 objects/s recovering 2024-05-15T13:29:13.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:13 smithi027 ceph-mon[39795]: 1.4 scrub starts 2024-05-15T13:29:13.984 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:13 smithi027 ceph-mon[39795]: 1.4 scrub ok 2024-05-15T13:29:14.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:13 smithi120 ceph-mon[43961]: pgmap v2459: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 308 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 739 op/s; 2645/14751 objects degraded (17.931%); 851/14751 objects misplaced (5.769%); 6.4 MiB/s, 166 objects/s recovering 2024-05-15T13:29:14.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:13 smithi120 ceph-mon[43961]: 1.4 scrub starts 2024-05-15T13:29:14.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:13 smithi120 ceph-mon[43961]: 1.4 scrub ok 2024-05-15T13:29:14.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:13 smithi177 ceph-mon[44303]: pgmap v2459: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 308 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 739 op/s; 2645/14751 objects degraded (17.931%); 851/14751 objects misplaced (5.769%); 6.4 MiB/s, 166 objects/s recovering 2024-05-15T13:29:14.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:13 smithi177 ceph-mon[44303]: 1.4 scrub starts 2024-05-15T13:29:14.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:13 smithi177 ceph-mon[44303]: 1.4 scrub ok 2024-05-15T13:29:14.558 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-05-15T13:29:14.559 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:29:14.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:14.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:29:14.665 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:29:14.718 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:14.759 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:29:14.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:14.763 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:29:14.863 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:29:14.898 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:15.001 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:14 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 2645/14751 objects degraded (17.931%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:29:15.001 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:14 smithi177 ceph-mon[44303]: 9.5 scrub starts 2024-05-15T13:29:15.001 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:14 smithi177 ceph-mon[44303]: 9.5 scrub ok 2024-05-15T13:29:15.011 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:29:15.106 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:29:15.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:15.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:29:15.210 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:29:15.244 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:15.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:14 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 2645/14751 objects degraded (17.931%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:29:15.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:14 smithi027 ceph-mon[39795]: 9.5 scrub starts 2024-05-15T13:29:15.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:14 smithi027 ceph-mon[39795]: 9.5 scrub ok 2024-05-15T13:29:15.299 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:15 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 2645/14751 objects degraded (17.931%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:29:15.299 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:15 smithi120 ceph-mon[43961]: 9.5 scrub starts 2024-05-15T13:29:15.299 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:15 smithi120 ceph-mon[43961]: 9.5 scrub ok 2024-05-15T13:29:15.452 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:29:15.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:15.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:29:15.559 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:29:15.598 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:15.728 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:29:15.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:15.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:29:15.832 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:29:15.894 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:15.935 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:29:15.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:15.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:29:16.020 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-05-15T13:29:16.040 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:29:16.081 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:16.113 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:29:16.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:16.118 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:29:16.219 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:29:16.252 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:16 smithi120 ceph-mon[43961]: pgmap v2460: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 240 MiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 722 op/s; 2673/11520 objects degraded (23.203%); 178/11520 objects misplaced (1.545%); 8.3 MiB/s, 191 objects/s recovering 2024-05-15T13:29:16.252 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:16 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/889786558' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:29:16.266 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:16.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:16 smithi027 ceph-mon[39795]: pgmap v2460: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 240 MiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 722 op/s; 2673/11520 objects degraded (23.203%); 178/11520 objects misplaced (1.545%); 8.3 MiB/s, 191 objects/s recovering 2024-05-15T13:29:16.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:16 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/889786558' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:29:16.300 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:29:16.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:16.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:29:16.404 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:29:16.444 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:16 smithi177 ceph-mon[44303]: pgmap v2460: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 240 MiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 722 op/s; 2673/11520 objects degraded (23.203%); 178/11520 objects misplaced (1.545%); 8.3 MiB/s, 191 objects/s recovering 2024-05-15T13:29:16.444 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:16 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/889786558' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T13:29:16.471 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:16.834 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T13:29:17.041 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.9 to 1 (10000) 2024-05-15T13:29:17.065 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:17 smithi027 ceph-mon[39795]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:29:17.065 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:17 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/889786558' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:29:17.065 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:17 smithi027 ceph-mon[39795]: osdmap e974: 12 total, 12 up, 11 in 2024-05-15T13:29:17.066 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:17 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1270129954' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:29:17.066 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:17 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:29:17.258 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:17 smithi120 ceph-mon[43961]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:29:17.258 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:17 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/889786558' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:29:17.258 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:17 smithi120 ceph-mon[43961]: osdmap e974: 12 total, 12 up, 11 in 2024-05-15T13:29:17.258 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:17 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1270129954' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:29:17.258 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:17 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:29:17.284 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Removed 120240 objects 2024-05-15T13:29:17.284 INFO:tasks.radosbench.radosbench.2.smithi177.stdout:Clean up completed and total clean up time :273.949 2024-05-15T13:29:17.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:17 smithi177 ceph-mon[44303]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:29:17.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:17 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/889786558' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T13:29:17.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:17 smithi177 ceph-mon[44303]: osdmap e974: 12 total, 12 up, 11 in 2024-05-15T13:29:17.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:17 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1270129954' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:29:17.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:17 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:29:17.783 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:29:17.784 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 10 1 2024-05-15T13:29:17.839 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:29:17.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:17.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:29:17.916 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:29:17.917 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T13:29:17.917 INFO:teuthology.orchestra.run.smithi120.stdout: "num_ops": 0 2024-05-15T13:29:17.917 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:29:17.943 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:29:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:18.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:18 smithi027 ceph-mon[39795]: pgmap v2462: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 141 MiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 2673/6750 objects degraded (39.600%); 178/6750 objects misplaced (2.637%); 4.8 MiB/s, 132 objects/s recovering 2024-05-15T13:29:18.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:18 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T13:29:18.023 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:18 smithi027 ceph-mon[39795]: osdmap e975: 12 total, 12 up, 11 in 2024-05-15T13:29:18.100 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:29:18.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:18.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:29:18.204 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:29:18.235 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:18.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:18 smithi120 ceph-mon[43961]: pgmap v2462: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 141 MiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 2673/6750 objects degraded (39.600%); 178/6750 objects misplaced (2.637%); 4.8 MiB/s, 132 objects/s recovering 2024-05-15T13:29:18.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:18 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T13:29:18.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:18 smithi120 ceph-mon[43961]: osdmap e975: 12 total, 12 up, 11 in 2024-05-15T13:29:18.326 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_7 2024-05-15T13:29:18.326 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool rm unique_pool_7 unique_pool_7 --yes-i-really-really-mean-it 2024-05-15T13:29:18.334 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:29:18.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:18.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:29:18.440 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0 2024-05-15T13:29:18.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:18 smithi177 ceph-mon[44303]: pgmap v2462: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 141 MiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 2673/6750 objects degraded (39.600%); 178/6750 objects misplaced (2.637%); 4.8 MiB/s, 132 objects/s recovering 2024-05-15T13:29:18.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:18 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T13:29:18.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:18 smithi177 ceph-mon[44303]: osdmap e975: 12 total, 12 up, 11 in 2024-05-15T13:29:18.471 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:18.840 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T13:29:19.938 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-05-15T13:29:20.233 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:29:20.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:20 smithi027 ceph-mon[39795]: pgmap v2464: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 32 MiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 802 op/s; 2677/1512 objects degraded (177.050%); 33/1512 objects misplaced (2.183%); 5.9 MiB/s, 210 objects/s recovering 2024-05-15T13:29:20.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:20 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 2677/1512 objects degraded (177.050%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:29:20.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:20 smithi120 ceph-mon[43961]: pgmap v2464: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 32 MiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 802 op/s; 2677/1512 objects degraded (177.050%); 33/1512 objects misplaced (2.183%); 5.9 MiB/s, 210 objects/s recovering 2024-05-15T13:29:20.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:20 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 2677/1512 objects degraded (177.050%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:29:20.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:20 smithi177 ceph-mon[44303]: pgmap v2464: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 32 MiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 802 op/s; 2677/1512 objects degraded (177.050%); 33/1512 objects misplaced (2.183%); 5.9 MiB/s, 210 objects/s recovering 2024-05-15T13:29:20.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:20 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 2677/1512 objects degraded (177.050%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:29:20.568 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:29:22.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:22 smithi120 ceph-mon[43961]: pgmap v2465: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 22 MiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 572 op/s; 2677/1017 objects degraded (263.225%); 33/1017 objects misplaced (3.245%); 4.4 MiB/s, 166 objects/s recovering 2024-05-15T13:29:22.345 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-05-15T13:29:22.345 INFO:teuthology.orchestra.run.smithi120.stdout: "ops": [], 2024-05-15T13:29:22.345 INFO:teuthology.orchestra.run.smithi120.stdout: "complaint_time": 30, 2024-05-15T13:29:22.345 INFO:teuthology.orchestra.run.smithi120.stdout: "num_blocked_ops": 0 2024-05-15T13:29:22.345 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-05-15T13:29:22.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:22 smithi177 ceph-mon[44303]: pgmap v2465: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 22 MiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 572 op/s; 2677/1017 objects degraded (263.225%); 33/1017 objects misplaced (3.245%); 4.4 MiB/s, 166 objects/s recovering 2024-05-15T13:29:22.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:22 smithi027 ceph-mon[39795]: pgmap v2465: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 22 MiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 572 op/s; 2677/1017 objects degraded (263.225%); 33/1017 objects misplaced (3.245%); 4.4 MiB/s, 166 objects/s recovering 2024-05-15T13:29:23.226 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-05-15T13:29:23.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:23 smithi120 ceph-mon[43961]: osdmap e976: 12 total, 12 up, 11 in 2024-05-15T13:29:23.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:23 smithi177 ceph-mon[44303]: osdmap e976: 12 total, 12 up, 11 in 2024-05-15T13:29:23.509 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T13:29:23.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:23 smithi027 ceph-mon[39795]: osdmap e976: 12 total, 12 up, 11 in 2024-05-15T13:29:24.002 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0" found: no such container 2024-05-15T13:29:24.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:24.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.0' 2024-05-15T13:29:24.125 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:24 smithi120 ceph-mon[43961]: pgmap v2467: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 500 op/s; 2499/6 objects degraded (41650.000%); 33/6 objects misplaced (550.000%); 1.3 MiB/s, 295 objects/s recovering 2024-05-15T13:29:24.125 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:24 smithi120 ceph-mon[43961]: osdmap e977: 12 total, 12 up, 11 in 2024-05-15T13:29:24.138 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:29:24.170 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:24.192 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:29:24.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:24.195 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:29:24.259 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:29:24.296 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:29:24.326 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:24.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:24 smithi177 ceph-mon[44303]: pgmap v2467: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 500 op/s; 2499/6 objects degraded (41650.000%); 33/6 objects misplaced (550.000%); 1.3 MiB/s, 295 objects/s recovering 2024-05-15T13:29:24.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:24 smithi177 ceph-mon[44303]: osdmap e977: 12 total, 12 up, 11 in 2024-05-15T13:29:24.474 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:29:24.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:24.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:29:24.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:24 smithi027 ceph-mon[39795]: pgmap v2467: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 500 op/s; 2499/6 objects degraded (41650.000%); 33/6 objects misplaced (550.000%); 1.3 MiB/s, 295 objects/s recovering 2024-05-15T13:29:24.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:24 smithi027 ceph-mon[39795]: osdmap e977: 12 total, 12 up, 11 in 2024-05-15T13:29:24.578 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:29:24.612 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:25.354 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:25 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 2499/6 objects degraded (41650.000%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:29:25.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:25 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 2499/6 objects degraded (41650.000%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:29:25.450 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:29:25.470 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:29:25.491 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:29:25.510 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:25 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 2499/6 objects degraded (41650.000%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-05-15T13:29:25.517 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:29:25.538 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:29:25.561 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:29:25.564 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:29:25.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:25.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:29:25.583 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:29:25.609 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:29:25.631 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:29:25.654 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:29:25.669 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:29:25.672 INFO:teuthology.orchestra.run.smithi120.stdout:ERROR: (22) Invalid argument 2024-05-15T13:29:25.672 INFO:teuthology.orchestra.run.smithi120.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:29:25.679 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:29:25.701 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:25.702 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:29:25.704 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:29:25.704 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:29:25.704 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:29:25.704 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:29:25.704 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:29:25.704 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:29:25.705 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:29:25.705 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:29:25.705 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:29:25.705 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:29:25.705 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:29:25.705 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:29:26.198 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:29:26.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:26 smithi027 ceph-mon[39795]: pgmap v2469: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 232 op/s; 2499/6 objects degraded (41650.000%); 33/6 objects misplaced (550.000%); 1.1 MiB/s, 395 objects/s recovering 2024-05-15T13:29:26.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:26 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3969979361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:29:26.405 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:26 smithi120 ceph-mon[43961]: pgmap v2469: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 232 op/s; 2499/6 objects degraded (41650.000%); 33/6 objects misplaced (550.000%); 1.1 MiB/s, 395 objects/s recovering 2024-05-15T13:29:26.406 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:26 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3969979361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:29:26.433 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-05-15T13:29:26.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:26 smithi177 ceph-mon[44303]: pgmap v2469: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 232 op/s; 2499/6 objects degraded (41650.000%); 33/6 objects misplaced (550.000%); 1.1 MiB/s, 395 objects/s recovering 2024-05-15T13:29:26.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:26 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3969979361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:29:26.705 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:29:27.054 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:29:27.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:27.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:29:27.164 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:29:27.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:27 smithi177 ceph-mon[44303]: osdmap e978: 12 total, 12 up, 11 in 2024-05-15T13:29:27.194 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:27.213 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:29:27.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:27.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:29:27.317 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:29:27.347 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:27.425 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:27 smithi027 ceph-mon[39795]: osdmap e978: 12 total, 12 up, 11 in 2024-05-15T13:29:27.445 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:29:27.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:27.448 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:29:27.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:27 smithi120 ceph-mon[43961]: osdmap e978: 12 total, 12 up, 11 in 2024-05-15T13:29:27.549 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:29:27.576 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:27.596 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:29:27.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:27.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:29:27.700 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:29:27.727 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:27.746 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:29:27.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:27.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:29:27.849 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:29:27.876 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:27.895 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:29:27.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:27.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:29:27.999 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:29:28.025 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:28.044 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:29:28.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:28.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:29:28.148 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:29:28.175 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:28.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:28 smithi177 ceph-mon[44303]: pgmap v2471: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 56 op/s; 2287/6 objects degraded (38116.667%); 33/6 objects misplaced (550.000%); 0 B/s, 443 objects/s recovering 2024-05-15T13:29:28.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:28 smithi177 ceph-mon[44303]: osdmap e979: 12 total, 12 up, 11 in 2024-05-15T13:29:28.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:28 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:29:28.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:28 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:29:28.249 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:29:28.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:28.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:29:28.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:28 smithi027 ceph-mon[39795]: pgmap v2471: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 56 op/s; 2287/6 objects degraded (38116.667%); 33/6 objects misplaced (550.000%); 0 B/s, 443 objects/s recovering 2024-05-15T13:29:28.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:28 smithi027 ceph-mon[39795]: osdmap e979: 12 total, 12 up, 11 in 2024-05-15T13:29:28.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:28 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:29:28.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:28 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:29:28.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:28 smithi120 ceph-mon[43961]: pgmap v2471: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 56 op/s; 2287/6 objects degraded (38116.667%); 33/6 objects misplaced (550.000%); 0 B/s, 443 objects/s recovering 2024-05-15T13:29:28.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:28 smithi120 ceph-mon[43961]: osdmap e979: 12 total, 12 up, 11 in 2024-05-15T13:29:28.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:28 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:29:28.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:28 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:29:28.353 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6 2024-05-15T13:29:28.379 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:28.399 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6" found: no such container 2024-05-15T13:29:28.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:28.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.6' 2024-05-15T13:29:28.503 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:29:28.513 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:29:28.513 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:29:28.534 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:28.744 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T13:29:29.113 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-05-15T13:29:29.377 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:29:29.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:29 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:29:29.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:29 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:29:29.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:29 smithi177 ceph-mon[44303]: osdmap e980: 12 total, 12 up, 11 in 2024-05-15T13:29:29.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:29 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:29:29.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:29 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:29:29.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:29 smithi027 ceph-mon[39795]: osdmap e980: 12 total, 12 up, 11 in 2024-05-15T13:29:29.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:29 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:29:29.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:29 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:29:29.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:29 smithi120 ceph-mon[43961]: osdmap e980: 12 total, 12 up, 11 in 2024-05-15T13:29:30.338 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:30 smithi177 ceph-mon[44303]: pgmap v2474: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 1373/6 objects degraded (22883.333%); 0 B/s, 305 objects/s recovering 2024-05-15T13:29:30.339 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:30 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:29:30.339 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:30 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:29:30.339 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:30 smithi177 ceph-mon[44303]: Health check update: Degraded data redundancy: 1373/6 objects degraded (22883.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-05-15T13:29:30.339 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:30 smithi177 ceph-mon[44303]: osdmap e981: 12 total, 12 up, 11 in 2024-05-15T13:29:30.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:30 smithi027 ceph-mon[39795]: pgmap v2474: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 1373/6 objects degraded (22883.333%); 0 B/s, 305 objects/s recovering 2024-05-15T13:29:30.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:30 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:29:30.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:30 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:29:30.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:30 smithi027 ceph-mon[39795]: Health check update: Degraded data redundancy: 1373/6 objects degraded (22883.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-05-15T13:29:30.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:30 smithi027 ceph-mon[39795]: osdmap e981: 12 total, 12 up, 11 in 2024-05-15T13:29:30.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:30 smithi120 ceph-mon[43961]: pgmap v2474: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 1373/6 objects degraded (22883.333%); 0 B/s, 305 objects/s recovering 2024-05-15T13:29:30.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:30 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:29:30.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:30 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:29:30.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:30 smithi120 ceph-mon[43961]: Health check update: Degraded data redundancy: 1373/6 objects degraded (22883.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-05-15T13:29:30.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:30 smithi120 ceph-mon[43961]: osdmap e981: 12 total, 12 up, 11 in 2024-05-15T13:29:31.164 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:29:31.164 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:29:31.827 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-05-15T13:29:32.090 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:29:32.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:32 smithi027 ceph-mon[39795]: pgmap v2476: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 1373/6 objects degraded (22883.333%); 0 B/s, 90 objects/s recovering 2024-05-15T13:29:32.351 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:32 smithi177 ceph-mon[44303]: pgmap v2476: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 1373/6 objects degraded (22883.333%); 0 B/s, 90 objects/s recovering 2024-05-15T13:29:32.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:32 smithi120 ceph-mon[43961]: pgmap v2476: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 1373/6 objects degraded (22883.333%); 0 B/s, 90 objects/s recovering 2024-05-15T13:29:32.904 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:29:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:29:33.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:33.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:29:33.227 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:29:33.254 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:33.273 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:29:33.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:33.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:29:33.377 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1 2024-05-15T13:29:33.409 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:33.759 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1" found: no such container 2024-05-15T13:29:33.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:33.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.1' 2024-05-15T13:29:33.806 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:29:33.806 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:29:33.864 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7 2024-05-15T13:29:33.890 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:33.909 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7" found: no such container 2024-05-15T13:29:33.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:33.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.7' 2024-05-15T13:29:34.013 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10 2024-05-15T13:29:34.043 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:34.063 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10" found: no such container 2024-05-15T13:29:34.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:34.067 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.10' 2024-05-15T13:29:34.168 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:29:34.195 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:34.292 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:34 smithi120 ceph-mon[43961]: pgmap v2477: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 1373/6 objects degraded (22883.333%); 0 B/s, 36 objects/s recovering 2024-05-15T13:29:34.392 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.10 to 1 (10000) 2024-05-15T13:29:34.420 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:34 smithi027 ceph-mon[39795]: pgmap v2477: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 1373/6 objects degraded (22883.333%); 0 B/s, 36 objects/s recovering 2024-05-15T13:29:34.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:34 smithi177 ceph-mon[44303]: pgmap v2477: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 1373/6 objects degraded (22883.333%); 0 B/s, 36 objects/s recovering 2024-05-15T13:29:34.544 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:29:34.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:34.547 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:29:34.550 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-05-15T13:29:34.648 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5 2024-05-15T13:29:34.674 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:34.692 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5" found: no such container 2024-05-15T13:29:34.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:34.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.5' 2024-05-15T13:29:34.796 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3 2024-05-15T13:29:34.826 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:34.849 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:29:35.252 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 7 1 2024-05-15T13:29:35.347 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:35 smithi177 ceph-mon[44303]: osdmap e982: 12 total, 12 up, 11 in 2024-05-15T13:29:35.347 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:35 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1121293194' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T13:29:35.347 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:35 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T13:29:35.347 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:35 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/352482970' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:29:35.347 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:35 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/445475294' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T13:29:35.347 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:35 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T13:29:35.348 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:35 smithi177 ceph-mon[44303]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:29:35.348 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:35 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T13:29:35.348 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:35 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/352482970' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:29:35.348 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:35 smithi177 ceph-mon[44303]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T13:29:35.348 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:35 smithi177 ceph-mon[44303]: osdmap e983: 12 total, 12 up, 11 in 2024-05-15T13:29:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-05-15T13:29:35.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:35 smithi027 ceph-mon[39795]: osdmap e982: 12 total, 12 up, 11 in 2024-05-15T13:29:35.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:35 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1121293194' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T13:29:35.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:35 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T13:29:35.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:35 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/352482970' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:29:35.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:35 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/445475294' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T13:29:35.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:35 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T13:29:35.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:35 smithi027 ceph-mon[39795]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:29:35.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:35 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T13:29:35.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:35 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/352482970' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:29:35.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:35 smithi027 ceph-mon[39795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T13:29:35.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:35 smithi027 ceph-mon[39795]: osdmap e983: 12 total, 12 up, 11 in 2024-05-15T13:29:35.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:35 smithi120 ceph-mon[43961]: osdmap e982: 12 total, 12 up, 11 in 2024-05-15T13:29:35.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:35 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1121293194' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T13:29:35.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:35 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T13:29:35.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:35 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/352482970' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:29:35.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:35 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/445475294' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T13:29:35.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:35 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T13:29:35.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:35 smithi120 ceph-mon[43961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:29:35.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:35 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T13:29:35.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:35 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/352482970' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T13:29:35.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:35 smithi120 ceph-mon[43961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T13:29:35.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:35 smithi120 ceph-mon[43961]: osdmap e983: 12 total, 12 up, 11 in 2024-05-15T13:29:35.978 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3" found: no such container 2024-05-15T13:29:35.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:35.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.3' 2024-05-15T13:29:36.083 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11 2024-05-15T13:29:36.094 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_7' removed 2024-05-15T13:29:36.115 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:36.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:36 smithi177 ceph-mon[44303]: pgmap v2480: 31 pgs: 31 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail 2024-05-15T13:29:36.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:36 smithi177 ceph-mon[44303]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1373/6 objects degraded (22883.333%), 1 pg degraded, 1 pg undersized) 2024-05-15T13:29:36.214 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11" found: no such container 2024-05-15T13:29:36.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:36.217 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.11' 2024-05-15T13:29:36.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:36 smithi027 ceph-mon[39795]: pgmap v2480: 31 pgs: 31 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail 2024-05-15T13:29:36.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:36 smithi027 ceph-mon[39795]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1373/6 objects degraded (22883.333%), 1 pg degraded, 1 pg undersized) 2024-05-15T13:29:36.318 DEBUG:teuthology.orchestra.run.smithi120:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4 2024-05-15T13:29:36.349 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:36 smithi120 ceph-mon[43961]: pgmap v2480: 31 pgs: 31 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail 2024-05-15T13:29:36.349 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:36 smithi120 ceph-mon[43961]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1373/6 objects degraded (22883.333%), 1 pg degraded, 1 pg undersized) 2024-05-15T13:29:36.351 INFO:teuthology.orchestra.run.smithi120.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:36.373 INFO:teuthology.orchestra.run.smithi120.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4" found: no such container 2024-05-15T13:29:36.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:36.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi120 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.4' 2024-05-15T13:29:36.477 DEBUG:teuthology.orchestra.run.smithi177:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8 2024-05-15T13:29:36.507 INFO:teuthology.orchestra.run.smithi177.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:36.528 INFO:teuthology.orchestra.run.smithi177.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8" found: no such container 2024-05-15T13:29:36.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:36.532 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi177 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.8' 2024-05-15T13:29:36.633 DEBUG:teuthology.orchestra.run.smithi027:> sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2 2024-05-15T13:29:36.668 INFO:teuthology.orchestra.run.smithi027.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T13:29:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:Error: no container with name or ID "ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2" found: no such container 2024-05-15T13:29:36.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T13:29:36.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi027 with status 125: 'sudo docker kill -s 1 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd.2' 2024-05-15T13:29:36.775 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-05-15T13:29:36.775 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T13:29:36.793 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-05-15T13:29:36.803 INFO:tasks.thrashosds:joining thrashosds 2024-05-15T13:29:37.169 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:29:37.617 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-05-15T13:29:37.877 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:29:37.958 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset noscrub 2024-05-15T13:29:38.253 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:38 smithi027 ceph-mon[39795]: pgmap v2481: 31 pgs: 31 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail 2024-05-15T13:29:38.253 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:38 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1731149546' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:29:38.366 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:38 smithi177 ceph-mon[44303]: pgmap v2481: 31 pgs: 31 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail 2024-05-15T13:29:38.366 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:38 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1731149546' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:29:38.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:38 smithi120 ceph-mon[43961]: pgmap v2481: 31 pgs: 31 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail 2024-05-15T13:29:38.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:38 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1731149546' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T13:29:38.931 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-05-15T13:29:38.932 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-05-15T13:29:38.932 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-05-15T13:29:38.932 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-05-15T13:29:38.932 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-05-15T13:29:38.932 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-05-15T13:29:38.932 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-05-15T13:29:38.932 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-05-15T13:29:38.932 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-05-15T13:29:38.932 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-05-15T13:29:38.932 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-05-15T13:29:38.932 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-05-15T13:29:38.933 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T13:29:38.933 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T13:29:38.933 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T13:29:38.933 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T13:29:38.933 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T13:29:38.933 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T13:29:38.933 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T13:29:38.933 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T13:29:38.933 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T13:29:38.933 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T13:29:38.933 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T13:29:38.933 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T13:29:39.714 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:29:39.715 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T13:29:39.715 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-05-15T13:29:39.715 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-05-15T13:29:39.715 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:29:39.838 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:29:40.103 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.7 to 1 (10000) 2024-05-15T13:29:40.405 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:40 smithi177 ceph-mon[44303]: pgmap v2482: 31 pgs: 31 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail 2024-05-15T13:29:40.405 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:40 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/18853717' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T13:29:40.434 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-05-15T13:29:40.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:40 smithi027 ceph-mon[39795]: pgmap v2482: 31 pgs: 31 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail 2024-05-15T13:29:40.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:40 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/18853717' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T13:29:40.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:40 smithi120 ceph-mon[43961]: pgmap v2482: 31 pgs: 31 active+clean; 590 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail 2024-05-15T13:29:40.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:40 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/18853717' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T13:29:40.723 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:29:41.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:41 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/18853717' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T13:29:41.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:41 smithi027 ceph-mon[39795]: osdmap e984: 12 total, 12 up, 11 in 2024-05-15T13:29:41.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:41 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:29:41.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:41 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:29:41.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:41 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:29:41.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:41 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:29:41.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:41 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/18853717' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T13:29:41.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:41 smithi177 ceph-mon[44303]: osdmap e984: 12 total, 12 up, 11 in 2024-05-15T13:29:41.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:41 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:29:41.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:41 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:29:41.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:41 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:29:41.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:41 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:29:41.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:41 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/18853717' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T13:29:41.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:41 smithi120 ceph-mon[43961]: osdmap e984: 12 total, 12 up, 11 in 2024-05-15T13:29:41.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:41 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:29:41.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:41 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:29:41.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:41 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:29:41.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:41 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:29:41.778 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 4 1 2024-05-15T13:29:42.358 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:29:42.413 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:29:42.413 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-05-15T13:29:42.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-05-15T13:29:42.413 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T13:29:42.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.413 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:193592 9.8 9:1d1e1382:::benchmark_data_smithi177_159919_object73349:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e936)", 2024-05-15T13:29:42.413 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:27:57.586668+0000", 2024-05-15T13:29:42.413 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 104.822093334, 2024-05-15T13:29:42.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9987109030000001, 2024-05-15T13:29:42.413 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:42.413 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:42.414 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:42.414 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:42.414 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:42.414 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 193592 2024-05-15T13:29:42.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.414 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:42.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:42.414 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:57.586668+0000", 2024-05-15T13:29:42.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:42.414 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:57.586668+0000", 2024-05-15T13:29:42.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:42.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:57.586670+0000", 2024-05-15T13:29:42.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.345e-06 2024-05-15T13:29:42.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:42.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:57.586676+0000", 2024-05-15T13:29:42.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3770000000000001e-06 2024-05-15T13:29:42.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:42.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:57.586690+0000", 2024-05-15T13:29:42.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4177e-05 2024-05-15T13:29:42.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:42.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:57.586698+0000", 2024-05-15T13:29:42.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2679999999999994e-06 2024-05-15T13:29:42.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:57.586752+0000", 2024-05-15T13:29:42.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3783000000000003e-05 2024-05-15T13:29:42.417 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:29:42.417 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:57.586760+0000", 2024-05-15T13:29:42.417 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9670000000000001e-06 2024-05-15T13:29:42.417 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.417 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.422254+0000", 2024-05-15T13:29:42.417 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.83549349399999995 2024-05-15T13:29:42.417 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:29:42.418 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.422266+0000", 2024-05-15T13:29:42.418 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2284e-05 2024-05-15T13:29:42.418 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.418 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.418 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.418 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.569280+0000", 2024-05-15T13:29:42.418 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.147013598 2024-05-15T13:29:42.418 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.418 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.418 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:42.418 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.569364+0000", 2024-05-15T13:29:42.418 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4715999999999993e-05 2024-05-15T13:29:42.418 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.418 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.419 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7,11", 2024-05-15T13:29:42.419 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.569454+0000", 2024-05-15T13:29:42.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9952999999999997e-05 2024-05-15T13:29:42.419 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.419 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.419 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.419 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.574514+0000", 2024-05-15T13:29:42.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0050600269999999999 2024-05-15T13:29:42.419 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.419 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.419 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.419 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.585124+0000", 2024-05-15T13:29:42.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.01060942 2024-05-15T13:29:42.419 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.420 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.420 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.420 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.585195+0000", 2024-05-15T13:29:42.420 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1824999999999998e-05 2024-05-15T13:29:42.420 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.420 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.420 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:42.420 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.585247+0000", 2024-05-15T13:29:42.420 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1762999999999999e-05 2024-05-15T13:29:42.420 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.420 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.420 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:42.420 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.585282+0000", 2024-05-15T13:29:42.420 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5144000000000001e-05 2024-05-15T13:29:42.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.421 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.421 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:42.421 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.585379+0000", 2024-05-15T13:29:42.421 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6762000000000006e-05 2024-05-15T13:29:42.421 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.421 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:42.421 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.421 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.421 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:193595 9.8 9:11a84886:::benchmark_data_smithi177_159919_object73352:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e937)", 2024-05-15T13:29:42.421 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:27:58.599292+0000", 2024-05-15T13:29:42.421 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 103.809469208, 2024-05-15T13:29:42.421 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.99668268100000001, 2024-05-15T13:29:42.421 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:42.422 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:42.422 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:42.422 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:42.422 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:42.422 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 193595 2024-05-15T13:29:42.422 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.422 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:42.422 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.422 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:42.422 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.599292+0000", 2024-05-15T13:29:42.422 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.422 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.422 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.422 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:42.423 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.599292+0000", 2024-05-15T13:29:42.423 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.423 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.423 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.423 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:42.423 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.599294+0000", 2024-05-15T13:29:42.423 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.519e-06 2024-05-15T13:29:42.423 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.423 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.423 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:42.423 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.599300+0000", 2024-05-15T13:29:42.423 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8449999999999999e-06 2024-05-15T13:29:42.423 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.423 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.423 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:42.424 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.599315+0000", 2024-05-15T13:29:42.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5248999999999999e-05 2024-05-15T13:29:42.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.424 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:42.424 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.599323+0000", 2024-05-15T13:29:42.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2230000000000007e-06 2024-05-15T13:29:42.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.424 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.424 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.599388+0000", 2024-05-15T13:29:42.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4754999999999999e-05 2024-05-15T13:29:42.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.424 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:29:42.425 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.599400+0000", 2024-05-15T13:29:42.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2021999999999999e-05 2024-05-15T13:29:42.425 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.425 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.425 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.574555+0000", 2024-05-15T13:29:42.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.97515491399999998 2024-05-15T13:29:42.425 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.425 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:42.425 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.574702+0000", 2024-05-15T13:29:42.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014717300000000001 2024-05-15T13:29:42.425 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.425 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7,11", 2024-05-15T13:29:42.426 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.574851+0000", 2024-05-15T13:29:42.426 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000148708 2024-05-15T13:29:42.426 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.426 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.426 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.426 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.585238+0000", 2024-05-15T13:29:42.426 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010387504000000001 2024-05-15T13:29:42.426 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.426 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.426 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:42.426 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.585395+0000", 2024-05-15T13:29:42.426 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015664300000000001 2024-05-15T13:29:42.426 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.426 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.427 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.427 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.585540+0000", 2024-05-15T13:29:42.427 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014509899999999999 2024-05-15T13:29:42.427 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.427 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.427 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.427 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.595804+0000", 2024-05-15T13:29:42.427 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010263572 2024-05-15T13:29:42.427 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.427 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.427 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:42.427 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.595865+0000", 2024-05-15T13:29:42.427 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1086000000000004e-05 2024-05-15T13:29:42.427 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.428 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.428 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:42.428 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.595975+0000", 2024-05-15T13:29:42.428 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000110369 2024-05-15T13:29:42.428 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.428 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:42.428 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.428 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.428 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.428 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:193895 9.8 9:1a3ebb9d:::benchmark_data_smithi177_159919_object73652:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-05-15T13:29:42.428 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:00.503900+0000", 2024-05-15T13:29:42.428 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 101.904861768, 2024-05-15T13:29:42.428 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.431474778, 2024-05-15T13:29:42.428 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:42.428 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:42.429 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:42.429 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:42.429 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:42.429 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 193895 2024-05-15T13:29:42.429 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.429 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:42.429 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.429 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:42.429 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.503900+0000", 2024-05-15T13:29:42.429 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.429 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.429 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.429 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:42.429 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.503900+0000", 2024-05-15T13:29:42.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.430 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.430 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.430 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:42.430 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.503902+0000", 2024-05-15T13:29:42.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7120000000000001e-06 2024-05-15T13:29:42.430 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.430 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.430 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:42.430 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.503909+0000", 2024-05-15T13:29:42.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.413e-06 2024-05-15T13:29:42.430 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.430 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.430 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:42.431 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.503925+0000", 2024-05-15T13:29:42.431 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5639000000000001e-05 2024-05-15T13:29:42.431 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.431 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.431 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:42.431 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.503933+0000", 2024-05-15T13:29:42.431 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8249999999999994e-06 2024-05-15T13:29:42.431 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.431 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.431 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.431 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.711387+0000", 2024-05-15T13:29:42.431 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20745359199999999 2024-05-15T13:29:42.431 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.431 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.431 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:42.432 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.711509+0000", 2024-05-15T13:29:42.432 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012236699999999999 2024-05-15T13:29:42.432 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.432 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.432 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7,11", 2024-05-15T13:29:42.432 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.711645+0000", 2024-05-15T13:29:42.432 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013535500000000001 2024-05-15T13:29:42.432 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.432 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.432 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.432 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.935052+0000", 2024-05-15T13:29:42.432 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.22340774599999999 2024-05-15T13:29:42.432 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.432 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.433 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.433 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.935109+0000", 2024-05-15T13:29:42.433 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6391999999999998e-05 2024-05-15T13:29:42.433 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.433 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.433 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:42.433 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.935125+0000", 2024-05-15T13:29:42.433 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5936000000000001e-05 2024-05-15T13:29:42.433 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.433 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.433 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.433 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.935170+0000", 2024-05-15T13:29:42.433 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5666e-05 2024-05-15T13:29:42.433 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.434 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.434 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:42.434 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.935235+0000", 2024-05-15T13:29:42.434 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4589000000000001e-05 2024-05-15T13:29:42.434 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.434 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.434 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:42.434 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.935375+0000", 2024-05-15T13:29:42.434 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000139546 2024-05-15T13:29:42.434 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.434 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:42.434 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.434 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.435 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.435 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:193905 9.8 9:13f25616:::benchmark_data_smithi177_159919_object73662:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-05-15T13:29:42.435 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:00.521556+0000", 2024-05-15T13:29:42.435 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 101.887205065, 2024-05-15T13:29:42.435 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.43324601000000001, 2024-05-15T13:29:42.435 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:42.435 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:42.435 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:42.435 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:42.435 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:42.435 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 193905 2024-05-15T13:29:42.435 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.435 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:42.435 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.435 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:42.436 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.521556+0000", 2024-05-15T13:29:42.436 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.436 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.436 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.436 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:42.436 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.521556+0000", 2024-05-15T13:29:42.436 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.436 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.436 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.436 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:42.436 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.521558+0000", 2024-05-15T13:29:42.436 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7999999999999999e-06 2024-05-15T13:29:42.436 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.436 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.437 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:42.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.521573+0000", 2024-05-15T13:29:42.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5038999999999999e-05 2024-05-15T13:29:42.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.437 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:42.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.521587+0000", 2024-05-15T13:29:42.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3456e-05 2024-05-15T13:29:42.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.437 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:42.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.521596+0000", 2024-05-15T13:29:42.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7299999999999994e-06 2024-05-15T13:29:42.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.438 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.935817+0000", 2024-05-15T13:29:42.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.41422135599999999 2024-05-15T13:29:42.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.438 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:42.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.935958+0000", 2024-05-15T13:29:42.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014087600000000001 2024-05-15T13:29:42.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.438 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7,11", 2024-05-15T13:29:42.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.936115+0000", 2024-05-15T13:29:42.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015755000000000001 2024-05-15T13:29:42.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.439 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.439 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.439 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.945207+0000", 2024-05-15T13:29:42.439 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0090913950000000004 2024-05-15T13:29:42.439 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.439 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.439 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.439 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.945261+0000", 2024-05-15T13:29:42.439 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4135999999999998e-05 2024-05-15T13:29:42.439 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.439 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.439 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.439 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.945310+0000", 2024-05-15T13:29:42.439 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8878999999999997e-05 2024-05-15T13:29:42.440 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.440 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.440 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:42.440 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.954672+0000", 2024-05-15T13:29:42.440 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0093623490000000007 2024-05-15T13:29:42.440 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.440 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.440 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:42.440 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.954706+0000", 2024-05-15T13:29:42.440 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4337e-05 2024-05-15T13:29:42.440 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.440 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.440 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:42.440 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.954803+0000", 2024-05-15T13:29:42.441 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6107000000000007e-05 2024-05-15T13:29:42.441 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.441 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:42.441 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.441 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.441 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.441 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:193934 9.8 9:1f33d6a1:::benchmark_data_smithi177_159919_object73691:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-05-15T13:29:42.441 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:00.590908+0000", 2024-05-15T13:29:42.441 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 101.81785394400001, 2024-05-15T13:29:42.441 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.36414381200000001, 2024-05-15T13:29:42.441 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:42.441 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:42.441 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:42.441 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:42.441 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:42.442 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 193934 2024-05-15T13:29:42.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.442 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:42.442 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.442 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:42.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.590908+0000", 2024-05-15T13:29:42.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.442 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.442 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:42.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.590908+0000", 2024-05-15T13:29:42.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.443 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:42.443 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.590909+0000", 2024-05-15T13:29:42.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8390000000000001e-06 2024-05-15T13:29:42.443 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.443 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:42.443 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.590917+0000", 2024-05-15T13:29:42.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2300000000000002e-06 2024-05-15T13:29:42.443 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.443 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:42.443 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.590931+0000", 2024-05-15T13:29:42.444 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4153e-05 2024-05-15T13:29:42.444 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.444 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.444 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:42.444 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.590939+0000", 2024-05-15T13:29:42.444 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1929999999999993e-06 2024-05-15T13:29:42.444 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.444 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.444 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.444 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.940809+0000", 2024-05-15T13:29:42.444 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.34986952399999999 2024-05-15T13:29:42.444 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.444 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.444 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:42.444 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.940944+0000", 2024-05-15T13:29:42.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013523699999999999 2024-05-15T13:29:42.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7,11", 2024-05-15T13:29:42.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.941039+0000", 2024-05-15T13:29:42.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5433999999999995e-05 2024-05-15T13:29:42.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.945361+0000", 2024-05-15T13:29:42.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0043216870000000003 2024-05-15T13:29:42.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.954655+0000", 2024-05-15T13:29:42.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0092939129999999991 2024-05-15T13:29:42.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:42.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.954810+0000", 2024-05-15T13:29:42.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015468 2024-05-15T13:29:42.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.954877+0000", 2024-05-15T13:29:42.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7578999999999998e-05 2024-05-15T13:29:42.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:42.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.954928+0000", 2024-05-15T13:29:42.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0423000000000002e-05 2024-05-15T13:29:42.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:42.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.955051+0000", 2024-05-15T13:29:42.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012391999999999999 2024-05-15T13:29:42.447 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.447 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:42.447 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.447 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:193984 9.8 9:13f44193:::benchmark_data_smithi177_159919_object73741:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-05-15T13:29:42.448 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:00.776380+0000", 2024-05-15T13:29:42.448 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 101.632381712, 2024-05-15T13:29:42.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.400334315, 2024-05-15T13:29:42.448 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:42.448 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:42.448 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:42.448 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:42.448 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:42.448 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 193984 2024-05-15T13:29:42.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.448 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:42.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.448 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:42.448 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.776380+0000", 2024-05-15T13:29:42.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:42.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.776380+0000", 2024-05-15T13:29:42.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:42.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.776381+0000", 2024-05-15T13:29:42.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4199999999999995e-07 2024-05-15T13:29:42.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:42.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.776385+0000", 2024-05-15T13:29:42.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4710000000000001e-06 2024-05-15T13:29:42.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:42.450 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.776397+0000", 2024-05-15T13:29:42.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2218000000000001e-05 2024-05-15T13:29:42.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:42.450 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.776405+0000", 2024-05-15T13:29:42.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2180000000000002e-06 2024-05-15T13:29:42.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.450 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.945402+0000", 2024-05-15T13:29:42.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.168997485 2024-05-15T13:29:42.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.451 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:42.451 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.945570+0000", 2024-05-15T13:29:42.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000167994 2024-05-15T13:29:42.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.451 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7,11", 2024-05-15T13:29:42.451 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.946266+0000", 2024-05-15T13:29:42.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00069625500000000001 2024-05-15T13:29:42.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.451 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:42.451 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.954816+0000", 2024-05-15T13:29:42.452 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0085497339999999998 2024-05-15T13:29:42.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.955115+0000", 2024-05-15T13:29:42.452 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000299389 2024-05-15T13:29:42.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.176524+0000", 2024-05-15T13:29:42.452 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.22140817199999999 2024-05-15T13:29:42.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.176551+0000", 2024-05-15T13:29:42.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7478999999999999e-05 2024-05-15T13:29:42.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.453 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.453 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:42.453 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.176602+0000", 2024-05-15T13:29:42.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0559000000000002e-05 2024-05-15T13:29:42.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.453 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.453 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:42.453 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.176714+0000", 2024-05-15T13:29:42.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000112599 2024-05-15T13:29:42.453 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.453 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:42.453 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.454 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.454 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.454 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:194010 9.8 9:15a0be37:::benchmark_data_smithi177_159919_object73767:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-05-15T13:29:42.454 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:00.837516+0000", 2024-05-15T13:29:42.454 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 101.57124558300001, 2024-05-15T13:29:42.454 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33943757099999999, 2024-05-15T13:29:42.454 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:42.454 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:42.454 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:42.454 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:42.454 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:42.454 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 194010 2024-05-15T13:29:42.454 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.454 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:42.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.455 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:42.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.837516+0000", 2024-05-15T13:29:42.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.455 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:42.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.837516+0000", 2024-05-15T13:29:42.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.455 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:42.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.837517+0000", 2024-05-15T13:29:42.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.412e-06 2024-05-15T13:29:42.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:42.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.837523+0000", 2024-05-15T13:29:42.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7270000000000004e-06 2024-05-15T13:29:42.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:42.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.837538+0000", 2024-05-15T13:29:42.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4457999999999999e-05 2024-05-15T13:29:42.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:42.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.837545+0000", 2024-05-15T13:29:42.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2629999999999997e-06 2024-05-15T13:29:42.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.457 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.457 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.950796+0000", 2024-05-15T13:29:42.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11325104699999999 2024-05-15T13:29:42.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.457 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:42.457 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.950907+0000", 2024-05-15T13:29:42.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111366 2024-05-15T13:29:42.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.457 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7,11", 2024-05-15T13:29:42.457 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.950997+0000", 2024-05-15T13:29:42.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9868000000000001e-05 2024-05-15T13:29:42.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:42.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.955132+0000", 2024-05-15T13:29:42.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0041352100000000003 2024-05-15T13:29:42.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:00.955190+0000", 2024-05-15T13:29:42.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.749e-05 2024-05-15T13:29:42.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.176801+0000", 2024-05-15T13:29:42.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.22161120100000001 2024-05-15T13:29:42.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.176833+0000", 2024-05-15T13:29:42.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1511999999999999e-05 2024-05-15T13:29:42.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:42.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.176868+0000", 2024-05-15T13:29:42.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5626000000000001e-05 2024-05-15T13:29:42.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:42.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.176954+0000", 2024-05-15T13:29:42.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5390999999999996e-05 2024-05-15T13:29:42.460 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.460 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:42.460 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.460 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:194070 9.8 9:1a1f44bd:::benchmark_data_smithi177_159919_object73827:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-05-15T13:29:42.460 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:01.026445+0000", 2024-05-15T13:29:42.460 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 101.382317032, 2024-05-15T13:29:42.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.37748531899999999, 2024-05-15T13:29:42.460 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:42.460 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:42.460 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:42.460 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:42.460 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:42.460 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 194070 2024-05-15T13:29:42.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.461 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:42.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:42.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.026445+0000", 2024-05-15T13:29:42.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:42.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.026445+0000", 2024-05-15T13:29:42.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:42.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.026446+0000", 2024-05-15T13:29:42.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.592e-06 2024-05-15T13:29:42.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:42.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.026453+0000", 2024-05-15T13:29:42.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5429999999999999e-06 2024-05-15T13:29:42.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:42.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.026465+0000", 2024-05-15T13:29:42.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2717999999999999e-05 2024-05-15T13:29:42.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:42.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.026474+0000", 2024-05-15T13:29:42.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4929999999999998e-06 2024-05-15T13:29:42.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.463 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.177669+0000", 2024-05-15T13:29:42.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.151195146 2024-05-15T13:29:42.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:42.463 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.177816+0000", 2024-05-15T13:29:42.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014657599999999999 2024-05-15T13:29:42.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7,11", 2024-05-15T13:29:42.463 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.177908+0000", 2024-05-15T13:29:42.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2408e-05 2024-05-15T13:29:42.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.189721+0000", 2024-05-15T13:29:42.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011813362000000001 2024-05-15T13:29:42.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:42.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.189735+0000", 2024-05-15T13:29:42.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3377000000000001e-05 2024-05-15T13:29:42.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.403757+0000", 2024-05-15T13:29:42.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.214022288 2024-05-15T13:29:42.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.403785+0000", 2024-05-15T13:29:42.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7679999999999999e-05 2024-05-15T13:29:42.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:42.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.403823+0000", 2024-05-15T13:29:42.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8488999999999997e-05 2024-05-15T13:29:42.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:42.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.403930+0000", 2024-05-15T13:29:42.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000106647 2024-05-15T13:29:42.466 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.466 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:42.466 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.466 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:194074 9.8 9:1f7bbfce:::benchmark_data_smithi177_159919_object73831:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-05-15T13:29:42.466 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:01.036338+0000", 2024-05-15T13:29:42.466 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 101.37242374500001, 2024-05-15T13:29:42.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.368006738, 2024-05-15T13:29:42.466 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:42.466 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:42.466 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:42.466 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:42.466 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:42.467 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 194074 2024-05-15T13:29:42.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.467 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:42.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.467 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:42.467 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.036338+0000", 2024-05-15T13:29:42.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.467 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:42.467 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.036338+0000", 2024-05-15T13:29:42.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:42.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.036339+0000", 2024-05-15T13:29:42.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4389999999999999e-06 2024-05-15T13:29:42.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:42.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.036345+0000", 2024-05-15T13:29:42.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.682e-06 2024-05-15T13:29:42.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:42.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.036355+0000", 2024-05-15T13:29:42.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0399e-05 2024-05-15T13:29:42.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:42.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.036363+0000", 2024-05-15T13:29:42.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.525e-06 2024-05-15T13:29:42.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.181917+0000", 2024-05-15T13:29:42.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.14555417500000001 2024-05-15T13:29:42.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:42.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.182064+0000", 2024-05-15T13:29:42.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014701300000000001 2024-05-15T13:29:42.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7,11", 2024-05-15T13:29:42.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.182212+0000", 2024-05-15T13:29:42.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014807000000000001 2024-05-15T13:29:42.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.189772+0000", 2024-05-15T13:29:42.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0075603750000000003 2024-05-15T13:29:42.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:42.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.193611+0000", 2024-05-15T13:29:42.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0038386959999999999 2024-05-15T13:29:42.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.404158+0000", 2024-05-15T13:29:42.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21054682299999999 2024-05-15T13:29:42.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.404216+0000", 2024-05-15T13:29:42.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7855000000000001e-05 2024-05-15T13:29:42.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:42.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.404268+0000", 2024-05-15T13:29:42.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2611999999999997e-05 2024-05-15T13:29:42.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.472 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:42.472 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.404345+0000", 2024-05-15T13:29:42.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6074000000000004e-05 2024-05-15T13:29:42.472 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.472 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:42.472 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.472 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:194099 9.8 9:156b4137:::benchmark_data_smithi177_159919_object73856:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-05-15T13:29:42.472 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:01.088198+0000", 2024-05-15T13:29:42.472 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 101.320563721, 2024-05-15T13:29:42.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.31642107800000002, 2024-05-15T13:29:42.473 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:42.473 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:42.473 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:42.473 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:42.473 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:42.473 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 194099 2024-05-15T13:29:42.473 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.473 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:42.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.473 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:42.473 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.088198+0000", 2024-05-15T13:29:42.473 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.473 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.473 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:42.474 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.088198+0000", 2024-05-15T13:29:42.474 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.474 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.474 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.474 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:42.474 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.088199+0000", 2024-05-15T13:29:42.474 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2160000000000001e-06 2024-05-15T13:29:42.474 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.474 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.474 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:42.474 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.088205+0000", 2024-05-15T13:29:42.474 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9519999999999999e-06 2024-05-15T13:29:42.474 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.474 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:42.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.088217+0000", 2024-05-15T13:29:42.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2337999999999999e-05 2024-05-15T13:29:42.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:42.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.088235+0000", 2024-05-15T13:29:42.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7697e-05 2024-05-15T13:29:42.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.186450+0000", 2024-05-15T13:29:42.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.098214535000000006 2024-05-15T13:29:42.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:42.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.186547+0000", 2024-05-15T13:29:42.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7330999999999994e-05 2024-05-15T13:29:42.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7,11", 2024-05-15T13:29:42.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.186642+0000", 2024-05-15T13:29:42.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5377000000000005e-05 2024-05-15T13:29:42.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:42.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.193623+0000", 2024-05-15T13:29:42.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0069803419999999996 2024-05-15T13:29:42.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.477 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.477 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.403730+0000", 2024-05-15T13:29:42.477 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.21010743400000001 2024-05-15T13:29:42.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.477 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.477 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.404407+0000", 2024-05-15T13:29:42.477 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00067688999999999998 2024-05-15T13:29:42.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.477 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.477 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.404457+0000", 2024-05-15T13:29:42.477 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0145000000000002e-05 2024-05-15T13:29:42.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.478 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:42.478 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.404504+0000", 2024-05-15T13:29:42.478 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7086000000000003e-05 2024-05-15T13:29:42.478 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.478 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.478 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:42.478 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:01.404619+0000", 2024-05-15T13:29:42.478 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011473500000000001 2024-05-15T13:29:42.478 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.478 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:42.478 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.478 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.478 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.478 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:194717 9.8 9:192ed2e0:::benchmark_data_smithi177_159919_object74474:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-05-15T13:29:42.478 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:02.460874+0000", 2024-05-15T13:29:42.479 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 99.947887800000004, 2024-05-15T13:29:42.479 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0629718880000001, 2024-05-15T13:29:42.479 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:42.479 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:42.479 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:42.479 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:42.479 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:42.479 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 194717 2024-05-15T13:29:42.479 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.479 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:42.479 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.479 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:42.479 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.460874+0000", 2024-05-15T13:29:42.480 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.480 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.480 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.480 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:42.480 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.460874+0000", 2024-05-15T13:29:42.480 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.480 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.480 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.480 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:42.480 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.460875+0000", 2024-05-15T13:29:42.480 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.643e-06 2024-05-15T13:29:42.480 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.480 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.480 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:42.480 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.460880+0000", 2024-05-15T13:29:42.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.826e-06 2024-05-15T13:29:42.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:42.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.460891+0000", 2024-05-15T13:29:42.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0803e-05 2024-05-15T13:29:42.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:42.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.460897+0000", 2024-05-15T13:29:42.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0820000000000004e-06 2024-05-15T13:29:42.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.460946+0000", 2024-05-15T13:29:42.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8603999999999997e-05 2024-05-15T13:29:42.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:29:42.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.460957+0000", 2024-05-15T13:29:42.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0889e-05 2024-05-15T13:29:42.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.507731+0000", 2024-05-15T13:29:42.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0467743359999999 2024-05-15T13:29:42.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:42.483 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.507836+0000", 2024-05-15T13:29:42.483 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000104861 2024-05-15T13:29:42.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.483 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,11", 2024-05-15T13:29:42.483 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.507919+0000", 2024-05-15T13:29:42.483 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3533000000000001e-05 2024-05-15T13:29:42.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.483 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.483 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.523612+0000", 2024-05-15T13:29:42.483 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015692718000000001 2024-05-15T13:29:42.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.483 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.484 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.523666+0000", 2024-05-15T13:29:42.484 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4098999999999998e-05 2024-05-15T13:29:42.484 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.484 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.484 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:42.484 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.523737+0000", 2024-05-15T13:29:42.484 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0956000000000003e-05 2024-05-15T13:29:42.484 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.484 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.484 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:42.484 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.523792+0000", 2024-05-15T13:29:42.484 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4434000000000001e-05 2024-05-15T13:29:42.484 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.484 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.484 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:42.485 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.523846+0000", 2024-05-15T13:29:42.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4104000000000002e-05 2024-05-15T13:29:42.485 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.485 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:42.485 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.485 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.485 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:194722 9.8 9:16952174:::benchmark_data_smithi177_159919_object74479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-05-15T13:29:42.485 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:02.477206+0000", 2024-05-15T13:29:42.485 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 99.931556036000003, 2024-05-15T13:29:42.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.046799212, 2024-05-15T13:29:42.485 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:42.485 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:42.485 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:42.486 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:42.486 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:42.486 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 194722 2024-05-15T13:29:42.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.486 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:42.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:42.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.477206+0000", 2024-05-15T13:29:42.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:42.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.477206+0000", 2024-05-15T13:29:42.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.487 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:42.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.477207+0000", 2024-05-15T13:29:42.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3260000000000001e-06 2024-05-15T13:29:42.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.487 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:42.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.477213+0000", 2024-05-15T13:29:42.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2129999999999996e-06 2024-05-15T13:29:42.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.487 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:42.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.477225+0000", 2024-05-15T13:29:42.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1827999999999999e-05 2024-05-15T13:29:42.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.488 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:42.488 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.477234+0000", 2024-05-15T13:29:42.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1740000000000006e-06 2024-05-15T13:29:42.488 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.488 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.488 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.477289+0000", 2024-05-15T13:29:42.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5129e-05 2024-05-15T13:29:42.488 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.488 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:29:42.488 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.477298+0000", 2024-05-15T13:29:42.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1989999999999995e-06 2024-05-15T13:29:42.488 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.513239+0000", 2024-05-15T13:29:42.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.035940656 2024-05-15T13:29:42.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:42.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.513367+0000", 2024-05-15T13:29:42.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012786899999999999 2024-05-15T13:29:42.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,11", 2024-05-15T13:29:42.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.513496+0000", 2024-05-15T13:29:42.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000128985 2024-05-15T13:29:42.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.523721+0000", 2024-05-15T13:29:42.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010225438 2024-05-15T13:29:42.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.523876+0000", 2024-05-15T13:29:42.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015429500000000001 2024-05-15T13:29:42.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:42.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.523953+0000", 2024-05-15T13:29:42.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7218000000000005e-05 2024-05-15T13:29:42.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:42.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.523980+0000", 2024-05-15T13:29:42.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7481e-05 2024-05-15T13:29:42.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:42.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.524005+0000", 2024-05-15T13:29:42.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4400999999999999e-05 2024-05-15T13:29:42.491 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.491 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:42.491 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.492 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:194729 9.8 9:1e9b5921:::benchmark_data_smithi177_159919_object74486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-05-15T13:29:42.492 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:02.503167+0000", 2024-05-15T13:29:42.492 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 99.905594937000004, 2024-05-15T13:29:42.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.030134828, 2024-05-15T13:29:42.492 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:42.492 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:42.492 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:42.492 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:42.492 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:42.492 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 194729 2024-05-15T13:29:42.492 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.492 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:42.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.493 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:42.493 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.503167+0000", 2024-05-15T13:29:42.493 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.493 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.493 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.493 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:42.493 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.503167+0000", 2024-05-15T13:29:42.493 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.493 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.493 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.493 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:42.493 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.503168+0000", 2024-05-15T13:29:42.493 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1200000000000001e-06 2024-05-15T13:29:42.493 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:42.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.503173+0000", 2024-05-15T13:29:42.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1420000000000001e-06 2024-05-15T13:29:42.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:42.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.503185+0000", 2024-05-15T13:29:42.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2347999999999999e-05 2024-05-15T13:29:42.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:42.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.503193+0000", 2024-05-15T13:29:42.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1380000000000007e-06 2024-05-15T13:29:42.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.503238+0000", 2024-05-15T13:29:42.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4824999999999999e-05 2024-05-15T13:29:42.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:29:42.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.503244+0000", 2024-05-15T13:29:42.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.959e-06 2024-05-15T13:29:42.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.518425+0000", 2024-05-15T13:29:42.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.015180653 2024-05-15T13:29:42.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:42.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.518557+0000", 2024-05-15T13:29:42.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013191799999999999 2024-05-15T13:29:42.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,11", 2024-05-15T13:29:42.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.518638+0000", 2024-05-15T13:29:42.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1298000000000007e-05 2024-05-15T13:29:42.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.524065+0000", 2024-05-15T13:29:42.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0054273969999999996 2024-05-15T13:29:42.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:42.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.529115+0000", 2024-05-15T13:29:42.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0050496559999999996 2024-05-15T13:29:42.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.533210+0000", 2024-05-15T13:29:42.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0040950179999999997 2024-05-15T13:29:42.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:42.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.533250+0000", 2024-05-15T13:29:42.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9783999999999997e-05 2024-05-15T13:29:42.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.498 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:42.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.533301+0000", 2024-05-15T13:29:42.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1572e-05 2024-05-15T13:29:42.498 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.498 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:42.498 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.498 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:194737 9.8 9:1775bed3:::benchmark_data_smithi177_159919_object74494:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-05-15T13:29:42.498 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:02.511792+0000", 2024-05-15T13:29:42.498 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 99.896969990000002, 2024-05-15T13:29:42.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.041376554, 2024-05-15T13:29:42.499 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:42.499 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:42.499 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:42.499 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:42.499 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:42.499 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 194737 2024-05-15T13:29:42.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.499 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:42.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:42.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.511792+0000", 2024-05-15T13:29:42.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:42.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.511792+0000", 2024-05-15T13:29:42.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:42.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.511793+0000", 2024-05-15T13:29:42.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.271e-06 2024-05-15T13:29:42.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:42.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.511799+0000", 2024-05-15T13:29:42.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8139999999999997e-06 2024-05-15T13:29:42.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:42.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.511812+0000", 2024-05-15T13:29:42.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3655999999999999e-05 2024-05-15T13:29:42.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:42.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.511820+0000", 2024-05-15T13:29:42.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0499999999999992e-06 2024-05-15T13:29:42.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.511880+0000", 2024-05-15T13:29:42.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9351999999999997e-05 2024-05-15T13:29:42.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:29:42.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.511889+0000", 2024-05-15T13:29:42.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4210000000000001e-06 2024-05-15T13:29:42.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.524116+0000", 2024-05-15T13:29:42.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0122272990000001 2024-05-15T13:29:42.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:42.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.524217+0000", 2024-05-15T13:29:42.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000100407 2024-05-15T13:29:42.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,11", 2024-05-15T13:29:42.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.524298+0000", 2024-05-15T13:29:42.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0792999999999997e-05 2024-05-15T13:29:42.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.533347+0000", 2024-05-15T13:29:42.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.009049507 2024-05-15T13:29:42.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:42.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.538168+0000", 2024-05-15T13:29:42.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0048212799999999998 2024-05-15T13:29:42.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.553097+0000", 2024-05-15T13:29:42.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014928733 2024-05-15T13:29:42.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:42.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.553131+0000", 2024-05-15T13:29:42.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4121000000000001e-05 2024-05-15T13:29:42.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:42.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.553168+0000", 2024-05-15T13:29:42.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6850000000000001e-05 2024-05-15T13:29:42.505 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.505 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:42.505 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.505 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:194758 9.8 9:1ec96b9e:::benchmark_data_smithi177_159919_object74515:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-05-15T13:29:42.505 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:02.558665+0000", 2024-05-15T13:29:42.505 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 99.850096094999998, 2024-05-15T13:29:42.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.99466317400000004, 2024-05-15T13:29:42.505 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:42.505 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:42.505 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:42.505 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:42.505 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:42.506 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 194758 2024-05-15T13:29:42.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.506 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:42.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.506 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:42.506 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.558665+0000", 2024-05-15T13:29:42.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.506 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:42.506 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.558665+0000", 2024-05-15T13:29:42.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:42.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.558667+0000", 2024-05-15T13:29:42.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5799999999999999e-06 2024-05-15T13:29:42.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:42.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.558674+0000", 2024-05-15T13:29:42.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9040000000000003e-06 2024-05-15T13:29:42.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:42.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.558688+0000", 2024-05-15T13:29:42.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4194e-05 2024-05-15T13:29:42.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:42.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.558697+0000", 2024-05-15T13:29:42.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3230000000000004e-06 2024-05-15T13:29:42.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.558759+0000", 2024-05-15T13:29:42.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1185000000000005e-05 2024-05-15T13:29:42.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:29:42.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.558769+0000", 2024-05-15T13:29:42.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0778000000000001e-05 2024-05-15T13:29:42.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.529150+0000", 2024-05-15T13:29:42.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.970380197 2024-05-15T13:29:42.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:42.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.529233+0000", 2024-05-15T13:29:42.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3855000000000003e-05 2024-05-15T13:29:42.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,11", 2024-05-15T13:29:42.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.529314+0000", 2024-05-15T13:29:42.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0564999999999999e-05 2024-05-15T13:29:42.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:42.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.538175+0000", 2024-05-15T13:29:42.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.008860471 2024-05-15T13:29:42.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.544091+0000", 2024-05-15T13:29:42.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.005916653 2024-05-15T13:29:42.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.553230+0000", 2024-05-15T13:29:42.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0091392120000000007 2024-05-15T13:29:42.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:42.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.553289+0000", 2024-05-15T13:29:42.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8675000000000003e-05 2024-05-15T13:29:42.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:42.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.553329+0000", 2024-05-15T13:29:42.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9582000000000002e-05 2024-05-15T13:29:42.511 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.511 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:42.511 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.512 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:194788 9.8 9:1bcf6210:::benchmark_data_smithi177_159919_object74545:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-05-15T13:29:42.512 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:02.616873+0000", 2024-05-15T13:29:42.512 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 99.791889061000006, 2024-05-15T13:29:42.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96749579200000002, 2024-05-15T13:29:42.512 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:42.512 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:42.512 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:42.512 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:42.512 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:42.512 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 194788 2024-05-15T13:29:42.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.512 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:42.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:42.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.616873+0000", 2024-05-15T13:29:42.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:42.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.616873+0000", 2024-05-15T13:29:42.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:42.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.616874+0000", 2024-05-15T13:29:42.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1230000000000001e-06 2024-05-15T13:29:42.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:42.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.616879+0000", 2024-05-15T13:29:42.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6799999999999998e-06 2024-05-15T13:29:42.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:42.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.616890+0000", 2024-05-15T13:29:42.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0987e-05 2024-05-15T13:29:42.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:42.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.616898+0000", 2024-05-15T13:29:42.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0770000000000008e-06 2024-05-15T13:29:42.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.616962+0000", 2024-05-15T13:29:42.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4006999999999999e-05 2024-05-15T13:29:42.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:29:42.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.616973+0000", 2024-05-15T13:29:42.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0625000000000001e-05 2024-05-15T13:29:42.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.533401+0000", 2024-05-15T13:29:42.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.91642807599999998 2024-05-15T13:29:42.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:42.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.533509+0000", 2024-05-15T13:29:42.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000107451 2024-05-15T13:29:42.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,11", 2024-05-15T13:29:42.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.533600+0000", 2024-05-15T13:29:42.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0996999999999996e-05 2024-05-15T13:29:42.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:42.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.544104+0000", 2024-05-15T13:29:42.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010504037000000001 2024-05-15T13:29:42.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.544142+0000", 2024-05-15T13:29:42.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8127999999999998e-05 2024-05-15T13:29:42.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.584273+0000", 2024-05-15T13:29:42.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.040131501 2024-05-15T13:29:42.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:42.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.584324+0000", 2024-05-15T13:29:42.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0973000000000001e-05 2024-05-15T13:29:42.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:42.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.584368+0000", 2024-05-15T13:29:42.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4129999999999999e-05 2024-05-15T13:29:42.518 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.518 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:42.518 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.518 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:194789 9.8 9:1a36f6d1:::benchmark_data_smithi177_159919_object74546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-05-15T13:29:42.518 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:02.616930+0000", 2024-05-15T13:29:42.518 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 99.791832002999996, 2024-05-15T13:29:42.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.96756435100000004, 2024-05-15T13:29:42.518 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:42.519 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:42.519 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:42.519 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:42.519 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:42.519 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 194789 2024-05-15T13:29:42.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.519 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:42.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:42.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.616930+0000", 2024-05-15T13:29:42.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:42.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.616930+0000", 2024-05-15T13:29:42.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:42.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.616930+0000", 2024-05-15T13:29:42.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9100000000000004e-07 2024-05-15T13:29:42.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:42.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.616933+0000", 2024-05-15T13:29:42.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2499999999999998e-06 2024-05-15T13:29:42.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:42.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.616941+0000", 2024-05-15T13:29:42.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0409999999999998e-06 2024-05-15T13:29:42.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:42.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.616948+0000", 2024-05-15T13:29:42.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3090000000000001e-06 2024-05-15T13:29:42.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.616995+0000", 2024-05-15T13:29:42.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7454999999999999e-05 2024-05-15T13:29:42.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:29:42.522 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.617001+0000", 2024-05-15T13:29:42.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7320000000000003e-06 2024-05-15T13:29:42.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.522 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.538210+0000", 2024-05-15T13:29:42.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.92120857 2024-05-15T13:29:42.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:42.522 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.538322+0000", 2024-05-15T13:29:42.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000112061 2024-05-15T13:29:42.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,11", 2024-05-15T13:29:42.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.538440+0000", 2024-05-15T13:29:42.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011815300000000001 2024-05-15T13:29:42.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:42.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.553342+0000", 2024-05-15T13:29:42.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014902442 2024-05-15T13:29:42.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.558725+0000", 2024-05-15T13:29:42.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0053832740000000004 2024-05-15T13:29:42.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.584413+0000", 2024-05-15T13:29:42.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.025687617999999999 2024-05-15T13:29:42.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:42.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.584461+0000", 2024-05-15T13:29:42.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7682999999999997e-05 2024-05-15T13:29:42.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:42.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.584494+0000", 2024-05-15T13:29:42.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3172e-05 2024-05-15T13:29:42.525 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.525 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:42.525 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.525 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:194798 9.8 9:1224807a:::benchmark_data_smithi177_159919_object74555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-05-15T13:29:42.525 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:02.639000+0000", 2024-05-15T13:29:42.525 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 99.769761775000006, 2024-05-15T13:29:42.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.94563272899999995, 2024-05-15T13:29:42.525 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:42.525 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:42.525 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:42.525 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:42.525 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:42.525 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 194798 2024-05-15T13:29:42.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.526 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:42.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:42.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.639000+0000", 2024-05-15T13:29:42.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:42.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.639000+0000", 2024-05-15T13:29:42.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:42.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.639001+0000", 2024-05-15T13:29:42.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1370000000000001e-06 2024-05-15T13:29:42.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.527 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:42.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.639006+0000", 2024-05-15T13:29:42.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4199999999999998e-06 2024-05-15T13:29:42.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.527 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:42.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.639017+0000", 2024-05-15T13:29:42.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0572e-05 2024-05-15T13:29:42.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.527 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:42.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.639024+0000", 2024-05-15T13:29:42.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.131e-06 2024-05-15T13:29:42.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.639075+0000", 2024-05-15T13:29:42.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1041000000000001e-05 2024-05-15T13:29:42.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:29:42.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.639082+0000", 2024-05-15T13:29:42.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5200000000000003e-06 2024-05-15T13:29:42.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.544170+0000", 2024-05-15T13:29:42.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.90508855899999996 2024-05-15T13:29:42.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:42.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.544286+0000", 2024-05-15T13:29:42.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115371 2024-05-15T13:29:42.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,11", 2024-05-15T13:29:42.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.544404+0000", 2024-05-15T13:29:42.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000118118 2024-05-15T13:29:42.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:42.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.557456+0000", 2024-05-15T13:29:42.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013051884999999999 2024-05-15T13:29:42.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.558781+0000", 2024-05-15T13:29:42.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013252629999999999 2024-05-15T13:29:42.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.584538+0000", 2024-05-15T13:29:42.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.025756863000000001 2024-05-15T13:29:42.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:42.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.584597+0000", 2024-05-15T13:29:42.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9070000000000002e-05 2024-05-15T13:29:42.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:42.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.584633+0000", 2024-05-15T13:29:42.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5778999999999997e-05 2024-05-15T13:29:42.531 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.531 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:42.531 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.531 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:194813 9.8 9:175e5813:::benchmark_data_smithi177_159919_object74570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-05-15T13:29:42.531 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:02.697111+0000", 2024-05-15T13:29:42.532 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 99.711650650999999, 2024-05-15T13:29:42.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.88765318199999999, 2024-05-15T13:29:42.532 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:42.532 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:42.532 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:42.532 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:42.532 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:42.532 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 194813 2024-05-15T13:29:42.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.532 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:42.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.532 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:42.532 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.697111+0000", 2024-05-15T13:29:42.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:42.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.697111+0000", 2024-05-15T13:29:42.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:42.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.697113+0000", 2024-05-15T13:29:42.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.59e-06 2024-05-15T13:29:42.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:42.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.697118+0000", 2024-05-15T13:29:42.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3410000000000002e-06 2024-05-15T13:29:42.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:42.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.697130+0000", 2024-05-15T13:29:42.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2347999999999999e-05 2024-05-15T13:29:42.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:42.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.697137+0000", 2024-05-15T13:29:42.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9739999999999996e-06 2024-05-15T13:29:42.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.697186+0000", 2024-05-15T13:29:42.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8639e-05 2024-05-15T13:29:42.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:29:42.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.697193+0000", 2024-05-15T13:29:42.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7209999999999997e-06 2024-05-15T13:29:42.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.548925+0000", 2024-05-15T13:29:42.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.85173268499999999 2024-05-15T13:29:42.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:42.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.549038+0000", 2024-05-15T13:29:42.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011314100000000001 2024-05-15T13:29:42.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,11", 2024-05-15T13:29:42.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.549120+0000", 2024-05-15T13:29:42.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2119000000000004e-05 2024-05-15T13:29:42.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:42.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.557461+0000", 2024-05-15T13:29:42.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0083401480000000004 2024-05-15T13:29:42.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.561518+0000", 2024-05-15T13:29:42.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0040576850000000001 2024-05-15T13:29:42.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.584677+0000", 2024-05-15T13:29:42.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.023159183 2024-05-15T13:29:42.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:42.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.584722+0000", 2024-05-15T13:29:42.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4721000000000001e-05 2024-05-15T13:29:42.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:42.538 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.584764+0000", 2024-05-15T13:29:42.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1887000000000001e-05 2024-05-15T13:29:42.538 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.538 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:42.538 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.538 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:194820 9.8 9:1981c359:::benchmark_data_smithi177_159919_object74577:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-05-15T13:29:42.538 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:02.732783+0000", 2024-05-15T13:29:42.538 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 99.675979002000005, 2024-05-15T13:29:42.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.85211658000000001, 2024-05-15T13:29:42.538 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:42.538 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:42.538 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:42.539 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:42.539 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:42.539 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 194820 2024-05-15T13:29:42.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.539 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:42.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:42.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.732783+0000", 2024-05-15T13:29:42.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:42.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.732783+0000", 2024-05-15T13:29:42.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:42.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:42.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.732785+0000", 2024-05-15T13:29:42.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1500000000000002e-06 2024-05-15T13:29:42.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:42.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.732790+0000", 2024-05-15T13:29:42.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5330000000000002e-06 2024-05-15T13:29:42.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:42.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.732804+0000", 2024-05-15T13:29:42.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3867e-05 2024-05-15T13:29:42.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:42.541 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.732812+0000", 2024-05-15T13:29:42.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1089999999999998e-06 2024-05-15T13:29:42.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.541 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.732864+0000", 2024-05-15T13:29:42.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2138000000000001e-05 2024-05-15T13:29:42.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:29:42.541 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:02.732871+0000", 2024-05-15T13:29:42.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7599999999999997e-06 2024-05-15T13:29:42.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:42.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.553369+0000", 2024-05-15T13:29:42.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.82049768199999995 2024-05-15T13:29:42.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:42.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.553489+0000", 2024-05-15T13:29:42.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120399 2024-05-15T13:29:42.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,11", 2024-05-15T13:29:42.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.553622+0000", 2024-05-15T13:29:42.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013295399999999999 2024-05-15T13:29:42.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:42.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.557548+0000", 2024-05-15T13:29:42.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003925942 2024-05-15T13:29:42.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.561634+0000", 2024-05-15T13:29:42.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004086381 2024-05-15T13:29:42.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:42.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.584818+0000", 2024-05-15T13:29:42.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.023183754000000001 2024-05-15T13:29:42.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:42.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.584865+0000", 2024-05-15T13:29:42.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7123000000000003e-05 2024-05-15T13:29:42.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:42.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:42.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:42.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:03.584899+0000", 2024-05-15T13:29:42.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3788000000000002e-05 2024-05-15T13:29:42.544 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.544 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:42.544 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.544 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:42.545 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:42.545 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:29:42.545 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:42 smithi120 ceph-mon[43961]: pgmap v2484: 31 pgs: 31 active+clean; 590 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:42.546 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:42 smithi027 ceph-mon[39795]: pgmap v2484: 31 pgs: 31 active+clean; 590 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:42.546 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:42 smithi177 ceph-mon[44303]: pgmap v2484: 31 pgs: 31 active+clean; 590 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:43.122 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-05-15T13:29:43.161 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-05-15T13:29:43.424 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:29:43.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:43 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3703666913' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:29:43.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:43 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3703666913' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:29:43.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:43 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3703666913' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T13:29:43.931 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T13:29:44.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:44 smithi027 ceph-mon[39795]: pgmap v2485: 31 pgs: 31 active+clean; 590 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:44.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:44 smithi027 ceph-mon[39795]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:29:44.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:44 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3703666913' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:29:44.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:44 smithi027 ceph-mon[39795]: osdmap e985: 12 total, 12 up, 11 in 2024-05-15T13:29:44.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:44 smithi027 ceph-mon[39795]: 1.5 scrub starts 2024-05-15T13:29:44.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:44 smithi027 ceph-mon[39795]: 1.5 scrub ok 2024-05-15T13:29:44.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:44 smithi177 ceph-mon[44303]: pgmap v2485: 31 pgs: 31 active+clean; 590 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:44.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:44 smithi177 ceph-mon[44303]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:29:44.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:44 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3703666913' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:29:44.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:44 smithi177 ceph-mon[44303]: osdmap e985: 12 total, 12 up, 11 in 2024-05-15T13:29:44.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:44 smithi177 ceph-mon[44303]: 1.5 scrub starts 2024-05-15T13:29:44.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:44 smithi177 ceph-mon[44303]: 1.5 scrub ok 2024-05-15T13:29:44.443 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:29:44.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:44 smithi120 ceph-mon[43961]: pgmap v2485: 31 pgs: 31 active+clean; 590 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:44.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:44 smithi120 ceph-mon[43961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T13:29:44.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:44 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3703666913' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T13:29:44.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:44 smithi120 ceph-mon[43961]: osdmap e985: 12 total, 12 up, 11 in 2024-05-15T13:29:44.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:44 smithi120 ceph-mon[43961]: 1.5 scrub starts 2024-05-15T13:29:44.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:44 smithi120 ceph-mon[43961]: 1.5 scrub ok 2024-05-15T13:29:45.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:45 smithi027 ceph-mon[39795]: 1.c scrub starts 2024-05-15T13:29:45.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:45 smithi027 ceph-mon[39795]: 1.c scrub ok 2024-05-15T13:29:45.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:45 smithi027 ceph-mon[39795]: 1.1 scrub starts 2024-05-15T13:29:45.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:45 smithi027 ceph-mon[39795]: 1.1 scrub ok 2024-05-15T13:29:45.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:45 smithi027 ceph-mon[39795]: 1.1c scrub starts 2024-05-15T13:29:45.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:45 smithi027 ceph-mon[39795]: 1.1c scrub ok 2024-05-15T13:29:45.357 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:29:45.357 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T13:29:45.357 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-05-15T13:29:45.357 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:29:45.378 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:45 smithi177 ceph-mon[44303]: 1.c scrub starts 2024-05-15T13:29:45.378 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:45 smithi177 ceph-mon[44303]: 1.c scrub ok 2024-05-15T13:29:45.378 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:45 smithi177 ceph-mon[44303]: 1.1 scrub starts 2024-05-15T13:29:45.378 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:45 smithi177 ceph-mon[44303]: 1.1 scrub ok 2024-05-15T13:29:45.378 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:45 smithi177 ceph-mon[44303]: 1.1c scrub starts 2024-05-15T13:29:45.378 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:45 smithi177 ceph-mon[44303]: 1.1c scrub ok 2024-05-15T13:29:45.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:45 smithi120 ceph-mon[43961]: 1.c scrub starts 2024-05-15T13:29:45.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:45 smithi120 ceph-mon[43961]: 1.c scrub ok 2024-05-15T13:29:45.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:45 smithi120 ceph-mon[43961]: 1.1 scrub starts 2024-05-15T13:29:45.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:45 smithi120 ceph-mon[43961]: 1.1 scrub ok 2024-05-15T13:29:45.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:45 smithi120 ceph-mon[43961]: 1.1c scrub starts 2024-05-15T13:29:45.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:45 smithi120 ceph-mon[43961]: 1.1c scrub ok 2024-05-15T13:29:46.124 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-05-15T13:29:46.129 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.4 to 1 (10000) 2024-05-15T13:29:46.384 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:29:46.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:46 smithi177 ceph-mon[44303]: 1.8 scrub starts 2024-05-15T13:29:46.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:46 smithi177 ceph-mon[44303]: 1.8 scrub ok 2024-05-15T13:29:46.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:46 smithi177 ceph-mon[44303]: pgmap v2487: 31 pgs: 31 active+clean; 590 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:46.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:46 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3743971310' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T13:29:46.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:46 smithi177 ceph-mon[44303]: 1.19 scrub starts 2024-05-15T13:29:46.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:46 smithi177 ceph-mon[44303]: 1.19 scrub ok 2024-05-15T13:29:46.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:46 smithi027 ceph-mon[39795]: 1.8 scrub starts 2024-05-15T13:29:46.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:46 smithi027 ceph-mon[39795]: 1.8 scrub ok 2024-05-15T13:29:46.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:46 smithi027 ceph-mon[39795]: pgmap v2487: 31 pgs: 31 active+clean; 590 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:46.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:46 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3743971310' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T13:29:46.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:46 smithi027 ceph-mon[39795]: 1.19 scrub starts 2024-05-15T13:29:46.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:46 smithi027 ceph-mon[39795]: 1.19 scrub ok 2024-05-15T13:29:46.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:46 smithi120 ceph-mon[43961]: 1.8 scrub starts 2024-05-15T13:29:46.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:46 smithi120 ceph-mon[43961]: 1.8 scrub ok 2024-05-15T13:29:46.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:46 smithi120 ceph-mon[43961]: pgmap v2487: 31 pgs: 31 active+clean; 590 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:46.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:46 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3743971310' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T13:29:46.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:46 smithi120 ceph-mon[43961]: 1.19 scrub starts 2024-05-15T13:29:46.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:46 smithi120 ceph-mon[43961]: 1.19 scrub ok 2024-05-15T13:29:47.094 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 8 1 2024-05-15T13:29:47.142 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-05-15T13:29:47.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:47 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3743971310' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T13:29:47.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:47 smithi177 ceph-mon[44303]: osdmap e986: 12 total, 12 up, 11 in 2024-05-15T13:29:47.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:47 smithi177 ceph-mon[44303]: 1.1b scrub starts 2024-05-15T13:29:47.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:47 smithi177 ceph-mon[44303]: 1.1b scrub ok 2024-05-15T13:29:47.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:47 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/353810947' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:29:47.459 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:47 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3743971310' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T13:29:47.459 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:47 smithi027 ceph-mon[39795]: osdmap e986: 12 total, 12 up, 11 in 2024-05-15T13:29:47.459 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:47 smithi027 ceph-mon[39795]: 1.1b scrub starts 2024-05-15T13:29:47.459 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:47 smithi027 ceph-mon[39795]: 1.1b scrub ok 2024-05-15T13:29:47.459 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:47 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/353810947' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:29:47.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:47 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3743971310' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T13:29:47.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:47 smithi120 ceph-mon[43961]: osdmap e986: 12 total, 12 up, 11 in 2024-05-15T13:29:47.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:47 smithi120 ceph-mon[43961]: 1.1b scrub starts 2024-05-15T13:29:47.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:47 smithi120 ceph-mon[43961]: 1.1b scrub ok 2024-05-15T13:29:47.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:47 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/353810947' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T13:29:47.811 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:29:48.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:48 smithi177 ceph-mon[44303]: 1.18 scrub starts 2024-05-15T13:29:48.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:48 smithi177 ceph-mon[44303]: 1.18 scrub ok 2024-05-15T13:29:48.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:48 smithi177 ceph-mon[44303]: pgmap v2489: 31 pgs: 31 active+clean; 590 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:48.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:48 smithi177 ceph-mon[44303]: 1.a scrub starts 2024-05-15T13:29:48.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:48 smithi177 ceph-mon[44303]: 1.a scrub ok 2024-05-15T13:29:48.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:48 smithi177 ceph-mon[44303]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:29:48.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:48 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T13:29:48.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:48 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/353810947' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:29:48.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:48 smithi177 ceph-mon[44303]: osdmap e987: 12 total, 12 up, 11 in 2024-05-15T13:29:48.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:48 smithi177 ceph-mon[44303]: 1.1d scrub starts 2024-05-15T13:29:48.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:48 smithi177 ceph-mon[44303]: 1.1d scrub ok 2024-05-15T13:29:48.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:48 smithi177 ceph-mon[44303]: 1.17 scrub starts 2024-05-15T13:29:48.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:48 smithi177 ceph-mon[44303]: 1.17 scrub ok 2024-05-15T13:29:48.205 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:29:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T13:29:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-05-15T13:29:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-05-15T13:29:48.205 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:29:48.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:48 smithi027 ceph-mon[39795]: 1.18 scrub starts 2024-05-15T13:29:48.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:48 smithi027 ceph-mon[39795]: 1.18 scrub ok 2024-05-15T13:29:48.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:48 smithi027 ceph-mon[39795]: pgmap v2489: 31 pgs: 31 active+clean; 590 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:48.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:48 smithi027 ceph-mon[39795]: 1.a scrub starts 2024-05-15T13:29:48.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:48 smithi027 ceph-mon[39795]: 1.a scrub ok 2024-05-15T13:29:48.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:48 smithi027 ceph-mon[39795]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:29:48.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:48 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T13:29:48.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:48 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/353810947' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:29:48.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:48 smithi027 ceph-mon[39795]: osdmap e987: 12 total, 12 up, 11 in 2024-05-15T13:29:48.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:48 smithi027 ceph-mon[39795]: 1.1d scrub starts 2024-05-15T13:29:48.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:48 smithi027 ceph-mon[39795]: 1.1d scrub ok 2024-05-15T13:29:48.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:48 smithi027 ceph-mon[39795]: 1.17 scrub starts 2024-05-15T13:29:48.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:48 smithi027 ceph-mon[39795]: 1.17 scrub ok 2024-05-15T13:29:48.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:48 smithi120 ceph-mon[43961]: 1.18 scrub starts 2024-05-15T13:29:48.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:48 smithi120 ceph-mon[43961]: 1.18 scrub ok 2024-05-15T13:29:48.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:48 smithi120 ceph-mon[43961]: pgmap v2489: 31 pgs: 31 active+clean; 590 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:48.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:48 smithi120 ceph-mon[43961]: 1.a scrub starts 2024-05-15T13:29:48.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:48 smithi120 ceph-mon[43961]: 1.a scrub ok 2024-05-15T13:29:48.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:48 smithi120 ceph-mon[43961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T13:29:48.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:48 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T13:29:48.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:48 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/353810947' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T13:29:48.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:48 smithi120 ceph-mon[43961]: osdmap e987: 12 total, 12 up, 11 in 2024-05-15T13:29:48.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:48 smithi120 ceph-mon[43961]: 1.1d scrub starts 2024-05-15T13:29:48.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:48 smithi120 ceph-mon[43961]: 1.1d scrub ok 2024-05-15T13:29:48.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:48 smithi120 ceph-mon[43961]: 1.17 scrub starts 2024-05-15T13:29:48.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:48 smithi120 ceph-mon[43961]: 1.17 scrub ok 2024-05-15T13:29:48.875 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-05-15T13:29:49.140 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:29:49.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:49 smithi177 ceph-mon[44303]: 1.12 scrub starts 2024-05-15T13:29:49.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:49 smithi177 ceph-mon[44303]: 1.12 scrub ok 2024-05-15T13:29:49.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:49 smithi177 ceph-mon[44303]: 1.16 scrub starts 2024-05-15T13:29:49.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:49 smithi177 ceph-mon[44303]: 1.16 scrub ok 2024-05-15T13:29:49.405 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:49 smithi027 ceph-mon[39795]: 1.12 scrub starts 2024-05-15T13:29:49.405 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:49 smithi027 ceph-mon[39795]: 1.12 scrub ok 2024-05-15T13:29:49.405 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:49 smithi027 ceph-mon[39795]: 1.16 scrub starts 2024-05-15T13:29:49.405 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:49 smithi027 ceph-mon[39795]: 1.16 scrub ok 2024-05-15T13:29:49.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:49 smithi120 ceph-mon[43961]: 1.12 scrub starts 2024-05-15T13:29:49.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:49 smithi120 ceph-mon[43961]: 1.12 scrub ok 2024-05-15T13:29:49.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:49 smithi120 ceph-mon[43961]: 1.16 scrub starts 2024-05-15T13:29:49.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:49 smithi120 ceph-mon[43961]: 1.16 scrub ok 2024-05-15T13:29:50.385 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.8 to 1 (10000) 2024-05-15T13:29:50.406 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:50 smithi027 ceph-mon[39795]: pgmap v2491: 31 pgs: 31 active+clean; 590 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:50.406 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:50 smithi027 ceph-mon[39795]: 1.1a scrub starts 2024-05-15T13:29:50.406 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:50 smithi027 ceph-mon[39795]: 1.1a scrub ok 2024-05-15T13:29:50.407 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:50 smithi027 ceph-mon[39795]: 1.11 scrub starts 2024-05-15T13:29:50.407 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:50 smithi027 ceph-mon[39795]: 1.11 scrub ok 2024-05-15T13:29:50.407 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:50 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2434167424' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T13:29:50.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:50 smithi177 ceph-mon[44303]: pgmap v2491: 31 pgs: 31 active+clean; 590 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:50.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:50 smithi177 ceph-mon[44303]: 1.1a scrub starts 2024-05-15T13:29:50.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:50 smithi177 ceph-mon[44303]: 1.1a scrub ok 2024-05-15T13:29:50.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:50 smithi177 ceph-mon[44303]: 1.11 scrub starts 2024-05-15T13:29:50.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:50 smithi177 ceph-mon[44303]: 1.11 scrub ok 2024-05-15T13:29:50.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:50 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2434167424' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T13:29:50.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:50 smithi120 ceph-mon[43961]: pgmap v2491: 31 pgs: 31 active+clean; 590 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:50.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:50 smithi120 ceph-mon[43961]: 1.1a scrub starts 2024-05-15T13:29:50.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:50 smithi120 ceph-mon[43961]: 1.1a scrub ok 2024-05-15T13:29:50.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:50 smithi120 ceph-mon[43961]: 1.11 scrub starts 2024-05-15T13:29:50.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:50 smithi120 ceph-mon[43961]: 1.11 scrub ok 2024-05-15T13:29:50.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:50 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2434167424' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T13:29:51.074 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 11 1 2024-05-15T13:29:51.222 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:29:51.223 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-05-15T13:29:51.223 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-05-15T13:29:51.223 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T13:29:51.223 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.223 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:1523 9.7 9:e3dfccba:::benchmark_data_smithi177_159919_object1522:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-05-15T13:29:51.223 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:23:12.787806+0000", 2024-05-15T13:29:51.223 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 398.430893846, 2024-05-15T13:29:51.223 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.203535101, 2024-05-15T13:29:51.223 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:51.223 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:51.223 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:51.223 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:51.223 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:51.224 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1523 2024-05-15T13:29:51.224 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.224 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:51.224 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.224 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:51.224 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.787806+0000", 2024-05-15T13:29:51.224 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.224 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.224 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.224 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:51.224 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.787806+0000", 2024-05-15T13:29:51.224 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.224 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.224 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.224 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:51.225 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.787807+0000", 2024-05-15T13:29:51.225 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.009e-06 2024-05-15T13:29:51.225 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.225 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.225 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:51.225 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.787824+0000", 2024-05-15T13:29:51.225 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7255999999999999e-05 2024-05-15T13:29:51.225 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.225 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.225 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:51.225 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.787834+0000", 2024-05-15T13:29:51.225 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0148e-05 2024-05-15T13:29:51.225 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.225 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.226 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:51.226 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.787841+0000", 2024-05-15T13:29:51.226 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3420000000000002e-06 2024-05-15T13:29:51.226 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.226 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.226 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:51.226 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.787900+0000", 2024-05-15T13:29:51.226 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9199e-05 2024-05-15T13:29:51.226 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.226 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.226 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:51.226 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.788048+0000", 2024-05-15T13:29:51.227 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000147801 2024-05-15T13:29:51.227 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.227 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.227 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,7", 2024-05-15T13:29:51.227 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.788170+0000", 2024-05-15T13:29:51.227 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000121833 2024-05-15T13:29:51.227 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.227 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.227 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:51.227 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.788530+0000", 2024-05-15T13:29:51.227 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00036018299999999998 2024-05-15T13:29:51.227 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.227 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.227 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.227 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.790804+0000", 2024-05-15T13:29:51.228 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0022733720000000001 2024-05-15T13:29:51.228 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.228 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.228 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.228 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.991234+0000", 2024-05-15T13:29:51.228 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20043037499999999 2024-05-15T13:29:51.228 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.228 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.228 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:51.228 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.991283+0000", 2024-05-15T13:29:51.228 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9398999999999999e-05 2024-05-15T13:29:51.228 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.229 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.229 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:51.229 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.991341+0000", 2024-05-15T13:29:51.229 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7184e-05 2024-05-15T13:29:51.229 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.229 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:51.229 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.229 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.229 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.229 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:1529 9.7 9:e52d796a:::benchmark_data_smithi177_159919_object1528:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-05-15T13:29:51.229 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:23:12.788845+0000", 2024-05-15T13:29:51.229 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 398.42985472599997, 2024-05-15T13:29:51.229 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20353689999999999, 2024-05-15T13:29:51.229 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:51.230 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:51.230 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:51.230 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:51.230 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:51.230 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1529 2024-05-15T13:29:51.230 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.230 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:51.230 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.230 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:51.230 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.788845+0000", 2024-05-15T13:29:51.230 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.230 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.230 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.231 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:51.231 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.788845+0000", 2024-05-15T13:29:51.231 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.231 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.231 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.231 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:51.231 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.788845+0000", 2024-05-15T13:29:51.231 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1099999999999995e-07 2024-05-15T13:29:51.231 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.231 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.231 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:51.231 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.788865+0000", 2024-05-15T13:29:51.231 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9967e-05 2024-05-15T13:29:51.231 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.232 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.232 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:51.232 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.788876+0000", 2024-05-15T13:29:51.232 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0406e-05 2024-05-15T13:29:51.232 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.232 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.232 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:51.232 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.788883+0000", 2024-05-15T13:29:51.232 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.294e-06 2024-05-15T13:29:51.232 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.232 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.232 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:51.232 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.788928+0000", 2024-05-15T13:29:51.232 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4530999999999998e-05 2024-05-15T13:29:51.232 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.233 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.233 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:51.233 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.789020+0000", 2024-05-15T13:29:51.233 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2836000000000004e-05 2024-05-15T13:29:51.233 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.233 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.233 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,7", 2024-05-15T13:29:51.233 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.789153+0000", 2024-05-15T13:29:51.233 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013306799999999999 2024-05-15T13:29:51.233 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.233 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.233 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:51.233 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.789537+0000", 2024-05-15T13:29:51.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00038396799999999999 2024-05-15T13:29:51.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.234 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.791579+0000", 2024-05-15T13:29:51.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020412479999999998 2024-05-15T13:29:51.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.234 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.992306+0000", 2024-05-15T13:29:51.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20072783 2024-05-15T13:29:51.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.234 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:51.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.992340+0000", 2024-05-15T13:29:51.235 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3549e-05 2024-05-15T13:29:51.235 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.235 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.235 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:51.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.992382+0000", 2024-05-15T13:29:51.235 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1492000000000002e-05 2024-05-15T13:29:51.235 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.235 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:51.235 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.235 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.235 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.235 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:1545 9.7 9:e45a13b4:::benchmark_data_smithi177_159919_object1544:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-05-15T13:29:51.235 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:23:12.794444+0000", 2024-05-15T13:29:51.236 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 398.42425525800002, 2024-05-15T13:29:51.236 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19834131099999999, 2024-05-15T13:29:51.236 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:51.236 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:51.236 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:51.236 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:51.236 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:51.236 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1545 2024-05-15T13:29:51.236 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.236 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:51.236 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.236 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:51.236 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.794444+0000", 2024-05-15T13:29:51.236 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.237 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.237 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.237 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:51.237 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.794444+0000", 2024-05-15T13:29:51.237 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.237 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.237 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.237 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:51.237 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.794445+0000", 2024-05-15T13:29:51.237 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1500000000000003e-07 2024-05-15T13:29:51.237 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.237 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.237 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:51.237 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.794466+0000", 2024-05-15T13:29:51.238 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1183999999999999e-05 2024-05-15T13:29:51.238 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.238 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.238 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:51.238 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.794480+0000", 2024-05-15T13:29:51.238 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3511e-05 2024-05-15T13:29:51.238 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.238 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.238 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:51.238 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.794488+0000", 2024-05-15T13:29:51.238 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0749999999999998e-06 2024-05-15T13:29:51.238 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.238 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.238 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:51.239 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.794538+0000", 2024-05-15T13:29:51.239 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0617e-05 2024-05-15T13:29:51.239 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.239 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.239 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:51.239 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.794641+0000", 2024-05-15T13:29:51.239 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102415 2024-05-15T13:29:51.239 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.239 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.239 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,7", 2024-05-15T13:29:51.239 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.794737+0000", 2024-05-15T13:29:51.239 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6358000000000001e-05 2024-05-15T13:29:51.239 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.240 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.240 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:51.240 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.795050+0000", 2024-05-15T13:29:51.240 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031315899999999997 2024-05-15T13:29:51.240 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.240 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.240 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.240 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.804206+0000", 2024-05-15T13:29:51.240 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0091558939999999995 2024-05-15T13:29:51.240 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.240 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.240 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.240 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.992719+0000", 2024-05-15T13:29:51.240 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.188512665 2024-05-15T13:29:51.240 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.241 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.241 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:51.241 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.992758+0000", 2024-05-15T13:29:51.241 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9026999999999998e-05 2024-05-15T13:29:51.241 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.241 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.241 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:51.241 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.992785+0000", 2024-05-15T13:29:51.241 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7491000000000002e-05 2024-05-15T13:29:51.241 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.241 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:51.241 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.241 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.241 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.242 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:1558 9.7 9:e0427c0e:::benchmark_data_smithi177_159919_object1557:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-05-15T13:29:51.242 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:23:12.800668+0000", 2024-05-15T13:29:51.242 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 398.41803140799999, 2024-05-15T13:29:51.242 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19223637499999999, 2024-05-15T13:29:51.242 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:51.242 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:51.242 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:51.242 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:51.242 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:51.242 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1558 2024-05-15T13:29:51.242 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.242 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:51.242 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.243 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:51.243 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.800668+0000", 2024-05-15T13:29:51.243 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.243 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.243 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.243 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:51.243 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.800668+0000", 2024-05-15T13:29:51.243 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.243 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.243 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.243 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:51.243 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.800669+0000", 2024-05-15T13:29:51.243 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0839999999999999e-06 2024-05-15T13:29:51.243 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.243 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.244 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:51.244 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.800691+0000", 2024-05-15T13:29:51.244 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2306e-05 2024-05-15T13:29:51.244 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.244 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.244 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:51.244 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.800703+0000", 2024-05-15T13:29:51.244 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1787000000000001e-05 2024-05-15T13:29:51.244 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.244 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.244 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:51.244 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.800709+0000", 2024-05-15T13:29:51.244 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7610000000000004e-06 2024-05-15T13:29:51.244 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.245 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.245 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:51.245 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.800749+0000", 2024-05-15T13:29:51.245 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0513000000000003e-05 2024-05-15T13:29:51.245 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.245 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.245 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:51.245 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.800844+0000", 2024-05-15T13:29:51.245 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4419000000000005e-05 2024-05-15T13:29:51.245 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.245 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.245 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,7", 2024-05-15T13:29:51.245 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.800932+0000", 2024-05-15T13:29:51.245 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7782999999999996e-05 2024-05-15T13:29:51.246 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.246 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.246 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:51.246 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.801281+0000", 2024-05-15T13:29:51.246 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00034985000000000002 2024-05-15T13:29:51.246 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.246 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.246 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.246 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.804252+0000", 2024-05-15T13:29:51.246 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0029703170000000001 2024-05-15T13:29:51.246 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.246 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.246 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.247 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.992823+0000", 2024-05-15T13:29:51.247 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.188570871 2024-05-15T13:29:51.247 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.247 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.247 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:51.247 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.992867+0000", 2024-05-15T13:29:51.247 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4066000000000001e-05 2024-05-15T13:29:51.247 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.247 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.247 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:51.247 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.992904+0000", 2024-05-15T13:29:51.247 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7617999999999998e-05 2024-05-15T13:29:51.247 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.247 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:51.247 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.248 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.248 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.248 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:1589 9.7 9:e641b6b3:::benchmark_data_smithi177_159919_object1588:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-05-15T13:29:51.248 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:23:12.813212+0000", 2024-05-15T13:29:51.248 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 398.405487145, 2024-05-15T13:29:51.248 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.18011100399999999, 2024-05-15T13:29:51.248 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:51.248 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:51.248 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:51.248 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:51.248 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:51.248 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 1589 2024-05-15T13:29:51.248 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.248 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:51.248 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.249 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:51.249 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.813212+0000", 2024-05-15T13:29:51.249 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.249 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.249 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.249 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:51.249 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.813212+0000", 2024-05-15T13:29:51.249 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.249 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.249 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.249 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:51.249 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.813213+0000", 2024-05-15T13:29:51.249 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.161e-06 2024-05-15T13:29:51.249 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.249 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.250 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:51.250 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.813235+0000", 2024-05-15T13:29:51.250 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.198e-05 2024-05-15T13:29:51.250 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.250 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.250 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:51.250 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.813249+0000", 2024-05-15T13:29:51.250 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3916e-05 2024-05-15T13:29:51.250 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.250 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.250 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:51.250 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.813258+0000", 2024-05-15T13:29:51.250 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2579999999999995e-06 2024-05-15T13:29:51.251 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.251 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.251 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:51.251 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.813325+0000", 2024-05-15T13:29:51.251 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7528999999999997e-05 2024-05-15T13:29:51.251 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.251 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.251 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:51.251 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.813472+0000", 2024-05-15T13:29:51.251 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014731100000000001 2024-05-15T13:29:51.251 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.251 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.251 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,7", 2024-05-15T13:29:51.251 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.813654+0000", 2024-05-15T13:29:51.252 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018127 2024-05-15T13:29:51.252 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.252 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.252 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:51.252 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.814123+0000", 2024-05-15T13:29:51.252 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00046956 2024-05-15T13:29:51.252 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.252 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.252 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.252 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.814549+0000", 2024-05-15T13:29:51.252 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00042548899999999998 2024-05-15T13:29:51.252 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.252 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.252 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.252 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.993235+0000", 2024-05-15T13:29:51.253 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.178686284 2024-05-15T13:29:51.253 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.253 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.253 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:51.253 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.993288+0000", 2024-05-15T13:29:51.253 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3375000000000003e-05 2024-05-15T13:29:51.253 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.253 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.253 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:51.253 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:23:12.993323+0000", 2024-05-15T13:29:51.253 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4870999999999999e-05 2024-05-15T13:29:51.253 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.253 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:51.253 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.253 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.254 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.254 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:89435 9.7 9:e3219ff8:::benchmark_data_smithi177_159919_object89434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e840)", 2024-05-15T13:29:51.254 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:24:11.536487+0000", 2024-05-15T13:29:51.254 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 339.68221281400002, 2024-05-15T13:29:51.254 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.103357382, 2024-05-15T13:29:51.254 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:51.254 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:51.254 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:51.254 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:51.254 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:51.254 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 89435 2024-05-15T13:29:51.254 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.254 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:51.254 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.255 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:51.255 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.536487+0000", 2024-05-15T13:29:51.255 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.255 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.255 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.255 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:51.255 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.536487+0000", 2024-05-15T13:29:51.255 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.255 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.255 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.255 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:51.255 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.536488+0000", 2024-05-15T13:29:51.255 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3239999999999999e-06 2024-05-15T13:29:51.256 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.256 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.256 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:51.256 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.536514+0000", 2024-05-15T13:29:51.256 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6401e-05 2024-05-15T13:29:51.256 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.256 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.256 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:51.256 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.536527+0000", 2024-05-15T13:29:51.256 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2669e-05 2024-05-15T13:29:51.256 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.256 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.256 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:51.256 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.536534+0000", 2024-05-15T13:29:51.256 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2740000000000002e-06 2024-05-15T13:29:51.257 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.257 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.257 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:51.257 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.536587+0000", 2024-05-15T13:29:51.257 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2825999999999999e-05 2024-05-15T13:29:51.257 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.257 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.257 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:51.257 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.536722+0000", 2024-05-15T13:29:51.257 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013495299999999999 2024-05-15T13:29:51.257 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.257 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.257 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,7", 2024-05-15T13:29:51.257 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.536821+0000", 2024-05-15T13:29:51.258 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9448000000000001e-05 2024-05-15T13:29:51.258 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.258 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.258 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:51.258 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.540994+0000", 2024-05-15T13:29:51.258 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0041729610000000002 2024-05-15T13:29:51.258 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.258 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.258 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.258 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.541044+0000", 2024-05-15T13:29:51.258 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9784999999999999e-05 2024-05-15T13:29:51.258 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.258 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.259 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.259 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.639784+0000", 2024-05-15T13:29:51.259 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.098739387999999997 2024-05-15T13:29:51.259 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.259 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.259 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:51.259 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.639817+0000", 2024-05-15T13:29:51.259 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3633000000000001e-05 2024-05-15T13:29:51.259 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.259 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.259 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:51.259 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.639844+0000", 2024-05-15T13:29:51.259 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6720000000000002e-05 2024-05-15T13:29:51.259 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.259 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:51.260 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.260 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.260 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.260 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:89436 9.7 9:ee9f6f09:::benchmark_data_smithi177_159919_object89435:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e840)", 2024-05-15T13:29:51.260 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:24:11.536549+0000", 2024-05-15T13:29:51.260 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 339.68215039299997, 2024-05-15T13:29:51.260 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.10342407300000001, 2024-05-15T13:29:51.260 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:51.260 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:51.260 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:51.260 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:51.260 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:51.260 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 89436 2024-05-15T13:29:51.260 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.260 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:51.261 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.261 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:51.261 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.536549+0000", 2024-05-15T13:29:51.261 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.261 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.261 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.261 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:51.261 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.536549+0000", 2024-05-15T13:29:51.261 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.261 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.261 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.261 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:51.261 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.536550+0000", 2024-05-15T13:29:51.261 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7400000000000003e-07 2024-05-15T13:29:51.261 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.262 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.262 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:51.262 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.536573+0000", 2024-05-15T13:29:51.262 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3856999999999999e-05 2024-05-15T13:29:51.262 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.262 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.262 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:51.262 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.536581+0000", 2024-05-15T13:29:51.262 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5340000000000002e-06 2024-05-15T13:29:51.262 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.262 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.262 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:51.263 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.536588+0000", 2024-05-15T13:29:51.263 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8059999999999997e-06 2024-05-15T13:29:51.263 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.263 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.263 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:51.263 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.538970+0000", 2024-05-15T13:29:51.263 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0023825640000000002 2024-05-15T13:29:51.263 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.263 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.263 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:51.263 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.539124+0000", 2024-05-15T13:29:51.263 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015413499999999999 2024-05-15T13:29:51.263 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.263 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.264 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,7", 2024-05-15T13:29:51.264 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.539278+0000", 2024-05-15T13:29:51.264 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015335699999999999 2024-05-15T13:29:51.264 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.264 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.264 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.264 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.541079+0000", 2024-05-15T13:29:51.264 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0018009740000000001 2024-05-15T13:29:51.264 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.264 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.264 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:51.264 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.541113+0000", 2024-05-15T13:29:51.264 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4119999999999999e-05 2024-05-15T13:29:51.264 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.265 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.265 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.265 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.639891+0000", 2024-05-15T13:29:51.265 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.09877814 2024-05-15T13:29:51.265 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.265 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.265 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:51.265 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.639939+0000", 2024-05-15T13:29:51.265 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7479999999999999e-05 2024-05-15T13:29:51.265 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.265 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.265 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:51.265 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:24:11.639973+0000", 2024-05-15T13:29:51.265 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4532000000000001e-05 2024-05-15T13:29:51.265 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.266 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:51.266 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.266 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.266 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.266 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:155755 9.9 9:9e70e3fc:::benchmark_data_smithi177_159919_object35512:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e901)", 2024-05-15T13:29:51.266 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:26:32.558170+0000", 2024-05-15T13:29:51.266 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 198.660529279, 2024-05-15T13:29:51.266 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.092896834999999997, 2024-05-15T13:29:51.266 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:51.266 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:51.266 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:51.266 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:51.266 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:51.266 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 155755 2024-05-15T13:29:51.267 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.267 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:51.267 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.267 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:51.267 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:32.558170+0000", 2024-05-15T13:29:51.267 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.267 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.267 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.267 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:51.267 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:32.558170+0000", 2024-05-15T13:29:51.267 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.267 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.267 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.267 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:51.267 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:32.558172+0000", 2024-05-15T13:29:51.268 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5680000000000001e-06 2024-05-15T13:29:51.268 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.268 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.268 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:51.268 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:32.558178+0000", 2024-05-15T13:29:51.268 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.511e-06 2024-05-15T13:29:51.268 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.268 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.268 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:51.268 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:32.558193+0000", 2024-05-15T13:29:51.268 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4620000000000001e-05 2024-05-15T13:29:51.268 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.268 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.268 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:51.269 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:32.558199+0000", 2024-05-15T13:29:51.269 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9880000000000001e-06 2024-05-15T13:29:51.269 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.269 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.269 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:51.269 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:32.558259+0000", 2024-05-15T13:29:51.269 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0721999999999999e-05 2024-05-15T13:29:51.269 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.269 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.269 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:51.269 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:32.558393+0000", 2024-05-15T13:29:51.269 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013354299999999999 2024-05-15T13:29:51.269 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.269 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.269 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,11", 2024-05-15T13:29:51.270 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:32.558466+0000", 2024-05-15T13:29:51.270 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2910000000000005e-05 2024-05-15T13:29:51.270 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.270 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.270 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:51.270 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:32.563541+0000", 2024-05-15T13:29:51.270 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0050749360000000004 2024-05-15T13:29:51.270 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.270 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.270 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.271 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:32.567823+0000", 2024-05-15T13:29:51.271 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0042824830000000001 2024-05-15T13:29:51.271 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.271 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.271 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.271 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:32.650989+0000", 2024-05-15T13:29:51.271 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.083165779999999995 2024-05-15T13:29:51.271 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.271 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.271 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:51.271 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:32.651029+0000", 2024-05-15T13:29:51.271 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9601000000000003e-05 2024-05-15T13:29:51.271 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.272 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.272 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:51.272 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:32.651067+0000", 2024-05-15T13:29:51.272 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8173000000000002e-05 2024-05-15T13:29:51.272 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.272 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:51.272 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.272 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.272 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.272 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:181970 9.9 9:966c04b6:::benchmark_data_smithi177_159919_object61727:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e925)", 2024-05-15T13:29:51.272 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:27:32.557829+0000", 2024-05-15T13:29:51.272 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 138.660870012, 2024-05-15T13:29:51.272 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.162904415, 2024-05-15T13:29:51.272 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:51.273 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:51.273 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:51.273 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:51.273 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:51.273 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 181970 2024-05-15T13:29:51.273 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.273 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:51.273 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.273 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:51.273 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.557829+0000", 2024-05-15T13:29:51.273 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.273 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.273 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.273 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:51.273 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.557829+0000", 2024-05-15T13:29:51.274 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.274 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.274 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.274 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:51.274 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.557831+0000", 2024-05-15T13:29:51.274 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5340000000000001e-06 2024-05-15T13:29:51.274 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.274 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.274 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:51.274 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.557838+0000", 2024-05-15T13:29:51.274 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9419999999999998e-06 2024-05-15T13:29:51.274 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.274 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.275 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:51.275 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.557863+0000", 2024-05-15T13:29:51.275 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4694000000000001e-05 2024-05-15T13:29:51.275 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.275 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.275 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:51.275 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.557874+0000", 2024-05-15T13:29:51.275 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1222e-05 2024-05-15T13:29:51.275 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.275 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.275 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:51.275 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.557934+0000", 2024-05-15T13:29:51.275 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0631999999999998e-05 2024-05-15T13:29:51.275 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.275 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.275 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:51.276 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.558087+0000", 2024-05-15T13:29:51.276 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015286299999999999 2024-05-15T13:29:51.276 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.276 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.276 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,5,11", 2024-05-15T13:29:51.276 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.558241+0000", 2024-05-15T13:29:51.276 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015405 2024-05-15T13:29:51.276 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.276 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.276 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.276 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.563504+0000", 2024-05-15T13:29:51.276 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.005263032 2024-05-15T13:29:51.276 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.276 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.277 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:51.277 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.563600+0000", 2024-05-15T13:29:51.277 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5734000000000002e-05 2024-05-15T13:29:51.277 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.277 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.277 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.277 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.563948+0000", 2024-05-15T13:29:51.277 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000348031 2024-05-15T13:29:51.277 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.277 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.277 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.277 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.720548+0000", 2024-05-15T13:29:51.277 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.15659953700000001 2024-05-15T13:29:51.277 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.277 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.278 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:51.278 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.720613+0000", 2024-05-15T13:29:51.278 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5238000000000001e-05 2024-05-15T13:29:51.278 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.278 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.278 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:51.278 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.720734+0000", 2024-05-15T13:29:51.278 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120906 2024-05-15T13:29:51.278 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.278 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:51.278 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.278 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.278 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.279 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:181976 9.7 9:e816fe93:::benchmark_data_smithi177_159919_object61733:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e925)", 2024-05-15T13:29:51.279 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:27:32.589989+0000", 2024-05-15T13:29:51.279 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 138.62871071500001, 2024-05-15T13:29:51.279 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13017503499999999, 2024-05-15T13:29:51.279 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:51.279 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:51.279 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:51.279 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:51.279 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:51.279 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 181976 2024-05-15T13:29:51.279 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.279 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:51.280 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.280 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:51.280 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.589989+0000", 2024-05-15T13:29:51.280 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.280 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.280 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.280 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:51.280 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.589989+0000", 2024-05-15T13:29:51.280 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.280 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.280 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.280 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:51.280 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.589990+0000", 2024-05-15T13:29:51.280 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6589999999999999e-06 2024-05-15T13:29:51.281 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.281 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.281 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:51.281 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.589997+0000", 2024-05-15T13:29:51.281 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1280000000000002e-06 2024-05-15T13:29:51.281 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.281 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.281 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:51.281 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.590014+0000", 2024-05-15T13:29:51.281 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6113e-05 2024-05-15T13:29:51.281 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.281 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.281 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:51.281 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.590023+0000", 2024-05-15T13:29:51.281 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1980000000000007e-06 2024-05-15T13:29:51.282 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.282 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.282 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:51.282 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.590080+0000", 2024-05-15T13:29:51.282 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7221e-05 2024-05-15T13:29:51.282 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.282 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.282 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:51.282 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.590171+0000", 2024-05-15T13:29:51.282 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1254000000000004e-05 2024-05-15T13:29:51.282 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.282 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.282 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,7", 2024-05-15T13:29:51.282 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.590249+0000", 2024-05-15T13:29:51.283 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8097999999999997e-05 2024-05-15T13:29:51.283 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.283 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.283 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:51.283 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.594729+0000", 2024-05-15T13:29:51.283 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0044792089999999996 2024-05-15T13:29:51.283 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.283 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.283 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.283 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.596881+0000", 2024-05-15T13:29:51.284 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002151993 2024-05-15T13:29:51.284 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.284 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.284 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.284 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.720082+0000", 2024-05-15T13:29:51.284 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.123201332 2024-05-15T13:29:51.284 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.284 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.284 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:51.284 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.720122+0000", 2024-05-15T13:29:51.284 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9650999999999997e-05 2024-05-15T13:29:51.284 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.284 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.284 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:51.284 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.720164+0000", 2024-05-15T13:29:51.285 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2178999999999997e-05 2024-05-15T13:29:51.285 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.285 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:51.285 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.285 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.285 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.285 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:181978 9.7 9:eb4b7035:::benchmark_data_smithi177_159919_object61735:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e925)", 2024-05-15T13:29:51.285 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:27:32.613714+0000", 2024-05-15T13:29:51.285 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 138.60498493399999, 2024-05-15T13:29:51.285 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.112145536, 2024-05-15T13:29:51.285 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:51.285 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:51.286 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:51.286 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:51.286 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:51.286 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 181978 2024-05-15T13:29:51.286 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.286 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:51.287 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.287 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:51.287 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.613714+0000", 2024-05-15T13:29:51.287 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.287 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.287 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.287 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:51.288 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.613714+0000", 2024-05-15T13:29:51.288 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.288 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.288 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.288 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:51.288 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.613716+0000", 2024-05-15T13:29:51.288 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5600000000000001e-06 2024-05-15T13:29:51.288 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.289 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.289 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:51.289 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.613725+0000", 2024-05-15T13:29:51.289 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8440000000000004e-06 2024-05-15T13:29:51.289 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.289 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.289 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:51.290 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.613747+0000", 2024-05-15T13:29:51.290 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1878e-05 2024-05-15T13:29:51.290 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.290 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.290 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:51.290 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.613756+0000", 2024-05-15T13:29:51.291 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8519999999999993e-06 2024-05-15T13:29:51.291 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.291 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.291 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:51.291 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.613815+0000", 2024-05-15T13:29:51.291 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9125000000000001e-05 2024-05-15T13:29:51.291 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.292 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.292 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:51.292 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.613952+0000", 2024-05-15T13:29:51.292 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013726999999999999 2024-05-15T13:29:51.292 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.292 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.293 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 3,7", 2024-05-15T13:29:51.293 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.614076+0000", 2024-05-15T13:29:51.293 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012430399999999999 2024-05-15T13:29:51.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.293 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.293 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.293 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.621917+0000", 2024-05-15T13:29:51.293 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0078403819999999999 2024-05-15T13:29:51.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.293 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.293 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:51.293 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.623472+0000", 2024-05-15T13:29:51.293 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015557909999999999 2024-05-15T13:29:51.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.293 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.294 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.294 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.725754+0000", 2024-05-15T13:29:51.294 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.102281448 2024-05-15T13:29:51.294 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.294 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.294 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:51.294 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.725811+0000", 2024-05-15T13:29:51.294 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6932000000000003e-05 2024-05-15T13:29:51.294 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.294 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.295 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:51.295 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:32.725860+0000", 2024-05-15T13:29:51.295 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9150000000000002e-05 2024-05-15T13:29:51.295 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.295 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:51.295 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.295 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.295 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.295 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:193556 9.9 9:9f498655:::benchmark_data_smithi177_159919_object73313:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e936)", 2024-05-15T13:29:51.295 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:27:57.416281+0000", 2024-05-15T13:29:51.295 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 113.802417948, 2024-05-15T13:29:51.295 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0616089450000001, 2024-05-15T13:29:51.295 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:51.295 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:51.295 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:51.296 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:51.296 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:51.296 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 193556 2024-05-15T13:29:51.296 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.296 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:51.296 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.296 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:51.296 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:57.416281+0000", 2024-05-15T13:29:51.296 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.296 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.296 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.296 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:51.296 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:57.416281+0000", 2024-05-15T13:29:51.297 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.297 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.297 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.297 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:51.297 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:57.416283+0000", 2024-05-15T13:29:51.297 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.296e-06 2024-05-15T13:29:51.297 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.297 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.297 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:51.297 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:57.416288+0000", 2024-05-15T13:29:51.297 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6740000000000002e-06 2024-05-15T13:29:51.297 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.297 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.297 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:51.298 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:57.416305+0000", 2024-05-15T13:29:51.298 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6290000000000002e-05 2024-05-15T13:29:51.298 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.298 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.298 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:51.298 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:57.416313+0000", 2024-05-15T13:29:51.298 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4880000000000007e-06 2024-05-15T13:29:51.298 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.298 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.298 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:51.298 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:57.416366+0000", 2024-05-15T13:29:51.298 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3143000000000003e-05 2024-05-15T13:29:51.299 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.299 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.299 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:29:51.299 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:57.416376+0000", 2024-05-15T13:29:51.299 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2930000000000006e-06 2024-05-15T13:29:51.299 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.299 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.300 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:51.300 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.418130+0000", 2024-05-15T13:29:51.300 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0017548169999999 2024-05-15T13:29:51.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.300 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.300 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:29:51.300 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.418140+0000", 2024-05-15T13:29:51.300 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0079e-05 2024-05-15T13:29:51.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.300 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.300 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:51.300 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.457891+0000", 2024-05-15T13:29:51.300 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.039750578 2024-05-15T13:29:51.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.300 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.301 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:51.301 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.458025+0000", 2024-05-15T13:29:51.301 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000133685 2024-05-15T13:29:51.301 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.301 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.301 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,11", 2024-05-15T13:29:51.301 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.458154+0000", 2024-05-15T13:29:51.301 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012931299999999999 2024-05-15T13:29:51.301 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.301 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.301 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:51.301 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.463832+0000", 2024-05-15T13:29:51.301 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0056779939999999996 2024-05-15T13:29:51.301 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.301 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.302 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.302 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.468946+0000", 2024-05-15T13:29:51.302 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.005114164 2024-05-15T13:29:51.302 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.302 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.302 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.302 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.477743+0000", 2024-05-15T13:29:51.302 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0087971869999999997 2024-05-15T13:29:51.302 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.302 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.302 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:51.302 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.477839+0000", 2024-05-15T13:29:51.302 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5230999999999997e-05 2024-05-15T13:29:51.303 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.303 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.303 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:51.303 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.477890+0000", 2024-05-15T13:29:51.303 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1712999999999998e-05 2024-05-15T13:29:51.303 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.303 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:51.303 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.303 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.303 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.303 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:224738 9.9 9:980bdbf2:::benchmark_data_smithi177_159919_object104495:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e967)", 2024-05-15T13:29:51.303 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:54.786847+0000", 2024-05-15T13:29:51.303 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 56.431852290999998, 2024-05-15T13:29:51.303 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13367820999999999, 2024-05-15T13:29:51.303 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:51.304 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:51.304 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:51.304 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:51.304 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:51.304 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 224738 2024-05-15T13:29:51.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.304 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:51.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:51.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.786847+0000", 2024-05-15T13:29:51.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:51.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.786847+0000", 2024-05-15T13:29:51.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.305 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:51.305 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.786849+0000", 2024-05-15T13:29:51.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2400000000000002e-06 2024-05-15T13:29:51.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.305 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:51.305 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.786856+0000", 2024-05-15T13:29:51.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.613e-06 2024-05-15T13:29:51.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.306 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:51.306 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.786873+0000", 2024-05-15T13:29:51.306 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6860000000000001e-05 2024-05-15T13:29:51.306 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.306 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.306 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:51.306 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.786882+0000", 2024-05-15T13:29:51.306 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3340000000000008e-06 2024-05-15T13:29:51.306 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.306 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.306 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:51.306 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.786960+0000", 2024-05-15T13:29:51.306 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7980999999999998e-05 2024-05-15T13:29:51.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.307 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:51.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.787099+0000", 2024-05-15T13:29:51.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013859699999999999 2024-05-15T13:29:51.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.307 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,11", 2024-05-15T13:29:51.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.787219+0000", 2024-05-15T13:29:51.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012048999999999999 2024-05-15T13:29:51.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.307 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.796092+0000", 2024-05-15T13:29:51.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0088723729999999994 2024-05-15T13:29:51.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:51.308 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.797197+0000", 2024-05-15T13:29:51.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001105808 2024-05-15T13:29:51.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.308 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.920421+0000", 2024-05-15T13:29:51.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.123223285 2024-05-15T13:29:51.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:51.308 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.920475+0000", 2024-05-15T13:29:51.309 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4659e-05 2024-05-15T13:29:51.309 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.309 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.309 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:51.309 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.920525+0000", 2024-05-15T13:29:51.309 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9969999999999998e-05 2024-05-15T13:29:51.309 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.309 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:51.309 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.309 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.309 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.309 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:224740 9.9 9:914d9f9b:::benchmark_data_smithi177_159919_object104497:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e967)", 2024-05-15T13:29:51.309 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:54.787063+0000", 2024-05-15T13:29:51.309 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 56.431636183000002, 2024-05-15T13:29:51.310 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.13391133999999999, 2024-05-15T13:29:51.310 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:51.310 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:51.310 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:51.310 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:51.310 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:51.310 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 224740 2024-05-15T13:29:51.310 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.310 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:51.310 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.310 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:51.310 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.787063+0000", 2024-05-15T13:29:51.310 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:51.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.787063+0000", 2024-05-15T13:29:51.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:51.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.787064+0000", 2024-05-15T13:29:51.311 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1999999999999999e-07 2024-05-15T13:29:51.311 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.311 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.311 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:51.311 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.787068+0000", 2024-05-15T13:29:51.312 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4349999999999999e-06 2024-05-15T13:29:51.312 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.312 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.312 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:51.312 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.787077+0000", 2024-05-15T13:29:51.312 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1419999999999999e-06 2024-05-15T13:29:51.312 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.312 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.312 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:51.312 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.787084+0000", 2024-05-15T13:29:51.312 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5180000000000001e-06 2024-05-15T13:29:51.312 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.312 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.312 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:51.312 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.791936+0000", 2024-05-15T13:29:51.313 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0048524579999999996 2024-05-15T13:29:51.313 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.313 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.313 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:51.313 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.792018+0000", 2024-05-15T13:29:51.313 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2033999999999994e-05 2024-05-15T13:29:51.313 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.313 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.313 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,11", 2024-05-15T13:29:51.313 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.792107+0000", 2024-05-15T13:29:51.313 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8735999999999999e-05 2024-05-15T13:29:51.313 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.313 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.313 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:51.314 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.797217+0000", 2024-05-15T13:29:51.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0051095539999999997 2024-05-15T13:29:51.314 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.314 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.314 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.314 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.797480+0000", 2024-05-15T13:29:51.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026294000000000003 2024-05-15T13:29:51.314 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.314 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.314 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:51.314 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.920889+0000", 2024-05-15T13:29:51.314 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.1234089 2024-05-15T13:29:51.314 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.314 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.315 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:51.315 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.920937+0000", 2024-05-15T13:29:51.315 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8241000000000001e-05 2024-05-15T13:29:51.315 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.315 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.315 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:51.315 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:54.920975+0000", 2024-05-15T13:29:51.315 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7662e-05 2024-05-15T13:29:51.315 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.315 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:51.315 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.315 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.315 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.316 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDScrubReserve(1.10,QREQUEST,e:987,from:5,reservation_nonce:11)", 2024-05-15T13:29:51.316 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:29:49.762855+0000", 2024-05-15T13:29:51.316 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.455844599, 2024-05-15T13:29:51.316 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018812399999999999, 2024-05-15T13:29:51.316 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:51.316 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T13:29:51.316 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:51.316 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.316 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:51.316 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.762855+0000", 2024-05-15T13:29:51.316 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.316 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.316 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.316 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:51.316 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.762855+0000", 2024-05-15T13:29:51.317 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.317 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.317 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.317 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:51.317 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.762856+0000", 2024-05-15T13:29:51.317 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.73e-06 2024-05-15T13:29:51.317 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.317 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.317 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:51.317 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.762867+0000", 2024-05-15T13:29:51.317 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0686999999999999e-05 2024-05-15T13:29:51.317 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.317 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.317 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:51.318 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.762894+0000", 2024-05-15T13:29:51.318 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7155e-05 2024-05-15T13:29:51.318 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.318 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.318 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:51.318 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.762909+0000", 2024-05-15T13:29:51.318 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.417e-05 2024-05-15T13:29:51.318 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.318 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.318 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:51.319 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.762997+0000", 2024-05-15T13:29:51.319 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8596000000000004e-05 2024-05-15T13:29:51.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.319 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.319 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:51.319 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.763011+0000", 2024-05-15T13:29:51.319 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4002000000000001e-05 2024-05-15T13:29:51.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.319 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.319 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:51.319 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.763043+0000", 2024-05-15T13:29:51.319 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1783999999999999e-05 2024-05-15T13:29:51.319 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.319 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:51.320 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.320 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.320 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.320 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "replica_scrub(pg: 1.10,from:0'0,to:0'0,epoch:987/761,start:1:08000000::::0,end:MAX,chunky:1,deep:1,version:9,allow_preemption:1,priority=5)", 2024-05-15T13:29:51.320 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:29:49.763667+0000", 2024-05-15T13:29:51.320 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.4550325660000001, 2024-05-15T13:29:51.320 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114882, 2024-05-15T13:29:51.320 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:51.320 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T13:29:51.320 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:51.320 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.320 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:51.320 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.763667+0000", 2024-05-15T13:29:51.320 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.320 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.321 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.321 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:51.321 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.763667+0000", 2024-05-15T13:29:51.321 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.321 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.321 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.321 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:51.321 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.763667+0000", 2024-05-15T13:29:51.321 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.46e-07 2024-05-15T13:29:51.321 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.321 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.321 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:51.321 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.763671+0000", 2024-05-15T13:29:51.321 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9090000000000002e-06 2024-05-15T13:29:51.321 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.322 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.322 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:51.322 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.763679+0000", 2024-05-15T13:29:51.322 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0129999999999993e-06 2024-05-15T13:29:51.322 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.322 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.322 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:51.322 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.763685+0000", 2024-05-15T13:29:51.322 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7540000000000003e-06 2024-05-15T13:29:51.325 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.325 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.325 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:51.326 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.763754+0000", 2024-05-15T13:29:51.326 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9682000000000001e-05 2024-05-15T13:29:51.326 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.326 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.326 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:51.327 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.763763+0000", 2024-05-15T13:29:51.327 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8869999999999998e-06 2024-05-15T13:29:51.327 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.327 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.327 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:51.327 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.763782+0000", 2024-05-15T13:29:51.327 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8291000000000002e-05 2024-05-15T13:29:51.327 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.327 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:51.327 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.327 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.327 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.327 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDScrubReserve(1.10,RELEASE,e:987,from:5,reservation_nonce:0)", 2024-05-15T13:29:51.328 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:29:49.796115+0000", 2024-05-15T13:29:51.328 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 1.422584472, 2024-05-15T13:29:51.328 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000163604, 2024-05-15T13:29:51.328 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:51.328 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T13:29:51.328 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:51.328 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.328 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:51.328 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.796115+0000", 2024-05-15T13:29:51.328 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.328 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.329 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:51.329 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.796115+0000", 2024-05-15T13:29:51.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:51.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.329 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:51.329 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.796115+0000", 2024-05-15T13:29:51.329 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7100000000000001e-07 2024-05-15T13:29:51.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.329 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.329 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:51.329 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.796135+0000", 2024-05-15T13:29:51.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9459000000000001e-05 2024-05-15T13:29:51.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:51.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.796139+0000", 2024-05-15T13:29:51.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8790000000000002e-06 2024-05-15T13:29:51.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:51.330 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.796148+0000", 2024-05-15T13:29:51.330 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3210000000000005e-06 2024-05-15T13:29:51.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.330 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.330 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:51.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.796242+0000", 2024-05-15T13:29:51.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4367e-05 2024-05-15T13:29:51.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:51.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.796258+0000", 2024-05-15T13:29:51.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5914e-05 2024-05-15T13:29:51.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:51.331 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:51.331 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:51.331 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.796278+0000", 2024-05-15T13:29:51.331 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0492999999999998e-05 2024-05-15T13:29:51.331 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.331 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:51.332 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.332 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:51.332 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:51.332 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:29:51.332 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:51 smithi177 ceph-mon[44303]: 1.10 deep-scrub starts 2024-05-15T13:29:51.333 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:51 smithi177 ceph-mon[44303]: 1.10 deep-scrub ok 2024-05-15T13:29:51.333 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:51 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2434167424' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T13:29:51.333 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:51 smithi177 ceph-mon[44303]: osdmap e988: 12 total, 12 up, 11 in 2024-05-15T13:29:51.357 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:29:51.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:51 smithi027 ceph-mon[39795]: 1.10 deep-scrub starts 2024-05-15T13:29:51.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:51 smithi027 ceph-mon[39795]: 1.10 deep-scrub ok 2024-05-15T13:29:51.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:51 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2434167424' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T13:29:51.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:51 smithi027 ceph-mon[39795]: osdmap e988: 12 total, 12 up, 11 in 2024-05-15T13:29:51.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:51 smithi120 ceph-mon[43961]: 1.10 deep-scrub starts 2024-05-15T13:29:51.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:51 smithi120 ceph-mon[43961]: 1.10 deep-scrub ok 2024-05-15T13:29:51.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:51 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2434167424' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T13:29:51.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:51 smithi120 ceph-mon[43961]: osdmap e988: 12 total, 12 up, 11 in 2024-05-15T13:29:51.898 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-05-15T13:29:52.162 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:29:52.420 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:52 smithi177 ceph-mon[44303]: pgmap v2493: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:52.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:52 smithi027 ceph-mon[39795]: pgmap v2493: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:52.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:52 smithi120 ceph-mon[43961]: pgmap v2493: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:54.005 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:29:54.005 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T13:29:54.005 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-05-15T13:29:54.005 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:29:54.155 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.11 to 1 (10000) 2024-05-15T13:29:54.335 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:54 smithi177 ceph-mon[44303]: pgmap v2494: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:54.335 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:54 smithi177 ceph-mon[44303]: 1.14 scrub starts 2024-05-15T13:29:54.335 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:54 smithi177 ceph-mon[44303]: 1.14 scrub ok 2024-05-15T13:29:54.335 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:54 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3334457608' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T13:29:54.498 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:54 smithi027 ceph-mon[39795]: pgmap v2494: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:54.498 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:54 smithi027 ceph-mon[39795]: 1.14 scrub starts 2024-05-15T13:29:54.498 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:54 smithi027 ceph-mon[39795]: 1.14 scrub ok 2024-05-15T13:29:54.498 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:54 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3334457608' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T13:29:54.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:54 smithi120 ceph-mon[43961]: pgmap v2494: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:54.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:54 smithi120 ceph-mon[43961]: 1.14 scrub starts 2024-05-15T13:29:54.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:54 smithi120 ceph-mon[43961]: 1.14 scrub ok 2024-05-15T13:29:54.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:54 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3334457608' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T13:29:54.750 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-05-15T13:29:54.797 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-05-15T13:29:54.798 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:29:55.011 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:29:55.073 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:29:55.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:55 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3334457608' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T13:29:55.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:55 smithi177 ceph-mon[44303]: osdmap e989: 12 total, 12 up, 11 in 2024-05-15T13:29:55.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:55 smithi177 ceph-mon[44303]: 1.e scrub starts 2024-05-15T13:29:55.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:55 smithi177 ceph-mon[44303]: 1.e scrub ok 2024-05-15T13:29:55.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:55 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3334457608' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T13:29:55.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:55 smithi027 ceph-mon[39795]: osdmap e989: 12 total, 12 up, 11 in 2024-05-15T13:29:55.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:55 smithi027 ceph-mon[39795]: 1.e scrub starts 2024-05-15T13:29:55.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:55 smithi027 ceph-mon[39795]: 1.e scrub ok 2024-05-15T13:29:55.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:55 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3334457608' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T13:29:55.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:55 smithi120 ceph-mon[43961]: osdmap e989: 12 total, 12 up, 11 in 2024-05-15T13:29:55.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:55 smithi120 ceph-mon[43961]: 1.e scrub starts 2024-05-15T13:29:55.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:55 smithi120 ceph-mon[43961]: 1.e scrub ok 2024-05-15T13:29:56.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:56 smithi027 ceph-mon[39795]: pgmap v2496: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:56.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:56 smithi177 ceph-mon[44303]: pgmap v2496: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:56.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:56 smithi120 ceph-mon[43961]: pgmap v2496: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:56.843 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:29:56.843 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-05-15T13:29:56.843 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-05-15T13:29:56.843 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-05-15T13:29:56.843 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:29:57.020 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:29:57.022 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:29:57.643 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-05-15T13:29:57.644 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2497,"stamp":"2024-05-15T13:29:56.795300+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":9173424,"kb_used_data":25840,"kb_used_omap":17,"kb_used_meta":9147438,"kb_avail":1022023248,"statfs":{"total":1055945392128,"available":1046551805952,"internally_reserved":0,"allocated":26460160,"data_stored":17488851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17546,"internal_metadata":9366977398},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-3,"num_read_kb":-3,"num_write":-4,"num_write_kb":-67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":-84148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001740"},"pg_stats":[{"pgid":"1.1e","version":"94'139","reported_seq":1136,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-05-15T13:29:50.381332+0000","last_change":"2024-05-15T13:29:11.559382+0000","last_active":"2024-05-15T13:29:50.381332+0000","last_peered":"2024-05-15T13:29:50.381332+0000","last_clean":"2024-05-15T13:29:50.381332+0000","last_became_active":"2024-05-15T13:28:01.363009+0000","last_became_peered":"2024-05-15T13:28:01.363009+0000","last_unstale":"2024-05-15T13:29:50.381332+0000","last_undegraded":"2024-05-15T13:29:50.381332+0000","last_fullsized":"2024-05-15T13:29:50.381332+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:11.559293+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:29:11.559293+0000","last_clean_scrub_stamp":"2024-05-15T13:29:11.559293+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:18.127074+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"94'139","reported_seq":1031,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-05-15T13:29:50.381333+0000","last_change":"2024-05-15T13:29:47.555278+0000","last_active":"2024-05-15T13:29:50.381333+0000","last_peered":"2024-05-15T13:29:50.381333+0000","last_clean":"2024-05-15T13:29:50.381333+0000","last_became_active":"2024-05-15T13:28:01.353254+0000","last_became_peered":"2024-05-15T13:28:01.353254+0000","last_unstale":"2024-05-15T13:29:50.381333+0000","last_undegraded":"2024-05-15T13:29:50.381333+0000","last_fullsized":"2024-05-15T13:29:50.381333+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:47.555173+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:23:03.856199+0000","last_clean_scrub_stamp":"2024-05-15T13:29:47.555173+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:49.032312+0000","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"94'139","reported_seq":896,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.259117+0000","last_change":"2024-05-15T13:29:44.708851+0000","last_active":"2024-05-15T13:29:54.259117+0000","last_peered":"2024-05-15T13:29:54.259117+0000","last_clean":"2024-05-15T13:29:54.259117+0000","last_became_active":"2024-05-15T13:28:01.366096+0000","last_became_peered":"2024-05-15T13:28:01.366096+0000","last_unstale":"2024-05-15T13:29:54.259117+0000","last_undegraded":"2024-05-15T13:29:54.259117+0000","last_fullsized":"2024-05-15T13:29:54.259117+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:44.708741+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:29:44.708741+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:51.133096+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,11],"acting":[8,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"94'139","reported_seq":1297,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.258829+0000","last_change":"2024-05-15T13:29:46.810878+0000","last_active":"2024-05-15T13:29:54.258829+0000","last_peered":"2024-05-15T13:29:54.258829+0000","last_clean":"2024-05-15T13:29:54.258829+0000","last_became_active":"2024-05-15T13:27:32.380219+0000","last_became_peered":"2024-05-15T13:27:32.380219+0000","last_unstale":"2024-05-15T13:29:54.258829+0000","last_undegraded":"2024-05-15T13:29:54.258829+0000","last_fullsized":"2024-05-15T13:29:54.258829+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:46.810775+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:29:46.810775+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:54.360083+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"94'139","reported_seq":1337,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-05-15T13:29:51.554579+0000","last_change":"2024-05-15T13:29:49.560177+0000","last_active":"2024-05-15T13:29:51.554579+0000","last_peered":"2024-05-15T13:29:51.554579+0000","last_clean":"2024-05-15T13:29:51.554579+0000","last_became_active":"2024-05-15T13:28:01.357553+0000","last_became_peered":"2024-05-15T13:28:01.357553+0000","last_unstale":"2024-05-15T13:29:51.554579+0000","last_undegraded":"2024-05-15T13:29:51.554579+0000","last_fullsized":"2024-05-15T13:29:51.554579+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:49.560066+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:22:03.418345+0000","last_clean_scrub_stamp":"2024-05-15T13:29:49.560066+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:31:18.793201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,11],"acting":[4,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"94'139","reported_seq":1122,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-05-15T13:29:50.439142+0000","last_change":"2024-05-15T13:29:45.745089+0000","last_active":"2024-05-15T13:29:50.439142+0000","last_peered":"2024-05-15T13:29:50.439142+0000","last_clean":"2024-05-15T13:29:50.439142+0000","last_became_active":"2024-05-15T13:24:11.374739+0000","last_became_peered":"2024-05-15T13:24:11.374739+0000","last_unstale":"2024-05-15T13:29:50.439142+0000","last_undegraded":"2024-05-15T13:29:50.439142+0000","last_fullsized":"2024-05-15T13:29:50.439142+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:45.744986+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:29:45.744986+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:31:00.748332+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"94'139","reported_seq":1946,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.151708+0000","last_change":"2024-05-15T13:29:46.768771+0000","last_active":"2024-05-15T13:29:54.151708+0000","last_peered":"2024-05-15T13:29:54.151708+0000","last_clean":"2024-05-15T13:29:54.151708+0000","last_became_active":"2024-05-15T13:24:11.368347+0000","last_became_peered":"2024-05-15T13:24:11.368347+0000","last_unstale":"2024-05-15T13:29:54.151708+0000","last_undegraded":"2024-05-15T13:29:54.151708+0000","last_fullsized":"2024-05-15T13:29:54.151708+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:46.768629+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:29:46.768629+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:31:04.930151+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"94'139","reported_seq":680,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.258734+0000","last_change":"2024-05-15T13:29:47.808074+0000","last_active":"2024-05-15T13:29:54.258734+0000","last_peered":"2024-05-15T13:29:54.258734+0000","last_clean":"2024-05-15T13:29:54.258734+0000","last_became_active":"2024-05-15T13:27:32.410771+0000","last_became_peered":"2024-05-15T13:27:32.410771+0000","last_unstale":"2024-05-15T13:29:54.258734+0000","last_undegraded":"2024-05-15T13:29:54.258734+0000","last_fullsized":"2024-05-15T13:29:54.258734+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:47.807948+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:29:47.807948+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:31:01.998503+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"94'139","reported_seq":1874,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.259203+0000","last_change":"2024-05-15T13:29:48.659515+0000","last_active":"2024-05-15T13:29:54.259203+0000","last_peered":"2024-05-15T13:29:54.259203+0000","last_clean":"2024-05-15T13:29:54.259203+0000","last_became_active":"2024-05-15T13:28:01.356182+0000","last_became_peered":"2024-05-15T13:28:01.356182+0000","last_unstale":"2024-05-15T13:29:54.259203+0000","last_undegraded":"2024-05-15T13:29:54.259203+0000","last_fullsized":"2024-05-15T13:29:54.259203+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:48.659406+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:29:48.659406+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:55.624977+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,11],"acting":[0,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"94'139","reported_seq":875,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-05-15T13:29:50.943186+0000","last_change":"2024-05-15T13:29:01.707292+0000","last_active":"2024-05-15T13:29:50.943186+0000","last_peered":"2024-05-15T13:29:50.943186+0000","last_clean":"2024-05-15T13:29:50.943186+0000","last_became_active":"2024-05-15T13:27:32.397907+0000","last_became_peered":"2024-05-15T13:27:32.397907+0000","last_unstale":"2024-05-15T13:29:50.943186+0000","last_undegraded":"2024-05-15T13:29:50.943186+0000","last_fullsized":"2024-05-15T13:29:50.943186+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:01.707172+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:18:24.970285+0000","last_clean_scrub_stamp":"2024-05-15T13:29:01.707172+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:27.496391+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"94'139","reported_seq":1382,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-05-15T13:29:52.099093+0000","last_change":"2024-05-15T13:28:16.746468+0000","last_active":"2024-05-15T13:29:52.099093+0000","last_peered":"2024-05-15T13:29:52.099093+0000","last_clean":"2024-05-15T13:29:52.099093+0000","last_became_active":"2024-05-15T13:19:41.190796+0000","last_became_peered":"2024-05-15T13:19:41.190796+0000","last_unstale":"2024-05-15T13:29:52.099093+0000","last_undegraded":"2024-05-15T13:29:52.099093+0000","last_fullsized":"2024-05-15T13:29:52.099093+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:28:16.746362+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:23:11.573492+0000","last_clean_scrub_stamp":"2024-05-15T13:28:16.746362+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":1946,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-05-15T13:29:50.942945+0000","last_change":"2024-05-15T13:28:48.463286+0000","last_active":"2024-05-15T13:29:50.942945+0000","last_peered":"2024-05-15T13:29:50.942945+0000","last_clean":"2024-05-15T13:29:50.942945+0000","last_became_active":"2024-05-15T13:28:01.370095+0000","last_became_peered":"2024-05-15T13:28:01.370095+0000","last_unstale":"2024-05-15T13:29:50.942945+0000","last_undegraded":"2024-05-15T13:29:50.942945+0000","last_fullsized":"2024-05-15T13:29:50.942945+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:28:48.463167+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:28:48.463167+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:29:59.226265+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"94'139","reported_seq":1657,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-05-15T13:29:50.381285+0000","last_change":"2024-05-15T13:29:13.552444+0000","last_active":"2024-05-15T13:29:50.381285+0000","last_peered":"2024-05-15T13:29:50.381285+0000","last_clean":"2024-05-15T13:29:50.381285+0000","last_became_active":"2024-05-15T13:28:01.357970+0000","last_became_peered":"2024-05-15T13:28:01.357970+0000","last_unstale":"2024-05-15T13:29:50.381285+0000","last_undegraded":"2024-05-15T13:29:50.381285+0000","last_fullsized":"2024-05-15T13:29:50.381285+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:13.552305+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:24:35.178049+0000","last_clean_scrub_stamp":"2024-05-15T13:29:13.552305+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:21.901694+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"94'139","reported_seq":1690,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-05-15T13:29:50.439154+0000","last_change":"2024-05-15T13:29:43.815139+0000","last_active":"2024-05-15T13:29:50.439154+0000","last_peered":"2024-05-15T13:29:50.439154+0000","last_clean":"2024-05-15T13:29:50.439154+0000","last_became_active":"2024-05-15T13:24:06.307598+0000","last_became_peered":"2024-05-15T13:24:06.307598+0000","last_unstale":"2024-05-15T13:29:50.439154+0000","last_undegraded":"2024-05-15T13:29:50.439154+0000","last_fullsized":"2024-05-15T13:29:50.439154+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:43.815033+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:18:24.970285+0000","last_clean_scrub_stamp":"2024-05-15T13:29:43.815033+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:31:01.907870+0000","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"94'139","reported_seq":1954,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-05-15T13:29:52.099192+0000","last_change":"2024-05-15T13:29:01.128994+0000","last_active":"2024-05-15T13:29:52.099192+0000","last_peered":"2024-05-15T13:29:52.099192+0000","last_clean":"2024-05-15T13:29:52.099192+0000","last_became_active":"2024-05-15T13:27:32.378350+0000","last_became_peered":"2024-05-15T13:27:32.378350+0000","last_unstale":"2024-05-15T13:29:52.099192+0000","last_undegraded":"2024-05-15T13:29:52.099192+0000","last_fullsized":"2024-05-15T13:29:52.099192+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:01.128890+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:29:01.128890+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:13.820028+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,8],"acting":[2,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"94'139","reported_seq":1705,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.195662+0000","last_change":"2024-05-15T13:28:53.019558+0000","last_active":"2024-05-15T13:29:54.195662+0000","last_peered":"2024-05-15T13:29:54.195662+0000","last_clean":"2024-05-15T13:29:54.195662+0000","last_became_active":"2024-05-15T13:24:06.312706+0000","last_became_peered":"2024-05-15T13:24:06.312706+0000","last_unstale":"2024-05-15T13:29:54.195662+0000","last_undegraded":"2024-05-15T13:29:54.195662+0000","last_fullsized":"2024-05-15T13:29:54.195662+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:28:53.019440+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:28:53.019440+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:14.627453+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"94'139","reported_seq":1705,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.258865+0000","last_change":"2024-05-15T13:29:44.140963+0000","last_active":"2024-05-15T13:29:54.258865+0000","last_peered":"2024-05-15T13:29:54.258865+0000","last_clean":"2024-05-15T13:29:54.258865+0000","last_became_active":"2024-05-15T13:15:43.614970+0000","last_became_peered":"2024-05-15T13:15:43.614970+0000","last_unstale":"2024-05-15T13:29:54.258865+0000","last_undegraded":"2024-05-15T13:29:54.258865+0000","last_fullsized":"2024-05-15T13:29:54.258865+0000","mapping_epoch":678,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:44.140861+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:29:44.140861+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:31:05.284727+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,0],"acting":[9,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.2","version":"94'139","reported_seq":1965,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-05-15T13:29:50.943099+0000","last_change":"2024-05-15T13:29:03.616520+0000","last_active":"2024-05-15T13:29:50.943099+0000","last_peered":"2024-05-15T13:29:50.943099+0000","last_clean":"2024-05-15T13:29:50.943099+0000","last_became_active":"2024-05-15T13:25:58.961079+0000","last_became_peered":"2024-05-15T13:25:58.961079+0000","last_unstale":"2024-05-15T13:29:50.943099+0000","last_undegraded":"2024-05-15T13:29:50.943099+0000","last_fullsized":"2024-05-15T13:29:50.943099+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:03.616403+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:29:03.616403+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:08.670665+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"94'139","reported_seq":1524,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.195576+0000","last_change":"2024-05-15T13:29:12.170940+0000","last_active":"2024-05-15T13:29:54.195576+0000","last_peered":"2024-05-15T13:29:54.195576+0000","last_clean":"2024-05-15T13:29:54.195576+0000","last_became_active":"2024-05-15T13:27:32.402442+0000","last_became_peered":"2024-05-15T13:27:32.402442+0000","last_unstale":"2024-05-15T13:29:54.195576+0000","last_undegraded":"2024-05-15T13:29:54.195576+0000","last_fullsized":"2024-05-15T13:29:54.195576+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:12.170850+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:29:12.170850+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:31.103516+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,8],"acting":[6,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"94'139","reported_seq":1934,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.151847+0000","last_change":"2024-05-15T13:29:44.746060+0000","last_active":"2024-05-15T13:29:54.151847+0000","last_peered":"2024-05-15T13:29:54.151847+0000","last_clean":"2024-05-15T13:29:54.151847+0000","last_became_active":"2024-05-15T13:24:06.307697+0000","last_became_peered":"2024-05-15T13:24:06.307697+0000","last_unstale":"2024-05-15T13:29:54.151847+0000","last_undegraded":"2024-05-15T13:29:54.151847+0000","last_fullsized":"2024-05-15T13:29:54.151847+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:44.745969+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:29:44.745969+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:56.706313+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"94'139","reported_seq":1305,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-05-15T13:29:50.943010+0000","last_change":"2024-05-15T13:28:50.642772+0000","last_active":"2024-05-15T13:29:50.943010+0000","last_peered":"2024-05-15T13:29:50.943010+0000","last_clean":"2024-05-15T13:29:50.943010+0000","last_became_active":"2024-05-15T13:24:06.300367+0000","last_became_peered":"2024-05-15T13:24:06.300367+0000","last_unstale":"2024-05-15T13:29:50.943010+0000","last_undegraded":"2024-05-15T13:29:50.943010+0000","last_fullsized":"2024-05-15T13:29:50.943010+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:28:50.642667+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:28:50.642667+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:00.558703+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"94'139","reported_seq":1731,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.258756+0000","last_change":"2024-05-15T13:29:47.060397+0000","last_active":"2024-05-15T13:29:54.258756+0000","last_peered":"2024-05-15T13:29:54.258756+0000","last_clean":"2024-05-15T13:29:54.258756+0000","last_became_active":"2024-05-15T13:25:58.956902+0000","last_became_peered":"2024-05-15T13:25:58.956902+0000","last_unstale":"2024-05-15T13:29:54.258756+0000","last_undegraded":"2024-05-15T13:29:54.258756+0000","last_fullsized":"2024-05-15T13:29:54.258756+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:47.060313+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:22:03.418345+0000","last_clean_scrub_stamp":"2024-05-15T13:29:47.060313+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:31:09.315835+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.e","version":"94'139","reported_seq":1186,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-05-15T13:29:51.554393+0000","last_change":"2024-05-15T13:25:59.104518+0000","last_active":"2024-05-15T13:29:51.554393+0000","last_peered":"2024-05-15T13:29:51.554393+0000","last_clean":"2024-05-15T13:29:51.554393+0000","last_became_active":"2024-05-15T13:25:58.962490+0000","last_became_peered":"2024-05-15T13:25:58.962490+0000","last_unstale":"2024-05-15T13:29:51.554393+0000","last_undegraded":"2024-05-15T13:29:51.554393+0000","last_fullsized":"2024-05-15T13:29:51.554393+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:25:59.104385+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:23:06.819156+0000","last_clean_scrub_stamp":"2024-05-15T13:25:59.104385+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"94'139","reported_seq":1710,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.195687+0000","last_change":"2024-05-15T13:28:51.011599+0000","last_active":"2024-05-15T13:29:54.195687+0000","last_peered":"2024-05-15T13:29:54.195687+0000","last_clean":"2024-05-15T13:29:54.195687+0000","last_became_active":"2024-05-15T13:25:58.982018+0000","last_became_peered":"2024-05-15T13:25:58.982018+0000","last_unstale":"2024-05-15T13:29:54.195687+0000","last_undegraded":"2024-05-15T13:29:54.195687+0000","last_fullsized":"2024-05-15T13:29:54.195687+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:28:51.011461+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:28:51.011461+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:02.255673+0000","scrub_duration":251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"94'139","reported_seq":1355,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-05-15T13:29:51.327064+0000","last_change":"2024-05-15T13:29:43.516245+0000","last_active":"2024-05-15T13:29:51.327064+0000","last_peered":"2024-05-15T13:29:51.327064+0000","last_clean":"2024-05-15T13:29:51.327064+0000","last_became_active":"2024-05-15T13:24:06.308068+0000","last_became_peered":"2024-05-15T13:24:06.308068+0000","last_unstale":"2024-05-15T13:29:51.327064+0000","last_undegraded":"2024-05-15T13:29:51.327064+0000","last_fullsized":"2024-05-15T13:29:51.327064+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:43.516140+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:29:43.516140+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:47.647778+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"94'139","reported_seq":1399,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-05-15T13:29:50.943205+0000","last_change":"2024-05-15T13:28:53.617167+0000","last_active":"2024-05-15T13:29:50.943205+0000","last_peered":"2024-05-15T13:29:50.943205+0000","last_clean":"2024-05-15T13:29:50.943205+0000","last_became_active":"2024-05-15T13:27:32.402681+0000","last_became_peered":"2024-05-15T13:27:32.402681+0000","last_unstale":"2024-05-15T13:29:50.943205+0000","last_undegraded":"2024-05-15T13:29:50.943205+0000","last_fullsized":"2024-05-15T13:29:50.943205+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:28:53.617058+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:23:03.856199+0000","last_clean_scrub_stamp":"2024-05-15T13:28:53.617058+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:07.185251+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,8],"acting":[7,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"94'139","reported_seq":1312,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-05-15T13:29:50.381197+0000","last_change":"2024-05-15T13:29:09.677866+0000","last_active":"2024-05-15T13:29:50.381197+0000","last_peered":"2024-05-15T13:29:50.381197+0000","last_clean":"2024-05-15T13:29:50.381197+0000","last_became_active":"2024-05-15T13:28:01.353258+0000","last_became_peered":"2024-05-15T13:28:01.353258+0000","last_unstale":"2024-05-15T13:29:50.381197+0000","last_undegraded":"2024-05-15T13:29:50.381197+0000","last_fullsized":"2024-05-15T13:29:50.381197+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:09.677728+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:29:09.677728+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:15.946272+0000","scrub_duration":225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"94'139","reported_seq":1952,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.151712+0000","last_change":"2024-05-15T13:29:49.795759+0000","last_active":"2024-05-15T13:29:54.151712+0000","last_peered":"2024-05-15T13:29:54.151712+0000","last_clean":"2024-05-15T13:29:54.151712+0000","last_became_active":"2024-05-15T13:19:41.190923+0000","last_became_peered":"2024-05-15T13:19:41.190923+0000","last_unstale":"2024-05-15T13:29:54.151712+0000","last_undegraded":"2024-05-15T13:29:54.151712+0000","last_fullsized":"2024-05-15T13:29:54.151712+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:49.795649+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:29:49.795649+0000","last_clean_scrub_stamp":"2024-05-15T13:29:49.795649+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:56.535421+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"94'139","reported_seq":1728,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.259469+0000","last_change":"2024-05-15T13:29:49.687124+0000","last_active":"2024-05-15T13:29:54.259469+0000","last_peered":"2024-05-15T13:29:54.259469+0000","last_clean":"2024-05-15T13:29:54.259469+0000","last_became_active":"2024-05-15T13:27:32.378438+0000","last_became_peered":"2024-05-15T13:27:32.378438+0000","last_unstale":"2024-05-15T13:29:54.259469+0000","last_undegraded":"2024-05-15T13:29:54.259469+0000","last_fullsized":"2024-05-15T13:29:54.259469+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:49.687008+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:29:49.687008+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:31:11.219965+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"94'139","reported_seq":1968,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.151718+0000","last_change":"2024-05-15T13:29:47.804385+0000","last_active":"2024-05-15T13:29:54.151718+0000","last_peered":"2024-05-15T13:29:54.151718+0000","last_clean":"2024-05-15T13:29:54.151718+0000","last_became_active":"2024-05-15T13:25:15.377001+0000","last_became_peered":"2024-05-15T13:25:15.377001+0000","last_unstale":"2024-05-15T13:29:54.151718+0000","last_undegraded":"2024-05-15T13:29:54.151718+0000","last_fullsized":"2024-05-15T13:29:54.151718+0000","mapping_epoch":868,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":869,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:47.803958+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:18:06.340555+0000","last_clean_scrub_stamp":"2024-05-15T13:29:47.803958+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:31:17.272715+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"94'139","reported_seq":1259,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-05-15T13:29:50.381230+0000","last_change":"2024-05-15T13:29:10.647683+0000","last_active":"2024-05-15T13:29:50.381230+0000","last_peered":"2024-05-15T13:29:50.381230+0000","last_clean":"2024-05-15T13:29:50.381230+0000","last_became_active":"2024-05-15T13:28:01.353822+0000","last_became_peered":"2024-05-15T13:28:01.353822+0000","last_unstale":"2024-05-15T13:29:50.381230+0000","last_undegraded":"2024-05-15T13:29:50.381230+0000","last_fullsized":"2024-05-15T13:29:50.381230+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:10.647536+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:29:10.647536+0000","last_clean_scrub_stamp":"2024-05-15T13:29:10.647536+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:26.229464+0000","scrub_duration":188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,0],"acting":[11,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057174,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":935784,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":933566,"kb_avail":92809368,"statfs":{"total":95995035648,"available":95036792832,"internally_reserved":0,"allocated":2256896,"data_stored":1443612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":955972038},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351514,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":944104,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":941886,"kb_avail":92801048,"statfs":{"total":95995035648,"available":95028273152,"internally_reserved":0,"allocated":2256896,"data_stored":1443612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":964491712},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417515043,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1178740,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":1176510,"kb_avail":92566412,"statfs":{"total":95995035648,"available":94788005888,"internally_reserved":0,"allocated":2269184,"data_stored":1451057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":1204746688},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874789,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":538740,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93206412,"statfs":{"total":95995035648,"available":95443365888,"internally_reserved":0,"allocated":2269184,"data_stored":1451057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169130,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":416752,"kb_used_data":2404,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93328400,"statfs":{"total":95995035648,"available":95568281600,"internally_reserved":0,"allocated":2461696,"data_stored":1644860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":424278474},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430769,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":795880,"kb_used_data":2524,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":92949272,"statfs":{"total":95995035648,"available":95180054528,"internally_reserved":0,"allocated":2584576,"data_stored":1771292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":812382662},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575434,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738963,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":890292,"kb_used_data":2536,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92854860,"statfs":{"total":95995035648,"available":95083376640,"internally_reserved":0,"allocated":2596864,"data_stored":1778737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":909048261},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411909,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":625008,"kb_used_data":2404,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93120144,"statfs":{"total":95995035648,"available":95355027456,"internally_reserved":0,"allocated":2461696,"data_stored":1644860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150271,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1192936,"kb_used_data":2524,"kb_used_omap":1,"kb_used_meta":1190398,"kb_avail":92552216,"statfs":{"total":95995035648,"available":94773469184,"internally_reserved":0,"allocated":2584576,"data_stored":1771292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1218968007},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":887,"seq":3809635992114,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":696984,"kb_used_data":2192,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93048168,"statfs":{"total":95995035648,"available":95281324032,"internally_reserved":0,"allocated":2244608,"data_stored":1436167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692406,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":958204,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":955774,"kb_avail":92786948,"statfs":{"total":95995035648,"available":95013834752,"internally_reserved":0,"allocated":2473984,"data_stored":1652305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":978713029},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:29:57.646 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:29:57.904 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:29:57.923 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:29:58.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:58 smithi177 ceph-mon[44303]: pgmap v2497: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:58.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:29:58 smithi177 ceph-mon[44303]: from='client.17094 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:29:58.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:58 smithi027 ceph-mon[39795]: pgmap v2497: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:58.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:29:58 smithi027 ceph-mon[39795]: from='client.17094 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:29:58.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:58 smithi120 ceph-mon[43961]: pgmap v2497: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:29:58.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:29:58 smithi120 ceph-mon[43961]: from='client.17094 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:29:59.743 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:29:59.743 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-05-15T13:29:59.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-05-15T13:29:59.743 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-05-15T13:29:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:144394 9.5 9:a6570608:::benchmark_data_smithi177_159919_object24151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e893)", 2024-05-15T13:29:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:26:07.428112+0000", 2024-05-15T13:29:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 232.311112536, 2024-05-15T13:29:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.089181659999999996, 2024-05-15T13:29:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 144394 2024-05-15T13:29:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:59.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:07.428112+0000", 2024-05-15T13:29:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:07.428112+0000", 2024-05-15T13:29:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:07.428113+0000", 2024-05-15T13:29:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.57e-06 2024-05-15T13:29:59.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:07.428119+0000", 2024-05-15T13:29:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7230000000000001e-06 2024-05-15T13:29:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:07.428134+0000", 2024-05-15T13:29:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4647000000000001e-05 2024-05-15T13:29:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:07.428141+0000", 2024-05-15T13:29:59.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3889999999999999e-06 2024-05-15T13:29:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:07.428187+0000", 2024-05-15T13:29:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6393000000000001e-05 2024-05-15T13:29:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:07.428274+0000", 2024-05-15T13:29:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6268999999999997e-05 2024-05-15T13:29:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T13:29:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:07.428348+0000", 2024-05-15T13:29:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3886000000000004e-05 2024-05-15T13:29:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:07.433087+0000", 2024-05-15T13:29:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0047393419999999997 2024-05-15T13:29:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:07.454485+0000", 2024-05-15T13:29:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021398127999999999 2024-05-15T13:29:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:07.517190+0000", 2024-05-15T13:29:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.062705005999999994 2024-05-15T13:29:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:07.517249+0000", 2024-05-15T13:29:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8400999999999999e-05 2024-05-15T13:29:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:26:07.517293+0000", 2024-05-15T13:29:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4906e-05 2024-05-15T13:29:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.749 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:169860 9.5 9:abf44cd0:::benchmark_data_smithi177_159919_object49617:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e909)", 2024-05-15T13:29:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:27:03.104684+0000", 2024-05-15T13:29:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 176.63454018799999, 2024-05-15T13:29:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.087180856000000001, 2024-05-15T13:29:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:59.750 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 169860 2024-05-15T13:29:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:03.104684+0000", 2024-05-15T13:29:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:03.104684+0000", 2024-05-15T13:29:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.751 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:03.104686+0000", 2024-05-15T13:29:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6729999999999999e-06 2024-05-15T13:29:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:03.104703+0000", 2024-05-15T13:29:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7260000000000001e-05 2024-05-15T13:29:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:03.104722+0000", 2024-05-15T13:29:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9026999999999999e-05 2024-05-15T13:29:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.752 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:03.104731+0000", 2024-05-15T13:29:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.0999999999999993e-06 2024-05-15T13:29:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:03.104793+0000", 2024-05-15T13:29:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2099000000000003e-05 2024-05-15T13:29:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:03.104942+0000", 2024-05-15T13:29:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014869500000000001 2024-05-15T13:29:59.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T13:29:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:03.105081+0000", 2024-05-15T13:29:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013871599999999999 2024-05-15T13:29:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:03.109215+0000", 2024-05-15T13:29:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0041345959999999999 2024-05-15T13:29:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:03.115125+0000", 2024-05-15T13:29:59.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0059100919999999996 2024-05-15T13:29:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:03.191753+0000", 2024-05-15T13:29:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.076627727000000007 2024-05-15T13:29:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:03.191810+0000", 2024-05-15T13:29:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6697999999999998e-05 2024-05-15T13:29:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:59.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:03.191865+0000", 2024-05-15T13:29:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5173000000000002e-05 2024-05-15T13:29:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:193547 9.5 9:a23f45c2:::benchmark_data_smithi177_159919_object73304:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e937)", 2024-05-15T13:29:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:27:58.413410+0000", 2024-05-15T13:29:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 121.325814561, 2024-05-15T13:29:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.085635226, 2024-05-15T13:29:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-05-15T13:29:59.756 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 193547 2024-05-15T13:29:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413410+0000", 2024-05-15T13:29:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413410+0000", 2024-05-15T13:29:59.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413412+0000", 2024-05-15T13:29:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3010000000000001e-06 2024-05-15T13:29:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413421+0000", 2024-05-15T13:29:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7909999999999993e-06 2024-05-15T13:29:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:59.759 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413444+0000", 2024-05-15T13:29:59.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2796999999999999e-05 2024-05-15T13:29:59.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.759 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:59.759 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413455+0000", 2024-05-15T13:29:59.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1015e-05 2024-05-15T13:29:59.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.759 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.759 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413529+0000", 2024-05-15T13:29:59.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4033000000000001e-05 2024-05-15T13:29:59.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op must wait for map", 2024-05-15T13:29:59.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413537+0000", 2024-05-15T13:29:59.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1319999999999994e-06 2024-05-15T13:29:59.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.419096+0000", 2024-05-15T13:29:59.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0055591210000000002 2024-05-15T13:29:59.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:29:59.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.419113+0000", 2024-05-15T13:29:59.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7085999999999999e-05 2024-05-15T13:29:59.761 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.466102+0000", 2024-05-15T13:29:59.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.046988858 2024-05-15T13:29:59.761 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for missing object", 2024-05-15T13:29:59.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.466225+0000", 2024-05-15T13:29:59.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000123335 2024-05-15T13:29:59.761 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.762 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.762 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.498987+0000", 2024-05-15T13:29:59.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.032761361000000003 2024-05-15T13:29:59.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.762 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:59.762 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.499045+0000", 2024-05-15T13:29:59.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8396000000000001e-05 2024-05-15T13:29:59.762 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.762 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:59.762 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.762 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:193550 9.5 9:a2e3715c:::benchmark_data_smithi177_159919_object73307:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e937)", 2024-05-15T13:29:59.763 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:27:58.413479+0000", 2024-05-15T13:29:59.763 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 121.325744804, 2024-05-15T13:29:59.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0849820299999999, 2024-05-15T13:29:59.763 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:59.763 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:59.763 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:59.763 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:59.763 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:59.763 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 193550 2024-05-15T13:29:59.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.763 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:59.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.763 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:59.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413479+0000", 2024-05-15T13:29:59.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:59.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413479+0000", 2024-05-15T13:29:59.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:59.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413480+0000", 2024-05-15T13:29:59.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7300000000000001e-07 2024-05-15T13:29:59.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:59.765 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413484+0000", 2024-05-15T13:29:59.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.459e-06 2024-05-15T13:29:59.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:59.765 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413491+0000", 2024-05-15T13:29:59.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9290000000000001e-06 2024-05-15T13:29:59.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:59.765 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413498+0000", 2024-05-15T13:29:59.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9450000000000004e-06 2024-05-15T13:29:59.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413610+0000", 2024-05-15T13:29:59.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111845 2024-05-15T13:29:59.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting_for_map not empty", 2024-05-15T13:29:59.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413618+0000", 2024-05-15T13:29:59.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8380000000000002e-06 2024-05-15T13:29:59.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.419149+0000", 2024-05-15T13:29:59.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0055310489999999997 2024-05-15T13:29:59.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:29:59.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.419158+0000", 2024-05-15T13:29:59.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5839999999999995e-06 2024-05-15T13:29:59.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.466500+0000", 2024-05-15T13:29:59.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0473427930000001 2024-05-15T13:29:59.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:59.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.466635+0000", 2024-05-15T13:29:59.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013507100000000001 2024-05-15T13:29:59.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T13:29:59.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.466772+0000", 2024-05-15T13:29:59.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000136384 2024-05-15T13:29:59.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.493271+0000", 2024-05-15T13:29:59.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.026498767999999999 2024-05-15T13:29:59.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.493340+0000", 2024-05-15T13:29:59.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8887999999999995e-05 2024-05-15T13:29:59.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:59.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.498286+0000", 2024-05-15T13:29:59.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0049460980000000003 2024-05-15T13:29:59.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:59.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.498352+0000", 2024-05-15T13:29:59.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6567000000000001e-05 2024-05-15T13:29:59.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:59.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.498462+0000", 2024-05-15T13:29:59.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109339 2024-05-15T13:29:59.770 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.770 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:59.770 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.770 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:193555 9.5 9:a38770fd:::benchmark_data_smithi177_159919_object73312:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e937)", 2024-05-15T13:29:59.770 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:27:58.413508+0000", 2024-05-15T13:29:59.770 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 121.325716329, 2024-05-15T13:29:59.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0852762730000001, 2024-05-15T13:29:59.771 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:59.771 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:59.771 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:59.771 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:59.771 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:59.771 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 193555 2024-05-15T13:29:59.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.771 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:59.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.771 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:59.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413508+0000", 2024-05-15T13:29:59.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:59.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413508+0000", 2024-05-15T13:29:59.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:59.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413508+0000", 2024-05-15T13:29:59.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9799999999999999e-07 2024-05-15T13:29:59.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:59.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413521+0000", 2024-05-15T13:29:59.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2381999999999999e-05 2024-05-15T13:29:59.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.773 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:59.773 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413528+0000", 2024-05-15T13:29:59.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1799999999999999e-06 2024-05-15T13:29:59.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.773 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:59.773 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413534+0000", 2024-05-15T13:29:59.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.3450000000000002e-06 2024-05-15T13:29:59.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.773 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413643+0000", 2024-05-15T13:29:59.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108575 2024-05-15T13:29:59.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting_for_map not empty", 2024-05-15T13:29:59.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413647+0000", 2024-05-15T13:29:59.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8859999999999997e-06 2024-05-15T13:29:59.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.419188+0000", 2024-05-15T13:29:59.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0055412969999999997 2024-05-15T13:29:59.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:29:59.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.419195+0000", 2024-05-15T13:29:59.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5679999999999996e-06 2024-05-15T13:29:59.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.472030+0000", 2024-05-15T13:29:59.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0528350829999999 2024-05-15T13:29:59.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:59.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.472158+0000", 2024-05-15T13:29:59.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012871200000000001 2024-05-15T13:29:59.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T13:29:59.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.472297+0000", 2024-05-15T13:29:59.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000138348 2024-05-15T13:29:59.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:59.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.498472+0000", 2024-05-15T13:29:59.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.026175120999999999 2024-05-15T13:29:59.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.498560+0000", 2024-05-15T13:29:59.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8522000000000004e-05 2024-05-15T13:29:59.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.498630+0000", 2024-05-15T13:29:59.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9330000000000002e-05 2024-05-15T13:29:59.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:59.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.498678+0000", 2024-05-15T13:29:59.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8248999999999998e-05 2024-05-15T13:29:59.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:59.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.498784+0000", 2024-05-15T13:29:59.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000106277 2024-05-15T13:29:59.778 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.778 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:59.778 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.778 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.778 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:193564 9.5 9:ae9a364b:::benchmark_data_smithi177_159919_object73321:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e937)", 2024-05-15T13:29:59.778 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:27:58.413542+0000", 2024-05-15T13:29:59.778 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 121.325682456, 2024-05-15T13:29:59.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0946678219999999, 2024-05-15T13:29:59.779 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:59.779 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:59.779 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:59.779 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:59.779 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:59.779 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 193564 2024-05-15T13:29:59.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.779 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:59.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:59.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413542+0000", 2024-05-15T13:29:59.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:59.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413542+0000", 2024-05-15T13:29:59.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:59.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413542+0000", 2024-05-15T13:29:59.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2599999999999998e-07 2024-05-15T13:29:59.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:59.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413545+0000", 2024-05-15T13:29:59.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4760000000000001e-06 2024-05-15T13:29:59.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:59.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413551+0000", 2024-05-15T13:29:59.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2400000000000004e-06 2024-05-15T13:29:59.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:59.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413556+0000", 2024-05-15T13:29:59.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2240000000000003e-06 2024-05-15T13:29:59.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413671+0000", 2024-05-15T13:29:59.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000115165 2024-05-15T13:29:59.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting_for_map not empty", 2024-05-15T13:29:59.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413675+0000", 2024-05-15T13:29:59.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7270000000000001e-06 2024-05-15T13:29:59.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.419232+0000", 2024-05-15T13:29:59.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0055565950000000001 2024-05-15T13:29:59.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:29:59.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.419237+0000", 2024-05-15T13:29:59.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8269999999999997e-06 2024-05-15T13:29:59.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.477270+0000", 2024-05-15T13:29:59.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.058033792 2024-05-15T13:29:59.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.783 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:59.783 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.477401+0000", 2024-05-15T13:29:59.783 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013085399999999999 2024-05-15T13:29:59.783 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.783 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T13:29:59.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.477538+0000", 2024-05-15T13:29:59.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013656399999999999 2024-05-15T13:29:59.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:59.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.498480+0000", 2024-05-15T13:29:59.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.020941946999999999 2024-05-15T13:29:59.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.784 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.784 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.784 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.499139+0000", 2024-05-15T13:29:59.784 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00065919600000000002 2024-05-15T13:29:59.784 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.508070+0000", 2024-05-15T13:29:59.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0089308749999999996 2024-05-15T13:29:59.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:59.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.508134+0000", 2024-05-15T13:29:59.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4008000000000001e-05 2024-05-15T13:29:59.785 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.785 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.785 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:59.785 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.508210+0000", 2024-05-15T13:29:59.785 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5906000000000001e-05 2024-05-15T13:29:59.786 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.786 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:59.786 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.786 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.786 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.786 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:193575 9.5 9:a597f500:::benchmark_data_smithi177_159919_object73332:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e937)", 2024-05-15T13:29:59.786 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:27:58.413563+0000", 2024-05-15T13:29:59.786 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 121.325661311, 2024-05-15T13:29:59.786 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0948674819999999, 2024-05-15T13:29:59.786 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:59.786 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:59.786 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:59.786 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:59.786 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:59.787 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 193575 2024-05-15T13:29:59.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.787 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:59.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:59.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413563+0000", 2024-05-15T13:29:59.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.787 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:59.787 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413563+0000", 2024-05-15T13:29:59.787 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.787 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.787 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:59.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413572+0000", 2024-05-15T13:29:59.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1139999999999995e-06 2024-05-15T13:29:59.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:59.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413575+0000", 2024-05-15T13:29:59.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6740000000000001e-06 2024-05-15T13:29:59.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.788 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:59.788 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413582+0000", 2024-05-15T13:29:59.788 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3780000000000003e-06 2024-05-15T13:29:59.788 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.788 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:59.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413588+0000", 2024-05-15T13:29:59.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6810000000000003e-06 2024-05-15T13:29:59.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413697+0000", 2024-05-15T13:29:59.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108707 2024-05-15T13:29:59.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.789 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.789 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting_for_map not empty", 2024-05-15T13:29:59.789 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413700+0000", 2024-05-15T13:29:59.789 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.45e-06 2024-05-15T13:29:59.789 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.419276+0000", 2024-05-15T13:29:59.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0055760389999999996 2024-05-15T13:29:59.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:29:59.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.419283+0000", 2024-05-15T13:29:59.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5629999999999997e-06 2024-05-15T13:29:59.790 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.790 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.790 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.790 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.482574+0000", 2024-05-15T13:29:59.790 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0632911229999999 2024-05-15T13:29:59.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:59.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.482697+0000", 2024-05-15T13:29:59.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000123471 2024-05-15T13:29:59.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T13:29:59.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.482826+0000", 2024-05-15T13:29:59.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000128374 2024-05-15T13:29:59.791 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.791 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.791 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:59.791 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.498486+0000", 2024-05-15T13:29:59.791 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015660838 2024-05-15T13:29:59.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.499198+0000", 2024-05-15T13:29:59.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00071176699999999998 2024-05-15T13:29:59.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.508339+0000", 2024-05-15T13:29:59.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.009140825 2024-05-15T13:29:59.792 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.792 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.792 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:59.792 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.508368+0000", 2024-05-15T13:29:59.792 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8668000000000001e-05 2024-05-15T13:29:59.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.793 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:59.793 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.508430+0000", 2024-05-15T13:29:59.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2810000000000003e-05 2024-05-15T13:29:59.793 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.793 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:59.793 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.793 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.793 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.793 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:193589 9.5 9:a9e11dc1:::benchmark_data_smithi177_159919_object73346:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e937)", 2024-05-15T13:29:59.793 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:27:58.413595+0000", 2024-05-15T13:29:59.793 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 121.32562956699999, 2024-05-15T13:29:59.793 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0950129529999999, 2024-05-15T13:29:59.794 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:59.794 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:59.794 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:59.794 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:59.794 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:59.794 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 193589 2024-05-15T13:29:59.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.794 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:59.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.794 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:59.794 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413595+0000", 2024-05-15T13:29:59.794 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.794 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.794 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:59.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413595+0000", 2024-05-15T13:29:59.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:59.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413595+0000", 2024-05-15T13:29:59.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.01e-07 2024-05-15T13:29:59.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.795 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:59.795 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413597+0000", 2024-05-15T13:29:59.795 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1799999999999999e-06 2024-05-15T13:29:59.795 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.795 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:59.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413605+0000", 2024-05-15T13:29:59.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3590000000000002e-06 2024-05-15T13:29:59.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:59.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413611+0000", 2024-05-15T13:29:59.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.319e-06 2024-05-15T13:29:59.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.796 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.796 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.796 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413715+0000", 2024-05-15T13:29:59.796 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000103569 2024-05-15T13:29:59.796 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting_for_map not empty", 2024-05-15T13:29:59.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.413718+0000", 2024-05-15T13:29:59.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.484e-06 2024-05-15T13:29:59.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.419311+0000", 2024-05-15T13:29:59.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0055926700000000001 2024-05-15T13:29:59.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.797 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:29:59.797 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.419315+0000", 2024-05-15T13:29:59.797 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1439999999999996e-06 2024-05-15T13:29:59.797 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.797 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.487875+0000", 2024-05-15T13:29:59.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0685604879999999 2024-05-15T13:29:59.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:59.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.488015+0000", 2024-05-15T13:29:59.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013935400000000001 2024-05-15T13:29:59.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.798 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.798 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T13:29:59.798 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.488161+0000", 2024-05-15T13:29:59.798 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000145909 2024-05-15T13:29:59.798 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:59.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.498796+0000", 2024-05-15T13:29:59.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010635498 2024-05-15T13:29:59.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.499251+0000", 2024-05-15T13:29:59.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00045538399999999998 2024-05-15T13:29:59.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.799 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.799 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.799 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.508470+0000", 2024-05-15T13:29:59.799 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0092188830000000006 2024-05-15T13:29:59.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:59.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.508506+0000", 2024-05-15T13:29:59.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5635e-05 2024-05-15T13:29:59.800 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.800 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.800 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:59.800 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.508608+0000", 2024-05-15T13:29:59.800 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010167599999999999 2024-05-15T13:29:59.800 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.800 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:59.800 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.801 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:193597 9.5 9:a043ff3d:::benchmark_data_smithi177_159919_object73354:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e937)", 2024-05-15T13:29:59.801 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:27:58.618157+0000", 2024-05-15T13:29:59.801 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 121.12106744, 2024-05-15T13:29:59.801 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.89716840399999997, 2024-05-15T13:29:59.801 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:59.801 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:59.801 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:59.801 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:59.801 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:59.801 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 193597 2024-05-15T13:29:59.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.801 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:59.801 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:59.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.618157+0000", 2024-05-15T13:29:59.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:59.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.618157+0000", 2024-05-15T13:29:59.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.802 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.802 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:59.802 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.618158+0000", 2024-05-15T13:29:59.802 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.604e-06 2024-05-15T13:29:59.802 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:59.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.618167+0000", 2024-05-15T13:29:59.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1310000000000006e-06 2024-05-15T13:29:59.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:59.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.618184+0000", 2024-05-15T13:29:59.803 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7453999999999999e-05 2024-05-15T13:29:59.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.803 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.803 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:59.803 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.618192+0000", 2024-05-15T13:29:59.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5560000000000002e-06 2024-05-15T13:29:59.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.618248+0000", 2024-05-15T13:29:59.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6382999999999999e-05 2024-05-15T13:29:59.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.804 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:29:59.804 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.618255+0000", 2024-05-15T13:29:59.804 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4109999999999999e-06 2024-05-15T13:29:59.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.804 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.499280+0000", 2024-05-15T13:29:59.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.88102414200000001 2024-05-15T13:29:59.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:59.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.499357+0000", 2024-05-15T13:29:59.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7096999999999997e-05 2024-05-15T13:29:59.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.805 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.805 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T13:29:59.805 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.499435+0000", 2024-05-15T13:29:59.805 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8372000000000001e-05 2024-05-15T13:29:59.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:59.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.508622+0000", 2024-05-15T13:29:59.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0091872399999999993 2024-05-15T13:29:59.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.511230+0000", 2024-05-15T13:29:59.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0026080130000000002 2024-05-15T13:29:59.806 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.806 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.806 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.806 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.515214+0000", 2024-05-15T13:29:59.806 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003983487 2024-05-15T13:29:59.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:59.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.515274+0000", 2024-05-15T13:29:59.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0535000000000002e-05 2024-05-15T13:29:59.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.807 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.807 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:59.807 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.515325+0000", 2024-05-15T13:29:59.807 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0979000000000001e-05 2024-05-15T13:29:59.807 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.807 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:59.807 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.807 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.808 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:193602 9.5 9:a39b9226:::benchmark_data_smithi177_159919_object73359:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e937)", 2024-05-15T13:29:59.808 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:27:58.660558+0000", 2024-05-15T13:29:59.808 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 121.07866604, 2024-05-15T13:29:59.808 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.86084806999999997, 2024-05-15T13:29:59.808 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:59.808 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:59.808 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:59.808 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:59.808 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:59.808 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 193602 2024-05-15T13:29:59.808 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.808 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:59.808 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.808 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:59.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.660558+0000", 2024-05-15T13:29:59.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:59.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.660558+0000", 2024-05-15T13:29:59.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.809 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:59.809 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.660560+0000", 2024-05-15T13:29:59.809 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3990000000000001e-06 2024-05-15T13:29:59.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.809 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:59.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.660575+0000", 2024-05-15T13:29:59.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.487e-05 2024-05-15T13:29:59.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:59.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.660592+0000", 2024-05-15T13:29:59.810 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7275999999999999e-05 2024-05-15T13:29:59.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.810 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.810 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:59.810 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.660599+0000", 2024-05-15T13:29:59.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4830000000000002e-06 2024-05-15T13:29:59.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.811 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.660658+0000", 2024-05-15T13:29:59.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8863999999999997e-05 2024-05-15T13:29:59.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-05-15T13:29:59.811 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:58.660666+0000", 2024-05-15T13:29:59.811 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0099999999999995e-06 2024-05-15T13:29:59.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.811 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.811 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.503552+0000", 2024-05-15T13:29:59.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.84288591199999996 2024-05-15T13:29:59.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:59.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.503701+0000", 2024-05-15T13:29:59.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014919 2024-05-15T13:29:59.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.812 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T13:29:59.812 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.503815+0000", 2024-05-15T13:29:59.812 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000113415 2024-05-15T13:29:59.812 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.812 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:59.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.508627+0000", 2024-05-15T13:29:59.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.004812513 2024-05-15T13:29:59.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.511285+0000", 2024-05-15T13:29:59.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0026579709999999999 2024-05-15T13:29:59.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.813 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.813 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.813 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.521290+0000", 2024-05-15T13:29:59.813 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010004608 2024-05-15T13:29:59.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:59.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.521351+0000", 2024-05-15T13:29:59.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1131999999999997e-05 2024-05-15T13:29:59.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.814 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.814 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:59.814 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:27:59.521406+0000", 2024-05-15T13:29:59.814 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.5427000000000003e-05 2024-05-15T13:29:59.814 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.814 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:59.814 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.815 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:208393 9.5 9:a3477c69:::benchmark_data_smithi177_159919_object88150:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e954)", 2024-05-15T13:29:59.815 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:27.806879+0000", 2024-05-15T13:29:59.815 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 91.932345159999997, 2024-05-15T13:29:59.815 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.084499307999999995, 2024-05-15T13:29:59.815 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:59.815 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:59.815 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:59.815 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:59.815 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:59.815 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 208393 2024-05-15T13:29:59.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.815 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:59.815 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.815 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:59.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:27.806879+0000", 2024-05-15T13:29:59.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:59.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:27.806879+0000", 2024-05-15T13:29:59.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.816 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:59.816 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:27.806880+0000", 2024-05-15T13:29:59.816 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3179999999999999e-06 2024-05-15T13:29:59.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.816 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:59.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:27.806886+0000", 2024-05-15T13:29:59.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3539999999999999e-06 2024-05-15T13:29:59.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:59.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:27.806897+0000", 2024-05-15T13:29:59.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1308000000000001e-05 2024-05-15T13:29:59.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.817 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.817 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:59.817 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:27.806906+0000", 2024-05-15T13:29:59.817 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3500000000000003e-06 2024-05-15T13:29:59.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:27.806967+0000", 2024-05-15T13:29:59.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0844999999999998e-05 2024-05-15T13:29:59.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:59.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:27.807085+0000", 2024-05-15T13:29:59.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011799299999999999 2024-05-15T13:29:59.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.818 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.818 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T13:29:59.818 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:27.807197+0000", 2024-05-15T13:29:59.818 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011160200000000001 2024-05-15T13:29:59.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:59.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:27.815976+0000", 2024-05-15T13:29:59.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.008778704 2024-05-15T13:29:59.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.819 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:27.819898+0000", 2024-05-15T13:29:59.819 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0039225299999999996 2024-05-15T13:29:59.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.819 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.819 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:27.891263+0000", 2024-05-15T13:29:59.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.071365003999999996 2024-05-15T13:29:59.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:59.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:27.891326+0000", 2024-05-15T13:29:59.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2877000000000001e-05 2024-05-15T13:29:59.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.820 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.820 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:59.820 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:27.891378+0000", 2024-05-15T13:29:59.820 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2423000000000003e-05 2024-05-15T13:29:59.820 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.820 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:59.821 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.821 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.821 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:208861 9.5 9:a650cd7c:::benchmark_data_smithi177_159919_object88618:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e954)", 2024-05-15T13:29:59.821 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:28.610173+0000", 2024-05-15T13:29:59.821 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 91.129050931999998, 2024-05-15T13:29:59.821 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.087666418999999995, 2024-05-15T13:29:59.821 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:59.821 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:59.821 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:59.821 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:59.821 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:59.821 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 208861 2024-05-15T13:29:59.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.822 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:59.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:59.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.610173+0000", 2024-05-15T13:29:59.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:59.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.610173+0000", 2024-05-15T13:29:59.822 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.822 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.822 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:59.822 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.610174+0000", 2024-05-15T13:29:59.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.61e-07 2024-05-15T13:29:59.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:59.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.610177+0000", 2024-05-15T13:29:59.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8729999999999999e-06 2024-05-15T13:29:59.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:59.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.610188+0000", 2024-05-15T13:29:59.823 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0949e-05 2024-05-15T13:29:59.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.823 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.823 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:59.823 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.610195+0000", 2024-05-15T13:29:59.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7780000000000002e-06 2024-05-15T13:29:59.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.614485+0000", 2024-05-15T13:29:59.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0042906610000000003 2024-05-15T13:29:59.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:59.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.614596+0000", 2024-05-15T13:29:59.824 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000110678 2024-05-15T13:29:59.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.824 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.824 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T13:29:59.824 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.614703+0000", 2024-05-15T13:29:59.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000107336 2024-05-15T13:29:59.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:59.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.623540+0000", 2024-05-15T13:29:59.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0088361480000000003 2024-05-15T13:29:59.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.825 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.637409+0000", 2024-05-15T13:29:59.825 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013869312999999999 2024-05-15T13:29:59.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.825 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.825 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.697677+0000", 2024-05-15T13:29:59.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.060268173000000001 2024-05-15T13:29:59.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:59.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.697791+0000", 2024-05-15T13:29:59.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000113793 2024-05-15T13:29:59.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.826 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.826 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:59.826 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.697840+0000", 2024-05-15T13:29:59.826 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8955999999999997e-05 2024-05-15T13:29:59.826 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.826 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:59.827 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.827 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.827 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:208864 9.5 9:a67a22d5:::benchmark_data_smithi177_159919_object88621:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e954)", 2024-05-15T13:29:59.827 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:28.610333+0000", 2024-05-15T13:29:59.827 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 91.128891577000005, 2024-05-15T13:29:59.827 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.087690586000000001, 2024-05-15T13:29:59.827 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:59.827 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:59.827 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:59.827 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:59.827 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:59.827 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 208864 2024-05-15T13:29:59.827 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.827 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:59.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:59.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.610333+0000", 2024-05-15T13:29:59.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:59.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.610333+0000", 2024-05-15T13:29:59.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.828 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.828 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.828 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:59.828 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.610333+0000", 2024-05-15T13:29:59.828 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7000000000000004e-07 2024-05-15T13:29:59.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:59.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.610338+0000", 2024-05-15T13:29:59.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2590000000000002e-06 2024-05-15T13:29:59.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:59.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.610347+0000", 2024-05-15T13:29:59.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9840000000000007e-06 2024-05-15T13:29:59.829 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.829 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.829 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:59.829 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.610354+0000", 2024-05-15T13:29:59.829 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0899999999999999e-06 2024-05-15T13:29:59.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.619611+0000", 2024-05-15T13:29:59.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0092571389999999993 2024-05-15T13:29:59.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:59.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.619708+0000", 2024-05-15T13:29:59.830 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6775999999999996e-05 2024-05-15T13:29:59.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.830 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.830 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T13:29:59.830 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.619782+0000", 2024-05-15T13:29:59.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4043999999999998e-05 2024-05-15T13:29:59.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:59.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.623679+0000", 2024-05-15T13:29:59.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0038976990000000001 2024-05-15T13:29:59.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.831 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.637445+0000", 2024-05-15T13:29:59.831 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013765937000000001 2024-05-15T13:29:59.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.831 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.831 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.697925+0000", 2024-05-15T13:29:59.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.060479131999999998 2024-05-15T13:29:59.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:59.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.697978+0000", 2024-05-15T13:29:59.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3542999999999999e-05 2024-05-15T13:29:59.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.832 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.832 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:59.832 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:28.698023+0000", 2024-05-15T13:29:59.832 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5213000000000002e-05 2024-05-15T13:29:59.832 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.833 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:59.833 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.833 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.833 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.833 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:211236 9.5 9:ab2ffba5:::benchmark_data_smithi177_159919_object90993:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e956)", 2024-05-15T13:29:59.833 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:32.677040+0000", 2024-05-15T13:29:59.833 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 87.062184368000004, 2024-05-15T13:29:59.833 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.085233719999999999, 2024-05-15T13:29:59.833 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:59.833 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:59.833 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:59.833 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:59.833 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:59.833 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 211236 2024-05-15T13:29:59.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.834 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:59.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:59.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:32.677040+0000", 2024-05-15T13:29:59.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.834 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:59.834 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:32.677040+0000", 2024-05-15T13:29:59.834 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.834 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.834 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:59.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:32.677041+0000", 2024-05-15T13:29:59.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.514e-06 2024-05-15T13:29:59.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:59.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:32.677047+0000", 2024-05-15T13:29:59.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4049999999999999e-06 2024-05-15T13:29:59.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.835 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.835 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:59.835 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:32.677060+0000", 2024-05-15T13:29:59.835 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3602e-05 2024-05-15T13:29:59.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:59.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:32.677068+0000", 2024-05-15T13:29:59.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6629999999999993e-06 2024-05-15T13:29:59.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:32.677115+0000", 2024-05-15T13:29:59.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7088e-05 2024-05-15T13:29:59.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.836 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.836 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:59.836 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:32.677211+0000", 2024-05-15T13:29:59.836 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5895000000000003e-05 2024-05-15T13:29:59.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T13:29:59.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:32.677291+0000", 2024-05-15T13:29:59.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9826000000000004e-05 2024-05-15T13:29:59.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:32.682310+0000", 2024-05-15T13:29:59.837 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0050186329999999998 2024-05-15T13:29:59.837 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.837 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.837 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:59.837 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:32.682528+0000", 2024-05-15T13:29:59.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021825699999999999 2024-05-15T13:29:59.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:32.762203+0000", 2024-05-15T13:29:59.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.079675428000000006 2024-05-15T13:29:59.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:59.838 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:32.762240+0000", 2024-05-15T13:29:59.838 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6436000000000001e-05 2024-05-15T13:29:59.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.838 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.838 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:59.839 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:32.762274+0000", 2024-05-15T13:29:59.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3973000000000002e-05 2024-05-15T13:29:59.839 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.839 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:59.839 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.839 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.839 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:222954 9.5 9:a6561452:::benchmark_data_smithi177_159919_object102711:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e966)", 2024-05-15T13:29:59.839 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:51.925730+0000", 2024-05-15T13:29:59.839 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 67.813494575999997, 2024-05-15T13:29:59.839 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.090029707, 2024-05-15T13:29:59.839 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:59.839 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:59.840 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:59.840 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:59.840 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:59.840 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 222954 2024-05-15T13:29:59.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.840 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:59.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:59.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:51.925730+0000", 2024-05-15T13:29:59.840 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.840 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.840 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:59.840 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:51.925730+0000", 2024-05-15T13:29:59.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:59.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:51.925731+0000", 2024-05-15T13:29:59.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3960000000000001e-06 2024-05-15T13:29:59.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.841 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.841 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:59.841 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:51.925737+0000", 2024-05-15T13:29:59.841 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2489999999999998e-06 2024-05-15T13:29:59.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:59.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:51.925749+0000", 2024-05-15T13:29:59.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1348e-05 2024-05-15T13:29:59.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:59.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:51.925756+0000", 2024-05-15T13:29:59.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3699999999999997e-06 2024-05-15T13:29:59.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.842 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.842 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.842 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:51.925804+0000", 2024-05-15T13:29:59.842 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8245999999999999e-05 2024-05-15T13:29:59.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:59.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:51.925919+0000", 2024-05-15T13:29:59.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011503 2024-05-15T13:29:59.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T13:29:59.843 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:51.926029+0000", 2024-05-15T13:29:59.843 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109289 2024-05-15T13:29:59.843 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.843 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.843 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:59.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:51.935480+0000", 2024-05-15T13:29:59.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0094514790000000005 2024-05-15T13:29:59.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:51.935506+0000", 2024-05-15T13:29:59.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5698999999999999e-05 2024-05-15T13:29:59.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.844 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:52.015661+0000", 2024-05-15T13:29:59.844 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.080155468999999993 2024-05-15T13:29:59.844 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.844 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.844 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:59.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:52.015711+0000", 2024-05-15T13:29:59.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9657000000000002e-05 2024-05-15T13:29:59.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.845 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:59.845 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:52.015759+0000", 2024-05-15T13:29:59.845 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8474999999999999e-05 2024-05-15T13:29:59.845 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.845 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:59.845 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.845 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.845 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.845 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.26536.0:222955 9.5 9:a5c70ebc:::benchmark_data_smithi177_159919_object102712:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e966)", 2024-05-15T13:29:59.845 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:28:51.930367+0000", 2024-05-15T13:29:59.846 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 67.808856876999997, 2024-05-15T13:29:59.846 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.093334901999999997, 2024-05-15T13:29:59.846 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:59.846 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T13:29:59.846 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-05-15T13:29:59.846 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.26536", 2024-05-15T13:29:59.846 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.177:0/2117757008", 2024-05-15T13:29:59.846 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 222955 2024-05-15T13:29:59.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.846 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:59.846 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.846 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:59.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:51.930367+0000", 2024-05-15T13:29:59.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:59.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:51.930367+0000", 2024-05-15T13:29:59.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.847 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:59.847 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:51.930369+0000", 2024-05-15T13:29:59.847 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.455e-06 2024-05-15T13:29:59.847 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.847 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:59.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:51.930375+0000", 2024-05-15T13:29:59.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1399999999999997e-06 2024-05-15T13:29:59.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:59.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:51.930391+0000", 2024-05-15T13:29:59.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6288000000000001e-05 2024-05-15T13:29:59.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.848 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.848 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:59.848 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:51.930399+0000", 2024-05-15T13:29:59.848 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5830000000000001e-06 2024-05-15T13:29:59.848 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:51.930451+0000", 2024-05-15T13:29:59.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1829000000000001e-05 2024-05-15T13:29:59.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:59.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:51.930593+0000", 2024-05-15T13:29:59.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014235699999999999 2024-05-15T13:29:59.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.849 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.849 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 5,7", 2024-05-15T13:29:59.849 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:51.930718+0000", 2024-05-15T13:29:59.849 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012507600000000001 2024-05-15T13:29:59.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-05-15T13:29:59.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:51.935620+0000", 2024-05-15T13:29:59.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0049018400000000002 2024-05-15T13:29:59.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:51.939685+0000", 2024-05-15T13:29:59.850 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0040654580000000001 2024-05-15T13:29:59.850 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.850 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.850 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-05-15T13:29:59.850 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:52.023596+0000", 2024-05-15T13:29:59.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.083910988000000006 2024-05-15T13:29:59.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-05-15T13:29:59.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:52.023653+0000", 2024-05-15T13:29:59.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6632999999999998e-05 2024-05-15T13:29:59.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.851 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.851 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:59.851 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:28:52.023702+0000", 2024-05-15T13:29:59.851 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9255000000000002e-05 2024-05-15T13:29:59.851 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.851 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:59.851 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.851 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.852 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDScrubReserve(1.1d,GRANT,e:987,from:5,reservation_nonce:0)", 2024-05-15T13:29:59.852 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:29:47.433180+0000", 2024-05-15T13:29:59.852 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 12.306044684, 2024-05-15T13:29:59.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024527300000000001, 2024-05-15T13:29:59.852 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:59.852 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T13:29:59.852 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:59.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.852 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:59.852 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.433180+0000", 2024-05-15T13:29:59.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.852 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.853 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:59.853 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.433180+0000", 2024-05-15T13:29:59.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.853 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:59.853 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.433180+0000", 2024-05-15T13:29:59.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6400000000000002e-07 2024-05-15T13:29:59.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.853 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:59.853 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.433184+0000", 2024-05-15T13:29:59.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4470000000000002e-06 2024-05-15T13:29:59.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:59.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.433193+0000", 2024-05-15T13:29:59.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.5920000000000007e-06 2024-05-15T13:29:59.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:59.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.433215+0000", 2024-05-15T13:29:59.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1736999999999999e-05 2024-05-15T13:29:59.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.433287+0000", 2024-05-15T13:29:59.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1673000000000003e-05 2024-05-15T13:29:59.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:59.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.433302+0000", 2024-05-15T13:29:59.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5158e-05 2024-05-15T13:29:59.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:59.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.433425+0000", 2024-05-15T13:29:59.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012300200000000001 2024-05-15T13:29:59.855 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.855 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:59.855 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.855 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "rep_scrubmap(1.1d e939 from shard 5)", 2024-05-15T13:29:59.856 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:29:47.528089+0000", 2024-05-15T13:29:59.856 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 12.211135407, 2024-05-15T13:29:59.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0098962670000000003, 2024-05-15T13:29:59.856 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:59.856 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T13:29:59.856 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:59.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:59.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.528089+0000", 2024-05-15T13:29:59.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:59.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.528089+0000", 2024-05-15T13:29:59.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:59.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.528091+0000", 2024-05-15T13:29:59.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6819999999999999e-06 2024-05-15T13:29:59.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:59.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.528099+0000", 2024-05-15T13:29:59.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5779999999999999e-06 2024-05-15T13:29:59.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:59.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.528119+0000", 2024-05-15T13:29:59.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9945999999999998e-05 2024-05-15T13:29:59.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:59.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.528130+0000", 2024-05-15T13:29:59.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0677e-05 2024-05-15T13:29:59.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.858 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.537964+0000", 2024-05-15T13:29:59.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0098338809999999992 2024-05-15T13:29:59.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.858 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.858 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:59.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.537973+0000", 2024-05-15T13:29:59.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6239999999999997e-06 2024-05-15T13:29:59.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:59.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.537985+0000", 2024-05-15T13:29:59.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1878999999999999e-05 2024-05-15T13:29:59.859 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.859 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:59.859 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.859 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "rep_scrubmap(1.1d e939 from shard 2)", 2024-05-15T13:29:59.860 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:29:47.554671+0000", 2024-05-15T13:29:59.860 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 12.18455352, 2024-05-15T13:29:59.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000225777, 2024-05-15T13:29:59.860 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:59.860 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T13:29:59.860 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:59.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:59.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.554671+0000", 2024-05-15T13:29:59.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:59.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.554671+0000", 2024-05-15T13:29:59.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:59.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.554673+0000", 2024-05-15T13:29:59.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.843e-06 2024-05-15T13:29:59.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:59.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.554684+0000", 2024-05-15T13:29:59.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0923e-05 2024-05-15T13:29:59.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:59.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.554709+0000", 2024-05-15T13:29:59.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5466e-05 2024-05-15T13:29:59.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:59.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.554723+0000", 2024-05-15T13:29:59.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3763e-05 2024-05-15T13:29:59.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.554828+0000", 2024-05-15T13:29:59.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105064 2024-05-15T13:29:59.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:59.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.554846+0000", 2024-05-15T13:29:59.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8641999999999999e-05 2024-05-15T13:29:59.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:59.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:47.554897+0000", 2024-05-15T13:29:59.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0076e-05 2024-05-15T13:29:59.864 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.864 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:59.864 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.864 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDScrubReserve(1.1a,QREQUEST,e:987,from:4,reservation_nonce:3)", 2024-05-15T13:29:59.864 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-05-15T13:29:49.558501+0000", 2024-05-15T13:29:59.864 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.180723742, 2024-05-15T13:29:59.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00019451200000000001, 2024-05-15T13:29:59.864 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-05-15T13:29:59.864 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-05-15T13:29:59.864 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-05-15T13:29:59.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-05-15T13:29:59.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.558501+0000", 2024-05-15T13:29:59.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-05-15T13:29:59.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.558501+0000", 2024-05-15T13:29:59.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-05-15T13:29:59.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.865 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-05-15T13:29:59.865 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.558503+0000", 2024-05-15T13:29:59.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1169999999999998e-06 2024-05-15T13:29:59.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-05-15T13:29:59.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.558517+0000", 2024-05-15T13:29:59.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3913e-05 2024-05-15T13:29:59.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-05-15T13:29:59.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.558554+0000", 2024-05-15T13:29:59.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6916999999999999e-05 2024-05-15T13:29:59.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-05-15T13:29:59.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.558571+0000", 2024-05-15T13:29:59.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7158000000000001e-05 2024-05-15T13:29:59.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-05-15T13:29:59.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.558638+0000", 2024-05-15T13:29:59.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7167000000000003e-05 2024-05-15T13:29:59.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-05-15T13:29:59.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.558654+0000", 2024-05-15T13:29:59.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6463000000000001e-05 2024-05-15T13:29:59.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-05-15T13:29:59.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-05-15T13:29:59.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-05-15T13:29:59.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-05-15T13:29:49.558695+0000", 2024-05-15T13:29:59.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0776999999999999e-05 2024-05-15T13:29:59.868 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.868 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:59.868 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.868 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-05-15T13:29:59.868 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-05-15T13:29:59.868 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:29:59.870 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:29:59.870 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:30:00.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:00 smithi177 ceph-mon[44303]: pgmap v2498: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:30:00.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:00 smithi177 ceph-mon[44303]: overall HEALTH_OK 2024-05-15T13:30:00.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:00 smithi027 ceph-mon[39795]: pgmap v2498: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:30:00.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:00 smithi027 ceph-mon[39795]: overall HEALTH_OK 2024-05-15T13:30:00.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:00 smithi120 ceph-mon[43961]: pgmap v2498: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:30:00.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:00 smithi120 ceph-mon[43961]: overall HEALTH_OK 2024-05-15T13:30:00.567 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2498,"stamp":"2024-05-15T13:29:58.795642+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":9173516,"kb_used_data":25924,"kb_used_omap":17,"kb_used_meta":9147438,"kb_avail":1022023156,"statfs":{"total":1055945392128,"available":1046551711744,"internally_reserved":0,"allocated":26546176,"data_stored":17540966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17546,"internal_metadata":9366977398},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-3,"num_read_kb":-3,"num_write":-4,"num_write_kb":-67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":-84148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.415302"},"pg_stats":[{"pgid":"1.1e","version":"94'139","reported_seq":1139,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.153607+0000","last_change":"2024-05-15T13:29:11.559382+0000","last_active":"2024-05-15T13:29:54.153607+0000","last_peered":"2024-05-15T13:29:54.153607+0000","last_clean":"2024-05-15T13:29:54.153607+0000","last_became_active":"2024-05-15T13:28:01.363009+0000","last_became_peered":"2024-05-15T13:28:01.363009+0000","last_unstale":"2024-05-15T13:29:54.153607+0000","last_undegraded":"2024-05-15T13:29:54.153607+0000","last_fullsized":"2024-05-15T13:29:54.153607+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:11.559293+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:29:11.559293+0000","last_clean_scrub_stamp":"2024-05-15T13:29:11.559293+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:18.127074+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"94'139","reported_seq":1033,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.153556+0000","last_change":"2024-05-15T13:29:47.555278+0000","last_active":"2024-05-15T13:29:54.153556+0000","last_peered":"2024-05-15T13:29:54.153556+0000","last_clean":"2024-05-15T13:29:54.153556+0000","last_became_active":"2024-05-15T13:28:01.353254+0000","last_became_peered":"2024-05-15T13:28:01.353254+0000","last_unstale":"2024-05-15T13:29:54.153556+0000","last_undegraded":"2024-05-15T13:29:54.153556+0000","last_fullsized":"2024-05-15T13:29:54.153556+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:47.555173+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:23:03.856199+0000","last_clean_scrub_stamp":"2024-05-15T13:29:47.555173+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:49.032312+0000","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"94'139","reported_seq":896,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.259117+0000","last_change":"2024-05-15T13:29:44.708851+0000","last_active":"2024-05-15T13:29:54.259117+0000","last_peered":"2024-05-15T13:29:54.259117+0000","last_clean":"2024-05-15T13:29:54.259117+0000","last_became_active":"2024-05-15T13:28:01.366096+0000","last_became_peered":"2024-05-15T13:28:01.366096+0000","last_unstale":"2024-05-15T13:29:54.259117+0000","last_undegraded":"2024-05-15T13:29:54.259117+0000","last_fullsized":"2024-05-15T13:29:54.259117+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:44.708741+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:29:44.708741+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:51.133096+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,11],"acting":[8,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"94'139","reported_seq":1297,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.258829+0000","last_change":"2024-05-15T13:29:46.810878+0000","last_active":"2024-05-15T13:29:54.258829+0000","last_peered":"2024-05-15T13:29:54.258829+0000","last_clean":"2024-05-15T13:29:54.258829+0000","last_became_active":"2024-05-15T13:27:32.380219+0000","last_became_peered":"2024-05-15T13:27:32.380219+0000","last_unstale":"2024-05-15T13:29:54.258829+0000","last_undegraded":"2024-05-15T13:29:54.258829+0000","last_fullsized":"2024-05-15T13:29:54.258829+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:46.810775+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:29:46.810775+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:54.360083+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"94'139","reported_seq":1339,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.154003+0000","last_change":"2024-05-15T13:29:49.560177+0000","last_active":"2024-05-15T13:29:54.154003+0000","last_peered":"2024-05-15T13:29:54.154003+0000","last_clean":"2024-05-15T13:29:54.154003+0000","last_became_active":"2024-05-15T13:28:01.357553+0000","last_became_peered":"2024-05-15T13:28:01.357553+0000","last_unstale":"2024-05-15T13:29:54.154003+0000","last_undegraded":"2024-05-15T13:29:54.154003+0000","last_fullsized":"2024-05-15T13:29:54.154003+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:49.560066+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:22:03.418345+0000","last_clean_scrub_stamp":"2024-05-15T13:29:49.560066+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:31:18.793201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,11],"acting":[4,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"94'139","reported_seq":1124,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.258697+0000","last_change":"2024-05-15T13:29:45.745089+0000","last_active":"2024-05-15T13:29:54.258697+0000","last_peered":"2024-05-15T13:29:54.258697+0000","last_clean":"2024-05-15T13:29:54.258697+0000","last_became_active":"2024-05-15T13:24:11.374739+0000","last_became_peered":"2024-05-15T13:24:11.374739+0000","last_unstale":"2024-05-15T13:29:54.258697+0000","last_undegraded":"2024-05-15T13:29:54.258697+0000","last_fullsized":"2024-05-15T13:29:54.258697+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:45.744986+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:29:45.744986+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:31:00.748332+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"94'139","reported_seq":1946,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.151708+0000","last_change":"2024-05-15T13:29:46.768771+0000","last_active":"2024-05-15T13:29:54.151708+0000","last_peered":"2024-05-15T13:29:54.151708+0000","last_clean":"2024-05-15T13:29:54.151708+0000","last_became_active":"2024-05-15T13:24:11.368347+0000","last_became_peered":"2024-05-15T13:24:11.368347+0000","last_unstale":"2024-05-15T13:29:54.151708+0000","last_undegraded":"2024-05-15T13:29:54.151708+0000","last_fullsized":"2024-05-15T13:29:54.151708+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:46.768629+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:29:46.768629+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:31:04.930151+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"94'139","reported_seq":680,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.258734+0000","last_change":"2024-05-15T13:29:47.808074+0000","last_active":"2024-05-15T13:29:54.258734+0000","last_peered":"2024-05-15T13:29:54.258734+0000","last_clean":"2024-05-15T13:29:54.258734+0000","last_became_active":"2024-05-15T13:27:32.410771+0000","last_became_peered":"2024-05-15T13:27:32.410771+0000","last_unstale":"2024-05-15T13:29:54.258734+0000","last_undegraded":"2024-05-15T13:29:54.258734+0000","last_fullsized":"2024-05-15T13:29:54.258734+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:47.807948+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:29:47.807948+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:31:01.998503+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"94'139","reported_seq":1874,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.259203+0000","last_change":"2024-05-15T13:29:48.659515+0000","last_active":"2024-05-15T13:29:54.259203+0000","last_peered":"2024-05-15T13:29:54.259203+0000","last_clean":"2024-05-15T13:29:54.259203+0000","last_became_active":"2024-05-15T13:28:01.356182+0000","last_became_peered":"2024-05-15T13:28:01.356182+0000","last_unstale":"2024-05-15T13:29:54.259203+0000","last_undegraded":"2024-05-15T13:29:54.259203+0000","last_fullsized":"2024-05-15T13:29:54.259203+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:48.659406+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:29:48.659406+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:55.624977+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,11],"acting":[0,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"94'139","reported_seq":877,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.259501+0000","last_change":"2024-05-15T13:29:01.707292+0000","last_active":"2024-05-15T13:29:54.259501+0000","last_peered":"2024-05-15T13:29:54.259501+0000","last_clean":"2024-05-15T13:29:54.259501+0000","last_became_active":"2024-05-15T13:27:32.397907+0000","last_became_peered":"2024-05-15T13:27:32.397907+0000","last_unstale":"2024-05-15T13:29:54.259501+0000","last_undegraded":"2024-05-15T13:29:54.259501+0000","last_fullsized":"2024-05-15T13:29:54.259501+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:01.707172+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:18:24.970285+0000","last_clean_scrub_stamp":"2024-05-15T13:29:01.707172+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:27.496391+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"94'139","reported_seq":1392,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.259129+0000","last_change":"2024-05-15T13:29:53.302523+0000","last_active":"2024-05-15T13:29:54.259129+0000","last_peered":"2024-05-15T13:29:54.259129+0000","last_clean":"2024-05-15T13:29:54.259129+0000","last_became_active":"2024-05-15T13:19:41.190796+0000","last_became_peered":"2024-05-15T13:19:41.190796+0000","last_unstale":"2024-05-15T13:29:54.259129+0000","last_undegraded":"2024-05-15T13:29:54.259129+0000","last_fullsized":"2024-05-15T13:29:54.259129+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:53.302388+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:23:11.573492+0000","last_clean_scrub_stamp":"2024-05-15T13:29:53.302388+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:31:13.355555+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":1948,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.259302+0000","last_change":"2024-05-15T13:28:48.463286+0000","last_active":"2024-05-15T13:29:54.259302+0000","last_peered":"2024-05-15T13:29:54.259302+0000","last_clean":"2024-05-15T13:29:54.259302+0000","last_became_active":"2024-05-15T13:28:01.370095+0000","last_became_peered":"2024-05-15T13:28:01.370095+0000","last_unstale":"2024-05-15T13:29:54.259302+0000","last_undegraded":"2024-05-15T13:29:54.259302+0000","last_fullsized":"2024-05-15T13:29:54.259302+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:28:48.463167+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:28:48.463167+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:29:59.226265+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"94'139","reported_seq":1659,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.153532+0000","last_change":"2024-05-15T13:29:13.552444+0000","last_active":"2024-05-15T13:29:54.153532+0000","last_peered":"2024-05-15T13:29:54.153532+0000","last_clean":"2024-05-15T13:29:54.153532+0000","last_became_active":"2024-05-15T13:28:01.357970+0000","last_became_peered":"2024-05-15T13:28:01.357970+0000","last_unstale":"2024-05-15T13:29:54.153532+0000","last_undegraded":"2024-05-15T13:29:54.153532+0000","last_fullsized":"2024-05-15T13:29:54.153532+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:13.552305+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:24:35.178049+0000","last_clean_scrub_stamp":"2024-05-15T13:29:13.552305+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:21.901694+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"94'139","reported_seq":1692,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.258981+0000","last_change":"2024-05-15T13:29:43.815139+0000","last_active":"2024-05-15T13:29:54.258981+0000","last_peered":"2024-05-15T13:29:54.258981+0000","last_clean":"2024-05-15T13:29:54.258981+0000","last_became_active":"2024-05-15T13:24:06.307598+0000","last_became_peered":"2024-05-15T13:24:06.307598+0000","last_unstale":"2024-05-15T13:29:54.258981+0000","last_undegraded":"2024-05-15T13:29:54.258981+0000","last_fullsized":"2024-05-15T13:29:54.258981+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:43.815033+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:18:24.970285+0000","last_clean_scrub_stamp":"2024-05-15T13:29:43.815033+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:31:01.907870+0000","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"94'139","reported_seq":1956,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.259247+0000","last_change":"2024-05-15T13:29:01.128994+0000","last_active":"2024-05-15T13:29:54.259247+0000","last_peered":"2024-05-15T13:29:54.259247+0000","last_clean":"2024-05-15T13:29:54.259247+0000","last_became_active":"2024-05-15T13:27:32.378350+0000","last_became_peered":"2024-05-15T13:27:32.378350+0000","last_unstale":"2024-05-15T13:29:54.259247+0000","last_undegraded":"2024-05-15T13:29:54.259247+0000","last_fullsized":"2024-05-15T13:29:54.259247+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:01.128890+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:29:01.128890+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:13.820028+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,8],"acting":[2,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"94'139","reported_seq":1705,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.195662+0000","last_change":"2024-05-15T13:28:53.019558+0000","last_active":"2024-05-15T13:29:54.195662+0000","last_peered":"2024-05-15T13:29:54.195662+0000","last_clean":"2024-05-15T13:29:54.195662+0000","last_became_active":"2024-05-15T13:24:06.312706+0000","last_became_peered":"2024-05-15T13:24:06.312706+0000","last_unstale":"2024-05-15T13:29:54.195662+0000","last_undegraded":"2024-05-15T13:29:54.195662+0000","last_fullsized":"2024-05-15T13:29:54.195662+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:28:53.019440+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:28:53.019440+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:14.627453+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"94'139","reported_seq":1705,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.258865+0000","last_change":"2024-05-15T13:29:44.140963+0000","last_active":"2024-05-15T13:29:54.258865+0000","last_peered":"2024-05-15T13:29:54.258865+0000","last_clean":"2024-05-15T13:29:54.258865+0000","last_became_active":"2024-05-15T13:15:43.614970+0000","last_became_peered":"2024-05-15T13:15:43.614970+0000","last_unstale":"2024-05-15T13:29:54.258865+0000","last_undegraded":"2024-05-15T13:29:54.258865+0000","last_fullsized":"2024-05-15T13:29:54.258865+0000","mapping_epoch":678,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:44.140861+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:29:44.140861+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:31:05.284727+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,0],"acting":[9,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.2","version":"94'139","reported_seq":1967,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.259513+0000","last_change":"2024-05-15T13:29:03.616520+0000","last_active":"2024-05-15T13:29:54.259513+0000","last_peered":"2024-05-15T13:29:54.259513+0000","last_clean":"2024-05-15T13:29:54.259513+0000","last_became_active":"2024-05-15T13:25:58.961079+0000","last_became_peered":"2024-05-15T13:25:58.961079+0000","last_unstale":"2024-05-15T13:29:54.259513+0000","last_undegraded":"2024-05-15T13:29:54.259513+0000","last_fullsized":"2024-05-15T13:29:54.259513+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:03.616403+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:29:03.616403+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:08.670665+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"94'139","reported_seq":1524,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.195576+0000","last_change":"2024-05-15T13:29:12.170940+0000","last_active":"2024-05-15T13:29:54.195576+0000","last_peered":"2024-05-15T13:29:54.195576+0000","last_clean":"2024-05-15T13:29:54.195576+0000","last_became_active":"2024-05-15T13:27:32.402442+0000","last_became_peered":"2024-05-15T13:27:32.402442+0000","last_unstale":"2024-05-15T13:29:54.195576+0000","last_undegraded":"2024-05-15T13:29:54.195576+0000","last_fullsized":"2024-05-15T13:29:54.195576+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:12.170850+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:29:12.170850+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:31.103516+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,8],"acting":[6,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"94'139","reported_seq":1934,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.151847+0000","last_change":"2024-05-15T13:29:44.746060+0000","last_active":"2024-05-15T13:29:54.151847+0000","last_peered":"2024-05-15T13:29:54.151847+0000","last_clean":"2024-05-15T13:29:54.151847+0000","last_became_active":"2024-05-15T13:24:06.307697+0000","last_became_peered":"2024-05-15T13:24:06.307697+0000","last_unstale":"2024-05-15T13:29:54.151847+0000","last_undegraded":"2024-05-15T13:29:54.151847+0000","last_fullsized":"2024-05-15T13:29:54.151847+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:44.745969+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:29:44.745969+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:56.706313+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"94'139","reported_seq":1307,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.259360+0000","last_change":"2024-05-15T13:28:50.642772+0000","last_active":"2024-05-15T13:29:54.259360+0000","last_peered":"2024-05-15T13:29:54.259360+0000","last_clean":"2024-05-15T13:29:54.259360+0000","last_became_active":"2024-05-15T13:24:06.300367+0000","last_became_peered":"2024-05-15T13:24:06.300367+0000","last_unstale":"2024-05-15T13:29:54.259360+0000","last_undegraded":"2024-05-15T13:29:54.259360+0000","last_fullsized":"2024-05-15T13:29:54.259360+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:28:50.642667+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:28:50.642667+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:00.558703+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"94'139","reported_seq":1731,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.258756+0000","last_change":"2024-05-15T13:29:47.060397+0000","last_active":"2024-05-15T13:29:54.258756+0000","last_peered":"2024-05-15T13:29:54.258756+0000","last_clean":"2024-05-15T13:29:54.258756+0000","last_became_active":"2024-05-15T13:25:58.956902+0000","last_became_peered":"2024-05-15T13:25:58.956902+0000","last_unstale":"2024-05-15T13:29:54.258756+0000","last_undegraded":"2024-05-15T13:29:54.258756+0000","last_fullsized":"2024-05-15T13:29:54.258756+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:47.060313+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:22:03.418345+0000","last_clean_scrub_stamp":"2024-05-15T13:29:47.060313+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:31:09.315835+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.e","version":"94'139","reported_seq":1196,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.781276+0000","last_change":"2024-05-15T13:29:54.781305+0000","last_active":"2024-05-15T13:29:54.781276+0000","last_peered":"2024-05-15T13:29:54.781276+0000","last_clean":"2024-05-15T13:29:54.781276+0000","last_became_active":"2024-05-15T13:25:58.962490+0000","last_became_peered":"2024-05-15T13:25:58.962490+0000","last_unstale":"2024-05-15T13:29:54.781276+0000","last_undegraded":"2024-05-15T13:29:54.781276+0000","last_fullsized":"2024-05-15T13:29:54.781276+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:54.781189+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:23:06.819156+0000","last_clean_scrub_stamp":"2024-05-15T13:29:54.781189+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:31:15.675830+0000","scrub_duration":168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"94'139","reported_seq":1710,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.195687+0000","last_change":"2024-05-15T13:28:51.011599+0000","last_active":"2024-05-15T13:29:54.195687+0000","last_peered":"2024-05-15T13:29:54.195687+0000","last_clean":"2024-05-15T13:29:54.195687+0000","last_became_active":"2024-05-15T13:25:58.982018+0000","last_became_peered":"2024-05-15T13:25:58.982018+0000","last_unstale":"2024-05-15T13:29:54.195687+0000","last_undegraded":"2024-05-15T13:29:54.195687+0000","last_fullsized":"2024-05-15T13:29:54.195687+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:28:51.011461+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:28:51.011461+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:02.255673+0000","scrub_duration":251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"94'139","reported_seq":1357,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.259201+0000","last_change":"2024-05-15T13:29:43.516245+0000","last_active":"2024-05-15T13:29:54.259201+0000","last_peered":"2024-05-15T13:29:54.259201+0000","last_clean":"2024-05-15T13:29:54.259201+0000","last_became_active":"2024-05-15T13:24:06.308068+0000","last_became_peered":"2024-05-15T13:24:06.308068+0000","last_unstale":"2024-05-15T13:29:54.259201+0000","last_undegraded":"2024-05-15T13:29:54.259201+0000","last_fullsized":"2024-05-15T13:29:54.259201+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:43.516140+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:29:43.516140+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:47.647778+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"94'139","reported_seq":1401,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.259539+0000","last_change":"2024-05-15T13:28:53.617167+0000","last_active":"2024-05-15T13:29:54.259539+0000","last_peered":"2024-05-15T13:29:54.259539+0000","last_clean":"2024-05-15T13:29:54.259539+0000","last_became_active":"2024-05-15T13:27:32.402681+0000","last_became_peered":"2024-05-15T13:27:32.402681+0000","last_unstale":"2024-05-15T13:29:54.259539+0000","last_undegraded":"2024-05-15T13:29:54.259539+0000","last_fullsized":"2024-05-15T13:29:54.259539+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:28:53.617058+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:23:03.856199+0000","last_clean_scrub_stamp":"2024-05-15T13:28:53.617058+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:07.185251+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,8],"acting":[7,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"94'139","reported_seq":1314,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.153380+0000","last_change":"2024-05-15T13:29:09.677866+0000","last_active":"2024-05-15T13:29:54.153380+0000","last_peered":"2024-05-15T13:29:54.153380+0000","last_clean":"2024-05-15T13:29:54.153380+0000","last_became_active":"2024-05-15T13:28:01.353258+0000","last_became_peered":"2024-05-15T13:28:01.353258+0000","last_unstale":"2024-05-15T13:29:54.153380+0000","last_undegraded":"2024-05-15T13:29:54.153380+0000","last_fullsized":"2024-05-15T13:29:54.153380+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:09.677728+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:29:09.677728+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:15.946272+0000","scrub_duration":225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"94'139","reported_seq":1952,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.151712+0000","last_change":"2024-05-15T13:29:49.795759+0000","last_active":"2024-05-15T13:29:54.151712+0000","last_peered":"2024-05-15T13:29:54.151712+0000","last_clean":"2024-05-15T13:29:54.151712+0000","last_became_active":"2024-05-15T13:19:41.190923+0000","last_became_peered":"2024-05-15T13:19:41.190923+0000","last_unstale":"2024-05-15T13:29:54.151712+0000","last_undegraded":"2024-05-15T13:29:54.151712+0000","last_fullsized":"2024-05-15T13:29:54.151712+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:49.795649+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:29:49.795649+0000","last_clean_scrub_stamp":"2024-05-15T13:29:49.795649+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:56.535421+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"94'139","reported_seq":1728,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.259469+0000","last_change":"2024-05-15T13:29:49.687124+0000","last_active":"2024-05-15T13:29:54.259469+0000","last_peered":"2024-05-15T13:29:54.259469+0000","last_clean":"2024-05-15T13:29:54.259469+0000","last_became_active":"2024-05-15T13:27:32.378438+0000","last_became_peered":"2024-05-15T13:27:32.378438+0000","last_unstale":"2024-05-15T13:29:54.259469+0000","last_undegraded":"2024-05-15T13:29:54.259469+0000","last_fullsized":"2024-05-15T13:29:54.259469+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:49.687008+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:29:49.687008+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:31:11.219965+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"94'139","reported_seq":1968,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.151718+0000","last_change":"2024-05-15T13:29:47.804385+0000","last_active":"2024-05-15T13:29:54.151718+0000","last_peered":"2024-05-15T13:29:54.151718+0000","last_clean":"2024-05-15T13:29:54.151718+0000","last_became_active":"2024-05-15T13:25:15.377001+0000","last_became_peered":"2024-05-15T13:25:15.377001+0000","last_unstale":"2024-05-15T13:29:54.151718+0000","last_undegraded":"2024-05-15T13:29:54.151718+0000","last_fullsized":"2024-05-15T13:29:54.151718+0000","mapping_epoch":868,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":869,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:47.803958+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:18:06.340555+0000","last_clean_scrub_stamp":"2024-05-15T13:29:47.803958+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:31:17.272715+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"94'139","reported_seq":1261,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-05-15T13:29:54.153392+0000","last_change":"2024-05-15T13:29:10.647683+0000","last_active":"2024-05-15T13:29:54.153392+0000","last_peered":"2024-05-15T13:29:54.153392+0000","last_clean":"2024-05-15T13:29:54.153392+0000","last_became_active":"2024-05-15T13:28:01.353822+0000","last_became_peered":"2024-05-15T13:28:01.353822+0000","last_unstale":"2024-05-15T13:29:54.153392+0000","last_undegraded":"2024-05-15T13:29:54.153392+0000","last_fullsized":"2024-05-15T13:29:54.153392+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:29:10.647536+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:29:10.647536+0000","last_clean_scrub_stamp":"2024-05-15T13:29:10.647536+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:30:26.229464+0000","scrub_duration":188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,0],"acting":[11,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057175,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":935796,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":933566,"kb_avail":92809356,"statfs":{"total":95995035648,"available":95036780544,"internally_reserved":0,"allocated":2269184,"data_stored":1451057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":955972038},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351515,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":944116,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":941886,"kb_avail":92801036,"statfs":{"total":95995035648,"available":95028260864,"internally_reserved":0,"allocated":2269184,"data_stored":1451057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":964491712},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417515043,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1178740,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":1176510,"kb_avail":92566412,"statfs":{"total":95995035648,"available":94788005888,"internally_reserved":0,"allocated":2269184,"data_stored":1451057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":1204746688},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874789,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":538740,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93206412,"statfs":{"total":95995035648,"available":95443365888,"internally_reserved":0,"allocated":2269184,"data_stored":1451057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169131,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":416764,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93328388,"statfs":{"total":95995035648,"available":95568269312,"internally_reserved":0,"allocated":2473984,"data_stored":1652305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":424278474},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430769,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":795880,"kb_used_data":2524,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":92949272,"statfs":{"total":95995035648,"available":95180054528,"internally_reserved":0,"allocated":2584576,"data_stored":1771292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":812382662},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575434,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738963,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":890292,"kb_used_data":2536,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92854860,"statfs":{"total":95995035648,"available":95083376640,"internally_reserved":0,"allocated":2596864,"data_stored":1778737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":909048261},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411910,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":625024,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93120128,"statfs":{"total":95995035648,"available":95355011072,"internally_reserved":0,"allocated":2473984,"data_stored":1652305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150272,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1192948,"kb_used_data":2536,"kb_used_omap":1,"kb_used_meta":1190398,"kb_avail":92552204,"statfs":{"total":95995035648,"available":94773456896,"internally_reserved":0,"allocated":2596864,"data_stored":1778737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1218968007},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":887,"seq":3809635992115,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":697012,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93048140,"statfs":{"total":95995035648,"available":95281295360,"internally_reserved":0,"allocated":2269184,"data_stored":1451057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692406,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":958204,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":955774,"kb_avail":92786948,"statfs":{"total":95995035648,"available":95013834752,"internally_reserved":0,"allocated":2473984,"data_stored":1652305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":978713029},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:30:00.570 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-05-15T13:30:00.571 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T13:30:00.571 INFO:tasks.thrashosds.thrasher:inject_pause on osd.9 2024-05-15T13:30:00.571 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-15T13:30:00.571 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-15T13:30:00.572 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph daemon osd.9 config set filestore_inject_stall 3 2024-05-15T13:30:00.836 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.c/config 2024-05-15T13:30:01.354 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:01 smithi177 ceph-mon[44303]: from='client.17100 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:30:01.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:01 smithi027 ceph-mon[39795]: from='client.17100 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:30:01.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:01 smithi120 ceph-mon[43961]: from='client.17100 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:30:02.310 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:02 smithi177 ceph-mon[44303]: pgmap v2499: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:30:02.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:02 smithi027 ceph-mon[39795]: pgmap v2499: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:30:02.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:02 smithi120 ceph-mon[43961]: pgmap v2499: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:30:02.613 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-05-15T13:30:02.613 INFO:teuthology.orchestra.run.smithi177.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-05-15T13:30:02.613 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-05-15T13:30:03.352 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:03 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T13:30:03.352 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:03 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T13:30:03.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:03 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T13:30:03.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:03 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T13:30:03.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:03 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T13:30:03.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:03 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T13:30:04.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:04 smithi177 ceph-mon[44303]: pgmap v2500: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:30:04.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:04 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:30:04.441 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:04 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:30:04.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:04 smithi177 ceph-mon[44303]: osdmap e990: 12 total, 12 up, 11 in 2024-05-15T13:30:04.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:04 smithi177 ceph-mon[44303]: 1.9 scrub starts 2024-05-15T13:30:04.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:04 smithi177 ceph-mon[44303]: 1.9 scrub ok 2024-05-15T13:30:04.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:04 smithi027 ceph-mon[39795]: pgmap v2500: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:30:04.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:04 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:30:04.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:04 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:30:04.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:04 smithi027 ceph-mon[39795]: osdmap e990: 12 total, 12 up, 11 in 2024-05-15T13:30:04.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:04 smithi027 ceph-mon[39795]: 1.9 scrub starts 2024-05-15T13:30:04.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:04 smithi027 ceph-mon[39795]: 1.9 scrub ok 2024-05-15T13:30:04.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:04 smithi120 ceph-mon[43961]: pgmap v2500: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:30:04.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:04 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:30:04.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:04 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:30:04.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:04 smithi120 ceph-mon[43961]: osdmap e990: 12 total, 12 up, 11 in 2024-05-15T13:30:04.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:04 smithi120 ceph-mon[43961]: 1.9 scrub starts 2024-05-15T13:30:04.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:04 smithi120 ceph-mon[43961]: 1.9 scrub ok 2024-05-15T13:30:05.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:05 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:30:05.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:05 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:30:05.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:05 smithi027 ceph-mon[39795]: osdmap e991: 12 total, 12 up, 11 in 2024-05-15T13:30:05.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:05 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:30:05.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:05 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:30:05.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:05 smithi120 ceph-mon[43961]: osdmap e991: 12 total, 12 up, 11 in 2024-05-15T13:30:05.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:05 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:30:05.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:05 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:30:05.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:05 smithi177 ceph-mon[44303]: osdmap e991: 12 total, 12 up, 11 in 2024-05-15T13:30:06.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:06 smithi027 ceph-mon[39795]: pgmap v2503: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:30:06.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:06 smithi027 ceph-mon[39795]: 1.f scrub starts 2024-05-15T13:30:06.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:06 smithi027 ceph-mon[39795]: 1.f scrub ok 2024-05-15T13:30:06.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:06 smithi120 ceph-mon[43961]: pgmap v2503: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:30:06.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:06 smithi120 ceph-mon[43961]: 1.f scrub starts 2024-05-15T13:30:06.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:06 smithi120 ceph-mon[43961]: 1.f scrub ok 2024-05-15T13:30:06.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:06 smithi177 ceph-mon[44303]: pgmap v2503: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:30:06.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:06 smithi177 ceph-mon[44303]: 1.f scrub starts 2024-05-15T13:30:06.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:06 smithi177 ceph-mon[44303]: 1.f scrub ok 2024-05-15T13:30:07.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:07 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:30:07.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:07 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:30:07.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:07 smithi027 ceph-mon[39795]: osdmap e992: 12 total, 12 up, 11 in 2024-05-15T13:30:07.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:07 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:30:07.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:07 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:30:07.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:07 smithi120 ceph-mon[43961]: osdmap e992: 12 total, 12 up, 11 in 2024-05-15T13:30:07.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:07 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:30:07.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:07 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:30:07.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:07 smithi177 ceph-mon[44303]: osdmap e992: 12 total, 12 up, 11 in 2024-05-15T13:30:08.384 INFO:tasks.thrashosds.thrasher:inning osd 2024-05-15T13:30:08.384 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-05-15T13:30:08.385 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd in 1 2024-05-15T13:30:08.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:08 smithi027 ceph-mon[39795]: pgmap v2505: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:30:08.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:08 smithi120 ceph-mon[43961]: pgmap v2505: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:30:08.680 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:30:08.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:08 smithi177 ceph-mon[44303]: pgmap v2505: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:30:09.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:09 smithi027 ceph-mon[39795]: 1.d scrub starts 2024-05-15T13:30:09.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:09 smithi027 ceph-mon[39795]: 1.d scrub ok 2024-05-15T13:30:09.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:09 smithi120 ceph-mon[43961]: 1.d scrub starts 2024-05-15T13:30:09.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:09 smithi120 ceph-mon[43961]: 1.d scrub ok 2024-05-15T13:30:09.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:09 smithi177 ceph-mon[44303]: 1.d scrub starts 2024-05-15T13:30:09.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:09 smithi177 ceph-mon[44303]: 1.d scrub ok 2024-05-15T13:30:10.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:10 smithi027 ceph-mon[39795]: pgmap v2506: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:30:10.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:10 smithi027 ceph-mon[39795]: osdmap e993: 12 total, 12 up, 11 in 2024-05-15T13:30:10.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:10 smithi120 ceph-mon[43961]: pgmap v2506: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:30:10.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:10 smithi120 ceph-mon[43961]: osdmap e993: 12 total, 12 up, 11 in 2024-05-15T13:30:10.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:10 smithi177 ceph-mon[44303]: pgmap v2506: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:30:10.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:10 smithi177 ceph-mon[44303]: osdmap e993: 12 total, 12 up, 11 in 2024-05-15T13:30:11.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:10 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-a[39772]: 2024-05-15T13:30:10.648+0000 7f8daffd2700 -1 mon.a@0(leader).osd e993 definitely_dead 0 2024-05-15T13:30:11.216 INFO:teuthology.orchestra.run.smithi027.stderr:marked in osd.1. 2024-05-15T13:30:11.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:11 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:30:11.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:11 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:30:11.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:11 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1368105296' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-05-15T13:30:11.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:11 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:30:11.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:11 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:30:11.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:11 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1368105296' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-05-15T13:30:11.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:11 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:30:11.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:11 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:30:11.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:11 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1368105296' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-05-15T13:30:11.970 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-05-15T13:30:11.970 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-05-15T13:30:11.970 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd pool set .mgr pgp_num 31 2024-05-15T13:30:12.220 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:12 smithi027 ceph-mon[39795]: pgmap v2508: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:30:12.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:12 smithi027 ceph-mon[39795]: 1.2 scrub starts 2024-05-15T13:30:12.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:12 smithi027 ceph-mon[39795]: 1.2 scrub ok 2024-05-15T13:30:12.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:12 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1368105296' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-05-15T13:30:12.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:12 smithi027 ceph-mon[39795]: osdmap e994: 12 total, 12 up, 12 in 2024-05-15T13:30:12.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:12 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:30:12.221 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:12 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:30:12.254 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:30:12.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:12 smithi120 ceph-mon[43961]: pgmap v2508: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:30:12.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:12 smithi120 ceph-mon[43961]: 1.2 scrub starts 2024-05-15T13:30:12.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:12 smithi120 ceph-mon[43961]: 1.2 scrub ok 2024-05-15T13:30:12.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:12 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1368105296' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-05-15T13:30:12.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:12 smithi120 ceph-mon[43961]: osdmap e994: 12 total, 12 up, 12 in 2024-05-15T13:30:12.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:12 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:30:12.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:12 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:30:12.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:12 smithi177 ceph-mon[44303]: pgmap v2508: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-05-15T13:30:12.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:12 smithi177 ceph-mon[44303]: 1.2 scrub starts 2024-05-15T13:30:12.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:12 smithi177 ceph-mon[44303]: 1.2 scrub ok 2024-05-15T13:30:12.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:12 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1368105296' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-05-15T13:30:12.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:12 smithi177 ceph-mon[44303]: osdmap e994: 12 total, 12 up, 12 in 2024-05-15T13:30:12.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:12 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:30:12.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:12 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:30:13.477 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:13 smithi027 ceph-mon[39795]: osdmap e995: 12 total, 12 up, 12 in 2024-05-15T13:30:13.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:13 smithi120 ceph-mon[43961]: osdmap e995: 12 total, 12 up, 12 in 2024-05-15T13:30:13.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:13 smithi177 ceph-mon[44303]: osdmap e995: 12 total, 12 up, 12 in 2024-05-15T13:30:14.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:14 smithi027 ceph-mon[39795]: pgmap v2511: 31 pgs: 2 remapped+peering, 1 peering, 4 unknown, 24 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:14.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:14 smithi027 ceph-mon[39795]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-05-15T13:30:14.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:14 smithi027 ceph-mon[39795]: osdmap e996: 12 total, 12 up, 12 in 2024-05-15T13:30:14.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:14 smithi120 ceph-mon[43961]: pgmap v2511: 31 pgs: 2 remapped+peering, 1 peering, 4 unknown, 24 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:14.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:14 smithi120 ceph-mon[43961]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-05-15T13:30:14.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:14 smithi120 ceph-mon[43961]: osdmap e996: 12 total, 12 up, 12 in 2024-05-15T13:30:14.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:14 smithi177 ceph-mon[44303]: pgmap v2511: 31 pgs: 2 remapped+peering, 1 peering, 4 unknown, 24 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:14.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:14 smithi177 ceph-mon[44303]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-05-15T13:30:14.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:14 smithi177 ceph-mon[44303]: osdmap e996: 12 total, 12 up, 12 in 2024-05-15T13:30:15.245 INFO:teuthology.orchestra.run.smithi027.stderr:set pool 1 pgp_num to 31 2024-05-15T13:30:15.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:15 smithi027 ceph-mon[39795]: 1.6 scrub starts 2024-05-15T13:30:15.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:15 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:30:15.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:15 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:30:15.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:15 smithi027 ceph-mon[39795]: osdmap e997: 12 total, 12 up, 12 in 2024-05-15T13:30:15.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:15 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/624682932' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]: dispatch 2024-05-15T13:30:15.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:15 smithi027 ceph-mon[39795]: 1.6 scrub ok 2024-05-15T13:30:15.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:15 smithi120 ceph-mon[43961]: 1.6 scrub starts 2024-05-15T13:30:15.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:15 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:30:15.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:15 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:30:15.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:15 smithi120 ceph-mon[43961]: osdmap e997: 12 total, 12 up, 12 in 2024-05-15T13:30:15.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:15 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/624682932' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]: dispatch 2024-05-15T13:30:15.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:15 smithi120 ceph-mon[43961]: 1.6 scrub ok 2024-05-15T13:30:15.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:15 smithi177 ceph-mon[44303]: 1.6 scrub starts 2024-05-15T13:30:15.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:15 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:30:15.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:15 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:30:15.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:15 smithi177 ceph-mon[44303]: osdmap e997: 12 total, 12 up, 12 in 2024-05-15T13:30:15.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:15 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/624682932' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]: dispatch 2024-05-15T13:30:15.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:15 smithi177 ceph-mon[44303]: 1.6 scrub ok 2024-05-15T13:30:15.877 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-05-15T13:30:16.155 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:30:16.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:16 smithi027 ceph-mon[39795]: pgmap v2514: 31 pgs: 3 remapped+peering, 1 peering, 3 unknown, 24 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:16.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:16 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/624682932' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-05-15T13:30:16.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:16 smithi027 ceph-mon[39795]: osdmap e998: 12 total, 12 up, 12 in 2024-05-15T13:30:16.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:16 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T13:30:16.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:16 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T13:30:16.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:16 smithi120 ceph-mon[43961]: pgmap v2514: 31 pgs: 3 remapped+peering, 1 peering, 3 unknown, 24 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:16.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:16 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/624682932' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-05-15T13:30:16.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:16 smithi120 ceph-mon[43961]: osdmap e998: 12 total, 12 up, 12 in 2024-05-15T13:30:16.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:16 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T13:30:16.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:16 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T13:30:16.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:16 smithi177 ceph-mon[44303]: pgmap v2514: 31 pgs: 3 remapped+peering, 1 peering, 3 unknown, 24 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:16.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:16 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/624682932' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-05-15T13:30:16.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:16 smithi177 ceph-mon[44303]: osdmap e998: 12 total, 12 up, 12 in 2024-05-15T13:30:16.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:16 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T13:30:16.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:16 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T13:30:17.473 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:17 smithi027 ceph-mon[39795]: 1.13 scrub starts 2024-05-15T13:30:17.474 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:17 smithi027 ceph-mon[39795]: 1.13 scrub ok 2024-05-15T13:30:17.474 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:17 smithi027 ceph-mon[39795]: pgmap v2516: 31 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 1 peering, 28 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:17.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:17 smithi120 ceph-mon[43961]: 1.13 scrub starts 2024-05-15T13:30:17.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:17 smithi120 ceph-mon[43961]: 1.13 scrub ok 2024-05-15T13:30:17.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:17 smithi120 ceph-mon[43961]: pgmap v2516: 31 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 1 peering, 28 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:17.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:17 smithi177 ceph-mon[44303]: 1.13 scrub starts 2024-05-15T13:30:17.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:17 smithi177 ceph-mon[44303]: 1.13 scrub ok 2024-05-15T13:30:17.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:17 smithi177 ceph-mon[44303]: pgmap v2516: 31 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 1 peering, 28 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:18.063 INFO:teuthology.orchestra.run.smithi027.stdout:mon.a: {} 2024-05-15T13:30:18.073 INFO:teuthology.orchestra.run.smithi027.stdout:mon.c: {} 2024-05-15T13:30:18.084 INFO:teuthology.orchestra.run.smithi027.stdout:mon.b: {} 2024-05-15T13:30:18.085 INFO:teuthology.orchestra.run.smithi027.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-05-15T13:30:18.086 INFO:teuthology.orchestra.run.smithi027.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-05-15T13:30:18.086 INFO:teuthology.orchestra.run.smithi027.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-05-15T13:30:18.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:18 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:30:18.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:18 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:30:18.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:18 smithi027 ceph-mon[39795]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-05-15T13:30:18.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:18 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T13:30:18.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:18 smithi027 ceph-mon[39795]: osdmap e999: 12 total, 12 up, 12 in 2024-05-15T13:30:18.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:18 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1834521774' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T13:30:18.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:18 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:30:18.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:18 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:30:18.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:18 smithi120 ceph-mon[43961]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-05-15T13:30:18.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:18 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T13:30:18.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:18 smithi120 ceph-mon[43961]: osdmap e999: 12 total, 12 up, 12 in 2024-05-15T13:30:18.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:18 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1834521774' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T13:30:18.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:18 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:30:18.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:18 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:30:18.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:18 smithi177 ceph-mon[44303]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-05-15T13:30:18.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:18 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T13:30:18.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:18 smithi177 ceph-mon[44303]: osdmap e999: 12 total, 12 up, 12 in 2024-05-15T13:30:18.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:18 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1834521774' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T13:30:18.803 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 0 1 2024-05-15T13:30:19.080 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:30:19.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:19 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:30:19.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:19 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:30:19.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:19 smithi027 ceph-mon[39795]: osdmap e1000: 12 total, 12 up, 12 in 2024-05-15T13:30:19.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:19 smithi027 ceph-mon[39795]: pgmap v2519: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:19.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:19 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:30:19.542 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:19 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:30:19.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:19 smithi120 ceph-mon[43961]: osdmap e1000: 12 total, 12 up, 12 in 2024-05-15T13:30:19.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:19 smithi120 ceph-mon[43961]: pgmap v2519: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:19.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:19 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:30:19.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:19 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:30:19.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:19 smithi177 ceph-mon[44303]: osdmap e1000: 12 total, 12 up, 12 in 2024-05-15T13:30:19.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:19 smithi177 ceph-mon[44303]: pgmap v2519: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:20.636 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:20 smithi027 ceph-mon[39795]: osdmap e1001: 12 total, 12 up, 12 in 2024-05-15T13:30:20.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:20 smithi177 ceph-mon[44303]: osdmap e1001: 12 total, 12 up, 12 in 2024-05-15T13:30:20.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:20 smithi120 ceph-mon[43961]: osdmap e1001: 12 total, 12 up, 12 in 2024-05-15T13:30:21.397 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.0 to 1 (10000) 2024-05-15T13:30:21.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:21 smithi177 ceph-mon[44303]: 1.3 scrub starts 2024-05-15T13:30:21.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:21 smithi177 ceph-mon[44303]: 1.3 scrub ok 2024-05-15T13:30:21.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:21 smithi177 ceph-mon[44303]: pgmap v2521: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:21.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:21 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/182207828' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:30:21.762 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:21 smithi027 ceph-mon[39795]: 1.3 scrub starts 2024-05-15T13:30:21.763 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:21 smithi027 ceph-mon[39795]: 1.3 scrub ok 2024-05-15T13:30:21.763 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:21 smithi027 ceph-mon[39795]: pgmap v2521: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:21.763 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:21 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/182207828' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:30:21.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:21 smithi120 ceph-mon[43961]: 1.3 scrub starts 2024-05-15T13:30:21.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:21 smithi120 ceph-mon[43961]: 1.3 scrub ok 2024-05-15T13:30:21.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:21 smithi120 ceph-mon[43961]: pgmap v2521: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:21.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:21 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/182207828' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:30:22.108 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd primary-affinity 0 1 2024-05-15T13:30:22.392 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:30:22.649 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:22 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/182207828' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T13:30:22.649 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:22 smithi027 ceph-mon[39795]: osdmap e1002: 12 total, 12 up, 12 in 2024-05-15T13:30:22.649 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:22 smithi027 ceph-mon[39795]: 1.7 scrub starts 2024-05-15T13:30:22.649 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:22 smithi027 ceph-mon[39795]: 1.7 scrub ok 2024-05-15T13:30:22.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:22 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/182207828' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T13:30:22.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:22 smithi177 ceph-mon[44303]: osdmap e1002: 12 total, 12 up, 12 in 2024-05-15T13:30:22.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:22 smithi177 ceph-mon[44303]: 1.7 scrub starts 2024-05-15T13:30:22.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:22 smithi177 ceph-mon[44303]: 1.7 scrub ok 2024-05-15T13:30:22.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:22 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/182207828' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T13:30:22.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:22 smithi120 ceph-mon[43961]: osdmap e1002: 12 total, 12 up, 12 in 2024-05-15T13:30:22.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:22 smithi120 ceph-mon[43961]: 1.7 scrub starts 2024-05-15T13:30:22.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:22 smithi120 ceph-mon[43961]: 1.7 scrub ok 2024-05-15T13:30:23.667 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:23 smithi027 ceph-mon[39795]: pgmap v2523: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:23.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:23 smithi177 ceph-mon[44303]: pgmap v2523: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:23.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:23 smithi120 ceph-mon[43961]: pgmap v2523: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:24.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:24 smithi177 ceph-mon[44303]: 1.1e scrub starts 2024-05-15T13:30:24.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:24 smithi177 ceph-mon[44303]: 1.1e scrub ok 2024-05-15T13:30:24.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:24 smithi177 ceph-mon[44303]: osdmap e1003: 12 total, 12 up, 12 in 2024-05-15T13:30:24.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:24 smithi027 ceph-mon[39795]: 1.1e scrub starts 2024-05-15T13:30:24.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:24 smithi027 ceph-mon[39795]: 1.1e scrub ok 2024-05-15T13:30:24.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:24 smithi027 ceph-mon[39795]: osdmap e1003: 12 total, 12 up, 12 in 2024-05-15T13:30:24.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:24 smithi120 ceph-mon[43961]: 1.1e scrub starts 2024-05-15T13:30:24.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:24 smithi120 ceph-mon[43961]: 1.1e scrub ok 2024-05-15T13:30:24.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:24 smithi120 ceph-mon[43961]: osdmap e1003: 12 total, 12 up, 12 in 2024-05-15T13:30:25.415 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-05-15T13:30:25.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:25 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3964924245' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:30:25.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:25 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:30:25.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:25 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:30:25.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:25 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:30:25.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:25 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:30:25.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:25 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:30:25.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:25 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:30:25.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:25 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:30:25.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:25 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3964924245' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:30:25.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:25 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:30:25.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:25 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:30:25.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:25 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:30:25.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:25 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:30:25.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:25 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:30:25.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:25 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:30:25.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:25 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:30:25.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:25 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3964924245' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T13:30:25.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:25 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T13:30:25.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:25 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T13:30:25.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:25 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T13:30:25.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:25 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:30:25.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:25 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:30:25.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:25 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:30:25.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:25 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:30:26.106 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 1 1 2024-05-15T13:30:26.390 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:30:26.680 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3964924245' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-05-15T13:30:26.680 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: osdmap e1004: 12 total, 12 up, 12 in 2024-05-15T13:30:26.680 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3743411588,config=1059521453,health=3897446751,kv=1035976652}) 2024-05-15T13:30:26.680 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=2207597936,logm=1849357557}) 2024-05-15T13:30:26.680 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3828784962}) 2024-05-15T13:30:26.680 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=962123619}) 2024-05-15T13:30:26.680 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2162620428}) 2024-05-15T13:30:26.680 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3555711982}) 2024-05-15T13:30:26.680 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3296490706}) 2024-05-15T13:30:26.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2714131814}) 2024-05-15T13:30:26.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3242969588}) 2024-05-15T13:30:26.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3438031892}) 2024-05-15T13:30:26.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1876350762}) 2024-05-15T13:30:26.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2579330522}) 2024-05-15T13:30:26.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3291359001}) 2024-05-15T13:30:26.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4111381431}) 2024-05-15T13:30:26.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1537842641}) 2024-05-15T13:30:26.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4236582740}) 2024-05-15T13:30:26.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2148582861}) 2024-05-15T13:30:26.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=636403271}) 2024-05-15T13:30:26.681 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1846778652}) 2024-05-15T13:30:26.682 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1737682994}) 2024-05-15T13:30:26.682 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2638735323}) 2024-05-15T13:30:26.682 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2542976639}) 2024-05-15T13:30:26.682 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2310644765}) 2024-05-15T13:30:26.682 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2931522595}) 2024-05-15T13:30:26.682 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719564258}) 2024-05-15T13:30:26.682 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2906778451}) 2024-05-15T13:30:26.682 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1906790861}) 2024-05-15T13:30:26.682 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3026946210}) 2024-05-15T13:30:26.682 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3594117137}) 2024-05-15T13:30:26.682 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4046712697}) 2024-05-15T13:30:26.682 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2718431200}) 2024-05-15T13:30:26.683 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4245538346}) 2024-05-15T13:30:26.683 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2213539098}) 2024-05-15T13:30:26.683 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1383584359}) 2024-05-15T13:30:26.683 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4032755801}) 2024-05-15T13:30:26.683 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2543911988}) 2024-05-15T13:30:26.683 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=448359296}) 2024-05-15T13:30:26.683 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=269001170}) 2024-05-15T13:30:26.683 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3726111232}) 2024-05-15T13:30:26.683 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1600926213}) 2024-05-15T13:30:26.683 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=879830249}) 2024-05-15T13:30:26.683 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3818322086}) 2024-05-15T13:30:26.683 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4179618337}) 2024-05-15T13:30:26.683 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1441934373}) 2024-05-15T13:30:26.684 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=536293986}) 2024-05-15T13:30:26.684 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1901508105}) 2024-05-15T13:30:26.684 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2513189039}) 2024-05-15T13:30:26.684 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=44461367}) 2024-05-15T13:30:26.684 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2623350307}) 2024-05-15T13:30:26.684 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3406906867}) 2024-05-15T13:30:26.684 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2364742732}) 2024-05-15T13:30:26.684 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1193353674}) 2024-05-15T13:30:26.684 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3478141404}) 2024-05-15T13:30:26.684 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4163423894}) 2024-05-15T13:30:26.684 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=835504684}) 2024-05-15T13:30:26.684 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4147876796}) 2024-05-15T13:30:26.684 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3585505121}) 2024-05-15T13:30:26.684 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3969141092}) 2024-05-15T13:30:26.684 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1181928348}) 2024-05-15T13:30:26.685 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1546158549}) 2024-05-15T13:30:26.685 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1882124549}) 2024-05-15T13:30:26.685 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4105125614}) 2024-05-15T13:30:26.685 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=290555439}) 2024-05-15T13:30:26.685 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3017519971}) 2024-05-15T13:30:26.685 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1708390703}) 2024-05-15T13:30:26.685 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1448163833}) 2024-05-15T13:30:26.685 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3091820379}) 2024-05-15T13:30:26.685 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=655461036}) 2024-05-15T13:30:26.685 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=284316238}) 2024-05-15T13:30:26.685 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3685794362}) 2024-05-15T13:30:26.685 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2846394306}) 2024-05-15T13:30:26.685 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3886962996}) 2024-05-15T13:30:26.685 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2876152109}) 2024-05-15T13:30:26.686 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1997380932}) 2024-05-15T13:30:26.686 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3831034936}) 2024-05-15T13:30:26.686 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3984417917}) 2024-05-15T13:30:26.686 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1616304705}) 2024-05-15T13:30:26.686 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3629548111}) 2024-05-15T13:30:26.686 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=628712242}) 2024-05-15T13:30:26.686 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2476776664}) 2024-05-15T13:30:26.686 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2618376024}) 2024-05-15T13:30:26.686 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2276395741}) 2024-05-15T13:30:26.686 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {logm=17,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=16,mon_config_key=43} crc {logm=874561438,mdsmap=3984425268,mgr=76093979,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=543308118,mon_config_key=4116137559}) 2024-05-15T13:30:26.686 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=687949025}) 2024-05-15T13:30:26.686 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2806157720}) 2024-05-15T13:30:26.686 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=32,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=50} crc {mon_config_key=3795027177,monmap=2435304556,osd_metadata=2987740824,osd_pg_creating=2973052468,osdmap=1520884268}) 2024-05-15T13:30:26.687 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=133776063}) 2024-05-15T13:30:26.687 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2431737174}) 2024-05-15T13:30:26.687 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:26 smithi027 ceph-mon[39795]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=19} crc {osdmap=1546563216}) 2024-05-15T13:30:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3964924245' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-05-15T13:30:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: osdmap e1004: 12 total, 12 up, 12 in 2024-05-15T13:30:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3743411588,config=1059521453,health=3897446751,kv=1035976652}) 2024-05-15T13:30:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=2207597936,logm=1849357557}) 2024-05-15T13:30:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3828784962}) 2024-05-15T13:30:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=962123619}) 2024-05-15T13:30:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2162620428}) 2024-05-15T13:30:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3555711982}) 2024-05-15T13:30:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3296490706}) 2024-05-15T13:30:26.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2714131814}) 2024-05-15T13:30:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3242969588}) 2024-05-15T13:30:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3438031892}) 2024-05-15T13:30:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1876350762}) 2024-05-15T13:30:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2579330522}) 2024-05-15T13:30:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3291359001}) 2024-05-15T13:30:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4111381431}) 2024-05-15T13:30:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1537842641}) 2024-05-15T13:30:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4236582740}) 2024-05-15T13:30:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2148582861}) 2024-05-15T13:30:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=636403271}) 2024-05-15T13:30:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1846778652}) 2024-05-15T13:30:26.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1737682994}) 2024-05-15T13:30:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2638735323}) 2024-05-15T13:30:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2542976639}) 2024-05-15T13:30:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2310644765}) 2024-05-15T13:30:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2931522595}) 2024-05-15T13:30:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719564258}) 2024-05-15T13:30:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2906778451}) 2024-05-15T13:30:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1906790861}) 2024-05-15T13:30:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3026946210}) 2024-05-15T13:30:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3594117137}) 2024-05-15T13:30:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4046712697}) 2024-05-15T13:30:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2718431200}) 2024-05-15T13:30:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4245538346}) 2024-05-15T13:30:26.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2213539098}) 2024-05-15T13:30:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1383584359}) 2024-05-15T13:30:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4032755801}) 2024-05-15T13:30:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2543911988}) 2024-05-15T13:30:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=448359296}) 2024-05-15T13:30:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=269001170}) 2024-05-15T13:30:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3726111232}) 2024-05-15T13:30:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1600926213}) 2024-05-15T13:30:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=879830249}) 2024-05-15T13:30:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3818322086}) 2024-05-15T13:30:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4179618337}) 2024-05-15T13:30:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1441934373}) 2024-05-15T13:30:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=536293986}) 2024-05-15T13:30:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1901508105}) 2024-05-15T13:30:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2513189039}) 2024-05-15T13:30:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=44461367}) 2024-05-15T13:30:26.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2623350307}) 2024-05-15T13:30:26.696 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3406906867}) 2024-05-15T13:30:26.696 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2364742732}) 2024-05-15T13:30:26.696 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1193353674}) 2024-05-15T13:30:26.696 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3478141404}) 2024-05-15T13:30:26.696 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4163423894}) 2024-05-15T13:30:26.696 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=835504684}) 2024-05-15T13:30:26.696 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4147876796}) 2024-05-15T13:30:26.696 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3585505121}) 2024-05-15T13:30:26.696 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3969141092}) 2024-05-15T13:30:26.696 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1181928348}) 2024-05-15T13:30:26.696 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1546158549}) 2024-05-15T13:30:26.696 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1882124549}) 2024-05-15T13:30:26.696 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4105125614}) 2024-05-15T13:30:26.696 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=290555439}) 2024-05-15T13:30:26.696 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3017519971}) 2024-05-15T13:30:26.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1708390703}) 2024-05-15T13:30:26.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1448163833}) 2024-05-15T13:30:26.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3091820379}) 2024-05-15T13:30:26.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=655461036}) 2024-05-15T13:30:26.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=284316238}) 2024-05-15T13:30:26.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3685794362}) 2024-05-15T13:30:26.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2846394306}) 2024-05-15T13:30:26.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3886962996}) 2024-05-15T13:30:26.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2876152109}) 2024-05-15T13:30:26.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1997380932}) 2024-05-15T13:30:26.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3831034936}) 2024-05-15T13:30:26.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3984417917}) 2024-05-15T13:30:26.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1616304705}) 2024-05-15T13:30:26.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3629548111}) 2024-05-15T13:30:26.698 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=628712242}) 2024-05-15T13:30:26.698 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2476776664}) 2024-05-15T13:30:26.698 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2618376024}) 2024-05-15T13:30:26.698 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2276395741}) 2024-05-15T13:30:26.698 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {logm=17,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=16,mon_config_key=43} crc {logm=874561438,mdsmap=3984425268,mgr=76093979,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=543308118,mon_config_key=4116137559}) 2024-05-15T13:30:26.698 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=687949025}) 2024-05-15T13:30:26.698 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2806157720}) 2024-05-15T13:30:26.698 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=32,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=50} crc {mon_config_key=3795027177,monmap=2435304556,osd_metadata=2987740824,osd_pg_creating=2973052468,osdmap=1520884268}) 2024-05-15T13:30:26.698 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=133776063}) 2024-05-15T13:30:26.698 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2431737174}) 2024-05-15T13:30:26.698 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:26 smithi177 ceph-mon[44303]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=19} crc {osdmap=1546563216}) 2024-05-15T13:30:26.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3964924245' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-05-15T13:30:26.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: osdmap e1004: 12 total, 12 up, 12 in 2024-05-15T13:30:26.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3743411588,config=1059521453,health=3897446751,kv=1035976652}) 2024-05-15T13:30:26.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=2207597936,logm=1849357557}) 2024-05-15T13:30:26.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3828784962}) 2024-05-15T13:30:26.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=962123619}) 2024-05-15T13:30:26.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2162620428}) 2024-05-15T13:30:26.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3555711982}) 2024-05-15T13:30:26.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3296490706}) 2024-05-15T13:30:26.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2714131814}) 2024-05-15T13:30:26.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3242969588}) 2024-05-15T13:30:26.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3438031892}) 2024-05-15T13:30:26.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1876350762}) 2024-05-15T13:30:26.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2579330522}) 2024-05-15T13:30:26.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3291359001}) 2024-05-15T13:30:26.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4111381431}) 2024-05-15T13:30:26.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1537842641}) 2024-05-15T13:30:26.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4236582740}) 2024-05-15T13:30:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2148582861}) 2024-05-15T13:30:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=636403271}) 2024-05-15T13:30:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1846778652}) 2024-05-15T13:30:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1737682994}) 2024-05-15T13:30:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2638735323}) 2024-05-15T13:30:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2542976639}) 2024-05-15T13:30:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2310644765}) 2024-05-15T13:30:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2931522595}) 2024-05-15T13:30:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719564258}) 2024-05-15T13:30:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2906778451}) 2024-05-15T13:30:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1906790861}) 2024-05-15T13:30:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3026946210}) 2024-05-15T13:30:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3594117137}) 2024-05-15T13:30:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4046712697}) 2024-05-15T13:30:26.795 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2718431200}) 2024-05-15T13:30:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4245538346}) 2024-05-15T13:30:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2213539098}) 2024-05-15T13:30:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1383584359}) 2024-05-15T13:30:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4032755801}) 2024-05-15T13:30:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2543911988}) 2024-05-15T13:30:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=448359296}) 2024-05-15T13:30:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=269001170}) 2024-05-15T13:30:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3726111232}) 2024-05-15T13:30:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1600926213}) 2024-05-15T13:30:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=879830249}) 2024-05-15T13:30:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3818322086}) 2024-05-15T13:30:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4179618337}) 2024-05-15T13:30:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1441934373}) 2024-05-15T13:30:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=536293986}) 2024-05-15T13:30:26.796 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1901508105}) 2024-05-15T13:30:26.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2513189039}) 2024-05-15T13:30:26.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=44461367}) 2024-05-15T13:30:26.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2623350307}) 2024-05-15T13:30:26.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3406906867}) 2024-05-15T13:30:26.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2364742732}) 2024-05-15T13:30:26.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1193353674}) 2024-05-15T13:30:26.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3478141404}) 2024-05-15T13:30:26.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4163423894}) 2024-05-15T13:30:26.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=835504684}) 2024-05-15T13:30:26.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4147876796}) 2024-05-15T13:30:26.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3585505121}) 2024-05-15T13:30:26.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3969141092}) 2024-05-15T13:30:26.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1181928348}) 2024-05-15T13:30:26.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1546158549}) 2024-05-15T13:30:26.797 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1882124549}) 2024-05-15T13:30:26.798 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4105125614}) 2024-05-15T13:30:26.798 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=290555439}) 2024-05-15T13:30:26.798 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3017519971}) 2024-05-15T13:30:26.798 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1708390703}) 2024-05-15T13:30:26.798 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1448163833}) 2024-05-15T13:30:26.798 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3091820379}) 2024-05-15T13:30:26.798 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=655461036}) 2024-05-15T13:30:26.798 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=284316238}) 2024-05-15T13:30:26.798 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3685794362}) 2024-05-15T13:30:26.798 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2846394306}) 2024-05-15T13:30:26.798 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3886962996}) 2024-05-15T13:30:26.798 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2876152109}) 2024-05-15T13:30:26.798 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1997380932}) 2024-05-15T13:30:26.799 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3831034936}) 2024-05-15T13:30:26.799 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3984417917}) 2024-05-15T13:30:26.799 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1616304705}) 2024-05-15T13:30:26.799 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3629548111}) 2024-05-15T13:30:26.799 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=628712242}) 2024-05-15T13:30:26.799 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2476776664}) 2024-05-15T13:30:26.799 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2618376024}) 2024-05-15T13:30:26.799 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2276395741}) 2024-05-15T13:30:26.799 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {logm=17,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=16,mon_config_key=43} crc {logm=874561438,mdsmap=3984425268,mgr=76093979,mgr_command_descs=708342470,mgr_metadata=4034416921,mgrstat=543308118,mon_config_key=4116137559}) 2024-05-15T13:30:26.799 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=687949025}) 2024-05-15T13:30:26.799 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2806157720}) 2024-05-15T13:30:26.799 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=32,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=50} crc {mon_config_key=3795027177,monmap=2435304556,osd_metadata=2987740824,osd_pg_creating=2973052468,osdmap=1520884268}) 2024-05-15T13:30:26.799 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=133776063}) 2024-05-15T13:30:26.799 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2431737174}) 2024-05-15T13:30:26.800 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:26 smithi120 ceph-mon[43961]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=19} crc {osdmap=1546563216}) 2024-05-15T13:30:27.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:27 smithi027 ceph-mon[39795]: pgmap v2527: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:27.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:27 smithi177 ceph-mon[44303]: pgmap v2527: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:27.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:27 smithi120 ceph-mon[43961]: pgmap v2527: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:28.428 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.1 to 1 (10000) 2024-05-15T13:30:28.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:28 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:30:28.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:28 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:30:28.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:28 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1272615212' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T13:30:28.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:28 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:30:28.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:28 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:30:28.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:28 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1272615212' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T13:30:28.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:28 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:30:28.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:28 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:30:28.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:28 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1272615212' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T13:30:29.208 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd primary-affinity 1 1 2024-05-15T13:30:29.430 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:29 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1272615212' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T13:30:29.430 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:29 smithi027 ceph-mon[39795]: osdmap e1005: 12 total, 12 up, 12 in 2024-05-15T13:30:29.430 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:29 smithi027 ceph-mon[39795]: pgmap v2529: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:29.487 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:30:29.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:29 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1272615212' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T13:30:29.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:29 smithi177 ceph-mon[44303]: osdmap e1005: 12 total, 12 up, 12 in 2024-05-15T13:30:29.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:29 smithi177 ceph-mon[44303]: pgmap v2529: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:29.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:29 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1272615212' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T13:30:29.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:29 smithi120 ceph-mon[43961]: osdmap e1005: 12 total, 12 up, 12 in 2024-05-15T13:30:29.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:29 smithi120 ceph-mon[43961]: pgmap v2529: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:31.862 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-05-15T13:30:32.170 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:31 smithi027 ceph-mon[39795]: pgmap v2530: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:32.171 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:31 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3180536176' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T13:30:32.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:31 smithi177 ceph-mon[44303]: pgmap v2530: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:32.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:31 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3180536176' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T13:30:32.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:31 smithi120 ceph-mon[43961]: pgmap v2530: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:32.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:31 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3180536176' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T13:30:32.571 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 2 1 2024-05-15T13:30:32.853 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:30:32.874 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:32 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3180536176' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-05-15T13:30:32.874 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:32 smithi027 ceph-mon[39795]: osdmap e1006: 12 total, 12 up, 12 in 2024-05-15T13:30:33.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:32 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3180536176' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-05-15T13:30:33.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:32 smithi177 ceph-mon[44303]: osdmap e1006: 12 total, 12 up, 12 in 2024-05-15T13:30:33.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:32 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3180536176' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-05-15T13:30:33.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:32 smithi120 ceph-mon[43961]: osdmap e1006: 12 total, 12 up, 12 in 2024-05-15T13:30:34.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:33 smithi177 ceph-mon[44303]: pgmap v2532: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:34.268 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:33 smithi027 ceph-mon[39795]: pgmap v2532: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:34.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:33 smithi120 ceph-mon[43961]: pgmap v2532: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:35.404 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.2 to 1 (10000) 2024-05-15T13:30:35.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:35 smithi177 ceph-mon[44303]: 1.b scrub starts 2024-05-15T13:30:35.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:35 smithi177 ceph-mon[44303]: osdmap e1007: 12 total, 12 up, 12 in 2024-05-15T13:30:35.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:35 smithi177 ceph-mon[44303]: 1.b scrub ok 2024-05-15T13:30:35.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:35 smithi177 ceph-mon[44303]: pgmap v2534: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:35.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:35 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1822686878' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:30:35.768 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:35 smithi027 ceph-mon[39795]: 1.b scrub starts 2024-05-15T13:30:35.768 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:35 smithi027 ceph-mon[39795]: osdmap e1007: 12 total, 12 up, 12 in 2024-05-15T13:30:35.768 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:35 smithi027 ceph-mon[39795]: 1.b scrub ok 2024-05-15T13:30:35.768 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:35 smithi027 ceph-mon[39795]: pgmap v2534: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:35.768 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:35 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1822686878' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:30:35.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:35 smithi120 ceph-mon[43961]: 1.b scrub starts 2024-05-15T13:30:35.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:35 smithi120 ceph-mon[43961]: osdmap e1007: 12 total, 12 up, 12 in 2024-05-15T13:30:35.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:35 smithi120 ceph-mon[43961]: 1.b scrub ok 2024-05-15T13:30:35.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:35 smithi120 ceph-mon[43961]: pgmap v2534: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:35.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:35 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1822686878' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:30:36.152 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd primary-affinity 2 1 2024-05-15T13:30:36.437 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:30:36.661 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:36 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1822686878' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T13:30:36.661 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:36 smithi027 ceph-mon[39795]: osdmap e1008: 12 total, 12 up, 12 in 2024-05-15T13:30:36.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:36 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1822686878' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T13:30:36.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:36 smithi177 ceph-mon[44303]: osdmap e1008: 12 total, 12 up, 12 in 2024-05-15T13:30:36.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:36 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1822686878' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T13:30:36.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:36 smithi120 ceph-mon[43961]: osdmap e1008: 12 total, 12 up, 12 in 2024-05-15T13:30:37.691 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:37 smithi027 ceph-mon[39795]: 1.4 scrub starts 2024-05-15T13:30:37.691 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:37 smithi027 ceph-mon[39795]: 1.4 scrub ok 2024-05-15T13:30:37.691 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:37 smithi027 ceph-mon[39795]: pgmap v2536: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:37.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:37 smithi177 ceph-mon[44303]: 1.4 scrub starts 2024-05-15T13:30:37.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:37 smithi177 ceph-mon[44303]: 1.4 scrub ok 2024-05-15T13:30:37.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:37 smithi177 ceph-mon[44303]: pgmap v2536: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:37.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:37 smithi120 ceph-mon[43961]: 1.4 scrub starts 2024-05-15T13:30:37.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:37 smithi120 ceph-mon[43961]: 1.4 scrub ok 2024-05-15T13:30:37.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:37 smithi120 ceph-mon[43961]: pgmap v2536: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:39.414 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-05-15T13:30:39.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:39 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:30:39.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:39 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:30:39.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:39 smithi177 ceph-mon[44303]: osdmap e1009: 12 total, 12 up, 12 in 2024-05-15T13:30:39.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:39 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2115978395' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:30:39.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:39 smithi177 ceph-mon[44303]: pgmap v2538: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:39.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:39 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:30:39.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:39 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:30:39.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:39 smithi027 ceph-mon[39795]: osdmap e1009: 12 total, 12 up, 12 in 2024-05-15T13:30:39.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:39 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2115978395' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:30:39.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:39 smithi027 ceph-mon[39795]: pgmap v2538: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:39.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:39 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:30:39.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:39 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:30:39.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:39 smithi120 ceph-mon[43961]: osdmap e1009: 12 total, 12 up, 12 in 2024-05-15T13:30:39.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:39 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2115978395' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T13:30:39.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:39 smithi120 ceph-mon[43961]: pgmap v2538: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:40.138 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 3 1 2024-05-15T13:30:40.419 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:30:40.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:40 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2115978395' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-05-15T13:30:40.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:40 smithi177 ceph-mon[44303]: osdmap e1010: 12 total, 12 up, 12 in 2024-05-15T13:30:40.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:40 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2115978395' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-05-15T13:30:40.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:40 smithi027 ceph-mon[39795]: osdmap e1010: 12 total, 12 up, 12 in 2024-05-15T13:30:40.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:40 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2115978395' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-05-15T13:30:40.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:40 smithi120 ceph-mon[43961]: osdmap e1010: 12 total, 12 up, 12 in 2024-05-15T13:30:41.532 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:41 smithi027 ceph-mon[39795]: pgmap v2540: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:41.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:41 smithi177 ceph-mon[44303]: pgmap v2540: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:41.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:41 smithi120 ceph-mon[43961]: pgmap v2540: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:42.430 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.3 to 1 (10000) 2024-05-15T13:30:42.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:42 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/469938390' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:30:42.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:42 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/469938390' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:30:42.792 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:42 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/469938390' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:30:43.194 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd primary-affinity 3 1 2024-05-15T13:30:43.471 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:30:43.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:43 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/469938390' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T13:30:43.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:43 smithi177 ceph-mon[44303]: osdmap e1011: 12 total, 12 up, 12 in 2024-05-15T13:30:43.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:43 smithi177 ceph-mon[44303]: pgmap v2542: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:43.715 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:43 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/469938390' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T13:30:43.716 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:43 smithi027 ceph-mon[39795]: osdmap e1011: 12 total, 12 up, 12 in 2024-05-15T13:30:43.716 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:43 smithi027 ceph-mon[39795]: pgmap v2542: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:43.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:43 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/469938390' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T13:30:43.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:43 smithi120 ceph-mon[43961]: osdmap e1011: 12 total, 12 up, 12 in 2024-05-15T13:30:43.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:43 smithi120 ceph-mon[43961]: pgmap v2542: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:45.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:45 smithi177 ceph-mon[44303]: 1.15 deep-scrub starts 2024-05-15T13:30:45.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:45 smithi177 ceph-mon[44303]: 1.15 deep-scrub ok 2024-05-15T13:30:45.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:45 smithi177 ceph-mon[44303]: pgmap v2543: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:45.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:45 smithi027 ceph-mon[39795]: 1.15 deep-scrub starts 2024-05-15T13:30:45.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:45 smithi027 ceph-mon[39795]: 1.15 deep-scrub ok 2024-05-15T13:30:45.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:45 smithi027 ceph-mon[39795]: pgmap v2543: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:45.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:45 smithi120 ceph-mon[43961]: 1.15 deep-scrub starts 2024-05-15T13:30:45.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:45 smithi120 ceph-mon[43961]: 1.15 deep-scrub ok 2024-05-15T13:30:45.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:45 smithi120 ceph-mon[43961]: pgmap v2543: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:46.411 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-05-15T13:30:46.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:46 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/187410376' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:30:46.751 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:46 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/187410376' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:30:46.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:46 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/187410376' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T13:30:47.165 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 4 1 2024-05-15T13:30:47.449 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:30:47.679 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:47 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/187410376' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-05-15T13:30:47.679 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:47 smithi027 ceph-mon[39795]: osdmap e1012: 12 total, 12 up, 12 in 2024-05-15T13:30:47.679 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:47 smithi027 ceph-mon[39795]: pgmap v2545: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:47.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:47 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/187410376' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-05-15T13:30:47.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:47 smithi177 ceph-mon[44303]: osdmap e1012: 12 total, 12 up, 12 in 2024-05-15T13:30:47.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:47 smithi177 ceph-mon[44303]: pgmap v2545: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:47.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:47 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/187410376' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-05-15T13:30:47.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:47 smithi120 ceph-mon[43961]: osdmap e1012: 12 total, 12 up, 12 in 2024-05-15T13:30:47.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:47 smithi120 ceph-mon[43961]: pgmap v2545: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:48.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:48 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:30:48.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:48 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:30:48.723 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:48 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:30:48.723 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:48 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:30:48.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:48 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:30:48.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:48 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:30:49.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:49 smithi177 ceph-mon[44303]: osdmap e1013: 12 total, 12 up, 12 in 2024-05-15T13:30:49.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:49 smithi177 ceph-mon[44303]: pgmap v2547: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:49.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:49 smithi027 ceph-mon[39795]: osdmap e1013: 12 total, 12 up, 12 in 2024-05-15T13:30:49.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:49 smithi027 ceph-mon[39795]: pgmap v2547: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:49.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:49 smithi120 ceph-mon[43961]: osdmap e1013: 12 total, 12 up, 12 in 2024-05-15T13:30:49.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:49 smithi120 ceph-mon[43961]: pgmap v2547: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:50.443 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.4 to 1 (10000) 2024-05-15T13:30:50.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:50 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3153834119' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T13:30:50.748 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:50 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3153834119' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T13:30:50.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:50 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3153834119' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T13:30:51.161 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd primary-affinity 4 1 2024-05-15T13:30:51.442 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:30:51.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:51 smithi027 ceph-mon[39795]: 1.1d scrub starts 2024-05-15T13:30:51.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:51 smithi027 ceph-mon[39795]: 1.1d scrub ok 2024-05-15T13:30:51.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:51 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3153834119' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T13:30:51.701 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:51 smithi027 ceph-mon[39795]: osdmap e1014: 12 total, 12 up, 12 in 2024-05-15T13:30:51.702 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:51 smithi027 ceph-mon[39795]: pgmap v2549: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:51.702 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:51 smithi027 ceph-mon[39795]: 1.c scrub starts 2024-05-15T13:30:51.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:51 smithi120 ceph-mon[43961]: 1.1d scrub starts 2024-05-15T13:30:51.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:51 smithi120 ceph-mon[43961]: 1.1d scrub ok 2024-05-15T13:30:51.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:51 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3153834119' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T13:30:51.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:51 smithi120 ceph-mon[43961]: osdmap e1014: 12 total, 12 up, 12 in 2024-05-15T13:30:51.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:51 smithi120 ceph-mon[43961]: pgmap v2549: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:51.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:51 smithi120 ceph-mon[43961]: 1.c scrub starts 2024-05-15T13:30:51.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:51 smithi177 ceph-mon[44303]: 1.1d scrub starts 2024-05-15T13:30:51.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:51 smithi177 ceph-mon[44303]: 1.1d scrub ok 2024-05-15T13:30:51.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:51 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3153834119' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T13:30:51.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:51 smithi177 ceph-mon[44303]: osdmap e1014: 12 total, 12 up, 12 in 2024-05-15T13:30:51.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:51 smithi177 ceph-mon[44303]: pgmap v2549: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:51.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:51 smithi177 ceph-mon[44303]: 1.c scrub starts 2024-05-15T13:30:52.728 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:52 smithi027 ceph-mon[39795]: 1.c scrub ok 2024-05-15T13:30:52.728 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:52 smithi027 ceph-mon[39795]: 1.1c scrub starts 2024-05-15T13:30:52.729 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:52 smithi027 ceph-mon[39795]: 1.1c scrub ok 2024-05-15T13:30:52.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:52 smithi120 ceph-mon[43961]: 1.c scrub ok 2024-05-15T13:30:52.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:52 smithi120 ceph-mon[43961]: 1.1c scrub starts 2024-05-15T13:30:52.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:52 smithi120 ceph-mon[43961]: 1.1c scrub ok 2024-05-15T13:30:52.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:52 smithi177 ceph-mon[44303]: 1.c scrub ok 2024-05-15T13:30:52.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:52 smithi177 ceph-mon[44303]: 1.1c scrub starts 2024-05-15T13:30:52.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:52 smithi177 ceph-mon[44303]: 1.1c scrub ok 2024-05-15T13:30:53.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:53 smithi027 ceph-mon[39795]: 1.5 scrub starts 2024-05-15T13:30:53.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:53 smithi027 ceph-mon[39795]: 1.5 scrub ok 2024-05-15T13:30:53.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:53 smithi027 ceph-mon[39795]: 1.18 scrub starts 2024-05-15T13:30:53.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:53 smithi027 ceph-mon[39795]: 1.18 scrub ok 2024-05-15T13:30:53.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:53 smithi027 ceph-mon[39795]: pgmap v2550: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:53.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:53 smithi120 ceph-mon[43961]: 1.5 scrub starts 2024-05-15T13:30:53.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:53 smithi120 ceph-mon[43961]: 1.5 scrub ok 2024-05-15T13:30:53.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:53 smithi120 ceph-mon[43961]: 1.18 scrub starts 2024-05-15T13:30:53.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:53 smithi120 ceph-mon[43961]: 1.18 scrub ok 2024-05-15T13:30:53.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:53 smithi120 ceph-mon[43961]: pgmap v2550: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:53.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:53 smithi177 ceph-mon[44303]: 1.5 scrub starts 2024-05-15T13:30:53.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:53 smithi177 ceph-mon[44303]: 1.5 scrub ok 2024-05-15T13:30:53.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:53 smithi177 ceph-mon[44303]: 1.18 scrub starts 2024-05-15T13:30:53.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:53 smithi177 ceph-mon[44303]: 1.18 scrub ok 2024-05-15T13:30:53.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:53 smithi177 ceph-mon[44303]: pgmap v2550: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-05-15T13:30:54.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:54 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4267176095' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T13:30:54.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:54 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/4267176095' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-05-15T13:30:54.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:54 smithi027 ceph-mon[39795]: osdmap e1015: 12 total, 12 up, 12 in 2024-05-15T13:30:54.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:54 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4267176095' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T13:30:54.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:54 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/4267176095' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-05-15T13:30:54.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:54 smithi120 ceph-mon[43961]: osdmap e1015: 12 total, 12 up, 12 in 2024-05-15T13:30:54.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:54 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4267176095' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T13:30:54.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:54 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/4267176095' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-05-15T13:30:54.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:54 smithi177 ceph-mon[44303]: osdmap e1015: 12 total, 12 up, 12 in 2024-05-15T13:30:55.195 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 5 1 2024-05-15T13:30:55.460 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:55 smithi027 ceph-mon[39795]: 1.16 scrub starts 2024-05-15T13:30:55.460 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:55 smithi027 ceph-mon[39795]: 1.16 scrub ok 2024-05-15T13:30:55.460 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:55 smithi027 ceph-mon[39795]: pgmap v2552: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:55.480 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:30:55.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:55 smithi120 ceph-mon[43961]: 1.16 scrub starts 2024-05-15T13:30:55.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:55 smithi120 ceph-mon[43961]: 1.16 scrub ok 2024-05-15T13:30:55.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:55 smithi120 ceph-mon[43961]: pgmap v2552: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:55.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:55 smithi177 ceph-mon[44303]: 1.16 scrub starts 2024-05-15T13:30:55.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:55 smithi177 ceph-mon[44303]: 1.16 scrub ok 2024-05-15T13:30:55.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:55 smithi177 ceph-mon[44303]: pgmap v2552: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:57.867 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.5 to 1 (10000) 2024-05-15T13:30:58.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:57 smithi177 ceph-mon[44303]: 1.19 scrub starts 2024-05-15T13:30:58.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:57 smithi177 ceph-mon[44303]: 1.19 scrub ok 2024-05-15T13:30:58.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:57 smithi177 ceph-mon[44303]: pgmap v2553: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:58.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:57 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/486555453' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T13:30:58.218 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:57 smithi027 ceph-mon[39795]: 1.19 scrub starts 2024-05-15T13:30:58.218 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:57 smithi027 ceph-mon[39795]: 1.19 scrub ok 2024-05-15T13:30:58.218 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:57 smithi027 ceph-mon[39795]: pgmap v2553: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:58.218 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:57 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/486555453' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T13:30:58.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:57 smithi120 ceph-mon[43961]: 1.19 scrub starts 2024-05-15T13:30:58.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:57 smithi120 ceph-mon[43961]: 1.19 scrub ok 2024-05-15T13:30:58.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:57 smithi120 ceph-mon[43961]: pgmap v2553: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:30:58.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:57 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/486555453' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T13:30:58.561 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd primary-affinity 5 1 2024-05-15T13:30:58.838 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:30:59.145 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:58 smithi027 ceph-mon[39795]: 1.1a scrub starts 2024-05-15T13:30:59.145 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:58 smithi027 ceph-mon[39795]: 1.1a scrub ok 2024-05-15T13:30:59.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:58 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/486555453' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T13:30:59.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:58 smithi027 ceph-mon[39795]: osdmap e1016: 12 total, 12 up, 12 in 2024-05-15T13:30:59.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:58 smithi027 ceph-mon[39795]: 1.1b scrub starts 2024-05-15T13:30:59.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:58 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:30:59.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:58 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:30:59.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:58 smithi027 ceph-mon[39795]: 1.1b scrub ok 2024-05-15T13:30:59.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:58 smithi177 ceph-mon[44303]: 1.1a scrub starts 2024-05-15T13:30:59.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:58 smithi177 ceph-mon[44303]: 1.1a scrub ok 2024-05-15T13:30:59.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:58 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/486555453' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T13:30:59.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:58 smithi177 ceph-mon[44303]: osdmap e1016: 12 total, 12 up, 12 in 2024-05-15T13:30:59.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:58 smithi177 ceph-mon[44303]: 1.1b scrub starts 2024-05-15T13:30:59.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:58 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:30:59.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:58 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:30:59.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:58 smithi177 ceph-mon[44303]: 1.1b scrub ok 2024-05-15T13:30:59.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:58 smithi120 ceph-mon[43961]: 1.1a scrub starts 2024-05-15T13:30:59.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:58 smithi120 ceph-mon[43961]: 1.1a scrub ok 2024-05-15T13:30:59.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:58 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/486555453' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T13:30:59.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:58 smithi120 ceph-mon[43961]: osdmap e1016: 12 total, 12 up, 12 in 2024-05-15T13:30:59.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:58 smithi120 ceph-mon[43961]: 1.1b scrub starts 2024-05-15T13:30:59.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:58 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:30:59.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:58 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:30:59.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:58 smithi120 ceph-mon[43961]: 1.1b scrub ok 2024-05-15T13:31:00.014 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:59 smithi027 ceph-mon[39795]: pgmap v2555: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:00.014 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:30:59 smithi027 ceph-mon[39795]: osdmap e1017: 12 total, 12 up, 12 in 2024-05-15T13:31:00.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:59 smithi177 ceph-mon[44303]: pgmap v2555: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:00.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:30:59 smithi177 ceph-mon[44303]: osdmap e1017: 12 total, 12 up, 12 in 2024-05-15T13:31:00.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:59 smithi120 ceph-mon[43961]: pgmap v2555: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:00.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:30:59 smithi120 ceph-mon[43961]: osdmap e1017: 12 total, 12 up, 12 in 2024-05-15T13:31:00.883 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-05-15T13:31:01.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:00 smithi177 ceph-mon[44303]: 1.8 scrub starts 2024-05-15T13:31:01.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:00 smithi177 ceph-mon[44303]: 1.8 scrub ok 2024-05-15T13:31:01.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:00 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1459701314' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T13:31:01.223 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:00 smithi027 ceph-mon[39795]: 1.8 scrub starts 2024-05-15T13:31:01.223 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:00 smithi027 ceph-mon[39795]: 1.8 scrub ok 2024-05-15T13:31:01.224 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:00 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1459701314' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T13:31:01.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:00 smithi120 ceph-mon[43961]: 1.8 scrub starts 2024-05-15T13:31:01.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:00 smithi120 ceph-mon[43961]: 1.8 scrub ok 2024-05-15T13:31:01.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:00 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1459701314' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T13:31:01.591 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 6 1 2024-05-15T13:31:01.866 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:02.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:01 smithi027 ceph-mon[39795]: 1.10 scrub starts 2024-05-15T13:31:02.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:01 smithi027 ceph-mon[39795]: 1.10 scrub ok 2024-05-15T13:31:02.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:01 smithi027 ceph-mon[39795]: pgmap v2557: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:02.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:01 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1459701314' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-05-15T13:31:02.143 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:01 smithi027 ceph-mon[39795]: osdmap e1018: 12 total, 12 up, 12 in 2024-05-15T13:31:02.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:01 smithi177 ceph-mon[44303]: 1.10 scrub starts 2024-05-15T13:31:02.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:01 smithi177 ceph-mon[44303]: 1.10 scrub ok 2024-05-15T13:31:02.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:01 smithi177 ceph-mon[44303]: pgmap v2557: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:02.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:01 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1459701314' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-05-15T13:31:02.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:01 smithi177 ceph-mon[44303]: osdmap e1018: 12 total, 12 up, 12 in 2024-05-15T13:31:02.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:01 smithi120 ceph-mon[43961]: 1.10 scrub starts 2024-05-15T13:31:02.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:01 smithi120 ceph-mon[43961]: 1.10 scrub ok 2024-05-15T13:31:02.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:01 smithi120 ceph-mon[43961]: pgmap v2557: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:02.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:01 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1459701314' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-05-15T13:31:02.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:01 smithi120 ceph-mon[43961]: osdmap e1018: 12 total, 12 up, 12 in 2024-05-15T13:31:03.888 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.6 to 1 (10000) 2024-05-15T13:31:04.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:03 smithi177 ceph-mon[44303]: pgmap v2559: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:04.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:03 smithi177 ceph-mon[44303]: 1.17 scrub starts 2024-05-15T13:31:04.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:03 smithi177 ceph-mon[44303]: 1.17 scrub ok 2024-05-15T13:31:04.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:03 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/963511629' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T13:31:04.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:03 smithi027 ceph-mon[39795]: pgmap v2559: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:04.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:03 smithi027 ceph-mon[39795]: 1.17 scrub starts 2024-05-15T13:31:04.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:03 smithi027 ceph-mon[39795]: 1.17 scrub ok 2024-05-15T13:31:04.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:03 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/963511629' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T13:31:04.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:03 smithi120 ceph-mon[43961]: pgmap v2559: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:04.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:03 smithi120 ceph-mon[43961]: 1.17 scrub starts 2024-05-15T13:31:04.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:03 smithi120 ceph-mon[43961]: 1.17 scrub ok 2024-05-15T13:31:04.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:03 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/963511629' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T13:31:04.737 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd primary-affinity 6 1 2024-05-15T13:31:05.017 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:05.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:04 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/963511629' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T13:31:05.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:04 smithi027 ceph-mon[39795]: osdmap e1019: 12 total, 12 up, 12 in 2024-05-15T13:31:05.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:04 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/963511629' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T13:31:05.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:04 smithi177 ceph-mon[44303]: osdmap e1019: 12 total, 12 up, 12 in 2024-05-15T13:31:05.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:04 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/963511629' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T13:31:05.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:04 smithi120 ceph-mon[43961]: osdmap e1019: 12 total, 12 up, 12 in 2024-05-15T13:31:06.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:05 smithi177 ceph-mon[44303]: pgmap v2561: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:06.220 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:05 smithi027 ceph-mon[39795]: pgmap v2561: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:06.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:05 smithi120 ceph-mon[43961]: pgmap v2561: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:07.907 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-05-15T13:31:08.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:07 smithi177 ceph-mon[44303]: pgmap v2562: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:08.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:07 smithi177 ceph-mon[44303]: 1.1 scrub starts 2024-05-15T13:31:08.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:07 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/257868994' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T13:31:08.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:07 smithi177 ceph-mon[44303]: 1.1 scrub ok 2024-05-15T13:31:08.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:07 smithi027 ceph-mon[39795]: pgmap v2562: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:08.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:07 smithi027 ceph-mon[39795]: 1.1 scrub starts 2024-05-15T13:31:08.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:07 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/257868994' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T13:31:08.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:07 smithi027 ceph-mon[39795]: 1.1 scrub ok 2024-05-15T13:31:08.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:07 smithi120 ceph-mon[43961]: pgmap v2562: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:08.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:07 smithi120 ceph-mon[43961]: 1.1 scrub starts 2024-05-15T13:31:08.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:07 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/257868994' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T13:31:08.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:07 smithi120 ceph-mon[43961]: 1.1 scrub ok 2024-05-15T13:31:08.747 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 7 1 2024-05-15T13:31:09.030 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:09.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:08 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:31:09.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:08 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:31:09.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:08 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/257868994' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-05-15T13:31:09.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:08 smithi027 ceph-mon[39795]: osdmap e1020: 12 total, 12 up, 12 in 2024-05-15T13:31:09.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:08 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:31:09.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:08 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:31:09.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:08 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/257868994' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-05-15T13:31:09.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:08 smithi177 ceph-mon[44303]: osdmap e1020: 12 total, 12 up, 12 in 2024-05-15T13:31:09.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:08 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:31:09.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:08 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:31:09.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:08 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/257868994' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-05-15T13:31:09.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:08 smithi120 ceph-mon[43961]: osdmap e1020: 12 total, 12 up, 12 in 2024-05-15T13:31:10.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:09 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:31:10.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:09 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:31:10.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:09 smithi177 ceph-mon[44303]: pgmap v2564: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:10.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:09 smithi177 ceph-mon[44303]: osdmap e1021: 12 total, 12 up, 12 in 2024-05-15T13:31:10.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:09 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:31:10.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:09 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:31:10.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:09 smithi027 ceph-mon[39795]: pgmap v2564: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:10.284 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:09 smithi027 ceph-mon[39795]: osdmap e1021: 12 total, 12 up, 12 in 2024-05-15T13:31:10.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:09 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:31:10.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:09 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:31:10.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:09 smithi120 ceph-mon[43961]: pgmap v2564: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:10.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:09 smithi120 ceph-mon[43961]: osdmap e1021: 12 total, 12 up, 12 in 2024-05-15T13:31:11.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:10 smithi177 ceph-mon[44303]: 1.11 scrub starts 2024-05-15T13:31:11.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:10 smithi177 ceph-mon[44303]: 1.11 scrub ok 2024-05-15T13:31:11.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:10 smithi027 ceph-mon[39795]: 1.11 scrub starts 2024-05-15T13:31:11.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:10 smithi027 ceph-mon[39795]: 1.11 scrub ok 2024-05-15T13:31:11.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:10 smithi120 ceph-mon[43961]: 1.11 scrub starts 2024-05-15T13:31:11.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:10 smithi120 ceph-mon[43961]: 1.11 scrub ok 2024-05-15T13:31:11.924 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.7 to 1 (10000) 2024-05-15T13:31:12.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:11 smithi177 ceph-mon[44303]: pgmap v2566: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:12.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:11 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3788377461' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T13:31:12.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:11 smithi027 ceph-mon[39795]: pgmap v2566: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:12.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:11 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3788377461' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T13:31:12.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:11 smithi120 ceph-mon[43961]: pgmap v2566: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:12.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:11 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3788377461' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T13:31:12.774 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd primary-affinity 7 1 2024-05-15T13:31:13.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:12 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3788377461' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T13:31:13.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:12 smithi027 ceph-mon[39795]: osdmap e1022: 12 total, 12 up, 12 in 2024-05-15T13:31:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:13.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:12 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3788377461' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T13:31:13.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:12 smithi177 ceph-mon[44303]: osdmap e1022: 12 total, 12 up, 12 in 2024-05-15T13:31:13.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:12 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3788377461' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T13:31:13.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:12 smithi120 ceph-mon[43961]: osdmap e1022: 12 total, 12 up, 12 in 2024-05-15T13:31:14.171 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:13 smithi027 ceph-mon[39795]: pgmap v2568: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:14.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:13 smithi177 ceph-mon[44303]: pgmap v2568: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:14.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:13 smithi120 ceph-mon[43961]: pgmap v2568: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:15.414 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-05-15T13:31:15.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:15 smithi177 ceph-mon[44303]: pgmap v2569: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:15.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:15 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/300365575' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T13:31:15.764 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:15 smithi027 ceph-mon[39795]: pgmap v2569: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:15.764 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:15 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/300365575' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T13:31:15.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:15 smithi120 ceph-mon[43961]: pgmap v2569: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:15.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:15 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/300365575' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T13:31:16.138 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 8 1 2024-05-15T13:31:16.415 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:16.674 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:16 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/300365575' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-05-15T13:31:16.675 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:16 smithi027 ceph-mon[39795]: osdmap e1023: 12 total, 12 up, 12 in 2024-05-15T13:31:16.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:16 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/300365575' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-05-15T13:31:16.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:16 smithi177 ceph-mon[44303]: osdmap e1023: 12 total, 12 up, 12 in 2024-05-15T13:31:16.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:16 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/300365575' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-05-15T13:31:16.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:16 smithi120 ceph-mon[43961]: osdmap e1023: 12 total, 12 up, 12 in 2024-05-15T13:31:17.518 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:17 smithi027 ceph-mon[39795]: 1.d scrub starts 2024-05-15T13:31:17.518 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:17 smithi027 ceph-mon[39795]: 1.d scrub ok 2024-05-15T13:31:17.518 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:17 smithi027 ceph-mon[39795]: pgmap v2571: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:17.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:17 smithi177 ceph-mon[44303]: 1.d scrub starts 2024-05-15T13:31:17.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:17 smithi177 ceph-mon[44303]: 1.d scrub ok 2024-05-15T13:31:17.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:17 smithi177 ceph-mon[44303]: pgmap v2571: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:17.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:17 smithi120 ceph-mon[43961]: 1.d scrub starts 2024-05-15T13:31:17.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:17 smithi120 ceph-mon[43961]: 1.d scrub ok 2024-05-15T13:31:17.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:17 smithi120 ceph-mon[43961]: pgmap v2571: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:18.425 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.8 to 1 (10000) 2024-05-15T13:31:18.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:18 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:31:18.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:18 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:31:18.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:18 smithi177 ceph-mon[44303]: osdmap e1024: 12 total, 12 up, 12 in 2024-05-15T13:31:18.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:18 smithi177 ceph-mon[44303]: 1.a scrub starts 2024-05-15T13:31:18.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:18 smithi177 ceph-mon[44303]: 1.a scrub ok 2024-05-15T13:31:18.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:18 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2440199201' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T13:31:18.769 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:18 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:31:18.769 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:18 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:31:18.769 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:18 smithi027 ceph-mon[39795]: osdmap e1024: 12 total, 12 up, 12 in 2024-05-15T13:31:18.769 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:18 smithi027 ceph-mon[39795]: 1.a scrub starts 2024-05-15T13:31:18.769 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:18 smithi027 ceph-mon[39795]: 1.a scrub ok 2024-05-15T13:31:18.769 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:18 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2440199201' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T13:31:18.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:18 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:31:18.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:18 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:31:18.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:18 smithi120 ceph-mon[43961]: osdmap e1024: 12 total, 12 up, 12 in 2024-05-15T13:31:18.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:18 smithi120 ceph-mon[43961]: 1.a scrub starts 2024-05-15T13:31:18.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:18 smithi120 ceph-mon[43961]: 1.a scrub ok 2024-05-15T13:31:18.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:18 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2440199201' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T13:31:19.127 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd primary-affinity 8 1 2024-05-15T13:31:19.403 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:19.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:19 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2440199201' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T13:31:19.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:19 smithi177 ceph-mon[44303]: osdmap e1025: 12 total, 12 up, 12 in 2024-05-15T13:31:19.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:19 smithi177 ceph-mon[44303]: 1.12 scrub starts 2024-05-15T13:31:19.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:19 smithi177 ceph-mon[44303]: 1.12 scrub ok 2024-05-15T13:31:19.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:19 smithi177 ceph-mon[44303]: pgmap v2574: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:19.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:19 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2440199201' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T13:31:19.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:19 smithi027 ceph-mon[39795]: osdmap e1025: 12 total, 12 up, 12 in 2024-05-15T13:31:19.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:19 smithi027 ceph-mon[39795]: 1.12 scrub starts 2024-05-15T13:31:19.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:19 smithi027 ceph-mon[39795]: 1.12 scrub ok 2024-05-15T13:31:19.698 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:19 smithi027 ceph-mon[39795]: pgmap v2574: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:19.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:19 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2440199201' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T13:31:19.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:19 smithi120 ceph-mon[43961]: osdmap e1025: 12 total, 12 up, 12 in 2024-05-15T13:31:19.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:19 smithi120 ceph-mon[43961]: 1.12 scrub starts 2024-05-15T13:31:19.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:19 smithi120 ceph-mon[43961]: 1.12 scrub ok 2024-05-15T13:31:19.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:19 smithi120 ceph-mon[43961]: pgmap v2574: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:20.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:20 smithi177 ceph-mon[44303]: 1.e scrub starts 2024-05-15T13:31:20.692 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:20 smithi027 ceph-mon[39795]: 1.e scrub starts 2024-05-15T13:31:20.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:20 smithi120 ceph-mon[43961]: 1.e scrub starts 2024-05-15T13:31:21.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:21 smithi177 ceph-mon[44303]: 1.e scrub ok 2024-05-15T13:31:21.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:21 smithi177 ceph-mon[44303]: pgmap v2575: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:21.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:21 smithi027 ceph-mon[39795]: 1.e scrub ok 2024-05-15T13:31:21.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:21 smithi027 ceph-mon[39795]: pgmap v2575: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:21.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:21 smithi120 ceph-mon[43961]: 1.e scrub ok 2024-05-15T13:31:21.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:21 smithi120 ceph-mon[43961]: pgmap v2575: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:22.449 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-05-15T13:31:22.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:22 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3848245029' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T13:31:22.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:22 smithi177 ceph-mon[44303]: 1.14 deep-scrub starts 2024-05-15T13:31:22.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:22 smithi177 ceph-mon[44303]: 1.14 deep-scrub ok 2024-05-15T13:31:22.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:22 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3848245029' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T13:31:22.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:22 smithi027 ceph-mon[39795]: 1.14 deep-scrub starts 2024-05-15T13:31:22.776 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:22 smithi027 ceph-mon[39795]: 1.14 deep-scrub ok 2024-05-15T13:31:22.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:22 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3848245029' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T13:31:22.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:22 smithi120 ceph-mon[43961]: 1.14 deep-scrub starts 2024-05-15T13:31:22.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:22 smithi120 ceph-mon[43961]: 1.14 deep-scrub ok 2024-05-15T13:31:23.224 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 9 1 2024-05-15T13:31:23.509 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:23.711 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:23 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3848245029' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-05-15T13:31:23.711 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:23 smithi027 ceph-mon[39795]: osdmap e1026: 12 total, 12 up, 12 in 2024-05-15T13:31:23.711 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:23 smithi027 ceph-mon[39795]: pgmap v2577: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:23.711 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:23 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:31:23.711 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:23 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:31:23.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:23 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3848245029' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-05-15T13:31:23.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:23 smithi120 ceph-mon[43961]: osdmap e1026: 12 total, 12 up, 12 in 2024-05-15T13:31:23.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:23 smithi120 ceph-mon[43961]: pgmap v2577: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:23.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:23 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:31:23.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:23 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:31:23.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:23 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3848245029' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-05-15T13:31:23.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:23 smithi177 ceph-mon[44303]: osdmap e1026: 12 total, 12 up, 12 in 2024-05-15T13:31:23.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:23 smithi177 ceph-mon[44303]: pgmap v2577: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:23.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:23 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:31:23.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:23 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:31:24.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:24 smithi027 ceph-mon[39795]: osdmap e1027: 12 total, 12 up, 12 in 2024-05-15T13:31:24.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:24 smithi120 ceph-mon[43961]: osdmap e1027: 12 total, 12 up, 12 in 2024-05-15T13:31:24.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:24 smithi177 ceph-mon[44303]: osdmap e1027: 12 total, 12 up, 12 in 2024-05-15T13:31:25.457 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:25 smithi027 ceph-mon[39795]: pgmap v2579: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:25.457 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:25 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:31:25.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:25 smithi120 ceph-mon[43961]: pgmap v2579: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:25.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:25 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:31:25.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:25 smithi177 ceph-mon[44303]: pgmap v2579: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:25.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:25 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:31:26.466 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.9 to 1 (10000) 2024-05-15T13:31:26.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:26 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/465440180' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:31:26.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:26 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:31:26.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:26 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:31:26.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:26 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:31:26.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:26 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/465440180' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:31:26.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:26 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:31:26.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:26 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:31:26.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:26 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:31:26.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:26 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/465440180' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:31:26.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:26 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:31:26.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:26 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:31:26.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:26 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:31:27.195 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd primary-affinity 9 1 2024-05-15T13:31:27.467 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:27 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/465440180' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T13:31:27.467 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:27 smithi027 ceph-mon[39795]: osdmap e1028: 12 total, 12 up, 12 in 2024-05-15T13:31:27.468 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:27 smithi027 ceph-mon[39795]: pgmap v2581: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:27.476 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:27.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:27 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/465440180' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T13:31:27.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:27 smithi120 ceph-mon[43961]: osdmap e1028: 12 total, 12 up, 12 in 2024-05-15T13:31:27.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:27 smithi120 ceph-mon[43961]: pgmap v2581: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:27.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:27 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/465440180' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T13:31:27.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:27 smithi177 ceph-mon[44303]: osdmap e1028: 12 total, 12 up, 12 in 2024-05-15T13:31:27.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:27 smithi177 ceph-mon[44303]: pgmap v2581: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:29.875 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-05-15T13:31:30.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:29 smithi177 ceph-mon[44303]: pgmap v2582: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:30.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:29 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3847987903' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:31:30.262 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:29 smithi027 ceph-mon[39795]: pgmap v2582: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:30.262 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:29 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3847987903' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:31:30.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:29 smithi120 ceph-mon[43961]: pgmap v2582: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:30.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:29 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3847987903' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T13:31:30.661 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 10 1 2024-05-15T13:31:30.936 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:31.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:30 smithi027 ceph-mon[39795]: 1.9 scrub starts 2024-05-15T13:31:31.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:30 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3847987903' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-05-15T13:31:31.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:30 smithi027 ceph-mon[39795]: osdmap e1029: 12 total, 12 up, 12 in 2024-05-15T13:31:31.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:30 smithi027 ceph-mon[39795]: 1.9 scrub ok 2024-05-15T13:31:31.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:30 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:31:31.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:30 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:31:31.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:30 smithi177 ceph-mon[44303]: 1.9 scrub starts 2024-05-15T13:31:31.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:30 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3847987903' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-05-15T13:31:31.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:30 smithi177 ceph-mon[44303]: osdmap e1029: 12 total, 12 up, 12 in 2024-05-15T13:31:31.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:30 smithi177 ceph-mon[44303]: 1.9 scrub ok 2024-05-15T13:31:31.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:30 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:31:31.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:30 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:31:31.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:30 smithi120 ceph-mon[43961]: 1.9 scrub starts 2024-05-15T13:31:31.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:30 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3847987903' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-05-15T13:31:31.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:30 smithi120 ceph-mon[43961]: osdmap e1029: 12 total, 12 up, 12 in 2024-05-15T13:31:31.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:30 smithi120 ceph-mon[43961]: 1.9 scrub ok 2024-05-15T13:31:31.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:30 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:31:31.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:30 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:31:32.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:31 smithi027 ceph-mon[39795]: 1.3 scrub starts 2024-05-15T13:31:32.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:31 smithi027 ceph-mon[39795]: 1.3 scrub ok 2024-05-15T13:31:32.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:31 smithi027 ceph-mon[39795]: pgmap v2584: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:32.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:31 smithi027 ceph-mon[39795]: osdmap e1030: 12 total, 12 up, 12 in 2024-05-15T13:31:32.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:31 smithi177 ceph-mon[44303]: 1.3 scrub starts 2024-05-15T13:31:32.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:31 smithi177 ceph-mon[44303]: 1.3 scrub ok 2024-05-15T13:31:32.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:31 smithi177 ceph-mon[44303]: pgmap v2584: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:32.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:31 smithi177 ceph-mon[44303]: osdmap e1030: 12 total, 12 up, 12 in 2024-05-15T13:31:32.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:31 smithi120 ceph-mon[43961]: 1.3 scrub starts 2024-05-15T13:31:32.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:31 smithi120 ceph-mon[43961]: 1.3 scrub ok 2024-05-15T13:31:32.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:31 smithi120 ceph-mon[43961]: pgmap v2584: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:32.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:31 smithi120 ceph-mon[43961]: osdmap e1030: 12 total, 12 up, 12 in 2024-05-15T13:31:32.891 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.10 to 1 (10000) 2024-05-15T13:31:33.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:32 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/720750021' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T13:31:33.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:32 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/720750021' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T13:31:33.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:32 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/720750021' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T13:31:33.661 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd primary-affinity 10 1 2024-05-15T13:31:33.939 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:34.162 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:33 smithi027 ceph-mon[39795]: pgmap v2586: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:34.162 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:33 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/720750021' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T13:31:34.163 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:33 smithi027 ceph-mon[39795]: osdmap e1031: 12 total, 12 up, 12 in 2024-05-15T13:31:34.163 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:33 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:31:34.163 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:33 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:31:34.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:33 smithi177 ceph-mon[44303]: pgmap v2586: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:34.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:33 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/720750021' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T13:31:34.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:33 smithi177 ceph-mon[44303]: osdmap e1031: 12 total, 12 up, 12 in 2024-05-15T13:31:34.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:33 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:31:34.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:33 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:31:34.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:33 smithi120 ceph-mon[43961]: pgmap v2586: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:34.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:33 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/720750021' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T13:31:34.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:33 smithi120 ceph-mon[43961]: osdmap e1031: 12 total, 12 up, 12 in 2024-05-15T13:31:34.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:33 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:31:34.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:33 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:31:35.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:34 smithi177 ceph-mon[44303]: osdmap e1032: 12 total, 12 up, 12 in 2024-05-15T13:31:35.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:34 smithi027 ceph-mon[39795]: osdmap e1032: 12 total, 12 up, 12 in 2024-05-15T13:31:35.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:34 smithi120 ceph-mon[43961]: osdmap e1032: 12 total, 12 up, 12 in 2024-05-15T13:31:36.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:35 smithi177 ceph-mon[44303]: pgmap v2589: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:36.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:35 smithi177 ceph-mon[44303]: 1.f scrub starts 2024-05-15T13:31:36.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:35 smithi027 ceph-mon[39795]: pgmap v2589: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:36.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:35 smithi027 ceph-mon[39795]: 1.f scrub starts 2024-05-15T13:31:36.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:35 smithi120 ceph-mon[43961]: pgmap v2589: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:36.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:35 smithi120 ceph-mon[43961]: 1.f scrub starts 2024-05-15T13:31:36.908 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-05-15T13:31:37.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:36 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:31:37.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:36 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:31:37.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:36 smithi177 ceph-mon[44303]: 1.0 scrub starts 2024-05-15T13:31:37.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:36 smithi177 ceph-mon[44303]: 1.0 scrub ok 2024-05-15T13:31:37.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:36 smithi177 ceph-mon[44303]: 1.f scrub ok 2024-05-15T13:31:37.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:36 smithi177 ceph-mon[44303]: osdmap e1033: 12 total, 12 up, 12 in 2024-05-15T13:31:37.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:36 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/746846449' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T13:31:37.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:36 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:31:37.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:36 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:31:37.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:36 smithi027 ceph-mon[39795]: 1.0 scrub starts 2024-05-15T13:31:37.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:36 smithi027 ceph-mon[39795]: 1.0 scrub ok 2024-05-15T13:31:37.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:36 smithi027 ceph-mon[39795]: 1.f scrub ok 2024-05-15T13:31:37.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:36 smithi027 ceph-mon[39795]: osdmap e1033: 12 total, 12 up, 12 in 2024-05-15T13:31:37.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:36 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/746846449' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T13:31:37.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:36 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:31:37.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:36 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:31:37.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:36 smithi120 ceph-mon[43961]: 1.0 scrub starts 2024-05-15T13:31:37.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:36 smithi120 ceph-mon[43961]: 1.0 scrub ok 2024-05-15T13:31:37.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:36 smithi120 ceph-mon[43961]: 1.f scrub ok 2024-05-15T13:31:37.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:36 smithi120 ceph-mon[43961]: osdmap e1033: 12 total, 12 up, 12 in 2024-05-15T13:31:37.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:36 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/746846449' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T13:31:37.662 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd reweight 11 1 2024-05-15T13:31:37.944 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:38.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:37 smithi177 ceph-mon[44303]: 1.13 scrub starts 2024-05-15T13:31:38.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:37 smithi177 ceph-mon[44303]: 1.13 scrub ok 2024-05-15T13:31:38.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:37 smithi177 ceph-mon[44303]: pgmap v2591: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:38.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:37 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/746846449' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-05-15T13:31:38.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:37 smithi177 ceph-mon[44303]: osdmap e1034: 12 total, 12 up, 12 in 2024-05-15T13:31:38.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:37 smithi027 ceph-mon[39795]: 1.13 scrub starts 2024-05-15T13:31:38.203 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:37 smithi027 ceph-mon[39795]: 1.13 scrub ok 2024-05-15T13:31:38.204 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:37 smithi027 ceph-mon[39795]: pgmap v2591: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:38.204 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:37 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/746846449' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-05-15T13:31:38.204 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:37 smithi027 ceph-mon[39795]: osdmap e1034: 12 total, 12 up, 12 in 2024-05-15T13:31:38.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:37 smithi120 ceph-mon[43961]: 1.13 scrub starts 2024-05-15T13:31:38.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:37 smithi120 ceph-mon[43961]: 1.13 scrub ok 2024-05-15T13:31:38.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:37 smithi120 ceph-mon[43961]: pgmap v2591: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:38.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:37 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/746846449' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-05-15T13:31:38.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:37 smithi120 ceph-mon[43961]: osdmap e1034: 12 total, 12 up, 12 in 2024-05-15T13:31:39.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:38 smithi177 ceph-mon[44303]: osdmap e1035: 12 total, 12 up, 12 in 2024-05-15T13:31:39.265 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:38 smithi027 ceph-mon[39795]: osdmap e1035: 12 total, 12 up, 12 in 2024-05-15T13:31:39.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:38 smithi120 ceph-mon[43961]: osdmap e1035: 12 total, 12 up, 12 in 2024-05-15T13:31:40.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:39 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:31:40.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:39 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:31:40.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:39 smithi177 ceph-mon[44303]: 1.1e scrub starts 2024-05-15T13:31:40.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:39 smithi177 ceph-mon[44303]: 1.1e scrub ok 2024-05-15T13:31:40.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:39 smithi177 ceph-mon[44303]: pgmap v2594: 31 pgs: 2 peering, 29 active+clean; 616 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 41 KiB/s wr, 4 op/s; 50 KiB/s, 0 objects/s recovering 2024-05-15T13:31:40.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:39 smithi177 ceph-mon[44303]: 1.6 scrub starts 2024-05-15T13:31:40.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:39 smithi177 ceph-mon[44303]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-05-15T13:31:40.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:39 smithi177 ceph-mon[44303]: 1.6 scrub ok 2024-05-15T13:31:40.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:39 smithi177 ceph-mon[44303]: osdmap e1036: 12 total, 12 up, 12 in 2024-05-15T13:31:40.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:39 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:31:40.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:39 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:31:40.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:39 smithi027 ceph-mon[39795]: 1.1e scrub starts 2024-05-15T13:31:40.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:39 smithi027 ceph-mon[39795]: 1.1e scrub ok 2024-05-15T13:31:40.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:39 smithi027 ceph-mon[39795]: pgmap v2594: 31 pgs: 2 peering, 29 active+clean; 616 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 41 KiB/s wr, 4 op/s; 50 KiB/s, 0 objects/s recovering 2024-05-15T13:31:40.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:39 smithi027 ceph-mon[39795]: 1.6 scrub starts 2024-05-15T13:31:40.284 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:39 smithi027 ceph-mon[39795]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-05-15T13:31:40.284 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:39 smithi027 ceph-mon[39795]: 1.6 scrub ok 2024-05-15T13:31:40.284 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:39 smithi027 ceph-mon[39795]: osdmap e1036: 12 total, 12 up, 12 in 2024-05-15T13:31:40.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:39 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:31:40.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:39 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:31:40.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:39 smithi120 ceph-mon[43961]: 1.1e scrub starts 2024-05-15T13:31:40.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:39 smithi120 ceph-mon[43961]: 1.1e scrub ok 2024-05-15T13:31:40.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:39 smithi120 ceph-mon[43961]: pgmap v2594: 31 pgs: 2 peering, 29 active+clean; 616 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 41 KiB/s wr, 4 op/s; 50 KiB/s, 0 objects/s recovering 2024-05-15T13:31:40.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:39 smithi120 ceph-mon[43961]: 1.6 scrub starts 2024-05-15T13:31:40.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:39 smithi120 ceph-mon[43961]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-05-15T13:31:40.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:39 smithi120 ceph-mon[43961]: 1.6 scrub ok 2024-05-15T13:31:40.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:39 smithi120 ceph-mon[43961]: osdmap e1036: 12 total, 12 up, 12 in 2024-05-15T13:31:40.924 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.11 to 1 (10000) 2024-05-15T13:31:41.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:40 smithi177 ceph-mon[44303]: 1.1 deep-scrub starts 2024-05-15T13:31:41.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:40 smithi177 ceph-mon[44303]: 1.1 deep-scrub ok 2024-05-15T13:31:41.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:40 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3000174174' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T13:31:41.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:40 smithi027 ceph-mon[39795]: 1.1 deep-scrub starts 2024-05-15T13:31:41.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:40 smithi027 ceph-mon[39795]: 1.1 deep-scrub ok 2024-05-15T13:31:41.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:40 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3000174174' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T13:31:41.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:40 smithi120 ceph-mon[43961]: 1.1 deep-scrub starts 2024-05-15T13:31:41.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:40 smithi120 ceph-mon[43961]: 1.1 deep-scrub ok 2024-05-15T13:31:41.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:40 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3000174174' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T13:31:41.760 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd primary-affinity 11 1 2024-05-15T13:31:42.032 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:41 smithi027 ceph-mon[39795]: pgmap v2596: 31 pgs: 2 peering, 29 active+clean; 616 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 41 KiB/s wr, 4 op/s; 50 KiB/s, 0 objects/s recovering 2024-05-15T13:31:42.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:41 smithi027 ceph-mon[39795]: 1.2 scrub starts 2024-05-15T13:31:42.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:41 smithi027 ceph-mon[39795]: 1.2 scrub ok 2024-05-15T13:31:42.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:41 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3000174174' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T13:31:42.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:41 smithi027 ceph-mon[39795]: osdmap e1037: 12 total, 12 up, 12 in 2024-05-15T13:31:42.043 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:42.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:41 smithi177 ceph-mon[44303]: pgmap v2596: 31 pgs: 2 peering, 29 active+clean; 616 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 41 KiB/s wr, 4 op/s; 50 KiB/s, 0 objects/s recovering 2024-05-15T13:31:42.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:41 smithi177 ceph-mon[44303]: 1.2 scrub starts 2024-05-15T13:31:42.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:41 smithi177 ceph-mon[44303]: 1.2 scrub ok 2024-05-15T13:31:42.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:41 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3000174174' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T13:31:42.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:41 smithi177 ceph-mon[44303]: osdmap e1037: 12 total, 12 up, 12 in 2024-05-15T13:31:42.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:41 smithi120 ceph-mon[43961]: pgmap v2596: 31 pgs: 2 peering, 29 active+clean; 616 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 41 KiB/s wr, 4 op/s; 50 KiB/s, 0 objects/s recovering 2024-05-15T13:31:42.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:41 smithi120 ceph-mon[43961]: 1.2 scrub starts 2024-05-15T13:31:42.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:41 smithi120 ceph-mon[43961]: 1.2 scrub ok 2024-05-15T13:31:42.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:41 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3000174174' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T13:31:42.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:41 smithi120 ceph-mon[43961]: osdmap e1037: 12 total, 12 up, 12 in 2024-05-15T13:31:43.254 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:42 smithi027 ceph-mon[39795]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-05-15T13:31:43.254 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:42 smithi027 ceph-mon[39795]: Cluster is now healthy 2024-05-15T13:31:43.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:42 smithi120 ceph-mon[43961]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-05-15T13:31:43.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:42 smithi120 ceph-mon[43961]: Cluster is now healthy 2024-05-15T13:31:43.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:42 smithi177 ceph-mon[44303]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-05-15T13:31:43.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:42 smithi177 ceph-mon[44303]: Cluster is now healthy 2024-05-15T13:31:44.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:43 smithi027 ceph-mon[39795]: pgmap v2598: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:44.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:43 smithi120 ceph-mon[43961]: pgmap v2598: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:44.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:43 smithi177 ceph-mon[44303]: pgmap v2598: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:44.989 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-05-15T13:31:45.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:44 smithi027 ceph-mon[39795]: 1.7 scrub starts 2024-05-15T13:31:45.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:44 smithi027 ceph-mon[39795]: 1.7 scrub ok 2024-05-15T13:31:45.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:44 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2527699729' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T13:31:45.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:44 smithi120 ceph-mon[43961]: 1.7 scrub starts 2024-05-15T13:31:45.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:44 smithi120 ceph-mon[43961]: 1.7 scrub ok 2024-05-15T13:31:45.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:44 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2527699729' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T13:31:45.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:44 smithi177 ceph-mon[44303]: 1.7 scrub starts 2024-05-15T13:31:45.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:44 smithi177 ceph-mon[44303]: 1.7 scrub ok 2024-05-15T13:31:45.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:44 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2527699729' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T13:31:46.069 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-05-15T13:31:46.069 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-05-15T13:31:46.070 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-05-15T13:31:46.070 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-05-15T13:31:46.070 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-15T13:31:46.070 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-15T13:31:46.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:45 smithi027 ceph-mon[39795]: pgmap v2599: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:46.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:45 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2527699729' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-05-15T13:31:46.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:45 smithi027 ceph-mon[39795]: osdmap e1038: 12 total, 12 up, 12 in 2024-05-15T13:31:46.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:45 smithi120 ceph-mon[43961]: pgmap v2599: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:46.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:45 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2527699729' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-05-15T13:31:46.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:45 smithi120 ceph-mon[43961]: osdmap e1038: 12 total, 12 up, 12 in 2024-05-15T13:31:46.351 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:46.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:45 smithi177 ceph-mon[44303]: pgmap v2599: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:46.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:45 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2527699729' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-05-15T13:31:46.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:45 smithi177 ceph-mon[44303]: osdmap e1038: 12 total, 12 up, 12 in 2024-05-15T13:31:47.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:46 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:31:47.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:46 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:31:47.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:46 smithi027 ceph-mon[39795]: osdmap e1039: 12 total, 12 up, 12 in 2024-05-15T13:31:47.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:46 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:31:47.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:46 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:31:47.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:46 smithi120 ceph-mon[43961]: osdmap e1039: 12 total, 12 up, 12 in 2024-05-15T13:31:47.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:46 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:31:47.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:46 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:31:47.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:46 smithi177 ceph-mon[44303]: osdmap e1039: 12 total, 12 up, 12 in 2024-05-15T13:31:48.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:47 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:31:48.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:47 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:31:48.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:47 smithi027 ceph-mon[39795]: pgmap v2602: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:48.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:47 smithi027 ceph-mon[39795]: osdmap e1040: 12 total, 12 up, 12 in 2024-05-15T13:31:48.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:47 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:31:48.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:47 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:31:48.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:47 smithi120 ceph-mon[43961]: pgmap v2602: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:48.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:47 smithi120 ceph-mon[43961]: osdmap e1040: 12 total, 12 up, 12 in 2024-05-15T13:31:48.418 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:31:48.419 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":1040,"fsid":"deaea04e-12b7-11ef-bc98-c7b262605968","created":"2024-05-15T12:38:21.244556+0000","modified":"2024-05-15T13:31:47.410827+0000","last_up_change":"2024-05-15T13:25:57.927520+0000","last_in_change":"2024-05-15T13:30:11.205814+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T12:41:57.948613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"998","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"834","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.940000057220459,"score_stable":1.940000057220459,"optimal_score":1,"raw_score_acting":1.940000057220459,"raw_score_stable":1.940000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"64fb3ad2-08a7-4106-a07a-6c776c877faf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":996,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4239116935},{"type":"v1","addr":"172.21.15.27:6803","nonce":4239116935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4239116935},{"type":"v1","addr":"172.21.15.27:6805","nonce":4239116935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4239116935},{"type":"v1","addr":"172.21.15.27:6809","nonce":4239116935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4239116935},{"type":"v1","addr":"172.21.15.27:6807","nonce":4239116935}]},"public_addr":"172.21.15.27:6803/4239116935","cluster_addr":"172.21.15.27:6805/4239116935","heartbeat_back_addr":"172.21.15.27:6809/4239116935","heartbeat_front_addr":"172.21.15.27:6807/4239116935","state":["exists","up"]},{"osd":1,"uuid":"9c16dd90-9d51-48b8-a296-1c8879ba1bc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":997,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1466270637},{"type":"v1","addr":"172.21.15.27:6811","nonce":1466270637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1466270637},{"type":"v1","addr":"172.21.15.27:6813","nonce":1466270637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1466270637},{"type":"v1","addr":"172.21.15.27:6817","nonce":1466270637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1466270637},{"type":"v1","addr":"172.21.15.27:6815","nonce":1466270637}]},"public_addr":"172.21.15.27:6811/1466270637","cluster_addr":"172.21.15.27:6813/1466270637","heartbeat_back_addr":"172.21.15.27:6817/1466270637","heartbeat_front_addr":"172.21.15.27:6815/1466270637","state":["exists","up"]},{"osd":2,"uuid":"cbe3ba13-3ea3-4891-8d2d-6b85581a732c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":996,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3285413897},{"type":"v1","addr":"172.21.15.27:6819","nonce":3285413897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3285413897},{"type":"v1","addr":"172.21.15.27:6821","nonce":3285413897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3285413897},{"type":"v1","addr":"172.21.15.27:6825","nonce":3285413897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3285413897},{"type":"v1","addr":"172.21.15.27:6823","nonce":3285413897}]},"public_addr":"172.21.15.27:6819/3285413897","cluster_addr":"172.21.15.27:6821/3285413897","heartbeat_back_addr":"172.21.15.27:6825/3285413897","heartbeat_front_addr":"172.21.15.27:6823/3285413897","state":["exists","up"]},{"osd":3,"uuid":"2e83735b-b944-4f87-8080-b994d4b96897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":982,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3008975244},{"type":"v1","addr":"172.21.15.27:6827","nonce":3008975244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3008975244},{"type":"v1","addr":"172.21.15.27:6829","nonce":3008975244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3008975244},{"type":"v1","addr":"172.21.15.27:6833","nonce":3008975244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3008975244},{"type":"v1","addr":"172.21.15.27:6831","nonce":3008975244}]},"public_addr":"172.21.15.27:6827/3008975244","cluster_addr":"172.21.15.27:6829/3008975244","heartbeat_back_addr":"172.21.15.27:6833/3008975244","heartbeat_front_addr":"172.21.15.27:6831/3008975244","state":["exists","up"]},{"osd":4,"uuid":"7c39c1dd-3867-46f7-8372-7688fa13b16a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":31,"last_clean_end":886,"up_from":887,"up_thru":997,"down_at":883,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3877842957},{"type":"v1","addr":"172.21.15.120:6801","nonce":3877842957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":3878842957},{"type":"v1","addr":"172.21.15.120:6833","nonce":3878842957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3877842957},{"type":"v1","addr":"172.21.15.120:6807","nonce":3877842957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3877842957},{"type":"v1","addr":"172.21.15.120:6805","nonce":3877842957}]},"public_addr":"172.21.15.120:6801/3877842957","cluster_addr":"172.21.15.120:6833/3878842957","heartbeat_back_addr":"172.21.15.120:6807/3877842957","heartbeat_front_addr":"172.21.15.120:6805/3877842957","state":["exists","up"]},{"osd":5,"uuid":"a6dc220f-aec0-4454-9695-b44c0b9770e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":997,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1600179183},{"type":"v1","addr":"172.21.15.120:6809","nonce":1600179183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1600179183},{"type":"v1","addr":"172.21.15.120:6811","nonce":1600179183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1600179183},{"type":"v1","addr":"172.21.15.120:6815","nonce":1600179183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1600179183},{"type":"v1","addr":"172.21.15.120:6813","nonce":1600179183}]},"public_addr":"172.21.15.120:6809/1600179183","cluster_addr":"172.21.15.120:6811/1600179183","heartbeat_back_addr":"172.21.15.120:6815/1600179183","heartbeat_front_addr":"172.21.15.120:6813/1600179183","state":["exists","up"]},{"osd":6,"uuid":"9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":996,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":3323706735},{"type":"v1","addr":"172.21.15.120:6817","nonce":3323706735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":3323706735},{"type":"v1","addr":"172.21.15.120:6819","nonce":3323706735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":3323706735},{"type":"v1","addr":"172.21.15.120:6823","nonce":3323706735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":3323706735},{"type":"v1","addr":"172.21.15.120:6821","nonce":3323706735}]},"public_addr":"172.21.15.120:6817/3323706735","cluster_addr":"172.21.15.120:6819/3323706735","heartbeat_back_addr":"172.21.15.120:6823/3323706735","heartbeat_front_addr":"172.21.15.120:6821/3323706735","state":["exists","up"]},{"osd":7,"uuid":"6ef085d8-f59b-4033-8280-2c8f78693e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":997,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2960616153},{"type":"v1","addr":"172.21.15.120:6825","nonce":2960616153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2960616153},{"type":"v1","addr":"172.21.15.120:6827","nonce":2960616153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2960616153},{"type":"v1","addr":"172.21.15.120:6831","nonce":2960616153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2960616153},{"type":"v1","addr":"172.21.15.120:6829","nonce":2960616153}]},"public_addr":"172.21.15.120:6825/2960616153","cluster_addr":"172.21.15.120:6827/2960616153","heartbeat_back_addr":"172.21.15.120:6831/2960616153","heartbeat_front_addr":"172.21.15.120:6829/2960616153","state":["exists","up"]},{"osd":8,"uuid":"26514b11-d02b-46e9-a4e8-3da193fcdbcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":997,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1132459315},{"type":"v1","addr":"172.21.15.177:6801","nonce":1132459315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1132459315},{"type":"v1","addr":"172.21.15.177:6803","nonce":1132459315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1132459315},{"type":"v1","addr":"172.21.15.177:6807","nonce":1132459315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1132459315},{"type":"v1","addr":"172.21.15.177:6805","nonce":1132459315}]},"public_addr":"172.21.15.177:6801/1132459315","cluster_addr":"172.21.15.177:6803/1132459315","heartbeat_back_addr":"172.21.15.177:6807/1132459315","heartbeat_front_addr":"172.21.15.177:6805/1132459315","state":["exists","up"]},{"osd":9,"uuid":"6f9497bc-e962-409d-8a2c-0c634cc1124f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":941,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2710457646},{"type":"v1","addr":"172.21.15.177:6809","nonce":2710457646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2710457646},{"type":"v1","addr":"172.21.15.177:6811","nonce":2710457646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2710457646},{"type":"v1","addr":"172.21.15.177:6815","nonce":2710457646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2710457646},{"type":"v1","addr":"172.21.15.177:6813","nonce":2710457646}]},"public_addr":"172.21.15.177:6809/2710457646","cluster_addr":"172.21.15.177:6811/2710457646","heartbeat_back_addr":"172.21.15.177:6815/2710457646","heartbeat_front_addr":"172.21.15.177:6813/2710457646","state":["exists","up"]},{"osd":10,"uuid":"f852876f-c7d1-4039-90c4-5ab09dbd3e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":1034,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":3236900263},{"type":"v1","addr":"172.21.15.177:6817","nonce":3236900263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3236900263},{"type":"v1","addr":"172.21.15.177:6819","nonce":3236900263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3236900263},{"type":"v1","addr":"172.21.15.177:6823","nonce":3236900263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3236900263},{"type":"v1","addr":"172.21.15.177:6821","nonce":3236900263}]},"public_addr":"172.21.15.177:6817/3236900263","cluster_addr":"172.21.15.177:6819/3236900263","heartbeat_back_addr":"172.21.15.177:6823/3236900263","heartbeat_front_addr":"172.21.15.177:6821/3236900263","state":["exists","up"]},{"osd":11,"uuid":"cccc53c5-8476-4d55-b160-4bdc5cf92902","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":90,"up_thru":939,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":681011628},{"type":"v1","addr":"172.21.15.177:6825","nonce":681011628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":681011628},{"type":"v1","addr":"172.21.15.177:6827","nonce":681011628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":681011628},{"type":"v1","addr":"172.21.15.177:6831","nonce":681011628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":681011628},{"type":"v1","addr":"172.21.15.177:6829","nonce":681011628}]},"public_addr":"172.21.15.177:6825/681011628","cluster_addr":"172.21.15.177:6827/681011628","heartbeat_back_addr":"172.21.15.177:6831/681011628","heartbeat_front_addr":"172.21.15.177:6829/681011628","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T13:31:35.216120+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T13:31:08.208039+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T13:31:22.865813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T13:30:57.937068+0000","dead_epoch":0},{"osd":4,"down_stamp":"2024-05-15T13:25:51.817062+0000","laggy_probability":0.30000001192092896,"laggy_interval":1,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T13:30:47.551975+0000","dead_epoch":883},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T13:31:07.863080+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T13:31:33.009970+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T13:31:17.284160+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T13:31:45.956960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T13:31:30.164295+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T13:31:46.414979+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T13:31:38.447809+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[1,6,10]}],"pg_upmap_items":[{"pgid":"1.6","mappings":[{"from":9,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6801/2530617473":"2024-05-16T12:39:19.409195+0000","172.21.15.27:6800/2530617473":"2024-05-16T12:39:19.409195+0000","172.21.15.27:0/1880261204":"2024-05-16T12:39:19.409195+0000","172.21.15.27:6801/3441136471":"2024-05-16T12:38:50.931011+0000","172.21.15.27:0/4291446247":"2024-05-16T12:39:19.409195+0000","172.21.15.27:0/580128209":"2024-05-16T12:38:50.931011+0000","172.21.15.27:6800/3441136471":"2024-05-16T12:38:50.931011+0000","172.21.15.27:0/3011229857":"2024-05-16T12:38:50.931011+0000","172.21.15.27:0/2109693913":"2024-05-16T12:38:50.931011+0000","172.21.15.27:0/2445663024":"2024-05-16T12:39:19.409195+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T13:31:48.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:47 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:31:48.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:47 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:31:48.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:47 smithi177 ceph-mon[44303]: pgmap v2602: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:48.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:47 smithi177 ceph-mon[44303]: osdmap e1040: 12 total, 12 up, 12 in 2024-05-15T13:31:49.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:48 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/3660911259' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T13:31:49.172 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-15T13:31:49.172 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-15T13:31:49.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:48 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/3660911259' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T13:31:49.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:48 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/3660911259' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T13:31:49.451 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:50.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:49 smithi027 ceph-mon[39795]: pgmap v2604: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:50.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:49 smithi027 ceph-mon[39795]: 1.15 scrub starts 2024-05-15T13:31:50.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:49 smithi027 ceph-mon[39795]: 1.15 scrub ok 2024-05-15T13:31:50.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:49 smithi120 ceph-mon[43961]: pgmap v2604: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:50.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:49 smithi120 ceph-mon[43961]: 1.15 scrub starts 2024-05-15T13:31:50.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:49 smithi120 ceph-mon[43961]: 1.15 scrub ok 2024-05-15T13:31:50.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:49 smithi177 ceph-mon[44303]: pgmap v2604: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:50.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:49 smithi177 ceph-mon[44303]: 1.15 scrub starts 2024-05-15T13:31:50.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:49 smithi177 ceph-mon[44303]: 1.15 scrub ok 2024-05-15T13:31:51.456 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:31:51.456 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":1040,"fsid":"deaea04e-12b7-11ef-bc98-c7b262605968","created":"2024-05-15T12:38:21.244556+0000","modified":"2024-05-15T13:31:47.410827+0000","last_up_change":"2024-05-15T13:25:57.927520+0000","last_in_change":"2024-05-15T13:30:11.205814+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T12:41:57.948613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"998","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"834","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.940000057220459,"score_stable":1.940000057220459,"optimal_score":1,"raw_score_acting":1.940000057220459,"raw_score_stable":1.940000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"64fb3ad2-08a7-4106-a07a-6c776c877faf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":996,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":4239116935},{"type":"v1","addr":"172.21.15.27:6803","nonce":4239116935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":4239116935},{"type":"v1","addr":"172.21.15.27:6805","nonce":4239116935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4239116935},{"type":"v1","addr":"172.21.15.27:6809","nonce":4239116935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":4239116935},{"type":"v1","addr":"172.21.15.27:6807","nonce":4239116935}]},"public_addr":"172.21.15.27:6803/4239116935","cluster_addr":"172.21.15.27:6805/4239116935","heartbeat_back_addr":"172.21.15.27:6809/4239116935","heartbeat_front_addr":"172.21.15.27:6807/4239116935","state":["exists","up"]},{"osd":1,"uuid":"9c16dd90-9d51-48b8-a296-1c8879ba1bc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":997,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1466270637},{"type":"v1","addr":"172.21.15.27:6811","nonce":1466270637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1466270637},{"type":"v1","addr":"172.21.15.27:6813","nonce":1466270637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1466270637},{"type":"v1","addr":"172.21.15.27:6817","nonce":1466270637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1466270637},{"type":"v1","addr":"172.21.15.27:6815","nonce":1466270637}]},"public_addr":"172.21.15.27:6811/1466270637","cluster_addr":"172.21.15.27:6813/1466270637","heartbeat_back_addr":"172.21.15.27:6817/1466270637","heartbeat_front_addr":"172.21.15.27:6815/1466270637","state":["exists","up"]},{"osd":2,"uuid":"cbe3ba13-3ea3-4891-8d2d-6b85581a732c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":996,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3285413897},{"type":"v1","addr":"172.21.15.27:6819","nonce":3285413897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3285413897},{"type":"v1","addr":"172.21.15.27:6821","nonce":3285413897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3285413897},{"type":"v1","addr":"172.21.15.27:6825","nonce":3285413897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3285413897},{"type":"v1","addr":"172.21.15.27:6823","nonce":3285413897}]},"public_addr":"172.21.15.27:6819/3285413897","cluster_addr":"172.21.15.27:6821/3285413897","heartbeat_back_addr":"172.21.15.27:6825/3285413897","heartbeat_front_addr":"172.21.15.27:6823/3285413897","state":["exists","up"]},{"osd":3,"uuid":"2e83735b-b944-4f87-8080-b994d4b96897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":982,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3008975244},{"type":"v1","addr":"172.21.15.27:6827","nonce":3008975244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3008975244},{"type":"v1","addr":"172.21.15.27:6829","nonce":3008975244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3008975244},{"type":"v1","addr":"172.21.15.27:6833","nonce":3008975244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3008975244},{"type":"v1","addr":"172.21.15.27:6831","nonce":3008975244}]},"public_addr":"172.21.15.27:6827/3008975244","cluster_addr":"172.21.15.27:6829/3008975244","heartbeat_back_addr":"172.21.15.27:6833/3008975244","heartbeat_front_addr":"172.21.15.27:6831/3008975244","state":["exists","up"]},{"osd":4,"uuid":"7c39c1dd-3867-46f7-8372-7688fa13b16a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":31,"last_clean_end":886,"up_from":887,"up_thru":997,"down_at":883,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3877842957},{"type":"v1","addr":"172.21.15.120:6801","nonce":3877842957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":3878842957},{"type":"v1","addr":"172.21.15.120:6833","nonce":3878842957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3877842957},{"type":"v1","addr":"172.21.15.120:6807","nonce":3877842957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3877842957},{"type":"v1","addr":"172.21.15.120:6805","nonce":3877842957}]},"public_addr":"172.21.15.120:6801/3877842957","cluster_addr":"172.21.15.120:6833/3878842957","heartbeat_back_addr":"172.21.15.120:6807/3877842957","heartbeat_front_addr":"172.21.15.120:6805/3877842957","state":["exists","up"]},{"osd":5,"uuid":"a6dc220f-aec0-4454-9695-b44c0b9770e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":997,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1600179183},{"type":"v1","addr":"172.21.15.120:6809","nonce":1600179183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1600179183},{"type":"v1","addr":"172.21.15.120:6811","nonce":1600179183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1600179183},{"type":"v1","addr":"172.21.15.120:6815","nonce":1600179183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1600179183},{"type":"v1","addr":"172.21.15.120:6813","nonce":1600179183}]},"public_addr":"172.21.15.120:6809/1600179183","cluster_addr":"172.21.15.120:6811/1600179183","heartbeat_back_addr":"172.21.15.120:6815/1600179183","heartbeat_front_addr":"172.21.15.120:6813/1600179183","state":["exists","up"]},{"osd":6,"uuid":"9c6b5d0d-5e3e-439c-a6d9-d7f3c503980b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":996,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":3323706735},{"type":"v1","addr":"172.21.15.120:6817","nonce":3323706735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":3323706735},{"type":"v1","addr":"172.21.15.120:6819","nonce":3323706735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":3323706735},{"type":"v1","addr":"172.21.15.120:6823","nonce":3323706735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":3323706735},{"type":"v1","addr":"172.21.15.120:6821","nonce":3323706735}]},"public_addr":"172.21.15.120:6817/3323706735","cluster_addr":"172.21.15.120:6819/3323706735","heartbeat_back_addr":"172.21.15.120:6823/3323706735","heartbeat_front_addr":"172.21.15.120:6821/3323706735","state":["exists","up"]},{"osd":7,"uuid":"6ef085d8-f59b-4033-8280-2c8f78693e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":997,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2960616153},{"type":"v1","addr":"172.21.15.120:6825","nonce":2960616153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2960616153},{"type":"v1","addr":"172.21.15.120:6827","nonce":2960616153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2960616153},{"type":"v1","addr":"172.21.15.120:6831","nonce":2960616153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2960616153},{"type":"v1","addr":"172.21.15.120:6829","nonce":2960616153}]},"public_addr":"172.21.15.120:6825/2960616153","cluster_addr":"172.21.15.120:6827/2960616153","heartbeat_back_addr":"172.21.15.120:6831/2960616153","heartbeat_front_addr":"172.21.15.120:6829/2960616153","state":["exists","up"]},{"osd":8,"uuid":"26514b11-d02b-46e9-a4e8-3da193fcdbcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":997,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1132459315},{"type":"v1","addr":"172.21.15.177:6801","nonce":1132459315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1132459315},{"type":"v1","addr":"172.21.15.177:6803","nonce":1132459315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1132459315},{"type":"v1","addr":"172.21.15.177:6807","nonce":1132459315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1132459315},{"type":"v1","addr":"172.21.15.177:6805","nonce":1132459315}]},"public_addr":"172.21.15.177:6801/1132459315","cluster_addr":"172.21.15.177:6803/1132459315","heartbeat_back_addr":"172.21.15.177:6807/1132459315","heartbeat_front_addr":"172.21.15.177:6805/1132459315","state":["exists","up"]},{"osd":9,"uuid":"6f9497bc-e962-409d-8a2c-0c634cc1124f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":941,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2710457646},{"type":"v1","addr":"172.21.15.177:6809","nonce":2710457646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2710457646},{"type":"v1","addr":"172.21.15.177:6811","nonce":2710457646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2710457646},{"type":"v1","addr":"172.21.15.177:6815","nonce":2710457646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2710457646},{"type":"v1","addr":"172.21.15.177:6813","nonce":2710457646}]},"public_addr":"172.21.15.177:6809/2710457646","cluster_addr":"172.21.15.177:6811/2710457646","heartbeat_back_addr":"172.21.15.177:6815/2710457646","heartbeat_front_addr":"172.21.15.177:6813/2710457646","state":["exists","up"]},{"osd":10,"uuid":"f852876f-c7d1-4039-90c4-5ab09dbd3e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":1034,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":3236900263},{"type":"v1","addr":"172.21.15.177:6817","nonce":3236900263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3236900263},{"type":"v1","addr":"172.21.15.177:6819","nonce":3236900263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3236900263},{"type":"v1","addr":"172.21.15.177:6823","nonce":3236900263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3236900263},{"type":"v1","addr":"172.21.15.177:6821","nonce":3236900263}]},"public_addr":"172.21.15.177:6817/3236900263","cluster_addr":"172.21.15.177:6819/3236900263","heartbeat_back_addr":"172.21.15.177:6823/3236900263","heartbeat_front_addr":"172.21.15.177:6821/3236900263","state":["exists","up"]},{"osd":11,"uuid":"cccc53c5-8476-4d55-b160-4bdc5cf92902","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":90,"up_thru":939,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":681011628},{"type":"v1","addr":"172.21.15.177:6825","nonce":681011628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":681011628},{"type":"v1","addr":"172.21.15.177:6827","nonce":681011628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":681011628},{"type":"v1","addr":"172.21.15.177:6831","nonce":681011628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":681011628},{"type":"v1","addr":"172.21.15.177:6829","nonce":681011628}]},"public_addr":"172.21.15.177:6825/681011628","cluster_addr":"172.21.15.177:6827/681011628","heartbeat_back_addr":"172.21.15.177:6831/681011628","heartbeat_front_addr":"172.21.15.177:6829/681011628","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T13:31:35.216120+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T13:31:08.208039+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T13:31:22.865813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T13:30:57.937068+0000","dead_epoch":0},{"osd":4,"down_stamp":"2024-05-15T13:25:51.817062+0000","laggy_probability":0.30000001192092896,"laggy_interval":1,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T13:30:47.551975+0000","dead_epoch":883},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T13:31:07.863080+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T13:31:33.009970+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T13:31:17.284160+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T13:31:45.956960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T13:31:30.164295+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T13:31:46.414979+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T13:31:38.447809+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[1,6,10]}],"pg_upmap_items":[{"pgid":"1.6","mappings":[{"from":9,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6801/2530617473":"2024-05-16T12:39:19.409195+0000","172.21.15.27:6800/2530617473":"2024-05-16T12:39:19.409195+0000","172.21.15.27:0/1880261204":"2024-05-16T12:39:19.409195+0000","172.21.15.27:6801/3441136471":"2024-05-16T12:38:50.931011+0000","172.21.15.27:0/4291446247":"2024-05-16T12:39:19.409195+0000","172.21.15.27:0/580128209":"2024-05-16T12:38:50.931011+0000","172.21.15.27:6800/3441136471":"2024-05-16T12:38:50.931011+0000","172.21.15.27:0/3011229857":"2024-05-16T12:38:50.931011+0000","172.21.15.27:0/2109693913":"2024-05-16T12:38:50.931011+0000","172.21.15.27:0/2445663024":"2024-05-16T12:39:19.409195+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T13:31:52.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:51 smithi027 ceph-mon[39795]: pgmap v2605: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:52.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:51 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2653818218' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T13:31:52.095 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-15T13:31:52.096 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-15T13:31:52.096 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-15T13:31:52.096 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-15T13:31:52.097 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-15T13:31:52.097 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-15T13:31:52.098 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-15T13:31:52.098 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-15T13:31:52.098 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-05-15T13:31:52.099 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-05-15T13:31:52.099 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-05-15T13:31:52.100 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-05-15T13:31:52.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:51 smithi120 ceph-mon[43961]: pgmap v2605: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:52.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:51 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2653818218' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T13:31:52.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:51 smithi177 ceph-mon[44303]: pgmap v2605: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:52.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:51 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2653818218' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T13:31:52.828 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:53.161 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:53.161 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:53.162 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:53.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:53 smithi027 ceph-mon[39795]: 1.b scrub starts 2024-05-15T13:31:53.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:53 smithi120 ceph-mon[43961]: 1.b scrub starts 2024-05-15T13:31:53.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:53 smithi177 ceph-mon[44303]: 1.b scrub starts 2024-05-15T13:31:54.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:54 smithi027 ceph-mon[39795]: pgmap v2606: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:54.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:54 smithi027 ceph-mon[39795]: 1.b scrub ok 2024-05-15T13:31:54.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:54 smithi120 ceph-mon[43961]: pgmap v2606: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:54.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:54 smithi120 ceph-mon[43961]: 1.b scrub ok 2024-05-15T13:31:54.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:54 smithi177 ceph-mon[44303]: pgmap v2606: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:54.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:54 smithi177 ceph-mon[44303]: 1.b scrub ok 2024-05-15T13:31:54.898 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:54.899 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:54.899 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:54.903 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:54.904 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:54.904 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:54.911 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:31:55.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:55 smithi027 ceph-mon[39795]: pgmap v2607: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:55.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:55 smithi177 ceph-mon[44303]: pgmap v2607: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:55.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:55 smithi120 ceph-mon[43961]: pgmap v2607: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:58.139 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:57 smithi027 ceph-mon[39795]: pgmap v2608: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:58.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:57 smithi177 ceph-mon[44303]: pgmap v2608: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:31:58.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:57 smithi120 ceph-mon[43961]: pgmap v2608: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:00.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:31:59 smithi027 ceph-mon[39795]: pgmap v2609: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:00.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:31:59 smithi177 ceph-mon[44303]: pgmap v2609: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:00.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:31:59 smithi120 ceph-mon[43961]: pgmap v2609: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:00.882 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:00 smithi027 ceph-mon[39795]: 1.5 scrub starts 2024-05-15T13:32:00.882 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:00 smithi027 ceph-mon[39795]: 1.5 scrub ok 2024-05-15T13:32:01.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:00 smithi177 ceph-mon[44303]: 1.5 scrub starts 2024-05-15T13:32:01.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:00 smithi177 ceph-mon[44303]: 1.5 scrub ok 2024-05-15T13:32:01.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:00 smithi120 ceph-mon[43961]: 1.5 scrub starts 2024-05-15T13:32:01.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:00 smithi120 ceph-mon[43961]: 1.5 scrub ok 2024-05-15T13:32:02.194 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:01 smithi177 ceph-mon[44303]: pgmap v2610: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:02.194 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:01 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:32:02.194 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:01 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:32:02.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:01 smithi027 ceph-mon[39795]: pgmap v2610: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:02.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:01 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:32:02.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:01 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:32:02.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:01 smithi120 ceph-mon[43961]: pgmap v2610: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:02.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:01 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:32:02.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:01 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:32:03.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:02 smithi027 ceph-mon[39795]: osdmap e1041: 12 total, 12 up, 12 in 2024-05-15T13:32:03.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:02 smithi177 ceph-mon[44303]: osdmap e1041: 12 total, 12 up, 12 in 2024-05-15T13:32:03.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:02 smithi120 ceph-mon[43961]: osdmap e1041: 12 total, 12 up, 12 in 2024-05-15T13:32:03.977 INFO:teuthology.orchestra.run.smithi027.stdout:77309411935 2024-05-15T13:32:03.978 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-15T13:32:04.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:03 smithi177 ceph-mon[44303]: 1.8 deep-scrub starts 2024-05-15T13:32:04.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:03 smithi177 ceph-mon[44303]: 1.8 deep-scrub ok 2024-05-15T13:32:04.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:03 smithi177 ceph-mon[44303]: pgmap v2612: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:04.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:03 smithi027 ceph-mon[39795]: 1.8 deep-scrub starts 2024-05-15T13:32:04.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:03 smithi027 ceph-mon[39795]: 1.8 deep-scrub ok 2024-05-15T13:32:04.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:03 smithi027 ceph-mon[39795]: pgmap v2612: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:04.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:03 smithi120 ceph-mon[43961]: 1.8 deep-scrub starts 2024-05-15T13:32:04.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:03 smithi120 ceph-mon[43961]: 1.8 deep-scrub ok 2024-05-15T13:32:04.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:03 smithi120 ceph-mon[43961]: pgmap v2612: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:05.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:05 smithi177 ceph-mon[44303]: 1.1b deep-scrub starts 2024-05-15T13:32:05.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:05 smithi177 ceph-mon[44303]: pgmap v2613: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:05.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:05 smithi177 ceph-mon[44303]: 1.1b deep-scrub ok 2024-05-15T13:32:05.735 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:05 smithi027 ceph-mon[39795]: 1.1b deep-scrub starts 2024-05-15T13:32:05.735 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:05 smithi027 ceph-mon[39795]: pgmap v2613: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:05.735 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:05 smithi027 ceph-mon[39795]: 1.1b deep-scrub ok 2024-05-15T13:32:05.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:05 smithi120 ceph-mon[43961]: 1.1b deep-scrub starts 2024-05-15T13:32:05.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:05 smithi120 ceph-mon[43961]: pgmap v2613: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:05.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:05 smithi120 ceph-mon[43961]: 1.1b deep-scrub ok 2024-05-15T13:32:07.833 INFO:teuthology.orchestra.run.smithi027.stdout:386547057200 2024-05-15T13:32:07.834 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-05-15T13:32:08.142 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:07 smithi027 ceph-mon[39795]: pgmap v2614: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:08.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:07 smithi177 ceph-mon[44303]: pgmap v2614: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:08.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:07 smithi120 ceph-mon[43961]: pgmap v2614: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:09.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:08 smithi027 ceph-mon[39795]: 1.4 deep-scrub starts 2024-05-15T13:32:09.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:08 smithi027 ceph-mon[39795]: 1.4 deep-scrub ok 2024-05-15T13:32:09.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:08 smithi027 ceph-mon[39795]: 1.18 scrub starts 2024-05-15T13:32:09.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:08 smithi027 ceph-mon[39795]: 1.18 scrub ok 2024-05-15T13:32:09.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:08 smithi177 ceph-mon[44303]: 1.4 deep-scrub starts 2024-05-15T13:32:09.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:08 smithi177 ceph-mon[44303]: 1.4 deep-scrub ok 2024-05-15T13:32:09.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:08 smithi177 ceph-mon[44303]: 1.18 scrub starts 2024-05-15T13:32:09.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:08 smithi177 ceph-mon[44303]: 1.18 scrub ok 2024-05-15T13:32:09.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:08 smithi120 ceph-mon[43961]: 1.4 deep-scrub starts 2024-05-15T13:32:09.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:08 smithi120 ceph-mon[43961]: 1.4 deep-scrub ok 2024-05-15T13:32:09.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:08 smithi120 ceph-mon[43961]: 1.18 scrub starts 2024-05-15T13:32:09.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:08 smithi120 ceph-mon[43961]: 1.18 scrub ok 2024-05-15T13:32:09.963 INFO:teuthology.orchestra.run.smithi027.stdout:240518169158 2024-05-15T13:32:09.964 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-15T13:32:10.039 INFO:teuthology.orchestra.run.smithi027.stdout:326417515069 2024-05-15T13:32:10.040 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-05-15T13:32:10.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:09 smithi177 ceph-mon[44303]: 1.1d deep-scrub starts 2024-05-15T13:32:10.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:09 smithi177 ceph-mon[44303]: 1.1d deep-scrub ok 2024-05-15T13:32:10.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:09 smithi177 ceph-mon[44303]: pgmap v2615: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:10.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:09 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:32:10.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:09 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:32:10.229 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:09 smithi027 ceph-mon[39795]: 1.1d deep-scrub starts 2024-05-15T13:32:10.229 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:09 smithi027 ceph-mon[39795]: 1.1d deep-scrub ok 2024-05-15T13:32:10.229 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:09 smithi027 ceph-mon[39795]: pgmap v2615: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:10.229 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:09 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:32:10.229 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:09 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:32:10.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:09 smithi120 ceph-mon[43961]: 1.1d deep-scrub starts 2024-05-15T13:32:10.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:09 smithi120 ceph-mon[43961]: 1.1d deep-scrub ok 2024-05-15T13:32:10.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:09 smithi120 ceph-mon[43961]: pgmap v2615: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:10.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:09 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:32:10.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:09 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:32:10.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:10 smithi027 ceph-mon[39795]: osdmap e1042: 12 total, 12 up, 12 in 2024-05-15T13:32:11.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:10 smithi177 ceph-mon[44303]: osdmap e1042: 12 total, 12 up, 12 in 2024-05-15T13:32:11.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:10 smithi120 ceph-mon[43961]: osdmap e1042: 12 total, 12 up, 12 in 2024-05-15T13:32:11.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:11 smithi027 ceph-mon[39795]: pgmap v2617: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:12.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:11 smithi177 ceph-mon[44303]: pgmap v2617: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:12.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:11 smithi120 ceph-mon[43961]: pgmap v2617: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:12.589 INFO:teuthology.orchestra.run.smithi027.stdout:34359738989 2024-05-15T13:32:12.589 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-15T13:32:13.503 INFO:teuthology.orchestra.run.smithi027.stdout:279172874817 2024-05-15T13:32:13.503 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-05-15T13:32:14.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:13 smithi177 ceph-mon[44303]: pgmap v2618: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:14.228 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:13 smithi027 ceph-mon[39795]: pgmap v2618: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:14.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:13 smithi120 ceph-mon[43961]: pgmap v2618: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:15.677 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:15 smithi027 ceph-mon[39795]: pgmap v2619: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:15.691 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:15 smithi177 ceph-mon[44303]: pgmap v2619: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:15.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:15 smithi120 ceph-mon[43961]: pgmap v2619: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:17.193 INFO:teuthology.orchestra.run.smithi027.stdout:111669150298 2024-05-15T13:32:17.194 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-15T13:32:17.667 INFO:teuthology.orchestra.run.smithi027.stdout:55834575462 2024-05-15T13:32:17.667 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-15T13:32:18.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:17 smithi027 ceph-mon[39795]: 1.1a scrub starts 2024-05-15T13:32:18.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:17 smithi027 ceph-mon[39795]: 1.1a scrub ok 2024-05-15T13:32:18.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:17 smithi027 ceph-mon[39795]: pgmap v2620: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:18.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:17 smithi177 ceph-mon[44303]: 1.1a scrub starts 2024-05-15T13:32:18.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:17 smithi177 ceph-mon[44303]: 1.1a scrub ok 2024-05-15T13:32:18.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:17 smithi177 ceph-mon[44303]: pgmap v2620: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:18.216 INFO:teuthology.orchestra.run.smithi027.stdout:171798692434 2024-05-15T13:32:18.217 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-15T13:32:18.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:17 smithi120 ceph-mon[43961]: 1.1a scrub starts 2024-05-15T13:32:18.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:17 smithi120 ceph-mon[43961]: 1.1a scrub ok 2024-05-15T13:32:18.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:17 smithi120 ceph-mon[43961]: pgmap v2620: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:19.137 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:18 smithi027 ceph-mon[39795]: 1.19 deep-scrub starts 2024-05-15T13:32:19.138 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:18 smithi027 ceph-mon[39795]: 1.19 deep-scrub ok 2024-05-15T13:32:19.163 INFO:teuthology.orchestra.run.smithi027.stdout:206158430798 2024-05-15T13:32:19.163 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-15T13:32:19.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:18 smithi177 ceph-mon[44303]: 1.19 deep-scrub starts 2024-05-15T13:32:19.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:18 smithi177 ceph-mon[44303]: 1.19 deep-scrub ok 2024-05-15T13:32:19.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:18 smithi120 ceph-mon[43961]: 1.19 deep-scrub starts 2024-05-15T13:32:19.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:18 smithi120 ceph-mon[43961]: 1.19 deep-scrub ok 2024-05-15T13:32:19.580 INFO:teuthology.orchestra.run.smithi027.stdout:347892351542 2024-05-15T13:32:19.580 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-05-15T13:32:20.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:19 smithi177 ceph-mon[44303]: pgmap v2621: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:20.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:19 smithi027 ceph-mon[39795]: pgmap v2621: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:20.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:19 smithi120 ceph-mon[43961]: pgmap v2621: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:20.333 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:32:20.333 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:32:20.345 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:32:20.382 INFO:teuthology.orchestra.run.smithi027.stdout:3809635992144 2024-05-15T13:32:20.382 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-15T13:32:20.628 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:32:22.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:21 smithi027 ceph-mon[39795]: pgmap v2622: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:22.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:21 smithi177 ceph-mon[44303]: pgmap v2622: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:22.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:21 smithi120 ceph-mon[43961]: pgmap v2622: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:22.599 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:32:22.599 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:32:22.605 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:32:22.999 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:32:22.999 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:32:22.999 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:32:23.001 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:32:23.163 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:22 smithi027 ceph-mon[39795]: 1.d scrub starts 2024-05-15T13:32:23.163 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:22 smithi027 ceph-mon[39795]: 1.d scrub ok 2024-05-15T13:32:23.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:22 smithi177 ceph-mon[44303]: 1.d scrub starts 2024-05-15T13:32:23.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:22 smithi177 ceph-mon[44303]: 1.d scrub ok 2024-05-15T13:32:23.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:22 smithi120 ceph-mon[43961]: 1.d scrub starts 2024-05-15T13:32:23.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:22 smithi120 ceph-mon[43961]: 1.d scrub ok 2024-05-15T13:32:24.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:23 smithi177 ceph-mon[44303]: pgmap v2623: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:24.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:23 smithi027 ceph-mon[39795]: pgmap v2623: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:24.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:23 smithi120 ceph-mon[43961]: pgmap v2623: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:24.678 INFO:teuthology.orchestra.run.smithi027.stdout:326417515074 2024-05-15T13:32:25.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:24 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1738431479' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-15T13:32:25.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:24 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1738431479' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-15T13:32:25.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:24 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1738431479' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-15T13:32:26.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:25 smithi177 ceph-mon[44303]: 1.16 scrub starts 2024-05-15T13:32:26.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:25 smithi177 ceph-mon[44303]: 1.16 scrub ok 2024-05-15T13:32:26.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:25 smithi177 ceph-mon[44303]: pgmap v2624: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:26.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:25 smithi027 ceph-mon[39795]: 1.16 scrub starts 2024-05-15T13:32:26.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:25 smithi027 ceph-mon[39795]: 1.16 scrub ok 2024-05-15T13:32:26.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:25 smithi027 ceph-mon[39795]: pgmap v2624: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:26.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:25 smithi120 ceph-mon[43961]: 1.16 scrub starts 2024-05-15T13:32:26.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:25 smithi120 ceph-mon[43961]: 1.16 scrub ok 2024-05-15T13:32:26.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:25 smithi120 ceph-mon[43961]: pgmap v2624: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:27.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:26 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:32:27.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:26 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:32:27.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:26 smithi027 ceph-mon[39795]: osdmap e1043: 12 total, 12 up, 12 in 2024-05-15T13:32:27.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:26 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:32:27.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:26 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:32:27.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:26 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:32:27.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:26 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:32:27.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:26 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:32:27.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:26 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:32:27.034 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:26 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:32:27.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:26 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:32:27.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:26 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:32:27.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:26 smithi177 ceph-mon[44303]: osdmap e1043: 12 total, 12 up, 12 in 2024-05-15T13:32:27.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:26 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:32:27.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:26 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:32:27.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:26 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:32:27.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:26 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:32:27.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:26 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:32:27.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:26 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:32:27.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:26 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:32:27.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:26 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:32:27.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:26 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:32:27.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:26 smithi120 ceph-mon[43961]: osdmap e1043: 12 total, 12 up, 12 in 2024-05-15T13:32:27.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:26 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:32:27.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:26 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:32:27.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:26 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:32:27.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:26 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:32:27.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:26 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:32:27.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:26 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:32:27.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:26 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:32:27.727 INFO:teuthology.orchestra.run.smithi027.stdout:55834575465 2024-05-15T13:32:28.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:27 smithi027 ceph-mon[39795]: 1.c deep-scrub starts 2024-05-15T13:32:28.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:27 smithi027 ceph-mon[39795]: 1.c deep-scrub ok 2024-05-15T13:32:28.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:27 smithi027 ceph-mon[39795]: pgmap v2626: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:28.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:27 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:32:28.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:27 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:32:28.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:27 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:32:28.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:27 smithi027 ceph-mon[39795]: 1.e scrub starts 2024-05-15T13:32:28.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:27 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1855336617' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-15T13:32:28.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:27 smithi027 ceph-mon[39795]: 1.e scrub ok 2024-05-15T13:32:28.195 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:27 smithi177 ceph-mon[44303]: 1.c deep-scrub starts 2024-05-15T13:32:28.195 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:27 smithi177 ceph-mon[44303]: 1.c deep-scrub ok 2024-05-15T13:32:28.195 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:27 smithi177 ceph-mon[44303]: pgmap v2626: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:28.195 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:27 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:32:28.195 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:27 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:32:28.195 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:27 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:32:28.196 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:27 smithi177 ceph-mon[44303]: 1.e scrub starts 2024-05-15T13:32:28.196 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:27 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1855336617' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-15T13:32:28.196 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:27 smithi177 ceph-mon[44303]: 1.e scrub ok 2024-05-15T13:32:28.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:27 smithi120 ceph-mon[43961]: 1.c deep-scrub starts 2024-05-15T13:32:28.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:27 smithi120 ceph-mon[43961]: 1.c deep-scrub ok 2024-05-15T13:32:28.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:27 smithi120 ceph-mon[43961]: pgmap v2626: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:28.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:27 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:32:28.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:27 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:32:28.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:27 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:32:28.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:27 smithi120 ceph-mon[43961]: 1.e scrub starts 2024-05-15T13:32:28.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:27 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1855336617' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-15T13:32:28.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:27 smithi120 ceph-mon[43961]: 1.e scrub ok 2024-05-15T13:32:30.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:29 smithi177 ceph-mon[44303]: pgmap v2627: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:30.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:29 smithi177 ceph-mon[44303]: 1.a scrub starts 2024-05-15T13:32:30.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:29 smithi177 ceph-mon[44303]: 1.a scrub ok 2024-05-15T13:32:30.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:29 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:32:30.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:29 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:32:30.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:29 smithi027 ceph-mon[39795]: pgmap v2627: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:30.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:29 smithi027 ceph-mon[39795]: 1.a scrub starts 2024-05-15T13:32:30.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:29 smithi027 ceph-mon[39795]: 1.a scrub ok 2024-05-15T13:32:30.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:29 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:32:30.284 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:29 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:32:30.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:29 smithi120 ceph-mon[43961]: pgmap v2627: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:30.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:29 smithi120 ceph-mon[43961]: 1.a scrub starts 2024-05-15T13:32:30.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:29 smithi120 ceph-mon[43961]: 1.a scrub ok 2024-05-15T13:32:30.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:29 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:32:30.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:29 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:32:31.086 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:30 smithi027 ceph-mon[39795]: 1.10 scrub starts 2024-05-15T13:32:31.086 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:30 smithi027 ceph-mon[39795]: 1.10 scrub ok 2024-05-15T13:32:31.086 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:30 smithi027 ceph-mon[39795]: osdmap e1044: 12 total, 12 up, 12 in 2024-05-15T13:32:31.110 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575462 got 55834575465 for osd.1 2024-05-15T13:32:31.111 DEBUG:teuthology.parallel:result is None 2024-05-15T13:32:31.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:30 smithi177 ceph-mon[44303]: 1.10 scrub starts 2024-05-15T13:32:31.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:30 smithi177 ceph-mon[44303]: 1.10 scrub ok 2024-05-15T13:32:31.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:30 smithi177 ceph-mon[44303]: osdmap e1044: 12 total, 12 up, 12 in 2024-05-15T13:32:31.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:30 smithi120 ceph-mon[43961]: 1.10 scrub starts 2024-05-15T13:32:31.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:30 smithi120 ceph-mon[43961]: 1.10 scrub ok 2024-05-15T13:32:31.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:30 smithi120 ceph-mon[43961]: osdmap e1044: 12 total, 12 up, 12 in 2024-05-15T13:32:31.388 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:32:32.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:31 smithi177 ceph-mon[44303]: 1.12 deep-scrub starts 2024-05-15T13:32:32.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:31 smithi177 ceph-mon[44303]: pgmap v2629: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:32.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:31 smithi177 ceph-mon[44303]: 1.12 deep-scrub ok 2024-05-15T13:32:32.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:31 smithi177 ceph-mon[44303]: 1.1c scrub starts 2024-05-15T13:32:32.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:31 smithi177 ceph-mon[44303]: 1.1c scrub ok 2024-05-15T13:32:32.285 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:31 smithi027 ceph-mon[39795]: 1.12 deep-scrub starts 2024-05-15T13:32:32.286 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:31 smithi027 ceph-mon[39795]: pgmap v2629: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:32.286 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:31 smithi027 ceph-mon[39795]: 1.12 deep-scrub ok 2024-05-15T13:32:32.286 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:31 smithi027 ceph-mon[39795]: 1.1c scrub starts 2024-05-15T13:32:32.286 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:31 smithi027 ceph-mon[39795]: 1.1c scrub ok 2024-05-15T13:32:32.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:31 smithi120 ceph-mon[43961]: 1.12 deep-scrub starts 2024-05-15T13:32:32.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:31 smithi120 ceph-mon[43961]: pgmap v2629: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:32.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:31 smithi120 ceph-mon[43961]: 1.12 deep-scrub ok 2024-05-15T13:32:32.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:31 smithi120 ceph-mon[43961]: 1.1c scrub starts 2024-05-15T13:32:32.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:31 smithi120 ceph-mon[43961]: 1.1c scrub ok 2024-05-15T13:32:32.562 INFO:teuthology.orchestra.run.smithi027.stdout:240518169162 2024-05-15T13:32:33.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:32 smithi027 ceph-mon[39795]: 1.17 deep-scrub starts 2024-05-15T13:32:33.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:32 smithi027 ceph-mon[39795]: 1.17 deep-scrub ok 2024-05-15T13:32:33.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:32 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2107927703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-15T13:32:33.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:32 smithi177 ceph-mon[44303]: 1.17 deep-scrub starts 2024-05-15T13:32:33.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:32 smithi177 ceph-mon[44303]: 1.17 deep-scrub ok 2024-05-15T13:32:33.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:32 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2107927703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-15T13:32:33.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:32 smithi120 ceph-mon[43961]: 1.17 deep-scrub starts 2024-05-15T13:32:33.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:32 smithi120 ceph-mon[43961]: 1.17 deep-scrub ok 2024-05-15T13:32:33.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:32 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2107927703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-15T13:32:33.583 INFO:teuthology.orchestra.run.smithi027.stdout:386547057206 2024-05-15T13:32:34.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:33 smithi027 ceph-mon[39795]: 1.11 deep-scrub starts 2024-05-15T13:32:34.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:33 smithi027 ceph-mon[39795]: 1.11 deep-scrub ok 2024-05-15T13:32:34.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:33 smithi027 ceph-mon[39795]: pgmap v2630: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:34.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:33 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:32:34.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:33 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:32:34.213 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:33 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2700158136' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-15T13:32:34.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:33 smithi120 ceph-mon[43961]: 1.11 deep-scrub starts 2024-05-15T13:32:34.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:33 smithi120 ceph-mon[43961]: 1.11 deep-scrub ok 2024-05-15T13:32:34.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:33 smithi120 ceph-mon[43961]: pgmap v2630: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:34.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:33 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:32:34.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:33 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:32:34.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:33 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2700158136' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-15T13:32:34.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:33 smithi177 ceph-mon[44303]: 1.11 deep-scrub starts 2024-05-15T13:32:34.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:33 smithi177 ceph-mon[44303]: 1.11 deep-scrub ok 2024-05-15T13:32:34.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:33 smithi177 ceph-mon[44303]: pgmap v2630: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:34.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:33 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:32:34.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:33 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:32:34.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:33 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2700158136' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-15T13:32:35.206 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:34 smithi027 ceph-mon[39795]: osdmap e1045: 12 total, 12 up, 12 in 2024-05-15T13:32:35.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:34 smithi120 ceph-mon[43961]: osdmap e1045: 12 total, 12 up, 12 in 2024-05-15T13:32:35.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:34 smithi177 ceph-mon[44303]: osdmap e1045: 12 total, 12 up, 12 in 2024-05-15T13:32:35.680 INFO:teuthology.orchestra.run.smithi027.stdout:77309411941 2024-05-15T13:32:36.227 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:35 smithi027 ceph-mon[39795]: pgmap v2632: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:36.228 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:35 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1946745067' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-15T13:32:36.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:35 smithi120 ceph-mon[43961]: pgmap v2632: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:36.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:35 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1946745067' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-15T13:32:36.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:35 smithi177 ceph-mon[44303]: pgmap v2632: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:36.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:35 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1946745067' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-15T13:32:36.786 INFO:teuthology.orchestra.run.smithi027.stdout:171798692438 2024-05-15T13:32:36.911 INFO:tasks.cephadm.ceph_manager.ceph:need seq 326417515069 got 326417515074 for osd.9 2024-05-15T13:32:36.911 DEBUG:teuthology.parallel:result is None 2024-05-15T13:32:37.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:36 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:32:37.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:36 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:32:37.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:36 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:32:37.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:36 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:32:37.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:36 smithi027 ceph-mon[39795]: osdmap e1046: 12 total, 12 up, 12 in 2024-05-15T13:32:37.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:36 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/441597921' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-15T13:32:37.094 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:36 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/1561092601' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-15T13:32:37.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:36 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:32:37.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:36 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:32:37.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:36 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:32:37.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:36 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:32:37.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:36 smithi120 ceph-mon[43961]: osdmap e1046: 12 total, 12 up, 12 in 2024-05-15T13:32:37.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:36 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/441597921' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-15T13:32:37.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:36 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/1561092601' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-15T13:32:37.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:36 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:32:37.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:36 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:32:37.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:36 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:32:37.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:36 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:32:37.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:36 smithi177 ceph-mon[44303]: osdmap e1046: 12 total, 12 up, 12 in 2024-05-15T13:32:37.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:36 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/441597921' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-15T13:32:37.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:36 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/1561092601' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-15T13:32:37.537 INFO:teuthology.orchestra.run.smithi027.stdout:111669150304 2024-05-15T13:32:37.758 INFO:teuthology.orchestra.run.smithi027.stdout:347892351547 2024-05-15T13:32:38.116 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:37 smithi027 ceph-mon[39795]: pgmap v2634: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:38.116 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:37 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2086493120' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-15T13:32:38.116 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:37 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/892571489' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-15T13:32:38.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:37 smithi120 ceph-mon[43961]: pgmap v2634: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:38.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:37 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2086493120' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-15T13:32:38.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:37 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/892571489' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-15T13:32:38.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:37 smithi177 ceph-mon[44303]: pgmap v2634: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:38.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:37 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2086493120' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-15T13:32:38.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:37 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/892571489' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-15T13:32:38.539 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518169158 got 240518169162 for osd.7 2024-05-15T13:32:38.539 DEBUG:teuthology.parallel:result is None 2024-05-15T13:32:38.985 INFO:teuthology.orchestra.run.smithi027.stdout:206158430801 2024-05-15T13:32:39.619 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430798 got 206158430801 for osd.6 2024-05-15T13:32:39.619 DEBUG:teuthology.parallel:result is None 2024-05-15T13:32:39.698 INFO:tasks.cephadm.ceph_manager.ceph:need seq 386547057200 got 386547057206 for osd.11 2024-05-15T13:32:39.698 DEBUG:teuthology.parallel:result is None 2024-05-15T13:32:40.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:39 smithi027 ceph-mon[39795]: pgmap v2635: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:40.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:39 smithi120 ceph-mon[43961]: pgmap v2635: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:40.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:39 smithi177 ceph-mon[44303]: pgmap v2635: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:41.802 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411935 got 77309411941 for osd.2 2024-05-15T13:32:41.802 DEBUG:teuthology.parallel:result is None 2024-05-15T13:32:42.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:41 smithi027 ceph-mon[39795]: pgmap v2636: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:42.141 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669150298 got 111669150304 for osd.3 2024-05-15T13:32:42.141 DEBUG:teuthology.parallel:result is None 2024-05-15T13:32:42.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:41 smithi120 ceph-mon[43961]: pgmap v2636: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:42.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:41 smithi177 ceph-mon[44303]: pgmap v2636: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:43.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:42 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:32:43.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:42 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:32:43.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:42 smithi027 ceph-mon[39795]: osdmap e1047: 12 total, 12 up, 12 in 2024-05-15T13:32:43.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:42 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:32:43.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:42 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:32:43.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:42 smithi120 ceph-mon[43961]: osdmap e1047: 12 total, 12 up, 12 in 2024-05-15T13:32:43.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:42 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:32:43.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:42 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:32:43.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:42 smithi177 ceph-mon[44303]: osdmap e1047: 12 total, 12 up, 12 in 2024-05-15T13:32:43.614 INFO:teuthology.orchestra.run.smithi027.stdout:279172874823 2024-05-15T13:32:44.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:43 smithi027 ceph-mon[39795]: pgmap v2638: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:44.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:43 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2746872169' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-15T13:32:44.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:43 smithi120 ceph-mon[43961]: pgmap v2638: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:44.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:43 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2746872169' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-15T13:32:44.420 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874817 got 279172874823 for osd.8 2024-05-15T13:32:44.421 DEBUG:teuthology.parallel:result is None 2024-05-15T13:32:44.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:43 smithi177 ceph-mon[44303]: pgmap v2638: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:44.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:43 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2746872169' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-15T13:32:44.520 INFO:teuthology.orchestra.run.smithi027.stdout:34359738997 2024-05-15T13:32:44.819 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892351542 got 347892351547 for osd.10 2024-05-15T13:32:44.819 DEBUG:teuthology.parallel:result is None 2024-05-15T13:32:45.188 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:44 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/2161090260' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-15T13:32:45.213 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738989 got 34359738997 for osd.0 2024-05-15T13:32:45.214 DEBUG:teuthology.parallel:result is None 2024-05-15T13:32:45.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:44 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/2161090260' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-15T13:32:45.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:44 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/2161090260' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-15T13:32:45.574 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798692434 got 171798692438 for osd.5 2024-05-15T13:32:45.574 DEBUG:teuthology.parallel:result is None 2024-05-15T13:32:45.657 INFO:teuthology.orchestra.run.smithi027.stdout:3809635992149 2024-05-15T13:32:46.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:45 smithi027 ceph-mon[39795]: pgmap v2639: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:46.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:45 smithi027 ceph-mon[39795]: 1.14 scrub starts 2024-05-15T13:32:46.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:45 smithi027 ceph-mon[39795]: 1.14 scrub ok 2024-05-15T13:32:46.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:45 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/425916772' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-15T13:32:46.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:45 smithi120 ceph-mon[43961]: pgmap v2639: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:46.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:45 smithi120 ceph-mon[43961]: 1.14 scrub starts 2024-05-15T13:32:46.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:45 smithi120 ceph-mon[43961]: 1.14 scrub ok 2024-05-15T13:32:46.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:45 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/425916772' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-15T13:32:46.338 INFO:tasks.cephadm.ceph_manager.ceph:need seq 3809635992144 got 3809635992149 for osd.4 2024-05-15T13:32:46.338 DEBUG:teuthology.parallel:result is None 2024-05-15T13:32:46.338 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-05-15T13:32:46.339 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:32:46.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:45 smithi177 ceph-mon[44303]: pgmap v2639: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:46.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:45 smithi177 ceph-mon[44303]: 1.14 scrub starts 2024-05-15T13:32:46.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:45 smithi177 ceph-mon[44303]: 1.14 scrub ok 2024-05-15T13:32:46.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:45 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/425916772' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-15T13:32:46.618 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:32:48.126 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:47 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:32:48.126 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:47 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:32:48.126 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:47 smithi027 ceph-mon[39795]: pgmap v2640: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:48.126 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:47 smithi027 ceph-mon[39795]: osdmap e1048: 12 total, 12 up, 12 in 2024-05-15T13:32:48.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:47 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:32:48.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:47 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:32:48.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:47 smithi120 ceph-mon[43961]: pgmap v2640: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:48.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:47 smithi120 ceph-mon[43961]: osdmap e1048: 12 total, 12 up, 12 in 2024-05-15T13:32:48.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:47 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:32:48.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:47 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:32:48.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:47 smithi177 ceph-mon[44303]: pgmap v2640: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:48.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:47 smithi177 ceph-mon[44303]: osdmap e1048: 12 total, 12 up, 12 in 2024-05-15T13:32:48.559 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:32:48.561 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:32:49.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:48 smithi027 ceph-mon[39795]: 1.f deep-scrub starts 2024-05-15T13:32:49.233 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:48 smithi027 ceph-mon[39795]: 1.f deep-scrub ok 2024-05-15T13:32:49.262 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2641,"stamp":"2024-05-15T13:32:46.974614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10284904,"kb_used_data":10776,"kb_used_omap":18,"kb_used_meta":10273965,"kb_avail":1114656920,"statfs":{"total":1151940427776,"available":1141408686080,"internally_reserved":0,"allocated":11034624,"data_stored":7084444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":10520540475},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.142731"},"pg_stats":[{"pgid":"1.1e","version":"94'139","reported_seq":1271,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:41.962246+0000","last_change":"2024-05-15T13:31:38.748656+0000","last_active":"2024-05-15T13:32:41.962246+0000","last_peered":"2024-05-15T13:32:41.962246+0000","last_clean":"2024-05-15T13:32:41.962246+0000","last_became_active":"2024-05-15T13:28:01.363009+0000","last_became_peered":"2024-05-15T13:28:01.363009+0000","last_unstale":"2024-05-15T13:32:41.962246+0000","last_undegraded":"2024-05-15T13:32:41.962246+0000","last_fullsized":"2024-05-15T13:32:41.962246+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:38.748530+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:29:11.559293+0000","last_clean_scrub_stamp":"2024-05-15T13:31:38.748530+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:32:55.235801+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"94'139","reported_seq":1049,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:43.186033+0000","last_change":"2024-05-15T13:32:08.435536+0000","last_active":"2024-05-15T13:32:43.186033+0000","last_peered":"2024-05-15T13:32:43.186033+0000","last_clean":"2024-05-15T13:32:43.186033+0000","last_became_active":"2024-05-15T13:30:15.243303+0000","last_became_peered":"2024-05-15T13:30:15.243303+0000","last_unstale":"2024-05-15T13:32:43.186033+0000","last_undegraded":"2024-05-15T13:32:43.186033+0000","last_fullsized":"2024-05-15T13:32:43.186033+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:08.435428+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:32:08.435428+0000","last_clean_scrub_stamp":"2024-05-15T13:32:08.435428+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:32.056821+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"94'139","reported_seq":1038,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:41.960391+0000","last_change":"2024-05-15T13:32:30.925839+0000","last_active":"2024-05-15T13:32:41.960391+0000","last_peered":"2024-05-15T13:32:41.960391+0000","last_clean":"2024-05-15T13:32:41.960391+0000","last_became_active":"2024-05-15T13:30:15.244002+0000","last_became_peered":"2024-05-15T13:30:15.244002+0000","last_unstale":"2024-05-15T13:32:41.960391+0000","last_undegraded":"2024-05-15T13:32:41.960391+0000","last_fullsized":"2024-05-15T13:32:41.960391+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:30.925686+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:32:30.925686+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:59.260209+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"94'139","reported_seq":1431,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:41.960378+0000","last_change":"2024-05-15T13:32:04.836769+0000","last_active":"2024-05-15T13:32:41.960378+0000","last_peered":"2024-05-15T13:32:41.960378+0000","last_clean":"2024-05-15T13:32:41.960378+0000","last_became_active":"2024-05-15T13:27:32.380219+0000","last_became_peered":"2024-05-15T13:27:32.380219+0000","last_unstale":"2024-05-15T13:32:41.960378+0000","last_undegraded":"2024-05-15T13:32:41.960378+0000","last_fullsized":"2024-05-15T13:32:41.960378+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:04.836624+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:32:04.836624+0000","last_clean_scrub_stamp":"2024-05-15T13:32:04.836624+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:06.947250+0000","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"94'139","reported_seq":1479,"reported_epoch":1046,"state":"active+clean","last_fresh":"2024-05-15T13:32:36.430161+0000","last_change":"2024-05-15T13:32:16.616896+0000","last_active":"2024-05-15T13:32:36.430161+0000","last_peered":"2024-05-15T13:32:36.430161+0000","last_clean":"2024-05-15T13:32:36.430161+0000","last_became_active":"2024-05-15T13:30:15.241152+0000","last_became_peered":"2024-05-15T13:30:15.241152+0000","last_unstale":"2024-05-15T13:32:36.430161+0000","last_undegraded":"2024-05-15T13:32:36.430161+0000","last_fullsized":"2024-05-15T13:32:36.430161+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:16.616790+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:22:03.418345+0000","last_clean_scrub_stamp":"2024-05-15T13:32:16.616790+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:20.235908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"94'139","reported_seq":1255,"reported_epoch":1046,"state":"active+clean","last_fresh":"2024-05-15T13:32:36.433603+0000","last_change":"2024-05-15T13:32:17.841086+0000","last_active":"2024-05-15T13:32:36.433603+0000","last_peered":"2024-05-15T13:32:36.433603+0000","last_clean":"2024-05-15T13:32:36.433603+0000","last_became_active":"2024-05-15T13:24:11.374739+0000","last_became_peered":"2024-05-15T13:24:11.374739+0000","last_unstale":"2024-05-15T13:32:36.433603+0000","last_undegraded":"2024-05-15T13:32:36.433603+0000","last_fullsized":"2024-05-15T13:32:36.433603+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:17.840955+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:32:17.840955+0000","last_clean_scrub_stamp":"2024-05-15T13:32:17.840955+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:37.140927+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"94'139","reported_seq":2079,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:44.456959+0000","last_change":"2024-05-15T13:32:07.854793+0000","last_active":"2024-05-15T13:32:44.456959+0000","last_peered":"2024-05-15T13:32:44.456959+0000","last_clean":"2024-05-15T13:32:44.456959+0000","last_became_active":"2024-05-15T13:24:11.368347+0000","last_became_peered":"2024-05-15T13:24:11.368347+0000","last_unstale":"2024-05-15T13:32:44.456959+0000","last_undegraded":"2024-05-15T13:32:44.456959+0000","last_fullsized":"2024-05-15T13:32:44.456959+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:07.854670+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:32:07.854670+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:15.750799+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"94'139","reported_seq":814,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:41.960474+0000","last_change":"2024-05-15T13:32:32.104855+0000","last_active":"2024-05-15T13:32:41.960474+0000","last_peered":"2024-05-15T13:32:41.960474+0000","last_clean":"2024-05-15T13:32:41.960474+0000","last_became_active":"2024-05-15T13:27:32.410771+0000","last_became_peered":"2024-05-15T13:27:32.410771+0000","last_unstale":"2024-05-15T13:32:41.960474+0000","last_undegraded":"2024-05-15T13:32:41.960474+0000","last_fullsized":"2024-05-15T13:32:41.960474+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:32.104724+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:32:32.104724+0000","last_clean_scrub_stamp":"2024-05-15T13:32:32.104724+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:48.717633+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"94'139","reported_seq":2009,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:43.186031+0000","last_change":"2024-05-15T13:32:24.225813+0000","last_active":"2024-05-15T13:32:43.186031+0000","last_peered":"2024-05-15T13:32:43.186031+0000","last_clean":"2024-05-15T13:32:43.186031+0000","last_became_active":"2024-05-15T13:30:15.242583+0000","last_became_peered":"2024-05-15T13:30:15.242583+0000","last_unstale":"2024-05-15T13:32:43.186031+0000","last_undegraded":"2024-05-15T13:32:43.186031+0000","last_fullsized":"2024-05-15T13:32:43.186031+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:24.225676+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:32:24.225676+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:43.891194+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"94'139","reported_seq":1010,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:44.273120+0000","last_change":"2024-05-15T13:31:48.952667+0000","last_active":"2024-05-15T13:32:44.273120+0000","last_peered":"2024-05-15T13:32:44.273120+0000","last_clean":"2024-05-15T13:32:44.273120+0000","last_became_active":"2024-05-15T13:27:32.397907+0000","last_became_peered":"2024-05-15T13:27:32.397907+0000","last_unstale":"2024-05-15T13:32:44.273120+0000","last_undegraded":"2024-05-15T13:32:44.273120+0000","last_fullsized":"2024-05-15T13:32:44.273120+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:48.952562+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:30:44.113972+0000","last_clean_scrub_stamp":"2024-05-15T13:31:48.952562+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:32:54.381731+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"94'139","reported_seq":1515,"reported_epoch":1046,"state":"active+clean","last_fresh":"2024-05-15T13:32:37.870986+0000","last_change":"2024-05-15T13:31:21.705531+0000","last_active":"2024-05-15T13:32:37.870986+0000","last_peered":"2024-05-15T13:32:37.870986+0000","last_clean":"2024-05-15T13:32:37.870986+0000","last_became_active":"2024-05-15T13:19:41.190796+0000","last_became_peered":"2024-05-15T13:19:41.190796+0000","last_unstale":"2024-05-15T13:32:37.870986+0000","last_undegraded":"2024-05-15T13:32:37.870986+0000","last_fullsized":"2024-05-15T13:32:37.870986+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:21.705423+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:31:21.705423+0000","last_clean_scrub_stamp":"2024-05-15T13:31:21.705423+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:32:42.288696+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":665,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:43.185912+0000","last_change":"2024-05-15T13:31:35.322228+0000","last_active":"2024-05-15T13:32:43.185912+0000","last_peered":"2024-05-15T13:32:43.185912+0000","last_clean":"2024-05-15T13:32:43.185912+0000","last_became_active":"2024-05-15T13:30:15.245704+0000","last_became_peered":"2024-05-15T13:30:15.245704+0000","last_unstale":"2024-05-15T13:32:43.185912+0000","last_undegraded":"2024-05-15T13:32:43.185912+0000","last_fullsized":"2024-05-15T13:32:43.185912+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:35.322078+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:31:35.322078+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"94'139","reported_seq":1792,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:41.962210+0000","last_change":"2024-05-15T13:32:06.944608+0000","last_active":"2024-05-15T13:32:41.962210+0000","last_peered":"2024-05-15T13:32:41.962210+0000","last_clean":"2024-05-15T13:32:41.962210+0000","last_became_active":"2024-05-15T13:28:01.357970+0000","last_became_peered":"2024-05-15T13:28:01.357970+0000","last_unstale":"2024-05-15T13:32:41.962210+0000","last_undegraded":"2024-05-15T13:32:41.962210+0000","last_fullsized":"2024-05-15T13:32:41.962210+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:06.944458+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:32:06.944458+0000","last_clean_scrub_stamp":"2024-05-15T13:32:06.944458+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:08.024542+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"94'139","reported_seq":1823,"reported_epoch":1046,"state":"active+clean","last_fresh":"2024-05-15T13:32:36.433603+0000","last_change":"2024-05-15T13:31:59.808857+0000","last_active":"2024-05-15T13:32:36.433603+0000","last_peered":"2024-05-15T13:32:36.433603+0000","last_clean":"2024-05-15T13:32:36.433603+0000","last_became_active":"2024-05-15T13:24:06.307598+0000","last_became_peered":"2024-05-15T13:24:06.307598+0000","last_unstale":"2024-05-15T13:32:36.433603+0000","last_undegraded":"2024-05-15T13:32:36.433603+0000","last_fullsized":"2024-05-15T13:32:36.433603+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:59.808705+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:18:24.970285+0000","last_clean_scrub_stamp":"2024-05-15T13:31:59.808705+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:15.174182+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"94'139","reported_seq":567,"reported_epoch":1046,"state":"active+clean","last_fresh":"2024-05-15T13:32:36.433636+0000","last_change":"2024-05-15T13:31:38.917101+0000","last_active":"2024-05-15T13:32:36.433636+0000","last_peered":"2024-05-15T13:32:36.433636+0000","last_clean":"2024-05-15T13:32:36.433636+0000","last_became_active":"2024-05-15T13:31:37.910916+0000","last_became_peered":"2024-05-15T13:31:37.910916+0000","last_unstale":"2024-05-15T13:32:36.433636+0000","last_undegraded":"2024-05-15T13:32:36.433636+0000","last_fullsized":"2024-05-15T13:32:36.433636+0000","mapping_epoch":1034,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":1035,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:38.916950+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:31:38.916950+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:00.059064+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,8],"acting":[10,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"94'139","reported_seq":1840,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:41.962665+0000","last_change":"2024-05-15T13:31:43.658069+0000","last_active":"2024-05-15T13:32:41.962665+0000","last_peered":"2024-05-15T13:32:41.962665+0000","last_clean":"2024-05-15T13:32:41.962665+0000","last_became_active":"2024-05-15T13:24:06.312706+0000","last_became_peered":"2024-05-15T13:24:06.312706+0000","last_unstale":"2024-05-15T13:32:41.962665+0000","last_undegraded":"2024-05-15T13:32:41.962665+0000","last_fullsized":"2024-05-15T13:32:41.962665+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:43.657969+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:31:43.657969+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:05.004139+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"94'139","reported_seq":567,"reported_epoch":1046,"state":"active+clean","last_fresh":"2024-05-15T13:32:36.433714+0000","last_change":"2024-05-15T13:31:39.936236+0000","last_active":"2024-05-15T13:32:36.433714+0000","last_peered":"2024-05-15T13:32:36.433714+0000","last_clean":"2024-05-15T13:32:36.433714+0000","last_became_active":"2024-05-15T13:31:37.910962+0000","last_became_peered":"2024-05-15T13:31:37.910962+0000","last_unstale":"2024-05-15T13:32:36.433714+0000","last_undegraded":"2024-05-15T13:32:36.433714+0000","last_fullsized":"2024-05-15T13:32:36.433714+0000","mapping_epoch":1034,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":1035,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:39.936110+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:31:39.936110+0000","last_clean_scrub_stamp":"2024-05-15T13:31:39.936110+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:32:58.379439+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"94'139","reported_seq":2101,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:44.273009+0000","last_change":"2024-05-15T13:31:40.877440+0000","last_active":"2024-05-15T13:32:44.273009+0000","last_peered":"2024-05-15T13:32:44.273009+0000","last_clean":"2024-05-15T13:32:44.273009+0000","last_became_active":"2024-05-15T13:25:58.961079+0000","last_became_peered":"2024-05-15T13:25:58.961079+0000","last_unstale":"2024-05-15T13:32:44.273009+0000","last_undegraded":"2024-05-15T13:32:44.273009+0000","last_fullsized":"2024-05-15T13:32:44.273009+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:40.877342+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:31:40.877342+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:06.601679+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"94'139","reported_seq":1809,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:43.185924+0000","last_change":"2024-05-15T13:31:30.381607+0000","last_active":"2024-05-15T13:32:43.185924+0000","last_peered":"2024-05-15T13:32:43.185924+0000","last_clean":"2024-05-15T13:32:43.185924+0000","last_became_active":"2024-05-15T13:30:15.244321+0000","last_became_peered":"2024-05-15T13:30:15.244321+0000","last_unstale":"2024-05-15T13:32:43.185924+0000","last_undegraded":"2024-05-15T13:32:43.185924+0000","last_fullsized":"2024-05-15T13:32:43.185924+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:30.381473+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:31:30.381473+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:32:56.084958+0000","scrub_duration":238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"94'139","reported_seq":2067,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:44.457143+0000","last_change":"2024-05-15T13:32:02.783642+0000","last_active":"2024-05-15T13:32:44.457143+0000","last_peered":"2024-05-15T13:32:44.457143+0000","last_clean":"2024-05-15T13:32:44.457143+0000","last_became_active":"2024-05-15T13:24:06.307697+0000","last_became_peered":"2024-05-15T13:24:06.307697+0000","last_unstale":"2024-05-15T13:32:44.457143+0000","last_undegraded":"2024-05-15T13:32:44.457143+0000","last_fullsized":"2024-05-15T13:32:44.457143+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:02.783509+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:32:02.783509+0000","last_clean_scrub_stamp":"2024-05-15T13:32:02.783509+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:19.902420+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"94'139","reported_seq":1441,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:44.272976+0000","last_change":"2024-05-15T13:31:29.988433+0000","last_active":"2024-05-15T13:32:44.272976+0000","last_peered":"2024-05-15T13:32:44.272976+0000","last_clean":"2024-05-15T13:32:44.272976+0000","last_became_active":"2024-05-15T13:24:06.300367+0000","last_became_peered":"2024-05-15T13:24:06.300367+0000","last_unstale":"2024-05-15T13:32:44.272976+0000","last_undegraded":"2024-05-15T13:32:44.272976+0000","last_fullsized":"2024-05-15T13:32:44.272976+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:29.988318+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:31:29.988318+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:32:46.787263+0000","scrub_duration":165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"94'139","reported_seq":1865,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:42.610208+0000","last_change":"2024-05-15T13:32:28.970505+0000","last_active":"2024-05-15T13:32:42.610208+0000","last_peered":"2024-05-15T13:32:42.610208+0000","last_clean":"2024-05-15T13:32:42.610208+0000","last_became_active":"2024-05-15T13:25:58.956902+0000","last_became_peered":"2024-05-15T13:25:58.956902+0000","last_unstale":"2024-05-15T13:32:42.610208+0000","last_undegraded":"2024-05-15T13:32:42.610208+0000","last_fullsized":"2024-05-15T13:32:42.610208+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:28.970402+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:22:03.418345+0000","last_clean_scrub_stamp":"2024-05-15T13:32:28.970402+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:44.742009+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.e","version":"94'139","reported_seq":1328,"reported_epoch":1046,"state":"active+clean","last_fresh":"2024-05-15T13:32:36.430132+0000","last_change":"2024-05-15T13:32:27.808010+0000","last_active":"2024-05-15T13:32:36.430132+0000","last_peered":"2024-05-15T13:32:36.430132+0000","last_clean":"2024-05-15T13:32:36.430132+0000","last_became_active":"2024-05-15T13:25:58.962490+0000","last_became_peered":"2024-05-15T13:25:58.962490+0000","last_unstale":"2024-05-15T13:32:36.430132+0000","last_undegraded":"2024-05-15T13:32:36.430132+0000","last_fullsized":"2024-05-15T13:32:36.430132+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:27.807901+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:23:06.819156+0000","last_clean_scrub_stamp":"2024-05-15T13:32:27.807901+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:32.269313+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"94'139","reported_seq":1844,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:41.962672+0000","last_change":"2024-05-15T13:31:35.563524+0000","last_active":"2024-05-15T13:32:41.962672+0000","last_peered":"2024-05-15T13:32:41.962672+0000","last_clean":"2024-05-15T13:32:41.962672+0000","last_became_active":"2024-05-15T13:25:58.982018+0000","last_became_peered":"2024-05-15T13:25:58.982018+0000","last_unstale":"2024-05-15T13:32:41.962672+0000","last_undegraded":"2024-05-15T13:32:41.962672+0000","last_fullsized":"2024-05-15T13:32:41.962672+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:35.563348+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:31:35.563348+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:32:44.570636+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"94'139","reported_seq":1487,"reported_epoch":1046,"state":"active+clean","last_fresh":"2024-05-15T13:32:37.870821+0000","last_change":"2024-05-15T13:32:26.772250+0000","last_active":"2024-05-15T13:32:37.870821+0000","last_peered":"2024-05-15T13:32:37.870821+0000","last_clean":"2024-05-15T13:32:37.870821+0000","last_became_active":"2024-05-15T13:24:06.308068+0000","last_became_peered":"2024-05-15T13:24:06.308068+0000","last_unstale":"2024-05-15T13:32:37.870821+0000","last_undegraded":"2024-05-15T13:32:37.870821+0000","last_fullsized":"2024-05-15T13:32:37.870821+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:26.772075+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:32:26.772075+0000","last_clean_scrub_stamp":"2024-05-15T13:32:26.772075+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:36.853392+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"94'139","reported_seq":1550,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:44.273129+0000","last_change":"2024-05-15T13:32:22.183549+0000","last_active":"2024-05-15T13:32:44.273129+0000","last_peered":"2024-05-15T13:32:44.273129+0000","last_clean":"2024-05-15T13:32:44.273129+0000","last_became_active":"2024-05-15T13:30:15.242131+0000","last_became_peered":"2024-05-15T13:30:15.242131+0000","last_unstale":"2024-05-15T13:32:44.273129+0000","last_undegraded":"2024-05-15T13:32:44.273129+0000","last_fullsized":"2024-05-15T13:32:44.273129+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:22.183442+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:23:03.856199+0000","last_clean_scrub_stamp":"2024-05-15T13:32:22.183442+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:30.820445+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"94'139","reported_seq":1447,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:41.962205+0000","last_change":"2024-05-15T13:31:53.033654+0000","last_active":"2024-05-15T13:32:41.962205+0000","last_peered":"2024-05-15T13:32:41.962205+0000","last_clean":"2024-05-15T13:32:41.962205+0000","last_became_active":"2024-05-15T13:28:01.353258+0000","last_became_peered":"2024-05-15T13:28:01.353258+0000","last_unstale":"2024-05-15T13:32:41.962205+0000","last_undegraded":"2024-05-15T13:32:41.962205+0000","last_fullsized":"2024-05-15T13:32:41.962205+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:53.033534+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:31:53.033534+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:08.511951+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"94'139","reported_seq":2085,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:44.457097+0000","last_change":"2024-05-15T13:32:29.807888+0000","last_active":"2024-05-15T13:32:44.457097+0000","last_peered":"2024-05-15T13:32:44.457097+0000","last_clean":"2024-05-15T13:32:44.457097+0000","last_became_active":"2024-05-15T13:19:41.190923+0000","last_became_peered":"2024-05-15T13:19:41.190923+0000","last_unstale":"2024-05-15T13:32:44.457097+0000","last_undegraded":"2024-05-15T13:32:44.457097+0000","last_fullsized":"2024-05-15T13:32:44.457097+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:29.807776+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:29:49.795649+0000","last_clean_scrub_stamp":"2024-05-15T13:32:29.807776+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:43.066650+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"94'139","reported_seq":1862,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:42.312915+0000","last_change":"2024-05-15T13:32:32.279549+0000","last_active":"2024-05-15T13:32:42.312915+0000","last_peered":"2024-05-15T13:32:42.312915+0000","last_clean":"2024-05-15T13:32:42.312915+0000","last_became_active":"2024-05-15T13:27:32.378438+0000","last_became_peered":"2024-05-15T13:27:32.378438+0000","last_unstale":"2024-05-15T13:32:42.312915+0000","last_undegraded":"2024-05-15T13:32:42.312915+0000","last_fullsized":"2024-05-15T13:32:42.312915+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:32.279440+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:32:32.279440+0000","last_clean_scrub_stamp":"2024-05-15T13:32:32.279440+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:41.509221+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"94'139","reported_seq":2109,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:44.457232+0000","last_change":"2024-05-15T13:32:30.862428+0000","last_active":"2024-05-15T13:32:44.457232+0000","last_peered":"2024-05-15T13:32:44.457232+0000","last_clean":"2024-05-15T13:32:44.457232+0000","last_became_active":"2024-05-15T13:30:15.244033+0000","last_became_peered":"2024-05-15T13:30:15.244033+0000","last_unstale":"2024-05-15T13:32:44.457232+0000","last_undegraded":"2024-05-15T13:32:44.457232+0000","last_fullsized":"2024-05-15T13:32:44.457232+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:30.862279+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:32:30.862279+0000","last_clean_scrub_stamp":"2024-05-15T13:32:30.862279+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:48.030677+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"94'139","reported_seq":1277,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:43.186029+0000","last_change":"2024-05-15T13:31:36.554068+0000","last_active":"2024-05-15T13:32:43.186029+0000","last_peered":"2024-05-15T13:32:43.186029+0000","last_clean":"2024-05-15T13:32:43.186029+0000","last_became_active":"2024-05-15T13:30:15.245741+0000","last_became_peered":"2024-05-15T13:30:15.245741+0000","last_unstale":"2024-05-15T13:32:43.186029+0000","last_undegraded":"2024-05-15T13:32:43.186029+0000","last_fullsized":"2024-05-15T13:32:43.186029+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:36.553956+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:29:10.647536+0000","last_clean_scrub_stamp":"2024-05-15T13:31:36.553956+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057209,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":934352,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":933566,"kb_avail":92810800,"statfs":{"total":95995035648,"available":95038259200,"internally_reserved":0,"allocated":790528,"data_stored":460634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":955972038},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351549,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":942660,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":941886,"kb_avail":92802492,"statfs":{"total":95995035648,"available":95029751808,"internally_reserved":0,"allocated":778240,"data_stored":453147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":964491712},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417515079,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1181456,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":1180670,"kb_avail":92563696,"statfs":{"total":95995035648,"available":94785224704,"internally_reserved":0,"allocated":790528,"data_stored":460634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":1209006528},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874824,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":537296,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93207856,"statfs":{"total":95995035648,"available":95444844544,"internally_reserved":0,"allocated":790528,"data_stored":460634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169166,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":415320,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93329832,"statfs":{"total":95995035648,"available":95569747968,"internally_reserved":0,"allocated":995328,"data_stored":661882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":424278474},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430804,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":794448,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":92950704,"statfs":{"total":95995035648,"available":95181520896,"internally_reserved":0,"allocated":1118208,"data_stored":788314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":812382662},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575469,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1118992,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":1118206,"kb_avail":92626160,"statfs":{"total":95995035648,"available":94849187840,"internally_reserved":0,"allocated":790528,"data_stored":460634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1145043397},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738998,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":888848,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92856304,"statfs":{"total":95995035648,"available":95084855296,"internally_reserved":0,"allocated":1118208,"data_stored":788314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":909048261},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411944,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":623564,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93121588,"statfs":{"total":95995035648,"available":95356506112,"internally_reserved":0,"allocated":983040,"data_stored":654395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150306,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1195652,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":1194558,"kb_avail":92549500,"statfs":{"total":95995035648,"available":94770688000,"internally_reserved":0,"allocated":1105920,"data_stored":780827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1223227847},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":887,"seq":3809635992149,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":695556,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93049596,"statfs":{"total":95995035648,"available":95282786304,"internally_reserved":0,"allocated":778240,"data_stored":453147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692441,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":956760,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":955774,"kb_avail":92788392,"statfs":{"total":95995035648,"available":95015313408,"internally_reserved":0,"allocated":995328,"data_stored":661882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":978713029},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:32:49.265 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T13:32:49.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:48 smithi120 ceph-mon[43961]: 1.f deep-scrub starts 2024-05-15T13:32:49.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:48 smithi120 ceph-mon[43961]: 1.f deep-scrub ok 2024-05-15T13:32:49.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:48 smithi177 ceph-mon[44303]: 1.f deep-scrub starts 2024-05-15T13:32:49.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:48 smithi177 ceph-mon[44303]: 1.f deep-scrub ok 2024-05-15T13:32:49.550 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:32:50.033 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:49 smithi027 ceph-mon[39795]: purged_snaps scrub starts 2024-05-15T13:32:50.034 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:49 smithi027 ceph-mon[39795]: purged_snaps scrub ok 2024-05-15T13:32:50.034 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:49 smithi027 ceph-mon[39795]: from='client.17430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:32:50.034 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:49 smithi027 ceph-mon[39795]: pgmap v2642: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:50.034 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:49 smithi027 ceph-mon[39795]: osdmap e1049: 12 total, 12 up, 12 in 2024-05-15T13:32:50.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:49 smithi120 ceph-mon[43961]: purged_snaps scrub starts 2024-05-15T13:32:50.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:49 smithi120 ceph-mon[43961]: purged_snaps scrub ok 2024-05-15T13:32:50.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:49 smithi120 ceph-mon[43961]: from='client.17430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:32:50.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:49 smithi120 ceph-mon[43961]: pgmap v2642: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:50.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:49 smithi120 ceph-mon[43961]: osdmap e1049: 12 total, 12 up, 12 in 2024-05-15T13:32:50.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:49 smithi177 ceph-mon[44303]: purged_snaps scrub starts 2024-05-15T13:32:50.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:49 smithi177 ceph-mon[44303]: purged_snaps scrub ok 2024-05-15T13:32:50.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:49 smithi177 ceph-mon[44303]: from='client.17430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:32:50.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:49 smithi177 ceph-mon[44303]: pgmap v2642: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:50.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:49 smithi177 ceph-mon[44303]: osdmap e1049: 12 total, 12 up, 12 in 2024-05-15T13:32:51.079 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:50 smithi027 ceph-mon[39795]: 1.9 scrub starts 2024-05-15T13:32:51.079 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:50 smithi027 ceph-mon[39795]: 1.9 scrub ok 2024-05-15T13:32:51.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:50 smithi120 ceph-mon[43961]: 1.9 scrub starts 2024-05-15T13:32:51.294 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:50 smithi120 ceph-mon[43961]: 1.9 scrub ok 2024-05-15T13:32:51.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:50 smithi177 ceph-mon[44303]: 1.9 scrub starts 2024-05-15T13:32:51.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:50 smithi177 ceph-mon[44303]: 1.9 scrub ok 2024-05-15T13:32:51.562 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-05-15T13:32:51.563 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-05-15T13:32:52.180 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:51 smithi027 ceph-mon[39795]: pgmap v2644: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:52.206 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2644,"stamp":"2024-05-15T13:32:50.834427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10284932,"kb_used_data":10200,"kb_used_omap":18,"kb_used_meta":10273965,"kb_avail":1114656892,"statfs":{"total":1151940427776,"available":1141408657408,"internally_reserved":0,"allocated":10444800,"data_stored":6719644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":10520540475},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001748"},"pg_stats":[{"pgid":"1.1e","version":"94'139","reported_seq":1273,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-05-15T13:32:47.358165+0000","last_change":"2024-05-15T13:31:38.748656+0000","last_active":"2024-05-15T13:32:47.358165+0000","last_peered":"2024-05-15T13:32:47.358165+0000","last_clean":"2024-05-15T13:32:47.358165+0000","last_became_active":"2024-05-15T13:28:01.363009+0000","last_became_peered":"2024-05-15T13:28:01.363009+0000","last_unstale":"2024-05-15T13:32:47.358165+0000","last_undegraded":"2024-05-15T13:32:47.358165+0000","last_fullsized":"2024-05-15T13:32:47.358165+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:38.748530+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:29:11.559293+0000","last_clean_scrub_stamp":"2024-05-15T13:31:38.748530+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:32:55.235801+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"94'139","reported_seq":1049,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:43.186033+0000","last_change":"2024-05-15T13:32:08.435536+0000","last_active":"2024-05-15T13:32:43.186033+0000","last_peered":"2024-05-15T13:32:43.186033+0000","last_clean":"2024-05-15T13:32:43.186033+0000","last_became_active":"2024-05-15T13:30:15.243303+0000","last_became_peered":"2024-05-15T13:30:15.243303+0000","last_unstale":"2024-05-15T13:32:43.186033+0000","last_undegraded":"2024-05-15T13:32:43.186033+0000","last_fullsized":"2024-05-15T13:32:43.186033+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:08.435428+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:32:08.435428+0000","last_clean_scrub_stamp":"2024-05-15T13:32:08.435428+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:32.056821+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"94'139","reported_seq":1042,"reported_epoch":1049,"state":"active+clean","last_fresh":"2024-05-15T13:32:49.091975+0000","last_change":"2024-05-15T13:32:30.925839+0000","last_active":"2024-05-15T13:32:49.091975+0000","last_peered":"2024-05-15T13:32:49.091975+0000","last_clean":"2024-05-15T13:32:49.091975+0000","last_became_active":"2024-05-15T13:30:15.244002+0000","last_became_peered":"2024-05-15T13:30:15.244002+0000","last_unstale":"2024-05-15T13:32:49.091975+0000","last_undegraded":"2024-05-15T13:32:49.091975+0000","last_fullsized":"2024-05-15T13:32:49.091975+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:30.925686+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:11:04.688445+0000","last_clean_scrub_stamp":"2024-05-15T13:32:30.925686+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:59.260209+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"94'139","reported_seq":1435,"reported_epoch":1049,"state":"active+clean","last_fresh":"2024-05-15T13:32:49.091616+0000","last_change":"2024-05-15T13:32:04.836769+0000","last_active":"2024-05-15T13:32:49.091616+0000","last_peered":"2024-05-15T13:32:49.091616+0000","last_clean":"2024-05-15T13:32:49.091616+0000","last_became_active":"2024-05-15T13:27:32.380219+0000","last_became_peered":"2024-05-15T13:27:32.380219+0000","last_unstale":"2024-05-15T13:32:49.091616+0000","last_undegraded":"2024-05-15T13:32:49.091616+0000","last_fullsized":"2024-05-15T13:32:49.091616+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:04.836624+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:32:04.836624+0000","last_clean_scrub_stamp":"2024-05-15T13:32:04.836624+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:06.947250+0000","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"94'139","reported_seq":1483,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-05-15T13:32:47.336620+0000","last_change":"2024-05-15T13:32:16.616896+0000","last_active":"2024-05-15T13:32:47.336620+0000","last_peered":"2024-05-15T13:32:47.336620+0000","last_clean":"2024-05-15T13:32:47.336620+0000","last_became_active":"2024-05-15T13:30:15.241152+0000","last_became_peered":"2024-05-15T13:30:15.241152+0000","last_unstale":"2024-05-15T13:32:47.336620+0000","last_undegraded":"2024-05-15T13:32:47.336620+0000","last_fullsized":"2024-05-15T13:32:47.336620+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:16.616790+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:22:03.418345+0000","last_clean_scrub_stamp":"2024-05-15T13:32:16.616790+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:20.235908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"94'139","reported_seq":1257,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:42.372618+0000","last_change":"2024-05-15T13:32:17.841086+0000","last_active":"2024-05-15T13:32:42.372618+0000","last_peered":"2024-05-15T13:32:42.372618+0000","last_clean":"2024-05-15T13:32:42.372618+0000","last_became_active":"2024-05-15T13:24:11.374739+0000","last_became_peered":"2024-05-15T13:24:11.374739+0000","last_unstale":"2024-05-15T13:32:42.372618+0000","last_undegraded":"2024-05-15T13:32:42.372618+0000","last_fullsized":"2024-05-15T13:32:42.372618+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:17.840955+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:32:17.840955+0000","last_clean_scrub_stamp":"2024-05-15T13:32:17.840955+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:37.140927+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"94'139","reported_seq":2079,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:44.456959+0000","last_change":"2024-05-15T13:32:07.854793+0000","last_active":"2024-05-15T13:32:44.456959+0000","last_peered":"2024-05-15T13:32:44.456959+0000","last_clean":"2024-05-15T13:32:44.456959+0000","last_became_active":"2024-05-15T13:24:11.368347+0000","last_became_peered":"2024-05-15T13:24:11.368347+0000","last_unstale":"2024-05-15T13:32:44.456959+0000","last_undegraded":"2024-05-15T13:32:44.456959+0000","last_fullsized":"2024-05-15T13:32:44.456959+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:07.854670+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:32:07.854670+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:15.750799+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"94'139","reported_seq":818,"reported_epoch":1049,"state":"active+clean","last_fresh":"2024-05-15T13:32:49.091857+0000","last_change":"2024-05-15T13:32:32.104855+0000","last_active":"2024-05-15T13:32:49.091857+0000","last_peered":"2024-05-15T13:32:49.091857+0000","last_clean":"2024-05-15T13:32:49.091857+0000","last_became_active":"2024-05-15T13:27:32.410771+0000","last_became_peered":"2024-05-15T13:27:32.410771+0000","last_unstale":"2024-05-15T13:32:49.091857+0000","last_undegraded":"2024-05-15T13:32:49.091857+0000","last_fullsized":"2024-05-15T13:32:49.091857+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:32.104724+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:32:32.104724+0000","last_clean_scrub_stamp":"2024-05-15T13:32:32.104724+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:48.717633+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"94'139","reported_seq":2009,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:43.186031+0000","last_change":"2024-05-15T13:32:24.225813+0000","last_active":"2024-05-15T13:32:43.186031+0000","last_peered":"2024-05-15T13:32:43.186031+0000","last_clean":"2024-05-15T13:32:43.186031+0000","last_became_active":"2024-05-15T13:30:15.242583+0000","last_became_peered":"2024-05-15T13:30:15.242583+0000","last_unstale":"2024-05-15T13:32:43.186031+0000","last_undegraded":"2024-05-15T13:32:43.186031+0000","last_fullsized":"2024-05-15T13:32:43.186031+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:24.225676+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:32:24.225676+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:43.891194+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"94'139","reported_seq":1010,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:44.273120+0000","last_change":"2024-05-15T13:31:48.952667+0000","last_active":"2024-05-15T13:32:44.273120+0000","last_peered":"2024-05-15T13:32:44.273120+0000","last_clean":"2024-05-15T13:32:44.273120+0000","last_became_active":"2024-05-15T13:27:32.397907+0000","last_became_peered":"2024-05-15T13:27:32.397907+0000","last_unstale":"2024-05-15T13:32:44.273120+0000","last_undegraded":"2024-05-15T13:32:44.273120+0000","last_fullsized":"2024-05-15T13:32:44.273120+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":834,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:48.952562+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:30:44.113972+0000","last_clean_scrub_stamp":"2024-05-15T13:31:48.952562+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:32:54.381731+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"94'139","reported_seq":1527,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-05-15T13:32:46.980510+0000","last_change":"2024-05-15T13:32:44.895587+0000","last_active":"2024-05-15T13:32:46.980510+0000","last_peered":"2024-05-15T13:32:46.980510+0000","last_clean":"2024-05-15T13:32:46.980510+0000","last_became_active":"2024-05-15T13:19:41.190796+0000","last_became_peered":"2024-05-15T13:19:41.190796+0000","last_unstale":"2024-05-15T13:32:46.980510+0000","last_undegraded":"2024-05-15T13:32:46.980510+0000","last_fullsized":"2024-05-15T13:32:46.980510+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:44.895483+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:31:21.705423+0000","last_clean_scrub_stamp":"2024-05-15T13:32:44.895483+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:34:14.777711+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"94'139","reported_seq":665,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:43.185912+0000","last_change":"2024-05-15T13:31:35.322228+0000","last_active":"2024-05-15T13:32:43.185912+0000","last_peered":"2024-05-15T13:32:43.185912+0000","last_clean":"2024-05-15T13:32:43.185912+0000","last_became_active":"2024-05-15T13:30:15.245704+0000","last_became_peered":"2024-05-15T13:30:15.245704+0000","last_unstale":"2024-05-15T13:32:43.185912+0000","last_undegraded":"2024-05-15T13:32:43.185912+0000","last_fullsized":"2024-05-15T13:32:43.185912+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:35.322078+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:31:35.322078+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,10],"acting":[1,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"94'139","reported_seq":1794,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-05-15T13:32:47.357992+0000","last_change":"2024-05-15T13:32:06.944608+0000","last_active":"2024-05-15T13:32:47.357992+0000","last_peered":"2024-05-15T13:32:47.357992+0000","last_clean":"2024-05-15T13:32:47.357992+0000","last_became_active":"2024-05-15T13:28:01.357970+0000","last_became_peered":"2024-05-15T13:28:01.357970+0000","last_unstale":"2024-05-15T13:32:47.357992+0000","last_undegraded":"2024-05-15T13:32:47.357992+0000","last_fullsized":"2024-05-15T13:32:47.357992+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:06.944458+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:32:06.944458+0000","last_clean_scrub_stamp":"2024-05-15T13:32:06.944458+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:08.024542+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"94'139","reported_seq":1825,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:42.372654+0000","last_change":"2024-05-15T13:31:59.808857+0000","last_active":"2024-05-15T13:32:42.372654+0000","last_peered":"2024-05-15T13:32:42.372654+0000","last_clean":"2024-05-15T13:32:42.372654+0000","last_became_active":"2024-05-15T13:24:06.307598+0000","last_became_peered":"2024-05-15T13:24:06.307598+0000","last_unstale":"2024-05-15T13:32:42.372654+0000","last_undegraded":"2024-05-15T13:32:42.372654+0000","last_fullsized":"2024-05-15T13:32:42.372654+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:59.808705+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:18:24.970285+0000","last_clean_scrub_stamp":"2024-05-15T13:31:59.808705+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:15.174182+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"94'139","reported_seq":569,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:42.372653+0000","last_change":"2024-05-15T13:31:38.917101+0000","last_active":"2024-05-15T13:32:42.372653+0000","last_peered":"2024-05-15T13:32:42.372653+0000","last_clean":"2024-05-15T13:32:42.372653+0000","last_became_active":"2024-05-15T13:31:37.910916+0000","last_became_peered":"2024-05-15T13:31:37.910916+0000","last_unstale":"2024-05-15T13:32:42.372653+0000","last_undegraded":"2024-05-15T13:32:42.372653+0000","last_fullsized":"2024-05-15T13:32:42.372653+0000","mapping_epoch":1034,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":1035,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:38.916950+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:31:38.916950+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:00.059064+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,8],"acting":[10,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"94'139","reported_seq":1844,"reported_epoch":1049,"state":"active+clean","last_fresh":"2024-05-15T13:32:49.091644+0000","last_change":"2024-05-15T13:31:43.658069+0000","last_active":"2024-05-15T13:32:49.091644+0000","last_peered":"2024-05-15T13:32:49.091644+0000","last_clean":"2024-05-15T13:32:49.091644+0000","last_became_active":"2024-05-15T13:24:06.312706+0000","last_became_peered":"2024-05-15T13:24:06.312706+0000","last_unstale":"2024-05-15T13:32:49.091644+0000","last_undegraded":"2024-05-15T13:32:49.091644+0000","last_fullsized":"2024-05-15T13:32:49.091644+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:43.657969+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:31:43.657969+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:05.004139+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"94'139","reported_seq":569,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:42.372634+0000","last_change":"2024-05-15T13:31:39.936236+0000","last_active":"2024-05-15T13:32:42.372634+0000","last_peered":"2024-05-15T13:32:42.372634+0000","last_clean":"2024-05-15T13:32:42.372634+0000","last_became_active":"2024-05-15T13:31:37.910962+0000","last_became_peered":"2024-05-15T13:31:37.910962+0000","last_unstale":"2024-05-15T13:32:42.372634+0000","last_undegraded":"2024-05-15T13:32:42.372634+0000","last_fullsized":"2024-05-15T13:32:42.372634+0000","mapping_epoch":1034,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":1035,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:39.936110+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:31:39.936110+0000","last_clean_scrub_stamp":"2024-05-15T13:31:39.936110+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:32:58.379439+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"94'139","reported_seq":2101,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:44.273009+0000","last_change":"2024-05-15T13:31:40.877440+0000","last_active":"2024-05-15T13:32:44.273009+0000","last_peered":"2024-05-15T13:32:44.273009+0000","last_clean":"2024-05-15T13:32:44.273009+0000","last_became_active":"2024-05-15T13:25:58.961079+0000","last_became_peered":"2024-05-15T13:25:58.961079+0000","last_unstale":"2024-05-15T13:32:44.273009+0000","last_undegraded":"2024-05-15T13:32:44.273009+0000","last_fullsized":"2024-05-15T13:32:44.273009+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:40.877342+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:31:40.877342+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:06.601679+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"94'139","reported_seq":1809,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:43.185924+0000","last_change":"2024-05-15T13:31:30.381607+0000","last_active":"2024-05-15T13:32:43.185924+0000","last_peered":"2024-05-15T13:32:43.185924+0000","last_clean":"2024-05-15T13:32:43.185924+0000","last_became_active":"2024-05-15T13:30:15.244321+0000","last_became_peered":"2024-05-15T13:30:15.244321+0000","last_unstale":"2024-05-15T13:32:43.185924+0000","last_undegraded":"2024-05-15T13:32:43.185924+0000","last_fullsized":"2024-05-15T13:32:43.185924+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":3,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:30.381473+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:31:30.381473+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:32:56.084958+0000","scrub_duration":238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"94'139","reported_seq":2067,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:44.457143+0000","last_change":"2024-05-15T13:32:02.783642+0000","last_active":"2024-05-15T13:32:44.457143+0000","last_peered":"2024-05-15T13:32:44.457143+0000","last_clean":"2024-05-15T13:32:44.457143+0000","last_became_active":"2024-05-15T13:24:06.307697+0000","last_became_peered":"2024-05-15T13:24:06.307697+0000","last_unstale":"2024-05-15T13:32:44.457143+0000","last_undegraded":"2024-05-15T13:32:44.457143+0000","last_fullsized":"2024-05-15T13:32:44.457143+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:02.783509+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:32:02.783509+0000","last_clean_scrub_stamp":"2024-05-15T13:32:02.783509+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:19.902420+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"94'139","reported_seq":1441,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:44.272976+0000","last_change":"2024-05-15T13:31:29.988433+0000","last_active":"2024-05-15T13:32:44.272976+0000","last_peered":"2024-05-15T13:32:44.272976+0000","last_clean":"2024-05-15T13:32:44.272976+0000","last_became_active":"2024-05-15T13:24:06.300367+0000","last_became_peered":"2024-05-15T13:24:06.300367+0000","last_unstale":"2024-05-15T13:32:44.272976+0000","last_undegraded":"2024-05-15T13:32:44.272976+0000","last_fullsized":"2024-05-15T13:32:44.272976+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:29.988318+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:31:29.988318+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:32:46.787263+0000","scrub_duration":165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"94'139","reported_seq":1865,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:42.610208+0000","last_change":"2024-05-15T13:32:28.970505+0000","last_active":"2024-05-15T13:32:42.610208+0000","last_peered":"2024-05-15T13:32:42.610208+0000","last_clean":"2024-05-15T13:32:42.610208+0000","last_became_active":"2024-05-15T13:25:58.956902+0000","last_became_peered":"2024-05-15T13:25:58.956902+0000","last_unstale":"2024-05-15T13:32:42.610208+0000","last_undegraded":"2024-05-15T13:32:42.610208+0000","last_fullsized":"2024-05-15T13:32:42.610208+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:28.970402+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:22:03.418345+0000","last_clean_scrub_stamp":"2024-05-15T13:32:28.970402+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:44.742009+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.e","version":"94'139","reported_seq":1332,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-05-15T13:32:47.336689+0000","last_change":"2024-05-15T13:32:27.808010+0000","last_active":"2024-05-15T13:32:47.336689+0000","last_peered":"2024-05-15T13:32:47.336689+0000","last_clean":"2024-05-15T13:32:47.336689+0000","last_became_active":"2024-05-15T13:25:58.962490+0000","last_became_peered":"2024-05-15T13:25:58.962490+0000","last_unstale":"2024-05-15T13:32:47.336689+0000","last_undegraded":"2024-05-15T13:32:47.336689+0000","last_fullsized":"2024-05-15T13:32:47.336689+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:27.807901+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:23:06.819156+0000","last_clean_scrub_stamp":"2024-05-15T13:32:27.807901+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:32.269313+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"94'139","reported_seq":1856,"reported_epoch":1049,"state":"active+clean","last_fresh":"2024-05-15T13:32:49.091768+0000","last_change":"2024-05-15T13:32:47.667969+0000","last_active":"2024-05-15T13:32:49.091768+0000","last_peered":"2024-05-15T13:32:49.091768+0000","last_clean":"2024-05-15T13:32:49.091768+0000","last_became_active":"2024-05-15T13:25:58.982018+0000","last_became_peered":"2024-05-15T13:25:58.982018+0000","last_unstale":"2024-05-15T13:32:49.091768+0000","last_undegraded":"2024-05-15T13:32:49.091768+0000","last_fullsized":"2024-05-15T13:32:49.091768+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:47.667861+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:32:47.667861+0000","last_clean_scrub_stamp":"2024-05-15T13:32:47.667861+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:51.907539+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"94'139","reported_seq":1491,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-05-15T13:32:47.357828+0000","last_change":"2024-05-15T13:32:26.772250+0000","last_active":"2024-05-15T13:32:47.357828+0000","last_peered":"2024-05-15T13:32:47.357828+0000","last_clean":"2024-05-15T13:32:47.357828+0000","last_became_active":"2024-05-15T13:24:06.308068+0000","last_became_peered":"2024-05-15T13:24:06.308068+0000","last_unstale":"2024-05-15T13:32:47.357828+0000","last_undegraded":"2024-05-15T13:32:47.357828+0000","last_fullsized":"2024-05-15T13:32:47.357828+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:26.772075+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:32:26.772075+0000","last_clean_scrub_stamp":"2024-05-15T13:32:26.772075+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:36.853392+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"94'139","reported_seq":1550,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:44.273129+0000","last_change":"2024-05-15T13:32:22.183549+0000","last_active":"2024-05-15T13:32:44.273129+0000","last_peered":"2024-05-15T13:32:44.273129+0000","last_clean":"2024-05-15T13:32:44.273129+0000","last_became_active":"2024-05-15T13:30:15.242131+0000","last_became_peered":"2024-05-15T13:30:15.242131+0000","last_unstale":"2024-05-15T13:32:44.273129+0000","last_undegraded":"2024-05-15T13:32:44.273129+0000","last_fullsized":"2024-05-15T13:32:44.273129+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:22.183442+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:23:03.856199+0000","last_clean_scrub_stamp":"2024-05-15T13:32:22.183442+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:30.820445+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"94'139","reported_seq":1449,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-05-15T13:32:47.358408+0000","last_change":"2024-05-15T13:31:53.033654+0000","last_active":"2024-05-15T13:32:47.358408+0000","last_peered":"2024-05-15T13:32:47.358408+0000","last_clean":"2024-05-15T13:32:47.358408+0000","last_became_active":"2024-05-15T13:28:01.353258+0000","last_became_peered":"2024-05-15T13:28:01.353258+0000","last_unstale":"2024-05-15T13:32:47.358408+0000","last_undegraded":"2024-05-15T13:32:47.358408+0000","last_fullsized":"2024-05-15T13:32:47.358408+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":4,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:53.033534+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T12:45:57.126994+0000","last_clean_scrub_stamp":"2024-05-15T13:31:53.033534+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:08.511951+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"94'139","reported_seq":2085,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:44.457097+0000","last_change":"2024-05-15T13:32:29.807888+0000","last_active":"2024-05-15T13:32:44.457097+0000","last_peered":"2024-05-15T13:32:44.457097+0000","last_clean":"2024-05-15T13:32:44.457097+0000","last_became_active":"2024-05-15T13:19:41.190923+0000","last_became_peered":"2024-05-15T13:19:41.190923+0000","last_unstale":"2024-05-15T13:32:44.457097+0000","last_undegraded":"2024-05-15T13:32:44.457097+0000","last_fullsized":"2024-05-15T13:32:44.457097+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:29.807776+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:29:49.795649+0000","last_clean_scrub_stamp":"2024-05-15T13:32:29.807776+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:43.066650+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"94'139","reported_seq":1866,"reported_epoch":1049,"state":"active+clean","last_fresh":"2024-05-15T13:32:48.986483+0000","last_change":"2024-05-15T13:32:32.279549+0000","last_active":"2024-05-15T13:32:48.986483+0000","last_peered":"2024-05-15T13:32:48.986483+0000","last_clean":"2024-05-15T13:32:48.986483+0000","last_became_active":"2024-05-15T13:27:32.378438+0000","last_became_peered":"2024-05-15T13:27:32.378438+0000","last_unstale":"2024-05-15T13:32:48.986483+0000","last_undegraded":"2024-05-15T13:32:48.986483+0000","last_fullsized":"2024-05-15T13:32:48.986483+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:32.279440+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:32:32.279440+0000","last_clean_scrub_stamp":"2024-05-15T13:32:32.279440+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:41.509221+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"94'139","reported_seq":2109,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:44.457232+0000","last_change":"2024-05-15T13:32:30.862428+0000","last_active":"2024-05-15T13:32:44.457232+0000","last_peered":"2024-05-15T13:32:44.457232+0000","last_clean":"2024-05-15T13:32:44.457232+0000","last_became_active":"2024-05-15T13:30:15.244033+0000","last_became_peered":"2024-05-15T13:30:15.244033+0000","last_unstale":"2024-05-15T13:32:44.457232+0000","last_undegraded":"2024-05-15T13:32:44.457232+0000","last_fullsized":"2024-05-15T13:32:44.457232+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:32:30.862279+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:32:30.862279+0000","last_clean_scrub_stamp":"2024-05-15T13:32:30.862279+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T13:33:48.030677+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"94'139","reported_seq":1277,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-05-15T13:32:43.186029+0000","last_change":"2024-05-15T13:31:36.554068+0000","last_active":"2024-05-15T13:32:43.186029+0000","last_peered":"2024-05-15T13:32:43.186029+0000","last_clean":"2024-05-15T13:32:43.186029+0000","last_became_active":"2024-05-15T13:30:15.245741+0000","last_became_peered":"2024-05-15T13:30:15.245741+0000","last_unstale":"2024-05-15T13:32:43.186029+0000","last_undegraded":"2024-05-15T13:32:43.186029+0000","last_fullsized":"2024-05-15T13:32:43.186029+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'139","last_scrub_stamp":"2024-05-15T13:31:36.553956+0000","last_deep_scrub":"94'139","last_deep_scrub_stamp":"2024-05-15T13:29:10.647536+0000","last_clean_scrub_stamp":"2024-05-15T13:31:36.553956+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057210,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":934364,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":933566,"kb_avail":92810788,"statfs":{"total":95995035648,"available":95038246912,"internally_reserved":0,"allocated":679936,"data_stored":392347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":955972038},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":81,"seq":347892351550,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":942672,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":941886,"kb_avail":92802480,"statfs":{"total":95995035648,"available":95029739520,"internally_reserved":0,"allocated":790528,"data_stored":460634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":964491712},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417515079,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1181456,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":1180670,"kb_avail":92563696,"statfs":{"total":95995035648,"available":94785224704,"internally_reserved":0,"allocated":790528,"data_stored":460634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":1209006528},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874825,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":537308,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93207844,"statfs":{"total":95995035648,"available":95444832256,"internally_reserved":0,"allocated":679936,"data_stored":392347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169166,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":415320,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93329832,"statfs":{"total":95995035648,"available":95569747968,"internally_reserved":0,"allocated":995328,"data_stored":661882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":424278474},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430805,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":794476,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":92950676,"statfs":{"total":95995035648,"available":95181492224,"internally_reserved":0,"allocated":1019904,"data_stored":727514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":812382662},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575469,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1118992,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":1118206,"kb_avail":92626160,"statfs":{"total":95995035648,"available":94849187840,"internally_reserved":0,"allocated":790528,"data_stored":460634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1145043397},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738999,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":888752,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92856400,"statfs":{"total":95995035648,"available":95084953600,"internally_reserved":0,"allocated":1019904,"data_stored":727514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":909048261},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411945,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":623588,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93121564,"statfs":{"total":95995035648,"available":95356481536,"internally_reserved":0,"allocated":884736,"data_stored":593595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150307,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1195676,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":1194558,"kb_avail":92549476,"statfs":{"total":95995035648,"available":94770663424,"internally_reserved":0,"allocated":1007616,"data_stored":720027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1223227847},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":887,"seq":3809635992150,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":695568,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93049584,"statfs":{"total":95995035648,"available":95282774016,"internally_reserved":0,"allocated":790528,"data_stored":460634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692441,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":956760,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":955774,"kb_avail":92788392,"statfs":{"total":95995035648,"available":95015313408,"internally_reserved":0,"allocated":995328,"data_stored":661882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":978713029},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:32:52.208 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-05-15T13:32:52.208 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid deaea04e-12b7-11ef-bc98-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-05-15T13:32:52.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:51 smithi120 ceph-mon[43961]: pgmap v2644: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:52.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:51 smithi177 ceph-mon[44303]: pgmap v2644: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:52.483 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T13:32:53.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:52 smithi027 ceph-mon[39795]: from='client.17436 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:32:53.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:52 smithi120 ceph-mon[43961]: from='client.17436 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:32:53.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:52 smithi177 ceph-mon[44303]: from='client.17436 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T13:32:54.283 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:53 smithi027 ceph-mon[39795]: pgmap v2645: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:54.293 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:53 smithi120 ceph-mon[43961]: pgmap v2645: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:54.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:53 smithi177 ceph-mon[44303]: pgmap v2645: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:55.182 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-15T13:32:55.192 INFO:tasks.cephadm:Teardown begin 2024-05-15T13:32:55.193 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-15T13:32:55.310 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-15T13:32:55.410 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-15T13:32:55.460 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-15T13:32:55.460 DEBUG:teuthology.orchestra.run.smithi027:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-15T13:32:55.494 DEBUG:teuthology.orchestra.run.smithi120:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-15T13:32:55.532 DEBUG:teuthology.orchestra.run.smithi177:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-15T13:32:55.560 INFO:tasks.cephadm:Stopping all daemons... 2024-05-15T13:32:55.560 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-05-15T13:32:55.560 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mon.a 2024-05-15T13:32:55.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:55 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/406094267' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-05-15T13:32:55.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:55 smithi177 ceph-mon[44303]: from='client.? 172.21.15.27:0/406094267' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-05-15T13:32:55.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:55 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:32:55.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:55 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:32:55.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:55 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:32:55.692 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:55 smithi177 ceph-mon[44303]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:32:55.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:55 smithi177 ceph-mon[44303]: pgmap v2646: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:55.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:55 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/406094267' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-05-15T13:32:55.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:55 smithi027 ceph-mon[39795]: from='client.? 172.21.15.27:0/406094267' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-05-15T13:32:55.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:55 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:32:55.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:55 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:32:55.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:55 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:32:55.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:55 smithi027 ceph-mon[39795]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:32:55.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:55 smithi027 ceph-mon[39795]: pgmap v2646: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:55.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:55 smithi027 systemd[1]: Stopping Ceph mon.a for deaea04e-12b7-11ef-bc98-c7b262605968... 2024-05-15T13:32:55.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:55 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/406094267' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-05-15T13:32:55.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:55 smithi120 ceph-mon[43961]: from='client.? 172.21.15.27:0/406094267' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-05-15T13:32:55.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:55 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T13:32:55.793 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:55 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T13:32:55.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:55 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T13:32:55.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:55 smithi120 ceph-mon[43961]: from='mgr.14148 172.21.15.27:0/928711067' entity='mgr.y' 2024-05-15T13:32:55.794 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:55 smithi120 ceph-mon[43961]: pgmap v2646: 31 pgs: 31 active+clean; 577 KiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T13:32:56.242 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:55 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-a[39772]: 2024-05-15T13:32:55.877+0000 7f8db57dd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T13:32:56.242 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:55 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-a[39772]: 2024-05-15T13:32:55.877+0000 7f8db57dd700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-05-15T13:32:56.533 INFO:journalctl@ceph.mon.a.smithi027.stdout:May 15 13:32:56 smithi027 bash[314789]: ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-a 2024-05-15T13:32:56.833 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mon.a.service' 2024-05-15T13:32:56.908 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T13:32:56.909 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-05-15T13:32:56.909 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-05-15T13:32:56.909 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mon.b 2024-05-15T13:32:57.242 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:56 smithi120 systemd[1]: Stopping Ceph mon.b for deaea04e-12b7-11ef-bc98-c7b262605968... 2024-05-15T13:32:57.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:57 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-b[43938]: 2024-05-15T13:32:57.238+0000 7f4592655700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T13:32:57.543 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:57 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-b[43938]: 2024-05-15T13:32:57.238+0000 7f4592655700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-05-15T13:32:58.043 INFO:journalctl@ceph.mon.b.smithi120.stdout:May 15 13:32:57 smithi120 bash[170596]: ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-b 2024-05-15T13:32:58.189 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mon.b.service' 2024-05-15T13:32:58.261 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T13:32:58.262 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-05-15T13:32:58.262 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-05-15T13:32:58.262 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mon.c 2024-05-15T13:32:58.615 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:58 smithi177 systemd[1]: Stopping Ceph mon.c for deaea04e-12b7-11ef-bc98-c7b262605968... 2024-05-15T13:32:58.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:58 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-c[44280]: 2024-05-15T13:32:58.610+0000 7f2244c8f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T13:32:58.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:58 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-c[44280]: 2024-05-15T13:32:58.610+0000 7f2244c8f700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-05-15T13:32:59.442 INFO:journalctl@ceph.mon.c.smithi177.stdout:May 15 13:32:59 smithi177 bash[174610]: ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mon-c 2024-05-15T13:32:59.573 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mon.c.service' 2024-05-15T13:32:59.632 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T13:32:59.632 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-05-15T13:32:59.632 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-05-15T13:32:59.633 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mgr.y 2024-05-15T13:32:59.936 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 13:32:59 smithi027 systemd[1]: Stopping Ceph mgr.y for deaea04e-12b7-11ef-bc98-c7b262605968... 2024-05-15T13:33:00.254 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 13:32:59 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y[40041]: 2024-05-15T13:32:59.932+0000 7f6ab1de7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T13:33:00.533 INFO:journalctl@ceph.mgr.y.smithi027.stdout:May 15 13:33:00 smithi027 bash[314933]: ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-y 2024-05-15T13:33:00.812 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mgr.y.service' 2024-05-15T13:33:00.850 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T13:33:00.850 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-05-15T13:33:00.850 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-05-15T13:33:00.851 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mgr.x 2024-05-15T13:33:01.293 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 13:33:00 smithi120 systemd[1]: Stopping Ceph mgr.x for deaea04e-12b7-11ef-bc98-c7b262605968... 2024-05-15T13:33:01.793 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 13:33:01 smithi120 bash[170752]: ceph-deaea04e-12b7-11ef-bc98-c7b262605968-mgr-x 2024-05-15T13:33:02.042 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 13:33:01 smithi120 systemd[1]: ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-05-15T13:33:02.042 INFO:journalctl@ceph.mgr.x.smithi120.stdout:May 15 13:33:02 smithi120 systemd[1]: ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-05-15T13:33:02.045 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@mgr.x.service' 2024-05-15T13:33:02.079 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T13:33:02.079 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-05-15T13:33:02.079 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-15T13:33:02.080 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.0 2024-05-15T13:33:02.533 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 15 13:33:02 smithi027 systemd[1]: Stopping Ceph osd.0 for deaea04e-12b7-11ef-bc98-c7b262605968... 2024-05-15T13:33:02.533 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 15 13:33:02 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-0[58372]: 2024-05-15T13:33:02.352+0000 7f8982bf9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T13:33:02.534 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 15 13:33:02 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-0[58372]: 2024-05-15T13:33:02.352+0000 7f8982bf9700 -1 osd.0 1049 *** Got signal Terminated *** 2024-05-15T13:33:02.534 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 15 13:33:02 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-0[58372]: 2024-05-15T13:33:02.352+0000 7f8982bf9700 -1 osd.0 1049 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T13:33:08.534 INFO:journalctl@ceph.osd.0.smithi027.stdout:May 15 13:33:08 smithi027 bash[315080]: ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-0 2024-05-15T13:33:10.325 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.0.service' 2024-05-15T13:33:10.367 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T13:33:10.367 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-15T13:33:10.367 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-15T13:33:10.367 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.1 2024-05-15T13:33:10.684 INFO:journalctl@ceph.osd.1.smithi027.stdout:May 15 13:33:10 smithi027 systemd[1]: Stopping Ceph osd.1 for deaea04e-12b7-11ef-bc98-c7b262605968... 2024-05-15T13:33:11.033 INFO:journalctl@ceph.osd.1.smithi027.stdout:May 15 13:33:10 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-1[66219]: 2024-05-15T13:33:10.680+0000 7f419d3a5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T13:33:11.033 INFO:journalctl@ceph.osd.1.smithi027.stdout:May 15 13:33:10 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-1[66219]: 2024-05-15T13:33:10.680+0000 7f419d3a5700 -1 osd.1 1049 *** Got signal Terminated *** 2024-05-15T13:33:11.033 INFO:journalctl@ceph.osd.1.smithi027.stdout:May 15 13:33:10 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-1[66219]: 2024-05-15T13:33:10.680+0000 7f419d3a5700 -1 osd.1 1049 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T13:33:17.033 INFO:journalctl@ceph.osd.1.smithi027.stdout:May 15 13:33:16 smithi027 bash[315356]: ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-1 2024-05-15T13:33:18.605 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.1.service' 2024-05-15T13:33:18.657 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T13:33:18.657 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-15T13:33:18.657 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-15T13:33:18.657 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.2 2024-05-15T13:33:18.980 INFO:journalctl@ceph.osd.2.smithi027.stdout:May 15 13:33:18 smithi027 systemd[1]: Stopping Ceph osd.2 for deaea04e-12b7-11ef-bc98-c7b262605968... 2024-05-15T13:33:19.285 INFO:journalctl@ceph.osd.2.smithi027.stdout:May 15 13:33:18 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-2[74111]: 2024-05-15T13:33:18.975+0000 7f97c6dc9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T13:33:19.285 INFO:journalctl@ceph.osd.2.smithi027.stdout:May 15 13:33:18 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-2[74111]: 2024-05-15T13:33:18.975+0000 7f97c6dc9700 -1 osd.2 1049 *** Got signal Terminated *** 2024-05-15T13:33:19.285 INFO:journalctl@ceph.osd.2.smithi027.stdout:May 15 13:33:18 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-2[74111]: 2024-05-15T13:33:18.975+0000 7f97c6dc9700 -1 osd.2 1049 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T13:33:25.283 INFO:journalctl@ceph.osd.2.smithi027.stdout:May 15 13:33:25 smithi027 bash[315620]: ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-2 2024-05-15T13:33:26.766 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.2.service' 2024-05-15T13:33:26.810 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T13:33:26.811 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-15T13:33:26.811 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-15T13:33:26.811 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.3 2024-05-15T13:33:27.033 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 15 13:33:26 smithi027 systemd[1]: Stopping Ceph osd.3 for deaea04e-12b7-11ef-bc98-c7b262605968... 2024-05-15T13:33:27.533 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 15 13:33:27 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-3[81304]: 2024-05-15T13:33:27.127+0000 7fd2b4032700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T13:33:27.533 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 15 13:33:27 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-3[81304]: 2024-05-15T13:33:27.127+0000 7fd2b4032700 -1 osd.3 1049 *** Got signal Terminated *** 2024-05-15T13:33:27.534 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 15 13:33:27 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-3[81304]: 2024-05-15T13:33:27.127+0000 7fd2b4032700 -1 osd.3 1049 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T13:33:28.942 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:28 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:28.673+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:29.294 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:28 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:28.869+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:29.872 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:29 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:29.575+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:29.942 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:29 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:29.626+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:30.283 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 15 13:33:29 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-3[81304]: 2024-05-15T13:33:29.803+0000 7fd2ac61e700 -1 osd.3 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:04.536889+0000 front 2024-05-15T13:33:04.536876+0000 (oldest deadline 2024-05-15T13:33:29.237041+0000) 2024-05-15T13:33:30.293 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:29 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:29.869+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:30.877 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:30 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:30.546+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:30.942 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:30 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:30.652+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:31.283 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 15 13:33:30 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-3[81304]: 2024-05-15T13:33:30.783+0000 7fd2ac61e700 -1 osd.3 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:04.536889+0000 front 2024-05-15T13:33:04.536876+0000 (oldest deadline 2024-05-15T13:33:29.237041+0000) 2024-05-15T13:33:31.293 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:30 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:30.874+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:31.834 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:31 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:31.559+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:31.942 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:31 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:31.621+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:32.033 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 15 13:33:31 smithi027 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-3[81304]: 2024-05-15T13:33:31.778+0000 7fd2ac61e700 -1 osd.3 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:04.536889+0000 front 2024-05-15T13:33:04.536876+0000 (oldest deadline 2024-05-15T13:33:29.237041+0000) 2024-05-15T13:33:32.293 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:31 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:31.831+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:32.667 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:32 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:32.243+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:32.793 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:32 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:32.589+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:32.794 INFO:journalctl@ceph.osd.4.smithi120.stdout:May 15 13:33:32 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-4[49322]: 2024-05-15T13:33:32.350+0000 7f9fa19a0700 -1 osd.4 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:06.241071+0000 front 2024-05-15T13:33:06.241092+0000 (oldest deadline 2024-05-15T13:33:32.141043+0000) 2024-05-15T13:33:32.942 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:32 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:32.663+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:32.942 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:32 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:32.869+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:33.293 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:32 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:32.834+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:33.371 INFO:journalctl@ceph.osd.3.smithi027.stdout:May 15 13:33:33 smithi027 bash[315884]: ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-3 2024-05-15T13:33:33.442 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:33 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:33.249+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:33.443 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:33 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:33.065+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:33.793 INFO:journalctl@ceph.osd.4.smithi120.stdout:May 15 13:33:33 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-4[49322]: 2024-05-15T13:33:33.358+0000 7f9fa19a0700 -1 osd.4 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:06.241071+0000 front 2024-05-15T13:33:06.241092+0000 (oldest deadline 2024-05-15T13:33:32.141043+0000) 2024-05-15T13:33:33.794 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:33 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:33.542+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:33.942 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:33 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:33.636+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:33.943 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:33 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:33.834+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:34.293 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:33 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:33.840+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:34.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:34 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:34.114+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:34.442 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:34 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:34.262+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:34.794 INFO:journalctl@ceph.osd.4.smithi120.stdout:May 15 13:33:34 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-4[49322]: 2024-05-15T13:33:34.331+0000 7f9fa19a0700 -1 osd.4 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:06.241071+0000 front 2024-05-15T13:33:06.241092+0000 (oldest deadline 2024-05-15T13:33:32.141043+0000) 2024-05-15T13:33:34.794 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:34 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:34.537+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:34.942 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:34 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:34.683+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:34.942 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:34 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:34.829+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:35.121 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.3.service' 2024-05-15T13:33:35.164 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T13:33:35.164 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-15T13:33:35.164 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-15T13:33:35.164 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.4 2024-05-15T13:33:35.196 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:34 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:34.885+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:35.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:35 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:35.112+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:35.442 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:35 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:35.227+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:35.492 INFO:journalctl@ceph.osd.4.smithi120.stdout:May 15 13:33:35 smithi120 systemd[1]: Stopping Ceph osd.4 for deaea04e-12b7-11ef-bc98-c7b262605968... 2024-05-15T13:33:35.492 INFO:journalctl@ceph.osd.4.smithi120.stdout:May 15 13:33:35 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-4[49322]: 2024-05-15T13:33:35.368+0000 7f9fa19a0700 -1 osd.4 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:06.241071+0000 front 2024-05-15T13:33:06.241092+0000 (oldest deadline 2024-05-15T13:33:32.141043+0000) 2024-05-15T13:33:35.492 INFO:journalctl@ceph.osd.4.smithi120.stdout:May 15 13:33:35 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-4[49322]: 2024-05-15T13:33:35.434+0000 7f9fa93b4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T13:33:35.492 INFO:journalctl@ceph.osd.4.smithi120.stdout:May 15 13:33:35 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-4[49322]: 2024-05-15T13:33:35.434+0000 7f9fa93b4700 -1 osd.4 1049 *** Got signal Terminated *** 2024-05-15T13:33:35.493 INFO:journalctl@ceph.osd.4.smithi120.stdout:May 15 13:33:35 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-4[49322]: 2024-05-15T13:33:35.434+0000 7f9fa93b4700 -1 osd.4 1049 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T13:33:35.793 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:35 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:35.489+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:35.942 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:35 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:35.653+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:35.942 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:35 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:35.868+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:36.293 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:35 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:35.841+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:36.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:36 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:36.148+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:36.442 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:36 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:36.210+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:36.793 INFO:journalctl@ceph.osd.4.smithi120.stdout:May 15 13:33:36 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-4[49322]: 2024-05-15T13:33:36.376+0000 7f9fa19a0700 -1 osd.4 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:06.241071+0000 front 2024-05-15T13:33:06.241092+0000 (oldest deadline 2024-05-15T13:33:32.141043+0000) 2024-05-15T13:33:36.794 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:36 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:36.503+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:36.942 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:36 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:36.611+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:36.942 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:36 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:36.860+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:37.293 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:36 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:36.824+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:37.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:37 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:37.138+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:37.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:37 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:37.138+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:33:37.442 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:37 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:37.203+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:37.793 INFO:journalctl@ceph.osd.4.smithi120.stdout:May 15 13:33:37 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-4[49322]: 2024-05-15T13:33:37.341+0000 7f9fa19a0700 -1 osd.4 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:06.241071+0000 front 2024-05-15T13:33:06.241092+0000 (oldest deadline 2024-05-15T13:33:32.141043+0000) 2024-05-15T13:33:37.794 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:37 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:37.481+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:37.942 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:37 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:37.600+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:37.942 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:37 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:37.831+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:38.293 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:37 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:37.807+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:38.294 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:37 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:37.807+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:33:38.443 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:38 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:38.187+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:38.443 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:38 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:38.187+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:33:38.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:38 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:38.186+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:38.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:38 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:38.186+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:33:38.787 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:38 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:38.473+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:38.787 INFO:journalctl@ceph.osd.4.smithi120.stdout:May 15 13:33:38 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-4[49322]: 2024-05-15T13:33:38.296+0000 7f9fa19a0700 -1 osd.4 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:06.241071+0000 front 2024-05-15T13:33:06.241092+0000 (oldest deadline 2024-05-15T13:33:32.141043+0000) 2024-05-15T13:33:38.787 INFO:journalctl@ceph.osd.4.smithi120.stdout:May 15 13:33:38 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-4[49322]: 2024-05-15T13:33:38.296+0000 7f9fa19a0700 -1 osd.4 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:15.443188+0000 front 2024-05-15T13:33:15.443369+0000 (oldest deadline 2024-05-15T13:33:37.743258+0000) 2024-05-15T13:33:38.824 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:38 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:38.560+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:39.043 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:38 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:38.784+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:39.053 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:38 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:38.784+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:33:39.150 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:38 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:38.821+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:39.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:39 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:39.173+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:39.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:39 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:39.173+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:33:39.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:39 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:39.147+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:39.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:39 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:39.147+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:33:39.754 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:39 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:39.514+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:39.754 INFO:journalctl@ceph.osd.4.smithi120.stdout:May 15 13:33:39 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-4[49322]: 2024-05-15T13:33:39.304+0000 7f9fa19a0700 -1 osd.4 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:06.241071+0000 front 2024-05-15T13:33:06.241092+0000 (oldest deadline 2024-05-15T13:33:32.141043+0000) 2024-05-15T13:33:39.754 INFO:journalctl@ceph.osd.4.smithi120.stdout:May 15 13:33:39 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-4[49322]: 2024-05-15T13:33:39.304+0000 7f9fa19a0700 -1 osd.4 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:15.443188+0000 front 2024-05-15T13:33:15.443369+0000 (oldest deadline 2024-05-15T13:33:37.743258+0000) 2024-05-15T13:33:39.854 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:39 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:39.567+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:40.043 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:39 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:39.751+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:40.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:39 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:39.751+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:33:40.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:39 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:39.851+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:40.533 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:40 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:40.201+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:40.533 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:40 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:40.201+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:33:40.533 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:40 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:40.196+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:40.534 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:40 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:40.196+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:33:40.732 INFO:journalctl@ceph.osd.4.smithi120.stdout:May 15 13:33:40 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-4[49322]: 2024-05-15T13:33:40.324+0000 7f9fa19a0700 -1 osd.4 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:06.241071+0000 front 2024-05-15T13:33:06.241092+0000 (oldest deadline 2024-05-15T13:33:32.141043+0000) 2024-05-15T13:33:40.732 INFO:journalctl@ceph.osd.4.smithi120.stdout:May 15 13:33:40 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-4[49322]: 2024-05-15T13:33:40.324+0000 7f9fa19a0700 -1 osd.4 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:15.443188+0000 front 2024-05-15T13:33:15.443369+0000 (oldest deadline 2024-05-15T13:33:37.743258+0000) 2024-05-15T13:33:40.733 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:40 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:40.541+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:40.816 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:40 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:40.529+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:41.043 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:40 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:40.729+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:41.043 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:40 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:40.729+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:33:41.184 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:40 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:40.812+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:41.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:41 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:41.206+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:41.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:41 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:41.206+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:33:41.442 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:41 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:41.181+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:41.442 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:41 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:41.181+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:33:41.793 INFO:journalctl@ceph.osd.4.smithi120.stdout:May 15 13:33:41 smithi120 bash[170903]: ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-4 2024-05-15T13:33:41.794 INFO:journalctl@ceph.osd.5.smithi120.stdout:May 15 13:33:41 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-5[56546]: 2024-05-15T13:33:41.745+0000 7f0f46d17700 -1 osd.5 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:14.602713+0000 front 2024-05-15T13:33:14.602751+0000 (oldest deadline 2024-05-15T13:33:41.603663+0000) 2024-05-15T13:33:41.794 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:41 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:41.516+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:41.794 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:41 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:41.694+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:41.794 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:41 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:41.694+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:33:41.812 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:41 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:41.552+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:42.181 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:41 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:41.809+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:42.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:42 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:42.201+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:42.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:42 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:42.201+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:33:42.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:42 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:42.178+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:42.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:42 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:42.178+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:33:42.942 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:42 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:42.560+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:42.942 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:42 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:42.788+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:43.044 INFO:journalctl@ceph.osd.5.smithi120.stdout:May 15 13:33:42 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-5[56546]: 2024-05-15T13:33:42.763+0000 7f0f46d17700 -1 osd.5 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:14.602713+0000 front 2024-05-15T13:33:14.602751+0000 (oldest deadline 2024-05-15T13:33:41.603663+0000) 2024-05-15T13:33:43.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:42 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:42.691+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:43.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:42 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:42.691+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:33:43.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:42 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:42.542+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:43.045 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:42 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:42.542+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.834583+0000 front 2024-05-15T13:33:12.834595+0000 (oldest deadline 2024-05-15T13:33:41.634974+0000) 2024-05-15T13:33:43.544 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:43 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:43.223+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:43.544 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:43 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:43.223+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:33:43.544 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:43 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:43.207+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:43.544 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:43 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:43.207+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:33:43.882 INFO:journalctl@ceph.osd.5.smithi120.stdout:May 15 13:33:43 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-5[56546]: 2024-05-15T13:33:43.808+0000 7f0f46d17700 -1 osd.5 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:14.602713+0000 front 2024-05-15T13:33:14.602751+0000 (oldest deadline 2024-05-15T13:33:41.603663+0000) 2024-05-15T13:33:43.882 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:43 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:43.564+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:43.882 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:43 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:43.564+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.834583+0000 front 2024-05-15T13:33:12.834595+0000 (oldest deadline 2024-05-15T13:33:41.634974+0000) 2024-05-15T13:33:43.882 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:43 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:43.731+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:43.883 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:43 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:43.731+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:33:43.888 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.4.service' 2024-05-15T13:33:43.942 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:43 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:43.540+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:43.942 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:43 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:43.749+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:43.968 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T13:33:43.968 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-15T13:33:43.968 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-15T13:33:43.969 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.5 2024-05-15T13:33:44.255 INFO:journalctl@ceph.osd.5.smithi120.stdout:May 15 13:33:44 smithi120 systemd[1]: Stopping Ceph osd.5 for deaea04e-12b7-11ef-bc98-c7b262605968... 2024-05-15T13:33:44.255 INFO:journalctl@ceph.osd.5.smithi120.stdout:May 15 13:33:44 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-5[56546]: 2024-05-15T13:33:44.252+0000 7f0f4a723700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T13:33:44.504 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:44 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:44.267+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:44.504 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:44 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:44.267+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:33:44.504 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:44 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:44.247+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:44.504 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:44 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:44.247+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:33:44.543 INFO:journalctl@ceph.osd.5.smithi120.stdout:May 15 13:33:44 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-5[56546]: 2024-05-15T13:33:44.252+0000 7f0f4a723700 -1 osd.5 1049 *** Got signal Terminated *** 2024-05-15T13:33:44.543 INFO:journalctl@ceph.osd.5.smithi120.stdout:May 15 13:33:44 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-5[56546]: 2024-05-15T13:33:44.252+0000 7f0f4a723700 -1 osd.5 1049 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T13:33:44.942 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:44 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:44.501+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:44.942 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:44 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:44.721+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:45.043 INFO:journalctl@ceph.osd.5.smithi120.stdout:May 15 13:33:44 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-5[56546]: 2024-05-15T13:33:44.838+0000 7f0f46d17700 -1 osd.5 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:14.602713+0000 front 2024-05-15T13:33:14.602751+0000 (oldest deadline 2024-05-15T13:33:41.603663+0000) 2024-05-15T13:33:45.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:44 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:44.596+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:45.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:44 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:44.596+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.834583+0000 front 2024-05-15T13:33:12.834595+0000 (oldest deadline 2024-05-15T13:33:41.634974+0000) 2024-05-15T13:33:45.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:44 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:44.753+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:45.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:44 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:44.753+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:33:45.692 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:45 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:45.211+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:45.692 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:45 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:45.211+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:33:45.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:45 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:45.455+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:45.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:45 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:45.455+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:33:45.693 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:45 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:45.251+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:45.693 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:45 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:45.251+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:33:46.043 INFO:journalctl@ceph.osd.5.smithi120.stdout:May 15 13:33:45 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-5[56546]: 2024-05-15T13:33:45.800+0000 7f0f46d17700 -1 osd.5 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:14.602713+0000 front 2024-05-15T13:33:14.602751+0000 (oldest deadline 2024-05-15T13:33:41.603663+0000) 2024-05-15T13:33:46.044 INFO:journalctl@ceph.osd.5.smithi120.stdout:May 15 13:33:45 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-5[56546]: 2024-05-15T13:33:45.800+0000 7f0f46d17700 -1 osd.5 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.704936+0000 front 2024-05-15T13:33:22.705029+0000 (oldest deadline 2024-05-15T13:33:45.005035+0000) 2024-05-15T13:33:46.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:45 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:45.580+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:46.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:45 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:45.580+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.834583+0000 front 2024-05-15T13:33:12.834595+0000 (oldest deadline 2024-05-15T13:33:41.634974+0000) 2024-05-15T13:33:46.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:45 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:45.733+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:46.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:45 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:45.733+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:33:46.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:45 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:45.766+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:46.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:45 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:45.766+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:33:46.692 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:46 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:46.476+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:46.692 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:46 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:46.476+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:33:46.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:46 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:46.250+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:46.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:46 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:46.250+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:33:46.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:46 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:46.250+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:46.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:46 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:46.250+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:33:47.043 INFO:journalctl@ceph.osd.5.smithi120.stdout:May 15 13:33:46 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-5[56546]: 2024-05-15T13:33:46.772+0000 7f0f46d17700 -1 osd.5 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:14.602713+0000 front 2024-05-15T13:33:14.602751+0000 (oldest deadline 2024-05-15T13:33:41.603663+0000) 2024-05-15T13:33:47.044 INFO:journalctl@ceph.osd.5.smithi120.stdout:May 15 13:33:46 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-5[56546]: 2024-05-15T13:33:46.772+0000 7f0f46d17700 -1 osd.5 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.704936+0000 front 2024-05-15T13:33:22.705029+0000 (oldest deadline 2024-05-15T13:33:45.005035+0000) 2024-05-15T13:33:47.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:46 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:46.541+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:47.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:46 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:46.541+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.834583+0000 front 2024-05-15T13:33:12.834595+0000 (oldest deadline 2024-05-15T13:33:41.634974+0000) 2024-05-15T13:33:47.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:46 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:46.734+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:47.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:46 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:46.734+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:33:47.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:46 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:46.738+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:47.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:46 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:46.738+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:33:47.492 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:47 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:47.263+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:47.492 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:47 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:47.263+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:33:47.492 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:47 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:47.220+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:47.492 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:47 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:47.220+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:33:47.789 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:47 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:47.489+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:47.789 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:47 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:47.489+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:33:47.793 INFO:journalctl@ceph.osd.5.smithi120.stdout:May 15 13:33:47 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-5[56546]: 2024-05-15T13:33:47.722+0000 7f0f46d17700 -1 osd.5 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:14.602713+0000 front 2024-05-15T13:33:14.602751+0000 (oldest deadline 2024-05-15T13:33:41.603663+0000) 2024-05-15T13:33:47.794 INFO:journalctl@ceph.osd.5.smithi120.stdout:May 15 13:33:47 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-5[56546]: 2024-05-15T13:33:47.722+0000 7f0f46d17700 -1 osd.5 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.704936+0000 front 2024-05-15T13:33:22.705029+0000 (oldest deadline 2024-05-15T13:33:45.005035+0000) 2024-05-15T13:33:47.794 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:47 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:47.507+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:47.794 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:47 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:47.507+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.834583+0000 front 2024-05-15T13:33:12.834595+0000 (oldest deadline 2024-05-15T13:33:41.634974+0000) 2024-05-15T13:33:47.794 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:47 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:47.726+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:47.794 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:47 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:47.726+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:33:47.794 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:47 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:47.726+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.879654+0000 front 2024-05-15T13:33:23.879744+0000 (oldest deadline 2024-05-15T13:33:46.779721+0000) 2024-05-15T13:33:48.190 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:47 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:47.786+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:48.191 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:47 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:47.786+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:33:48.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:48 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:48.285+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:48.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:48 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:48.285+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:33:48.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:48 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:48.188+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:48.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:48 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:48.188+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:33:48.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:48 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:48.188+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:33:48.794 INFO:journalctl@ceph.osd.5.smithi120.stdout:May 15 13:33:48 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-5[56546]: 2024-05-15T13:33:48.735+0000 7f0f46d17700 -1 osd.5 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:14.602713+0000 front 2024-05-15T13:33:14.602751+0000 (oldest deadline 2024-05-15T13:33:41.603663+0000) 2024-05-15T13:33:48.794 INFO:journalctl@ceph.osd.5.smithi120.stdout:May 15 13:33:48 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-5[56546]: 2024-05-15T13:33:48.735+0000 7f0f46d17700 -1 osd.5 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.704936+0000 front 2024-05-15T13:33:22.705029+0000 (oldest deadline 2024-05-15T13:33:45.005035+0000) 2024-05-15T13:33:48.795 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:48 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:48.526+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:48.795 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:48 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:48.526+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.834583+0000 front 2024-05-15T13:33:12.834595+0000 (oldest deadline 2024-05-15T13:33:41.634974+0000) 2024-05-15T13:33:48.795 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:48 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:48.746+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:48.796 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:48 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:48.746+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:33:48.796 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:48 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:48.746+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.879654+0000 front 2024-05-15T13:33:23.879744+0000 (oldest deadline 2024-05-15T13:33:46.779721+0000) 2024-05-15T13:33:48.802 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:48 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:48.468+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:48.802 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:48 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:48.468+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:33:49.171 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:48 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:48.799+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:49.171 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:48 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:48.799+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:33:49.443 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:49 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:49.266+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:49.443 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:49 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:49.266+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:33:49.444 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:49 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:49.168+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:49.444 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:49 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:49.168+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:33:49.444 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:49 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:49.168+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:33:49.800 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:49 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:49.453+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:49.800 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:49 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:49.453+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:33:50.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:49 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:49.574+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:50.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:49 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:49.574+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.834583+0000 front 2024-05-15T13:33:12.834595+0000 (oldest deadline 2024-05-15T13:33:41.634974+0000) 2024-05-15T13:33:50.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:49 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:49.574+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.236674+0000 front 2024-05-15T13:33:23.236785+0000 (oldest deadline 2024-05-15T13:33:49.136776+0000) 2024-05-15T13:33:50.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:49 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:49.730+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:50.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:49 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:49.730+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:33:50.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:49 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:49.730+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.879654+0000 front 2024-05-15T13:33:23.879744+0000 (oldest deadline 2024-05-15T13:33:46.779721+0000) 2024-05-15T13:33:50.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:49 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:49.797+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:50.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:49 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:49.797+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:33:50.488 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:50 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:50.262+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:50.488 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:50 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:50.262+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:33:50.488 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:50 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:50.212+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:50.488 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:50 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:50.212+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:33:50.488 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:50 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:50.212+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:33:50.558 INFO:journalctl@ceph.osd.5.smithi120.stdout:May 15 13:33:50 smithi120 bash[171169]: ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-5 2024-05-15T13:33:50.781 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:50 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:50.485+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:50.781 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:50 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:50.485+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:33:51.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:50 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:50.554+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:51.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:50 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:50.554+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.834583+0000 front 2024-05-15T13:33:12.834595+0000 (oldest deadline 2024-05-15T13:33:41.634974+0000) 2024-05-15T13:33:51.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:50 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:50.554+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.236674+0000 front 2024-05-15T13:33:23.236785+0000 (oldest deadline 2024-05-15T13:33:49.136776+0000) 2024-05-15T13:33:51.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:50 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:50.760+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:51.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:50 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:50.760+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:33:51.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:50 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:50.760+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.879654+0000 front 2024-05-15T13:33:23.879744+0000 (oldest deadline 2024-05-15T13:33:46.779721+0000) 2024-05-15T13:33:51.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:50 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:50.778+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:51.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:50 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:50.778+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:33:51.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:51 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:51.271+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:51.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:51 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:51.271+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:33:51.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:51 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:51.468+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:51.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:51 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:51.468+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:33:51.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:51 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:51.237+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:51.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:51 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:51.237+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:33:51.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:51 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:51.237+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:33:51.794 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:51 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:51.561+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:51.794 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:51 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:51.561+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.834583+0000 front 2024-05-15T13:33:12.834595+0000 (oldest deadline 2024-05-15T13:33:41.634974+0000) 2024-05-15T13:33:51.794 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:51 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:51.561+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.236674+0000 front 2024-05-15T13:33:23.236785+0000 (oldest deadline 2024-05-15T13:33:49.136776+0000) 2024-05-15T13:33:52.166 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:51 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:51.809+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:52.166 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:51 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:51.809+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:33:52.166 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:51 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:51.809+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.879654+0000 front 2024-05-15T13:33:23.879744+0000 (oldest deadline 2024-05-15T13:33:46.779721+0000) 2024-05-15T13:33:52.175 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.5.service' 2024-05-15T13:33:52.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:51 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:51.824+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:52.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:51 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:51.824+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:33:52.217 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T13:33:52.217 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-15T13:33:52.217 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-05-15T13:33:52.217 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.6 2024-05-15T13:33:52.543 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:52 smithi120 systemd[1]: Stopping Ceph osd.6 for deaea04e-12b7-11ef-bc98-c7b262605968... 2024-05-15T13:33:52.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:52 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:52.279+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:52.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:52 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:52.279+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:33:52.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:52 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:52.495+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:52.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:52 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:52.495+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:33:52.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:52 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:52.282+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:52.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:52 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:52.282+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:33:52.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:52 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:52.282+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:33:52.849 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:52 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:52.545+0000 7fa32c44a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T13:33:52.849 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:52 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:52.545+0000 7fa32c44a700 -1 osd.6 1049 *** Got signal Terminated *** 2024-05-15T13:33:52.849 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:52 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:52.545+0000 7fa32c44a700 -1 osd.6 1049 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T13:33:52.849 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:52 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:52.605+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:52.849 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:52 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:52.605+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.834583+0000 front 2024-05-15T13:33:12.834595+0000 (oldest deadline 2024-05-15T13:33:41.634974+0000) 2024-05-15T13:33:52.849 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:52 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:52.605+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.236674+0000 front 2024-05-15T13:33:23.236785+0000 (oldest deadline 2024-05-15T13:33:49.136776+0000) 2024-05-15T13:33:53.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:52 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:52.869+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:53.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:52 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:52.869+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:33:53.294 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:52 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:52.846+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:53.294 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:52 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:52.846+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:33:53.294 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:52 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:52.846+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.879654+0000 front 2024-05-15T13:33:23.879744+0000 (oldest deadline 2024-05-15T13:33:46.779721+0000) 2024-05-15T13:33:53.693 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:53 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:53.256+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:53.693 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:53 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:53.256+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:33:53.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:53 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:53.275+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:53.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:53 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:53.275+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:33:53.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:53 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:53.275+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:33:53.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:53 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:53.447+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:53.694 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:53 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:53.447+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:33:54.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:53 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:53.649+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:54.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:53 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:53.649+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.834583+0000 front 2024-05-15T13:33:12.834595+0000 (oldest deadline 2024-05-15T13:33:41.634974+0000) 2024-05-15T13:33:54.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:53 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:53.649+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.236674+0000 front 2024-05-15T13:33:23.236785+0000 (oldest deadline 2024-05-15T13:33:49.136776+0000) 2024-05-15T13:33:54.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:53 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:53.810+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:54.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:53 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:53.810+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:33:54.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:53 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:53.810+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.879654+0000 front 2024-05-15T13:33:23.879744+0000 (oldest deadline 2024-05-15T13:33:46.779721+0000) 2024-05-15T13:33:54.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:53 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:53.810+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:30.281285+0000 front 2024-05-15T13:33:30.281128+0000 (oldest deadline 2024-05-15T13:33:53.181040+0000) 2024-05-15T13:33:54.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:53 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:53.902+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:54.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:53 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:53.902+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:33:54.693 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:54 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:54.274+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:54.693 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:54 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:54.274+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:33:54.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:54 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:54.250+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:54.694 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:54 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:54.250+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:33:54.694 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:54 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:54.250+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:33:54.694 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:54 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:54.452+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:54.694 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:54 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:54.452+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:33:55.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:54 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:54.630+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:55.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:54 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:54.630+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.834583+0000 front 2024-05-15T13:33:12.834595+0000 (oldest deadline 2024-05-15T13:33:41.634974+0000) 2024-05-15T13:33:55.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:54 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:54.630+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.236674+0000 front 2024-05-15T13:33:23.236785+0000 (oldest deadline 2024-05-15T13:33:49.136776+0000) 2024-05-15T13:33:55.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:54 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:54.771+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:55.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:54 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:54.771+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:33:55.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:54 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:54.771+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.879654+0000 front 2024-05-15T13:33:23.879744+0000 (oldest deadline 2024-05-15T13:33:46.779721+0000) 2024-05-15T13:33:55.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:54 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:54.771+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:30.281285+0000 front 2024-05-15T13:33:30.281128+0000 (oldest deadline 2024-05-15T13:33:53.181040+0000) 2024-05-15T13:33:55.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:54 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:54.859+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:55.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:54 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:54.859+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:33:55.496 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:55 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:55.206+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:55.496 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:55 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:55.206+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:33:55.496 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:55 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:55.206+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:33:55.496 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:55 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:55.303+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:55.496 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:55 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:55.303+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:33:55.826 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:55 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:55.493+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:55.826 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:55 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:55.493+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:33:56.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:55 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:55.649+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:56.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:55 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:55.649+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.834583+0000 front 2024-05-15T13:33:12.834595+0000 (oldest deadline 2024-05-15T13:33:41.634974+0000) 2024-05-15T13:33:56.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:55 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:55.649+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.236674+0000 front 2024-05-15T13:33:23.236785+0000 (oldest deadline 2024-05-15T13:33:49.136776+0000) 2024-05-15T13:33:56.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:55 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:55.649+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:29.137273+0000 front 2024-05-15T13:33:29.137330+0000 (oldest deadline 2024-05-15T13:33:55.037187+0000) 2024-05-15T13:33:56.045 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:55 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:55.810+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:56.045 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:55 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:55.810+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:33:56.045 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:55 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:55.810+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.879654+0000 front 2024-05-15T13:33:23.879744+0000 (oldest deadline 2024-05-15T13:33:46.779721+0000) 2024-05-15T13:33:56.045 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:55 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:55.810+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:30.281285+0000 front 2024-05-15T13:33:30.281128+0000 (oldest deadline 2024-05-15T13:33:53.181040+0000) 2024-05-15T13:33:56.170 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:55 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:55.823+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:56.171 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:55 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:55.823+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:33:56.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:56 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:56.339+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:56.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:56 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:56.339+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:33:56.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:56 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:56.168+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:56.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:56 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:56.168+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:33:56.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:56 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:56.168+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:33:56.832 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:56 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:56.538+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:56.832 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:56 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:56.538+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:33:56.833 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:56 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:56.538+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:33:57.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:56 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:56.667+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:05.233323+0000 front 2024-05-15T13:33:05.233133+0000 (oldest deadline 2024-05-15T13:33:29.333199+0000) 2024-05-15T13:33:57.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:56 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:56.667+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.834583+0000 front 2024-05-15T13:33:12.834595+0000 (oldest deadline 2024-05-15T13:33:41.634974+0000) 2024-05-15T13:33:57.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:56 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:56.667+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.236674+0000 front 2024-05-15T13:33:23.236785+0000 (oldest deadline 2024-05-15T13:33:49.136776+0000) 2024-05-15T13:33:57.044 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:56 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6[64436]: 2024-05-15T13:33:56.667+0000 7fa328a3e700 -1 osd.6 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:29.137273+0000 front 2024-05-15T13:33:29.137330+0000 (oldest deadline 2024-05-15T13:33:55.037187+0000) 2024-05-15T13:33:57.045 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:56 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:56.803+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:57.045 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:56 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:56.803+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:33:57.045 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:56 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:56.803+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.879654+0000 front 2024-05-15T13:33:23.879744+0000 (oldest deadline 2024-05-15T13:33:46.779721+0000) 2024-05-15T13:33:57.045 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:56 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:56.803+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:30.281285+0000 front 2024-05-15T13:33:30.281128+0000 (oldest deadline 2024-05-15T13:33:53.181040+0000) 2024-05-15T13:33:57.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:56 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:56.830+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:57.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:56 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:56.830+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:33:57.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:56 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:56.830+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:33:57.530 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:57 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:57.322+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:57.531 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:57 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:57.322+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:33:57.531 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:57 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:57.322+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:29.362499+0000 front 2024-05-15T13:33:29.362441+0000 (oldest deadline 2024-05-15T13:33:56.362927+0000) 2024-05-15T13:33:57.531 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:57 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:57.193+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:57.531 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:57 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:57.193+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:33:57.531 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:57 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:57.193+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:33:57.531 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:57 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:57.193+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:33:57.807 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:57 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:57.528+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:57.807 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:57 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:57.528+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:33:57.807 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:57 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:57.528+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:33:58.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:57 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:57.805+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:58.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:57 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:57.805+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:33:58.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:57 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:57.805+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:33:58.294 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:57 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:57.801+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:58.294 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:57 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:57.801+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:33:58.294 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:57 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:57.801+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.879654+0000 front 2024-05-15T13:33:23.879744+0000 (oldest deadline 2024-05-15T13:33:46.779721+0000) 2024-05-15T13:33:58.294 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:57 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:57.801+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:30.281285+0000 front 2024-05-15T13:33:30.281128+0000 (oldest deadline 2024-05-15T13:33:53.181040+0000) 2024-05-15T13:33:58.525 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:58 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:58.195+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:58.525 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:58 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:58.195+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:33:58.525 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:58 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:58.195+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:33:58.525 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:58 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:58.195+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:33:58.525 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:58 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:58.289+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:58.525 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:58 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:58.289+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:33:58.525 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:58 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:58.289+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:29.362499+0000 front 2024-05-15T13:33:29.362441+0000 (oldest deadline 2024-05-15T13:33:56.362927+0000) 2024-05-15T13:33:58.794 INFO:journalctl@ceph.osd.6.smithi120.stdout:May 15 13:33:58 smithi120 bash[171437]: ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-6 2024-05-15T13:33:58.942 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:58 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:58.771+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:58.942 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:58 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:58.771+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:33:58.943 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:58 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:58.771+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:33:58.943 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:58 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:58.522+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:58.943 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:58 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:58.522+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:33:58.943 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:58 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:58.522+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:33:59.294 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:58 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:58.837+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:33:59.294 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:58 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:58.837+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:33:59.294 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:58 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:58.838+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.879654+0000 front 2024-05-15T13:33:23.879744+0000 (oldest deadline 2024-05-15T13:33:46.779721+0000) 2024-05-15T13:33:59.294 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:58 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:58.838+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:30.281285+0000 front 2024-05-15T13:33:30.281128+0000 (oldest deadline 2024-05-15T13:33:53.181040+0000) 2024-05-15T13:33:59.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:59 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:59.276+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:33:59.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:59 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:59.276+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:33:59.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:33:59 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:33:59.276+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:29.362499+0000 front 2024-05-15T13:33:29.362441+0000 (oldest deadline 2024-05-15T13:33:56.362927+0000) 2024-05-15T13:33:59.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:59 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:59.176+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:33:59.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:59 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:59.176+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:33:59.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:59 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:59.176+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:33:59.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:33:59 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:33:59.176+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:33:59.942 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:59 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:59.723+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:33:59.942 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:59 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:59.723+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:33:59.943 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:33:59 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:33:59.723+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:33:59.943 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:59 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:59.545+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:33:59.943 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:59 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:59.545+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:33:59.943 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:33:59 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:33:59.546+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:00.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:59 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:59.850+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:34:00.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:59 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:59.850+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:34:00.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:59 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:59.850+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.879654+0000 front 2024-05-15T13:33:23.879744+0000 (oldest deadline 2024-05-15T13:33:46.779721+0000) 2024-05-15T13:34:00.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:33:59 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:33:59.850+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:30.281285+0000 front 2024-05-15T13:33:30.281128+0000 (oldest deadline 2024-05-15T13:33:53.181040+0000) 2024-05-15T13:34:00.433 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.6.service' 2024-05-15T13:34:00.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:00 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:00.275+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:34:00.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:00 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:00.275+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:34:00.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:00 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:00.275+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:29.362499+0000 front 2024-05-15T13:33:29.362441+0000 (oldest deadline 2024-05-15T13:33:56.362927+0000) 2024-05-15T13:34:00.442 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:00 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:00.142+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:34:00.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:00 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:00.142+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:34:00.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:00 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:00.142+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:34:00.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:00 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:00.142+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:34:00.475 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T13:34:00.475 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-05-15T13:34:00.475 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-05-15T13:34:00.475 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.7 2024-05-15T13:34:00.760 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:00 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:00.504+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:00.761 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:00 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:00.504+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:00.761 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:00 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:00.504+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:00.783 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:00 smithi120 systemd[1]: Stopping Ceph osd.7 for deaea04e-12b7-11ef-bc98-c7b262605968... 2024-05-15T13:34:00.783 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:00 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:00.779+0000 7f26e810f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T13:34:01.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:00 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:00.779+0000 7f26e810f700 -1 osd.7 1049 *** Got signal Terminated *** 2024-05-15T13:34:01.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:00 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:00.779+0000 7f26e810f700 -1 osd.7 1049 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T13:34:01.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:00 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:00.840+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:34:01.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:00 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:00.840+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:34:01.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:00 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:00.840+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.879654+0000 front 2024-05-15T13:33:23.879744+0000 (oldest deadline 2024-05-15T13:33:46.779721+0000) 2024-05-15T13:34:01.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:00 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:00.840+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:30.281285+0000 front 2024-05-15T13:33:30.281128+0000 (oldest deadline 2024-05-15T13:33:53.181040+0000) 2024-05-15T13:34:01.190 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:00 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:00.758+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:01.191 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:00 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:00.758+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:01.191 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:00 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:00.758+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:01.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:01 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:01.288+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:34:01.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:01 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:01.288+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:34:01.443 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:01 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:01.288+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:29.362499+0000 front 2024-05-15T13:33:29.362441+0000 (oldest deadline 2024-05-15T13:33:56.362927+0000) 2024-05-15T13:34:01.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:01 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:01.188+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:34:01.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:01 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:01.188+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:34:01.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:01 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:01.188+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:34:01.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:01 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:01.188+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:34:01.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:01 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:01.188+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.417350+0000 front 2024-05-15T13:33:36.417365+0000 (oldest deadline 2024-05-15T13:34:01.117249+0000) 2024-05-15T13:34:01.789 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:01 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:01.454+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:01.790 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:01 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:01.454+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:01.790 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:01 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:01.454+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:02.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:01 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:01.787+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:02.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:01 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:01.787+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:02.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:01 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:01.787+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:02.294 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:01 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:01.792+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:34:02.294 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:01 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:01.792+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:34:02.294 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:01 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:01.792+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.879654+0000 front 2024-05-15T13:33:23.879744+0000 (oldest deadline 2024-05-15T13:33:46.779721+0000) 2024-05-15T13:34:02.294 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:01 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:01.792+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:30.281285+0000 front 2024-05-15T13:33:30.281128+0000 (oldest deadline 2024-05-15T13:33:53.181040+0000) 2024-05-15T13:34:02.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:02 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:02.300+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:34:02.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:02 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:02.300+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:34:02.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:02 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:02.300+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:29.362499+0000 front 2024-05-15T13:33:29.362441+0000 (oldest deadline 2024-05-15T13:33:56.362927+0000) 2024-05-15T13:34:02.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:02 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:02.232+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:34:02.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:02 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:02.232+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:34:02.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:02 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:02.232+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:34:02.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:02 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:02.232+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:34:02.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:02 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:02.232+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.417350+0000 front 2024-05-15T13:33:36.417365+0000 (oldest deadline 2024-05-15T13:34:01.117249+0000) 2024-05-15T13:34:02.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:02 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:02.409+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:02.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:02 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:02.409+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:02.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:02 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:02.409+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:02.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:02 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:02.409+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:03.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:02 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:02.787+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:03.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:02 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:02.787+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:03.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:02 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:02.787+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:03.293 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:02 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:02.800+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:34:03.294 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:02 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:02.800+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:34:03.294 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:02 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:02.800+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.879654+0000 front 2024-05-15T13:33:23.879744+0000 (oldest deadline 2024-05-15T13:33:46.779721+0000) 2024-05-15T13:34:03.294 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:02 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:02.800+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:30.281285+0000 front 2024-05-15T13:33:30.281128+0000 (oldest deadline 2024-05-15T13:33:53.181040+0000) 2024-05-15T13:34:03.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:03 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:03.347+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:34:03.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:03 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:03.347+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:34:03.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:03 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:03.347+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:29.362499+0000 front 2024-05-15T13:33:29.362441+0000 (oldest deadline 2024-05-15T13:33:56.362927+0000) 2024-05-15T13:34:03.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:03 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:03.347+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.265313+0000 front 2024-05-15T13:33:40.265371+0000 (oldest deadline 2024-05-15T13:34:02.565175+0000) 2024-05-15T13:34:03.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:03 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:03.192+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:34:03.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:03 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:03.192+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:34:03.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:03 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:03.192+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:34:03.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:03 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:03.192+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:34:03.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:03 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:03.192+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.417350+0000 front 2024-05-15T13:33:36.417365+0000 (oldest deadline 2024-05-15T13:34:01.117249+0000) 2024-05-15T13:34:03.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:03 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:03.401+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:03.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:03 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:03.401+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:03.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:03 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:03.401+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:03.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:03 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:03.401+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:04.043 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:03 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:03.762+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:34:04.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:03 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:03.762+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:34:04.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:03 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:03.762+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.879654+0000 front 2024-05-15T13:33:23.879744+0000 (oldest deadline 2024-05-15T13:33:46.779721+0000) 2024-05-15T13:34:04.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:03 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:03.762+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:30.281285+0000 front 2024-05-15T13:33:30.281128+0000 (oldest deadline 2024-05-15T13:33:53.181040+0000) 2024-05-15T13:34:04.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:03 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:03.762+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:37.182444+0000 front 2024-05-15T13:33:37.182576+0000 (oldest deadline 2024-05-15T13:34:03.082604+0000) 2024-05-15T13:34:04.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:03 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:03.830+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:04.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:03 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:03.831+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:04.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:03 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:03.831+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:04.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:04 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:04.352+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:34:04.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:04 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:04.352+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:34:04.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:04 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:04.352+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:29.362499+0000 front 2024-05-15T13:33:29.362441+0000 (oldest deadline 2024-05-15T13:33:56.362927+0000) 2024-05-15T13:34:04.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:04 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:04.352+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.265313+0000 front 2024-05-15T13:33:40.265371+0000 (oldest deadline 2024-05-15T13:34:02.565175+0000) 2024-05-15T13:34:04.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:04 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:04.227+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:34:04.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:04 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:04.227+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:34:04.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:04 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:04.227+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:34:04.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:04 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:04.227+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:34:04.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:04 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:04.227+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.417350+0000 front 2024-05-15T13:33:36.417365+0000 (oldest deadline 2024-05-15T13:34:01.117249+0000) 2024-05-15T13:34:04.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:04 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:04.373+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:04.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:04 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:04.373+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:04.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:04 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:04.373+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:04.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:04 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:04.373+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:05.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:04 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:04.713+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:34:05.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:04 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:04.713+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:34:05.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:04 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:04.713+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.879654+0000 front 2024-05-15T13:33:23.879744+0000 (oldest deadline 2024-05-15T13:33:46.779721+0000) 2024-05-15T13:34:05.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:04 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:04.713+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:30.281285+0000 front 2024-05-15T13:33:30.281128+0000 (oldest deadline 2024-05-15T13:33:53.181040+0000) 2024-05-15T13:34:05.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:04 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:04.713+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:37.182444+0000 front 2024-05-15T13:33:37.182576+0000 (oldest deadline 2024-05-15T13:34:03.082604+0000) 2024-05-15T13:34:05.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:04 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:04.793+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:05.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:04 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:04.793+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:05.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:04 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:04.793+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:05.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:05 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:05.324+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:34:05.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:05 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:05.324+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:34:05.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:05 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:05.324+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:29.362499+0000 front 2024-05-15T13:33:29.362441+0000 (oldest deadline 2024-05-15T13:33:56.362927+0000) 2024-05-15T13:34:05.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:05 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:05.324+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.265313+0000 front 2024-05-15T13:33:40.265371+0000 (oldest deadline 2024-05-15T13:34:02.565175+0000) 2024-05-15T13:34:05.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:05 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:05.224+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:34:05.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:05 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:05.224+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:34:05.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:05 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:05.224+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:34:05.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:05 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:05.224+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:34:05.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:05 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:05.224+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.417350+0000 front 2024-05-15T13:33:36.417365+0000 (oldest deadline 2024-05-15T13:34:01.117249+0000) 2024-05-15T13:34:05.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:05 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:05.329+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:05.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:05 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:05.329+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:05.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:05 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:05.329+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:05.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:05 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:05.329+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:06.043 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:05 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:05.690+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:02.276075+0000 front 2024-05-15T13:33:02.275872+0000 (oldest deadline 2024-05-15T13:33:28.176007+0000) 2024-05-15T13:34:06.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:05 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:05.690+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.177886+0000 front 2024-05-15T13:33:12.177861+0000 (oldest deadline 2024-05-15T13:33:36.877992+0000) 2024-05-15T13:34:06.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:05 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:05.690+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:23.879654+0000 front 2024-05-15T13:33:23.879744+0000 (oldest deadline 2024-05-15T13:33:46.779721+0000) 2024-05-15T13:34:06.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:05 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:05.690+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:30.281285+0000 front 2024-05-15T13:33:30.281128+0000 (oldest deadline 2024-05-15T13:33:53.181040+0000) 2024-05-15T13:34:06.044 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:05 smithi120 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7[72467]: 2024-05-15T13:34:05.690+0000 7f26e06fb700 -1 osd.7 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:37.182444+0000 front 2024-05-15T13:33:37.182576+0000 (oldest deadline 2024-05-15T13:34:03.082604+0000) 2024-05-15T13:34:06.187 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:05 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:05.757+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:06.187 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:05 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:05.757+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:06.187 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:05 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:05.757+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:06.187 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:05 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:05.757+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:06.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:06 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:06.327+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:34:06.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:06 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:06.327+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:34:06.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:06 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:06.327+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:29.362499+0000 front 2024-05-15T13:33:29.362441+0000 (oldest deadline 2024-05-15T13:33:56.362927+0000) 2024-05-15T13:34:06.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:06 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:06.327+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.265313+0000 front 2024-05-15T13:33:40.265371+0000 (oldest deadline 2024-05-15T13:34:02.565175+0000) 2024-05-15T13:34:06.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:06 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:06.185+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:34:06.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:06 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:06.185+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:34:06.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:06 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:06.185+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:34:06.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:06 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:06.185+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:34:06.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:06 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:06.185+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.417350+0000 front 2024-05-15T13:33:36.417365+0000 (oldest deadline 2024-05-15T13:34:01.117249+0000) 2024-05-15T13:34:06.443 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:06 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:06.369+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:06.443 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:06 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:06.369+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:06.443 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:06 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:06.369+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:06.444 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:06 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:06.369+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:06.920 INFO:journalctl@ceph.osd.7.smithi120.stdout:May 15 13:34:06 smithi120 bash[171711]: ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-7 2024-05-15T13:34:07.190 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:06 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:06.744+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:07.190 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:06 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:06.744+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:07.190 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:06 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:06.744+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:07.190 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:06 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:06.744+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:07.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:07 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:07.346+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:34:07.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:07 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:07.346+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:34:07.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:07 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:07.346+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:29.362499+0000 front 2024-05-15T13:33:29.362441+0000 (oldest deadline 2024-05-15T13:33:56.362927+0000) 2024-05-15T13:34:07.443 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:07 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:07.346+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.265313+0000 front 2024-05-15T13:33:40.265371+0000 (oldest deadline 2024-05-15T13:34:02.565175+0000) 2024-05-15T13:34:07.443 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:07 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:07.375+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:07.443 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:07 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:07.375+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:07.443 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:07 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:07.375+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:07.443 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:07 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:07.375+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:07.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:07 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:07.187+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:34:07.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:07 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:07.187+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:34:07.444 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:07 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:07.187+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:34:07.444 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:07 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:07.187+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:34:07.444 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:07 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:07.187+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.417350+0000 front 2024-05-15T13:33:36.417365+0000 (oldest deadline 2024-05-15T13:34:01.117249+0000) 2024-05-15T13:34:08.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:07 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:07.735+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:08.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:07 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:07.735+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:08.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:07 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:07.735+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:08.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:07 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:07.735+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:08.667 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.7.service' 2024-05-15T13:34:08.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:08 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:08.317+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:34:08.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:08 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:08.317+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:34:08.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:08 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:08.317+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:29.362499+0000 front 2024-05-15T13:33:29.362441+0000 (oldest deadline 2024-05-15T13:33:56.362927+0000) 2024-05-15T13:34:08.692 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:08 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:08.317+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.265313+0000 front 2024-05-15T13:33:40.265371+0000 (oldest deadline 2024-05-15T13:34:02.565175+0000) 2024-05-15T13:34:08.692 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:08 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:08.191+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:34:08.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:08 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:08.192+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:34:08.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:08 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:08.192+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:34:08.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:08 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:08.192+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:34:08.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:08 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:08.192+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.417350+0000 front 2024-05-15T13:33:36.417365+0000 (oldest deadline 2024-05-15T13:34:01.117249+0000) 2024-05-15T13:34:08.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:08 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:08.418+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:08.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:08 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:08.418+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:08.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:08 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:08.418+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:08.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:08 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:08.418+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:08.708 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T13:34:08.709 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-05-15T13:34:08.709 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-05-15T13:34:08.709 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.8 2024-05-15T13:34:09.119 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:08 smithi177 systemd[1]: Stopping Ceph osd.8 for deaea04e-12b7-11ef-bc98-c7b262605968... 2024-05-15T13:34:09.120 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:08 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:08.765+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:09.120 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:08 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:08.765+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:09.120 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:08 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:08.765+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:09.120 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:08 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:08.765+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:09.442 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:09 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:09.151+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:34:09.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:09 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:09.151+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:34:09.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:09 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:09.151+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:34:09.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:09 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:09.151+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:34:09.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:09 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:09.151+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.417350+0000 front 2024-05-15T13:33:36.417365+0000 (oldest deadline 2024-05-15T13:34:01.117249+0000) 2024-05-15T13:34:09.443 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:09 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:09.116+0000 7fce954b0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T13:34:09.443 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:09 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:09.116+0000 7fce954b0700 -1 osd.8 1049 *** Got signal Terminated *** 2024-05-15T13:34:09.444 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:09 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:09.116+0000 7fce954b0700 -1 osd.8 1049 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T13:34:09.444 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:09 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:09.339+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:34:09.444 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:09 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:09.339+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:34:09.444 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:09 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:09.340+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:29.362499+0000 front 2024-05-15T13:33:29.362441+0000 (oldest deadline 2024-05-15T13:33:56.362927+0000) 2024-05-15T13:34:09.444 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:09 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:09.340+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.265313+0000 front 2024-05-15T13:33:40.265371+0000 (oldest deadline 2024-05-15T13:34:02.565175+0000) 2024-05-15T13:34:09.782 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:09 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:09.444+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:09.782 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:09 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:09.444+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:09.782 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:09 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:09.444+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:09.782 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:09 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:09.444+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:10.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:09 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:09.778+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:10.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:09 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:09.778+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:10.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:09 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:09.778+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:10.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:09 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:09.778+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:10.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:10 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:10.325+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:34:10.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:10 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:10.325+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:34:10.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:10 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:10.325+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:29.362499+0000 front 2024-05-15T13:33:29.362441+0000 (oldest deadline 2024-05-15T13:33:56.362927+0000) 2024-05-15T13:34:10.443 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:10 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:10.325+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.265313+0000 front 2024-05-15T13:33:40.265371+0000 (oldest deadline 2024-05-15T13:34:02.565175+0000) 2024-05-15T13:34:10.443 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:10 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:10.408+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:10.443 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:10 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:10.408+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:10.443 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:10 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:10.408+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:10.443 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:10 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:10.408+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:10.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:10 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:10.173+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:34:10.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:10 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:10.173+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:34:10.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:10 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:10.173+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:34:10.444 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:10 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:10.173+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:34:10.444 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:10 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:10.173+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.417350+0000 front 2024-05-15T13:33:36.417365+0000 (oldest deadline 2024-05-15T13:34:01.117249+0000) 2024-05-15T13:34:11.169 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:10 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:10.789+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:11.169 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:10 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:10.789+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:11.170 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:10 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:10.789+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:11.170 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:10 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:10.789+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:11.437 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:11 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:11.330+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:34:11.438 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:11 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:11.330+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:34:11.438 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:11 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:11.330+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:29.362499+0000 front 2024-05-15T13:33:29.362441+0000 (oldest deadline 2024-05-15T13:33:56.362927+0000) 2024-05-15T13:34:11.438 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:11 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:11.330+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.265313+0000 front 2024-05-15T13:33:40.265371+0000 (oldest deadline 2024-05-15T13:34:02.565175+0000) 2024-05-15T13:34:11.438 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:11 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:11.166+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:34:11.438 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:11 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:11.166+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:34:11.438 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:11 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:11.166+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:34:11.438 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:11 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:11.166+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:34:11.438 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:11 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:11.166+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.417350+0000 front 2024-05-15T13:33:36.417365+0000 (oldest deadline 2024-05-15T13:34:01.117249+0000) 2024-05-15T13:34:11.692 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:11 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:11.434+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:11.692 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:11 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:11.434+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:11.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:11 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:11.434+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:11.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:11 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:11.434+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:11.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:11 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:11.434+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:45.584858+0000 front 2024-05-15T13:33:45.584881+0000 (oldest deadline 2024-05-15T13:34:11.385203+0000) 2024-05-15T13:34:12.162 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:11 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:11.817+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:12.162 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:11 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:11.817+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:12.162 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:11 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:11.817+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:12.162 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:11 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:11.817+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:12.162 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:11 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:11.817+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:12.442 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:12 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:12.159+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:34:12.442 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:12 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:12.159+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:34:12.442 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:12 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:12.159+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:34:12.442 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:12 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:12.159+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:34:12.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:12 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:12.159+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.417350+0000 front 2024-05-15T13:33:36.417365+0000 (oldest deadline 2024-05-15T13:34:01.117249+0000) 2024-05-15T13:34:12.443 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:12 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:12.357+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:34:12.443 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:12 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:12.357+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:34:12.443 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:12 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:12.357+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:29.362499+0000 front 2024-05-15T13:33:29.362441+0000 (oldest deadline 2024-05-15T13:33:56.362927+0000) 2024-05-15T13:34:12.443 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:12 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:12.357+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.265313+0000 front 2024-05-15T13:33:40.265371+0000 (oldest deadline 2024-05-15T13:34:02.565175+0000) 2024-05-15T13:34:12.786 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:12 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:12.483+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:12.786 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:12 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:12.483+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:12.786 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:12 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:12.483+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:12.786 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:12 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:12.483+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:12.786 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:12 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:12.483+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:45.584858+0000 front 2024-05-15T13:33:45.584881+0000 (oldest deadline 2024-05-15T13:34:11.385203+0000) 2024-05-15T13:34:13.173 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:12 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:12.782+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:13.173 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:12 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:12.782+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:13.173 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:12 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:12.782+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:13.173 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:12 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:12.782+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:13.173 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:12 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:12.782+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:13.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:13 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:13.399+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.059923+0000 front 2024-05-15T13:33:07.059958+0000 (oldest deadline 2024-05-15T13:33:32.359910+0000) 2024-05-15T13:34:13.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:13 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:13.399+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:12.360382+0000 front 2024-05-15T13:33:12.360483+0000 (oldest deadline 2024-05-15T13:33:36.460362+0000) 2024-05-15T13:34:13.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:13 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:13.399+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:29.362499+0000 front 2024-05-15T13:33:29.362441+0000 (oldest deadline 2024-05-15T13:33:56.362927+0000) 2024-05-15T13:34:13.443 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:13 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8[50178]: 2024-05-15T13:34:13.399+0000 7fce91aa4700 -1 osd.8 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.265313+0000 front 2024-05-15T13:33:40.265371+0000 (oldest deadline 2024-05-15T13:34:02.565175+0000) 2024-05-15T13:34:13.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:13 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:13.169+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:34:13.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:13 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:13.169+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:34:13.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:13 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:13.169+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:34:13.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:13 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:13.169+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:34:13.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:13 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:13.169+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.417350+0000 front 2024-05-15T13:33:36.417365+0000 (oldest deadline 2024-05-15T13:34:01.117249+0000) 2024-05-15T13:34:13.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:13 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:13.169+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.018305+0000 front 2024-05-15T13:33:47.018347+0000 (oldest deadline 2024-05-15T13:34:12.318261+0000) 2024-05-15T13:34:13.769 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:13 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:13.483+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:13.769 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:13 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:13.483+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:13.769 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:13 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:13.483+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:13.769 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:13 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:13.483+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:13.770 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:13 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:13.483+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:45.584858+0000 front 2024-05-15T13:33:45.584881+0000 (oldest deadline 2024-05-15T13:34:11.385203+0000) 2024-05-15T13:34:14.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:13 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:13.766+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:14.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:13 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:13.766+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:14.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:13 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:13.766+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:14.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:13 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:13.766+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:14.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:13 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:13.766+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:14.500 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:14 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:14.206+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:34:14.500 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:14 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:14.206+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:34:14.500 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:14 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:14.206+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:34:14.500 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:14 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:14.206+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:34:14.501 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:14 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:14.206+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.417350+0000 front 2024-05-15T13:33:36.417365+0000 (oldest deadline 2024-05-15T13:34:01.117249+0000) 2024-05-15T13:34:14.501 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:14 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:14.206+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.018305+0000 front 2024-05-15T13:33:47.018347+0000 (oldest deadline 2024-05-15T13:34:12.318261+0000) 2024-05-15T13:34:14.757 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:14 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:14.496+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:14.757 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:14 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:14.496+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:14.757 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:14 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:14.496+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:14.757 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:14 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:14.496+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:14.757 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:14 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:14.496+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:45.584858+0000 front 2024-05-15T13:33:45.584881+0000 (oldest deadline 2024-05-15T13:34:11.385203+0000) 2024-05-15T13:34:15.007 INFO:journalctl@ceph.osd.8.smithi177.stdout:May 15 13:34:15 smithi177 bash[174784]: ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-8 2024-05-15T13:34:15.007 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:14 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:14.753+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:15.007 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:14 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:14.753+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:15.007 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:14 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:14.753+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:15.007 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:14 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:14.753+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:15.007 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:14 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:14.753+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:15.692 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:15 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:15.461+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:15.692 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:15 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:15.461+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:15.692 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:15 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:15.461+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:15.692 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:15 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:15.461+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:15.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:15 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:15.461+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:45.584858+0000 front 2024-05-15T13:33:45.584881+0000 (oldest deadline 2024-05-15T13:34:11.385203+0000) 2024-05-15T13:34:15.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:15 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:15.247+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:34:15.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:15 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:15.247+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:34:15.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:15 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:15.247+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:34:15.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:15 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:15.247+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:34:15.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:15 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:15.247+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.417350+0000 front 2024-05-15T13:33:36.417365+0000 (oldest deadline 2024-05-15T13:34:01.117249+0000) 2024-05-15T13:34:15.693 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:15 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:15.247+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.018305+0000 front 2024-05-15T13:33:47.018347+0000 (oldest deadline 2024-05-15T13:34:12.318261+0000) 2024-05-15T13:34:16.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:15 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:15.785+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:16.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:15 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:15.785+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:16.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:15 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:15.785+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:16.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:15 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:15.785+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:16.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:15 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:15.785+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:16.503 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:16 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:16.203+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:34:16.503 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:16 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:16.203+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:34:16.503 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:16 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:16.203+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:34:16.503 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:16 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:16.203+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:34:16.503 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:16 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:16.203+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.417350+0000 front 2024-05-15T13:33:36.417365+0000 (oldest deadline 2024-05-15T13:34:01.117249+0000) 2024-05-15T13:34:16.503 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:16 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:16.203+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.018305+0000 front 2024-05-15T13:33:47.018347+0000 (oldest deadline 2024-05-15T13:34:12.318261+0000) 2024-05-15T13:34:16.799 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:16 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:16.498+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:16.799 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:16 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:16.498+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:16.799 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:16 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:16.498+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:16.799 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:16 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:16.498+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:16.799 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:16 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:16.498+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:45.584858+0000 front 2024-05-15T13:33:45.584881+0000 (oldest deadline 2024-05-15T13:34:11.385203+0000) 2024-05-15T13:34:16.913 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.8.service' 2024-05-15T13:34:16.952 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T13:34:16.952 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-05-15T13:34:16.952 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-05-15T13:34:16.952 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.9 2024-05-15T13:34:17.170 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:17 smithi177 systemd[1]: Stopping Ceph osd.9 for deaea04e-12b7-11ef-bc98-c7b262605968... 2024-05-15T13:34:17.170 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:16 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:16.795+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:17.170 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:16 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:16.795+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:17.170 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:16 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:16.795+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:17.170 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:16 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:16.795+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:17.170 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:16 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:16.795+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:17.442 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:17 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:17.165+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:34:17.442 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:17 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:17.165+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:34:17.442 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:17 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:17.165+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:34:17.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:17 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:17.165+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:34:17.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:17 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:17.165+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.417350+0000 front 2024-05-15T13:33:36.417365+0000 (oldest deadline 2024-05-15T13:34:01.117249+0000) 2024-05-15T13:34:17.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:17 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:17.165+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.018305+0000 front 2024-05-15T13:33:47.018347+0000 (oldest deadline 2024-05-15T13:34:12.318261+0000) 2024-05-15T13:34:17.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:17 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:17.299+0000 7f798b093700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T13:34:17.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:17 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:17.299+0000 7f798b093700 -1 osd.9 1049 *** Got signal Terminated *** 2024-05-15T13:34:17.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:17 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:17.299+0000 7f798b093700 -1 osd.9 1049 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T13:34:17.843 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:17 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:17.453+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:17.843 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:17 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:17.453+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:17.843 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:17 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:17.453+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:17.843 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:17 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:17.453+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:17.843 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:17 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:17.453+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:45.584858+0000 front 2024-05-15T13:33:45.584881+0000 (oldest deadline 2024-05-15T13:34:11.385203+0000) 2024-05-15T13:34:18.151 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:17 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:17.840+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:18.151 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:17 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:17.840+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:18.151 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:17 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:17.840+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:18.152 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:17 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:17.840+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:18.152 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:17 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:17.840+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:18.442 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:18 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:18.147+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:34:18.442 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:18 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:18.147+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:34:18.442 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:18 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:18.147+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:34:18.442 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:18 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:18.147+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:34:18.442 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:18 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:18.147+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.417350+0000 front 2024-05-15T13:33:36.417365+0000 (oldest deadline 2024-05-15T13:34:01.117249+0000) 2024-05-15T13:34:18.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:18 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:18.147+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.018305+0000 front 2024-05-15T13:33:47.018347+0000 (oldest deadline 2024-05-15T13:34:12.318261+0000) 2024-05-15T13:34:18.867 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:18 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:18.461+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:18.868 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:18 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:18.461+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:18.868 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:18 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:18.461+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:18.868 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:18 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:18.461+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:18.868 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:18 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:18.461+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:45.584858+0000 front 2024-05-15T13:33:45.584881+0000 (oldest deadline 2024-05-15T13:34:11.385203+0000) 2024-05-15T13:34:19.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:18 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:18.863+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:19.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:18 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:18.863+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:19.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:18 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:18.863+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:19.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:18 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:18.863+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:19.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:18 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:18.863+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:19.193 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:19 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:19.103+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:34:19.193 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:19 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:19.103+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:34:19.193 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:19 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:19.103+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:34:19.193 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:19 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:19.103+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:34:19.193 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:19 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:19.103+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.417350+0000 front 2024-05-15T13:33:36.417365+0000 (oldest deadline 2024-05-15T13:34:01.117249+0000) 2024-05-15T13:34:19.193 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:19 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:19.103+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.018305+0000 front 2024-05-15T13:33:47.018347+0000 (oldest deadline 2024-05-15T13:34:12.318261+0000) 2024-05-15T13:34:19.849 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:19 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:19.504+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:19.850 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:19 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:19.504+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:19.850 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:19 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:19.504+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:19.850 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:19 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:19.504+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:19.850 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:19 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:19.504+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:45.584858+0000 front 2024-05-15T13:33:45.584881+0000 (oldest deadline 2024-05-15T13:34:11.385203+0000) 2024-05-15T13:34:19.850 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:19 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:19.504+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:56.586853+0000 front 2024-05-15T13:33:56.586888+0000 (oldest deadline 2024-05-15T13:34:19.486713+0000) 2024-05-15T13:34:20.138 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:19 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:19.845+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:20.139 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:19 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:19.845+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:20.139 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:19 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:19.845+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:20.139 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:19 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:19.845+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:20.139 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:19 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:19.846+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:20.139 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:19 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:19.846+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:54.902049+0000 front 2024-05-15T13:33:54.902150+0000 (oldest deadline 2024-05-15T13:34:19.002003+0000) 2024-05-15T13:34:20.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:20 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:20.136+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:34:20.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:20 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:20.136+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:34:20.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:20 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:20.136+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:34:20.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:20 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:20.136+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:34:20.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:20 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:20.136+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.417350+0000 front 2024-05-15T13:33:36.417365+0000 (oldest deadline 2024-05-15T13:34:01.117249+0000) 2024-05-15T13:34:20.443 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:20 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:20.136+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.018305+0000 front 2024-05-15T13:33:47.018347+0000 (oldest deadline 2024-05-15T13:34:12.318261+0000) 2024-05-15T13:34:20.897 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:20 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:20.523+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:20.897 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:20 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:20.523+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:20.897 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:20 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:20.523+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:20.897 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:20 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:20.523+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:20.897 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:20 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:20.523+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:45.584858+0000 front 2024-05-15T13:33:45.584881+0000 (oldest deadline 2024-05-15T13:34:11.385203+0000) 2024-05-15T13:34:20.898 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:20 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:20.523+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:56.586853+0000 front 2024-05-15T13:33:56.586888+0000 (oldest deadline 2024-05-15T13:34:19.486713+0000) 2024-05-15T13:34:21.193 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:21 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:21.093+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:34:21.193 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:21 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:21.093+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:34:21.193 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:21 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:21.093+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:34:21.193 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:21 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:21.093+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:34:21.193 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:21 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:21.093+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.417350+0000 front 2024-05-15T13:33:36.417365+0000 (oldest deadline 2024-05-15T13:34:01.117249+0000) 2024-05-15T13:34:21.193 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:21 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:21.093+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.018305+0000 front 2024-05-15T13:33:47.018347+0000 (oldest deadline 2024-05-15T13:34:12.318261+0000) 2024-05-15T13:34:21.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:20 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:20.893+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:21.194 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:20 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:20.893+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:21.194 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:20 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:20.893+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:21.194 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:20 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:20.893+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:21.194 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:20 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:20.893+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:21.194 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:20 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:20.893+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:54.902049+0000 front 2024-05-15T13:33:54.902150+0000 (oldest deadline 2024-05-15T13:34:19.002003+0000) 2024-05-15T13:34:21.889 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:21 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:21.525+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:21.889 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:21 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:21.525+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:21.889 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:21 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:21.525+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:21.889 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:21 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:21.525+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:21.889 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:21 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:21.525+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:45.584858+0000 front 2024-05-15T13:33:45.584881+0000 (oldest deadline 2024-05-15T13:34:11.385203+0000) 2024-05-15T13:34:21.889 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:21 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:21.525+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:56.586853+0000 front 2024-05-15T13:33:56.586888+0000 (oldest deadline 2024-05-15T13:34:19.486713+0000) 2024-05-15T13:34:22.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:21 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:21.885+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:22.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:21 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:21.885+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:22.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:21 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:21.885+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:22.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:21 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:21.885+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:22.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:21 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:21.885+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:22.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:21 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:21.885+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:54.902049+0000 front 2024-05-15T13:33:54.902150+0000 (oldest deadline 2024-05-15T13:34:19.002003+0000) 2024-05-15T13:34:22.193 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:22 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:22.098+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.113240+0000 front 2024-05-15T13:33:07.113190+0000 (oldest deadline 2024-05-15T13:33:31.813286+0000) 2024-05-15T13:34:22.193 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:22 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:22.098+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-05-15T13:33:11.813951+0000 front 2024-05-15T13:33:11.813812+0000 (oldest deadline 2024-05-15T13:33:37.713865+0000) 2024-05-15T13:34:22.193 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:22 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:22.098+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.415017+0000 front 2024-05-15T13:33:22.415072+0000 (oldest deadline 2024-05-15T13:33:47.615625+0000) 2024-05-15T13:34:22.193 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:22 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:22.098+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.716934+0000 front 2024-05-15T13:33:31.716961+0000 (oldest deadline 2024-05-15T13:33:56.416829+0000) 2024-05-15T13:34:22.193 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:22 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:22.098+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.417350+0000 front 2024-05-15T13:33:36.417365+0000 (oldest deadline 2024-05-15T13:34:01.117249+0000) 2024-05-15T13:34:22.193 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:22 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:22.098+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.018305+0000 front 2024-05-15T13:33:47.018347+0000 (oldest deadline 2024-05-15T13:34:12.318261+0000) 2024-05-15T13:34:22.193 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:22 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9[57213]: 2024-05-15T13:34:22.098+0000 7f7987687700 -1 osd.9 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:56.419370+0000 front 2024-05-15T13:33:56.419403+0000 (oldest deadline 2024-05-15T13:34:21.119343+0000) 2024-05-15T13:34:22.914 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:22 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:22.523+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:22.915 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:22 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:22.523+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:22.915 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:22 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:22.523+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:22.915 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:22 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:22.523+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:22.915 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:22 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:22.523+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:45.584858+0000 front 2024-05-15T13:33:45.584881+0000 (oldest deadline 2024-05-15T13:34:11.385203+0000) 2024-05-15T13:34:22.915 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:22 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:22.523+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:56.586853+0000 front 2024-05-15T13:33:56.586888+0000 (oldest deadline 2024-05-15T13:34:19.486713+0000) 2024-05-15T13:34:23.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:22 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:22.910+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:23.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:22 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:22.910+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:23.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:22 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:22.910+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:23.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:22 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:22.910+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:23.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:22 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:22.910+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:23.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:22 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:22.910+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:54.902049+0000 front 2024-05-15T13:33:54.902150+0000 (oldest deadline 2024-05-15T13:34:19.002003+0000) 2024-05-15T13:34:23.692 INFO:journalctl@ceph.osd.9.smithi177.stdout:May 15 13:34:23 smithi177 bash[175048]: ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-9 2024-05-15T13:34:23.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:23 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:23.488+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:23.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:23 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:23.488+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:23.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:23 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:23.488+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:23.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:23 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:23.488+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:23.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:23 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:23.488+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:45.584858+0000 front 2024-05-15T13:33:45.584881+0000 (oldest deadline 2024-05-15T13:34:11.385203+0000) 2024-05-15T13:34:23.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:23 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:23.488+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:56.586853+0000 front 2024-05-15T13:33:56.586888+0000 (oldest deadline 2024-05-15T13:34:19.486713+0000) 2024-05-15T13:34:24.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:23 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:23.925+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:24.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:23 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:23.925+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:24.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:23 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:23.925+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:24.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:23 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:23.925+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:24.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:23 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:23.925+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:24.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:23 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:23.925+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:54.902049+0000 front 2024-05-15T13:33:54.902150+0000 (oldest deadline 2024-05-15T13:34:19.002003+0000) 2024-05-15T13:34:24.871 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:24 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:24.533+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:24.871 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:24 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:24.533+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:24.872 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:24 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:24.533+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:24.872 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:24 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:24.533+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:24.872 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:24 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:24.533+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:45.584858+0000 front 2024-05-15T13:33:45.584881+0000 (oldest deadline 2024-05-15T13:34:11.385203+0000) 2024-05-15T13:34:24.872 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:24 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:24.533+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:56.586853+0000 front 2024-05-15T13:33:56.586888+0000 (oldest deadline 2024-05-15T13:34:19.486713+0000) 2024-05-15T13:34:25.182 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:24 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:24.893+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:25.182 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:24 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:24.893+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:25.183 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:24 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:24.893+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:25.183 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:24 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:24.893+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:25.183 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:24 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:24.893+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:25.183 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:24 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:24.893+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:54.902049+0000 front 2024-05-15T13:33:54.902150+0000 (oldest deadline 2024-05-15T13:34:19.002003+0000) 2024-05-15T13:34:25.190 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.9.service' 2024-05-15T13:34:25.230 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T13:34:25.231 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-05-15T13:34:25.231 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-05-15T13:34:25.231 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.10 2024-05-15T13:34:25.442 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:25 smithi177 systemd[1]: Stopping Ceph osd.10 for deaea04e-12b7-11ef-bc98-c7b262605968... 2024-05-15T13:34:25.909 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:25 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:25.518+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:25.909 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:25 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:25.518+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:25.909 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:25 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:25.518+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:25.909 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:25 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:25.518+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:25.909 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:25 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:25.518+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:45.584858+0000 front 2024-05-15T13:33:45.584881+0000 (oldest deadline 2024-05-15T13:34:11.385203+0000) 2024-05-15T13:34:25.909 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:25 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:25.518+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:56.586853+0000 front 2024-05-15T13:33:56.586888+0000 (oldest deadline 2024-05-15T13:34:19.486713+0000) 2024-05-15T13:34:25.909 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:25 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:25.543+0000 7f5873c84700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T13:34:25.910 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:25 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:25.543+0000 7f5873c84700 -1 osd.10 1049 *** Got signal Terminated *** 2024-05-15T13:34:25.910 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:25 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:25.543+0000 7f5873c84700 -1 osd.10 1049 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T13:34:26.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:25 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:25.906+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:26.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:25 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:25.906+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:26.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:25 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:25.906+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:26.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:25 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:25.906+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:26.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:25 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:25.906+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:26.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:25 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:25.906+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:54.902049+0000 front 2024-05-15T13:33:54.902150+0000 (oldest deadline 2024-05-15T13:34:19.002003+0000) 2024-05-15T13:34:26.942 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:26 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:26.477+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:26.942 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:26 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:26.477+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:26.942 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:26 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:26.477+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:26.942 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:26 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:26.477+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:26.943 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:26 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:26.477+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:45.584858+0000 front 2024-05-15T13:33:45.584881+0000 (oldest deadline 2024-05-15T13:34:11.385203+0000) 2024-05-15T13:34:26.943 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:26 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:26.477+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:56.586853+0000 front 2024-05-15T13:33:56.586888+0000 (oldest deadline 2024-05-15T13:34:19.486713+0000) 2024-05-15T13:34:27.442 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:26 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:26.940+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:27.442 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:26 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:26.940+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:27.442 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:26 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:26.940+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:27.442 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:26 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:26.940+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:27.443 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:26 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:26.940+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:27.443 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:26 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:26.940+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:54.902049+0000 front 2024-05-15T13:33:54.902150+0000 (oldest deadline 2024-05-15T13:34:19.002003+0000) 2024-05-15T13:34:27.942 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:27 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:27.443+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:27.942 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:27 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:27.443+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:27.942 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:27 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:27.443+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:27.943 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:27 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:27.443+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:27.943 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:27 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:27.443+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:45.584858+0000 front 2024-05-15T13:33:45.584881+0000 (oldest deadline 2024-05-15T13:34:11.385203+0000) 2024-05-15T13:34:27.943 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:27 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:27.443+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:56.586853+0000 front 2024-05-15T13:33:56.586888+0000 (oldest deadline 2024-05-15T13:34:19.486713+0000) 2024-05-15T13:34:28.426 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:27 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:27.956+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:28.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:27 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:27.956+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:28.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:27 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:27.956+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:28.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:27 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:27.956+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:28.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:27 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:27.956+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:28.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:27 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:27.956+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:54.902049+0000 front 2024-05-15T13:33:54.902150+0000 (oldest deadline 2024-05-15T13:34:19.002003+0000) 2024-05-15T13:34:28.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:27 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:27.956+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6828 osd.7 since back 2024-05-15T13:34:04.303320+0000 front 2024-05-15T13:34:04.303309+0000 (oldest deadline 2024-05-15T13:34:27.203240+0000) 2024-05-15T13:34:28.692 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:28 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:28.424+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:28.692 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:28 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:28.424+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:28.692 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:28 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:28.424+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:28.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:28 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:28.424+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:28.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:28 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:28.424+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:45.584858+0000 front 2024-05-15T13:33:45.584881+0000 (oldest deadline 2024-05-15T13:34:11.385203+0000) 2024-05-15T13:34:28.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:28 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:28.424+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:56.586853+0000 front 2024-05-15T13:33:56.586888+0000 (oldest deadline 2024-05-15T13:34:19.486713+0000) 2024-05-15T13:34:29.426 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:28 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:28.955+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:29.426 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:28 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:28.955+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:29.426 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:28 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:28.955+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:29.426 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:28 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:28.955+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:29.426 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:28 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:28.955+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:29.426 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:28 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:28.955+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:54.902049+0000 front 2024-05-15T13:33:54.902150+0000 (oldest deadline 2024-05-15T13:34:19.002003+0000) 2024-05-15T13:34:29.426 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:28 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:28.955+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6828 osd.7 since back 2024-05-15T13:34:04.303320+0000 front 2024-05-15T13:34:04.303309+0000 (oldest deadline 2024-05-15T13:34:27.203240+0000) 2024-05-15T13:34:29.692 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:29 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:29.423+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:29.692 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:29 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:29.423+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:29.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:29 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:29.423+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:29.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:29 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:29.423+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:29.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:29 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:29.423+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:45.584858+0000 front 2024-05-15T13:33:45.584881+0000 (oldest deadline 2024-05-15T13:34:11.385203+0000) 2024-05-15T13:34:29.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:29 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:29.423+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:56.586853+0000 front 2024-05-15T13:33:56.586888+0000 (oldest deadline 2024-05-15T13:34:19.486713+0000) 2024-05-15T13:34:29.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:29 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:29.423+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6828 osd.7 since back 2024-05-15T13:34:01.688296+0000 front 2024-05-15T13:34:01.688346+0000 (oldest deadline 2024-05-15T13:34:29.288730+0000) 2024-05-15T13:34:30.409 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:29 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:29.951+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:30.410 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:29 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:29.951+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:30.410 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:29 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:29.951+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:30.410 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:29 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:29.951+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:30.410 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:29 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:29.951+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:30.410 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:29 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:29.951+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:54.902049+0000 front 2024-05-15T13:33:54.902150+0000 (oldest deadline 2024-05-15T13:34:19.002003+0000) 2024-05-15T13:34:30.410 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:29 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:29.951+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6828 osd.7 since back 2024-05-15T13:34:04.303320+0000 front 2024-05-15T13:34:04.303309+0000 (oldest deadline 2024-05-15T13:34:27.203240+0000) 2024-05-15T13:34:30.692 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:30 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:30.407+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:03.875991+0000 front 2024-05-15T13:33:03.876011+0000 (oldest deadline 2024-05-15T13:33:27.876594+0000) 2024-05-15T13:34:30.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:30 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:30.407+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:22.880766+0000 front 2024-05-15T13:33:22.880882+0000 (oldest deadline 2024-05-15T13:33:44.580880+0000) 2024-05-15T13:34:30.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:30 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:30.407+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:26.881879+0000 front 2024-05-15T13:33:26.881851+0000 (oldest deadline 2024-05-15T13:33:56.282443+0000) 2024-05-15T13:34:30.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:30 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:30.407+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:36.283160+0000 front 2024-05-15T13:33:36.283012+0000 (oldest deadline 2024-05-15T13:34:02.182962+0000) 2024-05-15T13:34:30.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:30 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:30.407+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:45.584858+0000 front 2024-05-15T13:33:45.584881+0000 (oldest deadline 2024-05-15T13:34:11.385203+0000) 2024-05-15T13:34:30.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:30 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:30.407+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:56.586853+0000 front 2024-05-15T13:33:56.586888+0000 (oldest deadline 2024-05-15T13:34:19.486713+0000) 2024-05-15T13:34:30.693 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:30 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10[64387]: 2024-05-15T13:34:30.407+0000 7f5870278700 -1 osd.10 1049 heartbeat_check: no reply from 172.21.15.120:6828 osd.7 since back 2024-05-15T13:34:01.688296+0000 front 2024-05-15T13:34:01.688346+0000 (oldest deadline 2024-05-15T13:34:29.288730+0000) 2024-05-15T13:34:31.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:30 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:30.938+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:31.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:30 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:30.938+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:31.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:30 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:30.938+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:31.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:30 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:30.938+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:31.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:30 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:30.938+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:31.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:30 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:30.938+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:54.902049+0000 front 2024-05-15T13:33:54.902150+0000 (oldest deadline 2024-05-15T13:34:19.002003+0000) 2024-05-15T13:34:31.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:30 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:30.938+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6828 osd.7 since back 2024-05-15T13:34:04.303320+0000 front 2024-05-15T13:34:04.303309+0000 (oldest deadline 2024-05-15T13:34:27.203240+0000) 2024-05-15T13:34:31.692 INFO:journalctl@ceph.osd.10.smithi177.stdout:May 15 13:34:31 smithi177 bash[175316]: ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-10 2024-05-15T13:34:32.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:31 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:31.915+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:32.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:31 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:31.915+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:32.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:31 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:31.915+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:32.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:31 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:31.915+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:32.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:31 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:31.915+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:32.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:31 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:31.915+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:54.902049+0000 front 2024-05-15T13:33:54.902150+0000 (oldest deadline 2024-05-15T13:34:19.002003+0000) 2024-05-15T13:34:32.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:31 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:31.915+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6828 osd.7 since back 2024-05-15T13:34:04.303320+0000 front 2024-05-15T13:34:04.303309+0000 (oldest deadline 2024-05-15T13:34:27.203240+0000) 2024-05-15T13:34:33.040 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:32 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:32.921+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:33.040 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:32 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:32.921+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:33.041 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:32 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:32.921+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:33.041 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:32 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:32.921+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:33.041 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:32 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:32.921+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:33.041 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:32 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:32.921+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:54.902049+0000 front 2024-05-15T13:33:54.902150+0000 (oldest deadline 2024-05-15T13:34:19.002003+0000) 2024-05-15T13:34:33.041 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:32 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:32.921+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6828 osd.7 since back 2024-05-15T13:34:04.303320+0000 front 2024-05-15T13:34:04.303309+0000 (oldest deadline 2024-05-15T13:34:27.203240+0000) 2024-05-15T13:34:33.409 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.10.service' 2024-05-15T13:34:33.449 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T13:34:33.450 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-05-15T13:34:33.450 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-05-15T13:34:33.450 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.11 2024-05-15T13:34:33.692 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:33 smithi177 systemd[1]: Stopping Ceph osd.11 for deaea04e-12b7-11ef-bc98-c7b262605968... 2024-05-15T13:34:34.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:33 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:33.747+0000 7fc21a47c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T13:34:34.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:33 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:33.747+0000 7fc21a47c700 -1 osd.11 1049 *** Got signal Terminated *** 2024-05-15T13:34:34.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:33 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:33.747+0000 7fc21a47c700 -1 osd.11 1049 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T13:34:34.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:33 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:33.925+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:34.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:33 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:33.925+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:34.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:33 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:33.925+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:34.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:33 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:33.925+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:34.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:33 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:33.925+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:34.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:33 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:33.925+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:54.902049+0000 front 2024-05-15T13:33:54.902150+0000 (oldest deadline 2024-05-15T13:34:19.002003+0000) 2024-05-15T13:34:34.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:33 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:33.925+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6828 osd.7 since back 2024-05-15T13:34:04.303320+0000 front 2024-05-15T13:34:04.303309+0000 (oldest deadline 2024-05-15T13:34:27.203240+0000) 2024-05-15T13:34:35.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:34 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:34.919+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:35.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:34 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:34.919+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:35.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:34 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:34.919+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:35.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:34 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:34.919+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:35.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:34 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:34.919+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:35.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:34 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:34.919+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:54.902049+0000 front 2024-05-15T13:33:54.902150+0000 (oldest deadline 2024-05-15T13:34:19.002003+0000) 2024-05-15T13:34:35.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:34 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:34.919+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6828 osd.7 since back 2024-05-15T13:34:04.303320+0000 front 2024-05-15T13:34:04.303309+0000 (oldest deadline 2024-05-15T13:34:27.203240+0000) 2024-05-15T13:34:36.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:35 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:35.906+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:36.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:35 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:35.906+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:36.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:35 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:35.906+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:36.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:35 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:35.906+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:36.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:35 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:35.906+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:36.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:35 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:35.906+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:54.902049+0000 front 2024-05-15T13:33:54.902150+0000 (oldest deadline 2024-05-15T13:34:19.002003+0000) 2024-05-15T13:34:36.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:35 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:35.906+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6828 osd.7 since back 2024-05-15T13:34:04.303320+0000 front 2024-05-15T13:34:04.303309+0000 (oldest deadline 2024-05-15T13:34:27.203240+0000) 2024-05-15T13:34:37.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:36 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:36.903+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:37.192 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:36 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:36.903+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:37.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:36 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:36.903+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:37.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:36 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:36.903+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:37.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:36 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:36.903+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:37.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:36 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:36.903+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:54.902049+0000 front 2024-05-15T13:33:54.902150+0000 (oldest deadline 2024-05-15T13:34:19.002003+0000) 2024-05-15T13:34:37.193 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:36 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:36.903+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6828 osd.7 since back 2024-05-15T13:34:04.303320+0000 front 2024-05-15T13:34:04.303309+0000 (oldest deadline 2024-05-15T13:34:27.203240+0000) 2024-05-15T13:34:38.442 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:37 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:37.948+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-05-15T13:33:07.292371+0000 front 2024-05-15T13:33:07.292444+0000 (oldest deadline 2024-05-15T13:33:31.992506+0000) 2024-05-15T13:34:38.443 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:37 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:37.948+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-05-15T13:33:21.696099+0000 front 2024-05-15T13:33:21.696231+0000 (oldest deadline 2024-05-15T13:33:45.597229+0000) 2024-05-15T13:34:38.443 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:37 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:37.948+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.27:6830 osd.3 since back 2024-05-15T13:33:31.498673+0000 front 2024-05-15T13:33:31.498685+0000 (oldest deadline 2024-05-15T13:33:56.198500+0000) 2024-05-15T13:34:38.443 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:37 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:37.948+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6804 osd.4 since back 2024-05-15T13:33:40.299695+0000 front 2024-05-15T13:33:40.299857+0000 (oldest deadline 2024-05-15T13:34:04.999749+0000) 2024-05-15T13:34:38.443 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:37 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:37.948+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-05-15T13:33:47.300789+0000 front 2024-05-15T13:33:47.300885+0000 (oldest deadline 2024-05-15T13:34:11.400785+0000) 2024-05-15T13:34:38.443 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:37 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:37.948+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6820 osd.6 since back 2024-05-15T13:33:54.902049+0000 front 2024-05-15T13:33:54.902150+0000 (oldest deadline 2024-05-15T13:34:19.002003+0000) 2024-05-15T13:34:38.443 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:37 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:37.948+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.120:6828 osd.7 since back 2024-05-15T13:34:04.303320+0000 front 2024-05-15T13:34:04.303309+0000 (oldest deadline 2024-05-15T13:34:27.203240+0000) 2024-05-15T13:34:38.443 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:37 smithi177 ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11[71458]: 2024-05-15T13:34:37.948+0000 7fc216a70700 -1 osd.11 1049 heartbeat_check: no reply from 172.21.15.177:6804 osd.8 since back 2024-05-15T13:34:12.404758+0000 front 2024-05-15T13:34:12.404852+0000 (oldest deadline 2024-05-15T13:34:37.104820+0000) 2024-05-15T13:34:39.778 INFO:journalctl@ceph.osd.11.smithi177.stdout:May 15 13:34:39 smithi177 bash[175580]: ceph-deaea04e-12b7-11ef-bc98-c7b262605968-osd-11 2024-05-15T13:34:41.507 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-deaea04e-12b7-11ef-bc98-c7b262605968@osd.11.service' 2024-05-15T13:34:41.548 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T13:34:41.548 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-05-15T13:34:41.548 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid deaea04e-12b7-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-15T13:34:41.795 INFO:teuthology.orchestra.run.smithi027.stdout:Deleting cluster with fsid: deaea04e-12b7-11ef-bc98-c7b262605968 2024-05-15T13:34:43.118 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid deaea04e-12b7-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-15T13:34:43.347 INFO:teuthology.orchestra.run.smithi120.stdout:Deleting cluster with fsid: deaea04e-12b7-11ef-bc98-c7b262605968 2024-05-15T13:34:44.507 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid deaea04e-12b7-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-15T13:34:44.732 INFO:teuthology.orchestra.run.smithi177.stdout:Deleting cluster with fsid: deaea04e-12b7-11ef-bc98-c7b262605968 2024-05-15T13:34:45.685 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-15T13:34:45.722 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-15T13:34:45.759 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-15T13:34:45.788 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-15T13:34:45.790 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707109/remote/smithi027/crash 2024-05-15T13:34:45.791 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/crash -- . 2024-05-15T13:34:45.854 INFO:teuthology.orchestra.run.smithi027.stderr:tar: /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-15T13:34:45.854 INFO:teuthology.orchestra.run.smithi027.stderr:tar: Error is not recoverable: exiting now 2024-05-15T13:34:45.856 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707109/remote/smithi120/crash 2024-05-15T13:34:45.857 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/crash -- . 2024-05-15T13:34:45.934 INFO:teuthology.orchestra.run.smithi120.stderr:tar: /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-15T13:34:45.934 INFO:teuthology.orchestra.run.smithi120.stderr:tar: Error is not recoverable: exiting now 2024-05-15T13:34:45.936 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707109/remote/smithi177/crash 2024-05-15T13:34:45.937 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/crash -- . 2024-05-15T13:34:46.000 INFO:teuthology.orchestra.run.smithi177.stderr:tar: /var/lib/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-15T13:34:46.000 INFO:teuthology.orchestra.run.smithi177.stderr:tar: Error is not recoverable: exiting now 2024-05-15T13:34:46.001 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-15T13:34:46.001 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | head -n 1 2024-05-15T13:34:46.132 INFO:teuthology.orchestra.run.smithi027.stdout:2024-05-15T12:50:00.000151+0000 mon.a (mon.0) 1068 : cluster 3 [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-05-15T13:34:46.133 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-05-15T13:34:46.133 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[SEC\]' /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | head -n 1 2024-05-15T13:34:46.173 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[ERR\]' /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | head -n 1 2024-05-15T13:34:46.251 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[WRN\]' /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | head -n 1 2024-05-15T13:34:46.326 INFO:teuthology.orchestra.run.smithi027.stdout:2024-05-15T12:50:00.000151+0000 mon.a (mon.0) 1068 : cluster 3 [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-05-15T13:34:46.327 INFO:tasks.cephadm:Compressing logs... 2024-05-15T13:34:46.327 DEBUG:teuthology.orchestra.run.smithi027:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-15T13:34:46.369 DEBUG:teuthology.orchestra.run.smithi120:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-15T13:34:46.372 DEBUG:teuthology.orchestra.run.smithi177:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-15T13:34:46.412 INFO:teuthology.orchestra.run.smithi177.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-15T13:34:46.416 INFO:teuthology.orchestra.run.smithi027.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-15T13:34:46.416 INFO:teuthology.orchestra.run.smithi027.stderr:: No such file or directory 2024-05-15T13:34:46.419 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-15T13:34:46.421 INFO:teuthology.orchestra.run.smithi120.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-15T13:34:46.422 INFO:teuthology.orchestra.run.smithi120.stderr:: No such file or directory 2024-05-15T13:34:46.424 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.11.log 2024-05-15T13:34:46.425 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-volume.log 2024-05-15T13:34:46.425 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.audit.log 2024-05-15T13:34:46.425 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.cephadm.log 2024-05-15T13:34:46.425 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.10.log 2024-05-15T13:34:46.425 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.8.log 2024-05-15T13:34:46.425 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-mon.c.log 2024-05-15T13:34:46.425 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.9.log 2024-05-15T13:34:46.425 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.8.log: /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.log 2024-05-15T13:34:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-volume.log 2024-05-15T13:34:46.427 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.2.log 2024-05-15T13:34:46.427 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-mgr.y.log 2024-05-15T13:34:46.427 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.audit.log 2024-05-15T13:34:46.428 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.cephadm.log 2024-05-15T13:34:46.429 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.3.log 2024-05-15T13:34:46.429 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.0.log 2024-05-15T13:34:46.429 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-mon.a.log 2024-05-15T13:34:46.429 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.log 2024-05-15T13:34:46.429 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.1.log 2024-05-15T13:34:46.431 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.6.log 2024-05-15T13:34:46.431 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-volume.log 2024-05-15T13:34:46.431 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.audit.log 2024-05-15T13:34:46.431 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.6.log: /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.cephadm.log 2024-05-15T13:34:46.431 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.7.log 2024-05-15T13:34:46.432 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.audit.log: /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-mgr.x.log 2024-05-15T13:34:46.432 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-mon.b.log 2024-05-15T13:34:46.432 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.log 2024-05-15T13:34:46.432 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.4.log 2024-05-15T13:34:46.432 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.9.log: /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.log: 95.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-15T13:34:46.433 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.5.log 2024-05-15T13:34:46.437 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.log: /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.1.log: 85.8% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.log.gz 2024-05-15T13:34:46.451 INFO:teuthology.orchestra.run.smithi027.stderr: 91.4% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-volume.log.gz 2024-05-15T13:34:46.535 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.4.log: /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.5.log: 93.3% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-mgr.x.log.gz 2024-05-15T13:34:46.584 INFO:teuthology.orchestra.run.smithi027.stderr: 82.6% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.cephadm.log.gz 2024-05-15T13:34:46.598 INFO:teuthology.orchestra.run.smithi027.stderr: 89.7% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.audit.log.gz 2024-05-15T13:34:46.700 INFO:teuthology.orchestra.run.smithi177.stderr: 82.3% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.cephadm.log.gz 2024-05-15T13:34:46.786 INFO:teuthology.orchestra.run.smithi177.stderr: 89.8% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.audit.log.gz 2024-05-15T13:34:46.966 INFO:teuthology.orchestra.run.smithi177.stderr: 85.9% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.log.gz 2024-05-15T13:34:47.016 INFO:teuthology.orchestra.run.smithi177.stderr: 91.4% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-volume.log.gz 2024-05-15T13:34:47.608 INFO:teuthology.orchestra.run.smithi120.stderr: 91.4% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-volume.log.gz 2024-05-15T13:34:47.823 INFO:teuthology.orchestra.run.smithi120.stderr: 95.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-15T13:34:47.980 INFO:teuthology.orchestra.run.smithi027.stderr: 95.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-15T13:34:48.082 INFO:teuthology.orchestra.run.smithi177.stderr: 91.3% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-mon.c.log.gz 2024-05-15T13:34:48.214 INFO:teuthology.orchestra.run.smithi027.stderr: 91.1% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-mgr.y.log.gz 2024-05-15T13:34:48.703 INFO:teuthology.orchestra.run.smithi120.stderr: 82.4% 85.8% 89.7% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.log.gz -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.cephadm.log.gz -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph.audit.log.gz 2024-05-15T13:34:48.704 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-05-15T13:34:48.704 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-05-15T13:34:51.469 INFO:teuthology.orchestra.run.smithi120.stderr: 91.1% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-mon.b.log.gz 2024-05-15T13:35:01.309 INFO:teuthology.orchestra.run.smithi027.stderr: 90.0% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-mon.a.log.gz 2024-05-15T13:37:16.294 INFO:teuthology.orchestra.run.smithi177.stderr: 94.0% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.10.log.gz 2024-05-15T13:37:48.203 INFO:teuthology.orchestra.run.smithi177.stderr: 93.9% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.9.log.gz 2024-05-15T13:37:49.631 INFO:teuthology.orchestra.run.smithi177.stderr: 93.0% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.8.log.gz 2024-05-15T13:38:16.200 INFO:teuthology.orchestra.run.smithi177.stderr: 92.8% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.11.log.gz 2024-05-15T13:38:16.202 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-05-15T13:38:16.202 INFO:teuthology.orchestra.run.smithi177.stderr:real 3m29.813s 2024-05-15T13:38:16.202 INFO:teuthology.orchestra.run.smithi177.stderr:user 5m2.187s 2024-05-15T13:38:16.202 INFO:teuthology.orchestra.run.smithi177.stderr:sys 0m13.072s 2024-05-15T13:39:14.644 INFO:teuthology.orchestra.run.smithi120.stderr: 93.1% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.4.log.gz 2024-05-15T13:41:27.987 INFO:teuthology.orchestra.run.smithi027.stderr: 92.1% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.2.log.gz 2024-05-15T13:42:08.530 INFO:teuthology.orchestra.run.smithi120.stderr: 90.7% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.7.log.gz 2024-05-15T13:43:16.533 INFO:teuthology.orchestra.run.smithi120.stderr: 91.1% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.6.log.gz 2024-05-15T13:44:10.815 INFO:teuthology.orchestra.run.smithi027.stderr: 91.9% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.3.log.gz 2024-05-15T13:44:38.381 INFO:teuthology.orchestra.run.smithi027.stderr: 91.6% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.0.log.gz 2024-05-15T13:44:47.654 INFO:teuthology.orchestra.run.smithi027.stderr: 91.5% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.1.log.gz 2024-05-15T13:44:47.656 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-05-15T13:44:47.657 INFO:teuthology.orchestra.run.smithi027.stderr:real 10m1.270s 2024-05-15T13:44:47.657 INFO:teuthology.orchestra.run.smithi027.stderr:user 13m16.373s 2024-05-15T13:44:47.657 INFO:teuthology.orchestra.run.smithi027.stderr:sys 0m28.477s 2024-05-15T13:44:49.151 INFO:teuthology.orchestra.run.smithi120.stderr: 91.6% -- replaced with /var/log/ceph/deaea04e-12b7-11ef-bc98-c7b262605968/ceph-osd.5.log.gz 2024-05-15T13:44:49.153 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-05-15T13:44:49.153 INFO:teuthology.orchestra.run.smithi120.stderr:real 10m2.762s 2024-05-15T13:44:49.154 INFO:teuthology.orchestra.run.smithi120.stderr:user 11m39.255s 2024-05-15T13:44:49.154 INFO:teuthology.orchestra.run.smithi120.stderr:sys 0m27.975s 2024-05-15T13:44:49.154 INFO:tasks.cephadm:Archiving logs... 2024-05-15T13:44:49.154 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/log/ceph to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707109/remote/smithi027/log 2024-05-15T13:44:49.155 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-15T13:47:12.454 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/log/ceph to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707109/remote/smithi120/log 2024-05-15T13:47:12.455 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-15T13:49:29.895 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/log/ceph to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707109/remote/smithi177/log 2024-05-15T13:49:29.896 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-15T13:50:19.971 INFO:tasks.cephadm:Removing cluster... 2024-05-15T13:50:19.972 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid deaea04e-12b7-11ef-bc98-c7b262605968 --force 2024-05-15T13:50:20.318 INFO:teuthology.orchestra.run.smithi027.stdout:Deleting cluster with fsid: deaea04e-12b7-11ef-bc98-c7b262605968 2024-05-15T13:50:21.349 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid deaea04e-12b7-11ef-bc98-c7b262605968 --force 2024-05-15T13:50:21.870 INFO:teuthology.orchestra.run.smithi120.stdout:Deleting cluster with fsid: deaea04e-12b7-11ef-bc98-c7b262605968 2024-05-15T13:50:22.876 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid deaea04e-12b7-11ef-bc98-c7b262605968 --force 2024-05-15T13:50:23.457 INFO:teuthology.orchestra.run.smithi177.stdout:Deleting cluster with fsid: deaea04e-12b7-11ef-bc98-c7b262605968 2024-05-15T13:50:24.079 INFO:tasks.cephadm:Removing cephadm ... 2024-05-15T13:50:24.080 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-15T13:50:24.120 DEBUG:teuthology.orchestra.run.smithi120:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-15T13:50:24.208 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-15T13:50:24.236 INFO:tasks.cephadm:Teardown complete 2024-05-15T13:50:24.237 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-15T13:50:24.246 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-15T13:50:24.246 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-15T13:50:24.249 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-15T13:50:24.251 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-15T13:50:24.394 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-15T13:50:24.403 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-15T13:50:24.403 DEBUG:teuthology.orchestra.run.smithi027:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T13:50:24.406 DEBUG:teuthology.orchestra.run.smithi120:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T13:50:24.408 DEBUG:teuthology.orchestra.run.smithi177:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T13:50:24.424 INFO:teuthology.orchestra.run.smithi027.stderr:bash: ntpq: command not found 2024-05-15T13:50:24.425 INFO:teuthology.orchestra.run.smithi177.stderr:bash: ntpq: command not found 2024-05-15T13:50:24.426 INFO:teuthology.orchestra.run.smithi120.stderr:bash: ntpq: command not found 2024-05-15T13:50:24.449 INFO:teuthology.orchestra.run.smithi120.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-15T13:50:24.449 INFO:teuthology.orchestra.run.smithi120.stdout:=============================================================================== 2024-05-15T13:50:24.450 INFO:teuthology.orchestra.run.smithi120.stdout:^* hv01.front.sepia.ceph.com 3 6 377 42 +1034us[+1066us] +/- 55ms 2024-05-15T13:50:24.450 INFO:teuthology.orchestra.run.smithi120.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 45 -861us[ -829us] +/- 60ms 2024-05-15T13:50:24.450 INFO:teuthology.orchestra.run.smithi120.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 -262us[ -230us] +/- 66ms 2024-05-15T13:50:24.450 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T13:50:24.450 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-15T13:50:24.450 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-05-15T13:50:24.450 INFO:teuthology.orchestra.run.smithi177.stdout:^* hv01.front.sepia.ceph.com 3 6 377 44 +1029us[+1021us] +/- 55ms 2024-05-15T13:50:24.450 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 43 -845us[ -845us] +/- 60ms 2024-05-15T13:50:24.450 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 -262us[ -270us] +/- 66ms 2024-05-15T13:50:24.451 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T13:50:24.472 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-15T13:50:24.472 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-05-15T13:50:24.472 INFO:teuthology.orchestra.run.smithi027.stdout:^* hv01.front.sepia.ceph.com 3 6 377 45 +1044us[+1053us] +/- 55ms 2024-05-15T13:50:24.472 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 49 -812us[ -802us] +/- 60ms 2024-05-15T13:50:24.472 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 -216us[ -216us] +/- 66ms 2024-05-15T13:50:24.472 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T13:50:24.473 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-15T13:50:24.482 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-15T13:50:24.483 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-15T13:50:24.491 DEBUG:teuthology.orchestra.run.smithi027:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-15T13:50:24.631 DEBUG:teuthology.orchestra.run.smithi120:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-15T13:50:24.856 DEBUG:teuthology.orchestra.run.smithi177:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-15T13:50:25.063 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-15T13:50:25.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T13:50:25.099 DEBUG:teuthology.orchestra.run.smithi120:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-15T13:50:25.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T13:50:25.139 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-15T13:50:25.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T13:50:25.174 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-15T13:50:25.184 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-15T13:50:25.192 INFO:teuthology.task.internal:Duration was 4943.390892 seconds 2024-05-15T13:50:25.193 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-15T13:50:25.201 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-15T13:50:25.201 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-15T13:50:25.204 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-15T13:50:25.207 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-15T13:50:25.286 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-15T13:50:25.302 INFO:teuthology.orchestra.run.smithi120.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-15T13:50:25.338 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-15T13:50:25.577 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-15T13:50:25.577 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi027.front.sepia.ceph.com 2024-05-15T13:50:25.577 DEBUG:teuthology.orchestra.run.smithi027:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-15T13:50:25.609 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi120.front.sepia.ceph.com 2024-05-15T13:50:25.609 DEBUG:teuthology.orchestra.run.smithi120:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-15T13:50:25.645 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi177.front.sepia.ceph.com 2024-05-15T13:50:25.645 DEBUG:teuthology.orchestra.run.smithi177:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-15T13:50:25.680 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-15T13:50:25.680 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-15T13:50:25.683 DEBUG:teuthology.orchestra.run.smithi120:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-15T13:50:25.687 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-15T13:50:25.841 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-15T13:50:25.841 DEBUG:teuthology.orchestra.run.smithi027:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-15T13:50:25.843 DEBUG:teuthology.orchestra.run.smithi120:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-15T13:50:25.846 DEBUG:teuthology.orchestra.run.smithi177:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-15T13:50:26.179 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-15T13:50:26.192 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-15T13:50:26.192 DEBUG:teuthology.orchestra.run.smithi027:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-15T13:50:26.244 DEBUG:teuthology.orchestra.run.smithi120:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-15T13:50:26.296 DEBUG:teuthology.orchestra.run.smithi177:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-15T13:50:26.353 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-15T13:50:26.364 DEBUG:teuthology.orchestra.run.smithi027:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:50:26.366 DEBUG:teuthology.orchestra.run.smithi120:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:50:26.369 DEBUG:teuthology.orchestra.run.smithi177:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:50:26.401 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = core 2024-05-15T13:50:26.403 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = core 2024-05-15T13:50:26.407 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = core 2024-05-15T13:50:26.421 DEBUG:teuthology.orchestra.run.smithi027:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:50:26.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T13:50:26.479 DEBUG:teuthology.orchestra.run.smithi120:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:50:26.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T13:50:26.497 DEBUG:teuthology.orchestra.run.smithi177:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:50:26.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T13:50:26.514 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-15T13:50:26.528 INFO:teuthology.task.internal:Transferring archived files... 2024-05-15T13:50:26.529 DEBUG:teuthology.misc:Transferring archived files from smithi027:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707109/remote/smithi027 2024-05-15T13:50:26.530 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-15T13:50:26.597 DEBUG:teuthology.misc:Transferring archived files from smithi120:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707109/remote/smithi120 2024-05-15T13:50:26.597 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-15T13:50:26.647 DEBUG:teuthology.misc:Transferring archived files from smithi177:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707109/remote/smithi177 2024-05-15T13:50:26.647 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-15T13:50:26.698 INFO:teuthology.task.internal:Removing archive directory... 2024-05-15T13:50:26.698 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-15T13:50:26.701 DEBUG:teuthology.orchestra.run.smithi120:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-15T13:50:26.703 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-15T13:50:26.746 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-15T13:50:26.757 INFO:teuthology.task.internal:Not uploading archives. 2024-05-15T13:50:26.757 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-15T13:50:26.836 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-15T13:50:26.836 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-15T13:50:26.839 DEBUG:teuthology.orchestra.run.smithi120:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-15T13:50:26.841 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-15T13:50:26.854 INFO:teuthology.orchestra.run.smithi027.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 15 13:50 /home/ubuntu/cephtest 2024-05-15T13:50:26.856 INFO:teuthology.orchestra.run.smithi120.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 15 13:50 /home/ubuntu/cephtest 2024-05-15T13:50:26.858 INFO:teuthology.orchestra.run.smithi177.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 15 13:50 /home/ubuntu/cephtest 2024-05-15T13:50:26.860 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-15T13:50:26.870 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-15T13:50:26.890 INFO:teuthology.nuke:Checking targets against current locks 2024-05-15T13:50:26.917 DEBUG:teuthology.nuke:shortname: smithi027 2024-05-15T13:50:26.917 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-15T13:50:26.943 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707109', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-15 12:17:44.501041', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIVYUlhD9dqWRehm4Q46t1Kk7sxw0EQbzqqTnMx+36BFfa6KGUui96G6YBRSKXjGnhrX7oJQFjQaiPcv1Fvg01s='} 2024-05-15T13:50:26.944 DEBUG:teuthology.nuke:shortname: smithi120 2024-05-15T13:50:26.944 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-15T13:50:26.967 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi120.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707109', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-15 12:17:44.497013', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFuIH7kElwDHODlT/U2ii20ssku7k+Dlhp7Zk81sm9NOThO8WXeOC9rd+th2GofnTY+X4Guzw70dftyZK/zqeZ4='} 2024-05-15T13:50:26.972 DEBUG:teuthology.nuke:shortname: smithi177 2024-05-15T13:50:26.973 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-15T13:50:27.002 INFO:teuthology.orchestra.console.smithi027:Power off 2024-05-15T13:50:27.002 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-15T13:50:27.013 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707109', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-15 12:17:44.499480', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJSPjjHqUwsGcplWj9kv5whuGv6DCIPW7AoVbpKVSJfWctMFTZLktyilQaz8si1tBzwk1wAcWxORKGVom4prL+g='} 2024-05-15T13:50:27.028 DEBUG:teuthology.orchestra.console.smithi027:power off output: Chassis Power Control: Down/Off 2024-05-15T13:50:27.028 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:50:27.048 INFO:teuthology.orchestra.console.smithi120:Power off 2024-05-15T13:50:27.048 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-15T13:50:27.059 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2024-05-15T13:50:27.072 DEBUG:teuthology.orchestra.console.smithi120:power off output: Chassis Power Control: Down/Off 2024-05-15T13:50:27.072 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:50:27.089 INFO:teuthology.orchestra.console.smithi177:Power off 2024-05-15T13:50:27.090 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-15T13:50:27.098 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is on 2024-05-15T13:50:27.112 DEBUG:teuthology.orchestra.console.smithi177:power off output: Chassis Power Control: Down/Off 2024-05-15T13:50:27.113 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:50:27.135 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-05-15T13:50:31.060 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:50:31.100 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:50:31.137 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:50:31.170 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2024-05-15T13:50:31.215 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is on 2024-05-15T13:50:31.247 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-05-15T13:50:35.175 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:50:35.216 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:50:35.247 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:50:35.285 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2024-05-15T13:50:35.327 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is on 2024-05-15T13:50:35.357 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-05-15T13:50:39.290 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:50:39.329 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:50:39.358 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:50:39.400 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is off 2024-05-15T13:50:39.439 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is off 2024-05-15T13:50:39.467 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off 2024-05-15T13:50:39.502 INFO:teuthology.orchestra.console.smithi027:Power off completed 2024-05-15T13:50:39.540 INFO:teuthology.orchestra.console.smithi120:Power off completed 2024-05-15T13:50:39.569 INFO:teuthology.orchestra.console.smithi177:Power off completed 2024-05-15T13:50:39.674 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} duration: 4943.390891551971 failure_reason: '"2024-05-15T12:50:00.000151+0000 mon.a (mon.0) 1068 : cluster 3 [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn''t resolve itself): 1 pg backfill_toofull" in cluster log' flavor: default owner: scheduled_rzarzynski@teuthology success: false 2024-05-15T13:50:39.674 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-15T13:50:39.769 INFO:teuthology.run:FAIL